2024-03-19T02:12:09.136 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-19T02:12:09.137 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609408 --owner scheduled_pdonnell@teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} --name pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi --verbose -- /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609408/orig.config.yaml 2024-03-19T02:12:09.195 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-19T02:12:09.285 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609408 branch: wip-batrick-testing-20240318.181317 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} email: pdonnell@redhat.com first_in_suite: false job_id: '7609408' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240318.181317 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - PG_AVAILABILITY - PG_DEGRADED - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount dry_run: true install: ceph: flavor: default sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 kclient: client.0: mount_subvol_num: 1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240318.181317 sha1: e48b1f6c0608b2615e1890cc46286c3877d6e910 owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 3005 sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 sleep_before_teardown: 0 subset: 433/512 suite: fs suite_branch: wip-batrick-testing-20240318.181317 suite_path: /home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph suite_sha1: e48b1f6c0608b2615e1890cc46286c3877d6e910 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi033.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJzq0qnp859Um6mXLmhNb1b49Ng59y4fRCIOrHnHTHcIf6mgk4lvvvbIbPq1uSBQAInIpaTsdJROOs1V07ujsYU= smithi096.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIhib3wbRbcPlEUnyWqrOG0jfwV6I0Xc3kK/vHWmavG4Dl7nQrkYBBGps05H+an/B7xEk8l0mXnuRd6TmMCyLRw= smithi155.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAmfURHQb3AafSy0yHeZWbFutUFquk1+c3wDFUcl75szpwFM2Z41eARRdgpOZE0tdaRbVRlcoWVVw6f0cQs/SAQ= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - ceph-fuse: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - workunit: clients: all: - suites/iozone.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-18_21:05:49 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-19T02:12:09.285 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa; will attempt to use it 2024-03-19T02:12:09.286 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks 2024-03-19T02:12:09.286 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-19T02:12:09.287 INFO:teuthology.task.internal:Checking packages... 2024-03-19T02:12:09.310 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'b6ae24918b03b1b8a19c9263a50655bba80bb810' 2024-03-19T02:12:09.311 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-19T02:12:09.311 INFO:teuthology.packaging:ref: None 2024-03-19T02:12:09.311 INFO:teuthology.packaging:tag: None 2024-03-19T02:12:09.311 INFO:teuthology.packaging:branch: wip-batrick-testing-20240318.181317 2024-03-19T02:12:09.311 INFO:teuthology.packaging:sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:12:09.311 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240318.181317 2024-03-19T02:12:09.454 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2206.gb6ae2491 2024-03-19T02:12:09.455 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-19T02:12:09.467 INFO:teuthology.task.internal:no buildpackages task found 2024-03-19T02:12:09.467 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-19T02:12:09.479 INFO:teuthology.task.internal:Saving configuration 2024-03-19T02:12:09.500 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-19T02:12:09.507 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-19T02:12:09.528 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi033.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609408', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 02:07:00.733364', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJzq0qnp859Um6mXLmhNb1b49Ng59y4fRCIOrHnHTHcIf6mgk4lvvvbIbPq1uSBQAInIpaTsdJROOs1V07ujsYU='} 2024-03-19T02:12:09.549 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi096.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609408', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 02:07:00.736686', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIhib3wbRbcPlEUnyWqrOG0jfwV6I0Xc3kK/vHWmavG4Dl7nQrkYBBGps05H+an/B7xEk8l0mXnuRd6TmMCyLRw='} 2024-03-19T02:12:09.571 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi155.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609408', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 02:07:00.735120', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAmfURHQb3AafSy0yHeZWbFutUFquk1+c3wDFUcl75szpwFM2Z41eARRdgpOZE0tdaRbVRlcoWVVw6f0cQs/SAQ='} 2024-03-19T02:12:09.571 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-19T02:12:09.579 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-19T02:12:09.579 INFO:teuthology.task.internal:roles: ubuntu@smithi096.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-19T02:12:09.580 INFO:teuthology.task.internal:roles: ubuntu@smithi155.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-19T02:12:09.580 INFO:teuthology.run_tasks:Running task console_log... 2024-03-19T02:12:09.701 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f44a355d160>, signals=[15]) 2024-03-19T02:12:09.701 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-19T02:12:09.708 INFO:teuthology.task.internal:Opening connections... 2024-03-19T02:12:09.708 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi033.front.sepia.ceph.com 2024-03-19T02:12:09.710 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:12:09.818 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi096.front.sepia.ceph.com 2024-03-19T02:12:09.819 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:12:09.909 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi155.front.sepia.ceph.com 2024-03-19T02:12:09.910 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:12:09.999 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-19T02:12:10.005 DEBUG:teuthology.orchestra.run.smithi033:> uname -m 2024-03-19T02:12:10.024 INFO:teuthology.orchestra.run.smithi033.stdout:x86_64 2024-03-19T02:12:10.024 DEBUG:teuthology.orchestra.run.smithi033:> cat /etc/os-release 2024-03-19T02:12:10.082 INFO:teuthology.orchestra.run.smithi033.stdout:NAME="CentOS Stream" 2024-03-19T02:12:10.083 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION="9" 2024-03-19T02:12:10.083 INFO:teuthology.orchestra.run.smithi033.stdout:ID="centos" 2024-03-19T02:12:10.083 INFO:teuthology.orchestra.run.smithi033.stdout:ID_LIKE="rhel fedora" 2024-03-19T02:12:10.083 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_ID="9" 2024-03-19T02:12:10.083 INFO:teuthology.orchestra.run.smithi033.stdout:PLATFORM_ID="platform:el9" 2024-03-19T02:12:10.083 INFO:teuthology.orchestra.run.smithi033.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-19T02:12:10.083 INFO:teuthology.orchestra.run.smithi033.stdout:ANSI_COLOR="0;31" 2024-03-19T02:12:10.083 INFO:teuthology.orchestra.run.smithi033.stdout:LOGO="fedora-logo-icon" 2024-03-19T02:12:10.083 INFO:teuthology.orchestra.run.smithi033.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-19T02:12:10.083 INFO:teuthology.orchestra.run.smithi033.stdout:HOME_URL="https://centos.org/" 2024-03-19T02:12:10.083 INFO:teuthology.orchestra.run.smithi033.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-19T02:12:10.083 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-19T02:12:10.084 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-19T02:12:10.084 INFO:teuthology.lock.ops:Updating smithi033.front.sepia.ceph.com on lock server 2024-03-19T02:12:10.103 DEBUG:teuthology.orchestra.run.smithi096:> uname -m 2024-03-19T02:12:10.120 INFO:teuthology.orchestra.run.smithi096.stdout:x86_64 2024-03-19T02:12:10.121 DEBUG:teuthology.orchestra.run.smithi096:> cat /etc/os-release 2024-03-19T02:12:10.177 INFO:teuthology.orchestra.run.smithi096.stdout:NAME="CentOS Stream" 2024-03-19T02:12:10.177 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION="9" 2024-03-19T02:12:10.177 INFO:teuthology.orchestra.run.smithi096.stdout:ID="centos" 2024-03-19T02:12:10.177 INFO:teuthology.orchestra.run.smithi096.stdout:ID_LIKE="rhel fedora" 2024-03-19T02:12:10.177 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION_ID="9" 2024-03-19T02:12:10.177 INFO:teuthology.orchestra.run.smithi096.stdout:PLATFORM_ID="platform:el9" 2024-03-19T02:12:10.177 INFO:teuthology.orchestra.run.smithi096.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-19T02:12:10.178 INFO:teuthology.orchestra.run.smithi096.stdout:ANSI_COLOR="0;31" 2024-03-19T02:12:10.178 INFO:teuthology.orchestra.run.smithi096.stdout:LOGO="fedora-logo-icon" 2024-03-19T02:12:10.178 INFO:teuthology.orchestra.run.smithi096.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-19T02:12:10.178 INFO:teuthology.orchestra.run.smithi096.stdout:HOME_URL="https://centos.org/" 2024-03-19T02:12:10.178 INFO:teuthology.orchestra.run.smithi096.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-19T02:12:10.178 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-19T02:12:10.178 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-19T02:12:10.178 INFO:teuthology.lock.ops:Updating smithi096.front.sepia.ceph.com on lock server 2024-03-19T02:12:10.204 DEBUG:teuthology.orchestra.run.smithi155:> uname -m 2024-03-19T02:12:10.222 INFO:teuthology.orchestra.run.smithi155.stdout:x86_64 2024-03-19T02:12:10.222 DEBUG:teuthology.orchestra.run.smithi155:> cat /etc/os-release 2024-03-19T02:12:10.280 INFO:teuthology.orchestra.run.smithi155.stdout:NAME="CentOS Stream" 2024-03-19T02:12:10.281 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION="9" 2024-03-19T02:12:10.281 INFO:teuthology.orchestra.run.smithi155.stdout:ID="centos" 2024-03-19T02:12:10.281 INFO:teuthology.orchestra.run.smithi155.stdout:ID_LIKE="rhel fedora" 2024-03-19T02:12:10.281 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_ID="9" 2024-03-19T02:12:10.281 INFO:teuthology.orchestra.run.smithi155.stdout:PLATFORM_ID="platform:el9" 2024-03-19T02:12:10.281 INFO:teuthology.orchestra.run.smithi155.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-19T02:12:10.281 INFO:teuthology.orchestra.run.smithi155.stdout:ANSI_COLOR="0;31" 2024-03-19T02:12:10.281 INFO:teuthology.orchestra.run.smithi155.stdout:LOGO="fedora-logo-icon" 2024-03-19T02:12:10.281 INFO:teuthology.orchestra.run.smithi155.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-19T02:12:10.281 INFO:teuthology.orchestra.run.smithi155.stdout:HOME_URL="https://centos.org/" 2024-03-19T02:12:10.281 INFO:teuthology.orchestra.run.smithi155.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-19T02:12:10.281 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-19T02:12:10.281 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-19T02:12:10.282 INFO:teuthology.lock.ops:Updating smithi155.front.sepia.ceph.com on lock server 2024-03-19T02:12:10.303 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-19T02:12:10.326 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-19T02:12:10.331 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-19T02:12:10.332 DEBUG:teuthology.orchestra.run.smithi033:> test '!' -e /home/ubuntu/cephtest 2024-03-19T02:12:10.334 DEBUG:teuthology.orchestra.run.smithi096:> test '!' -e /home/ubuntu/cephtest 2024-03-19T02:12:10.336 DEBUG:teuthology.orchestra.run.smithi155:> test '!' -e /home/ubuntu/cephtest 2024-03-19T02:12:10.352 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-19T02:12:10.358 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-19T02:12:10.358 DEBUG:teuthology.orchestra.run.smithi033:> test -z $(ls -A /var/lib/ceph) 2024-03-19T02:12:10.391 DEBUG:teuthology.orchestra.run.smithi096:> test -z $(ls -A /var/lib/ceph) 2024-03-19T02:12:10.393 DEBUG:teuthology.orchestra.run.smithi155:> test -z $(ls -A /var/lib/ceph) 2024-03-19T02:12:10.429 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-19T02:12:10.511 INFO:teuthology.run_tasks:Running task kernel... 2024-03-19T02:12:10.526 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-19T02:12:10.526 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-19T02:12:10.527 DEBUG:teuthology.orchestra.run.smithi033:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-19T02:12:10.527 DEBUG:teuthology.orchestra.run.smithi096:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-19T02:12:10.528 DEBUG:teuthology.orchestra.run.smithi155:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-19T02:12:10.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T02:12:10.543 DEBUG:teuthology.orchestra.run.smithi096:> uname -r 2024-03-19T02:12:10.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T02:12:10.544 DEBUG:teuthology.orchestra.run.smithi155:> uname -r 2024-03-19T02:12:10.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T02:12:10.546 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2024-03-19T02:12:10.600 INFO:teuthology.orchestra.run.smithi096.stdout:5.14.0-361.el9.x86_64 2024-03-19T02:12:10.600 INFO:teuthology.task.kernel:Running kernel on smithi096: 5.14.0-361.el9.x86_64 2024-03-19T02:12:10.601 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum install -y kernel 2024-03-19T02:12:10.602 INFO:teuthology.orchestra.run.smithi155.stdout:5.14.0-361.el9.x86_64 2024-03-19T02:12:10.602 INFO:teuthology.task.kernel:Running kernel on smithi155: 5.14.0-361.el9.x86_64 2024-03-19T02:12:10.602 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum install -y kernel 2024-03-19T02:12:10.603 INFO:teuthology.orchestra.run.smithi033.stdout:5.14.0-361.el9.x86_64 2024-03-19T02:12:10.603 INFO:teuthology.task.kernel:Running kernel on smithi033: 5.14.0-361.el9.x86_64 2024-03-19T02:12:10.604 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum install -y kernel 2024-03-19T02:12:12.982 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS-9 - AppStream 8.8 MB/s | 6.7 MB 00:00 2024-03-19T02:12:13.255 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS-9 - AppStream 7.0 MB/s | 6.7 MB 00:00 2024-03-19T02:12:13.331 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS-9 - AppStream 6.6 MB/s | 6.7 MB 00:01 2024-03-19T02:12:14.781 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS-9 - BaseOS 2.9 MB/s | 2.0 MB 00:00 2024-03-19T02:12:14.964 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-19T02:12:14.970 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-03-19T02:12:15.763 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-03-19T02:12:15.846 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-19T02:12:20.880 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - AppStream 4.7 MB/s | 19 MB 00:04 2024-03-19T02:12:24.798 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-19T02:12:24.838 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - AppStream 2.3 MB/s | 19 MB 00:08 2024-03-19T02:12:26.248 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - Extras packages 47 kB/s | 16 kB 00:00 2024-03-19T02:12:27.910 INFO:teuthology.orchestra.run.smithi096.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-19T02:12:30.547 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 6.1 MB 00:02 2024-03-19T02:12:32.064 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 16 kB 00:00 2024-03-19T02:12:32.857 INFO:teuthology.orchestra.run.smithi155.stdout:Extra Packages for Enterprise Linux 30 MB/s | 21 MB 00:00 2024-03-19T02:12:32.858 INFO:teuthology.orchestra.run.smithi096.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-19T02:12:35.022 INFO:teuthology.orchestra.run.smithi096.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-19T02:12:35.022 INFO:teuthology.orchestra.run.smithi096.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-19T02:12:35.121 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-19T02:12:35.133 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-19T02:12:35.133 INFO:teuthology.orchestra.run.smithi096.stdout: Package Architecture Version Repository Size 2024-03-19T02:12:35.133 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-19T02:12:35.133 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-19T02:12:35.133 INFO:teuthology.orchestra.run.smithi096.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-19T02:12:35.133 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-03-19T02:12:35.133 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-19T02:12:35.133 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-19T02:12:35.133 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-19T02:12:35.133 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-19T02:12:35.134 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-19T02:12:35.134 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-03-19T02:12:35.134 INFO:teuthology.orchestra.run.smithi096.stdout:Install 4 Packages 2024-03-19T02:12:35.134 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-19T02:12:35.134 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 76 M 2024-03-19T02:12:35.134 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 123 M 2024-03-19T02:12:35.134 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-19T02:12:37.724 INFO:teuthology.orchestra.run.smithi155.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-19T02:12:39.839 INFO:teuthology.orchestra.run.smithi155.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-19T02:12:39.839 INFO:teuthology.orchestra.run.smithi155.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-19T02:12:39.932 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-19T02:12:39.936 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-19T02:12:39.936 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-03-19T02:12:39.936 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-19T02:12:39.936 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-19T02:12:39.936 INFO:teuthology.orchestra.run.smithi155.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-19T02:12:39.936 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-19T02:12:39.936 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-19T02:12:39.936 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-19T02:12:39.936 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-19T02:12:39.937 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-19T02:12:39.937 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-19T02:12:39.937 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-03-19T02:12:39.937 INFO:teuthology.orchestra.run.smithi155.stdout:Install 4 Packages 2024-03-19T02:12:39.937 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-19T02:12:39.937 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 76 M 2024-03-19T02:12:39.937 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 123 M 2024-03-19T02:12:39.937 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-19T02:14:46.211 INFO:teuthology.orchestra.run.smithi096.stdout:[MIRROR] kernel-5.14.0-430.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-430.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T02:14:46.211 INFO:teuthology.orchestra.run.smithi096.stdout:[MIRROR] kernel-core-5.14.0-430.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-430.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T02:14:46.212 INFO:teuthology.orchestra.run.smithi096.stdout:[MIRROR] kernel-modules-5.14.0-430.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-430.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T02:14:46.573 INFO:teuthology.orchestra.run.smithi096.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 347 B/s | 44 kB 02:11 2024-03-19T02:14:51.848 INFO:teuthology.orchestra.run.smithi155.stdout:[MIRROR] kernel-5.14.0-430.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-430.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T02:14:51.848 INFO:teuthology.orchestra.run.smithi155.stdout:[MIRROR] kernel-core-5.14.0-430.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-430.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T02:14:51.848 INFO:teuthology.orchestra.run.smithi155.stdout:[MIRROR] kernel-modules-5.14.0-430.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-430.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T02:14:52.199 INFO:teuthology.orchestra.run.smithi155.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 344 B/s | 44 kB 02:12 2024-03-19T02:14:57.672 INFO:teuthology.orchestra.run.smithi096.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 107 kB/s | 15 MB 02:22 2024-03-19T02:15:03.864 INFO:teuthology.orchestra.run.smithi155.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 106 kB/s | 15 MB 02:23 2024-03-19T02:15:08.845 INFO:teuthology.orchestra.run.smithi096.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 1.2 MB/s | 27 MB 00:22 2024-03-19T02:15:13.103 INFO:teuthology.orchestra.run.smithi096.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 217 kB/s | 33 MB 02:37 2024-03-19T02:15:13.103 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:15:13.103 INFO:teuthology.orchestra.run.smithi096.stdout:Total 492 kB/s | 76 MB 02:37 2024-03-19T02:15:13.766 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-19T02:15:14.140 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-19T02:15:14.140 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-19T02:15:15.021 INFO:teuthology.orchestra.run.smithi155.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 1.2 MB/s | 27 MB 00:22 2024-03-19T02:15:15.143 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-19T02:15:15.143 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-19T02:15:17.608 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-19T02:15:18.214 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-19T02:15:18.304 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-19T02:15:18.763 INFO:teuthology.orchestra.run.smithi155.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 216 kB/s | 33 MB 02:38 2024-03-19T02:15:18.763 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:15:18.763 INFO:teuthology.orchestra.run.smithi155.stdout:Total 489 kB/s | 76 MB 02:38 2024-03-19T02:15:19.420 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-19T02:15:19.824 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-19T02:15:19.824 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-19T02:15:20.830 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-19T02:15:20.830 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-19T02:15:21.061 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-19T02:15:21.205 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-19T02:15:23.322 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-19T02:15:23.913 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-19T02:15:24.012 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-19T02:15:25.917 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-19T02:15:25.954 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-19T02:15:26.805 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-19T02:15:26.947 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-19T02:15:30.675 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-19T02:15:31.622 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-19T02:15:31.738 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-19T02:15:36.470 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-19T02:16:00.088 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-19T02:16:00.089 INFO:teuthology.orchestra.run.smithi096.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-19T02:16:00.089 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-19T02:16:00.132 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-19T02:16:00.473 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-19T02:16:00.473 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-19T02:16:00.473 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-19T02:16:00.473 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-19T02:16:00.936 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-19T02:16:00.936 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-19T02:16:00.936 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-19T02:16:00.936 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-19T02:16:00.936 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-19T02:16:00.936 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-19T02:16:00.937 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-19T02:16:00.937 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-19T02:16:00.937 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-19T02:16:01.296 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q kernel | sort -rV | head -n 1 2024-03-19T02:16:01.443 INFO:teuthology.orchestra.run.smithi096.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-19T02:16:01.444 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-19T02:16:01.444 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-19T02:16:01.444 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-19T02:16:01.444 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-19T02:16:01.444 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-19T02:16:01.444 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi096.front.sepia.ceph.com, path=None, version=distro) 2024-03-19T02:16:01.444 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q kernel | sort -rV | head -n 1 2024-03-19T02:16:01.524 INFO:teuthology.orchestra.run.smithi096.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-19T02:16:01.524 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-19T02:16:01.524 DEBUG:teuthology.orchestra.run.smithi096:> sudo rpm -qi grub2-tools 2024-03-19T02:16:01.558 INFO:teuthology.orchestra.run.smithi096.stdout:Name : grub2-tools 2024-03-19T02:16:01.558 INFO:teuthology.orchestra.run.smithi096.stdout:Epoch : 1 2024-03-19T02:16:01.558 INFO:teuthology.orchestra.run.smithi096.stdout:Version : 2.06 2024-03-19T02:16:01.558 INFO:teuthology.orchestra.run.smithi096.stdout:Release : 68.el9 2024-03-19T02:16:01.558 INFO:teuthology.orchestra.run.smithi096.stdout:Architecture: x86_64 2024-03-19T02:16:01.558 INFO:teuthology.orchestra.run.smithi096.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-19T02:16:01.558 INFO:teuthology.orchestra.run.smithi096.stdout:Group : Unspecified 2024-03-19T02:16:01.558 INFO:teuthology.orchestra.run.smithi096.stdout:Size : 8264254 2024-03-19T02:16:01.559 INFO:teuthology.orchestra.run.smithi096.stdout:License : GPLv3+ 2024-03-19T02:16:01.559 INFO:teuthology.orchestra.run.smithi096.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-19T02:16:01.559 INFO:teuthology.orchestra.run.smithi096.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-19T02:16:01.559 INFO:teuthology.orchestra.run.smithi096.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-19T02:16:01.559 INFO:teuthology.orchestra.run.smithi096.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-19T02:16:01.559 INFO:teuthology.orchestra.run.smithi096.stdout:Packager : builder@centos.org 2024-03-19T02:16:01.559 INFO:teuthology.orchestra.run.smithi096.stdout:Vendor : CentOS 2024-03-19T02:16:01.559 INFO:teuthology.orchestra.run.smithi096.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-19T02:16:01.559 INFO:teuthology.orchestra.run.smithi096.stdout:Summary : Support tools for GRUB. 2024-03-19T02:16:01.559 INFO:teuthology.orchestra.run.smithi096.stdout:Description : 2024-03-19T02:16:01.559 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-19T02:16:01.559 INFO:teuthology.orchestra.run.smithi096.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-19T02:16:01.559 INFO:teuthology.orchestra.run.smithi096.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-19T02:16:01.559 INFO:teuthology.orchestra.run.smithi096.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-19T02:16:01.559 INFO:teuthology.orchestra.run.smithi096.stdout:hardware devices. 2024-03-19T02:16:01.560 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-19T02:16:01.560 INFO:teuthology.orchestra.run.smithi096.stdout:This subpackage provides tools for support of all platforms. 2024-03-19T02:16:01.560 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-19T02:16:01.561 INFO:teuthology.task.kernel:Updating grub on smithi096 to boot 5.14.0-430.el9.x86_64 2024-03-19T02:16:01.561 DEBUG:teuthology.orchestra.run.smithi096:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-19T02:16:02.187 INFO:teuthology.orchestra.run.smithi096.stderr:Generating grub configuration file ... 2024-03-19T02:16:03.620 INFO:teuthology.orchestra.run.smithi096.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-19T02:16:03.669 INFO:teuthology.orchestra.run.smithi096.stderr:done 2024-03-19T02:16:03.672 DEBUG:teuthology.orchestra.run.smithi096:> mktemp 2024-03-19T02:16:03.687 INFO:teuthology.orchestra.run.smithi096.stdout:/tmp/tmp.zWmCRLnaqF 2024-03-19T02:16:03.687 DEBUG:teuthology.orchestra.run.smithi096:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.zWmCRLnaqF 2024-03-19T02:16:03.755 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0666 /tmp/tmp.zWmCRLnaqF 2024-03-19T02:16:03.900 DEBUG:teuthology.orchestra.remote:smithi096:/tmp/tmp.zWmCRLnaqF is 6KB 2024-03-19T02:16:03.949 DEBUG:teuthology.orchestra.run.smithi096:> rm -fr /tmp/tmp.zWmCRLnaqF 2024-03-19T02:16:03.963 DEBUG:teuthology.orchestra.run.smithi096:> sudo /bin/ls /boot/loader/entries || true 2024-03-19T02:16:04.027 INFO:teuthology.orchestra.run.smithi096.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-19T02:16:04.028 INFO:teuthology.orchestra.run.smithi096.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-19T02:16:04.028 INFO:teuthology.orchestra.run.smithi096.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-19T02:16:04.028 INFO:teuthology.orchestra.run.smithi096.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-19T02:16:04.029 DEBUG:teuthology.orchestra.run.smithi096:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-19T02:16:04.182 DEBUG:teuthology.orchestra.run.smithi096:> sudo shutdown -r now 2024-03-19T02:16:05.912 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-19T02:16:05.912 INFO:teuthology.orchestra.run.smithi155.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-19T02:16:05.912 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-19T02:16:05.955 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-19T02:16:06.358 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-19T02:16:06.358 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-19T02:16:06.358 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-19T02:16:06.358 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-19T02:16:06.788 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-19T02:16:06.788 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-19T02:16:06.788 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-19T02:16:06.788 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-19T02:16:06.789 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-19T02:16:06.789 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-19T02:16:06.789 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-19T02:16:06.789 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-19T02:16:06.789 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-19T02:16:07.145 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q kernel | sort -rV | head -n 1 2024-03-19T02:16:07.255 INFO:teuthology.orchestra.run.smithi155.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-19T02:16:07.255 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-19T02:16:07.256 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-19T02:16:07.256 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-19T02:16:07.256 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-19T02:16:07.256 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-03-19T02:16:07.256 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi155.front.sepia.ceph.com, path=None, version=distro) 2024-03-19T02:16:07.256 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q kernel | sort -rV | head -n 1 2024-03-19T02:16:07.335 INFO:teuthology.orchestra.run.smithi155.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-19T02:16:07.335 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-19T02:16:07.336 DEBUG:teuthology.orchestra.run.smithi155:> sudo rpm -qi grub2-tools 2024-03-19T02:16:07.369 INFO:teuthology.orchestra.run.smithi155.stdout:Name : grub2-tools 2024-03-19T02:16:07.370 INFO:teuthology.orchestra.run.smithi155.stdout:Epoch : 1 2024-03-19T02:16:07.370 INFO:teuthology.orchestra.run.smithi155.stdout:Version : 2.06 2024-03-19T02:16:07.370 INFO:teuthology.orchestra.run.smithi155.stdout:Release : 68.el9 2024-03-19T02:16:07.370 INFO:teuthology.orchestra.run.smithi155.stdout:Architecture: x86_64 2024-03-19T02:16:07.370 INFO:teuthology.orchestra.run.smithi155.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-19T02:16:07.370 INFO:teuthology.orchestra.run.smithi155.stdout:Group : Unspecified 2024-03-19T02:16:07.370 INFO:teuthology.orchestra.run.smithi155.stdout:Size : 8264254 2024-03-19T02:16:07.370 INFO:teuthology.orchestra.run.smithi155.stdout:License : GPLv3+ 2024-03-19T02:16:07.370 INFO:teuthology.orchestra.run.smithi155.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-19T02:16:07.370 INFO:teuthology.orchestra.run.smithi155.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-19T02:16:07.370 INFO:teuthology.orchestra.run.smithi155.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-19T02:16:07.370 INFO:teuthology.orchestra.run.smithi155.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-19T02:16:07.370 INFO:teuthology.orchestra.run.smithi155.stdout:Packager : builder@centos.org 2024-03-19T02:16:07.370 INFO:teuthology.orchestra.run.smithi155.stdout:Vendor : CentOS 2024-03-19T02:16:07.371 INFO:teuthology.orchestra.run.smithi155.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-19T02:16:07.371 INFO:teuthology.orchestra.run.smithi155.stdout:Summary : Support tools for GRUB. 2024-03-19T02:16:07.371 INFO:teuthology.orchestra.run.smithi155.stdout:Description : 2024-03-19T02:16:07.371 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-19T02:16:07.371 INFO:teuthology.orchestra.run.smithi155.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-19T02:16:07.371 INFO:teuthology.orchestra.run.smithi155.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-19T02:16:07.371 INFO:teuthology.orchestra.run.smithi155.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-19T02:16:07.371 INFO:teuthology.orchestra.run.smithi155.stdout:hardware devices. 2024-03-19T02:16:07.371 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-19T02:16:07.371 INFO:teuthology.orchestra.run.smithi155.stdout:This subpackage provides tools for support of all platforms. 2024-03-19T02:16:07.372 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-19T02:16:07.372 INFO:teuthology.task.kernel:Updating grub on smithi155 to boot 5.14.0-430.el9.x86_64 2024-03-19T02:16:07.372 DEBUG:teuthology.orchestra.run.smithi155:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-19T02:16:08.007 INFO:teuthology.orchestra.run.smithi155.stderr:Generating grub configuration file ... 2024-03-19T02:16:09.361 INFO:teuthology.orchestra.run.smithi155.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-19T02:16:09.397 INFO:teuthology.orchestra.run.smithi155.stderr:done 2024-03-19T02:16:09.400 DEBUG:teuthology.orchestra.run.smithi155:> mktemp 2024-03-19T02:16:09.416 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.04ncalL7N9 2024-03-19T02:16:09.416 DEBUG:teuthology.orchestra.run.smithi155:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.04ncalL7N9 2024-03-19T02:16:09.484 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.04ncalL7N9 2024-03-19T02:16:09.626 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.04ncalL7N9 is 6KB 2024-03-19T02:16:09.646 DEBUG:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.04ncalL7N9 2024-03-19T02:16:09.660 DEBUG:teuthology.orchestra.run.smithi155:> sudo /bin/ls /boot/loader/entries || true 2024-03-19T02:16:09.730 INFO:teuthology.orchestra.run.smithi155.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-19T02:16:09.731 INFO:teuthology.orchestra.run.smithi155.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-19T02:16:09.731 INFO:teuthology.orchestra.run.smithi155.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-19T02:16:09.731 INFO:teuthology.orchestra.run.smithi155.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-19T02:16:09.732 DEBUG:teuthology.orchestra.run.smithi155:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-19T02:16:09.868 DEBUG:teuthology.orchestra.run.smithi155:> sudo shutdown -r now 2024-03-19T02:16:34.185 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-19T02:16:34.185 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-03-19T02:16:34.186 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:16:39.871 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-03-19T02:16:39.871 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-19T02:16:39.871 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:16:41.628 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - BaseOS 31 kB/s | 8.0 MB 04:26 2024-03-19T02:16:43.396 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - AppStream 28 MB/s | 19 MB 00:00 2024-03-19T02:16:52.510 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.96 2024-03-19T02:16:58.402 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-19T02:17:01.511 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-03-19T02:17:01.512 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:17:04.574 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.96 2024-03-19T02:17:07.403 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-19T02:17:07.404 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:17:10.462 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-19T02:17:16.575 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-03-19T02:17:16.576 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:17:22.464 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-19T02:17:22.464 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:17:40.894 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-03-19T02:17:48.509 DEBUG:teuthology.orchestra.run.smithi096:> true 2024-03-19T02:17:48.906 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-03-19T02:17:48.906 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-19T02:17:48.906 DEBUG:teuthology.orchestra.run.smithi096:> uname -r 2024-03-19T02:17:48.920 INFO:teuthology.orchestra.run.smithi096.stdout:5.14.0-430.el9.x86_64 2024-03-19T02:17:48.921 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-19T02:17:48.921 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-19T02:17:48.921 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-19T02:17:49.921 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-19T02:17:49.922 DEBUG:teuthology.orchestra.run.smithi096:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-19T02:17:50.008 INFO:teuthology.orchestra.run.smithi096.stdout:ttyS1 2024-03-19T02:17:50.046 DEBUG:teuthology.parallel:result is None 2024-03-19T02:17:55.895 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-19T02:17:55.896 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:17:56.235 DEBUG:teuthology.orchestra.run.smithi155:> true 2024-03-19T02:17:56.654 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-19T02:17:56.654 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-430.el9.x86_64"... 2024-03-19T02:17:56.655 DEBUG:teuthology.orchestra.run.smithi155:> uname -r 2024-03-19T02:17:56.668 INFO:teuthology.orchestra.run.smithi155.stdout:5.14.0-430.el9.x86_64 2024-03-19T02:17:56.668 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-19T02:17:56.668 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-19T02:17:56.669 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-19T02:17:57.670 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-03-19T02:17:57.670 DEBUG:teuthology.orchestra.run.smithi155:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-19T02:17:57.733 INFO:teuthology.orchestra.run.smithi155.stdout:ttyS1 2024-03-19T02:17:57.768 DEBUG:teuthology.parallel:result is None 2024-03-19T02:21:15.763 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - CRB 23 kB/s | 6.1 MB 04:28 2024-03-19T02:21:17.147 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - Extras packages 82 kB/s | 16 kB 00:00 2024-03-19T02:21:18.949 INFO:teuthology.orchestra.run.smithi033.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-19T02:21:23.990 INFO:teuthology.orchestra.run.smithi033.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-19T02:21:26.197 INFO:teuthology.orchestra.run.smithi033.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-19T02:21:26.197 INFO:teuthology.orchestra.run.smithi033.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-19T02:21:26.290 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-19T02:21:26.297 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-19T02:21:26.297 INFO:teuthology.orchestra.run.smithi033.stdout: Package Architecture Version Repository Size 2024-03-19T02:21:26.297 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-19T02:21:26.297 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-19T02:21:26.297 INFO:teuthology.orchestra.run.smithi033.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-19T02:21:26.297 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-19T02:21:26.297 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-19T02:21:26.297 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-19T02:21:26.297 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-19T02:21:26.298 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:21:26.298 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-19T02:21:26.298 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-19T02:21:26.298 INFO:teuthology.orchestra.run.smithi033.stdout:Install 4 Packages 2024-03-19T02:21:26.298 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:21:26.298 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 76 M 2024-03-19T02:21:26.298 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 123 M 2024-03-19T02:21:26.298 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-19T02:21:26.508 INFO:teuthology.orchestra.run.smithi033.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 479 kB/s | 44 kB 00:00 2024-03-19T02:21:26.876 INFO:teuthology.orchestra.run.smithi033.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 32 MB/s | 15 MB 00:00 2024-03-19T02:21:27.292 INFO:teuthology.orchestra.run.smithi033.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 35 MB/s | 27 MB 00:00 2024-03-19T02:21:27.600 INFO:teuthology.orchestra.run.smithi033.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 28 MB/s | 33 MB 00:01 2024-03-19T02:21:27.600 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:21:27.601 INFO:teuthology.orchestra.run.smithi033.stdout:Total 58 MB/s | 76 MB 00:01 2024-03-19T02:21:28.296 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-19T02:21:28.722 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-19T02:21:28.722 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-19T02:21:29.780 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-19T02:21:29.781 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-19T02:21:32.387 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-19T02:21:33.022 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-19T02:21:33.100 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-19T02:21:35.964 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-19T02:21:36.103 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-19T02:21:41.202 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-19T02:21:41.244 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-19T02:21:46.373 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-19T02:22:17.397 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-19T02:22:17.397 INFO:teuthology.orchestra.run.smithi033.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-19T02:22:17.397 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:22:17.442 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-19T02:22:17.824 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-19T02:22:17.824 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-19T02:22:17.824 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-19T02:22:17.824 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-19T02:22:18.229 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-19T02:22:18.229 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:22:18.229 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-19T02:22:18.229 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-19T02:22:18.229 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-19T02:22:18.229 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-19T02:22:18.229 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-19T02:22:18.230 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:22:18.230 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-19T02:22:18.583 DEBUG:teuthology.orchestra.run.smithi033:> rpm -q kernel | sort -rV | head -n 1 2024-03-19T02:22:18.689 INFO:teuthology.orchestra.run.smithi033.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-19T02:22:18.689 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-19T02:22:18.689 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-19T02:22:18.690 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-19T02:22:18.690 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-19T02:22:18.690 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-19T02:22:18.690 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi033.front.sepia.ceph.com, path=None, version=distro) 2024-03-19T02:22:18.690 DEBUG:teuthology.orchestra.run.smithi033:> rpm -q kernel | sort -rV | head -n 1 2024-03-19T02:22:18.773 INFO:teuthology.orchestra.run.smithi033.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-19T02:22:18.773 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-19T02:22:18.774 DEBUG:teuthology.orchestra.run.smithi033:> sudo rpm -qi grub2-tools 2024-03-19T02:22:18.822 INFO:teuthology.orchestra.run.smithi033.stdout:Name : grub2-tools 2024-03-19T02:22:18.822 INFO:teuthology.orchestra.run.smithi033.stdout:Epoch : 1 2024-03-19T02:22:18.822 INFO:teuthology.orchestra.run.smithi033.stdout:Version : 2.06 2024-03-19T02:22:18.822 INFO:teuthology.orchestra.run.smithi033.stdout:Release : 68.el9 2024-03-19T02:22:18.822 INFO:teuthology.orchestra.run.smithi033.stdout:Architecture: x86_64 2024-03-19T02:22:18.822 INFO:teuthology.orchestra.run.smithi033.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-19T02:22:18.822 INFO:teuthology.orchestra.run.smithi033.stdout:Group : Unspecified 2024-03-19T02:22:18.822 INFO:teuthology.orchestra.run.smithi033.stdout:Size : 8264254 2024-03-19T02:22:18.823 INFO:teuthology.orchestra.run.smithi033.stdout:License : GPLv3+ 2024-03-19T02:22:18.823 INFO:teuthology.orchestra.run.smithi033.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-19T02:22:18.823 INFO:teuthology.orchestra.run.smithi033.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-19T02:22:18.823 INFO:teuthology.orchestra.run.smithi033.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-19T02:22:18.823 INFO:teuthology.orchestra.run.smithi033.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-19T02:22:18.823 INFO:teuthology.orchestra.run.smithi033.stdout:Packager : builder@centos.org 2024-03-19T02:22:18.823 INFO:teuthology.orchestra.run.smithi033.stdout:Vendor : CentOS 2024-03-19T02:22:18.823 INFO:teuthology.orchestra.run.smithi033.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-19T02:22:18.823 INFO:teuthology.orchestra.run.smithi033.stdout:Summary : Support tools for GRUB. 2024-03-19T02:22:18.823 INFO:teuthology.orchestra.run.smithi033.stdout:Description : 2024-03-19T02:22:18.823 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:22:18.823 INFO:teuthology.orchestra.run.smithi033.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-19T02:22:18.823 INFO:teuthology.orchestra.run.smithi033.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-19T02:22:18.823 INFO:teuthology.orchestra.run.smithi033.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-19T02:22:18.823 INFO:teuthology.orchestra.run.smithi033.stdout:hardware devices. 2024-03-19T02:22:18.823 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:22:18.824 INFO:teuthology.orchestra.run.smithi033.stdout:This subpackage provides tools for support of all platforms. 2024-03-19T02:22:18.824 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-19T02:22:18.824 INFO:teuthology.task.kernel:Updating grub on smithi033 to boot 5.14.0-430.el9.x86_64 2024-03-19T02:22:18.824 DEBUG:teuthology.orchestra.run.smithi033:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-19T02:22:19.360 INFO:teuthology.orchestra.run.smithi033.stderr:Generating grub configuration file ... 2024-03-19T02:22:20.849 INFO:teuthology.orchestra.run.smithi033.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-19T02:22:20.886 INFO:teuthology.orchestra.run.smithi033.stderr:done 2024-03-19T02:22:20.889 DEBUG:teuthology.orchestra.run.smithi033:> mktemp 2024-03-19T02:22:20.905 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/tmp.4B7rrk1iJF 2024-03-19T02:22:20.905 DEBUG:teuthology.orchestra.run.smithi033:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.4B7rrk1iJF 2024-03-19T02:22:20.979 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0666 /tmp/tmp.4B7rrk1iJF 2024-03-19T02:22:21.121 DEBUG:teuthology.orchestra.remote:smithi033:/tmp/tmp.4B7rrk1iJF is 6KB 2024-03-19T02:22:21.169 DEBUG:teuthology.orchestra.run.smithi033:> rm -fr /tmp/tmp.4B7rrk1iJF 2024-03-19T02:22:21.184 DEBUG:teuthology.orchestra.run.smithi033:> sudo /bin/ls /boot/loader/entries || true 2024-03-19T02:22:21.252 INFO:teuthology.orchestra.run.smithi033.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-19T02:22:21.252 INFO:teuthology.orchestra.run.smithi033.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-19T02:22:21.252 INFO:teuthology.orchestra.run.smithi033.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-19T02:22:21.252 INFO:teuthology.orchestra.run.smithi033.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-19T02:22:21.253 DEBUG:teuthology.orchestra.run.smithi033:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-19T02:22:21.407 DEBUG:teuthology.orchestra.run.smithi033:> sudo shutdown -r now 2024-03-19T02:22:51.410 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-19T02:22:51.411 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-03-19T02:22:51.412 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:23:09.858 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.33 2024-03-19T02:23:18.859 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-03-19T02:23:18.860 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:23:37.246 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.33 2024-03-19T02:23:49.248 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-03-19T02:23:49.249 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:23:49.687 DEBUG:teuthology.orchestra.run.smithi033:> true 2024-03-19T02:23:50.153 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-03-19T02:23:50.153 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-19T02:23:50.153 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2024-03-19T02:23:50.167 INFO:teuthology.orchestra.run.smithi033.stdout:5.14.0-430.el9.x86_64 2024-03-19T02:23:50.167 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-19T02:23:50.167 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-19T02:23:50.167 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-19T02:23:51.168 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-19T02:23:51.169 DEBUG:teuthology.orchestra.run.smithi033:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-19T02:23:51.250 INFO:teuthology.orchestra.run.smithi033.stdout:ttyS1 2024-03-19T02:23:51.279 DEBUG:teuthology.parallel:result is None 2024-03-19T02:23:51.280 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-19T02:23:51.308 INFO:teuthology.task.internal:Creating test directory... 2024-03-19T02:23:51.309 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-19T02:23:51.311 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-19T02:23:51.314 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-19T02:23:51.333 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-19T02:23:51.341 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-19T02:23:51.404 INFO:teuthology.task.internal:Creating archive directory... 2024-03-19T02:23:51.405 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-19T02:23:51.408 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-19T02:23:51.410 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-19T02:23:51.522 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-19T02:23:51.530 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-19T02:23:51.530 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-19T02:23:51.532 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-19T02:23:51.534 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-19T02:23:51.563 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T02:23:51.574 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T02:23:51.576 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T02:23:51.577 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T02:23:51.588 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T02:23:51.591 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T02:23:51.593 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-19T02:23:51.606 INFO:teuthology.task.internal:Configuring sudo... 2024-03-19T02:23:51.606 DEBUG:teuthology.orchestra.run.smithi033:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-19T02:23:51.620 DEBUG:teuthology.orchestra.run.smithi096:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-19T02:23:51.622 DEBUG:teuthology.orchestra.run.smithi155:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-19T02:23:51.660 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-19T02:23:51.675 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-19T02:23:51.675 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-19T02:23:51.688 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-19T02:23:51.693 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-19T02:23:51.716 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T02:23:51.783 DEBUG:teuthology.orchestra.run.smithi033:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T02:23:51.829 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T02:23:51.912 DEBUG:teuthology.orchestra.run.smithi033:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T02:23:51.954 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-19T02:23:51.954 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-19T02:23:52.023 DEBUG:teuthology.orchestra.run.smithi096:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T02:23:52.066 DEBUG:teuthology.orchestra.run.smithi096:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T02:23:52.129 DEBUG:teuthology.orchestra.run.smithi096:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T02:23:52.174 DEBUG:teuthology.orchestra.run.smithi096:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T02:23:52.217 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-19T02:23:52.217 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-19T02:23:52.287 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T02:23:52.332 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T02:23:52.392 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T02:23:52.434 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T02:23:52.480 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-19T02:23:52.480 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-19T02:23:52.549 DEBUG:teuthology.orchestra.run.smithi033:> sudo service rsyslog restart 2024-03-19T02:23:52.551 DEBUG:teuthology.orchestra.run.smithi096:> sudo service rsyslog restart 2024-03-19T02:23:52.553 DEBUG:teuthology.orchestra.run.smithi155:> sudo service rsyslog restart 2024-03-19T02:23:52.595 INFO:teuthology.orchestra.run.smithi033.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-19T02:23:52.604 INFO:teuthology.orchestra.run.smithi096.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-19T02:23:52.642 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-19T02:23:53.048 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-19T02:23:53.057 INFO:teuthology.task.internal:Starting timer... 2024-03-19T02:23:53.057 INFO:teuthology.run_tasks:Running task pcp... 2024-03-19T02:23:53.071 INFO:teuthology.run_tasks:Running task selinux... 2024-03-19T02:23:53.079 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-19T02:23:53.080 DEBUG:teuthology.orchestra.run.smithi033:> sudo service auditd rotate 2024-03-19T02:23:53.154 INFO:teuthology.orchestra.run.smithi033.stdout:Rotating logs: 2024-03-19T02:23:53.156 DEBUG:teuthology.orchestra.run.smithi096:> sudo service auditd rotate 2024-03-19T02:23:53.241 INFO:teuthology.orchestra.run.smithi096.stdout:Rotating logs: 2024-03-19T02:23:53.243 DEBUG:teuthology.orchestra.run.smithi155:> sudo service auditd rotate 2024-03-19T02:23:53.304 INFO:teuthology.orchestra.run.smithi155.stdout:Rotating logs: 2024-03-19T02:23:53.306 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-19T02:23:53.306 DEBUG:teuthology.orchestra.run.smithi033:> /usr/sbin/getenforce 2024-03-19T02:23:53.348 INFO:teuthology.orchestra.run.smithi033.stdout:Permissive 2024-03-19T02:23:53.349 DEBUG:teuthology.orchestra.run.smithi096:> /usr/sbin/getenforce 2024-03-19T02:23:53.379 INFO:teuthology.orchestra.run.smithi096.stdout:Permissive 2024-03-19T02:23:53.379 DEBUG:teuthology.orchestra.run.smithi155:> /usr/sbin/getenforce 2024-03-19T02:23:53.399 INFO:teuthology.orchestra.run.smithi155.stdout:Permissive 2024-03-19T02:23:53.399 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi033.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi096.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi155.front.sepia.ceph.com': 'permissive'} 2024-03-19T02:23:53.399 DEBUG:teuthology.orchestra.run.smithi033:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-19T02:23:53.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T02:23:53.429 DEBUG:teuthology.orchestra.run.smithi096:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-19T02:23:53.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T02:23:53.457 DEBUG:teuthology.orchestra.run.smithi155:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-19T02:23:53.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T02:23:53.487 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-19T02:23:53.488 DEBUG:teuthology.orchestra.run.smithi033:> sudo /usr/sbin/setenforce permissive 2024-03-19T02:23:53.517 DEBUG:teuthology.orchestra.run.smithi096:> sudo /usr/sbin/setenforce permissive 2024-03-19T02:23:53.547 DEBUG:teuthology.orchestra.run.smithi155:> sudo /usr/sbin/setenforce permissive 2024-03-19T02:23:53.585 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-19T02:23:53.594 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-19T02:23:53.611 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-19T02:23:53.612 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi033.front.sepia.ceph.com,smithi096.front.sepia.ceph.com,smithi155.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-19T02:32:29.560 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi033.front.sepia.ceph.com'), Remote(name='ubuntu@smithi096.front.sepia.ceph.com'), Remote(name='ubuntu@smithi155.front.sepia.ceph.com')] 2024-03-19T02:32:29.561 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-03-19T02:32:29.563 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:32:29.655 DEBUG:teuthology.orchestra.run.smithi033:> true 2024-03-19T02:32:29.733 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-03-19T02:32:29.734 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-03-19T02:32:29.734 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:32:29.825 DEBUG:teuthology.orchestra.run.smithi096:> true 2024-03-19T02:32:29.910 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-03-19T02:32:29.910 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-19T02:32:29.911 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:32:29.999 DEBUG:teuthology.orchestra.run.smithi155:> true 2024-03-19T02:32:30.099 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-03-19T02:32:30.099 INFO:teuthology.run_tasks:Running task clock... 2024-03-19T02:32:30.110 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-19T02:32:30.110 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-19T02:32:30.110 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T02:32:30.113 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-19T02:32:30.113 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T02:32:30.115 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-19T02:32:30.115 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T02:32:30.143 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-19T02:32:30.146 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-19T02:32:30.160 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-19T02:32:30.164 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-19T02:32:30.175 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-19T02:32:30.190 INFO:teuthology.orchestra.run.smithi033.stderr:sudo: ntpd: command not found 2024-03-19T02:32:30.193 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-19T02:32:30.194 INFO:teuthology.orchestra.run.smithi096.stderr:sudo: ntpd: command not found 2024-03-19T02:32:30.204 INFO:teuthology.orchestra.run.smithi033.stdout:506 Cannot talk to daemon 2024-03-19T02:32:30.208 INFO:teuthology.orchestra.run.smithi096.stdout:506 Cannot talk to daemon 2024-03-19T02:32:30.221 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-19T02:32:30.224 INFO:teuthology.orchestra.run.smithi155.stderr:sudo: ntpd: command not found 2024-03-19T02:32:30.225 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-19T02:32:30.237 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-19T02:32:30.238 INFO:teuthology.orchestra.run.smithi155.stdout:506 Cannot talk to daemon 2024-03-19T02:32:30.242 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-19T02:32:30.256 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-19T02:32:30.315 INFO:teuthology.orchestra.run.smithi096.stderr:bash: line 1: ntpq: command not found 2024-03-19T02:32:30.315 INFO:teuthology.orchestra.run.smithi033.stderr:bash: line 1: ntpq: command not found 2024-03-19T02:32:30.315 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-19T02:32:30.316 INFO:teuthology.orchestra.run.smithi096.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-19T02:32:30.317 INFO:teuthology.orchestra.run.smithi096.stdout:=============================================================================== 2024-03-19T02:32:30.317 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:32:30.317 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:32:30.317 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:32:30.317 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:32:30.317 INFO:teuthology.orchestra.run.smithi033.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-19T02:32:30.317 INFO:teuthology.orchestra.run.smithi033.stdout:=============================================================================== 2024-03-19T02:32:30.317 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:32:30.317 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:32:30.318 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:32:30.318 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:32:30.325 INFO:teuthology.orchestra.run.smithi155.stderr:bash: line 1: ntpq: command not found 2024-03-19T02:32:30.329 INFO:teuthology.orchestra.run.smithi155.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-19T02:32:30.329 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================== 2024-03-19T02:32:30.329 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:32:30.329 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:32:30.329 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:32:30.329 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T02:32:30.331 INFO:teuthology.run_tasks:Running task install... 2024-03-19T02:32:30.338 DEBUG:teuthology.task.install:project ceph 2024-03-19T02:32:30.339 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810'}} 2024-03-19T02:32:30.339 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810'} 2024-03-19T02:32:30.339 INFO:teuthology.task.install:Using flavor: default 2024-03-19T02:32:30.345 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-19T02:32:30.345 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-19T02:32:30.345 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'tag': None, 'wait_for_package': False} 2024-03-19T02:32:30.346 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:32:30.348 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'tag': None, 'wait_for_package': False} 2024-03-19T02:32:30.348 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:32:30.349 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'tag': None, 'wait_for_package': False} 2024-03-19T02:32:30.349 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:32:30.565 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/ 2024-03-19T02:32:30.565 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2206.gb6ae2491 2024-03-19T02:32:30.571 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/ 2024-03-19T02:32:30.571 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2206.gb6ae2491 2024-03-19T02:32:30.576 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/ 2024-03-19T02:32:30.576 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2206.gb6ae2491 2024-03-19T02:32:30.688 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-19T02:32:30.688 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-19T02:32:30.688 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-19T02:32:30.716 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-19T02:32:30.716 DEBUG:teuthology.orchestra.run.smithi096:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b6ae24918b03b1b8a19c9263a50655bba80bb810/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-19T02:32:30.747 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-19T02:32:30.747 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-19T02:32:30.747 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-19T02:32:30.749 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-19T02:32:30.749 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-19T02:32:30.749 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-19T02:32:30.777 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-19T02:32:30.778 DEBUG:teuthology.orchestra.run.smithi033:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b6ae24918b03b1b8a19c9263a50655bba80bb810/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-19T02:32:30.779 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-19T02:32:30.779 DEBUG:teuthology.orchestra.run.smithi155:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b6ae24918b03b1b8a19c9263a50655bba80bb810/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-19T02:32:30.784 DEBUG:teuthology.orchestra.run.smithi096:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-19T02:32:30.846 DEBUG:teuthology.orchestra.run.smithi033:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-19T02:32:30.849 DEBUG:teuthology.orchestra.run.smithi155:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-19T02:32:30.863 DEBUG:teuthology.orchestra.run.smithi096:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-19T02:32:30.927 DEBUG:teuthology.orchestra.run.smithi033:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-19T02:32:30.930 DEBUG:teuthology.orchestra.run.smithi155:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-19T02:32:30.930 INFO:teuthology.orchestra.run.smithi096.stdout:check_obsoletes = 1 2024-03-19T02:32:30.931 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum clean all 2024-03-19T02:32:30.994 INFO:teuthology.orchestra.run.smithi033.stdout:check_obsoletes = 1 2024-03-19T02:32:30.995 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum clean all 2024-03-19T02:32:30.997 INFO:teuthology.orchestra.run.smithi155.stdout:check_obsoletes = 1 2024-03-19T02:32:30.998 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum clean all 2024-03-19T02:32:31.183 INFO:teuthology.orchestra.run.smithi096.stdout:74 files removed 2024-03-19T02:32:31.210 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-19T02:32:31.248 INFO:teuthology.orchestra.run.smithi033.stdout:74 files removed 2024-03-19T02:32:31.254 INFO:teuthology.orchestra.run.smithi155.stdout:74 files removed 2024-03-19T02:32:31.276 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-19T02:32:31.283 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-19T02:32:31.872 INFO:teuthology.orchestra.run.smithi096.stdout:ceph packages for x86_64 230 kB/s | 83 kB 00:00 2024-03-19T02:32:31.987 INFO:teuthology.orchestra.run.smithi155.stdout:ceph packages for x86_64 205 kB/s | 83 kB 00:00 2024-03-19T02:32:32.039 INFO:teuthology.orchestra.run.smithi033.stdout:ceph packages for x86_64 181 kB/s | 83 kB 00:00 2024-03-19T02:32:32.289 INFO:teuthology.orchestra.run.smithi096.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-19T02:32:32.413 INFO:teuthology.orchestra.run.smithi155.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-19T02:32:32.440 INFO:teuthology.orchestra.run.smithi033.stdout:ceph noarch packages 29 kB/s | 10 kB 00:00 2024-03-19T02:32:32.648 INFO:teuthology.orchestra.run.smithi096.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-03-19T02:32:32.789 INFO:teuthology.orchestra.run.smithi155.stdout:ceph source packages 5.6 kB/s | 1.9 kB 00:00 2024-03-19T02:32:32.874 INFO:teuthology.orchestra.run.smithi033.stdout:ceph source packages 4.8 kB/s | 1.9 kB 00:00 2024-03-19T02:32:33.058 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-19T02:32:33.191 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-19T02:32:33.293 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-19T02:32:34.650 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS-9 - BaseOS 5.5 MB/s | 2.0 MB 00:00 2024-03-19T02:32:34.792 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS-9 - BaseOS 3.2 MB/s | 2.0 MB 00:00 2024-03-19T02:32:34.902 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS-9 - BaseOS 5.5 MB/s | 2.0 MB 00:00 2024-03-19T02:32:35.684 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.0 MB 00:00 2024-03-19T02:32:35.891 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-03-19T02:32:37.508 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - BaseOS 3.2 MB/s | 8.0 MB 00:02 2024-03-19T02:32:41.258 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - AppStream 6.9 MB/s | 19 MB 00:02 2024-03-19T02:32:45.067 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.1 MB 00:00 2024-03-19T02:32:46.677 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-03-19T02:32:49.705 INFO:teuthology.orchestra.run.smithi155.stdout:Extra Packages for Enterprise Linux 7.1 MB/s | 21 MB 00:02 2024-03-19T02:32:50.342 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - AppStream 1.4 MB/s | 19 MB 00:13 2024-03-19T02:32:51.198 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 19 MB 00:14 2024-03-19T02:32:54.544 INFO:teuthology.orchestra.run.smithi155.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-19T02:32:56.933 INFO:teuthology.orchestra.run.smithi155.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-19T02:32:56.934 INFO:teuthology.orchestra.run.smithi155.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-19T02:32:56.948 INFO:teuthology.orchestra.run.smithi155.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-19T02:32:56.949 INFO:teuthology.orchestra.run.smithi155.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-19T02:32:56.950 INFO:teuthology.orchestra.run.smithi155.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-19T02:32:56.951 INFO:teuthology.orchestra.run.smithi155.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-19T02:32:56.953 INFO:teuthology.orchestra.run.smithi155.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-19T02:32:56.956 INFO:teuthology.orchestra.run.smithi155.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-19T02:32:56.957 INFO:teuthology.orchestra.run.smithi155.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-19T02:32:57.036 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-03-19T02:32:57.053 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================================ 2024-03-19T02:32:57.053 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-03-19T02:32:57.053 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================================ 2024-03-19T02:32:57.053 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-03-19T02:32:57.053 INFO:teuthology.orchestra.run.smithi155.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-19T02:32:57.053 INFO:teuthology.orchestra.run.smithi155.stdout: ceph x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 6.6 k 2024-03-19T02:32:57.053 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.4 M 2024-03-19T02:32:57.053 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 891 k 2024-03-19T02:32:57.053 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 145 k 2024-03-19T02:32:57.053 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 1.5 M 2024-03-19T02:32:57.053 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 139 k 2024-03-19T02:32:57.054 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 1.8 M 2024-03-19T02:32:57.054 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 7.4 M 2024-03-19T02:32:57.054 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 49 k 2024-03-19T02:32:57.054 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 9.8 M 2024-03-19T02:32:57.054 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 48 M 2024-03-19T02:32:57.054 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-volume noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 264 k 2024-03-19T02:32:57.054 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 749 k 2024-03-19T02:32:57.054 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 221 k 2024-03-19T02:32:57.054 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-shell x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 27 k 2024-03-19T02:32:57.054 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-top noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 18 k 2024-03-19T02:32:57.054 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-19T02:32:57.054 INFO:teuthology.orchestra.run.smithi155.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-19T02:32:57.055 INFO:teuthology.orchestra.run.smithi155.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-19T02:32:57.055 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 32 k 2024-03-19T02:32:57.055 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 746 k 2024-03-19T02:32:57.055 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 127 k 2024-03-19T02:32:57.055 INFO:teuthology.orchestra.run.smithi155.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-19T02:32:57.055 INFO:teuthology.orchestra.run.smithi155.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-19T02:32:57.055 INFO:teuthology.orchestra.run.smithi155.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-19T02:32:57.055 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-19T02:32:57.055 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-19T02:32:57.055 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-19T02:32:57.055 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 162 k 2024-03-19T02:32:57.055 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 321 k 2024-03-19T02:32:57.055 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 298 k 2024-03-19T02:32:57.055 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 99 k 2024-03-19T02:32:57.056 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 85 k 2024-03-19T02:32:57.056 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-19T02:32:57.056 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 170 k 2024-03-19T02:32:57.056 INFO:teuthology.orchestra.run.smithi155.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-19T02:32:57.056 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrading: 2024-03-19T02:32:57.056 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-19T02:32:57.056 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-19T02:32:57.056 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-19T02:32:57.056 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-19T02:32:57.056 INFO:teuthology.orchestra.run.smithi155.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-19T02:32:57.056 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-19T02:32:57.056 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-19T02:32:57.056 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-19T02:32:57.056 INFO:teuthology.orchestra.run.smithi155.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-19T02:32:57.056 INFO:teuthology.orchestra.run.smithi155.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-19T02:32:57.057 INFO:teuthology.orchestra.run.smithi155.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-19T02:32:57.057 INFO:teuthology.orchestra.run.smithi155.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-19T02:32:57.057 INFO:teuthology.orchestra.run.smithi155.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-19T02:32:57.057 INFO:teuthology.orchestra.run.smithi155.stdout: librados2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.4 M 2024-03-19T02:32:57.057 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-19T02:32:57.057 INFO:teuthology.orchestra.run.smithi155.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-19T02:32:57.057 INFO:teuthology.orchestra.run.smithi155.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-19T02:32:57.057 INFO:teuthology.orchestra.run.smithi155.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-19T02:32:57.057 INFO:teuthology.orchestra.run.smithi155.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-19T02:32:57.057 INFO:teuthology.orchestra.run.smithi155.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-19T02:32:57.057 INFO:teuthology.orchestra.run.smithi155.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-19T02:32:57.057 INFO:teuthology.orchestra.run.smithi155.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-19T02:32:57.057 INFO:teuthology.orchestra.run.smithi155.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-19T02:32:57.058 INFO:teuthology.orchestra.run.smithi155.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-19T02:32:57.058 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-03-19T02:32:57.058 INFO:teuthology.orchestra.run.smithi155.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-19T02:32:57.058 INFO:teuthology.orchestra.run.smithi155.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-19T02:32:57.058 INFO:teuthology.orchestra.run.smithi155.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-19T02:32:57.058 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 21 M 2024-03-19T02:32:57.058 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 34 k 2024-03-19T02:32:57.058 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 2.4 M 2024-03-19T02:32:57.058 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 248 k 2024-03-19T02:32:57.058 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 4.7 M 2024-03-19T02:32:57.058 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 17 M 2024-03-19T02:32:57.058 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 16 k 2024-03-19T02:32:57.058 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 25 k 2024-03-19T02:32:57.059 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-19T02:32:57.059 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-19T02:32:57.059 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-19T02:32:57.059 INFO:teuthology.orchestra.run.smithi155.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-19T02:32:57.059 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-19T02:32:57.059 INFO:teuthology.orchestra.run.smithi155.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-19T02:32:57.059 INFO:teuthology.orchestra.run.smithi155.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-19T02:32:57.059 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-19T02:32:57.059 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-19T02:32:57.059 INFO:teuthology.orchestra.run.smithi155.stdout: libcephsqlite x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 163 k 2024-03-19T02:32:57.059 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-19T02:32:57.059 INFO:teuthology.orchestra.run.smithi155.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-19T02:32:57.059 INFO:teuthology.orchestra.run.smithi155.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-19T02:32:57.059 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-19T02:32:57.060 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-19T02:32:57.060 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 501 k 2024-03-19T02:32:57.060 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-19T02:32:57.060 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.1 M 2024-03-19T02:32:57.060 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-19T02:32:57.060 INFO:teuthology.orchestra.run.smithi155.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-19T02:32:57.060 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-19T02:32:57.060 INFO:teuthology.orchestra.run.smithi155.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-19T02:32:57.060 INFO:teuthology.orchestra.run.smithi155.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-19T02:32:57.060 INFO:teuthology.orchestra.run.smithi155.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-19T02:32:57.060 INFO:teuthology.orchestra.run.smithi155.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-19T02:32:57.060 INFO:teuthology.orchestra.run.smithi155.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-19T02:32:57.060 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-19T02:32:57.060 INFO:teuthology.orchestra.run.smithi155.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-19T02:32:57.061 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-19T02:32:57.061 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-19T02:32:57.061 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-19T02:32:57.061 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-19T02:32:57.061 INFO:teuthology.orchestra.run.smithi155.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-19T02:32:57.061 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-19T02:32:57.061 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-19T02:32:57.061 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-19T02:32:57.061 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 45 k 2024-03-19T02:32:57.061 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 131 k 2024-03-19T02:32:57.061 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-19T02:32:57.061 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-19T02:32:57.061 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-19T02:32:57.061 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-19T02:32:57.062 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-19T02:32:57.062 INFO:teuthology.orchestra.run.smithi155.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-19T02:32:57.062 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-19T02:32:57.062 INFO:teuthology.orchestra.run.smithi155.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-19T02:32:57.062 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-19T02:32:57.062 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-19T02:32:57.062 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-19T02:32:57.062 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-19T02:32:57.062 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-19T02:32:57.062 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-19T02:32:57.062 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-19T02:32:57.062 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-19T02:32:57.062 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-19T02:32:57.062 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-19T02:32:57.063 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-19T02:32:57.063 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-19T02:32:57.063 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-19T02:32:57.063 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-19T02:32:57.063 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-19T02:32:57.063 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-19T02:32:57.063 INFO:teuthology.orchestra.run.smithi155.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-19T02:32:57.063 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-19T02:32:57.063 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-19T02:32:57.063 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-19T02:32:57.063 INFO:teuthology.orchestra.run.smithi155.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-19T02:32:57.063 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-19T02:32:57.063 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-19T02:32:57.063 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-19T02:32:57.063 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-19T02:32:57.063 INFO:teuthology.orchestra.run.smithi155.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-19T02:32:57.064 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-19T02:32:57.064 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-19T02:32:57.064 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-19T02:32:57.064 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-19T02:32:57.064 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-19T02:32:57.064 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-19T02:32:57.064 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-19T02:32:57.064 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-19T02:32:57.064 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-19T02:32:57.064 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-19T02:32:57.064 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-19T02:32:57.064 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-19T02:32:57.064 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-19T02:32:57.065 INFO:teuthology.orchestra.run.smithi155.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-19T02:32:57.065 INFO:teuthology.orchestra.run.smithi155.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-19T02:32:57.065 INFO:teuthology.orchestra.run.smithi155.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-19T02:32:57.065 INFO:teuthology.orchestra.run.smithi155.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-19T02:32:57.065 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-19T02:32:57.065 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-19T02:32:57.065 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-19T02:32:57.065 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-19T02:32:57.065 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-19T02:32:57.065 INFO:teuthology.orchestra.run.smithi155.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-19T02:32:57.065 INFO:teuthology.orchestra.run.smithi155.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-19T02:32:57.065 INFO:teuthology.orchestra.run.smithi155.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-19T02:32:57.065 INFO:teuthology.orchestra.run.smithi155.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-19T02:32:57.065 INFO:teuthology.orchestra.run.smithi155.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-19T02:32:57.065 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-19T02:32:57.066 INFO:teuthology.orchestra.run.smithi155.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-19T02:32:57.066 INFO:teuthology.orchestra.run.smithi155.stdout:Installing weak dependencies: 2024-03-19T02:32:57.066 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-19T02:32:57.066 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-19T02:32:57.066 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-03-19T02:32:57.066 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================================ 2024-03-19T02:32:57.066 INFO:teuthology.orchestra.run.smithi155.stdout:Install 150 Packages 2024-03-19T02:32:57.066 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrade 24 Packages 2024-03-19T02:32:57.066 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-19T02:32:57.067 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 224 M 2024-03-19T02:32:57.067 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-03-19T02:32:57.904 INFO:teuthology.orchestra.run.smithi155.stdout:(1/174): ceph-19.0.0-2206.gb6ae2491.el9.x86_64. 16 kB/s | 6.6 kB 00:00 2024-03-19T02:32:58.338 INFO:teuthology.orchestra.run.smithi155.stdout:(2/174): ceph-fuse-19.0.0-2206.gb6ae2491.el9.x8 2.0 MB/s | 891 kB 00:00 2024-03-19T02:32:58.413 INFO:teuthology.orchestra.run.smithi155.stdout:(3/174): ceph-immutable-object-cache-19.0.0-220 1.9 MB/s | 145 kB 00:00 2024-03-19T02:32:58.838 INFO:teuthology.orchestra.run.smithi155.stdout:(4/174): ceph-mds-19.0.0-2206.gb6ae2491.el9.x86 5.6 MB/s | 2.4 MB 00:00 2024-03-19T02:32:59.122 INFO:teuthology.orchestra.run.smithi155.stdout:(5/174): ceph-mgr-19.0.0-2206.gb6ae2491.el9.x86 5.3 MB/s | 1.5 MB 00:00 2024-03-19T02:32:59.780 INFO:teuthology.orchestra.run.smithi155.stdout:(6/174): ceph-mon-19.0.0-2206.gb6ae2491.el9.x86 7.1 MB/s | 4.7 MB 00:00 2024-03-19T02:33:00.239 INFO:teuthology.orchestra.run.smithi155.stdout:(7/174): ceph-base-19.0.0-2206.gb6ae2491.el9.x8 2.0 MB/s | 5.4 MB 00:02 2024-03-19T02:33:01.514 INFO:teuthology.orchestra.run.smithi155.stdout:(8/174): ceph-common-19.0.0-2206.gb6ae2491.el9. 5.1 MB/s | 21 MB 00:04 2024-03-19T02:33:01.672 INFO:teuthology.orchestra.run.smithi155.stdout:(9/174): ceph-osd-19.0.0-2206.gb6ae2491.el9.x86 8.9 MB/s | 17 MB 00:01 2024-03-19T02:33:01.698 INFO:teuthology.orchestra.run.smithi155.stdout:(10/174): ceph-selinux-19.0.0-2206.gb6ae2491.el 137 kB/s | 25 kB 00:00 2024-03-19T02:33:01.839 INFO:teuthology.orchestra.run.smithi155.stdout:(11/174): cephfs-mirror-19.0.0-2206.gb6ae2491.e 1.5 MB/s | 221 kB 00:00 2024-03-19T02:33:01.915 INFO:teuthology.orchestra.run.smithi155.stdout:(12/174): cephfs-shell-19.0.0-2206.gb6ae2491.el 361 kB/s | 27 kB 00:00 2024-03-19T02:33:01.990 INFO:teuthology.orchestra.run.smithi155.stdout:(13/174): libcephfs-devel-19.0.0-2206.gb6ae2491 425 kB/s | 32 kB 00:00 2024-03-19T02:33:02.207 INFO:teuthology.orchestra.run.smithi155.stdout:(14/174): libcephfs2-19.0.0-2206.gb6ae2491.el9. 3.4 MB/s | 746 kB 00:00 2024-03-19T02:33:02.315 INFO:teuthology.orchestra.run.smithi155.stdout:(15/174): libcephsqlite-19.0.0-2206.gb6ae2491.e 1.5 MB/s | 163 kB 00:00 2024-03-19T02:33:02.366 INFO:teuthology.orchestra.run.smithi155.stdout:(16/174): librados-devel-19.0.0-2206.gb6ae2491. 2.5 MB/s | 127 kB 00:00 2024-03-19T02:33:02.491 INFO:teuthology.orchestra.run.smithi155.stdout:(17/174): libradosstriper1-19.0.0-2206.gb6ae249 3.9 MB/s | 501 kB 00:00 2024-03-19T02:33:03.466 INFO:teuthology.orchestra.run.smithi155.stdout:(18/174): librgw2-19.0.0-2206.gb6ae2491.el9.x86 5.3 MB/s | 5.1 MB 00:00 2024-03-19T02:33:03.542 INFO:teuthology.orchestra.run.smithi155.stdout:(19/174): python3-ceph-argparse-19.0.0-2206.gb6 603 kB/s | 45 kB 00:00 2024-03-19T02:33:03.650 INFO:teuthology.orchestra.run.smithi155.stdout:(20/174): ceph-radosgw-19.0.0-2206.gb6ae2491.el 2.9 MB/s | 9.8 MB 00:03 2024-03-19T02:33:03.675 INFO:teuthology.orchestra.run.smithi155.stdout:(21/174): python3-ceph-common-19.0.0-2206.gb6ae 983 kB/s | 131 kB 00:00 2024-03-19T02:33:03.759 INFO:teuthology.orchestra.run.smithi155.stdout:(22/174): python3-rados-19.0.0-2206.gb6ae2491.e 3.8 MB/s | 321 kB 00:00 2024-03-19T02:33:03.785 INFO:teuthology.orchestra.run.smithi155.stdout:(23/174): python3-cephfs-19.0.0-2206.gb6ae2491. 1.2 MB/s | 162 kB 00:00 2024-03-19T02:33:03.843 INFO:teuthology.orchestra.run.smithi155.stdout:(24/174): python3-rbd-19.0.0-2206.gb6ae2491.el9 3.5 MB/s | 298 kB 00:00 2024-03-19T02:33:03.885 INFO:teuthology.orchestra.run.smithi155.stdout:(25/174): python3-rgw-19.0.0-2206.gb6ae2491.el9 993 kB/s | 99 kB 00:00 2024-03-19T02:33:03.919 INFO:teuthology.orchestra.run.smithi155.stdout:(26/174): rbd-fuse-19.0.0-2206.gb6ae2491.el9.x8 1.1 MB/s | 85 kB 00:00 2024-03-19T02:33:03.994 INFO:teuthology.orchestra.run.smithi155.stdout:(27/174): rbd-nbd-19.0.0-2206.gb6ae2491.el9.x86 2.2 MB/s | 170 kB 00:00 2024-03-19T02:33:04.069 INFO:teuthology.orchestra.run.smithi155.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2206.g 453 kB/s | 34 kB 00:00 2024-03-19T02:33:04.137 INFO:teuthology.orchestra.run.smithi155.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2206.gb6ae249 2.0 MB/s | 139 kB 00:00 2024-03-19T02:33:04.396 INFO:teuthology.orchestra.run.smithi155.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2206.gb6ae2 7.0 MB/s | 1.8 MB 00:00 2024-03-19T02:33:04.729 INFO:teuthology.orchestra.run.smithi155.stdout:(31/174): rbd-mirror-19.0.0-2206.gb6ae2491.el9. 3.7 MB/s | 3.1 MB 00:00 2024-03-19T02:33:04.879 INFO:teuthology.orchestra.run.smithi155.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2206.gb6 1.6 MB/s | 248 kB 00:00 2024-03-19T02:33:04.954 INFO:teuthology.orchestra.run.smithi155.stdout:(33/174): ceph-mgr-rook-19.0.0-2206.gb6ae2491.e 659 kB/s | 49 kB 00:00 2024-03-19T02:33:05.046 INFO:teuthology.orchestra.run.smithi155.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2206.gb 180 kB/s | 16 kB 00:00 2024-03-19T02:33:05.221 INFO:teuthology.orchestra.run.smithi155.stdout:(35/174): ceph-volume-19.0.0-2206.gb6ae2491.el9 1.5 MB/s | 264 kB 00:00 2024-03-19T02:33:05.530 INFO:teuthology.orchestra.run.smithi155.stdout:(36/174): ceph-mgr-diskprediction-local-19.0.0- 6.5 MB/s | 7.4 MB 00:01 2024-03-19T02:33:05.563 INFO:teuthology.orchestra.run.smithi155.stdout:(37/174): cephadm-19.0.0-2206.gb6ae2491.el9.noa 2.1 MB/s | 749 kB 00:00 2024-03-19T02:33:05.580 INFO:teuthology.orchestra.run.smithi155.stdout:(38/174): cephfs-top-19.0.0-2206.gb6ae2491.el9. 363 kB/s | 18 kB 00:00 2024-03-19T02:33:05.731 INFO:teuthology.orchestra.run.smithi155.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 3.9 MB/s | 662 kB 00:00 2024-03-19T02:33:05.772 INFO:teuthology.orchestra.run.smithi155.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 4.7 MB/s | 920 kB 00:00 2024-03-19T02:33:05.798 INFO:teuthology.orchestra.run.smithi155.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-19T02:33:05.823 INFO:teuthology.orchestra.run.smithi155.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-19T02:33:05.848 INFO:teuthology.orchestra.run.smithi155.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 636 kB/s | 32 kB 00:00 2024-03-19T02:33:05.932 INFO:teuthology.orchestra.run.smithi155.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 28 MB/s | 3.0 MB 00:00 2024-03-19T02:33:05.957 INFO:teuthology.orchestra.run.smithi155.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 160 kB/s | 17 kB 00:00 2024-03-19T02:33:05.982 INFO:teuthology.orchestra.run.smithi155.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 200 kB/s | 10 kB 00:00 2024-03-19T02:33:06.008 INFO:teuthology.orchestra.run.smithi155.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-19T02:33:06.033 INFO:teuthology.orchestra.run.smithi155.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 880 kB/s | 44 kB 00:00 2024-03-19T02:33:06.059 INFO:teuthology.orchestra.run.smithi155.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-19T02:33:06.092 INFO:teuthology.orchestra.run.smithi155.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.8 MB/s | 578 kB 00:00 2024-03-19T02:33:06.147 INFO:teuthology.orchestra.run.smithi155.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 529 kB/s | 46 kB 00:00 2024-03-19T02:33:06.176 INFO:teuthology.orchestra.run.smithi155.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-03-19T02:33:06.201 INFO:teuthology.orchestra.run.smithi155.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.4 MB/s | 187 kB 00:00 2024-03-19T02:33:06.235 INFO:teuthology.orchestra.run.smithi155.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2024-03-19T02:33:06.252 INFO:teuthology.orchestra.run.smithi155.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 683 kB/s | 34 kB 00:00 2024-03-19T02:33:06.661 INFO:teuthology.orchestra.run.smithi155.stdout:(56/174): ceph-test-19.0.0-2206.gb6ae2491.el9.x 9.5 MB/s | 48 MB 00:04 2024-03-19T02:33:06.711 INFO:teuthology.orchestra.run.smithi155.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 3.3 MB/s | 1.5 MB 00:00 2024-03-19T02:33:06.770 INFO:teuthology.orchestra.run.smithi155.stdout:(58/174): openblas-openmp-0.3.21-2.el9.x86_64.r 8.9 MB/s | 4.7 MB 00:00 2024-03-19T02:33:06.795 INFO:teuthology.orchestra.run.smithi155.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 6.0 MB/s | 811 kB 00:00 2024-03-19T02:33:06.820 INFO:teuthology.orchestra.run.smithi155.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.2 MB/s | 134 kB 00:00 2024-03-19T02:33:06.862 INFO:teuthology.orchestra.run.smithi155.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-03-19T02:33:06.979 INFO:teuthology.orchestra.run.smithi155.stdout:(62/174): postgresql-server-13.11-1.el9.x86_64. 31 MB/s | 5.7 MB 00:00 2024-03-19T02:33:07.004 INFO:teuthology.orchestra.run.smithi155.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 428 kB/s | 78 kB 00:00 2024-03-19T02:33:07.088 INFO:teuthology.orchestra.run.smithi155.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 26 MB/s | 5.8 MB 00:00 2024-03-19T02:33:07.105 INFO:teuthology.orchestra.run.smithi155.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 199 kB/s | 24 kB 00:00 2024-03-19T02:33:07.122 INFO:teuthology.orchestra.run.smithi155.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 158 kB/s | 18 kB 00:00 2024-03-19T02:33:07.147 INFO:teuthology.orchestra.run.smithi155.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-19T02:33:07.172 INFO:teuthology.orchestra.run.smithi155.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 636 kB/s | 32 kB 00:00 2024-03-19T02:33:07.264 INFO:teuthology.orchestra.run.smithi155.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-19T02:33:07.290 INFO:teuthology.orchestra.run.smithi155.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 217 kB 00:00 2024-03-19T02:33:07.314 INFO:teuthology.orchestra.run.smithi155.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 174 kB 00:00 2024-03-19T02:33:07.340 INFO:teuthology.orchestra.run.smithi155.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 920 kB/s | 69 kB 00:00 2024-03-19T02:33:07.365 INFO:teuthology.orchestra.run.smithi155.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 823 kB/s | 41 kB 00:00 2024-03-19T02:33:07.390 INFO:teuthology.orchestra.run.smithi155.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-19T02:33:07.416 INFO:teuthology.orchestra.run.smithi155.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-19T02:33:07.449 INFO:teuthology.orchestra.run.smithi155.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.5 MB/s | 211 kB 00:00 2024-03-19T02:33:07.474 INFO:teuthology.orchestra.run.smithi155.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 569 kB/s | 47 kB 00:00 2024-03-19T02:33:07.499 INFO:teuthology.orchestra.run.smithi155.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 515 kB/s | 43 kB 00:00 2024-03-19T02:33:07.525 INFO:teuthology.orchestra.run.smithi155.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 773 kB/s | 39 kB 00:00 2024-03-19T02:33:07.625 INFO:teuthology.orchestra.run.smithi155.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 327 kB/s | 41 kB 00:00 2024-03-19T02:33:07.701 INFO:teuthology.orchestra.run.smithi155.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.7 MB/s | 300 kB 00:00 2024-03-19T02:33:07.859 INFO:teuthology.orchestra.run.smithi155.stdout:(82/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-03-19T02:33:07.884 INFO:teuthology.orchestra.run.smithi155.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 220 kB/s | 56 kB 00:00 2024-03-19T02:33:07.910 INFO:teuthology.orchestra.run.smithi155.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 302 kB/s | 63 kB 00:00 2024-03-19T02:33:07.935 INFO:teuthology.orchestra.run.smithi155.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 883 kB/s | 44 kB 00:00 2024-03-19T02:33:07.960 INFO:teuthology.orchestra.run.smithi155.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-19T02:33:07.994 INFO:teuthology.orchestra.run.smithi155.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.8 MB/s | 241 kB 00:00 2024-03-19T02:33:08.019 INFO:teuthology.orchestra.run.smithi155.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.7 MB/s | 103 kB 00:00 2024-03-19T02:33:08.053 INFO:teuthology.orchestra.run.smithi155.stdout:(89/174): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.1 MB 00:00 2024-03-19T02:33:08.078 INFO:teuthology.orchestra.run.smithi155.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.5 MB/s | 124 kB 00:00 2024-03-19T02:33:08.170 INFO:teuthology.orchestra.run.smithi155.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 1.2 MB/s | 187 kB 00:00 2024-03-19T02:33:08.295 INFO:teuthology.orchestra.run.smithi155.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 865 kB/s | 187 kB 00:00 2024-03-19T02:33:08.320 INFO:teuthology.orchestra.run.smithi155.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-19T02:33:08.346 INFO:teuthology.orchestra.run.smithi155.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-03-19T02:33:08.379 INFO:teuthology.orchestra.run.smithi155.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 803 kB 00:00 2024-03-19T02:33:08.430 INFO:teuthology.orchestra.run.smithi155.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 198 kB/s | 21 kB 00:00 2024-03-19T02:33:08.446 INFO:teuthology.orchestra.run.smithi155.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 253 kB/s | 25 kB 00:00 2024-03-19T02:33:08.547 INFO:teuthology.orchestra.run.smithi155.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.1 MB/s | 245 kB 00:00 2024-03-19T02:33:08.589 INFO:teuthology.orchestra.run.smithi155.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 1.7 MB/s | 248 kB 00:00 2024-03-19T02:33:08.631 INFO:teuthology.orchestra.run.smithi155.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 2.5 MB/s | 216 kB 00:00 2024-03-19T02:33:08.656 INFO:teuthology.orchestra.run.smithi155.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 523 kB/s | 35 kB 00:00 2024-03-19T02:33:08.706 INFO:teuthology.orchestra.run.smithi155.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-03-19T02:33:08.757 INFO:teuthology.orchestra.run.smithi155.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 446 kB/s | 22 kB 00:00 2024-03-19T02:33:08.799 INFO:teuthology.orchestra.run.smithi155.stdout:(104/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 266 kB/s | 44 kB 00:00 2024-03-19T02:33:08.941 INFO:teuthology.orchestra.run.smithi155.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 4.6 MB/s | 863 kB 00:00 2024-03-19T02:33:09.082 INFO:teuthology.orchestra.run.smithi155.stdout:(106/174): openssl-devel-3.0.7-27.el9.x86_64.rp 5.9 MB/s | 4.1 MB 00:00 2024-03-19T02:35:20.894 INFO:teuthology.orchestra.run.smithi155.stdout:[MIRROR] abseil-cpp-20211102.0-3.el9.x86_64.rpm: Curl error (28): Timeout was reached for https://ftp-nyc.osuosl.org/pub/fedora-epel/9/Everything/x86_64/Packages/a/abseil-cpp-20211102.0-3.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 443: Connection timed out] 2024-03-19T02:35:20.894 INFO:teuthology.orchestra.run.smithi155.stdout:[MIRROR] gperftools-libs-2.9.1-2.el9.x86_64.rpm: Curl error (28): Timeout was reached for https://ftp-nyc.osuosl.org/pub/fedora-epel/9/Everything/x86_64/Packages/g/gperftools-libs-2.9.1-2.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 443: Connection timed out] 2024-03-19T02:35:20.894 INFO:teuthology.orchestra.run.smithi155.stdout:[MIRROR] grpc-data-1.46.7-10.el9.noarch.rpm: Curl error (28): Timeout was reached for https://ftp-nyc.osuosl.org/pub/fedora-epel/9/Everything/x86_64/Packages/g/grpc-data-1.46.7-10.el9.noarch.rpm [Failed to connect to ftp-nyc.osuosl.org port 443: Connection timed out] 2024-03-19T02:35:21.190 INFO:teuthology.orchestra.run.smithi155.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 150 B/s | 19 kB 02:12 2024-03-19T02:35:21.399 INFO:teuthology.orchestra.run.smithi155.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 2.3 kB/s | 309 kB 02:12 2024-03-19T02:35:21.432 INFO:teuthology.orchestra.run.smithi155.stdout:(109/174): abseil-cpp-20211102.0-3.el9.x86_64.r 4.1 kB/s | 548 kB 02:12 2024-03-19T02:35:21.466 INFO:teuthology.orchestra.run.smithi155.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 389 kB/s | 26 kB 00:00 2024-03-19T02:35:21.499 INFO:teuthology.orchestra.run.smithi155.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 727 kB/s | 49 kB 00:00 2024-03-19T02:35:21.550 INFO:teuthology.orchestra.run.smithi155.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 806 kB/s | 67 kB 00:00 2024-03-19T02:35:21.617 INFO:teuthology.orchestra.run.smithi155.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2024-03-19T02:35:21.784 INFO:teuthology.orchestra.run.smithi155.stdout:(114/174): libarrow-9.0.0-10.el9.x86_64.rpm 7.4 MB/s | 4.4 MB 00:00 2024-03-19T02:35:21.809 INFO:teuthology.orchestra.run.smithi155.stdout:(115/174): python3-asyncssh-2.13.2-4.el9.noarch 2.8 MB/s | 548 kB 00:00 2024-03-19T02:35:21.842 INFO:teuthology.orchestra.run.smithi155.stdout:(116/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.8 MB/s | 838 kB 00:00 2024-03-19T02:35:21.909 INFO:teuthology.orchestra.run.smithi155.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 321 kB/s | 32 kB 00:00 2024-03-19T02:35:21.935 INFO:teuthology.orchestra.run.smithi155.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 163 kB/s | 15 kB 00:00 2024-03-19T02:35:21.960 INFO:teuthology.orchestra.run.smithi155.stdout:(119/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 247 kB/s | 43 kB 00:00 2024-03-19T02:35:21.994 INFO:teuthology.orchestra.run.smithi155.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-03-19T02:35:22.035 INFO:teuthology.orchestra.run.smithi155.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-03-19T02:35:22.060 INFO:teuthology.orchestra.run.smithi155.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.9 MB/s | 303 kB 00:00 2024-03-19T02:35:22.086 INFO:teuthology.orchestra.run.smithi155.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 383 kB/s | 35 kB 00:00 2024-03-19T02:35:22.136 INFO:teuthology.orchestra.run.smithi155.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 2.0 MB/s | 203 kB 00:00 2024-03-19T02:35:22.220 INFO:teuthology.orchestra.run.smithi155.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 13 MB/s | 2.0 MB 00:00 2024-03-19T02:35:22.245 INFO:teuthology.orchestra.run.smithi155.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 98 kB/s | 11 kB 00:00 2024-03-19T02:35:22.273 INFO:teuthology.orchestra.run.smithi155.stdout:(127/174): python3-grpcio-tools-1.46.7-10.el9.x 784 kB/s | 144 kB 00:00 2024-03-19T02:35:22.296 INFO:teuthology.orchestra.run.smithi155.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 234 kB/s | 18 kB 00:00 2024-03-19T02:35:22.321 INFO:teuthology.orchestra.run.smithi155.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 307 kB/s | 23 kB 00:00 2024-03-19T02:35:22.346 INFO:teuthology.orchestra.run.smithi155.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 258 kB/s | 19 kB 00:00 2024-03-19T02:35:22.371 INFO:teuthology.orchestra.run.smithi155.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 261 kB/s | 20 kB 00:00 2024-03-19T02:35:22.421 INFO:teuthology.orchestra.run.smithi155.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 10 MB/s | 1.0 MB 00:00 2024-03-19T02:35:22.447 INFO:teuthology.orchestra.run.smithi155.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 460 kB/s | 46 kB 00:00 2024-03-19T02:35:22.472 INFO:teuthology.orchestra.run.smithi155.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 782 kB/s | 79 kB 00:00 2024-03-19T02:35:22.498 INFO:teuthology.orchestra.run.smithi155.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 761 kB/s | 58 kB 00:00 2024-03-19T02:35:22.556 INFO:teuthology.orchestra.run.smithi155.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-03-19T02:35:22.582 INFO:teuthology.orchestra.run.smithi155.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 151 kB/s | 16 kB 00:00 2024-03-19T02:35:22.607 INFO:teuthology.orchestra.run.smithi155.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 826 kB/s | 90 kB 00:00 2024-03-19T02:35:22.649 INFO:teuthology.orchestra.run.smithi155.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 306 kB/s | 28 kB 00:00 2024-03-19T02:35:22.674 INFO:teuthology.orchestra.run.smithi155.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 334 kB/s | 31 kB 00:00 2024-03-19T02:35:22.716 INFO:teuthology.orchestra.run.smithi155.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-03-19T02:35:22.741 INFO:teuthology.orchestra.run.smithi155.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 641 kB/s | 59 kB 00:00 2024-03-19T02:35:22.766 INFO:teuthology.orchestra.run.smithi155.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 388 kB/s | 36 kB 00:00 2024-03-19T02:35:22.800 INFO:teuthology.orchestra.run.smithi155.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 608 kB/s | 51 kB 00:00 2024-03-19T02:35:22.834 INFO:teuthology.orchestra.run.smithi155.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-19T02:35:22.859 INFO:teuthology.orchestra.run.smithi155.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 974 kB/s | 90 kB 00:00 2024-03-19T02:35:22.884 INFO:teuthology.orchestra.run.smithi155.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-03-19T02:35:22.910 INFO:teuthology.orchestra.run.smithi155.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 264 kB/s | 20 kB 00:00 2024-03-19T02:35:22.943 INFO:teuthology.orchestra.run.smithi155.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-03-19T02:35:23.010 INFO:teuthology.orchestra.run.smithi155.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-03-19T02:35:23.177 INFO:teuthology.orchestra.run.smithi155.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 1.9 MB/s | 321 kB 00:00 2024-03-19T02:35:23.369 INFO:teuthology.orchestra.run.smithi155.stdout:(152/174): librados2-19.0.0-2206.gb6ae2491.el9. 7.4 MB/s | 3.4 MB 00:00 2024-03-19T02:35:23.427 INFO:teuthology.orchestra.run.smithi155.stdout:(153/174): NetworkManager-1.47.2-1.el9.x86_64.r 9.2 MB/s | 2.3 MB 00:00 2024-03-19T02:35:23.469 INFO:teuthology.orchestra.run.smithi155.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 1.0 MB/s | 42 kB 00:00 2024-03-19T02:35:23.511 INFO:teuthology.orchestra.run.smithi155.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 5.8 MB/s | 248 kB 00:00 2024-03-19T02:35:23.553 INFO:teuthology.orchestra.run.smithi155.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.7 MB/s | 71 kB 00:00 2024-03-19T02:35:23.595 INFO:teuthology.orchestra.run.smithi155.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 872 kB/s | 36 kB 00:00 2024-03-19T02:35:23.637 INFO:teuthology.orchestra.run.smithi155.stdout:(158/174): NetworkManager-libnm-1.47.2-1.el9.x8 6.8 MB/s | 1.8 MB 00:00 2024-03-19T02:35:23.656 INFO:teuthology.orchestra.run.smithi155.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 3.1 MB/s | 192 kB 00:00 2024-03-19T02:35:23.682 INFO:teuthology.orchestra.run.smithi155.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 5.6 MB/s | 256 kB 00:00 2024-03-19T02:35:23.707 INFO:teuthology.orchestra.run.smithi155.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 458 kB/s | 23 kB 00:00 2024-03-19T02:35:23.732 INFO:teuthology.orchestra.run.smithi155.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 108 kB 00:00 2024-03-19T02:35:23.758 INFO:teuthology.orchestra.run.smithi155.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 3.0 MB/s | 155 kB 00:00 2024-03-19T02:35:23.783 INFO:teuthology.orchestra.run.smithi155.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 136 kB 00:00 2024-03-19T02:35:23.808 INFO:teuthology.orchestra.run.smithi155.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 64 kB 00:00 2024-03-19T02:35:23.834 INFO:teuthology.orchestra.run.smithi155.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 570 kB/s | 29 kB 00:00 2024-03-19T02:35:23.868 INFO:teuthology.orchestra.run.smithi155.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 21 MB/s | 1.2 MB 00:00 2024-03-19T02:35:23.926 INFO:teuthology.orchestra.run.smithi155.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 23 MB/s | 2.2 MB 00:00 2024-03-19T02:35:23.977 INFO:teuthology.orchestra.run.smithi155.stdout:(169/174): util-linux-2.37.4-18.el9.x86_64.rpm 21 MB/s | 2.3 MB 00:00 2024-03-19T02:35:24.027 INFO:teuthology.orchestra.run.smithi155.stdout:(170/174): librbd1-19.0.0-2206.gb6ae2491.el9.x8 3.0 MB/s | 3.1 MB 00:01 2024-03-19T02:35:24.052 INFO:teuthology.orchestra.run.smithi155.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 413 kB/s | 31 kB 00:00 2024-03-19T02:35:24.086 INFO:teuthology.orchestra.run.smithi155.stdout:(172/174): util-linux-core-2.37.4-18.el9.x86_64 2.9 MB/s | 465 kB 00:00 2024-03-19T02:35:24.211 INFO:teuthology.orchestra.run.smithi155.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 5.8 MB/s | 1.1 MB 00:00 2024-03-19T02:35:24.236 INFO:teuthology.orchestra.run.smithi155.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 405 kB/s | 74 kB 00:00 2024-03-19T02:35:24.240 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:35:24.240 INFO:teuthology.orchestra.run.smithi155.stdout:Total 1.5 MB/s | 224 MB 02:27 2024-03-19T02:35:24.693 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-03-19T02:35:24.970 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-03-19T02:35:24.970 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-03-19T02:35:27.034 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-03-19T02:35:27.034 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-03-19T02:35:33.958 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-03-19T02:35:34.049 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-19T02:35:34.084 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-19T02:35:34.130 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-19T02:35:34.139 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-19T02:35:34.190 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-19T02:35:34.242 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-19T02:35:34.275 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-19T02:35:34.300 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-19T02:35:34.357 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-19T02:35:34.684 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-19T02:35:34.751 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T02:35:34.873 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T02:35:34.902 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T02:35:34.994 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T02:35:35.033 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-19T02:35:35.088 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-19T02:35:35.285 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-19T02:35:35.346 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-19T02:35:35.374 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T02:35:35.477 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T02:35:35.500 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-19T02:35:35.566 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-19T02:35:35.604 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-19T02:35:35.627 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-19T02:35:35.642 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-19T02:35:35.780 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-19T02:35:35.808 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-19T02:35:35.848 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-19T02:35:36.030 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-19T02:35:36.150 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-19T02:35:36.210 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-19T02:35:36.310 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T02:35:36.362 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T02:35:36.591 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T02:35:36.705 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-19T02:35:36.804 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-19T02:35:36.850 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-19T02:35:36.885 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-19T02:35:36.929 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-19T02:35:37.023 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-19T02:35:37.082 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-19T02:35:37.120 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-19T02:35:37.193 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-19T02:35:37.228 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-19T02:35:37.279 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-19T02:35:37.321 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-19T02:35:37.348 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-19T02:35:37.417 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-19T02:35:37.453 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 39/198 2024-03-19T02:35:37.496 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 40/198 2024-03-19T02:35:37.525 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 41/198 2024-03-19T02:35:37.594 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-19T02:35:37.640 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-19T02:35:37.721 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-19T02:35:37.863 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-19T02:35:37.919 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-19T02:35:38.445 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-19T02:35:38.531 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-19T02:35:38.607 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-19T02:35:38.660 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-19T02:35:38.668 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-19T02:35:38.716 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-19T02:35:38.758 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-19T02:35:38.796 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 53/198 2024-03-19T02:35:38.827 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-19T02:35:39.111 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-19T02:35:39.286 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-19T02:35:39.371 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-19T02:35:39.414 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-19T02:35:39.568 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-19T02:35:39.617 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-19T02:35:39.672 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-19T02:35:39.745 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-19T02:35:39.777 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-19T02:35:39.814 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-19T02:35:39.860 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-19T02:35:39.896 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-19T02:35:39.948 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-19T02:35:39.998 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-19T02:35:40.131 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-19T02:35:40.170 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-19T02:35:40.201 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-19T02:35:40.242 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-19T02:35:40.336 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-19T02:35:40.393 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-19T02:35:40.433 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-19T02:35:40.459 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-19T02:35:40.483 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-19T02:35:40.561 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-19T02:35:40.692 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-19T02:35:40.800 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-19T02:35:40.842 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-19T02:35:40.899 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-19T02:35:40.963 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-19T02:35:41.155 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-19T02:35:41.213 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-19T02:35:41.341 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-19T02:35:41.387 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-19T02:35:41.418 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-19T02:35:41.450 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-19T02:35:41.478 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-19T02:35:41.512 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-19T02:35:41.551 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-19T02:35:41.583 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-19T02:35:41.613 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-19T02:35:41.689 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-19T02:35:41.732 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-19T02:35:41.762 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-19T02:35:41.803 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-19T02:35:41.838 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-19T02:35:41.927 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-19T02:35:42.017 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-19T02:35:42.061 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-19T02:35:42.092 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-19T02:35:42.139 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-19T02:35:42.238 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-19T02:35:42.486 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-19T02:35:43.383 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-19T02:35:43.433 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-19T02:35:43.467 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-19T02:35:43.515 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-19T02:35:43.602 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-19T02:35:43.635 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-19T02:35:43.813 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-19T02:35:43.849 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-19T02:35:43.928 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-19T02:35:44.572 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-19T02:35:44.687 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-19T02:35:45.904 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-19T02:35:45.993 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-19T02:35:46.033 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-19T02:35:46.064 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-19T02:35:46.091 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-19T02:35:46.403 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-19T02:35:46.430 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-19T02:35:46.521 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-19T02:35:46.551 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 125/198 2024-03-19T02:35:46.596 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-19T02:35:47.074 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-19T02:35:47.108 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-19T02:35:47.179 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-19T02:35:47.202 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 129/198 2024-03-19T02:35:49.485 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T02:35:49.515 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T02:35:50.076 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T02:35:50.110 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-19T02:35:50.202 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-19T02:35:50.203 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-19T02:35:50.203 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-19T02:35:50.203 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-19T02:35:50.237 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-19T02:36:01.939 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-19T02:36:01.939 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /sys 2024-03-19T02:36:01.939 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /proc 2024-03-19T02:36:01.939 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /mnt 2024-03-19T02:36:01.940 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /var/tmp 2024-03-19T02:36:01.940 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /home 2024-03-19T02:36:01.940 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /root 2024-03-19T02:36:01.940 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /tmp 2024-03-19T02:36:01.940 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-19T02:36:02.110 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-19T02:36:02.125 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-19T02:36:02.125 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:36:02.125 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-19T02:36:02.125 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-19T02:36:03.597 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-19T02:36:03.622 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-19T02:36:03.623 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:36:03.623 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-19T02:36:03.623 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-19T02:36:03.623 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-19T02:36:03.623 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-19T02:36:03.872 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-19T02:36:03.894 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-19T02:36:03.894 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:36:03.894 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-19T02:36:03.894 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-19T02:36:03.894 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-19T02:36:03.895 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-19T02:36:04.340 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-19T02:36:04.362 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-19T02:36:04.363 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:36:04.363 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-19T02:36:04.363 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-19T02:36:04.363 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-19T02:36:04.363 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-19T02:36:04.459 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-19T02:36:04.936 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-19T02:36:04.966 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-19T02:36:04.998 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 139/198 2024-03-19T02:36:05.057 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 140/198 2024-03-19T02:36:05.081 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-19T02:36:05.313 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-19T02:36:05.348 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-19T02:36:06.318 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-19T02:36:06.349 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-19T02:36:06.448 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-19T02:36:06.613 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 144/198 2024-03-19T02:36:06.643 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-19T02:36:06.666 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-19T02:36:06.666 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:36:06.666 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-19T02:36:06.666 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-19T02:36:06.666 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-19T02:36:06.666 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-19T02:36:06.710 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-19T02:36:06.720 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-19T02:36:06.778 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 147/198 2024-03-19T02:36:07.673 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-19T02:36:07.721 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-19T02:36:07.743 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-19T02:36:07.743 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:36:07.743 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-19T02:36:07.744 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-19T02:36:07.744 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-19T02:36:07.744 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-19T02:36:07.788 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-19T02:36:07.809 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-19T02:36:07.810 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:36:07.810 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-19T02:36:07.810 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-19T02:36:07.864 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T02:36:07.886 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T02:36:07.886 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:36:07.886 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-19T02:36:07.886 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-19T02:36:07.886 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-19T02:36:07.886 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-19T02:36:08.198 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-19T02:36:08.220 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-19T02:36:08.220 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:36:08.220 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-19T02:36:08.220 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T02:36:08.220 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T02:36:08.220 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-19T02:36:12.669 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-19T02:36:12.727 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 154/198 2024-03-19T02:36:12.758 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 155/198 2024-03-19T02:36:12.785 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-19T02:36:12.965 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 157/198 2024-03-19T02:36:13.076 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-19T02:36:13.125 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 159/198 2024-03-19T02:36:13.179 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-19T02:36:13.251 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-19T02:36:13.286 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-19T02:36:13.318 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 163/198 2024-03-19T02:36:13.351 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-19T02:36:13.381 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-19T02:36:13.447 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-19T02:36:13.522 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-19T02:36:13.562 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-19T02:36:14.147 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T02:36:14.186 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T02:36:14.206 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T02:36:15.024 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 170/198 2024-03-19T02:36:15.117 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-19T02:36:15.158 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-19T02:36:15.254 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-19T02:36:15.288 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-19T02:36:15.313 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-19T02:36:15.338 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-19T02:36:15.395 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-19T02:36:15.395 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-19T02:36:15.469 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-19T02:36:15.508 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-19T02:36:15.508 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-19T02:36:15.592 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-19T02:36:15.619 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-19T02:36:15.652 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-19T02:36:15.678 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-19T02:36:15.702 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-19T02:36:15.737 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-19T02:36:15.784 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-19T02:36:15.816 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-19T02:36:15.841 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-19T02:36:15.875 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-19T02:36:15.900 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-19T02:36:15.925 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-19T02:36:15.931 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T02:36:15.932 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T02:36:16.129 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T02:36:16.129 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-19T02:36:16.211 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-19T02:36:16.236 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-19T02:36:16.262 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-19T02:36:16.295 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-19T02:36:16.321 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-19T02:36:16.386 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-19T02:36:21.241 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-19T02:36:21.241 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 1/198 2024-03-19T02:36:21.241 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2/198 2024-03-19T02:36:21.241 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 3/198 2024-03-19T02:36:21.241 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 4/198 2024-03-19T02:36:21.241 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 5/198 2024-03-19T02:36:21.241 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 6/198 2024-03-19T02:36:21.241 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 7/198 2024-03-19T02:36:21.241 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 8/198 2024-03-19T02:36:21.242 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 9/198 2024-03-19T02:36:21.242 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T02:36:21.242 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T02:36:21.242 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-19T02:36:21.242 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 13/198 2024-03-19T02:36:21.242 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 14/198 2024-03-19T02:36:21.242 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 15/198 2024-03-19T02:36:21.242 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T02:36:21.242 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 17/198 2024-03-19T02:36:21.242 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 18/198 2024-03-19T02:36:21.242 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 19/198 2024-03-19T02:36:21.242 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 20/198 2024-03-19T02:36:21.242 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 21/198 2024-03-19T02:36:21.243 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 22/198 2024-03-19T02:36:21.243 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 23/198 2024-03-19T02:36:21.243 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 24/198 2024-03-19T02:36:21.243 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 25/198 2024-03-19T02:36:21.243 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 26/198 2024-03-19T02:36:21.243 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 27/198 2024-03-19T02:36:21.243 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 28/198 2024-03-19T02:36:21.243 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 29/198 2024-03-19T02:36:21.258 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 30/198 2024-03-19T02:36:21.258 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 31/198 2024-03-19T02:36:21.258 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 32/198 2024-03-19T02:36:21.258 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 33/198 2024-03-19T02:36:21.259 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 34/198 2024-03-19T02:36:21.259 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 35/198 2024-03-19T02:36:21.259 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 36/198 2024-03-19T02:36:21.259 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 37/198 2024-03-19T02:36:21.259 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 38/198 2024-03-19T02:36:21.259 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 39/198 2024-03-19T02:36:21.259 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-19T02:36:21.259 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-19T02:36:21.259 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-19T02:36:21.259 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-19T02:36:21.259 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-19T02:36:21.259 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-19T02:36:21.259 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-19T02:36:21.260 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-19T02:36:21.260 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-19T02:36:21.260 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-19T02:36:21.260 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-19T02:36:21.260 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-19T02:36:21.260 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-19T02:36:21.260 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-19T02:36:21.260 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-19T02:36:21.260 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-19T02:36:21.260 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-19T02:36:21.260 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-19T02:36:21.260 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-19T02:36:21.260 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-19T02:36:21.260 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-19T02:36:21.261 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-19T02:36:21.261 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-19T02:36:21.261 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-19T02:36:21.261 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-19T02:36:21.261 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-19T02:36:21.261 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-19T02:36:21.261 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-19T02:36:21.261 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-19T02:36:21.261 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-19T02:36:21.261 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-19T02:36:21.261 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-19T02:36:21.261 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-19T02:36:21.261 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-19T02:36:21.262 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-19T02:36:21.262 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-19T02:36:21.262 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-19T02:36:21.262 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-19T02:36:21.262 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-19T02:36:21.262 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-19T02:36:21.262 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-19T02:36:21.262 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-19T02:36:21.262 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-19T02:36:21.262 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-19T02:36:21.262 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-19T02:36:21.262 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-19T02:36:21.262 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-19T02:36:21.262 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-19T02:36:21.263 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-19T02:36:21.263 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-19T02:36:21.263 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-19T02:36:21.263 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-19T02:36:21.263 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-19T02:36:21.263 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-19T02:36:21.263 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-19T02:36:21.263 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-19T02:36:21.263 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-19T02:36:21.263 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-19T02:36:21.263 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-19T02:36:21.263 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-19T02:36:21.263 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-19T02:36:21.264 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-19T02:36:21.264 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-19T02:36:21.264 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-19T02:36:21.264 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-19T02:36:21.264 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-19T02:36:21.264 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-19T02:36:21.264 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-19T02:36:21.264 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-19T02:36:21.264 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-19T02:36:21.264 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-19T02:36:21.264 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-19T02:36:21.264 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-19T02:36:21.264 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-19T02:36:21.264 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-19T02:36:21.265 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-19T02:36:21.265 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-19T02:36:21.265 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-19T02:36:21.265 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-19T02:36:21.265 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-19T02:36:21.265 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-19T02:36:21.265 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-19T02:36:21.265 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-19T02:36:21.265 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-19T02:36:21.265 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-19T02:36:21.265 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-19T02:36:21.265 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-19T02:36:21.265 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-19T02:36:21.265 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-19T02:36:21.265 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-19T02:36:21.266 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-19T02:36:21.266 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-19T02:36:21.266 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-19T02:36:21.266 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-19T02:36:21.266 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-19T02:36:21.266 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-19T02:36:21.266 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-19T02:36:21.266 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-19T02:36:21.266 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-19T02:36:21.266 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-19T02:36:21.266 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-19T02:36:21.266 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-19T02:36:21.266 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-19T02:36:21.266 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-19T02:36:21.267 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-19T02:36:21.267 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-19T02:36:21.267 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-19T02:36:21.267 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-19T02:36:21.267 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-19T02:36:21.267 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-19T02:36:21.267 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-19T02:36:21.267 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T02:36:21.267 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-19T02:36:21.267 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-19T02:36:21.267 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-19T02:36:21.267 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-19T02:36:21.267 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-19T02:36:21.267 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-19T02:36:21.267 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-19T02:36:21.268 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-19T02:36:21.268 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-19T02:36:21.268 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-19T02:36:21.268 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-19T02:36:21.268 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-19T02:36:21.268 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-19T02:36:21.268 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-19T02:36:21.268 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-19T02:36:21.268 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-19T02:36:21.268 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-19T02:36:21.268 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-19T02:36:21.268 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-19T02:36:21.268 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-19T02:36:21.268 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-19T02:36:21.268 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-19T02:36:21.269 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-19T02:36:21.269 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-19T02:36:21.269 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-19T02:36:21.269 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-19T02:36:21.269 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-19T02:36:21.269 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-19T02:36:21.269 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-19T02:36:21.269 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-19T02:36:21.269 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-19T02:36:21.269 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-19T02:36:21.269 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-19T02:36:21.269 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-19T02:36:21.269 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-19T02:36:21.269 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-19T02:36:21.270 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-19T02:36:21.270 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-19T02:36:21.270 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-19T02:36:21.270 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-19T02:36:21.270 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-19T02:36:21.270 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-19T02:36:21.270 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-19T02:36:21.270 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-19T02:36:21.270 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-19T02:36:21.270 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-19T02:36:24.410 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-19T02:36:24.410 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-19T02:36:24.410 INFO:teuthology.orchestra.run.smithi155.stdout:Upgraded: 2024-03-19T02:36:24.410 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-19T02:36:24.410 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-19T02:36:24.410 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-19T02:36:24.411 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-19T02:36:24.411 INFO:teuthology.orchestra.run.smithi155.stdout: acl-2.3.1-4.el9.x86_64 2024-03-19T02:36:24.411 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-19T02:36:24.411 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-19T02:36:24.411 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-19T02:36:24.411 INFO:teuthology.orchestra.run.smithi155.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-19T02:36:24.411 INFO:teuthology.orchestra.run.smithi155.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-19T02:36:24.411 INFO:teuthology.orchestra.run.smithi155.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-19T02:36:24.411 INFO:teuthology.orchestra.run.smithi155.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-19T02:36:24.411 INFO:teuthology.orchestra.run.smithi155.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-19T02:36:24.411 INFO:teuthology.orchestra.run.smithi155.stdout: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:36:24.411 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:36:24.411 INFO:teuthology.orchestra.run.smithi155.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-19T02:36:24.411 INFO:teuthology.orchestra.run.smithi155.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-19T02:36:24.412 INFO:teuthology.orchestra.run.smithi155.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-19T02:36:24.412 INFO:teuthology.orchestra.run.smithi155.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-19T02:36:24.412 INFO:teuthology.orchestra.run.smithi155.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-19T02:36:24.412 INFO:teuthology.orchestra.run.smithi155.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-19T02:36:24.412 INFO:teuthology.orchestra.run.smithi155.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-19T02:36:24.412 INFO:teuthology.orchestra.run.smithi155.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-19T02:36:24.412 INFO:teuthology.orchestra.run.smithi155.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-19T02:36:24.412 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-03-19T02:36:24.412 INFO:teuthology.orchestra.run.smithi155.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-19T02:36:24.412 INFO:teuthology.orchestra.run.smithi155.stdout: automake-1.16.2-8.el9.noarch 2024-03-19T02:36:24.413 INFO:teuthology.orchestra.run.smithi155.stdout: bison-3.7.4-5.el9.x86_64 2024-03-19T02:36:24.413 INFO:teuthology.orchestra.run.smithi155.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-19T02:36:24.413 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:36:24.413 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:36:24.413 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:36:24.413 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:36:24.413 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:36:24.413 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:36:24.413 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:36:24.413 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:36:24.413 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:36:24.413 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:36:24.413 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:36:24.414 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:36:24.414 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:36:24.414 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:36:24.414 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:36:24.414 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:36:24.414 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:36:24.414 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:36:24.414 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:36:24.414 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:36:24.414 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:36:24.414 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:36:24.415 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:36:24.415 INFO:teuthology.orchestra.run.smithi155.stdout: cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:36:24.415 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-19T02:36:24.415 INFO:teuthology.orchestra.run.smithi155.stdout: flex-2.6.4-9.el9.x86_64 2024-03-19T02:36:24.415 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-19T02:36:24.415 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-19T02:36:24.415 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-19T02:36:24.415 INFO:teuthology.orchestra.run.smithi155.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-19T02:36:24.415 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-19T02:36:24.415 INFO:teuthology.orchestra.run.smithi155.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-19T02:36:24.415 INFO:teuthology.orchestra.run.smithi155.stdout: jq-1.6-16.el9.x86_64 2024-03-19T02:36:24.415 INFO:teuthology.orchestra.run.smithi155.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-19T02:36:24.415 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-19T02:36:24.416 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-19T02:36:24.416 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:36:24.416 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:36:24.416 INFO:teuthology.orchestra.run.smithi155.stdout: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:36:24.416 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-19T02:36:24.416 INFO:teuthology.orchestra.run.smithi155.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-19T02:36:24.416 INFO:teuthology.orchestra.run.smithi155.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-19T02:36:24.416 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-19T02:36:24.416 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-19T02:36:24.416 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:36:24.416 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:36:24.417 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-19T02:36:24.417 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:36:24.417 INFO:teuthology.orchestra.run.smithi155.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-19T02:36:24.417 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-19T02:36:24.417 INFO:teuthology.orchestra.run.smithi155.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-19T02:36:24.417 INFO:teuthology.orchestra.run.smithi155.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-19T02:36:24.417 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-19T02:36:24.417 INFO:teuthology.orchestra.run.smithi155.stdout: lua-5.4.4-4.el9.x86_64 2024-03-19T02:36:24.417 INFO:teuthology.orchestra.run.smithi155.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-19T02:36:24.417 INFO:teuthology.orchestra.run.smithi155.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-19T02:36:24.417 INFO:teuthology.orchestra.run.smithi155.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-19T02:36:24.417 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-19T02:36:24.417 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-19T02:36:24.417 INFO:teuthology.orchestra.run.smithi155.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-19T02:36:24.418 INFO:teuthology.orchestra.run.smithi155.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-19T02:36:24.418 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-13.11-1.el9.x86_64 2024-03-19T02:36:24.418 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-19T02:36:24.418 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-19T02:36:24.418 INFO:teuthology.orchestra.run.smithi155.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-19T02:36:24.418 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-19T02:36:24.418 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-19T02:36:24.418 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-19T02:36:24.418 INFO:teuthology.orchestra.run.smithi155.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-19T02:36:24.418 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-19T02:36:24.419 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-19T02:36:24.419 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-19T02:36:24.419 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:36:24.419 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:36:24.419 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:36:24.419 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-19T02:36:24.419 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-19T02:36:24.419 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-19T02:36:24.419 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-19T02:36:24.419 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-19T02:36:24.419 INFO:teuthology.orchestra.run.smithi155.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-19T02:36:24.419 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-19T02:36:24.420 INFO:teuthology.orchestra.run.smithi155.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-19T02:36:24.420 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-19T02:36:24.420 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-19T02:36:24.420 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-19T02:36:24.420 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-19T02:36:24.420 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-19T02:36:24.420 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-19T02:36:24.420 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-19T02:36:24.420 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-19T02:36:24.420 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-19T02:36:24.420 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-19T02:36:24.421 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-19T02:36:24.421 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-19T02:36:24.421 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-19T02:36:24.421 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-19T02:36:24.421 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-19T02:36:24.421 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-19T02:36:24.421 INFO:teuthology.orchestra.run.smithi155.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-19T02:36:24.421 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-19T02:36:24.422 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-19T02:36:24.422 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-19T02:36:24.422 INFO:teuthology.orchestra.run.smithi155.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-19T02:36:24.422 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-19T02:36:24.422 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ply-3.11-14.el9.noarch 2024-03-19T02:36:24.422 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-19T02:36:24.422 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-19T02:36:24.422 INFO:teuthology.orchestra.run.smithi155.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-19T02:36:24.422 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-19T02:36:24.422 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-19T02:36:24.422 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-19T02:36:24.422 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-19T02:36:24.423 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-19T02:36:24.423 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-19T02:36:24.423 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:36:24.423 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:36:24.423 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-19T02:36:24.423 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-19T02:36:24.423 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-19T02:36:24.423 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:36:24.423 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-19T02:36:24.423 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-19T02:36:24.423 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-19T02:36:24.423 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-19T02:36:24.423 INFO:teuthology.orchestra.run.smithi155.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-19T02:36:24.423 INFO:teuthology.orchestra.run.smithi155.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-19T02:36:24.423 INFO:teuthology.orchestra.run.smithi155.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-19T02:36:24.424 INFO:teuthology.orchestra.run.smithi155.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-19T02:36:24.424 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-19T02:36:24.424 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-19T02:36:24.424 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-19T02:36:24.424 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-19T02:36:24.424 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-19T02:36:24.424 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-19T02:36:24.424 INFO:teuthology.orchestra.run.smithi155.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-19T02:36:24.424 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:36:24.424 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:36:24.424 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:36:24.424 INFO:teuthology.orchestra.run.smithi155.stdout: re2-1:20211101-3.el9.x86_64 2024-03-19T02:36:24.424 INFO:teuthology.orchestra.run.smithi155.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-19T02:36:24.425 INFO:teuthology.orchestra.run.smithi155.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-19T02:36:24.425 INFO:teuthology.orchestra.run.smithi155.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-19T02:36:24.425 INFO:teuthology.orchestra.run.smithi155.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-19T02:36:24.425 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-19T02:36:24.425 INFO:teuthology.orchestra.run.smithi155.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-19T02:36:24.426 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-19T02:36:24.426 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-03-19T02:36:24.929 DEBUG:teuthology.parallel:result is None 2024-03-19T02:37:20.595 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - CRB 24 kB/s | 6.1 MB 04:25 2024-03-19T02:37:21.887 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - CRB 23 kB/s | 6.1 MB 04:27 2024-03-19T02:37:22.044 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 16 kB 00:00 2024-03-19T02:37:23.362 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 16 kB 00:00 2024-03-19T02:37:24.747 INFO:teuthology.orchestra.run.smithi033.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-19T02:37:29.780 INFO:teuthology.orchestra.run.smithi033.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-03-19T02:37:32.254 INFO:teuthology.orchestra.run.smithi033.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-19T02:37:32.255 INFO:teuthology.orchestra.run.smithi033.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-19T02:37:32.270 INFO:teuthology.orchestra.run.smithi033.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-19T02:37:32.272 INFO:teuthology.orchestra.run.smithi033.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-19T02:37:32.273 INFO:teuthology.orchestra.run.smithi033.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-19T02:37:32.274 INFO:teuthology.orchestra.run.smithi033.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-19T02:37:32.276 INFO:teuthology.orchestra.run.smithi033.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-19T02:37:32.279 INFO:teuthology.orchestra.run.smithi033.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-19T02:37:32.280 INFO:teuthology.orchestra.run.smithi033.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-19T02:37:32.366 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-19T02:37:32.382 INFO:teuthology.orchestra.run.smithi033.stdout:============================================================================================ 2024-03-19T02:37:32.382 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-19T02:37:32.382 INFO:teuthology.orchestra.run.smithi033.stdout:============================================================================================ 2024-03-19T02:37:32.382 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-19T02:37:32.382 INFO:teuthology.orchestra.run.smithi033.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-19T02:37:32.382 INFO:teuthology.orchestra.run.smithi033.stdout: ceph x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 6.6 k 2024-03-19T02:37:32.383 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-base x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.4 M 2024-03-19T02:37:32.383 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 891 k 2024-03-19T02:37:32.383 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 145 k 2024-03-19T02:37:32.383 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 1.5 M 2024-03-19T02:37:32.383 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 139 k 2024-03-19T02:37:32.383 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 1.8 M 2024-03-19T02:37:32.383 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 7.4 M 2024-03-19T02:37:32.383 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-rook noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 49 k 2024-03-19T02:37:32.383 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-radosgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 9.8 M 2024-03-19T02:37:32.383 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-test x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 48 M 2024-03-19T02:37:32.384 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-volume noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 264 k 2024-03-19T02:37:32.384 INFO:teuthology.orchestra.run.smithi033.stdout: cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 749 k 2024-03-19T02:37:32.384 INFO:teuthology.orchestra.run.smithi033.stdout: cephfs-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 221 k 2024-03-19T02:37:32.384 INFO:teuthology.orchestra.run.smithi033.stdout: cephfs-shell x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 27 k 2024-03-19T02:37:32.384 INFO:teuthology.orchestra.run.smithi033.stdout: cephfs-top noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 18 k 2024-03-19T02:37:32.384 INFO:teuthology.orchestra.run.smithi033.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-19T02:37:32.384 INFO:teuthology.orchestra.run.smithi033.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-19T02:37:32.384 INFO:teuthology.orchestra.run.smithi033.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-19T02:37:32.384 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 32 k 2024-03-19T02:37:32.384 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 746 k 2024-03-19T02:37:32.385 INFO:teuthology.orchestra.run.smithi033.stdout: librados-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 127 k 2024-03-19T02:37:32.385 INFO:teuthology.orchestra.run.smithi033.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-19T02:37:32.385 INFO:teuthology.orchestra.run.smithi033.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-19T02:37:32.385 INFO:teuthology.orchestra.run.smithi033.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-19T02:37:32.385 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-19T02:37:32.385 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-19T02:37:32.385 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-19T02:37:32.385 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cephfs x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 162 k 2024-03-19T02:37:32.385 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rados x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 321 k 2024-03-19T02:37:32.385 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 298 k 2024-03-19T02:37:32.386 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 99 k 2024-03-19T02:37:32.386 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 85 k 2024-03-19T02:37:32.386 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-19T02:37:32.386 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-nbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 170 k 2024-03-19T02:37:32.386 INFO:teuthology.orchestra.run.smithi033.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-19T02:37:32.386 INFO:teuthology.orchestra.run.smithi033.stdout:Upgrading: 2024-03-19T02:37:32.386 INFO:teuthology.orchestra.run.smithi033.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-19T02:37:32.386 INFO:teuthology.orchestra.run.smithi033.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-19T02:37:32.386 INFO:teuthology.orchestra.run.smithi033.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-19T02:37:32.386 INFO:teuthology.orchestra.run.smithi033.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-19T02:37:32.386 INFO:teuthology.orchestra.run.smithi033.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-19T02:37:32.386 INFO:teuthology.orchestra.run.smithi033.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-19T02:37:32.387 INFO:teuthology.orchestra.run.smithi033.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-19T02:37:32.387 INFO:teuthology.orchestra.run.smithi033.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-19T02:37:32.387 INFO:teuthology.orchestra.run.smithi033.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-19T02:37:32.387 INFO:teuthology.orchestra.run.smithi033.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-19T02:37:32.387 INFO:teuthology.orchestra.run.smithi033.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-19T02:37:32.387 INFO:teuthology.orchestra.run.smithi033.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-19T02:37:32.387 INFO:teuthology.orchestra.run.smithi033.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-19T02:37:32.387 INFO:teuthology.orchestra.run.smithi033.stdout: librados2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.4 M 2024-03-19T02:37:32.387 INFO:teuthology.orchestra.run.smithi033.stdout: librbd1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-19T02:37:32.387 INFO:teuthology.orchestra.run.smithi033.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-19T02:37:32.387 INFO:teuthology.orchestra.run.smithi033.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-19T02:37:32.387 INFO:teuthology.orchestra.run.smithi033.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-19T02:37:32.387 INFO:teuthology.orchestra.run.smithi033.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-19T02:37:32.388 INFO:teuthology.orchestra.run.smithi033.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-19T02:37:32.388 INFO:teuthology.orchestra.run.smithi033.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-19T02:37:32.388 INFO:teuthology.orchestra.run.smithi033.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-19T02:37:32.388 INFO:teuthology.orchestra.run.smithi033.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-19T02:37:32.388 INFO:teuthology.orchestra.run.smithi033.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-19T02:37:32.388 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-19T02:37:32.388 INFO:teuthology.orchestra.run.smithi033.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-19T02:37:32.388 INFO:teuthology.orchestra.run.smithi033.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-19T02:37:32.388 INFO:teuthology.orchestra.run.smithi033.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-19T02:37:32.388 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 21 M 2024-03-19T02:37:32.388 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 34 k 2024-03-19T02:37:32.389 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mds x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 2.4 M 2024-03-19T02:37:32.389 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 248 k 2024-03-19T02:37:32.389 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mon x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 4.7 M 2024-03-19T02:37:32.389 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-osd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 17 M 2024-03-19T02:37:32.389 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 16 k 2024-03-19T02:37:32.389 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-selinux x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 25 k 2024-03-19T02:37:32.389 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-19T02:37:32.389 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-19T02:37:32.389 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-19T02:37:32.389 INFO:teuthology.orchestra.run.smithi033.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-19T02:37:32.389 INFO:teuthology.orchestra.run.smithi033.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-19T02:37:32.389 INFO:teuthology.orchestra.run.smithi033.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-19T02:37:32.389 INFO:teuthology.orchestra.run.smithi033.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-19T02:37:32.390 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-19T02:37:32.390 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-19T02:37:32.390 INFO:teuthology.orchestra.run.smithi033.stdout: libcephsqlite x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 163 k 2024-03-19T02:37:32.390 INFO:teuthology.orchestra.run.smithi033.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-19T02:37:32.390 INFO:teuthology.orchestra.run.smithi033.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-19T02:37:32.390 INFO:teuthology.orchestra.run.smithi033.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-19T02:37:32.390 INFO:teuthology.orchestra.run.smithi033.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-19T02:37:32.390 INFO:teuthology.orchestra.run.smithi033.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-19T02:37:32.390 INFO:teuthology.orchestra.run.smithi033.stdout: libradosstriper1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 501 k 2024-03-19T02:37:32.390 INFO:teuthology.orchestra.run.smithi033.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-19T02:37:32.390 INFO:teuthology.orchestra.run.smithi033.stdout: librgw2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.1 M 2024-03-19T02:37:32.390 INFO:teuthology.orchestra.run.smithi033.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-19T02:37:32.390 INFO:teuthology.orchestra.run.smithi033.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-19T02:37:32.390 INFO:teuthology.orchestra.run.smithi033.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-19T02:37:32.391 INFO:teuthology.orchestra.run.smithi033.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-19T02:37:32.391 INFO:teuthology.orchestra.run.smithi033.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-19T02:37:32.391 INFO:teuthology.orchestra.run.smithi033.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-19T02:37:32.391 INFO:teuthology.orchestra.run.smithi033.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-19T02:37:32.391 INFO:teuthology.orchestra.run.smithi033.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-19T02:37:32.391 INFO:teuthology.orchestra.run.smithi033.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-19T02:37:32.391 INFO:teuthology.orchestra.run.smithi033.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-19T02:37:32.392 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-19T02:37:32.392 INFO:teuthology.orchestra.run.smithi033.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-19T02:37:32.392 INFO:teuthology.orchestra.run.smithi033.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-19T02:37:32.392 INFO:teuthology.orchestra.run.smithi033.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-19T02:37:32.392 INFO:teuthology.orchestra.run.smithi033.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-19T02:37:32.392 INFO:teuthology.orchestra.run.smithi033.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-19T02:37:32.392 INFO:teuthology.orchestra.run.smithi033.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-19T02:37:32.392 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-19T02:37:32.392 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-argparse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 45 k 2024-03-19T02:37:32.392 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 131 k 2024-03-19T02:37:32.392 INFO:teuthology.orchestra.run.smithi033.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-19T02:37:32.392 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-19T02:37:32.392 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-19T02:37:32.392 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-19T02:37:32.393 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-19T02:37:32.393 INFO:teuthology.orchestra.run.smithi033.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-19T02:37:32.393 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-19T02:37:32.393 INFO:teuthology.orchestra.run.smithi033.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-19T02:37:32.393 INFO:teuthology.orchestra.run.smithi033.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-19T02:37:32.393 INFO:teuthology.orchestra.run.smithi033.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-19T02:37:32.393 INFO:teuthology.orchestra.run.smithi033.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-19T02:37:32.393 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-19T02:37:32.393 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-19T02:37:32.393 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-19T02:37:32.393 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-19T02:37:32.393 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-19T02:37:32.393 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-19T02:37:32.394 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-19T02:37:32.394 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-19T02:37:32.394 INFO:teuthology.orchestra.run.smithi033.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-19T02:37:32.394 INFO:teuthology.orchestra.run.smithi033.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-19T02:37:32.394 INFO:teuthology.orchestra.run.smithi033.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-19T02:37:32.394 INFO:teuthology.orchestra.run.smithi033.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-19T02:37:32.394 INFO:teuthology.orchestra.run.smithi033.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-19T02:37:32.394 INFO:teuthology.orchestra.run.smithi033.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-19T02:37:32.394 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-19T02:37:32.394 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-19T02:37:32.394 INFO:teuthology.orchestra.run.smithi033.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-19T02:37:32.394 INFO:teuthology.orchestra.run.smithi033.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-19T02:37:32.394 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-19T02:37:32.394 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-19T02:37:32.395 INFO:teuthology.orchestra.run.smithi033.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-19T02:37:32.395 INFO:teuthology.orchestra.run.smithi033.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-19T02:37:32.395 INFO:teuthology.orchestra.run.smithi033.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-19T02:37:32.395 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-19T02:37:32.395 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-19T02:37:32.395 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-19T02:37:32.395 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-19T02:37:32.395 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-19T02:37:32.395 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-19T02:37:32.395 INFO:teuthology.orchestra.run.smithi033.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-19T02:37:32.396 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-19T02:37:32.396 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-19T02:37:32.396 INFO:teuthology.orchestra.run.smithi033.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-19T02:37:32.396 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-19T02:37:32.396 INFO:teuthology.orchestra.run.smithi033.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-19T02:37:32.396 INFO:teuthology.orchestra.run.smithi033.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-19T02:37:32.396 INFO:teuthology.orchestra.run.smithi033.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-19T02:37:32.396 INFO:teuthology.orchestra.run.smithi033.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-19T02:37:32.396 INFO:teuthology.orchestra.run.smithi033.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-19T02:37:32.396 INFO:teuthology.orchestra.run.smithi033.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-19T02:37:32.397 INFO:teuthology.orchestra.run.smithi033.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-19T02:37:32.397 INFO:teuthology.orchestra.run.smithi033.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-19T02:37:32.397 INFO:teuthology.orchestra.run.smithi033.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-19T02:37:32.397 INFO:teuthology.orchestra.run.smithi033.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-19T02:37:32.397 INFO:teuthology.orchestra.run.smithi033.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-19T02:37:32.397 INFO:teuthology.orchestra.run.smithi033.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-19T02:37:32.397 INFO:teuthology.orchestra.run.smithi033.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-19T02:37:32.397 INFO:teuthology.orchestra.run.smithi033.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-19T02:37:32.397 INFO:teuthology.orchestra.run.smithi033.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-19T02:37:32.397 INFO:teuthology.orchestra.run.smithi033.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-19T02:37:32.397 INFO:teuthology.orchestra.run.smithi033.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-19T02:37:32.397 INFO:teuthology.orchestra.run.smithi033.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-19T02:37:32.397 INFO:teuthology.orchestra.run.smithi033.stdout:Installing weak dependencies: 2024-03-19T02:37:32.398 INFO:teuthology.orchestra.run.smithi033.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-19T02:37:32.398 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:37:32.398 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-19T02:37:32.398 INFO:teuthology.orchestra.run.smithi033.stdout:============================================================================================ 2024-03-19T02:37:32.398 INFO:teuthology.orchestra.run.smithi033.stdout:Install 150 Packages 2024-03-19T02:37:32.398 INFO:teuthology.orchestra.run.smithi033.stdout:Upgrade 24 Packages 2024-03-19T02:37:32.398 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:37:32.399 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 224 M 2024-03-19T02:37:32.399 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-19T02:37:33.190 INFO:teuthology.orchestra.run.smithi033.stdout:(1/174): ceph-19.0.0-2206.gb6ae2491.el9.x86_64. 38 kB/s | 6.6 kB 00:00 2024-03-19T02:37:33.398 INFO:teuthology.orchestra.run.smithi033.stdout:(2/174): ceph-fuse-19.0.0-2206.gb6ae2491.el9.x8 4.2 MB/s | 891 kB 00:00 2024-03-19T02:37:33.448 INFO:teuthology.orchestra.run.smithi033.stdout:(3/174): ceph-immutable-object-cache-19.0.0-220 2.8 MB/s | 145 kB 00:00 2024-03-19T02:37:33.682 INFO:teuthology.orchestra.run.smithi033.stdout:(4/174): ceph-base-19.0.0-2206.gb6ae2491.el9.x8 8.1 MB/s | 5.4 MB 00:00 2024-03-19T02:37:33.749 INFO:teuthology.orchestra.run.smithi033.stdout:(5/174): ceph-mds-19.0.0-2206.gb6ae2491.el9.x86 7.9 MB/s | 2.4 MB 00:00 2024-03-19T02:37:33.841 INFO:teuthology.orchestra.run.smithi033.stdout:(6/174): ceph-mgr-19.0.0-2206.gb6ae2491.el9.x86 9.4 MB/s | 1.5 MB 00:00 2024-03-19T02:37:34.191 INFO:teuthology.orchestra.run.smithi033.stdout:(7/174): ceph-mon-19.0.0-2206.gb6ae2491.el9.x86 11 MB/s | 4.7 MB 00:00 2024-03-19T02:37:34.402 INFO:teuthology.orchestra.run.smithi096.stdout:Extra Packages for Enterprise Linux 1.7 MB/s | 21 MB 00:12 2024-03-19T02:37:35.050 INFO:teuthology.orchestra.run.smithi033.stdout:(8/174): ceph-osd-19.0.0-2206.gb6ae2491.el9.x86 14 MB/s | 17 MB 00:01 2024-03-19T02:37:35.167 INFO:teuthology.orchestra.run.smithi033.stdout:(9/174): ceph-radosgw-19.0.0-2206.gb6ae2491.el9 10 MB/s | 9.8 MB 00:00 2024-03-19T02:37:35.192 INFO:teuthology.orchestra.run.smithi033.stdout:(10/174): ceph-selinux-19.0.0-2206.gb6ae2491.el 178 kB/s | 25 kB 00:00 2024-03-19T02:37:35.250 INFO:teuthology.orchestra.run.smithi033.stdout:(11/174): cephfs-mirror-19.0.0-2206.gb6ae2491.e 3.7 MB/s | 221 kB 00:00 2024-03-19T02:37:35.309 INFO:teuthology.orchestra.run.smithi033.stdout:(12/174): cephfs-shell-19.0.0-2206.gb6ae2491.el 465 kB/s | 27 kB 00:00 2024-03-19T02:37:35.368 INFO:teuthology.orchestra.run.smithi033.stdout:(13/174): libcephfs-devel-19.0.0-2206.gb6ae2491 542 kB/s | 32 kB 00:00 2024-03-19T02:37:35.451 INFO:teuthology.orchestra.run.smithi033.stdout:(14/174): libcephfs2-19.0.0-2206.gb6ae2491.el9. 8.8 MB/s | 746 kB 00:00 2024-03-19T02:37:35.527 INFO:teuthology.orchestra.run.smithi033.stdout:(15/174): libcephsqlite-19.0.0-2206.gb6ae2491.e 2.1 MB/s | 163 kB 00:00 2024-03-19T02:37:35.594 INFO:teuthology.orchestra.run.smithi033.stdout:(16/174): librados-devel-19.0.0-2206.gb6ae2491. 1.9 MB/s | 127 kB 00:00 2024-03-19T02:37:35.669 INFO:teuthology.orchestra.run.smithi033.stdout:(17/174): libradosstriper1-19.0.0-2206.gb6ae249 6.5 MB/s | 501 kB 00:00 2024-03-19T02:37:36.103 INFO:teuthology.orchestra.run.smithi033.stdout:(18/174): librgw2-19.0.0-2206.gb6ae2491.el9.x86 12 MB/s | 5.1 MB 00:00 2024-03-19T02:37:36.153 INFO:teuthology.orchestra.run.smithi033.stdout:(19/174): python3-ceph-argparse-19.0.0-2206.gb6 906 kB/s | 45 kB 00:00 2024-03-19T02:37:36.220 INFO:teuthology.orchestra.run.smithi033.stdout:(20/174): python3-ceph-common-19.0.0-2206.gb6ae 1.9 MB/s | 131 kB 00:00 2024-03-19T02:37:36.287 INFO:teuthology.orchestra.run.smithi033.stdout:(21/174): python3-cephfs-19.0.0-2206.gb6ae2491. 2.4 MB/s | 162 kB 00:00 2024-03-19T02:37:36.371 INFO:teuthology.orchestra.run.smithi033.stdout:(22/174): python3-rados-19.0.0-2206.gb6ae2491.e 3.8 MB/s | 321 kB 00:00 2024-03-19T02:37:36.446 INFO:teuthology.orchestra.run.smithi033.stdout:(23/174): python3-rbd-19.0.0-2206.gb6ae2491.el9 3.9 MB/s | 298 kB 00:00 2024-03-19T02:37:36.521 INFO:teuthology.orchestra.run.smithi033.stdout:(24/174): python3-rgw-19.0.0-2206.gb6ae2491.el9 1.3 MB/s | 99 kB 00:00 2024-03-19T02:37:36.588 INFO:teuthology.orchestra.run.smithi033.stdout:(25/174): rbd-fuse-19.0.0-2206.gb6ae2491.el9.x8 1.3 MB/s | 85 kB 00:00 2024-03-19T02:37:36.888 INFO:teuthology.orchestra.run.smithi033.stdout:(26/174): rbd-mirror-19.0.0-2206.gb6ae2491.el9. 10 MB/s | 3.1 MB 00:00 2024-03-19T02:37:36.955 INFO:teuthology.orchestra.run.smithi033.stdout:(27/174): rbd-nbd-19.0.0-2206.gb6ae2491.el9.x86 2.5 MB/s | 170 kB 00:00 2024-03-19T02:37:37.014 INFO:teuthology.orchestra.run.smithi033.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2206.g 582 kB/s | 34 kB 00:00 2024-03-19T02:37:37.106 INFO:teuthology.orchestra.run.smithi033.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2206.gb6ae249 1.5 MB/s | 139 kB 00:00 2024-03-19T02:37:37.281 INFO:teuthology.orchestra.run.smithi033.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2206.gb6ae2 10 MB/s | 1.8 MB 00:00 2024-03-19T02:37:37.898 INFO:teuthology.orchestra.run.smithi033.stdout:(31/174): ceph-mgr-diskprediction-local-19.0.0- 12 MB/s | 7.4 MB 00:00 2024-03-19T02:37:38.315 INFO:teuthology.orchestra.run.smithi033.stdout:(32/174): ceph-test-19.0.0-2206.gb6ae2491.el9.x 15 MB/s | 48 MB 00:03 2024-03-19T02:37:38.340 INFO:teuthology.orchestra.run.smithi033.stdout:(33/174): ceph-mgr-modules-core-19.0.0-2206.gb6 562 kB/s | 248 kB 00:00 2024-03-19T02:37:38.366 INFO:teuthology.orchestra.run.smithi033.stdout:(34/174): ceph-mgr-rook-19.0.0-2206.gb6ae2491.e 987 kB/s | 49 kB 00:00 2024-03-19T02:37:38.383 INFO:teuthology.orchestra.run.smithi033.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2206.gb 394 kB/s | 16 kB 00:00 2024-03-19T02:37:38.416 INFO:teuthology.orchestra.run.smithi033.stdout:(36/174): ceph-volume-19.0.0-2206.gb6ae2491.el9 5.1 MB/s | 264 kB 00:00 2024-03-19T02:37:38.467 INFO:teuthology.orchestra.run.smithi033.stdout:(37/174): cephadm-19.0.0-2206.gb6ae2491.el9.noa 8.7 MB/s | 749 kB 00:00 2024-03-19T02:37:38.492 INFO:teuthology.orchestra.run.smithi033.stdout:(38/174): cephfs-top-19.0.0-2206.gb6ae2491.el9. 243 kB/s | 18 kB 00:00 2024-03-19T02:37:38.725 INFO:teuthology.orchestra.run.smithi033.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 3.9 MB/s | 920 kB 00:00 2024-03-19T02:37:38.751 INFO:teuthology.orchestra.run.smithi033.stdout:(40/174): automake-1.16.2-8.el9.noarch.rpm 2.3 MB/s | 662 kB 00:00 2024-03-19T02:37:38.776 INFO:teuthology.orchestra.run.smithi033.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 106 kB 00:00 2024-03-19T02:37:38.801 INFO:teuthology.orchestra.run.smithi033.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-19T02:37:38.827 INFO:teuthology.orchestra.run.smithi033.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 637 kB/s | 32 kB 00:00 2024-03-19T02:37:38.910 INFO:teuthology.orchestra.run.smithi033.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 207 kB/s | 17 kB 00:00 2024-03-19T02:37:38.961 INFO:teuthology.orchestra.run.smithi033.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 19 MB/s | 3.0 MB 00:00 2024-03-19T02:37:38.986 INFO:teuthology.orchestra.run.smithi033.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 136 kB/s | 10 kB 00:00 2024-03-19T02:37:39.011 INFO:teuthology.orchestra.run.smithi033.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-19T02:37:39.036 INFO:teuthology.orchestra.run.smithi033.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 888 kB/s | 44 kB 00:00 2024-03-19T02:37:39.070 INFO:teuthology.orchestra.run.smithi033.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-19T02:37:39.096 INFO:teuthology.orchestra.run.smithi033.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.8 MB/s | 578 kB 00:00 2024-03-19T02:37:39.121 INFO:teuthology.orchestra.run.smithi033.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 947 kB/s | 46 kB 00:00 2024-03-19T02:37:39.146 INFO:teuthology.orchestra.run.smithi033.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-19T02:37:39.172 INFO:teuthology.orchestra.run.smithi033.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-19T02:37:39.197 INFO:teuthology.orchestra.run.smithi033.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-19T02:37:39.222 INFO:teuthology.orchestra.run.smithi033.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 690 kB/s | 34 kB 00:00 2024-03-19T02:37:39.301 INFO:teuthology.orchestra.run.smithi096.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-19T02:37:39.373 INFO:teuthology.orchestra.run.smithi033.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 27 MB/s | 4.7 MB 00:00 2024-03-19T02:37:39.415 INFO:teuthology.orchestra.run.smithi033.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 8.0 MB/s | 1.5 MB 00:00 2024-03-19T02:37:39.456 INFO:teuthology.orchestra.run.smithi033.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 9.6 MB/s | 811 kB 00:00 2024-03-19T02:37:39.482 INFO:teuthology.orchestra.run.smithi033.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.0 MB/s | 134 kB 00:00 2024-03-19T02:37:39.640 INFO:teuthology.orchestra.run.smithi033.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 31 MB/s | 5.7 MB 00:00 2024-03-19T02:37:39.674 INFO:teuthology.orchestra.run.smithi033.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.4 MB/s | 1.0 MB 00:00 2024-03-19T02:37:39.699 INFO:teuthology.orchestra.run.smithi033.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-19T02:37:39.875 INFO:teuthology.orchestra.run.smithi033.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 29 MB/s | 5.8 MB 00:00 2024-03-19T02:37:39.900 INFO:teuthology.orchestra.run.smithi033.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 122 kB/s | 24 kB 00:00 2024-03-19T02:37:39.925 INFO:teuthology.orchestra.run.smithi033.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 361 kB/s | 18 kB 00:00 2024-03-19T02:37:39.950 INFO:teuthology.orchestra.run.smithi033.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-19T02:37:39.967 INFO:teuthology.orchestra.run.smithi033.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 758 kB/s | 32 kB 00:00 2024-03-19T02:37:40.151 INFO:teuthology.orchestra.run.smithi033.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 24 MB/s | 4.8 MB 00:00 2024-03-19T02:37:40.176 INFO:teuthology.orchestra.run.smithi033.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.0 MB/s | 217 kB 00:00 2024-03-19T02:37:40.202 INFO:teuthology.orchestra.run.smithi033.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-19T02:37:40.227 INFO:teuthology.orchestra.run.smithi033.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-19T02:37:40.252 INFO:teuthology.orchestra.run.smithi033.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 822 kB/s | 41 kB 00:00 2024-03-19T02:37:40.278 INFO:teuthology.orchestra.run.smithi033.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-19T02:37:40.303 INFO:teuthology.orchestra.run.smithi033.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-19T02:37:40.328 INFO:teuthology.orchestra.run.smithi033.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-19T02:37:40.354 INFO:teuthology.orchestra.run.smithi033.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 950 kB/s | 47 kB 00:00 2024-03-19T02:37:40.387 INFO:teuthology.orchestra.run.smithi033.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 733 kB/s | 43 kB 00:00 2024-03-19T02:37:40.613 INFO:teuthology.orchestra.run.smithi033.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 174 kB/s | 39 kB 00:00 2024-03-19T02:37:40.855 INFO:teuthology.orchestra.run.smithi033.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-03-19T02:37:40.880 INFO:teuthology.orchestra.run.smithi033.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 155 kB/s | 41 kB 00:00 2024-03-19T02:37:40.905 INFO:teuthology.orchestra.run.smithi033.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-19T02:37:40.930 INFO:teuthology.orchestra.run.smithi033.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-19T02:37:40.956 INFO:teuthology.orchestra.run.smithi033.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-19T02:37:40.981 INFO:teuthology.orchestra.run.smithi033.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 881 kB/s | 44 kB 00:00 2024-03-19T02:37:41.006 INFO:teuthology.orchestra.run.smithi033.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-19T02:37:41.031 INFO:teuthology.orchestra.run.smithi033.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-19T02:37:41.090 INFO:teuthology.orchestra.run.smithi033.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-19T02:37:41.116 INFO:teuthology.orchestra.run.smithi033.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-19T02:37:41.141 INFO:teuthology.orchestra.run.smithi033.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-19T02:37:41.166 INFO:teuthology.orchestra.run.smithi033.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-19T02:37:41.675 INFO:teuthology.orchestra.run.smithi033.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 351 kB/s | 187 kB 00:00 2024-03-19T02:37:41.706 INFO:teuthology.orchestra.run.smithi096.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-19T02:37:41.706 INFO:teuthology.orchestra.run.smithi096.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-19T02:37:41.717 INFO:teuthology.orchestra.run.smithi096.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-19T02:37:41.718 INFO:teuthology.orchestra.run.smithi096.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-19T02:37:41.719 INFO:teuthology.orchestra.run.smithi096.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-19T02:37:41.720 INFO:teuthology.orchestra.run.smithi096.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-19T02:37:41.721 INFO:teuthology.orchestra.run.smithi096.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-19T02:37:41.724 INFO:teuthology.orchestra.run.smithi096.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-19T02:37:41.725 INFO:teuthology.orchestra.run.smithi096.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-19T02:37:41.803 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-03-19T02:37:41.828 INFO:teuthology.orchestra.run.smithi096.stdout:============================================================================================ 2024-03-19T02:37:41.828 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-03-19T02:37:41.828 INFO:teuthology.orchestra.run.smithi096.stdout:============================================================================================ 2024-03-19T02:37:41.828 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-03-19T02:37:41.829 INFO:teuthology.orchestra.run.smithi096.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-19T02:37:41.829 INFO:teuthology.orchestra.run.smithi096.stdout: ceph x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 6.6 k 2024-03-19T02:37:41.829 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-base x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.4 M 2024-03-19T02:37:41.829 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 891 k 2024-03-19T02:37:41.829 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 145 k 2024-03-19T02:37:41.829 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 1.5 M 2024-03-19T02:37:41.829 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 139 k 2024-03-19T02:37:41.829 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 1.8 M 2024-03-19T02:37:41.829 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 7.4 M 2024-03-19T02:37:41.829 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-rook noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 49 k 2024-03-19T02:37:41.829 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-radosgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 9.8 M 2024-03-19T02:37:41.829 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-test x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 48 M 2024-03-19T02:37:41.830 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-volume noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 264 k 2024-03-19T02:37:41.830 INFO:teuthology.orchestra.run.smithi096.stdout: cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 749 k 2024-03-19T02:37:41.830 INFO:teuthology.orchestra.run.smithi096.stdout: cephfs-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 221 k 2024-03-19T02:37:41.830 INFO:teuthology.orchestra.run.smithi096.stdout: cephfs-shell x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 27 k 2024-03-19T02:37:41.830 INFO:teuthology.orchestra.run.smithi096.stdout: cephfs-top noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 18 k 2024-03-19T02:37:41.830 INFO:teuthology.orchestra.run.smithi096.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-19T02:37:41.830 INFO:teuthology.orchestra.run.smithi096.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-19T02:37:41.830 INFO:teuthology.orchestra.run.smithi096.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-19T02:37:41.830 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 32 k 2024-03-19T02:37:41.830 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 746 k 2024-03-19T02:37:41.830 INFO:teuthology.orchestra.run.smithi096.stdout: librados-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 127 k 2024-03-19T02:37:41.830 INFO:teuthology.orchestra.run.smithi096.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-19T02:37:41.830 INFO:teuthology.orchestra.run.smithi096.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-19T02:37:41.831 INFO:teuthology.orchestra.run.smithi096.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-19T02:37:41.831 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-19T02:37:41.831 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-19T02:37:41.831 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-19T02:37:41.831 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cephfs x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 162 k 2024-03-19T02:37:41.831 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rados x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 321 k 2024-03-19T02:37:41.831 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 298 k 2024-03-19T02:37:41.831 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 99 k 2024-03-19T02:37:41.831 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 85 k 2024-03-19T02:37:41.831 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-19T02:37:41.831 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-nbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 170 k 2024-03-19T02:37:41.831 INFO:teuthology.orchestra.run.smithi096.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-19T02:37:41.831 INFO:teuthology.orchestra.run.smithi096.stdout:Upgrading: 2024-03-19T02:37:41.832 INFO:teuthology.orchestra.run.smithi096.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-19T02:37:41.832 INFO:teuthology.orchestra.run.smithi096.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-19T02:37:41.832 INFO:teuthology.orchestra.run.smithi096.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-19T02:37:41.832 INFO:teuthology.orchestra.run.smithi096.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-19T02:37:41.832 INFO:teuthology.orchestra.run.smithi096.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-19T02:37:41.832 INFO:teuthology.orchestra.run.smithi096.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-19T02:37:41.832 INFO:teuthology.orchestra.run.smithi096.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-19T02:37:41.832 INFO:teuthology.orchestra.run.smithi096.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-19T02:37:41.832 INFO:teuthology.orchestra.run.smithi096.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-19T02:37:41.832 INFO:teuthology.orchestra.run.smithi096.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-19T02:37:41.832 INFO:teuthology.orchestra.run.smithi096.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-19T02:37:41.832 INFO:teuthology.orchestra.run.smithi096.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-19T02:37:41.833 INFO:teuthology.orchestra.run.smithi096.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-19T02:37:41.833 INFO:teuthology.orchestra.run.smithi096.stdout: librados2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.4 M 2024-03-19T02:37:41.833 INFO:teuthology.orchestra.run.smithi096.stdout: librbd1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-19T02:37:41.833 INFO:teuthology.orchestra.run.smithi096.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-19T02:37:41.833 INFO:teuthology.orchestra.run.smithi096.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-19T02:37:41.833 INFO:teuthology.orchestra.run.smithi096.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-19T02:37:41.833 INFO:teuthology.orchestra.run.smithi096.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-19T02:37:41.833 INFO:teuthology.orchestra.run.smithi096.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-19T02:37:41.833 INFO:teuthology.orchestra.run.smithi096.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-19T02:37:41.833 INFO:teuthology.orchestra.run.smithi096.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-19T02:37:41.833 INFO:teuthology.orchestra.run.smithi096.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-19T02:37:41.833 INFO:teuthology.orchestra.run.smithi096.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-19T02:37:41.833 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-03-19T02:37:41.834 INFO:teuthology.orchestra.run.smithi096.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-19T02:37:41.834 INFO:teuthology.orchestra.run.smithi096.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-19T02:37:41.834 INFO:teuthology.orchestra.run.smithi096.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-19T02:37:41.834 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 21 M 2024-03-19T02:37:41.834 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 34 k 2024-03-19T02:37:41.834 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mds x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 2.4 M 2024-03-19T02:37:41.834 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 248 k 2024-03-19T02:37:41.834 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mon x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 4.7 M 2024-03-19T02:37:41.834 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-osd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 17 M 2024-03-19T02:37:41.834 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 16 k 2024-03-19T02:37:41.834 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-selinux x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 25 k 2024-03-19T02:37:41.834 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-19T02:37:41.834 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-19T02:37:41.835 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-19T02:37:41.835 INFO:teuthology.orchestra.run.smithi096.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-19T02:37:41.835 INFO:teuthology.orchestra.run.smithi096.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-19T02:37:41.835 INFO:teuthology.orchestra.run.smithi096.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-19T02:37:41.835 INFO:teuthology.orchestra.run.smithi096.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-19T02:37:41.835 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-19T02:37:41.835 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-19T02:37:41.835 INFO:teuthology.orchestra.run.smithi096.stdout: libcephsqlite x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 163 k 2024-03-19T02:37:41.835 INFO:teuthology.orchestra.run.smithi096.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-19T02:37:41.835 INFO:teuthology.orchestra.run.smithi096.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-19T02:37:41.835 INFO:teuthology.orchestra.run.smithi096.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-19T02:37:41.835 INFO:teuthology.orchestra.run.smithi096.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-19T02:37:41.835 INFO:teuthology.orchestra.run.smithi096.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-19T02:37:41.836 INFO:teuthology.orchestra.run.smithi096.stdout: libradosstriper1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 501 k 2024-03-19T02:37:41.836 INFO:teuthology.orchestra.run.smithi096.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-19T02:37:41.836 INFO:teuthology.orchestra.run.smithi096.stdout: librgw2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.1 M 2024-03-19T02:37:41.836 INFO:teuthology.orchestra.run.smithi096.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-19T02:37:41.836 INFO:teuthology.orchestra.run.smithi096.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-19T02:37:41.836 INFO:teuthology.orchestra.run.smithi096.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-19T02:37:41.836 INFO:teuthology.orchestra.run.smithi096.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-19T02:37:41.836 INFO:teuthology.orchestra.run.smithi096.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-19T02:37:41.836 INFO:teuthology.orchestra.run.smithi096.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-19T02:37:41.836 INFO:teuthology.orchestra.run.smithi096.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-19T02:37:41.836 INFO:teuthology.orchestra.run.smithi096.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-19T02:37:41.836 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-19T02:37:41.836 INFO:teuthology.orchestra.run.smithi096.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-19T02:37:41.836 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-19T02:37:41.837 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-19T02:37:41.837 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-19T02:37:41.837 INFO:teuthology.orchestra.run.smithi096.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-19T02:37:41.837 INFO:teuthology.orchestra.run.smithi096.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-19T02:37:41.837 INFO:teuthology.orchestra.run.smithi096.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-19T02:37:41.837 INFO:teuthology.orchestra.run.smithi096.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-19T02:37:41.837 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-19T02:37:41.837 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-argparse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 45 k 2024-03-19T02:37:41.837 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 131 k 2024-03-19T02:37:41.837 INFO:teuthology.orchestra.run.smithi096.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-19T02:37:41.837 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-19T02:37:41.837 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-19T02:37:41.837 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-19T02:37:41.838 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-19T02:37:41.838 INFO:teuthology.orchestra.run.smithi096.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-19T02:37:41.838 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-19T02:37:41.838 INFO:teuthology.orchestra.run.smithi096.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-19T02:37:41.838 INFO:teuthology.orchestra.run.smithi096.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-19T02:37:41.838 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-19T02:37:41.838 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-19T02:37:41.838 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-19T02:37:41.838 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-19T02:37:41.838 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-19T02:37:41.838 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-19T02:37:41.838 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-19T02:37:41.838 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-19T02:37:41.839 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-19T02:37:41.839 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-19T02:37:41.839 INFO:teuthology.orchestra.run.smithi096.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-19T02:37:41.839 INFO:teuthology.orchestra.run.smithi096.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-19T02:37:41.839 INFO:teuthology.orchestra.run.smithi096.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-19T02:37:41.839 INFO:teuthology.orchestra.run.smithi096.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-19T02:37:41.839 INFO:teuthology.orchestra.run.smithi096.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-19T02:37:41.839 INFO:teuthology.orchestra.run.smithi096.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-19T02:37:41.839 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-19T02:37:41.839 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-19T02:37:41.839 INFO:teuthology.orchestra.run.smithi096.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-19T02:37:41.839 INFO:teuthology.orchestra.run.smithi096.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-19T02:37:41.839 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-19T02:37:41.839 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-19T02:37:41.839 INFO:teuthology.orchestra.run.smithi096.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-19T02:37:41.840 INFO:teuthology.orchestra.run.smithi096.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-19T02:37:41.840 INFO:teuthology.orchestra.run.smithi096.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-19T02:37:41.840 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-19T02:37:41.840 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-19T02:37:41.840 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-19T02:37:41.840 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-19T02:37:41.840 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-19T02:37:41.840 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-19T02:37:41.840 INFO:teuthology.orchestra.run.smithi096.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-19T02:37:41.840 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-19T02:37:41.840 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-19T02:37:41.840 INFO:teuthology.orchestra.run.smithi096.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-19T02:37:41.840 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-19T02:37:41.841 INFO:teuthology.orchestra.run.smithi096.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-19T02:37:41.841 INFO:teuthology.orchestra.run.smithi096.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-19T02:37:41.841 INFO:teuthology.orchestra.run.smithi096.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-19T02:37:41.841 INFO:teuthology.orchestra.run.smithi096.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-19T02:37:41.841 INFO:teuthology.orchestra.run.smithi096.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-19T02:37:41.841 INFO:teuthology.orchestra.run.smithi096.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-19T02:37:41.841 INFO:teuthology.orchestra.run.smithi096.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-19T02:37:41.841 INFO:teuthology.orchestra.run.smithi096.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-19T02:37:41.841 INFO:teuthology.orchestra.run.smithi096.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-19T02:37:41.841 INFO:teuthology.orchestra.run.smithi096.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-19T02:37:41.841 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-19T02:37:41.841 INFO:teuthology.orchestra.run.smithi096.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-19T02:37:41.841 INFO:teuthology.orchestra.run.smithi096.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-19T02:37:41.841 INFO:teuthology.orchestra.run.smithi096.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-19T02:37:41.842 INFO:teuthology.orchestra.run.smithi096.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-19T02:37:41.842 INFO:teuthology.orchestra.run.smithi096.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-19T02:37:41.842 INFO:teuthology.orchestra.run.smithi096.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-19T02:37:41.842 INFO:teuthology.orchestra.run.smithi096.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-19T02:37:41.842 INFO:teuthology.orchestra.run.smithi096.stdout:Installing weak dependencies: 2024-03-19T02:37:41.842 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-19T02:37:41.842 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-19T02:37:41.842 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-03-19T02:37:41.842 INFO:teuthology.orchestra.run.smithi096.stdout:============================================================================================ 2024-03-19T02:37:41.842 INFO:teuthology.orchestra.run.smithi096.stdout:Install 150 Packages 2024-03-19T02:37:41.842 INFO:teuthology.orchestra.run.smithi096.stdout:Upgrade 24 Packages 2024-03-19T02:37:41.842 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-19T02:37:41.843 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 224 M 2024-03-19T02:37:41.843 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-03-19T02:37:41.875 INFO:teuthology.orchestra.run.smithi033.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 977 kB/s | 195 kB 00:00 2024-03-19T02:37:42.109 INFO:teuthology.orchestra.run.smithi033.stdout:(93/174): ceph-common-19.0.0-2206.gb6ae2491.el9 2.3 MB/s | 21 MB 00:09 2024-03-19T02:37:42.142 INFO:teuthology.orchestra.run.smithi033.stdout:(94/174): libgfortran-11.4.1-3.el9.x86_64.rpm 823 kB/s | 803 kB 00:00 2024-03-19T02:37:42.159 INFO:teuthology.orchestra.run.smithi033.stdout:(95/174): python3-requests-2.25.1-8.el9.noarch. 440 kB/s | 125 kB 00:00 2024-03-19T02:37:42.218 INFO:teuthology.orchestra.run.smithi033.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 198 kB/s | 21 kB 00:00 2024-03-19T02:37:42.235 INFO:teuthology.orchestra.run.smithi033.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 275 kB/s | 25 kB 00:00 2024-03-19T02:37:42.352 INFO:teuthology.orchestra.run.smithi033.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.8 MB/s | 245 kB 00:00 2024-03-19T02:37:42.394 INFO:teuthology.orchestra.run.smithi033.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 1.5 MB/s | 248 kB 00:00 2024-03-19T02:37:42.444 INFO:teuthology.orchestra.run.smithi033.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 2.3 MB/s | 216 kB 00:00 2024-03-19T02:37:42.461 INFO:teuthology.orchestra.run.smithi033.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 522 kB/s | 35 kB 00:00 2024-03-19T02:37:42.486 INFO:teuthology.orchestra.run.smithi033.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-19T02:37:42.512 INFO:teuthology.orchestra.run.smithi033.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-03-19T02:37:42.595 INFO:teuthology.orchestra.run.smithi033.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 206 kB/s | 22 kB 00:00 2024-03-19T02:37:42.870 INFO:teuthology.orchestra.run.smithi033.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.4 MB/s | 863 kB 00:00 2024-03-19T02:37:42.896 INFO:teuthology.orchestra.run.smithi033.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.8 MB/s | 548 kB 00:00 2024-03-19T02:37:42.946 INFO:teuthology.orchestra.run.smithi033.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 388 kB/s | 19 kB 00:00 2024-03-19T02:37:43.121 INFO:teuthology.orchestra.run.smithi033.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.2 MB/s | 309 kB 00:00 2024-03-19T02:37:43.188 INFO:teuthology.orchestra.run.smithi033.stdout:(109/174): libarrow-9.0.0-10.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-03-19T02:37:43.214 INFO:teuthology.orchestra.run.smithi033.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 282 kB/s | 26 kB 00:00 2024-03-19T02:37:43.219 INFO:teuthology.orchestra.run.smithi096.stdout:(1/174): ceph-19.0.0-2206.gb6ae2491.el9.x86_64. 13 kB/s | 6.6 kB 00:00 2024-03-19T02:37:43.272 INFO:teuthology.orchestra.run.smithi033.stdout:(111/174): openssl-devel-3.0.7-27.el9.x86_64.rp 3.7 MB/s | 4.1 MB 00:01 2024-03-19T02:37:43.298 INFO:teuthology.orchestra.run.smithi033.stdout:(112/174): liboath-2.6.7-2.el9.x86_64.rpm 446 kB/s | 49 kB 00:00 2024-03-19T02:37:43.323 INFO:teuthology.orchestra.run.smithi033.stdout:(113/174): libunwind-1.6.2-1.el9.x86_64.rpm 618 kB/s | 67 kB 00:00 2024-03-19T02:37:43.365 INFO:teuthology.orchestra.run.smithi033.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2024-03-19T02:37:43.415 INFO:teuthology.orchestra.run.smithi033.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 873 kB/s | 43 kB 00:00 2024-03-19T02:37:43.440 INFO:teuthology.orchestra.run.smithi033.stdout:(116/174): python3-asyncssh-2.13.2-4.el9.noarch 4.6 MB/s | 548 kB 00:00 2024-03-19T02:37:43.466 INFO:teuthology.orchestra.run.smithi033.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 640 kB/s | 32 kB 00:00 2024-03-19T02:37:43.491 INFO:teuthology.orchestra.run.smithi033.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 298 kB/s | 15 kB 00:00 2024-03-19T02:37:43.516 INFO:teuthology.orchestra.run.smithi033.stdout:(119/174): luarocks-3.9.2-1.el9.noarch.rpm 621 kB/s | 151 kB 00:00 2024-03-19T02:37:43.519 INFO:teuthology.orchestra.run.smithi096.stdout:(2/174): ceph-base-19.0.0-2206.gb6ae2491.el9.x8 6.5 MB/s | 5.4 MB 00:00 2024-03-19T02:37:43.542 INFO:teuthology.orchestra.run.smithi033.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-19T02:37:43.567 INFO:teuthology.orchestra.run.smithi033.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 4.6 MB/s | 358 kB 00:00 2024-03-19T02:37:43.569 INFO:teuthology.orchestra.run.smithi096.stdout:(3/174): ceph-immutable-object-cache-19.0.0-220 2.8 MB/s | 145 kB 00:00 2024-03-19T02:37:43.592 INFO:teuthology.orchestra.run.smithi033.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 701 kB/s | 35 kB 00:00 2024-03-19T02:37:43.625 INFO:teuthology.orchestra.run.smithi033.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 3.4 MB/s | 203 kB 00:00 2024-03-19T02:37:43.651 INFO:teuthology.orchestra.run.smithi033.stdout:(124/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.2 MB/s | 303 kB 00:00 2024-03-19T02:37:43.685 INFO:teuthology.orchestra.run.smithi033.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-03-19T02:37:43.710 INFO:teuthology.orchestra.run.smithi033.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 181 kB/s | 11 kB 00:00 2024-03-19T02:37:43.728 INFO:teuthology.orchestra.run.smithi096.stdout:(4/174): ceph-mds-19.0.0-2206.gb6ae2491.el9.x86 15 MB/s | 2.4 MB 00:00 2024-03-19T02:37:43.769 INFO:teuthology.orchestra.run.smithi033.stdout:(127/174): python3-grpcio-1.46.7-10.el9.x86_64. 12 MB/s | 2.0 MB 00:00 2024-03-19T02:37:43.795 INFO:teuthology.orchestra.run.smithi033.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 162 kB/s | 18 kB 00:00 2024-03-19T02:37:43.819 INFO:teuthology.orchestra.run.smithi033.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 213 kB/s | 23 kB 00:00 2024-03-19T02:37:43.837 INFO:teuthology.orchestra.run.smithi096.stdout:(5/174): ceph-mgr-19.0.0-2206.gb6ae2491.el9.x86 14 MB/s | 1.5 MB 00:00 2024-03-19T02:37:43.844 INFO:teuthology.orchestra.run.smithi033.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 258 kB/s | 19 kB 00:00 2024-03-19T02:37:43.870 INFO:teuthology.orchestra.run.smithi033.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-03-19T02:37:43.870 INFO:teuthology.orchestra.run.smithi096.stdout:(6/174): ceph-fuse-19.0.0-2206.gb6ae2491.el9.x8 1.3 MB/s | 891 kB 00:00 2024-03-19T02:37:43.895 INFO:teuthology.orchestra.run.smithi033.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 922 kB/s | 46 kB 00:00 2024-03-19T02:37:43.930 INFO:teuthology.orchestra.run.smithi033.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-03-19T02:37:43.954 INFO:teuthology.orchestra.run.smithi033.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 983 kB/s | 58 kB 00:00 2024-03-19T02:37:43.988 INFO:teuthology.orchestra.run.smithi033.stdout:(135/174): python3-kubernetes-26.1.0-2.el9.noar 6.1 MB/s | 1.0 MB 00:00 2024-03-19T02:37:44.013 INFO:teuthology.orchestra.run.smithi033.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 281 kB/s | 16 kB 00:00 2024-03-19T02:37:44.038 INFO:teuthology.orchestra.run.smithi033.stdout:(137/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-03-19T02:37:44.063 INFO:teuthology.orchestra.run.smithi033.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-03-19T02:37:44.080 INFO:teuthology.orchestra.run.smithi033.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 420 kB/s | 28 kB 00:00 2024-03-19T02:37:44.097 INFO:teuthology.orchestra.run.smithi033.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 522 kB/s | 31 kB 00:00 2024-03-19T02:37:44.112 INFO:teuthology.orchestra.run.smithi096.stdout:(7/174): ceph-mon-19.0.0-2206.gb6ae2491.el9.x86 17 MB/s | 4.7 MB 00:00 2024-03-19T02:37:44.114 INFO:teuthology.orchestra.run.smithi033.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 3.6 MB/s | 188 kB 00:00 2024-03-19T02:37:44.131 INFO:teuthology.orchestra.run.smithi033.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-03-19T02:37:44.149 INFO:teuthology.orchestra.run.smithi033.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 709 kB/s | 36 kB 00:00 2024-03-19T02:37:44.165 INFO:teuthology.orchestra.run.smithi033.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-03-19T02:37:44.190 INFO:teuthology.orchestra.run.smithi033.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-03-19T02:37:44.215 INFO:teuthology.orchestra.run.smithi033.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-03-19T02:37:44.241 INFO:teuthology.orchestra.run.smithi033.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.5 MB/s | 427 kB 00:00 2024-03-19T02:37:44.266 INFO:teuthology.orchestra.run.smithi033.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2024-03-19T02:37:44.291 INFO:teuthology.orchestra.run.smithi033.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-19T02:37:44.350 INFO:teuthology.orchestra.run.smithi033.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-03-19T02:37:44.417 INFO:teuthology.orchestra.run.smithi033.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 4.7 MB/s | 321 kB 00:00 2024-03-19T02:37:44.704 INFO:teuthology.orchestra.run.smithi096.stdout:(8/174): ceph-radosgw-19.0.0-2206.gb6ae2491.el9 17 MB/s | 9.8 MB 00:00 2024-03-19T02:37:44.746 INFO:teuthology.orchestra.run.smithi096.stdout:(9/174): ceph-selinux-19.0.0-2206.gb6ae2491.el9 608 kB/s | 25 kB 00:00 2024-03-19T02:37:44.842 INFO:teuthology.orchestra.run.smithi033.stdout:(152/174): librados2-19.0.0-2206.gb6ae2491.el9. 5.9 MB/s | 3.4 MB 00:00 2024-03-19T02:37:45.284 INFO:teuthology.orchestra.run.smithi033.stdout:(153/174): NetworkManager-1.47.2-1.el9.x86_64.r 2.7 MB/s | 2.3 MB 00:00 2024-03-19T02:37:45.401 INFO:teuthology.orchestra.run.smithi033.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 361 kB/s | 42 kB 00:00 2024-03-19T02:37:45.527 INFO:teuthology.orchestra.run.smithi033.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 1.9 MB/s | 248 kB 00:00 2024-03-19T02:37:45.577 INFO:teuthology.orchestra.run.smithi033.stdout:(156/174): librbd1-19.0.0-2206.gb6ae2491.el9.x8 2.4 MB/s | 3.1 MB 00:01 2024-03-19T02:37:45.635 INFO:teuthology.orchestra.run.smithi033.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 658 kB/s | 71 kB 00:00 2024-03-19T02:37:45.752 INFO:teuthology.orchestra.run.smithi033.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.6 MB/s | 192 kB 00:00 2024-03-19T02:37:45.803 INFO:teuthology.orchestra.run.smithi033.stdout:(159/174): NetworkManager-libnm-1.47.2-1.el9.x8 1.9 MB/s | 1.8 MB 00:00 2024-03-19T02:37:45.864 INFO:teuthology.orchestra.run.smithi033.stdout:(160/174): elfutils-debuginfod-client-0.190-2.e 127 kB/s | 36 kB 00:00 2024-03-19T02:37:45.890 INFO:teuthology.orchestra.run.smithi033.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 1.8 MB/s | 256 kB 00:00 2024-03-19T02:37:45.915 INFO:teuthology.orchestra.run.smithi033.stdout:(162/174): libacl-2.3.1-4.el9.x86_64.rpm 205 kB/s | 23 kB 00:00 2024-03-19T02:37:45.999 INFO:teuthology.orchestra.run.smithi033.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 155 kB 00:00 2024-03-19T02:37:46.033 INFO:teuthology.orchestra.run.smithi033.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 136 kB 00:00 2024-03-19T02:37:46.116 INFO:teuthology.orchestra.run.smithi033.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 544 kB/s | 64 kB 00:00 2024-03-19T02:37:46.142 INFO:teuthology.orchestra.run.smithi033.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 264 kB/s | 29 kB 00:00 2024-03-19T02:37:46.167 INFO:teuthology.orchestra.run.smithi033.stdout:(167/174): libblkid-2.37.4-18.el9.x86_64.rpm 359 kB/s | 108 kB 00:00 2024-03-19T02:37:46.242 INFO:teuthology.orchestra.run.smithi033.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 9.7 MB/s | 1.2 MB 00:00 2024-03-19T02:37:46.384 INFO:teuthology.orchestra.run.smithi033.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 3.2 MB/s | 465 kB 00:00 2024-03-19T02:37:46.426 INFO:teuthology.orchestra.run.smithi033.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 7.6 MB/s | 2.2 MB 00:00 2024-03-19T02:37:46.485 INFO:teuthology.orchestra.run.smithi033.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 311 kB/s | 31 kB 00:00 2024-03-19T02:37:46.552 INFO:teuthology.orchestra.run.smithi033.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 8.6 MB/s | 1.1 MB 00:00 2024-03-19T02:37:46.593 INFO:teuthology.orchestra.run.smithi033.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 685 kB/s | 74 kB 00:00 2024-03-19T02:37:46.810 INFO:teuthology.orchestra.run.smithi033.stdout:(174/174): util-linux-2.37.4-18.el9.x86_64.rpm 3.6 MB/s | 2.3 MB 00:00 2024-03-19T02:37:46.814 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:37:46.815 INFO:teuthology.orchestra.run.smithi033.stdout:Total 16 MB/s | 224 MB 00:14 2024-03-19T02:37:46.888 INFO:teuthology.orchestra.run.smithi096.stdout:(10/174): ceph-osd-19.0.0-2206.gb6ae2491.el9.x8 5.6 MB/s | 17 MB 00:03 2024-03-19T02:37:47.105 INFO:teuthology.orchestra.run.smithi096.stdout:(11/174): ceph-common-19.0.0-2206.gb6ae2491.el9 4.7 MB/s | 21 MB 00:04 2024-03-19T02:37:47.130 INFO:teuthology.orchestra.run.smithi096.stdout:(12/174): cephfs-mirror-19.0.0-2206.gb6ae2491.e 916 kB/s | 221 kB 00:00 2024-03-19T02:37:47.163 INFO:teuthology.orchestra.run.smithi096.stdout:(13/174): cephfs-shell-19.0.0-2206.gb6ae2491.el 465 kB/s | 27 kB 00:00 2024-03-19T02:37:47.189 INFO:teuthology.orchestra.run.smithi096.stdout:(14/174): libcephfs-devel-19.0.0-2206.gb6ae2491 541 kB/s | 32 kB 00:00 2024-03-19T02:37:47.278 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-19T02:37:47.280 INFO:teuthology.orchestra.run.smithi096.stdout:(15/174): libcephsqlite-19.0.0-2206.gb6ae2491.e 1.7 MB/s | 163 kB 00:00 2024-03-19T02:37:47.356 INFO:teuthology.orchestra.run.smithi096.stdout:(16/174): libcephfs2-19.0.0-2206.gb6ae2491.el9. 3.8 MB/s | 746 kB 00:00 2024-03-19T02:37:47.381 INFO:teuthology.orchestra.run.smithi096.stdout:(17/174): librados-devel-19.0.0-2206.gb6ae2491. 1.2 MB/s | 127 kB 00:00 2024-03-19T02:37:47.523 INFO:teuthology.orchestra.run.smithi096.stdout:(18/174): libradosstriper1-19.0.0-2206.gb6ae249 2.9 MB/s | 501 kB 00:00 2024-03-19T02:37:47.574 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-19T02:37:47.574 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-19T02:37:47.615 INFO:teuthology.orchestra.run.smithi096.stdout:(19/174): python3-ceph-argparse-19.0.0-2206.gb6 492 kB/s | 45 kB 00:00 2024-03-19T02:37:47.724 INFO:teuthology.orchestra.run.smithi096.stdout:(20/174): python3-ceph-common-19.0.0-2206.gb6ae 1.2 MB/s | 131 kB 00:00 2024-03-19T02:37:47.874 INFO:teuthology.orchestra.run.smithi096.stdout:(21/174): librgw2-19.0.0-2206.gb6ae2491.el9.x86 10 MB/s | 5.1 MB 00:00 2024-03-19T02:37:47.899 INFO:teuthology.orchestra.run.smithi096.stdout:(22/174): python3-cephfs-19.0.0-2206.gb6ae2491. 923 kB/s | 162 kB 00:00 2024-03-19T02:37:47.958 INFO:teuthology.orchestra.run.smithi096.stdout:(23/174): python3-rados-19.0.0-2206.gb6ae2491.e 3.8 MB/s | 321 kB 00:00 2024-03-19T02:37:48.000 INFO:teuthology.orchestra.run.smithi096.stdout:(24/174): python3-rbd-19.0.0-2206.gb6ae2491.el9 2.9 MB/s | 298 kB 00:00 2024-03-19T02:37:48.409 INFO:teuthology.orchestra.run.smithi096.stdout:(25/174): ceph-test-19.0.0-2206.gb6ae2491.el9.x 13 MB/s | 48 MB 00:03 2024-03-19T02:37:48.434 INFO:teuthology.orchestra.run.smithi096.stdout:(26/174): rbd-fuse-19.0.0-2206.gb6ae2491.el9.x8 197 kB/s | 85 kB 00:00 2024-03-19T02:37:48.459 INFO:teuthology.orchestra.run.smithi096.stdout:(27/174): python3-rgw-19.0.0-2206.gb6ae2491.el9 198 kB/s | 99 kB 00:00 2024-03-19T02:37:48.509 INFO:teuthology.orchestra.run.smithi096.stdout:(28/174): rbd-nbd-19.0.0-2206.gb6ae2491.el9.x86 2.2 MB/s | 170 kB 00:00 2024-03-19T02:37:48.543 INFO:teuthology.orchestra.run.smithi096.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2206.g 406 kB/s | 34 kB 00:00 2024-03-19T02:37:48.602 INFO:teuthology.orchestra.run.smithi096.stdout:(30/174): rbd-mirror-19.0.0-2206.gb6ae2491.el9. 16 MB/s | 3.1 MB 00:00 2024-03-19T02:37:48.626 INFO:teuthology.orchestra.run.smithi096.stdout:(31/174): ceph-mgr-cephadm-19.0.0-2206.gb6ae249 1.2 MB/s | 139 kB 00:00 2024-03-19T02:37:48.710 INFO:teuthology.orchestra.run.smithi096.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2206.gb6 2.9 MB/s | 248 kB 00:00 2024-03-19T02:37:48.802 INFO:teuthology.orchestra.run.smithi096.stdout:(33/174): ceph-mgr-rook-19.0.0-2206.gb6ae2491.e 539 kB/s | 49 kB 00:00 2024-03-19T02:37:48.894 INFO:teuthology.orchestra.run.smithi096.stdout:(34/174): ceph-mgr-dashboard-19.0.0-2206.gb6ae2 5.1 MB/s | 1.8 MB 00:00 2024-03-19T02:37:48.919 INFO:teuthology.orchestra.run.smithi096.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2206.gb 141 kB/s | 16 kB 00:00 2024-03-19T02:37:49.020 INFO:teuthology.orchestra.run.smithi096.stdout:(36/174): ceph-mgr-diskprediction-local-19.0.0- 18 MB/s | 7.4 MB 00:00 2024-03-19T02:37:49.045 INFO:teuthology.orchestra.run.smithi096.stdout:(37/174): ceph-volume-19.0.0-2206.gb6ae2491.el9 1.7 MB/s | 264 kB 00:00 2024-03-19T02:37:49.070 INFO:teuthology.orchestra.run.smithi096.stdout:(38/174): cephfs-top-19.0.0-2206.gb6ae2491.el9. 365 kB/s | 18 kB 00:00 2024-03-19T02:37:49.104 INFO:teuthology.orchestra.run.smithi096.stdout:(39/174): cephadm-19.0.0-2206.gb6ae2491.el9.noa 4.0 MB/s | 749 kB 00:00 2024-03-19T02:37:49.296 INFO:teuthology.orchestra.run.smithi096.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 552 kB/s | 106 kB 00:00 2024-03-19T02:37:49.329 INFO:teuthology.orchestra.run.smithi096.stdout:(41/174): automake-1.16.2-8.el9.noarch.rpm 2.3 MB/s | 662 kB 00:00 2024-03-19T02:37:49.363 INFO:teuthology.orchestra.run.smithi096.stdout:(42/174): bison-3.7.4-5.el9.x86_64.rpm 3.1 MB/s | 920 kB 00:00 2024-03-19T02:37:49.446 INFO:teuthology.orchestra.run.smithi096.stdout:(43/174): flex-2.6.4-9.el9.x86_64.rpm 2.0 MB/s | 307 kB 00:00 2024-03-19T02:37:49.472 INFO:teuthology.orchestra.run.smithi096.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 223 kB/s | 32 kB 00:00 2024-03-19T02:37:49.564 INFO:teuthology.orchestra.run.smithi096.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 147 kB/s | 17 kB 00:00 2024-03-19T02:37:49.589 INFO:teuthology.orchestra.run.smithi096.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 87 kB/s | 10 kB 00:00 2024-03-19T02:37:49.639 INFO:teuthology.orchestra.run.smithi096.stdout:(47/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 11 MB/s | 3.0 MB 00:00 2024-03-19T02:37:49.643 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-19T02:37:49.644 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-19T02:37:49.665 INFO:teuthology.orchestra.run.smithi096.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 159 kB 00:00 2024-03-19T02:37:49.689 INFO:teuthology.orchestra.run.smithi096.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 444 kB/s | 44 kB 00:00 2024-03-19T02:37:49.715 INFO:teuthology.orchestra.run.smithi096.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-19T02:37:49.741 INFO:teuthology.orchestra.run.smithi096.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 938 kB/s | 46 kB 00:00 2024-03-19T02:37:49.766 INFO:teuthology.orchestra.run.smithi096.stdout:(52/174): libtool-2.4.6-45.el9.x86_64.rpm 7.6 MB/s | 578 kB 00:00 2024-03-19T02:37:49.791 INFO:teuthology.orchestra.run.smithi096.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-19T02:37:49.816 INFO:teuthology.orchestra.run.smithi096.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 2.5 MB/s | 187 kB 00:00 2024-03-19T02:37:49.833 INFO:teuthology.orchestra.run.smithi096.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 829 kB/s | 34 kB 00:00 2024-03-19T02:37:49.867 INFO:teuthology.orchestra.run.smithi096.stdout:(56/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-03-19T02:37:50.059 INFO:teuthology.orchestra.run.smithi096.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 20 MB/s | 4.7 MB 00:00 2024-03-19T02:37:50.101 INFO:teuthology.orchestra.run.smithi096.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 5.7 MB/s | 1.5 MB 00:00 2024-03-19T02:37:50.134 INFO:teuthology.orchestra.run.smithi096.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 3.0 MB/s | 811 kB 00:00 2024-03-19T02:37:50.160 INFO:teuthology.orchestra.run.smithi096.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.3 MB/s | 134 kB 00:00 2024-03-19T02:37:50.251 INFO:teuthology.orchestra.run.smithi096.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 8.7 MB/s | 1.0 MB 00:00 2024-03-19T02:37:50.360 INFO:teuthology.orchestra.run.smithi096.stdout:(62/174): postgresql-server-13.11-1.el9.x86_64. 25 MB/s | 5.7 MB 00:00 2024-03-19T02:37:50.385 INFO:teuthology.orchestra.run.smithi096.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 346 kB/s | 78 kB 00:00 2024-03-19T02:37:50.469 INFO:teuthology.orchestra.run.smithi096.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 27 MB/s | 5.8 MB 00:00 2024-03-19T02:37:50.494 INFO:teuthology.orchestra.run.smithi096.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 186 kB/s | 24 kB 00:00 2024-03-19T02:37:50.519 INFO:teuthology.orchestra.run.smithi096.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 138 kB/s | 18 kB 00:00 2024-03-19T02:37:50.553 INFO:teuthology.orchestra.run.smithi096.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 144 kB 00:00 2024-03-19T02:37:50.578 INFO:teuthology.orchestra.run.smithi096.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 542 kB/s | 32 kB 00:00 2024-03-19T02:37:50.678 INFO:teuthology.orchestra.run.smithi096.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 30 MB/s | 4.8 MB 00:00 2024-03-19T02:37:50.704 INFO:teuthology.orchestra.run.smithi096.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2024-03-19T02:37:50.729 INFO:teuthology.orchestra.run.smithi096.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 174 kB 00:00 2024-03-19T02:37:50.754 INFO:teuthology.orchestra.run.smithi096.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 924 kB/s | 69 kB 00:00 2024-03-19T02:37:50.779 INFO:teuthology.orchestra.run.smithi096.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 825 kB/s | 41 kB 00:00 2024-03-19T02:37:50.805 INFO:teuthology.orchestra.run.smithi096.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-19T02:37:50.830 INFO:teuthology.orchestra.run.smithi096.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-19T02:37:50.855 INFO:teuthology.orchestra.run.smithi096.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 955 kB/s | 47 kB 00:00 2024-03-19T02:37:50.890 INFO:teuthology.orchestra.run.smithi096.stdout:(77/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 211 kB 00:00 2024-03-19T02:37:50.930 INFO:teuthology.orchestra.run.smithi096.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 427 kB/s | 43 kB 00:00 2024-03-19T02:37:51.156 INFO:teuthology.orchestra.run.smithi096.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 174 kB/s | 39 kB 00:00 2024-03-19T02:37:51.181 INFO:teuthology.orchestra.run.smithi096.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 166 kB/s | 41 kB 00:00 2024-03-19T02:37:51.431 INFO:teuthology.orchestra.run.smithi096.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 27 MB/s | 16 MB 00:00 2024-03-19T02:37:51.465 INFO:teuthology.orchestra.run.smithi096.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 200 kB/s | 56 kB 00:00 2024-03-19T02:37:51.490 INFO:teuthology.orchestra.run.smithi096.stdout:(83/174): socat-1.7.4.1-5.el9.x86_64.rpm 977 kB/s | 300 kB 00:00 2024-03-19T02:37:51.515 INFO:teuthology.orchestra.run.smithi096.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 748 kB/s | 63 kB 00:00 2024-03-19T02:37:51.566 INFO:teuthology.orchestra.run.smithi096.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 442 kB/s | 44 kB 00:00 2024-03-19T02:37:51.591 INFO:teuthology.orchestra.run.smithi096.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-19T02:37:51.624 INFO:teuthology.orchestra.run.smithi096.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.2 MB/s | 241 kB 00:00 2024-03-19T02:37:51.666 INFO:teuthology.orchestra.run.smithi096.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.1 MB 00:00 2024-03-19T02:37:51.692 INFO:teuthology.orchestra.run.smithi096.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 103 kB 00:00 2024-03-19T02:37:51.717 INFO:teuthology.orchestra.run.smithi096.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 124 kB 00:00 2024-03-19T02:37:51.742 INFO:teuthology.orchestra.run.smithi096.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2024-03-19T02:37:56.768 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-19T02:37:56.856 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-19T02:37:56.899 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-19T02:37:56.929 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-19T02:37:56.939 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-19T02:37:56.987 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-19T02:37:57.042 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-19T02:37:57.083 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-19T02:37:57.117 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-19T02:37:57.171 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-19T02:37:57.519 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-19T02:37:57.568 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T02:37:57.692 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T02:37:57.727 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T02:37:57.820 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T02:37:57.847 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-19T02:37:57.900 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-19T02:37:58.089 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-19T02:37:58.268 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-19T02:37:58.294 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T02:37:58.400 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T02:37:58.429 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-19T02:37:58.494 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-19T02:37:58.535 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-19T02:37:58.566 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-19T02:37:58.581 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-19T02:37:58.733 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-19T02:37:58.764 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-19T02:37:58.796 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-19T02:37:58.979 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-19T02:37:59.097 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-19T02:37:59.149 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-19T02:37:59.275 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T02:37:59.326 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T02:37:59.532 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T02:38:00.847 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-19T02:38:00.847 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-19T02:38:00.847 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-19T02:38:00.847 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-19T02:38:00.847 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-19T02:38:00.847 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-19T02:38:00.847 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-19T02:38:00.847 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-19T02:38:00.847 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-19T02:38:00.847 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-19T02:38:00.847 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-19T02:38:00.848 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-19T02:38:00.848 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-19T02:38:00.848 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-19T02:38:00.848 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 39/198 2024-03-19T02:38:00.848 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 40/198 2024-03-19T02:38:00.848 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 41/198 2024-03-19T02:38:00.848 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-19T02:38:00.848 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-19T02:38:00.848 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-19T02:38:00.869 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-19T02:38:00.923 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-19T02:38:01.477 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-19T02:38:01.554 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-19T02:38:01.635 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-19T02:38:01.679 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-19T02:38:01.687 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-19T02:38:01.731 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-19T02:38:01.777 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-19T02:38:01.814 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 53/198 2024-03-19T02:38:01.845 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-19T02:38:02.120 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-19T02:38:02.589 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-19T02:38:02.672 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-19T02:38:02.718 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-19T02:38:02.859 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-19T02:38:02.919 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-19T02:38:02.975 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-19T02:38:03.048 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-19T02:38:03.086 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-19T02:38:03.143 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-19T02:38:03.196 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-19T02:38:03.230 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-19T02:38:03.274 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-19T02:38:03.315 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-19T02:38:03.466 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-19T02:38:03.512 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-19T02:38:03.543 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-19T02:38:03.593 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-19T02:38:03.691 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-19T02:38:03.726 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-19T02:38:03.766 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-19T02:38:03.791 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-19T02:38:03.815 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-19T02:38:03.896 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-19T02:38:04.023 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-19T02:38:04.137 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-19T02:38:04.183 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-19T02:38:04.249 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-19T02:38:04.305 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-19T02:38:04.500 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-19T02:38:04.553 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-19T02:38:04.706 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-19T02:38:04.751 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-19T02:38:04.783 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-19T02:38:04.814 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-19T02:38:04.833 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-19T02:38:04.868 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-19T02:38:04.908 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-19T02:38:04.939 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-19T02:38:04.977 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-19T02:38:05.056 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-19T02:38:05.087 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-19T02:38:05.121 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-19T02:38:05.162 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-19T02:38:05.205 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-19T02:38:05.300 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-19T02:38:05.388 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-19T02:38:05.445 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-19T02:38:05.476 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-19T02:38:05.524 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-19T02:38:05.635 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-19T02:38:05.694 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-19T02:38:06.464 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-19T02:38:06.507 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-19T02:38:06.541 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-19T02:38:06.590 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-19T02:38:06.681 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-19T02:38:06.717 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-19T02:38:06.901 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-19T02:38:07.156 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-19T02:38:07.241 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-19T02:38:07.946 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-19T02:38:08.057 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-19T02:38:09.362 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-19T02:38:09.448 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-19T02:38:09.479 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-19T02:38:09.510 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-19T02:38:09.537 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-19T02:38:09.869 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-19T02:38:09.909 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-19T02:38:10.001 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-19T02:38:10.031 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 125/198 2024-03-19T02:38:10.068 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-19T02:38:10.557 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-19T02:38:10.587 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-19T02:38:10.642 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-19T02:38:10.672 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 129/198 2024-03-19T02:38:13.012 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T02:38:13.043 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T02:38:13.621 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T02:38:13.655 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-19T02:38:13.731 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-19T02:38:13.731 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-19T02:38:13.731 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-19T02:38:13.732 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:38:13.772 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-19T02:38:25.841 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-19T02:38:25.841 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /sys 2024-03-19T02:38:25.841 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /proc 2024-03-19T02:38:25.841 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /mnt 2024-03-19T02:38:25.841 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /var/tmp 2024-03-19T02:38:25.841 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /home 2024-03-19T02:38:25.842 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /root 2024-03-19T02:38:25.842 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /tmp 2024-03-19T02:38:25.842 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:38:26.008 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-19T02:38:26.023 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-19T02:38:26.023 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:38:26.023 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-19T02:38:26.023 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:38:27.569 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-19T02:38:27.594 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-19T02:38:27.594 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:38:27.594 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-19T02:38:27.594 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-19T02:38:27.595 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-19T02:38:27.595 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:38:27.829 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-19T02:38:27.851 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-19T02:38:27.851 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:38:27.852 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-19T02:38:27.852 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-19T02:38:27.852 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-19T02:38:27.852 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:38:28.323 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-19T02:38:28.345 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-19T02:38:28.345 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:38:28.345 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-19T02:38:28.345 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-19T02:38:28.345 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-19T02:38:28.345 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:38:28.432 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-19T02:38:28.850 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-19T02:38:28.881 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-19T02:38:28.914 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 139/198 2024-03-19T02:38:28.975 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 140/198 2024-03-19T02:38:29.004 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-19T02:38:29.252 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-19T02:38:29.280 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-19T02:38:30.302 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-19T02:38:30.322 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-19T02:38:30.427 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-19T02:38:30.601 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 144/198 2024-03-19T02:38:30.632 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-19T02:38:30.655 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-19T02:38:30.655 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:38:30.655 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-19T02:38:30.655 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-19T02:38:30.655 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-19T02:38:30.655 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:38:30.699 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-19T02:38:30.726 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-19T02:38:30.777 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 147/198 2024-03-19T02:38:31.708 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-19T02:38:31.744 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-19T02:38:31.767 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-19T02:38:31.767 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:38:31.767 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-19T02:38:31.768 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-19T02:38:31.768 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-19T02:38:31.768 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:38:31.811 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-19T02:38:31.832 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-19T02:38:31.832 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:38:31.832 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-19T02:38:31.832 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:38:31.886 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T02:38:31.908 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T02:38:31.908 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:38:31.908 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-19T02:38:31.908 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-19T02:38:31.908 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-19T02:38:31.908 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:38:32.220 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-19T02:38:32.242 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-19T02:38:32.243 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:38:32.243 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-19T02:38:32.243 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T02:38:32.243 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T02:38:32.243 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:38:37.008 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-19T02:38:37.050 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 154/198 2024-03-19T02:38:37.080 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 155/198 2024-03-19T02:38:37.107 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-19T02:38:37.297 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 157/198 2024-03-19T02:38:37.419 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-19T02:38:37.464 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 159/198 2024-03-19T02:38:37.519 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-19T02:38:37.571 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-19T02:38:37.615 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-19T02:38:37.647 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 163/198 2024-03-19T02:38:37.679 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-19T02:38:37.709 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-19T02:38:37.770 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-19T02:38:37.837 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-19T02:38:37.874 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-19T02:38:38.421 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T02:38:38.457 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T02:38:38.476 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T02:38:39.359 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 170/198 2024-03-19T02:38:39.452 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-19T02:38:39.494 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-19T02:38:39.586 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-19T02:38:39.630 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-19T02:38:39.655 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-19T02:38:39.681 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-19T02:38:39.707 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-19T02:38:39.707 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-19T02:38:39.795 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-19T02:38:39.833 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-19T02:38:39.833 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-19T02:38:39.951 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-19T02:38:39.978 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-19T02:38:40.010 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-19T02:38:40.036 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-19T02:38:40.061 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-19T02:38:40.120 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-19T02:38:40.167 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-19T02:38:40.199 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-19T02:38:40.224 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-19T02:38:40.249 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-19T02:38:40.275 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-19T02:38:40.320 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-19T02:38:40.326 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T02:38:40.326 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T02:38:40.511 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T02:38:40.511 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-19T02:38:40.586 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-19T02:38:40.610 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-19T02:38:40.642 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-19T02:38:40.670 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-19T02:38:40.696 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-19T02:38:40.763 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-19T02:38:45.502 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-19T02:38:45.502 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 1/198 2024-03-19T02:38:45.502 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2/198 2024-03-19T02:38:45.502 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 3/198 2024-03-19T02:38:45.502 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 4/198 2024-03-19T02:38:45.502 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 5/198 2024-03-19T02:38:45.502 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 6/198 2024-03-19T02:38:45.502 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 7/198 2024-03-19T02:38:45.502 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 8/198 2024-03-19T02:38:45.503 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 9/198 2024-03-19T02:38:45.503 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T02:38:45.503 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T02:38:45.503 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-19T02:38:45.503 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 13/198 2024-03-19T02:38:45.503 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 14/198 2024-03-19T02:38:45.503 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 15/198 2024-03-19T02:38:45.504 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T02:38:45.504 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 17/198 2024-03-19T02:38:45.504 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 18/198 2024-03-19T02:38:45.504 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 19/198 2024-03-19T02:38:45.504 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 20/198 2024-03-19T02:38:45.504 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 21/198 2024-03-19T02:38:45.504 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 22/198 2024-03-19T02:38:45.504 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 23/198 2024-03-19T02:38:45.504 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 24/198 2024-03-19T02:38:45.504 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 25/198 2024-03-19T02:38:45.505 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 26/198 2024-03-19T02:38:45.505 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 27/198 2024-03-19T02:38:45.505 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 28/198 2024-03-19T02:38:45.505 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 29/198 2024-03-19T02:38:45.505 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 30/198 2024-03-19T02:38:45.505 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 31/198 2024-03-19T02:38:45.505 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 32/198 2024-03-19T02:38:45.506 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 33/198 2024-03-19T02:38:45.506 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 34/198 2024-03-19T02:38:45.506 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 35/198 2024-03-19T02:38:45.506 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 36/198 2024-03-19T02:38:45.506 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 37/198 2024-03-19T02:38:45.506 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 38/198 2024-03-19T02:38:45.506 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 39/198 2024-03-19T02:38:45.506 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-19T02:38:45.507 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-19T02:38:45.507 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-19T02:38:45.507 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-19T02:38:45.507 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-19T02:38:45.507 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-19T02:38:45.507 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-19T02:38:45.507 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-19T02:38:45.507 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-19T02:38:45.508 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-19T02:38:45.508 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-19T02:38:45.508 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-19T02:38:45.508 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-19T02:38:45.508 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-19T02:38:45.508 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-19T02:38:45.508 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-19T02:38:45.509 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-19T02:38:45.509 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-19T02:38:45.509 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-19T02:38:45.509 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-19T02:38:45.509 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-19T02:38:45.509 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-19T02:38:45.509 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-19T02:38:45.510 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-19T02:38:45.510 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-19T02:38:45.510 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-19T02:38:45.510 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-19T02:38:45.510 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-19T02:38:45.510 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-19T02:38:45.510 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-19T02:38:45.510 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-19T02:38:45.510 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-19T02:38:45.511 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-19T02:38:45.511 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-19T02:38:45.511 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-19T02:38:45.511 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-19T02:38:45.511 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-19T02:38:45.511 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-19T02:38:45.511 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-19T02:38:45.511 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-19T02:38:45.511 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-19T02:38:45.511 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-19T02:38:45.511 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-19T02:38:45.512 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-19T02:38:45.512 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-19T02:38:45.512 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-19T02:38:45.512 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-19T02:38:45.512 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-19T02:38:45.512 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-19T02:38:45.512 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-19T02:38:45.512 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-19T02:38:45.512 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-19T02:38:45.512 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-19T02:38:45.512 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-19T02:38:45.512 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-19T02:38:45.512 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-19T02:38:45.512 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-19T02:38:45.513 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-19T02:38:45.513 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-19T02:38:45.513 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-19T02:38:45.513 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-19T02:38:45.513 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-19T02:38:45.513 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-19T02:38:45.513 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-19T02:38:45.513 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-19T02:38:45.513 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-19T02:38:45.513 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-19T02:38:45.513 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-19T02:38:45.513 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-19T02:38:45.513 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-19T02:38:45.514 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-19T02:38:45.514 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-19T02:38:45.514 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-19T02:38:45.514 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-19T02:38:45.514 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-19T02:38:45.514 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-19T02:38:45.514 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-19T02:38:45.514 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-19T02:38:45.514 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-19T02:38:45.514 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-19T02:38:45.514 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-19T02:38:45.514 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-19T02:38:45.514 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-19T02:38:45.515 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-19T02:38:45.515 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-19T02:38:45.515 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-19T02:38:45.515 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-19T02:38:45.515 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-19T02:38:45.515 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-19T02:38:45.515 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-19T02:38:45.515 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-19T02:38:45.515 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-19T02:38:45.515 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-19T02:38:45.515 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-19T02:38:45.515 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-19T02:38:45.515 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-19T02:38:45.515 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-19T02:38:45.515 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-19T02:38:45.516 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-19T02:38:45.516 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-19T02:38:45.516 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-19T02:38:45.516 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-19T02:38:45.516 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-19T02:38:45.516 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-19T02:38:45.516 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-19T02:38:45.516 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-19T02:38:45.516 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-19T02:38:45.516 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-19T02:38:45.516 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-19T02:38:45.516 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-19T02:38:45.516 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-19T02:38:45.516 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T02:38:45.517 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-19T02:38:45.517 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-19T02:38:45.517 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-19T02:38:45.517 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-19T02:38:45.517 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-19T02:38:45.517 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-19T02:38:45.517 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-19T02:38:45.517 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-19T02:38:45.517 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-19T02:38:45.517 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-19T02:38:45.517 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-19T02:38:45.517 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-19T02:38:45.517 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-19T02:38:45.517 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-19T02:38:45.517 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-19T02:38:45.518 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-19T02:38:45.518 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-19T02:38:45.518 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-19T02:38:45.518 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-19T02:38:45.518 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-19T02:38:45.518 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-19T02:38:45.518 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-19T02:38:45.518 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-19T02:38:45.518 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-19T02:38:45.518 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-19T02:38:45.518 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-19T02:38:45.518 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-19T02:38:45.519 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-19T02:38:45.519 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-19T02:38:45.519 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-19T02:38:45.519 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-19T02:38:45.519 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-19T02:38:45.519 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-19T02:38:45.519 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-19T02:38:45.519 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-19T02:38:45.519 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-19T02:38:45.519 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-19T02:38:45.519 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-19T02:38:45.519 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-19T02:38:45.519 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-19T02:38:45.519 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-19T02:38:45.520 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-19T02:38:45.520 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-19T02:38:45.520 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-19T02:38:45.520 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-19T02:38:45.520 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-19T02:38:48.564 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-19T02:38:48.564 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:38:48.564 INFO:teuthology.orchestra.run.smithi033.stdout:Upgraded: 2024-03-19T02:38:48.564 INFO:teuthology.orchestra.run.smithi033.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-19T02:38:48.565 INFO:teuthology.orchestra.run.smithi033.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-19T02:38:48.565 INFO:teuthology.orchestra.run.smithi033.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-19T02:38:48.565 INFO:teuthology.orchestra.run.smithi033.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-19T02:38:48.565 INFO:teuthology.orchestra.run.smithi033.stdout: acl-2.3.1-4.el9.x86_64 2024-03-19T02:38:48.565 INFO:teuthology.orchestra.run.smithi033.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-19T02:38:48.565 INFO:teuthology.orchestra.run.smithi033.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-19T02:38:48.565 INFO:teuthology.orchestra.run.smithi033.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-19T02:38:48.565 INFO:teuthology.orchestra.run.smithi033.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-19T02:38:48.565 INFO:teuthology.orchestra.run.smithi033.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-19T02:38:48.565 INFO:teuthology.orchestra.run.smithi033.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-19T02:38:48.565 INFO:teuthology.orchestra.run.smithi033.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-19T02:38:48.565 INFO:teuthology.orchestra.run.smithi033.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-19T02:38:48.565 INFO:teuthology.orchestra.run.smithi033.stdout: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:38:48.566 INFO:teuthology.orchestra.run.smithi033.stdout: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:38:48.566 INFO:teuthology.orchestra.run.smithi033.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-19T02:38:48.566 INFO:teuthology.orchestra.run.smithi033.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-19T02:38:48.566 INFO:teuthology.orchestra.run.smithi033.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-19T02:38:48.566 INFO:teuthology.orchestra.run.smithi033.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-19T02:38:48.566 INFO:teuthology.orchestra.run.smithi033.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-19T02:38:48.566 INFO:teuthology.orchestra.run.smithi033.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-19T02:38:48.566 INFO:teuthology.orchestra.run.smithi033.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-19T02:38:48.566 INFO:teuthology.orchestra.run.smithi033.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-19T02:38:48.566 INFO:teuthology.orchestra.run.smithi033.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-19T02:38:48.566 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-19T02:38:48.566 INFO:teuthology.orchestra.run.smithi033.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-19T02:38:48.566 INFO:teuthology.orchestra.run.smithi033.stdout: automake-1.16.2-8.el9.noarch 2024-03-19T02:38:48.566 INFO:teuthology.orchestra.run.smithi033.stdout: bison-3.7.4-5.el9.x86_64 2024-03-19T02:38:48.567 INFO:teuthology.orchestra.run.smithi033.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-19T02:38:48.567 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:38:48.567 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:38:48.567 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:38:48.567 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:38:48.567 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:38:48.567 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:38:48.567 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:38:48.568 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:38:48.568 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:38:48.568 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:38:48.568 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:38:48.568 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:38:48.568 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:38:48.568 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:38:48.568 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:38:48.569 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:38:48.569 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:38:48.569 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:38:48.569 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:38:48.569 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:38:48.569 INFO:teuthology.orchestra.run.smithi033.stdout: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:38:48.569 INFO:teuthology.orchestra.run.smithi033.stdout: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:38:48.569 INFO:teuthology.orchestra.run.smithi033.stdout: cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:38:48.570 INFO:teuthology.orchestra.run.smithi033.stdout: cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:38:48.570 INFO:teuthology.orchestra.run.smithi033.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-19T02:38:48.570 INFO:teuthology.orchestra.run.smithi033.stdout: flex-2.6.4-9.el9.x86_64 2024-03-19T02:38:48.570 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-19T02:38:48.570 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-19T02:38:48.570 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-19T02:38:48.570 INFO:teuthology.orchestra.run.smithi033.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-19T02:38:48.570 INFO:teuthology.orchestra.run.smithi033.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-19T02:38:48.570 INFO:teuthology.orchestra.run.smithi033.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-19T02:38:48.571 INFO:teuthology.orchestra.run.smithi033.stdout: jq-1.6-16.el9.x86_64 2024-03-19T02:38:48.571 INFO:teuthology.orchestra.run.smithi033.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-19T02:38:48.571 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-19T02:38:48.571 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-19T02:38:48.571 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:38:48.571 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:38:48.571 INFO:teuthology.orchestra.run.smithi033.stdout: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:38:48.571 INFO:teuthology.orchestra.run.smithi033.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-19T02:38:48.571 INFO:teuthology.orchestra.run.smithi033.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-19T02:38:48.572 INFO:teuthology.orchestra.run.smithi033.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-19T02:38:48.572 INFO:teuthology.orchestra.run.smithi033.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-19T02:38:48.572 INFO:teuthology.orchestra.run.smithi033.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-19T02:38:48.572 INFO:teuthology.orchestra.run.smithi033.stdout: librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:38:48.572 INFO:teuthology.orchestra.run.smithi033.stdout: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:38:48.572 INFO:teuthology.orchestra.run.smithi033.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-19T02:38:48.572 INFO:teuthology.orchestra.run.smithi033.stdout: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:38:48.572 INFO:teuthology.orchestra.run.smithi033.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-19T02:38:48.572 INFO:teuthology.orchestra.run.smithi033.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-19T02:38:48.572 INFO:teuthology.orchestra.run.smithi033.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-19T02:38:48.572 INFO:teuthology.orchestra.run.smithi033.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-19T02:38:48.573 INFO:teuthology.orchestra.run.smithi033.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-19T02:38:48.573 INFO:teuthology.orchestra.run.smithi033.stdout: lua-5.4.4-4.el9.x86_64 2024-03-19T02:38:48.573 INFO:teuthology.orchestra.run.smithi033.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-19T02:38:48.573 INFO:teuthology.orchestra.run.smithi033.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-19T02:38:48.573 INFO:teuthology.orchestra.run.smithi033.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-19T02:38:48.573 INFO:teuthology.orchestra.run.smithi033.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-19T02:38:48.573 INFO:teuthology.orchestra.run.smithi033.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-19T02:38:48.573 INFO:teuthology.orchestra.run.smithi033.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-19T02:38:48.573 INFO:teuthology.orchestra.run.smithi033.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-19T02:38:48.573 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-13.11-1.el9.x86_64 2024-03-19T02:38:48.574 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-19T02:38:48.574 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-19T02:38:48.574 INFO:teuthology.orchestra.run.smithi033.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-19T02:38:48.574 INFO:teuthology.orchestra.run.smithi033.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-19T02:38:48.574 INFO:teuthology.orchestra.run.smithi033.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-19T02:38:48.574 INFO:teuthology.orchestra.run.smithi033.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-19T02:38:48.574 INFO:teuthology.orchestra.run.smithi033.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-19T02:38:48.574 INFO:teuthology.orchestra.run.smithi033.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-19T02:38:48.574 INFO:teuthology.orchestra.run.smithi033.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-19T02:38:48.574 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-19T02:38:48.574 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:38:48.574 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:38:48.575 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:38:48.575 INFO:teuthology.orchestra.run.smithi033.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-19T02:38:48.575 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-19T02:38:48.575 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-19T02:38:48.575 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-19T02:38:48.575 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-19T02:38:48.575 INFO:teuthology.orchestra.run.smithi033.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-19T02:38:48.575 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-19T02:38:48.575 INFO:teuthology.orchestra.run.smithi033.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-19T02:38:48.576 INFO:teuthology.orchestra.run.smithi033.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-19T02:38:48.576 INFO:teuthology.orchestra.run.smithi033.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-19T02:38:48.576 INFO:teuthology.orchestra.run.smithi033.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-19T02:38:48.576 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-19T02:38:48.576 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-19T02:38:48.576 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-19T02:38:48.576 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-19T02:38:48.576 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-19T02:38:48.576 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-19T02:38:48.576 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-19T02:38:48.576 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-19T02:38:48.577 INFO:teuthology.orchestra.run.smithi033.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-19T02:38:48.577 INFO:teuthology.orchestra.run.smithi033.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-19T02:38:48.577 INFO:teuthology.orchestra.run.smithi033.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-19T02:38:48.577 INFO:teuthology.orchestra.run.smithi033.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-19T02:38:48.577 INFO:teuthology.orchestra.run.smithi033.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-19T02:38:48.577 INFO:teuthology.orchestra.run.smithi033.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-19T02:38:48.577 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-19T02:38:48.577 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-19T02:38:48.577 INFO:teuthology.orchestra.run.smithi033.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-19T02:38:48.577 INFO:teuthology.orchestra.run.smithi033.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-19T02:38:48.577 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-19T02:38:48.577 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ply-3.11-14.el9.noarch 2024-03-19T02:38:48.577 INFO:teuthology.orchestra.run.smithi033.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-19T02:38:48.577 INFO:teuthology.orchestra.run.smithi033.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-19T02:38:48.577 INFO:teuthology.orchestra.run.smithi033.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-19T02:38:48.578 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-19T02:38:48.578 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-19T02:38:48.578 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-19T02:38:48.578 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-19T02:38:48.578 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-19T02:38:48.578 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-19T02:38:48.578 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:38:48.578 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:38:48.578 INFO:teuthology.orchestra.run.smithi033.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-19T02:38:48.578 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-19T02:38:48.578 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-19T02:38:48.578 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:38:48.578 INFO:teuthology.orchestra.run.smithi033.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-19T02:38:48.579 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-19T02:38:48.579 INFO:teuthology.orchestra.run.smithi033.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-19T02:38:48.579 INFO:teuthology.orchestra.run.smithi033.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-19T02:38:48.579 INFO:teuthology.orchestra.run.smithi033.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-19T02:38:48.579 INFO:teuthology.orchestra.run.smithi033.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-19T02:38:48.579 INFO:teuthology.orchestra.run.smithi033.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-19T02:38:48.579 INFO:teuthology.orchestra.run.smithi033.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-19T02:38:48.579 INFO:teuthology.orchestra.run.smithi033.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-19T02:38:48.579 INFO:teuthology.orchestra.run.smithi033.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-19T02:38:48.579 INFO:teuthology.orchestra.run.smithi033.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-19T02:38:48.579 INFO:teuthology.orchestra.run.smithi033.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-19T02:38:48.579 INFO:teuthology.orchestra.run.smithi033.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-19T02:38:48.579 INFO:teuthology.orchestra.run.smithi033.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-19T02:38:48.579 INFO:teuthology.orchestra.run.smithi033.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-19T02:38:48.579 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:38:48.580 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:38:48.580 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:38:48.580 INFO:teuthology.orchestra.run.smithi033.stdout: re2-1:20211101-3.el9.x86_64 2024-03-19T02:38:48.580 INFO:teuthology.orchestra.run.smithi033.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-19T02:38:48.580 INFO:teuthology.orchestra.run.smithi033.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-19T02:38:48.580 INFO:teuthology.orchestra.run.smithi033.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-19T02:38:48.580 INFO:teuthology.orchestra.run.smithi033.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-19T02:38:48.580 INFO:teuthology.orchestra.run.smithi033.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-19T02:38:48.581 INFO:teuthology.orchestra.run.smithi033.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-19T02:38:48.581 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:38:48.581 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-19T02:38:49.064 DEBUG:teuthology.parallel:result is None 2024-03-19T02:40:02.787 INFO:teuthology.orchestra.run.smithi096.stdout:[MIRROR] jq-1.6-16.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/jq-1.6-16.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T02:40:02.787 INFO:teuthology.orchestra.run.smithi096.stdout:[MIRROR] libgfortran-11.4.1-3.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.4.1-3.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T02:40:02.787 INFO:teuthology.orchestra.run.smithi096.stdout:[MIRROR] libquadmath-11.4.1-3.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.4.1-3.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T02:40:03.449 INFO:teuthology.orchestra.run.smithi096.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 1.4 kB/s | 187 kB 02:11 2024-03-19T02:40:03.474 INFO:teuthology.orchestra.run.smithi096.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.5 kB/s | 195 kB 02:11 2024-03-19T02:40:03.732 INFO:teuthology.orchestra.run.smithi096.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 441 kB/s | 125 kB 00:00 2024-03-19T02:40:03.883 INFO:teuthology.orchestra.run.smithi096.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 53 kB/s | 21 kB 00:00 2024-03-19T02:40:04.216 INFO:teuthology.orchestra.run.smithi096.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 53 kB/s | 25 kB 00:00 2024-03-19T02:40:04.400 INFO:teuthology.orchestra.run.smithi096.stdout:(97/174): libgfortran-11.4.1-3.el9.x86_64.rpm 6.1 kB/s | 803 kB 02:12 2024-03-19T02:40:04.842 INFO:teuthology.orchestra.run.smithi096.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 392 kB/s | 245 kB 00:00 2024-03-19T02:40:04.976 INFO:teuthology.orchestra.run.smithi096.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 1.6 MB/s | 216 kB 00:00 2024-03-19T02:40:05.034 INFO:teuthology.orchestra.run.smithi096.stdout:(100/174): python3-jinja2-2.11.3-5.el9.noarch.r 391 kB/s | 248 kB 00:00 2024-03-19T02:40:05.117 INFO:teuthology.orchestra.run.smithi096.stdout:(101/174): openssl-devel-3.0.7-27.el9.x86_64.rp 3.3 MB/s | 4.1 MB 00:01 2024-03-19T02:40:05.143 INFO:teuthology.orchestra.run.smithi096.stdout:(102/174): qatlib-service-23.11.0-1.el9.x86_64. 208 kB/s | 35 kB 00:00 2024-03-19T02:40:05.168 INFO:teuthology.orchestra.run.smithi096.stdout:(103/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 332 kB/s | 44 kB 00:00 2024-03-19T02:40:05.227 INFO:teuthology.orchestra.run.smithi096.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 860 kB/s | 94 kB 00:00 2024-03-19T02:40:05.252 INFO:teuthology.orchestra.run.smithi096.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 205 kB/s | 22 kB 00:00 2024-03-19T02:40:05.477 INFO:teuthology.orchestra.run.smithi096.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.7 MB/s | 863 kB 00:00 2024-03-19T02:40:05.911 INFO:teuthology.orchestra.run.smithi096.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 45 kB/s | 19 kB 00:00 2024-03-19T02:40:06.053 INFO:teuthology.orchestra.run.smithi096.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 386 kB/s | 309 kB 00:00 2024-03-19T02:40:06.114 INFO:teuthology.orchestra.run.smithi096.stdout:(109/174): abseil-cpp-20211102.0-3.el9.x86_64.r 620 kB/s | 548 kB 00:00 2024-03-19T02:40:06.162 INFO:teuthology.orchestra.run.smithi096.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 239 kB/s | 26 kB 00:00 2024-03-19T02:40:06.237 INFO:teuthology.orchestra.run.smithi096.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 388 kB/s | 49 kB 00:00 2024-03-19T02:40:06.296 INFO:teuthology.orchestra.run.smithi096.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 505 kB/s | 67 kB 00:00 2024-03-19T02:40:06.387 INFO:teuthology.orchestra.run.smithi096.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-03-19T02:40:06.597 INFO:teuthology.orchestra.run.smithi096.stdout:(114/174): python3-asyncssh-2.13.2-4.el9.noarch 2.6 MB/s | 548 kB 00:00 2024-03-19T02:40:06.638 INFO:teuthology.orchestra.run.smithi096.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.4 MB/s | 838 kB 00:00 2024-03-19T02:40:06.713 INFO:teuthology.orchestra.run.smithi096.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 371 kB/s | 43 kB 00:00 2024-03-19T02:40:06.763 INFO:teuthology.orchestra.run.smithi096.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 257 kB/s | 32 kB 00:00 2024-03-19T02:40:06.831 INFO:teuthology.orchestra.run.smithi096.stdout:(118/174): libarrow-9.0.0-10.el9.x86_64.rpm 4.8 MB/s | 4.4 MB 00:00 2024-03-19T02:40:06.856 INFO:teuthology.orchestra.run.smithi096.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 105 kB/s | 15 kB 00:00 2024-03-19T02:40:06.889 INFO:teuthology.orchestra.run.smithi096.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.3 MB/s | 172 kB 00:00 2024-03-19T02:40:06.998 INFO:teuthology.orchestra.run.smithi096.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.1 MB/s | 303 kB 00:00 2024-03-19T02:40:07.023 INFO:teuthology.orchestra.run.smithi096.stdout:(122/174): python3-cherrypy-18.6.1-2.el9.noarch 1.8 MB/s | 358 kB 00:00 2024-03-19T02:40:07.048 INFO:teuthology.orchestra.run.smithi096.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 222 kB/s | 35 kB 00:00 2024-03-19T02:40:07.124 INFO:teuthology.orchestra.run.smithi096.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 1.6 MB/s | 203 kB 00:00 2024-03-19T02:40:07.190 INFO:teuthology.orchestra.run.smithi096.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.0 MB/s | 144 kB 00:00 2024-03-19T02:40:07.266 INFO:teuthology.orchestra.run.smithi096.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 75 kB/s | 11 kB 00:00 2024-03-19T02:40:07.324 INFO:teuthology.orchestra.run.smithi096.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 132 kB/s | 18 kB 00:00 2024-03-19T02:40:07.391 INFO:teuthology.orchestra.run.smithi096.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 185 kB/s | 23 kB 00:00 2024-03-19T02:40:07.442 INFO:teuthology.orchestra.run.smithi096.stdout:(129/174): python3-grpcio-1.46.7-10.el9.x86_64. 4.9 MB/s | 2.0 MB 00:00 2024-03-19T02:40:07.467 INFO:teuthology.orchestra.run.smithi096.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 136 kB/s | 19 kB 00:00 2024-03-19T02:40:07.509 INFO:teuthology.orchestra.run.smithi096.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 168 kB/s | 20 kB 00:00 2024-03-19T02:40:07.609 INFO:teuthology.orchestra.run.smithi096.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 326 kB/s | 46 kB 00:00 2024-03-19T02:40:07.643 INFO:teuthology.orchestra.run.smithi096.stdout:(133/174): python3-kubernetes-26.1.0-2.el9.noar 5.1 MB/s | 1.0 MB 00:00 2024-03-19T02:40:07.671 INFO:teuthology.orchestra.run.smithi096.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 487 kB/s | 79 kB 00:00 2024-03-19T02:40:07.738 INFO:teuthology.orchestra.run.smithi096.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 448 kB/s | 58 kB 00:00 2024-03-19T02:40:07.780 INFO:teuthology.orchestra.run.smithi096.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-03-19T02:40:07.798 INFO:teuthology.orchestra.run.smithi096.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 130 kB/s | 16 kB 00:00 2024-03-19T02:40:07.881 INFO:teuthology.orchestra.run.smithi096.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 630 kB/s | 90 kB 00:00 2024-03-19T02:40:07.906 INFO:teuthology.orchestra.run.smithi096.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 224 kB/s | 28 kB 00:00 2024-03-19T02:40:07.932 INFO:teuthology.orchestra.run.smithi096.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 229 kB/s | 31 kB 00:00 2024-03-19T02:40:08.032 INFO:teuthology.orchestra.run.smithi096.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.2 MB/s | 188 kB 00:00 2024-03-19T02:40:08.057 INFO:teuthology.orchestra.run.smithi096.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 393 kB/s | 59 kB 00:00 2024-03-19T02:40:08.082 INFO:teuthology.orchestra.run.smithi096.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 237 kB/s | 36 kB 00:00 2024-03-19T02:40:08.174 INFO:teuthology.orchestra.run.smithi096.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 359 kB/s | 51 kB 00:00 2024-03-19T02:40:08.208 INFO:teuthology.orchestra.run.smithi096.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.5 MB/s | 230 kB 00:00 2024-03-19T02:40:08.341 INFO:teuthology.orchestra.run.smithi096.stdout:(146/174): python3-zc-lockfile-2.0-10.el9.noarc 150 kB/s | 20 kB 00:00 2024-03-19T02:40:08.370 INFO:teuthology.orchestra.run.smithi096.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.2 MB/s | 427 kB 00:00 2024-03-19T02:40:08.459 INFO:teuthology.orchestra.run.smithi096.stdout:(148/174): python3-websocket-client-1.2.3-2.el9 238 kB/s | 90 kB 00:00 2024-03-19T02:40:08.484 INFO:teuthology.orchestra.run.smithi096.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-19T02:40:08.670 INFO:teuthology.orchestra.run.smithi096.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 5.3 MB/s | 1.6 MB 00:00 2024-03-19T02:40:08.835 INFO:teuthology.orchestra.run.smithi096.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 1.9 MB/s | 321 kB 00:00 2024-03-19T02:40:08.927 INFO:teuthology.orchestra.run.smithi096.stdout:(152/174): librbd1-19.0.0-2206.gb6ae2491.el9.x8 7.1 MB/s | 3.1 MB 00:00 2024-03-19T02:40:09.262 INFO:teuthology.orchestra.run.smithi096.stdout:(153/174): librados2-19.0.0-2206.gb6ae2491.el9. 4.2 MB/s | 3.4 MB 00:00 2024-03-19T02:40:09.553 INFO:teuthology.orchestra.run.smithi096.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 144 kB/s | 42 kB 00:00 2024-03-19T02:40:09.919 INFO:teuthology.orchestra.run.smithi096.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 678 kB/s | 248 kB 00:00 2024-03-19T02:40:10.028 INFO:teuthology.orchestra.run.smithi096.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 657 kB/s | 71 kB 00:00 2024-03-19T02:40:10.138 INFO:teuthology.orchestra.run.smithi096.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 336 kB/s | 36 kB 00:00 2024-03-19T02:40:10.445 INFO:teuthology.orchestra.run.smithi096.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 623 kB/s | 192 kB 00:00 2024-03-19T02:40:11.153 INFO:teuthology.orchestra.run.smithi096.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 362 kB/s | 256 kB 00:00 2024-03-19T02:40:11.295 INFO:teuthology.orchestra.run.smithi096.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 163 kB/s | 23 kB 00:00 2024-03-19T02:40:11.537 INFO:teuthology.orchestra.run.smithi096.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 448 kB/s | 108 kB 00:00 2024-03-19T02:40:11.638 INFO:teuthology.orchestra.run.smithi096.stdout:(162/174): NetworkManager-1.47.2-1.el9.x86_64.r 842 kB/s | 2.3 MB 00:02 2024-03-19T02:40:11.738 INFO:teuthology.orchestra.run.smithi096.stdout:(163/174): NetworkManager-libnm-1.47.2-1.el9.x8 668 kB/s | 1.8 MB 00:02 2024-03-19T02:40:12.130 INFO:teuthology.orchestra.run.smithi096.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 262 kB/s | 155 kB 00:00 2024-03-19T02:40:12.463 INFO:teuthology.orchestra.run.smithi096.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 164 kB/s | 136 kB 00:00 2024-03-19T02:40:12.680 INFO:teuthology.orchestra.run.smithi096.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 68 kB/s | 64 kB 00:00 2024-03-19T02:40:12.839 INFO:teuthology.orchestra.run.smithi096.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 40 kB/s | 29 kB 00:00 2024-03-19T02:40:14.056 INFO:teuthology.orchestra.run.smithi096.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 778 kB/s | 1.2 MB 00:01 2024-03-19T02:40:14.614 INFO:teuthology.orchestra.run.smithi096.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 833 kB/s | 465 kB 00:00 2024-03-19T02:40:14.723 INFO:teuthology.orchestra.run.smithi096.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 287 kB/s | 31 kB 00:00 2024-03-19T02:40:15.848 INFO:teuthology.orchestra.run.smithi096.stdout:(171/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 977 kB/s | 1.1 MB 00:01 2024-03-19T02:40:16.248 INFO:teuthology.orchestra.run.smithi096.stdout:(172/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 619 kB/s | 2.2 MB 00:03 2024-03-19T02:40:16.273 INFO:teuthology.orchestra.run.smithi096.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 175 kB/s | 74 kB 00:00 2024-03-19T02:40:18.873 INFO:teuthology.orchestra.run.smithi096.stdout:(174/174): util-linux-2.37.4-18.el9.x86_64.rpm 387 kB/s | 2.3 MB 00:06 2024-03-19T02:40:18.876 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:40:18.876 INFO:teuthology.orchestra.run.smithi096.stdout:Total 1.4 MB/s | 224 MB 02:37 2024-03-19T02:40:19.336 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-03-19T02:40:19.613 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-03-19T02:40:19.613 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-03-19T02:40:21.700 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-03-19T02:40:21.700 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-03-19T02:40:28.592 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-03-19T02:40:28.702 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-19T02:40:28.731 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-19T02:40:28.769 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-19T02:40:28.778 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-19T02:40:28.833 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-19T02:40:28.889 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-19T02:40:28.922 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-19T02:40:28.956 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-19T02:40:29.010 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-19T02:40:29.352 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-19T02:40:29.416 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T02:40:29.519 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T02:40:29.542 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T02:40:29.616 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T02:40:29.655 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-19T02:40:29.711 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-19T02:40:29.898 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-19T02:40:29.945 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-19T02:40:29.973 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T02:40:30.075 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T02:40:30.099 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-19T02:40:30.147 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-19T02:40:30.186 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-19T02:40:30.217 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-19T02:40:30.232 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-19T02:40:30.429 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-19T02:40:30.466 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-19T02:40:30.514 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-19T02:40:30.711 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-19T02:40:30.841 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-19T02:40:30.901 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-19T02:40:31.005 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T02:40:31.052 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T02:40:31.254 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T02:40:31.357 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-19T02:40:31.451 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-19T02:40:31.483 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-19T02:40:31.510 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-19T02:40:31.553 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-19T02:40:31.626 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-19T02:40:31.689 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-19T02:40:31.728 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-19T02:40:31.791 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-19T02:40:31.844 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-19T02:40:31.887 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-19T02:40:31.922 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-19T02:40:31.948 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-19T02:40:32.017 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-19T02:40:32.062 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 39/198 2024-03-19T02:40:32.113 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 40/198 2024-03-19T02:40:32.142 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 41/198 2024-03-19T02:40:32.210 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-19T02:40:32.257 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-19T02:40:32.330 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-19T02:40:32.470 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-19T02:40:32.516 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-19T02:40:33.038 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-19T02:40:33.120 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-19T02:40:33.200 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-19T02:40:33.258 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-19T02:40:33.266 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-19T02:40:33.322 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-19T02:40:33.381 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-19T02:40:33.418 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 53/198 2024-03-19T02:40:33.450 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-19T02:40:33.754 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-19T02:40:33.950 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-19T02:40:34.060 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-19T02:40:34.104 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-19T02:40:34.245 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-19T02:40:34.299 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-19T02:40:34.361 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-19T02:40:34.442 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-19T02:40:34.474 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-19T02:40:34.511 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-19T02:40:34.560 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-19T02:40:34.754 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-19T02:40:34.889 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-19T02:40:35.030 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-19T02:40:35.171 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-19T02:40:35.227 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-19T02:40:35.257 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-19T02:40:35.298 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-19T02:40:35.392 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-19T02:40:35.432 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-19T02:40:35.472 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-19T02:40:35.498 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-19T02:40:35.522 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-19T02:40:35.608 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-19T02:40:35.731 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-19T02:40:35.840 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-19T02:40:35.889 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-19T02:40:35.938 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-19T02:40:35.993 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-19T02:40:36.185 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-19T02:40:36.236 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-19T02:40:36.372 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-19T02:40:36.423 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-19T02:40:36.453 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-19T02:40:36.485 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-19T02:40:36.513 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-19T02:40:36.547 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-19T02:40:36.586 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-19T02:40:36.618 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-19T02:40:36.648 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-19T02:40:36.733 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-19T02:40:36.775 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-19T02:40:36.809 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-19T02:40:36.850 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-19T02:40:36.885 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-19T02:40:36.975 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-19T02:40:37.064 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-19T02:40:37.107 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-19T02:40:37.149 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-19T02:40:37.194 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-19T02:40:37.291 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-19T02:40:37.340 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-19T02:40:38.064 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-19T02:40:38.113 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-19T02:40:38.164 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-19T02:40:38.227 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-19T02:40:38.300 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-19T02:40:38.340 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-19T02:40:38.516 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-19T02:40:38.546 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-19T02:40:38.632 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-19T02:40:39.265 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-19T02:40:39.375 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-19T02:40:40.587 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-19T02:40:40.680 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-19T02:40:40.715 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-19T02:40:40.743 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-19T02:40:40.770 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-19T02:40:41.082 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-19T02:40:41.127 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-19T02:40:41.200 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-19T02:40:41.239 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 125/198 2024-03-19T02:40:41.283 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-19T02:40:41.742 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-19T02:40:41.794 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-19T02:40:41.849 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-19T02:40:41.872 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 129/198 2024-03-19T02:40:44.184 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T02:40:44.221 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T02:40:44.764 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T02:40:44.808 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-19T02:40:44.883 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-19T02:40:44.883 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-19T02:40:44.883 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-19T02:40:44.883 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-19T02:40:44.917 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-19T02:40:56.673 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-19T02:40:56.673 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /sys 2024-03-19T02:40:56.674 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /proc 2024-03-19T02:40:56.674 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /mnt 2024-03-19T02:40:56.674 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /var/tmp 2024-03-19T02:40:56.674 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /home 2024-03-19T02:40:56.674 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /root 2024-03-19T02:40:56.674 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /tmp 2024-03-19T02:40:56.674 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-19T02:40:56.839 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-19T02:40:56.855 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-19T02:40:56.855 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:40:56.855 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-19T02:40:56.855 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-19T02:40:58.359 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-19T02:40:58.384 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-19T02:40:58.384 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:40:58.384 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-19T02:40:58.384 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-19T02:40:58.384 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-19T02:40:58.384 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-19T02:40:58.613 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-19T02:40:58.635 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-19T02:40:58.635 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:40:58.635 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-19T02:40:58.635 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-19T02:40:58.635 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-19T02:40:58.635 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-19T02:40:59.077 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-19T02:40:59.099 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-19T02:40:59.099 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:40:59.099 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-19T02:40:59.099 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-19T02:40:59.099 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-19T02:40:59.099 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-19T02:40:59.196 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-19T02:40:59.590 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-19T02:40:59.621 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-19T02:40:59.653 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 139/198 2024-03-19T02:40:59.714 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 140/198 2024-03-19T02:40:59.736 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-19T02:40:59.970 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-19T02:41:00.004 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-19T02:41:00.975 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-19T02:41:01.012 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-19T02:41:01.112 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-19T02:41:01.295 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 144/198 2024-03-19T02:41:01.331 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-19T02:41:01.353 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-19T02:41:01.353 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:41:01.353 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-19T02:41:01.354 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-19T02:41:01.354 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-19T02:41:01.354 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-19T02:41:01.398 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-19T02:41:01.409 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-19T02:41:01.459 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 147/198 2024-03-19T02:41:02.322 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-19T02:41:02.351 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-19T02:41:02.373 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-19T02:41:02.374 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:41:02.374 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-19T02:41:02.374 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-19T02:41:02.374 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-19T02:41:02.374 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-19T02:41:02.419 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-19T02:41:02.440 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-19T02:41:02.440 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:41:02.440 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-19T02:41:02.440 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-19T02:41:02.494 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T02:41:02.516 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T02:41:02.516 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:41:02.516 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-19T02:41:02.516 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-19T02:41:02.517 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-19T02:41:02.517 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-19T02:41:02.819 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-19T02:41:02.841 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-19T02:41:02.841 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T02:41:02.842 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-19T02:41:02.842 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T02:41:02.842 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T02:41:02.842 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-19T02:41:07.307 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-19T02:41:07.348 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 154/198 2024-03-19T02:41:07.379 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 155/198 2024-03-19T02:41:07.406 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-19T02:41:07.590 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 157/198 2024-03-19T02:41:07.714 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-19T02:41:07.765 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 159/198 2024-03-19T02:41:07.816 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-19T02:41:07.872 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-19T02:41:07.915 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-19T02:41:07.956 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 163/198 2024-03-19T02:41:08.047 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-19T02:41:08.076 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-19T02:41:08.143 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-19T02:41:08.213 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-19T02:41:08.250 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-19T02:41:08.859 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T02:41:08.898 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T02:41:08.918 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T02:41:09.736 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 170/198 2024-03-19T02:41:09.819 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-19T02:41:09.868 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-19T02:41:09.957 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-19T02:41:09.998 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-19T02:41:10.023 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-19T02:41:10.048 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-19T02:41:10.075 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-19T02:41:10.075 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-19T02:41:10.166 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-19T02:41:10.201 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-19T02:41:10.201 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-19T02:41:10.277 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-19T02:41:10.304 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-19T02:41:10.337 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-19T02:41:10.362 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-19T02:41:10.387 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-19T02:41:10.422 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-19T02:41:10.468 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-19T02:41:10.509 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-19T02:41:10.534 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-19T02:41:10.560 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-19T02:41:10.585 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-19T02:41:10.618 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-19T02:41:10.625 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T02:41:10.625 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T02:41:10.805 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T02:41:10.805 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-19T02:41:10.888 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-19T02:41:10.913 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-19T02:41:10.944 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-19T02:41:10.972 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-19T02:41:10.997 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-19T02:41:11.064 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-19T02:41:16.010 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-19T02:41:16.010 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 1/198 2024-03-19T02:41:16.010 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2/198 2024-03-19T02:41:16.010 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 3/198 2024-03-19T02:41:16.010 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 4/198 2024-03-19T02:41:16.011 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 5/198 2024-03-19T02:41:16.011 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 6/198 2024-03-19T02:41:16.011 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 7/198 2024-03-19T02:41:16.011 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 8/198 2024-03-19T02:41:16.011 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 9/198 2024-03-19T02:41:16.011 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T02:41:16.011 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T02:41:16.011 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-19T02:41:16.011 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 13/198 2024-03-19T02:41:16.011 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 14/198 2024-03-19T02:41:16.011 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 15/198 2024-03-19T02:41:16.011 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T02:41:16.011 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 17/198 2024-03-19T02:41:16.011 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 18/198 2024-03-19T02:41:16.012 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 19/198 2024-03-19T02:41:16.012 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 20/198 2024-03-19T02:41:16.012 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 21/198 2024-03-19T02:41:16.012 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 22/198 2024-03-19T02:41:16.012 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 23/198 2024-03-19T02:41:16.012 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 24/198 2024-03-19T02:41:16.012 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 25/198 2024-03-19T02:41:16.012 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 26/198 2024-03-19T02:41:16.012 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 27/198 2024-03-19T02:41:16.012 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 28/198 2024-03-19T02:41:16.012 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 29/198 2024-03-19T02:41:16.012 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 30/198 2024-03-19T02:41:16.012 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 31/198 2024-03-19T02:41:16.012 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 32/198 2024-03-19T02:41:16.013 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 33/198 2024-03-19T02:41:16.013 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 34/198 2024-03-19T02:41:16.013 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 35/198 2024-03-19T02:41:16.013 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 36/198 2024-03-19T02:41:16.013 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 37/198 2024-03-19T02:41:16.013 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 38/198 2024-03-19T02:41:16.013 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 39/198 2024-03-19T02:41:16.013 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-19T02:41:16.013 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-19T02:41:16.013 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-19T02:41:16.013 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-19T02:41:16.013 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-19T02:41:16.013 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-19T02:41:16.013 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-19T02:41:16.014 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-19T02:41:16.014 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-19T02:41:16.014 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-19T02:41:16.014 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-19T02:41:16.014 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-19T02:41:16.014 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-19T02:41:16.014 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-19T02:41:16.014 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-19T02:41:16.014 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-19T02:41:16.014 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-19T02:41:16.014 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-19T02:41:16.014 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-19T02:41:16.014 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-19T02:41:16.014 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-19T02:41:16.015 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-19T02:41:16.015 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-19T02:41:16.015 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-19T02:41:16.015 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-19T02:41:16.015 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-19T02:41:16.015 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-19T02:41:16.015 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-19T02:41:16.015 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-19T02:41:16.015 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-19T02:41:16.015 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-19T02:41:16.015 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-19T02:41:16.015 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-19T02:41:16.016 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-19T02:41:16.016 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-19T02:41:16.016 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-19T02:41:16.016 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-19T02:41:16.016 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-19T02:41:16.016 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-19T02:41:16.016 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-19T02:41:16.016 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-19T02:41:16.016 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-19T02:41:16.016 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-19T02:41:16.016 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-19T02:41:16.016 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-19T02:41:16.016 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-19T02:41:16.017 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-19T02:41:16.017 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-19T02:41:16.017 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-19T02:41:16.017 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-19T02:41:16.017 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-19T02:41:16.017 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-19T02:41:16.017 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-19T02:41:16.017 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-19T02:41:16.017 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-19T02:41:16.017 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-19T02:41:16.017 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-19T02:41:16.017 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-19T02:41:16.017 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-19T02:41:16.017 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-19T02:41:16.017 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-19T02:41:16.018 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-19T02:41:16.018 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-19T02:41:16.018 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-19T02:41:16.018 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-19T02:41:16.018 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-19T02:41:16.018 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-19T02:41:16.018 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-19T02:41:16.018 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-19T02:41:16.018 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-19T02:41:16.018 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-19T02:41:16.018 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-19T02:41:16.018 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-19T02:41:16.019 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-19T02:41:16.019 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-19T02:41:16.019 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-19T02:41:16.019 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-19T02:41:16.019 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-19T02:41:16.019 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-19T02:41:16.019 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-19T02:41:16.019 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-19T02:41:16.019 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-19T02:41:16.019 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-19T02:41:16.019 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-19T02:41:16.019 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-19T02:41:16.019 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-19T02:41:16.019 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-19T02:41:16.019 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-19T02:41:16.020 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-19T02:41:16.020 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-19T02:41:16.020 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-19T02:41:16.020 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-19T02:41:16.020 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-19T02:41:16.020 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-19T02:41:16.020 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-19T02:41:16.020 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-19T02:41:16.020 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-19T02:41:16.020 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-19T02:41:16.020 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-19T02:41:16.020 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-19T02:41:16.020 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-19T02:41:16.021 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-19T02:41:16.021 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-19T02:41:16.021 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-19T02:41:16.021 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-19T02:41:16.021 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-19T02:41:16.021 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-19T02:41:16.021 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-19T02:41:16.021 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-19T02:41:16.021 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-19T02:41:16.021 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-19T02:41:16.021 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T02:41:16.021 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-19T02:41:16.021 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-19T02:41:16.021 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-19T02:41:16.022 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-19T02:41:16.022 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-19T02:41:16.022 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-19T02:41:16.022 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-19T02:41:16.022 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-19T02:41:16.022 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-19T02:41:16.022 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-19T02:41:16.022 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-19T02:41:16.022 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-19T02:41:16.022 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-19T02:41:16.022 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-19T02:41:16.022 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-19T02:41:16.022 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-19T02:41:16.023 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-19T02:41:16.023 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-19T02:41:16.023 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-19T02:41:16.023 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-19T02:41:16.023 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-19T02:41:16.023 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-19T02:41:16.023 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-19T02:41:16.023 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-19T02:41:16.023 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-19T02:41:16.023 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-19T02:41:16.023 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-19T02:41:16.023 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-19T02:41:16.023 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-19T02:41:16.023 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-19T02:41:16.024 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-19T02:41:16.024 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-19T02:41:16.024 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-19T02:41:16.024 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-19T02:41:16.024 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-19T02:41:16.024 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-19T02:41:16.024 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-19T02:41:16.024 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-19T02:41:16.024 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-19T02:41:16.024 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-19T02:41:16.024 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-19T02:41:16.024 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-19T02:41:16.024 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-19T02:41:16.025 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-19T02:41:16.025 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-19T02:41:16.025 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-19T02:41:18.900 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-19T02:41:18.901 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-19T02:41:18.901 INFO:teuthology.orchestra.run.smithi096.stdout:Upgraded: 2024-03-19T02:41:18.901 INFO:teuthology.orchestra.run.smithi096.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-19T02:41:18.901 INFO:teuthology.orchestra.run.smithi096.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-19T02:41:18.901 INFO:teuthology.orchestra.run.smithi096.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-19T02:41:18.901 INFO:teuthology.orchestra.run.smithi096.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-19T02:41:18.901 INFO:teuthology.orchestra.run.smithi096.stdout: acl-2.3.1-4.el9.x86_64 2024-03-19T02:41:18.901 INFO:teuthology.orchestra.run.smithi096.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-19T02:41:18.901 INFO:teuthology.orchestra.run.smithi096.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-19T02:41:18.901 INFO:teuthology.orchestra.run.smithi096.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-19T02:41:18.901 INFO:teuthology.orchestra.run.smithi096.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-19T02:41:18.901 INFO:teuthology.orchestra.run.smithi096.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-19T02:41:18.901 INFO:teuthology.orchestra.run.smithi096.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-19T02:41:18.902 INFO:teuthology.orchestra.run.smithi096.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-19T02:41:18.902 INFO:teuthology.orchestra.run.smithi096.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-19T02:41:18.902 INFO:teuthology.orchestra.run.smithi096.stdout: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:41:18.902 INFO:teuthology.orchestra.run.smithi096.stdout: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:41:18.902 INFO:teuthology.orchestra.run.smithi096.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-19T02:41:18.902 INFO:teuthology.orchestra.run.smithi096.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-19T02:41:18.902 INFO:teuthology.orchestra.run.smithi096.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-19T02:41:18.902 INFO:teuthology.orchestra.run.smithi096.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-19T02:41:18.902 INFO:teuthology.orchestra.run.smithi096.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-19T02:41:18.902 INFO:teuthology.orchestra.run.smithi096.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-19T02:41:18.902 INFO:teuthology.orchestra.run.smithi096.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-19T02:41:18.902 INFO:teuthology.orchestra.run.smithi096.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-19T02:41:18.902 INFO:teuthology.orchestra.run.smithi096.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-19T02:41:18.903 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-03-19T02:41:18.903 INFO:teuthology.orchestra.run.smithi096.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-19T02:41:18.903 INFO:teuthology.orchestra.run.smithi096.stdout: automake-1.16.2-8.el9.noarch 2024-03-19T02:41:18.903 INFO:teuthology.orchestra.run.smithi096.stdout: bison-3.7.4-5.el9.x86_64 2024-03-19T02:41:18.903 INFO:teuthology.orchestra.run.smithi096.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-19T02:41:18.903 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:41:18.903 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:41:18.903 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:41:18.903 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:41:18.903 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:41:18.903 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:41:18.903 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:41:18.903 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:41:18.903 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:41:18.903 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:41:18.904 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:41:18.904 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:41:18.904 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:41:18.904 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:41:18.904 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:41:18.904 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:41:18.904 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:41:18.904 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:41:18.904 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:41:18.904 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:41:18.904 INFO:teuthology.orchestra.run.smithi096.stdout: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:41:18.904 INFO:teuthology.orchestra.run.smithi096.stdout: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:41:18.905 INFO:teuthology.orchestra.run.smithi096.stdout: cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:41:18.905 INFO:teuthology.orchestra.run.smithi096.stdout: cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T02:41:18.905 INFO:teuthology.orchestra.run.smithi096.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-19T02:41:18.905 INFO:teuthology.orchestra.run.smithi096.stdout: flex-2.6.4-9.el9.x86_64 2024-03-19T02:41:18.905 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-19T02:41:18.905 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-19T02:41:18.905 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-19T02:41:18.905 INFO:teuthology.orchestra.run.smithi096.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-19T02:41:18.905 INFO:teuthology.orchestra.run.smithi096.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-19T02:41:18.905 INFO:teuthology.orchestra.run.smithi096.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-19T02:41:18.906 INFO:teuthology.orchestra.run.smithi096.stdout: jq-1.6-16.el9.x86_64 2024-03-19T02:41:18.906 INFO:teuthology.orchestra.run.smithi096.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-19T02:41:18.906 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-19T02:41:18.906 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-19T02:41:18.906 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:41:18.906 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:41:18.906 INFO:teuthology.orchestra.run.smithi096.stdout: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:41:18.906 INFO:teuthology.orchestra.run.smithi096.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-19T02:41:18.906 INFO:teuthology.orchestra.run.smithi096.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-19T02:41:18.906 INFO:teuthology.orchestra.run.smithi096.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-19T02:41:18.907 INFO:teuthology.orchestra.run.smithi096.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-19T02:41:18.907 INFO:teuthology.orchestra.run.smithi096.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-19T02:41:18.907 INFO:teuthology.orchestra.run.smithi096.stdout: librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:41:18.907 INFO:teuthology.orchestra.run.smithi096.stdout: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:41:18.907 INFO:teuthology.orchestra.run.smithi096.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-19T02:41:18.907 INFO:teuthology.orchestra.run.smithi096.stdout: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:41:18.907 INFO:teuthology.orchestra.run.smithi096.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-19T02:41:18.907 INFO:teuthology.orchestra.run.smithi096.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-19T02:41:18.907 INFO:teuthology.orchestra.run.smithi096.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-19T02:41:18.907 INFO:teuthology.orchestra.run.smithi096.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-19T02:41:18.907 INFO:teuthology.orchestra.run.smithi096.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-19T02:41:18.907 INFO:teuthology.orchestra.run.smithi096.stdout: lua-5.4.4-4.el9.x86_64 2024-03-19T02:41:18.907 INFO:teuthology.orchestra.run.smithi096.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-19T02:41:18.907 INFO:teuthology.orchestra.run.smithi096.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-19T02:41:18.908 INFO:teuthology.orchestra.run.smithi096.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-19T02:41:18.908 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-19T02:41:18.908 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-19T02:41:18.908 INFO:teuthology.orchestra.run.smithi096.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-19T02:41:18.908 INFO:teuthology.orchestra.run.smithi096.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-19T02:41:18.908 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-13.11-1.el9.x86_64 2024-03-19T02:41:18.908 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-19T02:41:18.908 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-19T02:41:18.908 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-19T02:41:18.908 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-19T02:41:18.908 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-19T02:41:18.908 INFO:teuthology.orchestra.run.smithi096.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-19T02:41:18.908 INFO:teuthology.orchestra.run.smithi096.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-19T02:41:18.908 INFO:teuthology.orchestra.run.smithi096.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-19T02:41:18.908 INFO:teuthology.orchestra.run.smithi096.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-19T02:41:18.909 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-19T02:41:18.909 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:41:18.909 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:41:18.909 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:41:18.909 INFO:teuthology.orchestra.run.smithi096.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-19T02:41:18.909 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-19T02:41:18.909 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-19T02:41:18.909 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-19T02:41:18.909 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-19T02:41:18.909 INFO:teuthology.orchestra.run.smithi096.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-19T02:41:18.909 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-19T02:41:18.909 INFO:teuthology.orchestra.run.smithi096.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-19T02:41:18.909 INFO:teuthology.orchestra.run.smithi096.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-19T02:41:18.910 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-19T02:41:18.910 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-19T02:41:18.910 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-19T02:41:18.910 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-19T02:41:18.910 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-19T02:41:18.910 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-19T02:41:18.910 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-19T02:41:18.910 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-19T02:41:18.910 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-19T02:41:18.910 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-19T02:41:18.910 INFO:teuthology.orchestra.run.smithi096.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-19T02:41:18.910 INFO:teuthology.orchestra.run.smithi096.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-19T02:41:18.910 INFO:teuthology.orchestra.run.smithi096.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-19T02:41:18.910 INFO:teuthology.orchestra.run.smithi096.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-19T02:41:18.911 INFO:teuthology.orchestra.run.smithi096.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-19T02:41:18.911 INFO:teuthology.orchestra.run.smithi096.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-19T02:41:18.911 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-19T02:41:18.911 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-19T02:41:18.911 INFO:teuthology.orchestra.run.smithi096.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-19T02:41:18.911 INFO:teuthology.orchestra.run.smithi096.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-19T02:41:18.911 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-19T02:41:18.911 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ply-3.11-14.el9.noarch 2024-03-19T02:41:18.911 INFO:teuthology.orchestra.run.smithi096.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-19T02:41:18.911 INFO:teuthology.orchestra.run.smithi096.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-19T02:41:18.911 INFO:teuthology.orchestra.run.smithi096.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-19T02:41:18.912 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-19T02:41:18.912 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-19T02:41:18.912 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-19T02:41:18.912 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-19T02:41:18.912 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-19T02:41:18.912 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-19T02:41:18.912 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:41:18.912 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:41:18.912 INFO:teuthology.orchestra.run.smithi096.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-19T02:41:18.912 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-19T02:41:18.912 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-19T02:41:18.912 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:41:18.912 INFO:teuthology.orchestra.run.smithi096.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-19T02:41:18.913 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-19T02:41:18.913 INFO:teuthology.orchestra.run.smithi096.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-19T02:41:18.913 INFO:teuthology.orchestra.run.smithi096.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-19T02:41:18.913 INFO:teuthology.orchestra.run.smithi096.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-19T02:41:18.913 INFO:teuthology.orchestra.run.smithi096.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-19T02:41:18.913 INFO:teuthology.orchestra.run.smithi096.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-19T02:41:18.913 INFO:teuthology.orchestra.run.smithi096.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-19T02:41:18.913 INFO:teuthology.orchestra.run.smithi096.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-19T02:41:18.913 INFO:teuthology.orchestra.run.smithi096.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-19T02:41:18.913 INFO:teuthology.orchestra.run.smithi096.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-19T02:41:18.913 INFO:teuthology.orchestra.run.smithi096.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-19T02:41:18.913 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-19T02:41:18.913 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-19T02:41:18.913 INFO:teuthology.orchestra.run.smithi096.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-19T02:41:18.914 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:41:18.914 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:41:18.914 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T02:41:18.914 INFO:teuthology.orchestra.run.smithi096.stdout: re2-1:20211101-3.el9.x86_64 2024-03-19T02:41:18.914 INFO:teuthology.orchestra.run.smithi096.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-19T02:41:18.914 INFO:teuthology.orchestra.run.smithi096.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-19T02:41:18.914 INFO:teuthology.orchestra.run.smithi096.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-19T02:41:18.914 INFO:teuthology.orchestra.run.smithi096.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-19T02:41:18.914 INFO:teuthology.orchestra.run.smithi096.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-19T02:41:18.914 INFO:teuthology.orchestra.run.smithi096.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-19T02:41:18.915 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-03-19T02:41:18.915 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-03-19T02:41:19.460 DEBUG:teuthology.parallel:result is None 2024-03-19T02:41:19.460 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:41:19.688 DEBUG:teuthology.orchestra.run.smithi033:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-19T02:41:19.716 INFO:teuthology.orchestra.run.smithi033.stdout:19.0.0-2206.gb6ae2491.el9 2024-03-19T02:41:19.717 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2206.gb6ae2491.el9 2024-03-19T02:41:19.717 INFO:teuthology.task.install:The correct ceph version 19.0.0-2206.gb6ae2491 is installed. 2024-03-19T02:41:19.719 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:41:19.865 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-19T02:41:19.886 INFO:teuthology.orchestra.run.smithi096.stdout:19.0.0-2206.gb6ae2491.el9 2024-03-19T02:41:19.886 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2206.gb6ae2491.el9 2024-03-19T02:41:19.886 INFO:teuthology.task.install:The correct ceph version 19.0.0-2206.gb6ae2491 is installed. 2024-03-19T02:41:19.888 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:41:20.105 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-19T02:41:20.132 INFO:teuthology.orchestra.run.smithi155.stdout:19.0.0-2206.gb6ae2491.el9 2024-03-19T02:41:20.132 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2206.gb6ae2491.el9 2024-03-19T02:41:20.133 INFO:teuthology.task.install:The correct ceph version 19.0.0-2206.gb6ae2491 is installed. 2024-03-19T02:41:20.134 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-19T02:41:20.135 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-19T02:41:20.135 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-19T02:41:20.172 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-19T02:41:20.172 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-19T02:41:20.202 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-19T02:41:20.202 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-19T02:41:20.236 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-19T02:41:20.236 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-19T02:41:20.236 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/daemon-helper 2024-03-19T02:41:20.267 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-19T02:41:20.336 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-19T02:41:20.336 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/daemon-helper 2024-03-19T02:41:20.363 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-19T02:41:20.430 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-19T02:41:20.430 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/daemon-helper 2024-03-19T02:41:20.462 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-19T02:41:20.536 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-19T02:41:20.536 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-19T02:41:20.536 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-19T02:41:20.565 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-19T02:41:20.633 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-19T02:41:20.634 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-19T02:41:20.660 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-19T02:41:20.727 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-19T02:41:20.727 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-19T02:41:20.755 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-19T02:41:20.822 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-19T02:41:20.823 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-19T02:41:20.823 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/stdin-killer 2024-03-19T02:41:20.852 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-19T02:41:20.921 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-19T02:41:20.922 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/stdin-killer 2024-03-19T02:41:20.949 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-19T02:41:21.016 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-19T02:41:21.016 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/stdin-killer 2024-03-19T02:41:21.043 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-19T02:41:21.111 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-19T02:41:21.223 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'PG_AVAILABILITY', 'PG_DEGRADED', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-03-19T02:41:21.223 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:41:21.224 INFO:tasks.cephadm:Cluster fsid is 2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T02:41:21.224 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-19T02:41:21.224 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.33', 'mon.b': '172.21.15.96', 'mon.c': '172.21.15.155'} 2024-03-19T02:41:21.224 INFO:tasks.cephadm:First mon is mon.a on smithi033 2024-03-19T02:41:21.224 INFO:tasks.cephadm:First mgr is x 2024-03-19T02:41:21.224 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-19T02:41:21.224 DEBUG:teuthology.orchestra.run.smithi033:> sudo hostname $(hostname -s) 2024-03-19T02:41:21.255 DEBUG:teuthology.orchestra.run.smithi096:> sudo hostname $(hostname -s) 2024-03-19T02:41:21.283 DEBUG:teuthology.orchestra.run.smithi155:> sudo hostname $(hostname -s) 2024-03-19T02:41:21.311 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-19T02:41:21.311 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:41:21.451 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77736/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2206-gb6ae2491', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2206.gb6ae2491'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-18 20:42:27.872142', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-batrick-testing-20240318.181317', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-19T02:41:21.674 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-batrick-testing-20240318.181317, sha1 b6ae24918b03b1b8a19c9263a50655bba80bb810 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:41:21.675 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/x86_64/flavors/default/cephadm 2024-03-19T02:41:21.675 DEBUG:teuthology.orchestra.run.smithi033:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-19T02:41:22.135 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-r--r--. 1 ubuntu ubuntu 767404 Mar 19 02:41 /home/ubuntu/cephtest/cephadm 2024-03-19T02:41:22.135 DEBUG:teuthology.orchestra.run.smithi096:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-19T02:41:22.621 INFO:teuthology.orchestra.run.smithi096.stdout:-rw-r--r--. 1 ubuntu ubuntu 767404 Mar 19 02:41 /home/ubuntu/cephtest/cephadm 2024-03-19T02:41:22.622 DEBUG:teuthology.orchestra.run.smithi155:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-19T02:41:23.487 INFO:teuthology.orchestra.run.smithi155.stdout:-rw-r--r--. 1 ubuntu ubuntu 767404 Mar 19 02:41 /home/ubuntu/cephtest/cephadm 2024-03-19T02:41:23.488 DEBUG:teuthology.orchestra.run.smithi033:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-19T02:41:23.507 DEBUG:teuthology.orchestra.run.smithi096:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-19T02:41:23.526 DEBUG:teuthology.orchestra.run.smithi155:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-19T02:41:23.568 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 on all hosts... 2024-03-19T02:41:23.568 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 pull 2024-03-19T02:41:23.571 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 pull 2024-03-19T02:41:23.575 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 pull 2024-03-19T02:41:24.398 INFO:teuthology.orchestra.run.smithi033.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810... 2024-03-19T02:41:24.478 INFO:teuthology.orchestra.run.smithi155.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810... 2024-03-19T02:41:24.522 INFO:teuthology.orchestra.run.smithi096.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810... 2024-03-19T02:42:26.224 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-03-19T02:42:26.224 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_version": "ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev)", 2024-03-19T02:42:26.225 INFO:teuthology.orchestra.run.smithi155.stdout: "image_id": "2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c", 2024-03-19T02:42:26.225 INFO:teuthology.orchestra.run.smithi155.stdout: "repo_digests": [ 2024-03-19T02:42:26.225 INFO:teuthology.orchestra.run.smithi155.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639" 2024-03-19T02:42:26.225 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-03-19T02:42:26.225 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-03-19T02:42:28.020 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:42:28.020 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_version": "ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev)", 2024-03-19T02:42:28.020 INFO:teuthology.orchestra.run.smithi033.stdout: "image_id": "2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c", 2024-03-19T02:42:28.021 INFO:teuthology.orchestra.run.smithi033.stdout: "repo_digests": [ 2024-03-19T02:42:28.021 INFO:teuthology.orchestra.run.smithi033.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639" 2024-03-19T02:42:28.021 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-03-19T02:42:28.021 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:42:29.089 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-03-19T02:42:29.090 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_version": "ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev)", 2024-03-19T02:42:29.090 INFO:teuthology.orchestra.run.smithi096.stdout: "image_id": "2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c", 2024-03-19T02:42:29.090 INFO:teuthology.orchestra.run.smithi096.stdout: "repo_digests": [ 2024-03-19T02:42:29.090 INFO:teuthology.orchestra.run.smithi096.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639" 2024-03-19T02:42:29.090 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-03-19T02:42:29.090 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-03-19T02:42:29.115 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /etc/ceph 2024-03-19T02:42:29.150 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /etc/ceph 2024-03-19T02:42:29.180 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /etc/ceph 2024-03-19T02:42:29.215 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 777 /etc/ceph 2024-03-19T02:42:29.246 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 777 /etc/ceph 2024-03-19T02:42:29.274 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 777 /etc/ceph 2024-03-19T02:42:29.306 INFO:tasks.cephadm:Writing seed config... 2024-03-19T02:42:29.307 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-19T02:42:29.307 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-19T02:42:29.307 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-19T02:42:29.307 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-19T02:42:29.308 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-19T02:42:29.308 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-19T02:42:29.308 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-19T02:42:29.308 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-19T02:42:29.308 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-19T02:42:29.308 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-19T02:42:29.308 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-19T02:42:29.308 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-19T02:42:29.308 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-19T02:42:29.308 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-19T02:42:29.308 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-19T02:42:29.308 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-19T02:42:29.308 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-03-19T02:42:29.308 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-19T02:42:29.308 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-19T02:42:29.308 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-19T02:42:29.309 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-19T02:42:29.309 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-03-19T02:42:29.309 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-19T02:42:29.309 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-19T02:42:29.309 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-19T02:42:29.309 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-19T02:42:29.309 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-19T02:42:29.309 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-19T02:42:29.309 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-19T02:42:29.309 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-19T02:42:29.309 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-19T02:42:29.309 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-19T02:42:29.309 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-19T02:42:29.309 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-19T02:42:29.309 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-19T02:42:29.310 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-19T02:42:29.310 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-19T02:42:29.310 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-19T02:42:29.310 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-19T02:42:29.310 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-19T02:42:29.310 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-19T02:42:29.310 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-19T02:42:29.310 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-03-19T02:42:29.311 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-19T02:42:29.311 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-19T02:42:29.329 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 2b7a8938-e59a-11ee-95c9-87774f69a715 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-19T02:42:29.329 DEBUG:teuthology.orchestra.run.smithi033:mon.a> sudo journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mon.a.service 2024-03-19T02:42:29.372 DEBUG:teuthology.orchestra.run.smithi033:mgr.x> sudo journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mgr.x.service 2024-03-19T02:42:29.415 INFO:tasks.cephadm:Bootstrapping... 2024-03-19T02:42:29.415 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 -v bootstrap --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.33 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-19T02:42:29.616 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:42:29.616 INFO:teuthology.orchestra.run.smithi033.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810', '-v', 'bootstrap', '--fsid', '2b7a8938-e59a-11ee-95c9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.33', '--skip-admin-label'] 2024-03-19T02:42:29.642 INFO:teuthology.orchestra.run.smithi033.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-19T02:42:29.642 INFO:teuthology.orchestra.run.smithi033.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-19T02:42:29.643 INFO:teuthology.orchestra.run.smithi033.stdout:Verifying podman|docker is present... 2024-03-19T02:42:29.666 INFO:teuthology.orchestra.run.smithi033.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-19T02:42:29.666 INFO:teuthology.orchestra.run.smithi033.stdout:Verifying lvm2 is present... 2024-03-19T02:42:29.667 INFO:teuthology.orchestra.run.smithi033.stdout:Verifying time synchronization is in place... 2024-03-19T02:42:29.672 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-19T02:42:29.672 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-19T02:42:29.678 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-19T02:42:29.678 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout inactive 2024-03-19T02:42:29.684 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout enabled 2024-03-19T02:42:29.690 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout active 2024-03-19T02:42:29.690 INFO:teuthology.orchestra.run.smithi033.stdout:Unit chronyd.service is enabled and running 2024-03-19T02:42:29.690 INFO:teuthology.orchestra.run.smithi033.stdout:Repeating the final host check... 2024-03-19T02:42:29.712 INFO:teuthology.orchestra.run.smithi033.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-19T02:42:29.712 INFO:teuthology.orchestra.run.smithi033.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-19T02:42:29.712 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl is present 2024-03-19T02:42:29.712 INFO:teuthology.orchestra.run.smithi033.stdout:lvcreate is present 2024-03-19T02:42:29.718 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-19T02:42:29.719 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-19T02:42:29.724 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-19T02:42:29.724 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout inactive 2024-03-19T02:42:29.730 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout enabled 2024-03-19T02:42:29.736 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout active 2024-03-19T02:42:29.736 INFO:teuthology.orchestra.run.smithi033.stdout:Unit chronyd.service is enabled and running 2024-03-19T02:42:29.736 INFO:teuthology.orchestra.run.smithi033.stdout:Host looks OK 2024-03-19T02:42:29.736 INFO:teuthology.orchestra.run.smithi033.stdout:Cluster fsid: 2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T02:42:29.736 INFO:teuthology.orchestra.run.smithi033.stdout:Acquiring lock 140401638432672 on /run/cephadm/2b7a8938-e59a-11ee-95c9-87774f69a715.lock 2024-03-19T02:42:29.737 INFO:teuthology.orchestra.run.smithi033.stdout:Lock 140401638432672 acquired on /run/cephadm/2b7a8938-e59a-11ee-95c9-87774f69a715.lock 2024-03-19T02:42:29.737 INFO:teuthology.orchestra.run.smithi033.stdout:Verifying IP 172.21.15.33 port 3300 ... 2024-03-19T02:42:29.737 INFO:teuthology.orchestra.run.smithi033.stdout:Verifying IP 172.21.15.33 port 6789 ... 2024-03-19T02:42:29.737 INFO:teuthology.orchestra.run.smithi033.stdout:Base mon IP(s) is [172.21.15.33:3300, 172.21.15.33:6789], mon addrv is [v2:172.21.15.33:3300,v1:172.21.15.33:6789] 2024-03-19T02:42:29.741 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.33 metric 100 2024-03-19T02:42:29.741 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.33 metric 100 2024-03-19T02:42:29.743 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-19T02:42:29.743 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-19T02:42:29.743 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-19T02:42:29.745 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-19T02:42:29.746 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-19T02:42:29.746 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-19T02:42:29.746 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-19T02:42:29.746 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fea5:d294/64 scope link noprefixroute 2024-03-19T02:42:29.746 INFO:teuthology.orchestra.run.smithi033.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-19T02:42:29.746 INFO:teuthology.orchestra.run.smithi033.stdout:Mon IP `172.21.15.33` is in CIDR network `172.21.0.0/20` 2024-03-19T02:42:29.746 INFO:teuthology.orchestra.run.smithi033.stdout:Mon IP `172.21.15.33` is in CIDR network `172.21.0.0/20` 2024-03-19T02:42:29.747 INFO:teuthology.orchestra.run.smithi033.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-19T02:42:29.747 INFO:teuthology.orchestra.run.smithi033.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-19T02:42:29.748 INFO:teuthology.orchestra.run.smithi033.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810... 2024-03-19T02:42:30.213 INFO:teuthology.orchestra.run.smithi033.stdout:/bin/podman: stdout 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c 2024-03-19T02:42:30.214 INFO:teuthology.orchestra.run.smithi033.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810... 2024-03-19T02:42:30.214 INFO:teuthology.orchestra.run.smithi033.stdout:/bin/podman: stderr Getting image source signatures 2024-03-19T02:42:30.214 INFO:teuthology.orchestra.run.smithi033.stdout:/bin/podman: stderr Copying blob sha256:93b5a8d362aeee7e8b1f0bbf18238e1bb256c50573ad5b72aff9e382b1bb3ccb 2024-03-19T02:42:30.214 INFO:teuthology.orchestra.run.smithi033.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-19T02:42:30.214 INFO:teuthology.orchestra.run.smithi033.stdout:/bin/podman: stderr Copying config sha256:2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c 2024-03-19T02:42:30.214 INFO:teuthology.orchestra.run.smithi033.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-19T02:42:31.127 INFO:teuthology.orchestra.run.smithi033.stdout:ceph: stdout ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev) 2024-03-19T02:42:31.127 INFO:teuthology.orchestra.run.smithi033.stdout:Ceph version: ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev) 2024-03-19T02:42:31.127 INFO:teuthology.orchestra.run.smithi033.stdout:Extracting ceph user uid/gid from container image... 2024-03-19T02:42:32.083 INFO:teuthology.orchestra.run.smithi033.stdout:stat: stdout 167 167 2024-03-19T02:42:32.083 INFO:teuthology.orchestra.run.smithi033.stdout:Creating initial keys... 2024-03-19T02:42:33.005 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph-authtool: stdout AQCY+/hlOzdgGBAAT+gHzAG3dDxqLtnidwlcXg== 2024-03-19T02:42:33.929 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph-authtool: stdout AQCZ+/hlSES+EBAAQqTxhL9cp3pxmRsP4VSOMA== 2024-03-19T02:42:34.828 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph-authtool: stdout AQCa+/hlfFdIDhAAYW5L0E+5alLV93+ysqzI4Q== 2024-03-19T02:42:34.829 INFO:teuthology.orchestra.run.smithi033.stdout:Creating initial monmap... 2024-03-19T02:42:35.617 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-19T02:42:35.617 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-19T02:42:35.617 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T02:42:35.617 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-19T02:42:35.618 INFO:teuthology.orchestra.run.smithi033.stdout:monmaptool for a [v2:172.21.15.33:3300,v1:172.21.15.33:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-19T02:42:35.618 INFO:teuthology.orchestra.run.smithi033.stdout:setting min_mon_release = quincy 2024-03-19T02:42:35.618 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/monmaptool: set fsid to 2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T02:42:35.618 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-19T02:42:35.618 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:42:35.618 INFO:teuthology.orchestra.run.smithi033.stdout:Creating mon... 2024-03-19T02:42:36.432 INFO:teuthology.orchestra.run.smithi033.stdout:create mon.a on 2024-03-19T02:42:36.635 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-19T02:42:36.772 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-19T02:42:36.939 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-2b7a8938-e59a-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-2b7a8938-e59a-11ee-95c9-87774f69a715.target. 2024-03-19T02:42:36.940 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-2b7a8938-e59a-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-2b7a8938-e59a-11ee-95c9-87774f69a715.target. 2024-03-19T02:42:37.186 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mon.a 2024-03-19T02:42:37.186 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stderr Failed to reset failed state of unit ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mon.a.service: Unit ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mon.a.service not loaded. 2024-03-19T02:42:37.354 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-2b7a8938-e59a-11ee-95c9-87774f69a715.target.wants/ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mon.a.service → /etc/systemd/system/ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@.service. 2024-03-19T02:42:37.940 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-19T02:42:37.940 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout disabled 2024-03-19T02:42:37.947 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-19T02:42:37.947 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout inactive 2024-03-19T02:42:37.947 INFO:teuthology.orchestra.run.smithi033.stdout:firewalld.service is not enabled 2024-03-19T02:42:37.947 INFO:teuthology.orchestra.run.smithi033.stdout:Not possible to enable service . firewalld.service is not available 2024-03-19T02:42:37.948 INFO:teuthology.orchestra.run.smithi033.stdout:Waiting for mon to start... 2024-03-19T02:42:37.948 INFO:teuthology.orchestra.run.smithi033.stdout:Waiting for mon... 2024-03-19T02:42:39.009 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout cluster: 2024-03-19T02:42:39.009 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout id: 2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T02:42:39.009 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-19T02:42:39.009 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-03-19T02:42:39.009 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout services: 2024-03-19T02:42:39.009 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.49798s) 2024-03-19T02:42:39.009 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-19T02:42:39.009 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-19T02:42:39.009 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-03-19T02:42:39.009 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout data: 2024-03-19T02:42:39.009 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-19T02:42:39.009 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-19T02:42:39.010 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-19T02:42:39.010 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout pgs: 2024-03-19T02:42:39.010 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-03-19T02:42:39.010 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.328+0000 7fde54b6e700 1 Processor -- start 2024-03-19T02:42:39.010 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.328+0000 7fde54b6e700 1 -- start start 2024-03-19T02:42:39.010 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.328+0000 7fde54b6e700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde50108580 0x7fde5010a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:39.010 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.328+0000 7fde54b6e700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fde5007be20 con 0x7fde50108580 2024-03-19T02:42:39.010 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.329+0000 7fde4e59c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde50108580 0x7fde5010a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:39.010 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.329+0000 7fde4e59c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde50108580 0x7fde5010a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50196/0 (socket says 172.21.15.33:50196) 2024-03-19T02:42:39.010 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.329+0000 7fde4e59c700 1 -- 172.21.15.33:0/1224398923 learned_addr learned my addr 172.21.15.33:0/1224398923 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:42:39.010 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.329+0000 7fde4e59c700 1 -- 172.21.15.33:0/1224398923 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde5010aeb0 con 0x7fde50108580 2024-03-19T02:42:39.010 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.329+0000 7fde4e59c700 1 --2- 172.21.15.33:0/1224398923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde50108580 0x7fde5010a970 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fde400098a0 tx=0x7fde40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdae5fc08f6b4724 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:39.010 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.329+0000 7fde4d59a700 1 -- 172.21.15.33:0/1224398923 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde40004160 con 0x7fde50108580 2024-03-19T02:42:39.011 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.329+0000 7fde4d59a700 1 -- 172.21.15.33:0/1224398923 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde40004300 con 0x7fde50108580 2024-03-19T02:42:39.011 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.329+0000 7fde4d59a700 1 -- 172.21.15.33:0/1224398923 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde400036a0 con 0x7fde50108580 2024-03-19T02:42:39.011 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.330+0000 7fde54b6e700 1 -- 172.21.15.33:0/1224398923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde50108580 msgr2=0x7fde5010a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:39.011 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.330+0000 7fde54b6e700 1 --2- 172.21.15.33:0/1224398923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde50108580 0x7fde5010a970 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fde400098a0 tx=0x7fde40009bb0 comp rx=0 tx=0).stop 2024-03-19T02:42:39.011 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.330+0000 7fde54b6e700 1 -- 172.21.15.33:0/1224398923 shutdown_connections 2024-03-19T02:42:39.011 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.330+0000 7fde54b6e700 1 --2- 172.21.15.33:0/1224398923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde50108580 0x7fde5010a970 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:39.011 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.330+0000 7fde54b6e700 1 -- 172.21.15.33:0/1224398923 >> 172.21.15.33:0/1224398923 conn(0x7fde50101fb0 msgr2=0x7fde501043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:39.011 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.330+0000 7fde54b6e700 1 -- 172.21.15.33:0/1224398923 shutdown_connections 2024-03-19T02:42:39.011 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.330+0000 7fde54b6e700 1 -- 172.21.15.33:0/1224398923 wait complete. 2024-03-19T02:42:39.011 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.331+0000 7fde54b6e700 1 Processor -- start 2024-03-19T02:42:39.011 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.331+0000 7fde54b6e700 1 -- start start 2024-03-19T02:42:39.011 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.331+0000 7fde54b6e700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde50108580 0x7fde5019b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:39.011 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.331+0000 7fde54b6e700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fde5010b320 con 0x7fde50108580 2024-03-19T02:42:39.012 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.331+0000 7fde4e59c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde50108580 0x7fde5019b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:39.012 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.331+0000 7fde4e59c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde50108580 0x7fde5019b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50206/0 (socket says 172.21.15.33:50206) 2024-03-19T02:42:39.012 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.331+0000 7fde4e59c700 1 -- 172.21.15.33:0/375884261 learned_addr learned my addr 172.21.15.33:0/375884261 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:42:39.012 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.331+0000 7fde4e59c700 1 -- 172.21.15.33:0/375884261 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde5019b590 con 0x7fde50108580 2024-03-19T02:42:39.012 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.332+0000 7fde4e59c700 1 --2- 172.21.15.33:0/375884261 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde50108580 0x7fde5019b050 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fde40000c00 tx=0x7fde40004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:39.012 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.332+0000 7fde3f7fe700 1 -- 172.21.15.33:0/375884261 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde40003db0 con 0x7fde50108580 2024-03-19T02:42:39.012 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.332+0000 7fde3f7fe700 1 -- 172.21.15.33:0/375884261 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde40011630 con 0x7fde50108580 2024-03-19T02:42:39.012 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.332+0000 7fde54b6e700 1 -- 172.21.15.33:0/375884261 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde5019b820 con 0x7fde50108580 2024-03-19T02:42:39.012 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.332+0000 7fde3f7fe700 1 -- 172.21.15.33:0/375884261 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde40011950 con 0x7fde50108580 2024-03-19T02:42:39.012 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.332+0000 7fde54b6e700 1 -- 172.21.15.33:0/375884261 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde5019e510 con 0x7fde50108580 2024-03-19T02:42:39.012 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.332+0000 7fde3f7fe700 1 -- 172.21.15.33:0/375884261 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fde40011af0 con 0x7fde50108580 2024-03-19T02:42:39.012 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.332+0000 7fde3f7fe700 1 -- 172.21.15.33:0/375884261 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fde4001c790 con 0x7fde50108580 2024-03-19T02:42:39.012 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.333+0000 7fde54b6e700 1 -- 172.21.15.33:0/375884261 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde50063940 con 0x7fde50108580 2024-03-19T02:42:39.013 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.335+0000 7fde3f7fe700 1 -- 172.21.15.33:0/375884261 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fde40032020 con 0x7fde50108580 2024-03-19T02:42:39.013 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.408+0000 7fde54b6e700 1 -- 172.21.15.33:0/375884261 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fde5019e780 con 0x7fde50108580 2024-03-19T02:42:39.013 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.408+0000 7fde3f7fe700 1 -- 172.21.15.33:0/375884261 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+316 (secure 0 0 0) 0x7fde40004260 con 0x7fde50108580 2024-03-19T02:42:39.013 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.409+0000 7fde54b6e700 1 -- 172.21.15.33:0/375884261 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde50108580 msgr2=0x7fde5019b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:39.013 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.409+0000 7fde54b6e700 1 --2- 172.21.15.33:0/375884261 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde50108580 0x7fde5019b050 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fde40000c00 tx=0x7fde40004750 comp rx=0 tx=0).stop 2024-03-19T02:42:39.013 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.409+0000 7fde54b6e700 1 -- 172.21.15.33:0/375884261 shutdown_connections 2024-03-19T02:42:39.013 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.409+0000 7fde54b6e700 1 --2- 172.21.15.33:0/375884261 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde50108580 0x7fde5019b050 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:39.013 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.409+0000 7fde54b6e700 1 -- 172.21.15.33:0/375884261 >> 172.21.15.33:0/375884261 conn(0x7fde50101fb0 msgr2=0x7fde50102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:39.013 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.409+0000 7fde54b6e700 1 -- 172.21.15.33:0/375884261 shutdown_connections 2024-03-19T02:42:39.013 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:38.409+0000 7fde54b6e700 1 -- 172.21.15.33:0/375884261 wait complete. 2024-03-19T02:42:39.013 INFO:teuthology.orchestra.run.smithi033.stdout:mon is available 2024-03-19T02:42:39.013 INFO:teuthology.orchestra.run.smithi033.stdout:Assimilating anything we can from ceph.conf... 2024-03-19T02:42:40.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:40 smithi033 ceph-mon[31595]: from='client.? 172.21.15.33:0/3086036375' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-19T02:42:46.339 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-03-19T02:42:46.339 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout [global] 2024-03-19T02:42:46.339 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout fsid = 2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T02:42:46.339 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.33:3300,v1:172.21.15.33:6789] 2024-03-19T02:42:46.339 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-19T02:42:46.339 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-19T02:42:46.339 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-19T02:42:46.339 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-19T02:42:46.339 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-03-19T02:42:46.339 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-19T02:42:46.339 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-19T02:42:46.350 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-03-19T02:42:46.350 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout [osd] 2024-03-19T02:42:46.350 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-19T02:42:46.350 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-19T02:42:46.351 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-19T02:42:46.351 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-19T02:42:46.351 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-19T02:42:46.351 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.569+0000 7fd0a8b54700 1 Processor -- start 2024-03-19T02:42:46.351 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.569+0000 7fd0a8b54700 1 -- start start 2024-03-19T02:42:46.351 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.569+0000 7fd0a8b54700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0a4108500 0x7fd0a410a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:46.351 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.569+0000 7fd0a8b54700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a407be80 con 0x7fd0a4108500 2024-03-19T02:42:46.352 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.569+0000 7fd0a259c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0a4108500 0x7fd0a410a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:46.352 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.569+0000 7fd0a259c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0a4108500 0x7fd0a410a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50214/0 (socket says 172.21.15.33:50214) 2024-03-19T02:42:46.352 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.570+0000 7fd0a259c700 1 -- 172.21.15.33:0/2673017621 learned_addr learned my addr 172.21.15.33:0/2673017621 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:42:46.352 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.570+0000 7fd0a259c700 1 -- 172.21.15.33:0/2673017621 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0a410ae30 con 0x7fd0a4108500 2024-03-19T02:42:46.352 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.570+0000 7fd0a259c700 1 --2- 172.21.15.33:0/2673017621 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0a4108500 0x7fd0a410a8f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fd0940098a0 tx=0x7fd094009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f9d197ec7762043 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:46.352 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.570+0000 7fd0a159a700 1 -- 172.21.15.33:0/2673017621 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd094004160 con 0x7fd0a4108500 2024-03-19T02:42:46.352 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.570+0000 7fd0a159a700 1 -- 172.21.15.33:0/2673017621 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd094004300 con 0x7fd0a4108500 2024-03-19T02:42:46.353 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.570+0000 7fd0a159a700 1 -- 172.21.15.33:0/2673017621 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd0940036a0 con 0x7fd0a4108500 2024-03-19T02:42:46.353 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.571+0000 7fd0a8b54700 1 -- 172.21.15.33:0/2673017621 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0a4108500 msgr2=0x7fd0a410a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:46.353 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.571+0000 7fd0a8b54700 1 --2- 172.21.15.33:0/2673017621 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0a4108500 0x7fd0a410a8f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fd0940098a0 tx=0x7fd094009bb0 comp rx=0 tx=0).stop 2024-03-19T02:42:46.353 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.571+0000 7fd0a8b54700 1 -- 172.21.15.33:0/2673017621 shutdown_connections 2024-03-19T02:42:46.353 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.571+0000 7fd0a8b54700 1 --2- 172.21.15.33:0/2673017621 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0a4108500 0x7fd0a410a8f0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:46.353 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.571+0000 7fd0a8b54700 1 -- 172.21.15.33:0/2673017621 >> 172.21.15.33:0/2673017621 conn(0x7fd0a4101f30 msgr2=0x7fd0a4104370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:46.354 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.571+0000 7fd0a8b54700 1 -- 172.21.15.33:0/2673017621 shutdown_connections 2024-03-19T02:42:46.354 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.571+0000 7fd0a8b54700 1 -- 172.21.15.33:0/2673017621 wait complete. 2024-03-19T02:42:46.354 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.572+0000 7fd0a8b54700 1 Processor -- start 2024-03-19T02:42:46.354 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.572+0000 7fd0a8b54700 1 -- start start 2024-03-19T02:42:46.354 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.572+0000 7fd0a8b54700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0a4108500 0x7fd0a419b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:46.354 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.572+0000 7fd0a8b54700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a410b300 con 0x7fd0a4108500 2024-03-19T02:42:46.354 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.572+0000 7fd0a259c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0a4108500 0x7fd0a419b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:46.354 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.572+0000 7fd0a259c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0a4108500 0x7fd0a419b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50226/0 (socket says 172.21.15.33:50226) 2024-03-19T02:42:46.355 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.572+0000 7fd0a259c700 1 -- 172.21.15.33:0/3086036375 learned_addr learned my addr 172.21.15.33:0/3086036375 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:42:46.355 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.572+0000 7fd0a259c700 1 -- 172.21.15.33:0/3086036375 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0a419b5a0 con 0x7fd0a4108500 2024-03-19T02:42:46.355 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.572+0000 7fd0a259c700 1 --2- 172.21.15.33:0/3086036375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0a4108500 0x7fd0a419b060 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fd094000c00 tx=0x7fd094004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:46.355 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.572+0000 7fd0937fe700 1 -- 172.21.15.33:0/3086036375 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd094003db0 con 0x7fd0a4108500 2024-03-19T02:42:46.355 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.572+0000 7fd0937fe700 1 -- 172.21.15.33:0/3086036375 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd094011630 con 0x7fd0a4108500 2024-03-19T02:42:46.355 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.572+0000 7fd0a8b54700 1 -- 172.21.15.33:0/3086036375 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0a419b830 con 0x7fd0a4108500 2024-03-19T02:42:46.355 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.573+0000 7fd0937fe700 1 -- 172.21.15.33:0/3086036375 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd094011950 con 0x7fd0a4108500 2024-03-19T02:42:46.355 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.573+0000 7fd0a8b54700 1 -- 172.21.15.33:0/3086036375 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0a419e520 con 0x7fd0a4108500 2024-03-19T02:42:46.355 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.573+0000 7fd0937fe700 1 -- 172.21.15.33:0/3086036375 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd094011af0 con 0x7fd0a4108500 2024-03-19T02:42:46.356 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.573+0000 7fd0937fe700 1 -- 172.21.15.33:0/3086036375 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd09401c450 con 0x7fd0a4108500 2024-03-19T02:42:46.356 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.574+0000 7fd0a8b54700 1 -- 172.21.15.33:0/3086036375 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0a4063940 con 0x7fd0a4108500 2024-03-19T02:42:46.356 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.576+0000 7fd0937fe700 1 -- 172.21.15.33:0/3086036375 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fd094032020 con 0x7fd0a4108500 2024-03-19T02:42:46.356 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.647+0000 7fd0a8b54700 1 -- 172.21.15.33:0/3086036375 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fd0a419e6f0 con 0x7fd0a4108500 2024-03-19T02:42:46.356 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.651+0000 7fd0937fe700 1 -- 172.21.15.33:0/3086036375 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7fd094004260 con 0x7fd0a4108500 2024-03-19T02:42:46.356 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.651+0000 7fd0937fe700 1 -- 172.21.15.33:0/3086036375 <== mon.0 v2:172.21.15.33:3300/0 8 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fd09401c6a0 con 0x7fd0a4108500 2024-03-19T02:42:46.356 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.652+0000 7fd0a8b54700 1 -- 172.21.15.33:0/3086036375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0a4108500 msgr2=0x7fd0a419b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:46.356 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.652+0000 7fd0a8b54700 1 --2- 172.21.15.33:0/3086036375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0a4108500 0x7fd0a419b060 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fd094000c00 tx=0x7fd094004750 comp rx=0 tx=0).stop 2024-03-19T02:42:46.356 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.652+0000 7fd0a8b54700 1 -- 172.21.15.33:0/3086036375 shutdown_connections 2024-03-19T02:42:46.357 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.652+0000 7fd0a8b54700 1 --2- 172.21.15.33:0/3086036375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0a4108500 0x7fd0a419b060 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:46.357 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.652+0000 7fd0a8b54700 1 -- 172.21.15.33:0/3086036375 >> 172.21.15.33:0/3086036375 conn(0x7fd0a4101f30 msgr2=0x7fd0a4102c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:46.357 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.652+0000 7fd0a8b54700 1 -- 172.21.15.33:0/3086036375 shutdown_connections 2024-03-19T02:42:46.357 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:40.652+0000 7fd0a8b54700 1 -- 172.21.15.33:0/3086036375 wait complete. 2024-03-19T02:42:46.357 INFO:teuthology.orchestra.run.smithi033.stdout:Generating new minimal ceph.conf... 2024-03-19T02:42:47.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:47 smithi033 ceph-mon[31595]: from='client.? 172.21.15.33:0/50679438' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:42:48.060 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.461+0000 7f7ce8565700 1 Processor -- start 2024-03-19T02:42:48.061 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.462+0000 7f7ce8565700 1 -- start start 2024-03-19T02:42:48.061 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.462+0000 7f7ce8565700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ce007de50 0x7f7ce007c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:48.061 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.462+0000 7f7ce8565700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce007c880 con 0x7f7ce007de50 2024-03-19T02:42:48.061 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.462+0000 7f7ce6301700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ce007de50 0x7f7ce007c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:48.061 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.462+0000 7f7ce6301700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ce007de50 0x7f7ce007c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33714/0 (socket says 172.21.15.33:33714) 2024-03-19T02:42:48.061 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.462+0000 7f7ce6301700 1 -- 172.21.15.33:0/3861922220 learned_addr learned my addr 172.21.15.33:0/3861922220 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:42:48.061 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.463+0000 7f7ce6301700 1 -- 172.21.15.33:0/3861922220 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ce007ca00 con 0x7f7ce007de50 2024-03-19T02:42:48.061 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.463+0000 7f7ce6301700 1 --2- 172.21.15.33:0/3861922220 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ce007de50 0x7f7ce007c2b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7cd40098a0 tx=0x7f7cd4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf3b3133b14cf871 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:48.061 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.463+0000 7f7ce52ff700 1 -- 172.21.15.33:0/3861922220 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7cd4004160 con 0x7f7ce007de50 2024-03-19T02:42:48.061 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.463+0000 7f7ce52ff700 1 -- 172.21.15.33:0/3861922220 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f7cd4004300 con 0x7f7ce007de50 2024-03-19T02:42:48.061 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.463+0000 7f7ce52ff700 1 -- 172.21.15.33:0/3861922220 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7cd4011a80 con 0x7f7ce007de50 2024-03-19T02:42:48.062 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.463+0000 7f7ce8565700 1 -- 172.21.15.33:0/3861922220 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ce007de50 msgr2=0x7f7ce007c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:48.062 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.463+0000 7f7ce8565700 1 --2- 172.21.15.33:0/3861922220 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ce007de50 0x7f7ce007c2b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7cd40098a0 tx=0x7f7cd4009bb0 comp rx=0 tx=0).stop 2024-03-19T02:42:48.062 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.464+0000 7f7ce8565700 1 -- 172.21.15.33:0/3861922220 shutdown_connections 2024-03-19T02:42:48.062 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.464+0000 7f7ce8565700 1 --2- 172.21.15.33:0/3861922220 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ce007de50 0x7f7ce007c2b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:48.062 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.464+0000 7f7ce8565700 1 -- 172.21.15.33:0/3861922220 >> 172.21.15.33:0/3861922220 conn(0x7f7ce0103020 msgr2=0x7f7ce0105480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:48.062 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.464+0000 7f7ce8565700 1 -- 172.21.15.33:0/3861922220 shutdown_connections 2024-03-19T02:42:48.062 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.464+0000 7f7ce8565700 1 -- 172.21.15.33:0/3861922220 wait complete. 2024-03-19T02:42:48.062 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.464+0000 7f7ce8565700 1 Processor -- start 2024-03-19T02:42:48.062 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.464+0000 7f7ce8565700 1 -- start start 2024-03-19T02:42:48.062 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.464+0000 7f7ce8565700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ce007de50 0x7f7ce019db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:48.063 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.464+0000 7f7ce8565700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce010e070 con 0x7f7ce007de50 2024-03-19T02:42:48.063 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.465+0000 7f7ce6301700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ce007de50 0x7f7ce019db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:48.063 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.465+0000 7f7ce6301700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ce007de50 0x7f7ce019db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33724/0 (socket says 172.21.15.33:33724) 2024-03-19T02:42:48.063 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.465+0000 7f7ce6301700 1 -- 172.21.15.33:0/50679438 learned_addr learned my addr 172.21.15.33:0/50679438 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:42:48.063 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.465+0000 7f7ce6301700 1 -- 172.21.15.33:0/50679438 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ce019e050 con 0x7f7ce007de50 2024-03-19T02:42:48.063 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.465+0000 7f7ce6301700 1 --2- 172.21.15.33:0/50679438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ce007de50 0x7f7ce019db10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4006af0 tx=0x7f7cd4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:48.063 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.465+0000 7f7cd37fe700 1 -- 172.21.15.33:0/50679438 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7cd40039e0 con 0x7f7ce007de50 2024-03-19T02:42:48.063 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.465+0000 7f7cd37fe700 1 -- 172.21.15.33:0/50679438 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f7cd4003b80 con 0x7f7ce007de50 2024-03-19T02:42:48.063 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.465+0000 7f7ce8565700 1 -- 172.21.15.33:0/50679438 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ce019e2e0 con 0x7f7ce007de50 2024-03-19T02:42:48.063 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.465+0000 7f7cd37fe700 1 -- 172.21.15.33:0/50679438 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7cd40196a0 con 0x7f7ce007de50 2024-03-19T02:42:48.063 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.465+0000 7f7ce8565700 1 -- 172.21.15.33:0/50679438 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ce01a0fd0 con 0x7f7ce007de50 2024-03-19T02:42:48.063 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.465+0000 7f7cd37fe700 1 -- 172.21.15.33:0/50679438 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7cd4019d00 con 0x7f7ce007de50 2024-03-19T02:42:48.063 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.465+0000 7f7cd37fe700 1 -- 172.21.15.33:0/50679438 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7cd4024730 con 0x7f7ce007de50 2024-03-19T02:42:48.063 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.466+0000 7f7ce8565700 1 -- 172.21.15.33:0/50679438 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ce0063940 con 0x7f7ce007de50 2024-03-19T02:42:48.064 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.468+0000 7f7cd37fe700 1 -- 172.21.15.33:0/50679438 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f7cd4043dd0 con 0x7f7ce007de50 2024-03-19T02:42:48.064 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.539+0000 7f7ce8565700 1 -- 172.21.15.33:0/50679438 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f7ce01a1580 con 0x7f7ce007de50 2024-03-19T02:42:48.064 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.539+0000 7f7cd37fe700 1 -- 172.21.15.33:0/50679438 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f7cd4032020 con 0x7f7ce007de50 2024-03-19T02:42:48.064 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.540+0000 7f7ce8565700 1 -- 172.21.15.33:0/50679438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ce007de50 msgr2=0x7f7ce019db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:48.064 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.540+0000 7f7ce8565700 1 --2- 172.21.15.33:0/50679438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ce007de50 0x7f7ce019db10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4006af0 tx=0x7f7cd4011c40 comp rx=0 tx=0).stop 2024-03-19T02:42:48.064 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.540+0000 7f7ce8565700 1 -- 172.21.15.33:0/50679438 shutdown_connections 2024-03-19T02:42:48.064 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.540+0000 7f7ce8565700 1 --2- 172.21.15.33:0/50679438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ce007de50 0x7f7ce019db10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:48.064 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.540+0000 7f7ce8565700 1 -- 172.21.15.33:0/50679438 >> 172.21.15.33:0/50679438 conn(0x7f7ce0103020 msgr2=0x7f7ce0103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:48.064 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.540+0000 7f7ce8565700 1 -- 172.21.15.33:0/50679438 shutdown_connections 2024-03-19T02:42:48.064 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:47.540+0000 7f7ce8565700 1 -- 172.21.15.33:0/50679438 wait complete. 2024-03-19T02:42:48.064 INFO:teuthology.orchestra.run.smithi033.stdout:Restarting the monitor... 2024-03-19T02:42:48.184 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:48 smithi033 systemd[1]: Stopping Ceph mon.a for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T02:42:48.483 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:48 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mon-a[31591]: 2024-03-19T02:42:48.315+0000 7ffa14815700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T02:42:48.483 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:48 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mon-a[31591]: 2024-03-19T02:42:48.315+0000 7ffa14815700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-19T02:42:48.758 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:48 smithi033 podman[31845]: 2024-03-19 02:42:48.499718769 +0000 UTC m=+0.285552911 container died 9dbb0ea83b7a91473b80d2bb5ac5161a101072c4640d7de5c873c403a0d61572 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mon-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-19T02:42:49.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:48 smithi033 podman[31845]: 2024-03-19 02:42:48.84893179 +0000 UTC m=+0.634765916 container cleanup 9dbb0ea83b7a91473b80d2bb5ac5161a101072c4640d7de5c873c403a0d61572 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mon-a, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-19T02:42:49.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:48 smithi033 bash[31845]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mon-a 2024-03-19T02:42:49.467 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 podman[31859]: 2024-03-19 02:42:49.116814564 +0000 UTC m=+0.616414881 container remove 9dbb0ea83b7a91473b80d2bb5ac5161a101072c4640d7de5c873c403a0d61572 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mon-a, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-19T02:42:49.468 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 systemd[1]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-19T02:42:49.468 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 systemd[1]: Stopped Ceph mon.a for 2b7a8938-e59a-11ee-95c9-87774f69a715. 2024-03-19T02:42:49.468 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 systemd[1]: Starting Ceph mon.a for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T02:42:49.719 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 podman[31954]: 2024-03-19 02:42:49.467686971 +0000 UTC m=+0.101593077 container create 24ae7c13ccaa60b50b98a22bf85d87f102639af4a19cc16cb6f753ec99f1ba13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mon-a, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-19T02:42:49.720 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 podman[31954]: 2024-03-19 02:42:49.383304444 +0000 UTC m=+0.017210546 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:42:49.720 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 podman[31954]: 2024-03-19 02:42:49.720091659 +0000 UTC m=+0.353997753 container init 24ae7c13ccaa60b50b98a22bf85d87f102639af4a19cc16cb6f753ec99f1ba13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mon-a, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-19T02:42:49.807 INFO:teuthology.orchestra.run.smithi033.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-19T02:42:50.006 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 podman[31954]: 2024-03-19 02:42:49.722904573 +0000 UTC m=+0.356810637 container start 24ae7c13ccaa60b50b98a22bf85d87f102639af4a19cc16cb6f753ec99f1ba13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mon-a, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-19T02:42:50.007 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: set uid:gid to 167:167 (ceph:ceph) 2024-03-19T02:42:50.007 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev), process ceph-mon, pid 2 2024-03-19T02:42:50.007 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: pidfile_write: ignore empty --pid-file 2024-03-19T02:42:50.007 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: load: jerasure load: lrc 2024-03-19T02:42:50.007 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: RocksDB version: 7.9.2 2024-03-19T02:42:50.007 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Git sha 0 2024-03-19T02:42:50.007 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Compile date 2024-03-18 18:57:12 2024-03-19T02:42:50.007 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: DB SUMMARY 2024-03-19T02:42:50.007 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: DB Session ID: 88HWEPR1UYN77VC7FC7L 2024-03-19T02:42:50.007 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: CURRENT file: CURRENT 2024-03-19T02:42:50.007 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: IDENTITY file: IDENTITY 2024-03-19T02:42:50.007 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-19T02:42:50.007 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-19T02:42:50.008 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 103656 ; 2024-03-19T02:42:50.008 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.error_if_exists: 0 2024-03-19T02:42:50.008 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.create_if_missing: 0 2024-03-19T02:42:50.008 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.paranoid_checks: 1 2024-03-19T02:42:50.008 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-19T02:42:50.008 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-19T02:42:50.008 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-19T02:42:50.008 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.env: 0x556901550c60 2024-03-19T02:42:50.008 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.fs: PosixFileSystem 2024-03-19T02:42:50.008 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.info_log: 0x556903bc7ae0 2024-03-19T02:42:50.008 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.max_file_opening_threads: 16 2024-03-19T02:42:50.008 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.statistics: (nil) 2024-03-19T02:42:50.008 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.use_fsync: 0 2024-03-19T02:42:50.008 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.max_log_file_size: 0 2024-03-19T02:42:50.009 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-19T02:42:50.009 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-19T02:42:50.009 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.keep_log_file_num: 1000 2024-03-19T02:42:50.009 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.recycle_log_file_num: 0 2024-03-19T02:42:50.009 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.allow_fallocate: 1 2024-03-19T02:42:50.010 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.allow_mmap_reads: 0 2024-03-19T02:42:50.010 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.allow_mmap_writes: 0 2024-03-19T02:42:50.010 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.use_direct_reads: 0 2024-03-19T02:42:50.010 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-19T02:42:50.010 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.create_missing_column_families: 0 2024-03-19T02:42:50.010 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.db_log_dir: 2024-03-19T02:42:50.010 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.wal_dir: 2024-03-19T02:42:50.010 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-19T02:42:50.010 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-19T02:42:50.010 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-19T02:42:50.011 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-19T02:42:50.011 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-19T02:42:50.011 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-19T02:42:50.011 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.advise_random_on_open: 1 2024-03-19T02:42:50.011 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.db_write_buffer_size: 0 2024-03-19T02:42:50.011 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.write_buffer_manager: 0x556903b9a1e0 2024-03-19T02:42:50.011 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-19T02:42:50.011 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-19T02:42:50.011 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-19T02:42:50.011 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.rate_limiter: (nil) 2024-03-19T02:42:50.011 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-19T02:42:50.011 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.wal_recovery_mode: 2 2024-03-19T02:42:50.011 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.enable_thread_tracking: 0 2024-03-19T02:42:50.011 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.enable_pipelined_write: 0 2024-03-19T02:42:50.012 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.unordered_write: 0 2024-03-19T02:42:50.012 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-19T02:42:50.012 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-19T02:42:50.012 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-19T02:42:50.012 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-19T02:42:50.012 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.row_cache: None 2024-03-19T02:42:50.012 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.wal_filter: None 2024-03-19T02:42:50.012 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-19T02:42:50.012 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.allow_ingest_behind: 0 2024-03-19T02:42:50.012 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.two_write_queues: 0 2024-03-19T02:42:50.012 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.manual_wal_flush: 0 2024-03-19T02:42:50.012 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.wal_compression: 0 2024-03-19T02:42:50.012 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.atomic_flush: 0 2024-03-19T02:42:50.012 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-19T02:42:50.013 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-19T02:42:50.013 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-19T02:42:50.013 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.log_readahead_size: 0 2024-03-19T02:42:50.013 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-19T02:42:50.013 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.best_efforts_recovery: 0 2024-03-19T02:42:50.013 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-19T02:42:50.013 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-19T02:42:50.013 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.allow_data_in_errors: 0 2024-03-19T02:42:50.013 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.db_host_id: __hostname__ 2024-03-19T02:42:50.013 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-19T02:42:50.013 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.max_background_jobs: 2 2024-03-19T02:42:50.013 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.max_background_compactions: -1 2024-03-19T02:42:50.013 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.max_subcompactions: 1 2024-03-19T02:42:50.014 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-19T02:42:50.014 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-19T02:42:50.014 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-19T02:42:50.014 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.max_total_wal_size: 0 2024-03-19T02:42:50.014 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-19T02:42:50.014 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-19T02:42:50.014 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-19T02:42:50.014 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-19T02:42:50.014 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.max_open_files: -1 2024-03-19T02:42:50.014 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.bytes_per_sync: 0 2024-03-19T02:42:50.014 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-19T02:42:50.014 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-19T02:42:50.014 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.compaction_readahead_size: 0 2024-03-19T02:42:50.014 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.max_background_flushes: -1 2024-03-19T02:42:50.015 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Compression algorithms supported: 2024-03-19T02:42:50.015 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-19T02:42:50.015 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: kZSTD supported: 0 2024-03-19T02:42:50.015 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: kXpressCompression supported: 0 2024-03-19T02:42:50.015 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: kLZ4HCCompression supported: 1 2024-03-19T02:42:50.015 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: kZlibCompression supported: 1 2024-03-19T02:42:50.015 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: kSnappyCompression supported: 1 2024-03-19T02:42:50.015 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: kLZ4Compression supported: 1 2024-03-19T02:42:50.015 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: kBZip2Compression supported: 0 2024-03-19T02:42:50.015 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-19T02:42:50.015 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-19T02:42:50.015 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-19T02:42:50.015 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-19T02:42:50.015 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-19T02:42:50.015 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.merge_operator: 2024-03-19T02:42:50.016 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.compaction_filter: None 2024-03-19T02:42:50.016 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.compaction_filter_factory: None 2024-03-19T02:42:50.016 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.sst_partitioner_factory: None 2024-03-19T02:42:50.016 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-19T02:42:50.016 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-19T02:42:50.016 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556903bc7be0) 2024-03-19T02:42:50.016 INFO:journalctl@ceph.mon.a.smithi033.stdout: cache_index_and_filter_blocks: 1 2024-03-19T02:42:50.016 INFO:journalctl@ceph.mon.a.smithi033.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-19T02:42:50.016 INFO:journalctl@ceph.mon.a.smithi033.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-19T02:42:50.016 INFO:journalctl@ceph.mon.a.smithi033.stdout: pin_top_level_index_and_filter: 1 2024-03-19T02:42:50.016 INFO:journalctl@ceph.mon.a.smithi033.stdout: index_type: 0 2024-03-19T02:42:50.016 INFO:journalctl@ceph.mon.a.smithi033.stdout: data_block_index_type: 0 2024-03-19T02:42:50.016 INFO:journalctl@ceph.mon.a.smithi033.stdout: index_shortening: 1 2024-03-19T02:42:50.016 INFO:journalctl@ceph.mon.a.smithi033.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-19T02:42:50.017 INFO:journalctl@ceph.mon.a.smithi033.stdout: checksum: 4 2024-03-19T02:42:50.017 INFO:journalctl@ceph.mon.a.smithi033.stdout: no_block_cache: 0 2024-03-19T02:42:50.017 INFO:journalctl@ceph.mon.a.smithi033.stdout: block_cache: 0x556903bcd090 2024-03-19T02:42:50.017 INFO:journalctl@ceph.mon.a.smithi033.stdout: block_cache_name: BinnedLRUCache 2024-03-19T02:42:50.017 INFO:journalctl@ceph.mon.a.smithi033.stdout: block_cache_options: 2024-03-19T02:42:50.017 INFO:journalctl@ceph.mon.a.smithi033.stdout: capacity : 536870912 2024-03-19T02:42:50.017 INFO:journalctl@ceph.mon.a.smithi033.stdout: num_shard_bits : 4 2024-03-19T02:42:50.017 INFO:journalctl@ceph.mon.a.smithi033.stdout: strict_capacity_limit : 0 2024-03-19T02:42:50.017 INFO:journalctl@ceph.mon.a.smithi033.stdout: high_pri_pool_ratio: 0.000 2024-03-19T02:42:50.017 INFO:journalctl@ceph.mon.a.smithi033.stdout: block_cache_compressed: (nil) 2024-03-19T02:42:50.017 INFO:journalctl@ceph.mon.a.smithi033.stdout: persistent_cache: (nil) 2024-03-19T02:42:50.017 INFO:journalctl@ceph.mon.a.smithi033.stdout: block_size: 4096 2024-03-19T02:42:50.017 INFO:journalctl@ceph.mon.a.smithi033.stdout: block_size_deviation: 10 2024-03-19T02:42:50.017 INFO:journalctl@ceph.mon.a.smithi033.stdout: block_restart_interval: 16 2024-03-19T02:42:50.018 INFO:journalctl@ceph.mon.a.smithi033.stdout: index_block_restart_interval: 1 2024-03-19T02:42:50.018 INFO:journalctl@ceph.mon.a.smithi033.stdout: metadata_block_size: 4096 2024-03-19T02:42:50.018 INFO:journalctl@ceph.mon.a.smithi033.stdout: partition_filters: 0 2024-03-19T02:42:50.018 INFO:journalctl@ceph.mon.a.smithi033.stdout: use_delta_encoding: 1 2024-03-19T02:42:50.018 INFO:journalctl@ceph.mon.a.smithi033.stdout: filter_policy: bloomfilter 2024-03-19T02:42:50.018 INFO:journalctl@ceph.mon.a.smithi033.stdout: whole_key_filtering: 1 2024-03-19T02:42:50.018 INFO:journalctl@ceph.mon.a.smithi033.stdout: verify_compression: 0 2024-03-19T02:42:50.018 INFO:journalctl@ceph.mon.a.smithi033.stdout: read_amp_bytes_per_bit: 0 2024-03-19T02:42:50.018 INFO:journalctl@ceph.mon.a.smithi033.stdout: format_version: 5 2024-03-19T02:42:50.018 INFO:journalctl@ceph.mon.a.smithi033.stdout: enable_index_compression: 1 2024-03-19T02:42:50.018 INFO:journalctl@ceph.mon.a.smithi033.stdout: block_align: 0 2024-03-19T02:42:50.018 INFO:journalctl@ceph.mon.a.smithi033.stdout: max_auto_readahead_size: 262144 2024-03-19T02:42:50.018 INFO:journalctl@ceph.mon.a.smithi033.stdout: prepopulate_block_cache: 0 2024-03-19T02:42:50.018 INFO:journalctl@ceph.mon.a.smithi033.stdout: initial_auto_readahead_size: 8192 2024-03-19T02:42:50.019 INFO:journalctl@ceph.mon.a.smithi033.stdout: num_file_reads_for_auto_readahead: 2 2024-03-19T02:42:50.019 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.write_buffer_size: 33554432 2024-03-19T02:42:50.019 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.max_write_buffer_number: 2 2024-03-19T02:42:50.019 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.compression: NoCompression 2024-03-19T02:42:50.019 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.bottommost_compression: Disabled 2024-03-19T02:42:50.019 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.prefix_extractor: nullptr 2024-03-19T02:42:50.019 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-19T02:42:50.019 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.num_levels: 7 2024-03-19T02:42:50.019 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-19T02:42:50.020 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-19T02:42:50.020 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-19T02:42:50.020 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-19T02:42:50.020 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-19T02:42:50.020 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-19T02:42:50.020 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-19T02:42:50.020 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-19T02:42:50.020 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-19T02:42:50.020 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-19T02:42:50.020 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-19T02:42:50.020 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-19T02:42:50.020 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-19T02:42:50.020 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.compression_opts.level: 32767 2024-03-19T02:42:50.020 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.compression_opts.strategy: 0 2024-03-19T02:42:50.021 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-19T02:42:50.021 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-19T02:42:50.021 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-19T02:42:50.021 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-19T02:42:50.021 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.compression_opts.enabled: false 2024-03-19T02:42:50.021 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-19T02:42:50.021 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-19T02:42:50.021 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-19T02:42:50.021 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-19T02:42:50.021 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.target_file_size_base: 67108864 2024-03-19T02:42:50.021 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-19T02:42:50.021 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-19T02:42:50.021 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-19T02:42:50.021 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-19T02:42:50.021 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-19T02:42:50.022 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-19T02:42:50.022 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-19T02:42:50.022 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-19T02:42:50.022 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-19T02:42:50.022 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-19T02:42:50.022 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-19T02:42:50.022 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-19T02:42:50.022 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-19T02:42:50.022 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-19T02:42:50.023 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.arena_block_size: 1048576 2024-03-19T02:42:50.023 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-19T02:42:50.023 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-19T02:42:50.023 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.disable_auto_compactions: 0 2024-03-19T02:42:50.023 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-19T02:42:50.023 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-19T02:42:50.023 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-19T02:42:50.023 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-19T02:42:50.023 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-19T02:42:50.024 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-19T02:42:50.024 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-19T02:42:50.024 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-19T02:42:50.024 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-19T02:42:50.024 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-19T02:42:50.024 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.table_properties_collectors: 2024-03-19T02:42:50.024 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.inplace_update_support: 0 2024-03-19T02:42:50.024 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-19T02:42:50.024 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-19T02:42:50.024 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-19T02:42:50.024 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-19T02:42:50.024 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.bloom_locality: 0 2024-03-19T02:42:50.024 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.max_successive_merges: 0 2024-03-19T02:42:50.024 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-19T02:42:50.024 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.paranoid_file_checks: 0 2024-03-19T02:42:50.025 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.force_consistency_checks: 1 2024-03-19T02:42:50.025 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.report_bg_io_stats: 0 2024-03-19T02:42:50.025 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.ttl: 2592000 2024-03-19T02:42:50.025 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-19T02:42:50.025 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-19T02:42:50.025 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-19T02:42:50.025 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.enable_blob_files: false 2024-03-19T02:42:50.025 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.min_blob_size: 0 2024-03-19T02:42:50.025 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.blob_file_size: 268435456 2024-03-19T02:42:50.025 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-19T02:42:50.025 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-19T02:42:50.025 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-19T02:42:50.025 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-19T02:42:50.025 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-19T02:42:50.025 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.blob_file_starting_level: 0 2024-03-19T02:42:50.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-19T02:42:50.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-19T02:42:50.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-19T02:42:50.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d4063a72-a6e2-46de-9fd0-efd228a3bb1a 2024-03-19T02:42:50.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710816169751723, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-19T02:42:50.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-19T02:42:50.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710816169752698, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 98865, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 320, "table_properties": {"data_size": 96844, "index_size": 303, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13812, "raw_average_key_size": 49, "raw_value_size": 89319, "raw_average_value_size": 320, "num_data_blocks": 12, "num_entries": 279, "num_filter_entries": 279, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710816169, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d4063a72-a6e2-46de-9fd0-efd228a3bb1a", "db_session_id": "88HWEPR1UYN77VC7FC7L", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-19T02:42:50.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710816169752783, "job": 1, "event": "recovery_finished"} 2024-03-19T02:42:50.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-19T02:42:50.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-19T02:42:50.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x556903cc0000 2024-03-19T02:42:50.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: DB pointer 0x556903caa000 2024-03-19T02:42:50.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-19T02:42:50.027 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-19T02:42:50.027 INFO:journalctl@ceph.mon.a.smithi033.stdout: ** DB Stats ** 2024-03-19T02:42:50.027 INFO:journalctl@ceph.mon.a.smithi033.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-19T02:42:50.027 INFO:journalctl@ceph.mon.a.smithi033.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-19T02:42:50.027 INFO:journalctl@ceph.mon.a.smithi033.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-19T02:42:50.027 INFO:journalctl@ceph.mon.a.smithi033.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-19T02:42:50.027 INFO:journalctl@ceph.mon.a.smithi033.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-19T02:42:50.027 INFO:journalctl@ceph.mon.a.smithi033.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-19T02:42:50.027 INFO:journalctl@ceph.mon.a.smithi033.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-19T02:42:50.027 INFO:journalctl@ceph.mon.a.smithi033.stdout: 2024-03-19T02:42:50.027 INFO:journalctl@ceph.mon.a.smithi033.stdout: ** Compaction Stats [default] ** 2024-03-19T02:42:50.027 INFO:journalctl@ceph.mon.a.smithi033.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-19T02:42:50.027 INFO:journalctl@ceph.mon.a.smithi033.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-19T02:42:50.027 INFO:journalctl@ceph.mon.a.smithi033.stdout: L0 2/0 98.38 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-19T02:42:50.027 INFO:journalctl@ceph.mon.a.smithi033.stdout: Sum 2/0 98.38 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-19T02:42:50.028 INFO:journalctl@ceph.mon.a.smithi033.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-19T02:42:50.028 INFO:journalctl@ceph.mon.a.smithi033.stdout: 2024-03-19T02:42:50.028 INFO:journalctl@ceph.mon.a.smithi033.stdout: ** Compaction Stats [default] ** 2024-03-19T02:42:50.028 INFO:journalctl@ceph.mon.a.smithi033.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-19T02:42:50.028 INFO:journalctl@ceph.mon.a.smithi033.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-19T02:42:50.028 INFO:journalctl@ceph.mon.a.smithi033.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 150.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-19T02:42:50.028 INFO:journalctl@ceph.mon.a.smithi033.stdout: 2024-03-19T02:42:50.028 INFO:journalctl@ceph.mon.a.smithi033.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-19T02:42:50.028 INFO:journalctl@ceph.mon.a.smithi033.stdout: 2024-03-19T02:42:50.028 INFO:journalctl@ceph.mon.a.smithi033.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-19T02:42:50.028 INFO:journalctl@ceph.mon.a.smithi033.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-19T02:42:50.028 INFO:journalctl@ceph.mon.a.smithi033.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-19T02:42:50.028 INFO:journalctl@ceph.mon.a.smithi033.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-19T02:42:50.028 INFO:journalctl@ceph.mon.a.smithi033.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-19T02:42:50.028 INFO:journalctl@ceph.mon.a.smithi033.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-19T02:42:50.029 INFO:journalctl@ceph.mon.a.smithi033.stdout: Cumulative compaction: 0.00 GB write, 32.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-19T02:42:50.029 INFO:journalctl@ceph.mon.a.smithi033.stdout: Interval compaction: 0.00 GB write, 32.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-19T02:42:50.029 INFO:journalctl@ceph.mon.a.smithi033.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-19T02:42:50.029 INFO:journalctl@ceph.mon.a.smithi033.stdout: Block cache BinnedLRUCache@0x556903bcd090#2 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-19T02:42:50.029 INFO:journalctl@ceph.mon.a.smithi033.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-19T02:42:50.029 INFO:journalctl@ceph.mon.a.smithi033.stdout: 2024-03-19T02:42:50.029 INFO:journalctl@ceph.mon.a.smithi033.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-19T02:42:50.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: starting mon.a rank 0 at public addrs [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] at bind addrs [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T02:42:50.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: mon.a@-1(???) e1 preinit fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T02:42:50.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: mon.a@-1(???).mds e1 new map 2024-03-19T02:42:50.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: mon.a@-1(???).mds e1 print_map 2024-03-19T02:42:50.029 INFO:journalctl@ceph.mon.a.smithi033.stdout: e1 2024-03-19T02:42:50.029 INFO:journalctl@ceph.mon.a.smithi033.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-19T02:42:50.029 INFO:journalctl@ceph.mon.a.smithi033.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-19T02:42:50.030 INFO:journalctl@ceph.mon.a.smithi033.stdout: legacy client fscid: -1 2024-03-19T02:42:50.030 INFO:journalctl@ceph.mon.a.smithi033.stdout: 2024-03-19T02:42:50.030 INFO:journalctl@ceph.mon.a.smithi033.stdout: No filesystems configured 2024-03-19T02:42:50.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-19T02:42:50.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-19T02:42:50.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-19T02:42:50.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-19T02:42:50.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-19T02:42:50.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 bash[31954]: 24ae7c13ccaa60b50b98a22bf85d87f102639af4a19cc16cb6f753ec99f1ba13 2024-03-19T02:42:50.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 systemd[1]: Started Ceph mon.a for 2b7a8938-e59a-11ee-95c9-87774f69a715. 2024-03-19T02:42:50.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-19T02:42:50.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: monmap epoch 1 2024-03-19T02:42:50.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T02:42:50.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: last_changed 2024-03-19T02:42:35.143659+0000 2024-03-19T02:42:50.031 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: created 2024-03-19T02:42:35.143659+0000 2024-03-19T02:42:50.031 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: min_mon_release 19 (squid) 2024-03-19T02:42:50.031 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: election_strategy: 1 2024-03-19T02:42:50.031 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: 0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2024-03-19T02:42:50.031 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: fsmap 2024-03-19T02:42:50.031 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: osdmap e1: 0 total, 0 up, 0 in 2024-03-19T02:42:50.031 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:49 smithi033 ceph-mon[31973]: mgrmap e1: no daemons active 2024-03-19T02:42:50.950 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.211+0000 7f9deb546700 1 Processor -- start 2024-03-19T02:42:50.950 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.211+0000 7f9deb546700 1 -- start start 2024-03-19T02:42:50.950 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.211+0000 7f9deb546700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9de4107a60 0x7f9de4107e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:50.950 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.211+0000 7f9deb546700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9de4108450 con 0x7f9de4107a60 2024-03-19T02:42:50.950 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.211+0000 7f9de92e2700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9de4107a60 0x7f9de4107e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:50.950 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.211+0000 7f9de92e2700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9de4107a60 0x7f9de4107e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33740/0 (socket says 172.21.15.33:33740) 2024-03-19T02:42:50.950 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.211+0000 7f9de92e2700 1 -- 172.21.15.33:0/3608186625 learned_addr learned my addr 172.21.15.33:0/3608186625 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:42:50.950 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.212+0000 7f9de92e2700 1 -- 172.21.15.33:0/3608186625 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9de4108c60 con 0x7f9de4107a60 2024-03-19T02:42:50.951 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.212+0000 7f9de92e2700 1 --2- 172.21.15.33:0/3608186625 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9de4107a60 0x7f9de4107e80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9de00098a0 tx=0x7f9de0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=782988f87d1a84d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:50.951 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.212+0000 7f9ddbfff700 1 -- 172.21.15.33:0/3608186625 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9de0004160 con 0x7f9de4107a60 2024-03-19T02:42:50.951 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.212+0000 7f9ddbfff700 1 -- 172.21.15.33:0/3608186625 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f9de0004300 con 0x7f9de4107a60 2024-03-19T02:42:50.951 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.212+0000 7f9ddbfff700 1 -- 172.21.15.33:0/3608186625 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9de0011a80 con 0x7f9de4107a60 2024-03-19T02:42:50.951 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.212+0000 7f9deb546700 1 -- 172.21.15.33:0/3608186625 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9de4107a60 msgr2=0x7f9de4107e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:50.951 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.212+0000 7f9deb546700 1 --2- 172.21.15.33:0/3608186625 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9de4107a60 0x7f9de4107e80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9de00098a0 tx=0x7f9de0009bb0 comp rx=0 tx=0).stop 2024-03-19T02:42:50.951 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.212+0000 7f9deb546700 1 -- 172.21.15.33:0/3608186625 shutdown_connections 2024-03-19T02:42:50.951 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.212+0000 7f9deb546700 1 --2- 172.21.15.33:0/3608186625 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9de4107a60 0x7f9de4107e80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:50.951 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.212+0000 7f9deb546700 1 -- 172.21.15.33:0/3608186625 >> 172.21.15.33:0/3608186625 conn(0x7f9de4103020 msgr2=0x7f9de4105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:50.951 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.213+0000 7f9deb546700 1 -- 172.21.15.33:0/3608186625 shutdown_connections 2024-03-19T02:42:50.951 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.213+0000 7f9deb546700 1 -- 172.21.15.33:0/3608186625 wait complete. 2024-03-19T02:42:50.951 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.213+0000 7f9deb546700 1 Processor -- start 2024-03-19T02:42:50.951 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.213+0000 7f9deb546700 1 -- start start 2024-03-19T02:42:50.951 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.213+0000 7f9deb546700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9de4107a60 0x7f9de407db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:50.951 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.213+0000 7f9deb546700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9de410c010 con 0x7f9de4107a60 2024-03-19T02:42:50.952 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.213+0000 7f9de92e2700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9de4107a60 0x7f9de407db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:50.952 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.213+0000 7f9de92e2700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9de4107a60 0x7f9de407db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33754/0 (socket says 172.21.15.33:33754) 2024-03-19T02:42:50.952 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.213+0000 7f9de92e2700 1 -- 172.21.15.33:0/4027781134 learned_addr learned my addr 172.21.15.33:0/4027781134 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:42:50.952 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.214+0000 7f9de92e2700 1 -- 172.21.15.33:0/4027781134 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9de407e080 con 0x7f9de4107a60 2024-03-19T02:42:50.952 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.214+0000 7f9de92e2700 1 --2- 172.21.15.33:0/4027781134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9de4107a60 0x7f9de407db40 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9de0006af0 tx=0x7f9de0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:50.952 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.214+0000 7f9dda7fc700 1 -- 172.21.15.33:0/4027781134 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9de00039e0 con 0x7f9de4107a60 2024-03-19T02:42:50.952 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.214+0000 7f9dda7fc700 1 -- 172.21.15.33:0/4027781134 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f9de0003b80 con 0x7f9de4107a60 2024-03-19T02:42:50.952 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.214+0000 7f9deb546700 1 -- 172.21.15.33:0/4027781134 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9de407c270 con 0x7f9de4107a60 2024-03-19T02:42:50.952 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.214+0000 7f9dda7fc700 1 -- 172.21.15.33:0/4027781134 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9de00196a0 con 0x7f9de4107a60 2024-03-19T02:42:50.952 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.214+0000 7f9deb546700 1 -- 172.21.15.33:0/4027781134 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9de407c7b0 con 0x7f9de4107a60 2024-03-19T02:42:50.952 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.214+0000 7f9dda7fc700 1 -- 172.21.15.33:0/4027781134 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9de0019ce0 con 0x7f9de4107a60 2024-03-19T02:42:50.952 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.214+0000 7f9dda7fc700 1 -- 172.21.15.33:0/4027781134 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9de0024730 con 0x7f9de4107a60 2024-03-19T02:42:50.952 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.215+0000 7f9deb546700 1 -- 172.21.15.33:0/4027781134 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9de4063940 con 0x7f9de4107a60 2024-03-19T02:42:50.952 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.217+0000 7f9dda7fc700 1 -- 172.21.15.33:0/4027781134 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f9de0043dd0 con 0x7f9de4107a60 2024-03-19T02:42:50.953 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.287+0000 7f9deb546700 1 -- 172.21.15.33:0/4027781134 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f9de407caf0 con 0x7f9de4107a60 2024-03-19T02:42:50.953 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.292+0000 7f9dda7fc700 1 -- 172.21.15.33:0/4027781134 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f9de0032020 con 0x7f9de4107a60 2024-03-19T02:42:50.953 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.292+0000 7f9deb546700 1 -- 172.21.15.33:0/4027781134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9de4107a60 msgr2=0x7f9de407db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:50.953 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.292+0000 7f9deb546700 1 --2- 172.21.15.33:0/4027781134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9de4107a60 0x7f9de407db40 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9de0006af0 tx=0x7f9de0011c40 comp rx=0 tx=0).stop 2024-03-19T02:42:50.953 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.292+0000 7f9deb546700 1 -- 172.21.15.33:0/4027781134 shutdown_connections 2024-03-19T02:42:50.953 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.292+0000 7f9deb546700 1 --2- 172.21.15.33:0/4027781134 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9de4107a60 0x7f9de407db40 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:50.953 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.292+0000 7f9deb546700 1 -- 172.21.15.33:0/4027781134 >> 172.21.15.33:0/4027781134 conn(0x7f9de4103020 msgr2=0x7f9de4103a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:50.953 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.292+0000 7f9deb546700 1 -- 172.21.15.33:0/4027781134 shutdown_connections 2024-03-19T02:42:50.953 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:50.293+0000 7f9deb546700 1 -- 172.21.15.33:0/4027781134 wait complete. 2024-03-19T02:42:50.953 INFO:teuthology.orchestra.run.smithi033.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-19T02:42:50.977 INFO:teuthology.orchestra.run.smithi033.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-19T02:42:50.977 INFO:teuthology.orchestra.run.smithi033.stdout:Creating mgr... 2024-03-19T02:42:50.977 INFO:teuthology.orchestra.run.smithi033.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-19T02:42:50.977 INFO:teuthology.orchestra.run.smithi033.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-19T02:42:51.227 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mgr.x 2024-03-19T02:42:51.227 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stderr Failed to reset failed state of unit ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mgr.x.service: Unit ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mgr.x.service not loaded. 2024-03-19T02:42:51.379 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-2b7a8938-e59a-11ee-95c9-87774f69a715.target.wants/ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@.service. 2024-03-19T02:42:51.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:51 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/4027781134' entity='client.admin' 2024-03-19T02:42:51.528 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:42:51 smithi033 systemd[1]: Starting Ceph mgr.x for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T02:42:51.915 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:42:51 smithi033 podman[32189]: 2024-03-19 02:42:51.520042432 +0000 UTC m=+0.018497578 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:42:51.915 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:42:51 smithi033 podman[32189]: 2024-03-19 02:42:51.629252823 +0000 UTC m=+0.127708007 container create ed11015124be9cc4cc29ee053e312c8fc9f394c5a33d6aa081bfd911c744dbfd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True) 2024-03-19T02:42:51.915 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:42:51 smithi033 podman[32189]: 2024-03-19 02:42:51.848091857 +0000 UTC m=+0.346547002 container init ed11015124be9cc4cc29ee053e312c8fc9f394c5a33d6aa081bfd911c744dbfd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307) 2024-03-19T02:42:51.915 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:42:51 smithi033 podman[32189]: 2024-03-19 02:42:51.850657826 +0000 UTC m=+0.349112980 container start ed11015124be9cc4cc29ee053e312c8fc9f394c5a33d6aa081bfd911c744dbfd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T02:42:51.915 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:42:51 smithi033 bash[32189]: ed11015124be9cc4cc29ee053e312c8fc9f394c5a33d6aa081bfd911c744dbfd 2024-03-19T02:42:51.927 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-19T02:42:51.927 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout disabled 2024-03-19T02:42:51.934 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-19T02:42:51.934 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout inactive 2024-03-19T02:42:51.934 INFO:teuthology.orchestra.run.smithi033.stdout:firewalld.service is not enabled 2024-03-19T02:42:51.934 INFO:teuthology.orchestra.run.smithi033.stdout:Not possible to enable service . firewalld.service is not available 2024-03-19T02:42:51.942 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-19T02:42:51.942 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout disabled 2024-03-19T02:42:51.950 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-19T02:42:51.950 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout inactive 2024-03-19T02:42:51.950 INFO:teuthology.orchestra.run.smithi033.stdout:firewalld.service is not enabled 2024-03-19T02:42:51.950 INFO:teuthology.orchestra.run.smithi033.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-19T02:42:51.950 INFO:teuthology.orchestra.run.smithi033.stdout:Waiting for mgr to start... 2024-03-19T02:42:51.950 INFO:teuthology.orchestra.run.smithi033.stdout:Waiting for mgr... 2024-03-19T02:42:52.235 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:42:51 smithi033 systemd[1]: Started Ceph mgr.x for 2b7a8938-e59a-11ee-95c9-87774f69a715. 2024-03-19T02:42:52.518 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:52 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/185452579' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-19T02:42:52.519 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:42:52 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:42:52.418+0000 7f7e0097f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T02:42:52.519 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:42:52 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:42:52.517+0000 7f7e0097f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T02:42:53.528 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:42:53 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:42:53.182+0000 7f7e0097f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T02:42:53.528 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:42:53 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:42:53.277+0000 7f7e0097f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T02:42:53.528 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:42:53 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:42:53.420+0000 7f7e0097f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T02:42:53.871 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:42:53 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:42:53.518+0000 7f7e0097f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T02:42:53.871 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:42:53 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:42:53.602+0000 7f7e0097f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T02:42:53.871 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:42:53 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:42:53.871+0000 7f7e0097f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-19T02:42:54.168 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:42:54 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:42:53.983+0000 7f7e0097f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T02:42:54.168 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:42:54 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:42:54.071+0000 7f7e0097f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T02:42:54.168 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:42:54 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:42:54.168+0000 7f7e0097f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T02:42:54.528 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:42:54 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:42:54.346+0000 7f7e0097f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T02:42:55.778 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:42:55 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:42:55.350+0000 7f7e0097f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-19T02:42:55.778 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:42:55 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:42:55.484+0000 7f7e0097f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T02:42:56.114 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:42:55 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:42:55.822+0000 7f7e0097f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T02:42:56.115 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:42:55 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:42:55.913+0000 7f7e0097f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T02:42:56.115 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:42:56 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:42:56.114+0000 7f7e0097f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:42:56.498 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:42:56 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:42:56.302+0000 7f7e0097f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T02:42:56.501 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-03-19T02:42:56.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout { 2024-03-19T02:42:56.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "fsid": "2b7a8938-e59a-11ee-95c9-87774f69a715", 2024-03-19T02:42:56.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "health": { 2024-03-19T02:42:56.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-19T02:42:56.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-19T02:42:56.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-19T02:42:56.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:42:56.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-19T02:42:56.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-19T02:42:56.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 0 2024-03-19T02:42:56.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ], 2024-03-19T02:42:56.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-19T02:42:56.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "a" 2024-03-19T02:42:56.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ], 2024-03-19T02:42:56.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-19T02:42:56.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-19T02:42:56.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T02:42:56.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-19T02:42:56.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-19T02:42:56.513 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:42:56.513 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-19T02:42:56.513 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T02:42:56.513 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-19T02:42:56.513 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-19T02:42:56.513 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-19T02:42:56.513 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-19T02:42:56.513 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-19T02:42:56.513 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-19T02:42:56.513 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:42:56.513 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-19T02:42:56.513 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-19T02:42:56.513 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-19T02:42:56.513 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-19T02:42:56.514 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-19T02:42:56.514 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-19T02:42:56.514 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-19T02:42:56.514 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-19T02:42:56.514 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-19T02:42:56.514 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:42:56.514 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-19T02:42:56.514 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T02:42:56.514 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-19T02:42:56.514 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-19T02:42:56.514 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:42:56.514 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-19T02:42:56.514 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-19T02:42:56.514 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-19T02:42:56.515 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-19T02:42:56.515 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-19T02:42:56.515 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-19T02:42:56.515 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "restful" 2024-03-19T02:42:56.515 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ], 2024-03-19T02:42:56.515 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T02:42:56.515 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:42:56.515 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-19T02:42:56.515 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T02:42:56.515 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "modified": "2024-03-19T02:42:37.912664+0000", 2024-03-19T02:42:56.515 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T02:42:56.515 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:42:56.515 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-19T02:42:56.515 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout } 2024-03-19T02:42:56.516 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.335+0000 7fa126a15700 1 Processor -- start 2024-03-19T02:42:56.516 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.335+0000 7fa126a15700 1 -- start start 2024-03-19T02:42:56.516 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.335+0000 7fa126a15700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa120108580 0x7fa12010a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.516 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.335+0000 7fa126a15700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa12007be00 con 0x7fa120108580 2024-03-19T02:42:56.516 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.335+0000 7fa11ffff700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa120108580 0x7fa12010a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.516 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.335+0000 7fa11ffff700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa120108580 0x7fa12010a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33794/0 (socket says 172.21.15.33:33794) 2024-03-19T02:42:56.516 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.335+0000 7fa11ffff700 1 -- 172.21.15.33:0/2612845317 learned_addr learned my addr 172.21.15.33:0/2612845317 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:42:56.516 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.336+0000 7fa11ffff700 1 -- 172.21.15.33:0/2612845317 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa12010aeb0 con 0x7fa120108580 2024-03-19T02:42:56.516 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.336+0000 7fa11ffff700 1 --2- 172.21.15.33:0/2612845317 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa120108580 0x7fa12010a970 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa1100098a0 tx=0x7fa110009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9c9cdaa389722ac server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:56.516 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.336+0000 7fa11effd700 1 -- 172.21.15.33:0/2612845317 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa110004160 con 0x7fa120108580 2024-03-19T02:42:56.516 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.336+0000 7fa11effd700 1 -- 172.21.15.33:0/2612845317 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fa110004300 con 0x7fa120108580 2024-03-19T02:42:56.516 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.336+0000 7fa11effd700 1 -- 172.21.15.33:0/2612845317 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa110011a80 con 0x7fa120108580 2024-03-19T02:42:56.516 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.337+0000 7fa126a15700 1 -- 172.21.15.33:0/2612845317 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa120108580 msgr2=0x7fa12010a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.516 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.337+0000 7fa126a15700 1 --2- 172.21.15.33:0/2612845317 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa120108580 0x7fa12010a970 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa1100098a0 tx=0x7fa110009bb0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.517 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.337+0000 7fa126a15700 1 -- 172.21.15.33:0/2612845317 shutdown_connections 2024-03-19T02:42:56.517 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.337+0000 7fa126a15700 1 --2- 172.21.15.33:0/2612845317 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa120108580 0x7fa12010a970 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.517 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.337+0000 7fa126a15700 1 -- 172.21.15.33:0/2612845317 >> 172.21.15.33:0/2612845317 conn(0x7fa120101f30 msgr2=0x7fa120104370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.517 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.337+0000 7fa126a15700 1 -- 172.21.15.33:0/2612845317 shutdown_connections 2024-03-19T02:42:56.517 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.337+0000 7fa126a15700 1 -- 172.21.15.33:0/2612845317 wait complete. 2024-03-19T02:42:56.517 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.337+0000 7fa126a15700 1 Processor -- start 2024-03-19T02:42:56.517 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.338+0000 7fa126a15700 1 -- start start 2024-03-19T02:42:56.517 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.338+0000 7fa126a15700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa120108580 0x7fa12019b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:56.517 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.338+0000 7fa126a15700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa12010be10 con 0x7fa120108580 2024-03-19T02:42:56.517 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.338+0000 7fa11ffff700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa120108580 0x7fa12019b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:56.517 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.338+0000 7fa11ffff700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa120108580 0x7fa12019b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33808/0 (socket says 172.21.15.33:33808) 2024-03-19T02:42:56.517 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.338+0000 7fa11ffff700 1 -- 172.21.15.33:0/185452579 learned_addr learned my addr 172.21.15.33:0/185452579 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:42:56.517 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.338+0000 7fa11ffff700 1 -- 172.21.15.33:0/185452579 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa12019be10 con 0x7fa120108580 2024-03-19T02:42:56.517 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.338+0000 7fa11ffff700 1 --2- 172.21.15.33:0/185452579 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa120108580 0x7fa12019b8d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa110006af0 tx=0x7fa110011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:56.518 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.339+0000 7fa11d7fa700 1 -- 172.21.15.33:0/185452579 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa1100039e0 con 0x7fa120108580 2024-03-19T02:42:56.518 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.339+0000 7fa11d7fa700 1 -- 172.21.15.33:0/185452579 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fa110003b80 con 0x7fa120108580 2024-03-19T02:42:56.518 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.339+0000 7fa126a15700 1 -- 172.21.15.33:0/185452579 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa12019c0a0 con 0x7fa120108580 2024-03-19T02:42:56.518 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.339+0000 7fa11d7fa700 1 -- 172.21.15.33:0/185452579 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa1100196a0 con 0x7fa120108580 2024-03-19T02:42:56.518 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.339+0000 7fa126a15700 1 -- 172.21.15.33:0/185452579 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa12019ed90 con 0x7fa120108580 2024-03-19T02:42:56.518 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.339+0000 7fa11d7fa700 1 -- 172.21.15.33:0/185452579 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa110019ce0 con 0x7fa120108580 2024-03-19T02:42:56.518 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.339+0000 7fa11d7fa700 1 -- 172.21.15.33:0/185452579 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa110024730 con 0x7fa120108580 2024-03-19T02:42:56.518 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.340+0000 7fa126a15700 1 -- 172.21.15.33:0/185452579 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa120063940 con 0x7fa120108580 2024-03-19T02:42:56.518 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.344+0000 7fa11d7fa700 1 -- 172.21.15.33:0/185452579 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fa110043dd0 con 0x7fa120108580 2024-03-19T02:42:56.518 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.417+0000 7fa126a15700 1 -- 172.21.15.33:0/185452579 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fa12019ef60 con 0x7fa120108580 2024-03-19T02:42:56.518 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.418+0000 7fa11d7fa700 1 -- 172.21.15.33:0/185452579 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fa110032020 con 0x7fa120108580 2024-03-19T02:42:56.518 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.419+0000 7fa126a15700 1 -- 172.21.15.33:0/185452579 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa120108580 msgr2=0x7fa12019b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:56.518 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.419+0000 7fa126a15700 1 --2- 172.21.15.33:0/185452579 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa120108580 0x7fa12019b8d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa110006af0 tx=0x7fa110011c40 comp rx=0 tx=0).stop 2024-03-19T02:42:56.519 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.419+0000 7fa126a15700 1 -- 172.21.15.33:0/185452579 shutdown_connections 2024-03-19T02:42:56.519 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.419+0000 7fa126a15700 1 --2- 172.21.15.33:0/185452579 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa120108580 0x7fa12019b8d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:56.519 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.419+0000 7fa126a15700 1 -- 172.21.15.33:0/185452579 >> 172.21.15.33:0/185452579 conn(0x7fa120101f30 msgr2=0x7fa120102980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:56.519 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.419+0000 7fa126a15700 1 -- 172.21.15.33:0/185452579 shutdown_connections 2024-03-19T02:42:56.519 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:52.419+0000 7fa126a15700 1 -- 172.21.15.33:0/185452579 wait complete. 2024-03-19T02:42:56.519 INFO:teuthology.orchestra.run.smithi033.stdout:mgr not available, waiting (1/15)... 2024-03-19T02:42:56.778 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:42:56 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:42:56.592+0000 7f7e0097f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:42:57.278 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:42:56 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:42:56.853+0000 7f7e0097f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T02:42:57.278 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:42:57 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:42:57.096+0000 7f7e0097f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T02:42:58.278 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:42:57 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:42:57.938+0000 7f7e0097f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T02:42:58.778 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:42:58 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:42:58.524+0000 7f7e0097f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T02:42:58.779 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:42:58 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:42:58.625+0000 7f7e0097f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T02:42:59.278 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:42:58 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:42:58.849+0000 7f7e0097f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T02:42:59.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:58 smithi033 ceph-mon[31973]: Activating manager daemon x 2024-03-19T02:42:59.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:58 smithi033 ceph-mon[31973]: mgrmap e2: x(active, starting, since 0.00324389s) 2024-03-19T02:42:59.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:58 smithi033 ceph-mon[31973]: from='mgr.14100 172.21.15.33:0/1380968957' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-19T02:42:59.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:58 smithi033 ceph-mon[31973]: from='mgr.14100 172.21.15.33:0/1380968957' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-19T02:42:59.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:58 smithi033 ceph-mon[31973]: from='mgr.14100 172.21.15.33:0/1380968957' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-19T02:42:59.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:58 smithi033 ceph-mon[31973]: from='mgr.14100 172.21.15.33:0/1380968957' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:42:59.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:58 smithi033 ceph-mon[31973]: from='mgr.14100 172.21.15.33:0/1380968957' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-19T02:42:59.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:58 smithi033 ceph-mon[31973]: Manager daemon x is now available 2024-03-19T02:42:59.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:58 smithi033 ceph-mon[31973]: from='mgr.14100 172.21.15.33:0/1380968957' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-19T02:42:59.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:58 smithi033 ceph-mon[31973]: from='mgr.14100 172.21.15.33:0/1380968957' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-19T02:42:59.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:58 smithi033 ceph-mon[31973]: from='mgr.14100 172.21.15.33:0/1380968957' entity='mgr.x' 2024-03-19T02:42:59.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:58 smithi033 ceph-mon[31973]: from='mgr.14100 172.21.15.33:0/1380968957' entity='mgr.x' 2024-03-19T02:42:59.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:58 smithi033 ceph-mon[31973]: from='mgr.14100 172.21.15.33:0/1380968957' entity='mgr.x' 2024-03-19T02:42:59.616 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-03-19T02:42:59.626 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout { 2024-03-19T02:42:59.626 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "fsid": "2b7a8938-e59a-11ee-95c9-87774f69a715", 2024-03-19T02:42:59.626 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "health": { 2024-03-19T02:42:59.627 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-19T02:42:59.627 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-19T02:42:59.627 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-19T02:42:59.627 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:42:59.627 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-19T02:42:59.627 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-19T02:42:59.627 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 0 2024-03-19T02:42:59.627 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ], 2024-03-19T02:42:59.627 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-19T02:42:59.627 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "a" 2024-03-19T02:42:59.627 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ], 2024-03-19T02:42:59.627 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-03-19T02:42:59.627 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-19T02:42:59.627 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T02:42:59.627 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-19T02:42:59.628 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-19T02:42:59.628 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:42:59.628 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-19T02:42:59.628 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T02:42:59.628 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-19T02:42:59.628 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-19T02:42:59.628 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-19T02:42:59.628 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-19T02:42:59.628 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-19T02:42:59.628 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-19T02:42:59.628 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:42:59.628 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-19T02:42:59.628 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-19T02:42:59.628 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-19T02:42:59.629 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-19T02:42:59.629 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-19T02:42:59.629 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-19T02:42:59.629 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-19T02:42:59.629 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-19T02:42:59.629 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-19T02:42:59.629 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:42:59.629 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-19T02:42:59.629 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T02:42:59.629 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-19T02:42:59.629 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-19T02:42:59.629 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:42:59.629 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-19T02:42:59.629 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-19T02:42:59.629 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-19T02:42:59.629 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-19T02:42:59.630 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-19T02:42:59.630 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-19T02:42:59.630 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "restful" 2024-03-19T02:42:59.630 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ], 2024-03-19T02:42:59.630 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T02:42:59.630 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:42:59.630 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-19T02:42:59.630 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T02:42:59.630 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "modified": "2024-03-19T02:42:37.912664+0000", 2024-03-19T02:42:59.630 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T02:42:59.630 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:42:59.630 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-19T02:42:59.630 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout } 2024-03-19T02:42:59.630 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.946+0000 7facfffbf700 1 Processor -- start 2024-03-19T02:42:59.631 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.946+0000 7facfffbf700 1 -- start start 2024-03-19T02:42:59.631 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.946+0000 7facfffbf700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facf8108590 0x7facf810a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:59.631 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.946+0000 7facfffbf700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7facf807be20 con 0x7facf8108590 2024-03-19T02:42:59.631 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.946+0000 7facfdd5b700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facf8108590 0x7facf810a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:59.631 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.946+0000 7facfdd5b700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facf8108590 0x7facf810a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51854/0 (socket says 172.21.15.33:51854) 2024-03-19T02:42:59.631 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.946+0000 7facfdd5b700 1 -- 172.21.15.33:0/483415700 learned_addr learned my addr 172.21.15.33:0/483415700 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:42:59.631 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.947+0000 7facfdd5b700 1 -- 172.21.15.33:0/483415700 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf810aec0 con 0x7facf8108590 2024-03-19T02:42:59.631 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.947+0000 7facfdd5b700 1 --2- 172.21.15.33:0/483415700 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facf8108590 0x7facf810a980 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7facf40098a0 tx=0x7facf4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=473b01bb48f280b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:59.631 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.947+0000 7facfcd59700 1 -- 172.21.15.33:0/483415700 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7facf4004160 con 0x7facf8108590 2024-03-19T02:42:59.631 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.947+0000 7facfcd59700 1 -- 172.21.15.33:0/483415700 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7facf4004300 con 0x7facf8108590 2024-03-19T02:42:59.631 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.947+0000 7facfcd59700 1 -- 172.21.15.33:0/483415700 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7facf4011a80 con 0x7facf8108590 2024-03-19T02:42:59.631 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.948+0000 7facfffbf700 1 -- 172.21.15.33:0/483415700 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facf8108590 msgr2=0x7facf810a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:59.631 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.948+0000 7facfffbf700 1 --2- 172.21.15.33:0/483415700 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facf8108590 0x7facf810a980 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7facf40098a0 tx=0x7facf4009bb0 comp rx=0 tx=0).stop 2024-03-19T02:42:59.631 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.948+0000 7facfffbf700 1 -- 172.21.15.33:0/483415700 shutdown_connections 2024-03-19T02:42:59.632 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.948+0000 7facfffbf700 1 --2- 172.21.15.33:0/483415700 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facf8108590 0x7facf810a980 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:59.632 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.948+0000 7facfffbf700 1 -- 172.21.15.33:0/483415700 >> 172.21.15.33:0/483415700 conn(0x7facf8101f60 msgr2=0x7facf81043a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:59.632 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.948+0000 7facfffbf700 1 -- 172.21.15.33:0/483415700 shutdown_connections 2024-03-19T02:42:59.632 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.948+0000 7facfffbf700 1 -- 172.21.15.33:0/483415700 wait complete. 2024-03-19T02:42:59.632 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.948+0000 7facfffbf700 1 Processor -- start 2024-03-19T02:42:59.632 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.948+0000 7facfffbf700 1 -- start start 2024-03-19T02:42:59.632 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.949+0000 7facfffbf700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facf8108590 0x7facf819b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:42:59.632 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.949+0000 7facfffbf700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7facf810be30 con 0x7facf8108590 2024-03-19T02:42:59.632 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.949+0000 7facfdd5b700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facf8108590 0x7facf819b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:42:59.632 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.949+0000 7facfdd5b700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facf8108590 0x7facf819b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51860/0 (socket says 172.21.15.33:51860) 2024-03-19T02:42:59.632 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.949+0000 7facfdd5b700 1 -- 172.21.15.33:0/422161576 learned_addr learned my addr 172.21.15.33:0/422161576 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:42:59.632 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.949+0000 7facfdd5b700 1 -- 172.21.15.33:0/422161576 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf819be80 con 0x7facf8108590 2024-03-19T02:42:59.632 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.949+0000 7facfdd5b700 1 --2- 172.21.15.33:0/422161576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facf8108590 0x7facf819b940 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7facf4011a50 tx=0x7facf4004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:42:59.632 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.949+0000 7faceeffd700 1 -- 172.21.15.33:0/422161576 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7facf40039f0 con 0x7facf8108590 2024-03-19T02:42:59.633 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.949+0000 7facfffbf700 1 -- 172.21.15.33:0/422161576 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facf819c110 con 0x7facf8108590 2024-03-19T02:42:59.633 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.949+0000 7faceeffd700 1 -- 172.21.15.33:0/422161576 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7facf4003b90 con 0x7facf8108590 2024-03-19T02:42:59.633 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.949+0000 7faceeffd700 1 -- 172.21.15.33:0/422161576 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7facf401a600 con 0x7facf8108590 2024-03-19T02:42:59.633 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.949+0000 7facfffbf700 1 -- 172.21.15.33:0/422161576 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facf819ee00 con 0x7facf8108590 2024-03-19T02:42:59.633 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.950+0000 7faceeffd700 1 -- 172.21.15.33:0/422161576 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 2) ==== 45862+0+0 (secure 0 0 0) 0x7facf4024a40 con 0x7facf8108590 2024-03-19T02:42:59.633 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.950+0000 7facfffbf700 1 -- 172.21.15.33:0/422161576 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facf8063940 con 0x7facf8108590 2024-03-19T02:42:59.633 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.950+0000 7faceeffd700 1 -- 172.21.15.33:0/422161576 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7facf404cec0 con 0x7facf8108590 2024-03-19T02:42:59.633 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:58.953+0000 7faceeffd700 1 -- 172.21.15.33:0/422161576 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7facf4024440 con 0x7facf8108590 2024-03-19T02:42:59.633 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:59.026+0000 7facfffbf700 1 -- 172.21.15.33:0/422161576 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7facf819efd0 con 0x7facf8108590 2024-03-19T02:42:59.633 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:59.026+0000 7faceeffd700 1 -- 172.21.15.33:0/422161576 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7facf819efd0 con 0x7facf8108590 2024-03-19T02:42:59.633 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:59.027+0000 7facfffbf700 1 -- 172.21.15.33:0/422161576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facf8108590 msgr2=0x7facf819b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:42:59.633 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:59.027+0000 7facfffbf700 1 --2- 172.21.15.33:0/422161576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facf8108590 0x7facf819b940 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7facf4011a50 tx=0x7facf4004290 comp rx=0 tx=0).stop 2024-03-19T02:42:59.633 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:59.027+0000 7facfffbf700 1 -- 172.21.15.33:0/422161576 shutdown_connections 2024-03-19T02:42:59.633 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:59.027+0000 7facfffbf700 1 --2- 172.21.15.33:0/422161576 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7facf8108590 0x7facf819b940 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:42:59.634 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:59.027+0000 7facfffbf700 1 -- 172.21.15.33:0/422161576 >> 172.21.15.33:0/422161576 conn(0x7facf8101f60 msgr2=0x7facf81029b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:42:59.634 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:59.027+0000 7facfffbf700 1 -- 172.21.15.33:0/422161576 shutdown_connections 2024-03-19T02:42:59.634 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:42:59.027+0000 7facfffbf700 1 -- 172.21.15.33:0/422161576 wait complete. 2024-03-19T02:42:59.634 INFO:teuthology.orchestra.run.smithi033.stdout:mgr not available, waiting (2/15)... 2024-03-19T02:43:00.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:59 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/422161576' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-19T02:43:00.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:42:59 smithi033 ceph-mon[31973]: mgrmap e3: x(active, since 1.00713s) 2024-03-19T02:43:02.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:01 smithi033 ceph-mon[31973]: mgrmap e4: x(active, since 2s) 2024-03-19T02:43:02.891 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-03-19T02:43:02.891 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout { 2024-03-19T02:43:02.892 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "fsid": "2b7a8938-e59a-11ee-95c9-87774f69a715", 2024-03-19T02:43:02.892 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "health": { 2024-03-19T02:43:02.892 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-19T02:43:02.892 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-19T02:43:02.892 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-19T02:43:02.892 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:43:02.892 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-19T02:43:02.892 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-19T02:43:02.892 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 0 2024-03-19T02:43:02.892 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ], 2024-03-19T02:43:02.892 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-19T02:43:02.892 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "a" 2024-03-19T02:43:02.892 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ], 2024-03-19T02:43:02.892 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-19T02:43:02.892 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-19T02:43:02.893 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T02:43:02.893 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-19T02:43:02.893 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-19T02:43:02.893 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:43:02.893 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-19T02:43:02.893 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T02:43:02.893 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-19T02:43:02.893 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-19T02:43:02.893 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-19T02:43:02.901 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-19T02:43:02.901 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-19T02:43:02.901 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-19T02:43:02.901 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:43:02.902 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-19T02:43:02.902 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-19T02:43:02.902 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-19T02:43:02.902 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-19T02:43:02.902 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-19T02:43:02.902 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-19T02:43:02.902 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-19T02:43:02.902 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-19T02:43:02.902 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-19T02:43:02.902 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:43:02.902 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-19T02:43:02.902 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T02:43:02.902 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-19T02:43:02.902 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-19T02:43:02.903 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:43:02.903 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-19T02:43:02.903 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-19T02:43:02.903 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-19T02:43:02.903 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-19T02:43:02.903 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-19T02:43:02.903 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-19T02:43:02.903 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "restful" 2024-03-19T02:43:02.903 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ], 2024-03-19T02:43:02.903 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T02:43:02.903 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:43:02.903 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-19T02:43:02.903 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T02:43:02.903 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "modified": "2024-03-19T02:42:37.912664+0000", 2024-03-19T02:43:02.903 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T02:43:02.904 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout }, 2024-03-19T02:43:02.904 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-19T02:43:02.904 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout } 2024-03-19T02:43:02.904 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.054+0000 7f6603c5e700 1 Processor -- start 2024-03-19T02:43:02.904 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.054+0000 7f6603c5e700 1 -- start start 2024-03-19T02:43:02.904 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.054+0000 7f6603c5e700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65fc107a60 0x7f65fc107e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:02.904 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.054+0000 7f6603c5e700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f65fc108450 con 0x7f65fc107a60 2024-03-19T02:43:02.904 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.054+0000 7f66019fa700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65fc107a60 0x7f65fc107e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:02.904 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.054+0000 7f66019fa700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65fc107a60 0x7f65fc107e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51876/0 (socket says 172.21.15.33:51876) 2024-03-19T02:43:02.904 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.054+0000 7f66019fa700 1 -- 172.21.15.33:0/4211125831 learned_addr learned my addr 172.21.15.33:0/4211125831 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:02.904 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.055+0000 7f66019fa700 1 -- 172.21.15.33:0/4211125831 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65fc108c60 con 0x7f65fc107a60 2024-03-19T02:43:02.904 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.055+0000 7f66019fa700 1 --2- 172.21.15.33:0/4211125831 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65fc107a60 0x7f65fc107e80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f65f80098a0 tx=0x7f65f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a0463afa449f858 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:02.904 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.055+0000 7f66009f8700 1 -- 172.21.15.33:0/4211125831 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65f8004160 con 0x7f65fc107a60 2024-03-19T02:43:02.904 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.055+0000 7f66009f8700 1 -- 172.21.15.33:0/4211125831 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f65f8004300 con 0x7f65fc107a60 2024-03-19T02:43:02.904 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.055+0000 7f66009f8700 1 -- 172.21.15.33:0/4211125831 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65f8011a80 con 0x7f65fc107a60 2024-03-19T02:43:02.905 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.056+0000 7f6603c5e700 1 -- 172.21.15.33:0/4211125831 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65fc107a60 msgr2=0x7f65fc107e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:02.905 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.056+0000 7f6603c5e700 1 --2- 172.21.15.33:0/4211125831 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65fc107a60 0x7f65fc107e80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f65f80098a0 tx=0x7f65f8009bb0 comp rx=0 tx=0).stop 2024-03-19T02:43:02.905 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.056+0000 7f6603c5e700 1 -- 172.21.15.33:0/4211125831 shutdown_connections 2024-03-19T02:43:02.905 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.056+0000 7f6603c5e700 1 --2- 172.21.15.33:0/4211125831 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65fc107a60 0x7f65fc107e80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:02.905 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.056+0000 7f6603c5e700 1 -- 172.21.15.33:0/4211125831 >> 172.21.15.33:0/4211125831 conn(0x7f65fc103000 msgr2=0x7f65fc105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:02.905 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.056+0000 7f6603c5e700 1 -- 172.21.15.33:0/4211125831 shutdown_connections 2024-03-19T02:43:02.905 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.056+0000 7f6603c5e700 1 -- 172.21.15.33:0/4211125831 wait complete. 2024-03-19T02:43:02.905 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.056+0000 7f6603c5e700 1 Processor -- start 2024-03-19T02:43:02.905 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.056+0000 7f6603c5e700 1 -- start start 2024-03-19T02:43:02.905 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.056+0000 7f6603c5e700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65fc107a60 0x7f65fc07db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:02.905 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.056+0000 7f6603c5e700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f65fc10c010 con 0x7f65fc107a60 2024-03-19T02:43:02.905 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.057+0000 7f66019fa700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65fc107a60 0x7f65fc07db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:02.905 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.057+0000 7f66019fa700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65fc107a60 0x7f65fc07db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51888/0 (socket says 172.21.15.33:51888) 2024-03-19T02:43:02.905 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.057+0000 7f66019fa700 1 -- 172.21.15.33:0/530207038 learned_addr learned my addr 172.21.15.33:0/530207038 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:02.905 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.057+0000 7f66019fa700 1 -- 172.21.15.33:0/530207038 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65fc07e080 con 0x7f65fc107a60 2024-03-19T02:43:02.906 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.057+0000 7f66019fa700 1 --2- 172.21.15.33:0/530207038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65fc107a60 0x7f65fc07db40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f65f8006af0 tx=0x7f65f8011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:02.906 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.057+0000 7f65f2ffd700 1 -- 172.21.15.33:0/530207038 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65f8003930 con 0x7f65fc107a60 2024-03-19T02:43:02.906 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.057+0000 7f65f2ffd700 1 -- 172.21.15.33:0/530207038 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f65f8003ad0 con 0x7f65fc107a60 2024-03-19T02:43:02.906 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.057+0000 7f65f2ffd700 1 -- 172.21.15.33:0/530207038 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65f80195e0 con 0x7f65fc107a60 2024-03-19T02:43:02.906 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.057+0000 7f6603c5e700 1 -- 172.21.15.33:0/530207038 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65fc07c270 con 0x7f65fc107a60 2024-03-19T02:43:02.906 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.057+0000 7f6603c5e700 1 -- 172.21.15.33:0/530207038 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65fc07c7b0 con 0x7f65fc107a60 2024-03-19T02:43:02.906 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.058+0000 7f65f2ffd700 1 -- 172.21.15.33:0/530207038 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7f65f8019890 con 0x7f65fc107a60 2024-03-19T02:43:02.906 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.058+0000 7f6603c5e700 1 -- 172.21.15.33:0/530207038 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65fc063940 con 0x7f65fc107a60 2024-03-19T02:43:02.906 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.058+0000 7f65f2ffd700 1 --2- 172.21.15.33:0/530207038 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7f65e8038f90 0x7f65e803b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:02.906 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.058+0000 7f65f2ffd700 1 -- 172.21.15.33:0/530207038 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f65f804d460 con 0x7f65fc107a60 2024-03-19T02:43:02.906 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.058+0000 7f66011f9700 1 --2- 172.21.15.33:0/530207038 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7f65e8038f90 0x7f65e803b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:02.906 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.059+0000 7f66011f9700 1 --2- 172.21.15.33:0/530207038 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7f65e8038f90 0x7f65e803b450 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f65ec006fd0 tx=0x7f65ec006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:02.906 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.064+0000 7f65f2ffd700 1 -- 172.21.15.33:0/530207038 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f65f804ce80 con 0x7f65fc107a60 2024-03-19T02:43:02.907 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.350+0000 7f6603c5e700 1 -- 172.21.15.33:0/530207038 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f65fc07ca30 con 0x7f65fc107a60 2024-03-19T02:43:02.907 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.350+0000 7f65f2ffd700 1 -- 172.21.15.33:0/530207038 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f65f802d9c0 con 0x7f65fc107a60 2024-03-19T02:43:02.907 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.353+0000 7f6603c5e700 1 -- 172.21.15.33:0/530207038 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7f65e8038f90 msgr2=0x7f65e803b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:02.907 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.353+0000 7f6603c5e700 1 --2- 172.21.15.33:0/530207038 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7f65e8038f90 0x7f65e803b450 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f65ec006fd0 tx=0x7f65ec006e40 comp rx=0 tx=0).stop 2024-03-19T02:43:02.907 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.353+0000 7f6603c5e700 1 -- 172.21.15.33:0/530207038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65fc107a60 msgr2=0x7f65fc07db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:02.907 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.353+0000 7f6603c5e700 1 --2- 172.21.15.33:0/530207038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65fc107a60 0x7f65fc07db40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f65f8006af0 tx=0x7f65f8011b70 comp rx=0 tx=0).stop 2024-03-19T02:43:02.907 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.353+0000 7f6603c5e700 1 -- 172.21.15.33:0/530207038 shutdown_connections 2024-03-19T02:43:02.907 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.353+0000 7f6603c5e700 1 --2- 172.21.15.33:0/530207038 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7f65e8038f90 0x7f65e803b450 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:02.907 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.353+0000 7f6603c5e700 1 --2- 172.21.15.33:0/530207038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f65fc107a60 0x7f65fc07db40 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:02.907 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.353+0000 7f6603c5e700 1 -- 172.21.15.33:0/530207038 >> 172.21.15.33:0/530207038 conn(0x7f65fc103000 msgr2=0x7f65fc103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:02.907 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.353+0000 7f6603c5e700 1 -- 172.21.15.33:0/530207038 shutdown_connections 2024-03-19T02:43:02.907 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:02.353+0000 7f6603c5e700 1 -- 172.21.15.33:0/530207038 wait complete. 2024-03-19T02:43:02.907 INFO:teuthology.orchestra.run.smithi033.stdout:mgr is available 2024-03-19T02:43:03.012 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:02 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/530207038' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-19T02:43:04.069 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:03 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1552795990' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-19T02:43:04.078 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-03-19T02:43:04.078 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout [global] 2024-03-19T02:43:04.078 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout fsid = 2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T02:43:04.078 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-19T02:43:04.078 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-19T02:43:04.078 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-19T02:43:04.078 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-19T02:43:04.078 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-03-19T02:43:04.078 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-19T02:43:04.079 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-19T02:43:04.079 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 2024-03-19T02:43:04.079 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout [osd] 2024-03-19T02:43:04.079 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-19T02:43:04.079 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-19T02:43:04.079 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-19T02:43:04.079 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-19T02:43:04.079 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-19T02:43:04.079 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.289+0000 7efccb2ef700 1 Processor -- start 2024-03-19T02:43:04.079 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.289+0000 7efccb2ef700 1 -- start start 2024-03-19T02:43:04.079 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.289+0000 7efccb2ef700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcc4107aa0 0x7efcc4107ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:04.079 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.289+0000 7efccb2ef700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efcc4108490 con 0x7efcc4107aa0 2024-03-19T02:43:04.079 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.289+0000 7efcc908b700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcc4107aa0 0x7efcc4107ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:04.079 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.289+0000 7efcc908b700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcc4107aa0 0x7efcc4107ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51894/0 (socket says 172.21.15.33:51894) 2024-03-19T02:43:04.079 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.289+0000 7efcc908b700 1 -- 172.21.15.33:0/913028457 learned_addr learned my addr 172.21.15.33:0/913028457 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:04.080 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.290+0000 7efcc908b700 1 -- 172.21.15.33:0/913028457 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcc4108ca0 con 0x7efcc4107aa0 2024-03-19T02:43:04.080 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.290+0000 7efcc908b700 1 --2- 172.21.15.33:0/913028457 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcc4107aa0 0x7efcc4107ec0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7efcc00098a0 tx=0x7efcc0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9419c532ff5e7d33 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:04.080 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.290+0000 7efcbbfff700 1 -- 172.21.15.33:0/913028457 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcc0004160 con 0x7efcc4107aa0 2024-03-19T02:43:04.080 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.290+0000 7efcbbfff700 1 -- 172.21.15.33:0/913028457 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7efcc0004300 con 0x7efcc4107aa0 2024-03-19T02:43:04.080 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.290+0000 7efcbbfff700 1 -- 172.21.15.33:0/913028457 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcc0011a80 con 0x7efcc4107aa0 2024-03-19T02:43:04.080 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.290+0000 7efccb2ef700 1 -- 172.21.15.33:0/913028457 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcc4107aa0 msgr2=0x7efcc4107ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:04.080 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.290+0000 7efccb2ef700 1 --2- 172.21.15.33:0/913028457 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcc4107aa0 0x7efcc4107ec0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7efcc00098a0 tx=0x7efcc0009bb0 comp rx=0 tx=0).stop 2024-03-19T02:43:04.080 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.291+0000 7efccb2ef700 1 -- 172.21.15.33:0/913028457 shutdown_connections 2024-03-19T02:43:04.080 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.291+0000 7efccb2ef700 1 --2- 172.21.15.33:0/913028457 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcc4107aa0 0x7efcc4107ec0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:04.080 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.291+0000 7efccb2ef700 1 -- 172.21.15.33:0/913028457 >> 172.21.15.33:0/913028457 conn(0x7efcc4103020 msgr2=0x7efcc4105480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:04.080 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.291+0000 7efccb2ef700 1 -- 172.21.15.33:0/913028457 shutdown_connections 2024-03-19T02:43:04.080 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.291+0000 7efccb2ef700 1 -- 172.21.15.33:0/913028457 wait complete. 2024-03-19T02:43:04.080 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.291+0000 7efccb2ef700 1 Processor -- start 2024-03-19T02:43:04.080 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.291+0000 7efccb2ef700 1 -- start start 2024-03-19T02:43:04.080 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.291+0000 7efccb2ef700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcc4107aa0 0x7efcc419b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:04.081 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.291+0000 7efccb2ef700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efcc410c050 con 0x7efcc4107aa0 2024-03-19T02:43:04.081 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.292+0000 7efcc908b700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcc4107aa0 0x7efcc419b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:04.081 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.292+0000 7efcc908b700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcc4107aa0 0x7efcc419b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51910/0 (socket says 172.21.15.33:51910) 2024-03-19T02:43:04.081 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.292+0000 7efcc908b700 1 -- 172.21.15.33:0/1552795990 learned_addr learned my addr 172.21.15.33:0/1552795990 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:04.081 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.292+0000 7efcc908b700 1 -- 172.21.15.33:0/1552795990 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcc419be50 con 0x7efcc4107aa0 2024-03-19T02:43:04.081 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.292+0000 7efcc908b700 1 --2- 172.21.15.33:0/1552795990 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcc4107aa0 0x7efcc419b910 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efcc0006af0 tx=0x7efcc0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:04.081 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.292+0000 7efcba7fc700 1 -- 172.21.15.33:0/1552795990 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcc00039e0 con 0x7efcc4107aa0 2024-03-19T02:43:04.081 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.292+0000 7efcba7fc700 1 -- 172.21.15.33:0/1552795990 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7efcc0003b80 con 0x7efcc4107aa0 2024-03-19T02:43:04.081 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.292+0000 7efccb2ef700 1 -- 172.21.15.33:0/1552795990 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcc419c0e0 con 0x7efcc4107aa0 2024-03-19T02:43:04.081 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.292+0000 7efcba7fc700 1 -- 172.21.15.33:0/1552795990 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcc00196a0 con 0x7efcc4107aa0 2024-03-19T02:43:04.081 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.292+0000 7efccb2ef700 1 -- 172.21.15.33:0/1552795990 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcc419edd0 con 0x7efcc4107aa0 2024-03-19T02:43:04.081 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.293+0000 7efcba7fc700 1 -- 172.21.15.33:0/1552795990 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7efcc0019950 con 0x7efcc4107aa0 2024-03-19T02:43:04.081 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.293+0000 7efccb2ef700 1 -- 172.21.15.33:0/1552795990 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcc4063940 con 0x7efcc4107aa0 2024-03-19T02:43:04.081 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.293+0000 7efcba7fc700 1 --2- 172.21.15.33:0/1552795990 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7efcb0038f90 0x7efcb003b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:04.082 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.293+0000 7efcba7fc700 1 -- 172.21.15.33:0/1552795990 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7efcc004d280 con 0x7efcc4107aa0 2024-03-19T02:43:04.082 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.294+0000 7efcc888a700 1 --2- 172.21.15.33:0/1552795990 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7efcb0038f90 0x7efcb003b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:04.082 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.294+0000 7efcc888a700 1 --2- 172.21.15.33:0/1552795990 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7efcb0038f90 0x7efcb003b450 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7efcb4006fd0 tx=0x7efcb4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:04.082 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.298+0000 7efcba7fc700 1 -- 172.21.15.33:0/1552795990 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efcc0023e50 con 0x7efcc4107aa0 2024-03-19T02:43:04.082 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.506+0000 7efccb2ef700 1 -- 172.21.15.33:0/1552795990 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7efcc419efa0 con 0x7efcc4107aa0 2024-03-19T02:43:04.082 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.506+0000 7efcba7fc700 1 -- 172.21.15.33:0/1552795990 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7efcc419efa0 con 0x7efcc4107aa0 2024-03-19T02:43:04.082 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.508+0000 7efccb2ef700 1 -- 172.21.15.33:0/1552795990 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7efcb0038f90 msgr2=0x7efcb003b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:04.082 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.508+0000 7efccb2ef700 1 --2- 172.21.15.33:0/1552795990 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7efcb0038f90 0x7efcb003b450 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7efcb4006fd0 tx=0x7efcb4006e40 comp rx=0 tx=0).stop 2024-03-19T02:43:04.082 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.508+0000 7efccb2ef700 1 -- 172.21.15.33:0/1552795990 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcc4107aa0 msgr2=0x7efcc419b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:04.082 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.508+0000 7efccb2ef700 1 --2- 172.21.15.33:0/1552795990 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcc4107aa0 0x7efcc419b910 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efcc0006af0 tx=0x7efcc0011c40 comp rx=0 tx=0).stop 2024-03-19T02:43:04.082 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.509+0000 7efccb2ef700 1 -- 172.21.15.33:0/1552795990 shutdown_connections 2024-03-19T02:43:04.082 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.509+0000 7efccb2ef700 1 --2- 172.21.15.33:0/1552795990 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7efcb0038f90 0x7efcb003b450 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:04.082 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.509+0000 7efccb2ef700 1 --2- 172.21.15.33:0/1552795990 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efcc4107aa0 0x7efcc419b910 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:04.082 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.509+0000 7efccb2ef700 1 -- 172.21.15.33:0/1552795990 >> 172.21.15.33:0/1552795990 conn(0x7efcc4103020 msgr2=0x7efcc4103a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:04.082 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.509+0000 7efccb2ef700 1 -- 172.21.15.33:0/1552795990 shutdown_connections 2024-03-19T02:43:04.083 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:03.509+0000 7efccb2ef700 1 -- 172.21.15.33:0/1552795990 wait complete. 2024-03-19T02:43:04.083 INFO:teuthology.orchestra.run.smithi033.stdout:Enabling cephadm module... 2024-03-19T02:43:05.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:04 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/415908847' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-19T02:43:05.028 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:04 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: ignoring --setuser ceph since I am not root 2024-03-19T02:43:05.028 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:04 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: ignoring --setgroup ceph since I am not root 2024-03-19T02:43:05.509 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.458+0000 7fae0f1b6700 1 Processor -- start 2024-03-19T02:43:05.509 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.458+0000 7fae0f1b6700 1 -- start start 2024-03-19T02:43:05.509 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.458+0000 7fae0f1b6700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae08107a60 0x7fae08107e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:05.509 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.458+0000 7fae0f1b6700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fae08108450 con 0x7fae08107a60 2024-03-19T02:43:05.509 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.459+0000 7fae0cf52700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae08107a60 0x7fae08107e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:05.509 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.459+0000 7fae0cf52700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae08107a60 0x7fae08107e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51982/0 (socket says 172.21.15.33:51982) 2024-03-19T02:43:05.509 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.459+0000 7fae0cf52700 1 -- 172.21.15.33:0/2404569624 learned_addr learned my addr 172.21.15.33:0/2404569624 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:05.509 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.459+0000 7fae0cf52700 1 -- 172.21.15.33:0/2404569624 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae08108c60 con 0x7fae08107a60 2024-03-19T02:43:05.509 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.459+0000 7fae0cf52700 1 --2- 172.21.15.33:0/2404569624 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae08107a60 0x7fae08107e80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fadfc0098a0 tx=0x7fadfc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75b90275b3bed91c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:05.509 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.459+0000 7fae077fe700 1 -- 172.21.15.33:0/2404569624 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fadfc004160 con 0x7fae08107a60 2024-03-19T02:43:05.509 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.459+0000 7fae077fe700 1 -- 172.21.15.33:0/2404569624 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fadfc004300 con 0x7fae08107a60 2024-03-19T02:43:05.509 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.459+0000 7fae077fe700 1 -- 172.21.15.33:0/2404569624 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fadfc011a80 con 0x7fae08107a60 2024-03-19T02:43:05.510 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.460+0000 7fae0f1b6700 1 -- 172.21.15.33:0/2404569624 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae08107a60 msgr2=0x7fae08107e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:05.510 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.460+0000 7fae0f1b6700 1 --2- 172.21.15.33:0/2404569624 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae08107a60 0x7fae08107e80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fadfc0098a0 tx=0x7fadfc009bb0 comp rx=0 tx=0).stop 2024-03-19T02:43:05.510 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.460+0000 7fae0f1b6700 1 -- 172.21.15.33:0/2404569624 shutdown_connections 2024-03-19T02:43:05.510 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.460+0000 7fae0f1b6700 1 --2- 172.21.15.33:0/2404569624 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae08107a60 0x7fae08107e80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:05.510 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.460+0000 7fae0f1b6700 1 -- 172.21.15.33:0/2404569624 >> 172.21.15.33:0/2404569624 conn(0x7fae08103020 msgr2=0x7fae08105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:05.510 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.460+0000 7fae0f1b6700 1 -- 172.21.15.33:0/2404569624 shutdown_connections 2024-03-19T02:43:05.510 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.460+0000 7fae0f1b6700 1 -- 172.21.15.33:0/2404569624 wait complete. 2024-03-19T02:43:05.510 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.460+0000 7fae0f1b6700 1 Processor -- start 2024-03-19T02:43:05.510 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.461+0000 7fae0f1b6700 1 -- start start 2024-03-19T02:43:05.510 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.461+0000 7fae0f1b6700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae08107a60 0x7fae0807db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:05.510 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.461+0000 7fae0f1b6700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fae0810c010 con 0x7fae08107a60 2024-03-19T02:43:05.510 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.461+0000 7fae0cf52700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae08107a60 0x7fae0807db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:05.510 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.461+0000 7fae0cf52700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae08107a60 0x7fae0807db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51994/0 (socket says 172.21.15.33:51994) 2024-03-19T02:43:05.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.461+0000 7fae0cf52700 1 -- 172.21.15.33:0/415908847 learned_addr learned my addr 172.21.15.33:0/415908847 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:05.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.461+0000 7fae0cf52700 1 -- 172.21.15.33:0/415908847 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae0807e080 con 0x7fae08107a60 2024-03-19T02:43:05.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.461+0000 7fae0cf52700 1 --2- 172.21.15.33:0/415908847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae08107a60 0x7fae0807db40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fadfc006af0 tx=0x7fadfc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:05.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.461+0000 7fae05ffb700 1 -- 172.21.15.33:0/415908847 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fadfc0039e0 con 0x7fae08107a60 2024-03-19T02:43:05.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.461+0000 7fae05ffb700 1 -- 172.21.15.33:0/415908847 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fadfc003b80 con 0x7fae08107a60 2024-03-19T02:43:05.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.461+0000 7fae0f1b6700 1 -- 172.21.15.33:0/415908847 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae0807c270 con 0x7fae08107a60 2024-03-19T02:43:05.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.461+0000 7fae05ffb700 1 -- 172.21.15.33:0/415908847 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fadfc0196a0 con 0x7fae08107a60 2024-03-19T02:43:05.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.462+0000 7fae0f1b6700 1 -- 172.21.15.33:0/415908847 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae0807c7b0 con 0x7fae08107a60 2024-03-19T02:43:05.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.462+0000 7fae05ffb700 1 -- 172.21.15.33:0/415908847 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7fadfc0199a0 con 0x7fae08107a60 2024-03-19T02:43:05.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.463+0000 7fae05ffb700 1 --2- 172.21.15.33:0/415908847 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7fadf8038f40 0x7fadf803b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:05.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.463+0000 7fae0f1b6700 1 -- 172.21.15.33:0/415908847 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae08063940 con 0x7fae08107a60 2024-03-19T02:43:05.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.463+0000 7fae05ffb700 1 -- 172.21.15.33:0/415908847 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fadfc04d5a0 con 0x7fae08107a60 2024-03-19T02:43:05.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.463+0000 7fae07fff700 1 --2- 172.21.15.33:0/415908847 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7fadf8038f40 0x7fadf803b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:05.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.463+0000 7fae07fff700 1 --2- 172.21.15.33:0/415908847 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7fadf8038f40 0x7fadf803b400 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fadf4006fd0 tx=0x7fadf4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:05.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.468+0000 7fae05ffb700 1 -- 172.21.15.33:0/415908847 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fadfc00f2e0 con 0x7fae08107a60 2024-03-19T02:43:05.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.728+0000 7fae0f1b6700 1 -- 172.21.15.33:0/415908847 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fae0807ca30 con 0x7fae08107a60 2024-03-19T02:43:05.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.925+0000 7fae05ffb700 1 -- 172.21.15.33:0/415908847 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7fadfc029260 con 0x7fae08107a60 2024-03-19T02:43:05.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.926+0000 7fae05ffb700 1 -- 172.21.15.33:0/415908847 <== mon.0 v2:172.21.15.33:3300/0 8 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7fadfc02d960 con 0x7fae08107a60 2024-03-19T02:43:05.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.928+0000 7fae0f1b6700 1 -- 172.21.15.33:0/415908847 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7fadf8038f40 msgr2=0x7fadf803b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:05.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.928+0000 7fae0f1b6700 1 --2- 172.21.15.33:0/415908847 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7fadf8038f40 0x7fadf803b400 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fadf4006fd0 tx=0x7fadf4006e40 comp rx=0 tx=0).stop 2024-03-19T02:43:05.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.928+0000 7fae0f1b6700 1 -- 172.21.15.33:0/415908847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae08107a60 msgr2=0x7fae0807db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:05.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.928+0000 7fae0f1b6700 1 --2- 172.21.15.33:0/415908847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae08107a60 0x7fae0807db40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fadfc006af0 tx=0x7fadfc011c40 comp rx=0 tx=0).stop 2024-03-19T02:43:05.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.928+0000 7fae0f1b6700 1 -- 172.21.15.33:0/415908847 shutdown_connections 2024-03-19T02:43:05.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.928+0000 7fae0f1b6700 1 --2- 172.21.15.33:0/415908847 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7fadf8038f40 0x7fadf803b400 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:05.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.928+0000 7fae0f1b6700 1 --2- 172.21.15.33:0/415908847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae08107a60 0x7fae0807db40 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:05.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.928+0000 7fae0f1b6700 1 -- 172.21.15.33:0/415908847 >> 172.21.15.33:0/415908847 conn(0x7fae08103020 msgr2=0x7fae08103a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:05.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.928+0000 7fae0f1b6700 1 -- 172.21.15.33:0/415908847 shutdown_connections 2024-03-19T02:43:05.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:04.928+0000 7fae0f1b6700 1 -- 172.21.15.33:0/415908847 wait complete. 2024-03-19T02:43:05.778 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:05 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:05.493+0000 7f07ad828200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T02:43:05.778 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:05 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:05.592+0000 7f07ad828200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T02:43:06.125 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:05 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/415908847' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-19T02:43:06.126 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:05 smithi033 ceph-mon[31973]: mgrmap e5: x(active, since 6s) 2024-03-19T02:43:06.499 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:06 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:06.268+0000 7f07ad828200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T02:43:06.499 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:06 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:06.363+0000 7f07ad828200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T02:43:06.778 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:06 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:06.508+0000 7f07ad828200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T02:43:06.778 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:06 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:06.606+0000 7f07ad828200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T02:43:06.778 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:06 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:06.690+0000 7f07ad828200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T02:43:06.849 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout { 2024-03-19T02:43:06.850 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-19T02:43:06.850 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-19T02:43:06.850 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-19T02:43:06.850 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-19T02:43:06.850 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout } 2024-03-19T02:43:06.850 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.919+0000 7f8ee4ee1700 1 Processor -- start 2024-03-19T02:43:06.850 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.919+0000 7f8ee4ee1700 1 -- start start 2024-03-19T02:43:06.850 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.920+0000 7f8ee4ee1700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ee0108580 0x7f8ee010a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:06.850 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.920+0000 7f8ee4ee1700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee007be20 con 0x7f8ee0108580 2024-03-19T02:43:06.850 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.920+0000 7f8ede59c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ee0108580 0x7f8ee010a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:06.850 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.920+0000 7f8ede59c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ee0108580 0x7f8ee010a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52022/0 (socket says 172.21.15.33:52022) 2024-03-19T02:43:06.850 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.920+0000 7f8ede59c700 1 -- 172.21.15.33:0/635978927 learned_addr learned my addr 172.21.15.33:0/635978927 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:06.850 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.920+0000 7f8ede59c700 1 -- 172.21.15.33:0/635978927 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ee010aeb0 con 0x7f8ee0108580 2024-03-19T02:43:06.851 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.920+0000 7f8ede59c700 1 --2- 172.21.15.33:0/635978927 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ee0108580 0x7f8ee010a970 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8ed40098a0 tx=0x7f8ed4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc2c1a60f6439b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:06.851 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.920+0000 7f8edd59a700 1 -- 172.21.15.33:0/635978927 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ed4004160 con 0x7f8ee0108580 2024-03-19T02:43:06.851 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.920+0000 7f8edd59a700 1 -- 172.21.15.33:0/635978927 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f8ed4004300 con 0x7f8ee0108580 2024-03-19T02:43:06.851 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.920+0000 7f8edd59a700 1 -- 172.21.15.33:0/635978927 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ed4011a80 con 0x7f8ee0108580 2024-03-19T02:43:06.851 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.921+0000 7f8ee4ee1700 1 -- 172.21.15.33:0/635978927 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ee0108580 msgr2=0x7f8ee010a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:06.851 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.921+0000 7f8ee4ee1700 1 --2- 172.21.15.33:0/635978927 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ee0108580 0x7f8ee010a970 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8ed40098a0 tx=0x7f8ed4009bb0 comp rx=0 tx=0).stop 2024-03-19T02:43:06.851 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.921+0000 7f8ee4ee1700 1 -- 172.21.15.33:0/635978927 shutdown_connections 2024-03-19T02:43:06.851 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.921+0000 7f8ee4ee1700 1 --2- 172.21.15.33:0/635978927 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ee0108580 0x7f8ee010a970 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:06.851 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.921+0000 7f8ee4ee1700 1 -- 172.21.15.33:0/635978927 >> 172.21.15.33:0/635978927 conn(0x7f8ee0101f90 msgr2=0x7f8ee01043b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:06.851 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.921+0000 7f8ee4ee1700 1 -- 172.21.15.33:0/635978927 shutdown_connections 2024-03-19T02:43:06.851 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.921+0000 7f8ee4ee1700 1 -- 172.21.15.33:0/635978927 wait complete. 2024-03-19T02:43:06.851 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.922+0000 7f8ee4ee1700 1 Processor -- start 2024-03-19T02:43:06.851 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.922+0000 7f8ee4ee1700 1 -- start start 2024-03-19T02:43:06.851 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.922+0000 7f8ee4ee1700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ee0108580 0x7f8ee0107e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:06.851 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.922+0000 7f8ee4ee1700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee010be20 con 0x7f8ee0108580 2024-03-19T02:43:06.852 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.922+0000 7f8ede59c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ee0108580 0x7f8ee0107e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:06.852 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.922+0000 7f8ede59c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ee0108580 0x7f8ee0107e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52032/0 (socket says 172.21.15.33:52032) 2024-03-19T02:43:06.852 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.922+0000 7f8ede59c700 1 -- 172.21.15.33:0/2566759441 learned_addr learned my addr 172.21.15.33:0/2566759441 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:06.852 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.923+0000 7f8ede59c700 1 -- 172.21.15.33:0/2566759441 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ee0108340 con 0x7f8ee0108580 2024-03-19T02:43:06.852 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.923+0000 7f8ede59c700 1 --2- 172.21.15.33:0/2566759441 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ee0108580 0x7f8ee0107e00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8ed4011a50 tx=0x7f8ed4011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:06.852 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.923+0000 7f8ecf7fe700 1 -- 172.21.15.33:0/2566759441 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ed4003a80 con 0x7f8ee0108580 2024-03-19T02:43:06.852 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.923+0000 7f8ee4ee1700 1 -- 172.21.15.33:0/2566759441 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ee0106530 con 0x7f8ee0108580 2024-03-19T02:43:06.852 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.923+0000 7f8ecf7fe700 1 -- 172.21.15.33:0/2566759441 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f8ed4003c20 con 0x7f8ee0108580 2024-03-19T02:43:06.852 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.923+0000 7f8ecf7fe700 1 -- 172.21.15.33:0/2566759441 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ed401a650 con 0x7f8ee0108580 2024-03-19T02:43:06.852 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.923+0000 7f8ee4ee1700 1 -- 172.21.15.33:0/2566759441 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ee0106a70 con 0x7f8ee0108580 2024-03-19T02:43:06.852 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.924+0000 7f8ecf7fe700 1 -- 172.21.15.33:0/2566759441 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7f8ed401a970 con 0x7f8ee0108580 2024-03-19T02:43:06.852 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.924+0000 7f8ee4ee1700 1 -- 172.21.15.33:0/2566759441 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ee0063940 con 0x7f8ee0108580 2024-03-19T02:43:06.852 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.924+0000 7f8ecf7fe700 1 --2- 172.21.15.33:0/2566759441 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7f8ec8038fe0 0x7f8ec803b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:06.852 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.924+0000 7f8ecf7fe700 1 -- 172.21.15.33:0/2566759441 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8ed404d290 con 0x7f8ee0108580 2024-03-19T02:43:06.852 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.924+0000 7f8eddd9b700 1 -- 172.21.15.33:0/2566759441 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7f8ec8038fe0 msgr2=0x7f8ec803b4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/3342667183 2024-03-19T02:43:06.853 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.924+0000 7f8eddd9b700 1 --2- 172.21.15.33:0/2566759441 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7f8ec8038fe0 0x7f8ec803b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T02:43:06.853 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:05.929+0000 7f8ecf7fe700 1 -- 172.21.15.33:0/2566759441 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8ed4019c70 con 0x7f8ee0108580 2024-03-19T02:43:06.853 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:06.125+0000 7f8eddd9b700 1 -- 172.21.15.33:0/2566759441 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7f8ec8038fe0 msgr2=0x7f8ec803b4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/3342667183 2024-03-19T02:43:06.853 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:06.125+0000 7f8eddd9b700 1 --2- 172.21.15.33:0/2566759441 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7f8ec8038fe0 0x7f8ec803b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T02:43:06.853 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:06.206+0000 7f8ee4ee1700 1 -- 172.21.15.33:0/2566759441 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f8ee0107260 con 0x7f8ee0108580 2024-03-19T02:43:06.853 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:06.206+0000 7f8ecf7fe700 1 -- 172.21.15.33:0/2566759441 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f8ed401ad10 con 0x7f8ee0108580 2024-03-19T02:43:06.853 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:06.208+0000 7f8ee4ee1700 1 -- 172.21.15.33:0/2566759441 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7f8ec8038fe0 msgr2=0x7f8ec803b4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T02:43:06.853 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:06.208+0000 7f8ee4ee1700 1 --2- 172.21.15.33:0/2566759441 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7f8ec8038fe0 0x7f8ec803b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:06.853 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:06.208+0000 7f8ee4ee1700 1 -- 172.21.15.33:0/2566759441 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ee0108580 msgr2=0x7f8ee0107e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:06.853 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:06.208+0000 7f8ee4ee1700 1 --2- 172.21.15.33:0/2566759441 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ee0108580 0x7f8ee0107e00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8ed4011a50 tx=0x7f8ed4011b30 comp rx=0 tx=0).stop 2024-03-19T02:43:06.853 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:06.208+0000 7f8ee4ee1700 1 -- 172.21.15.33:0/2566759441 shutdown_connections 2024-03-19T02:43:06.853 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:06.208+0000 7f8ee4ee1700 1 --2- 172.21.15.33:0/2566759441 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7f8ec8038fe0 0x7f8ec803b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:06.853 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:06.208+0000 7f8ee4ee1700 1 --2- 172.21.15.33:0/2566759441 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ee0108580 0x7f8ee0107e00 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:06.853 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:06.208+0000 7f8ee4ee1700 1 -- 172.21.15.33:0/2566759441 >> 172.21.15.33:0/2566759441 conn(0x7f8ee0101f90 msgr2=0x7f8ee0102940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:06.853 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:06.208+0000 7f8ee4ee1700 1 -- 172.21.15.33:0/2566759441 shutdown_connections 2024-03-19T02:43:06.854 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:06.208+0000 7f8ee4ee1700 1 -- 172.21.15.33:0/2566759441 wait complete. 2024-03-19T02:43:06.854 INFO:teuthology.orchestra.run.smithi033.stdout:Waiting for the mgr to restart... 2024-03-19T02:43:06.854 INFO:teuthology.orchestra.run.smithi033.stdout:Waiting for mgr epoch 5... 2024-03-19T02:43:07.093 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:06 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2566759441' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-19T02:43:07.094 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:06 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:06.962+0000 7f07ad828200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-19T02:43:07.094 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:07 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:07.075+0000 7f07ad828200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T02:43:07.446 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:07 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:07.167+0000 7f07ad828200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T02:43:07.446 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:07 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:07.266+0000 7f07ad828200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T02:43:07.778 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:07 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:07.445+0000 7f07ad828200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T02:43:08.712 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:08 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:08.440+0000 7f07ad828200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-19T02:43:08.712 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:08 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:08.575+0000 7f07ad828200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T02:43:09.005 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:08 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:08.913+0000 7f07ad828200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T02:43:09.005 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:09 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:09.004+0000 7f07ad828200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T02:43:09.278 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:09 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:09.204+0000 7f07ad828200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:43:09.682 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:09 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:09.392+0000 7f07ad828200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T02:43:09.941 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:09 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:09.681+0000 7f07ad828200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:43:09.941 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:09 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:09.940+0000 7f07ad828200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T02:43:10.278 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:10 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:10.182+0000 7f07ad828200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T02:43:11.278 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:11 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:11.005+0000 7f07ad828200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T02:43:11.904 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:11 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:11.583+0000 7f07ad828200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T02:43:11.905 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:11 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:11.683+0000 7f07ad828200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T02:43:12.278 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:11 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:11.903+0000 7f07ad828200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T02:43:12.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:11 smithi033 ceph-mon[31973]: Active manager daemon x restarted 2024-03-19T02:43:12.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:11 smithi033 ceph-mon[31973]: Activating manager daemon x 2024-03-19T02:43:12.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:11 smithi033 ceph-mon[31973]: osdmap e2: 0 total, 0 up, 0 in 2024-03-19T02:43:12.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:11 smithi033 ceph-mon[31973]: mgrmap e6: x(active, starting, since 0.00322596s) 2024-03-19T02:43:12.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:11 smithi033 ceph-mon[31973]: from='mgr.14118 172.21.15.33:0/3216842381' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:43:12.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:11 smithi033 ceph-mon[31973]: from='mgr.14118 172.21.15.33:0/3216842381' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-19T02:43:12.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:11 smithi033 ceph-mon[31973]: from='mgr.14118 172.21.15.33:0/3216842381' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-19T02:43:12.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:11 smithi033 ceph-mon[31973]: from='mgr.14118 172.21.15.33:0/3216842381' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-19T02:43:12.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:11 smithi033 ceph-mon[31973]: from='mgr.14118 172.21.15.33:0/3216842381' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-19T02:43:12.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:11 smithi033 ceph-mon[31973]: Manager daemon x is now available 2024-03-19T02:43:12.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:11 smithi033 ceph-mon[31973]: from='mgr.14118 172.21.15.33:0/3216842381' entity='mgr.x' 2024-03-19T02:43:12.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:11 smithi033 ceph-mon[31973]: from='mgr.14118 172.21.15.33:0/3216842381' entity='mgr.x' 2024-03-19T02:43:12.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:11 smithi033 ceph-mon[31973]: from='mgr.14118 172.21.15.33:0/3216842381' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:43:12.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:11 smithi033 ceph-mon[31973]: from='mgr.14118 172.21.15.33:0/3216842381' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:43:13.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:12 smithi033 ceph-mon[31973]: Found migration_current of "None". Setting to last migration. 2024-03-19T02:43:13.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:12 smithi033 ceph-mon[31973]: from='mgr.14118 172.21.15.33:0/3216842381' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:43:13.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:12 smithi033 ceph-mon[31973]: from='mgr.14118 172.21.15.33:0/3216842381' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-19T02:43:13.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:12 smithi033 ceph-mon[31973]: from='mgr.14118 172.21.15.33:0/3216842381' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-19T02:43:13.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:12 smithi033 ceph-mon[31973]: from='mgr.14118 172.21.15.33:0/3216842381' entity='mgr.x' 2024-03-19T02:43:13.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:12 smithi033 ceph-mon[31973]: from='mgr.14118 172.21.15.33:0/3216842381' entity='mgr.x' 2024-03-19T02:43:13.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:12 smithi033 ceph-mon[31973]: mgrmap e7: x(active, since 1.00555s) 2024-03-19T02:43:13.659 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout { 2024-03-19T02:43:13.659 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-19T02:43:13.660 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-19T02:43:13.660 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout } 2024-03-19T02:43:13.660 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.304+0000 7f2389588700 1 Processor -- start 2024-03-19T02:43:13.660 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.304+0000 7f2389588700 1 -- start start 2024-03-19T02:43:13.660 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.304+0000 7f2389588700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2384106980 0x7f2384106da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:13.660 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.304+0000 7f2389588700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2384107370 con 0x7f2384106980 2024-03-19T02:43:13.660 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.304+0000 7f2382ffd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2384106980 0x7f2384106da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:13.660 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.304+0000 7f2382ffd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2384106980 0x7f2384106da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52048/0 (socket says 172.21.15.33:52048) 2024-03-19T02:43:13.660 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.304+0000 7f2382ffd700 1 -- 172.21.15.33:0/3212458178 learned_addr learned my addr 172.21.15.33:0/3212458178 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:13.660 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.304+0000 7f2382ffd700 1 -- 172.21.15.33:0/3212458178 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2384107b80 con 0x7f2384106980 2024-03-19T02:43:13.660 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.305+0000 7f2382ffd700 1 --2- 172.21.15.33:0/3212458178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2384106980 0x7f2384106da0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f23740098a0 tx=0x7f2374009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8586362846847411 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:13.660 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.305+0000 7f2381ffb700 1 -- 172.21.15.33:0/3212458178 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2374004160 con 0x7f2384106980 2024-03-19T02:43:13.660 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.305+0000 7f2381ffb700 1 -- 172.21.15.33:0/3212458178 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f2374004300 con 0x7f2384106980 2024-03-19T02:43:13.661 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.305+0000 7f2381ffb700 1 -- 172.21.15.33:0/3212458178 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2374011a80 con 0x7f2384106980 2024-03-19T02:43:13.661 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.305+0000 7f2389588700 1 -- 172.21.15.33:0/3212458178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2384106980 msgr2=0x7f2384106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:13.661 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.305+0000 7f2389588700 1 --2- 172.21.15.33:0/3212458178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2384106980 0x7f2384106da0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f23740098a0 tx=0x7f2374009bb0 comp rx=0 tx=0).stop 2024-03-19T02:43:13.661 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.305+0000 7f2389588700 1 -- 172.21.15.33:0/3212458178 shutdown_connections 2024-03-19T02:43:13.661 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.305+0000 7f2389588700 1 --2- 172.21.15.33:0/3212458178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2384106980 0x7f2384106da0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:13.661 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.305+0000 7f2389588700 1 -- 172.21.15.33:0/3212458178 >> 172.21.15.33:0/3212458178 conn(0x7f2384101f40 msgr2=0x7f2384104360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:13.661 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.306+0000 7f2389588700 1 -- 172.21.15.33:0/3212458178 shutdown_connections 2024-03-19T02:43:13.661 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.306+0000 7f2389588700 1 -- 172.21.15.33:0/3212458178 wait complete. 2024-03-19T02:43:13.661 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.306+0000 7f2389588700 1 Processor -- start 2024-03-19T02:43:13.661 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.307+0000 7f2389588700 1 -- start start 2024-03-19T02:43:13.661 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.307+0000 7f2389588700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2384106980 0x7f238407ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:13.661 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.307+0000 7f2389588700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f238410af30 con 0x7f2384106980 2024-03-19T02:43:13.661 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.307+0000 7f2382ffd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2384106980 0x7f238407ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:13.662 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.307+0000 7f2382ffd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2384106980 0x7f238407ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52060/0 (socket says 172.21.15.33:52060) 2024-03-19T02:43:13.662 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.307+0000 7f2382ffd700 1 -- 172.21.15.33:0/2822010109 learned_addr learned my addr 172.21.15.33:0/2822010109 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:13.662 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.307+0000 7f2382ffd700 1 -- 172.21.15.33:0/2822010109 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f238419e670 con 0x7f2384106980 2024-03-19T02:43:13.662 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.307+0000 7f2382ffd700 1 --2- 172.21.15.33:0/2822010109 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2384106980 0x7f238407ba40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2374000c00 tx=0x7f2374004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:13.662 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.307+0000 7f236bfff700 1 -- 172.21.15.33:0/2822010109 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f23740039c0 con 0x7f2384106980 2024-03-19T02:43:13.662 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.308+0000 7f236bfff700 1 -- 172.21.15.33:0/2822010109 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f2374003b60 con 0x7f2384106980 2024-03-19T02:43:13.662 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.308+0000 7f2389588700 1 -- 172.21.15.33:0/2822010109 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f238419e840 con 0x7f2384106980 2024-03-19T02:43:13.662 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.308+0000 7f2389588700 1 -- 172.21.15.33:0/2822010109 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f238419ec60 con 0x7f2384106980 2024-03-19T02:43:13.662 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.308+0000 7f236bfff700 1 -- 172.21.15.33:0/2822010109 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f237401a6f0 con 0x7f2384106980 2024-03-19T02:43:13.662 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.308+0000 7f236bfff700 1 -- 172.21.15.33:0/2822010109 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7f237401a9a0 con 0x7f2384106980 2024-03-19T02:43:13.662 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.309+0000 7f236bfff700 1 --2- 172.21.15.33:0/2822010109 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7f236c038fe0 0x7f236c03b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:13.662 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.309+0000 7f236bfff700 1 -- 172.21.15.33:0/2822010109 --> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f236c03bbb0 con 0x7f236c038fe0 2024-03-19T02:43:13.662 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.309+0000 7f236bfff700 1 -- 172.21.15.33:0/2822010109 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f237404dd40 con 0x7f2384106980 2024-03-19T02:43:13.663 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.309+0000 7f23827fc700 1 -- 172.21.15.33:0/2822010109 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7f236c038fe0 msgr2=0x7f236c03b4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/3342667183 2024-03-19T02:43:13.663 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.309+0000 7f23827fc700 1 --2- 172.21.15.33:0/2822010109 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7f236c038fe0 0x7f236c03b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T02:43:13.663 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.510+0000 7f23827fc700 1 -- 172.21.15.33:0/2822010109 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7f236c038fe0 msgr2=0x7f236c03b4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/3342667183 2024-03-19T02:43:13.663 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.510+0000 7f23827fc700 1 --2- 172.21.15.33:0/2822010109 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7f236c038fe0 0x7f236c03b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T02:43:13.663 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.911+0000 7f23827fc700 1 -- 172.21.15.33:0/2822010109 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7f236c038fe0 msgr2=0x7f236c03b4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/3342667183 2024-03-19T02:43:13.663 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:07.911+0000 7f23827fc700 1 --2- 172.21.15.33:0/2822010109 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7f236c038fe0 0x7f236c03b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-19T02:43:13.663 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:08.712+0000 7f23827fc700 1 -- 172.21.15.33:0/2822010109 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7f236c038fe0 msgr2=0x7f236c03b4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/3342667183 2024-03-19T02:43:13.663 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:08.712+0000 7f23827fc700 1 --2- 172.21.15.33:0/2822010109 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7f236c038fe0 0x7f236c03b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-19T02:43:13.663 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:10.313+0000 7f23827fc700 1 -- 172.21.15.33:0/2822010109 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7f236c038fe0 msgr2=0x7f236c03b4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/3342667183 2024-03-19T02:43:13.663 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:10.313+0000 7f23827fc700 1 --2- 172.21.15.33:0/2822010109 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7f236c038fe0 0x7f236c03b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-19T02:43:13.663 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:11.919+0000 7f236bfff700 1 -- 172.21.15.33:0/2822010109 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mgrmap(e 6) ==== 45873+0+0 (secure 0 0 0) 0x7f237402d910 con 0x7f2384106980 2024-03-19T02:43:13.663 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:11.919+0000 7f236bfff700 1 -- 172.21.15.33:0/2822010109 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7f236c038fe0 msgr2=0x7f236c03b4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T02:43:13.663 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:11.919+0000 7f236bfff700 1 --2- 172.21.15.33:0/2822010109 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7f236c038fe0 0x7f236c03b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:13.663 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:12.920+0000 7f236bfff700 1 -- 172.21.15.33:0/2822010109 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mgrmap(e 7) ==== 46000+0+0 (secure 0 0 0) 0x7f237402d3f0 con 0x7f2384106980 2024-03-19T02:43:13.664 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:12.920+0000 7f236bfff700 1 --2- 172.21.15.33:0/2822010109 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f236c03c8d0 0x7f236c03ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:13.664 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:12.920+0000 7f236bfff700 1 -- 172.21.15.33:0/2822010109 --> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f236c03bbb0 con 0x7f236c03c8d0 2024-03-19T02:43:13.664 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:12.920+0000 7f23827fc700 1 --2- 172.21.15.33:0/2822010109 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f236c03c8d0 0x7f236c03ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:13.664 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:12.921+0000 7f23827fc700 1 --2- 172.21.15.33:0/2822010109 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f236c03c8d0 0x7f236c03ecc0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2378003a10 tx=0x7f2378009290 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:13.664 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:12.922+0000 7f236bfff700 1 -- 172.21.15.33:0/2822010109 <== mgr.14118 v2:172.21.15.33:6800/906645748 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f236c03bbb0 con 0x7f236c03c8d0 2024-03-19T02:43:13.664 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:12.928+0000 7f2389588700 1 -- 172.21.15.33:0/2822010109 --> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f2370002630 con 0x7f236c03c8d0 2024-03-19T02:43:13.664 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:12.928+0000 7f236bfff700 1 -- 172.21.15.33:0/2822010109 <== mgr.14118 v2:172.21.15.33:6800/906645748 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f2370002630 con 0x7f236c03c8d0 2024-03-19T02:43:13.664 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:12.929+0000 7f2389588700 1 -- 172.21.15.33:0/2822010109 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f236c03c8d0 msgr2=0x7f236c03ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:13.664 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:12.929+0000 7f2389588700 1 --2- 172.21.15.33:0/2822010109 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f236c03c8d0 0x7f236c03ecc0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2378003a10 tx=0x7f2378009290 comp rx=0 tx=0).stop 2024-03-19T02:43:13.664 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:12.929+0000 7f2389588700 1 -- 172.21.15.33:0/2822010109 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2384106980 msgr2=0x7f238407ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:13.664 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:12.929+0000 7f2389588700 1 --2- 172.21.15.33:0/2822010109 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2384106980 0x7f238407ba40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2374000c00 tx=0x7f2374004290 comp rx=0 tx=0).stop 2024-03-19T02:43:13.664 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:12.929+0000 7f2389588700 1 -- 172.21.15.33:0/2822010109 shutdown_connections 2024-03-19T02:43:13.664 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:12.929+0000 7f2389588700 1 --2- 172.21.15.33:0/2822010109 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f236c03c8d0 0x7f236c03ecc0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:13.664 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:12.929+0000 7f2389588700 1 --2- 172.21.15.33:0/2822010109 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2384106980 0x7f238407ba40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:13.664 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:12.929+0000 7f2389588700 1 --2- 172.21.15.33:0/2822010109 >> [v2:172.21.15.33:6800/3342667183,v1:172.21.15.33:6801/3342667183] conn(0x7f236c038fe0 0x7f236c03b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:13.665 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:12.929+0000 7f2389588700 1 -- 172.21.15.33:0/2822010109 >> 172.21.15.33:0/2822010109 conn(0x7f2384101f40 msgr2=0x7f2384102a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:13.665 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:12.929+0000 7f2389588700 1 -- 172.21.15.33:0/2822010109 shutdown_connections 2024-03-19T02:43:13.665 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:12.929+0000 7f2389588700 1 -- 172.21.15.33:0/2822010109 wait complete. 2024-03-19T02:43:13.665 INFO:teuthology.orchestra.run.smithi033.stdout:mgr epoch 5 is available 2024-03-19T02:43:13.665 INFO:teuthology.orchestra.run.smithi033.stdout:Setting orchestrator backend to cephadm... 2024-03-19T02:43:14.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:13 smithi033 ceph-mon[31973]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-19T02:43:14.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:13 smithi033 ceph-mon[31973]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-19T02:43:14.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:13 smithi033 ceph-mon[31973]: from='mgr.14118 172.21.15.33:0/3216842381' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:43:14.956 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.115+0000 7f6b92898700 1 Processor -- start 2024-03-19T02:43:14.956 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.115+0000 7f6b92898700 1 -- start start 2024-03-19T02:43:14.956 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.115+0000 7f6b92898700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b8c106390 0x7f6b8c1067b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:14.956 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.115+0000 7f6b92898700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8c07bde0 con 0x7f6b8c106390 2024-03-19T02:43:14.956 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.115+0000 7f6b8bfff700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b8c106390 0x7f6b8c1067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:14.956 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.115+0000 7f6b8bfff700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b8c106390 0x7f6b8c1067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55376/0 (socket says 172.21.15.33:55376) 2024-03-19T02:43:14.956 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.115+0000 7f6b8bfff700 1 -- 172.21.15.33:0/2083460774 learned_addr learned my addr 172.21.15.33:0/2083460774 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:14.956 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.115+0000 7f6b8bfff700 1 -- 172.21.15.33:0/2083460774 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b8c106cf0 con 0x7f6b8c106390 2024-03-19T02:43:14.956 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.116+0000 7f6b8bfff700 1 --2- 172.21.15.33:0/2083460774 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b8c106390 0x7f6b8c1067b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c0098a0 tx=0x7f6b7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee2546c0e9fefd34 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:14.957 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.116+0000 7f6b8affd700 1 -- 172.21.15.33:0/2083460774 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b7c004160 con 0x7f6b8c106390 2024-03-19T02:43:14.957 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.116+0000 7f6b8affd700 1 -- 172.21.15.33:0/2083460774 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f6b7c004300 con 0x7f6b8c106390 2024-03-19T02:43:14.957 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.116+0000 7f6b8affd700 1 -- 172.21.15.33:0/2083460774 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b7c011a80 con 0x7f6b8c106390 2024-03-19T02:43:14.957 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.116+0000 7f6b92898700 1 -- 172.21.15.33:0/2083460774 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b8c106390 msgr2=0x7f6b8c1067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:14.957 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.116+0000 7f6b92898700 1 --2- 172.21.15.33:0/2083460774 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b8c106390 0x7f6b8c1067b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c0098a0 tx=0x7f6b7c009bb0 comp rx=0 tx=0).stop 2024-03-19T02:43:14.957 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.116+0000 7f6b92898700 1 -- 172.21.15.33:0/2083460774 shutdown_connections 2024-03-19T02:43:14.957 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.116+0000 7f6b92898700 1 --2- 172.21.15.33:0/2083460774 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b8c106390 0x7f6b8c1067b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:14.957 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.116+0000 7f6b92898700 1 -- 172.21.15.33:0/2083460774 >> 172.21.15.33:0/2083460774 conn(0x7f6b8c101f30 msgr2=0x7f6b8c104370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:14.957 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.116+0000 7f6b92898700 1 -- 172.21.15.33:0/2083460774 shutdown_connections 2024-03-19T02:43:14.957 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.116+0000 7f6b92898700 1 -- 172.21.15.33:0/2083460774 wait complete. 2024-03-19T02:43:14.957 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.117+0000 7f6b92898700 1 Processor -- start 2024-03-19T02:43:14.957 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.117+0000 7f6b92898700 1 -- start start 2024-03-19T02:43:14.957 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.117+0000 7f6b92898700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b8c106390 0x7f6b8c19dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:14.957 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.117+0000 7f6b92898700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8c107d20 con 0x7f6b8c106390 2024-03-19T02:43:14.958 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.117+0000 7f6b8bfff700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b8c106390 0x7f6b8c19dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:14.958 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.117+0000 7f6b8bfff700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b8c106390 0x7f6b8c19dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55390/0 (socket says 172.21.15.33:55390) 2024-03-19T02:43:14.958 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.117+0000 7f6b8bfff700 1 -- 172.21.15.33:0/3768974176 learned_addr learned my addr 172.21.15.33:0/3768974176 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:14.958 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.118+0000 7f6b8bfff700 1 -- 172.21.15.33:0/3768974176 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b8c19e010 con 0x7f6b8c106390 2024-03-19T02:43:14.958 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.118+0000 7f6b8bfff700 1 --2- 172.21.15.33:0/3768974176 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b8c106390 0x7f6b8c19dad0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c006af0 tx=0x7f6b7c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:14.958 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.118+0000 7f6b897fa700 1 -- 172.21.15.33:0/3768974176 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b7c0039e0 con 0x7f6b8c106390 2024-03-19T02:43:14.958 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.118+0000 7f6b897fa700 1 -- 172.21.15.33:0/3768974176 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f6b7c003b80 con 0x7f6b8c106390 2024-03-19T02:43:14.958 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.118+0000 7f6b92898700 1 -- 172.21.15.33:0/3768974176 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b8c19e2a0 con 0x7f6b8c106390 2024-03-19T02:43:14.958 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.118+0000 7f6b897fa700 1 -- 172.21.15.33:0/3768974176 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b7c0196a0 con 0x7f6b8c106390 2024-03-19T02:43:14.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.118+0000 7f6b92898700 1 -- 172.21.15.33:0/3768974176 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b8c1a0f90 con 0x7f6b8c106390 2024-03-19T02:43:14.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.119+0000 7f6b897fa700 1 -- 172.21.15.33:0/3768974176 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f6b7c019950 con 0x7f6b8c106390 2024-03-19T02:43:14.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.119+0000 7f6b92898700 1 -- 172.21.15.33:0/3768974176 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b8c063940 con 0x7f6b8c106390 2024-03-19T02:43:14.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.119+0000 7f6b897fa700 1 --2- 172.21.15.33:0/3768974176 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f6b74038fe0 0x7f6b7403b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:14.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.119+0000 7f6b897fa700 1 -- 172.21.15.33:0/3768974176 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6b7c04d510 con 0x7f6b8c106390 2024-03-19T02:43:14.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.119+0000 7f6b8b7fe700 1 --2- 172.21.15.33:0/3768974176 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f6b74038fe0 0x7f6b7403b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:14.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.120+0000 7f6b8b7fe700 1 --2- 172.21.15.33:0/3768974176 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f6b74038fe0 0x7f6b7403b4a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6b80006fd0 tx=0x7f6b80006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:14.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.124+0000 7f6b897fa700 1 -- 172.21.15.33:0/3768974176 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6b7c019d40 con 0x7f6b8c106390 2024-03-19T02:43:14.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.346+0000 7f6b92898700 1 -- 172.21.15.33:0/3768974176 --> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f6b8c0008d0 con 0x7f6b74038fe0 2024-03-19T02:43:14.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.352+0000 7f6b897fa700 1 -- 172.21.15.33:0/3768974176 <== mgr.14118 v2:172.21.15.33:6800/906645748 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6b8c0008d0 con 0x7f6b74038fe0 2024-03-19T02:43:14.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.354+0000 7f6b92898700 1 -- 172.21.15.33:0/3768974176 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f6b74038fe0 msgr2=0x7f6b7403b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:14.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.354+0000 7f6b92898700 1 --2- 172.21.15.33:0/3768974176 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f6b74038fe0 0x7f6b7403b4a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6b80006fd0 tx=0x7f6b80006e40 comp rx=0 tx=0).stop 2024-03-19T02:43:14.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.354+0000 7f6b92898700 1 -- 172.21.15.33:0/3768974176 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b8c106390 msgr2=0x7f6b8c19dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:14.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.354+0000 7f6b92898700 1 --2- 172.21.15.33:0/3768974176 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b8c106390 0x7f6b8c19dad0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c006af0 tx=0x7f6b7c011c40 comp rx=0 tx=0).stop 2024-03-19T02:43:14.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.354+0000 7f6b92898700 1 -- 172.21.15.33:0/3768974176 shutdown_connections 2024-03-19T02:43:14.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.354+0000 7f6b92898700 1 --2- 172.21.15.33:0/3768974176 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f6b74038fe0 0x7f6b7403b4a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:14.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.354+0000 7f6b92898700 1 --2- 172.21.15.33:0/3768974176 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6b8c106390 0x7f6b8c19dad0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:14.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.354+0000 7f6b92898700 1 -- 172.21.15.33:0/3768974176 >> 172.21.15.33:0/3768974176 conn(0x7f6b8c101f30 msgr2=0x7f6b8c1947b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:14.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.355+0000 7f6b92898700 1 -- 172.21.15.33:0/3768974176 shutdown_connections 2024-03-19T02:43:14.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:14.355+0000 7f6b92898700 1 -- 172.21.15.33:0/3768974176 wait complete. 2024-03-19T02:43:15.259 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:14 smithi033 ceph-mon[31973]: [19/Mar/2024:02:43:13] ENGINE Bus STARTING 2024-03-19T02:43:15.259 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:14 smithi033 ceph-mon[31973]: [19/Mar/2024:02:43:13] ENGINE Serving on http://172.21.15.33:8765 2024-03-19T02:43:15.259 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:14 smithi033 ceph-mon[31973]: [19/Mar/2024:02:43:13] ENGINE Serving on https://172.21.15.33:7150 2024-03-19T02:43:15.259 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:14 smithi033 ceph-mon[31973]: [19/Mar/2024:02:43:13] ENGINE Bus STARTED 2024-03-19T02:43:15.259 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:14 smithi033 ceph-mon[31973]: mgrmap e8: x(active, since 2s) 2024-03-19T02:43:15.259 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:14 smithi033 ceph-mon[31973]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:43:15.259 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:14 smithi033 ceph-mon[31973]: from='mgr.14118 172.21.15.33:0/3216842381' entity='mgr.x' 2024-03-19T02:43:15.259 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:14 smithi033 ceph-mon[31973]: from='mgr.14118 172.21.15.33:0/3216842381' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:43:16.138 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:15 smithi033 ceph-mon[31973]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:43:16.148 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-19T02:43:16.148 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.365+0000 7f00ec6c3700 1 Processor -- start 2024-03-19T02:43:16.148 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.365+0000 7f00ec6c3700 1 -- start start 2024-03-19T02:43:16.148 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.366+0000 7f00ec6c3700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00e4107aa0 0x7f00e4107ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:16.148 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.366+0000 7f00ec6c3700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f00e4108490 con 0x7f00e4107aa0 2024-03-19T02:43:16.148 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.366+0000 7f00ea45f700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00e4107aa0 0x7f00e4107ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:16.149 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.366+0000 7f00ea45f700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00e4107aa0 0x7f00e4107ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55398/0 (socket says 172.21.15.33:55398) 2024-03-19T02:43:16.149 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.366+0000 7f00ea45f700 1 -- 172.21.15.33:0/3830385044 learned_addr learned my addr 172.21.15.33:0/3830385044 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:16.149 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.366+0000 7f00ea45f700 1 -- 172.21.15.33:0/3830385044 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00e4108ca0 con 0x7f00e4107aa0 2024-03-19T02:43:16.149 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.366+0000 7f00ea45f700 1 --2- 172.21.15.33:0/3830385044 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00e4107aa0 0x7f00e4107ec0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f00e00098a0 tx=0x7f00e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4601b0dd1a3c8da server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:16.149 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.366+0000 7f00e945d700 1 -- 172.21.15.33:0/3830385044 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00e0004160 con 0x7f00e4107aa0 2024-03-19T02:43:16.149 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.366+0000 7f00e945d700 1 -- 172.21.15.33:0/3830385044 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f00e0004300 con 0x7f00e4107aa0 2024-03-19T02:43:16.149 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.366+0000 7f00e945d700 1 -- 172.21.15.33:0/3830385044 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00e0011a80 con 0x7f00e4107aa0 2024-03-19T02:43:16.149 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.367+0000 7f00ec6c3700 1 -- 172.21.15.33:0/3830385044 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00e4107aa0 msgr2=0x7f00e4107ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:16.149 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.367+0000 7f00ec6c3700 1 --2- 172.21.15.33:0/3830385044 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00e4107aa0 0x7f00e4107ec0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f00e00098a0 tx=0x7f00e0009bb0 comp rx=0 tx=0).stop 2024-03-19T02:43:16.149 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.367+0000 7f00ec6c3700 1 -- 172.21.15.33:0/3830385044 shutdown_connections 2024-03-19T02:43:16.149 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.367+0000 7f00ec6c3700 1 --2- 172.21.15.33:0/3830385044 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00e4107aa0 0x7f00e4107ec0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:16.149 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.367+0000 7f00ec6c3700 1 -- 172.21.15.33:0/3830385044 >> 172.21.15.33:0/3830385044 conn(0x7f00e4103020 msgr2=0x7f00e4105480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:16.149 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.367+0000 7f00ec6c3700 1 -- 172.21.15.33:0/3830385044 shutdown_connections 2024-03-19T02:43:16.150 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.367+0000 7f00ec6c3700 1 -- 172.21.15.33:0/3830385044 wait complete. 2024-03-19T02:43:16.150 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.368+0000 7f00ec6c3700 1 Processor -- start 2024-03-19T02:43:16.150 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.368+0000 7f00ec6c3700 1 -- start start 2024-03-19T02:43:16.150 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.368+0000 7f00ec6c3700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00e4107aa0 0x7f00e419b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:16.150 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.368+0000 7f00ec6c3700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f00e410c050 con 0x7f00e4107aa0 2024-03-19T02:43:16.150 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.368+0000 7f00ea45f700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00e4107aa0 0x7f00e419b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:16.150 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.368+0000 7f00ea45f700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00e4107aa0 0x7f00e419b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55414/0 (socket says 172.21.15.33:55414) 2024-03-19T02:43:16.150 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.368+0000 7f00ea45f700 1 -- 172.21.15.33:0/1113402713 learned_addr learned my addr 172.21.15.33:0/1113402713 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:16.150 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.368+0000 7f00ea45f700 1 -- 172.21.15.33:0/1113402713 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00e419be50 con 0x7f00e4107aa0 2024-03-19T02:43:16.150 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.368+0000 7f00ea45f700 1 --2- 172.21.15.33:0/1113402713 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00e4107aa0 0x7f00e419b910 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f00e0000c00 tx=0x7f00e0011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:16.151 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.369+0000 7f00d77fe700 1 -- 172.21.15.33:0/1113402713 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00e00039f0 con 0x7f00e4107aa0 2024-03-19T02:43:16.151 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.369+0000 7f00d77fe700 1 -- 172.21.15.33:0/1113402713 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f00e0003b90 con 0x7f00e4107aa0 2024-03-19T02:43:16.151 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.369+0000 7f00ec6c3700 1 -- 172.21.15.33:0/1113402713 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00e419c0e0 con 0x7f00e4107aa0 2024-03-19T02:43:16.151 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.369+0000 7f00d77fe700 1 -- 172.21.15.33:0/1113402713 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00e001a740 con 0x7f00e4107aa0 2024-03-19T02:43:16.151 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.369+0000 7f00ec6c3700 1 -- 172.21.15.33:0/1113402713 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00e419edd0 con 0x7f00e4107aa0 2024-03-19T02:43:16.151 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.369+0000 7f00d77fe700 1 -- 172.21.15.33:0/1113402713 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f00e001a9f0 con 0x7f00e4107aa0 2024-03-19T02:43:16.151 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.370+0000 7f00d77fe700 1 --2- 172.21.15.33:0/1113402713 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f00d0039030 0x7f00d003b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:16.151 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.370+0000 7f00ec6c3700 1 -- 172.21.15.33:0/1113402713 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00e4063940 con 0x7f00e4107aa0 2024-03-19T02:43:16.151 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.370+0000 7f00d77fe700 1 -- 172.21.15.33:0/1113402713 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f00e004d3b0 con 0x7f00e4107aa0 2024-03-19T02:43:16.152 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.370+0000 7f00e9c5e700 1 --2- 172.21.15.33:0/1113402713 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f00d0039030 0x7f00d003b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:16.152 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.370+0000 7f00e9c5e700 1 --2- 172.21.15.33:0/1113402713 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f00d0039030 0x7f00d003b4f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f00d8006fd0 tx=0x7f00d8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:16.152 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.375+0000 7f00d77fe700 1 -- 172.21.15.33:0/1113402713 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f00e001ace0 con 0x7f00e4107aa0 2024-03-19T02:43:16.152 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.584+0000 7f00ec6c3700 1 -- 172.21.15.33:0/1113402713 --> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f00e41041a0 con 0x7f00d0039030 2024-03-19T02:43:16.152 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.585+0000 7f00d77fe700 1 -- 172.21.15.33:0/1113402713 <== mgr.14118 v2:172.21.15.33:6800/906645748 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f00e41041a0 con 0x7f00d0039030 2024-03-19T02:43:16.152 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.586+0000 7f00ec6c3700 1 -- 172.21.15.33:0/1113402713 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f00d0039030 msgr2=0x7f00d003b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:16.152 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.586+0000 7f00ec6c3700 1 --2- 172.21.15.33:0/1113402713 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f00d0039030 0x7f00d003b4f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f00d8006fd0 tx=0x7f00d8006e40 comp rx=0 tx=0).stop 2024-03-19T02:43:16.152 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.586+0000 7f00ec6c3700 1 -- 172.21.15.33:0/1113402713 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00e4107aa0 msgr2=0x7f00e419b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:16.152 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.586+0000 7f00ec6c3700 1 --2- 172.21.15.33:0/1113402713 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00e4107aa0 0x7f00e419b910 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f00e0000c00 tx=0x7f00e0011b00 comp rx=0 tx=0).stop 2024-03-19T02:43:16.152 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.587+0000 7f00ec6c3700 1 -- 172.21.15.33:0/1113402713 shutdown_connections 2024-03-19T02:43:16.152 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.587+0000 7f00ec6c3700 1 --2- 172.21.15.33:0/1113402713 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f00d0039030 0x7f00d003b4f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:16.152 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.587+0000 7f00ec6c3700 1 --2- 172.21.15.33:0/1113402713 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f00e4107aa0 0x7f00e419b910 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:16.152 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.587+0000 7f00ec6c3700 1 -- 172.21.15.33:0/1113402713 >> 172.21.15.33:0/1113402713 conn(0x7f00e4103020 msgr2=0x7f00e4103a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:16.153 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.587+0000 7f00ec6c3700 1 -- 172.21.15.33:0/1113402713 shutdown_connections 2024-03-19T02:43:16.153 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:15.587+0000 7f00ec6c3700 1 -- 172.21.15.33:0/1113402713 wait complete. 2024-03-19T02:43:16.153 INFO:teuthology.orchestra.run.smithi033.stdout:Generating ssh key... 2024-03-19T02:43:17.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:16 smithi033 ceph-mon[31973]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:43:17.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:16 smithi033 ceph-mon[31973]: Generating ssh key... 2024-03-19T02:43:17.494 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:17 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: Generating public/private rsa key pair. 2024-03-19T02:43:17.495 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:17 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: Your identification has been saved in /tmp/tmpg34lyevc/key. 2024-03-19T02:43:17.495 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:17 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: Your public key has been saved in /tmp/tmpg34lyevc/key.pub. 2024-03-19T02:43:17.495 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:17 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: The key fingerprint is: 2024-03-19T02:43:17.495 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:17 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: SHA256:Evx3/WsJJAQ9vyyyvykzPuN7XT0C+oRVFE0QhLkkrfY ceph-2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T02:43:17.495 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:17 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: The key's randomart image is: 2024-03-19T02:43:17.495 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:17 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: +---[RSA 3072]----+ 2024-03-19T02:43:17.495 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:17 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: | .+ =**. | 2024-03-19T02:43:17.495 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:17 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: | . . O . . | 2024-03-19T02:43:17.495 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:17 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: | o = = | 2024-03-19T02:43:17.495 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:17 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: | o o = + | 2024-03-19T02:43:17.495 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:17 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: | . S * * o .| 2024-03-19T02:43:17.495 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:17 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: | . = E = +.| 2024-03-19T02:43:17.495 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:17 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: | = o + +| 2024-03-19T02:43:17.495 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:17 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: | B o.. o.| 2024-03-19T02:43:17.496 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:17 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: | o+X+. .. | 2024-03-19T02:43:17.496 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:17 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: +----[SHA256]-----+ 2024-03-19T02:43:17.761 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.540+0000 7f9559381700 1 Processor -- start 2024-03-19T02:43:17.761 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.540+0000 7f9559381700 1 -- start start 2024-03-19T02:43:17.761 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.540+0000 7f9559381700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9554107a60 0x7f9554107e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:17.761 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.540+0000 7f9559381700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9554108450 con 0x7f9554107a60 2024-03-19T02:43:17.761 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.540+0000 7f9552ffd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9554107a60 0x7f9554107e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:17.761 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.540+0000 7f9552ffd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9554107a60 0x7f9554107e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55422/0 (socket says 172.21.15.33:55422) 2024-03-19T02:43:17.761 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.540+0000 7f9552ffd700 1 -- 172.21.15.33:0/3501504167 learned_addr learned my addr 172.21.15.33:0/3501504167 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:17.761 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.540+0000 7f9552ffd700 1 -- 172.21.15.33:0/3501504167 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9554108c60 con 0x7f9554107a60 2024-03-19T02:43:17.761 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.540+0000 7f9552ffd700 1 --2- 172.21.15.33:0/3501504167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9554107a60 0x7f9554107e80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f95440098a0 tx=0x7f9544009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8619f6b90cc45193 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:17.761 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.541+0000 7f9551ffb700 1 -- 172.21.15.33:0/3501504167 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9544004160 con 0x7f9554107a60 2024-03-19T02:43:17.761 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.541+0000 7f9551ffb700 1 -- 172.21.15.33:0/3501504167 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f9544004300 con 0x7f9554107a60 2024-03-19T02:43:17.762 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.541+0000 7f9551ffb700 1 -- 172.21.15.33:0/3501504167 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9544011a80 con 0x7f9554107a60 2024-03-19T02:43:17.762 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.541+0000 7f9559381700 1 -- 172.21.15.33:0/3501504167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9554107a60 msgr2=0x7f9554107e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:17.762 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.541+0000 7f9559381700 1 --2- 172.21.15.33:0/3501504167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9554107a60 0x7f9554107e80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f95440098a0 tx=0x7f9544009bb0 comp rx=0 tx=0).stop 2024-03-19T02:43:17.762 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.541+0000 7f9559381700 1 -- 172.21.15.33:0/3501504167 shutdown_connections 2024-03-19T02:43:17.762 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.541+0000 7f9559381700 1 --2- 172.21.15.33:0/3501504167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9554107a60 0x7f9554107e80 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:17.762 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.541+0000 7f9559381700 1 -- 172.21.15.33:0/3501504167 >> 172.21.15.33:0/3501504167 conn(0x7f9554103020 msgr2=0x7f9554105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:17.762 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.541+0000 7f9559381700 1 -- 172.21.15.33:0/3501504167 shutdown_connections 2024-03-19T02:43:17.762 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.541+0000 7f9559381700 1 -- 172.21.15.33:0/3501504167 wait complete. 2024-03-19T02:43:17.762 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.542+0000 7f9559381700 1 Processor -- start 2024-03-19T02:43:17.762 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.542+0000 7f9559381700 1 -- start start 2024-03-19T02:43:17.762 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.542+0000 7f9559381700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9554107a60 0x7f955407db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:17.762 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.542+0000 7f9559381700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f955410c010 con 0x7f9554107a60 2024-03-19T02:43:17.762 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.542+0000 7f9552ffd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9554107a60 0x7f955407db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:17.762 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.542+0000 7f9552ffd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9554107a60 0x7f955407db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55424/0 (socket says 172.21.15.33:55424) 2024-03-19T02:43:17.763 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.542+0000 7f9552ffd700 1 -- 172.21.15.33:0/2065080165 learned_addr learned my addr 172.21.15.33:0/2065080165 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:17.763 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.542+0000 7f9552ffd700 1 -- 172.21.15.33:0/2065080165 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f955407e080 con 0x7f9554107a60 2024-03-19T02:43:17.763 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.542+0000 7f9552ffd700 1 --2- 172.21.15.33:0/2065080165 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9554107a60 0x7f955407db40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9544006af0 tx=0x7f9544011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:17.763 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.543+0000 7f953bfff700 1 -- 172.21.15.33:0/2065080165 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f95440039e0 con 0x7f9554107a60 2024-03-19T02:43:17.763 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.543+0000 7f953bfff700 1 -- 172.21.15.33:0/2065080165 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f9544003b80 con 0x7f9554107a60 2024-03-19T02:43:17.763 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.543+0000 7f9559381700 1 -- 172.21.15.33:0/2065080165 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f955407c270 con 0x7f9554107a60 2024-03-19T02:43:17.763 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.543+0000 7f953bfff700 1 -- 172.21.15.33:0/2065080165 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f95440196a0 con 0x7f9554107a60 2024-03-19T02:43:17.763 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.543+0000 7f9559381700 1 -- 172.21.15.33:0/2065080165 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f955407c7b0 con 0x7f9554107a60 2024-03-19T02:43:17.763 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.544+0000 7f953bfff700 1 -- 172.21.15.33:0/2065080165 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f95440199a0 con 0x7f9554107a60 2024-03-19T02:43:17.763 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.544+0000 7f9559381700 1 -- 172.21.15.33:0/2065080165 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9554063940 con 0x7f9554107a60 2024-03-19T02:43:17.763 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.544+0000 7f953bfff700 1 --2- 172.21.15.33:0/2065080165 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f953c038f90 0x7f953c03b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:17.763 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.544+0000 7f953bfff700 1 -- 172.21.15.33:0/2065080165 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f954404d570 con 0x7f9554107a60 2024-03-19T02:43:17.763 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.544+0000 7f95527fc700 1 --2- 172.21.15.33:0/2065080165 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f953c038f90 0x7f953c03b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:17.764 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.544+0000 7f95527fc700 1 --2- 172.21.15.33:0/2065080165 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f953c038f90 0x7f953c03b450 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9548006fd0 tx=0x7f9548006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:17.764 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.549+0000 7f953bfff700 1 -- 172.21.15.33:0/2065080165 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f954402d960 con 0x7f9554107a60 2024-03-19T02:43:17.764 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:16.755+0000 7f9559381700 1 -- 172.21.15.33:0/2065080165 --> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f9554104130 con 0x7f953c038f90 2024-03-19T02:43:17.764 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:17.188+0000 7f953bfff700 1 -- 172.21.15.33:0/2065080165 <== mgr.14118 v2:172.21.15.33:6800/906645748 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f9554104130 con 0x7f953c038f90 2024-03-19T02:43:17.764 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:17.190+0000 7f9559381700 1 -- 172.21.15.33:0/2065080165 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f953c038f90 msgr2=0x7f953c03b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:17.764 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:17.190+0000 7f9559381700 1 --2- 172.21.15.33:0/2065080165 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f953c038f90 0x7f953c03b450 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9548006fd0 tx=0x7f9548006e40 comp rx=0 tx=0).stop 2024-03-19T02:43:17.764 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:17.190+0000 7f9559381700 1 -- 172.21.15.33:0/2065080165 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9554107a60 msgr2=0x7f955407db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:17.764 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:17.190+0000 7f9559381700 1 --2- 172.21.15.33:0/2065080165 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9554107a60 0x7f955407db40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9544006af0 tx=0x7f9544011c40 comp rx=0 tx=0).stop 2024-03-19T02:43:17.764 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:17.190+0000 7f9559381700 1 -- 172.21.15.33:0/2065080165 shutdown_connections 2024-03-19T02:43:17.764 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:17.190+0000 7f9559381700 1 --2- 172.21.15.33:0/2065080165 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f953c038f90 0x7f953c03b450 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:17.764 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:17.190+0000 7f9559381700 1 --2- 172.21.15.33:0/2065080165 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9554107a60 0x7f955407db40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:17.764 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:17.190+0000 7f9559381700 1 -- 172.21.15.33:0/2065080165 >> 172.21.15.33:0/2065080165 conn(0x7f9554103020 msgr2=0x7f9554103a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:17.764 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:17.190+0000 7f9559381700 1 -- 172.21.15.33:0/2065080165 shutdown_connections 2024-03-19T02:43:17.765 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:17.190+0000 7f9559381700 1 -- 172.21.15.33:0/2065080165 wait complete. 2024-03-19T02:43:18.402 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:18 smithi033 ceph-mon[31973]: from='mgr.14118 172.21.15.33:0/3216842381' entity='mgr.x' 2024-03-19T02:43:18.402 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:18 smithi033 ceph-mon[31973]: from='mgr.14118 172.21.15.33:0/3216842381' entity='mgr.x' 2024-03-19T02:43:18.976 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9kiSC/4znDfhLrn3PHj6eqEVvOkdZ3IXVQRDbC420s/zEhdnDmIULUl+Wa0sCPnps1IBvJRpVls8wbIJg6oM7TcD9rvdhd4PJlOn5+UJMIPieTWM1hFaVcdvChj8XveijjD7V9R4pd78joeIMErB2ggVkiq4lwZTYVLMA8ppmEouuWzza2ldIC6n4X0ietPTSwv28Wl+JtxCN48g2x92T4ljLoICxNQ9LpJgkINshU6KU74iT+sxCRtZr50fUZGM8D4Mi9uKrLpWqiuzuvsIxsvap2Fod8XppxSSgOnJZPtg6ruYw6qaxsD5U5TUpzfadFIplheNgUUPArFE+bI3aA5XSJXBski4e++jhoxBHMvUpVKrPFofexxxe45MEYdxYSE33GKQO8IAcwLCO82iwtMhZ26+PGJIUZgsa6FwAd/Mtk3pBtSFCT/8MMAf9XExIMfzXHCM8xfWwPWcmEXn/EI5TYDZOgyQcEvsdYWe6vzD/zaMElcevSISszNbfGcU= ceph-2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T02:43:18.976 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.183+0000 7fb9ef46c700 1 Processor -- start 2024-03-19T02:43:18.976 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.184+0000 7fb9ef46c700 1 -- start start 2024-03-19T02:43:18.977 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.184+0000 7fb9ef46c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9e8106380 0x7fb9e81067a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:18.977 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.184+0000 7fb9ef46c700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e807be00 con 0x7fb9e8106380 2024-03-19T02:43:18.977 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.184+0000 7fb9ed208700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9e8106380 0x7fb9e81067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:18.977 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.184+0000 7fb9ed208700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9e8106380 0x7fb9e81067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55434/0 (socket says 172.21.15.33:55434) 2024-03-19T02:43:18.977 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.184+0000 7fb9ed208700 1 -- 172.21.15.33:0/489061952 learned_addr learned my addr 172.21.15.33:0/489061952 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:18.977 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.184+0000 7fb9ed208700 1 -- 172.21.15.33:0/489061952 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9e8106ce0 con 0x7fb9e8106380 2024-03-19T02:43:18.977 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.185+0000 7fb9ed208700 1 --2- 172.21.15.33:0/489061952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9e8106380 0x7fb9e81067a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb9e40098a0 tx=0x7fb9e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c35e0887178ee4e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:18.977 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.186+0000 7fb9dbfff700 1 -- 172.21.15.33:0/489061952 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9e4004160 con 0x7fb9e8106380 2024-03-19T02:43:18.977 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.186+0000 7fb9dbfff700 1 -- 172.21.15.33:0/489061952 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fb9e4004300 con 0x7fb9e8106380 2024-03-19T02:43:18.977 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.186+0000 7fb9dbfff700 1 -- 172.21.15.33:0/489061952 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9e4011a80 con 0x7fb9e8106380 2024-03-19T02:43:18.977 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.187+0000 7fb9ef46c700 1 -- 172.21.15.33:0/489061952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9e8106380 msgr2=0x7fb9e81067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:18.977 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.187+0000 7fb9ef46c700 1 --2- 172.21.15.33:0/489061952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9e8106380 0x7fb9e81067a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb9e40098a0 tx=0x7fb9e4009bb0 comp rx=0 tx=0).stop 2024-03-19T02:43:18.978 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.187+0000 7fb9ef46c700 1 -- 172.21.15.33:0/489061952 shutdown_connections 2024-03-19T02:43:18.978 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.187+0000 7fb9ef46c700 1 --2- 172.21.15.33:0/489061952 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9e8106380 0x7fb9e81067a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:18.978 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.187+0000 7fb9ef46c700 1 -- 172.21.15.33:0/489061952 >> 172.21.15.33:0/489061952 conn(0x7fb9e8101f80 msgr2=0x7fb9e81043a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:18.978 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.187+0000 7fb9ef46c700 1 -- 172.21.15.33:0/489061952 shutdown_connections 2024-03-19T02:43:18.978 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.187+0000 7fb9ef46c700 1 -- 172.21.15.33:0/489061952 wait complete. 2024-03-19T02:43:18.978 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.188+0000 7fb9ef46c700 1 Processor -- start 2024-03-19T02:43:18.978 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.188+0000 7fb9ef46c700 1 -- start start 2024-03-19T02:43:18.978 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.188+0000 7fb9ef46c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9e8106380 0x7fb9e819db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:18.978 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.188+0000 7fb9ef46c700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e8107d10 con 0x7fb9e8106380 2024-03-19T02:43:18.978 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.188+0000 7fb9ed208700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9e8106380 0x7fb9e819db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:18.978 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.188+0000 7fb9ed208700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9e8106380 0x7fb9e819db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55446/0 (socket says 172.21.15.33:55446) 2024-03-19T02:43:18.978 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.188+0000 7fb9ed208700 1 -- 172.21.15.33:0/634662532 learned_addr learned my addr 172.21.15.33:0/634662532 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:18.978 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.188+0000 7fb9ed208700 1 -- 172.21.15.33:0/634662532 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9e819e0a0 con 0x7fb9e8106380 2024-03-19T02:43:18.979 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.188+0000 7fb9ed208700 1 --2- 172.21.15.33:0/634662532 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9e8106380 0x7fb9e819db60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb9e4011a50 tx=0x7fb9e4004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:18.979 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.188+0000 7fb9da7fc700 1 -- 172.21.15.33:0/634662532 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9e40039f0 con 0x7fb9e8106380 2024-03-19T02:43:18.979 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.189+0000 7fb9da7fc700 1 -- 172.21.15.33:0/634662532 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fb9e4003b90 con 0x7fb9e8106380 2024-03-19T02:43:18.979 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.189+0000 7fb9ef46c700 1 -- 172.21.15.33:0/634662532 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9e819e330 con 0x7fb9e8106380 2024-03-19T02:43:18.979 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.189+0000 7fb9da7fc700 1 -- 172.21.15.33:0/634662532 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9e401a600 con 0x7fb9e8106380 2024-03-19T02:43:18.979 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.189+0000 7fb9ef46c700 1 -- 172.21.15.33:0/634662532 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9e81a1020 con 0x7fb9e8106380 2024-03-19T02:43:18.979 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.189+0000 7fb9da7fc700 1 -- 172.21.15.33:0/634662532 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fb9e401a8d0 con 0x7fb9e8106380 2024-03-19T02:43:18.979 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.189+0000 7fb9ef46c700 1 -- 172.21.15.33:0/634662532 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9e8063940 con 0x7fb9e8106380 2024-03-19T02:43:18.980 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.190+0000 7fb9da7fc700 1 --2- 172.21.15.33:0/634662532 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7fb9d4039030 0x7fb9d403b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:18.980 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.190+0000 7fb9da7fc700 1 -- 172.21.15.33:0/634662532 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb9e404d420 con 0x7fb9e8106380 2024-03-19T02:43:18.980 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.190+0000 7fb9eca07700 1 --2- 172.21.15.33:0/634662532 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7fb9d4039030 0x7fb9d403b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:18.980 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.190+0000 7fb9eca07700 1 --2- 172.21.15.33:0/634662532 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7fb9d4039030 0x7fb9d403b4f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc006fd0 tx=0x7fb9dc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:18.980 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.195+0000 7fb9da7fc700 1 -- 172.21.15.33:0/634662532 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb9e4024a40 con 0x7fb9e8106380 2024-03-19T02:43:18.980 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.401+0000 7fb9ef46c700 1 -- 172.21.15.33:0/634662532 --> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fb9e80008d0 con 0x7fb9d4039030 2024-03-19T02:43:18.980 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.402+0000 7fb9da7fc700 1 -- 172.21.15.33:0/634662532 <== mgr.14118 v2:172.21.15.33:6800/906645748 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fb9e80008d0 con 0x7fb9d4039030 2024-03-19T02:43:18.980 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.404+0000 7fb9ef46c700 1 -- 172.21.15.33:0/634662532 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7fb9d4039030 msgr2=0x7fb9d403b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:18.980 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.404+0000 7fb9ef46c700 1 --2- 172.21.15.33:0/634662532 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7fb9d4039030 0x7fb9d403b4f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc006fd0 tx=0x7fb9dc006e40 comp rx=0 tx=0).stop 2024-03-19T02:43:18.980 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.404+0000 7fb9ef46c700 1 -- 172.21.15.33:0/634662532 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9e8106380 msgr2=0x7fb9e819db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:18.980 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.404+0000 7fb9ef46c700 1 --2- 172.21.15.33:0/634662532 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9e8106380 0x7fb9e819db60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb9e4011a50 tx=0x7fb9e4004290 comp rx=0 tx=0).stop 2024-03-19T02:43:18.980 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.404+0000 7fb9ef46c700 1 -- 172.21.15.33:0/634662532 shutdown_connections 2024-03-19T02:43:18.980 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.404+0000 7fb9ef46c700 1 --2- 172.21.15.33:0/634662532 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7fb9d4039030 0x7fb9d403b4f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:18.981 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.404+0000 7fb9ef46c700 1 --2- 172.21.15.33:0/634662532 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9e8106380 0x7fb9e819db60 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:18.981 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.404+0000 7fb9ef46c700 1 -- 172.21.15.33:0/634662532 >> 172.21.15.33:0/634662532 conn(0x7fb9e8101f80 msgr2=0x7fb9e81947a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:18.981 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.404+0000 7fb9ef46c700 1 -- 172.21.15.33:0/634662532 shutdown_connections 2024-03-19T02:43:18.981 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:18.404+0000 7fb9ef46c700 1 -- 172.21.15.33:0/634662532 wait complete. 2024-03-19T02:43:18.981 INFO:teuthology.orchestra.run.smithi033.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-19T02:43:18.981 INFO:teuthology.orchestra.run.smithi033.stdout:Adding key to root@localhost authorized_keys... 2024-03-19T02:43:18.981 INFO:teuthology.orchestra.run.smithi033.stdout:Adding host smithi033... 2024-03-19T02:43:19.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:19 smithi033 ceph-mon[31973]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:43:20.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:20 smithi033 ceph-mon[31973]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi033", "addr": "172.21.15.33", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:43:21.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:21 smithi033 ceph-mon[31973]: Deploying cephadm binary to smithi033 2024-03-19T02:43:24.510 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout Added host 'smithi033' with addr '172.21.15.33' 2024-03-19T02:43:24.510 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.400+0000 7f1b4d858700 1 Processor -- start 2024-03-19T02:43:24.510 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.400+0000 7f1b4d858700 1 -- start start 2024-03-19T02:43:24.510 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.400+0000 7f1b4d858700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b48107aa0 0x7f1b48107ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:24.510 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.400+0000 7f1b4d858700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1b48108490 con 0x7f1b48107aa0 2024-03-19T02:43:24.510 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.400+0000 7f1b46ffd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b48107aa0 0x7f1b48107ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:24.510 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.400+0000 7f1b46ffd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b48107aa0 0x7f1b48107ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55452/0 (socket says 172.21.15.33:55452) 2024-03-19T02:43:24.510 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.400+0000 7f1b46ffd700 1 -- 172.21.15.33:0/1227847915 learned_addr learned my addr 172.21.15.33:0/1227847915 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:24.510 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.401+0000 7f1b46ffd700 1 -- 172.21.15.33:0/1227847915 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b48108ca0 con 0x7f1b48107aa0 2024-03-19T02:43:24.510 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.401+0000 7f1b46ffd700 1 --2- 172.21.15.33:0/1227847915 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b48107aa0 0x7f1b48107ec0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1b38009e80 tx=0x7f1b3800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fedbfb023d0cadc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:24.510 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.401+0000 7f1b45ffb700 1 -- 172.21.15.33:0/1227847915 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b38017070 con 0x7f1b48107aa0 2024-03-19T02:43:24.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.401+0000 7f1b45ffb700 1 -- 172.21.15.33:0/1227847915 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f1b3800b8c0 con 0x7f1b48107aa0 2024-03-19T02:43:24.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.401+0000 7f1b45ffb700 1 -- 172.21.15.33:0/1227847915 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b38004520 con 0x7f1b48107aa0 2024-03-19T02:43:24.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.402+0000 7f1b4d858700 1 -- 172.21.15.33:0/1227847915 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b48107aa0 msgr2=0x7f1b48107ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:24.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.402+0000 7f1b4d858700 1 --2- 172.21.15.33:0/1227847915 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b48107aa0 0x7f1b48107ec0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1b38009e80 tx=0x7f1b3800b330 comp rx=0 tx=0).stop 2024-03-19T02:43:24.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.402+0000 7f1b4d858700 1 -- 172.21.15.33:0/1227847915 shutdown_connections 2024-03-19T02:43:24.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.402+0000 7f1b4d858700 1 --2- 172.21.15.33:0/1227847915 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b48107aa0 0x7f1b48107ec0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:24.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.402+0000 7f1b4d858700 1 -- 172.21.15.33:0/1227847915 >> 172.21.15.33:0/1227847915 conn(0x7f1b48103020 msgr2=0x7f1b48105480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:24.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.402+0000 7f1b4d858700 1 -- 172.21.15.33:0/1227847915 shutdown_connections 2024-03-19T02:43:24.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.402+0000 7f1b4d858700 1 -- 172.21.15.33:0/1227847915 wait complete. 2024-03-19T02:43:24.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.402+0000 7f1b4d858700 1 Processor -- start 2024-03-19T02:43:24.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.403+0000 7f1b4d858700 1 -- start start 2024-03-19T02:43:24.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.403+0000 7f1b4d858700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b48107aa0 0x7f1b4819b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:24.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.403+0000 7f1b4d858700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1b4810c050 con 0x7f1b48107aa0 2024-03-19T02:43:24.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.403+0000 7f1b46ffd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b48107aa0 0x7f1b4819b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:24.511 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.403+0000 7f1b46ffd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b48107aa0 0x7f1b4819b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55456/0 (socket says 172.21.15.33:55456) 2024-03-19T02:43:24.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.403+0000 7f1b46ffd700 1 -- 172.21.15.33:0/3954806459 learned_addr learned my addr 172.21.15.33:0/3954806459 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:24.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.403+0000 7f1b46ffd700 1 -- 172.21.15.33:0/3954806459 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b4819be00 con 0x7f1b48107aa0 2024-03-19T02:43:24.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.403+0000 7f1b46ffd700 1 --2- 172.21.15.33:0/3954806459 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b48107aa0 0x7f1b4819b8c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1b38009e50 tx=0x7f1b38004670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:24.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.403+0000 7f1b4c856700 1 -- 172.21.15.33:0/3954806459 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b38020020 con 0x7f1b48107aa0 2024-03-19T02:43:24.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.403+0000 7f1b4c856700 1 -- 172.21.15.33:0/3954806459 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f1b3800bba0 con 0x7f1b48107aa0 2024-03-19T02:43:24.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.403+0000 7f1b4c856700 1 -- 172.21.15.33:0/3954806459 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b38017040 con 0x7f1b48107aa0 2024-03-19T02:43:24.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.403+0000 7f1b4d858700 1 -- 172.21.15.33:0/3954806459 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b4819c090 con 0x7f1b48107aa0 2024-03-19T02:43:24.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.404+0000 7f1b4d858700 1 -- 172.21.15.33:0/3954806459 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b4819ed80 con 0x7f1b48107aa0 2024-03-19T02:43:24.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.404+0000 7f1b4c856700 1 -- 172.21.15.33:0/3954806459 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f1b38023020 con 0x7f1b48107aa0 2024-03-19T02:43:24.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.405+0000 7f1b4c856700 1 --2- 172.21.15.33:0/3954806459 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f1b34038f40 0x7f1b3403b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:24.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.405+0000 7f1b4d858700 1 -- 172.21.15.33:0/3954806459 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b48063940 con 0x7f1b48107aa0 2024-03-19T02:43:24.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.405+0000 7f1b4c856700 1 -- 172.21.15.33:0/3954806459 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1b3800f050 con 0x7f1b48107aa0 2024-03-19T02:43:24.512 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.405+0000 7f1b467fc700 1 --2- 172.21.15.33:0/3954806459 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f1b34038f40 0x7f1b3403b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:24.513 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.405+0000 7f1b467fc700 1 --2- 172.21.15.33:0/3954806459 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f1b34038f40 0x7f1b3403b400 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1b30006fd0 tx=0x7f1b30006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:24.513 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.410+0000 7f1b4c856700 1 -- 172.21.15.33:0/3954806459 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1b3802a020 con 0x7f1b48107aa0 2024-03-19T02:43:24.513 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:19.618+0000 7f1b4d858700 1 -- 172.21.15.33:0/3954806459 --> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi033", "addr": "172.21.15.33", "target": ["mon-mgr", ""]}) -- 0x7f1b48104120 con 0x7f1b34038f40 2024-03-19T02:43:24.513 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:23.932+0000 7f1b4c856700 1 -- 172.21.15.33:0/3954806459 <== mgr.14118 v2:172.21.15.33:6800/906645748 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f1b48104120 con 0x7f1b34038f40 2024-03-19T02:43:24.513 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:23.935+0000 7f1b4d858700 1 -- 172.21.15.33:0/3954806459 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f1b34038f40 msgr2=0x7f1b3403b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:24.513 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:23.935+0000 7f1b4d858700 1 --2- 172.21.15.33:0/3954806459 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f1b34038f40 0x7f1b3403b400 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1b30006fd0 tx=0x7f1b30006e40 comp rx=0 tx=0).stop 2024-03-19T02:43:24.513 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:23.935+0000 7f1b4d858700 1 -- 172.21.15.33:0/3954806459 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b48107aa0 msgr2=0x7f1b4819b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:24.513 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:23.935+0000 7f1b4d858700 1 --2- 172.21.15.33:0/3954806459 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b48107aa0 0x7f1b4819b8c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1b38009e50 tx=0x7f1b38004670 comp rx=0 tx=0).stop 2024-03-19T02:43:24.513 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:23.935+0000 7f1b4d858700 1 -- 172.21.15.33:0/3954806459 shutdown_connections 2024-03-19T02:43:24.513 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:23.935+0000 7f1b4d858700 1 --2- 172.21.15.33:0/3954806459 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f1b34038f40 0x7f1b3403b400 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:24.513 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:23.935+0000 7f1b4d858700 1 --2- 172.21.15.33:0/3954806459 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1b48107aa0 0x7f1b4819b8c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:24.513 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:23.935+0000 7f1b4d858700 1 -- 172.21.15.33:0/3954806459 >> 172.21.15.33:0/3954806459 conn(0x7f1b48103020 msgr2=0x7f1b48103a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:24.513 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:23.935+0000 7f1b4d858700 1 -- 172.21.15.33:0/3954806459 shutdown_connections 2024-03-19T02:43:24.513 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:23.935+0000 7f1b4d858700 1 -- 172.21.15.33:0/3954806459 wait complete. 2024-03-19T02:43:24.513 INFO:teuthology.orchestra.run.smithi033.stdout:Deploying unmanaged mon service... 2024-03-19T02:43:25.162 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:24 smithi033 ceph-mon[31973]: from='mgr.14118 172.21.15.33:0/3216842381' entity='mgr.x' 2024-03-19T02:43:25.163 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:24 smithi033 ceph-mon[31973]: Added host smithi033 2024-03-19T02:43:25.163 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:24 smithi033 ceph-mon[31973]: from='mgr.14118 172.21.15.33:0/3216842381' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:43:25.828 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-19T02:43:25.828 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.933+0000 7f6942cb5700 1 Processor -- start 2024-03-19T02:43:25.828 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.933+0000 7f6942cb5700 1 -- start start 2024-03-19T02:43:25.828 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.934+0000 7f6942cb5700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f693c1069b0 0x7f693c106dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:25.829 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.934+0000 7f6942cb5700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f693c1073a0 con 0x7f693c1069b0 2024-03-19T02:43:25.829 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.934+0000 7f6940a51700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f693c1069b0 0x7f693c106dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:25.829 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.934+0000 7f6940a51700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f693c1069b0 0x7f693c106dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50754/0 (socket says 172.21.15.33:50754) 2024-03-19T02:43:25.829 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.934+0000 7f6940a51700 1 -- 172.21.15.33:0/4293834867 learned_addr learned my addr 172.21.15.33:0/4293834867 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:25.829 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.934+0000 7f6940a51700 1 -- 172.21.15.33:0/4293834867 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f693c107bb0 con 0x7f693c1069b0 2024-03-19T02:43:25.829 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.934+0000 7f6940a51700 1 --2- 172.21.15.33:0/4293834867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f693c1069b0 0x7f693c106dd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f69300098a0 tx=0x7f6930009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79db1d18d93bf977 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:25.829 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.934+0000 7f693b7fe700 1 -- 172.21.15.33:0/4293834867 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6930004160 con 0x7f693c1069b0 2024-03-19T02:43:25.829 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.934+0000 7f693b7fe700 1 -- 172.21.15.33:0/4293834867 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f6930004300 con 0x7f693c1069b0 2024-03-19T02:43:25.829 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.934+0000 7f693b7fe700 1 -- 172.21.15.33:0/4293834867 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6930011a80 con 0x7f693c1069b0 2024-03-19T02:43:25.829 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.935+0000 7f6942cb5700 1 -- 172.21.15.33:0/4293834867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f693c1069b0 msgr2=0x7f693c106dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:25.829 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.935+0000 7f6942cb5700 1 --2- 172.21.15.33:0/4293834867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f693c1069b0 0x7f693c106dd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f69300098a0 tx=0x7f6930009bb0 comp rx=0 tx=0).stop 2024-03-19T02:43:25.829 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.935+0000 7f6942cb5700 1 -- 172.21.15.33:0/4293834867 shutdown_connections 2024-03-19T02:43:25.829 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.935+0000 7f6942cb5700 1 --2- 172.21.15.33:0/4293834867 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f693c1069b0 0x7f693c106dd0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:25.829 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.935+0000 7f6942cb5700 1 -- 172.21.15.33:0/4293834867 >> 172.21.15.33:0/4293834867 conn(0x7f693c101fb0 msgr2=0x7f693c1043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:25.829 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.935+0000 7f6942cb5700 1 -- 172.21.15.33:0/4293834867 shutdown_connections 2024-03-19T02:43:25.829 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.935+0000 7f6942cb5700 1 -- 172.21.15.33:0/4293834867 wait complete. 2024-03-19T02:43:25.830 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.936+0000 7f6942cb5700 1 Processor -- start 2024-03-19T02:43:25.830 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.936+0000 7f6942cb5700 1 -- start start 2024-03-19T02:43:25.830 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.936+0000 7f6942cb5700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f693c1069b0 0x7f693c104360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:25.830 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.936+0000 7f6942cb5700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f693c10af60 con 0x7f693c1069b0 2024-03-19T02:43:25.830 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.936+0000 7f6940a51700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f693c1069b0 0x7f693c104360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:25.830 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.936+0000 7f6940a51700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f693c1069b0 0x7f693c104360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50766/0 (socket says 172.21.15.33:50766) 2024-03-19T02:43:25.830 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.936+0000 7f6940a51700 1 -- 172.21.15.33:0/186745324 learned_addr learned my addr 172.21.15.33:0/186745324 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:25.830 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.936+0000 7f6940a51700 1 -- 172.21.15.33:0/186745324 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f693c1048a0 con 0x7f693c1069b0 2024-03-19T02:43:25.830 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.936+0000 7f6940a51700 1 --2- 172.21.15.33:0/186745324 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f693c1069b0 0x7f693c104360 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6930006af0 tx=0x7f6930011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:25.830 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.937+0000 7f6939ffb700 1 -- 172.21.15.33:0/186745324 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f69300039e0 con 0x7f693c1069b0 2024-03-19T02:43:25.830 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.937+0000 7f6939ffb700 1 -- 172.21.15.33:0/186745324 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f6930003b80 con 0x7f693c1069b0 2024-03-19T02:43:25.830 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.937+0000 7f6942cb5700 1 -- 172.21.15.33:0/186745324 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f693c102a90 con 0x7f693c1069b0 2024-03-19T02:43:25.830 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.937+0000 7f6939ffb700 1 -- 172.21.15.33:0/186745324 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f69300196a0 con 0x7f693c1069b0 2024-03-19T02:43:25.830 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.937+0000 7f6942cb5700 1 -- 172.21.15.33:0/186745324 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f693c102fd0 con 0x7f693c1069b0 2024-03-19T02:43:25.830 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.938+0000 7f6939ffb700 1 -- 172.21.15.33:0/186745324 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f6930019950 con 0x7f693c1069b0 2024-03-19T02:43:25.831 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.938+0000 7f6942cb5700 1 -- 172.21.15.33:0/186745324 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f693c063940 con 0x7f693c1069b0 2024-03-19T02:43:25.831 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.938+0000 7f6939ffb700 1 --2- 172.21.15.33:0/186745324 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f692c038fe0 0x7f692c03b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:25.831 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.938+0000 7f6939ffb700 1 -- 172.21.15.33:0/186745324 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f693004d3e0 con 0x7f693c1069b0 2024-03-19T02:43:25.831 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.938+0000 7f693bfff700 1 --2- 172.21.15.33:0/186745324 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f692c038fe0 0x7f692c03b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:25.831 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.938+0000 7f693bfff700 1 --2- 172.21.15.33:0/186745324 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f692c038fe0 0x7f692c03b4a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6928006fd0 tx=0x7f6928006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:25.831 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:24.943+0000 7f6939ffb700 1 -- 172.21.15.33:0/186745324 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6930019d40 con 0x7f693c1069b0 2024-03-19T02:43:25.831 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:25.162+0000 7f6942cb5700 1 -- 172.21.15.33:0/186745324 --> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f693c195270 con 0x7f692c038fe0 2024-03-19T02:43:25.831 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:25.166+0000 7f6939ffb700 1 -- 172.21.15.33:0/186745324 <== mgr.14118 v2:172.21.15.33:6800/906645748 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f693c195270 con 0x7f692c038fe0 2024-03-19T02:43:25.831 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:25.168+0000 7f6942cb5700 1 -- 172.21.15.33:0/186745324 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f692c038fe0 msgr2=0x7f692c03b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:25.831 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:25.168+0000 7f6942cb5700 1 --2- 172.21.15.33:0/186745324 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f692c038fe0 0x7f692c03b4a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6928006fd0 tx=0x7f6928006e40 comp rx=0 tx=0).stop 2024-03-19T02:43:25.831 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:25.168+0000 7f6942cb5700 1 -- 172.21.15.33:0/186745324 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f693c1069b0 msgr2=0x7f693c104360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:25.831 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:25.168+0000 7f6942cb5700 1 --2- 172.21.15.33:0/186745324 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f693c1069b0 0x7f693c104360 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6930006af0 tx=0x7f6930011c40 comp rx=0 tx=0).stop 2024-03-19T02:43:25.831 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:25.168+0000 7f6942cb5700 1 -- 172.21.15.33:0/186745324 shutdown_connections 2024-03-19T02:43:25.831 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:25.168+0000 7f6942cb5700 1 --2- 172.21.15.33:0/186745324 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f692c038fe0 0x7f692c03b4a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:25.832 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:25.168+0000 7f6942cb5700 1 --2- 172.21.15.33:0/186745324 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f693c1069b0 0x7f693c104360 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:25.832 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:25.168+0000 7f6942cb5700 1 -- 172.21.15.33:0/186745324 >> 172.21.15.33:0/186745324 conn(0x7f693c101fb0 msgr2=0x7f693c1924f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:25.832 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:25.168+0000 7f6942cb5700 1 -- 172.21.15.33:0/186745324 shutdown_connections 2024-03-19T02:43:25.832 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:25.168+0000 7f6942cb5700 1 -- 172.21.15.33:0/186745324 wait complete. 2024-03-19T02:43:25.832 INFO:teuthology.orchestra.run.smithi033.stdout:Deploying unmanaged mgr service... 2024-03-19T02:43:26.366 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:26 smithi033 ceph-mon[31973]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:43:26.366 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:26 smithi033 ceph-mon[31973]: Saving service mon spec with placement count:5 2024-03-19T02:43:26.366 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:26 smithi033 ceph-mon[31973]: from='mgr.14118 172.21.15.33:0/3216842381' entity='mgr.x' 2024-03-19T02:43:27.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:27 smithi033 ceph-mon[31973]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:43:27.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:27 smithi033 ceph-mon[31973]: Saving service mgr spec with placement count:2 2024-03-19T02:43:27.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:27 smithi033 ceph-mon[31973]: from='mgr.14118 172.21.15.33:0/3216842381' entity='mgr.x' 2024-03-19T02:43:27.958 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-19T02:43:27.958 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.217+0000 7f4cebf62700 1 Processor -- start 2024-03-19T02:43:27.958 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.217+0000 7f4cebf62700 1 -- start start 2024-03-19T02:43:27.958 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.218+0000 7f4cebf62700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ce4106ea0 0x7f4ce41072c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:27.958 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.218+0000 7f4cebf62700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce4107890 con 0x7f4ce4106ea0 2024-03-19T02:43:27.958 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.218+0000 7f4ce9cfe700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ce4106ea0 0x7f4ce41072c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:27.958 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.218+0000 7f4ce9cfe700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ce4106ea0 0x7f4ce41072c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50780/0 (socket says 172.21.15.33:50780) 2024-03-19T02:43:27.958 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.218+0000 7f4ce9cfe700 1 -- 172.21.15.33:0/592078149 learned_addr learned my addr 172.21.15.33:0/592078149 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:27.958 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.218+0000 7f4ce9cfe700 1 -- 172.21.15.33:0/592078149 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ce4107bc0 con 0x7f4ce4106ea0 2024-03-19T02:43:27.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.218+0000 7f4ce9cfe700 1 --2- 172.21.15.33:0/592078149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ce4106ea0 0x7f4ce41072c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4009e80 tx=0x7f4cd400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=377845e2948a261 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:27.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.218+0000 7f4ce8cfc700 1 -- 172.21.15.33:0/592078149 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4cd4017070 con 0x7f4ce4106ea0 2024-03-19T02:43:27.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.218+0000 7f4ce8cfc700 1 -- 172.21.15.33:0/592078149 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f4cd400b8c0 con 0x7f4ce4106ea0 2024-03-19T02:43:27.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.218+0000 7f4ce8cfc700 1 -- 172.21.15.33:0/592078149 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4cd4004520 con 0x7f4ce4106ea0 2024-03-19T02:43:27.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.219+0000 7f4cebf62700 1 -- 172.21.15.33:0/592078149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ce4106ea0 msgr2=0x7f4ce41072c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:27.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.219+0000 7f4cebf62700 1 --2- 172.21.15.33:0/592078149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ce4106ea0 0x7f4ce41072c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4009e80 tx=0x7f4cd400b330 comp rx=0 tx=0).stop 2024-03-19T02:43:27.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.219+0000 7f4cebf62700 1 -- 172.21.15.33:0/592078149 shutdown_connections 2024-03-19T02:43:27.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.219+0000 7f4cebf62700 1 --2- 172.21.15.33:0/592078149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ce4106ea0 0x7f4ce41072c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:27.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.219+0000 7f4cebf62700 1 -- 172.21.15.33:0/592078149 >> 172.21.15.33:0/592078149 conn(0x7f4ce4101720 msgr2=0x7f4ce4103b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:27.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.219+0000 7f4cebf62700 1 -- 172.21.15.33:0/592078149 shutdown_connections 2024-03-19T02:43:27.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.219+0000 7f4cebf62700 1 -- 172.21.15.33:0/592078149 wait complete. 2024-03-19T02:43:27.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.220+0000 7f4cebf62700 1 Processor -- start 2024-03-19T02:43:27.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.220+0000 7f4cebf62700 1 -- start start 2024-03-19T02:43:27.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.220+0000 7f4cebf62700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ce4106ea0 0x7f4ce419b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:27.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.220+0000 7f4cebf62700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce410af70 con 0x7f4ce4106ea0 2024-03-19T02:43:27.959 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.220+0000 7f4ce9cfe700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ce4106ea0 0x7f4ce419b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:27.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.220+0000 7f4ce9cfe700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ce4106ea0 0x7f4ce419b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50794/0 (socket says 172.21.15.33:50794) 2024-03-19T02:43:27.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.220+0000 7f4ce9cfe700 1 -- 172.21.15.33:0/1635861310 learned_addr learned my addr 172.21.15.33:0/1635861310 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:27.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.220+0000 7f4ce9cfe700 1 -- 172.21.15.33:0/1635861310 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ce419bdf0 con 0x7f4ce4106ea0 2024-03-19T02:43:27.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.220+0000 7f4ce9cfe700 1 --2- 172.21.15.33:0/1635861310 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ce4106ea0 0x7f4ce419b8b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4009e50 tx=0x7f4cd4004670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:27.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.221+0000 7f4ce2ffd700 1 -- 172.21.15.33:0/1635861310 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4cd4020020 con 0x7f4ce4106ea0 2024-03-19T02:43:27.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.221+0000 7f4ce2ffd700 1 -- 172.21.15.33:0/1635861310 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f4cd400bba0 con 0x7f4ce4106ea0 2024-03-19T02:43:27.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.221+0000 7f4cebf62700 1 -- 172.21.15.33:0/1635861310 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ce419c080 con 0x7f4ce4106ea0 2024-03-19T02:43:27.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.221+0000 7f4ce2ffd700 1 -- 172.21.15.33:0/1635861310 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4cd4017040 con 0x7f4ce4106ea0 2024-03-19T02:43:27.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.221+0000 7f4cebf62700 1 -- 172.21.15.33:0/1635861310 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ce419ed20 con 0x7f4ce4106ea0 2024-03-19T02:43:27.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.221+0000 7f4ce2ffd700 1 -- 172.21.15.33:0/1635861310 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f4cd4023020 con 0x7f4ce4106ea0 2024-03-19T02:43:27.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.222+0000 7f4cebf62700 1 -- 172.21.15.33:0/1635861310 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ce4063940 con 0x7f4ce4106ea0 2024-03-19T02:43:27.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.222+0000 7f4ce2ffd700 1 --2- 172.21.15.33:0/1635861310 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f4ccc038fe0 0x7f4ccc03b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:27.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.222+0000 7f4ce2ffd700 1 -- 172.21.15.33:0/1635861310 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4cd400f050 con 0x7f4ce4106ea0 2024-03-19T02:43:27.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.222+0000 7f4ce94fd700 1 --2- 172.21.15.33:0/1635861310 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f4ccc038fe0 0x7f4ccc03b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:27.960 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.222+0000 7f4ce94fd700 1 --2- 172.21.15.33:0/1635861310 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f4ccc038fe0 0x7f4ccc03b4a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8006fd0 tx=0x7f4cd8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:27.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.227+0000 7f4ce2ffd700 1 -- 172.21.15.33:0/1635861310 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4cd402a020 con 0x7f4ce4106ea0 2024-03-19T02:43:27.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.448+0000 7f4cebf62700 1 -- 172.21.15.33:0/1635861310 --> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f4ce402df90 con 0x7f4ccc038fe0 2024-03-19T02:43:27.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.455+0000 7f4ce2ffd700 1 -- 172.21.15.33:0/1635861310 <== mgr.14118 v2:172.21.15.33:6800/906645748 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f4ce402df90 con 0x7f4ccc038fe0 2024-03-19T02:43:27.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.457+0000 7f4cebf62700 1 -- 172.21.15.33:0/1635861310 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f4ccc038fe0 msgr2=0x7f4ccc03b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:27.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.457+0000 7f4cebf62700 1 --2- 172.21.15.33:0/1635861310 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f4ccc038fe0 0x7f4ccc03b4a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8006fd0 tx=0x7f4cd8006e40 comp rx=0 tx=0).stop 2024-03-19T02:43:27.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.457+0000 7f4cebf62700 1 -- 172.21.15.33:0/1635861310 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ce4106ea0 msgr2=0x7f4ce419b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:27.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.457+0000 7f4cebf62700 1 --2- 172.21.15.33:0/1635861310 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ce4106ea0 0x7f4ce419b8b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4009e50 tx=0x7f4cd4004670 comp rx=0 tx=0).stop 2024-03-19T02:43:27.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.457+0000 7f4cebf62700 1 -- 172.21.15.33:0/1635861310 shutdown_connections 2024-03-19T02:43:27.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.458+0000 7f4cebf62700 1 --2- 172.21.15.33:0/1635861310 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f4ccc038fe0 0x7f4ccc03b4a0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:27.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.458+0000 7f4cebf62700 1 --2- 172.21.15.33:0/1635861310 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ce4106ea0 0x7f4ce419b8b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:27.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.458+0000 7f4cebf62700 1 -- 172.21.15.33:0/1635861310 >> 172.21.15.33:0/1635861310 conn(0x7f4ce4101720 msgr2=0x7f4ce4192510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:27.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.458+0000 7f4cebf62700 1 -- 172.21.15.33:0/1635861310 shutdown_connections 2024-03-19T02:43:27.961 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:26.458+0000 7f4cebf62700 1 -- 172.21.15.33:0/1635861310 wait complete. 2024-03-19T02:43:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:28 smithi033 ceph-mon[31973]: from='mgr.14118 172.21.15.33:0/3216842381' entity='mgr.x' 2024-03-19T02:43:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:28 smithi033 ceph-mon[31973]: from='mgr.14118 172.21.15.33:0/3216842381' entity='mgr.x' 2024-03-19T02:43:29.911 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:29 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3524706526' entity='client.admin' 2024-03-19T02:43:29.911 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:29 smithi033 ceph-mon[31973]: from='mgr.14118 172.21.15.33:0/3216842381' entity='mgr.x' 2024-03-19T02:43:29.918 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.372+0000 7f7e8351b700 1 Processor -- start 2024-03-19T02:43:29.919 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.372+0000 7f7e8351b700 1 -- start start 2024-03-19T02:43:29.919 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.372+0000 7f7e8351b700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e7c1069b0 0x7f7e7c106dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.919 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.372+0000 7f7e8351b700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7c1073a0 con 0x7f7e7c1069b0 2024-03-19T02:43:29.919 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.372+0000 7f7e812b7700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e7c1069b0 0x7f7e7c106dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.919 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.372+0000 7f7e812b7700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e7c1069b0 0x7f7e7c106dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50806/0 (socket says 172.21.15.33:50806) 2024-03-19T02:43:29.919 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.372+0000 7f7e812b7700 1 -- 172.21.15.33:0/2950113185 learned_addr learned my addr 172.21.15.33:0/2950113185 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:29.919 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.373+0000 7f7e812b7700 1 -- 172.21.15.33:0/2950113185 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e7c107bb0 con 0x7f7e7c1069b0 2024-03-19T02:43:29.919 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.373+0000 7f7e812b7700 1 --2- 172.21.15.33:0/2950113185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e7c1069b0 0x7f7e7c106dd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7e780098a0 tx=0x7f7e78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa5e23328b3cd70c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.919 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.373+0000 7f7e73fff700 1 -- 172.21.15.33:0/2950113185 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e78004160 con 0x7f7e7c1069b0 2024-03-19T02:43:29.919 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.373+0000 7f7e73fff700 1 -- 172.21.15.33:0/2950113185 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e78004300 con 0x7f7e7c1069b0 2024-03-19T02:43:29.919 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.373+0000 7f7e73fff700 1 -- 172.21.15.33:0/2950113185 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e78011a50 con 0x7f7e7c1069b0 2024-03-19T02:43:29.919 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.374+0000 7f7e8351b700 1 -- 172.21.15.33:0/2950113185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e7c1069b0 msgr2=0x7f7e7c106dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.919 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.374+0000 7f7e8351b700 1 --2- 172.21.15.33:0/2950113185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e7c1069b0 0x7f7e7c106dd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7e780098a0 tx=0x7f7e78009bb0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.919 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.374+0000 7f7e8351b700 1 -- 172.21.15.33:0/2950113185 shutdown_connections 2024-03-19T02:43:29.919 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.374+0000 7f7e8351b700 1 --2- 172.21.15.33:0/2950113185 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e7c1069b0 0x7f7e7c106dd0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.920 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.374+0000 7f7e8351b700 1 -- 172.21.15.33:0/2950113185 >> 172.21.15.33:0/2950113185 conn(0x7f7e7c101fb0 msgr2=0x7f7e7c1043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:29.920 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.374+0000 7f7e8351b700 1 -- 172.21.15.33:0/2950113185 shutdown_connections 2024-03-19T02:43:29.920 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.374+0000 7f7e8351b700 1 -- 172.21.15.33:0/2950113185 wait complete. 2024-03-19T02:43:29.920 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.375+0000 7f7e8351b700 1 Processor -- start 2024-03-19T02:43:29.920 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.375+0000 7f7e8351b700 1 -- start start 2024-03-19T02:43:29.920 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.375+0000 7f7e8351b700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e7c1069b0 0x7f7e7c104230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.920 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.375+0000 7f7e8351b700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7c10af60 con 0x7f7e7c1069b0 2024-03-19T02:43:29.920 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.375+0000 7f7e812b7700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e7c1069b0 0x7f7e7c104230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.920 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.375+0000 7f7e812b7700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e7c1069b0 0x7f7e7c104230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50820/0 (socket says 172.21.15.33:50820) 2024-03-19T02:43:29.920 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.375+0000 7f7e812b7700 1 -- 172.21.15.33:0/3524706526 learned_addr learned my addr 172.21.15.33:0/3524706526 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:29.920 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.375+0000 7f7e812b7700 1 -- 172.21.15.33:0/3524706526 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e7c104770 con 0x7f7e7c1069b0 2024-03-19T02:43:29.920 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.376+0000 7f7e812b7700 1 --2- 172.21.15.33:0/3524706526 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e7c1069b0 0x7f7e7c104230 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7e78006ba0 tx=0x7f7e78011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.920 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.376+0000 7f7e727fc700 1 -- 172.21.15.33:0/3524706526 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e78003af0 con 0x7f7e7c1069b0 2024-03-19T02:43:29.920 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.376+0000 7f7e8351b700 1 -- 172.21.15.33:0/3524706526 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e7c1029b0 con 0x7f7e7c1069b0 2024-03-19T02:43:29.920 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.376+0000 7f7e727fc700 1 -- 172.21.15.33:0/3524706526 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e7801a8e0 con 0x7f7e7c1069b0 2024-03-19T02:43:29.920 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.376+0000 7f7e727fc700 1 -- 172.21.15.33:0/3524706526 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e78024900 con 0x7f7e7c1069b0 2024-03-19T02:43:29.921 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.376+0000 7f7e8351b700 1 -- 172.21.15.33:0/3524706526 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e7c102e90 con 0x7f7e7c1069b0 2024-03-19T02:43:29.921 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.377+0000 7f7e8351b700 1 -- 172.21.15.33:0/3524706526 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e7c063940 con 0x7f7e7c1069b0 2024-03-19T02:43:29.921 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.377+0000 7f7e727fc700 1 -- 172.21.15.33:0/3524706526 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f7e78022020 con 0x7f7e7c1069b0 2024-03-19T02:43:29.921 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.377+0000 7f7e727fc700 1 --2- 172.21.15.33:0/3524706526 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f7e68038f90 0x7f7e6803b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:29.921 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.377+0000 7f7e727fc700 1 -- 172.21.15.33:0/3524706526 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7e7804d250 con 0x7f7e7c1069b0 2024-03-19T02:43:29.921 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.377+0000 7f7e80ab6700 1 --2- 172.21.15.33:0/3524706526 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f7e68038f90 0x7f7e6803b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:29.921 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.378+0000 7f7e80ab6700 1 --2- 172.21.15.33:0/3524706526 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f7e68038f90 0x7f7e6803b450 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c006fd0 tx=0x7f7e6c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:29.921 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.384+0000 7f7e727fc700 1 -- 172.21.15.33:0/3524706526 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7e7800b5a0 con 0x7f7e7c1069b0 2024-03-19T02:43:29.921 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.630+0000 7f7e8351b700 1 -- 172.21.15.33:0/3524706526 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f7e7c1951c0 con 0x7f7e7c1069b0 2024-03-19T02:43:29.921 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.636+0000 7f7e727fc700 1 -- 172.21.15.33:0/3524706526 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f7e7802dc90 con 0x7f7e7c1069b0 2024-03-19T02:43:29.921 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.639+0000 7f7e8351b700 1 -- 172.21.15.33:0/3524706526 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f7e68038f90 msgr2=0x7f7e6803b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.921 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.639+0000 7f7e8351b700 1 --2- 172.21.15.33:0/3524706526 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f7e68038f90 0x7f7e6803b450 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c006fd0 tx=0x7f7e6c006e40 comp rx=0 tx=0).stop 2024-03-19T02:43:29.921 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.639+0000 7f7e8351b700 1 -- 172.21.15.33:0/3524706526 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e7c1069b0 msgr2=0x7f7e7c104230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:29.921 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.639+0000 7f7e8351b700 1 --2- 172.21.15.33:0/3524706526 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e7c1069b0 0x7f7e7c104230 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7e78006ba0 tx=0x7f7e78011ad0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.921 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.639+0000 7f7e8351b700 1 -- 172.21.15.33:0/3524706526 shutdown_connections 2024-03-19T02:43:29.921 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.639+0000 7f7e8351b700 1 --2- 172.21.15.33:0/3524706526 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f7e68038f90 0x7f7e6803b450 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.922 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.639+0000 7f7e8351b700 1 --2- 172.21.15.33:0/3524706526 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e7c1069b0 0x7f7e7c104230 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:29.922 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.639+0000 7f7e8351b700 1 -- 172.21.15.33:0/3524706526 >> 172.21.15.33:0/3524706526 conn(0x7f7e7c101fb0 msgr2=0x7f7e7c192490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:29.922 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.639+0000 7f7e8351b700 1 -- 172.21.15.33:0/3524706526 shutdown_connections 2024-03-19T02:43:29.922 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:28.639+0000 7f7e8351b700 1 -- 172.21.15.33:0/3524706526 wait complete. 2024-03-19T02:43:31.185 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.380+0000 7f077038a700 1 Processor -- start 2024-03-19T02:43:31.185 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.380+0000 7f077038a700 1 -- start start 2024-03-19T02:43:31.185 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.380+0000 7f077038a700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0768107b00 0x7f0768107f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.185 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.380+0000 7f077038a700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f07681084f0 con 0x7f0768107b00 2024-03-19T02:43:31.185 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.381+0000 7f076e126700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0768107b00 0x7f0768107f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.185 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.381+0000 7f076e126700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0768107b00 0x7f0768107f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50832/0 (socket says 172.21.15.33:50832) 2024-03-19T02:43:31.185 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.381+0000 7f076e126700 1 -- 172.21.15.33:0/2771372286 learned_addr learned my addr 172.21.15.33:0/2771372286 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:31.186 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.381+0000 7f076e126700 1 -- 172.21.15.33:0/2771372286 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0768108cb0 con 0x7f0768107b00 2024-03-19T02:43:31.186 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.381+0000 7f076e126700 1 --2- 172.21.15.33:0/2771372286 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0768107b00 0x7f0768107f20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f07600098a0 tx=0x7f0760009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=706fd8b63c3b822f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:31.186 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.382+0000 7f076d124700 1 -- 172.21.15.33:0/2771372286 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0760004160 con 0x7f0768107b00 2024-03-19T02:43:31.186 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.382+0000 7f076d124700 1 -- 172.21.15.33:0/2771372286 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0760004300 con 0x7f0768107b00 2024-03-19T02:43:31.186 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.382+0000 7f076d124700 1 -- 172.21.15.33:0/2771372286 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0760011a50 con 0x7f0768107b00 2024-03-19T02:43:31.186 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.382+0000 7f077038a700 1 -- 172.21.15.33:0/2771372286 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0768107b00 msgr2=0x7f0768107f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:31.186 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.382+0000 7f077038a700 1 --2- 172.21.15.33:0/2771372286 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0768107b00 0x7f0768107f20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f07600098a0 tx=0x7f0760009bb0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.186 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.382+0000 7f077038a700 1 -- 172.21.15.33:0/2771372286 shutdown_connections 2024-03-19T02:43:31.186 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.382+0000 7f077038a700 1 --2- 172.21.15.33:0/2771372286 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0768107b00 0x7f0768107f20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.186 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.382+0000 7f077038a700 1 -- 172.21.15.33:0/2771372286 >> 172.21.15.33:0/2771372286 conn(0x7f07681030c0 msgr2=0x7f07681054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:31.186 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.382+0000 7f077038a700 1 -- 172.21.15.33:0/2771372286 shutdown_connections 2024-03-19T02:43:31.186 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.382+0000 7f077038a700 1 -- 172.21.15.33:0/2771372286 wait complete. 2024-03-19T02:43:31.186 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.383+0000 7f077038a700 1 Processor -- start 2024-03-19T02:43:31.187 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.383+0000 7f077038a700 1 -- start start 2024-03-19T02:43:31.187 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.383+0000 7f077038a700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0768107b00 0x7f076819b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.187 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.383+0000 7f077038a700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f076810c060 con 0x7f0768107b00 2024-03-19T02:43:31.187 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.383+0000 7f076e126700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0768107b00 0x7f076819b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.187 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.383+0000 7f076e126700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0768107b00 0x7f076819b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50846/0 (socket says 172.21.15.33:50846) 2024-03-19T02:43:31.187 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.383+0000 7f076e126700 1 -- 172.21.15.33:0/2845836245 learned_addr learned my addr 172.21.15.33:0/2845836245 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:31.187 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.384+0000 7f076e126700 1 -- 172.21.15.33:0/2845836245 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f076819be30 con 0x7f0768107b00 2024-03-19T02:43:31.187 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.384+0000 7f076e126700 1 --2- 172.21.15.33:0/2845836245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0768107b00 0x7f076819b8f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0760011710 tx=0x7f0760011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:31.187 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.384+0000 7f07677fe700 1 -- 172.21.15.33:0/2845836245 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f07600039c0 con 0x7f0768107b00 2024-03-19T02:43:31.187 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.384+0000 7f07677fe700 1 -- 172.21.15.33:0/2845836245 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0760003b60 con 0x7f0768107b00 2024-03-19T02:43:31.187 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.384+0000 7f077038a700 1 -- 172.21.15.33:0/2845836245 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f076819c0c0 con 0x7f0768107b00 2024-03-19T02:43:31.187 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.384+0000 7f077038a700 1 -- 172.21.15.33:0/2845836245 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f076819edb0 con 0x7f0768107b00 2024-03-19T02:43:31.187 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.384+0000 7f07677fe700 1 -- 172.21.15.33:0/2845836245 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f076001a690 con 0x7f0768107b00 2024-03-19T02:43:31.187 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.385+0000 7f07677fe700 1 -- 172.21.15.33:0/2845836245 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f076001aa50 con 0x7f0768107b00 2024-03-19T02:43:31.188 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.385+0000 7f07677fe700 1 --2- 172.21.15.33:0/2845836245 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f0750038f90 0x7f075003b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:31.188 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.385+0000 7f07677fe700 1 -- 172.21.15.33:0/2845836245 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f076004d440 con 0x7f0768107b00 2024-03-19T02:43:31.188 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.385+0000 7f076d925700 1 --2- 172.21.15.33:0/2845836245 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f0750038f90 0x7f075003b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:31.188 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.385+0000 7f077038a700 1 -- 172.21.15.33:0/2845836245 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0754005360 con 0x7f0768107b00 2024-03-19T02:43:31.188 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.386+0000 7f076d925700 1 --2- 172.21.15.33:0/2845836245 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f0750038f90 0x7f075003b450 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f075c006fd0 tx=0x7f075c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:31.188 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.391+0000 7f07677fe700 1 -- 172.21.15.33:0/2845836245 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f076000bc40 con 0x7f0768107b00 2024-03-19T02:43:31.188 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.595+0000 7f077038a700 1 -- 172.21.15.33:0/2845836245 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f0754005650 con 0x7f0768107b00 2024-03-19T02:43:31.188 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.599+0000 7f07677fe700 1 -- 172.21.15.33:0/2845836245 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f076001ad40 con 0x7f0768107b00 2024-03-19T02:43:31.188 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.601+0000 7f077038a700 1 -- 172.21.15.33:0/2845836245 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f0750038f90 msgr2=0x7f075003b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:31.188 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.601+0000 7f077038a700 1 --2- 172.21.15.33:0/2845836245 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f0750038f90 0x7f075003b450 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f075c006fd0 tx=0x7f075c006e40 comp rx=0 tx=0).stop 2024-03-19T02:43:31.188 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.601+0000 7f077038a700 1 -- 172.21.15.33:0/2845836245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0768107b00 msgr2=0x7f076819b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:31.188 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.601+0000 7f077038a700 1 --2- 172.21.15.33:0/2845836245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0768107b00 0x7f076819b8f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0760011710 tx=0x7f0760011ad0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.188 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.601+0000 7f077038a700 1 -- 172.21.15.33:0/2845836245 shutdown_connections 2024-03-19T02:43:31.188 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.601+0000 7f077038a700 1 --2- 172.21.15.33:0/2845836245 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f0750038f90 0x7f075003b450 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.188 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.601+0000 7f077038a700 1 --2- 172.21.15.33:0/2845836245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0768107b00 0x7f076819b8f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:31.189 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.601+0000 7f077038a700 1 -- 172.21.15.33:0/2845836245 >> 172.21.15.33:0/2845836245 conn(0x7f07681030c0 msgr2=0x7f0768103a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:31.189 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.601+0000 7f077038a700 1 -- 172.21.15.33:0/2845836245 shutdown_connections 2024-03-19T02:43:31.189 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:30.601+0000 7f077038a700 1 -- 172.21.15.33:0/2845836245 wait complete. 2024-03-19T02:43:31.189 INFO:teuthology.orchestra.run.smithi033.stdout:Enabling the dashboard module... 2024-03-19T02:43:31.766 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:31 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2845836245' entity='client.admin' 2024-03-19T02:43:31.766 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:31 smithi033 ceph-mon[31973]: from='mgr.14118 172.21.15.33:0/3216842381' entity='mgr.x' 2024-03-19T02:43:32.916 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:32 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2441193076' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-19T02:43:32.916 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:32 smithi033 ceph-mon[31973]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:43:32.916 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:32 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: ignoring --setuser ceph since I am not root 2024-03-19T02:43:32.916 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:32 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: ignoring --setgroup ceph since I am not root 2024-03-19T02:43:33.215 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.606+0000 7f3324cae700 1 Processor -- start 2024-03-19T02:43:33.215 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.606+0000 7f3324cae700 1 -- start start 2024-03-19T02:43:33.215 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.606+0000 7f3324cae700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3320107b00 0x7f3320107f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:33.216 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.606+0000 7f3324cae700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f33201084f0 con 0x7f3320107b00 2024-03-19T02:43:33.216 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.607+0000 7f331e59c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3320107b00 0x7f3320107f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:33.216 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.607+0000 7f331e59c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3320107b00 0x7f3320107f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50862/0 (socket says 172.21.15.33:50862) 2024-03-19T02:43:33.216 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.607+0000 7f331e59c700 1 -- 172.21.15.33:0/1660088051 learned_addr learned my addr 172.21.15.33:0/1660088051 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:33.216 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.607+0000 7f331e59c700 1 -- 172.21.15.33:0/1660088051 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3320108cb0 con 0x7f3320107b00 2024-03-19T02:43:33.216 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.607+0000 7f331e59c700 1 --2- 172.21.15.33:0/1660088051 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3320107b00 0x7f3320107f20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3310009e80 tx=0x7f331000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=269a901ec06def36 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:33.216 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.607+0000 7f331d59a700 1 -- 172.21.15.33:0/1660088051 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3310017070 con 0x7f3320107b00 2024-03-19T02:43:33.216 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.607+0000 7f331d59a700 1 -- 172.21.15.33:0/1660088051 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f331000b8c0 con 0x7f3320107b00 2024-03-19T02:43:33.216 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.607+0000 7f331d59a700 1 -- 172.21.15.33:0/1660088051 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3310004490 con 0x7f3320107b00 2024-03-19T02:43:33.216 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.608+0000 7f3324cae700 1 -- 172.21.15.33:0/1660088051 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3320107b00 msgr2=0x7f3320107f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:33.216 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.608+0000 7f3324cae700 1 --2- 172.21.15.33:0/1660088051 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3320107b00 0x7f3320107f20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3310009e80 tx=0x7f331000b330 comp rx=0 tx=0).stop 2024-03-19T02:43:33.216 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.608+0000 7f3324cae700 1 -- 172.21.15.33:0/1660088051 shutdown_connections 2024-03-19T02:43:33.216 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.608+0000 7f3324cae700 1 --2- 172.21.15.33:0/1660088051 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3320107b00 0x7f3320107f20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:33.216 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.608+0000 7f3324cae700 1 -- 172.21.15.33:0/1660088051 >> 172.21.15.33:0/1660088051 conn(0x7f33201030c0 msgr2=0x7f33201054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:33.217 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.608+0000 7f3324cae700 1 -- 172.21.15.33:0/1660088051 shutdown_connections 2024-03-19T02:43:33.217 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.608+0000 7f3324cae700 1 -- 172.21.15.33:0/1660088051 wait complete. 2024-03-19T02:43:33.217 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.609+0000 7f3324cae700 1 Processor -- start 2024-03-19T02:43:33.217 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.609+0000 7f3324cae700 1 -- start start 2024-03-19T02:43:33.217 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.609+0000 7f3324cae700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3320107b00 0x7f332019b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:33.217 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.609+0000 7f3324cae700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f332010c060 con 0x7f3320107b00 2024-03-19T02:43:33.217 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.609+0000 7f331e59c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3320107b00 0x7f332019b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:33.217 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.609+0000 7f331e59c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3320107b00 0x7f332019b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50870/0 (socket says 172.21.15.33:50870) 2024-03-19T02:43:33.217 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.610+0000 7f331e59c700 1 -- 172.21.15.33:0/2441193076 learned_addr learned my addr 172.21.15.33:0/2441193076 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:33.217 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.610+0000 7f331e59c700 1 -- 172.21.15.33:0/2441193076 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f332019be80 con 0x7f3320107b00 2024-03-19T02:43:33.217 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.610+0000 7f331e59c700 1 --2- 172.21.15.33:0/2441193076 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3320107b00 0x7f332019b940 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f331000b7b0 tx=0x7f3310003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:33.217 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.610+0000 7f33177fe700 1 -- 172.21.15.33:0/2441193076 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3310020020 con 0x7f3320107b00 2024-03-19T02:43:33.217 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.610+0000 7f33177fe700 1 -- 172.21.15.33:0/2441193076 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3310012430 con 0x7f3320107b00 2024-03-19T02:43:33.217 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.610+0000 7f3324cae700 1 -- 172.21.15.33:0/2441193076 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f332019c110 con 0x7f3320107b00 2024-03-19T02:43:33.218 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.610+0000 7f33177fe700 1 -- 172.21.15.33:0/2441193076 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3310017040 con 0x7f3320107b00 2024-03-19T02:43:33.218 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.610+0000 7f3324cae700 1 -- 172.21.15.33:0/2441193076 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f332019ee00 con 0x7f3320107b00 2024-03-19T02:43:33.218 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.611+0000 7f33177fe700 1 -- 172.21.15.33:0/2441193076 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f3310023020 con 0x7f3320107b00 2024-03-19T02:43:33.218 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.611+0000 7f3324cae700 1 -- 172.21.15.33:0/2441193076 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3320063940 con 0x7f3320107b00 2024-03-19T02:43:33.218 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.611+0000 7f33177fe700 1 --2- 172.21.15.33:0/2441193076 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f330c039030 0x7f330c03b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:33.218 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.611+0000 7f33177fe700 1 -- 172.21.15.33:0/2441193076 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f331000f050 con 0x7f3320107b00 2024-03-19T02:43:33.218 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.612+0000 7f331dd9b700 1 --2- 172.21.15.33:0/2441193076 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f330c039030 0x7f330c03b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:33.218 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.612+0000 7f331dd9b700 1 --2- 172.21.15.33:0/2441193076 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f330c039030 0x7f330c03b4f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3308006fd0 tx=0x7f3308006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:33.218 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.618+0000 7f33177fe700 1 -- 172.21.15.33:0/2441193076 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f331002a020 con 0x7f3320107b00 2024-03-19T02:43:33.218 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:31.894+0000 7f3324cae700 1 -- 172.21.15.33:0/2441193076 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f332019efd0 con 0x7f3320107b00 2024-03-19T02:43:33.218 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:32.601+0000 7f33177fe700 1 -- 172.21.15.33:0/2441193076 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f332019efd0 con 0x7f3320107b00 2024-03-19T02:43:33.218 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:32.604+0000 7f33177fe700 1 -- 172.21.15.33:0/2441193076 <== mon.0 v2:172.21.15.33:3300/0 8 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f3310025400 con 0x7f3320107b00 2024-03-19T02:43:33.218 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:32.606+0000 7f3324cae700 1 -- 172.21.15.33:0/2441193076 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f330c039030 msgr2=0x7f330c03b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:33.219 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:32.606+0000 7f3324cae700 1 --2- 172.21.15.33:0/2441193076 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f330c039030 0x7f330c03b4f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3308006fd0 tx=0x7f3308006e40 comp rx=0 tx=0).stop 2024-03-19T02:43:33.219 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:32.606+0000 7f3324cae700 1 -- 172.21.15.33:0/2441193076 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3320107b00 msgr2=0x7f332019b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:33.219 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:32.606+0000 7f3324cae700 1 --2- 172.21.15.33:0/2441193076 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3320107b00 0x7f332019b940 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f331000b7b0 tx=0x7f3310003ce0 comp rx=0 tx=0).stop 2024-03-19T02:43:33.219 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:32.606+0000 7f3324cae700 1 -- 172.21.15.33:0/2441193076 shutdown_connections 2024-03-19T02:43:33.219 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:32.606+0000 7f3324cae700 1 --2- 172.21.15.33:0/2441193076 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f330c039030 0x7f330c03b4f0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:33.219 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:32.606+0000 7f3324cae700 1 --2- 172.21.15.33:0/2441193076 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3320107b00 0x7f332019b940 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:33.219 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:32.606+0000 7f3324cae700 1 -- 172.21.15.33:0/2441193076 >> 172.21.15.33:0/2441193076 conn(0x7f33201030c0 msgr2=0x7f3320103b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:33.219 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:32.606+0000 7f3324cae700 1 -- 172.21.15.33:0/2441193076 shutdown_connections 2024-03-19T02:43:33.219 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:32.606+0000 7f3324cae700 1 -- 172.21.15.33:0/2441193076 wait complete. 2024-03-19T02:43:33.454 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:33 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:33.181+0000 7f01237fe200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T02:43:33.455 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:33 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:33.280+0000 7f01237fe200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T02:43:33.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:33 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2441193076' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-19T02:43:33.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:33 smithi033 ceph-mon[31973]: mgrmap e9: x(active, since 20s) 2024-03-19T02:43:34.194 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:33 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:33.952+0000 7f01237fe200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T02:43:34.195 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:34 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:34.048+0000 7f01237fe200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T02:43:34.195 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:34 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:34.193+0000 7f01237fe200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T02:43:34.524 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:34 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:34.292+0000 7f01237fe200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T02:43:34.524 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:34 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:34.377+0000 7f01237fe200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T02:43:34.526 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout { 2024-03-19T02:43:34.532 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-19T02:43:34.533 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-19T02:43:34.533 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-19T02:43:34.533 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-19T02:43:34.533 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout } 2024-03-19T02:43:34.533 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.652+0000 7f365cf29700 1 Processor -- start 2024-03-19T02:43:34.533 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.652+0000 7f365cf29700 1 -- start start 2024-03-19T02:43:34.533 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.653+0000 7f365cf29700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3658108580 0x7f365810a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:34.533 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.653+0000 7f365cf29700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f365807be20 con 0x7f3658108580 2024-03-19T02:43:34.533 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.653+0000 7f365659c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3658108580 0x7f365810a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:34.533 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.653+0000 7f365659c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3658108580 0x7f365810a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34328/0 (socket says 172.21.15.33:34328) 2024-03-19T02:43:34.533 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.653+0000 7f365659c700 1 -- 172.21.15.33:0/4237625513 learned_addr learned my addr 172.21.15.33:0/4237625513 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:34.533 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.653+0000 7f365659c700 1 -- 172.21.15.33:0/4237625513 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f365810aeb0 con 0x7f3658108580 2024-03-19T02:43:34.533 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.653+0000 7f365659c700 1 --2- 172.21.15.33:0/4237625513 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3658108580 0x7f365810a970 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f364c0098a0 tx=0x7f364c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81e47e107891330d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:34.533 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.653+0000 7f365559a700 1 -- 172.21.15.33:0/4237625513 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f364c004160 con 0x7f3658108580 2024-03-19T02:43:34.533 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.654+0000 7f365559a700 1 -- 172.21.15.33:0/4237625513 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f364c004300 con 0x7f3658108580 2024-03-19T02:43:34.534 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.654+0000 7f365559a700 1 -- 172.21.15.33:0/4237625513 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f364c011a50 con 0x7f3658108580 2024-03-19T02:43:34.534 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.654+0000 7f365cf29700 1 -- 172.21.15.33:0/4237625513 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3658108580 msgr2=0x7f365810a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:34.534 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.654+0000 7f365cf29700 1 --2- 172.21.15.33:0/4237625513 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3658108580 0x7f365810a970 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f364c0098a0 tx=0x7f364c009bb0 comp rx=0 tx=0).stop 2024-03-19T02:43:34.534 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.654+0000 7f365cf29700 1 -- 172.21.15.33:0/4237625513 shutdown_connections 2024-03-19T02:43:34.534 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.654+0000 7f365cf29700 1 --2- 172.21.15.33:0/4237625513 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3658108580 0x7f365810a970 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:34.534 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.654+0000 7f365cf29700 1 -- 172.21.15.33:0/4237625513 >> 172.21.15.33:0/4237625513 conn(0x7f3658101fb0 msgr2=0x7f36581043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:34.534 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.654+0000 7f365cf29700 1 -- 172.21.15.33:0/4237625513 shutdown_connections 2024-03-19T02:43:34.534 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.654+0000 7f365cf29700 1 -- 172.21.15.33:0/4237625513 wait complete. 2024-03-19T02:43:34.534 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.655+0000 7f365cf29700 1 Processor -- start 2024-03-19T02:43:34.534 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.655+0000 7f365cf29700 1 -- start start 2024-03-19T02:43:34.534 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.655+0000 7f365cf29700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3658108580 0x7f365819b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:34.534 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.655+0000 7f365cf29700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f365810be60 con 0x7f3658108580 2024-03-19T02:43:34.534 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.655+0000 7f365659c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3658108580 0x7f365819b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:34.534 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.656+0000 7f365659c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3658108580 0x7f365819b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34338/0 (socket says 172.21.15.33:34338) 2024-03-19T02:43:34.535 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.656+0000 7f365659c700 1 -- 172.21.15.33:0/1676324502 learned_addr learned my addr 172.21.15.33:0/1676324502 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:34.535 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.656+0000 7f365659c700 1 -- 172.21.15.33:0/1676324502 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f365819bd80 con 0x7f3658108580 2024-03-19T02:43:34.535 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.656+0000 7f365659c700 1 --2- 172.21.15.33:0/1676324502 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3658108580 0x7f365819b840 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f364c006b20 tx=0x7f364c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:34.535 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.656+0000 7f36477fe700 1 -- 172.21.15.33:0/1676324502 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f364c003b70 con 0x7f3658108580 2024-03-19T02:43:34.535 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.656+0000 7f36477fe700 1 -- 172.21.15.33:0/1676324502 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f364c01a8e0 con 0x7f3658108580 2024-03-19T02:43:34.535 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.656+0000 7f365cf29700 1 -- 172.21.15.33:0/1676324502 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f365819c010 con 0x7f3658108580 2024-03-19T02:43:34.535 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.656+0000 7f36477fe700 1 -- 172.21.15.33:0/1676324502 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f364c0249b0 con 0x7f3658108580 2024-03-19T02:43:34.535 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.656+0000 7f365cf29700 1 -- 172.21.15.33:0/1676324502 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f365819ed00 con 0x7f3658108580 2024-03-19T02:43:34.535 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.657+0000 7f36477fe700 1 -- 172.21.15.33:0/1676324502 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f364c022020 con 0x7f3658108580 2024-03-19T02:43:34.535 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.657+0000 7f36477fe700 1 --2- 172.21.15.33:0/1676324502 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f3640039090 0x7f364003b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:34.535 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.657+0000 7f36477fe700 1 -- 172.21.15.33:0/1676324502 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f364c04e540 con 0x7f3658108580 2024-03-19T02:43:34.535 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.657+0000 7f3655d9b700 1 -- 172.21.15.33:0/1676324502 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f3640039090 msgr2=0x7f364003b550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/906645748 2024-03-19T02:43:34.535 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.657+0000 7f365cf29700 1 -- 172.21.15.33:0/1676324502 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3658063940 con 0x7f3658108580 2024-03-19T02:43:34.536 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.657+0000 7f3655d9b700 1 --2- 172.21.15.33:0/1676324502 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f3640039090 0x7f364003b550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T02:43:34.536 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.662+0000 7f36477fe700 1 -- 172.21.15.33:0/1676324502 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f364c00b3e0 con 0x7f3658108580 2024-03-19T02:43:34.536 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.858+0000 7f3655d9b700 1 -- 172.21.15.33:0/1676324502 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f3640039090 msgr2=0x7f364003b550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/906645748 2024-03-19T02:43:34.536 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.858+0000 7f3655d9b700 1 --2- 172.21.15.33:0/1676324502 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f3640039090 0x7f364003b550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T02:43:34.536 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.927+0000 7f365cf29700 1 -- 172.21.15.33:0/1676324502 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f365819eed0 con 0x7f3658108580 2024-03-19T02:43:34.536 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.927+0000 7f36477fe700 1 -- 172.21.15.33:0/1676324502 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f364c0172b0 con 0x7f3658108580 2024-03-19T02:43:34.536 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.930+0000 7f365cf29700 1 -- 172.21.15.33:0/1676324502 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f3640039090 msgr2=0x7f364003b550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T02:43:34.536 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.930+0000 7f365cf29700 1 --2- 172.21.15.33:0/1676324502 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f3640039090 0x7f364003b550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:34.536 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.930+0000 7f365cf29700 1 -- 172.21.15.33:0/1676324502 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3658108580 msgr2=0x7f365819b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:34.536 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.930+0000 7f365cf29700 1 --2- 172.21.15.33:0/1676324502 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3658108580 0x7f365819b840 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f364c006b20 tx=0x7f364c011ad0 comp rx=0 tx=0).stop 2024-03-19T02:43:34.536 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.930+0000 7f365cf29700 1 -- 172.21.15.33:0/1676324502 shutdown_connections 2024-03-19T02:43:34.536 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.930+0000 7f365cf29700 1 --2- 172.21.15.33:0/1676324502 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f3640039090 0x7f364003b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:34.536 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.930+0000 7f365cf29700 1 --2- 172.21.15.33:0/1676324502 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3658108580 0x7f365819b840 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:34.536 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.930+0000 7f365cf29700 1 -- 172.21.15.33:0/1676324502 >> 172.21.15.33:0/1676324502 conn(0x7f3658101fb0 msgr2=0x7f3658102930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:34.536 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.930+0000 7f365cf29700 1 -- 172.21.15.33:0/1676324502 shutdown_connections 2024-03-19T02:43:34.536 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:33.930+0000 7f365cf29700 1 -- 172.21.15.33:0/1676324502 wait complete. 2024-03-19T02:43:34.537 INFO:teuthology.orchestra.run.smithi033.stdout:Waiting for the mgr to restart... 2024-03-19T02:43:34.537 INFO:teuthology.orchestra.run.smithi033.stdout:Waiting for mgr epoch 9... 2024-03-19T02:43:34.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:34 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1676324502' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-19T02:43:34.778 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:34 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:34.651+0000 7f01237fe200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-19T02:43:34.778 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:34 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:34.765+0000 7f01237fe200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T02:43:35.135 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:34 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:34.854+0000 7f01237fe200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T02:43:35.136 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:34 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:34.954+0000 7f01237fe200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T02:43:35.136 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:35 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:35.135+0000 7f01237fe200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T02:43:36.386 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:36 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:36.121+0000 7f01237fe200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-19T02:43:36.387 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:36 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:36.255+0000 7f01237fe200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T02:43:36.686 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:36 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:36.593+0000 7f01237fe200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T02:43:36.686 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:36 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:36.685+0000 7f01237fe200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T02:43:37.028 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:36 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:36.886+0000 7f01237fe200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:43:37.368 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:37 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:37.075+0000 7f01237fe200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T02:43:37.629 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:37 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:37.367+0000 7f01237fe200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:43:37.630 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:37 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:37.629+0000 7f01237fe200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T02:43:37.987 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:37 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:37.871+0000 7f01237fe200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T02:43:39.028 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:38 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:38.697+0000 7f01237fe200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T02:43:39.609 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:39 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:39.280+0000 7f01237fe200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T02:43:39.609 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:39 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:39.385+0000 7f01237fe200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T02:43:39.609 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:43:39 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:43:39.608+0000 7f01237fe200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T02:43:40.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:39 smithi033 ceph-mon[31973]: Active manager daemon x restarted 2024-03-19T02:43:40.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:39 smithi033 ceph-mon[31973]: Activating manager daemon x 2024-03-19T02:43:40.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:39 smithi033 ceph-mon[31973]: osdmap e3: 0 total, 0 up, 0 in 2024-03-19T02:43:40.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:39 smithi033 ceph-mon[31973]: mgrmap e10: x(active, starting, since 0.00309136s) 2024-03-19T02:43:40.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:39 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:43:40.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:39 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-19T02:43:40.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:39 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-19T02:43:40.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:39 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-19T02:43:40.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:39 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-19T02:43:40.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:39 smithi033 ceph-mon[31973]: Manager daemon x is now available 2024-03-19T02:43:40.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:39 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:43:40.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:39 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:43:40.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:39 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-19T02:43:40.860 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-19T02:43:40.861 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:40 smithi033 ceph-mon[31973]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-19T02:43:40.861 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:40 smithi033 ceph-mon[31973]: mgrmap e11: x(active, since 1.00637s) 2024-03-19T02:43:40.861 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:40 smithi033 ceph-mon[31973]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-19T02:43:41.270 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout { 2024-03-19T02:43:41.270 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-19T02:43:41.270 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-19T02:43:41.270 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout } 2024-03-19T02:43:41.270 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.977+0000 7f9034f51700 1 Processor -- start 2024-03-19T02:43:41.270 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.977+0000 7f9034f51700 1 -- start start 2024-03-19T02:43:41.270 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.978+0000 7f9034f51700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f903007de40 0x7f903007c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:41.270 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.978+0000 7f9034f51700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f903007c8c0 con 0x7f903007de40 2024-03-19T02:43:41.270 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.978+0000 7f902e59c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f903007de40 0x7f903007c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:41.270 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.978+0000 7f902e59c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f903007de40 0x7f903007c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34346/0 (socket says 172.21.15.33:34346) 2024-03-19T02:43:41.270 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.978+0000 7f902e59c700 1 -- 172.21.15.33:0/3630257763 learned_addr learned my addr 172.21.15.33:0/3630257763 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:41.271 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.978+0000 7f902e59c700 1 -- 172.21.15.33:0/3630257763 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f903007ca40 con 0x7f903007de40 2024-03-19T02:43:41.271 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.979+0000 7f902e59c700 1 --2- 172.21.15.33:0/3630257763 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f903007de40 0x7f903007c2f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f90200098a0 tx=0x7f9020009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69c4d2f94adc1bb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:41.271 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.979+0000 7f902d59a700 1 -- 172.21.15.33:0/3630257763 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9020004160 con 0x7f903007de40 2024-03-19T02:43:41.271 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.979+0000 7f902d59a700 1 -- 172.21.15.33:0/3630257763 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9020004300 con 0x7f903007de40 2024-03-19T02:43:41.271 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.979+0000 7f902d59a700 1 -- 172.21.15.33:0/3630257763 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9020011a50 con 0x7f903007de40 2024-03-19T02:43:41.271 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.980+0000 7f9034f51700 1 -- 172.21.15.33:0/3630257763 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f903007de40 msgr2=0x7f903007c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:41.271 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.980+0000 7f9034f51700 1 --2- 172.21.15.33:0/3630257763 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f903007de40 0x7f903007c2f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f90200098a0 tx=0x7f9020009bb0 comp rx=0 tx=0).stop 2024-03-19T02:43:41.271 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.980+0000 7f9034f51700 1 -- 172.21.15.33:0/3630257763 shutdown_connections 2024-03-19T02:43:41.271 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.980+0000 7f9034f51700 1 --2- 172.21.15.33:0/3630257763 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f903007de40 0x7f903007c2f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:41.271 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.980+0000 7f9034f51700 1 -- 172.21.15.33:0/3630257763 >> 172.21.15.33:0/3630257763 conn(0x7f90301030c0 msgr2=0x7f90301054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:41.271 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.980+0000 7f9034f51700 1 -- 172.21.15.33:0/3630257763 shutdown_connections 2024-03-19T02:43:41.271 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.980+0000 7f9034f51700 1 -- 172.21.15.33:0/3630257763 wait complete. 2024-03-19T02:43:41.271 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.980+0000 7f9034f51700 1 Processor -- start 2024-03-19T02:43:41.271 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.981+0000 7f9034f51700 1 -- start start 2024-03-19T02:43:41.272 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.981+0000 7f9034f51700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f903007de40 0x7f903019dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:41.272 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.981+0000 7f9034f51700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f903010e080 con 0x7f903007de40 2024-03-19T02:43:41.272 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.981+0000 7f902e59c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f903007de40 0x7f903019dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:41.272 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.981+0000 7f902e59c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f903007de40 0x7f903019dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34360/0 (socket says 172.21.15.33:34360) 2024-03-19T02:43:41.272 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.981+0000 7f902e59c700 1 -- 172.21.15.33:0/2901201084 learned_addr learned my addr 172.21.15.33:0/2901201084 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:41.272 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.981+0000 7f902e59c700 1 -- 172.21.15.33:0/2901201084 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f903019e010 con 0x7f903007de40 2024-03-19T02:43:41.272 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.981+0000 7f902e59c700 1 --2- 172.21.15.33:0/2901201084 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f903007de40 0x7f903019dad0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9020011a20 tx=0x7f9020011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:41.272 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.981+0000 7f901f7fe700 1 -- 172.21.15.33:0/2901201084 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f90200039c0 con 0x7f903007de40 2024-03-19T02:43:41.272 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.981+0000 7f901f7fe700 1 -- 172.21.15.33:0/2901201084 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9020003b60 con 0x7f903007de40 2024-03-19T02:43:41.272 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.981+0000 7f9034f51700 1 -- 172.21.15.33:0/2901201084 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f903019e2a0 con 0x7f903007de40 2024-03-19T02:43:41.272 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.981+0000 7f901f7fe700 1 -- 172.21.15.33:0/2901201084 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f902001a6b0 con 0x7f903007de40 2024-03-19T02:43:41.272 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.982+0000 7f9034f51700 1 -- 172.21.15.33:0/2901201084 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90301a0f90 con 0x7f903007de40 2024-03-19T02:43:41.272 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.982+0000 7f901f7fe700 1 -- 172.21.15.33:0/2901201084 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f902001a9b0 con 0x7f903007de40 2024-03-19T02:43:41.272 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.982+0000 7f901f7fe700 1 --2- 172.21.15.33:0/2901201084 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f9018038fe0 0x7f901803b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:41.273 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.982+0000 7f901f7fe700 1 -- 172.21.15.33:0/2901201084 --> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f901803bbb0 con 0x7f9018038fe0 2024-03-19T02:43:41.273 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.983+0000 7f901f7fe700 1 -- 172.21.15.33:0/2901201084 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f902004e310 con 0x7f903007de40 2024-03-19T02:43:41.273 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.983+0000 7f902dd9b700 1 -- 172.21.15.33:0/2901201084 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f9018038fe0 msgr2=0x7f901803b4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/906645748 2024-03-19T02:43:41.273 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:34.983+0000 7f902dd9b700 1 --2- 172.21.15.33:0/2901201084 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f9018038fe0 0x7f901803b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T02:43:41.273 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:35.184+0000 7f902dd9b700 1 -- 172.21.15.33:0/2901201084 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f9018038fe0 msgr2=0x7f901803b4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/906645748 2024-03-19T02:43:41.273 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:35.184+0000 7f902dd9b700 1 --2- 172.21.15.33:0/2901201084 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f9018038fe0 0x7f901803b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T02:43:41.273 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:35.585+0000 7f902dd9b700 1 -- 172.21.15.33:0/2901201084 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f9018038fe0 msgr2=0x7f901803b4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/906645748 2024-03-19T02:43:41.273 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:35.585+0000 7f902dd9b700 1 --2- 172.21.15.33:0/2901201084 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f9018038fe0 0x7f901803b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-19T02:43:41.273 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:36.386+0000 7f902dd9b700 1 -- 172.21.15.33:0/2901201084 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f9018038fe0 msgr2=0x7f901803b4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/906645748 2024-03-19T02:43:41.273 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:36.386+0000 7f902dd9b700 1 --2- 172.21.15.33:0/2901201084 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f9018038fe0 0x7f901803b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-19T02:43:41.273 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:37.987+0000 7f902dd9b700 1 -- 172.21.15.33:0/2901201084 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f9018038fe0 msgr2=0x7f901803b4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/906645748 2024-03-19T02:43:41.273 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:37.987+0000 7f902dd9b700 1 --2- 172.21.15.33:0/2901201084 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f9018038fe0 0x7f901803b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-19T02:43:41.273 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:39.623+0000 7f901f7fe700 1 -- 172.21.15.33:0/2901201084 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mgrmap(e 10) ==== 45886+0+0 (secure 0 0 0) 0x7f902002d910 con 0x7f903007de40 2024-03-19T02:43:41.273 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:39.623+0000 7f901f7fe700 1 -- 172.21.15.33:0/2901201084 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f9018038fe0 msgr2=0x7f901803b4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T02:43:41.273 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:39.623+0000 7f901f7fe700 1 --2- 172.21.15.33:0/2901201084 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f9018038fe0 0x7f901803b4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:41.274 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:40.625+0000 7f901f7fe700 1 -- 172.21.15.33:0/2901201084 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mgrmap(e 11) ==== 46013+0+0 (secure 0 0 0) 0x7f902004f360 con 0x7f903007de40 2024-03-19T02:43:41.274 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:40.625+0000 7f901f7fe700 1 --2- 172.21.15.33:0/2901201084 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f901803ca60 0x7f901803ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:41.274 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:40.625+0000 7f901f7fe700 1 -- 172.21.15.33:0/2901201084 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9020025e70 con 0x7f901803ca60 2024-03-19T02:43:41.274 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:40.625+0000 7f902dd9b700 1 --2- 172.21.15.33:0/2901201084 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f901803ca60 0x7f901803ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:41.274 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:40.626+0000 7f902dd9b700 1 --2- 172.21.15.33:0/2901201084 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f901803ca60 0x7f901803ee50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9024003a10 tx=0x7f90240092b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:41.274 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:40.627+0000 7f901f7fe700 1 -- 172.21.15.33:0/2901201084 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f9020025e70 con 0x7f901803ca60 2024-03-19T02:43:41.278 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:40.634+0000 7f9034f51700 1 -- 172.21.15.33:0/2901201084 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f9030063940 con 0x7f901803ca60 2024-03-19T02:43:41.278 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:40.634+0000 7f901f7fe700 1 -- 172.21.15.33:0/2901201084 <== mgr.14150 v2:172.21.15.33:6800/1453067701 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f9030063940 con 0x7f901803ca60 2024-03-19T02:43:41.278 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:40.634+0000 7f9034f51700 1 -- 172.21.15.33:0/2901201084 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f901803ca60 msgr2=0x7f901803ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:41.278 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:40.634+0000 7f9034f51700 1 --2- 172.21.15.33:0/2901201084 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f901803ca60 0x7f901803ee50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9024003a10 tx=0x7f90240092b0 comp rx=0 tx=0).stop 2024-03-19T02:43:41.278 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:40.634+0000 7f9034f51700 1 -- 172.21.15.33:0/2901201084 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f903007de40 msgr2=0x7f903019dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:41.278 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:40.634+0000 7f9034f51700 1 --2- 172.21.15.33:0/2901201084 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f903007de40 0x7f903019dad0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9020011a20 tx=0x7f9020011b00 comp rx=0 tx=0).stop 2024-03-19T02:43:41.278 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:40.634+0000 7f9034f51700 1 -- 172.21.15.33:0/2901201084 shutdown_connections 2024-03-19T02:43:41.278 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:40.634+0000 7f9034f51700 1 --2- 172.21.15.33:0/2901201084 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f901803ca60 0x7f901803ee50 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:41.278 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:40.634+0000 7f9034f51700 1 --2- 172.21.15.33:0/2901201084 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f903007de40 0x7f903019dad0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:41.278 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:40.634+0000 7f9034f51700 1 --2- 172.21.15.33:0/2901201084 >> [v2:172.21.15.33:6800/906645748,v1:172.21.15.33:6801/906645748] conn(0x7f9018038fe0 0x7f901803b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:41.278 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:40.635+0000 7f9034f51700 1 -- 172.21.15.33:0/2901201084 >> 172.21.15.33:0/2901201084 conn(0x7f90301030c0 msgr2=0x7f9030103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:41.278 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:40.635+0000 7f9034f51700 1 -- 172.21.15.33:0/2901201084 shutdown_connections 2024-03-19T02:43:41.279 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:40.635+0000 7f9034f51700 1 -- 172.21.15.33:0/2901201084 wait complete. 2024-03-19T02:43:41.279 INFO:teuthology.orchestra.run.smithi033.stdout:mgr epoch 9 is available 2024-03-19T02:43:41.279 INFO:teuthology.orchestra.run.smithi033.stdout:Generating a dashboard self-signed certificate... 2024-03-19T02:43:42.742 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-19T02:43:42.742 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.939+0000 7f5bc70ac700 1 Processor -- start 2024-03-19T02:43:42.742 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.939+0000 7f5bc70ac700 1 -- start start 2024-03-19T02:43:42.742 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.940+0000 7f5bc70ac700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bc0108520 0x7f5bc010a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:42.742 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.940+0000 7f5bc70ac700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc007be80 con 0x7f5bc0108520 2024-03-19T02:43:42.742 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.940+0000 7f5bc4e48700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bc0108520 0x7f5bc010a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:42.742 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.940+0000 7f5bc4e48700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bc0108520 0x7f5bc010a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34436/0 (socket says 172.21.15.33:34436) 2024-03-19T02:43:42.742 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.940+0000 7f5bc4e48700 1 -- 172.21.15.33:0/3597829294 learned_addr learned my addr 172.21.15.33:0/3597829294 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:42.743 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.940+0000 7f5bc4e48700 1 -- 172.21.15.33:0/3597829294 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc010ae50 con 0x7f5bc0108520 2024-03-19T02:43:42.743 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.940+0000 7f5bc4e48700 1 --2- 172.21.15.33:0/3597829294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bc0108520 0x7f5bc010a910 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5bb40098a0 tx=0x7f5bb4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b1f724a6e948d1a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:42.743 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.940+0000 7f5bbf7fe700 1 -- 172.21.15.33:0/3597829294 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5bb4004160 con 0x7f5bc0108520 2024-03-19T02:43:42.743 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.941+0000 7f5bbf7fe700 1 -- 172.21.15.33:0/3597829294 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5bb4004300 con 0x7f5bc0108520 2024-03-19T02:43:42.743 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.941+0000 7f5bbf7fe700 1 -- 172.21.15.33:0/3597829294 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5bb4011a50 con 0x7f5bc0108520 2024-03-19T02:43:42.743 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.941+0000 7f5bc70ac700 1 -- 172.21.15.33:0/3597829294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bc0108520 msgr2=0x7f5bc010a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:42.743 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.941+0000 7f5bc70ac700 1 --2- 172.21.15.33:0/3597829294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bc0108520 0x7f5bc010a910 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5bb40098a0 tx=0x7f5bb4009bb0 comp rx=0 tx=0).stop 2024-03-19T02:43:42.743 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.941+0000 7f5bc70ac700 1 -- 172.21.15.33:0/3597829294 shutdown_connections 2024-03-19T02:43:42.743 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.941+0000 7f5bc70ac700 1 --2- 172.21.15.33:0/3597829294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bc0108520 0x7f5bc010a910 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:42.743 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.941+0000 7f5bc70ac700 1 -- 172.21.15.33:0/3597829294 >> 172.21.15.33:0/3597829294 conn(0x7f5bc0101fb0 msgr2=0x7f5bc01043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:42.743 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.941+0000 7f5bc70ac700 1 -- 172.21.15.33:0/3597829294 shutdown_connections 2024-03-19T02:43:42.743 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.941+0000 7f5bc70ac700 1 -- 172.21.15.33:0/3597829294 wait complete. 2024-03-19T02:43:42.743 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.942+0000 7f5bc70ac700 1 Processor -- start 2024-03-19T02:43:42.743 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.942+0000 7f5bc70ac700 1 -- start start 2024-03-19T02:43:42.744 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.942+0000 7f5bc70ac700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bc0108520 0x7f5bc0107da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:42.744 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.942+0000 7f5bc70ac700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc010be60 con 0x7f5bc0108520 2024-03-19T02:43:42.744 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.942+0000 7f5bc4e48700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bc0108520 0x7f5bc0107da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:42.744 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.942+0000 7f5bc4e48700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bc0108520 0x7f5bc0107da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34450/0 (socket says 172.21.15.33:34450) 2024-03-19T02:43:42.744 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.942+0000 7f5bc4e48700 1 -- 172.21.15.33:0/1365362230 learned_addr learned my addr 172.21.15.33:0/1365362230 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:42.744 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.942+0000 7f5bc4e48700 1 -- 172.21.15.33:0/1365362230 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc01082e0 con 0x7f5bc0108520 2024-03-19T02:43:42.744 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.943+0000 7f5bc4e48700 1 --2- 172.21.15.33:0/1365362230 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bc0108520 0x7f5bc0107da0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5bb4011a20 tx=0x7f5bb4011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:42.744 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.943+0000 7f5bbdffb700 1 -- 172.21.15.33:0/1365362230 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5bb4003b70 con 0x7f5bc0108520 2024-03-19T02:43:42.744 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.943+0000 7f5bbdffb700 1 -- 172.21.15.33:0/1365362230 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5bb401a8e0 con 0x7f5bc0108520 2024-03-19T02:43:42.744 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.943+0000 7f5bc70ac700 1 -- 172.21.15.33:0/1365362230 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bc01064d0 con 0x7f5bc0108520 2024-03-19T02:43:42.744 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.943+0000 7f5bbdffb700 1 -- 172.21.15.33:0/1365362230 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5bb4024990 con 0x7f5bc0108520 2024-03-19T02:43:42.744 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.943+0000 7f5bc70ac700 1 -- 172.21.15.33:0/1365362230 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bc0106a10 con 0x7f5bc0108520 2024-03-19T02:43:42.744 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.944+0000 7f5bc70ac700 1 -- 172.21.15.33:0/1365362230 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bc0063940 con 0x7f5bc0108520 2024-03-19T02:43:42.744 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.944+0000 7f5bbdffb700 1 -- 172.21.15.33:0/1365362230 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 11) ==== 46013+0+0 (secure 0 0 0) 0x7f5bb4022020 con 0x7f5bc0108520 2024-03-19T02:43:42.744 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.944+0000 7f5bbdffb700 1 --2- 172.21.15.33:0/1365362230 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5bb0038f20 0x7f5bb003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:42.744 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.944+0000 7f5bbdffb700 1 -- 172.21.15.33:0/1365362230 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5bb404d0e0 con 0x7f5bc0108520 2024-03-19T02:43:42.745 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.944+0000 7f5bbffff700 1 --2- 172.21.15.33:0/1365362230 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5bb0038f20 0x7f5bb003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:42.745 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.945+0000 7f5bbffff700 1 --2- 172.21.15.33:0/1365362230 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5bb0038f20 0x7f5bb003b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5bac006fd0 tx=0x7f5bac006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:42.745 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:41.949+0000 7f5bbdffb700 1 -- 172.21.15.33:0/1365362230 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5bb4024cf0 con 0x7f5bc0108520 2024-03-19T02:43:42.745 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:42.169+0000 7f5bc70ac700 1 -- 172.21.15.33:0/1365362230 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f5bc0107320 con 0x7f5bb0038f20 2024-03-19T02:43:42.745 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:42.252+0000 7f5bbdffb700 1 -- 172.21.15.33:0/1365362230 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f5bc0107320 con 0x7f5bb0038f20 2024-03-19T02:43:42.745 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:42.255+0000 7f5bc70ac700 1 -- 172.21.15.33:0/1365362230 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5bb0038f20 msgr2=0x7f5bb003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:42.745 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:42.255+0000 7f5bc70ac700 1 --2- 172.21.15.33:0/1365362230 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5bb0038f20 0x7f5bb003b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5bac006fd0 tx=0x7f5bac006e40 comp rx=0 tx=0).stop 2024-03-19T02:43:42.745 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:42.255+0000 7f5bc70ac700 1 -- 172.21.15.33:0/1365362230 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bc0108520 msgr2=0x7f5bc0107da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:42.745 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:42.255+0000 7f5bc70ac700 1 --2- 172.21.15.33:0/1365362230 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bc0108520 0x7f5bc0107da0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5bb4011a20 tx=0x7f5bb4011b00 comp rx=0 tx=0).stop 2024-03-19T02:43:42.745 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:42.255+0000 7f5bc70ac700 1 -- 172.21.15.33:0/1365362230 shutdown_connections 2024-03-19T02:43:42.745 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:42.255+0000 7f5bc70ac700 1 --2- 172.21.15.33:0/1365362230 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5bb0038f20 0x7f5bb003b3e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:42.745 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:42.255+0000 7f5bc70ac700 1 --2- 172.21.15.33:0/1365362230 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bc0108520 0x7f5bc0107da0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:42.745 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:42.255+0000 7f5bc70ac700 1 -- 172.21.15.33:0/1365362230 >> 172.21.15.33:0/1365362230 conn(0x7f5bc0101fb0 msgr2=0x7f5bc01028c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:42.745 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:42.255+0000 7f5bc70ac700 1 -- 172.21.15.33:0/1365362230 shutdown_connections 2024-03-19T02:43:42.745 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:42.255+0000 7f5bc70ac700 1 -- 172.21.15.33:0/1365362230 wait complete. 2024-03-19T02:43:42.745 INFO:teuthology.orchestra.run.smithi033.stdout:Creating initial admin user... 2024-03-19T02:43:42.862 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:42 smithi033 ceph-mon[31973]: [19/Mar/2024:02:43:41] ENGINE Bus STARTING 2024-03-19T02:43:42.862 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:42 smithi033 ceph-mon[31973]: [19/Mar/2024:02:43:41] ENGINE Serving on https://172.21.15.33:7150 2024-03-19T02:43:42.862 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:42 smithi033 ceph-mon[31973]: [19/Mar/2024:02:43:41] ENGINE Serving on http://172.21.15.33:8765 2024-03-19T02:43:42.862 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:42 smithi033 ceph-mon[31973]: [19/Mar/2024:02:43:41] ENGINE Bus STARTED 2024-03-19T02:43:42.862 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:42 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:43:42.862 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:42 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:43:42.862 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:42 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:43:43.667 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:43 smithi033 ceph-mon[31973]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:43:43.668 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:43 smithi033 ceph-mon[31973]: mgrmap e12: x(active, since 2s) 2024-03-19T02:43:43.668 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:43 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:43:44.755 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:44 smithi033 ceph-mon[31973]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:43:44.755 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:44 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:43:45.072 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$J5O/1CSm9RNLv.a4VScXnOYreGGf8QbGkRa2LFIMxF90RDtL1ktwS", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710816223, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-19T02:43:45.072 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.166+0000 7f599047d700 1 Processor -- start 2024-03-19T02:43:45.072 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.166+0000 7f599047d700 1 -- start start 2024-03-19T02:43:45.072 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.166+0000 7f599047d700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5988106990 0x7f5988106db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:45.072 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.167+0000 7f599047d700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5988107380 con 0x7f5988106990 2024-03-19T02:43:45.072 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.167+0000 7f598e219700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5988106990 0x7f5988106db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:45.072 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.167+0000 7f598e219700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5988106990 0x7f5988106db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34464/0 (socket says 172.21.15.33:34464) 2024-03-19T02:43:45.073 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.167+0000 7f598e219700 1 -- 172.21.15.33:0/761223432 learned_addr learned my addr 172.21.15.33:0/761223432 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:45.073 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.167+0000 7f598e219700 1 -- 172.21.15.33:0/761223432 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5988107b90 con 0x7f5988106990 2024-03-19T02:43:45.073 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.167+0000 7f598e219700 1 --2- 172.21.15.33:0/761223432 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5988106990 0x7f5988106db0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f59780098a0 tx=0x7f5978009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef5e8dac13d7bf63 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:45.073 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.167+0000 7f598d217700 1 -- 172.21.15.33:0/761223432 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5978004160 con 0x7f5988106990 2024-03-19T02:43:45.073 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.167+0000 7f598d217700 1 -- 172.21.15.33:0/761223432 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5978004300 con 0x7f5988106990 2024-03-19T02:43:45.073 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.167+0000 7f598d217700 1 -- 172.21.15.33:0/761223432 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5978011a50 con 0x7f5988106990 2024-03-19T02:43:45.073 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.168+0000 7f599047d700 1 -- 172.21.15.33:0/761223432 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5988106990 msgr2=0x7f5988106db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:45.073 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.168+0000 7f599047d700 1 --2- 172.21.15.33:0/761223432 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5988106990 0x7f5988106db0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f59780098a0 tx=0x7f5978009bb0 comp rx=0 tx=0).stop 2024-03-19T02:43:45.073 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.168+0000 7f599047d700 1 -- 172.21.15.33:0/761223432 shutdown_connections 2024-03-19T02:43:45.073 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.168+0000 7f599047d700 1 --2- 172.21.15.33:0/761223432 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5988106990 0x7f5988106db0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:45.073 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.168+0000 7f599047d700 1 -- 172.21.15.33:0/761223432 >> 172.21.15.33:0/761223432 conn(0x7f5988101f30 msgr2=0x7f5988104370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:45.073 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.168+0000 7f599047d700 1 -- 172.21.15.33:0/761223432 shutdown_connections 2024-03-19T02:43:45.074 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.168+0000 7f599047d700 1 -- 172.21.15.33:0/761223432 wait complete. 2024-03-19T02:43:45.074 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.169+0000 7f599047d700 1 Processor -- start 2024-03-19T02:43:45.074 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.169+0000 7f599047d700 1 -- start start 2024-03-19T02:43:45.074 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.169+0000 7f599047d700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5988106990 0x7f598819daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:45.074 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.169+0000 7f599047d700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f598807afa0 con 0x7f5988106990 2024-03-19T02:43:45.074 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.169+0000 7f598e219700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5988106990 0x7f598819daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:45.074 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.169+0000 7f598e219700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5988106990 0x7f598819daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34478/0 (socket says 172.21.15.33:34478) 2024-03-19T02:43:45.074 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.169+0000 7f598e219700 1 -- 172.21.15.33:0/2239215020 learned_addr learned my addr 172.21.15.33:0/2239215020 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:45.074 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.169+0000 7f598e219700 1 -- 172.21.15.33:0/2239215020 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f598819dfe0 con 0x7f5988106990 2024-03-19T02:43:45.074 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.169+0000 7f598e219700 1 --2- 172.21.15.33:0/2239215020 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5988106990 0x7f598819daa0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5978006ba0 tx=0x7f5978011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:45.074 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.169+0000 7f597f7fe700 1 -- 172.21.15.33:0/2239215020 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5978003af0 con 0x7f5988106990 2024-03-19T02:43:45.074 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.170+0000 7f597f7fe700 1 -- 172.21.15.33:0/2239215020 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f597801a8e0 con 0x7f5988106990 2024-03-19T02:43:45.074 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.170+0000 7f599047d700 1 -- 172.21.15.33:0/2239215020 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f598819e270 con 0x7f5988106990 2024-03-19T02:43:45.075 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.170+0000 7f597f7fe700 1 -- 172.21.15.33:0/2239215020 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5978024900 con 0x7f5988106990 2024-03-19T02:43:45.075 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.170+0000 7f599047d700 1 -- 172.21.15.33:0/2239215020 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59881a0f60 con 0x7f5988106990 2024-03-19T02:43:45.075 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.170+0000 7f597f7fe700 1 -- 172.21.15.33:0/2239215020 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7f5978022020 con 0x7f5988106990 2024-03-19T02:43:45.075 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.171+0000 7f597f7fe700 1 --2- 172.21.15.33:0/2239215020 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5974039030 0x7f597403b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:45.075 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.171+0000 7f597f7fe700 1 -- 172.21.15.33:0/2239215020 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f597804d320 con 0x7f5988106990 2024-03-19T02:43:45.075 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.171+0000 7f598da18700 1 --2- 172.21.15.33:0/2239215020 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5974039030 0x7f597403b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:45.075 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.171+0000 7f599047d700 1 -- 172.21.15.33:0/2239215020 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5988063940 con 0x7f5988106990 2024-03-19T02:43:45.075 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.171+0000 7f598da18700 1 --2- 172.21.15.33:0/2239215020 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5974039030 0x7f597403b4f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5984006fd0 tx=0x7f5984006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:45.075 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.176+0000 7f597f7fe700 1 -- 172.21.15.33:0/2239215020 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f597800fd80 con 0x7f5988106990 2024-03-19T02:43:45.075 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.417+0000 7f599047d700 1 -- 172.21.15.33:0/2239215020 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f5988191450 con 0x7f5974039030 2024-03-19T02:43:45.075 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.664+0000 7f597f7fe700 1 -- 172.21.15.33:0/2239215020 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f5988191450 con 0x7f5974039030 2024-03-19T02:43:45.075 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.666+0000 7f599047d700 1 -- 172.21.15.33:0/2239215020 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5974039030 msgr2=0x7f597403b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:45.075 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.666+0000 7f599047d700 1 --2- 172.21.15.33:0/2239215020 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5974039030 0x7f597403b4f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5984006fd0 tx=0x7f5984006e40 comp rx=0 tx=0).stop 2024-03-19T02:43:45.076 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.666+0000 7f599047d700 1 -- 172.21.15.33:0/2239215020 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5988106990 msgr2=0x7f598819daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:45.076 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.666+0000 7f599047d700 1 --2- 172.21.15.33:0/2239215020 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5988106990 0x7f598819daa0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5978006ba0 tx=0x7f5978011ad0 comp rx=0 tx=0).stop 2024-03-19T02:43:45.076 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.666+0000 7f599047d700 1 -- 172.21.15.33:0/2239215020 shutdown_connections 2024-03-19T02:43:45.076 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.667+0000 7f599047d700 1 --2- 172.21.15.33:0/2239215020 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5974039030 0x7f597403b4f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:45.076 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.667+0000 7f599047d700 1 --2- 172.21.15.33:0/2239215020 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5988106990 0x7f598819daa0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:45.076 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.667+0000 7f599047d700 1 -- 172.21.15.33:0/2239215020 >> 172.21.15.33:0/2239215020 conn(0x7f5988101f30 msgr2=0x7f598818fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:45.076 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.667+0000 7f599047d700 1 -- 172.21.15.33:0/2239215020 shutdown_connections 2024-03-19T02:43:45.076 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:43.667+0000 7f599047d700 1 -- 172.21.15.33:0/2239215020 wait complete. 2024-03-19T02:43:45.076 INFO:teuthology.orchestra.run.smithi033.stdout:Fetching dashboard port number... 2024-03-19T02:43:45.993 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:45 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3541858407' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-19T02:43:45.994 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:45 smithi033 ceph-mon[31973]: mgrmap e13: x(active, since 6s) 2024-03-19T02:43:46.244 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stdout 8443 2024-03-19T02:43:46.244 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.476+0000 7f1cfd930700 1 Processor -- start 2024-03-19T02:43:46.244 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.476+0000 7f1cfd930700 1 -- start start 2024-03-19T02:43:46.244 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.476+0000 7f1cfd930700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1cf8107a60 0x7f1cf8107e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:46.244 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.476+0000 7f1cfd930700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf8108450 con 0x7f1cf8107a60 2024-03-19T02:43:46.244 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.476+0000 7f1cf6ffd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1cf8107a60 0x7f1cf8107e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:46.244 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.476+0000 7f1cf6ffd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1cf8107a60 0x7f1cf8107e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46396/0 (socket says 172.21.15.33:46396) 2024-03-19T02:43:46.244 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.476+0000 7f1cf6ffd700 1 -- 172.21.15.33:0/708353243 learned_addr learned my addr 172.21.15.33:0/708353243 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:46.244 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.476+0000 7f1cf6ffd700 1 -- 172.21.15.33:0/708353243 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cf8108c60 con 0x7f1cf8107a60 2024-03-19T02:43:46.245 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.476+0000 7f1cf6ffd700 1 --2- 172.21.15.33:0/708353243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1cf8107a60 0x7f1cf8107e80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1ce80098a0 tx=0x7f1ce8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6329c1af0830378c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:46.245 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.477+0000 7f1cf5ffb700 1 -- 172.21.15.33:0/708353243 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ce8004160 con 0x7f1cf8107a60 2024-03-19T02:43:46.245 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.477+0000 7f1cf5ffb700 1 -- 172.21.15.33:0/708353243 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ce8004300 con 0x7f1cf8107a60 2024-03-19T02:43:46.245 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.477+0000 7f1cf5ffb700 1 -- 172.21.15.33:0/708353243 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ce8011a50 con 0x7f1cf8107a60 2024-03-19T02:43:46.245 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.477+0000 7f1cfd930700 1 -- 172.21.15.33:0/708353243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1cf8107a60 msgr2=0x7f1cf8107e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:46.245 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.477+0000 7f1cfd930700 1 --2- 172.21.15.33:0/708353243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1cf8107a60 0x7f1cf8107e80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1ce80098a0 tx=0x7f1ce8009bb0 comp rx=0 tx=0).stop 2024-03-19T02:43:46.245 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.477+0000 7f1cfd930700 1 -- 172.21.15.33:0/708353243 shutdown_connections 2024-03-19T02:43:46.245 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.477+0000 7f1cfd930700 1 --2- 172.21.15.33:0/708353243 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1cf8107a60 0x7f1cf8107e80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:46.245 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.477+0000 7f1cfd930700 1 -- 172.21.15.33:0/708353243 >> 172.21.15.33:0/708353243 conn(0x7f1cf8103020 msgr2=0x7f1cf8105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:46.245 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.477+0000 7f1cfd930700 1 -- 172.21.15.33:0/708353243 shutdown_connections 2024-03-19T02:43:46.245 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.477+0000 7f1cfd930700 1 -- 172.21.15.33:0/708353243 wait complete. 2024-03-19T02:43:46.245 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.478+0000 7f1cfd930700 1 Processor -- start 2024-03-19T02:43:46.245 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.478+0000 7f1cfd930700 1 -- start start 2024-03-19T02:43:46.245 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.478+0000 7f1cfd930700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1cf8107a60 0x7f1cf819b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:46.245 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.478+0000 7f1cfd930700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf810c010 con 0x7f1cf8107a60 2024-03-19T02:43:46.246 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.478+0000 7f1cf6ffd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1cf8107a60 0x7f1cf819b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:46.246 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.478+0000 7f1cf6ffd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1cf8107a60 0x7f1cf819b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46398/0 (socket says 172.21.15.33:46398) 2024-03-19T02:43:46.246 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.478+0000 7f1cf6ffd700 1 -- 172.21.15.33:0/3541858407 learned_addr learned my addr 172.21.15.33:0/3541858407 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:46.246 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.479+0000 7f1cf6ffd700 1 -- 172.21.15.33:0/3541858407 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cf819bde0 con 0x7f1cf8107a60 2024-03-19T02:43:46.246 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.479+0000 7f1cf6ffd700 1 --2- 172.21.15.33:0/3541858407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1cf8107a60 0x7f1cf819b8a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8006ba0 tx=0x7f1ce8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:46.246 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.479+0000 7f1cfc92e700 1 -- 172.21.15.33:0/3541858407 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ce8003af0 con 0x7f1cf8107a60 2024-03-19T02:43:46.246 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.479+0000 7f1cfc92e700 1 -- 172.21.15.33:0/3541858407 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ce801a8e0 con 0x7f1cf8107a60 2024-03-19T02:43:46.246 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.479+0000 7f1cfc92e700 1 -- 172.21.15.33:0/3541858407 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ce8024900 con 0x7f1cf8107a60 2024-03-19T02:43:46.246 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.479+0000 7f1cfd930700 1 -- 172.21.15.33:0/3541858407 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cf819c070 con 0x7f1cf8107a60 2024-03-19T02:43:46.246 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.479+0000 7f1cfd930700 1 -- 172.21.15.33:0/3541858407 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cf819ed60 con 0x7f1cf8107a60 2024-03-19T02:43:46.246 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.480+0000 7f1cfc92e700 1 -- 172.21.15.33:0/3541858407 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7f1ce8022020 con 0x7f1cf8107a60 2024-03-19T02:43:46.246 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.480+0000 7f1cfd930700 1 -- 172.21.15.33:0/3541858407 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cf8063940 con 0x7f1cf8107a60 2024-03-19T02:43:46.247 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.480+0000 7f1cfc92e700 1 --2- 172.21.15.33:0/3541858407 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f1ce0038fe0 0x7f1ce003b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:46.247 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.480+0000 7f1cfc92e700 1 -- 172.21.15.33:0/3541858407 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f1ce804d380 con 0x7f1cf8107a60 2024-03-19T02:43:46.247 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.480+0000 7f1cf67fc700 1 --2- 172.21.15.33:0/3541858407 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f1ce0038fe0 0x7f1ce003b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:46.247 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.481+0000 7f1cf67fc700 1 --2- 172.21.15.33:0/3541858407 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f1ce0038fe0 0x7f1ce003b4a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1cec006fd0 tx=0x7f1cec006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:46.247 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.485+0000 7f1cfc92e700 1 -- 172.21.15.33:0/3541858407 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1ce802c3a0 con 0x7f1cf8107a60 2024-03-19T02:43:46.247 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.689+0000 7f1cfd930700 1 -- 172.21.15.33:0/3541858407 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f1cf819ef30 con 0x7f1cf8107a60 2024-03-19T02:43:46.247 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.690+0000 7f1cfc92e700 1 -- 172.21.15.33:0/3541858407 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f1ce8024bc0 con 0x7f1cf8107a60 2024-03-19T02:43:46.247 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.692+0000 7f1cfd930700 1 -- 172.21.15.33:0/3541858407 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f1ce0038fe0 msgr2=0x7f1ce003b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:46.247 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.692+0000 7f1cfd930700 1 --2- 172.21.15.33:0/3541858407 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f1ce0038fe0 0x7f1ce003b4a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1cec006fd0 tx=0x7f1cec006e40 comp rx=0 tx=0).stop 2024-03-19T02:43:46.247 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.692+0000 7f1cfd930700 1 -- 172.21.15.33:0/3541858407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1cf8107a60 msgr2=0x7f1cf819b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:46.247 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.692+0000 7f1cfd930700 1 --2- 172.21.15.33:0/3541858407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1cf8107a60 0x7f1cf819b8a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8006ba0 tx=0x7f1ce8011ad0 comp rx=0 tx=0).stop 2024-03-19T02:43:46.247 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.692+0000 7f1cfd930700 1 -- 172.21.15.33:0/3541858407 shutdown_connections 2024-03-19T02:43:46.247 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.692+0000 7f1cfd930700 1 --2- 172.21.15.33:0/3541858407 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f1ce0038fe0 0x7f1ce003b4a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:46.247 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.692+0000 7f1cfd930700 1 --2- 172.21.15.33:0/3541858407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1cf8107a60 0x7f1cf819b8a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:46.248 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.692+0000 7f1cfd930700 1 -- 172.21.15.33:0/3541858407 >> 172.21.15.33:0/3541858407 conn(0x7f1cf8103020 msgr2=0x7f1cf81039f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:46.248 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.692+0000 7f1cfd930700 1 -- 172.21.15.33:0/3541858407 shutdown_connections 2024-03-19T02:43:46.248 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:45.692+0000 7f1cfd930700 1 -- 172.21.15.33:0/3541858407 wait complete. 2024-03-19T02:43:46.251 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-19T02:43:46.251 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout disabled 2024-03-19T02:43:46.259 INFO:teuthology.orchestra.run.smithi033.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-19T02:43:46.259 INFO:teuthology.orchestra.run.smithi033.stdout:systemctl: stdout inactive 2024-03-19T02:43:46.259 INFO:teuthology.orchestra.run.smithi033.stdout:firewalld.service is not enabled 2024-03-19T02:43:46.259 INFO:teuthology.orchestra.run.smithi033.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-19T02:43:46.261 INFO:teuthology.orchestra.run.smithi033.stdout:Ceph Dashboard is now available at: 2024-03-19T02:43:46.261 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:43:46.261 INFO:teuthology.orchestra.run.smithi033.stdout: URL: https://smithi033.front.sepia.ceph.com:8443/ 2024-03-19T02:43:46.261 INFO:teuthology.orchestra.run.smithi033.stdout: User: admin 2024-03-19T02:43:46.261 INFO:teuthology.orchestra.run.smithi033.stdout: Password: 44xd1i29km 2024-03-19T02:43:46.261 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:43:46.262 INFO:teuthology.orchestra.run.smithi033.stdout:Saving cluster configuration to /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config directory 2024-03-19T02:43:46.262 INFO:teuthology.orchestra.run.smithi033.stdout:Enabling autotune for osd_memory_target 2024-03-19T02:43:47.408 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.635+0000 7f7b991a7700 1 Processor -- start 2024-03-19T02:43:47.408 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.636+0000 7f7b991a7700 1 -- start start 2024-03-19T02:43:47.408 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.636+0000 7f7b991a7700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b94108540 0x7f7b9410a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:47.408 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.636+0000 7f7b991a7700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7b9407be40 con 0x7f7b94108540 2024-03-19T02:43:47.408 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.636+0000 7f7b92d9d700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b94108540 0x7f7b9410a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:47.408 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.636+0000 7f7b92d9d700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b94108540 0x7f7b9410a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46412/0 (socket says 172.21.15.33:46412) 2024-03-19T02:43:47.408 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.636+0000 7f7b92d9d700 1 -- 172.21.15.33:0/2722143587 learned_addr learned my addr 172.21.15.33:0/2722143587 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:47.409 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.636+0000 7f7b92d9d700 1 -- 172.21.15.33:0/2722143587 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b9410ae70 con 0x7f7b94108540 2024-03-19T02:43:47.409 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.637+0000 7f7b92d9d700 1 --2- 172.21.15.33:0/2722143587 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b94108540 0x7f7b9410a930 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7b840098a0 tx=0x7f7b84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf7ad9aa56e89019 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:47.409 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.637+0000 7f7b91d9b700 1 -- 172.21.15.33:0/2722143587 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b84004160 con 0x7f7b94108540 2024-03-19T02:43:47.409 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.637+0000 7f7b91d9b700 1 -- 172.21.15.33:0/2722143587 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7b84004300 con 0x7f7b94108540 2024-03-19T02:43:47.409 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.637+0000 7f7b91d9b700 1 -- 172.21.15.33:0/2722143587 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b84011a50 con 0x7f7b94108540 2024-03-19T02:43:47.409 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.637+0000 7f7b991a7700 1 -- 172.21.15.33:0/2722143587 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b94108540 msgr2=0x7f7b9410a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:47.409 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.637+0000 7f7b991a7700 1 --2- 172.21.15.33:0/2722143587 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b94108540 0x7f7b9410a930 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7b840098a0 tx=0x7f7b84009bb0 comp rx=0 tx=0).stop 2024-03-19T02:43:47.409 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.638+0000 7f7b991a7700 1 -- 172.21.15.33:0/2722143587 shutdown_connections 2024-03-19T02:43:47.409 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.638+0000 7f7b991a7700 1 --2- 172.21.15.33:0/2722143587 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b94108540 0x7f7b9410a930 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:47.409 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.638+0000 7f7b991a7700 1 -- 172.21.15.33:0/2722143587 >> 172.21.15.33:0/2722143587 conn(0x7f7b94101f30 msgr2=0x7f7b94104370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:47.409 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.638+0000 7f7b991a7700 1 -- 172.21.15.33:0/2722143587 shutdown_connections 2024-03-19T02:43:47.409 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.638+0000 7f7b991a7700 1 -- 172.21.15.33:0/2722143587 wait complete. 2024-03-19T02:43:47.409 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.638+0000 7f7b991a7700 1 Processor -- start 2024-03-19T02:43:47.410 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.638+0000 7f7b991a7700 1 -- start start 2024-03-19T02:43:47.410 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.639+0000 7f7b991a7700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b94108540 0x7f7b9419b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:47.410 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.639+0000 7f7b991a7700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7b9410be70 con 0x7f7b94108540 2024-03-19T02:43:47.410 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.639+0000 7f7b92d9d700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b94108540 0x7f7b9419b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:47.410 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.639+0000 7f7b92d9d700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b94108540 0x7f7b9419b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46416/0 (socket says 172.21.15.33:46416) 2024-03-19T02:43:47.410 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.639+0000 7f7b92d9d700 1 -- 172.21.15.33:0/3637266852 learned_addr learned my addr 172.21.15.33:0/3637266852 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:47.410 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.639+0000 7f7b92d9d700 1 -- 172.21.15.33:0/3637266852 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b9419bda0 con 0x7f7b94108540 2024-03-19T02:43:47.410 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.639+0000 7f7b92d9d700 1 --2- 172.21.15.33:0/3637266852 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b94108540 0x7f7b9419b860 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7b84011a20 tx=0x7f7b84011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:47.410 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.639+0000 7f7b7bfff700 1 -- 172.21.15.33:0/3637266852 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b84003b70 con 0x7f7b94108540 2024-03-19T02:43:47.410 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.639+0000 7f7b991a7700 1 -- 172.21.15.33:0/3637266852 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b9419c030 con 0x7f7b94108540 2024-03-19T02:43:47.410 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.639+0000 7f7b7bfff700 1 -- 172.21.15.33:0/3637266852 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7b8401a8e0 con 0x7f7b94108540 2024-03-19T02:43:47.410 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.639+0000 7f7b7bfff700 1 -- 172.21.15.33:0/3637266852 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b84024990 con 0x7f7b94108540 2024-03-19T02:43:47.410 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.639+0000 7f7b991a7700 1 -- 172.21.15.33:0/3637266852 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b9419ed20 con 0x7f7b94108540 2024-03-19T02:43:47.411 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.640+0000 7f7b7bfff700 1 -- 172.21.15.33:0/3637266852 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f7b84022020 con 0x7f7b94108540 2024-03-19T02:43:47.411 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.640+0000 7f7b7bfff700 1 --2- 172.21.15.33:0/3637266852 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7b7c0390e0 0x7f7b7c03b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:47.411 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.640+0000 7f7b991a7700 1 -- 172.21.15.33:0/3637266852 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b94063940 con 0x7f7b94108540 2024-03-19T02:43:47.411 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.640+0000 7f7b7bfff700 1 -- 172.21.15.33:0/3637266852 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f7b8404d760 con 0x7f7b94108540 2024-03-19T02:43:47.411 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.641+0000 7f7b9259c700 1 --2- 172.21.15.33:0/3637266852 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7b7c0390e0 0x7f7b7c03b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:47.411 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.641+0000 7f7b9259c700 1 --2- 172.21.15.33:0/3637266852 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7b7c0390e0 0x7f7b7c03b5a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7b88006fd0 tx=0x7f7b88006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:47.411 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.646+0000 7f7b7bfff700 1 -- 172.21.15.33:0/3637266852 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7b8400b3e0 con 0x7f7b94108540 2024-03-19T02:43:47.411 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.851+0000 7f7b991a7700 1 -- 172.21.15.33:0/3637266852 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f7b9419eef0 con 0x7f7b94108540 2024-03-19T02:43:47.411 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.851+0000 7f7b7bfff700 1 -- 172.21.15.33:0/3637266852 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f7b840172b0 con 0x7f7b94108540 2024-03-19T02:43:47.411 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.853+0000 7f7b991a7700 1 -- 172.21.15.33:0/3637266852 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7b7c0390e0 msgr2=0x7f7b7c03b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:47.411 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.853+0000 7f7b991a7700 1 --2- 172.21.15.33:0/3637266852 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7b7c0390e0 0x7f7b7c03b5a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7b88006fd0 tx=0x7f7b88006e40 comp rx=0 tx=0).stop 2024-03-19T02:43:47.411 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.853+0000 7f7b991a7700 1 -- 172.21.15.33:0/3637266852 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b94108540 msgr2=0x7f7b9419b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:47.411 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.853+0000 7f7b991a7700 1 --2- 172.21.15.33:0/3637266852 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b94108540 0x7f7b9419b860 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7b84011a20 tx=0x7f7b84011b00 comp rx=0 tx=0).stop 2024-03-19T02:43:47.412 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.853+0000 7f7b991a7700 1 -- 172.21.15.33:0/3637266852 shutdown_connections 2024-03-19T02:43:47.412 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.853+0000 7f7b991a7700 1 --2- 172.21.15.33:0/3637266852 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7b7c0390e0 0x7f7b7c03b5a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:47.412 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.853+0000 7f7b991a7700 1 --2- 172.21.15.33:0/3637266852 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b94108540 0x7f7b9419b860 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:47.412 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.853+0000 7f7b991a7700 1 -- 172.21.15.33:0/3637266852 >> 172.21.15.33:0/3637266852 conn(0x7f7b94101f30 msgr2=0x7f7b94102900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:47.412 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.853+0000 7f7b991a7700 1 -- 172.21.15.33:0/3637266852 shutdown_connections 2024-03-19T02:43:47.412 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:46.853+0000 7f7b991a7700 1 -- 172.21.15.33:0/3637266852 wait complete. 2024-03-19T02:43:48.773 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.872+0000 7f0d0ab47700 1 Processor -- start 2024-03-19T02:43:48.773 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.872+0000 7f0d0ab47700 1 -- start start 2024-03-19T02:43:48.773 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.872+0000 7f0d0ab47700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d04108580 0x7f0d0410a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:48.773 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.872+0000 7f0d0ab47700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0d0407be20 con 0x7f0d04108580 2024-03-19T02:43:48.774 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.872+0000 7f0d088e3700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d04108580 0x7f0d0410a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:48.774 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.872+0000 7f0d088e3700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d04108580 0x7f0d0410a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46432/0 (socket says 172.21.15.33:46432) 2024-03-19T02:43:48.774 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.872+0000 7f0d088e3700 1 -- 172.21.15.33:0/2428086079 learned_addr learned my addr 172.21.15.33:0/2428086079 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:48.774 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.873+0000 7f0d088e3700 1 -- 172.21.15.33:0/2428086079 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d0410aeb0 con 0x7f0d04108580 2024-03-19T02:43:48.774 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.873+0000 7f0d088e3700 1 --2- 172.21.15.33:0/2428086079 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d04108580 0x7f0d0410a970 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0cf80098a0 tx=0x7f0cf8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23d614d7e88cf4d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:48.774 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.873+0000 7f0d037fe700 1 -- 172.21.15.33:0/2428086079 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0cf8004160 con 0x7f0d04108580 2024-03-19T02:43:48.774 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.873+0000 7f0d037fe700 1 -- 172.21.15.33:0/2428086079 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0cf8004300 con 0x7f0d04108580 2024-03-19T02:43:48.774 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.873+0000 7f0d037fe700 1 -- 172.21.15.33:0/2428086079 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0cf8011a50 con 0x7f0d04108580 2024-03-19T02:43:48.774 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.873+0000 7f0d0ab47700 1 -- 172.21.15.33:0/2428086079 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d04108580 msgr2=0x7f0d0410a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:48.774 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.874+0000 7f0d0ab47700 1 --2- 172.21.15.33:0/2428086079 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d04108580 0x7f0d0410a970 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0cf80098a0 tx=0x7f0cf8009bb0 comp rx=0 tx=0).stop 2024-03-19T02:43:48.774 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.874+0000 7f0d0ab47700 1 -- 172.21.15.33:0/2428086079 shutdown_connections 2024-03-19T02:43:48.774 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.874+0000 7f0d0ab47700 1 --2- 172.21.15.33:0/2428086079 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d04108580 0x7f0d0410a970 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:48.774 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.874+0000 7f0d0ab47700 1 -- 172.21.15.33:0/2428086079 >> 172.21.15.33:0/2428086079 conn(0x7f0d04101fb0 msgr2=0x7f0d041043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:48.774 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.874+0000 7f0d0ab47700 1 -- 172.21.15.33:0/2428086079 shutdown_connections 2024-03-19T02:43:48.775 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.874+0000 7f0d0ab47700 1 -- 172.21.15.33:0/2428086079 wait complete. 2024-03-19T02:43:48.775 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.874+0000 7f0d0ab47700 1 Processor -- start 2024-03-19T02:43:48.775 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.874+0000 7f0d0ab47700 1 -- start start 2024-03-19T02:43:48.775 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.874+0000 7f0d0ab47700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d04108580 0x7f0d04107e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:48.775 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.874+0000 7f0d0ab47700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0d0410be60 con 0x7f0d04108580 2024-03-19T02:43:48.775 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.874+0000 7f0d088e3700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d04108580 0x7f0d04107e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:48.775 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.875+0000 7f0d088e3700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d04108580 0x7f0d04107e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46448/0 (socket says 172.21.15.33:46448) 2024-03-19T02:43:48.775 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.875+0000 7f0d088e3700 1 -- 172.21.15.33:0/1516102130 learned_addr learned my addr 172.21.15.33:0/1516102130 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:48.775 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.875+0000 7f0d088e3700 1 -- 172.21.15.33:0/1516102130 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d04108340 con 0x7f0d04108580 2024-03-19T02:43:48.775 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.875+0000 7f0d088e3700 1 --2- 172.21.15.33:0/1516102130 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d04108580 0x7f0d04107e00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8011a20 tx=0x7f0cf8011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:48.775 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.875+0000 7f0d01ffb700 1 -- 172.21.15.33:0/1516102130 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0cf8003b70 con 0x7f0d04108580 2024-03-19T02:43:48.775 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.875+0000 7f0d01ffb700 1 -- 172.21.15.33:0/1516102130 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0cf801a8e0 con 0x7f0d04108580 2024-03-19T02:43:48.775 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.875+0000 7f0d0ab47700 1 -- 172.21.15.33:0/1516102130 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d04106530 con 0x7f0d04108580 2024-03-19T02:43:48.775 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.875+0000 7f0d01ffb700 1 -- 172.21.15.33:0/1516102130 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0cf8024990 con 0x7f0d04108580 2024-03-19T02:43:48.775 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.875+0000 7f0d0ab47700 1 -- 172.21.15.33:0/1516102130 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d04106a70 con 0x7f0d04108580 2024-03-19T02:43:48.776 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.876+0000 7f0d01ffb700 1 -- 172.21.15.33:0/1516102130 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f0cf8022020 con 0x7f0d04108580 2024-03-19T02:43:48.776 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.876+0000 7f0d01ffb700 1 --2- 172.21.15.33:0/1516102130 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f0cf40390e0 0x7f0cf403b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:48.776 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.876+0000 7f0d01ffb700 1 -- 172.21.15.33:0/1516102130 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0cf804d370 con 0x7f0d04108580 2024-03-19T02:43:48.776 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.876+0000 7f0d0ab47700 1 -- 172.21.15.33:0/1516102130 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d04063940 con 0x7f0d04108580 2024-03-19T02:43:48.776 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.876+0000 7f0d03fff700 1 --2- 172.21.15.33:0/1516102130 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f0cf40390e0 0x7f0cf403b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:48.776 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.877+0000 7f0d03fff700 1 --2- 172.21.15.33:0/1516102130 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f0cf40390e0 0x7f0cf403b5a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0006fd0 tx=0x7f0cf0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:48.776 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:47.881+0000 7f0d01ffb700 1 -- 172.21.15.33:0/1516102130 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0cf802d910 con 0x7f0d04108580 2024-03-19T02:43:48.776 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:48.195+0000 7f0d0ab47700 1 -- 172.21.15.33:0/1516102130 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f0d04107110 con 0x7f0d04108580 2024-03-19T02:43:48.776 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:48.197+0000 7f0d01ffb700 1 -- 172.21.15.33:0/1516102130 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f0cf802d3f0 con 0x7f0d04108580 2024-03-19T02:43:48.776 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:48.199+0000 7f0d0ab47700 1 -- 172.21.15.33:0/1516102130 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f0cf40390e0 msgr2=0x7f0cf403b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:48.776 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:48.199+0000 7f0d0ab47700 1 --2- 172.21.15.33:0/1516102130 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f0cf40390e0 0x7f0cf403b5a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0006fd0 tx=0x7f0cf0006e40 comp rx=0 tx=0).stop 2024-03-19T02:43:48.776 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:48.199+0000 7f0d0ab47700 1 -- 172.21.15.33:0/1516102130 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d04108580 msgr2=0x7f0d04107e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:48.776 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:48.199+0000 7f0d0ab47700 1 --2- 172.21.15.33:0/1516102130 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d04108580 0x7f0d04107e00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8011a20 tx=0x7f0cf8011b00 comp rx=0 tx=0).stop 2024-03-19T02:43:48.776 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:48.199+0000 7f0d0ab47700 1 -- 172.21.15.33:0/1516102130 shutdown_connections 2024-03-19T02:43:48.776 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:48.199+0000 7f0d0ab47700 1 --2- 172.21.15.33:0/1516102130 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f0cf40390e0 0x7f0cf403b5a0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:48.777 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:48.199+0000 7f0d0ab47700 1 --2- 172.21.15.33:0/1516102130 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0d04108580 0x7f0d04107e00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:48.777 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:48.199+0000 7f0d0ab47700 1 -- 172.21.15.33:0/1516102130 >> 172.21.15.33:0/1516102130 conn(0x7f0d04101fb0 msgr2=0x7f0d04102930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:48.777 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:48.199+0000 7f0d0ab47700 1 -- 172.21.15.33:0/1516102130 shutdown_connections 2024-03-19T02:43:48.777 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr 2024-03-19T02:43:48.199+0000 7f0d0ab47700 1 -- 172.21.15.33:0/1516102130 wait complete. 2024-03-19T02:43:48.777 INFO:teuthology.orchestra.run.smithi033.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-19T02:43:48.777 INFO:teuthology.orchestra.run.smithi033.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-19T02:43:48.777 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:43:48.777 INFO:teuthology.orchestra.run.smithi033.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-19T02:43:48.777 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:43:48.777 INFO:teuthology.orchestra.run.smithi033.stdout:Or, if you are only running a single cluster on this host: 2024-03-19T02:43:48.777 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:43:48.777 INFO:teuthology.orchestra.run.smithi033.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-19T02:43:48.777 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:43:48.777 INFO:teuthology.orchestra.run.smithi033.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-19T02:43:48.777 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:43:48.777 INFO:teuthology.orchestra.run.smithi033.stdout: ceph telemetry on 2024-03-19T02:43:48.777 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:43:48.778 INFO:teuthology.orchestra.run.smithi033.stdout:For more information see: 2024-03-19T02:43:48.778 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:43:48.778 INFO:teuthology.orchestra.run.smithi033.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-19T02:43:48.778 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:43:48.778 INFO:teuthology.orchestra.run.smithi033.stdout:Bootstrap complete. 2024-03-19T02:43:48.814 INFO:tasks.cephadm:Fetching config... 2024-03-19T02:43:48.814 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-19T02:43:48.814 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-19T02:43:48.832 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-19T02:43:48.832 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-19T02:43:48.832 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-19T02:43:48.886 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-19T02:43:48.886 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-19T02:43:48.886 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-19T02:43:48.953 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-19T02:43:48.953 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-19T02:43:48.953 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-19T02:43:49.009 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-19T02:43:49.009 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9kiSC/4znDfhLrn3PHj6eqEVvOkdZ3IXVQRDbC420s/zEhdnDmIULUl+Wa0sCPnps1IBvJRpVls8wbIJg6oM7TcD9rvdhd4PJlOn5+UJMIPieTWM1hFaVcdvChj8XveijjD7V9R4pd78joeIMErB2ggVkiq4lwZTYVLMA8ppmEouuWzza2ldIC6n4X0ietPTSwv28Wl+JtxCN48g2x92T4ljLoICxNQ9LpJgkINshU6KU74iT+sxCRtZr50fUZGM8D4Mi9uKrLpWqiuzuvsIxsvap2Fod8XppxSSgOnJZPtg6ruYw6qaxsD5U5TUpzfadFIplheNgUUPArFE+bI3aA5XSJXBski4e++jhoxBHMvUpVKrPFofexxxe45MEYdxYSE33GKQO8IAcwLCO82iwtMhZ26+PGJIUZgsa6FwAd/Mtk3pBtSFCT/8MMAf9XExIMfzXHCM8xfWwPWcmEXn/EI5TYDZOgyQcEvsdYWe6vzD/zaMElcevSISszNbfGcU= ceph-2b7a8938-e59a-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-19T02:43:49.087 INFO:teuthology.orchestra.run.smithi033.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9kiSC/4znDfhLrn3PHj6eqEVvOkdZ3IXVQRDbC420s/zEhdnDmIULUl+Wa0sCPnps1IBvJRpVls8wbIJg6oM7TcD9rvdhd4PJlOn5+UJMIPieTWM1hFaVcdvChj8XveijjD7V9R4pd78joeIMErB2ggVkiq4lwZTYVLMA8ppmEouuWzza2ldIC6n4X0ietPTSwv28Wl+JtxCN48g2x92T4ljLoICxNQ9LpJgkINshU6KU74iT+sxCRtZr50fUZGM8D4Mi9uKrLpWqiuzuvsIxsvap2Fod8XppxSSgOnJZPtg6ruYw6qaxsD5U5TUpzfadFIplheNgUUPArFE+bI3aA5XSJXBski4e++jhoxBHMvUpVKrPFofexxxe45MEYdxYSE33GKQO8IAcwLCO82iwtMhZ26+PGJIUZgsa6FwAd/Mtk3pBtSFCT/8MMAf9XExIMfzXHCM8xfWwPWcmEXn/EI5TYDZOgyQcEvsdYWe6vzD/zaMElcevSISszNbfGcU= ceph-2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T02:43:49.103 DEBUG:teuthology.orchestra.run.smithi096:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9kiSC/4znDfhLrn3PHj6eqEVvOkdZ3IXVQRDbC420s/zEhdnDmIULUl+Wa0sCPnps1IBvJRpVls8wbIJg6oM7TcD9rvdhd4PJlOn5+UJMIPieTWM1hFaVcdvChj8XveijjD7V9R4pd78joeIMErB2ggVkiq4lwZTYVLMA8ppmEouuWzza2ldIC6n4X0ietPTSwv28Wl+JtxCN48g2x92T4ljLoICxNQ9LpJgkINshU6KU74iT+sxCRtZr50fUZGM8D4Mi9uKrLpWqiuzuvsIxsvap2Fod8XppxSSgOnJZPtg6ruYw6qaxsD5U5TUpzfadFIplheNgUUPArFE+bI3aA5XSJXBski4e++jhoxBHMvUpVKrPFofexxxe45MEYdxYSE33GKQO8IAcwLCO82iwtMhZ26+PGJIUZgsa6FwAd/Mtk3pBtSFCT/8MMAf9XExIMfzXHCM8xfWwPWcmEXn/EI5TYDZOgyQcEvsdYWe6vzD/zaMElcevSISszNbfGcU= ceph-2b7a8938-e59a-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-19T02:43:49.148 INFO:teuthology.orchestra.run.smithi096.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9kiSC/4znDfhLrn3PHj6eqEVvOkdZ3IXVQRDbC420s/zEhdnDmIULUl+Wa0sCPnps1IBvJRpVls8wbIJg6oM7TcD9rvdhd4PJlOn5+UJMIPieTWM1hFaVcdvChj8XveijjD7V9R4pd78joeIMErB2ggVkiq4lwZTYVLMA8ppmEouuWzza2ldIC6n4X0ietPTSwv28Wl+JtxCN48g2x92T4ljLoICxNQ9LpJgkINshU6KU74iT+sxCRtZr50fUZGM8D4Mi9uKrLpWqiuzuvsIxsvap2Fod8XppxSSgOnJZPtg6ruYw6qaxsD5U5TUpzfadFIplheNgUUPArFE+bI3aA5XSJXBski4e++jhoxBHMvUpVKrPFofexxxe45MEYdxYSE33GKQO8IAcwLCO82iwtMhZ26+PGJIUZgsa6FwAd/Mtk3pBtSFCT/8MMAf9XExIMfzXHCM8xfWwPWcmEXn/EI5TYDZOgyQcEvsdYWe6vzD/zaMElcevSISszNbfGcU= ceph-2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T02:43:49.162 DEBUG:teuthology.orchestra.run.smithi155:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9kiSC/4znDfhLrn3PHj6eqEVvOkdZ3IXVQRDbC420s/zEhdnDmIULUl+Wa0sCPnps1IBvJRpVls8wbIJg6oM7TcD9rvdhd4PJlOn5+UJMIPieTWM1hFaVcdvChj8XveijjD7V9R4pd78joeIMErB2ggVkiq4lwZTYVLMA8ppmEouuWzza2ldIC6n4X0ietPTSwv28Wl+JtxCN48g2x92T4ljLoICxNQ9LpJgkINshU6KU74iT+sxCRtZr50fUZGM8D4Mi9uKrLpWqiuzuvsIxsvap2Fod8XppxSSgOnJZPtg6ruYw6qaxsD5U5TUpzfadFIplheNgUUPArFE+bI3aA5XSJXBski4e++jhoxBHMvUpVKrPFofexxxe45MEYdxYSE33GKQO8IAcwLCO82iwtMhZ26+PGJIUZgsa6FwAd/Mtk3pBtSFCT/8MMAf9XExIMfzXHCM8xfWwPWcmEXn/EI5TYDZOgyQcEvsdYWe6vzD/zaMElcevSISszNbfGcU= ceph-2b7a8938-e59a-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-19T02:43:49.208 INFO:teuthology.orchestra.run.smithi155.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9kiSC/4znDfhLrn3PHj6eqEVvOkdZ3IXVQRDbC420s/zEhdnDmIULUl+Wa0sCPnps1IBvJRpVls8wbIJg6oM7TcD9rvdhd4PJlOn5+UJMIPieTWM1hFaVcdvChj8XveijjD7V9R4pd78joeIMErB2ggVkiq4lwZTYVLMA8ppmEouuWzza2ldIC6n4X0ietPTSwv28Wl+JtxCN48g2x92T4ljLoICxNQ9LpJgkINshU6KU74iT+sxCRtZr50fUZGM8D4Mi9uKrLpWqiuzuvsIxsvap2Fod8XppxSSgOnJZPtg6ruYw6qaxsD5U5TUpzfadFIplheNgUUPArFE+bI3aA5XSJXBski4e++jhoxBHMvUpVKrPFofexxxe45MEYdxYSE33GKQO8IAcwLCO82iwtMhZ26+PGJIUZgsa6FwAd/Mtk3pBtSFCT/8MMAf9XExIMfzXHCM8xfWwPWcmEXn/EI5TYDZOgyQcEvsdYWe6vzD/zaMElcevSISszNbfGcU= ceph-2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T02:43:49.223 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-19T02:43:49.464 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:43:49.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:49 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1516102130' entity='client.admin' 2024-03-19T02:43:51.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:51.926+0000 7fbe9907b700 1 -- 172.21.15.33:0/2225972795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe94100c40 msgr2=0x7fbe94101060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:51.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:51.926+0000 7fbe9907b700 1 --2- 172.21.15.33:0/2225972795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe94100c40 0x7fbe94101060 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c00b0a0 tx=0x7fbe8c00b3b0 comp rx=0 tx=0).stop 2024-03-19T02:43:51.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:51.926+0000 7fbe9907b700 1 -- 172.21.15.33:0/2225972795 shutdown_connections 2024-03-19T02:43:51.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:51.926+0000 7fbe9907b700 1 --2- 172.21.15.33:0/2225972795 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe94100c40 0x7fbe94101060 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:51.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:51.926+0000 7fbe9907b700 1 -- 172.21.15.33:0/2225972795 >> 172.21.15.33:0/2225972795 conn(0x7fbe940fc1e0 msgr2=0x7fbe940fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:51.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:51.926+0000 7fbe9907b700 1 -- 172.21.15.33:0/2225972795 shutdown_connections 2024-03-19T02:43:51.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:51.926+0000 7fbe9907b700 1 -- 172.21.15.33:0/2225972795 wait complete. 2024-03-19T02:43:51.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:51.926+0000 7fbe9907b700 1 Processor -- start 2024-03-19T02:43:51.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:51.927+0000 7fbe9907b700 1 -- start start 2024-03-19T02:43:51.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:51.927+0000 7fbe9907b700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe94100c40 0x7fbe9406a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:51.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:51.927+0000 7fbe9907b700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbe94106b80 con 0x7fbe94100c40 2024-03-19T02:43:51.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:51.927+0000 7fbe92d9d700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe94100c40 0x7fbe9406a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:51.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:51.927+0000 7fbe92d9d700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe94100c40 0x7fbe9406a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46468/0 (socket says 172.21.15.33:46468) 2024-03-19T02:43:51.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:51.927+0000 7fbe92d9d700 1 -- 172.21.15.33:0/2974005307 learned_addr learned my addr 172.21.15.33:0/2974005307 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:51.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:51.927+0000 7fbe92d9d700 1 -- 172.21.15.33:0/2974005307 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe94197e10 con 0x7fbe94100c40 2024-03-19T02:43:51.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:51.927+0000 7fbe92d9d700 1 --2- 172.21.15.33:0/2974005307 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe94100c40 0x7fbe9406a180 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c00b830 tx=0x7fbe8c003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:51.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:51.927+0000 7fbe8bfff700 1 -- 172.21.15.33:0/2974005307 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe8c017040 con 0x7fbe94100c40 2024-03-19T02:43:51.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:51.928+0000 7fbe8bfff700 1 -- 172.21.15.33:0/2974005307 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe8c0044f0 con 0x7fbe94100c40 2024-03-19T02:43:51.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:51.928+0000 7fbe9907b700 1 -- 172.21.15.33:0/2974005307 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe9419c420 con 0x7fbe94100c40 2024-03-19T02:43:51.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:51.928+0000 7fbe8bfff700 1 -- 172.21.15.33:0/2974005307 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe8c01d8e0 con 0x7fbe94100c40 2024-03-19T02:43:51.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:51.928+0000 7fbe9907b700 1 -- 172.21.15.33:0/2974005307 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe9419c7c0 con 0x7fbe94100c40 2024-03-19T02:43:51.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:51.928+0000 7fbe8bfff700 1 -- 172.21.15.33:0/2974005307 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fbe8c024020 con 0x7fbe94100c40 2024-03-19T02:43:51.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:51.929+0000 7fbe8bfff700 1 --2- 172.21.15.33:0/2974005307 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbe80039010 0x7fbe8003b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:51.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:51.929+0000 7fbe8bfff700 1 -- 172.21.15.33:0/2974005307 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fbe8c022020 con 0x7fbe94100c40 2024-03-19T02:43:51.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:51.929+0000 7fbe9259c700 1 --2- 172.21.15.33:0/2974005307 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbe80039010 0x7fbe8003b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:51.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:51.929+0000 7fbe9907b700 1 -- 172.21.15.33:0/2974005307 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe94107300 con 0x7fbe94100c40 2024-03-19T02:43:51.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:51.929+0000 7fbe9259c700 1 --2- 172.21.15.33:0/2974005307 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbe80039010 0x7fbe8003b4d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c006fd0 tx=0x7fbe7c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:51.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:51.934+0000 7fbe8bfff700 1 -- 172.21.15.33:0/2974005307 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbe8c011c20 con 0x7fbe94100c40 2024-03-19T02:43:52.086 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:51 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:43:52.086 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:51 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:43:52.086 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:51 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi033", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:43:52.086 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:51 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:43:52.086 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:51 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:43:52.086 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:51 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:43:52.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:52.139+0000 7fbe9907b700 1 -- 172.21.15.33:0/2974005307 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7fbe9402dcf0 con 0x7fbe94100c40 2024-03-19T02:43:52.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:52.141+0000 7fbe8bfff700 1 -- 172.21.15.33:0/2974005307 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7fbe8c01a380 con 0x7fbe94100c40 2024-03-19T02:43:52.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:52.143+0000 7fbe9907b700 1 -- 172.21.15.33:0/2974005307 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbe80039010 msgr2=0x7fbe8003b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:52.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:52.143+0000 7fbe9907b700 1 --2- 172.21.15.33:0/2974005307 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbe80039010 0x7fbe8003b4d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c006fd0 tx=0x7fbe7c006e40 comp rx=0 tx=0).stop 2024-03-19T02:43:52.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:52.143+0000 7fbe9907b700 1 -- 172.21.15.33:0/2974005307 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe94100c40 msgr2=0x7fbe9406a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:52.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:52.143+0000 7fbe9907b700 1 --2- 172.21.15.33:0/2974005307 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe94100c40 0x7fbe9406a180 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c00b830 tx=0x7fbe8c003ce0 comp rx=0 tx=0).stop 2024-03-19T02:43:52.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:52.144+0000 7fbe9907b700 1 -- 172.21.15.33:0/2974005307 shutdown_connections 2024-03-19T02:43:52.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:52.144+0000 7fbe9907b700 1 --2- 172.21.15.33:0/2974005307 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbe80039010 0x7fbe8003b4d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:52.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:52.144+0000 7fbe9907b700 1 --2- 172.21.15.33:0/2974005307 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe94100c40 0x7fbe9406a180 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:52.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:52.144+0000 7fbe9907b700 1 -- 172.21.15.33:0/2974005307 >> 172.21.15.33:0/2974005307 conn(0x7fbe940fc1e0 msgr2=0x7fbe940fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:52.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:52.144+0000 7fbe9907b700 1 -- 172.21.15.33:0/2974005307 shutdown_connections 2024-03-19T02:43:52.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:52.144+0000 7fbe9907b700 1 -- 172.21.15.33:0/2974005307 wait complete. 2024-03-19T02:43:52.858 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-19T02:43:52.859 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-19T02:43:53.403 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:53 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2974005307' entity='client.admin' 2024-03-19T02:43:53.403 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:53 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:43:53.404 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:53 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:43:53.526 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:43:55.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.051+0000 7f74daebc700 1 -- 172.21.15.33:0/1436186232 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74d4101d60 msgr2=0x7f74d4102180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:55.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.051+0000 7f74daebc700 1 --2- 172.21.15.33:0/1436186232 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74d4101d60 0x7f74d4102180 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f74c8009910 tx=0x7f74c8009c20 comp rx=0 tx=0).stop 2024-03-19T02:43:55.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.051+0000 7f74daebc700 1 -- 172.21.15.33:0/1436186232 shutdown_connections 2024-03-19T02:43:55.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.051+0000 7f74daebc700 1 --2- 172.21.15.33:0/1436186232 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74d4101d60 0x7f74d4102180 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:55.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.051+0000 7f74daebc700 1 -- 172.21.15.33:0/1436186232 >> 172.21.15.33:0/1436186232 conn(0x7f74d40fd300 msgr2=0x7f74d40ff740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:55.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.051+0000 7f74daebc700 1 -- 172.21.15.33:0/1436186232 shutdown_connections 2024-03-19T02:43:55.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.051+0000 7f74daebc700 1 -- 172.21.15.33:0/1436186232 wait complete. 2024-03-19T02:43:55.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.052+0000 7f74daebc700 1 Processor -- start 2024-03-19T02:43:55.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.052+0000 7f74daebc700 1 -- start start 2024-03-19T02:43:55.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.052+0000 7f74daebc700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74d4101d60 0x7f74d41973a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:55.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.052+0000 7f74daebc700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f74d4107ca0 con 0x7f74d4101d60 2024-03-19T02:43:55.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.052+0000 7f74d8c58700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74d4101d60 0x7f74d41973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:55.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.053+0000 7f74d8c58700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74d4101d60 0x7f74d41973a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53538/0 (socket says 172.21.15.33:53538) 2024-03-19T02:43:55.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.053+0000 7f74d8c58700 1 -- 172.21.15.33:0/6184115 learned_addr learned my addr 172.21.15.33:0/6184115 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:55.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.053+0000 7f74d8c58700 1 -- 172.21.15.33:0/6184115 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74d41978e0 con 0x7f74d4101d60 2024-03-19T02:43:55.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.053+0000 7f74d8c58700 1 --2- 172.21.15.33:0/6184115 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74d4101d60 0x7f74d41973a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f74c8005950 tx=0x7f74c8004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:55.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.053+0000 7f74d1ffb700 1 -- 172.21.15.33:0/6184115 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74c8026070 con 0x7f74d4101d60 2024-03-19T02:43:55.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.053+0000 7f74d1ffb700 1 -- 172.21.15.33:0/6184115 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f74c800bc00 con 0x7f74d4101d60 2024-03-19T02:43:55.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.053+0000 7f74daebc700 1 -- 172.21.15.33:0/6184115 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74d4197b70 con 0x7f74d4101d60 2024-03-19T02:43:55.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.053+0000 7f74d1ffb700 1 -- 172.21.15.33:0/6184115 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74c80039a0 con 0x7f74d4101d60 2024-03-19T02:43:55.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.053+0000 7f74daebc700 1 -- 172.21.15.33:0/6184115 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74d419a860 con 0x7f74d4101d60 2024-03-19T02:43:55.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.054+0000 7f74d1ffb700 1 -- 172.21.15.33:0/6184115 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f74c8029070 con 0x7f74d4101d60 2024-03-19T02:43:55.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.054+0000 7f74daebc700 1 -- 172.21.15.33:0/6184115 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74d4108420 con 0x7f74d4101d60 2024-03-19T02:43:55.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.054+0000 7f74d1ffb700 1 --2- 172.21.15.33:0/6184115 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f74c4038fc0 0x7f74c403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:55.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.054+0000 7f74d1ffb700 1 -- 172.21.15.33:0/6184115 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f74c801e070 con 0x7f74d4101d60 2024-03-19T02:43:55.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.054+0000 7f74d3fff700 1 --2- 172.21.15.33:0/6184115 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f74c4038fc0 0x7f74c403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:55.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.055+0000 7f74d3fff700 1 --2- 172.21.15.33:0/6184115 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f74c4038fc0 0x7f74c403b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f74c0006fd0 tx=0x7f74c0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:55.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.059+0000 7f74d1ffb700 1 -- 172.21.15.33:0/6184115 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f74c8013070 con 0x7f74d4101d60 2024-03-19T02:43:55.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.265+0000 7f74daebc700 1 -- 172.21.15.33:0/6184115 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f74d402dcc0 con 0x7f74c4038fc0 2024-03-19T02:43:55.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.269+0000 7f74d1ffb700 1 -- 172.21.15.33:0/6184115 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f74d402dcc0 con 0x7f74c4038fc0 2024-03-19T02:43:55.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.271+0000 7f74daebc700 1 -- 172.21.15.33:0/6184115 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f74c4038fc0 msgr2=0x7f74c403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:55.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.271+0000 7f74daebc700 1 --2- 172.21.15.33:0/6184115 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f74c4038fc0 0x7f74c403b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f74c0006fd0 tx=0x7f74c0006e40 comp rx=0 tx=0).stop 2024-03-19T02:43:55.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.271+0000 7f74daebc700 1 -- 172.21.15.33:0/6184115 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74d4101d60 msgr2=0x7f74d41973a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:55.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.271+0000 7f74daebc700 1 --2- 172.21.15.33:0/6184115 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74d4101d60 0x7f74d41973a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f74c8005950 tx=0x7f74c8004fa0 comp rx=0 tx=0).stop 2024-03-19T02:43:55.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.271+0000 7f74daebc700 1 -- 172.21.15.33:0/6184115 shutdown_connections 2024-03-19T02:43:55.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.271+0000 7f74daebc700 1 --2- 172.21.15.33:0/6184115 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f74c4038fc0 0x7f74c403b480 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:55.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.271+0000 7f74daebc700 1 --2- 172.21.15.33:0/6184115 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74d4101d60 0x7f74d41973a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:55.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.271+0000 7f74daebc700 1 -- 172.21.15.33:0/6184115 >> 172.21.15.33:0/6184115 conn(0x7f74d40fd300 msgr2=0x7f74d40fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:55.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.271+0000 7f74daebc700 1 -- 172.21.15.33:0/6184115 shutdown_connections 2024-03-19T02:43:55.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:55.271+0000 7f74daebc700 1 -- 172.21.15.33:0/6184115 wait complete. 2024-03-19T02:43:55.979 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi096 2024-03-19T02:43:55.979 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-19T02:43:55.979 DEBUG:teuthology.orchestra.run.smithi096:> dd of=/etc/ceph/ceph.conf 2024-03-19T02:43:55.999 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-19T02:43:55.999 DEBUG:teuthology.orchestra.run.smithi096:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:43:56.056 INFO:tasks.cephadm:Adding host smithi096 to orchestrator... 2024-03-19T02:43:56.056 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph orch host add smithi096 2024-03-19T02:43:56.299 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:43:56.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:56 smithi033 ceph-mon[31973]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:43:56.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:56 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:43:56.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:56 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:43:56.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:56 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:43:56.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:56 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:43:56.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:56 smithi033 ceph-mon[31973]: Updating smithi033:/etc/ceph/ceph.conf 2024-03-19T02:43:56.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:56 smithi033 ceph-mon[31973]: Updating smithi033:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:43:56.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:56 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:43:56.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:56 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:43:56.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:56 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:43:57.507 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:57 smithi033 ceph-mon[31973]: Updating smithi033:/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:43:57.507 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:57 smithi033 ceph-mon[31973]: Updating smithi033:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T02:43:57.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:57.715+0000 7fd957c39700 1 -- 172.21.15.33:0/3019150339 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd950101f50 msgr2=0x7fd950102330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:43:57.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:57.715+0000 7fd957c39700 1 --2- 172.21.15.33:0/3019150339 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd950101f50 0x7fd950102330 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd94c009910 tx=0x7fd94c009c20 comp rx=0 tx=0).stop 2024-03-19T02:43:57.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:57.715+0000 7fd957c39700 1 -- 172.21.15.33:0/3019150339 shutdown_connections 2024-03-19T02:43:57.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:57.715+0000 7fd957c39700 1 --2- 172.21.15.33:0/3019150339 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd950101f50 0x7fd950102330 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:43:57.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:57.715+0000 7fd957c39700 1 -- 172.21.15.33:0/3019150339 >> 172.21.15.33:0/3019150339 conn(0x7fd9500fd820 msgr2=0x7fd9500ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:43:57.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:57.715+0000 7fd957c39700 1 -- 172.21.15.33:0/3019150339 shutdown_connections 2024-03-19T02:43:57.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:57.715+0000 7fd957c39700 1 -- 172.21.15.33:0/3019150339 wait complete. 2024-03-19T02:43:57.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:57.715+0000 7fd957c39700 1 Processor -- start 2024-03-19T02:43:57.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:57.715+0000 7fd957c39700 1 -- start start 2024-03-19T02:43:57.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:57.716+0000 7fd957c39700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd950101f50 0x7fd95019d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:57.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:57.716+0000 7fd957c39700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd95010c1d0 con 0x7fd950101f50 2024-03-19T02:43:57.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:57.716+0000 7fd9559d5700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd950101f50 0x7fd95019d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:57.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:57.716+0000 7fd9559d5700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd950101f50 0x7fd95019d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53554/0 (socket says 172.21.15.33:53554) 2024-03-19T02:43:57.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:57.716+0000 7fd9559d5700 1 -- 172.21.15.33:0/2456996489 learned_addr learned my addr 172.21.15.33:0/2456996489 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:43:57.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:57.716+0000 7fd9559d5700 1 -- 172.21.15.33:0/2456996489 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd95019d910 con 0x7fd950101f50 2024-03-19T02:43:57.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:57.716+0000 7fd9559d5700 1 --2- 172.21.15.33:0/2456996489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd950101f50 0x7fd95019d3d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd94c0056d0 tx=0x7fd94c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:57.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:57.716+0000 7fd946ffd700 1 -- 172.21.15.33:0/2456996489 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd94c026070 con 0x7fd950101f50 2024-03-19T02:43:57.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:57.716+0000 7fd946ffd700 1 -- 172.21.15.33:0/2456996489 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd94c00bba0 con 0x7fd950101f50 2024-03-19T02:43:57.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:57.716+0000 7fd957c39700 1 -- 172.21.15.33:0/2456996489 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd95019dc00 con 0x7fd950101f50 2024-03-19T02:43:57.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:57.716+0000 7fd946ffd700 1 -- 172.21.15.33:0/2456996489 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd94c0037f0 con 0x7fd950101f50 2024-03-19T02:43:57.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:57.716+0000 7fd957c39700 1 -- 172.21.15.33:0/2456996489 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd950197590 con 0x7fd950101f50 2024-03-19T02:43:57.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:57.717+0000 7fd946ffd700 1 -- 172.21.15.33:0/2456996489 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fd94c029070 con 0x7fd950101f50 2024-03-19T02:43:57.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:57.717+0000 7fd946ffd700 1 --2- 172.21.15.33:0/2456996489 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fd93c039020 0x7fd93c03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:43:57.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:57.717+0000 7fd946ffd700 1 -- 172.21.15.33:0/2456996489 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd94c01e070 con 0x7fd950101f50 2024-03-19T02:43:57.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:57.717+0000 7fd957c39700 1 -- 172.21.15.33:0/2456996489 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd950103430 con 0x7fd950101f50 2024-03-19T02:43:57.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:57.718+0000 7fd9551d4700 1 --2- 172.21.15.33:0/2456996489 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fd93c039020 0x7fd93c03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:43:57.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:57.718+0000 7fd9551d4700 1 --2- 172.21.15.33:0/2456996489 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fd93c039020 0x7fd93c03b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd940006fd0 tx=0x7fd940006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:43:57.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:57.723+0000 7fd946ffd700 1 -- 172.21.15.33:0/2456996489 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd94c030070 con 0x7fd950101f50 2024-03-19T02:43:57.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:43:57.930+0000 7fd957c39700 1 -- 172.21.15.33:0/2456996489 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi096", "target": ["mon-mgr", ""]}) -- 0x7fd950063940 con 0x7fd93c039020 2024-03-19T02:43:58.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:58 smithi033 ceph-mon[31973]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi096", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:44:00.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:59 smithi033 ceph-mon[31973]: Deploying cephadm binary to smithi096 2024-03-19T02:44:00.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:43:59 smithi033 ceph-mon[31973]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:02.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:01 smithi033 ceph-mon[31973]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:02.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:02.445+0000 7fd946ffd700 1 -- 172.21.15.33:0/2456996489 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fd950063940 con 0x7fd93c039020 2024-03-19T02:44:02.447 INFO:teuthology.orchestra.run.smithi033.stdout:Added host 'smithi096' with addr '172.21.15.96' 2024-03-19T02:44:02.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:02.448+0000 7fd957c39700 1 -- 172.21.15.33:0/2456996489 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fd93c039020 msgr2=0x7fd93c03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:02.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:02.448+0000 7fd957c39700 1 --2- 172.21.15.33:0/2456996489 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fd93c039020 0x7fd93c03b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd940006fd0 tx=0x7fd940006e40 comp rx=0 tx=0).stop 2024-03-19T02:44:02.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:02.448+0000 7fd957c39700 1 -- 172.21.15.33:0/2456996489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd950101f50 msgr2=0x7fd95019d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:02.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:02.448+0000 7fd957c39700 1 --2- 172.21.15.33:0/2456996489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd950101f50 0x7fd95019d3d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd94c0056d0 tx=0x7fd94c0057b0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:02.448+0000 7fd957c39700 1 -- 172.21.15.33:0/2456996489 shutdown_connections 2024-03-19T02:44:02.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:02.448+0000 7fd957c39700 1 --2- 172.21.15.33:0/2456996489 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fd93c039020 0x7fd93c03b4e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:02.448+0000 7fd957c39700 1 --2- 172.21.15.33:0/2456996489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd950101f50 0x7fd95019d3d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:02.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:02.448+0000 7fd957c39700 1 -- 172.21.15.33:0/2456996489 >> 172.21.15.33:0/2456996489 conn(0x7fd9500fd820 msgr2=0x7fd950103920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:02.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:02.448+0000 7fd957c39700 1 -- 172.21.15.33:0/2456996489 shutdown_connections 2024-03-19T02:44:02.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:02.448+0000 7fd957c39700 1 -- 172.21.15.33:0/2456996489 wait complete. 2024-03-19T02:44:03.055 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-19T02:44:03.302 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:44:03.674 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:03 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:03 smithi033 ceph-mon[31973]: Added host smithi096 2024-03-19T02:44:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:03 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:44:03.675 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:03 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:04.715 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:04 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:04.716 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:04 smithi033 ceph-mon[31973]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:04.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:04.959+0000 7f1725c56700 1 -- 172.21.15.33:0/2455587996 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1720100e40 msgr2=0x7f1720101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:04.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:04.959+0000 7f1725c56700 1 --2- 172.21.15.33:0/2455587996 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1720100e40 0x7f1720101220 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1710009910 tx=0x7f1710009c20 comp rx=0 tx=0).stop 2024-03-19T02:44:04.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:04.959+0000 7f1725c56700 1 -- 172.21.15.33:0/2455587996 shutdown_connections 2024-03-19T02:44:04.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:04.959+0000 7f1725c56700 1 --2- 172.21.15.33:0/2455587996 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1720100e40 0x7f1720101220 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:04.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:04.959+0000 7f1725c56700 1 -- 172.21.15.33:0/2455587996 >> 172.21.15.33:0/2455587996 conn(0x7f1720077850 msgr2=0x7f17200feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:04.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:04.959+0000 7f1725c56700 1 -- 172.21.15.33:0/2455587996 shutdown_connections 2024-03-19T02:44:04.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:04.959+0000 7f1725c56700 1 -- 172.21.15.33:0/2455587996 wait complete. 2024-03-19T02:44:04.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:04.960+0000 7f1725c56700 1 Processor -- start 2024-03-19T02:44:04.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:04.960+0000 7f1725c56700 1 -- start start 2024-03-19T02:44:04.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:04.960+0000 7f1725c56700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1720100e40 0x7f172019d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:04.960+0000 7f1725c56700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f172010b0c0 con 0x7f1720100e40 2024-03-19T02:44:04.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:04.960+0000 7f171f7fe700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1720100e40 0x7f172019d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:04.960+0000 7f171f7fe700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1720100e40 0x7f172019d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45392/0 (socket says 172.21.15.33:45392) 2024-03-19T02:44:04.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:04.960+0000 7f171f7fe700 1 -- 172.21.15.33:0/3450144465 learned_addr learned my addr 172.21.15.33:0/3450144465 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:44:04.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:04.960+0000 7f171f7fe700 1 -- 172.21.15.33:0/3450144465 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f172019d910 con 0x7f1720100e40 2024-03-19T02:44:04.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:04.960+0000 7f171f7fe700 1 --2- 172.21.15.33:0/3450144465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1720100e40 0x7f172019d3d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f17100056d0 tx=0x7f17100057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:04.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:04.961+0000 7f171cff9700 1 -- 172.21.15.33:0/3450144465 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1710026070 con 0x7f1720100e40 2024-03-19T02:44:04.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:04.961+0000 7f171cff9700 1 -- 172.21.15.33:0/3450144465 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f171000bba0 con 0x7f1720100e40 2024-03-19T02:44:04.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:04.961+0000 7f1725c56700 1 -- 172.21.15.33:0/3450144465 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f172019dc00 con 0x7f1720100e40 2024-03-19T02:44:04.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:04.961+0000 7f171cff9700 1 -- 172.21.15.33:0/3450144465 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f17100037f0 con 0x7f1720100e40 2024-03-19T02:44:04.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:04.961+0000 7f1725c56700 1 -- 172.21.15.33:0/3450144465 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1720197590 con 0x7f1720100e40 2024-03-19T02:44:04.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:04.961+0000 7f171cff9700 1 -- 172.21.15.33:0/3450144465 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f1710029070 con 0x7f1720100e40 2024-03-19T02:44:04.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:04.962+0000 7f1725c56700 1 -- 172.21.15.33:0/3450144465 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1720102320 con 0x7f1720100e40 2024-03-19T02:44:04.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:04.962+0000 7f171cff9700 1 --2- 172.21.15.33:0/3450144465 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f1708038fc0 0x7f170803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:04.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:04.962+0000 7f171cff9700 1 -- 172.21.15.33:0/3450144465 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f171001e070 con 0x7f1720100e40 2024-03-19T02:44:04.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:04.962+0000 7f171effd700 1 --2- 172.21.15.33:0/3450144465 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f1708038fc0 0x7f170803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:04.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:04.962+0000 7f171effd700 1 --2- 172.21.15.33:0/3450144465 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f1708038fc0 0x7f170803b480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1714006fd0 tx=0x7f1714006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:04.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:04.967+0000 7f171cff9700 1 -- 172.21.15.33:0/3450144465 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1710013070 con 0x7f1720100e40 2024-03-19T02:44:05.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:05.177+0000 7f1725c56700 1 -- 172.21.15.33:0/3450144465 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1720198ac0 con 0x7f1708038fc0 2024-03-19T02:44:05.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:05.178+0000 7f171cff9700 1 -- 172.21.15.33:0/3450144465 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7f1720198ac0 con 0x7f1708038fc0 2024-03-19T02:44:05.179 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:44:05.179 INFO:teuthology.orchestra.run.smithi033.stdout:[{"addr": "172.21.15.33", "hostname": "smithi033", "labels": [], "status": ""}, {"addr": "172.21.15.96", "hostname": "smithi096", "labels": [], "status": ""}] 2024-03-19T02:44:05.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:05.180+0000 7f1725c56700 1 -- 172.21.15.33:0/3450144465 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f1708038fc0 msgr2=0x7f170803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:05.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:05.180+0000 7f1725c56700 1 --2- 172.21.15.33:0/3450144465 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f1708038fc0 0x7f170803b480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1714006fd0 tx=0x7f1714006e40 comp rx=0 tx=0).stop 2024-03-19T02:44:05.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:05.181+0000 7f1725c56700 1 -- 172.21.15.33:0/3450144465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1720100e40 msgr2=0x7f172019d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:05.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:05.181+0000 7f1725c56700 1 --2- 172.21.15.33:0/3450144465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1720100e40 0x7f172019d3d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f17100056d0 tx=0x7f17100057b0 comp rx=0 tx=0).stop 2024-03-19T02:44:05.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:05.181+0000 7f1725c56700 1 -- 172.21.15.33:0/3450144465 shutdown_connections 2024-03-19T02:44:05.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:05.181+0000 7f1725c56700 1 --2- 172.21.15.33:0/3450144465 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f1708038fc0 0x7f170803b480 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:05.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:05.181+0000 7f1725c56700 1 --2- 172.21.15.33:0/3450144465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1720100e40 0x7f172019d3d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:05.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:05.181+0000 7f1725c56700 1 -- 172.21.15.33:0/3450144465 >> 172.21.15.33:0/3450144465 conn(0x7f1720077850 msgr2=0x7f1720069700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:05.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:05.181+0000 7f1725c56700 1 -- 172.21.15.33:0/3450144465 shutdown_connections 2024-03-19T02:44:05.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:05.181+0000 7f1725c56700 1 -- 172.21.15.33:0/3450144465 wait complete. 2024-03-19T02:44:05.865 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi155 2024-03-19T02:44:05.865 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-19T02:44:05.865 DEBUG:teuthology.orchestra.run.smithi155:> dd of=/etc/ceph/ceph.conf 2024-03-19T02:44:05.885 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-19T02:44:05.885 DEBUG:teuthology.orchestra.run.smithi155:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:44:05.943 INFO:tasks.cephadm:Adding host smithi155 to orchestrator... 2024-03-19T02:44:05.943 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph orch host add smithi155 2024-03-19T02:44:06.188 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:44:06.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:06 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:06.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:06 smithi033 ceph-mon[31973]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:44:06.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:06 smithi033 ceph-mon[31973]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:07.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:07.700+0000 7f9d753df700 1 -- 172.21.15.33:0/3891145903 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d70100e30 msgr2=0x7f9d70101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:07.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:07.700+0000 7f9d753df700 1 --2- 172.21.15.33:0/3891145903 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d70100e30 0x7f9d70101210 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9d68009910 tx=0x7f9d68009c20 comp rx=0 tx=0).stop 2024-03-19T02:44:07.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:07.700+0000 7f9d753df700 1 -- 172.21.15.33:0/3891145903 shutdown_connections 2024-03-19T02:44:07.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:07.700+0000 7f9d753df700 1 --2- 172.21.15.33:0/3891145903 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d70100e30 0x7f9d70101210 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:07.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:07.700+0000 7f9d753df700 1 -- 172.21.15.33:0/3891145903 >> 172.21.15.33:0/3891145903 conn(0x7f9d700fc6c0 msgr2=0x7f9d700feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:07.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:07.700+0000 7f9d753df700 1 -- 172.21.15.33:0/3891145903 shutdown_connections 2024-03-19T02:44:07.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:07.700+0000 7f9d753df700 1 -- 172.21.15.33:0/3891145903 wait complete. 2024-03-19T02:44:07.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:07.700+0000 7f9d753df700 1 Processor -- start 2024-03-19T02:44:07.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:07.701+0000 7f9d753df700 1 -- start start 2024-03-19T02:44:07.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:07.701+0000 7f9d753df700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d70100e30 0x7f9d70196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:07.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:07.701+0000 7f9d753df700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7010b0b0 con 0x7f9d70100e30 2024-03-19T02:44:07.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:07.701+0000 7f9d6effd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d70100e30 0x7f9d70196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:07.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:07.701+0000 7f9d6effd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d70100e30 0x7f9d70196380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:45414/0 (socket says 172.21.15.33:45414) 2024-03-19T02:44:07.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:07.701+0000 7f9d6effd700 1 -- 172.21.15.33:0/3028206803 learned_addr learned my addr 172.21.15.33:0/3028206803 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:44:07.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:07.701+0000 7f9d6effd700 1 -- 172.21.15.33:0/3028206803 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d7006ad90 con 0x7f9d70100e30 2024-03-19T02:44:07.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:07.701+0000 7f9d6effd700 1 --2- 172.21.15.33:0/3028206803 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d70100e30 0x7f9d70196380 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9d68006010 tx=0x7f9d68005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:07.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:07.701+0000 7f9d67fff700 1 -- 172.21.15.33:0/3028206803 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9d68026070 con 0x7f9d70100e30 2024-03-19T02:44:07.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:07.701+0000 7f9d67fff700 1 -- 172.21.15.33:0/3028206803 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9d6800bba0 con 0x7f9d70100e30 2024-03-19T02:44:07.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:07.701+0000 7f9d753df700 1 -- 172.21.15.33:0/3028206803 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d701968c0 con 0x7f9d70100e30 2024-03-19T02:44:07.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:07.701+0000 7f9d67fff700 1 -- 172.21.15.33:0/3028206803 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9d680038a0 con 0x7f9d70100e30 2024-03-19T02:44:07.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:07.702+0000 7f9d753df700 1 -- 172.21.15.33:0/3028206803 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d70196d80 con 0x7f9d70100e30 2024-03-19T02:44:07.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:07.702+0000 7f9d67fff700 1 -- 172.21.15.33:0/3028206803 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f9d68029070 con 0x7f9d70100e30 2024-03-19T02:44:07.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:07.702+0000 7f9d753df700 1 -- 172.21.15.33:0/3028206803 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d70102310 con 0x7f9d70100e30 2024-03-19T02:44:07.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:07.703+0000 7f9d67fff700 1 --2- 172.21.15.33:0/3028206803 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9d5c038fc0 0x7f9d5c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:07.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:07.703+0000 7f9d67fff700 1 -- 172.21.15.33:0/3028206803 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f9d6801e070 con 0x7f9d70100e30 2024-03-19T02:44:07.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:07.703+0000 7f9d6e7fc700 1 --2- 172.21.15.33:0/3028206803 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9d5c038fc0 0x7f9d5c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:07.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:07.703+0000 7f9d6e7fc700 1 --2- 172.21.15.33:0/3028206803 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9d5c038fc0 0x7f9d5c03b480 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9d58006fd0 tx=0x7f9d58006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:07.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:07.707+0000 7f9d67fff700 1 -- 172.21.15.33:0/3028206803 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9d68013070 con 0x7f9d70100e30 2024-03-19T02:44:07.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:07 smithi033 ceph-mon[31973]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:07.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:07.918+0000 7f9d753df700 1 -- 172.21.15.33:0/3028206803 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi155", "target": ["mon-mgr", ""]}) -- 0x7f9d7006a580 con 0x7f9d5c038fc0 2024-03-19T02:44:09.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:09 smithi033 ceph-mon[31973]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi155", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:44:09.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:09 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:09.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:09 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:09.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:09 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:09.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:09 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:09.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:09 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi096", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:44:09.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:09 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:09.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:09 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:44:09.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:09 smithi033 ceph-mon[31973]: Updating smithi096:/etc/ceph/ceph.conf 2024-03-19T02:44:09.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:09 smithi033 ceph-mon[31973]: Updating smithi096:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:44:09.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:09 smithi033 ceph-mon[31973]: Deploying cephadm binary to smithi155 2024-03-19T02:44:10.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:10 smithi033 ceph-mon[31973]: Updating smithi096:/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:44:10.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:10 smithi033 ceph-mon[31973]: Updating smithi096:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T02:44:10.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:10 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:10.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:10 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:10.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:10 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:10.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:10 smithi033 ceph-mon[31973]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:12.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:11 smithi033 ceph-mon[31973]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:12.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:12.329+0000 7f9d67fff700 1 -- 172.21.15.33:0/3028206803 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f9d7006a580 con 0x7f9d5c038fc0 2024-03-19T02:44:12.330 INFO:teuthology.orchestra.run.smithi033.stdout:Added host 'smithi155' with addr '172.21.15.155' 2024-03-19T02:44:12.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:12.332+0000 7f9d753df700 1 -- 172.21.15.33:0/3028206803 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9d5c038fc0 msgr2=0x7f9d5c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:12.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:12.332+0000 7f9d753df700 1 --2- 172.21.15.33:0/3028206803 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9d5c038fc0 0x7f9d5c03b480 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9d58006fd0 tx=0x7f9d58006e40 comp rx=0 tx=0).stop 2024-03-19T02:44:12.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:12.332+0000 7f9d753df700 1 -- 172.21.15.33:0/3028206803 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d70100e30 msgr2=0x7f9d70196380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:12.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:12.332+0000 7f9d753df700 1 --2- 172.21.15.33:0/3028206803 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d70100e30 0x7f9d70196380 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9d68006010 tx=0x7f9d68005780 comp rx=0 tx=0).stop 2024-03-19T02:44:12.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:12.332+0000 7f9d753df700 1 -- 172.21.15.33:0/3028206803 shutdown_connections 2024-03-19T02:44:12.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:12.332+0000 7f9d753df700 1 --2- 172.21.15.33:0/3028206803 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9d5c038fc0 0x7f9d5c03b480 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:12.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:12.332+0000 7f9d753df700 1 --2- 172.21.15.33:0/3028206803 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9d70100e30 0x7f9d70196380 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:12.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:12.332+0000 7f9d753df700 1 -- 172.21.15.33:0/3028206803 >> 172.21.15.33:0/3028206803 conn(0x7f9d700fc6c0 msgr2=0x7f9d70102800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:12.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:12.333+0000 7f9d753df700 1 -- 172.21.15.33:0/3028206803 shutdown_connections 2024-03-19T02:44:12.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:12.333+0000 7f9d753df700 1 -- 172.21.15.33:0/3028206803 wait complete. 2024-03-19T02:44:12.963 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-19T02:44:13.207 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:44:13.658 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:13 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:13.658 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:13 smithi033 ceph-mon[31973]: Added host smithi155 2024-03-19T02:44:13.658 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:13 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:44:13.658 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:13 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:14.634 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:14 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:14.634 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:14 smithi033 ceph-mon[31973]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:14.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:14.859+0000 7f74d29a0700 1 -- 172.21.15.33:0/1173787587 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74cc106a40 msgr2=0x7f74cc106e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:14.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:14.859+0000 7f74d29a0700 1 --2- 172.21.15.33:0/1173787587 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74cc106a40 0x7f74cc106e20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f74bc009910 tx=0x7f74bc009c20 comp rx=0 tx=0).stop 2024-03-19T02:44:14.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:14.859+0000 7f74d29a0700 1 -- 172.21.15.33:0/1173787587 shutdown_connections 2024-03-19T02:44:14.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:14.859+0000 7f74d29a0700 1 --2- 172.21.15.33:0/1173787587 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74cc106a40 0x7f74cc106e20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:14.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:14.859+0000 7f74d29a0700 1 -- 172.21.15.33:0/1173787587 >> 172.21.15.33:0/1173787587 conn(0x7f74cc0fc590 msgr2=0x7f74cc0fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:14.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:14.859+0000 7f74d29a0700 1 -- 172.21.15.33:0/1173787587 shutdown_connections 2024-03-19T02:44:14.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:14.859+0000 7f74d29a0700 1 -- 172.21.15.33:0/1173787587 wait complete. 2024-03-19T02:44:14.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:14.860+0000 7f74d29a0700 1 Processor -- start 2024-03-19T02:44:14.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:14.860+0000 7f74d29a0700 1 -- start start 2024-03-19T02:44:14.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:14.860+0000 7f74d29a0700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74cc106a40 0x7f74cc0fca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:14.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:14.860+0000 7f74d29a0700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f74cc10af30 con 0x7f74cc106a40 2024-03-19T02:44:14.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:14.860+0000 7f74d199e700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74cc106a40 0x7f74cc0fca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:14.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:14.860+0000 7f74d199e700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74cc106a40 0x7f74cc0fca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50916/0 (socket says 172.21.15.33:50916) 2024-03-19T02:44:14.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:14.860+0000 7f74d199e700 1 -- 172.21.15.33:0/2759926849 learned_addr learned my addr 172.21.15.33:0/2759926849 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:44:14.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:14.860+0000 7f74d199e700 1 -- 172.21.15.33:0/2759926849 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74cc0fe8e0 con 0x7f74cc106a40 2024-03-19T02:44:14.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:14.860+0000 7f74d199e700 1 --2- 172.21.15.33:0/2759926849 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74cc106a40 0x7f74cc0fca40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f74bc006010 tx=0x7f74bc005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:14.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:14.861+0000 7f74c2ffd700 1 -- 172.21.15.33:0/2759926849 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74bc026070 con 0x7f74cc106a40 2024-03-19T02:44:14.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:14.861+0000 7f74c2ffd700 1 -- 172.21.15.33:0/2759926849 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f74bc00bc00 con 0x7f74cc106a40 2024-03-19T02:44:14.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:14.861+0000 7f74d29a0700 1 -- 172.21.15.33:0/2759926849 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74cc0fcf80 con 0x7f74cc106a40 2024-03-19T02:44:14.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:14.861+0000 7f74c2ffd700 1 -- 172.21.15.33:0/2759926849 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74bc003930 con 0x7f74cc106a40 2024-03-19T02:44:14.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:14.861+0000 7f74d29a0700 1 -- 172.21.15.33:0/2759926849 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74cc0fd440 con 0x7f74cc106a40 2024-03-19T02:44:14.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:14.862+0000 7f74c2ffd700 1 -- 172.21.15.33:0/2759926849 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f74bc029070 con 0x7f74cc106a40 2024-03-19T02:44:14.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:14.862+0000 7f74d29a0700 1 -- 172.21.15.33:0/2759926849 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74cc102180 con 0x7f74cc106a40 2024-03-19T02:44:14.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:14.862+0000 7f74c2ffd700 1 --2- 172.21.15.33:0/2759926849 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f74b8038fc0 0x7f74b803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:14.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:14.862+0000 7f74c2ffd700 1 -- 172.21.15.33:0/2759926849 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f74bc01e070 con 0x7f74cc106a40 2024-03-19T02:44:14.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:14.862+0000 7f74d119d700 1 --2- 172.21.15.33:0/2759926849 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f74b8038fc0 0x7f74b803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:14.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:14.862+0000 7f74d119d700 1 --2- 172.21.15.33:0/2759926849 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f74b8038fc0 0x7f74b803b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f74c8006fd0 tx=0x7f74c8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:14.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:14.867+0000 7f74c2ffd700 1 -- 172.21.15.33:0/2759926849 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f74bc013070 con 0x7f74cc106a40 2024-03-19T02:44:15.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:15.090+0000 7f74d29a0700 1 -- 172.21.15.33:0/2759926849 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f74cc0fdf20 con 0x7f74b8038fc0 2024-03-19T02:44:15.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:15.091+0000 7f74c2ffd700 1 -- 172.21.15.33:0/2759926849 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7f74cc0fdf20 con 0x7f74b8038fc0 2024-03-19T02:44:15.092 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:44:15.092 INFO:teuthology.orchestra.run.smithi033.stdout:[{"addr": "172.21.15.33", "hostname": "smithi033", "labels": [], "status": ""}, {"addr": "172.21.15.96", "hostname": "smithi096", "labels": [], "status": ""}, {"addr": "172.21.15.155", "hostname": "smithi155", "labels": [], "status": ""}] 2024-03-19T02:44:15.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:15.093+0000 7f74d29a0700 1 -- 172.21.15.33:0/2759926849 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f74b8038fc0 msgr2=0x7f74b803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:15.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:15.093+0000 7f74d29a0700 1 --2- 172.21.15.33:0/2759926849 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f74b8038fc0 0x7f74b803b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f74c8006fd0 tx=0x7f74c8006e40 comp rx=0 tx=0).stop 2024-03-19T02:44:15.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:15.093+0000 7f74d29a0700 1 -- 172.21.15.33:0/2759926849 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74cc106a40 msgr2=0x7f74cc0fca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:15.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:15.093+0000 7f74d29a0700 1 --2- 172.21.15.33:0/2759926849 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74cc106a40 0x7f74cc0fca40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f74bc006010 tx=0x7f74bc005780 comp rx=0 tx=0).stop 2024-03-19T02:44:15.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:15.093+0000 7f74d29a0700 1 -- 172.21.15.33:0/2759926849 shutdown_connections 2024-03-19T02:44:15.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:15.093+0000 7f74d29a0700 1 --2- 172.21.15.33:0/2759926849 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f74b8038fc0 0x7f74b803b480 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:15.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:15.093+0000 7f74d29a0700 1 --2- 172.21.15.33:0/2759926849 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f74cc106a40 0x7f74cc0fca40 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:15.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:15.093+0000 7f74d29a0700 1 -- 172.21.15.33:0/2759926849 >> 172.21.15.33:0/2759926849 conn(0x7f74cc0fc590 msgr2=0x7f74cc069f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:15.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:15.093+0000 7f74d29a0700 1 -- 172.21.15.33:0/2759926849 shutdown_connections 2024-03-19T02:44:15.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:15.094+0000 7f74d29a0700 1 -- 172.21.15.33:0/2759926849 wait complete. 2024-03-19T02:44:15.678 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-19T02:44:15.679 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph osd crush tunables default 2024-03-19T02:44:15.916 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:44:16.010 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:15 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:16.010 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:15 smithi033 ceph-mon[31973]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:44:16.011 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:15 smithi033 ceph-mon[31973]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:17.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.368+0000 7f5ee3561700 1 -- 172.21.15.33:0/2097310894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5edc101f50 msgr2=0x7f5edc102330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:17.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.368+0000 7f5ee3561700 1 --2- 172.21.15.33:0/2097310894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5edc101f50 0x7f5edc102330 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8009910 tx=0x7f5ed8009c20 comp rx=0 tx=0).stop 2024-03-19T02:44:17.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.368+0000 7f5ee3561700 1 -- 172.21.15.33:0/2097310894 shutdown_connections 2024-03-19T02:44:17.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.368+0000 7f5ee3561700 1 --2- 172.21.15.33:0/2097310894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5edc101f50 0x7f5edc102330 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:17.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.368+0000 7f5ee3561700 1 -- 172.21.15.33:0/2097310894 >> 172.21.15.33:0/2097310894 conn(0x7f5edc0fd820 msgr2=0x7f5edc0ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:17.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.368+0000 7f5ee3561700 1 -- 172.21.15.33:0/2097310894 shutdown_connections 2024-03-19T02:44:17.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.368+0000 7f5ee3561700 1 -- 172.21.15.33:0/2097310894 wait complete. 2024-03-19T02:44:17.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.369+0000 7f5ee3561700 1 Processor -- start 2024-03-19T02:44:17.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.369+0000 7f5ee3561700 1 -- start start 2024-03-19T02:44:17.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.369+0000 7f5ee3561700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5edc101f50 0x7f5edc19d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:17.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.369+0000 7f5ee3561700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5edc10c1d0 con 0x7f5edc101f50 2024-03-19T02:44:17.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.369+0000 7f5ee12fd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5edc101f50 0x7f5edc19d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:17.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.369+0000 7f5ee12fd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5edc101f50 0x7f5edc19d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50928/0 (socket says 172.21.15.33:50928) 2024-03-19T02:44:17.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.369+0000 7f5ee12fd700 1 -- 172.21.15.33:0/2708956598 learned_addr learned my addr 172.21.15.33:0/2708956598 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:44:17.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.369+0000 7f5ee12fd700 1 -- 172.21.15.33:0/2708956598 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5edc19d910 con 0x7f5edc101f50 2024-03-19T02:44:17.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.370+0000 7f5ee12fd700 1 --2- 172.21.15.33:0/2708956598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5edc101f50 0x7f5edc19d3d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5ed80056d0 tx=0x7f5ed80057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:17.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.370+0000 7f5ed27fc700 1 -- 172.21.15.33:0/2708956598 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ed8026070 con 0x7f5edc101f50 2024-03-19T02:44:17.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.370+0000 7f5ed27fc700 1 -- 172.21.15.33:0/2708956598 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ed800bba0 con 0x7f5edc101f50 2024-03-19T02:44:17.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.370+0000 7f5ed27fc700 1 -- 172.21.15.33:0/2708956598 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ed80037f0 con 0x7f5edc101f50 2024-03-19T02:44:17.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.370+0000 7f5ee3561700 1 -- 172.21.15.33:0/2708956598 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5edc19dc00 con 0x7f5edc101f50 2024-03-19T02:44:17.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.370+0000 7f5ee3561700 1 -- 172.21.15.33:0/2708956598 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5edc197590 con 0x7f5edc101f50 2024-03-19T02:44:17.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.371+0000 7f5ed27fc700 1 -- 172.21.15.33:0/2708956598 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f5ed8029070 con 0x7f5edc101f50 2024-03-19T02:44:17.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.371+0000 7f5ee3561700 1 -- 172.21.15.33:0/2708956598 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5edc103430 con 0x7f5edc101f50 2024-03-19T02:44:17.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.371+0000 7f5ed27fc700 1 --2- 172.21.15.33:0/2708956598 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5ec8039020 0x7f5ec803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:17.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.371+0000 7f5ed27fc700 1 -- 172.21.15.33:0/2708956598 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5ed801e070 con 0x7f5edc101f50 2024-03-19T02:44:17.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.371+0000 7f5ee0afc700 1 --2- 172.21.15.33:0/2708956598 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5ec8039020 0x7f5ec803b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:17.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.371+0000 7f5ee0afc700 1 --2- 172.21.15.33:0/2708956598 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5ec8039020 0x7f5ec803b4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc006fd0 tx=0x7f5ecc006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:17.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.376+0000 7f5ed27fc700 1 -- 172.21.15.33:0/2708956598 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5ed8013070 con 0x7f5edc101f50 2024-03-19T02:44:17.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.584+0000 7f5ee3561700 1 -- 172.21.15.33:0/2708956598 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f5edc063940 con 0x7f5edc101f50 2024-03-19T02:44:17.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.635+0000 7f5ed27fc700 1 -- 172.21.15.33:0/2708956598 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f5ed8030090 con 0x7f5edc101f50 2024-03-19T02:44:17.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.638+0000 7f5ee3561700 1 -- 172.21.15.33:0/2708956598 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5ec8039020 msgr2=0x7f5ec803b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:17.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.638+0000 7f5ee3561700 1 --2- 172.21.15.33:0/2708956598 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5ec8039020 0x7f5ec803b4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc006fd0 tx=0x7f5ecc006e40 comp rx=0 tx=0).stop 2024-03-19T02:44:17.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.638+0000 7f5ee3561700 1 -- 172.21.15.33:0/2708956598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5edc101f50 msgr2=0x7f5edc19d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:17.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.638+0000 7f5ee3561700 1 --2- 172.21.15.33:0/2708956598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5edc101f50 0x7f5edc19d3d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5ed80056d0 tx=0x7f5ed80057b0 comp rx=0 tx=0).stop 2024-03-19T02:44:17.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.638+0000 7f5ee3561700 1 -- 172.21.15.33:0/2708956598 shutdown_connections 2024-03-19T02:44:17.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.638+0000 7f5ee3561700 1 --2- 172.21.15.33:0/2708956598 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5ec8039020 0x7f5ec803b4e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:17.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.638+0000 7f5ee3561700 1 --2- 172.21.15.33:0/2708956598 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5edc101f50 0x7f5edc19d3d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:17.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.638+0000 7f5ee3561700 1 -- 172.21.15.33:0/2708956598 >> 172.21.15.33:0/2708956598 conn(0x7f5edc0fd820 msgr2=0x7f5edc103920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:17.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.638+0000 7f5ee3561700 1 -- 172.21.15.33:0/2708956598 shutdown_connections 2024-03-19T02:44:17.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:17.638+0000 7f5ee3561700 1 -- 172.21.15.33:0/2708956598 wait complete. 2024-03-19T02:44:17.640 INFO:teuthology.orchestra.run.smithi033.stderr:adjusted tunables profile to default 2024-03-19T02:44:18.013 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:17 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2708956598' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-19T02:44:18.306 INFO:tasks.cephadm:Adding mon.a on smithi033 2024-03-19T02:44:18.307 INFO:tasks.cephadm:Adding mon.b on smithi096 2024-03-19T02:44:18.307 INFO:tasks.cephadm:Adding mon.c on smithi155 2024-03-19T02:44:18.307 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph orch apply mon '3;smithi033:172.21.15.33=a;smithi096:172.21.15.96=b;smithi155:172.21.15.155=c' 2024-03-19T02:44:18.543 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:44:19.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:18 smithi033 ceph-mon[31973]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:19.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:18 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2708956598' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-19T02:44:19.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:18 smithi033 ceph-mon[31973]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T02:44:19.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:18 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:19.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:18 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:19.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:18 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:19.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:18 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:19.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:18 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi155", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:44:19.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:18 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:19.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:18 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:44:20.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:19 smithi033 ceph-mon[31973]: Updating smithi155:/etc/ceph/ceph.conf 2024-03-19T02:44:20.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:19 smithi033 ceph-mon[31973]: Updating smithi155:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:44:20.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:19 smithi033 ceph-mon[31973]: Updating smithi155:/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:44:20.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:19 smithi033 ceph-mon[31973]: Updating smithi155:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T02:44:20.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:19 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:20.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:19 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:20.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:19 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:20.043 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.042+0000 7f854bc8b700 1 -- 172.21.15.155:0/1993063488 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85440746b0 msgr2=0x7f8544074a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:20.044 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.042+0000 7f854bc8b700 1 --2- 172.21.15.155:0/1993063488 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85440746b0 0x7f8544074a90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8540009910 tx=0x7f8540009c20 comp rx=0 tx=0).stop 2024-03-19T02:44:20.044 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.042+0000 7f854bc8b700 1 -- 172.21.15.155:0/1993063488 shutdown_connections 2024-03-19T02:44:20.044 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.042+0000 7f854bc8b700 1 --2- 172.21.15.155:0/1993063488 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85440746b0 0x7f8544074a90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:20.044 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.042+0000 7f854bc8b700 1 -- 172.21.15.155:0/1993063488 >> 172.21.15.155:0/1993063488 conn(0x7f85440fd820 msgr2=0x7f85440ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:20.044 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.042+0000 7f854bc8b700 1 -- 172.21.15.155:0/1993063488 shutdown_connections 2024-03-19T02:44:20.044 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.042+0000 7f854bc8b700 1 -- 172.21.15.155:0/1993063488 wait complete. 2024-03-19T02:44:20.044 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.043+0000 7f854bc8b700 1 Processor -- start 2024-03-19T02:44:20.044 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.043+0000 7f854bc8b700 1 -- start start 2024-03-19T02:44:20.044 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.043+0000 7f854bc8b700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85440746b0 0x7f854419dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:20.044 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.043+0000 7f854bc8b700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8544112840 con 0x7f85440746b0 2024-03-19T02:44:20.045 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.043+0000 7f8549a27700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85440746b0 0x7f854419dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:20.045 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.043+0000 7f8549a27700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85440746b0 0x7f854419dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.155:54164/0 (socket says 172.21.15.155:54164) 2024-03-19T02:44:20.045 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.043+0000 7f8549a27700 1 -- 172.21.15.155:0/3064292160 learned_addr learned my addr 172.21.15.155:0/3064292160 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-19T02:44:20.045 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.044+0000 7f8549a27700 1 -- 172.21.15.155:0/3064292160 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f854419e2c0 con 0x7f85440746b0 2024-03-19T02:44:20.046 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.044+0000 7f8549a27700 1 --2- 172.21.15.155:0/3064292160 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85440746b0 0x7f854419dd80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f85400056d0 tx=0x7f85400057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:20.046 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.044+0000 7f853affd700 1 -- 172.21.15.155:0/3064292160 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8540026070 con 0x7f85440746b0 2024-03-19T02:44:20.046 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.044+0000 7f854bc8b700 1 -- 172.21.15.155:0/3064292160 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f854419e550 con 0x7f85440746b0 2024-03-19T02:44:20.046 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.044+0000 7f854bc8b700 1 -- 172.21.15.155:0/3064292160 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f854419e9b0 con 0x7f85440746b0 2024-03-19T02:44:20.046 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.044+0000 7f853affd700 1 -- 172.21.15.155:0/3064292160 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8540004370 con 0x7f85440746b0 2024-03-19T02:44:20.046 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.044+0000 7f853affd700 1 -- 172.21.15.155:0/3064292160 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8540003990 con 0x7f85440746b0 2024-03-19T02:44:20.047 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.045+0000 7f853affd700 1 -- 172.21.15.155:0/3064292160 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f8540029070 con 0x7f85440746b0 2024-03-19T02:44:20.047 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.045+0000 7f853affd700 1 --2- 172.21.15.155:0/3064292160 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8530038fc0 0x7f853003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:20.047 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.045+0000 7f853affd700 1 -- 172.21.15.155:0/3064292160 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f854001e070 con 0x7f85440746b0 2024-03-19T02:44:20.047 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.045+0000 7f854bc8b700 1 -- 172.21.15.155:0/3064292160 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8528005330 con 0x7f85440746b0 2024-03-19T02:44:20.047 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.046+0000 7f8549226700 1 --2- 172.21.15.155:0/3064292160 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8530038fc0 0x7f853003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:20.047 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.046+0000 7f8549226700 1 --2- 172.21.15.155:0/3064292160 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8530038fc0 0x7f853003b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8534006fd0 tx=0x7f8534006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:20.051 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.051+0000 7f853affd700 1 -- 172.21.15.155:0/3064292160 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8540013070 con 0x7f85440746b0 2024-03-19T02:44:20.259 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.258+0000 7f854bc8b700 1 -- 172.21.15.155:0/3064292160 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi033:172.21.15.33=a;smithi096:172.21.15.96=b;smithi155:172.21.15.155=c", "target": ["mon-mgr", ""]}) -- 0x7f8528000c80 con 0x7f8530038fc0 2024-03-19T02:44:20.266 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.265+0000 7f853affd700 1 -- 172.21.15.155:0/3064292160 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f8528000c80 con 0x7f8530038fc0 2024-03-19T02:44:20.266 INFO:teuthology.orchestra.run.smithi155.stdout:Scheduled mon update... 2024-03-19T02:44:20.268 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.268+0000 7f854bc8b700 1 -- 172.21.15.155:0/3064292160 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8530038fc0 msgr2=0x7f853003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:20.268 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.268+0000 7f854bc8b700 1 --2- 172.21.15.155:0/3064292160 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8530038fc0 0x7f853003b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8534006fd0 tx=0x7f8534006e40 comp rx=0 tx=0).stop 2024-03-19T02:44:20.268 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.268+0000 7f854bc8b700 1 -- 172.21.15.155:0/3064292160 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85440746b0 msgr2=0x7f854419dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:20.269 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.268+0000 7f854bc8b700 1 --2- 172.21.15.155:0/3064292160 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85440746b0 0x7f854419dd80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f85400056d0 tx=0x7f85400057b0 comp rx=0 tx=0).stop 2024-03-19T02:44:20.269 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.268+0000 7f854bc8b700 1 -- 172.21.15.155:0/3064292160 shutdown_connections 2024-03-19T02:44:20.269 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.268+0000 7f854bc8b700 1 --2- 172.21.15.155:0/3064292160 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8530038fc0 0x7f853003b480 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:20.269 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.268+0000 7f854bc8b700 1 --2- 172.21.15.155:0/3064292160 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85440746b0 0x7f854419dd80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:20.269 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.268+0000 7f854bc8b700 1 -- 172.21.15.155:0/3064292160 >> 172.21.15.155:0/3064292160 conn(0x7f85440fd820 msgr2=0x7f8544103a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:20.269 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.268+0000 7f854bc8b700 1 -- 172.21.15.155:0/3064292160 shutdown_connections 2024-03-19T02:44:20.269 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:20.268+0000 7f854bc8b700 1 -- 172.21.15.155:0/3064292160 wait complete. 2024-03-19T02:44:20.866 DEBUG:teuthology.orchestra.run.smithi096:mon.b> sudo journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mon.b.service 2024-03-19T02:44:20.869 DEBUG:teuthology.orchestra.run.smithi155:mon.c> sudo journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mon.c.service 2024-03-19T02:44:20.872 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-19T02:44:20.872 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-19T02:44:21.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:20 smithi033 ceph-mon[31973]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:21.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:20 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:21.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:20 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:44:21.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:20 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:21.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:20 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:44:21.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:20 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:21.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:20 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T02:44:21.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:20 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:22.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:21 smithi033 ceph-mon[31973]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi033:172.21.15.33=a;smithi096:172.21.15.96=b;smithi155:172.21.15.155=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:44:22.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:21 smithi033 ceph-mon[31973]: Saving service mon spec with placement smithi033:172.21.15.33=a;smithi096:172.21.15.96=b;smithi155:172.21.15.155=c;count:3 2024-03-19T02:44:22.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:21 smithi033 ceph-mon[31973]: Deploying daemon mon.c on smithi155 2024-03-19T02:44:22.038 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:44:23.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:22 smithi033 ceph-mon[31973]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:23.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:23.990+0000 7ff8f9416700 1 -- 172.21.15.155:0/2002635560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8f4102f40 msgr2=0x7ff8f4103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:23.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:23.990+0000 7ff8f9416700 1 --2- 172.21.15.155:0/2002635560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8f4102f40 0x7ff8f4103320 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff8e4009910 tx=0x7ff8e4009c20 comp rx=0 tx=0).stop 2024-03-19T02:44:23.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:23.990+0000 7ff8f9416700 1 -- 172.21.15.155:0/2002635560 shutdown_connections 2024-03-19T02:44:23.993 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:23.990+0000 7ff8f9416700 1 --2- 172.21.15.155:0/2002635560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8f4102f40 0x7ff8f4103320 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:23.993 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:23.990+0000 7ff8f9416700 1 -- 172.21.15.155:0/2002635560 >> 172.21.15.155:0/2002635560 conn(0x7ff8f40fc6d0 msgr2=0x7ff8f40feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:23.993 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:23.990+0000 7ff8f9416700 1 -- 172.21.15.155:0/2002635560 shutdown_connections 2024-03-19T02:44:23.993 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:23.990+0000 7ff8f9416700 1 -- 172.21.15.155:0/2002635560 wait complete. 2024-03-19T02:44:23.993 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:23.990+0000 7ff8f9416700 1 Processor -- start 2024-03-19T02:44:23.993 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:23.991+0000 7ff8f9416700 1 -- start start 2024-03-19T02:44:23.993 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:23.991+0000 7ff8f9416700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8f4102f40 0x7ff8f419d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:23.993 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:23.991+0000 7ff8f9416700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f410c240 con 0x7ff8f4102f40 2024-03-19T02:44:23.993 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:23.991+0000 7ff8f2ffd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8f4102f40 0x7ff8f419d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:23.993 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:23.991+0000 7ff8f2ffd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8f4102f40 0x7ff8f419d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.155:54182/0 (socket says 172.21.15.155:54182) 2024-03-19T02:44:23.993 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:23.991+0000 7ff8f2ffd700 1 -- 172.21.15.155:0/2189142395 learned_addr learned my addr 172.21.15.155:0/2189142395 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-19T02:44:23.993 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:23.991+0000 7ff8f2ffd700 1 -- 172.21.15.155:0/2189142395 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8f419d940 con 0x7ff8f4102f40 2024-03-19T02:44:23.994 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:23.992+0000 7ff8f2ffd700 1 --2- 172.21.15.155:0/2189142395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8f4102f40 0x7ff8f419d400 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff8e40056d0 tx=0x7ff8e40057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:23.994 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:23.992+0000 7ff8dbfff700 1 -- 172.21.15.155:0/2189142395 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff8e4026070 con 0x7ff8f4102f40 2024-03-19T02:44:23.994 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:23.992+0000 7ff8f9416700 1 -- 172.21.15.155:0/2189142395 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8f419dbd0 con 0x7ff8f4102f40 2024-03-19T02:44:23.994 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:23.992+0000 7ff8dbfff700 1 -- 172.21.15.155:0/2189142395 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff8e4004370 con 0x7ff8f4102f40 2024-03-19T02:44:23.994 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:23.992+0000 7ff8dbfff700 1 -- 172.21.15.155:0/2189142395 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff8e40039b0 con 0x7ff8f4102f40 2024-03-19T02:44:23.994 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:23.992+0000 7ff8f9416700 1 -- 172.21.15.155:0/2189142395 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8f41975c0 con 0x7ff8f4102f40 2024-03-19T02:44:23.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:23.993+0000 7ff8f9416700 1 -- 172.21.15.155:0/2189142395 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8f406a930 con 0x7ff8f4102f40 2024-03-19T02:44:23.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:23.993+0000 7ff8dbfff700 1 -- 172.21.15.155:0/2189142395 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7ff8e4029070 con 0x7ff8f4102f40 2024-03-19T02:44:23.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:23.993+0000 7ff8dbfff700 1 --2- 172.21.15.155:0/2189142395 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff8dc038fc0 0x7ff8dc03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:23.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:23.993+0000 7ff8dbfff700 1 -- 172.21.15.155:0/2189142395 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff8e401e070 con 0x7ff8f4102f40 2024-03-19T02:44:23.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:23.993+0000 7ff8f27fc700 1 --2- 172.21.15.155:0/2189142395 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff8dc038fc0 0x7ff8dc03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:23.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:23.994+0000 7ff8f27fc700 1 --2- 172.21.15.155:0/2189142395 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff8dc038fc0 0x7ff8dc03b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8006fd0 tx=0x7ff8e8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:23.998 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:23.998+0000 7ff8dbfff700 1 -- 172.21.15.155:0/2189142395 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff8e4051070 con 0x7ff8f4102f40 2024-03-19T02:44:24.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:24.277+0000 7ff8f9416700 1 -- 172.21.15.155:0/2189142395 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7ff8f4063940 con 0x7ff8f4102f40 2024-03-19T02:44:24.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:24.278+0000 7ff8dbfff700 1 -- 172.21.15.155:0/2189142395 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7ff8e4030020 con 0x7ff8f4102f40 2024-03-19T02:44:24.279 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-19T02:44:24.279 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"2b7a8938-e59a-11ee-95c9-87774f69a715","modified":"2024-03-19T02:42:35.143659Z","created":"2024-03-19T02:42:35.143659Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:3300","nonce":0},{"type":"v1","addr":"172.21.15.33:6789","nonce":0}]},"addr":"172.21.15.33:6789/0","public_addr":"172.21.15.33:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-19T02:44:24.280 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:24.280+0000 7ff8f9416700 1 -- 172.21.15.155:0/2189142395 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff8dc038fc0 msgr2=0x7ff8dc03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:24.280 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:24.280+0000 7ff8f9416700 1 --2- 172.21.15.155:0/2189142395 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff8dc038fc0 0x7ff8dc03b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8006fd0 tx=0x7ff8e8006e40 comp rx=0 tx=0).stop 2024-03-19T02:44:24.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:24.280+0000 7ff8f9416700 1 -- 172.21.15.155:0/2189142395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8f4102f40 msgr2=0x7ff8f419d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:24.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:24.280+0000 7ff8f9416700 1 --2- 172.21.15.155:0/2189142395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8f4102f40 0x7ff8f419d400 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff8e40056d0 tx=0x7ff8e40057b0 comp rx=0 tx=0).stop 2024-03-19T02:44:24.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:24.280+0000 7ff8f9416700 1 -- 172.21.15.155:0/2189142395 shutdown_connections 2024-03-19T02:44:24.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:24.280+0000 7ff8f9416700 1 --2- 172.21.15.155:0/2189142395 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff8dc038fc0 0x7ff8dc03b480 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:24.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:24.280+0000 7ff8f9416700 1 --2- 172.21.15.155:0/2189142395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8f4102f40 0x7ff8f419d400 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:24.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:24.280+0000 7ff8f9416700 1 -- 172.21.15.155:0/2189142395 >> 172.21.15.155:0/2189142395 conn(0x7ff8f40fc6d0 msgr2=0x7ff8f4103cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:24.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:24.280+0000 7ff8f9416700 1 -- 172.21.15.155:0/2189142395 shutdown_connections 2024-03-19T02:44:24.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:24.280+0000 7ff8f9416700 1 -- 172.21.15.155:0/2189142395 wait complete. 2024-03-19T02:44:24.282 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-03-19T02:44:25.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:24 smithi033 ceph-mon[31973]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:25.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:24 smithi033 ceph-mon[31973]: from='client.? 172.21.15.155:0/2189142395' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T02:44:25.924 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-19T02:44:25.924 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-19T02:44:26.612 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T02:44:31.782 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.779+0000 7f4ce002b700 1 -- 172.21.15.155:0/1423518176 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4cc40035c0 msgr2=0x7f4cc4005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:31.782 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.779+0000 7f4ce002b700 1 --2- 172.21.15.155:0/1423518176 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4cc40035c0 0x7f4cc4005a50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8078510 tx=0x7f4cd0009e50 comp rx=0 tx=0).stop 2024-03-19T02:44:31.782 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.779+0000 7f4ce002b700 1 -- 172.21.15.155:0/1423518176 shutdown_connections 2024-03-19T02:44:31.782 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.779+0000 7f4ce002b700 1 --2- 172.21.15.155:0/1423518176 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4cc40035c0 0x7f4cc4005a50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:31.782 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.779+0000 7f4ce002b700 1 -- 172.21.15.155:0/1423518176 >> 172.21.15.155:0/1423518176 conn(0x7f4cd8077300 msgr2=0x7f4cd8077710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:31.782 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.779+0000 7f4ce002b700 1 -- 172.21.15.155:0/1423518176 shutdown_connections 2024-03-19T02:44:31.782 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.779+0000 7f4ce002b700 1 -- 172.21.15.155:0/1423518176 wait complete. 2024-03-19T02:44:31.782 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.780+0000 7f4ce002b700 1 Processor -- start 2024-03-19T02:44:31.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.780+0000 7f4ce002b700 1 -- start start 2024-03-19T02:44:31.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.780+0000 7f4ce002b700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4cc40035c0 0x7f4cd819f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:31.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.780+0000 7f4ce002b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4cd819fbf0 0x7f4cd8199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:31.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.780+0000 7f4ce002b700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd81026b0 con 0x7f4cc40035c0 2024-03-19T02:44:31.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.780+0000 7f4ce002b700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd8102530 con 0x7f4cd819fbf0 2024-03-19T02:44:31.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.780+0000 7f4cdd5c6700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4cd819fbf0 0x7f4cd8199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:31.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.780+0000 7f4cdd5c6700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4cd819fbf0 0x7f4cd8199880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:33394/0 (socket says 172.21.15.155:33394) 2024-03-19T02:44:31.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.780+0000 7f4cdd5c6700 1 -- 172.21.15.155:0/1487961750 learned_addr learned my addr 172.21.15.155:0/1487961750 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-19T02:44:31.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.781+0000 7f4cdd5c6700 1 -- 172.21.15.155:0/1487961750 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4cd819fbf0 msgr2=0x7f4cd8199880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-19T02:44:31.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.781+0000 7f4cdd5c6700 1 -- 172.21.15.155:0/1487961750 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4cd819fbf0 msgr2=0x7f4cd8199880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-19T02:44:31.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.781+0000 7f4cdd5c6700 1 --2- 172.21.15.155:0/1487961750 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4cd819fbf0 0x7f4cd8199880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T02:44:31.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.781+0000 7f4cdd5c6700 1 --2- 172.21.15.155:0/1487961750 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4cd819fbf0 0x7f4cd8199880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T02:44:31.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.781+0000 7f4cdddc7700 1 --2- 172.21.15.155:0/1487961750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4cc40035c0 0x7f4cd819f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:31.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.781+0000 7f4cdddc7700 1 -- 172.21.15.155:0/1487961750 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4cd819fbf0 msgr2=0x7f4cd8199880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T02:44:31.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.781+0000 7f4cdddc7700 1 --2- 172.21.15.155:0/1487961750 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4cd819fbf0 0x7f4cd8199880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:31.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.781+0000 7f4cdddc7700 1 -- 172.21.15.155:0/1487961750 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cd81a0230 con 0x7f4cc40035c0 2024-03-19T02:44:31.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.781+0000 7f4cdddc7700 1 --2- 172.21.15.155:0/1487961750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4cc40035c0 0x7f4cd819f6b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8078510 tx=0x7f4cd0003e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:31.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.781+0000 7f4cceffd700 1 -- 172.21.15.155:0/1487961750 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f4cd001ab80 con 0x7f4cc40035c0 2024-03-19T02:44:31.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.781+0000 7f4ce002b700 1 -- 172.21.15.155:0/1487961750 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cd8199e20 con 0x7f4cc40035c0 2024-03-19T02:44:31.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.781+0000 7f4cceffd700 1 -- 172.21.15.155:0/1487961750 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4cd001ad20 con 0x7f4cc40035c0 2024-03-19T02:44:31.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.781+0000 7f4cceffd700 1 -- 172.21.15.155:0/1487961750 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f4cd002a9f0 con 0x7f4cc40035c0 2024-03-19T02:44:31.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.781+0000 7f4ce002b700 1 -- 172.21.15.155:0/1487961750 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cd819a380 con 0x7f4cc40035c0 2024-03-19T02:44:31.785 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.782+0000 7f4cceffd700 1 -- 172.21.15.155:0/1487961750 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f4cd0032070 con 0x7f4cc40035c0 2024-03-19T02:44:31.785 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.783+0000 7f4cceffd700 1 --2- 172.21.15.155:0/1487961750 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4cbc038d60 0x7f4cbc03b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:31.785 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.783+0000 7f4cceffd700 1 -- 172.21.15.155:0/1487961750 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f4cd0027070 con 0x7f4cc40035c0 2024-03-19T02:44:31.786 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.783+0000 7f4ce002b700 1 -- 172.21.15.155:0/1487961750 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cd806a200 con 0x7f4cc40035c0 2024-03-19T02:44:31.786 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.783+0000 7f4cdd5c6700 1 --2- 172.21.15.155:0/1487961750 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4cbc038d60 0x7f4cbc03b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:31.786 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.783+0000 7f4cdd5c6700 1 --2- 172.21.15.155:0/1487961750 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4cbc038d60 0x7f4cbc03b220 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8009180 tx=0x7f4cc8008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:31.788 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:31.788+0000 7f4cceffd700 1 -- 172.21.15.155:0/1487961750 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4cd001d070 con 0x7f4cc40035c0 2024-03-19T02:44:32.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:31 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:44:32.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:31 smithi033 ceph-mon[31973]: mon.a calling monitor election 2024-03-19T02:44:32.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:31 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:44:32.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:31 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:44:32.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:31 smithi033 ceph-mon[31973]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:32.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:31 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:44:32.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:31 smithi033 ceph-mon[31973]: mon.c calling monitor election 2024-03-19T02:44:32.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:31 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:44:32.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:31 smithi033 ceph-mon[31973]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:32.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:31 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:44:32.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:31 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:44:32.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:31 smithi033 ceph-mon[31973]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-19T02:44:32.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:31 smithi033 ceph-mon[31973]: monmap epoch 2 2024-03-19T02:44:32.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:31 smithi033 ceph-mon[31973]: fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T02:44:32.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:31 smithi033 ceph-mon[31973]: last_changed 2024-03-19T02:44:26.566826+0000 2024-03-19T02:44:32.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:31 smithi033 ceph-mon[31973]: created 2024-03-19T02:42:35.143659+0000 2024-03-19T02:44:32.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:31 smithi033 ceph-mon[31973]: min_mon_release 19 (squid) 2024-03-19T02:44:32.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:31 smithi033 ceph-mon[31973]: election_strategy: 1 2024-03-19T02:44:32.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:31 smithi033 ceph-mon[31973]: 0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2024-03-19T02:44:32.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:31 smithi033 ceph-mon[31973]: 1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.c 2024-03-19T02:44:32.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:31 smithi033 ceph-mon[31973]: fsmap 2024-03-19T02:44:32.031 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:31 smithi033 ceph-mon[31973]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T02:44:32.031 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:31 smithi033 ceph-mon[31973]: mgrmap e13: x(active, since 51s) 2024-03-19T02:44:32.031 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:31 smithi033 ceph-mon[31973]: overall HEALTH_OK 2024-03-19T02:44:32.031 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:31 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:32.031 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:31 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:32.031 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:31 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:32.031 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:31 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T02:44:32.031 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:31 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:32.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:32.065+0000 7f4ce002b700 1 -- 172.21.15.155:0/1487961750 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f4cd819ad60 con 0x7f4cc40035c0 2024-03-19T02:44:32.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:32.066+0000 7f4cceffd700 1 -- 172.21.15.155:0/1487961750 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f4cd0039410 con 0x7f4cc40035c0 2024-03-19T02:44:32.066 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-19T02:44:32.066 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":2,"fsid":"2b7a8938-e59a-11ee-95c9-87774f69a715","modified":"2024-03-19T02:44:26.566826Z","created":"2024-03-19T02:42:35.143659Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:3300","nonce":0},{"type":"v1","addr":"172.21.15.33:6789","nonce":0}]},"addr":"172.21.15.33:6789/0","public_addr":"172.21.15.33:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-19T02:44:32.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:32.068+0000 7f4ce002b700 1 -- 172.21.15.155:0/1487961750 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4cbc038d60 msgr2=0x7f4cbc03b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:32.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:32.068+0000 7f4ce002b700 1 --2- 172.21.15.155:0/1487961750 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4cbc038d60 0x7f4cbc03b220 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8009180 tx=0x7f4cc8008040 comp rx=0 tx=0).stop 2024-03-19T02:44:32.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:32.068+0000 7f4ce002b700 1 -- 172.21.15.155:0/1487961750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4cc40035c0 msgr2=0x7f4cd819f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:32.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:32.068+0000 7f4ce002b700 1 --2- 172.21.15.155:0/1487961750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4cc40035c0 0x7f4cd819f6b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8078510 tx=0x7f4cd0003e20 comp rx=0 tx=0).stop 2024-03-19T02:44:32.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:32.068+0000 7f4ce002b700 1 -- 172.21.15.155:0/1487961750 shutdown_connections 2024-03-19T02:44:32.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:32.068+0000 7f4ce002b700 1 --2- 172.21.15.155:0/1487961750 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4cbc038d60 0x7f4cbc03b220 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:32.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:32.068+0000 7f4ce002b700 1 --2- 172.21.15.155:0/1487961750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4cc40035c0 0x7f4cd819f6b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:32.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:32.068+0000 7f4ce002b700 1 --2- 172.21.15.155:0/1487961750 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4cd819fbf0 0x7f4cd8199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:32.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:32.068+0000 7f4ce002b700 1 -- 172.21.15.155:0/1487961750 >> 172.21.15.155:0/1487961750 conn(0x7f4cd8077300 msgr2=0x7f4cd80fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:32.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:32.068+0000 7f4ce002b700 1 -- 172.21.15.155:0/1487961750 shutdown_connections 2024-03-19T02:44:32.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:32.068+0000 7f4ce002b700 1 -- 172.21.15.155:0/1487961750 wait complete. 2024-03-19T02:44:32.069 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 2 2024-03-19T02:44:33.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:32 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:44:33.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:32 smithi033 ceph-mon[31973]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:33.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:32 smithi033 ceph-mon[31973]: Deploying daemon mon.b on smithi096 2024-03-19T02:44:33.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:32 smithi033 ceph-mon[31973]: from='client.? 172.21.15.155:0/1487961750' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T02:44:33.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:32 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:44:33.631 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-19T02:44:33.631 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-19T02:44:33.870 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T02:44:33.992 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:33 smithi155 ceph-mon[35804]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:34.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:33 smithi033 ceph-mon[31973]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:35.191 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 systemd[1]: Starting Ceph mon.b for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T02:44:35.323 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.320+0000 7ff5cbb7b700 1 -- 172.21.15.155:0/2944842173 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5c4102ff0 msgr2=0x7ff5c41033d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:35.323 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.320+0000 7ff5cbb7b700 1 --2- 172.21.15.155:0/2944842173 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5c4102ff0 0x7ff5c41033d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc00b0a0 tx=0x7ff5bc00b3b0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.323 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.320+0000 7ff5cbb7b700 1 -- 172.21.15.155:0/2944842173 shutdown_connections 2024-03-19T02:44:35.323 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.320+0000 7ff5cbb7b700 1 --2- 172.21.15.155:0/2944842173 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5c4102ff0 0x7ff5c41033d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.323 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.320+0000 7ff5cbb7b700 1 -- 172.21.15.155:0/2944842173 >> 172.21.15.155:0/2944842173 conn(0x7ff5c40fc7c0 msgr2=0x7ff5c40febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:35.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.321+0000 7ff5cbb7b700 1 -- 172.21.15.155:0/2944842173 shutdown_connections 2024-03-19T02:44:35.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.321+0000 7ff5cbb7b700 1 -- 172.21.15.155:0/2944842173 wait complete. 2024-03-19T02:44:35.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.321+0000 7ff5cbb7b700 1 Processor -- start 2024-03-19T02:44:35.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.321+0000 7ff5cbb7b700 1 -- start start 2024-03-19T02:44:35.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.321+0000 7ff5cbb7b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff5c4102ff0 0x7ff5c419d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.321+0000 7ff5cbb7b700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5c419dbe0 0x7ff5c4197860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.321+0000 7ff5cbb7b700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff5c410c320 con 0x7ff5c419dbe0 2024-03-19T02:44:35.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.321+0000 7ff5cbb7b700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff5c410c1a0 con 0x7ff5c4102ff0 2024-03-19T02:44:35.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.322+0000 7ff5c9917700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff5c4102ff0 0x7ff5c419d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.322+0000 7ff5c9917700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff5c4102ff0 0x7ff5c419d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:33398/0 (socket says 172.21.15.155:33398) 2024-03-19T02:44:35.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.322+0000 7ff5c9917700 1 -- 172.21.15.155:0/927661695 learned_addr learned my addr 172.21.15.155:0/927661695 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-19T02:44:35.325 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.322+0000 7ff5c9116700 1 --2- 172.21.15.155:0/927661695 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5c419dbe0 0x7ff5c4197860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.325 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.322+0000 7ff5c9917700 1 -- 172.21.15.155:0/927661695 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff5c4102ff0 msgr2=0x7ff5c419d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-19T02:44:35.325 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.322+0000 7ff5c9917700 1 -- 172.21.15.155:0/927661695 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff5c4102ff0 msgr2=0x7ff5c419d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-19T02:44:35.325 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.322+0000 7ff5c9917700 1 --2- 172.21.15.155:0/927661695 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff5c4102ff0 0x7ff5c419d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T02:44:35.325 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.322+0000 7ff5c9917700 1 --2- 172.21.15.155:0/927661695 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff5c4102ff0 0x7ff5c419d6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T02:44:35.325 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.322+0000 7ff5c9116700 1 -- 172.21.15.155:0/927661695 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff5c4102ff0 msgr2=0x7ff5c419d6a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T02:44:35.325 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.322+0000 7ff5c9116700 1 --2- 172.21.15.155:0/927661695 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff5c4102ff0 0x7ff5c419d6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.325 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.322+0000 7ff5c9116700 1 -- 172.21.15.155:0/927661695 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5c419e220 con 0x7ff5c419dbe0 2024-03-19T02:44:35.325 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.322+0000 7ff5c9116700 1 --2- 172.21.15.155:0/927661695 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5c419dbe0 0x7ff5c4197860 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff5b800b4d0 tx=0x7ff5b800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:35.325 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.322+0000 7ff5c2ffd700 1 -- 172.21.15.155:0/927661695 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7ff5b8011630 con 0x7ff5c419dbe0 2024-03-19T02:44:35.325 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.322+0000 7ff5c2ffd700 1 -- 172.21.15.155:0/927661695 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff5b8011c80 con 0x7ff5c419dbe0 2024-03-19T02:44:35.325 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.322+0000 7ff5cbb7b700 1 -- 172.21.15.155:0/927661695 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5c4197da0 con 0x7ff5c419dbe0 2024-03-19T02:44:35.326 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.322+0000 7ff5c2ffd700 1 -- 172.21.15.155:0/927661695 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7ff5b8010da0 con 0x7ff5c419dbe0 2024-03-19T02:44:35.326 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.322+0000 7ff5cbb7b700 1 -- 172.21.15.155:0/927661695 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5c4198330 con 0x7ff5c419dbe0 2024-03-19T02:44:35.327 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.323+0000 7ff5cbb7b700 1 -- 172.21.15.155:0/927661695 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5c406a4e0 con 0x7ff5c419dbe0 2024-03-19T02:44:35.327 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.323+0000 7ff5c2ffd700 1 -- 172.21.15.155:0/927661695 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7ff5b8018020 con 0x7ff5c419dbe0 2024-03-19T02:44:35.327 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.324+0000 7ff5c2ffd700 1 --2- 172.21.15.155:0/927661695 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff5ac038d60 0x7ff5ac03b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:35.327 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.324+0000 7ff5c2ffd700 1 -- 172.21.15.155:0/927661695 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff5b804ba90 con 0x7ff5c419dbe0 2024-03-19T02:44:35.327 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.324+0000 7ff5c9917700 1 --2- 172.21.15.155:0/927661695 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff5ac038d60 0x7ff5ac03b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:35.327 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.324+0000 7ff5c9917700 1 --2- 172.21.15.155:0/927661695 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff5ac038d60 0x7ff5ac03b220 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc009e60 tx=0x7ff5bc006230 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:35.329 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.329+0000 7ff5c2ffd700 1 -- 172.21.15.155:0/927661695 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff5b801f020 con 0x7ff5c419dbe0 2024-03-19T02:44:35.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 podman[35926]: 2024-03-19 02:44:35.375171253 +0000 UTC m=+0.110688638 container create 787ee6c286fe9c4ac3783c5fb2474b6e9374f6768b9d40a8271e16690532e09d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T02:44:35.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 podman[35926]: 2024-03-19 02:44:35.283648121 +0000 UTC m=+0.019165516 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:44:35.602 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.601+0000 7ff5cbb7b700 1 -- 172.21.15.155:0/927661695 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7ff5c4198d90 con 0x7ff5c419dbe0 2024-03-19T02:44:35.602 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.602+0000 7ff5c2ffd700 1 -- 172.21.15.155:0/927661695 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7ff5b8018770 con 0x7ff5c419dbe0 2024-03-19T02:44:35.603 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-19T02:44:35.603 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":2,"fsid":"2b7a8938-e59a-11ee-95c9-87774f69a715","modified":"2024-03-19T02:44:26.566826Z","created":"2024-03-19T02:42:35.143659Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:3300","nonce":0},{"type":"v1","addr":"172.21.15.33:6789","nonce":0}]},"addr":"172.21.15.33:6789/0","public_addr":"172.21.15.33:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-19T02:44:35.604 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.604+0000 7ff5cbb7b700 1 -- 172.21.15.155:0/927661695 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff5ac038d60 msgr2=0x7ff5ac03b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:35.604 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.604+0000 7ff5cbb7b700 1 --2- 172.21.15.155:0/927661695 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff5ac038d60 0x7ff5ac03b220 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc009e60 tx=0x7ff5bc006230 comp rx=0 tx=0).stop 2024-03-19T02:44:35.604 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.604+0000 7ff5cbb7b700 1 -- 172.21.15.155:0/927661695 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5c419dbe0 msgr2=0x7ff5c4197860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:35.604 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.604+0000 7ff5cbb7b700 1 --2- 172.21.15.155:0/927661695 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5c419dbe0 0x7ff5c4197860 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff5b800b4d0 tx=0x7ff5b800b890 comp rx=0 tx=0).stop 2024-03-19T02:44:35.604 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.604+0000 7ff5cbb7b700 1 -- 172.21.15.155:0/927661695 shutdown_connections 2024-03-19T02:44:35.604 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.604+0000 7ff5cbb7b700 1 --2- 172.21.15.155:0/927661695 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff5ac038d60 0x7ff5ac03b220 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.605 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.604+0000 7ff5cbb7b700 1 --2- 172.21.15.155:0/927661695 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff5c4102ff0 0x7ff5c419d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.605 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.604+0000 7ff5cbb7b700 1 --2- 172.21.15.155:0/927661695 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff5c419dbe0 0x7ff5c4197860 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:35.605 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.604+0000 7ff5cbb7b700 1 -- 172.21.15.155:0/927661695 >> 172.21.15.155:0/927661695 conn(0x7ff5c40fc7c0 msgr2=0x7ff5c4103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:35.605 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.604+0000 7ff5cbb7b700 1 -- 172.21.15.155:0/927661695 shutdown_connections 2024-03-19T02:44:35.605 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:35.604+0000 7ff5cbb7b700 1 -- 172.21.15.155:0/927661695 wait complete. 2024-03-19T02:44:35.605 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 2 2024-03-19T02:44:35.758 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 podman[35926]: 2024-03-19 02:44:35.569010215 +0000 UTC m=+0.304527613 container init 787ee6c286fe9c4ac3783c5fb2474b6e9374f6768b9d40a8271e16690532e09d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mon-b, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T02:44:35.758 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 podman[35926]: 2024-03-19 02:44:35.57184328 +0000 UTC m=+0.307360664 container start 787ee6c286fe9c4ac3783c5fb2474b6e9374f6768b9d40a8271e16690532e09d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mon-b, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T02:44:35.758 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: set uid:gid to 167:167 (ceph:ceph) 2024-03-19T02:44:35.758 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev), process ceph-mon, pid 2 2024-03-19T02:44:35.759 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: pidfile_write: ignore empty --pid-file 2024-03-19T02:44:35.759 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: load: jerasure load: lrc 2024-03-19T02:44:35.759 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: RocksDB version: 7.9.2 2024-03-19T02:44:35.759 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Git sha 0 2024-03-19T02:44:35.759 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Compile date 2024-03-18 18:57:12 2024-03-19T02:44:35.759 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: DB SUMMARY 2024-03-19T02:44:35.759 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: DB Session ID: UV7PBQVHQS4P2TORG0VY 2024-03-19T02:44:35.759 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: CURRENT file: CURRENT 2024-03-19T02:44:35.759 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: IDENTITY file: IDENTITY 2024-03-19T02:44:35.759 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-19T02:44:35.759 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-19T02:44:35.759 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-19T02:44:35.759 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.error_if_exists: 0 2024-03-19T02:44:35.760 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.create_if_missing: 0 2024-03-19T02:44:35.760 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.paranoid_checks: 1 2024-03-19T02:44:35.760 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-19T02:44:35.760 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-19T02:44:35.760 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-19T02:44:35.760 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.env: 0x556f2fd08c60 2024-03-19T02:44:35.760 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.fs: PosixFileSystem 2024-03-19T02:44:35.760 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.info_log: 0x556f3124fae0 2024-03-19T02:44:35.760 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.max_file_opening_threads: 16 2024-03-19T02:44:35.760 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.statistics: (nil) 2024-03-19T02:44:35.760 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.use_fsync: 0 2024-03-19T02:44:35.760 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.max_log_file_size: 0 2024-03-19T02:44:35.760 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-19T02:44:35.761 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-19T02:44:35.761 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.keep_log_file_num: 1000 2024-03-19T02:44:35.761 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.recycle_log_file_num: 0 2024-03-19T02:44:35.761 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.allow_fallocate: 1 2024-03-19T02:44:35.761 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.allow_mmap_reads: 0 2024-03-19T02:44:35.761 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.allow_mmap_writes: 0 2024-03-19T02:44:35.761 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.use_direct_reads: 0 2024-03-19T02:44:35.761 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-19T02:44:35.761 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.create_missing_column_families: 0 2024-03-19T02:44:35.761 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.db_log_dir: 2024-03-19T02:44:35.761 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.wal_dir: 2024-03-19T02:44:35.761 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-19T02:44:35.761 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-19T02:44:35.761 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-19T02:44:35.761 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-19T02:44:35.761 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-19T02:44:35.762 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-19T02:44:35.762 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.advise_random_on_open: 1 2024-03-19T02:44:35.762 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.db_write_buffer_size: 0 2024-03-19T02:44:35.762 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.write_buffer_manager: 0x556f312221e0 2024-03-19T02:44:35.762 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-19T02:44:35.762 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-19T02:44:35.762 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-19T02:44:35.762 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.rate_limiter: (nil) 2024-03-19T02:44:35.762 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-19T02:44:35.762 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.wal_recovery_mode: 2 2024-03-19T02:44:35.762 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.enable_thread_tracking: 0 2024-03-19T02:44:35.762 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.enable_pipelined_write: 0 2024-03-19T02:44:35.762 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.unordered_write: 0 2024-03-19T02:44:35.763 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-19T02:44:35.763 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-19T02:44:35.763 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-19T02:44:35.763 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-19T02:44:35.763 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.row_cache: None 2024-03-19T02:44:35.763 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.wal_filter: None 2024-03-19T02:44:35.763 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-19T02:44:35.763 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.allow_ingest_behind: 0 2024-03-19T02:44:35.763 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.two_write_queues: 0 2024-03-19T02:44:35.763 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.manual_wal_flush: 0 2024-03-19T02:44:35.763 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.wal_compression: 0 2024-03-19T02:44:35.763 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.atomic_flush: 0 2024-03-19T02:44:35.763 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-19T02:44:35.763 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-19T02:44:35.763 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-19T02:44:35.763 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.log_readahead_size: 0 2024-03-19T02:44:35.764 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-19T02:44:35.764 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.best_efforts_recovery: 0 2024-03-19T02:44:35.764 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-19T02:44:35.764 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-19T02:44:35.764 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.allow_data_in_errors: 0 2024-03-19T02:44:35.764 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.db_host_id: __hostname__ 2024-03-19T02:44:35.764 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-19T02:44:35.764 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.max_background_jobs: 2 2024-03-19T02:44:35.764 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.max_background_compactions: -1 2024-03-19T02:44:35.764 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.max_subcompactions: 1 2024-03-19T02:44:35.764 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-19T02:44:35.764 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-19T02:44:35.764 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-19T02:44:35.764 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.max_total_wal_size: 0 2024-03-19T02:44:35.764 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-19T02:44:35.764 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-19T02:44:35.765 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-19T02:44:35.765 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-19T02:44:35.765 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.max_open_files: -1 2024-03-19T02:44:35.765 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.bytes_per_sync: 0 2024-03-19T02:44:35.765 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-19T02:44:35.765 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-19T02:44:35.765 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.compaction_readahead_size: 0 2024-03-19T02:44:35.765 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.max_background_flushes: -1 2024-03-19T02:44:35.765 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Compression algorithms supported: 2024-03-19T02:44:35.765 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-19T02:44:35.765 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: kZSTD supported: 0 2024-03-19T02:44:35.765 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: kXpressCompression supported: 0 2024-03-19T02:44:35.765 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: kLZ4HCCompression supported: 1 2024-03-19T02:44:35.765 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: kZlibCompression supported: 1 2024-03-19T02:44:35.766 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: kSnappyCompression supported: 1 2024-03-19T02:44:35.766 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: kLZ4Compression supported: 1 2024-03-19T02:44:35.766 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: kBZip2Compression supported: 0 2024-03-19T02:44:35.766 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-19T02:44:35.766 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-19T02:44:35.766 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-19T02:44:35.766 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-19T02:44:35.766 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-19T02:44:35.766 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.merge_operator: 2024-03-19T02:44:35.766 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.compaction_filter: None 2024-03-19T02:44:35.766 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.compaction_filter_factory: None 2024-03-19T02:44:35.766 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.sst_partitioner_factory: None 2024-03-19T02:44:35.766 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-19T02:44:35.766 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-19T02:44:35.766 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556f3124fbe0) 2024-03-19T02:44:35.767 INFO:journalctl@ceph.mon.b.smithi096.stdout: cache_index_and_filter_blocks: 1 2024-03-19T02:44:35.767 INFO:journalctl@ceph.mon.b.smithi096.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-19T02:44:35.767 INFO:journalctl@ceph.mon.b.smithi096.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-19T02:44:35.767 INFO:journalctl@ceph.mon.b.smithi096.stdout: pin_top_level_index_and_filter: 1 2024-03-19T02:44:35.767 INFO:journalctl@ceph.mon.b.smithi096.stdout: index_type: 0 2024-03-19T02:44:35.767 INFO:journalctl@ceph.mon.b.smithi096.stdout: data_block_index_type: 0 2024-03-19T02:44:35.767 INFO:journalctl@ceph.mon.b.smithi096.stdout: index_shortening: 1 2024-03-19T02:44:35.767 INFO:journalctl@ceph.mon.b.smithi096.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-19T02:44:35.767 INFO:journalctl@ceph.mon.b.smithi096.stdout: checksum: 4 2024-03-19T02:44:35.767 INFO:journalctl@ceph.mon.b.smithi096.stdout: no_block_cache: 0 2024-03-19T02:44:35.767 INFO:journalctl@ceph.mon.b.smithi096.stdout: block_cache: 0x556f31255090 2024-03-19T02:44:35.767 INFO:journalctl@ceph.mon.b.smithi096.stdout: block_cache_name: BinnedLRUCache 2024-03-19T02:44:35.767 INFO:journalctl@ceph.mon.b.smithi096.stdout: block_cache_options: 2024-03-19T02:44:35.767 INFO:journalctl@ceph.mon.b.smithi096.stdout: capacity : 536870912 2024-03-19T02:44:35.767 INFO:journalctl@ceph.mon.b.smithi096.stdout: num_shard_bits : 4 2024-03-19T02:44:35.768 INFO:journalctl@ceph.mon.b.smithi096.stdout: strict_capacity_limit : 0 2024-03-19T02:44:35.768 INFO:journalctl@ceph.mon.b.smithi096.stdout: high_pri_pool_ratio: 0.000 2024-03-19T02:44:35.768 INFO:journalctl@ceph.mon.b.smithi096.stdout: block_cache_compressed: (nil) 2024-03-19T02:44:35.768 INFO:journalctl@ceph.mon.b.smithi096.stdout: persistent_cache: (nil) 2024-03-19T02:44:35.768 INFO:journalctl@ceph.mon.b.smithi096.stdout: block_size: 4096 2024-03-19T02:44:35.768 INFO:journalctl@ceph.mon.b.smithi096.stdout: block_size_deviation: 10 2024-03-19T02:44:35.768 INFO:journalctl@ceph.mon.b.smithi096.stdout: block_restart_interval: 16 2024-03-19T02:44:35.768 INFO:journalctl@ceph.mon.b.smithi096.stdout: index_block_restart_interval: 1 2024-03-19T02:44:35.768 INFO:journalctl@ceph.mon.b.smithi096.stdout: metadata_block_size: 4096 2024-03-19T02:44:35.768 INFO:journalctl@ceph.mon.b.smithi096.stdout: partition_filters: 0 2024-03-19T02:44:35.768 INFO:journalctl@ceph.mon.b.smithi096.stdout: use_delta_encoding: 1 2024-03-19T02:44:35.768 INFO:journalctl@ceph.mon.b.smithi096.stdout: filter_policy: bloomfilter 2024-03-19T02:44:35.768 INFO:journalctl@ceph.mon.b.smithi096.stdout: whole_key_filtering: 1 2024-03-19T02:44:35.768 INFO:journalctl@ceph.mon.b.smithi096.stdout: verify_compression: 0 2024-03-19T02:44:35.768 INFO:journalctl@ceph.mon.b.smithi096.stdout: read_amp_bytes_per_bit: 0 2024-03-19T02:44:35.769 INFO:journalctl@ceph.mon.b.smithi096.stdout: format_version: 5 2024-03-19T02:44:35.769 INFO:journalctl@ceph.mon.b.smithi096.stdout: enable_index_compression: 1 2024-03-19T02:44:35.769 INFO:journalctl@ceph.mon.b.smithi096.stdout: block_align: 0 2024-03-19T02:44:35.769 INFO:journalctl@ceph.mon.b.smithi096.stdout: max_auto_readahead_size: 262144 2024-03-19T02:44:35.769 INFO:journalctl@ceph.mon.b.smithi096.stdout: prepopulate_block_cache: 0 2024-03-19T02:44:35.769 INFO:journalctl@ceph.mon.b.smithi096.stdout: initial_auto_readahead_size: 8192 2024-03-19T02:44:35.770 INFO:journalctl@ceph.mon.b.smithi096.stdout: num_file_reads_for_auto_readahead: 2 2024-03-19T02:44:35.770 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.write_buffer_size: 33554432 2024-03-19T02:44:35.770 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.max_write_buffer_number: 2 2024-03-19T02:44:35.770 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.compression: NoCompression 2024-03-19T02:44:35.770 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.bottommost_compression: Disabled 2024-03-19T02:44:35.770 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.prefix_extractor: nullptr 2024-03-19T02:44:35.770 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-19T02:44:35.770 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.num_levels: 7 2024-03-19T02:44:35.771 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-19T02:44:35.771 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-19T02:44:35.771 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-19T02:44:35.771 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-19T02:44:35.771 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-19T02:44:35.771 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-19T02:44:35.771 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-19T02:44:35.771 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-19T02:44:35.771 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-19T02:44:35.771 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-19T02:44:35.771 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-19T02:44:35.771 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-19T02:44:35.771 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-19T02:44:35.772 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.compression_opts.level: 32767 2024-03-19T02:44:35.772 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.compression_opts.strategy: 0 2024-03-19T02:44:35.772 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-19T02:44:35.772 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-19T02:44:35.772 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-19T02:44:35.772 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-19T02:44:35.772 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.compression_opts.enabled: false 2024-03-19T02:44:35.772 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-19T02:44:35.772 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-19T02:44:35.772 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-19T02:44:35.772 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-19T02:44:35.772 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.target_file_size_base: 67108864 2024-03-19T02:44:35.772 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-19T02:44:35.772 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-19T02:44:35.773 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-19T02:44:35.773 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-19T02:44:35.773 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-19T02:44:35.773 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-19T02:44:35.773 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-19T02:44:35.773 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-19T02:44:35.773 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-19T02:44:35.773 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-19T02:44:35.773 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-19T02:44:35.773 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-19T02:44:35.773 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-19T02:44:35.773 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-19T02:44:35.773 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.arena_block_size: 1048576 2024-03-19T02:44:35.773 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-19T02:44:35.773 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-19T02:44:35.774 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.disable_auto_compactions: 0 2024-03-19T02:44:35.774 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-19T02:44:35.774 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-19T02:44:35.774 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-19T02:44:35.774 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-19T02:44:35.774 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-19T02:44:35.774 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-19T02:44:35.774 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-19T02:44:35.774 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-19T02:44:35.774 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-19T02:44:35.774 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-19T02:44:35.774 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.table_properties_collectors: 2024-03-19T02:44:35.774 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.inplace_update_support: 0 2024-03-19T02:44:35.774 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-19T02:44:35.774 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-19T02:44:35.775 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-19T02:44:35.775 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-19T02:44:35.775 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.bloom_locality: 0 2024-03-19T02:44:35.775 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.max_successive_merges: 0 2024-03-19T02:44:35.775 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-19T02:44:35.775 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.paranoid_file_checks: 0 2024-03-19T02:44:35.775 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.force_consistency_checks: 1 2024-03-19T02:44:35.775 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.report_bg_io_stats: 0 2024-03-19T02:44:35.775 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.ttl: 2592000 2024-03-19T02:44:35.775 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-19T02:44:35.775 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-19T02:44:35.775 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-19T02:44:35.775 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.enable_blob_files: false 2024-03-19T02:44:35.775 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.min_blob_size: 0 2024-03-19T02:44:35.776 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.blob_file_size: 268435456 2024-03-19T02:44:35.776 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-19T02:44:35.776 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-19T02:44:35.776 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-19T02:44:35.776 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-19T02:44:35.776 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-19T02:44:35.776 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.blob_file_starting_level: 0 2024-03-19T02:44:35.776 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-19T02:44:35.776 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-19T02:44:35.776 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-19T02:44:35.776 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 99861656-5d02-4b55-9436-5a6dc8a8989c 2024-03-19T02:44:35.776 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710816275601715, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-19T02:44:35.776 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-19T02:44:35.776 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710816275602165, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710816275, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "99861656-5d02-4b55-9436-5a6dc8a8989c", "db_session_id": "UV7PBQVHQS4P2TORG0VY", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-19T02:44:35.777 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710816275602244, "job": 1, "event": "recovery_finished"} 2024-03-19T02:44:35.777 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-19T02:44:35.777 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-19T02:44:35.777 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x556f31348000 2024-03-19T02:44:35.777 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: DB pointer 0x556f31332000 2024-03-19T02:44:35.777 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-19T02:44:35.777 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-19T02:44:35.777 INFO:journalctl@ceph.mon.b.smithi096.stdout: ** DB Stats ** 2024-03-19T02:44:35.777 INFO:journalctl@ceph.mon.b.smithi096.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-19T02:44:35.777 INFO:journalctl@ceph.mon.b.smithi096.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-19T02:44:35.777 INFO:journalctl@ceph.mon.b.smithi096.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-19T02:44:35.777 INFO:journalctl@ceph.mon.b.smithi096.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-19T02:44:35.777 INFO:journalctl@ceph.mon.b.smithi096.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-19T02:44:35.777 INFO:journalctl@ceph.mon.b.smithi096.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-19T02:44:35.777 INFO:journalctl@ceph.mon.b.smithi096.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-19T02:44:35.777 INFO:journalctl@ceph.mon.b.smithi096.stdout: 2024-03-19T02:44:35.778 INFO:journalctl@ceph.mon.b.smithi096.stdout: ** Compaction Stats [default] ** 2024-03-19T02:44:35.778 INFO:journalctl@ceph.mon.b.smithi096.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-19T02:44:35.778 INFO:journalctl@ceph.mon.b.smithi096.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-19T02:44:35.778 INFO:journalctl@ceph.mon.b.smithi096.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-19T02:44:35.778 INFO:journalctl@ceph.mon.b.smithi096.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-19T02:44:35.778 INFO:journalctl@ceph.mon.b.smithi096.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-19T02:44:35.778 INFO:journalctl@ceph.mon.b.smithi096.stdout: 2024-03-19T02:44:35.778 INFO:journalctl@ceph.mon.b.smithi096.stdout: ** Compaction Stats [default] ** 2024-03-19T02:44:35.778 INFO:journalctl@ceph.mon.b.smithi096.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-19T02:44:35.778 INFO:journalctl@ceph.mon.b.smithi096.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-19T02:44:35.778 INFO:journalctl@ceph.mon.b.smithi096.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-19T02:44:35.778 INFO:journalctl@ceph.mon.b.smithi096.stdout: 2024-03-19T02:44:35.778 INFO:journalctl@ceph.mon.b.smithi096.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-19T02:44:35.778 INFO:journalctl@ceph.mon.b.smithi096.stdout: 2024-03-19T02:44:35.779 INFO:journalctl@ceph.mon.b.smithi096.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-19T02:44:35.779 INFO:journalctl@ceph.mon.b.smithi096.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-19T02:44:35.779 INFO:journalctl@ceph.mon.b.smithi096.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-19T02:44:35.779 INFO:journalctl@ceph.mon.b.smithi096.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-19T02:44:35.779 INFO:journalctl@ceph.mon.b.smithi096.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-19T02:44:35.779 INFO:journalctl@ceph.mon.b.smithi096.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-19T02:44:35.779 INFO:journalctl@ceph.mon.b.smithi096.stdout: Cumulative compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-19T02:44:35.779 INFO:journalctl@ceph.mon.b.smithi096.stdout: Interval compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-19T02:44:35.779 INFO:journalctl@ceph.mon.b.smithi096.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-19T02:44:35.779 INFO:journalctl@ceph.mon.b.smithi096.stdout: Block cache BinnedLRUCache@0x556f31255090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-19T02:44:35.779 INFO:journalctl@ceph.mon.b.smithi096.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-19T02:44:35.779 INFO:journalctl@ceph.mon.b.smithi096.stdout: 2024-03-19T02:44:35.779 INFO:journalctl@ceph.mon.b.smithi096.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-19T02:44:35.779 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-19T02:44:35.780 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: using public_addr v2:172.21.15.96:0/0 -> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] 2024-03-19T02:44:35.780 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: starting mon.b rank -1 at public addrs [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] at bind addrs [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T02:44:35.780 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: mon.b@-1(???) e0 preinit fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T02:44:35.780 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: mon.b@-1(synchronizing).mds e1 new map 2024-03-19T02:44:35.780 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-19T02:44:35.780 INFO:journalctl@ceph.mon.b.smithi096.stdout: e1 2024-03-19T02:44:35.780 INFO:journalctl@ceph.mon.b.smithi096.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-19T02:44:35.780 INFO:journalctl@ceph.mon.b.smithi096.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-19T02:44:35.780 INFO:journalctl@ceph.mon.b.smithi096.stdout: legacy client fscid: -1 2024-03-19T02:44:35.780 INFO:journalctl@ceph.mon.b.smithi096.stdout: 2024-03-19T02:44:35.780 INFO:journalctl@ceph.mon.b.smithi096.stdout: No filesystems configured 2024-03-19T02:44:35.780 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-19T02:44:35.780 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-19T02:44:35.780 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-19T02:44:35.780 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-19T02:44:35.781 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-19T02:44:35.781 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-19T02:44:35.781 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-19T02:44:35.781 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-19T02:44:35.781 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-19T02:44:35.781 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-19T02:44:35.781 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:35.781 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:44:35.781 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:35.781 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:35.781 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi155", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:44:35.781 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:35.781 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:35.781 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:35.782 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:35.782 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi096", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:44:35.782 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:35.782 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:44:35.782 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: Updating smithi096:/etc/ceph/ceph.conf 2024-03-19T02:44:35.782 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: Updating smithi096:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:44:35.782 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: Deploying cephadm binary to smithi155 2024-03-19T02:44:35.782 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: Updating smithi096:/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:44:35.782 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: Updating smithi096:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T02:44:35.782 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:35.782 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:35.782 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:35.782 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:35.782 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:35.783 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:35.783 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: Added host smithi155 2024-03-19T02:44:35.783 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:44:35.783 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:35.783 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:35.783 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:35.783 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:35.783 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:44:35.783 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:35.783 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2708956598' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-19T02:44:35.783 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:35.783 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2708956598' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-19T02:44:35.783 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T02:44:35.783 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:35.783 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:35.783 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:35.784 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:35.784 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi155", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:44:35.784 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:35.784 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:44:35.784 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: Updating smithi155:/etc/ceph/ceph.conf 2024-03-19T02:44:35.784 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: Updating smithi155:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:44:35.784 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: Updating smithi155:/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:44:35.784 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: Updating smithi155:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T02:44:35.784 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:35.784 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:35.784 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:35.784 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:35.784 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:35.784 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:44:35.784 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:35.785 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:44:35.785 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:35.785 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T02:44:35.785 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:35.785 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi033:172.21.15.33=a;smithi096:172.21.15.96=b;smithi155:172.21.15.155=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:44:35.785 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: Saving service mon spec with placement smithi033:172.21.15.33=a;smithi096:172.21.15.96=b;smithi155:172.21.15.155=c;count:3 2024-03-19T02:44:35.785 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: Deploying daemon mon.c on smithi155 2024-03-19T02:44:35.785 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:35.785 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:35.785 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='client.? 172.21.15.155:0/2189142395' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T02:44:35.785 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:44:35.785 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: mon.a calling monitor election 2024-03-19T02:44:35.785 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:44:35.785 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:44:35.785 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:35.785 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:44:35.786 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: mon.c calling monitor election 2024-03-19T02:44:35.786 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:44:35.786 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:35.786 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:44:35.786 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:44:35.786 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-19T02:44:35.786 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: monmap epoch 2 2024-03-19T02:44:35.786 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T02:44:35.786 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: last_changed 2024-03-19T02:44:26.566826+0000 2024-03-19T02:44:35.786 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: created 2024-03-19T02:42:35.143659+0000 2024-03-19T02:44:35.786 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: min_mon_release 19 (squid) 2024-03-19T02:44:35.786 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: election_strategy: 1 2024-03-19T02:44:35.786 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: 0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2024-03-19T02:44:35.786 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: 1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.c 2024-03-19T02:44:35.786 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: fsmap 2024-03-19T02:44:35.786 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T02:44:35.787 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: mgrmap e13: x(active, since 51s) 2024-03-19T02:44:35.787 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: overall HEALTH_OK 2024-03-19T02:44:35.787 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:35.787 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:35.787 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:35.787 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T02:44:35.787 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:35.787 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:35.787 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: Deploying daemon mon.b on smithi096 2024-03-19T02:44:35.787 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='client.? 172.21.15.155:0/1487961750' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T02:44:35.787 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:44:35.787 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:35.787 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 ceph-mon[35946]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-19T02:44:35.787 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 bash[35926]: 787ee6c286fe9c4ac3783c5fb2474b6e9374f6768b9d40a8271e16690532e09d 2024-03-19T02:44:35.787 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:35 smithi096 systemd[1]: Started Ceph mon.b for 2b7a8938-e59a-11ee-95c9-87774f69a715. 2024-03-19T02:44:37.137 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-19T02:44:37.138 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-19T02:44:37.381 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T02:44:41.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:44:41.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: mon.a calling monitor election 2024-03-19T02:44:41.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: mon.c calling monitor election 2024-03-19T02:44:41.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:41.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:44:41.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:44:41.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:44:41.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:44:41.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:41.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:44:41.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:44:41.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:41.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:44:41.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-19T02:44:41.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: monmap epoch 3 2024-03-19T02:44:41.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T02:44:41.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: last_changed 2024-03-19T02:44:35.629697+0000 2024-03-19T02:44:41.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: created 2024-03-19T02:42:35.143659+0000 2024-03-19T02:44:41.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: min_mon_release 19 (squid) 2024-03-19T02:44:41.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: election_strategy: 1 2024-03-19T02:44:41.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: 0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2024-03-19T02:44:41.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: 1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.c 2024-03-19T02:44:41.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: 2: [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon.b 2024-03-19T02:44:41.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: fsmap 2024-03-19T02:44:41.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T02:44:41.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: mgrmap e13: x(active, since 61s) 2024-03-19T02:44:41.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-19T02:44:41.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-19T02:44:41.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-19T02:44:41.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: mon.b (rank 2) addr [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] is down (out of quorum) 2024-03-19T02:44:41.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:41.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:41.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:41.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:44:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: mon.a calling monitor election 2024-03-19T02:44:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: mon.c calling monitor election 2024-03-19T02:44:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:41.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:44:41.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:44:41.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:44:41.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:44:41.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:41.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:44:41.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:44:41.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:41.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:44:41.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-19T02:44:41.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: monmap epoch 3 2024-03-19T02:44:41.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T02:44:41.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: last_changed 2024-03-19T02:44:35.629697+0000 2024-03-19T02:44:41.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: created 2024-03-19T02:42:35.143659+0000 2024-03-19T02:44:41.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: min_mon_release 19 (squid) 2024-03-19T02:44:41.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: election_strategy: 1 2024-03-19T02:44:41.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: 0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2024-03-19T02:44:41.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: 1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.c 2024-03-19T02:44:41.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: 2: [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon.b 2024-03-19T02:44:41.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: fsmap 2024-03-19T02:44:41.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T02:44:41.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: mgrmap e13: x(active, since 61s) 2024-03-19T02:44:41.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-19T02:44:41.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-19T02:44:41.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-19T02:44:41.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: mon.b (rank 2) addr [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] is down (out of quorum) 2024-03-19T02:44:41.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:41.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:41.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:41.031 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:41.971 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.971+0000 7feb90266700 1 -- 172.21.15.155:0/3459906118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feb88102050 msgr2=0x7feb74005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:41.972 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.971+0000 7feb90266700 1 --2- 172.21.15.155:0/3459906118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feb88102050 0x7feb74005610 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7feb84005fa0 tx=0x7feb8400ffc0 comp rx=0 tx=0).stop 2024-03-19T02:44:41.972 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.971+0000 7feb90266700 1 -- 172.21.15.155:0/3459906118 shutdown_connections 2024-03-19T02:44:41.972 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.971+0000 7feb90266700 1 --2- 172.21.15.155:0/3459906118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feb88102050 0x7feb74005610 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:41.972 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.971+0000 7feb90266700 1 -- 172.21.15.155:0/3459906118 >> 172.21.15.155:0/3459906118 conn(0x7feb880fd8e0 msgr2=0x7feb880ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:41.972 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.971+0000 7feb90266700 1 -- 172.21.15.155:0/3459906118 shutdown_connections 2024-03-19T02:44:41.972 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.971+0000 7feb90266700 1 -- 172.21.15.155:0/3459906118 wait complete. 2024-03-19T02:44:41.972 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:41 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:44:41.973 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:41 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:44:41.973 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:41 smithi155 ceph-mon[35804]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:41.975 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.972+0000 7feb90266700 1 Processor -- start 2024-03-19T02:44:41.975 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.972+0000 7feb90266700 1 -- start start 2024-03-19T02:44:41.975 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.972+0000 7feb90266700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feb88102050 0x7feb8819d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:41.975 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.972+0000 7feb90266700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb8819dbb0 0x7feb88197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:41.975 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.972+0000 7feb90266700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb88197c80 0x7feb88198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:41.975 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.972+0000 7feb90266700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7feb88109ce0 con 0x7feb88102050 2024-03-19T02:44:41.975 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.972+0000 7feb90266700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7feb88109b60 con 0x7feb8819dbb0 2024-03-19T02:44:41.975 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.972+0000 7feb90266700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7feb88109e60 con 0x7feb88197c80 2024-03-19T02:44:41.976 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.972+0000 7feb8e002700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feb88102050 0x7feb8819d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:41.976 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.972+0000 7feb8e002700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feb88102050 0x7feb8819d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.155:49934/0 (socket says 172.21.15.155:49934) 2024-03-19T02:44:41.976 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.972+0000 7feb8e002700 1 -- 172.21.15.155:0/2123201604 learned_addr learned my addr 172.21.15.155:0/2123201604 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-19T02:44:41.976 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.972+0000 7feb8e803700 1 --2- 172.21.15.155:0/2123201604 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb88197c80 0x7feb88198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:41.976 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.972+0000 7feb8d801700 1 --2- 172.21.15.155:0/2123201604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb8819dbb0 0x7feb88197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:41.976 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.973+0000 7feb8e002700 1 -- 172.21.15.155:0/2123201604 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb88197c80 msgr2=0x7feb88198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:41.976 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.973+0000 7feb8d801700 1 -- 172.21.15.155:0/2123201604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb8819dbb0 msgr2=0x7feb88197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-19T02:44:41.976 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.973+0000 7feb8e002700 1 --2- 172.21.15.155:0/2123201604 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb88197c80 0x7feb88198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:41.976 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.973+0000 7feb8d801700 1 -- 172.21.15.155:0/2123201604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb8819dbb0 msgr2=0x7feb88197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-19T02:44:41.976 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.973+0000 7feb8d801700 1 --2- 172.21.15.155:0/2123201604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb8819dbb0 0x7feb88197740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T02:44:41.976 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.973+0000 7feb8e002700 1 -- 172.21.15.155:0/2123201604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb8819dbb0 msgr2=0x7feb88197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:41.976 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.973+0000 7feb8d801700 1 --2- 172.21.15.155:0/2123201604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb8819dbb0 0x7feb88197740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T02:44:41.976 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.973+0000 7feb8e002700 1 --2- 172.21.15.155:0/2123201604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb8819dbb0 0x7feb88197740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:41.976 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.973+0000 7feb8e002700 1 -- 172.21.15.155:0/2123201604 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb881a4420 con 0x7feb88102050 2024-03-19T02:44:41.977 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.973+0000 7feb8e002700 1 --2- 172.21.15.155:0/2123201604 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feb88102050 0x7feb8819d670 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7feb840137f0 tx=0x7feb84005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:41.977 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.973+0000 7feb7effd700 1 -- 172.21.15.155:0/2123201604 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb84004ba0 con 0x7feb88102050 2024-03-19T02:44:41.977 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.973+0000 7feb7effd700 1 -- 172.21.15.155:0/2123201604 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb8401b8e0 con 0x7feb88102050 2024-03-19T02:44:41.977 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.973+0000 7feb90266700 1 -- 172.21.15.155:0/2123201604 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb881a4650 con 0x7feb88102050 2024-03-19T02:44:41.977 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.973+0000 7feb7effd700 1 -- 172.21.15.155:0/2123201604 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb84011890 con 0x7feb88102050 2024-03-19T02:44:41.977 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.973+0000 7feb90266700 1 -- 172.21.15.155:0/2123201604 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb881a4be0 con 0x7feb88102050 2024-03-19T02:44:41.977 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.974+0000 7feb7effd700 1 -- 172.21.15.155:0/2123201604 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7feb84011a30 con 0x7feb88102050 2024-03-19T02:44:41.977 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.974+0000 7feb90266700 1 -- 172.21.15.155:0/2123201604 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb88103530 con 0x7feb88102050 2024-03-19T02:44:41.977 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.974+0000 7feb7effd700 1 --2- 172.21.15.155:0/2123201604 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7feb6c039450 0x7feb6c03b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:41.977 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.974+0000 7feb7effd700 1 -- 172.21.15.155:0/2123201604 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7feb8404e560 con 0x7feb88102050 2024-03-19T02:44:41.977 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.974+0000 7feb8d801700 1 --2- 172.21.15.155:0/2123201604 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7feb6c039450 0x7feb6c03b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:41.978 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.975+0000 7feb8d801700 1 --2- 172.21.15.155:0/2123201604 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7feb6c039450 0x7feb6c03b910 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7feb78009850 tx=0x7feb78008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:41.980 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:41.980+0000 7feb7effd700 1 -- 172.21.15.155:0/2123201604 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7feb84022070 con 0x7feb88102050 2024-03-19T02:44:42.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:41 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:44:42.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:41 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:44:42.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:41 smithi033 ceph-mon[31973]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:42.270 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:42.269+0000 7feb90266700 1 -- 172.21.15.155:0/2123201604 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7feb8802dbb0 con 0x7feb88102050 2024-03-19T02:44:42.270 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:42.270+0000 7feb7effd700 1 -- 172.21.15.155:0/2123201604 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1279 (secure 0 0 0) 0x7feb84019bf0 con 0x7feb88102050 2024-03-19T02:44:42.270 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-03-19T02:44:42.270 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":3,"fsid":"2b7a8938-e59a-11ee-95c9-87774f69a715","modified":"2024-03-19T02:44:35.629697Z","created":"2024-03-19T02:42:35.143659Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:3300","nonce":0},{"type":"v1","addr":"172.21.15.33:6789","nonce":0}]},"addr":"172.21.15.33:6789/0","public_addr":"172.21.15.33:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:3300","nonce":0},{"type":"v1","addr":"172.21.15.96:6789","nonce":0}]},"addr":"172.21.15.96:6789/0","public_addr":"172.21.15.96:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-19T02:44:42.273 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:42.272+0000 7feb90266700 1 -- 172.21.15.155:0/2123201604 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7feb6c039450 msgr2=0x7feb6c03b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:42.273 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:42.272+0000 7feb90266700 1 --2- 172.21.15.155:0/2123201604 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7feb6c039450 0x7feb6c03b910 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7feb78009850 tx=0x7feb78008040 comp rx=0 tx=0).stop 2024-03-19T02:44:42.273 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:42.272+0000 7feb90266700 1 -- 172.21.15.155:0/2123201604 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feb88102050 msgr2=0x7feb8819d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:42.273 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:42.272+0000 7feb90266700 1 --2- 172.21.15.155:0/2123201604 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feb88102050 0x7feb8819d670 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7feb840137f0 tx=0x7feb84005250 comp rx=0 tx=0).stop 2024-03-19T02:44:42.273 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:42.272+0000 7feb90266700 1 -- 172.21.15.155:0/2123201604 shutdown_connections 2024-03-19T02:44:42.273 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:42.272+0000 7feb90266700 1 --2- 172.21.15.155:0/2123201604 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7feb6c039450 0x7feb6c03b910 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:42.273 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:42.272+0000 7feb90266700 1 --2- 172.21.15.155:0/2123201604 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb88197c80 0x7feb88198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:42.273 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:42.272+0000 7feb90266700 1 --2- 172.21.15.155:0/2123201604 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feb88102050 0x7feb8819d670 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:42.273 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:42.272+0000 7feb90266700 1 --2- 172.21.15.155:0/2123201604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feb8819dbb0 0x7feb88197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:42.273 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:42.272+0000 7feb90266700 1 -- 172.21.15.155:0/2123201604 >> 172.21.15.155:0/2123201604 conn(0x7feb880fd8e0 msgr2=0x7feb880febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:42.273 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:42.272+0000 7feb90266700 1 -- 172.21.15.155:0/2123201604 shutdown_connections 2024-03-19T02:44:42.273 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:42.272+0000 7feb90266700 1 -- 172.21.15.155:0/2123201604 wait complete. 2024-03-19T02:44:42.274 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 3 2024-03-19T02:44:42.875 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-19T02:44:42.876 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph config generate-minimal-conf 2024-03-19T02:44:42.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:44:42.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: mon.a calling monitor election 2024-03-19T02:44:42.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: mon.c calling monitor election 2024-03-19T02:44:42.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:42.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:44:42.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:44:42.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:44:42.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:44:42.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:42.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:44:42.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:44:42.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:42.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:44:42.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-19T02:44:42.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: monmap epoch 3 2024-03-19T02:44:42.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T02:44:42.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: last_changed 2024-03-19T02:44:35.629697+0000 2024-03-19T02:44:42.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: created 2024-03-19T02:42:35.143659+0000 2024-03-19T02:44:42.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: min_mon_release 19 (squid) 2024-03-19T02:44:42.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: election_strategy: 1 2024-03-19T02:44:42.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: 0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2024-03-19T02:44:42.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: 1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.c 2024-03-19T02:44:42.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: 2: [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon.b 2024-03-19T02:44:42.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: fsmap 2024-03-19T02:44:42.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T02:44:42.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: mgrmap e13: x(active, since 61s) 2024-03-19T02:44:42.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-19T02:44:42.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-19T02:44:42.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-19T02:44:42.968 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: mon.b (rank 2) addr [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] is down (out of quorum) 2024-03-19T02:44:42.968 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:42.968 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:42.968 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:42.968 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:42.968 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:44:42.968 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:44:42.968 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:42 smithi096 ceph-mon[35946]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:43.119 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:44:43.923 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: mon.b calling monitor election 2024-03-19T02:44:43.923 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: Updating smithi033:/etc/ceph/ceph.conf 2024-03-19T02:44:43.923 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: Updating smithi096:/etc/ceph/ceph.conf 2024-03-19T02:44:43.923 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: Updating smithi155:/etc/ceph/ceph.conf 2024-03-19T02:44:43.923 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: Updating smithi033:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:44:43.923 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: Updating smithi096:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:44:43.923 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: Updating smithi155:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:44:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: mon.b calling monitor election 2024-03-19T02:44:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: mon.c calling monitor election 2024-03-19T02:44:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: mon.a calling monitor election 2024-03-19T02:44:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-19T02:44:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: monmap epoch 3 2024-03-19T02:44:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T02:44:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: last_changed 2024-03-19T02:44:35.629697+0000 2024-03-19T02:44:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: created 2024-03-19T02:42:35.143659+0000 2024-03-19T02:44:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: min_mon_release 19 (squid) 2024-03-19T02:44:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: election_strategy: 1 2024-03-19T02:44:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: 0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2024-03-19T02:44:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: 1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.c 2024-03-19T02:44:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: 2: [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon.b 2024-03-19T02:44:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: fsmap 2024-03-19T02:44:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T02:44:43.924 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: mgrmap e13: x(active, since 63s) 2024-03-19T02:44:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-19T02:44:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: Cluster is now healthy 2024-03-19T02:44:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: overall HEALTH_OK 2024-03-19T02:44:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: Reconfiguring mon.a (unknown last config time)... 2024-03-19T02:44:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T02:44:43.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T02:44:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: Reconfiguring daemon mon.a on smithi033 2024-03-19T02:44:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:44:43.926 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:43 smithi033 ceph-mon[31973]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:43.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: mon.b calling monitor election 2024-03-19T02:44:43.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: Updating smithi033:/etc/ceph/ceph.conf 2024-03-19T02:44:43.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: Updating smithi096:/etc/ceph/ceph.conf 2024-03-19T02:44:43.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: Updating smithi155:/etc/ceph/ceph.conf 2024-03-19T02:44:43.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: Updating smithi033:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:44:43.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: Updating smithi096:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:44:43.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: Updating smithi155:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:44:43.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: mon.b calling monitor election 2024-03-19T02:44:43.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: mon.c calling monitor election 2024-03-19T02:44:43.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: mon.a calling monitor election 2024-03-19T02:44:43.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-19T02:44:43.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: monmap epoch 3 2024-03-19T02:44:43.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T02:44:43.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: last_changed 2024-03-19T02:44:35.629697+0000 2024-03-19T02:44:43.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: created 2024-03-19T02:42:35.143659+0000 2024-03-19T02:44:43.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: min_mon_release 19 (squid) 2024-03-19T02:44:43.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: election_strategy: 1 2024-03-19T02:44:43.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: 0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2024-03-19T02:44:43.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: 1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.c 2024-03-19T02:44:43.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: 2: [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon.b 2024-03-19T02:44:43.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: fsmap 2024-03-19T02:44:43.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T02:44:43.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: mgrmap e13: x(active, since 63s) 2024-03-19T02:44:43.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-19T02:44:43.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: Cluster is now healthy 2024-03-19T02:44:43.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: overall HEALTH_OK 2024-03-19T02:44:43.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:43.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:43.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:43.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:43.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:43.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:43.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:43.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:43.968 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:43.968 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:43.968 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:43.968 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: Reconfiguring mon.a (unknown last config time)... 2024-03-19T02:44:43.968 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T02:44:43.968 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T02:44:43.968 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:43.968 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: Reconfiguring daemon mon.a on smithi033 2024-03-19T02:44:43.968 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:44:43.968 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:43 smithi096 ceph-mon[35946]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:44.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: mon.b calling monitor election 2024-03-19T02:44:44.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: Updating smithi033:/etc/ceph/ceph.conf 2024-03-19T02:44:44.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: Updating smithi096:/etc/ceph/ceph.conf 2024-03-19T02:44:44.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: Updating smithi155:/etc/ceph/ceph.conf 2024-03-19T02:44:44.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: Updating smithi033:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:44:44.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: Updating smithi096:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:44:44.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: Updating smithi155:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:44:44.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: mon.b calling monitor election 2024-03-19T02:44:44.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: mon.c calling monitor election 2024-03-19T02:44:44.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: mon.a calling monitor election 2024-03-19T02:44:44.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-19T02:44:44.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: monmap epoch 3 2024-03-19T02:44:44.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T02:44:44.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: last_changed 2024-03-19T02:44:35.629697+0000 2024-03-19T02:44:44.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: created 2024-03-19T02:42:35.143659+0000 2024-03-19T02:44:44.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: min_mon_release 19 (squid) 2024-03-19T02:44:44.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: election_strategy: 1 2024-03-19T02:44:44.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: 0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2024-03-19T02:44:44.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: 1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.c 2024-03-19T02:44:44.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: 2: [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon.b 2024-03-19T02:44:44.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: fsmap 2024-03-19T02:44:44.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T02:44:44.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: mgrmap e13: x(active, since 63s) 2024-03-19T02:44:44.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-19T02:44:44.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: Cluster is now healthy 2024-03-19T02:44:44.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: overall HEALTH_OK 2024-03-19T02:44:44.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:44.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:44.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:44.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:44.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:44.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:44.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:44.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:44.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:44.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:44.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:44.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: Reconfiguring mon.a (unknown last config time)... 2024-03-19T02:44:44.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T02:44:44.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T02:44:44.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:44.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: Reconfiguring daemon mon.a on smithi033 2024-03-19T02:44:44.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:44:44.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:43 smithi155 ceph-mon[35804]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:45.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.660+0000 7f6da0790700 1 -- 172.21.15.33:0/2058157476 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6d98102d20 msgr2=0x7f6d9810f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:45.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.660+0000 7f6da0790700 1 --2- 172.21.15.33:0/2058157476 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6d98102d20 0x7f6d9810f1c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6d9000b0a0 tx=0x7f6d9000b3b0 comp rx=0 tx=0).stop 2024-03-19T02:44:45.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.660+0000 7f6da0790700 1 -- 172.21.15.33:0/2058157476 shutdown_connections 2024-03-19T02:44:45.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.660+0000 7f6da0790700 1 --2- 172.21.15.33:0/2058157476 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6d98102d20 0x7f6d9810f1c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:45.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.660+0000 7f6da0790700 1 --2- 172.21.15.33:0/2058157476 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d98108410 0x7f6d981087f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:45.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.660+0000 7f6da0790700 1 --2- 172.21.15.33:0/2058157476 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d98102360 0x7f6d981027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:45.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.660+0000 7f6da0790700 1 -- 172.21.15.33:0/2058157476 >> 172.21.15.33:0/2058157476 conn(0x7f6d98079440 msgr2=0x7f6d98079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:45.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.660+0000 7f6da0790700 1 -- 172.21.15.33:0/2058157476 shutdown_connections 2024-03-19T02:44:45.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.660+0000 7f6da0790700 1 -- 172.21.15.33:0/2058157476 wait complete. 2024-03-19T02:44:45.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.660+0000 7f6da0790700 1 Processor -- start 2024-03-19T02:44:45.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.660+0000 7f6da0790700 1 -- start start 2024-03-19T02:44:45.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.661+0000 7f6da0790700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d98102360 0x7f6d9819e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:45.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.661+0000 7f6da0790700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6d98102d20 0x7f6d9819eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:45.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.661+0000 7f6da0790700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d98108410 0x7f6d981987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:45.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.661+0000 7f6da0790700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6d98112550 con 0x7f6d98102d20 2024-03-19T02:44:45.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.661+0000 7f6da0790700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6d981123d0 con 0x7f6d98102360 2024-03-19T02:44:45.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.661+0000 7f6da0790700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6d981126d0 con 0x7f6d98108410 2024-03-19T02:44:45.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.661+0000 7f6d9dd2b700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6d98102d20 0x7f6d9819eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:45.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.661+0000 7f6d9dd2b700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6d98102d20 0x7f6d9819eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58320/0 (socket says 172.21.15.33:58320) 2024-03-19T02:44:45.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.661+0000 7f6d9dd2b700 1 -- 172.21.15.33:0/1332068913 learned_addr learned my addr 172.21.15.33:0/1332068913 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:44:45.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.661+0000 7f6d9e52c700 1 --2- 172.21.15.33:0/1332068913 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d98102360 0x7f6d9819e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:45.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.661+0000 7f6d9ed2d700 1 --2- 172.21.15.33:0/1332068913 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d98108410 0x7f6d981987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:45.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.661+0000 7f6d9dd2b700 1 -- 172.21.15.33:0/1332068913 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d98108410 msgr2=0x7f6d981987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:45.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.661+0000 7f6d9dd2b700 1 --2- 172.21.15.33:0/1332068913 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d98108410 0x7f6d981987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:45.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.661+0000 7f6d9dd2b700 1 -- 172.21.15.33:0/1332068913 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d98102360 msgr2=0x7f6d9819e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:45.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.661+0000 7f6d9dd2b700 1 --2- 172.21.15.33:0/1332068913 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d98102360 0x7f6d9819e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:45.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.661+0000 7f6d9dd2b700 1 -- 172.21.15.33:0/1332068913 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d98198f90 con 0x7f6d98102d20 2024-03-19T02:44:45.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.661+0000 7f6d9dd2b700 1 --2- 172.21.15.33:0/1332068913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6d98102d20 0x7f6d9819eb70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6d8800b530 tx=0x7f6d8800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:45.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.661+0000 7f6d8f7fe700 1 -- 172.21.15.33:0/1332068913 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d88015020 con 0x7f6d98102d20 2024-03-19T02:44:45.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.661+0000 7f6da0790700 1 -- 172.21.15.33:0/1332068913 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d98199280 con 0x7f6d98102d20 2024-03-19T02:44:45.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.661+0000 7f6d8f7fe700 1 -- 172.21.15.33:0/1332068913 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6d88011ce0 con 0x7f6d98102d20 2024-03-19T02:44:45.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.661+0000 7f6d8f7fe700 1 -- 172.21.15.33:0/1332068913 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d88010e40 con 0x7f6d98102d20 2024-03-19T02:44:45.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.661+0000 7f6da0790700 1 -- 172.21.15.33:0/1332068913 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d981a5490 con 0x7f6d98102d20 2024-03-19T02:44:45.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.662+0000 7f6d8f7fe700 1 -- 172.21.15.33:0/1332068913 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f6d88018020 con 0x7f6d98102d20 2024-03-19T02:44:45.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.662+0000 7f6da0790700 1 -- 172.21.15.33:0/1332068913 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d98103a50 con 0x7f6d98102d20 2024-03-19T02:44:45.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.662+0000 7f6d8f7fe700 1 --2- 172.21.15.33:0/1332068913 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f6d84039200 0x7f6d8403b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:45.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.663+0000 7f6d8f7fe700 1 -- 172.21.15.33:0/1332068913 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6d8804b990 con 0x7f6d98102d20 2024-03-19T02:44:45.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.663+0000 7f6d9e52c700 1 --2- 172.21.15.33:0/1332068913 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f6d84039200 0x7f6d8403b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:45.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.663+0000 7f6d9e52c700 1 --2- 172.21.15.33:0/1332068913 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f6d84039200 0x7f6d8403b6c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6d94005950 tx=0x7f6d9400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:45.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.668+0000 7f6d8f7fe700 1 -- 172.21.15.33:0/1332068913 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6d8801f020 con 0x7f6d98102d20 2024-03-19T02:44:45.774 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:45 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:45.774 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:45 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:45.774 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:45 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:45.774 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:45 smithi096 ceph-mon[35946]: Reconfiguring mon.b (monmap changed)... 2024-03-19T02:44:45.774 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:45 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T02:44:45.774 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:45 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T02:44:45.775 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:45 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:45.775 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:45 smithi096 ceph-mon[35946]: Reconfiguring daemon mon.b on smithi096 2024-03-19T02:44:45.775 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:45 smithi096 ceph-mon[35946]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:45.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.873+0000 7f6da0790700 1 -- 172.21.15.33:0/1332068913 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f6d9802dbb0 con 0x7f6d98102d20 2024-03-19T02:44:45.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.873+0000 7f6d8f7fe700 1 -- 172.21.15.33:0/1332068913 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7f6d88017dd0 con 0x7f6d98102d20 2024-03-19T02:44:45.874 INFO:teuthology.orchestra.run.smithi033.stdout:# minimal ceph.conf for 2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T02:44:45.874 INFO:teuthology.orchestra.run.smithi033.stdout:[global] 2024-03-19T02:44:45.874 INFO:teuthology.orchestra.run.smithi033.stdout: fsid = 2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T02:44:45.875 INFO:teuthology.orchestra.run.smithi033.stdout: mon_host = [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-03-19T02:44:45.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.875+0000 7f6da0790700 1 -- 172.21.15.33:0/1332068913 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f6d84039200 msgr2=0x7f6d8403b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:45.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.875+0000 7f6da0790700 1 --2- 172.21.15.33:0/1332068913 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f6d84039200 0x7f6d8403b6c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6d94005950 tx=0x7f6d9400b410 comp rx=0 tx=0).stop 2024-03-19T02:44:45.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.875+0000 7f6da0790700 1 -- 172.21.15.33:0/1332068913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6d98102d20 msgr2=0x7f6d9819eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:45.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.875+0000 7f6da0790700 1 --2- 172.21.15.33:0/1332068913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6d98102d20 0x7f6d9819eb70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6d8800b530 tx=0x7f6d8800b840 comp rx=0 tx=0).stop 2024-03-19T02:44:45.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.875+0000 7f6da0790700 1 -- 172.21.15.33:0/1332068913 shutdown_connections 2024-03-19T02:44:45.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.876+0000 7f6da0790700 1 --2- 172.21.15.33:0/1332068913 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f6d84039200 0x7f6d8403b6c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:45.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.876+0000 7f6da0790700 1 --2- 172.21.15.33:0/1332068913 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d98108410 0x7f6d981987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:45.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.876+0000 7f6da0790700 1 --2- 172.21.15.33:0/1332068913 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d98102360 0x7f6d9819e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:45.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.876+0000 7f6da0790700 1 --2- 172.21.15.33:0/1332068913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6d98102d20 0x7f6d9819eb70 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:45.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.876+0000 7f6da0790700 1 -- 172.21.15.33:0/1332068913 >> 172.21.15.33:0/1332068913 conn(0x7f6d98079440 msgr2=0x7f6d981011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:45.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.876+0000 7f6da0790700 1 -- 172.21.15.33:0/1332068913 shutdown_connections 2024-03-19T02:44:45.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:45.876+0000 7f6da0790700 1 -- 172.21.15.33:0/1332068913 wait complete. 2024-03-19T02:44:46.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:45 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:46.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:45 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:46.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:45 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:46.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:45 smithi155 ceph-mon[35804]: Reconfiguring mon.b (monmap changed)... 2024-03-19T02:44:46.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:45 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T02:44:46.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:45 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T02:44:46.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:45 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:46.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:45 smithi155 ceph-mon[35804]: Reconfiguring daemon mon.b on smithi096 2024-03-19T02:44:46.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:45 smithi155 ceph-mon[35804]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:46.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:45 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:46.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:45 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:46.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:45 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:46.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:45 smithi033 ceph-mon[31973]: Reconfiguring mon.b (monmap changed)... 2024-03-19T02:44:46.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:45 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T02:44:46.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:45 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T02:44:46.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:45 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:46.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:45 smithi033 ceph-mon[31973]: Reconfiguring daemon mon.b on smithi096 2024-03-19T02:44:46.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:45 smithi033 ceph-mon[31973]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:46.478 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-19T02:44:46.478 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-19T02:44:46.478 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/ceph.conf 2024-03-19T02:44:46.509 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-19T02:44:46.509 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:44:46.577 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-19T02:44:46.577 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/ceph/ceph.conf 2024-03-19T02:44:46.607 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-19T02:44:46.607 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:44:46.674 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-19T02:44:46.674 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/ceph/ceph.conf 2024-03-19T02:44:46.704 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-19T02:44:46.704 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:44:46.773 INFO:tasks.cephadm:Adding mgr.x on smithi033 2024-03-19T02:44:46.773 INFO:tasks.cephadm:Adding mgr.y on smithi096 2024-03-19T02:44:46.773 INFO:tasks.cephadm:Adding mgr.z on smithi155 2024-03-19T02:44:46.773 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph orch apply mgr '3;smithi033=x;smithi096=y;smithi155=z' 2024-03-19T02:44:46.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:46 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1332068913' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:46.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:46 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:46.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:46 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:46.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:46 smithi096 ceph-mon[35946]: Reconfiguring mon.c (monmap changed)... 2024-03-19T02:44:46.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:46 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T02:44:46.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:46 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T02:44:46.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:46 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:46.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:46 smithi096 ceph-mon[35946]: Reconfiguring daemon mon.c on smithi155 2024-03-19T02:44:47.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:46 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1332068913' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:47.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:46 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:47.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:46 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:47.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:46 smithi155 ceph-mon[35804]: Reconfiguring mon.c (monmap changed)... 2024-03-19T02:44:47.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:46 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T02:44:47.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:46 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T02:44:47.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:46 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:47.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:46 smithi155 ceph-mon[35804]: Reconfiguring daemon mon.c on smithi155 2024-03-19T02:44:47.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:46 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1332068913' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:47.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:46 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:47.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:46 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:47.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:46 smithi033 ceph-mon[31973]: Reconfiguring mon.c (monmap changed)... 2024-03-19T02:44:47.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:46 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T02:44:47.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:46 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T02:44:47.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:46 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:47.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:46 smithi033 ceph-mon[31973]: Reconfiguring daemon mon.c on smithi155 2024-03-19T02:44:47.584 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T02:44:48.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:47 smithi033 ceph-mon[31973]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:48.137 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:47 smithi155 ceph-mon[35804]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:48.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:47 smithi096 ceph-mon[35946]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:49.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:49 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:49.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:49 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:49.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:49 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:44:49.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:49 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:49.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:49 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:44:49.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:49 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:49.324 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:49 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:49.325 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:49 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:49.325 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:49 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:44:49.325 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:49 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:49.325 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:49 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:44:49.325 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:49 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:49.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:49 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:49.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:49 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:49.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:49 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:44:49.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:49 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:49.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:49 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:44:49.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:49 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:49.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.508+0000 7f98d96a0700 1 -- 172.21.15.155:0/159920816 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98d4103540 msgr2=0x7f98d41039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:49.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.508+0000 7f98d96a0700 1 --2- 172.21.15.155:0/159920816 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98d4103540 0x7f98d41039a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f98c4009990 tx=0x7f98c4009ca0 comp rx=0 tx=0).stop 2024-03-19T02:44:49.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.508+0000 7f98d96a0700 1 -- 172.21.15.155:0/159920816 shutdown_connections 2024-03-19T02:44:49.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.508+0000 7f98d96a0700 1 --2- 172.21.15.155:0/159920816 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98d4103ee0 0x7f98d410a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:49.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.508+0000 7f98d96a0700 1 --2- 172.21.15.155:0/159920816 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f98d4102340 0x7f98d4102760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:49.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.508+0000 7f98d96a0700 1 --2- 172.21.15.155:0/159920816 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98d4103540 0x7f98d41039a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:49.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.508+0000 7f98d96a0700 1 -- 172.21.15.155:0/159920816 >> 172.21.15.155:0/159920816 conn(0x7f98d40fd920 msgr2=0x7f98d40ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:49.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.508+0000 7f98d96a0700 1 -- 172.21.15.155:0/159920816 shutdown_connections 2024-03-19T02:44:49.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.508+0000 7f98d96a0700 1 -- 172.21.15.155:0/159920816 wait complete. 2024-03-19T02:44:49.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.508+0000 7f98d96a0700 1 Processor -- start 2024-03-19T02:44:49.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.508+0000 7f98d96a0700 1 -- start start 2024-03-19T02:44:49.512 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.509+0000 7f98d96a0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98d4102340 0x7f98d41984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:49.512 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.509+0000 7f98d96a0700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f98d4103540 0x7f98d4198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:49.512 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.509+0000 7f98d96a0700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98d4103ee0 0x7f98d419fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:49.512 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.509+0000 7f98d96a0700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f98d410de30 con 0x7f98d4103540 2024-03-19T02:44:49.512 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.509+0000 7f98d96a0700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f98d410dcb0 con 0x7f98d4102340 2024-03-19T02:44:49.512 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.509+0000 7f98d96a0700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f98d410dfb0 con 0x7f98d4103ee0 2024-03-19T02:44:49.512 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.509+0000 7f98d37fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98d4103ee0 0x7f98d419fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:49.512 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.509+0000 7f98d37fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98d4103ee0 0x7f98d419fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:54114/0 (socket says 172.21.15.155:54114) 2024-03-19T02:44:49.512 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.509+0000 7f98d37fe700 1 -- 172.21.15.155:0/307776075 learned_addr learned my addr 172.21.15.155:0/307776075 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-19T02:44:49.512 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.509+0000 7f98d2ffd700 1 --2- 172.21.15.155:0/307776075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98d4102340 0x7f98d41984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:49.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.509+0000 7f98d27fc700 1 --2- 172.21.15.155:0/307776075 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f98d4103540 0x7f98d4198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:49.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.509+0000 7f98d37fe700 1 -- 172.21.15.155:0/307776075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98d4102340 msgr2=0x7f98d41984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:49.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.509+0000 7f98d37fe700 1 --2- 172.21.15.155:0/307776075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98d4102340 0x7f98d41984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:49.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.509+0000 7f98d37fe700 1 -- 172.21.15.155:0/307776075 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f98d4103540 msgr2=0x7f98d4198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:49.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.509+0000 7f98d37fe700 1 --2- 172.21.15.155:0/307776075 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f98d4103540 0x7f98d4198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:49.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.509+0000 7f98d37fe700 1 -- 172.21.15.155:0/307776075 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98d41a0130 con 0x7f98d4103ee0 2024-03-19T02:44:49.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.509+0000 7f98d27fc700 1 --2- 172.21.15.155:0/307776075 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f98d4103540 0x7f98d4198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:44:49.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.509+0000 7f98d37fe700 1 --2- 172.21.15.155:0/307776075 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98d4103ee0 0x7f98d419fa90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f98cc00ef60 tx=0x7f98cc00ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:49.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.509+0000 7f98bbfff700 1 -- 172.21.15.155:0/307776075 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98cc0092e0 con 0x7f98d4103ee0 2024-03-19T02:44:49.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.509+0000 7f98bbfff700 1 -- 172.21.15.155:0/307776075 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f98cc004bb0 con 0x7f98d4103ee0 2024-03-19T02:44:49.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.509+0000 7f98bbfff700 1 -- 172.21.15.155:0/307776075 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98cc004280 con 0x7f98d4103ee0 2024-03-19T02:44:49.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.509+0000 7f98d96a0700 1 -- 172.21.15.155:0/307776075 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98d41a03c0 con 0x7f98d4103ee0 2024-03-19T02:44:49.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.509+0000 7f98d96a0700 1 -- 172.21.15.155:0/307776075 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98d41a0950 con 0x7f98d4103ee0 2024-03-19T02:44:49.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.510+0000 7f98bbfff700 1 -- 172.21.15.155:0/307776075 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f98cc01b030 con 0x7f98d4103ee0 2024-03-19T02:44:49.514 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.510+0000 7f98d96a0700 1 -- 172.21.15.155:0/307776075 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98d4068fd0 con 0x7f98d4103ee0 2024-03-19T02:44:49.514 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.510+0000 7f98bbfff700 1 --2- 172.21.15.155:0/307776075 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f98bc039200 0x7f98bc03b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:49.514 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.511+0000 7f98bbfff700 1 -- 172.21.15.155:0/307776075 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f98cc04bda0 con 0x7f98d4103ee0 2024-03-19T02:44:49.514 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.511+0000 7f98d2ffd700 1 --2- 172.21.15.155:0/307776075 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f98bc039200 0x7f98bc03b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:49.514 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.511+0000 7f98d2ffd700 1 --2- 172.21.15.155:0/307776075 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f98bc039200 0x7f98bc03b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f98d41033a0 tx=0x7f98c8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:49.516 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.516+0000 7f98bbfff700 1 -- 172.21.15.155:0/307776075 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f98cc022020 con 0x7f98d4103ee0 2024-03-19T02:44:49.721 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.721+0000 7f98d96a0700 1 -- 172.21.15.155:0/307776075 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi033=x;smithi096=y;smithi155=z", "target": ["mon-mgr", ""]}) -- 0x7f98d4062810 con 0x7f98bc039200 2024-03-19T02:44:49.727 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.727+0000 7f98bbfff700 1 -- 172.21.15.155:0/307776075 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f98d4062810 con 0x7f98bc039200 2024-03-19T02:44:49.728 INFO:teuthology.orchestra.run.smithi155.stdout:Scheduled mgr update... 2024-03-19T02:44:49.730 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.729+0000 7f98d96a0700 1 -- 172.21.15.155:0/307776075 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f98bc039200 msgr2=0x7f98bc03b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:49.730 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.729+0000 7f98d96a0700 1 --2- 172.21.15.155:0/307776075 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f98bc039200 0x7f98bc03b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f98d41033a0 tx=0x7f98c8009260 comp rx=0 tx=0).stop 2024-03-19T02:44:49.730 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.729+0000 7f98d96a0700 1 -- 172.21.15.155:0/307776075 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98d4103ee0 msgr2=0x7f98d419fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:49.730 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.729+0000 7f98d96a0700 1 --2- 172.21.15.155:0/307776075 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98d4103ee0 0x7f98d419fa90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f98cc00ef60 tx=0x7f98cc00ef90 comp rx=0 tx=0).stop 2024-03-19T02:44:49.730 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.729+0000 7f98d96a0700 1 -- 172.21.15.155:0/307776075 shutdown_connections 2024-03-19T02:44:49.730 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.729+0000 7f98d96a0700 1 --2- 172.21.15.155:0/307776075 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f98bc039200 0x7f98bc03b6c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:49.730 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.729+0000 7f98d96a0700 1 --2- 172.21.15.155:0/307776075 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98d4103ee0 0x7f98d419fa90 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:49.731 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.729+0000 7f98d96a0700 1 --2- 172.21.15.155:0/307776075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f98d4102340 0x7f98d41984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:49.731 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.729+0000 7f98d96a0700 1 --2- 172.21.15.155:0/307776075 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f98d4103540 0x7f98d4198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:49.731 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.729+0000 7f98d96a0700 1 -- 172.21.15.155:0/307776075 >> 172.21.15.155:0/307776075 conn(0x7f98d40fd920 msgr2=0x7f98d41068e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:49.731 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.729+0000 7f98d96a0700 1 -- 172.21.15.155:0/307776075 shutdown_connections 2024-03-19T02:44:49.731 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:44:49.729+0000 7f98d96a0700 1 -- 172.21.15.155:0/307776075 wait complete. 2024-03-19T02:44:50.336 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:50 smithi155 ceph-mon[35804]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:50.337 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:50 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:50.337 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:50 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:44:50.337 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:50 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:50.337 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:50 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:44:50.337 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:50 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:50.337 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:50 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T02:44:50.337 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:50 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-19T02:44:50.337 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:50 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T02:44:50.337 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:50 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:50.364 DEBUG:teuthology.orchestra.run.smithi096:mgr.y> sudo journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mgr.y.service 2024-03-19T02:44:50.367 DEBUG:teuthology.orchestra.run.smithi155:mgr.z> sudo journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mgr.z.service 2024-03-19T02:44:50.369 INFO:tasks.cephadm:Deploying OSDs... 2024-03-19T02:44:50.369 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-19T02:44:50.369 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/scratch_devs of=/dev/stdout 2024-03-19T02:44:50.388 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-19T02:44:50.388 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_1 2024-03-19T02:44:50.392 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:50 smithi096 ceph-mon[35946]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:50.392 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:50 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:50.392 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:50 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:44:50.392 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:50 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:50.392 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:50 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:44:50.392 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:50 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:50.393 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:50 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T02:44:50.393 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:50 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-19T02:44:50.393 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:50 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T02:44:50.393 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:50 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:50.445 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-19T02:44:50.446 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T02:44:50.446 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-19T02:44:50.446 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:44:50.446 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T02:44:50.446 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-03-19 02:43:52.532036513 +0000 2024-03-19T02:44:50.446 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-03-19 02:30:00.417176692 +0000 2024-03-19T02:44:50.446 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-03-19 02:30:00.417176692 +0000 2024-03-19T02:44:50.446 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-03-19 02:30:00.417176692 +0000 2024-03-19T02:44:50.447 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-19T02:44:50.512 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:50 smithi033 ceph-mon[31973]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:50.512 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:50 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:50.512 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:50 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:44:50.512 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:50 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:50.512 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:50 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:44:50.512 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:50 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:50.512 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:50 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T02:44:50.513 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:50 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-19T02:44:50.513 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:50 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T02:44:50.513 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:50 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:50.515 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-03-19T02:44:50.515 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-03-19T02:44:50.516 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000238756 s, 2.1 MB/s 2024-03-19T02:44:50.517 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-19T02:44:50.574 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_2 2024-03-19T02:44:50.630 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-19T02:44:50.630 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T02:44:50.631 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-19T02:44:50.631 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:44:50.631 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T02:44:50.631 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-03-19 02:43:52.562036005 +0000 2024-03-19T02:44:50.631 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-03-19 02:30:00.768170936 +0000 2024-03-19T02:44:50.631 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-03-19 02:30:00.768170936 +0000 2024-03-19T02:44:50.631 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-03-19 02:30:00.768170936 +0000 2024-03-19T02:44:50.631 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-19T02:44:50.698 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-03-19T02:44:50.698 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-03-19T02:44:50.698 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.00020309 s, 2.5 MB/s 2024-03-19T02:44:50.699 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-19T02:44:50.756 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_3 2024-03-19T02:44:50.812 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-19T02:44:50.812 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T02:44:50.812 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-19T02:44:50.812 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:44:50.812 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T02:44:50.812 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-03-19 02:43:52.591035514 +0000 2024-03-19T02:44:50.812 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-03-19 02:30:01.120165164 +0000 2024-03-19T02:44:50.812 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-03-19 02:30:01.120165164 +0000 2024-03-19T02:44:50.812 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-03-19 02:30:01.120165164 +0000 2024-03-19T02:44:50.812 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-19T02:44:50.878 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-03-19T02:44:50.878 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-03-19T02:44:50.878 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000212503 s, 2.4 MB/s 2024-03-19T02:44:50.879 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-19T02:44:50.935 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_4 2024-03-19T02:44:50.990 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-19T02:44:50.990 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T02:44:50.990 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-19T02:44:50.990 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:44:50.990 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T02:44:50.991 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-03-19 02:43:52.620035023 +0000 2024-03-19T02:44:50.991 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-03-19 02:30:01.458159622 +0000 2024-03-19T02:44:50.991 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-03-19 02:30:01.458159622 +0000 2024-03-19T02:44:50.991 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-03-19 02:30:01.458159622 +0000 2024-03-19T02:44:50.991 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-19T02:44:51.056 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-03-19T02:44:51.056 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-03-19T02:44:51.056 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000223493 s, 2.3 MB/s 2024-03-19T02:44:51.057 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-19T02:44:51.114 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-03-19T02:44:51.114 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/scratch_devs of=/dev/stdout 2024-03-19T02:44:51.133 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-19T02:44:51.133 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_1 2024-03-19T02:44:51.191 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-19T02:44:51.191 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T02:44:51.191 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-19T02:44:51.191 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:44:51.191 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T02:44:51.191 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-03-19 02:44:35.080058146 +0000 2024-03-19T02:44:51.191 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-03-19 02:32:07.170293983 +0000 2024-03-19T02:44:51.191 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-03-19 02:32:07.170293983 +0000 2024-03-19T02:44:51.191 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-03-19 02:32:07.170293983 +0000 2024-03-19T02:44:51.192 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-19T02:44:51.261 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-03-19T02:44:51.261 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-03-19T02:44:51.261 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000287892 s, 1.8 MB/s 2024-03-19T02:44:51.262 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-19T02:44:51.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:51 smithi033 ceph-mon[31973]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi033=x;smithi096=y;smithi155=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:44:51.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:51 smithi033 ceph-mon[31973]: Saving service mgr spec with placement smithi033=x;smithi096=y;smithi155=z;count:3 2024-03-19T02:44:51.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:51 smithi033 ceph-mon[31973]: Deploying daemon mgr.z on smithi155 2024-03-19T02:44:51.320 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_2 2024-03-19T02:44:51.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:51 smithi155 ceph-mon[35804]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi033=x;smithi096=y;smithi155=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:44:51.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:51 smithi155 ceph-mon[35804]: Saving service mgr spec with placement smithi033=x;smithi096=y;smithi155=z;count:3 2024-03-19T02:44:51.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:51 smithi155 ceph-mon[35804]: Deploying daemon mgr.z on smithi155 2024-03-19T02:44:51.377 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-19T02:44:51.377 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T02:44:51.377 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-19T02:44:51.377 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:44:51.377 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T02:44:51.378 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-03-19 02:44:35.080058146 +0000 2024-03-19T02:44:51.378 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-03-19 02:32:07.505285839 +0000 2024-03-19T02:44:51.378 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-03-19 02:32:07.505285839 +0000 2024-03-19T02:44:51.378 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-03-19 02:32:07.505285839 +0000 2024-03-19T02:44:51.378 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-19T02:44:51.444 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:51 smithi096 ceph-mon[35946]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi033=x;smithi096=y;smithi155=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:44:51.444 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:51 smithi096 ceph-mon[35946]: Saving service mgr spec with placement smithi033=x;smithi096=y;smithi155=z;count:3 2024-03-19T02:44:51.444 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:51 smithi096 ceph-mon[35946]: Deploying daemon mgr.z on smithi155 2024-03-19T02:44:51.446 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-03-19T02:44:51.446 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-03-19T02:44:51.446 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000204494 s, 2.5 MB/s 2024-03-19T02:44:51.448 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-19T02:44:51.505 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_3 2024-03-19T02:44:51.560 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-19T02:44:51.560 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T02:44:51.560 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-19T02:44:51.560 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:44:51.560 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T02:44:51.560 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-03-19 02:44:35.080058146 +0000 2024-03-19T02:44:51.561 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-03-19 02:32:07.857277282 +0000 2024-03-19T02:44:51.561 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-03-19 02:32:07.857277282 +0000 2024-03-19T02:44:51.561 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-03-19 02:32:07.857277282 +0000 2024-03-19T02:44:51.561 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-19T02:44:51.626 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-03-19T02:44:51.626 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-03-19T02:44:51.626 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000191327 s, 2.7 MB/s 2024-03-19T02:44:51.627 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-19T02:44:51.683 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_4 2024-03-19T02:44:51.738 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-19T02:44:51.738 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T02:44:51.738 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-19T02:44:51.738 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:44:51.738 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T02:44:51.739 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-03-19 02:44:35.080058146 +0000 2024-03-19T02:44:51.739 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-03-19 02:32:08.189269210 +0000 2024-03-19T02:44:51.739 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-03-19 02:32:08.189269210 +0000 2024-03-19T02:44:51.739 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-03-19 02:32:08.189269210 +0000 2024-03-19T02:44:51.739 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-19T02:44:51.804 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-03-19T02:44:51.804 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-03-19T02:44:51.804 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000186556 s, 2.7 MB/s 2024-03-19T02:44:51.805 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-19T02:44:51.861 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-03-19T02:44:51.861 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/scratch_devs of=/dev/stdout 2024-03-19T02:44:51.878 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-19T02:44:51.878 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_1 2024-03-19T02:44:51.934 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-19T02:44:51.934 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T02:44:51.935 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-19T02:44:51.935 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:44:51.935 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T02:44:51.935 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-03-19 02:44:51.753967222 +0000 2024-03-19T02:44:51.935 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-03-19 02:27:46.142782355 +0000 2024-03-19T02:44:51.935 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-03-19 02:27:46.142782355 +0000 2024-03-19T02:44:51.935 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-03-19 02:27:46.142782355 +0000 2024-03-19T02:44:51.935 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-19T02:44:52.000 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:51 smithi155 systemd[1]: Starting Ceph mgr.z for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T02:44:52.003 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-03-19T02:44:52.003 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-03-19T02:44:52.003 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000284855 s, 1.8 MB/s 2024-03-19T02:44:52.005 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-19T02:44:52.061 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_2 2024-03-19T02:44:52.116 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-19T02:44:52.116 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T02:44:52.116 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-19T02:44:52.116 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:44:52.116 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T02:44:52.116 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-03-19 02:44:51.754967202 +0000 2024-03-19T02:44:52.116 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-03-19 02:27:46.483775355 +0000 2024-03-19T02:44:52.116 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-03-19 02:27:46.483775355 +0000 2024-03-19T02:44:52.116 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-03-19 02:27:46.483775355 +0000 2024-03-19T02:44:52.117 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-19T02:44:52.211 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-03-19T02:44:52.211 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-03-19T02:44:52.211 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000192579 s, 2.7 MB/s 2024-03-19T02:44:52.212 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-19T02:44:52.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:52 smithi155 ceph-mon[35804]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:52.266 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:52 smithi155 podman[37586]: 2024-03-19 02:44:52.019780777 +0000 UTC m=+0.103053984 container create eabf96d16ca6f70d8350d501a4d2e02b1c2c309d16113004bddaa6671132566a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True) 2024-03-19T02:44:52.266 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:52 smithi155 podman[37586]: 2024-03-19 02:44:51.934837429 +0000 UTC m=+0.018110644 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:44:52.266 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:52 smithi155 podman[37586]: 2024-03-19 02:44:52.239267648 +0000 UTC m=+0.322540864 container init eabf96d16ca6f70d8350d501a4d2e02b1c2c309d16113004bddaa6671132566a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-19T02:44:52.266 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:52 smithi155 podman[37586]: 2024-03-19 02:44:52.241995463 +0000 UTC m=+0.325268671 container start eabf96d16ca6f70d8350d501a4d2e02b1c2c309d16113004bddaa6671132566a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-19T02:44:52.269 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_3 2024-03-19T02:44:52.324 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-19T02:44:52.324 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T02:44:52.324 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-19T02:44:52.324 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:44:52.324 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T02:44:52.324 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-03-19 02:44:51.754967202 +0000 2024-03-19T02:44:52.324 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-03-19 02:27:46.819768464 +0000 2024-03-19T02:44:52.324 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-03-19 02:27:46.819768464 +0000 2024-03-19T02:44:52.324 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-03-19 02:27:46.819768464 +0000 2024-03-19T02:44:52.325 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-19T02:44:52.392 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-03-19T02:44:52.392 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-03-19T02:44:52.393 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000230805 s, 2.2 MB/s 2024-03-19T02:44:52.394 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-19T02:44:52.449 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_4 2024-03-19T02:44:52.464 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:52 smithi096 ceph-mon[35946]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:52.504 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-19T02:44:52.504 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T02:44:52.505 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-19T02:44:52.505 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:44:52.505 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T02:44:52.505 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-03-19 02:44:51.755967181 +0000 2024-03-19T02:44:52.505 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-03-19 02:27:47.165761367 +0000 2024-03-19T02:44:52.505 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-03-19 02:27:47.165761367 +0000 2024-03-19T02:44:52.505 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-03-19 02:27:47.165761367 +0000 2024-03-19T02:44:52.505 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-19T02:44:52.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:52 smithi033 ceph-mon[31973]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:52.569 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:52 smithi155 ceph-mgr[37692]: -- 172.21.15.155:0/2282072565 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5567f6c765a0 con 0x5567f6c4a800 2024-03-19T02:44:52.569 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:52 smithi155 bash[37586]: eabf96d16ca6f70d8350d501a4d2e02b1c2c309d16113004bddaa6671132566a 2024-03-19T02:44:52.569 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:52 smithi155 systemd[1]: Started Ceph mgr.z for 2b7a8938-e59a-11ee-95c9-87774f69a715. 2024-03-19T02:44:52.571 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-03-19T02:44:52.571 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-03-19T02:44:52.571 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000218673 s, 2.3 MB/s 2024-03-19T02:44:52.572 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-19T02:44:52.628 INFO:tasks.cephadm:Deploying osd.0 on smithi033 with /dev/vg_nvme/lv_4... 2024-03-19T02:44:52.628 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-19T02:44:52.874 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:44:52.905 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:52 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:44:52.810+0000 7f0d4c87b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T02:44:52.905 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:52 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:44:52.904+0000 7f0d4c87b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T02:44:53.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:53 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:53.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:53 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:53.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:53 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:53.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:53 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T02:44:53.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:53 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-19T02:44:53.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:53 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T02:44:53.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:53 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:53.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:53 smithi033 ceph-mon[31973]: Deploying daemon mgr.y on smithi096 2024-03-19T02:44:53.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:53 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:53.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:53 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:53.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:53 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:53.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:53 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T02:44:53.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:53 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-19T02:44:53.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:53 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T02:44:53.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:53 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:53.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:53 smithi096 ceph-mon[35946]: Deploying daemon mgr.y on smithi096 2024-03-19T02:44:53.766 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:53 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:44:53.550+0000 7f0d4c87b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T02:44:53.766 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:53 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:44:53.643+0000 7f0d4c87b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T02:44:53.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:53 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:53.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:53 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:53.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:53 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:53.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:53 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T02:44:53.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:53 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-19T02:44:53.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:53 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T02:44:53.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:53 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:53.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:53 smithi155 ceph-mon[35804]: Deploying daemon mgr.y on smithi096 2024-03-19T02:44:54.231 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:53 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:44:53.787+0000 7f0d4c87b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T02:44:54.231 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:53 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:44:53.881+0000 7f0d4c87b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T02:44:54.231 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:53 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:44:53.964+0000 7f0d4c87b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T02:44:54.231 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:54 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:44:54.229+0000 7f0d4c87b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-19T02:44:54.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:54 smithi155 ceph-mon[35804]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:54.516 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:54 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:44:54.338+0000 7f0d4c87b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T02:44:54.516 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:54 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:44:54.425+0000 7f0d4c87b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T02:44:54.709 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-19T02:44:54.709 INFO:teuthology.orchestra.run.smithi033.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-19T02:44:54.709 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10+0 records in 2024-03-19T02:44:54.709 INFO:teuthology.orchestra.run.smithi033.stderr:10+0 records out 2024-03-19T02:44:54.709 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111424 s, 941 MB/s 2024-03-19T02:44:54.709 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping successful for: 2024-03-19T02:44:54.710 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:54 smithi033 ceph-mon[31973]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:54.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:54 smithi096 ceph-mon[35946]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:54.715 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:44:54 smithi096 podman[37351]: 2024-03-19 02:44:54.642823512 +0000 UTC m=+0.116229771 container create 9e45c9a1db3572c2db31df1181993964559fd9132323770875dd591f45127423 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-19T02:44:55.016 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:54 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:44:54.521+0000 7f0d4c87b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T02:44:55.016 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:54 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:44:54.699+0000 7f0d4c87b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T02:44:55.215 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:44:54 smithi096 podman[37351]: 2024-03-19 02:44:54.887132626 +0000 UTC m=+0.360538884 container init 9e45c9a1db3572c2db31df1181993964559fd9132323770875dd591f45127423 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T02:44:55.215 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:44:54 smithi096 podman[37351]: 2024-03-19 02:44:54.889658527 +0000 UTC m=+0.363064796 container start 9e45c9a1db3572c2db31df1181993964559fd9132323770875dd591f45127423 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-19T02:44:55.215 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:44:54 smithi096 ceph-mgr[37370]: -- 172.21.15.96:0/1804047405 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55817ea9c5a0 con 0x55817ea70c00 2024-03-19T02:44:55.215 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:44:54 smithi096 bash[37351]: 9e45c9a1db3572c2db31df1181993964559fd9132323770875dd591f45127423 2024-03-19T02:44:55.215 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:44:54 smithi096 systemd[1]: Started Ceph mgr.y for 2b7a8938-e59a-11ee-95c9-87774f69a715. 2024-03-19T02:44:55.329 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi033:vg_nvme/lv_4 2024-03-19T02:44:55.566 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:44:55.715 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:44:55 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:44:55.453+0000 7fcf7ba60200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T02:44:55.715 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:44:55 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:44:55.549+0000 7fcf7ba60200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T02:44:56.016 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:55 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:44:55.732+0000 7f0d4c87b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-19T02:44:56.016 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:55 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:44:55.872+0000 7f0d4c87b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T02:44:56.235 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:56 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:56.235 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:56 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:56.235 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:56 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:56.235 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:56 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:56.235 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:56 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:44:56.235 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:56 smithi096 ceph-mon[35946]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:56.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:56 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:56.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:56 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:56.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:56 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:56.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:56 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:56.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:56 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:44:56.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:56 smithi033 ceph-mon[31973]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:56.313 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:56 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:44:56.221+0000 7f0d4c87b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T02:44:56.313 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:56 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:44:56.312+0000 7f0d4c87b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T02:44:56.313 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:56 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:56.313 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:56 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:56.313 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:56 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:56.313 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:56 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:56.313 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:56 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:44:56.313 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:56 smithi155 ceph-mon[35804]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:56.575 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:44:56 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:44:56.234+0000 7fcf7ba60200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T02:44:56.575 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:44:56 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:44:56.327+0000 7fcf7ba60200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T02:44:56.575 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:44:56 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:44:56.478+0000 7fcf7ba60200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T02:44:56.575 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:44:56 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:44:56.574+0000 7fcf7ba60200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T02:44:56.710 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:56 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:44:56.520+0000 7f0d4c87b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:44:56.710 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:56 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:44:56.709+0000 7f0d4c87b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T02:44:56.927 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:44:56 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:44:56.659+0000 7fcf7ba60200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T02:44:56.927 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:44:56 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:44:56.926+0000 7fcf7ba60200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-19T02:44:57.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.129+0000 7f9a8f171700 1 -- 172.21.15.33:0/3583091627 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a88102dd0 msgr2=0x7f9a88109680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:57.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.129+0000 7f9a8f171700 1 --2- 172.21.15.33:0/3583091627 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a88102dd0 0x7f9a88109680 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9a80009fc0 tx=0x7f9a8000b330 comp rx=0 tx=0).stop 2024-03-19T02:44:57.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.129+0000 7f9a8f171700 1 -- 172.21.15.33:0/3583091627 shutdown_connections 2024-03-19T02:44:57.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.129+0000 7f9a8f171700 1 --2- 172.21.15.33:0/3583091627 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a88102dd0 0x7f9a88109680 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:57.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.129+0000 7f9a8f171700 1 --2- 172.21.15.33:0/3583091627 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a88101230 0x7f9a88101650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:57.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.129+0000 7f9a8f171700 1 --2- 172.21.15.33:0/3583091627 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9a88102430 0x7f9a88102890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:57.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.129+0000 7f9a8f171700 1 -- 172.21.15.33:0/3583091627 >> 172.21.15.33:0/3583091627 conn(0x7f9a880fc7d0 msgr2=0x7f9a880fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:44:57.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.129+0000 7f9a8f171700 1 -- 172.21.15.33:0/3583091627 shutdown_connections 2024-03-19T02:44:57.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.129+0000 7f9a8f171700 1 -- 172.21.15.33:0/3583091627 wait complete. 2024-03-19T02:44:57.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.130+0000 7f9a8f171700 1 Processor -- start 2024-03-19T02:44:57.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.130+0000 7f9a8f171700 1 -- start start 2024-03-19T02:44:57.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.130+0000 7f9a8f171700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a88101230 0x7f9a8806b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:57.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.130+0000 7f9a8f171700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9a88102430 0x7f9a8819d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:57.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.130+0000 7f9a8f171700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a88102dd0 0x7f9a8819f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:57.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.130+0000 7f9a8f171700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8810cd50 con 0x7f9a88102dd0 2024-03-19T02:44:57.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.130+0000 7f9a8f171700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8810cbd0 con 0x7f9a88101230 2024-03-19T02:44:57.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.130+0000 7f9a8f171700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8810ced0 con 0x7f9a88102430 2024-03-19T02:44:57.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.131+0000 7f9a8d70e700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a88102dd0 0x7f9a8819f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:57.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.131+0000 7f9a8cf0d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a88101230 0x7f9a8806b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:57.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.131+0000 7f9a8d70e700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a88102dd0 0x7f9a8819f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52338/0 (socket says 172.21.15.33:52338) 2024-03-19T02:44:57.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.131+0000 7f9a8d70e700 1 -- 172.21.15.33:0/491383442 learned_addr learned my addr 172.21.15.33:0/491383442 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:44:57.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.131+0000 7f9a8cf0d700 1 --2- 172.21.15.33:0/491383442 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a88101230 0x7f9a8806b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:47168/0 (socket says 172.21.15.33:47168) 2024-03-19T02:44:57.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.131+0000 7f9a87fff700 1 --2- 172.21.15.33:0/491383442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9a88102430 0x7f9a8819d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:57.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.131+0000 7f9a8d70e700 1 -- 172.21.15.33:0/491383442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9a88102430 msgr2=0x7f9a8819d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:57.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.131+0000 7f9a8d70e700 1 --2- 172.21.15.33:0/491383442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9a88102430 0x7f9a8819d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:57.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.131+0000 7f9a8d70e700 1 -- 172.21.15.33:0/491383442 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a88101230 msgr2=0x7f9a8806b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:44:57.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.131+0000 7f9a8d70e700 1 --2- 172.21.15.33:0/491383442 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a88101230 0x7f9a8806b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:44:57.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.131+0000 7f9a8d70e700 1 -- 172.21.15.33:0/491383442 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a881a00a0 con 0x7f9a88102dd0 2024-03-19T02:44:57.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.131+0000 7f9a8d70e700 1 --2- 172.21.15.33:0/491383442 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a88102dd0 0x7f9a8819f7d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9a80007f20 tx=0x7f9a80009b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:57.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.131+0000 7f9a85ffb700 1 -- 172.21.15.33:0/491383442 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a80021070 con 0x7f9a88102dd0 2024-03-19T02:44:57.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.131+0000 7f9a85ffb700 1 -- 172.21.15.33:0/491383442 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9a80007ac0 con 0x7f9a88102dd0 2024-03-19T02:44:57.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.131+0000 7f9a8f171700 1 -- 172.21.15.33:0/491383442 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a881a0330 con 0x7f9a88102dd0 2024-03-19T02:44:57.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.131+0000 7f9a85ffb700 1 -- 172.21.15.33:0/491383442 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a80005330 con 0x7f9a88102dd0 2024-03-19T02:44:57.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.132+0000 7f9a8f171700 1 -- 172.21.15.33:0/491383442 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a881a0830 con 0x7f9a88102dd0 2024-03-19T02:44:57.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.132+0000 7f9a85ffb700 1 -- 172.21.15.33:0/491383442 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f9a80024070 con 0x7f9a88102dd0 2024-03-19T02:44:57.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.132+0000 7f9a85ffb700 1 --2- 172.21.15.33:0/491383442 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9a78039200 0x7f9a7803b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:44:57.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.133+0000 7f9a85ffb700 1 -- 172.21.15.33:0/491383442 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f9a8000f020 con 0x7f9a88102dd0 2024-03-19T02:44:57.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.133+0000 7f9a8cf0d700 1 --2- 172.21.15.33:0/491383442 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9a78039200 0x7f9a7803b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:44:57.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.133+0000 7f9a8f171700 1 -- 172.21.15.33:0/491383442 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a88068fd0 con 0x7f9a88102dd0 2024-03-19T02:44:57.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.133+0000 7f9a8cf0d700 1 --2- 172.21.15.33:0/491383442 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9a78039200 0x7f9a7803b6c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9a88102290 tx=0x7f9a7c009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:44:57.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.142+0000 7f9a85ffb700 1 -- 172.21.15.33:0/491383442 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9a8002b020 con 0x7f9a88102dd0 2024-03-19T02:44:57.215 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:44:57 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:44:57.036+0000 7fcf7ba60200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T02:44:57.215 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:44:57 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:44:57.122+0000 7fcf7ba60200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T02:44:57.256 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:56 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:44:56.992+0000 7f0d4c87b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:44:57.256 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:57 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:44:57.255+0000 7f0d4c87b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T02:44:57.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:57.371+0000 7f9a8f171700 1 -- 172.21.15.33:0/491383442 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f9a88062750 con 0x7f9a78039200 2024-03-19T02:44:57.516 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:57 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:44:57.493+0000 7f0d4c87b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T02:44:57.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:57 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:57.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:57 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:57.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:57 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:57.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:57 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:57.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:57 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:57.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:57 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:44:57.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:57 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:57.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:57 smithi155 ceph-mon[35804]: Reconfiguring mgr.x (unknown last config time)... 2024-03-19T02:44:57.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:57 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T02:44:57.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:57 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T02:44:57.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:57 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:57.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:57 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:44:57.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:57 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:44:57.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:57 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:57.715 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:44:57 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:44:57.216+0000 7fcf7ba60200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T02:44:57.715 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:44:57 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:44:57.390+0000 7fcf7ba60200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T02:44:57.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:57 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:57.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:57 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:57.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:57 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:57.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:57 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:57.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:57 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:57.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:57 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:44:57.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:57 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:57.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:57 smithi096 ceph-mon[35946]: Reconfiguring mgr.x (unknown last config time)... 2024-03-19T02:44:57.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:57 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T02:44:57.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:57 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T02:44:57.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:57 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:57.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:57 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:44:57.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:57 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:44:57.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:57 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:57.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:57 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:57.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:57 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:57.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:57 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:57.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:57 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:57.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:57 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:57.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:57 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:44:57.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:57 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:57.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:57 smithi033 ceph-mon[31973]: Reconfiguring mgr.x (unknown last config time)... 2024-03-19T02:44:57.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:57 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T02:44:57.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:57 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T02:44:57.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:57 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:57.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:57 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:44:57.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:57 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:44:57.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:57 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:58.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:58 smithi033 ceph-mon[31973]: Reconfiguring daemon mgr.x on smithi033 2024-03-19T02:44:58.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:58 smithi033 ceph-mon[31973]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:44:58.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:58 smithi033 ceph-mon[31973]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:58.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:58 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:58.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:58 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:58.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:58 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:44:58.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:58 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:58.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:58 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:44:58.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:58 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:58.715 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:44:58 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:44:58.343+0000 7fcf7ba60200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-19T02:44:58.715 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:44:58 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:44:58.476+0000 7fcf7ba60200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T02:44:58.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:58 smithi096 ceph-mon[35946]: Reconfiguring daemon mgr.x on smithi033 2024-03-19T02:44:58.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:58 smithi096 ceph-mon[35946]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:44:58.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:58 smithi096 ceph-mon[35946]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:58.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:58 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:58.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:58 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:58.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:58 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:44:58.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:58 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:58.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:58 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:44:58.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:58 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:58.766 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:58 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:44:58.297+0000 7f0d4c87b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T02:44:58.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:58 smithi155 ceph-mon[35804]: Reconfiguring daemon mgr.x on smithi033 2024-03-19T02:44:58.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:58 smithi155 ceph-mon[35804]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:44:58.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:58 smithi155 ceph-mon[35804]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:44:58.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:58 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:58.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:58 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:58.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:58 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:44:58.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:58 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:44:58.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:58 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:44:58.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:58 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:44:59.089 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:44:58 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:44:58.805+0000 7fcf7ba60200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T02:44:59.089 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:44:58 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:44:58.893+0000 7fcf7ba60200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T02:44:59.089 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:44:59 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:44:59.088+0000 7fcf7ba60200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:44:59.188 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:58 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:44:58.876+0000 7f0d4c87b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T02:44:59.188 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:58 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:44:58.972+0000 7f0d4c87b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T02:44:59.188 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:44:59 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:44:59.187+0000 7f0d4c87b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T02:44:59.454 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:44:59 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:44:59.273+0000 7fcf7ba60200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T02:44:59.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:44:59.461+0000 7f9a85ffb700 1 -- 172.21.15.33:0/491383442 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mgrmap(e 14) ==== 91698+0+0 (secure 0 0 0) 0x7f9a80026450 con 0x7f9a88102dd0 2024-03-19T02:44:59.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:59 smithi033 ceph-mon[31973]: Standby manager daemon z started 2024-03-19T02:44:59.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:59 smithi033 ceph-mon[31973]: from='mgr.? 172.21.15.155:0/3135398398' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-19T02:44:59.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:59 smithi033 ceph-mon[31973]: from='mgr.? 172.21.15.155:0/3135398398' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T02:44:59.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:59 smithi033 ceph-mon[31973]: from='mgr.? 172.21.15.155:0/3135398398' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-19T02:44:59.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:44:59 smithi033 ceph-mon[31973]: from='mgr.? 172.21.15.155:0/3135398398' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T02:44:59.715 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:44:59 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:44:59.558+0000 7fcf7ba60200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:44:59.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:59 smithi096 ceph-mon[35946]: Standby manager daemon z started 2024-03-19T02:44:59.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:59 smithi096 ceph-mon[35946]: from='mgr.? 172.21.15.155:0/3135398398' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-19T02:44:59.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:59 smithi096 ceph-mon[35946]: from='mgr.? 172.21.15.155:0/3135398398' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T02:44:59.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:59 smithi096 ceph-mon[35946]: from='mgr.? 172.21.15.155:0/3135398398' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-19T02:44:59.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:44:59 smithi096 ceph-mon[35946]: from='mgr.? 172.21.15.155:0/3135398398' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T02:44:59.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:59 smithi155 ceph-mon[35804]: Standby manager daemon z started 2024-03-19T02:44:59.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:59 smithi155 ceph-mon[35804]: from='mgr.? 172.21.15.155:0/3135398398' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-19T02:44:59.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:59 smithi155 ceph-mon[35804]: from='mgr.? 172.21.15.155:0/3135398398' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T02:44:59.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:59 smithi155 ceph-mon[35804]: from='mgr.? 172.21.15.155:0/3135398398' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-19T02:44:59.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:44:59 smithi155 ceph-mon[35804]: from='mgr.? 172.21.15.155:0/3135398398' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T02:45:00.215 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:44:59 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:44:59.813+0000 7fcf7ba60200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T02:45:00.215 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:45:00 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:45:00.046+0000 7fcf7ba60200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T02:45:00.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:00 smithi096 ceph-mon[35946]: mgrmap e14: x(active, since 79s), standbys: z 2024-03-19T02:45:00.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:00 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-19T02:45:00.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:00 smithi096 ceph-mon[35946]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:00.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:00 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:00.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:00 smithi155 ceph-mon[35804]: mgrmap e14: x(active, since 79s), standbys: z 2024-03-19T02:45:00.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:00 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-19T02:45:00.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:00 smithi155 ceph-mon[35804]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:00.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:00 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:00.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:00 smithi033 ceph-mon[31973]: mgrmap e14: x(active, since 79s), standbys: z 2024-03-19T02:45:00.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:00 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-19T02:45:00.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:00 smithi033 ceph-mon[31973]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:00.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:00 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:01.215 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:45:00 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:45:00.839+0000 7fcf7ba60200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T02:45:01.709 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:01 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/609021964' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "090a89ab-6b7b-4317-8823-178d4212b2f4"}]: dispatch 2024-03-19T02:45:01.709 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:01 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/609021964' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "090a89ab-6b7b-4317-8823-178d4212b2f4"}]': finished 2024-03-19T02:45:01.709 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:01 smithi096 ceph-mon[35946]: osdmap e5: 1 total, 0 up, 1 in 2024-03-19T02:45:01.709 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:01 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:45:01.709 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:01 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2110184434' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:45:01.710 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:45:01 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:45:01.397+0000 7fcf7ba60200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T02:45:01.710 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:45:01 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:45:01.495+0000 7fcf7ba60200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T02:45:01.710 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:45:01 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:45:01.708+0000 7fcf7ba60200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T02:45:01.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:01 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/609021964' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "090a89ab-6b7b-4317-8823-178d4212b2f4"}]: dispatch 2024-03-19T02:45:01.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:01 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/609021964' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "090a89ab-6b7b-4317-8823-178d4212b2f4"}]': finished 2024-03-19T02:45:01.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:01 smithi155 ceph-mon[35804]: osdmap e5: 1 total, 0 up, 1 in 2024-03-19T02:45:01.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:01 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:45:01.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:01 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2110184434' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:45:01.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:01 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/609021964' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "090a89ab-6b7b-4317-8823-178d4212b2f4"}]: dispatch 2024-03-19T02:45:01.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:01 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/609021964' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "090a89ab-6b7b-4317-8823-178d4212b2f4"}]': finished 2024-03-19T02:45:01.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:01 smithi033 ceph-mon[31973]: osdmap e5: 1 total, 0 up, 1 in 2024-03-19T02:45:01.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:01 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:45:01.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:01 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2110184434' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:45:02.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:45:02.476+0000 7f9a85ffb700 1 -- 172.21.15.33:0/491383442 <== mon.0 v2:172.21.15.33:3300/0 8 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f9a80005850 con 0x7f9a88102dd0 2024-03-19T02:45:02.744 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:02 smithi096 ceph-mon[35946]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:02.744 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:02 smithi096 ceph-mon[35946]: Standby manager daemon y started 2024-03-19T02:45:02.744 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:02 smithi096 ceph-mon[35946]: from='mgr.? 172.21.15.96:0/1905396549' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-19T02:45:02.744 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:02 smithi096 ceph-mon[35946]: from='mgr.? 172.21.15.96:0/1905396549' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T02:45:02.744 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:02 smithi096 ceph-mon[35946]: from='mgr.? 172.21.15.96:0/1905396549' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-19T02:45:02.744 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:02 smithi096 ceph-mon[35946]: from='mgr.? 172.21.15.96:0/1905396549' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T02:45:02.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:02 smithi155 ceph-mon[35804]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:02.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:02 smithi155 ceph-mon[35804]: Standby manager daemon y started 2024-03-19T02:45:02.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:02 smithi155 ceph-mon[35804]: from='mgr.? 172.21.15.96:0/1905396549' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-19T02:45:02.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:02 smithi155 ceph-mon[35804]: from='mgr.? 172.21.15.96:0/1905396549' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T02:45:02.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:02 smithi155 ceph-mon[35804]: from='mgr.? 172.21.15.96:0/1905396549' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-19T02:45:02.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:02 smithi155 ceph-mon[35804]: from='mgr.? 172.21.15.96:0/1905396549' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T02:45:02.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:02 smithi033 ceph-mon[31973]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:02.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:02 smithi033 ceph-mon[31973]: Standby manager daemon y started 2024-03-19T02:45:02.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:02 smithi033 ceph-mon[31973]: from='mgr.? 172.21.15.96:0/1905396549' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-19T02:45:02.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:02 smithi033 ceph-mon[31973]: from='mgr.? 172.21.15.96:0/1905396549' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T02:45:02.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:02 smithi033 ceph-mon[31973]: from='mgr.? 172.21.15.96:0/1905396549' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-19T02:45:02.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:02 smithi033 ceph-mon[31973]: from='mgr.? 172.21.15.96:0/1905396549' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T02:45:03.722 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:03 smithi155 ceph-mon[35804]: mgrmap e15: x(active, since 82s), standbys: z, y 2024-03-19T02:45:03.723 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:03 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-19T02:45:03.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:03 smithi033 ceph-mon[31973]: mgrmap e15: x(active, since 82s), standbys: z, y 2024-03-19T02:45:03.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:03 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-19T02:45:03.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:03 smithi096 ceph-mon[35946]: mgrmap e15: x(active, since 82s), standbys: z, y 2024-03-19T02:45:03.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:03 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-19T02:45:04.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:04 smithi155 ceph-mon[35804]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:04.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:04 smithi033 ceph-mon[31973]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:04.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:04 smithi096 ceph-mon[35946]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:06.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:05 smithi155 ceph-mon[35804]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:06.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:05 smithi033 ceph-mon[31973]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:06.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:05 smithi096 ceph-mon[35946]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:08.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:07 smithi155 ceph-mon[35804]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:08.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:07 smithi033 ceph-mon[31973]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:08.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:07 smithi096 ceph-mon[35946]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:08.953 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:08 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-19T02:45:08.954 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:08 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:45:08.954 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:08 smithi033 ceph-mon[31973]: Deploying daemon osd.0 on smithi033 2024-03-19T02:45:09.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:08 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-19T02:45:09.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:08 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:45:09.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:08 smithi155 ceph-mon[35804]: Deploying daemon osd.0 on smithi033 2024-03-19T02:45:09.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:08 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-19T02:45:09.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:08 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:45:09.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:08 smithi096 ceph-mon[35946]: Deploying daemon osd.0 on smithi033 2024-03-19T02:45:10.012 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:09 smithi033 ceph-mon[31973]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:10.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:09 smithi155 ceph-mon[35804]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:10.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:09 smithi096 ceph-mon[35946]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:11.981 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:11 smithi033 ceph-mon[31973]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:12.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:11 smithi155 ceph-mon[35804]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:12.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:11 smithi096 ceph-mon[35946]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:14.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:14 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:14.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:14 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:14.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:14 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:45:14.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:14 smithi033 ceph-mon[31973]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:14.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:14 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:14.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:14 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:14.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:14 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:45:14.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:14 smithi096 ceph-mon[35946]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:14.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:14 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:14.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:14 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:14.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:14 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:45:14.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:14 smithi155 ceph-mon[35804]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:16.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:15 smithi155 ceph-mon[35804]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:16.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:15 smithi033 ceph-mon[31973]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:16.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:15 smithi096 ceph-mon[35946]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:16.447 INFO:teuthology.orchestra.run.smithi033.stdout:Created osd(s) 0 on host 'smithi033' 2024-03-19T02:45:16.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:45:16.445+0000 7f9a85ffb700 1 -- 172.21.15.33:0/491383442 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9a88062750 con 0x7f9a78039200 2024-03-19T02:45:16.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:45:16.448+0000 7f9a8f171700 1 -- 172.21.15.33:0/491383442 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9a78039200 msgr2=0x7f9a7803b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:16.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:45:16.448+0000 7f9a8f171700 1 --2- 172.21.15.33:0/491383442 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9a78039200 0x7f9a7803b6c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9a88102290 tx=0x7f9a7c009310 comp rx=0 tx=0).stop 2024-03-19T02:45:16.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:45:16.448+0000 7f9a8f171700 1 -- 172.21.15.33:0/491383442 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a88102dd0 msgr2=0x7f9a8819f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:16.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:45:16.448+0000 7f9a8f171700 1 --2- 172.21.15.33:0/491383442 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a88102dd0 0x7f9a8819f7d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9a80007f20 tx=0x7f9a80009b40 comp rx=0 tx=0).stop 2024-03-19T02:45:16.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:45:16.448+0000 7f9a8f171700 1 -- 172.21.15.33:0/491383442 shutdown_connections 2024-03-19T02:45:16.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:45:16.448+0000 7f9a8f171700 1 --2- 172.21.15.33:0/491383442 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9a78039200 0x7f9a7803b6c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:16.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:45:16.448+0000 7f9a8f171700 1 --2- 172.21.15.33:0/491383442 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a88102dd0 0x7f9a8819f7d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:16.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:45:16.448+0000 7f9a8f171700 1 --2- 172.21.15.33:0/491383442 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a88101230 0x7f9a8806b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:16.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:45:16.448+0000 7f9a8f171700 1 --2- 172.21.15.33:0/491383442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9a88102430 0x7f9a8819d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:16.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:45:16.448+0000 7f9a8f171700 1 -- 172.21.15.33:0/491383442 >> 172.21.15.33:0/491383442 conn(0x7f9a880fc7d0 msgr2=0x7f9a88105730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:16.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:45:16.448+0000 7f9a8f171700 1 -- 172.21.15.33:0/491383442 shutdown_connections 2024-03-19T02:45:16.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:45:16.448+0000 7f9a8f171700 1 -- 172.21.15.33:0/491383442 wait complete. 2024-03-19T02:45:17.016 DEBUG:teuthology.orchestra.run.smithi033:osd.0> sudo journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.0.service 2024-03-19T02:45:17.018 INFO:tasks.cephadm:Deploying osd.1 on smithi096 with /dev/vg_nvme/lv_4... 2024-03-19T02:45:17.018 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-19T02:45:17.262 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T02:45:17.656 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:17 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:17.657 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:17 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:17.657 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:17 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:17.657 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:17 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:17.657 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:17 smithi096 ceph-mon[35946]: from='osd.0 [v2:172.21.15.33:6802/1633124995,v1:172.21.15.33:6803/1633124995]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-19T02:45:17.749 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:17 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:17.749 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:17 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:17.749 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:17 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:17.749 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:17 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:17.750 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:17 smithi033 ceph-mon[31973]: from='osd.0 [v2:172.21.15.33:6802/1633124995,v1:172.21.15.33:6803/1633124995]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-19T02:45:17.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:17 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:17.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:17 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:17.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:17 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:17.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:17 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:17.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:17 smithi155 ceph-mon[35804]: from='osd.0 [v2:172.21.15.33:6802/1633124995,v1:172.21.15.33:6803/1633124995]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-19T02:45:18.561 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:18 smithi033 ceph-mon[31973]: from='osd.0 [v2:172.21.15.33:6802/1633124995,v1:172.21.15.33:6803/1633124995]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-19T02:45:18.561 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:18 smithi033 ceph-mon[31973]: osdmap e6: 1 total, 0 up, 1 in 2024-03-19T02:45:18.561 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:18 smithi033 ceph-mon[31973]: from='osd.0 [v2:172.21.15.33:6802/1633124995,v1:172.21.15.33:6803/1633124995]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-19T02:45:18.673 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:18 smithi096 ceph-mon[35946]: from='osd.0 [v2:172.21.15.33:6802/1633124995,v1:172.21.15.33:6803/1633124995]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-19T02:45:18.674 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:18 smithi096 ceph-mon[35946]: osdmap e6: 1 total, 0 up, 1 in 2024-03-19T02:45:18.674 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:18 smithi096 ceph-mon[35946]: from='osd.0 [v2:172.21.15.33:6802/1633124995,v1:172.21.15.33:6803/1633124995]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-19T02:45:18.674 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:18 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:45:18.674 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:18 smithi096 ceph-mon[35946]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:19.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:18 smithi155 ceph-mon[35804]: from='osd.0 [v2:172.21.15.33:6802/1633124995,v1:172.21.15.33:6803/1633124995]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-19T02:45:19.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:18 smithi155 ceph-mon[35804]: osdmap e6: 1 total, 0 up, 1 in 2024-03-19T02:45:19.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:18 smithi155 ceph-mon[35804]: from='osd.0 [v2:172.21.15.33:6802/1633124995,v1:172.21.15.33:6803/1633124995]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-19T02:45:19.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:18 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:45:19.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:18 smithi155 ceph-mon[35804]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:19.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:18 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:45:19.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:18 smithi033 ceph-mon[31973]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:19.029 INFO:journalctl@ceph.osd.0.smithi033.stdout:Mar 19 02:45:18 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-0[46485]: 2024-03-19T02:45:18.563+0000 7fb1036d2700 -1 osd.0 0 waiting for initial osdmap 2024-03-19T02:45:19.029 INFO:journalctl@ceph.osd.0.smithi033.stdout:Mar 19 02:45:18 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-0[46485]: 2024-03-19T02:45:18.566+0000 7fb0ff4c5700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T02:45:19.220 INFO:teuthology.orchestra.run.smithi096.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-19T02:45:19.221 INFO:teuthology.orchestra.run.smithi096.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-19T02:45:19.221 INFO:teuthology.orchestra.run.smithi096.stderr: stderr: 10+0 records in 2024-03-19T02:45:19.221 INFO:teuthology.orchestra.run.smithi096.stderr:10+0 records out 2024-03-19T02:45:19.221 INFO:teuthology.orchestra.run.smithi096.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114293 s, 917 MB/s 2024-03-19T02:45:19.221 INFO:teuthology.orchestra.run.smithi096.stderr:--> Zapping successful for: 2024-03-19T02:45:19.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:19 smithi033 ceph-mon[31973]: from='osd.0 [v2:172.21.15.33:6802/1633124995,v1:172.21.15.33:6803/1633124995]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-03-19T02:45:19.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:19 smithi033 ceph-mon[31973]: osdmap e7: 1 total, 0 up, 1 in 2024-03-19T02:45:19.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:19 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:45:19.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:19 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:45:19.841 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:19 smithi096 ceph-mon[35946]: from='osd.0 [v2:172.21.15.33:6802/1633124995,v1:172.21.15.33:6803/1633124995]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-03-19T02:45:19.841 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:19 smithi096 ceph-mon[35946]: osdmap e7: 1 total, 0 up, 1 in 2024-03-19T02:45:19.841 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:19 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:45:19.841 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:19 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:45:19.868 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi096:vg_nvme/lv_4 2024-03-19T02:45:19.917 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:19 smithi155 ceph-mon[35804]: from='osd.0 [v2:172.21.15.33:6802/1633124995,v1:172.21.15.33:6803/1633124995]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-03-19T02:45:19.917 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:19 smithi155 ceph-mon[35804]: osdmap e7: 1 total, 0 up, 1 in 2024-03-19T02:45:19.917 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:19 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:45:19.917 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:19 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:45:20.114 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T02:45:20.880 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:20 smithi096 ceph-mon[35946]: purged_snaps scrub starts 2024-03-19T02:45:20.880 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:20 smithi096 ceph-mon[35946]: purged_snaps scrub ok 2024-03-19T02:45:20.880 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:20 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:45:20.880 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:20 smithi096 ceph-mon[35946]: osd.0 [v2:172.21.15.33:6802/1633124995,v1:172.21.15.33:6803/1633124995] boot 2024-03-19T02:45:20.880 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:20 smithi096 ceph-mon[35946]: osdmap e8: 1 total, 1 up, 1 in 2024-03-19T02:45:20.880 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:20 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:45:20.880 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:20 smithi096 ceph-mon[35946]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:20.880 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:20 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:20.881 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:20 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:20.881 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:20 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:45:20.881 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:20 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:20.881 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:20 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:45:20.881 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:20 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:45:20.881 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:20 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:20.881 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:20 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:45:21.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:20 smithi155 ceph-mon[35804]: purged_snaps scrub starts 2024-03-19T02:45:21.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:20 smithi155 ceph-mon[35804]: purged_snaps scrub ok 2024-03-19T02:45:21.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:20 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:45:21.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:20 smithi155 ceph-mon[35804]: osd.0 [v2:172.21.15.33:6802/1633124995,v1:172.21.15.33:6803/1633124995] boot 2024-03-19T02:45:21.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:20 smithi155 ceph-mon[35804]: osdmap e8: 1 total, 1 up, 1 in 2024-03-19T02:45:21.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:20 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:45:21.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:20 smithi155 ceph-mon[35804]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:21.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:20 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:21.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:20 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:21.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:20 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:45:21.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:20 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:21.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:20 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:45:21.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:20 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:45:21.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:20 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:21.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:20 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:45:21.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:20 smithi033 ceph-mon[31973]: purged_snaps scrub starts 2024-03-19T02:45:21.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:20 smithi033 ceph-mon[31973]: purged_snaps scrub ok 2024-03-19T02:45:21.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:20 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:45:21.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:20 smithi033 ceph-mon[31973]: osd.0 [v2:172.21.15.33:6802/1633124995,v1:172.21.15.33:6803/1633124995] boot 2024-03-19T02:45:21.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:20 smithi033 ceph-mon[31973]: osdmap e8: 1 total, 1 up, 1 in 2024-03-19T02:45:21.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:20 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:45:21.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:20 smithi033 ceph-mon[31973]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T02:45:21.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:20 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:21.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:20 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:21.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:20 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:45:21.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:20 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:21.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:20 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:45:21.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:20 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:45:21.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:20 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:21.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:20 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:45:21.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.627+0000 7f4d81dd6700 1 -- 172.21.15.96:0/3799328192 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d7c105780 msgr2=0x7f4d7c107b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:21.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.627+0000 7f4d81dd6700 1 --2- 172.21.15.96:0/3799328192 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d7c105780 0x7f4d7c107b70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4d64009990 tx=0x7f4d64009ca0 comp rx=0 tx=0).stop 2024-03-19T02:45:21.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.627+0000 7f4d81dd6700 1 -- 172.21.15.96:0/3799328192 shutdown_connections 2024-03-19T02:45:21.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.627+0000 7f4d81dd6700 1 --2- 172.21.15.96:0/3799328192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d7c1080b0 0x7f4d7c10a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:21.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.627+0000 7f4d81dd6700 1 --2- 172.21.15.96:0/3799328192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d7c102e50 0x7f4d7c105240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:21.629 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.627+0000 7f4d81dd6700 1 --2- 172.21.15.96:0/3799328192 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d7c105780 0x7f4d7c107b70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:21.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.627+0000 7f4d81dd6700 1 -- 172.21.15.96:0/3799328192 >> 172.21.15.96:0/3799328192 conn(0x7f4d7c0fc7e0 msgr2=0x7f4d7c0fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:21.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.627+0000 7f4d81dd6700 1 -- 172.21.15.96:0/3799328192 shutdown_connections 2024-03-19T02:45:21.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.627+0000 7f4d81dd6700 1 -- 172.21.15.96:0/3799328192 wait complete. 2024-03-19T02:45:21.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.628+0000 7f4d81dd6700 1 Processor -- start 2024-03-19T02:45:21.630 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.628+0000 7f4d81dd6700 1 -- start start 2024-03-19T02:45:21.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.628+0000 7f4d81dd6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d7c102e50 0x7f4d7c198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:21.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.628+0000 7f4d81dd6700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d7c105780 0x7f4d7c198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:21.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.628+0000 7f4d81dd6700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d7c1080b0 0x7f4d7c19fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:21.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.628+0000 7f4d81dd6700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c10dec0 con 0x7f4d7c1080b0 2024-03-19T02:45:21.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.628+0000 7f4d81dd6700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c10dd40 con 0x7f4d7c102e50 2024-03-19T02:45:21.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.628+0000 7f4d81dd6700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c10e040 con 0x7f4d7c105780 2024-03-19T02:45:21.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.628+0000 7f4d7b7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d7c102e50 0x7f4d7c198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:21.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.628+0000 7f4d7b7fe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d7c102e50 0x7f4d7c198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:41348/0 (socket says 172.21.15.96:41348) 2024-03-19T02:45:21.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.628+0000 7f4d7b7fe700 1 -- 172.21.15.96:0/1279879823 learned_addr learned my addr 172.21.15.96:0/1279879823 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-03-19T02:45:21.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.628+0000 7f4d72ffd700 1 --2- 172.21.15.96:0/1279879823 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d7c105780 0x7f4d7c198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:21.632 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.628+0000 7f4d7bfff700 1 --2- 172.21.15.96:0/1279879823 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d7c1080b0 0x7f4d7c19fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:21.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.629+0000 7f4d7b7fe700 1 -- 172.21.15.96:0/1279879823 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d7c105780 msgr2=0x7f4d7c198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:21.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.629+0000 7f4d7b7fe700 1 --2- 172.21.15.96:0/1279879823 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d7c105780 0x7f4d7c198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:21.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.629+0000 7f4d7b7fe700 1 -- 172.21.15.96:0/1279879823 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d7c1080b0 msgr2=0x7f4d7c19fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:21.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.629+0000 7f4d7b7fe700 1 --2- 172.21.15.96:0/1279879823 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d7c1080b0 0x7f4d7c19fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:21.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.629+0000 7f4d7b7fe700 1 -- 172.21.15.96:0/1279879823 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d7c1a01f0 con 0x7f4d7c102e50 2024-03-19T02:45:21.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.629+0000 7f4d72ffd700 1 --2- 172.21.15.96:0/1279879823 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d7c105780 0x7f4d7c198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:45:21.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.629+0000 7f4d7bfff700 1 --2- 172.21.15.96:0/1279879823 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d7c1080b0 0x7f4d7c19fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:45:21.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.629+0000 7f4d7b7fe700 1 --2- 172.21.15.96:0/1279879823 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d7c102e50 0x7f4d7c198590 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c00b530 tx=0x7f4d6c00b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:21.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.629+0000 7f4d797fa700 1 -- 172.21.15.96:0/1279879823 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c015020 con 0x7f4d7c102e50 2024-03-19T02:45:21.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.629+0000 7f4d797fa700 1 -- 172.21.15.96:0/1279879823 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4d6c011ce0 con 0x7f4d7c102e50 2024-03-19T02:45:21.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.629+0000 7f4d81dd6700 1 -- 172.21.15.96:0/1279879823 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d7c1a04e0 con 0x7f4d7c102e50 2024-03-19T02:45:21.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.629+0000 7f4d797fa700 1 -- 172.21.15.96:0/1279879823 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c010d90 con 0x7f4d7c102e50 2024-03-19T02:45:21.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.629+0000 7f4d81dd6700 1 -- 172.21.15.96:0/1279879823 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d7c1a0a50 con 0x7f4d7c102e50 2024-03-19T02:45:21.633 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.630+0000 7f4d81dd6700 1 -- 172.21.15.96:0/1279879823 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d7c068fd0 con 0x7f4d7c102e50 2024-03-19T02:45:21.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.631+0000 7f4d797fa700 1 -- 172.21.15.96:0/1279879823 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f4d6c018020 con 0x7f4d7c102e50 2024-03-19T02:45:21.634 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.632+0000 7f4d797fa700 1 --2- 172.21.15.96:0/1279879823 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4d680a2800 0x7f4d680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:21.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.632+0000 7f4d797fa700 1 -- 172.21.15.96:0/1279879823 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f4d6c0ccdf0 con 0x7f4d7c102e50 2024-03-19T02:45:21.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.632+0000 7f4d72ffd700 1 --2- 172.21.15.96:0/1279879823 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4d680a2800 0x7f4d680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:21.635 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.632+0000 7f4d72ffd700 1 --2- 172.21.15.96:0/1279879823 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4d680a2800 0x7f4d680a4cc0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4d6400b5c0 tx=0x7f4d64005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:21.637 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.636+0000 7f4d797fa700 1 -- 172.21.15.96:0/1279879823 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4d6c01f020 con 0x7f4d7c102e50 2024-03-19T02:45:21.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:21 smithi096 ceph-mon[35946]: Detected new or changed devices on smithi033 2024-03-19T02:45:21.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:21 smithi096 ceph-mon[35946]: Adjusting osd_memory_target on smithi033 to 17093M 2024-03-19T02:45:21.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:21 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:45:21.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:21 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:45:21.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:21 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:21.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:21 smithi096 ceph-mon[35946]: osdmap e9: 1 total, 1 up, 1 in 2024-03-19T02:45:21.848 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:21.846+0000 7f4d81dd6700 1 -- 172.21.15.96:0/1279879823 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f4d7c062750 con 0x7f4d680a2800 2024-03-19T02:45:22.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:21 smithi155 ceph-mon[35804]: Detected new or changed devices on smithi033 2024-03-19T02:45:22.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:21 smithi155 ceph-mon[35804]: Adjusting osd_memory_target on smithi033 to 17093M 2024-03-19T02:45:22.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:21 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:45:22.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:21 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:45:22.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:21 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:22.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:21 smithi155 ceph-mon[35804]: osdmap e9: 1 total, 1 up, 1 in 2024-03-19T02:45:22.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:21 smithi033 ceph-mon[31973]: Detected new or changed devices on smithi033 2024-03-19T02:45:22.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:21 smithi033 ceph-mon[31973]: Adjusting osd_memory_target on smithi033 to 17093M 2024-03-19T02:45:22.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:21 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:45:22.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:21 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:45:22.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:21 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:22.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:21 smithi033 ceph-mon[31973]: osdmap e9: 1 total, 1 up, 1 in 2024-03-19T02:45:22.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:22 smithi096 ceph-mon[35946]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:22.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:22 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:45:22.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:22 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:45:22.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:22 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:45:23.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:22 smithi155 ceph-mon[35804]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:23.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:22 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:45:23.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:22 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:45:23.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:22 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:45:23.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:22 smithi033 ceph-mon[31973]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:23.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:22 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:45:23.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:22 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:45:23.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:22 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:45:23.759 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:23 smithi096 ceph-mon[35946]: from='client.24152 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:45:24.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:23 smithi155 ceph-mon[35804]: from='client.24152 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:45:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:23 smithi033 ceph-mon[31973]: from='client.24152 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:45:24.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:24 smithi096 ceph-mon[35946]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:24.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:24 smithi096 ceph-mon[35946]: from='client.? 172.21.15.96:0/2190335563' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f50eec55-1956-4486-8593-f2c8fa610e88"}]: dispatch 2024-03-19T02:45:24.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:24 smithi096 ceph-mon[35946]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f50eec55-1956-4486-8593-f2c8fa610e88"}]: dispatch 2024-03-19T02:45:24.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:24 smithi096 ceph-mon[35946]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f50eec55-1956-4486-8593-f2c8fa610e88"}]': finished 2024-03-19T02:45:24.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:24 smithi096 ceph-mon[35946]: osdmap e10: 2 total, 1 up, 2 in 2024-03-19T02:45:24.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:24 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:45:25.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:24 smithi155 ceph-mon[35804]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:25.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:24 smithi155 ceph-mon[35804]: from='client.? 172.21.15.96:0/2190335563' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f50eec55-1956-4486-8593-f2c8fa610e88"}]: dispatch 2024-03-19T02:45:25.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:24 smithi155 ceph-mon[35804]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f50eec55-1956-4486-8593-f2c8fa610e88"}]: dispatch 2024-03-19T02:45:25.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:24 smithi155 ceph-mon[35804]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f50eec55-1956-4486-8593-f2c8fa610e88"}]': finished 2024-03-19T02:45:25.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:24 smithi155 ceph-mon[35804]: osdmap e10: 2 total, 1 up, 2 in 2024-03-19T02:45:25.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:24 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:45:25.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:24 smithi033 ceph-mon[31973]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:25.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:24 smithi033 ceph-mon[31973]: from='client.? 172.21.15.96:0/2190335563' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f50eec55-1956-4486-8593-f2c8fa610e88"}]: dispatch 2024-03-19T02:45:25.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:24 smithi033 ceph-mon[31973]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f50eec55-1956-4486-8593-f2c8fa610e88"}]: dispatch 2024-03-19T02:45:25.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:24 smithi033 ceph-mon[31973]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f50eec55-1956-4486-8593-f2c8fa610e88"}]': finished 2024-03-19T02:45:25.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:24 smithi033 ceph-mon[31973]: osdmap e10: 2 total, 1 up, 2 in 2024-03-19T02:45:25.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:24 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:45:25.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:25 smithi096 ceph-mon[35946]: from='client.? 172.21.15.96:0/579388368' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:45:26.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:25 smithi155 ceph-mon[35804]: from='client.? 172.21.15.96:0/579388368' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:45:26.017 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:25 smithi033 ceph-mon[31973]: from='client.? 172.21.15.96:0/579388368' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:45:26.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:26 smithi096 ceph-mon[35946]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:27.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:26 smithi155 ceph-mon[35804]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:27.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:26 smithi033 ceph-mon[31973]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:28.008 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:27 smithi096 ceph-mon[35946]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:28.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:27 smithi155 ceph-mon[35804]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:28.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:27 smithi033 ceph-mon[31973]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:29.960 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:29 smithi096 ceph-mon[35946]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:30.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:29 smithi155 ceph-mon[35804]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:30.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:29 smithi033 ceph-mon[31973]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:31.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:31 smithi096 ceph-mon[35946]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:32.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:31 smithi155 ceph-mon[35804]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:32.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:31 smithi033 ceph-mon[31973]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:32.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:32 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-19T02:45:32.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:32 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:45:32.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:32 smithi096 ceph-mon[35946]: Deploying daemon osd.1 on smithi096 2024-03-19T02:45:33.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:32 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-19T02:45:33.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:32 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:45:33.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:32 smithi155 ceph-mon[35804]: Deploying daemon osd.1 on smithi096 2024-03-19T02:45:33.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:32 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-19T02:45:33.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:32 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:45:33.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:32 smithi033 ceph-mon[31973]: Deploying daemon osd.1 on smithi096 2024-03-19T02:45:33.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:33 smithi096 ceph-mon[35946]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:34.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:33 smithi155 ceph-mon[35804]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:34.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:33 smithi033 ceph-mon[31973]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:35.993 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:35 smithi096 ceph-mon[35946]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:36.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:35 smithi155 ceph-mon[35804]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:36.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:35 smithi033 ceph-mon[31973]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:38.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:38 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:38.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:38 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:38.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:38 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:45:38.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:38 smithi155 ceph-mon[35804]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:38.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:38 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:38.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:38 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:38.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:38 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:45:38.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:38 smithi033 ceph-mon[31973]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:38.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:38 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:38.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:38 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:38.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:38 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:45:38.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:38 smithi096 ceph-mon[35946]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:40.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:39 smithi155 ceph-mon[35804]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:40.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:39 smithi033 ceph-mon[31973]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:40.163 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:39 smithi096 ceph-mon[35946]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:40.916 INFO:teuthology.orchestra.run.smithi096.stdout:Created osd(s) 1 on host 'smithi096' 2024-03-19T02:45:40.916 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:40.915+0000 7f4d797fa700 1 -- 172.21.15.96:0/1279879823 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4d7c062750 con 0x7f4d680a2800 2024-03-19T02:45:40.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:40.917+0000 7f4d81dd6700 1 -- 172.21.15.96:0/1279879823 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4d680a2800 msgr2=0x7f4d680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:40.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:40.917+0000 7f4d81dd6700 1 --2- 172.21.15.96:0/1279879823 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4d680a2800 0x7f4d680a4cc0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4d6400b5c0 tx=0x7f4d64005fb0 comp rx=0 tx=0).stop 2024-03-19T02:45:40.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:40.917+0000 7f4d81dd6700 1 -- 172.21.15.96:0/1279879823 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d7c102e50 msgr2=0x7f4d7c198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:40.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:40.917+0000 7f4d81dd6700 1 --2- 172.21.15.96:0/1279879823 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d7c102e50 0x7f4d7c198590 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c00b530 tx=0x7f4d6c00b8f0 comp rx=0 tx=0).stop 2024-03-19T02:45:40.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:40.917+0000 7f4d81dd6700 1 -- 172.21.15.96:0/1279879823 shutdown_connections 2024-03-19T02:45:40.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:40.917+0000 7f4d81dd6700 1 --2- 172.21.15.96:0/1279879823 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4d680a2800 0x7f4d680a4cc0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:40.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:40.917+0000 7f4d81dd6700 1 --2- 172.21.15.96:0/1279879823 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d7c1080b0 0x7f4d7c19fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:40.918 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:40.917+0000 7f4d81dd6700 1 --2- 172.21.15.96:0/1279879823 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d7c102e50 0x7f4d7c198590 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:40.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:40.917+0000 7f4d81dd6700 1 --2- 172.21.15.96:0/1279879823 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d7c105780 0x7f4d7c198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:40.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:40.917+0000 7f4d81dd6700 1 -- 172.21.15.96:0/1279879823 >> 172.21.15.96:0/1279879823 conn(0x7f4d7c0fc7e0 msgr2=0x7f4d7c103890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:40.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:40.917+0000 7f4d81dd6700 1 -- 172.21.15.96:0/1279879823 shutdown_connections 2024-03-19T02:45:40.919 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:45:40.917+0000 7f4d81dd6700 1 -- 172.21.15.96:0/1279879823 wait complete. 2024-03-19T02:45:41.146 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:40 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:41.146 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:40 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:41.146 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:40 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:45:41.146 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:40 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:45:41.146 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:40 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:41.147 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:40 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:41.147 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:40 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:41.147 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:40 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:45:41.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:41.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:41.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:45:41.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:45:41.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:41.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:41.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:41.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:45:41.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:41.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:41.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:45:41.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:45:41.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:41.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:41.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:41.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:45:41.558 DEBUG:teuthology.orchestra.run.smithi096:osd.1> sudo journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.1.service 2024-03-19T02:45:41.560 INFO:tasks.cephadm:Deploying osd.2 on smithi155 with /dev/vg_nvme/lv_4... 2024-03-19T02:45:41.561 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-19T02:45:41.807 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T02:45:42.201 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:41 smithi155 ceph-mon[35804]: from='osd.1 [v2:172.21.15.96:6800/340144398,v1:172.21.15.96:6801/340144398]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-19T02:45:42.201 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:41 smithi155 ceph-mon[35804]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-19T02:45:42.201 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:41 smithi155 ceph-mon[35804]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:42.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:41 smithi096 ceph-mon[35946]: from='osd.1 [v2:172.21.15.96:6800/340144398,v1:172.21.15.96:6801/340144398]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-19T02:45:42.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:41 smithi096 ceph-mon[35946]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-19T02:45:42.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:41 smithi096 ceph-mon[35946]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:42.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:41 smithi033 ceph-mon[31973]: from='osd.1 [v2:172.21.15.96:6800/340144398,v1:172.21.15.96:6801/340144398]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-19T02:45:42.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:41 smithi033 ceph-mon[31973]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-19T02:45:42.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:41 smithi033 ceph-mon[31973]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:42.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:42 smithi096 ceph-mon[35946]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-19T02:45:42.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:42 smithi096 ceph-mon[35946]: from='osd.1 [v2:172.21.15.96:6800/340144398,v1:172.21.15.96:6801/340144398]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-03-19T02:45:42.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:42 smithi096 ceph-mon[35946]: osdmap e11: 2 total, 1 up, 2 in 2024-03-19T02:45:42.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:42 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:45:42.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:42 smithi096 ceph-mon[35946]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-03-19T02:45:43.151 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:42 smithi155 ceph-mon[35804]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-19T02:45:43.151 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:42 smithi155 ceph-mon[35804]: from='osd.1 [v2:172.21.15.96:6800/340144398,v1:172.21.15.96:6801/340144398]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-03-19T02:45:43.151 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:42 smithi155 ceph-mon[35804]: osdmap e11: 2 total, 1 up, 2 in 2024-03-19T02:45:43.151 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:42 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:45:43.151 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:42 smithi155 ceph-mon[35804]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-03-19T02:45:43.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:42 smithi033 ceph-mon[31973]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-19T02:45:43.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:42 smithi033 ceph-mon[31973]: from='osd.1 [v2:172.21.15.96:6800/340144398,v1:172.21.15.96:6801/340144398]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-03-19T02:45:43.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:42 smithi033 ceph-mon[31973]: osdmap e11: 2 total, 1 up, 2 in 2024-03-19T02:45:43.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:42 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:45:43.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:42 smithi033 ceph-mon[31973]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-03-19T02:45:43.750 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-19T02:45:43.750 INFO:teuthology.orchestra.run.smithi155.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-19T02:45:43.750 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10+0 records in 2024-03-19T02:45:43.750 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2024-03-19T02:45:43.750 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112259 s, 934 MB/s 2024-03-19T02:45:43.750 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping successful for: 2024-03-19T02:45:43.922 INFO:journalctl@ceph.osd.1.smithi096.stdout:Mar 19 02:45:43 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-1[40541]: 2024-03-19T02:45:43.656+0000 7f00bbc1c700 -1 osd.1 0 waiting for initial osdmap 2024-03-19T02:45:43.923 INFO:journalctl@ceph.osd.1.smithi096.stdout:Mar 19 02:45:43 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-1[40541]: 2024-03-19T02:45:43.659+0000 7f00b3a07700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T02:45:44.110 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:43 smithi155 ceph-mon[35804]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-03-19T02:45:44.111 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:43 smithi155 ceph-mon[35804]: osdmap e12: 2 total, 1 up, 2 in 2024-03-19T02:45:44.111 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:43 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:45:44.111 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:43 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:45:44.111 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:43 smithi155 ceph-mon[35804]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:44.111 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:43 smithi155 ceph-mon[35804]: from='osd.1 ' entity='osd.1' 2024-03-19T02:45:44.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:43 smithi096 ceph-mon[35946]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-03-19T02:45:44.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:43 smithi096 ceph-mon[35946]: osdmap e12: 2 total, 1 up, 2 in 2024-03-19T02:45:44.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:43 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:45:44.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:43 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:45:44.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:43 smithi096 ceph-mon[35946]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:44.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:43 smithi096 ceph-mon[35946]: from='osd.1 ' entity='osd.1' 2024-03-19T02:45:44.251 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:43 smithi033 ceph-mon[31973]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-03-19T02:45:44.251 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:43 smithi033 ceph-mon[31973]: osdmap e12: 2 total, 1 up, 2 in 2024-03-19T02:45:44.251 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:43 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:45:44.251 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:43 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:45:44.251 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:43 smithi033 ceph-mon[31973]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T02:45:44.252 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:43 smithi033 ceph-mon[31973]: from='osd.1 ' entity='osd.1' 2024-03-19T02:45:44.384 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi155:vg_nvme/lv_4 2024-03-19T02:45:44.636 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T02:45:45.217 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:44 smithi096 ceph-mon[35946]: purged_snaps scrub starts 2024-03-19T02:45:45.218 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:44 smithi096 ceph-mon[35946]: purged_snaps scrub ok 2024-03-19T02:45:45.218 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:44 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:45:45.218 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:44 smithi096 ceph-mon[35946]: Detected new or changed devices on smithi096 2024-03-19T02:45:45.218 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:44 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:45.218 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:44 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:45.218 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:44 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:45:45.218 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:44 smithi096 ceph-mon[35946]: Adjusting osd_memory_target on smithi096 to 17088M 2024-03-19T02:45:45.218 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:44 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:45.218 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:44 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:45:45.218 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:44 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:45:45.218 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:44 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:45.219 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:44 smithi096 ceph-mon[35946]: osd.1 [v2:172.21.15.96:6800/340144398,v1:172.21.15.96:6801/340144398] boot 2024-03-19T02:45:45.219 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:44 smithi096 ceph-mon[35946]: osdmap e13: 2 total, 2 up, 2 in 2024-03-19T02:45:45.219 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:44 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:45:45.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:44 smithi155 ceph-mon[35804]: purged_snaps scrub starts 2024-03-19T02:45:45.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:44 smithi155 ceph-mon[35804]: purged_snaps scrub ok 2024-03-19T02:45:45.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:44 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:45:45.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:44 smithi155 ceph-mon[35804]: Detected new or changed devices on smithi096 2024-03-19T02:45:45.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:44 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:45.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:44 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:45.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:44 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:45:45.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:44 smithi155 ceph-mon[35804]: Adjusting osd_memory_target on smithi096 to 17088M 2024-03-19T02:45:45.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:44 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:45.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:44 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:45:45.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:44 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:45:45.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:44 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:45.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:44 smithi155 ceph-mon[35804]: osd.1 [v2:172.21.15.96:6800/340144398,v1:172.21.15.96:6801/340144398] boot 2024-03-19T02:45:45.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:44 smithi155 ceph-mon[35804]: osdmap e13: 2 total, 2 up, 2 in 2024-03-19T02:45:45.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:44 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:45:45.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:44 smithi033 ceph-mon[31973]: purged_snaps scrub starts 2024-03-19T02:45:45.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:44 smithi033 ceph-mon[31973]: purged_snaps scrub ok 2024-03-19T02:45:45.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:44 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:45:45.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:44 smithi033 ceph-mon[31973]: Detected new or changed devices on smithi096 2024-03-19T02:45:45.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:44 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:45.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:44 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:45.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:44 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:45:45.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:44 smithi033 ceph-mon[31973]: Adjusting osd_memory_target on smithi096 to 17088M 2024-03-19T02:45:45.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:44 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:45.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:44 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:45:45.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:44 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:45:45.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:44 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:45:45.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:44 smithi033 ceph-mon[31973]: osd.1 [v2:172.21.15.96:6800/340144398,v1:172.21.15.96:6801/340144398] boot 2024-03-19T02:45:45.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:44 smithi033 ceph-mon[31973]: osdmap e13: 2 total, 2 up, 2 in 2024-03-19T02:45:45.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:44 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:45:46.020 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.016+0000 7f2bf1265700 1 -- 172.21.15.155:0/330676327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2bec102e50 msgr2=0x7f2bec109700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:46.020 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.016+0000 7f2bf1265700 1 --2- 172.21.15.155:0/330676327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2bec102e50 0x7f2bec109700 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2be400b0a0 tx=0x7f2be400b3b0 comp rx=0 tx=0).stop 2024-03-19T02:45:46.020 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.016+0000 7f2bf1265700 1 -- 172.21.15.155:0/330676327 shutdown_connections 2024-03-19T02:45:46.020 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.016+0000 7f2bf1265700 1 --2- 172.21.15.155:0/330676327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2bec102e50 0x7f2bec109700 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:46.020 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.016+0000 7f2bf1265700 1 --2- 172.21.15.155:0/330676327 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2bec1012b0 0x7f2bec1016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:46.020 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.016+0000 7f2bf1265700 1 --2- 172.21.15.155:0/330676327 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2bec1024b0 0x7f2bec102910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:46.020 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.016+0000 7f2bf1265700 1 -- 172.21.15.155:0/330676327 >> 172.21.15.155:0/330676327 conn(0x7f2bec0fc870 msgr2=0x7f2bec0fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:45:46.020 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.017+0000 7f2bf1265700 1 -- 172.21.15.155:0/330676327 shutdown_connections 2024-03-19T02:45:46.020 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.017+0000 7f2bf1265700 1 -- 172.21.15.155:0/330676327 wait complete. 2024-03-19T02:45:46.020 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.017+0000 7f2bf1265700 1 Processor -- start 2024-03-19T02:45:46.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.017+0000 7f2bf1265700 1 -- start start 2024-03-19T02:45:46.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.018+0000 7f2bf1265700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2bec1012b0 0x7f2bec06b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:46.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.018+0000 7f2bf1265700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2bec1024b0 0x7f2bec19d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:46.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.018+0000 7f2bf1265700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2bec102e50 0x7f2bec19f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:46.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.018+0000 7f2bf1265700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2bec10cd40 con 0x7f2bec1012b0 2024-03-19T02:45:46.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.018+0000 7f2bf1265700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2bec10cbc0 con 0x7f2bec102e50 2024-03-19T02:45:46.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.018+0000 7f2bf1265700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2bec10cec0 con 0x7f2bec1024b0 2024-03-19T02:45:46.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.018+0000 7f2bea7fc700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2bec1024b0 0x7f2bec19d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:46.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.018+0000 7f2bea7fc700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2bec1024b0 0x7f2bec19d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:49420/0 (socket says 172.21.15.155:49420) 2024-03-19T02:45:46.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.018+0000 7f2bea7fc700 1 -- 172.21.15.155:0/617227649 learned_addr learned my addr 172.21.15.155:0/617227649 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-19T02:45:46.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.018+0000 7f2beaffd700 1 --2- 172.21.15.155:0/617227649 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2bec1012b0 0x7f2bec06b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:46.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.018+0000 7f2beb7fe700 1 --2- 172.21.15.155:0/617227649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2bec102e50 0x7f2bec19f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:46.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.018+0000 7f2bea7fc700 1 -- 172.21.15.155:0/617227649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2bec102e50 msgr2=0x7f2bec19f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:46.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.018+0000 7f2bea7fc700 1 --2- 172.21.15.155:0/617227649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2bec102e50 0x7f2bec19f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:46.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.018+0000 7f2bea7fc700 1 -- 172.21.15.155:0/617227649 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2bec1012b0 msgr2=0x7f2bec06b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:45:46.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.018+0000 7f2bea7fc700 1 --2- 172.21.15.155:0/617227649 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2bec1012b0 0x7f2bec06b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:45:46.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.018+0000 7f2bea7fc700 1 -- 172.21.15.155:0/617227649 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bec1a0080 con 0x7f2bec1024b0 2024-03-19T02:45:46.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.018+0000 7f2beaffd700 1 --2- 172.21.15.155:0/617227649 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2bec1012b0 0x7f2bec06b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:45:46.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.018+0000 7f2bea7fc700 1 --2- 172.21.15.155:0/617227649 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2bec1024b0 0x7f2bec19d120 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2be000b530 tx=0x7f2be000b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:46.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.018+0000 7f2bd3fff700 1 -- 172.21.15.155:0/617227649 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be0015020 con 0x7f2bec1024b0 2024-03-19T02:45:46.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.018+0000 7f2bd3fff700 1 -- 172.21.15.155:0/617227649 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2be0011ce0 con 0x7f2bec1024b0 2024-03-19T02:45:46.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.018+0000 7f2bf1265700 1 -- 172.21.15.155:0/617227649 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bec1a0370 con 0x7f2bec1024b0 2024-03-19T02:45:46.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.018+0000 7f2bd3fff700 1 -- 172.21.15.155:0/617227649 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be0010d60 con 0x7f2bec1024b0 2024-03-19T02:45:46.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.019+0000 7f2bf1265700 1 -- 172.21.15.155:0/617227649 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bec1a0980 con 0x7f2bec1024b0 2024-03-19T02:45:46.023 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.019+0000 7f2bf1265700 1 -- 172.21.15.155:0/617227649 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bec068fd0 con 0x7f2bec1024b0 2024-03-19T02:45:46.023 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.021+0000 7f2bd3fff700 1 -- 172.21.15.155:0/617227649 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f2be0018020 con 0x7f2bec1024b0 2024-03-19T02:45:46.023 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.022+0000 7f2bd3fff700 1 --2- 172.21.15.155:0/617227649 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2bd40a2880 0x7f2bd40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:45:46.023 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.022+0000 7f2bd3fff700 1 -- 172.21.15.155:0/617227649 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f2be00ccf90 con 0x7f2bec1024b0 2024-03-19T02:45:46.023 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.022+0000 7f2beaffd700 1 --2- 172.21.15.155:0/617227649 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2bd40a2880 0x7f2bd40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:45:46.023 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.023+0000 7f2beaffd700 1 --2- 172.21.15.155:0/617227649 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2bd40a2880 0x7f2bd40a4d40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc005fd0 tx=0x7f2bdc005e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:45:46.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.026+0000 7f2bd3fff700 1 -- 172.21.15.155:0/617227649 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2be001f020 con 0x7f2bec1024b0 2024-03-19T02:45:46.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:45 smithi096 ceph-mon[35946]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:45:46.228 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:45:46.227+0000 7f2bf1265700 1 -- 172.21.15.155:0/617227649 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f2bec062750 con 0x7f2bd40a2880 2024-03-19T02:45:46.228 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:45 smithi155 ceph-mon[35804]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:45:46.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:45 smithi033 ceph-mon[31973]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:45:47.123 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:46 smithi155 ceph-mon[35804]: osdmap e14: 2 total, 2 up, 2 in 2024-03-19T02:45:47.123 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:46 smithi155 ceph-mon[35804]: from='client.24154 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:45:47.123 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:46 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:45:47.123 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:46 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:45:47.123 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:46 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:45:47.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:46 smithi096 ceph-mon[35946]: osdmap e14: 2 total, 2 up, 2 in 2024-03-19T02:45:47.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:46 smithi096 ceph-mon[35946]: from='client.24154 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:45:47.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:46 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:45:47.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:46 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:45:47.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:46 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:45:47.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:46 smithi033 ceph-mon[31973]: osdmap e14: 2 total, 2 up, 2 in 2024-03-19T02:45:47.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:46 smithi033 ceph-mon[31973]: from='client.24154 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:45:47.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:46 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:45:47.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:46 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:45:47.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:46 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:45:48.013 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:47 smithi155 ceph-mon[35804]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:45:48.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:47 smithi033 ceph-mon[31973]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:45:48.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:47 smithi096 ceph-mon[35946]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:45:49.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:48 smithi155 ceph-mon[35804]: from='client.? 172.21.15.155:0/4285960993' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "893a2d62-80d9-4e5e-bc6a-a1c8c681d652"}]: dispatch 2024-03-19T02:45:49.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:48 smithi155 ceph-mon[35804]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "893a2d62-80d9-4e5e-bc6a-a1c8c681d652"}]: dispatch 2024-03-19T02:45:49.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:48 smithi155 ceph-mon[35804]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "893a2d62-80d9-4e5e-bc6a-a1c8c681d652"}]': finished 2024-03-19T02:45:49.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:48 smithi155 ceph-mon[35804]: osdmap e15: 3 total, 2 up, 3 in 2024-03-19T02:45:49.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:48 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:45:49.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:48 smithi033 ceph-mon[31973]: from='client.? 172.21.15.155:0/4285960993' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "893a2d62-80d9-4e5e-bc6a-a1c8c681d652"}]: dispatch 2024-03-19T02:45:49.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:48 smithi033 ceph-mon[31973]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "893a2d62-80d9-4e5e-bc6a-a1c8c681d652"}]: dispatch 2024-03-19T02:45:49.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:48 smithi033 ceph-mon[31973]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "893a2d62-80d9-4e5e-bc6a-a1c8c681d652"}]': finished 2024-03-19T02:45:49.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:48 smithi033 ceph-mon[31973]: osdmap e15: 3 total, 2 up, 3 in 2024-03-19T02:45:49.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:48 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:45:49.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:48 smithi096 ceph-mon[35946]: from='client.? 172.21.15.155:0/4285960993' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "893a2d62-80d9-4e5e-bc6a-a1c8c681d652"}]: dispatch 2024-03-19T02:45:49.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:48 smithi096 ceph-mon[35946]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "893a2d62-80d9-4e5e-bc6a-a1c8c681d652"}]: dispatch 2024-03-19T02:45:49.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:48 smithi096 ceph-mon[35946]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "893a2d62-80d9-4e5e-bc6a-a1c8c681d652"}]': finished 2024-03-19T02:45:49.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:48 smithi096 ceph-mon[35946]: osdmap e15: 3 total, 2 up, 3 in 2024-03-19T02:45:49.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:48 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:45:50.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:49 smithi155 ceph-mon[35804]: from='client.? 172.21.15.155:0/3890261229' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:45:50.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:49 smithi155 ceph-mon[35804]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:45:50.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:49 smithi033 ceph-mon[31973]: from='client.? 172.21.15.155:0/3890261229' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:45:50.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:49 smithi033 ceph-mon[31973]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:45:50.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:49 smithi096 ceph-mon[35946]: from='client.? 172.21.15.155:0/3890261229' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:45:50.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:49 smithi096 ceph-mon[35946]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:45:52.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:51 smithi155 ceph-mon[35804]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:45:52.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:51 smithi033 ceph-mon[31973]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:45:52.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:51 smithi096 ceph-mon[35946]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:45:53.929 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:53 smithi155 ceph-mon[35804]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:45:54.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:53 smithi033 ceph-mon[31973]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:45:54.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:53 smithi096 ceph-mon[35946]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:45:55.755 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:55 smithi155 ceph-mon[35804]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:45:56.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:55 smithi033 ceph-mon[31973]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:45:56.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:55 smithi096 ceph-mon[35946]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:45:57.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:56 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-19T02:45:57.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:56 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:45:57.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:56 smithi155 ceph-mon[35804]: Deploying daemon osd.2 on smithi155 2024-03-19T02:45:57.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:56 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-19T02:45:57.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:56 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:45:57.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:56 smithi033 ceph-mon[31973]: Deploying daemon osd.2 on smithi155 2024-03-19T02:45:57.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:56 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-19T02:45:57.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:56 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:45:57.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:56 smithi096 ceph-mon[35946]: Deploying daemon osd.2 on smithi155 2024-03-19T02:45:57.972 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:57 smithi155 ceph-mon[35804]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:45:58.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:57 smithi033 ceph-mon[31973]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:45:58.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:57 smithi096 ceph-mon[35946]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:46:00.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:45:59 smithi155 ceph-mon[35804]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:46:00.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:45:59 smithi033 ceph-mon[31973]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:46:00.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:45:59 smithi096 ceph-mon[35946]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:46:02.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:02 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:02.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:02 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:02.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:02 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:46:02.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:02 smithi155 ceph-mon[35804]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:46:02.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:02 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:02.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:02 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:02.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:02 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:46:02.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:02 smithi096 ceph-mon[35946]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:46:02.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:02 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:02.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:02 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:02.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:02 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:46:02.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:02 smithi033 ceph-mon[31973]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:46:04.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:04 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:04.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:04 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:04.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:04 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:46:04.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:04 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:46:04.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:04 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:04.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:04 smithi033 ceph-mon[31973]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:46:04.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:04 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:04.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:04 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:04.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:04 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:46:04.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:04 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:46:04.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:04 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:04.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:04 smithi096 ceph-mon[35946]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:46:04.487 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:04 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:04.487 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:04 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:04.487 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:04 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:46:04.487 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:04 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:46:04.487 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:04 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:04.487 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:04 smithi155 ceph-mon[35804]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:46:04.893 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:46:04.892+0000 7f2bd3fff700 1 -- 172.21.15.155:0/617227649 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2bec062750 con 0x7f2bd40a2880 2024-03-19T02:46:04.893 INFO:teuthology.orchestra.run.smithi155.stdout:Created osd(s) 2 on host 'smithi155' 2024-03-19T02:46:04.895 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:46:04.894+0000 7f2bf1265700 1 -- 172.21.15.155:0/617227649 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2bd40a2880 msgr2=0x7f2bd40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:04.896 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:46:04.894+0000 7f2bf1265700 1 --2- 172.21.15.155:0/617227649 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2bd40a2880 0x7f2bd40a4d40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc005fd0 tx=0x7f2bdc005e40 comp rx=0 tx=0).stop 2024-03-19T02:46:04.896 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:46:04.894+0000 7f2bf1265700 1 -- 172.21.15.155:0/617227649 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2bec1024b0 msgr2=0x7f2bec19d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:04.896 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:46:04.894+0000 7f2bf1265700 1 --2- 172.21.15.155:0/617227649 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2bec1024b0 0x7f2bec19d120 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2be000b530 tx=0x7f2be000b8f0 comp rx=0 tx=0).stop 2024-03-19T02:46:04.896 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:46:04.894+0000 7f2bf1265700 1 -- 172.21.15.155:0/617227649 shutdown_connections 2024-03-19T02:46:04.896 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:46:04.894+0000 7f2bf1265700 1 --2- 172.21.15.155:0/617227649 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2bd40a2880 0x7f2bd40a4d40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:04.896 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:46:04.894+0000 7f2bf1265700 1 --2- 172.21.15.155:0/617227649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2bec102e50 0x7f2bec19f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:04.897 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:46:04.894+0000 7f2bf1265700 1 --2- 172.21.15.155:0/617227649 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2bec1012b0 0x7f2bec06b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:04.897 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:46:04.894+0000 7f2bf1265700 1 --2- 172.21.15.155:0/617227649 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2bec1024b0 0x7f2bec19d120 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:04.897 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:46:04.894+0000 7f2bf1265700 1 -- 172.21.15.155:0/617227649 >> 172.21.15.155:0/617227649 conn(0x7f2bec0fc870 msgr2=0x7f2bec1057b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:04.897 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:46:04.894+0000 7f2bf1265700 1 -- 172.21.15.155:0/617227649 shutdown_connections 2024-03-19T02:46:04.897 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:46:04.894+0000 7f2bf1265700 1 -- 172.21.15.155:0/617227649 wait complete. 2024-03-19T02:46:05.508 DEBUG:teuthology.orchestra.run.smithi155:osd.2> sudo journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.2.service 2024-03-19T02:46:05.510 INFO:tasks.cephadm:Deploying osd.3 on smithi033 with /dev/vg_nvme/lv_3... 2024-03-19T02:46:05.510 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-19T02:46:05.764 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:46:06.149 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:05 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:06.149 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:05 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:06.149 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:05 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:46:06.149 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:05 smithi033 ceph-mon[31973]: from='osd.2 [v2:172.21.15.155:6800/3388743876,v1:172.21.15.155:6801/3388743876]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-19T02:46:06.149 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:05 smithi033 ceph-mon[31973]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-19T02:46:06.149 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:05 smithi033 ceph-mon[31973]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:46:06.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:05 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:06.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:05 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:06.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:05 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:46:06.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:05 smithi096 ceph-mon[35946]: from='osd.2 [v2:172.21.15.155:6800/3388743876,v1:172.21.15.155:6801/3388743876]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-19T02:46:06.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:05 smithi096 ceph-mon[35946]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-19T02:46:06.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:05 smithi096 ceph-mon[35946]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:46:06.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:05 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:06.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:05 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:06.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:05 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:46:06.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:05 smithi155 ceph-mon[35804]: from='osd.2 [v2:172.21.15.155:6800/3388743876,v1:172.21.15.155:6801/3388743876]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-19T02:46:06.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:05 smithi155 ceph-mon[35804]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-19T02:46:06.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:05 smithi155 ceph-mon[35804]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:46:07.159 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:06 smithi155 ceph-mon[35804]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-19T02:46:07.160 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:06 smithi155 ceph-mon[35804]: osdmap e16: 3 total, 2 up, 3 in 2024-03-19T02:46:07.160 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:06 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:46:07.160 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:06 smithi155 ceph-mon[35804]: from='osd.2 [v2:172.21.15.155:6800/3388743876,v1:172.21.15.155:6801/3388743876]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-19T02:46:07.160 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:06 smithi155 ceph-mon[35804]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-19T02:46:07.160 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 19 02:46:06 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-2[41086]: 2024-03-19T02:46:06.865+0000 7fb20b588700 -1 osd.2 0 waiting for initial osdmap 2024-03-19T02:46:07.160 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 19 02:46:06 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-2[41086]: 2024-03-19T02:46:06.868+0000 7fb20737b700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T02:46:07.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:06 smithi096 ceph-mon[35946]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-19T02:46:07.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:06 smithi096 ceph-mon[35946]: osdmap e16: 3 total, 2 up, 3 in 2024-03-19T02:46:07.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:06 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:46:07.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:06 smithi096 ceph-mon[35946]: from='osd.2 [v2:172.21.15.155:6800/3388743876,v1:172.21.15.155:6801/3388743876]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-19T02:46:07.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:06 smithi096 ceph-mon[35946]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-19T02:46:07.221 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:06 smithi033 ceph-mon[31973]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-19T02:46:07.221 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:06 smithi033 ceph-mon[31973]: osdmap e16: 3 total, 2 up, 3 in 2024-03-19T02:46:07.221 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:06 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:46:07.221 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:06 smithi033 ceph-mon[31973]: from='osd.2 [v2:172.21.15.155:6800/3388743876,v1:172.21.15.155:6801/3388743876]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-19T02:46:07.221 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:06 smithi033 ceph-mon[31973]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-19T02:46:07.956 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-19T02:46:07.956 INFO:teuthology.orchestra.run.smithi033.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-19T02:46:07.956 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10+0 records in 2024-03-19T02:46:07.956 INFO:teuthology.orchestra.run.smithi033.stderr:10+0 records out 2024-03-19T02:46:07.956 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111165 s, 943 MB/s 2024-03-19T02:46:07.956 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping successful for: 2024-03-19T02:46:08.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:07 smithi155 ceph-mon[35804]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-03-19T02:46:08.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:07 smithi155 ceph-mon[35804]: osdmap e17: 3 total, 2 up, 3 in 2024-03-19T02:46:08.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:07 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:46:08.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:07 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:46:08.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:07 smithi155 ceph-mon[35804]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:46:08.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:07 smithi096 ceph-mon[35946]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-03-19T02:46:08.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:07 smithi096 ceph-mon[35946]: osdmap e17: 3 total, 2 up, 3 in 2024-03-19T02:46:08.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:07 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:46:08.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:07 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:46:08.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:07 smithi096 ceph-mon[35946]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:46:08.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:07 smithi033 ceph-mon[31973]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-03-19T02:46:08.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:07 smithi033 ceph-mon[31973]: osdmap e17: 3 total, 2 up, 3 in 2024-03-19T02:46:08.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:07 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:46:08.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:07 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:46:08.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:07 smithi033 ceph-mon[31973]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T02:46:08.633 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi033:vg_nvme/lv_3 2024-03-19T02:46:08.886 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:46:09.144 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:08 smithi033 ceph-mon[31973]: purged_snaps scrub starts 2024-03-19T02:46:09.144 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:08 smithi033 ceph-mon[31973]: purged_snaps scrub ok 2024-03-19T02:46:09.144 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:08 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:46:09.144 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:08 smithi033 ceph-mon[31973]: osd.2 [v2:172.21.15.155:6800/3388743876,v1:172.21.15.155:6801/3388743876] boot 2024-03-19T02:46:09.144 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:08 smithi033 ceph-mon[31973]: osdmap e18: 3 total, 3 up, 3 in 2024-03-19T02:46:09.144 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:08 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:46:09.144 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:08 smithi033 ceph-mon[31973]: Detected new or changed devices on smithi155 2024-03-19T02:46:09.144 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:08 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:09.145 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:08 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:09.145 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:08 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:46:09.145 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:08 smithi033 ceph-mon[31973]: Adjusting osd_memory_target on smithi155 to 17088M 2024-03-19T02:46:09.145 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:08 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:09.145 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:08 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:46:09.145 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:08 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:46:09.145 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:08 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:09.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:08 smithi096 ceph-mon[35946]: purged_snaps scrub starts 2024-03-19T02:46:09.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:08 smithi096 ceph-mon[35946]: purged_snaps scrub ok 2024-03-19T02:46:09.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:08 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:46:09.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:08 smithi096 ceph-mon[35946]: osd.2 [v2:172.21.15.155:6800/3388743876,v1:172.21.15.155:6801/3388743876] boot 2024-03-19T02:46:09.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:08 smithi096 ceph-mon[35946]: osdmap e18: 3 total, 3 up, 3 in 2024-03-19T02:46:09.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:08 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:46:09.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:08 smithi096 ceph-mon[35946]: Detected new or changed devices on smithi155 2024-03-19T02:46:09.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:08 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:09.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:08 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:09.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:08 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:46:09.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:08 smithi096 ceph-mon[35946]: Adjusting osd_memory_target on smithi155 to 17088M 2024-03-19T02:46:09.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:08 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:09.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:08 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:46:09.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:08 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:46:09.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:08 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:09.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:08 smithi155 ceph-mon[35804]: purged_snaps scrub starts 2024-03-19T02:46:09.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:08 smithi155 ceph-mon[35804]: purged_snaps scrub ok 2024-03-19T02:46:09.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:08 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:46:09.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:08 smithi155 ceph-mon[35804]: osd.2 [v2:172.21.15.155:6800/3388743876,v1:172.21.15.155:6801/3388743876] boot 2024-03-19T02:46:09.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:08 smithi155 ceph-mon[35804]: osdmap e18: 3 total, 3 up, 3 in 2024-03-19T02:46:09.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:08 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:46:09.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:08 smithi155 ceph-mon[35804]: Detected new or changed devices on smithi155 2024-03-19T02:46:09.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:08 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:09.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:08 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:09.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:08 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:46:09.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:08 smithi155 ceph-mon[35804]: Adjusting osd_memory_target on smithi155 to 17088M 2024-03-19T02:46:09.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:08 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:09.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:08 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:46:09.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:08 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:46:09.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:08 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:10.199 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:09 smithi033 ceph-mon[31973]: osdmap e19: 3 total, 3 up, 3 in 2024-03-19T02:46:10.199 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:09 smithi033 ceph-mon[31973]: pgmap v84: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:10.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:09 smithi155 ceph-mon[35804]: osdmap e19: 3 total, 3 up, 3 in 2024-03-19T02:46:10.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:09 smithi155 ceph-mon[35804]: pgmap v84: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:10.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.406+0000 7f4b0982c700 1 -- 172.21.15.33:0/3758496509 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b04105760 msgr2=0x7f4b04107b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:10.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.406+0000 7f4b0982c700 1 --2- 172.21.15.33:0/3758496509 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b04105760 0x7f4b04107b50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4af8009990 tx=0x7f4af8009ca0 comp rx=0 tx=0).stop 2024-03-19T02:46:10.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.406+0000 7f4b0982c700 1 -- 172.21.15.33:0/3758496509 shutdown_connections 2024-03-19T02:46:10.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.406+0000 7f4b0982c700 1 --2- 172.21.15.33:0/3758496509 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b04108090 0x7f4b0410a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:10.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.406+0000 7f4b0982c700 1 --2- 172.21.15.33:0/3758496509 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b04102e30 0x7f4b04105220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:10.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.406+0000 7f4b0982c700 1 --2- 172.21.15.33:0/3758496509 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b04105760 0x7f4b04107b50 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:10.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.406+0000 7f4b0982c700 1 -- 172.21.15.33:0/3758496509 >> 172.21.15.33:0/3758496509 conn(0x7f4b040fc800 msgr2=0x7f4b040fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:10.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.406+0000 7f4b0982c700 1 -- 172.21.15.33:0/3758496509 shutdown_connections 2024-03-19T02:46:10.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.406+0000 7f4b0982c700 1 -- 172.21.15.33:0/3758496509 wait complete. 2024-03-19T02:46:10.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.407+0000 7f4b0982c700 1 Processor -- start 2024-03-19T02:46:10.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.407+0000 7f4b0982c700 1 -- start start 2024-03-19T02:46:10.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.407+0000 7f4b0982c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b04102e30 0x7f4b04102790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:10.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.407+0000 7f4b0982c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b04105760 0x7f4b04100de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:10.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.407+0000 7f4b0982c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b04108090 0x7f4b04101320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:10.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.407+0000 7f4b0982c700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0410de00 con 0x7f4b04105760 2024-03-19T02:46:10.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.407+0000 7f4b0982c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0410dc80 con 0x7f4b04108090 2024-03-19T02:46:10.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.407+0000 7f4b0982c700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0410df80 con 0x7f4b04102e30 2024-03-19T02:46:10.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.407+0000 7f4b027fc700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b04105760 0x7f4b04100de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:10.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.407+0000 7f4b027fc700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b04105760 0x7f4b04100de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50996/0 (socket says 172.21.15.33:50996) 2024-03-19T02:46:10.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.407+0000 7f4b027fc700 1 -- 172.21.15.33:0/3499177113 learned_addr learned my addr 172.21.15.33:0/3499177113 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:46:10.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.407+0000 7f4b037fe700 1 --2- 172.21.15.33:0/3499177113 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b04108090 0x7f4b04101320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:10.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.407+0000 7f4b02ffd700 1 --2- 172.21.15.33:0/3499177113 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b04102e30 0x7f4b04102790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:10.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.408+0000 7f4b027fc700 1 -- 172.21.15.33:0/3499177113 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b04102e30 msgr2=0x7f4b04102790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:10.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.408+0000 7f4b027fc700 1 --2- 172.21.15.33:0/3499177113 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b04102e30 0x7f4b04102790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:10.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.408+0000 7f4b027fc700 1 -- 172.21.15.33:0/3499177113 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b04108090 msgr2=0x7f4b04101320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:10.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.408+0000 7f4b027fc700 1 --2- 172.21.15.33:0/3499177113 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b04108090 0x7f4b04101320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:10.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.408+0000 7f4b027fc700 1 -- 172.21.15.33:0/3499177113 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b04101bd0 con 0x7f4b04105760 2024-03-19T02:46:10.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.408+0000 7f4b027fc700 1 --2- 172.21.15.33:0/3499177113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b04105760 0x7f4b04100de0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4af800b5c0 tx=0x7f4af80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:10.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.408+0000 7f4b0882a700 1 -- 172.21.15.33:0/3499177113 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af8003ca0 con 0x7f4b04105760 2024-03-19T02:46:10.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.408+0000 7f4b0882a700 1 -- 172.21.15.33:0/3499177113 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4af8004830 con 0x7f4b04105760 2024-03-19T02:46:10.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.408+0000 7f4b0982c700 1 -- 172.21.15.33:0/3499177113 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b041a25e0 con 0x7f4b04105760 2024-03-19T02:46:10.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.408+0000 7f4b0882a700 1 -- 172.21.15.33:0/3499177113 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af8005480 con 0x7f4b04105760 2024-03-19T02:46:10.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.408+0000 7f4b0982c700 1 -- 172.21.15.33:0/3499177113 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b041a29d0 con 0x7f4b04105760 2024-03-19T02:46:10.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.409+0000 7f4b0982c700 1 -- 172.21.15.33:0/3499177113 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b04068fd0 con 0x7f4b04105760 2024-03-19T02:46:10.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.410+0000 7f4b0882a700 1 -- 172.21.15.33:0/3499177113 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f4af8029060 con 0x7f4b04105760 2024-03-19T02:46:10.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.411+0000 7f4b0882a700 1 --2- 172.21.15.33:0/3499177113 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4aec0a2800 0x7f4aec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:10.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.411+0000 7f4b02ffd700 1 --2- 172.21.15.33:0/3499177113 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4aec0a2800 0x7f4aec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:10.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.411+0000 7f4b0882a700 1 -- 172.21.15.33:0/3499177113 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3226+0+0 (secure 0 0 0) 0x7f4af80ce9f0 con 0x7f4b04105760 2024-03-19T02:46:10.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.412+0000 7f4b02ffd700 1 --2- 172.21.15.33:0/3499177113 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4aec0a2800 0x7f4aec0a4cc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4af4005950 tx=0x7f4af40058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:10.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.415+0000 7f4b0882a700 1 -- 172.21.15.33:0/3499177113 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4af8013070 con 0x7f4b04105760 2024-03-19T02:46:10.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:09 smithi096 ceph-mon[35946]: osdmap e19: 3 total, 3 up, 3 in 2024-03-19T02:46:10.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:09 smithi096 ceph-mon[35946]: pgmap v84: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:10.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:10.629+0000 7f4b0982c700 1 -- 172.21.15.33:0/3499177113 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f4b04062750 con 0x7f4aec0a2800 2024-03-19T02:46:11.253 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:10 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-19T02:46:11.253 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:10 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-19T02:46:11.253 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:10 smithi033 ceph-mon[31973]: osdmap e20: 3 total, 3 up, 3 in 2024-03-19T02:46:11.253 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:10 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-19T02:46:11.253 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:10 smithi033 ceph-mon[31973]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:46:11.253 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:10 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:46:11.253 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:10 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:46:11.253 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:10 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:46:11.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:10 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-19T02:46:11.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:10 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-19T02:46:11.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:10 smithi155 ceph-mon[35804]: osdmap e20: 3 total, 3 up, 3 in 2024-03-19T02:46:11.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:10 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-19T02:46:11.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:10 smithi155 ceph-mon[35804]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:46:11.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:10 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:46:11.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:10 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:46:11.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:10 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:46:11.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:10 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-19T02:46:11.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:10 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-19T02:46:11.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:10 smithi096 ceph-mon[35946]: osdmap e20: 3 total, 3 up, 3 in 2024-03-19T02:46:11.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:10 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-19T02:46:11.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:10 smithi096 ceph-mon[35946]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:46:11.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:10 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:46:11.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:10 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:46:11.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:10 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:46:11.528 INFO:journalctl@ceph.osd.0.smithi033.stdout:Mar 19 02:46:11 smithi033 sudo[51306]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-19T02:46:11.528 INFO:journalctl@ceph.osd.0.smithi033.stdout:Mar 19 02:46:11 smithi033 sudo[51306]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T02:46:11.528 INFO:journalctl@ceph.osd.0.smithi033.stdout:Mar 19 02:46:11 smithi033 sudo[51306]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T02:46:11.528 INFO:journalctl@ceph.osd.0.smithi033.stdout:Mar 19 02:46:11 smithi033 sudo[51306]: pam_unix(sudo:session): session closed for user root 2024-03-19T02:46:12.026 INFO:journalctl@ceph.osd.0.smithi033.stdout:Mar 19 02:46:11 smithi033 sudo[51325]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-19T02:46:12.026 INFO:journalctl@ceph.osd.0.smithi033.stdout:Mar 19 02:46:11 smithi033 sudo[51325]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T02:46:12.026 INFO:journalctl@ceph.osd.0.smithi033.stdout:Mar 19 02:46:11 smithi033 sudo[51325]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T02:46:12.026 INFO:journalctl@ceph.osd.0.smithi033.stdout:Mar 19 02:46:11 smithi033 sudo[51325]: pam_unix(sudo:session): session closed for user root 2024-03-19T02:46:12.156 INFO:journalctl@ceph.osd.1.smithi096.stdout:Mar 19 02:46:11 smithi096 sudo[44257]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-19T02:46:12.156 INFO:journalctl@ceph.osd.1.smithi096.stdout:Mar 19 02:46:11 smithi096 sudo[44257]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T02:46:12.157 INFO:journalctl@ceph.osd.1.smithi096.stdout:Mar 19 02:46:11 smithi096 sudo[44257]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T02:46:12.157 INFO:journalctl@ceph.osd.1.smithi096.stdout:Mar 19 02:46:11 smithi096 sudo[44257]: pam_unix(sudo:session): session closed for user root 2024-03-19T02:46:12.157 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:12 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-19T02:46:12.157 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:12 smithi096 ceph-mon[35946]: osdmap e21: 3 total, 3 up, 3 in 2024-03-19T02:46:12.157 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:12 smithi096 ceph-mon[35946]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:12.157 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:12 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:12.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:12 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-19T02:46:12.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:12 smithi033 ceph-mon[31973]: osdmap e21: 3 total, 3 up, 3 in 2024-03-19T02:46:12.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:12 smithi033 ceph-mon[31973]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:12.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:12 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:12.423 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:12 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-19T02:46:12.423 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:12 smithi155 ceph-mon[35804]: osdmap e21: 3 total, 3 up, 3 in 2024-03-19T02:46:12.423 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:12 smithi155 ceph-mon[35804]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:12.423 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:12 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:12.465 INFO:journalctl@ceph.osd.1.smithi096.stdout:Mar 19 02:46:12 smithi096 sudo[44260]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-19T02:46:12.465 INFO:journalctl@ceph.osd.1.smithi096.stdout:Mar 19 02:46:12 smithi096 sudo[44260]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T02:46:12.465 INFO:journalctl@ceph.osd.1.smithi096.stdout:Mar 19 02:46:12 smithi096 sudo[44260]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T02:46:12.465 INFO:journalctl@ceph.osd.1.smithi096.stdout:Mar 19 02:46:12 smithi096 sudo[44260]: pam_unix(sudo:session): session closed for user root 2024-03-19T02:46:12.680 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 19 02:46:12 smithi155 sudo[44686]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-19T02:46:12.680 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 19 02:46:12 smithi155 sudo[44686]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T02:46:12.680 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 19 02:46:12 smithi155 sudo[44686]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T02:46:12.680 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 19 02:46:12 smithi155 sudo[44686]: pam_unix(sudo:session): session closed for user root 2024-03-19T02:46:13.016 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 19 02:46:12 smithi155 sudo[44689]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-19T02:46:13.016 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 19 02:46:12 smithi155 sudo[44689]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T02:46:13.016 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 19 02:46:12 smithi155 sudo[44689]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T02:46:13.016 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 19 02:46:12 smithi155 sudo[44689]: pam_unix(sudo:session): session closed for user root 2024-03-19T02:46:13.223 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:12 smithi033 sudo[51374]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-19T02:46:13.224 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:12 smithi033 sudo[51374]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T02:46:13.224 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:12 smithi033 sudo[51374]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T02:46:13.224 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:12 smithi033 sudo[51374]: pam_unix(sudo:session): session closed for user root 2024-03-19T02:46:13.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:13.241+0000 7f4b0882a700 1 -- 172.21.15.33:0/3499177113 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f4af8098f50 con 0x7f4b04105760 2024-03-19T02:46:13.487 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:13 smithi155 ceph-mon[35804]: osdmap e22: 3 total, 3 up, 3 in 2024-03-19T02:46:13.487 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:13 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:13.487 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:13 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:13.487 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:13 smithi155 ceph-mon[35804]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T02:46:13.487 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:13 smithi155 sudo[44692]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-19T02:46:13.488 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:13 smithi155 sudo[44692]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T02:46:13.488 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:13 smithi155 sudo[44692]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T02:46:13.488 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:13 smithi155 sudo[44692]: pam_unix(sudo:session): session closed for user root 2024-03-19T02:46:13.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:13 smithi033 sudo[51406]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-19T02:46:13.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:13 smithi033 sudo[51406]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T02:46:13.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:13 smithi033 sudo[51406]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T02:46:13.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:13 smithi033 sudo[51406]: pam_unix(sudo:session): session closed for user root 2024-03-19T02:46:13.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:13 smithi033 ceph-mon[31973]: osdmap e22: 3 total, 3 up, 3 in 2024-03-19T02:46:13.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:13 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:13.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:13 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:13.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:13 smithi033 ceph-mon[31973]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T02:46:13.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:13 smithi096 ceph-mon[35946]: osdmap e22: 3 total, 3 up, 3 in 2024-03-19T02:46:13.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:13 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:13.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:13 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:13.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:13 smithi096 ceph-mon[35946]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T02:46:13.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:13 smithi155 sudo[44695]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-19T02:46:13.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:13 smithi155 sudo[44695]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T02:46:13.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:13 smithi155 sudo[44695]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T02:46:13.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:13 smithi155 sudo[44695]: pam_unix(sudo:session): session closed for user root 2024-03-19T02:46:14.220 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:13 smithi096 sudo[44263]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-19T02:46:14.220 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:13 smithi096 sudo[44263]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T02:46:14.220 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:13 smithi096 sudo[44263]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T02:46:14.220 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:13 smithi096 sudo[44263]: pam_unix(sudo:session): session closed for user root 2024-03-19T02:46:14.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:14 smithi155 ceph-mon[35804]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T02:46:14.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:14 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:46:14.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:14 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:46:14.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:14 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:46:14.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:14 smithi155 ceph-mon[35804]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T02:46:14.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:14 smithi155 ceph-mon[35804]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-19T02:46:14.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:14 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:46:14.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:14 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:46:14.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:14 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:46:14.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:14 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1736670239' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b07e0482-4281-4d32-9dfe-9995d7246175"}]: dispatch 2024-03-19T02:46:14.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:14 smithi155 ceph-mon[35804]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b07e0482-4281-4d32-9dfe-9995d7246175"}]: dispatch 2024-03-19T02:46:14.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:14 smithi155 ceph-mon[35804]: pgmap v89: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:14.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:14 smithi155 ceph-mon[35804]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T02:46:14.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:14 smithi155 ceph-mon[35804]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T02:46:14.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:14 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:46:14.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:14 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:46:14.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:14 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:46:14.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:14 smithi155 ceph-mon[35804]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b07e0482-4281-4d32-9dfe-9995d7246175"}]': finished 2024-03-19T02:46:14.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:14 smithi155 ceph-mon[35804]: osdmap e23: 4 total, 3 up, 4 in 2024-03-19T02:46:14.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:14 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:46:14.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:14 smithi155 ceph-mon[35804]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T02:46:14.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:14 smithi033 ceph-mon[31973]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T02:46:14.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:14 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:46:14.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:14 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:46:14.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:14 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:46:14.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:14 smithi033 ceph-mon[31973]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T02:46:14.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:14 smithi033 ceph-mon[31973]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-19T02:46:14.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:14 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:46:14.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:14 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:46:14.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:14 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:46:14.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:14 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1736670239' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b07e0482-4281-4d32-9dfe-9995d7246175"}]: dispatch 2024-03-19T02:46:14.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:14 smithi033 ceph-mon[31973]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b07e0482-4281-4d32-9dfe-9995d7246175"}]: dispatch 2024-03-19T02:46:14.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:14 smithi033 ceph-mon[31973]: pgmap v89: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:14.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:14 smithi033 ceph-mon[31973]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T02:46:14.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:14 smithi033 ceph-mon[31973]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T02:46:14.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:14 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:46:14.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:14 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:46:14.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:14 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:46:14.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:14 smithi033 ceph-mon[31973]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b07e0482-4281-4d32-9dfe-9995d7246175"}]': finished 2024-03-19T02:46:14.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:14 smithi033 ceph-mon[31973]: osdmap e23: 4 total, 3 up, 4 in 2024-03-19T02:46:14.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:14 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:46:14.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:14 smithi033 ceph-mon[31973]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T02:46:14.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:14 smithi096 sudo[44266]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-19T02:46:14.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:14 smithi096 sudo[44266]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T02:46:14.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:14 smithi096 sudo[44266]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T02:46:14.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:14 smithi096 sudo[44266]: pam_unix(sudo:session): session closed for user root 2024-03-19T02:46:14.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:14 smithi096 ceph-mon[35946]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T02:46:14.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:14 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:46:14.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:14 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:46:14.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:14 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:46:14.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:14 smithi096 ceph-mon[35946]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T02:46:14.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:14 smithi096 ceph-mon[35946]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-19T02:46:14.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:14 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:46:14.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:14 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:46:14.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:14 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:46:14.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:14 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1736670239' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b07e0482-4281-4d32-9dfe-9995d7246175"}]: dispatch 2024-03-19T02:46:14.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:14 smithi096 ceph-mon[35946]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b07e0482-4281-4d32-9dfe-9995d7246175"}]: dispatch 2024-03-19T02:46:14.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:14 smithi096 ceph-mon[35946]: pgmap v89: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:14.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:14 smithi096 ceph-mon[35946]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T02:46:14.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:14 smithi096 ceph-mon[35946]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T02:46:14.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:14 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:46:14.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:14 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:46:14.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:14 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:46:14.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:14 smithi096 ceph-mon[35946]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b07e0482-4281-4d32-9dfe-9995d7246175"}]': finished 2024-03-19T02:46:14.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:14 smithi096 ceph-mon[35946]: osdmap e23: 4 total, 3 up, 4 in 2024-03-19T02:46:14.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:14 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:46:14.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:14 smithi096 ceph-mon[35946]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T02:46:15.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:15 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/183886688' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:46:15.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:15 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/183886688' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:46:15.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:15 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/183886688' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:46:16.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:16 smithi155 ceph-mon[35804]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:16.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:16 smithi033 ceph-mon[31973]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:16.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:16 smithi096 ceph-mon[35946]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:18.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:17 smithi155 ceph-mon[35804]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:18.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:17 smithi033 ceph-mon[31973]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:18.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:17 smithi096 ceph-mon[35946]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:19.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:19 smithi033 ceph-mon[31973]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:20.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:19 smithi155 ceph-mon[35804]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:20.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:19 smithi096 ceph-mon[35946]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:22.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:21 smithi155 ceph-mon[35804]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:22.017 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:21 smithi096 ceph-mon[35946]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:22.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:21 smithi033 ceph-mon[31973]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:23.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:22 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-19T02:46:23.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:22 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:46:23.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:22 smithi155 ceph-mon[35804]: Deploying daemon osd.3 on smithi033 2024-03-19T02:46:23.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:22 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-19T02:46:23.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:22 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:46:23.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:22 smithi033 ceph-mon[31973]: Deploying daemon osd.3 on smithi033 2024-03-19T02:46:23.039 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:22 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-19T02:46:23.039 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:22 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:46:23.039 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:22 smithi096 ceph-mon[35946]: Deploying daemon osd.3 on smithi033 2024-03-19T02:46:24.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:23 smithi155 ceph-mon[35804]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:24.020 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:23 smithi033 ceph-mon[31973]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:24.209 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:23 smithi096 ceph-mon[35946]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:25.999 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:25 smithi033 ceph-mon[31973]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:26.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:25 smithi155 ceph-mon[35804]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:26.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:25 smithi096 ceph-mon[35946]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:28.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:28 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:28.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:28 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:28.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:28 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:46:28.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:28 smithi096 ceph-mon[35946]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:28.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:28 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:28.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:28 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:28.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:28 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:46:28.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:28 smithi155 ceph-mon[35804]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:28 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:28 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:28 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:46:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:28 smithi033 ceph-mon[31973]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:30.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:29 smithi155 ceph-mon[35804]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:30.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:29 smithi033 ceph-mon[31973]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:30.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:29 smithi096 ceph-mon[35946]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:30.842 INFO:teuthology.orchestra.run.smithi033.stdout:Created osd(s) 3 on host 'smithi033' 2024-03-19T02:46:30.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:30.841+0000 7f4b0882a700 1 -- 172.21.15.33:0/3499177113 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4b04062750 con 0x7f4aec0a2800 2024-03-19T02:46:30.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:30.843+0000 7f4b0982c700 1 -- 172.21.15.33:0/3499177113 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4aec0a2800 msgr2=0x7f4aec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:30.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:30.844+0000 7f4b0982c700 1 --2- 172.21.15.33:0/3499177113 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4aec0a2800 0x7f4aec0a4cc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4af4005950 tx=0x7f4af40058e0 comp rx=0 tx=0).stop 2024-03-19T02:46:30.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:30.844+0000 7f4b0982c700 1 -- 172.21.15.33:0/3499177113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b04105760 msgr2=0x7f4b04100de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:30.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:30.844+0000 7f4b0982c700 1 --2- 172.21.15.33:0/3499177113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b04105760 0x7f4b04100de0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4af800b5c0 tx=0x7f4af80047b0 comp rx=0 tx=0).stop 2024-03-19T02:46:30.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:30.844+0000 7f4b0982c700 1 -- 172.21.15.33:0/3499177113 shutdown_connections 2024-03-19T02:46:30.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:30.844+0000 7f4b0982c700 1 --2- 172.21.15.33:0/3499177113 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4aec0a2800 0x7f4aec0a4cc0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:30.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:30.844+0000 7f4b0982c700 1 --2- 172.21.15.33:0/3499177113 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b04108090 0x7f4b04101320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:30.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:30.844+0000 7f4b0982c700 1 --2- 172.21.15.33:0/3499177113 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b04102e30 0x7f4b04102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:30.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:30.844+0000 7f4b0982c700 1 --2- 172.21.15.33:0/3499177113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b04105760 0x7f4b04100de0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:30.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:30.844+0000 7f4b0982c700 1 -- 172.21.15.33:0/3499177113 >> 172.21.15.33:0/3499177113 conn(0x7f4b040fc800 msgr2=0x7f4b04104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:30.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:30.844+0000 7f4b0982c700 1 -- 172.21.15.33:0/3499177113 shutdown_connections 2024-03-19T02:46:30.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:46:30.844+0000 7f4b0982c700 1 -- 172.21.15.33:0/3499177113 wait complete. 2024-03-19T02:46:31.179 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:30 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:31.179 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:30 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:31.179 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:30 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:46:31.179 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:30 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:46:31.179 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:30 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:31.179 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:30 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:31.179 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:30 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:46:31.180 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:30 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:31.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:30 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:31.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:30 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:31.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:30 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:46:31.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:30 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:46:31.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:30 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:31.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:30 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:31.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:30 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:46:31.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:30 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:31.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:30 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:31.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:30 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:31.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:30 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:46:31.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:30 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:46:31.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:30 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:31.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:30 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:31.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:30 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:46:31.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:30 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:31.488 DEBUG:teuthology.orchestra.run.smithi033:osd.3> sudo journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.3.service 2024-03-19T02:46:31.491 INFO:tasks.cephadm:Deploying osd.4 on smithi096 with /dev/vg_nvme/lv_3... 2024-03-19T02:46:31.491 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-19T02:46:31.738 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T02:46:32.186 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:31 smithi096 ceph-mon[35946]: from='osd.3 [v2:172.21.15.33:6810/677007171,v1:172.21.15.33:6811/677007171]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-19T02:46:32.186 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:31 smithi096 ceph-mon[35946]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:32.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:31 smithi155 ceph-mon[35804]: from='osd.3 [v2:172.21.15.33:6810/677007171,v1:172.21.15.33:6811/677007171]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-19T02:46:32.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:31 smithi155 ceph-mon[35804]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:32.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:31 smithi033 ceph-mon[31973]: from='osd.3 [v2:172.21.15.33:6810/677007171,v1:172.21.15.33:6811/677007171]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-19T02:46:32.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:31 smithi033 ceph-mon[31973]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:33.133 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:32 smithi096 ceph-mon[35946]: from='osd.3 [v2:172.21.15.33:6810/677007171,v1:172.21.15.33:6811/677007171]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-19T02:46:33.133 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:32 smithi096 ceph-mon[35946]: osdmap e24: 4 total, 3 up, 4 in 2024-03-19T02:46:33.133 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:32 smithi096 ceph-mon[35946]: from='osd.3 [v2:172.21.15.33:6810/677007171,v1:172.21.15.33:6811/677007171]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-19T02:46:33.133 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:32 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:46:33.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:32 smithi155 ceph-mon[35804]: from='osd.3 [v2:172.21.15.33:6810/677007171,v1:172.21.15.33:6811/677007171]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-19T02:46:33.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:32 smithi155 ceph-mon[35804]: osdmap e24: 4 total, 3 up, 4 in 2024-03-19T02:46:33.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:32 smithi155 ceph-mon[35804]: from='osd.3 [v2:172.21.15.33:6810/677007171,v1:172.21.15.33:6811/677007171]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-19T02:46:33.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:32 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:46:33.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:32 smithi033 ceph-mon[31973]: from='osd.3 [v2:172.21.15.33:6810/677007171,v1:172.21.15.33:6811/677007171]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-19T02:46:33.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:32 smithi033 ceph-mon[31973]: osdmap e24: 4 total, 3 up, 4 in 2024-03-19T02:46:33.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:32 smithi033 ceph-mon[31973]: from='osd.3 [v2:172.21.15.33:6810/677007171,v1:172.21.15.33:6811/677007171]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-19T02:46:33.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:32 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:46:33.279 INFO:journalctl@ceph.osd.3.smithi033.stdout:Mar 19 02:46:32 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-3[53426]: 2024-03-19T02:46:32.977+0000 7f4428cf2700 -1 osd.3 0 waiting for initial osdmap 2024-03-19T02:46:33.279 INFO:journalctl@ceph.osd.3.smithi033.stdout:Mar 19 02:46:32 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-3[53426]: 2024-03-19T02:46:32.982+0000 7f4424ae5700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T02:46:33.694 INFO:teuthology.orchestra.run.smithi096.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-19T02:46:33.694 INFO:teuthology.orchestra.run.smithi096.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-19T02:46:33.694 INFO:teuthology.orchestra.run.smithi096.stderr: stderr: 10+0 records in 2024-03-19T02:46:33.694 INFO:teuthology.orchestra.run.smithi096.stderr:10+0 records out 2024-03-19T02:46:33.695 INFO:teuthology.orchestra.run.smithi096.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114775 s, 914 MB/s 2024-03-19T02:46:33.695 INFO:teuthology.orchestra.run.smithi096.stderr:--> Zapping successful for: 2024-03-19T02:46:34.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:33 smithi155 ceph-mon[35804]: from='osd.3 [v2:172.21.15.33:6810/677007171,v1:172.21.15.33:6811/677007171]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-03-19T02:46:34.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:33 smithi155 ceph-mon[35804]: osdmap e25: 4 total, 3 up, 4 in 2024-03-19T02:46:34.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:33 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:46:34.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:33 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:46:34.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:33 smithi155 ceph-mon[35804]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:34.269 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:33 smithi096 ceph-mon[35946]: from='osd.3 [v2:172.21.15.33:6810/677007171,v1:172.21.15.33:6811/677007171]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-03-19T02:46:34.269 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:33 smithi096 ceph-mon[35946]: osdmap e25: 4 total, 3 up, 4 in 2024-03-19T02:46:34.269 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:33 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:46:34.270 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:33 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:46:34.270 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:33 smithi096 ceph-mon[35946]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:34.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:33 smithi033 ceph-mon[31973]: from='osd.3 [v2:172.21.15.33:6810/677007171,v1:172.21.15.33:6811/677007171]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-03-19T02:46:34.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:33 smithi033 ceph-mon[31973]: osdmap e25: 4 total, 3 up, 4 in 2024-03-19T02:46:34.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:33 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:46:34.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:33 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:46:34.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:33 smithi033 ceph-mon[31973]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-19T02:46:34.341 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi096:vg_nvme/lv_3 2024-03-19T02:46:34.589 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T02:46:35.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:34 smithi096 ceph-mon[35946]: purged_snaps scrub starts 2024-03-19T02:46:35.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:34 smithi096 ceph-mon[35946]: purged_snaps scrub ok 2024-03-19T02:46:35.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:34 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:46:35.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:34 smithi096 ceph-mon[35946]: osd.3 [v2:172.21.15.33:6810/677007171,v1:172.21.15.33:6811/677007171] boot 2024-03-19T02:46:35.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:34 smithi096 ceph-mon[35946]: osdmap e26: 4 total, 4 up, 4 in 2024-03-19T02:46:35.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:34 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:46:35.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:34 smithi096 ceph-mon[35946]: Detected new or changed devices on smithi033 2024-03-19T02:46:35.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:34 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:35.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:34 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:35.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:34 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:46:35.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:34 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:46:35.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:34 smithi096 ceph-mon[35946]: Adjusting osd_memory_target on smithi033 to 8546M 2024-03-19T02:46:35.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:34 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:35.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:34 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:46:35.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:34 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:46:35.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:34 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:35.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:34 smithi096 ceph-mon[35946]: osdmap e27: 4 total, 4 up, 4 in 2024-03-19T02:46:35.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:34 smithi155 ceph-mon[35804]: purged_snaps scrub starts 2024-03-19T02:46:35.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:34 smithi155 ceph-mon[35804]: purged_snaps scrub ok 2024-03-19T02:46:35.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:34 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:46:35.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:34 smithi155 ceph-mon[35804]: osd.3 [v2:172.21.15.33:6810/677007171,v1:172.21.15.33:6811/677007171] boot 2024-03-19T02:46:35.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:34 smithi155 ceph-mon[35804]: osdmap e26: 4 total, 4 up, 4 in 2024-03-19T02:46:35.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:34 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:46:35.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:34 smithi155 ceph-mon[35804]: Detected new or changed devices on smithi033 2024-03-19T02:46:35.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:34 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:35.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:34 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:35.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:34 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:46:35.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:34 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:46:35.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:34 smithi155 ceph-mon[35804]: Adjusting osd_memory_target on smithi033 to 8546M 2024-03-19T02:46:35.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:34 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:35.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:34 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:46:35.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:34 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:46:35.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:34 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:35.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:34 smithi155 ceph-mon[35804]: osdmap e27: 4 total, 4 up, 4 in 2024-03-19T02:46:35.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:34 smithi033 ceph-mon[31973]: purged_snaps scrub starts 2024-03-19T02:46:35.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:34 smithi033 ceph-mon[31973]: purged_snaps scrub ok 2024-03-19T02:46:35.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:34 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:46:35.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:34 smithi033 ceph-mon[31973]: osd.3 [v2:172.21.15.33:6810/677007171,v1:172.21.15.33:6811/677007171] boot 2024-03-19T02:46:35.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:34 smithi033 ceph-mon[31973]: osdmap e26: 4 total, 4 up, 4 in 2024-03-19T02:46:35.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:34 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:46:35.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:34 smithi033 ceph-mon[31973]: Detected new or changed devices on smithi033 2024-03-19T02:46:35.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:34 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:35.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:34 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:35.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:34 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:46:35.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:34 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:46:35.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:34 smithi033 ceph-mon[31973]: Adjusting osd_memory_target on smithi033 to 8546M 2024-03-19T02:46:35.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:34 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:35.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:34 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:46:35.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:34 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:46:35.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:34 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:35.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:34 smithi033 ceph-mon[31973]: osdmap e27: 4 total, 4 up, 4 in 2024-03-19T02:46:36.156 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:35 smithi096 ceph-mon[35946]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:46:36.157 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:35 smithi096 ceph-mon[35946]: osdmap e28: 4 total, 4 up, 4 in 2024-03-19T02:46:36.260 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.258+0000 7faea311c700 1 -- 172.21.15.96:0/867358271 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae9c101270 msgr2=0x7fae9c101690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:36.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.258+0000 7faea311c700 1 --2- 172.21.15.96:0/867358271 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae9c101270 0x7fae9c101690 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fae900099c0 tx=0x7fae90009cd0 comp rx=0 tx=0).stop 2024-03-19T02:46:36.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.258+0000 7faea311c700 1 -- 172.21.15.96:0/867358271 shutdown_connections 2024-03-19T02:46:36.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.258+0000 7faea311c700 1 --2- 172.21.15.96:0/867358271 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae9c102e10 0x7fae9c1096c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:36.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.258+0000 7faea311c700 1 --2- 172.21.15.96:0/867358271 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae9c101270 0x7fae9c101690 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:36.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.258+0000 7faea311c700 1 --2- 172.21.15.96:0/867358271 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae9c102470 0x7fae9c1028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:36.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.258+0000 7faea311c700 1 -- 172.21.15.96:0/867358271 >> 172.21.15.96:0/867358271 conn(0x7fae9c0fc830 msgr2=0x7fae9c0fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:36.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.258+0000 7faea311c700 1 -- 172.21.15.96:0/867358271 shutdown_connections 2024-03-19T02:46:36.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.258+0000 7faea311c700 1 -- 172.21.15.96:0/867358271 wait complete. 2024-03-19T02:46:36.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.259+0000 7faea311c700 1 Processor -- start 2024-03-19T02:46:36.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.259+0000 7faea311c700 1 -- start start 2024-03-19T02:46:36.261 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.259+0000 7faea311c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae9c101270 0x7fae9c19a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:36.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.259+0000 7faea311c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae9c102470 0x7fae9c19ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:36.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.259+0000 7faea311c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae9c102e10 0x7fae9c1a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:36.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.259+0000 7faea311c700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fae9c10cd60 con 0x7fae9c102470 2024-03-19T02:46:36.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.259+0000 7faea311c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fae9c10cbe0 con 0x7fae9c101270 2024-03-19T02:46:36.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.259+0000 7faea311c700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fae9c10cee0 con 0x7fae9c102e10 2024-03-19T02:46:36.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.259+0000 7faea0eb8700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae9c101270 0x7fae9c19a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:36.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.259+0000 7faea0eb8700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae9c101270 0x7fae9c19a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:36580/0 (socket says 172.21.15.96:36580) 2024-03-19T02:46:36.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.259+0000 7faea0eb8700 1 -- 172.21.15.96:0/2232213337 learned_addr learned my addr 172.21.15.96:0/2232213337 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-03-19T02:46:36.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.259+0000 7fae9bfff700 1 --2- 172.21.15.96:0/2232213337 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae9c102470 0x7fae9c19ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:36.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.259+0000 7faea16b9700 1 --2- 172.21.15.96:0/2232213337 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae9c102e10 0x7fae9c1a1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:36.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.259+0000 7faea0eb8700 1 -- 172.21.15.96:0/2232213337 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae9c102e10 msgr2=0x7fae9c1a1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:36.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.259+0000 7faea0eb8700 1 --2- 172.21.15.96:0/2232213337 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae9c102e10 0x7fae9c1a1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:36.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.259+0000 7faea0eb8700 1 -- 172.21.15.96:0/2232213337 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae9c102470 msgr2=0x7fae9c19ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:36.262 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.259+0000 7faea0eb8700 1 --2- 172.21.15.96:0/2232213337 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae9c102470 0x7fae9c19ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:36.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.259+0000 7faea0eb8700 1 -- 172.21.15.96:0/2232213337 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae9c1a2330 con 0x7fae9c101270 2024-03-19T02:46:36.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.259+0000 7faea0eb8700 1 --2- 172.21.15.96:0/2232213337 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae9c101270 0x7fae9c19a6d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fae900098e0 tx=0x7fae900057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:36.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.260+0000 7fae99ffb700 1 -- 172.21.15.96:0/2232213337 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae90027070 con 0x7fae9c101270 2024-03-19T02:46:36.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.260+0000 7fae99ffb700 1 -- 172.21.15.96:0/2232213337 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae90004050 con 0x7fae9c101270 2024-03-19T02:46:36.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.260+0000 7fae99ffb700 1 -- 172.21.15.96:0/2232213337 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae90022dd0 con 0x7fae9c101270 2024-03-19T02:46:36.263 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.260+0000 7faea311c700 1 -- 172.21.15.96:0/2232213337 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae9c1a25c0 con 0x7fae9c101270 2024-03-19T02:46:36.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.260+0000 7faea311c700 1 -- 172.21.15.96:0/2232213337 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae9c1a29c0 con 0x7fae9c101270 2024-03-19T02:46:36.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.261+0000 7faea311c700 1 -- 172.21.15.96:0/2232213337 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae9c068fd0 con 0x7fae9c101270 2024-03-19T02:46:36.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.262+0000 7fae99ffb700 1 -- 172.21.15.96:0/2232213337 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fae9002a070 con 0x7fae9c101270 2024-03-19T02:46:36.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.262+0000 7fae99ffb700 1 --2- 172.21.15.96:0/2232213337 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fae8c0a2890 0x7fae8c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:46:36.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.263+0000 7fae99ffb700 1 -- 172.21.15.96:0/2232213337 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3698+0+0 (secure 0 0 0) 0x7fae900cece0 con 0x7fae9c101270 2024-03-19T02:46:36.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.263+0000 7fae9bfff700 1 --2- 172.21.15.96:0/2232213337 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fae8c0a2890 0x7fae8c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:46:36.264 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.263+0000 7fae9bfff700 1 --2- 172.21.15.96:0/2232213337 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fae8c0a2890 0x7fae8c0a4d50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fae88007c60 tx=0x7fae880073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:46:36.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:35 smithi155 ceph-mon[35804]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:46:36.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:35 smithi155 ceph-mon[35804]: osdmap e28: 4 total, 4 up, 4 in 2024-03-19T02:46:36.267 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.266+0000 7fae99ffb700 1 -- 172.21.15.96:0/2232213337 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fae90031020 con 0x7fae9c101270 2024-03-19T02:46:36.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:35 smithi033 ceph-mon[31973]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:46:36.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:35 smithi033 ceph-mon[31973]: osdmap e28: 4 total, 4 up, 4 in 2024-03-19T02:46:36.471 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:36.470+0000 7faea311c700 1 -- 172.21.15.96:0/2232213337 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fae9c06b160 con 0x7fae8c0a2890 2024-03-19T02:46:37.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:36 smithi155 ceph-mon[35804]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:46:37.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:36 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:46:37.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:36 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:46:37.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:36 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:46:37.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:36 smithi033 ceph-mon[31973]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:46:37.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:36 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:46:37.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:36 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:46:37.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:36 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:46:37.291 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:36 smithi096 ceph-mon[35946]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:46:37.291 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:36 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:46:37.291 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:36 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:46:37.292 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:36 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:46:38.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:38 smithi155 ceph-mon[35804]: osdmap e29: 4 total, 4 up, 4 in 2024-03-19T02:46:38.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:38 smithi155 ceph-mon[35804]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:46:38.274 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:38 smithi096 ceph-mon[35946]: osdmap e29: 4 total, 4 up, 4 in 2024-03-19T02:46:38.274 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:38 smithi096 ceph-mon[35946]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:46:38.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:38 smithi033 ceph-mon[31973]: osdmap e29: 4 total, 4 up, 4 in 2024-03-19T02:46:38.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:38 smithi033 ceph-mon[31973]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:46:39.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:39 smithi155 ceph-mon[35804]: osdmap e30: 4 total, 4 up, 4 in 2024-03-19T02:46:39.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:39 smithi033 ceph-mon[31973]: osdmap e30: 4 total, 4 up, 4 in 2024-03-19T02:46:39.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:39 smithi096 ceph-mon[35946]: osdmap e30: 4 total, 4 up, 4 in 2024-03-19T02:46:40.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:40 smithi155 ceph-mon[35804]: from='client.? 172.21.15.96:0/2357394815' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a988384b-af6b-47c6-8c07-5c9673acee45"}]: dispatch 2024-03-19T02:46:40.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:40 smithi155 ceph-mon[35804]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a988384b-af6b-47c6-8c07-5c9673acee45"}]: dispatch 2024-03-19T02:46:40.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:40 smithi155 ceph-mon[35804]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a988384b-af6b-47c6-8c07-5c9673acee45"}]': finished 2024-03-19T02:46:40.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:40 smithi155 ceph-mon[35804]: osdmap e31: 5 total, 4 up, 5 in 2024-03-19T02:46:40.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:46:40.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:40 smithi155 ceph-mon[35804]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:46:40.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:40 smithi155 ceph-mon[35804]: from='client.? 172.21.15.96:0/2918055493' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:46:40.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:40 smithi033 ceph-mon[31973]: from='client.? 172.21.15.96:0/2357394815' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a988384b-af6b-47c6-8c07-5c9673acee45"}]: dispatch 2024-03-19T02:46:40.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:40 smithi033 ceph-mon[31973]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a988384b-af6b-47c6-8c07-5c9673acee45"}]: dispatch 2024-03-19T02:46:40.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:40 smithi033 ceph-mon[31973]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a988384b-af6b-47c6-8c07-5c9673acee45"}]': finished 2024-03-19T02:46:40.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:40 smithi033 ceph-mon[31973]: osdmap e31: 5 total, 4 up, 5 in 2024-03-19T02:46:40.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:46:40.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:40 smithi033 ceph-mon[31973]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:46:40.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:40 smithi033 ceph-mon[31973]: from='client.? 172.21.15.96:0/2918055493' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:46:40.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:40 smithi096 ceph-mon[35946]: from='client.? 172.21.15.96:0/2357394815' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a988384b-af6b-47c6-8c07-5c9673acee45"}]: dispatch 2024-03-19T02:46:40.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:40 smithi096 ceph-mon[35946]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a988384b-af6b-47c6-8c07-5c9673acee45"}]: dispatch 2024-03-19T02:46:40.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:40 smithi096 ceph-mon[35946]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a988384b-af6b-47c6-8c07-5c9673acee45"}]': finished 2024-03-19T02:46:40.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:40 smithi096 ceph-mon[35946]: osdmap e31: 5 total, 4 up, 5 in 2024-03-19T02:46:40.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:40 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:46:40.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:40 smithi096 ceph-mon[35946]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:46:40.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:40 smithi096 ceph-mon[35946]: from='client.? 172.21.15.96:0/2918055493' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:46:42.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:41 smithi155 ceph-mon[35804]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-19T02:46:42.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:41 smithi033 ceph-mon[31973]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-19T02:46:42.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:41 smithi096 ceph-mon[35946]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-19T02:46:44.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:43 smithi155 ceph-mon[35804]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-19T02:46:44.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:43 smithi033 ceph-mon[31973]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-19T02:46:44.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:43 smithi096 ceph-mon[35946]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-19T02:46:45.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:45 smithi096 ceph-mon[35946]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-19T02:46:46.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:45 smithi155 ceph-mon[35804]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-19T02:46:46.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:45 smithi033 ceph-mon[31973]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-19T02:46:47.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:46 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-19T02:46:47.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:46 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:46:47.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:46 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-19T02:46:47.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:46 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:46:47.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:46 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-19T02:46:47.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:46 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:46:48.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:47 smithi096 ceph-mon[35946]: Deploying daemon osd.4 on smithi096 2024-03-19T02:46:48.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:47 smithi096 ceph-mon[35946]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-19T02:46:48.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:47 smithi155 ceph-mon[35804]: Deploying daemon osd.4 on smithi096 2024-03-19T02:46:48.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:47 smithi155 ceph-mon[35804]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-19T02:46:48.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:47 smithi033 ceph-mon[31973]: Deploying daemon osd.4 on smithi096 2024-03-19T02:46:48.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:47 smithi033 ceph-mon[31973]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-19T02:46:49.824 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:49 smithi096 ceph-mon[35946]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-03-19T02:46:50.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:49 smithi155 ceph-mon[35804]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-03-19T02:46:50.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:49 smithi033 ceph-mon[31973]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-03-19T02:46:51.881 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:51 smithi096 ceph-mon[35946]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-19T02:46:52.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:51 smithi155 ceph-mon[35804]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-19T02:46:52.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:51 smithi033 ceph-mon[31973]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-19T02:46:53.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:53 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:53.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:53 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:53.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:53 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:46:53.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:53 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:53.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:53 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:53.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:53 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:46:53.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:53 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:53.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:53 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:53.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:53 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:46:54.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:54 smithi155 ceph-mon[35804]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:46:54.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:54 smithi033 ceph-mon[31973]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:46:54.864 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:54 smithi096 ceph-mon[35946]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:46:56.339 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:56.339+0000 7fae99ffb700 1 -- 172.21.15.96:0/2232213337 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fae9c06b160 con 0x7fae8c0a2890 2024-03-19T02:46:56.340 INFO:teuthology.orchestra.run.smithi096.stdout:Created osd(s) 4 on host 'smithi096' 2024-03-19T02:46:56.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:56.341+0000 7faea311c700 1 -- 172.21.15.96:0/2232213337 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fae8c0a2890 msgr2=0x7fae8c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:56.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:56.341+0000 7faea311c700 1 --2- 172.21.15.96:0/2232213337 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fae8c0a2890 0x7fae8c0a4d50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fae88007c60 tx=0x7fae880073d0 comp rx=0 tx=0).stop 2024-03-19T02:46:56.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:56.341+0000 7faea311c700 1 -- 172.21.15.96:0/2232213337 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae9c101270 msgr2=0x7fae9c19a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:46:56.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:56.341+0000 7faea311c700 1 --2- 172.21.15.96:0/2232213337 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae9c101270 0x7fae9c19a6d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fae900098e0 tx=0x7fae900057c0 comp rx=0 tx=0).stop 2024-03-19T02:46:56.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:56.341+0000 7faea311c700 1 -- 172.21.15.96:0/2232213337 shutdown_connections 2024-03-19T02:46:56.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:56.341+0000 7faea311c700 1 --2- 172.21.15.96:0/2232213337 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fae8c0a2890 0x7fae8c0a4d50 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:56.342 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:56.341+0000 7faea311c700 1 --2- 172.21.15.96:0/2232213337 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae9c102e10 0x7fae9c1a1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:56.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:56.341+0000 7faea311c700 1 --2- 172.21.15.96:0/2232213337 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae9c101270 0x7fae9c19a6d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:56.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:56.341+0000 7faea311c700 1 --2- 172.21.15.96:0/2232213337 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae9c102470 0x7fae9c19ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:46:56.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:56.341+0000 7faea311c700 1 -- 172.21.15.96:0/2232213337 >> 172.21.15.96:0/2232213337 conn(0x7fae9c0fc830 msgr2=0x7fae9c06aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:46:56.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:56.341+0000 7faea311c700 1 -- 172.21.15.96:0/2232213337 shutdown_connections 2024-03-19T02:46:56.343 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:46:56.342+0000 7faea311c700 1 -- 172.21.15.96:0/2232213337 wait complete. 2024-03-19T02:46:56.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:56 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:56.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:56 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:56.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:56 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:46:56.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:56 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:46:56.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:56 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:56.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:56 smithi096 ceph-mon[35946]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:46:56.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:56 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:56.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:56 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:56.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:56 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:46:56.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:56 smithi096 ceph-mon[35946]: from='osd.4 [v2:172.21.15.96:6808/3106158554,v1:172.21.15.96:6809/3106158554]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-19T02:46:56.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:56 smithi096 ceph-mon[35946]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-19T02:46:56.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:56 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:56.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:56 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:56.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:56 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:46:56.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:56 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:46:56.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:56 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:56.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:56 smithi155 ceph-mon[35804]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:46:56.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:56 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:56.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:56 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:56.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:56 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:46:56.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:56 smithi155 ceph-mon[35804]: from='osd.4 [v2:172.21.15.96:6808/3106158554,v1:172.21.15.96:6809/3106158554]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-19T02:46:56.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:56 smithi155 ceph-mon[35804]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-19T02:46:56.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:56 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:56.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:56 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:56.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:56 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:46:56.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:56 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:46:56.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:56 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:56.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:56 smithi033 ceph-mon[31973]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:46:56.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:56 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:56.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:56 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:46:56.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:56 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:46:56.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:56 smithi033 ceph-mon[31973]: from='osd.4 [v2:172.21.15.96:6808/3106158554,v1:172.21.15.96:6809/3106158554]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-19T02:46:56.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:56 smithi033 ceph-mon[31973]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-19T02:46:57.166 DEBUG:teuthology.orchestra.run.smithi096:osd.4> sudo journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.4.service 2024-03-19T02:46:57.168 INFO:tasks.cephadm:Deploying osd.5 on smithi155 with /dev/vg_nvme/lv_3... 2024-03-19T02:46:57.168 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-19T02:46:57.416 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T02:46:58.655 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:58 smithi155 ceph-mon[35804]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-19T02:46:58.655 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:58 smithi155 ceph-mon[35804]: from='osd.4 [v2:172.21.15.96:6808/3106158554,v1:172.21.15.96:6809/3106158554]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-03-19T02:46:58.655 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:58 smithi155 ceph-mon[35804]: osdmap e32: 5 total, 4 up, 5 in 2024-03-19T02:46:58.655 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:58 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:46:58.655 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:58 smithi155 ceph-mon[35804]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-03-19T02:46:58.656 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:58 smithi155 ceph-mon[35804]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:46:58.695 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:58 smithi096 ceph-mon[35946]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-19T02:46:58.695 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:58 smithi096 ceph-mon[35946]: from='osd.4 [v2:172.21.15.96:6808/3106158554,v1:172.21.15.96:6809/3106158554]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-03-19T02:46:58.695 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:58 smithi096 ceph-mon[35946]: osdmap e32: 5 total, 4 up, 5 in 2024-03-19T02:46:58.695 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:58 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:46:58.695 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:58 smithi096 ceph-mon[35946]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-03-19T02:46:58.695 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:58 smithi096 ceph-mon[35946]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:46:58.695 INFO:journalctl@ceph.osd.4.smithi096.stdout:Mar 19 02:46:58 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-4[47074]: 2024-03-19T02:46:58.325+0000 7f7bebf94700 -1 osd.4 0 waiting for initial osdmap 2024-03-19T02:46:58.696 INFO:journalctl@ceph.osd.4.smithi096.stdout:Mar 19 02:46:58 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-4[47074]: 2024-03-19T02:46:58.334+0000 7f7be7d87700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T02:46:58.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:58 smithi033 ceph-mon[31973]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-19T02:46:58.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:58 smithi033 ceph-mon[31973]: from='osd.4 [v2:172.21.15.96:6808/3106158554,v1:172.21.15.96:6809/3106158554]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-03-19T02:46:58.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:58 smithi033 ceph-mon[31973]: osdmap e32: 5 total, 4 up, 5 in 2024-03-19T02:46:58.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:58 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:46:58.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:58 smithi033 ceph-mon[31973]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-03-19T02:46:58.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:58 smithi033 ceph-mon[31973]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:46:59.377 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-19T02:46:59.378 INFO:teuthology.orchestra.run.smithi155.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-19T02:46:59.378 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10+0 records in 2024-03-19T02:46:59.378 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2024-03-19T02:46:59.378 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113113 s, 927 MB/s 2024-03-19T02:46:59.378 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping successful for: 2024-03-19T02:46:59.708 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:59 smithi155 ceph-mon[35804]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-03-19T02:46:59.708 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:59 smithi155 ceph-mon[35804]: osdmap e33: 5 total, 4 up, 5 in 2024-03-19T02:46:59.708 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:59 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:46:59.708 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:46:59 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:46:59.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:59 smithi096 ceph-mon[35946]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-03-19T02:46:59.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:59 smithi096 ceph-mon[35946]: osdmap e33: 5 total, 4 up, 5 in 2024-03-19T02:46:59.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:59 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:46:59.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:46:59 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:46:59.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:59 smithi033 ceph-mon[31973]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-03-19T02:46:59.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:59 smithi033 ceph-mon[31973]: osdmap e33: 5 total, 4 up, 5 in 2024-03-19T02:46:59.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:59 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:46:59.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:46:59 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:47:00.028 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi155:vg_nvme/lv_3 2024-03-19T02:47:00.265 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T02:47:00.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:00 smithi096 ceph-mon[35946]: purged_snaps scrub starts 2024-03-19T02:47:00.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:00 smithi096 ceph-mon[35946]: purged_snaps scrub ok 2024-03-19T02:47:00.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:00 smithi096 ceph-mon[35946]: osd.4 [v2:172.21.15.96:6808/3106158554,v1:172.21.15.96:6809/3106158554] boot 2024-03-19T02:47:00.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:00 smithi096 ceph-mon[35946]: osdmap e34: 5 total, 5 up, 5 in 2024-03-19T02:47:00.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:00 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:47:00.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:00 smithi096 ceph-mon[35946]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:47:00.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:00 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:00.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:00 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:00.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:00 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:47:00.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:00 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:47:00.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:00 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:00.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:00 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:00.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:00 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:47:00.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:00 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:00.637 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:00 smithi155 ceph-mon[35804]: purged_snaps scrub starts 2024-03-19T02:47:00.637 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:00 smithi155 ceph-mon[35804]: purged_snaps scrub ok 2024-03-19T02:47:00.637 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:00 smithi155 ceph-mon[35804]: osd.4 [v2:172.21.15.96:6808/3106158554,v1:172.21.15.96:6809/3106158554] boot 2024-03-19T02:47:00.637 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:00 smithi155 ceph-mon[35804]: osdmap e34: 5 total, 5 up, 5 in 2024-03-19T02:47:00.637 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:00 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:47:00.638 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:00 smithi155 ceph-mon[35804]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:47:00.638 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:00 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:00.638 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:00 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:00.638 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:00 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:47:00.638 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:00 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:47:00.638 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:00 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:00.638 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:00 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:00.638 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:00 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:47:00.638 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:00 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:00.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:00 smithi033 ceph-mon[31973]: purged_snaps scrub starts 2024-03-19T02:47:00.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:00 smithi033 ceph-mon[31973]: purged_snaps scrub ok 2024-03-19T02:47:00.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:00 smithi033 ceph-mon[31973]: osd.4 [v2:172.21.15.96:6808/3106158554,v1:172.21.15.96:6809/3106158554] boot 2024-03-19T02:47:00.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:00 smithi033 ceph-mon[31973]: osdmap e34: 5 total, 5 up, 5 in 2024-03-19T02:47:00.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:00 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:47:00.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:00 smithi033 ceph-mon[31973]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-19T02:47:00.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:00 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:00.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:00 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:00.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:00 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:47:00.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:00 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:47:00.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:00 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:00.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:00 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:00.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:00 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:47:00.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:00 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:01.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:01 smithi155 ceph-mon[35804]: Detected new or changed devices on smithi096 2024-03-19T02:47:01.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:01 smithi155 ceph-mon[35804]: Adjusting osd_memory_target on smithi096 to 8544M 2024-03-19T02:47:01.615 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:01 smithi155 ceph-mon[35804]: osdmap e35: 5 total, 5 up, 5 in 2024-03-19T02:47:01.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:01 smithi096 ceph-mon[35946]: Detected new or changed devices on smithi096 2024-03-19T02:47:01.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:01 smithi096 ceph-mon[35946]: Adjusting osd_memory_target on smithi096 to 8544M 2024-03-19T02:47:01.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:01 smithi096 ceph-mon[35946]: osdmap e35: 5 total, 5 up, 5 in 2024-03-19T02:47:01.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:01 smithi033 ceph-mon[31973]: Detected new or changed devices on smithi096 2024-03-19T02:47:01.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:01 smithi033 ceph-mon[31973]: Adjusting osd_memory_target on smithi096 to 8544M 2024-03-19T02:47:01.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:01 smithi033 ceph-mon[31973]: osdmap e35: 5 total, 5 up, 5 in 2024-03-19T02:47:01.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.829+0000 7fb3d6c1e700 1 -- 172.21.15.155:0/3933948860 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3d0102440 msgr2=0x7fb3d01028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:01.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.829+0000 7fb3d6c1e700 1 --2- 172.21.15.155:0/3933948860 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3d0102440 0x7fb3d01028a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc009990 tx=0x7fb3bc009ca0 comp rx=0 tx=0).stop 2024-03-19T02:47:01.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.829+0000 7fb3d6c1e700 1 -- 172.21.15.155:0/3933948860 shutdown_connections 2024-03-19T02:47:01.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.829+0000 7fb3d6c1e700 1 --2- 172.21.15.155:0/3933948860 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3d0102de0 0x7fb3d0109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:01.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.829+0000 7fb3d6c1e700 1 --2- 172.21.15.155:0/3933948860 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3d0101240 0x7fb3d0101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:01.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.829+0000 7fb3d6c1e700 1 --2- 172.21.15.155:0/3933948860 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3d0102440 0x7fb3d01028a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:01.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.829+0000 7fb3d6c1e700 1 -- 172.21.15.155:0/3933948860 >> 172.21.15.155:0/3933948860 conn(0x7fb3d00fc7e0 msgr2=0x7fb3d00fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:01.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.829+0000 7fb3d6c1e700 1 -- 172.21.15.155:0/3933948860 shutdown_connections 2024-03-19T02:47:01.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.829+0000 7fb3d6c1e700 1 -- 172.21.15.155:0/3933948860 wait complete. 2024-03-19T02:47:01.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.830+0000 7fb3d6c1e700 1 Processor -- start 2024-03-19T02:47:01.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.830+0000 7fb3d6c1e700 1 -- start start 2024-03-19T02:47:01.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.830+0000 7fb3d6c1e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3d0101240 0x7fb3d0198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:01.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.830+0000 7fb3d6c1e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3d0102440 0x7fb3d0198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:01.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.830+0000 7fb3d6c1e700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3d0102de0 0x7fb3d019fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:01.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.830+0000 7fb3d6c1e700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d010ccd0 con 0x7fb3d0102de0 2024-03-19T02:47:01.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.830+0000 7fb3d6c1e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d010cb50 con 0x7fb3d0101240 2024-03-19T02:47:01.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.830+0000 7fb3d6c1e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d010ce50 con 0x7fb3d0102440 2024-03-19T02:47:01.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.830+0000 7fb3cffff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3d0102440 0x7fb3d0198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:01.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.830+0000 7fb3cffff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3d0102440 0x7fb3d0198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:43478/0 (socket says 172.21.15.155:43478) 2024-03-19T02:47:01.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.830+0000 7fb3cffff700 1 -- 172.21.15.155:0/864389752 learned_addr learned my addr 172.21.15.155:0/864389752 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-19T02:47:01.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.830+0000 7fb3d51bb700 1 --2- 172.21.15.155:0/864389752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3d0102de0 0x7fb3d019fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:01.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.830+0000 7fb3d49ba700 1 --2- 172.21.15.155:0/864389752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3d0101240 0x7fb3d0198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:01.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.830+0000 7fb3cffff700 1 -- 172.21.15.155:0/864389752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3d0101240 msgr2=0x7fb3d0198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:01.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.830+0000 7fb3cffff700 1 --2- 172.21.15.155:0/864389752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3d0101240 0x7fb3d0198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:01.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.830+0000 7fb3cffff700 1 -- 172.21.15.155:0/864389752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3d0102de0 msgr2=0x7fb3d019fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:01.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.830+0000 7fb3cffff700 1 --2- 172.21.15.155:0/864389752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3d0102de0 0x7fb3d019fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:01.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.830+0000 7fb3cffff700 1 -- 172.21.15.155:0/864389752 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3d01a0120 con 0x7fb3d0102440 2024-03-19T02:47:01.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.831+0000 7fb3cffff700 1 --2- 172.21.15.155:0/864389752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3d0102440 0x7fb3d0198a50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc000c00 tx=0x7fb3bc00bb10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:01.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.831+0000 7fb3cdffb700 1 -- 172.21.15.155:0/864389752 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3bc027070 con 0x7fb3d0102440 2024-03-19T02:47:01.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.831+0000 7fb3cdffb700 1 -- 172.21.15.155:0/864389752 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb3bc003d40 con 0x7fb3d0102440 2024-03-19T02:47:01.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.831+0000 7fb3cdffb700 1 -- 172.21.15.155:0/864389752 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3bc00f460 con 0x7fb3d0102440 2024-03-19T02:47:01.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.831+0000 7fb3d6c1e700 1 -- 172.21.15.155:0/864389752 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3d01a03b0 con 0x7fb3d0102440 2024-03-19T02:47:01.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.831+0000 7fb3d6c1e700 1 -- 172.21.15.155:0/864389752 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3d01a07b0 con 0x7fb3d0102440 2024-03-19T02:47:01.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.832+0000 7fb3d6c1e700 1 -- 172.21.15.155:0/864389752 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3d0069000 con 0x7fb3d0102440 2024-03-19T02:47:01.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.833+0000 7fb3cdffb700 1 -- 172.21.15.155:0/864389752 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb3bc02a070 con 0x7fb3d0102440 2024-03-19T02:47:01.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.833+0000 7fb3cdffb700 1 --2- 172.21.15.155:0/864389752 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb3c00a2890 0x7fb3c00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:01.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.834+0000 7fb3cdffb700 1 -- 172.21.15.155:0/864389752 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 4101+0+0 (secure 0 0 0) 0x7fb3bc0ce810 con 0x7fb3d0102440 2024-03-19T02:47:01.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.834+0000 7fb3d49ba700 1 --2- 172.21.15.155:0/864389752 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb3c00a2890 0x7fb3c00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:01.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.834+0000 7fb3d49ba700 1 --2- 172.21.15.155:0/864389752 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb3c00a2890 0x7fb3c00a4d50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb3d01022a0 tx=0x7fb3c4007400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:01.839 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:01.838+0000 7fb3cdffb700 1 -- 172.21.15.155:0/864389752 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb3d0069000 con 0x7fb3d0102440 2024-03-19T02:47:02.043 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:02.042+0000 7fb3d6c1e700 1 -- 172.21.15.155:0/864389752 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fb3d006a700 con 0x7fb3c00a2890 2024-03-19T02:47:02.645 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:02 smithi155 ceph-mon[35804]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:02.645 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:02 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:47:02.645 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:02 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:47:02.645 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:02 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:02.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:02 smithi033 ceph-mon[31973]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:02.679 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:02 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:47:02.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:02 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:47:02.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:02 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:02.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:02 smithi096 ceph-mon[35946]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:02.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:02 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:47:02.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:02 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:47:02.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:02 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:03.663 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:03 smithi155 ceph-mon[35804]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:47:03.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:03 smithi096 ceph-mon[35946]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:47:03.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:03 smithi033 ceph-mon[31973]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:47:04.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:04 smithi096 ceph-mon[35946]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:04.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:04 smithi155 ceph-mon[35804]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:04.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:04 smithi033 ceph-mon[31973]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:05.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:05 smithi096 ceph-mon[35946]: from='client.? 172.21.15.155:0/969839162' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "198dc992-96f8-4c52-96ee-1fd5de148bb8"}]: dispatch 2024-03-19T02:47:05.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:05 smithi096 ceph-mon[35946]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "198dc992-96f8-4c52-96ee-1fd5de148bb8"}]: dispatch 2024-03-19T02:47:05.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:05 smithi096 ceph-mon[35946]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "198dc992-96f8-4c52-96ee-1fd5de148bb8"}]': finished 2024-03-19T02:47:05.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:05 smithi096 ceph-mon[35946]: osdmap e36: 6 total, 5 up, 6 in 2024-03-19T02:47:05.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:05 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:47:05.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:05 smithi096 ceph-mon[35946]: from='client.? 172.21.15.155:0/3880840611' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:47:05.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:05 smithi155 ceph-mon[35804]: from='client.? 172.21.15.155:0/969839162' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "198dc992-96f8-4c52-96ee-1fd5de148bb8"}]: dispatch 2024-03-19T02:47:05.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:05 smithi155 ceph-mon[35804]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "198dc992-96f8-4c52-96ee-1fd5de148bb8"}]: dispatch 2024-03-19T02:47:05.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:05 smithi155 ceph-mon[35804]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "198dc992-96f8-4c52-96ee-1fd5de148bb8"}]': finished 2024-03-19T02:47:05.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:05 smithi155 ceph-mon[35804]: osdmap e36: 6 total, 5 up, 6 in 2024-03-19T02:47:05.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:05 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:47:05.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:05 smithi155 ceph-mon[35804]: from='client.? 172.21.15.155:0/3880840611' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:47:05.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:05 smithi033 ceph-mon[31973]: from='client.? 172.21.15.155:0/969839162' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "198dc992-96f8-4c52-96ee-1fd5de148bb8"}]: dispatch 2024-03-19T02:47:05.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:05 smithi033 ceph-mon[31973]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "198dc992-96f8-4c52-96ee-1fd5de148bb8"}]: dispatch 2024-03-19T02:47:05.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:05 smithi033 ceph-mon[31973]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "198dc992-96f8-4c52-96ee-1fd5de148bb8"}]': finished 2024-03-19T02:47:05.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:05 smithi033 ceph-mon[31973]: osdmap e36: 6 total, 5 up, 6 in 2024-03-19T02:47:05.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:05 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:47:05.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:05 smithi033 ceph-mon[31973]: from='client.? 172.21.15.155:0/3880840611' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:47:06.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:06 smithi096 ceph-mon[35946]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:06.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:06 smithi155 ceph-mon[35804]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:06.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:06 smithi033 ceph-mon[31973]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:08.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:07 smithi155 ceph-mon[35804]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:08.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:07 smithi033 ceph-mon[31973]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:08.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:07 smithi096 ceph-mon[35946]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:10.010 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:09 smithi155 ceph-mon[35804]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:10.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:09 smithi033 ceph-mon[31973]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:10.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:09 smithi096 ceph-mon[35946]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:11.988 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:11 smithi155 ceph-mon[35804]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:12.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:11 smithi033 ceph-mon[31973]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:12.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:11 smithi096 ceph-mon[35946]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:12.940 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:12 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-19T02:47:12.941 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:12 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:12.941 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:12 smithi155 ceph-mon[35804]: Deploying daemon osd.5 on smithi155 2024-03-19T02:47:13.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:12 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-19T02:47:13.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:12 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:13.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:12 smithi033 ceph-mon[31973]: Deploying daemon osd.5 on smithi155 2024-03-19T02:47:13.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:12 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-19T02:47:13.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:12 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:13.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:12 smithi096 ceph-mon[35946]: Deploying daemon osd.5 on smithi155 2024-03-19T02:47:13.877 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:13 smithi155 ceph-mon[35804]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:14.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:13 smithi033 ceph-mon[31973]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:14.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:13 smithi096 ceph-mon[35946]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:15.922 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:15 smithi155 ceph-mon[35804]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:16.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:15 smithi033 ceph-mon[31973]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:16.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:15 smithi096 ceph-mon[35946]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:18.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:18 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:18.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:18 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:18.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:18 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:47:18.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:18 smithi155 ceph-mon[35804]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:18.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:18 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:18.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:18 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:18.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:18 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:47:18.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:18 smithi033 ceph-mon[31973]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:18.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:18 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:18.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:18 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:18.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:18 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:47:18.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:18 smithi096 ceph-mon[35946]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:20.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:19 smithi155 ceph-mon[35804]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:20.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:19 smithi033 ceph-mon[31973]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:20.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:19 smithi096 ceph-mon[35946]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:21.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:21.308+0000 7fb3cdffb700 1 -- 172.21.15.155:0/864389752 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb3d006a700 con 0x7fb3c00a2890 2024-03-19T02:47:21.310 INFO:teuthology.orchestra.run.smithi155.stdout:Created osd(s) 5 on host 'smithi155' 2024-03-19T02:47:21.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:21.311+0000 7fb3d6c1e700 1 -- 172.21.15.155:0/864389752 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb3c00a2890 msgr2=0x7fb3c00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:21.311+0000 7fb3d6c1e700 1 --2- 172.21.15.155:0/864389752 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb3c00a2890 0x7fb3c00a4d50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb3d01022a0 tx=0x7fb3c4007400 comp rx=0 tx=0).stop 2024-03-19T02:47:21.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:21.311+0000 7fb3d6c1e700 1 -- 172.21.15.155:0/864389752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3d0102440 msgr2=0x7fb3d0198a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:21.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:21.311+0000 7fb3d6c1e700 1 --2- 172.21.15.155:0/864389752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3d0102440 0x7fb3d0198a50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc000c00 tx=0x7fb3bc00bb10 comp rx=0 tx=0).stop 2024-03-19T02:47:21.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:21.311+0000 7fb3d6c1e700 1 -- 172.21.15.155:0/864389752 shutdown_connections 2024-03-19T02:47:21.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:21.311+0000 7fb3d6c1e700 1 --2- 172.21.15.155:0/864389752 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb3c00a2890 0x7fb3c00a4d50 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:21.311+0000 7fb3d6c1e700 1 --2- 172.21.15.155:0/864389752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb3d0102de0 0x7fb3d019fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:21.311+0000 7fb3d6c1e700 1 --2- 172.21.15.155:0/864389752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3d0101240 0x7fb3d0198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:21.311+0000 7fb3d6c1e700 1 --2- 172.21.15.155:0/864389752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3d0102440 0x7fb3d0198a50 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:21.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:21.311+0000 7fb3d6c1e700 1 -- 172.21.15.155:0/864389752 >> 172.21.15.155:0/864389752 conn(0x7fb3d00fc7e0 msgr2=0x7fb3d0069ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:21.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:21.311+0000 7fb3d6c1e700 1 -- 172.21.15.155:0/864389752 shutdown_connections 2024-03-19T02:47:21.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:47:21.311+0000 7fb3d6c1e700 1 -- 172.21.15.155:0/864389752 wait complete. 2024-03-19T02:47:21.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:21 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:21.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:21 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:21.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:21 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:21.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:21 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:47:21.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:21 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:21.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:21 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:21.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:21 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:21.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:21 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:21.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:21 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:47:21.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:21 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:21.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:21 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:21.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:21 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:21.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:21 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:21.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:21 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:47:21.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:21 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:22.134 DEBUG:teuthology.orchestra.run.smithi155:osd.5> sudo journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.5.service 2024-03-19T02:47:22.137 INFO:tasks.cephadm:Deploying osd.6 on smithi033 with /dev/vg_nvme/lv_2... 2024-03-19T02:47:22.137 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-19T02:47:22.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:22 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:22.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:22 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:22.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:22 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:47:22.367 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:22 smithi155 ceph-mon[35804]: from='osd.5 [v2:172.21.15.155:6808/327293324,v1:172.21.15.155:6809/327293324]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-19T02:47:22.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:22 smithi155 ceph-mon[35804]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-19T02:47:22.368 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:22 smithi155 ceph-mon[35804]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:22.393 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:47:22.515 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:22 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:22.515 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:22 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:22.515 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:22 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:47:22.515 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:22 smithi033 ceph-mon[31973]: from='osd.5 [v2:172.21.15.155:6808/327293324,v1:172.21.15.155:6809/327293324]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-19T02:47:22.515 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:22 smithi033 ceph-mon[31973]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-19T02:47:22.515 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:22 smithi033 ceph-mon[31973]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:22.617 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:22 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:22.617 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:22 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:22.617 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:22 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:47:22.618 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:22 smithi096 ceph-mon[35946]: from='osd.5 [v2:172.21.15.155:6808/327293324,v1:172.21.15.155:6809/327293324]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-19T02:47:22.618 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:22 smithi096 ceph-mon[35946]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-19T02:47:22.618 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:22 smithi096 ceph-mon[35946]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:23.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:23 smithi155 ceph-mon[35804]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-19T02:47:23.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:23 smithi155 ceph-mon[35804]: from='osd.5 [v2:172.21.15.155:6808/327293324,v1:172.21.15.155:6809/327293324]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-19T02:47:23.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:23 smithi155 ceph-mon[35804]: osdmap e37: 6 total, 5 up, 6 in 2024-03-19T02:47:23.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:23 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:47:23.366 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:23 smithi155 ceph-mon[35804]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-19T02:47:23.367 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 02:47:23 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T02:47:23.287+0000 7f09d958e700 -1 osd.5 0 waiting for initial osdmap 2024-03-19T02:47:23.367 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 02:47:23 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T02:47:23.296+0000 7f09d5381700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T02:47:23.607 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:23 smithi033 ceph-mon[31973]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-19T02:47:23.607 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:23 smithi033 ceph-mon[31973]: from='osd.5 [v2:172.21.15.155:6808/327293324,v1:172.21.15.155:6809/327293324]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-19T02:47:23.607 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:23 smithi033 ceph-mon[31973]: osdmap e37: 6 total, 5 up, 6 in 2024-03-19T02:47:23.607 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:23 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:47:23.608 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:23 smithi033 ceph-mon[31973]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-19T02:47:23.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:23 smithi096 ceph-mon[35946]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-19T02:47:23.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:23 smithi096 ceph-mon[35946]: from='osd.5 [v2:172.21.15.155:6808/327293324,v1:172.21.15.155:6809/327293324]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-19T02:47:23.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:23 smithi096 ceph-mon[35946]: osdmap e37: 6 total, 5 up, 6 in 2024-03-19T02:47:23.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:23 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:47:23.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:23 smithi096 ceph-mon[35946]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-19T02:47:24.558 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:24 smithi033 ceph-mon[31973]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-03-19T02:47:24.558 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:24 smithi033 ceph-mon[31973]: osdmap e38: 6 total, 5 up, 6 in 2024-03-19T02:47:24.558 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:24 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:47:24.558 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:24 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:47:24.558 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:24 smithi033 ceph-mon[31973]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:24.560 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-19T02:47:24.560 INFO:teuthology.orchestra.run.smithi033.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-19T02:47:24.560 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10+0 records in 2024-03-19T02:47:24.560 INFO:teuthology.orchestra.run.smithi033.stderr:10+0 records out 2024-03-19T02:47:24.561 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110711 s, 947 MB/s 2024-03-19T02:47:24.561 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping successful for: 2024-03-19T02:47:24.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:24 smithi096 ceph-mon[35946]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-03-19T02:47:24.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:24 smithi096 ceph-mon[35946]: osdmap e38: 6 total, 5 up, 6 in 2024-03-19T02:47:24.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:24 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:47:24.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:24 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:47:24.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:24 smithi096 ceph-mon[35946]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:24.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:24 smithi155 ceph-mon[35804]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-03-19T02:47:24.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:24 smithi155 ceph-mon[35804]: osdmap e38: 6 total, 5 up, 6 in 2024-03-19T02:47:24.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:24 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:47:24.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:24 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:47:24.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:24 smithi155 ceph-mon[35804]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-19T02:47:25.252 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi033:vg_nvme/lv_2 2024-03-19T02:47:25.504 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:47:25.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:25 smithi155 ceph-mon[35804]: purged_snaps scrub starts 2024-03-19T02:47:25.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:25 smithi155 ceph-mon[35804]: purged_snaps scrub ok 2024-03-19T02:47:25.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:25 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:47:25.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:25 smithi155 ceph-mon[35804]: osd.5 [v2:172.21.15.155:6808/327293324,v1:172.21.15.155:6809/327293324] boot 2024-03-19T02:47:25.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:25 smithi155 ceph-mon[35804]: osdmap e39: 6 total, 6 up, 6 in 2024-03-19T02:47:25.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:25 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:47:25.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:25 smithi155 ceph-mon[35804]: osdmap e40: 6 total, 6 up, 6 in 2024-03-19T02:47:25.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:25 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:25.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:25 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:25.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:25 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:47:25.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:25 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:47:25.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:25 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:25.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:25 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:25.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:25 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:47:25.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:25 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:25.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:25 smithi033 ceph-mon[31973]: purged_snaps scrub starts 2024-03-19T02:47:25.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:25 smithi033 ceph-mon[31973]: purged_snaps scrub ok 2024-03-19T02:47:25.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:25 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:47:25.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:25 smithi033 ceph-mon[31973]: osd.5 [v2:172.21.15.155:6808/327293324,v1:172.21.15.155:6809/327293324] boot 2024-03-19T02:47:25.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:25 smithi033 ceph-mon[31973]: osdmap e39: 6 total, 6 up, 6 in 2024-03-19T02:47:25.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:25 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:47:25.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:25 smithi033 ceph-mon[31973]: osdmap e40: 6 total, 6 up, 6 in 2024-03-19T02:47:25.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:25 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:25.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:25 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:25.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:25 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:47:25.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:25 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:47:25.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:25 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:25.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:25 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:25.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:25 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:47:25.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:25 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:25.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:25 smithi096 ceph-mon[35946]: purged_snaps scrub starts 2024-03-19T02:47:25.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:25 smithi096 ceph-mon[35946]: purged_snaps scrub ok 2024-03-19T02:47:25.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:25 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:47:25.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:25 smithi096 ceph-mon[35946]: osd.5 [v2:172.21.15.155:6808/327293324,v1:172.21.15.155:6809/327293324] boot 2024-03-19T02:47:25.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:25 smithi096 ceph-mon[35946]: osdmap e39: 6 total, 6 up, 6 in 2024-03-19T02:47:25.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:25 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:47:25.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:25 smithi096 ceph-mon[35946]: osdmap e40: 6 total, 6 up, 6 in 2024-03-19T02:47:25.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:25 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:25.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:25 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:25.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:25 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:47:25.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:25 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:47:25.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:25 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:25.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:25 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:25.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:25 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:47:25.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:25 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:26.472 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:26 smithi033 ceph-mon[31973]: Detected new or changed devices on smithi155 2024-03-19T02:47:26.472 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:26 smithi033 ceph-mon[31973]: Adjusting osd_memory_target on smithi155 to 8544M 2024-03-19T02:47:26.472 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:26 smithi033 ceph-mon[31973]: pgmap v143: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:47:26.472 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:26 smithi033 ceph-mon[31973]: osdmap e41: 6 total, 6 up, 6 in 2024-03-19T02:47:26.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:26 smithi096 ceph-mon[35946]: Detected new or changed devices on smithi155 2024-03-19T02:47:26.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:26 smithi096 ceph-mon[35946]: Adjusting osd_memory_target on smithi155 to 8544M 2024-03-19T02:47:26.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:26 smithi096 ceph-mon[35946]: pgmap v143: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:47:26.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:26 smithi096 ceph-mon[35946]: osdmap e41: 6 total, 6 up, 6 in 2024-03-19T02:47:26.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:26 smithi155 ceph-mon[35804]: Detected new or changed devices on smithi155 2024-03-19T02:47:26.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:26 smithi155 ceph-mon[35804]: Adjusting osd_memory_target on smithi155 to 8544M 2024-03-19T02:47:26.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:26 smithi155 ceph-mon[35804]: pgmap v143: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:47:26.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:26 smithi155 ceph-mon[35804]: osdmap e41: 6 total, 6 up, 6 in 2024-03-19T02:47:27.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.008+0000 7f0a1c8b2700 1 -- 172.21.15.33:0/1591192301 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a18101240 msgr2=0x7f0a18101660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:27.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.008+0000 7f0a1c8b2700 1 --2- 172.21.15.33:0/1591192301 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a18101240 0x7f0a18101660 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0a080099c0 tx=0x7f0a08009cd0 comp rx=0 tx=0).stop 2024-03-19T02:47:27.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.009+0000 7f0a1c8b2700 1 -- 172.21.15.33:0/1591192301 shutdown_connections 2024-03-19T02:47:27.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.009+0000 7f0a1c8b2700 1 --2- 172.21.15.33:0/1591192301 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a18102de0 0x7f0a18109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:27.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.009+0000 7f0a1c8b2700 1 --2- 172.21.15.33:0/1591192301 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a18101240 0x7f0a18101660 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:27.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.009+0000 7f0a1c8b2700 1 --2- 172.21.15.33:0/1591192301 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a18102440 0x7f0a181028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:27.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.009+0000 7f0a1c8b2700 1 -- 172.21.15.33:0/1591192301 >> 172.21.15.33:0/1591192301 conn(0x7f0a180fc800 msgr2=0x7f0a180fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:27.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.009+0000 7f0a1c8b2700 1 -- 172.21.15.33:0/1591192301 shutdown_connections 2024-03-19T02:47:27.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.009+0000 7f0a1c8b2700 1 -- 172.21.15.33:0/1591192301 wait complete. 2024-03-19T02:47:27.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.009+0000 7f0a1c8b2700 1 Processor -- start 2024-03-19T02:47:27.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.009+0000 7f0a1c8b2700 1 -- start start 2024-03-19T02:47:27.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.010+0000 7f0a1c8b2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a18101240 0x7f0a181985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:27.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.010+0000 7f0a1c8b2700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a18102440 0x7f0a18198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:27.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.010+0000 7f0a1c8b2700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a18102de0 0x7f0a1819fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:27.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.010+0000 7f0a1c8b2700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0a1810cdb0 con 0x7f0a18102440 2024-03-19T02:47:27.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.010+0000 7f0a1c8b2700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0a1810cc30 con 0x7f0a18101240 2024-03-19T02:47:27.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.010+0000 7f0a1c8b2700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f0a1810cf30 con 0x7f0a18102de0 2024-03-19T02:47:27.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.010+0000 7f0a15d9b700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a18102440 0x7f0a18198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:27.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.010+0000 7f0a15d9b700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a18102440 0x7f0a18198b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:52690/0 (socket says 172.21.15.33:52690) 2024-03-19T02:47:27.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.010+0000 7f0a15d9b700 1 -- 172.21.15.33:0/2012796611 learned_addr learned my addr 172.21.15.33:0/2012796611 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:47:27.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.010+0000 7f0a1659c700 1 --2- 172.21.15.33:0/2012796611 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a18101240 0x7f0a181985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:27.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.010+0000 7f0a16d9d700 1 --2- 172.21.15.33:0/2012796611 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a18102de0 0x7f0a1819fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:27.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.010+0000 7f0a15d9b700 1 -- 172.21.15.33:0/2012796611 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a18102de0 msgr2=0x7f0a1819fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:27.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.010+0000 7f0a15d9b700 1 --2- 172.21.15.33:0/2012796611 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a18102de0 0x7f0a1819fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:27.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.010+0000 7f0a15d9b700 1 -- 172.21.15.33:0/2012796611 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a18101240 msgr2=0x7f0a181985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:27.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.010+0000 7f0a15d9b700 1 --2- 172.21.15.33:0/2012796611 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a18101240 0x7f0a181985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:27.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.010+0000 7f0a15d9b700 1 -- 172.21.15.33:0/2012796611 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a181a0220 con 0x7f0a18102440 2024-03-19T02:47:27.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.010+0000 7f0a1659c700 1 --2- 172.21.15.33:0/2012796611 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a18101240 0x7f0a181985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:47:27.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.010+0000 7f0a15d9b700 1 --2- 172.21.15.33:0/2012796611 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a18102440 0x7f0a18198b00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c00b560 tx=0x7f0a0c00b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:27.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.010+0000 7f0a077fe700 1 -- 172.21.15.33:0/2012796611 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c015020 con 0x7f0a18102440 2024-03-19T02:47:27.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.010+0000 7f0a077fe700 1 -- 172.21.15.33:0/2012796611 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0a0c011ce0 con 0x7f0a18102440 2024-03-19T02:47:27.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.010+0000 7f0a1c8b2700 1 -- 172.21.15.33:0/2012796611 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a181a0510 con 0x7f0a18102440 2024-03-19T02:47:27.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.010+0000 7f0a077fe700 1 -- 172.21.15.33:0/2012796611 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c010da0 con 0x7f0a18102440 2024-03-19T02:47:27.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.011+0000 7f0a1c8b2700 1 -- 172.21.15.33:0/2012796611 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a181a0ad0 con 0x7f0a18102440 2024-03-19T02:47:27.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.012+0000 7f0a1c8b2700 1 -- 172.21.15.33:0/2012796611 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a18068fd0 con 0x7f0a18102440 2024-03-19T02:47:27.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.013+0000 7f0a077fe700 1 -- 172.21.15.33:0/2012796611 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f0a0c018020 con 0x7f0a18102440 2024-03-19T02:47:27.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.013+0000 7f0a077fe700 1 --2- 172.21.15.33:0/2012796611 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f0a000a2890 0x7f0a000a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:27.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.013+0000 7f0a1659c700 1 --2- 172.21.15.33:0/2012796611 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f0a000a2890 0x7f0a000a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:27.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.014+0000 7f0a077fe700 1 -- 172.21.15.33:0/2012796611 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4562+0+0 (secure 0 0 0) 0x7f0a0c0cd430 con 0x7f0a18102440 2024-03-19T02:47:27.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.014+0000 7f0a1659c700 1 --2- 172.21.15.33:0/2012796611 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f0a000a2890 0x7f0a000a4d50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0a08000c00 tx=0x7f0a08005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:27.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.018+0000 7f0a077fe700 1 -- 172.21.15.33:0/2012796611 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0a0c01f020 con 0x7f0a18102440 2024-03-19T02:47:27.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:27.234+0000 7f0a1c8b2700 1 -- 172.21.15.33:0/2012796611 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f0a181a0d90 con 0x7f0a000a2890 2024-03-19T02:47:27.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:27 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:47:27.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:27 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:47:27.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:27 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:27.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:27 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:47:27.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:27 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:47:27.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:27 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:27.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:27 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:47:27.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:27 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:47:27.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:27 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:28 smithi033 ceph-mon[31973]: from='client.14355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:47:28.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:28 smithi033 ceph-mon[31973]: osdmap e42: 6 total, 6 up, 6 in 2024-03-19T02:47:28.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:28 smithi033 ceph-mon[31973]: pgmap v146: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:47:28.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:28 smithi096 ceph-mon[35946]: from='client.14355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:47:28.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:28 smithi096 ceph-mon[35946]: osdmap e42: 6 total, 6 up, 6 in 2024-03-19T02:47:28.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:28 smithi096 ceph-mon[35946]: pgmap v146: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:47:28.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:28 smithi155 ceph-mon[35804]: from='client.14355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:47:28.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:28 smithi155 ceph-mon[35804]: osdmap e42: 6 total, 6 up, 6 in 2024-03-19T02:47:28.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:28 smithi155 ceph-mon[35804]: pgmap v146: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:47:29.428 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:29 smithi033 ceph-mon[31973]: osdmap e43: 6 total, 6 up, 6 in 2024-03-19T02:47:29.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:29 smithi096 ceph-mon[35946]: osdmap e43: 6 total, 6 up, 6 in 2024-03-19T02:47:29.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:29 smithi155 ceph-mon[35804]: osdmap e43: 6 total, 6 up, 6 in 2024-03-19T02:47:30.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:30 smithi096 ceph-mon[35946]: pgmap v148: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:47:30.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:30 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/269559715' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7278ec0-ab2e-49ed-be53-84d8b9e24177"}]: dispatch 2024-03-19T02:47:30.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:30 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/269559715' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a7278ec0-ab2e-49ed-be53-84d8b9e24177"}]': finished 2024-03-19T02:47:30.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:30 smithi096 ceph-mon[35946]: osdmap e44: 7 total, 6 up, 7 in 2024-03-19T02:47:30.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:30 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:47:30.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:30 smithi155 ceph-mon[35804]: pgmap v148: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:47:30.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:30 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/269559715' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7278ec0-ab2e-49ed-be53-84d8b9e24177"}]: dispatch 2024-03-19T02:47:30.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:30 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/269559715' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a7278ec0-ab2e-49ed-be53-84d8b9e24177"}]': finished 2024-03-19T02:47:30.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:30 smithi155 ceph-mon[35804]: osdmap e44: 7 total, 6 up, 7 in 2024-03-19T02:47:30.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:30 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:47:30.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:30 smithi033 ceph-mon[31973]: pgmap v148: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:47:30.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:30 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/269559715' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7278ec0-ab2e-49ed-be53-84d8b9e24177"}]: dispatch 2024-03-19T02:47:30.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:30 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/269559715' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a7278ec0-ab2e-49ed-be53-84d8b9e24177"}]': finished 2024-03-19T02:47:30.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:30 smithi033 ceph-mon[31973]: osdmap e44: 7 total, 6 up, 7 in 2024-03-19T02:47:30.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:30 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:47:31.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:31 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2303206307' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:47:31.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:31 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2303206307' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:47:31.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:31 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2303206307' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:47:32.680 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:32 smithi033 ceph-mon[31973]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-19T02:47:32.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:32 smithi096 ceph-mon[35946]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-19T02:47:32.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:32 smithi155 ceph-mon[35804]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-19T02:47:34.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:33 smithi155 ceph-mon[35804]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-19T02:47:34.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:33 smithi033 ceph-mon[31973]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-19T02:47:34.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:33 smithi096 ceph-mon[35946]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-19T02:47:36.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:35 smithi155 ceph-mon[35804]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-19T02:47:36.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:35 smithi033 ceph-mon[31973]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-19T02:47:36.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:35 smithi096 ceph-mon[35946]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-19T02:47:38.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:37 smithi155 ceph-mon[35804]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-19T02:47:38.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:37 smithi033 ceph-mon[31973]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-19T02:47:38.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:37 smithi096 ceph-mon[35946]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-19T02:47:39.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:38 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-19T02:47:39.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:38 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:39.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:38 smithi155 ceph-mon[35804]: Deploying daemon osd.6 on smithi033 2024-03-19T02:47:39.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:38 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-19T02:47:39.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:38 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:39.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:38 smithi033 ceph-mon[31973]: Deploying daemon osd.6 on smithi033 2024-03-19T02:47:39.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:38 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-19T02:47:39.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:38 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:39.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:38 smithi096 ceph-mon[35946]: Deploying daemon osd.6 on smithi033 2024-03-19T02:47:40.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:39 smithi155 ceph-mon[35804]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-19T02:47:40.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:39 smithi033 ceph-mon[31973]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-19T02:47:40.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:39 smithi096 ceph-mon[35946]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-19T02:47:42.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:41 smithi155 ceph-mon[35804]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:47:42.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:41 smithi033 ceph-mon[31973]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:47:42.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:41 smithi096 ceph-mon[35946]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:47:44.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:44 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:44.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:44 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:44.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:44 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:47:44.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:44 smithi096 ceph-mon[35946]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:47:45.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:44 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:45.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:44 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:45.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:44 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:47:45.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:44 smithi155 ceph-mon[35804]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:47:45.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:44 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:45.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:44 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:45.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:44 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:47:45.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:44 smithi033 ceph-mon[31973]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:47:46.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:45 smithi155 ceph-mon[35804]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:47:46.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:45 smithi033 ceph-mon[31973]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:47:46.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:45 smithi096 ceph-mon[35946]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:47:47.494 INFO:teuthology.orchestra.run.smithi033.stdout:Created osd(s) 6 on host 'smithi033' 2024-03-19T02:47:47.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:47.494+0000 7f0a077fe700 1 -- 172.21.15.33:0/2012796611 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0a181a0d90 con 0x7f0a000a2890 2024-03-19T02:47:47.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:47.495+0000 7f0a1c8b2700 1 -- 172.21.15.33:0/2012796611 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f0a000a2890 msgr2=0x7f0a000a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:47.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:47.495+0000 7f0a1c8b2700 1 --2- 172.21.15.33:0/2012796611 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f0a000a2890 0x7f0a000a4d50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0a08000c00 tx=0x7f0a08005fd0 comp rx=0 tx=0).stop 2024-03-19T02:47:47.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:47.495+0000 7f0a1c8b2700 1 -- 172.21.15.33:0/2012796611 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a18102440 msgr2=0x7f0a18198b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:47.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:47.495+0000 7f0a1c8b2700 1 --2- 172.21.15.33:0/2012796611 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a18102440 0x7f0a18198b00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c00b560 tx=0x7f0a0c00b920 comp rx=0 tx=0).stop 2024-03-19T02:47:47.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:47.495+0000 7f0a1c8b2700 1 -- 172.21.15.33:0/2012796611 shutdown_connections 2024-03-19T02:47:47.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:47.495+0000 7f0a1c8b2700 1 --2- 172.21.15.33:0/2012796611 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f0a000a2890 0x7f0a000a4d50 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:47.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:47.495+0000 7f0a1c8b2700 1 --2- 172.21.15.33:0/2012796611 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a18102de0 0x7f0a1819fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:47.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:47.495+0000 7f0a1c8b2700 1 --2- 172.21.15.33:0/2012796611 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a18101240 0x7f0a181985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:47.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:47.495+0000 7f0a1c8b2700 1 --2- 172.21.15.33:0/2012796611 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a18102440 0x7f0a18198b00 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:47.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:47.496+0000 7f0a1c8b2700 1 -- 172.21.15.33:0/2012796611 >> 172.21.15.33:0/2012796611 conn(0x7f0a180fc800 msgr2=0x7f0a1806aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:47.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:47.496+0000 7f0a1c8b2700 1 -- 172.21.15.33:0/2012796611 shutdown_connections 2024-03-19T02:47:47.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:47:47.496+0000 7f0a1c8b2700 1 -- 172.21.15.33:0/2012796611 wait complete. 2024-03-19T02:47:47.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:47 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:47.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:47 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:47.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:47 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:47.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:47 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:47:47.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:47 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:47.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:47 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:47.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:47 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:47.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:47 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:47:47.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:47 smithi033 ceph-mon[31973]: from='osd.6 [v2:172.21.15.33:6818/873871509,v1:172.21.15.33:6819/873871509]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-19T02:47:47.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:47 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:47.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:47 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:47.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:47 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:47.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:47 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:47:47.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:47 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:47.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:47 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:47.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:47 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:47.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:47 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:47:47.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:47 smithi096 ceph-mon[35946]: from='osd.6 [v2:172.21.15.33:6818/873871509,v1:172.21.15.33:6819/873871509]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-19T02:47:48.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:47 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:48.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:47 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:48.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:47 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:48.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:47 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:47:48.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:47 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:48.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:47 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:48.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:47 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:48.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:47 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:47:48.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:47 smithi155 ceph-mon[35804]: from='osd.6 [v2:172.21.15.33:6818/873871509,v1:172.21.15.33:6819/873871509]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-19T02:47:48.158 DEBUG:teuthology.orchestra.run.smithi033:osd.6> sudo journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.6.service 2024-03-19T02:47:48.161 INFO:tasks.cephadm:Deploying osd.7 on smithi096 with /dev/vg_nvme/lv_2... 2024-03-19T02:47:48.162 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-19T02:47:48.408 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T02:47:48.825 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:48 smithi096 ceph-mon[35946]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:47:48.825 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:48 smithi096 ceph-mon[35946]: from='osd.6 [v2:172.21.15.33:6818/873871509,v1:172.21.15.33:6819/873871509]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-19T02:47:48.825 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:48 smithi096 ceph-mon[35946]: osdmap e45: 7 total, 6 up, 7 in 2024-03-19T02:47:48.825 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:48 smithi096 ceph-mon[35946]: from='osd.6 [v2:172.21.15.33:6818/873871509,v1:172.21.15.33:6819/873871509]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-19T02:47:48.825 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:48 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:47:49.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:48 smithi155 ceph-mon[35804]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:47:49.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:48 smithi155 ceph-mon[35804]: from='osd.6 [v2:172.21.15.33:6818/873871509,v1:172.21.15.33:6819/873871509]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-19T02:47:49.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:48 smithi155 ceph-mon[35804]: osdmap e45: 7 total, 6 up, 7 in 2024-03-19T02:47:49.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:48 smithi155 ceph-mon[35804]: from='osd.6 [v2:172.21.15.33:6818/873871509,v1:172.21.15.33:6819/873871509]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-19T02:47:49.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:48 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:47:49.022 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:48 smithi033 ceph-mon[31973]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:47:49.023 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:48 smithi033 ceph-mon[31973]: from='osd.6 [v2:172.21.15.33:6818/873871509,v1:172.21.15.33:6819/873871509]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-19T02:47:49.023 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:48 smithi033 ceph-mon[31973]: osdmap e45: 7 total, 6 up, 7 in 2024-03-19T02:47:49.023 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:48 smithi033 ceph-mon[31973]: from='osd.6 [v2:172.21.15.33:6818/873871509,v1:172.21.15.33:6819/873871509]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-19T02:47:49.023 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:48 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:47:49.740 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 02:47:49 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6[60108]: 2024-03-19T02:47:49.485+0000 7f163feaa700 -1 osd.6 0 waiting for initial osdmap 2024-03-19T02:47:49.740 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 02:47:49 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6[60108]: 2024-03-19T02:47:49.494+0000 7f163bc9d700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T02:47:50.459 INFO:teuthology.orchestra.run.smithi096.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-19T02:47:50.459 INFO:teuthology.orchestra.run.smithi096.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-19T02:47:50.459 INFO:teuthology.orchestra.run.smithi096.stderr: stderr: 10+0 records in 2024-03-19T02:47:50.459 INFO:teuthology.orchestra.run.smithi096.stderr:10+0 records out 2024-03-19T02:47:50.459 INFO:teuthology.orchestra.run.smithi096.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0115535 s, 908 MB/s 2024-03-19T02:47:50.459 INFO:teuthology.orchestra.run.smithi096.stderr:--> Zapping successful for: 2024-03-19T02:47:50.758 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:50 smithi096 ceph-mon[35946]: from='osd.6 [v2:172.21.15.33:6818/873871509,v1:172.21.15.33:6819/873871509]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-03-19T02:47:50.759 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:50 smithi096 ceph-mon[35946]: osdmap e46: 7 total, 6 up, 7 in 2024-03-19T02:47:50.759 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:50 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:47:50.759 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:50 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:47:50.759 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:50 smithi096 ceph-mon[35946]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:47:50.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:50 smithi155 ceph-mon[35804]: from='osd.6 [v2:172.21.15.33:6818/873871509,v1:172.21.15.33:6819/873871509]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-03-19T02:47:50.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:50 smithi155 ceph-mon[35804]: osdmap e46: 7 total, 6 up, 7 in 2024-03-19T02:47:50.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:50 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:47:50.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:50 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:47:50.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:50 smithi155 ceph-mon[35804]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:47:50.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:50 smithi033 ceph-mon[31973]: from='osd.6 [v2:172.21.15.33:6818/873871509,v1:172.21.15.33:6819/873871509]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-03-19T02:47:50.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:50 smithi033 ceph-mon[31973]: osdmap e46: 7 total, 6 up, 7 in 2024-03-19T02:47:50.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:50 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:47:50.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:50 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:47:50.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:50 smithi033 ceph-mon[31973]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-19T02:47:51.172 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi096:vg_nvme/lv_2 2024-03-19T02:47:51.413 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T02:47:51.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:51 smithi155 ceph-mon[35804]: purged_snaps scrub starts 2024-03-19T02:47:51.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:51 smithi155 ceph-mon[35804]: purged_snaps scrub ok 2024-03-19T02:47:51.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:51 smithi155 ceph-mon[35804]: osd.6 [v2:172.21.15.33:6818/873871509,v1:172.21.15.33:6819/873871509] boot 2024-03-19T02:47:51.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:51 smithi155 ceph-mon[35804]: osdmap e47: 7 total, 7 up, 7 in 2024-03-19T02:47:51.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:51 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:47:51.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:51 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:51.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:51 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:51.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:51 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:47:51.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:51 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:47:51.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:51 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:47:51.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:51 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:51.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:51 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:51.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:51 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:47:51.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:51 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:51.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:51 smithi033 ceph-mon[31973]: purged_snaps scrub starts 2024-03-19T02:47:51.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:51 smithi033 ceph-mon[31973]: purged_snaps scrub ok 2024-03-19T02:47:51.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:51 smithi033 ceph-mon[31973]: osd.6 [v2:172.21.15.33:6818/873871509,v1:172.21.15.33:6819/873871509] boot 2024-03-19T02:47:51.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:51 smithi033 ceph-mon[31973]: osdmap e47: 7 total, 7 up, 7 in 2024-03-19T02:47:51.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:51 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:47:51.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:51 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:51.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:51 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:51.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:51 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:47:51.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:51 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:47:51.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:51 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:47:51.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:51 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:51.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:51 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:51.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:51 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:47:51.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:51 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:51.797 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:51 smithi096 ceph-mon[35946]: purged_snaps scrub starts 2024-03-19T02:47:51.797 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:51 smithi096 ceph-mon[35946]: purged_snaps scrub ok 2024-03-19T02:47:51.797 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:51 smithi096 ceph-mon[35946]: osd.6 [v2:172.21.15.33:6818/873871509,v1:172.21.15.33:6819/873871509] boot 2024-03-19T02:47:51.797 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:51 smithi096 ceph-mon[35946]: osdmap e47: 7 total, 7 up, 7 in 2024-03-19T02:47:51.797 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:51 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:47:51.797 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:51 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:51.797 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:51 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:51.797 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:51 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:47:51.797 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:51 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:47:51.797 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:51 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:47:51.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:51 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:51.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:51 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:51.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:51 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:47:51.798 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:51 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:47:52.761 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:52 smithi096 ceph-mon[35946]: Detected new or changed devices on smithi033 2024-03-19T02:47:52.761 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:52 smithi096 ceph-mon[35946]: Adjusting osd_memory_target on smithi033 to 5697M 2024-03-19T02:47:52.761 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:52 smithi096 ceph-mon[35946]: osdmap e48: 7 total, 7 up, 7 in 2024-03-19T02:47:52.761 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:52 smithi096 ceph-mon[35946]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:47:52.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:52 smithi155 ceph-mon[35804]: Detected new or changed devices on smithi033 2024-03-19T02:47:52.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:52 smithi155 ceph-mon[35804]: Adjusting osd_memory_target on smithi033 to 5697M 2024-03-19T02:47:52.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:52 smithi155 ceph-mon[35804]: osdmap e48: 7 total, 7 up, 7 in 2024-03-19T02:47:52.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:52 smithi155 ceph-mon[35804]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:47:52.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:52 smithi033 ceph-mon[31973]: Detected new or changed devices on smithi033 2024-03-19T02:47:52.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:52 smithi033 ceph-mon[31973]: Adjusting osd_memory_target on smithi033 to 5697M 2024-03-19T02:47:52.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:52 smithi033 ceph-mon[31973]: osdmap e48: 7 total, 7 up, 7 in 2024-03-19T02:47:52.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:52 smithi033 ceph-mon[31973]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:47:53.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.149+0000 7f40433a3700 1 -- 172.21.15.96:0/3270556531 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f403c102440 msgr2=0x7f403c1028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:53.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.149+0000 7f40433a3700 1 --2- 172.21.15.96:0/3270556531 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f403c102440 0x7f403c1028a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f40300099c0 tx=0x7f4030009cd0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.149+0000 7f40433a3700 1 -- 172.21.15.96:0/3270556531 shutdown_connections 2024-03-19T02:47:53.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.149+0000 7f40433a3700 1 --2- 172.21.15.96:0/3270556531 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f403c102de0 0x7f403c109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.152 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.149+0000 7f40433a3700 1 --2- 172.21.15.96:0/3270556531 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f403c101240 0x7f403c101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.149+0000 7f40433a3700 1 --2- 172.21.15.96:0/3270556531 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f403c102440 0x7f403c1028a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.149+0000 7f40433a3700 1 -- 172.21.15.96:0/3270556531 >> 172.21.15.96:0/3270556531 conn(0x7f403c0fc7e0 msgr2=0x7f403c0fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:47:53.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.149+0000 7f40433a3700 1 -- 172.21.15.96:0/3270556531 shutdown_connections 2024-03-19T02:47:53.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.149+0000 7f40433a3700 1 -- 172.21.15.96:0/3270556531 wait complete. 2024-03-19T02:47:53.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.150+0000 7f40433a3700 1 Processor -- start 2024-03-19T02:47:53.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.150+0000 7f40433a3700 1 -- start start 2024-03-19T02:47:53.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.150+0000 7f40433a3700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f403c101240 0x7f403c198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.150+0000 7f40433a3700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f403c102440 0x7f403c198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.150+0000 7f40433a3700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f403c102de0 0x7f403c19faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.150+0000 7f40433a3700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f403c10cd40 con 0x7f403c101240 2024-03-19T02:47:53.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.150+0000 7f40433a3700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f403c10cbc0 con 0x7f403c102440 2024-03-19T02:47:53.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.150+0000 7f40433a3700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f403c10cec0 con 0x7f403c102de0 2024-03-19T02:47:53.153 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.150+0000 7f404093e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f403c102440 0x7f403c198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.150+0000 7f404093e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f403c102440 0x7f403c198a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:56708/0 (socket says 172.21.15.96:56708) 2024-03-19T02:47:53.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.150+0000 7f404113f700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f403c101240 0x7f403c198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.150+0000 7f404093e700 1 -- 172.21.15.96:0/2735121129 learned_addr learned my addr 172.21.15.96:0/2735121129 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-03-19T02:47:53.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.150+0000 7f4041940700 1 --2- 172.21.15.96:0/2735121129 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f403c102de0 0x7f403c19faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.150+0000 7f404093e700 1 -- 172.21.15.96:0/2735121129 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f403c102de0 msgr2=0x7f403c19faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:53.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.150+0000 7f404093e700 1 --2- 172.21.15.96:0/2735121129 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f403c102de0 0x7f403c19faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.150+0000 7f404093e700 1 -- 172.21.15.96:0/2735121129 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f403c101240 msgr2=0x7f403c198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:47:53.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.150+0000 7f404093e700 1 --2- 172.21.15.96:0/2735121129 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f403c101240 0x7f403c198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:47:53.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.150+0000 7f404093e700 1 -- 172.21.15.96:0/2735121129 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f403c1a0190 con 0x7f403c102440 2024-03-19T02:47:53.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.150+0000 7f4041940700 1 --2- 172.21.15.96:0/2735121129 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f403c102de0 0x7f403c19faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:47:53.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.150+0000 7f404093e700 1 --2- 172.21.15.96:0/2735121129 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f403c102440 0x7f403c198a70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4030006010 tx=0x7f40300047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:53.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.151+0000 7f403a7fc700 1 -- 172.21.15.96:0/2735121129 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4030027070 con 0x7f403c102440 2024-03-19T02:47:53.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.151+0000 7f403a7fc700 1 -- 172.21.15.96:0/2735121129 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4030004050 con 0x7f403c102440 2024-03-19T02:47:53.154 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.151+0000 7f40433a3700 1 -- 172.21.15.96:0/2735121129 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f403c1a0420 con 0x7f403c102440 2024-03-19T02:47:53.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.151+0000 7f403a7fc700 1 -- 172.21.15.96:0/2735121129 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4030022d60 con 0x7f403c102440 2024-03-19T02:47:53.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.151+0000 7f40433a3700 1 -- 172.21.15.96:0/2735121129 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f403c1a08f0 con 0x7f403c102440 2024-03-19T02:47:53.155 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.152+0000 7f40433a3700 1 -- 172.21.15.96:0/2735121129 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f403c068fd0 con 0x7f403c102440 2024-03-19T02:47:53.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.153+0000 7f403a7fc700 1 -- 172.21.15.96:0/2735121129 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f403002a070 con 0x7f403c102440 2024-03-19T02:47:53.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.153+0000 7f403a7fc700 1 --2- 172.21.15.96:0/2735121129 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f40240a2890 0x7f40240a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:47:53.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.154+0000 7f403a7fc700 1 -- 172.21.15.96:0/2735121129 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4965+0+0 (secure 0 0 0) 0x7f40300cf070 con 0x7f403c102440 2024-03-19T02:47:53.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.154+0000 7f404113f700 1 --2- 172.21.15.96:0/2735121129 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f40240a2890 0x7f40240a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:47:53.156 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.154+0000 7f404113f700 1 --2- 172.21.15.96:0/2735121129 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f40240a2890 0x7f40240a4d50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f402c009550 tx=0x7f402c0091b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:47:53.157 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.157+0000 7f403a7fc700 1 -- 172.21.15.96:0/2735121129 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4030031070 con 0x7f403c102440 2024-03-19T02:47:53.361 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:47:53.360+0000 7f40433a3700 1 -- 172.21.15.96:0/2735121129 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f403c06a6d0 con 0x7f40240a2890 2024-03-19T02:47:53.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:53 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:47:53.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:53 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:47:53.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:53 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:53.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:53 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:47:53.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:53 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:47:53.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:53 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:53.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:53 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:47:53.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:53 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:47:53.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:53 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:47:54.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:54 smithi155 ceph-mon[35804]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:47:54.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:54 smithi155 ceph-mon[35804]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:47:54.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:54 smithi033 ceph-mon[31973]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:47:54.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:54 smithi033 ceph-mon[31973]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:47:54.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:54 smithi096 ceph-mon[35946]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:47:54.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:54 smithi096 ceph-mon[35946]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:47:55.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:55 smithi096 ceph-mon[35946]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:47:56.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:55 smithi155 ceph-mon[35804]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:47:56.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:55 smithi033 ceph-mon[31973]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:47:56.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:56 smithi096 ceph-mon[35946]: from='client.? 172.21.15.96:0/3034672169' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "727e133d-aa24-4ffc-90e2-266eb2aab8c5"}]: dispatch 2024-03-19T02:47:56.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:56 smithi096 ceph-mon[35946]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "727e133d-aa24-4ffc-90e2-266eb2aab8c5"}]: dispatch 2024-03-19T02:47:56.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:56 smithi096 ceph-mon[35946]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "727e133d-aa24-4ffc-90e2-266eb2aab8c5"}]': finished 2024-03-19T02:47:56.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:56 smithi096 ceph-mon[35946]: osdmap e49: 8 total, 7 up, 8 in 2024-03-19T02:47:56.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:56 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:47:57.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:56 smithi155 ceph-mon[35804]: from='client.? 172.21.15.96:0/3034672169' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "727e133d-aa24-4ffc-90e2-266eb2aab8c5"}]: dispatch 2024-03-19T02:47:57.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:56 smithi155 ceph-mon[35804]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "727e133d-aa24-4ffc-90e2-266eb2aab8c5"}]: dispatch 2024-03-19T02:47:57.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:56 smithi155 ceph-mon[35804]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "727e133d-aa24-4ffc-90e2-266eb2aab8c5"}]': finished 2024-03-19T02:47:57.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:56 smithi155 ceph-mon[35804]: osdmap e49: 8 total, 7 up, 8 in 2024-03-19T02:47:57.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:56 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:47:57.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:56 smithi033 ceph-mon[31973]: from='client.? 172.21.15.96:0/3034672169' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "727e133d-aa24-4ffc-90e2-266eb2aab8c5"}]: dispatch 2024-03-19T02:47:57.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:56 smithi033 ceph-mon[31973]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "727e133d-aa24-4ffc-90e2-266eb2aab8c5"}]: dispatch 2024-03-19T02:47:57.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:56 smithi033 ceph-mon[31973]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "727e133d-aa24-4ffc-90e2-266eb2aab8c5"}]': finished 2024-03-19T02:47:57.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:56 smithi033 ceph-mon[31973]: osdmap e49: 8 total, 7 up, 8 in 2024-03-19T02:47:57.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:56 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:47:58.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:57 smithi155 ceph-mon[35804]: from='client.? 172.21.15.96:0/3377999478' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:47:58.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:57 smithi155 ceph-mon[35804]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:47:58.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:57 smithi033 ceph-mon[31973]: from='client.? 172.21.15.96:0/3377999478' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:47:58.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:57 smithi033 ceph-mon[31973]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:47:58.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:57 smithi096 ceph-mon[35946]: from='client.? 172.21.15.96:0/3377999478' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:47:58.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:57 smithi096 ceph-mon[35946]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:48:00.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:47:59 smithi155 ceph-mon[35804]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:48:00.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:47:59 smithi033 ceph-mon[31973]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:48:00.202 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:47:59 smithi096 ceph-mon[35946]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:48:02.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:01 smithi155 ceph-mon[35804]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:48:02.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:01 smithi033 ceph-mon[31973]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:48:02.202 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:01 smithi096 ceph-mon[35946]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:48:04.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:03 smithi155 ceph-mon[35804]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:48:04.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:03 smithi033 ceph-mon[31973]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:48:04.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:03 smithi096 ceph-mon[35946]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:48:05.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:04 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-19T02:48:05.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:04 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:48:05.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:04 smithi155 ceph-mon[35804]: Deploying daemon osd.7 on smithi096 2024-03-19T02:48:05.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:04 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-19T02:48:05.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:04 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:48:05.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:04 smithi033 ceph-mon[31973]: Deploying daemon osd.7 on smithi096 2024-03-19T02:48:05.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:04 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-19T02:48:05.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:04 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:48:05.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:04 smithi096 ceph-mon[35946]: Deploying daemon osd.7 on smithi096 2024-03-19T02:48:06.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:05 smithi155 ceph-mon[35804]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:48:06.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:05 smithi033 ceph-mon[31973]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:48:06.103 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:05 smithi096 ceph-mon[35946]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:48:07.786 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:07 smithi096 ceph-mon[35946]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:48:08.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:07 smithi155 ceph-mon[35804]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:48:08.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:07 smithi033 ceph-mon[31973]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:48:10.951 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:10 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:10.952 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:10 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:10.952 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:10 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:48:10.952 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:10 smithi096 ceph-mon[35946]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:48:11.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:10 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:11.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:10 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:11.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:10 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:48:11.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:10 smithi155 ceph-mon[35804]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:48:11.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:10 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:11.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:10 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:11.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:10 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:48:11.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:10 smithi033 ceph-mon[31973]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:48:11.856 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:11 smithi096 ceph-mon[35946]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:48:12.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:11 smithi155 ceph-mon[35804]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:48:12.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:11 smithi033 ceph-mon[31973]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:48:13.531 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:48:13.530+0000 7f403a7fc700 1 -- 172.21.15.96:0/2735121129 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f403c06a6d0 con 0x7f40240a2890 2024-03-19T02:48:13.531 INFO:teuthology.orchestra.run.smithi096.stdout:Created osd(s) 7 on host 'smithi096' 2024-03-19T02:48:13.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:48:13.532+0000 7f40433a3700 1 -- 172.21.15.96:0/2735121129 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f40240a2890 msgr2=0x7f40240a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:13.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:48:13.532+0000 7f40433a3700 1 --2- 172.21.15.96:0/2735121129 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f40240a2890 0x7f40240a4d50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f402c009550 tx=0x7f402c0091b0 comp rx=0 tx=0).stop 2024-03-19T02:48:13.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:48:13.532+0000 7f40433a3700 1 -- 172.21.15.96:0/2735121129 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f403c102440 msgr2=0x7f403c198a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:13.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:48:13.532+0000 7f40433a3700 1 --2- 172.21.15.96:0/2735121129 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f403c102440 0x7f403c198a70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4030006010 tx=0x7f40300047b0 comp rx=0 tx=0).stop 2024-03-19T02:48:13.533 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:48:13.532+0000 7f40433a3700 1 -- 172.21.15.96:0/2735121129 shutdown_connections 2024-03-19T02:48:13.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:48:13.532+0000 7f40433a3700 1 --2- 172.21.15.96:0/2735121129 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f40240a2890 0x7f40240a4d50 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:13.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:48:13.532+0000 7f40433a3700 1 --2- 172.21.15.96:0/2735121129 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f403c102de0 0x7f403c19faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:13.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:48:13.532+0000 7f40433a3700 1 --2- 172.21.15.96:0/2735121129 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f403c101240 0x7f403c198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:13.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:48:13.532+0000 7f40433a3700 1 --2- 172.21.15.96:0/2735121129 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f403c102440 0x7f403c198a70 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:13.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:48:13.532+0000 7f40433a3700 1 -- 172.21.15.96:0/2735121129 >> 172.21.15.96:0/2735121129 conn(0x7f403c0fc7e0 msgr2=0x7f403c069fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:13.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:48:13.532+0000 7f40433a3700 1 -- 172.21.15.96:0/2735121129 shutdown_connections 2024-03-19T02:48:13.534 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:48:13.532+0000 7f40433a3700 1 -- 172.21.15.96:0/2735121129 wait complete. 2024-03-19T02:48:13.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:13 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:13.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:13 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:13.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:13 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:48:13.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:13 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:48:13.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:13 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:13.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:13 smithi096 ceph-mon[35946]: from='osd.7 [v2:172.21.15.96:6816/2334123891,v1:172.21.15.96:6817/2334123891]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-19T02:48:13.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:13 smithi096 ceph-mon[35946]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-19T02:48:13.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:13 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:13.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:13 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:13.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:13 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:48:14.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:13 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:14.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:13 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:14.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:13 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:48:14.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:13 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:48:14.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:13 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:14.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:13 smithi155 ceph-mon[35804]: from='osd.7 [v2:172.21.15.96:6816/2334123891,v1:172.21.15.96:6817/2334123891]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-19T02:48:14.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:13 smithi155 ceph-mon[35804]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-19T02:48:14.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:13 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:14.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:13 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:14.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:13 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:48:14.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:13 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:14.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:13 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:14.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:13 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:48:14.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:13 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:48:14.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:13 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:14.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:13 smithi033 ceph-mon[31973]: from='osd.7 [v2:172.21.15.96:6816/2334123891,v1:172.21.15.96:6817/2334123891]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-19T02:48:14.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:13 smithi033 ceph-mon[31973]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-19T02:48:14.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:13 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:14.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:13 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:14.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:13 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:48:14.231 DEBUG:teuthology.orchestra.run.smithi096:osd.7> sudo journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.7.service 2024-03-19T02:48:14.233 INFO:tasks.cephadm:Deploying osd.8 on smithi155 with /dev/vg_nvme/lv_2... 2024-03-19T02:48:14.233 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-19T02:48:14.481 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T02:48:14.955 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:14 smithi155 ceph-mon[35804]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-19T02:48:14.955 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:14 smithi155 ceph-mon[35804]: osdmap e50: 8 total, 7 up, 8 in 2024-03-19T02:48:14.955 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:14 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:48:14.955 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:14 smithi155 ceph-mon[35804]: from='osd.7 [v2:172.21.15.96:6816/2334123891,v1:172.21.15.96:6817/2334123891]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-03-19T02:48:14.955 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:14 smithi155 ceph-mon[35804]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-03-19T02:48:14.956 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:14 smithi155 ceph-mon[35804]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:48:14.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:14 smithi096 ceph-mon[35946]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-19T02:48:14.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:14 smithi096 ceph-mon[35946]: osdmap e50: 8 total, 7 up, 8 in 2024-03-19T02:48:14.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:14 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:48:14.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:14 smithi096 ceph-mon[35946]: from='osd.7 [v2:172.21.15.96:6816/2334123891,v1:172.21.15.96:6817/2334123891]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-03-19T02:48:14.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:14 smithi096 ceph-mon[35946]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-03-19T02:48:14.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:14 smithi096 ceph-mon[35946]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:48:15.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:14 smithi033 ceph-mon[31973]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-19T02:48:15.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:14 smithi033 ceph-mon[31973]: osdmap e50: 8 total, 7 up, 8 in 2024-03-19T02:48:15.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:14 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:48:15.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:14 smithi033 ceph-mon[31973]: from='osd.7 [v2:172.21.15.96:6816/2334123891,v1:172.21.15.96:6817/2334123891]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-03-19T02:48:15.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:14 smithi033 ceph-mon[31973]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-03-19T02:48:15.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:14 smithi033 ceph-mon[31973]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:48:15.590 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 02:48:15 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T02:48:15.278+0000 7f248e7af700 -1 osd.7 0 waiting for initial osdmap 2024-03-19T02:48:15.591 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 02:48:15 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T02:48:15.287+0000 7f248659a700 -1 osd.7 52 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T02:48:15.872 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:15 smithi155 ceph-mon[35804]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-03-19T02:48:15.873 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:15 smithi155 ceph-mon[35804]: osdmap e51: 8 total, 7 up, 8 in 2024-03-19T02:48:15.873 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:15 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:48:15.873 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:15 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:48:15.873 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:15 smithi155 ceph-mon[35804]: osdmap e52: 8 total, 7 up, 8 in 2024-03-19T02:48:15.873 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:15 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:48:15.873 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:15 smithi155 ceph-mon[35804]: from='osd.7 ' entity='osd.7' 2024-03-19T02:48:15.938 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:15 smithi096 ceph-mon[35946]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-03-19T02:48:15.938 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:15 smithi096 ceph-mon[35946]: osdmap e51: 8 total, 7 up, 8 in 2024-03-19T02:48:15.938 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:15 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:48:15.938 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:15 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:48:15.938 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:15 smithi096 ceph-mon[35946]: osdmap e52: 8 total, 7 up, 8 in 2024-03-19T02:48:15.938 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:15 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:48:15.938 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:15 smithi096 ceph-mon[35946]: from='osd.7 ' entity='osd.7' 2024-03-19T02:48:16.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:15 smithi033 ceph-mon[31973]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-03-19T02:48:16.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:15 smithi033 ceph-mon[31973]: osdmap e51: 8 total, 7 up, 8 in 2024-03-19T02:48:16.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:15 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:48:16.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:15 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:48:16.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:15 smithi033 ceph-mon[31973]: osdmap e52: 8 total, 7 up, 8 in 2024-03-19T02:48:16.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:15 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:48:16.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:15 smithi033 ceph-mon[31973]: from='osd.7 ' entity='osd.7' 2024-03-19T02:48:16.711 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-19T02:48:16.712 INFO:teuthology.orchestra.run.smithi155.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-19T02:48:16.712 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10+0 records in 2024-03-19T02:48:16.712 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2024-03-19T02:48:16.712 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115841 s, 905 MB/s 2024-03-19T02:48:16.712 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping successful for: 2024-03-19T02:48:16.712 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:16 smithi155 ceph-mon[35804]: purged_snaps scrub starts 2024-03-19T02:48:16.712 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:16 smithi155 ceph-mon[35804]: purged_snaps scrub ok 2024-03-19T02:48:16.712 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:16 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:48:16.712 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:16 smithi155 ceph-mon[35804]: pgmap v180: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:48:16.712 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:16 smithi155 ceph-mon[35804]: osd.7 [v2:172.21.15.96:6816/2334123891,v1:172.21.15.96:6817/2334123891] boot 2024-03-19T02:48:16.712 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:16 smithi155 ceph-mon[35804]: osdmap e53: 8 total, 8 up, 8 in 2024-03-19T02:48:16.712 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:16 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:48:16.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:16 smithi096 ceph-mon[35946]: purged_snaps scrub starts 2024-03-19T02:48:16.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:16 smithi096 ceph-mon[35946]: purged_snaps scrub ok 2024-03-19T02:48:16.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:16 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:48:16.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:16 smithi096 ceph-mon[35946]: pgmap v180: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:48:16.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:16 smithi096 ceph-mon[35946]: osd.7 [v2:172.21.15.96:6816/2334123891,v1:172.21.15.96:6817/2334123891] boot 2024-03-19T02:48:16.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:16 smithi096 ceph-mon[35946]: osdmap e53: 8 total, 8 up, 8 in 2024-03-19T02:48:16.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:16 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:48:17.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:16 smithi033 ceph-mon[31973]: purged_snaps scrub starts 2024-03-19T02:48:17.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:16 smithi033 ceph-mon[31973]: purged_snaps scrub ok 2024-03-19T02:48:17.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:16 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:48:17.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:16 smithi033 ceph-mon[31973]: pgmap v180: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-19T02:48:17.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:16 smithi033 ceph-mon[31973]: osd.7 [v2:172.21.15.96:6816/2334123891,v1:172.21.15.96:6817/2334123891] boot 2024-03-19T02:48:17.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:16 smithi033 ceph-mon[31973]: osdmap e53: 8 total, 8 up, 8 in 2024-03-19T02:48:17.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:16 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:48:17.426 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi155:vg_nvme/lv_2 2024-03-19T02:48:17.667 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T02:48:18.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:17 smithi155 ceph-mon[35804]: osdmap e54: 8 total, 8 up, 8 in 2024-03-19T02:48:18.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:17 smithi155 ceph-mon[35804]: Detected new or changed devices on smithi096 2024-03-19T02:48:18.119 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:17 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:18.120 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:17 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:18.120 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:17 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:18.120 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:17 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:18.120 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:17 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:18.120 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:17 smithi155 ceph-mon[35804]: Adjusting osd_memory_target on smithi096 to 5696M 2024-03-19T02:48:18.120 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:17 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:18.120 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:17 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:48:18.120 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:17 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:48:18.120 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:17 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:18.120 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:17 smithi155 ceph-mon[35804]: pgmap v183: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:18.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:17 smithi033 ceph-mon[31973]: osdmap e54: 8 total, 8 up, 8 in 2024-03-19T02:48:18.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:17 smithi033 ceph-mon[31973]: Detected new or changed devices on smithi096 2024-03-19T02:48:18.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:17 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:18.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:17 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:18.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:17 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:18.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:17 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:18.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:17 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:18.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:17 smithi033 ceph-mon[31973]: Adjusting osd_memory_target on smithi096 to 5696M 2024-03-19T02:48:18.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:17 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:18.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:17 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:48:18.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:17 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:48:18.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:17 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:18.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:17 smithi033 ceph-mon[31973]: pgmap v183: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:18.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:17 smithi096 ceph-mon[35946]: osdmap e54: 8 total, 8 up, 8 in 2024-03-19T02:48:18.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:17 smithi096 ceph-mon[35946]: Detected new or changed devices on smithi096 2024-03-19T02:48:18.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:17 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:18.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:17 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:18.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:17 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:18.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:17 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:18.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:17 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:18.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:17 smithi096 ceph-mon[35946]: Adjusting osd_memory_target on smithi096 to 5696M 2024-03-19T02:48:18.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:17 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:18.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:17 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:48:18.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:17 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:48:18.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:17 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:18.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:17 smithi096 ceph-mon[35946]: pgmap v183: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:19.131 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:19 smithi155 ceph-mon[35804]: osdmap e55: 8 total, 8 up, 8 in 2024-03-19T02:48:19.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:19 smithi033 ceph-mon[31973]: osdmap e55: 8 total, 8 up, 8 in 2024-03-19T02:48:19.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.365+0000 7f55bfe9e700 1 -- 172.21.15.155:0/3231430305 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f55b8102e20 msgr2=0x7f55b8105210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:19.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.365+0000 7f55bfe9e700 1 --2- 172.21.15.155:0/3231430305 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f55b8102e20 0x7f55b8105210 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f55b4009990 tx=0x7f55b4009ca0 comp rx=0 tx=0).stop 2024-03-19T02:48:19.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.365+0000 7f55bfe9e700 1 -- 172.21.15.155:0/3231430305 shutdown_connections 2024-03-19T02:48:19.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.365+0000 7f55bfe9e700 1 --2- 172.21.15.155:0/3231430305 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f55b8108080 0x7f55b810a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:19.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.365+0000 7f55bfe9e700 1 --2- 172.21.15.155:0/3231430305 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f55b8102e20 0x7f55b8105210 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:19.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.365+0000 7f55bfe9e700 1 --2- 172.21.15.155:0/3231430305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55b8105750 0x7f55b8107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:19.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.365+0000 7f55bfe9e700 1 -- 172.21.15.155:0/3231430305 >> 172.21.15.155:0/3231430305 conn(0x7f55b80fc810 msgr2=0x7f55b80fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:19.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.365+0000 7f55bfe9e700 1 -- 172.21.15.155:0/3231430305 shutdown_connections 2024-03-19T02:48:19.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.365+0000 7f55bfe9e700 1 -- 172.21.15.155:0/3231430305 wait complete. 2024-03-19T02:48:19.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.366+0000 7f55bfe9e700 1 Processor -- start 2024-03-19T02:48:19.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.366+0000 7f55bfe9e700 1 -- start start 2024-03-19T02:48:19.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.366+0000 7f55bfe9e700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f55b8102e20 0x7f55b8198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:19.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.366+0000 7f55bfe9e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55b8105750 0x7f55b8198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:19.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.366+0000 7f55bfe9e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f55b8108080 0x7f55b819fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:19.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.366+0000 7f55bfe9e700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f55b810df00 con 0x7f55b8102e20 2024-03-19T02:48:19.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.366+0000 7f55bfe9e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f55b810dd80 con 0x7f55b8105750 2024-03-19T02:48:19.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.366+0000 7f55bfe9e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f55b810e080 con 0x7f55b8108080 2024-03-19T02:48:19.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.366+0000 7f55be43b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f55b8108080 0x7f55b819fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:19.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.366+0000 7f55be43b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f55b8108080 0x7f55b819fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:47984/0 (socket says 172.21.15.155:47984) 2024-03-19T02:48:19.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.366+0000 7f55be43b700 1 -- 172.21.15.155:0/3587659928 learned_addr learned my addr 172.21.15.155:0/3587659928 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-19T02:48:19.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.366+0000 7f55bd439700 1 --2- 172.21.15.155:0/3587659928 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55b8105750 0x7f55b8198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:19.370 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.366+0000 7f55bdc3a700 1 --2- 172.21.15.155:0/3587659928 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f55b8102e20 0x7f55b8198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:19.370 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.366+0000 7f55be43b700 1 -- 172.21.15.155:0/3587659928 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55b8105750 msgr2=0x7f55b8198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:19.370 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.366+0000 7f55be43b700 1 --2- 172.21.15.155:0/3587659928 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55b8105750 0x7f55b8198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:19.370 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.366+0000 7f55be43b700 1 -- 172.21.15.155:0/3587659928 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f55b8102e20 msgr2=0x7f55b8198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:19.370 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.366+0000 7f55be43b700 1 --2- 172.21.15.155:0/3587659928 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f55b8102e20 0x7f55b8198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:19.370 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.366+0000 7f55be43b700 1 -- 172.21.15.155:0/3587659928 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55b81a0170 con 0x7f55b8108080 2024-03-19T02:48:19.370 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.366+0000 7f55be43b700 1 --2- 172.21.15.155:0/3587659928 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f55b8108080 0x7f55b819fad0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f55b000ba10 tx=0x7f55b000bd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:19.370 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.367+0000 7f55aeffd700 1 -- 172.21.15.155:0/3587659928 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b0019070 con 0x7f55b8108080 2024-03-19T02:48:19.370 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.367+0000 7f55aeffd700 1 -- 172.21.15.155:0/3587659928 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55b000cde0 con 0x7f55b8108080 2024-03-19T02:48:19.370 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.367+0000 7f55aeffd700 1 -- 172.21.15.155:0/3587659928 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b0012340 con 0x7f55b8108080 2024-03-19T02:48:19.371 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.367+0000 7f55bfe9e700 1 -- 172.21.15.155:0/3587659928 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55b81a0460 con 0x7f55b8108080 2024-03-19T02:48:19.371 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.367+0000 7f55bfe9e700 1 -- 172.21.15.155:0/3587659928 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55b81a0950 con 0x7f55b8108080 2024-03-19T02:48:19.371 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.368+0000 7f55bfe9e700 1 -- 172.21.15.155:0/3587659928 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55b8068fd0 con 0x7f55b8108080 2024-03-19T02:48:19.371 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.369+0000 7f55aeffd700 1 -- 172.21.15.155:0/3587659928 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f55b001c070 con 0x7f55b8108080 2024-03-19T02:48:19.372 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.370+0000 7f55aeffd700 1 --2- 172.21.15.155:0/3587659928 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f55a40a2890 0x7f55a40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:19.372 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.370+0000 7f55aeffd700 1 -- 172.21.15.155:0/3587659928 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5397+0+0 (secure 0 0 0) 0x7f55b00cdaf0 con 0x7f55b8108080 2024-03-19T02:48:19.372 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.370+0000 7f55bdc3a700 1 --2- 172.21.15.155:0/3587659928 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f55a40a2890 0x7f55a40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:19.372 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.371+0000 7f55bdc3a700 1 --2- 172.21.15.155:0/3587659928 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f55a40a2890 0x7f55a40a4d50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f55b40036e0 tx=0x7f55b4005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:19.374 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.374+0000 7f55aeffd700 1 -- 172.21.15.155:0/3587659928 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f55b0023070 con 0x7f55b8108080 2024-03-19T02:48:19.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:19 smithi096 ceph-mon[35946]: osdmap e55: 8 total, 8 up, 8 in 2024-03-19T02:48:19.581 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:19.580+0000 7f55bfe9e700 1 -- 172.21.15.155:0/3587659928 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f55b8062750 con 0x7f55a40a2890 2024-03-19T02:48:20.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:20 smithi033 ceph-mon[31973]: osdmap e56: 8 total, 8 up, 8 in 2024-03-19T02:48:20.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:20 smithi033 ceph-mon[31973]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:48:20.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:20 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:48:20.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:20 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:48:20.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:20 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:48:20.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:20 smithi033 ceph-mon[31973]: pgmap v186: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:20.304 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:20 smithi155 ceph-mon[35804]: osdmap e56: 8 total, 8 up, 8 in 2024-03-19T02:48:20.305 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:20 smithi155 ceph-mon[35804]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:48:20.305 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:20 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:48:20.305 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:20 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:48:20.305 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:20 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:48:20.305 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:20 smithi155 ceph-mon[35804]: pgmap v186: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:20.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:20 smithi096 ceph-mon[35946]: osdmap e56: 8 total, 8 up, 8 in 2024-03-19T02:48:20.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:20 smithi096 ceph-mon[35946]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:48:20.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:20 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:48:20.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:20 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:48:20.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:20 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:48:20.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:20 smithi096 ceph-mon[35946]: pgmap v186: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:21.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:21 smithi033 ceph-mon[31973]: osdmap e57: 8 total, 8 up, 8 in 2024-03-19T02:48:21.309 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:21 smithi155 ceph-mon[35804]: osdmap e57: 8 total, 8 up, 8 in 2024-03-19T02:48:21.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:21 smithi096 ceph-mon[35946]: osdmap e57: 8 total, 8 up, 8 in 2024-03-19T02:48:22.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:22 smithi033 ceph-mon[31973]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:22.366 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:22 smithi096 ceph-mon[35946]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:22.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:22 smithi155 ceph-mon[35804]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:23.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:23 smithi033 ceph-mon[31973]: from='client.? 172.21.15.155:0/766438379' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb20eec0-39e5-4b86-9db5-486b1840cdad"}]: dispatch 2024-03-19T02:48:23.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:23 smithi033 ceph-mon[31973]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb20eec0-39e5-4b86-9db5-486b1840cdad"}]: dispatch 2024-03-19T02:48:23.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:23 smithi033 ceph-mon[31973]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bb20eec0-39e5-4b86-9db5-486b1840cdad"}]': finished 2024-03-19T02:48:23.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:23 smithi033 ceph-mon[31973]: osdmap e58: 9 total, 8 up, 9 in 2024-03-19T02:48:23.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:23 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:48:23.383 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:23 smithi096 ceph-mon[35946]: from='client.? 172.21.15.155:0/766438379' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb20eec0-39e5-4b86-9db5-486b1840cdad"}]: dispatch 2024-03-19T02:48:23.383 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:23 smithi096 ceph-mon[35946]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb20eec0-39e5-4b86-9db5-486b1840cdad"}]: dispatch 2024-03-19T02:48:23.383 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:23 smithi096 ceph-mon[35946]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bb20eec0-39e5-4b86-9db5-486b1840cdad"}]': finished 2024-03-19T02:48:23.383 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:23 smithi096 ceph-mon[35946]: osdmap e58: 9 total, 8 up, 9 in 2024-03-19T02:48:23.384 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:23 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:48:23.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:23 smithi155 ceph-mon[35804]: from='client.? 172.21.15.155:0/766438379' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb20eec0-39e5-4b86-9db5-486b1840cdad"}]: dispatch 2024-03-19T02:48:23.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:23 smithi155 ceph-mon[35804]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb20eec0-39e5-4b86-9db5-486b1840cdad"}]: dispatch 2024-03-19T02:48:23.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:23 smithi155 ceph-mon[35804]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bb20eec0-39e5-4b86-9db5-486b1840cdad"}]': finished 2024-03-19T02:48:23.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:23 smithi155 ceph-mon[35804]: osdmap e58: 9 total, 8 up, 9 in 2024-03-19T02:48:23.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:23 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:48:24.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:24 smithi096 ceph-mon[35946]: from='client.? 172.21.15.155:0/3691512521' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:48:24.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:24 smithi096 ceph-mon[35946]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:24.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:24 smithi155 ceph-mon[35804]: from='client.? 172.21.15.155:0/3691512521' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:48:24.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:24 smithi155 ceph-mon[35804]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:24.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:24 smithi033 ceph-mon[31973]: from='client.? 172.21.15.155:0/3691512521' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:48:24.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:24 smithi033 ceph-mon[31973]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:26.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:25 smithi155 ceph-mon[35804]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:26.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:25 smithi033 ceph-mon[31973]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:26.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:25 smithi096 ceph-mon[35946]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:28.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:27 smithi155 ceph-mon[35804]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:28.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:27 smithi033 ceph-mon[31973]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:28.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:27 smithi096 ceph-mon[35946]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:30.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:29 smithi155 ceph-mon[35804]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:30.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:29 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-19T02:48:30.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:29 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:48:30.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:29 smithi033 ceph-mon[31973]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:30.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:29 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-19T02:48:30.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:29 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:48:30.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:29 smithi096 ceph-mon[35946]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:30.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:29 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-19T02:48:30.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:29 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:48:30.962 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:30 smithi155 ceph-mon[35804]: Deploying daemon osd.8 on smithi155 2024-03-19T02:48:31.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:30 smithi033 ceph-mon[31973]: Deploying daemon osd.8 on smithi155 2024-03-19T02:48:31.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:30 smithi096 ceph-mon[35946]: Deploying daemon osd.8 on smithi155 2024-03-19T02:48:31.937 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:31 smithi155 ceph-mon[35804]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:32.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:31 smithi033 ceph-mon[31973]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:32.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:31 smithi096 ceph-mon[35946]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:34.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:33 smithi155 ceph-mon[35804]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:34.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:33 smithi033 ceph-mon[31973]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:34.078 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:33 smithi096 ceph-mon[35946]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:36.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:36 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:36.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:36 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:36.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:36 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:48:36.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:36 smithi155 ceph-mon[35804]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:36.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:36 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:36.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:36 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:36.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:36 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:48:36.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:36 smithi096 ceph-mon[35946]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:36.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:36 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:36.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:36 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:36.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:36 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:48:36.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:36 smithi033 ceph-mon[31973]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:37.959 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:37 smithi155 ceph-mon[35804]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:38.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:37 smithi033 ceph-mon[31973]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:38.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:37 smithi096 ceph-mon[35946]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:39.249 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:39.248+0000 7f55aeffd700 1 -- 172.21.15.155:0/3587659928 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f55b8062750 con 0x7f55a40a2890 2024-03-19T02:48:39.250 INFO:teuthology.orchestra.run.smithi155.stdout:Created osd(s) 8 on host 'smithi155' 2024-03-19T02:48:39.251 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:39.250+0000 7f55bfe9e700 1 -- 172.21.15.155:0/3587659928 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f55a40a2890 msgr2=0x7f55a40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:39.252 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:39.250+0000 7f55bfe9e700 1 --2- 172.21.15.155:0/3587659928 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f55a40a2890 0x7f55a40a4d50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f55b40036e0 tx=0x7f55b4005fd0 comp rx=0 tx=0).stop 2024-03-19T02:48:39.252 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:39.250+0000 7f55bfe9e700 1 -- 172.21.15.155:0/3587659928 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f55b8108080 msgr2=0x7f55b819fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:39.252 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:39.250+0000 7f55bfe9e700 1 --2- 172.21.15.155:0/3587659928 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f55b8108080 0x7f55b819fad0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f55b000ba10 tx=0x7f55b000bd20 comp rx=0 tx=0).stop 2024-03-19T02:48:39.252 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:39.250+0000 7f55bfe9e700 1 -- 172.21.15.155:0/3587659928 shutdown_connections 2024-03-19T02:48:39.252 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:39.250+0000 7f55bfe9e700 1 --2- 172.21.15.155:0/3587659928 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f55a40a2890 0x7f55a40a4d50 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:39.252 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:39.250+0000 7f55bfe9e700 1 --2- 172.21.15.155:0/3587659928 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f55b8108080 0x7f55b819fad0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:39.252 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:39.250+0000 7f55bfe9e700 1 --2- 172.21.15.155:0/3587659928 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f55b8102e20 0x7f55b8198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:39.252 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:39.250+0000 7f55bfe9e700 1 --2- 172.21.15.155:0/3587659928 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55b8105750 0x7f55b8198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:39.252 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:39.250+0000 7f55bfe9e700 1 -- 172.21.15.155:0/3587659928 >> 172.21.15.155:0/3587659928 conn(0x7f55b80fc810 msgr2=0x7f55b8103760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:39.252 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:39.251+0000 7f55bfe9e700 1 -- 172.21.15.155:0/3587659928 shutdown_connections 2024-03-19T02:48:39.252 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:48:39.251+0000 7f55bfe9e700 1 -- 172.21.15.155:0/3587659928 wait complete. 2024-03-19T02:48:39.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:39 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:39.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:39 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:39.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:39 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:48:39.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:39 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:48:39.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:39 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:39.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:39 smithi155 ceph-mon[35804]: from='osd.8 [v2:172.21.15.155:6816/3498516903,v1:172.21.15.155:6817/3498516903]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-19T02:48:39.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:39 smithi155 ceph-mon[35804]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-19T02:48:39.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:39 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:39.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:39 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:48:39.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:39 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:39.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:39 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:39.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:39 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:39.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:39 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:48:39.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:39 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:48:39.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:39 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:39.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:39 smithi096 ceph-mon[35946]: from='osd.8 [v2:172.21.15.155:6816/3498516903,v1:172.21.15.155:6817/3498516903]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-19T02:48:39.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:39 smithi096 ceph-mon[35946]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-19T02:48:39.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:39 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:39.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:39 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:48:39.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:39 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:39.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:39 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:39.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:39 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:39.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:39 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:48:39.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:39 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:48:39.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:39 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:39.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:39 smithi033 ceph-mon[31973]: from='osd.8 [v2:172.21.15.155:6816/3498516903,v1:172.21.15.155:6817/3498516903]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-19T02:48:39.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:39 smithi033 ceph-mon[31973]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-19T02:48:39.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:39 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:39.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:39 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:48:39.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:39 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:39.894 DEBUG:teuthology.orchestra.run.smithi155:osd.8> sudo journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.8.service 2024-03-19T02:48:39.896 INFO:tasks.cephadm:Deploying osd.9 on smithi033 with /dev/vg_nvme/lv_1... 2024-03-19T02:48:39.897 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-19T02:48:40.151 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:48:40.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:40 smithi033 ceph-mon[31973]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-19T02:48:40.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:40 smithi033 ceph-mon[31973]: from='osd.8 [v2:172.21.15.155:6816/3498516903,v1:172.21.15.155:6817/3498516903]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-19T02:48:40.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:40 smithi033 ceph-mon[31973]: osdmap e59: 9 total, 8 up, 9 in 2024-03-19T02:48:40.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:48:40.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:40 smithi033 ceph-mon[31973]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-19T02:48:40.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:40 smithi033 ceph-mon[31973]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:40.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:40 smithi096 ceph-mon[35946]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-19T02:48:40.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:40 smithi096 ceph-mon[35946]: from='osd.8 [v2:172.21.15.155:6816/3498516903,v1:172.21.15.155:6817/3498516903]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-19T02:48:40.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:40 smithi096 ceph-mon[35946]: osdmap e59: 9 total, 8 up, 9 in 2024-03-19T02:48:40.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:40 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:48:40.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:40 smithi096 ceph-mon[35946]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-19T02:48:40.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:40 smithi096 ceph-mon[35946]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:40.718 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:40 smithi155 ceph-mon[35804]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-19T02:48:40.718 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:40 smithi155 ceph-mon[35804]: from='osd.8 [v2:172.21.15.155:6816/3498516903,v1:172.21.15.155:6817/3498516903]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-19T02:48:40.718 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:40 smithi155 ceph-mon[35804]: osdmap e59: 9 total, 8 up, 9 in 2024-03-19T02:48:40.718 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:48:40.718 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:40 smithi155 ceph-mon[35804]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-19T02:48:40.718 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:40 smithi155 ceph-mon[35804]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T02:48:41.369 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 02:48:41 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T02:48:41.069+0000 7fec64e94700 -1 osd.8 0 waiting for initial osdmap 2024-03-19T02:48:41.369 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 02:48:41 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T02:48:41.082+0000 7fec5cc7f700 -1 osd.8 60 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T02:48:41.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:41 smithi033 ceph-mon[31973]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-03-19T02:48:41.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:41 smithi033 ceph-mon[31973]: osdmap e60: 9 total, 8 up, 9 in 2024-03-19T02:48:41.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:41 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:48:41.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:41 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:48:41.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:41 smithi033 ceph-mon[31973]: from='osd.8 ' entity='osd.8' 2024-03-19T02:48:41.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:41 smithi096 ceph-mon[35946]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-03-19T02:48:41.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:41 smithi096 ceph-mon[35946]: osdmap e60: 9 total, 8 up, 9 in 2024-03-19T02:48:41.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:41 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:48:41.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:41 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:48:41.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:41 smithi096 ceph-mon[35946]: from='osd.8 ' entity='osd.8' 2024-03-19T02:48:41.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:41 smithi155 ceph-mon[35804]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-03-19T02:48:41.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:41 smithi155 ceph-mon[35804]: osdmap e60: 9 total, 8 up, 9 in 2024-03-19T02:48:41.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:41 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:48:41.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:41 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:48:41.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:41 smithi155 ceph-mon[35804]: from='osd.8 ' entity='osd.8' 2024-03-19T02:48:42.231 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-19T02:48:42.231 INFO:teuthology.orchestra.run.smithi033.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-19T02:48:42.231 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10+0 records in 2024-03-19T02:48:42.231 INFO:teuthology.orchestra.run.smithi033.stderr:10+0 records out 2024-03-19T02:48:42.231 INFO:teuthology.orchestra.run.smithi033.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0109315 s, 959 MB/s 2024-03-19T02:48:42.231 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 2024-03-19T02:48:42.231 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping successful for: 2024-03-19T02:48:42.511 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:42 smithi155 ceph-mon[35804]: purged_snaps scrub starts 2024-03-19T02:48:42.581 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:42 smithi155 ceph-mon[35804]: purged_snaps scrub ok 2024-03-19T02:48:42.581 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:42 smithi155 ceph-mon[35804]: osd.8 [v2:172.21.15.155:6816/3498516903,v1:172.21.15.155:6817/3498516903] boot 2024-03-19T02:48:42.582 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:42 smithi155 ceph-mon[35804]: osdmap e61: 9 total, 9 up, 9 in 2024-03-19T02:48:42.582 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:42 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:48:42.582 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:42 smithi155 ceph-mon[35804]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:48:42.582 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:42 smithi155 ceph-mon[35804]: osdmap e62: 9 total, 9 up, 9 in 2024-03-19T02:48:42.582 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:42 smithi033 ceph-mon[31973]: purged_snaps scrub starts 2024-03-19T02:48:42.583 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:42 smithi033 ceph-mon[31973]: purged_snaps scrub ok 2024-03-19T02:48:42.583 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:42 smithi033 ceph-mon[31973]: osd.8 [v2:172.21.15.155:6816/3498516903,v1:172.21.15.155:6817/3498516903] boot 2024-03-19T02:48:42.583 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:42 smithi033 ceph-mon[31973]: osdmap e61: 9 total, 9 up, 9 in 2024-03-19T02:48:42.583 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:42 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:48:42.583 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:42 smithi033 ceph-mon[31973]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:48:42.583 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:42 smithi033 ceph-mon[31973]: osdmap e62: 9 total, 9 up, 9 in 2024-03-19T02:48:42.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:42 smithi096 ceph-mon[35946]: purged_snaps scrub starts 2024-03-19T02:48:42.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:42 smithi096 ceph-mon[35946]: purged_snaps scrub ok 2024-03-19T02:48:42.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:42 smithi096 ceph-mon[35946]: osd.8 [v2:172.21.15.155:6816/3498516903,v1:172.21.15.155:6817/3498516903] boot 2024-03-19T02:48:42.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:42 smithi096 ceph-mon[35946]: osdmap e61: 9 total, 9 up, 9 in 2024-03-19T02:48:42.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:42 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:48:42.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:42 smithi096 ceph-mon[35946]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:48:42.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:42 smithi096 ceph-mon[35946]: osdmap e62: 9 total, 9 up, 9 in 2024-03-19T02:48:43.136 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi033:vg_nvme/lv_1 2024-03-19T02:48:43.378 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:48:44.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:43 smithi033 ceph-mon[31973]: Detected new or changed devices on smithi155 2024-03-19T02:48:44.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:43 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:44.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:43 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:44.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:43 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:44.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:43 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:44.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:43 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:44.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:43 smithi033 ceph-mon[31973]: Adjusting osd_memory_target on smithi155 to 5696M 2024-03-19T02:48:44.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:43 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:44.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:43 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:48:44.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:43 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:48:44.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:43 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:44.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:43 smithi033 ceph-mon[31973]: osdmap e63: 9 total, 9 up, 9 in 2024-03-19T02:48:44.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:43 smithi033 ceph-mon[31973]: pgmap v205: 1 pgs: 1 remapped+peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:48:44.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:43 smithi096 ceph-mon[35946]: Detected new or changed devices on smithi155 2024-03-19T02:48:44.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:43 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:44.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:43 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:44.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:43 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:44.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:43 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:44.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:43 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:44.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:43 smithi096 ceph-mon[35946]: Adjusting osd_memory_target on smithi155 to 5696M 2024-03-19T02:48:44.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:43 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:44.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:43 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:48:44.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:43 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:48:44.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:43 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:44.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:43 smithi096 ceph-mon[35946]: osdmap e63: 9 total, 9 up, 9 in 2024-03-19T02:48:44.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:43 smithi096 ceph-mon[35946]: pgmap v205: 1 pgs: 1 remapped+peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:48:44.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:43 smithi155 ceph-mon[35804]: Detected new or changed devices on smithi155 2024-03-19T02:48:44.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:43 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:44.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:43 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:44.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:43 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:44.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:43 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:44.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:43 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:48:44.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:43 smithi155 ceph-mon[35804]: Adjusting osd_memory_target on smithi155 to 5696M 2024-03-19T02:48:44.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:43 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:44.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:43 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:48:44.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:43 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:48:44.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:43 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:48:44.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:43 smithi155 ceph-mon[35804]: osdmap e63: 9 total, 9 up, 9 in 2024-03-19T02:48:44.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:43 smithi155 ceph-mon[35804]: pgmap v205: 1 pgs: 1 remapped+peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:48:45.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.163+0000 7f76aaeb6700 1 -- 172.21.15.33:0/3452508014 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76a4102de0 msgr2=0x7f76a4109690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:45.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.163+0000 7f76aaeb6700 1 --2- 172.21.15.33:0/3452508014 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76a4102de0 0x7f76a4109690 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f768c0099c0 tx=0x7f768c009cd0 comp rx=0 tx=0).stop 2024-03-19T02:48:45.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.163+0000 7f76aaeb6700 1 -- 172.21.15.33:0/3452508014 shutdown_connections 2024-03-19T02:48:45.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.163+0000 7f76aaeb6700 1 --2- 172.21.15.33:0/3452508014 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76a4102de0 0x7f76a4109690 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:45.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.163+0000 7f76aaeb6700 1 --2- 172.21.15.33:0/3452508014 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76a4101240 0x7f76a4101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:45.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.163+0000 7f76aaeb6700 1 --2- 172.21.15.33:0/3452508014 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76a4102440 0x7f76a41028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:45.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.163+0000 7f76aaeb6700 1 -- 172.21.15.33:0/3452508014 >> 172.21.15.33:0/3452508014 conn(0x7f76a40fc7e0 msgr2=0x7f76a40fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:48:45.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.164+0000 7f76aaeb6700 1 -- 172.21.15.33:0/3452508014 shutdown_connections 2024-03-19T02:48:45.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.164+0000 7f76aaeb6700 1 -- 172.21.15.33:0/3452508014 wait complete. 2024-03-19T02:48:45.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.164+0000 7f76aaeb6700 1 Processor -- start 2024-03-19T02:48:45.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.164+0000 7f76aaeb6700 1 -- start start 2024-03-19T02:48:45.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.164+0000 7f76aaeb6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76a4101240 0x7f76a419a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:45.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.164+0000 7f76aaeb6700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76a4102440 0x7f76a419ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:45.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.164+0000 7f76aaeb6700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76a4102de0 0x7f76a41a1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:45.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.164+0000 7f76aaeb6700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f76a410ccd0 con 0x7f76a4102440 2024-03-19T02:48:45.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.164+0000 7f76aaeb6700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f76a410cb50 con 0x7f76a4101240 2024-03-19T02:48:45.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.164+0000 7f76aaeb6700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f76a410ce50 con 0x7f76a4102de0 2024-03-19T02:48:45.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.164+0000 7f76a3fff700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76a4102440 0x7f76a419ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:45.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.165+0000 7f76a3fff700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76a4102440 0x7f76a419ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47966/0 (socket says 172.21.15.33:47966) 2024-03-19T02:48:45.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.165+0000 7f76a3fff700 1 -- 172.21.15.33:0/2217887173 learned_addr learned my addr 172.21.15.33:0/2217887173 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:48:45.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.165+0000 7f76a8c52700 1 --2- 172.21.15.33:0/2217887173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76a4101240 0x7f76a419a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:45.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.165+0000 7f76a9453700 1 --2- 172.21.15.33:0/2217887173 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76a4102de0 0x7f76a41a1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:45.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.165+0000 7f76a3fff700 1 -- 172.21.15.33:0/2217887173 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76a4102de0 msgr2=0x7f76a41a1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:45.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.165+0000 7f76a3fff700 1 --2- 172.21.15.33:0/2217887173 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76a4102de0 0x7f76a41a1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:45.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.165+0000 7f76a3fff700 1 -- 172.21.15.33:0/2217887173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76a4101240 msgr2=0x7f76a419a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:48:45.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.165+0000 7f76a3fff700 1 --2- 172.21.15.33:0/2217887173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76a4101240 0x7f76a419a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:48:45.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.165+0000 7f76a3fff700 1 -- 172.21.15.33:0/2217887173 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76a41a2230 con 0x7f76a4102440 2024-03-19T02:48:45.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.165+0000 7f76a3fff700 1 --2- 172.21.15.33:0/2217887173 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76a4102440 0x7f76a419ab10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f769400c9e0 tx=0x7f769400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:45.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.165+0000 7f76a1ffb700 1 -- 172.21.15.33:0/2217887173 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7694007ac0 con 0x7f76a4102440 2024-03-19T02:48:45.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.165+0000 7f76a1ffb700 1 -- 172.21.15.33:0/2217887173 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7694007c60 con 0x7f76a4102440 2024-03-19T02:48:45.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.165+0000 7f76aaeb6700 1 -- 172.21.15.33:0/2217887173 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76a41a2520 con 0x7f76a4102440 2024-03-19T02:48:45.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.165+0000 7f76a1ffb700 1 -- 172.21.15.33:0/2217887173 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7694018660 con 0x7f76a4102440 2024-03-19T02:48:45.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.165+0000 7f76aaeb6700 1 -- 172.21.15.33:0/2217887173 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76a41a2a60 con 0x7f76a4102440 2024-03-19T02:48:45.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.166+0000 7f76aaeb6700 1 -- 172.21.15.33:0/2217887173 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76a4068fd0 con 0x7f76a4102440 2024-03-19T02:48:45.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.167+0000 7f76a1ffb700 1 -- 172.21.15.33:0/2217887173 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f769400f450 con 0x7f76a4102440 2024-03-19T02:48:45.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.168+0000 7f76a1ffb700 1 --2- 172.21.15.33:0/2217887173 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f76900a2890 0x7f76900a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:48:45.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.168+0000 7f76a8c52700 1 --2- 172.21.15.33:0/2217887173 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f76900a2890 0x7f76900a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:48:45.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.168+0000 7f76a1ffb700 1 -- 172.21.15.33:0/2217887173 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5829+0+0 (secure 0 0 0) 0x7f76940ce030 con 0x7f76a4102440 2024-03-19T02:48:45.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.168+0000 7f76a8c52700 1 --2- 172.21.15.33:0/2217887173 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f76900a2890 0x7f76900a4d50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f76a41022a0 tx=0x7f769c009180 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:48:45.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.173+0000 7f76a1ffb700 1 -- 172.21.15.33:0/2217887173 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7694014030 con 0x7f76a4102440 2024-03-19T02:48:45.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:48:45.386+0000 7f76aaeb6700 1 -- 172.21.15.33:0/2217887173 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f76a406a6d0 con 0x7f76900a2890 2024-03-19T02:48:46.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:45 smithi096 ceph-mon[35946]: osdmap e64: 9 total, 9 up, 9 in 2024-03-19T02:48:46.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:45 smithi096 ceph-mon[35946]: from='client.14406 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:48:46.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:45 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:48:46.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:45 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:48:46.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:45 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:48:46.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:45 smithi096 ceph-mon[35946]: pgmap v207: 1 pgs: 1 remapped+peering; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:48:46.224 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:45 smithi033 ceph-mon[31973]: osdmap e64: 9 total, 9 up, 9 in 2024-03-19T02:48:46.224 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:45 smithi033 ceph-mon[31973]: from='client.14406 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:48:46.224 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:45 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:48:46.224 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:45 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:48:46.224 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:45 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:48:46.224 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:45 smithi033 ceph-mon[31973]: pgmap v207: 1 pgs: 1 remapped+peering; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:48:46.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:45 smithi155 ceph-mon[35804]: osdmap e64: 9 total, 9 up, 9 in 2024-03-19T02:48:46.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:45 smithi155 ceph-mon[35804]: from='client.14406 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:48:46.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:45 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:48:46.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:45 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:48:46.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:45 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:48:46.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:45 smithi155 ceph-mon[35804]: pgmap v207: 1 pgs: 1 remapped+peering; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:48:47.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:46 smithi096 ceph-mon[35946]: osdmap e65: 9 total, 9 up, 9 in 2024-03-19T02:48:47.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:46 smithi155 ceph-mon[35804]: osdmap e65: 9 total, 9 up, 9 in 2024-03-19T02:48:47.277 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:46 smithi033 ceph-mon[31973]: osdmap e65: 9 total, 9 up, 9 in 2024-03-19T02:48:48.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:47 smithi096 ceph-mon[35946]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 804 GiB / 805 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-03-19T02:48:48.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:47 smithi155 ceph-mon[35804]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 804 GiB / 805 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-03-19T02:48:48.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:47 smithi033 ceph-mon[31973]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 804 GiB / 805 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-03-19T02:48:49.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:48 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1927019559' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "179c6228-3e23-4f65-91a6-1ef0005d6904"}]: dispatch 2024-03-19T02:48:49.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:48 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1927019559' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "179c6228-3e23-4f65-91a6-1ef0005d6904"}]': finished 2024-03-19T02:48:49.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:48 smithi096 ceph-mon[35946]: osdmap e66: 10 total, 9 up, 10 in 2024-03-19T02:48:49.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:48 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:48:49.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:48 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1891186041' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:48:49.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:48 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1927019559' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "179c6228-3e23-4f65-91a6-1ef0005d6904"}]: dispatch 2024-03-19T02:48:49.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:48 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1927019559' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "179c6228-3e23-4f65-91a6-1ef0005d6904"}]': finished 2024-03-19T02:48:49.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:48 smithi155 ceph-mon[35804]: osdmap e66: 10 total, 9 up, 10 in 2024-03-19T02:48:49.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:48 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:48:49.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:48 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1891186041' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:48:49.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:48 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1927019559' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "179c6228-3e23-4f65-91a6-1ef0005d6904"}]: dispatch 2024-03-19T02:48:49.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:48 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1927019559' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "179c6228-3e23-4f65-91a6-1ef0005d6904"}]': finished 2024-03-19T02:48:49.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:48 smithi033 ceph-mon[31973]: osdmap e66: 10 total, 9 up, 10 in 2024-03-19T02:48:49.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:48 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:48:49.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:48 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1891186041' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:48:50.167 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:49 smithi155 ceph-mon[35804]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-19T02:48:50.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:49 smithi096 ceph-mon[35946]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-19T02:48:50.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:49 smithi033 ceph-mon[31973]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-19T02:48:52.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:51 smithi155 ceph-mon[35804]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-19T02:48:52.017 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:51 smithi096 ceph-mon[35946]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-19T02:48:52.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:51 smithi033 ceph-mon[31973]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-19T02:48:53.948 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:53 smithi033 ceph-mon[31973]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-19T02:48:54.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:53 smithi155 ceph-mon[35804]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-19T02:48:54.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:53 smithi096 ceph-mon[35946]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-19T02:48:56.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:55 smithi155 ceph-mon[35804]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-19T02:48:56.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:55 smithi033 ceph-mon[31973]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-19T02:48:56.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:55 smithi096 ceph-mon[35946]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-19T02:48:57.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:56 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-19T02:48:57.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:56 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:48:57.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:56 smithi155 ceph-mon[35804]: Deploying daemon osd.9 on smithi033 2024-03-19T02:48:57.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:56 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-19T02:48:57.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:56 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:48:57.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:56 smithi033 ceph-mon[31973]: Deploying daemon osd.9 on smithi033 2024-03-19T02:48:57.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:56 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-19T02:48:57.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:56 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:48:57.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:56 smithi096 ceph-mon[35946]: Deploying daemon osd.9 on smithi033 2024-03-19T02:48:58.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:57 smithi033 ceph-mon[31973]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:48:58.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:57 smithi096 ceph-mon[35946]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:48:58.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:57 smithi155 ceph-mon[35804]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:48:59.867 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:48:59 smithi033 ceph-mon[31973]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:49:00.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:48:59 smithi096 ceph-mon[35946]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:49:00.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:48:59 smithi155 ceph-mon[35804]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:49:02.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:02 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:02.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:02 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:02.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:02 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:49:02.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:02 smithi096 ceph-mon[35946]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:49:02.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:02 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:02.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:02 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:02.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:02 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:49:02.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:02 smithi155 ceph-mon[35804]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:49:02.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:02 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:02.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:02 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:02.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:02 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:49:02.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:02 smithi033 ceph-mon[31973]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:49:04.021 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:03 smithi033 ceph-mon[31973]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:49:04.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:03 smithi096 ceph-mon[35946]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:49:04.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:03 smithi155 ceph-mon[35804]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:49:05.563 INFO:teuthology.orchestra.run.smithi033.stdout:Created osd(s) 9 on host 'smithi033' 2024-03-19T02:49:05.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:49:05.562+0000 7f76a1ffb700 1 -- 172.21.15.33:0/2217887173 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f76a406a6d0 con 0x7f76900a2890 2024-03-19T02:49:05.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:49:05.564+0000 7f76aaeb6700 1 -- 172.21.15.33:0/2217887173 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f76900a2890 msgr2=0x7f76900a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:05.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:49:05.564+0000 7f76aaeb6700 1 --2- 172.21.15.33:0/2217887173 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f76900a2890 0x7f76900a4d50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f76a41022a0 tx=0x7f769c009180 comp rx=0 tx=0).stop 2024-03-19T02:49:05.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:49:05.565+0000 7f76aaeb6700 1 -- 172.21.15.33:0/2217887173 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76a4102440 msgr2=0x7f76a419ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:05.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:49:05.565+0000 7f76aaeb6700 1 --2- 172.21.15.33:0/2217887173 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76a4102440 0x7f76a419ab10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f769400c9e0 tx=0x7f769400ccf0 comp rx=0 tx=0).stop 2024-03-19T02:49:05.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:49:05.565+0000 7f76aaeb6700 1 -- 172.21.15.33:0/2217887173 shutdown_connections 2024-03-19T02:49:05.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:49:05.565+0000 7f76aaeb6700 1 --2- 172.21.15.33:0/2217887173 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f76900a2890 0x7f76900a4d50 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:05.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:49:05.565+0000 7f76aaeb6700 1 --2- 172.21.15.33:0/2217887173 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f76a4102de0 0x7f76a41a1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:05.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:49:05.565+0000 7f76aaeb6700 1 --2- 172.21.15.33:0/2217887173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f76a4101240 0x7f76a419a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:05.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:49:05.565+0000 7f76aaeb6700 1 --2- 172.21.15.33:0/2217887173 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f76a4102440 0x7f76a419ab10 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:05.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:49:05.565+0000 7f76aaeb6700 1 -- 172.21.15.33:0/2217887173 >> 172.21.15.33:0/2217887173 conn(0x7f76a40fc7e0 msgr2=0x7f76a4069fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:05.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:49:05.565+0000 7f76aaeb6700 1 -- 172.21.15.33:0/2217887173 shutdown_connections 2024-03-19T02:49:05.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:49:05.565+0000 7f76aaeb6700 1 -- 172.21.15.33:0/2217887173 wait complete. 2024-03-19T02:49:05.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:05 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:05.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:05 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:05.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:05 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:49:05.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:05 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:49:05.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:05 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:05.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:05 smithi033 ceph-mon[31973]: from='osd.9 [v2:172.21.15.33:6826/760777213,v1:172.21.15.33:6827/760777213]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-19T02:49:05.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:05 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:05.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:05 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:05.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:05 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:49:05.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:05 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:05.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:05 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:05.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:05 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:49:05.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:05 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:49:05.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:05 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:05.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:05 smithi096 ceph-mon[35946]: from='osd.9 [v2:172.21.15.33:6826/760777213,v1:172.21.15.33:6827/760777213]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-19T02:49:05.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:05 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:05.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:05 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:05.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:05 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:49:06.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:05 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:06.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:05 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:06.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:05 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:49:06.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:05 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:49:06.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:05 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:06.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:05 smithi155 ceph-mon[35804]: from='osd.9 [v2:172.21.15.33:6826/760777213,v1:172.21.15.33:6827/760777213]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-19T02:49:06.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:05 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:06.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:05 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:06.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:05 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:49:06.189 DEBUG:teuthology.orchestra.run.smithi033:osd.9> sudo journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.9.service 2024-03-19T02:49:06.191 INFO:tasks.cephadm:Deploying osd.10 on smithi096 with /dev/vg_nvme/lv_1... 2024-03-19T02:49:06.191 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-19T02:49:06.437 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T02:49:06.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:06 smithi033 ceph-mon[31973]: from='osd.9 [v2:172.21.15.33:6826/760777213,v1:172.21.15.33:6827/760777213]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-19T02:49:06.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:06 smithi033 ceph-mon[31973]: osdmap e67: 10 total, 9 up, 10 in 2024-03-19T02:49:06.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:06 smithi033 ceph-mon[31973]: from='osd.9 [v2:172.21.15.33:6826/760777213,v1:172.21.15.33:6827/760777213]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-19T02:49:06.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:06 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:49:06.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:06 smithi033 ceph-mon[31973]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:49:06.814 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:06 smithi096 ceph-mon[35946]: from='osd.9 [v2:172.21.15.33:6826/760777213,v1:172.21.15.33:6827/760777213]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-19T02:49:06.817 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:06 smithi096 ceph-mon[35946]: osdmap e67: 10 total, 9 up, 10 in 2024-03-19T02:49:06.817 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:06 smithi096 ceph-mon[35946]: from='osd.9 [v2:172.21.15.33:6826/760777213,v1:172.21.15.33:6827/760777213]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-19T02:49:06.818 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:06 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:49:06.818 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:06 smithi096 ceph-mon[35946]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:49:07.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:06 smithi155 ceph-mon[35804]: from='osd.9 [v2:172.21.15.33:6826/760777213,v1:172.21.15.33:6827/760777213]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-19T02:49:07.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:06 smithi155 ceph-mon[35804]: osdmap e67: 10 total, 9 up, 10 in 2024-03-19T02:49:07.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:06 smithi155 ceph-mon[35804]: from='osd.9 [v2:172.21.15.33:6826/760777213,v1:172.21.15.33:6827/760777213]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-03-19T02:49:07.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:06 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:49:07.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:06 smithi155 ceph-mon[35804]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:49:07.598 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 02:49:07 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T02:49:07.305+0000 7fe2d1b01700 -1 osd.9 0 waiting for initial osdmap 2024-03-19T02:49:07.598 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 02:49:07 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T02:49:07.319+0000 7fe2c98ec700 -1 osd.9 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T02:49:07.918 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:07 smithi033 ceph-mon[31973]: from='osd.9 [v2:172.21.15.33:6826/760777213,v1:172.21.15.33:6827/760777213]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-03-19T02:49:07.918 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:07 smithi033 ceph-mon[31973]: osdmap e68: 10 total, 9 up, 10 in 2024-03-19T02:49:07.918 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:07 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:49:07.918 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:07 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:49:07.918 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:07 smithi033 ceph-mon[31973]: from='osd.9 [v2:172.21.15.33:6826/760777213,v1:172.21.15.33:6827/760777213]' entity='osd.9' 2024-03-19T02:49:07.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:07 smithi096 ceph-mon[35946]: from='osd.9 [v2:172.21.15.33:6826/760777213,v1:172.21.15.33:6827/760777213]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-03-19T02:49:07.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:07 smithi096 ceph-mon[35946]: osdmap e68: 10 total, 9 up, 10 in 2024-03-19T02:49:07.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:07 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:49:07.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:07 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:49:07.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:07 smithi096 ceph-mon[35946]: from='osd.9 [v2:172.21.15.33:6826/760777213,v1:172.21.15.33:6827/760777213]' entity='osd.9' 2024-03-19T02:49:08.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:07 smithi155 ceph-mon[35804]: from='osd.9 [v2:172.21.15.33:6826/760777213,v1:172.21.15.33:6827/760777213]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-03-19T02:49:08.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:07 smithi155 ceph-mon[35804]: osdmap e68: 10 total, 9 up, 10 in 2024-03-19T02:49:08.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:07 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:49:08.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:07 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:49:08.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:07 smithi155 ceph-mon[35804]: from='osd.9 [v2:172.21.15.33:6826/760777213,v1:172.21.15.33:6827/760777213]' entity='osd.9' 2024-03-19T02:49:08.708 INFO:teuthology.orchestra.run.smithi096.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-19T02:49:08.709 INFO:teuthology.orchestra.run.smithi096.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-19T02:49:08.709 INFO:teuthology.orchestra.run.smithi096.stderr: stderr: 10+0 records in 2024-03-19T02:49:08.709 INFO:teuthology.orchestra.run.smithi096.stderr:10+0 records out 2024-03-19T02:49:08.709 INFO:teuthology.orchestra.run.smithi096.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113506 s, 924 MB/s 2024-03-19T02:49:08.709 INFO:teuthology.orchestra.run.smithi096.stderr:--> Zapping successful for: 2024-03-19T02:49:08.847 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:08 smithi033 ceph-mon[31973]: purged_snaps scrub starts 2024-03-19T02:49:08.847 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:08 smithi033 ceph-mon[31973]: purged_snaps scrub ok 2024-03-19T02:49:08.847 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:08 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:49:08.847 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:08 smithi033 ceph-mon[31973]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:49:08.848 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:08 smithi033 ceph-mon[31973]: osd.9 [v2:172.21.15.33:6826/760777213,v1:172.21.15.33:6827/760777213] boot 2024-03-19T02:49:08.848 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:08 smithi033 ceph-mon[31973]: osdmap e69: 10 total, 10 up, 10 in 2024-03-19T02:49:08.848 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:08 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:49:08.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:08 smithi096 ceph-mon[35946]: purged_snaps scrub starts 2024-03-19T02:49:08.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:08 smithi096 ceph-mon[35946]: purged_snaps scrub ok 2024-03-19T02:49:08.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:08 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:49:08.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:08 smithi096 ceph-mon[35946]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:49:08.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:08 smithi096 ceph-mon[35946]: osd.9 [v2:172.21.15.33:6826/760777213,v1:172.21.15.33:6827/760777213] boot 2024-03-19T02:49:08.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:08 smithi096 ceph-mon[35946]: osdmap e69: 10 total, 10 up, 10 in 2024-03-19T02:49:08.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:08 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:49:09.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:08 smithi155 ceph-mon[35804]: purged_snaps scrub starts 2024-03-19T02:49:09.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:08 smithi155 ceph-mon[35804]: purged_snaps scrub ok 2024-03-19T02:49:09.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:08 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:49:09.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:08 smithi155 ceph-mon[35804]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-19T02:49:09.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:08 smithi155 ceph-mon[35804]: osd.9 [v2:172.21.15.33:6826/760777213,v1:172.21.15.33:6827/760777213] boot 2024-03-19T02:49:09.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:08 smithi155 ceph-mon[35804]: osdmap e69: 10 total, 10 up, 10 in 2024-03-19T02:49:09.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:08 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:49:09.439 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi096:vg_nvme/lv_1 2024-03-19T02:49:09.674 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T02:49:10.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:10 smithi096 ceph-mon[35946]: Detected new or changed devices on smithi033 2024-03-19T02:49:10.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:10 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:10.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:10 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:10.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:10 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:49:10.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:10 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:49:10.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:10 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:49:10.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:10 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:49:10.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:10 smithi096 ceph-mon[35946]: Adjusting osd_memory_target on smithi033 to 4273M 2024-03-19T02:49:10.467 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:10 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:10.467 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:10 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:49:10.467 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:10 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:49:10.467 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:10 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:10.467 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:10 smithi096 ceph-mon[35946]: osdmap e70: 10 total, 10 up, 10 in 2024-03-19T02:49:10.467 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:10 smithi096 ceph-mon[35946]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:10.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:10 smithi155 ceph-mon[35804]: Detected new or changed devices on smithi033 2024-03-19T02:49:10.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:10 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:10.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:10 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:10.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:10 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:49:10.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:10 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:49:10.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:10 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:49:10.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:10 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:49:10.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:10 smithi155 ceph-mon[35804]: Adjusting osd_memory_target on smithi033 to 4273M 2024-03-19T02:49:10.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:10 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:10.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:10 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:49:10.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:10 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:49:10.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:10 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:10.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:10 smithi155 ceph-mon[35804]: osdmap e70: 10 total, 10 up, 10 in 2024-03-19T02:49:10.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:10 smithi155 ceph-mon[35804]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:10.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:10 smithi033 ceph-mon[31973]: Detected new or changed devices on smithi033 2024-03-19T02:49:10.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:10 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:10.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:10 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:10.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:10 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:49:10.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:10 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:49:10.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:10 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:49:10.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:10 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:49:10.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:10 smithi033 ceph-mon[31973]: Adjusting osd_memory_target on smithi033 to 4273M 2024-03-19T02:49:10.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:10 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:10.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:10 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:49:10.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:10 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:49:10.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:10 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:10.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:10 smithi033 ceph-mon[31973]: osdmap e70: 10 total, 10 up, 10 in 2024-03-19T02:49:10.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:10 smithi033 ceph-mon[31973]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:11.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.387+0000 7fe4c3cc1700 1 -- 172.21.15.96:0/3274978731 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe4bc102de0 msgr2=0x7fe4bc109690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:11.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.387+0000 7fe4c3cc1700 1 --2- 172.21.15.96:0/3274978731 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe4bc102de0 0x7fe4bc109690 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe4b400b0d0 tx=0x7fe4b400b3e0 comp rx=0 tx=0).stop 2024-03-19T02:49:11.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.387+0000 7fe4c3cc1700 1 -- 172.21.15.96:0/3274978731 shutdown_connections 2024-03-19T02:49:11.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.387+0000 7fe4c3cc1700 1 --2- 172.21.15.96:0/3274978731 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe4bc102de0 0x7fe4bc109690 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:11.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.387+0000 7fe4c3cc1700 1 --2- 172.21.15.96:0/3274978731 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4bc101240 0x7fe4bc101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:11.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.387+0000 7fe4c3cc1700 1 --2- 172.21.15.96:0/3274978731 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe4bc102440 0x7fe4bc1028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:11.390 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.387+0000 7fe4c3cc1700 1 -- 172.21.15.96:0/3274978731 >> 172.21.15.96:0/3274978731 conn(0x7fe4bc0fc7e0 msgr2=0x7fe4bc0fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:11.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.387+0000 7fe4c3cc1700 1 -- 172.21.15.96:0/3274978731 shutdown_connections 2024-03-19T02:49:11.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.387+0000 7fe4c3cc1700 1 -- 172.21.15.96:0/3274978731 wait complete. 2024-03-19T02:49:11.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.388+0000 7fe4c3cc1700 1 Processor -- start 2024-03-19T02:49:11.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.388+0000 7fe4c3cc1700 1 -- start start 2024-03-19T02:49:11.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.388+0000 7fe4c3cc1700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe4bc101240 0x7fe4bc06b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:11.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.388+0000 7fe4c3cc1700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe4bc102440 0x7fe4bc19d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:11.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.388+0000 7fe4c3cc1700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4bc102de0 0x7fe4bc19f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:11.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.388+0000 7fe4c3cc1700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe4bc10cd80 con 0x7fe4bc101240 2024-03-19T02:49:11.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.388+0000 7fe4c3cc1700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe4bc10cc00 con 0x7fe4bc102440 2024-03-19T02:49:11.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.388+0000 7fe4c3cc1700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe4bc10cf00 con 0x7fe4bc102de0 2024-03-19T02:49:11.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.388+0000 7fe4c125c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe4bc102440 0x7fe4bc19d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:11.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.388+0000 7fe4c125c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe4bc102440 0x7fe4bc19d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:57316/0 (socket says 172.21.15.96:57316) 2024-03-19T02:49:11.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.388+0000 7fe4c125c700 1 -- 172.21.15.96:0/4192575467 learned_addr learned my addr 172.21.15.96:0/4192575467 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-03-19T02:49:11.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.388+0000 7fe4c225e700 1 --2- 172.21.15.96:0/4192575467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4bc102de0 0x7fe4bc19f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:11.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.388+0000 7fe4c1a5d700 1 --2- 172.21.15.96:0/4192575467 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe4bc101240 0x7fe4bc06b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:11.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.388+0000 7fe4c125c700 1 -- 172.21.15.96:0/4192575467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4bc102de0 msgr2=0x7fe4bc19f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:11.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.388+0000 7fe4c125c700 1 --2- 172.21.15.96:0/4192575467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4bc102de0 0x7fe4bc19f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:11.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.388+0000 7fe4c125c700 1 -- 172.21.15.96:0/4192575467 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe4bc101240 msgr2=0x7fe4bc06b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:11.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.388+0000 7fe4c125c700 1 --2- 172.21.15.96:0/4192575467 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe4bc101240 0x7fe4bc06b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:11.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.388+0000 7fe4c125c700 1 -- 172.21.15.96:0/4192575467 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4bc1a0120 con 0x7fe4bc102440 2024-03-19T02:49:11.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.388+0000 7fe4c1a5d700 1 --2- 172.21.15.96:0/4192575467 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe4bc101240 0x7fe4bc06b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:49:11.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.389+0000 7fe4c125c700 1 --2- 172.21.15.96:0/4192575467 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe4bc102440 0x7fe4bc19d200 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac00b530 tx=0x7fe4ac00b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:11.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.389+0000 7fe4b2ffd700 1 -- 172.21.15.96:0/4192575467 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4ac015020 con 0x7fe4bc102440 2024-03-19T02:49:11.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.389+0000 7fe4c3cc1700 1 -- 172.21.15.96:0/4192575467 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4bc1a0410 con 0x7fe4bc102440 2024-03-19T02:49:11.392 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.389+0000 7fe4b2ffd700 1 -- 172.21.15.96:0/4192575467 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe4ac011ce0 con 0x7fe4bc102440 2024-03-19T02:49:11.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.389+0000 7fe4b2ffd700 1 -- 172.21.15.96:0/4192575467 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4ac010d60 con 0x7fe4bc102440 2024-03-19T02:49:11.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.389+0000 7fe4c3cc1700 1 -- 172.21.15.96:0/4192575467 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4bc1a0a20 con 0x7fe4bc102440 2024-03-19T02:49:11.393 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.390+0000 7fe4c3cc1700 1 -- 172.21.15.96:0/4192575467 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4bc068fd0 con 0x7fe4bc102440 2024-03-19T02:49:11.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.391+0000 7fe4b2ffd700 1 -- 172.21.15.96:0/4192575467 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fe4ac018020 con 0x7fe4bc102440 2024-03-19T02:49:11.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.391+0000 7fe4b2ffd700 1 --2- 172.21.15.96:0/4192575467 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe4a80a2890 0x7fe4a80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:11.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.392+0000 7fe4b2ffd700 1 -- 172.21.15.96:0/4192575467 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 6261+0+0 (secure 0 0 0) 0x7fe4ac0cddf0 con 0x7fe4bc102440 2024-03-19T02:49:11.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.392+0000 7fe4c1a5d700 1 --2- 172.21.15.96:0/4192575467 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe4a80a2890 0x7fe4a80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:11.394 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.392+0000 7fe4c1a5d700 1 --2- 172.21.15.96:0/4192575467 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe4a80a2890 0x7fe4a80a4d50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8005fd0 tx=0x7fe4b8005dc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:11.396 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.395+0000 7fe4b2ffd700 1 -- 172.21.15.96:0/4192575467 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe4ac01f030 con 0x7fe4bc102440 2024-03-19T02:49:11.600 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:11.599+0000 7fe4c3cc1700 1 -- 172.21.15.96:0/4192575467 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fe4bc062750 con 0x7fe4a80a2890 2024-03-19T02:49:11.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:11 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:49:11.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:11 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:49:11.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:11 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:49:12.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:11 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:49:12.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:11 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:49:12.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:11 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:49:12.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:11 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:49:12.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:11 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:49:12.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:11 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:49:12.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:12 smithi096 ceph-mon[35946]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:49:12.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:12 smithi096 ceph-mon[35946]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:13.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:12 smithi155 ceph-mon[35804]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:49:13.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:12 smithi155 ceph-mon[35804]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:13.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:12 smithi033 ceph-mon[31973]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi096:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:49:13.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:12 smithi033 ceph-mon[31973]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:14.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:13 smithi033 ceph-mon[31973]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:14.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:13 smithi096 ceph-mon[35946]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:14.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:13 smithi155 ceph-mon[35804]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:15.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:14 smithi033 ceph-mon[31973]: from='client.? 172.21.15.96:0/1686793495' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be17b478-3c2e-41fb-bede-958333acdf1d"}]: dispatch 2024-03-19T02:49:15.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:14 smithi033 ceph-mon[31973]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be17b478-3c2e-41fb-bede-958333acdf1d"}]: dispatch 2024-03-19T02:49:15.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:14 smithi033 ceph-mon[31973]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "be17b478-3c2e-41fb-bede-958333acdf1d"}]': finished 2024-03-19T02:49:15.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:14 smithi033 ceph-mon[31973]: osdmap e71: 11 total, 10 up, 11 in 2024-03-19T02:49:15.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:14 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:49:15.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:14 smithi096 ceph-mon[35946]: from='client.? 172.21.15.96:0/1686793495' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be17b478-3c2e-41fb-bede-958333acdf1d"}]: dispatch 2024-03-19T02:49:15.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:14 smithi096 ceph-mon[35946]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be17b478-3c2e-41fb-bede-958333acdf1d"}]: dispatch 2024-03-19T02:49:15.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:14 smithi096 ceph-mon[35946]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "be17b478-3c2e-41fb-bede-958333acdf1d"}]': finished 2024-03-19T02:49:15.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:14 smithi096 ceph-mon[35946]: osdmap e71: 11 total, 10 up, 11 in 2024-03-19T02:49:15.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:14 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:49:15.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:14 smithi155 ceph-mon[35804]: from='client.? 172.21.15.96:0/1686793495' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be17b478-3c2e-41fb-bede-958333acdf1d"}]: dispatch 2024-03-19T02:49:15.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:14 smithi155 ceph-mon[35804]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be17b478-3c2e-41fb-bede-958333acdf1d"}]: dispatch 2024-03-19T02:49:15.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:14 smithi155 ceph-mon[35804]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "be17b478-3c2e-41fb-bede-958333acdf1d"}]': finished 2024-03-19T02:49:15.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:14 smithi155 ceph-mon[35804]: osdmap e71: 11 total, 10 up, 11 in 2024-03-19T02:49:15.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:14 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:49:16.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:15 smithi033 ceph-mon[31973]: from='client.? 172.21.15.96:0/3026353240' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:49:16.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:15 smithi033 ceph-mon[31973]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:16.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:15 smithi096 ceph-mon[35946]: from='client.? 172.21.15.96:0/3026353240' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:49:16.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:15 smithi096 ceph-mon[35946]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:16.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:15 smithi155 ceph-mon[35804]: from='client.? 172.21.15.96:0/3026353240' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:49:16.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:15 smithi155 ceph-mon[35804]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:18.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:17 smithi033 ceph-mon[31973]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:18.177 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:17 smithi096 ceph-mon[35946]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:18.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:17 smithi155 ceph-mon[35804]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:20.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:19 smithi033 ceph-mon[31973]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:20.068 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:19 smithi155 ceph-mon[35804]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:20.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:19 smithi096 ceph-mon[35946]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:22.024 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:21 smithi096 ceph-mon[35946]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:22.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:21 smithi033 ceph-mon[31973]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:22.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:21 smithi155 ceph-mon[35804]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:23.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:22 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-19T02:49:23.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:22 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:49:23.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:22 smithi033 ceph-mon[31973]: Deploying daemon osd.10 on smithi096 2024-03-19T02:49:23.070 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:22 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-19T02:49:23.070 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:22 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:49:23.070 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:22 smithi096 ceph-mon[35946]: Deploying daemon osd.10 on smithi096 2024-03-19T02:49:23.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:22 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-19T02:49:23.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:22 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:49:23.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:22 smithi155 ceph-mon[35804]: Deploying daemon osd.10 on smithi096 2024-03-19T02:49:24.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:23 smithi033 ceph-mon[31973]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:24.081 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:23 smithi096 ceph-mon[35946]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:24.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:23 smithi155 ceph-mon[35804]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:26.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:25 smithi033 ceph-mon[31973]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:26.059 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:25 smithi096 ceph-mon[35946]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:26.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:25 smithi155 ceph-mon[35804]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:27.862 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:27 smithi096 ceph-mon[35946]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:28.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:27 smithi033 ceph-mon[31973]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:28.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:27 smithi155 ceph-mon[35804]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:29.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:29 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:29.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:29 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:29.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:29 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:49:29.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:29 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:29.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:29 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:29.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:29 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:49:29.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:29 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:29.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:29 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:29.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:29 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:49:30.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:30 smithi033 ceph-mon[31973]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:30.386 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:30 smithi096 ceph-mon[35946]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:30.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:30 smithi155 ceph-mon[35804]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:31.669 INFO:teuthology.orchestra.run.smithi096.stdout:Created osd(s) 10 on host 'smithi096' 2024-03-19T02:49:31.670 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:31.668+0000 7fe4b2ffd700 1 -- 172.21.15.96:0/4192575467 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fe4bc062750 con 0x7fe4a80a2890 2024-03-19T02:49:31.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:31.670+0000 7fe4c3cc1700 1 -- 172.21.15.96:0/4192575467 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe4a80a2890 msgr2=0x7fe4a80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:31.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:31.670+0000 7fe4c3cc1700 1 --2- 172.21.15.96:0/4192575467 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe4a80a2890 0x7fe4a80a4d50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8005fd0 tx=0x7fe4b8005dc0 comp rx=0 tx=0).stop 2024-03-19T02:49:31.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:31.670+0000 7fe4c3cc1700 1 -- 172.21.15.96:0/4192575467 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe4bc102440 msgr2=0x7fe4bc19d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:31.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:31.670+0000 7fe4c3cc1700 1 --2- 172.21.15.96:0/4192575467 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe4bc102440 0x7fe4bc19d200 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac00b530 tx=0x7fe4ac00b8f0 comp rx=0 tx=0).stop 2024-03-19T02:49:31.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:31.670+0000 7fe4c3cc1700 1 -- 172.21.15.96:0/4192575467 shutdown_connections 2024-03-19T02:49:31.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:31.670+0000 7fe4c3cc1700 1 --2- 172.21.15.96:0/4192575467 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe4a80a2890 0x7fe4a80a4d50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:31.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:31.670+0000 7fe4c3cc1700 1 --2- 172.21.15.96:0/4192575467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4bc102de0 0x7fe4bc19f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:31.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:31.670+0000 7fe4c3cc1700 1 --2- 172.21.15.96:0/4192575467 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe4bc101240 0x7fe4bc06b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:31.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:31.670+0000 7fe4c3cc1700 1 --2- 172.21.15.96:0/4192575467 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe4bc102440 0x7fe4bc19d200 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:31.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:31.670+0000 7fe4c3cc1700 1 -- 172.21.15.96:0/4192575467 >> 172.21.15.96:0/4192575467 conn(0x7fe4bc0fc7e0 msgr2=0x7fe4bc105790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:31.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:31.670+0000 7fe4c3cc1700 1 -- 172.21.15.96:0/4192575467 shutdown_connections 2024-03-19T02:49:31.672 INFO:teuthology.orchestra.run.smithi096.stderr:2024-03-19T02:49:31.670+0000 7fe4c3cc1700 1 -- 172.21.15.96:0/4192575467 wait complete. 2024-03-19T02:49:32.501 DEBUG:teuthology.orchestra.run.smithi096:osd.10> sudo journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.10.service 2024-03-19T02:49:32.503 INFO:tasks.cephadm:Deploying osd.11 on smithi155 with /dev/vg_nvme/lv_1... 2024-03-19T02:49:32.503 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-19T02:49:32.757 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T02:49:32.878 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:32 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:32.878 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:32 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:32.879 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:32 smithi155 ceph-mon[35804]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:32.879 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:32 smithi155 ceph-mon[35804]: from='osd.10 [v2:172.21.15.96:6824/1404898482,v1:172.21.15.96:6825/1404898482]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-19T02:49:32.879 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:32 smithi155 ceph-mon[35804]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-19T02:49:32.879 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:32 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:32.879 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:32 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:32.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:32 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:32.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:32 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:32.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:32 smithi096 ceph-mon[35946]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:32.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:32 smithi096 ceph-mon[35946]: from='osd.10 [v2:172.21.15.96:6824/1404898482,v1:172.21.15.96:6825/1404898482]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-19T02:49:32.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:32 smithi096 ceph-mon[35946]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-19T02:49:32.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:32 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:32.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:32 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:33.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:32 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:33.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:32 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:33.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:32 smithi033 ceph-mon[31973]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:33.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:32 smithi033 ceph-mon[31973]: from='osd.10 [v2:172.21.15.96:6824/1404898482,v1:172.21.15.96:6825/1404898482]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-19T02:49:33.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:32 smithi033 ceph-mon[31973]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-19T02:49:33.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:32 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:33.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:32 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:33.930 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:33 smithi033 ceph-mon[31973]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-19T02:49:33.930 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:33 smithi033 ceph-mon[31973]: osdmap e72: 11 total, 10 up, 11 in 2024-03-19T02:49:33.930 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:33 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:49:33.930 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:33 smithi033 ceph-mon[31973]: from='osd.10 [v2:172.21.15.96:6824/1404898482,v1:172.21.15.96:6825/1404898482]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-03-19T02:49:33.930 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:33 smithi033 ceph-mon[31973]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-03-19T02:49:33.962 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:33 smithi096 ceph-mon[35946]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-19T02:49:33.963 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:33 smithi096 ceph-mon[35946]: osdmap e72: 11 total, 10 up, 11 in 2024-03-19T02:49:33.963 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:33 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:49:33.963 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:33 smithi096 ceph-mon[35946]: from='osd.10 [v2:172.21.15.96:6824/1404898482,v1:172.21.15.96:6825/1404898482]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-03-19T02:49:33.963 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:33 smithi096 ceph-mon[35946]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-03-19T02:49:33.963 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 02:49:33 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T02:49:33.666+0000 7f25a8a88700 -1 osd.10 0 waiting for initial osdmap 2024-03-19T02:49:33.963 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 02:49:33 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T02:49:33.677+0000 7f25a487b700 -1 osd.10 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T02:49:34.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:33 smithi155 ceph-mon[35804]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-19T02:49:34.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:33 smithi155 ceph-mon[35804]: osdmap e72: 11 total, 10 up, 11 in 2024-03-19T02:49:34.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:33 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:49:34.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:33 smithi155 ceph-mon[35804]: from='osd.10 [v2:172.21.15.96:6824/1404898482,v1:172.21.15.96:6825/1404898482]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-03-19T02:49:34.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:33 smithi155 ceph-mon[35804]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]: dispatch 2024-03-19T02:49:34.814 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-19T02:49:34.814 INFO:teuthology.orchestra.run.smithi155.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-19T02:49:34.814 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10+0 records in 2024-03-19T02:49:34.814 INFO:teuthology.orchestra.run.smithi155.stderr:10+0 records out 2024-03-19T02:49:34.815 INFO:teuthology.orchestra.run.smithi155.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128611 s, 815 MB/s 2024-03-19T02:49:34.815 INFO:teuthology.orchestra.run.smithi155.stderr:--> Zapping successful for: 2024-03-19T02:49:34.951 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:34 smithi096 ceph-mon[35946]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-03-19T02:49:34.951 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:34 smithi096 ceph-mon[35946]: osdmap e73: 11 total, 10 up, 11 in 2024-03-19T02:49:34.951 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:34 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:49:34.951 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:34 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:49:34.951 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:34 smithi096 ceph-mon[35946]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:34.951 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:34 smithi096 ceph-mon[35946]: osd.10 [v2:172.21.15.96:6824/1404898482,v1:172.21.15.96:6825/1404898482] boot 2024-03-19T02:49:34.952 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:34 smithi096 ceph-mon[35946]: osdmap e74: 11 total, 11 up, 11 in 2024-03-19T02:49:34.952 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:34 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:49:35.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:34 smithi155 ceph-mon[35804]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-03-19T02:49:35.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:34 smithi155 ceph-mon[35804]: osdmap e73: 11 total, 10 up, 11 in 2024-03-19T02:49:35.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:34 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:49:35.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:34 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:49:35.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:34 smithi155 ceph-mon[35804]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:35.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:34 smithi155 ceph-mon[35804]: osd.10 [v2:172.21.15.96:6824/1404898482,v1:172.21.15.96:6825/1404898482] boot 2024-03-19T02:49:35.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:34 smithi155 ceph-mon[35804]: osdmap e74: 11 total, 11 up, 11 in 2024-03-19T02:49:35.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:34 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:49:35.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:34 smithi033 ceph-mon[31973]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi096", "root=default"]}]': finished 2024-03-19T02:49:35.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:34 smithi033 ceph-mon[31973]: osdmap e73: 11 total, 10 up, 11 in 2024-03-19T02:49:35.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:34 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:49:35.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:34 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:49:35.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:34 smithi033 ceph-mon[31973]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-03-19T02:49:35.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:34 smithi033 ceph-mon[31973]: osd.10 [v2:172.21.15.96:6824/1404898482,v1:172.21.15.96:6825/1404898482] boot 2024-03-19T02:49:35.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:34 smithi033 ceph-mon[31973]: osdmap e74: 11 total, 11 up, 11 in 2024-03-19T02:49:35.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:34 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:49:35.581 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi155:vg_nvme/lv_1 2024-03-19T02:49:35.834 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T02:49:35.861 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:35 smithi155 ceph-mon[35804]: purged_snaps scrub starts 2024-03-19T02:49:35.861 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:35 smithi155 ceph-mon[35804]: purged_snaps scrub ok 2024-03-19T02:49:35.861 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:35 smithi155 ceph-mon[35804]: osdmap e75: 11 total, 11 up, 11 in 2024-03-19T02:49:35.963 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:35 smithi096 ceph-mon[35946]: purged_snaps scrub starts 2024-03-19T02:49:35.964 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:35 smithi096 ceph-mon[35946]: purged_snaps scrub ok 2024-03-19T02:49:35.964 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:35 smithi096 ceph-mon[35946]: osdmap e75: 11 total, 11 up, 11 in 2024-03-19T02:49:36.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:35 smithi033 ceph-mon[31973]: purged_snaps scrub starts 2024-03-19T02:49:36.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:35 smithi033 ceph-mon[31973]: purged_snaps scrub ok 2024-03-19T02:49:36.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:35 smithi033 ceph-mon[31973]: osdmap e75: 11 total, 11 up, 11 in 2024-03-19T02:49:36.925 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:36 smithi155 ceph-mon[35804]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:49:36.925 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:36 smithi155 ceph-mon[35804]: osdmap e76: 11 total, 11 up, 11 in 2024-03-19T02:49:36.925 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:36 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:36.925 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:36 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:36.925 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:36 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:49:36.925 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:36 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:49:36.925 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:36 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:49:36.925 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:36 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:49:36.925 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:36 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:36.925 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:36 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:49:36.926 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:36 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:49:36.926 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:36 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:36.926 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:36 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:49:36.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:36 smithi096 ceph-mon[35946]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:49:36.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:36 smithi096 ceph-mon[35946]: osdmap e76: 11 total, 11 up, 11 in 2024-03-19T02:49:36.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:36 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:36.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:36 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:36.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:36 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:49:36.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:36 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:49:36.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:36 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:49:36.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:36 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:49:36.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:36 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:36.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:36 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:49:36.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:36 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:49:36.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:36 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:36.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:36 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:49:37.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:36 smithi033 ceph-mon[31973]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:49:37.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:36 smithi033 ceph-mon[31973]: osdmap e76: 11 total, 11 up, 11 in 2024-03-19T02:49:37.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:36 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:37.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:36 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:37.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:36 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:49:37.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:36 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:49:37.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:36 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:49:37.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:36 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:49:37.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:36 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:37.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:36 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:49:37.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:36 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:49:37.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:36 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:37.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:36 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:49:37.458 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.455+0000 7f2481bdc700 1 -- 172.21.15.155:0/803976503 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f247c1012b0 msgr2=0x7f247c1016d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:37.458 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.455+0000 7f2481bdc700 1 --2- 172.21.15.155:0/803976503 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f247c1012b0 0x7f247c1016d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f246c009990 tx=0x7f246c009ca0 comp rx=0 tx=0).stop 2024-03-19T02:49:37.458 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.455+0000 7f2481bdc700 1 -- 172.21.15.155:0/803976503 shutdown_connections 2024-03-19T02:49:37.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.455+0000 7f2481bdc700 1 --2- 172.21.15.155:0/803976503 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f247c102e50 0x7f247c109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:37.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.455+0000 7f2481bdc700 1 --2- 172.21.15.155:0/803976503 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f247c1012b0 0x7f247c1016d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:37.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.455+0000 7f2481bdc700 1 --2- 172.21.15.155:0/803976503 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f247c1024b0 0x7f247c102910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:37.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.455+0000 7f2481bdc700 1 -- 172.21.15.155:0/803976503 >> 172.21.15.155:0/803976503 conn(0x7f247c0fc870 msgr2=0x7f247c0fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:37.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.455+0000 7f2481bdc700 1 -- 172.21.15.155:0/803976503 shutdown_connections 2024-03-19T02:49:37.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.455+0000 7f2481bdc700 1 -- 172.21.15.155:0/803976503 wait complete. 2024-03-19T02:49:37.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.456+0000 7f2481bdc700 1 Processor -- start 2024-03-19T02:49:37.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.456+0000 7f2481bdc700 1 -- start start 2024-03-19T02:49:37.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.456+0000 7f2481bdc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f247c1012b0 0x7f247c1984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:37.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.456+0000 7f2481bdc700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f247c1024b0 0x7f247c1989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:37.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.456+0000 7f2481bdc700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f247c102e50 0x7f247c19fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:37.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.456+0000 7f2481bdc700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f247c10ccd0 con 0x7f247c102e50 2024-03-19T02:49:37.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.456+0000 7f2481bdc700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f247c10cb50 con 0x7f247c1012b0 2024-03-19T02:49:37.460 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.456+0000 7f2481bdc700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f247c10ce50 con 0x7f247c1024b0 2024-03-19T02:49:37.460 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.457+0000 7f2472ffd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f247c1024b0 0x7f247c1989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:37.460 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.457+0000 7f2472ffd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f247c1024b0 0x7f247c1989e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:40582/0 (socket says 172.21.15.155:40582) 2024-03-19T02:49:37.460 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.457+0000 7f2472ffd700 1 -- 172.21.15.155:0/1688143766 learned_addr learned my addr 172.21.15.155:0/1688143766 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-19T02:49:37.460 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.457+0000 7f247b7fe700 1 --2- 172.21.15.155:0/1688143766 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f247c1012b0 0x7f247c1984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:37.460 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.457+0000 7f247bfff700 1 --2- 172.21.15.155:0/1688143766 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f247c102e50 0x7f247c19fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:37.460 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.457+0000 7f2472ffd700 1 -- 172.21.15.155:0/1688143766 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f247c1012b0 msgr2=0x7f247c1984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:37.460 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.457+0000 7f2472ffd700 1 --2- 172.21.15.155:0/1688143766 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f247c1012b0 0x7f247c1984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:37.460 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.457+0000 7f2472ffd700 1 -- 172.21.15.155:0/1688143766 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f247c102e50 msgr2=0x7f247c19fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:37.460 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.457+0000 7f2472ffd700 1 --2- 172.21.15.155:0/1688143766 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f247c102e50 0x7f247c19fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:37.460 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.457+0000 7f2472ffd700 1 -- 172.21.15.155:0/1688143766 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f247c1a00b0 con 0x7f247c1024b0 2024-03-19T02:49:37.460 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.457+0000 7f2472ffd700 1 --2- 172.21.15.155:0/1688143766 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f247c1024b0 0x7f247c1989e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f246400d9b0 tx=0x7f246400dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:37.460 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.457+0000 7f24797fa700 1 -- 172.21.15.155:0/1688143766 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2464017070 con 0x7f247c1024b0 2024-03-19T02:49:37.460 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.457+0000 7f24797fa700 1 -- 172.21.15.155:0/1688143766 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2464005020 con 0x7f247c1024b0 2024-03-19T02:49:37.460 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.457+0000 7f2481bdc700 1 -- 172.21.15.155:0/1688143766 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f247c1a03a0 con 0x7f247c1024b0 2024-03-19T02:49:37.461 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.457+0000 7f24797fa700 1 -- 172.21.15.155:0/1688143766 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2464009a20 con 0x7f247c1024b0 2024-03-19T02:49:37.462 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.457+0000 7f2481bdc700 1 -- 172.21.15.155:0/1688143766 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f247c1a0890 con 0x7f247c1024b0 2024-03-19T02:49:37.462 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.458+0000 7f2481bdc700 1 -- 172.21.15.155:0/1688143766 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f247c068fd0 con 0x7f247c1024b0 2024-03-19T02:49:37.462 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.459+0000 7f24797fa700 1 -- 172.21.15.155:0/1688143766 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f24640051d0 con 0x7f247c1024b0 2024-03-19T02:49:37.462 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.460+0000 7f24797fa700 1 --2- 172.21.15.155:0/1688143766 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f24680a2890 0x7f24680a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:49:37.462 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.460+0000 7f24797fa700 1 -- 172.21.15.155:0/1688143766 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6723+0+0 (secure 0 0 0) 0x7f24640ce630 con 0x7f247c1024b0 2024-03-19T02:49:37.462 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.460+0000 7f247b7fe700 1 --2- 172.21.15.155:0/1688143766 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f24680a2890 0x7f24680a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:49:37.462 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.461+0000 7f247b7fe700 1 --2- 172.21.15.155:0/1688143766 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f24680a2890 0x7f24680a4d50 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f246c0037f0 tx=0x7f246c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:49:37.464 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.464+0000 7f24797fa700 1 -- 172.21.15.155:0/1688143766 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2464020020 con 0x7f247c1024b0 2024-03-19T02:49:37.667 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:37.666+0000 7f2481bdc700 1 -- 172.21.15.155:0/1688143766 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f247c0fea60 con 0x7f24680a2890 2024-03-19T02:49:37.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:37 smithi096 ceph-mon[35946]: Detected new or changed devices on smithi096 2024-03-19T02:49:37.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:37 smithi096 ceph-mon[35946]: Adjusting osd_memory_target on smithi096 to 4272M 2024-03-19T02:49:37.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:37 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:49:37.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:37 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:49:37.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:37 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:37.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:37 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:49:37.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:37 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:49:37.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:37 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:49:38.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:37 smithi155 ceph-mon[35804]: Detected new or changed devices on smithi096 2024-03-19T02:49:38.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:37 smithi155 ceph-mon[35804]: Adjusting osd_memory_target on smithi096 to 4272M 2024-03-19T02:49:38.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:37 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:49:38.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:37 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:49:38.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:37 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:38.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:37 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:49:38.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:37 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:49:38.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:37 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:49:38.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:37 smithi033 ceph-mon[31973]: Detected new or changed devices on smithi096 2024-03-19T02:49:38.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:37 smithi033 ceph-mon[31973]: Adjusting osd_memory_target on smithi096 to 4272M 2024-03-19T02:49:38.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:37 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:49:38.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:37 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:49:38.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:37 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:38.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:37 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T02:49:38.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:37 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T02:49:38.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:37 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:49:38.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:38 smithi096 ceph-mon[35946]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:49:38.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:38 smithi096 ceph-mon[35946]: osdmap e77: 11 total, 11 up, 11 in 2024-03-19T02:49:38.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:38 smithi096 ceph-mon[35946]: pgmap v246: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-03-19T02:49:39.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:38 smithi155 ceph-mon[35804]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:49:39.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:38 smithi155 ceph-mon[35804]: osdmap e77: 11 total, 11 up, 11 in 2024-03-19T02:49:39.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:38 smithi155 ceph-mon[35804]: pgmap v246: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-03-19T02:49:39.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:38 smithi033 ceph-mon[31973]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi155:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:49:39.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:38 smithi033 ceph-mon[31973]: osdmap e77: 11 total, 11 up, 11 in 2024-03-19T02:49:39.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:38 smithi033 ceph-mon[31973]: pgmap v246: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-03-19T02:49:39.823 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:39 smithi155 ceph-mon[35804]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-19T02:49:39.823 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:39 smithi155 ceph-mon[35804]: osdmap e78: 11 total, 11 up, 11 in 2024-03-19T02:49:39.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:39 smithi096 ceph-mon[35946]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-19T02:49:39.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:39 smithi096 ceph-mon[35946]: osdmap e78: 11 total, 11 up, 11 in 2024-03-19T02:49:40.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:39 smithi033 ceph-mon[31973]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-19T02:49:40.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:39 smithi033 ceph-mon[31973]: osdmap e78: 11 total, 11 up, 11 in 2024-03-19T02:49:40.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:40 smithi096 ceph-mon[35946]: pgmap v248: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-03-19T02:49:40.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:40 smithi096 ceph-mon[35946]: from='client.? 172.21.15.155:0/1304776608' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0903ed2-cc1e-4c18-8731-1f0ee4dd0457"}]: dispatch 2024-03-19T02:49:40.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:40 smithi096 ceph-mon[35946]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0903ed2-cc1e-4c18-8731-1f0ee4dd0457"}]: dispatch 2024-03-19T02:49:40.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:40 smithi096 ceph-mon[35946]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0903ed2-cc1e-4c18-8731-1f0ee4dd0457"}]': finished 2024-03-19T02:49:40.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:40 smithi096 ceph-mon[35946]: osdmap e79: 12 total, 11 up, 12 in 2024-03-19T02:49:40.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:40 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:49:41.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:40 smithi155 ceph-mon[35804]: pgmap v248: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-03-19T02:49:41.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:40 smithi155 ceph-mon[35804]: from='client.? 172.21.15.155:0/1304776608' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0903ed2-cc1e-4c18-8731-1f0ee4dd0457"}]: dispatch 2024-03-19T02:49:41.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:40 smithi155 ceph-mon[35804]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0903ed2-cc1e-4c18-8731-1f0ee4dd0457"}]: dispatch 2024-03-19T02:49:41.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:40 smithi155 ceph-mon[35804]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0903ed2-cc1e-4c18-8731-1f0ee4dd0457"}]': finished 2024-03-19T02:49:41.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:40 smithi155 ceph-mon[35804]: osdmap e79: 12 total, 11 up, 12 in 2024-03-19T02:49:41.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:49:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:40 smithi033 ceph-mon[31973]: pgmap v248: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-03-19T02:49:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:40 smithi033 ceph-mon[31973]: from='client.? 172.21.15.155:0/1304776608' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0903ed2-cc1e-4c18-8731-1f0ee4dd0457"}]: dispatch 2024-03-19T02:49:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:40 smithi033 ceph-mon[31973]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0903ed2-cc1e-4c18-8731-1f0ee4dd0457"}]: dispatch 2024-03-19T02:49:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:40 smithi033 ceph-mon[31973]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0903ed2-cc1e-4c18-8731-1f0ee4dd0457"}]': finished 2024-03-19T02:49:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:40 smithi033 ceph-mon[31973]: osdmap e79: 12 total, 11 up, 12 in 2024-03-19T02:49:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:49:41.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:41 smithi096 ceph-mon[35946]: from='client.? 172.21.15.155:0/3688034954' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:49:42.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:41 smithi155 ceph-mon[35804]: from='client.? 172.21.15.155:0/3688034954' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:49:42.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:41 smithi033 ceph-mon[31973]: from='client.? 172.21.15.155:0/3688034954' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T02:49:42.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:42 smithi096 ceph-mon[35946]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-19T02:49:42.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:42 smithi096 ceph-mon[35946]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-19T02:49:42.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:42 smithi096 ceph-mon[35946]: Cluster is now healthy 2024-03-19T02:49:43.019 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:42 smithi155 ceph-mon[35804]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-19T02:49:43.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:42 smithi155 ceph-mon[35804]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-19T02:49:43.020 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:42 smithi155 ceph-mon[35804]: Cluster is now healthy 2024-03-19T02:49:43.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:42 smithi033 ceph-mon[31973]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-19T02:49:43.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:42 smithi033 ceph-mon[31973]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-19T02:49:43.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:42 smithi033 ceph-mon[31973]: Cluster is now healthy 2024-03-19T02:49:45.013 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:44 smithi155 ceph-mon[35804]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-19T02:49:45.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:44 smithi033 ceph-mon[31973]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-19T02:49:45.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:44 smithi096 ceph-mon[35946]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-19T02:49:46.981 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:46 smithi155 ceph-mon[35804]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-19T02:49:47.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:46 smithi033 ceph-mon[31973]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-19T02:49:47.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:46 smithi096 ceph-mon[35946]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-19T02:49:48.981 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:48 smithi155 ceph-mon[35804]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-19T02:49:48.981 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:48 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-19T02:49:48.981 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:48 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:49:49.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:48 smithi033 ceph-mon[31973]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-19T02:49:49.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:48 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-19T02:49:49.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:48 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:49:49.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:48 smithi096 ceph-mon[35946]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-19T02:49:49.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:48 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-19T02:49:49.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:48 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:49:49.936 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:49 smithi155 ceph-mon[35804]: Deploying daemon osd.11 on smithi155 2024-03-19T02:49:50.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:49 smithi033 ceph-mon[31973]: Deploying daemon osd.11 on smithi155 2024-03-19T02:49:50.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:49 smithi096 ceph-mon[35946]: Deploying daemon osd.11 on smithi155 2024-03-19T02:49:51.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:50 smithi155 ceph-mon[35804]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-19T02:49:51.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:50 smithi033 ceph-mon[31973]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-19T02:49:51.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:50 smithi096 ceph-mon[35946]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-19T02:49:51.929 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:51 smithi155 ceph-mon[35804]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-19T02:49:52.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:51 smithi033 ceph-mon[31973]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-19T02:49:52.116 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:51 smithi096 ceph-mon[35946]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-19T02:49:53.928 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:53 smithi155 ceph-mon[35804]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:49:54.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:53 smithi033 ceph-mon[31973]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:49:54.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:53 smithi096 ceph-mon[35946]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:49:55.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:55 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:55.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:55 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:55.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:55 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:49:55.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:55 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:55.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:55 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:55.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:55 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:49:55.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:55 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:55.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:55 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:55.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:55 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:49:56.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:56 smithi096 ceph-mon[35946]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:49:56.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:56 smithi155 ceph-mon[35804]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:49:56.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:56 smithi033 ceph-mon[31973]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:49:57.786 INFO:teuthology.orchestra.run.smithi155.stdout:Created osd(s) 11 on host 'smithi155' 2024-03-19T02:49:57.787 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:57.785+0000 7f24797fa700 1 -- 172.21.15.155:0/1688143766 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f247c0fea60 con 0x7f24680a2890 2024-03-19T02:49:57.789 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:57.788+0000 7f2481bdc700 1 -- 172.21.15.155:0/1688143766 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f24680a2890 msgr2=0x7f24680a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:57.789 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:57.788+0000 7f2481bdc700 1 --2- 172.21.15.155:0/1688143766 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f24680a2890 0x7f24680a4d50 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f246c0037f0 tx=0x7f246c005fd0 comp rx=0 tx=0).stop 2024-03-19T02:49:57.789 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:57.788+0000 7f2481bdc700 1 -- 172.21.15.155:0/1688143766 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f247c1024b0 msgr2=0x7f247c1989e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:49:57.789 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:57.788+0000 7f2481bdc700 1 --2- 172.21.15.155:0/1688143766 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f247c1024b0 0x7f247c1989e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f246400d9b0 tx=0x7f246400dcc0 comp rx=0 tx=0).stop 2024-03-19T02:49:57.789 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:57.788+0000 7f2481bdc700 1 -- 172.21.15.155:0/1688143766 shutdown_connections 2024-03-19T02:49:57.789 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:57.788+0000 7f2481bdc700 1 --2- 172.21.15.155:0/1688143766 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f24680a2890 0x7f24680a4d50 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:57.789 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:57.788+0000 7f2481bdc700 1 --2- 172.21.15.155:0/1688143766 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f247c102e50 0x7f247c19fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:57.789 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:57.788+0000 7f2481bdc700 1 --2- 172.21.15.155:0/1688143766 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f247c1012b0 0x7f247c1984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:57.789 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:57.788+0000 7f2481bdc700 1 --2- 172.21.15.155:0/1688143766 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f247c1024b0 0x7f247c1989e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:49:57.789 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:57.788+0000 7f2481bdc700 1 -- 172.21.15.155:0/1688143766 >> 172.21.15.155:0/1688143766 conn(0x7f247c0fc870 msgr2=0x7f247c0fe350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:49:57.790 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:57.788+0000 7f2481bdc700 1 -- 172.21.15.155:0/1688143766 shutdown_connections 2024-03-19T02:49:57.790 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:49:57.788+0000 7f2481bdc700 1 -- 172.21.15.155:0/1688143766 wait complete. 2024-03-19T02:49:58.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:57 smithi155 ceph-mon[35804]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:49:58.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:57 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:58.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:57 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:58.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:57 smithi096 ceph-mon[35946]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:49:58.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:57 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:58.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:57 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:58.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:57 smithi033 ceph-mon[31973]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:49:58.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:57 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:58.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:57 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:58.414 DEBUG:teuthology.orchestra.run.smithi155:osd.11> sudo journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.11.service 2024-03-19T02:49:58.417 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-19T02:49:58.417 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-19T02:49:58.684 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:49:59.117 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:58 smithi033 ceph-mon[31973]: from='osd.11 [v2:172.21.15.155:6824/792352827,v1:172.21.15.155:6825/792352827]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-19T02:49:59.117 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:58 smithi033 ceph-mon[31973]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-19T02:49:59.117 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:58 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:59.117 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:58 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:59.179 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:58 smithi155 ceph-mon[35804]: from='osd.11 [v2:172.21.15.155:6824/792352827,v1:172.21.15.155:6825/792352827]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-19T02:49:59.179 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:58 smithi155 ceph-mon[35804]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-19T02:49:59.179 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:58 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:59.179 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:58 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:59.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:58 smithi096 ceph-mon[35946]: from='osd.11 [v2:172.21.15.155:6824/792352827,v1:172.21.15.155:6825/792352827]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-19T02:49:59.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:58 smithi096 ceph-mon[35946]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-19T02:49:59.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:58 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:49:59.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:58 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:00.061 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:59 smithi033 ceph-mon[31973]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-19T02:50:00.061 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:59 smithi033 ceph-mon[31973]: osdmap e80: 12 total, 11 up, 12 in 2024-03-19T02:50:00.061 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:59 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:50:00.061 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:59 smithi033 ceph-mon[31973]: from='osd.11 [v2:172.21.15.155:6824/792352827,v1:172.21.15.155:6825/792352827]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-19T02:50:00.061 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:59 smithi033 ceph-mon[31973]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-19T02:50:00.061 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:49:59 smithi033 ceph-mon[31973]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:50:00.154 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:59 smithi155 ceph-mon[35804]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-19T02:50:00.155 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:59 smithi155 ceph-mon[35804]: osdmap e80: 12 total, 11 up, 12 in 2024-03-19T02:50:00.155 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:59 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:50:00.155 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:59 smithi155 ceph-mon[35804]: from='osd.11 [v2:172.21.15.155:6824/792352827,v1:172.21.15.155:6825/792352827]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-19T02:50:00.155 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:59 smithi155 ceph-mon[35804]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-19T02:50:00.155 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:49:59 smithi155 ceph-mon[35804]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:50:00.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:59 smithi096 ceph-mon[35946]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-19T02:50:00.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:59 smithi096 ceph-mon[35946]: osdmap e80: 12 total, 11 up, 12 in 2024-03-19T02:50:00.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:59 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:50:00.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:59 smithi096 ceph-mon[35946]: from='osd.11 [v2:172.21.15.155:6824/792352827,v1:172.21.15.155:6825/792352827]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-19T02:50:00.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:59 smithi096 ceph-mon[35946]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]: dispatch 2024-03-19T02:50:00.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:49:59 smithi096 ceph-mon[35946]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:50:00.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.441+0000 7fd071311700 1 -- 172.21.15.33:0/2840950758 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd06c101240 msgr2=0x7fd06c101660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:00.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.441+0000 7fd071311700 1 --2- 172.21.15.33:0/2840950758 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd06c101240 0x7fd06c101660 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd05c0099c0 tx=0x7fd05c009cd0 comp rx=0 tx=0).stop 2024-03-19T02:50:00.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.441+0000 7fd071311700 1 -- 172.21.15.33:0/2840950758 shutdown_connections 2024-03-19T02:50:00.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.441+0000 7fd071311700 1 --2- 172.21.15.33:0/2840950758 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd06c102de0 0x7fd06c109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:00.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.441+0000 7fd071311700 1 --2- 172.21.15.33:0/2840950758 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd06c101240 0x7fd06c101660 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:00.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.441+0000 7fd071311700 1 --2- 172.21.15.33:0/2840950758 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd06c102440 0x7fd06c1028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:00.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.441+0000 7fd071311700 1 -- 172.21.15.33:0/2840950758 >> 172.21.15.33:0/2840950758 conn(0x7fd06c0fc7e0 msgr2=0x7fd06c0fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:00.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.441+0000 7fd071311700 1 -- 172.21.15.33:0/2840950758 shutdown_connections 2024-03-19T02:50:00.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.441+0000 7fd071311700 1 -- 172.21.15.33:0/2840950758 wait complete. 2024-03-19T02:50:00.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.442+0000 7fd071311700 1 Processor -- start 2024-03-19T02:50:00.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.442+0000 7fd071311700 1 -- start start 2024-03-19T02:50:00.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.442+0000 7fd071311700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd06c101240 0x7fd06c06aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:00.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.442+0000 7fd071311700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd06c102440 0x7fd06c19d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:00.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.442+0000 7fd071311700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd06c102de0 0x7fd06c19f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:00.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.442+0000 7fd071311700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c10ccb0 con 0x7fd06c102de0 2024-03-19T02:50:00.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.442+0000 7fd071311700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c10cb30 con 0x7fd06c101240 2024-03-19T02:50:00.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.442+0000 7fd071311700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c10ce30 con 0x7fd06c102440 2024-03-19T02:50:00.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.442+0000 7fd06a7fc700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd06c102440 0x7fd06c19d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:00.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.442+0000 7fd06b7fe700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd06c102de0 0x7fd06c19f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:00.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.442+0000 7fd06a7fc700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd06c102440 0x7fd06c19d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:46288/0 (socket says 172.21.15.33:46288) 2024-03-19T02:50:00.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.442+0000 7fd06a7fc700 1 -- 172.21.15.33:0/2754707519 learned_addr learned my addr 172.21.15.33:0/2754707519 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:00.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.442+0000 7fd06affd700 1 --2- 172.21.15.33:0/2754707519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd06c101240 0x7fd06c06aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:00.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.443+0000 7fd06a7fc700 1 -- 172.21.15.33:0/2754707519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd06c101240 msgr2=0x7fd06c06aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:00.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.443+0000 7fd06a7fc700 1 --2- 172.21.15.33:0/2754707519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd06c101240 0x7fd06c06aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:00.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.443+0000 7fd06a7fc700 1 -- 172.21.15.33:0/2754707519 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd06c102de0 msgr2=0x7fd06c19f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:00.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.443+0000 7fd06a7fc700 1 --2- 172.21.15.33:0/2754707519 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd06c102de0 0x7fd06c19f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:00.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.443+0000 7fd06a7fc700 1 -- 172.21.15.33:0/2754707519 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd06c1a0160 con 0x7fd06c102440 2024-03-19T02:50:00.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.443+0000 7fd06b7fe700 1 --2- 172.21.15.33:0/2754707519 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd06c102de0 0x7fd06c19f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:00.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.443+0000 7fd06a7fc700 1 --2- 172.21.15.33:0/2754707519 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd06c102440 0x7fd06c19d2a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd05400e9a0 tx=0x7fd05400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:00.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.443+0000 7fd063fff700 1 -- 172.21.15.33:0/2754707519 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05400ca10 con 0x7fd06c102440 2024-03-19T02:50:00.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.443+0000 7fd071311700 1 -- 172.21.15.33:0/2754707519 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd06c1a0450 con 0x7fd06c102440 2024-03-19T02:50:00.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.443+0000 7fd063fff700 1 -- 172.21.15.33:0/2754707519 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd05400cbb0 con 0x7fd06c102440 2024-03-19T02:50:00.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.443+0000 7fd063fff700 1 -- 172.21.15.33:0/2754707519 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd054018680 con 0x7fd06c102440 2024-03-19T02:50:00.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.443+0000 7fd071311700 1 -- 172.21.15.33:0/2754707519 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd06c1a09e0 con 0x7fd06c102440 2024-03-19T02:50:00.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.444+0000 7fd071311700 1 -- 172.21.15.33:0/2754707519 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd06c068fd0 con 0x7fd06c102440 2024-03-19T02:50:00.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.445+0000 7fd063fff700 1 -- 172.21.15.33:0/2754707519 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fd054010430 con 0x7fd06c102440 2024-03-19T02:50:00.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.446+0000 7fd063fff700 1 --2- 172.21.15.33:0/2754707519 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fd0580a2910 0x7fd0580a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:00.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.446+0000 7fd06affd700 1 --2- 172.21.15.33:0/2754707519 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fd0580a2910 0x7fd0580a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:00.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.446+0000 7fd063fff700 1 -- 172.21.15.33:0/2754707519 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 6847+0+0 (secure 0 0 0) 0x7fd0540ce910 con 0x7fd06c102440 2024-03-19T02:50:00.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.446+0000 7fd06affd700 1 --2- 172.21.15.33:0/2754707519 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fd0580a2910 0x7fd0580a4dd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd05c000c00 tx=0x7fd05c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:00.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.454+0000 7fd063fff700 1 -- 172.21.15.33:0/2754707519 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd05409bed0 con 0x7fd06c102440 2024-03-19T02:50:00.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.663+0000 7fd071311700 1 -- 172.21.15.33:0/2754707519 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fd06c02dbb0 con 0x7fd06c102440 2024-03-19T02:50:00.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.664+0000 7fd063fff700 1 -- 172.21.15.33:0/2754707519 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7fd05409b7d0 con 0x7fd06c102440 2024-03-19T02:50:00.665 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:50:00.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.666+0000 7fd071311700 1 -- 172.21.15.33:0/2754707519 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fd0580a2910 msgr2=0x7fd0580a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:00.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.666+0000 7fd071311700 1 --2- 172.21.15.33:0/2754707519 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fd0580a2910 0x7fd0580a4dd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd05c000c00 tx=0x7fd05c005fb0 comp rx=0 tx=0).stop 2024-03-19T02:50:00.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.666+0000 7fd071311700 1 -- 172.21.15.33:0/2754707519 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd06c102440 msgr2=0x7fd06c19d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:00.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.666+0000 7fd071311700 1 --2- 172.21.15.33:0/2754707519 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd06c102440 0x7fd06c19d2a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd05400e9a0 tx=0x7fd05400ecb0 comp rx=0 tx=0).stop 2024-03-19T02:50:00.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.666+0000 7fd071311700 1 -- 172.21.15.33:0/2754707519 shutdown_connections 2024-03-19T02:50:00.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.666+0000 7fd071311700 1 --2- 172.21.15.33:0/2754707519 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fd0580a2910 0x7fd0580a4dd0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:00.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.666+0000 7fd071311700 1 --2- 172.21.15.33:0/2754707519 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd06c102de0 0x7fd06c19f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:00.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.666+0000 7fd071311700 1 --2- 172.21.15.33:0/2754707519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd06c101240 0x7fd06c06aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:00.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.666+0000 7fd071311700 1 --2- 172.21.15.33:0/2754707519 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd06c102440 0x7fd06c19d2a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:00.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.666+0000 7fd071311700 1 -- 172.21.15.33:0/2754707519 >> 172.21.15.33:0/2754707519 conn(0x7fd06c0fc7e0 msgr2=0x7fd06c105900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:00.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.666+0000 7fd071311700 1 -- 172.21.15.33:0/2754707519 shutdown_connections 2024-03-19T02:50:00.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:00.666+0000 7fd071311700 1 -- 172.21.15.33:0/2754707519 wait complete. 2024-03-19T02:50:00.767 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 02:50:00 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T02:50:00.536+0000 7f0a8c0cb700 -1 osd.11 0 waiting for initial osdmap 2024-03-19T02:50:00.767 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 02:50:00 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T02:50:00.552+0000 7f0a83eb6700 -1 osd.11 81 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T02:50:00.990 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:00 smithi033 ceph-mon[31973]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-03-19T02:50:00.990 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:00 smithi033 ceph-mon[31973]: osdmap e81: 12 total, 11 up, 12 in 2024-03-19T02:50:00.991 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:00 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:50:00.991 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:00 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:50:00.991 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:00 smithi033 ceph-mon[31973]: overall HEALTH_OK 2024-03-19T02:50:00.991 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:00 smithi033 ceph-mon[31973]: from='osd.11 ' entity='osd.11' 2024-03-19T02:50:00.991 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:00 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2754707519' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-19T02:50:01.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:00 smithi096 ceph-mon[35946]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-03-19T02:50:01.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:00 smithi096 ceph-mon[35946]: osdmap e81: 12 total, 11 up, 12 in 2024-03-19T02:50:01.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:00 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:50:01.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:00 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:50:01.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:00 smithi096 ceph-mon[35946]: overall HEALTH_OK 2024-03-19T02:50:01.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:00 smithi096 ceph-mon[35946]: from='osd.11 ' entity='osd.11' 2024-03-19T02:50:01.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:00 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2754707519' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-19T02:50:01.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:00 smithi155 ceph-mon[35804]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi155", "root=default"]}]': finished 2024-03-19T02:50:01.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:00 smithi155 ceph-mon[35804]: osdmap e81: 12 total, 11 up, 12 in 2024-03-19T02:50:01.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:00 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:50:01.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:00 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:50:01.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:00 smithi155 ceph-mon[35804]: overall HEALTH_OK 2024-03-19T02:50:01.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:00 smithi155 ceph-mon[35804]: from='osd.11 ' entity='osd.11' 2024-03-19T02:50:01.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:00 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2754707519' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-19T02:50:01.269 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":81,"num_osds":12,"num_up_osds":11,"osd_up_since":1710816574,"num_in_osds":12,"osd_in_since":1710816580,"num_remapped_pgs":0} 2024-03-19T02:50:01.955 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:01 smithi155 ceph-mon[35804]: purged_snaps scrub starts 2024-03-19T02:50:01.955 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:01 smithi155 ceph-mon[35804]: purged_snaps scrub ok 2024-03-19T02:50:01.955 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:01 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:50:01.955 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:01 smithi155 ceph-mon[35804]: osd.11 [v2:172.21.15.155:6824/792352827,v1:172.21.15.155:6825/792352827] boot 2024-03-19T02:50:01.955 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:01 smithi155 ceph-mon[35804]: osdmap e82: 12 total, 12 up, 12 in 2024-03-19T02:50:01.955 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:01 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:50:01.955 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:01 smithi155 ceph-mon[35804]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:50:02.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:01 smithi096 ceph-mon[35946]: purged_snaps scrub starts 2024-03-19T02:50:02.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:01 smithi096 ceph-mon[35946]: purged_snaps scrub ok 2024-03-19T02:50:02.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:01 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:50:02.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:01 smithi096 ceph-mon[35946]: osd.11 [v2:172.21.15.155:6824/792352827,v1:172.21.15.155:6825/792352827] boot 2024-03-19T02:50:02.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:01 smithi096 ceph-mon[35946]: osdmap e82: 12 total, 12 up, 12 in 2024-03-19T02:50:02.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:01 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:50:02.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:01 smithi096 ceph-mon[35946]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:50:02.270 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-19T02:50:02.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:01 smithi033 ceph-mon[31973]: purged_snaps scrub starts 2024-03-19T02:50:02.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:01 smithi033 ceph-mon[31973]: purged_snaps scrub ok 2024-03-19T02:50:02.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:01 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:50:02.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:01 smithi033 ceph-mon[31973]: osd.11 [v2:172.21.15.155:6824/792352827,v1:172.21.15.155:6825/792352827] boot 2024-03-19T02:50:02.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:01 smithi033 ceph-mon[31973]: osdmap e82: 12 total, 12 up, 12 in 2024-03-19T02:50:02.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:01 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:50:02.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:01 smithi033 ceph-mon[31973]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-19T02:50:02.521 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:50:03.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:03 smithi096 ceph-mon[35946]: Detected new or changed devices on smithi155 2024-03-19T02:50:03.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:03 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:03.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:03 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:03.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:03 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:50:03.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:03 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:50:03.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:03 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:50:03.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:03 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:50:03.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:03 smithi096 ceph-mon[35946]: Adjusting osd_memory_target on smithi155 to 4272M 2024-03-19T02:50:03.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:03 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:03.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:03 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:03.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:03 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:50:03.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:03 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:03.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:03 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:50:03.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:03 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:03.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:03 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:50:03.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:03 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:03.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:03 smithi155 ceph-mon[35804]: Detected new or changed devices on smithi155 2024-03-19T02:50:03.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:03 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:03.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:03 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:03.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:03 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:50:03.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:03 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:50:03.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:03 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:50:03.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:03 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:50:03.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:03 smithi155 ceph-mon[35804]: Adjusting osd_memory_target on smithi155 to 4272M 2024-03-19T02:50:03.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:03 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:03.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:03 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:03.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:03 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:50:03.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:03 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:03.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:03 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:50:03.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:03 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:03.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:03 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:50:03.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:03 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:03.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:03 smithi033 ceph-mon[31973]: Detected new or changed devices on smithi155 2024-03-19T02:50:03.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:03 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:03.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:03 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:03 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:50:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:03 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:50:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:03 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:50:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:03 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:50:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:03 smithi033 ceph-mon[31973]: Adjusting osd_memory_target on smithi155 to 4272M 2024-03-19T02:50:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:03 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:03 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:03 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:50:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:03 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:03 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:50:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:03 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:03 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:50:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:03 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:04.155 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:04 smithi033 ceph-mon[31973]: osdmap e83: 12 total, 12 up, 12 in 2024-03-19T02:50:04.156 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:04 smithi033 ceph-mon[31973]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:04.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.258+0000 7f2d07f0f700 1 -- 172.21.15.33:0/4206297122 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d00103560 msgr2=0x7f2d001039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:04.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.258+0000 7f2d07f0f700 1 --2- 172.21.15.33:0/4206297122 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d00103560 0x7f2d001039c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2cf40099c0 tx=0x7f2cf4009cd0 comp rx=0 tx=0).stop 2024-03-19T02:50:04.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.258+0000 7f2d07f0f700 1 -- 172.21.15.33:0/4206297122 shutdown_connections 2024-03-19T02:50:04.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.258+0000 7f2d07f0f700 1 --2- 172.21.15.33:0/4206297122 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d00103f00 0x7f2d0010a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:04.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.258+0000 7f2d07f0f700 1 --2- 172.21.15.33:0/4206297122 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d00102360 0x7f2d00102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:04.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.258+0000 7f2d07f0f700 1 --2- 172.21.15.33:0/4206297122 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d00103560 0x7f2d001039c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:04.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.258+0000 7f2d07f0f700 1 -- 172.21.15.33:0/4206297122 >> 172.21.15.33:0/4206297122 conn(0x7f2d000fd920 msgr2=0x7f2d000ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:04.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.258+0000 7f2d07f0f700 1 -- 172.21.15.33:0/4206297122 shutdown_connections 2024-03-19T02:50:04.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.258+0000 7f2d07f0f700 1 -- 172.21.15.33:0/4206297122 wait complete. 2024-03-19T02:50:04.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.259+0000 7f2d07f0f700 1 Processor -- start 2024-03-19T02:50:04.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.259+0000 7f2d07f0f700 1 -- start start 2024-03-19T02:50:04.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.259+0000 7f2d07f0f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d00102360 0x7f2d00198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:04.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.259+0000 7f2d07f0f700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d00103560 0x7f2d001989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:04.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.259+0000 7f2d07f0f700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d00103f00 0x7f2d0019fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:04.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.259+0000 7f2d07f0f700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2d0010de50 con 0x7f2d00103f00 2024-03-19T02:50:04.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.259+0000 7f2d07f0f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2d0010dcd0 con 0x7f2d00102360 2024-03-19T02:50:04.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.259+0000 7f2d07f0f700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2d0010dfd0 con 0x7f2d00103560 2024-03-19T02:50:04.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.259+0000 7f2d064ac700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d00103f00 0x7f2d0019fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:04.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.259+0000 7f2d064ac700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d00103f00 0x7f2d0019fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59336/0 (socket says 172.21.15.33:59336) 2024-03-19T02:50:04.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.259+0000 7f2d064ac700 1 -- 172.21.15.33:0/3828471905 learned_addr learned my addr 172.21.15.33:0/3828471905 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:04.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.259+0000 7f2d054aa700 1 --2- 172.21.15.33:0/3828471905 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d00103560 0x7f2d001989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:04.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.259+0000 7f2d05cab700 1 --2- 172.21.15.33:0/3828471905 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d00102360 0x7f2d00198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:04.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.259+0000 7f2d064ac700 1 -- 172.21.15.33:0/3828471905 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d00103560 msgr2=0x7f2d001989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:04.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.259+0000 7f2d064ac700 1 --2- 172.21.15.33:0/3828471905 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d00103560 0x7f2d001989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:04.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.259+0000 7f2d064ac700 1 -- 172.21.15.33:0/3828471905 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d00102360 msgr2=0x7f2d00198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:04.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.259+0000 7f2d064ac700 1 --2- 172.21.15.33:0/3828471905 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d00102360 0x7f2d00198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:04.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.259+0000 7f2d064ac700 1 -- 172.21.15.33:0/3828471905 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d001a00f0 con 0x7f2d00103f00 2024-03-19T02:50:04.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.260+0000 7f2d064ac700 1 --2- 172.21.15.33:0/3828471905 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d00103f00 0x7f2d0019fa50 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2cf800b580 tx=0x7f2cf800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:04.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.260+0000 7f2cfeffd700 1 -- 172.21.15.33:0/3828471905 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf8015020 con 0x7f2d00103f00 2024-03-19T02:50:04.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.260+0000 7f2cfeffd700 1 -- 172.21.15.33:0/3828471905 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2cf8011ce0 con 0x7f2d00103f00 2024-03-19T02:50:04.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.260+0000 7f2d07f0f700 1 -- 172.21.15.33:0/3828471905 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d001a03e0 con 0x7f2d00103f00 2024-03-19T02:50:04.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.260+0000 7f2cfeffd700 1 -- 172.21.15.33:0/3828471905 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf8010dd0 con 0x7f2d00103f00 2024-03-19T02:50:04.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.260+0000 7f2d07f0f700 1 -- 172.21.15.33:0/3828471905 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d001a09f0 con 0x7f2d00103f00 2024-03-19T02:50:04.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.261+0000 7f2d07f0f700 1 -- 172.21.15.33:0/3828471905 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d00068fd0 con 0x7f2d00103f00 2024-03-19T02:50:04.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.262+0000 7f2cfeffd700 1 -- 172.21.15.33:0/3828471905 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f2cf8018020 con 0x7f2d00103f00 2024-03-19T02:50:04.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.262+0000 7f2cfeffd700 1 --2- 172.21.15.33:0/3828471905 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2ce80a2910 0x7f2ce80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:04.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.263+0000 7f2d05cab700 1 --2- 172.21.15.33:0/3828471905 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2ce80a2910 0x7f2ce80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:04.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.263+0000 7f2cfeffd700 1 -- 172.21.15.33:0/3828471905 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f2cf80cdc60 con 0x7f2d00103f00 2024-03-19T02:50:04.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.263+0000 7f2d05cab700 1 --2- 172.21.15.33:0/3828471905 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2ce80a2910 0x7f2ce80a4dd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0007720 tx=0x7f2cf00076b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:04.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.267+0000 7f2cfeffd700 1 -- 172.21.15.33:0/3828471905 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2cf801f020 con 0x7f2d00103f00 2024-03-19T02:50:04.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:04 smithi096 ceph-mon[35946]: osdmap e83: 12 total, 12 up, 12 in 2024-03-19T02:50:04.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:04 smithi096 ceph-mon[35946]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:04.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.474+0000 7f2d07f0f700 1 -- 172.21.15.33:0/3828471905 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f2d0002dbb0 con 0x7f2d00103f00 2024-03-19T02:50:04.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.474+0000 7f2cfeffd700 1 -- 172.21.15.33:0/3828471905 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v83) ==== 74+0+133 (secure 0 0 0) 0x7f2cf809b1a0 con 0x7f2d00103f00 2024-03-19T02:50:04.475 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:50:04.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.476+0000 7f2d07f0f700 1 -- 172.21.15.33:0/3828471905 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2ce80a2910 msgr2=0x7f2ce80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:04.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.476+0000 7f2d07f0f700 1 --2- 172.21.15.33:0/3828471905 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2ce80a2910 0x7f2ce80a4dd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0007720 tx=0x7f2cf00076b0 comp rx=0 tx=0).stop 2024-03-19T02:50:04.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.476+0000 7f2d07f0f700 1 -- 172.21.15.33:0/3828471905 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d00103f00 msgr2=0x7f2d0019fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:04.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.476+0000 7f2d07f0f700 1 --2- 172.21.15.33:0/3828471905 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d00103f00 0x7f2d0019fa50 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2cf800b580 tx=0x7f2cf800b890 comp rx=0 tx=0).stop 2024-03-19T02:50:04.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.476+0000 7f2d07f0f700 1 -- 172.21.15.33:0/3828471905 shutdown_connections 2024-03-19T02:50:04.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.476+0000 7f2d07f0f700 1 --2- 172.21.15.33:0/3828471905 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2ce80a2910 0x7f2ce80a4dd0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:04.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.476+0000 7f2d07f0f700 1 --2- 172.21.15.33:0/3828471905 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d00103f00 0x7f2d0019fa50 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:04.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.476+0000 7f2d07f0f700 1 --2- 172.21.15.33:0/3828471905 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d00102360 0x7f2d00198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:04.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.476+0000 7f2d07f0f700 1 --2- 172.21.15.33:0/3828471905 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d00103560 0x7f2d001989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:04.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.476+0000 7f2d07f0f700 1 -- 172.21.15.33:0/3828471905 >> 172.21.15.33:0/3828471905 conn(0x7f2d000fd920 msgr2=0x7f2d00106860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:04.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.477+0000 7f2d07f0f700 1 -- 172.21.15.33:0/3828471905 shutdown_connections 2024-03-19T02:50:04.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:04.477+0000 7f2d07f0f700 1 -- 172.21.15.33:0/3828471905 wait complete. 2024-03-19T02:50:04.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:04 smithi155 ceph-mon[35804]: osdmap e83: 12 total, 12 up, 12 in 2024-03-19T02:50:04.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:04 smithi155 ceph-mon[35804]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:05.186 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:05 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3828471905' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-19T02:50:05.215 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":83,"num_osds":12,"num_up_osds":12,"osd_up_since":1710816601,"num_in_osds":12,"osd_in_since":1710816580,"num_remapped_pgs":0} 2024-03-19T02:50:05.216 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-19T02:50:05.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:05 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3828471905' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-19T02:50:05.500 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:50:05.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:05 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3828471905' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-19T02:50:06.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:06 smithi033 ceph-mon[31973]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:06.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:06 smithi096 ceph-mon[35946]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:06.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:06 smithi155 ceph-mon[35804]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:07.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.189+0000 7fadc7e51700 1 -- 172.21.15.33:0/2219979714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fadc0100c50 msgr2=0x7fadc01010d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:07.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.189+0000 7fadc7e51700 1 --2- 172.21.15.33:0/2219979714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fadc0100c50 0x7fadc01010d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fadb0009990 tx=0x7fadb0009ca0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.189+0000 7fadc7e51700 1 -- 172.21.15.33:0/2219979714 shutdown_connections 2024-03-19T02:50:07.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.189+0000 7fadc7e51700 1 --2- 172.21.15.33:0/2219979714 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fadc0101610 0x7fadc010d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.189+0000 7fadc7e51700 1 --2- 172.21.15.33:0/2219979714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fadc0106a30 0x7fadc0106e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.189+0000 7fadc7e51700 1 --2- 172.21.15.33:0/2219979714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fadc0100c50 0x7fadc01010d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.189+0000 7fadc7e51700 1 -- 172.21.15.33:0/2219979714 >> 172.21.15.33:0/2219979714 conn(0x7fadc0076570 msgr2=0x7fadc0076980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:07.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.189+0000 7fadc7e51700 1 -- 172.21.15.33:0/2219979714 shutdown_connections 2024-03-19T02:50:07.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.189+0000 7fadc7e51700 1 -- 172.21.15.33:0/2219979714 wait complete. 2024-03-19T02:50:07.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.190+0000 7fadc7e51700 1 Processor -- start 2024-03-19T02:50:07.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.190+0000 7fadc7e51700 1 -- start start 2024-03-19T02:50:07.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.190+0000 7fadc7e51700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fadc0100c50 0x7fadc019e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.190+0000 7fadc7e51700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fadc0101610 0x7fadc019ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.190+0000 7fadc7e51700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fadc0106a30 0x7fadc01987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.190+0000 7fadc7e51700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fadc01102d0 con 0x7fadc0106a30 2024-03-19T02:50:07.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.190+0000 7fadc7e51700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fadc0110150 con 0x7fadc0101610 2024-03-19T02:50:07.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.190+0000 7fadc7e51700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fadc0110450 con 0x7fadc0100c50 2024-03-19T02:50:07.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.190+0000 7fadc63ee700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fadc0106a30 0x7fadc01987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.190+0000 7fadc63ee700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fadc0106a30 0x7fadc01987a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59356/0 (socket says 172.21.15.33:59356) 2024-03-19T02:50:07.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.190+0000 7fadc63ee700 1 -- 172.21.15.33:0/1089535488 learned_addr learned my addr 172.21.15.33:0/1089535488 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:07.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.190+0000 7fadc53ec700 1 --2- 172.21.15.33:0/1089535488 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fadc0101610 0x7fadc019ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.190+0000 7fadc5bed700 1 --2- 172.21.15.33:0/1089535488 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fadc0100c50 0x7fadc019e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.190+0000 7fadc63ee700 1 -- 172.21.15.33:0/1089535488 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fadc0100c50 msgr2=0x7fadc019e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:07.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.190+0000 7fadc63ee700 1 --2- 172.21.15.33:0/1089535488 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fadc0100c50 0x7fadc019e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.190+0000 7fadc63ee700 1 -- 172.21.15.33:0/1089535488 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fadc0101610 msgr2=0x7fadc019ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:07.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.190+0000 7fadc63ee700 1 --2- 172.21.15.33:0/1089535488 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fadc0101610 0x7fadc019ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.190+0000 7fadc63ee700 1 -- 172.21.15.33:0/1089535488 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadc0199080 con 0x7fadc0106a30 2024-03-19T02:50:07.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.191+0000 7fadc63ee700 1 --2- 172.21.15.33:0/1089535488 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fadc0106a30 0x7fadc01987a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fadb8009fa0 tx=0x7fadb80076c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:07.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.191+0000 7fadb6ffd700 1 -- 172.21.15.33:0/1089535488 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb8019070 con 0x7fadc0106a30 2024-03-19T02:50:07.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.191+0000 7fadb6ffd700 1 -- 172.21.15.33:0/1089535488 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fadb8003c20 con 0x7fadc0106a30 2024-03-19T02:50:07.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.191+0000 7fadc7e51700 1 -- 172.21.15.33:0/1089535488 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadc0199370 con 0x7fadc0106a30 2024-03-19T02:50:07.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.191+0000 7fadb6ffd700 1 -- 172.21.15.33:0/1089535488 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb80146e0 con 0x7fadc0106a30 2024-03-19T02:50:07.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.191+0000 7fadc7e51700 1 -- 172.21.15.33:0/1089535488 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadc01a5600 con 0x7fadc0106a30 2024-03-19T02:50:07.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.192+0000 7fadc7e51700 1 -- 172.21.15.33:0/1089535488 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadc01020c0 con 0x7fadc0106a30 2024-03-19T02:50:07.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.193+0000 7fadb6ffd700 1 -- 172.21.15.33:0/1089535488 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fadb801c070 con 0x7fadc0106a30 2024-03-19T02:50:07.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.193+0000 7fadb6ffd700 1 --2- 172.21.15.33:0/1089535488 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fadac0a2910 0x7fadac0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:07.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.193+0000 7fadc5bed700 1 --2- 172.21.15.33:0/1089535488 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fadac0a2910 0x7fadac0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:07.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.193+0000 7fadb6ffd700 1 -- 172.21.15.33:0/1089535488 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fadb80cdfb0 con 0x7fadc0106a30 2024-03-19T02:50:07.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.194+0000 7fadc5bed700 1 --2- 172.21.15.33:0/1089535488 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fadac0a2910 0x7fadac0a4dd0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fadbc005950 tx=0x7fadbc0058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:07.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.198+0000 7fadb6ffd700 1 -- 172.21.15.33:0/1089535488 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fadb8011070 con 0x7fadc0106a30 2024-03-19T02:50:07.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.401+0000 7fadc7e51700 1 -- 172.21.15.33:0/1089535488 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fadc0199c80 con 0x7fadc0106a30 2024-03-19T02:50:07.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.402+0000 7fadb6ffd700 1 -- 172.21.15.33:0/1089535488 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+18371 (secure 0 0 0) 0x7fadb809b4f0 con 0x7fadc0106a30 2024-03-19T02:50:07.403 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:50:07.403 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":83,"fsid":"2b7a8938-e59a-11ee-95c9-87774f69a715","created":"2024-03-19T02:42:37.912179+0000","modified":"2024-03-19T02:50:03.016974+0000","last_up_change":"2024-03-19T02:50:01.539392+0000","last_in_change":"2024-03-19T02:49:40.534415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:46:09.963737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"090a89ab-6b7b-4317-8823-178d4212b2f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6803","nonce":1633124995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6805","nonce":1633124995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6809","nonce":1633124995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6807","nonce":1633124995}]},"public_addr":"172.21.15.33:6803/1633124995","cluster_addr":"172.21.15.33:6805/1633124995","heartbeat_back_addr":"172.21.15.33:6809/1633124995","heartbeat_front_addr":"172.21.15.33:6807/1633124995","state":["exists","up"]},{"osd":1,"uuid":"f50eec55-1956-4486-8593-f2c8fa610e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6801","nonce":340144398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6803","nonce":340144398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6807","nonce":340144398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6805","nonce":340144398}]},"public_addr":"172.21.15.96:6801/340144398","cluster_addr":"172.21.15.96:6803/340144398","heartbeat_back_addr":"172.21.15.96:6807/340144398","heartbeat_front_addr":"172.21.15.96:6805/340144398","state":["exists","up"]},{"osd":2,"uuid":"893a2d62-80d9-4e5e-bc6a-a1c8c681d652","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6801","nonce":3388743876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6803","nonce":3388743876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6807","nonce":3388743876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6805","nonce":3388743876}]},"public_addr":"172.21.15.155:6801/3388743876","cluster_addr":"172.21.15.155:6803/3388743876","heartbeat_back_addr":"172.21.15.155:6807/3388743876","heartbeat_front_addr":"172.21.15.155:6805/3388743876","state":["exists","up"]},{"osd":3,"uuid":"b07e0482-4281-4d32-9dfe-9995d7246175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6811","nonce":677007171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6813","nonce":677007171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6817","nonce":677007171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6815","nonce":677007171}]},"public_addr":"172.21.15.33:6811/677007171","cluster_addr":"172.21.15.33:6813/677007171","heartbeat_back_addr":"172.21.15.33:6817/677007171","heartbeat_front_addr":"172.21.15.33:6815/677007171","state":["exists","up"]},{"osd":4,"uuid":"a988384b-af6b-47c6-8c07-5c9673acee45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6809","nonce":3106158554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6811","nonce":3106158554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6815","nonce":3106158554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6813","nonce":3106158554}]},"public_addr":"172.21.15.96:6809/3106158554","cluster_addr":"172.21.15.96:6811/3106158554","heartbeat_back_addr":"172.21.15.96:6815/3106158554","heartbeat_front_addr":"172.21.15.96:6813/3106158554","state":["exists","up"]},{"osd":5,"uuid":"198dc992-96f8-4c52-96ee-1fd5de148bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6809","nonce":327293324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6811","nonce":327293324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6815","nonce":327293324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6813","nonce":327293324}]},"public_addr":"172.21.15.155:6809/327293324","cluster_addr":"172.21.15.155:6811/327293324","heartbeat_back_addr":"172.21.15.155:6815/327293324","heartbeat_front_addr":"172.21.15.155:6813/327293324","state":["exists","up"]},{"osd":6,"uuid":"a7278ec0-ab2e-49ed-be53-84d8b9e24177","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6819","nonce":873871509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6821","nonce":873871509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6825","nonce":873871509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6823","nonce":873871509}]},"public_addr":"172.21.15.33:6819/873871509","cluster_addr":"172.21.15.33:6821/873871509","heartbeat_back_addr":"172.21.15.33:6825/873871509","heartbeat_front_addr":"172.21.15.33:6823/873871509","state":["exists","up"]},{"osd":7,"uuid":"727e133d-aa24-4ffc-90e2-266eb2aab8c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6817","nonce":2334123891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6819","nonce":2334123891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6823","nonce":2334123891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6821","nonce":2334123891}]},"public_addr":"172.21.15.96:6817/2334123891","cluster_addr":"172.21.15.96:6819/2334123891","heartbeat_back_addr":"172.21.15.96:6823/2334123891","heartbeat_front_addr":"172.21.15.96:6821/2334123891","state":["exists","up"]},{"osd":8,"uuid":"bb20eec0-39e5-4b86-9db5-486b1840cdad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6817","nonce":3498516903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6819","nonce":3498516903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6823","nonce":3498516903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6821","nonce":3498516903}]},"public_addr":"172.21.15.155:6817/3498516903","cluster_addr":"172.21.15.155:6819/3498516903","heartbeat_back_addr":"172.21.15.155:6823/3498516903","heartbeat_front_addr":"172.21.15.155:6821/3498516903","state":["exists","up"]},{"osd":9,"uuid":"179c6228-3e23-4f65-91a6-1ef0005d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6827","nonce":760777213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6829","nonce":760777213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6833","nonce":760777213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6831","nonce":760777213}]},"public_addr":"172.21.15.33:6827/760777213","cluster_addr":"172.21.15.33:6829/760777213","heartbeat_back_addr":"172.21.15.33:6833/760777213","heartbeat_front_addr":"172.21.15.33:6831/760777213","state":["exists","up"]},{"osd":10,"uuid":"be17b478-3c2e-41fb-bede-958333acdf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6825","nonce":1404898482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6827","nonce":1404898482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6831","nonce":1404898482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6829","nonce":1404898482}]},"public_addr":"172.21.15.96:6825/1404898482","cluster_addr":"172.21.15.96:6827/1404898482","heartbeat_back_addr":"172.21.15.96:6831/1404898482","heartbeat_front_addr":"172.21.15.96:6829/1404898482","state":["exists","up"]},{"osd":11,"uuid":"c0903ed2-cc1e-4c18-8731-1f0ee4dd0457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6825","nonce":792352827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6827","nonce":792352827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6831","nonce":792352827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6829","nonce":792352827}]},"public_addr":"172.21.15.155:6825/792352827","cluster_addr":"172.21.15.155:6827/792352827","heartbeat_back_addr":"172.21.15.155:6831/792352827","heartbeat_front_addr":"172.21.15.155:6829/792352827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:17.942801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:42.109449+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:05.981599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:32.069574+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:57.431015+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:22.448592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:48.526515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:14.498352+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:40.175426+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:06.364006+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:32.853161+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:59.073538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:6801/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:6800/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/1067768047":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/729379453":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2586588742":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/2932814165":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6801/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2212564963":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6800/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/3275126058":"2024-03-20T02:43:39.622218+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:50:07.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.404+0000 7fadc7e51700 1 -- 172.21.15.33:0/1089535488 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fadac0a2910 msgr2=0x7fadac0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:07.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.404+0000 7fadc7e51700 1 --2- 172.21.15.33:0/1089535488 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fadac0a2910 0x7fadac0a4dd0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fadbc005950 tx=0x7fadbc0058e0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.404+0000 7fadc7e51700 1 -- 172.21.15.33:0/1089535488 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fadc0106a30 msgr2=0x7fadc01987a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:07.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.404+0000 7fadc7e51700 1 --2- 172.21.15.33:0/1089535488 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fadc0106a30 0x7fadc01987a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fadb8009fa0 tx=0x7fadb80076c0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.404+0000 7fadc7e51700 1 -- 172.21.15.33:0/1089535488 shutdown_connections 2024-03-19T02:50:07.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.404+0000 7fadc7e51700 1 --2- 172.21.15.33:0/1089535488 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fadac0a2910 0x7fadac0a4dd0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.404+0000 7fadc7e51700 1 --2- 172.21.15.33:0/1089535488 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fadc0106a30 0x7fadc01987a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.404+0000 7fadc7e51700 1 --2- 172.21.15.33:0/1089535488 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fadc0100c50 0x7fadc019e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.404+0000 7fadc7e51700 1 --2- 172.21.15.33:0/1089535488 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fadc0101610 0x7fadc019ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:07.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.404+0000 7fadc7e51700 1 -- 172.21.15.33:0/1089535488 >> 172.21.15.33:0/1089535488 conn(0x7fadc0076570 msgr2=0x7fadc00feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:07.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.404+0000 7fadc7e51700 1 -- 172.21.15.33:0/1089535488 shutdown_connections 2024-03-19T02:50:07.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:07.404+0000 7fadc7e51700 1 -- 172.21.15.33:0/1089535488 wait complete. 2024-03-19T02:50:07.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:07 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1089535488' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:50:07.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:07 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1089535488' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:50:07.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:07 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1089535488' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:50:08.216 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:46:09.963737+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T02:50:08.216 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-19T02:50:08.458 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:50:08.712 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:08 smithi033 ceph-mon[31973]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:08.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:08 smithi096 ceph-mon[35946]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:08.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:08 smithi155 ceph-mon[35804]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:09.962 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:09 smithi033 ceph-mon[31973]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:10.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.176+0000 7f4f915af700 1 -- 172.21.15.33:0/3553940638 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f8c1014a0 msgr2=0x7f4f8c101880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:10.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.176+0000 7f4f915af700 1 --2- 172.21.15.33:0/3553940638 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f8c1014a0 0x7f4f8c101880 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c009990 tx=0x7f4f7c009ca0 comp rx=0 tx=0).stop 2024-03-19T02:50:10.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.176+0000 7f4f915af700 1 -- 172.21.15.33:0/3553940638 shutdown_connections 2024-03-19T02:50:10.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.176+0000 7f4f915af700 1 --2- 172.21.15.33:0/3553940638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f8c106520 0x7f4f8c10e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:10.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.176+0000 7f4f915af700 1 --2- 172.21.15.33:0/3553940638 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f8c1014a0 0x7f4f8c101880 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:10.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.176+0000 7f4f915af700 1 --2- 172.21.15.33:0/3553940638 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4f8c101e50 0x7f4f8c105e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:10.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.176+0000 7f4f915af700 1 -- 172.21.15.33:0/3553940638 >> 172.21.15.33:0/3553940638 conn(0x7f4f8c077250 msgr2=0x7f4f8c077660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:10.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.176+0000 7f4f915af700 1 -- 172.21.15.33:0/3553940638 shutdown_connections 2024-03-19T02:50:10.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.176+0000 7f4f915af700 1 -- 172.21.15.33:0/3553940638 wait complete. 2024-03-19T02:50:10.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.177+0000 7f4f915af700 1 Processor -- start 2024-03-19T02:50:10.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.177+0000 7f4f915af700 1 -- start start 2024-03-19T02:50:10.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.177+0000 7f4f915af700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f8c1014a0 0x7f4f8c19e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:10.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.177+0000 7f4f915af700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4f8c101e50 0x7f4f8c19ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:10.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.177+0000 7f4f915af700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f8c106520 0x7f4f8c1988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:10.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.177+0000 7f4f915af700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4f8c111490 con 0x7f4f8c106520 2024-03-19T02:50:10.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.177+0000 7f4f915af700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4f8c111310 con 0x7f4f8c1014a0 2024-03-19T02:50:10.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.177+0000 7f4f915af700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4f8c111610 con 0x7f4f8c101e50 2024-03-19T02:50:10.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.177+0000 7f4f8b7fe700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f8c106520 0x7f4f8c1988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:10.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.177+0000 7f4f8b7fe700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f8c106520 0x7f4f8c1988a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59382/0 (socket says 172.21.15.33:59382) 2024-03-19T02:50:10.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.177+0000 7f4f8b7fe700 1 -- 172.21.15.33:0/3754503201 learned_addr learned my addr 172.21.15.33:0/3754503201 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:10.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.177+0000 7f4f8affd700 1 --2- 172.21.15.33:0/3754503201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f8c1014a0 0x7f4f8c19e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:10.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.178+0000 7f4f8a7fc700 1 --2- 172.21.15.33:0/3754503201 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4f8c101e50 0x7f4f8c19ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:10.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.178+0000 7f4f8b7fe700 1 -- 172.21.15.33:0/3754503201 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4f8c101e50 msgr2=0x7f4f8c19ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:10.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.178+0000 7f4f8b7fe700 1 --2- 172.21.15.33:0/3754503201 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4f8c101e50 0x7f4f8c19ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:10.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.178+0000 7f4f8b7fe700 1 -- 172.21.15.33:0/3754503201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f8c1014a0 msgr2=0x7f4f8c19e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:10.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.178+0000 7f4f8b7fe700 1 --2- 172.21.15.33:0/3754503201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f8c1014a0 0x7f4f8c19e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:10.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.178+0000 7f4f8b7fe700 1 -- 172.21.15.33:0/3754503201 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f8c199000 con 0x7f4f8c106520 2024-03-19T02:50:10.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.178+0000 7f4f8b7fe700 1 --2- 172.21.15.33:0/3754503201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f8c106520 0x7f4f8c1988a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4f8400b970 tx=0x7f4f8400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:10.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.178+0000 7f4f83fff700 1 -- 172.21.15.33:0/3754503201 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f84019070 con 0x7f4f8c106520 2024-03-19T02:50:10.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.178+0000 7f4f83fff700 1 -- 172.21.15.33:0/3754503201 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4f8400cde0 con 0x7f4f8c106520 2024-03-19T02:50:10.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.178+0000 7f4f915af700 1 -- 172.21.15.33:0/3754503201 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f8c1992f0 con 0x7f4f8c106520 2024-03-19T02:50:10.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.178+0000 7f4f83fff700 1 -- 172.21.15.33:0/3754503201 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f84007370 con 0x7f4f8c106520 2024-03-19T02:50:10.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.178+0000 7f4f915af700 1 -- 172.21.15.33:0/3754503201 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f8c1a55b0 con 0x7f4f8c106520 2024-03-19T02:50:10.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.179+0000 7f4f915af700 1 -- 172.21.15.33:0/3754503201 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f8c068fd0 con 0x7f4f8c106520 2024-03-19T02:50:10.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.181+0000 7f4f83fff700 1 -- 172.21.15.33:0/3754503201 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f4f8401c070 con 0x7f4f8c106520 2024-03-19T02:50:10.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.182+0000 7f4f83fff700 1 --2- 172.21.15.33:0/3754503201 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4f780a2890 0x7f4f780a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:10.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.182+0000 7f4f8affd700 1 --2- 172.21.15.33:0/3754503201 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4f780a2890 0x7f4f780a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:10.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.182+0000 7f4f83fff700 1 -- 172.21.15.33:0/3754503201 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f4f840ce260 con 0x7f4f8c106520 2024-03-19T02:50:10.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.182+0000 7f4f8affd700 1 --2- 172.21.15.33:0/3754503201 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4f780a2890 0x7f4f780a4d50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c00b5c0 tx=0x7f4f7c005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:10.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.185+0000 7f4f83fff700 1 -- 172.21.15.33:0/3754503201 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4f84023070 con 0x7f4f8c106520 2024-03-19T02:50:10.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:09 smithi096 ceph-mon[35946]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:10.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:09 smithi155 ceph-mon[35804]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:10.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.390+0000 7f4f915af700 1 -- 172.21.15.33:0/3754503201 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4f8c199dd0 con 0x7f4f8c106520 2024-03-19T02:50:10.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.391+0000 7f4f83fff700 1 -- 172.21.15.33:0/3754503201 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7f4f8409b700 con 0x7f4f8c106520 2024-03-19T02:50:10.392 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 1 2024-03-19T02:50:10.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.393+0000 7f4f915af700 1 -- 172.21.15.33:0/3754503201 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4f780a2890 msgr2=0x7f4f780a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:10.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.393+0000 7f4f915af700 1 --2- 172.21.15.33:0/3754503201 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4f780a2890 0x7f4f780a4d50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c00b5c0 tx=0x7f4f7c005960 comp rx=0 tx=0).stop 2024-03-19T02:50:10.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.393+0000 7f4f915af700 1 -- 172.21.15.33:0/3754503201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f8c106520 msgr2=0x7f4f8c1988a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:10.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.393+0000 7f4f915af700 1 --2- 172.21.15.33:0/3754503201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f8c106520 0x7f4f8c1988a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4f8400b970 tx=0x7f4f8400bc80 comp rx=0 tx=0).stop 2024-03-19T02:50:10.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.393+0000 7f4f915af700 1 -- 172.21.15.33:0/3754503201 shutdown_connections 2024-03-19T02:50:10.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.393+0000 7f4f915af700 1 --2- 172.21.15.33:0/3754503201 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4f780a2890 0x7f4f780a4d50 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:10.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.393+0000 7f4f915af700 1 --2- 172.21.15.33:0/3754503201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4f8c106520 0x7f4f8c1988a0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:10.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.393+0000 7f4f915af700 1 --2- 172.21.15.33:0/3754503201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f8c1014a0 0x7f4f8c19e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:10.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.393+0000 7f4f915af700 1 --2- 172.21.15.33:0/3754503201 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4f8c101e50 0x7f4f8c19ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:10.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.393+0000 7f4f915af700 1 -- 172.21.15.33:0/3754503201 >> 172.21.15.33:0/3754503201 conn(0x7f4f8c077250 msgr2=0x7f4f8c06aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:10.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.393+0000 7f4f915af700 1 -- 172.21.15.33:0/3754503201 shutdown_connections 2024-03-19T02:50:10.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:10.393+0000 7f4f915af700 1 -- 172.21.15.33:0/3754503201 wait complete. 2024-03-19T02:50:11.043 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:10 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3754503201' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:50:11.073 INFO:tasks.cephadm:Adding mds.a on smithi033 2024-03-19T02:50:11.073 INFO:tasks.cephadm:Adding mds.d on smithi033 2024-03-19T02:50:11.073 INFO:tasks.cephadm:Adding mds.g on smithi033 2024-03-19T02:50:11.073 INFO:tasks.cephadm:Adding mds.j on smithi033 2024-03-19T02:50:11.073 INFO:tasks.cephadm:Adding mds.b on smithi096 2024-03-19T02:50:11.073 INFO:tasks.cephadm:Adding mds.e on smithi096 2024-03-19T02:50:11.073 INFO:tasks.cephadm:Adding mds.h on smithi096 2024-03-19T02:50:11.073 INFO:tasks.cephadm:Adding mds.k on smithi096 2024-03-19T02:50:11.073 INFO:tasks.cephadm:Adding mds.c on smithi155 2024-03-19T02:50:11.073 INFO:tasks.cephadm:Adding mds.f on smithi155 2024-03-19T02:50:11.074 INFO:tasks.cephadm:Adding mds.i on smithi155 2024-03-19T02:50:11.074 INFO:tasks.cephadm:Adding mds.l on smithi155 2024-03-19T02:50:11.074 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph orch apply mds all '12;smithi033=a;smithi033=d;smithi033=g;smithi033=j;smithi096=b;smithi096=e;smithi096=h;smithi096=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l' 2024-03-19T02:50:11.104 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:10 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3754503201' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:50:11.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:10 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3754503201' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:50:11.325 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T02:50:12.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:11 smithi155 ceph-mon[35804]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:12.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:11 smithi096 ceph-mon[35946]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:12.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:11 smithi033 ceph-mon[31973]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:12.998 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.995+0000 7efd77f49700 1 -- 172.21.15.155:0/149299246 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd701035e0 msgr2=0x7efd70103a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:12.998 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.995+0000 7efd77f49700 1 --2- 172.21.15.155:0/149299246 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd701035e0 0x7efd70103a40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7efd6c009990 tx=0x7efd6c009ca0 comp rx=0 tx=0).stop 2024-03-19T02:50:12.998 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.995+0000 7efd77f49700 1 -- 172.21.15.155:0/149299246 shutdown_connections 2024-03-19T02:50:12.998 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.995+0000 7efd77f49700 1 --2- 172.21.15.155:0/149299246 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd70103f80 0x7efd7010a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:12.998 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.995+0000 7efd77f49700 1 --2- 172.21.15.155:0/149299246 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd70100600 0x7efd70100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:12.998 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.995+0000 7efd77f49700 1 --2- 172.21.15.155:0/149299246 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd701035e0 0x7efd70103a40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:12.998 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.995+0000 7efd77f49700 1 -- 172.21.15.155:0/149299246 >> 172.21.15.155:0/149299246 conn(0x7efd700fbf20 msgr2=0x7efd700fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:12.998 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.996+0000 7efd77f49700 1 -- 172.21.15.155:0/149299246 shutdown_connections 2024-03-19T02:50:12.999 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.996+0000 7efd77f49700 1 -- 172.21.15.155:0/149299246 wait complete. 2024-03-19T02:50:12.999 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.996+0000 7efd77f49700 1 Processor -- start 2024-03-19T02:50:12.999 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.996+0000 7efd77f49700 1 -- start start 2024-03-19T02:50:12.999 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.996+0000 7efd77f49700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd70100600 0x7efd70102ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:12.999 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.996+0000 7efd77f49700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd701035e0 0x7efd70101590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:12.999 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.996+0000 7efd77f49700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd70103f80 0x7efd70101ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:12.999 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.996+0000 7efd77f49700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efd7010de50 con 0x7efd70103f80 2024-03-19T02:50:12.999 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.996+0000 7efd77f49700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efd7010dcd0 con 0x7efd701035e0 2024-03-19T02:50:12.999 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.996+0000 7efd77f49700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7efd7010dfd0 con 0x7efd70100600 2024-03-19T02:50:12.999 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.996+0000 7efd75ce5700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd70100600 0x7efd70102ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:12.999 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.996+0000 7efd75ce5700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd70100600 0x7efd70102ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:38528/0 (socket says 172.21.15.155:38528) 2024-03-19T02:50:12.999 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.996+0000 7efd75ce5700 1 -- 172.21.15.155:0/2312534157 learned_addr learned my addr 172.21.15.155:0/2312534157 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-03-19T02:50:12.999 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.996+0000 7efd754e4700 1 --2- 172.21.15.155:0/2312534157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd701035e0 0x7efd70101590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:13.000 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.997+0000 7efd764e6700 1 --2- 172.21.15.155:0/2312534157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd70103f80 0x7efd70101ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:13.000 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.997+0000 7efd75ce5700 1 -- 172.21.15.155:0/2312534157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd701035e0 msgr2=0x7efd70101590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:13.000 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.997+0000 7efd75ce5700 1 --2- 172.21.15.155:0/2312534157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd701035e0 0x7efd70101590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:13.000 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.997+0000 7efd75ce5700 1 -- 172.21.15.155:0/2312534157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd70103f80 msgr2=0x7efd70101ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:13.000 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.997+0000 7efd75ce5700 1 --2- 172.21.15.155:0/2312534157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd70103f80 0x7efd70101ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:13.000 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.997+0000 7efd75ce5700 1 -- 172.21.15.155:0/2312534157 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd701023a0 con 0x7efd70100600 2024-03-19T02:50:13.000 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.997+0000 7efd75ce5700 1 --2- 172.21.15.155:0/2312534157 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd70100600 0x7efd70102ef0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7efd6000b500 tx=0x7efd6000b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:13.000 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.997+0000 7efd66ffd700 1 -- 172.21.15.155:0/2312534157 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd60015020 con 0x7efd70100600 2024-03-19T02:50:13.000 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.997+0000 7efd77f49700 1 -- 172.21.15.155:0/2312534157 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd701a26a0 con 0x7efd70100600 2024-03-19T02:50:13.000 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.997+0000 7efd66ffd700 1 -- 172.21.15.155:0/2312534157 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efd60011ce0 con 0x7efd70100600 2024-03-19T02:50:13.000 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.997+0000 7efd66ffd700 1 -- 172.21.15.155:0/2312534157 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd60010c80 con 0x7efd70100600 2024-03-19T02:50:13.000 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.997+0000 7efd77f49700 1 -- 172.21.15.155:0/2312534157 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd701a2bb0 con 0x7efd70100600 2024-03-19T02:50:13.001 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.998+0000 7efd77f49700 1 -- 172.21.15.155:0/2312534157 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd70068fd0 con 0x7efd70100600 2024-03-19T02:50:13.001 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.999+0000 7efd66ffd700 1 -- 172.21.15.155:0/2312534157 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7efd60018020 con 0x7efd70100600 2024-03-19T02:50:13.001 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:12.999+0000 7efd66ffd700 1 --2- 172.21.15.155:0/2312534157 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7efd5c0a2890 0x7efd5c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:13.001 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:13.000+0000 7efd66ffd700 1 -- 172.21.15.155:0/2312534157 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7efd600cdd60 con 0x7efd70100600 2024-03-19T02:50:13.001 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:13.000+0000 7efd754e4700 1 --2- 172.21.15.155:0/2312534157 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7efd5c0a2890 0x7efd5c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:13.001 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:13.000+0000 7efd754e4700 1 --2- 172.21.15.155:0/2312534157 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7efd5c0a2890 0x7efd5c0a4d50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efd6c00b5c0 tx=0x7efd6c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:13.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:13.004+0000 7efd66ffd700 1 -- 172.21.15.155:0/2312534157 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efd6001f020 con 0x7efd70100600 2024-03-19T02:50:13.215 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:13.213+0000 7efd77f49700 1 -- 172.21.15.155:0/2312534157 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi033=a;smithi033=d;smithi033=g;smithi033=j;smithi096=b;smithi096=e;smithi096=h;smithi096=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l", "target": ["mon-mgr", ""]}) -- 0x7efd70062910 con 0x7efd5c0a2890 2024-03-19T02:50:13.223 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:13.222+0000 7efd66ffd700 1 -- 172.21.15.155:0/2312534157 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7efd70062910 con 0x7efd5c0a2890 2024-03-19T02:50:13.223 INFO:teuthology.orchestra.run.smithi155.stdout:Scheduled mds.all update... 2024-03-19T02:50:13.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:13.224+0000 7efd77f49700 1 -- 172.21.15.155:0/2312534157 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7efd5c0a2890 msgr2=0x7efd5c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:13.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:13.224+0000 7efd77f49700 1 --2- 172.21.15.155:0/2312534157 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7efd5c0a2890 0x7efd5c0a4d50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efd6c00b5c0 tx=0x7efd6c005fd0 comp rx=0 tx=0).stop 2024-03-19T02:50:13.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:13.224+0000 7efd77f49700 1 -- 172.21.15.155:0/2312534157 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd70100600 msgr2=0x7efd70102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:13.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:13.224+0000 7efd77f49700 1 --2- 172.21.15.155:0/2312534157 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd70100600 0x7efd70102ef0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7efd6000b500 tx=0x7efd6000b810 comp rx=0 tx=0).stop 2024-03-19T02:50:13.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:13.224+0000 7efd77f49700 1 -- 172.21.15.155:0/2312534157 shutdown_connections 2024-03-19T02:50:13.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:13.224+0000 7efd77f49700 1 --2- 172.21.15.155:0/2312534157 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7efd5c0a2890 0x7efd5c0a4d50 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:13.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:13.224+0000 7efd77f49700 1 --2- 172.21.15.155:0/2312534157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd70103f80 0x7efd70101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:13.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:13.224+0000 7efd77f49700 1 --2- 172.21.15.155:0/2312534157 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd70100600 0x7efd70102ef0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:13.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:13.224+0000 7efd77f49700 1 --2- 172.21.15.155:0/2312534157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd701035e0 0x7efd70101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:13.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:13.224+0000 7efd77f49700 1 -- 172.21.15.155:0/2312534157 >> 172.21.15.155:0/2312534157 conn(0x7efd700fbf20 msgr2=0x7efd70106a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:13.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:13.224+0000 7efd77f49700 1 -- 172.21.15.155:0/2312534157 shutdown_connections 2024-03-19T02:50:13.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-03-19T02:50:13.224+0000 7efd77f49700 1 -- 172.21.15.155:0/2312534157 wait complete. 2024-03-19T02:50:13.889 DEBUG:teuthology.orchestra.run.smithi033:mds.a> sudo journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.a.service 2024-03-19T02:50:13.892 DEBUG:teuthology.orchestra.run.smithi033:mds.d> sudo journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.d.service 2024-03-19T02:50:13.937 DEBUG:teuthology.orchestra.run.smithi033:mds.g> sudo journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.g.service 2024-03-19T02:50:13.980 DEBUG:teuthology.orchestra.run.smithi033:mds.j> sudo journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.j.service 2024-03-19T02:50:14.024 DEBUG:teuthology.orchestra.run.smithi096:mds.b> sudo journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.b.service 2024-03-19T02:50:14.026 DEBUG:teuthology.orchestra.run.smithi096:mds.e> sudo journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.e.service 2024-03-19T02:50:14.070 DEBUG:teuthology.orchestra.run.smithi096:mds.h> sudo journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.h.service 2024-03-19T02:50:14.112 DEBUG:teuthology.orchestra.run.smithi096:mds.k> sudo journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.k.service 2024-03-19T02:50:14.156 DEBUG:teuthology.orchestra.run.smithi155:mds.c> sudo journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.c.service 2024-03-19T02:50:14.158 DEBUG:teuthology.orchestra.run.smithi155:mds.f> sudo journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.f.service 2024-03-19T02:50:14.200 DEBUG:teuthology.orchestra.run.smithi155:mds.i> sudo journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.i.service 2024-03-19T02:50:14.243 DEBUG:teuthology.orchestra.run.smithi155:mds.l> sudo journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.l.service 2024-03-19T02:50:14.286 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-19T02:50:14.287 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:50:14.383 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:14 smithi155 ceph-mon[35804]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi033=a;smithi033=d;smithi033=g;smithi033=j;smithi096=b;smithi096=e;smithi096=h;smithi096=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:50:14.383 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:14 smithi155 ceph-mon[35804]: Saving service mds.all spec with placement smithi033=a;smithi033=d;smithi033=g;smithi033=j;smithi096=b;smithi096=e;smithi096=h;smithi096=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l;count:12 2024-03-19T02:50:14.383 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:14 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:14.384 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:14 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:50:14.384 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:14 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:14.384 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:14 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:50:14.384 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:14 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:14.384 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:14 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:14.384 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:14 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:14.384 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:14 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:14.384 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:14 smithi155 ceph-mon[35804]: Deploying daemon mds.l on smithi155 2024-03-19T02:50:14.384 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:14 smithi155 ceph-mon[35804]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:14.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.396+0000 7f883a231640 1 -- 172.21.15.33:0/706549542 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8834102310 msgr2=0x7f8834108ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:14.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.396+0000 7f883a231640 1 --2- 172.21.15.33:0/706549542 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8834102310 0x7f8834108ba0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f882c009ca0 tx=0x7f882c02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:50:14.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.396+0000 7f883a231640 1 -- 172.21.15.33:0/706549542 shutdown_connections 2024-03-19T02:50:14.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.396+0000 7f883a231640 1 --2- 172.21.15.33:0/706549542 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8834102310 0x7f8834108ba0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:14.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.396+0000 7f883a231640 1 --2- 172.21.15.33:0/706549542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8834101950 0x7f8834101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:14.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.396+0000 7f883a231640 1 --2- 172.21.15.33:0/706549542 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8834100760 0x7f8834100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:14.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.396+0000 7f883a231640 1 -- 172.21.15.33:0/706549542 >> 172.21.15.33:0/706549542 conn(0x7f88340fbcd0 msgr2=0x7f88340fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:14.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.396+0000 7f883a231640 1 -- 172.21.15.33:0/706549542 shutdown_connections 2024-03-19T02:50:14.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.396+0000 7f883a231640 1 -- 172.21.15.33:0/706549542 wait complete. 2024-03-19T02:50:14.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.397+0000 7f883a231640 1 Processor -- start 2024-03-19T02:50:14.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.397+0000 7f883a231640 1 -- start start 2024-03-19T02:50:14.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.397+0000 7f883a231640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8834100760 0x7f883410b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:14.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.397+0000 7f883a231640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8834101950 0x7f8834109720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:14.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.397+0000 7f883a231640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8834102310 0x7f8834109c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:14.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.397+0000 7f883a231640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f883410d490 con 0x7f8834101950 2024-03-19T02:50:14.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.397+0000 7f883a231640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f883410d310 con 0x7f8834102310 2024-03-19T02:50:14.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.397+0000 7f883a231640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f883410d610 con 0x7f8834100760 2024-03-19T02:50:14.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.397+0000 7f8832ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8834101950 0x7f8834109720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:14.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.397+0000 7f8832ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8834101950 0x7f8834109720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58940/0 (socket says 172.21.15.33:58940) 2024-03-19T02:50:14.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.397+0000 7f8832ffd640 1 -- 172.21.15.33:0/3067317338 learned_addr learned my addr 172.21.15.33:0/3067317338 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:14.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.397+0000 7f88337fe640 1 --2- 172.21.15.33:0/3067317338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8834100760 0x7f883410b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:14.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.397+0000 7f8832ffd640 1 -- 172.21.15.33:0/3067317338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8834100760 msgr2=0x7f883410b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:14.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.397+0000 7f8832ffd640 1 --2- 172.21.15.33:0/3067317338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8834100760 0x7f883410b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:14.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.397+0000 7f8832ffd640 1 -- 172.21.15.33:0/3067317338 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8834102310 msgr2=0x7f8834109c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:14.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.397+0000 7f8832ffd640 1 --2- 172.21.15.33:0/3067317338 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8834102310 0x7f8834109c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:14.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.397+0000 7f8832ffd640 1 -- 172.21.15.33:0/3067317338 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f883410a4f0 con 0x7f8834101950 2024-03-19T02:50:14.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.398+0000 7f8832ffd640 1 --2- 172.21.15.33:0/3067317338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8834101950 0x7f8834109720 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f882800b4a0 tx=0x7f882800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:14.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.398+0000 7f8830ff9640 1 -- 172.21.15.33:0/3067317338 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8828015020 con 0x7f8834101950 2024-03-19T02:50:14.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.398+0000 7f883a231640 1 -- 172.21.15.33:0/3067317338 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88341a1d10 con 0x7f8834101950 2024-03-19T02:50:14.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.398+0000 7f8830ff9640 1 -- 172.21.15.33:0/3067317338 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8828011ce0 con 0x7f8834101950 2024-03-19T02:50:14.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.398+0000 7f8830ff9640 1 -- 172.21.15.33:0/3067317338 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8828010dc0 con 0x7f8834101950 2024-03-19T02:50:14.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.398+0000 7f883a231640 1 -- 172.21.15.33:0/3067317338 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88341a2270 con 0x7f8834101950 2024-03-19T02:50:14.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.399+0000 7f883a231640 1 -- 172.21.15.33:0/3067317338 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8834107bf0 con 0x7f8834101950 2024-03-19T02:50:14.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.400+0000 7f8830ff9640 1 -- 172.21.15.33:0/3067317338 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f8828018020 con 0x7f8834101950 2024-03-19T02:50:14.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.400+0000 7f8830ff9640 1 --2- 172.21.15.33:0/3067317338 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f881c0a2800 0x7f881c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:14.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.400+0000 7f88337fe640 1 --2- 172.21.15.33:0/3067317338 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f881c0a2800 0x7f881c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:14.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.400+0000 7f8830ff9640 1 -- 172.21.15.33:0/3067317338 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f88280cded0 con 0x7f8834101950 2024-03-19T02:50:14.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.401+0000 7f88337fe640 1 --2- 172.21.15.33:0/3067317338 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f881c0a2800 0x7f881c0a4cc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f88341017b0 tx=0x7f8824009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:14.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.405+0000 7f8830ff9640 1 -- 172.21.15.33:0/3067317338 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f882801f020 con 0x7f8834101950 2024-03-19T02:50:14.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:14 smithi096 ceph-mon[35946]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi033=a;smithi033=d;smithi033=g;smithi033=j;smithi096=b;smithi096=e;smithi096=h;smithi096=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:50:14.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:14 smithi096 ceph-mon[35946]: Saving service mds.all spec with placement smithi033=a;smithi033=d;smithi033=g;smithi033=j;smithi096=b;smithi096=e;smithi096=h;smithi096=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l;count:12 2024-03-19T02:50:14.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:14 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:14.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:14 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:50:14.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:14 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:14.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:14 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:50:14.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:14 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:14.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:14 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:14.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:14 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:14.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:14 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:14.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:14 smithi096 ceph-mon[35946]: Deploying daemon mds.l on smithi155 2024-03-19T02:50:14.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:14 smithi096 ceph-mon[35946]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:14.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:14 smithi033 ceph-mon[31973]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi033=a;smithi033=d;smithi033=g;smithi033=j;smithi096=b;smithi096=e;smithi096=h;smithi096=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:50:14.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:14 smithi033 ceph-mon[31973]: Saving service mds.all spec with placement smithi033=a;smithi033=d;smithi033=g;smithi033=j;smithi096=b;smithi096=e;smithi096=h;smithi096=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l;count:12 2024-03-19T02:50:14.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:14 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:14.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:14 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:50:14.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:14 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:14.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:14 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:50:14.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:14 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:14.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:14 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:14.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:14 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:14.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:14 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:14.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:14 smithi033 ceph-mon[31973]: Deploying daemon mds.l on smithi155 2024-03-19T02:50:14.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:14 smithi033 ceph-mon[31973]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:14.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.590+0000 7f883a231640 1 -- 172.21.15.33:0/3067317338 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f883402d3c0 con 0x7f8834101950 2024-03-19T02:50:14.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.591+0000 7f8830ff9640 1 -- 172.21.15.33:0/3067317338 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+18371 (secure 0 0 0) 0x7f882809b410 con 0x7f8834101950 2024-03-19T02:50:14.593 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:50:14.593 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":83,"fsid":"2b7a8938-e59a-11ee-95c9-87774f69a715","created":"2024-03-19T02:42:37.912179+0000","modified":"2024-03-19T02:50:03.016974+0000","last_up_change":"2024-03-19T02:50:01.539392+0000","last_in_change":"2024-03-19T02:49:40.534415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:46:09.963737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"090a89ab-6b7b-4317-8823-178d4212b2f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6803","nonce":1633124995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6805","nonce":1633124995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6809","nonce":1633124995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6807","nonce":1633124995}]},"public_addr":"172.21.15.33:6803/1633124995","cluster_addr":"172.21.15.33:6805/1633124995","heartbeat_back_addr":"172.21.15.33:6809/1633124995","heartbeat_front_addr":"172.21.15.33:6807/1633124995","state":["exists","up"]},{"osd":1,"uuid":"f50eec55-1956-4486-8593-f2c8fa610e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6801","nonce":340144398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6803","nonce":340144398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6807","nonce":340144398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6805","nonce":340144398}]},"public_addr":"172.21.15.96:6801/340144398","cluster_addr":"172.21.15.96:6803/340144398","heartbeat_back_addr":"172.21.15.96:6807/340144398","heartbeat_front_addr":"172.21.15.96:6805/340144398","state":["exists","up"]},{"osd":2,"uuid":"893a2d62-80d9-4e5e-bc6a-a1c8c681d652","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6801","nonce":3388743876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6803","nonce":3388743876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6807","nonce":3388743876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6805","nonce":3388743876}]},"public_addr":"172.21.15.155:6801/3388743876","cluster_addr":"172.21.15.155:6803/3388743876","heartbeat_back_addr":"172.21.15.155:6807/3388743876","heartbeat_front_addr":"172.21.15.155:6805/3388743876","state":["exists","up"]},{"osd":3,"uuid":"b07e0482-4281-4d32-9dfe-9995d7246175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6811","nonce":677007171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6813","nonce":677007171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6817","nonce":677007171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6815","nonce":677007171}]},"public_addr":"172.21.15.33:6811/677007171","cluster_addr":"172.21.15.33:6813/677007171","heartbeat_back_addr":"172.21.15.33:6817/677007171","heartbeat_front_addr":"172.21.15.33:6815/677007171","state":["exists","up"]},{"osd":4,"uuid":"a988384b-af6b-47c6-8c07-5c9673acee45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6809","nonce":3106158554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6811","nonce":3106158554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6815","nonce":3106158554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6813","nonce":3106158554}]},"public_addr":"172.21.15.96:6809/3106158554","cluster_addr":"172.21.15.96:6811/3106158554","heartbeat_back_addr":"172.21.15.96:6815/3106158554","heartbeat_front_addr":"172.21.15.96:6813/3106158554","state":["exists","up"]},{"osd":5,"uuid":"198dc992-96f8-4c52-96ee-1fd5de148bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6809","nonce":327293324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6811","nonce":327293324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6815","nonce":327293324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6813","nonce":327293324}]},"public_addr":"172.21.15.155:6809/327293324","cluster_addr":"172.21.15.155:6811/327293324","heartbeat_back_addr":"172.21.15.155:6815/327293324","heartbeat_front_addr":"172.21.15.155:6813/327293324","state":["exists","up"]},{"osd":6,"uuid":"a7278ec0-ab2e-49ed-be53-84d8b9e24177","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6819","nonce":873871509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6821","nonce":873871509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6825","nonce":873871509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6823","nonce":873871509}]},"public_addr":"172.21.15.33:6819/873871509","cluster_addr":"172.21.15.33:6821/873871509","heartbeat_back_addr":"172.21.15.33:6825/873871509","heartbeat_front_addr":"172.21.15.33:6823/873871509","state":["exists","up"]},{"osd":7,"uuid":"727e133d-aa24-4ffc-90e2-266eb2aab8c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6817","nonce":2334123891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6819","nonce":2334123891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6823","nonce":2334123891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6821","nonce":2334123891}]},"public_addr":"172.21.15.96:6817/2334123891","cluster_addr":"172.21.15.96:6819/2334123891","heartbeat_back_addr":"172.21.15.96:6823/2334123891","heartbeat_front_addr":"172.21.15.96:6821/2334123891","state":["exists","up"]},{"osd":8,"uuid":"bb20eec0-39e5-4b86-9db5-486b1840cdad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6817","nonce":3498516903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6819","nonce":3498516903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6823","nonce":3498516903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6821","nonce":3498516903}]},"public_addr":"172.21.15.155:6817/3498516903","cluster_addr":"172.21.15.155:6819/3498516903","heartbeat_back_addr":"172.21.15.155:6823/3498516903","heartbeat_front_addr":"172.21.15.155:6821/3498516903","state":["exists","up"]},{"osd":9,"uuid":"179c6228-3e23-4f65-91a6-1ef0005d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6827","nonce":760777213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6829","nonce":760777213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6833","nonce":760777213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6831","nonce":760777213}]},"public_addr":"172.21.15.33:6827/760777213","cluster_addr":"172.21.15.33:6829/760777213","heartbeat_back_addr":"172.21.15.33:6833/760777213","heartbeat_front_addr":"172.21.15.33:6831/760777213","state":["exists","up"]},{"osd":10,"uuid":"be17b478-3c2e-41fb-bede-958333acdf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6825","nonce":1404898482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6827","nonce":1404898482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6831","nonce":1404898482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6829","nonce":1404898482}]},"public_addr":"172.21.15.96:6825/1404898482","cluster_addr":"172.21.15.96:6827/1404898482","heartbeat_back_addr":"172.21.15.96:6831/1404898482","heartbeat_front_addr":"172.21.15.96:6829/1404898482","state":["exists","up"]},{"osd":11,"uuid":"c0903ed2-cc1e-4c18-8731-1f0ee4dd0457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6825","nonce":792352827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6827","nonce":792352827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6831","nonce":792352827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6829","nonce":792352827}]},"public_addr":"172.21.15.155:6825/792352827","cluster_addr":"172.21.15.155:6827/792352827","heartbeat_back_addr":"172.21.15.155:6831/792352827","heartbeat_front_addr":"172.21.15.155:6829/792352827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:17.942801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:42.109449+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:05.981599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:32.069574+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:57.431015+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:22.448592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:48.526515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:14.498352+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:40.175426+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:06.364006+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:32.853161+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:59.073538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:6801/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:6800/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/1067768047":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/729379453":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2586588742":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/2932814165":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6801/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2212564963":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6800/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/3275126058":"2024-03-20T02:43:39.622218+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:50:14.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.594+0000 7f883a231640 1 -- 172.21.15.33:0/3067317338 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f881c0a2800 msgr2=0x7f881c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:14.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.594+0000 7f883a231640 1 --2- 172.21.15.33:0/3067317338 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f881c0a2800 0x7f881c0a4cc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f88341017b0 tx=0x7f8824009110 comp rx=0 tx=0).stop 2024-03-19T02:50:14.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.594+0000 7f883a231640 1 -- 172.21.15.33:0/3067317338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8834101950 msgr2=0x7f8834109720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:14.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.594+0000 7f883a231640 1 --2- 172.21.15.33:0/3067317338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8834101950 0x7f8834109720 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f882800b4a0 tx=0x7f882800b8a0 comp rx=0 tx=0).stop 2024-03-19T02:50:14.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.594+0000 7f883a231640 1 -- 172.21.15.33:0/3067317338 shutdown_connections 2024-03-19T02:50:14.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.594+0000 7f883a231640 1 --2- 172.21.15.33:0/3067317338 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f881c0a2800 0x7f881c0a4cc0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:14.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.594+0000 7f883a231640 1 --2- 172.21.15.33:0/3067317338 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8834102310 0x7f8834109c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:14.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.594+0000 7f883a231640 1 --2- 172.21.15.33:0/3067317338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8834101950 0x7f8834109720 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:14.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.594+0000 7f883a231640 1 --2- 172.21.15.33:0/3067317338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8834100760 0x7f883410b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:14.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.594+0000 7f883a231640 1 -- 172.21.15.33:0/3067317338 >> 172.21.15.33:0/3067317338 conn(0x7f88340fbcd0 msgr2=0x7f8834104dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:14.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.594+0000 7f883a231640 1 -- 172.21.15.33:0/3067317338 shutdown_connections 2024-03-19T02:50:14.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.594+0000 7f883a231640 1 -- 172.21.15.33:0/3067317338 wait complete. 2024-03-19T02:50:14.606 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:46:09.963737+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T02:50:14.606 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T02:50:14.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.712+0000 7fe260f5b640 1 -- 172.21.15.33:0/203072023 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe25c1022f0 msgr2=0x7fe25c108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:14.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.712+0000 7fe260f5b640 1 --2- 172.21.15.33:0/203072023 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe25c1022f0 0x7fe25c108b80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe254009ca0 tx=0x7fe25402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:50:14.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.712+0000 7fe260f5b640 1 -- 172.21.15.33:0/203072023 shutdown_connections 2024-03-19T02:50:14.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.712+0000 7fe260f5b640 1 --2- 172.21.15.33:0/203072023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe25c101930 0x7fe25c101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:14.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.712+0000 7fe260f5b640 1 --2- 172.21.15.33:0/203072023 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe25c1022f0 0x7fe25c108b80 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:14.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.712+0000 7fe260f5b640 1 --2- 172.21.15.33:0/203072023 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe25c100740 0x7fe25c100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:14.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.712+0000 7fe260f5b640 1 -- 172.21.15.33:0/203072023 >> 172.21.15.33:0/203072023 conn(0x7fe25c0fbcd0 msgr2=0x7fe25c0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:14.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.712+0000 7fe260f5b640 1 -- 172.21.15.33:0/203072023 shutdown_connections 2024-03-19T02:50:14.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.712+0000 7fe260f5b640 1 -- 172.21.15.33:0/203072023 wait complete. 2024-03-19T02:50:14.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.712+0000 7fe260f5b640 1 Processor -- start 2024-03-19T02:50:14.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.712+0000 7fe260f5b640 1 -- start start 2024-03-19T02:50:14.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.713+0000 7fe260f5b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe25c100740 0x7fe25c06a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:14.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.713+0000 7fe260f5b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe25c101930 0x7fe25c19c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:14.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.713+0000 7fe260f5b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe25c1022f0 0x7fe25c19ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:14.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.713+0000 7fe260f5b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe25c10c2d0 con 0x7fe25c1022f0 2024-03-19T02:50:14.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.713+0000 7fe260f5b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe25c10c150 con 0x7fe25c100740 2024-03-19T02:50:14.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.713+0000 7fe260f5b640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe25c10c450 con 0x7fe25c101930 2024-03-19T02:50:14.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.713+0000 7fe25ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe25c1022f0 0x7fe25c19ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:14.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.713+0000 7fe25ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe25c1022f0 0x7fe25c19ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58964/0 (socket says 172.21.15.33:58964) 2024-03-19T02:50:14.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.713+0000 7fe25ad9d640 1 -- 172.21.15.33:0/1791173736 learned_addr learned my addr 172.21.15.33:0/1791173736 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:14.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.713+0000 7fe253fff640 1 --2- 172.21.15.33:0/1791173736 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe25c101930 0x7fe25c19c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:14.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.713+0000 7fe25a59c640 1 --2- 172.21.15.33:0/1791173736 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe25c100740 0x7fe25c06a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:14.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.713+0000 7fe25ad9d640 1 -- 172.21.15.33:0/1791173736 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe25c101930 msgr2=0x7fe25c19c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:14.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.713+0000 7fe25ad9d640 1 --2- 172.21.15.33:0/1791173736 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe25c101930 0x7fe25c19c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:14.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.713+0000 7fe25ad9d640 1 -- 172.21.15.33:0/1791173736 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe25c100740 msgr2=0x7fe25c06a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:14.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.713+0000 7fe25ad9d640 1 --2- 172.21.15.33:0/1791173736 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe25c100740 0x7fe25c06a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:14.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.713+0000 7fe25ad9d640 1 -- 172.21.15.33:0/1791173736 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe25c19f660 con 0x7fe25c1022f0 2024-03-19T02:50:14.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.713+0000 7fe253fff640 1 --2- 172.21.15.33:0/1791173736 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe25c101930 0x7fe25c19c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:50:14.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.713+0000 7fe25a59c640 1 --2- 172.21.15.33:0/1791173736 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe25c100740 0x7fe25c06a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:50:14.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.713+0000 7fe25ad9d640 1 --2- 172.21.15.33:0/1791173736 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe25c1022f0 0x7fe25c19ed50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe254037040 tx=0x7fe2540328f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:14.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.713+0000 7fe2537fe640 1 -- 172.21.15.33:0/1791173736 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe254041020 con 0x7fe25c1022f0 2024-03-19T02:50:14.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.713+0000 7fe260f5b640 1 -- 172.21.15.33:0/1791173736 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe25c19f890 con 0x7fe25c1022f0 2024-03-19T02:50:14.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.713+0000 7fe2537fe640 1 -- 172.21.15.33:0/1791173736 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe254003e20 con 0x7fe25c1022f0 2024-03-19T02:50:14.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.713+0000 7fe2537fe640 1 -- 172.21.15.33:0/1791173736 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe254039040 con 0x7fe25c1022f0 2024-03-19T02:50:14.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.713+0000 7fe260f5b640 1 -- 172.21.15.33:0/1791173736 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe25c19fe50 con 0x7fe25c1022f0 2024-03-19T02:50:14.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.714+0000 7fe260f5b640 1 -- 172.21.15.33:0/1791173736 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe25c107a60 con 0x7fe25c1022f0 2024-03-19T02:50:14.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.716+0000 7fe2537fe640 1 -- 172.21.15.33:0/1791173736 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fe254044020 con 0x7fe25c1022f0 2024-03-19T02:50:14.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.716+0000 7fe2537fe640 1 --2- 172.21.15.33:0/1791173736 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe2480a2800 0x7fe2480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:14.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.716+0000 7fe25a59c640 1 --2- 172.21.15.33:0/1791173736 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe2480a2800 0x7fe2480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:14.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.717+0000 7fe2537fe640 1 -- 172.21.15.33:0/1791173736 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fe2540f0f70 con 0x7fe25c1022f0 2024-03-19T02:50:14.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.717+0000 7fe25a59c640 1 --2- 172.21.15.33:0/1791173736 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe2480a2800 0x7fe2480a4cc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe25c101790 tx=0x7fe24c005db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:14.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.720+0000 7fe2537fe640 1 -- 172.21.15.33:0/1791173736 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe25404b020 con 0x7fe25c1022f0 2024-03-19T02:50:14.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.903+0000 7fe260f5b640 1 -- 172.21.15.33:0/1791173736 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe25c02d380 con 0x7fe25c1022f0 2024-03-19T02:50:14.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.903+0000 7fe2537fe640 1 -- 172.21.15.33:0/1791173736 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7fe2540be4b0 con 0x7fe25c1022f0 2024-03-19T02:50:14.905 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 1 2024-03-19T02:50:14.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.905+0000 7fe260f5b640 1 -- 172.21.15.33:0/1791173736 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe2480a2800 msgr2=0x7fe2480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:14.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.905+0000 7fe260f5b640 1 --2- 172.21.15.33:0/1791173736 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe2480a2800 0x7fe2480a4cc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe25c101790 tx=0x7fe24c005db0 comp rx=0 tx=0).stop 2024-03-19T02:50:14.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.905+0000 7fe260f5b640 1 -- 172.21.15.33:0/1791173736 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe25c1022f0 msgr2=0x7fe25c19ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:14.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.905+0000 7fe260f5b640 1 --2- 172.21.15.33:0/1791173736 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe25c1022f0 0x7fe25c19ed50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe254037040 tx=0x7fe2540328f0 comp rx=0 tx=0).stop 2024-03-19T02:50:14.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.905+0000 7fe260f5b640 1 -- 172.21.15.33:0/1791173736 shutdown_connections 2024-03-19T02:50:14.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.905+0000 7fe260f5b640 1 --2- 172.21.15.33:0/1791173736 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe2480a2800 0x7fe2480a4cc0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:14.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.905+0000 7fe260f5b640 1 --2- 172.21.15.33:0/1791173736 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe25c101930 0x7fe25c19c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:14.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.905+0000 7fe260f5b640 1 --2- 172.21.15.33:0/1791173736 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe25c1022f0 0x7fe25c19ed50 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:14.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.905+0000 7fe260f5b640 1 --2- 172.21.15.33:0/1791173736 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe25c100740 0x7fe25c06a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:14.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.905+0000 7fe260f5b640 1 -- 172.21.15.33:0/1791173736 >> 172.21.15.33:0/1791173736 conn(0x7fe25c0fbcd0 msgr2=0x7fe25c104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:14.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.905+0000 7fe260f5b640 1 -- 172.21.15.33:0/1791173736 shutdown_connections 2024-03-19T02:50:14.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:14.905+0000 7fe260f5b640 1 -- 172.21.15.33:0/1791173736 wait complete. 2024-03-19T02:50:14.917 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:50:15.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.019+0000 7fcd4ac85640 1 -- 172.21.15.33:0/1034727814 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd44101830 msgr2=0x7fcd44101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:15.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.019+0000 7fcd4ac85640 1 --2- 172.21.15.33:0/1034727814 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd44101830 0x7fcd44101c50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c009ca0 tx=0x7fcd3c02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:50:15.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.019+0000 7fcd4ac85640 1 -- 172.21.15.33:0/1034727814 shutdown_connections 2024-03-19T02:50:15.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.019+0000 7fcd4ac85640 1 --2- 172.21.15.33:0/1034727814 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd44102a20 0x7fcd44102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:15.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.019+0000 7fcd4ac85640 1 --2- 172.21.15.33:0/1034727814 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd441033e0 0x7fcd44109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:15.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.019+0000 7fcd4ac85640 1 --2- 172.21.15.33:0/1034727814 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd44101830 0x7fcd44101c50 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:15.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.019+0000 7fcd4ac85640 1 -- 172.21.15.33:0/1034727814 >> 172.21.15.33:0/1034727814 conn(0x7fcd440fce00 msgr2=0x7fcd440ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:15.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.019+0000 7fcd4ac85640 1 -- 172.21.15.33:0/1034727814 shutdown_connections 2024-03-19T02:50:15.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.019+0000 7fcd4ac85640 1 -- 172.21.15.33:0/1034727814 wait complete. 2024-03-19T02:50:15.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.020+0000 7fcd4ac85640 1 Processor -- start 2024-03-19T02:50:15.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.020+0000 7fcd4ac85640 1 -- start start 2024-03-19T02:50:15.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.020+0000 7fcd4ac85640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd44101830 0x7fcd44197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:15.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.020+0000 7fcd4ac85640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd44102a20 0x7fcd44198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:15.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.020+0000 7fcd4ac85640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd441033e0 0x7fcd4419f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:15.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.020+0000 7fcd4ac85640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcd4410d350 con 0x7fcd441033e0 2024-03-19T02:50:15.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.020+0000 7fcd4ac85640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcd4410d1d0 con 0x7fcd44102a20 2024-03-19T02:50:15.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.020+0000 7fcd4ac85640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fcd4410d4d0 con 0x7fcd44101830 2024-03-19T02:50:15.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.020+0000 7fcd49222640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd441033e0 0x7fcd4419f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:15.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.020+0000 7fcd49222640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd441033e0 0x7fcd4419f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58972/0 (socket says 172.21.15.33:58972) 2024-03-19T02:50:15.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.020+0000 7fcd49222640 1 -- 172.21.15.33:0/2455449427 learned_addr learned my addr 172.21.15.33:0/2455449427 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:15.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.020+0000 7fcd48a21640 1 --2- 172.21.15.33:0/2455449427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd44101830 0x7fcd44197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:15.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.020+0000 7fcd43fff640 1 --2- 172.21.15.33:0/2455449427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd44102a20 0x7fcd44198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:15.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.020+0000 7fcd49222640 1 -- 172.21.15.33:0/2455449427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd44101830 msgr2=0x7fcd44197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:15.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.020+0000 7fcd49222640 1 --2- 172.21.15.33:0/2455449427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd44101830 0x7fcd44197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:15.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.020+0000 7fcd49222640 1 -- 172.21.15.33:0/2455449427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd44102a20 msgr2=0x7fcd44198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:15.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.020+0000 7fcd49222640 1 --2- 172.21.15.33:0/2455449427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd44102a20 0x7fcd44198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:15.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.020+0000 7fcd49222640 1 -- 172.21.15.33:0/2455449427 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd4419f700 con 0x7fcd441033e0 2024-03-19T02:50:15.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.021+0000 7fcd43fff640 1 --2- 172.21.15.33:0/2455449427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd44102a20 0x7fcd44198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:50:15.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.021+0000 7fcd49222640 1 --2- 172.21.15.33:0/2455449427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd441033e0 0x7fcd4419f090 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcd3400c970 tx=0x7fcd3400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:15.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.021+0000 7fcd41ffb640 1 -- 172.21.15.33:0/2455449427 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd34007bb0 con 0x7fcd441033e0 2024-03-19T02:50:15.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.021+0000 7fcd4ac85640 1 -- 172.21.15.33:0/2455449427 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd4419f9f0 con 0x7fcd441033e0 2024-03-19T02:50:15.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.021+0000 7fcd41ffb640 1 -- 172.21.15.33:0/2455449427 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcd34007d50 con 0x7fcd441033e0 2024-03-19T02:50:15.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.021+0000 7fcd41ffb640 1 -- 172.21.15.33:0/2455449427 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd34004e10 con 0x7fcd441033e0 2024-03-19T02:50:15.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.021+0000 7fcd4ac85640 1 -- 172.21.15.33:0/2455449427 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd4419ff40 con 0x7fcd441033e0 2024-03-19T02:50:15.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.022+0000 7fcd4ac85640 1 -- 172.21.15.33:0/2455449427 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd44108cc0 con 0x7fcd441033e0 2024-03-19T02:50:15.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.023+0000 7fcd41ffb640 1 -- 172.21.15.33:0/2455449427 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fcd34005020 con 0x7fcd441033e0 2024-03-19T02:50:15.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.023+0000 7fcd41ffb640 1 --2- 172.21.15.33:0/2455449427 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fcd2c0a2800 0x7fcd2c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:15.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.024+0000 7fcd41ffb640 1 -- 172.21.15.33:0/2455449427 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fcd340ce260 con 0x7fcd441033e0 2024-03-19T02:50:15.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.024+0000 7fcd48a21640 1 --2- 172.21.15.33:0/2455449427 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fcd2c0a2800 0x7fcd2c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:15.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.024+0000 7fcd48a21640 1 --2- 172.21.15.33:0/2455449427 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fcd2c0a2800 0x7fcd2c0a4cc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c032380 tx=0x7fcd3c000f40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:15.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.028+0000 7fcd41ffb640 1 -- 172.21.15.33:0/2455449427 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcd34014030 con 0x7fcd441033e0 2024-03-19T02:50:15.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.250+0000 7fcd4ac85640 1 -- 172.21.15.33:0/2455449427 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcd44062420 con 0x7fcd441033e0 2024-03-19T02:50:15.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.251+0000 7fcd41ffb640 1 -- 172.21.15.33:0/2455449427 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7fcd3409b7a0 con 0x7fcd441033e0 2024-03-19T02:50:15.252 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:50:15.252 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-19T02:50:15.252 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 1 2024-03-19T02:50:15.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.253+0000 7fcd4ac85640 1 -- 172.21.15.33:0/2455449427 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fcd2c0a2800 msgr2=0x7fcd2c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:15.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.253+0000 7fcd4ac85640 1 --2- 172.21.15.33:0/2455449427 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fcd2c0a2800 0x7fcd2c0a4cc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c032380 tx=0x7fcd3c000f40 comp rx=0 tx=0).stop 2024-03-19T02:50:15.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.253+0000 7fcd4ac85640 1 -- 172.21.15.33:0/2455449427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd441033e0 msgr2=0x7fcd4419f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:15.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.253+0000 7fcd4ac85640 1 --2- 172.21.15.33:0/2455449427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd441033e0 0x7fcd4419f090 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcd3400c970 tx=0x7fcd3400ce30 comp rx=0 tx=0).stop 2024-03-19T02:50:15.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.253+0000 7fcd4ac85640 1 -- 172.21.15.33:0/2455449427 shutdown_connections 2024-03-19T02:50:15.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.253+0000 7fcd4ac85640 1 --2- 172.21.15.33:0/2455449427 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fcd2c0a2800 0x7fcd2c0a4cc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:15.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.253+0000 7fcd4ac85640 1 --2- 172.21.15.33:0/2455449427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd441033e0 0x7fcd4419f090 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:15.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.253+0000 7fcd4ac85640 1 --2- 172.21.15.33:0/2455449427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd44102a20 0x7fcd44198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:15.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.253+0000 7fcd4ac85640 1 --2- 172.21.15.33:0/2455449427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd44101830 0x7fcd44197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:15.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.253+0000 7fcd4ac85640 1 -- 172.21.15.33:0/2455449427 >> 172.21.15.33:0/2455449427 conn(0x7fcd440fce00 msgr2=0x7fcd44105e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:15.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.253+0000 7fcd4ac85640 1 -- 172.21.15.33:0/2455449427 shutdown_connections 2024-03-19T02:50:15.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:15.253+0000 7fcd4ac85640 1 -- 172.21.15.33:0/2455449427 wait complete. 2024-03-19T02:50:15.423 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:15 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3067317338' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:50:15.423 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:15 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1791173736' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:50:15.423 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:15 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2455449427' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:15.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:15 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3067317338' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:50:15.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:15 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1791173736' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:50:15.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:15 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2455449427' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:15.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:15 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3067317338' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:50:15.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:15 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1791173736' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:50:15.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:15 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2455449427' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:15.768 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 19 02:50:15 smithi155 systemd[1]: Starting Ceph mds.l for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T02:50:16.109 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 19 02:50:15 smithi155 podman[65817]: 2024-03-19 02:50:15.720201475 +0000 UTC m=+0.018349946 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:50:16.109 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 19 02:50:15 smithi155 podman[65817]: 2024-03-19 02:50:15.847953844 +0000 UTC m=+0.146102309 container create 0c647f0e69330a905303cd215ef449acf725d25b152b54998b3dcbf5b2140cf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-19T02:50:16.109 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 19 02:50:16 smithi155 podman[65817]: 2024-03-19 02:50:16.108986047 +0000 UTC m=+0.407134516 container init 0c647f0e69330a905303cd215ef449acf725d25b152b54998b3dcbf5b2140cf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-l, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True) 2024-03-19T02:50:16.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:16 smithi155 ceph-mon[35804]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:16.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:16 smithi155 ceph-mon[35804]: mds.? [v2:172.21.15.155:6832/268774752,v1:172.21.15.155:6833/268774752] up:boot 2024-03-19T02:50:16.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:16 smithi155 ceph-mon[35804]: fsmap 1 up:standby 2024-03-19T02:50:16.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:16 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-19T02:50:16.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:16 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:16.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:16 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:16.517 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 19 02:50:16 smithi155 podman[65817]: 2024-03-19 02:50:16.111582388 +0000 UTC m=+0.409730851 container start 0c647f0e69330a905303cd215ef449acf725d25b152b54998b3dcbf5b2140cf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-l, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-19T02:50:16.517 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 19 02:50:16 smithi155 ceph-mds[65836]: -- 172.21.15.155:0/1778759464 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56195276ba40 con 0x56195273f000 2024-03-19T02:50:16.517 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 19 02:50:16 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-l[65832]: starting mds.l at 2024-03-19T02:50:16.517 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 19 02:50:16 smithi155 bash[65817]: 0c647f0e69330a905303cd215ef449acf725d25b152b54998b3dcbf5b2140cf5 2024-03-19T02:50:16.517 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 19 02:50:16 smithi155 systemd[1]: Started Ceph mds.l for 2b7a8938-e59a-11ee-95c9-87774f69a715. 2024-03-19T02:50:16.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:16 smithi096 ceph-mon[35946]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:16.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:16 smithi096 ceph-mon[35946]: mds.? [v2:172.21.15.155:6832/268774752,v1:172.21.15.155:6833/268774752] up:boot 2024-03-19T02:50:16.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:16 smithi096 ceph-mon[35946]: fsmap 1 up:standby 2024-03-19T02:50:16.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:16 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-19T02:50:16.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:16 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:16.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:16 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:16.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:16 smithi033 ceph-mon[31973]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:16.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:16 smithi033 ceph-mon[31973]: mds.? [v2:172.21.15.155:6832/268774752,v1:172.21.15.155:6833/268774752] up:boot 2024-03-19T02:50:16.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:16 smithi033 ceph-mon[31973]: fsmap 1 up:standby 2024-03-19T02:50:16.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:16 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-19T02:50:16.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:16 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:16.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:16 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:17.266 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:50:17.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.381+0000 7f203bc12640 1 -- 172.21.15.33:0/1011175521 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2034101930 msgr2=0x7f2034101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:17.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.381+0000 7f203bc12640 1 --2- 172.21.15.33:0/1011175521 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2034101930 0x7f2034101db0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2024009900 tx=0x7f202402bc30 comp rx=0 tx=0).stop 2024-03-19T02:50:17.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.381+0000 7f203bc12640 1 -- 172.21.15.33:0/1011175521 shutdown_connections 2024-03-19T02:50:17.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.381+0000 7f203bc12640 1 --2- 172.21.15.33:0/1011175521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20341022f0 0x7f2034108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:17.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.381+0000 7f203bc12640 1 --2- 172.21.15.33:0/1011175521 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2034101930 0x7f2034101db0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:17.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.381+0000 7f203bc12640 1 --2- 172.21.15.33:0/1011175521 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2034100740 0x7f2034100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:17.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.381+0000 7f203bc12640 1 -- 172.21.15.33:0/1011175521 >> 172.21.15.33:0/1011175521 conn(0x7f20340fbcb0 msgr2=0x7f20340fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:17.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.381+0000 7f203bc12640 1 -- 172.21.15.33:0/1011175521 shutdown_connections 2024-03-19T02:50:17.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.381+0000 7f203bc12640 1 -- 172.21.15.33:0/1011175521 wait complete. 2024-03-19T02:50:17.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.381+0000 7f203bc12640 1 Processor -- start 2024-03-19T02:50:17.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.381+0000 7f203bc12640 1 -- start start 2024-03-19T02:50:17.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.382+0000 7f203bc12640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2034100740 0x7f2034069510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:17.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.382+0000 7f203bc12640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2034101930 0x7f203419c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:17.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.382+0000 7f203bc12640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20341022f0 0x7f203419ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:17.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.382+0000 7f203bc12640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f203410c270 con 0x7f20341022f0 2024-03-19T02:50:17.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.382+0000 7f203bc12640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f203410c0f0 con 0x7f2034100740 2024-03-19T02:50:17.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.382+0000 7f203bc12640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f203410c3f0 con 0x7f2034101930 2024-03-19T02:50:17.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.382+0000 7f203a1af640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20341022f0 0x7f203419ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:17.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.382+0000 7f20399ae640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2034100740 0x7f2034069510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:17.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.382+0000 7f20399ae640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2034100740 0x7f2034069510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:42164/0 (socket says 172.21.15.33:42164) 2024-03-19T02:50:17.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.382+0000 7f203a1af640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20341022f0 0x7f203419ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58992/0 (socket says 172.21.15.33:58992) 2024-03-19T02:50:17.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.382+0000 7f20399ae640 1 -- 172.21.15.33:0/3929304401 learned_addr learned my addr 172.21.15.33:0/3929304401 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:17.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.382+0000 7f20391ad640 1 --2- 172.21.15.33:0/3929304401 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2034101930 0x7f203419c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:17.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.382+0000 7f20399ae640 1 -- 172.21.15.33:0/3929304401 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2034101930 msgr2=0x7f203419c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:17.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.382+0000 7f20399ae640 1 --2- 172.21.15.33:0/3929304401 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2034101930 0x7f203419c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:17.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.382+0000 7f20399ae640 1 -- 172.21.15.33:0/3929304401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20341022f0 msgr2=0x7f203419ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:17.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.382+0000 7f20399ae640 1 --2- 172.21.15.33:0/3929304401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20341022f0 0x7f203419ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:17.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.382+0000 7f20399ae640 1 -- 172.21.15.33:0/3929304401 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f203419f6c0 con 0x7f2034100740 2024-03-19T02:50:17.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.382+0000 7f203a1af640 1 --2- 172.21.15.33:0/3929304401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20341022f0 0x7f203419ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:17.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.382+0000 7f20399ae640 1 --2- 172.21.15.33:0/3929304401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2034100740 0x7f2034069510 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f203000d950 tx=0x7f203000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:17.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.382+0000 7f202affd640 1 -- 172.21.15.33:0/3929304401 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2030017070 con 0x7f2034100740 2024-03-19T02:50:17.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.383+0000 7f202affd640 1 -- 172.21.15.33:0/3929304401 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2030009c30 con 0x7f2034100740 2024-03-19T02:50:17.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.383+0000 7f203bc12640 1 -- 172.21.15.33:0/3929304401 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f203419f9b0 con 0x7f2034100740 2024-03-19T02:50:17.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.383+0000 7f202affd640 1 -- 172.21.15.33:0/3929304401 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f203000f620 con 0x7f2034100740 2024-03-19T02:50:17.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.383+0000 7f203bc12640 1 -- 172.21.15.33:0/3929304401 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f203419fe80 con 0x7f2034100740 2024-03-19T02:50:17.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.383+0000 7f203bc12640 1 -- 172.21.15.33:0/3929304401 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2034107b30 con 0x7f2034100740 2024-03-19T02:50:17.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.384+0000 7f202affd640 1 -- 172.21.15.33:0/3929304401 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f203001a070 con 0x7f2034100740 2024-03-19T02:50:17.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.385+0000 7f202affd640 1 --2- 172.21.15.33:0/3929304401 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f20200a2800 0x7f20200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:17.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.385+0000 7f202affd640 1 -- 172.21.15.33:0/3929304401 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f20300cdfe0 con 0x7f2034100740 2024-03-19T02:50:17.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.385+0000 7f20391ad640 1 --2- 172.21.15.33:0/3929304401 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f20200a2800 0x7f20200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:17.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.385+0000 7f20391ad640 1 --2- 172.21.15.33:0/3929304401 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f20200a2800 0x7f20200a4cc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2024009230 tx=0x7f2024005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:17.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.390+0000 7f202affd640 1 -- 172.21.15.33:0/3929304401 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2030009de0 con 0x7f2034100740 2024-03-19T02:50:17.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:17 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:17.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:17 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:17.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:17 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:17.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:17 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:17.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:17 smithi096 ceph-mon[35946]: Deploying daemon mds.k on smithi096 2024-03-19T02:50:17.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.611+0000 7f203bc12640 1 -- 172.21.15.33:0/3929304401 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f203402d450 con 0x7f2034100740 2024-03-19T02:50:17.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.612+0000 7f202affd640 1 -- 172.21.15.33:0/3929304401 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1393 (secure 0 0 0) 0x7f203009ae30 con 0x7f2034100740 2024-03-19T02:50:17.613 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:50:17.613 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-19T02:50:17.613 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 2 2024-03-19T02:50:17.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.614+0000 7f203bc12640 1 -- 172.21.15.33:0/3929304401 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f20200a2800 msgr2=0x7f20200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:17.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.614+0000 7f203bc12640 1 --2- 172.21.15.33:0/3929304401 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f20200a2800 0x7f20200a4cc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2024009230 tx=0x7f2024005fd0 comp rx=0 tx=0).stop 2024-03-19T02:50:17.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.614+0000 7f203bc12640 1 -- 172.21.15.33:0/3929304401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2034100740 msgr2=0x7f2034069510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:17.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.614+0000 7f203bc12640 1 --2- 172.21.15.33:0/3929304401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2034100740 0x7f2034069510 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f203000d950 tx=0x7f203000de10 comp rx=0 tx=0).stop 2024-03-19T02:50:17.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.614+0000 7f203bc12640 1 -- 172.21.15.33:0/3929304401 shutdown_connections 2024-03-19T02:50:17.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.614+0000 7f203bc12640 1 --2- 172.21.15.33:0/3929304401 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f20200a2800 0x7f20200a4cc0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:17.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.614+0000 7f203bc12640 1 --2- 172.21.15.33:0/3929304401 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2034101930 0x7f203419c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:17.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.614+0000 7f203bc12640 1 --2- 172.21.15.33:0/3929304401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20341022f0 0x7f203419ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:17.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.614+0000 7f203bc12640 1 --2- 172.21.15.33:0/3929304401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2034100740 0x7f2034069510 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:17.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.614+0000 7f203bc12640 1 -- 172.21.15.33:0/3929304401 >> 172.21.15.33:0/3929304401 conn(0x7f20340fbcb0 msgr2=0x7f2034104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:17.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.614+0000 7f203bc12640 1 -- 172.21.15.33:0/3929304401 shutdown_connections 2024-03-19T02:50:17.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:17.614+0000 7f203bc12640 1 -- 172.21.15.33:0/3929304401 wait complete. 2024-03-19T02:50:17.625 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:17 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:17.625 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:17 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:17.625 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:17 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:17.625 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:17 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:17.625 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:17 smithi033 ceph-mon[31973]: Deploying daemon mds.k on smithi096 2024-03-19T02:50:17.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:17 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:17.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:17 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:17.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:17 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:17.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:17 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:17.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:17 smithi155 ceph-mon[35804]: Deploying daemon mds.k on smithi096 2024-03-19T02:50:18.439 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:18 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3929304401' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:18.439 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:18 smithi096 ceph-mon[35946]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:18.716 INFO:journalctl@ceph.mds.k.smithi096.stdout:Mar 19 02:50:18 smithi096 systemd[1]: Starting Ceph mds.k for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T02:50:18.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:18 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3929304401' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:18.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:18 smithi155 ceph-mon[35804]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:18.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:18 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3929304401' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:18.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:18 smithi033 ceph-mon[31973]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:19.141 INFO:journalctl@ceph.mds.k.smithi096.stdout:Mar 19 02:50:18 smithi096 podman[64513]: 2024-03-19 02:50:18.732561063 +0000 UTC m=+0.018020721 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:50:19.141 INFO:journalctl@ceph.mds.k.smithi096.stdout:Mar 19 02:50:18 smithi096 podman[64513]: 2024-03-19 02:50:18.871110951 +0000 UTC m=+0.156570632 container create b6f149b36ab370a3c8d993026ef26705a6d08f861c1bc99996279d20fa6ebf8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-k, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True) 2024-03-19T02:50:19.141 INFO:journalctl@ceph.mds.k.smithi096.stdout:Mar 19 02:50:19 smithi096 podman[64513]: 2024-03-19 02:50:19.140726731 +0000 UTC m=+0.426186391 container init b6f149b36ab370a3c8d993026ef26705a6d08f861c1bc99996279d20fa6ebf8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-k, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-19T02:50:19.465 INFO:journalctl@ceph.mds.k.smithi096.stdout:Mar 19 02:50:19 smithi096 podman[64513]: 2024-03-19 02:50:19.143455428 +0000 UTC m=+0.428915067 container start b6f149b36ab370a3c8d993026ef26705a6d08f861c1bc99996279d20fa6ebf8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-k, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-19T02:50:19.466 INFO:journalctl@ceph.mds.k.smithi096.stdout:Mar 19 02:50:19 smithi096 ceph-mds[64532]: -- 172.21.15.96:0/1627807018 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d62cadba40 con 0x55d62caae800 2024-03-19T02:50:19.466 INFO:journalctl@ceph.mds.k.smithi096.stdout:Mar 19 02:50:19 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-k[64528]: starting mds.k at 2024-03-19T02:50:19.466 INFO:journalctl@ceph.mds.k.smithi096.stdout:Mar 19 02:50:19 smithi096 bash[64513]: b6f149b36ab370a3c8d993026ef26705a6d08f861c1bc99996279d20fa6ebf8b 2024-03-19T02:50:19.466 INFO:journalctl@ceph.mds.k.smithi096.stdout:Mar 19 02:50:19 smithi096 systemd[1]: Started Ceph mds.k for 2b7a8938-e59a-11ee-95c9-87774f69a715. 2024-03-19T02:50:19.627 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:50:19.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.730+0000 7f4b9b9a9640 1 -- 172.21.15.33:0/1718040559 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b94101930 msgr2=0x7f4b94101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:19.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.730+0000 7f4b9b9a9640 1 --2- 172.21.15.33:0/1718040559 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b94101930 0x7f4b94101db0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4b84009900 tx=0x7f4b8402bc30 comp rx=0 tx=0).stop 2024-03-19T02:50:19.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.730+0000 7f4b9b9a9640 1 -- 172.21.15.33:0/1718040559 shutdown_connections 2024-03-19T02:50:19.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.730+0000 7f4b9b9a9640 1 --2- 172.21.15.33:0/1718040559 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b941022f0 0x7f4b94108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:19.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.730+0000 7f4b9b9a9640 1 --2- 172.21.15.33:0/1718040559 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b94101930 0x7f4b94101db0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:19.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.730+0000 7f4b9b9a9640 1 --2- 172.21.15.33:0/1718040559 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b94100740 0x7f4b94100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:19.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.730+0000 7f4b9b9a9640 1 -- 172.21.15.33:0/1718040559 >> 172.21.15.33:0/1718040559 conn(0x7f4b940fbcb0 msgr2=0x7f4b940fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:19.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.730+0000 7f4b9b9a9640 1 -- 172.21.15.33:0/1718040559 shutdown_connections 2024-03-19T02:50:19.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.730+0000 7f4b9b9a9640 1 -- 172.21.15.33:0/1718040559 wait complete. 2024-03-19T02:50:19.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.731+0000 7f4b9b9a9640 1 Processor -- start 2024-03-19T02:50:19.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.731+0000 7f4b9b9a9640 1 -- start start 2024-03-19T02:50:19.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.731+0000 7f4b9b9a9640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b94100740 0x7f4b941979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:19.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.731+0000 7f4b9b9a9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b94101930 0x7f4b94197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:19.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.731+0000 7f4b9b9a9640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b941022f0 0x7f4b9419ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:19.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.731+0000 7f4b9b9a9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4b9410c250 con 0x7f4b94101930 2024-03-19T02:50:19.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.731+0000 7f4b9b9a9640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4b9410c0d0 con 0x7f4b941022f0 2024-03-19T02:50:19.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.731+0000 7f4b9b9a9640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4b9410c3d0 con 0x7f4b94100740 2024-03-19T02:50:19.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.731+0000 7f4b98f44640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b94101930 0x7f4b94197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:19.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.731+0000 7f4b98f44640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b94101930 0x7f4b94197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59002/0 (socket says 172.21.15.33:59002) 2024-03-19T02:50:19.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.731+0000 7f4b98f44640 1 -- 172.21.15.33:0/3326141830 learned_addr learned my addr 172.21.15.33:0/3326141830 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:19.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.731+0000 7f4b99745640 1 --2- 172.21.15.33:0/3326141830 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b94100740 0x7f4b941979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:19.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.731+0000 7f4b99f46640 1 --2- 172.21.15.33:0/3326141830 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b941022f0 0x7f4b9419ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:19.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.732+0000 7f4b98f44640 1 -- 172.21.15.33:0/3326141830 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b94100740 msgr2=0x7f4b941979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:19.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.732+0000 7f4b98f44640 1 --2- 172.21.15.33:0/3326141830 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b94100740 0x7f4b941979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:19.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.732+0000 7f4b98f44640 1 -- 172.21.15.33:0/3326141830 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b941022f0 msgr2=0x7f4b9419ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:19.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.732+0000 7f4b98f44640 1 --2- 172.21.15.33:0/3326141830 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b941022f0 0x7f4b9419ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:19.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.732+0000 7f4b98f44640 1 -- 172.21.15.33:0/3326141830 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b9419f600 con 0x7f4b94101930 2024-03-19T02:50:19.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.732+0000 7f4b98f44640 1 --2- 172.21.15.33:0/3326141830 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b94101930 0x7f4b94197f20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4b84006010 tx=0x7f4b840042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:19.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.732+0000 7f4b8a7fc640 1 -- 172.21.15.33:0/3326141830 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b84043020 con 0x7f4b94101930 2024-03-19T02:50:19.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.732+0000 7f4b9b9a9640 1 -- 172.21.15.33:0/3326141830 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b9419f890 con 0x7f4b94101930 2024-03-19T02:50:19.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.732+0000 7f4b8a7fc640 1 -- 172.21.15.33:0/3326141830 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b84004aa0 con 0x7f4b94101930 2024-03-19T02:50:19.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.732+0000 7f4b8a7fc640 1 -- 172.21.15.33:0/3326141830 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b84003c60 con 0x7f4b94101930 2024-03-19T02:50:19.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.732+0000 7f4b9b9a9640 1 -- 172.21.15.33:0/3326141830 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b9419fde0 con 0x7f4b94101930 2024-03-19T02:50:19.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.733+0000 7f4b9b9a9640 1 -- 172.21.15.33:0/3326141830 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b941a0040 con 0x7f4b94101930 2024-03-19T02:50:19.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.735+0000 7f4b8a7fc640 1 -- 172.21.15.33:0/3326141830 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f4b84046020 con 0x7f4b94101930 2024-03-19T02:50:19.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.736+0000 7f4b8a7fc640 1 --2- 172.21.15.33:0/3326141830 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4b800a2800 0x7f4b800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:19.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.736+0000 7f4b99745640 1 --2- 172.21.15.33:0/3326141830 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4b800a2800 0x7f4b800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:19.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.736+0000 7f4b8a7fc640 1 -- 172.21.15.33:0/3326141830 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f4b840f0990 con 0x7f4b94101930 2024-03-19T02:50:19.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.736+0000 7f4b99745640 1 --2- 172.21.15.33:0/3326141830 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4b800a2800 0x7f4b800a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4b94101790 tx=0x7f4b90005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:19.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.740+0000 7f4b8a7fc640 1 -- 172.21.15.33:0/3326141830 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4b8404d020 con 0x7f4b94101930 2024-03-19T02:50:19.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.978+0000 7f4b9b9a9640 1 -- 172.21.15.33:0/3326141830 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b941a0210 con 0x7f4b94101930 2024-03-19T02:50:19.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.978+0000 7f4b8a7fc640 1 -- 172.21.15.33:0/3326141830 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2218 (secure 0 0 0) 0x7f4b840bde30 con 0x7f4b94101930 2024-03-19T02:50:19.980 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:50:19.980 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-19T02:50:19.980 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 3 2024-03-19T02:50:19.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.980+0000 7f4b9b9a9640 1 -- 172.21.15.33:0/3326141830 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4b800a2800 msgr2=0x7f4b800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:19.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.980+0000 7f4b9b9a9640 1 --2- 172.21.15.33:0/3326141830 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4b800a2800 0x7f4b800a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4b94101790 tx=0x7f4b90005f90 comp rx=0 tx=0).stop 2024-03-19T02:50:19.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.980+0000 7f4b9b9a9640 1 -- 172.21.15.33:0/3326141830 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b94101930 msgr2=0x7f4b94197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:19.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.980+0000 7f4b9b9a9640 1 --2- 172.21.15.33:0/3326141830 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b94101930 0x7f4b94197f20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4b84006010 tx=0x7f4b840042f0 comp rx=0 tx=0).stop 2024-03-19T02:50:19.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.980+0000 7f4b9b9a9640 1 -- 172.21.15.33:0/3326141830 shutdown_connections 2024-03-19T02:50:19.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.980+0000 7f4b9b9a9640 1 --2- 172.21.15.33:0/3326141830 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4b800a2800 0x7f4b800a4cc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:19.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.980+0000 7f4b9b9a9640 1 --2- 172.21.15.33:0/3326141830 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b941022f0 0x7f4b9419ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:19.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.980+0000 7f4b9b9a9640 1 --2- 172.21.15.33:0/3326141830 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b94101930 0x7f4b94197f20 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:19.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.980+0000 7f4b9b9a9640 1 --2- 172.21.15.33:0/3326141830 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b94100740 0x7f4b941979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:19.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.980+0000 7f4b9b9a9640 1 -- 172.21.15.33:0/3326141830 >> 172.21.15.33:0/3326141830 conn(0x7f4b940fbcb0 msgr2=0x7f4b940fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:19.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.981+0000 7f4b9b9a9640 1 -- 172.21.15.33:0/3326141830 shutdown_connections 2024-03-19T02:50:19.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:19.981+0000 7f4b9b9a9640 1 -- 172.21.15.33:0/3326141830 wait complete. 2024-03-19T02:50:20.568 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:20 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:20.569 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:20 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:20.569 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:20 smithi155 ceph-mon[35804]: mds.? [v2:172.21.15.96:6832/2137965374,v1:172.21.15.96:6833/2137965374] up:boot 2024-03-19T02:50:20.569 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:20 smithi155 ceph-mon[35804]: fsmap 2 up:standby 2024-03-19T02:50:20.569 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:20 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-19T02:50:20.569 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:20 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:20.569 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:20 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:20.569 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:20 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:20.569 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:20 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:20.569 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:20 smithi155 ceph-mon[35804]: Deploying daemon mds.j on smithi033 2024-03-19T02:50:20.569 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:20 smithi155 ceph-mon[35804]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:20.569 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:20 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3326141830' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:20.572 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:20 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:20.572 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:20 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:20.572 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:20 smithi096 ceph-mon[35946]: mds.? [v2:172.21.15.96:6832/2137965374,v1:172.21.15.96:6833/2137965374] up:boot 2024-03-19T02:50:20.572 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:20 smithi096 ceph-mon[35946]: fsmap 2 up:standby 2024-03-19T02:50:20.572 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:20 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-19T02:50:20.572 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:20 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:20.572 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:20 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:20.572 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:20 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:20.572 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:20 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:20.573 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:20 smithi096 ceph-mon[35946]: Deploying daemon mds.j on smithi033 2024-03-19T02:50:20.573 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:20 smithi096 ceph-mon[35946]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:20.573 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:20 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3326141830' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:20.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:20 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:20.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:20 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:20.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:20 smithi033 ceph-mon[31973]: mds.? [v2:172.21.15.96:6832/2137965374,v1:172.21.15.96:6833/2137965374] up:boot 2024-03-19T02:50:20.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:20 smithi033 ceph-mon[31973]: fsmap 2 up:standby 2024-03-19T02:50:20.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:20 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-19T02:50:20.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:20 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:20.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:20 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:20.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:20 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:20.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:20 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:20.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:20 smithi033 ceph-mon[31973]: Deploying daemon mds.j on smithi033 2024-03-19T02:50:20.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:20 smithi033 ceph-mon[31973]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:20.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:20 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3326141830' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:21.698 INFO:journalctl@ceph.mds.j.smithi033.stdout:Mar 19 02:50:21 smithi033 systemd[1]: Starting Ceph mds.j for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T02:50:21.962 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:21 smithi033 ceph-mon[31973]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:21.962 INFO:journalctl@ceph.mds.j.smithi033.stdout:Mar 19 02:50:21 smithi033 podman[72479]: 2024-03-19 02:50:21.600499553 +0000 UTC m=+0.018345325 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:50:21.962 INFO:journalctl@ceph.mds.j.smithi033.stdout:Mar 19 02:50:21 smithi033 podman[72479]: 2024-03-19 02:50:21.820645716 +0000 UTC m=+0.238491532 container create deeb835197b670b7212587fbe8589c99e8b85a812c1fc46ada0d21cb5dffcfda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-j, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T02:50:21.993 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:50:22.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:21 smithi155 ceph-mon[35804]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:22.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.100+0000 7f9db9bf3640 1 -- 172.21.15.33:0/308599059 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9db41022f0 msgr2=0x7f9db4108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:22.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.100+0000 7f9db9bf3640 1 --2- 172.21.15.33:0/308599059 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9db41022f0 0x7f9db4108b80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9dac009ca0 tx=0x7f9dac02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:50:22.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.100+0000 7f9db9bf3640 1 -- 172.21.15.33:0/308599059 shutdown_connections 2024-03-19T02:50:22.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.100+0000 7f9db9bf3640 1 --2- 172.21.15.33:0/308599059 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9db4101930 0x7f9db4101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:22.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.100+0000 7f9db9bf3640 1 --2- 172.21.15.33:0/308599059 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9db41022f0 0x7f9db4108b80 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:22.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.100+0000 7f9db9bf3640 1 --2- 172.21.15.33:0/308599059 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9db4100740 0x7f9db4100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:22.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.100+0000 7f9db9bf3640 1 -- 172.21.15.33:0/308599059 >> 172.21.15.33:0/308599059 conn(0x7f9db40fbcd0 msgr2=0x7f9db40fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:22.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.100+0000 7f9db9bf3640 1 -- 172.21.15.33:0/308599059 shutdown_connections 2024-03-19T02:50:22.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.100+0000 7f9db9bf3640 1 -- 172.21.15.33:0/308599059 wait complete. 2024-03-19T02:50:22.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.100+0000 7f9db9bf3640 1 Processor -- start 2024-03-19T02:50:22.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.100+0000 7f9db9bf3640 1 -- start start 2024-03-19T02:50:22.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.100+0000 7f9db9bf3640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9db4100740 0x7f9db4197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:22.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.100+0000 7f9db9bf3640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9db4101930 0x7f9db4197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:22.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.100+0000 7f9db9bf3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9db41022f0 0x7f9db419efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:22.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.100+0000 7f9db9bf3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9db410d390 con 0x7f9db41022f0 2024-03-19T02:50:22.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.100+0000 7f9db9bf3640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9db410d210 con 0x7f9db4100740 2024-03-19T02:50:22.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.100+0000 7f9db9bf3640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9db410d510 con 0x7f9db4101930 2024-03-19T02:50:22.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.101+0000 7f9db3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9db41022f0 0x7f9db419efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:22.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.101+0000 7f9db37fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9db4100740 0x7f9db4197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:22.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.101+0000 7f9db3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9db41022f0 0x7f9db419efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59030/0 (socket says 172.21.15.33:59030) 2024-03-19T02:50:22.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.101+0000 7f9db3fff640 1 -- 172.21.15.33:0/199974812 learned_addr learned my addr 172.21.15.33:0/199974812 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:22.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.101+0000 7f9db37fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9db4100740 0x7f9db4197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:42210/0 (socket says 172.21.15.33:42210) 2024-03-19T02:50:22.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.101+0000 7f9db2ffd640 1 --2- 172.21.15.33:0/199974812 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9db4101930 0x7f9db4197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:22.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.101+0000 7f9db3fff640 1 -- 172.21.15.33:0/199974812 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9db4101930 msgr2=0x7f9db4197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:22.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.101+0000 7f9db3fff640 1 --2- 172.21.15.33:0/199974812 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9db4101930 0x7f9db4197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:22.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.101+0000 7f9db3fff640 1 -- 172.21.15.33:0/199974812 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9db4100740 msgr2=0x7f9db4197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:22.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.101+0000 7f9db3fff640 1 --2- 172.21.15.33:0/199974812 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9db4100740 0x7f9db4197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:22.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.101+0000 7f9db3fff640 1 -- 172.21.15.33:0/199974812 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9db419f640 con 0x7f9db41022f0 2024-03-19T02:50:22.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.101+0000 7f9db37fe640 1 --2- 172.21.15.33:0/199974812 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9db4100740 0x7f9db4197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:22.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.101+0000 7f9db3fff640 1 --2- 172.21.15.33:0/199974812 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9db41022f0 0x7f9db419efd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9dac02c700 tx=0x7f9dac0328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:22.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.101+0000 7f9db0ff9640 1 -- 172.21.15.33:0/199974812 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dac041020 con 0x7f9db41022f0 2024-03-19T02:50:22.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.101+0000 7f9db9bf3640 1 -- 172.21.15.33:0/199974812 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9db419f8d0 con 0x7f9db41022f0 2024-03-19T02:50:22.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.101+0000 7f9db0ff9640 1 -- 172.21.15.33:0/199974812 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9dac003e20 con 0x7f9db41022f0 2024-03-19T02:50:22.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.101+0000 7f9db0ff9640 1 -- 172.21.15.33:0/199974812 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dac039040 con 0x7f9db41022f0 2024-03-19T02:50:22.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.101+0000 7f9db9bf3640 1 -- 172.21.15.33:0/199974812 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9db419fe10 con 0x7f9db41022f0 2024-03-19T02:50:22.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.102+0000 7f9db9bf3640 1 -- 172.21.15.33:0/199974812 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9db4107ab0 con 0x7f9db41022f0 2024-03-19T02:50:22.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.104+0000 7f9db0ff9640 1 -- 172.21.15.33:0/199974812 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f9dac044020 con 0x7f9db41022f0 2024-03-19T02:50:22.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.105+0000 7f9db0ff9640 1 --2- 172.21.15.33:0/199974812 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9d9c0a2800 0x7f9d9c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:22.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.105+0000 7f9db0ff9640 1 -- 172.21.15.33:0/199974812 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f9dac0f0c90 con 0x7f9db41022f0 2024-03-19T02:50:22.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.105+0000 7f9db37fe640 1 --2- 172.21.15.33:0/199974812 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9d9c0a2800 0x7f9d9c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:22.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.105+0000 7f9db37fe640 1 --2- 172.21.15.33:0/199974812 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9d9c0a2800 0x7f9d9c0a4cc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9db4101790 tx=0x7f9da4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:22.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.108+0000 7f9db0ff9640 1 -- 172.21.15.33:0/199974812 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9dac04b020 con 0x7f9db41022f0 2024-03-19T02:50:22.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:21 smithi096 ceph-mon[35946]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:22.278 INFO:journalctl@ceph.mds.j.smithi033.stdout:Mar 19 02:50:22 smithi033 podman[72479]: 2024-03-19 02:50:22.082262071 +0000 UTC m=+0.500107843 container init deeb835197b670b7212587fbe8589c99e8b85a812c1fc46ada0d21cb5dffcfda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-j, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True) 2024-03-19T02:50:22.279 INFO:journalctl@ceph.mds.j.smithi033.stdout:Mar 19 02:50:22 smithi033 podman[72479]: 2024-03-19 02:50:22.085476215 +0000 UTC m=+0.503321986 container start deeb835197b670b7212587fbe8589c99e8b85a812c1fc46ada0d21cb5dffcfda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-j, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-19T02:50:22.279 INFO:journalctl@ceph.mds.j.smithi033.stdout:Mar 19 02:50:22 smithi033 ceph-mds[72518]: -- 172.21.15.33:0/767452273 <== mon.0 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f8cfe45a40 con 0x55f8cfe18800 2024-03-19T02:50:22.279 INFO:journalctl@ceph.mds.j.smithi033.stdout:Mar 19 02:50:22 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-j[72494]: starting mds.j at 2024-03-19T02:50:22.279 INFO:journalctl@ceph.mds.j.smithi033.stdout:Mar 19 02:50:22 smithi033 bash[72479]: deeb835197b670b7212587fbe8589c99e8b85a812c1fc46ada0d21cb5dffcfda 2024-03-19T02:50:22.279 INFO:journalctl@ceph.mds.j.smithi033.stdout:Mar 19 02:50:22 smithi033 systemd[1]: Started Ceph mds.j for 2b7a8938-e59a-11ee-95c9-87774f69a715. 2024-03-19T02:50:22.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.338+0000 7f9db9bf3640 1 -- 172.21.15.33:0/199974812 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9db4062420 con 0x7f9db41022f0 2024-03-19T02:50:22.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.339+0000 7f9db0ff9640 1 -- 172.21.15.33:0/199974812 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2218 (secure 0 0 0) 0x7f9dac0be130 con 0x7f9db41022f0 2024-03-19T02:50:22.340 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:50:22.340 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-19T02:50:22.340 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 3 2024-03-19T02:50:22.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.341+0000 7f9db9bf3640 1 -- 172.21.15.33:0/199974812 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9d9c0a2800 msgr2=0x7f9d9c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:22.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.341+0000 7f9db9bf3640 1 --2- 172.21.15.33:0/199974812 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9d9c0a2800 0x7f9d9c0a4cc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9db4101790 tx=0x7f9da4005f90 comp rx=0 tx=0).stop 2024-03-19T02:50:22.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.341+0000 7f9db9bf3640 1 -- 172.21.15.33:0/199974812 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9db41022f0 msgr2=0x7f9db419efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:22.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.341+0000 7f9db9bf3640 1 --2- 172.21.15.33:0/199974812 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9db41022f0 0x7f9db419efd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9dac02c700 tx=0x7f9dac0328a0 comp rx=0 tx=0).stop 2024-03-19T02:50:22.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.341+0000 7f9db9bf3640 1 -- 172.21.15.33:0/199974812 shutdown_connections 2024-03-19T02:50:22.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.341+0000 7f9db9bf3640 1 --2- 172.21.15.33:0/199974812 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9d9c0a2800 0x7f9d9c0a4cc0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:22.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.341+0000 7f9db9bf3640 1 --2- 172.21.15.33:0/199974812 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9db4101930 0x7f9db4197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:22.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.341+0000 7f9db9bf3640 1 --2- 172.21.15.33:0/199974812 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9db41022f0 0x7f9db419efd0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:22.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.341+0000 7f9db9bf3640 1 --2- 172.21.15.33:0/199974812 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9db4100740 0x7f9db4197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:22.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.341+0000 7f9db9bf3640 1 -- 172.21.15.33:0/199974812 >> 172.21.15.33:0/199974812 conn(0x7f9db40fbcd0 msgr2=0x7f9db4104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:22.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.341+0000 7f9db9bf3640 1 -- 172.21.15.33:0/199974812 shutdown_connections 2024-03-19T02:50:22.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:22.341+0000 7f9db9bf3640 1 -- 172.21.15.33:0/199974812 wait complete. 2024-03-19T02:50:23.483 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:23 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:23.483 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:23 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:23.483 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:23 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:23.483 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:23 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:23.483 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:23 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:23.483 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:23 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:23.483 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:23 smithi033 ceph-mon[31973]: Deploying daemon mds.d on smithi033 2024-03-19T02:50:23.483 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:23 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/199974812' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:23.483 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:23 smithi033 ceph-mon[31973]: mds.? [v2:172.21.15.33:6834/973717150,v1:172.21.15.33:6835/973717150] up:boot 2024-03-19T02:50:23.483 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:23 smithi033 ceph-mon[31973]: fsmap 3 up:standby 2024-03-19T02:50:23.484 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:23 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-19T02:50:23.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:23 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:23.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:23 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:23.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:23 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:23.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:23 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:23.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:23 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:23.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:23 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:23.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:23 smithi155 ceph-mon[35804]: Deploying daemon mds.d on smithi033 2024-03-19T02:50:23.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:23 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/199974812' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:23.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:23 smithi155 ceph-mon[35804]: mds.? [v2:172.21.15.33:6834/973717150,v1:172.21.15.33:6835/973717150] up:boot 2024-03-19T02:50:23.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:23 smithi155 ceph-mon[35804]: fsmap 3 up:standby 2024-03-19T02:50:23.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:23 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-19T02:50:23.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:23 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:23.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:23 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:23.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:23 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:23.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:23 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:23.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:23 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:23.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:23 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:23.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:23 smithi096 ceph-mon[35946]: Deploying daemon mds.d on smithi033 2024-03-19T02:50:23.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:23 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/199974812' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:23.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:23 smithi096 ceph-mon[35946]: mds.? [v2:172.21.15.33:6834/973717150,v1:172.21.15.33:6835/973717150] up:boot 2024-03-19T02:50:23.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:23 smithi096 ceph-mon[35946]: fsmap 3 up:standby 2024-03-19T02:50:23.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:23 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-19T02:50:24.354 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:50:24.380 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:24 smithi033 ceph-mon[31973]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:24.380 INFO:journalctl@ceph.mds.d.smithi033.stdout:Mar 19 02:50:24 smithi033 systemd[1]: Starting Ceph mds.d for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T02:50:24.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.460+0000 7f404e2d1640 1 -- 172.21.15.33:0/3243402566 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4048101930 msgr2=0x7f4048101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:24.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.460+0000 7f404e2d1640 1 --2- 172.21.15.33:0/3243402566 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4048101930 0x7f4048101db0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4038009900 tx=0x7f403802bc30 comp rx=0 tx=0).stop 2024-03-19T02:50:24.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.460+0000 7f404e2d1640 1 -- 172.21.15.33:0/3243402566 shutdown_connections 2024-03-19T02:50:24.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.460+0000 7f404e2d1640 1 --2- 172.21.15.33:0/3243402566 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f40481022f0 0x7f4048108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:24.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.460+0000 7f404e2d1640 1 --2- 172.21.15.33:0/3243402566 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4048101930 0x7f4048101db0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:24.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.460+0000 7f404e2d1640 1 --2- 172.21.15.33:0/3243402566 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4048100740 0x7f4048100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:24.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.460+0000 7f404e2d1640 1 -- 172.21.15.33:0/3243402566 >> 172.21.15.33:0/3243402566 conn(0x7f40480fbcb0 msgr2=0x7f40480fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:24.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.460+0000 7f404e2d1640 1 -- 172.21.15.33:0/3243402566 shutdown_connections 2024-03-19T02:50:24.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.460+0000 7f404e2d1640 1 -- 172.21.15.33:0/3243402566 wait complete. 2024-03-19T02:50:24.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.460+0000 7f404e2d1640 1 Processor -- start 2024-03-19T02:50:24.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.460+0000 7f404e2d1640 1 -- start start 2024-03-19T02:50:24.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.461+0000 7f404e2d1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4048100740 0x7f4048069510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:24.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.461+0000 7f404e2d1640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4048101930 0x7f404819c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:24.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.461+0000 7f404e2d1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40481022f0 0x7f404819ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:24.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.461+0000 7f404e2d1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f404810c270 con 0x7f4048100740 2024-03-19T02:50:24.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.461+0000 7f404e2d1640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f404810c0f0 con 0x7f40481022f0 2024-03-19T02:50:24.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.461+0000 7f404e2d1640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f404810c3f0 con 0x7f4048101930 2024-03-19T02:50:24.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.461+0000 7f4047fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4048100740 0x7f4048069510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:24.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.461+0000 7f4047fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4048100740 0x7f4048069510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57288/0 (socket says 172.21.15.33:57288) 2024-03-19T02:50:24.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.461+0000 7f4047fff640 1 -- 172.21.15.33:0/3805967880 learned_addr learned my addr 172.21.15.33:0/3805967880 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:24.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.461+0000 7f40477fe640 1 --2- 172.21.15.33:0/3805967880 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4048101930 0x7f404819c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:24.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.461+0000 7f4047fff640 1 -- 172.21.15.33:0/3805967880 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4048101930 msgr2=0x7f404819c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:24.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.461+0000 7f4047fff640 1 --2- 172.21.15.33:0/3805967880 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4048101930 0x7f404819c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:24.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.461+0000 7f4047fff640 1 -- 172.21.15.33:0/3805967880 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40481022f0 msgr2=0x7f404819ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:24.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.461+0000 7f4047fff640 1 --2- 172.21.15.33:0/3805967880 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40481022f0 0x7f404819ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:24.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.461+0000 7f4047fff640 1 -- 172.21.15.33:0/3805967880 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f404819f6e0 con 0x7f4048100740 2024-03-19T02:50:24.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.461+0000 7f40477fe640 1 --2- 172.21.15.33:0/3805967880 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4048101930 0x7f404819c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:50:24.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.461+0000 7f4047fff640 1 --2- 172.21.15.33:0/3805967880 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4048100740 0x7f4048069510 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f403c00e9c0 tx=0x7f403c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:24.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.461+0000 7f40457fa640 1 -- 172.21.15.33:0/3805967880 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f403c00cd80 con 0x7f4048100740 2024-03-19T02:50:24.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.461+0000 7f40457fa640 1 -- 172.21.15.33:0/3805967880 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f403c0035f0 con 0x7f4048100740 2024-03-19T02:50:24.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.461+0000 7f404e2d1640 1 -- 172.21.15.33:0/3805967880 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f404819f9d0 con 0x7f4048100740 2024-03-19T02:50:24.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.461+0000 7f40457fa640 1 -- 172.21.15.33:0/3805967880 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f403c010690 con 0x7f4048100740 2024-03-19T02:50:24.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.461+0000 7f404e2d1640 1 -- 172.21.15.33:0/3805967880 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f404819ff20 con 0x7f4048100740 2024-03-19T02:50:24.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.462+0000 7f404e2d1640 1 -- 172.21.15.33:0/3805967880 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4048107ad0 con 0x7f4048100740 2024-03-19T02:50:24.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.463+0000 7f40457fa640 1 -- 172.21.15.33:0/3805967880 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f403c0037a0 con 0x7f4048100740 2024-03-19T02:50:24.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.464+0000 7f40457fa640 1 --2- 172.21.15.33:0/3805967880 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f40300a2800 0x7f40300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:24.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.464+0000 7f40477fe640 1 --2- 172.21.15.33:0/3805967880 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f40300a2800 0x7f40300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:24.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.464+0000 7f40457fa640 1 -- 172.21.15.33:0/3805967880 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f403c0cea90 con 0x7f4048100740 2024-03-19T02:50:24.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.464+0000 7f40477fe640 1 --2- 172.21.15.33:0/3805967880 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f40300a2800 0x7f40300a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f40380096c0 tx=0x7f4038000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:24.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.469+0000 7f40457fa640 1 -- 172.21.15.33:0/3805967880 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f403c09bfd0 con 0x7f4048100740 2024-03-19T02:50:24.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:24 smithi155 ceph-mon[35804]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:24.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.707+0000 7f404e2d1640 1 -- 172.21.15.33:0/3805967880 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f404802d1f0 con 0x7f4048100740 2024-03-19T02:50:24.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.707+0000 7f40457fa640 1 -- 172.21.15.33:0/3805967880 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3040 (secure 0 0 0) 0x7f403c09b8d0 con 0x7f4048100740 2024-03-19T02:50:24.709 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:50:24.709 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-19T02:50:24.709 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 4 2024-03-19T02:50:24.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.710+0000 7f404e2d1640 1 -- 172.21.15.33:0/3805967880 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f40300a2800 msgr2=0x7f40300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:24.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.710+0000 7f404e2d1640 1 --2- 172.21.15.33:0/3805967880 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f40300a2800 0x7f40300a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f40380096c0 tx=0x7f4038000bc0 comp rx=0 tx=0).stop 2024-03-19T02:50:24.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.710+0000 7f404e2d1640 1 -- 172.21.15.33:0/3805967880 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4048100740 msgr2=0x7f4048069510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:24.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.710+0000 7f404e2d1640 1 --2- 172.21.15.33:0/3805967880 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4048100740 0x7f4048069510 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f403c00e9c0 tx=0x7f403c00ee80 comp rx=0 tx=0).stop 2024-03-19T02:50:24.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.710+0000 7f404e2d1640 1 -- 172.21.15.33:0/3805967880 shutdown_connections 2024-03-19T02:50:24.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.710+0000 7f404e2d1640 1 --2- 172.21.15.33:0/3805967880 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f40300a2800 0x7f40300a4cc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:24.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.710+0000 7f404e2d1640 1 --2- 172.21.15.33:0/3805967880 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4048101930 0x7f404819c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:24.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.710+0000 7f404e2d1640 1 --2- 172.21.15.33:0/3805967880 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40481022f0 0x7f404819ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:24.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.710+0000 7f404e2d1640 1 --2- 172.21.15.33:0/3805967880 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4048100740 0x7f4048069510 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:24.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.710+0000 7f404e2d1640 1 -- 172.21.15.33:0/3805967880 >> 172.21.15.33:0/3805967880 conn(0x7f40480fbcb0 msgr2=0x7f4048104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:24.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.710+0000 7f404e2d1640 1 -- 172.21.15.33:0/3805967880 shutdown_connections 2024-03-19T02:50:24.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:24.710+0000 7f404e2d1640 1 -- 172.21.15.33:0/3805967880 wait complete. 2024-03-19T02:50:24.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:24 smithi096 ceph-mon[35946]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:24.721 INFO:journalctl@ceph.mds.d.smithi033.stdout:Mar 19 02:50:24 smithi033 podman[72831]: 2024-03-19 02:50:24.360311412 +0000 UTC m=+0.017386799 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:50:24.721 INFO:journalctl@ceph.mds.d.smithi033.stdout:Mar 19 02:50:24 smithi033 podman[72831]: 2024-03-19 02:50:24.494198698 +0000 UTC m=+0.151274063 container create fb33411f4f2db1d6ce6a0e823652a54fa2086ded7737949f63daedc0c51972fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-d, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True) 2024-03-19T02:50:25.029 INFO:journalctl@ceph.mds.d.smithi033.stdout:Mar 19 02:50:24 smithi033 podman[72831]: 2024-03-19 02:50:24.72218136 +0000 UTC m=+0.379256730 container init fb33411f4f2db1d6ce6a0e823652a54fa2086ded7737949f63daedc0c51972fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-d, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-19T02:50:25.029 INFO:journalctl@ceph.mds.d.smithi033.stdout:Mar 19 02:50:24 smithi033 podman[72831]: 2024-03-19 02:50:24.724906298 +0000 UTC m=+0.381981667 container start fb33411f4f2db1d6ce6a0e823652a54fa2086ded7737949f63daedc0c51972fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-d, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux ) 2024-03-19T02:50:25.029 INFO:journalctl@ceph.mds.d.smithi033.stdout:Mar 19 02:50:24 smithi033 ceph-mds[72890]: -- 172.21.15.33:0/982121190 <== mon.0 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c62e9dfa40 con 0x55c62e9b2c00 2024-03-19T02:50:25.029 INFO:journalctl@ceph.mds.d.smithi033.stdout:Mar 19 02:50:24 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-d[72884]: starting mds.d at 2024-03-19T02:50:25.029 INFO:journalctl@ceph.mds.d.smithi033.stdout:Mar 19 02:50:24 smithi033 bash[72831]: fb33411f4f2db1d6ce6a0e823652a54fa2086ded7737949f63daedc0c51972fa 2024-03-19T02:50:25.029 INFO:journalctl@ceph.mds.d.smithi033.stdout:Mar 19 02:50:24 smithi033 systemd[1]: Started Ceph mds.d for 2b7a8938-e59a-11ee-95c9-87774f69a715. 2024-03-19T02:50:25.515 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:25 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3805967880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:25.515 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:25 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:25.515 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:25 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:25.515 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:25 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:25.515 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:25 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:25.515 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:25 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:25.515 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:25 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:25.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:25 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3805967880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:25.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:25 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:25.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:25 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:25.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:25 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:25.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:25 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:25.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:25 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:25.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:25 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:25.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:25 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3805967880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:25.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:25 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:25.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:25 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:25.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:25 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:25.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:25 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:25.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:25 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:25.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:25 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:26.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:26 smithi155 ceph-mon[35804]: Deploying daemon mds.e on smithi096 2024-03-19T02:50:26.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:26 smithi155 ceph-mon[35804]: mds.? [v2:172.21.15.33:6836/3063370040,v1:172.21.15.33:6837/3063370040] up:boot 2024-03-19T02:50:26.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:26 smithi155 ceph-mon[35804]: fsmap 4 up:standby 2024-03-19T02:50:26.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:26 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-19T02:50:26.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:26 smithi155 ceph-mon[35804]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:26.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:26 smithi033 ceph-mon[31973]: Deploying daemon mds.e on smithi096 2024-03-19T02:50:26.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:26 smithi033 ceph-mon[31973]: mds.? [v2:172.21.15.33:6836/3063370040,v1:172.21.15.33:6837/3063370040] up:boot 2024-03-19T02:50:26.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:26 smithi033 ceph-mon[31973]: fsmap 4 up:standby 2024-03-19T02:50:26.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:26 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-19T02:50:26.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:26 smithi033 ceph-mon[31973]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:26.544 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:26 smithi096 ceph-mon[35946]: Deploying daemon mds.e on smithi096 2024-03-19T02:50:26.544 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:26 smithi096 ceph-mon[35946]: mds.? [v2:172.21.15.33:6836/3063370040,v1:172.21.15.33:6837/3063370040] up:boot 2024-03-19T02:50:26.544 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:26 smithi096 ceph-mon[35946]: fsmap 4 up:standby 2024-03-19T02:50:26.545 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:26 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-19T02:50:26.545 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:26 smithi096 ceph-mon[35946]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:26.723 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:50:26.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.838+0000 7f97ad23f640 1 -- 172.21.15.33:0/4139313055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97a8100740 msgr2=0x7f97a8100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:26.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.838+0000 7f97ad23f640 1 --2- 172.21.15.33:0/4139313055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97a8100740 0x7f97a8100b60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f97a0009900 tx=0x7f97a002bc30 comp rx=0 tx=0).stop 2024-03-19T02:50:26.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.838+0000 7f97ad23f640 1 -- 172.21.15.33:0/4139313055 shutdown_connections 2024-03-19T02:50:26.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.838+0000 7f97ad23f640 1 --2- 172.21.15.33:0/4139313055 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97a8101930 0x7f97a8101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:26.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.838+0000 7f97ad23f640 1 --2- 172.21.15.33:0/4139313055 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97a81022f0 0x7f97a8108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:26.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.838+0000 7f97ad23f640 1 --2- 172.21.15.33:0/4139313055 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97a8100740 0x7f97a8100b60 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:26.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.838+0000 7f97ad23f640 1 -- 172.21.15.33:0/4139313055 >> 172.21.15.33:0/4139313055 conn(0x7f97a80fbcb0 msgr2=0x7f97a80fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:26.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.838+0000 7f97ad23f640 1 -- 172.21.15.33:0/4139313055 shutdown_connections 2024-03-19T02:50:26.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.838+0000 7f97ad23f640 1 -- 172.21.15.33:0/4139313055 wait complete. 2024-03-19T02:50:26.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.838+0000 7f97ad23f640 1 Processor -- start 2024-03-19T02:50:26.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.839+0000 7f97ad23f640 1 -- start start 2024-03-19T02:50:26.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.839+0000 7f97ad23f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97a8100740 0x7f97a81969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:26.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.839+0000 7f97ad23f640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97a8101930 0x7f97a8196f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:26.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.839+0000 7f97ad23f640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97a81022f0 0x7f97a819dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:26.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.839+0000 7f97ad23f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f97a810c290 con 0x7f97a8100740 2024-03-19T02:50:26.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.839+0000 7f97ad23f640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f97a810c110 con 0x7f97a8101930 2024-03-19T02:50:26.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.839+0000 7f97ad23f640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f97a810c410 con 0x7f97a81022f0 2024-03-19T02:50:26.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.839+0000 7f97a6d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97a8100740 0x7f97a81969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:26.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.839+0000 7f97a6d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97a8100740 0x7f97a81969f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57322/0 (socket says 172.21.15.33:57322) 2024-03-19T02:50:26.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.839+0000 7f97a6d9d640 1 -- 172.21.15.33:0/2695741522 learned_addr learned my addr 172.21.15.33:0/2695741522 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:26.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.839+0000 7f97a659c640 1 --2- 172.21.15.33:0/2695741522 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97a8101930 0x7f97a8196f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:26.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.839+0000 7f97a6d9d640 1 -- 172.21.15.33:0/2695741522 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97a81022f0 msgr2=0x7f97a819dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:26.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.839+0000 7f97a6d9d640 1 --2- 172.21.15.33:0/2695741522 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97a81022f0 0x7f97a819dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:26.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.839+0000 7f97a6d9d640 1 -- 172.21.15.33:0/2695741522 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97a8101930 msgr2=0x7f97a8196f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:26.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.839+0000 7f97a6d9d640 1 --2- 172.21.15.33:0/2695741522 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97a8101930 0x7f97a8196f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:26.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.839+0000 7f97a6d9d640 1 -- 172.21.15.33:0/2695741522 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97a819e610 con 0x7f97a8100740 2024-03-19T02:50:26.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.840+0000 7f97a6d9d640 1 --2- 172.21.15.33:0/2695741522 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97a8100740 0x7f97a81969f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f97a0005ab0 tx=0x7f97a00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:26.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.840+0000 7f979ffff640 1 -- 172.21.15.33:0/2695741522 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a0043020 con 0x7f97a8100740 2024-03-19T02:50:26.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.840+0000 7f97ad23f640 1 -- 172.21.15.33:0/2695741522 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97a819e8a0 con 0x7f97a8100740 2024-03-19T02:50:26.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.840+0000 7f979ffff640 1 -- 172.21.15.33:0/2695741522 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f97a00054c0 con 0x7f97a8100740 2024-03-19T02:50:26.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.840+0000 7f979ffff640 1 -- 172.21.15.33:0/2695741522 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a0003d10 con 0x7f97a8100740 2024-03-19T02:50:26.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.840+0000 7f97ad23f640 1 -- 172.21.15.33:0/2695741522 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97a819ee40 con 0x7f97a8100740 2024-03-19T02:50:26.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.841+0000 7f97ad23f640 1 -- 172.21.15.33:0/2695741522 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97a8107b80 con 0x7f97a8100740 2024-03-19T02:50:26.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.842+0000 7f979ffff640 1 -- 172.21.15.33:0/2695741522 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f97a0046020 con 0x7f97a8100740 2024-03-19T02:50:26.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.842+0000 7f979ffff640 1 --2- 172.21.15.33:0/2695741522 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f97940a2800 0x7f97940a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:26.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.843+0000 7f97a659c640 1 --2- 172.21.15.33:0/2695741522 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f97940a2800 0x7f97940a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:26.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.843+0000 7f979ffff640 1 -- 172.21.15.33:0/2695741522 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f97a00f0a90 con 0x7f97a8100740 2024-03-19T02:50:26.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.843+0000 7f97a659c640 1 --2- 172.21.15.33:0/2695741522 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f97940a2800 0x7f97940a4cc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f97a8197f00 tx=0x7f979000a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:26.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:26.847+0000 7f979ffff640 1 -- 172.21.15.33:0/2695741522 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f97a004d020 con 0x7f97a8100740 2024-03-19T02:50:27.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:27.068+0000 7f97ad23f640 1 -- 172.21.15.33:0/2695741522 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97a8062420 con 0x7f97a8100740 2024-03-19T02:50:27.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:27.069+0000 7f979ffff640 1 -- 172.21.15.33:0/2695741522 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3865 (secure 0 0 0) 0x7f97a00be050 con 0x7f97a8100740 2024-03-19T02:50:27.070 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:50:27.070 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-19T02:50:27.070 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 5 2024-03-19T02:50:27.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:27.071+0000 7f97ad23f640 1 -- 172.21.15.33:0/2695741522 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f97940a2800 msgr2=0x7f97940a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:27.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:27.071+0000 7f97ad23f640 1 --2- 172.21.15.33:0/2695741522 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f97940a2800 0x7f97940a4cc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f97a8197f00 tx=0x7f979000a2d0 comp rx=0 tx=0).stop 2024-03-19T02:50:27.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:27.071+0000 7f97ad23f640 1 -- 172.21.15.33:0/2695741522 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97a8100740 msgr2=0x7f97a81969f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:27.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:27.071+0000 7f97ad23f640 1 --2- 172.21.15.33:0/2695741522 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97a8100740 0x7f97a81969f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f97a0005ab0 tx=0x7f97a00042f0 comp rx=0 tx=0).stop 2024-03-19T02:50:27.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:27.071+0000 7f97ad23f640 1 -- 172.21.15.33:0/2695741522 shutdown_connections 2024-03-19T02:50:27.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:27.071+0000 7f97ad23f640 1 --2- 172.21.15.33:0/2695741522 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f97940a2800 0x7f97940a4cc0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:27.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:27.071+0000 7f97ad23f640 1 --2- 172.21.15.33:0/2695741522 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97a81022f0 0x7f97a819dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:27.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:27.071+0000 7f97ad23f640 1 --2- 172.21.15.33:0/2695741522 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97a8101930 0x7f97a8196f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:27.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:27.071+0000 7f97ad23f640 1 --2- 172.21.15.33:0/2695741522 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f97a8100740 0x7f97a81969f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:27.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:27.071+0000 7f97ad23f640 1 -- 172.21.15.33:0/2695741522 >> 172.21.15.33:0/2695741522 conn(0x7f97a80fbcb0 msgr2=0x7f97a8104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:27.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:27.071+0000 7f97ad23f640 1 -- 172.21.15.33:0/2695741522 shutdown_connections 2024-03-19T02:50:27.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:27.071+0000 7f97ad23f640 1 -- 172.21.15.33:0/2695741522 wait complete. 2024-03-19T02:50:27.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:27 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2695741522' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:27.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:27 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2695741522' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:27.600 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:27 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2695741522' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:28.215 INFO:journalctl@ceph.mds.e.smithi096.stdout:Mar 19 02:50:27 smithi096 systemd[1]: Started Ceph mds.e for 2b7a8938-e59a-11ee-95c9-87774f69a715. 2024-03-19T02:50:28.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:28 smithi155 ceph-mon[35804]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:28.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:28 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:28.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:28 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:28.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:28 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:28.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:28 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:28.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:28 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:28.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:28 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:28 smithi033 ceph-mon[31973]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:28 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:28.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:28 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:28.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:28 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:28.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:28 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:28.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:28 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:28.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:28 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:28.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:28 smithi096 ceph-mon[35946]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:28.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:28 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:28.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:28 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:28.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:28 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:28.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:28 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:28.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:28 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:28.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:28 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:29.084 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:50:29.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.198+0000 7feeb2cb2640 1 -- 172.21.15.33:0/2916584357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feeac1022e0 msgr2=0x7feeac108b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:29.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.198+0000 7feeb2cb2640 1 --2- 172.21.15.33:0/2916584357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feeac1022e0 0x7feeac108b70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7feea4009ca0 tx=0x7feea402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:50:29.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.198+0000 7feeb2cb2640 1 -- 172.21.15.33:0/2916584357 shutdown_connections 2024-03-19T02:50:29.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.198+0000 7feeb2cb2640 1 --2- 172.21.15.33:0/2916584357 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feeac101920 0x7feeac101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:29.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.198+0000 7feeb2cb2640 1 --2- 172.21.15.33:0/2916584357 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feeac1022e0 0x7feeac108b70 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:29.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.198+0000 7feeb2cb2640 1 --2- 172.21.15.33:0/2916584357 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feeac100730 0x7feeac100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:29.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.198+0000 7feeb2cb2640 1 -- 172.21.15.33:0/2916584357 >> 172.21.15.33:0/2916584357 conn(0x7feeac0fbca0 msgr2=0x7feeac0fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:29.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.198+0000 7feeb2cb2640 1 -- 172.21.15.33:0/2916584357 shutdown_connections 2024-03-19T02:50:29.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.198+0000 7feeb2cb2640 1 -- 172.21.15.33:0/2916584357 wait complete. 2024-03-19T02:50:29.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.199+0000 7feeb2cb2640 1 Processor -- start 2024-03-19T02:50:29.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.199+0000 7feeb2cb2640 1 -- start start 2024-03-19T02:50:29.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.199+0000 7feeb2cb2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feeac100730 0x7feeac06aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:29.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.199+0000 7feeb2cb2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feeac101920 0x7feeac19c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:29.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.199+0000 7feeb2cb2640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feeac1022e0 0x7feeac19ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:29.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.199+0000 7feeb2cb2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7feeac10c260 con 0x7feeac101920 2024-03-19T02:50:29.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.199+0000 7feeb2cb2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7feeac10c0e0 con 0x7feeac100730 2024-03-19T02:50:29.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.199+0000 7feeb2cb2640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7feeac10c3e0 con 0x7feeac1022e0 2024-03-19T02:50:29.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.199+0000 7feeabfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feeac101920 0x7feeac19c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:29.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.199+0000 7feeabfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feeac101920 0x7feeac19c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57342/0 (socket says 172.21.15.33:57342) 2024-03-19T02:50:29.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.199+0000 7feeb0a4e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feeac100730 0x7feeac06aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:29.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.199+0000 7feeabfff640 1 -- 172.21.15.33:0/3105044426 learned_addr learned my addr 172.21.15.33:0/3105044426 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:29.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.199+0000 7feeb124f640 1 --2- 172.21.15.33:0/3105044426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feeac1022e0 0x7feeac19ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:29.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.199+0000 7feeabfff640 1 -- 172.21.15.33:0/3105044426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feeac1022e0 msgr2=0x7feeac19ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:29.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.199+0000 7feeabfff640 1 --2- 172.21.15.33:0/3105044426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feeac1022e0 0x7feeac19ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:29.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.199+0000 7feeabfff640 1 -- 172.21.15.33:0/3105044426 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feeac100730 msgr2=0x7feeac06aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:29.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.199+0000 7feeabfff640 1 --2- 172.21.15.33:0/3105044426 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feeac100730 0x7feeac06aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:29.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.199+0000 7feeabfff640 1 -- 172.21.15.33:0/3105044426 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeac19f620 con 0x7feeac101920 2024-03-19T02:50:29.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.200+0000 7feeabfff640 1 --2- 172.21.15.33:0/3105044426 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feeac101920 0x7feeac19c6f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fee9800e9f0 tx=0x7fee9800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:29.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.200+0000 7feea9ffb640 1 -- 172.21.15.33:0/3105044426 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee9800cd80 con 0x7feeac101920 2024-03-19T02:50:29.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.200+0000 7feea9ffb640 1 -- 172.21.15.33:0/3105044426 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee980054d0 con 0x7feeac101920 2024-03-19T02:50:29.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.200+0000 7feeb2cb2640 1 -- 172.21.15.33:0/3105044426 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feeac19f910 con 0x7feeac101920 2024-03-19T02:50:29.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.200+0000 7feea9ffb640 1 -- 172.21.15.33:0/3105044426 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee98010490 con 0x7feeac101920 2024-03-19T02:50:29.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.200+0000 7feeb2cb2640 1 -- 172.21.15.33:0/3105044426 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feeac19fe10 con 0x7feeac101920 2024-03-19T02:50:29.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.201+0000 7feeb2cb2640 1 -- 172.21.15.33:0/3105044426 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feeac107ad0 con 0x7feeac101920 2024-03-19T02:50:29.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.202+0000 7feea9ffb640 1 -- 172.21.15.33:0/3105044426 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fee98005020 con 0x7feeac101920 2024-03-19T02:50:29.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.203+0000 7feea9ffb640 1 --2- 172.21.15.33:0/3105044426 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fee9c0a2800 0x7fee9c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:29.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.203+0000 7feeb0a4e640 1 --2- 172.21.15.33:0/3105044426 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fee9c0a2800 0x7fee9c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:29.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.203+0000 7feea9ffb640 1 -- 172.21.15.33:0/3105044426 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fee980ce940 con 0x7feeac101920 2024-03-19T02:50:29.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.203+0000 7feeb0a4e640 1 --2- 172.21.15.33:0/3105044426 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fee9c0a2800 0x7fee9c0a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7feeac101780 tx=0x7feea000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:29.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.207+0000 7feea9ffb640 1 -- 172.21.15.33:0/3105044426 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fee9809be80 con 0x7feeac101920 2024-03-19T02:50:29.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.430+0000 7feeb2cb2640 1 -- 172.21.15.33:0/3105044426 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feeac02d1f0 con 0x7feeac101920 2024-03-19T02:50:29.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.430+0000 7feea9ffb640 1 -- 172.21.15.33:0/3105044426 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4690 (secure 0 0 0) 0x7fee9809b780 con 0x7feeac101920 2024-03-19T02:50:29.431 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:50:29.431 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-19T02:50:29.432 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 6 2024-03-19T02:50:29.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.432+0000 7feeb2cb2640 1 -- 172.21.15.33:0/3105044426 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fee9c0a2800 msgr2=0x7fee9c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:29.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.432+0000 7feeb2cb2640 1 --2- 172.21.15.33:0/3105044426 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fee9c0a2800 0x7fee9c0a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7feeac101780 tx=0x7feea000b390 comp rx=0 tx=0).stop 2024-03-19T02:50:29.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.432+0000 7feeb2cb2640 1 -- 172.21.15.33:0/3105044426 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feeac101920 msgr2=0x7feeac19c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:29.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.432+0000 7feeb2cb2640 1 --2- 172.21.15.33:0/3105044426 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feeac101920 0x7feeac19c6f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fee9800e9f0 tx=0x7fee9800edf0 comp rx=0 tx=0).stop 2024-03-19T02:50:29.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.432+0000 7feeb2cb2640 1 -- 172.21.15.33:0/3105044426 shutdown_connections 2024-03-19T02:50:29.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.432+0000 7feeb2cb2640 1 --2- 172.21.15.33:0/3105044426 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fee9c0a2800 0x7fee9c0a4cc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:29.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.432+0000 7feeb2cb2640 1 --2- 172.21.15.33:0/3105044426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feeac1022e0 0x7feeac19ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:29.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.432+0000 7feeb2cb2640 1 --2- 172.21.15.33:0/3105044426 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7feeac101920 0x7feeac19c6f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:29.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.432+0000 7feeb2cb2640 1 --2- 172.21.15.33:0/3105044426 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7feeac100730 0x7feeac06aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:29.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.432+0000 7feeb2cb2640 1 -- 172.21.15.33:0/3105044426 >> 172.21.15.33:0/3105044426 conn(0x7feeac0fbca0 msgr2=0x7feeac104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:29.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.432+0000 7feeb2cb2640 1 -- 172.21.15.33:0/3105044426 shutdown_connections 2024-03-19T02:50:29.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:29.432+0000 7feeb2cb2640 1 -- 172.21.15.33:0/3105044426 wait complete. 2024-03-19T02:50:29.443 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:29 smithi033 ceph-mon[31973]: Deploying daemon mds.f on smithi155 2024-03-19T02:50:29.443 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:29 smithi033 ceph-mon[31973]: mds.? [v2:172.21.15.96:6834/3551193617,v1:172.21.15.96:6835/3551193617] up:boot 2024-03-19T02:50:29.443 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:29 smithi033 ceph-mon[31973]: fsmap 5 up:standby 2024-03-19T02:50:29.443 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:29 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-19T02:50:29.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:29 smithi155 ceph-mon[35804]: Deploying daemon mds.f on smithi155 2024-03-19T02:50:29.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:29 smithi155 ceph-mon[35804]: mds.? [v2:172.21.15.96:6834/3551193617,v1:172.21.15.96:6835/3551193617] up:boot 2024-03-19T02:50:29.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:29 smithi155 ceph-mon[35804]: fsmap 5 up:standby 2024-03-19T02:50:29.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:29 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-19T02:50:29.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:29 smithi096 ceph-mon[35946]: Deploying daemon mds.f on smithi155 2024-03-19T02:50:29.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:29 smithi096 ceph-mon[35946]: mds.? [v2:172.21.15.96:6834/3551193617,v1:172.21.15.96:6835/3551193617] up:boot 2024-03-19T02:50:29.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:29 smithi096 ceph-mon[35946]: fsmap 5 up:standby 2024-03-19T02:50:29.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:29 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-19T02:50:30.261 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 19 02:50:30 smithi155 systemd[1]: Starting Ceph mds.f for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T02:50:30.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:30 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3105044426' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:30.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:30 smithi155 ceph-mon[35804]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:30.516 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 19 02:50:30 smithi155 podman[66125]: 2024-03-19 02:50:30.165428238 +0000 UTC m=+0.017701468 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:50:30.517 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 19 02:50:30 smithi155 podman[66125]: 2024-03-19 02:50:30.284876293 +0000 UTC m=+0.137149517 container create 3b39aaf94f1ae1f636a573c0b0fb761c1499b25145fbbea728475deae1bf5bb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-f, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-19T02:50:30.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:30 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3105044426' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:30.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:30 smithi033 ceph-mon[31973]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:30.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:30 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3105044426' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:30.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:30 smithi096 ceph-mon[35946]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:31.016 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 19 02:50:30 smithi155 podman[66125]: 2024-03-19 02:50:30.54972742 +0000 UTC m=+0.402000646 container init 3b39aaf94f1ae1f636a573c0b0fb761c1499b25145fbbea728475deae1bf5bb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-f, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-19T02:50:31.017 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 19 02:50:30 smithi155 podman[66125]: 2024-03-19 02:50:30.552412748 +0000 UTC m=+0.404685982 container start 3b39aaf94f1ae1f636a573c0b0fb761c1499b25145fbbea728475deae1bf5bb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-f, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T02:50:31.017 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 19 02:50:30 smithi155 ceph-mds[66144]: -- 172.21.15.155:0/550291771 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ad3639ba40 con 0x55ad3636f000 2024-03-19T02:50:31.017 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 19 02:50:30 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-f[66140]: starting mds.f at 2024-03-19T02:50:31.017 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 19 02:50:30 smithi155 bash[66125]: 3b39aaf94f1ae1f636a573c0b0fb761c1499b25145fbbea728475deae1bf5bb0 2024-03-19T02:50:31.017 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 19 02:50:30 smithi155 systemd[1]: Started Ceph mds.f for 2b7a8938-e59a-11ee-95c9-87774f69a715. 2024-03-19T02:50:31.445 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:50:31.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.559+0000 7f57d0f97640 1 -- 172.21.15.33:0/3234530390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f57cc1019b0 msgr2=0x7f57cc101e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:31.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.559+0000 7f57d0f97640 1 --2- 172.21.15.33:0/3234530390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f57cc1019b0 0x7f57cc101e30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f57b4009900 tx=0x7f57b402bc30 comp rx=0 tx=0).stop 2024-03-19T02:50:31.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.559+0000 7f57d0f97640 1 -- 172.21.15.33:0/3234530390 shutdown_connections 2024-03-19T02:50:31.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.559+0000 7f57d0f97640 1 --2- 172.21.15.33:0/3234530390 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f57cc102370 0x7f57cc108c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:31.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.559+0000 7f57d0f97640 1 --2- 172.21.15.33:0/3234530390 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f57cc1019b0 0x7f57cc101e30 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:31.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.559+0000 7f57d0f97640 1 --2- 172.21.15.33:0/3234530390 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57cc1007c0 0x7f57cc100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:31.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.559+0000 7f57d0f97640 1 -- 172.21.15.33:0/3234530390 >> 172.21.15.33:0/3234530390 conn(0x7f57cc0fbd50 msgr2=0x7f57cc0fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:31.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.559+0000 7f57d0f97640 1 -- 172.21.15.33:0/3234530390 shutdown_connections 2024-03-19T02:50:31.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.559+0000 7f57d0f97640 1 -- 172.21.15.33:0/3234530390 wait complete. 2024-03-19T02:50:31.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.560+0000 7f57d0f97640 1 Processor -- start 2024-03-19T02:50:31.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.560+0000 7f57d0f97640 1 -- start start 2024-03-19T02:50:31.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.560+0000 7f57d0f97640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f57cc1007c0 0x7f57cc104750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:31.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.560+0000 7f57d0f97640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57cc1019b0 0x7f57cc102d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:31.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.560+0000 7f57d0f97640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f57cc102370 0x7f57cc1032d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:31.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.560+0000 7f57d0f97640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f57cc10c260 con 0x7f57cc1007c0 2024-03-19T02:50:31.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.560+0000 7f57d0f97640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f57cc10c0e0 con 0x7f57cc1019b0 2024-03-19T02:50:31.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.560+0000 7f57d0f97640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f57cc10c3e0 con 0x7f57cc102370 2024-03-19T02:50:31.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.560+0000 7f57ca59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f57cc1007c0 0x7f57cc104750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:31.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.560+0000 7f57ca59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f57cc1007c0 0x7f57cc104750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57360/0 (socket says 172.21.15.33:57360) 2024-03-19T02:50:31.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.560+0000 7f57ca59c640 1 -- 172.21.15.33:0/2997653797 learned_addr learned my addr 172.21.15.33:0/2997653797 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:31.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.560+0000 7f57cad9d640 1 --2- 172.21.15.33:0/2997653797 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f57cc102370 0x7f57cc1032d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:31.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.560+0000 7f57c3fff640 1 --2- 172.21.15.33:0/2997653797 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57cc1019b0 0x7f57cc102d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:31.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.561+0000 7f57ca59c640 1 -- 172.21.15.33:0/2997653797 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f57cc102370 msgr2=0x7f57cc1032d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:31.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.561+0000 7f57ca59c640 1 --2- 172.21.15.33:0/2997653797 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f57cc102370 0x7f57cc1032d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:31.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.561+0000 7f57ca59c640 1 -- 172.21.15.33:0/2997653797 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57cc1019b0 msgr2=0x7f57cc102d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:31.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.561+0000 7f57ca59c640 1 --2- 172.21.15.33:0/2997653797 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57cc1019b0 0x7f57cc102d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:31.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.561+0000 7f57ca59c640 1 -- 172.21.15.33:0/2997653797 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57cc103b60 con 0x7f57cc1007c0 2024-03-19T02:50:31.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.561+0000 7f57ca59c640 1 --2- 172.21.15.33:0/2997653797 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f57cc1007c0 0x7f57cc104750 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f57bc00c970 tx=0x7f57bc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:31.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.561+0000 7f57c37fe640 1 -- 172.21.15.33:0/2997653797 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57bc007c10 con 0x7f57cc1007c0 2024-03-19T02:50:31.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.561+0000 7f57c37fe640 1 -- 172.21.15.33:0/2997653797 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57bc007db0 con 0x7f57cc1007c0 2024-03-19T02:50:31.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.561+0000 7f57d0f97640 1 -- 172.21.15.33:0/2997653797 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57cc1a1b20 con 0x7f57cc1007c0 2024-03-19T02:50:31.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.561+0000 7f57c37fe640 1 -- 172.21.15.33:0/2997653797 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57bc00f870 con 0x7f57cc1007c0 2024-03-19T02:50:31.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.561+0000 7f57d0f97640 1 -- 172.21.15.33:0/2997653797 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57cc1a2040 con 0x7f57cc1007c0 2024-03-19T02:50:31.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.562+0000 7f57d0f97640 1 -- 172.21.15.33:0/2997653797 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57cc068a60 con 0x7f57cc1007c0 2024-03-19T02:50:31.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.563+0000 7f57c37fe640 1 -- 172.21.15.33:0/2997653797 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f57bc00fa10 con 0x7f57cc1007c0 2024-03-19T02:50:31.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.564+0000 7f57c37fe640 1 --2- 172.21.15.33:0/2997653797 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f57b80a2800 0x7f57b80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:31.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.564+0000 7f57c3fff640 1 --2- 172.21.15.33:0/2997653797 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f57b80a2800 0x7f57b80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:31.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.564+0000 7f57c37fe640 1 -- 172.21.15.33:0/2997653797 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f57bc0ce900 con 0x7f57cc1007c0 2024-03-19T02:50:31.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.564+0000 7f57c3fff640 1 --2- 172.21.15.33:0/2997653797 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f57b80a2800 0x7f57b80a4cc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f57b4009230 tx=0x7f57b402c740 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:31.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.568+0000 7f57c37fe640 1 -- 172.21.15.33:0/2997653797 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f57bc09bed0 con 0x7f57cc1007c0 2024-03-19T02:50:31.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:31 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:31.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:31 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:31.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:31 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:31.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:31 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:31.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:31 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:31.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:31 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:31.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:31 smithi033 ceph-mon[31973]: mds.? [v2:172.21.15.155:6834/1274854977,v1:172.21.15.155:6835/1274854977] up:boot 2024-03-19T02:50:31.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:31 smithi033 ceph-mon[31973]: fsmap 6 up:standby 2024-03-19T02:50:31.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:31 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-19T02:50:31.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.793+0000 7f57d0f97640 1 -- 172.21.15.33:0/2997653797 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f57cc1a2210 con 0x7f57cc1007c0 2024-03-19T02:50:31.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.794+0000 7f57c37fe640 1 -- 172.21.15.33:0/2997653797 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5518 (secure 0 0 0) 0x7f57bc09b7d0 con 0x7f57cc1007c0 2024-03-19T02:50:31.795 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:50:31.795 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-19T02:50:31.795 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 7 2024-03-19T02:50:31.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.796+0000 7f57d0f97640 1 -- 172.21.15.33:0/2997653797 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f57b80a2800 msgr2=0x7f57b80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:31.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.796+0000 7f57d0f97640 1 --2- 172.21.15.33:0/2997653797 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f57b80a2800 0x7f57b80a4cc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f57b4009230 tx=0x7f57b402c740 comp rx=0 tx=0).stop 2024-03-19T02:50:31.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.796+0000 7f57d0f97640 1 -- 172.21.15.33:0/2997653797 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f57cc1007c0 msgr2=0x7f57cc104750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:31.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.796+0000 7f57d0f97640 1 --2- 172.21.15.33:0/2997653797 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f57cc1007c0 0x7f57cc104750 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f57bc00c970 tx=0x7f57bc00ce30 comp rx=0 tx=0).stop 2024-03-19T02:50:31.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.796+0000 7f57d0f97640 1 -- 172.21.15.33:0/2997653797 shutdown_connections 2024-03-19T02:50:31.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.796+0000 7f57d0f97640 1 --2- 172.21.15.33:0/2997653797 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f57b80a2800 0x7f57b80a4cc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:31.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.796+0000 7f57d0f97640 1 --2- 172.21.15.33:0/2997653797 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f57cc102370 0x7f57cc1032d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:31.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.796+0000 7f57d0f97640 1 --2- 172.21.15.33:0/2997653797 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57cc1019b0 0x7f57cc102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:31.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.796+0000 7f57d0f97640 1 --2- 172.21.15.33:0/2997653797 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f57cc1007c0 0x7f57cc104750 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:31.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.796+0000 7f57d0f97640 1 -- 172.21.15.33:0/2997653797 >> 172.21.15.33:0/2997653797 conn(0x7f57cc0fbd50 msgr2=0x7f57cc0fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:31.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.796+0000 7f57d0f97640 1 -- 172.21.15.33:0/2997653797 shutdown_connections 2024-03-19T02:50:31.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:31.796+0000 7f57d0f97640 1 -- 172.21.15.33:0/2997653797 wait complete. 2024-03-19T02:50:31.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:31 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:31.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:31 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:31.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:31 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:31.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:31 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:31.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:31 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:31.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:31 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:31.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:31 smithi096 ceph-mon[35946]: mds.? [v2:172.21.15.155:6834/1274854977,v1:172.21.15.155:6835/1274854977] up:boot 2024-03-19T02:50:31.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:31 smithi096 ceph-mon[35946]: fsmap 6 up:standby 2024-03-19T02:50:31.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:31 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-19T02:50:32.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:31 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:32.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:31 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:32.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:31 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:32.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:31 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:32.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:31 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:32.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:31 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:32.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:31 smithi155 ceph-mon[35804]: mds.? [v2:172.21.15.155:6834/1274854977,v1:172.21.15.155:6835/1274854977] up:boot 2024-03-19T02:50:32.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:31 smithi155 ceph-mon[35804]: fsmap 6 up:standby 2024-03-19T02:50:32.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:31 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-19T02:50:32.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:32 smithi096 ceph-mon[35946]: Deploying daemon mds.c on smithi155 2024-03-19T02:50:32.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:32 smithi096 ceph-mon[35946]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:32.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:32 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2997653797' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:33.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:32 smithi155 ceph-mon[35804]: Deploying daemon mds.c on smithi155 2024-03-19T02:50:33.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:32 smithi155 ceph-mon[35804]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:33.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:32 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2997653797' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:33.017 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 19 02:50:32 smithi155 systemd[1]: Starting Ceph mds.c for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T02:50:33.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:32 smithi033 ceph-mon[31973]: Deploying daemon mds.c on smithi155 2024-03-19T02:50:33.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:32 smithi033 ceph-mon[31973]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:33.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:32 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2997653797' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:33.369 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 19 02:50:33 smithi155 podman[66431]: 2024-03-19 02:50:32.999321285 +0000 UTC m=+0.017687948 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:50:33.369 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 19 02:50:33 smithi155 podman[66431]: 2024-03-19 02:50:33.114775531 +0000 UTC m=+0.133142194 container create b6f010f3fa3f0f9ef09dc6674cd81105a9d4d58e479bd90fbd77391726e6f423 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T02:50:33.369 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 19 02:50:33 smithi155 podman[66431]: 2024-03-19 02:50:33.342619458 +0000 UTC m=+0.360986121 container init b6f010f3fa3f0f9ef09dc6674cd81105a9d4d58e479bd90fbd77391726e6f423 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-19T02:50:33.369 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 19 02:50:33 smithi155 podman[66431]: 2024-03-19 02:50:33.345160192 +0000 UTC m=+0.363526872 container start b6f010f3fa3f0f9ef09dc6674cd81105a9d4d58e479bd90fbd77391726e6f423 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-c, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-19T02:50:33.766 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 19 02:50:33 smithi155 ceph-mds[66450]: -- 172.21.15.155:0/2521048913 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e4a1099a40 con 0x55e4a106cc00 2024-03-19T02:50:33.766 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 19 02:50:33 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-c[66446]: starting mds.c at 2024-03-19T02:50:33.766 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 19 02:50:33 smithi155 bash[66431]: b6f010f3fa3f0f9ef09dc6674cd81105a9d4d58e479bd90fbd77391726e6f423 2024-03-19T02:50:33.766 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 19 02:50:33 smithi155 systemd[1]: Started Ceph mds.c for 2b7a8938-e59a-11ee-95c9-87774f69a715. 2024-03-19T02:50:33.809 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:50:33.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.925+0000 7f003b89a640 1 -- 172.21.15.33:0/1254926183 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00341022f0 msgr2=0x7f0034108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:33.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.925+0000 7f003b89a640 1 --2- 172.21.15.33:0/1254926183 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00341022f0 0x7f0034108b80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f002c009be0 tx=0x7f002c02bef0 comp rx=0 tx=0).stop 2024-03-19T02:50:33.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.925+0000 7f003b89a640 1 -- 172.21.15.33:0/1254926183 shutdown_connections 2024-03-19T02:50:33.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.925+0000 7f003b89a640 1 --2- 172.21.15.33:0/1254926183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0034101930 0x7f0034101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:33.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.925+0000 7f003b89a640 1 --2- 172.21.15.33:0/1254926183 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00341022f0 0x7f0034108b80 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:33.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.925+0000 7f003b89a640 1 --2- 172.21.15.33:0/1254926183 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0034100740 0x7f0034100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:33.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.925+0000 7f003b89a640 1 -- 172.21.15.33:0/1254926183 >> 172.21.15.33:0/1254926183 conn(0x7f00340fbcb0 msgr2=0x7f00340fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:33.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.925+0000 7f003b89a640 1 -- 172.21.15.33:0/1254926183 shutdown_connections 2024-03-19T02:50:33.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.925+0000 7f003b89a640 1 -- 172.21.15.33:0/1254926183 wait complete. 2024-03-19T02:50:33.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.925+0000 7f003b89a640 1 Processor -- start 2024-03-19T02:50:33.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.926+0000 7f003b89a640 1 -- start start 2024-03-19T02:50:33.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.926+0000 7f003b89a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0034100740 0x7f00340694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:33.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.926+0000 7f003b89a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0034101930 0x7f003419c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:33.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.926+0000 7f003b89a640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f00341022f0 0x7f003419edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:33.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.926+0000 7f003b89a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f003410c270 con 0x7f0034101930 2024-03-19T02:50:33.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.926+0000 7f003b89a640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f003410c0f0 con 0x7f0034100740 2024-03-19T02:50:33.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.926+0000 7f003b89a640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f003410c3f0 con 0x7f00341022f0 2024-03-19T02:50:33.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.926+0000 7f0038e35640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0034101930 0x7f003419c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:33.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.926+0000 7f0038e35640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0034101930 0x7f003419c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49276/0 (socket says 172.21.15.33:49276) 2024-03-19T02:50:33.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.926+0000 7f0038e35640 1 -- 172.21.15.33:0/125996354 learned_addr learned my addr 172.21.15.33:0/125996354 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:33.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.926+0000 7f0039636640 1 --2- 172.21.15.33:0/125996354 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0034100740 0x7f00340694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:33.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.926+0000 7f0038e35640 1 -- 172.21.15.33:0/125996354 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f00341022f0 msgr2=0x7f003419edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:33.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.926+0000 7f0039e37640 1 --2- 172.21.15.33:0/125996354 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f00341022f0 0x7f003419edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:33.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.926+0000 7f0038e35640 1 --2- 172.21.15.33:0/125996354 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f00341022f0 0x7f003419edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:33.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.926+0000 7f0038e35640 1 -- 172.21.15.33:0/125996354 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0034100740 msgr2=0x7f00340694f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:33.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.926+0000 7f0038e35640 1 --2- 172.21.15.33:0/125996354 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0034100740 0x7f00340694f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:33.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.926+0000 7f0038e35640 1 -- 172.21.15.33:0/125996354 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f003419f6d0 con 0x7f0034101930 2024-03-19T02:50:33.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.926+0000 7f0039636640 1 --2- 172.21.15.33:0/125996354 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0034100740 0x7f00340694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:33.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.926+0000 7f0039e37640 1 --2- 172.21.15.33:0/125996354 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f00341022f0 0x7f003419edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:50:33.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.926+0000 7f0038e35640 1 --2- 172.21.15.33:0/125996354 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0034101930 0x7f003419c7a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f002400e9f0 tx=0x7f002400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:33.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.927+0000 7f002a7fc640 1 -- 172.21.15.33:0/125996354 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002400cd80 con 0x7f0034101930 2024-03-19T02:50:33.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.927+0000 7f003b89a640 1 -- 172.21.15.33:0/125996354 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f003419f960 con 0x7f0034101930 2024-03-19T02:50:33.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.927+0000 7f002a7fc640 1 -- 172.21.15.33:0/125996354 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f00240054d0 con 0x7f0034101930 2024-03-19T02:50:33.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.927+0000 7f002a7fc640 1 -- 172.21.15.33:0/125996354 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0024010490 con 0x7f0034101930 2024-03-19T02:50:33.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.927+0000 7f003b89a640 1 -- 172.21.15.33:0/125996354 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f003419fe60 con 0x7f0034101930 2024-03-19T02:50:33.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.928+0000 7f003b89a640 1 -- 172.21.15.33:0/125996354 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0034107a80 con 0x7f0034101930 2024-03-19T02:50:33.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.929+0000 7f002a7fc640 1 -- 172.21.15.33:0/125996354 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f0024005020 con 0x7f0034101930 2024-03-19T02:50:33.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.929+0000 7f002a7fc640 1 --2- 172.21.15.33:0/125996354 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f00200a2800 0x7f00200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:33.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.930+0000 7f0039636640 1 --2- 172.21.15.33:0/125996354 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f00200a2800 0x7f00200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:33.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.930+0000 7f002a7fc640 1 -- 172.21.15.33:0/125996354 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f00240ce9f0 con 0x7f0034101930 2024-03-19T02:50:33.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.930+0000 7f0039636640 1 --2- 172.21.15.33:0/125996354 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f00200a2800 0x7f00200a4cc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f00300096c0 tx=0x7f0030009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:33.935 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:33.934+0000 7f002a7fc640 1 -- 172.21.15.33:0/125996354 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f002409bf30 con 0x7f0034101930 2024-03-19T02:50:34.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:34.159+0000 7f003b89a640 1 -- 172.21.15.33:0/125996354 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f003402d3f0 con 0x7f0034101930 2024-03-19T02:50:34.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:34.160+0000 7f002a7fc640 1 -- 172.21.15.33:0/125996354 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6346 (secure 0 0 0) 0x7f002409b830 con 0x7f0034101930 2024-03-19T02:50:34.161 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:50:34.161 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24436,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-19T02:50:34.161 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 8 2024-03-19T02:50:34.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:34.162+0000 7f003b89a640 1 -- 172.21.15.33:0/125996354 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f00200a2800 msgr2=0x7f00200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:34.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:34.162+0000 7f003b89a640 1 --2- 172.21.15.33:0/125996354 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f00200a2800 0x7f00200a4cc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f00300096c0 tx=0x7f0030009260 comp rx=0 tx=0).stop 2024-03-19T02:50:34.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:34.162+0000 7f003b89a640 1 -- 172.21.15.33:0/125996354 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0034101930 msgr2=0x7f003419c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:34.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:34.162+0000 7f003b89a640 1 --2- 172.21.15.33:0/125996354 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0034101930 0x7f003419c7a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f002400e9f0 tx=0x7f002400edf0 comp rx=0 tx=0).stop 2024-03-19T02:50:34.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:34.162+0000 7f003b89a640 1 -- 172.21.15.33:0/125996354 shutdown_connections 2024-03-19T02:50:34.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:34.162+0000 7f003b89a640 1 --2- 172.21.15.33:0/125996354 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f00200a2800 0x7f00200a4cc0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:34.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:34.162+0000 7f003b89a640 1 --2- 172.21.15.33:0/125996354 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f00341022f0 0x7f003419edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:34.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:34.162+0000 7f003b89a640 1 --2- 172.21.15.33:0/125996354 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0034101930 0x7f003419c7a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:34.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:34.162+0000 7f003b89a640 1 --2- 172.21.15.33:0/125996354 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0034100740 0x7f00340694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:34.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:34.162+0000 7f003b89a640 1 -- 172.21.15.33:0/125996354 >> 172.21.15.33:0/125996354 conn(0x7f00340fbcb0 msgr2=0x7f0034104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:34.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:34.162+0000 7f003b89a640 1 -- 172.21.15.33:0/125996354 shutdown_connections 2024-03-19T02:50:34.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:34.162+0000 7f003b89a640 1 -- 172.21.15.33:0/125996354 wait complete. 2024-03-19T02:50:34.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:34 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:34.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:34 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:34.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:34 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:34.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:34 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:34.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:34 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:34.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:34 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:34.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:34 smithi096 ceph-mon[35946]: Deploying daemon mds.h on smithi096 2024-03-19T02:50:34.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:34 smithi096 ceph-mon[35946]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:34.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:34 smithi096 ceph-mon[35946]: mds.? [v2:172.21.15.155:6836/1429349330,v1:172.21.15.155:6837/1429349330] up:boot 2024-03-19T02:50:34.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:34 smithi096 ceph-mon[35946]: fsmap 7 up:standby 2024-03-19T02:50:34.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:34 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-19T02:50:34.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:34 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/125996354' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:34.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:34 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:34.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:34 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:34.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:34 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:34.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:34 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:34.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:34 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:34.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:34 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:34.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:34 smithi155 ceph-mon[35804]: Deploying daemon mds.h on smithi096 2024-03-19T02:50:34.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:34 smithi155 ceph-mon[35804]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:34.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:34 smithi155 ceph-mon[35804]: mds.? [v2:172.21.15.155:6836/1429349330,v1:172.21.15.155:6837/1429349330] up:boot 2024-03-19T02:50:34.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:34 smithi155 ceph-mon[35804]: fsmap 7 up:standby 2024-03-19T02:50:34.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:34 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-19T02:50:34.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:34 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/125996354' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:34.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:34 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:34.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:34 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:34.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:34 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:34.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:34 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:34.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:34 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:34.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:34 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:34.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:34 smithi033 ceph-mon[31973]: Deploying daemon mds.h on smithi096 2024-03-19T02:50:34.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:34 smithi033 ceph-mon[31973]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:34.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:34 smithi033 ceph-mon[31973]: mds.? [v2:172.21.15.155:6836/1429349330,v1:172.21.15.155:6837/1429349330] up:boot 2024-03-19T02:50:34.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:34 smithi033 ceph-mon[31973]: fsmap 7 up:standby 2024-03-19T02:50:34.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:34 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-19T02:50:34.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:34 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/125996354' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:35.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:35 smithi096 ceph-mon[35946]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:36.175 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:50:36.206 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:35 smithi033 ceph-mon[31973]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:36.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:35 smithi155 ceph-mon[35804]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:36.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.292+0000 7f101c019640 1 -- 172.21.15.33:0/3438813395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1014102a30 msgr2=0x7f1014102eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:36.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.292+0000 7f101c019640 1 --2- 172.21.15.33:0/3438813395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1014102a30 0x7f1014102eb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1004009900 tx=0x7f100402bc30 comp rx=0 tx=0).stop 2024-03-19T02:50:36.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.292+0000 7f101c019640 1 -- 172.21.15.33:0/3438813395 shutdown_connections 2024-03-19T02:50:36.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.292+0000 7f101c019640 1 --2- 172.21.15.33:0/3438813395 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10141033f0 0x7f1014109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:36.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.292+0000 7f101c019640 1 --2- 172.21.15.33:0/3438813395 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1014102a30 0x7f1014102eb0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:36.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.292+0000 7f101c019640 1 --2- 172.21.15.33:0/3438813395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1014101840 0x7f1014101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:36.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.292+0000 7f101c019640 1 -- 172.21.15.33:0/3438813395 >> 172.21.15.33:0/3438813395 conn(0x7f10140fcdd0 msgr2=0x7f10140ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:36.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.292+0000 7f101c019640 1 -- 172.21.15.33:0/3438813395 shutdown_connections 2024-03-19T02:50:36.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.292+0000 7f101c019640 1 -- 172.21.15.33:0/3438813395 wait complete. 2024-03-19T02:50:36.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.293+0000 7f101c019640 1 Processor -- start 2024-03-19T02:50:36.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.293+0000 7f101c019640 1 -- start start 2024-03-19T02:50:36.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.293+0000 7f101c019640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1014101840 0x7f1014199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:36.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.293+0000 7f101c019640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1014102a30 0x7f101419a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:36.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.293+0000 7f101c019640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10141033f0 0x7f10141a1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:36.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.293+0000 7f101c019640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f101410d370 con 0x7f1014102a30 2024-03-19T02:50:36.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.293+0000 7f101c019640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f101410d1f0 con 0x7f1014101840 2024-03-19T02:50:36.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.293+0000 7f101c019640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f101410d4f0 con 0x7f10141033f0 2024-03-19T02:50:36.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.293+0000 7f10195b4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1014102a30 0x7f101419a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:36.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.293+0000 7f10195b4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1014102a30 0x7f101419a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49296/0 (socket says 172.21.15.33:49296) 2024-03-19T02:50:36.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.293+0000 7f1019db5640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1014101840 0x7f1014199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:36.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.293+0000 7f10195b4640 1 -- 172.21.15.33:0/3889572396 learned_addr learned my addr 172.21.15.33:0/3889572396 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:36.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.294+0000 7f101a5b6640 1 --2- 172.21.15.33:0/3889572396 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10141033f0 0x7f10141a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:36.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.294+0000 7f10195b4640 1 -- 172.21.15.33:0/3889572396 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10141033f0 msgr2=0x7f10141a1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:36.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.294+0000 7f10195b4640 1 --2- 172.21.15.33:0/3889572396 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10141033f0 0x7f10141a1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:36.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.294+0000 7f10195b4640 1 -- 172.21.15.33:0/3889572396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1014101840 msgr2=0x7f1014199cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:36.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.294+0000 7f10195b4640 1 --2- 172.21.15.33:0/3889572396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1014101840 0x7f1014199cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:36.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.294+0000 7f10195b4640 1 -- 172.21.15.33:0/3889572396 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10141a18e0 con 0x7f1014102a30 2024-03-19T02:50:36.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.294+0000 7f101a5b6640 1 --2- 172.21.15.33:0/3889572396 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10141033f0 0x7f10141a1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:50:36.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.294+0000 7f10195b4640 1 --2- 172.21.15.33:0/3889572396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1014102a30 0x7f101419a230 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1004006010 tx=0x7f1004004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:36.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.294+0000 7f100affd640 1 -- 172.21.15.33:0/3889572396 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1004043020 con 0x7f1014102a30 2024-03-19T02:50:36.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.294+0000 7f100affd640 1 -- 172.21.15.33:0/3889572396 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1004004440 con 0x7f1014102a30 2024-03-19T02:50:36.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.294+0000 7f101c019640 1 -- 172.21.15.33:0/3889572396 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10141a1b70 con 0x7f1014102a30 2024-03-19T02:50:36.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.294+0000 7f100affd640 1 -- 172.21.15.33:0/3889572396 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1004003bf0 con 0x7f1014102a30 2024-03-19T02:50:36.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.294+0000 7f101c019640 1 -- 172.21.15.33:0/3889572396 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10141a20c0 con 0x7f1014102a30 2024-03-19T02:50:36.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.295+0000 7f101c019640 1 -- 172.21.15.33:0/3889572396 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10141a22f0 con 0x7f1014102a30 2024-03-19T02:50:36.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.296+0000 7f100affd640 1 -- 172.21.15.33:0/3889572396 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f1004046020 con 0x7f1014102a30 2024-03-19T02:50:36.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.297+0000 7f100affd640 1 --2- 172.21.15.33:0/3889572396 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f10000a2800 0x7f10000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:36.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.297+0000 7f1019db5640 1 --2- 172.21.15.33:0/3889572396 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f10000a2800 0x7f10000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:36.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.297+0000 7f100affd640 1 -- 172.21.15.33:0/3889572396 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f10040f1010 con 0x7f1014102a30 2024-03-19T02:50:36.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.297+0000 7f1019db5640 1 --2- 172.21.15.33:0/3889572396 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f10000a2800 0x7f10000a4cc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f10140ff4d0 tx=0x7f1010005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:36.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.301+0000 7f100affd640 1 -- 172.21.15.33:0/3889572396 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f100404d020 con 0x7f1014102a30 2024-03-19T02:50:36.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.545+0000 7f101c019640 1 -- 172.21.15.33:0/3889572396 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f10141a24c0 con 0x7f1014102a30 2024-03-19T02:50:36.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.545+0000 7f100affd640 1 -- 172.21.15.33:0/3889572396 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6346 (secure 0 0 0) 0x7f10040be4b0 con 0x7f1014102a30 2024-03-19T02:50:36.547 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:50:36.547 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24436,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-19T02:50:36.547 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 8 2024-03-19T02:50:36.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.547+0000 7f101c019640 1 -- 172.21.15.33:0/3889572396 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f10000a2800 msgr2=0x7f10000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:36.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.547+0000 7f101c019640 1 --2- 172.21.15.33:0/3889572396 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f10000a2800 0x7f10000a4cc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f10140ff4d0 tx=0x7f1010005f90 comp rx=0 tx=0).stop 2024-03-19T02:50:36.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.547+0000 7f101c019640 1 -- 172.21.15.33:0/3889572396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1014102a30 msgr2=0x7f101419a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:36.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.547+0000 7f101c019640 1 --2- 172.21.15.33:0/3889572396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1014102a30 0x7f101419a230 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1004006010 tx=0x7f1004004040 comp rx=0 tx=0).stop 2024-03-19T02:50:36.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.547+0000 7f101c019640 1 -- 172.21.15.33:0/3889572396 shutdown_connections 2024-03-19T02:50:36.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.547+0000 7f101c019640 1 --2- 172.21.15.33:0/3889572396 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f10000a2800 0x7f10000a4cc0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:36.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.547+0000 7f101c019640 1 --2- 172.21.15.33:0/3889572396 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10141033f0 0x7f10141a1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:36.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.547+0000 7f101c019640 1 --2- 172.21.15.33:0/3889572396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1014102a30 0x7f101419a230 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:36.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.547+0000 7f101c019640 1 --2- 172.21.15.33:0/3889572396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1014101840 0x7f1014199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:36.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.547+0000 7f101c019640 1 -- 172.21.15.33:0/3889572396 >> 172.21.15.33:0/3889572396 conn(0x7f10140fcdd0 msgr2=0x7f1014073a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:36.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.548+0000 7f101c019640 1 -- 172.21.15.33:0/3889572396 shutdown_connections 2024-03-19T02:50:36.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:36.548+0000 7f101c019640 1 -- 172.21.15.33:0/3889572396 wait complete. 2024-03-19T02:50:36.715 INFO:journalctl@ceph.mds.h.smithi096.stdout:Mar 19 02:50:36 smithi096 systemd[1]: Started Ceph mds.h for 2b7a8938-e59a-11ee-95c9-87774f69a715. 2024-03-19T02:50:37.681 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:37 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:37.681 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:37 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:37.681 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:37 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:37.681 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:37 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:37.681 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:37 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:37.681 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:37 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:37.681 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:37 smithi155 ceph-mon[35804]: Deploying daemon mds.i on smithi155 2024-03-19T02:50:37.681 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:37 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3889572396' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:37.682 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:37 smithi155 ceph-mon[35804]: mds.? [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] up:boot 2024-03-19T02:50:37.682 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:37 smithi155 ceph-mon[35804]: fsmap 8 up:standby 2024-03-19T02:50:37.682 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:37 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-19T02:50:37.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:37 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:37.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:37 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:37.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:37 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:37.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:37 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:37.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:37 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:37.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:37 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:37.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:37 smithi096 ceph-mon[35946]: Deploying daemon mds.i on smithi155 2024-03-19T02:50:37.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:37 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3889572396' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:37.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:37 smithi096 ceph-mon[35946]: mds.? [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] up:boot 2024-03-19T02:50:37.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:37 smithi096 ceph-mon[35946]: fsmap 8 up:standby 2024-03-19T02:50:37.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:37 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-19T02:50:37.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:37 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:37.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:37 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:37.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:37 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:37.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:37 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:37.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:37 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:37.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:37 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:37.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:37 smithi033 ceph-mon[31973]: Deploying daemon mds.i on smithi155 2024-03-19T02:50:37.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:37 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3889572396' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:37.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:37 smithi033 ceph-mon[31973]: mds.? [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] up:boot 2024-03-19T02:50:37.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:37 smithi033 ceph-mon[31973]: fsmap 8 up:standby 2024-03-19T02:50:37.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:37 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-19T02:50:38.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:38 smithi155 ceph-mon[35804]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:38.561 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:50:38.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.675+0000 7f1e4ab10640 1 -- 172.21.15.33:0/1884493975 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1e44101840 msgr2=0x7f1e44101c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.675+0000 7f1e4ab10640 1 --2- 172.21.15.33:0/1884493975 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1e44101840 0x7f1e44101c60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c009ca0 tx=0x7f1e3c02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.675+0000 7f1e4ab10640 1 -- 172.21.15.33:0/1884493975 shutdown_connections 2024-03-19T02:50:38.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.675+0000 7f1e4ab10640 1 --2- 172.21.15.33:0/1884493975 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e44102a30 0x7f1e44102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.675+0000 7f1e4ab10640 1 --2- 172.21.15.33:0/1884493975 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e441033f0 0x7f1e44109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.675+0000 7f1e4ab10640 1 --2- 172.21.15.33:0/1884493975 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1e44101840 0x7f1e44101c60 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.675+0000 7f1e4ab10640 1 -- 172.21.15.33:0/1884493975 >> 172.21.15.33:0/1884493975 conn(0x7f1e440fcdd0 msgr2=0x7f1e440ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:38.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.675+0000 7f1e4ab10640 1 -- 172.21.15.33:0/1884493975 shutdown_connections 2024-03-19T02:50:38.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.675+0000 7f1e4ab10640 1 -- 172.21.15.33:0/1884493975 wait complete. 2024-03-19T02:50:38.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.676+0000 7f1e4ab10640 1 Processor -- start 2024-03-19T02:50:38.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.676+0000 7f1e4ab10640 1 -- start start 2024-03-19T02:50:38.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.676+0000 7f1e4ab10640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1e44101840 0x7f1e44197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.676+0000 7f1e4ab10640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e44102a30 0x7f1e44197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.676+0000 7f1e4ab10640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e441033f0 0x7f1e4419efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.676+0000 7f1e4ab10640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1e4410d360 con 0x7f1e44101840 2024-03-19T02:50:38.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.676+0000 7f1e4ab10640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1e4410d1e0 con 0x7f1e441033f0 2024-03-19T02:50:38.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.676+0000 7f1e4ab10640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1e4410d4e0 con 0x7f1e44102a30 2024-03-19T02:50:38.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.676+0000 7f1e488ac640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1e44101840 0x7f1e44197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.676+0000 7f1e488ac640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1e44101840 0x7f1e44197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49320/0 (socket says 172.21.15.33:49320) 2024-03-19T02:50:38.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.676+0000 7f1e488ac640 1 -- 172.21.15.33:0/2881794752 learned_addr learned my addr 172.21.15.33:0/2881794752 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:38.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.676+0000 7f1e43fff640 1 --2- 172.21.15.33:0/2881794752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e44102a30 0x7f1e44197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.676+0000 7f1e488ac640 1 -- 172.21.15.33:0/2881794752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e44102a30 msgr2=0x7f1e44197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.676+0000 7f1e488ac640 1 --2- 172.21.15.33:0/2881794752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e44102a30 0x7f1e44197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.676+0000 7f1e490ad640 1 --2- 172.21.15.33:0/2881794752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e441033f0 0x7f1e4419efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.676+0000 7f1e488ac640 1 -- 172.21.15.33:0/2881794752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e441033f0 msgr2=0x7f1e4419efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.676+0000 7f1e488ac640 1 --2- 172.21.15.33:0/2881794752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e441033f0 0x7f1e4419efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.676+0000 7f1e488ac640 1 -- 172.21.15.33:0/2881794752 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e4419f650 con 0x7f1e44101840 2024-03-19T02:50:38.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.677+0000 7f1e488ac640 1 --2- 172.21.15.33:0/2881794752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1e44101840 0x7f1e44197a30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c032380 tx=0x7f1e3c032820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:38.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.677+0000 7f1e41ffb640 1 -- 172.21.15.33:0/2881794752 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c041020 con 0x7f1e44101840 2024-03-19T02:50:38.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.677+0000 7f1e41ffb640 1 -- 172.21.15.33:0/2881794752 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1e3c003e20 con 0x7f1e44101840 2024-03-19T02:50:38.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.677+0000 7f1e4ab10640 1 -- 172.21.15.33:0/2881794752 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e4419f8e0 con 0x7f1e44101840 2024-03-19T02:50:38.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.677+0000 7f1e41ffb640 1 -- 172.21.15.33:0/2881794752 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c039040 con 0x7f1e44101840 2024-03-19T02:50:38.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.677+0000 7f1e4ab10640 1 -- 172.21.15.33:0/2881794752 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e4419fdd0 con 0x7f1e44101840 2024-03-19T02:50:38.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.678+0000 7f1e4ab10640 1 -- 172.21.15.33:0/2881794752 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e44108b60 con 0x7f1e44101840 2024-03-19T02:50:38.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.680+0000 7f1e41ffb640 1 -- 172.21.15.33:0/2881794752 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f1e3c044020 con 0x7f1e44101840 2024-03-19T02:50:38.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.680+0000 7f1e41ffb640 1 --2- 172.21.15.33:0/2881794752 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f1e2c0a2800 0x7f1e2c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:38.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.680+0000 7f1e43fff640 1 --2- 172.21.15.33:0/2881794752 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f1e2c0a2800 0x7f1e2c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:38.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.680+0000 7f1e41ffb640 1 -- 172.21.15.33:0/2881794752 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f1e3c0f0b50 con 0x7f1e44101840 2024-03-19T02:50:38.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.681+0000 7f1e43fff640 1 --2- 172.21.15.33:0/2881794752 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f1e2c0a2800 0x7f1e2c0a4cc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1e44198f40 tx=0x7f1e38005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:38.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.684+0000 7f1e41ffb640 1 -- 172.21.15.33:0/2881794752 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1e3c04b020 con 0x7f1e44101840 2024-03-19T02:50:38.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:38 smithi096 ceph-mon[35946]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:38.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:38 smithi033 ceph-mon[31973]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:38.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.907+0000 7f1e4ab10640 1 -- 172.21.15.33:0/2881794752 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e4402d1f0 con 0x7f1e44101840 2024-03-19T02:50:38.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.908+0000 7f1e41ffb640 1 -- 172.21.15.33:0/2881794752 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7168 (secure 0 0 0) 0x7f1e3c0bdff0 con 0x7f1e44101840 2024-03-19T02:50:38.909 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:50:38.909 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24436,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-19T02:50:38.910 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 9 2024-03-19T02:50:38.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.910+0000 7f1e4ab10640 1 -- 172.21.15.33:0/2881794752 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f1e2c0a2800 msgr2=0x7f1e2c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.910+0000 7f1e4ab10640 1 --2- 172.21.15.33:0/2881794752 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f1e2c0a2800 0x7f1e2c0a4cc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1e44198f40 tx=0x7f1e38005f90 comp rx=0 tx=0).stop 2024-03-19T02:50:38.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.910+0000 7f1e4ab10640 1 -- 172.21.15.33:0/2881794752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1e44101840 msgr2=0x7f1e44197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:38.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.910+0000 7f1e4ab10640 1 --2- 172.21.15.33:0/2881794752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1e44101840 0x7f1e44197a30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c032380 tx=0x7f1e3c032820 comp rx=0 tx=0).stop 2024-03-19T02:50:38.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.910+0000 7f1e4ab10640 1 -- 172.21.15.33:0/2881794752 shutdown_connections 2024-03-19T02:50:38.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.910+0000 7f1e4ab10640 1 --2- 172.21.15.33:0/2881794752 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f1e2c0a2800 0x7f1e2c0a4cc0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.910+0000 7f1e4ab10640 1 --2- 172.21.15.33:0/2881794752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e44102a30 0x7f1e44197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.910+0000 7f1e4ab10640 1 --2- 172.21.15.33:0/2881794752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e441033f0 0x7f1e4419efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.910+0000 7f1e4ab10640 1 --2- 172.21.15.33:0/2881794752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1e44101840 0x7f1e44197a30 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:38.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.910+0000 7f1e4ab10640 1 -- 172.21.15.33:0/2881794752 >> 172.21.15.33:0/2881794752 conn(0x7f1e440fcdd0 msgr2=0x7f1e44105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:38.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.910+0000 7f1e4ab10640 1 -- 172.21.15.33:0/2881794752 shutdown_connections 2024-03-19T02:50:38.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:38.910+0000 7f1e4ab10640 1 -- 172.21.15.33:0/2881794752 wait complete. 2024-03-19T02:50:39.266 INFO:journalctl@ceph.mds.i.smithi155.stdout:Mar 19 02:50:39 smithi155 systemd[1]: Started Ceph mds.i for 2b7a8938-e59a-11ee-95c9-87774f69a715. 2024-03-19T02:50:39.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:39 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2881794752' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:39.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:39 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:39.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:39 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:39.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:39 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:39.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:39 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:39.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:39 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:39.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:39 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:39.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:39 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2881794752' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:39.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:39 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:39.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:39 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:39.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:39 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:39.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:39 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:39.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:39 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:39.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:39 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:39.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:39 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2881794752' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:39.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:39 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:39.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:39 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:39.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:39 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:39.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:39 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:39.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:39 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:39.777 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:39 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:40.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:40 smithi096 ceph-mon[35946]: Deploying daemon mds.g on smithi033 2024-03-19T02:50:40.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:40 smithi096 ceph-mon[35946]: mds.? [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] up:boot 2024-03-19T02:50:40.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:40 smithi096 ceph-mon[35946]: fsmap 9 up:standby 2024-03-19T02:50:40.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:40 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-19T02:50:40.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:40 smithi096 ceph-mon[35946]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:40.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:40 smithi155 ceph-mon[35804]: Deploying daemon mds.g on smithi033 2024-03-19T02:50:40.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:40 smithi155 ceph-mon[35804]: mds.? [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] up:boot 2024-03-19T02:50:40.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:40 smithi155 ceph-mon[35804]: fsmap 9 up:standby 2024-03-19T02:50:40.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-19T02:50:40.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:40 smithi155 ceph-mon[35804]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:40.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:40 smithi033 ceph-mon[31973]: Deploying daemon mds.g on smithi033 2024-03-19T02:50:40.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:40 smithi033 ceph-mon[31973]: mds.? [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] up:boot 2024-03-19T02:50:40.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:40 smithi033 ceph-mon[31973]: fsmap 9 up:standby 2024-03-19T02:50:40.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-19T02:50:40.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:40 smithi033 ceph-mon[31973]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:40.922 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:50:41.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.151+0000 7f620ec44640 1 -- 172.21.15.33:0/1278919453 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6208100740 msgr2=0x7f6208100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:41.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.151+0000 7f620ec44640 1 --2- 172.21.15.33:0/1278919453 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6208100740 0x7f6208100b60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6204009900 tx=0x7f620402bc30 comp rx=0 tx=0).stop 2024-03-19T02:50:41.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.151+0000 7f620ec44640 1 -- 172.21.15.33:0/1278919453 shutdown_connections 2024-03-19T02:50:41.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.151+0000 7f620ec44640 1 --2- 172.21.15.33:0/1278919453 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6208101930 0x7f6208101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:41.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.151+0000 7f620ec44640 1 --2- 172.21.15.33:0/1278919453 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62081022f0 0x7f6208108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:41.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.151+0000 7f620ec44640 1 --2- 172.21.15.33:0/1278919453 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6208100740 0x7f6208100b60 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:41.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.151+0000 7f620ec44640 1 -- 172.21.15.33:0/1278919453 >> 172.21.15.33:0/1278919453 conn(0x7f62080fbcb0 msgr2=0x7f62080fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:41.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.151+0000 7f620ec44640 1 -- 172.21.15.33:0/1278919453 shutdown_connections 2024-03-19T02:50:41.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.151+0000 7f620ec44640 1 -- 172.21.15.33:0/1278919453 wait complete. 2024-03-19T02:50:41.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.151+0000 7f620ec44640 1 Processor -- start 2024-03-19T02:50:41.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.151+0000 7f620ec44640 1 -- start start 2024-03-19T02:50:41.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.151+0000 7f620ec44640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6208100740 0x7f62081046d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:41.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.151+0000 7f620ec44640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6208101930 0x7f6208102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:41.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.151+0000 7f620ec44640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62081022f0 0x7f6208103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:41.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.151+0000 7f620ec44640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f620810c240 con 0x7f6208101930 2024-03-19T02:50:41.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.152+0000 7f620ec44640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f620810c0c0 con 0x7f6208100740 2024-03-19T02:50:41.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.152+0000 7f620ec44640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f620810c3c0 con 0x7f62081022f0 2024-03-19T02:50:41.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.152+0000 7f61fffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6208101930 0x7f6208102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:41.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.152+0000 7f61fffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6208101930 0x7f6208102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49342/0 (socket says 172.21.15.33:49342) 2024-03-19T02:50:41.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.152+0000 7f61fffff640 1 -- 172.21.15.33:0/3794788599 learned_addr learned my addr 172.21.15.33:0/3794788599 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:41.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.152+0000 7f620d1e1640 1 --2- 172.21.15.33:0/3794788599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62081022f0 0x7f6208103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:41.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.152+0000 7f620c9e0640 1 --2- 172.21.15.33:0/3794788599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6208100740 0x7f62081046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:41.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.152+0000 7f61fffff640 1 -- 172.21.15.33:0/3794788599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62081022f0 msgr2=0x7f6208103250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:41.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.152+0000 7f61fffff640 1 --2- 172.21.15.33:0/3794788599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62081022f0 0x7f6208103250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:41.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.152+0000 7f61fffff640 1 -- 172.21.15.33:0/3794788599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6208100740 msgr2=0x7f62081046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:41.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.152+0000 7f61fffff640 1 --2- 172.21.15.33:0/3794788599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6208100740 0x7f62081046d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:41.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.152+0000 7f61fffff640 1 -- 172.21.15.33:0/3794788599 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6208103ae0 con 0x7f6208101930 2024-03-19T02:50:41.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.152+0000 7f620c9e0640 1 --2- 172.21.15.33:0/3794788599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6208100740 0x7f62081046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:50:41.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.152+0000 7f61fffff640 1 --2- 172.21.15.33:0/3794788599 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6208101930 0x7f6208102d10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f61f800d950 tx=0x7f61f800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:41.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.152+0000 7f61fdffb640 1 -- 172.21.15.33:0/3794788599 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f8017030 con 0x7f6208101930 2024-03-19T02:50:41.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.152+0000 7f61fdffb640 1 -- 172.21.15.33:0/3794788599 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f61f8004020 con 0x7f6208101930 2024-03-19T02:50:41.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.152+0000 7f620ec44640 1 -- 172.21.15.33:0/3794788599 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62081a1c40 con 0x7f6208101930 2024-03-19T02:50:41.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.152+0000 7f61fdffb640 1 -- 172.21.15.33:0/3794788599 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f80047a0 con 0x7f6208101930 2024-03-19T02:50:41.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.152+0000 7f620ec44640 1 -- 172.21.15.33:0/3794788599 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62081a2190 con 0x7f6208101930 2024-03-19T02:50:41.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.153+0000 7f620ec44640 1 -- 172.21.15.33:0/3794788599 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62081a23c0 con 0x7f6208101930 2024-03-19T02:50:41.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.154+0000 7f61fdffb640 1 -- 172.21.15.33:0/3794788599 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f61f80041d0 con 0x7f6208101930 2024-03-19T02:50:41.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.155+0000 7f61fdffb640 1 --2- 172.21.15.33:0/3794788599 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f61f00a2800 0x7f61f00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:41.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.155+0000 7f61fdffb640 1 -- 172.21.15.33:0/3794788599 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f61f80ce120 con 0x7f6208101930 2024-03-19T02:50:41.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.155+0000 7f620c9e0640 1 --2- 172.21.15.33:0/3794788599 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f61f00a2800 0x7f61f00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:41.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.155+0000 7f620c9e0640 1 --2- 172.21.15.33:0/3794788599 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f61f00a2800 0x7f61f00a4cc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6204006010 tx=0x7f620402c7e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:41.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.159+0000 7f61fdffb640 1 -- 172.21.15.33:0/3794788599 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f61f809b660 con 0x7f6208101930 2024-03-19T02:50:41.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.388+0000 7f620ec44640 1 -- 172.21.15.33:0/3794788599 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f62081a2590 con 0x7f6208101930 2024-03-19T02:50:41.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.389+0000 7f61fdffb640 1 -- 172.21.15.33:0/3794788599 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+7998 (secure 0 0 0) 0x7f61f809af60 con 0x7f6208101930 2024-03-19T02:50:41.394 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:50:41.394 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24436,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-19T02:50:41.395 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 10 2024-03-19T02:50:41.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.391+0000 7f620ec44640 1 -- 172.21.15.33:0/3794788599 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f61f00a2800 msgr2=0x7f61f00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:41.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.391+0000 7f620ec44640 1 --2- 172.21.15.33:0/3794788599 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f61f00a2800 0x7f61f00a4cc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6204006010 tx=0x7f620402c7e0 comp rx=0 tx=0).stop 2024-03-19T02:50:41.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.391+0000 7f620ec44640 1 -- 172.21.15.33:0/3794788599 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6208101930 msgr2=0x7f6208102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:41.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.391+0000 7f620ec44640 1 --2- 172.21.15.33:0/3794788599 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6208101930 0x7f6208102d10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f61f800d950 tx=0x7f61f800de10 comp rx=0 tx=0).stop 2024-03-19T02:50:41.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.391+0000 7f620ec44640 1 -- 172.21.15.33:0/3794788599 shutdown_connections 2024-03-19T02:50:41.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.391+0000 7f620ec44640 1 --2- 172.21.15.33:0/3794788599 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f61f00a2800 0x7f61f00a4cc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:41.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.391+0000 7f620ec44640 1 --2- 172.21.15.33:0/3794788599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62081022f0 0x7f6208103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:41.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.391+0000 7f620ec44640 1 --2- 172.21.15.33:0/3794788599 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6208101930 0x7f6208102d10 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:41.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.391+0000 7f620ec44640 1 --2- 172.21.15.33:0/3794788599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6208100740 0x7f62081046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:41.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.391+0000 7f620ec44640 1 -- 172.21.15.33:0/3794788599 >> 172.21.15.33:0/3794788599 conn(0x7f62080fbcb0 msgr2=0x7f62080689d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:41.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.391+0000 7f620ec44640 1 -- 172.21.15.33:0/3794788599 shutdown_connections 2024-03-19T02:50:41.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:41.391+0000 7f620ec44640 1 -- 172.21.15.33:0/3794788599 wait complete. 2024-03-19T02:50:41.403 INFO:journalctl@ceph.mds.g.smithi033.stdout:Mar 19 02:50:41 smithi033 systemd[1]: Starting Ceph mds.g for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T02:50:41.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:41 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3794788599' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:41.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:41 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3794788599' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:41.769 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:41 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3794788599' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:41.770 INFO:journalctl@ceph.mds.g.smithi033.stdout:Mar 19 02:50:41 smithi033 podman[73472]: 2024-03-19 02:50:41.398340805 +0000 UTC m=+0.018570154 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:50:41.770 INFO:journalctl@ceph.mds.g.smithi033.stdout:Mar 19 02:50:41 smithi033 podman[73472]: 2024-03-19 02:50:41.500206856 +0000 UTC m=+0.120436186 container create d62af71f141093b3d3e6eef5f216257908dcf8fed9baa4c28d1e088ebc440c95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-g, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T02:50:41.770 INFO:journalctl@ceph.mds.g.smithi033.stdout:Mar 19 02:50:41 smithi033 podman[73472]: 2024-03-19 02:50:41.76996059 +0000 UTC m=+0.390189942 container init d62af71f141093b3d3e6eef5f216257908dcf8fed9baa4c28d1e088ebc440c95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-g, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-19T02:50:42.028 INFO:journalctl@ceph.mds.g.smithi033.stdout:Mar 19 02:50:41 smithi033 podman[73472]: 2024-03-19 02:50:41.772905151 +0000 UTC m=+0.393134494 container start d62af71f141093b3d3e6eef5f216257908dcf8fed9baa4c28d1e088ebc440c95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-g, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-19T02:50:42.028 INFO:journalctl@ceph.mds.g.smithi033.stdout:Mar 19 02:50:41 smithi033 ceph-mds[73494]: -- 172.21.15.33:0/3695882085 <== mon.0 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564cbef19a40 con 0x564cbeeec800 2024-03-19T02:50:42.028 INFO:journalctl@ceph.mds.g.smithi033.stdout:Mar 19 02:50:41 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-g[73490]: starting mds.g at 2024-03-19T02:50:42.029 INFO:journalctl@ceph.mds.g.smithi033.stdout:Mar 19 02:50:41 smithi033 bash[73472]: d62af71f141093b3d3e6eef5f216257908dcf8fed9baa4c28d1e088ebc440c95 2024-03-19T02:50:42.029 INFO:journalctl@ceph.mds.g.smithi033.stdout:Mar 19 02:50:41 smithi033 systemd[1]: Started Ceph mds.g for 2b7a8938-e59a-11ee-95c9-87774f69a715. 2024-03-19T02:50:42.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:42 smithi155 ceph-mon[35804]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:42.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:42 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:42.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:42 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:42.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:42 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:42.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:42 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:42.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:42 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:42.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:42 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:42.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:42 smithi033 ceph-mon[31973]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:42.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:42 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:42.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:42 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:42.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:42 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:42.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:42 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:42.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:42 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:42.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:42 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:42.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:42 smithi096 ceph-mon[35946]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:42.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:42 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:42.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:42 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:42.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:42 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:42.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:42 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:42.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:42 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:42.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:42 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:43.405 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:50:43.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.525+0000 7fb23cbb9640 1 -- 172.21.15.33:0/2912301005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb238104be0 msgr2=0x7fb238106fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:43.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.525+0000 7fb23cbb9640 1 --2- 172.21.15.33:0/2912301005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb238104be0 0x7fb238106fd0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb220009900 tx=0x7fb22002bc30 comp rx=0 tx=0).stop 2024-03-19T02:50:43.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.525+0000 7fb23cbb9640 1 -- 172.21.15.33:0/2912301005 shutdown_connections 2024-03-19T02:50:43.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.525+0000 7fb23cbb9640 1 --2- 172.21.15.33:0/2912301005 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb238107510 0x7fb238109960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:43.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.525+0000 7fb23cbb9640 1 --2- 172.21.15.33:0/2912301005 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb238104be0 0x7fb238106fd0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:43.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.525+0000 7fb23cbb9640 1 --2- 172.21.15.33:0/2912301005 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2381022b0 0x7fb2381046a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:43.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.525+0000 7fb23cbb9640 1 -- 172.21.15.33:0/2912301005 >> 172.21.15.33:0/2912301005 conn(0x7fb2380fbcc0 msgr2=0x7fb2380fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:43.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.525+0000 7fb23cbb9640 1 -- 172.21.15.33:0/2912301005 shutdown_connections 2024-03-19T02:50:43.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.525+0000 7fb23cbb9640 1 -- 172.21.15.33:0/2912301005 wait complete. 2024-03-19T02:50:43.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.526+0000 7fb23cbb9640 1 Processor -- start 2024-03-19T02:50:43.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.526+0000 7fb23cbb9640 1 -- start start 2024-03-19T02:50:43.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.526+0000 7fb23cbb9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb2381022b0 0x7fb238197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:43.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.526+0000 7fb23cbb9640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb238104be0 0x7fb238198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:43.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.526+0000 7fb23cbb9640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb238107510 0x7fb23819f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:43.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.526+0000 7fb23cbb9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb23810d370 con 0x7fb2381022b0 2024-03-19T02:50:43.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.526+0000 7fb23cbb9640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb23810d1f0 con 0x7fb238104be0 2024-03-19T02:50:43.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.526+0000 7fb23cbb9640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb23810d4f0 con 0x7fb238107510 2024-03-19T02:50:43.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.526+0000 7fb23659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb2381022b0 0x7fb238197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:43.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.526+0000 7fb23659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb2381022b0 0x7fb238197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33536/0 (socket says 172.21.15.33:33536) 2024-03-19T02:50:43.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.526+0000 7fb23659c640 1 -- 172.21.15.33:0/3604147880 learned_addr learned my addr 172.21.15.33:0/3604147880 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:43.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.526+0000 7fb235d9b640 1 --2- 172.21.15.33:0/3604147880 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb238104be0 0x7fb238198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:43.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.526+0000 7fb236d9d640 1 --2- 172.21.15.33:0/3604147880 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb238107510 0x7fb23819f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:43.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.526+0000 7fb23659c640 1 -- 172.21.15.33:0/3604147880 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb238107510 msgr2=0x7fb23819f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:43.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.526+0000 7fb23659c640 1 --2- 172.21.15.33:0/3604147880 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb238107510 0x7fb23819f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:43.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.526+0000 7fb23659c640 1 -- 172.21.15.33:0/3604147880 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb238104be0 msgr2=0x7fb238198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:43.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.526+0000 7fb23659c640 1 --2- 172.21.15.33:0/3604147880 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb238104be0 0x7fb238198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:43.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.526+0000 7fb23659c640 1 -- 172.21.15.33:0/3604147880 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb23819f700 con 0x7fb2381022b0 2024-03-19T02:50:43.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.526+0000 7fb235d9b640 1 --2- 172.21.15.33:0/3604147880 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb238104be0 0x7fb238198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:50:43.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.526+0000 7fb236d9d640 1 --2- 172.21.15.33:0/3604147880 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb238107510 0x7fb23819f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:50:43.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.526+0000 7fb23659c640 1 --2- 172.21.15.33:0/3604147880 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb2381022b0 0x7fb238197ae0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb22800ea80 tx=0x7fb22800ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:43.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.527+0000 7fb22f7fe640 1 -- 172.21.15.33:0/3604147880 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22800ce40 con 0x7fb2381022b0 2024-03-19T02:50:43.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.527+0000 7fb23cbb9640 1 -- 172.21.15.33:0/3604147880 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb23819f9f0 con 0x7fb2381022b0 2024-03-19T02:50:43.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.527+0000 7fb22f7fe640 1 -- 172.21.15.33:0/3604147880 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb228005020 con 0x7fb2381022b0 2024-03-19T02:50:43.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.527+0000 7fb22f7fe640 1 -- 172.21.15.33:0/3604147880 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2280116d0 con 0x7fb2381022b0 2024-03-19T02:50:43.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.527+0000 7fb23cbb9640 1 -- 172.21.15.33:0/3604147880 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb23819fec0 con 0x7fb2381022b0 2024-03-19T02:50:43.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:43 smithi033 ceph-mon[31973]: Deploying daemon mds.a on smithi033 2024-03-19T02:50:43.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:43 smithi033 ceph-mon[31973]: mds.? [v2:172.21.15.33:6838/1789650898,v1:172.21.15.33:6839/1789650898] up:boot 2024-03-19T02:50:43.531 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:43 smithi033 ceph-mon[31973]: fsmap 10 up:standby 2024-03-19T02:50:43.532 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:43 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-19T02:50:43.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.528+0000 7fb23cbb9640 1 -- 172.21.15.33:0/3604147880 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2380fd7b0 con 0x7fb2381022b0 2024-03-19T02:50:43.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.529+0000 7fb22f7fe640 1 -- 172.21.15.33:0/3604147880 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb2280051d0 con 0x7fb2381022b0 2024-03-19T02:50:43.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.529+0000 7fb22f7fe640 1 --2- 172.21.15.33:0/3604147880 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb2240a2800 0x7fb2240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:43.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.529+0000 7fb235d9b640 1 --2- 172.21.15.33:0/3604147880 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb2240a2800 0x7fb2240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:43.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.529+0000 7fb22f7fe640 1 -- 172.21.15.33:0/3604147880 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fb2280ce960 con 0x7fb2381022b0 2024-03-19T02:50:43.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.530+0000 7fb235d9b640 1 --2- 172.21.15.33:0/3604147880 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb2240a2800 0x7fb2240a4cc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb220006010 tx=0x7fb22002c7e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:43.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.534+0000 7fb22f7fe640 1 -- 172.21.15.33:0/3604147880 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb228016030 con 0x7fb2381022b0 2024-03-19T02:50:43.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:43 smithi155 ceph-mon[35804]: Deploying daemon mds.a on smithi033 2024-03-19T02:50:43.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:43 smithi155 ceph-mon[35804]: mds.? [v2:172.21.15.33:6838/1789650898,v1:172.21.15.33:6839/1789650898] up:boot 2024-03-19T02:50:43.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:43 smithi155 ceph-mon[35804]: fsmap 10 up:standby 2024-03-19T02:50:43.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:43 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-19T02:50:43.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.769+0000 7fb23cbb9640 1 -- 172.21.15.33:0/3604147880 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb23802d300 con 0x7fb2381022b0 2024-03-19T02:50:43.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.770+0000 7fb22f7fe640 1 -- 172.21.15.33:0/3604147880 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8824 (secure 0 0 0) 0x7fb22809bea0 con 0x7fb2381022b0 2024-03-19T02:50:43.771 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:50:43.771 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24436,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-19T02:50:43.771 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 11 2024-03-19T02:50:43.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.772+0000 7fb23cbb9640 1 -- 172.21.15.33:0/3604147880 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb2240a2800 msgr2=0x7fb2240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:43.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.772+0000 7fb23cbb9640 1 --2- 172.21.15.33:0/3604147880 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb2240a2800 0x7fb2240a4cc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb220006010 tx=0x7fb22002c7e0 comp rx=0 tx=0).stop 2024-03-19T02:50:43.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.772+0000 7fb23cbb9640 1 -- 172.21.15.33:0/3604147880 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb2381022b0 msgr2=0x7fb238197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:43.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.772+0000 7fb23cbb9640 1 --2- 172.21.15.33:0/3604147880 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb2381022b0 0x7fb238197ae0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb22800ea80 tx=0x7fb22800ef40 comp rx=0 tx=0).stop 2024-03-19T02:50:43.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.772+0000 7fb23cbb9640 1 -- 172.21.15.33:0/3604147880 shutdown_connections 2024-03-19T02:50:43.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.772+0000 7fb23cbb9640 1 --2- 172.21.15.33:0/3604147880 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb2240a2800 0x7fb2240a4cc0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:43.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.772+0000 7fb23cbb9640 1 --2- 172.21.15.33:0/3604147880 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb238107510 0x7fb23819f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:43.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.772+0000 7fb23cbb9640 1 --2- 172.21.15.33:0/3604147880 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb238104be0 0x7fb238198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:43.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.772+0000 7fb23cbb9640 1 --2- 172.21.15.33:0/3604147880 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb2381022b0 0x7fb238197ae0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:43.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.772+0000 7fb23cbb9640 1 -- 172.21.15.33:0/3604147880 >> 172.21.15.33:0/3604147880 conn(0x7fb2380fbcc0 msgr2=0x7fb238102bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:43.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.772+0000 7fb23cbb9640 1 -- 172.21.15.33:0/3604147880 shutdown_connections 2024-03-19T02:50:43.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:43.772+0000 7fb23cbb9640 1 -- 172.21.15.33:0/3604147880 wait complete. 2024-03-19T02:50:43.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:43 smithi096 ceph-mon[35946]: Deploying daemon mds.a on smithi033 2024-03-19T02:50:43.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:43 smithi096 ceph-mon[35946]: mds.? [v2:172.21.15.33:6838/1789650898,v1:172.21.15.33:6839/1789650898] up:boot 2024-03-19T02:50:43.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:43 smithi096 ceph-mon[35946]: fsmap 10 up:standby 2024-03-19T02:50:43.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:43 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-19T02:50:44.279 INFO:journalctl@ceph.mds.a.smithi033.stdout:Mar 19 02:50:44 smithi033 systemd[1]: Starting Ceph mds.a for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T02:50:44.643 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:44 smithi033 ceph-mon[31973]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:44.643 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:44 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3604147880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:44.643 INFO:journalctl@ceph.mds.a.smithi033.stdout:Mar 19 02:50:44 smithi033 podman[73821]: 2024-03-19 02:50:44.244123275 +0000 UTC m=+0.017737130 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:50:44.643 INFO:journalctl@ceph.mds.a.smithi033.stdout:Mar 19 02:50:44 smithi033 podman[73821]: 2024-03-19 02:50:44.373389251 +0000 UTC m=+0.147003108 container create 4b3e4c2dbe2052015d1f9f1c0cfb27f97035c965bf34418708ec3f5ec7bb5806 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-03-19T02:50:44.643 INFO:journalctl@ceph.mds.a.smithi033.stdout:Mar 19 02:50:44 smithi033 podman[73821]: 2024-03-19 02:50:44.642830635 +0000 UTC m=+0.416444490 container init 4b3e4c2dbe2052015d1f9f1c0cfb27f97035c965bf34418708ec3f5ec7bb5806 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-a, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T02:50:44.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:44 smithi155 ceph-mon[35804]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:44.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:44 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3604147880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:44.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:44 smithi096 ceph-mon[35946]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:44.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:44 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3604147880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:45.028 INFO:journalctl@ceph.mds.a.smithi033.stdout:Mar 19 02:50:44 smithi033 podman[73821]: 2024-03-19 02:50:44.645772829 +0000 UTC m=+0.419386702 container start 4b3e4c2dbe2052015d1f9f1c0cfb27f97035c965bf34418708ec3f5ec7bb5806 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-a, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-19T02:50:45.029 INFO:journalctl@ceph.mds.a.smithi033.stdout:Mar 19 02:50:44 smithi033 ceph-mds[73840]: -- 172.21.15.33:0/3958659586 <== mon.0 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b742f89a40 con 0x55b742f5c800 2024-03-19T02:50:45.029 INFO:journalctl@ceph.mds.a.smithi033.stdout:Mar 19 02:50:44 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-a[73836]: starting mds.a at 2024-03-19T02:50:45.029 INFO:journalctl@ceph.mds.a.smithi033.stdout:Mar 19 02:50:44 smithi033 bash[73821]: 4b3e4c2dbe2052015d1f9f1c0cfb27f97035c965bf34418708ec3f5ec7bb5806 2024-03-19T02:50:45.029 INFO:journalctl@ceph.mds.a.smithi033.stdout:Mar 19 02:50:44 smithi033 systemd[1]: Started Ceph mds.a for 2b7a8938-e59a-11ee-95c9-87774f69a715. 2024-03-19T02:50:45.788 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:50:45.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.906+0000 7fdcfa0ff640 1 -- 172.21.15.33:0/2410489441 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdcf4101950 msgr2=0x7fdcf4101dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:45.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.906+0000 7fdcfa0ff640 1 --2- 172.21.15.33:0/2410489441 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdcf4101950 0x7fdcf4101dd0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdce8009900 tx=0x7fdce802bc30 comp rx=0 tx=0).stop 2024-03-19T02:50:45.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.906+0000 7fdcfa0ff640 1 -- 172.21.15.33:0/2410489441 shutdown_connections 2024-03-19T02:50:45.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.906+0000 7fdcfa0ff640 1 --2- 172.21.15.33:0/2410489441 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdcf4102310 0x7fdcf4108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:45.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.906+0000 7fdcfa0ff640 1 --2- 172.21.15.33:0/2410489441 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdcf4101950 0x7fdcf4101dd0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:45.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.906+0000 7fdcfa0ff640 1 --2- 172.21.15.33:0/2410489441 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdcf4100760 0x7fdcf4100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:45.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.906+0000 7fdcfa0ff640 1 -- 172.21.15.33:0/2410489441 >> 172.21.15.33:0/2410489441 conn(0x7fdcf40fbcd0 msgr2=0x7fdcf40fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:45.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.906+0000 7fdcfa0ff640 1 -- 172.21.15.33:0/2410489441 shutdown_connections 2024-03-19T02:50:45.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.906+0000 7fdcfa0ff640 1 -- 172.21.15.33:0/2410489441 wait complete. 2024-03-19T02:50:45.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.907+0000 7fdcfa0ff640 1 Processor -- start 2024-03-19T02:50:45.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.907+0000 7fdcfa0ff640 1 -- start start 2024-03-19T02:50:45.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.907+0000 7fdcfa0ff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdcf4100760 0x7fdcf4196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:45.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.907+0000 7fdcfa0ff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdcf4101950 0x7fdcf4196f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:45.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.907+0000 7fdcfa0ff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdcf4102310 0x7fdcf419dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:45.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.907+0000 7fdcfa0ff640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf410c270 con 0x7fdcf4101950 2024-03-19T02:50:45.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.907+0000 7fdcfa0ff640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf410c0f0 con 0x7fdcf4102310 2024-03-19T02:50:45.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.907+0000 7fdcfa0ff640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf410c3f0 con 0x7fdcf4100760 2024-03-19T02:50:45.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.907+0000 7fdcf2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdcf4101950 0x7fdcf4196f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:45.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.907+0000 7fdcf2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdcf4101950 0x7fdcf4196f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33560/0 (socket says 172.21.15.33:33560) 2024-03-19T02:50:45.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.907+0000 7fdcf2ffd640 1 -- 172.21.15.33:0/4220023775 learned_addr learned my addr 172.21.15.33:0/4220023775 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:45.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.908+0000 7fdcf3fff640 1 --2- 172.21.15.33:0/4220023775 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdcf4102310 0x7fdcf419dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:45.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.908+0000 7fdcf37fe640 1 --2- 172.21.15.33:0/4220023775 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdcf4100760 0x7fdcf4196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:45.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.908+0000 7fdcf2ffd640 1 -- 172.21.15.33:0/4220023775 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdcf4100760 msgr2=0x7fdcf4196a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:45.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.908+0000 7fdcf2ffd640 1 --2- 172.21.15.33:0/4220023775 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdcf4100760 0x7fdcf4196a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:45.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.908+0000 7fdcf2ffd640 1 -- 172.21.15.33:0/4220023775 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdcf4102310 msgr2=0x7fdcf419dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:45.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.908+0000 7fdcf2ffd640 1 --2- 172.21.15.33:0/4220023775 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdcf4102310 0x7fdcf419dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:45.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.908+0000 7fdcf2ffd640 1 -- 172.21.15.33:0/4220023775 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcf4069e50 con 0x7fdcf4101950 2024-03-19T02:50:45.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.908+0000 7fdcf3fff640 1 --2- 172.21.15.33:0/4220023775 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdcf4102310 0x7fdcf419dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:50:45.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.908+0000 7fdcf2ffd640 1 --2- 172.21.15.33:0/4220023775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdcf4101950 0x7fdcf4196f40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdce8006010 tx=0x7fdce8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:45.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.908+0000 7fdcf0ff9640 1 -- 172.21.15.33:0/4220023775 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce8043020 con 0x7fdcf4101950 2024-03-19T02:50:45.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.908+0000 7fdcf0ff9640 1 -- 172.21.15.33:0/4220023775 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdce8004440 con 0x7fdcf4101950 2024-03-19T02:50:45.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.908+0000 7fdcf0ff9640 1 -- 172.21.15.33:0/4220023775 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce8003bf0 con 0x7fdcf4101950 2024-03-19T02:50:45.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.908+0000 7fdcfa0ff640 1 -- 172.21.15.33:0/4220023775 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcf406a0e0 con 0x7fdcf4101950 2024-03-19T02:50:45.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.908+0000 7fdcfa0ff640 1 -- 172.21.15.33:0/4220023775 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcf406a680 con 0x7fdcf4101950 2024-03-19T02:50:45.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.909+0000 7fdcfa0ff640 1 -- 172.21.15.33:0/4220023775 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcf4107aa0 con 0x7fdcf4101950 2024-03-19T02:50:45.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.910+0000 7fdcf0ff9640 1 -- 172.21.15.33:0/4220023775 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fdce8046020 con 0x7fdcf4101950 2024-03-19T02:50:45.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.911+0000 7fdcf0ff9640 1 --2- 172.21.15.33:0/4220023775 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fdcdc0a2880 0x7fdcdc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:45.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.911+0000 7fdcf37fe640 1 --2- 172.21.15.33:0/4220023775 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fdcdc0a2880 0x7fdcdc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:45.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.911+0000 7fdcf0ff9640 1 -- 172.21.15.33:0/4220023775 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fdce80f1050 con 0x7fdcf4101950 2024-03-19T02:50:45.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.911+0000 7fdcf37fe640 1 --2- 172.21.15.33:0/4220023775 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fdcdc0a2880 0x7fdcdc0a4d40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fdcf41017b0 tx=0x7fdce4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:45.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:45.915+0000 7fdcf0ff9640 1 -- 172.21.15.33:0/4220023775 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdce804d020 con 0x7fdcf4101950 2024-03-19T02:50:46.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:45 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:46.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:45 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:46.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:45 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:46.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:45 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:46.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:45 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:46.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:45 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:46.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:45 smithi033 ceph-mon[31973]: Deploying daemon mds.b on smithi096 2024-03-19T02:50:46.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:45 smithi033 ceph-mon[31973]: mds.? [v2:172.21.15.33:6840/1349324757,v1:172.21.15.33:6841/1349324757] up:boot 2024-03-19T02:50:46.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:45 smithi033 ceph-mon[31973]: fsmap 11 up:standby 2024-03-19T02:50:46.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:45 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-19T02:50:46.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:45 smithi033 ceph-mon[31973]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:46.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:46.137+0000 7fdcfa0ff640 1 -- 172.21.15.33:0/4220023775 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdcf402d1f0 con 0x7fdcf4101950 2024-03-19T02:50:46.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:46.137+0000 7fdcf0ff9640 1 -- 172.21.15.33:0/4220023775 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9650 (secure 0 0 0) 0x7fdce803e450 con 0x7fdcf4101950 2024-03-19T02:50:46.139 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:50:46.139 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24436,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-19T02:50:46.139 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 12 2024-03-19T02:50:46.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:46.139+0000 7fdcfa0ff640 1 -- 172.21.15.33:0/4220023775 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fdcdc0a2880 msgr2=0x7fdcdc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:46.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:46.139+0000 7fdcfa0ff640 1 --2- 172.21.15.33:0/4220023775 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fdcdc0a2880 0x7fdcdc0a4d40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fdcf41017b0 tx=0x7fdce4005f90 comp rx=0 tx=0).stop 2024-03-19T02:50:46.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:46.139+0000 7fdcfa0ff640 1 -- 172.21.15.33:0/4220023775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdcf4101950 msgr2=0x7fdcf4196f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:46.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:46.139+0000 7fdcfa0ff640 1 --2- 172.21.15.33:0/4220023775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdcf4101950 0x7fdcf4196f40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdce8006010 tx=0x7fdce8004040 comp rx=0 tx=0).stop 2024-03-19T02:50:46.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:46.139+0000 7fdcfa0ff640 1 -- 172.21.15.33:0/4220023775 shutdown_connections 2024-03-19T02:50:46.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:46.139+0000 7fdcfa0ff640 1 --2- 172.21.15.33:0/4220023775 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fdcdc0a2880 0x7fdcdc0a4d40 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:46.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:46.139+0000 7fdcfa0ff640 1 --2- 172.21.15.33:0/4220023775 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdcf4102310 0x7fdcf419dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:46.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:46.139+0000 7fdcfa0ff640 1 --2- 172.21.15.33:0/4220023775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdcf4101950 0x7fdcf4196f40 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:46.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:46.139+0000 7fdcfa0ff640 1 --2- 172.21.15.33:0/4220023775 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdcf4100760 0x7fdcf4196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:46.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:46.139+0000 7fdcfa0ff640 1 -- 172.21.15.33:0/4220023775 >> 172.21.15.33:0/4220023775 conn(0x7fdcf40fbcd0 msgr2=0x7fdcf4104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:46.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:46.140+0000 7fdcfa0ff640 1 -- 172.21.15.33:0/4220023775 shutdown_connections 2024-03-19T02:50:46.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:46.140+0000 7fdcfa0ff640 1 -- 172.21.15.33:0/4220023775 wait complete. 2024-03-19T02:50:46.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:45 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:46.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:45 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:46.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:45 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:46.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:45 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:46.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:45 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:46.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:45 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:46.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:45 smithi096 ceph-mon[35946]: Deploying daemon mds.b on smithi096 2024-03-19T02:50:46.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:45 smithi096 ceph-mon[35946]: mds.? [v2:172.21.15.33:6840/1349324757,v1:172.21.15.33:6841/1349324757] up:boot 2024-03-19T02:50:46.217 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:45 smithi096 ceph-mon[35946]: fsmap 11 up:standby 2024-03-19T02:50:46.217 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:45 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-19T02:50:46.217 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:45 smithi096 ceph-mon[35946]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:46.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:45 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:46.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:45 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:46.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:45 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:46.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:45 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T02:50:46.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:45 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T02:50:46.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:45 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:46.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:45 smithi155 ceph-mon[35804]: Deploying daemon mds.b on smithi096 2024-03-19T02:50:46.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:45 smithi155 ceph-mon[35804]: mds.? [v2:172.21.15.33:6840/1349324757,v1:172.21.15.33:6841/1349324757] up:boot 2024-03-19T02:50:46.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:45 smithi155 ceph-mon[35804]: fsmap 11 up:standby 2024-03-19T02:50:46.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:45 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-19T02:50:46.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:45 smithi155 ceph-mon[35804]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:47.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:46 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/4220023775' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:47.123 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:46 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/4220023775' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:47.123 INFO:journalctl@ceph.mds.b.smithi096.stdout:Mar 19 02:50:47 smithi096 systemd[1]: Starting Ceph mds.b for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T02:50:47.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:46 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/4220023775' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:47.465 INFO:journalctl@ceph.mds.b.smithi096.stdout:Mar 19 02:50:47 smithi096 podman[65444]: 2024-03-19 02:50:47.241135977 +0000 UTC m=+0.018191351 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T02:50:47.466 INFO:journalctl@ceph.mds.b.smithi096.stdout:Mar 19 02:50:47 smithi096 podman[65444]: 2024-03-19 02:50:47.355073307 +0000 UTC m=+0.132128716 container create a09bdf94d17e0c7a9e940c0065916e55bfd61c2039eba3d1fdabd8938242f271 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-b, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-19T02:50:47.965 INFO:journalctl@ceph.mds.b.smithi096.stdout:Mar 19 02:50:47 smithi096 podman[65444]: 2024-03-19 02:50:47.591256847 +0000 UTC m=+0.368312213 container init a09bdf94d17e0c7a9e940c0065916e55bfd61c2039eba3d1fdabd8938242f271 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-19T02:50:47.965 INFO:journalctl@ceph.mds.b.smithi096.stdout:Mar 19 02:50:47 smithi096 podman[65444]: 2024-03-19 02:50:47.593718137 +0000 UTC m=+0.370773506 container start a09bdf94d17e0c7a9e940c0065916e55bfd61c2039eba3d1fdabd8938242f271 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-19T02:50:47.966 INFO:journalctl@ceph.mds.b.smithi096.stdout:Mar 19 02:50:47 smithi096 ceph-mds[65463]: -- 172.21.15.96:0/3938786350 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f34d039a40 con 0x55f34d00c800 2024-03-19T02:50:47.966 INFO:journalctl@ceph.mds.b.smithi096.stdout:Mar 19 02:50:47 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-b[65459]: starting mds.b at 2024-03-19T02:50:47.966 INFO:journalctl@ceph.mds.b.smithi096.stdout:Mar 19 02:50:47 smithi096 bash[65444]: a09bdf94d17e0c7a9e940c0065916e55bfd61c2039eba3d1fdabd8938242f271 2024-03-19T02:50:47.966 INFO:journalctl@ceph.mds.b.smithi096.stdout:Mar 19 02:50:47 smithi096 systemd[1]: Started Ceph mds.b for 2b7a8938-e59a-11ee-95c9-87774f69a715. 2024-03-19T02:50:48.152 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:50:48.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.255+0000 7fa2b656f640 1 -- 172.21.15.33:0/996046578 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2b01022f0 msgr2=0x7fa2b0108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:48.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.255+0000 7fa2b656f640 1 --2- 172.21.15.33:0/996046578 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2b01022f0 0x7fa2b0108b80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa2a8009ca0 tx=0x7fa2a802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:50:48.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.255+0000 7fa2b656f640 1 -- 172.21.15.33:0/996046578 shutdown_connections 2024-03-19T02:50:48.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.255+0000 7fa2b656f640 1 --2- 172.21.15.33:0/996046578 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2b01022f0 0x7fa2b0108b80 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:48.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.255+0000 7fa2b656f640 1 --2- 172.21.15.33:0/996046578 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2b0101930 0x7fa2b0101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:48.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.255+0000 7fa2b656f640 1 --2- 172.21.15.33:0/996046578 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2b0100740 0x7fa2b0100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:48.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.255+0000 7fa2b656f640 1 -- 172.21.15.33:0/996046578 >> 172.21.15.33:0/996046578 conn(0x7fa2b00fbcb0 msgr2=0x7fa2b00fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:48.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.255+0000 7fa2b656f640 1 -- 172.21.15.33:0/996046578 shutdown_connections 2024-03-19T02:50:48.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.256+0000 7fa2b656f640 1 -- 172.21.15.33:0/996046578 wait complete. 2024-03-19T02:50:48.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.256+0000 7fa2b656f640 1 Processor -- start 2024-03-19T02:50:48.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.256+0000 7fa2b656f640 1 -- start start 2024-03-19T02:50:48.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.256+0000 7fa2b656f640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2b0100740 0x7fa2b0196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:48.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.256+0000 7fa2b656f640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2b0101930 0x7fa2b0196e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:48.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.256+0000 7fa2b656f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2b01022f0 0x7fa2b019dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:48.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.256+0000 7fa2b656f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b010c270 con 0x7fa2b01022f0 2024-03-19T02:50:48.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.256+0000 7fa2b656f640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b010c0f0 con 0x7fa2b0100740 2024-03-19T02:50:48.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.256+0000 7fa2b656f640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b010c3f0 con 0x7fa2b0101930 2024-03-19T02:50:48.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.256+0000 7fa2b4b0c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2b01022f0 0x7fa2b019dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:48.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.256+0000 7fa2b4b0c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2b01022f0 0x7fa2b019dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33582/0 (socket says 172.21.15.33:33582) 2024-03-19T02:50:48.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.257+0000 7fa2b4b0c640 1 -- 172.21.15.33:0/2103974887 learned_addr learned my addr 172.21.15.33:0/2103974887 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:48.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.257+0000 7fa2affff640 1 --2- 172.21.15.33:0/2103974887 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2b0100740 0x7fa2b0196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:48.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.257+0000 7fa2af7fe640 1 --2- 172.21.15.33:0/2103974887 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2b0101930 0x7fa2b0196e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:48.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.257+0000 7fa2b4b0c640 1 -- 172.21.15.33:0/2103974887 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2b0101930 msgr2=0x7fa2b0196e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:48.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.257+0000 7fa2b4b0c640 1 --2- 172.21.15.33:0/2103974887 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2b0101930 0x7fa2b0196e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:48.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.257+0000 7fa2b4b0c640 1 -- 172.21.15.33:0/2103974887 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2b0100740 msgr2=0x7fa2b0196910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:48.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.257+0000 7fa2b4b0c640 1 --2- 172.21.15.33:0/2103974887 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2b0100740 0x7fa2b0196910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:48.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.257+0000 7fa2b4b0c640 1 -- 172.21.15.33:0/2103974887 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2b019e530 con 0x7fa2b01022f0 2024-03-19T02:50:48.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.257+0000 7fa2b4b0c640 1 --2- 172.21.15.33:0/2103974887 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2b01022f0 0x7fa2b019dec0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa2a802c700 tx=0x7fa2a80328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:48.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.257+0000 7fa2ad7fa640 1 -- 172.21.15.33:0/2103974887 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a8041020 con 0x7fa2b01022f0 2024-03-19T02:50:48.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.257+0000 7fa2b656f640 1 -- 172.21.15.33:0/2103974887 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2b019e7c0 con 0x7fa2b01022f0 2024-03-19T02:50:48.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.257+0000 7fa2ad7fa640 1 -- 172.21.15.33:0/2103974887 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa2a8003e20 con 0x7fa2b01022f0 2024-03-19T02:50:48.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.257+0000 7fa2ad7fa640 1 -- 172.21.15.33:0/2103974887 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a8039040 con 0x7fa2b01022f0 2024-03-19T02:50:48.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.257+0000 7fa2b656f640 1 -- 172.21.15.33:0/2103974887 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2b019ecb0 con 0x7fa2b01022f0 2024-03-19T02:50:48.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.258+0000 7fa2b656f640 1 -- 172.21.15.33:0/2103974887 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2b0107ae0 con 0x7fa2b01022f0 2024-03-19T02:50:48.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.259+0000 7fa2ad7fa640 1 -- 172.21.15.33:0/2103974887 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fa2a8044020 con 0x7fa2b01022f0 2024-03-19T02:50:48.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.260+0000 7fa2ad7fa640 1 --2- 172.21.15.33:0/2103974887 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fa2980a2800 0x7fa2980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:48.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.260+0000 7fa2affff640 1 --2- 172.21.15.33:0/2103974887 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fa2980a2800 0x7fa2980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:48.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.260+0000 7fa2ad7fa640 1 -- 172.21.15.33:0/2103974887 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fa2a80f0860 con 0x7fa2b01022f0 2024-03-19T02:50:48.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.260+0000 7fa2affff640 1 --2- 172.21.15.33:0/2103974887 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fa2980a2800 0x7fa2980a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0101790 tx=0x7fa2a0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:48.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.264+0000 7fa2ad7fa640 1 -- 172.21.15.33:0/2103974887 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa2a804b020 con 0x7fa2b01022f0 2024-03-19T02:50:48.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.494+0000 7fa2b656f640 1 -- 172.21.15.33:0/2103974887 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2b002d3c0 con 0x7fa2b01022f0 2024-03-19T02:50:48.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.495+0000 7fa2ad7fa640 1 -- 172.21.15.33:0/2103974887 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10476 (secure 0 0 0) 0x7fa2a80bdda0 con 0x7fa2b01022f0 2024-03-19T02:50:48.496 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:50:48.496 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24436,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24509,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6839/3731232169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":3731232169},{"type":"v1","addr":"172.21.15.96:6839","nonce":3731232169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-03-19T02:50:48.496 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 13 2024-03-19T02:50:48.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.497+0000 7fa2b656f640 1 -- 172.21.15.33:0/2103974887 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fa2980a2800 msgr2=0x7fa2980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:48.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.497+0000 7fa2b656f640 1 --2- 172.21.15.33:0/2103974887 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fa2980a2800 0x7fa2980a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0101790 tx=0x7fa2a0005f90 comp rx=0 tx=0).stop 2024-03-19T02:50:48.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.497+0000 7fa2b656f640 1 -- 172.21.15.33:0/2103974887 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2b01022f0 msgr2=0x7fa2b019dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:48.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.497+0000 7fa2b656f640 1 --2- 172.21.15.33:0/2103974887 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2b01022f0 0x7fa2b019dec0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa2a802c700 tx=0x7fa2a80328a0 comp rx=0 tx=0).stop 2024-03-19T02:50:48.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.497+0000 7fa2b656f640 1 -- 172.21.15.33:0/2103974887 shutdown_connections 2024-03-19T02:50:48.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.497+0000 7fa2b656f640 1 --2- 172.21.15.33:0/2103974887 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fa2980a2800 0x7fa2980a4cc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:48.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.497+0000 7fa2b656f640 1 --2- 172.21.15.33:0/2103974887 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2b0101930 0x7fa2b0196e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:48.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.497+0000 7fa2b656f640 1 --2- 172.21.15.33:0/2103974887 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa2b01022f0 0x7fa2b019dec0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:48.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.497+0000 7fa2b656f640 1 --2- 172.21.15.33:0/2103974887 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2b0100740 0x7fa2b0196910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:48.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.497+0000 7fa2b656f640 1 -- 172.21.15.33:0/2103974887 >> 172.21.15.33:0/2103974887 conn(0x7fa2b00fbcb0 msgr2=0x7fa2b0104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:48.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.498+0000 7fa2b656f640 1 -- 172.21.15.33:0/2103974887 shutdown_connections 2024-03-19T02:50:48.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.498+0000 7fa2b656f640 1 -- 172.21.15.33:0/2103974887 wait complete. 2024-03-19T02:50:48.511 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:50:48.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.625+0000 7f8ffdf13640 1 -- 172.21.15.33:0/3042062737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ff81034a0 msgr2=0x7f8ff8109d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:48.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.625+0000 7f8ffdf13640 1 --2- 172.21.15.33:0/3042062737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ff81034a0 0x7f8ff8109d30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0009cd0 tx=0x7f8ff002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:50:48.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.625+0000 7f8ffdf13640 1 -- 172.21.15.33:0/3042062737 shutdown_connections 2024-03-19T02:50:48.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.625+0000 7f8ffdf13640 1 --2- 172.21.15.33:0/3042062737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ff81034a0 0x7f8ff8109d30 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:48.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.625+0000 7f8ffdf13640 1 --2- 172.21.15.33:0/3042062737 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ff8102ae0 0x7f8ff8102f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:48.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.625+0000 7f8ffdf13640 1 --2- 172.21.15.33:0/3042062737 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ff80ffb10 0x7f8ff80fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:48.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.625+0000 7f8ffdf13640 1 -- 172.21.15.33:0/3042062737 >> 172.21.15.33:0/3042062737 conn(0x7f8ff80fb440 msgr2=0x7f8ff80fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:48.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.625+0000 7f8ffdf13640 1 -- 172.21.15.33:0/3042062737 shutdown_connections 2024-03-19T02:50:48.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.625+0000 7f8ffdf13640 1 -- 172.21.15.33:0/3042062737 wait complete. 2024-03-19T02:50:48.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.625+0000 7f8ffdf13640 1 Processor -- start 2024-03-19T02:50:48.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.626+0000 7f8ffdf13640 1 -- start start 2024-03-19T02:50:48.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.626+0000 7f8ffdf13640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ff80ffb10 0x7f8ff8102450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:48.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.626+0000 7f8ffdf13640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ff8102ae0 0x7f8ff8100a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:48.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.626+0000 7f8ffdf13640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ff81034a0 0x7f8ff8100fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:48.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.626+0000 7f8ffdf13640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff810d410 con 0x7f8ff81034a0 2024-03-19T02:50:48.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.626+0000 7f8ffdf13640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff810d290 con 0x7f8ff8102ae0 2024-03-19T02:50:48.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.626+0000 7f8ffdf13640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff810d590 con 0x7f8ff80ffb10 2024-03-19T02:50:48.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.626+0000 7f8ff7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ff81034a0 0x7f8ff8100fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:48.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.626+0000 7f8ff7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ff81034a0 0x7f8ff8100fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33606/0 (socket says 172.21.15.33:33606) 2024-03-19T02:50:48.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.626+0000 7f8ff7fff640 1 -- 172.21.15.33:0/990260040 learned_addr learned my addr 172.21.15.33:0/990260040 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:48.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.626+0000 7f8feeffd640 1 --2- 172.21.15.33:0/990260040 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ff8102ae0 0x7f8ff8100a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:48.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.626+0000 7f8ff77fe640 1 --2- 172.21.15.33:0/990260040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ff80ffb10 0x7f8ff8102450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:48.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.626+0000 7f8ff7fff640 1 -- 172.21.15.33:0/990260040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ff80ffb10 msgr2=0x7f8ff8102450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:48.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.626+0000 7f8ff7fff640 1 --2- 172.21.15.33:0/990260040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ff80ffb10 0x7f8ff8102450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:48.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.626+0000 7f8ff7fff640 1 -- 172.21.15.33:0/990260040 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ff8102ae0 msgr2=0x7f8ff8100a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:48.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.626+0000 7f8ff7fff640 1 --2- 172.21.15.33:0/990260040 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ff8102ae0 0x7f8ff8100a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:48.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.626+0000 7f8ff7fff640 1 -- 172.21.15.33:0/990260040 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ff8101860 con 0x7f8ff81034a0 2024-03-19T02:50:48.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.626+0000 7f8ff7fff640 1 --2- 172.21.15.33:0/990260040 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ff81034a0 0x7f8ff8100fd0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8ff002c630 tx=0x7f8ff0032910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:48.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.627+0000 7f8ff57fa640 1 -- 172.21.15.33:0/990260040 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff0039040 con 0x7f8ff81034a0 2024-03-19T02:50:48.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.627+0000 7f8ff57fa640 1 -- 172.21.15.33:0/990260040 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8ff0044070 con 0x7f8ff81034a0 2024-03-19T02:50:48.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.627+0000 7f8ffdf13640 1 -- 172.21.15.33:0/990260040 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ff81a1bd0 con 0x7f8ff81034a0 2024-03-19T02:50:48.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.627+0000 7f8ff57fa640 1 -- 172.21.15.33:0/990260040 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff00045d0 con 0x7f8ff81034a0 2024-03-19T02:50:48.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.627+0000 7f8ffdf13640 1 -- 172.21.15.33:0/990260040 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ff81a2120 con 0x7f8ff81034a0 2024-03-19T02:50:48.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.628+0000 7f8ffdf13640 1 -- 172.21.15.33:0/990260040 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ff8108d70 con 0x7f8ff81034a0 2024-03-19T02:50:48.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.629+0000 7f8ff57fa640 1 -- 172.21.15.33:0/990260040 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f8ff0003970 con 0x7f8ff81034a0 2024-03-19T02:50:48.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.629+0000 7f8ff57fa640 1 --2- 172.21.15.33:0/990260040 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8fe40a2800 0x7f8fe40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:48.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.629+0000 7f8ff77fe640 1 --2- 172.21.15.33:0/990260040 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8fe40a2800 0x7f8fe40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:48.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.629+0000 7f8ff57fa640 1 -- 172.21.15.33:0/990260040 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f8ff00f0e80 con 0x7f8ff81034a0 2024-03-19T02:50:48.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.630+0000 7f8ff77fe640 1 --2- 172.21.15.33:0/990260040 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8fe40a2800 0x7f8fe40a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8ff8069c50 tx=0x7f8fe8009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:48.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.634+0000 7f8ff57fa640 1 -- 172.21.15.33:0/990260040 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8ff00be3c0 con 0x7f8ff81034a0 2024-03-19T02:50:48.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:48 smithi033 ceph-mon[31973]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:48.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:48 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:48.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:48 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:48.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:48 smithi033 ceph-mon[31973]: mds.? [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] up:boot 2024-03-19T02:50:48.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:48 smithi033 ceph-mon[31973]: fsmap 12 up:standby 2024-03-19T02:50:48.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:48 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-19T02:50:48.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:48 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:48.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:48 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:48.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:48 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:48.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:48 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:50:48.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:48 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2103974887' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:48.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.822+0000 7f8ffdf13640 1 -- 172.21.15.33:0/990260040 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8ff802d440 con 0x7f8ff81034a0 2024-03-19T02:50:48.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.823+0000 7f8ff57fa640 1 -- 172.21.15.33:0/990260040 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+18371 (secure 0 0 0) 0x7f8ff00bdcc0 con 0x7f8ff81034a0 2024-03-19T02:50:48.825 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:50:48.825 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":83,"fsid":"2b7a8938-e59a-11ee-95c9-87774f69a715","created":"2024-03-19T02:42:37.912179+0000","modified":"2024-03-19T02:50:03.016974+0000","last_up_change":"2024-03-19T02:50:01.539392+0000","last_in_change":"2024-03-19T02:49:40.534415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:46:09.963737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"090a89ab-6b7b-4317-8823-178d4212b2f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6803","nonce":1633124995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6805","nonce":1633124995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6809","nonce":1633124995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6807","nonce":1633124995}]},"public_addr":"172.21.15.33:6803/1633124995","cluster_addr":"172.21.15.33:6805/1633124995","heartbeat_back_addr":"172.21.15.33:6809/1633124995","heartbeat_front_addr":"172.21.15.33:6807/1633124995","state":["exists","up"]},{"osd":1,"uuid":"f50eec55-1956-4486-8593-f2c8fa610e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6801","nonce":340144398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6803","nonce":340144398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6807","nonce":340144398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6805","nonce":340144398}]},"public_addr":"172.21.15.96:6801/340144398","cluster_addr":"172.21.15.96:6803/340144398","heartbeat_back_addr":"172.21.15.96:6807/340144398","heartbeat_front_addr":"172.21.15.96:6805/340144398","state":["exists","up"]},{"osd":2,"uuid":"893a2d62-80d9-4e5e-bc6a-a1c8c681d652","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6801","nonce":3388743876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6803","nonce":3388743876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6807","nonce":3388743876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6805","nonce":3388743876}]},"public_addr":"172.21.15.155:6801/3388743876","cluster_addr":"172.21.15.155:6803/3388743876","heartbeat_back_addr":"172.21.15.155:6807/3388743876","heartbeat_front_addr":"172.21.15.155:6805/3388743876","state":["exists","up"]},{"osd":3,"uuid":"b07e0482-4281-4d32-9dfe-9995d7246175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6811","nonce":677007171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6813","nonce":677007171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6817","nonce":677007171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6815","nonce":677007171}]},"public_addr":"172.21.15.33:6811/677007171","cluster_addr":"172.21.15.33:6813/677007171","heartbeat_back_addr":"172.21.15.33:6817/677007171","heartbeat_front_addr":"172.21.15.33:6815/677007171","state":["exists","up"]},{"osd":4,"uuid":"a988384b-af6b-47c6-8c07-5c9673acee45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6809","nonce":3106158554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6811","nonce":3106158554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6815","nonce":3106158554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6813","nonce":3106158554}]},"public_addr":"172.21.15.96:6809/3106158554","cluster_addr":"172.21.15.96:6811/3106158554","heartbeat_back_addr":"172.21.15.96:6815/3106158554","heartbeat_front_addr":"172.21.15.96:6813/3106158554","state":["exists","up"]},{"osd":5,"uuid":"198dc992-96f8-4c52-96ee-1fd5de148bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6809","nonce":327293324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6811","nonce":327293324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6815","nonce":327293324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6813","nonce":327293324}]},"public_addr":"172.21.15.155:6809/327293324","cluster_addr":"172.21.15.155:6811/327293324","heartbeat_back_addr":"172.21.15.155:6815/327293324","heartbeat_front_addr":"172.21.15.155:6813/327293324","state":["exists","up"]},{"osd":6,"uuid":"a7278ec0-ab2e-49ed-be53-84d8b9e24177","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6819","nonce":873871509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6821","nonce":873871509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6825","nonce":873871509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6823","nonce":873871509}]},"public_addr":"172.21.15.33:6819/873871509","cluster_addr":"172.21.15.33:6821/873871509","heartbeat_back_addr":"172.21.15.33:6825/873871509","heartbeat_front_addr":"172.21.15.33:6823/873871509","state":["exists","up"]},{"osd":7,"uuid":"727e133d-aa24-4ffc-90e2-266eb2aab8c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6817","nonce":2334123891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6819","nonce":2334123891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6823","nonce":2334123891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6821","nonce":2334123891}]},"public_addr":"172.21.15.96:6817/2334123891","cluster_addr":"172.21.15.96:6819/2334123891","heartbeat_back_addr":"172.21.15.96:6823/2334123891","heartbeat_front_addr":"172.21.15.96:6821/2334123891","state":["exists","up"]},{"osd":8,"uuid":"bb20eec0-39e5-4b86-9db5-486b1840cdad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6817","nonce":3498516903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6819","nonce":3498516903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6823","nonce":3498516903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6821","nonce":3498516903}]},"public_addr":"172.21.15.155:6817/3498516903","cluster_addr":"172.21.15.155:6819/3498516903","heartbeat_back_addr":"172.21.15.155:6823/3498516903","heartbeat_front_addr":"172.21.15.155:6821/3498516903","state":["exists","up"]},{"osd":9,"uuid":"179c6228-3e23-4f65-91a6-1ef0005d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6827","nonce":760777213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6829","nonce":760777213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6833","nonce":760777213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6831","nonce":760777213}]},"public_addr":"172.21.15.33:6827/760777213","cluster_addr":"172.21.15.33:6829/760777213","heartbeat_back_addr":"172.21.15.33:6833/760777213","heartbeat_front_addr":"172.21.15.33:6831/760777213","state":["exists","up"]},{"osd":10,"uuid":"be17b478-3c2e-41fb-bede-958333acdf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6825","nonce":1404898482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6827","nonce":1404898482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6831","nonce":1404898482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6829","nonce":1404898482}]},"public_addr":"172.21.15.96:6825/1404898482","cluster_addr":"172.21.15.96:6827/1404898482","heartbeat_back_addr":"172.21.15.96:6831/1404898482","heartbeat_front_addr":"172.21.15.96:6829/1404898482","state":["exists","up"]},{"osd":11,"uuid":"c0903ed2-cc1e-4c18-8731-1f0ee4dd0457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6825","nonce":792352827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6827","nonce":792352827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6831","nonce":792352827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6829","nonce":792352827}]},"public_addr":"172.21.15.155:6825/792352827","cluster_addr":"172.21.15.155:6827/792352827","heartbeat_back_addr":"172.21.15.155:6831/792352827","heartbeat_front_addr":"172.21.15.155:6829/792352827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:17.942801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:42.109449+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:05.981599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:32.069574+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:57.431015+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:22.448592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:48.526515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:14.498352+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:40.175426+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:06.364006+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:32.853161+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:59.073538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:6801/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:6800/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/1067768047":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/729379453":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2586588742":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/2932814165":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6801/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2212564963":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6800/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/3275126058":"2024-03-20T02:43:39.622218+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:50:48.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.825+0000 7f8ffdf13640 1 -- 172.21.15.33:0/990260040 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8fe40a2800 msgr2=0x7f8fe40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:48.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.825+0000 7f8ffdf13640 1 --2- 172.21.15.33:0/990260040 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8fe40a2800 0x7f8fe40a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8ff8069c50 tx=0x7f8fe8009110 comp rx=0 tx=0).stop 2024-03-19T02:50:48.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.825+0000 7f8ffdf13640 1 -- 172.21.15.33:0/990260040 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ff81034a0 msgr2=0x7f8ff8100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:48.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.825+0000 7f8ffdf13640 1 --2- 172.21.15.33:0/990260040 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ff81034a0 0x7f8ff8100fd0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8ff002c630 tx=0x7f8ff0032910 comp rx=0 tx=0).stop 2024-03-19T02:50:48.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.825+0000 7f8ffdf13640 1 -- 172.21.15.33:0/990260040 shutdown_connections 2024-03-19T02:50:48.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.825+0000 7f8ffdf13640 1 --2- 172.21.15.33:0/990260040 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8fe40a2800 0x7f8fe40a4cc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:48.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.825+0000 7f8ffdf13640 1 --2- 172.21.15.33:0/990260040 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ff81034a0 0x7f8ff8100fd0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:48.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.825+0000 7f8ffdf13640 1 --2- 172.21.15.33:0/990260040 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ff8102ae0 0x7f8ff8100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:48.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.826+0000 7f8ffdf13640 1 --2- 172.21.15.33:0/990260040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ff80ffb10 0x7f8ff8102450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:48.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.826+0000 7f8ffdf13640 1 -- 172.21.15.33:0/990260040 >> 172.21.15.33:0/990260040 conn(0x7f8ff80fb440 msgr2=0x7f8ff8105f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:48.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.826+0000 7f8ffdf13640 1 -- 172.21.15.33:0/990260040 shutdown_connections 2024-03-19T02:50:48.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.826+0000 7f8ffdf13640 1 -- 172.21.15.33:0/990260040 wait complete. 2024-03-19T02:50:48.837 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:46:09.963737+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T02:50:48.838 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T02:50:48.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.946+0000 7f7269a4b640 1 -- 172.21.15.33:0/4292392931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72641022d0 msgr2=0x7f7264108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:48.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.946+0000 7f7269a4b640 1 --2- 172.21.15.33:0/4292392931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72641022d0 0x7f7264108b60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f725c009ca0 tx=0x7f725c02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:50:48.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.946+0000 7f7269a4b640 1 -- 172.21.15.33:0/4292392931 shutdown_connections 2024-03-19T02:50:48.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.946+0000 7f7269a4b640 1 --2- 172.21.15.33:0/4292392931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f72641022d0 0x7f7264108b60 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:48.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.946+0000 7f7269a4b640 1 --2- 172.21.15.33:0/4292392931 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7264101910 0x7f7264101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:48.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.946+0000 7f7269a4b640 1 --2- 172.21.15.33:0/4292392931 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7264100720 0x7f7264100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:48.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.946+0000 7f7269a4b640 1 -- 172.21.15.33:0/4292392931 >> 172.21.15.33:0/4292392931 conn(0x7f72640fbcb0 msgr2=0x7f72640fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:48.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.946+0000 7f7269a4b640 1 -- 172.21.15.33:0/4292392931 shutdown_connections 2024-03-19T02:50:48.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.946+0000 7f7269a4b640 1 -- 172.21.15.33:0/4292392931 wait complete. 2024-03-19T02:50:48.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.947+0000 7f7269a4b640 1 Processor -- start 2024-03-19T02:50:48.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.947+0000 7f7269a4b640 1 -- start start 2024-03-19T02:50:48.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.947+0000 7f7269a4b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7264100720 0x7f7264196970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:48.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.947+0000 7f7269a4b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7264101910 0x7f7264196eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:48.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.947+0000 7f7269a4b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72641022d0 0x7f726419df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:48.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.947+0000 7f7269a4b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f726410c250 con 0x7f7264100720 2024-03-19T02:50:48.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.947+0000 7f7269a4b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f726410c0d0 con 0x7f7264101910 2024-03-19T02:50:48.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.947+0000 7f7269a4b640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f726410c3d0 con 0x7f72641022d0 2024-03-19T02:50:48.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.947+0000 7f7262ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7264100720 0x7f7264196970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:48.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.947+0000 7f7262ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7264100720 0x7f7264196970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33626/0 (socket says 172.21.15.33:33626) 2024-03-19T02:50:48.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.947+0000 7f7262ffd640 1 -- 172.21.15.33:0/2849068220 learned_addr learned my addr 172.21.15.33:0/2849068220 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:48.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.947+0000 7f72627fc640 1 --2- 172.21.15.33:0/2849068220 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7264101910 0x7f7264196eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:48.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.947+0000 7f72637fe640 1 --2- 172.21.15.33:0/2849068220 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72641022d0 0x7f726419df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:48.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.947+0000 7f7262ffd640 1 -- 172.21.15.33:0/2849068220 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72641022d0 msgr2=0x7f726419df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:48.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.947+0000 7f7262ffd640 1 --2- 172.21.15.33:0/2849068220 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72641022d0 0x7f726419df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:48.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.947+0000 7f7262ffd640 1 -- 172.21.15.33:0/2849068220 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7264101910 msgr2=0x7f7264196eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:48.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.947+0000 7f7262ffd640 1 --2- 172.21.15.33:0/2849068220 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7264101910 0x7f7264196eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:48.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.947+0000 7f7262ffd640 1 -- 172.21.15.33:0/2849068220 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f726419e590 con 0x7f7264100720 2024-03-19T02:50:48.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.948+0000 7f7262ffd640 1 --2- 172.21.15.33:0/2849068220 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7264100720 0x7f7264196970 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f725800e9c0 tx=0x7f725800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:48.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.948+0000 7f7268a49640 1 -- 172.21.15.33:0/2849068220 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725800cd80 con 0x7f7264100720 2024-03-19T02:50:48.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.948+0000 7f7268a49640 1 -- 172.21.15.33:0/2849068220 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72580035f0 con 0x7f7264100720 2024-03-19T02:50:48.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.948+0000 7f7268a49640 1 -- 172.21.15.33:0/2849068220 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7258010690 con 0x7f7264100720 2024-03-19T02:50:48.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.948+0000 7f7269a4b640 1 -- 172.21.15.33:0/2849068220 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f726419e880 con 0x7f7264100720 2024-03-19T02:50:48.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.948+0000 7f7269a4b640 1 -- 172.21.15.33:0/2849068220 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f726419ed80 con 0x7f7264100720 2024-03-19T02:50:48.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.949+0000 7f7269a4b640 1 -- 172.21.15.33:0/2849068220 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7264107b60 con 0x7f7264100720 2024-03-19T02:50:48.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.950+0000 7f7268a49640 1 -- 172.21.15.33:0/2849068220 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f72580037a0 con 0x7f7264100720 2024-03-19T02:50:48.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.950+0000 7f7268a49640 1 --2- 172.21.15.33:0/2849068220 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f724c0a2800 0x7f724c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:48.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.950+0000 7f7268a49640 1 -- 172.21.15.33:0/2849068220 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f72580cec60 con 0x7f7264100720 2024-03-19T02:50:48.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.950+0000 7f72627fc640 1 --2- 172.21.15.33:0/2849068220 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f724c0a2800 0x7f724c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:48.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.951+0000 7f72627fc640 1 --2- 172.21.15.33:0/2849068220 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f724c0a2800 0x7f724c0a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7254005fd0 tx=0x7f7254005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:48.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:48.955+0000 7f7268a49640 1 -- 172.21.15.33:0/2849068220 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f725809c1a0 con 0x7f7264100720 2024-03-19T02:50:49.014 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:48 smithi155 ceph-mon[35804]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:49.015 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:48 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:49.015 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:48 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:49.015 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:48 smithi155 ceph-mon[35804]: mds.? [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] up:boot 2024-03-19T02:50:49.015 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:48 smithi155 ceph-mon[35804]: fsmap 12 up:standby 2024-03-19T02:50:49.015 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:48 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-19T02:50:49.015 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:48 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:49.015 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:48 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:49.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:48 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:49.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:48 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:50:49.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:48 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2103974887' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:49.045 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:48 smithi096 ceph-mon[35946]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:49.045 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:48 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:49.045 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:48 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:49.045 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:48 smithi096 ceph-mon[35946]: mds.? [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] up:boot 2024-03-19T02:50:49.046 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:48 smithi096 ceph-mon[35946]: fsmap 12 up:standby 2024-03-19T02:50:49.046 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:48 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-19T02:50:49.046 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:48 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:49.046 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:48 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:49.046 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:48 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:49.046 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:48 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:50:49.047 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:48 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2103974887' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:49.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.140+0000 7f7269a4b640 1 -- 172.21.15.33:0/2849068220 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7264062450 con 0x7f7264100720 2024-03-19T02:50:49.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.140+0000 7f7268a49640 1 -- 172.21.15.33:0/2849068220 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7f725809baa0 con 0x7f7264100720 2024-03-19T02:50:49.142 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 1 2024-03-19T02:50:49.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.142+0000 7f7269a4b640 1 -- 172.21.15.33:0/2849068220 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f724c0a2800 msgr2=0x7f724c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:49.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.142+0000 7f7269a4b640 1 --2- 172.21.15.33:0/2849068220 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f724c0a2800 0x7f724c0a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7254005fd0 tx=0x7f7254005850 comp rx=0 tx=0).stop 2024-03-19T02:50:49.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.142+0000 7f7269a4b640 1 -- 172.21.15.33:0/2849068220 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7264100720 msgr2=0x7f7264196970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:49.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.142+0000 7f7269a4b640 1 --2- 172.21.15.33:0/2849068220 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7264100720 0x7f7264196970 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f725800e9c0 tx=0x7f725800ee80 comp rx=0 tx=0).stop 2024-03-19T02:50:49.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.142+0000 7f7269a4b640 1 -- 172.21.15.33:0/2849068220 shutdown_connections 2024-03-19T02:50:49.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.142+0000 7f7269a4b640 1 --2- 172.21.15.33:0/2849068220 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f724c0a2800 0x7f724c0a4cc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:49.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.142+0000 7f7269a4b640 1 --2- 172.21.15.33:0/2849068220 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72641022d0 0x7f726419df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:49.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.142+0000 7f7269a4b640 1 --2- 172.21.15.33:0/2849068220 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7264101910 0x7f7264196eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:49.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.142+0000 7f7269a4b640 1 --2- 172.21.15.33:0/2849068220 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7264100720 0x7f7264196970 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:49.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.142+0000 7f7269a4b640 1 -- 172.21.15.33:0/2849068220 >> 172.21.15.33:0/2849068220 conn(0x7f72640fbcb0 msgr2=0x7f7264104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:49.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.142+0000 7f7269a4b640 1 -- 172.21.15.33:0/2849068220 shutdown_connections 2024-03-19T02:50:49.148 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.142+0000 7f7269a4b640 1 -- 172.21.15.33:0/2849068220 wait complete. 2024-03-19T02:50:49.155 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-19T02:50:49.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.257+0000 7fb38e162640 1 -- 172.21.15.33:0/3349972371 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb388102a40 msgr2=0x7fb388102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:49.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.257+0000 7fb38e162640 1 --2- 172.21.15.33:0/3349972371 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb388102a40 0x7fb388102ec0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb378009900 tx=0x7fb37802bc30 comp rx=0 tx=0).stop 2024-03-19T02:50:49.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.257+0000 7fb38e162640 1 -- 172.21.15.33:0/3349972371 shutdown_connections 2024-03-19T02:50:49.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.257+0000 7fb38e162640 1 --2- 172.21.15.33:0/3349972371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb388103400 0x7fb388109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:49.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.257+0000 7fb38e162640 1 --2- 172.21.15.33:0/3349972371 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb388102a40 0x7fb388102ec0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:49.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.257+0000 7fb38e162640 1 --2- 172.21.15.33:0/3349972371 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb388101850 0x7fb388101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:49.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.257+0000 7fb38e162640 1 -- 172.21.15.33:0/3349972371 >> 172.21.15.33:0/3349972371 conn(0x7fb3880fcdc0 msgr2=0x7fb3880ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:49.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.257+0000 7fb38e162640 1 -- 172.21.15.33:0/3349972371 shutdown_connections 2024-03-19T02:50:49.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.257+0000 7fb38e162640 1 -- 172.21.15.33:0/3349972371 wait complete. 2024-03-19T02:50:49.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.258+0000 7fb38e162640 1 Processor -- start 2024-03-19T02:50:49.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.258+0000 7fb38e162640 1 -- start start 2024-03-19T02:50:49.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.258+0000 7fb38e162640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb388101850 0x7fb388197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:49.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.258+0000 7fb38e162640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb388102a40 0x7fb388197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:49.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.258+0000 7fb38e162640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb388103400 0x7fb38819f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:49.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.258+0000 7fb38e162640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb38810d380 con 0x7fb388103400 2024-03-19T02:50:49.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.258+0000 7fb38e162640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb38810d200 con 0x7fb388101850 2024-03-19T02:50:49.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.258+0000 7fb38e162640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb38810d500 con 0x7fb388102a40 2024-03-19T02:50:49.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.258+0000 7fb387fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb388103400 0x7fb38819f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:49.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.258+0000 7fb387fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb388103400 0x7fb38819f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33642/0 (socket says 172.21.15.33:33642) 2024-03-19T02:50:49.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.258+0000 7fb387fff640 1 -- 172.21.15.33:0/3150569864 learned_addr learned my addr 172.21.15.33:0/3150569864 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:49.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.259+0000 7fb386ffd640 1 --2- 172.21.15.33:0/3150569864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb388102a40 0x7fb388197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:49.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.259+0000 7fb3877fe640 1 --2- 172.21.15.33:0/3150569864 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb388101850 0x7fb388197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:49.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.259+0000 7fb387fff640 1 -- 172.21.15.33:0/3150569864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb388102a40 msgr2=0x7fb388197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:49.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.259+0000 7fb387fff640 1 --2- 172.21.15.33:0/3150569864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb388102a40 0x7fb388197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:49.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.259+0000 7fb387fff640 1 -- 172.21.15.33:0/3150569864 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb388101850 msgr2=0x7fb388197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:49.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.259+0000 7fb387fff640 1 --2- 172.21.15.33:0/3150569864 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb388101850 0x7fb388197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:49.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.259+0000 7fb387fff640 1 -- 172.21.15.33:0/3150569864 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb38819f6a0 con 0x7fb388103400 2024-03-19T02:50:49.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.259+0000 7fb387fff640 1 --2- 172.21.15.33:0/3150569864 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb388103400 0x7fb38819f030 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb38000eb10 tx=0x7fb38000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:49.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.259+0000 7fb384ff9640 1 -- 172.21.15.33:0/3150569864 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb380004b20 con 0x7fb388103400 2024-03-19T02:50:49.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.259+0000 7fb384ff9640 1 -- 172.21.15.33:0/3150569864 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb380004cc0 con 0x7fb388103400 2024-03-19T02:50:49.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.259+0000 7fb38e162640 1 -- 172.21.15.33:0/3150569864 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb38819f990 con 0x7fb388103400 2024-03-19T02:50:49.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.259+0000 7fb384ff9640 1 -- 172.21.15.33:0/3150569864 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb380010690 con 0x7fb388103400 2024-03-19T02:50:49.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.259+0000 7fb38e162640 1 -- 172.21.15.33:0/3150569864 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb38819feb0 con 0x7fb388103400 2024-03-19T02:50:49.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.261+0000 7fb38e162640 1 -- 172.21.15.33:0/3150569864 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb388108be0 con 0x7fb388103400 2024-03-19T02:50:49.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.261+0000 7fb384ff9640 1 -- 172.21.15.33:0/3150569864 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb380005020 con 0x7fb388103400 2024-03-19T02:50:49.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.262+0000 7fb384ff9640 1 --2- 172.21.15.33:0/3150569864 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb3700a2800 0x7fb3700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:49.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.262+0000 7fb3877fe640 1 --2- 172.21.15.33:0/3150569864 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb3700a2800 0x7fb3700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:49.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.262+0000 7fb384ff9640 1 -- 172.21.15.33:0/3150569864 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fb3800ced20 con 0x7fb388103400 2024-03-19T02:50:49.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.262+0000 7fb3877fe640 1 --2- 172.21.15.33:0/3150569864 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb3700a2800 0x7fb3700a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb3881028a0 tx=0x7fb37c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:49.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.268+0000 7fb384ff9640 1 -- 172.21.15.33:0/3150569864 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb38009c1c0 con 0x7fb388103400 2024-03-19T02:50:49.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.482+0000 7fb38e162640 1 -- 172.21.15.33:0/3150569864 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fb388062420 con 0x7fb388103400 2024-03-19T02:50:49.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.482+0000 7fb384ff9640 1 -- 172.21.15.33:0/3150569864 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v83) ==== 84+0+65 (secure 0 0 0) 0x7fb38009bac0 con 0x7fb388103400 2024-03-19T02:50:49.482 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:50:49.482 INFO:teuthology.orchestra.run.smithi033.stdout:[ 2024-03-19T02:50:49.482 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T02:50:49.482 INFO:teuthology.orchestra.run.smithi033.stdout: "poolnum": 1, 2024-03-19T02:50:49.483 INFO:teuthology.orchestra.run.smithi033.stdout: "poolname": ".mgr" 2024-03-19T02:50:49.483 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T02:50:49.483 INFO:teuthology.orchestra.run.smithi033.stdout:] 2024-03-19T02:50:49.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.484+0000 7fb38e162640 1 -- 172.21.15.33:0/3150569864 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb3700a2800 msgr2=0x7fb3700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:49.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.484+0000 7fb38e162640 1 --2- 172.21.15.33:0/3150569864 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb3700a2800 0x7fb3700a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb3881028a0 tx=0x7fb37c005f90 comp rx=0 tx=0).stop 2024-03-19T02:50:49.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.484+0000 7fb38e162640 1 -- 172.21.15.33:0/3150569864 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb388103400 msgr2=0x7fb38819f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:49.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.484+0000 7fb38e162640 1 --2- 172.21.15.33:0/3150569864 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb388103400 0x7fb38819f030 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb38000eb10 tx=0x7fb38000c2d0 comp rx=0 tx=0).stop 2024-03-19T02:50:49.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.484+0000 7fb38e162640 1 -- 172.21.15.33:0/3150569864 shutdown_connections 2024-03-19T02:50:49.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.484+0000 7fb38e162640 1 --2- 172.21.15.33:0/3150569864 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb3700a2800 0x7fb3700a4cc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:49.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.484+0000 7fb38e162640 1 --2- 172.21.15.33:0/3150569864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb388102a40 0x7fb388197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:49.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.484+0000 7fb38e162640 1 --2- 172.21.15.33:0/3150569864 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb388103400 0x7fb38819f030 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:49.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.484+0000 7fb38e162640 1 --2- 172.21.15.33:0/3150569864 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb388101850 0x7fb388197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:49.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.484+0000 7fb38e162640 1 -- 172.21.15.33:0/3150569864 >> 172.21.15.33:0/3150569864 conn(0x7fb3880fcdc0 msgr2=0x7fb388105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:49.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.484+0000 7fb38e162640 1 -- 172.21.15.33:0/3150569864 shutdown_connections 2024-03-19T02:50:49.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.484+0000 7fb38e162640 1 -- 172.21.15.33:0/3150569864 wait complete. 2024-03-19T02:50:49.497 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-19T02:50:49.497 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-19T02:50:49.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.611+0000 7f62e4590640 1 -- 172.21.15.33:0/1902225779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62dc1009a0 msgr2=0x7f62dc100e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:49.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.611+0000 7f62e4590640 1 --2- 172.21.15.33:0/1902225779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62dc1009a0 0x7f62dc100e20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f62d8009900 tx=0x7f62d802bc30 comp rx=0 tx=0).stop 2024-03-19T02:50:49.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.611+0000 7f62e4590640 1 -- 172.21.15.33:0/1902225779 shutdown_connections 2024-03-19T02:50:49.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.611+0000 7f62e4590640 1 --2- 172.21.15.33:0/1902225779 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62dc101360 0x7f62dc108c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:49.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.611+0000 7f62e4590640 1 --2- 172.21.15.33:0/1902225779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62dc1009a0 0x7f62dc100e20 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:49.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.611+0000 7f62e4590640 1 --2- 172.21.15.33:0/1902225779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62dc101d50 0x7f62dc102170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:49.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.611+0000 7f62e4590640 1 -- 172.21.15.33:0/1902225779 >> 172.21.15.33:0/1902225779 conn(0x7f62dc0fb450 msgr2=0x7f62dc0fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:49.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.611+0000 7f62e4590640 1 -- 172.21.15.33:0/1902225779 shutdown_connections 2024-03-19T02:50:49.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.611+0000 7f62e4590640 1 -- 172.21.15.33:0/1902225779 wait complete. 2024-03-19T02:50:49.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.612+0000 7f62e4590640 1 Processor -- start 2024-03-19T02:50:49.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.612+0000 7f62e4590640 1 -- start start 2024-03-19T02:50:49.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.612+0000 7f62e4590640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62dc1009a0 0x7f62dc104940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:49.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.612+0000 7f62e4590640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62dc101360 0x7f62dc102f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:49.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.612+0000 7f62e4590640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62dc101d50 0x7f62dc1034c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:49.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.612+0000 7f62e4590640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f62dc10c290 con 0x7f62dc101d50 2024-03-19T02:50:49.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.612+0000 7f62e4590640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f62dc10c110 con 0x7f62dc1009a0 2024-03-19T02:50:49.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.612+0000 7f62e4590640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f62dc10c410 con 0x7f62dc101360 2024-03-19T02:50:49.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.612+0000 7f62e2b2d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62dc101d50 0x7f62dc1034c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:49.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.612+0000 7f62e2b2d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62dc101d50 0x7f62dc1034c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33660/0 (socket says 172.21.15.33:33660) 2024-03-19T02:50:49.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.612+0000 7f62e2b2d640 1 -- 172.21.15.33:0/729620059 learned_addr learned my addr 172.21.15.33:0/729620059 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:49.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.612+0000 7f62e232c640 1 --2- 172.21.15.33:0/729620059 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62dc1009a0 0x7f62dc104940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:49.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.613+0000 7f62e1b2b640 1 --2- 172.21.15.33:0/729620059 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62dc101360 0x7f62dc102f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:49.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.613+0000 7f62e2b2d640 1 -- 172.21.15.33:0/729620059 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62dc101360 msgr2=0x7f62dc102f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:49.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.613+0000 7f62e2b2d640 1 --2- 172.21.15.33:0/729620059 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62dc101360 0x7f62dc102f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:49.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.613+0000 7f62e2b2d640 1 -- 172.21.15.33:0/729620059 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62dc1009a0 msgr2=0x7f62dc104940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:49.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.613+0000 7f62e2b2d640 1 --2- 172.21.15.33:0/729620059 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62dc1009a0 0x7f62dc104940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:49.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.613+0000 7f62e2b2d640 1 -- 172.21.15.33:0/729620059 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62dc103d50 con 0x7f62dc101d50 2024-03-19T02:50:49.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.613+0000 7f62e2b2d640 1 --2- 172.21.15.33:0/729620059 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62dc101d50 0x7f62dc1034c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f62d400eaa0 tx=0x7f62d400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:49.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.613+0000 7f62d37fe640 1 -- 172.21.15.33:0/729620059 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d400cdb0 con 0x7f62dc101d50 2024-03-19T02:50:49.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.613+0000 7f62e4590640 1 -- 172.21.15.33:0/729620059 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62dc1a1bb0 con 0x7f62dc101d50 2024-03-19T02:50:49.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.613+0000 7f62d37fe640 1 -- 172.21.15.33:0/729620059 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62d4005020 con 0x7f62dc101d50 2024-03-19T02:50:49.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.613+0000 7f62d37fe640 1 -- 172.21.15.33:0/729620059 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d40116d0 con 0x7f62dc101d50 2024-03-19T02:50:49.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.613+0000 7f62e4590640 1 -- 172.21.15.33:0/729620059 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62dc1a20d0 con 0x7f62dc101d50 2024-03-19T02:50:49.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.614+0000 7f62e4590640 1 -- 172.21.15.33:0/729620059 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62dc068a60 con 0x7f62dc101d50 2024-03-19T02:50:49.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.615+0000 7f62d37fe640 1 -- 172.21.15.33:0/729620059 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f62d40051d0 con 0x7f62dc101d50 2024-03-19T02:50:49.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.616+0000 7f62d37fe640 1 --2- 172.21.15.33:0/729620059 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f62c80a2800 0x7f62c80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:49.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.616+0000 7f62e232c640 1 --2- 172.21.15.33:0/729620059 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f62c80a2800 0x7f62c80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:49.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.616+0000 7f62d37fe640 1 -- 172.21.15.33:0/729620059 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f62d40ceae0 con 0x7f62dc101d50 2024-03-19T02:50:49.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.616+0000 7f62e232c640 1 --2- 172.21.15.33:0/729620059 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f62c80a2800 0x7f62c80a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f62dc069c50 tx=0x7f62cc00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:49.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.620+0000 7f62d37fe640 1 -- 172.21.15.33:0/729620059 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f62d4016030 con 0x7f62dc101d50 2024-03-19T02:50:49.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:49 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/990260040' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:50:49.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:49 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2849068220' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:50:49.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:49 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3150569864' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-19T02:50:49.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:49 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:49.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:49 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:49.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:49 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:49.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:49 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:49.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:49 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:49.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:49.821+0000 7f62e4590640 1 -- 172.21.15.33:0/729620059 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f62dc1a2550 con 0x7f62dc101d50 2024-03-19T02:50:50.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:49 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/990260040' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:50:50.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:49 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2849068220' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:50:50.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:49 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3150569864' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-19T02:50:50.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:49 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:50.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:49 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:50.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:49 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:50.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:49 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:50.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:49 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:50.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:49 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/990260040' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:50:50.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:49 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2849068220' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:50:50.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:49 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3150569864' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-19T02:50:50.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:49 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:50.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:49 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:50.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:49 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:50.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:49 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:50.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:49 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:50.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.771+0000 7f62d37fe640 1 -- 172.21.15.33:0/729620059 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v84) ==== 138+0+0 (secure 0 0 0) 0x7f62d409b9d0 con 0x7f62dc101d50 2024-03-19T02:50:50.771 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'cephfs_metadata' created 2024-03-19T02:50:50.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.773+0000 7f62e4590640 1 -- 172.21.15.33:0/729620059 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f62c80a2800 msgr2=0x7f62c80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:50.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.773+0000 7f62e4590640 1 --2- 172.21.15.33:0/729620059 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f62c80a2800 0x7f62c80a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f62dc069c50 tx=0x7f62cc00b390 comp rx=0 tx=0).stop 2024-03-19T02:50:50.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.773+0000 7f62e4590640 1 -- 172.21.15.33:0/729620059 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62dc101d50 msgr2=0x7f62dc1034c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:50.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.773+0000 7f62e4590640 1 --2- 172.21.15.33:0/729620059 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62dc101d50 0x7f62dc1034c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f62d400eaa0 tx=0x7f62d400ef60 comp rx=0 tx=0).stop 2024-03-19T02:50:50.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.773+0000 7f62e4590640 1 -- 172.21.15.33:0/729620059 shutdown_connections 2024-03-19T02:50:50.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.773+0000 7f62e4590640 1 --2- 172.21.15.33:0/729620059 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f62c80a2800 0x7f62c80a4cc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:50.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.773+0000 7f62e4590640 1 --2- 172.21.15.33:0/729620059 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62dc101360 0x7f62dc102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:50.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.773+0000 7f62e4590640 1 --2- 172.21.15.33:0/729620059 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62dc101d50 0x7f62dc1034c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:50.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.773+0000 7f62e4590640 1 --2- 172.21.15.33:0/729620059 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62dc1009a0 0x7f62dc104940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:50.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.773+0000 7f62e4590640 1 -- 172.21.15.33:0/729620059 >> 172.21.15.33:0/729620059 conn(0x7f62dc0fb450 msgr2=0x7f62dc0ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:50.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.774+0000 7f62e4590640 1 -- 172.21.15.33:0/729620059 shutdown_connections 2024-03-19T02:50:50.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.774+0000 7f62e4590640 1 -- 172.21.15.33:0/729620059 wait complete. 2024-03-19T02:50:50.788 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-19T02:50:50.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.898+0000 7f8bdabc4640 1 -- 172.21.15.33:0/2489510237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bd41022d0 msgr2=0x7f8bd4108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:50.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.898+0000 7f8bdabc4640 1 --2- 172.21.15.33:0/2489510237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bd41022d0 0x7f8bd4108b60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4009900 tx=0x7f8bc402bc30 comp rx=0 tx=0).stop 2024-03-19T02:50:50.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.898+0000 7f8bdabc4640 1 -- 172.21.15.33:0/2489510237 shutdown_connections 2024-03-19T02:50:50.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.898+0000 7f8bdabc4640 1 --2- 172.21.15.33:0/2489510237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bd4101910 0x7f8bd4101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:50.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.898+0000 7f8bdabc4640 1 --2- 172.21.15.33:0/2489510237 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bd41022d0 0x7f8bd4108b60 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:50.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.898+0000 7f8bdabc4640 1 --2- 172.21.15.33:0/2489510237 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bd4100720 0x7f8bd4100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:50.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.898+0000 7f8bdabc4640 1 -- 172.21.15.33:0/2489510237 >> 172.21.15.33:0/2489510237 conn(0x7f8bd40fbcb0 msgr2=0x7f8bd40fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:50.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.899+0000 7f8bdabc4640 1 -- 172.21.15.33:0/2489510237 shutdown_connections 2024-03-19T02:50:50.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.899+0000 7f8bdabc4640 1 -- 172.21.15.33:0/2489510237 wait complete. 2024-03-19T02:50:50.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.899+0000 7f8bdabc4640 1 Processor -- start 2024-03-19T02:50:50.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.899+0000 7f8bdabc4640 1 -- start start 2024-03-19T02:50:50.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.899+0000 7f8bdabc4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bd4100720 0x7f8bd4197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:50.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.899+0000 7f8bdabc4640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bd4101910 0x7f8bd4197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:50.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.899+0000 7f8bdabc4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bd41022d0 0x7f8bd419f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:50.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.899+0000 7f8bdabc4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd410c230 con 0x7f8bd4100720 2024-03-19T02:50:50.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.899+0000 7f8bdabc4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd410c0b0 con 0x7f8bd41022d0 2024-03-19T02:50:50.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.899+0000 7f8bdabc4640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd410c3b0 con 0x7f8bd4101910 2024-03-19T02:50:50.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.899+0000 7f8bd8960640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bd4100720 0x7f8bd4197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:50.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.899+0000 7f8bd8960640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bd4100720 0x7f8bd4197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33678/0 (socket says 172.21.15.33:33678) 2024-03-19T02:50:50.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.899+0000 7f8bd8960640 1 -- 172.21.15.33:0/3468970048 learned_addr learned my addr 172.21.15.33:0/3468970048 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:50.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.899+0000 7f8bd3fff640 1 --2- 172.21.15.33:0/3468970048 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bd4101910 0x7f8bd4197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:50.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.899+0000 7f8bd9161640 1 --2- 172.21.15.33:0/3468970048 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bd41022d0 0x7f8bd419f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:50.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.900+0000 7f8bd8960640 1 -- 172.21.15.33:0/3468970048 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bd4101910 msgr2=0x7f8bd4197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:50.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.900+0000 7f8bd8960640 1 --2- 172.21.15.33:0/3468970048 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bd4101910 0x7f8bd4197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:50.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.900+0000 7f8bd8960640 1 -- 172.21.15.33:0/3468970048 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bd41022d0 msgr2=0x7f8bd419f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:50.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.900+0000 7f8bd8960640 1 --2- 172.21.15.33:0/3468970048 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bd41022d0 0x7f8bd419f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:50.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.900+0000 7f8bd8960640 1 -- 172.21.15.33:0/3468970048 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd419f6a0 con 0x7f8bd4100720 2024-03-19T02:50:50.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.900+0000 7f8bd8960640 1 --2- 172.21.15.33:0/3468970048 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bd4100720 0x7f8bd4197a80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc00ebc0 tx=0x7f8bcc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:50.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.900+0000 7f8bd1ffb640 1 -- 172.21.15.33:0/3468970048 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc003b70 con 0x7f8bd4100720 2024-03-19T02:50:50.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.900+0000 7f8bd1ffb640 1 -- 172.21.15.33:0/3468970048 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8bcc003d10 con 0x7f8bd4100720 2024-03-19T02:50:50.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.900+0000 7f8bdabc4640 1 -- 172.21.15.33:0/3468970048 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bd419f990 con 0x7f8bd4100720 2024-03-19T02:50:50.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.900+0000 7f8bd1ffb640 1 -- 172.21.15.33:0/3468970048 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc019840 con 0x7f8bd4100720 2024-03-19T02:50:50.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.900+0000 7f8bdabc4640 1 -- 172.21.15.33:0/3468970048 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bd419fee0 con 0x7f8bd4100720 2024-03-19T02:50:50.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.901+0000 7f8bdabc4640 1 -- 172.21.15.33:0/3468970048 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bd41a0170 con 0x7f8bd4100720 2024-03-19T02:50:50.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.903+0000 7f8bd1ffb640 1 -- 172.21.15.33:0/3468970048 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f8bcc0199e0 con 0x7f8bd4100720 2024-03-19T02:50:50.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.903+0000 7f8bd1ffb640 1 --2- 172.21.15.33:0/3468970048 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8bbc0a2800 0x7f8bbc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:50.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.903+0000 7f8bd1ffb640 1 -- 172.21.15.33:0/3468970048 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7497+0+0 (secure 0 0 0) 0x7f8bcc0cf170 con 0x7f8bd4100720 2024-03-19T02:50:50.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.903+0000 7f8bd3fff640 1 --2- 172.21.15.33:0/3468970048 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8bbc0a2800 0x7f8bbc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:50.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.904+0000 7f8bd3fff640 1 --2- 172.21.15.33:0/3468970048 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8bbc0a2800 0x7f8bbc0a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4198f90 tx=0x7f8bc8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:50.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:50.907+0000 7f8bd1ffb640 1 -- 172.21.15.33:0/3468970048 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8bcc09c530 con 0x7f8bd4100720 2024-03-19T02:50:51.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:50 smithi155 ceph-mon[35804]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:51.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:50 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:51.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:50 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:51.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:50 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/729620059' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-19T02:50:51.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:50 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:51.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:50 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:50:51.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:50 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:51.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:50 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:50:51.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:50 smithi033 ceph-mon[31973]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:51.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:50 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:51.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:50 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:51.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:50 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/729620059' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-19T02:50:51.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:50 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:51.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:50 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:50:51.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:50 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:51.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:50 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:50:51.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.135+0000 7f8bdabc4640 1 -- 172.21.15.33:0/3468970048 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f8bd41a0340 con 0x7f8bd4100720 2024-03-19T02:50:51.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:50 smithi096 ceph-mon[35946]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:51.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:50 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:51.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:50 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:51.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:50 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/729620059' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-19T02:50:51.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:50 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:51.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:50 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:50:51.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:50 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:51.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:50 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:50:51.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.778+0000 7f8bd1ffb640 1 -- 172.21.15.33:0/3468970048 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v85) ==== 171+0+0 (secure 0 0 0) 0x7f8bcc09be30 con 0x7f8bd4100720 2024-03-19T02:50:51.778 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'cephfs_data' created 2024-03-19T02:50:51.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.781+0000 7f8bdabc4640 1 -- 172.21.15.33:0/3468970048 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8bbc0a2800 msgr2=0x7f8bbc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:51.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.781+0000 7f8bdabc4640 1 --2- 172.21.15.33:0/3468970048 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8bbc0a2800 0x7f8bbc0a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4198f90 tx=0x7f8bc8005f90 comp rx=0 tx=0).stop 2024-03-19T02:50:51.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.781+0000 7f8bdabc4640 1 -- 172.21.15.33:0/3468970048 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bd4100720 msgr2=0x7f8bd4197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:51.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.781+0000 7f8bdabc4640 1 --2- 172.21.15.33:0/3468970048 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bd4100720 0x7f8bd4197a80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc00ebc0 tx=0x7f8bcc00c6a0 comp rx=0 tx=0).stop 2024-03-19T02:50:51.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.782+0000 7f8bdabc4640 1 -- 172.21.15.33:0/3468970048 shutdown_connections 2024-03-19T02:50:51.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.782+0000 7f8bdabc4640 1 --2- 172.21.15.33:0/3468970048 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8bbc0a2800 0x7f8bbc0a4cc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:51.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.782+0000 7f8bdabc4640 1 --2- 172.21.15.33:0/3468970048 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bd4101910 0x7f8bd4197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:51.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.782+0000 7f8bdabc4640 1 --2- 172.21.15.33:0/3468970048 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bd41022d0 0x7f8bd419f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:51.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.782+0000 7f8bdabc4640 1 --2- 172.21.15.33:0/3468970048 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bd4100720 0x7f8bd4197a80 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:51.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.782+0000 7f8bdabc4640 1 -- 172.21.15.33:0/3468970048 >> 172.21.15.33:0/3468970048 conn(0x7f8bd40fbcb0 msgr2=0x7f8bd40fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:51.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.784+0000 7f8bdabc4640 1 -- 172.21.15.33:0/3468970048 shutdown_connections 2024-03-19T02:50:51.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.789+0000 7f8bdabc4640 1 -- 172.21.15.33:0/3468970048 wait complete. 2024-03-19T02:50:51.814 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-19T02:50:51.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.926+0000 7f91d3ade640 1 -- 172.21.15.33:0/4260610282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91cc1022f0 msgr2=0x7f91cc108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:51.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.926+0000 7f91d3ade640 1 --2- 172.21.15.33:0/4260610282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91cc1022f0 0x7f91cc108b80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f91c4009ca0 tx=0x7f91c402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:50:51.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.926+0000 7f91d3ade640 1 -- 172.21.15.33:0/4260610282 shutdown_connections 2024-03-19T02:50:51.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.926+0000 7f91d3ade640 1 --2- 172.21.15.33:0/4260610282 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91cc1022f0 0x7f91cc108b80 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:51.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.926+0000 7f91d3ade640 1 --2- 172.21.15.33:0/4260610282 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91cc101930 0x7f91cc101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:51.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.926+0000 7f91d3ade640 1 --2- 172.21.15.33:0/4260610282 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f91cc100740 0x7f91cc100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:51.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.926+0000 7f91d3ade640 1 -- 172.21.15.33:0/4260610282 >> 172.21.15.33:0/4260610282 conn(0x7f91cc0fbcd0 msgr2=0x7f91cc0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:51.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.926+0000 7f91d3ade640 1 -- 172.21.15.33:0/4260610282 shutdown_connections 2024-03-19T02:50:51.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.926+0000 7f91d3ade640 1 -- 172.21.15.33:0/4260610282 wait complete. 2024-03-19T02:50:51.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.927+0000 7f91d3ade640 1 Processor -- start 2024-03-19T02:50:51.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.927+0000 7f91d3ade640 1 -- start start 2024-03-19T02:50:51.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.927+0000 7f91d3ade640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91cc100740 0x7f91cc196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:51.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.927+0000 7f91d3ade640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f91cc101930 0x7f91cc196e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:51.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.927+0000 7f91d3ade640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91cc1022f0 0x7f91cc19ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:51.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.927+0000 7f91d3ade640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f91cc10c270 con 0x7f91cc1022f0 2024-03-19T02:50:51.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.927+0000 7f91d3ade640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f91cc10c0f0 con 0x7f91cc100740 2024-03-19T02:50:51.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.927+0000 7f91d3ade640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f91cc10c3f0 con 0x7f91cc101930 2024-03-19T02:50:51.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.927+0000 7f91d207b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91cc1022f0 0x7f91cc19ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:51.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.927+0000 7f91d207b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91cc1022f0 0x7f91cc19ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33694/0 (socket says 172.21.15.33:33694) 2024-03-19T02:50:51.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.927+0000 7f91d207b640 1 -- 172.21.15.33:0/3412016773 learned_addr learned my addr 172.21.15.33:0/3412016773 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:51.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.927+0000 7f91d187a640 1 --2- 172.21.15.33:0/3412016773 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91cc100740 0x7f91cc196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:51.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.927+0000 7f91d1079640 1 --2- 172.21.15.33:0/3412016773 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f91cc101930 0x7f91cc196e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:51.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.927+0000 7f91d207b640 1 -- 172.21.15.33:0/3412016773 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f91cc101930 msgr2=0x7f91cc196e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:51.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.927+0000 7f91d207b640 1 --2- 172.21.15.33:0/3412016773 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f91cc101930 0x7f91cc196e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:51.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.927+0000 7f91d207b640 1 -- 172.21.15.33:0/3412016773 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91cc100740 msgr2=0x7f91cc196920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:51.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.927+0000 7f91d207b640 1 --2- 172.21.15.33:0/3412016773 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91cc100740 0x7f91cc196920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:51.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.927+0000 7f91d207b640 1 -- 172.21.15.33:0/3412016773 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91cc19e540 con 0x7f91cc1022f0 2024-03-19T02:50:51.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.927+0000 7f91d187a640 1 --2- 172.21.15.33:0/3412016773 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91cc100740 0x7f91cc196920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:50:51.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.928+0000 7f91d207b640 1 --2- 172.21.15.33:0/3412016773 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91cc1022f0 0x7f91cc19ded0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f91c402c700 tx=0x7f91c40328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:51.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.928+0000 7f91c2ffd640 1 -- 172.21.15.33:0/3412016773 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c4041020 con 0x7f91cc1022f0 2024-03-19T02:50:51.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.928+0000 7f91c2ffd640 1 -- 172.21.15.33:0/3412016773 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f91c4003e20 con 0x7f91cc1022f0 2024-03-19T02:50:51.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.928+0000 7f91d3ade640 1 -- 172.21.15.33:0/3412016773 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91cc19e7d0 con 0x7f91cc1022f0 2024-03-19T02:50:51.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.928+0000 7f91c2ffd640 1 -- 172.21.15.33:0/3412016773 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c4039040 con 0x7f91cc1022f0 2024-03-19T02:50:51.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.928+0000 7f91d3ade640 1 -- 172.21.15.33:0/3412016773 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91cc19ecc0 con 0x7f91cc1022f0 2024-03-19T02:50:51.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.929+0000 7f91d3ade640 1 -- 172.21.15.33:0/3412016773 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91cc107ae0 con 0x7f91cc1022f0 2024-03-19T02:50:51.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.930+0000 7f91c2ffd640 1 -- 172.21.15.33:0/3412016773 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f91c4044020 con 0x7f91cc1022f0 2024-03-19T02:50:51.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.931+0000 7f91c2ffd640 1 --2- 172.21.15.33:0/3412016773 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f91b80a2800 0x7f91b80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:51.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.931+0000 7f91d187a640 1 --2- 172.21.15.33:0/3412016773 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f91b80a2800 0x7f91b80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:51.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.931+0000 7f91c2ffd640 1 -- 172.21.15.33:0/3412016773 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7879+0+0 (secure 0 0 0) 0x7f91c40f0bf0 con 0x7f91cc1022f0 2024-03-19T02:50:51.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.931+0000 7f91d187a640 1 --2- 172.21.15.33:0/3412016773 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f91b80a2800 0x7f91b80a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f91c8005fd0 tx=0x7f91c800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:51.936 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:51.936+0000 7f91c2ffd640 1 -- 172.21.15.33:0/3412016773 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f91c404b020 con 0x7f91cc1022f0 2024-03-19T02:50:51.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:51 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/729620059' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-19T02:50:51.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:51 smithi096 ceph-mon[35946]: osdmap e84: 12 total, 12 up, 12 in 2024-03-19T02:50:51.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:51 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3468970048' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-19T02:50:51.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:51 smithi096 ceph-mon[35946]: pgmap v290: 33 pgs: 9 creating+peering, 23 unknown, 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:52.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:52.201+0000 7f91d3ade640 1 -- 172.21.15.33:0/3412016773 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f91cc062420 con 0x7f91cc1022f0 2024-03-19T02:50:52.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:51 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/729620059' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-19T02:50:52.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:51 smithi155 ceph-mon[35804]: osdmap e84: 12 total, 12 up, 12 in 2024-03-19T02:50:52.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:51 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3468970048' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-19T02:50:52.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:51 smithi155 ceph-mon[35804]: pgmap v290: 33 pgs: 9 creating+peering, 23 unknown, 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:52.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:51 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/729620059' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-19T02:50:52.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:51 smithi033 ceph-mon[31973]: osdmap e84: 12 total, 12 up, 12 in 2024-03-19T02:50:52.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:51 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3468970048' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-19T02:50:52.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:51 smithi033 ceph-mon[31973]: pgmap v290: 33 pgs: 9 creating+peering, 23 unknown, 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:50:53.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.197+0000 7f91c2ffd640 1 -- 172.21.15.33:0/3412016773 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f91c40bdda0 con 0x7f91cc1022f0 2024-03-19T02:50:53.199 INFO:teuthology.orchestra.run.smithi033.stderr:new fs with metadata pool 2 and data pool 3 2024-03-19T02:50:53.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.200+0000 7f91d3ade640 1 -- 172.21.15.33:0/3412016773 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f91b80a2800 msgr2=0x7f91b80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:53.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.200+0000 7f91d3ade640 1 --2- 172.21.15.33:0/3412016773 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f91b80a2800 0x7f91b80a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f91c8005fd0 tx=0x7f91c800b390 comp rx=0 tx=0).stop 2024-03-19T02:50:53.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.200+0000 7f91d3ade640 1 -- 172.21.15.33:0/3412016773 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91cc1022f0 msgr2=0x7f91cc19ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:53.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.200+0000 7f91d3ade640 1 --2- 172.21.15.33:0/3412016773 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91cc1022f0 0x7f91cc19ded0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f91c402c700 tx=0x7f91c40328a0 comp rx=0 tx=0).stop 2024-03-19T02:50:53.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.200+0000 7f91d3ade640 1 -- 172.21.15.33:0/3412016773 shutdown_connections 2024-03-19T02:50:53.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.200+0000 7f91d3ade640 1 --2- 172.21.15.33:0/3412016773 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f91b80a2800 0x7f91b80a4cc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:53.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.200+0000 7f91d3ade640 1 --2- 172.21.15.33:0/3412016773 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f91cc101930 0x7f91cc196e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:53.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.200+0000 7f91d3ade640 1 --2- 172.21.15.33:0/3412016773 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91cc1022f0 0x7f91cc19ded0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:53.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.200+0000 7f91d3ade640 1 --2- 172.21.15.33:0/3412016773 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91cc100740 0x7f91cc196920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:53.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.200+0000 7f91d3ade640 1 -- 172.21.15.33:0/3412016773 >> 172.21.15.33:0/3412016773 conn(0x7f91cc0fbcd0 msgr2=0x7f91cc104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:53.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.200+0000 7f91d3ade640 1 -- 172.21.15.33:0/3412016773 shutdown_connections 2024-03-19T02:50:53.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.200+0000 7f91d3ade640 1 -- 172.21.15.33:0/3412016773 wait complete. 2024-03-19T02:50:53.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:52 smithi096 ceph-mon[35946]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-19T02:50:53.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:52 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3468970048' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-19T02:50:53.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:52 smithi096 ceph-mon[35946]: osdmap e85: 12 total, 12 up, 12 in 2024-03-19T02:50:53.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:52 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:53.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:52 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:53.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:52 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:53.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:52 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3412016773' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-19T02:50:53.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:52 smithi096 ceph-mon[35946]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-19T02:50:53.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:52 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:53.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:52 smithi096 ceph-mon[35946]: osdmap e86: 12 total, 12 up, 12 in 2024-03-19T02:50:53.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:52 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:53.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:52 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:53.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:52 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:53.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:52 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:50:53.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:52 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:53.217 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:52 smithi033 ceph-mon[31973]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-19T02:50:53.217 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:52 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3468970048' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-19T02:50:53.217 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:52 smithi033 ceph-mon[31973]: osdmap e85: 12 total, 12 up, 12 in 2024-03-19T02:50:53.218 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:52 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:53.218 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:52 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:53.218 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:52 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:53.218 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:52 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3412016773' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-19T02:50:53.218 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:52 smithi033 ceph-mon[31973]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-19T02:50:53.218 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:52 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:53.218 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:52 smithi033 ceph-mon[31973]: osdmap e86: 12 total, 12 up, 12 in 2024-03-19T02:50:53.218 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:52 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:53.218 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:52 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:53.218 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:52 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:53.218 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:52 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:50:53.218 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:52 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:53.219 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-19T02:50:53.219 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-19T02:50:53.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:52 smithi155 ceph-mon[35804]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-19T02:50:53.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:52 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3468970048' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-19T02:50:53.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:52 smithi155 ceph-mon[35804]: osdmap e85: 12 total, 12 up, 12 in 2024-03-19T02:50:53.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:52 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:53.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:52 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:53.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:52 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:53.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:52 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3412016773' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-19T02:50:53.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:52 smithi155 ceph-mon[35804]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-19T02:50:53.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:52 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:53.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:52 smithi155 ceph-mon[35804]: osdmap e86: 12 total, 12 up, 12 in 2024-03-19T02:50:53.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:52 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:53.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:52 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:53.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:52 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:50:53.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:52 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:50:53.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:52 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:50:53.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.332+0000 7fe015e80640 1 -- 172.21.15.33:0/1939955028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe010101920 msgr2=0x7fe010101da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:53.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.332+0000 7fe015e80640 1 --2- 172.21.15.33:0/1939955028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe010101920 0x7fe010101da0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe004009900 tx=0x7fe00402bc30 comp rx=0 tx=0).stop 2024-03-19T02:50:53.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.332+0000 7fe015e80640 1 -- 172.21.15.33:0/1939955028 shutdown_connections 2024-03-19T02:50:53.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.332+0000 7fe015e80640 1 --2- 172.21.15.33:0/1939955028 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0101022e0 0x7fe010108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:53.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.332+0000 7fe015e80640 1 --2- 172.21.15.33:0/1939955028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe010101920 0x7fe010101da0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:53.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.332+0000 7fe015e80640 1 --2- 172.21.15.33:0/1939955028 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe010100730 0x7fe010100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:53.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.332+0000 7fe015e80640 1 -- 172.21.15.33:0/1939955028 >> 172.21.15.33:0/1939955028 conn(0x7fe0100fbcc0 msgr2=0x7fe0100fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:53.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.332+0000 7fe015e80640 1 -- 172.21.15.33:0/1939955028 shutdown_connections 2024-03-19T02:50:53.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.332+0000 7fe015e80640 1 -- 172.21.15.33:0/1939955028 wait complete. 2024-03-19T02:50:53.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.332+0000 7fe015e80640 1 Processor -- start 2024-03-19T02:50:53.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.333+0000 7fe015e80640 1 -- start start 2024-03-19T02:50:53.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.333+0000 7fe015e80640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe010100730 0x7fe01006a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:53.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.333+0000 7fe015e80640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe010101920 0x7fe01019c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:53.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.333+0000 7fe015e80640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0101022e0 0x7fe01019ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:53.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.333+0000 7fe015e80640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe01010c2d0 con 0x7fe0101022e0 2024-03-19T02:50:53.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.333+0000 7fe015e80640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe01010c150 con 0x7fe010100730 2024-03-19T02:50:53.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.333+0000 7fe015e80640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe01010c450 con 0x7fe010101920 2024-03-19T02:50:53.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.333+0000 7fe00ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0101022e0 0x7fe01019ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:53.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.333+0000 7fe00ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0101022e0 0x7fe01019ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33718/0 (socket says 172.21.15.33:33718) 2024-03-19T02:50:53.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.333+0000 7fe00ffff640 1 -- 172.21.15.33:0/1426264497 learned_addr learned my addr 172.21.15.33:0/1426264497 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:53.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.333+0000 7fe00effd640 1 --2- 172.21.15.33:0/1426264497 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe010101920 0x7fe01019c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:53.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.333+0000 7fe00f7fe640 1 --2- 172.21.15.33:0/1426264497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe010100730 0x7fe01006a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:53.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.333+0000 7fe00ffff640 1 -- 172.21.15.33:0/1426264497 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe010101920 msgr2=0x7fe01019c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:53.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.333+0000 7fe00ffff640 1 --2- 172.21.15.33:0/1426264497 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe010101920 0x7fe01019c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:53.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.333+0000 7fe00ffff640 1 -- 172.21.15.33:0/1426264497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe010100730 msgr2=0x7fe01006a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:53.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.333+0000 7fe00ffff640 1 --2- 172.21.15.33:0/1426264497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe010100730 0x7fe01006a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:53.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.333+0000 7fe00ffff640 1 -- 172.21.15.33:0/1426264497 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe01019f6f0 con 0x7fe0101022e0 2024-03-19T02:50:53.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.333+0000 7fe00effd640 1 --2- 172.21.15.33:0/1426264497 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe010101920 0x7fe01019c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:50:53.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.333+0000 7fe00ffff640 1 --2- 172.21.15.33:0/1426264497 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0101022e0 0x7fe01019ee30 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe00800cce0 tx=0x7fe00800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:53.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.333+0000 7fe00cff9640 1 -- 172.21.15.33:0/1426264497 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe00800f450 con 0x7fe0101022e0 2024-03-19T02:50:53.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.333+0000 7fe00cff9640 1 -- 172.21.15.33:0/1426264497 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe00800faa0 con 0x7fe0101022e0 2024-03-19T02:50:53.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.333+0000 7fe015e80640 1 -- 172.21.15.33:0/1426264497 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe01019f9e0 con 0x7fe0101022e0 2024-03-19T02:50:53.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.333+0000 7fe00cff9640 1 -- 172.21.15.33:0/1426264497 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe008018b70 con 0x7fe0101022e0 2024-03-19T02:50:53.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.333+0000 7fe015e80640 1 -- 172.21.15.33:0/1426264497 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe01019ff60 con 0x7fe0101022e0 2024-03-19T02:50:53.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.335+0000 7fe015e80640 1 -- 172.21.15.33:0/1426264497 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe010107ad0 con 0x7fe0101022e0 2024-03-19T02:50:53.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.335+0000 7fe00cff9640 1 -- 172.21.15.33:0/1426264497 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fe008018d10 con 0x7fe0101022e0 2024-03-19T02:50:53.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.336+0000 7fe00cff9640 1 --2- 172.21.15.33:0/1426264497 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fdff80a2800 0x7fdff80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:53.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.336+0000 7fe00f7fe640 1 --2- 172.21.15.33:0/1426264497 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fdff80a2800 0x7fdff80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:53.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.336+0000 7fe00cff9640 1 -- 172.21.15.33:0/1426264497 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fe0080cede0 con 0x7fe0101022e0 2024-03-19T02:50:53.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.336+0000 7fe00f7fe640 1 --2- 172.21.15.33:0/1426264497 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fdff80a2800 0x7fdff80a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe010101780 tx=0x7fe000008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:53.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.340+0000 7fe00cff9640 1 -- 172.21.15.33:0/1426264497 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe008014030 con 0x7fe0101022e0 2024-03-19T02:50:53.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:53.534+0000 7fe015e80640 1 -- 172.21.15.33:0/1426264497 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7fe01002d1f0 con 0x7fe0101022e0 2024-03-19T02:50:54.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.213+0000 7fe00cff9640 1 -- 172.21.15.33:0/1426264497 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v88) ==== 157+0+0 (secure 0 0 0) 0x7fe00809bfd0 con 0x7fe0101022e0 2024-03-19T02:50:54.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.216+0000 7fe015e80640 1 -- 172.21.15.33:0/1426264497 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fdff80a2800 msgr2=0x7fdff80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:54.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.216+0000 7fe015e80640 1 --2- 172.21.15.33:0/1426264497 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fdff80a2800 0x7fdff80a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe010101780 tx=0x7fe000008040 comp rx=0 tx=0).stop 2024-03-19T02:50:54.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.216+0000 7fe015e80640 1 -- 172.21.15.33:0/1426264497 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0101022e0 msgr2=0x7fe01019ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:54.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.216+0000 7fe015e80640 1 --2- 172.21.15.33:0/1426264497 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0101022e0 0x7fe01019ee30 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe00800cce0 tx=0x7fe00800a620 comp rx=0 tx=0).stop 2024-03-19T02:50:54.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.216+0000 7fe015e80640 1 -- 172.21.15.33:0/1426264497 shutdown_connections 2024-03-19T02:50:54.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.216+0000 7fe015e80640 1 --2- 172.21.15.33:0/1426264497 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fdff80a2800 0x7fdff80a4cc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:54.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.216+0000 7fe015e80640 1 --2- 172.21.15.33:0/1426264497 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe010101920 0x7fe01019c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:54.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.216+0000 7fe015e80640 1 --2- 172.21.15.33:0/1426264497 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe0101022e0 0x7fe01019ee30 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:54.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.216+0000 7fe015e80640 1 --2- 172.21.15.33:0/1426264497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe010100730 0x7fe01006a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:54.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.216+0000 7fe015e80640 1 -- 172.21.15.33:0/1426264497 >> 172.21.15.33:0/1426264497 conn(0x7fe0100fbcc0 msgr2=0x7fe010104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:54.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.216+0000 7fe015e80640 1 -- 172.21.15.33:0/1426264497 shutdown_connections 2024-03-19T02:50:54.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.216+0000 7fe015e80640 1 -- 172.21.15.33:0/1426264497 wait complete. 2024-03-19T02:50:54.231 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-19T02:50:54.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.347+0000 7fddbd5b2640 1 -- 172.21.15.33:0/174297481 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fddb8104c70 msgr2=0x7fddb8107060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:54.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.347+0000 7fddbd5b2640 1 --2- 172.21.15.33:0/174297481 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fddb8104c70 0x7fddb8107060 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdda0009900 tx=0x7fdda002bc30 comp rx=0 tx=0).stop 2024-03-19T02:50:54.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.347+0000 7fddbd5b2640 1 -- 172.21.15.33:0/174297481 shutdown_connections 2024-03-19T02:50:54.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.347+0000 7fddbd5b2640 1 --2- 172.21.15.33:0/174297481 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fddb81075a0 0x7fddb81099f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:54.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.347+0000 7fddbd5b2640 1 --2- 172.21.15.33:0/174297481 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fddb8104c70 0x7fddb8107060 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:54.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.347+0000 7fddbd5b2640 1 --2- 172.21.15.33:0/174297481 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fddb8102340 0x7fddb8104730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:54.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.347+0000 7fddbd5b2640 1 -- 172.21.15.33:0/174297481 >> 172.21.15.33:0/174297481 conn(0x7fddb80fbd50 msgr2=0x7fddb80fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:54.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.347+0000 7fddbd5b2640 1 -- 172.21.15.33:0/174297481 shutdown_connections 2024-03-19T02:50:54.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.347+0000 7fddbd5b2640 1 -- 172.21.15.33:0/174297481 wait complete. 2024-03-19T02:50:54.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.347+0000 7fddbd5b2640 1 Processor -- start 2024-03-19T02:50:54.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.348+0000 7fddbd5b2640 1 -- start start 2024-03-19T02:50:54.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.348+0000 7fddbd5b2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fddb8102340 0x7fddb8197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:54.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.348+0000 7fddbd5b2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fddb8104c70 0x7fddb8198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:54.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.348+0000 7fddbd5b2640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fddb81075a0 0x7fddb819f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:54.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.348+0000 7fddbd5b2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fddb810d450 con 0x7fddb8104c70 2024-03-19T02:50:54.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.348+0000 7fddbd5b2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fddb810d2d0 con 0x7fddb8102340 2024-03-19T02:50:54.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.348+0000 7fddbd5b2640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fddb810d5d0 con 0x7fddb81075a0 2024-03-19T02:50:54.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.348+0000 7fddae7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fddb8104c70 0x7fddb8198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:54.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.348+0000 7fddae7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fddb8104c70 0x7fddb8198120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41280/0 (socket says 172.21.15.33:41280) 2024-03-19T02:50:54.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.348+0000 7fddae7fc640 1 -- 172.21.15.33:0/3817317632 learned_addr learned my addr 172.21.15.33:0/3817317632 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:54.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.348+0000 7fddb6ffd640 1 --2- 172.21.15.33:0/3817317632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fddb8102340 0x7fddb8197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:54.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.348+0000 7fddb77fe640 1 --2- 172.21.15.33:0/3817317632 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fddb81075a0 0x7fddb819f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:54.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.348+0000 7fddae7fc640 1 -- 172.21.15.33:0/3817317632 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fddb81075a0 msgr2=0x7fddb819f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:54.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.348+0000 7fddae7fc640 1 --2- 172.21.15.33:0/3817317632 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fddb81075a0 0x7fddb819f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:54.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.348+0000 7fddae7fc640 1 -- 172.21.15.33:0/3817317632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fddb8102340 msgr2=0x7fddb8197be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:54.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.348+0000 7fddae7fc640 1 --2- 172.21.15.33:0/3817317632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fddb8102340 0x7fddb8197be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:54.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.348+0000 7fddae7fc640 1 -- 172.21.15.33:0/3817317632 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddb819f800 con 0x7fddb8104c70 2024-03-19T02:50:54.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.348+0000 7fddb6ffd640 1 --2- 172.21.15.33:0/3817317632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fddb8102340 0x7fddb8197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:50:54.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.348+0000 7fddb77fe640 1 --2- 172.21.15.33:0/3817317632 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fddb81075a0 0x7fddb819f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:50:54.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.348+0000 7fddae7fc640 1 --2- 172.21.15.33:0/3817317632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fddb8104c70 0x7fddb8198120 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdda0036040 tx=0x7fdda00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:54.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.349+0000 7fddb4ff9640 1 -- 172.21.15.33:0/3817317632 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda0043020 con 0x7fddb8104c70 2024-03-19T02:50:54.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.349+0000 7fddb4ff9640 1 -- 172.21.15.33:0/3817317632 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdda0004aa0 con 0x7fddb8104c70 2024-03-19T02:50:54.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.349+0000 7fddbd5b2640 1 -- 172.21.15.33:0/3817317632 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddb819fa90 con 0x7fddb8104c70 2024-03-19T02:50:54.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.349+0000 7fddb4ff9640 1 -- 172.21.15.33:0/3817317632 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda0003de0 con 0x7fddb8104c70 2024-03-19T02:50:54.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.349+0000 7fddbd5b2640 1 -- 172.21.15.33:0/3817317632 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddb81a0030 con 0x7fddb8104c70 2024-03-19T02:50:54.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.350+0000 7fddbd5b2640 1 -- 172.21.15.33:0/3817317632 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddb80fd840 con 0x7fddb8104c70 2024-03-19T02:50:54.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.351+0000 7fddb4ff9640 1 -- 172.21.15.33:0/3817317632 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fdda0046020 con 0x7fddb8104c70 2024-03-19T02:50:54.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.351+0000 7fddb4ff9640 1 --2- 172.21.15.33:0/3817317632 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fdda40a2880 0x7fdda40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:54.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.351+0000 7fddb6ffd640 1 --2- 172.21.15.33:0/3817317632 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fdda40a2880 0x7fdda40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:54.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.351+0000 7fddb4ff9640 1 -- 172.21.15.33:0/3817317632 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8277+0+0 (secure 0 0 0) 0x7fdda00f0ee0 con 0x7fddb8104c70 2024-03-19T02:50:54.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.352+0000 7fddb6ffd640 1 --2- 172.21.15.33:0/3817317632 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fdda40a2880 0x7fdda40a4d40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdda8009930 tx=0x7fdda8008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:54.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.356+0000 7fddb4ff9640 1 -- 172.21.15.33:0/3817317632 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdda004d020 con 0x7fddb8104c70 2024-03-19T02:50:54.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:54 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3412016773' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-19T02:50:54.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:54 smithi096 ceph-mon[35946]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-19T02:50:54.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:54 smithi096 ceph-mon[35946]: osdmap e87: 12 total, 12 up, 12 in 2024-03-19T02:50:54.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:54 smithi096 ceph-mon[35946]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-19T02:50:54.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:54 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1426264497' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-19T02:50:54.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:54 smithi096 ceph-mon[35946]: pgmap v294: 97 pgs: 16 creating+peering, 75 unknown, 6 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-19T02:50:54.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:54 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3412016773' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-19T02:50:54.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:54 smithi155 ceph-mon[35804]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-19T02:50:54.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:54 smithi155 ceph-mon[35804]: osdmap e87: 12 total, 12 up, 12 in 2024-03-19T02:50:54.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:54 smithi155 ceph-mon[35804]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-19T02:50:54.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:54 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1426264497' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-19T02:50:54.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:54 smithi155 ceph-mon[35804]: pgmap v294: 97 pgs: 16 creating+peering, 75 unknown, 6 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-19T02:50:54.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:54 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3412016773' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-19T02:50:54.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:54 smithi033 ceph-mon[31973]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-19T02:50:54.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:54 smithi033 ceph-mon[31973]: osdmap e87: 12 total, 12 up, 12 in 2024-03-19T02:50:54.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:54 smithi033 ceph-mon[31973]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-19T02:50:54.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:54 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1426264497' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-19T02:50:54.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:54 smithi033 ceph-mon[31973]: pgmap v294: 97 pgs: 16 creating+peering, 75 unknown, 6 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-19T02:50:54.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:54.539+0000 7fddbd5b2640 1 -- 172.21.15.33:0/3817317632 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fddb8062450 con 0x7fddb8104c70 2024-03-19T02:50:55.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:55 smithi096 ceph-mon[35946]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-19T02:50:55.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:55 smithi096 ceph-mon[35946]: Cluster is now healthy 2024-03-19T02:50:55.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:55 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1426264497' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-19T02:50:55.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:55 smithi096 ceph-mon[35946]: mds.? [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] up:active 2024-03-19T02:50:55.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:55 smithi096 ceph-mon[35946]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T02:50:55.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:55 smithi096 ceph-mon[35946]: osdmap e88: 12 total, 12 up, 12 in 2024-03-19T02:50:55.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:55 smithi096 ceph-mon[35946]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T02:50:55.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:55 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3817317632' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-19T02:50:55.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:55 smithi155 ceph-mon[35804]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-19T02:50:55.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:55 smithi155 ceph-mon[35804]: Cluster is now healthy 2024-03-19T02:50:55.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:55 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1426264497' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-19T02:50:55.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:55 smithi155 ceph-mon[35804]: mds.? [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] up:active 2024-03-19T02:50:55.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:55 smithi155 ceph-mon[35804]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T02:50:55.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:55 smithi155 ceph-mon[35804]: osdmap e88: 12 total, 12 up, 12 in 2024-03-19T02:50:55.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:55 smithi155 ceph-mon[35804]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T02:50:55.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:55 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3817317632' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-19T02:50:55.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:55 smithi033 ceph-mon[31973]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-19T02:50:55.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:55 smithi033 ceph-mon[31973]: Cluster is now healthy 2024-03-19T02:50:55.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:55 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1426264497' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-19T02:50:55.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:55 smithi033 ceph-mon[31973]: mds.? [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] up:active 2024-03-19T02:50:55.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:55 smithi033 ceph-mon[31973]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T02:50:55.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:55 smithi033 ceph-mon[31973]: osdmap e88: 12 total, 12 up, 12 in 2024-03-19T02:50:55.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:55 smithi033 ceph-mon[31973]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T02:50:55.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:55 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3817317632' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-19T02:50:56.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.226+0000 7fddb4ff9640 1 -- 172.21.15.33:0/3817317632 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v90) ==== 228+0+0 (secure 0 0 0) 0x7fdda00bdf50 con 0x7fddb8104c70 2024-03-19T02:50:56.227 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'cephfs_data_ec' created 2024-03-19T02:50:56.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.229+0000 7fddbd5b2640 1 -- 172.21.15.33:0/3817317632 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fdda40a2880 msgr2=0x7fdda40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:56.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.229+0000 7fddbd5b2640 1 --2- 172.21.15.33:0/3817317632 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fdda40a2880 0x7fdda40a4d40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdda8009930 tx=0x7fdda8008040 comp rx=0 tx=0).stop 2024-03-19T02:50:56.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.229+0000 7fddbd5b2640 1 -- 172.21.15.33:0/3817317632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fddb8104c70 msgr2=0x7fddb8198120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:56.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.229+0000 7fddbd5b2640 1 --2- 172.21.15.33:0/3817317632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fddb8104c70 0x7fddb8198120 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdda0036040 tx=0x7fdda00042f0 comp rx=0 tx=0).stop 2024-03-19T02:50:56.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.229+0000 7fddbd5b2640 1 -- 172.21.15.33:0/3817317632 shutdown_connections 2024-03-19T02:50:56.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.229+0000 7fddbd5b2640 1 --2- 172.21.15.33:0/3817317632 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fdda40a2880 0x7fdda40a4d40 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:56.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.229+0000 7fddbd5b2640 1 --2- 172.21.15.33:0/3817317632 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fddb81075a0 0x7fddb819f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:56.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.230+0000 7fddbd5b2640 1 --2- 172.21.15.33:0/3817317632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fddb8104c70 0x7fddb8198120 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:56.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.230+0000 7fddbd5b2640 1 --2- 172.21.15.33:0/3817317632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fddb8102340 0x7fddb8197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:56.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.230+0000 7fddbd5b2640 1 -- 172.21.15.33:0/3817317632 >> 172.21.15.33:0/3817317632 conn(0x7fddb80fbd50 msgr2=0x7fddb8102e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:56.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.230+0000 7fddbd5b2640 1 -- 172.21.15.33:0/3817317632 shutdown_connections 2024-03-19T02:50:56.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.230+0000 7fddbd5b2640 1 -- 172.21.15.33:0/3817317632 wait complete. 2024-03-19T02:50:56.249 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-19T02:50:56.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.360+0000 7fcef25c6640 1 -- 172.21.15.33:0/1026757492 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fceec100720 msgr2=0x7fceec100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:56.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.360+0000 7fcef25c6640 1 --2- 172.21.15.33:0/1026757492 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fceec100720 0x7fceec100b40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcedc009860 tx=0x7fcedc02bb70 comp rx=0 tx=0).stop 2024-03-19T02:50:56.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.360+0000 7fcef25c6640 1 -- 172.21.15.33:0/1026757492 shutdown_connections 2024-03-19T02:50:56.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.360+0000 7fcef25c6640 1 --2- 172.21.15.33:0/1026757492 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fceec1022d0 0x7fceec108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:56.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.360+0000 7fcef25c6640 1 --2- 172.21.15.33:0/1026757492 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fceec101910 0x7fceec101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:56.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.360+0000 7fcef25c6640 1 --2- 172.21.15.33:0/1026757492 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fceec100720 0x7fceec100b40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:56.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.361+0000 7fcef25c6640 1 -- 172.21.15.33:0/1026757492 >> 172.21.15.33:0/1026757492 conn(0x7fceec0fbcb0 msgr2=0x7fceec0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:56.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.361+0000 7fcef25c6640 1 -- 172.21.15.33:0/1026757492 shutdown_connections 2024-03-19T02:50:56.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.361+0000 7fcef25c6640 1 -- 172.21.15.33:0/1026757492 wait complete. 2024-03-19T02:50:56.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.361+0000 7fcef25c6640 1 Processor -- start 2024-03-19T02:50:56.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.361+0000 7fcef25c6640 1 -- start start 2024-03-19T02:50:56.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.361+0000 7fcef25c6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fceec100720 0x7fceec069510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:56.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.361+0000 7fcef25c6640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fceec101910 0x7fceec19c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:56.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.361+0000 7fcef25c6640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fceec1022d0 0x7fceec19ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:56.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.361+0000 7fcef25c6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fceec10c220 con 0x7fceec100720 2024-03-19T02:50:56.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.361+0000 7fcef25c6640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fceec10c0a0 con 0x7fceec101910 2024-03-19T02:50:56.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.361+0000 7fcef25c6640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fceec10c3a0 con 0x7fceec1022d0 2024-03-19T02:50:56.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.361+0000 7fceebfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fceec100720 0x7fceec069510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:56.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.362+0000 7fceebfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fceec100720 0x7fceec069510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41292/0 (socket says 172.21.15.33:41292) 2024-03-19T02:50:56.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.362+0000 7fceebfff640 1 -- 172.21.15.33:0/2105285046 learned_addr learned my addr 172.21.15.33:0/2105285046 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:56.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.362+0000 7fceeb7fe640 1 --2- 172.21.15.33:0/2105285046 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fceec101910 0x7fceec19c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:56.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.362+0000 7fcef0b63640 1 --2- 172.21.15.33:0/2105285046 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fceec1022d0 0x7fceec19ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:56.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.362+0000 7fceebfff640 1 -- 172.21.15.33:0/2105285046 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fceec1022d0 msgr2=0x7fceec19ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:56.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.362+0000 7fceebfff640 1 --2- 172.21.15.33:0/2105285046 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fceec1022d0 0x7fceec19ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:56.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.362+0000 7fceebfff640 1 -- 172.21.15.33:0/2105285046 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fceec101910 msgr2=0x7fceec19c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:56.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.362+0000 7fceebfff640 1 --2- 172.21.15.33:0/2105285046 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fceec101910 0x7fceec19c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:56.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.362+0000 7fceebfff640 1 -- 172.21.15.33:0/2105285046 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fceec19f6c0 con 0x7fceec100720 2024-03-19T02:50:56.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.362+0000 7fceebfff640 1 --2- 172.21.15.33:0/2105285046 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fceec100720 0x7fceec069510 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcedc005940 tx=0x7fcedc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:56.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.362+0000 7fcee97fa640 1 -- 172.21.15.33:0/2105285046 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcedc041020 con 0x7fceec100720 2024-03-19T02:50:56.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.362+0000 7fcee97fa640 1 -- 172.21.15.33:0/2105285046 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcedc0035f0 con 0x7fceec100720 2024-03-19T02:50:56.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.362+0000 7fcef25c6640 1 -- 172.21.15.33:0/2105285046 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fceec19f950 con 0x7fceec100720 2024-03-19T02:50:56.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.362+0000 7fcee97fa640 1 -- 172.21.15.33:0/2105285046 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcedc039040 con 0x7fceec100720 2024-03-19T02:50:56.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.362+0000 7fcef25c6640 1 -- 172.21.15.33:0/2105285046 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fceec19fe40 con 0x7fceec100720 2024-03-19T02:50:56.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.363+0000 7fcef25c6640 1 -- 172.21.15.33:0/2105285046 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fceec107c00 con 0x7fceec100720 2024-03-19T02:50:56.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.364+0000 7fcee97fa640 1 -- 172.21.15.33:0/2105285046 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fcedc044020 con 0x7fceec100720 2024-03-19T02:50:56.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.365+0000 7fcee97fa640 1 --2- 172.21.15.33:0/2105285046 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fced80a2800 0x7fced80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:56.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.365+0000 7fceeb7fe640 1 --2- 172.21.15.33:0/2105285046 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fced80a2800 0x7fced80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:56.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.365+0000 7fcee97fa640 1 -- 172.21.15.33:0/2105285046 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8770+0+0 (secure 0 0 0) 0x7fcedc0f0700 con 0x7fceec100720 2024-03-19T02:50:56.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.365+0000 7fceeb7fe640 1 --2- 172.21.15.33:0/2105285046 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fced80a2800 0x7fced80a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fceec069280 tx=0x7fced4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:56.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.370+0000 7fcee97fa640 1 -- 172.21.15.33:0/2105285046 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcedc04b020 con 0x7fceec100720 2024-03-19T02:50:56.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:56 smithi155 ceph-mon[35804]: osdmap e89: 12 total, 12 up, 12 in 2024-03-19T02:50:56.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:56 smithi155 ceph-mon[35804]: pgmap v297: 97 pgs: 16 creating+peering, 12 unknown, 69 active+clean; 579 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s wr, 6 op/s 2024-03-19T02:50:56.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:56 smithi033 ceph-mon[31973]: osdmap e89: 12 total, 12 up, 12 in 2024-03-19T02:50:56.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:56 smithi033 ceph-mon[31973]: pgmap v297: 97 pgs: 16 creating+peering, 12 unknown, 69 active+clean; 579 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s wr, 6 op/s 2024-03-19T02:50:56.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:56.551+0000 7fcef25c6640 1 -- 172.21.15.33:0/2105285046 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fceec062490 con 0x7fceec100720 2024-03-19T02:50:56.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:56 smithi096 ceph-mon[35946]: osdmap e89: 12 total, 12 up, 12 in 2024-03-19T02:50:56.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:56 smithi096 ceph-mon[35946]: pgmap v297: 97 pgs: 16 creating+peering, 12 unknown, 69 active+clean; 579 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s wr, 6 op/s 2024-03-19T02:50:57.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.228+0000 7fcee97fa640 1 -- 172.21.15.33:0/2105285046 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v91) ==== 169+0+0 (secure 0 0 0) 0x7fcedc0bd520 con 0x7fceec100720 2024-03-19T02:50:57.229 INFO:teuthology.orchestra.run.smithi033.stderr:set pool 4 allow_ec_overwrites to true 2024-03-19T02:50:57.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.232+0000 7fcef25c6640 1 -- 172.21.15.33:0/2105285046 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fced80a2800 msgr2=0x7fced80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:57.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.232+0000 7fcef25c6640 1 --2- 172.21.15.33:0/2105285046 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fced80a2800 0x7fced80a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fceec069280 tx=0x7fced4009260 comp rx=0 tx=0).stop 2024-03-19T02:50:57.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.232+0000 7fcef25c6640 1 -- 172.21.15.33:0/2105285046 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fceec100720 msgr2=0x7fceec069510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:57.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.232+0000 7fcef25c6640 1 --2- 172.21.15.33:0/2105285046 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fceec100720 0x7fceec069510 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcedc005940 tx=0x7fcedc0042f0 comp rx=0 tx=0).stop 2024-03-19T02:50:57.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.232+0000 7fcef25c6640 1 -- 172.21.15.33:0/2105285046 shutdown_connections 2024-03-19T02:50:57.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.232+0000 7fcef25c6640 1 --2- 172.21.15.33:0/2105285046 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fced80a2800 0x7fced80a4cc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:57.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.232+0000 7fcef25c6640 1 --2- 172.21.15.33:0/2105285046 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fceec1022d0 0x7fceec19ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:57.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.232+0000 7fcef25c6640 1 --2- 172.21.15.33:0/2105285046 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fceec101910 0x7fceec19c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:57.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.232+0000 7fcef25c6640 1 --2- 172.21.15.33:0/2105285046 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fceec100720 0x7fceec069510 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:57.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.232+0000 7fcef25c6640 1 -- 172.21.15.33:0/2105285046 >> 172.21.15.33:0/2105285046 conn(0x7fceec0fbcb0 msgr2=0x7fceec104dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:57.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.232+0000 7fcef25c6640 1 -- 172.21.15.33:0/2105285046 shutdown_connections 2024-03-19T02:50:57.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.232+0000 7fcef25c6640 1 -- 172.21.15.33:0/2105285046 wait complete. 2024-03-19T02:50:57.258 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-19T02:50:57.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.366+0000 7f7399713640 1 -- 172.21.15.33:0/3121217916 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7394101910 msgr2=0x7f7394101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:57.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.366+0000 7f7399713640 1 --2- 172.21.15.33:0/3121217916 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7394101910 0x7f7394101d90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f737c009900 tx=0x7f737c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:50:57.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.366+0000 7f7399713640 1 -- 172.21.15.33:0/3121217916 shutdown_connections 2024-03-19T02:50:57.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.366+0000 7f7399713640 1 --2- 172.21.15.33:0/3121217916 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f73941022d0 0x7f7394108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:57.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.366+0000 7f7399713640 1 --2- 172.21.15.33:0/3121217916 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7394101910 0x7f7394101d90 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:57.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.366+0000 7f7399713640 1 --2- 172.21.15.33:0/3121217916 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7394100720 0x7f7394100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:57.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.366+0000 7f7399713640 1 -- 172.21.15.33:0/3121217916 >> 172.21.15.33:0/3121217916 conn(0x7f73940fbcb0 msgr2=0x7f73940fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:57.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.366+0000 7f7399713640 1 -- 172.21.15.33:0/3121217916 shutdown_connections 2024-03-19T02:50:57.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.367+0000 7f7399713640 1 -- 172.21.15.33:0/3121217916 wait complete. 2024-03-19T02:50:57.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.367+0000 7f7399713640 1 Processor -- start 2024-03-19T02:50:57.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.367+0000 7f7399713640 1 -- start start 2024-03-19T02:50:57.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.367+0000 7f7399713640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7394100720 0x7f73940694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:57.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.367+0000 7f7399713640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7394101910 0x7f739419c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:57.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.367+0000 7f7399713640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73941022d0 0x7f739419edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:57.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.367+0000 7f7399713640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f739410c250 con 0x7f7394100720 2024-03-19T02:50:57.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.367+0000 7f7399713640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f739410c0d0 con 0x7f73941022d0 2024-03-19T02:50:57.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.367+0000 7f7399713640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f739410c3d0 con 0x7f7394101910 2024-03-19T02:50:57.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.367+0000 7f7392ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7394100720 0x7f73940694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:57.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.367+0000 7f7392ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7394100720 0x7f73940694f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41302/0 (socket says 172.21.15.33:41302) 2024-03-19T02:50:57.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.367+0000 7f7392ffd640 1 -- 172.21.15.33:0/2180227573 learned_addr learned my addr 172.21.15.33:0/2180227573 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:57.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.367+0000 7f73937fe640 1 --2- 172.21.15.33:0/2180227573 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73941022d0 0x7f739419edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:57.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.368+0000 7f73927fc640 1 --2- 172.21.15.33:0/2180227573 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7394101910 0x7f739419c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:57.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.368+0000 7f7392ffd640 1 -- 172.21.15.33:0/2180227573 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7394101910 msgr2=0x7f739419c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:57.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.368+0000 7f7392ffd640 1 --2- 172.21.15.33:0/2180227573 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7394101910 0x7f739419c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:57.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.368+0000 7f7392ffd640 1 -- 172.21.15.33:0/2180227573 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73941022d0 msgr2=0x7f739419edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:57.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.368+0000 7f7392ffd640 1 --2- 172.21.15.33:0/2180227573 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73941022d0 0x7f739419edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:57.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.368+0000 7f7392ffd640 1 -- 172.21.15.33:0/2180227573 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f739419f700 con 0x7f7394100720 2024-03-19T02:50:57.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.368+0000 7f7392ffd640 1 --2- 172.21.15.33:0/2180227573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7394100720 0x7f73940694f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f738c00e9c0 tx=0x7f738c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:57.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.368+0000 7f738bfff640 1 -- 172.21.15.33:0/2180227573 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f738c00cde0 con 0x7f7394100720 2024-03-19T02:50:57.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.368+0000 7f738bfff640 1 -- 172.21.15.33:0/2180227573 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f738c0108e0 con 0x7f7394100720 2024-03-19T02:50:57.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.368+0000 7f7399713640 1 -- 172.21.15.33:0/2180227573 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f739419f990 con 0x7f7394100720 2024-03-19T02:50:57.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.368+0000 7f738bfff640 1 -- 172.21.15.33:0/2180227573 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f738c0188e0 con 0x7f7394100720 2024-03-19T02:50:57.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.368+0000 7f7399713640 1 -- 172.21.15.33:0/2180227573 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f739419fee0 con 0x7f7394100720 2024-03-19T02:50:57.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.369+0000 7f7399713640 1 -- 172.21.15.33:0/2180227573 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7394107ab0 con 0x7f7394100720 2024-03-19T02:50:57.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.370+0000 7f738bfff640 1 -- 172.21.15.33:0/2180227573 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f738c01f020 con 0x7f7394100720 2024-03-19T02:50:57.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.371+0000 7f738bfff640 1 --2- 172.21.15.33:0/2180227573 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f73800a2800 0x7f73800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:57.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.371+0000 7f738bfff640 1 -- 172.21.15.33:0/2180227573 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8770+0+0 (secure 0 0 0) 0x7f738c0ceef0 con 0x7f7394100720 2024-03-19T02:50:57.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.371+0000 7f73927fc640 1 --2- 172.21.15.33:0/2180227573 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f73800a2800 0x7f73800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:57.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.371+0000 7f73927fc640 1 --2- 172.21.15.33:0/2180227573 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f73800a2800 0x7f73800a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f737c036040 tx=0x7f737c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:57.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.375+0000 7f738bfff640 1 -- 172.21.15.33:0/2180227573 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f738c09bd70 con 0x7f7394100720 2024-03-19T02:50:57.526 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:57 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3817317632' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-19T02:50:57.526 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:57 smithi155 ceph-mon[35804]: osdmap e90: 12 total, 12 up, 12 in 2024-03-19T02:50:57.526 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:57 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2105285046' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-19T02:50:57.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:57 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3817317632' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-19T02:50:57.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:57 smithi033 ceph-mon[31973]: osdmap e90: 12 total, 12 up, 12 in 2024-03-19T02:50:57.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:57 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2105285046' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-19T02:50:57.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:57.600+0000 7f7399713640 1 -- 172.21.15.33:0/2180227573 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f739402d1f0 con 0x7f7394100720 2024-03-19T02:50:57.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:57 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3817317632' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-19T02:50:57.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:57 smithi096 ceph-mon[35946]: osdmap e90: 12 total, 12 up, 12 in 2024-03-19T02:50:57.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:57 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2105285046' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-19T02:50:58.215 INFO:journalctl@ceph.mds.b.smithi096.stdout:Mar 19 02:50:57 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-b[65459]: 2024-03-19T02:50:57.788+0000 7f393d23b700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-19T02:50:58.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.246+0000 7f738bfff640 1 -- 172.21.15.33:0/2180227573 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f738c09b670 con 0x7f7394100720 2024-03-19T02:50:58.247 INFO:teuthology.orchestra.run.smithi033.stderr:added data pool 4 to fsmap 2024-03-19T02:50:58.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.249+0000 7f7399713640 1 -- 172.21.15.33:0/2180227573 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f73800a2800 msgr2=0x7f73800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:58.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.249+0000 7f7399713640 1 --2- 172.21.15.33:0/2180227573 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f73800a2800 0x7f73800a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f737c036040 tx=0x7f737c000bc0 comp rx=0 tx=0).stop 2024-03-19T02:50:58.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.249+0000 7f7399713640 1 -- 172.21.15.33:0/2180227573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7394100720 msgr2=0x7f73940694f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:58.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.249+0000 7f7399713640 1 --2- 172.21.15.33:0/2180227573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7394100720 0x7f73940694f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f738c00e9c0 tx=0x7f738c00ee80 comp rx=0 tx=0).stop 2024-03-19T02:50:58.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.249+0000 7f7399713640 1 -- 172.21.15.33:0/2180227573 shutdown_connections 2024-03-19T02:50:58.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.249+0000 7f7399713640 1 --2- 172.21.15.33:0/2180227573 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f73800a2800 0x7f73800a4cc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:58.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.249+0000 7f7399713640 1 --2- 172.21.15.33:0/2180227573 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7394101910 0x7f739419c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:58.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.249+0000 7f7399713640 1 --2- 172.21.15.33:0/2180227573 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f73941022d0 0x7f739419edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:58.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.249+0000 7f7399713640 1 --2- 172.21.15.33:0/2180227573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7394100720 0x7f73940694f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:58.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.249+0000 7f7399713640 1 -- 172.21.15.33:0/2180227573 >> 172.21.15.33:0/2180227573 conn(0x7f73940fbcb0 msgr2=0x7f7394104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:58.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.249+0000 7f7399713640 1 -- 172.21.15.33:0/2180227573 shutdown_connections 2024-03-19T02:50:58.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.249+0000 7f7399713640 1 -- 172.21.15.33:0/2180227573 wait complete. 2024-03-19T02:50:58.263 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:50:58.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.372+0000 7f26f0819640 1 -- 172.21.15.33:0/1352232366 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26ec101930 msgr2=0x7f26ec101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:58.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.372+0000 7f26f0819640 1 --2- 172.21.15.33:0/1352232366 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26ec101930 0x7f26ec101db0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f26d4009860 tx=0x7f26d402bb70 comp rx=0 tx=0).stop 2024-03-19T02:50:58.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.372+0000 7f26f0819640 1 -- 172.21.15.33:0/1352232366 shutdown_connections 2024-03-19T02:50:58.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.372+0000 7f26f0819640 1 --2- 172.21.15.33:0/1352232366 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26ec1022f0 0x7f26ec108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:58.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.372+0000 7f26f0819640 1 --2- 172.21.15.33:0/1352232366 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26ec101930 0x7f26ec101db0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:58.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.372+0000 7f26f0819640 1 --2- 172.21.15.33:0/1352232366 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f26ec100740 0x7f26ec100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:58.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.372+0000 7f26f0819640 1 -- 172.21.15.33:0/1352232366 >> 172.21.15.33:0/1352232366 conn(0x7f26ec0fbcb0 msgr2=0x7f26ec0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:58.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.372+0000 7f26f0819640 1 -- 172.21.15.33:0/1352232366 shutdown_connections 2024-03-19T02:50:58.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.372+0000 7f26f0819640 1 -- 172.21.15.33:0/1352232366 wait complete. 2024-03-19T02:50:58.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.372+0000 7f26f0819640 1 Processor -- start 2024-03-19T02:50:58.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.372+0000 7f26f0819640 1 -- start start 2024-03-19T02:50:58.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.373+0000 7f26f0819640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26ec100740 0x7f26ec197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:58.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.373+0000 7f26f0819640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26ec101930 0x7f26ec198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:58.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.373+0000 7f26f0819640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f26ec1022f0 0x7f26ec19f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:58.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.373+0000 7f26f0819640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f26ec10c270 con 0x7f26ec101930 2024-03-19T02:50:58.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.373+0000 7f26f0819640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f26ec10c0f0 con 0x7f26ec100740 2024-03-19T02:50:58.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.373+0000 7f26f0819640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f26ec10c3f0 con 0x7f26ec1022f0 2024-03-19T02:50:58.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.373+0000 7f26e3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26ec101930 0x7f26ec198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:58.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.373+0000 7f26e3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26ec101930 0x7f26ec198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41322/0 (socket says 172.21.15.33:41322) 2024-03-19T02:50:58.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.373+0000 7f26e3fff640 1 -- 172.21.15.33:0/177750341 learned_addr learned my addr 172.21.15.33:0/177750341 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:58.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.373+0000 7f26ea59c640 1 --2- 172.21.15.33:0/177750341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26ec100740 0x7f26ec197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:58.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.373+0000 7f26e3fff640 1 -- 172.21.15.33:0/177750341 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f26ec1022f0 msgr2=0x7f26ec19f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:58.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.373+0000 7f26ead9d640 1 --2- 172.21.15.33:0/177750341 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f26ec1022f0 0x7f26ec19f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:58.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.373+0000 7f26e3fff640 1 --2- 172.21.15.33:0/177750341 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f26ec1022f0 0x7f26ec19f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:58.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.373+0000 7f26e3fff640 1 -- 172.21.15.33:0/177750341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26ec100740 msgr2=0x7f26ec197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:58.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.373+0000 7f26e3fff640 1 --2- 172.21.15.33:0/177750341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26ec100740 0x7f26ec197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:58.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.373+0000 7f26e3fff640 1 -- 172.21.15.33:0/177750341 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26ec19f6e0 con 0x7f26ec101930 2024-03-19T02:50:58.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.373+0000 7f26e3fff640 1 --2- 172.21.15.33:0/177750341 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26ec101930 0x7f26ec198000 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f26d4005940 tx=0x7f26d4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:58.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.373+0000 7f26e37fe640 1 -- 172.21.15.33:0/177750341 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d4041020 con 0x7f26ec101930 2024-03-19T02:50:58.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.373+0000 7f26e37fe640 1 -- 172.21.15.33:0/177750341 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f26d402c9c0 con 0x7f26ec101930 2024-03-19T02:50:58.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.373+0000 7f26f0819640 1 -- 172.21.15.33:0/177750341 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26ec19f970 con 0x7f26ec101930 2024-03-19T02:50:58.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.373+0000 7f26e37fe640 1 -- 172.21.15.33:0/177750341 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d4039040 con 0x7f26ec101930 2024-03-19T02:50:58.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.374+0000 7f26f0819640 1 -- 172.21.15.33:0/177750341 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26ec19fe60 con 0x7f26ec101930 2024-03-19T02:50:58.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.374+0000 7f26f0819640 1 -- 172.21.15.33:0/177750341 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26ec1a0030 con 0x7f26ec101930 2024-03-19T02:50:58.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.375+0000 7f26e37fe640 1 -- 172.21.15.33:0/177750341 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f26d4044020 con 0x7f26ec101930 2024-03-19T02:50:58.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.376+0000 7f26e37fe640 1 --2- 172.21.15.33:0/177750341 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f26d80a2800 0x7f26d80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:58.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.376+0000 7f26ea59c640 1 --2- 172.21.15.33:0/177750341 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f26d80a2800 0x7f26d80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:58.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.376+0000 7f26e37fe640 1 -- 172.21.15.33:0/177750341 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f26d40f0690 con 0x7f26ec101930 2024-03-19T02:50:58.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.376+0000 7f26ea59c640 1 --2- 172.21.15.33:0/177750341 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f26d80a2800 0x7f26d80a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f26dc0058c0 tx=0x7f26dc00a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:58.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.380+0000 7f26e37fe640 1 -- 172.21.15.33:0/177750341 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f26d404b020 con 0x7f26ec101930 2024-03-19T02:50:58.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:58 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2105285046' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-19T02:50:58.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:58 smithi155 ceph-mon[35804]: osdmap e91: 12 total, 12 up, 12 in 2024-03-19T02:50:58.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:58 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2180227573' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-19T02:50:58.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:58 smithi155 ceph-mon[35804]: pgmap v300: 129 pgs: 5 creating+peering, 27 unknown, 97 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 9 op/s 2024-03-19T02:50:58.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:58 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2105285046' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-19T02:50:58.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:58 smithi033 ceph-mon[31973]: osdmap e91: 12 total, 12 up, 12 in 2024-03-19T02:50:58.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:58 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2180227573' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-19T02:50:58.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:58 smithi033 ceph-mon[31973]: pgmap v300: 129 pgs: 5 creating+peering, 27 unknown, 97 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 9 op/s 2024-03-19T02:50:58.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.605+0000 7f26f0819640 1 -- 172.21.15.33:0/177750341 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f26ec1a03c0 con 0x7f26ec101930 2024-03-19T02:50:58.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.606+0000 7f26e37fe640 1 -- 172.21.15.33:0/177750341 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11794 (secure 0 0 0) 0x7f26d40bd4f0 con 0x7f26ec101930 2024-03-19T02:50:58.607 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:50:58.607 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24436,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:50:52.201798+0000","modified":"2024-03-19T02:50:57.601788+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24509},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/3731232169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":3731232169},{"type":"v1","addr":"172.21.15.96:6839","nonce":3731232169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24509,"qdb_cluster":[24509]},"id":1}]} 2024-03-19T02:50:58.607 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 17 2024-03-19T02:50:58.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.608+0000 7f26f0819640 1 -- 172.21.15.33:0/177750341 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f26d80a2800 msgr2=0x7f26d80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:58.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.608+0000 7f26f0819640 1 --2- 172.21.15.33:0/177750341 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f26d80a2800 0x7f26d80a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f26dc0058c0 tx=0x7f26dc00a2d0 comp rx=0 tx=0).stop 2024-03-19T02:50:58.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.608+0000 7f26f0819640 1 -- 172.21.15.33:0/177750341 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26ec101930 msgr2=0x7f26ec198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:58.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.608+0000 7f26f0819640 1 --2- 172.21.15.33:0/177750341 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26ec101930 0x7f26ec198000 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f26d4005940 tx=0x7f26d4004270 comp rx=0 tx=0).stop 2024-03-19T02:50:58.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.608+0000 7f26f0819640 1 -- 172.21.15.33:0/177750341 shutdown_connections 2024-03-19T02:50:58.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.608+0000 7f26f0819640 1 --2- 172.21.15.33:0/177750341 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f26d80a2800 0x7f26d80a4cc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:58.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.608+0000 7f26f0819640 1 --2- 172.21.15.33:0/177750341 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f26ec1022f0 0x7f26ec19f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:58.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.608+0000 7f26f0819640 1 --2- 172.21.15.33:0/177750341 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f26ec101930 0x7f26ec198000 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:58.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.608+0000 7f26f0819640 1 --2- 172.21.15.33:0/177750341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26ec100740 0x7f26ec197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:58.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.608+0000 7f26f0819640 1 -- 172.21.15.33:0/177750341 >> 172.21.15.33:0/177750341 conn(0x7f26ec0fbcb0 msgr2=0x7f26ec068a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:58.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.608+0000 7f26f0819640 1 -- 172.21.15.33:0/177750341 shutdown_connections 2024-03-19T02:50:58.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.608+0000 7f26f0819640 1 -- 172.21.15.33:0/177750341 wait complete. 2024-03-19T02:50:58.621 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:50:58.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:58 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2105285046' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-19T02:50:58.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:58 smithi096 ceph-mon[35946]: osdmap e91: 12 total, 12 up, 12 in 2024-03-19T02:50:58.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:58 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2180227573' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-19T02:50:58.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:58 smithi096 ceph-mon[35946]: pgmap v300: 129 pgs: 5 creating+peering, 27 unknown, 97 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 9 op/s 2024-03-19T02:50:58.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.722+0000 7f121a9b9640 1 -- 172.21.15.33:0/2615119061 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f12141075a0 msgr2=0x7f1214109990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:58.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.722+0000 7f121a9b9640 1 --2- 172.21.15.33:0/2615119061 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f12141075a0 0x7f1214109990 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f120c009ca0 tx=0x7f120c02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:50:58.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.722+0000 7f121a9b9640 1 -- 172.21.15.33:0/2615119061 shutdown_connections 2024-03-19T02:50:58.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.722+0000 7f121a9b9640 1 --2- 172.21.15.33:0/2615119061 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1214104c70 0x7f1214107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:58.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.722+0000 7f121a9b9640 1 --2- 172.21.15.33:0/2615119061 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f12141075a0 0x7f1214109990 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:58.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.722+0000 7f121a9b9640 1 --2- 172.21.15.33:0/2615119061 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1214102340 0x7f1214104730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:58.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.722+0000 7f121a9b9640 1 -- 172.21.15.33:0/2615119061 >> 172.21.15.33:0/2615119061 conn(0x7f12140fbcd0 msgr2=0x7f12140fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:58.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.722+0000 7f121a9b9640 1 -- 172.21.15.33:0/2615119061 shutdown_connections 2024-03-19T02:50:58.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.722+0000 7f121a9b9640 1 -- 172.21.15.33:0/2615119061 wait complete. 2024-03-19T02:50:58.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.722+0000 7f121a9b9640 1 Processor -- start 2024-03-19T02:50:58.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.722+0000 7f121a9b9640 1 -- start start 2024-03-19T02:50:58.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.723+0000 7f121a9b9640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1214102340 0x7f1214197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:58.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.723+0000 7f121a9b9640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1214104c70 0x7f1214198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:58.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.723+0000 7f121a9b9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f12141075a0 0x7f121419f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:58.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.723+0000 7f121a9b9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f121410d400 con 0x7f12141075a0 2024-03-19T02:50:58.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.723+0000 7f121a9b9640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f121410d280 con 0x7f1214102340 2024-03-19T02:50:58.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.723+0000 7f121a9b9640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f121410d580 con 0x7f1214104c70 2024-03-19T02:50:58.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.723+0000 7f1218f56640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f12141075a0 0x7f121419f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:58.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.723+0000 7f1218f56640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f12141075a0 0x7f121419f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41346/0 (socket says 172.21.15.33:41346) 2024-03-19T02:50:58.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.723+0000 7f1218f56640 1 -- 172.21.15.33:0/234557893 learned_addr learned my addr 172.21.15.33:0/234557893 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:58.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.723+0000 7f1213fff640 1 --2- 172.21.15.33:0/234557893 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1214102340 0x7f1214197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:58.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.723+0000 7f12137fe640 1 --2- 172.21.15.33:0/234557893 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1214104c70 0x7f1214198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:58.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.723+0000 7f1218f56640 1 -- 172.21.15.33:0/234557893 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1214104c70 msgr2=0x7f1214198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:58.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.723+0000 7f1218f56640 1 --2- 172.21.15.33:0/234557893 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1214104c70 0x7f1214198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:58.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.723+0000 7f1218f56640 1 -- 172.21.15.33:0/234557893 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1214102340 msgr2=0x7f1214197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:58.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.723+0000 7f1218f56640 1 --2- 172.21.15.33:0/234557893 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1214102340 0x7f1214197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:58.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.723+0000 7f1218f56640 1 -- 172.21.15.33:0/234557893 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f121419f750 con 0x7f12141075a0 2024-03-19T02:50:58.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.723+0000 7f1218f56640 1 --2- 172.21.15.33:0/234557893 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f12141075a0 0x7f121419f0e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f120c037040 tx=0x7f120c032820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:58.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.723+0000 7f12117fa640 1 -- 172.21.15.33:0/234557893 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f120c041020 con 0x7f12141075a0 2024-03-19T02:50:58.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.723+0000 7f121a9b9640 1 -- 172.21.15.33:0/234557893 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f121419f9e0 con 0x7f12141075a0 2024-03-19T02:50:58.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.723+0000 7f12117fa640 1 -- 172.21.15.33:0/234557893 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f120c003e20 con 0x7f12141075a0 2024-03-19T02:50:58.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.723+0000 7f12117fa640 1 -- 172.21.15.33:0/234557893 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f120c039040 con 0x7f12141075a0 2024-03-19T02:50:58.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.723+0000 7f121a9b9640 1 -- 172.21.15.33:0/234557893 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f121419ff50 con 0x7f12141075a0 2024-03-19T02:50:58.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.724+0000 7f121a9b9640 1 -- 172.21.15.33:0/234557893 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1214067db0 con 0x7f12141075a0 2024-03-19T02:50:58.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.725+0000 7f12117fa640 1 -- 172.21.15.33:0/234557893 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f120c044020 con 0x7f12141075a0 2024-03-19T02:50:58.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.726+0000 7f12117fa640 1 --2- 172.21.15.33:0/234557893 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f11fc0a2800 0x7f11fc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:58.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.726+0000 7f1213fff640 1 --2- 172.21.15.33:0/234557893 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f11fc0a2800 0x7f11fc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:58.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.726+0000 7f12117fa640 1 -- 172.21.15.33:0/234557893 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f120c0f0e90 con 0x7f12141075a0 2024-03-19T02:50:58.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.726+0000 7f1213fff640 1 --2- 172.21.15.33:0/234557893 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f11fc0a2800 0x7f11fc0a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1204009bd0 tx=0x7f1204005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:58.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.730+0000 7f12117fa640 1 -- 172.21.15.33:0/234557893 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1214067db0 con 0x7f12141075a0 2024-03-19T02:50:58.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.913+0000 7f121a9b9640 1 -- 172.21.15.33:0/234557893 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1214062450 con 0x7f12141075a0 2024-03-19T02:50:58.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.914+0000 7f12117fa640 1 -- 172.21.15.33:0/234557893 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24056 (secure 0 0 0) 0x7f120c04b020 con 0x7f12141075a0 2024-03-19T02:50:58.916 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:50:58.916 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":92,"fsid":"2b7a8938-e59a-11ee-95c9-87774f69a715","created":"2024-03-19T02:42:37.912179+0000","modified":"2024-03-19T02:50:57.601593+0000","last_up_change":"2024-03-19T02:50:01.539392+0000","last_in_change":"2024-03-19T02:49:40.534415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:46:09.963737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:50:50.069641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:50:51.381064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:50:55.476667+0000","flags":32773,"flags_names":"hashpspool,ec_overwrites,creating","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"090a89ab-6b7b-4317-8823-178d4212b2f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6803","nonce":1633124995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6805","nonce":1633124995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6809","nonce":1633124995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6807","nonce":1633124995}]},"public_addr":"172.21.15.33:6803/1633124995","cluster_addr":"172.21.15.33:6805/1633124995","heartbeat_back_addr":"172.21.15.33:6809/1633124995","heartbeat_front_addr":"172.21.15.33:6807/1633124995","state":["exists","up"]},{"osd":1,"uuid":"f50eec55-1956-4486-8593-f2c8fa610e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6801","nonce":340144398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6803","nonce":340144398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6807","nonce":340144398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6805","nonce":340144398}]},"public_addr":"172.21.15.96:6801/340144398","cluster_addr":"172.21.15.96:6803/340144398","heartbeat_back_addr":"172.21.15.96:6807/340144398","heartbeat_front_addr":"172.21.15.96:6805/340144398","state":["exists","up"]},{"osd":2,"uuid":"893a2d62-80d9-4e5e-bc6a-a1c8c681d652","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6801","nonce":3388743876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6803","nonce":3388743876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6807","nonce":3388743876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6805","nonce":3388743876}]},"public_addr":"172.21.15.155:6801/3388743876","cluster_addr":"172.21.15.155:6803/3388743876","heartbeat_back_addr":"172.21.15.155:6807/3388743876","heartbeat_front_addr":"172.21.15.155:6805/3388743876","state":["exists","up"]},{"osd":3,"uuid":"b07e0482-4281-4d32-9dfe-9995d7246175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6811","nonce":677007171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6813","nonce":677007171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6817","nonce":677007171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6815","nonce":677007171}]},"public_addr":"172.21.15.33:6811/677007171","cluster_addr":"172.21.15.33:6813/677007171","heartbeat_back_addr":"172.21.15.33:6817/677007171","heartbeat_front_addr":"172.21.15.33:6815/677007171","state":["exists","up"]},{"osd":4,"uuid":"a988384b-af6b-47c6-8c07-5c9673acee45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6809","nonce":3106158554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6811","nonce":3106158554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6815","nonce":3106158554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6813","nonce":3106158554}]},"public_addr":"172.21.15.96:6809/3106158554","cluster_addr":"172.21.15.96:6811/3106158554","heartbeat_back_addr":"172.21.15.96:6815/3106158554","heartbeat_front_addr":"172.21.15.96:6813/3106158554","state":["exists","up"]},{"osd":5,"uuid":"198dc992-96f8-4c52-96ee-1fd5de148bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6809","nonce":327293324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6811","nonce":327293324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6815","nonce":327293324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6813","nonce":327293324}]},"public_addr":"172.21.15.155:6809/327293324","cluster_addr":"172.21.15.155:6811/327293324","heartbeat_back_addr":"172.21.15.155:6815/327293324","heartbeat_front_addr":"172.21.15.155:6813/327293324","state":["exists","up"]},{"osd":6,"uuid":"a7278ec0-ab2e-49ed-be53-84d8b9e24177","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6819","nonce":873871509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6821","nonce":873871509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6825","nonce":873871509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6823","nonce":873871509}]},"public_addr":"172.21.15.33:6819/873871509","cluster_addr":"172.21.15.33:6821/873871509","heartbeat_back_addr":"172.21.15.33:6825/873871509","heartbeat_front_addr":"172.21.15.33:6823/873871509","state":["exists","up"]},{"osd":7,"uuid":"727e133d-aa24-4ffc-90e2-266eb2aab8c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6817","nonce":2334123891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6819","nonce":2334123891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6823","nonce":2334123891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6821","nonce":2334123891}]},"public_addr":"172.21.15.96:6817/2334123891","cluster_addr":"172.21.15.96:6819/2334123891","heartbeat_back_addr":"172.21.15.96:6823/2334123891","heartbeat_front_addr":"172.21.15.96:6821/2334123891","state":["exists","up"]},{"osd":8,"uuid":"bb20eec0-39e5-4b86-9db5-486b1840cdad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6817","nonce":3498516903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6819","nonce":3498516903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6823","nonce":3498516903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6821","nonce":3498516903}]},"public_addr":"172.21.15.155:6817/3498516903","cluster_addr":"172.21.15.155:6819/3498516903","heartbeat_back_addr":"172.21.15.155:6823/3498516903","heartbeat_front_addr":"172.21.15.155:6821/3498516903","state":["exists","up"]},{"osd":9,"uuid":"179c6228-3e23-4f65-91a6-1ef0005d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6827","nonce":760777213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6829","nonce":760777213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6833","nonce":760777213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6831","nonce":760777213}]},"public_addr":"172.21.15.33:6827/760777213","cluster_addr":"172.21.15.33:6829/760777213","heartbeat_back_addr":"172.21.15.33:6833/760777213","heartbeat_front_addr":"172.21.15.33:6831/760777213","state":["exists","up"]},{"osd":10,"uuid":"be17b478-3c2e-41fb-bede-958333acdf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6825","nonce":1404898482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6827","nonce":1404898482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6831","nonce":1404898482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6829","nonce":1404898482}]},"public_addr":"172.21.15.96:6825/1404898482","cluster_addr":"172.21.15.96:6827/1404898482","heartbeat_back_addr":"172.21.15.96:6831/1404898482","heartbeat_front_addr":"172.21.15.96:6829/1404898482","state":["exists","up"]},{"osd":11,"uuid":"c0903ed2-cc1e-4c18-8731-1f0ee4dd0457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6825","nonce":792352827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6827","nonce":792352827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6831","nonce":792352827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6829","nonce":792352827}]},"public_addr":"172.21.15.155:6825/792352827","cluster_addr":"172.21.15.155:6827/792352827","heartbeat_back_addr":"172.21.15.155:6831/792352827","heartbeat_front_addr":"172.21.15.155:6829/792352827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:17.942801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:42.109449+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:05.981599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:32.069574+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:57.431015+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:22.448592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:48.526515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:14.498352+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:40.175426+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:06.364006+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:32.853161+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:59.073538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:6801/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:6800/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/1067768047":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/729379453":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2586588742":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/2932814165":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6801/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2212564963":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6800/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/3275126058":"2024-03-20T02:43:39.622218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:50:58.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.916+0000 7f121a9b9640 1 -- 172.21.15.33:0/234557893 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f11fc0a2800 msgr2=0x7f11fc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:58.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.916+0000 7f121a9b9640 1 --2- 172.21.15.33:0/234557893 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f11fc0a2800 0x7f11fc0a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1204009bd0 tx=0x7f1204005f90 comp rx=0 tx=0).stop 2024-03-19T02:50:58.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.916+0000 7f121a9b9640 1 -- 172.21.15.33:0/234557893 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f12141075a0 msgr2=0x7f121419f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:58.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.916+0000 7f121a9b9640 1 --2- 172.21.15.33:0/234557893 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f12141075a0 0x7f121419f0e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f120c037040 tx=0x7f120c032820 comp rx=0 tx=0).stop 2024-03-19T02:50:58.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.916+0000 7f121a9b9640 1 -- 172.21.15.33:0/234557893 shutdown_connections 2024-03-19T02:50:58.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.916+0000 7f121a9b9640 1 --2- 172.21.15.33:0/234557893 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f11fc0a2800 0x7f11fc0a4cc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:58.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.916+0000 7f121a9b9640 1 --2- 172.21.15.33:0/234557893 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1214104c70 0x7f1214198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:58.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.916+0000 7f121a9b9640 1 --2- 172.21.15.33:0/234557893 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f12141075a0 0x7f121419f0e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:58.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.916+0000 7f121a9b9640 1 --2- 172.21.15.33:0/234557893 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1214102340 0x7f1214197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:58.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.916+0000 7f121a9b9640 1 -- 172.21.15.33:0/234557893 >> 172.21.15.33:0/234557893 conn(0x7f12140fbcd0 msgr2=0x7f1214102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:58.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.917+0000 7f121a9b9640 1 -- 172.21.15.33:0/234557893 shutdown_connections 2024-03-19T02:50:58.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:58.917+0000 7f121a9b9640 1 -- 172.21.15.33:0/234557893 wait complete. 2024-03-19T02:50:58.930 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:50:59.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.032+0000 7f2d8edc4640 1 -- 172.21.15.33:0/1374630892 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d88102310 msgr2=0x7f2d88108ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:59.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.032+0000 7f2d8edc4640 1 --2- 172.21.15.33:0/1374630892 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d88102310 0x7f2d88108ba0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2d80009be0 tx=0x7f2d8002bef0 comp rx=0 tx=0).stop 2024-03-19T02:50:59.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.032+0000 7f2d8edc4640 1 -- 172.21.15.33:0/1374630892 shutdown_connections 2024-03-19T02:50:59.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.032+0000 7f2d8edc4640 1 --2- 172.21.15.33:0/1374630892 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d88101950 0x7f2d88101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:59.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.032+0000 7f2d8edc4640 1 --2- 172.21.15.33:0/1374630892 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d88102310 0x7f2d88108ba0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:59.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.032+0000 7f2d8edc4640 1 --2- 172.21.15.33:0/1374630892 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d88100760 0x7f2d88100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:59.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.032+0000 7f2d8edc4640 1 -- 172.21.15.33:0/1374630892 >> 172.21.15.33:0/1374630892 conn(0x7f2d880fbcd0 msgr2=0x7f2d880fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:59.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.032+0000 7f2d8edc4640 1 -- 172.21.15.33:0/1374630892 shutdown_connections 2024-03-19T02:50:59.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.032+0000 7f2d8edc4640 1 -- 172.21.15.33:0/1374630892 wait complete. 2024-03-19T02:50:59.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.033+0000 7f2d8edc4640 1 Processor -- start 2024-03-19T02:50:59.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.033+0000 7f2d8edc4640 1 -- start start 2024-03-19T02:50:59.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.033+0000 7f2d8edc4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d88100760 0x7f2d88197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:59.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.033+0000 7f2d8edc4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d88101950 0x7f2d88198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:59.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.033+0000 7f2d8edc4640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d88102310 0x7f2d8819f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:59.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.033+0000 7f2d8edc4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8810c2f0 con 0x7f2d88101950 2024-03-19T02:50:59.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.033+0000 7f2d8edc4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8810c170 con 0x7f2d88100760 2024-03-19T02:50:59.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.033+0000 7f2d8edc4640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8810c470 con 0x7f2d88102310 2024-03-19T02:50:59.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.033+0000 7f2d87fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d88101950 0x7f2d88198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:59.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.033+0000 7f2d87fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d88101950 0x7f2d88198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41360/0 (socket says 172.21.15.33:41360) 2024-03-19T02:50:59.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.033+0000 7f2d87fff640 1 -- 172.21.15.33:0/1583226695 learned_addr learned my addr 172.21.15.33:0/1583226695 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:59.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.033+0000 7f2d8cb60640 1 --2- 172.21.15.33:0/1583226695 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d88100760 0x7f2d88197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:59.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.033+0000 7f2d87fff640 1 -- 172.21.15.33:0/1583226695 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d88102310 msgr2=0x7f2d8819f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:59.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.033+0000 7f2d8d361640 1 --2- 172.21.15.33:0/1583226695 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d88102310 0x7f2d8819f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:59.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.033+0000 7f2d87fff640 1 --2- 172.21.15.33:0/1583226695 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d88102310 0x7f2d8819f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:59.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.033+0000 7f2d87fff640 1 -- 172.21.15.33:0/1583226695 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d88100760 msgr2=0x7f2d88197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:59.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.033+0000 7f2d87fff640 1 --2- 172.21.15.33:0/1583226695 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d88100760 0x7f2d88197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:59.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.033+0000 7f2d87fff640 1 -- 172.21.15.33:0/1583226695 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d8819f6f0 con 0x7f2d88101950 2024-03-19T02:50:59.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.033+0000 7f2d87fff640 1 --2- 172.21.15.33:0/1583226695 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d88101950 0x7f2d88198080 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2d7400b4a0 tx=0x7f2d7400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:59.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.033+0000 7f2d85ffb640 1 -- 172.21.15.33:0/1583226695 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d74014070 con 0x7f2d88101950 2024-03-19T02:50:59.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.033+0000 7f2d8edc4640 1 -- 172.21.15.33:0/1583226695 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d8819f9e0 con 0x7f2d88101950 2024-03-19T02:50:59.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.033+0000 7f2d85ffb640 1 -- 172.21.15.33:0/1583226695 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2d74010ac0 con 0x7f2d88101950 2024-03-19T02:50:59.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.034+0000 7f2d85ffb640 1 -- 172.21.15.33:0/1583226695 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d74017c30 con 0x7f2d88101950 2024-03-19T02:50:59.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.034+0000 7f2d8edc4640 1 -- 172.21.15.33:0/1583226695 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d8819ffb0 con 0x7f2d88101950 2024-03-19T02:50:59.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.035+0000 7f2d8edc4640 1 -- 172.21.15.33:0/1583226695 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d881a0240 con 0x7f2d88101950 2024-03-19T02:50:59.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.036+0000 7f2d85ffb640 1 -- 172.21.15.33:0/1583226695 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f2d74017070 con 0x7f2d88101950 2024-03-19T02:50:59.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.036+0000 7f2d85ffb640 1 --2- 172.21.15.33:0/1583226695 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2d780a2800 0x7f2d780a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:59.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.036+0000 7f2d8cb60640 1 --2- 172.21.15.33:0/1583226695 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2d780a2800 0x7f2d780a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:59.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.036+0000 7f2d85ffb640 1 -- 172.21.15.33:0/1583226695 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8802+0+0 (secure 0 0 0) 0x7f2d740ce020 con 0x7f2d88101950 2024-03-19T02:50:59.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.037+0000 7f2d8cb60640 1 --2- 172.21.15.33:0/1583226695 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2d780a2800 0x7f2d780a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c009530 tx=0x7f2d7c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:59.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.041+0000 7f2d85ffb640 1 -- 172.21.15.33:0/1583226695 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2d7409ae80 con 0x7f2d88101950 2024-03-19T02:50:59.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.226+0000 7f2d8edc4640 1 -- 172.21.15.33:0/1583226695 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2d881a0410 con 0x7f2d88101950 2024-03-19T02:50:59.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.227+0000 7f2d85ffb640 1 -- 172.21.15.33:0/1583226695 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24056 (secure 0 0 0) 0x7f2d7409a780 con 0x7f2d88101950 2024-03-19T02:50:59.228 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:50:59.229 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":92,"fsid":"2b7a8938-e59a-11ee-95c9-87774f69a715","created":"2024-03-19T02:42:37.912179+0000","modified":"2024-03-19T02:50:57.601593+0000","last_up_change":"2024-03-19T02:50:01.539392+0000","last_in_change":"2024-03-19T02:49:40.534415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:46:09.963737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:50:50.069641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:50:51.381064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:50:55.476667+0000","flags":32773,"flags_names":"hashpspool,ec_overwrites,creating","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"090a89ab-6b7b-4317-8823-178d4212b2f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6803","nonce":1633124995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6805","nonce":1633124995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6809","nonce":1633124995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6807","nonce":1633124995}]},"public_addr":"172.21.15.33:6803/1633124995","cluster_addr":"172.21.15.33:6805/1633124995","heartbeat_back_addr":"172.21.15.33:6809/1633124995","heartbeat_front_addr":"172.21.15.33:6807/1633124995","state":["exists","up"]},{"osd":1,"uuid":"f50eec55-1956-4486-8593-f2c8fa610e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6801","nonce":340144398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6803","nonce":340144398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6807","nonce":340144398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6805","nonce":340144398}]},"public_addr":"172.21.15.96:6801/340144398","cluster_addr":"172.21.15.96:6803/340144398","heartbeat_back_addr":"172.21.15.96:6807/340144398","heartbeat_front_addr":"172.21.15.96:6805/340144398","state":["exists","up"]},{"osd":2,"uuid":"893a2d62-80d9-4e5e-bc6a-a1c8c681d652","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6801","nonce":3388743876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6803","nonce":3388743876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6807","nonce":3388743876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6805","nonce":3388743876}]},"public_addr":"172.21.15.155:6801/3388743876","cluster_addr":"172.21.15.155:6803/3388743876","heartbeat_back_addr":"172.21.15.155:6807/3388743876","heartbeat_front_addr":"172.21.15.155:6805/3388743876","state":["exists","up"]},{"osd":3,"uuid":"b07e0482-4281-4d32-9dfe-9995d7246175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6811","nonce":677007171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6813","nonce":677007171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6817","nonce":677007171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6815","nonce":677007171}]},"public_addr":"172.21.15.33:6811/677007171","cluster_addr":"172.21.15.33:6813/677007171","heartbeat_back_addr":"172.21.15.33:6817/677007171","heartbeat_front_addr":"172.21.15.33:6815/677007171","state":["exists","up"]},{"osd":4,"uuid":"a988384b-af6b-47c6-8c07-5c9673acee45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6809","nonce":3106158554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6811","nonce":3106158554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6815","nonce":3106158554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6813","nonce":3106158554}]},"public_addr":"172.21.15.96:6809/3106158554","cluster_addr":"172.21.15.96:6811/3106158554","heartbeat_back_addr":"172.21.15.96:6815/3106158554","heartbeat_front_addr":"172.21.15.96:6813/3106158554","state":["exists","up"]},{"osd":5,"uuid":"198dc992-96f8-4c52-96ee-1fd5de148bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6809","nonce":327293324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6811","nonce":327293324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6815","nonce":327293324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6813","nonce":327293324}]},"public_addr":"172.21.15.155:6809/327293324","cluster_addr":"172.21.15.155:6811/327293324","heartbeat_back_addr":"172.21.15.155:6815/327293324","heartbeat_front_addr":"172.21.15.155:6813/327293324","state":["exists","up"]},{"osd":6,"uuid":"a7278ec0-ab2e-49ed-be53-84d8b9e24177","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6819","nonce":873871509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6821","nonce":873871509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6825","nonce":873871509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6823","nonce":873871509}]},"public_addr":"172.21.15.33:6819/873871509","cluster_addr":"172.21.15.33:6821/873871509","heartbeat_back_addr":"172.21.15.33:6825/873871509","heartbeat_front_addr":"172.21.15.33:6823/873871509","state":["exists","up"]},{"osd":7,"uuid":"727e133d-aa24-4ffc-90e2-266eb2aab8c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6817","nonce":2334123891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6819","nonce":2334123891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6823","nonce":2334123891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6821","nonce":2334123891}]},"public_addr":"172.21.15.96:6817/2334123891","cluster_addr":"172.21.15.96:6819/2334123891","heartbeat_back_addr":"172.21.15.96:6823/2334123891","heartbeat_front_addr":"172.21.15.96:6821/2334123891","state":["exists","up"]},{"osd":8,"uuid":"bb20eec0-39e5-4b86-9db5-486b1840cdad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6817","nonce":3498516903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6819","nonce":3498516903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6823","nonce":3498516903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6821","nonce":3498516903}]},"public_addr":"172.21.15.155:6817/3498516903","cluster_addr":"172.21.15.155:6819/3498516903","heartbeat_back_addr":"172.21.15.155:6823/3498516903","heartbeat_front_addr":"172.21.15.155:6821/3498516903","state":["exists","up"]},{"osd":9,"uuid":"179c6228-3e23-4f65-91a6-1ef0005d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6827","nonce":760777213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6829","nonce":760777213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6833","nonce":760777213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6831","nonce":760777213}]},"public_addr":"172.21.15.33:6827/760777213","cluster_addr":"172.21.15.33:6829/760777213","heartbeat_back_addr":"172.21.15.33:6833/760777213","heartbeat_front_addr":"172.21.15.33:6831/760777213","state":["exists","up"]},{"osd":10,"uuid":"be17b478-3c2e-41fb-bede-958333acdf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6825","nonce":1404898482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6827","nonce":1404898482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6831","nonce":1404898482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6829","nonce":1404898482}]},"public_addr":"172.21.15.96:6825/1404898482","cluster_addr":"172.21.15.96:6827/1404898482","heartbeat_back_addr":"172.21.15.96:6831/1404898482","heartbeat_front_addr":"172.21.15.96:6829/1404898482","state":["exists","up"]},{"osd":11,"uuid":"c0903ed2-cc1e-4c18-8731-1f0ee4dd0457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6825","nonce":792352827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6827","nonce":792352827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6831","nonce":792352827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6829","nonce":792352827}]},"public_addr":"172.21.15.155:6825/792352827","cluster_addr":"172.21.15.155:6827/792352827","heartbeat_back_addr":"172.21.15.155:6831/792352827","heartbeat_front_addr":"172.21.15.155:6829/792352827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:17.942801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:42.109449+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:05.981599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:32.069574+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:57.431015+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:22.448592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:48.526515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:14.498352+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:40.175426+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:06.364006+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:32.853161+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:59.073538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:6801/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:6800/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/1067768047":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/729379453":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2586588742":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/2932814165":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6801/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2212564963":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6800/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/3275126058":"2024-03-20T02:43:39.622218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:50:59.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.229+0000 7f2d8edc4640 1 -- 172.21.15.33:0/1583226695 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2d780a2800 msgr2=0x7f2d780a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:59.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.229+0000 7f2d8edc4640 1 --2- 172.21.15.33:0/1583226695 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2d780a2800 0x7f2d780a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c009530 tx=0x7f2d7c009260 comp rx=0 tx=0).stop 2024-03-19T02:50:59.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.229+0000 7f2d8edc4640 1 -- 172.21.15.33:0/1583226695 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d88101950 msgr2=0x7f2d88198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:59.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.229+0000 7f2d8edc4640 1 --2- 172.21.15.33:0/1583226695 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d88101950 0x7f2d88198080 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2d7400b4a0 tx=0x7f2d7400b960 comp rx=0 tx=0).stop 2024-03-19T02:50:59.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.229+0000 7f2d8edc4640 1 -- 172.21.15.33:0/1583226695 shutdown_connections 2024-03-19T02:50:59.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.229+0000 7f2d8edc4640 1 --2- 172.21.15.33:0/1583226695 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2d780a2800 0x7f2d780a4cc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:59.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.229+0000 7f2d8edc4640 1 --2- 172.21.15.33:0/1583226695 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d88102310 0x7f2d8819f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:59.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.229+0000 7f2d8edc4640 1 --2- 172.21.15.33:0/1583226695 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d88101950 0x7f2d88198080 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:59.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.229+0000 7f2d8edc4640 1 --2- 172.21.15.33:0/1583226695 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d88100760 0x7f2d88197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:59.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.229+0000 7f2d8edc4640 1 -- 172.21.15.33:0/1583226695 >> 172.21.15.33:0/1583226695 conn(0x7f2d880fbcd0 msgr2=0x7f2d880fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:59.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.229+0000 7f2d8edc4640 1 -- 172.21.15.33:0/1583226695 shutdown_connections 2024-03-19T02:50:59.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.229+0000 7f2d8edc4640 1 -- 172.21.15.33:0/1583226695 wait complete. 2024-03-19T02:50:59.260 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:50:59.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.362+0000 7fc012ed1640 1 -- 172.21.15.33:0/296987522 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc00c101930 msgr2=0x7fc00c101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:59.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.362+0000 7fc012ed1640 1 --2- 172.21.15.33:0/296987522 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc00c101930 0x7fc00c101db0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbff8009900 tx=0x7fbff802bc30 comp rx=0 tx=0).stop 2024-03-19T02:50:59.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.362+0000 7fc012ed1640 1 -- 172.21.15.33:0/296987522 shutdown_connections 2024-03-19T02:50:59.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.362+0000 7fc012ed1640 1 --2- 172.21.15.33:0/296987522 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc00c1022f0 0x7fc00c108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:59.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.362+0000 7fc012ed1640 1 --2- 172.21.15.33:0/296987522 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc00c101930 0x7fc00c101db0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:59.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.362+0000 7fc012ed1640 1 --2- 172.21.15.33:0/296987522 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc00c100740 0x7fc00c100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:59.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.362+0000 7fc012ed1640 1 -- 172.21.15.33:0/296987522 >> 172.21.15.33:0/296987522 conn(0x7fc00c0fbcb0 msgr2=0x7fc00c0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:59.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.362+0000 7fc012ed1640 1 -- 172.21.15.33:0/296987522 shutdown_connections 2024-03-19T02:50:59.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.362+0000 7fc012ed1640 1 -- 172.21.15.33:0/296987522 wait complete. 2024-03-19T02:50:59.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.363+0000 7fc012ed1640 1 Processor -- start 2024-03-19T02:50:59.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.363+0000 7fc012ed1640 1 -- start start 2024-03-19T02:50:59.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.363+0000 7fc012ed1640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc00c100740 0x7fc00c1046d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:59.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.363+0000 7fc012ed1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc00c101930 0x7fc00c102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:59.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.363+0000 7fc012ed1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc00c1022f0 0x7fc00c103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:59.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.363+0000 7fc012ed1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc00c10c250 con 0x7fc00c101930 2024-03-19T02:50:59.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.363+0000 7fc012ed1640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc00c10c0d0 con 0x7fc00c1022f0 2024-03-19T02:50:59.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.363+0000 7fc012ed1640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc00c10c3d0 con 0x7fc00c100740 2024-03-19T02:50:59.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.363+0000 7fc00bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc00c101930 0x7fc00c102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:59.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.363+0000 7fc00bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc00c101930 0x7fc00c102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41388/0 (socket says 172.21.15.33:41388) 2024-03-19T02:50:59.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.363+0000 7fc00bfff640 1 -- 172.21.15.33:0/956107841 learned_addr learned my addr 172.21.15.33:0/956107841 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:59.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.363+0000 7fc01146e640 1 --2- 172.21.15.33:0/956107841 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc00c1022f0 0x7fc00c103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:59.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.363+0000 7fc010c6d640 1 --2- 172.21.15.33:0/956107841 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc00c100740 0x7fc00c1046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:59.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.363+0000 7fc00bfff640 1 -- 172.21.15.33:0/956107841 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc00c100740 msgr2=0x7fc00c1046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:59.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.363+0000 7fc00bfff640 1 --2- 172.21.15.33:0/956107841 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc00c100740 0x7fc00c1046d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:59.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.363+0000 7fc00bfff640 1 -- 172.21.15.33:0/956107841 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc00c1022f0 msgr2=0x7fc00c103250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:59.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.363+0000 7fc00bfff640 1 --2- 172.21.15.33:0/956107841 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc00c1022f0 0x7fc00c103250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:59.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.363+0000 7fc00bfff640 1 -- 172.21.15.33:0/956107841 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc00c103ae0 con 0x7fc00c101930 2024-03-19T02:50:59.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.363+0000 7fc00bfff640 1 --2- 172.21.15.33:0/956107841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc00c101930 0x7fc00c102d10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbff8006010 tx=0x7fbff8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:59.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.364+0000 7fc009ffb640 1 -- 172.21.15.33:0/956107841 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff8043020 con 0x7fc00c101930 2024-03-19T02:50:59.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.364+0000 7fc009ffb640 1 -- 172.21.15.33:0/956107841 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbff8004440 con 0x7fc00c101930 2024-03-19T02:50:59.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.364+0000 7fc012ed1640 1 -- 172.21.15.33:0/956107841 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc00c1a1b00 con 0x7fc00c101930 2024-03-19T02:50:59.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.364+0000 7fc009ffb640 1 -- 172.21.15.33:0/956107841 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff8003bf0 con 0x7fc00c101930 2024-03-19T02:50:59.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.364+0000 7fc012ed1640 1 -- 172.21.15.33:0/956107841 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc00c1a20a0 con 0x7fc00c101930 2024-03-19T02:50:59.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.365+0000 7fc012ed1640 1 -- 172.21.15.33:0/956107841 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc00c1a2270 con 0x7fc00c101930 2024-03-19T02:50:59.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.366+0000 7fc009ffb640 1 -- 172.21.15.33:0/956107841 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fbff8046020 con 0x7fc00c101930 2024-03-19T02:50:59.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.366+0000 7fc009ffb640 1 --2- 172.21.15.33:0/956107841 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbffc0a2880 0x7fbffc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:59.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.367+0000 7fc010c6d640 1 --2- 172.21.15.33:0/956107841 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbffc0a2880 0x7fbffc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:59.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.367+0000 7fc009ffb640 1 -- 172.21.15.33:0/956107841 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7fbff80f15d0 con 0x7fc00c101930 2024-03-19T02:50:59.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.367+0000 7fc010c6d640 1 --2- 172.21.15.33:0/956107841 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbffc0a2880 0x7fbffc0a4d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc000005fd0 tx=0x7fc000005ce0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:59.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.370+0000 7fc009ffb640 1 -- 172.21.15.33:0/956107841 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbff804d020 con 0x7fc00c101930 2024-03-19T02:50:59.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:59 smithi155 ceph-mon[35804]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-19T02:50:59.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:59 smithi155 ceph-mon[35804]: osdmap e92: 12 total, 12 up, 12 in 2024-03-19T02:50:59.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:59 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2180227573' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-19T02:50:59.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:59 smithi155 ceph-mon[35804]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T02:50:59.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:59 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/177750341' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:59.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:59 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/234557893' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:50:59.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:59 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1583226695' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:50:59.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:50:59 smithi155 ceph-mon[35804]: osdmap e93: 12 total, 12 up, 12 in 2024-03-19T02:50:59.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:59 smithi033 ceph-mon[31973]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-19T02:50:59.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:59 smithi033 ceph-mon[31973]: osdmap e92: 12 total, 12 up, 12 in 2024-03-19T02:50:59.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:59 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2180227573' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-19T02:50:59.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:59 smithi033 ceph-mon[31973]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T02:50:59.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:59 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/177750341' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:59.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:59 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/234557893' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:50:59.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:59 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1583226695' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:50:59.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:50:59 smithi033 ceph-mon[31973]: osdmap e93: 12 total, 12 up, 12 in 2024-03-19T02:50:59.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.591+0000 7fc012ed1640 1 -- 172.21.15.33:0/956107841 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc00c1a2440 con 0x7fc00c101930 2024-03-19T02:50:59.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.592+0000 7fc009ffb640 1 -- 172.21.15.33:0/956107841 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11794 (secure 0 0 0) 0x7fbff80be3d0 con 0x7fc00c101930 2024-03-19T02:50:59.593 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:50:59.593 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24436,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24457,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T02:50:52.201798+0000","modified":"2024-03-19T02:50:57.601788+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24509},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/3731232169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":3731232169},{"type":"v1","addr":"172.21.15.96:6839","nonce":3731232169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24509,"qdb_cluster":[24509]},"id":1}]} 2024-03-19T02:50:59.593 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 17 2024-03-19T02:50:59.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.594+0000 7fc012ed1640 1 -- 172.21.15.33:0/956107841 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbffc0a2880 msgr2=0x7fbffc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:59.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.594+0000 7fc012ed1640 1 --2- 172.21.15.33:0/956107841 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbffc0a2880 0x7fbffc0a4d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc000005fd0 tx=0x7fc000005ce0 comp rx=0 tx=0).stop 2024-03-19T02:50:59.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.594+0000 7fc012ed1640 1 -- 172.21.15.33:0/956107841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc00c101930 msgr2=0x7fc00c102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:59.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.594+0000 7fc012ed1640 1 --2- 172.21.15.33:0/956107841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc00c101930 0x7fc00c102d10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbff8006010 tx=0x7fbff8004040 comp rx=0 tx=0).stop 2024-03-19T02:50:59.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.594+0000 7fc012ed1640 1 -- 172.21.15.33:0/956107841 shutdown_connections 2024-03-19T02:50:59.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.594+0000 7fc012ed1640 1 --2- 172.21.15.33:0/956107841 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbffc0a2880 0x7fbffc0a4d40 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:59.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.594+0000 7fc012ed1640 1 --2- 172.21.15.33:0/956107841 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc00c1022f0 0x7fc00c103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:59.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.594+0000 7fc012ed1640 1 --2- 172.21.15.33:0/956107841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc00c101930 0x7fc00c102d10 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:59.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.594+0000 7fc012ed1640 1 --2- 172.21.15.33:0/956107841 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc00c100740 0x7fc00c1046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:59.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.594+0000 7fc012ed1640 1 -- 172.21.15.33:0/956107841 >> 172.21.15.33:0/956107841 conn(0x7fc00c0fbcb0 msgr2=0x7fc00c0fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:59.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.594+0000 7fc012ed1640 1 -- 172.21.15.33:0/956107841 shutdown_connections 2024-03-19T02:50:59.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.594+0000 7fc012ed1640 1 -- 172.21.15.33:0/956107841 wait complete. 2024-03-19T02:50:59.607 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:50:59.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.710+0000 7ff51e06a640 1 -- 172.21.15.33:0/3843647093 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff518102320 msgr2=0x7ff518104710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:59.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.710+0000 7ff51e06a640 1 --2- 172.21.15.33:0/3843647093 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff518102320 0x7ff518104710 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff508009900 tx=0x7ff50802bc30 comp rx=0 tx=0).stop 2024-03-19T02:50:59.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.710+0000 7ff51e06a640 1 -- 172.21.15.33:0/3843647093 shutdown_connections 2024-03-19T02:50:59.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.710+0000 7ff51e06a640 1 --2- 172.21.15.33:0/3843647093 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff518104c50 0x7ff518107040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:59.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.710+0000 7ff51e06a640 1 --2- 172.21.15.33:0/3843647093 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff518107580 0x7ff518109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:59.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.710+0000 7ff51e06a640 1 --2- 172.21.15.33:0/3843647093 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff518102320 0x7ff518104710 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:59.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.710+0000 7ff51e06a640 1 -- 172.21.15.33:0/3843647093 >> 172.21.15.33:0/3843647093 conn(0x7ff5180fbcb0 msgr2=0x7ff5180fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:59.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.710+0000 7ff51e06a640 1 -- 172.21.15.33:0/3843647093 shutdown_connections 2024-03-19T02:50:59.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.710+0000 7ff51e06a640 1 -- 172.21.15.33:0/3843647093 wait complete. 2024-03-19T02:50:59.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.711+0000 7ff51e06a640 1 Processor -- start 2024-03-19T02:50:59.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.711+0000 7ff51e06a640 1 -- start start 2024-03-19T02:50:59.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.711+0000 7ff51e06a640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff518102320 0x7ff518197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:59.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.711+0000 7ff51e06a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff518104c50 0x7ff518198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:59.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.711+0000 7ff51e06a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff518107580 0x7ff51819f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:59.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.711+0000 7ff51e06a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff51810d3e0 con 0x7ff518107580 2024-03-19T02:50:59.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.711+0000 7ff51e06a640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff51810d260 con 0x7ff518104c50 2024-03-19T02:50:59.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.711+0000 7ff51e06a640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff51810d560 con 0x7ff518102320 2024-03-19T02:50:59.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.711+0000 7ff517fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff518107580 0x7ff51819f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:59.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.711+0000 7ff517fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff518107580 0x7ff51819f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41398/0 (socket says 172.21.15.33:41398) 2024-03-19T02:50:59.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.711+0000 7ff517fff640 1 -- 172.21.15.33:0/997474931 learned_addr learned my addr 172.21.15.33:0/997474931 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:50:59.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.711+0000 7ff516ffd640 1 --2- 172.21.15.33:0/997474931 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff518104c50 0x7ff518198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:59.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.711+0000 7ff517fff640 1 -- 172.21.15.33:0/997474931 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff518102320 msgr2=0x7ff518197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:59.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.711+0000 7ff517fff640 1 --2- 172.21.15.33:0/997474931 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff518102320 0x7ff518197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:59.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.711+0000 7ff517fff640 1 -- 172.21.15.33:0/997474931 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff518104c50 msgr2=0x7ff518198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:59.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.711+0000 7ff517fff640 1 --2- 172.21.15.33:0/997474931 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff518104c50 0x7ff518198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:59.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.711+0000 7ff517fff640 1 -- 172.21.15.33:0/997474931 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff51819f730 con 0x7ff518107580 2024-03-19T02:50:59.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.712+0000 7ff517fff640 1 --2- 172.21.15.33:0/997474931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff518107580 0x7ff51819f0c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff51000eb10 tx=0x7ff51000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:59.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.712+0000 7ff514ff9640 1 -- 172.21.15.33:0/997474931 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff51000ce20 con 0x7ff518107580 2024-03-19T02:50:59.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.712+0000 7ff514ff9640 1 -- 172.21.15.33:0/997474931 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff510005020 con 0x7ff518107580 2024-03-19T02:50:59.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.712+0000 7ff51e06a640 1 -- 172.21.15.33:0/997474931 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff51819fa20 con 0x7ff518107580 2024-03-19T02:50:59.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.712+0000 7ff514ff9640 1 -- 172.21.15.33:0/997474931 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff510010690 con 0x7ff518107580 2024-03-19T02:50:59.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.712+0000 7ff51e06a640 1 -- 172.21.15.33:0/997474931 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff51819ffa0 con 0x7ff518107580 2024-03-19T02:50:59.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.713+0000 7ff51e06a640 1 -- 172.21.15.33:0/997474931 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5180fd7a0 con 0x7ff518107580 2024-03-19T02:50:59.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.714+0000 7ff514ff9640 1 -- 172.21.15.33:0/997474931 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff5100051d0 con 0x7ff518107580 2024-03-19T02:50:59.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.715+0000 7ff514ff9640 1 --2- 172.21.15.33:0/997474931 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff5000a2800 0x7ff5000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:50:59.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.715+0000 7ff5177fe640 1 --2- 172.21.15.33:0/997474931 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff5000a2800 0x7ff5000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:50:59.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.715+0000 7ff514ff9640 1 -- 172.21.15.33:0/997474931 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7ff5100cf550 con 0x7ff518107580 2024-03-19T02:50:59.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.715+0000 7ff5177fe640 1 --2- 172.21.15.33:0/997474931 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff5000a2800 0x7ff5000a4cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff508036040 tx=0x7ff508000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:50:59.718 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:59 smithi096 ceph-mon[35946]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-19T02:50:59.719 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:59 smithi096 ceph-mon[35946]: osdmap e92: 12 total, 12 up, 12 in 2024-03-19T02:50:59.719 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:59 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2180227573' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-19T02:50:59.719 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:59 smithi096 ceph-mon[35946]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T02:50:59.719 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:59 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/177750341' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:50:59.719 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:59 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/234557893' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:50:59.719 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:59 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1583226695' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:50:59.719 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:50:59 smithi096 ceph-mon[35946]: osdmap e93: 12 total, 12 up, 12 in 2024-03-19T02:50:59.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.718+0000 7ff514ff9640 1 -- 172.21.15.33:0/997474931 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff510053740 con 0x7ff518107580 2024-03-19T02:50:59.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.905+0000 7ff51e06a640 1 -- 172.21.15.33:0/997474931 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff518062420 con 0x7ff518107580 2024-03-19T02:50:59.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.907+0000 7ff514ff9640 1 -- 172.21.15.33:0/997474931 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24043 (secure 0 0 0) 0x7ff510016330 con 0x7ff518107580 2024-03-19T02:50:59.908 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:50:59.908 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":93,"fsid":"2b7a8938-e59a-11ee-95c9-87774f69a715","created":"2024-03-19T02:42:37.912179+0000","modified":"2024-03-19T02:50:59.241668+0000","last_up_change":"2024-03-19T02:50:01.539392+0000","last_in_change":"2024-03-19T02:49:40.534415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:46:09.963737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:50:50.069641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:50:51.381064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:50:55.476667+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"090a89ab-6b7b-4317-8823-178d4212b2f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6803","nonce":1633124995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6805","nonce":1633124995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6809","nonce":1633124995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6807","nonce":1633124995}]},"public_addr":"172.21.15.33:6803/1633124995","cluster_addr":"172.21.15.33:6805/1633124995","heartbeat_back_addr":"172.21.15.33:6809/1633124995","heartbeat_front_addr":"172.21.15.33:6807/1633124995","state":["exists","up"]},{"osd":1,"uuid":"f50eec55-1956-4486-8593-f2c8fa610e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6801","nonce":340144398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6803","nonce":340144398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6807","nonce":340144398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6805","nonce":340144398}]},"public_addr":"172.21.15.96:6801/340144398","cluster_addr":"172.21.15.96:6803/340144398","heartbeat_back_addr":"172.21.15.96:6807/340144398","heartbeat_front_addr":"172.21.15.96:6805/340144398","state":["exists","up"]},{"osd":2,"uuid":"893a2d62-80d9-4e5e-bc6a-a1c8c681d652","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6801","nonce":3388743876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6803","nonce":3388743876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6807","nonce":3388743876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6805","nonce":3388743876}]},"public_addr":"172.21.15.155:6801/3388743876","cluster_addr":"172.21.15.155:6803/3388743876","heartbeat_back_addr":"172.21.15.155:6807/3388743876","heartbeat_front_addr":"172.21.15.155:6805/3388743876","state":["exists","up"]},{"osd":3,"uuid":"b07e0482-4281-4d32-9dfe-9995d7246175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6811","nonce":677007171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6813","nonce":677007171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6817","nonce":677007171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6815","nonce":677007171}]},"public_addr":"172.21.15.33:6811/677007171","cluster_addr":"172.21.15.33:6813/677007171","heartbeat_back_addr":"172.21.15.33:6817/677007171","heartbeat_front_addr":"172.21.15.33:6815/677007171","state":["exists","up"]},{"osd":4,"uuid":"a988384b-af6b-47c6-8c07-5c9673acee45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6809","nonce":3106158554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6811","nonce":3106158554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6815","nonce":3106158554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6813","nonce":3106158554}]},"public_addr":"172.21.15.96:6809/3106158554","cluster_addr":"172.21.15.96:6811/3106158554","heartbeat_back_addr":"172.21.15.96:6815/3106158554","heartbeat_front_addr":"172.21.15.96:6813/3106158554","state":["exists","up"]},{"osd":5,"uuid":"198dc992-96f8-4c52-96ee-1fd5de148bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6809","nonce":327293324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6811","nonce":327293324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6815","nonce":327293324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6813","nonce":327293324}]},"public_addr":"172.21.15.155:6809/327293324","cluster_addr":"172.21.15.155:6811/327293324","heartbeat_back_addr":"172.21.15.155:6815/327293324","heartbeat_front_addr":"172.21.15.155:6813/327293324","state":["exists","up"]},{"osd":6,"uuid":"a7278ec0-ab2e-49ed-be53-84d8b9e24177","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6819","nonce":873871509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6821","nonce":873871509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6825","nonce":873871509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6823","nonce":873871509}]},"public_addr":"172.21.15.33:6819/873871509","cluster_addr":"172.21.15.33:6821/873871509","heartbeat_back_addr":"172.21.15.33:6825/873871509","heartbeat_front_addr":"172.21.15.33:6823/873871509","state":["exists","up"]},{"osd":7,"uuid":"727e133d-aa24-4ffc-90e2-266eb2aab8c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6817","nonce":2334123891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6819","nonce":2334123891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6823","nonce":2334123891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6821","nonce":2334123891}]},"public_addr":"172.21.15.96:6817/2334123891","cluster_addr":"172.21.15.96:6819/2334123891","heartbeat_back_addr":"172.21.15.96:6823/2334123891","heartbeat_front_addr":"172.21.15.96:6821/2334123891","state":["exists","up"]},{"osd":8,"uuid":"bb20eec0-39e5-4b86-9db5-486b1840cdad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6817","nonce":3498516903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6819","nonce":3498516903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6823","nonce":3498516903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6821","nonce":3498516903}]},"public_addr":"172.21.15.155:6817/3498516903","cluster_addr":"172.21.15.155:6819/3498516903","heartbeat_back_addr":"172.21.15.155:6823/3498516903","heartbeat_front_addr":"172.21.15.155:6821/3498516903","state":["exists","up"]},{"osd":9,"uuid":"179c6228-3e23-4f65-91a6-1ef0005d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6827","nonce":760777213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6829","nonce":760777213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6833","nonce":760777213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6831","nonce":760777213}]},"public_addr":"172.21.15.33:6827/760777213","cluster_addr":"172.21.15.33:6829/760777213","heartbeat_back_addr":"172.21.15.33:6833/760777213","heartbeat_front_addr":"172.21.15.33:6831/760777213","state":["exists","up"]},{"osd":10,"uuid":"be17b478-3c2e-41fb-bede-958333acdf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6825","nonce":1404898482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6827","nonce":1404898482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6831","nonce":1404898482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6829","nonce":1404898482}]},"public_addr":"172.21.15.96:6825/1404898482","cluster_addr":"172.21.15.96:6827/1404898482","heartbeat_back_addr":"172.21.15.96:6831/1404898482","heartbeat_front_addr":"172.21.15.96:6829/1404898482","state":["exists","up"]},{"osd":11,"uuid":"c0903ed2-cc1e-4c18-8731-1f0ee4dd0457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6825","nonce":792352827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6827","nonce":792352827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6831","nonce":792352827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6829","nonce":792352827}]},"public_addr":"172.21.15.155:6825/792352827","cluster_addr":"172.21.15.155:6827/792352827","heartbeat_back_addr":"172.21.15.155:6831/792352827","heartbeat_front_addr":"172.21.15.155:6829/792352827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:17.942801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:42.109449+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:05.981599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:32.069574+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:57.431015+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:22.448592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:48.526515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:14.498352+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:40.175426+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:06.364006+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:32.853161+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:59.073538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:6801/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:6800/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/1067768047":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/729379453":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2586588742":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/2932814165":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6801/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2212564963":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6800/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/3275126058":"2024-03-20T02:43:39.622218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:50:59.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.909+0000 7ff51e06a640 1 -- 172.21.15.33:0/997474931 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff5000a2800 msgr2=0x7ff5000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:59.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.909+0000 7ff51e06a640 1 --2- 172.21.15.33:0/997474931 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff5000a2800 0x7ff5000a4cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff508036040 tx=0x7ff508000bc0 comp rx=0 tx=0).stop 2024-03-19T02:50:59.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.909+0000 7ff51e06a640 1 -- 172.21.15.33:0/997474931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff518107580 msgr2=0x7ff51819f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:50:59.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.909+0000 7ff51e06a640 1 --2- 172.21.15.33:0/997474931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff518107580 0x7ff51819f0c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff51000eb10 tx=0x7ff51000ef10 comp rx=0 tx=0).stop 2024-03-19T02:50:59.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.909+0000 7ff51e06a640 1 -- 172.21.15.33:0/997474931 shutdown_connections 2024-03-19T02:50:59.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.909+0000 7ff51e06a640 1 --2- 172.21.15.33:0/997474931 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff5000a2800 0x7ff5000a4cc0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:59.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.909+0000 7ff51e06a640 1 --2- 172.21.15.33:0/997474931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff518107580 0x7ff51819f0c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:59.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.909+0000 7ff51e06a640 1 --2- 172.21.15.33:0/997474931 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff518104c50 0x7ff518198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:59.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.909+0000 7ff51e06a640 1 --2- 172.21.15.33:0/997474931 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff518102320 0x7ff518197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:50:59.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.909+0000 7ff51e06a640 1 -- 172.21.15.33:0/997474931 >> 172.21.15.33:0/997474931 conn(0x7ff5180fbcb0 msgr2=0x7ff518102e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:50:59.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.909+0000 7ff51e06a640 1 -- 172.21.15.33:0/997474931 shutdown_connections 2024-03-19T02:50:59.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:50:59.909+0000 7ff51e06a640 1 -- 172.21.15.33:0/997474931 wait complete. 2024-03-19T02:50:59.922 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T02:50:52.201798+0000', 'modified': '2024-03-19T02:50:57.601788+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24509}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24509': {'gid': 24509, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.96:6839/3731232169', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6838', 'nonce': 3731232169}, {'type': 'v1', 'addr': '172.21.15.96:6839', 'nonce': 3731232169}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24509, 'qdb_cluster': [24509]} 2024-03-19T02:50:59.922 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-19T02:50:59.922 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-19T02:51:00.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.025+0000 7f7bfe553640 1 -- 172.21.15.33:0/1547777579 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bf81022d0 msgr2=0x7f7bf8108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:00.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.025+0000 7f7bfe553640 1 --2- 172.21.15.33:0/1547777579 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bf81022d0 0x7f7bf8108b60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7bf0009ca0 tx=0x7f7bf002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.025+0000 7f7bfe553640 1 -- 172.21.15.33:0/1547777579 shutdown_connections 2024-03-19T02:51:00.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.025+0000 7f7bfe553640 1 --2- 172.21.15.33:0/1547777579 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bf81022d0 0x7f7bf8108b60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.025+0000 7f7bfe553640 1 --2- 172.21.15.33:0/1547777579 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7bf8101910 0x7f7bf8101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.025+0000 7f7bfe553640 1 --2- 172.21.15.33:0/1547777579 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7bf8100720 0x7f7bf8100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.025+0000 7f7bfe553640 1 -- 172.21.15.33:0/1547777579 >> 172.21.15.33:0/1547777579 conn(0x7f7bf80fbcb0 msgr2=0x7f7bf80fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:00.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.025+0000 7f7bfe553640 1 -- 172.21.15.33:0/1547777579 shutdown_connections 2024-03-19T02:51:00.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.025+0000 7f7bfe553640 1 -- 172.21.15.33:0/1547777579 wait complete. 2024-03-19T02:51:00.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.026+0000 7f7bfe553640 1 Processor -- start 2024-03-19T02:51:00.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.026+0000 7f7bfe553640 1 -- start start 2024-03-19T02:51:00.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.026+0000 7f7bfe553640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7bf8100720 0x7f7bf8197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:00.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.026+0000 7f7bfe553640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bf8101910 0x7f7bf8198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:00.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.026+0000 7f7bfe553640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7bf81022d0 0x7f7bf819f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:00.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.026+0000 7f7bfe553640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf810c250 con 0x7f7bf8101910 2024-03-19T02:51:00.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.026+0000 7f7bfe553640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf810c0d0 con 0x7f7bf81022d0 2024-03-19T02:51:00.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.026+0000 7f7bfe553640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf810c3d0 con 0x7f7bf8100720 2024-03-19T02:51:00.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.026+0000 7f7bf77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bf8101910 0x7f7bf8198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:00.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.026+0000 7f7bf77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bf8101910 0x7f7bf8198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41428/0 (socket says 172.21.15.33:41428) 2024-03-19T02:51:00.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.026+0000 7f7bf77fe640 1 -- 172.21.15.33:0/2050599817 learned_addr learned my addr 172.21.15.33:0/2050599817 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:00.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.026+0000 7f7bf7fff640 1 --2- 172.21.15.33:0/2050599817 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7bf8100720 0x7f7bf8197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:00.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.026+0000 7f7bfcaf0640 1 --2- 172.21.15.33:0/2050599817 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7bf81022d0 0x7f7bf819f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:00.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.027+0000 7f7bf77fe640 1 -- 172.21.15.33:0/2050599817 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7bf8100720 msgr2=0x7f7bf8197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:00.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.027+0000 7f7bf77fe640 1 --2- 172.21.15.33:0/2050599817 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7bf8100720 0x7f7bf8197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.027+0000 7f7bf77fe640 1 -- 172.21.15.33:0/2050599817 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7bf81022d0 msgr2=0x7f7bf819f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:00.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.027+0000 7f7bf77fe640 1 --2- 172.21.15.33:0/2050599817 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7bf81022d0 0x7f7bf819f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.027+0000 7f7bf77fe640 1 -- 172.21.15.33:0/2050599817 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bf819f6b0 con 0x7f7bf8101910 2024-03-19T02:51:00.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.027+0000 7f7bfcaf0640 1 --2- 172.21.15.33:0/2050599817 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7bf81022d0 0x7f7bf819f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:00.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.027+0000 7f7bf77fe640 1 --2- 172.21.15.33:0/2050599817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bf8101910 0x7f7bf8198020 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7bec00e9f0 tx=0x7f7bec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:00.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.027+0000 7f7bf57fa640 1 -- 172.21.15.33:0/2050599817 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bec00cd80 con 0x7f7bf8101910 2024-03-19T02:51:00.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.027+0000 7f7bf57fa640 1 -- 172.21.15.33:0/2050599817 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7bec0108e0 con 0x7f7bf8101910 2024-03-19T02:51:00.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.027+0000 7f7bfe553640 1 -- 172.21.15.33:0/2050599817 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bf819f9a0 con 0x7f7bf8101910 2024-03-19T02:51:00.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.027+0000 7f7bf57fa640 1 -- 172.21.15.33:0/2050599817 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bec018920 con 0x7f7bf8101910 2024-03-19T02:51:00.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.027+0000 7f7bfe553640 1 -- 172.21.15.33:0/2050599817 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bf819fef0 con 0x7f7bf8101910 2024-03-19T02:51:00.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.029+0000 7f7bf57fa640 1 -- 172.21.15.33:0/2050599817 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f7bec010a90 con 0x7f7bf8101910 2024-03-19T02:51:00.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.029+0000 7f7c0121b740 1 Processor -- start 2024-03-19T02:51:00.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.029+0000 7f7c0121b740 1 -- start start 2024-03-19T02:51:00.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.030+0000 7f7bf57fa640 1 --2- 172.21.15.33:0/2050599817 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7be00a2800 0x7f7be00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:00.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.030+0000 7f7c0121b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eca3136ff0 0x55eca3137410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:00.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.030+0000 7f7c0121b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eca31379e0 0x55eca313c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:00.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.030+0000 7f7c0121b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eca313cb50 0x55eca313efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:00.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.030+0000 7f7c0121b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55eca309bde0 con 0x55eca31379e0 2024-03-19T02:51:00.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.030+0000 7f7c0121b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55eca3044c80 con 0x55eca313cb50 2024-03-19T02:51:00.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.030+0000 7f7bf57fa640 1 -- 172.21.15.33:0/2050599817 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7f7bec0cef90 con 0x7f7bf8101910 2024-03-19T02:51:00.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.030+0000 7f7c0121b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55eca3097e80 con 0x55eca3136ff0 2024-03-19T02:51:00.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.030+0000 7f7bfcaf0640 1 --2- 172.21.15.33:0/2050599817 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7be00a2800 0x7f7be00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:00.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.030+0000 7f7bf77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eca31379e0 0x55eca313c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:00.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.030+0000 7f7bf77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eca31379e0 0x55eca313c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41438/0 (socket says 172.21.15.33:41438) 2024-03-19T02:51:00.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.030+0000 7f7bf77fe640 1 -- 172.21.15.33:0/332738551 learned_addr learned my addr 172.21.15.33:0/332738551 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:00.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.030+0000 7f7bfcaf0640 1 --2- 172.21.15.33:0/332738551 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eca313cb50 0x55eca313efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:00.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.030+0000 7f7bf7fff640 1 --2- 172.21.15.33:0/332738551 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eca3136ff0 0x55eca3137410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:00.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.030+0000 7f7bf77fe640 1 -- 172.21.15.33:0/332738551 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eca3136ff0 msgr2=0x55eca3137410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:00.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.030+0000 7f7bf77fe640 1 --2- 172.21.15.33:0/332738551 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eca3136ff0 0x55eca3137410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.030+0000 7f7bf77fe640 1 -- 172.21.15.33:0/332738551 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eca313cb50 msgr2=0x55eca313efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:00.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.030+0000 7f7bf77fe640 1 --2- 172.21.15.33:0/332738551 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eca313cb50 0x55eca313efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.030+0000 7f7bf77fe640 1 -- 172.21.15.33:0/332738551 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eca3056fb0 con 0x55eca31379e0 2024-03-19T02:51:00.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.030+0000 7f7bf7fff640 1 --2- 172.21.15.33:0/332738551 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eca3136ff0 0x55eca3137410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:51:00.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.030+0000 7f7bfcaf0640 1 --2- 172.21.15.33:0/2050599817 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7be00a2800 0x7f7be00a4cc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7bf0006010 tx=0x7f7bf0009930 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:00.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.030+0000 7f7bf77fe640 1 --2- 172.21.15.33:0/332738551 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eca31379e0 0x55eca313c4e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7bec09b730 tx=0x7f7bec010ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40ab6731675b3018 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:00.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.030+0000 7f7bde7fc640 1 -- 172.21.15.33:0/332738551 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bec0a3e90 con 0x55eca31379e0 2024-03-19T02:51:00.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.031+0000 7f7bde7fc640 1 -- 172.21.15.33:0/332738551 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7bec099aa0 con 0x55eca31379e0 2024-03-19T02:51:00.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.031+0000 7f7bde7fc640 1 -- 172.21.15.33:0/332738551 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bec0974f0 con 0x55eca31379e0 2024-03-19T02:51:00.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.031+0000 7f7c0121b740 1 -- 172.21.15.33:0/332738551 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eca31379e0 msgr2=0x55eca313c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:00.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.031+0000 7f7c0121b740 1 --2- 172.21.15.33:0/332738551 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eca31379e0 0x55eca313c4e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7bec09b730 tx=0x7f7bec010ea0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.031+0000 7f7c0121b740 1 -- 172.21.15.33:0/332738551 shutdown_connections 2024-03-19T02:51:00.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.031+0000 7f7c0121b740 1 --2- 172.21.15.33:0/332738551 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eca313cb50 0x55eca313efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.031+0000 7f7c0121b740 1 --2- 172.21.15.33:0/332738551 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eca31379e0 0x55eca313c4e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.031+0000 7f7c0121b740 1 --2- 172.21.15.33:0/332738551 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eca3136ff0 0x55eca3137410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.031+0000 7f7c0121b740 1 -- 172.21.15.33:0/332738551 >> 172.21.15.33:0/332738551 conn(0x55eca3132560 msgr2=0x55eca3132960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:00.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.031+0000 7f7c0121b740 1 -- 172.21.15.33:0/332738551 shutdown_connections 2024-03-19T02:51:00.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.031+0000 7f7c0121b740 1 -- 172.21.15.33:0/332738551 wait complete. 2024-03-19T02:51:00.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.031+0000 7f7c0121b740 1 -- start start 2024-03-19T02:51:00.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.032+0000 7f7c0121b740 1 Processor -- start 2024-03-19T02:51:00.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.032+0000 7f7c0121b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eca3136ff0 0x55eca3141540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:00.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.032+0000 7f7c0121b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eca31379e0 0x55eca3141a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:00.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.032+0000 7f7c0121b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eca313cb50 0x55eca3148ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:00.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.032+0000 7f7c0121b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55eca313faa0 con 0x55eca31379e0 2024-03-19T02:51:00.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.032+0000 7f7c0121b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55eca3057fc0 con 0x55eca3136ff0 2024-03-19T02:51:00.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.032+0000 7f7c0121b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55eca313fc20 con 0x55eca313cb50 2024-03-19T02:51:00.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.032+0000 7f7c0121b740 10 client.0 _ll_register_callbacks cb 0x55eca3093a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:00.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.032+0000 7f7bf77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eca31379e0 0x55eca3141a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:00.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.032+0000 7f7bf77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eca31379e0 0x55eca3141a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41452/0 (socket says 172.21.15.33:41452) 2024-03-19T02:51:00.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.032+0000 7f7bf77fe640 1 -- 172.21.15.33:0/2699466168 learned_addr learned my addr 172.21.15.33:0/2699466168 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:00.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.032+0000 7f7bf7fff640 1 --2- 172.21.15.33:0/2699466168 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eca3136ff0 0x55eca3141540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:00.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.032+0000 7f7bf77fe640 1 -- 172.21.15.33:0/2699466168 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eca313cb50 msgr2=0x55eca3148ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:00.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.032+0000 7f7bf77fe640 1 --2- 172.21.15.33:0/2699466168 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eca313cb50 0x55eca3148ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.032+0000 7f7bf77fe640 1 -- 172.21.15.33:0/2699466168 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eca3136ff0 msgr2=0x55eca3141540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:00.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.032+0000 7f7bf77fe640 1 --2- 172.21.15.33:0/2699466168 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eca3136ff0 0x55eca3141540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.032+0000 7f7bf77fe640 1 -- 172.21.15.33:0/2699466168 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eca2db02d0 con 0x55eca31379e0 2024-03-19T02:51:00.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.032+0000 7f7bf77fe640 1 --2- 172.21.15.33:0/2699466168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eca31379e0 0x55eca3141a80 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7bec00c6a0 tx=0x7f7bec0973d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f37109f136e8968 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:00.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.032+0000 7f7bd77fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:51:00.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.033+0000 7f7bd77fe640 1 -- 172.21.15.33:0/2699466168 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bec0b9060 con 0x55eca31379e0 2024-03-19T02:51:00.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.033+0000 7f7c0121b740 1 -- 172.21.15.33:0/2699466168 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bec099aa0 con 0x55eca31379e0 2024-03-19T02:51:00.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.033+0000 7f7bd77fe640 1 -- 172.21.15.33:0/2699466168 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7f7bec010630 con 0x55eca31379e0 2024-03-19T02:51:00.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.033+0000 7f7bd77fe640 1 -- 172.21.15.33:0/2699466168 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f7bec0b3070 con 0x55eca31379e0 2024-03-19T02:51:00.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.033+0000 7f7c0121b740 10 client.14730 fetch_fsmap learned FSMap version 17 2024-03-19T02:51:00.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.033+0000 7f7c0121b740 1 -- 172.21.15.33:0/2699466168 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55eca2e7a6b0 con 0x55eca31379e0 2024-03-19T02:51:00.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.033+0000 7f7bd77fe640 1 -- 172.21.15.33:0/2699466168 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7bec099aa0 con 0x55eca31379e0 2024-03-19T02:51:00.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.033+0000 7f7bd77fe640 1 -- 172.21.15.33:0/2699466168 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f7bec018ac0 con 0x55eca31379e0 2024-03-19T02:51:00.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.033+0000 7f7c0121b740 10 client.14730 fetch_fsmap finished waiting for FSMap version 17 2024-03-19T02:51:00.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.033+0000 7f7c0121b740 10 client.14730 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-19T02:51:00.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.033+0000 7f7c0121b740 20 client.14730 populate_metadata read hostname 'smithi033' 2024-03-19T02:51:00.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.033+0000 7f7c0121b740 1 --2- 172.21.15.33:0/2699466168 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55eca314de60 0x55eca314f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:00.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.033+0000 7f7c0121b740 4 client.14730 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:51:00.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.033+0000 7f7c0121b740 1 -- 172.21.15.33:0/2699466168 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eca30a9320 con 0x55eca314de60 2024-03-19T02:51:00.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.033+0000 7f7bf7fff640 1 --2- 172.21.15.33:0/2699466168 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55eca314de60 0x55eca314f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:00.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.034+0000 7f7bf7fff640 1 --2- 172.21.15.33:0/2699466168 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55eca314de60 0x55eca314f2d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d3e566b5a406a69 server_cookie=2e5ec486e5f696e3 in_seq=0 out_seq=0 2024-03-19T02:51:00.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.034+0000 7f7bd77fe640 10 client.14730 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:51:00.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.035+0000 7f7bd77fe640 1 -- 172.21.15.33:0/2699466168 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55eca30a9320 con 0x55eca314de60 2024-03-19T02:51:00.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.035+0000 7f7bd77fe640 10 client.14730 handle_command_reply: tid=0 2024-03-19T02:51:00.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.035+0000 7f7c0121b740 1 client.14730 shutdown 2024-03-19T02:51:00.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.035+0000 7f7c0121b740 1 -- 172.21.15.33:0/2699466168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eca31379e0 msgr2=0x55eca3141a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:00.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.035+0000 7f7c0121b740 1 --2- 172.21.15.33:0/2699466168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eca31379e0 0x55eca3141a80 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7bec00c6a0 tx=0x7f7bec0973d0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.035+0000 7f7c0121b740 1 -- 172.21.15.33:0/2699466168 shutdown_connections 2024-03-19T02:51:00.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.035+0000 7f7c0121b740 1 --2- 172.21.15.33:0/2699466168 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55eca314de60 0x55eca314f2d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.035+0000 7f7c0121b740 1 --2- 172.21.15.33:0/2699466168 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eca313cb50 0x55eca3148ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.035+0000 7f7bd77fe640 0 client.14730 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:51:00.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.035+0000 7f7c0121b740 1 --2- 172.21.15.33:0/2699466168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eca31379e0 0x55eca3141a80 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.035+0000 7f7c0121b740 1 --2- 172.21.15.33:0/2699466168 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eca3136ff0 0x55eca3141540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.035+0000 7f7c0121b740 1 -- 172.21.15.33:0/2699466168 >> 172.21.15.33:0/2699466168 conn(0x55eca3132560 msgr2=0x55eca31317e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:00.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.035+0000 7f7c0121b740 1 -- 172.21.15.33:0/2699466168 shutdown_connections 2024-03-19T02:51:00.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.035+0000 7f7c0121b740 1 -- 172.21.15.33:0/2699466168 wait complete. 2024-03-19T02:51:00.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.035+0000 7f7c0121b740 20 client.14730 trim_cache size 0 max 16384 2024-03-19T02:51:00.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.053+0000 7f7c0121b740 1 Processor -- start 2024-03-19T02:51:00.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.053+0000 7f7c0121b740 1 -- start start 2024-03-19T02:51:00.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.053+0000 7f7c0121b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eca3136ff0 0x55eca314cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:00.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.053+0000 7f7c0121b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eca31379e0 0x55eca3148a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:00.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.053+0000 7f7c0121b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eca313cb50 0x55eca313ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:00.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.053+0000 7f7c0121b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55eca314d610 con 0x55eca31379e0 2024-03-19T02:51:00.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.053+0000 7f7c0121b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55eca314dce0 con 0x55eca313cb50 2024-03-19T02:51:00.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.053+0000 7f7c0121b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55eca314e3b0 con 0x55eca3136ff0 2024-03-19T02:51:00.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.054+0000 7f7bf77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eca31379e0 0x55eca3148a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:00.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.054+0000 7f7bf77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eca31379e0 0x55eca3148a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41462/0 (socket says 172.21.15.33:41462) 2024-03-19T02:51:00.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.054+0000 7f7bf77fe640 1 -- 172.21.15.33:0/3911568725 learned_addr learned my addr 172.21.15.33:0/3911568725 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:00.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.054+0000 7f7bfcaf0640 1 --2- 172.21.15.33:0/3911568725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eca313cb50 0x55eca313ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:00.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.054+0000 7f7bf77fe640 1 -- 172.21.15.33:0/3911568725 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eca3136ff0 msgr2=0x55eca314cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:00.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.054+0000 7f7bf77fe640 1 --2- 172.21.15.33:0/3911568725 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eca3136ff0 0x55eca314cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.054+0000 7f7bf77fe640 1 -- 172.21.15.33:0/3911568725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eca313cb50 msgr2=0x55eca313ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:00.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.054+0000 7f7bf77fe640 1 --2- 172.21.15.33:0/3911568725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eca313cb50 0x55eca313ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.054+0000 7f7bf77fe640 1 -- 172.21.15.33:0/3911568725 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eca312dfc0 con 0x55eca31379e0 2024-03-19T02:51:00.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.054+0000 7f7bf77fe640 1 --2- 172.21.15.33:0/3911568725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eca31379e0 0x55eca3148a80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7bec09b4a0 tx=0x7f7bec097440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21cdfa99acc7a2d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:00.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.054+0000 7f7bddffb640 1 -- 172.21.15.33:0/3911568725 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bec0b9030 con 0x55eca31379e0 2024-03-19T02:51:00.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.054+0000 7f7bddffb640 1 -- 172.21.15.33:0/3911568725 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7bec099aa0 con 0x55eca31379e0 2024-03-19T02:51:00.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.054+0000 7f7bddffb640 1 -- 172.21.15.33:0/3911568725 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bec0afad0 con 0x55eca31379e0 2024-03-19T02:51:00.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.055+0000 7f7c0121b740 1 -- 172.21.15.33:0/3911568725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eca31379e0 msgr2=0x55eca3148a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:00.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.055+0000 7f7c0121b740 1 --2- 172.21.15.33:0/3911568725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eca31379e0 0x55eca3148a80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7bec09b4a0 tx=0x7f7bec097440 comp rx=0 tx=0).stop 2024-03-19T02:51:00.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.055+0000 7f7c0121b740 1 -- 172.21.15.33:0/3911568725 shutdown_connections 2024-03-19T02:51:00.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.055+0000 7f7c0121b740 1 --2- 172.21.15.33:0/3911568725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eca313cb50 0x55eca313ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.055+0000 7f7c0121b740 1 --2- 172.21.15.33:0/3911568725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eca31379e0 0x55eca3148a80 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.055+0000 7f7c0121b740 1 --2- 172.21.15.33:0/3911568725 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eca3136ff0 0x55eca314cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.055+0000 7f7c0121b740 1 -- 172.21.15.33:0/3911568725 >> 172.21.15.33:0/3911568725 conn(0x55eca3132560 msgr2=0x55eca313fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:00.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.055+0000 7f7c0121b740 1 -- 172.21.15.33:0/3911568725 shutdown_connections 2024-03-19T02:51:00.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.055+0000 7f7c0121b740 1 -- 172.21.15.33:0/3911568725 wait complete. 2024-03-19T02:51:00.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.055+0000 7f7c0121b740 1 -- start start 2024-03-19T02:51:00.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.055+0000 7f7c0121b740 1 Processor -- start 2024-03-19T02:51:00.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.055+0000 7f7c0121b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eca3136ff0 0x55eca313b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:00.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.055+0000 7f7c0121b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eca31379e0 0x55eca3148a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:00.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.055+0000 7f7c0121b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eca313cb50 0x55eca31b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:00.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.055+0000 7f7c0121b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55eca314f150 con 0x55eca3136ff0 2024-03-19T02:51:00.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.055+0000 7f7c0121b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55eca314ea80 con 0x55eca31379e0 2024-03-19T02:51:00.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.055+0000 7f7c0121b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55eca314f820 con 0x55eca313cb50 2024-03-19T02:51:00.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.055+0000 7f7c0121b740 10 client.0 _ll_register_callbacks cb 0x7f7bec005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:51:00.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.055+0000 7f7bf7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eca3136ff0 0x55eca313b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:00.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.055+0000 7f7bf7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eca3136ff0 0x55eca313b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41474/0 (socket says 172.21.15.33:41474) 2024-03-19T02:51:00.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.055+0000 7f7bf7fff640 1 -- 172.21.15.33:0/3547546716 learned_addr learned my addr 172.21.15.33:0/3547546716 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:00.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.055+0000 7f7bf77fe640 1 --2- 172.21.15.33:0/3547546716 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eca31379e0 0x55eca3148a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:00.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.056+0000 7f7bfcaf0640 1 --2- 172.21.15.33:0/3547546716 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eca313cb50 0x55eca31b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:00.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.056+0000 7f7bf7fff640 1 -- 172.21.15.33:0/3547546716 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eca313cb50 msgr2=0x55eca31b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:00.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.056+0000 7f7bf7fff640 1 --2- 172.21.15.33:0/3547546716 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eca313cb50 0x55eca31b1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.056+0000 7f7bf7fff640 1 -- 172.21.15.33:0/3547546716 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eca31379e0 msgr2=0x55eca3148a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:51:00.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.056+0000 7f7bf7fff640 1 --2- 172.21.15.33:0/3547546716 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eca31379e0 0x55eca3148a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.056+0000 7f7bf7fff640 1 -- 172.21.15.33:0/3547546716 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eca31b23f0 con 0x55eca3136ff0 2024-03-19T02:51:00.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.056+0000 7f7bf7fff640 1 --2- 172.21.15.33:0/3547546716 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eca3136ff0 0x55eca313b9f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7be8005fd0 tx=0x7f7be8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b561b4eb6de1200 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:00.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.056+0000 7f7bd77fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:51:00.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.056+0000 7f7bd77fe640 1 -- 172.21.15.33:0/3547546716 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be801de60 con 0x55eca3136ff0 2024-03-19T02:51:00.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.056+0000 7f7bd77fe640 1 -- 172.21.15.33:0/3547546716 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7f7be801d630 con 0x55eca3136ff0 2024-03-19T02:51:00.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.056+0000 7f7c0121b740 1 -- 172.21.15.33:0/3547546716 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bec099aa0 con 0x55eca3136ff0 2024-03-19T02:51:00.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.056+0000 7f7bd77fe640 1 -- 172.21.15.33:0/3547546716 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f7be802d050 con 0x55eca3136ff0 2024-03-19T02:51:00.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.056+0000 7f7c0121b740 10 client.14736 fetch_fsmap learned FSMap version 17 2024-03-19T02:51:00.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.056+0000 7f7c0121b740 1 -- 172.21.15.33:0/3547546716 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55eca31b37c0 con 0x55eca3136ff0 2024-03-19T02:51:00.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.056+0000 7f7bd77fe640 1 -- 172.21.15.33:0/3547546716 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7bec099aa0 con 0x55eca3136ff0 2024-03-19T02:51:00.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.056+0000 7f7bd77fe640 1 -- 172.21.15.33:0/3547546716 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f7be801c390 con 0x55eca3136ff0 2024-03-19T02:51:00.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.056+0000 7f7c0121b740 10 client.14736 fetch_fsmap finished waiting for FSMap version 17 2024-03-19T02:51:00.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.056+0000 7f7c0121b740 10 client.14736 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-19T02:51:00.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.056+0000 7f7c0121b740 20 client.14736 populate_metadata read hostname 'smithi033' 2024-03-19T02:51:00.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.057+0000 7f7c0121b740 1 --2- 172.21.15.33:0/3547546716 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55eca31b83c0 0x55eca31b70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:00.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.057+0000 7f7c0121b740 4 client.14736 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T02:51:00.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.057+0000 7f7c0121b740 1 -- 172.21.15.33:0/3547546716 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "status"}) -- 0x55eca30a3670 con 0x55eca31b83c0 2024-03-19T02:51:00.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.057+0000 7f7bf77fe640 1 --2- 172.21.15.33:0/3547546716 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55eca31b83c0 0x55eca31b70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:00.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.057+0000 7f7bd77fe640 10 client.14736 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:51:00.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.057+0000 7f7bf77fe640 1 --2- 172.21.15.33:0/3547546716 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55eca31b83c0 0x55eca31b70d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3255754f47bc4e69 server_cookie=daa194c261d7f879 in_seq=0 out_seq=0 2024-03-19T02:51:00.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.057+0000 7f7bd77fe640 1 -- 172.21.15.33:0/3547546716 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55eca30a3670 con 0x55eca31b83c0 2024-03-19T02:51:00.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.057+0000 7f7bd77fe640 10 client.14736 handle_command_reply: tid=0 2024-03-19T02:51:00.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.057+0000 7f7c0121b740 1 client.14736 shutdown 2024-03-19T02:51:00.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.057+0000 7f7c0121b740 1 -- 172.21.15.33:0/3547546716 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eca3136ff0 msgr2=0x55eca313b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:00.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.057+0000 7f7c0121b740 1 --2- 172.21.15.33:0/3547546716 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eca3136ff0 0x55eca313b9f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7be8005fd0 tx=0x7f7be8011810 comp rx=0 tx=0).stop 2024-03-19T02:51:00.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.057+0000 7f7c0121b740 1 -- 172.21.15.33:0/3547546716 shutdown_connections 2024-03-19T02:51:00.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.057+0000 7f7c0121b740 1 --2- 172.21.15.33:0/3547546716 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55eca31b83c0 0x55eca31b70d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.057+0000 7f7c0121b740 1 --2- 172.21.15.33:0/3547546716 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eca313cb50 0x55eca31b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.057+0000 7f7c0121b740 1 --2- 172.21.15.33:0/3547546716 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eca31379e0 0x55eca3148a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.057+0000 7f7bd77fe640 0 client.14736 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:51:00.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.057+0000 7f7c0121b740 1 --2- 172.21.15.33:0/3547546716 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eca3136ff0 0x55eca313b9f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.057+0000 7f7c0121b740 1 -- 172.21.15.33:0/3547546716 >> 172.21.15.33:0/3547546716 conn(0x55eca3132560 msgr2=0x55eca3138270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:00.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.057+0000 7f7c0121b740 1 -- 172.21.15.33:0/3547546716 shutdown_connections 2024-03-19T02:51:00.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.057+0000 7f7c0121b740 1 -- 172.21.15.33:0/3547546716 wait complete. 2024-03-19T02:51:00.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.058+0000 7f7c0121b740 20 client.14736 trim_cache size 0 max 16384 2024-03-19T02:51:00.066 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:51:00.066 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_fsid": "2b7a8938-e59a-11ee-95c9-87774f69a715", 2024-03-19T02:51:00.066 INFO:teuthology.orchestra.run.smithi033.stdout: "whoami": 0, 2024-03-19T02:51:00.066 INFO:teuthology.orchestra.run.smithi033.stdout: "id": 24509, 2024-03-19T02:51:00.066 INFO:teuthology.orchestra.run.smithi033.stdout: "want_state": "up:active", 2024-03-19T02:51:00.066 INFO:teuthology.orchestra.run.smithi033.stdout: "state": "up:active", 2024-03-19T02:51:00.066 INFO:teuthology.orchestra.run.smithi033.stdout: "fs_name": "cephfs", 2024-03-19T02:51:00.066 INFO:teuthology.orchestra.run.smithi033.stdout: "rank_uptime": 6.8584789349999999, 2024-03-19T02:51:00.066 INFO:teuthology.orchestra.run.smithi033.stdout: "mdsmap_epoch": 17, 2024-03-19T02:51:00.066 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch": 93, 2024-03-19T02:51:00.066 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch_barrier": 92, 2024-03-19T02:51:00.067 INFO:teuthology.orchestra.run.smithi033.stdout: "uptime": 12.439223449 2024-03-19T02:51:00.067 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:51:00.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.058+0000 7f7bfe553640 1 -- 172.21.15.33:0/2050599817 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7be00a2800 msgr2=0x7f7be00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:00.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.058+0000 7f7bfe553640 1 --2- 172.21.15.33:0/2050599817 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7be00a2800 0x7f7be00a4cc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7bf0006010 tx=0x7f7bf0009930 comp rx=0 tx=0).stop 2024-03-19T02:51:00.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.058+0000 7f7bfe553640 1 -- 172.21.15.33:0/2050599817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bf8101910 msgr2=0x7f7bf8198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:00.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.058+0000 7f7bfe553640 1 --2- 172.21.15.33:0/2050599817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bf8101910 0x7f7bf8198020 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7bec00e9f0 tx=0x7f7bec00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.058+0000 7f7bfe553640 1 -- 172.21.15.33:0/2050599817 shutdown_connections 2024-03-19T02:51:00.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.058+0000 7f7bfe553640 1 --2- 172.21.15.33:0/2050599817 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7be00a2800 0x7f7be00a4cc0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.058+0000 7f7bfe553640 1 --2- 172.21.15.33:0/2050599817 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7bf81022d0 0x7f7bf819f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.058+0000 7f7bfe553640 1 --2- 172.21.15.33:0/2050599817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bf8101910 0x7f7bf8198020 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.058+0000 7f7bfe553640 1 --2- 172.21.15.33:0/2050599817 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7bf8100720 0x7f7bf8197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:00.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.058+0000 7f7bfe553640 1 -- 172.21.15.33:0/2050599817 >> 172.21.15.33:0/2050599817 conn(0x7f7bf80fbcb0 msgr2=0x7f7bf80fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:00.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.058+0000 7f7bfe553640 1 -- 172.21.15.33:0/2050599817 shutdown_connections 2024-03-19T02:51:00.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:00.058+0000 7f7bfe553640 1 -- 172.21.15.33:0/2050599817 wait complete. 2024-03-19T02:51:00.070 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-19T02:51:00.070 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-19T02:51:00.070 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-19T02:51:00.317 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:51:00.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:00 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/956107841' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:51:00.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:00 smithi155 ceph-mon[35804]: pgmap v303: 129 pgs: 5 creating+peering, 19 unknown, 105 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 5 op/s 2024-03-19T02:51:00.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:00 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/997474931' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:51:00.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:00 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/956107841' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:51:00.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:00 smithi033 ceph-mon[31973]: pgmap v303: 129 pgs: 5 creating+peering, 19 unknown, 105 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 5 op/s 2024-03-19T02:51:00.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:00 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/997474931' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:51:00.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:00 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/956107841' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:51:00.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:00 smithi096 ceph-mon[35946]: pgmap v303: 129 pgs: 5 creating+peering, 19 unknown, 105 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 5 op/s 2024-03-19T02:51:00.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:00 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/997474931' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:51:01.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:01 smithi033 ceph-mon[31973]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-19T02:51:01.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:01 smithi033 ceph-mon[31973]: Cluster is now healthy 2024-03-19T02:51:01.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:01 smithi155 ceph-mon[35804]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-19T02:51:01.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:01 smithi155 ceph-mon[35804]: Cluster is now healthy 2024-03-19T02:51:01.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:01 smithi096 ceph-mon[35946]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-19T02:51:01.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:01 smithi096 ceph-mon[35946]: Cluster is now healthy 2024-03-19T02:51:01.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.800+0000 7f1e5fd9c700 1 -- 172.21.15.33:0/3640602890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1e581014d0 msgr2=0x7f1e581018b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:01.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.800+0000 7f1e5fd9c700 1 --2- 172.21.15.33:0/3640602890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1e581014d0 0x7f1e581018b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1e4800b0d0 tx=0x7f1e4800b3e0 comp rx=0 tx=0).stop 2024-03-19T02:51:01.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.800+0000 7f1e5fd9c700 1 -- 172.21.15.33:0/3640602890 shutdown_connections 2024-03-19T02:51:01.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.800+0000 7f1e5fd9c700 1 --2- 172.21.15.33:0/3640602890 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e58106520 0x7f1e5810e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:01.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.800+0000 7f1e5fd9c700 1 --2- 172.21.15.33:0/3640602890 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1e581014d0 0x7f1e581018b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:01.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.800+0000 7f1e5fd9c700 1 --2- 172.21.15.33:0/3640602890 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e58101e80 0x7f1e58105e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:01.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.800+0000 7f1e5fd9c700 1 -- 172.21.15.33:0/3640602890 >> 172.21.15.33:0/3640602890 conn(0x7f1e58077260 msgr2=0x7f1e58077670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:01.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.800+0000 7f1e5fd9c700 1 -- 172.21.15.33:0/3640602890 shutdown_connections 2024-03-19T02:51:01.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.800+0000 7f1e5fd9c700 1 -- 172.21.15.33:0/3640602890 wait complete. 2024-03-19T02:51:01.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.801+0000 7f1e5fd9c700 1 Processor -- start 2024-03-19T02:51:01.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.801+0000 7f1e5fd9c700 1 -- start start 2024-03-19T02:51:01.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.801+0000 7f1e5fd9c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1e581014d0 0x7f1e58102ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:01.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.801+0000 7f1e5fd9c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e58101e80 0x7f1e58103420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:01.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.801+0000 7f1e5fd9c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e58106520 0x7f1e58103960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:01.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.801+0000 7f1e5fd9c700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1e58111430 con 0x7f1e581014d0 2024-03-19T02:51:01.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.801+0000 7f1e5fd9c700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1e581112b0 con 0x7f1e58101e80 2024-03-19T02:51:01.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.801+0000 7f1e5fd9c700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1e581115b0 con 0x7f1e58106520 2024-03-19T02:51:01.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.801+0000 7f1e5db38700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1e581014d0 0x7f1e58102ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:01.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.801+0000 7f1e5db38700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1e581014d0 0x7f1e58102ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41502/0 (socket says 172.21.15.33:41502) 2024-03-19T02:51:01.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.801+0000 7f1e5db38700 1 -- 172.21.15.33:0/3517098326 learned_addr learned my addr 172.21.15.33:0/3517098326 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:01.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.801+0000 7f1e5d337700 1 --2- 172.21.15.33:0/3517098326 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e58101e80 0x7f1e58103420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:01.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.801+0000 7f1e5e339700 1 --2- 172.21.15.33:0/3517098326 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e58106520 0x7f1e58103960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:01.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.801+0000 7f1e5db38700 1 -- 172.21.15.33:0/3517098326 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e58106520 msgr2=0x7f1e58103960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:01.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.801+0000 7f1e5db38700 1 --2- 172.21.15.33:0/3517098326 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e58106520 0x7f1e58103960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:01.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.801+0000 7f1e5db38700 1 -- 172.21.15.33:0/3517098326 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e58101e80 msgr2=0x7f1e58103420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:01.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.801+0000 7f1e5db38700 1 --2- 172.21.15.33:0/3517098326 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e58101e80 0x7f1e58103420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:01.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.801+0000 7f1e5db38700 1 -- 172.21.15.33:0/3517098326 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e581a6f90 con 0x7f1e581014d0 2024-03-19T02:51:01.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.801+0000 7f1e5e339700 1 --2- 172.21.15.33:0/3517098326 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e58106520 0x7f1e58103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:51:01.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.802+0000 7f1e5db38700 1 --2- 172.21.15.33:0/3517098326 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1e581014d0 0x7f1e58102ee0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1e4800d010 tx=0x7f1e48007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:01.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.802+0000 7f1e56ffd700 1 -- 172.21.15.33:0/3517098326 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e4801f070 con 0x7f1e581014d0 2024-03-19T02:51:01.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.802+0000 7f1e56ffd700 1 -- 172.21.15.33:0/3517098326 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1e48003c00 con 0x7f1e581014d0 2024-03-19T02:51:01.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.802+0000 7f1e5fd9c700 1 -- 172.21.15.33:0/3517098326 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e581a7220 con 0x7f1e581014d0 2024-03-19T02:51:01.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.802+0000 7f1e56ffd700 1 -- 172.21.15.33:0/3517098326 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e4801b6a0 con 0x7f1e581014d0 2024-03-19T02:51:01.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.802+0000 7f1e5fd9c700 1 -- 172.21.15.33:0/3517098326 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e581a7670 con 0x7f1e581014d0 2024-03-19T02:51:01.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.803+0000 7f1e5fd9c700 1 -- 172.21.15.33:0/3517098326 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e58068fd0 con 0x7f1e581014d0 2024-03-19T02:51:01.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.804+0000 7f1e56ffd700 1 -- 172.21.15.33:0/3517098326 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f1e48022070 con 0x7f1e581014d0 2024-03-19T02:51:01.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.805+0000 7f1e56ffd700 1 --2- 172.21.15.33:0/3517098326 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f1e400a2890 0x7f1e400a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:01.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.805+0000 7f1e5d337700 1 --2- 172.21.15.33:0/3517098326 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f1e400a2890 0x7f1e400a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:01.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.805+0000 7f1e56ffd700 1 -- 172.21.15.33:0/3517098326 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7f1e4800f020 con 0x7f1e581014d0 2024-03-19T02:51:01.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.805+0000 7f1e5d337700 1 --2- 172.21.15.33:0/3517098326 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f1e400a2890 0x7f1e400a4d50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1e58101010 tx=0x7f1e4c009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:01.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:01.809+0000 7f1e56ffd700 1 -- 172.21.15.33:0/3517098326 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1e48029020 con 0x7f1e581014d0 2024-03-19T02:51:02.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:02.057+0000 7f1e5fd9c700 1 -- 172.21.15.33:0/3517098326 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f1e5802dd70 con 0x7f1e581014d0 2024-03-19T02:51:02.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:02.058+0000 7f1e56ffd700 1 -- 172.21.15.33:0/3517098326 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f1e48011390 con 0x7f1e581014d0 2024-03-19T02:51:02.059 INFO:teuthology.orchestra.run.smithi033.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-19T02:51:02.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:02.060+0000 7f1e5fd9c700 1 -- 172.21.15.33:0/3517098326 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f1e400a2890 msgr2=0x7f1e400a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:02.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:02.060+0000 7f1e5fd9c700 1 --2- 172.21.15.33:0/3517098326 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f1e400a2890 0x7f1e400a4d50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1e58101010 tx=0x7f1e4c009190 comp rx=0 tx=0).stop 2024-03-19T02:51:02.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:02.060+0000 7f1e5fd9c700 1 -- 172.21.15.33:0/3517098326 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1e581014d0 msgr2=0x7f1e58102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:02.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:02.060+0000 7f1e5fd9c700 1 --2- 172.21.15.33:0/3517098326 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1e581014d0 0x7f1e58102ee0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1e4800d010 tx=0x7f1e48007a20 comp rx=0 tx=0).stop 2024-03-19T02:51:02.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:02.060+0000 7f1e5fd9c700 1 -- 172.21.15.33:0/3517098326 shutdown_connections 2024-03-19T02:51:02.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:02.060+0000 7f1e5fd9c700 1 --2- 172.21.15.33:0/3517098326 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f1e400a2890 0x7f1e400a4d50 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:02.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:02.060+0000 7f1e5fd9c700 1 --2- 172.21.15.33:0/3517098326 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e58106520 0x7f1e58103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:02.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:02.060+0000 7f1e5fd9c700 1 --2- 172.21.15.33:0/3517098326 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1e581014d0 0x7f1e58102ee0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:02.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:02.060+0000 7f1e5fd9c700 1 --2- 172.21.15.33:0/3517098326 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1e58101e80 0x7f1e58103420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:02.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:02.060+0000 7f1e5fd9c700 1 -- 172.21.15.33:0/3517098326 >> 172.21.15.33:0/3517098326 conn(0x7f1e58077260 msgr2=0x7f1e5806b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:02.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:02.060+0000 7f1e5fd9c700 1 -- 172.21.15.33:0/3517098326 shutdown_connections 2024-03-19T02:51:02.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:02.060+0000 7f1e5fd9c700 1 -- 172.21.15.33:0/3517098326 wait complete. 2024-03-19T02:51:02.319 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:02 smithi033 ceph-mon[31973]: pgmap v304: 129 pgs: 3 creating+peering, 126 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 186 B/s wr, 3 op/s 2024-03-19T02:51:02.319 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:02 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3517098326' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-19T02:51:02.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:02 smithi155 ceph-mon[35804]: pgmap v304: 129 pgs: 3 creating+peering, 126 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 186 B/s wr, 3 op/s 2024-03-19T02:51:02.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:02 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3517098326' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-19T02:51:02.641 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-19T02:51:02.641 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-19T02:51:02.641 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi033.front.sepia.ceph.com 2024-03-19T02:51:02.641 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-19T02:51:02.642 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-19T02:51:02.642 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-19T02:51:02.642 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-19T02:51:02.642 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-19T02:51:02.642 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-19T02:51:02.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:51:02.642 DEBUG:teuthology.orchestra.run.smithi033:> ip addr 2024-03-19T02:51:02.658 INFO:teuthology.orchestra.run.smithi033.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-19T02:51:02.658 INFO:teuthology.orchestra.run.smithi033.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-19T02:51:02.658 INFO:teuthology.orchestra.run.smithi033.stdout: inet 127.0.0.1/8 scope host lo 2024-03-19T02:51:02.659 INFO:teuthology.orchestra.run.smithi033.stdout: valid_lft forever preferred_lft forever 2024-03-19T02:51:02.659 INFO:teuthology.orchestra.run.smithi033.stdout: inet6 ::1/128 scope host 2024-03-19T02:51:02.659 INFO:teuthology.orchestra.run.smithi033.stdout: valid_lft forever preferred_lft forever 2024-03-19T02:51:02.659 INFO:teuthology.orchestra.run.smithi033.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-19T02:51:02.659 INFO:teuthology.orchestra.run.smithi033.stdout: link/ether 0c:c4:7a:6b:fd:14 brd ff:ff:ff:ff:ff:ff 2024-03-19T02:51:02.659 INFO:teuthology.orchestra.run.smithi033.stdout: altname enp7s0f0 2024-03-19T02:51:02.659 INFO:teuthology.orchestra.run.smithi033.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-19T02:51:02.659 INFO:teuthology.orchestra.run.smithi033.stdout: link/ether 0c:c4:7a:6b:fd:15 brd ff:ff:ff:ff:ff:ff 2024-03-19T02:51:02.659 INFO:teuthology.orchestra.run.smithi033.stdout: altname enp7s0f1 2024-03-19T02:51:02.659 INFO:teuthology.orchestra.run.smithi033.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-19T02:51:02.659 INFO:teuthology.orchestra.run.smithi033.stdout: link/ether ac:1f:6b:a5:d2:94 brd ff:ff:ff:ff:ff:ff 2024-03-19T02:51:02.659 INFO:teuthology.orchestra.run.smithi033.stdout: altname enp1s0f0 2024-03-19T02:51:02.659 INFO:teuthology.orchestra.run.smithi033.stdout: inet 172.21.15.33/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-19T02:51:02.659 INFO:teuthology.orchestra.run.smithi033.stdout: valid_lft 41565sec preferred_lft 41565sec 2024-03-19T02:51:02.659 INFO:teuthology.orchestra.run.smithi033.stdout: inet6 fe80::ae1f:6bff:fea5:d294/64 scope link noprefixroute 2024-03-19T02:51:02.660 INFO:teuthology.orchestra.run.smithi033.stdout: valid_lft forever preferred_lft forever 2024-03-19T02:51:02.660 INFO:teuthology.orchestra.run.smithi033.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-19T02:51:02.660 INFO:teuthology.orchestra.run.smithi033.stdout: link/ether ac:1f:6b:a5:d2:95 brd ff:ff:ff:ff:ff:ff 2024-03-19T02:51:02.660 INFO:teuthology.orchestra.run.smithi033.stdout: altname enp1s0f1 2024-03-19T02:51:02.660 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-19T02:51:02.660 DEBUG:teuthology.orchestra.run.smithi033:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T02:51:02.660 DEBUG:teuthology.orchestra.run.smithi033:> set -e 2024-03-19T02:51:02.661 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip link add name ceph-brx type bridge 2024-03-19T02:51:02.661 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip addr flush dev ceph-brx 2024-03-19T02:51:02.661 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip link set ceph-brx up 2024-03-19T02:51:02.661 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-19T02:51:02.661 DEBUG:teuthology.orchestra.run.smithi033:> ') 2024-03-19T02:51:02.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:02 smithi096 ceph-mon[35946]: pgmap v304: 129 pgs: 3 creating+peering, 126 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 186 B/s wr, 3 op/s 2024-03-19T02:51:02.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:02 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3517098326' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-19T02:51:02.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T02:51:02.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:51:02.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:51:02.870 DEBUG:teuthology.orchestra.run.smithi033:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-19T02:51:02.897 INFO:teuthology.orchestra.run.smithi033.stdout:1 2024-03-19T02:51:02.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:51:02.899 DEBUG:teuthology.orchestra.run.smithi033:> route 2024-03-19T02:51:02.972 INFO:teuthology.orchestra.run.smithi033.stdout:Kernel IP routing table 2024-03-19T02:51:02.973 INFO:teuthology.orchestra.run.smithi033.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-19T02:51:02.973 INFO:teuthology.orchestra.run.smithi033.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-19T02:51:02.973 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-19T02:51:02.973 INFO:teuthology.orchestra.run.smithi033.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-19T02:51:02.974 DEBUG:teuthology.orchestra.run.smithi033:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T02:51:02.974 DEBUG:teuthology.orchestra.run.smithi033:> set -e 2024-03-19T02:51:02.974 DEBUG:teuthology.orchestra.run.smithi033:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-19T02:51:02.974 DEBUG:teuthology.orchestra.run.smithi033:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-19T02:51:02.974 DEBUG:teuthology.orchestra.run.smithi033:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-19T02:51:02.974 DEBUG:teuthology.orchestra.run.smithi033:> ') 2024-03-19T02:51:03.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T02:51:03.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:51:03.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:51:03.290 DEBUG:teuthology.orchestra.run.smithi033:> ip netns list 2024-03-19T02:51:03.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:51:03.305 DEBUG:teuthology.orchestra.run.smithi033:> ip netns list-id 2024-03-19T02:51:03.360 DEBUG:teuthology.orchestra.run.smithi033:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T02:51:03.361 DEBUG:teuthology.orchestra.run.smithi033:> set -e 2024-03-19T02:51:03.361 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-19T02:51:03.361 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-19T02:51:03.361 DEBUG:teuthology.orchestra.run.smithi033:> ') 2024-03-19T02:51:03.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T02:51:03.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:51:03.482 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-19T02:51:03.482 DEBUG:teuthology.orchestra.run.smithi033:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T02:51:03.482 DEBUG:teuthology.orchestra.run.smithi033:> set -e 2024-03-19T02:51:03.482 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-19T02:51:03.482 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-19T02:51:03.482 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-19T02:51:03.482 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-19T02:51:03.482 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-19T02:51:03.482 DEBUG:teuthology.orchestra.run.smithi033:> ') 2024-03-19T02:51:03.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T02:51:03.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:51:03.621 DEBUG:teuthology.orchestra.run.smithi033:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T02:51:03.621 DEBUG:teuthology.orchestra.run.smithi033:> set -e 2024-03-19T02:51:03.621 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip link set brx.0 up 2024-03-19T02:51:03.622 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip link set dev brx.0 master ceph-brx 2024-03-19T02:51:03.622 DEBUG:teuthology.orchestra.run.smithi033:> ') 2024-03-19T02:51:03.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T02:51:03.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:51:03.704 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-19T02:51:03.704 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-19T02:51:03.704 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-19T02:51:03.718 INFO:teuthology.orchestra.run.smithi033.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-19T02:51:03.718 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-19T02:51:03.718 DEBUG:teuthology.orchestra.run.smithi033:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-19T02:51:03.773 DEBUG:teuthology.orchestra.run.smithi033:> sudo modprobe fuse 2024-03-19T02:51:03.840 DEBUG:teuthology.orchestra.run.smithi033:> cat /proc/self/mounts | awk '{print $2}' 2024-03-19T02:51:03.896 INFO:teuthology.orchestra.run.smithi033.stdout:/proc 2024-03-19T02:51:03.896 INFO:teuthology.orchestra.run.smithi033.stdout:/sys 2024-03-19T02:51:03.897 INFO:teuthology.orchestra.run.smithi033.stdout:/dev 2024-03-19T02:51:03.897 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/kernel/security 2024-03-19T02:51:03.897 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/shm 2024-03-19T02:51:03.897 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/pts 2024-03-19T02:51:03.897 INFO:teuthology.orchestra.run.smithi033.stdout:/run 2024-03-19T02:51:03.897 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/fs/cgroup 2024-03-19T02:51:03.897 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/fs/pstore 2024-03-19T02:51:03.897 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/fs/bpf 2024-03-19T02:51:03.897 INFO:teuthology.orchestra.run.smithi033.stdout:/ 2024-03-19T02:51:03.897 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/fs/selinux 2024-03-19T02:51:03.897 INFO:teuthology.orchestra.run.smithi033.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T02:51:03.897 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/mqueue 2024-03-19T02:51:03.897 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/hugepages 2024-03-19T02:51:03.898 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/kernel/debug 2024-03-19T02:51:03.898 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/kernel/tracing 2024-03-19T02:51:03.898 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/kernel/config 2024-03-19T02:51:03.898 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/fs/fuse/connections 2024-03-19T02:51:03.898 INFO:teuthology.orchestra.run.smithi033.stdout:/run/credentials/systemd-sysctl.service 2024-03-19T02:51:03.898 INFO:teuthology.orchestra.run.smithi033.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-19T02:51:03.898 INFO:teuthology.orchestra.run.smithi033.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-19T02:51:03.898 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/nfs/rpc_pipefs 2024-03-19T02:51:03.898 INFO:teuthology.orchestra.run.smithi033.stdout:/run/user/1000 2024-03-19T02:51:03.898 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/ceph 2024-03-19T02:51:03.898 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay 2024-03-19T02:51:03.898 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/d40815713b6d8abf13551e5b03e4726efec7d920162aacc3167eb822ccd4f341/merged 2024-03-19T02:51:03.898 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/f26473e65fe17f5cc9a064aa0a0705b7cf6cdfb5557e691edb195c421fe9311e/merged 2024-03-19T02:51:03.898 INFO:teuthology.orchestra.run.smithi033.stdout:/run/user/0 2024-03-19T02:51:03.899 INFO:teuthology.orchestra.run.smithi033.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T02:51:03.899 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/458cde7783a31ad935025ecce83257fe525e84dfc943bc5ddf3711c44dc9cb47/merged 2024-03-19T02:51:03.899 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/4781736ffa71eef74ba07d8e57fa30a2a7dc9c2ea614310d94ac0a0ba14c808c/merged 2024-03-19T02:51:03.899 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/a6b836646400b0531dc3e7adb38a6372d800021d4cfe7dcdc74b23fb4bff575d/merged 2024-03-19T02:51:03.899 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/a0b05b7ceab9ec0d57bb302cc867219b12bfe23a430eb417e4c02683d743e203/merged 2024-03-19T02:51:03.899 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/264ba1fe315f2d024742741205d4d32695fa53dc0074ea52b5dac59a93dcc6b7/merged 2024-03-19T02:51:03.899 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/a592735a9ef96a00521c4d49ff5be4085daca00927f58bcae50da2dd3c634979/merged 2024-03-19T02:51:03.899 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/547c2dfe92bf1ab04750857a84566c04e128f7d6f517b355478a8dfa63b23f88/merged 2024-03-19T02:51:03.899 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/4613b7ef0ee9acd7aadf0117330f3af918303ea30b265b69578eb1c58954e1b9/merged 2024-03-19T02:51:03.899 INFO:teuthology.orchestra.run.smithi033.stdout:/run/netns 2024-03-19T02:51:03.899 INFO:teuthology.orchestra.run.smithi033.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-19T02:51:03.899 INFO:teuthology.orchestra.run.smithi033.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-19T02:51:03.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:51:03.900 DEBUG:teuthology.orchestra.run.smithi033:> ls /sys/fs/fuse/connections 2024-03-19T02:51:03.951 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-19T02:51:03.951 DEBUG:teuthology.orchestra.run.smithi033:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-19T02:51:03.994 DEBUG:teuthology.orchestra.run.smithi033:> sudo modprobe fuse 2024-03-19T02:51:04.034 DEBUG:teuthology.orchestra.run.smithi033:> cat /proc/self/mounts | awk '{print $2}' 2024-03-19T02:51:04.090 INFO:teuthology.orchestra.run.smithi033.stdout:/proc 2024-03-19T02:51:04.090 INFO:teuthology.orchestra.run.smithi033.stdout:/sys 2024-03-19T02:51:04.090 INFO:teuthology.orchestra.run.smithi033.stdout:/dev 2024-03-19T02:51:04.090 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/kernel/security 2024-03-19T02:51:04.090 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/shm 2024-03-19T02:51:04.090 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/pts 2024-03-19T02:51:04.090 INFO:teuthology.orchestra.run.smithi033.stdout:/run 2024-03-19T02:51:04.090 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/fs/cgroup 2024-03-19T02:51:04.090 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/fs/pstore 2024-03-19T02:51:04.090 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/fs/bpf 2024-03-19T02:51:04.091 INFO:teuthology.orchestra.run.smithi033.stdout:/ 2024-03-19T02:51:04.091 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/fs/selinux 2024-03-19T02:51:04.091 INFO:teuthology.orchestra.run.smithi033.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T02:51:04.091 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/mqueue 2024-03-19T02:51:04.091 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/hugepages 2024-03-19T02:51:04.091 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/kernel/debug 2024-03-19T02:51:04.091 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/kernel/tracing 2024-03-19T02:51:04.091 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/kernel/config 2024-03-19T02:51:04.091 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/fs/fuse/connections 2024-03-19T02:51:04.091 INFO:teuthology.orchestra.run.smithi033.stdout:/run/credentials/systemd-sysctl.service 2024-03-19T02:51:04.091 INFO:teuthology.orchestra.run.smithi033.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-19T02:51:04.091 INFO:teuthology.orchestra.run.smithi033.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-19T02:51:04.091 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/nfs/rpc_pipefs 2024-03-19T02:51:04.091 INFO:teuthology.orchestra.run.smithi033.stdout:/run/user/1000 2024-03-19T02:51:04.092 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/ceph 2024-03-19T02:51:04.092 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay 2024-03-19T02:51:04.092 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/d40815713b6d8abf13551e5b03e4726efec7d920162aacc3167eb822ccd4f341/merged 2024-03-19T02:51:04.092 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/f26473e65fe17f5cc9a064aa0a0705b7cf6cdfb5557e691edb195c421fe9311e/merged 2024-03-19T02:51:04.092 INFO:teuthology.orchestra.run.smithi033.stdout:/run/user/0 2024-03-19T02:51:04.092 INFO:teuthology.orchestra.run.smithi033.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T02:51:04.092 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/458cde7783a31ad935025ecce83257fe525e84dfc943bc5ddf3711c44dc9cb47/merged 2024-03-19T02:51:04.092 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/4781736ffa71eef74ba07d8e57fa30a2a7dc9c2ea614310d94ac0a0ba14c808c/merged 2024-03-19T02:51:04.092 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/a6b836646400b0531dc3e7adb38a6372d800021d4cfe7dcdc74b23fb4bff575d/merged 2024-03-19T02:51:04.092 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/a0b05b7ceab9ec0d57bb302cc867219b12bfe23a430eb417e4c02683d743e203/merged 2024-03-19T02:51:04.092 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/264ba1fe315f2d024742741205d4d32695fa53dc0074ea52b5dac59a93dcc6b7/merged 2024-03-19T02:51:04.092 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/a592735a9ef96a00521c4d49ff5be4085daca00927f58bcae50da2dd3c634979/merged 2024-03-19T02:51:04.092 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/547c2dfe92bf1ab04750857a84566c04e128f7d6f517b355478a8dfa63b23f88/merged 2024-03-19T02:51:04.092 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/4613b7ef0ee9acd7aadf0117330f3af918303ea30b265b69578eb1c58954e1b9/merged 2024-03-19T02:51:04.093 INFO:teuthology.orchestra.run.smithi033.stdout:/run/netns 2024-03-19T02:51:04.093 INFO:teuthology.orchestra.run.smithi033.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-19T02:51:04.093 INFO:teuthology.orchestra.run.smithi033.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-19T02:51:04.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:51:04.093 DEBUG:teuthology.orchestra.run.smithi033:> ls /sys/fs/fuse/connections 2024-03-19T02:51:04.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi033.stderr:2024-03-19T02:51:04.094+0000 7f8acd7f6480 -1 init, newargv = 0x55b00dfe10a0 newargc=15 2024-03-19T02:51:04.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi033.stderr:ceph-fuse[76045]: starting ceph client 2024-03-19T02:51:04.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi033.stderr:ceph-fuse[76045]: starting fuse 2024-03-19T02:51:04.110 INFO:teuthology.orchestra.run.smithi033.stdout:119 2024-03-19T02:51:04.110 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-03-19T02:51:04.110 DEBUG:teuthology.orchestra.run.smithi033:> sudo stdin-killer -- python3 -c ' 2024-03-19T02:51:04.111 DEBUG:teuthology.orchestra.run.smithi033:> import glob 2024-03-19T02:51:04.111 DEBUG:teuthology.orchestra.run.smithi033:> import re 2024-03-19T02:51:04.111 DEBUG:teuthology.orchestra.run.smithi033:> import os 2024-03-19T02:51:04.111 DEBUG:teuthology.orchestra.run.smithi033:> import subprocess 2024-03-19T02:51:04.111 DEBUG:teuthology.orchestra.run.smithi033:> 2024-03-19T02:51:04.111 DEBUG:teuthology.orchestra.run.smithi033:> def _find_admin_socket(client_name): 2024-03-19T02:51:04.111 DEBUG:teuthology.orchestra.run.smithi033:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-19T02:51:04.111 DEBUG:teuthology.orchestra.run.smithi033:> files = glob.glob(asok_path) 2024-03-19T02:51:04.111 DEBUG:teuthology.orchestra.run.smithi033:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-19T02:51:04.111 DEBUG:teuthology.orchestra.run.smithi033:> 2024-03-19T02:51:04.111 DEBUG:teuthology.orchestra.run.smithi033:> # Given a non-glob path, it better be there 2024-03-19T02:51:04.111 DEBUG:teuthology.orchestra.run.smithi033:> if "*" not in asok_path: 2024-03-19T02:51:04.112 DEBUG:teuthology.orchestra.run.smithi033:> assert(len(files) == 1) 2024-03-19T02:51:04.112 DEBUG:teuthology.orchestra.run.smithi033:> return files[0] 2024-03-19T02:51:04.112 DEBUG:teuthology.orchestra.run.smithi033:> 2024-03-19T02:51:04.112 DEBUG:teuthology.orchestra.run.smithi033:> for f in files: 2024-03-19T02:51:04.112 DEBUG:teuthology.orchestra.run.smithi033:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-19T02:51:04.112 DEBUG:teuthology.orchestra.run.smithi033:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-19T02:51:04.112 DEBUG:teuthology.orchestra.run.smithi033:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-19T02:51:04.112 DEBUG:teuthology.orchestra.run.smithi033:> contents = proc_f.read() 2024-03-19T02:51:04.112 DEBUG:teuthology.orchestra.run.smithi033:> if mountpoint in contents: 2024-03-19T02:51:04.112 DEBUG:teuthology.orchestra.run.smithi033:> return f 2024-03-19T02:51:04.112 DEBUG:teuthology.orchestra.run.smithi033:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-19T02:51:04.112 DEBUG:teuthology.orchestra.run.smithi033:> 2024-03-19T02:51:04.112 DEBUG:teuthology.orchestra.run.smithi033:> print(_find_admin_socket("client.admin")) 2024-03-19T02:51:04.112 DEBUG:teuthology.orchestra.run.smithi033:> ' 2024-03-19T02:51:04.174 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:03 smithi033 ceph-mon[31973]: pgmap v305: 129 pgs: 129 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-19T02:51:04.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:03 smithi096 ceph-mon[35946]: pgmap v305: 129 pgs: 129 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-19T02:51:04.232 INFO:teuthology.orchestra.run.smithi033.stdout:/var/run/ceph/ceph-client.admin.76045.asok 2024-03-19T02:51:04.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:51:04.243 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.76045.asok 2024-03-19T02:51:04.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:51:04.243 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.76045.asok status 2024-03-19T02:51:04.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:03 smithi155 ceph-mon[35804]: pgmap v305: 129 pgs: 129 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-19T02:51:04.382 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:51:04.383 INFO:teuthology.orchestra.run.smithi033.stdout: "metadata": { 2024-03-19T02:51:04.383 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_sha1": "b6ae24918b03b1b8a19c9263a50655bba80bb810", 2024-03-19T02:51:04.383 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_version": "ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev)", 2024-03-19T02:51:04.383 INFO:teuthology.orchestra.run.smithi033.stdout: "entity_id": "admin", 2024-03-19T02:51:04.383 INFO:teuthology.orchestra.run.smithi033.stdout: "hostname": "smithi033", 2024-03-19T02:51:04.383 INFO:teuthology.orchestra.run.smithi033.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-19T02:51:04.383 INFO:teuthology.orchestra.run.smithi033.stdout: "pid": "76045", 2024-03-19T02:51:04.383 INFO:teuthology.orchestra.run.smithi033.stdout: "root": "/" 2024-03-19T02:51:04.383 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T02:51:04.383 INFO:teuthology.orchestra.run.smithi033.stdout: "dentry_count": 0, 2024-03-19T02:51:04.383 INFO:teuthology.orchestra.run.smithi033.stdout: "dentry_pinned_count": 0, 2024-03-19T02:51:04.383 INFO:teuthology.orchestra.run.smithi033.stdout: "id": 14748, 2024-03-19T02:51:04.383 INFO:teuthology.orchestra.run.smithi033.stdout: "inst": { 2024-03-19T02:51:04.384 INFO:teuthology.orchestra.run.smithi033.stdout: "name": { 2024-03-19T02:51:04.384 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "client", 2024-03-19T02:51:04.384 INFO:teuthology.orchestra.run.smithi033.stdout: "num": 14748 2024-03-19T02:51:04.384 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T02:51:04.384 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": { 2024-03-19T02:51:04.384 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-03-19T02:51:04.384 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "192.168.0.1:0", 2024-03-19T02:51:04.384 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3449994219 2024-03-19T02:51:04.384 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T02:51:04.384 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T02:51:04.384 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": { 2024-03-19T02:51:04.384 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-03-19T02:51:04.384 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "192.168.0.1:0", 2024-03-19T02:51:04.384 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3449994219 2024-03-19T02:51:04.384 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T02:51:04.384 INFO:teuthology.orchestra.run.smithi033.stdout: "inst_str": "client.14748 192.168.0.1:0/3449994219", 2024-03-19T02:51:04.385 INFO:teuthology.orchestra.run.smithi033.stdout: "addr_str": "192.168.0.1:0/3449994219", 2024-03-19T02:51:04.385 INFO:teuthology.orchestra.run.smithi033.stdout: "inode_count": 1, 2024-03-19T02:51:04.385 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_epoch": 17, 2024-03-19T02:51:04.385 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_epoch": 93, 2024-03-19T02:51:04.385 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_epoch_barrier": 0, 2024-03-19T02:51:04.385 INFO:teuthology.orchestra.run.smithi033.stdout: "blocklisted": false, 2024-03-19T02:51:04.385 INFO:teuthology.orchestra.run.smithi033.stdout: "fs_name": "cephfs" 2024-03-19T02:51:04.385 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:51:04.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:51:04.392 DEBUG:teuthology.orchestra.run.smithi033:> stat --file-system '--printf=%T 2024-03-19T02:51:04.393 DEBUG:teuthology.orchestra.run.smithi033:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-19T02:51:04.408 INFO:teuthology.orchestra.run.smithi033.stdout:fuseblk 2024-03-19T02:51:04.408 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-19T02:51:04.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:51:04.408 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-19T02:51:04.480 DEBUG:teuthology.orchestra.run.smithi033:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-19T02:51:04.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-19T02:51:04.588 INFO:teuthology.orchestra.run.smithi033.stdout:# file: . 2024-03-19T02:51:04.588 INFO:teuthology.orchestra.run.smithi033.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-19T02:51:04.588 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:51:04.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:51:04.594 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-19T02:51:04.594 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T02:51:04.610 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:51:04.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.754+0000 7efe23589640 1 -- 172.21.15.33:0/1394814568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe1c1022f0 msgr2=0x7efe1c108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:04.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.754+0000 7efe23589640 1 --2- 172.21.15.33:0/1394814568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe1c1022f0 0x7efe1c108b80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7efe0c009900 tx=0x7efe0c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:51:04.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.754+0000 7efe23589640 1 -- 172.21.15.33:0/1394814568 shutdown_connections 2024-03-19T02:51:04.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.754+0000 7efe23589640 1 --2- 172.21.15.33:0/1394814568 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe1c101930 0x7efe1c101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:04.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.754+0000 7efe23589640 1 --2- 172.21.15.33:0/1394814568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe1c1022f0 0x7efe1c108b80 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:04.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.754+0000 7efe23589640 1 --2- 172.21.15.33:0/1394814568 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe1c100740 0x7efe1c100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:04.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.754+0000 7efe23589640 1 -- 172.21.15.33:0/1394814568 >> 172.21.15.33:0/1394814568 conn(0x7efe1c0fbcb0 msgr2=0x7efe1c0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:04.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.754+0000 7efe23589640 1 -- 172.21.15.33:0/1394814568 shutdown_connections 2024-03-19T02:51:04.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.754+0000 7efe23589640 1 -- 172.21.15.33:0/1394814568 wait complete. 2024-03-19T02:51:04.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.754+0000 7efe23589640 1 Processor -- start 2024-03-19T02:51:04.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.754+0000 7efe23589640 1 -- start start 2024-03-19T02:51:04.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.755+0000 7efe23589640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe1c100740 0x7efe1c199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:04.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.755+0000 7efe23589640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe1c101930 0x7efe1c19a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:04.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.755+0000 7efe23589640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe1c1022f0 0x7efe1c1a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:04.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.755+0000 7efe23589640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efe1c109b10 con 0x7efe1c101930 2024-03-19T02:51:04.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.755+0000 7efe23589640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efe1c109990 con 0x7efe1c1022f0 2024-03-19T02:51:04.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.755+0000 7efe23589640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7efe1c109c90 con 0x7efe1c100740 2024-03-19T02:51:04.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.755+0000 7efe20b24640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe1c101930 0x7efe1c19a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:04.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.755+0000 7efe20b24640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe1c101930 0x7efe1c19a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57226/0 (socket says 172.21.15.33:57226) 2024-03-19T02:51:04.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.755+0000 7efe20b24640 1 -- 172.21.15.33:0/1930598171 learned_addr learned my addr 172.21.15.33:0/1930598171 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:04.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.755+0000 7efe21b26640 1 --2- 172.21.15.33:0/1930598171 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe1c1022f0 0x7efe1c1a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:04.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.755+0000 7efe21325640 1 --2- 172.21.15.33:0/1930598171 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe1c100740 0x7efe1c199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:04.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.755+0000 7efe20b24640 1 -- 172.21.15.33:0/1930598171 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe1c100740 msgr2=0x7efe1c199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:04.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.755+0000 7efe20b24640 1 --2- 172.21.15.33:0/1930598171 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe1c100740 0x7efe1c199c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:04.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.755+0000 7efe20b24640 1 -- 172.21.15.33:0/1930598171 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe1c1022f0 msgr2=0x7efe1c1a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:04.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.755+0000 7efe20b24640 1 --2- 172.21.15.33:0/1930598171 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe1c1022f0 0x7efe1c1a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:04.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.755+0000 7efe20b24640 1 -- 172.21.15.33:0/1930598171 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe1c1a18d0 con 0x7efe1c101930 2024-03-19T02:51:04.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.755+0000 7efe20b24640 1 --2- 172.21.15.33:0/1930598171 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe1c101930 0x7efe1c19a1d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7efe1000e9f0 tx=0x7efe1000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:04.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.755+0000 7efe1a7fc640 1 -- 172.21.15.33:0/1930598171 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe1000cd80 con 0x7efe1c101930 2024-03-19T02:51:04.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.755+0000 7efe1a7fc640 1 -- 172.21.15.33:0/1930598171 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe100054d0 con 0x7efe1c101930 2024-03-19T02:51:04.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.755+0000 7efe23589640 1 -- 172.21.15.33:0/1930598171 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe1c1a1bc0 con 0x7efe1c101930 2024-03-19T02:51:04.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.756+0000 7efe1a7fc640 1 -- 172.21.15.33:0/1930598171 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe10010490 con 0x7efe1c101930 2024-03-19T02:51:04.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.756+0000 7efe23589640 1 -- 172.21.15.33:0/1930598171 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe1c1a20c0 con 0x7efe1c101930 2024-03-19T02:51:04.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.756+0000 7efe23589640 1 -- 172.21.15.33:0/1930598171 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe1c1a22f0 con 0x7efe1c101930 2024-03-19T02:51:04.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.758+0000 7efe1a7fc640 1 -- 172.21.15.33:0/1930598171 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7efe10005020 con 0x7efe1c101930 2024-03-19T02:51:04.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.759+0000 7efe1a7fc640 1 --2- 172.21.15.33:0/1930598171 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7efe040a2800 0x7efe040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:04.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.759+0000 7efe21325640 1 --2- 172.21.15.33:0/1930598171 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7efe040a2800 0x7efe040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:04.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.759+0000 7efe1a7fc640 1 -- 172.21.15.33:0/1930598171 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7efe100cf000 con 0x7efe1c101930 2024-03-19T02:51:04.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.759+0000 7efe21325640 1 --2- 172.21.15.33:0/1930598171 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7efe040a2800 0x7efe040a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7efe1c19ab60 tx=0x7efe14009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:04.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.763+0000 7efe1a7fc640 1 -- 172.21.15.33:0/1930598171 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efe1009be60 con 0x7efe1c101930 2024-03-19T02:51:04.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.949+0000 7efe23589640 1 -- 172.21.15.33:0/1930598171 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efe1c1a2560 con 0x7efe1c101930 2024-03-19T02:51:04.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.950+0000 7efe1a7fc640 1 -- 172.21.15.33:0/1930598171 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24043 (secure 0 0 0) 0x7efe1009b760 con 0x7efe1c101930 2024-03-19T02:51:04.951 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:51:04.952 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":93,"fsid":"2b7a8938-e59a-11ee-95c9-87774f69a715","created":"2024-03-19T02:42:37.912179+0000","modified":"2024-03-19T02:50:59.241668+0000","last_up_change":"2024-03-19T02:50:01.539392+0000","last_in_change":"2024-03-19T02:49:40.534415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:46:09.963737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:50:50.069641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:50:51.381064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:50:55.476667+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"090a89ab-6b7b-4317-8823-178d4212b2f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6803","nonce":1633124995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6805","nonce":1633124995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6809","nonce":1633124995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6807","nonce":1633124995}]},"public_addr":"172.21.15.33:6803/1633124995","cluster_addr":"172.21.15.33:6805/1633124995","heartbeat_back_addr":"172.21.15.33:6809/1633124995","heartbeat_front_addr":"172.21.15.33:6807/1633124995","state":["exists","up"]},{"osd":1,"uuid":"f50eec55-1956-4486-8593-f2c8fa610e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6801","nonce":340144398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6803","nonce":340144398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6807","nonce":340144398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6805","nonce":340144398}]},"public_addr":"172.21.15.96:6801/340144398","cluster_addr":"172.21.15.96:6803/340144398","heartbeat_back_addr":"172.21.15.96:6807/340144398","heartbeat_front_addr":"172.21.15.96:6805/340144398","state":["exists","up"]},{"osd":2,"uuid":"893a2d62-80d9-4e5e-bc6a-a1c8c681d652","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6801","nonce":3388743876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6803","nonce":3388743876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6807","nonce":3388743876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6805","nonce":3388743876}]},"public_addr":"172.21.15.155:6801/3388743876","cluster_addr":"172.21.15.155:6803/3388743876","heartbeat_back_addr":"172.21.15.155:6807/3388743876","heartbeat_front_addr":"172.21.15.155:6805/3388743876","state":["exists","up"]},{"osd":3,"uuid":"b07e0482-4281-4d32-9dfe-9995d7246175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6811","nonce":677007171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6813","nonce":677007171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6817","nonce":677007171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6815","nonce":677007171}]},"public_addr":"172.21.15.33:6811/677007171","cluster_addr":"172.21.15.33:6813/677007171","heartbeat_back_addr":"172.21.15.33:6817/677007171","heartbeat_front_addr":"172.21.15.33:6815/677007171","state":["exists","up"]},{"osd":4,"uuid":"a988384b-af6b-47c6-8c07-5c9673acee45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6809","nonce":3106158554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6811","nonce":3106158554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6815","nonce":3106158554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6813","nonce":3106158554}]},"public_addr":"172.21.15.96:6809/3106158554","cluster_addr":"172.21.15.96:6811/3106158554","heartbeat_back_addr":"172.21.15.96:6815/3106158554","heartbeat_front_addr":"172.21.15.96:6813/3106158554","state":["exists","up"]},{"osd":5,"uuid":"198dc992-96f8-4c52-96ee-1fd5de148bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6809","nonce":327293324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6811","nonce":327293324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6815","nonce":327293324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6813","nonce":327293324}]},"public_addr":"172.21.15.155:6809/327293324","cluster_addr":"172.21.15.155:6811/327293324","heartbeat_back_addr":"172.21.15.155:6815/327293324","heartbeat_front_addr":"172.21.15.155:6813/327293324","state":["exists","up"]},{"osd":6,"uuid":"a7278ec0-ab2e-49ed-be53-84d8b9e24177","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6819","nonce":873871509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6821","nonce":873871509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6825","nonce":873871509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6823","nonce":873871509}]},"public_addr":"172.21.15.33:6819/873871509","cluster_addr":"172.21.15.33:6821/873871509","heartbeat_back_addr":"172.21.15.33:6825/873871509","heartbeat_front_addr":"172.21.15.33:6823/873871509","state":["exists","up"]},{"osd":7,"uuid":"727e133d-aa24-4ffc-90e2-266eb2aab8c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6817","nonce":2334123891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6819","nonce":2334123891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6823","nonce":2334123891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6821","nonce":2334123891}]},"public_addr":"172.21.15.96:6817/2334123891","cluster_addr":"172.21.15.96:6819/2334123891","heartbeat_back_addr":"172.21.15.96:6823/2334123891","heartbeat_front_addr":"172.21.15.96:6821/2334123891","state":["exists","up"]},{"osd":8,"uuid":"bb20eec0-39e5-4b86-9db5-486b1840cdad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6817","nonce":3498516903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6819","nonce":3498516903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6823","nonce":3498516903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6821","nonce":3498516903}]},"public_addr":"172.21.15.155:6817/3498516903","cluster_addr":"172.21.15.155:6819/3498516903","heartbeat_back_addr":"172.21.15.155:6823/3498516903","heartbeat_front_addr":"172.21.15.155:6821/3498516903","state":["exists","up"]},{"osd":9,"uuid":"179c6228-3e23-4f65-91a6-1ef0005d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6827","nonce":760777213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6829","nonce":760777213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6833","nonce":760777213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6831","nonce":760777213}]},"public_addr":"172.21.15.33:6827/760777213","cluster_addr":"172.21.15.33:6829/760777213","heartbeat_back_addr":"172.21.15.33:6833/760777213","heartbeat_front_addr":"172.21.15.33:6831/760777213","state":["exists","up"]},{"osd":10,"uuid":"be17b478-3c2e-41fb-bede-958333acdf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6825","nonce":1404898482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6827","nonce":1404898482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6831","nonce":1404898482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6829","nonce":1404898482}]},"public_addr":"172.21.15.96:6825/1404898482","cluster_addr":"172.21.15.96:6827/1404898482","heartbeat_back_addr":"172.21.15.96:6831/1404898482","heartbeat_front_addr":"172.21.15.96:6829/1404898482","state":["exists","up"]},{"osd":11,"uuid":"c0903ed2-cc1e-4c18-8731-1f0ee4dd0457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6825","nonce":792352827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6827","nonce":792352827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6831","nonce":792352827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6829","nonce":792352827}]},"public_addr":"172.21.15.155:6825/792352827","cluster_addr":"172.21.15.155:6827/792352827","heartbeat_back_addr":"172.21.15.155:6831/792352827","heartbeat_front_addr":"172.21.15.155:6829/792352827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:17.942801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:42.109449+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:05.981599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:32.069574+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:57.431015+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:22.448592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:48.526515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:14.498352+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:40.175426+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:06.364006+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:32.853161+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:59.073538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:6801/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:6800/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/1067768047":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/729379453":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2586588742":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/2932814165":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6801/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2212564963":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6800/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/3275126058":"2024-03-20T02:43:39.622218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:51:04.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.952+0000 7efe23589640 1 -- 172.21.15.33:0/1930598171 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7efe040a2800 msgr2=0x7efe040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:04.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.952+0000 7efe23589640 1 --2- 172.21.15.33:0/1930598171 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7efe040a2800 0x7efe040a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7efe1c19ab60 tx=0x7efe14009500 comp rx=0 tx=0).stop 2024-03-19T02:51:04.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.952+0000 7efe23589640 1 -- 172.21.15.33:0/1930598171 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe1c101930 msgr2=0x7efe1c19a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:04.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.952+0000 7efe23589640 1 --2- 172.21.15.33:0/1930598171 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe1c101930 0x7efe1c19a1d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7efe1000e9f0 tx=0x7efe1000edf0 comp rx=0 tx=0).stop 2024-03-19T02:51:04.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.952+0000 7efe23589640 1 -- 172.21.15.33:0/1930598171 shutdown_connections 2024-03-19T02:51:04.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.952+0000 7efe23589640 1 --2- 172.21.15.33:0/1930598171 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7efe040a2800 0x7efe040a4cc0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:04.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.952+0000 7efe23589640 1 --2- 172.21.15.33:0/1930598171 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe1c1022f0 0x7efe1c1a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:04.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.952+0000 7efe23589640 1 --2- 172.21.15.33:0/1930598171 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe1c101930 0x7efe1c19a1d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:04.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.952+0000 7efe23589640 1 --2- 172.21.15.33:0/1930598171 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe1c100740 0x7efe1c199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:04.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.952+0000 7efe23589640 1 -- 172.21.15.33:0/1930598171 >> 172.21.15.33:0/1930598171 conn(0x7efe1c0fbcb0 msgr2=0x7efe1c0fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:04.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.952+0000 7efe23589640 1 -- 172.21.15.33:0/1930598171 shutdown_connections 2024-03-19T02:51:04.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:04.952+0000 7efe23589640 1 -- 172.21.15.33:0/1930598171 wait complete. 2024-03-19T02:51:04.964 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:46:09.963737+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T02:50:50.069641+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T02:50:51.381064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T02:50:55.476667+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T02:51:04.964 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T02:51:05.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.070+0000 7f05feb76640 1 -- 172.21.15.33:0/454834433 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f05f8100720 msgr2=0x7f05f8100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:05.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.070+0000 7f05feb76640 1 --2- 172.21.15.33:0/454834433 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f05f8100720 0x7f05f8100b40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f05ec009900 tx=0x7f05ec02bc30 comp rx=0 tx=0).stop 2024-03-19T02:51:05.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.070+0000 7f05feb76640 1 -- 172.21.15.33:0/454834433 shutdown_connections 2024-03-19T02:51:05.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.070+0000 7f05feb76640 1 --2- 172.21.15.33:0/454834433 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05f8101910 0x7f05f8101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:05.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.070+0000 7f05feb76640 1 --2- 172.21.15.33:0/454834433 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05f81022d0 0x7f05f8108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:05.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.070+0000 7f05feb76640 1 --2- 172.21.15.33:0/454834433 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f05f8100720 0x7f05f8100b40 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:05.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.070+0000 7f05feb76640 1 -- 172.21.15.33:0/454834433 >> 172.21.15.33:0/454834433 conn(0x7f05f80fbcb0 msgr2=0x7f05f80fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:05.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.071+0000 7f05feb76640 1 -- 172.21.15.33:0/454834433 shutdown_connections 2024-03-19T02:51:05.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.071+0000 7f05feb76640 1 -- 172.21.15.33:0/454834433 wait complete. 2024-03-19T02:51:05.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.071+0000 7f05feb76640 1 Processor -- start 2024-03-19T02:51:05.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.071+0000 7f05feb76640 1 -- start start 2024-03-19T02:51:05.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.071+0000 7f05feb76640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f05f8100720 0x7f05f80694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:05.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.071+0000 7f05feb76640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05f8101910 0x7f05f819c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:05.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.071+0000 7f05feb76640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05f81022d0 0x7f05f819ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:05.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.071+0000 7f05feb76640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f05f810c270 con 0x7f05f8100720 2024-03-19T02:51:05.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.071+0000 7f05feb76640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f05f810c0f0 con 0x7f05f8101910 2024-03-19T02:51:05.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.071+0000 7f05feb76640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f05f810c3f0 con 0x7f05f81022d0 2024-03-19T02:51:05.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.071+0000 7f05fc912640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f05f8100720 0x7f05f80694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:05.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.071+0000 7f05fc912640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f05f8100720 0x7f05f80694f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57240/0 (socket says 172.21.15.33:57240) 2024-03-19T02:51:05.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.071+0000 7f05fc912640 1 -- 172.21.15.33:0/1056373741 learned_addr learned my addr 172.21.15.33:0/1056373741 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:05.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.071+0000 7f05f7fff640 1 --2- 172.21.15.33:0/1056373741 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05f8101910 0x7f05f819c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:05.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.072+0000 7f05fc912640 1 -- 172.21.15.33:0/1056373741 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05f81022d0 msgr2=0x7f05f819ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:05.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.072+0000 7f05fc912640 1 --2- 172.21.15.33:0/1056373741 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05f81022d0 0x7f05f819ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:05.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.072+0000 7f05fc912640 1 -- 172.21.15.33:0/1056373741 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05f8101910 msgr2=0x7f05f819c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:05.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.072+0000 7f05fc912640 1 --2- 172.21.15.33:0/1056373741 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05f8101910 0x7f05f819c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:05.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.072+0000 7f05fc912640 1 -- 172.21.15.33:0/1056373741 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05f819f6f0 con 0x7f05f8100720 2024-03-19T02:51:05.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.072+0000 7f05fc912640 1 --2- 172.21.15.33:0/1056373741 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f05f8100720 0x7f05f80694f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f05ec000c00 tx=0x7f05ec0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:05.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.072+0000 7f05f5ffb640 1 -- 172.21.15.33:0/1056373741 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05ec043020 con 0x7f05f8100720 2024-03-19T02:51:05.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.072+0000 7f05f5ffb640 1 -- 172.21.15.33:0/1056373741 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f05ec0044c0 con 0x7f05f8100720 2024-03-19T02:51:05.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.072+0000 7f05f5ffb640 1 -- 172.21.15.33:0/1056373741 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05ec003b40 con 0x7f05f8100720 2024-03-19T02:51:05.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.072+0000 7f05feb76640 1 -- 172.21.15.33:0/1056373741 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05f819f920 con 0x7f05f8100720 2024-03-19T02:51:05.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.072+0000 7f05feb76640 1 -- 172.21.15.33:0/1056373741 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05f819fe70 con 0x7f05f8100720 2024-03-19T02:51:05.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.073+0000 7f05feb76640 1 -- 172.21.15.33:0/1056373741 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05f8107b10 con 0x7f05f8100720 2024-03-19T02:51:05.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.074+0000 7f05f5ffb640 1 -- 172.21.15.33:0/1056373741 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f05ec046020 con 0x7f05f8100720 2024-03-19T02:51:05.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.075+0000 7f05f5ffb640 1 --2- 172.21.15.33:0/1056373741 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f05e80a2800 0x7f05e80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:05.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.075+0000 7f05f7fff640 1 --2- 172.21.15.33:0/1056373741 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f05e80a2800 0x7f05e80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:05.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.075+0000 7f05f5ffb640 1 -- 172.21.15.33:0/1056373741 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7f05ec0f0f30 con 0x7f05f8100720 2024-03-19T02:51:05.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.076+0000 7f05f7fff640 1 --2- 172.21.15.33:0/1056373741 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f05e80a2800 0x7f05e80a4cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f05f8069260 tx=0x7f05e400a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:05.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.080+0000 7f05f5ffb640 1 -- 172.21.15.33:0/1056373741 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f05ec04d020 con 0x7f05f8100720 2024-03-19T02:51:05.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:05 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1930598171' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:51:05.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.266+0000 7f05feb76640 1 -- 172.21.15.33:0/1056373741 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f05f80624c0 con 0x7f05f8100720 2024-03-19T02:51:05.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.266+0000 7f05f5ffb640 1 -- 172.21.15.33:0/1056373741 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f05ec03c350 con 0x7f05f8100720 2024-03-19T02:51:05.267 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 1 2024-03-19T02:51:05.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.268+0000 7f05feb76640 1 -- 172.21.15.33:0/1056373741 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f05e80a2800 msgr2=0x7f05e80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:05.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.268+0000 7f05feb76640 1 --2- 172.21.15.33:0/1056373741 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f05e80a2800 0x7f05e80a4cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f05f8069260 tx=0x7f05e400a2d0 comp rx=0 tx=0).stop 2024-03-19T02:51:05.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.268+0000 7f05feb76640 1 -- 172.21.15.33:0/1056373741 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f05f8100720 msgr2=0x7f05f80694f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:05.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.268+0000 7f05feb76640 1 --2- 172.21.15.33:0/1056373741 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f05f8100720 0x7f05f80694f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f05ec000c00 tx=0x7f05ec0040c0 comp rx=0 tx=0).stop 2024-03-19T02:51:05.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.268+0000 7f05feb76640 1 -- 172.21.15.33:0/1056373741 shutdown_connections 2024-03-19T02:51:05.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.268+0000 7f05feb76640 1 --2- 172.21.15.33:0/1056373741 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f05e80a2800 0x7f05e80a4cc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:05.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.268+0000 7f05feb76640 1 --2- 172.21.15.33:0/1056373741 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05f81022d0 0x7f05f819ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:05.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.268+0000 7f05feb76640 1 --2- 172.21.15.33:0/1056373741 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05f8101910 0x7f05f819c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:05.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.268+0000 7f05feb76640 1 --2- 172.21.15.33:0/1056373741 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f05f8100720 0x7f05f80694f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:05.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.268+0000 7f05feb76640 1 -- 172.21.15.33:0/1056373741 >> 172.21.15.33:0/1056373741 conn(0x7f05f80fbcb0 msgr2=0x7f05f8104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:05.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.268+0000 7f05feb76640 1 -- 172.21.15.33:0/1056373741 shutdown_connections 2024-03-19T02:51:05.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.269+0000 7f05feb76640 1 -- 172.21.15.33:0/1056373741 wait complete. 2024-03-19T02:51:05.272 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:05 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1930598171' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:51:05.280 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T02:51:05.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.385+0000 7f6f30eb8640 1 -- 172.21.15.33:0/890101609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f2c100740 msgr2=0x7f6f2c100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:05.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.385+0000 7f6f30eb8640 1 --2- 172.21.15.33:0/890101609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f2c100740 0x7f6f2c100b60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c009ca0 tx=0x7f6f1c02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:51:05.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.385+0000 7f6f30eb8640 1 -- 172.21.15.33:0/890101609 shutdown_connections 2024-03-19T02:51:05.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.385+0000 7f6f30eb8640 1 --2- 172.21.15.33:0/890101609 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f2c101930 0x7f6f2c101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:05.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.385+0000 7f6f30eb8640 1 --2- 172.21.15.33:0/890101609 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f2c1022f0 0x7f6f2c108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:05.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.385+0000 7f6f30eb8640 1 --2- 172.21.15.33:0/890101609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f2c100740 0x7f6f2c100b60 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:05.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.385+0000 7f6f30eb8640 1 -- 172.21.15.33:0/890101609 >> 172.21.15.33:0/890101609 conn(0x7f6f2c0fbcd0 msgr2=0x7f6f2c0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:05.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.385+0000 7f6f30eb8640 1 -- 172.21.15.33:0/890101609 shutdown_connections 2024-03-19T02:51:05.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.385+0000 7f6f30eb8640 1 -- 172.21.15.33:0/890101609 wait complete. 2024-03-19T02:51:05.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.386+0000 7f6f30eb8640 1 Processor -- start 2024-03-19T02:51:05.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.386+0000 7f6f30eb8640 1 -- start start 2024-03-19T02:51:05.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.386+0000 7f6f30eb8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f2c100740 0x7f6f2c197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:05.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.386+0000 7f6f30eb8640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f2c101930 0x7f6f2c198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:05.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.386+0000 7f6f30eb8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f2c1022f0 0x7f6f2c19f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:05.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.386+0000 7f6f30eb8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6f2c10c240 con 0x7f6f2c1022f0 2024-03-19T02:51:05.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.386+0000 7f6f30eb8640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6f2c10c0c0 con 0x7f6f2c100740 2024-03-19T02:51:05.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.386+0000 7f6f30eb8640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6f2c10c3c0 con 0x7f6f2c101930 2024-03-19T02:51:05.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.386+0000 7f6f2ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f2c1022f0 0x7f6f2c19f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:05.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.386+0000 7f6f2a59c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f2c100740 0x7f6f2c197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:05.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.387+0000 7f6f2ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f2c1022f0 0x7f6f2c19f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57254/0 (socket says 172.21.15.33:57254) 2024-03-19T02:51:05.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.387+0000 7f6f2ad9d640 1 -- 172.21.15.33:0/2741769431 learned_addr learned my addr 172.21.15.33:0/2741769431 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:05.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.387+0000 7f6f2a59c640 1 --2- 172.21.15.33:0/2741769431 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f2c100740 0x7f6f2c197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:60322/0 (socket says 172.21.15.33:60322) 2024-03-19T02:51:05.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.387+0000 7f6f29d9b640 1 --2- 172.21.15.33:0/2741769431 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f2c101930 0x7f6f2c198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:05.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.387+0000 7f6f2a59c640 1 -- 172.21.15.33:0/2741769431 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f2c101930 msgr2=0x7f6f2c198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:05.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.387+0000 7f6f2a59c640 1 --2- 172.21.15.33:0/2741769431 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f2c101930 0x7f6f2c198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:05.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.387+0000 7f6f2a59c640 1 -- 172.21.15.33:0/2741769431 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f2c1022f0 msgr2=0x7f6f2c19f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:05.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.387+0000 7f6f2a59c640 1 --2- 172.21.15.33:0/2741769431 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f2c1022f0 0x7f6f2c19f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:05.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.387+0000 7f6f2a59c640 1 -- 172.21.15.33:0/2741769431 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f2c19f6e0 con 0x7f6f2c100740 2024-03-19T02:51:05.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.387+0000 7f6f2ad9d640 1 --2- 172.21.15.33:0/2741769431 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f2c1022f0 0x7f6f2c19f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:05.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.387+0000 7f6f2a59c640 1 --2- 172.21.15.33:0/2741769431 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f2c100740 0x7f6f2c197ac0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c02c630 tx=0x7f6f1c0328a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:05.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.387+0000 7f6f237fe640 1 -- 172.21.15.33:0/2741769431 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c041020 con 0x7f6f2c100740 2024-03-19T02:51:05.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.387+0000 7f6f30eb8640 1 -- 172.21.15.33:0/2741769431 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f2c19f970 con 0x7f6f2c100740 2024-03-19T02:51:05.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.387+0000 7f6f237fe640 1 -- 172.21.15.33:0/2741769431 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f1c003e20 con 0x7f6f2c100740 2024-03-19T02:51:05.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.387+0000 7f6f237fe640 1 -- 172.21.15.33:0/2741769431 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c039040 con 0x7f6f2c100740 2024-03-19T02:51:05.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.387+0000 7f6f30eb8640 1 -- 172.21.15.33:0/2741769431 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f2c19fe30 con 0x7f6f2c100740 2024-03-19T02:51:05.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.388+0000 7f6f30eb8640 1 -- 172.21.15.33:0/2741769431 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f2c1a0000 con 0x7f6f2c100740 2024-03-19T02:51:05.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.389+0000 7f6f237fe640 1 -- 172.21.15.33:0/2741769431 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f6f1c044020 con 0x7f6f2c100740 2024-03-19T02:51:05.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.390+0000 7f6f237fe640 1 --2- 172.21.15.33:0/2741769431 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f6f180a2800 0x7f6f180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:05.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.390+0000 7f6f29d9b640 1 --2- 172.21.15.33:0/2741769431 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f6f180a2800 0x7f6f180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:05.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.390+0000 7f6f237fe640 1 -- 172.21.15.33:0/2741769431 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7f6f1c0f11c0 con 0x7f6f2c100740 2024-03-19T02:51:05.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.390+0000 7f6f29d9b640 1 --2- 172.21.15.33:0/2741769431 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f6f180a2800 0x7f6f180a4cc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6f14005fd0 tx=0x7f6f14005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:05.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.398+0000 7f6f237fe640 1 -- 172.21.15.33:0/2741769431 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6f1c04b020 con 0x7f6f2c100740 2024-03-19T02:51:05.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:05 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1930598171' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:51:05.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.581+0000 7f6f30eb8640 1 -- 172.21.15.33:0/2741769431 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6f2c1a0390 con 0x7f6f2c100740 2024-03-19T02:51:05.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.581+0000 7f6f237fe640 1 -- 172.21.15.33:0/2741769431 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f6f1c0bdfc0 con 0x7f6f2c100740 2024-03-19T02:51:05.583 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 32 2024-03-19T02:51:05.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.583+0000 7f6f30eb8640 1 -- 172.21.15.33:0/2741769431 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f6f180a2800 msgr2=0x7f6f180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:05.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.583+0000 7f6f30eb8640 1 --2- 172.21.15.33:0/2741769431 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f6f180a2800 0x7f6f180a4cc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6f14005fd0 tx=0x7f6f14005850 comp rx=0 tx=0).stop 2024-03-19T02:51:05.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.583+0000 7f6f30eb8640 1 -- 172.21.15.33:0/2741769431 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f2c100740 msgr2=0x7f6f2c197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:05.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.583+0000 7f6f30eb8640 1 --2- 172.21.15.33:0/2741769431 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f2c100740 0x7f6f2c197ac0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c02c630 tx=0x7f6f1c0328a0 comp rx=0 tx=0).stop 2024-03-19T02:51:05.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.584+0000 7f6f30eb8640 1 -- 172.21.15.33:0/2741769431 shutdown_connections 2024-03-19T02:51:05.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.584+0000 7f6f30eb8640 1 --2- 172.21.15.33:0/2741769431 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f6f180a2800 0x7f6f180a4cc0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:05.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.584+0000 7f6f30eb8640 1 --2- 172.21.15.33:0/2741769431 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f2c101930 0x7f6f2c198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:05.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.584+0000 7f6f30eb8640 1 --2- 172.21.15.33:0/2741769431 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6f2c1022f0 0x7f6f2c19f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:05.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.584+0000 7f6f30eb8640 1 --2- 172.21.15.33:0/2741769431 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6f2c100740 0x7f6f2c197ac0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:05.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.584+0000 7f6f30eb8640 1 -- 172.21.15.33:0/2741769431 >> 172.21.15.33:0/2741769431 conn(0x7f6f2c0fbcd0 msgr2=0x7f6f2c069e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:05.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.584+0000 7f6f30eb8640 1 -- 172.21.15.33:0/2741769431 shutdown_connections 2024-03-19T02:51:05.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.584+0000 7f6f30eb8640 1 -- 172.21.15.33:0/2741769431 wait complete. 2024-03-19T02:51:05.595 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T02:51:05.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.700+0000 7f53655b9640 1 -- 172.21.15.33:0/1769822491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f53601022e0 msgr2=0x7f5360108b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:05.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.700+0000 7f53655b9640 1 --2- 172.21.15.33:0/1769822491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f53601022e0 0x7f5360108b70 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5350009900 tx=0x7f535002bc30 comp rx=0 tx=0).stop 2024-03-19T02:51:05.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.700+0000 7f53655b9640 1 -- 172.21.15.33:0/1769822491 shutdown_connections 2024-03-19T02:51:05.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.700+0000 7f53655b9640 1 --2- 172.21.15.33:0/1769822491 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5360101920 0x7f5360101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:05.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.700+0000 7f53655b9640 1 --2- 172.21.15.33:0/1769822491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f53601022e0 0x7f5360108b70 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:05.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.700+0000 7f53655b9640 1 --2- 172.21.15.33:0/1769822491 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5360100730 0x7f5360100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:05.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.700+0000 7f53655b9640 1 -- 172.21.15.33:0/1769822491 >> 172.21.15.33:0/1769822491 conn(0x7f53600fbcc0 msgr2=0x7f53600fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:05.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.700+0000 7f53655b9640 1 -- 172.21.15.33:0/1769822491 shutdown_connections 2024-03-19T02:51:05.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.700+0000 7f53655b9640 1 -- 172.21.15.33:0/1769822491 wait complete. 2024-03-19T02:51:05.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.700+0000 7f53655b9640 1 Processor -- start 2024-03-19T02:51:05.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.701+0000 7f53655b9640 1 -- start start 2024-03-19T02:51:05.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.701+0000 7f53655b9640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5360100730 0x7f5360196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:05.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.701+0000 7f53655b9640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5360101920 0x7f5360196e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:05.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.701+0000 7f53655b9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f53601022e0 0x7f536019dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:05.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.701+0000 7f53655b9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f536010c260 con 0x7f53601022e0 2024-03-19T02:51:05.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.701+0000 7f53655b9640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f536010c0e0 con 0x7f5360100730 2024-03-19T02:51:05.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.701+0000 7f53655b9640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f536010c3e0 con 0x7f5360101920 2024-03-19T02:51:05.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.701+0000 7f535f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f53601022e0 0x7f536019dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:05.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.701+0000 7f535f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f53601022e0 0x7f536019dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57280/0 (socket says 172.21.15.33:57280) 2024-03-19T02:51:05.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.701+0000 7f535f7fe640 1 -- 172.21.15.33:0/13828432 learned_addr learned my addr 172.21.15.33:0/13828432 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:05.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.701+0000 7f535effd640 1 --2- 172.21.15.33:0/13828432 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5360100730 0x7f5360196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:05.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.701+0000 7f535e7fc640 1 --2- 172.21.15.33:0/13828432 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5360101920 0x7f5360196e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:05.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.701+0000 7f535f7fe640 1 -- 172.21.15.33:0/13828432 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5360101920 msgr2=0x7f5360196e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:05.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.701+0000 7f535f7fe640 1 --2- 172.21.15.33:0/13828432 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5360101920 0x7f5360196e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:05.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.701+0000 7f535f7fe640 1 -- 172.21.15.33:0/13828432 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5360100730 msgr2=0x7f5360196930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:05.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.701+0000 7f535f7fe640 1 --2- 172.21.15.33:0/13828432 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5360100730 0x7f5360196930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:05.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.701+0000 7f535f7fe640 1 -- 172.21.15.33:0/13828432 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f536019e550 con 0x7f53601022e0 2024-03-19T02:51:05.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.701+0000 7f535effd640 1 --2- 172.21.15.33:0/13828432 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5360100730 0x7f5360196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:05.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.701+0000 7f535f7fe640 1 --2- 172.21.15.33:0/13828432 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f53601022e0 0x7f536019dee0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5350009230 tx=0x7f53500040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:05.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.701+0000 7f5347fff640 1 -- 172.21.15.33:0/13828432 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5350043020 con 0x7f53601022e0 2024-03-19T02:51:05.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.701+0000 7f5347fff640 1 -- 172.21.15.33:0/13828432 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f53500044c0 con 0x7f53601022e0 2024-03-19T02:51:05.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.702+0000 7f53655b9640 1 -- 172.21.15.33:0/13828432 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f536019e7e0 con 0x7f53601022e0 2024-03-19T02:51:05.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.702+0000 7f5347fff640 1 -- 172.21.15.33:0/13828432 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5350003bc0 con 0x7f53601022e0 2024-03-19T02:51:05.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.702+0000 7f53655b9640 1 -- 172.21.15.33:0/13828432 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5360069df0 con 0x7f53601022e0 2024-03-19T02:51:05.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.703+0000 7f53655b9640 1 -- 172.21.15.33:0/13828432 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f536006a7b0 con 0x7f53601022e0 2024-03-19T02:51:05.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.704+0000 7f5347fff640 1 -- 172.21.15.33:0/13828432 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f5350046020 con 0x7f53601022e0 2024-03-19T02:51:05.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.704+0000 7f5347fff640 1 --2- 172.21.15.33:0/13828432 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f53480a4a90 0x7f53480a6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:05.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.705+0000 7f535effd640 1 --2- 172.21.15.33:0/13828432 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f53480a4a90 0x7f53480a6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:05.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.705+0000 7f5347fff640 1 -- 172.21.15.33:0/13828432 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7f53500f0f80 con 0x7f53601022e0 2024-03-19T02:51:05.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.705+0000 7f535effd640 1 --2- 172.21.15.33:0/13828432 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f53480a4a90 0x7f53480a6f50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f53580099a0 tx=0x7f5358009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:05.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.709+0000 7f5347fff640 1 -- 172.21.15.33:0/13828432 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f535004d020 con 0x7f53601022e0 2024-03-19T02:51:05.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.892+0000 7f53655b9640 1 -- 172.21.15.33:0/13828432 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5360107540 con 0x7f53601022e0 2024-03-19T02:51:05.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.892+0000 7f5347fff640 1 -- 172.21.15.33:0/13828432 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f53500bdde0 con 0x7f53601022e0 2024-03-19T02:51:05.893 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 64 2024-03-19T02:51:05.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.895+0000 7f53655b9640 1 -- 172.21.15.33:0/13828432 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f53480a4a90 msgr2=0x7f53480a6f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:05.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.895+0000 7f53655b9640 1 --2- 172.21.15.33:0/13828432 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f53480a4a90 0x7f53480a6f50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f53580099a0 tx=0x7f5358009500 comp rx=0 tx=0).stop 2024-03-19T02:51:05.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.895+0000 7f53655b9640 1 -- 172.21.15.33:0/13828432 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f53601022e0 msgr2=0x7f536019dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:05.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.895+0000 7f53655b9640 1 --2- 172.21.15.33:0/13828432 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f53601022e0 0x7f536019dee0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5350009230 tx=0x7f53500040c0 comp rx=0 tx=0).stop 2024-03-19T02:51:05.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.895+0000 7f53655b9640 1 -- 172.21.15.33:0/13828432 shutdown_connections 2024-03-19T02:51:05.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.895+0000 7f53655b9640 1 --2- 172.21.15.33:0/13828432 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f53480a4a90 0x7f53480a6f50 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:05.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.895+0000 7f53655b9640 1 --2- 172.21.15.33:0/13828432 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5360101920 0x7f5360196e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:05.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.895+0000 7f53655b9640 1 --2- 172.21.15.33:0/13828432 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f53601022e0 0x7f536019dee0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:05.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.895+0000 7f53655b9640 1 --2- 172.21.15.33:0/13828432 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5360100730 0x7f5360196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:05.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.895+0000 7f53655b9640 1 -- 172.21.15.33:0/13828432 >> 172.21.15.33:0/13828432 conn(0x7f53600fbcc0 msgr2=0x7f5360104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:05.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.895+0000 7f53655b9640 1 -- 172.21.15.33:0/13828432 shutdown_connections 2024-03-19T02:51:05.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:05.895+0000 7f53655b9640 1 -- 172.21.15.33:0/13828432 wait complete. 2024-03-19T02:51:05.906 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T02:51:06.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.014+0000 7f709b332640 1 -- 172.21.15.33:0/2793359753 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f70941022d0 msgr2=0x7f7094108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:06.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.014+0000 7f709b332640 1 --2- 172.21.15.33:0/2793359753 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f70941022d0 0x7f7094108b60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7084009ca0 tx=0x7f708402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:51:06.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.014+0000 7f709b332640 1 -- 172.21.15.33:0/2793359753 shutdown_connections 2024-03-19T02:51:06.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.014+0000 7f709b332640 1 --2- 172.21.15.33:0/2793359753 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7094101910 0x7f7094101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:06.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.014+0000 7f709b332640 1 --2- 172.21.15.33:0/2793359753 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f70941022d0 0x7f7094108b60 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:06.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.014+0000 7f709b332640 1 --2- 172.21.15.33:0/2793359753 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7094100720 0x7f7094100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:06.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.014+0000 7f709b332640 1 -- 172.21.15.33:0/2793359753 >> 172.21.15.33:0/2793359753 conn(0x7f70940fbcb0 msgr2=0x7f70940fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:06.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.014+0000 7f709b332640 1 -- 172.21.15.33:0/2793359753 shutdown_connections 2024-03-19T02:51:06.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.014+0000 7f709b332640 1 -- 172.21.15.33:0/2793359753 wait complete. 2024-03-19T02:51:06.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.015+0000 7f709b332640 1 Processor -- start 2024-03-19T02:51:06.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.015+0000 7f709b332640 1 -- start start 2024-03-19T02:51:06.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.015+0000 7f709b332640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7094100720 0x7f7094197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:06.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.015+0000 7f709b332640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7094101910 0x7f7094197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:06.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.015+0000 7f709b332640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70941022d0 0x7f709419efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:06.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.015+0000 7f709b332640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f709410c250 con 0x7f7094101910 2024-03-19T02:51:06.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.015+0000 7f709b332640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f709410c0d0 con 0x7f70941022d0 2024-03-19T02:51:06.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.015+0000 7f709b332640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f709410c3d0 con 0x7f7094100720 2024-03-19T02:51:06.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.015+0000 7f70988cd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7094101910 0x7f7094197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:06.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.015+0000 7f70988cd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7094101910 0x7f7094197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57298/0 (socket says 172.21.15.33:57298) 2024-03-19T02:51:06.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.015+0000 7f70988cd640 1 -- 172.21.15.33:0/2469138355 learned_addr learned my addr 172.21.15.33:0/2469138355 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:06.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.015+0000 7f70998cf640 1 --2- 172.21.15.33:0/2469138355 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70941022d0 0x7f709419efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:06.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.015+0000 7f70990ce640 1 --2- 172.21.15.33:0/2469138355 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7094100720 0x7f7094197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:06.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.016+0000 7f70988cd640 1 -- 172.21.15.33:0/2469138355 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7094100720 msgr2=0x7f7094197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:06.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.016+0000 7f70988cd640 1 --2- 172.21.15.33:0/2469138355 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7094100720 0x7f7094197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:06.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.016+0000 7f70988cd640 1 -- 172.21.15.33:0/2469138355 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70941022d0 msgr2=0x7f709419efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:06.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.016+0000 7f70988cd640 1 --2- 172.21.15.33:0/2469138355 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70941022d0 0x7f709419efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:06.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.016+0000 7f70988cd640 1 -- 172.21.15.33:0/2469138355 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f709419f640 con 0x7f7094101910 2024-03-19T02:51:06.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.016+0000 7f70988cd640 1 --2- 172.21.15.33:0/2469138355 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7094101910 0x7f7094197f60 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f708800e9f0 tx=0x7f708800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:06.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.016+0000 7f70927fc640 1 -- 172.21.15.33:0/2469138355 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708800cd80 con 0x7f7094101910 2024-03-19T02:51:06.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.016+0000 7f70927fc640 1 -- 172.21.15.33:0/2469138355 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f70880054d0 con 0x7f7094101910 2024-03-19T02:51:06.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.016+0000 7f709b332640 1 -- 172.21.15.33:0/2469138355 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f709419f930 con 0x7f7094101910 2024-03-19T02:51:06.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.016+0000 7f70927fc640 1 -- 172.21.15.33:0/2469138355 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7088010490 con 0x7f7094101910 2024-03-19T02:51:06.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.016+0000 7f709b332640 1 -- 172.21.15.33:0/2469138355 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f709419fe80 con 0x7f7094101910 2024-03-19T02:51:06.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.017+0000 7f709b332640 1 -- 172.21.15.33:0/2469138355 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70941a0110 con 0x7f7094101910 2024-03-19T02:51:06.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.018+0000 7f70927fc640 1 -- 172.21.15.33:0/2469138355 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f7088005020 con 0x7f7094101910 2024-03-19T02:51:06.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.019+0000 7f70927fc640 1 --2- 172.21.15.33:0/2469138355 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f707c0a2880 0x7f707c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:06.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.019+0000 7f70990ce640 1 --2- 172.21.15.33:0/2469138355 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f707c0a2880 0x7f707c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:06.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.019+0000 7f70927fc640 1 -- 172.21.15.33:0/2469138355 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7f70880cef70 con 0x7f7094101910 2024-03-19T02:51:06.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.019+0000 7f70990ce640 1 --2- 172.21.15.33:0/2469138355 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f707c0a2880 0x7f707c0a4d40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7094101770 tx=0x7f708c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:06.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.023+0000 7f70927fc640 1 -- 172.21.15.33:0/2469138355 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f708809bdd0 con 0x7f7094101910 2024-03-19T02:51:06.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.206+0000 7f709b332640 1 -- 172.21.15.33:0/2469138355 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f70941a02e0 con 0x7f7094101910 2024-03-19T02:51:06.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.206+0000 7f70927fc640 1 -- 172.21.15.33:0/2469138355 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f708809b6d0 con 0x7f7094101910 2024-03-19T02:51:06.208 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 32 2024-03-19T02:51:06.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.208+0000 7f709b332640 1 -- 172.21.15.33:0/2469138355 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f707c0a2880 msgr2=0x7f707c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:06.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.208+0000 7f709b332640 1 --2- 172.21.15.33:0/2469138355 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f707c0a2880 0x7f707c0a4d40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7094101770 tx=0x7f708c005f90 comp rx=0 tx=0).stop 2024-03-19T02:51:06.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.208+0000 7f709b332640 1 -- 172.21.15.33:0/2469138355 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7094101910 msgr2=0x7f7094197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:06.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.208+0000 7f709b332640 1 --2- 172.21.15.33:0/2469138355 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7094101910 0x7f7094197f60 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f708800e9f0 tx=0x7f708800edf0 comp rx=0 tx=0).stop 2024-03-19T02:51:06.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.208+0000 7f709b332640 1 -- 172.21.15.33:0/2469138355 shutdown_connections 2024-03-19T02:51:06.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.208+0000 7f709b332640 1 --2- 172.21.15.33:0/2469138355 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f707c0a2880 0x7f707c0a4d40 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:06.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.209+0000 7f709b332640 1 --2- 172.21.15.33:0/2469138355 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70941022d0 0x7f709419efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:06.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.209+0000 7f709b332640 1 --2- 172.21.15.33:0/2469138355 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7094101910 0x7f7094197f60 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:06.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.209+0000 7f709b332640 1 --2- 172.21.15.33:0/2469138355 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7094100720 0x7f7094197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:06.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.209+0000 7f709b332640 1 -- 172.21.15.33:0/2469138355 >> 172.21.15.33:0/2469138355 conn(0x7f70940fbcb0 msgr2=0x7f70940fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:06.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.209+0000 7f709b332640 1 -- 172.21.15.33:0/2469138355 shutdown_connections 2024-03-19T02:51:06.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.209+0000 7f709b332640 1 -- 172.21.15.33:0/2469138355 wait complete. 2024-03-19T02:51:06.219 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:06 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1056373741' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:51:06.219 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:06 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2741769431' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:51:06.219 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:06 smithi033 ceph-mon[31973]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s wr, 0 op/s 2024-03-19T02:51:06.219 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:06 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/13828432' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:51:06.220 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-19T02:51:06.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:06 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1056373741' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:51:06.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:06 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2741769431' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:51:06.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:06 smithi155 ceph-mon[35804]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s wr, 0 op/s 2024-03-19T02:51:06.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:06 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/13828432' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:51:06.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.323+0000 7fae4ae20640 1 -- 172.21.15.33:0/3610136896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae44101050 msgr2=0x7fae441014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:06.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.323+0000 7fae4ae20640 1 --2- 172.21.15.33:0/3610136896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae44101050 0x7fae441014d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fae30009900 tx=0x7fae3002bc30 comp rx=0 tx=0).stop 2024-03-19T02:51:06.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.323+0000 7fae4ae20640 1 -- 172.21.15.33:0/3610136896 shutdown_connections 2024-03-19T02:51:06.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.323+0000 7fae4ae20640 1 --2- 172.21.15.33:0/3610136896 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae44101a10 0x7fae44108330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:06.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.323+0000 7fae4ae20640 1 --2- 172.21.15.33:0/3610136896 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae44101050 0x7fae441014d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:06.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.323+0000 7fae4ae20640 1 --2- 172.21.15.33:0/3610136896 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae440ffe70 0x7fae44100290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:06.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.323+0000 7fae4ae20640 1 -- 172.21.15.33:0/3610136896 >> 172.21.15.33:0/3610136896 conn(0x7fae440fb430 msgr2=0x7fae440fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:06.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.323+0000 7fae4ae20640 1 -- 172.21.15.33:0/3610136896 shutdown_connections 2024-03-19T02:51:06.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.323+0000 7fae4ae20640 1 -- 172.21.15.33:0/3610136896 wait complete. 2024-03-19T02:51:06.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.324+0000 7fae4ae20640 1 Processor -- start 2024-03-19T02:51:06.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.324+0000 7fae4ae20640 1 -- start start 2024-03-19T02:51:06.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.324+0000 7fae4ae20640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae440ffe70 0x7fae44197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:06.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.324+0000 7fae4ae20640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae44101050 0x7fae44198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:06.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.324+0000 7fae4ae20640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae44101a10 0x7fae4419f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:06.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.324+0000 7fae4ae20640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fae4410d410 con 0x7fae44101050 2024-03-19T02:51:06.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.324+0000 7fae4ae20640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fae4410d290 con 0x7fae44101a10 2024-03-19T02:51:06.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.324+0000 7fae4ae20640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fae4410d590 con 0x7fae440ffe70 2024-03-19T02:51:06.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.324+0000 7fae43fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae44101050 0x7fae44198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:06.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.324+0000 7fae43fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae44101050 0x7fae44198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57316/0 (socket says 172.21.15.33:57316) 2024-03-19T02:51:06.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.324+0000 7fae43fff640 1 -- 172.21.15.33:0/913570873 learned_addr learned my addr 172.21.15.33:0/913570873 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:06.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.324+0000 7fae493bd640 1 --2- 172.21.15.33:0/913570873 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae44101a10 0x7fae4419f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:06.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.324+0000 7fae48bbc640 1 --2- 172.21.15.33:0/913570873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae440ffe70 0x7fae44197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:06.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.324+0000 7fae43fff640 1 -- 172.21.15.33:0/913570873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae440ffe70 msgr2=0x7fae44197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:06.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.324+0000 7fae43fff640 1 --2- 172.21.15.33:0/913570873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae440ffe70 0x7fae44197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:06.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.324+0000 7fae43fff640 1 -- 172.21.15.33:0/913570873 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae44101a10 msgr2=0x7fae4419f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:06.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.324+0000 7fae43fff640 1 --2- 172.21.15.33:0/913570873 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae44101a10 0x7fae4419f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:06.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.324+0000 7fae43fff640 1 -- 172.21.15.33:0/913570873 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae4419f730 con 0x7fae44101050 2024-03-19T02:51:06.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.324+0000 7fae48bbc640 1 --2- 172.21.15.33:0/913570873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae440ffe70 0x7fae44197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:51:06.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.324+0000 7fae43fff640 1 --2- 172.21.15.33:0/913570873 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae44101050 0x7fae44198050 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fae30006010 tx=0x7fae30004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:06.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.325+0000 7fae41ffb640 1 -- 172.21.15.33:0/913570873 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae30043020 con 0x7fae44101050 2024-03-19T02:51:06.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.325+0000 7fae41ffb640 1 -- 172.21.15.33:0/913570873 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae30004440 con 0x7fae44101050 2024-03-19T02:51:06.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.325+0000 7fae4ae20640 1 -- 172.21.15.33:0/913570873 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae4419f9c0 con 0x7fae44101050 2024-03-19T02:51:06.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.325+0000 7fae41ffb640 1 -- 172.21.15.33:0/913570873 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae30003bf0 con 0x7fae44101050 2024-03-19T02:51:06.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.325+0000 7fae4ae20640 1 -- 172.21.15.33:0/913570873 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae4419ff90 con 0x7fae44101050 2024-03-19T02:51:06.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.326+0000 7fae4ae20640 1 -- 172.21.15.33:0/913570873 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae44068a60 con 0x7fae44101050 2024-03-19T02:51:06.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.327+0000 7fae41ffb640 1 -- 172.21.15.33:0/913570873 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fae30046020 con 0x7fae44101050 2024-03-19T02:51:06.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.327+0000 7fae41ffb640 1 --2- 172.21.15.33:0/913570873 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fae340a2800 0x7fae340a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:06.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.327+0000 7fae41ffb640 1 -- 172.21.15.33:0/913570873 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7fae300f1040 con 0x7fae44101050 2024-03-19T02:51:06.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.327+0000 7fae48bbc640 1 --2- 172.21.15.33:0/913570873 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fae340a2800 0x7fae340a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:06.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.328+0000 7fae48bbc640 1 --2- 172.21.15.33:0/913570873 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fae340a2800 0x7fae340a4cc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fae44100bc0 tx=0x7fae38008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:06.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.332+0000 7fae41ffb640 1 -- 172.21.15.33:0/913570873 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fae3004d020 con 0x7fae44101050 2024-03-19T02:51:06.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:06 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1056373741' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:51:06.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:06 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2741769431' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:51:06.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:06 smithi096 ceph-mon[35946]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s wr, 0 op/s 2024-03-19T02:51:06.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:06 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/13828432' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:51:06.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.527+0000 7fae4ae20640 1 -- 172.21.15.33:0/913570873 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fae44062420 con 0x7fae44101050 2024-03-19T02:51:06.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.527+0000 7fae41ffb640 1 -- 172.21.15.33:0/913570873 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v93) ==== 81+0+589 (secure 0 0 0) 0x7fae300bdea0 con 0x7fae44101050 2024-03-19T02:51:06.529 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:6801/906645748 2024-03-20T02:43:39.622218+0000 2024-03-19T02:51:06.529 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:6800/906645748 2024-03-20T02:43:39.622218+0000 2024-03-19T02:51:06.529 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/1067768047 2024-03-20T02:43:39.622218+0000 2024-03-19T02:51:06.529 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/729379453 2024-03-20T02:43:11.918076+0000 2024-03-19T02:51:06.529 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/2586588742 2024-03-20T02:43:39.622218+0000 2024-03-19T02:51:06.529 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/2932814165 2024-03-20T02:43:11.918076+0000 2024-03-19T02:51:06.529 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:6801/3342667183 2024-03-20T02:43:11.918076+0000 2024-03-19T02:51:06.529 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/2212564963 2024-03-20T02:43:11.918076+0000 2024-03-19T02:51:06.529 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:6800/3342667183 2024-03-20T02:43:11.918076+0000 2024-03-19T02:51:06.529 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/3275126058 2024-03-20T02:43:39.622218+0000 2024-03-19T02:51:06.530 INFO:teuthology.orchestra.run.smithi033.stderr:listed 10 entries 2024-03-19T02:51:06.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.529+0000 7fae4ae20640 1 -- 172.21.15.33:0/913570873 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fae340a2800 msgr2=0x7fae340a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:06.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.529+0000 7fae4ae20640 1 --2- 172.21.15.33:0/913570873 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fae340a2800 0x7fae340a4cc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fae44100bc0 tx=0x7fae38008040 comp rx=0 tx=0).stop 2024-03-19T02:51:06.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.529+0000 7fae4ae20640 1 -- 172.21.15.33:0/913570873 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae44101050 msgr2=0x7fae44198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:06.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.529+0000 7fae4ae20640 1 --2- 172.21.15.33:0/913570873 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae44101050 0x7fae44198050 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fae30006010 tx=0x7fae30004040 comp rx=0 tx=0).stop 2024-03-19T02:51:06.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.529+0000 7fae4ae20640 1 -- 172.21.15.33:0/913570873 shutdown_connections 2024-03-19T02:51:06.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.529+0000 7fae4ae20640 1 --2- 172.21.15.33:0/913570873 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fae340a2800 0x7fae340a4cc0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:06.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.529+0000 7fae4ae20640 1 --2- 172.21.15.33:0/913570873 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fae44101a10 0x7fae4419f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:06.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.529+0000 7fae4ae20640 1 --2- 172.21.15.33:0/913570873 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fae44101050 0x7fae44198050 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:06.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.529+0000 7fae4ae20640 1 --2- 172.21.15.33:0/913570873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fae440ffe70 0x7fae44197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:06.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.529+0000 7fae4ae20640 1 -- 172.21.15.33:0/913570873 >> 172.21.15.33:0/913570873 conn(0x7fae440fb430 msgr2=0x7fae44104500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:06.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.530+0000 7fae4ae20640 1 -- 172.21.15.33:0/913570873 shutdown_connections 2024-03-19T02:51:06.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.530+0000 7fae4ae20640 1 -- 172.21.15.33:0/913570873 wait complete. 2024-03-19T02:51:06.541 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-19T02:51:06.541 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T02:51:06.557 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:51:06.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.698+0000 7f7ad2dd4640 1 -- 172.21.15.33:0/221198338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7acc102310 msgr2=0x7f7acc108ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:06.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.698+0000 7f7ad2dd4640 1 --2- 172.21.15.33:0/221198338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7acc102310 0x7f7acc108ba0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4009900 tx=0x7f7ab402bc30 comp rx=0 tx=0).stop 2024-03-19T02:51:06.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.698+0000 7f7ad2dd4640 1 -- 172.21.15.33:0/221198338 shutdown_connections 2024-03-19T02:51:06.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.698+0000 7f7ad2dd4640 1 --2- 172.21.15.33:0/221198338 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7acc102310 0x7f7acc108ba0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:06.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.698+0000 7f7ad2dd4640 1 --2- 172.21.15.33:0/221198338 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7acc101950 0x7f7acc101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:06.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.698+0000 7f7ad2dd4640 1 --2- 172.21.15.33:0/221198338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7acc100760 0x7f7acc100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:06.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.698+0000 7f7ad2dd4640 1 -- 172.21.15.33:0/221198338 >> 172.21.15.33:0/221198338 conn(0x7f7acc0fbcd0 msgr2=0x7f7acc0fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:06.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.698+0000 7f7ad2dd4640 1 -- 172.21.15.33:0/221198338 shutdown_connections 2024-03-19T02:51:06.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.698+0000 7f7ad2dd4640 1 -- 172.21.15.33:0/221198338 wait complete. 2024-03-19T02:51:06.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.698+0000 7f7ad2dd4640 1 Processor -- start 2024-03-19T02:51:06.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.698+0000 7f7ad2dd4640 1 -- start start 2024-03-19T02:51:06.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.698+0000 7f7ad2dd4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7acc100760 0x7f7acc1046f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:06.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.699+0000 7f7ad2dd4640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7acc101950 0x7f7acc102d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:06.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.699+0000 7f7ad2dd4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7acc102310 0x7f7acc103270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:06.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.699+0000 7f7ad2dd4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7acc10c270 con 0x7f7acc102310 2024-03-19T02:51:06.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.699+0000 7f7ad2dd4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7acc10c0f0 con 0x7f7acc100760 2024-03-19T02:51:06.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.699+0000 7f7ad2dd4640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7acc10c3f0 con 0x7f7acc101950 2024-03-19T02:51:06.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.699+0000 7f7ad1371640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7acc102310 0x7f7acc103270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:06.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.699+0000 7f7ad1371640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7acc102310 0x7f7acc103270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57338/0 (socket says 172.21.15.33:57338) 2024-03-19T02:51:06.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.699+0000 7f7ad1371640 1 -- 172.21.15.33:0/3549893468 learned_addr learned my addr 172.21.15.33:0/3549893468 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:06.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.699+0000 7f7ad0b70640 1 --2- 172.21.15.33:0/3549893468 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7acc100760 0x7f7acc1046f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:06.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.699+0000 7f7acbfff640 1 --2- 172.21.15.33:0/3549893468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7acc101950 0x7f7acc102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:06.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.699+0000 7f7ad1371640 1 -- 172.21.15.33:0/3549893468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7acc101950 msgr2=0x7f7acc102d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:06.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.699+0000 7f7ad1371640 1 --2- 172.21.15.33:0/3549893468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7acc101950 0x7f7acc102d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:06.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.699+0000 7f7ad1371640 1 -- 172.21.15.33:0/3549893468 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7acc100760 msgr2=0x7f7acc1046f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:06.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.699+0000 7f7ad1371640 1 --2- 172.21.15.33:0/3549893468 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7acc100760 0x7f7acc1046f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:06.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.699+0000 7f7ad1371640 1 -- 172.21.15.33:0/3549893468 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7acc103b00 con 0x7f7acc102310 2024-03-19T02:51:06.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.699+0000 7f7ad1371640 1 --2- 172.21.15.33:0/3549893468 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7acc102310 0x7f7acc103270 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4009230 tx=0x7f7ab40040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:06.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.699+0000 7f7ac9ffb640 1 -- 172.21.15.33:0/3549893468 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab4043020 con 0x7f7acc102310 2024-03-19T02:51:06.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.699+0000 7f7ac9ffb640 1 -- 172.21.15.33:0/3549893468 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ab40044c0 con 0x7f7acc102310 2024-03-19T02:51:06.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.699+0000 7f7ad2dd4640 1 -- 172.21.15.33:0/3549893468 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7acc1a1b20 con 0x7f7acc102310 2024-03-19T02:51:06.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.699+0000 7f7ac9ffb640 1 -- 172.21.15.33:0/3549893468 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab4003bc0 con 0x7f7acc102310 2024-03-19T02:51:06.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.700+0000 7f7ad2dd4640 1 -- 172.21.15.33:0/3549893468 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7acc1a2070 con 0x7f7acc102310 2024-03-19T02:51:06.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.700+0000 7f7ad2dd4640 1 -- 172.21.15.33:0/3549893468 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7acc068a60 con 0x7f7acc102310 2024-03-19T02:51:06.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.702+0000 7f7ac9ffb640 1 -- 172.21.15.33:0/3549893468 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f7ab4046020 con 0x7f7acc102310 2024-03-19T02:51:06.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.702+0000 7f7ac9ffb640 1 --2- 172.21.15.33:0/3549893468 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7ab80a2800 0x7f7ab80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:06.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.702+0000 7f7ac9ffb640 1 -- 172.21.15.33:0/3549893468 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7f7ab40f1330 con 0x7f7acc102310 2024-03-19T02:51:06.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.702+0000 7f7ad0b70640 1 --2- 172.21.15.33:0/3549893468 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7ab80a2800 0x7f7ab80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:06.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.703+0000 7f7ad0b70640 1 --2- 172.21.15.33:0/3549893468 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7ab80a2800 0x7f7ab80a4cc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7ac4005fd0 tx=0x7f7ac400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:06.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.706+0000 7f7ac9ffb640 1 -- 172.21.15.33:0/3549893468 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7ab404d020 con 0x7f7acc102310 2024-03-19T02:51:06.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.889+0000 7f7ad2dd4640 1 -- 172.21.15.33:0/3549893468 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7acc1a2450 con 0x7f7acc102310 2024-03-19T02:51:06.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.890+0000 7f7ac9ffb640 1 -- 172.21.15.33:0/3549893468 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24043 (secure 0 0 0) 0x7f7ab40be190 con 0x7f7acc102310 2024-03-19T02:51:06.892 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:51:06.892 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":93,"fsid":"2b7a8938-e59a-11ee-95c9-87774f69a715","created":"2024-03-19T02:42:37.912179+0000","modified":"2024-03-19T02:50:59.241668+0000","last_up_change":"2024-03-19T02:50:01.539392+0000","last_in_change":"2024-03-19T02:49:40.534415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:46:09.963737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:50:50.069641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:50:51.381064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:50:55.476667+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"090a89ab-6b7b-4317-8823-178d4212b2f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6803","nonce":1633124995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6805","nonce":1633124995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6809","nonce":1633124995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6807","nonce":1633124995}]},"public_addr":"172.21.15.33:6803/1633124995","cluster_addr":"172.21.15.33:6805/1633124995","heartbeat_back_addr":"172.21.15.33:6809/1633124995","heartbeat_front_addr":"172.21.15.33:6807/1633124995","state":["exists","up"]},{"osd":1,"uuid":"f50eec55-1956-4486-8593-f2c8fa610e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6801","nonce":340144398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6803","nonce":340144398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6807","nonce":340144398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6805","nonce":340144398}]},"public_addr":"172.21.15.96:6801/340144398","cluster_addr":"172.21.15.96:6803/340144398","heartbeat_back_addr":"172.21.15.96:6807/340144398","heartbeat_front_addr":"172.21.15.96:6805/340144398","state":["exists","up"]},{"osd":2,"uuid":"893a2d62-80d9-4e5e-bc6a-a1c8c681d652","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6801","nonce":3388743876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6803","nonce":3388743876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6807","nonce":3388743876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6805","nonce":3388743876}]},"public_addr":"172.21.15.155:6801/3388743876","cluster_addr":"172.21.15.155:6803/3388743876","heartbeat_back_addr":"172.21.15.155:6807/3388743876","heartbeat_front_addr":"172.21.15.155:6805/3388743876","state":["exists","up"]},{"osd":3,"uuid":"b07e0482-4281-4d32-9dfe-9995d7246175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6811","nonce":677007171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6813","nonce":677007171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6817","nonce":677007171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6815","nonce":677007171}]},"public_addr":"172.21.15.33:6811/677007171","cluster_addr":"172.21.15.33:6813/677007171","heartbeat_back_addr":"172.21.15.33:6817/677007171","heartbeat_front_addr":"172.21.15.33:6815/677007171","state":["exists","up"]},{"osd":4,"uuid":"a988384b-af6b-47c6-8c07-5c9673acee45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6809","nonce":3106158554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6811","nonce":3106158554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6815","nonce":3106158554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6813","nonce":3106158554}]},"public_addr":"172.21.15.96:6809/3106158554","cluster_addr":"172.21.15.96:6811/3106158554","heartbeat_back_addr":"172.21.15.96:6815/3106158554","heartbeat_front_addr":"172.21.15.96:6813/3106158554","state":["exists","up"]},{"osd":5,"uuid":"198dc992-96f8-4c52-96ee-1fd5de148bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6809","nonce":327293324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6811","nonce":327293324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6815","nonce":327293324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6813","nonce":327293324}]},"public_addr":"172.21.15.155:6809/327293324","cluster_addr":"172.21.15.155:6811/327293324","heartbeat_back_addr":"172.21.15.155:6815/327293324","heartbeat_front_addr":"172.21.15.155:6813/327293324","state":["exists","up"]},{"osd":6,"uuid":"a7278ec0-ab2e-49ed-be53-84d8b9e24177","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6819","nonce":873871509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6821","nonce":873871509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6825","nonce":873871509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6823","nonce":873871509}]},"public_addr":"172.21.15.33:6819/873871509","cluster_addr":"172.21.15.33:6821/873871509","heartbeat_back_addr":"172.21.15.33:6825/873871509","heartbeat_front_addr":"172.21.15.33:6823/873871509","state":["exists","up"]},{"osd":7,"uuid":"727e133d-aa24-4ffc-90e2-266eb2aab8c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6817","nonce":2334123891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6819","nonce":2334123891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6823","nonce":2334123891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6821","nonce":2334123891}]},"public_addr":"172.21.15.96:6817/2334123891","cluster_addr":"172.21.15.96:6819/2334123891","heartbeat_back_addr":"172.21.15.96:6823/2334123891","heartbeat_front_addr":"172.21.15.96:6821/2334123891","state":["exists","up"]},{"osd":8,"uuid":"bb20eec0-39e5-4b86-9db5-486b1840cdad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6817","nonce":3498516903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6819","nonce":3498516903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6823","nonce":3498516903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6821","nonce":3498516903}]},"public_addr":"172.21.15.155:6817/3498516903","cluster_addr":"172.21.15.155:6819/3498516903","heartbeat_back_addr":"172.21.15.155:6823/3498516903","heartbeat_front_addr":"172.21.15.155:6821/3498516903","state":["exists","up"]},{"osd":9,"uuid":"179c6228-3e23-4f65-91a6-1ef0005d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6827","nonce":760777213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6829","nonce":760777213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6833","nonce":760777213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6831","nonce":760777213}]},"public_addr":"172.21.15.33:6827/760777213","cluster_addr":"172.21.15.33:6829/760777213","heartbeat_back_addr":"172.21.15.33:6833/760777213","heartbeat_front_addr":"172.21.15.33:6831/760777213","state":["exists","up"]},{"osd":10,"uuid":"be17b478-3c2e-41fb-bede-958333acdf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6825","nonce":1404898482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6827","nonce":1404898482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6831","nonce":1404898482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6829","nonce":1404898482}]},"public_addr":"172.21.15.96:6825/1404898482","cluster_addr":"172.21.15.96:6827/1404898482","heartbeat_back_addr":"172.21.15.96:6831/1404898482","heartbeat_front_addr":"172.21.15.96:6829/1404898482","state":["exists","up"]},{"osd":11,"uuid":"c0903ed2-cc1e-4c18-8731-1f0ee4dd0457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6825","nonce":792352827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6827","nonce":792352827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6831","nonce":792352827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6829","nonce":792352827}]},"public_addr":"172.21.15.155:6825/792352827","cluster_addr":"172.21.15.155:6827/792352827","heartbeat_back_addr":"172.21.15.155:6831/792352827","heartbeat_front_addr":"172.21.15.155:6829/792352827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:17.942801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:42.109449+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:05.981599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:32.069574+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:57.431015+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:22.448592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:48.526515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:14.498352+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:40.175426+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:06.364006+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:32.853161+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:59.073538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:6801/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:6800/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/1067768047":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/729379453":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2586588742":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/2932814165":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6801/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2212564963":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6800/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/3275126058":"2024-03-20T02:43:39.622218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:51:06.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.892+0000 7f7ad2dd4640 1 -- 172.21.15.33:0/3549893468 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7ab80a2800 msgr2=0x7f7ab80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:06.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.892+0000 7f7ad2dd4640 1 --2- 172.21.15.33:0/3549893468 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7ab80a2800 0x7f7ab80a4cc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7ac4005fd0 tx=0x7f7ac400b410 comp rx=0 tx=0).stop 2024-03-19T02:51:06.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.892+0000 7f7ad2dd4640 1 -- 172.21.15.33:0/3549893468 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7acc102310 msgr2=0x7f7acc103270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:06.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.892+0000 7f7ad2dd4640 1 --2- 172.21.15.33:0/3549893468 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7acc102310 0x7f7acc103270 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4009230 tx=0x7f7ab40040c0 comp rx=0 tx=0).stop 2024-03-19T02:51:06.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.892+0000 7f7ad2dd4640 1 -- 172.21.15.33:0/3549893468 shutdown_connections 2024-03-19T02:51:06.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.892+0000 7f7ad2dd4640 1 --2- 172.21.15.33:0/3549893468 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7ab80a2800 0x7f7ab80a4cc0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:06.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.892+0000 7f7ad2dd4640 1 --2- 172.21.15.33:0/3549893468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7acc101950 0x7f7acc102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:06.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.892+0000 7f7ad2dd4640 1 --2- 172.21.15.33:0/3549893468 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7acc102310 0x7f7acc103270 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:06.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.892+0000 7f7ad2dd4640 1 --2- 172.21.15.33:0/3549893468 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7acc100760 0x7f7acc1046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:06.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.892+0000 7f7ad2dd4640 1 -- 172.21.15.33:0/3549893468 >> 172.21.15.33:0/3549893468 conn(0x7f7acc0fbcd0 msgr2=0x7f7acc0fd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:06.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.892+0000 7f7ad2dd4640 1 -- 172.21.15.33:0/3549893468 shutdown_connections 2024-03-19T02:51:06.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:06.892+0000 7f7ad2dd4640 1 -- 172.21.15.33:0/3549893468 wait complete. 2024-03-19T02:51:06.904 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:46:09.963737+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T02:50:50.069641+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T02:50:51.381064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T02:50:55.476667+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T02:51:06.905 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T02:51:07.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.010+0000 7ff771261640 1 -- 172.21.15.33:0/1229408702 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff76c074280 msgr2=0x7ff76c074700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:07.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.010+0000 7ff771261640 1 --2- 172.21.15.33:0/1229408702 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff76c074280 0x7ff76c074700 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff754009900 tx=0x7ff75402bc30 comp rx=0 tx=0).stop 2024-03-19T02:51:07.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.010+0000 7ff771261640 1 -- 172.21.15.33:0/1229408702 shutdown_connections 2024-03-19T02:51:07.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.010+0000 7ff771261640 1 --2- 172.21.15.33:0/1229408702 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff76c1096f0 0x7ff76c10bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.010+0000 7ff771261640 1 --2- 172.21.15.33:0/1229408702 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff76c074280 0x7ff76c074700 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.010+0000 7ff771261640 1 --2- 172.21.15.33:0/1229408702 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff76c075860 0x7ff76c073cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.010+0000 7ff771261640 1 -- 172.21.15.33:0/1229408702 >> 172.21.15.33:0/1229408702 conn(0x7ff76c0fcdc0 msgr2=0x7ff76c0ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:07.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.010+0000 7ff771261640 1 -- 172.21.15.33:0/1229408702 shutdown_connections 2024-03-19T02:51:07.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.010+0000 7ff771261640 1 -- 172.21.15.33:0/1229408702 wait complete. 2024-03-19T02:51:07.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.010+0000 7ff771261640 1 Processor -- start 2024-03-19T02:51:07.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.011+0000 7ff771261640 1 -- start start 2024-03-19T02:51:07.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.011+0000 7ff771261640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff76c074280 0x7ff76c199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:07.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.011+0000 7ff771261640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff76c075860 0x7ff76c19a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:07.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.011+0000 7ff771261640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff76c1096f0 0x7ff76c1a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:07.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.011+0000 7ff771261640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff76c10f540 con 0x7ff76c074280 2024-03-19T02:51:07.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.011+0000 7ff771261640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff76c10f3c0 con 0x7ff76c075860 2024-03-19T02:51:07.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.011+0000 7ff771261640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff76c10f6c0 con 0x7ff76c1096f0 2024-03-19T02:51:07.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.011+0000 7ff76affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff76c074280 0x7ff76c199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:07.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.011+0000 7ff76affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff76c074280 0x7ff76c199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57348/0 (socket says 172.21.15.33:57348) 2024-03-19T02:51:07.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.011+0000 7ff76affd640 1 -- 172.21.15.33:0/3497139049 learned_addr learned my addr 172.21.15.33:0/3497139049 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:07.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.011+0000 7ff763fff640 1 --2- 172.21.15.33:0/3497139049 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff76c075860 0x7ff76c19a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:07.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.011+0000 7ff76b7fe640 1 --2- 172.21.15.33:0/3497139049 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff76c1096f0 0x7ff76c1a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:07.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.011+0000 7ff76affd640 1 -- 172.21.15.33:0/3497139049 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff76c1096f0 msgr2=0x7ff76c1a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:07.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.011+0000 7ff76affd640 1 --2- 172.21.15.33:0/3497139049 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff76c1096f0 0x7ff76c1a11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.011+0000 7ff76affd640 1 -- 172.21.15.33:0/3497139049 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff76c075860 msgr2=0x7ff76c19a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:07.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.011+0000 7ff76affd640 1 --2- 172.21.15.33:0/3497139049 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff76c075860 0x7ff76c19a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.011+0000 7ff76affd640 1 -- 172.21.15.33:0/3497139049 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff76c1a1890 con 0x7ff76c074280 2024-03-19T02:51:07.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.011+0000 7ff763fff640 1 --2- 172.21.15.33:0/3497139049 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff76c075860 0x7ff76c19a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:07.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.011+0000 7ff76affd640 1 --2- 172.21.15.33:0/3497139049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff76c074280 0x7ff76c199c80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff75c00e9f0 tx=0x7ff75c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:07.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.015+0000 7ff768ff9640 1 -- 172.21.15.33:0/3497139049 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff75c00cd80 con 0x7ff76c074280 2024-03-19T02:51:07.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.015+0000 7ff771261640 1 -- 172.21.15.33:0/3497139049 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff76c1a1b80 con 0x7ff76c074280 2024-03-19T02:51:07.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.015+0000 7ff768ff9640 1 -- 172.21.15.33:0/3497139049 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff75c0108e0 con 0x7ff76c074280 2024-03-19T02:51:07.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.015+0000 7ff768ff9640 1 -- 172.21.15.33:0/3497139049 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff75c018920 con 0x7ff76c074280 2024-03-19T02:51:07.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.015+0000 7ff771261640 1 -- 172.21.15.33:0/3497139049 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff76c1a20d0 con 0x7ff76c074280 2024-03-19T02:51:07.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.016+0000 7ff771261640 1 -- 172.21.15.33:0/3497139049 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff76c108420 con 0x7ff76c074280 2024-03-19T02:51:07.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.016+0000 7ff768ff9640 1 -- 172.21.15.33:0/3497139049 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff75c010a90 con 0x7ff76c074280 2024-03-19T02:51:07.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.017+0000 7ff768ff9640 1 --2- 172.21.15.33:0/3497139049 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff7580a2800 0x7ff7580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:07.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.017+0000 7ff763fff640 1 --2- 172.21.15.33:0/3497139049 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff7580a2800 0x7ff7580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:07.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.017+0000 7ff768ff9640 1 -- 172.21.15.33:0/3497139049 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7ff75c0cf020 con 0x7ff76c074280 2024-03-19T02:51:07.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.018+0000 7ff763fff640 1 --2- 172.21.15.33:0/3497139049 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff7580a2800 0x7ff7580a4cc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff754009230 tx=0x7ff754005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:07.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.022+0000 7ff768ff9640 1 -- 172.21.15.33:0/3497139049 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff75c09b830 con 0x7ff76c074280 2024-03-19T02:51:07.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.204+0000 7ff771261640 1 -- 172.21.15.33:0/3497139049 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff76c062450 con 0x7ff76c074280 2024-03-19T02:51:07.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.205+0000 7ff768ff9640 1 -- 172.21.15.33:0/3497139049 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7ff75c09f8d0 con 0x7ff76c074280 2024-03-19T02:51:07.206 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 1 2024-03-19T02:51:07.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.207+0000 7ff771261640 1 -- 172.21.15.33:0/3497139049 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff7580a2800 msgr2=0x7ff7580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:07.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.207+0000 7ff771261640 1 --2- 172.21.15.33:0/3497139049 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff7580a2800 0x7ff7580a4cc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff754009230 tx=0x7ff754005fd0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.207+0000 7ff771261640 1 -- 172.21.15.33:0/3497139049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff76c074280 msgr2=0x7ff76c199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:07.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.207+0000 7ff771261640 1 --2- 172.21.15.33:0/3497139049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff76c074280 0x7ff76c199c80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff75c00e9f0 tx=0x7ff75c00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.207+0000 7ff771261640 1 -- 172.21.15.33:0/3497139049 shutdown_connections 2024-03-19T02:51:07.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.207+0000 7ff771261640 1 --2- 172.21.15.33:0/3497139049 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff7580a2800 0x7ff7580a4cc0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.207+0000 7ff771261640 1 --2- 172.21.15.33:0/3497139049 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff76c1096f0 0x7ff76c1a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.207+0000 7ff771261640 1 --2- 172.21.15.33:0/3497139049 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff76c075860 0x7ff76c19a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.207+0000 7ff771261640 1 --2- 172.21.15.33:0/3497139049 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff76c074280 0x7ff76c199c80 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.207+0000 7ff771261640 1 -- 172.21.15.33:0/3497139049 >> 172.21.15.33:0/3497139049 conn(0x7ff76c0fcdc0 msgr2=0x7ff76c1055f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:07.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.207+0000 7ff771261640 1 -- 172.21.15.33:0/3497139049 shutdown_connections 2024-03-19T02:51:07.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.207+0000 7ff771261640 1 -- 172.21.15.33:0/3497139049 wait complete. 2024-03-19T02:51:07.217 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:07 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2469138355' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:51:07.217 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:07 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/913570873' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T02:51:07.217 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:07 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3549893468' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:51:07.218 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T02:51:07.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:07 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2469138355' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:51:07.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:07 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/913570873' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T02:51:07.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:07 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3549893468' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:51:07.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.322+0000 7fad5bd3b640 1 -- 172.21.15.33:0/2815322507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad54100730 msgr2=0x7fad54100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:07.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.322+0000 7fad5bd3b640 1 --2- 172.21.15.33:0/2815322507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad54100730 0x7fad54100b50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fad50009900 tx=0x7fad5002bc30 comp rx=0 tx=0).stop 2024-03-19T02:51:07.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.322+0000 7fad5bd3b640 1 -- 172.21.15.33:0/2815322507 shutdown_connections 2024-03-19T02:51:07.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.322+0000 7fad5bd3b640 1 --2- 172.21.15.33:0/2815322507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fad541022e0 0x7fad54108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.322+0000 7fad5bd3b640 1 --2- 172.21.15.33:0/2815322507 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad54101920 0x7fad54101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.323+0000 7fad5bd3b640 1 --2- 172.21.15.33:0/2815322507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad54100730 0x7fad54100b50 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.323+0000 7fad5bd3b640 1 -- 172.21.15.33:0/2815322507 >> 172.21.15.33:0/2815322507 conn(0x7fad540fbcc0 msgr2=0x7fad540fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:07.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.323+0000 7fad5bd3b640 1 -- 172.21.15.33:0/2815322507 shutdown_connections 2024-03-19T02:51:07.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.323+0000 7fad5bd3b640 1 -- 172.21.15.33:0/2815322507 wait complete. 2024-03-19T02:51:07.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.323+0000 7fad5bd3b640 1 Processor -- start 2024-03-19T02:51:07.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.323+0000 7fad5bd3b640 1 -- start start 2024-03-19T02:51:07.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.323+0000 7fad5bd3b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fad54100730 0x7fad54197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:07.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.323+0000 7fad5bd3b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad54101920 0x7fad54197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:07.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.323+0000 7fad5bd3b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad541022e0 0x7fad5419f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:07.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.323+0000 7fad5bd3b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fad5410d380 con 0x7fad541022e0 2024-03-19T02:51:07.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.323+0000 7fad5bd3b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fad5410d200 con 0x7fad54101920 2024-03-19T02:51:07.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.323+0000 7fad5bd3b640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fad5410d500 con 0x7fad54100730 2024-03-19T02:51:07.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.323+0000 7fad5a2d8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad541022e0 0x7fad5419f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:07.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.323+0000 7fad5a2d8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad541022e0 0x7fad5419f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57366/0 (socket says 172.21.15.33:57366) 2024-03-19T02:51:07.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.324+0000 7fad5a2d8640 1 -- 172.21.15.33:0/3640900550 learned_addr learned my addr 172.21.15.33:0/3640900550 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:07.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.324+0000 7fad592d6640 1 --2- 172.21.15.33:0/3640900550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad54101920 0x7fad54197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:07.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.324+0000 7fad59ad7640 1 --2- 172.21.15.33:0/3640900550 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fad54100730 0x7fad54197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:07.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.324+0000 7fad5a2d8640 1 -- 172.21.15.33:0/3640900550 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fad54100730 msgr2=0x7fad54197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:07.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.324+0000 7fad5a2d8640 1 --2- 172.21.15.33:0/3640900550 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fad54100730 0x7fad54197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.324+0000 7fad5a2d8640 1 -- 172.21.15.33:0/3640900550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad54101920 msgr2=0x7fad54197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:07.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.324+0000 7fad5a2d8640 1 --2- 172.21.15.33:0/3640900550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad54101920 0x7fad54197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.324+0000 7fad5a2d8640 1 -- 172.21.15.33:0/3640900550 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad5419f6d0 con 0x7fad541022e0 2024-03-19T02:51:07.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.324+0000 7fad5a2d8640 1 --2- 172.21.15.33:0/3640900550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad541022e0 0x7fad5419f060 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fad4c00eaa0 tx=0x7fad4c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:07.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.324+0000 7fad4affd640 1 -- 172.21.15.33:0/3640900550 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad4c00cdb0 con 0x7fad541022e0 2024-03-19T02:51:07.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.324+0000 7fad4affd640 1 -- 172.21.15.33:0/3640900550 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad4c005020 con 0x7fad541022e0 2024-03-19T02:51:07.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.324+0000 7fad5bd3b640 1 -- 172.21.15.33:0/3640900550 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad5419f9c0 con 0x7fad541022e0 2024-03-19T02:51:07.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.324+0000 7fad4affd640 1 -- 172.21.15.33:0/3640900550 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad4c0116d0 con 0x7fad541022e0 2024-03-19T02:51:07.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.324+0000 7fad5bd3b640 1 -- 172.21.15.33:0/3640900550 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad5419ff60 con 0x7fad541022e0 2024-03-19T02:51:07.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.325+0000 7fad5bd3b640 1 -- 172.21.15.33:0/3640900550 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad54107b10 con 0x7fad541022e0 2024-03-19T02:51:07.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.326+0000 7fad4affd640 1 -- 172.21.15.33:0/3640900550 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fad4c0051d0 con 0x7fad541022e0 2024-03-19T02:51:07.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.327+0000 7fad4affd640 1 --2- 172.21.15.33:0/3640900550 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fad400a2880 0x7fad400a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:07.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.327+0000 7fad59ad7640 1 --2- 172.21.15.33:0/3640900550 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fad400a2880 0x7fad400a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:07.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.327+0000 7fad4affd640 1 -- 172.21.15.33:0/3640900550 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7fad4c0cf1e0 con 0x7fad541022e0 2024-03-19T02:51:07.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.327+0000 7fad59ad7640 1 --2- 172.21.15.33:0/3640900550 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fad400a2880 0x7fad400a4d40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fad50036040 tx=0x7fad50000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:07.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.330+0000 7fad4affd640 1 -- 172.21.15.33:0/3640900550 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fad4c016030 con 0x7fad541022e0 2024-03-19T02:51:07.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:07 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2469138355' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:51:07.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:07 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/913570873' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T02:51:07.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:07 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3549893468' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:51:07.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.510+0000 7fad5bd3b640 1 -- 172.21.15.33:0/3640900550 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fad5402d1f0 con 0x7fad541022e0 2024-03-19T02:51:07.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.511+0000 7fad4affd640 1 -- 172.21.15.33:0/3640900550 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7fad4c09c060 con 0x7fad541022e0 2024-03-19T02:51:07.512 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 32 2024-03-19T02:51:07.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.513+0000 7fad5bd3b640 1 -- 172.21.15.33:0/3640900550 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fad400a2880 msgr2=0x7fad400a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:07.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.513+0000 7fad5bd3b640 1 --2- 172.21.15.33:0/3640900550 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fad400a2880 0x7fad400a4d40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fad50036040 tx=0x7fad50000bc0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.513+0000 7fad5bd3b640 1 -- 172.21.15.33:0/3640900550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad541022e0 msgr2=0x7fad5419f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:07.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.513+0000 7fad5bd3b640 1 --2- 172.21.15.33:0/3640900550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad541022e0 0x7fad5419f060 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fad4c00eaa0 tx=0x7fad4c00ef60 comp rx=0 tx=0).stop 2024-03-19T02:51:07.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.513+0000 7fad5bd3b640 1 -- 172.21.15.33:0/3640900550 shutdown_connections 2024-03-19T02:51:07.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.513+0000 7fad5bd3b640 1 --2- 172.21.15.33:0/3640900550 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fad400a2880 0x7fad400a4d40 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.513+0000 7fad5bd3b640 1 --2- 172.21.15.33:0/3640900550 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fad541022e0 0x7fad5419f060 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.513+0000 7fad5bd3b640 1 --2- 172.21.15.33:0/3640900550 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad54101920 0x7fad54197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.513+0000 7fad5bd3b640 1 --2- 172.21.15.33:0/3640900550 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fad54100730 0x7fad54197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.513+0000 7fad5bd3b640 1 -- 172.21.15.33:0/3640900550 >> 172.21.15.33:0/3640900550 conn(0x7fad540fbcc0 msgr2=0x7fad54104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:07.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.513+0000 7fad5bd3b640 1 -- 172.21.15.33:0/3640900550 shutdown_connections 2024-03-19T02:51:07.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.513+0000 7fad5bd3b640 1 -- 172.21.15.33:0/3640900550 wait complete. 2024-03-19T02:51:07.524 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T02:51:07.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.628+0000 7fd5ce90c640 1 -- 172.21.15.33:0/1531939069 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd5c81022d0 msgr2=0x7fd5c8108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:07.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.628+0000 7fd5ce90c640 1 --2- 172.21.15.33:0/1531939069 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd5c81022d0 0x7fd5c8108b60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd5c0009ca0 tx=0x7fd5c002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.628+0000 7fd5ce90c640 1 -- 172.21.15.33:0/1531939069 shutdown_connections 2024-03-19T02:51:07.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.628+0000 7fd5ce90c640 1 --2- 172.21.15.33:0/1531939069 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5c8101910 0x7fd5c8101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.628+0000 7fd5ce90c640 1 --2- 172.21.15.33:0/1531939069 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd5c81022d0 0x7fd5c8108b60 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.628+0000 7fd5ce90c640 1 --2- 172.21.15.33:0/1531939069 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5c8100720 0x7fd5c8100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.628+0000 7fd5ce90c640 1 -- 172.21.15.33:0/1531939069 >> 172.21.15.33:0/1531939069 conn(0x7fd5c80fbcb0 msgr2=0x7fd5c80fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:07.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.628+0000 7fd5ce90c640 1 -- 172.21.15.33:0/1531939069 shutdown_connections 2024-03-19T02:51:07.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.628+0000 7fd5ce90c640 1 -- 172.21.15.33:0/1531939069 wait complete. 2024-03-19T02:51:07.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.629+0000 7fd5ce90c640 1 Processor -- start 2024-03-19T02:51:07.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.629+0000 7fd5ce90c640 1 -- start start 2024-03-19T02:51:07.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.629+0000 7fd5ce90c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5c8100720 0x7fd5c8199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:07.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.629+0000 7fd5ce90c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd5c8101910 0x7fd5c819a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:07.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.629+0000 7fd5ce90c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5c81022d0 0x7fd5c81a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:07.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.629+0000 7fd5ce90c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c810c250 con 0x7fd5c8101910 2024-03-19T02:51:07.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.629+0000 7fd5ce90c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c810c0d0 con 0x7fd5c81022d0 2024-03-19T02:51:07.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.629+0000 7fd5ce90c640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c810c3d0 con 0x7fd5c8100720 2024-03-19T02:51:07.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.629+0000 7fd5c77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd5c8101910 0x7fd5c819a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:07.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.629+0000 7fd5c77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd5c8101910 0x7fd5c819a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57386/0 (socket says 172.21.15.33:57386) 2024-03-19T02:51:07.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.629+0000 7fd5c77fe640 1 -- 172.21.15.33:0/2893150343 learned_addr learned my addr 172.21.15.33:0/2893150343 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:07.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.629+0000 7fd5ccea9640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5c81022d0 0x7fd5c81a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:07.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.630+0000 7fd5c7fff640 1 --2- 172.21.15.33:0/2893150343 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5c8100720 0x7fd5c8199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:07.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.630+0000 7fd5ccea9640 1 -- 172.21.15.33:0/2893150343 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5c8100720 msgr2=0x7fd5c8199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:07.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.630+0000 7fd5ccea9640 1 --2- 172.21.15.33:0/2893150343 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5c8100720 0x7fd5c8199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.630+0000 7fd5ccea9640 1 -- 172.21.15.33:0/2893150343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd5c8101910 msgr2=0x7fd5c819a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:07.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.630+0000 7fd5ccea9640 1 --2- 172.21.15.33:0/2893150343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd5c8101910 0x7fd5c819a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.630+0000 7fd5ccea9640 1 -- 172.21.15.33:0/2893150343 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5c81a1880 con 0x7fd5c81022d0 2024-03-19T02:51:07.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.630+0000 7fd5c77fe640 1 --2- 172.21.15.33:0/2893150343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd5c8101910 0x7fd5c819a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:07.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.630+0000 7fd5ccea9640 1 --2- 172.21.15.33:0/2893150343 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5c81022d0 0x7fd5c81a11e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd5c002c630 tx=0x7fd5c00328a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:07.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.630+0000 7fd5c57fa640 1 -- 172.21.15.33:0/2893150343 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c0041020 con 0x7fd5c81022d0 2024-03-19T02:51:07.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.630+0000 7fd5ce90c640 1 -- 172.21.15.33:0/2893150343 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5c81a1b10 con 0x7fd5c81022d0 2024-03-19T02:51:07.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.630+0000 7fd5c57fa640 1 -- 172.21.15.33:0/2893150343 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5c0003e20 con 0x7fd5c81022d0 2024-03-19T02:51:07.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.630+0000 7fd5c57fa640 1 -- 172.21.15.33:0/2893150343 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c0039040 con 0x7fd5c81022d0 2024-03-19T02:51:07.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.630+0000 7fd5ce90c640 1 -- 172.21.15.33:0/2893150343 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5c81a2050 con 0x7fd5c81022d0 2024-03-19T02:51:07.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.631+0000 7fd5ce90c640 1 -- 172.21.15.33:0/2893150343 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5c81a2220 con 0x7fd5c81022d0 2024-03-19T02:51:07.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.632+0000 7fd5c57fa640 1 -- 172.21.15.33:0/2893150343 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fd5c0044020 con 0x7fd5c81022d0 2024-03-19T02:51:07.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.633+0000 7fd5c57fa640 1 --2- 172.21.15.33:0/2893150343 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fd5b00a2800 0x7fd5b00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:07.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.633+0000 7fd5c7fff640 1 --2- 172.21.15.33:0/2893150343 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fd5b00a2800 0x7fd5b00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:07.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.633+0000 7fd5c57fa640 1 -- 172.21.15.33:0/2893150343 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7fd5c00f1310 con 0x7fd5c81022d0 2024-03-19T02:51:07.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.633+0000 7fd5c7fff640 1 --2- 172.21.15.33:0/2893150343 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fd5b00a2800 0x7fd5b00a4cc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd5c819b140 tx=0x7fd5b8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:07.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.638+0000 7fd5c57fa640 1 -- 172.21.15.33:0/2893150343 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd5c004b020 con 0x7fd5c81022d0 2024-03-19T02:51:07.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.823+0000 7fd5ce90c640 1 -- 172.21.15.33:0/2893150343 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd5c81a23f0 con 0x7fd5c81022d0 2024-03-19T02:51:07.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.824+0000 7fd5c57fa640 1 -- 172.21.15.33:0/2893150343 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7fd5c00be170 con 0x7fd5c81022d0 2024-03-19T02:51:07.825 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 64 2024-03-19T02:51:07.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.826+0000 7fd5ce90c640 1 -- 172.21.15.33:0/2893150343 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fd5b00a2800 msgr2=0x7fd5b00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:07.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.826+0000 7fd5ce90c640 1 --2- 172.21.15.33:0/2893150343 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fd5b00a2800 0x7fd5b00a4cc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd5c819b140 tx=0x7fd5b8005f90 comp rx=0 tx=0).stop 2024-03-19T02:51:07.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.826+0000 7fd5ce90c640 1 -- 172.21.15.33:0/2893150343 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5c81022d0 msgr2=0x7fd5c81a11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:07.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.826+0000 7fd5ce90c640 1 --2- 172.21.15.33:0/2893150343 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5c81022d0 0x7fd5c81a11e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd5c002c630 tx=0x7fd5c00328a0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.826+0000 7fd5ce90c640 1 -- 172.21.15.33:0/2893150343 shutdown_connections 2024-03-19T02:51:07.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.826+0000 7fd5ce90c640 1 --2- 172.21.15.33:0/2893150343 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fd5b00a2800 0x7fd5b00a4cc0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.826+0000 7fd5ce90c640 1 --2- 172.21.15.33:0/2893150343 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5c81022d0 0x7fd5c81a11e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.826+0000 7fd5ce90c640 1 --2- 172.21.15.33:0/2893150343 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd5c8101910 0x7fd5c819a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.826+0000 7fd5ce90c640 1 --2- 172.21.15.33:0/2893150343 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5c8100720 0x7fd5c8199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.826+0000 7fd5ce90c640 1 -- 172.21.15.33:0/2893150343 >> 172.21.15.33:0/2893150343 conn(0x7fd5c80fbcb0 msgr2=0x7fd5c80fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:07.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.826+0000 7fd5ce90c640 1 -- 172.21.15.33:0/2893150343 shutdown_connections 2024-03-19T02:51:07.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.826+0000 7fd5ce90c640 1 -- 172.21.15.33:0/2893150343 wait complete. 2024-03-19T02:51:07.838 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T02:51:07.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.942+0000 7fc622a9a640 1 -- 172.21.15.33:0/1395159913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc61c101930 msgr2=0x7fc61c101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:07.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.942+0000 7fc622a9a640 1 --2- 172.21.15.33:0/1395159913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc61c101930 0x7fc61c101db0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc608009900 tx=0x7fc60802bc30 comp rx=0 tx=0).stop 2024-03-19T02:51:07.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.942+0000 7fc622a9a640 1 -- 172.21.15.33:0/1395159913 shutdown_connections 2024-03-19T02:51:07.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.942+0000 7fc622a9a640 1 --2- 172.21.15.33:0/1395159913 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc61c1022f0 0x7fc61c108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.942+0000 7fc622a9a640 1 --2- 172.21.15.33:0/1395159913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc61c101930 0x7fc61c101db0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.942+0000 7fc622a9a640 1 --2- 172.21.15.33:0/1395159913 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc61c100740 0x7fc61c100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.942+0000 7fc622a9a640 1 -- 172.21.15.33:0/1395159913 >> 172.21.15.33:0/1395159913 conn(0x7fc61c0fbcd0 msgr2=0x7fc61c0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:07.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.942+0000 7fc622a9a640 1 -- 172.21.15.33:0/1395159913 shutdown_connections 2024-03-19T02:51:07.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.942+0000 7fc622a9a640 1 -- 172.21.15.33:0/1395159913 wait complete. 2024-03-19T02:51:07.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.942+0000 7fc622a9a640 1 Processor -- start 2024-03-19T02:51:07.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.942+0000 7fc622a9a640 1 -- start start 2024-03-19T02:51:07.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.943+0000 7fc622a9a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc61c100740 0x7fc61c197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:07.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.943+0000 7fc622a9a640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc61c101930 0x7fc61c197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:07.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.943+0000 7fc622a9a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc61c1022f0 0x7fc61c19f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:07.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.943+0000 7fc622a9a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c10c250 con 0x7fc61c100740 2024-03-19T02:51:07.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.943+0000 7fc622a9a640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c10c0d0 con 0x7fc61c1022f0 2024-03-19T02:51:07.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.943+0000 7fc622a9a640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c10c3d0 con 0x7fc61c101930 2024-03-19T02:51:07.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.943+0000 7fc620836640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc61c100740 0x7fc61c197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:07.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.943+0000 7fc620836640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc61c100740 0x7fc61c197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57404/0 (socket says 172.21.15.33:57404) 2024-03-19T02:51:07.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.943+0000 7fc620836640 1 -- 172.21.15.33:0/3289253210 learned_addr learned my addr 172.21.15.33:0/3289253210 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:07.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.943+0000 7fc621037640 1 --2- 172.21.15.33:0/3289253210 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc61c1022f0 0x7fc61c19f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:07.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.943+0000 7fc61bfff640 1 --2- 172.21.15.33:0/3289253210 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc61c101930 0x7fc61c197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:07.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.943+0000 7fc620836640 1 -- 172.21.15.33:0/3289253210 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc61c101930 msgr2=0x7fc61c197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:07.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.943+0000 7fc620836640 1 --2- 172.21.15.33:0/3289253210 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc61c101930 0x7fc61c197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.943+0000 7fc620836640 1 -- 172.21.15.33:0/3289253210 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc61c1022f0 msgr2=0x7fc61c19f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:07.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.943+0000 7fc620836640 1 --2- 172.21.15.33:0/3289253210 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc61c1022f0 0x7fc61c19f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:07.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.943+0000 7fc620836640 1 -- 172.21.15.33:0/3289253210 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc61c19f6d0 con 0x7fc61c100740 2024-03-19T02:51:07.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.943+0000 7fc61bfff640 1 --2- 172.21.15.33:0/3289253210 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc61c101930 0x7fc61c197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:51:07.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.943+0000 7fc620836640 1 --2- 172.21.15.33:0/3289253210 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc61c100740 0x7fc61c197ab0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc61000e9c0 tx=0x7fc61000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:07.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.943+0000 7fc619ffb640 1 -- 172.21.15.33:0/3289253210 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc61000cd80 con 0x7fc61c100740 2024-03-19T02:51:07.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.943+0000 7fc619ffb640 1 -- 172.21.15.33:0/3289253210 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc6100035f0 con 0x7fc61c100740 2024-03-19T02:51:07.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.943+0000 7fc622a9a640 1 -- 172.21.15.33:0/3289253210 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc61c19f9c0 con 0x7fc61c100740 2024-03-19T02:51:07.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.943+0000 7fc619ffb640 1 -- 172.21.15.33:0/3289253210 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc610010690 con 0x7fc61c100740 2024-03-19T02:51:07.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.944+0000 7fc622a9a640 1 -- 172.21.15.33:0/3289253210 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc61c19ff10 con 0x7fc61c100740 2024-03-19T02:51:07.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.945+0000 7fc622a9a640 1 -- 172.21.15.33:0/3289253210 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc61c1a01a0 con 0x7fc61c100740 2024-03-19T02:51:07.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.945+0000 7fc619ffb640 1 -- 172.21.15.33:0/3289253210 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc6100037a0 con 0x7fc61c100740 2024-03-19T02:51:07.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.946+0000 7fc619ffb640 1 --2- 172.21.15.33:0/3289253210 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc60c0a2800 0x7fc60c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:07.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.946+0000 7fc61bfff640 1 --2- 172.21.15.33:0/3289253210 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc60c0a2800 0x7fc60c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:07.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.946+0000 7fc619ffb640 1 -- 172.21.15.33:0/3289253210 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7fc6100cf140 con 0x7fc61c100740 2024-03-19T02:51:07.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.947+0000 7fc61bfff640 1 --2- 172.21.15.33:0/3289253210 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc60c0a2800 0x7fc60c0a4cc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc61c198fc0 tx=0x7fc60802c7e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:07.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:07.951+0000 7fc619ffb640 1 -- 172.21.15.33:0/3289253210 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc61009bf40 con 0x7fc61c100740 2024-03-19T02:51:08.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.140+0000 7fc622a9a640 1 -- 172.21.15.33:0/3289253210 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc61c1a0370 con 0x7fc61c100740 2024-03-19T02:51:08.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.140+0000 7fc619ffb640 1 -- 172.21.15.33:0/3289253210 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fc61009b840 con 0x7fc61c100740 2024-03-19T02:51:08.141 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 32 2024-03-19T02:51:08.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.142+0000 7fc622a9a640 1 -- 172.21.15.33:0/3289253210 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc60c0a2800 msgr2=0x7fc60c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:08.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.142+0000 7fc622a9a640 1 --2- 172.21.15.33:0/3289253210 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc60c0a2800 0x7fc60c0a4cc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc61c198fc0 tx=0x7fc60802c7e0 comp rx=0 tx=0).stop 2024-03-19T02:51:08.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.142+0000 7fc622a9a640 1 -- 172.21.15.33:0/3289253210 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc61c100740 msgr2=0x7fc61c197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:08.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.142+0000 7fc622a9a640 1 --2- 172.21.15.33:0/3289253210 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc61c100740 0x7fc61c197ab0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc61000e9c0 tx=0x7fc61000ee80 comp rx=0 tx=0).stop 2024-03-19T02:51:08.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.142+0000 7fc622a9a640 1 -- 172.21.15.33:0/3289253210 shutdown_connections 2024-03-19T02:51:08.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.142+0000 7fc622a9a640 1 --2- 172.21.15.33:0/3289253210 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc60c0a2800 0x7fc60c0a4cc0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:08.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.142+0000 7fc622a9a640 1 --2- 172.21.15.33:0/3289253210 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc61c101930 0x7fc61c197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:08.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.142+0000 7fc622a9a640 1 --2- 172.21.15.33:0/3289253210 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc61c1022f0 0x7fc61c19f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:08.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.142+0000 7fc622a9a640 1 --2- 172.21.15.33:0/3289253210 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc61c100740 0x7fc61c197ab0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:08.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.142+0000 7fc622a9a640 1 -- 172.21.15.33:0/3289253210 >> 172.21.15.33:0/3289253210 conn(0x7fc61c0fbcd0 msgr2=0x7fc61c069de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:08.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.142+0000 7fc622a9a640 1 -- 172.21.15.33:0/3289253210 shutdown_connections 2024-03-19T02:51:08.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.142+0000 7fc622a9a640 1 -- 172.21.15.33:0/3289253210 wait complete. 2024-03-19T02:51:08.153 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:08 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3497139049' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:51:08.153 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:08 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3640900550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:51:08.153 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:08 smithi033 ceph-mon[31973]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 324 B/s wr, 0 op/s 2024-03-19T02:51:08.153 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:08 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2893150343' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:51:08.154 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-19T02:51:08.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.258+0000 7f62c9110640 1 -- 172.21.15.33:0/569351839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62c41034a0 msgr2=0x7f62c4109d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:08.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.258+0000 7f62c9110640 1 --2- 172.21.15.33:0/569351839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62c41034a0 0x7f62c4109d30 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f62bc009cd0 tx=0x7f62bc02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:51:08.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.258+0000 7f62c9110640 1 -- 172.21.15.33:0/569351839 shutdown_connections 2024-03-19T02:51:08.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.258+0000 7f62c9110640 1 --2- 172.21.15.33:0/569351839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62c41034a0 0x7f62c4109d30 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:08.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.258+0000 7f62c9110640 1 --2- 172.21.15.33:0/569351839 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62c4102ae0 0x7f62c4102f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:08.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.258+0000 7f62c9110640 1 --2- 172.21.15.33:0/569351839 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62c40ffb10 0x7f62c40fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:08.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.258+0000 7f62c9110640 1 -- 172.21.15.33:0/569351839 >> 172.21.15.33:0/569351839 conn(0x7f62c40fb420 msgr2=0x7f62c40fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:08.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.258+0000 7f62c9110640 1 -- 172.21.15.33:0/569351839 shutdown_connections 2024-03-19T02:51:08.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.259+0000 7f62c9110640 1 -- 172.21.15.33:0/569351839 wait complete. 2024-03-19T02:51:08.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.259+0000 7f62c9110640 1 Processor -- start 2024-03-19T02:51:08.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.259+0000 7f62c9110640 1 -- start start 2024-03-19T02:51:08.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.259+0000 7f62c9110640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62c40ffb10 0x7f62c4197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:08.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.259+0000 7f62c9110640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62c4102ae0 0x7f62c4197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:08.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.259+0000 7f62c9110640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62c41034a0 0x7f62c419f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:08.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.259+0000 7f62c9110640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f62c410d390 con 0x7f62c4102ae0 2024-03-19T02:51:08.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.259+0000 7f62c9110640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f62c410d210 con 0x7f62c40ffb10 2024-03-19T02:51:08.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.259+0000 7f62c9110640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f62c410d510 con 0x7f62c41034a0 2024-03-19T02:51:08.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.259+0000 7f62bbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62c4102ae0 0x7f62c4197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:08.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.259+0000 7f62bbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62c4102ae0 0x7f62c4197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57416/0 (socket says 172.21.15.33:57416) 2024-03-19T02:51:08.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.259+0000 7f62bbfff640 1 -- 172.21.15.33:0/4138330917 learned_addr learned my addr 172.21.15.33:0/4138330917 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:08.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.260+0000 7f62c2d9d640 1 --2- 172.21.15.33:0/4138330917 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62c40ffb10 0x7f62c4197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:08.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.260+0000 7f62c359e640 1 --2- 172.21.15.33:0/4138330917 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62c41034a0 0x7f62c419f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:08.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.260+0000 7f62bbfff640 1 -- 172.21.15.33:0/4138330917 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62c41034a0 msgr2=0x7f62c419f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:08.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.260+0000 7f62bbfff640 1 --2- 172.21.15.33:0/4138330917 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62c41034a0 0x7f62c419f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:08.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.260+0000 7f62bbfff640 1 -- 172.21.15.33:0/4138330917 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62c40ffb10 msgr2=0x7f62c4197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:08.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.260+0000 7f62bbfff640 1 --2- 172.21.15.33:0/4138330917 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62c40ffb10 0x7f62c4197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:08.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.260+0000 7f62bbfff640 1 -- 172.21.15.33:0/4138330917 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62c419f680 con 0x7f62c4102ae0 2024-03-19T02:51:08.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.260+0000 7f62c2d9d640 1 --2- 172.21.15.33:0/4138330917 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62c40ffb10 0x7f62c4197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:08.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.260+0000 7f62bbfff640 1 --2- 172.21.15.33:0/4138330917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62c4102ae0 0x7f62c4197fa0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f62ac00ea60 tx=0x7f62ac00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:08.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.260+0000 7f62c0d99640 1 -- 172.21.15.33:0/4138330917 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ac00cde0 con 0x7f62c4102ae0 2024-03-19T02:51:08.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.260+0000 7f62c0d99640 1 -- 172.21.15.33:0/4138330917 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62ac0035f0 con 0x7f62c4102ae0 2024-03-19T02:51:08.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.260+0000 7f62c9110640 1 -- 172.21.15.33:0/4138330917 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62c419f970 con 0x7f62c4102ae0 2024-03-19T02:51:08.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.260+0000 7f62c0d99640 1 -- 172.21.15.33:0/4138330917 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ac010690 con 0x7f62c4102ae0 2024-03-19T02:51:08.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.260+0000 7f62c9110640 1 -- 172.21.15.33:0/4138330917 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62c419fec0 con 0x7f62c4102ae0 2024-03-19T02:51:08.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.261+0000 7f62c9110640 1 -- 172.21.15.33:0/4138330917 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62c4108d30 con 0x7f62c4102ae0 2024-03-19T02:51:08.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.262+0000 7f62c0d99640 1 -- 172.21.15.33:0/4138330917 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f62ac0037a0 con 0x7f62c4102ae0 2024-03-19T02:51:08.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.263+0000 7f62c0d99640 1 --2- 172.21.15.33:0/4138330917 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f62b00a2800 0x7f62b00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:08.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.263+0000 7f62c0d99640 1 -- 172.21.15.33:0/4138330917 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7f62ac0cf210 con 0x7f62c4102ae0 2024-03-19T02:51:08.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.263+0000 7f62c2d9d640 1 --2- 172.21.15.33:0/4138330917 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f62b00a2800 0x7f62b00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:08.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.263+0000 7f62c2d9d640 1 --2- 172.21.15.33:0/4138330917 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f62b00a2800 0x7f62b00a4cc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f62b4005fd0 tx=0x7f62b400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:08.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.266+0000 7f62c0d99640 1 -- 172.21.15.33:0/4138330917 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f62ac053740 con 0x7f62c4102ae0 2024-03-19T02:51:08.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.461+0000 7f62c9110640 1 -- 172.21.15.33:0/4138330917 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f62c402d1f0 con 0x7f62c4102ae0 2024-03-19T02:51:08.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.461+0000 7f62c0d99640 1 -- 172.21.15.33:0/4138330917 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v93) ==== 81+0+589 (secure 0 0 0) 0x7f62c402d1f0 con 0x7f62c4102ae0 2024-03-19T02:51:08.462 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:6801/906645748 2024-03-20T02:43:39.622218+0000 2024-03-19T02:51:08.463 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:6800/906645748 2024-03-20T02:43:39.622218+0000 2024-03-19T02:51:08.463 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/1067768047 2024-03-20T02:43:39.622218+0000 2024-03-19T02:51:08.463 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/729379453 2024-03-20T02:43:11.918076+0000 2024-03-19T02:51:08.463 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/2586588742 2024-03-20T02:43:39.622218+0000 2024-03-19T02:51:08.463 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/2932814165 2024-03-20T02:43:11.918076+0000 2024-03-19T02:51:08.463 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:6801/3342667183 2024-03-20T02:43:11.918076+0000 2024-03-19T02:51:08.463 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/2212564963 2024-03-20T02:43:11.918076+0000 2024-03-19T02:51:08.463 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:6800/3342667183 2024-03-20T02:43:11.918076+0000 2024-03-19T02:51:08.463 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/3275126058 2024-03-20T02:43:39.622218+0000 2024-03-19T02:51:08.463 INFO:teuthology.orchestra.run.smithi033.stderr:listed 10 entries 2024-03-19T02:51:08.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.463+0000 7f62c9110640 1 -- 172.21.15.33:0/4138330917 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f62b00a2800 msgr2=0x7f62b00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:08.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.463+0000 7f62c9110640 1 --2- 172.21.15.33:0/4138330917 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f62b00a2800 0x7f62b00a4cc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f62b4005fd0 tx=0x7f62b400b390 comp rx=0 tx=0).stop 2024-03-19T02:51:08.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.463+0000 7f62c9110640 1 -- 172.21.15.33:0/4138330917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62c4102ae0 msgr2=0x7f62c4197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:08.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.463+0000 7f62c9110640 1 --2- 172.21.15.33:0/4138330917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62c4102ae0 0x7f62c4197fa0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f62ac00ea60 tx=0x7f62ac00ef20 comp rx=0 tx=0).stop 2024-03-19T02:51:08.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.463+0000 7f62c9110640 1 -- 172.21.15.33:0/4138330917 shutdown_connections 2024-03-19T02:51:08.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.463+0000 7f62c9110640 1 --2- 172.21.15.33:0/4138330917 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f62b00a2800 0x7f62b00a4cc0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:08.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.463+0000 7f62c9110640 1 --2- 172.21.15.33:0/4138330917 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62c41034a0 0x7f62c419f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:08.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.463+0000 7f62c9110640 1 --2- 172.21.15.33:0/4138330917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f62c4102ae0 0x7f62c4197fa0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:08.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.463+0000 7f62c9110640 1 --2- 172.21.15.33:0/4138330917 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62c40ffb10 0x7f62c4197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:08.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.463+0000 7f62c9110640 1 -- 172.21.15.33:0/4138330917 >> 172.21.15.33:0/4138330917 conn(0x7f62c40fb420 msgr2=0x7f62c4105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:08.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.463+0000 7f62c9110640 1 -- 172.21.15.33:0/4138330917 shutdown_connections 2024-03-19T02:51:08.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:08.463+0000 7f62c9110640 1 -- 172.21.15.33:0/4138330917 wait complete. 2024-03-19T02:51:08.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:08 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3497139049' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:51:08.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:08 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3640900550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:51:08.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:08 smithi096 ceph-mon[35946]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 324 B/s wr, 0 op/s 2024-03-19T02:51:08.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:08 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2893150343' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:51:08.475 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi033.front.sepia.ceph.com... 2024-03-19T02:51:08.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:51:08.475 DEBUG:teuthology.orchestra.run.smithi033:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-19T02:51:08.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi033.stderr:ceph-fuse[76045]: fuse finished with error 0 and tester_r 0 2024-03-19T02:51:08.502 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T02:51:08.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:08 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3497139049' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:51:08.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:08 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3640900550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:51:08.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:08 smithi155 ceph-mon[35804]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 324 B/s wr, 0 op/s 2024-03-19T02:51:08.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:08 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2893150343' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:51:09.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:09 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3289253210' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:51:09.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:09 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/4138330917' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T02:51:09.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:09 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3289253210' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:51:09.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:09 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/4138330917' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T02:51:09.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:09 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3289253210' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:51:09.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:09 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/4138330917' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T02:51:10.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:10 smithi096 ceph-mon[35946]: pgmap v308: 129 pgs: 129 active+clean; 581 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 489 B/s wr, 0 op/s 2024-03-19T02:51:10.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:10 smithi155 ceph-mon[35804]: pgmap v308: 129 pgs: 129 active+clean; 581 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 489 B/s wr, 0 op/s 2024-03-19T02:51:10.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:10 smithi033 ceph-mon[31973]: pgmap v308: 129 pgs: 129 active+clean; 581 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 489 B/s wr, 0 op/s 2024-03-19T02:51:12.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:11 smithi096 ceph-mon[35946]: pgmap v309: 129 pgs: 129 active+clean; 581 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-19T02:51:12.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:11 smithi155 ceph-mon[35804]: pgmap v309: 129 pgs: 129 active+clean; 581 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-19T02:51:12.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:11 smithi033 ceph-mon[31973]: pgmap v309: 129 pgs: 129 active+clean; 581 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-19T02:51:14.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:13 smithi096 ceph-mon[35946]: pgmap v310: 129 pgs: 129 active+clean; 581 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-19T02:51:14.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:13 smithi155 ceph-mon[35804]: pgmap v310: 129 pgs: 129 active+clean; 581 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-19T02:51:14.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:13 smithi033 ceph-mon[31973]: pgmap v310: 129 pgs: 129 active+clean; 581 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-19T02:51:14.604 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi033.front.sepia.ceph.com 2024-03-19T02:51:14.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:51:14.604 DEBUG:teuthology.orchestra.run.smithi033:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-19T02:51:14.635 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:51:14.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.792+0000 7f44f1f59640 1 -- 172.21.15.33:0/1124244578 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44ec0757e0 msgr2=0x7f44ec073c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:14.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.792+0000 7f44f1f59640 1 --2- 172.21.15.33:0/1124244578 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44ec0757e0 0x7f44ec073c30 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f44e0009900 tx=0x7f44e002bcd0 comp rx=0 tx=0).stop 2024-03-19T02:51:14.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.792+0000 7f44f1f59640 1 -- 172.21.15.33:0/1124244578 shutdown_connections 2024-03-19T02:51:14.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.792+0000 7f44f1f59640 1 --2- 172.21.15.33:0/1124244578 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44ec1096e0 0x7f44ec10bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:14.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.792+0000 7f44f1f59640 1 --2- 172.21.15.33:0/1124244578 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44ec074200 0x7f44ec074660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:14.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.792+0000 7f44f1f59640 1 --2- 172.21.15.33:0/1124244578 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44ec0757e0 0x7f44ec073c30 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:14.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.792+0000 7f44f1f59640 1 -- 172.21.15.33:0/1124244578 >> 172.21.15.33:0/1124244578 conn(0x7f44ec0fce00 msgr2=0x7f44ec0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:14.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.792+0000 7f44f1f59640 1 -- 172.21.15.33:0/1124244578 shutdown_connections 2024-03-19T02:51:14.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.792+0000 7f44f1f59640 1 -- 172.21.15.33:0/1124244578 wait complete. 2024-03-19T02:51:14.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.793+0000 7f44f1f59640 1 Processor -- start 2024-03-19T02:51:14.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.793+0000 7f44f1f59640 1 -- start start 2024-03-19T02:51:14.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.793+0000 7f44f1f59640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44ec074200 0x7f44ec199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:14.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.793+0000 7f44f1f59640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44ec0757e0 0x7f44ec19a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:14.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.793+0000 7f44f1f59640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44ec1096e0 0x7f44ec1a1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:14.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.793+0000 7f44f1f59640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f44ec10f560 con 0x7f44ec1096e0 2024-03-19T02:51:14.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.793+0000 7f44f1f59640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f44ec10f3e0 con 0x7f44ec074200 2024-03-19T02:51:14.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.793+0000 7f44f1f59640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f44ec10f6e0 con 0x7f44ec0757e0 2024-03-19T02:51:14.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.793+0000 7f44ebfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44ec1096e0 0x7f44ec1a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:14.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.793+0000 7f44ebfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44ec1096e0 0x7f44ec1a1220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49894/0 (socket says 172.21.15.33:49894) 2024-03-19T02:51:14.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.793+0000 7f44ebfff640 1 -- 172.21.15.33:0/2736002883 learned_addr learned my addr 172.21.15.33:0/2736002883 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:14.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.793+0000 7f44eb7fe640 1 --2- 172.21.15.33:0/2736002883 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44ec074200 0x7f44ec199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:14.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.793+0000 7f44eaffd640 1 --2- 172.21.15.33:0/2736002883 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44ec0757e0 0x7f44ec19a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:14.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.794+0000 7f44ebfff640 1 -- 172.21.15.33:0/2736002883 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44ec0757e0 msgr2=0x7f44ec19a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:14.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.794+0000 7f44ebfff640 1 --2- 172.21.15.33:0/2736002883 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44ec0757e0 0x7f44ec19a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:14.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.794+0000 7f44ebfff640 1 -- 172.21.15.33:0/2736002883 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44ec074200 msgr2=0x7f44ec199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:14.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.794+0000 7f44ebfff640 1 --2- 172.21.15.33:0/2736002883 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44ec074200 0x7f44ec199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:14.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.794+0000 7f44ebfff640 1 -- 172.21.15.33:0/2736002883 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44ec1a18c0 con 0x7f44ec1096e0 2024-03-19T02:51:14.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.794+0000 7f44ebfff640 1 --2- 172.21.15.33:0/2736002883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44ec1096e0 0x7f44ec1a1220 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f44e400cad0 tx=0x7f44e400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:14.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.794+0000 7f44e8ff9640 1 -- 172.21.15.33:0/2736002883 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e400ae20 con 0x7f44ec1096e0 2024-03-19T02:51:14.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.794+0000 7f44f1f59640 1 -- 172.21.15.33:0/2736002883 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44ec1a1bb0 con 0x7f44ec1096e0 2024-03-19T02:51:14.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.794+0000 7f44e8ff9640 1 -- 172.21.15.33:0/2736002883 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f44e4005020 con 0x7f44ec1096e0 2024-03-19T02:51:14.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.794+0000 7f44e8ff9640 1 -- 172.21.15.33:0/2736002883 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e400f6b0 con 0x7f44ec1096e0 2024-03-19T02:51:14.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.794+0000 7f44f1f59640 1 -- 172.21.15.33:0/2736002883 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44ec1a2100 con 0x7f44ec1096e0 2024-03-19T02:51:14.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.795+0000 7f44f1f59640 1 -- 172.21.15.33:0/2736002883 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44ec1084a0 con 0x7f44ec1096e0 2024-03-19T02:51:14.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.796+0000 7f44e8ff9640 1 -- 172.21.15.33:0/2736002883 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f44e40051d0 con 0x7f44ec1096e0 2024-03-19T02:51:14.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.797+0000 7f44e8ff9640 1 --2- 172.21.15.33:0/2736002883 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f44d40a2800 0x7f44d40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:14.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.797+0000 7f44e8ff9640 1 -- 172.21.15.33:0/2736002883 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7f44e40cecc0 con 0x7f44ec1096e0 2024-03-19T02:51:14.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.797+0000 7f44eb7fe640 1 --2- 172.21.15.33:0/2736002883 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f44d40a2800 0x7f44d40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:14.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.797+0000 7f44eb7fe640 1 --2- 172.21.15.33:0/2736002883 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f44d40a2800 0x7f44d40a4cc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f44ec19ab50 tx=0x7f44e0005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:14.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.801+0000 7f44e8ff9640 1 -- 172.21.15.33:0/2736002883 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f44e4014030 con 0x7f44ec1096e0 2024-03-19T02:51:14.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.987+0000 7f44f1f59640 1 -- 172.21.15.33:0/2736002883 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f44ec0624c0 con 0x7f44ec1096e0 2024-03-19T02:51:14.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.989+0000 7f44e8ff9640 1 -- 172.21.15.33:0/2736002883 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24043 (secure 0 0 0) 0x7f44e409bb20 con 0x7f44ec1096e0 2024-03-19T02:51:14.991 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:51:14.991 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":93,"fsid":"2b7a8938-e59a-11ee-95c9-87774f69a715","created":"2024-03-19T02:42:37.912179+0000","modified":"2024-03-19T02:50:59.241668+0000","last_up_change":"2024-03-19T02:50:01.539392+0000","last_in_change":"2024-03-19T02:49:40.534415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:46:09.963737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:50:50.069641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:50:51.381064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:50:55.476667+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"090a89ab-6b7b-4317-8823-178d4212b2f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6803","nonce":1633124995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6805","nonce":1633124995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6809","nonce":1633124995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6807","nonce":1633124995}]},"public_addr":"172.21.15.33:6803/1633124995","cluster_addr":"172.21.15.33:6805/1633124995","heartbeat_back_addr":"172.21.15.33:6809/1633124995","heartbeat_front_addr":"172.21.15.33:6807/1633124995","state":["exists","up"]},{"osd":1,"uuid":"f50eec55-1956-4486-8593-f2c8fa610e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6801","nonce":340144398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6803","nonce":340144398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6807","nonce":340144398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6805","nonce":340144398}]},"public_addr":"172.21.15.96:6801/340144398","cluster_addr":"172.21.15.96:6803/340144398","heartbeat_back_addr":"172.21.15.96:6807/340144398","heartbeat_front_addr":"172.21.15.96:6805/340144398","state":["exists","up"]},{"osd":2,"uuid":"893a2d62-80d9-4e5e-bc6a-a1c8c681d652","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6801","nonce":3388743876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6803","nonce":3388743876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6807","nonce":3388743876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6805","nonce":3388743876}]},"public_addr":"172.21.15.155:6801/3388743876","cluster_addr":"172.21.15.155:6803/3388743876","heartbeat_back_addr":"172.21.15.155:6807/3388743876","heartbeat_front_addr":"172.21.15.155:6805/3388743876","state":["exists","up"]},{"osd":3,"uuid":"b07e0482-4281-4d32-9dfe-9995d7246175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6811","nonce":677007171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6813","nonce":677007171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6817","nonce":677007171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6815","nonce":677007171}]},"public_addr":"172.21.15.33:6811/677007171","cluster_addr":"172.21.15.33:6813/677007171","heartbeat_back_addr":"172.21.15.33:6817/677007171","heartbeat_front_addr":"172.21.15.33:6815/677007171","state":["exists","up"]},{"osd":4,"uuid":"a988384b-af6b-47c6-8c07-5c9673acee45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6809","nonce":3106158554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6811","nonce":3106158554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6815","nonce":3106158554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6813","nonce":3106158554}]},"public_addr":"172.21.15.96:6809/3106158554","cluster_addr":"172.21.15.96:6811/3106158554","heartbeat_back_addr":"172.21.15.96:6815/3106158554","heartbeat_front_addr":"172.21.15.96:6813/3106158554","state":["exists","up"]},{"osd":5,"uuid":"198dc992-96f8-4c52-96ee-1fd5de148bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6809","nonce":327293324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6811","nonce":327293324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6815","nonce":327293324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6813","nonce":327293324}]},"public_addr":"172.21.15.155:6809/327293324","cluster_addr":"172.21.15.155:6811/327293324","heartbeat_back_addr":"172.21.15.155:6815/327293324","heartbeat_front_addr":"172.21.15.155:6813/327293324","state":["exists","up"]},{"osd":6,"uuid":"a7278ec0-ab2e-49ed-be53-84d8b9e24177","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6819","nonce":873871509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6821","nonce":873871509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6825","nonce":873871509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6823","nonce":873871509}]},"public_addr":"172.21.15.33:6819/873871509","cluster_addr":"172.21.15.33:6821/873871509","heartbeat_back_addr":"172.21.15.33:6825/873871509","heartbeat_front_addr":"172.21.15.33:6823/873871509","state":["exists","up"]},{"osd":7,"uuid":"727e133d-aa24-4ffc-90e2-266eb2aab8c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6817","nonce":2334123891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6819","nonce":2334123891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6823","nonce":2334123891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6821","nonce":2334123891}]},"public_addr":"172.21.15.96:6817/2334123891","cluster_addr":"172.21.15.96:6819/2334123891","heartbeat_back_addr":"172.21.15.96:6823/2334123891","heartbeat_front_addr":"172.21.15.96:6821/2334123891","state":["exists","up"]},{"osd":8,"uuid":"bb20eec0-39e5-4b86-9db5-486b1840cdad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6817","nonce":3498516903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6819","nonce":3498516903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6823","nonce":3498516903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6821","nonce":3498516903}]},"public_addr":"172.21.15.155:6817/3498516903","cluster_addr":"172.21.15.155:6819/3498516903","heartbeat_back_addr":"172.21.15.155:6823/3498516903","heartbeat_front_addr":"172.21.15.155:6821/3498516903","state":["exists","up"]},{"osd":9,"uuid":"179c6228-3e23-4f65-91a6-1ef0005d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6827","nonce":760777213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6829","nonce":760777213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6833","nonce":760777213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6831","nonce":760777213}]},"public_addr":"172.21.15.33:6827/760777213","cluster_addr":"172.21.15.33:6829/760777213","heartbeat_back_addr":"172.21.15.33:6833/760777213","heartbeat_front_addr":"172.21.15.33:6831/760777213","state":["exists","up"]},{"osd":10,"uuid":"be17b478-3c2e-41fb-bede-958333acdf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6825","nonce":1404898482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6827","nonce":1404898482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6831","nonce":1404898482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6829","nonce":1404898482}]},"public_addr":"172.21.15.96:6825/1404898482","cluster_addr":"172.21.15.96:6827/1404898482","heartbeat_back_addr":"172.21.15.96:6831/1404898482","heartbeat_front_addr":"172.21.15.96:6829/1404898482","state":["exists","up"]},{"osd":11,"uuid":"c0903ed2-cc1e-4c18-8731-1f0ee4dd0457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6825","nonce":792352827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6827","nonce":792352827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6831","nonce":792352827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6829","nonce":792352827}]},"public_addr":"172.21.15.155:6825/792352827","cluster_addr":"172.21.15.155:6827/792352827","heartbeat_back_addr":"172.21.15.155:6831/792352827","heartbeat_front_addr":"172.21.15.155:6829/792352827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:17.942801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:42.109449+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:05.981599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:32.069574+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:57.431015+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:22.448592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:48.526515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:14.498352+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:40.175426+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:06.364006+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:32.853161+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:59.073538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:6801/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:6800/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/1067768047":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/729379453":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2586588742":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/2932814165":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6801/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2212564963":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6800/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/3275126058":"2024-03-20T02:43:39.622218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:51:14.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.991+0000 7f44f1f59640 1 -- 172.21.15.33:0/2736002883 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f44d40a2800 msgr2=0x7f44d40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:14.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.991+0000 7f44f1f59640 1 --2- 172.21.15.33:0/2736002883 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f44d40a2800 0x7f44d40a4cc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f44ec19ab50 tx=0x7f44e0005fd0 comp rx=0 tx=0).stop 2024-03-19T02:51:14.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.991+0000 7f44f1f59640 1 -- 172.21.15.33:0/2736002883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44ec1096e0 msgr2=0x7f44ec1a1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:14.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.991+0000 7f44f1f59640 1 --2- 172.21.15.33:0/2736002883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44ec1096e0 0x7f44ec1a1220 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f44e400cad0 tx=0x7f44e400cf90 comp rx=0 tx=0).stop 2024-03-19T02:51:14.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.991+0000 7f44f1f59640 1 -- 172.21.15.33:0/2736002883 shutdown_connections 2024-03-19T02:51:14.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.991+0000 7f44f1f59640 1 --2- 172.21.15.33:0/2736002883 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f44d40a2800 0x7f44d40a4cc0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:14.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.991+0000 7f44f1f59640 1 --2- 172.21.15.33:0/2736002883 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44ec0757e0 0x7f44ec19a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:14.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.991+0000 7f44f1f59640 1 --2- 172.21.15.33:0/2736002883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44ec1096e0 0x7f44ec1a1220 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:14.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.991+0000 7f44f1f59640 1 --2- 172.21.15.33:0/2736002883 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44ec074200 0x7f44ec199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:14.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.991+0000 7f44f1f59640 1 -- 172.21.15.33:0/2736002883 >> 172.21.15.33:0/2736002883 conn(0x7f44ec0fce00 msgr2=0x7f44ec105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:14.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.991+0000 7f44f1f59640 1 -- 172.21.15.33:0/2736002883 shutdown_connections 2024-03-19T02:51:14.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:14.991+0000 7f44f1f59640 1 -- 172.21.15.33:0/2736002883 wait complete. 2024-03-19T02:51:15.003 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:51:15.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.107+0000 7f2718836640 1 -- 172.21.15.33:0/3211360699 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2714101950 msgr2=0x7f2714101dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:15.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.107+0000 7f2718836640 1 --2- 172.21.15.33:0/3211360699 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2714101950 0x7f2714101dd0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f26fc009900 tx=0x7f26fc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:51:15.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.107+0000 7f2718836640 1 -- 172.21.15.33:0/3211360699 shutdown_connections 2024-03-19T02:51:15.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.107+0000 7f2718836640 1 --2- 172.21.15.33:0/3211360699 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2714102310 0x7f2714108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:15.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.107+0000 7f2718836640 1 --2- 172.21.15.33:0/3211360699 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2714101950 0x7f2714101dd0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:15.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.107+0000 7f2718836640 1 --2- 172.21.15.33:0/3211360699 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2714100760 0x7f2714100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:15.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.107+0000 7f2718836640 1 -- 172.21.15.33:0/3211360699 >> 172.21.15.33:0/3211360699 conn(0x7f27140fbcd0 msgr2=0x7f27140fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:15.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.107+0000 7f2718836640 1 -- 172.21.15.33:0/3211360699 shutdown_connections 2024-03-19T02:51:15.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.107+0000 7f2718836640 1 -- 172.21.15.33:0/3211360699 wait complete. 2024-03-19T02:51:15.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.108+0000 7f2718836640 1 Processor -- start 2024-03-19T02:51:15.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.108+0000 7f2718836640 1 -- start start 2024-03-19T02:51:15.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.108+0000 7f2718836640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2714100760 0x7f271410b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:15.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.108+0000 7f2718836640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2714101950 0x7f2714109720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:15.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.108+0000 7f2718836640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2714102310 0x7f2714109c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:15.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.108+0000 7f2718836640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f271410d3b0 con 0x7f2714102310 2024-03-19T02:51:15.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.108+0000 7f2718836640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f271410d230 con 0x7f2714101950 2024-03-19T02:51:15.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.108+0000 7f2718836640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f271410d530 con 0x7f2714100760 2024-03-19T02:51:15.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.108+0000 7f2712d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2714102310 0x7f2714109c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:15.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.108+0000 7f2712d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2714102310 0x7f2714109c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49914/0 (socket says 172.21.15.33:49914) 2024-03-19T02:51:15.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.108+0000 7f2712d9d640 1 -- 172.21.15.33:0/452844266 learned_addr learned my addr 172.21.15.33:0/452844266 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:15.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.108+0000 7f270bfff640 1 --2- 172.21.15.33:0/452844266 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2714101950 0x7f2714109720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:15.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.108+0000 7f271259c640 1 --2- 172.21.15.33:0/452844266 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2714100760 0x7f271410b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:15.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.108+0000 7f2712d9d640 1 -- 172.21.15.33:0/452844266 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2714100760 msgr2=0x7f271410b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:15.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.108+0000 7f2712d9d640 1 --2- 172.21.15.33:0/452844266 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2714100760 0x7f271410b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:15.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.108+0000 7f2712d9d640 1 -- 172.21.15.33:0/452844266 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2714101950 msgr2=0x7f2714109720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:15.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.108+0000 7f2712d9d640 1 --2- 172.21.15.33:0/452844266 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2714101950 0x7f2714109720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:15.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.108+0000 7f2712d9d640 1 -- 172.21.15.33:0/452844266 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f271410a4f0 con 0x7f2714102310 2024-03-19T02:51:15.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.109+0000 7f2712d9d640 1 --2- 172.21.15.33:0/452844266 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2714102310 0x7f2714109c60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f270c00cad0 tx=0x7f270c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:15.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.109+0000 7f270b7fe640 1 -- 172.21.15.33:0/452844266 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f270c00ae20 con 0x7f2714102310 2024-03-19T02:51:15.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.109+0000 7f270b7fe640 1 -- 172.21.15.33:0/452844266 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f270c005020 con 0x7f2714102310 2024-03-19T02:51:15.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.109+0000 7f2718836640 1 -- 172.21.15.33:0/452844266 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27141a1bb0 con 0x7f2714102310 2024-03-19T02:51:15.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.109+0000 7f270b7fe640 1 -- 172.21.15.33:0/452844266 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f270c00f6b0 con 0x7f2714102310 2024-03-19T02:51:15.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.109+0000 7f2718836640 1 -- 172.21.15.33:0/452844266 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27141a20d0 con 0x7f2714102310 2024-03-19T02:51:15.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.110+0000 7f2718836640 1 -- 172.21.15.33:0/452844266 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2714068a60 con 0x7f2714102310 2024-03-19T02:51:15.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.111+0000 7f270b7fe640 1 -- 172.21.15.33:0/452844266 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f270c0051d0 con 0x7f2714102310 2024-03-19T02:51:15.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.112+0000 7f270b7fe640 1 --2- 172.21.15.33:0/452844266 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f27000a2800 0x7f27000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:15.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.112+0000 7f271259c640 1 --2- 172.21.15.33:0/452844266 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f27000a2800 0x7f27000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:15.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.112+0000 7f270b7fe640 1 -- 172.21.15.33:0/452844266 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7f270c0ced40 con 0x7f2714102310 2024-03-19T02:51:15.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.112+0000 7f271259c640 1 --2- 172.21.15.33:0/452844266 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f27000a2800 0x7f27000a4cc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f27141017b0 tx=0x7f2704005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:15.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.116+0000 7f270b7fe640 1 -- 172.21.15.33:0/452844266 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f270c014030 con 0x7f2714102310 2024-03-19T02:51:15.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:15 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2736002883' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:51:15.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.301+0000 7f2718836640 1 -- 172.21.15.33:0/452844266 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2714062420 con 0x7f2714102310 2024-03-19T02:51:15.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.303+0000 7f270b7fe640 1 -- 172.21.15.33:0/452844266 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24043 (secure 0 0 0) 0x7f270c09bb40 con 0x7f2714102310 2024-03-19T02:51:15.304 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:51:15.304 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":93,"fsid":"2b7a8938-e59a-11ee-95c9-87774f69a715","created":"2024-03-19T02:42:37.912179+0000","modified":"2024-03-19T02:50:59.241668+0000","last_up_change":"2024-03-19T02:50:01.539392+0000","last_in_change":"2024-03-19T02:49:40.534415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:46:09.963737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:50:50.069641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:50:51.381064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:50:55.476667+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"090a89ab-6b7b-4317-8823-178d4212b2f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6803","nonce":1633124995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6805","nonce":1633124995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6809","nonce":1633124995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6807","nonce":1633124995}]},"public_addr":"172.21.15.33:6803/1633124995","cluster_addr":"172.21.15.33:6805/1633124995","heartbeat_back_addr":"172.21.15.33:6809/1633124995","heartbeat_front_addr":"172.21.15.33:6807/1633124995","state":["exists","up"]},{"osd":1,"uuid":"f50eec55-1956-4486-8593-f2c8fa610e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6801","nonce":340144398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6803","nonce":340144398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6807","nonce":340144398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6805","nonce":340144398}]},"public_addr":"172.21.15.96:6801/340144398","cluster_addr":"172.21.15.96:6803/340144398","heartbeat_back_addr":"172.21.15.96:6807/340144398","heartbeat_front_addr":"172.21.15.96:6805/340144398","state":["exists","up"]},{"osd":2,"uuid":"893a2d62-80d9-4e5e-bc6a-a1c8c681d652","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6801","nonce":3388743876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6803","nonce":3388743876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6807","nonce":3388743876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6805","nonce":3388743876}]},"public_addr":"172.21.15.155:6801/3388743876","cluster_addr":"172.21.15.155:6803/3388743876","heartbeat_back_addr":"172.21.15.155:6807/3388743876","heartbeat_front_addr":"172.21.15.155:6805/3388743876","state":["exists","up"]},{"osd":3,"uuid":"b07e0482-4281-4d32-9dfe-9995d7246175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6811","nonce":677007171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6813","nonce":677007171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6817","nonce":677007171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6815","nonce":677007171}]},"public_addr":"172.21.15.33:6811/677007171","cluster_addr":"172.21.15.33:6813/677007171","heartbeat_back_addr":"172.21.15.33:6817/677007171","heartbeat_front_addr":"172.21.15.33:6815/677007171","state":["exists","up"]},{"osd":4,"uuid":"a988384b-af6b-47c6-8c07-5c9673acee45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6809","nonce":3106158554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6811","nonce":3106158554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6815","nonce":3106158554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6813","nonce":3106158554}]},"public_addr":"172.21.15.96:6809/3106158554","cluster_addr":"172.21.15.96:6811/3106158554","heartbeat_back_addr":"172.21.15.96:6815/3106158554","heartbeat_front_addr":"172.21.15.96:6813/3106158554","state":["exists","up"]},{"osd":5,"uuid":"198dc992-96f8-4c52-96ee-1fd5de148bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6809","nonce":327293324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6811","nonce":327293324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6815","nonce":327293324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6813","nonce":327293324}]},"public_addr":"172.21.15.155:6809/327293324","cluster_addr":"172.21.15.155:6811/327293324","heartbeat_back_addr":"172.21.15.155:6815/327293324","heartbeat_front_addr":"172.21.15.155:6813/327293324","state":["exists","up"]},{"osd":6,"uuid":"a7278ec0-ab2e-49ed-be53-84d8b9e24177","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6819","nonce":873871509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6821","nonce":873871509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6825","nonce":873871509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6823","nonce":873871509}]},"public_addr":"172.21.15.33:6819/873871509","cluster_addr":"172.21.15.33:6821/873871509","heartbeat_back_addr":"172.21.15.33:6825/873871509","heartbeat_front_addr":"172.21.15.33:6823/873871509","state":["exists","up"]},{"osd":7,"uuid":"727e133d-aa24-4ffc-90e2-266eb2aab8c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6817","nonce":2334123891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6819","nonce":2334123891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6823","nonce":2334123891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6821","nonce":2334123891}]},"public_addr":"172.21.15.96:6817/2334123891","cluster_addr":"172.21.15.96:6819/2334123891","heartbeat_back_addr":"172.21.15.96:6823/2334123891","heartbeat_front_addr":"172.21.15.96:6821/2334123891","state":["exists","up"]},{"osd":8,"uuid":"bb20eec0-39e5-4b86-9db5-486b1840cdad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6817","nonce":3498516903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6819","nonce":3498516903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6823","nonce":3498516903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6821","nonce":3498516903}]},"public_addr":"172.21.15.155:6817/3498516903","cluster_addr":"172.21.15.155:6819/3498516903","heartbeat_back_addr":"172.21.15.155:6823/3498516903","heartbeat_front_addr":"172.21.15.155:6821/3498516903","state":["exists","up"]},{"osd":9,"uuid":"179c6228-3e23-4f65-91a6-1ef0005d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6827","nonce":760777213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6829","nonce":760777213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6833","nonce":760777213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6831","nonce":760777213}]},"public_addr":"172.21.15.33:6827/760777213","cluster_addr":"172.21.15.33:6829/760777213","heartbeat_back_addr":"172.21.15.33:6833/760777213","heartbeat_front_addr":"172.21.15.33:6831/760777213","state":["exists","up"]},{"osd":10,"uuid":"be17b478-3c2e-41fb-bede-958333acdf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6825","nonce":1404898482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6827","nonce":1404898482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6831","nonce":1404898482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6829","nonce":1404898482}]},"public_addr":"172.21.15.96:6825/1404898482","cluster_addr":"172.21.15.96:6827/1404898482","heartbeat_back_addr":"172.21.15.96:6831/1404898482","heartbeat_front_addr":"172.21.15.96:6829/1404898482","state":["exists","up"]},{"osd":11,"uuid":"c0903ed2-cc1e-4c18-8731-1f0ee4dd0457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6825","nonce":792352827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6827","nonce":792352827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6831","nonce":792352827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6829","nonce":792352827}]},"public_addr":"172.21.15.155:6825/792352827","cluster_addr":"172.21.15.155:6827/792352827","heartbeat_back_addr":"172.21.15.155:6831/792352827","heartbeat_front_addr":"172.21.15.155:6829/792352827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:17.942801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:42.109449+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:05.981599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:32.069574+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:57.431015+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:22.448592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:48.526515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:14.498352+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:40.175426+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:06.364006+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:32.853161+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:59.073538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:6801/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:6800/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/1067768047":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/729379453":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2586588742":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/2932814165":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6801/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2212564963":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6800/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/3275126058":"2024-03-20T02:43:39.622218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:51:15.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.305+0000 7f2718836640 1 -- 172.21.15.33:0/452844266 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f27000a2800 msgr2=0x7f27000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:15.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.305+0000 7f2718836640 1 --2- 172.21.15.33:0/452844266 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f27000a2800 0x7f27000a4cc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f27141017b0 tx=0x7f2704005f90 comp rx=0 tx=0).stop 2024-03-19T02:51:15.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.305+0000 7f2718836640 1 -- 172.21.15.33:0/452844266 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2714102310 msgr2=0x7f2714109c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:15.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.305+0000 7f2718836640 1 --2- 172.21.15.33:0/452844266 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2714102310 0x7f2714109c60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f270c00cad0 tx=0x7f270c00cf90 comp rx=0 tx=0).stop 2024-03-19T02:51:15.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.305+0000 7f2718836640 1 -- 172.21.15.33:0/452844266 shutdown_connections 2024-03-19T02:51:15.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.305+0000 7f2718836640 1 --2- 172.21.15.33:0/452844266 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f27000a2800 0x7f27000a4cc0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:15.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.305+0000 7f2718836640 1 --2- 172.21.15.33:0/452844266 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2714102310 0x7f2714109c60 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:15.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.305+0000 7f2718836640 1 --2- 172.21.15.33:0/452844266 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2714101950 0x7f2714109720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:15.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.305+0000 7f2718836640 1 --2- 172.21.15.33:0/452844266 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2714100760 0x7f271410b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:15.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.305+0000 7f2718836640 1 -- 172.21.15.33:0/452844266 >> 172.21.15.33:0/452844266 conn(0x7f27140fbcd0 msgr2=0x7f2714104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:15.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.305+0000 7f2718836640 1 -- 172.21.15.33:0/452844266 shutdown_connections 2024-03-19T02:51:15.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.305+0000 7f2718836640 1 -- 172.21.15.33:0/452844266 wait complete. 2024-03-19T02:51:15.319 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-19T02:51:15.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.424+0000 7fb34bd53640 1 -- 172.21.15.33:0/1078759841 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3441033e0 msgr2=0x7fb344109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:15.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.424+0000 7fb34bd53640 1 --2- 172.21.15.33:0/1078759841 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3441033e0 0x7fb344109c70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb33c009be0 tx=0x7fb33c02bef0 comp rx=0 tx=0).stop 2024-03-19T02:51:15.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.424+0000 7fb34bd53640 1 -- 172.21.15.33:0/1078759841 shutdown_connections 2024-03-19T02:51:15.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.424+0000 7fb34bd53640 1 --2- 172.21.15.33:0/1078759841 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3441033e0 0x7fb344109c70 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:15.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.424+0000 7fb34bd53640 1 --2- 172.21.15.33:0/1078759841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb344102a20 0x7fb344102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:15.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.424+0000 7fb34bd53640 1 --2- 172.21.15.33:0/1078759841 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb344101830 0x7fb344101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:15.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.424+0000 7fb34bd53640 1 -- 172.21.15.33:0/1078759841 >> 172.21.15.33:0/1078759841 conn(0x7fb3440fcdc0 msgr2=0x7fb3440ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:15.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.424+0000 7fb34bd53640 1 -- 172.21.15.33:0/1078759841 shutdown_connections 2024-03-19T02:51:15.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.425+0000 7fb34bd53640 1 -- 172.21.15.33:0/1078759841 wait complete. 2024-03-19T02:51:15.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.425+0000 7fb34bd53640 1 Processor -- start 2024-03-19T02:51:15.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.425+0000 7fb34bd53640 1 -- start start 2024-03-19T02:51:15.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.425+0000 7fb34bd53640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb344101830 0x7fb344197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:15.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.425+0000 7fb34bd53640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb344102a20 0x7fb344197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:15.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.425+0000 7fb34bd53640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3441033e0 0x7fb34419efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:15.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.425+0000 7fb34bd53640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb34410d360 con 0x7fb344102a20 2024-03-19T02:51:15.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.425+0000 7fb34bd53640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb34410d1e0 con 0x7fb344101830 2024-03-19T02:51:15.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.425+0000 7fb34bd53640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb34410d4e0 con 0x7fb3441033e0 2024-03-19T02:51:15.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.425+0000 7fb3492ee640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb344102a20 0x7fb344197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:15.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.425+0000 7fb349aef640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb344101830 0x7fb344197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:15.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.425+0000 7fb3492ee640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb344102a20 0x7fb344197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49924/0 (socket says 172.21.15.33:49924) 2024-03-19T02:51:15.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.425+0000 7fb349aef640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb344101830 0x7fb344197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:58252/0 (socket says 172.21.15.33:58252) 2024-03-19T02:51:15.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.425+0000 7fb3492ee640 1 -- 172.21.15.33:0/1398856471 learned_addr learned my addr 172.21.15.33:0/1398856471 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:15.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.426+0000 7fb349aef640 1 -- 172.21.15.33:0/1398856471 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3441033e0 msgr2=0x7fb34419efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:15.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.426+0000 7fb34a2f0640 1 --2- 172.21.15.33:0/1398856471 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3441033e0 0x7fb34419efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:15.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.426+0000 7fb349aef640 1 --2- 172.21.15.33:0/1398856471 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3441033e0 0x7fb34419efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:15.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.426+0000 7fb349aef640 1 -- 172.21.15.33:0/1398856471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb344102a20 msgr2=0x7fb344197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:15.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.426+0000 7fb349aef640 1 --2- 172.21.15.33:0/1398856471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb344102a20 0x7fb344197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:15.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.426+0000 7fb349aef640 1 -- 172.21.15.33:0/1398856471 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb34419f630 con 0x7fb344101830 2024-03-19T02:51:15.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.426+0000 7fb3492ee640 1 --2- 172.21.15.33:0/1398856471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb344102a20 0x7fb344197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:15.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.426+0000 7fb349aef640 1 --2- 172.21.15.33:0/1398856471 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb344101830 0x7fb344197a10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb344102880 tx=0x7fb33400dbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:15.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.426+0000 7fb33affd640 1 -- 172.21.15.33:0/1398856471 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb334017070 con 0x7fb344101830 2024-03-19T02:51:15.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.426+0000 7fb33affd640 1 -- 172.21.15.33:0/1398856471 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb334010460 con 0x7fb344101830 2024-03-19T02:51:15.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.426+0000 7fb34bd53640 1 -- 172.21.15.33:0/1398856471 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb34419f920 con 0x7fb344101830 2024-03-19T02:51:15.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.426+0000 7fb33affd640 1 -- 172.21.15.33:0/1398856471 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb33400f600 con 0x7fb344101830 2024-03-19T02:51:15.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.426+0000 7fb34bd53640 1 -- 172.21.15.33:0/1398856471 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb34419fe20 con 0x7fb344101830 2024-03-19T02:51:15.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.427+0000 7fb34bd53640 1 -- 172.21.15.33:0/1398856471 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb344108bd0 con 0x7fb344101830 2024-03-19T02:51:15.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.428+0000 7fb33affd640 1 -- 172.21.15.33:0/1398856471 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb33401a070 con 0x7fb344101830 2024-03-19T02:51:15.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.428+0000 7fb33affd640 1 --2- 172.21.15.33:0/1398856471 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb3300a2800 0x7fb3300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:15.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.429+0000 7fb3492ee640 1 --2- 172.21.15.33:0/1398856471 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb3300a2800 0x7fb3300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:15.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.429+0000 7fb33affd640 1 -- 172.21.15.33:0/1398856471 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7fb3340ce690 con 0x7fb344101830 2024-03-19T02:51:15.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.429+0000 7fb3492ee640 1 --2- 172.21.15.33:0/1398856471 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb3300a2800 0x7fb3300a4cc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb3400096c0 tx=0x7fb340009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:15.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.437+0000 7fb33affd640 1 -- 172.21.15.33:0/1398856471 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb33409b490 con 0x7fb344101830 2024-03-19T02:51:15.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:15 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2736002883' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:51:15.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:15 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2736002883' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:51:15.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:15.658+0000 7fb34bd53640 1 -- 172.21.15.33:0/1398856471 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fb34402d1f0 con 0x7fb344101830 2024-03-19T02:51:16.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.048+0000 7fb33affd640 1 -- 172.21.15.33:0/1398856471 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7fb33409ad90 con 0x7fb344101830 2024-03-19T02:51:16.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.050+0000 7fb34bd53640 1 -- 172.21.15.33:0/1398856471 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb3300a2800 msgr2=0x7fb3300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:16.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.050+0000 7fb34bd53640 1 --2- 172.21.15.33:0/1398856471 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb3300a2800 0x7fb3300a4cc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb3400096c0 tx=0x7fb340009260 comp rx=0 tx=0).stop 2024-03-19T02:51:16.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.050+0000 7fb34bd53640 1 -- 172.21.15.33:0/1398856471 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb344101830 msgr2=0x7fb344197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:16.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.050+0000 7fb34bd53640 1 --2- 172.21.15.33:0/1398856471 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb344101830 0x7fb344197a10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb344102880 tx=0x7fb33400dbb0 comp rx=0 tx=0).stop 2024-03-19T02:51:16.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.050+0000 7fb34bd53640 1 -- 172.21.15.33:0/1398856471 shutdown_connections 2024-03-19T02:51:16.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.050+0000 7fb34bd53640 1 --2- 172.21.15.33:0/1398856471 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb3300a2800 0x7fb3300a4cc0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:16.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.050+0000 7fb34bd53640 1 --2- 172.21.15.33:0/1398856471 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3441033e0 0x7fb34419efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:16.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.050+0000 7fb34bd53640 1 --2- 172.21.15.33:0/1398856471 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb344102a20 0x7fb344197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:16.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.050+0000 7fb34bd53640 1 --2- 172.21.15.33:0/1398856471 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb344101830 0x7fb344197a10 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:16.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.050+0000 7fb34bd53640 1 -- 172.21.15.33:0/1398856471 >> 172.21.15.33:0/1398856471 conn(0x7fb3440fcdc0 msgr2=0x7fb344105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:16.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.050+0000 7fb34bd53640 1 -- 172.21.15.33:0/1398856471 shutdown_connections 2024-03-19T02:51:16.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.050+0000 7fb34bd53640 1 -- 172.21.15.33:0/1398856471 wait complete. 2024-03-19T02:51:16.063 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-03-19T02:51:16.063 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-19T02:51:16.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.169+0000 7f9e670f1640 1 -- 172.21.15.33:0/2105695806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e601034a0 msgr2=0x7f9e60109d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:16.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.169+0000 7f9e670f1640 1 --2- 172.21.15.33:0/2105695806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e601034a0 0x7f9e60109d30 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9e58009cd0 tx=0x7f9e5802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:51:16.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.169+0000 7f9e670f1640 1 -- 172.21.15.33:0/2105695806 shutdown_connections 2024-03-19T02:51:16.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.169+0000 7f9e670f1640 1 --2- 172.21.15.33:0/2105695806 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e60102ae0 0x7f9e60102f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:16.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.169+0000 7f9e670f1640 1 --2- 172.21.15.33:0/2105695806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e601034a0 0x7f9e60109d30 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:16.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.169+0000 7f9e670f1640 1 --2- 172.21.15.33:0/2105695806 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e600ffb10 0x7f9e600fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:16.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.169+0000 7f9e670f1640 1 -- 172.21.15.33:0/2105695806 >> 172.21.15.33:0/2105695806 conn(0x7f9e600fb420 msgr2=0x7f9e600fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:16.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.169+0000 7f9e670f1640 1 -- 172.21.15.33:0/2105695806 shutdown_connections 2024-03-19T02:51:16.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.169+0000 7f9e670f1640 1 -- 172.21.15.33:0/2105695806 wait complete. 2024-03-19T02:51:16.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.170+0000 7f9e670f1640 1 Processor -- start 2024-03-19T02:51:16.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.170+0000 7f9e670f1640 1 -- start start 2024-03-19T02:51:16.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.170+0000 7f9e670f1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e600ffb10 0x7f9e60197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:16.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.170+0000 7f9e670f1640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e60102ae0 0x7f9e60197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:16.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.170+0000 7f9e670f1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e601034a0 0x7f9e6019f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:16.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.170+0000 7f9e670f1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6010d3a0 con 0x7f9e601034a0 2024-03-19T02:51:16.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.170+0000 7f9e670f1640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6010d220 con 0x7f9e600ffb10 2024-03-19T02:51:16.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.170+0000 7f9e670f1640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6010d520 con 0x7f9e60102ae0 2024-03-19T02:51:16.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.170+0000 7f9e6568e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e601034a0 0x7f9e6019f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:16.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.170+0000 7f9e6568e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e601034a0 0x7f9e6019f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49938/0 (socket says 172.21.15.33:49938) 2024-03-19T02:51:16.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.170+0000 7f9e6568e640 1 -- 172.21.15.33:0/1391833397 learned_addr learned my addr 172.21.15.33:0/1391833397 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:16.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.170+0000 7f9e64e8d640 1 --2- 172.21.15.33:0/1391833397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e600ffb10 0x7f9e60197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:16.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.170+0000 7f9e5ffff640 1 --2- 172.21.15.33:0/1391833397 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e60102ae0 0x7f9e60197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:16.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.170+0000 7f9e6568e640 1 -- 172.21.15.33:0/1391833397 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e60102ae0 msgr2=0x7f9e60197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:16.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.170+0000 7f9e6568e640 1 --2- 172.21.15.33:0/1391833397 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e60102ae0 0x7f9e60197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:16.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.170+0000 7f9e6568e640 1 -- 172.21.15.33:0/1391833397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e600ffb10 msgr2=0x7f9e60197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:16.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.170+0000 7f9e6568e640 1 --2- 172.21.15.33:0/1391833397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e600ffb10 0x7f9e60197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:16.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.170+0000 7f9e6568e640 1 -- 172.21.15.33:0/1391833397 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e6019f680 con 0x7f9e601034a0 2024-03-19T02:51:16.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.170+0000 7f9e5ffff640 1 --2- 172.21.15.33:0/1391833397 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e60102ae0 0x7f9e60197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:51:16.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.170+0000 7f9e6568e640 1 --2- 172.21.15.33:0/1391833397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e601034a0 0x7f9e6019f010 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9e5802c630 tx=0x7f9e580328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:16.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.171+0000 7f9e5dffb640 1 -- 172.21.15.33:0/1391833397 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e58041020 con 0x7f9e601034a0 2024-03-19T02:51:16.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.171+0000 7f9e5dffb640 1 -- 172.21.15.33:0/1391833397 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e58003e20 con 0x7f9e601034a0 2024-03-19T02:51:16.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.171+0000 7f9e670f1640 1 -- 172.21.15.33:0/1391833397 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e6019f910 con 0x7f9e601034a0 2024-03-19T02:51:16.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.171+0000 7f9e5dffb640 1 -- 172.21.15.33:0/1391833397 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e58039040 con 0x7f9e601034a0 2024-03-19T02:51:16.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.171+0000 7f9e670f1640 1 -- 172.21.15.33:0/1391833397 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e6019fe50 con 0x7f9e601034a0 2024-03-19T02:51:16.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.172+0000 7f9e670f1640 1 -- 172.21.15.33:0/1391833397 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e60108c10 con 0x7f9e601034a0 2024-03-19T02:51:16.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.173+0000 7f9e5dffb640 1 -- 172.21.15.33:0/1391833397 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f9e58044020 con 0x7f9e601034a0 2024-03-19T02:51:16.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.173+0000 7f9e5dffb640 1 --2- 172.21.15.33:0/1391833397 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9e500a2880 0x7f9e500a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:16.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.174+0000 7f9e5dffb640 1 -- 172.21.15.33:0/1391833397 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7f9e580f1200 con 0x7f9e601034a0 2024-03-19T02:51:16.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.174+0000 7f9e64e8d640 1 --2- 172.21.15.33:0/1391833397 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9e500a2880 0x7f9e500a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:16.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.174+0000 7f9e64e8d640 1 --2- 172.21.15.33:0/1391833397 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9e500a2880 0x7f9e500a4d40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9e540095f0 tx=0x7f9e54005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:16.178 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.177+0000 7f9e5dffb640 1 -- 172.21.15.33:0/1391833397 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9e5804b020 con 0x7f9e601034a0 2024-03-19T02:51:16.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:16.403+0000 7f9e670f1640 1 -- 172.21.15.33:0/1391833397 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f9e6002d1f0 con 0x7f9e601034a0 2024-03-19T02:51:16.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:16 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/452844266' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:51:16.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:16 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1398856471' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-19T02:51:16.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:16 smithi096 ceph-mon[35946]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-19T02:51:16.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:16 smithi096 ceph-mon[35946]: pgmap v311: 129 pgs: 129 active+clean; 581 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-19T02:51:16.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:16 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/452844266' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:51:16.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:16 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1398856471' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-19T02:51:16.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:16 smithi155 ceph-mon[35804]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-19T02:51:16.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:16 smithi155 ceph-mon[35804]: pgmap v311: 129 pgs: 129 active+clean; 581 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-19T02:51:16.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:16 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/452844266' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:51:16.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:16 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1398856471' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-19T02:51:16.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:16 smithi033 ceph-mon[31973]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-19T02:51:16.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:16 smithi033 ceph-mon[31973]: pgmap v311: 129 pgs: 129 active+clean; 581 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-19T02:51:17.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.051+0000 7f9e5dffb640 1 -- 172.21.15.33:0/1391833397 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7f9e580be060 con 0x7f9e601034a0 2024-03-19T02:51:17.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.055+0000 7f9e670f1640 1 -- 172.21.15.33:0/1391833397 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9e500a2880 msgr2=0x7f9e500a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:17.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.055+0000 7f9e670f1640 1 --2- 172.21.15.33:0/1391833397 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9e500a2880 0x7f9e500a4d40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9e540095f0 tx=0x7f9e54005f90 comp rx=0 tx=0).stop 2024-03-19T02:51:17.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.055+0000 7f9e670f1640 1 -- 172.21.15.33:0/1391833397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e601034a0 msgr2=0x7f9e6019f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:17.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.055+0000 7f9e670f1640 1 --2- 172.21.15.33:0/1391833397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e601034a0 0x7f9e6019f010 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9e5802c630 tx=0x7f9e580328a0 comp rx=0 tx=0).stop 2024-03-19T02:51:17.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.055+0000 7f9e670f1640 1 -- 172.21.15.33:0/1391833397 shutdown_connections 2024-03-19T02:51:17.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.055+0000 7f9e670f1640 1 --2- 172.21.15.33:0/1391833397 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9e500a2880 0x7f9e500a4d40 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:17.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.055+0000 7f9e670f1640 1 --2- 172.21.15.33:0/1391833397 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9e60102ae0 0x7f9e60197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:17.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.055+0000 7f9e670f1640 1 --2- 172.21.15.33:0/1391833397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9e601034a0 0x7f9e6019f010 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:17.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.055+0000 7f9e670f1640 1 --2- 172.21.15.33:0/1391833397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9e600ffb10 0x7f9e60197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:17.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.055+0000 7f9e670f1640 1 -- 172.21.15.33:0/1391833397 >> 172.21.15.33:0/1391833397 conn(0x7f9e600fb420 msgr2=0x7f9e60105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:17.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.055+0000 7f9e670f1640 1 -- 172.21.15.33:0/1391833397 shutdown_connections 2024-03-19T02:51:17.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.055+0000 7f9e670f1640 1 -- 172.21.15.33:0/1391833397 wait complete. 2024-03-19T02:51:17.072 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-19T02:51:17.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.182+0000 7ff8d7e0b640 1 -- 172.21.15.33:0/3915297245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8d01075a0 msgr2=0x7ff8d01099f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:17.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.182+0000 7ff8d7e0b640 1 --2- 172.21.15.33:0/3915297245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8d01075a0 0x7ff8d01099f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff8c0009900 tx=0x7ff8c002bc30 comp rx=0 tx=0).stop 2024-03-19T02:51:17.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.182+0000 7ff8d7e0b640 1 -- 172.21.15.33:0/3915297245 shutdown_connections 2024-03-19T02:51:17.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.182+0000 7ff8d7e0b640 1 --2- 172.21.15.33:0/3915297245 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8d0104c70 0x7ff8d0107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:17.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.182+0000 7ff8d7e0b640 1 --2- 172.21.15.33:0/3915297245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8d01075a0 0x7ff8d01099f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:17.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.182+0000 7ff8d7e0b640 1 --2- 172.21.15.33:0/3915297245 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8d0102340 0x7ff8d0104730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:17.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.182+0000 7ff8d7e0b640 1 -- 172.21.15.33:0/3915297245 >> 172.21.15.33:0/3915297245 conn(0x7ff8d00fbd30 msgr2=0x7ff8d00fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:17.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.182+0000 7ff8d7e0b640 1 -- 172.21.15.33:0/3915297245 shutdown_connections 2024-03-19T02:51:17.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.182+0000 7ff8d7e0b640 1 -- 172.21.15.33:0/3915297245 wait complete. 2024-03-19T02:51:17.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.182+0000 7ff8d7e0b640 1 Processor -- start 2024-03-19T02:51:17.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.183+0000 7ff8d7e0b640 1 -- start start 2024-03-19T02:51:17.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.183+0000 7ff8d7e0b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8d0102340 0x7ff8d0197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:17.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.183+0000 7ff8d7e0b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8d0104c70 0x7ff8d0198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:17.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.183+0000 7ff8d7e0b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8d01075a0 0x7ff8d019f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:17.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.183+0000 7ff8d7e0b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff8d010d460 con 0x7ff8d0102340 2024-03-19T02:51:17.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.183+0000 7ff8d7e0b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff8d010d2e0 con 0x7ff8d0104c70 2024-03-19T02:51:17.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.183+0000 7ff8d7e0b640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff8d010d5e0 con 0x7ff8d01075a0 2024-03-19T02:51:17.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.183+0000 7ff8d5ba7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8d0102340 0x7ff8d0197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:17.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.183+0000 7ff8d5ba7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8d0102340 0x7ff8d0197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49952/0 (socket says 172.21.15.33:49952) 2024-03-19T02:51:17.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.183+0000 7ff8d5ba7640 1 -- 172.21.15.33:0/2441165827 learned_addr learned my addr 172.21.15.33:0/2441165827 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:17.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.183+0000 7ff8d53a6640 1 --2- 172.21.15.33:0/2441165827 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8d0104c70 0x7ff8d0198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:17.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.183+0000 7ff8d63a8640 1 --2- 172.21.15.33:0/2441165827 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8d01075a0 0x7ff8d019f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:17.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.183+0000 7ff8d5ba7640 1 -- 172.21.15.33:0/2441165827 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8d01075a0 msgr2=0x7ff8d019f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:17.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.183+0000 7ff8d5ba7640 1 --2- 172.21.15.33:0/2441165827 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8d01075a0 0x7ff8d019f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:17.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.183+0000 7ff8d5ba7640 1 -- 172.21.15.33:0/2441165827 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8d0104c70 msgr2=0x7ff8d0198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:17.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.183+0000 7ff8d5ba7640 1 --2- 172.21.15.33:0/2441165827 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8d0104c70 0x7ff8d0198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:17.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.183+0000 7ff8d5ba7640 1 -- 172.21.15.33:0/2441165827 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8d019f760 con 0x7ff8d0102340 2024-03-19T02:51:17.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.183+0000 7ff8d53a6640 1 --2- 172.21.15.33:0/2441165827 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8d0104c70 0x7ff8d0198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:51:17.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.183+0000 7ff8d5ba7640 1 --2- 172.21.15.33:0/2441165827 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8d0102340 0x7ff8d0197b40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff8c800ebc0 tx=0x7ff8c800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:17.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.183+0000 7ff8ceffd640 1 -- 172.21.15.33:0/2441165827 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c8010630 con 0x7ff8d0102340 2024-03-19T02:51:17.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.184+0000 7ff8ceffd640 1 -- 172.21.15.33:0/2441165827 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff8c8010c80 con 0x7ff8d0102340 2024-03-19T02:51:17.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.184+0000 7ff8d7e0b640 1 -- 172.21.15.33:0/2441165827 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8d019fa50 con 0x7ff8d0102340 2024-03-19T02:51:17.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.184+0000 7ff8ceffd640 1 -- 172.21.15.33:0/2441165827 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c8018b30 con 0x7ff8d0102340 2024-03-19T02:51:17.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.184+0000 7ff8d7e0b640 1 -- 172.21.15.33:0/2441165827 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8d019ffa0 con 0x7ff8d0102340 2024-03-19T02:51:17.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.184+0000 7ff8d7e0b640 1 -- 172.21.15.33:0/2441165827 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8d00fd820 con 0x7ff8d0102340 2024-03-19T02:51:17.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.186+0000 7ff8ceffd640 1 -- 172.21.15.33:0/2441165827 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff8c8018cd0 con 0x7ff8d0102340 2024-03-19T02:51:17.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.186+0000 7ff8ceffd640 1 --2- 172.21.15.33:0/2441165827 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff8b80a2800 0x7ff8b80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:17.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.186+0000 7ff8d53a6640 1 --2- 172.21.15.33:0/2441165827 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff8b80a2800 0x7ff8b80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:17.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.186+0000 7ff8ceffd640 1 -- 172.21.15.33:0/2441165827 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7ff8c80cf390 con 0x7ff8d0102340 2024-03-19T02:51:17.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.187+0000 7ff8d53a6640 1 --2- 172.21.15.33:0/2441165827 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff8b80a2800 0x7ff8b80a4cc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff8d0199050 tx=0x7ff8c4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:17.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.191+0000 7ff8ceffd640 1 -- 172.21.15.33:0/2441165827 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff8c809c1f0 con 0x7ff8d0102340 2024-03-19T02:51:17.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:17.413+0000 7ff8d7e0b640 1 -- 172.21.15.33:0/2441165827 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7ff8d002d1f0 con 0x7ff8d0102340 2024-03-19T02:51:17.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:17 smithi096 ceph-mon[35946]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-19T02:51:17.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:17 smithi096 ceph-mon[35946]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T02:51:17.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:17 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1391833397' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-19T02:51:17.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:17 smithi155 ceph-mon[35804]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-19T02:51:17.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:17 smithi155 ceph-mon[35804]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T02:51:17.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:17 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1391833397' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-19T02:51:17.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:17 smithi033 ceph-mon[31973]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-19T02:51:17.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:17 smithi033 ceph-mon[31973]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T02:51:17.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:17 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1391833397' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-19T02:51:18.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.064+0000 7ff8ceffd640 1 -- 172.21.15.33:0/2441165827 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) ==== 121+0+0 (secure 0 0 0) 0x7ff8d002d1f0 con 0x7ff8d0102340 2024-03-19T02:51:18.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.066+0000 7ff8d7e0b640 1 -- 172.21.15.33:0/2441165827 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff8b80a2800 msgr2=0x7ff8b80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:18.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.066+0000 7ff8d7e0b640 1 --2- 172.21.15.33:0/2441165827 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff8b80a2800 0x7ff8b80a4cc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff8d0199050 tx=0x7ff8c4008040 comp rx=0 tx=0).stop 2024-03-19T02:51:18.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.066+0000 7ff8d7e0b640 1 -- 172.21.15.33:0/2441165827 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8d0102340 msgr2=0x7ff8d0197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:18.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.066+0000 7ff8d7e0b640 1 --2- 172.21.15.33:0/2441165827 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8d0102340 0x7ff8d0197b40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff8c800ebc0 tx=0x7ff8c800c6a0 comp rx=0 tx=0).stop 2024-03-19T02:51:18.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.067+0000 7ff8d7e0b640 1 -- 172.21.15.33:0/2441165827 shutdown_connections 2024-03-19T02:51:18.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.067+0000 7ff8d7e0b640 1 --2- 172.21.15.33:0/2441165827 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff8b80a2800 0x7ff8b80a4cc0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:18.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.067+0000 7ff8d7e0b640 1 --2- 172.21.15.33:0/2441165827 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8d01075a0 0x7ff8d019f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:18.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.067+0000 7ff8d7e0b640 1 --2- 172.21.15.33:0/2441165827 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff8d0104c70 0x7ff8d0198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:18.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.067+0000 7ff8d7e0b640 1 --2- 172.21.15.33:0/2441165827 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff8d0102340 0x7ff8d0197b40 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:18.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.067+0000 7ff8d7e0b640 1 -- 172.21.15.33:0/2441165827 >> 172.21.15.33:0/2441165827 conn(0x7ff8d00fbd30 msgr2=0x7ff8d0102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:18.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.067+0000 7ff8d7e0b640 1 -- 172.21.15.33:0/2441165827 shutdown_connections 2024-03-19T02:51:18.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.067+0000 7ff8d7e0b640 1 -- 172.21.15.33:0/2441165827 wait complete. 2024-03-19T02:51:18.085 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-19T02:51:18.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.195+0000 7f8bd679d640 1 -- 172.21.15.33:0/2058849018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bd0101830 msgr2=0x7f8bd0101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:18.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.195+0000 7f8bd679d640 1 --2- 172.21.15.33:0/2058849018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bd0101830 0x7f8bd0101c50 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8bc0009900 tx=0x7f8bc002bc30 comp rx=0 tx=0).stop 2024-03-19T02:51:18.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.195+0000 7f8bd679d640 1 -- 172.21.15.33:0/2058849018 shutdown_connections 2024-03-19T02:51:18.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.195+0000 7f8bd679d640 1 --2- 172.21.15.33:0/2058849018 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bd01033e0 0x7f8bd0109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:18.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.195+0000 7f8bd679d640 1 --2- 172.21.15.33:0/2058849018 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bd0102a20 0x7f8bd0102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:18.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.195+0000 7f8bd679d640 1 --2- 172.21.15.33:0/2058849018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bd0101830 0x7f8bd0101c50 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:18.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.195+0000 7f8bd679d640 1 -- 172.21.15.33:0/2058849018 >> 172.21.15.33:0/2058849018 conn(0x7f8bd00fcdc0 msgr2=0x7f8bd00ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:18.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.195+0000 7f8bd679d640 1 -- 172.21.15.33:0/2058849018 shutdown_connections 2024-03-19T02:51:18.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.195+0000 7f8bd679d640 1 -- 172.21.15.33:0/2058849018 wait complete. 2024-03-19T02:51:18.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.196+0000 7f8bd679d640 1 Processor -- start 2024-03-19T02:51:18.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.196+0000 7f8bd679d640 1 -- start start 2024-03-19T02:51:18.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.196+0000 7f8bd679d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bd0101830 0x7f8bd0197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:18.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.196+0000 7f8bd679d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bd0102a20 0x7f8bd0197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:18.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.196+0000 7f8bd679d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bd01033e0 0x7f8bd019f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:18.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.196+0000 7f8bd679d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd010d360 con 0x7f8bd0101830 2024-03-19T02:51:18.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.196+0000 7f8bd679d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd010d1e0 con 0x7f8bd01033e0 2024-03-19T02:51:18.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.196+0000 7f8bd679d640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd010d4e0 con 0x7f8bd0102a20 2024-03-19T02:51:18.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.196+0000 7f8bcffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bd0101830 0x7f8bd0197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:18.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.196+0000 7f8bcffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bd0101830 0x7f8bd0197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49976/0 (socket says 172.21.15.33:49976) 2024-03-19T02:51:18.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.196+0000 7f8bcffff640 1 -- 172.21.15.33:0/2554887923 learned_addr learned my addr 172.21.15.33:0/2554887923 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:18.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.196+0000 7f8bd4d3a640 1 --2- 172.21.15.33:0/2554887923 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bd01033e0 0x7f8bd019f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:18.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.196+0000 7f8bcf7fe640 1 --2- 172.21.15.33:0/2554887923 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bd0102a20 0x7f8bd0197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:18.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.197+0000 7f8bcffff640 1 -- 172.21.15.33:0/2554887923 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bd0102a20 msgr2=0x7f8bd0197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:18.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.197+0000 7f8bcffff640 1 --2- 172.21.15.33:0/2554887923 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bd0102a20 0x7f8bd0197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:18.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.197+0000 7f8bcffff640 1 -- 172.21.15.33:0/2554887923 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bd01033e0 msgr2=0x7f8bd019f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:18.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.197+0000 7f8bcffff640 1 --2- 172.21.15.33:0/2554887923 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bd01033e0 0x7f8bd019f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:18.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.197+0000 7f8bcffff640 1 -- 172.21.15.33:0/2554887923 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd019f6c0 con 0x7f8bd0101830 2024-03-19T02:51:18.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.197+0000 7f8bcffff640 1 --2- 172.21.15.33:0/2554887923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bd0101830 0x7f8bd0197aa0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8bc0000c00 tx=0x7f8bc0004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:18.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.197+0000 7f8bcd7fa640 1 -- 172.21.15.33:0/2554887923 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc0043020 con 0x7f8bd0101830 2024-03-19T02:51:18.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.197+0000 7f8bcd7fa640 1 -- 172.21.15.33:0/2554887923 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8bc0004440 con 0x7f8bd0101830 2024-03-19T02:51:18.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.197+0000 7f8bd679d640 1 -- 172.21.15.33:0/2554887923 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bd019f950 con 0x7f8bd0101830 2024-03-19T02:51:18.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.197+0000 7f8bcd7fa640 1 -- 172.21.15.33:0/2554887923 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc0003c90 con 0x7f8bd0101830 2024-03-19T02:51:18.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.197+0000 7f8bd679d640 1 -- 172.21.15.33:0/2554887923 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bd019fef0 con 0x7f8bd0101830 2024-03-19T02:51:18.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.198+0000 7f8bd679d640 1 -- 172.21.15.33:0/2554887923 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bd0108c70 con 0x7f8bd0101830 2024-03-19T02:51:18.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.199+0000 7f8bcd7fa640 1 -- 172.21.15.33:0/2554887923 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f8bc0046020 con 0x7f8bd0101830 2024-03-19T02:51:18.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.199+0000 7f8bcd7fa640 1 --2- 172.21.15.33:0/2554887923 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8bb80a2880 0x7f8bb80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:18.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.200+0000 7f8bcf7fe640 1 --2- 172.21.15.33:0/2554887923 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8bb80a2880 0x7f8bb80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:18.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.200+0000 7f8bcd7fa640 1 -- 172.21.15.33:0/2554887923 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7f8bc00f0df0 con 0x7f8bd0101830 2024-03-19T02:51:18.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.200+0000 7f8bcf7fe640 1 --2- 172.21.15.33:0/2554887923 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8bb80a2880 0x7f8bb80a4d40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8bd0198fb0 tx=0x7f8bc4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:18.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.204+0000 7f8bcd7fa640 1 -- 172.21.15.33:0/2554887923 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8bc004d020 con 0x7f8bd0101830 2024-03-19T02:51:18.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:18.428+0000 7f8bd679d640 1 -- 172.21.15.33:0/2554887923 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f8bd0062420 con 0x7f8bd0101830 2024-03-19T02:51:18.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:18 smithi096 ceph-mon[35946]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-19T02:51:18.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:18 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1391833397' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-19T02:51:18.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:18 smithi096 ceph-mon[35946]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-19T02:51:18.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:18 smithi096 ceph-mon[35946]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T02:51:18.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:18 smithi096 ceph-mon[35946]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-03-19T02:51:18.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:18 smithi096 ceph-mon[35946]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-19T02:51:18.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:18 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2441165827' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-19T02:51:18.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:18 smithi096 ceph-mon[35946]: pgmap v312: 129 pgs: 129 active+clean; 581 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-03-19T02:51:18.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:18 smithi155 ceph-mon[35804]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-19T02:51:18.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:18 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1391833397' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-19T02:51:18.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:18 smithi155 ceph-mon[35804]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-19T02:51:18.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:18 smithi155 ceph-mon[35804]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T02:51:18.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:18 smithi155 ceph-mon[35804]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-03-19T02:51:18.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:18 smithi155 ceph-mon[35804]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-19T02:51:18.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:18 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2441165827' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-19T02:51:18.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:18 smithi155 ceph-mon[35804]: pgmap v312: 129 pgs: 129 active+clean; 581 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-03-19T02:51:18.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:18 smithi033 ceph-mon[31973]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-19T02:51:18.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:18 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1391833397' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-19T02:51:18.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:18 smithi033 ceph-mon[31973]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-19T02:51:18.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:18 smithi033 ceph-mon[31973]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T02:51:18.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:18 smithi033 ceph-mon[31973]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-03-19T02:51:18.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:18 smithi033 ceph-mon[31973]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-19T02:51:18.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:18 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2441165827' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-19T02:51:18.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:18 smithi033 ceph-mon[31973]: pgmap v312: 129 pgs: 129 active+clean; 581 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-03-19T02:51:19.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.071+0000 7f8bcd7fa640 1 -- 172.21.15.33:0/2554887923 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) ==== 115+0+0 (secure 0 0 0) 0x7f8bc00bdc50 con 0x7f8bd0101830 2024-03-19T02:51:19.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.074+0000 7f8bd679d640 1 -- 172.21.15.33:0/2554887923 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8bb80a2880 msgr2=0x7f8bb80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:19.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.074+0000 7f8bd679d640 1 --2- 172.21.15.33:0/2554887923 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8bb80a2880 0x7f8bb80a4d40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8bd0198fb0 tx=0x7f8bc4005f90 comp rx=0 tx=0).stop 2024-03-19T02:51:19.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.074+0000 7f8bd679d640 1 -- 172.21.15.33:0/2554887923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bd0101830 msgr2=0x7f8bd0197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:19.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.074+0000 7f8bd679d640 1 --2- 172.21.15.33:0/2554887923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bd0101830 0x7f8bd0197aa0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8bc0000c00 tx=0x7f8bc0004040 comp rx=0 tx=0).stop 2024-03-19T02:51:19.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.074+0000 7f8bd679d640 1 -- 172.21.15.33:0/2554887923 shutdown_connections 2024-03-19T02:51:19.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.074+0000 7f8bd679d640 1 --2- 172.21.15.33:0/2554887923 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8bb80a2880 0x7f8bb80a4d40 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:19.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.074+0000 7f8bd679d640 1 --2- 172.21.15.33:0/2554887923 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bd0102a20 0x7f8bd0197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:19.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.074+0000 7f8bd679d640 1 --2- 172.21.15.33:0/2554887923 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bd01033e0 0x7f8bd019f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:19.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.074+0000 7f8bd679d640 1 --2- 172.21.15.33:0/2554887923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8bd0101830 0x7f8bd0197aa0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:19.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.074+0000 7f8bd679d640 1 -- 172.21.15.33:0/2554887923 >> 172.21.15.33:0/2554887923 conn(0x7f8bd00fcdc0 msgr2=0x7f8bd0105e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:19.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.075+0000 7f8bd679d640 1 -- 172.21.15.33:0/2554887923 shutdown_connections 2024-03-19T02:51:19.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.075+0000 7f8bd679d640 1 -- 172.21.15.33:0/2554887923 wait complete. 2024-03-19T02:51:19.090 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-03-19T02:51:19.091 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-19T02:51:19.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.202+0000 7f283b7d4640 1 -- 172.21.15.33:0/1476714584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28341022d0 msgr2=0x7f2834108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:19.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.202+0000 7f283b7d4640 1 --2- 172.21.15.33:0/1476714584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28341022d0 0x7f2834108b60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f282c009be0 tx=0x7f282c02bef0 comp rx=0 tx=0).stop 2024-03-19T02:51:19.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.202+0000 7f283b7d4640 1 -- 172.21.15.33:0/1476714584 shutdown_connections 2024-03-19T02:51:19.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.202+0000 7f283b7d4640 1 --2- 172.21.15.33:0/1476714584 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2834101910 0x7f2834101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:19.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.202+0000 7f283b7d4640 1 --2- 172.21.15.33:0/1476714584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f28341022d0 0x7f2834108b60 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:19.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.202+0000 7f283b7d4640 1 --2- 172.21.15.33:0/1476714584 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2834100720 0x7f2834100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:19.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.202+0000 7f283b7d4640 1 -- 172.21.15.33:0/1476714584 >> 172.21.15.33:0/1476714584 conn(0x7f28340fbcb0 msgr2=0x7f28340fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:19.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.202+0000 7f283b7d4640 1 -- 172.21.15.33:0/1476714584 shutdown_connections 2024-03-19T02:51:19.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.202+0000 7f283b7d4640 1 -- 172.21.15.33:0/1476714584 wait complete. 2024-03-19T02:51:19.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.202+0000 7f283b7d4640 1 Processor -- start 2024-03-19T02:51:19.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.202+0000 7f283b7d4640 1 -- start start 2024-03-19T02:51:19.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.203+0000 7f283b7d4640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2834100720 0x7f283410b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:19.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.203+0000 7f283b7d4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2834101910 0x7f28341096e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:19.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.203+0000 7f283b7d4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28341022d0 0x7f2834109c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:19.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.203+0000 7f283b7d4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f283410d330 con 0x7f28341022d0 2024-03-19T02:51:19.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.203+0000 7f283b7d4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f283410d1b0 con 0x7f2834101910 2024-03-19T02:51:19.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.203+0000 7f283b7d4640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f283410d4b0 con 0x7f2834100720 2024-03-19T02:51:19.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.203+0000 7f2839d71640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28341022d0 0x7f2834109c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:19.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.203+0000 7f2839d71640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28341022d0 0x7f2834109c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49980/0 (socket says 172.21.15.33:49980) 2024-03-19T02:51:19.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.203+0000 7f2839d71640 1 -- 172.21.15.33:0/3697464158 learned_addr learned my addr 172.21.15.33:0/3697464158 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:19.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.203+0000 7f2839570640 1 --2- 172.21.15.33:0/3697464158 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2834100720 0x7f283410b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:19.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.203+0000 7f2838d6f640 1 --2- 172.21.15.33:0/3697464158 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2834101910 0x7f28341096e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:19.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.203+0000 7f2839d71640 1 -- 172.21.15.33:0/3697464158 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2834100720 msgr2=0x7f283410b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:19.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.203+0000 7f2839d71640 1 --2- 172.21.15.33:0/3697464158 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2834100720 0x7f283410b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:19.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.203+0000 7f2839d71640 1 -- 172.21.15.33:0/3697464158 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2834101910 msgr2=0x7f28341096e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:19.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.203+0000 7f2839d71640 1 --2- 172.21.15.33:0/3697464158 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2834101910 0x7f28341096e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:19.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.203+0000 7f2839d71640 1 -- 172.21.15.33:0/3697464158 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f283410a4b0 con 0x7f28341022d0 2024-03-19T02:51:19.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.203+0000 7f2839570640 1 --2- 172.21.15.33:0/3697464158 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2834100720 0x7f283410b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:19.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.203+0000 7f2838d6f640 1 --2- 172.21.15.33:0/3697464158 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2834101910 0x7f28341096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:19.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.203+0000 7f2839d71640 1 --2- 172.21.15.33:0/3697464158 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28341022d0 0x7f2834109c20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f282c004af0 tx=0x7f282c033c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:19.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.203+0000 7f282a7fc640 1 -- 172.21.15.33:0/3697464158 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f282c043020 con 0x7f28341022d0 2024-03-19T02:51:19.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.203+0000 7f283b7d4640 1 -- 172.21.15.33:0/3697464158 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28341a1b60 con 0x7f28341022d0 2024-03-19T02:51:19.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.203+0000 7f282a7fc640 1 -- 172.21.15.33:0/3697464158 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f282c02ca60 con 0x7f28341022d0 2024-03-19T02:51:19.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.204+0000 7f282a7fc640 1 -- 172.21.15.33:0/3697464158 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f282c034430 con 0x7f28341022d0 2024-03-19T02:51:19.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.204+0000 7f283b7d4640 1 -- 172.21.15.33:0/3697464158 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28341a20b0 con 0x7f28341022d0 2024-03-19T02:51:19.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.204+0000 7f283b7d4640 1 -- 172.21.15.33:0/3697464158 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2834107b30 con 0x7f28341022d0 2024-03-19T02:51:19.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.206+0000 7f282a7fc640 1 -- 172.21.15.33:0/3697464158 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f282c046020 con 0x7f28341022d0 2024-03-19T02:51:19.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.206+0000 7f282a7fc640 1 --2- 172.21.15.33:0/3697464158 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f28200a2800 0x7f28200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:19.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.206+0000 7f282a7fc640 1 -- 172.21.15.33:0/3697464158 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7f282c0f1b30 con 0x7f28341022d0 2024-03-19T02:51:19.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.206+0000 7f2839570640 1 --2- 172.21.15.33:0/3697464158 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f28200a2800 0x7f28200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:19.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.207+0000 7f2839570640 1 --2- 172.21.15.33:0/3697464158 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f28200a2800 0x7f28200a4cc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2834101770 tx=0x7f2830009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:19.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.210+0000 7f282a7fc640 1 -- 172.21.15.33:0/3697464158 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f282c04d020 con 0x7f28341022d0 2024-03-19T02:51:19.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.419+0000 7f283b7d4640 1 -- 172.21.15.33:0/3697464158 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f2834061c60 con 0x7f28200a2800 2024-03-19T02:51:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:19 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2441165827' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-19T02:51:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:19 smithi033 ceph-mon[31973]: mds.? [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] up:active 2024-03-19T02:51:19.426 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:19 smithi033 ceph-mon[31973]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-19T02:51:19.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:19 smithi033 ceph-mon[31973]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-19T02:51:19.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:19 smithi033 ceph-mon[31973]: Cluster is now healthy 2024-03-19T02:51:19.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:19 smithi033 ceph-mon[31973]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-03-19T02:51:19.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:19 smithi033 ceph-mon[31973]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-03-19T02:51:19.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:19 smithi033 ceph-mon[31973]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-19T02:51:19.427 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:19 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2554887923' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-19T02:51:19.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.437+0000 7f282a7fc640 1 -- 172.21.15.33:0/3697464158 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2834061c60 con 0x7f28200a2800 2024-03-19T02:51:19.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.439+0000 7f283b7d4640 1 -- 172.21.15.33:0/3697464158 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f28200a2800 msgr2=0x7f28200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:19.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.439+0000 7f283b7d4640 1 --2- 172.21.15.33:0/3697464158 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f28200a2800 0x7f28200a4cc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2834101770 tx=0x7f2830009260 comp rx=0 tx=0).stop 2024-03-19T02:51:19.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.439+0000 7f283b7d4640 1 -- 172.21.15.33:0/3697464158 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28341022d0 msgr2=0x7f2834109c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:19.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.439+0000 7f283b7d4640 1 --2- 172.21.15.33:0/3697464158 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28341022d0 0x7f2834109c20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f282c004af0 tx=0x7f282c033c90 comp rx=0 tx=0).stop 2024-03-19T02:51:19.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.439+0000 7f283b7d4640 1 -- 172.21.15.33:0/3697464158 shutdown_connections 2024-03-19T02:51:19.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.440+0000 7f283b7d4640 1 --2- 172.21.15.33:0/3697464158 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f28200a2800 0x7f28200a4cc0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:19.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.440+0000 7f283b7d4640 1 --2- 172.21.15.33:0/3697464158 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f28341022d0 0x7f2834109c20 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:19.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.440+0000 7f283b7d4640 1 --2- 172.21.15.33:0/3697464158 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2834101910 0x7f28341096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:19.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.440+0000 7f283b7d4640 1 --2- 172.21.15.33:0/3697464158 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2834100720 0x7f283410b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:19.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.440+0000 7f283b7d4640 1 -- 172.21.15.33:0/3697464158 >> 172.21.15.33:0/3697464158 conn(0x7f28340fbcb0 msgr2=0x7f2834104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:19.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.440+0000 7f283b7d4640 1 -- 172.21.15.33:0/3697464158 shutdown_connections 2024-03-19T02:51:19.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.440+0000 7f283b7d4640 1 -- 172.21.15.33:0/3697464158 wait complete. 2024-03-19T02:51:19.452 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-19T02:51:19.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:19 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2441165827' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-19T02:51:19.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:19 smithi096 ceph-mon[35946]: mds.? [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] up:active 2024-03-19T02:51:19.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:19 smithi096 ceph-mon[35946]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-19T02:51:19.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:19 smithi096 ceph-mon[35946]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-19T02:51:19.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:19 smithi096 ceph-mon[35946]: Cluster is now healthy 2024-03-19T02:51:19.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:19 smithi096 ceph-mon[35946]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-03-19T02:51:19.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:19 smithi096 ceph-mon[35946]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-03-19T02:51:19.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:19 smithi096 ceph-mon[35946]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-19T02:51:19.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:19 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2554887923' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-19T02:51:19.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:19 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2441165827' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-19T02:51:19.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:19 smithi155 ceph-mon[35804]: mds.? [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] up:active 2024-03-19T02:51:19.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:19 smithi155 ceph-mon[35804]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-19T02:51:19.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:19 smithi155 ceph-mon[35804]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-19T02:51:19.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:19 smithi155 ceph-mon[35804]: Cluster is now healthy 2024-03-19T02:51:19.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:19 smithi155 ceph-mon[35804]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-03-19T02:51:19.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:19 smithi155 ceph-mon[35804]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-03-19T02:51:19.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:19 smithi155 ceph-mon[35804]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-19T02:51:19.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:19 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2554887923' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-19T02:51:19.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.559+0000 7f1a58461640 1 -- 172.21.15.33:0/2275937590 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a50102a30 msgr2=0x7f1a50102eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:19.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.559+0000 7f1a58461640 1 --2- 172.21.15.33:0/2275937590 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a50102a30 0x7f1a50102eb0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1a40009900 tx=0x7f1a4002bc30 comp rx=0 tx=0).stop 2024-03-19T02:51:19.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.559+0000 7f1a58461640 1 -- 172.21.15.33:0/2275937590 shutdown_connections 2024-03-19T02:51:19.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.559+0000 7f1a58461640 1 --2- 172.21.15.33:0/2275937590 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a501033f0 0x7f1a50109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:19.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.559+0000 7f1a58461640 1 --2- 172.21.15.33:0/2275937590 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a50102a30 0x7f1a50102eb0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:19.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.559+0000 7f1a58461640 1 --2- 172.21.15.33:0/2275937590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a50101840 0x7f1a50101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:19.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.559+0000 7f1a58461640 1 -- 172.21.15.33:0/2275937590 >> 172.21.15.33:0/2275937590 conn(0x7f1a500fcd90 msgr2=0x7f1a500ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:19.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.559+0000 7f1a58461640 1 -- 172.21.15.33:0/2275937590 shutdown_connections 2024-03-19T02:51:19.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.559+0000 7f1a58461640 1 -- 172.21.15.33:0/2275937590 wait complete. 2024-03-19T02:51:19.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.560+0000 7f1a58461640 1 Processor -- start 2024-03-19T02:51:19.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.560+0000 7f1a58461640 1 -- start start 2024-03-19T02:51:19.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.560+0000 7f1a58461640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a50101840 0x7f1a500755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:19.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.560+0000 7f1a58461640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a50102a30 0x7f1a50073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:19.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.560+0000 7f1a58461640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a501033f0 0x7f1a50074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:19.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.560+0000 7f1a58461640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1a5010d350 con 0x7f1a50101840 2024-03-19T02:51:19.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.560+0000 7f1a58461640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1a5010d1d0 con 0x7f1a501033f0 2024-03-19T02:51:19.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.560+0000 7f1a58461640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1a5010d4d0 con 0x7f1a50102a30 2024-03-19T02:51:19.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.560+0000 7f1a561fd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a50101840 0x7f1a500755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:19.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.560+0000 7f1a561fd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a50101840 0x7f1a500755d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50034/0 (socket says 172.21.15.33:50034) 2024-03-19T02:51:19.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.560+0000 7f1a561fd640 1 -- 172.21.15.33:0/1023333488 learned_addr learned my addr 172.21.15.33:0/1023333488 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:19.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.560+0000 7f1a559fc640 1 --2- 172.21.15.33:0/1023333488 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a50102a30 0x7f1a50073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:19.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.560+0000 7f1a569fe640 1 --2- 172.21.15.33:0/1023333488 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a501033f0 0x7f1a50074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:19.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.560+0000 7f1a561fd640 1 -- 172.21.15.33:0/1023333488 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a50102a30 msgr2=0x7f1a50073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:19.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.560+0000 7f1a561fd640 1 --2- 172.21.15.33:0/1023333488 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a50102a30 0x7f1a50073c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:19.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.560+0000 7f1a561fd640 1 -- 172.21.15.33:0/1023333488 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a501033f0 msgr2=0x7f1a50074150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:19.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.560+0000 7f1a561fd640 1 --2- 172.21.15.33:0/1023333488 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a501033f0 0x7f1a50074150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:19.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.560+0000 7f1a561fd640 1 -- 172.21.15.33:0/1023333488 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a500749e0 con 0x7f1a50101840 2024-03-19T02:51:19.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.560+0000 7f1a569fe640 1 --2- 172.21.15.33:0/1023333488 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a501033f0 0x7f1a50074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:51:19.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.560+0000 7f1a561fd640 1 --2- 172.21.15.33:0/1023333488 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a50101840 0x7f1a500755d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c00e9c0 tx=0x7f1a4c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:19.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.561+0000 7f1a477fe640 1 -- 172.21.15.33:0/1023333488 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c00cd80 con 0x7f1a50101840 2024-03-19T02:51:19.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.561+0000 7f1a58461640 1 -- 172.21.15.33:0/1023333488 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a501a1bc0 con 0x7f1a50101840 2024-03-19T02:51:19.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.561+0000 7f1a477fe640 1 -- 172.21.15.33:0/1023333488 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a4c0035f0 con 0x7f1a50101840 2024-03-19T02:51:19.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.561+0000 7f1a477fe640 1 -- 172.21.15.33:0/1023333488 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c010690 con 0x7f1a50101840 2024-03-19T02:51:19.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.561+0000 7f1a58461640 1 -- 172.21.15.33:0/1023333488 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a501a20e0 con 0x7f1a50101840 2024-03-19T02:51:19.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.562+0000 7f1a58461640 1 -- 172.21.15.33:0/1023333488 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a50068a60 con 0x7f1a50101840 2024-03-19T02:51:19.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.563+0000 7f1a477fe640 1 -- 172.21.15.33:0/1023333488 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f1a4c0037a0 con 0x7f1a50101840 2024-03-19T02:51:19.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.563+0000 7f1a477fe640 1 --2- 172.21.15.33:0/1023333488 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f1a3c0a2800 0x7f1a3c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:19.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.564+0000 7f1a559fc640 1 --2- 172.21.15.33:0/1023333488 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f1a3c0a2800 0x7f1a3c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:19.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.564+0000 7f1a477fe640 1 -- 172.21.15.33:0/1023333488 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7f1a4c0cf120 con 0x7f1a50101840 2024-03-19T02:51:19.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.564+0000 7f1a559fc640 1 --2- 172.21.15.33:0/1023333488 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f1a3c0a2800 0x7f1a3c0a4cc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1a40009230 tx=0x7f1a40005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:19.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.568+0000 7f1a477fe640 1 -- 172.21.15.33:0/1023333488 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1a4c09bf80 con 0x7f1a50101840 2024-03-19T02:51:19.778 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:51:19 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:51:19.424+0000 7f0085cd5700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:51:19.778 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:51:19 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:51:19.424+0000 7f0085cd5700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:51:19.779 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:51:19 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:51:19.424+0000 7f0085cd5700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:51:19.779 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:51:19 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:51:19.424+0000 7f0085cd5700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:51:19.779 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:51:19 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:51:19.424+0000 7f0085cd5700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:51:19.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.778+0000 7f1a58461640 1 -- 172.21.15.33:0/1023333488 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f1a50061cf0 con 0x7f1a3c0a2800 2024-03-19T02:51:19.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.798+0000 7f1a477fe640 1 -- 172.21.15.33:0/1023333488 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f1a50061cf0 con 0x7f1a3c0a2800 2024-03-19T02:51:19.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.800+0000 7f1a58461640 1 -- 172.21.15.33:0/1023333488 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f1a3c0a2800 msgr2=0x7f1a3c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:19.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.800+0000 7f1a58461640 1 --2- 172.21.15.33:0/1023333488 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f1a3c0a2800 0x7f1a3c0a4cc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1a40009230 tx=0x7f1a40005fd0 comp rx=0 tx=0).stop 2024-03-19T02:51:19.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.800+0000 7f1a58461640 1 -- 172.21.15.33:0/1023333488 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a50101840 msgr2=0x7f1a500755d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:19.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.800+0000 7f1a58461640 1 --2- 172.21.15.33:0/1023333488 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a50101840 0x7f1a500755d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c00e9c0 tx=0x7f1a4c00ee80 comp rx=0 tx=0).stop 2024-03-19T02:51:19.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.800+0000 7f1a58461640 1 -- 172.21.15.33:0/1023333488 shutdown_connections 2024-03-19T02:51:19.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.800+0000 7f1a58461640 1 --2- 172.21.15.33:0/1023333488 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f1a3c0a2800 0x7f1a3c0a4cc0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:19.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.800+0000 7f1a58461640 1 --2- 172.21.15.33:0/1023333488 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a50102a30 0x7f1a50073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:19.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.800+0000 7f1a58461640 1 --2- 172.21.15.33:0/1023333488 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a501033f0 0x7f1a50074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:19.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.800+0000 7f1a58461640 1 --2- 172.21.15.33:0/1023333488 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a50101840 0x7f1a500755d0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:19.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.800+0000 7f1a58461640 1 -- 172.21.15.33:0/1023333488 >> 172.21.15.33:0/1023333488 conn(0x7f1a500fcd90 msgr2=0x7f1a50105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:19.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.800+0000 7f1a58461640 1 -- 172.21.15.33:0/1023333488 shutdown_connections 2024-03-19T02:51:19.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.800+0000 7f1a58461640 1 -- 172.21.15.33:0/1023333488 wait complete. 2024-03-19T02:51:19.812 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-19T02:51:19.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.918+0000 7f8e9bcac640 1 -- 172.21.15.33:0/2597290319 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e94101930 msgr2=0x7f8e94101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:19.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.918+0000 7f8e9bcac640 1 --2- 172.21.15.33:0/2597290319 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e94101930 0x7f8e94101db0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8e84009900 tx=0x7f8e8402bc30 comp rx=0 tx=0).stop 2024-03-19T02:51:19.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.918+0000 7f8e9bcac640 1 -- 172.21.15.33:0/2597290319 shutdown_connections 2024-03-19T02:51:19.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.918+0000 7f8e9bcac640 1 --2- 172.21.15.33:0/2597290319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e941022f0 0x7f8e94108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:19.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.918+0000 7f8e9bcac640 1 --2- 172.21.15.33:0/2597290319 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e94101930 0x7f8e94101db0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:19.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.918+0000 7f8e9bcac640 1 --2- 172.21.15.33:0/2597290319 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e94100740 0x7f8e94100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:19.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.918+0000 7f8e9bcac640 1 -- 172.21.15.33:0/2597290319 >> 172.21.15.33:0/2597290319 conn(0x7f8e940fbcd0 msgr2=0x7f8e940fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:19.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.918+0000 7f8e9bcac640 1 -- 172.21.15.33:0/2597290319 shutdown_connections 2024-03-19T02:51:19.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.918+0000 7f8e9bcac640 1 -- 172.21.15.33:0/2597290319 wait complete. 2024-03-19T02:51:19.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.919+0000 7f8e9bcac640 1 Processor -- start 2024-03-19T02:51:19.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.919+0000 7f8e9bcac640 1 -- start start 2024-03-19T02:51:19.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.919+0000 7f8e9bcac640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e94100740 0x7f8e94197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:19.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.919+0000 7f8e9bcac640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e94101930 0x7f8e94197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:19.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.919+0000 7f8e9bcac640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e941022f0 0x7f8e9419f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:19.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.919+0000 7f8e9bcac640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8e9410c250 con 0x7f8e941022f0 2024-03-19T02:51:19.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.919+0000 7f8e9bcac640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8e9410c0d0 con 0x7f8e94100740 2024-03-19T02:51:19.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.919+0000 7f8e9bcac640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8e9410c3d0 con 0x7f8e94101930 2024-03-19T02:51:19.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.919+0000 7f8e9a249640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e941022f0 0x7f8e9419f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:19.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.919+0000 7f8e99a48640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e94100740 0x7f8e94197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:19.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.919+0000 7f8e9a249640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e941022f0 0x7f8e9419f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50054/0 (socket says 172.21.15.33:50054) 2024-03-19T02:51:19.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.919+0000 7f8e9a249640 1 -- 172.21.15.33:0/2030375479 learned_addr learned my addr 172.21.15.33:0/2030375479 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:19.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.919+0000 7f8e99247640 1 --2- 172.21.15.33:0/2030375479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e94101930 0x7f8e94197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:19.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.919+0000 7f8e9a249640 1 -- 172.21.15.33:0/2030375479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e94101930 msgr2=0x7f8e94197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:19.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.919+0000 7f8e9a249640 1 --2- 172.21.15.33:0/2030375479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e94101930 0x7f8e94197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:19.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.919+0000 7f8e9a249640 1 -- 172.21.15.33:0/2030375479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e94100740 msgr2=0x7f8e94197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:19.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.919+0000 7f8e9a249640 1 --2- 172.21.15.33:0/2030375479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e94100740 0x7f8e94197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:19.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.919+0000 7f8e9a249640 1 -- 172.21.15.33:0/2030375479 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e9419f6c0 con 0x7f8e941022f0 2024-03-19T02:51:19.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.919+0000 7f8e99a48640 1 --2- 172.21.15.33:0/2030375479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e94100740 0x7f8e94197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:19.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.920+0000 7f8e9a249640 1 --2- 172.21.15.33:0/2030375479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e941022f0 0x7f8e9419f050 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c00eb10 tx=0x7f8e8c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:19.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.920+0000 7f8e8affd640 1 -- 172.21.15.33:0/2030375479 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e8c004b20 con 0x7f8e941022f0 2024-03-19T02:51:19.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.920+0000 7f8e8affd640 1 -- 172.21.15.33:0/2030375479 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e8c004cc0 con 0x7f8e941022f0 2024-03-19T02:51:19.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.920+0000 7f8e9bcac640 1 -- 172.21.15.33:0/2030375479 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e9419f9b0 con 0x7f8e941022f0 2024-03-19T02:51:19.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.920+0000 7f8e8affd640 1 -- 172.21.15.33:0/2030375479 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e8c010690 con 0x7f8e941022f0 2024-03-19T02:51:19.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.920+0000 7f8e9bcac640 1 -- 172.21.15.33:0/2030375479 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e9419fe80 con 0x7f8e941022f0 2024-03-19T02:51:19.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.921+0000 7f8e9bcac640 1 -- 172.21.15.33:0/2030375479 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e941a00b0 con 0x7f8e941022f0 2024-03-19T02:51:19.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.922+0000 7f8e8affd640 1 -- 172.21.15.33:0/2030375479 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f8e8c005020 con 0x7f8e941022f0 2024-03-19T02:51:19.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.922+0000 7f8e8affd640 1 --2- 172.21.15.33:0/2030375479 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8e800a2800 0x7f8e800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:19.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.922+0000 7f8e8affd640 1 -- 172.21.15.33:0/2030375479 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7f8e8c0cf230 con 0x7f8e941022f0 2024-03-19T02:51:19.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.922+0000 7f8e99a48640 1 --2- 172.21.15.33:0/2030375479 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8e800a2800 0x7f8e800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:19.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.923+0000 7f8e99a48640 1 --2- 172.21.15.33:0/2030375479 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8e800a2800 0x7f8e800a4cc0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8e94101790 tx=0x7f8e90005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:19.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:19.927+0000 7f8e8affd640 1 -- 172.21.15.33:0/2030375479 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8e8c09c090 con 0x7f8e941022f0 2024-03-19T02:51:20.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.146+0000 7f8e9bcac640 1 -- 172.21.15.33:0/2030375479 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f8e9406a570 con 0x7f8e800a2800 2024-03-19T02:51:20.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.150+0000 7f8e8affd640 1 -- 172.21.15.33:0/2030375479 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f8e9406a570 con 0x7f8e800a2800 2024-03-19T02:51:20.151 INFO:teuthology.orchestra.run.smithi033.stdout:/volumes/qa/sv_0/14730696-c69f-4c28-ba34-e177c7403ad4 2024-03-19T02:51:20.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.152+0000 7f8e9bcac640 1 -- 172.21.15.33:0/2030375479 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8e800a2800 msgr2=0x7f8e800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:20.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.152+0000 7f8e9bcac640 1 --2- 172.21.15.33:0/2030375479 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8e800a2800 0x7f8e800a4cc0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8e94101790 tx=0x7f8e90005f90 comp rx=0 tx=0).stop 2024-03-19T02:51:20.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.152+0000 7f8e9bcac640 1 -- 172.21.15.33:0/2030375479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e941022f0 msgr2=0x7f8e9419f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:20.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.152+0000 7f8e9bcac640 1 --2- 172.21.15.33:0/2030375479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e941022f0 0x7f8e9419f050 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c00eb10 tx=0x7f8e8c00c2d0 comp rx=0 tx=0).stop 2024-03-19T02:51:20.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.152+0000 7f8e9bcac640 1 -- 172.21.15.33:0/2030375479 shutdown_connections 2024-03-19T02:51:20.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.152+0000 7f8e9bcac640 1 --2- 172.21.15.33:0/2030375479 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8e800a2800 0x7f8e800a4cc0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:20.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.152+0000 7f8e9bcac640 1 --2- 172.21.15.33:0/2030375479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8e94101930 0x7f8e94197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:20.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.152+0000 7f8e9bcac640 1 --2- 172.21.15.33:0/2030375479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8e941022f0 0x7f8e9419f050 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:20.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.152+0000 7f8e9bcac640 1 --2- 172.21.15.33:0/2030375479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e94100740 0x7f8e94197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:20.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.152+0000 7f8e9bcac640 1 -- 172.21.15.33:0/2030375479 >> 172.21.15.33:0/2030375479 conn(0x7f8e940fbcd0 msgr2=0x7f8e94069e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:20.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.152+0000 7f8e9bcac640 1 -- 172.21.15.33:0/2030375479 shutdown_connections 2024-03-19T02:51:20.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.152+0000 7f8e9bcac640 1 -- 172.21.15.33:0/2030375479 wait complete. 2024-03-19T02:51:20.164 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-19T02:51:20.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.270+0000 7f0967c8a640 1 -- 172.21.15.33:0/2773540208 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0960101850 msgr2=0x7f0960101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:20.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.270+0000 7f0967c8a640 1 --2- 172.21.15.33:0/2773540208 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0960101850 0x7f0960101c70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0958009900 tx=0x7f095802bc30 comp rx=0 tx=0).stop 2024-03-19T02:51:20.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.270+0000 7f0967c8a640 1 -- 172.21.15.33:0/2773540208 shutdown_connections 2024-03-19T02:51:20.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.270+0000 7f0967c8a640 1 --2- 172.21.15.33:0/2773540208 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0960103400 0x7f0960109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:20.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.270+0000 7f0967c8a640 1 --2- 172.21.15.33:0/2773540208 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0960102a40 0x7f0960102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:20.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.270+0000 7f0967c8a640 1 --2- 172.21.15.33:0/2773540208 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0960101850 0x7f0960101c70 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:20.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.270+0000 7f0967c8a640 1 -- 172.21.15.33:0/2773540208 >> 172.21.15.33:0/2773540208 conn(0x7f09600fcdc0 msgr2=0x7f09600ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:20.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.270+0000 7f0967c8a640 1 -- 172.21.15.33:0/2773540208 shutdown_connections 2024-03-19T02:51:20.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.270+0000 7f0967c8a640 1 -- 172.21.15.33:0/2773540208 wait complete. 2024-03-19T02:51:20.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.271+0000 7f0967c8a640 1 Processor -- start 2024-03-19T02:51:20.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.271+0000 7f0967c8a640 1 -- start start 2024-03-19T02:51:20.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.271+0000 7f0967c8a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0960101850 0x7f0960197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:20.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.271+0000 7f0967c8a640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0960102a40 0x7f0960197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:20.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.271+0000 7f0967c8a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0960103400 0x7f096019f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:20.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.271+0000 7f0967c8a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f096010d380 con 0x7f0960103400 2024-03-19T02:51:20.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.271+0000 7f0967c8a640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f096010d200 con 0x7f0960101850 2024-03-19T02:51:20.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.271+0000 7f0967c8a640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f096010d500 con 0x7f0960102a40 2024-03-19T02:51:20.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.271+0000 7f0966227640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0960103400 0x7f096019f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:20.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.271+0000 7f0966227640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0960103400 0x7f096019f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50072/0 (socket says 172.21.15.33:50072) 2024-03-19T02:51:20.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.271+0000 7f0966227640 1 -- 172.21.15.33:0/375940493 learned_addr learned my addr 172.21.15.33:0/375940493 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:20.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.271+0000 7f0965a26640 1 --2- 172.21.15.33:0/375940493 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0960101850 0x7f0960197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:20.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.271+0000 7f0965225640 1 --2- 172.21.15.33:0/375940493 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0960102a40 0x7f0960197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:20.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.272+0000 7f0966227640 1 -- 172.21.15.33:0/375940493 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0960102a40 msgr2=0x7f0960197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:20.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.272+0000 7f0966227640 1 --2- 172.21.15.33:0/375940493 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0960102a40 0x7f0960197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:20.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.272+0000 7f0966227640 1 -- 172.21.15.33:0/375940493 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0960101850 msgr2=0x7f0960197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:20.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.272+0000 7f0966227640 1 --2- 172.21.15.33:0/375940493 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0960101850 0x7f0960197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:20.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.272+0000 7f0966227640 1 -- 172.21.15.33:0/375940493 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f096019f6a0 con 0x7f0960103400 2024-03-19T02:51:20.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.272+0000 7f0966227640 1 --2- 172.21.15.33:0/375940493 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0960103400 0x7f096019f030 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f095000eb10 tx=0x7f095000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:20.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.272+0000 7f095effd640 1 -- 172.21.15.33:0/375940493 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0950004b20 con 0x7f0960103400 2024-03-19T02:51:20.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.272+0000 7f095effd640 1 -- 172.21.15.33:0/375940493 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0950004cc0 con 0x7f0960103400 2024-03-19T02:51:20.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.272+0000 7f0967c8a640 1 -- 172.21.15.33:0/375940493 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f096019f990 con 0x7f0960103400 2024-03-19T02:51:20.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.272+0000 7f095effd640 1 -- 172.21.15.33:0/375940493 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0950010690 con 0x7f0960103400 2024-03-19T02:51:20.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.272+0000 7f0967c8a640 1 -- 172.21.15.33:0/375940493 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f096019ff30 con 0x7f0960103400 2024-03-19T02:51:20.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.273+0000 7f0967c8a640 1 -- 172.21.15.33:0/375940493 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0960108c10 con 0x7f0960103400 2024-03-19T02:51:20.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.274+0000 7f095effd640 1 -- 172.21.15.33:0/375940493 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f0950005020 con 0x7f0960103400 2024-03-19T02:51:20.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.274+0000 7f095effd640 1 --2- 172.21.15.33:0/375940493 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f09480a2880 0x7f09480a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:20.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.275+0000 7f0965a26640 1 --2- 172.21.15.33:0/375940493 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f09480a2880 0x7f09480a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:20.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.275+0000 7f095effd640 1 -- 172.21.15.33:0/375940493 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7f09500cf2e0 con 0x7f0960103400 2024-03-19T02:51:20.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.275+0000 7f0965a26640 1 --2- 172.21.15.33:0/375940493 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f09480a2880 0x7f09480a4d40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0958005ab0 tx=0x7f0958005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:20.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.279+0000 7f095effd640 1 -- 172.21.15.33:0/375940493 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f095009c140 con 0x7f0960103400 2024-03-19T02:51:20.417 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:20 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2554887923' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-19T02:51:20.417 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:20 smithi155 ceph-mon[35804]: mds.? [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] up:active 2024-03-19T02:51:20.417 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:20 smithi155 ceph-mon[35804]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-03-19T02:51:20.417 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:20 smithi155 ceph-mon[35804]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-19T02:51:20.417 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:20 smithi155 ceph-mon[35804]: from='client.14862 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:51:20.417 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:20 smithi155 ceph-mon[35804]: pgmap v313: 129 pgs: 129 active+clean; 588 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 1023 B/s wr, 3 op/s 2024-03-19T02:51:20.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:20 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2554887923' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-19T02:51:20.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:20 smithi096 ceph-mon[35946]: mds.? [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] up:active 2024-03-19T02:51:20.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:20 smithi096 ceph-mon[35946]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-03-19T02:51:20.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:20 smithi096 ceph-mon[35946]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-19T02:51:20.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:20 smithi096 ceph-mon[35946]: from='client.14862 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:51:20.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:20 smithi096 ceph-mon[35946]: pgmap v313: 129 pgs: 129 active+clean; 588 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 1023 B/s wr, 3 op/s 2024-03-19T02:51:20.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.491+0000 7f0967c8a640 1 -- 172.21.15.33:0/375940493 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f0960061c60 con 0x7f09480a2880 2024-03-19T02:51:20.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.514+0000 7f095effd640 1 -- 172.21.15.33:0/375940493 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f0960061c60 con 0x7f09480a2880 2024-03-19T02:51:20.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.516+0000 7f0967c8a640 1 -- 172.21.15.33:0/375940493 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f09480a2880 msgr2=0x7f09480a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:20.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.516+0000 7f0967c8a640 1 --2- 172.21.15.33:0/375940493 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f09480a2880 0x7f09480a4d40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0958005ab0 tx=0x7f0958005fd0 comp rx=0 tx=0).stop 2024-03-19T02:51:20.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.516+0000 7f0967c8a640 1 -- 172.21.15.33:0/375940493 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0960103400 msgr2=0x7f096019f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:20.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.516+0000 7f0967c8a640 1 --2- 172.21.15.33:0/375940493 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0960103400 0x7f096019f030 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f095000eb10 tx=0x7f095000c2d0 comp rx=0 tx=0).stop 2024-03-19T02:51:20.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.516+0000 7f0967c8a640 1 -- 172.21.15.33:0/375940493 shutdown_connections 2024-03-19T02:51:20.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.516+0000 7f0967c8a640 1 --2- 172.21.15.33:0/375940493 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f09480a2880 0x7f09480a4d40 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:20.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.516+0000 7f0967c8a640 1 --2- 172.21.15.33:0/375940493 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0960102a40 0x7f0960197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:20.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.516+0000 7f0967c8a640 1 --2- 172.21.15.33:0/375940493 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0960103400 0x7f096019f030 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:20.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.516+0000 7f0967c8a640 1 --2- 172.21.15.33:0/375940493 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0960101850 0x7f0960197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:20.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.516+0000 7f0967c8a640 1 -- 172.21.15.33:0/375940493 >> 172.21.15.33:0/375940493 conn(0x7f09600fcdc0 msgr2=0x7f0960105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:20.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.516+0000 7f0967c8a640 1 -- 172.21.15.33:0/375940493 shutdown_connections 2024-03-19T02:51:20.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.516+0000 7f0967c8a640 1 -- 172.21.15.33:0/375940493 wait complete. 2024-03-19T02:51:20.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:20 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2554887923' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-19T02:51:20.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:20 smithi033 ceph-mon[31973]: mds.? [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] up:active 2024-03-19T02:51:20.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:20 smithi033 ceph-mon[31973]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-03-19T02:51:20.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:20 smithi033 ceph-mon[31973]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-19T02:51:20.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:20 smithi033 ceph-mon[31973]: from='client.14862 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:51:20.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:20 smithi033 ceph-mon[31973]: pgmap v313: 129 pgs: 129 active+clean; 588 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 1023 B/s wr, 3 op/s 2024-03-19T02:51:20.528 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-19T02:51:20.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.635+0000 7f8a7ec44640 1 -- 172.21.15.33:0/3548947974 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a78101ce0 msgr2=0x7f8a78102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:20.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.635+0000 7f8a7ec44640 1 --2- 172.21.15.33:0/3548947974 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a78101ce0 0x7f8a78102100 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8a70009ca0 tx=0x7f8a7002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:51:20.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.635+0000 7f8a7ec44640 1 -- 172.21.15.33:0/3548947974 shutdown_connections 2024-03-19T02:51:20.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.635+0000 7f8a7ec44640 1 --2- 172.21.15.33:0/3548947974 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a78100a10 0x7f8a78100e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:20.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.635+0000 7f8a7ec44640 1 --2- 172.21.15.33:0/3548947974 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a781013d0 0x7f8a78108c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:20.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.635+0000 7f8a7ec44640 1 --2- 172.21.15.33:0/3548947974 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a78101ce0 0x7f8a78102100 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:20.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.635+0000 7f8a7ec44640 1 -- 172.21.15.33:0/3548947974 >> 172.21.15.33:0/3548947974 conn(0x7f8a780fb420 msgr2=0x7f8a780fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:20.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.635+0000 7f8a7ec44640 1 -- 172.21.15.33:0/3548947974 shutdown_connections 2024-03-19T02:51:20.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.635+0000 7f8a7ec44640 1 -- 172.21.15.33:0/3548947974 wait complete. 2024-03-19T02:51:20.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.636+0000 7f8a7ec44640 1 Processor -- start 2024-03-19T02:51:20.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.636+0000 7f8a7ec44640 1 -- start start 2024-03-19T02:51:20.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.636+0000 7f8a7ec44640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a78100a10 0x7f8a78199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:20.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.636+0000 7f8a7ec44640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a781013d0 0x7f8a7819a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:20.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.636+0000 7f8a7ec44640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a78101ce0 0x7f8a781a1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:20.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.636+0000 7f8a7ec44640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7810af40 con 0x7f8a78101ce0 2024-03-19T02:51:20.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.636+0000 7f8a7ec44640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7810adc0 con 0x7f8a78100a10 2024-03-19T02:51:20.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.636+0000 7f8a7ec44640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7810b0c0 con 0x7f8a781013d0 2024-03-19T02:51:20.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.636+0000 7f8a7d1e1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a78101ce0 0x7f8a781a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:20.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.636+0000 7f8a7d1e1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a78101ce0 0x7f8a781a1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50104/0 (socket says 172.21.15.33:50104) 2024-03-19T02:51:20.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.636+0000 7f8a7d1e1640 1 -- 172.21.15.33:0/2321138712 learned_addr learned my addr 172.21.15.33:0/2321138712 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:20.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.636+0000 7f8a7c9e0640 1 --2- 172.21.15.33:0/2321138712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a78100a10 0x7f8a78199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:20.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.636+0000 7f8a77fff640 1 --2- 172.21.15.33:0/2321138712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a781013d0 0x7f8a7819a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:20.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.636+0000 7f8a7d1e1640 1 -- 172.21.15.33:0/2321138712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a781013d0 msgr2=0x7f8a7819a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:20.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.636+0000 7f8a7d1e1640 1 --2- 172.21.15.33:0/2321138712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a781013d0 0x7f8a7819a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:20.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.636+0000 7f8a7d1e1640 1 -- 172.21.15.33:0/2321138712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a78100a10 msgr2=0x7f8a78199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:20.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.636+0000 7f8a7d1e1640 1 --2- 172.21.15.33:0/2321138712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a78100a10 0x7f8a78199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:20.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.636+0000 7f8a7d1e1640 1 -- 172.21.15.33:0/2321138712 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a781a1930 con 0x7f8a78101ce0 2024-03-19T02:51:20.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.636+0000 7f8a7c9e0640 1 --2- 172.21.15.33:0/2321138712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a78100a10 0x7f8a78199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:20.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.636+0000 7f8a7d1e1640 1 --2- 172.21.15.33:0/2321138712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a78101ce0 0x7f8a781a1290 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8a6800e970 tx=0x7f8a6800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:20.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.637+0000 7f8a75ffb640 1 -- 172.21.15.33:0/2321138712 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6800ccc0 con 0x7f8a78101ce0 2024-03-19T02:51:20.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.637+0000 7f8a75ffb640 1 -- 172.21.15.33:0/2321138712 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a6800ce60 con 0x7f8a78101ce0 2024-03-19T02:51:20.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.637+0000 7f8a75ffb640 1 -- 172.21.15.33:0/2321138712 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a68010430 con 0x7f8a78101ce0 2024-03-19T02:51:20.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.637+0000 7f8a7ec44640 1 -- 172.21.15.33:0/2321138712 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a781a1c20 con 0x7f8a78101ce0 2024-03-19T02:51:20.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.637+0000 7f8a7ec44640 1 -- 172.21.15.33:0/2321138712 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a781a20f0 con 0x7f8a78101ce0 2024-03-19T02:51:20.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.638+0000 7f8a7ec44640 1 -- 172.21.15.33:0/2321138712 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a78107c20 con 0x7f8a78101ce0 2024-03-19T02:51:20.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.639+0000 7f8a75ffb640 1 -- 172.21.15.33:0/2321138712 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f8a68004ff0 con 0x7f8a78101ce0 2024-03-19T02:51:20.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.639+0000 7f8a75ffb640 1 --2- 172.21.15.33:0/2321138712 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8a600a2880 0x7f8a600a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:20.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.640+0000 7f8a7c9e0640 1 --2- 172.21.15.33:0/2321138712 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8a600a2880 0x7f8a600a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:20.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.640+0000 7f8a75ffb640 1 -- 172.21.15.33:0/2321138712 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7f8a680cf120 con 0x7f8a78101ce0 2024-03-19T02:51:20.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.640+0000 7f8a7c9e0640 1 --2- 172.21.15.33:0/2321138712 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8a600a2880 0x7f8a600a4d40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8a70032380 tx=0x7f8a70000f40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:20.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.644+0000 7f8a75ffb640 1 -- 172.21.15.33:0/2321138712 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8a6809bf20 con 0x7f8a78101ce0 2024-03-19T02:51:20.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.857+0000 7f8a7ec44640 1 -- 172.21.15.33:0/2321138712 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f8a78061c60 con 0x7f8a600a2880 2024-03-19T02:51:20.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.861+0000 7f8a75ffb640 1 -- 172.21.15.33:0/2321138712 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f8a78061c60 con 0x7f8a600a2880 2024-03-19T02:51:20.862 INFO:teuthology.orchestra.run.smithi033.stdout:/volumes/qa/sv_1/e63ac6a9-4a71-481c-90dc-c783fdcf66b5 2024-03-19T02:51:20.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.863+0000 7f8a7ec44640 1 -- 172.21.15.33:0/2321138712 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8a600a2880 msgr2=0x7f8a600a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:20.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.863+0000 7f8a7ec44640 1 --2- 172.21.15.33:0/2321138712 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8a600a2880 0x7f8a600a4d40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8a70032380 tx=0x7f8a70000f40 comp rx=0 tx=0).stop 2024-03-19T02:51:20.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.863+0000 7f8a7ec44640 1 -- 172.21.15.33:0/2321138712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a78101ce0 msgr2=0x7f8a781a1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:20.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.863+0000 7f8a7ec44640 1 --2- 172.21.15.33:0/2321138712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a78101ce0 0x7f8a781a1290 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8a6800e970 tx=0x7f8a6800ed70 comp rx=0 tx=0).stop 2024-03-19T02:51:20.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.863+0000 7f8a7ec44640 1 -- 172.21.15.33:0/2321138712 shutdown_connections 2024-03-19T02:51:20.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.863+0000 7f8a7ec44640 1 --2- 172.21.15.33:0/2321138712 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8a600a2880 0x7f8a600a4d40 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:20.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.863+0000 7f8a7ec44640 1 --2- 172.21.15.33:0/2321138712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a781013d0 0x7f8a7819a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:20.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.863+0000 7f8a7ec44640 1 --2- 172.21.15.33:0/2321138712 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8a78101ce0 0x7f8a781a1290 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:20.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.863+0000 7f8a7ec44640 1 --2- 172.21.15.33:0/2321138712 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8a78100a10 0x7f8a78199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:20.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.863+0000 7f8a7ec44640 1 -- 172.21.15.33:0/2321138712 >> 172.21.15.33:0/2321138712 conn(0x7f8a780fb420 msgr2=0x7f8a78104df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:20.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.863+0000 7f8a7ec44640 1 -- 172.21.15.33:0/2321138712 shutdown_connections 2024-03-19T02:51:20.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.863+0000 7f8a7ec44640 1 -- 172.21.15.33:0/2321138712 wait complete. 2024-03-19T02:51:20.875 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:51:20.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.977+0000 7f4ae19b3640 1 -- 172.21.15.33:0/1530043215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4adc102310 msgr2=0x7f4adc108ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:20.980 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.977+0000 7f4ae19b3640 1 --2- 172.21.15.33:0/1530043215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4adc102310 0x7f4adc108ba0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f4acc009900 tx=0x7f4acc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:51:20.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.977+0000 7f4ae19b3640 1 -- 172.21.15.33:0/1530043215 shutdown_connections 2024-03-19T02:51:20.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.977+0000 7f4ae19b3640 1 --2- 172.21.15.33:0/1530043215 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4adc101950 0x7f4adc101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:20.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.977+0000 7f4ae19b3640 1 --2- 172.21.15.33:0/1530043215 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4adc102310 0x7f4adc108ba0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:20.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.977+0000 7f4ae19b3640 1 --2- 172.21.15.33:0/1530043215 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4adc100760 0x7f4adc100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:20.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.977+0000 7f4ae19b3640 1 -- 172.21.15.33:0/1530043215 >> 172.21.15.33:0/1530043215 conn(0x7f4adc0fbcd0 msgr2=0x7f4adc0fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:20.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.977+0000 7f4ae19b3640 1 -- 172.21.15.33:0/1530043215 shutdown_connections 2024-03-19T02:51:20.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.977+0000 7f4ae19b3640 1 -- 172.21.15.33:0/1530043215 wait complete. 2024-03-19T02:51:20.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.977+0000 7f4ae19b3640 1 Processor -- start 2024-03-19T02:51:20.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.978+0000 7f4ae19b3640 1 -- start start 2024-03-19T02:51:20.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.978+0000 7f4ae19b3640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4adc100760 0x7f4adc197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:20.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.978+0000 7f4ae19b3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4adc101950 0x7f4adc197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:20.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.978+0000 7f4ae19b3640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4adc102310 0x7f4adc19eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:20.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.978+0000 7f4ae19b3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4adc10d370 con 0x7f4adc101950 2024-03-19T02:51:20.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.978+0000 7f4ae19b3640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4adc10d1f0 con 0x7f4adc102310 2024-03-19T02:51:20.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.978+0000 7f4ae19b3640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4adc10d4f0 con 0x7f4adc100760 2024-03-19T02:51:20.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.978+0000 7f4ada7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4adc101950 0x7f4adc197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:20.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.978+0000 7f4ada7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4adc101950 0x7f4adc197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50114/0 (socket says 172.21.15.33:50114) 2024-03-19T02:51:20.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.978+0000 7f4ada7fc640 1 -- 172.21.15.33:0/232614714 learned_addr learned my addr 172.21.15.33:0/232614714 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:20.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.978+0000 7f4adb7fe640 1 --2- 172.21.15.33:0/232614714 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4adc102310 0x7f4adc19eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:20.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.978+0000 7f4adaffd640 1 --2- 172.21.15.33:0/232614714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4adc100760 0x7f4adc197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:20.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.978+0000 7f4ada7fc640 1 -- 172.21.15.33:0/232614714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4adc100760 msgr2=0x7f4adc197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:20.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.978+0000 7f4ada7fc640 1 --2- 172.21.15.33:0/232614714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4adc100760 0x7f4adc197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:20.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.978+0000 7f4ada7fc640 1 -- 172.21.15.33:0/232614714 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4adc102310 msgr2=0x7f4adc19eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:20.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.978+0000 7f4ada7fc640 1 --2- 172.21.15.33:0/232614714 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4adc102310 0x7f4adc19eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:20.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.978+0000 7f4ada7fc640 1 -- 172.21.15.33:0/232614714 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4adc19f660 con 0x7f4adc101950 2024-03-19T02:51:20.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.978+0000 7f4ada7fc640 1 --2- 172.21.15.33:0/232614714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4adc101950 0x7f4adc197f80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4ac400d950 tx=0x7f4ac400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:20.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.978+0000 7f4ae09b1640 1 -- 172.21.15.33:0/232614714 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac4017030 con 0x7f4adc101950 2024-03-19T02:51:20.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.979+0000 7f4ae19b3640 1 -- 172.21.15.33:0/232614714 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4adc19f950 con 0x7f4adc101950 2024-03-19T02:51:20.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.979+0000 7f4ae09b1640 1 -- 172.21.15.33:0/232614714 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4ac4004020 con 0x7f4adc101950 2024-03-19T02:51:20.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.979+0000 7f4ae09b1640 1 -- 172.21.15.33:0/232614714 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac40047a0 con 0x7f4adc101950 2024-03-19T02:51:20.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.979+0000 7f4ae19b3640 1 -- 172.21.15.33:0/232614714 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4adc19fef0 con 0x7f4adc101950 2024-03-19T02:51:20.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.980+0000 7f4ae19b3640 1 -- 172.21.15.33:0/232614714 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4adc107ba0 con 0x7f4adc101950 2024-03-19T02:51:20.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.981+0000 7f4ae09b1640 1 -- 172.21.15.33:0/232614714 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f4ac40041d0 con 0x7f4adc101950 2024-03-19T02:51:20.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.981+0000 7f4ae09b1640 1 --2- 172.21.15.33:0/232614714 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4ac80a2800 0x7f4ac80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:20.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.981+0000 7f4adaffd640 1 --2- 172.21.15.33:0/232614714 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4ac80a2800 0x7f4ac80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:20.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.981+0000 7f4ae09b1640 1 -- 172.21.15.33:0/232614714 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7f4ac40ce6f0 con 0x7f4adc101950 2024-03-19T02:51:20.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.982+0000 7f4adaffd640 1 --2- 172.21.15.33:0/232614714 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4ac80a2800 0x7f4ac80a4cc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4ad40099a0 tx=0x7f4ad4009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:20.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:20.986+0000 7f4ae09b1640 1 -- 172.21.15.33:0/232614714 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4ac409b4f0 con 0x7f4adc101950 2024-03-19T02:51:21.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.099+0000 7f4ae09b1640 1 -- 172.21.15.33:0/232614714 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f4ac4097460 con 0x7f4adc101950 2024-03-19T02:51:21.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.215+0000 7f4ae19b3640 1 -- 172.21.15.33:0/232614714 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4adc02d5d0 con 0x7f4adc101950 2024-03-19T02:51:21.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.216+0000 7f4ae09b1640 1 -- 172.21.15.33:0/232614714 <== mon.0 v2:172.21.15.33:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11867 (secure 0 0 0) 0x7f4ac408b720 con 0x7f4adc101950 2024-03-19T02:51:21.217 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:51:21.217 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:50:52.201798+0000","modified":"2024-03-19T02:51:19.076317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24488,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/3731232169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":3731232169},{"type":"v1","addr":"172.21.15.96:6839","nonce":3731232169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24457,24509,24488]},"id":1}]} 2024-03-19T02:51:21.217 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 24 2024-03-19T02:51:21.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.218+0000 7f4ae19b3640 1 -- 172.21.15.33:0/232614714 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4ac80a2800 msgr2=0x7f4ac80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:21.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.218+0000 7f4ae19b3640 1 --2- 172.21.15.33:0/232614714 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4ac80a2800 0x7f4ac80a4cc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4ad40099a0 tx=0x7f4ad4009500 comp rx=0 tx=0).stop 2024-03-19T02:51:21.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.218+0000 7f4ae19b3640 1 -- 172.21.15.33:0/232614714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4adc101950 msgr2=0x7f4adc197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:21.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.218+0000 7f4ae19b3640 1 --2- 172.21.15.33:0/232614714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4adc101950 0x7f4adc197f80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4ac400d950 tx=0x7f4ac400de10 comp rx=0 tx=0).stop 2024-03-19T02:51:21.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.218+0000 7f4ae19b3640 1 -- 172.21.15.33:0/232614714 shutdown_connections 2024-03-19T02:51:21.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.218+0000 7f4ae19b3640 1 --2- 172.21.15.33:0/232614714 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4ac80a2800 0x7f4ac80a4cc0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:21.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.218+0000 7f4ae19b3640 1 --2- 172.21.15.33:0/232614714 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4adc102310 0x7f4adc19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:21.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.218+0000 7f4ae19b3640 1 --2- 172.21.15.33:0/232614714 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4adc101950 0x7f4adc197f80 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:21.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.218+0000 7f4ae19b3640 1 --2- 172.21.15.33:0/232614714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4adc100760 0x7f4adc197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:21.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.218+0000 7f4ae19b3640 1 -- 172.21.15.33:0/232614714 >> 172.21.15.33:0/232614714 conn(0x7f4adc0fbcd0 msgr2=0x7f4adc104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:21.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.218+0000 7f4ae19b3640 1 -- 172.21.15.33:0/232614714 shutdown_connections 2024-03-19T02:51:21.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.218+0000 7f4ae19b3640 1 -- 172.21.15.33:0/232614714 wait complete. 2024-03-19T02:51:21.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:21 smithi033 ceph-mon[31973]: from='client.14874 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:51:21.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:21 smithi033 ceph-mon[31973]: from='client.14880 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:51:21.229 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:21 smithi033 ceph-mon[31973]: from='client.14886 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:51:21.230 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:51:21.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.333+0000 7f8f93c44640 1 -- 172.21.15.33:0/2315148951 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f8c102a60 msgr2=0x7f8f8c102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:21.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.333+0000 7f8f93c44640 1 --2- 172.21.15.33:0/2315148951 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f8c102a60 0x7f8f8c102ee0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8f80009900 tx=0x7f8f8002bc30 comp rx=0 tx=0).stop 2024-03-19T02:51:21.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.333+0000 7f8f93c44640 1 -- 172.21.15.33:0/2315148951 shutdown_connections 2024-03-19T02:51:21.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.333+0000 7f8f93c44640 1 --2- 172.21.15.33:0/2315148951 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f8c103420 0x7f8f8c109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:21.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.333+0000 7f8f93c44640 1 --2- 172.21.15.33:0/2315148951 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f8c102a60 0x7f8f8c102ee0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:21.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.333+0000 7f8f93c44640 1 --2- 172.21.15.33:0/2315148951 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f8c101870 0x7f8f8c101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:21.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.333+0000 7f8f93c44640 1 -- 172.21.15.33:0/2315148951 >> 172.21.15.33:0/2315148951 conn(0x7f8f8c0fcdc0 msgr2=0x7f8f8c0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:21.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.333+0000 7f8f93c44640 1 -- 172.21.15.33:0/2315148951 shutdown_connections 2024-03-19T02:51:21.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.333+0000 7f8f93c44640 1 -- 172.21.15.33:0/2315148951 wait complete. 2024-03-19T02:51:21.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.333+0000 7f8f93c44640 1 Processor -- start 2024-03-19T02:51:21.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.333+0000 7f8f93c44640 1 -- start start 2024-03-19T02:51:21.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.333+0000 7f8f93c44640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f8c101870 0x7f8f8c197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:21.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.333+0000 7f8f93c44640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f8c102a60 0x7f8f8c197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:21.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.333+0000 7f8f93c44640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f8c103420 0x7f8f8c19f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:21.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.333+0000 7f8f93c44640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8f8c10d3a0 con 0x7f8f8c103420 2024-03-19T02:51:21.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.333+0000 7f8f93c44640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8f8c10d220 con 0x7f8f8c102a60 2024-03-19T02:51:21.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.333+0000 7f8f93c44640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8f8c10d520 con 0x7f8f8c101870 2024-03-19T02:51:21.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.334+0000 7f8f921e1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f8c103420 0x7f8f8c19f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:21.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.334+0000 7f8f921e1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f8c103420 0x7f8f8c19f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50140/0 (socket says 172.21.15.33:50140) 2024-03-19T02:51:21.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.334+0000 7f8f921e1640 1 -- 172.21.15.33:0/1633399558 learned_addr learned my addr 172.21.15.33:0/1633399558 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:21.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.334+0000 7f8f911df640 1 --2- 172.21.15.33:0/1633399558 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f8c102a60 0x7f8f8c197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:21.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.334+0000 7f8f919e0640 1 --2- 172.21.15.33:0/1633399558 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f8c101870 0x7f8f8c197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:21.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.334+0000 7f8f911df640 1 -- 172.21.15.33:0/1633399558 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f8c101870 msgr2=0x7f8f8c197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:21.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.334+0000 7f8f911df640 1 --2- 172.21.15.33:0/1633399558 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f8c101870 0x7f8f8c197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:21.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.334+0000 7f8f911df640 1 -- 172.21.15.33:0/1633399558 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f8c103420 msgr2=0x7f8f8c19f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:21.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.334+0000 7f8f911df640 1 --2- 172.21.15.33:0/1633399558 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f8c103420 0x7f8f8c19f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:21.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.334+0000 7f8f911df640 1 -- 172.21.15.33:0/1633399558 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f8c19f6c0 con 0x7f8f8c102a60 2024-03-19T02:51:21.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.334+0000 7f8f921e1640 1 --2- 172.21.15.33:0/1633399558 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f8c103420 0x7f8f8c19f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:21.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.334+0000 7f8f919e0640 1 --2- 172.21.15.33:0/1633399558 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f8c101870 0x7f8f8c197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:21.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.334+0000 7f8f911df640 1 --2- 172.21.15.33:0/1633399558 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f8c102a60 0x7f8f8c197fe0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8f80000c00 tx=0x7f8f80005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:21.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.334+0000 7f8f8affd640 1 -- 172.21.15.33:0/1633399558 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f80043020 con 0x7f8f8c102a60 2024-03-19T02:51:21.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.334+0000 7f8f93c44640 1 -- 172.21.15.33:0/1633399558 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f8c19f950 con 0x7f8f8c102a60 2024-03-19T02:51:21.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.334+0000 7f8f8affd640 1 -- 172.21.15.33:0/1633399558 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8f80004350 con 0x7f8f8c102a60 2024-03-19T02:51:21.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.334+0000 7f8f8affd640 1 -- 172.21.15.33:0/1633399558 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f80003b50 con 0x7f8f8c102a60 2024-03-19T02:51:21.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.334+0000 7f8f93c44640 1 -- 172.21.15.33:0/1633399558 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f8c19fef0 con 0x7f8f8c102a60 2024-03-19T02:51:21.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.335+0000 7f8f93c44640 1 -- 172.21.15.33:0/1633399558 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f8c108c00 con 0x7f8f8c102a60 2024-03-19T02:51:21.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.336+0000 7f8f8affd640 1 -- 172.21.15.33:0/1633399558 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f8f80046020 con 0x7f8f8c102a60 2024-03-19T02:51:21.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.337+0000 7f8f8affd640 1 --2- 172.21.15.33:0/1633399558 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8f740a2890 0x7f8f740a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:21.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.337+0000 7f8f8affd640 1 -- 172.21.15.33:0/1633399558 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7f8f800f1320 con 0x7f8f8c102a60 2024-03-19T02:51:21.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.337+0000 7f8f919e0640 1 --2- 172.21.15.33:0/1633399558 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8f740a2890 0x7f8f740a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:21.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.337+0000 7f8f919e0640 1 --2- 172.21.15.33:0/1633399558 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8f740a2890 0x7f8f740a4d50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c1028c0 tx=0x7f8f84009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:21.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.341+0000 7f8f8affd640 1 -- 172.21.15.33:0/1633399558 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8f8004d020 con 0x7f8f8c102a60 2024-03-19T02:51:21.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:21 smithi096 ceph-mon[35946]: from='client.14874 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:51:21.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:21 smithi096 ceph-mon[35946]: from='client.14880 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:51:21.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:21 smithi096 ceph-mon[35946]: from='client.14886 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:51:21.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:21 smithi155 ceph-mon[35804]: from='client.14874 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:51:21.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:21 smithi155 ceph-mon[35804]: from='client.14880 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:51:21.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:21 smithi155 ceph-mon[35804]: from='client.14886 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:51:21.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.525+0000 7f8f93c44640 1 -- 172.21.15.33:0/1633399558 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8f8c02d3f0 con 0x7f8f8c102a60 2024-03-19T02:51:21.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.526+0000 7f8f8affd640 1 -- 172.21.15.33:0/1633399558 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24043 (secure 0 0 0) 0x7f8f800be130 con 0x7f8f8c102a60 2024-03-19T02:51:21.528 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:51:21.528 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":93,"fsid":"2b7a8938-e59a-11ee-95c9-87774f69a715","created":"2024-03-19T02:42:37.912179+0000","modified":"2024-03-19T02:50:59.241668+0000","last_up_change":"2024-03-19T02:50:01.539392+0000","last_in_change":"2024-03-19T02:49:40.534415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:46:09.963737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:50:50.069641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:50:51.381064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:50:55.476667+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"090a89ab-6b7b-4317-8823-178d4212b2f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6803","nonce":1633124995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6805","nonce":1633124995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6809","nonce":1633124995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6807","nonce":1633124995}]},"public_addr":"172.21.15.33:6803/1633124995","cluster_addr":"172.21.15.33:6805/1633124995","heartbeat_back_addr":"172.21.15.33:6809/1633124995","heartbeat_front_addr":"172.21.15.33:6807/1633124995","state":["exists","up"]},{"osd":1,"uuid":"f50eec55-1956-4486-8593-f2c8fa610e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6801","nonce":340144398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6803","nonce":340144398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6807","nonce":340144398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6805","nonce":340144398}]},"public_addr":"172.21.15.96:6801/340144398","cluster_addr":"172.21.15.96:6803/340144398","heartbeat_back_addr":"172.21.15.96:6807/340144398","heartbeat_front_addr":"172.21.15.96:6805/340144398","state":["exists","up"]},{"osd":2,"uuid":"893a2d62-80d9-4e5e-bc6a-a1c8c681d652","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6801","nonce":3388743876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6803","nonce":3388743876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6807","nonce":3388743876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6805","nonce":3388743876}]},"public_addr":"172.21.15.155:6801/3388743876","cluster_addr":"172.21.15.155:6803/3388743876","heartbeat_back_addr":"172.21.15.155:6807/3388743876","heartbeat_front_addr":"172.21.15.155:6805/3388743876","state":["exists","up"]},{"osd":3,"uuid":"b07e0482-4281-4d32-9dfe-9995d7246175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6811","nonce":677007171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6813","nonce":677007171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6817","nonce":677007171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6815","nonce":677007171}]},"public_addr":"172.21.15.33:6811/677007171","cluster_addr":"172.21.15.33:6813/677007171","heartbeat_back_addr":"172.21.15.33:6817/677007171","heartbeat_front_addr":"172.21.15.33:6815/677007171","state":["exists","up"]},{"osd":4,"uuid":"a988384b-af6b-47c6-8c07-5c9673acee45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6809","nonce":3106158554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6811","nonce":3106158554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6815","nonce":3106158554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6813","nonce":3106158554}]},"public_addr":"172.21.15.96:6809/3106158554","cluster_addr":"172.21.15.96:6811/3106158554","heartbeat_back_addr":"172.21.15.96:6815/3106158554","heartbeat_front_addr":"172.21.15.96:6813/3106158554","state":["exists","up"]},{"osd":5,"uuid":"198dc992-96f8-4c52-96ee-1fd5de148bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6809","nonce":327293324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6811","nonce":327293324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6815","nonce":327293324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6813","nonce":327293324}]},"public_addr":"172.21.15.155:6809/327293324","cluster_addr":"172.21.15.155:6811/327293324","heartbeat_back_addr":"172.21.15.155:6815/327293324","heartbeat_front_addr":"172.21.15.155:6813/327293324","state":["exists","up"]},{"osd":6,"uuid":"a7278ec0-ab2e-49ed-be53-84d8b9e24177","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6819","nonce":873871509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6821","nonce":873871509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6825","nonce":873871509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6823","nonce":873871509}]},"public_addr":"172.21.15.33:6819/873871509","cluster_addr":"172.21.15.33:6821/873871509","heartbeat_back_addr":"172.21.15.33:6825/873871509","heartbeat_front_addr":"172.21.15.33:6823/873871509","state":["exists","up"]},{"osd":7,"uuid":"727e133d-aa24-4ffc-90e2-266eb2aab8c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6817","nonce":2334123891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6819","nonce":2334123891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6823","nonce":2334123891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6821","nonce":2334123891}]},"public_addr":"172.21.15.96:6817/2334123891","cluster_addr":"172.21.15.96:6819/2334123891","heartbeat_back_addr":"172.21.15.96:6823/2334123891","heartbeat_front_addr":"172.21.15.96:6821/2334123891","state":["exists","up"]},{"osd":8,"uuid":"bb20eec0-39e5-4b86-9db5-486b1840cdad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6817","nonce":3498516903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6819","nonce":3498516903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6823","nonce":3498516903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6821","nonce":3498516903}]},"public_addr":"172.21.15.155:6817/3498516903","cluster_addr":"172.21.15.155:6819/3498516903","heartbeat_back_addr":"172.21.15.155:6823/3498516903","heartbeat_front_addr":"172.21.15.155:6821/3498516903","state":["exists","up"]},{"osd":9,"uuid":"179c6228-3e23-4f65-91a6-1ef0005d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6827","nonce":760777213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6829","nonce":760777213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6833","nonce":760777213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6831","nonce":760777213}]},"public_addr":"172.21.15.33:6827/760777213","cluster_addr":"172.21.15.33:6829/760777213","heartbeat_back_addr":"172.21.15.33:6833/760777213","heartbeat_front_addr":"172.21.15.33:6831/760777213","state":["exists","up"]},{"osd":10,"uuid":"be17b478-3c2e-41fb-bede-958333acdf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6825","nonce":1404898482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6827","nonce":1404898482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6831","nonce":1404898482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6829","nonce":1404898482}]},"public_addr":"172.21.15.96:6825/1404898482","cluster_addr":"172.21.15.96:6827/1404898482","heartbeat_back_addr":"172.21.15.96:6831/1404898482","heartbeat_front_addr":"172.21.15.96:6829/1404898482","state":["exists","up"]},{"osd":11,"uuid":"c0903ed2-cc1e-4c18-8731-1f0ee4dd0457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6825","nonce":792352827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6827","nonce":792352827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6831","nonce":792352827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6829","nonce":792352827}]},"public_addr":"172.21.15.155:6825/792352827","cluster_addr":"172.21.15.155:6827/792352827","heartbeat_back_addr":"172.21.15.155:6831/792352827","heartbeat_front_addr":"172.21.15.155:6829/792352827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:17.942801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:42.109449+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:05.981599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:32.069574+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:57.431015+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:22.448592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:48.526515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:14.498352+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:40.175426+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:06.364006+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:32.853161+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:59.073538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:6801/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:6800/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/1067768047":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/729379453":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2586588742":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/2932814165":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6801/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2212564963":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6800/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/3275126058":"2024-03-20T02:43:39.622218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:51:21.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.528+0000 7f8f93c44640 1 -- 172.21.15.33:0/1633399558 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8f740a2890 msgr2=0x7f8f740a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:21.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.528+0000 7f8f93c44640 1 --2- 172.21.15.33:0/1633399558 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8f740a2890 0x7f8f740a4d50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c1028c0 tx=0x7f8f84009500 comp rx=0 tx=0).stop 2024-03-19T02:51:21.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.528+0000 7f8f93c44640 1 -- 172.21.15.33:0/1633399558 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f8c102a60 msgr2=0x7f8f8c197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:21.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.528+0000 7f8f93c44640 1 --2- 172.21.15.33:0/1633399558 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f8c102a60 0x7f8f8c197fe0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8f80000c00 tx=0x7f8f80005dc0 comp rx=0 tx=0).stop 2024-03-19T02:51:21.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.528+0000 7f8f93c44640 1 -- 172.21.15.33:0/1633399558 shutdown_connections 2024-03-19T02:51:21.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.528+0000 7f8f93c44640 1 --2- 172.21.15.33:0/1633399558 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8f740a2890 0x7f8f740a4d50 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:21.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.528+0000 7f8f93c44640 1 --2- 172.21.15.33:0/1633399558 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f8c103420 0x7f8f8c19f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:21.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.528+0000 7f8f93c44640 1 --2- 172.21.15.33:0/1633399558 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f8c102a60 0x7f8f8c197fe0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:21.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.528+0000 7f8f93c44640 1 --2- 172.21.15.33:0/1633399558 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f8c101870 0x7f8f8c197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:21.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.528+0000 7f8f93c44640 1 -- 172.21.15.33:0/1633399558 >> 172.21.15.33:0/1633399558 conn(0x7f8f8c0fcdc0 msgr2=0x7f8f8c105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:21.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.528+0000 7f8f93c44640 1 -- 172.21.15.33:0/1633399558 shutdown_connections 2024-03-19T02:51:21.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.528+0000 7f8f93c44640 1 -- 172.21.15.33:0/1633399558 wait complete. 2024-03-19T02:51:21.541 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-19T02:51:21.541 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T02:51:21.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.643+0000 7ff72811b640 1 -- 172.21.15.33:0/1737074208 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff720101840 msgr2=0x7ff720101c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:21.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.643+0000 7ff72811b640 1 --2- 172.21.15.33:0/1737074208 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff720101840 0x7ff720101c60 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff71c009900 tx=0x7ff71c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:51:21.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.643+0000 7ff72811b640 1 -- 172.21.15.33:0/1737074208 shutdown_connections 2024-03-19T02:51:21.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.643+0000 7ff72811b640 1 --2- 172.21.15.33:0/1737074208 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7201033f0 0x7ff720109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:21.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.643+0000 7ff72811b640 1 --2- 172.21.15.33:0/1737074208 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff720102a30 0x7ff720102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:21.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.643+0000 7ff72811b640 1 --2- 172.21.15.33:0/1737074208 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff720101840 0x7ff720101c60 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:21.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.643+0000 7ff72811b640 1 -- 172.21.15.33:0/1737074208 >> 172.21.15.33:0/1737074208 conn(0x7ff7200fcdd0 msgr2=0x7ff7200ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:21.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.643+0000 7ff72811b640 1 -- 172.21.15.33:0/1737074208 shutdown_connections 2024-03-19T02:51:21.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.643+0000 7ff72811b640 1 -- 172.21.15.33:0/1737074208 wait complete. 2024-03-19T02:51:21.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.643+0000 7ff72811b640 1 Processor -- start 2024-03-19T02:51:21.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.644+0000 7ff72811b640 1 -- start start 2024-03-19T02:51:21.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.644+0000 7ff72811b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff720101840 0x7ff720197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:21.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.644+0000 7ff72811b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff720102a30 0x7ff720197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:21.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.644+0000 7ff72811b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7201033f0 0x7ff72019f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:21.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.644+0000 7ff72811b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff72010d370 con 0x7ff720101840 2024-03-19T02:51:21.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.644+0000 7ff72811b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff72010d1f0 con 0x7ff720102a30 2024-03-19T02:51:21.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.644+0000 7ff72811b640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff72010d4f0 con 0x7ff7201033f0 2024-03-19T02:51:21.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.644+0000 7ff725eb7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff720101840 0x7ff720197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:21.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.644+0000 7ff725eb7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff720101840 0x7ff720197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50154/0 (socket says 172.21.15.33:50154) 2024-03-19T02:51:21.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.644+0000 7ff725eb7640 1 -- 172.21.15.33:0/3539953811 learned_addr learned my addr 172.21.15.33:0/3539953811 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:21.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.644+0000 7ff7256b6640 1 --2- 172.21.15.33:0/3539953811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff720102a30 0x7ff720197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:21.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.644+0000 7ff7266b8640 1 --2- 172.21.15.33:0/3539953811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7201033f0 0x7ff72019f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:21.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.644+0000 7ff725eb7640 1 -- 172.21.15.33:0/3539953811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7201033f0 msgr2=0x7ff72019f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:21.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.644+0000 7ff725eb7640 1 --2- 172.21.15.33:0/3539953811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7201033f0 0x7ff72019f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:21.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.644+0000 7ff725eb7640 1 -- 172.21.15.33:0/3539953811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff720102a30 msgr2=0x7ff720197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:21.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.644+0000 7ff725eb7640 1 --2- 172.21.15.33:0/3539953811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff720102a30 0x7ff720197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:21.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.644+0000 7ff725eb7640 1 -- 172.21.15.33:0/3539953811 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff72019f690 con 0x7ff720101840 2024-03-19T02:51:21.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.644+0000 7ff725eb7640 1 --2- 172.21.15.33:0/3539953811 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff720101840 0x7ff720197a70 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff71c000c00 tx=0x7ff71c005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:21.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.645+0000 7ff716ffd640 1 -- 172.21.15.33:0/3539953811 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff71c043020 con 0x7ff720101840 2024-03-19T02:51:21.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.645+0000 7ff716ffd640 1 -- 172.21.15.33:0/3539953811 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff71c004350 con 0x7ff720101840 2024-03-19T02:51:21.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.645+0000 7ff72811b640 1 -- 172.21.15.33:0/3539953811 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff72019f920 con 0x7ff720101840 2024-03-19T02:51:21.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.645+0000 7ff716ffd640 1 -- 172.21.15.33:0/3539953811 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff71c003b80 con 0x7ff720101840 2024-03-19T02:51:21.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.645+0000 7ff72811b640 1 -- 172.21.15.33:0/3539953811 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff72019fe70 con 0x7ff720101840 2024-03-19T02:51:21.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.646+0000 7ff72811b640 1 -- 172.21.15.33:0/3539953811 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff720068a60 con 0x7ff720101840 2024-03-19T02:51:21.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.646+0000 7ff716ffd640 1 -- 172.21.15.33:0/3539953811 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7ff71c046020 con 0x7ff720101840 2024-03-19T02:51:21.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.647+0000 7ff716ffd640 1 --2- 172.21.15.33:0/3539953811 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff70c0a2890 0x7ff70c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:21.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.647+0000 7ff7256b6640 1 --2- 172.21.15.33:0/3539953811 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff70c0a2890 0x7ff70c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:21.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.647+0000 7ff716ffd640 1 -- 172.21.15.33:0/3539953811 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7ff71c0f0ea0 con 0x7ff720101840 2024-03-19T02:51:21.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.648+0000 7ff7256b6640 1 --2- 172.21.15.33:0/3539953811 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff70c0a2890 0x7ff70c0a4d50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff720198f80 tx=0x7ff710005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:21.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.652+0000 7ff716ffd640 1 -- 172.21.15.33:0/3539953811 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff71c04d020 con 0x7ff720101840 2024-03-19T02:51:21.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.833+0000 7ff72811b640 1 -- 172.21.15.33:0/3539953811 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff720061c60 con 0x7ff70c0a2890 2024-03-19T02:51:21.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.843+0000 7ff716ffd640 1 -- 172.21.15.33:0/3539953811 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340348 (secure 0 0 0) 0x7ff720061c60 con 0x7ff70c0a2890 2024-03-19T02:51:21.849 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:51:21.851 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-03-19T02:51:21.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.847+0000 7ff72811b640 1 -- 172.21.15.33:0/3539953811 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff70c0a2890 msgr2=0x7ff70c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:21.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.847+0000 7ff72811b640 1 --2- 172.21.15.33:0/3539953811 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff70c0a2890 0x7ff70c0a4d50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff720198f80 tx=0x7ff710005f90 comp rx=0 tx=0).stop 2024-03-19T02:51:21.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.847+0000 7ff72811b640 1 -- 172.21.15.33:0/3539953811 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff720101840 msgr2=0x7ff720197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:21.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.847+0000 7ff72811b640 1 --2- 172.21.15.33:0/3539953811 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff720101840 0x7ff720197a70 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff71c000c00 tx=0x7ff71c005dc0 comp rx=0 tx=0).stop 2024-03-19T02:51:21.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.847+0000 7ff72811b640 1 -- 172.21.15.33:0/3539953811 shutdown_connections 2024-03-19T02:51:21.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.847+0000 7ff72811b640 1 --2- 172.21.15.33:0/3539953811 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff70c0a2890 0x7ff70c0a4d50 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:21.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.847+0000 7ff72811b640 1 --2- 172.21.15.33:0/3539953811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7201033f0 0x7ff72019f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:21.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.847+0000 7ff72811b640 1 --2- 172.21.15.33:0/3539953811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff720102a30 0x7ff720197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:21.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.847+0000 7ff72811b640 1 --2- 172.21.15.33:0/3539953811 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff720101840 0x7ff720197a70 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:21.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.847+0000 7ff72811b640 1 -- 172.21.15.33:0/3539953811 >> 172.21.15.33:0/3539953811 conn(0x7ff7200fcdd0 msgr2=0x7ff720105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:21.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.847+0000 7ff72811b640 1 -- 172.21.15.33:0/3539953811 shutdown_connections 2024-03-19T02:51:21.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:21.847+0000 7ff72811b640 1 -- 172.21.15.33:0/3539953811 wait complete. 2024-03-19T02:51:21.859 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":314,"stamp":"2024-03-19T02:51:21.710892+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":604537,"num_objects":62,"num_object_clones":0,"num_object_copies":187,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":162,"num_read_kb":143,"num_write":313,"num_write_kb":2990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":253,"ondisk_log_size":253,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":363252,"kb_used_data":14096,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124578572,"statfs":{"total":1151940427776,"available":1151568457728,"internally_reserved":0,"allocated":14434304,"data_stored":7854382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9529,"num_objects":38,"num_object_clones":0,"num_object_copies":115,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":26,"num_read_kb":24,"num_write":83,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003535"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.124670+0000","last_change":"2024-03-19T02:50:52.228368+0000","last_active":"2024-03-19T02:51:00.124670+0000","last_peered":"2024-03-19T02:51:00.124670+0000","last_clean":"2024-03-19T02:51:00.124670+0000","last_became_active":"2024-03-19T02:50:52.228069+0000","last_became_peered":"2024-03-19T02:50:52.228069+0000","last_unstale":"2024-03-19T02:51:00.124670+0000","last_undegraded":"2024-03-19T02:51:00.124670+0000","last_fullsized":"2024-03-19T02:51:00.124670+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:59:34.209095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.832634+0000","last_change":"2024-03-19T02:50:52.231358+0000","last_active":"2024-03-19T02:50:59.832634+0000","last_peered":"2024-03-19T02:50:59.832634+0000","last_clean":"2024-03-19T02:50:59.832634+0000","last_became_active":"2024-03-19T02:50:52.231029+0000","last_became_peered":"2024-03-19T02:50:52.231029+0000","last_unstale":"2024-03-19T02:50:59.832634+0000","last_undegraded":"2024-03-19T02:50:59.832634+0000","last_fullsized":"2024-03-19T02:50:59.832634+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:42:26.031392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.832451+0000","last_change":"2024-03-19T02:50:52.232685+0000","last_active":"2024-03-19T02:50:59.832451+0000","last_peered":"2024-03-19T02:50:59.832451+0000","last_clean":"2024-03-19T02:50:59.832451+0000","last_became_active":"2024-03-19T02:50:52.232361+0000","last_became_peered":"2024-03-19T02:50:52.232361+0000","last_unstale":"2024-03-19T02:50:59.832451+0000","last_undegraded":"2024-03-19T02:50:59.832451+0000","last_fullsized":"2024-03-19T02:50:59.832451+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:53.136200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245640+0000","last_change":"2024-03-19T02:50:52.227648+0000","last_active":"2024-03-19T02:50:59.245640+0000","last_peered":"2024-03-19T02:50:59.245640+0000","last_clean":"2024-03-19T02:50:59.245640+0000","last_became_active":"2024-03-19T02:50:52.226943+0000","last_became_peered":"2024-03-19T02:50:52.226943+0000","last_unstale":"2024-03-19T02:50:59.245640+0000","last_undegraded":"2024-03-19T02:50:59.245640+0000","last_fullsized":"2024-03-19T02:50:59.245640+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:03.293794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122803+0000","last_change":"2024-03-19T02:50:52.225329+0000","last_active":"2024-03-19T02:51:00.122803+0000","last_peered":"2024-03-19T02:51:00.122803+0000","last_clean":"2024-03-19T02:51:00.122803+0000","last_became_active":"2024-03-19T02:50:52.225012+0000","last_became_peered":"2024-03-19T02:50:52.225012+0000","last_unstale":"2024-03-19T02:51:00.122803+0000","last_undegraded":"2024-03-19T02:51:00.122803+0000","last_fullsized":"2024-03-19T02:51:00.122803+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:32:52.877704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.036857+0000","last_change":"2024-03-19T02:50:52.217399+0000","last_active":"2024-03-19T02:51:00.036857+0000","last_peered":"2024-03-19T02:51:00.036857+0000","last_clean":"2024-03-19T02:51:00.036857+0000","last_became_active":"2024-03-19T02:50:52.216987+0000","last_became_peered":"2024-03-19T02:50:52.216987+0000","last_unstale":"2024-03-19T02:51:00.036857+0000","last_undegraded":"2024-03-19T02:51:00.036857+0000","last_fullsized":"2024-03-19T02:51:00.036857+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:57:15.772546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122135+0000","last_change":"2024-03-19T02:50:52.226755+0000","last_active":"2024-03-19T02:51:00.122135+0000","last_peered":"2024-03-19T02:51:00.122135+0000","last_clean":"2024-03-19T02:51:00.122135+0000","last_became_active":"2024-03-19T02:50:52.226434+0000","last_became_peered":"2024-03-19T02:50:52.226434+0000","last_unstale":"2024-03-19T02:51:00.122135+0000","last_undegraded":"2024-03-19T02:51:00.122135+0000","last_fullsized":"2024-03-19T02:51:00.122135+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:49:42.134262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122957+0000","last_change":"2024-03-19T02:50:52.215848+0000","last_active":"2024-03-19T02:51:00.122957+0000","last_peered":"2024-03-19T02:51:00.122957+0000","last_clean":"2024-03-19T02:51:00.122957+0000","last_became_active":"2024-03-19T02:50:52.215494+0000","last_became_peered":"2024-03-19T02:50:52.215494+0000","last_unstale":"2024-03-19T02:51:00.122957+0000","last_undegraded":"2024-03-19T02:51:00.122957+0000","last_fullsized":"2024-03-19T02:51:00.122957+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:55:10.182359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122921+0000","last_change":"2024-03-19T02:50:52.233237+0000","last_active":"2024-03-19T02:51:00.122921+0000","last_peered":"2024-03-19T02:51:00.122921+0000","last_clean":"2024-03-19T02:51:00.122921+0000","last_became_active":"2024-03-19T02:50:52.232091+0000","last_became_peered":"2024-03-19T02:50:52.232091+0000","last_unstale":"2024-03-19T02:51:00.122921+0000","last_undegraded":"2024-03-19T02:51:00.122921+0000","last_fullsized":"2024-03-19T02:51:00.122921+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:08:20.361576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.037210+0000","last_change":"2024-03-19T02:50:52.219880+0000","last_active":"2024-03-19T02:51:00.037210+0000","last_peered":"2024-03-19T02:51:00.037210+0000","last_clean":"2024-03-19T02:51:00.037210+0000","last_became_active":"2024-03-19T02:50:52.219180+0000","last_became_peered":"2024-03-19T02:50:52.219180+0000","last_unstale":"2024-03-19T02:51:00.037210+0000","last_undegraded":"2024-03-19T02:51:00.037210+0000","last_fullsized":"2024-03-19T02:51:00.037210+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:47:29.861906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122755+0000","last_change":"2024-03-19T02:50:52.226314+0000","last_active":"2024-03-19T02:51:00.122755+0000","last_peered":"2024-03-19T02:51:00.122755+0000","last_clean":"2024-03-19T02:51:00.122755+0000","last_became_active":"2024-03-19T02:50:52.225919+0000","last_became_peered":"2024-03-19T02:50:52.225919+0000","last_unstale":"2024-03-19T02:51:00.122755+0000","last_undegraded":"2024-03-19T02:51:00.122755+0000","last_fullsized":"2024-03-19T02:51:00.122755+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:42:51.782592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245637+0000","last_change":"2024-03-19T02:50:52.227929+0000","last_active":"2024-03-19T02:50:59.245637+0000","last_peered":"2024-03-19T02:50:59.245637+0000","last_clean":"2024-03-19T02:50:59.245637+0000","last_became_active":"2024-03-19T02:50:52.227360+0000","last_became_peered":"2024-03-19T02:50:52.227360+0000","last_unstale":"2024-03-19T02:50:59.245637+0000","last_undegraded":"2024-03-19T02:50:59.245637+0000","last_fullsized":"2024-03-19T02:50:59.245637+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:24:44.386101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245208+0000","last_change":"2024-03-19T02:50:52.228088+0000","last_active":"2024-03-19T02:50:59.245208+0000","last_peered":"2024-03-19T02:50:59.245208+0000","last_clean":"2024-03-19T02:50:59.245208+0000","last_became_active":"2024-03-19T02:50:52.227754+0000","last_became_peered":"2024-03-19T02:50:52.227754+0000","last_unstale":"2024-03-19T02:50:59.245208+0000","last_undegraded":"2024-03-19T02:50:59.245208+0000","last_fullsized":"2024-03-19T02:50:59.245208+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:58:57.404241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.832111+0000","last_change":"2024-03-19T02:50:52.226768+0000","last_active":"2024-03-19T02:50:59.832111+0000","last_peered":"2024-03-19T02:50:59.832111+0000","last_clean":"2024-03-19T02:50:59.832111+0000","last_became_active":"2024-03-19T02:50:52.226452+0000","last_became_peered":"2024-03-19T02:50:52.226452+0000","last_unstale":"2024-03-19T02:50:59.832111+0000","last_undegraded":"2024-03-19T02:50:59.832111+0000","last_fullsized":"2024-03-19T02:50:59.832111+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:23:33.250773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.832082+0000","last_change":"2024-03-19T02:50:52.231132+0000","last_active":"2024-03-19T02:50:59.832082+0000","last_peered":"2024-03-19T02:50:59.832082+0000","last_clean":"2024-03-19T02:50:59.832082+0000","last_became_active":"2024-03-19T02:50:52.230778+0000","last_became_peered":"2024-03-19T02:50:52.230778+0000","last_unstale":"2024-03-19T02:50:59.832082+0000","last_undegraded":"2024-03-19T02:50:59.832082+0000","last_fullsized":"2024-03-19T02:50:59.832082+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:33:03.675210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122740+0000","last_change":"2024-03-19T02:50:52.228302+0000","last_active":"2024-03-19T02:51:00.122740+0000","last_peered":"2024-03-19T02:51:00.122740+0000","last_clean":"2024-03-19T02:51:00.122740+0000","last_became_active":"2024-03-19T02:50:52.228007+0000","last_became_peered":"2024-03-19T02:50:52.228007+0000","last_unstale":"2024-03-19T02:51:00.122740+0000","last_undegraded":"2024-03-19T02:51:00.122740+0000","last_fullsized":"2024-03-19T02:51:00.122740+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:59:28.988504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.124445+0000","last_change":"2024-03-19T02:50:52.228640+0000","last_active":"2024-03-19T02:51:00.124445+0000","last_peered":"2024-03-19T02:51:00.124445+0000","last_clean":"2024-03-19T02:51:00.124445+0000","last_became_active":"2024-03-19T02:50:52.228319+0000","last_became_peered":"2024-03-19T02:50:52.228319+0000","last_unstale":"2024-03-19T02:51:00.124445+0000","last_undegraded":"2024-03-19T02:51:00.124445+0000","last_fullsized":"2024-03-19T02:51:00.124445+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:30.356453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123774+0000","last_change":"2024-03-19T02:50:52.222300+0000","last_active":"2024-03-19T02:51:00.123774+0000","last_peered":"2024-03-19T02:51:00.123774+0000","last_clean":"2024-03-19T02:51:00.123774+0000","last_became_active":"2024-03-19T02:50:52.221978+0000","last_became_peered":"2024-03-19T02:50:52.221978+0000","last_unstale":"2024-03-19T02:51:00.123774+0000","last_undegraded":"2024-03-19T02:51:00.123774+0000","last_fullsized":"2024-03-19T02:51:00.123774+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:38:36.576987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.653257+0000","last_change":"2024-03-19T02:50:52.217640+0000","last_active":"2024-03-19T02:50:59.653257+0000","last_peered":"2024-03-19T02:50:59.653257+0000","last_clean":"2024-03-19T02:50:59.653257+0000","last_became_active":"2024-03-19T02:50:52.217325+0000","last_became_peered":"2024-03-19T02:50:52.217325+0000","last_unstale":"2024-03-19T02:50:59.653257+0000","last_undegraded":"2024-03-19T02:50:59.653257+0000","last_fullsized":"2024-03-19T02:50:59.653257+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:40:25.894843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.652853+0000","last_change":"2024-03-19T02:50:57.245323+0000","last_active":"2024-03-19T02:50:59.652853+0000","last_peered":"2024-03-19T02:50:59.652853+0000","last_clean":"2024-03-19T02:50:59.652853+0000","last_became_active":"2024-03-19T02:50:57.244886+0000","last_became_peered":"2024-03-19T02:50:57.244886+0000","last_unstale":"2024-03-19T02:50:59.652853+0000","last_undegraded":"2024-03-19T02:50:59.652853+0000","last_fullsized":"2024-03-19T02:50:59.652853+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:42:09.465738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"93'9","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:18.083329+0000","last_change":"2024-03-19T02:50:51.797160+0000","last_active":"2024-03-19T02:51:18.083329+0000","last_peered":"2024-03-19T02:51:18.083329+0000","last_clean":"2024-03-19T02:51:18.083329+0000","last_became_active":"2024-03-19T02:50:51.796783+0000","last_became_peered":"2024-03-19T02:50:51.796783+0000","last_unstale":"2024-03-19T02:51:18.083329+0000","last_undegraded":"2024-03-19T02:51:18.083329+0000","last_fullsized":"2024-03-19T02:51:18.083329+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:57:08.185379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123291+0000","last_change":"2024-03-19T02:50:52.225114+0000","last_active":"2024-03-19T02:51:00.123291+0000","last_peered":"2024-03-19T02:51:00.123291+0000","last_clean":"2024-03-19T02:51:00.123291+0000","last_became_active":"2024-03-19T02:50:52.220529+0000","last_became_peered":"2024-03-19T02:50:52.220529+0000","last_unstale":"2024-03-19T02:51:00.123291+0000","last_undegraded":"2024-03-19T02:51:00.123291+0000","last_fullsized":"2024-03-19T02:51:00.123291+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:12:51.408689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245260+0000","last_change":"2024-03-19T02:50:57.252827+0000","last_active":"2024-03-19T02:50:59.245260+0000","last_peered":"2024-03-19T02:50:59.245260+0000","last_clean":"2024-03-19T02:50:59.245260+0000","last_became_active":"2024-03-19T02:50:57.252502+0000","last_became_peered":"2024-03-19T02:50:57.252502+0000","last_unstale":"2024-03-19T02:50:59.245260+0000","last_undegraded":"2024-03-19T02:50:59.245260+0000","last_fullsized":"2024-03-19T02:50:59.245260+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:29:36.563533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"93'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122744+0000","last_change":"2024-03-19T02:50:51.797380+0000","last_active":"2024-03-19T02:51:00.122744+0000","last_peered":"2024-03-19T02:51:00.122744+0000","last_clean":"2024-03-19T02:51:00.122744+0000","last_became_active":"2024-03-19T02:50:51.797021+0000","last_became_peered":"2024-03-19T02:50:51.797021+0000","last_unstale":"2024-03-19T02:51:00.122744+0000","last_undegraded":"2024-03-19T02:51:00.122744+0000","last_fullsized":"2024-03-19T02:51:00.122744+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:25.400360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.832593+0000","last_change":"2024-03-19T02:50:52.232717+0000","last_active":"2024-03-19T02:50:59.832593+0000","last_peered":"2024-03-19T02:50:59.832593+0000","last_clean":"2024-03-19T02:50:59.832593+0000","last_became_active":"2024-03-19T02:50:52.232368+0000","last_became_peered":"2024-03-19T02:50:52.232368+0000","last_unstale":"2024-03-19T02:50:59.832593+0000","last_undegraded":"2024-03-19T02:50:59.832593+0000","last_fullsized":"2024-03-19T02:50:59.832593+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:55:38.027120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245388+0000","last_change":"2024-03-19T02:50:57.250840+0000","last_active":"2024-03-19T02:50:59.245388+0000","last_peered":"2024-03-19T02:50:59.245388+0000","last_clean":"2024-03-19T02:50:59.245388+0000","last_became_active":"2024-03-19T02:50:57.250333+0000","last_became_peered":"2024-03-19T02:50:57.250333+0000","last_unstale":"2024-03-19T02:50:59.245388+0000","last_undegraded":"2024-03-19T02:50:59.245388+0000","last_fullsized":"2024-03-19T02:50:59.245388+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:57:09.089939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"93'4","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:18.079717+0000","last_change":"2024-03-19T02:50:51.796889+0000","last_active":"2024-03-19T02:51:18.079717+0000","last_peered":"2024-03-19T02:51:18.079717+0000","last_clean":"2024-03-19T02:51:18.079717+0000","last_became_active":"2024-03-19T02:50:51.796570+0000","last_became_peered":"2024-03-19T02:50:51.796570+0000","last_unstale":"2024-03-19T02:51:18.079717+0000","last_undegraded":"2024-03-19T02:51:18.079717+0000","last_fullsized":"2024-03-19T02:51:18.079717+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:18:38.301905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123117+0000","last_change":"2024-03-19T02:50:52.224533+0000","last_active":"2024-03-19T02:51:00.123117+0000","last_peered":"2024-03-19T02:51:00.123117+0000","last_clean":"2024-03-19T02:51:00.123117+0000","last_became_active":"2024-03-19T02:50:52.224221+0000","last_became_peered":"2024-03-19T02:50:52.224221+0000","last_unstale":"2024-03-19T02:51:00.123117+0000","last_undegraded":"2024-03-19T02:51:00.123117+0000","last_fullsized":"2024-03-19T02:51:00.123117+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:36:43.335318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245257+0000","last_change":"2024-03-19T02:50:57.252949+0000","last_active":"2024-03-19T02:50:59.245257+0000","last_peered":"2024-03-19T02:50:59.245257+0000","last_clean":"2024-03-19T02:50:59.245257+0000","last_became_active":"2024-03-19T02:50:57.252563+0000","last_became_peered":"2024-03-19T02:50:57.252563+0000","last_unstale":"2024-03-19T02:50:59.245257+0000","last_undegraded":"2024-03-19T02:50:59.245257+0000","last_fullsized":"2024-03-19T02:50:59.245257+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:30:38.244158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122643+0000","last_change":"2024-03-19T02:50:51.794594+0000","last_active":"2024-03-19T02:51:00.122643+0000","last_peered":"2024-03-19T02:51:00.122643+0000","last_clean":"2024-03-19T02:51:00.122643+0000","last_became_active":"2024-03-19T02:50:51.794197+0000","last_became_peered":"2024-03-19T02:50:51.794197+0000","last_unstale":"2024-03-19T02:51:00.122643+0000","last_undegraded":"2024-03-19T02:51:00.122643+0000","last_fullsized":"2024-03-19T02:51:00.122643+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:58:44.534106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245719+0000","last_change":"2024-03-19T02:50:52.214167+0000","last_active":"2024-03-19T02:50:59.245719+0000","last_peered":"2024-03-19T02:50:59.245719+0000","last_clean":"2024-03-19T02:50:59.245719+0000","last_became_active":"2024-03-19T02:50:52.213873+0000","last_became_peered":"2024-03-19T02:50:52.213873+0000","last_unstale":"2024-03-19T02:50:59.245719+0000","last_undegraded":"2024-03-19T02:50:59.245719+0000","last_fullsized":"2024-03-19T02:50:59.245719+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:28:51.035477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.653068+0000","last_change":"2024-03-19T02:50:57.245139+0000","last_active":"2024-03-19T02:50:59.653068+0000","last_peered":"2024-03-19T02:50:59.653068+0000","last_clean":"2024-03-19T02:50:59.653068+0000","last_became_active":"2024-03-19T02:50:57.244636+0000","last_became_peered":"2024-03-19T02:50:57.244636+0000","last_unstale":"2024-03-19T02:50:59.653068+0000","last_undegraded":"2024-03-19T02:50:59.653068+0000","last_fullsized":"2024-03-19T02:50:59.653068+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:44:29.660221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'5","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:19.087270+0000","last_change":"2024-03-19T02:50:51.795969+0000","last_active":"2024-03-19T02:51:19.087270+0000","last_peered":"2024-03-19T02:51:19.087270+0000","last_clean":"2024-03-19T02:51:19.087270+0000","last_became_active":"2024-03-19T02:50:51.795628+0000","last_became_peered":"2024-03-19T02:50:51.795628+0000","last_unstale":"2024-03-19T02:51:19.087270+0000","last_undegraded":"2024-03-19T02:51:19.087270+0000","last_fullsized":"2024-03-19T02:51:19.087270+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:14:48.463282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123036+0000","last_change":"2024-03-19T02:50:52.223278+0000","last_active":"2024-03-19T02:51:00.123036+0000","last_peered":"2024-03-19T02:51:00.123036+0000","last_clean":"2024-03-19T02:51:00.123036+0000","last_became_active":"2024-03-19T02:50:52.222741+0000","last_became_peered":"2024-03-19T02:50:52.222741+0000","last_unstale":"2024-03-19T02:51:00.123036+0000","last_undegraded":"2024-03-19T02:51:00.123036+0000","last_fullsized":"2024-03-19T02:51:00.123036+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:32:09.841825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.653019+0000","last_change":"2024-03-19T02:50:57.253130+0000","last_active":"2024-03-19T02:50:59.653019+0000","last_peered":"2024-03-19T02:50:59.653019+0000","last_clean":"2024-03-19T02:50:59.653019+0000","last_became_active":"2024-03-19T02:50:57.252598+0000","last_became_peered":"2024-03-19T02:50:57.252598+0000","last_unstale":"2024-03-19T02:50:59.653019+0000","last_undegraded":"2024-03-19T02:50:59.653019+0000","last_fullsized":"2024-03-19T02:50:59.653019+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:09:38.183430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"93'3","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:18.079600+0000","last_change":"2024-03-19T02:50:51.794075+0000","last_active":"2024-03-19T02:51:18.079600+0000","last_peered":"2024-03-19T02:51:18.079600+0000","last_clean":"2024-03-19T02:51:18.079600+0000","last_became_active":"2024-03-19T02:50:51.793707+0000","last_became_peered":"2024-03-19T02:50:51.793707+0000","last_unstale":"2024-03-19T02:51:18.079600+0000","last_undegraded":"2024-03-19T02:51:18.079600+0000","last_fullsized":"2024-03-19T02:51:18.079600+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:52:07.180162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245655+0000","last_change":"2024-03-19T02:50:52.211112+0000","last_active":"2024-03-19T02:50:59.245655+0000","last_peered":"2024-03-19T02:50:59.245655+0000","last_clean":"2024-03-19T02:50:59.245655+0000","last_became_active":"2024-03-19T02:50:52.210790+0000","last_became_peered":"2024-03-19T02:50:52.210790+0000","last_unstale":"2024-03-19T02:50:59.245655+0000","last_undegraded":"2024-03-19T02:50:59.245655+0000","last_fullsized":"2024-03-19T02:50:59.245655+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:39:17.651585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.124501+0000","last_change":"2024-03-19T02:50:57.250397+0000","last_active":"2024-03-19T02:51:00.124501+0000","last_peered":"2024-03-19T02:51:00.124501+0000","last_clean":"2024-03-19T02:51:00.124501+0000","last_became_active":"2024-03-19T02:50:57.249968+0000","last_became_peered":"2024-03-19T02:50:57.249968+0000","last_unstale":"2024-03-19T02:51:00.124501+0000","last_undegraded":"2024-03-19T02:51:00.124501+0000","last_fullsized":"2024-03-19T02:51:00.124501+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:39.796799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"93'2","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:18.082766+0000","last_change":"2024-03-19T02:50:51.785347+0000","last_active":"2024-03-19T02:51:18.082766+0000","last_peered":"2024-03-19T02:51:18.082766+0000","last_clean":"2024-03-19T02:51:18.082766+0000","last_became_active":"2024-03-19T02:50:51.785004+0000","last_became_peered":"2024-03-19T02:50:51.785004+0000","last_unstale":"2024-03-19T02:51:18.082766+0000","last_undegraded":"2024-03-19T02:51:18.082766+0000","last_fullsized":"2024-03-19T02:51:18.082766+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:47.269503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123279+0000","last_change":"2024-03-19T02:50:52.221330+0000","last_active":"2024-03-19T02:51:00.123279+0000","last_peered":"2024-03-19T02:51:00.123279+0000","last_clean":"2024-03-19T02:51:00.123279+0000","last_became_active":"2024-03-19T02:50:52.219417+0000","last_became_peered":"2024-03-19T02:50:52.219417+0000","last_unstale":"2024-03-19T02:51:00.123279+0000","last_undegraded":"2024-03-19T02:51:00.123279+0000","last_fullsized":"2024-03-19T02:51:00.123279+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:28:37.856260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.832612+0000","last_change":"2024-03-19T02:50:57.258357+0000","last_active":"2024-03-19T02:50:59.832612+0000","last_peered":"2024-03-19T02:50:59.832612+0000","last_clean":"2024-03-19T02:50:59.832612+0000","last_became_active":"2024-03-19T02:50:57.257924+0000","last_became_peered":"2024-03-19T02:50:57.257924+0000","last_unstale":"2024-03-19T02:50:59.832612+0000","last_undegraded":"2024-03-19T02:50:59.832612+0000","last_fullsized":"2024-03-19T02:50:59.832612+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:46:05.340731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"93'1","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245236+0000","last_change":"2024-03-19T02:50:51.796425+0000","last_active":"2024-03-19T02:50:59.245236+0000","last_peered":"2024-03-19T02:50:59.245236+0000","last_clean":"2024-03-19T02:50:59.245236+0000","last_became_active":"2024-03-19T02:50:51.796139+0000","last_became_peered":"2024-03-19T02:50:51.796139+0000","last_unstale":"2024-03-19T02:50:59.245236+0000","last_undegraded":"2024-03-19T02:50:59.245236+0000","last_fullsized":"2024-03-19T02:50:59.245236+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:12:17.123556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122903+0000","last_change":"2024-03-19T02:50:52.219375+0000","last_active":"2024-03-19T02:51:00.122903+0000","last_peered":"2024-03-19T02:51:00.122903+0000","last_clean":"2024-03-19T02:51:00.122903+0000","last_became_active":"2024-03-19T02:50:52.219070+0000","last_became_peered":"2024-03-19T02:50:52.219070+0000","last_unstale":"2024-03-19T02:51:00.122903+0000","last_undegraded":"2024-03-19T02:51:00.122903+0000","last_fullsized":"2024-03-19T02:51:00.122903+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:31:58.847700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122836+0000","last_change":"2024-03-19T02:50:57.246502+0000","last_active":"2024-03-19T02:51:00.122836+0000","last_peered":"2024-03-19T02:51:00.122836+0000","last_clean":"2024-03-19T02:51:00.122836+0000","last_became_active":"2024-03-19T02:50:57.246089+0000","last_became_peered":"2024-03-19T02:50:57.246089+0000","last_unstale":"2024-03-19T02:51:00.122836+0000","last_undegraded":"2024-03-19T02:51:00.122836+0000","last_fullsized":"2024-03-19T02:51:00.122836+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:00.162308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'3","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:17.066781+0000","last_change":"2024-03-19T02:50:51.796448+0000","last_active":"2024-03-19T02:51:17.066781+0000","last_peered":"2024-03-19T02:51:17.066781+0000","last_clean":"2024-03-19T02:51:17.066781+0000","last_became_active":"2024-03-19T02:50:51.796132+0000","last_became_peered":"2024-03-19T02:50:51.796132+0000","last_unstale":"2024-03-19T02:51:17.066781+0000","last_undegraded":"2024-03-19T02:51:17.066781+0000","last_fullsized":"2024-03-19T02:51:17.066781+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:37:39.442772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123246+0000","last_change":"2024-03-19T02:50:52.223022+0000","last_active":"2024-03-19T02:51:00.123246+0000","last_peered":"2024-03-19T02:51:00.123246+0000","last_clean":"2024-03-19T02:51:00.123246+0000","last_became_active":"2024-03-19T02:50:52.221655+0000","last_became_peered":"2024-03-19T02:50:52.221655+0000","last_unstale":"2024-03-19T02:51:00.123246+0000","last_undegraded":"2024-03-19T02:51:00.123246+0000","last_fullsized":"2024-03-19T02:51:00.123246+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:34:36.054088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122905+0000","last_change":"2024-03-19T02:50:57.244003+0000","last_active":"2024-03-19T02:51:00.122905+0000","last_peered":"2024-03-19T02:51:00.122905+0000","last_clean":"2024-03-19T02:51:00.122905+0000","last_became_active":"2024-03-19T02:50:57.243567+0000","last_became_peered":"2024-03-19T02:50:57.243567+0000","last_unstale":"2024-03-19T02:51:00.122905+0000","last_undegraded":"2024-03-19T02:51:00.122905+0000","last_fullsized":"2024-03-19T02:51:00.122905+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:58:52.400237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"93'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:17.066294+0000","last_change":"2024-03-19T02:50:51.801259+0000","last_active":"2024-03-19T02:51:17.066294+0000","last_peered":"2024-03-19T02:51:17.066294+0000","last_clean":"2024-03-19T02:51:17.066294+0000","last_became_active":"2024-03-19T02:50:51.800868+0000","last_became_peered":"2024-03-19T02:50:51.800868+0000","last_unstale":"2024-03-19T02:51:17.066294+0000","last_undegraded":"2024-03-19T02:51:17.066294+0000","last_fullsized":"2024-03-19T02:51:17.066294+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:44:06.486687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.832608+0000","last_change":"2024-03-19T02:50:52.224973+0000","last_active":"2024-03-19T02:50:59.832608+0000","last_peered":"2024-03-19T02:50:59.832608+0000","last_clean":"2024-03-19T02:50:59.832608+0000","last_became_active":"2024-03-19T02:50:52.224636+0000","last_became_peered":"2024-03-19T02:50:52.224636+0000","last_unstale":"2024-03-19T02:50:59.832608+0000","last_undegraded":"2024-03-19T02:50:59.832608+0000","last_fullsized":"2024-03-19T02:50:59.832608+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:09:58.136715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245524+0000","last_change":"2024-03-19T02:50:57.238764+0000","last_active":"2024-03-19T02:50:59.245524+0000","last_peered":"2024-03-19T02:50:59.245524+0000","last_clean":"2024-03-19T02:50:59.245524+0000","last_became_active":"2024-03-19T02:50:57.238401+0000","last_became_peered":"2024-03-19T02:50:57.238401+0000","last_unstale":"2024-03-19T02:50:59.245524+0000","last_undegraded":"2024-03-19T02:50:59.245524+0000","last_fullsized":"2024-03-19T02:50:59.245524+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:20:46.131776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"93'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:17.066786+0000","last_change":"2024-03-19T02:50:51.797633+0000","last_active":"2024-03-19T02:51:17.066786+0000","last_peered":"2024-03-19T02:51:17.066786+0000","last_clean":"2024-03-19T02:51:17.066786+0000","last_became_active":"2024-03-19T02:50:51.797313+0000","last_became_peered":"2024-03-19T02:50:51.797313+0000","last_unstale":"2024-03-19T02:51:17.066786+0000","last_undegraded":"2024-03-19T02:51:17.066786+0000","last_fullsized":"2024-03-19T02:51:17.066786+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:43:04.659301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123225+0000","last_change":"2024-03-19T02:50:52.231486+0000","last_active":"2024-03-19T02:51:00.123225+0000","last_peered":"2024-03-19T02:51:00.123225+0000","last_clean":"2024-03-19T02:51:00.123225+0000","last_became_active":"2024-03-19T02:50:52.231172+0000","last_became_peered":"2024-03-19T02:50:52.231172+0000","last_unstale":"2024-03-19T02:51:00.123225+0000","last_undegraded":"2024-03-19T02:51:00.123225+0000","last_fullsized":"2024-03-19T02:51:00.123225+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:24:49.912633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123100+0000","last_change":"2024-03-19T02:50:57.247207+0000","last_active":"2024-03-19T02:51:00.123100+0000","last_peered":"2024-03-19T02:51:00.123100+0000","last_clean":"2024-03-19T02:51:00.123100+0000","last_became_active":"2024-03-19T02:50:57.246776+0000","last_became_peered":"2024-03-19T02:50:57.246776+0000","last_unstale":"2024-03-19T02:51:00.123100+0000","last_undegraded":"2024-03-19T02:51:00.123100+0000","last_fullsized":"2024-03-19T02:51:00.123100+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:02:24.118858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"93'15","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:19.433337+0000","last_change":"2024-03-19T02:50:51.792398+0000","last_active":"2024-03-19T02:51:19.433337+0000","last_peered":"2024-03-19T02:51:19.433337+0000","last_clean":"2024-03-19T02:51:19.433337+0000","last_became_active":"2024-03-19T02:50:51.792090+0000","last_became_peered":"2024-03-19T02:50:51.792090+0000","last_unstale":"2024-03-19T02:51:19.433337+0000","last_undegraded":"2024-03-19T02:51:19.433337+0000","last_fullsized":"2024-03-19T02:51:19.433337+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:02:58.175210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10442,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123668+0000","last_change":"2024-03-19T02:50:52.220755+0000","last_active":"2024-03-19T02:51:00.123668+0000","last_peered":"2024-03-19T02:51:00.123668+0000","last_clean":"2024-03-19T02:51:00.123668+0000","last_became_active":"2024-03-19T02:50:52.220453+0000","last_became_peered":"2024-03-19T02:50:52.220453+0000","last_unstale":"2024-03-19T02:51:00.123668+0000","last_undegraded":"2024-03-19T02:51:00.123668+0000","last_fullsized":"2024-03-19T02:51:00.123668+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:48:27.866863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.246094+0000","last_change":"2024-03-19T02:50:57.253843+0000","last_active":"2024-03-19T02:50:59.246094+0000","last_peered":"2024-03-19T02:50:59.246094+0000","last_clean":"2024-03-19T02:50:59.246094+0000","last_became_active":"2024-03-19T02:50:57.252633+0000","last_became_peered":"2024-03-19T02:50:57.252633+0000","last_unstale":"2024-03-19T02:50:59.246094+0000","last_undegraded":"2024-03-19T02:50:59.246094+0000","last_fullsized":"2024-03-19T02:50:59.246094+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:03:48.514219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"93'6","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:18.079468+0000","last_change":"2024-03-19T02:50:51.795041+0000","last_active":"2024-03-19T02:51:18.079468+0000","last_peered":"2024-03-19T02:51:18.079468+0000","last_clean":"2024-03-19T02:51:18.079468+0000","last_became_active":"2024-03-19T02:50:51.793485+0000","last_became_peered":"2024-03-19T02:50:51.793485+0000","last_unstale":"2024-03-19T02:51:18.079468+0000","last_undegraded":"2024-03-19T02:51:18.079468+0000","last_fullsized":"2024-03-19T02:51:18.079468+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:11:50.136394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245096+0000","last_change":"2024-03-19T02:50:52.227554+0000","last_active":"2024-03-19T02:50:59.245096+0000","last_peered":"2024-03-19T02:50:59.245096+0000","last_clean":"2024-03-19T02:50:59.245096+0000","last_became_active":"2024-03-19T02:50:52.227222+0000","last_became_peered":"2024-03-19T02:50:52.227222+0000","last_unstale":"2024-03-19T02:50:59.245096+0000","last_undegraded":"2024-03-19T02:50:59.245096+0000","last_fullsized":"2024-03-19T02:50:59.245096+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:56.323749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123844+0000","last_change":"2024-03-19T02:50:57.257504+0000","last_active":"2024-03-19T02:51:00.123844+0000","last_peered":"2024-03-19T02:51:00.123844+0000","last_clean":"2024-03-19T02:51:00.123844+0000","last_became_active":"2024-03-19T02:50:57.257120+0000","last_became_peered":"2024-03-19T02:50:57.257120+0000","last_unstale":"2024-03-19T02:51:00.123844+0000","last_undegraded":"2024-03-19T02:51:00.123844+0000","last_fullsized":"2024-03-19T02:51:00.123844+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:19:10.471666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"93'3","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:17.067046+0000","last_change":"2024-03-19T02:50:51.798049+0000","last_active":"2024-03-19T02:51:17.067046+0000","last_peered":"2024-03-19T02:51:17.067046+0000","last_clean":"2024-03-19T02:51:17.067046+0000","last_became_active":"2024-03-19T02:50:51.797134+0000","last_became_peered":"2024-03-19T02:50:51.797134+0000","last_unstale":"2024-03-19T02:51:17.067046+0000","last_undegraded":"2024-03-19T02:51:17.067046+0000","last_fullsized":"2024-03-19T02:51:17.067046+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:03:46.984240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122444+0000","last_change":"2024-03-19T02:50:52.219535+0000","last_active":"2024-03-19T02:51:00.122444+0000","last_peered":"2024-03-19T02:51:00.122444+0000","last_clean":"2024-03-19T02:51:00.122444+0000","last_became_active":"2024-03-19T02:50:52.219226+0000","last_became_peered":"2024-03-19T02:50:52.219226+0000","last_unstale":"2024-03-19T02:51:00.122444+0000","last_undegraded":"2024-03-19T02:51:00.122444+0000","last_fullsized":"2024-03-19T02:51:00.122444+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:27:26.875533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123841+0000","last_change":"2024-03-19T02:50:57.252873+0000","last_active":"2024-03-19T02:51:00.123841+0000","last_peered":"2024-03-19T02:51:00.123841+0000","last_clean":"2024-03-19T02:51:00.123841+0000","last_became_active":"2024-03-19T02:50:57.249551+0000","last_became_peered":"2024-03-19T02:50:57.249551+0000","last_unstale":"2024-03-19T02:51:00.123841+0000","last_undegraded":"2024-03-19T02:51:00.123841+0000","last_fullsized":"2024-03-19T02:51:00.123841+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:33:59.750606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"93'3","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:19.087751+0000","last_change":"2024-03-19T02:50:51.796478+0000","last_active":"2024-03-19T02:51:19.087751+0000","last_peered":"2024-03-19T02:51:19.087751+0000","last_clean":"2024-03-19T02:51:19.087751+0000","last_became_active":"2024-03-19T02:50:51.796126+0000","last_became_peered":"2024-03-19T02:50:51.796126+0000","last_unstale":"2024-03-19T02:51:19.087751+0000","last_undegraded":"2024-03-19T02:51:19.087751+0000","last_fullsized":"2024-03-19T02:51:19.087751+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:13:42.690481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122240+0000","last_change":"2024-03-19T02:50:52.228151+0000","last_active":"2024-03-19T02:51:00.122240+0000","last_peered":"2024-03-19T02:51:00.122240+0000","last_clean":"2024-03-19T02:51:00.122240+0000","last_became_active":"2024-03-19T02:50:52.227858+0000","last_became_peered":"2024-03-19T02:50:52.227858+0000","last_unstale":"2024-03-19T02:51:00.122240+0000","last_undegraded":"2024-03-19T02:51:00.122240+0000","last_fullsized":"2024-03-19T02:51:00.122240+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:10.285606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245298+0000","last_change":"2024-03-19T02:50:57.259269+0000","last_active":"2024-03-19T02:50:59.245298+0000","last_peered":"2024-03-19T02:50:59.245298+0000","last_clean":"2024-03-19T02:50:59.245298+0000","last_became_active":"2024-03-19T02:50:57.258778+0000","last_became_peered":"2024-03-19T02:50:57.258778+0000","last_unstale":"2024-03-19T02:50:59.245298+0000","last_undegraded":"2024-03-19T02:50:59.245298+0000","last_fullsized":"2024-03-19T02:50:59.245298+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:02:44.419512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"93'15","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:18.081718+0000","last_change":"2024-03-19T02:50:51.799246+0000","last_active":"2024-03-19T02:51:18.081718+0000","last_peered":"2024-03-19T02:51:18.081718+0000","last_clean":"2024-03-19T02:51:18.081718+0000","last_became_active":"2024-03-19T02:50:51.798935+0000","last_became_peered":"2024-03-19T02:50:51.798935+0000","last_unstale":"2024-03-19T02:51:18.081718+0000","last_undegraded":"2024-03-19T02:51:18.081718+0000","last_fullsized":"2024-03-19T02:51:18.081718+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:26:10.045267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245197+0000","last_change":"2024-03-19T02:50:52.215997+0000","last_active":"2024-03-19T02:50:59.245197+0000","last_peered":"2024-03-19T02:50:59.245197+0000","last_clean":"2024-03-19T02:50:59.245197+0000","last_became_active":"2024-03-19T02:50:52.215398+0000","last_became_peered":"2024-03-19T02:50:52.215398+0000","last_unstale":"2024-03-19T02:50:59.245197+0000","last_undegraded":"2024-03-19T02:50:59.245197+0000","last_fullsized":"2024-03-19T02:50:59.245197+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:49:14.010736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245331+0000","last_change":"2024-03-19T02:50:57.247254+0000","last_active":"2024-03-19T02:50:59.245331+0000","last_peered":"2024-03-19T02:50:59.245331+0000","last_clean":"2024-03-19T02:50:59.245331+0000","last_became_active":"2024-03-19T02:50:57.246861+0000","last_became_peered":"2024-03-19T02:50:57.246861+0000","last_unstale":"2024-03-19T02:50:59.245331+0000","last_undegraded":"2024-03-19T02:50:59.245331+0000","last_fullsized":"2024-03-19T02:50:59.245331+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:47:20.933550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"93'2","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:19.077099+0000","last_change":"2024-03-19T02:50:51.796246+0000","last_active":"2024-03-19T02:51:19.077099+0000","last_peered":"2024-03-19T02:51:19.077099+0000","last_clean":"2024-03-19T02:51:19.077099+0000","last_became_active":"2024-03-19T02:50:51.795823+0000","last_became_peered":"2024-03-19T02:50:51.795823+0000","last_unstale":"2024-03-19T02:51:19.077099+0000","last_undegraded":"2024-03-19T02:51:19.077099+0000","last_fullsized":"2024-03-19T02:51:19.077099+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:36:02.220891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.124142+0000","last_change":"2024-03-19T02:50:52.224119+0000","last_active":"2024-03-19T02:51:00.124142+0000","last_peered":"2024-03-19T02:51:00.124142+0000","last_clean":"2024-03-19T02:51:00.124142+0000","last_became_active":"2024-03-19T02:50:52.223773+0000","last_became_peered":"2024-03-19T02:50:52.223773+0000","last_unstale":"2024-03-19T02:51:00.124142+0000","last_undegraded":"2024-03-19T02:51:00.124142+0000","last_fullsized":"2024-03-19T02:51:00.124142+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:56:03.098858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123038+0000","last_change":"2024-03-19T02:50:57.652515+0000","last_active":"2024-03-19T02:51:00.123038+0000","last_peered":"2024-03-19T02:51:00.123038+0000","last_clean":"2024-03-19T02:51:00.123038+0000","last_became_active":"2024-03-19T02:50:57.651808+0000","last_became_peered":"2024-03-19T02:50:57.651808+0000","last_unstale":"2024-03-19T02:51:00.123038+0000","last_undegraded":"2024-03-19T02:51:00.123038+0000","last_fullsized":"2024-03-19T02:51:00.123038+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:16:34.550457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123394+0000","last_change":"2024-03-19T02:50:51.794282+0000","last_active":"2024-03-19T02:51:00.123394+0000","last_peered":"2024-03-19T02:51:00.123394+0000","last_clean":"2024-03-19T02:51:00.123394+0000","last_became_active":"2024-03-19T02:50:51.793947+0000","last_became_peered":"2024-03-19T02:50:51.793947+0000","last_unstale":"2024-03-19T02:51:00.123394+0000","last_undegraded":"2024-03-19T02:51:00.123394+0000","last_fullsized":"2024-03-19T02:51:00.123394+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:08:54.764915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.037018+0000","last_change":"2024-03-19T02:50:52.221707+0000","last_active":"2024-03-19T02:51:00.037018+0000","last_peered":"2024-03-19T02:51:00.037018+0000","last_clean":"2024-03-19T02:51:00.037018+0000","last_became_active":"2024-03-19T02:50:52.221417+0000","last_became_peered":"2024-03-19T02:50:52.221417+0000","last_unstale":"2024-03-19T02:51:00.037018+0000","last_undegraded":"2024-03-19T02:51:00.037018+0000","last_fullsized":"2024-03-19T02:51:00.037018+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:59:27.528346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245401+0000","last_change":"2024-03-19T02:50:57.257552+0000","last_active":"2024-03-19T02:50:59.245401+0000","last_peered":"2024-03-19T02:50:59.245401+0000","last_clean":"2024-03-19T02:50:59.245401+0000","last_became_active":"2024-03-19T02:50:57.257121+0000","last_became_peered":"2024-03-19T02:50:57.257121+0000","last_unstale":"2024-03-19T02:50:59.245401+0000","last_undegraded":"2024-03-19T02:50:59.245401+0000","last_fullsized":"2024-03-19T02:50:59.245401+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:43:47.145344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"93'1","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.121899+0000","last_change":"2024-03-19T02:50:51.804890+0000","last_active":"2024-03-19T02:51:00.121899+0000","last_peered":"2024-03-19T02:51:00.121899+0000","last_clean":"2024-03-19T02:51:00.121899+0000","last_became_active":"2024-03-19T02:50:51.804545+0000","last_became_peered":"2024-03-19T02:50:51.804545+0000","last_unstale":"2024-03-19T02:51:00.121899+0000","last_undegraded":"2024-03-19T02:51:00.121899+0000","last_fullsized":"2024-03-19T02:51:00.121899+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:16:14.945372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.036929+0000","last_change":"2024-03-19T02:50:52.224849+0000","last_active":"2024-03-19T02:51:00.036929+0000","last_peered":"2024-03-19T02:51:00.036929+0000","last_clean":"2024-03-19T02:51:00.036929+0000","last_became_active":"2024-03-19T02:50:52.224550+0000","last_became_peered":"2024-03-19T02:50:52.224550+0000","last_unstale":"2024-03-19T02:51:00.036929+0000","last_undegraded":"2024-03-19T02:51:00.036929+0000","last_fullsized":"2024-03-19T02:51:00.036929+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:28:59.683849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245196+0000","last_change":"2024-03-19T02:50:57.652425+0000","last_active":"2024-03-19T02:50:59.245196+0000","last_peered":"2024-03-19T02:50:59.245196+0000","last_clean":"2024-03-19T02:50:59.245196+0000","last_became_active":"2024-03-19T02:50:57.651975+0000","last_became_peered":"2024-03-19T02:50:57.651975+0000","last_unstale":"2024-03-19T02:50:59.245196+0000","last_undegraded":"2024-03-19T02:50:59.245196+0000","last_fullsized":"2024-03-19T02:50:59.245196+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:07:20.587918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245321+0000","last_change":"2024-03-19T02:50:52.218207+0000","last_active":"2024-03-19T02:50:59.245321+0000","last_peered":"2024-03-19T02:50:59.245321+0000","last_clean":"2024-03-19T02:50:59.245321+0000","last_became_active":"2024-03-19T02:50:52.217012+0000","last_became_peered":"2024-03-19T02:50:52.217012+0000","last_unstale":"2024-03-19T02:50:59.245321+0000","last_undegraded":"2024-03-19T02:50:59.245321+0000","last_fullsized":"2024-03-19T02:50:59.245321+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:07:47.366481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:19.085606+0000","last_change":"2024-03-19T02:50:51.801405+0000","last_active":"2024-03-19T02:51:19.085606+0000","last_peered":"2024-03-19T02:51:19.085606+0000","last_clean":"2024-03-19T02:51:19.085606+0000","last_became_active":"2024-03-19T02:50:51.801094+0000","last_became_peered":"2024-03-19T02:50:51.801094+0000","last_unstale":"2024-03-19T02:51:19.085606+0000","last_undegraded":"2024-03-19T02:51:19.085606+0000","last_fullsized":"2024-03-19T02:51:19.085606+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:53:39.534977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"93'1","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122679+0000","last_change":"2024-03-19T02:50:57.245508+0000","last_active":"2024-03-19T02:51:00.122679+0000","last_peered":"2024-03-19T02:51:00.122679+0000","last_clean":"2024-03-19T02:51:00.122679+0000","last_became_active":"2024-03-19T02:50:57.245109+0000","last_became_peered":"2024-03-19T02:50:57.245109+0000","last_unstale":"2024-03-19T02:51:00.122679+0000","last_undegraded":"2024-03-19T02:51:00.122679+0000","last_fullsized":"2024-03-19T02:51:00.122679+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:25:09.085074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122705+0000","last_change":"2024-03-19T02:50:52.221852+0000","last_active":"2024-03-19T02:51:00.122705+0000","last_peered":"2024-03-19T02:51:00.122705+0000","last_clean":"2024-03-19T02:51:00.122705+0000","last_became_active":"2024-03-19T02:50:52.221535+0000","last_became_peered":"2024-03-19T02:50:52.221535+0000","last_unstale":"2024-03-19T02:51:00.122705+0000","last_undegraded":"2024-03-19T02:51:00.122705+0000","last_fullsized":"2024-03-19T02:51:00.122705+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:37:18.030714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"93'3","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:19.083671+0000","last_change":"2024-03-19T02:50:51.790231+0000","last_active":"2024-03-19T02:51:19.083671+0000","last_peered":"2024-03-19T02:51:19.083671+0000","last_clean":"2024-03-19T02:51:19.083671+0000","last_became_active":"2024-03-19T02:50:51.789867+0000","last_became_peered":"2024-03-19T02:50:51.789867+0000","last_unstale":"2024-03-19T02:51:19.083671+0000","last_undegraded":"2024-03-19T02:51:19.083671+0000","last_fullsized":"2024-03-19T02:51:19.083671+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:46.099191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":361,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122201+0000","last_change":"2024-03-19T02:49:38.697673+0000","last_active":"2024-03-19T02:51:00.122201+0000","last_peered":"2024-03-19T02:51:00.122201+0000","last_clean":"2024-03-19T02:51:00.122201+0000","last_became_active":"2024-03-19T02:49:38.697161+0000","last_became_peered":"2024-03-19T02:49:38.697161+0000","last_unstale":"2024-03-19T02:51:00.122201+0000","last_undegraded":"2024-03-19T02:51:00.122201+0000","last_fullsized":"2024-03-19T02:51:00.122201+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:46:10.013962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:46:10.013962+0000","last_clean_scrub_stamp":"2024-03-19T02:46:10.013962+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:59:35.179938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245940+0000","last_change":"2024-03-19T02:50:57.652428+0000","last_active":"2024-03-19T02:50:59.245940+0000","last_peered":"2024-03-19T02:50:59.245940+0000","last_clean":"2024-03-19T02:50:59.245940+0000","last_became_active":"2024-03-19T02:50:57.651864+0000","last_became_peered":"2024-03-19T02:50:57.651864+0000","last_unstale":"2024-03-19T02:50:59.245940+0000","last_undegraded":"2024-03-19T02:50:59.245940+0000","last_fullsized":"2024-03-19T02:50:59.245940+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:52:19.007175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.244774+0000","last_change":"2024-03-19T02:50:52.218728+0000","last_active":"2024-03-19T02:50:59.244774+0000","last_peered":"2024-03-19T02:50:59.244774+0000","last_clean":"2024-03-19T02:50:59.244774+0000","last_became_active":"2024-03-19T02:50:52.218407+0000","last_became_peered":"2024-03-19T02:50:52.218407+0000","last_unstale":"2024-03-19T02:50:59.244774+0000","last_undegraded":"2024-03-19T02:50:59.244774+0000","last_fullsized":"2024-03-19T02:50:59.244774+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:30:49.304034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"93'3","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:18.079179+0000","last_change":"2024-03-19T02:50:51.802057+0000","last_active":"2024-03-19T02:51:18.079179+0000","last_peered":"2024-03-19T02:51:18.079179+0000","last_clean":"2024-03-19T02:51:18.079179+0000","last_became_active":"2024-03-19T02:50:51.801754+0000","last_became_peered":"2024-03-19T02:50:51.801754+0000","last_unstale":"2024-03-19T02:51:18.079179+0000","last_undegraded":"2024-03-19T02:51:18.079179+0000","last_fullsized":"2024-03-19T02:51:18.079179+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:23:15.401822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123530+0000","last_change":"2024-03-19T02:50:57.260303+0000","last_active":"2024-03-19T02:51:00.123530+0000","last_peered":"2024-03-19T02:51:00.123530+0000","last_clean":"2024-03-19T02:51:00.123530+0000","last_became_active":"2024-03-19T02:50:57.259792+0000","last_became_peered":"2024-03-19T02:50:57.259792+0000","last_unstale":"2024-03-19T02:51:00.123530+0000","last_undegraded":"2024-03-19T02:51:00.123530+0000","last_fullsized":"2024-03-19T02:51:00.123530+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:03.072161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"93'1","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:17.067713+0000","last_change":"2024-03-19T02:50:51.797949+0000","last_active":"2024-03-19T02:51:17.067713+0000","last_peered":"2024-03-19T02:51:17.067713+0000","last_clean":"2024-03-19T02:51:17.067713+0000","last_became_active":"2024-03-19T02:50:51.797522+0000","last_became_peered":"2024-03-19T02:50:51.797522+0000","last_unstale":"2024-03-19T02:51:17.067713+0000","last_undegraded":"2024-03-19T02:51:17.067713+0000","last_fullsized":"2024-03-19T02:51:17.067713+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:34:06.127993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245412+0000","last_change":"2024-03-19T02:50:52.224095+0000","last_active":"2024-03-19T02:50:59.245412+0000","last_peered":"2024-03-19T02:50:59.245412+0000","last_clean":"2024-03-19T02:50:59.245412+0000","last_became_active":"2024-03-19T02:50:52.223764+0000","last_became_peered":"2024-03-19T02:50:52.223764+0000","last_unstale":"2024-03-19T02:50:59.245412+0000","last_undegraded":"2024-03-19T02:50:59.245412+0000","last_fullsized":"2024-03-19T02:50:59.245412+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:07:41.503162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123123+0000","last_change":"2024-03-19T02:50:57.652274+0000","last_active":"2024-03-19T02:51:00.123123+0000","last_peered":"2024-03-19T02:51:00.123123+0000","last_clean":"2024-03-19T02:51:00.123123+0000","last_became_active":"2024-03-19T02:50:57.651885+0000","last_became_peered":"2024-03-19T02:50:57.651885+0000","last_unstale":"2024-03-19T02:51:00.123123+0000","last_undegraded":"2024-03-19T02:51:00.123123+0000","last_fullsized":"2024-03-19T02:51:00.123123+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:03:02.081143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'3","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:18.083250+0000","last_change":"2024-03-19T02:50:51.796308+0000","last_active":"2024-03-19T02:51:18.083250+0000","last_peered":"2024-03-19T02:51:18.083250+0000","last_clean":"2024-03-19T02:51:18.083250+0000","last_became_active":"2024-03-19T02:50:51.795990+0000","last_became_peered":"2024-03-19T02:50:51.795990+0000","last_unstale":"2024-03-19T02:51:18.083250+0000","last_undegraded":"2024-03-19T02:51:18.083250+0000","last_fullsized":"2024-03-19T02:51:18.083250+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:45:53.608817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123567+0000","last_change":"2024-03-19T02:50:52.221983+0000","last_active":"2024-03-19T02:51:00.123567+0000","last_peered":"2024-03-19T02:51:00.123567+0000","last_clean":"2024-03-19T02:51:00.123567+0000","last_became_active":"2024-03-19T02:50:52.220605+0000","last_became_peered":"2024-03-19T02:50:52.220605+0000","last_unstale":"2024-03-19T02:51:00.123567+0000","last_undegraded":"2024-03-19T02:51:00.123567+0000","last_fullsized":"2024-03-19T02:51:00.123567+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:12.834159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.652688+0000","last_change":"2024-03-19T02:50:57.242428+0000","last_active":"2024-03-19T02:50:59.652688+0000","last_peered":"2024-03-19T02:50:59.652688+0000","last_clean":"2024-03-19T02:50:59.652688+0000","last_became_active":"2024-03-19T02:50:57.242007+0000","last_became_peered":"2024-03-19T02:50:57.242007+0000","last_unstale":"2024-03-19T02:50:59.652688+0000","last_undegraded":"2024-03-19T02:50:59.652688+0000","last_fullsized":"2024-03-19T02:50:59.652688+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:23:30.987761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"93'4","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:17.066389+0000","last_change":"2024-03-19T02:50:51.791342+0000","last_active":"2024-03-19T02:51:17.066389+0000","last_peered":"2024-03-19T02:51:17.066389+0000","last_clean":"2024-03-19T02:51:17.066389+0000","last_became_active":"2024-03-19T02:50:51.791039+0000","last_became_peered":"2024-03-19T02:50:51.791039+0000","last_unstale":"2024-03-19T02:51:17.066389+0000","last_undegraded":"2024-03-19T02:51:17.066389+0000","last_fullsized":"2024-03-19T02:51:17.066389+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:50:38.216623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.832528+0000","last_change":"2024-03-19T02:50:52.224303+0000","last_active":"2024-03-19T02:50:59.832528+0000","last_peered":"2024-03-19T02:50:59.832528+0000","last_clean":"2024-03-19T02:50:59.832528+0000","last_became_active":"2024-03-19T02:50:52.223942+0000","last_became_peered":"2024-03-19T02:50:52.223942+0000","last_unstale":"2024-03-19T02:50:59.832528+0000","last_undegraded":"2024-03-19T02:50:59.832528+0000","last_fullsized":"2024-03-19T02:50:59.832528+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:57:04.175971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245562+0000","last_change":"2024-03-19T02:50:57.652126+0000","last_active":"2024-03-19T02:50:59.245562+0000","last_peered":"2024-03-19T02:50:59.245562+0000","last_clean":"2024-03-19T02:50:59.245562+0000","last_became_active":"2024-03-19T02:50:57.651713+0000","last_became_peered":"2024-03-19T02:50:57.651713+0000","last_unstale":"2024-03-19T02:50:59.245562+0000","last_undegraded":"2024-03-19T02:50:59.245562+0000","last_fullsized":"2024-03-19T02:50:59.245562+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:28:51.077437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"93'2","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:17.067090+0000","last_change":"2024-03-19T02:50:51.795793+0000","last_active":"2024-03-19T02:51:17.067090+0000","last_peered":"2024-03-19T02:51:17.067090+0000","last_clean":"2024-03-19T02:51:17.067090+0000","last_became_active":"2024-03-19T02:50:51.795448+0000","last_became_peered":"2024-03-19T02:50:51.795448+0000","last_unstale":"2024-03-19T02:51:17.067090+0000","last_undegraded":"2024-03-19T02:51:17.067090+0000","last_fullsized":"2024-03-19T02:51:17.067090+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:33:11.399247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122822+0000","last_change":"2024-03-19T02:50:52.221681+0000","last_active":"2024-03-19T02:51:00.122822+0000","last_peered":"2024-03-19T02:51:00.122822+0000","last_clean":"2024-03-19T02:51:00.122822+0000","last_became_active":"2024-03-19T02:50:52.220987+0000","last_became_peered":"2024-03-19T02:50:52.220987+0000","last_unstale":"2024-03-19T02:51:00.122822+0000","last_undegraded":"2024-03-19T02:51:00.122822+0000","last_fullsized":"2024-03-19T02:51:00.122822+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:16:36.561143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123978+0000","last_change":"2024-03-19T02:50:57.249830+0000","last_active":"2024-03-19T02:51:00.123978+0000","last_peered":"2024-03-19T02:51:00.123978+0000","last_clean":"2024-03-19T02:51:00.123978+0000","last_became_active":"2024-03-19T02:50:57.248879+0000","last_became_peered":"2024-03-19T02:50:57.248879+0000","last_unstale":"2024-03-19T02:51:00.123978+0000","last_undegraded":"2024-03-19T02:51:00.123978+0000","last_fullsized":"2024-03-19T02:51:00.123978+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:58:43.225841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"93'3","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:17.067683+0000","last_change":"2024-03-19T02:50:51.801374+0000","last_active":"2024-03-19T02:51:17.067683+0000","last_peered":"2024-03-19T02:51:17.067683+0000","last_clean":"2024-03-19T02:51:17.067683+0000","last_became_active":"2024-03-19T02:50:51.801041+0000","last_became_peered":"2024-03-19T02:50:51.801041+0000","last_unstale":"2024-03-19T02:51:17.067683+0000","last_undegraded":"2024-03-19T02:51:17.067683+0000","last_fullsized":"2024-03-19T02:51:17.067683+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:56:14.671718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123333+0000","last_change":"2024-03-19T02:50:52.225915+0000","last_active":"2024-03-19T02:51:00.123333+0000","last_peered":"2024-03-19T02:51:00.123333+0000","last_clean":"2024-03-19T02:51:00.123333+0000","last_became_active":"2024-03-19T02:50:52.225606+0000","last_became_peered":"2024-03-19T02:50:52.225606+0000","last_unstale":"2024-03-19T02:51:00.123333+0000","last_undegraded":"2024-03-19T02:51:00.123333+0000","last_fullsized":"2024-03-19T02:51:00.123333+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:27:29.549255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245549+0000","last_change":"2024-03-19T02:50:57.240747+0000","last_active":"2024-03-19T02:50:59.245549+0000","last_peered":"2024-03-19T02:50:59.245549+0000","last_clean":"2024-03-19T02:50:59.245549+0000","last_became_active":"2024-03-19T02:50:57.240373+0000","last_became_peered":"2024-03-19T02:50:57.240373+0000","last_unstale":"2024-03-19T02:50:59.245549+0000","last_undegraded":"2024-03-19T02:50:59.245549+0000","last_fullsized":"2024-03-19T02:50:59.245549+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:22:04.991497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.652699+0000","last_change":"2024-03-19T02:50:51.794071+0000","last_active":"2024-03-19T02:50:59.652699+0000","last_peered":"2024-03-19T02:50:59.652699+0000","last_clean":"2024-03-19T02:50:59.652699+0000","last_became_active":"2024-03-19T02:50:51.793570+0000","last_became_peered":"2024-03-19T02:50:51.793570+0000","last_unstale":"2024-03-19T02:50:59.652699+0000","last_undegraded":"2024-03-19T02:50:59.652699+0000","last_fullsized":"2024-03-19T02:50:59.652699+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:34:20.456515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123180+0000","last_change":"2024-03-19T02:50:52.223341+0000","last_active":"2024-03-19T02:51:00.123180+0000","last_peered":"2024-03-19T02:51:00.123180+0000","last_clean":"2024-03-19T02:51:00.123180+0000","last_became_active":"2024-03-19T02:50:52.220280+0000","last_became_peered":"2024-03-19T02:50:52.220280+0000","last_unstale":"2024-03-19T02:51:00.123180+0000","last_undegraded":"2024-03-19T02:51:00.123180+0000","last_fullsized":"2024-03-19T02:51:00.123180+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:04:15.830811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245369+0000","last_change":"2024-03-19T02:50:57.651293+0000","last_active":"2024-03-19T02:50:59.245369+0000","last_peered":"2024-03-19T02:50:59.245369+0000","last_clean":"2024-03-19T02:50:59.245369+0000","last_became_active":"2024-03-19T02:50:57.650893+0000","last_became_peered":"2024-03-19T02:50:57.650893+0000","last_unstale":"2024-03-19T02:50:59.245369+0000","last_undegraded":"2024-03-19T02:50:59.245369+0000","last_fullsized":"2024-03-19T02:50:59.245369+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:33:35.984728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"93'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:17.067529+0000","last_change":"2024-03-19T02:50:51.798297+0000","last_active":"2024-03-19T02:51:17.067529+0000","last_peered":"2024-03-19T02:51:17.067529+0000","last_clean":"2024-03-19T02:51:17.067529+0000","last_became_active":"2024-03-19T02:50:51.797516+0000","last_became_peered":"2024-03-19T02:50:51.797516+0000","last_unstale":"2024-03-19T02:51:17.067529+0000","last_undegraded":"2024-03-19T02:51:17.067529+0000","last_fullsized":"2024-03-19T02:51:17.067529+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:09:42.295094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122133+0000","last_change":"2024-03-19T02:50:52.224011+0000","last_active":"2024-03-19T02:51:00.122133+0000","last_peered":"2024-03-19T02:51:00.122133+0000","last_clean":"2024-03-19T02:51:00.122133+0000","last_became_active":"2024-03-19T02:50:52.223708+0000","last_became_peered":"2024-03-19T02:50:52.223708+0000","last_unstale":"2024-03-19T02:51:00.122133+0000","last_undegraded":"2024-03-19T02:51:00.122133+0000","last_fullsized":"2024-03-19T02:51:00.122133+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:17:49.305587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123035+0000","last_change":"2024-03-19T02:50:57.251134+0000","last_active":"2024-03-19T02:51:00.123035+0000","last_peered":"2024-03-19T02:51:00.123035+0000","last_clean":"2024-03-19T02:51:00.123035+0000","last_became_active":"2024-03-19T02:50:57.250761+0000","last_became_peered":"2024-03-19T02:50:57.250761+0000","last_unstale":"2024-03-19T02:51:00.123035+0000","last_undegraded":"2024-03-19T02:51:00.123035+0000","last_fullsized":"2024-03-19T02:51:00.123035+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:09:26.860235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"93'3","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:18.081704+0000","last_change":"2024-03-19T02:50:51.794532+0000","last_active":"2024-03-19T02:51:18.081704+0000","last_peered":"2024-03-19T02:51:18.081704+0000","last_clean":"2024-03-19T02:51:18.081704+0000","last_became_active":"2024-03-19T02:50:51.794215+0000","last_became_peered":"2024-03-19T02:50:51.794215+0000","last_unstale":"2024-03-19T02:51:18.081704+0000","last_undegraded":"2024-03-19T02:51:18.081704+0000","last_fullsized":"2024-03-19T02:51:18.081704+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:19:28.060014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122736+0000","last_change":"2024-03-19T02:50:52.224776+0000","last_active":"2024-03-19T02:51:00.122736+0000","last_peered":"2024-03-19T02:51:00.122736+0000","last_clean":"2024-03-19T02:51:00.122736+0000","last_became_active":"2024-03-19T02:50:52.224442+0000","last_became_peered":"2024-03-19T02:50:52.224442+0000","last_unstale":"2024-03-19T02:51:00.122736+0000","last_undegraded":"2024-03-19T02:51:00.122736+0000","last_fullsized":"2024-03-19T02:51:00.122736+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:11:13.722005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.832239+0000","last_change":"2024-03-19T02:50:57.250322+0000","last_active":"2024-03-19T02:50:59.832239+0000","last_peered":"2024-03-19T02:50:59.832239+0000","last_clean":"2024-03-19T02:50:59.832239+0000","last_became_active":"2024-03-19T02:50:57.249890+0000","last_became_peered":"2024-03-19T02:50:57.249890+0000","last_unstale":"2024-03-19T02:50:59.832239+0000","last_undegraded":"2024-03-19T02:50:59.832239+0000","last_fullsized":"2024-03-19T02:50:59.832239+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:36:14.169985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245870+0000","last_change":"2024-03-19T02:50:51.794898+0000","last_active":"2024-03-19T02:50:59.245870+0000","last_peered":"2024-03-19T02:50:59.245870+0000","last_clean":"2024-03-19T02:50:59.245870+0000","last_became_active":"2024-03-19T02:50:51.794562+0000","last_became_peered":"2024-03-19T02:50:51.794562+0000","last_unstale":"2024-03-19T02:50:59.245870+0000","last_undegraded":"2024-03-19T02:50:59.245870+0000","last_fullsized":"2024-03-19T02:50:59.245870+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:17:45.239395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122369+0000","last_change":"2024-03-19T02:50:52.227427+0000","last_active":"2024-03-19T02:51:00.122369+0000","last_peered":"2024-03-19T02:51:00.122369+0000","last_clean":"2024-03-19T02:51:00.122369+0000","last_became_active":"2024-03-19T02:50:52.227111+0000","last_became_peered":"2024-03-19T02:50:52.227111+0000","last_unstale":"2024-03-19T02:51:00.122369+0000","last_undegraded":"2024-03-19T02:51:00.122369+0000","last_fullsized":"2024-03-19T02:51:00.122369+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:40:53.295454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245066+0000","last_change":"2024-03-19T02:50:57.244449+0000","last_active":"2024-03-19T02:50:59.245066+0000","last_peered":"2024-03-19T02:50:59.245066+0000","last_clean":"2024-03-19T02:50:59.245066+0000","last_became_active":"2024-03-19T02:50:57.243991+0000","last_became_peered":"2024-03-19T02:50:57.243991+0000","last_unstale":"2024-03-19T02:50:59.245066+0000","last_undegraded":"2024-03-19T02:50:59.245066+0000","last_fullsized":"2024-03-19T02:50:59.245066+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:50:45.265773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245271+0000","last_change":"2024-03-19T02:50:51.799834+0000","last_active":"2024-03-19T02:50:59.245271+0000","last_peered":"2024-03-19T02:50:59.245271+0000","last_clean":"2024-03-19T02:50:59.245271+0000","last_became_active":"2024-03-19T02:50:51.799563+0000","last_became_peered":"2024-03-19T02:50:51.799563+0000","last_unstale":"2024-03-19T02:50:59.245271+0000","last_undegraded":"2024-03-19T02:50:59.245271+0000","last_fullsized":"2024-03-19T02:50:59.245271+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:51:00.284454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122953+0000","last_change":"2024-03-19T02:50:52.222710+0000","last_active":"2024-03-19T02:51:00.122953+0000","last_peered":"2024-03-19T02:51:00.122953+0000","last_clean":"2024-03-19T02:51:00.122953+0000","last_became_active":"2024-03-19T02:50:52.222321+0000","last_became_peered":"2024-03-19T02:50:52.222321+0000","last_unstale":"2024-03-19T02:51:00.122953+0000","last_undegraded":"2024-03-19T02:51:00.122953+0000","last_fullsized":"2024-03-19T02:51:00.122953+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:05:27.527696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123595+0000","last_change":"2024-03-19T02:50:52.226631+0000","last_active":"2024-03-19T02:51:00.123595+0000","last_peered":"2024-03-19T02:51:00.123595+0000","last_clean":"2024-03-19T02:51:00.123595+0000","last_became_active":"2024-03-19T02:50:52.226294+0000","last_became_peered":"2024-03-19T02:50:52.226294+0000","last_unstale":"2024-03-19T02:51:00.123595+0000","last_undegraded":"2024-03-19T02:51:00.123595+0000","last_fullsized":"2024-03-19T02:51:00.123595+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:01:31.350063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.653221+0000","last_change":"2024-03-19T02:50:52.221138+0000","last_active":"2024-03-19T02:50:59.653221+0000","last_peered":"2024-03-19T02:50:59.653221+0000","last_clean":"2024-03-19T02:50:59.653221+0000","last_became_active":"2024-03-19T02:50:52.220730+0000","last_became_peered":"2024-03-19T02:50:52.220730+0000","last_unstale":"2024-03-19T02:50:59.653221+0000","last_undegraded":"2024-03-19T02:50:59.653221+0000","last_fullsized":"2024-03-19T02:50:59.653221+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:26:13.059986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.652639+0000","last_change":"2024-03-19T02:50:52.223567+0000","last_active":"2024-03-19T02:50:59.652639+0000","last_peered":"2024-03-19T02:50:59.652639+0000","last_clean":"2024-03-19T02:50:59.652639+0000","last_became_active":"2024-03-19T02:50:52.223174+0000","last_became_peered":"2024-03-19T02:50:52.223174+0000","last_unstale":"2024-03-19T02:50:59.652639+0000","last_undegraded":"2024-03-19T02:50:59.652639+0000","last_fullsized":"2024-03-19T02:50:59.652639+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:23:06.383584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.832350+0000","last_change":"2024-03-19T02:50:52.229916+0000","last_active":"2024-03-19T02:50:59.832350+0000","last_peered":"2024-03-19T02:50:59.832350+0000","last_clean":"2024-03-19T02:50:59.832350+0000","last_became_active":"2024-03-19T02:50:52.229597+0000","last_became_peered":"2024-03-19T02:50:52.229597+0000","last_unstale":"2024-03-19T02:50:59.832350+0000","last_undegraded":"2024-03-19T02:50:59.832350+0000","last_fullsized":"2024-03-19T02:50:59.832350+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:19.331893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245325+0000","last_change":"2024-03-19T02:50:52.223331+0000","last_active":"2024-03-19T02:50:59.245325+0000","last_peered":"2024-03-19T02:50:59.245325+0000","last_clean":"2024-03-19T02:50:59.245325+0000","last_became_active":"2024-03-19T02:50:52.222975+0000","last_became_peered":"2024-03-19T02:50:52.222975+0000","last_unstale":"2024-03-19T02:50:59.245325+0000","last_undegraded":"2024-03-19T02:50:59.245325+0000","last_fullsized":"2024-03-19T02:50:59.245325+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:26:23.023749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.124357+0000","last_change":"2024-03-19T02:50:52.218073+0000","last_active":"2024-03-19T02:51:00.124357+0000","last_peered":"2024-03-19T02:51:00.124357+0000","last_clean":"2024-03-19T02:51:00.124357+0000","last_became_active":"2024-03-19T02:50:52.217742+0000","last_became_peered":"2024-03-19T02:50:52.217742+0000","last_unstale":"2024-03-19T02:51:00.124357+0000","last_undegraded":"2024-03-19T02:51:00.124357+0000","last_fullsized":"2024-03-19T02:51:00.124357+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:08:51.053153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.653317+0000","last_change":"2024-03-19T02:50:52.222108+0000","last_active":"2024-03-19T02:50:59.653317+0000","last_peered":"2024-03-19T02:50:59.653317+0000","last_clean":"2024-03-19T02:50:59.653317+0000","last_became_active":"2024-03-19T02:50:52.221748+0000","last_became_peered":"2024-03-19T02:50:52.221748+0000","last_unstale":"2024-03-19T02:50:59.653317+0000","last_undegraded":"2024-03-19T02:50:59.653317+0000","last_fullsized":"2024-03-19T02:50:59.653317+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:33:31.401939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123790+0000","last_change":"2024-03-19T02:50:52.221799+0000","last_active":"2024-03-19T02:51:00.123790+0000","last_peered":"2024-03-19T02:51:00.123790+0000","last_clean":"2024-03-19T02:51:00.123790+0000","last_became_active":"2024-03-19T02:50:52.220196+0000","last_became_peered":"2024-03-19T02:50:52.220196+0000","last_unstale":"2024-03-19T02:51:00.123790+0000","last_undegraded":"2024-03-19T02:51:00.123790+0000","last_fullsized":"2024-03-19T02:51:00.123790+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:24:01.541716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245221+0000","last_change":"2024-03-19T02:50:52.223102+0000","last_active":"2024-03-19T02:50:59.245221+0000","last_peered":"2024-03-19T02:50:59.245221+0000","last_clean":"2024-03-19T02:50:59.245221+0000","last_became_active":"2024-03-19T02:50:52.222801+0000","last_became_peered":"2024-03-19T02:50:52.222801+0000","last_unstale":"2024-03-19T02:50:59.245221+0000","last_undegraded":"2024-03-19T02:50:59.245221+0000","last_fullsized":"2024-03-19T02:50:59.245221+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:48:48.471048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123072+0000","last_change":"2024-03-19T02:50:52.221296+0000","last_active":"2024-03-19T02:51:00.123072+0000","last_peered":"2024-03-19T02:51:00.123072+0000","last_clean":"2024-03-19T02:51:00.123072+0000","last_became_active":"2024-03-19T02:50:52.219141+0000","last_became_peered":"2024-03-19T02:50:52.219141+0000","last_unstale":"2024-03-19T02:51:00.123072+0000","last_undegraded":"2024-03-19T02:51:00.123072+0000","last_fullsized":"2024-03-19T02:51:00.123072+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:19:48.345528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123187+0000","last_change":"2024-03-19T02:50:52.230847+0000","last_active":"2024-03-19T02:51:00.123187+0000","last_peered":"2024-03-19T02:51:00.123187+0000","last_clean":"2024-03-19T02:51:00.123187+0000","last_became_active":"2024-03-19T02:50:52.230478+0000","last_became_peered":"2024-03-19T02:50:52.230478+0000","last_unstale":"2024-03-19T02:51:00.123187+0000","last_undegraded":"2024-03-19T02:51:00.123187+0000","last_fullsized":"2024-03-19T02:51:00.123187+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:04:09.459446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245081+0000","last_change":"2024-03-19T02:50:52.223508+0000","last_active":"2024-03-19T02:50:59.245081+0000","last_peered":"2024-03-19T02:50:59.245081+0000","last_clean":"2024-03-19T02:50:59.245081+0000","last_became_active":"2024-03-19T02:50:52.223173+0000","last_became_peered":"2024-03-19T02:50:52.223173+0000","last_unstale":"2024-03-19T02:50:59.245081+0000","last_undegraded":"2024-03-19T02:50:59.245081+0000","last_fullsized":"2024-03-19T02:50:59.245081+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:53.320173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.124338+0000","last_change":"2024-03-19T02:50:52.229548+0000","last_active":"2024-03-19T02:51:00.124338+0000","last_peered":"2024-03-19T02:51:00.124338+0000","last_clean":"2024-03-19T02:51:00.124338+0000","last_became_active":"2024-03-19T02:50:52.229203+0000","last_became_peered":"2024-03-19T02:50:52.229203+0000","last_unstale":"2024-03-19T02:51:00.124338+0000","last_undegraded":"2024-03-19T02:51:00.124338+0000","last_fullsized":"2024-03-19T02:51:00.124338+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:48:45.884098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":14052,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":26,"num_read_kb":24,"num_write":131,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":47673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318290,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1032192,"data_stored":503210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579927,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28648,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716504,"statfs":{"total":95995035648,"available":95965700096,"internally_reserved":0,"allocated":1671168,"data_stored":1098998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743453,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1052672,"data_stored":504028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005090,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1056768,"data_stored":508044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266728,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32788,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712364,"statfs":{"total":95995035648,"available":95961460736,"internally_reserved":0,"allocated":1650688,"data_stored":1103562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462955,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":503300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1052672,"data_stored":513652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738443,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":504688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":508402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149756,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":503494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888119,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":503854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724594,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28636,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716516,"statfs":{"total":95995035648,"available":95965712384,"internally_reserved":0,"allocated":1658880,"data_stored":1099150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":23141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":12020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T02:51:21.866 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T02:51:22.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.001+0000 7f0de76dc640 1 -- 172.21.15.33:0/448390907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0de0102a30 msgr2=0x7f0de0102eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:22.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.001+0000 7f0de76dc640 1 --2- 172.21.15.33:0/448390907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0de0102a30 0x7f0de0102eb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0dd0009900 tx=0x7f0dd002bc30 comp rx=0 tx=0).stop 2024-03-19T02:51:22.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.001+0000 7f0de76dc640 1 -- 172.21.15.33:0/448390907 shutdown_connections 2024-03-19T02:51:22.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.001+0000 7f0de76dc640 1 --2- 172.21.15.33:0/448390907 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0de01033f0 0x7f0de0109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:22.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.001+0000 7f0de76dc640 1 --2- 172.21.15.33:0/448390907 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0de0102a30 0x7f0de0102eb0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:22.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.001+0000 7f0de76dc640 1 --2- 172.21.15.33:0/448390907 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0de0101840 0x7f0de0101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:22.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.001+0000 7f0de76dc640 1 -- 172.21.15.33:0/448390907 >> 172.21.15.33:0/448390907 conn(0x7f0de00fcdd0 msgr2=0x7f0de00ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:22.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.001+0000 7f0de76dc640 1 -- 172.21.15.33:0/448390907 shutdown_connections 2024-03-19T02:51:22.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.001+0000 7f0de76dc640 1 -- 172.21.15.33:0/448390907 wait complete. 2024-03-19T02:51:22.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.002+0000 7f0de76dc640 1 Processor -- start 2024-03-19T02:51:22.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.002+0000 7f0de76dc640 1 -- start start 2024-03-19T02:51:22.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.002+0000 7f0de76dc640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0de0101840 0x7f0de0197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:22.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.002+0000 7f0de76dc640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0de0102a30 0x7f0de0197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:22.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.002+0000 7f0de76dc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0de01033f0 0x7f0de019f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:22.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.002+0000 7f0de76dc640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0de010d370 con 0x7f0de01033f0 2024-03-19T02:51:22.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.002+0000 7f0de76dc640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0de010d1f0 con 0x7f0de0102a30 2024-03-19T02:51:22.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.002+0000 7f0de76dc640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f0de010d4f0 con 0x7f0de0101840 2024-03-19T02:51:22.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.002+0000 7f0de5c79640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0de01033f0 0x7f0de019f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:22.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.002+0000 7f0de5c79640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0de01033f0 0x7f0de019f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50164/0 (socket says 172.21.15.33:50164) 2024-03-19T02:51:22.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.002+0000 7f0de5c79640 1 -- 172.21.15.33:0/471214278 learned_addr learned my addr 172.21.15.33:0/471214278 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:22.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.002+0000 7f0de4c77640 1 --2- 172.21.15.33:0/471214278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0de0102a30 0x7f0de0197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:22.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.002+0000 7f0de5478640 1 --2- 172.21.15.33:0/471214278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0de0101840 0x7f0de0197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:22.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.002+0000 7f0de5c79640 1 -- 172.21.15.33:0/471214278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0de0101840 msgr2=0x7f0de0197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:22.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.002+0000 7f0de5c79640 1 --2- 172.21.15.33:0/471214278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0de0101840 0x7f0de0197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:22.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.002+0000 7f0de5c79640 1 -- 172.21.15.33:0/471214278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0de0102a30 msgr2=0x7f0de0197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:22.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.002+0000 7f0de5c79640 1 --2- 172.21.15.33:0/471214278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0de0102a30 0x7f0de0197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:22.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.002+0000 7f0de5c79640 1 -- 172.21.15.33:0/471214278 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0de019f6b0 con 0x7f0de01033f0 2024-03-19T02:51:22.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.002+0000 7f0de5c79640 1 --2- 172.21.15.33:0/471214278 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0de01033f0 0x7f0de019f040 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0dd800eb10 tx=0x7f0dd800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:22.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.003+0000 7f0dd67fc640 1 -- 172.21.15.33:0/471214278 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd8004b20 con 0x7f0de01033f0 2024-03-19T02:51:22.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.003+0000 7f0dd67fc640 1 -- 172.21.15.33:0/471214278 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0dd8004cc0 con 0x7f0de01033f0 2024-03-19T02:51:22.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.003+0000 7f0de76dc640 1 -- 172.21.15.33:0/471214278 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0de019f9a0 con 0x7f0de01033f0 2024-03-19T02:51:22.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.003+0000 7f0dd67fc640 1 -- 172.21.15.33:0/471214278 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd8010690 con 0x7f0de01033f0 2024-03-19T02:51:22.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.003+0000 7f0de76dc640 1 -- 172.21.15.33:0/471214278 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0de019fe70 con 0x7f0de01033f0 2024-03-19T02:51:22.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.004+0000 7f0de76dc640 1 -- 172.21.15.33:0/471214278 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0de0108c30 con 0x7f0de01033f0 2024-03-19T02:51:22.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.005+0000 7f0dd67fc640 1 -- 172.21.15.33:0/471214278 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f0dd8005020 con 0x7f0de01033f0 2024-03-19T02:51:22.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.005+0000 7f0dd67fc640 1 --2- 172.21.15.33:0/471214278 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f0dcc0a2890 0x7f0dcc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:22.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.005+0000 7f0de5478640 1 --2- 172.21.15.33:0/471214278 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f0dcc0a2890 0x7f0dcc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:22.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.006+0000 7f0dd67fc640 1 -- 172.21.15.33:0/471214278 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7f0dd80cf3a0 con 0x7f0de01033f0 2024-03-19T02:51:22.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.006+0000 7f0de5478640 1 --2- 172.21.15.33:0/471214278 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f0dcc0a2890 0x7f0dcc0a4d50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0de0102890 tx=0x7f0ddc005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:22.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.010+0000 7f0dd67fc640 1 -- 172.21.15.33:0/471214278 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0dd809c0a0 con 0x7f0de01033f0 2024-03-19T02:51:22.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.192+0000 7f0de76dc640 1 -- 172.21.15.33:0/471214278 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0de0061c60 con 0x7f0dcc0a2890 2024-03-19T02:51:22.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.201+0000 7f0dd67fc640 1 -- 172.21.15.33:0/471214278 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340348 (secure 0 0 0) 0x7f0de0061c60 con 0x7f0dcc0a2890 2024-03-19T02:51:22.206 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:51:22.207 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-03-19T02:51:22.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.205+0000 7f0de76dc640 1 -- 172.21.15.33:0/471214278 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f0dcc0a2890 msgr2=0x7f0dcc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:22.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.205+0000 7f0de76dc640 1 --2- 172.21.15.33:0/471214278 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f0dcc0a2890 0x7f0dcc0a4d50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0de0102890 tx=0x7f0ddc005f90 comp rx=0 tx=0).stop 2024-03-19T02:51:22.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.205+0000 7f0de76dc640 1 -- 172.21.15.33:0/471214278 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0de01033f0 msgr2=0x7f0de019f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:22.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.205+0000 7f0de76dc640 1 --2- 172.21.15.33:0/471214278 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0de01033f0 0x7f0de019f040 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0dd800eb10 tx=0x7f0dd800c2d0 comp rx=0 tx=0).stop 2024-03-19T02:51:22.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.205+0000 7f0de76dc640 1 -- 172.21.15.33:0/471214278 shutdown_connections 2024-03-19T02:51:22.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.205+0000 7f0de76dc640 1 --2- 172.21.15.33:0/471214278 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f0dcc0a2890 0x7f0dcc0a4d50 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:22.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.205+0000 7f0de76dc640 1 --2- 172.21.15.33:0/471214278 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0de01033f0 0x7f0de019f040 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:22.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.205+0000 7f0de76dc640 1 --2- 172.21.15.33:0/471214278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0de0102a30 0x7f0de0197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:22.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.205+0000 7f0de76dc640 1 --2- 172.21.15.33:0/471214278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0de0101840 0x7f0de0197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:22.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.205+0000 7f0de76dc640 1 -- 172.21.15.33:0/471214278 >> 172.21.15.33:0/471214278 conn(0x7f0de00fcdd0 msgr2=0x7f0de0105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:22.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.205+0000 7f0de76dc640 1 -- 172.21.15.33:0/471214278 shutdown_connections 2024-03-19T02:51:22.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:22.205+0000 7f0de76dc640 1 -- 172.21.15.33:0/471214278 wait complete. 2024-03-19T02:51:22.216 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:22 smithi033 ceph-mon[31973]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:51:22.216 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:22 smithi033 ceph-mon[31973]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-19T02:51:22.216 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:22 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/232614714' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:51:22.216 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:22 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1633399558' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:51:22.216 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:22 smithi033 ceph-mon[31973]: pgmap v314: 129 pgs: 129 active+clean; 590 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-03-19T02:51:22.217 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":314,"stamp":"2024-03-19T02:51:21.710892+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":604537,"num_objects":62,"num_object_clones":0,"num_object_copies":187,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":162,"num_read_kb":143,"num_write":313,"num_write_kb":2990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":253,"ondisk_log_size":253,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":363252,"kb_used_data":14096,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124578572,"statfs":{"total":1151940427776,"available":1151568457728,"internally_reserved":0,"allocated":14434304,"data_stored":7854382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9529,"num_objects":38,"num_object_clones":0,"num_object_copies":115,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":26,"num_read_kb":24,"num_write":83,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003535"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.124670+0000","last_change":"2024-03-19T02:50:52.228368+0000","last_active":"2024-03-19T02:51:00.124670+0000","last_peered":"2024-03-19T02:51:00.124670+0000","last_clean":"2024-03-19T02:51:00.124670+0000","last_became_active":"2024-03-19T02:50:52.228069+0000","last_became_peered":"2024-03-19T02:50:52.228069+0000","last_unstale":"2024-03-19T02:51:00.124670+0000","last_undegraded":"2024-03-19T02:51:00.124670+0000","last_fullsized":"2024-03-19T02:51:00.124670+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:59:34.209095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.832634+0000","last_change":"2024-03-19T02:50:52.231358+0000","last_active":"2024-03-19T02:50:59.832634+0000","last_peered":"2024-03-19T02:50:59.832634+0000","last_clean":"2024-03-19T02:50:59.832634+0000","last_became_active":"2024-03-19T02:50:52.231029+0000","last_became_peered":"2024-03-19T02:50:52.231029+0000","last_unstale":"2024-03-19T02:50:59.832634+0000","last_undegraded":"2024-03-19T02:50:59.832634+0000","last_fullsized":"2024-03-19T02:50:59.832634+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:42:26.031392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.832451+0000","last_change":"2024-03-19T02:50:52.232685+0000","last_active":"2024-03-19T02:50:59.832451+0000","last_peered":"2024-03-19T02:50:59.832451+0000","last_clean":"2024-03-19T02:50:59.832451+0000","last_became_active":"2024-03-19T02:50:52.232361+0000","last_became_peered":"2024-03-19T02:50:52.232361+0000","last_unstale":"2024-03-19T02:50:59.832451+0000","last_undegraded":"2024-03-19T02:50:59.832451+0000","last_fullsized":"2024-03-19T02:50:59.832451+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:53.136200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245640+0000","last_change":"2024-03-19T02:50:52.227648+0000","last_active":"2024-03-19T02:50:59.245640+0000","last_peered":"2024-03-19T02:50:59.245640+0000","last_clean":"2024-03-19T02:50:59.245640+0000","last_became_active":"2024-03-19T02:50:52.226943+0000","last_became_peered":"2024-03-19T02:50:52.226943+0000","last_unstale":"2024-03-19T02:50:59.245640+0000","last_undegraded":"2024-03-19T02:50:59.245640+0000","last_fullsized":"2024-03-19T02:50:59.245640+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:03.293794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122803+0000","last_change":"2024-03-19T02:50:52.225329+0000","last_active":"2024-03-19T02:51:00.122803+0000","last_peered":"2024-03-19T02:51:00.122803+0000","last_clean":"2024-03-19T02:51:00.122803+0000","last_became_active":"2024-03-19T02:50:52.225012+0000","last_became_peered":"2024-03-19T02:50:52.225012+0000","last_unstale":"2024-03-19T02:51:00.122803+0000","last_undegraded":"2024-03-19T02:51:00.122803+0000","last_fullsized":"2024-03-19T02:51:00.122803+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:32:52.877704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.036857+0000","last_change":"2024-03-19T02:50:52.217399+0000","last_active":"2024-03-19T02:51:00.036857+0000","last_peered":"2024-03-19T02:51:00.036857+0000","last_clean":"2024-03-19T02:51:00.036857+0000","last_became_active":"2024-03-19T02:50:52.216987+0000","last_became_peered":"2024-03-19T02:50:52.216987+0000","last_unstale":"2024-03-19T02:51:00.036857+0000","last_undegraded":"2024-03-19T02:51:00.036857+0000","last_fullsized":"2024-03-19T02:51:00.036857+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:57:15.772546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122135+0000","last_change":"2024-03-19T02:50:52.226755+0000","last_active":"2024-03-19T02:51:00.122135+0000","last_peered":"2024-03-19T02:51:00.122135+0000","last_clean":"2024-03-19T02:51:00.122135+0000","last_became_active":"2024-03-19T02:50:52.226434+0000","last_became_peered":"2024-03-19T02:50:52.226434+0000","last_unstale":"2024-03-19T02:51:00.122135+0000","last_undegraded":"2024-03-19T02:51:00.122135+0000","last_fullsized":"2024-03-19T02:51:00.122135+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:49:42.134262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122957+0000","last_change":"2024-03-19T02:50:52.215848+0000","last_active":"2024-03-19T02:51:00.122957+0000","last_peered":"2024-03-19T02:51:00.122957+0000","last_clean":"2024-03-19T02:51:00.122957+0000","last_became_active":"2024-03-19T02:50:52.215494+0000","last_became_peered":"2024-03-19T02:50:52.215494+0000","last_unstale":"2024-03-19T02:51:00.122957+0000","last_undegraded":"2024-03-19T02:51:00.122957+0000","last_fullsized":"2024-03-19T02:51:00.122957+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:55:10.182359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122921+0000","last_change":"2024-03-19T02:50:52.233237+0000","last_active":"2024-03-19T02:51:00.122921+0000","last_peered":"2024-03-19T02:51:00.122921+0000","last_clean":"2024-03-19T02:51:00.122921+0000","last_became_active":"2024-03-19T02:50:52.232091+0000","last_became_peered":"2024-03-19T02:50:52.232091+0000","last_unstale":"2024-03-19T02:51:00.122921+0000","last_undegraded":"2024-03-19T02:51:00.122921+0000","last_fullsized":"2024-03-19T02:51:00.122921+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:08:20.361576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.037210+0000","last_change":"2024-03-19T02:50:52.219880+0000","last_active":"2024-03-19T02:51:00.037210+0000","last_peered":"2024-03-19T02:51:00.037210+0000","last_clean":"2024-03-19T02:51:00.037210+0000","last_became_active":"2024-03-19T02:50:52.219180+0000","last_became_peered":"2024-03-19T02:50:52.219180+0000","last_unstale":"2024-03-19T02:51:00.037210+0000","last_undegraded":"2024-03-19T02:51:00.037210+0000","last_fullsized":"2024-03-19T02:51:00.037210+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:47:29.861906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122755+0000","last_change":"2024-03-19T02:50:52.226314+0000","last_active":"2024-03-19T02:51:00.122755+0000","last_peered":"2024-03-19T02:51:00.122755+0000","last_clean":"2024-03-19T02:51:00.122755+0000","last_became_active":"2024-03-19T02:50:52.225919+0000","last_became_peered":"2024-03-19T02:50:52.225919+0000","last_unstale":"2024-03-19T02:51:00.122755+0000","last_undegraded":"2024-03-19T02:51:00.122755+0000","last_fullsized":"2024-03-19T02:51:00.122755+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:42:51.782592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245637+0000","last_change":"2024-03-19T02:50:52.227929+0000","last_active":"2024-03-19T02:50:59.245637+0000","last_peered":"2024-03-19T02:50:59.245637+0000","last_clean":"2024-03-19T02:50:59.245637+0000","last_became_active":"2024-03-19T02:50:52.227360+0000","last_became_peered":"2024-03-19T02:50:52.227360+0000","last_unstale":"2024-03-19T02:50:59.245637+0000","last_undegraded":"2024-03-19T02:50:59.245637+0000","last_fullsized":"2024-03-19T02:50:59.245637+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:24:44.386101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245208+0000","last_change":"2024-03-19T02:50:52.228088+0000","last_active":"2024-03-19T02:50:59.245208+0000","last_peered":"2024-03-19T02:50:59.245208+0000","last_clean":"2024-03-19T02:50:59.245208+0000","last_became_active":"2024-03-19T02:50:52.227754+0000","last_became_peered":"2024-03-19T02:50:52.227754+0000","last_unstale":"2024-03-19T02:50:59.245208+0000","last_undegraded":"2024-03-19T02:50:59.245208+0000","last_fullsized":"2024-03-19T02:50:59.245208+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:58:57.404241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.832111+0000","last_change":"2024-03-19T02:50:52.226768+0000","last_active":"2024-03-19T02:50:59.832111+0000","last_peered":"2024-03-19T02:50:59.832111+0000","last_clean":"2024-03-19T02:50:59.832111+0000","last_became_active":"2024-03-19T02:50:52.226452+0000","last_became_peered":"2024-03-19T02:50:52.226452+0000","last_unstale":"2024-03-19T02:50:59.832111+0000","last_undegraded":"2024-03-19T02:50:59.832111+0000","last_fullsized":"2024-03-19T02:50:59.832111+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:23:33.250773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.832082+0000","last_change":"2024-03-19T02:50:52.231132+0000","last_active":"2024-03-19T02:50:59.832082+0000","last_peered":"2024-03-19T02:50:59.832082+0000","last_clean":"2024-03-19T02:50:59.832082+0000","last_became_active":"2024-03-19T02:50:52.230778+0000","last_became_peered":"2024-03-19T02:50:52.230778+0000","last_unstale":"2024-03-19T02:50:59.832082+0000","last_undegraded":"2024-03-19T02:50:59.832082+0000","last_fullsized":"2024-03-19T02:50:59.832082+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:33:03.675210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122740+0000","last_change":"2024-03-19T02:50:52.228302+0000","last_active":"2024-03-19T02:51:00.122740+0000","last_peered":"2024-03-19T02:51:00.122740+0000","last_clean":"2024-03-19T02:51:00.122740+0000","last_became_active":"2024-03-19T02:50:52.228007+0000","last_became_peered":"2024-03-19T02:50:52.228007+0000","last_unstale":"2024-03-19T02:51:00.122740+0000","last_undegraded":"2024-03-19T02:51:00.122740+0000","last_fullsized":"2024-03-19T02:51:00.122740+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:59:28.988504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.124445+0000","last_change":"2024-03-19T02:50:52.228640+0000","last_active":"2024-03-19T02:51:00.124445+0000","last_peered":"2024-03-19T02:51:00.124445+0000","last_clean":"2024-03-19T02:51:00.124445+0000","last_became_active":"2024-03-19T02:50:52.228319+0000","last_became_peered":"2024-03-19T02:50:52.228319+0000","last_unstale":"2024-03-19T02:51:00.124445+0000","last_undegraded":"2024-03-19T02:51:00.124445+0000","last_fullsized":"2024-03-19T02:51:00.124445+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:30.356453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123774+0000","last_change":"2024-03-19T02:50:52.222300+0000","last_active":"2024-03-19T02:51:00.123774+0000","last_peered":"2024-03-19T02:51:00.123774+0000","last_clean":"2024-03-19T02:51:00.123774+0000","last_became_active":"2024-03-19T02:50:52.221978+0000","last_became_peered":"2024-03-19T02:50:52.221978+0000","last_unstale":"2024-03-19T02:51:00.123774+0000","last_undegraded":"2024-03-19T02:51:00.123774+0000","last_fullsized":"2024-03-19T02:51:00.123774+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:38:36.576987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.653257+0000","last_change":"2024-03-19T02:50:52.217640+0000","last_active":"2024-03-19T02:50:59.653257+0000","last_peered":"2024-03-19T02:50:59.653257+0000","last_clean":"2024-03-19T02:50:59.653257+0000","last_became_active":"2024-03-19T02:50:52.217325+0000","last_became_peered":"2024-03-19T02:50:52.217325+0000","last_unstale":"2024-03-19T02:50:59.653257+0000","last_undegraded":"2024-03-19T02:50:59.653257+0000","last_fullsized":"2024-03-19T02:50:59.653257+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:40:25.894843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.652853+0000","last_change":"2024-03-19T02:50:57.245323+0000","last_active":"2024-03-19T02:50:59.652853+0000","last_peered":"2024-03-19T02:50:59.652853+0000","last_clean":"2024-03-19T02:50:59.652853+0000","last_became_active":"2024-03-19T02:50:57.244886+0000","last_became_peered":"2024-03-19T02:50:57.244886+0000","last_unstale":"2024-03-19T02:50:59.652853+0000","last_undegraded":"2024-03-19T02:50:59.652853+0000","last_fullsized":"2024-03-19T02:50:59.652853+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:42:09.465738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"93'9","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:18.083329+0000","last_change":"2024-03-19T02:50:51.797160+0000","last_active":"2024-03-19T02:51:18.083329+0000","last_peered":"2024-03-19T02:51:18.083329+0000","last_clean":"2024-03-19T02:51:18.083329+0000","last_became_active":"2024-03-19T02:50:51.796783+0000","last_became_peered":"2024-03-19T02:50:51.796783+0000","last_unstale":"2024-03-19T02:51:18.083329+0000","last_undegraded":"2024-03-19T02:51:18.083329+0000","last_fullsized":"2024-03-19T02:51:18.083329+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:57:08.185379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123291+0000","last_change":"2024-03-19T02:50:52.225114+0000","last_active":"2024-03-19T02:51:00.123291+0000","last_peered":"2024-03-19T02:51:00.123291+0000","last_clean":"2024-03-19T02:51:00.123291+0000","last_became_active":"2024-03-19T02:50:52.220529+0000","last_became_peered":"2024-03-19T02:50:52.220529+0000","last_unstale":"2024-03-19T02:51:00.123291+0000","last_undegraded":"2024-03-19T02:51:00.123291+0000","last_fullsized":"2024-03-19T02:51:00.123291+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:12:51.408689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245260+0000","last_change":"2024-03-19T02:50:57.252827+0000","last_active":"2024-03-19T02:50:59.245260+0000","last_peered":"2024-03-19T02:50:59.245260+0000","last_clean":"2024-03-19T02:50:59.245260+0000","last_became_active":"2024-03-19T02:50:57.252502+0000","last_became_peered":"2024-03-19T02:50:57.252502+0000","last_unstale":"2024-03-19T02:50:59.245260+0000","last_undegraded":"2024-03-19T02:50:59.245260+0000","last_fullsized":"2024-03-19T02:50:59.245260+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:29:36.563533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"93'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122744+0000","last_change":"2024-03-19T02:50:51.797380+0000","last_active":"2024-03-19T02:51:00.122744+0000","last_peered":"2024-03-19T02:51:00.122744+0000","last_clean":"2024-03-19T02:51:00.122744+0000","last_became_active":"2024-03-19T02:50:51.797021+0000","last_became_peered":"2024-03-19T02:50:51.797021+0000","last_unstale":"2024-03-19T02:51:00.122744+0000","last_undegraded":"2024-03-19T02:51:00.122744+0000","last_fullsized":"2024-03-19T02:51:00.122744+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:25.400360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.832593+0000","last_change":"2024-03-19T02:50:52.232717+0000","last_active":"2024-03-19T02:50:59.832593+0000","last_peered":"2024-03-19T02:50:59.832593+0000","last_clean":"2024-03-19T02:50:59.832593+0000","last_became_active":"2024-03-19T02:50:52.232368+0000","last_became_peered":"2024-03-19T02:50:52.232368+0000","last_unstale":"2024-03-19T02:50:59.832593+0000","last_undegraded":"2024-03-19T02:50:59.832593+0000","last_fullsized":"2024-03-19T02:50:59.832593+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:55:38.027120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245388+0000","last_change":"2024-03-19T02:50:57.250840+0000","last_active":"2024-03-19T02:50:59.245388+0000","last_peered":"2024-03-19T02:50:59.245388+0000","last_clean":"2024-03-19T02:50:59.245388+0000","last_became_active":"2024-03-19T02:50:57.250333+0000","last_became_peered":"2024-03-19T02:50:57.250333+0000","last_unstale":"2024-03-19T02:50:59.245388+0000","last_undegraded":"2024-03-19T02:50:59.245388+0000","last_fullsized":"2024-03-19T02:50:59.245388+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:57:09.089939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"93'4","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:18.079717+0000","last_change":"2024-03-19T02:50:51.796889+0000","last_active":"2024-03-19T02:51:18.079717+0000","last_peered":"2024-03-19T02:51:18.079717+0000","last_clean":"2024-03-19T02:51:18.079717+0000","last_became_active":"2024-03-19T02:50:51.796570+0000","last_became_peered":"2024-03-19T02:50:51.796570+0000","last_unstale":"2024-03-19T02:51:18.079717+0000","last_undegraded":"2024-03-19T02:51:18.079717+0000","last_fullsized":"2024-03-19T02:51:18.079717+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:18:38.301905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123117+0000","last_change":"2024-03-19T02:50:52.224533+0000","last_active":"2024-03-19T02:51:00.123117+0000","last_peered":"2024-03-19T02:51:00.123117+0000","last_clean":"2024-03-19T02:51:00.123117+0000","last_became_active":"2024-03-19T02:50:52.224221+0000","last_became_peered":"2024-03-19T02:50:52.224221+0000","last_unstale":"2024-03-19T02:51:00.123117+0000","last_undegraded":"2024-03-19T02:51:00.123117+0000","last_fullsized":"2024-03-19T02:51:00.123117+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:36:43.335318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245257+0000","last_change":"2024-03-19T02:50:57.252949+0000","last_active":"2024-03-19T02:50:59.245257+0000","last_peered":"2024-03-19T02:50:59.245257+0000","last_clean":"2024-03-19T02:50:59.245257+0000","last_became_active":"2024-03-19T02:50:57.252563+0000","last_became_peered":"2024-03-19T02:50:57.252563+0000","last_unstale":"2024-03-19T02:50:59.245257+0000","last_undegraded":"2024-03-19T02:50:59.245257+0000","last_fullsized":"2024-03-19T02:50:59.245257+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:30:38.244158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122643+0000","last_change":"2024-03-19T02:50:51.794594+0000","last_active":"2024-03-19T02:51:00.122643+0000","last_peered":"2024-03-19T02:51:00.122643+0000","last_clean":"2024-03-19T02:51:00.122643+0000","last_became_active":"2024-03-19T02:50:51.794197+0000","last_became_peered":"2024-03-19T02:50:51.794197+0000","last_unstale":"2024-03-19T02:51:00.122643+0000","last_undegraded":"2024-03-19T02:51:00.122643+0000","last_fullsized":"2024-03-19T02:51:00.122643+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:58:44.534106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245719+0000","last_change":"2024-03-19T02:50:52.214167+0000","last_active":"2024-03-19T02:50:59.245719+0000","last_peered":"2024-03-19T02:50:59.245719+0000","last_clean":"2024-03-19T02:50:59.245719+0000","last_became_active":"2024-03-19T02:50:52.213873+0000","last_became_peered":"2024-03-19T02:50:52.213873+0000","last_unstale":"2024-03-19T02:50:59.245719+0000","last_undegraded":"2024-03-19T02:50:59.245719+0000","last_fullsized":"2024-03-19T02:50:59.245719+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:28:51.035477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.653068+0000","last_change":"2024-03-19T02:50:57.245139+0000","last_active":"2024-03-19T02:50:59.653068+0000","last_peered":"2024-03-19T02:50:59.653068+0000","last_clean":"2024-03-19T02:50:59.653068+0000","last_became_active":"2024-03-19T02:50:57.244636+0000","last_became_peered":"2024-03-19T02:50:57.244636+0000","last_unstale":"2024-03-19T02:50:59.653068+0000","last_undegraded":"2024-03-19T02:50:59.653068+0000","last_fullsized":"2024-03-19T02:50:59.653068+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:44:29.660221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'5","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:19.087270+0000","last_change":"2024-03-19T02:50:51.795969+0000","last_active":"2024-03-19T02:51:19.087270+0000","last_peered":"2024-03-19T02:51:19.087270+0000","last_clean":"2024-03-19T02:51:19.087270+0000","last_became_active":"2024-03-19T02:50:51.795628+0000","last_became_peered":"2024-03-19T02:50:51.795628+0000","last_unstale":"2024-03-19T02:51:19.087270+0000","last_undegraded":"2024-03-19T02:51:19.087270+0000","last_fullsized":"2024-03-19T02:51:19.087270+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:14:48.463282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123036+0000","last_change":"2024-03-19T02:50:52.223278+0000","last_active":"2024-03-19T02:51:00.123036+0000","last_peered":"2024-03-19T02:51:00.123036+0000","last_clean":"2024-03-19T02:51:00.123036+0000","last_became_active":"2024-03-19T02:50:52.222741+0000","last_became_peered":"2024-03-19T02:50:52.222741+0000","last_unstale":"2024-03-19T02:51:00.123036+0000","last_undegraded":"2024-03-19T02:51:00.123036+0000","last_fullsized":"2024-03-19T02:51:00.123036+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:32:09.841825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.653019+0000","last_change":"2024-03-19T02:50:57.253130+0000","last_active":"2024-03-19T02:50:59.653019+0000","last_peered":"2024-03-19T02:50:59.653019+0000","last_clean":"2024-03-19T02:50:59.653019+0000","last_became_active":"2024-03-19T02:50:57.252598+0000","last_became_peered":"2024-03-19T02:50:57.252598+0000","last_unstale":"2024-03-19T02:50:59.653019+0000","last_undegraded":"2024-03-19T02:50:59.653019+0000","last_fullsized":"2024-03-19T02:50:59.653019+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:09:38.183430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"93'3","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:18.079600+0000","last_change":"2024-03-19T02:50:51.794075+0000","last_active":"2024-03-19T02:51:18.079600+0000","last_peered":"2024-03-19T02:51:18.079600+0000","last_clean":"2024-03-19T02:51:18.079600+0000","last_became_active":"2024-03-19T02:50:51.793707+0000","last_became_peered":"2024-03-19T02:50:51.793707+0000","last_unstale":"2024-03-19T02:51:18.079600+0000","last_undegraded":"2024-03-19T02:51:18.079600+0000","last_fullsized":"2024-03-19T02:51:18.079600+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:52:07.180162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245655+0000","last_change":"2024-03-19T02:50:52.211112+0000","last_active":"2024-03-19T02:50:59.245655+0000","last_peered":"2024-03-19T02:50:59.245655+0000","last_clean":"2024-03-19T02:50:59.245655+0000","last_became_active":"2024-03-19T02:50:52.210790+0000","last_became_peered":"2024-03-19T02:50:52.210790+0000","last_unstale":"2024-03-19T02:50:59.245655+0000","last_undegraded":"2024-03-19T02:50:59.245655+0000","last_fullsized":"2024-03-19T02:50:59.245655+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:39:17.651585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.124501+0000","last_change":"2024-03-19T02:50:57.250397+0000","last_active":"2024-03-19T02:51:00.124501+0000","last_peered":"2024-03-19T02:51:00.124501+0000","last_clean":"2024-03-19T02:51:00.124501+0000","last_became_active":"2024-03-19T02:50:57.249968+0000","last_became_peered":"2024-03-19T02:50:57.249968+0000","last_unstale":"2024-03-19T02:51:00.124501+0000","last_undegraded":"2024-03-19T02:51:00.124501+0000","last_fullsized":"2024-03-19T02:51:00.124501+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:39.796799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"93'2","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:18.082766+0000","last_change":"2024-03-19T02:50:51.785347+0000","last_active":"2024-03-19T02:51:18.082766+0000","last_peered":"2024-03-19T02:51:18.082766+0000","last_clean":"2024-03-19T02:51:18.082766+0000","last_became_active":"2024-03-19T02:50:51.785004+0000","last_became_peered":"2024-03-19T02:50:51.785004+0000","last_unstale":"2024-03-19T02:51:18.082766+0000","last_undegraded":"2024-03-19T02:51:18.082766+0000","last_fullsized":"2024-03-19T02:51:18.082766+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:47.269503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123279+0000","last_change":"2024-03-19T02:50:52.221330+0000","last_active":"2024-03-19T02:51:00.123279+0000","last_peered":"2024-03-19T02:51:00.123279+0000","last_clean":"2024-03-19T02:51:00.123279+0000","last_became_active":"2024-03-19T02:50:52.219417+0000","last_became_peered":"2024-03-19T02:50:52.219417+0000","last_unstale":"2024-03-19T02:51:00.123279+0000","last_undegraded":"2024-03-19T02:51:00.123279+0000","last_fullsized":"2024-03-19T02:51:00.123279+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:28:37.856260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.832612+0000","last_change":"2024-03-19T02:50:57.258357+0000","last_active":"2024-03-19T02:50:59.832612+0000","last_peered":"2024-03-19T02:50:59.832612+0000","last_clean":"2024-03-19T02:50:59.832612+0000","last_became_active":"2024-03-19T02:50:57.257924+0000","last_became_peered":"2024-03-19T02:50:57.257924+0000","last_unstale":"2024-03-19T02:50:59.832612+0000","last_undegraded":"2024-03-19T02:50:59.832612+0000","last_fullsized":"2024-03-19T02:50:59.832612+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:46:05.340731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"93'1","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245236+0000","last_change":"2024-03-19T02:50:51.796425+0000","last_active":"2024-03-19T02:50:59.245236+0000","last_peered":"2024-03-19T02:50:59.245236+0000","last_clean":"2024-03-19T02:50:59.245236+0000","last_became_active":"2024-03-19T02:50:51.796139+0000","last_became_peered":"2024-03-19T02:50:51.796139+0000","last_unstale":"2024-03-19T02:50:59.245236+0000","last_undegraded":"2024-03-19T02:50:59.245236+0000","last_fullsized":"2024-03-19T02:50:59.245236+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:12:17.123556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122903+0000","last_change":"2024-03-19T02:50:52.219375+0000","last_active":"2024-03-19T02:51:00.122903+0000","last_peered":"2024-03-19T02:51:00.122903+0000","last_clean":"2024-03-19T02:51:00.122903+0000","last_became_active":"2024-03-19T02:50:52.219070+0000","last_became_peered":"2024-03-19T02:50:52.219070+0000","last_unstale":"2024-03-19T02:51:00.122903+0000","last_undegraded":"2024-03-19T02:51:00.122903+0000","last_fullsized":"2024-03-19T02:51:00.122903+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:31:58.847700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122836+0000","last_change":"2024-03-19T02:50:57.246502+0000","last_active":"2024-03-19T02:51:00.122836+0000","last_peered":"2024-03-19T02:51:00.122836+0000","last_clean":"2024-03-19T02:51:00.122836+0000","last_became_active":"2024-03-19T02:50:57.246089+0000","last_became_peered":"2024-03-19T02:50:57.246089+0000","last_unstale":"2024-03-19T02:51:00.122836+0000","last_undegraded":"2024-03-19T02:51:00.122836+0000","last_fullsized":"2024-03-19T02:51:00.122836+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:00.162308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'3","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:17.066781+0000","last_change":"2024-03-19T02:50:51.796448+0000","last_active":"2024-03-19T02:51:17.066781+0000","last_peered":"2024-03-19T02:51:17.066781+0000","last_clean":"2024-03-19T02:51:17.066781+0000","last_became_active":"2024-03-19T02:50:51.796132+0000","last_became_peered":"2024-03-19T02:50:51.796132+0000","last_unstale":"2024-03-19T02:51:17.066781+0000","last_undegraded":"2024-03-19T02:51:17.066781+0000","last_fullsized":"2024-03-19T02:51:17.066781+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:37:39.442772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123246+0000","last_change":"2024-03-19T02:50:52.223022+0000","last_active":"2024-03-19T02:51:00.123246+0000","last_peered":"2024-03-19T02:51:00.123246+0000","last_clean":"2024-03-19T02:51:00.123246+0000","last_became_active":"2024-03-19T02:50:52.221655+0000","last_became_peered":"2024-03-19T02:50:52.221655+0000","last_unstale":"2024-03-19T02:51:00.123246+0000","last_undegraded":"2024-03-19T02:51:00.123246+0000","last_fullsized":"2024-03-19T02:51:00.123246+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:34:36.054088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122905+0000","last_change":"2024-03-19T02:50:57.244003+0000","last_active":"2024-03-19T02:51:00.122905+0000","last_peered":"2024-03-19T02:51:00.122905+0000","last_clean":"2024-03-19T02:51:00.122905+0000","last_became_active":"2024-03-19T02:50:57.243567+0000","last_became_peered":"2024-03-19T02:50:57.243567+0000","last_unstale":"2024-03-19T02:51:00.122905+0000","last_undegraded":"2024-03-19T02:51:00.122905+0000","last_fullsized":"2024-03-19T02:51:00.122905+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:58:52.400237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"93'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:17.066294+0000","last_change":"2024-03-19T02:50:51.801259+0000","last_active":"2024-03-19T02:51:17.066294+0000","last_peered":"2024-03-19T02:51:17.066294+0000","last_clean":"2024-03-19T02:51:17.066294+0000","last_became_active":"2024-03-19T02:50:51.800868+0000","last_became_peered":"2024-03-19T02:50:51.800868+0000","last_unstale":"2024-03-19T02:51:17.066294+0000","last_undegraded":"2024-03-19T02:51:17.066294+0000","last_fullsized":"2024-03-19T02:51:17.066294+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:44:06.486687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.832608+0000","last_change":"2024-03-19T02:50:52.224973+0000","last_active":"2024-03-19T02:50:59.832608+0000","last_peered":"2024-03-19T02:50:59.832608+0000","last_clean":"2024-03-19T02:50:59.832608+0000","last_became_active":"2024-03-19T02:50:52.224636+0000","last_became_peered":"2024-03-19T02:50:52.224636+0000","last_unstale":"2024-03-19T02:50:59.832608+0000","last_undegraded":"2024-03-19T02:50:59.832608+0000","last_fullsized":"2024-03-19T02:50:59.832608+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:09:58.136715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245524+0000","last_change":"2024-03-19T02:50:57.238764+0000","last_active":"2024-03-19T02:50:59.245524+0000","last_peered":"2024-03-19T02:50:59.245524+0000","last_clean":"2024-03-19T02:50:59.245524+0000","last_became_active":"2024-03-19T02:50:57.238401+0000","last_became_peered":"2024-03-19T02:50:57.238401+0000","last_unstale":"2024-03-19T02:50:59.245524+0000","last_undegraded":"2024-03-19T02:50:59.245524+0000","last_fullsized":"2024-03-19T02:50:59.245524+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:20:46.131776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"93'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:17.066786+0000","last_change":"2024-03-19T02:50:51.797633+0000","last_active":"2024-03-19T02:51:17.066786+0000","last_peered":"2024-03-19T02:51:17.066786+0000","last_clean":"2024-03-19T02:51:17.066786+0000","last_became_active":"2024-03-19T02:50:51.797313+0000","last_became_peered":"2024-03-19T02:50:51.797313+0000","last_unstale":"2024-03-19T02:51:17.066786+0000","last_undegraded":"2024-03-19T02:51:17.066786+0000","last_fullsized":"2024-03-19T02:51:17.066786+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:43:04.659301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123225+0000","last_change":"2024-03-19T02:50:52.231486+0000","last_active":"2024-03-19T02:51:00.123225+0000","last_peered":"2024-03-19T02:51:00.123225+0000","last_clean":"2024-03-19T02:51:00.123225+0000","last_became_active":"2024-03-19T02:50:52.231172+0000","last_became_peered":"2024-03-19T02:50:52.231172+0000","last_unstale":"2024-03-19T02:51:00.123225+0000","last_undegraded":"2024-03-19T02:51:00.123225+0000","last_fullsized":"2024-03-19T02:51:00.123225+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:24:49.912633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123100+0000","last_change":"2024-03-19T02:50:57.247207+0000","last_active":"2024-03-19T02:51:00.123100+0000","last_peered":"2024-03-19T02:51:00.123100+0000","last_clean":"2024-03-19T02:51:00.123100+0000","last_became_active":"2024-03-19T02:50:57.246776+0000","last_became_peered":"2024-03-19T02:50:57.246776+0000","last_unstale":"2024-03-19T02:51:00.123100+0000","last_undegraded":"2024-03-19T02:51:00.123100+0000","last_fullsized":"2024-03-19T02:51:00.123100+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:02:24.118858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"93'15","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:19.433337+0000","last_change":"2024-03-19T02:50:51.792398+0000","last_active":"2024-03-19T02:51:19.433337+0000","last_peered":"2024-03-19T02:51:19.433337+0000","last_clean":"2024-03-19T02:51:19.433337+0000","last_became_active":"2024-03-19T02:50:51.792090+0000","last_became_peered":"2024-03-19T02:50:51.792090+0000","last_unstale":"2024-03-19T02:51:19.433337+0000","last_undegraded":"2024-03-19T02:51:19.433337+0000","last_fullsized":"2024-03-19T02:51:19.433337+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:02:58.175210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10442,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123668+0000","last_change":"2024-03-19T02:50:52.220755+0000","last_active":"2024-03-19T02:51:00.123668+0000","last_peered":"2024-03-19T02:51:00.123668+0000","last_clean":"2024-03-19T02:51:00.123668+0000","last_became_active":"2024-03-19T02:50:52.220453+0000","last_became_peered":"2024-03-19T02:50:52.220453+0000","last_unstale":"2024-03-19T02:51:00.123668+0000","last_undegraded":"2024-03-19T02:51:00.123668+0000","last_fullsized":"2024-03-19T02:51:00.123668+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:48:27.866863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.246094+0000","last_change":"2024-03-19T02:50:57.253843+0000","last_active":"2024-03-19T02:50:59.246094+0000","last_peered":"2024-03-19T02:50:59.246094+0000","last_clean":"2024-03-19T02:50:59.246094+0000","last_became_active":"2024-03-19T02:50:57.252633+0000","last_became_peered":"2024-03-19T02:50:57.252633+0000","last_unstale":"2024-03-19T02:50:59.246094+0000","last_undegraded":"2024-03-19T02:50:59.246094+0000","last_fullsized":"2024-03-19T02:50:59.246094+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:03:48.514219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"93'6","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:18.079468+0000","last_change":"2024-03-19T02:50:51.795041+0000","last_active":"2024-03-19T02:51:18.079468+0000","last_peered":"2024-03-19T02:51:18.079468+0000","last_clean":"2024-03-19T02:51:18.079468+0000","last_became_active":"2024-03-19T02:50:51.793485+0000","last_became_peered":"2024-03-19T02:50:51.793485+0000","last_unstale":"2024-03-19T02:51:18.079468+0000","last_undegraded":"2024-03-19T02:51:18.079468+0000","last_fullsized":"2024-03-19T02:51:18.079468+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:11:50.136394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245096+0000","last_change":"2024-03-19T02:50:52.227554+0000","last_active":"2024-03-19T02:50:59.245096+0000","last_peered":"2024-03-19T02:50:59.245096+0000","last_clean":"2024-03-19T02:50:59.245096+0000","last_became_active":"2024-03-19T02:50:52.227222+0000","last_became_peered":"2024-03-19T02:50:52.227222+0000","last_unstale":"2024-03-19T02:50:59.245096+0000","last_undegraded":"2024-03-19T02:50:59.245096+0000","last_fullsized":"2024-03-19T02:50:59.245096+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:56.323749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123844+0000","last_change":"2024-03-19T02:50:57.257504+0000","last_active":"2024-03-19T02:51:00.123844+0000","last_peered":"2024-03-19T02:51:00.123844+0000","last_clean":"2024-03-19T02:51:00.123844+0000","last_became_active":"2024-03-19T02:50:57.257120+0000","last_became_peered":"2024-03-19T02:50:57.257120+0000","last_unstale":"2024-03-19T02:51:00.123844+0000","last_undegraded":"2024-03-19T02:51:00.123844+0000","last_fullsized":"2024-03-19T02:51:00.123844+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:19:10.471666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"93'3","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:17.067046+0000","last_change":"2024-03-19T02:50:51.798049+0000","last_active":"2024-03-19T02:51:17.067046+0000","last_peered":"2024-03-19T02:51:17.067046+0000","last_clean":"2024-03-19T02:51:17.067046+0000","last_became_active":"2024-03-19T02:50:51.797134+0000","last_became_peered":"2024-03-19T02:50:51.797134+0000","last_unstale":"2024-03-19T02:51:17.067046+0000","last_undegraded":"2024-03-19T02:51:17.067046+0000","last_fullsized":"2024-03-19T02:51:17.067046+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:03:46.984240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122444+0000","last_change":"2024-03-19T02:50:52.219535+0000","last_active":"2024-03-19T02:51:00.122444+0000","last_peered":"2024-03-19T02:51:00.122444+0000","last_clean":"2024-03-19T02:51:00.122444+0000","last_became_active":"2024-03-19T02:50:52.219226+0000","last_became_peered":"2024-03-19T02:50:52.219226+0000","last_unstale":"2024-03-19T02:51:00.122444+0000","last_undegraded":"2024-03-19T02:51:00.122444+0000","last_fullsized":"2024-03-19T02:51:00.122444+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:27:26.875533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123841+0000","last_change":"2024-03-19T02:50:57.252873+0000","last_active":"2024-03-19T02:51:00.123841+0000","last_peered":"2024-03-19T02:51:00.123841+0000","last_clean":"2024-03-19T02:51:00.123841+0000","last_became_active":"2024-03-19T02:50:57.249551+0000","last_became_peered":"2024-03-19T02:50:57.249551+0000","last_unstale":"2024-03-19T02:51:00.123841+0000","last_undegraded":"2024-03-19T02:51:00.123841+0000","last_fullsized":"2024-03-19T02:51:00.123841+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:33:59.750606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"93'3","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:19.087751+0000","last_change":"2024-03-19T02:50:51.796478+0000","last_active":"2024-03-19T02:51:19.087751+0000","last_peered":"2024-03-19T02:51:19.087751+0000","last_clean":"2024-03-19T02:51:19.087751+0000","last_became_active":"2024-03-19T02:50:51.796126+0000","last_became_peered":"2024-03-19T02:50:51.796126+0000","last_unstale":"2024-03-19T02:51:19.087751+0000","last_undegraded":"2024-03-19T02:51:19.087751+0000","last_fullsized":"2024-03-19T02:51:19.087751+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:13:42.690481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122240+0000","last_change":"2024-03-19T02:50:52.228151+0000","last_active":"2024-03-19T02:51:00.122240+0000","last_peered":"2024-03-19T02:51:00.122240+0000","last_clean":"2024-03-19T02:51:00.122240+0000","last_became_active":"2024-03-19T02:50:52.227858+0000","last_became_peered":"2024-03-19T02:50:52.227858+0000","last_unstale":"2024-03-19T02:51:00.122240+0000","last_undegraded":"2024-03-19T02:51:00.122240+0000","last_fullsized":"2024-03-19T02:51:00.122240+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:10.285606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245298+0000","last_change":"2024-03-19T02:50:57.259269+0000","last_active":"2024-03-19T02:50:59.245298+0000","last_peered":"2024-03-19T02:50:59.245298+0000","last_clean":"2024-03-19T02:50:59.245298+0000","last_became_active":"2024-03-19T02:50:57.258778+0000","last_became_peered":"2024-03-19T02:50:57.258778+0000","last_unstale":"2024-03-19T02:50:59.245298+0000","last_undegraded":"2024-03-19T02:50:59.245298+0000","last_fullsized":"2024-03-19T02:50:59.245298+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:02:44.419512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"93'15","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:18.081718+0000","last_change":"2024-03-19T02:50:51.799246+0000","last_active":"2024-03-19T02:51:18.081718+0000","last_peered":"2024-03-19T02:51:18.081718+0000","last_clean":"2024-03-19T02:51:18.081718+0000","last_became_active":"2024-03-19T02:50:51.798935+0000","last_became_peered":"2024-03-19T02:50:51.798935+0000","last_unstale":"2024-03-19T02:51:18.081718+0000","last_undegraded":"2024-03-19T02:51:18.081718+0000","last_fullsized":"2024-03-19T02:51:18.081718+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:26:10.045267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245197+0000","last_change":"2024-03-19T02:50:52.215997+0000","last_active":"2024-03-19T02:50:59.245197+0000","last_peered":"2024-03-19T02:50:59.245197+0000","last_clean":"2024-03-19T02:50:59.245197+0000","last_became_active":"2024-03-19T02:50:52.215398+0000","last_became_peered":"2024-03-19T02:50:52.215398+0000","last_unstale":"2024-03-19T02:50:59.245197+0000","last_undegraded":"2024-03-19T02:50:59.245197+0000","last_fullsized":"2024-03-19T02:50:59.245197+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:49:14.010736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245331+0000","last_change":"2024-03-19T02:50:57.247254+0000","last_active":"2024-03-19T02:50:59.245331+0000","last_peered":"2024-03-19T02:50:59.245331+0000","last_clean":"2024-03-19T02:50:59.245331+0000","last_became_active":"2024-03-19T02:50:57.246861+0000","last_became_peered":"2024-03-19T02:50:57.246861+0000","last_unstale":"2024-03-19T02:50:59.245331+0000","last_undegraded":"2024-03-19T02:50:59.245331+0000","last_fullsized":"2024-03-19T02:50:59.245331+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:47:20.933550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"93'2","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:19.077099+0000","last_change":"2024-03-19T02:50:51.796246+0000","last_active":"2024-03-19T02:51:19.077099+0000","last_peered":"2024-03-19T02:51:19.077099+0000","last_clean":"2024-03-19T02:51:19.077099+0000","last_became_active":"2024-03-19T02:50:51.795823+0000","last_became_peered":"2024-03-19T02:50:51.795823+0000","last_unstale":"2024-03-19T02:51:19.077099+0000","last_undegraded":"2024-03-19T02:51:19.077099+0000","last_fullsized":"2024-03-19T02:51:19.077099+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:36:02.220891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.124142+0000","last_change":"2024-03-19T02:50:52.224119+0000","last_active":"2024-03-19T02:51:00.124142+0000","last_peered":"2024-03-19T02:51:00.124142+0000","last_clean":"2024-03-19T02:51:00.124142+0000","last_became_active":"2024-03-19T02:50:52.223773+0000","last_became_peered":"2024-03-19T02:50:52.223773+0000","last_unstale":"2024-03-19T02:51:00.124142+0000","last_undegraded":"2024-03-19T02:51:00.124142+0000","last_fullsized":"2024-03-19T02:51:00.124142+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:56:03.098858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123038+0000","last_change":"2024-03-19T02:50:57.652515+0000","last_active":"2024-03-19T02:51:00.123038+0000","last_peered":"2024-03-19T02:51:00.123038+0000","last_clean":"2024-03-19T02:51:00.123038+0000","last_became_active":"2024-03-19T02:50:57.651808+0000","last_became_peered":"2024-03-19T02:50:57.651808+0000","last_unstale":"2024-03-19T02:51:00.123038+0000","last_undegraded":"2024-03-19T02:51:00.123038+0000","last_fullsized":"2024-03-19T02:51:00.123038+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:16:34.550457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123394+0000","last_change":"2024-03-19T02:50:51.794282+0000","last_active":"2024-03-19T02:51:00.123394+0000","last_peered":"2024-03-19T02:51:00.123394+0000","last_clean":"2024-03-19T02:51:00.123394+0000","last_became_active":"2024-03-19T02:50:51.793947+0000","last_became_peered":"2024-03-19T02:50:51.793947+0000","last_unstale":"2024-03-19T02:51:00.123394+0000","last_undegraded":"2024-03-19T02:51:00.123394+0000","last_fullsized":"2024-03-19T02:51:00.123394+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:08:54.764915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.037018+0000","last_change":"2024-03-19T02:50:52.221707+0000","last_active":"2024-03-19T02:51:00.037018+0000","last_peered":"2024-03-19T02:51:00.037018+0000","last_clean":"2024-03-19T02:51:00.037018+0000","last_became_active":"2024-03-19T02:50:52.221417+0000","last_became_peered":"2024-03-19T02:50:52.221417+0000","last_unstale":"2024-03-19T02:51:00.037018+0000","last_undegraded":"2024-03-19T02:51:00.037018+0000","last_fullsized":"2024-03-19T02:51:00.037018+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:59:27.528346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245401+0000","last_change":"2024-03-19T02:50:57.257552+0000","last_active":"2024-03-19T02:50:59.245401+0000","last_peered":"2024-03-19T02:50:59.245401+0000","last_clean":"2024-03-19T02:50:59.245401+0000","last_became_active":"2024-03-19T02:50:57.257121+0000","last_became_peered":"2024-03-19T02:50:57.257121+0000","last_unstale":"2024-03-19T02:50:59.245401+0000","last_undegraded":"2024-03-19T02:50:59.245401+0000","last_fullsized":"2024-03-19T02:50:59.245401+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:43:47.145344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"93'1","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.121899+0000","last_change":"2024-03-19T02:50:51.804890+0000","last_active":"2024-03-19T02:51:00.121899+0000","last_peered":"2024-03-19T02:51:00.121899+0000","last_clean":"2024-03-19T02:51:00.121899+0000","last_became_active":"2024-03-19T02:50:51.804545+0000","last_became_peered":"2024-03-19T02:50:51.804545+0000","last_unstale":"2024-03-19T02:51:00.121899+0000","last_undegraded":"2024-03-19T02:51:00.121899+0000","last_fullsized":"2024-03-19T02:51:00.121899+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:16:14.945372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.036929+0000","last_change":"2024-03-19T02:50:52.224849+0000","last_active":"2024-03-19T02:51:00.036929+0000","last_peered":"2024-03-19T02:51:00.036929+0000","last_clean":"2024-03-19T02:51:00.036929+0000","last_became_active":"2024-03-19T02:50:52.224550+0000","last_became_peered":"2024-03-19T02:50:52.224550+0000","last_unstale":"2024-03-19T02:51:00.036929+0000","last_undegraded":"2024-03-19T02:51:00.036929+0000","last_fullsized":"2024-03-19T02:51:00.036929+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:28:59.683849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245196+0000","last_change":"2024-03-19T02:50:57.652425+0000","last_active":"2024-03-19T02:50:59.245196+0000","last_peered":"2024-03-19T02:50:59.245196+0000","last_clean":"2024-03-19T02:50:59.245196+0000","last_became_active":"2024-03-19T02:50:57.651975+0000","last_became_peered":"2024-03-19T02:50:57.651975+0000","last_unstale":"2024-03-19T02:50:59.245196+0000","last_undegraded":"2024-03-19T02:50:59.245196+0000","last_fullsized":"2024-03-19T02:50:59.245196+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:07:20.587918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245321+0000","last_change":"2024-03-19T02:50:52.218207+0000","last_active":"2024-03-19T02:50:59.245321+0000","last_peered":"2024-03-19T02:50:59.245321+0000","last_clean":"2024-03-19T02:50:59.245321+0000","last_became_active":"2024-03-19T02:50:52.217012+0000","last_became_peered":"2024-03-19T02:50:52.217012+0000","last_unstale":"2024-03-19T02:50:59.245321+0000","last_undegraded":"2024-03-19T02:50:59.245321+0000","last_fullsized":"2024-03-19T02:50:59.245321+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:07:47.366481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:19.085606+0000","last_change":"2024-03-19T02:50:51.801405+0000","last_active":"2024-03-19T02:51:19.085606+0000","last_peered":"2024-03-19T02:51:19.085606+0000","last_clean":"2024-03-19T02:51:19.085606+0000","last_became_active":"2024-03-19T02:50:51.801094+0000","last_became_peered":"2024-03-19T02:50:51.801094+0000","last_unstale":"2024-03-19T02:51:19.085606+0000","last_undegraded":"2024-03-19T02:51:19.085606+0000","last_fullsized":"2024-03-19T02:51:19.085606+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:53:39.534977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"93'1","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122679+0000","last_change":"2024-03-19T02:50:57.245508+0000","last_active":"2024-03-19T02:51:00.122679+0000","last_peered":"2024-03-19T02:51:00.122679+0000","last_clean":"2024-03-19T02:51:00.122679+0000","last_became_active":"2024-03-19T02:50:57.245109+0000","last_became_peered":"2024-03-19T02:50:57.245109+0000","last_unstale":"2024-03-19T02:51:00.122679+0000","last_undegraded":"2024-03-19T02:51:00.122679+0000","last_fullsized":"2024-03-19T02:51:00.122679+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:25:09.085074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122705+0000","last_change":"2024-03-19T02:50:52.221852+0000","last_active":"2024-03-19T02:51:00.122705+0000","last_peered":"2024-03-19T02:51:00.122705+0000","last_clean":"2024-03-19T02:51:00.122705+0000","last_became_active":"2024-03-19T02:50:52.221535+0000","last_became_peered":"2024-03-19T02:50:52.221535+0000","last_unstale":"2024-03-19T02:51:00.122705+0000","last_undegraded":"2024-03-19T02:51:00.122705+0000","last_fullsized":"2024-03-19T02:51:00.122705+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:37:18.030714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"93'3","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:19.083671+0000","last_change":"2024-03-19T02:50:51.790231+0000","last_active":"2024-03-19T02:51:19.083671+0000","last_peered":"2024-03-19T02:51:19.083671+0000","last_clean":"2024-03-19T02:51:19.083671+0000","last_became_active":"2024-03-19T02:50:51.789867+0000","last_became_peered":"2024-03-19T02:50:51.789867+0000","last_unstale":"2024-03-19T02:51:19.083671+0000","last_undegraded":"2024-03-19T02:51:19.083671+0000","last_fullsized":"2024-03-19T02:51:19.083671+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:46.099191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":361,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122201+0000","last_change":"2024-03-19T02:49:38.697673+0000","last_active":"2024-03-19T02:51:00.122201+0000","last_peered":"2024-03-19T02:51:00.122201+0000","last_clean":"2024-03-19T02:51:00.122201+0000","last_became_active":"2024-03-19T02:49:38.697161+0000","last_became_peered":"2024-03-19T02:49:38.697161+0000","last_unstale":"2024-03-19T02:51:00.122201+0000","last_undegraded":"2024-03-19T02:51:00.122201+0000","last_fullsized":"2024-03-19T02:51:00.122201+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:46:10.013962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:46:10.013962+0000","last_clean_scrub_stamp":"2024-03-19T02:46:10.013962+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:59:35.179938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245940+0000","last_change":"2024-03-19T02:50:57.652428+0000","last_active":"2024-03-19T02:50:59.245940+0000","last_peered":"2024-03-19T02:50:59.245940+0000","last_clean":"2024-03-19T02:50:59.245940+0000","last_became_active":"2024-03-19T02:50:57.651864+0000","last_became_peered":"2024-03-19T02:50:57.651864+0000","last_unstale":"2024-03-19T02:50:59.245940+0000","last_undegraded":"2024-03-19T02:50:59.245940+0000","last_fullsized":"2024-03-19T02:50:59.245940+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:52:19.007175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.244774+0000","last_change":"2024-03-19T02:50:52.218728+0000","last_active":"2024-03-19T02:50:59.244774+0000","last_peered":"2024-03-19T02:50:59.244774+0000","last_clean":"2024-03-19T02:50:59.244774+0000","last_became_active":"2024-03-19T02:50:52.218407+0000","last_became_peered":"2024-03-19T02:50:52.218407+0000","last_unstale":"2024-03-19T02:50:59.244774+0000","last_undegraded":"2024-03-19T02:50:59.244774+0000","last_fullsized":"2024-03-19T02:50:59.244774+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:30:49.304034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"93'3","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:18.079179+0000","last_change":"2024-03-19T02:50:51.802057+0000","last_active":"2024-03-19T02:51:18.079179+0000","last_peered":"2024-03-19T02:51:18.079179+0000","last_clean":"2024-03-19T02:51:18.079179+0000","last_became_active":"2024-03-19T02:50:51.801754+0000","last_became_peered":"2024-03-19T02:50:51.801754+0000","last_unstale":"2024-03-19T02:51:18.079179+0000","last_undegraded":"2024-03-19T02:51:18.079179+0000","last_fullsized":"2024-03-19T02:51:18.079179+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:23:15.401822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123530+0000","last_change":"2024-03-19T02:50:57.260303+0000","last_active":"2024-03-19T02:51:00.123530+0000","last_peered":"2024-03-19T02:51:00.123530+0000","last_clean":"2024-03-19T02:51:00.123530+0000","last_became_active":"2024-03-19T02:50:57.259792+0000","last_became_peered":"2024-03-19T02:50:57.259792+0000","last_unstale":"2024-03-19T02:51:00.123530+0000","last_undegraded":"2024-03-19T02:51:00.123530+0000","last_fullsized":"2024-03-19T02:51:00.123530+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:03.072161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"93'1","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:17.067713+0000","last_change":"2024-03-19T02:50:51.797949+0000","last_active":"2024-03-19T02:51:17.067713+0000","last_peered":"2024-03-19T02:51:17.067713+0000","last_clean":"2024-03-19T02:51:17.067713+0000","last_became_active":"2024-03-19T02:50:51.797522+0000","last_became_peered":"2024-03-19T02:50:51.797522+0000","last_unstale":"2024-03-19T02:51:17.067713+0000","last_undegraded":"2024-03-19T02:51:17.067713+0000","last_fullsized":"2024-03-19T02:51:17.067713+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:34:06.127993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245412+0000","last_change":"2024-03-19T02:50:52.224095+0000","last_active":"2024-03-19T02:50:59.245412+0000","last_peered":"2024-03-19T02:50:59.245412+0000","last_clean":"2024-03-19T02:50:59.245412+0000","last_became_active":"2024-03-19T02:50:52.223764+0000","last_became_peered":"2024-03-19T02:50:52.223764+0000","last_unstale":"2024-03-19T02:50:59.245412+0000","last_undegraded":"2024-03-19T02:50:59.245412+0000","last_fullsized":"2024-03-19T02:50:59.245412+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:07:41.503162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123123+0000","last_change":"2024-03-19T02:50:57.652274+0000","last_active":"2024-03-19T02:51:00.123123+0000","last_peered":"2024-03-19T02:51:00.123123+0000","last_clean":"2024-03-19T02:51:00.123123+0000","last_became_active":"2024-03-19T02:50:57.651885+0000","last_became_peered":"2024-03-19T02:50:57.651885+0000","last_unstale":"2024-03-19T02:51:00.123123+0000","last_undegraded":"2024-03-19T02:51:00.123123+0000","last_fullsized":"2024-03-19T02:51:00.123123+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:03:02.081143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'3","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:18.083250+0000","last_change":"2024-03-19T02:50:51.796308+0000","last_active":"2024-03-19T02:51:18.083250+0000","last_peered":"2024-03-19T02:51:18.083250+0000","last_clean":"2024-03-19T02:51:18.083250+0000","last_became_active":"2024-03-19T02:50:51.795990+0000","last_became_peered":"2024-03-19T02:50:51.795990+0000","last_unstale":"2024-03-19T02:51:18.083250+0000","last_undegraded":"2024-03-19T02:51:18.083250+0000","last_fullsized":"2024-03-19T02:51:18.083250+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:45:53.608817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123567+0000","last_change":"2024-03-19T02:50:52.221983+0000","last_active":"2024-03-19T02:51:00.123567+0000","last_peered":"2024-03-19T02:51:00.123567+0000","last_clean":"2024-03-19T02:51:00.123567+0000","last_became_active":"2024-03-19T02:50:52.220605+0000","last_became_peered":"2024-03-19T02:50:52.220605+0000","last_unstale":"2024-03-19T02:51:00.123567+0000","last_undegraded":"2024-03-19T02:51:00.123567+0000","last_fullsized":"2024-03-19T02:51:00.123567+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:12.834159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.652688+0000","last_change":"2024-03-19T02:50:57.242428+0000","last_active":"2024-03-19T02:50:59.652688+0000","last_peered":"2024-03-19T02:50:59.652688+0000","last_clean":"2024-03-19T02:50:59.652688+0000","last_became_active":"2024-03-19T02:50:57.242007+0000","last_became_peered":"2024-03-19T02:50:57.242007+0000","last_unstale":"2024-03-19T02:50:59.652688+0000","last_undegraded":"2024-03-19T02:50:59.652688+0000","last_fullsized":"2024-03-19T02:50:59.652688+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:23:30.987761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"93'4","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:17.066389+0000","last_change":"2024-03-19T02:50:51.791342+0000","last_active":"2024-03-19T02:51:17.066389+0000","last_peered":"2024-03-19T02:51:17.066389+0000","last_clean":"2024-03-19T02:51:17.066389+0000","last_became_active":"2024-03-19T02:50:51.791039+0000","last_became_peered":"2024-03-19T02:50:51.791039+0000","last_unstale":"2024-03-19T02:51:17.066389+0000","last_undegraded":"2024-03-19T02:51:17.066389+0000","last_fullsized":"2024-03-19T02:51:17.066389+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:50:38.216623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.832528+0000","last_change":"2024-03-19T02:50:52.224303+0000","last_active":"2024-03-19T02:50:59.832528+0000","last_peered":"2024-03-19T02:50:59.832528+0000","last_clean":"2024-03-19T02:50:59.832528+0000","last_became_active":"2024-03-19T02:50:52.223942+0000","last_became_peered":"2024-03-19T02:50:52.223942+0000","last_unstale":"2024-03-19T02:50:59.832528+0000","last_undegraded":"2024-03-19T02:50:59.832528+0000","last_fullsized":"2024-03-19T02:50:59.832528+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:57:04.175971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245562+0000","last_change":"2024-03-19T02:50:57.652126+0000","last_active":"2024-03-19T02:50:59.245562+0000","last_peered":"2024-03-19T02:50:59.245562+0000","last_clean":"2024-03-19T02:50:59.245562+0000","last_became_active":"2024-03-19T02:50:57.651713+0000","last_became_peered":"2024-03-19T02:50:57.651713+0000","last_unstale":"2024-03-19T02:50:59.245562+0000","last_undegraded":"2024-03-19T02:50:59.245562+0000","last_fullsized":"2024-03-19T02:50:59.245562+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:28:51.077437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"93'2","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:17.067090+0000","last_change":"2024-03-19T02:50:51.795793+0000","last_active":"2024-03-19T02:51:17.067090+0000","last_peered":"2024-03-19T02:51:17.067090+0000","last_clean":"2024-03-19T02:51:17.067090+0000","last_became_active":"2024-03-19T02:50:51.795448+0000","last_became_peered":"2024-03-19T02:50:51.795448+0000","last_unstale":"2024-03-19T02:51:17.067090+0000","last_undegraded":"2024-03-19T02:51:17.067090+0000","last_fullsized":"2024-03-19T02:51:17.067090+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:33:11.399247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122822+0000","last_change":"2024-03-19T02:50:52.221681+0000","last_active":"2024-03-19T02:51:00.122822+0000","last_peered":"2024-03-19T02:51:00.122822+0000","last_clean":"2024-03-19T02:51:00.122822+0000","last_became_active":"2024-03-19T02:50:52.220987+0000","last_became_peered":"2024-03-19T02:50:52.220987+0000","last_unstale":"2024-03-19T02:51:00.122822+0000","last_undegraded":"2024-03-19T02:51:00.122822+0000","last_fullsized":"2024-03-19T02:51:00.122822+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:16:36.561143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123978+0000","last_change":"2024-03-19T02:50:57.249830+0000","last_active":"2024-03-19T02:51:00.123978+0000","last_peered":"2024-03-19T02:51:00.123978+0000","last_clean":"2024-03-19T02:51:00.123978+0000","last_became_active":"2024-03-19T02:50:57.248879+0000","last_became_peered":"2024-03-19T02:50:57.248879+0000","last_unstale":"2024-03-19T02:51:00.123978+0000","last_undegraded":"2024-03-19T02:51:00.123978+0000","last_fullsized":"2024-03-19T02:51:00.123978+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:58:43.225841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"93'3","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:17.067683+0000","last_change":"2024-03-19T02:50:51.801374+0000","last_active":"2024-03-19T02:51:17.067683+0000","last_peered":"2024-03-19T02:51:17.067683+0000","last_clean":"2024-03-19T02:51:17.067683+0000","last_became_active":"2024-03-19T02:50:51.801041+0000","last_became_peered":"2024-03-19T02:50:51.801041+0000","last_unstale":"2024-03-19T02:51:17.067683+0000","last_undegraded":"2024-03-19T02:51:17.067683+0000","last_fullsized":"2024-03-19T02:51:17.067683+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:56:14.671718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123333+0000","last_change":"2024-03-19T02:50:52.225915+0000","last_active":"2024-03-19T02:51:00.123333+0000","last_peered":"2024-03-19T02:51:00.123333+0000","last_clean":"2024-03-19T02:51:00.123333+0000","last_became_active":"2024-03-19T02:50:52.225606+0000","last_became_peered":"2024-03-19T02:50:52.225606+0000","last_unstale":"2024-03-19T02:51:00.123333+0000","last_undegraded":"2024-03-19T02:51:00.123333+0000","last_fullsized":"2024-03-19T02:51:00.123333+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:27:29.549255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245549+0000","last_change":"2024-03-19T02:50:57.240747+0000","last_active":"2024-03-19T02:50:59.245549+0000","last_peered":"2024-03-19T02:50:59.245549+0000","last_clean":"2024-03-19T02:50:59.245549+0000","last_became_active":"2024-03-19T02:50:57.240373+0000","last_became_peered":"2024-03-19T02:50:57.240373+0000","last_unstale":"2024-03-19T02:50:59.245549+0000","last_undegraded":"2024-03-19T02:50:59.245549+0000","last_fullsized":"2024-03-19T02:50:59.245549+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:22:04.991497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.652699+0000","last_change":"2024-03-19T02:50:51.794071+0000","last_active":"2024-03-19T02:50:59.652699+0000","last_peered":"2024-03-19T02:50:59.652699+0000","last_clean":"2024-03-19T02:50:59.652699+0000","last_became_active":"2024-03-19T02:50:51.793570+0000","last_became_peered":"2024-03-19T02:50:51.793570+0000","last_unstale":"2024-03-19T02:50:59.652699+0000","last_undegraded":"2024-03-19T02:50:59.652699+0000","last_fullsized":"2024-03-19T02:50:59.652699+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:34:20.456515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123180+0000","last_change":"2024-03-19T02:50:52.223341+0000","last_active":"2024-03-19T02:51:00.123180+0000","last_peered":"2024-03-19T02:51:00.123180+0000","last_clean":"2024-03-19T02:51:00.123180+0000","last_became_active":"2024-03-19T02:50:52.220280+0000","last_became_peered":"2024-03-19T02:50:52.220280+0000","last_unstale":"2024-03-19T02:51:00.123180+0000","last_undegraded":"2024-03-19T02:51:00.123180+0000","last_fullsized":"2024-03-19T02:51:00.123180+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:04:15.830811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245369+0000","last_change":"2024-03-19T02:50:57.651293+0000","last_active":"2024-03-19T02:50:59.245369+0000","last_peered":"2024-03-19T02:50:59.245369+0000","last_clean":"2024-03-19T02:50:59.245369+0000","last_became_active":"2024-03-19T02:50:57.650893+0000","last_became_peered":"2024-03-19T02:50:57.650893+0000","last_unstale":"2024-03-19T02:50:59.245369+0000","last_undegraded":"2024-03-19T02:50:59.245369+0000","last_fullsized":"2024-03-19T02:50:59.245369+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:33:35.984728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"93'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:17.067529+0000","last_change":"2024-03-19T02:50:51.798297+0000","last_active":"2024-03-19T02:51:17.067529+0000","last_peered":"2024-03-19T02:51:17.067529+0000","last_clean":"2024-03-19T02:51:17.067529+0000","last_became_active":"2024-03-19T02:50:51.797516+0000","last_became_peered":"2024-03-19T02:50:51.797516+0000","last_unstale":"2024-03-19T02:51:17.067529+0000","last_undegraded":"2024-03-19T02:51:17.067529+0000","last_fullsized":"2024-03-19T02:51:17.067529+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:09:42.295094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122133+0000","last_change":"2024-03-19T02:50:52.224011+0000","last_active":"2024-03-19T02:51:00.122133+0000","last_peered":"2024-03-19T02:51:00.122133+0000","last_clean":"2024-03-19T02:51:00.122133+0000","last_became_active":"2024-03-19T02:50:52.223708+0000","last_became_peered":"2024-03-19T02:50:52.223708+0000","last_unstale":"2024-03-19T02:51:00.122133+0000","last_undegraded":"2024-03-19T02:51:00.122133+0000","last_fullsized":"2024-03-19T02:51:00.122133+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:17:49.305587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123035+0000","last_change":"2024-03-19T02:50:57.251134+0000","last_active":"2024-03-19T02:51:00.123035+0000","last_peered":"2024-03-19T02:51:00.123035+0000","last_clean":"2024-03-19T02:51:00.123035+0000","last_became_active":"2024-03-19T02:50:57.250761+0000","last_became_peered":"2024-03-19T02:50:57.250761+0000","last_unstale":"2024-03-19T02:51:00.123035+0000","last_undegraded":"2024-03-19T02:51:00.123035+0000","last_fullsized":"2024-03-19T02:51:00.123035+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:09:26.860235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"93'3","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:18.081704+0000","last_change":"2024-03-19T02:50:51.794532+0000","last_active":"2024-03-19T02:51:18.081704+0000","last_peered":"2024-03-19T02:51:18.081704+0000","last_clean":"2024-03-19T02:51:18.081704+0000","last_became_active":"2024-03-19T02:50:51.794215+0000","last_became_peered":"2024-03-19T02:50:51.794215+0000","last_unstale":"2024-03-19T02:51:18.081704+0000","last_undegraded":"2024-03-19T02:51:18.081704+0000","last_fullsized":"2024-03-19T02:51:18.081704+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:19:28.060014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122736+0000","last_change":"2024-03-19T02:50:52.224776+0000","last_active":"2024-03-19T02:51:00.122736+0000","last_peered":"2024-03-19T02:51:00.122736+0000","last_clean":"2024-03-19T02:51:00.122736+0000","last_became_active":"2024-03-19T02:50:52.224442+0000","last_became_peered":"2024-03-19T02:50:52.224442+0000","last_unstale":"2024-03-19T02:51:00.122736+0000","last_undegraded":"2024-03-19T02:51:00.122736+0000","last_fullsized":"2024-03-19T02:51:00.122736+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:11:13.722005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.832239+0000","last_change":"2024-03-19T02:50:57.250322+0000","last_active":"2024-03-19T02:50:59.832239+0000","last_peered":"2024-03-19T02:50:59.832239+0000","last_clean":"2024-03-19T02:50:59.832239+0000","last_became_active":"2024-03-19T02:50:57.249890+0000","last_became_peered":"2024-03-19T02:50:57.249890+0000","last_unstale":"2024-03-19T02:50:59.832239+0000","last_undegraded":"2024-03-19T02:50:59.832239+0000","last_fullsized":"2024-03-19T02:50:59.832239+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:36:14.169985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245870+0000","last_change":"2024-03-19T02:50:51.794898+0000","last_active":"2024-03-19T02:50:59.245870+0000","last_peered":"2024-03-19T02:50:59.245870+0000","last_clean":"2024-03-19T02:50:59.245870+0000","last_became_active":"2024-03-19T02:50:51.794562+0000","last_became_peered":"2024-03-19T02:50:51.794562+0000","last_unstale":"2024-03-19T02:50:59.245870+0000","last_undegraded":"2024-03-19T02:50:59.245870+0000","last_fullsized":"2024-03-19T02:50:59.245870+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:17:45.239395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122369+0000","last_change":"2024-03-19T02:50:52.227427+0000","last_active":"2024-03-19T02:51:00.122369+0000","last_peered":"2024-03-19T02:51:00.122369+0000","last_clean":"2024-03-19T02:51:00.122369+0000","last_became_active":"2024-03-19T02:50:52.227111+0000","last_became_peered":"2024-03-19T02:50:52.227111+0000","last_unstale":"2024-03-19T02:51:00.122369+0000","last_undegraded":"2024-03-19T02:51:00.122369+0000","last_fullsized":"2024-03-19T02:51:00.122369+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:40:53.295454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245066+0000","last_change":"2024-03-19T02:50:57.244449+0000","last_active":"2024-03-19T02:50:59.245066+0000","last_peered":"2024-03-19T02:50:59.245066+0000","last_clean":"2024-03-19T02:50:59.245066+0000","last_became_active":"2024-03-19T02:50:57.243991+0000","last_became_peered":"2024-03-19T02:50:57.243991+0000","last_unstale":"2024-03-19T02:50:59.245066+0000","last_undegraded":"2024-03-19T02:50:59.245066+0000","last_fullsized":"2024-03-19T02:50:59.245066+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:50:45.265773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245271+0000","last_change":"2024-03-19T02:50:51.799834+0000","last_active":"2024-03-19T02:50:59.245271+0000","last_peered":"2024-03-19T02:50:59.245271+0000","last_clean":"2024-03-19T02:50:59.245271+0000","last_became_active":"2024-03-19T02:50:51.799563+0000","last_became_peered":"2024-03-19T02:50:51.799563+0000","last_unstale":"2024-03-19T02:50:59.245271+0000","last_undegraded":"2024-03-19T02:50:59.245271+0000","last_fullsized":"2024-03-19T02:50:59.245271+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:51:00.284454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.122953+0000","last_change":"2024-03-19T02:50:52.222710+0000","last_active":"2024-03-19T02:51:00.122953+0000","last_peered":"2024-03-19T02:51:00.122953+0000","last_clean":"2024-03-19T02:51:00.122953+0000","last_became_active":"2024-03-19T02:50:52.222321+0000","last_became_peered":"2024-03-19T02:50:52.222321+0000","last_unstale":"2024-03-19T02:51:00.122953+0000","last_undegraded":"2024-03-19T02:51:00.122953+0000","last_fullsized":"2024-03-19T02:51:00.122953+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:05:27.527696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123595+0000","last_change":"2024-03-19T02:50:52.226631+0000","last_active":"2024-03-19T02:51:00.123595+0000","last_peered":"2024-03-19T02:51:00.123595+0000","last_clean":"2024-03-19T02:51:00.123595+0000","last_became_active":"2024-03-19T02:50:52.226294+0000","last_became_peered":"2024-03-19T02:50:52.226294+0000","last_unstale":"2024-03-19T02:51:00.123595+0000","last_undegraded":"2024-03-19T02:51:00.123595+0000","last_fullsized":"2024-03-19T02:51:00.123595+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:01:31.350063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.653221+0000","last_change":"2024-03-19T02:50:52.221138+0000","last_active":"2024-03-19T02:50:59.653221+0000","last_peered":"2024-03-19T02:50:59.653221+0000","last_clean":"2024-03-19T02:50:59.653221+0000","last_became_active":"2024-03-19T02:50:52.220730+0000","last_became_peered":"2024-03-19T02:50:52.220730+0000","last_unstale":"2024-03-19T02:50:59.653221+0000","last_undegraded":"2024-03-19T02:50:59.653221+0000","last_fullsized":"2024-03-19T02:50:59.653221+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:26:13.059986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.652639+0000","last_change":"2024-03-19T02:50:52.223567+0000","last_active":"2024-03-19T02:50:59.652639+0000","last_peered":"2024-03-19T02:50:59.652639+0000","last_clean":"2024-03-19T02:50:59.652639+0000","last_became_active":"2024-03-19T02:50:52.223174+0000","last_became_peered":"2024-03-19T02:50:52.223174+0000","last_unstale":"2024-03-19T02:50:59.652639+0000","last_undegraded":"2024-03-19T02:50:59.652639+0000","last_fullsized":"2024-03-19T02:50:59.652639+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:23:06.383584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.832350+0000","last_change":"2024-03-19T02:50:52.229916+0000","last_active":"2024-03-19T02:50:59.832350+0000","last_peered":"2024-03-19T02:50:59.832350+0000","last_clean":"2024-03-19T02:50:59.832350+0000","last_became_active":"2024-03-19T02:50:52.229597+0000","last_became_peered":"2024-03-19T02:50:52.229597+0000","last_unstale":"2024-03-19T02:50:59.832350+0000","last_undegraded":"2024-03-19T02:50:59.832350+0000","last_fullsized":"2024-03-19T02:50:59.832350+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:19.331893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245325+0000","last_change":"2024-03-19T02:50:52.223331+0000","last_active":"2024-03-19T02:50:59.245325+0000","last_peered":"2024-03-19T02:50:59.245325+0000","last_clean":"2024-03-19T02:50:59.245325+0000","last_became_active":"2024-03-19T02:50:52.222975+0000","last_became_peered":"2024-03-19T02:50:52.222975+0000","last_unstale":"2024-03-19T02:50:59.245325+0000","last_undegraded":"2024-03-19T02:50:59.245325+0000","last_fullsized":"2024-03-19T02:50:59.245325+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:26:23.023749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.124357+0000","last_change":"2024-03-19T02:50:52.218073+0000","last_active":"2024-03-19T02:51:00.124357+0000","last_peered":"2024-03-19T02:51:00.124357+0000","last_clean":"2024-03-19T02:51:00.124357+0000","last_became_active":"2024-03-19T02:50:52.217742+0000","last_became_peered":"2024-03-19T02:50:52.217742+0000","last_unstale":"2024-03-19T02:51:00.124357+0000","last_undegraded":"2024-03-19T02:51:00.124357+0000","last_fullsized":"2024-03-19T02:51:00.124357+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:08:51.053153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.653317+0000","last_change":"2024-03-19T02:50:52.222108+0000","last_active":"2024-03-19T02:50:59.653317+0000","last_peered":"2024-03-19T02:50:59.653317+0000","last_clean":"2024-03-19T02:50:59.653317+0000","last_became_active":"2024-03-19T02:50:52.221748+0000","last_became_peered":"2024-03-19T02:50:52.221748+0000","last_unstale":"2024-03-19T02:50:59.653317+0000","last_undegraded":"2024-03-19T02:50:59.653317+0000","last_fullsized":"2024-03-19T02:50:59.653317+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:33:31.401939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123790+0000","last_change":"2024-03-19T02:50:52.221799+0000","last_active":"2024-03-19T02:51:00.123790+0000","last_peered":"2024-03-19T02:51:00.123790+0000","last_clean":"2024-03-19T02:51:00.123790+0000","last_became_active":"2024-03-19T02:50:52.220196+0000","last_became_peered":"2024-03-19T02:50:52.220196+0000","last_unstale":"2024-03-19T02:51:00.123790+0000","last_undegraded":"2024-03-19T02:51:00.123790+0000","last_fullsized":"2024-03-19T02:51:00.123790+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:24:01.541716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245221+0000","last_change":"2024-03-19T02:50:52.223102+0000","last_active":"2024-03-19T02:50:59.245221+0000","last_peered":"2024-03-19T02:50:59.245221+0000","last_clean":"2024-03-19T02:50:59.245221+0000","last_became_active":"2024-03-19T02:50:52.222801+0000","last_became_peered":"2024-03-19T02:50:52.222801+0000","last_unstale":"2024-03-19T02:50:59.245221+0000","last_undegraded":"2024-03-19T02:50:59.245221+0000","last_fullsized":"2024-03-19T02:50:59.245221+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:48:48.471048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123072+0000","last_change":"2024-03-19T02:50:52.221296+0000","last_active":"2024-03-19T02:51:00.123072+0000","last_peered":"2024-03-19T02:51:00.123072+0000","last_clean":"2024-03-19T02:51:00.123072+0000","last_became_active":"2024-03-19T02:50:52.219141+0000","last_became_peered":"2024-03-19T02:50:52.219141+0000","last_unstale":"2024-03-19T02:51:00.123072+0000","last_undegraded":"2024-03-19T02:51:00.123072+0000","last_fullsized":"2024-03-19T02:51:00.123072+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:19:48.345528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.123187+0000","last_change":"2024-03-19T02:50:52.230847+0000","last_active":"2024-03-19T02:51:00.123187+0000","last_peered":"2024-03-19T02:51:00.123187+0000","last_clean":"2024-03-19T02:51:00.123187+0000","last_became_active":"2024-03-19T02:50:52.230478+0000","last_became_peered":"2024-03-19T02:50:52.230478+0000","last_unstale":"2024-03-19T02:51:00.123187+0000","last_undegraded":"2024-03-19T02:51:00.123187+0000","last_fullsized":"2024-03-19T02:51:00.123187+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:04:09.459446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:50:59.245081+0000","last_change":"2024-03-19T02:50:52.223508+0000","last_active":"2024-03-19T02:50:59.245081+0000","last_peered":"2024-03-19T02:50:59.245081+0000","last_clean":"2024-03-19T02:50:59.245081+0000","last_became_active":"2024-03-19T02:50:52.223173+0000","last_became_peered":"2024-03-19T02:50:52.223173+0000","last_unstale":"2024-03-19T02:50:59.245081+0000","last_undegraded":"2024-03-19T02:50:59.245081+0000","last_fullsized":"2024-03-19T02:50:59.245081+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:53.320173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T02:51:00.124338+0000","last_change":"2024-03-19T02:50:52.229548+0000","last_active":"2024-03-19T02:51:00.124338+0000","last_peered":"2024-03-19T02:51:00.124338+0000","last_clean":"2024-03-19T02:51:00.124338+0000","last_became_active":"2024-03-19T02:50:52.229203+0000","last_became_peered":"2024-03-19T02:50:52.229203+0000","last_unstale":"2024-03-19T02:51:00.124338+0000","last_undegraded":"2024-03-19T02:51:00.124338+0000","last_fullsized":"2024-03-19T02:51:00.124338+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:48:45.884098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":14052,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":26,"num_read_kb":24,"num_write":131,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":47673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318290,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1032192,"data_stored":503210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579927,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28648,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716504,"statfs":{"total":95995035648,"available":95965700096,"internally_reserved":0,"allocated":1671168,"data_stored":1098998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743453,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1052672,"data_stored":504028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005090,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1056768,"data_stored":508044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266728,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32788,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712364,"statfs":{"total":95995035648,"available":95961460736,"internally_reserved":0,"allocated":1650688,"data_stored":1103562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462955,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":503300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1052672,"data_stored":513652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738443,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":504688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":508402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149756,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":503494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888119,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":503854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724594,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28636,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716516,"statfs":{"total":95995035648,"available":95965712384,"internally_reserved":0,"allocated":1658880,"data_stored":1099150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":23141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":12020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T02:51:22.222 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-19T02:51:22.222 INFO:tasks.cephadm:Setting up client nodes... 2024-03-19T02:51:22.223 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-19T02:51:22.366 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:22 smithi096 ceph-mon[35946]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:51:22.366 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:22 smithi096 ceph-mon[35946]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-19T02:51:22.366 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:22 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/232614714' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:51:22.366 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:22 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1633399558' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:51:22.366 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:22 smithi096 ceph-mon[35946]: pgmap v314: 129 pgs: 129 active+clean; 590 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-03-19T02:51:22.501 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:51:22.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:22 smithi155 ceph-mon[35804]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:51:22.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:22 smithi155 ceph-mon[35804]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-19T02:51:22.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:22 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/232614714' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:51:22.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:22 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1633399558' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:51:22.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:22 smithi155 ceph-mon[35804]: pgmap v314: 129 pgs: 129 active+clean; 590 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-03-19T02:51:23.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:23 smithi033 ceph-mon[31973]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:51:23.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:23 smithi033 ceph-mon[31973]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:51:23.380 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:23 smithi096 ceph-mon[35946]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:51:23.381 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:23 smithi096 ceph-mon[35946]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:51:23.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:23 smithi155 ceph-mon[35804]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:51:23.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:23 smithi155 ceph-mon[35804]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:51:24.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.216+0000 7f8885a87700 1 -- 172.21.15.33:0/300233710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8880101260 msgr2=0x7f8880101680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:24.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.216+0000 7f8885a87700 1 --2- 172.21.15.33:0/300233710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8880101260 0x7f8880101680 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f88700099c0 tx=0x7f8870009cd0 comp rx=0 tx=0).stop 2024-03-19T02:51:24.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.216+0000 7f8885a87700 1 -- 172.21.15.33:0/300233710 shutdown_connections 2024-03-19T02:51:24.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.216+0000 7f8885a87700 1 --2- 172.21.15.33:0/300233710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8880102e00 0x7f88801096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:24.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.216+0000 7f8885a87700 1 --2- 172.21.15.33:0/300233710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8880101260 0x7f8880101680 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:24.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.216+0000 7f8885a87700 1 --2- 172.21.15.33:0/300233710 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8880102460 0x7f88801028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:24.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.216+0000 7f8885a87700 1 -- 172.21.15.33:0/300233710 >> 172.21.15.33:0/300233710 conn(0x7f88800fc800 msgr2=0x7f88800fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:24.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.216+0000 7f8885a87700 1 -- 172.21.15.33:0/300233710 shutdown_connections 2024-03-19T02:51:24.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.216+0000 7f8885a87700 1 -- 172.21.15.33:0/300233710 wait complete. 2024-03-19T02:51:24.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.217+0000 7f8885a87700 1 Processor -- start 2024-03-19T02:51:24.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.217+0000 7f8885a87700 1 -- start start 2024-03-19T02:51:24.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.217+0000 7f8885a87700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8880101260 0x7f8880198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:24.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.217+0000 7f8885a87700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8880102460 0x7f8880198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:24.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.217+0000 7f8885a87700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8880102e00 0x7f888019f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:24.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.217+0000 7f8885a87700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f888010cd00 con 0x7f8880102460 2024-03-19T02:51:24.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.217+0000 7f8885a87700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f888010cb80 con 0x7f8880101260 2024-03-19T02:51:24.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.217+0000 7f8885a87700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f888010ce80 con 0x7f8880102e00 2024-03-19T02:51:24.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.218+0000 7f887effd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8880102460 0x7f8880198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:24.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.218+0000 7f887effd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8880102460 0x7f8880198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38336/0 (socket says 172.21.15.33:38336) 2024-03-19T02:51:24.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.218+0000 7f887effd700 1 -- 172.21.15.33:0/2910894718 learned_addr learned my addr 172.21.15.33:0/2910894718 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:24.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.218+0000 7f887ffff700 1 --2- 172.21.15.33:0/2910894718 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8880102e00 0x7f888019f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:24.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.218+0000 7f887f7fe700 1 --2- 172.21.15.33:0/2910894718 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8880101260 0x7f8880198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:24.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.218+0000 7f887effd700 1 -- 172.21.15.33:0/2910894718 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8880102e00 msgr2=0x7f888019f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:24.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.218+0000 7f887effd700 1 --2- 172.21.15.33:0/2910894718 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8880102e00 0x7f888019f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:24.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.218+0000 7f887effd700 1 -- 172.21.15.33:0/2910894718 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8880101260 msgr2=0x7f8880198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:24.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.218+0000 7f887effd700 1 --2- 172.21.15.33:0/2910894718 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8880101260 0x7f8880198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:24.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.218+0000 7f887effd700 1 -- 172.21.15.33:0/2910894718 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88801037d0 con 0x7f8880102460 2024-03-19T02:51:24.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.218+0000 7f887effd700 1 --2- 172.21.15.33:0/2910894718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8880102460 0x7f8880198970 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f887400e9d0 tx=0x7f887400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:24.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.219+0000 7f887cff9700 1 -- 172.21.15.33:0/2910894718 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f887400cbd0 con 0x7f8880102460 2024-03-19T02:51:24.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.219+0000 7f887cff9700 1 -- 172.21.15.33:0/2910894718 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f887400ee70 con 0x7f8880102460 2024-03-19T02:51:24.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.219+0000 7f8885a87700 1 -- 172.21.15.33:0/2910894718 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8880103ac0 con 0x7f8880102460 2024-03-19T02:51:24.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.219+0000 7f887cff9700 1 -- 172.21.15.33:0/2910894718 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8874018680 con 0x7f8880102460 2024-03-19T02:51:24.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.219+0000 7f8885a87700 1 -- 172.21.15.33:0/2910894718 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8880104050 con 0x7f8880102460 2024-03-19T02:51:24.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.220+0000 7f8885a87700 1 -- 172.21.15.33:0/2910894718 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8880050820 con 0x7f8880102460 2024-03-19T02:51:24.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.221+0000 7f887cff9700 1 -- 172.21.15.33:0/2910894718 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f8874010430 con 0x7f8880102460 2024-03-19T02:51:24.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.222+0000 7f887cff9700 1 --2- 172.21.15.33:0/2910894718 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f88680a2a50 0x7f88680a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:24.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.222+0000 7f887f7fe700 1 --2- 172.21.15.33:0/2910894718 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f88680a2a50 0x7f88680a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:24.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.222+0000 7f887cff9700 1 -- 172.21.15.33:0/2910894718 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7f88740cf000 con 0x7f8880102460 2024-03-19T02:51:24.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.223+0000 7f887f7fe700 1 --2- 172.21.15.33:0/2910894718 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f88680a2a50 0x7f88680a4f10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f887000b580 tx=0x7f8870005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:24.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.227+0000 7f887cff9700 1 -- 172.21.15.33:0/2910894718 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f887400cd70 con 0x7f8880102460 2024-03-19T02:51:24.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:24 smithi033 ceph-mon[31973]: pgmap v315: 129 pgs: 129 active+clean; 590 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3.0 KiB/s wr, 9 op/s 2024-03-19T02:51:24.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:24 smithi096 ceph-mon[35946]: pgmap v315: 129 pgs: 129 active+clean; 590 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3.0 KiB/s wr, 9 op/s 2024-03-19T02:51:24.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:24 smithi155 ceph-mon[35804]: pgmap v315: 129 pgs: 129 active+clean; 590 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3.0 KiB/s wr, 9 op/s 2024-03-19T02:51:24.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.518+0000 7f8885a87700 1 -- 172.21.15.33:0/2910894718 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f8880104620 con 0x7f8880102460 2024-03-19T02:51:24.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.520+0000 7f887cff9700 1 -- 172.21.15.33:0/2910894718 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f8880104620 con 0x7f8880102460 2024-03-19T02:51:24.521 INFO:teuthology.orchestra.run.smithi033.stdout:[client.0] 2024-03-19T02:51:24.521 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQCs/fhlJGb9HhAAdEvg1NmL/cjZ50Sb5kQLnQ== 2024-03-19T02:51:24.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.523+0000 7f8885a87700 1 -- 172.21.15.33:0/2910894718 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f88680a2a50 msgr2=0x7f88680a4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:24.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.523+0000 7f8885a87700 1 --2- 172.21.15.33:0/2910894718 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f88680a2a50 0x7f88680a4f10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f887000b580 tx=0x7f8870005fb0 comp rx=0 tx=0).stop 2024-03-19T02:51:24.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.523+0000 7f8885a87700 1 -- 172.21.15.33:0/2910894718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8880102460 msgr2=0x7f8880198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:24.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.523+0000 7f8885a87700 1 --2- 172.21.15.33:0/2910894718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8880102460 0x7f8880198970 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f887400e9d0 tx=0x7f887400ece0 comp rx=0 tx=0).stop 2024-03-19T02:51:24.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.523+0000 7f8885a87700 1 -- 172.21.15.33:0/2910894718 shutdown_connections 2024-03-19T02:51:24.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.523+0000 7f8885a87700 1 --2- 172.21.15.33:0/2910894718 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f88680a2a50 0x7f88680a4f10 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:24.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.523+0000 7f8885a87700 1 --2- 172.21.15.33:0/2910894718 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8880102e00 0x7f888019f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:24.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.523+0000 7f8885a87700 1 --2- 172.21.15.33:0/2910894718 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8880101260 0x7f8880198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:24.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.523+0000 7f8885a87700 1 --2- 172.21.15.33:0/2910894718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8880102460 0x7f8880198970 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:24.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.523+0000 7f8885a87700 1 -- 172.21.15.33:0/2910894718 >> 172.21.15.33:0/2910894718 conn(0x7f88800fc800 msgr2=0x7f888006aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:24.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.523+0000 7f8885a87700 1 -- 172.21.15.33:0/2910894718 shutdown_connections 2024-03-19T02:51:24.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:24.523+0000 7f8885a87700 1 -- 172.21.15.33:0/2910894718 wait complete. 2024-03-19T02:51:25.287 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-19T02:51:25.287 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-19T02:51:25.287 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-19T02:51:25.329 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-19T02:51:25.329 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-19T02:51:25.330 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph mgr dump --format=json 2024-03-19T02:51:25.394 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:25 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2910894718' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-19T02:51:25.394 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:25 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2910894718' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-19T02:51:25.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:25 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2910894718' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-19T02:51:25.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:25 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2910894718' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-19T02:51:25.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:25 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2910894718' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-19T02:51:25.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:25 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2910894718' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-19T02:51:25.627 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:51:26.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:26 smithi096 ceph-mon[35946]: pgmap v316: 129 pgs: 129 active+clean; 625 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 6.2 KiB/s wr, 13 op/s 2024-03-19T02:51:26.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:26 smithi155 ceph-mon[35804]: pgmap v316: 129 pgs: 129 active+clean; 625 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 6.2 KiB/s wr, 13 op/s 2024-03-19T02:51:26.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:26 smithi033 ceph-mon[31973]: pgmap v316: 129 pgs: 129 active+clean; 625 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 6.2 KiB/s wr, 13 op/s 2024-03-19T02:51:27.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.400+0000 7fbd59535700 1 -- 172.21.15.33:0/254496628 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd541014a0 msgr2=0x7fbd54101880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:27.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.400+0000 7fbd59535700 1 --2- 172.21.15.33:0/254496628 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd541014a0 0x7fbd54101880 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbd4400b0a0 tx=0x7fbd4400b3b0 comp rx=0 tx=0).stop 2024-03-19T02:51:27.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.400+0000 7fbd59535700 1 -- 172.21.15.33:0/254496628 shutdown_connections 2024-03-19T02:51:27.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.400+0000 7fbd59535700 1 --2- 172.21.15.33:0/254496628 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbd54106530 0x7fbd5410e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:27.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.400+0000 7fbd59535700 1 --2- 172.21.15.33:0/254496628 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd541014a0 0x7fbd54101880 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:27.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.400+0000 7fbd59535700 1 --2- 172.21.15.33:0/254496628 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd54101e50 0x7fbd54105e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:27.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.400+0000 7fbd59535700 1 -- 172.21.15.33:0/254496628 >> 172.21.15.33:0/254496628 conn(0x7fbd54077230 msgr2=0x7fbd54077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:27.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.400+0000 7fbd59535700 1 -- 172.21.15.33:0/254496628 shutdown_connections 2024-03-19T02:51:27.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.400+0000 7fbd59535700 1 -- 172.21.15.33:0/254496628 wait complete. 2024-03-19T02:51:27.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.401+0000 7fbd59535700 1 Processor -- start 2024-03-19T02:51:27.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.401+0000 7fbd59535700 1 -- start start 2024-03-19T02:51:27.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.401+0000 7fbd59535700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd541014a0 0x7fbd5419e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:27.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.401+0000 7fbd59535700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbd54101e50 0x7fbd5419ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:27.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.401+0000 7fbd59535700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd54106530 0x7fbd54198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:27.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.401+0000 7fbd59535700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbd541113c0 con 0x7fbd54101e50 2024-03-19T02:51:27.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.401+0000 7fbd59535700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbd54111240 con 0x7fbd541014a0 2024-03-19T02:51:27.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.401+0000 7fbd59535700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbd54111540 con 0x7fbd54106530 2024-03-19T02:51:27.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.401+0000 7fbd527fc700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbd54101e50 0x7fbd5419ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:27.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.401+0000 7fbd527fc700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbd54101e50 0x7fbd5419ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38352/0 (socket says 172.21.15.33:38352) 2024-03-19T02:51:27.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.401+0000 7fbd527fc700 1 -- 172.21.15.33:0/3859117487 learned_addr learned my addr 172.21.15.33:0/3859117487 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:27.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.401+0000 7fbd52ffd700 1 --2- 172.21.15.33:0/3859117487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd541014a0 0x7fbd5419e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:27.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.401+0000 7fbd537fe700 1 --2- 172.21.15.33:0/3859117487 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd54106530 0x7fbd54198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:27.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.402+0000 7fbd52ffd700 1 -- 172.21.15.33:0/3859117487 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd54106530 msgr2=0x7fbd54198830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:27.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.402+0000 7fbd52ffd700 1 --2- 172.21.15.33:0/3859117487 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd54106530 0x7fbd54198830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:27.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.402+0000 7fbd52ffd700 1 -- 172.21.15.33:0/3859117487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbd54101e50 msgr2=0x7fbd5419ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:27.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.402+0000 7fbd52ffd700 1 --2- 172.21.15.33:0/3859117487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbd54101e50 0x7fbd5419ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:27.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.402+0000 7fbd52ffd700 1 -- 172.21.15.33:0/3859117487 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd54199010 con 0x7fbd541014a0 2024-03-19T02:51:27.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.402+0000 7fbd527fc700 1 --2- 172.21.15.33:0/3859117487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbd54101e50 0x7fbd5419ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:27.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.402+0000 7fbd52ffd700 1 --2- 172.21.15.33:0/3859117487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd541014a0 0x7fbd5419e6b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbd4400b070 tx=0x7fbd44004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:27.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.402+0000 7fbd4bfff700 1 -- 172.21.15.33:0/3859117487 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd44021070 con 0x7fbd541014a0 2024-03-19T02:51:27.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.402+0000 7fbd59535700 1 -- 172.21.15.33:0/3859117487 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd541992a0 con 0x7fbd541014a0 2024-03-19T02:51:27.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.402+0000 7fbd4bfff700 1 -- 172.21.15.33:0/3859117487 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbd44007e30 con 0x7fbd541014a0 2024-03-19T02:51:27.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.402+0000 7fbd4bfff700 1 -- 172.21.15.33:0/3859117487 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd440045e0 con 0x7fbd541014a0 2024-03-19T02:51:27.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.402+0000 7fbd59535700 1 -- 172.21.15.33:0/3859117487 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd541a5510 con 0x7fbd541014a0 2024-03-19T02:51:27.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.403+0000 7fbd59535700 1 -- 172.21.15.33:0/3859117487 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd54102930 con 0x7fbd541014a0 2024-03-19T02:51:27.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.404+0000 7fbd4bfff700 1 -- 172.21.15.33:0/3859117487 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fbd44007980 con 0x7fbd541014a0 2024-03-19T02:51:27.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.404+0000 7fbd4bfff700 1 --2- 172.21.15.33:0/3859117487 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbd400a2920 0x7fbd400a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:27.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.405+0000 7fbd527fc700 1 --2- 172.21.15.33:0/3859117487 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbd400a2920 0x7fbd400a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:27.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.405+0000 7fbd4bfff700 1 -- 172.21.15.33:0/3859117487 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7fbd4400f020 con 0x7fbd541014a0 2024-03-19T02:51:27.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.405+0000 7fbd527fc700 1 --2- 172.21.15.33:0/3859117487 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbd400a2920 0x7fbd400a4de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbd54106090 tx=0x7fbd3c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:27.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.408+0000 7fbd4bfff700 1 -- 172.21.15.33:0/3859117487 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbd4402a020 con 0x7fbd541014a0 2024-03-19T02:51:27.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.671+0000 7fbd59535700 1 -- 172.21.15.33:0/3859117487 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fbd54199d60 con 0x7fbd541014a0 2024-03-19T02:51:27.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.676+0000 7fbd4bfff700 1 -- 172.21.15.33:0/3859117487 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+262660 (secure 0 0 0) 0x7fbd4409d1e0 con 0x7fbd541014a0 2024-03-19T02:51:27.681 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:51:27.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.680+0000 7fbd59535700 1 -- 172.21.15.33:0/3859117487 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbd400a2920 msgr2=0x7fbd400a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:27.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.680+0000 7fbd59535700 1 --2- 172.21.15.33:0/3859117487 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbd400a2920 0x7fbd400a4de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbd54106090 tx=0x7fbd3c009260 comp rx=0 tx=0).stop 2024-03-19T02:51:27.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.680+0000 7fbd59535700 1 -- 172.21.15.33:0/3859117487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd541014a0 msgr2=0x7fbd5419e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:27.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.680+0000 7fbd59535700 1 --2- 172.21.15.33:0/3859117487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd541014a0 0x7fbd5419e6b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbd4400b070 tx=0x7fbd44004d00 comp rx=0 tx=0).stop 2024-03-19T02:51:27.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.680+0000 7fbd59535700 1 -- 172.21.15.33:0/3859117487 shutdown_connections 2024-03-19T02:51:27.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.680+0000 7fbd59535700 1 --2- 172.21.15.33:0/3859117487 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbd400a2920 0x7fbd400a4de0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:27.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.680+0000 7fbd59535700 1 --2- 172.21.15.33:0/3859117487 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd54106530 0x7fbd54198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:27.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.680+0000 7fbd59535700 1 --2- 172.21.15.33:0/3859117487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd541014a0 0x7fbd5419e6b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:27.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.680+0000 7fbd59535700 1 --2- 172.21.15.33:0/3859117487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbd54101e50 0x7fbd5419ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:27.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.680+0000 7fbd59535700 1 -- 172.21.15.33:0/3859117487 >> 172.21.15.33:0/3859117487 conn(0x7fbd54077230 msgr2=0x7fbd5406ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:27.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.680+0000 7fbd59535700 1 -- 172.21.15.33:0/3859117487 shutdown_connections 2024-03-19T02:51:27.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:27.680+0000 7fbd59535700 1 -- 172.21.15.33:0/3859117487 wait complete. 2024-03-19T02:51:27.987 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:27 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3859117487' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-19T02:51:28.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:27 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3859117487' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-19T02:51:28.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:27 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3859117487' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-19T02:51:28.526 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1453067701},{"type":"v1","addr":"172.21.15.33:6801","nonce":1453067701}]},"active_addr":"172.21.15.33:6801/1453067701","active_change":"2024-03-19T02:43:39.622322+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24115,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24122,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.33:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":1950732854}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":176174888}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":4099467400}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":3554132372}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":2221692066}]}]} 2024-03-19T02:51:28.532 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-19T02:51:28.532 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-19T02:51:28.532 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-19T02:51:28.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:28 smithi033 ceph-mon[31973]: pgmap v317: 129 pgs: 129 active+clean; 625 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 6.2 KiB/s wr, 13 op/s 2024-03-19T02:51:28.780 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:51:29.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:28 smithi155 ceph-mon[35804]: pgmap v317: 129 pgs: 129 active+clean; 625 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 6.2 KiB/s wr, 13 op/s 2024-03-19T02:51:29.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:28 smithi096 ceph-mon[35946]: pgmap v317: 129 pgs: 129 active+clean; 625 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 6.2 KiB/s wr, 13 op/s 2024-03-19T02:51:30.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.367+0000 7fa5a07c0700 1 -- 172.21.15.33:0/463430971 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa5981014d0 msgr2=0x7fa5981018b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:30.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.367+0000 7fa5a07c0700 1 --2- 172.21.15.33:0/463430971 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa5981014d0 0x7fa5981018b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa5880099c0 tx=0x7fa588009cd0 comp rx=0 tx=0).stop 2024-03-19T02:51:30.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.367+0000 7fa5a07c0700 1 -- 172.21.15.33:0/463430971 shutdown_connections 2024-03-19T02:51:30.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.367+0000 7fa5a07c0700 1 --2- 172.21.15.33:0/463430971 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa598106520 0x7fa59810e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:30.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.367+0000 7fa5a07c0700 1 --2- 172.21.15.33:0/463430971 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa5981014d0 0x7fa5981018b0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:30.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.367+0000 7fa5a07c0700 1 --2- 172.21.15.33:0/463430971 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa598101e80 0x7fa598105e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:30.370 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.367+0000 7fa5a07c0700 1 -- 172.21.15.33:0/463430971 >> 172.21.15.33:0/463430971 conn(0x7fa598077260 msgr2=0x7fa598077670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:30.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.367+0000 7fa5a07c0700 1 -- 172.21.15.33:0/463430971 shutdown_connections 2024-03-19T02:51:30.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.367+0000 7fa5a07c0700 1 -- 172.21.15.33:0/463430971 wait complete. 2024-03-19T02:51:30.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.368+0000 7fa5a07c0700 1 Processor -- start 2024-03-19T02:51:30.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.368+0000 7fa5a07c0700 1 -- start start 2024-03-19T02:51:30.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.368+0000 7fa5a07c0700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5981014d0 0x7fa59819e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:30.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.368+0000 7fa5a07c0700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa598101e80 0x7fa59819ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:30.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.368+0000 7fa5a07c0700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa598106520 0x7fa598198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:30.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.368+0000 7fa5a07c0700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa598111420 con 0x7fa598101e80 2024-03-19T02:51:30.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.368+0000 7fa5a07c0700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa5981112a0 con 0x7fa5981014d0 2024-03-19T02:51:30.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.368+0000 7fa5a07c0700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa5981115a0 con 0x7fa598106520 2024-03-19T02:51:30.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.368+0000 7fa59dd5b700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa598101e80 0x7fa59819ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:30.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.368+0000 7fa59e55c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5981014d0 0x7fa59819e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:30.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.368+0000 7fa59e55c700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5981014d0 0x7fa59819e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:46438/0 (socket says 172.21.15.33:46438) 2024-03-19T02:51:30.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.368+0000 7fa59ed5d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa598106520 0x7fa598198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:30.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.368+0000 7fa59e55c700 1 -- 172.21.15.33:0/3391172164 learned_addr learned my addr 172.21.15.33:0/3391172164 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:30.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.368+0000 7fa59dd5b700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa598101e80 0x7fa59819ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38374/0 (socket says 172.21.15.33:38374) 2024-03-19T02:51:30.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.369+0000 7fa59e55c700 1 -- 172.21.15.33:0/3391172164 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa598106520 msgr2=0x7fa598198830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:30.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.369+0000 7fa59e55c700 1 --2- 172.21.15.33:0/3391172164 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa598106520 0x7fa598198830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:30.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.369+0000 7fa59e55c700 1 -- 172.21.15.33:0/3391172164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa598101e80 msgr2=0x7fa59819ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:30.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.369+0000 7fa59e55c700 1 --2- 172.21.15.33:0/3391172164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa598101e80 0x7fa59819ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:30.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.369+0000 7fa59e55c700 1 -- 172.21.15.33:0/3391172164 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa598199010 con 0x7fa5981014d0 2024-03-19T02:51:30.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.369+0000 7fa59dd5b700 1 --2- 172.21.15.33:0/3391172164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa598101e80 0x7fa59819ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:30.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.369+0000 7fa59ed5d700 1 --2- 172.21.15.33:0/3391172164 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa598106520 0x7fa598198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:30.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.369+0000 7fa59e55c700 1 --2- 172.21.15.33:0/3391172164 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5981014d0 0x7fa59819e6b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa588005910 tx=0x7fa588004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:30.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.369+0000 7fa58f7fe700 1 -- 172.21.15.33:0/3391172164 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa588026070 con 0x7fa5981014d0 2024-03-19T02:51:30.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.369+0000 7fa5a07c0700 1 -- 172.21.15.33:0/3391172164 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5981992a0 con 0x7fa5981014d0 2024-03-19T02:51:30.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.369+0000 7fa58f7fe700 1 -- 172.21.15.33:0/3391172164 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa588003c70 con 0x7fa5981014d0 2024-03-19T02:51:30.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.369+0000 7fa58f7fe700 1 -- 172.21.15.33:0/3391172164 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa58800fa40 con 0x7fa5981014d0 2024-03-19T02:51:30.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.369+0000 7fa5a07c0700 1 -- 172.21.15.33:0/3391172164 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5981a5510 con 0x7fa5981014d0 2024-03-19T02:51:30.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.370+0000 7fa5a07c0700 1 -- 172.21.15.33:0/3391172164 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa598102920 con 0x7fa5981014d0 2024-03-19T02:51:30.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.371+0000 7fa58f7fe700 1 -- 172.21.15.33:0/3391172164 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fa588029070 con 0x7fa5981014d0 2024-03-19T02:51:30.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.372+0000 7fa58f7fe700 1 --2- 172.21.15.33:0/3391172164 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fa5840a2920 0x7fa5840a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:30.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.372+0000 7fa59dd5b700 1 --2- 172.21.15.33:0/3391172164 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fa5840a2920 0x7fa5840a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:30.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.372+0000 7fa58f7fe700 1 -- 172.21.15.33:0/3391172164 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7fa5880cfbe0 con 0x7fa5981014d0 2024-03-19T02:51:30.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.372+0000 7fa59dd5b700 1 --2- 172.21.15.33:0/3391172164 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fa5840a2920 0x7fa5840a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa598101140 tx=0x7fa594009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:30.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.380+0000 7fa58f7fe700 1 -- 172.21.15.33:0/3391172164 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa588030080 con 0x7fa5981014d0 2024-03-19T02:51:30.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.588+0000 7fa5a07c0700 1 -- 172.21.15.33:0/3391172164 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa598199d60 con 0x7fa5981014d0 2024-03-19T02:51:30.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.589+0000 7fa58f7fe700 1 -- 172.21.15.33:0/3391172164 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24043 (secure 0 0 0) 0x7fa588013070 con 0x7fa5981014d0 2024-03-19T02:51:30.591 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:51:30.591 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":93,"fsid":"2b7a8938-e59a-11ee-95c9-87774f69a715","created":"2024-03-19T02:42:37.912179+0000","modified":"2024-03-19T02:50:59.241668+0000","last_up_change":"2024-03-19T02:50:01.539392+0000","last_in_change":"2024-03-19T02:49:40.534415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:46:09.963737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:50:50.069641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:50:51.381064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:50:55.476667+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"090a89ab-6b7b-4317-8823-178d4212b2f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6803","nonce":1633124995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6805","nonce":1633124995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6809","nonce":1633124995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6807","nonce":1633124995}]},"public_addr":"172.21.15.33:6803/1633124995","cluster_addr":"172.21.15.33:6805/1633124995","heartbeat_back_addr":"172.21.15.33:6809/1633124995","heartbeat_front_addr":"172.21.15.33:6807/1633124995","state":["exists","up"]},{"osd":1,"uuid":"f50eec55-1956-4486-8593-f2c8fa610e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6801","nonce":340144398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6803","nonce":340144398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6807","nonce":340144398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6805","nonce":340144398}]},"public_addr":"172.21.15.96:6801/340144398","cluster_addr":"172.21.15.96:6803/340144398","heartbeat_back_addr":"172.21.15.96:6807/340144398","heartbeat_front_addr":"172.21.15.96:6805/340144398","state":["exists","up"]},{"osd":2,"uuid":"893a2d62-80d9-4e5e-bc6a-a1c8c681d652","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6801","nonce":3388743876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6803","nonce":3388743876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6807","nonce":3388743876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6805","nonce":3388743876}]},"public_addr":"172.21.15.155:6801/3388743876","cluster_addr":"172.21.15.155:6803/3388743876","heartbeat_back_addr":"172.21.15.155:6807/3388743876","heartbeat_front_addr":"172.21.15.155:6805/3388743876","state":["exists","up"]},{"osd":3,"uuid":"b07e0482-4281-4d32-9dfe-9995d7246175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6811","nonce":677007171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6813","nonce":677007171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6817","nonce":677007171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6815","nonce":677007171}]},"public_addr":"172.21.15.33:6811/677007171","cluster_addr":"172.21.15.33:6813/677007171","heartbeat_back_addr":"172.21.15.33:6817/677007171","heartbeat_front_addr":"172.21.15.33:6815/677007171","state":["exists","up"]},{"osd":4,"uuid":"a988384b-af6b-47c6-8c07-5c9673acee45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6809","nonce":3106158554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6811","nonce":3106158554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6815","nonce":3106158554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6813","nonce":3106158554}]},"public_addr":"172.21.15.96:6809/3106158554","cluster_addr":"172.21.15.96:6811/3106158554","heartbeat_back_addr":"172.21.15.96:6815/3106158554","heartbeat_front_addr":"172.21.15.96:6813/3106158554","state":["exists","up"]},{"osd":5,"uuid":"198dc992-96f8-4c52-96ee-1fd5de148bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6809","nonce":327293324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6811","nonce":327293324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6815","nonce":327293324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6813","nonce":327293324}]},"public_addr":"172.21.15.155:6809/327293324","cluster_addr":"172.21.15.155:6811/327293324","heartbeat_back_addr":"172.21.15.155:6815/327293324","heartbeat_front_addr":"172.21.15.155:6813/327293324","state":["exists","up"]},{"osd":6,"uuid":"a7278ec0-ab2e-49ed-be53-84d8b9e24177","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6819","nonce":873871509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6821","nonce":873871509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6825","nonce":873871509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6823","nonce":873871509}]},"public_addr":"172.21.15.33:6819/873871509","cluster_addr":"172.21.15.33:6821/873871509","heartbeat_back_addr":"172.21.15.33:6825/873871509","heartbeat_front_addr":"172.21.15.33:6823/873871509","state":["exists","up"]},{"osd":7,"uuid":"727e133d-aa24-4ffc-90e2-266eb2aab8c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6817","nonce":2334123891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6819","nonce":2334123891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6823","nonce":2334123891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6821","nonce":2334123891}]},"public_addr":"172.21.15.96:6817/2334123891","cluster_addr":"172.21.15.96:6819/2334123891","heartbeat_back_addr":"172.21.15.96:6823/2334123891","heartbeat_front_addr":"172.21.15.96:6821/2334123891","state":["exists","up"]},{"osd":8,"uuid":"bb20eec0-39e5-4b86-9db5-486b1840cdad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6817","nonce":3498516903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6819","nonce":3498516903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6823","nonce":3498516903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6821","nonce":3498516903}]},"public_addr":"172.21.15.155:6817/3498516903","cluster_addr":"172.21.15.155:6819/3498516903","heartbeat_back_addr":"172.21.15.155:6823/3498516903","heartbeat_front_addr":"172.21.15.155:6821/3498516903","state":["exists","up"]},{"osd":9,"uuid":"179c6228-3e23-4f65-91a6-1ef0005d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6827","nonce":760777213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6829","nonce":760777213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6833","nonce":760777213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6831","nonce":760777213}]},"public_addr":"172.21.15.33:6827/760777213","cluster_addr":"172.21.15.33:6829/760777213","heartbeat_back_addr":"172.21.15.33:6833/760777213","heartbeat_front_addr":"172.21.15.33:6831/760777213","state":["exists","up"]},{"osd":10,"uuid":"be17b478-3c2e-41fb-bede-958333acdf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6825","nonce":1404898482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6827","nonce":1404898482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6831","nonce":1404898482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6829","nonce":1404898482}]},"public_addr":"172.21.15.96:6825/1404898482","cluster_addr":"172.21.15.96:6827/1404898482","heartbeat_back_addr":"172.21.15.96:6831/1404898482","heartbeat_front_addr":"172.21.15.96:6829/1404898482","state":["exists","up"]},{"osd":11,"uuid":"c0903ed2-cc1e-4c18-8731-1f0ee4dd0457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6825","nonce":792352827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6827","nonce":792352827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6831","nonce":792352827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6829","nonce":792352827}]},"public_addr":"172.21.15.155:6825/792352827","cluster_addr":"172.21.15.155:6827/792352827","heartbeat_back_addr":"172.21.15.155:6831/792352827","heartbeat_front_addr":"172.21.15.155:6829/792352827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:17.942801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:42.109449+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:05.981599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:32.069574+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:57.431015+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:22.448592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:48.526515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:14.498352+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:40.175426+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:06.364006+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:32.853161+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:59.073538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:6801/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:6800/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/1067768047":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/729379453":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2586588742":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/2932814165":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6801/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2212564963":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6800/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/3275126058":"2024-03-20T02:43:39.622218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:51:30.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.592+0000 7fa5a07c0700 1 -- 172.21.15.33:0/3391172164 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fa5840a2920 msgr2=0x7fa5840a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:30.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.592+0000 7fa5a07c0700 1 --2- 172.21.15.33:0/3391172164 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fa5840a2920 0x7fa5840a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa598101140 tx=0x7fa594009260 comp rx=0 tx=0).stop 2024-03-19T02:51:30.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.592+0000 7fa5a07c0700 1 -- 172.21.15.33:0/3391172164 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5981014d0 msgr2=0x7fa59819e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:30.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.592+0000 7fa5a07c0700 1 --2- 172.21.15.33:0/3391172164 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5981014d0 0x7fa59819e6b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa588005910 tx=0x7fa588004270 comp rx=0 tx=0).stop 2024-03-19T02:51:30.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.592+0000 7fa5a07c0700 1 -- 172.21.15.33:0/3391172164 shutdown_connections 2024-03-19T02:51:30.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.592+0000 7fa5a07c0700 1 --2- 172.21.15.33:0/3391172164 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fa5840a2920 0x7fa5840a4de0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:30.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.592+0000 7fa5a07c0700 1 --2- 172.21.15.33:0/3391172164 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa598106520 0x7fa598198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:30.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.592+0000 7fa5a07c0700 1 --2- 172.21.15.33:0/3391172164 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5981014d0 0x7fa59819e6b0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:30.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.592+0000 7fa5a07c0700 1 --2- 172.21.15.33:0/3391172164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa598101e80 0x7fa59819ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:30.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.592+0000 7fa5a07c0700 1 -- 172.21.15.33:0/3391172164 >> 172.21.15.33:0/3391172164 conn(0x7fa598077260 msgr2=0x7fa59806ad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:30.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.592+0000 7fa5a07c0700 1 -- 172.21.15.33:0/3391172164 shutdown_connections 2024-03-19T02:51:30.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:30.592+0000 7fa5a07c0700 1 -- 172.21.15.33:0/3391172164 wait complete. 2024-03-19T02:51:30.908 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:30 smithi033 ceph-mon[31973]: pgmap v318: 129 pgs: 129 active+clean; 637 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.6 KiB/s wr, 15 op/s 2024-03-19T02:51:30.909 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:30 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3391172164' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:51:31.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:30 smithi155 ceph-mon[35804]: pgmap v318: 129 pgs: 129 active+clean; 637 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.6 KiB/s wr, 15 op/s 2024-03-19T02:51:31.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:30 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3391172164' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:51:31.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:30 smithi096 ceph-mon[35946]: pgmap v318: 129 pgs: 129 active+clean; 637 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.6 KiB/s wr, 15 op/s 2024-03-19T02:51:31.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:30 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3391172164' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:51:31.295 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-19T02:51:31.295 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-19T02:51:31.545 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:51:32.985 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:32 smithi033 ceph-mon[31973]: pgmap v319: 129 pgs: 129 active+clean; 637 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.7 KiB/s wr, 13 op/s 2024-03-19T02:51:33.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:32 smithi155 ceph-mon[35804]: pgmap v319: 129 pgs: 129 active+clean; 637 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.7 KiB/s wr, 13 op/s 2024-03-19T02:51:33.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:32 smithi096 ceph-mon[35946]: pgmap v319: 129 pgs: 129 active+clean; 637 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.7 KiB/s wr, 13 op/s 2024-03-19T02:51:33.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.356+0000 7f3b23ee9700 1 -- 172.21.15.33:0/734904142 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b1c101210 msgr2=0x7f3b1c101670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:33.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.356+0000 7f3b23ee9700 1 --2- 172.21.15.33:0/734904142 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b1c101210 0x7f3b1c101670 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c009990 tx=0x7f3b0c009ca0 comp rx=0 tx=0).stop 2024-03-19T02:51:33.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.356+0000 7f3b23ee9700 1 -- 172.21.15.33:0/734904142 shutdown_connections 2024-03-19T02:51:33.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.356+0000 7f3b23ee9700 1 --2- 172.21.15.33:0/734904142 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3b1c101bb0 0x7f3b1c10e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:33.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.356+0000 7f3b23ee9700 1 --2- 172.21.15.33:0/734904142 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b1c107250 0x7f3b1c107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:33.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.356+0000 7f3b23ee9700 1 --2- 172.21.15.33:0/734904142 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b1c101210 0x7f3b1c101670 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:33.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.356+0000 7f3b23ee9700 1 -- 172.21.15.33:0/734904142 >> 172.21.15.33:0/734904142 conn(0x7f3b1c078140 msgr2=0x7f3b1c078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:33.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.356+0000 7f3b23ee9700 1 -- 172.21.15.33:0/734904142 shutdown_connections 2024-03-19T02:51:33.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.356+0000 7f3b23ee9700 1 -- 172.21.15.33:0/734904142 wait complete. 2024-03-19T02:51:33.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.357+0000 7f3b23ee9700 1 Processor -- start 2024-03-19T02:51:33.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.357+0000 7f3b23ee9700 1 -- start start 2024-03-19T02:51:33.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.357+0000 7f3b23ee9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b1c101210 0x7f3b1c10e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:33.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.357+0000 7f3b23ee9700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b1c101bb0 0x7f3b1c10eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:33.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.357+0000 7f3b23ee9700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3b1c107250 0x7f3b1c10f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:33.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.357+0000 7f3b23ee9700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3b1c1124a0 con 0x7f3b1c101bb0 2024-03-19T02:51:33.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.357+0000 7f3b23ee9700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3b1c112320 con 0x7f3b1c101210 2024-03-19T02:51:33.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.357+0000 7f3b23ee9700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3b1c112620 con 0x7f3b1c107250 2024-03-19T02:51:33.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.357+0000 7f3b21484700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b1c101bb0 0x7f3b1c10eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:33.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.357+0000 7f3b21484700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b1c101bb0 0x7f3b1c10eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38382/0 (socket says 172.21.15.33:38382) 2024-03-19T02:51:33.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.357+0000 7f3b21c85700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b1c101210 0x7f3b1c10e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:33.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.357+0000 7f3b21484700 1 -- 172.21.15.33:0/425044032 learned_addr learned my addr 172.21.15.33:0/425044032 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:33.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.357+0000 7f3b22486700 1 --2- 172.21.15.33:0/425044032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3b1c107250 0x7f3b1c10f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:33.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.358+0000 7f3b21c85700 1 -- 172.21.15.33:0/425044032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3b1c107250 msgr2=0x7f3b1c10f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:33.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.358+0000 7f3b21c85700 1 --2- 172.21.15.33:0/425044032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3b1c107250 0x7f3b1c10f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:33.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.358+0000 7f3b21c85700 1 -- 172.21.15.33:0/425044032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b1c101bb0 msgr2=0x7f3b1c10eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:33.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.358+0000 7f3b21c85700 1 --2- 172.21.15.33:0/425044032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b1c101bb0 0x7f3b1c10eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:33.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.358+0000 7f3b21c85700 1 -- 172.21.15.33:0/425044032 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b1c1a6f00 con 0x7f3b1c101210 2024-03-19T02:51:33.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.358+0000 7f3b21c85700 1 --2- 172.21.15.33:0/425044032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b1c101210 0x7f3b1c10e9a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3b1800e9f0 tx=0x7f3b1800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:33.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.358+0000 7f3b12ffd700 1 -- 172.21.15.33:0/425044032 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1800c9c0 con 0x7f3b1c101210 2024-03-19T02:51:33.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.358+0000 7f3b12ffd700 1 -- 172.21.15.33:0/425044032 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3b1800cb60 con 0x7f3b1c101210 2024-03-19T02:51:33.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.358+0000 7f3b23ee9700 1 -- 172.21.15.33:0/425044032 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b1c1a71f0 con 0x7f3b1c101210 2024-03-19T02:51:33.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.358+0000 7f3b12ffd700 1 -- 172.21.15.33:0/425044032 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b18018680 con 0x7f3b1c101210 2024-03-19T02:51:33.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.358+0000 7f3b23ee9700 1 -- 172.21.15.33:0/425044032 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b1c1a7730 con 0x7f3b1c101210 2024-03-19T02:51:33.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.359+0000 7f3b23ee9700 1 -- 172.21.15.33:0/425044032 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b1c068fd0 con 0x7f3b1c101210 2024-03-19T02:51:33.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.360+0000 7f3b12ffd700 1 -- 172.21.15.33:0/425044032 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f3b18010430 con 0x7f3b1c101210 2024-03-19T02:51:33.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.360+0000 7f3b12ffd700 1 --2- 172.21.15.33:0/425044032 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f3b080a2920 0x7f3b080a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:33.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.361+0000 7f3b21484700 1 --2- 172.21.15.33:0/425044032 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f3b080a2920 0x7f3b080a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:33.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.361+0000 7f3b12ffd700 1 -- 172.21.15.33:0/425044032 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8802+0+0 (secure 0 0 0) 0x7f3b180cee70 con 0x7f3b1c101210 2024-03-19T02:51:33.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.361+0000 7f3b21484700 1 --2- 172.21.15.33:0/425044032 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f3b080a2920 0x7f3b080a4de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c0ff990 tx=0x7f3b0c00b610 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:33.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.365+0000 7f3b12ffd700 1 -- 172.21.15.33:0/425044032 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3b1809bb70 con 0x7f3b1c101210 2024-03-19T02:51:33.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.570+0000 7f3b23ee9700 1 -- 172.21.15.33:0/425044032 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3b1c02dbb0 con 0x7f3b1c101210 2024-03-19T02:51:33.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.572+0000 7f3b12ffd700 1 -- 172.21.15.33:0/425044032 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24043 (secure 0 0 0) 0x7f3b1809b470 con 0x7f3b1c101210 2024-03-19T02:51:33.573 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:51:33.573 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":93,"fsid":"2b7a8938-e59a-11ee-95c9-87774f69a715","created":"2024-03-19T02:42:37.912179+0000","modified":"2024-03-19T02:50:59.241668+0000","last_up_change":"2024-03-19T02:50:01.539392+0000","last_in_change":"2024-03-19T02:49:40.534415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:46:09.963737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:50:50.069641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:50:51.381064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:50:55.476667+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"090a89ab-6b7b-4317-8823-178d4212b2f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6803","nonce":1633124995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6805","nonce":1633124995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6809","nonce":1633124995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6807","nonce":1633124995}]},"public_addr":"172.21.15.33:6803/1633124995","cluster_addr":"172.21.15.33:6805/1633124995","heartbeat_back_addr":"172.21.15.33:6809/1633124995","heartbeat_front_addr":"172.21.15.33:6807/1633124995","state":["exists","up"]},{"osd":1,"uuid":"f50eec55-1956-4486-8593-f2c8fa610e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6801","nonce":340144398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6803","nonce":340144398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6807","nonce":340144398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6805","nonce":340144398}]},"public_addr":"172.21.15.96:6801/340144398","cluster_addr":"172.21.15.96:6803/340144398","heartbeat_back_addr":"172.21.15.96:6807/340144398","heartbeat_front_addr":"172.21.15.96:6805/340144398","state":["exists","up"]},{"osd":2,"uuid":"893a2d62-80d9-4e5e-bc6a-a1c8c681d652","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6801","nonce":3388743876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6803","nonce":3388743876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6807","nonce":3388743876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6805","nonce":3388743876}]},"public_addr":"172.21.15.155:6801/3388743876","cluster_addr":"172.21.15.155:6803/3388743876","heartbeat_back_addr":"172.21.15.155:6807/3388743876","heartbeat_front_addr":"172.21.15.155:6805/3388743876","state":["exists","up"]},{"osd":3,"uuid":"b07e0482-4281-4d32-9dfe-9995d7246175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6811","nonce":677007171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6813","nonce":677007171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6817","nonce":677007171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6815","nonce":677007171}]},"public_addr":"172.21.15.33:6811/677007171","cluster_addr":"172.21.15.33:6813/677007171","heartbeat_back_addr":"172.21.15.33:6817/677007171","heartbeat_front_addr":"172.21.15.33:6815/677007171","state":["exists","up"]},{"osd":4,"uuid":"a988384b-af6b-47c6-8c07-5c9673acee45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6809","nonce":3106158554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6811","nonce":3106158554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6815","nonce":3106158554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6813","nonce":3106158554}]},"public_addr":"172.21.15.96:6809/3106158554","cluster_addr":"172.21.15.96:6811/3106158554","heartbeat_back_addr":"172.21.15.96:6815/3106158554","heartbeat_front_addr":"172.21.15.96:6813/3106158554","state":["exists","up"]},{"osd":5,"uuid":"198dc992-96f8-4c52-96ee-1fd5de148bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6809","nonce":327293324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6811","nonce":327293324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6815","nonce":327293324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6813","nonce":327293324}]},"public_addr":"172.21.15.155:6809/327293324","cluster_addr":"172.21.15.155:6811/327293324","heartbeat_back_addr":"172.21.15.155:6815/327293324","heartbeat_front_addr":"172.21.15.155:6813/327293324","state":["exists","up"]},{"osd":6,"uuid":"a7278ec0-ab2e-49ed-be53-84d8b9e24177","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6819","nonce":873871509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6821","nonce":873871509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6825","nonce":873871509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6823","nonce":873871509}]},"public_addr":"172.21.15.33:6819/873871509","cluster_addr":"172.21.15.33:6821/873871509","heartbeat_back_addr":"172.21.15.33:6825/873871509","heartbeat_front_addr":"172.21.15.33:6823/873871509","state":["exists","up"]},{"osd":7,"uuid":"727e133d-aa24-4ffc-90e2-266eb2aab8c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6817","nonce":2334123891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6819","nonce":2334123891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6823","nonce":2334123891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6821","nonce":2334123891}]},"public_addr":"172.21.15.96:6817/2334123891","cluster_addr":"172.21.15.96:6819/2334123891","heartbeat_back_addr":"172.21.15.96:6823/2334123891","heartbeat_front_addr":"172.21.15.96:6821/2334123891","state":["exists","up"]},{"osd":8,"uuid":"bb20eec0-39e5-4b86-9db5-486b1840cdad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6817","nonce":3498516903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6819","nonce":3498516903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6823","nonce":3498516903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6821","nonce":3498516903}]},"public_addr":"172.21.15.155:6817/3498516903","cluster_addr":"172.21.15.155:6819/3498516903","heartbeat_back_addr":"172.21.15.155:6823/3498516903","heartbeat_front_addr":"172.21.15.155:6821/3498516903","state":["exists","up"]},{"osd":9,"uuid":"179c6228-3e23-4f65-91a6-1ef0005d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6827","nonce":760777213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6829","nonce":760777213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6833","nonce":760777213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6831","nonce":760777213}]},"public_addr":"172.21.15.33:6827/760777213","cluster_addr":"172.21.15.33:6829/760777213","heartbeat_back_addr":"172.21.15.33:6833/760777213","heartbeat_front_addr":"172.21.15.33:6831/760777213","state":["exists","up"]},{"osd":10,"uuid":"be17b478-3c2e-41fb-bede-958333acdf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6825","nonce":1404898482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6827","nonce":1404898482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6831","nonce":1404898482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6829","nonce":1404898482}]},"public_addr":"172.21.15.96:6825/1404898482","cluster_addr":"172.21.15.96:6827/1404898482","heartbeat_back_addr":"172.21.15.96:6831/1404898482","heartbeat_front_addr":"172.21.15.96:6829/1404898482","state":["exists","up"]},{"osd":11,"uuid":"c0903ed2-cc1e-4c18-8731-1f0ee4dd0457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6825","nonce":792352827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6827","nonce":792352827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6831","nonce":792352827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6829","nonce":792352827}]},"public_addr":"172.21.15.155:6825/792352827","cluster_addr":"172.21.15.155:6827/792352827","heartbeat_back_addr":"172.21.15.155:6831/792352827","heartbeat_front_addr":"172.21.15.155:6829/792352827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:17.942801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:42.109449+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:05.981599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:32.069574+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:57.431015+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:22.448592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:48.526515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:14.498352+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:40.175426+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:06.364006+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:32.853161+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:59.073538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:6801/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:6800/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/1067768047":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/729379453":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2586588742":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/2932814165":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6801/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2212564963":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6800/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/3275126058":"2024-03-20T02:43:39.622218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:51:33.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.574+0000 7f3b23ee9700 1 -- 172.21.15.33:0/425044032 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f3b080a2920 msgr2=0x7f3b080a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:33.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.574+0000 7f3b23ee9700 1 --2- 172.21.15.33:0/425044032 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f3b080a2920 0x7f3b080a4de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c0ff990 tx=0x7f3b0c00b610 comp rx=0 tx=0).stop 2024-03-19T02:51:33.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.574+0000 7f3b23ee9700 1 -- 172.21.15.33:0/425044032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b1c101210 msgr2=0x7f3b1c10e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:33.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.574+0000 7f3b23ee9700 1 --2- 172.21.15.33:0/425044032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b1c101210 0x7f3b1c10e9a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3b1800e9f0 tx=0x7f3b1800ed00 comp rx=0 tx=0).stop 2024-03-19T02:51:33.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.574+0000 7f3b23ee9700 1 -- 172.21.15.33:0/425044032 shutdown_connections 2024-03-19T02:51:33.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.574+0000 7f3b23ee9700 1 --2- 172.21.15.33:0/425044032 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f3b080a2920 0x7f3b080a4de0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:33.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.574+0000 7f3b23ee9700 1 --2- 172.21.15.33:0/425044032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3b1c107250 0x7f3b1c10f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:33.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.574+0000 7f3b23ee9700 1 --2- 172.21.15.33:0/425044032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b1c101210 0x7f3b1c10e9a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:33.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.574+0000 7f3b23ee9700 1 --2- 172.21.15.33:0/425044032 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b1c101bb0 0x7f3b1c10eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:33.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.574+0000 7f3b23ee9700 1 -- 172.21.15.33:0/425044032 >> 172.21.15.33:0/425044032 conn(0x7f3b1c078140 msgr2=0x7f3b1c100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:33.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.574+0000 7f3b23ee9700 1 -- 172.21.15.33:0/425044032 shutdown_connections 2024-03-19T02:51:33.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:33.574+0000 7f3b23ee9700 1 -- 172.21.15.33:0/425044032 wait complete. 2024-03-19T02:51:34.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:33 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/425044032' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:51:34.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:33 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/425044032' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:51:34.130 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:33 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/425044032' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:51:34.457 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-19T02:51:34.457 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-19T02:51:34.458 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-19T02:51:34.458 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-19T02:51:34.459 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-19T02:51:34.459 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-19T02:51:34.459 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-19T02:51:34.460 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-19T02:51:34.460 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-19T02:51:34.461 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-19T02:51:34.461 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-19T02:51:34.461 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-19T02:51:35.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:34 smithi155 ceph-mon[35804]: pgmap v320: 129 pgs: 129 active+clean; 637 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 4.8 KiB/s wr, 7 op/s 2024-03-19T02:51:35.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:34 smithi033 ceph-mon[31973]: pgmap v320: 129 pgs: 129 active+clean; 637 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 4.8 KiB/s wr, 7 op/s 2024-03-19T02:51:35.146 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:51:35.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:34 smithi096 ceph-mon[35946]: pgmap v320: 129 pgs: 129 active+clean; 637 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 4.8 KiB/s wr, 7 op/s 2024-03-19T02:51:35.489 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:51:35.490 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:51:35.490 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:51:35.490 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:51:35.491 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:51:35.507 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:51:35.508 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:51:35.513 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:51:36.008 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:35 smithi033 ceph-mon[31973]: pgmap v321: 129 pgs: 129 active+clean; 637 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 4.6 KiB/s wr, 9 op/s 2024-03-19T02:51:36.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:35 smithi155 ceph-mon[35804]: pgmap v321: 129 pgs: 129 active+clean; 637 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 4.6 KiB/s wr, 9 op/s 2024-03-19T02:51:36.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:35 smithi096 ceph-mon[35946]: pgmap v321: 129 pgs: 129 active+clean; 637 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 4.6 KiB/s wr, 9 op/s 2024-03-19T02:51:36.543 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:51:36.544 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:51:38.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:37 smithi033 ceph-mon[31973]: pgmap v322: 129 pgs: 129 active+clean; 637 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-19T02:51:38.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:37 smithi096 ceph-mon[35946]: pgmap v322: 129 pgs: 129 active+clean; 637 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-19T02:51:38.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:37 smithi155 ceph-mon[35804]: pgmap v322: 129 pgs: 129 active+clean; 637 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-19T02:51:39.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:39 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-19T02:51:39.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:39 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-03-19T02:51:39.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:39 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-03-19T02:51:39.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:39 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-03-19T02:51:39.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:39 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 6]}]: dispatch 2024-03-19T02:51:39.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:39 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]: dispatch 2024-03-19T02:51:39.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:39 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-19T02:51:39.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:39 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-03-19T02:51:39.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:39 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]: dispatch 2024-03-19T02:51:39.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:39 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2024-03-19T02:51:40.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:39 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-19T02:51:40.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:39 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-03-19T02:51:40.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:39 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-03-19T02:51:40.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:39 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-03-19T02:51:40.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:39 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 6]}]: dispatch 2024-03-19T02:51:40.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:39 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]: dispatch 2024-03-19T02:51:40.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:39 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-19T02:51:40.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:39 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-03-19T02:51:40.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:39 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]: dispatch 2024-03-19T02:51:40.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:39 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2024-03-19T02:51:40.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:39 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-19T02:51:40.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:39 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-03-19T02:51:40.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:39 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-03-19T02:51:40.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:39 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-03-19T02:51:40.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:39 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 6]}]: dispatch 2024-03-19T02:51:40.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:39 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]: dispatch 2024-03-19T02:51:40.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:39 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-03-19T02:51:40.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:39 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-03-19T02:51:40.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:39 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]: dispatch 2024-03-19T02:51:40.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:39 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2024-03-19T02:51:40.520 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:51:41.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-19T02:51:41.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-03-19T02:51:41.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-03-19T02:51:41.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-03-19T02:51:41.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 6]}]': finished 2024-03-19T02:51:41.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]': finished 2024-03-19T02:51:41.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-19T02:51:41.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-03-19T02:51:41.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]': finished 2024-03-19T02:51:41.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2024-03-19T02:51:41.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:40 smithi155 ceph-mon[35804]: pgmap v324: 129 pgs: 129 active+clean; 641 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 409 B/s wr, 6 op/s 2024-03-19T02:51:41.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:40 smithi155 ceph-mon[35804]: osdmap e94: 12 total, 12 up, 12 in 2024-03-19T02:51:41.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:40 smithi155 ceph-mon[35804]: osdmap e95: 12 total, 12 up, 12 in 2024-03-19T02:51:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-19T02:51:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-03-19T02:51:41.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-03-19T02:51:41.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-03-19T02:51:41.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 6]}]': finished 2024-03-19T02:51:41.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]': finished 2024-03-19T02:51:41.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-19T02:51:41.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-03-19T02:51:41.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]': finished 2024-03-19T02:51:41.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2024-03-19T02:51:41.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:40 smithi033 ceph-mon[31973]: pgmap v324: 129 pgs: 129 active+clean; 641 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 409 B/s wr, 6 op/s 2024-03-19T02:51:41.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:40 smithi033 ceph-mon[31973]: osdmap e94: 12 total, 12 up, 12 in 2024-03-19T02:51:41.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:40 smithi033 ceph-mon[31973]: osdmap e95: 12 total, 12 up, 12 in 2024-03-19T02:51:41.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:40 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-19T02:51:41.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:40 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-03-19T02:51:41.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:40 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-03-19T02:51:41.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:40 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-03-19T02:51:41.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:40 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 6]}]': finished 2024-03-19T02:51:41.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:40 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]': finished 2024-03-19T02:51:41.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:40 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-03-19T02:51:41.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:40 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-03-19T02:51:41.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:40 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]': finished 2024-03-19T02:51:41.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:40 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2024-03-19T02:51:41.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:40 smithi096 ceph-mon[35946]: pgmap v324: 129 pgs: 129 active+clean; 641 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 409 B/s wr, 6 op/s 2024-03-19T02:51:41.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:40 smithi096 ceph-mon[35946]: osdmap e94: 12 total, 12 up, 12 in 2024-03-19T02:51:41.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:40 smithi096 ceph-mon[35946]: osdmap e95: 12 total, 12 up, 12 in 2024-03-19T02:51:43.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:42 smithi155 ceph-mon[35804]: pgmap v326: 129 pgs: 5 peering, 124 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 7 op/s; 14 B/s, 0 objects/s recovering 2024-03-19T02:51:43.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:42 smithi033 ceph-mon[31973]: pgmap v326: 129 pgs: 5 peering, 124 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 7 op/s; 14 B/s, 0 objects/s recovering 2024-03-19T02:51:43.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:42 smithi096 ceph-mon[35946]: pgmap v326: 129 pgs: 5 peering, 124 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 7 op/s; 14 B/s, 0 objects/s recovering 2024-03-19T02:51:45.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:44 smithi155 ceph-mon[35804]: pgmap v327: 129 pgs: 5 peering, 124 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 511 B/s wr, 3 op/s; 28 B/s, 1 objects/s recovering 2024-03-19T02:51:45.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:44 smithi033 ceph-mon[31973]: pgmap v327: 129 pgs: 5 peering, 124 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 511 B/s wr, 3 op/s; 28 B/s, 1 objects/s recovering 2024-03-19T02:51:45.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:44 smithi096 ceph-mon[35946]: pgmap v327: 129 pgs: 5 peering, 124 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 511 B/s wr, 3 op/s; 28 B/s, 1 objects/s recovering 2024-03-19T02:51:46.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.242+0000 7f5916afd700 1 -- 172.21.15.33:0/3739901904 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5910106530 msgr2=0x7f591010e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:46.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.242+0000 7f5916afd700 1 --2- 172.21.15.33:0/3739901904 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5910106530 0x7f591010e100 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f590800b0a0 tx=0x7f590800b3b0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.242+0000 7f5916afd700 1 -- 172.21.15.33:0/3739901904 shutdown_connections 2024-03-19T02:51:46.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.242+0000 7f5916afd700 1 --2- 172.21.15.33:0/3739901904 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5910106530 0x7f591010e100 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.242+0000 7f5916afd700 1 --2- 172.21.15.33:0/3739901904 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f59101014a0 0x7f5910101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.242+0000 7f5916afd700 1 --2- 172.21.15.33:0/3739901904 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5910101e50 0x7f5910105e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.242+0000 7f5916afd700 1 -- 172.21.15.33:0/3739901904 >> 172.21.15.33:0/3739901904 conn(0x7f5910077230 msgr2=0x7f5910077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:46.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.242+0000 7f5916afd700 1 -- 172.21.15.33:0/3739901904 shutdown_connections 2024-03-19T02:51:46.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.242+0000 7f5916afd700 1 -- 172.21.15.33:0/3739901904 wait complete. 2024-03-19T02:51:46.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.243+0000 7f5916afd700 1 Processor -- start 2024-03-19T02:51:46.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.243+0000 7f5916afd700 1 -- start start 2024-03-19T02:51:46.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.243+0000 7f5916afd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59101014a0 0x7f5910102ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.243+0000 7f5916afd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5910101e50 0x7f5910103430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.243+0000 7f5916afd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5910106530 0x7f5910103970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.243+0000 7f5916afd700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f59101113c0 con 0x7f5910101e50 2024-03-19T02:51:46.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.243+0000 7f5916afd700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5910111240 con 0x7f5910106530 2024-03-19T02:51:46.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.243+0000 7f5916afd700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5910111540 con 0x7f59101014a0 2024-03-19T02:51:46.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.243+0000 7f590ffff700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5910101e50 0x7f5910103430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.243+0000 7f590ffff700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5910101e50 0x7f5910103430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47596/0 (socket says 172.21.15.33:47596) 2024-03-19T02:51:46.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.243+0000 7f590ffff700 1 -- 172.21.15.33:0/3518711201 learned_addr learned my addr 172.21.15.33:0/3518711201 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:46.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.243+0000 7f591509a700 1 --2- 172.21.15.33:0/3518711201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5910106530 0x7f5910103970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.243+0000 7f5914899700 1 --2- 172.21.15.33:0/3518711201 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59101014a0 0x7f5910102ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.243+0000 7f590ffff700 1 -- 172.21.15.33:0/3518711201 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59101014a0 msgr2=0x7f5910102ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:46.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.243+0000 7f590ffff700 1 --2- 172.21.15.33:0/3518711201 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59101014a0 0x7f5910102ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.243+0000 7f590ffff700 1 -- 172.21.15.33:0/3518711201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5910106530 msgr2=0x7f5910103970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:46.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.243+0000 7f590ffff700 1 --2- 172.21.15.33:0/3518711201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5910106530 0x7f5910103970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.244+0000 7f590ffff700 1 -- 172.21.15.33:0/3518711201 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59101a6ef0 con 0x7f5910101e50 2024-03-19T02:51:46.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.244+0000 7f590ffff700 1 --2- 172.21.15.33:0/3518711201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5910101e50 0x7f5910103430 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f590400e970 tx=0x7f590400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:46.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.244+0000 7f590dffb700 1 -- 172.21.15.33:0/3518711201 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5904004a30 con 0x7f5910101e50 2024-03-19T02:51:46.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.244+0000 7f590dffb700 1 -- 172.21.15.33:0/3518711201 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f590400cc90 con 0x7f5910101e50 2024-03-19T02:51:46.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.244+0000 7f5916afd700 1 -- 172.21.15.33:0/3518711201 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59101a71e0 con 0x7f5910101e50 2024-03-19T02:51:46.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.244+0000 7f590dffb700 1 -- 172.21.15.33:0/3518711201 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59040055e0 con 0x7f5910101e50 2024-03-19T02:51:46.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.244+0000 7f5916afd700 1 -- 172.21.15.33:0/3518711201 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59101a7720 con 0x7f5910101e50 2024-03-19T02:51:46.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.245+0000 7f5916afd700 1 -- 172.21.15.33:0/3518711201 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5910062a20 con 0x7f5910101e50 2024-03-19T02:51:46.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.246+0000 7f590dffb700 1 -- 172.21.15.33:0/3518711201 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f590400c7e0 con 0x7f5910101e50 2024-03-19T02:51:46.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.246+0000 7f590dffb700 1 --2- 172.21.15.33:0/3518711201 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f58f80a2920 0x7f58f80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.247+0000 7f5914899700 1 --2- 172.21.15.33:0/3518711201 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f58f80a2920 0x7f58f80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.247+0000 7f590dffb700 1 -- 172.21.15.33:0/3518711201 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7f59040cf4d0 con 0x7f5910101e50 2024-03-19T02:51:46.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.247+0000 7f590dffb700 1 --2- 172.21.15.33:0/3518711201 >> [v2:172.21.15.155:6816/3498516903,v1:172.21.15.155:6817/3498516903] conn(0x7f58f80ada70 0x7f58f80aff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.247+0000 7f590dffb700 1 -- 172.21.15.33:0/3518711201 --> [v2:172.21.15.155:6816/3498516903,v1:172.21.15.155:6817/3498516903] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f58f80b05e0 con 0x7f58f80ada70 2024-03-19T02:51:46.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.247+0000 7f590dffb700 1 -- 172.21.15.33:0/3518711201 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7f59040cf920 con 0x7f5910101e50 2024-03-19T02:51:46.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.247+0000 7f5914899700 1 --2- 172.21.15.33:0/3518711201 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f58f80a2920 0x7f58f80a4de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5900009c60 tx=0x7f5900009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:46.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.247+0000 7f591509a700 1 --2- 172.21.15.33:0/3518711201 >> [v2:172.21.15.155:6816/3498516903,v1:172.21.15.155:6817/3498516903] conn(0x7f58f80ada70 0x7f58f80aff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.247+0000 7f591509a700 1 --2- 172.21.15.33:0/3518711201 >> [v2:172.21.15.155:6816/3498516903,v1:172.21.15.155:6817/3498516903] conn(0x7f58f80ada70 0x7f58f80aff10 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:46.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.248+0000 7f590dffb700 1 -- 172.21.15.33:0/3518711201 <== osd.8 v2:172.21.15.155:6816/3498516903 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f58f80b05e0 con 0x7f58f80ada70 2024-03-19T02:51:46.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.264+0000 7f5916afd700 1 -- 172.21.15.33:0/3518711201 --> [v2:172.21.15.155:6816/3498516903,v1:172.21.15.155:6817/3498516903] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5910062ea0 con 0x7f58f80ada70 2024-03-19T02:51:46.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.265+0000 7f590dffb700 1 -- 172.21.15.33:0/3518711201 <== osd.8 v2:172.21.15.155:6816/3498516903 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5910062ea0 con 0x7f58f80ada70 2024-03-19T02:51:46.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.265+0000 7f5916afd700 1 -- 172.21.15.33:0/3518711201 >> [v2:172.21.15.155:6816/3498516903,v1:172.21.15.155:6817/3498516903] conn(0x7f58f80ada70 msgr2=0x7f58f80aff10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:46.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.265+0000 7f5916afd700 1 --2- 172.21.15.33:0/3518711201 >> [v2:172.21.15.155:6816/3498516903,v1:172.21.15.155:6817/3498516903] conn(0x7f58f80ada70 0x7f58f80aff10 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.265+0000 7f5916afd700 1 -- 172.21.15.33:0/3518711201 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f58f80a2920 msgr2=0x7f58f80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:46.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.265+0000 7f5916afd700 1 --2- 172.21.15.33:0/3518711201 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f58f80a2920 0x7f58f80a4de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5900009c60 tx=0x7f5900009260 comp rx=0 tx=0).stop 2024-03-19T02:51:46.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.265+0000 7f5916afd700 1 -- 172.21.15.33:0/3518711201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5910101e50 msgr2=0x7f5910103430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:46.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.265+0000 7f5916afd700 1 --2- 172.21.15.33:0/3518711201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5910101e50 0x7f5910103430 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f590400e970 tx=0x7f590400ec80 comp rx=0 tx=0).stop 2024-03-19T02:51:46.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.265+0000 7f591509a700 1 -- 172.21.15.33:0/3518711201 reap_dead start 2024-03-19T02:51:46.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.265+0000 7f5916afd700 1 -- 172.21.15.33:0/3518711201 shutdown_connections 2024-03-19T02:51:46.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.265+0000 7f5916afd700 1 -- 172.21.15.33:0/3518711201 >> 172.21.15.33:0/3518711201 conn(0x7f5910077230 msgr2=0x7f591006add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:46.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.265+0000 7f5916afd700 1 -- 172.21.15.33:0/3518711201 shutdown_connections 2024-03-19T02:51:46.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.265+0000 7f5916afd700 1 -- 172.21.15.33:0/3518711201 wait complete. 2024-03-19T02:51:46.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.136+0000 7fb0f4280700 1 -- 172.21.15.33:0/2822102161 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0ec1012c0 msgr2=0x7fb0ec10f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:46.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.136+0000 7fb0f4280700 1 --2- 172.21.15.33:0/2822102161 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0ec1012c0 0x7fb0ec10f220 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb0e400b0a0 tx=0x7fb0e400b3b0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.136+0000 7fb0f4280700 1 -- 172.21.15.33:0/2822102161 shutdown_connections 2024-03-19T02:51:46.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.136+0000 7fb0f4280700 1 --2- 172.21.15.33:0/2822102161 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0ec1012c0 0x7fb0ec10f220 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.136+0000 7fb0f4280700 1 --2- 172.21.15.33:0/2822102161 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0ec107380 0x7fb0ec107760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.136+0000 7fb0f4280700 1 --2- 172.21.15.33:0/2822102161 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0ec100900 0x7fb0ec100d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.136+0000 7fb0f4280700 1 -- 172.21.15.33:0/2822102161 >> 172.21.15.33:0/2822102161 conn(0x7fb0ec077810 msgr2=0x7fb0ec0fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:46.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.136+0000 7fb0f4280700 1 -- 172.21.15.33:0/2822102161 shutdown_connections 2024-03-19T02:51:46.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.136+0000 7fb0f4280700 1 -- 172.21.15.33:0/2822102161 wait complete. 2024-03-19T02:51:46.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.137+0000 7fb0f4280700 1 Processor -- start 2024-03-19T02:51:46.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.137+0000 7fb0f4280700 1 -- start start 2024-03-19T02:51:46.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.137+0000 7fb0f4280700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0ec100900 0x7fb0ec101ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.137+0000 7fb0f4280700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0ec1012c0 0x7fb0ec102020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.137+0000 7fb0f4280700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0ec107380 0x7fb0ec102560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.137+0000 7fb0f4280700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb0ec112610 con 0x7fb0ec100900 2024-03-19T02:51:46.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.137+0000 7fb0f4280700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb0ec112490 con 0x7fb0ec107380 2024-03-19T02:51:46.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.137+0000 7fb0f4280700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb0ec112790 con 0x7fb0ec1012c0 2024-03-19T02:51:46.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.137+0000 7fb0f201c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0ec100900 0x7fb0ec101ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.137+0000 7fb0f201c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0ec100900 0x7fb0ec101ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47568/0 (socket says 172.21.15.33:47568) 2024-03-19T02:51:46.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.137+0000 7fb0f201c700 1 -- 172.21.15.33:0/604542731 learned_addr learned my addr 172.21.15.33:0/604542731 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:46.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.137+0000 7fb0f281d700 1 --2- 172.21.15.33:0/604542731 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0ec107380 0x7fb0ec102560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.137+0000 7fb0f181b700 1 --2- 172.21.15.33:0/604542731 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0ec1012c0 0x7fb0ec102020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.138+0000 7fb0f201c700 1 -- 172.21.15.33:0/604542731 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0ec1012c0 msgr2=0x7fb0ec102020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:46.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.138+0000 7fb0f201c700 1 --2- 172.21.15.33:0/604542731 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0ec1012c0 0x7fb0ec102020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.138+0000 7fb0f201c700 1 -- 172.21.15.33:0/604542731 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0ec107380 msgr2=0x7fb0ec102560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:46.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.138+0000 7fb0f201c700 1 --2- 172.21.15.33:0/604542731 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb0ec107380 0x7fb0ec102560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.362 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.138+0000 7fb0f201c700 1 -- 172.21.15.33:0/604542731 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0ec1a6f90 con 0x7fb0ec100900 2024-03-19T02:51:46.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.138+0000 7fb0f201c700 1 --2- 172.21.15.33:0/604542731 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0ec100900 0x7fb0ec101ae0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb0e800eb30 tx=0x7fb0e800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:46.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.138+0000 7fb0e37fe700 1 -- 172.21.15.33:0/604542731 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e800cc90 con 0x7fb0ec100900 2024-03-19T02:51:46.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.138+0000 7fb0f4280700 1 -- 172.21.15.33:0/604542731 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0ec1a7280 con 0x7fb0ec100900 2024-03-19T02:51:46.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.138+0000 7fb0e37fe700 1 -- 172.21.15.33:0/604542731 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb0e80044d0 con 0x7fb0ec100900 2024-03-19T02:51:46.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.138+0000 7fb0e37fe700 1 -- 172.21.15.33:0/604542731 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e8010430 con 0x7fb0ec100900 2024-03-19T02:51:46.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.138+0000 7fb0f4280700 1 -- 172.21.15.33:0/604542731 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0ec1a77c0 con 0x7fb0ec100900 2024-03-19T02:51:46.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.139+0000 7fb0f4280700 1 -- 172.21.15.33:0/604542731 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb0ec062a20 con 0x7fb0ec100900 2024-03-19T02:51:46.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.140+0000 7fb0e37fe700 1 -- 172.21.15.33:0/604542731 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fb0e8003ea0 con 0x7fb0ec100900 2024-03-19T02:51:46.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.141+0000 7fb0e37fe700 1 --2- 172.21.15.33:0/604542731 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb0d80a2920 0x7fb0d80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.141+0000 7fb0f181b700 1 --2- 172.21.15.33:0/604542731 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb0d80a2920 0x7fb0d80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.141+0000 7fb0e37fe700 1 -- 172.21.15.33:0/604542731 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7fb0e80cf5b0 con 0x7fb0ec100900 2024-03-19T02:51:46.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.141+0000 7fb0e37fe700 1 --2- 172.21.15.33:0/604542731 >> [v2:172.21.15.33:6818/873871509,v1:172.21.15.33:6819/873871509] conn(0x7fb0d80ada80 0x7fb0d80aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:46.363 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.141+0000 7fb0e37fe700 1 -- 172.21.15.33:0/604542731 --> [v2:172.21.15.33:6818/873871509,v1:172.21.15.33:6819/873871509] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb0d80b05f0 con 0x7fb0d80ada80 2024-03-19T02:51:46.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.141+0000 7fb0e37fe700 1 -- 172.21.15.33:0/604542731 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7fb0e80cfa00 con 0x7fb0ec100900 2024-03-19T02:51:46.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.141+0000 7fb0f281d700 1 --2- 172.21.15.33:0/604542731 >> [v2:172.21.15.33:6818/873871509,v1:172.21.15.33:6819/873871509] conn(0x7fb0d80ada80 0x7fb0d80aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:46.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.141+0000 7fb0f181b700 1 --2- 172.21.15.33:0/604542731 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb0d80a2920 0x7fb0d80a4de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec0ffe00 tx=0x7fb0dc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:46.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.142+0000 7fb0f281d700 1 --2- 172.21.15.33:0/604542731 >> [v2:172.21.15.33:6818/873871509,v1:172.21.15.33:6819/873871509] conn(0x7fb0d80ada80 0x7fb0d80aff20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:46.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.142+0000 7fb0e37fe700 1 -- 172.21.15.33:0/604542731 <== osd.6 v2:172.21.15.33:6818/873871509 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb0d80b05f0 con 0x7fb0d80ada80 2024-03-19T02:51:46.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.159+0000 7fb0f4280700 1 -- 172.21.15.33:0/604542731 --> [v2:172.21.15.33:6818/873871509,v1:172.21.15.33:6819/873871509] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb0ec062e20 con 0x7fb0d80ada80 2024-03-19T02:51:46.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.159+0000 7fb0e37fe700 1 -- 172.21.15.33:0/604542731 <== osd.6 v2:172.21.15.33:6818/873871509 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb0ec062e20 con 0x7fb0d80ada80 2024-03-19T02:51:46.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.160+0000 7fb0f4280700 1 -- 172.21.15.33:0/604542731 >> [v2:172.21.15.33:6818/873871509,v1:172.21.15.33:6819/873871509] conn(0x7fb0d80ada80 msgr2=0x7fb0d80aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:46.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.160+0000 7fb0f4280700 1 --2- 172.21.15.33:0/604542731 >> [v2:172.21.15.33:6818/873871509,v1:172.21.15.33:6819/873871509] conn(0x7fb0d80ada80 0x7fb0d80aff20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:46.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.160+0000 7fb0f4280700 1 -- 172.21.15.33:0/604542731 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb0d80a2920 msgr2=0x7fb0d80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:46.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.160+0000 7fb0f4280700 1 --2- 172.21.15.33:0/604542731 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb0d80a2920 0x7fb0d80a4de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec0ffe00 tx=0x7fb0dc009260 comp rx=0 tx=0).stop 2024-03-19T02:51:46.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.160+0000 7fb0f4280700 1 -- 172.21.15.33:0/604542731 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0ec100900 msgr2=0x7fb0ec101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:46.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.160+0000 7fb0f4280700 1 --2- 172.21.15.33:0/604542731 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb0ec100900 0x7fb0ec101ae0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb0e800eb30 tx=0x7fb0e800ee40 comp rx=0 tx=0).stop 2024-03-19T02:51:46.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.160+0000 7fb0f281d700 1 -- 172.21.15.33:0/604542731 reap_dead start 2024-03-19T02:51:46.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.160+0000 7fb0f4280700 1 -- 172.21.15.33:0/604542731 shutdown_connections 2024-03-19T02:51:46.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.160+0000 7fb0f4280700 1 -- 172.21.15.33:0/604542731 >> 172.21.15.33:0/604542731 conn(0x7fb0ec077810 msgr2=0x7fb0ec105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:46.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.160+0000 7fb0f4280700 1 -- 172.21.15.33:0/604542731 shutdown_connections 2024-03-19T02:51:46.366 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:46.160+0000 7fb0f4280700 1 -- 172.21.15.33:0/604542731 wait complete. 2024-03-19T02:51:46.909 INFO:teuthology.orchestra.run.smithi033.stdout:201863462961 2024-03-19T02:51:46.909 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-19T02:51:47.006 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:46 smithi033 ceph-mon[31973]: pgmap v328: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 511 B/s wr, 7 op/s; 101 B/s, 1 objects/s recovering 2024-03-19T02:51:47.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:46 smithi155 ceph-mon[35804]: pgmap v328: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 511 B/s wr, 7 op/s; 101 B/s, 1 objects/s recovering 2024-03-19T02:51:47.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:46 smithi096 ceph-mon[35946]: pgmap v328: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 511 B/s wr, 7 op/s; 101 B/s, 1 objects/s recovering 2024-03-19T02:51:47.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.770+0000 7f2355168700 1 -- 172.21.15.33:0/395639999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2350100900 msgr2=0x7f2350100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:47.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.770+0000 7f2355168700 1 --2- 172.21.15.33:0/395639999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2350100900 0x7f2350100d80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2344009990 tx=0x7f2344009ca0 comp rx=0 tx=0).stop 2024-03-19T02:51:47.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.770+0000 7f2355168700 1 -- 172.21.15.33:0/395639999 shutdown_connections 2024-03-19T02:51:47.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.770+0000 7f2355168700 1 --2- 172.21.15.33:0/395639999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f23501012c0 0x7f235010f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:47.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.770+0000 7f2355168700 1 --2- 172.21.15.33:0/395639999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2350107380 0x7f2350107760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:47.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.770+0000 7f2355168700 1 --2- 172.21.15.33:0/395639999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2350100900 0x7f2350100d80 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:47.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.770+0000 7f2355168700 1 -- 172.21.15.33:0/395639999 >> 172.21.15.33:0/395639999 conn(0x7f2350077810 msgr2=0x7f23500fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:47.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.771+0000 7f2355168700 1 -- 172.21.15.33:0/395639999 shutdown_connections 2024-03-19T02:51:47.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.771+0000 7f2355168700 1 -- 172.21.15.33:0/395639999 wait complete. 2024-03-19T02:51:47.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.771+0000 7f2355168700 1 Processor -- start 2024-03-19T02:51:47.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.771+0000 7f2355168700 1 -- start start 2024-03-19T02:51:47.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.772+0000 7f2355168700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2350100900 0x7f235019e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:47.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.772+0000 7f2355168700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f23501012c0 0x7f235019ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:47.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.772+0000 7f2355168700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2350107380 0x7f23501989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:47.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.772+0000 7f2355168700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2350112610 con 0x7f2350107380 2024-03-19T02:51:47.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.772+0000 7f2355168700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2350112490 con 0x7f2350100900 2024-03-19T02:51:47.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.772+0000 7f2355168700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2350112790 con 0x7f23501012c0 2024-03-19T02:51:47.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.772+0000 7f234f59e700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2350107380 0x7f23501989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:47.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.772+0000 7f234ed9d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2350100900 0x7f235019e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:47.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.772+0000 7f234f59e700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2350107380 0x7f23501989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47614/0 (socket says 172.21.15.33:47614) 2024-03-19T02:51:47.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.772+0000 7f234f59e700 1 -- 172.21.15.33:0/824645943 learned_addr learned my addr 172.21.15.33:0/824645943 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:47.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.772+0000 7f234e59c700 1 --2- 172.21.15.33:0/824645943 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f23501012c0 0x7f235019ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:47.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.772+0000 7f234f59e700 1 -- 172.21.15.33:0/824645943 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f23501012c0 msgr2=0x7f235019ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:47.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.772+0000 7f234f59e700 1 --2- 172.21.15.33:0/824645943 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f23501012c0 0x7f235019ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:47.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.772+0000 7f234f59e700 1 -- 172.21.15.33:0/824645943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2350100900 msgr2=0x7f235019e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:47.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.772+0000 7f234f59e700 1 --2- 172.21.15.33:0/824645943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2350100900 0x7f235019e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:47.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.772+0000 7f234f59e700 1 -- 172.21.15.33:0/824645943 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2350199110 con 0x7f2350107380 2024-03-19T02:51:47.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.772+0000 7f234ed9d700 1 --2- 172.21.15.33:0/824645943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2350100900 0x7f235019e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:47.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.772+0000 7f234f59e700 1 --2- 172.21.15.33:0/824645943 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2350107380 0x7f23501989b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f23480094f0 tx=0x7f234800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:47.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.772+0000 7f2337fff700 1 -- 172.21.15.33:0/824645943 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2348004db0 con 0x7f2350107380 2024-03-19T02:51:47.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.773+0000 7f2355168700 1 -- 172.21.15.33:0/824645943 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2350199400 con 0x7f2350107380 2024-03-19T02:51:47.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.773+0000 7f2337fff700 1 -- 172.21.15.33:0/824645943 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2348009e20 con 0x7f2350107380 2024-03-19T02:51:47.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.773+0000 7f2337fff700 1 -- 172.21.15.33:0/824645943 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2348015400 con 0x7f2350107380 2024-03-19T02:51:47.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.773+0000 7f2355168700 1 -- 172.21.15.33:0/824645943 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23501a56c0 con 0x7f2350107380 2024-03-19T02:51:47.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.773+0000 7f2355168700 1 -- 172.21.15.33:0/824645943 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2350062a20 con 0x7f2350107380 2024-03-19T02:51:47.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.775+0000 7f2337fff700 1 -- 172.21.15.33:0/824645943 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f23480052d0 con 0x7f2350107380 2024-03-19T02:51:47.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.775+0000 7f2337fff700 1 --2- 172.21.15.33:0/824645943 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f23380a2920 0x7f23380a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:47.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.775+0000 7f234ed9d700 1 --2- 172.21.15.33:0/824645943 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f23380a2920 0x7f23380a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:47.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.775+0000 7f2337fff700 1 -- 172.21.15.33:0/824645943 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7f23480cf4a0 con 0x7f2350107380 2024-03-19T02:51:47.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.775+0000 7f2337fff700 1 --2- 172.21.15.33:0/824645943 >> [v2:172.21.15.155:6808/327293324,v1:172.21.15.155:6809/327293324] conn(0x7f23380ada80 0x7f23380aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:47.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.775+0000 7f2337fff700 1 -- 172.21.15.33:0/824645943 --> [v2:172.21.15.155:6808/327293324,v1:172.21.15.155:6809/327293324] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2348010ec0 con 0x7f23380ada80 2024-03-19T02:51:47.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.776+0000 7f2337fff700 1 -- 172.21.15.33:0/824645943 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7f23480cf8f0 con 0x7f2350107380 2024-03-19T02:51:47.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.776+0000 7f234e59c700 1 --2- 172.21.15.33:0/824645943 >> [v2:172.21.15.155:6808/327293324,v1:172.21.15.155:6809/327293324] conn(0x7f23380ada80 0x7f23380aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:47.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.776+0000 7f234ed9d700 1 --2- 172.21.15.33:0/824645943 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f23380a2920 0x7f23380a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2350199af0 tx=0x7f23400073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:47.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.776+0000 7f234e59c700 1 --2- 172.21.15.33:0/824645943 >> [v2:172.21.15.155:6808/327293324,v1:172.21.15.155:6809/327293324] conn(0x7f23380ada80 0x7f23380aff20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:47.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.777+0000 7f2337fff700 1 -- 172.21.15.33:0/824645943 <== osd.5 v2:172.21.15.155:6808/327293324 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2348010ec0 con 0x7f23380ada80 2024-03-19T02:51:47.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.794+0000 7f2355168700 1 -- 172.21.15.33:0/824645943 --> [v2:172.21.15.155:6808/327293324,v1:172.21.15.155:6809/327293324] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2350062e20 con 0x7f23380ada80 2024-03-19T02:51:47.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.794+0000 7f2337fff700 1 -- 172.21.15.33:0/824645943 <== osd.5 v2:172.21.15.155:6808/327293324 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2350062e20 con 0x7f23380ada80 2024-03-19T02:51:47.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.795+0000 7f2355168700 1 -- 172.21.15.33:0/824645943 >> [v2:172.21.15.155:6808/327293324,v1:172.21.15.155:6809/327293324] conn(0x7f23380ada80 msgr2=0x7f23380aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:47.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.795+0000 7f2355168700 1 --2- 172.21.15.33:0/824645943 >> [v2:172.21.15.155:6808/327293324,v1:172.21.15.155:6809/327293324] conn(0x7f23380ada80 0x7f23380aff20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:47.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.795+0000 7f2355168700 1 -- 172.21.15.33:0/824645943 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f23380a2920 msgr2=0x7f23380a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:47.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.795+0000 7f2355168700 1 --2- 172.21.15.33:0/824645943 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f23380a2920 0x7f23380a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2350199af0 tx=0x7f23400073d0 comp rx=0 tx=0).stop 2024-03-19T02:51:47.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.795+0000 7f2355168700 1 -- 172.21.15.33:0/824645943 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2350107380 msgr2=0x7f23501989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:47.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.795+0000 7f2355168700 1 --2- 172.21.15.33:0/824645943 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2350107380 0x7f23501989b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f23480094f0 tx=0x7f234800d3b0 comp rx=0 tx=0).stop 2024-03-19T02:51:47.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.795+0000 7f234f59e700 1 -- 172.21.15.33:0/824645943 reap_dead start 2024-03-19T02:51:47.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.795+0000 7f2355168700 1 -- 172.21.15.33:0/824645943 shutdown_connections 2024-03-19T02:51:47.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.795+0000 7f2355168700 1 -- 172.21.15.33:0/824645943 >> 172.21.15.33:0/824645943 conn(0x7f2350077810 msgr2=0x7f23501053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:47.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.795+0000 7f2355168700 1 -- 172.21.15.33:0/824645943 shutdown_connections 2024-03-19T02:51:47.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:47.795+0000 7f2355168700 1 -- 172.21.15.33:0/824645943 wait complete. 2024-03-19T02:51:48.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:48 smithi033 ceph-mon[31973]: pgmap v329: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 101 B/s, 1 objects/s recovering 2024-03-19T02:51:49.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:48 smithi155 ceph-mon[35804]: pgmap v329: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 101 B/s, 1 objects/s recovering 2024-03-19T02:51:49.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.024+0000 7f173d1ad700 1 -- 172.21.15.33:0/1746805692 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1738108340 msgr2=0x7f1738108720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:49.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.024+0000 7f173d1ad700 1 --2- 172.21.15.33:0/1746805692 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1738108340 0x7f1738108720 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f17280099c0 tx=0x7f1728009cd0 comp rx=0 tx=0).stop 2024-03-19T02:51:49.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.024+0000 7f173d1ad700 1 -- 172.21.15.33:0/1746805692 shutdown_connections 2024-03-19T02:51:49.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.024+0000 7f173d1ad700 1 --2- 172.21.15.33:0/1746805692 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1738102d20 0x7f173810f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:49.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.024+0000 7f173d1ad700 1 --2- 172.21.15.33:0/1746805692 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1738108340 0x7f1738108720 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:49.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.024+0000 7f173d1ad700 1 --2- 172.21.15.33:0/1746805692 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1738102360 0x7f17381027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:49.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.024+0000 7f173d1ad700 1 -- 172.21.15.33:0/1746805692 >> 172.21.15.33:0/1746805692 conn(0x7f1738079440 msgr2=0x7f1738079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:49.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.024+0000 7f173d1ad700 1 -- 172.21.15.33:0/1746805692 shutdown_connections 2024-03-19T02:51:49.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.024+0000 7f173d1ad700 1 -- 172.21.15.33:0/1746805692 wait complete. 2024-03-19T02:51:49.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.025+0000 7f173d1ad700 1 Processor -- start 2024-03-19T02:51:49.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.026+0000 7f173d1ad700 1 -- start start 2024-03-19T02:51:49.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.026+0000 7f173d1ad700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1738102360 0x7f1738198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:49.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.026+0000 7f173d1ad700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1738102d20 0x7f1738198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:49.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.026+0000 7f173d1ad700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1738108340 0x7f173819d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:49.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.026+0000 7f173d1ad700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f17381124a0 con 0x7f1738102d20 2024-03-19T02:51:49.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.026+0000 7f173d1ad700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1738112320 con 0x7f1738108340 2024-03-19T02:51:49.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.026+0000 7f173d1ad700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1738112620 con 0x7f1738102360 2024-03-19T02:51:49.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.026+0000 7f173659c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1738102d20 0x7f1738198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:49.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.026+0000 7f173659c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1738102d20 0x7f1738198c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47634/0 (socket says 172.21.15.33:47634) 2024-03-19T02:51:49.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.026+0000 7f173659c700 1 -- 172.21.15.33:0/867839885 learned_addr learned my addr 172.21.15.33:0/867839885 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:49.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.026+0000 7f173759e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1738108340 0x7f173819d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:49.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.026+0000 7f1736d9d700 1 --2- 172.21.15.33:0/867839885 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1738102360 0x7f1738198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:49.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.026+0000 7f173759e700 1 -- 172.21.15.33:0/867839885 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1738102360 msgr2=0x7f1738198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:49.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.026+0000 7f173759e700 1 --2- 172.21.15.33:0/867839885 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1738102360 0x7f1738198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:49.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.026+0000 7f173759e700 1 -- 172.21.15.33:0/867839885 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1738102d20 msgr2=0x7f1738198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:49.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.026+0000 7f173759e700 1 --2- 172.21.15.33:0/867839885 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1738102d20 0x7f1738198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:49.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.026+0000 7f173759e700 1 -- 172.21.15.33:0/867839885 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f173819d760 con 0x7f1738108340 2024-03-19T02:51:49.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.026+0000 7f1736d9d700 1 --2- 172.21.15.33:0/867839885 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1738102360 0x7f1738198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:49.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.026+0000 7f173659c700 1 --2- 172.21.15.33:0/867839885 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1738102d20 0x7f1738198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:49.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.027+0000 7f173759e700 1 --2- 172.21.15.33:0/867839885 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1738108340 0x7f173819d020 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f173000efc0 tx=0x7f173000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:49.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.027+0000 7f171ffff700 1 -- 172.21.15.33:0/867839885 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17300092e0 con 0x7f1738108340 2024-03-19T02:51:49.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.027+0000 7f171ffff700 1 -- 172.21.15.33:0/867839885 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1730004bb0 con 0x7f1738108340 2024-03-19T02:51:49.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.027+0000 7f171ffff700 1 -- 172.21.15.33:0/867839885 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1730004020 con 0x7f1738108340 2024-03-19T02:51:49.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.027+0000 7f173d1ad700 1 -- 172.21.15.33:0/867839885 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f173819d9f0 con 0x7f1738108340 2024-03-19T02:51:49.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.027+0000 7f173d1ad700 1 -- 172.21.15.33:0/867839885 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f173819de20 con 0x7f1738108340 2024-03-19T02:51:49.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.029+0000 7f173d1ad700 1 -- 172.21.15.33:0/867839885 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1724001030 con 0x7f1738108340 2024-03-19T02:51:49.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.030+0000 7f171ffff700 1 -- 172.21.15.33:0/867839885 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f173001b030 con 0x7f1738108340 2024-03-19T02:51:49.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.031+0000 7f171ffff700 1 --2- 172.21.15.33:0/867839885 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f17200a29f0 0x7f17200a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:49.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.031+0000 7f1736d9d700 1 --2- 172.21.15.33:0/867839885 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f17200a29f0 0x7f17200a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:49.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.031+0000 7f171ffff700 1 -- 172.21.15.33:0/867839885 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7f17300ce8a0 con 0x7f1738108340 2024-03-19T02:51:49.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.031+0000 7f171ffff700 1 --2- 172.21.15.33:0/867839885 >> [v2:172.21.15.33:6810/677007171,v1:172.21.15.33:6811/677007171] conn(0x7f17200adb50 0x7f17200afff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:49.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.032+0000 7f171ffff700 1 -- 172.21.15.33:0/867839885 --> [v2:172.21.15.33:6810/677007171,v1:172.21.15.33:6811/677007171] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1730020e50 con 0x7f17200adb50 2024-03-19T02:51:49.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.032+0000 7f171ffff700 1 -- 172.21.15.33:0/867839885 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7f17300d32c0 con 0x7f1738108340 2024-03-19T02:51:49.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.032+0000 7f173659c700 1 --2- 172.21.15.33:0/867839885 >> [v2:172.21.15.33:6810/677007171,v1:172.21.15.33:6811/677007171] conn(0x7f17200adb50 0x7f17200afff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:49.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.032+0000 7f1736d9d700 1 --2- 172.21.15.33:0/867839885 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f17200a29f0 0x7f17200a4eb0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1728009350 tx=0x7f1728000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:49.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.032+0000 7f173659c700 1 --2- 172.21.15.33:0/867839885 >> [v2:172.21.15.33:6810/677007171,v1:172.21.15.33:6811/677007171] conn(0x7f17200adb50 0x7f17200afff0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:49.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.033+0000 7f171ffff700 1 -- 172.21.15.33:0/867839885 <== osd.3 v2:172.21.15.33:6810/677007171 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1730020e50 con 0x7f17200adb50 2024-03-19T02:51:49.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.056+0000 7f173d1ad700 1 -- 172.21.15.33:0/867839885 --> [v2:172.21.15.33:6810/677007171,v1:172.21.15.33:6811/677007171] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1724002e60 con 0x7f17200adb50 2024-03-19T02:51:49.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.057+0000 7f171ffff700 1 -- 172.21.15.33:0/867839885 <== osd.3 v2:172.21.15.33:6810/677007171 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1724002e60 con 0x7f17200adb50 2024-03-19T02:51:49.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.057+0000 7f173d1ad700 1 -- 172.21.15.33:0/867839885 >> [v2:172.21.15.33:6810/677007171,v1:172.21.15.33:6811/677007171] conn(0x7f17200adb50 msgr2=0x7f17200afff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:49.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.057+0000 7f173d1ad700 1 --2- 172.21.15.33:0/867839885 >> [v2:172.21.15.33:6810/677007171,v1:172.21.15.33:6811/677007171] conn(0x7f17200adb50 0x7f17200afff0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:49.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.057+0000 7f173d1ad700 1 -- 172.21.15.33:0/867839885 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f17200a29f0 msgr2=0x7f17200a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:49.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.057+0000 7f173d1ad700 1 --2- 172.21.15.33:0/867839885 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f17200a29f0 0x7f17200a4eb0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1728009350 tx=0x7f1728000bc0 comp rx=0 tx=0).stop 2024-03-19T02:51:49.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.057+0000 7f173d1ad700 1 -- 172.21.15.33:0/867839885 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1738108340 msgr2=0x7f173819d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:49.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.057+0000 7f173d1ad700 1 --2- 172.21.15.33:0/867839885 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1738108340 0x7f173819d020 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f173000efc0 tx=0x7f173000c3a0 comp rx=0 tx=0).stop 2024-03-19T02:51:49.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.057+0000 7f173759e700 1 -- 172.21.15.33:0/867839885 reap_dead start 2024-03-19T02:51:49.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.057+0000 7f173d1ad700 1 -- 172.21.15.33:0/867839885 shutdown_connections 2024-03-19T02:51:49.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.057+0000 7f173d1ad700 1 -- 172.21.15.33:0/867839885 >> 172.21.15.33:0/867839885 conn(0x7f1738079440 msgr2=0x7f1738100cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:49.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.058+0000 7f173d1ad700 1 -- 172.21.15.33:0/867839885 shutdown_connections 2024-03-19T02:51:49.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.058+0000 7f173d1ad700 1 -- 172.21.15.33:0/867839885 wait complete. 2024-03-19T02:51:49.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:48 smithi096 ceph-mon[35946]: pgmap v329: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 101 B/s, 1 objects/s recovering 2024-03-19T02:51:50.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.545+0000 7f4b1a274700 1 -- 172.21.15.33:0/535492287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b141011f0 msgr2=0x7f4b14101670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:50.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.545+0000 7f4b1a274700 1 --2- 172.21.15.33:0/535492287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b141011f0 0x7f4b14101670 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4b08009990 tx=0x7f4b08009ca0 comp rx=0 tx=0).stop 2024-03-19T02:51:50.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.545+0000 7f4b1a274700 1 -- 172.21.15.33:0/535492287 shutdown_connections 2024-03-19T02:51:50.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.545+0000 7f4b1a274700 1 --2- 172.21.15.33:0/535492287 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b14101bb0 0x7f4b1410e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:50.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.545+0000 7f4b1a274700 1 --2- 172.21.15.33:0/535492287 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b141072a0 0x7f4b14107680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:50.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.545+0000 7f4b1a274700 1 --2- 172.21.15.33:0/535492287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b141011f0 0x7f4b14101670 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:50.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.545+0000 7f4b1a274700 1 -- 172.21.15.33:0/535492287 >> 172.21.15.33:0/535492287 conn(0x7f4b14078220 msgr2=0x7f4b14078630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:50.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.545+0000 7f4b1a274700 1 -- 172.21.15.33:0/535492287 shutdown_connections 2024-03-19T02:51:50.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.545+0000 7f4b1a274700 1 -- 172.21.15.33:0/535492287 wait complete. 2024-03-19T02:51:50.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.546+0000 7f4b1a274700 1 Processor -- start 2024-03-19T02:51:50.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.546+0000 7f4b1a274700 1 -- start start 2024-03-19T02:51:50.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.546+0000 7f4b1a274700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b141011f0 0x7f4b141997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:50.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.546+0000 7f4b1a274700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b14101bb0 0x7f4b14199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:50.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.546+0000 7f4b1a274700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b141072a0 0x7f4b141a4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:50.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.546+0000 7f4b1a274700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4b14111390 con 0x7f4b14101bb0 2024-03-19T02:51:50.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.546+0000 7f4b1a274700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4b14111210 con 0x7f4b141072a0 2024-03-19T02:51:50.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.546+0000 7f4b1a274700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4b14111510 con 0x7f4b141011f0 2024-03-19T02:51:50.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.546+0000 7f4b137fe700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b14101bb0 0x7f4b14199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:50.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.546+0000 7f4b13fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b141011f0 0x7f4b141997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:50.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.546+0000 7f4b18811700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b141072a0 0x7f4b141a4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:50.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.546+0000 7f4b13fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b141011f0 0x7f4b141997b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:59118/0 (socket says 172.21.15.33:59118) 2024-03-19T02:51:50.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.546+0000 7f4b13fff700 1 -- 172.21.15.33:0/1019067693 learned_addr learned my addr 172.21.15.33:0/1019067693 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:50.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.546+0000 7f4b13fff700 1 -- 172.21.15.33:0/1019067693 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b141072a0 msgr2=0x7f4b141a4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:50.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.546+0000 7f4b13fff700 1 --2- 172.21.15.33:0/1019067693 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4b141072a0 0x7f4b141a4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:50.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.546+0000 7f4b13fff700 1 -- 172.21.15.33:0/1019067693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b14101bb0 msgr2=0x7f4b14199cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:50.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.546+0000 7f4b13fff700 1 --2- 172.21.15.33:0/1019067693 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4b14101bb0 0x7f4b14199cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:50.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.546+0000 7f4b13fff700 1 -- 172.21.15.33:0/1019067693 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b141a4d70 con 0x7f4b141011f0 2024-03-19T02:51:50.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.546+0000 7f4b13fff700 1 --2- 172.21.15.33:0/1019067693 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b141011f0 0x7f4b141997b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4b0400e9a0 tx=0x7f4b0400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:50.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.546+0000 7f4b117fa700 1 -- 172.21.15.33:0/1019067693 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b0400ca10 con 0x7f4b141011f0 2024-03-19T02:51:50.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.546+0000 7f4b117fa700 1 -- 172.21.15.33:0/1019067693 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b0400cbb0 con 0x7f4b141011f0 2024-03-19T02:51:50.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.546+0000 7f4b1a274700 1 -- 172.21.15.33:0/1019067693 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b141a4fa0 con 0x7f4b141011f0 2024-03-19T02:51:50.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.546+0000 7f4b117fa700 1 -- 172.21.15.33:0/1019067693 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b04018680 con 0x7f4b141011f0 2024-03-19T02:51:50.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.546+0000 7f4b1a274700 1 -- 172.21.15.33:0/1019067693 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b141a53a0 con 0x7f4b141011f0 2024-03-19T02:51:50.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.548+0000 7f4b1a274700 1 -- 172.21.15.33:0/1019067693 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4b14062a20 con 0x7f4b141011f0 2024-03-19T02:51:50.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.549+0000 7f4b117fa700 1 -- 172.21.15.33:0/1019067693 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f4b04010430 con 0x7f4b141011f0 2024-03-19T02:51:50.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.550+0000 7f4b117fa700 1 --2- 172.21.15.33:0/1019067693 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4afc0a2920 0x7f4afc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:50.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.550+0000 7f4b137fe700 1 --2- 172.21.15.33:0/1019067693 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4afc0a2920 0x7f4afc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:50.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.550+0000 7f4b117fa700 1 -- 172.21.15.33:0/1019067693 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7f4b040cf530 con 0x7f4b141011f0 2024-03-19T02:51:50.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.550+0000 7f4b117fa700 1 --2- 172.21.15.33:0/1019067693 >> [v2:172.21.15.96:6816/2334123891,v1:172.21.15.96:6817/2334123891] conn(0x7f4afc0ada80 0x7f4afc0aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:50.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.550+0000 7f4b117fa700 1 -- 172.21.15.33:0/1019067693 --> [v2:172.21.15.96:6816/2334123891,v1:172.21.15.96:6817/2334123891] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4b04016ec0 con 0x7f4afc0ada80 2024-03-19T02:51:50.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.550+0000 7f4b117fa700 1 -- 172.21.15.33:0/1019067693 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7f4b040cf8e0 con 0x7f4b141011f0 2024-03-19T02:51:50.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.550+0000 7f4b137fe700 1 --2- 172.21.15.33:0/1019067693 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4afc0a2920 0x7f4afc0a4de0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4b080098e0 tx=0x7f4b08005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:50.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.550+0000 7f4b18811700 1 --2- 172.21.15.33:0/1019067693 >> [v2:172.21.15.96:6816/2334123891,v1:172.21.15.96:6817/2334123891] conn(0x7f4afc0ada80 0x7f4afc0aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:50.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.550+0000 7f4b18811700 1 --2- 172.21.15.33:0/1019067693 >> [v2:172.21.15.96:6816/2334123891,v1:172.21.15.96:6817/2334123891] conn(0x7f4afc0ada80 0x7f4afc0aff20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:50.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.550+0000 7f4b117fa700 1 -- 172.21.15.33:0/1019067693 <== osd.7 v2:172.21.15.96:6816/2334123891 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4b04016ec0 con 0x7f4afc0ada80 2024-03-19T02:51:50.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.566+0000 7f4b1a274700 1 -- 172.21.15.33:0/1019067693 --> [v2:172.21.15.96:6816/2334123891,v1:172.21.15.96:6817/2334123891] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4b14062ea0 con 0x7f4afc0ada80 2024-03-19T02:51:50.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.568+0000 7f4b117fa700 1 -- 172.21.15.33:0/1019067693 <== osd.7 v2:172.21.15.96:6816/2334123891 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4b14062ea0 con 0x7f4afc0ada80 2024-03-19T02:51:50.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.568+0000 7f4b1a274700 1 -- 172.21.15.33:0/1019067693 >> [v2:172.21.15.96:6816/2334123891,v1:172.21.15.96:6817/2334123891] conn(0x7f4afc0ada80 msgr2=0x7f4afc0aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:50.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.568+0000 7f4b1a274700 1 --2- 172.21.15.33:0/1019067693 >> [v2:172.21.15.96:6816/2334123891,v1:172.21.15.96:6817/2334123891] conn(0x7f4afc0ada80 0x7f4afc0aff20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:50.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.568+0000 7f4b1a274700 1 -- 172.21.15.33:0/1019067693 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4afc0a2920 msgr2=0x7f4afc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:50.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.568+0000 7f4b1a274700 1 --2- 172.21.15.33:0/1019067693 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4afc0a2920 0x7f4afc0a4de0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4b080098e0 tx=0x7f4b08005960 comp rx=0 tx=0).stop 2024-03-19T02:51:50.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.568+0000 7f4b1a274700 1 -- 172.21.15.33:0/1019067693 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b141011f0 msgr2=0x7f4b141997b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:50.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.568+0000 7f4b1a274700 1 --2- 172.21.15.33:0/1019067693 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4b141011f0 0x7f4b141997b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4b0400e9a0 tx=0x7f4b0400ecb0 comp rx=0 tx=0).stop 2024-03-19T02:51:50.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.568+0000 7f4b18811700 1 -- 172.21.15.33:0/1019067693 reap_dead start 2024-03-19T02:51:50.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.568+0000 7f4b1a274700 1 -- 172.21.15.33:0/1019067693 shutdown_connections 2024-03-19T02:51:50.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.568+0000 7f4b1a274700 1 -- 172.21.15.33:0/1019067693 >> 172.21.15.33:0/1019067693 conn(0x7f4b14078220 msgr2=0x7f4b14100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:50.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.568+0000 7f4b1a274700 1 -- 172.21.15.33:0/1019067693 shutdown_connections 2024-03-19T02:51:50.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:49.568+0000 7f4b1a274700 1 -- 172.21.15.33:0/1019067693 wait complete. 2024-03-19T02:51:51.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:50 smithi155 ceph-mon[35804]: pgmap v330: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 307 B/s wr, 6 op/s; 81 B/s, 1 objects/s recovering 2024-03-19T02:51:51.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:50 smithi033 ceph-mon[31973]: pgmap v330: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 307 B/s wr, 6 op/s; 81 B/s, 1 objects/s recovering 2024-03-19T02:51:51.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:50 smithi096 ceph-mon[35946]: pgmap v330: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 307 B/s wr, 6 op/s; 81 B/s, 1 objects/s recovering 2024-03-19T02:51:51.448 INFO:teuthology.orchestra.run.smithi033.stdout:167503724600 2024-03-19T02:51:51.448 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-19T02:51:52.307 INFO:teuthology.orchestra.run.smithi033.stdout:111669149762 2024-03-19T02:51:52.308 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-19T02:51:52.994 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:52 smithi033 ceph-mon[31973]: pgmap v331: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 279 B/s wr, 5 op/s; 73 B/s, 1 objects/s recovering 2024-03-19T02:51:52.995 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:52 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:51:53.016 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:52 smithi096 ceph-mon[35946]: pgmap v331: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 279 B/s wr, 5 op/s; 73 B/s, 1 objects/s recovering 2024-03-19T02:51:53.016 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:52 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:51:53.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:52 smithi155 ceph-mon[35804]: pgmap v331: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 279 B/s wr, 5 op/s; 73 B/s, 1 objects/s recovering 2024-03-19T02:51:53.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:52 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:51:53.096 INFO:teuthology.orchestra.run.smithi033.stdout:261993005095 2024-03-19T02:51:53.096 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-19T02:51:53.489 INFO:teuthology.orchestra.run.smithi033.stdout:227633266734 2024-03-19T02:51:53.489 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-19T02:51:53.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:53 smithi096 ceph-mon[35946]: pgmap v332: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s; 57 B/s, 0 objects/s recovering 2024-03-19T02:51:53.981 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:53 smithi033 ceph-mon[31973]: pgmap v332: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s; 57 B/s, 0 objects/s recovering 2024-03-19T02:51:54.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:53 smithi155 ceph-mon[35804]: pgmap v332: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s; 57 B/s, 0 objects/s recovering 2024-03-19T02:51:54.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.297+0000 7f7b46c7d700 1 -- 172.21.15.33:0/1878697325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b401014a0 msgr2=0x7f7b40101880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:54.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.297+0000 7f7b46c7d700 1 --2- 172.21.15.33:0/1878697325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b401014a0 0x7f7b40101880 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7b340099c0 tx=0x7f7b34009cd0 comp rx=0 tx=0).stop 2024-03-19T02:51:54.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.297+0000 7f7b46c7d700 1 -- 172.21.15.33:0/1878697325 shutdown_connections 2024-03-19T02:51:54.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.297+0000 7f7b46c7d700 1 --2- 172.21.15.33:0/1878697325 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b401064f0 0x7f7b4010e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:54.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.297+0000 7f7b46c7d700 1 --2- 172.21.15.33:0/1878697325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b401014a0 0x7f7b40101880 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:54.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.297+0000 7f7b46c7d700 1 --2- 172.21.15.33:0/1878697325 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b40101e50 0x7f7b40105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:54.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.297+0000 7f7b46c7d700 1 -- 172.21.15.33:0/1878697325 >> 172.21.15.33:0/1878697325 conn(0x7f7b40077260 msgr2=0x7f7b40077670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:54.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.297+0000 7f7b46c7d700 1 -- 172.21.15.33:0/1878697325 shutdown_connections 2024-03-19T02:51:54.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.297+0000 7f7b46c7d700 1 -- 172.21.15.33:0/1878697325 wait complete. 2024-03-19T02:51:54.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.297+0000 7f7b46c7d700 1 Processor -- start 2024-03-19T02:51:54.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.298+0000 7f7b46c7d700 1 -- start start 2024-03-19T02:51:54.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.298+0000 7f7b46c7d700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b401014a0 0x7f7b40102eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:54.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.298+0000 7f7b46c7d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b40101e50 0x7f7b401033f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:54.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.298+0000 7f7b46c7d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b401064f0 0x7f7b40103930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:54.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.298+0000 7f7b46c7d700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7b40111380 con 0x7f7b401014a0 2024-03-19T02:51:54.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.298+0000 7f7b46c7d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7b40111200 con 0x7f7b40101e50 2024-03-19T02:51:54.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.298+0000 7f7b46c7d700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7b40111500 con 0x7f7b401064f0 2024-03-19T02:51:54.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.298+0000 7f7b44a19700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b401014a0 0x7f7b40102eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:54.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.298+0000 7f7b44a19700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b401014a0 0x7f7b40102eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50668/0 (socket says 172.21.15.33:50668) 2024-03-19T02:51:54.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.298+0000 7f7b44a19700 1 -- 172.21.15.33:0/2163630227 learned_addr learned my addr 172.21.15.33:0/2163630227 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:54.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.298+0000 7f7b3ffff700 1 --2- 172.21.15.33:0/2163630227 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b40101e50 0x7f7b401033f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:54.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.298+0000 7f7b4521a700 1 --2- 172.21.15.33:0/2163630227 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b401064f0 0x7f7b40103930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:54.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.298+0000 7f7b3ffff700 1 -- 172.21.15.33:0/2163630227 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b401064f0 msgr2=0x7f7b40103930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:54.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.298+0000 7f7b3ffff700 1 --2- 172.21.15.33:0/2163630227 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b401064f0 0x7f7b40103930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:54.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.298+0000 7f7b3ffff700 1 -- 172.21.15.33:0/2163630227 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b401014a0 msgr2=0x7f7b40102eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:54.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.298+0000 7f7b3ffff700 1 --2- 172.21.15.33:0/2163630227 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b401014a0 0x7f7b40102eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:54.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.298+0000 7f7b3ffff700 1 -- 172.21.15.33:0/2163630227 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b401a6f10 con 0x7f7b40101e50 2024-03-19T02:51:54.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.298+0000 7f7b3ffff700 1 --2- 172.21.15.33:0/2163630227 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b40101e50 0x7f7b401033f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c00e9c0 tx=0x7f7b2c00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:54.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.299+0000 7f7b3dffb700 1 -- 172.21.15.33:0/2163630227 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c00c9c0 con 0x7f7b40101e50 2024-03-19T02:51:54.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.299+0000 7f7b3dffb700 1 -- 172.21.15.33:0/2163630227 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7b2c00cb60 con 0x7f7b40101e50 2024-03-19T02:51:54.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.299+0000 7f7b46c7d700 1 -- 172.21.15.33:0/2163630227 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b401a7200 con 0x7f7b40101e50 2024-03-19T02:51:54.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.299+0000 7f7b3dffb700 1 -- 172.21.15.33:0/2163630227 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c018680 con 0x7f7b40101e50 2024-03-19T02:51:54.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.299+0000 7f7b46c7d700 1 -- 172.21.15.33:0/2163630227 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b401a7740 con 0x7f7b40101e50 2024-03-19T02:51:54.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.300+0000 7f7b46c7d700 1 -- 172.21.15.33:0/2163630227 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7b40062a20 con 0x7f7b40101e50 2024-03-19T02:51:54.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.301+0000 7f7b3dffb700 1 -- 172.21.15.33:0/2163630227 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f7b2c010430 con 0x7f7b40101e50 2024-03-19T02:51:54.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.302+0000 7f7b3dffb700 1 --2- 172.21.15.33:0/2163630227 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7b300a2920 0x7f7b300a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:54.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.302+0000 7f7b44a19700 1 --2- 172.21.15.33:0/2163630227 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7b300a2920 0x7f7b300a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:54.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.302+0000 7f7b3dffb700 1 -- 172.21.15.33:0/2163630227 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7f7b2c0cf660 con 0x7f7b40101e50 2024-03-19T02:51:54.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.302+0000 7f7b3dffb700 1 --2- 172.21.15.33:0/2163630227 >> [v2:172.21.15.155:6800/3388743876,v1:172.21.15.155:6801/3388743876] conn(0x7f7b300ada80 0x7f7b300aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:54.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.302+0000 7f7b3dffb700 1 -- 172.21.15.33:0/2163630227 --> [v2:172.21.15.155:6800/3388743876,v1:172.21.15.155:6801/3388743876] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7b2c01bec0 con 0x7f7b300ada80 2024-03-19T02:51:54.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.302+0000 7f7b3dffb700 1 -- 172.21.15.33:0/2163630227 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7f7b2c0cfa10 con 0x7f7b40101e50 2024-03-19T02:51:54.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.303+0000 7f7b4521a700 1 --2- 172.21.15.33:0/2163630227 >> [v2:172.21.15.155:6800/3388743876,v1:172.21.15.155:6801/3388743876] conn(0x7f7b300ada80 0x7f7b300aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:54.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.303+0000 7f7b44a19700 1 --2- 172.21.15.33:0/2163630227 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7b300a2920 0x7f7b300a4de0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7b34009990 tx=0x7f7b34005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:54.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.303+0000 7f7b4521a700 1 --2- 172.21.15.33:0/2163630227 >> [v2:172.21.15.155:6800/3388743876,v1:172.21.15.155:6801/3388743876] conn(0x7f7b300ada80 0x7f7b300aff20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:54.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.304+0000 7f7b3dffb700 1 -- 172.21.15.33:0/2163630227 <== osd.2 v2:172.21.15.155:6800/3388743876 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7b2c01bec0 con 0x7f7b300ada80 2024-03-19T02:51:54.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.321+0000 7f7b46c7d700 1 -- 172.21.15.33:0/2163630227 --> [v2:172.21.15.155:6800/3388743876,v1:172.21.15.155:6801/3388743876] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7b40062ea0 con 0x7f7b300ada80 2024-03-19T02:51:54.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.322+0000 7f7b3dffb700 1 -- 172.21.15.33:0/2163630227 <== osd.2 v2:172.21.15.155:6800/3388743876 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7b40062ea0 con 0x7f7b300ada80 2024-03-19T02:51:54.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.322+0000 7f7b46c7d700 1 -- 172.21.15.33:0/2163630227 >> [v2:172.21.15.155:6800/3388743876,v1:172.21.15.155:6801/3388743876] conn(0x7f7b300ada80 msgr2=0x7f7b300aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:54.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.322+0000 7f7b46c7d700 1 --2- 172.21.15.33:0/2163630227 >> [v2:172.21.15.155:6800/3388743876,v1:172.21.15.155:6801/3388743876] conn(0x7f7b300ada80 0x7f7b300aff20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:54.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.322+0000 7f7b46c7d700 1 -- 172.21.15.33:0/2163630227 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7b300a2920 msgr2=0x7f7b300a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:54.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.322+0000 7f7b46c7d700 1 --2- 172.21.15.33:0/2163630227 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7b300a2920 0x7f7b300a4de0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7b34009990 tx=0x7f7b34005960 comp rx=0 tx=0).stop 2024-03-19T02:51:54.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.322+0000 7f7b46c7d700 1 -- 172.21.15.33:0/2163630227 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b40101e50 msgr2=0x7f7b401033f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:54.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.322+0000 7f7b46c7d700 1 --2- 172.21.15.33:0/2163630227 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b40101e50 0x7f7b401033f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c00e9c0 tx=0x7f7b2c00ecd0 comp rx=0 tx=0).stop 2024-03-19T02:51:54.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.322+0000 7f7b4521a700 1 -- 172.21.15.33:0/2163630227 reap_dead start 2024-03-19T02:51:54.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.322+0000 7f7b46c7d700 1 -- 172.21.15.33:0/2163630227 shutdown_connections 2024-03-19T02:51:54.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.322+0000 7f7b46c7d700 1 -- 172.21.15.33:0/2163630227 >> 172.21.15.33:0/2163630227 conn(0x7f7b40077260 msgr2=0x7f7b401053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:54.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.322+0000 7f7b46c7d700 1 -- 172.21.15.33:0/2163630227 shutdown_connections 2024-03-19T02:51:54.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.322+0000 7f7b46c7d700 1 -- 172.21.15.33:0/2163630227 wait complete. 2024-03-19T02:51:55.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.199+0000 7f67c083b700 1 -- 172.21.15.33:0/3683605835 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f67bc102fa0 msgr2=0x7f67bc106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:55.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.199+0000 7f67c083b700 1 --2- 172.21.15.33:0/3683605835 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f67bc102fa0 0x7f67bc106f40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f67b0009990 tx=0x7f67b0009ca0 comp rx=0 tx=0).stop 2024-03-19T02:51:55.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.199+0000 7f67c083b700 1 -- 172.21.15.33:0/3683605835 shutdown_connections 2024-03-19T02:51:55.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.199+0000 7f67c083b700 1 --2- 172.21.15.33:0/3683605835 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67bc107620 0x7f67bc10f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:55.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.199+0000 7f67c083b700 1 --2- 172.21.15.33:0/3683605835 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f67bc1025f0 0x7f67bc1029d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:55.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.199+0000 7f67c083b700 1 --2- 172.21.15.33:0/3683605835 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f67bc102fa0 0x7f67bc106f40 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:55.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.199+0000 7f67c083b700 1 -- 172.21.15.33:0/3683605835 >> 172.21.15.33:0/3683605835 conn(0x7f67bc0793a0 msgr2=0x7f67bc0797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:55.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.199+0000 7f67c083b700 1 -- 172.21.15.33:0/3683605835 shutdown_connections 2024-03-19T02:51:55.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.199+0000 7f67c083b700 1 -- 172.21.15.33:0/3683605835 wait complete. 2024-03-19T02:51:55.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.200+0000 7f67c083b700 1 Processor -- start 2024-03-19T02:51:55.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.200+0000 7f67c083b700 1 -- start start 2024-03-19T02:51:55.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.200+0000 7f67c083b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f67bc1025f0 0x7f67bc074610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:55.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.200+0000 7f67c083b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67bc102fa0 0x7f67bc074b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:55.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.200+0000 7f67c083b700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f67bc107620 0x7f67bc075090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:55.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.200+0000 7f67c083b700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc112490 con 0x7f67bc107620 2024-03-19T02:51:55.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.200+0000 7f67c083b700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc112310 con 0x7f67bc102fa0 2024-03-19T02:51:55.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.200+0000 7f67c083b700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc112610 con 0x7f67bc1025f0 2024-03-19T02:51:55.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.200+0000 7f67b9d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67bc102fa0 0x7f67bc074b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:55.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.200+0000 7f67ba59c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f67bc1025f0 0x7f67bc074610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:55.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.200+0000 7f67ba59c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f67bc1025f0 0x7f67bc074610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:42426/0 (socket says 172.21.15.33:42426) 2024-03-19T02:51:55.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.200+0000 7f67b9d9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67bc102fa0 0x7f67bc074b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:56834/0 (socket says 172.21.15.33:56834) 2024-03-19T02:51:55.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.200+0000 7f67ba59c700 1 -- 172.21.15.33:0/3929884620 learned_addr learned my addr 172.21.15.33:0/3929884620 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:55.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.200+0000 7f67bad9d700 1 --2- 172.21.15.33:0/3929884620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f67bc107620 0x7f67bc075090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:55.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.200+0000 7f67ba59c700 1 -- 172.21.15.33:0/3929884620 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67bc102fa0 msgr2=0x7f67bc074b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:55.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.200+0000 7f67ba59c700 1 --2- 172.21.15.33:0/3929884620 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67bc102fa0 0x7f67bc074b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:55.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.200+0000 7f67ba59c700 1 -- 172.21.15.33:0/3929884620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f67bc107620 msgr2=0x7f67bc075090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:55.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.200+0000 7f67ba59c700 1 --2- 172.21.15.33:0/3929884620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f67bc107620 0x7f67bc075090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:55.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.200+0000 7f67ba59c700 1 -- 172.21.15.33:0/3929884620 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67bc1a6f10 con 0x7f67bc1025f0 2024-03-19T02:51:55.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.200+0000 7f67bad9d700 1 --2- 172.21.15.33:0/3929884620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f67bc107620 0x7f67bc075090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:55.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.201+0000 7f67ba59c700 1 --2- 172.21.15.33:0/3929884620 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f67bc1025f0 0x7f67bc074610 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f67ac00e9c0 tx=0x7f67ac00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:55.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.201+0000 7f67ab7fe700 1 -- 172.21.15.33:0/3929884620 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ac00cbd0 con 0x7f67bc1025f0 2024-03-19T02:51:55.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.201+0000 7f67ab7fe700 1 -- 172.21.15.33:0/3929884620 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f67ac00ee60 con 0x7f67bc1025f0 2024-03-19T02:51:55.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.201+0000 7f67c083b700 1 -- 172.21.15.33:0/3929884620 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67bc1a7200 con 0x7f67bc1025f0 2024-03-19T02:51:55.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.201+0000 7f67ab7fe700 1 -- 172.21.15.33:0/3929884620 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ac0055b0 con 0x7f67bc1025f0 2024-03-19T02:51:55.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.201+0000 7f67c083b700 1 -- 172.21.15.33:0/3929884620 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67bc1a7710 con 0x7f67bc1025f0 2024-03-19T02:51:55.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.202+0000 7f67c083b700 1 -- 172.21.15.33:0/3929884620 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f67bc062a20 con 0x7f67bc1025f0 2024-03-19T02:51:55.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.204+0000 7f67ab7fe700 1 -- 172.21.15.33:0/3929884620 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f67ac004750 con 0x7f67bc1025f0 2024-03-19T02:51:55.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.205+0000 7f67ab7fe700 1 --2- 172.21.15.33:0/3929884620 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f67a40a29a0 0x7f67a40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:55.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.205+0000 7f67b9d9b700 1 --2- 172.21.15.33:0/3929884620 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f67a40a29a0 0x7f67a40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:55.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.205+0000 7f67ab7fe700 1 -- 172.21.15.33:0/3929884620 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7f67ac0cf560 con 0x7f67bc1025f0 2024-03-19T02:51:55.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.205+0000 7f67ab7fe700 1 --2- 172.21.15.33:0/3929884620 >> [v2:172.21.15.96:6808/3106158554,v1:172.21.15.96:6809/3106158554] conn(0x7f67a40adb00 0x7f67a40affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:55.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.205+0000 7f67ab7fe700 1 -- 172.21.15.33:0/3929884620 --> [v2:172.21.15.96:6808/3106158554,v1:172.21.15.96:6809/3106158554] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f67a40b0670 con 0x7f67a40adb00 2024-03-19T02:51:55.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.205+0000 7f67ab7fe700 1 -- 172.21.15.33:0/3929884620 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7f67ac0cf9b0 con 0x7f67bc1025f0 2024-03-19T02:51:55.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.205+0000 7f67bad9d700 1 --2- 172.21.15.33:0/3929884620 >> [v2:172.21.15.96:6808/3106158554,v1:172.21.15.96:6809/3106158554] conn(0x7f67a40adb00 0x7f67a40affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:55.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.206+0000 7f67b9d9b700 1 --2- 172.21.15.33:0/3929884620 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f67a40a29a0 0x7f67a40a4e60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f67b0006010 tx=0x7f67b0005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:55.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.206+0000 7f67bad9d700 1 --2- 172.21.15.33:0/3929884620 >> [v2:172.21.15.96:6808/3106158554,v1:172.21.15.96:6809/3106158554] conn(0x7f67a40adb00 0x7f67a40affa0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:55.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.207+0000 7f67ab7fe700 1 -- 172.21.15.33:0/3929884620 <== osd.4 v2:172.21.15.96:6808/3106158554 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f67a40b0670 con 0x7f67a40adb00 2024-03-19T02:51:55.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.230+0000 7f67c083b700 1 -- 172.21.15.33:0/3929884620 --> [v2:172.21.15.96:6808/3106158554,v1:172.21.15.96:6809/3106158554] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f67bc062ea0 con 0x7f67a40adb00 2024-03-19T02:51:55.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.231+0000 7f67ab7fe700 1 -- 172.21.15.33:0/3929884620 <== osd.4 v2:172.21.15.96:6808/3106158554 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f67bc062ea0 con 0x7f67a40adb00 2024-03-19T02:51:55.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.231+0000 7f67c083b700 1 -- 172.21.15.33:0/3929884620 >> [v2:172.21.15.96:6808/3106158554,v1:172.21.15.96:6809/3106158554] conn(0x7f67a40adb00 msgr2=0x7f67a40affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:55.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.231+0000 7f67c083b700 1 --2- 172.21.15.33:0/3929884620 >> [v2:172.21.15.96:6808/3106158554,v1:172.21.15.96:6809/3106158554] conn(0x7f67a40adb00 0x7f67a40affa0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:55.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.232+0000 7f67c083b700 1 -- 172.21.15.33:0/3929884620 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f67a40a29a0 msgr2=0x7f67a40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:55.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.232+0000 7f67c083b700 1 --2- 172.21.15.33:0/3929884620 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f67a40a29a0 0x7f67a40a4e60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f67b0006010 tx=0x7f67b0005960 comp rx=0 tx=0).stop 2024-03-19T02:51:55.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.232+0000 7f67c083b700 1 -- 172.21.15.33:0/3929884620 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f67bc1025f0 msgr2=0x7f67bc074610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:55.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.232+0000 7f67c083b700 1 --2- 172.21.15.33:0/3929884620 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f67bc1025f0 0x7f67bc074610 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f67ac00e9c0 tx=0x7f67ac00ecd0 comp rx=0 tx=0).stop 2024-03-19T02:51:55.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.232+0000 7f67bad9d700 1 -- 172.21.15.33:0/3929884620 reap_dead start 2024-03-19T02:51:55.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.232+0000 7f67c083b700 1 -- 172.21.15.33:0/3929884620 shutdown_connections 2024-03-19T02:51:55.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.232+0000 7f67c083b700 1 -- 172.21.15.33:0/3929884620 >> 172.21.15.33:0/3929884620 conn(0x7f67bc0793a0 msgr2=0x7f67bc1087a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:55.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.232+0000 7f67c083b700 1 -- 172.21.15.33:0/3929884620 shutdown_connections 2024-03-19T02:51:55.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:54.232+0000 7f67c083b700 1 -- 172.21.15.33:0/3929884620 wait complete. 2024-03-19T02:51:55.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:55 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:51:55.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:55 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:51:55.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:55 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:51:55.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:55 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:51:55.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:55 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:51:55.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:55 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:51:55.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:55 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:51:55.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:55 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:51:55.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:55 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:51:55.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:55 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:51:55.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:55 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:51:55.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:55 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:51:55.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.602+0000 7f8eeed6d700 1 -- 172.21.15.33:0/3694346978 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ee81014d0 msgr2=0x7f8ee81018b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:55.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.602+0000 7f8eeed6d700 1 --2- 172.21.15.33:0/3694346978 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ee81014d0 0x7f8ee81018b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8ed80099c0 tx=0x7f8ed8009cd0 comp rx=0 tx=0).stop 2024-03-19T02:51:55.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.602+0000 7f8eeed6d700 1 -- 172.21.15.33:0/3694346978 shutdown_connections 2024-03-19T02:51:55.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.602+0000 7f8eeed6d700 1 --2- 172.21.15.33:0/3694346978 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ee8106520 0x7f8ee810e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:55.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.602+0000 7f8eeed6d700 1 --2- 172.21.15.33:0/3694346978 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ee81014d0 0x7f8ee81018b0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:55.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.602+0000 7f8eeed6d700 1 --2- 172.21.15.33:0/3694346978 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ee8101e80 0x7f8ee8105e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:55.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.602+0000 7f8eeed6d700 1 -- 172.21.15.33:0/3694346978 >> 172.21.15.33:0/3694346978 conn(0x7f8ee8077260 msgr2=0x7f8ee8077670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:55.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.602+0000 7f8eeed6d700 1 -- 172.21.15.33:0/3694346978 shutdown_connections 2024-03-19T02:51:55.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.602+0000 7f8eeed6d700 1 -- 172.21.15.33:0/3694346978 wait complete. 2024-03-19T02:51:55.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.603+0000 7f8eeed6d700 1 Processor -- start 2024-03-19T02:51:55.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.603+0000 7f8eeed6d700 1 -- start start 2024-03-19T02:51:55.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.603+0000 7f8eeed6d700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ee81014d0 0x7f8ee81a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:55.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.603+0000 7f8eeed6d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ee8101e80 0x7f8ee81a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:55.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.603+0000 7f8eeed6d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ee8106520 0x7f8ee819a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:55.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.603+0000 7f8eeed6d700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee81113e0 con 0x7f8ee81014d0 2024-03-19T02:51:55.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.603+0000 7f8eeed6d700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee8111260 con 0x7f8ee8106520 2024-03-19T02:51:55.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.603+0000 7f8eeed6d700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee8111560 con 0x7f8ee8101e80 2024-03-19T02:51:55.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.603+0000 7f8eecb09700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ee81014d0 0x7f8ee81a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:55.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.603+0000 7f8eecb09700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ee81014d0 0x7f8ee81a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50674/0 (socket says 172.21.15.33:50674) 2024-03-19T02:51:55.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.603+0000 7f8eecb09700 1 -- 172.21.15.33:0/2119094138 learned_addr learned my addr 172.21.15.33:0/2119094138 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:55.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.603+0000 7f8ee7fff700 1 --2- 172.21.15.33:0/2119094138 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ee8101e80 0x7f8ee81a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:55.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.603+0000 7f8eed30a700 1 --2- 172.21.15.33:0/2119094138 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ee8106520 0x7f8ee819a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:55.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.603+0000 7f8eecb09700 1 -- 172.21.15.33:0/2119094138 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ee8101e80 msgr2=0x7f8ee81a0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:55.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.603+0000 7f8eecb09700 1 --2- 172.21.15.33:0/2119094138 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ee8101e80 0x7f8ee81a0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:55.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.603+0000 7f8eecb09700 1 -- 172.21.15.33:0/2119094138 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ee8106520 msgr2=0x7f8ee819a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:55.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.603+0000 7f8eecb09700 1 --2- 172.21.15.33:0/2119094138 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ee8106520 0x7f8ee819a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:55.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.603+0000 7f8eecb09700 1 -- 172.21.15.33:0/2119094138 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ee819b130 con 0x7f8ee81014d0 2024-03-19T02:51:55.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.603+0000 7f8ee7fff700 1 --2- 172.21.15.33:0/2119094138 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ee8101e80 0x7f8ee81a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:55.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.604+0000 7f8eecb09700 1 --2- 172.21.15.33:0/2119094138 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ee81014d0 0x7f8ee81a0810 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8ed8006010 tx=0x7f8ed8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:55.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.604+0000 7f8ee5ffb700 1 -- 172.21.15.33:0/2119094138 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ed8026070 con 0x7f8ee81014d0 2024-03-19T02:51:55.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.604+0000 7f8ee5ffb700 1 -- 172.21.15.33:0/2119094138 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8ed8004630 con 0x7f8ee81014d0 2024-03-19T02:51:55.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.604+0000 7f8eeed6d700 1 -- 172.21.15.33:0/2119094138 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ee819b3c0 con 0x7f8ee81014d0 2024-03-19T02:51:55.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.604+0000 7f8ee5ffb700 1 -- 172.21.15.33:0/2119094138 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ed800fb20 con 0x7f8ee81014d0 2024-03-19T02:51:55.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.604+0000 7f8eeed6d700 1 -- 172.21.15.33:0/2119094138 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ee81a76e0 con 0x7f8ee81014d0 2024-03-19T02:51:55.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.605+0000 7f8eeed6d700 1 -- 172.21.15.33:0/2119094138 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8ee8100c40 con 0x7f8ee81014d0 2024-03-19T02:51:55.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.606+0000 7f8ee5ffb700 1 -- 172.21.15.33:0/2119094138 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f8ed8029070 con 0x7f8ee81014d0 2024-03-19T02:51:55.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.607+0000 7f8ee5ffb700 1 --2- 172.21.15.33:0/2119094138 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8ed00a2920 0x7f8ed00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:55.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.607+0000 7f8ee7fff700 1 --2- 172.21.15.33:0/2119094138 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8ed00a2920 0x7f8ed00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:55.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.607+0000 7f8ee5ffb700 1 -- 172.21.15.33:0/2119094138 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7f8ed80cfe10 con 0x7f8ee81014d0 2024-03-19T02:51:55.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.607+0000 7f8ee5ffb700 1 --2- 172.21.15.33:0/2119094138 >> [v2:172.21.15.155:6824/792352827,v1:172.21.15.155:6825/792352827] conn(0x7f8ed00ad950 0x7f8ed00afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:55.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.607+0000 7f8ee5ffb700 1 -- 172.21.15.33:0/2119094138 --> [v2:172.21.15.155:6824/792352827,v1:172.21.15.155:6825/792352827] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8ed00b04c0 con 0x7f8ed00ad950 2024-03-19T02:51:55.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.607+0000 7f8ee5ffb700 1 -- 172.21.15.33:0/2119094138 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7f8ed80d01c0 con 0x7f8ee81014d0 2024-03-19T02:51:55.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.607+0000 7f8eed30a700 1 --2- 172.21.15.33:0/2119094138 >> [v2:172.21.15.155:6824/792352827,v1:172.21.15.155:6825/792352827] conn(0x7f8ed00ad950 0x7f8ed00afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:55.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.607+0000 7f8ee7fff700 1 --2- 172.21.15.33:0/2119094138 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8ed00a2920 0x7f8ed00a4de0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8ee806ba60 tx=0x7f8edc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:55.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.607+0000 7f8eed30a700 1 --2- 172.21.15.33:0/2119094138 >> [v2:172.21.15.155:6824/792352827,v1:172.21.15.155:6825/792352827] conn(0x7f8ed00ad950 0x7f8ed00afdf0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:55.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.608+0000 7f8ee5ffb700 1 -- 172.21.15.33:0/2119094138 <== osd.11 v2:172.21.15.155:6824/792352827 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8ed00b04c0 con 0x7f8ed00ad950 2024-03-19T02:51:55.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.625+0000 7f8eeed6d700 1 -- 172.21.15.33:0/2119094138 --> [v2:172.21.15.155:6824/792352827,v1:172.21.15.155:6825/792352827] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8ee8102920 con 0x7f8ed00ad950 2024-03-19T02:51:55.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.625+0000 7f8ee5ffb700 1 -- 172.21.15.33:0/2119094138 <== osd.11 v2:172.21.15.155:6824/792352827 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8ee8102920 con 0x7f8ed00ad950 2024-03-19T02:51:55.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.626+0000 7f8eeed6d700 1 -- 172.21.15.33:0/2119094138 >> [v2:172.21.15.155:6824/792352827,v1:172.21.15.155:6825/792352827] conn(0x7f8ed00ad950 msgr2=0x7f8ed00afdf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:55.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.626+0000 7f8eeed6d700 1 --2- 172.21.15.33:0/2119094138 >> [v2:172.21.15.155:6824/792352827,v1:172.21.15.155:6825/792352827] conn(0x7f8ed00ad950 0x7f8ed00afdf0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:55.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.626+0000 7f8eeed6d700 1 -- 172.21.15.33:0/2119094138 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8ed00a2920 msgr2=0x7f8ed00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:55.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.626+0000 7f8eeed6d700 1 --2- 172.21.15.33:0/2119094138 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8ed00a2920 0x7f8ed00a4de0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8ee806ba60 tx=0x7f8edc009260 comp rx=0 tx=0).stop 2024-03-19T02:51:55.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.626+0000 7f8eeed6d700 1 -- 172.21.15.33:0/2119094138 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ee81014d0 msgr2=0x7f8ee81a0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:55.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.626+0000 7f8eeed6d700 1 --2- 172.21.15.33:0/2119094138 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8ee81014d0 0x7f8ee81a0810 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8ed8006010 tx=0x7f8ed8004270 comp rx=0 tx=0).stop 2024-03-19T02:51:55.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.626+0000 7f8eed30a700 1 -- 172.21.15.33:0/2119094138 reap_dead start 2024-03-19T02:51:55.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.626+0000 7f8eeed6d700 1 -- 172.21.15.33:0/2119094138 shutdown_connections 2024-03-19T02:51:55.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.626+0000 7f8eeed6d700 1 -- 172.21.15.33:0/2119094138 >> 172.21.15.33:0/2119094138 conn(0x7f8ee8077260 msgr2=0x7f8ee8107820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:55.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.626+0000 7f8eeed6d700 1 -- 172.21.15.33:0/2119094138 shutdown_connections 2024-03-19T02:51:55.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:55.626+0000 7f8eeed6d700 1 -- 172.21.15.33:0/2119094138 wait complete. 2024-03-19T02:51:56.455 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:56 smithi033 ceph-mon[31973]: pgmap v333: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s; 48 B/s, 0 objects/s recovering 2024-03-19T02:51:56.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:56 smithi096 ceph-mon[35946]: pgmap v333: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s; 48 B/s, 0 objects/s recovering 2024-03-19T02:51:56.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:56 smithi155 ceph-mon[35804]: pgmap v333: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s; 48 B/s, 0 objects/s recovering 2024-03-19T02:51:57.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.200+0000 7f856afa6700 1 -- 172.21.15.33:0/3175451018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8564101e50 msgr2=0x7f8564105e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:57.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.200+0000 7f856afa6700 1 --2- 172.21.15.33:0/3175451018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8564101e50 0x7f8564105e50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8550009990 tx=0x7f8550009ca0 comp rx=0 tx=0).stop 2024-03-19T02:51:57.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.200+0000 7f856afa6700 1 -- 172.21.15.33:0/3175451018 shutdown_connections 2024-03-19T02:51:57.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.200+0000 7f856afa6700 1 --2- 172.21.15.33:0/3175451018 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8564106530 0x7f856410e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:57.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.200+0000 7f856afa6700 1 --2- 172.21.15.33:0/3175451018 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85641014a0 0x7f8564101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:57.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.200+0000 7f856afa6700 1 --2- 172.21.15.33:0/3175451018 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8564101e50 0x7f8564105e50 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:57.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.200+0000 7f856afa6700 1 -- 172.21.15.33:0/3175451018 >> 172.21.15.33:0/3175451018 conn(0x7f8564077230 msgr2=0x7f8564077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:57.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.200+0000 7f856afa6700 1 -- 172.21.15.33:0/3175451018 shutdown_connections 2024-03-19T02:51:57.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.200+0000 7f856afa6700 1 -- 172.21.15.33:0/3175451018 wait complete. 2024-03-19T02:51:57.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.201+0000 7f856afa6700 1 Processor -- start 2024-03-19T02:51:57.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.201+0000 7f856afa6700 1 -- start start 2024-03-19T02:51:57.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.201+0000 7f856afa6700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85641014a0 0x7f8564102ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:57.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.201+0000 7f856afa6700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8564101e50 0x7f8564103430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:57.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.201+0000 7f856afa6700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8564106530 0x7f8564103970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:57.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.201+0000 7f856afa6700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f85641113a0 con 0x7f85641014a0 2024-03-19T02:51:57.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.201+0000 7f856afa6700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8564111220 con 0x7f8564106530 2024-03-19T02:51:57.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.201+0000 7f856afa6700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8564111520 con 0x7f8564101e50 2024-03-19T02:51:57.203 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.201+0000 7f8568d42700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85641014a0 0x7f8564102ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:57.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.201+0000 7f8568d42700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85641014a0 0x7f8564102ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50684/0 (socket says 172.21.15.33:50684) 2024-03-19T02:51:57.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.201+0000 7f8568d42700 1 -- 172.21.15.33:0/1176853191 learned_addr learned my addr 172.21.15.33:0/1176853191 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:57.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.201+0000 7f8569543700 1 --2- 172.21.15.33:0/1176853191 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8564106530 0x7f8564103970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:57.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.201+0000 7f8563fff700 1 --2- 172.21.15.33:0/1176853191 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8564101e50 0x7f8564103430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:57.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.201+0000 7f8568d42700 1 -- 172.21.15.33:0/1176853191 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8564101e50 msgr2=0x7f8564103430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:57.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.201+0000 7f8568d42700 1 --2- 172.21.15.33:0/1176853191 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8564101e50 0x7f8564103430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:57.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.201+0000 7f8568d42700 1 -- 172.21.15.33:0/1176853191 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8564106530 msgr2=0x7f8564103970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:57.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.201+0000 7f8568d42700 1 --2- 172.21.15.33:0/1176853191 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8564106530 0x7f8564103970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:57.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.201+0000 7f8568d42700 1 -- 172.21.15.33:0/1176853191 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85641a6f30 con 0x7f85641014a0 2024-03-19T02:51:57.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.202+0000 7f8568d42700 1 --2- 172.21.15.33:0/1176853191 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85641014a0 0x7f8564102ef0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f855800e9c0 tx=0x7f855800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:57.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.202+0000 7f8561ffb700 1 -- 172.21.15.33:0/1176853191 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f855800cbd0 con 0x7f85641014a0 2024-03-19T02:51:57.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.202+0000 7f8561ffb700 1 -- 172.21.15.33:0/1176853191 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f855800ee60 con 0x7f85641014a0 2024-03-19T02:51:57.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.202+0000 7f856afa6700 1 -- 172.21.15.33:0/1176853191 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85641a7220 con 0x7f85641014a0 2024-03-19T02:51:57.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.202+0000 7f8561ffb700 1 -- 172.21.15.33:0/1176853191 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85580055b0 con 0x7f85641014a0 2024-03-19T02:51:57.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.202+0000 7f856afa6700 1 -- 172.21.15.33:0/1176853191 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85641a76e0 con 0x7f85641014a0 2024-03-19T02:51:57.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.203+0000 7f856afa6700 1 -- 172.21.15.33:0/1176853191 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f85640629e0 con 0x7f85641014a0 2024-03-19T02:51:57.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.204+0000 7f8561ffb700 1 -- 172.21.15.33:0/1176853191 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f8558004750 con 0x7f85641014a0 2024-03-19T02:51:57.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.204+0000 7f8561ffb700 1 --2- 172.21.15.33:0/1176853191 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f85540a2920 0x7f85540a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:57.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.205+0000 7f8561ffb700 1 -- 172.21.15.33:0/1176853191 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7f85580cf610 con 0x7f85641014a0 2024-03-19T02:51:57.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.205+0000 7f8563fff700 1 --2- 172.21.15.33:0/1176853191 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f85540a2920 0x7f85540a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:57.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.205+0000 7f8561ffb700 1 --2- 172.21.15.33:0/1176853191 >> [v2:172.21.15.33:6802/1633124995,v1:172.21.15.33:6803/1633124995] conn(0x7f85540ada80 0x7f85540aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:57.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.205+0000 7f8561ffb700 1 -- 172.21.15.33:0/1176853191 --> [v2:172.21.15.33:6802/1633124995,v1:172.21.15.33:6803/1633124995] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f85540b05f0 con 0x7f85540ada80 2024-03-19T02:51:57.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.205+0000 7f8561ffb700 1 -- 172.21.15.33:0/1176853191 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7f85580cfa60 con 0x7f85641014a0 2024-03-19T02:51:57.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.205+0000 7f8569543700 1 --2- 172.21.15.33:0/1176853191 >> [v2:172.21.15.33:6802/1633124995,v1:172.21.15.33:6803/1633124995] conn(0x7f85540ada80 0x7f85540aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:57.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.205+0000 7f8563fff700 1 --2- 172.21.15.33:0/1176853191 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f85540a2920 0x7f85540a4de0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8550006010 tx=0x7f8550005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:57.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.205+0000 7f8569543700 1 --2- 172.21.15.33:0/1176853191 >> [v2:172.21.15.33:6802/1633124995,v1:172.21.15.33:6803/1633124995] conn(0x7f85540ada80 0x7f85540aff20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:57.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.206+0000 7f8561ffb700 1 -- 172.21.15.33:0/1176853191 <== osd.0 v2:172.21.15.33:6802/1633124995 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f85540b05f0 con 0x7f85540ada80 2024-03-19T02:51:57.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.222+0000 7f856afa6700 1 -- 172.21.15.33:0/1176853191 --> [v2:172.21.15.33:6802/1633124995,v1:172.21.15.33:6803/1633124995] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8564062e90 con 0x7f85540ada80 2024-03-19T02:51:57.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.223+0000 7f8561ffb700 1 -- 172.21.15.33:0/1176853191 <== osd.0 v2:172.21.15.33:6802/1633124995 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8564062e90 con 0x7f85540ada80 2024-03-19T02:51:57.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.223+0000 7f856afa6700 1 -- 172.21.15.33:0/1176853191 >> [v2:172.21.15.33:6802/1633124995,v1:172.21.15.33:6803/1633124995] conn(0x7f85540ada80 msgr2=0x7f85540aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:57.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.223+0000 7f856afa6700 1 --2- 172.21.15.33:0/1176853191 >> [v2:172.21.15.33:6802/1633124995,v1:172.21.15.33:6803/1633124995] conn(0x7f85540ada80 0x7f85540aff20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:57.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.223+0000 7f856afa6700 1 -- 172.21.15.33:0/1176853191 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f85540a2920 msgr2=0x7f85540a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:57.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.224+0000 7f856afa6700 1 --2- 172.21.15.33:0/1176853191 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f85540a2920 0x7f85540a4de0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8550006010 tx=0x7f8550005960 comp rx=0 tx=0).stop 2024-03-19T02:51:57.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.224+0000 7f856afa6700 1 -- 172.21.15.33:0/1176853191 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85641014a0 msgr2=0x7f8564102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:57.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.224+0000 7f856afa6700 1 --2- 172.21.15.33:0/1176853191 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85641014a0 0x7f8564102ef0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f855800e9c0 tx=0x7f855800ecd0 comp rx=0 tx=0).stop 2024-03-19T02:51:57.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.224+0000 7f8569543700 1 -- 172.21.15.33:0/1176853191 reap_dead start 2024-03-19T02:51:57.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.224+0000 7f856afa6700 1 -- 172.21.15.33:0/1176853191 shutdown_connections 2024-03-19T02:51:57.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.224+0000 7f856afa6700 1 -- 172.21.15.33:0/1176853191 >> 172.21.15.33:0/1176853191 conn(0x7f8564077230 msgr2=0x7f856406adb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:57.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.224+0000 7f856afa6700 1 -- 172.21.15.33:0/1176853191 shutdown_connections 2024-03-19T02:51:57.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.224+0000 7f856afa6700 1 -- 172.21.15.33:0/1176853191 wait complete. 2024-03-19T02:51:57.645 INFO:teuthology.orchestra.run.smithi033.stdout:146028888126 2024-03-19T02:51:57.645 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-19T02:51:57.928 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:57 smithi033 ceph-mon[31973]: pgmap v334: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-19T02:51:57.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.958+0000 7f167d322700 1 -- 172.21.15.33:0/3826695072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1678102550 msgr2=0x7f1678102930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:57.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.958+0000 7f167d322700 1 --2- 172.21.15.33:0/3826695072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1678102550 0x7f1678102930 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f16680099c0 tx=0x7f1668009cd0 comp rx=0 tx=0).stop 2024-03-19T02:51:57.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.958+0000 7f167d322700 1 -- 172.21.15.33:0/3826695072 shutdown_connections 2024-03-19T02:51:57.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.958+0000 7f167d322700 1 --2- 172.21.15.33:0/3826695072 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1678107630 0x7f167810f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:57.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.958+0000 7f167d322700 1 --2- 172.21.15.33:0/3826695072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1678102550 0x7f1678102930 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:57.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.958+0000 7f167d322700 1 --2- 172.21.15.33:0/3826695072 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1678102f00 0x7f1678106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:57.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.958+0000 7f167d322700 1 -- 172.21.15.33:0/3826695072 >> 172.21.15.33:0/3826695072 conn(0x7f16780fddc0 msgr2=0x7f16781001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:57.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.958+0000 7f167d322700 1 -- 172.21.15.33:0/3826695072 shutdown_connections 2024-03-19T02:51:57.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.958+0000 7f167d322700 1 -- 172.21.15.33:0/3826695072 wait complete. 2024-03-19T02:51:57.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.958+0000 7f167d322700 1 Processor -- start 2024-03-19T02:51:57.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.958+0000 7f167d322700 1 -- start start 2024-03-19T02:51:57.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.959+0000 7f167d322700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1678102550 0x7f167819e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:57.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.959+0000 7f167d322700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1678102f00 0x7f167819ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:57.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.959+0000 7f167d322700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1678107630 0x7f1678198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:57.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.959+0000 7f167d322700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f16781124f0 con 0x7f1678102550 2024-03-19T02:51:57.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.959+0000 7f167d322700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1678112370 con 0x7f1678102f00 2024-03-19T02:51:57.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.959+0000 7f167d322700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1678112670 con 0x7f1678107630 2024-03-19T02:51:57.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.959+0000 7f1676ffd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1678102550 0x7f167819e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:57.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.959+0000 7f1676ffd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1678102550 0x7f167819e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50696/0 (socket says 172.21.15.33:50696) 2024-03-19T02:51:57.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.959+0000 7f1676ffd700 1 -- 172.21.15.33:0/1569855041 learned_addr learned my addr 172.21.15.33:0/1569855041 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:57.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.959+0000 7f16767fc700 1 --2- 172.21.15.33:0/1569855041 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1678102f00 0x7f167819ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:57.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.959+0000 7f16777fe700 1 --2- 172.21.15.33:0/1569855041 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1678107630 0x7f1678198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:57.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.959+0000 7f16767fc700 1 -- 172.21.15.33:0/1569855041 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1678107630 msgr2=0x7f1678198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:57.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.959+0000 7f16767fc700 1 --2- 172.21.15.33:0/1569855041 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1678107630 0x7f1678198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:57.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.959+0000 7f16767fc700 1 -- 172.21.15.33:0/1569855041 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1678102550 msgr2=0x7f167819e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:57.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.959+0000 7f16767fc700 1 --2- 172.21.15.33:0/1569855041 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1678102550 0x7f167819e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:57.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.959+0000 7f16767fc700 1 -- 172.21.15.33:0/1569855041 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1678199030 con 0x7f1678102f00 2024-03-19T02:51:57.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.959+0000 7f1676ffd700 1 --2- 172.21.15.33:0/1569855041 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1678102550 0x7f167819e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:57.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.959+0000 7f16767fc700 1 --2- 172.21.15.33:0/1569855041 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1678102f00 0x7f167819ebc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f166c00e9c0 tx=0x7f166c00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:57.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.960+0000 7f165ffff700 1 -- 172.21.15.33:0/1569855041 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166c00c9c0 con 0x7f1678102f00 2024-03-19T02:51:57.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.960+0000 7f165ffff700 1 -- 172.21.15.33:0/1569855041 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f166c00cb60 con 0x7f1678102f00 2024-03-19T02:51:57.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.960+0000 7f165ffff700 1 -- 172.21.15.33:0/1569855041 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166c018680 con 0x7f1678102f00 2024-03-19T02:51:57.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.960+0000 7f167d322700 1 -- 172.21.15.33:0/1569855041 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1678199320 con 0x7f1678102f00 2024-03-19T02:51:57.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.960+0000 7f167d322700 1 -- 172.21.15.33:0/1569855041 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16781a54e0 con 0x7f1678102f00 2024-03-19T02:51:57.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.961+0000 7f167d322700 1 -- 172.21.15.33:0/1569855041 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1678062a60 con 0x7f1678102f00 2024-03-19T02:51:57.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.962+0000 7f165ffff700 1 -- 172.21.15.33:0/1569855041 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f166c047a90 con 0x7f1678102f00 2024-03-19T02:51:57.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.962+0000 7f165ffff700 1 --2- 172.21.15.33:0/1569855041 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f16600a2920 0x7f16600a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:57.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.962+0000 7f1676ffd700 1 --2- 172.21.15.33:0/1569855041 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f16600a2920 0x7f16600a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:57.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.962+0000 7f165ffff700 1 -- 172.21.15.33:0/1569855041 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7f166c0cf190 con 0x7f1678102f00 2024-03-19T02:51:57.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.963+0000 7f165ffff700 1 --2- 172.21.15.33:0/1569855041 >> [v2:172.21.15.96:6800/340144398,v1:172.21.15.96:6801/340144398] conn(0x7f16600ada80 0x7f16600aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:57.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.963+0000 7f165ffff700 1 -- 172.21.15.33:0/1569855041 --> [v2:172.21.15.96:6800/340144398,v1:172.21.15.96:6801/340144398] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f166c017e70 con 0x7f16600ada80 2024-03-19T02:51:57.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.963+0000 7f165ffff700 1 -- 172.21.15.33:0/1569855041 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7f166c0cf540 con 0x7f1678102f00 2024-03-19T02:51:57.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.963+0000 7f16777fe700 1 --2- 172.21.15.33:0/1569855041 >> [v2:172.21.15.96:6800/340144398,v1:172.21.15.96:6801/340144398] conn(0x7f16600ada80 0x7f16600aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:57.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.963+0000 7f1676ffd700 1 --2- 172.21.15.33:0/1569855041 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f16600a2920 0x7f16600a4de0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1668009990 tx=0x7f1668005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:57.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.963+0000 7f16777fe700 1 --2- 172.21.15.33:0/1569855041 >> [v2:172.21.15.96:6800/340144398,v1:172.21.15.96:6801/340144398] conn(0x7f16600ada80 0x7f16600aff20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:57.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.964+0000 7f165ffff700 1 -- 172.21.15.33:0/1569855041 <== osd.1 v2:172.21.15.96:6800/340144398 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f166c017e70 con 0x7f16600ada80 2024-03-19T02:51:57.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.980+0000 7f167d322700 1 -- 172.21.15.33:0/1569855041 --> [v2:172.21.15.96:6800/340144398,v1:172.21.15.96:6801/340144398] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1678062ee0 con 0x7f16600ada80 2024-03-19T02:51:57.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.981+0000 7f165ffff700 1 -- 172.21.15.33:0/1569855041 <== osd.1 v2:172.21.15.96:6800/340144398 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1678062ee0 con 0x7f16600ada80 2024-03-19T02:51:57.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.981+0000 7f167d322700 1 -- 172.21.15.33:0/1569855041 >> [v2:172.21.15.96:6800/340144398,v1:172.21.15.96:6801/340144398] conn(0x7f16600ada80 msgr2=0x7f16600aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:57.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.981+0000 7f167d322700 1 --2- 172.21.15.33:0/1569855041 >> [v2:172.21.15.96:6800/340144398,v1:172.21.15.96:6801/340144398] conn(0x7f16600ada80 0x7f16600aff20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:57.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.981+0000 7f167d322700 1 -- 172.21.15.33:0/1569855041 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f16600a2920 msgr2=0x7f16600a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:57.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.981+0000 7f167d322700 1 --2- 172.21.15.33:0/1569855041 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f16600a2920 0x7f16600a4de0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1668009990 tx=0x7f1668005960 comp rx=0 tx=0).stop 2024-03-19T02:51:57.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.981+0000 7f167d322700 1 -- 172.21.15.33:0/1569855041 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1678102f00 msgr2=0x7f167819ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:57.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.981+0000 7f167d322700 1 --2- 172.21.15.33:0/1569855041 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1678102f00 0x7f167819ebc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f166c00e9c0 tx=0x7f166c00ecd0 comp rx=0 tx=0).stop 2024-03-19T02:51:57.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.981+0000 7f16777fe700 1 -- 172.21.15.33:0/1569855041 reap_dead start 2024-03-19T02:51:57.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.981+0000 7f167d322700 1 -- 172.21.15.33:0/1569855041 shutdown_connections 2024-03-19T02:51:57.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.981+0000 7f167d322700 1 -- 172.21.15.33:0/1569855041 >> 172.21.15.33:0/1569855041 conn(0x7f16780fddc0 msgr2=0x7f1678100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:57.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.981+0000 7f167d322700 1 -- 172.21.15.33:0/1569855041 shutdown_connections 2024-03-19T02:51:57.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:57.981+0000 7f167d322700 1 -- 172.21.15.33:0/1569855041 wait complete. 2024-03-19T02:51:58.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:57 smithi096 ceph-mon[35946]: pgmap v334: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-19T02:51:58.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:57 smithi155 ceph-mon[35804]: pgmap v334: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-19T02:51:58.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.361+0000 7f7bfc7ef700 1 -- 172.21.15.33:0/2389158378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bf41065c0 msgr2=0x7f7bf410e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:58.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.361+0000 7f7bfc7ef700 1 --2- 172.21.15.33:0/2389158378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bf41065c0 0x7f7bf410e190 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7be400b0d0 tx=0x7f7be400b3e0 comp rx=0 tx=0).stop 2024-03-19T02:51:58.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.361+0000 7f7bfc7ef700 1 -- 172.21.15.33:0/2389158378 shutdown_connections 2024-03-19T02:51:58.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.361+0000 7f7bfc7ef700 1 --2- 172.21.15.33:0/2389158378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bf41065c0 0x7f7bf410e190 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:58.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.361+0000 7f7bfc7ef700 1 --2- 172.21.15.33:0/2389158378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7bf41014e0 0x7f7bf41018c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:58.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.361+0000 7f7bfc7ef700 1 --2- 172.21.15.33:0/2389158378 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7bf4101e90 0x7f7bf4105ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:58.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.361+0000 7f7bfc7ef700 1 -- 172.21.15.33:0/2389158378 >> 172.21.15.33:0/2389158378 conn(0x7f7bf4078340 msgr2=0x7f7bf4078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:58.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.361+0000 7f7bfc7ef700 1 -- 172.21.15.33:0/2389158378 shutdown_connections 2024-03-19T02:51:58.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.361+0000 7f7bfc7ef700 1 -- 172.21.15.33:0/2389158378 wait complete. 2024-03-19T02:51:58.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.362+0000 7f7bfc7ef700 1 Processor -- start 2024-03-19T02:51:58.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.362+0000 7f7bfc7ef700 1 -- start start 2024-03-19T02:51:58.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.362+0000 7f7bfc7ef700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7bf41014e0 0x7f7bf4199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:58.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.362+0000 7f7bfc7ef700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7bf4101e90 0x7f7bf4199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:58.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.362+0000 7f7bfc7ef700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bf41065c0 0x7f7bf41a48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:58.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.362+0000 7f7bfc7ef700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf4111460 con 0x7f7bf41065c0 2024-03-19T02:51:58.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.362+0000 7f7bfc7ef700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf41112e0 con 0x7f7bf4101e90 2024-03-19T02:51:58.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.362+0000 7f7bfc7ef700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf41115e0 con 0x7f7bf41014e0 2024-03-19T02:51:58.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.362+0000 7f7bfad8c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bf41065c0 0x7f7bf41a48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:58.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.362+0000 7f7bfad8c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bf41065c0 0x7f7bf41a48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50720/0 (socket says 172.21.15.33:50720) 2024-03-19T02:51:58.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.362+0000 7f7bfad8c700 1 -- 172.21.15.33:0/568646614 learned_addr learned my addr 172.21.15.33:0/568646614 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:51:58.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.362+0000 7f7bf9d8a700 1 --2- 172.21.15.33:0/568646614 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7bf4101e90 0x7f7bf4199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:58.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.362+0000 7f7bfa58b700 1 --2- 172.21.15.33:0/568646614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7bf41014e0 0x7f7bf4199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:58.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.362+0000 7f7bfad8c700 1 -- 172.21.15.33:0/568646614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7bf41014e0 msgr2=0x7f7bf4199860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:58.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.362+0000 7f7bfad8c700 1 --2- 172.21.15.33:0/568646614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7bf41014e0 0x7f7bf4199860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:58.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.362+0000 7f7bfad8c700 1 -- 172.21.15.33:0/568646614 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7bf4101e90 msgr2=0x7f7bf4199da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:58.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.362+0000 7f7bfad8c700 1 --2- 172.21.15.33:0/568646614 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7bf4101e90 0x7f7bf4199da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:58.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.362+0000 7f7bfad8c700 1 -- 172.21.15.33:0/568646614 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bf41a4e20 con 0x7f7bf41065c0 2024-03-19T02:51:58.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.362+0000 7f7bf9d8a700 1 --2- 172.21.15.33:0/568646614 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7bf4101e90 0x7f7bf4199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:51:58.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.362+0000 7f7bfad8c700 1 --2- 172.21.15.33:0/568646614 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bf41065c0 0x7f7bf41a48e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7be4007a00 tx=0x7f7be4007a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:58.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.363+0000 7f7bf37fe700 1 -- 172.21.15.33:0/568646614 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be401f070 con 0x7f7bf41065c0 2024-03-19T02:51:58.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.363+0000 7f7bf37fe700 1 -- 172.21.15.33:0/568646614 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7be4004c50 con 0x7f7bf41065c0 2024-03-19T02:51:58.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.363+0000 7f7bfc7ef700 1 -- 172.21.15.33:0/568646614 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bf41a4ff0 con 0x7f7bf41065c0 2024-03-19T02:51:58.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.363+0000 7f7bf37fe700 1 -- 172.21.15.33:0/568646614 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be401b690 con 0x7f7bf41065c0 2024-03-19T02:51:58.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.363+0000 7f7bfc7ef700 1 -- 172.21.15.33:0/568646614 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bf41a53e0 con 0x7f7bf41065c0 2024-03-19T02:51:58.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.363+0000 7f7bfc7ef700 1 -- 172.21.15.33:0/568646614 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7bf40629e0 con 0x7f7bf41065c0 2024-03-19T02:51:58.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.365+0000 7f7bf37fe700 1 -- 172.21.15.33:0/568646614 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f7be4022070 con 0x7f7bf41065c0 2024-03-19T02:51:58.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.365+0000 7f7bf37fe700 1 --2- 172.21.15.33:0/568646614 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7bdc0a2920 0x7f7bdc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:58.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.365+0000 7f7bfa58b700 1 --2- 172.21.15.33:0/568646614 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7bdc0a2920 0x7f7bdc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:58.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.365+0000 7f7bf37fe700 1 -- 172.21.15.33:0/568646614 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7f7be400f020 con 0x7f7bf41065c0 2024-03-19T02:51:58.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.366+0000 7f7bf37fe700 1 --2- 172.21.15.33:0/568646614 >> [v2:172.21.15.33:6826/760777213,v1:172.21.15.33:6827/760777213] conn(0x7f7bdc0ada80 0x7f7bdc0aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:51:58.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.366+0000 7f7bf37fe700 1 -- 172.21.15.33:0/568646614 --> [v2:172.21.15.33:6826/760777213,v1:172.21.15.33:6827/760777213] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7bdc0b05f0 con 0x7f7bdc0ada80 2024-03-19T02:51:58.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.366+0000 7f7bf37fe700 1 -- 172.21.15.33:0/568646614 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7f7be40d0630 con 0x7f7bf41065c0 2024-03-19T02:51:58.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.366+0000 7f7bf9d8a700 1 --2- 172.21.15.33:0/568646614 >> [v2:172.21.15.33:6826/760777213,v1:172.21.15.33:6827/760777213] conn(0x7f7bdc0ada80 0x7f7bdc0aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:51:58.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.366+0000 7f7bfa58b700 1 --2- 172.21.15.33:0/568646614 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7bdc0a2920 0x7f7bdc0a4de0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7bec009c60 tx=0x7f7bec009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:58.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.366+0000 7f7bf9d8a700 1 --2- 172.21.15.33:0/568646614 >> [v2:172.21.15.33:6826/760777213,v1:172.21.15.33:6827/760777213] conn(0x7f7bdc0ada80 0x7f7bdc0aff20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:51:58.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.367+0000 7f7bf37fe700 1 -- 172.21.15.33:0/568646614 <== osd.9 v2:172.21.15.33:6826/760777213 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7bdc0b05f0 con 0x7f7bdc0ada80 2024-03-19T02:51:58.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.383+0000 7f7bfc7ef700 1 -- 172.21.15.33:0/568646614 --> [v2:172.21.15.33:6826/760777213,v1:172.21.15.33:6827/760777213] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7bf4062de0 con 0x7f7bdc0ada80 2024-03-19T02:51:58.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.384+0000 7f7bf37fe700 1 -- 172.21.15.33:0/568646614 <== osd.9 v2:172.21.15.33:6826/760777213 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7bf4062de0 con 0x7f7bdc0ada80 2024-03-19T02:51:58.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.384+0000 7f7bfc7ef700 1 -- 172.21.15.33:0/568646614 >> [v2:172.21.15.33:6826/760777213,v1:172.21.15.33:6827/760777213] conn(0x7f7bdc0ada80 msgr2=0x7f7bdc0aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:58.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.384+0000 7f7bfc7ef700 1 --2- 172.21.15.33:0/568646614 >> [v2:172.21.15.33:6826/760777213,v1:172.21.15.33:6827/760777213] conn(0x7f7bdc0ada80 0x7f7bdc0aff20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:51:58.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.384+0000 7f7bfc7ef700 1 -- 172.21.15.33:0/568646614 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7bdc0a2920 msgr2=0x7f7bdc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:58.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.384+0000 7f7bfc7ef700 1 --2- 172.21.15.33:0/568646614 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7bdc0a2920 0x7f7bdc0a4de0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7bec009c60 tx=0x7f7bec009260 comp rx=0 tx=0).stop 2024-03-19T02:51:58.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.384+0000 7f7bfc7ef700 1 -- 172.21.15.33:0/568646614 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bf41065c0 msgr2=0x7f7bf41a48e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:51:58.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.384+0000 7f7bfc7ef700 1 --2- 172.21.15.33:0/568646614 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7bf41065c0 0x7f7bf41a48e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7be4007a00 tx=0x7f7be4007a30 comp rx=0 tx=0).stop 2024-03-19T02:51:58.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.384+0000 7f7bfad8c700 1 -- 172.21.15.33:0/568646614 reap_dead start 2024-03-19T02:51:58.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.384+0000 7f7bfc7ef700 1 -- 172.21.15.33:0/568646614 shutdown_connections 2024-03-19T02:51:58.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.384+0000 7f7bfc7ef700 1 -- 172.21.15.33:0/568646614 >> 172.21.15.33:0/568646614 conn(0x7f7bf4078340 msgr2=0x7f7bf40ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:51:58.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.384+0000 7f7bfc7ef700 1 -- 172.21.15.33:0/568646614 shutdown_connections 2024-03-19T02:51:58.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:51:58.385+0000 7f7bfc7ef700 1 -- 172.21.15.33:0/568646614 wait complete. 2024-03-19T02:51:59.885 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:51:59 smithi033 ceph-mon[31973]: pgmap v335: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-19T02:51:59.915 INFO:teuthology.orchestra.run.smithi033.stdout:352187318298 2024-03-19T02:51:59.915 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-19T02:52:00.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.099+0000 7fbdf7cb4700 1 -- 172.21.15.33:0/457369369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdf01083a0 msgr2=0x7fbdf0108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:00.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.099+0000 7fbdf7cb4700 1 --2- 172.21.15.33:0/457369369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdf01083a0 0x7fbdf0108780 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fbdec009990 tx=0x7fbdec009ca0 comp rx=0 tx=0).stop 2024-03-19T02:52:00.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.099+0000 7fbdf7cb4700 1 -- 172.21.15.33:0/457369369 shutdown_connections 2024-03-19T02:52:00.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.099+0000 7fbdf7cb4700 1 --2- 172.21.15.33:0/457369369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdf0102d30 0x7fbdf010f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:00.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.099+0000 7fbdf7cb4700 1 --2- 172.21.15.33:0/457369369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdf01083a0 0x7fbdf0108780 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:00.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.099+0000 7fbdf7cb4700 1 --2- 172.21.15.33:0/457369369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbdf0102390 0x7fbdf01027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:00.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.100+0000 7fbdf7cb4700 1 -- 172.21.15.33:0/457369369 >> 172.21.15.33:0/457369369 conn(0x7fbdf00793e0 msgr2=0x7fbdf00797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:00.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.100+0000 7fbdf7cb4700 1 -- 172.21.15.33:0/457369369 shutdown_connections 2024-03-19T02:52:00.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.100+0000 7fbdf7cb4700 1 -- 172.21.15.33:0/457369369 wait complete. 2024-03-19T02:52:00.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.100+0000 7fbdf7cb4700 1 Processor -- start 2024-03-19T02:52:00.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.100+0000 7fbdf7cb4700 1 -- start start 2024-03-19T02:52:00.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.100+0000 7fbdf7cb4700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbdf0102390 0x7fbdf0074610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:00.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.100+0000 7fbdf7cb4700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdf0102d30 0x7fbdf0074b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:00.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.100+0000 7fbdf7cb4700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdf01083a0 0x7fbdf0075090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:00.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.101+0000 7fbdf7cb4700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf01124b0 con 0x7fbdf01083a0 2024-03-19T02:52:00.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.101+0000 7fbdf7cb4700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf0112330 con 0x7fbdf0102d30 2024-03-19T02:52:00.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.101+0000 7fbdf7cb4700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf0112630 con 0x7fbdf0102390 2024-03-19T02:52:00.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.101+0000 7fbdf524f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdf0102d30 0x7fbdf0074b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:00.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.101+0000 7fbdf5a50700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbdf0102390 0x7fbdf0074610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:00.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.101+0000 7fbdf6251700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdf01083a0 0x7fbdf0075090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:00.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.101+0000 7fbdf524f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdf0102d30 0x7fbdf0074b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:56954/0 (socket says 172.21.15.33:56954) 2024-03-19T02:52:00.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.101+0000 7fbdf524f700 1 -- 172.21.15.33:0/1915023074 learned_addr learned my addr 172.21.15.33:0/1915023074 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:00.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.101+0000 7fbdf5a50700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbdf0102390 0x7fbdf0074610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:42536/0 (socket says 172.21.15.33:42536) 2024-03-19T02:52:00.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.101+0000 7fbdf524f700 1 -- 172.21.15.33:0/1915023074 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbdf0102390 msgr2=0x7fbdf0074610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:00.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.101+0000 7fbdf524f700 1 --2- 172.21.15.33:0/1915023074 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbdf0102390 0x7fbdf0074610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:00.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.101+0000 7fbdf524f700 1 -- 172.21.15.33:0/1915023074 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdf01083a0 msgr2=0x7fbdf0075090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:00.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.101+0000 7fbdf524f700 1 --2- 172.21.15.33:0/1915023074 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdf01083a0 0x7fbdf0075090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:00.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.101+0000 7fbdf524f700 1 -- 172.21.15.33:0/1915023074 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdf01a6f10 con 0x7fbdf0102d30 2024-03-19T02:52:00.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.101+0000 7fbdf6251700 1 --2- 172.21.15.33:0/1915023074 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbdf01083a0 0x7fbdf0075090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:00.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.101+0000 7fbdf5a50700 1 --2- 172.21.15.33:0/1915023074 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbdf0102390 0x7fbdf0074610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:00.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.101+0000 7fbdf524f700 1 --2- 172.21.15.33:0/1915023074 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdf0102d30 0x7fbdf0074b50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbde000e9c0 tx=0x7fbde000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:00.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.101+0000 7fbde6ffd700 1 -- 172.21.15.33:0/1915023074 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde000cbd0 con 0x7fbdf0102d30 2024-03-19T02:52:00.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.101+0000 7fbde6ffd700 1 -- 172.21.15.33:0/1915023074 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbde000ee60 con 0x7fbdf0102d30 2024-03-19T02:52:00.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.101+0000 7fbdf7cb4700 1 -- 172.21.15.33:0/1915023074 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdf01a7200 con 0x7fbdf0102d30 2024-03-19T02:52:00.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.102+0000 7fbde6ffd700 1 -- 172.21.15.33:0/1915023074 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde00055b0 con 0x7fbdf0102d30 2024-03-19T02:52:00.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.102+0000 7fbdf7cb4700 1 -- 172.21.15.33:0/1915023074 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdf01a76f0 con 0x7fbdf0102d30 2024-03-19T02:52:00.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.103+0000 7fbdf7cb4700 1 -- 172.21.15.33:0/1915023074 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbdf0062a20 con 0x7fbdf0102d30 2024-03-19T02:52:00.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.104+0000 7fbde6ffd700 1 -- 172.21.15.33:0/1915023074 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fbde0004830 con 0x7fbdf0102d30 2024-03-19T02:52:00.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.105+0000 7fbde6ffd700 1 --2- 172.21.15.33:0/1915023074 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbddc0a2920 0x7fbddc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:00.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.105+0000 7fbde6ffd700 1 -- 172.21.15.33:0/1915023074 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7fbde00cf250 con 0x7fbdf0102d30 2024-03-19T02:52:00.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.105+0000 7fbdf5a50700 1 --2- 172.21.15.33:0/1915023074 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbddc0a2920 0x7fbddc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:00.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.105+0000 7fbde6ffd700 1 --2- 172.21.15.33:0/1915023074 >> [v2:172.21.15.96:6824/1404898482,v1:172.21.15.96:6825/1404898482] conn(0x7fbddc0ada80 0x7fbddc0aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:00.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.105+0000 7fbde6ffd700 1 -- 172.21.15.33:0/1915023074 --> [v2:172.21.15.96:6824/1404898482,v1:172.21.15.96:6825/1404898482] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbddc0b05f0 con 0x7fbddc0ada80 2024-03-19T02:52:00.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.105+0000 7fbde6ffd700 1 -- 172.21.15.33:0/1915023074 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_get_version_reply(handle=1 version=95) ==== 24+0+0 (secure 0 0 0) 0x7fbde00d6050 con 0x7fbdf0102d30 2024-03-19T02:52:00.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.105+0000 7fbdf5a50700 1 --2- 172.21.15.33:0/1915023074 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbddc0a2920 0x7fbddc0a4de0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbdec00b2e0 tx=0x7fbdec01a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:00.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.105+0000 7fbdf6251700 1 --2- 172.21.15.33:0/1915023074 >> [v2:172.21.15.96:6824/1404898482,v1:172.21.15.96:6825/1404898482] conn(0x7fbddc0ada80 0x7fbddc0aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:00.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.106+0000 7fbdf6251700 1 --2- 172.21.15.33:0/1915023074 >> [v2:172.21.15.96:6824/1404898482,v1:172.21.15.96:6825/1404898482] conn(0x7fbddc0ada80 0x7fbddc0aff20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:00.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.107+0000 7fbde6ffd700 1 -- 172.21.15.33:0/1915023074 <== osd.10 v2:172.21.15.96:6824/1404898482 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbddc0b05f0 con 0x7fbddc0ada80 2024-03-19T02:52:00.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.123+0000 7fbdf7cb4700 1 -- 172.21.15.33:0/1915023074 --> [v2:172.21.15.96:6824/1404898482,v1:172.21.15.96:6825/1404898482] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbdf0062ea0 con 0x7fbddc0ada80 2024-03-19T02:52:00.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.124+0000 7fbde6ffd700 1 -- 172.21.15.33:0/1915023074 <== osd.10 v2:172.21.15.96:6824/1404898482 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fbdf0062ea0 con 0x7fbddc0ada80 2024-03-19T02:52:00.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.124+0000 7fbdf7cb4700 1 -- 172.21.15.33:0/1915023074 >> [v2:172.21.15.96:6824/1404898482,v1:172.21.15.96:6825/1404898482] conn(0x7fbddc0ada80 msgr2=0x7fbddc0aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:00.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.124+0000 7fbdf7cb4700 1 --2- 172.21.15.33:0/1915023074 >> [v2:172.21.15.96:6824/1404898482,v1:172.21.15.96:6825/1404898482] conn(0x7fbddc0ada80 0x7fbddc0aff20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:00.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.124+0000 7fbdf7cb4700 1 -- 172.21.15.33:0/1915023074 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbddc0a2920 msgr2=0x7fbddc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:00.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.124+0000 7fbdf7cb4700 1 --2- 172.21.15.33:0/1915023074 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbddc0a2920 0x7fbddc0a4de0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbdec00b2e0 tx=0x7fbdec01a040 comp rx=0 tx=0).stop 2024-03-19T02:52:00.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.124+0000 7fbdf7cb4700 1 -- 172.21.15.33:0/1915023074 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdf0102d30 msgr2=0x7fbdf0074b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:00.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.124+0000 7fbdf7cb4700 1 --2- 172.21.15.33:0/1915023074 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbdf0102d30 0x7fbdf0074b50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbde000e9c0 tx=0x7fbde000ecd0 comp rx=0 tx=0).stop 2024-03-19T02:52:00.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.124+0000 7fbdf6251700 1 -- 172.21.15.33:0/1915023074 reap_dead start 2024-03-19T02:52:00.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.124+0000 7fbdf7cb4700 1 -- 172.21.15.33:0/1915023074 shutdown_connections 2024-03-19T02:52:00.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.124+0000 7fbdf7cb4700 1 -- 172.21.15.33:0/1915023074 >> 172.21.15.33:0/1915023074 conn(0x7fbdf00793e0 msgr2=0x7fbdf0101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:00.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.124+0000 7fbdf7cb4700 1 -- 172.21.15.33:0/1915023074 shutdown_connections 2024-03-19T02:52:00.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:00.124+0000 7fbdf7cb4700 1 -- 172.21.15.33:0/1915023074 wait complete. 2024-03-19T02:52:00.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:51:59 smithi096 ceph-mon[35946]: pgmap v335: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-19T02:52:00.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:51:59 smithi155 ceph-mon[35804]: pgmap v335: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-19T02:52:00.460 INFO:teuthology.orchestra.run.smithi033.stdout:55834574926 2024-03-19T02:52:00.460 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-19T02:52:00.592 INFO:teuthology.orchestra.run.smithi033.stdout:77309411401 2024-03-19T02:52:00.592 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-19T02:52:01.221 INFO:teuthology.orchestra.run.smithi033.stdout:317827579936 2024-03-19T02:52:01.222 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-19T02:52:01.231 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:52:01.231 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:52:01.628 INFO:teuthology.orchestra.run.smithi033.stdout:296352743461 2024-03-19T02:52:01.628 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-19T02:52:01.767 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:52:01.768 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:52:01.771 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:52:01.814 INFO:teuthology.orchestra.run.smithi033.stdout:34359738451 2024-03-19T02:52:01.814 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-19T02:52:01.875 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:01 smithi033 ceph-mon[31973]: pgmap v336: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-19T02:52:02.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:01 smithi096 ceph-mon[35946]: pgmap v336: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-19T02:52:02.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:01 smithi155 ceph-mon[35804]: pgmap v336: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-19T02:52:03.966 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:03 smithi033 ceph-mon[31973]: pgmap v337: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:52:04.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:03 smithi096 ceph-mon[35946]: pgmap v337: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:52:04.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:03 smithi155 ceph-mon[35804]: pgmap v337: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:52:04.295 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:52:04.296 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:52:04.296 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:52:05.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.281+0000 7f4a7bbcc700 1 -- 172.21.15.33:0/1000098932 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a74101e10 msgr2=0x7f4a74105e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:05.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.281+0000 7f4a7bbcc700 1 --2- 172.21.15.33:0/1000098932 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a74101e10 0x7f4a74105e40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4a680098f0 tx=0x7f4a68009c00 comp rx=0 tx=0).stop 2024-03-19T02:52:05.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.281+0000 7f4a7bbcc700 1 -- 172.21.15.33:0/1000098932 shutdown_connections 2024-03-19T02:52:05.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.281+0000 7f4a7bbcc700 1 --2- 172.21.15.33:0/1000098932 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4a74106520 0x7f4a7410e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:05.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.281+0000 7f4a7bbcc700 1 --2- 172.21.15.33:0/1000098932 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a74101460 0x7f4a74101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:05.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.281+0000 7f4a7bbcc700 1 --2- 172.21.15.33:0/1000098932 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a74101e10 0x7f4a74105e40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:05.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.281+0000 7f4a7bbcc700 1 -- 172.21.15.33:0/1000098932 >> 172.21.15.33:0/1000098932 conn(0x7f4a74078180 msgr2=0x7f4a74078590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:05.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.281+0000 7f4a7bbcc700 1 -- 172.21.15.33:0/1000098932 shutdown_connections 2024-03-19T02:52:05.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.281+0000 7f4a7bbcc700 1 -- 172.21.15.33:0/1000098932 wait complete. 2024-03-19T02:52:05.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.282+0000 7f4a7bbcc700 1 Processor -- start 2024-03-19T02:52:05.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.282+0000 7f4a7bbcc700 1 -- start start 2024-03-19T02:52:05.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.282+0000 7f4a7bbcc700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4a74101460 0x7f4a74102ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:05.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.282+0000 7f4a7bbcc700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a74101e10 0x7f4a74103420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:05.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.282+0000 7f4a7bbcc700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a74106520 0x7f4a74103960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:05.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.282+0000 7f4a7bbcc700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4a74111390 con 0x7f4a74101460 2024-03-19T02:52:05.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.282+0000 7f4a7bbcc700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4a74111210 con 0x7f4a74101e10 2024-03-19T02:52:05.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.282+0000 7f4a7bbcc700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4a74111510 con 0x7f4a74106520 2024-03-19T02:52:05.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.282+0000 7f4a79968700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4a74101460 0x7f4a74102ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:05.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.282+0000 7f4a79968700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4a74101460 0x7f4a74102ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53430/0 (socket says 172.21.15.33:53430) 2024-03-19T02:52:05.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.282+0000 7f4a79968700 1 -- 172.21.15.33:0/921506898 learned_addr learned my addr 172.21.15.33:0/921506898 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:05.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.282+0000 7f4a79167700 1 --2- 172.21.15.33:0/921506898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a74101e10 0x7f4a74103420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:05.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.282+0000 7f4a7a169700 1 --2- 172.21.15.33:0/921506898 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a74106520 0x7f4a74103960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:05.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.282+0000 7f4a79968700 1 -- 172.21.15.33:0/921506898 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a74106520 msgr2=0x7f4a74103960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:05.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.282+0000 7f4a79968700 1 --2- 172.21.15.33:0/921506898 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a74106520 0x7f4a74103960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:05.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.282+0000 7f4a79968700 1 -- 172.21.15.33:0/921506898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a74101e10 msgr2=0x7f4a74103420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:05.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.282+0000 7f4a79968700 1 --2- 172.21.15.33:0/921506898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a74101e10 0x7f4a74103420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:05.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.282+0000 7f4a79968700 1 -- 172.21.15.33:0/921506898 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a741a6fb0 con 0x7f4a74101460 2024-03-19T02:52:05.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.283+0000 7f4a79968700 1 --2- 172.21.15.33:0/921506898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4a74101460 0x7f4a74102ee0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4a6400c110 tx=0x7f4a6400c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:05.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.283+0000 7f4a72ffd700 1 -- 172.21.15.33:0/921506898 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a640092e0 con 0x7f4a74101460 2024-03-19T02:52:05.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.283+0000 7f4a7bbcc700 1 -- 172.21.15.33:0/921506898 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a741a7240 con 0x7f4a74101460 2024-03-19T02:52:05.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.283+0000 7f4a72ffd700 1 -- 172.21.15.33:0/921506898 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4a64004bb0 con 0x7f4a74101460 2024-03-19T02:52:05.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.283+0000 7f4a72ffd700 1 -- 172.21.15.33:0/921506898 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a64007c70 con 0x7f4a74101460 2024-03-19T02:52:05.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.283+0000 7f4a7bbcc700 1 -- 172.21.15.33:0/921506898 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a741a7750 con 0x7f4a74101460 2024-03-19T02:52:05.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.284+0000 7f4a7bbcc700 1 -- 172.21.15.33:0/921506898 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a7406a260 con 0x7f4a74101460 2024-03-19T02:52:05.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.285+0000 7f4a72ffd700 1 -- 172.21.15.33:0/921506898 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f4a6401a020 con 0x7f4a74101460 2024-03-19T02:52:05.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.285+0000 7f4a72ffd700 1 --2- 172.21.15.33:0/921506898 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4a5c0a2920 0x7f4a5c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:05.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.285+0000 7f4a79167700 1 --2- 172.21.15.33:0/921506898 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4a5c0a2920 0x7f4a5c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:05.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.285+0000 7f4a72ffd700 1 -- 172.21.15.33:0/921506898 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7f4a640cf5d0 con 0x7f4a74101460 2024-03-19T02:52:05.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.286+0000 7f4a79167700 1 --2- 172.21.15.33:0/921506898 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4a5c0a2920 0x7f4a5c0a4de0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4a68009810 tx=0x7f4a68019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:05.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.289+0000 7f4a72ffd700 1 -- 172.21.15.33:0/921506898 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4a64010040 con 0x7f4a74101460 2024-03-19T02:52:05.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.498+0000 7f4a7bbcc700 1 -- 172.21.15.33:0/921506898 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f4a74104410 con 0x7f4a74101460 2024-03-19T02:52:05.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.498+0000 7f4a72ffd700 1 -- 172.21.15.33:0/921506898 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f4a6409c1b0 con 0x7f4a74101460 2024-03-19T02:52:05.499 INFO:teuthology.orchestra.run.smithi033.stdout:227633266737 2024-03-19T02:52:05.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.500+0000 7f4a7bbcc700 1 -- 172.21.15.33:0/921506898 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4a5c0a2920 msgr2=0x7f4a5c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:05.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.500+0000 7f4a7bbcc700 1 --2- 172.21.15.33:0/921506898 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4a5c0a2920 0x7f4a5c0a4de0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4a68009810 tx=0x7f4a68019040 comp rx=0 tx=0).stop 2024-03-19T02:52:05.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.500+0000 7f4a7bbcc700 1 -- 172.21.15.33:0/921506898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4a74101460 msgr2=0x7f4a74102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:05.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.500+0000 7f4a7bbcc700 1 --2- 172.21.15.33:0/921506898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4a74101460 0x7f4a74102ee0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4a6400c110 tx=0x7f4a6400c420 comp rx=0 tx=0).stop 2024-03-19T02:52:05.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.500+0000 7f4a7bbcc700 1 -- 172.21.15.33:0/921506898 shutdown_connections 2024-03-19T02:52:05.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.500+0000 7f4a7bbcc700 1 --2- 172.21.15.33:0/921506898 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f4a5c0a2920 0x7f4a5c0a4de0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:05.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.500+0000 7f4a7bbcc700 1 --2- 172.21.15.33:0/921506898 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4a74106520 0x7f4a74103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:05.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.500+0000 7f4a7bbcc700 1 --2- 172.21.15.33:0/921506898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4a74101460 0x7f4a74102ee0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:05.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.500+0000 7f4a7bbcc700 1 --2- 172.21.15.33:0/921506898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a74101e10 0x7f4a74103420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:05.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.500+0000 7f4a7bbcc700 1 -- 172.21.15.33:0/921506898 >> 172.21.15.33:0/921506898 conn(0x7f4a74078180 msgr2=0x7f4a7406adf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:05.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.500+0000 7f4a7bbcc700 1 -- 172.21.15.33:0/921506898 shutdown_connections 2024-03-19T02:52:05.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:05.500+0000 7f4a7bbcc700 1 -- 172.21.15.33:0/921506898 wait complete. 2024-03-19T02:52:05.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:05 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/921506898' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-19T02:52:05.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:05 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/921506898' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-19T02:52:06.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:05 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/921506898' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-19T02:52:06.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:06 smithi096 ceph-mon[35946]: pgmap v338: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:52:07.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:06 smithi155 ceph-mon[35804]: pgmap v338: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:52:07.020 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:06 smithi033 ceph-mon[31973]: pgmap v338: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:52:07.642 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633266734 got 227633266737 for osd.7 2024-03-19T02:52:07.643 DEBUG:teuthology.parallel:result is None 2024-03-19T02:52:07.859 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:07 smithi033 ceph-mon[31973]: pgmap v339: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:52:08.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:07 smithi096 ceph-mon[35946]: pgmap v339: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:52:08.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:07 smithi155 ceph-mon[35804]: pgmap v339: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:52:08.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.203+0000 7f6c8a45f700 1 -- 172.21.15.33:0/2709897314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c841083a0 msgr2=0x7f6c84108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:08.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.203+0000 7f6c8a45f700 1 --2- 172.21.15.33:0/2709897314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c841083a0 0x7f6c84108780 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f6c780099c0 tx=0x7f6c78009cd0 comp rx=0 tx=0).stop 2024-03-19T02:52:08.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.203+0000 7f6c8a45f700 1 -- 172.21.15.33:0/2709897314 shutdown_connections 2024-03-19T02:52:08.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.203+0000 7f6c8a45f700 1 --2- 172.21.15.33:0/2709897314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6c84102d50 0x7f6c8410f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:08.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.203+0000 7f6c8a45f700 1 --2- 172.21.15.33:0/2709897314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c841083a0 0x7f6c84108780 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:08.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.203+0000 7f6c8a45f700 1 --2- 172.21.15.33:0/2709897314 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c84102390 0x7f6c84102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:08.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.203+0000 7f6c8a45f700 1 -- 172.21.15.33:0/2709897314 >> 172.21.15.33:0/2709897314 conn(0x7f6c84079470 msgr2=0x7f6c84079880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:08.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.203+0000 7f6c8a45f700 1 -- 172.21.15.33:0/2709897314 shutdown_connections 2024-03-19T02:52:08.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.203+0000 7f6c8a45f700 1 -- 172.21.15.33:0/2709897314 wait complete. 2024-03-19T02:52:08.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.204+0000 7f6c8a45f700 1 Processor -- start 2024-03-19T02:52:08.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.204+0000 7f6c8a45f700 1 -- start start 2024-03-19T02:52:08.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.205+0000 7f6c8a45f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c84102390 0x7f6c841a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:08.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.205+0000 7f6c8a45f700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c84102d50 0x7f6c841a0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:08.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.205+0000 7f6c8a45f700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6c841083a0 0x7f6c8419a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:08.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.205+0000 7f6c8a45f700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6c84112550 con 0x7f6c84102d50 2024-03-19T02:52:08.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.205+0000 7f6c8a45f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6c841123d0 con 0x7f6c84102390 2024-03-19T02:52:08.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.205+0000 7f6c8a45f700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6c841126d0 con 0x7f6c841083a0 2024-03-19T02:52:08.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.205+0000 7f6c837fe700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c84102d50 0x7f6c841a0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:08.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.205+0000 7f6c837fe700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c84102d50 0x7f6c841a0d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53450/0 (socket says 172.21.15.33:53450) 2024-03-19T02:52:08.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.205+0000 7f6c837fe700 1 -- 172.21.15.33:0/1607963993 learned_addr learned my addr 172.21.15.33:0/1607963993 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:08.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.205+0000 7f6c83fff700 1 --2- 172.21.15.33:0/1607963993 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c84102390 0x7f6c841a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:08.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.205+0000 7f6c889fc700 1 --2- 172.21.15.33:0/1607963993 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6c841083a0 0x7f6c8419a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:08.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.205+0000 7f6c837fe700 1 -- 172.21.15.33:0/1607963993 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6c841083a0 msgr2=0x7f6c8419a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:08.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.205+0000 7f6c837fe700 1 --2- 172.21.15.33:0/1607963993 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6c841083a0 0x7f6c8419a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:08.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.205+0000 7f6c837fe700 1 -- 172.21.15.33:0/1607963993 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c84102390 msgr2=0x7f6c841a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:08.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.205+0000 7f6c837fe700 1 --2- 172.21.15.33:0/1607963993 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c84102390 0x7f6c841a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:08.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.205+0000 7f6c837fe700 1 -- 172.21.15.33:0/1607963993 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c8419b140 con 0x7f6c84102d50 2024-03-19T02:52:08.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.205+0000 7f6c83fff700 1 --2- 172.21.15.33:0/1607963993 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c84102390 0x7f6c841a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:08.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.205+0000 7f6c837fe700 1 --2- 172.21.15.33:0/1607963993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c84102d50 0x7f6c841a0d60 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6c7400eb30 tx=0x7f6c7400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:08.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.206+0000 7f6c817fa700 1 -- 172.21.15.33:0/1607963993 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c7400cbd0 con 0x7f6c84102d50 2024-03-19T02:52:08.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.206+0000 7f6c817fa700 1 -- 172.21.15.33:0/1607963993 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6c740044d0 con 0x7f6c84102d50 2024-03-19T02:52:08.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.206+0000 7f6c8a45f700 1 -- 172.21.15.33:0/1607963993 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c8419b430 con 0x7f6c84102d50 2024-03-19T02:52:08.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.206+0000 7f6c817fa700 1 -- 172.21.15.33:0/1607963993 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c74003e30 con 0x7f6c84102d50 2024-03-19T02:52:08.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.206+0000 7f6c8a45f700 1 -- 172.21.15.33:0/1607963993 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c841a76f0 con 0x7f6c84102d50 2024-03-19T02:52:08.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.207+0000 7f6c8a45f700 1 -- 172.21.15.33:0/1607963993 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c84068fd0 con 0x7f6c84102d50 2024-03-19T02:52:08.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.208+0000 7f6c817fa700 1 -- 172.21.15.33:0/1607963993 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f6c74004000 con 0x7f6c84102d50 2024-03-19T02:52:08.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.209+0000 7f6c817fa700 1 --2- 172.21.15.33:0/1607963993 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f6c6c0a2920 0x7f6c6c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:08.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.209+0000 7f6c83fff700 1 --2- 172.21.15.33:0/1607963993 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f6c6c0a2920 0x7f6c6c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:08.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.209+0000 7f6c817fa700 1 -- 172.21.15.33:0/1607963993 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7f6c740cf290 con 0x7f6c84102d50 2024-03-19T02:52:08.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.209+0000 7f6c83fff700 1 --2- 172.21.15.33:0/1607963993 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f6c6c0a2920 0x7f6c6c0a4de0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6c78000c00 tx=0x7f6c78005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:08.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.213+0000 7f6c817fa700 1 -- 172.21.15.33:0/1607963993 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6c740042f0 con 0x7f6c84102d50 2024-03-19T02:52:08.631 INFO:teuthology.orchestra.run.smithi033.stdout:261993005100 2024-03-19T02:52:08.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.421+0000 7f6c8a45f700 1 -- 172.21.15.33:0/1607963993 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f6c841a7980 con 0x7f6c84102d50 2024-03-19T02:52:08.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.421+0000 7f6c817fa700 1 -- 172.21.15.33:0/1607963993 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f6c7409be70 con 0x7f6c84102d50 2024-03-19T02:52:08.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.424+0000 7f6c8a45f700 1 -- 172.21.15.33:0/1607963993 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f6c6c0a2920 msgr2=0x7f6c6c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:08.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.424+0000 7f6c8a45f700 1 --2- 172.21.15.33:0/1607963993 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f6c6c0a2920 0x7f6c6c0a4de0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6c78000c00 tx=0x7f6c78005fb0 comp rx=0 tx=0).stop 2024-03-19T02:52:08.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.424+0000 7f6c8a45f700 1 -- 172.21.15.33:0/1607963993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c84102d50 msgr2=0x7f6c841a0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:08.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.424+0000 7f6c8a45f700 1 --2- 172.21.15.33:0/1607963993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c84102d50 0x7f6c841a0d60 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6c7400eb30 tx=0x7f6c7400ee40 comp rx=0 tx=0).stop 2024-03-19T02:52:08.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.424+0000 7f6c8a45f700 1 -- 172.21.15.33:0/1607963993 shutdown_connections 2024-03-19T02:52:08.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.424+0000 7f6c8a45f700 1 --2- 172.21.15.33:0/1607963993 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f6c6c0a2920 0x7f6c6c0a4de0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:08.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.424+0000 7f6c8a45f700 1 --2- 172.21.15.33:0/1607963993 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6c841083a0 0x7f6c8419a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:08.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.424+0000 7f6c8a45f700 1 --2- 172.21.15.33:0/1607963993 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c84102390 0x7f6c841a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:08.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.424+0000 7f6c8a45f700 1 --2- 172.21.15.33:0/1607963993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c84102d50 0x7f6c841a0d60 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:08.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.424+0000 7f6c8a45f700 1 -- 172.21.15.33:0/1607963993 >> 172.21.15.33:0/1607963993 conn(0x7f6c84079470 msgr2=0x7f6c84074770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:08.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.424+0000 7f6c8a45f700 1 -- 172.21.15.33:0/1607963993 shutdown_connections 2024-03-19T02:52:08.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:08.424+0000 7f6c8a45f700 1 -- 172.21.15.33:0/1607963993 wait complete. 2024-03-19T02:52:08.898 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:08 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1607963993' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-19T02:52:09.189 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:52:09.189 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:52:09.189 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:52:09.190 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:52:09.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:08 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1607963993' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-19T02:52:09.222 INFO:tasks.cephadm.ceph_manager.ceph:need seq 261993005095 got 261993005100 for osd.8 2024-03-19T02:52:09.222 DEBUG:teuthology.parallel:result is None 2024-03-19T02:52:09.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:08 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1607963993' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-19T02:52:10.023 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:09 smithi033 ceph-mon[31973]: pgmap v340: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-19T02:52:10.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:09 smithi096 ceph-mon[35946]: pgmap v340: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-19T02:52:10.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:09 smithi155 ceph-mon[35804]: pgmap v340: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-19T02:52:12.152 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:11 smithi033 ceph-mon[31973]: pgmap v341: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-19T02:52:12.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:11 smithi096 ceph-mon[35946]: pgmap v341: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-19T02:52:12.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:11 smithi155 ceph-mon[35804]: pgmap v341: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-19T02:52:13.925 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:13 smithi033 ceph-mon[31973]: pgmap v342: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:52:14.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.039+0000 7f7b38041700 1 -- 172.21.15.33:0/1091296688 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b301009d0 msgr2=0x7f7b30100e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:14.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.039+0000 7f7b38041700 1 --2- 172.21.15.33:0/1091296688 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b301009d0 0x7f7b30100e50 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7b20009990 tx=0x7f7b20009ca0 comp rx=0 tx=0).stop 2024-03-19T02:52:14.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.039+0000 7f7b38041700 1 -- 172.21.15.33:0/1091296688 shutdown_connections 2024-03-19T02:52:14.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.039+0000 7f7b38041700 1 --2- 172.21.15.33:0/1091296688 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b30101390 0x7f7b3010d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:14.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.039+0000 7f7b38041700 1 --2- 172.21.15.33:0/1091296688 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b301069d0 0x7f7b30106db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:14.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.039+0000 7f7b38041700 1 --2- 172.21.15.33:0/1091296688 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b301009d0 0x7f7b30100e50 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:14.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.039+0000 7f7b38041700 1 -- 172.21.15.33:0/1091296688 >> 172.21.15.33:0/1091296688 conn(0x7f7b30077810 msgr2=0x7f7b300fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:14.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.039+0000 7f7b38041700 1 -- 172.21.15.33:0/1091296688 shutdown_connections 2024-03-19T02:52:14.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.039+0000 7f7b38041700 1 -- 172.21.15.33:0/1091296688 wait complete. 2024-03-19T02:52:14.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.039+0000 7f7b38041700 1 Processor -- start 2024-03-19T02:52:14.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.039+0000 7f7b38041700 1 -- start start 2024-03-19T02:52:14.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.040+0000 7f7b38041700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b301009d0 0x7f7b30198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:14.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.040+0000 7f7b38041700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b30101390 0x7f7b30198f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:14.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.040+0000 7f7b38041700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b301069d0 0x7f7b3019d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:14.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.040+0000 7f7b38041700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3010e980 con 0x7f7b301069d0 2024-03-19T02:52:14.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.040+0000 7f7b38041700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3010e800 con 0x7f7b30101390 2024-03-19T02:52:14.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.040+0000 7f7b38041700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3010eb00 con 0x7f7b301009d0 2024-03-19T02:52:14.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.040+0000 7f7b365de700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b301069d0 0x7f7b3019d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:14.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.040+0000 7f7b365de700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b301069d0 0x7f7b3019d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40098/0 (socket says 172.21.15.33:40098) 2024-03-19T02:52:14.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.040+0000 7f7b365de700 1 -- 172.21.15.33:0/1650525966 learned_addr learned my addr 172.21.15.33:0/1650525966 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:14.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.040+0000 7f7b355dc700 1 --2- 172.21.15.33:0/1650525966 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b30101390 0x7f7b30198f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:14.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.040+0000 7f7b35ddd700 1 --2- 172.21.15.33:0/1650525966 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b301009d0 0x7f7b30198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:14.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.040+0000 7f7b365de700 1 -- 172.21.15.33:0/1650525966 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b301009d0 msgr2=0x7f7b30198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:14.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.040+0000 7f7b365de700 1 --2- 172.21.15.33:0/1650525966 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b301009d0 0x7f7b30198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:14.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.040+0000 7f7b365de700 1 -- 172.21.15.33:0/1650525966 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b30101390 msgr2=0x7f7b30198f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:14.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.040+0000 7f7b365de700 1 --2- 172.21.15.33:0/1650525966 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b30101390 0x7f7b30198f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:14.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.040+0000 7f7b365de700 1 -- 172.21.15.33:0/1650525966 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b3019da50 con 0x7f7b301069d0 2024-03-19T02:52:14.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.040+0000 7f7b355dc700 1 --2- 172.21.15.33:0/1650525966 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b30101390 0x7f7b30198f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:14.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.040+0000 7f7b365de700 1 --2- 172.21.15.33:0/1650525966 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b301069d0 0x7f7b3019d350 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f7b28007ee0 tx=0x7f7b2800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:14.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.041+0000 7f7b26ffd700 1 -- 172.21.15.33:0/1650525966 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b28004bb0 con 0x7f7b301069d0 2024-03-19T02:52:14.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.041+0000 7f7b26ffd700 1 -- 172.21.15.33:0/1650525966 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7b28004d50 con 0x7f7b301069d0 2024-03-19T02:52:14.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.041+0000 7f7b38041700 1 -- 172.21.15.33:0/1650525966 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b3019dd40 con 0x7f7b301069d0 2024-03-19T02:52:14.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.041+0000 7f7b26ffd700 1 -- 172.21.15.33:0/1650525966 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b280058e0 con 0x7f7b301069d0 2024-03-19T02:52:14.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.041+0000 7f7b38041700 1 -- 172.21.15.33:0/1650525966 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b30110cf0 con 0x7f7b301069d0 2024-03-19T02:52:14.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.042+0000 7f7b38041700 1 -- 172.21.15.33:0/1650525966 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b30068fd0 con 0x7f7b301069d0 2024-03-19T02:52:14.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.043+0000 7f7b26ffd700 1 -- 172.21.15.33:0/1650525966 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f7b2801a020 con 0x7f7b301069d0 2024-03-19T02:52:14.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.043+0000 7f7b26ffd700 1 --2- 172.21.15.33:0/1650525966 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7b1c0a29a0 0x7f7b1c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:14.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.044+0000 7f7b35ddd700 1 --2- 172.21.15.33:0/1650525966 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7b1c0a29a0 0x7f7b1c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:14.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.044+0000 7f7b26ffd700 1 -- 172.21.15.33:0/1650525966 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7f7b280cecf0 con 0x7f7b301069d0 2024-03-19T02:52:14.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.044+0000 7f7b35ddd700 1 --2- 172.21.15.33:0/1650525966 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7b1c0a29a0 0x7f7b1c0a4e60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c00abd0 tx=0x7f7b2c00a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:14.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.047+0000 7f7b26ffd700 1 -- 172.21.15.33:0/1650525966 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7b28010070 con 0x7f7b301069d0 2024-03-19T02:52:14.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:13 smithi096 ceph-mon[35946]: pgmap v342: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:52:14.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.255+0000 7f7b38041700 1 -- 172.21.15.33:0/1650525966 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f7b30110f20 con 0x7f7b301069d0 2024-03-19T02:52:14.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.255+0000 7f7b26ffd700 1 -- 172.21.15.33:0/1650525966 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f7b2809b870 con 0x7f7b301069d0 2024-03-19T02:52:14.257 INFO:teuthology.orchestra.run.smithi033.stdout:167503724605 2024-03-19T02:52:14.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.257+0000 7f7b38041700 1 -- 172.21.15.33:0/1650525966 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7b1c0a29a0 msgr2=0x7f7b1c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:14.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.257+0000 7f7b38041700 1 --2- 172.21.15.33:0/1650525966 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7b1c0a29a0 0x7f7b1c0a4e60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c00abd0 tx=0x7f7b2c00a380 comp rx=0 tx=0).stop 2024-03-19T02:52:14.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.257+0000 7f7b38041700 1 -- 172.21.15.33:0/1650525966 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b301069d0 msgr2=0x7f7b3019d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:14.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.257+0000 7f7b38041700 1 --2- 172.21.15.33:0/1650525966 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b301069d0 0x7f7b3019d350 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f7b28007ee0 tx=0x7f7b2800d3b0 comp rx=0 tx=0).stop 2024-03-19T02:52:14.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.258+0000 7f7b38041700 1 -- 172.21.15.33:0/1650525966 shutdown_connections 2024-03-19T02:52:14.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.258+0000 7f7b38041700 1 --2- 172.21.15.33:0/1650525966 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7b1c0a29a0 0x7f7b1c0a4e60 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:14.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.258+0000 7f7b38041700 1 --2- 172.21.15.33:0/1650525966 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b301069d0 0x7f7b3019d350 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:14.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.258+0000 7f7b38041700 1 --2- 172.21.15.33:0/1650525966 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b301009d0 0x7f7b30198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:14.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.258+0000 7f7b38041700 1 --2- 172.21.15.33:0/1650525966 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b30101390 0x7f7b30198f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:14.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.258+0000 7f7b38041700 1 -- 172.21.15.33:0/1650525966 >> 172.21.15.33:0/1650525966 conn(0x7f7b30077810 msgr2=0x7f7b30103360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:14.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.258+0000 7f7b38041700 1 -- 172.21.15.33:0/1650525966 shutdown_connections 2024-03-19T02:52:14.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:14.258+0000 7f7b38041700 1 -- 172.21.15.33:0/1650525966 wait complete. 2024-03-19T02:52:14.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:13 smithi155 ceph-mon[35804]: pgmap v342: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:52:15.094 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:14 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1650525966' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-19T02:52:15.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:14 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1650525966' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-19T02:52:15.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:14 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1650525966' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-19T02:52:15.892 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:15 smithi033 ceph-mon[31973]: pgmap v343: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:52:16.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:15 smithi096 ceph-mon[35946]: pgmap v343: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:52:16.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:15 smithi155 ceph-mon[35804]: pgmap v343: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:52:16.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.981+0000 7fec8480f700 1 -- 172.21.15.33:0/2498043263 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec80101210 msgr2=0x7fec80101670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:16.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.981+0000 7fec8480f700 1 --2- 172.21.15.33:0/2498043263 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec80101210 0x7fec80101670 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fec70009990 tx=0x7fec70009ca0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.981+0000 7fec8480f700 1 -- 172.21.15.33:0/2498043263 shutdown_connections 2024-03-19T02:52:16.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.981+0000 7fec8480f700 1 --2- 172.21.15.33:0/2498043263 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec80101bb0 0x7fec8010e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.981+0000 7fec8480f700 1 --2- 172.21.15.33:0/2498043263 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec80107250 0x7fec80107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.981+0000 7fec8480f700 1 --2- 172.21.15.33:0/2498043263 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec80101210 0x7fec80101670 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.981+0000 7fec8480f700 1 -- 172.21.15.33:0/2498043263 >> 172.21.15.33:0/2498043263 conn(0x7fec80078140 msgr2=0x7fec80078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:16.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.981+0000 7fec8480f700 1 -- 172.21.15.33:0/2498043263 shutdown_connections 2024-03-19T02:52:16.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.981+0000 7fec8480f700 1 -- 172.21.15.33:0/2498043263 wait complete. 2024-03-19T02:52:16.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.982+0000 7fec8480f700 1 Processor -- start 2024-03-19T02:52:16.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.982+0000 7fec8480f700 1 -- start start 2024-03-19T02:52:16.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.982+0000 7fec8480f700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec80101210 0x7fec80102ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:16.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.982+0000 7fec8480f700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec80101bb0 0x7fec80103400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:16.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.982+0000 7fec8480f700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec80107250 0x7fec80103940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:16.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.982+0000 7fec8480f700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fec80111360 con 0x7fec80101210 2024-03-19T02:52:16.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.982+0000 7fec8480f700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fec801111e0 con 0x7fec80107250 2024-03-19T02:52:16.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.982+0000 7fec8480f700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fec801114e0 con 0x7fec80101bb0 2024-03-19T02:52:16.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.982+0000 7fec7e59c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec80101210 0x7fec80102ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:16.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.982+0000 7fec7e59c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec80101210 0x7fec80102ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40126/0 (socket says 172.21.15.33:40126) 2024-03-19T02:52:16.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.982+0000 7fec7e59c700 1 -- 172.21.15.33:0/2233799407 learned_addr learned my addr 172.21.15.33:0/2233799407 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:16.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.982+0000 7fec7dd9b700 1 --2- 172.21.15.33:0/2233799407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec80101bb0 0x7fec80103400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:16.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.982+0000 7fec7ed9d700 1 --2- 172.21.15.33:0/2233799407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec80107250 0x7fec80103940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:16.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.982+0000 7fec7e59c700 1 -- 172.21.15.33:0/2233799407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec80101bb0 msgr2=0x7fec80103400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:16.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.982+0000 7fec7e59c700 1 --2- 172.21.15.33:0/2233799407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec80101bb0 0x7fec80103400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.982+0000 7fec7e59c700 1 -- 172.21.15.33:0/2233799407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec80107250 msgr2=0x7fec80103940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:16.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.982+0000 7fec7e59c700 1 --2- 172.21.15.33:0/2233799407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec80107250 0x7fec80103940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:16.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.982+0000 7fec7e59c700 1 -- 172.21.15.33:0/2233799407 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec801a6f10 con 0x7fec80101210 2024-03-19T02:52:16.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.983+0000 7fec7e59c700 1 --2- 172.21.15.33:0/2233799407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec80101210 0x7fec80102ec0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fec6800e9c0 tx=0x7fec6800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:16.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.983+0000 7fec777fe700 1 -- 172.21.15.33:0/2233799407 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec6800cbd0 con 0x7fec80101210 2024-03-19T02:52:16.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.983+0000 7fec8480f700 1 -- 172.21.15.33:0/2233799407 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec801a7200 con 0x7fec80101210 2024-03-19T02:52:16.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.983+0000 7fec777fe700 1 -- 172.21.15.33:0/2233799407 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fec6800ee60 con 0x7fec80101210 2024-03-19T02:52:16.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.983+0000 7fec777fe700 1 -- 172.21.15.33:0/2233799407 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec680055b0 con 0x7fec80101210 2024-03-19T02:52:16.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.983+0000 7fec8480f700 1 -- 172.21.15.33:0/2233799407 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec801a7710 con 0x7fec80101210 2024-03-19T02:52:16.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.984+0000 7fec8480f700 1 -- 172.21.15.33:0/2233799407 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec80102900 con 0x7fec80101210 2024-03-19T02:52:16.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.985+0000 7fec777fe700 1 -- 172.21.15.33:0/2233799407 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fec68004750 con 0x7fec80101210 2024-03-19T02:52:16.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.985+0000 7fec777fe700 1 --2- 172.21.15.33:0/2233799407 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fec6c0a29a0 0x7fec6c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:16.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.986+0000 7fec7dd9b700 1 --2- 172.21.15.33:0/2233799407 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fec6c0a29a0 0x7fec6c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:16.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.986+0000 7fec777fe700 1 -- 172.21.15.33:0/2233799407 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7fec680cf130 con 0x7fec80101210 2024-03-19T02:52:16.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.986+0000 7fec7dd9b700 1 --2- 172.21.15.33:0/2233799407 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fec6c0a29a0 0x7fec6c0a4e60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fec7000b2e0 tx=0x7fec70005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:16.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:16.990+0000 7fec777fe700 1 -- 172.21.15.33:0/2233799407 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fec6809bcb0 con 0x7fec80101210 2024-03-19T02:52:17.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:17.200+0000 7fec8480f700 1 -- 172.21.15.33:0/2233799407 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fec801a7a50 con 0x7fec80101210 2024-03-19T02:52:17.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:17.201+0000 7fec777fe700 1 -- 172.21.15.33:0/2233799407 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fec6809b5b0 con 0x7fec80101210 2024-03-19T02:52:17.202 INFO:teuthology.orchestra.run.smithi033.stdout:201863462967 2024-03-19T02:52:17.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:17.203+0000 7fec8480f700 1 -- 172.21.15.33:0/2233799407 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fec6c0a29a0 msgr2=0x7fec6c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:17.203+0000 7fec8480f700 1 --2- 172.21.15.33:0/2233799407 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fec6c0a29a0 0x7fec6c0a4e60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fec7000b2e0 tx=0x7fec70005960 comp rx=0 tx=0).stop 2024-03-19T02:52:17.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:17.203+0000 7fec8480f700 1 -- 172.21.15.33:0/2233799407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec80101210 msgr2=0x7fec80102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:17.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:17.203+0000 7fec8480f700 1 --2- 172.21.15.33:0/2233799407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec80101210 0x7fec80102ec0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fec6800e9c0 tx=0x7fec6800ecd0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:17.203+0000 7fec8480f700 1 -- 172.21.15.33:0/2233799407 shutdown_connections 2024-03-19T02:52:17.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:17.203+0000 7fec8480f700 1 --2- 172.21.15.33:0/2233799407 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fec6c0a29a0 0x7fec6c0a4e60 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:17.203+0000 7fec8480f700 1 --2- 172.21.15.33:0/2233799407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec80107250 0x7fec80103940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:17.203+0000 7fec8480f700 1 --2- 172.21.15.33:0/2233799407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec80101210 0x7fec80102ec0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:17.203+0000 7fec8480f700 1 --2- 172.21.15.33:0/2233799407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec80101bb0 0x7fec80103400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:17.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:17.203+0000 7fec8480f700 1 -- 172.21.15.33:0/2233799407 >> 172.21.15.33:0/2233799407 conn(0x7fec80078140 msgr2=0x7fec8006a220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:17.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:17.203+0000 7fec8480f700 1 -- 172.21.15.33:0/2233799407 shutdown_connections 2024-03-19T02:52:17.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:17.203+0000 7fec8480f700 1 -- 172.21.15.33:0/2233799407 wait complete. 2024-03-19T02:52:17.316 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:17 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2233799407' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-19T02:52:17.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:17 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2233799407' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-19T02:52:17.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:17 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2233799407' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-19T02:52:18.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:18 smithi155 ceph-mon[35804]: pgmap v344: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:52:18.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:18 smithi033 ceph-mon[31973]: pgmap v344: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:52:18.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:18 smithi096 ceph-mon[35946]: pgmap v344: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:52:18.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.883+0000 7f983a2a7700 1 -- 172.21.15.33:0/2663904987 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f983406aa60 msgr2=0x7f983406aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:18.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.883+0000 7f983a2a7700 1 --2- 172.21.15.33:0/2663904987 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f983406aa60 0x7f983406aee0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f98280099c0 tx=0x7f9828009cd0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.883+0000 7f983a2a7700 1 -- 172.21.15.33:0/2663904987 shutdown_connections 2024-03-19T02:52:18.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.883+0000 7f983a2a7700 1 --2- 172.21.15.33:0/2663904987 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f983406b420 0x7f983410f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.883+0000 7f983a2a7700 1 --2- 172.21.15.33:0/2663904987 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9834108340 0x7f9834108720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.883+0000 7f983a2a7700 1 --2- 172.21.15.33:0/2663904987 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f983406aa60 0x7f983406aee0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.883+0000 7f983a2a7700 1 -- 172.21.15.33:0/2663904987 >> 172.21.15.33:0/2663904987 conn(0x7f9834078220 msgr2=0x7f9834078630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:18.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.883+0000 7f983a2a7700 1 -- 172.21.15.33:0/2663904987 shutdown_connections 2024-03-19T02:52:18.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.883+0000 7f983a2a7700 1 -- 172.21.15.33:0/2663904987 wait complete. 2024-03-19T02:52:18.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.883+0000 7f983a2a7700 1 Processor -- start 2024-03-19T02:52:18.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.883+0000 7f983a2a7700 1 -- start start 2024-03-19T02:52:18.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.884+0000 7f983a2a7700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f983406aa60 0x7f983419e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.884+0000 7f983a2a7700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f983406b420 0x7f983419ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.884+0000 7f983a2a7700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9834108340 0x7f9834198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.884+0000 7f983a2a7700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f98341124e0 con 0x7f983406aa60 2024-03-19T02:52:18.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.884+0000 7f983a2a7700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9834112360 con 0x7f9834108340 2024-03-19T02:52:18.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.884+0000 7f983a2a7700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9834112660 con 0x7f983406b420 2024-03-19T02:52:18.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.884+0000 7f9833fff700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f983406aa60 0x7f983419e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.884+0000 7f9833fff700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f983406aa60 0x7f983419e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40150/0 (socket says 172.21.15.33:40150) 2024-03-19T02:52:18.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.884+0000 7f9833fff700 1 -- 172.21.15.33:0/3050750261 learned_addr learned my addr 172.21.15.33:0/3050750261 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:18.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.884+0000 7f9838844700 1 --2- 172.21.15.33:0/3050750261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9834108340 0x7f9834198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.884+0000 7f98337fe700 1 --2- 172.21.15.33:0/3050750261 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f983406b420 0x7f983419ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.884+0000 7f9833fff700 1 -- 172.21.15.33:0/3050750261 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f983406b420 msgr2=0x7f983419ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:18.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.884+0000 7f9833fff700 1 --2- 172.21.15.33:0/3050750261 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f983406b420 0x7f983419ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.884+0000 7f9833fff700 1 -- 172.21.15.33:0/3050750261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9834108340 msgr2=0x7f9834198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:18.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.884+0000 7f9833fff700 1 --2- 172.21.15.33:0/3050750261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9834108340 0x7f9834198800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:18.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.884+0000 7f9833fff700 1 -- 172.21.15.33:0/3050750261 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9834198fe0 con 0x7f983406aa60 2024-03-19T02:52:18.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.884+0000 7f9838844700 1 --2- 172.21.15.33:0/3050750261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9834108340 0x7f9834198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:18.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.884+0000 7f9833fff700 1 --2- 172.21.15.33:0/3050750261 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f983406aa60 0x7f983419e680 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f982400e9c0 tx=0x7f982400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:18.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.885+0000 7f98317fa700 1 -- 172.21.15.33:0/3050750261 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f982400cbd0 con 0x7f983406aa60 2024-03-19T02:52:18.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.885+0000 7f983a2a7700 1 -- 172.21.15.33:0/3050750261 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98341992d0 con 0x7f983406aa60 2024-03-19T02:52:18.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.885+0000 7f98317fa700 1 -- 172.21.15.33:0/3050750261 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f982400ee60 con 0x7f983406aa60 2024-03-19T02:52:18.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.885+0000 7f98317fa700 1 -- 172.21.15.33:0/3050750261 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98240055b0 con 0x7f983406aa60 2024-03-19T02:52:18.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.885+0000 7f983a2a7700 1 -- 172.21.15.33:0/3050750261 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98341a54e0 con 0x7f983406aa60 2024-03-19T02:52:18.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.886+0000 7f983a2a7700 1 -- 172.21.15.33:0/3050750261 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98341039f0 con 0x7f983406aa60 2024-03-19T02:52:18.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.887+0000 7f98317fa700 1 -- 172.21.15.33:0/3050750261 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f9824004750 con 0x7f983406aa60 2024-03-19T02:52:18.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.888+0000 7f98317fa700 1 --2- 172.21.15.33:0/3050750261 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f981c0a2920 0x7f981c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:18.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.888+0000 7f98337fe700 1 --2- 172.21.15.33:0/3050750261 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f981c0a2920 0x7f981c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:18.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.888+0000 7f98317fa700 1 -- 172.21.15.33:0/3050750261 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7f98240cf360 con 0x7f983406aa60 2024-03-19T02:52:18.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.888+0000 7f98337fe700 1 --2- 172.21.15.33:0/3050750261 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f981c0a2920 0x7f981c0a4de0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9834078800 tx=0x7f9828005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:18.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:18.891+0000 7f98317fa700 1 -- 172.21.15.33:0/3050750261 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9824053740 con 0x7f983406aa60 2024-03-19T02:52:19.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.102+0000 7f983a2a7700 1 -- 172.21.15.33:0/3050750261 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f9834199d30 con 0x7f983406aa60 2024-03-19T02:52:19.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.103+0000 7f98317fa700 1 -- 172.21.15.33:0/3050750261 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f982409bf40 con 0x7f983406aa60 2024-03-19T02:52:19.104 INFO:teuthology.orchestra.run.smithi033.stdout:317827579939 2024-03-19T02:52:19.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.105+0000 7f983a2a7700 1 -- 172.21.15.33:0/3050750261 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f981c0a2920 msgr2=0x7f981c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:19.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.105+0000 7f983a2a7700 1 --2- 172.21.15.33:0/3050750261 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f981c0a2920 0x7f981c0a4de0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9834078800 tx=0x7f9828005fb0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.105+0000 7f983a2a7700 1 -- 172.21.15.33:0/3050750261 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f983406aa60 msgr2=0x7f983419e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:19.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.105+0000 7f983a2a7700 1 --2- 172.21.15.33:0/3050750261 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f983406aa60 0x7f983419e680 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f982400e9c0 tx=0x7f982400ecd0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.105+0000 7f983a2a7700 1 -- 172.21.15.33:0/3050750261 shutdown_connections 2024-03-19T02:52:19.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.105+0000 7f983a2a7700 1 --2- 172.21.15.33:0/3050750261 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f981c0a2920 0x7f981c0a4de0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.105+0000 7f983a2a7700 1 --2- 172.21.15.33:0/3050750261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9834108340 0x7f9834198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.105+0000 7f983a2a7700 1 --2- 172.21.15.33:0/3050750261 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f983406aa60 0x7f983419e680 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.105+0000 7f983a2a7700 1 --2- 172.21.15.33:0/3050750261 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f983406b420 0x7f983419ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:19.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.105+0000 7f983a2a7700 1 -- 172.21.15.33:0/3050750261 >> 172.21.15.33:0/3050750261 conn(0x7f9834078220 msgr2=0x7f98340ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:19.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.105+0000 7f983a2a7700 1 -- 172.21.15.33:0/3050750261 shutdown_connections 2024-03-19T02:52:19.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.105+0000 7f983a2a7700 1 -- 172.21.15.33:0/3050750261 wait complete. 2024-03-19T02:52:19.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:19 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3050750261' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-19T02:52:19.527 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:19 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3050750261' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-19T02:52:19.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:19 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3050750261' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-19T02:52:19.837 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462961 got 201863462967 for osd.6 2024-03-19T02:52:19.837 DEBUG:teuthology.parallel:result is None 2024-03-19T02:52:20.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.310+0000 7f5e54bc2700 1 -- 172.21.15.33:0/4121159871 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e50107690 msgr2=0x7f5e5010f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:20.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.310+0000 7f5e54bc2700 1 --2- 172.21.15.33:0/4121159871 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e50107690 0x7f5e5010f210 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f5e480099c0 tx=0x7f5e48009cd0 comp rx=0 tx=0).stop 2024-03-19T02:52:20.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.310+0000 7f5e54bc2700 1 -- 172.21.15.33:0/4121159871 shutdown_connections 2024-03-19T02:52:20.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.310+0000 7f5e54bc2700 1 --2- 172.21.15.33:0/4121159871 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e50107690 0x7f5e5010f210 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:20.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.310+0000 7f5e54bc2700 1 --2- 172.21.15.33:0/4121159871 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e5006aab0 0x7f5e5006ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:20.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.310+0000 7f5e54bc2700 1 --2- 172.21.15.33:0/4121159871 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e5006b460 0x7f5e50106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:20.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.310+0000 7f5e54bc2700 1 -- 172.21.15.33:0/4121159871 >> 172.21.15.33:0/4121159871 conn(0x7f5e50077260 msgr2=0x7f5e50077670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:20.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.310+0000 7f5e54bc2700 1 -- 172.21.15.33:0/4121159871 shutdown_connections 2024-03-19T02:52:20.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.310+0000 7f5e54bc2700 1 -- 172.21.15.33:0/4121159871 wait complete. 2024-03-19T02:52:20.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.311+0000 7f5e54bc2700 1 Processor -- start 2024-03-19T02:52:20.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.311+0000 7f5e54bc2700 1 -- start start 2024-03-19T02:52:20.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.311+0000 7f5e54bc2700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e5006aab0 0x7f5e5019e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:20.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.311+0000 7f5e54bc2700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e5006b460 0x7f5e5019ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:20.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.311+0000 7f5e54bc2700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e50107690 0x7f5e50198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:20.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.311+0000 7f5e54bc2700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5e50112500 con 0x7f5e50107690 2024-03-19T02:52:20.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.311+0000 7f5e54bc2700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5e50112380 con 0x7f5e5006b460 2024-03-19T02:52:20.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.311+0000 7f5e54bc2700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5e50112680 con 0x7f5e5006aab0 2024-03-19T02:52:20.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.311+0000 7f5e4ed9d700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e50107690 0x7f5e50198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:20.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.311+0000 7f5e4dd9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e5006b460 0x7f5e5019ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:20.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.311+0000 7f5e4e59c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e5006aab0 0x7f5e5019e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:20.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.311+0000 7f5e4dd9b700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e5006b460 0x7f5e5019ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:56826/0 (socket says 172.21.15.33:56826) 2024-03-19T02:52:20.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.311+0000 7f5e4dd9b700 1 -- 172.21.15.33:0/261510437 learned_addr learned my addr 172.21.15.33:0/261510437 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:20.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.312+0000 7f5e4dd9b700 1 -- 172.21.15.33:0/261510437 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e5006aab0 msgr2=0x7f5e5019e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:20.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.312+0000 7f5e4dd9b700 1 --2- 172.21.15.33:0/261510437 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e5006aab0 0x7f5e5019e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:20.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.312+0000 7f5e4dd9b700 1 -- 172.21.15.33:0/261510437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e50107690 msgr2=0x7f5e50198880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:20.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.312+0000 7f5e4dd9b700 1 --2- 172.21.15.33:0/261510437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e50107690 0x7f5e50198880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:20.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.312+0000 7f5e4dd9b700 1 -- 172.21.15.33:0/261510437 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e50198fe0 con 0x7f5e5006b460 2024-03-19T02:52:20.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.312+0000 7f5e4ed9d700 1 --2- 172.21.15.33:0/261510437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e50107690 0x7f5e50198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:20.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.312+0000 7f5e4e59c700 1 --2- 172.21.15.33:0/261510437 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e5006aab0 0x7f5e5019e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:20.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.312+0000 7f5e4dd9b700 1 --2- 172.21.15.33:0/261510437 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e5006b460 0x7f5e5019ec00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5e38009fd0 tx=0x7f5e3800ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:20.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.312+0000 7f5e477fe700 1 -- 172.21.15.33:0/261510437 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3800eea0 con 0x7f5e5006b460 2024-03-19T02:52:20.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.312+0000 7f5e477fe700 1 -- 172.21.15.33:0/261510437 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e380044d0 con 0x7f5e5006b460 2024-03-19T02:52:20.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.312+0000 7f5e54bc2700 1 -- 172.21.15.33:0/261510437 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e501992d0 con 0x7f5e5006b460 2024-03-19T02:52:20.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.312+0000 7f5e477fe700 1 -- 172.21.15.33:0/261510437 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e38003dd0 con 0x7f5e5006b460 2024-03-19T02:52:20.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.312+0000 7f5e54bc2700 1 -- 172.21.15.33:0/261510437 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e501a5590 con 0x7f5e5006b460 2024-03-19T02:52:20.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.313+0000 7f5e54bc2700 1 -- 172.21.15.33:0/261510437 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e5010a840 con 0x7f5e5006b460 2024-03-19T02:52:20.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.314+0000 7f5e477fe700 1 -- 172.21.15.33:0/261510437 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f5e38004000 con 0x7f5e5006b460 2024-03-19T02:52:20.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.314+0000 7f5e477fe700 1 --2- 172.21.15.33:0/261510437 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5e3c0a2920 0x7f5e3c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:20.139 INFO:teuthology.orchestra.run.smithi033.stdout:34359738455 2024-03-19T02:52:20.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.314+0000 7f5e4e59c700 1 --2- 172.21.15.33:0/261510437 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5e3c0a2920 0x7f5e3c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:20.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.314+0000 7f5e477fe700 1 -- 172.21.15.33:0/261510437 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7f5e380cefa0 con 0x7f5e5006b460 2024-03-19T02:52:20.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.315+0000 7f5e4e59c700 1 --2- 172.21.15.33:0/261510437 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5e3c0a2920 0x7f5e3c0a4de0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5e501999c0 tx=0x7f5e4000b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:20.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.319+0000 7f5e477fe700 1 -- 172.21.15.33:0/261510437 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5e380042f0 con 0x7f5e5006b460 2024-03-19T02:52:20.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.526+0000 7f5e54bc2700 1 -- 172.21.15.33:0/261510437 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f5e50199db0 con 0x7f5e5006b460 2024-03-19T02:52:20.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.526+0000 7f5e477fe700 1 -- 172.21.15.33:0/261510437 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5e3809bb80 con 0x7f5e5006b460 2024-03-19T02:52:20.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.528+0000 7f5e54bc2700 1 -- 172.21.15.33:0/261510437 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5e3c0a2920 msgr2=0x7f5e3c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:20.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.528+0000 7f5e54bc2700 1 --2- 172.21.15.33:0/261510437 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5e3c0a2920 0x7f5e3c0a4de0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5e501999c0 tx=0x7f5e4000b040 comp rx=0 tx=0).stop 2024-03-19T02:52:20.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.528+0000 7f5e54bc2700 1 -- 172.21.15.33:0/261510437 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e5006b460 msgr2=0x7f5e5019ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:20.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.528+0000 7f5e54bc2700 1 --2- 172.21.15.33:0/261510437 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e5006b460 0x7f5e5019ec00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5e38009fd0 tx=0x7f5e3800ed10 comp rx=0 tx=0).stop 2024-03-19T02:52:20.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.528+0000 7f5e54bc2700 1 -- 172.21.15.33:0/261510437 shutdown_connections 2024-03-19T02:52:20.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.528+0000 7f5e54bc2700 1 --2- 172.21.15.33:0/261510437 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5e3c0a2920 0x7f5e3c0a4de0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:20.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.528+0000 7f5e54bc2700 1 --2- 172.21.15.33:0/261510437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e50107690 0x7f5e50198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:20.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.528+0000 7f5e54bc2700 1 --2- 172.21.15.33:0/261510437 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e5006aab0 0x7f5e5019e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:20.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.528+0000 7f5e54bc2700 1 --2- 172.21.15.33:0/261510437 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e5006b460 0x7f5e5019ec00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:20.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.528+0000 7f5e54bc2700 1 -- 172.21.15.33:0/261510437 >> 172.21.15.33:0/261510437 conn(0x7f5e50077260 msgr2=0x7f5e500ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:20.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.528+0000 7f5e54bc2700 1 -- 172.21.15.33:0/261510437 shutdown_connections 2024-03-19T02:52:20.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:19.528+0000 7f5e54bc2700 1 -- 172.21.15.33:0/261510437 wait complete. 2024-03-19T02:52:20.438 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:20 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/261510437' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-19T02:52:20.438 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:20 smithi033 ceph-mon[31973]: pgmap v345: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-19T02:52:20.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:20 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/261510437' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-19T02:52:20.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:20 smithi155 ceph-mon[35804]: pgmap v345: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-19T02:52:20.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.539+0000 7fec47c9e700 1 -- 172.21.15.33:0/3704303833 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec40101210 msgr2=0x7fec40101670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:20.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.539+0000 7fec47c9e700 1 --2- 172.21.15.33:0/3704303833 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec40101210 0x7fec40101670 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fec3c009990 tx=0x7fec3c009ca0 comp rx=0 tx=0).stop 2024-03-19T02:52:20.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.539+0000 7fec47c9e700 1 -- 172.21.15.33:0/3704303833 shutdown_connections 2024-03-19T02:52:20.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.539+0000 7fec47c9e700 1 --2- 172.21.15.33:0/3704303833 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec40101bb0 0x7fec4010e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:20.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.539+0000 7fec47c9e700 1 --2- 172.21.15.33:0/3704303833 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec40107250 0x7fec40107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:20.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.539+0000 7fec47c9e700 1 --2- 172.21.15.33:0/3704303833 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec40101210 0x7fec40101670 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:20.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.539+0000 7fec47c9e700 1 -- 172.21.15.33:0/3704303833 >> 172.21.15.33:0/3704303833 conn(0x7fec40078140 msgr2=0x7fec40078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:20.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.539+0000 7fec47c9e700 1 -- 172.21.15.33:0/3704303833 shutdown_connections 2024-03-19T02:52:20.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.539+0000 7fec47c9e700 1 -- 172.21.15.33:0/3704303833 wait complete. 2024-03-19T02:52:20.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.540+0000 7fec47c9e700 1 Processor -- start 2024-03-19T02:52:20.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.540+0000 7fec47c9e700 1 -- start start 2024-03-19T02:52:20.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.540+0000 7fec47c9e700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec40101210 0x7fec4019e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:20.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.540+0000 7fec47c9e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec40101bb0 0x7fec4019eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:20.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.540+0000 7fec47c9e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec40107250 0x7fec40198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:20.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.540+0000 7fec47c9e700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fec40111360 con 0x7fec40101210 2024-03-19T02:52:20.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.540+0000 7fec47c9e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fec401111e0 con 0x7fec40101bb0 2024-03-19T02:52:20.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.540+0000 7fec47c9e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fec401114e0 con 0x7fec40107250 2024-03-19T02:52:20.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.540+0000 7fec45a3a700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec40101210 0x7fec4019e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:20.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.540+0000 7fec45a3a700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec40101210 0x7fec4019e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40176/0 (socket says 172.21.15.33:40176) 2024-03-19T02:52:20.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.540+0000 7fec45a3a700 1 -- 172.21.15.33:0/472103391 learned_addr learned my addr 172.21.15.33:0/472103391 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:20.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.540+0000 7fec45239700 1 --2- 172.21.15.33:0/472103391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec40101bb0 0x7fec4019eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:20.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.540+0000 7fec4623b700 1 --2- 172.21.15.33:0/472103391 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec40107250 0x7fec40198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:20.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.540+0000 7fec45a3a700 1 -- 172.21.15.33:0/472103391 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec40107250 msgr2=0x7fec40198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:20.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.540+0000 7fec45a3a700 1 --2- 172.21.15.33:0/472103391 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec40107250 0x7fec40198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:20.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.540+0000 7fec45a3a700 1 -- 172.21.15.33:0/472103391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec40101bb0 msgr2=0x7fec4019eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:20.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.540+0000 7fec45a3a700 1 --2- 172.21.15.33:0/472103391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec40101bb0 0x7fec4019eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:20.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.540+0000 7fec45a3a700 1 -- 172.21.15.33:0/472103391 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec40198f60 con 0x7fec40101210 2024-03-19T02:52:20.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.541+0000 7fec45a3a700 1 --2- 172.21.15.33:0/472103391 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec40101210 0x7fec4019e600 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fec3000e9f0 tx=0x7fec3000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:20.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.541+0000 7fec36ffd700 1 -- 172.21.15.33:0/472103391 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec30004a30 con 0x7fec40101210 2024-03-19T02:52:20.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.541+0000 7fec36ffd700 1 -- 172.21.15.33:0/472103391 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fec300054d0 con 0x7fec40101210 2024-03-19T02:52:20.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.541+0000 7fec47c9e700 1 -- 172.21.15.33:0/472103391 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec40199250 con 0x7fec40101210 2024-03-19T02:52:20.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.541+0000 7fec36ffd700 1 -- 172.21.15.33:0/472103391 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3000cd00 con 0x7fec40101210 2024-03-19T02:52:20.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.541+0000 7fec47c9e700 1 -- 172.21.15.33:0/472103391 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec401a5460 con 0x7fec40101210 2024-03-19T02:52:20.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.542+0000 7fec47c9e700 1 -- 172.21.15.33:0/472103391 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec40102900 con 0x7fec40101210 2024-03-19T02:52:20.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.543+0000 7fec36ffd700 1 -- 172.21.15.33:0/472103391 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fec30004ed0 con 0x7fec40101210 2024-03-19T02:52:20.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.544+0000 7fec36ffd700 1 --2- 172.21.15.33:0/472103391 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fec2c0a2920 0x7fec2c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:20.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.544+0000 7fec45239700 1 --2- 172.21.15.33:0/472103391 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fec2c0a2920 0x7fec2c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:20.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.544+0000 7fec36ffd700 1 -- 172.21.15.33:0/472103391 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7fec300cf310 con 0x7fec40101210 2024-03-19T02:52:20.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.545+0000 7fec45239700 1 --2- 172.21.15.33:0/472103391 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fec2c0a2920 0x7fec2c0a4de0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fec3c00b2e0 tx=0x7fec3c005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:20.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.547+0000 7fec36ffd700 1 -- 172.21.15.33:0/472103391 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fec3009bf80 con 0x7fec40101210 2024-03-19T02:52:20.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:20 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/261510437' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-19T02:52:20.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:20 smithi096 ceph-mon[35946]: pgmap v345: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-19T02:52:20.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.733+0000 7f030ddef700 1 -- 172.21.15.33:0/1422782206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f03081014a0 msgr2=0x7f0308101880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:20.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.733+0000 7f030ddef700 1 --2- 172.21.15.33:0/1422782206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f03081014a0 0x7f0308101880 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f02f80099c0 tx=0x7f02f8009cd0 comp rx=0 tx=0).stop 2024-03-19T02:52:20.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.733+0000 7f030ddef700 1 -- 172.21.15.33:0/1422782206 shutdown_connections 2024-03-19T02:52:20.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.733+0000 7f030ddef700 1 --2- 172.21.15.33:0/1422782206 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03081064f0 0x7f030810e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:20.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.733+0000 7f030ddef700 1 --2- 172.21.15.33:0/1422782206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f03081014a0 0x7f0308101880 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:20.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.733+0000 7f030ddef700 1 --2- 172.21.15.33:0/1422782206 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0308101e50 0x7f0308105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:20.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.733+0000 7f030ddef700 1 -- 172.21.15.33:0/1422782206 >> 172.21.15.33:0/1422782206 conn(0x7f0308077260 msgr2=0x7f0308077670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:20.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.733+0000 7f030ddef700 1 -- 172.21.15.33:0/1422782206 shutdown_connections 2024-03-19T02:52:20.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.734+0000 7f030ddef700 1 -- 172.21.15.33:0/1422782206 wait complete. 2024-03-19T02:52:20.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.734+0000 7f030ddef700 1 Processor -- start 2024-03-19T02:52:20.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.735+0000 7f030ddef700 1 -- start start 2024-03-19T02:52:20.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.735+0000 7f030ddef700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f03081014a0 0x7f0308199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:20.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.735+0000 7f030ddef700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0308101e50 0x7f0308199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:20.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.735+0000 7f030ddef700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03081064f0 0x7f03081a4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:20.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.735+0000 7f030ddef700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f03081113a0 con 0x7f03081014a0 2024-03-19T02:52:20.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.735+0000 7f030ddef700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0308111220 con 0x7f0308101e50 2024-03-19T02:52:20.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.735+0000 7f030ddef700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f0308111520 con 0x7f03081064f0 2024-03-19T02:52:20.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.735+0000 7f03077fe700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f03081014a0 0x7f0308199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:20.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.735+0000 7f0306ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0308101e50 0x7f0308199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:20.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.735+0000 7f03077fe700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f03081014a0 0x7f0308199800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:40196/0 (socket says 172.21.15.33:40196) 2024-03-19T02:52:20.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.735+0000 7f03077fe700 1 -- 172.21.15.33:0/3054548133 learned_addr learned my addr 172.21.15.33:0/3054548133 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:20.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.735+0000 7f03077fe700 1 -- 172.21.15.33:0/3054548133 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03081064f0 msgr2=0x7f03081a4880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T02:52:20.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.736+0000 7f03077fe700 1 --2- 172.21.15.33:0/3054548133 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03081064f0 0x7f03081a4880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:20.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.736+0000 7f03077fe700 1 -- 172.21.15.33:0/3054548133 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0308101e50 msgr2=0x7f0308199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:20.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.736+0000 7f03077fe700 1 --2- 172.21.15.33:0/3054548133 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0308101e50 0x7f0308199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:20.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.736+0000 7f03077fe700 1 -- 172.21.15.33:0/3054548133 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03081a4dc0 con 0x7f03081014a0 2024-03-19T02:52:20.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.736+0000 7f0306ffd700 1 --2- 172.21.15.33:0/3054548133 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0308101e50 0x7f0308199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:20.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.736+0000 7f03077fe700 1 --2- 172.21.15.33:0/3054548133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f03081014a0 0x7f0308199800 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f02f8006010 tx=0x7f02f8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:20.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.736+0000 7f0304ff9700 1 -- 172.21.15.33:0/3054548133 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f8026070 con 0x7f03081014a0 2024-03-19T02:52:20.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.736+0000 7f030ddef700 1 -- 172.21.15.33:0/3054548133 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03081975f0 con 0x7f03081014a0 2024-03-19T02:52:20.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.736+0000 7f0304ff9700 1 -- 172.21.15.33:0/3054548133 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f02f8004660 con 0x7f03081014a0 2024-03-19T02:52:20.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.736+0000 7f030ddef700 1 -- 172.21.15.33:0/3054548133 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0308197b20 con 0x7f03081014a0 2024-03-19T02:52:20.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.736+0000 7f0304ff9700 1 -- 172.21.15.33:0/3054548133 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f800fb50 con 0x7f03081014a0 2024-03-19T02:52:20.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.737+0000 7f030ddef700 1 -- 172.21.15.33:0/3054548133 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0308068fd0 con 0x7f03081014a0 2024-03-19T02:52:20.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.739+0000 7f0304ff9700 1 -- 172.21.15.33:0/3054548133 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f02f8029070 con 0x7f03081014a0 2024-03-19T02:52:20.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.739+0000 7f0304ff9700 1 --2- 172.21.15.33:0/3054548133 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f02f00a29a0 0x7f02f00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:20.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.739+0000 7f0306ffd700 1 --2- 172.21.15.33:0/3054548133 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f02f00a29a0 0x7f02f00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:20.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.739+0000 7f0304ff9700 1 -- 172.21.15.33:0/3054548133 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7f02f80cfa10 con 0x7f03081014a0 2024-03-19T02:52:20.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.740+0000 7f0306ffd700 1 --2- 172.21.15.33:0/3054548133 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f02f00a29a0 0x7f02f00a4e60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0308100e70 tx=0x7f02fc012040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:20.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.747+0000 7f0304ff9700 1 -- 172.21.15.33:0/3054548133 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f02f8013070 con 0x7f03081014a0 2024-03-19T02:52:20.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.764+0000 7fec47c9e700 1 -- 172.21.15.33:0/472103391 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fec401a56f0 con 0x7fec40101210 2024-03-19T02:52:20.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.764+0000 7fec36ffd700 1 -- 172.21.15.33:0/472103391 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fec3009b880 con 0x7fec40101210 2024-03-19T02:52:20.765 INFO:teuthology.orchestra.run.smithi033.stdout:111669149768 2024-03-19T02:52:20.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.766+0000 7fec47c9e700 1 -- 172.21.15.33:0/472103391 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fec2c0a2920 msgr2=0x7fec2c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:20.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.766+0000 7fec47c9e700 1 --2- 172.21.15.33:0/472103391 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fec2c0a2920 0x7fec2c0a4de0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fec3c00b2e0 tx=0x7fec3c005960 comp rx=0 tx=0).stop 2024-03-19T02:52:20.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.766+0000 7fec47c9e700 1 -- 172.21.15.33:0/472103391 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec40101210 msgr2=0x7fec4019e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:20.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.766+0000 7fec47c9e700 1 --2- 172.21.15.33:0/472103391 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec40101210 0x7fec4019e600 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fec3000e9f0 tx=0x7fec3000ed00 comp rx=0 tx=0).stop 2024-03-19T02:52:20.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.766+0000 7fec47c9e700 1 -- 172.21.15.33:0/472103391 shutdown_connections 2024-03-19T02:52:20.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.766+0000 7fec47c9e700 1 --2- 172.21.15.33:0/472103391 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fec2c0a2920 0x7fec2c0a4de0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:20.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.766+0000 7fec47c9e700 1 --2- 172.21.15.33:0/472103391 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fec40107250 0x7fec40198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:20.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.766+0000 7fec47c9e700 1 --2- 172.21.15.33:0/472103391 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fec40101210 0x7fec4019e600 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:20.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.766+0000 7fec47c9e700 1 --2- 172.21.15.33:0/472103391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec40101bb0 0x7fec4019eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:20.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.766+0000 7fec47c9e700 1 -- 172.21.15.33:0/472103391 >> 172.21.15.33:0/472103391 conn(0x7fec40078140 msgr2=0x7fec401005b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:20.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.766+0000 7fec47c9e700 1 -- 172.21.15.33:0/472103391 shutdown_connections 2024-03-19T02:52:20.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.766+0000 7fec47c9e700 1 -- 172.21.15.33:0/472103391 wait complete. 2024-03-19T02:52:20.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.964+0000 7f030ddef700 1 -- 172.21.15.33:0/3054548133 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f030802dbb0 con 0x7f03081014a0 2024-03-19T02:52:20.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.965+0000 7f0304ff9700 1 -- 172.21.15.33:0/3054548133 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f02f809c680 con 0x7f03081014a0 2024-03-19T02:52:20.966 INFO:teuthology.orchestra.run.smithi033.stdout:55834574930 2024-03-19T02:52:20.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.967+0000 7f030ddef700 1 -- 172.21.15.33:0/3054548133 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f02f00a29a0 msgr2=0x7f02f00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:20.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.967+0000 7f030ddef700 1 --2- 172.21.15.33:0/3054548133 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f02f00a29a0 0x7f02f00a4e60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0308100e70 tx=0x7f02fc012040 comp rx=0 tx=0).stop 2024-03-19T02:52:20.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.967+0000 7f030ddef700 1 -- 172.21.15.33:0/3054548133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f03081014a0 msgr2=0x7f0308199800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:20.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.967+0000 7f030ddef700 1 --2- 172.21.15.33:0/3054548133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f03081014a0 0x7f0308199800 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f02f8006010 tx=0x7f02f8004270 comp rx=0 tx=0).stop 2024-03-19T02:52:20.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.967+0000 7f030ddef700 1 -- 172.21.15.33:0/3054548133 shutdown_connections 2024-03-19T02:52:20.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.967+0000 7f030ddef700 1 --2- 172.21.15.33:0/3054548133 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f02f00a29a0 0x7f02f00a4e60 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:20.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.967+0000 7f030ddef700 1 --2- 172.21.15.33:0/3054548133 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03081064f0 0x7f03081a4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:20.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.967+0000 7f030ddef700 1 --2- 172.21.15.33:0/3054548133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f03081014a0 0x7f0308199800 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:20.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.967+0000 7f030ddef700 1 --2- 172.21.15.33:0/3054548133 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0308101e50 0x7f0308199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:20.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.967+0000 7f030ddef700 1 -- 172.21.15.33:0/3054548133 >> 172.21.15.33:0/3054548133 conn(0x7f0308077260 msgr2=0x7f03080ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:20.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.967+0000 7f030ddef700 1 -- 172.21.15.33:0/3054548133 shutdown_connections 2024-03-19T02:52:20.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:20.967+0000 7f030ddef700 1 -- 172.21.15.33:0/3054548133 wait complete. 2024-03-19T02:52:21.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:21 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/472103391' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-19T02:52:21.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:21 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3054548133' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-19T02:52:21.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:21 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/472103391' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-19T02:52:21.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:21 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3054548133' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-19T02:52:21.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:21 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/472103391' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-19T02:52:21.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:21 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3054548133' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-19T02:52:22.165 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149762 got 111669149768 for osd.3 2024-03-19T02:52:22.165 DEBUG:teuthology.parallel:result is None 2024-03-19T02:52:22.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:22 smithi033 ceph-mon[31973]: pgmap v346: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-19T02:52:22.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.663+0000 7f9a91071700 1 -- 172.21.15.33:0/3524861110 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a8c10ca00 msgr2=0x7f9a8c10ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:22.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.663+0000 7f9a91071700 1 --2- 172.21.15.33:0/3524861110 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a8c10ca00 0x7f9a8c10ee30 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9a8400b0a0 tx=0x7f9a8400b3b0 comp rx=0 tx=0).stop 2024-03-19T02:52:22.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.663+0000 7f9a91071700 1 -- 172.21.15.33:0/3524861110 shutdown_connections 2024-03-19T02:52:22.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.663+0000 7f9a91071700 1 --2- 172.21.15.33:0/3524861110 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a8c10ca00 0x7f9a8c10ee30 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:22.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.663+0000 7f9a91071700 1 --2- 172.21.15.33:0/3524861110 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a8c103620 0x7f9a8c103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:22.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.663+0000 7f9a91071700 1 --2- 172.21.15.33:0/3524861110 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9a8c103f40 0x7f9a8c10c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:22.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.663+0000 7f9a91071700 1 -- 172.21.15.33:0/3524861110 >> 172.21.15.33:0/3524861110 conn(0x7f9a8c078340 msgr2=0x7f9a8c078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:22.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.663+0000 7f9a91071700 1 -- 172.21.15.33:0/3524861110 shutdown_connections 2024-03-19T02:52:22.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.663+0000 7f9a91071700 1 -- 172.21.15.33:0/3524861110 wait complete. 2024-03-19T02:52:22.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.664+0000 7f9a91071700 1 Processor -- start 2024-03-19T02:52:22.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.664+0000 7f9a91071700 1 -- start start 2024-03-19T02:52:22.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.664+0000 7f9a91071700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a8c103620 0x7f9a8c19e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:22.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.664+0000 7f9a91071700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9a8c103f40 0x7f9a8c19ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:22.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.664+0000 7f9a91071700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a8c10ca00 0x7f9a8c198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:22.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.664+0000 7f9a91071700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c112570 con 0x7f9a8c10ca00 2024-03-19T02:52:22.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.664+0000 7f9a91071700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c1123f0 con 0x7f9a8c103620 2024-03-19T02:52:22.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.664+0000 7f9a91071700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c1126f0 con 0x7f9a8c103f40 2024-03-19T02:52:22.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.664+0000 7f9a8ad9d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a8c103620 0x7f9a8c19e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:22.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.664+0000 7f9a8b59e700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a8c10ca00 0x7f9a8c198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:22.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.664+0000 7f9a8ad9d700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a8c103620 0x7f9a8c19e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:56876/0 (socket says 172.21.15.33:56876) 2024-03-19T02:52:22.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.664+0000 7f9a8a59c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9a8c103f40 0x7f9a8c19ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:22.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.664+0000 7f9a8ad9d700 1 -- 172.21.15.33:0/864649755 learned_addr learned my addr 172.21.15.33:0/864649755 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:22.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.665+0000 7f9a8ad9d700 1 -- 172.21.15.33:0/864649755 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9a8c103f40 msgr2=0x7f9a8c19ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:22.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.665+0000 7f9a8ad9d700 1 --2- 172.21.15.33:0/864649755 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9a8c103f40 0x7f9a8c19ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:22.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.665+0000 7f9a8ad9d700 1 -- 172.21.15.33:0/864649755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a8c10ca00 msgr2=0x7f9a8c198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:22.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.665+0000 7f9a8ad9d700 1 --2- 172.21.15.33:0/864649755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a8c10ca00 0x7f9a8c198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:22.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.665+0000 7f9a8ad9d700 1 -- 172.21.15.33:0/864649755 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a8c199070 con 0x7f9a8c103620 2024-03-19T02:52:22.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.665+0000 7f9a8ad9d700 1 --2- 172.21.15.33:0/864649755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a8c103620 0x7f9a8c19e6c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c00eb30 tx=0x7f9a7c00ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:22.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.665+0000 7f9a73fff700 1 -- 172.21.15.33:0/864649755 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c00cbd0 con 0x7f9a8c103620 2024-03-19T02:52:22.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.665+0000 7f9a73fff700 1 -- 172.21.15.33:0/864649755 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9a7c0044d0 con 0x7f9a8c103620 2024-03-19T02:52:22.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.665+0000 7f9a91071700 1 -- 172.21.15.33:0/864649755 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a8c199360 con 0x7f9a8c103620 2024-03-19T02:52:22.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.665+0000 7f9a73fff700 1 -- 172.21.15.33:0/864649755 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c003e30 con 0x7f9a8c103620 2024-03-19T02:52:22.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.665+0000 7f9a91071700 1 -- 172.21.15.33:0/864649755 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a8c1a5520 con 0x7f9a8c103620 2024-03-19T02:52:22.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.666+0000 7f9a91071700 1 -- 172.21.15.33:0/864649755 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a8c06b250 con 0x7f9a8c103620 2024-03-19T02:52:22.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.667+0000 7f9a73fff700 1 -- 172.21.15.33:0/864649755 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f9a7c004000 con 0x7f9a8c103620 2024-03-19T02:52:22.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.668+0000 7f9a73fff700 1 --2- 172.21.15.33:0/864649755 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9a740a29a0 0x7f9a740a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:22.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.668+0000 7f9a8a59c700 1 --2- 172.21.15.33:0/864649755 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9a740a29a0 0x7f9a740a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:22.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.668+0000 7f9a73fff700 1 -- 172.21.15.33:0/864649755 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7f9a7c0cf230 con 0x7f9a8c103620 2024-03-19T02:52:22.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.668+0000 7f9a8a59c700 1 --2- 172.21.15.33:0/864649755 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9a740a29a0 0x7f9a740a4e60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c10e650 tx=0x7f9a80009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:22.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.672+0000 7f9a73fff700 1 -- 172.21.15.33:0/864649755 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9a7c0042f0 con 0x7f9a8c103620 2024-03-19T02:52:22.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:22 smithi096 ceph-mon[35946]: pgmap v346: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-19T02:52:22.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:22 smithi155 ceph-mon[35804]: pgmap v346: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-19T02:52:22.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.882+0000 7f9a91071700 1 -- 172.21.15.33:0/864649755 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f9a8c02dbb0 con 0x7f9a8c103620 2024-03-19T02:52:22.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.883+0000 7f9a73fff700 1 -- 172.21.15.33:0/864649755 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9a7c09be10 con 0x7f9a8c103620 2024-03-19T02:52:22.884 INFO:teuthology.orchestra.run.smithi033.stdout:77309411406 2024-03-19T02:52:22.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.885+0000 7f9a91071700 1 -- 172.21.15.33:0/864649755 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9a740a29a0 msgr2=0x7f9a740a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:22.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.885+0000 7f9a91071700 1 --2- 172.21.15.33:0/864649755 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9a740a29a0 0x7f9a740a4e60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c10e650 tx=0x7f9a80009260 comp rx=0 tx=0).stop 2024-03-19T02:52:22.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.885+0000 7f9a91071700 1 -- 172.21.15.33:0/864649755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a8c103620 msgr2=0x7f9a8c19e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:22.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.885+0000 7f9a91071700 1 --2- 172.21.15.33:0/864649755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a8c103620 0x7f9a8c19e6c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c00eb30 tx=0x7f9a7c00ee40 comp rx=0 tx=0).stop 2024-03-19T02:52:22.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.885+0000 7f9a91071700 1 -- 172.21.15.33:0/864649755 shutdown_connections 2024-03-19T02:52:22.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.885+0000 7f9a91071700 1 --2- 172.21.15.33:0/864649755 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9a740a29a0 0x7f9a740a4e60 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:22.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.885+0000 7f9a91071700 1 --2- 172.21.15.33:0/864649755 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9a8c10ca00 0x7f9a8c198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:22.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.885+0000 7f9a91071700 1 --2- 172.21.15.33:0/864649755 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9a8c103620 0x7f9a8c19e6c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:22.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.885+0000 7f9a91071700 1 --2- 172.21.15.33:0/864649755 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9a8c103f40 0x7f9a8c19ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:22.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.885+0000 7f9a91071700 1 -- 172.21.15.33:0/864649755 >> 172.21.15.33:0/864649755 conn(0x7f9a8c078340 msgr2=0x7f9a8c0fe370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:22.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.885+0000 7f9a91071700 1 -- 172.21.15.33:0/864649755 shutdown_connections 2024-03-19T02:52:22.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:22.885+0000 7f9a91071700 1 -- 172.21.15.33:0/864649755 wait complete. 2024-03-19T02:52:23.087 INFO:tasks.cephadm.ceph_manager.ceph:need seq 317827579936 got 317827579939 for osd.10 2024-03-19T02:52:23.087 DEBUG:teuthology.parallel:result is None 2024-03-19T02:52:23.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:23 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/864649755' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-19T02:52:23.616 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:23 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/864649755' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-19T02:52:23.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:23 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/864649755' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-19T02:52:24.223 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724600 got 167503724605 for osd.5 2024-03-19T02:52:24.223 DEBUG:teuthology.parallel:result is None 2024-03-19T02:52:24.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.423+0000 7fc274a77700 1 -- 172.21.15.33:0/3596126443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc270108340 msgr2=0x7fc270108720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:24.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.423+0000 7fc274a77700 1 --2- 172.21.15.33:0/3596126443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc270108340 0x7fc270108720 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc2600099c0 tx=0x7fc260009cd0 comp rx=0 tx=0).stop 2024-03-19T02:52:24.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.424+0000 7fc274a77700 1 -- 172.21.15.33:0/3596126443 shutdown_connections 2024-03-19T02:52:24.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.424+0000 7fc274a77700 1 --2- 172.21.15.33:0/3596126443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc270102d20 0x7fc27010f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:24.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.424+0000 7fc274a77700 1 --2- 172.21.15.33:0/3596126443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc270108340 0x7fc270108720 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:24.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.424+0000 7fc274a77700 1 --2- 172.21.15.33:0/3596126443 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc270102360 0x7fc2701027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:24.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.424+0000 7fc274a77700 1 -- 172.21.15.33:0/3596126443 >> 172.21.15.33:0/3596126443 conn(0x7fc270079440 msgr2=0x7fc270079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:24.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.424+0000 7fc274a77700 1 -- 172.21.15.33:0/3596126443 shutdown_connections 2024-03-19T02:52:24.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.424+0000 7fc274a77700 1 -- 172.21.15.33:0/3596126443 wait complete. 2024-03-19T02:52:24.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.424+0000 7fc274a77700 1 Processor -- start 2024-03-19T02:52:24.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.424+0000 7fc274a77700 1 -- start start 2024-03-19T02:52:24.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.425+0000 7fc274a77700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc270102360 0x7fc270074610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:24.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.425+0000 7fc274a77700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc270102d20 0x7fc270074b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:24.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.425+0000 7fc274a77700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc270108340 0x7fc270075090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:24.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.425+0000 7fc274a77700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc2701124f0 con 0x7fc270102d20 2024-03-19T02:52:24.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.425+0000 7fc274a77700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc270112370 con 0x7fc270102360 2024-03-19T02:52:24.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.425+0000 7fc274a77700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc270112670 con 0x7fc270108340 2024-03-19T02:52:24.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.425+0000 7fc26dd9b700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc270102d20 0x7fc270074b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:24.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.425+0000 7fc26dd9b700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc270102d20 0x7fc270074b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38052/0 (socket says 172.21.15.33:38052) 2024-03-19T02:52:24.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.425+0000 7fc26dd9b700 1 -- 172.21.15.33:0/1927763485 learned_addr learned my addr 172.21.15.33:0/1927763485 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:24.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.425+0000 7fc26e59c700 1 --2- 172.21.15.33:0/1927763485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc270102360 0x7fc270074610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:24.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.425+0000 7fc26ed9d700 1 --2- 172.21.15.33:0/1927763485 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc270108340 0x7fc270075090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:24.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.425+0000 7fc26dd9b700 1 -- 172.21.15.33:0/1927763485 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc270108340 msgr2=0x7fc270075090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:24.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.425+0000 7fc26dd9b700 1 --2- 172.21.15.33:0/1927763485 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc270108340 0x7fc270075090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:24.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.425+0000 7fc26dd9b700 1 -- 172.21.15.33:0/1927763485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc270102360 msgr2=0x7fc270074610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:24.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.425+0000 7fc26dd9b700 1 --2- 172.21.15.33:0/1927763485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc270102360 0x7fc270074610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:24.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.425+0000 7fc26dd9b700 1 -- 172.21.15.33:0/1927763485 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2701a6f30 con 0x7fc270102d20 2024-03-19T02:52:24.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.425+0000 7fc26dd9b700 1 --2- 172.21.15.33:0/1927763485 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc270102d20 0x7fc270074b50 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc26400e9a0 tx=0x7fc26400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:24.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.425+0000 7fc25f7fe700 1 -- 172.21.15.33:0/1927763485 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc26400ca10 con 0x7fc270102d20 2024-03-19T02:52:24.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.425+0000 7fc274a77700 1 -- 172.21.15.33:0/1927763485 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2701a7220 con 0x7fc270102d20 2024-03-19T02:52:24.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.426+0000 7fc25f7fe700 1 -- 172.21.15.33:0/1927763485 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc26400cbb0 con 0x7fc270102d20 2024-03-19T02:52:24.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.426+0000 7fc25f7fe700 1 -- 172.21.15.33:0/1927763485 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc264018680 con 0x7fc270102d20 2024-03-19T02:52:24.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.426+0000 7fc274a77700 1 -- 172.21.15.33:0/1927763485 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2701a77b0 con 0x7fc270102d20 2024-03-19T02:52:24.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.427+0000 7fc274a77700 1 -- 172.21.15.33:0/1927763485 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc270068fd0 con 0x7fc270102d20 2024-03-19T02:52:24.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.428+0000 7fc25f7fe700 1 -- 172.21.15.33:0/1927763485 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fc264010430 con 0x7fc270102d20 2024-03-19T02:52:24.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.429+0000 7fc25f7fe700 1 --2- 172.21.15.33:0/1927763485 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc2580a2920 0x7fc2580a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:24.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.429+0000 7fc26e59c700 1 --2- 172.21.15.33:0/1927763485 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc2580a2920 0x7fc2580a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:24.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.429+0000 7fc25f7fe700 1 -- 172.21.15.33:0/1927763485 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7fc2640cf310 con 0x7fc270102d20 2024-03-19T02:52:24.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.429+0000 7fc26e59c700 1 --2- 172.21.15.33:0/1927763485 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc2580a2920 0x7fc2580a4de0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc260003760 tx=0x7fc260005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:24.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.433+0000 7fc25f7fe700 1 -- 172.21.15.33:0/1927763485 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc26409bef0 con 0x7fc270102d20 2024-03-19T02:52:24.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:24 smithi033 ceph-mon[31973]: pgmap v347: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:52:24.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:24 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3687472672' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-19T02:52:24.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.905+0000 7f52a4ae1700 1 -- 172.21.15.33:0/2000271377 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52a01014a0 msgr2=0x7f52a0101880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:24.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.905+0000 7f52a4ae1700 1 --2- 172.21.15.33:0/2000271377 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52a01014a0 0x7f52a0101880 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f529000b0d0 tx=0x7f529000b3e0 comp rx=0 tx=0).stop 2024-03-19T02:52:24.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.905+0000 7f52a4ae1700 1 -- 172.21.15.33:0/2000271377 shutdown_connections 2024-03-19T02:52:24.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.905+0000 7f52a4ae1700 1 --2- 172.21.15.33:0/2000271377 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f52a01064f0 0x7f52a010e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:24.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.905+0000 7f52a4ae1700 1 --2- 172.21.15.33:0/2000271377 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52a01014a0 0x7f52a0101880 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:24.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.905+0000 7f52a4ae1700 1 --2- 172.21.15.33:0/2000271377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52a0101e50 0x7f52a0105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:24.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.905+0000 7f52a4ae1700 1 -- 172.21.15.33:0/2000271377 >> 172.21.15.33:0/2000271377 conn(0x7f52a0077260 msgr2=0x7f52a0077670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:24.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.905+0000 7f52a4ae1700 1 -- 172.21.15.33:0/2000271377 shutdown_connections 2024-03-19T02:52:24.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.905+0000 7f52a4ae1700 1 -- 172.21.15.33:0/2000271377 wait complete. 2024-03-19T02:52:24.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.906+0000 7f52a4ae1700 1 Processor -- start 2024-03-19T02:52:24.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.906+0000 7f52a4ae1700 1 -- start start 2024-03-19T02:52:24.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.906+0000 7f52a4ae1700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52a01014a0 0x7f52a019d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:24.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.906+0000 7f52a4ae1700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52a0101e50 0x7f52a019db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:24.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.906+0000 7f52a4ae1700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f52a01064f0 0x7f52a01976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:24.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.906+0000 7f52a4ae1700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f52a0111380 con 0x7f52a01014a0 2024-03-19T02:52:24.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.906+0000 7f52a4ae1700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f52a0111200 con 0x7f52a01064f0 2024-03-19T02:52:24.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.906+0000 7f52a4ae1700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f52a0111500 con 0x7f52a0101e50 2024-03-19T02:52:24.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.906+0000 7f529e59c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52a01014a0 0x7f52a019d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:24.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.906+0000 7f529e59c700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52a01014a0 0x7f52a019d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38032/0 (socket says 172.21.15.33:38032) 2024-03-19T02:52:24.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.906+0000 7f529e59c700 1 -- 172.21.15.33:0/3687472672 learned_addr learned my addr 172.21.15.33:0/3687472672 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:24.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.907+0000 7f529ed9d700 1 --2- 172.21.15.33:0/3687472672 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f52a01064f0 0x7f52a01976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:24.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.907+0000 7f529dd9b700 1 --2- 172.21.15.33:0/3687472672 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52a0101e50 0x7f52a019db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:24.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.907+0000 7f529e59c700 1 -- 172.21.15.33:0/3687472672 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52a0101e50 msgr2=0x7f52a019db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:24.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.907+0000 7f529e59c700 1 --2- 172.21.15.33:0/3687472672 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52a0101e50 0x7f52a019db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:24.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.907+0000 7f529e59c700 1 -- 172.21.15.33:0/3687472672 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f52a01064f0 msgr2=0x7f52a01976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:24.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.907+0000 7f529e59c700 1 --2- 172.21.15.33:0/3687472672 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f52a01064f0 0x7f52a01976e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:24.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.907+0000 7f529e59c700 1 -- 172.21.15.33:0/3687472672 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52a0197fc0 con 0x7f52a01014a0 2024-03-19T02:52:24.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.907+0000 7f529ed9d700 1 --2- 172.21.15.33:0/3687472672 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f52a01064f0 0x7f52a01976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:24.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.907+0000 7f529e59c700 1 --2- 172.21.15.33:0/3687472672 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52a01014a0 0x7f52a019d610 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5290007a30 tx=0x7f5290007a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:24.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.907+0000 7f52977fe700 1 -- 172.21.15.33:0/3687472672 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5290004c50 con 0x7f52a01014a0 2024-03-19T02:52:24.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.907+0000 7f52977fe700 1 -- 172.21.15.33:0/3687472672 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5290004df0 con 0x7f52a01014a0 2024-03-19T02:52:24.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.907+0000 7f52977fe700 1 -- 172.21.15.33:0/3687472672 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5290005410 con 0x7f52a01014a0 2024-03-19T02:52:24.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.907+0000 7f52a4ae1700 1 -- 172.21.15.33:0/3687472672 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52a0198250 con 0x7f52a01014a0 2024-03-19T02:52:24.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.907+0000 7f52a4ae1700 1 -- 172.21.15.33:0/3687472672 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52a006a850 con 0x7f52a01014a0 2024-03-19T02:52:24.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.908+0000 7f52a4ae1700 1 -- 172.21.15.33:0/3687472672 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52a01028f0 con 0x7f52a01014a0 2024-03-19T02:52:24.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.910+0000 7f52977fe700 1 -- 172.21.15.33:0/3687472672 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f5290009b80 con 0x7f52a01014a0 2024-03-19T02:52:24.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.911+0000 7f52977fe700 1 --2- 172.21.15.33:0/3687472672 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f528c0a29a0 0x7f528c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:24.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.911+0000 7f529dd9b700 1 --2- 172.21.15.33:0/3687472672 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f528c0a29a0 0x7f528c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:24.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.911+0000 7f52977fe700 1 -- 172.21.15.33:0/3687472672 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7f529000f030 con 0x7f52a01014a0 2024-03-19T02:52:24.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.911+0000 7f529dd9b700 1 --2- 172.21.15.33:0/3687472672 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f528c0a29a0 0x7f528c0a4e60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f52a0198ea0 tx=0x7f5288009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:24.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:23.914+0000 7f52977fe700 1 -- 172.21.15.33:0/3687472672 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f529009eb60 con 0x7f52a01014a0 2024-03-19T02:52:24.535 INFO:teuthology.orchestra.run.smithi033.stdout:296352743466 2024-03-19T02:52:24.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.121+0000 7f52a4ae1700 1 -- 172.21.15.33:0/3687472672 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f52a006b2e0 con 0x7f52a01014a0 2024-03-19T02:52:24.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.121+0000 7f52977fe700 1 -- 172.21.15.33:0/3687472672 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f529002be30 con 0x7f52a01014a0 2024-03-19T02:52:24.535 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.123+0000 7f52a4ae1700 1 -- 172.21.15.33:0/3687472672 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f528c0a29a0 msgr2=0x7f528c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:24.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.123+0000 7f52a4ae1700 1 --2- 172.21.15.33:0/3687472672 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f528c0a29a0 0x7f528c0a4e60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f52a0198ea0 tx=0x7f5288009260 comp rx=0 tx=0).stop 2024-03-19T02:52:24.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.124+0000 7f52a4ae1700 1 -- 172.21.15.33:0/3687472672 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52a01014a0 msgr2=0x7f52a019d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:24.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.124+0000 7f52a4ae1700 1 --2- 172.21.15.33:0/3687472672 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52a01014a0 0x7f52a019d610 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5290007a30 tx=0x7f5290007a60 comp rx=0 tx=0).stop 2024-03-19T02:52:24.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.124+0000 7f52a4ae1700 1 -- 172.21.15.33:0/3687472672 shutdown_connections 2024-03-19T02:52:24.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.124+0000 7f52a4ae1700 1 --2- 172.21.15.33:0/3687472672 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f528c0a29a0 0x7f528c0a4e60 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:24.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.124+0000 7f52a4ae1700 1 --2- 172.21.15.33:0/3687472672 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f52a01064f0 0x7f52a01976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:24.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.124+0000 7f52a4ae1700 1 --2- 172.21.15.33:0/3687472672 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52a01014a0 0x7f52a019d610 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:24.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.124+0000 7f52a4ae1700 1 --2- 172.21.15.33:0/3687472672 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52a0101e50 0x7f52a019db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:24.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.124+0000 7f52a4ae1700 1 -- 172.21.15.33:0/3687472672 >> 172.21.15.33:0/3687472672 conn(0x7f52a0077260 msgr2=0x7f52a0105130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:24.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.124+0000 7f52a4ae1700 1 -- 172.21.15.33:0/3687472672 shutdown_connections 2024-03-19T02:52:24.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.124+0000 7f52a4ae1700 1 -- 172.21.15.33:0/3687472672 wait complete. 2024-03-19T02:52:24.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.646+0000 7fc274a77700 1 -- 172.21.15.33:0/1927763485 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fc27002dbb0 con 0x7fc270102d20 2024-03-19T02:52:24.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.647+0000 7fc25f7fe700 1 -- 172.21.15.33:0/1927763485 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc26409b7f0 con 0x7fc270102d20 2024-03-19T02:52:24.648 INFO:teuthology.orchestra.run.smithi033.stdout:146028888132 2024-03-19T02:52:24.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.649+0000 7fc274a77700 1 -- 172.21.15.33:0/1927763485 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc2580a2920 msgr2=0x7fc2580a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:24.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.649+0000 7fc274a77700 1 --2- 172.21.15.33:0/1927763485 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc2580a2920 0x7fc2580a4de0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc260003760 tx=0x7fc260005fb0 comp rx=0 tx=0).stop 2024-03-19T02:52:24.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.649+0000 7fc274a77700 1 -- 172.21.15.33:0/1927763485 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc270102d20 msgr2=0x7fc270074b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:24.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.649+0000 7fc274a77700 1 --2- 172.21.15.33:0/1927763485 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc270102d20 0x7fc270074b50 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc26400e9a0 tx=0x7fc26400ecb0 comp rx=0 tx=0).stop 2024-03-19T02:52:24.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.649+0000 7fc274a77700 1 -- 172.21.15.33:0/1927763485 shutdown_connections 2024-03-19T02:52:24.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.649+0000 7fc274a77700 1 --2- 172.21.15.33:0/1927763485 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc2580a2920 0x7fc2580a4de0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:24.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.649+0000 7fc274a77700 1 --2- 172.21.15.33:0/1927763485 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc270108340 0x7fc270075090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:24.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.649+0000 7fc274a77700 1 --2- 172.21.15.33:0/1927763485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc270102360 0x7fc270074610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:24.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.649+0000 7fc274a77700 1 --2- 172.21.15.33:0/1927763485 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc270102d20 0x7fc270074b50 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:24.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.649+0000 7fc274a77700 1 -- 172.21.15.33:0/1927763485 >> 172.21.15.33:0/1927763485 conn(0x7fc270079440 msgr2=0x7fc270101100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:24.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.649+0000 7fc274a77700 1 -- 172.21.15.33:0/1927763485 shutdown_connections 2024-03-19T02:52:24.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:24.649+0000 7fc274a77700 1 -- 172.21.15.33:0/1927763485 wait complete. 2024-03-19T02:52:24.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:24 smithi096 ceph-mon[35946]: pgmap v347: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:52:24.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:24 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3687472672' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-19T02:52:24.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:24 smithi155 ceph-mon[35804]: pgmap v347: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:52:24.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:24 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3687472672' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-19T02:52:25.390 INFO:tasks.cephadm.ceph_manager.ceph:need seq 296352743461 got 296352743466 for osd.9 2024-03-19T02:52:25.390 DEBUG:teuthology.parallel:result is None 2024-03-19T02:52:25.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.464+0000 7f24bb275700 1 -- 172.21.15.33:0/1199037510 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f24b4108340 msgr2=0x7f24b4108720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:25.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.464+0000 7f24bb275700 1 --2- 172.21.15.33:0/1199037510 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f24b4108340 0x7f24b4108720 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f24a40099c0 tx=0x7f24a4009cd0 comp rx=0 tx=0).stop 2024-03-19T02:52:25.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.464+0000 7f24bb275700 1 -- 172.21.15.33:0/1199037510 shutdown_connections 2024-03-19T02:52:25.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.464+0000 7f24bb275700 1 --2- 172.21.15.33:0/1199037510 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24b4102d20 0x7f24b410f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:25.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.464+0000 7f24bb275700 1 --2- 172.21.15.33:0/1199037510 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f24b4108340 0x7f24b4108720 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:25.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.464+0000 7f24bb275700 1 --2- 172.21.15.33:0/1199037510 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24b4102360 0x7f24b41027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:25.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.464+0000 7f24bb275700 1 -- 172.21.15.33:0/1199037510 >> 172.21.15.33:0/1199037510 conn(0x7f24b4079440 msgr2=0x7f24b4079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:25.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.465+0000 7f24bb275700 1 -- 172.21.15.33:0/1199037510 shutdown_connections 2024-03-19T02:52:25.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.465+0000 7f24bb275700 1 -- 172.21.15.33:0/1199037510 wait complete. 2024-03-19T02:52:25.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.466+0000 7f24bb275700 1 Processor -- start 2024-03-19T02:52:25.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.466+0000 7f24bb275700 1 -- start start 2024-03-19T02:52:25.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.466+0000 7f24bb275700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24b4102360 0x7f24b4074610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:25.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.466+0000 7f24bb275700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f24b4102d20 0x7f24b4074b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:25.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.466+0000 7f24bb275700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24b4108340 0x7f24b4075090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:25.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.466+0000 7f24bb275700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f24b4112470 con 0x7f24b4102d20 2024-03-19T02:52:25.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.466+0000 7f24bb275700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f24b41122f0 con 0x7f24b4108340 2024-03-19T02:52:25.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.466+0000 7f24bb275700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f24b41125f0 con 0x7f24b4102360 2024-03-19T02:52:25.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.466+0000 7f24b9812700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24b4108340 0x7f24b4075090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:25.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.466+0000 7f24b8810700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f24b4102d20 0x7f24b4074b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:25.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.466+0000 7f24b9011700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24b4102360 0x7f24b4074610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:25.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.466+0000 7f24b9812700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24b4108340 0x7f24b4075090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:38434/0 (socket says 172.21.15.33:38434) 2024-03-19T02:52:25.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.466+0000 7f24b9812700 1 -- 172.21.15.33:0/1642304586 learned_addr learned my addr 172.21.15.33:0/1642304586 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:25.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.467+0000 7f24b9812700 1 -- 172.21.15.33:0/1642304586 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24b4102360 msgr2=0x7f24b4074610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:25.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.467+0000 7f24b9812700 1 --2- 172.21.15.33:0/1642304586 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24b4102360 0x7f24b4074610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:25.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.467+0000 7f24b9812700 1 -- 172.21.15.33:0/1642304586 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f24b4102d20 msgr2=0x7f24b4074b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:25.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.467+0000 7f24b9812700 1 --2- 172.21.15.33:0/1642304586 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f24b4102d20 0x7f24b4074b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:25.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.467+0000 7f24b9812700 1 -- 172.21.15.33:0/1642304586 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24b41a6e40 con 0x7f24b4108340 2024-03-19T02:52:25.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.467+0000 7f24b9011700 1 --2- 172.21.15.33:0/1642304586 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24b4102360 0x7f24b4074610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:25.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.467+0000 7f24b9812700 1 --2- 172.21.15.33:0/1642304586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24b4108340 0x7f24b4075090 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f24ac00ef30 tx=0x7f24ac00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:25.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.467+0000 7f24aa7fc700 1 -- 172.21.15.33:0/1642304586 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24ac0092e0 con 0x7f24b4108340 2024-03-19T02:52:25.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.467+0000 7f24aa7fc700 1 -- 172.21.15.33:0/1642304586 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f24ac004bb0 con 0x7f24b4108340 2024-03-19T02:52:25.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.467+0000 7f24bb275700 1 -- 172.21.15.33:0/1642304586 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24b41a70d0 con 0x7f24b4108340 2024-03-19T02:52:25.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.467+0000 7f24aa7fc700 1 -- 172.21.15.33:0/1642304586 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24ac004020 con 0x7f24b4108340 2024-03-19T02:52:25.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.467+0000 7f24bb275700 1 -- 172.21.15.33:0/1642304586 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24b41a75c0 con 0x7f24b4108340 2024-03-19T02:52:25.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.468+0000 7f24bb275700 1 -- 172.21.15.33:0/1642304586 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24b4068fd0 con 0x7f24b4108340 2024-03-19T02:52:25.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.469+0000 7f24aa7fc700 1 -- 172.21.15.33:0/1642304586 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f24ac01b030 con 0x7f24b4108340 2024-03-19T02:52:25.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.470+0000 7f24aa7fc700 1 --2- 172.21.15.33:0/1642304586 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f24a00a2920 0x7f24a00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:25.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.470+0000 7f24b9011700 1 --2- 172.21.15.33:0/1642304586 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f24a00a2920 0x7f24a00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:25.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.470+0000 7f24aa7fc700 1 -- 172.21.15.33:0/1642304586 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7f24ac0ce9e0 con 0x7f24b4108340 2024-03-19T02:52:25.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.470+0000 7f24b9011700 1 --2- 172.21.15.33:0/1642304586 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f24a00a2920 0x7f24a00a4de0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f24a4003760 tx=0x7f24a4005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:25.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.478+0000 7f24aa7fc700 1 -- 172.21.15.33:0/1642304586 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f24ac022020 con 0x7f24b4108340 2024-03-19T02:52:25.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:25 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1927763485' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-19T02:52:25.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.686+0000 7f24bb275700 1 -- 172.21.15.33:0/1642304586 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f24b402de10 con 0x7f24b4108340 2024-03-19T02:52:25.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.687+0000 7f24aa7fc700 1 -- 172.21.15.33:0/1642304586 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f24ac010040 con 0x7f24b4108340 2024-03-19T02:52:25.688 INFO:teuthology.orchestra.run.smithi033.stdout:352187318304 2024-03-19T02:52:25.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.689+0000 7f24bb275700 1 -- 172.21.15.33:0/1642304586 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f24a00a2920 msgr2=0x7f24a00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:25.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.689+0000 7f24bb275700 1 --2- 172.21.15.33:0/1642304586 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f24a00a2920 0x7f24a00a4de0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f24a4003760 tx=0x7f24a4005fb0 comp rx=0 tx=0).stop 2024-03-19T02:52:25.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.689+0000 7f24bb275700 1 -- 172.21.15.33:0/1642304586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24b4108340 msgr2=0x7f24b4075090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:25.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.689+0000 7f24bb275700 1 --2- 172.21.15.33:0/1642304586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24b4108340 0x7f24b4075090 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f24ac00ef30 tx=0x7f24ac00c3a0 comp rx=0 tx=0).stop 2024-03-19T02:52:25.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.689+0000 7f24bb275700 1 -- 172.21.15.33:0/1642304586 shutdown_connections 2024-03-19T02:52:25.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.689+0000 7f24bb275700 1 --2- 172.21.15.33:0/1642304586 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f24a00a2920 0x7f24a00a4de0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:25.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.689+0000 7f24bb275700 1 --2- 172.21.15.33:0/1642304586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24b4108340 0x7f24b4075090 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:25.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.689+0000 7f24bb275700 1 --2- 172.21.15.33:0/1642304586 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24b4102360 0x7f24b4074610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:25.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.689+0000 7f24bb275700 1 --2- 172.21.15.33:0/1642304586 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f24b4102d20 0x7f24b4074b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:25.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.689+0000 7f24bb275700 1 -- 172.21.15.33:0/1642304586 >> 172.21.15.33:0/1642304586 conn(0x7f24b4079440 msgr2=0x7f24b41010e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:25.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.689+0000 7f24bb275700 1 -- 172.21.15.33:0/1642304586 shutdown_connections 2024-03-19T02:52:25.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:25.689+0000 7f24bb275700 1 -- 172.21.15.33:0/1642304586 wait complete. 2024-03-19T02:52:25.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:25 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1927763485' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-19T02:52:25.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:25 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1927763485' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-19T02:52:25.988 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411401 got 77309411406 for osd.2 2024-03-19T02:52:25.988 DEBUG:teuthology.parallel:result is None 2024-03-19T02:52:26.625 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:26 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1642304586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-19T02:52:26.625 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:26 smithi033 ceph-mon[31973]: pgmap v348: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:52:26.653 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888126 got 146028888132 for osd.4 2024-03-19T02:52:26.654 DEBUG:teuthology.parallel:result is None 2024-03-19T02:52:26.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:26 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1642304586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-19T02:52:26.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:26 smithi096 ceph-mon[35946]: pgmap v348: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:52:26.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:26 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1642304586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-19T02:52:26.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:26 smithi155 ceph-mon[35804]: pgmap v348: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:52:27.264 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574926 got 55834574930 for osd.1 2024-03-19T02:52:27.264 DEBUG:teuthology.parallel:result is None 2024-03-19T02:52:27.708 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738451 got 34359738455 for osd.0 2024-03-19T02:52:27.708 DEBUG:teuthology.parallel:result is None 2024-03-19T02:52:28.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:27 smithi033 ceph-mon[31973]: pgmap v349: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:52:28.082 INFO:tasks.cephadm.ceph_manager.ceph:need seq 352187318298 got 352187318304 for osd.11 2024-03-19T02:52:28.082 DEBUG:teuthology.parallel:result is None 2024-03-19T02:52:28.082 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-19T02:52:28.082 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-19T02:52:28.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:27 smithi096 ceph-mon[35946]: pgmap v349: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:52:28.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:27 smithi155 ceph-mon[35804]: pgmap v349: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:52:28.343 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:52:29.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:29 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:52:29.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:29 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:52:29.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:29 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:52:29.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:29 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:52:29.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:29 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:52:29.516 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:29 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:52:29.517 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:29 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:52:29.517 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:29 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:52:29.517 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:29 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:52:29.517 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:29 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:52:29.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:29 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:52:29.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:29 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:52:29.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:29 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:52:29.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:29 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:52:29.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:29 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' 2024-03-19T02:52:30.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.154+0000 7fa57b0d7700 1 -- 172.21.15.33:0/3583608505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa574101e90 msgr2=0x7fa574105ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:30.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.154+0000 7fa57b0d7700 1 --2- 172.21.15.33:0/3583608505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa574101e90 0x7fa574105ee0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa5680099c0 tx=0x7fa568009cd0 comp rx=0 tx=0).stop 2024-03-19T02:52:30.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.154+0000 7fa57b0d7700 1 -- 172.21.15.33:0/3583608505 shutdown_connections 2024-03-19T02:52:30.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.154+0000 7fa57b0d7700 1 --2- 172.21.15.33:0/3583608505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5741065c0 0x7fa57410e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:30.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.154+0000 7fa57b0d7700 1 --2- 172.21.15.33:0/3583608505 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5741014e0 0x7fa5741018c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:30.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.154+0000 7fa57b0d7700 1 --2- 172.21.15.33:0/3583608505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa574101e90 0x7fa574105ee0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:30.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.154+0000 7fa57b0d7700 1 -- 172.21.15.33:0/3583608505 >> 172.21.15.33:0/3583608505 conn(0x7fa574078340 msgr2=0x7fa574078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:30.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.154+0000 7fa57b0d7700 1 -- 172.21.15.33:0/3583608505 shutdown_connections 2024-03-19T02:52:30.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.154+0000 7fa57b0d7700 1 -- 172.21.15.33:0/3583608505 wait complete. 2024-03-19T02:52:30.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.155+0000 7fa57b0d7700 1 Processor -- start 2024-03-19T02:52:30.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.155+0000 7fa57b0d7700 1 -- start start 2024-03-19T02:52:30.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.155+0000 7fa57b0d7700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5741014e0 0x7fa57419d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:30.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.155+0000 7fa57b0d7700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa574101e90 0x7fa57419dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:30.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.155+0000 7fa57b0d7700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5741065c0 0x7fa574197840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:30.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.155+0000 7fa57b0d7700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa5741114a0 con 0x7fa574101e90 2024-03-19T02:52:30.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.155+0000 7fa57b0d7700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa574111320 con 0x7fa5741065c0 2024-03-19T02:52:30.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.155+0000 7fa57b0d7700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa574111620 con 0x7fa5741014e0 2024-03-19T02:52:30.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.156+0000 7fa573fff700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa574101e90 0x7fa57419dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:30.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.156+0000 7fa579674700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5741065c0 0x7fa574197840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:30.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.156+0000 7fa578e73700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5741014e0 0x7fa57419d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:30.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.156+0000 7fa579674700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5741065c0 0x7fa574197840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:38448/0 (socket says 172.21.15.33:38448) 2024-03-19T02:52:30.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.156+0000 7fa578e73700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5741014e0 0x7fa57419d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:58944/0 (socket says 172.21.15.33:58944) 2024-03-19T02:52:30.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.156+0000 7fa579674700 1 -- 172.21.15.33:0/958278402 learned_addr learned my addr 172.21.15.33:0/958278402 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:30.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.156+0000 7fa579674700 1 -- 172.21.15.33:0/958278402 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5741014e0 msgr2=0x7fa57419d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:30.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.156+0000 7fa579674700 1 --2- 172.21.15.33:0/958278402 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5741014e0 0x7fa57419d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:30.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.156+0000 7fa579674700 1 -- 172.21.15.33:0/958278402 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa574101e90 msgr2=0x7fa57419dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:30.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.156+0000 7fa579674700 1 --2- 172.21.15.33:0/958278402 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa574101e90 0x7fa57419dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:30.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.156+0000 7fa579674700 1 -- 172.21.15.33:0/958278402 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa574198020 con 0x7fa5741065c0 2024-03-19T02:52:30.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.156+0000 7fa579674700 1 --2- 172.21.15.33:0/958278402 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5741065c0 0x7fa574197840 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa56c00e580 tx=0x7fa56c00e890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:30.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.156+0000 7fa571ffb700 1 -- 172.21.15.33:0/958278402 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa56c004db0 con 0x7fa5741065c0 2024-03-19T02:52:30.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.156+0000 7fa571ffb700 1 -- 172.21.15.33:0/958278402 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa56c0090d0 con 0x7fa5741065c0 2024-03-19T02:52:30.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.156+0000 7fa571ffb700 1 -- 172.21.15.33:0/958278402 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa56c004640 con 0x7fa5741065c0 2024-03-19T02:52:30.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.156+0000 7fa57b0d7700 1 -- 172.21.15.33:0/958278402 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa574198310 con 0x7fa5741065c0 2024-03-19T02:52:30.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.157+0000 7fa57b0d7700 1 -- 172.21.15.33:0/958278402 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa57406aa50 con 0x7fa5741065c0 2024-03-19T02:52:30.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.157+0000 7fa57b0d7700 1 -- 172.21.15.33:0/958278402 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5741029c0 con 0x7fa5741065c0 2024-03-19T02:52:30.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.159+0000 7fa571ffb700 1 -- 172.21.15.33:0/958278402 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fa56c003a00 con 0x7fa5741065c0 2024-03-19T02:52:30.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.159+0000 7fa571ffb700 1 --2- 172.21.15.33:0/958278402 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fa55c0a29a0 0x7fa55c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:30.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.159+0000 7fa578e73700 1 --2- 172.21.15.33:0/958278402 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fa55c0a29a0 0x7fa55c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:30.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.159+0000 7fa571ffb700 1 -- 172.21.15.33:0/958278402 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7fa56c0d0fa0 con 0x7fa5741065c0 2024-03-19T02:52:30.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.160+0000 7fa578e73700 1 --2- 172.21.15.33:0/958278402 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fa55c0a29a0 0x7fa55c0a4e60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa564009c60 tx=0x7fa564009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:30.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.167+0000 7fa571ffb700 1 -- 172.21.15.33:0/958278402 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa56c09db80 con 0x7fa5741065c0 2024-03-19T02:52:30.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.366+0000 7fa57b0d7700 1 -- 172.21.15.33:0/958278402 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa57406b560 con 0x7fa55c0a29a0 2024-03-19T02:52:30.367 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:30 smithi033 ceph-mon[31973]: pgmap v350: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-19T02:52:30.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.378+0000 7fa571ffb700 1 -- 172.21.15.33:0/958278402 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340381 (secure 0 0 0) 0x7fa57406b560 con 0x7fa55c0a29a0 2024-03-19T02:52:30.384 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:52:30.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.382+0000 7fa57b0d7700 1 -- 172.21.15.33:0/958278402 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fa55c0a29a0 msgr2=0x7fa55c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:30.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.382+0000 7fa57b0d7700 1 --2- 172.21.15.33:0/958278402 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fa55c0a29a0 0x7fa55c0a4e60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa564009c60 tx=0x7fa564009260 comp rx=0 tx=0).stop 2024-03-19T02:52:30.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.382+0000 7fa57b0d7700 1 -- 172.21.15.33:0/958278402 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5741065c0 msgr2=0x7fa574197840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:30.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.382+0000 7fa57b0d7700 1 --2- 172.21.15.33:0/958278402 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5741065c0 0x7fa574197840 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa56c00e580 tx=0x7fa56c00e890 comp rx=0 tx=0).stop 2024-03-19T02:52:30.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.382+0000 7fa57b0d7700 1 -- 172.21.15.33:0/958278402 shutdown_connections 2024-03-19T02:52:30.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.382+0000 7fa57b0d7700 1 --2- 172.21.15.33:0/958278402 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fa55c0a29a0 0x7fa55c0a4e60 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:30.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.382+0000 7fa57b0d7700 1 --2- 172.21.15.33:0/958278402 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5741065c0 0x7fa574197840 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:30.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.382+0000 7fa57b0d7700 1 --2- 172.21.15.33:0/958278402 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5741014e0 0x7fa57419d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:30.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.382+0000 7fa57b0d7700 1 --2- 172.21.15.33:0/958278402 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa574101e90 0x7fa57419dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:30.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.382+0000 7fa57b0d7700 1 -- 172.21.15.33:0/958278402 >> 172.21.15.33:0/958278402 conn(0x7fa574078340 msgr2=0x7fa5740ff480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:30.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.382+0000 7fa57b0d7700 1 -- 172.21.15.33:0/958278402 shutdown_connections 2024-03-19T02:52:30.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:30.382+0000 7fa57b0d7700 1 -- 172.21.15.33:0/958278402 wait complete. 2024-03-19T02:52:30.387 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-03-19T02:52:30.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:30 smithi096 ceph-mon[35946]: pgmap v350: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-19T02:52:30.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:30 smithi155 ceph-mon[35804]: pgmap v350: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-19T02:52:31.058 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2024-03-19T02:52:29.730164+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657568,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":574,"num_read_kb":410,"num_write":343,"num_write_kb":3055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1181547,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":278,"ondisk_log_size":278,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":368044,"kb_used_data":14672,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124573780,"statfs":{"total":1151940427776,"available":1151563550720,"internally_reserved":0,"allocated":15024128,"data_stored":8270546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":75,"num_read_kb":40,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003040"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.717879+0000","last_change":"2024-03-19T02:50:52.228368+0000","last_active":"2024-03-19T02:51:40.717879+0000","last_peered":"2024-03-19T02:51:40.717879+0000","last_clean":"2024-03-19T02:51:40.717879+0000","last_became_active":"2024-03-19T02:50:52.228069+0000","last_became_peered":"2024-03-19T02:50:52.228069+0000","last_unstale":"2024-03-19T02:51:40.717879+0000","last_undegraded":"2024-03-19T02:51:40.717879+0000","last_fullsized":"2024-03-19T02:51:40.717879+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:59:34.209095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.720688+0000","last_change":"2024-03-19T02:50:52.231358+0000","last_active":"2024-03-19T02:51:40.720688+0000","last_peered":"2024-03-19T02:51:40.720688+0000","last_clean":"2024-03-19T02:51:40.720688+0000","last_became_active":"2024-03-19T02:50:52.231029+0000","last_became_peered":"2024-03-19T02:50:52.231029+0000","last_unstale":"2024-03-19T02:51:40.720688+0000","last_undegraded":"2024-03-19T02:51:40.720688+0000","last_fullsized":"2024-03-19T02:51:40.720688+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:42:26.031392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719930+0000","last_change":"2024-03-19T02:50:52.232685+0000","last_active":"2024-03-19T02:51:40.719930+0000","last_peered":"2024-03-19T02:51:40.719930+0000","last_clean":"2024-03-19T02:51:40.719930+0000","last_became_active":"2024-03-19T02:50:52.232361+0000","last_became_peered":"2024-03-19T02:50:52.232361+0000","last_unstale":"2024-03-19T02:51:40.719930+0000","last_undegraded":"2024-03-19T02:51:40.719930+0000","last_fullsized":"2024-03-19T02:51:40.719930+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:53.136200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.716118+0000","last_change":"2024-03-19T02:50:52.227648+0000","last_active":"2024-03-19T02:51:40.716118+0000","last_peered":"2024-03-19T02:51:40.716118+0000","last_clean":"2024-03-19T02:51:40.716118+0000","last_became_active":"2024-03-19T02:50:52.226943+0000","last_became_peered":"2024-03-19T02:50:52.226943+0000","last_unstale":"2024-03-19T02:51:40.716118+0000","last_undegraded":"2024-03-19T02:51:40.716118+0000","last_fullsized":"2024-03-19T02:51:40.716118+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:03.293794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715486+0000","last_change":"2024-03-19T02:50:52.225329+0000","last_active":"2024-03-19T02:51:40.715486+0000","last_peered":"2024-03-19T02:51:40.715486+0000","last_clean":"2024-03-19T02:51:40.715486+0000","last_became_active":"2024-03-19T02:50:52.225012+0000","last_became_peered":"2024-03-19T02:50:52.225012+0000","last_unstale":"2024-03-19T02:51:40.715486+0000","last_undegraded":"2024-03-19T02:51:40.715486+0000","last_fullsized":"2024-03-19T02:51:40.715486+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:32:52.877704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719154+0000","last_change":"2024-03-19T02:50:52.217399+0000","last_active":"2024-03-19T02:51:40.719154+0000","last_peered":"2024-03-19T02:51:40.719154+0000","last_clean":"2024-03-19T02:51:40.719154+0000","last_became_active":"2024-03-19T02:50:52.216987+0000","last_became_peered":"2024-03-19T02:50:52.216987+0000","last_unstale":"2024-03-19T02:51:40.719154+0000","last_undegraded":"2024-03-19T02:51:40.719154+0000","last_fullsized":"2024-03-19T02:51:40.719154+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:57:15.772546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.718346+0000","last_change":"2024-03-19T02:50:52.226755+0000","last_active":"2024-03-19T02:51:40.718346+0000","last_peered":"2024-03-19T02:51:40.718346+0000","last_clean":"2024-03-19T02:51:40.718346+0000","last_became_active":"2024-03-19T02:50:52.226434+0000","last_became_peered":"2024-03-19T02:50:52.226434+0000","last_unstale":"2024-03-19T02:51:40.718346+0000","last_undegraded":"2024-03-19T02:51:40.718346+0000","last_fullsized":"2024-03-19T02:51:40.718346+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:49:42.134262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715219+0000","last_change":"2024-03-19T02:50:52.215848+0000","last_active":"2024-03-19T02:51:40.715219+0000","last_peered":"2024-03-19T02:51:40.715219+0000","last_clean":"2024-03-19T02:51:40.715219+0000","last_became_active":"2024-03-19T02:50:52.215494+0000","last_became_peered":"2024-03-19T02:50:52.215494+0000","last_unstale":"2024-03-19T02:51:40.715219+0000","last_undegraded":"2024-03-19T02:51:40.715219+0000","last_fullsized":"2024-03-19T02:51:40.715219+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:55:10.182359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719368+0000","last_change":"2024-03-19T02:50:52.233237+0000","last_active":"2024-03-19T02:51:40.719368+0000","last_peered":"2024-03-19T02:51:40.719368+0000","last_clean":"2024-03-19T02:51:40.719368+0000","last_became_active":"2024-03-19T02:50:52.232091+0000","last_became_peered":"2024-03-19T02:50:52.232091+0000","last_unstale":"2024-03-19T02:51:40.719368+0000","last_undegraded":"2024-03-19T02:51:40.719368+0000","last_fullsized":"2024-03-19T02:51:40.719368+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:08:20.361576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.722463+0000","last_change":"2024-03-19T02:51:40.722463+0000","last_active":"2024-03-19T02:51:40.722463+0000","last_peered":"2024-03-19T02:51:40.722463+0000","last_clean":"2024-03-19T02:51:40.722463+0000","last_became_active":"2024-03-19T02:51:40.722120+0000","last_became_peered":"2024-03-19T02:51:40.722120+0000","last_unstale":"2024-03-19T02:51:40.722463+0000","last_undegraded":"2024-03-19T02:51:40.722463+0000","last_fullsized":"2024-03-19T02:51:40.722463+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:49:21.387064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.720478+0000","last_change":"2024-03-19T02:50:52.226314+0000","last_active":"2024-03-19T02:51:40.720478+0000","last_peered":"2024-03-19T02:51:40.720478+0000","last_clean":"2024-03-19T02:51:40.720478+0000","last_became_active":"2024-03-19T02:50:52.225919+0000","last_became_peered":"2024-03-19T02:50:52.225919+0000","last_unstale":"2024-03-19T02:51:40.720478+0000","last_undegraded":"2024-03-19T02:51:40.720478+0000","last_fullsized":"2024-03-19T02:51:40.720478+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:42:51.782592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.716200+0000","last_change":"2024-03-19T02:50:52.227929+0000","last_active":"2024-03-19T02:51:40.716200+0000","last_peered":"2024-03-19T02:51:40.716200+0000","last_clean":"2024-03-19T02:51:40.716200+0000","last_became_active":"2024-03-19T02:50:52.227360+0000","last_became_peered":"2024-03-19T02:50:52.227360+0000","last_unstale":"2024-03-19T02:51:40.716200+0000","last_undegraded":"2024-03-19T02:51:40.716200+0000","last_fullsized":"2024-03-19T02:51:40.716200+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:24:44.386101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719392+0000","last_change":"2024-03-19T02:50:52.228088+0000","last_active":"2024-03-19T02:51:40.719392+0000","last_peered":"2024-03-19T02:51:40.719392+0000","last_clean":"2024-03-19T02:51:40.719392+0000","last_became_active":"2024-03-19T02:50:52.227754+0000","last_became_peered":"2024-03-19T02:50:52.227754+0000","last_unstale":"2024-03-19T02:51:40.719392+0000","last_undegraded":"2024-03-19T02:51:40.719392+0000","last_fullsized":"2024-03-19T02:51:40.719392+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:58:57.404241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719885+0000","last_change":"2024-03-19T02:50:52.226768+0000","last_active":"2024-03-19T02:51:40.719885+0000","last_peered":"2024-03-19T02:51:40.719885+0000","last_clean":"2024-03-19T02:51:40.719885+0000","last_became_active":"2024-03-19T02:50:52.226452+0000","last_became_peered":"2024-03-19T02:50:52.226452+0000","last_unstale":"2024-03-19T02:51:40.719885+0000","last_undegraded":"2024-03-19T02:51:40.719885+0000","last_fullsized":"2024-03-19T02:51:40.719885+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:23:33.250773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719676+0000","last_change":"2024-03-19T02:50:52.231132+0000","last_active":"2024-03-19T02:51:40.719676+0000","last_peered":"2024-03-19T02:51:40.719676+0000","last_clean":"2024-03-19T02:51:40.719676+0000","last_became_active":"2024-03-19T02:50:52.230778+0000","last_became_peered":"2024-03-19T02:50:52.230778+0000","last_unstale":"2024-03-19T02:51:40.719676+0000","last_undegraded":"2024-03-19T02:51:40.719676+0000","last_fullsized":"2024-03-19T02:51:40.719676+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:33:03.675210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.720289+0000","last_change":"2024-03-19T02:50:52.228302+0000","last_active":"2024-03-19T02:51:40.720289+0000","last_peered":"2024-03-19T02:51:40.720289+0000","last_clean":"2024-03-19T02:51:40.720289+0000","last_became_active":"2024-03-19T02:50:52.228007+0000","last_became_peered":"2024-03-19T02:50:52.228007+0000","last_unstale":"2024-03-19T02:51:40.720289+0000","last_undegraded":"2024-03-19T02:51:40.720289+0000","last_fullsized":"2024-03-19T02:51:40.720289+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:59:28.988504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.717631+0000","last_change":"2024-03-19T02:50:52.228640+0000","last_active":"2024-03-19T02:51:40.717631+0000","last_peered":"2024-03-19T02:51:40.717631+0000","last_clean":"2024-03-19T02:51:40.717631+0000","last_became_active":"2024-03-19T02:50:52.228319+0000","last_became_peered":"2024-03-19T02:50:52.228319+0000","last_unstale":"2024-03-19T02:51:40.717631+0000","last_undegraded":"2024-03-19T02:51:40.717631+0000","last_fullsized":"2024-03-19T02:51:40.717631+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:30.356453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.718484+0000","last_change":"2024-03-19T02:50:52.222300+0000","last_active":"2024-03-19T02:51:40.718484+0000","last_peered":"2024-03-19T02:51:40.718484+0000","last_clean":"2024-03-19T02:51:40.718484+0000","last_became_active":"2024-03-19T02:50:52.221978+0000","last_became_peered":"2024-03-19T02:50:52.221978+0000","last_unstale":"2024-03-19T02:51:40.718484+0000","last_undegraded":"2024-03-19T02:51:40.718484+0000","last_fullsized":"2024-03-19T02:51:40.718484+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:38:36.576987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719330+0000","last_change":"2024-03-19T02:50:52.217640+0000","last_active":"2024-03-19T02:51:40.719330+0000","last_peered":"2024-03-19T02:51:40.719330+0000","last_clean":"2024-03-19T02:51:40.719330+0000","last_became_active":"2024-03-19T02:50:52.217325+0000","last_became_peered":"2024-03-19T02:50:52.217325+0000","last_unstale":"2024-03-19T02:51:40.719330+0000","last_undegraded":"2024-03-19T02:51:40.719330+0000","last_fullsized":"2024-03-19T02:51:40.719330+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:40:25.894843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.718707+0000","last_change":"2024-03-19T02:50:57.245323+0000","last_active":"2024-03-19T02:51:40.718707+0000","last_peered":"2024-03-19T02:51:40.718707+0000","last_clean":"2024-03-19T02:51:40.718707+0000","last_became_active":"2024-03-19T02:50:57.244886+0000","last_became_peered":"2024-03-19T02:50:57.244886+0000","last_unstale":"2024-03-19T02:51:40.718707+0000","last_undegraded":"2024-03-19T02:51:40.718707+0000","last_fullsized":"2024-03-19T02:51:40.718707+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:42:09.465738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"93'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719811+0000","last_change":"2024-03-19T02:50:51.797160+0000","last_active":"2024-03-19T02:51:40.719811+0000","last_peered":"2024-03-19T02:51:40.719811+0000","last_clean":"2024-03-19T02:51:40.719811+0000","last_became_active":"2024-03-19T02:50:51.796783+0000","last_became_peered":"2024-03-19T02:50:51.796783+0000","last_unstale":"2024-03-19T02:51:40.719811+0000","last_undegraded":"2024-03-19T02:51:40.719811+0000","last_fullsized":"2024-03-19T02:51:40.719811+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:57:08.185379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.716088+0000","last_change":"2024-03-19T02:50:52.225114+0000","last_active":"2024-03-19T02:51:40.716088+0000","last_peered":"2024-03-19T02:51:40.716088+0000","last_clean":"2024-03-19T02:51:40.716088+0000","last_became_active":"2024-03-19T02:50:52.220529+0000","last_became_peered":"2024-03-19T02:50:52.220529+0000","last_unstale":"2024-03-19T02:51:40.716088+0000","last_undegraded":"2024-03-19T02:51:40.716088+0000","last_fullsized":"2024-03-19T02:51:40.716088+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:12:51.408689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715363+0000","last_change":"2024-03-19T02:50:57.252827+0000","last_active":"2024-03-19T02:51:40.715363+0000","last_peered":"2024-03-19T02:51:40.715363+0000","last_clean":"2024-03-19T02:51:40.715363+0000","last_became_active":"2024-03-19T02:50:57.252502+0000","last_became_peered":"2024-03-19T02:50:57.252502+0000","last_unstale":"2024-03-19T02:51:40.715363+0000","last_undegraded":"2024-03-19T02:51:40.715363+0000","last_fullsized":"2024-03-19T02:51:40.715363+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:29:36.563533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"93'2","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.716120+0000","last_change":"2024-03-19T02:50:51.797380+0000","last_active":"2024-03-19T02:51:40.716120+0000","last_peered":"2024-03-19T02:51:40.716120+0000","last_clean":"2024-03-19T02:51:40.716120+0000","last_became_active":"2024-03-19T02:50:51.797021+0000","last_became_peered":"2024-03-19T02:50:51.797021+0000","last_unstale":"2024-03-19T02:51:40.716120+0000","last_undegraded":"2024-03-19T02:51:40.716120+0000","last_fullsized":"2024-03-19T02:51:40.716120+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:25.400360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.720551+0000","last_change":"2024-03-19T02:50:52.232717+0000","last_active":"2024-03-19T02:51:40.720551+0000","last_peered":"2024-03-19T02:51:40.720551+0000","last_clean":"2024-03-19T02:51:40.720551+0000","last_became_active":"2024-03-19T02:50:52.232368+0000","last_became_peered":"2024-03-19T02:50:52.232368+0000","last_unstale":"2024-03-19T02:51:40.720551+0000","last_undegraded":"2024-03-19T02:51:40.720551+0000","last_fullsized":"2024-03-19T02:51:40.720551+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:55:38.027120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715501+0000","last_change":"2024-03-19T02:50:57.250840+0000","last_active":"2024-03-19T02:51:40.715501+0000","last_peered":"2024-03-19T02:51:40.715501+0000","last_clean":"2024-03-19T02:51:40.715501+0000","last_became_active":"2024-03-19T02:50:57.250333+0000","last_became_peered":"2024-03-19T02:50:57.250333+0000","last_unstale":"2024-03-19T02:51:40.715501+0000","last_undegraded":"2024-03-19T02:51:40.715501+0000","last_fullsized":"2024-03-19T02:51:40.715501+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:57:09.089939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"93'4","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715764+0000","last_change":"2024-03-19T02:50:51.796889+0000","last_active":"2024-03-19T02:51:40.715764+0000","last_peered":"2024-03-19T02:51:40.715764+0000","last_clean":"2024-03-19T02:51:40.715764+0000","last_became_active":"2024-03-19T02:50:51.796570+0000","last_became_peered":"2024-03-19T02:50:51.796570+0000","last_unstale":"2024-03-19T02:51:40.715764+0000","last_undegraded":"2024-03-19T02:51:40.715764+0000","last_fullsized":"2024-03-19T02:51:40.715764+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:18:38.301905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.720023+0000","last_change":"2024-03-19T02:50:52.224533+0000","last_active":"2024-03-19T02:51:40.720023+0000","last_peered":"2024-03-19T02:51:40.720023+0000","last_clean":"2024-03-19T02:51:40.720023+0000","last_became_active":"2024-03-19T02:50:52.224221+0000","last_became_peered":"2024-03-19T02:50:52.224221+0000","last_unstale":"2024-03-19T02:51:40.720023+0000","last_undegraded":"2024-03-19T02:51:40.720023+0000","last_fullsized":"2024-03-19T02:51:40.720023+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:36:43.335318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715388+0000","last_change":"2024-03-19T02:50:57.252949+0000","last_active":"2024-03-19T02:51:40.715388+0000","last_peered":"2024-03-19T02:51:40.715388+0000","last_clean":"2024-03-19T02:51:40.715388+0000","last_became_active":"2024-03-19T02:50:57.252563+0000","last_became_peered":"2024-03-19T02:50:57.252563+0000","last_unstale":"2024-03-19T02:51:40.715388+0000","last_undegraded":"2024-03-19T02:51:40.715388+0000","last_fullsized":"2024-03-19T02:51:40.715388+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:30:38.244158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715620+0000","last_change":"2024-03-19T02:50:51.794594+0000","last_active":"2024-03-19T02:51:40.715620+0000","last_peered":"2024-03-19T02:51:40.715620+0000","last_clean":"2024-03-19T02:51:40.715620+0000","last_became_active":"2024-03-19T02:50:51.794197+0000","last_became_peered":"2024-03-19T02:50:51.794197+0000","last_unstale":"2024-03-19T02:51:40.715620+0000","last_undegraded":"2024-03-19T02:51:40.715620+0000","last_fullsized":"2024-03-19T02:51:40.715620+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:58:44.534106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.722212+0000","last_change":"2024-03-19T02:50:52.214167+0000","last_active":"2024-03-19T02:51:40.722212+0000","last_peered":"2024-03-19T02:51:40.722212+0000","last_clean":"2024-03-19T02:51:40.722212+0000","last_became_active":"2024-03-19T02:50:52.213873+0000","last_became_peered":"2024-03-19T02:50:52.213873+0000","last_unstale":"2024-03-19T02:51:40.722212+0000","last_undegraded":"2024-03-19T02:51:40.722212+0000","last_fullsized":"2024-03-19T02:51:40.722212+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:28:51.035477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719147+0000","last_change":"2024-03-19T02:50:57.245139+0000","last_active":"2024-03-19T02:51:40.719147+0000","last_peered":"2024-03-19T02:51:40.719147+0000","last_clean":"2024-03-19T02:51:40.719147+0000","last_became_active":"2024-03-19T02:50:57.244636+0000","last_became_peered":"2024-03-19T02:50:57.244636+0000","last_unstale":"2024-03-19T02:51:40.719147+0000","last_undegraded":"2024-03-19T02:51:40.719147+0000","last_fullsized":"2024-03-19T02:51:40.719147+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:44:29.660221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'5","reported_seq":105,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:52:24.152646+0000","last_change":"2024-03-19T02:50:51.795969+0000","last_active":"2024-03-19T02:52:24.152646+0000","last_peered":"2024-03-19T02:52:24.152646+0000","last_clean":"2024-03-19T02:52:24.152646+0000","last_became_active":"2024-03-19T02:50:51.795628+0000","last_became_peered":"2024-03-19T02:50:51.795628+0000","last_unstale":"2024-03-19T02:52:24.152646+0000","last_undegraded":"2024-03-19T02:52:24.152646+0000","last_fullsized":"2024-03-19T02:52:24.152646+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:14:48.463282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715506+0000","last_change":"2024-03-19T02:50:52.223278+0000","last_active":"2024-03-19T02:51:40.715506+0000","last_peered":"2024-03-19T02:51:40.715506+0000","last_clean":"2024-03-19T02:51:40.715506+0000","last_became_active":"2024-03-19T02:50:52.222741+0000","last_became_peered":"2024-03-19T02:50:52.222741+0000","last_unstale":"2024-03-19T02:51:40.715506+0000","last_undegraded":"2024-03-19T02:51:40.715506+0000","last_fullsized":"2024-03-19T02:51:40.715506+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:32:09.841825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719135+0000","last_change":"2024-03-19T02:50:57.253130+0000","last_active":"2024-03-19T02:51:40.719135+0000","last_peered":"2024-03-19T02:51:40.719135+0000","last_clean":"2024-03-19T02:51:40.719135+0000","last_became_active":"2024-03-19T02:50:57.252598+0000","last_became_peered":"2024-03-19T02:50:57.252598+0000","last_unstale":"2024-03-19T02:51:40.719135+0000","last_undegraded":"2024-03-19T02:51:40.719135+0000","last_fullsized":"2024-03-19T02:51:40.719135+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:09:38.183430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"93'3","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719095+0000","last_change":"2024-03-19T02:50:51.794075+0000","last_active":"2024-03-19T02:51:40.719095+0000","last_peered":"2024-03-19T02:51:40.719095+0000","last_clean":"2024-03-19T02:51:40.719095+0000","last_became_active":"2024-03-19T02:50:51.793707+0000","last_became_peered":"2024-03-19T02:50:51.793707+0000","last_unstale":"2024-03-19T02:51:40.719095+0000","last_undegraded":"2024-03-19T02:51:40.719095+0000","last_fullsized":"2024-03-19T02:51:40.719095+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:52:07.180162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.722243+0000","last_change":"2024-03-19T02:50:52.211112+0000","last_active":"2024-03-19T02:51:40.722243+0000","last_peered":"2024-03-19T02:51:40.722243+0000","last_clean":"2024-03-19T02:51:40.722243+0000","last_became_active":"2024-03-19T02:50:52.210790+0000","last_became_peered":"2024-03-19T02:50:52.210790+0000","last_unstale":"2024-03-19T02:51:40.722243+0000","last_undegraded":"2024-03-19T02:51:40.722243+0000","last_fullsized":"2024-03-19T02:51:40.722243+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:39:17.651585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":19,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.717585+0000","last_change":"2024-03-19T02:50:57.250397+0000","last_active":"2024-03-19T02:51:40.717585+0000","last_peered":"2024-03-19T02:51:40.717585+0000","last_clean":"2024-03-19T02:51:40.717585+0000","last_became_active":"2024-03-19T02:50:57.249968+0000","last_became_peered":"2024-03-19T02:50:57.249968+0000","last_unstale":"2024-03-19T02:51:40.717585+0000","last_undegraded":"2024-03-19T02:51:40.717585+0000","last_fullsized":"2024-03-19T02:51:40.717585+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:39.796799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"93'2","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.717379+0000","last_change":"2024-03-19T02:50:51.785347+0000","last_active":"2024-03-19T02:51:40.717379+0000","last_peered":"2024-03-19T02:51:40.717379+0000","last_clean":"2024-03-19T02:51:40.717379+0000","last_became_active":"2024-03-19T02:50:51.785004+0000","last_became_peered":"2024-03-19T02:50:51.785004+0000","last_unstale":"2024-03-19T02:51:40.717379+0000","last_undegraded":"2024-03-19T02:51:40.717379+0000","last_fullsized":"2024-03-19T02:51:40.717379+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:47.269503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715338+0000","last_change":"2024-03-19T02:50:52.221330+0000","last_active":"2024-03-19T02:51:40.715338+0000","last_peered":"2024-03-19T02:51:40.715338+0000","last_clean":"2024-03-19T02:51:40.715338+0000","last_became_active":"2024-03-19T02:50:52.219417+0000","last_became_peered":"2024-03-19T02:50:52.219417+0000","last_unstale":"2024-03-19T02:51:40.715338+0000","last_undegraded":"2024-03-19T02:51:40.715338+0000","last_fullsized":"2024-03-19T02:51:40.715338+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:28:37.856260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.720481+0000","last_change":"2024-03-19T02:50:57.258357+0000","last_active":"2024-03-19T02:51:40.720481+0000","last_peered":"2024-03-19T02:51:40.720481+0000","last_clean":"2024-03-19T02:51:40.720481+0000","last_became_active":"2024-03-19T02:50:57.257924+0000","last_became_peered":"2024-03-19T02:50:57.257924+0000","last_unstale":"2024-03-19T02:51:40.720481+0000","last_undegraded":"2024-03-19T02:51:40.720481+0000","last_fullsized":"2024-03-19T02:51:40.720481+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:46:05.340731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"93'1","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.723269+0000","last_change":"2024-03-19T02:50:51.796425+0000","last_active":"2024-03-19T02:51:40.723269+0000","last_peered":"2024-03-19T02:51:40.723269+0000","last_clean":"2024-03-19T02:51:40.723269+0000","last_became_active":"2024-03-19T02:50:51.796139+0000","last_became_peered":"2024-03-19T02:50:51.796139+0000","last_unstale":"2024-03-19T02:51:40.723269+0000","last_undegraded":"2024-03-19T02:51:40.723269+0000","last_fullsized":"2024-03-19T02:51:40.723269+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:12:17.123556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.717799+0000","last_change":"2024-03-19T02:50:52.219375+0000","last_active":"2024-03-19T02:51:40.717799+0000","last_peered":"2024-03-19T02:51:40.717799+0000","last_clean":"2024-03-19T02:51:40.717799+0000","last_became_active":"2024-03-19T02:50:52.219070+0000","last_became_peered":"2024-03-19T02:50:52.219070+0000","last_unstale":"2024-03-19T02:51:40.717799+0000","last_undegraded":"2024-03-19T02:51:40.717799+0000","last_fullsized":"2024-03-19T02:51:40.717799+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:31:58.847700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715282+0000","last_change":"2024-03-19T02:50:57.246502+0000","last_active":"2024-03-19T02:51:40.715282+0000","last_peered":"2024-03-19T02:51:40.715282+0000","last_clean":"2024-03-19T02:51:40.715282+0000","last_became_active":"2024-03-19T02:50:57.246089+0000","last_became_peered":"2024-03-19T02:50:57.246089+0000","last_unstale":"2024-03-19T02:51:40.715282+0000","last_undegraded":"2024-03-19T02:51:40.715282+0000","last_fullsized":"2024-03-19T02:51:40.715282+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:00.162308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'3","reported_seq":105,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:52:28.156316+0000","last_change":"2024-03-19T02:50:51.796448+0000","last_active":"2024-03-19T02:52:28.156316+0000","last_peered":"2024-03-19T02:52:28.156316+0000","last_clean":"2024-03-19T02:52:28.156316+0000","last_became_active":"2024-03-19T02:50:51.796132+0000","last_became_peered":"2024-03-19T02:50:51.796132+0000","last_unstale":"2024-03-19T02:52:28.156316+0000","last_undegraded":"2024-03-19T02:52:28.156316+0000","last_fullsized":"2024-03-19T02:52:28.156316+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:37:39.442772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":71,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.718528+0000","last_change":"2024-03-19T02:50:52.223022+0000","last_active":"2024-03-19T02:51:40.718528+0000","last_peered":"2024-03-19T02:51:40.718528+0000","last_clean":"2024-03-19T02:51:40.718528+0000","last_became_active":"2024-03-19T02:50:52.221655+0000","last_became_peered":"2024-03-19T02:50:52.221655+0000","last_unstale":"2024-03-19T02:51:40.718528+0000","last_undegraded":"2024-03-19T02:51:40.718528+0000","last_fullsized":"2024-03-19T02:51:40.718528+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:34:36.054088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.718116+0000","last_change":"2024-03-19T02:50:57.244003+0000","last_active":"2024-03-19T02:51:40.718116+0000","last_peered":"2024-03-19T02:51:40.718116+0000","last_clean":"2024-03-19T02:51:40.718116+0000","last_became_active":"2024-03-19T02:50:57.243567+0000","last_became_peered":"2024-03-19T02:50:57.243567+0000","last_unstale":"2024-03-19T02:51:40.718116+0000","last_undegraded":"2024-03-19T02:51:40.718116+0000","last_fullsized":"2024-03-19T02:51:40.718116+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:58:52.400237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"93'2","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719750+0000","last_change":"2024-03-19T02:50:51.801259+0000","last_active":"2024-03-19T02:51:40.719750+0000","last_peered":"2024-03-19T02:51:40.719750+0000","last_clean":"2024-03-19T02:51:40.719750+0000","last_became_active":"2024-03-19T02:50:51.800868+0000","last_became_peered":"2024-03-19T02:50:51.800868+0000","last_unstale":"2024-03-19T02:51:40.719750+0000","last_undegraded":"2024-03-19T02:51:40.719750+0000","last_fullsized":"2024-03-19T02:51:40.719750+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:44:06.486687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.724476+0000","last_change":"2024-03-19T02:51:40.724476+0000","last_active":"2024-03-19T02:51:40.724476+0000","last_peered":"2024-03-19T02:51:40.724476+0000","last_clean":"2024-03-19T02:51:40.724476+0000","last_became_active":"2024-03-19T02:51:40.724081+0000","last_became_peered":"2024-03-19T02:51:40.724081+0000","last_unstale":"2024-03-19T02:51:40.724476+0000","last_undegraded":"2024-03-19T02:51:40.724476+0000","last_fullsized":"2024-03-19T02:51:40.724476+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:04:15.684753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715647+0000","last_change":"2024-03-19T02:50:57.238764+0000","last_active":"2024-03-19T02:51:40.715647+0000","last_peered":"2024-03-19T02:51:40.715647+0000","last_clean":"2024-03-19T02:51:40.715647+0000","last_became_active":"2024-03-19T02:50:57.238401+0000","last_became_peered":"2024-03-19T02:50:57.238401+0000","last_unstale":"2024-03-19T02:51:40.715647+0000","last_undegraded":"2024-03-19T02:51:40.715647+0000","last_fullsized":"2024-03-19T02:51:40.715647+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:20:46.131776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"93'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.722102+0000","last_change":"2024-03-19T02:50:51.797633+0000","last_active":"2024-03-19T02:51:40.722102+0000","last_peered":"2024-03-19T02:51:40.722102+0000","last_clean":"2024-03-19T02:51:40.722102+0000","last_became_active":"2024-03-19T02:50:51.797313+0000","last_became_peered":"2024-03-19T02:50:51.797313+0000","last_unstale":"2024-03-19T02:51:40.722102+0000","last_undegraded":"2024-03-19T02:51:40.722102+0000","last_fullsized":"2024-03-19T02:51:40.722102+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:43:04.659301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.720178+0000","last_change":"2024-03-19T02:50:52.231486+0000","last_active":"2024-03-19T02:51:40.720178+0000","last_peered":"2024-03-19T02:51:40.720178+0000","last_clean":"2024-03-19T02:51:40.720178+0000","last_became_active":"2024-03-19T02:50:52.231172+0000","last_became_peered":"2024-03-19T02:50:52.231172+0000","last_unstale":"2024-03-19T02:51:40.720178+0000","last_undegraded":"2024-03-19T02:51:40.720178+0000","last_fullsized":"2024-03-19T02:51:40.720178+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:24:49.912633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.717867+0000","last_change":"2024-03-19T02:50:57.247207+0000","last_active":"2024-03-19T02:51:40.717867+0000","last_peered":"2024-03-19T02:51:40.717867+0000","last_clean":"2024-03-19T02:51:40.717867+0000","last_became_active":"2024-03-19T02:50:57.246776+0000","last_became_peered":"2024-03-19T02:50:57.246776+0000","last_unstale":"2024-03-19T02:51:40.717867+0000","last_undegraded":"2024-03-19T02:51:40.717867+0000","last_fullsized":"2024-03-19T02:51:40.717867+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:02:24.118858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"95'33","reported_seq":214,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:52:29.157221+0000","last_change":"2024-03-19T02:50:51.792398+0000","last_active":"2024-03-19T02:52:29.157221+0000","last_peered":"2024-03-19T02:52:29.157221+0000","last_clean":"2024-03-19T02:52:29.157221+0000","last_became_active":"2024-03-19T02:50:51.792090+0000","last_became_peered":"2024-03-19T02:50:51.792090+0000","last_unstale":"2024-03-19T02:52:29.157221+0000","last_undegraded":"2024-03-19T02:52:29.157221+0000","last_fullsized":"2024-03-19T02:52:29.157221+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:02:58.175210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63356,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":150,"num_read_kb":137,"num_write":34,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715596+0000","last_change":"2024-03-19T02:50:52.220755+0000","last_active":"2024-03-19T02:51:40.715596+0000","last_peered":"2024-03-19T02:51:40.715596+0000","last_clean":"2024-03-19T02:51:40.715596+0000","last_became_active":"2024-03-19T02:50:52.220453+0000","last_became_peered":"2024-03-19T02:50:52.220453+0000","last_unstale":"2024-03-19T02:51:40.715596+0000","last_undegraded":"2024-03-19T02:51:40.715596+0000","last_fullsized":"2024-03-19T02:51:40.715596+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:48:27.866863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.722499+0000","last_change":"2024-03-19T02:50:57.253843+0000","last_active":"2024-03-19T02:51:40.722499+0000","last_peered":"2024-03-19T02:51:40.722499+0000","last_clean":"2024-03-19T02:51:40.722499+0000","last_became_active":"2024-03-19T02:50:57.252633+0000","last_became_peered":"2024-03-19T02:50:57.252633+0000","last_unstale":"2024-03-19T02:51:40.722499+0000","last_undegraded":"2024-03-19T02:51:40.722499+0000","last_fullsized":"2024-03-19T02:51:40.722499+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:03:48.514219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"93'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719004+0000","last_change":"2024-03-19T02:50:51.795041+0000","last_active":"2024-03-19T02:51:40.719004+0000","last_peered":"2024-03-19T02:51:40.719004+0000","last_clean":"2024-03-19T02:51:40.719004+0000","last_became_active":"2024-03-19T02:50:51.793485+0000","last_became_peered":"2024-03-19T02:50:51.793485+0000","last_unstale":"2024-03-19T02:51:40.719004+0000","last_undegraded":"2024-03-19T02:51:40.719004+0000","last_fullsized":"2024-03-19T02:51:40.719004+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:11:50.136394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719177+0000","last_change":"2024-03-19T02:50:52.227554+0000","last_active":"2024-03-19T02:51:40.719177+0000","last_peered":"2024-03-19T02:51:40.719177+0000","last_clean":"2024-03-19T02:51:40.719177+0000","last_became_active":"2024-03-19T02:50:52.227222+0000","last_became_peered":"2024-03-19T02:50:52.227222+0000","last_unstale":"2024-03-19T02:51:40.719177+0000","last_undegraded":"2024-03-19T02:51:40.719177+0000","last_fullsized":"2024-03-19T02:51:40.719177+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:56.323749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":19,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.717521+0000","last_change":"2024-03-19T02:50:57.257504+0000","last_active":"2024-03-19T02:51:40.717521+0000","last_peered":"2024-03-19T02:51:40.717521+0000","last_clean":"2024-03-19T02:51:40.717521+0000","last_became_active":"2024-03-19T02:50:57.257120+0000","last_became_peered":"2024-03-19T02:50:57.257120+0000","last_unstale":"2024-03-19T02:51:40.717521+0000","last_undegraded":"2024-03-19T02:51:40.717521+0000","last_fullsized":"2024-03-19T02:51:40.717521+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:19:10.471666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"93'3","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.722290+0000","last_change":"2024-03-19T02:50:51.798049+0000","last_active":"2024-03-19T02:51:40.722290+0000","last_peered":"2024-03-19T02:51:40.722290+0000","last_clean":"2024-03-19T02:51:40.722290+0000","last_became_active":"2024-03-19T02:50:51.797134+0000","last_became_peered":"2024-03-19T02:50:51.797134+0000","last_unstale":"2024-03-19T02:51:40.722290+0000","last_undegraded":"2024-03-19T02:51:40.722290+0000","last_fullsized":"2024-03-19T02:51:40.722290+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:03:46.984240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719601+0000","last_change":"2024-03-19T02:50:52.219535+0000","last_active":"2024-03-19T02:51:40.719601+0000","last_peered":"2024-03-19T02:51:40.719601+0000","last_clean":"2024-03-19T02:51:40.719601+0000","last_became_active":"2024-03-19T02:50:52.219226+0000","last_became_peered":"2024-03-19T02:50:52.219226+0000","last_unstale":"2024-03-19T02:51:40.719601+0000","last_undegraded":"2024-03-19T02:51:40.719601+0000","last_fullsized":"2024-03-19T02:51:40.719601+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:27:26.875533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719542+0000","last_change":"2024-03-19T02:50:57.252873+0000","last_active":"2024-03-19T02:51:40.719542+0000","last_peered":"2024-03-19T02:51:40.719542+0000","last_clean":"2024-03-19T02:51:40.719542+0000","last_became_active":"2024-03-19T02:50:57.249551+0000","last_became_peered":"2024-03-19T02:50:57.249551+0000","last_unstale":"2024-03-19T02:51:40.719542+0000","last_undegraded":"2024-03-19T02:51:40.719542+0000","last_fullsized":"2024-03-19T02:51:40.719542+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:33:59.750606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"93'3","reported_seq":102,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:52:24.164485+0000","last_change":"2024-03-19T02:50:51.796478+0000","last_active":"2024-03-19T02:52:24.164485+0000","last_peered":"2024-03-19T02:52:24.164485+0000","last_clean":"2024-03-19T02:52:24.164485+0000","last_became_active":"2024-03-19T02:50:51.796126+0000","last_became_peered":"2024-03-19T02:50:51.796126+0000","last_unstale":"2024-03-19T02:52:24.164485+0000","last_undegraded":"2024-03-19T02:52:24.164485+0000","last_fullsized":"2024-03-19T02:52:24.164485+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:13:42.690481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719337+0000","last_change":"2024-03-19T02:50:52.228151+0000","last_active":"2024-03-19T02:51:40.719337+0000","last_peered":"2024-03-19T02:51:40.719337+0000","last_clean":"2024-03-19T02:51:40.719337+0000","last_became_active":"2024-03-19T02:50:52.227858+0000","last_became_peered":"2024-03-19T02:50:52.227858+0000","last_unstale":"2024-03-19T02:51:40.719337+0000","last_undegraded":"2024-03-19T02:51:40.719337+0000","last_fullsized":"2024-03-19T02:51:40.719337+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:10.285606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719376+0000","last_change":"2024-03-19T02:50:57.259269+0000","last_active":"2024-03-19T02:51:40.719376+0000","last_peered":"2024-03-19T02:51:40.719376+0000","last_clean":"2024-03-19T02:51:40.719376+0000","last_became_active":"2024-03-19T02:50:57.258778+0000","last_became_peered":"2024-03-19T02:50:57.258778+0000","last_unstale":"2024-03-19T02:51:40.719376+0000","last_undegraded":"2024-03-19T02:51:40.719376+0000","last_fullsized":"2024-03-19T02:51:40.719376+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:02:44.419512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"93'15","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.739965+0000","last_change":"2024-03-19T02:51:40.739522+0000","last_active":"2024-03-19T02:51:40.739965+0000","last_peered":"2024-03-19T02:51:40.739965+0000","last_clean":"2024-03-19T02:51:40.739965+0000","last_became_active":"2024-03-19T02:51:40.720675+0000","last_became_peered":"2024-03-19T02:51:40.720675+0000","last_unstale":"2024-03-19T02:51:40.739965+0000","last_undegraded":"2024-03-19T02:51:40.739965+0000","last_fullsized":"2024-03-19T02:51:40.739965+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:18:19.449850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.723163+0000","last_change":"2024-03-19T02:50:52.215997+0000","last_active":"2024-03-19T02:51:40.723163+0000","last_peered":"2024-03-19T02:51:40.723163+0000","last_clean":"2024-03-19T02:51:40.723163+0000","last_became_active":"2024-03-19T02:50:52.215398+0000","last_became_peered":"2024-03-19T02:50:52.215398+0000","last_unstale":"2024-03-19T02:51:40.723163+0000","last_undegraded":"2024-03-19T02:51:40.723163+0000","last_fullsized":"2024-03-19T02:51:40.723163+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:49:14.010736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.723079+0000","last_change":"2024-03-19T02:50:57.247254+0000","last_active":"2024-03-19T02:51:40.723079+0000","last_peered":"2024-03-19T02:51:40.723079+0000","last_clean":"2024-03-19T02:51:40.723079+0000","last_became_active":"2024-03-19T02:50:57.246861+0000","last_became_peered":"2024-03-19T02:50:57.246861+0000","last_unstale":"2024-03-19T02:51:40.723079+0000","last_undegraded":"2024-03-19T02:51:40.723079+0000","last_fullsized":"2024-03-19T02:51:40.723079+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:47:20.933550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"93'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715688+0000","last_change":"2024-03-19T02:50:51.796246+0000","last_active":"2024-03-19T02:51:40.715688+0000","last_peered":"2024-03-19T02:51:40.715688+0000","last_clean":"2024-03-19T02:51:40.715688+0000","last_became_active":"2024-03-19T02:50:51.795823+0000","last_became_peered":"2024-03-19T02:50:51.795823+0000","last_unstale":"2024-03-19T02:51:40.715688+0000","last_undegraded":"2024-03-19T02:51:40.715688+0000","last_fullsized":"2024-03-19T02:51:40.715688+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:36:02.220891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.717740+0000","last_change":"2024-03-19T02:50:52.224119+0000","last_active":"2024-03-19T02:51:40.717740+0000","last_peered":"2024-03-19T02:51:40.717740+0000","last_clean":"2024-03-19T02:51:40.717740+0000","last_became_active":"2024-03-19T02:50:52.223773+0000","last_became_peered":"2024-03-19T02:50:52.223773+0000","last_unstale":"2024-03-19T02:51:40.717740+0000","last_undegraded":"2024-03-19T02:51:40.717740+0000","last_fullsized":"2024-03-19T02:51:40.717740+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:56:03.098858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715398+0000","last_change":"2024-03-19T02:50:57.652515+0000","last_active":"2024-03-19T02:51:40.715398+0000","last_peered":"2024-03-19T02:51:40.715398+0000","last_clean":"2024-03-19T02:51:40.715398+0000","last_became_active":"2024-03-19T02:50:57.651808+0000","last_became_peered":"2024-03-19T02:50:57.651808+0000","last_unstale":"2024-03-19T02:51:40.715398+0000","last_undegraded":"2024-03-19T02:51:40.715398+0000","last_fullsized":"2024-03-19T02:51:40.715398+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:16:34.550457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.718884+0000","last_change":"2024-03-19T02:50:51.794282+0000","last_active":"2024-03-19T02:51:40.718884+0000","last_peered":"2024-03-19T02:51:40.718884+0000","last_clean":"2024-03-19T02:51:40.718884+0000","last_became_active":"2024-03-19T02:50:51.793947+0000","last_became_peered":"2024-03-19T02:50:51.793947+0000","last_unstale":"2024-03-19T02:51:40.718884+0000","last_undegraded":"2024-03-19T02:51:40.718884+0000","last_fullsized":"2024-03-19T02:51:40.718884+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:08:54.764915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.718639+0000","last_change":"2024-03-19T02:50:52.221707+0000","last_active":"2024-03-19T02:51:40.718639+0000","last_peered":"2024-03-19T02:51:40.718639+0000","last_clean":"2024-03-19T02:51:40.718639+0000","last_became_active":"2024-03-19T02:50:52.221417+0000","last_became_peered":"2024-03-19T02:50:52.221417+0000","last_unstale":"2024-03-19T02:51:40.718639+0000","last_undegraded":"2024-03-19T02:51:40.718639+0000","last_fullsized":"2024-03-19T02:51:40.718639+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:59:27.528346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719877+0000","last_change":"2024-03-19T02:50:57.257552+0000","last_active":"2024-03-19T02:51:40.719877+0000","last_peered":"2024-03-19T02:51:40.719877+0000","last_clean":"2024-03-19T02:51:40.719877+0000","last_became_active":"2024-03-19T02:50:57.257121+0000","last_became_peered":"2024-03-19T02:50:57.257121+0000","last_unstale":"2024-03-19T02:51:40.719877+0000","last_undegraded":"2024-03-19T02:51:40.719877+0000","last_fullsized":"2024-03-19T02:51:40.719877+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:43:47.145344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"93'1","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.718626+0000","last_change":"2024-03-19T02:50:51.804890+0000","last_active":"2024-03-19T02:51:40.718626+0000","last_peered":"2024-03-19T02:51:40.718626+0000","last_clean":"2024-03-19T02:51:40.718626+0000","last_became_active":"2024-03-19T02:50:51.804545+0000","last_became_peered":"2024-03-19T02:50:51.804545+0000","last_unstale":"2024-03-19T02:51:40.718626+0000","last_undegraded":"2024-03-19T02:51:40.718626+0000","last_fullsized":"2024-03-19T02:51:40.718626+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:16:14.945372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.718269+0000","last_change":"2024-03-19T02:50:52.224849+0000","last_active":"2024-03-19T02:51:40.718269+0000","last_peered":"2024-03-19T02:51:40.718269+0000","last_clean":"2024-03-19T02:51:40.718269+0000","last_became_active":"2024-03-19T02:50:52.224550+0000","last_became_peered":"2024-03-19T02:50:52.224550+0000","last_unstale":"2024-03-19T02:51:40.718269+0000","last_undegraded":"2024-03-19T02:51:40.718269+0000","last_fullsized":"2024-03-19T02:51:40.718269+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:28:59.683849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.720381+0000","last_change":"2024-03-19T02:50:57.652425+0000","last_active":"2024-03-19T02:51:40.720381+0000","last_peered":"2024-03-19T02:51:40.720381+0000","last_clean":"2024-03-19T02:51:40.720381+0000","last_became_active":"2024-03-19T02:50:57.651975+0000","last_became_peered":"2024-03-19T02:50:57.651975+0000","last_unstale":"2024-03-19T02:51:40.720381+0000","last_undegraded":"2024-03-19T02:51:40.720381+0000","last_fullsized":"2024-03-19T02:51:40.720381+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:07:20.587918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.723382+0000","last_change":"2024-03-19T02:50:52.218207+0000","last_active":"2024-03-19T02:51:40.723382+0000","last_peered":"2024-03-19T02:51:40.723382+0000","last_clean":"2024-03-19T02:51:40.723382+0000","last_became_active":"2024-03-19T02:50:52.217012+0000","last_became_peered":"2024-03-19T02:50:52.217012+0000","last_unstale":"2024-03-19T02:51:40.723382+0000","last_undegraded":"2024-03-19T02:51:40.723382+0000","last_fullsized":"2024-03-19T02:51:40.723382+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:07:47.366481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'4","reported_seq":101,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:52:24.164157+0000","last_change":"2024-03-19T02:50:51.801405+0000","last_active":"2024-03-19T02:52:24.164157+0000","last_peered":"2024-03-19T02:52:24.164157+0000","last_clean":"2024-03-19T02:52:24.164157+0000","last_became_active":"2024-03-19T02:50:51.801094+0000","last_became_peered":"2024-03-19T02:50:51.801094+0000","last_unstale":"2024-03-19T02:52:24.164157+0000","last_undegraded":"2024-03-19T02:52:24.164157+0000","last_fullsized":"2024-03-19T02:52:24.164157+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:53:39.534977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":67,"num_read_kb":67,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"93'1","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.730836+0000","last_change":"2024-03-19T02:51:40.730836+0000","last_active":"2024-03-19T02:51:40.730836+0000","last_peered":"2024-03-19T02:51:40.730836+0000","last_clean":"2024-03-19T02:51:40.730836+0000","last_became_active":"2024-03-19T02:51:40.723984+0000","last_became_peered":"2024-03-19T02:51:40.723984+0000","last_unstale":"2024-03-19T02:51:40.730836+0000","last_undegraded":"2024-03-19T02:51:40.730836+0000","last_fullsized":"2024-03-19T02:51:40.730836+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:29:57.301066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715624+0000","last_change":"2024-03-19T02:50:52.221852+0000","last_active":"2024-03-19T02:51:40.715624+0000","last_peered":"2024-03-19T02:51:40.715624+0000","last_clean":"2024-03-19T02:51:40.715624+0000","last_became_active":"2024-03-19T02:50:52.221535+0000","last_became_peered":"2024-03-19T02:50:52.221535+0000","last_unstale":"2024-03-19T02:51:40.715624+0000","last_undegraded":"2024-03-19T02:51:40.715624+0000","last_fullsized":"2024-03-19T02:51:40.715624+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:37:18.030714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"93'3","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715607+0000","last_change":"2024-03-19T02:50:51.790231+0000","last_active":"2024-03-19T02:51:40.715607+0000","last_peered":"2024-03-19T02:51:40.715607+0000","last_clean":"2024-03-19T02:51:40.715607+0000","last_became_active":"2024-03-19T02:50:51.789867+0000","last_became_peered":"2024-03-19T02:50:51.789867+0000","last_unstale":"2024-03-19T02:51:40.715607+0000","last_undegraded":"2024-03-19T02:51:40.715607+0000","last_fullsized":"2024-03-19T02:51:40.715607+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:46.099191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":366,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719128+0000","last_change":"2024-03-19T02:49:38.697673+0000","last_active":"2024-03-19T02:51:40.719128+0000","last_peered":"2024-03-19T02:51:40.719128+0000","last_clean":"2024-03-19T02:51:40.719128+0000","last_became_active":"2024-03-19T02:49:38.697161+0000","last_became_peered":"2024-03-19T02:49:38.697161+0000","last_unstale":"2024-03-19T02:51:40.719128+0000","last_undegraded":"2024-03-19T02:51:40.719128+0000","last_fullsized":"2024-03-19T02:51:40.719128+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:46:10.013962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:46:10.013962+0000","last_clean_scrub_stamp":"2024-03-19T02:46:10.013962+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:59:35.179938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.722715+0000","last_change":"2024-03-19T02:50:57.652428+0000","last_active":"2024-03-19T02:51:40.722715+0000","last_peered":"2024-03-19T02:51:40.722715+0000","last_clean":"2024-03-19T02:51:40.722715+0000","last_became_active":"2024-03-19T02:50:57.651864+0000","last_became_peered":"2024-03-19T02:50:57.651864+0000","last_unstale":"2024-03-19T02:51:40.722715+0000","last_undegraded":"2024-03-19T02:51:40.722715+0000","last_fullsized":"2024-03-19T02:51:40.722715+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:52:19.007175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.717092+0000","last_change":"2024-03-19T02:50:52.218728+0000","last_active":"2024-03-19T02:51:40.717092+0000","last_peered":"2024-03-19T02:51:40.717092+0000","last_clean":"2024-03-19T02:51:40.717092+0000","last_became_active":"2024-03-19T02:50:52.218407+0000","last_became_peered":"2024-03-19T02:50:52.218407+0000","last_unstale":"2024-03-19T02:51:40.717092+0000","last_undegraded":"2024-03-19T02:51:40.717092+0000","last_fullsized":"2024-03-19T02:51:40.717092+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:30:49.304034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"93'3","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719242+0000","last_change":"2024-03-19T02:50:51.802057+0000","last_active":"2024-03-19T02:51:40.719242+0000","last_peered":"2024-03-19T02:51:40.719242+0000","last_clean":"2024-03-19T02:51:40.719242+0000","last_became_active":"2024-03-19T02:50:51.801754+0000","last_became_peered":"2024-03-19T02:50:51.801754+0000","last_unstale":"2024-03-19T02:51:40.719242+0000","last_undegraded":"2024-03-19T02:51:40.719242+0000","last_fullsized":"2024-03-19T02:51:40.719242+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:23:15.401822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":19,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.717091+0000","last_change":"2024-03-19T02:50:57.260303+0000","last_active":"2024-03-19T02:51:40.717091+0000","last_peered":"2024-03-19T02:51:40.717091+0000","last_clean":"2024-03-19T02:51:40.717091+0000","last_became_active":"2024-03-19T02:50:57.259792+0000","last_became_peered":"2024-03-19T02:50:57.259792+0000","last_unstale":"2024-03-19T02:51:40.717091+0000","last_undegraded":"2024-03-19T02:51:40.717091+0000","last_fullsized":"2024-03-19T02:51:40.717091+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:03.072161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"93'1","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715677+0000","last_change":"2024-03-19T02:50:51.797949+0000","last_active":"2024-03-19T02:51:40.715677+0000","last_peered":"2024-03-19T02:51:40.715677+0000","last_clean":"2024-03-19T02:51:40.715677+0000","last_became_active":"2024-03-19T02:50:51.797522+0000","last_became_peered":"2024-03-19T02:50:51.797522+0000","last_unstale":"2024-03-19T02:51:40.715677+0000","last_undegraded":"2024-03-19T02:51:40.715677+0000","last_fullsized":"2024-03-19T02:51:40.715677+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:34:06.127993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.720131+0000","last_change":"2024-03-19T02:50:52.224095+0000","last_active":"2024-03-19T02:51:40.720131+0000","last_peered":"2024-03-19T02:51:40.720131+0000","last_clean":"2024-03-19T02:51:40.720131+0000","last_became_active":"2024-03-19T02:50:52.223764+0000","last_became_peered":"2024-03-19T02:50:52.223764+0000","last_unstale":"2024-03-19T02:51:40.720131+0000","last_undegraded":"2024-03-19T02:51:40.720131+0000","last_fullsized":"2024-03-19T02:51:40.720131+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:07:41.503162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715305+0000","last_change":"2024-03-19T02:50:57.652274+0000","last_active":"2024-03-19T02:51:40.715305+0000","last_peered":"2024-03-19T02:51:40.715305+0000","last_clean":"2024-03-19T02:51:40.715305+0000","last_became_active":"2024-03-19T02:50:57.651885+0000","last_became_peered":"2024-03-19T02:50:57.651885+0000","last_unstale":"2024-03-19T02:51:40.715305+0000","last_undegraded":"2024-03-19T02:51:40.715305+0000","last_fullsized":"2024-03-19T02:51:40.715305+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:03:02.081143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'8","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719329+0000","last_change":"2024-03-19T02:50:51.796308+0000","last_active":"2024-03-19T02:51:40.719329+0000","last_peered":"2024-03-19T02:51:40.719329+0000","last_clean":"2024-03-19T02:51:40.719329+0000","last_became_active":"2024-03-19T02:50:51.795990+0000","last_became_peered":"2024-03-19T02:50:51.795990+0000","last_unstale":"2024-03-19T02:51:40.719329+0000","last_undegraded":"2024-03-19T02:51:40.719329+0000","last_fullsized":"2024-03-19T02:51:40.719329+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:45:53.608817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.716308+0000","last_change":"2024-03-19T02:50:52.221983+0000","last_active":"2024-03-19T02:51:40.716308+0000","last_peered":"2024-03-19T02:51:40.716308+0000","last_clean":"2024-03-19T02:51:40.716308+0000","last_became_active":"2024-03-19T02:50:52.220605+0000","last_became_peered":"2024-03-19T02:50:52.220605+0000","last_unstale":"2024-03-19T02:51:40.716308+0000","last_undegraded":"2024-03-19T02:51:40.716308+0000","last_fullsized":"2024-03-19T02:51:40.716308+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:12.834159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.718148+0000","last_change":"2024-03-19T02:50:57.242428+0000","last_active":"2024-03-19T02:51:40.718148+0000","last_peered":"2024-03-19T02:51:40.718148+0000","last_clean":"2024-03-19T02:51:40.718148+0000","last_became_active":"2024-03-19T02:50:57.242007+0000","last_became_peered":"2024-03-19T02:50:57.242007+0000","last_unstale":"2024-03-19T02:51:40.718148+0000","last_undegraded":"2024-03-19T02:51:40.718148+0000","last_fullsized":"2024-03-19T02:51:40.718148+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:23:30.987761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"93'4","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715977+0000","last_change":"2024-03-19T02:50:51.791342+0000","last_active":"2024-03-19T02:51:40.715977+0000","last_peered":"2024-03-19T02:51:40.715977+0000","last_clean":"2024-03-19T02:51:40.715977+0000","last_became_active":"2024-03-19T02:50:51.791039+0000","last_became_peered":"2024-03-19T02:50:51.791039+0000","last_unstale":"2024-03-19T02:51:40.715977+0000","last_undegraded":"2024-03-19T02:51:40.715977+0000","last_fullsized":"2024-03-19T02:51:40.715977+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:50:38.216623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.720588+0000","last_change":"2024-03-19T02:50:52.224303+0000","last_active":"2024-03-19T02:51:40.720588+0000","last_peered":"2024-03-19T02:51:40.720588+0000","last_clean":"2024-03-19T02:51:40.720588+0000","last_became_active":"2024-03-19T02:50:52.223942+0000","last_became_peered":"2024-03-19T02:50:52.223942+0000","last_unstale":"2024-03-19T02:51:40.720588+0000","last_undegraded":"2024-03-19T02:51:40.720588+0000","last_fullsized":"2024-03-19T02:51:40.720588+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:57:04.175971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715762+0000","last_change":"2024-03-19T02:50:57.652126+0000","last_active":"2024-03-19T02:51:40.715762+0000","last_peered":"2024-03-19T02:51:40.715762+0000","last_clean":"2024-03-19T02:51:40.715762+0000","last_became_active":"2024-03-19T02:50:57.651713+0000","last_became_peered":"2024-03-19T02:50:57.651713+0000","last_unstale":"2024-03-19T02:51:40.715762+0000","last_undegraded":"2024-03-19T02:51:40.715762+0000","last_fullsized":"2024-03-19T02:51:40.715762+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:28:51.077437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"93'2","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.741378+0000","last_change":"2024-03-19T02:51:40.741378+0000","last_active":"2024-03-19T02:51:40.741378+0000","last_peered":"2024-03-19T02:51:40.741378+0000","last_clean":"2024-03-19T02:51:40.741378+0000","last_became_active":"2024-03-19T02:51:40.724510+0000","last_became_peered":"2024-03-19T02:51:40.724510+0000","last_unstale":"2024-03-19T02:51:40.741378+0000","last_undegraded":"2024-03-19T02:51:40.741378+0000","last_fullsized":"2024-03-19T02:51:40.741378+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:30:16.228635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715268+0000","last_change":"2024-03-19T02:50:52.221681+0000","last_active":"2024-03-19T02:51:40.715268+0000","last_peered":"2024-03-19T02:51:40.715268+0000","last_clean":"2024-03-19T02:51:40.715268+0000","last_became_active":"2024-03-19T02:50:52.220987+0000","last_became_peered":"2024-03-19T02:50:52.220987+0000","last_unstale":"2024-03-19T02:51:40.715268+0000","last_undegraded":"2024-03-19T02:51:40.715268+0000","last_fullsized":"2024-03-19T02:51:40.715268+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:16:36.561143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719714+0000","last_change":"2024-03-19T02:50:57.249830+0000","last_active":"2024-03-19T02:51:40.719714+0000","last_peered":"2024-03-19T02:51:40.719714+0000","last_clean":"2024-03-19T02:51:40.719714+0000","last_became_active":"2024-03-19T02:50:57.248879+0000","last_became_peered":"2024-03-19T02:50:57.248879+0000","last_unstale":"2024-03-19T02:51:40.719714+0000","last_undegraded":"2024-03-19T02:51:40.719714+0000","last_fullsized":"2024-03-19T02:51:40.719714+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:58:43.225841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"93'3","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715344+0000","last_change":"2024-03-19T02:50:51.801374+0000","last_active":"2024-03-19T02:51:40.715344+0000","last_peered":"2024-03-19T02:51:40.715344+0000","last_clean":"2024-03-19T02:51:40.715344+0000","last_became_active":"2024-03-19T02:50:51.801041+0000","last_became_peered":"2024-03-19T02:50:51.801041+0000","last_unstale":"2024-03-19T02:51:40.715344+0000","last_undegraded":"2024-03-19T02:51:40.715344+0000","last_fullsized":"2024-03-19T02:51:40.715344+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:56:14.671718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.717377+0000","last_change":"2024-03-19T02:50:52.225915+0000","last_active":"2024-03-19T02:51:40.717377+0000","last_peered":"2024-03-19T02:51:40.717377+0000","last_clean":"2024-03-19T02:51:40.717377+0000","last_became_active":"2024-03-19T02:50:52.225606+0000","last_became_peered":"2024-03-19T02:50:52.225606+0000","last_unstale":"2024-03-19T02:51:40.717377+0000","last_undegraded":"2024-03-19T02:51:40.717377+0000","last_fullsized":"2024-03-19T02:51:40.717377+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:27:29.549255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"93'1","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.716380+0000","last_change":"2024-03-19T02:50:57.240747+0000","last_active":"2024-03-19T02:51:40.716380+0000","last_peered":"2024-03-19T02:51:40.716380+0000","last_clean":"2024-03-19T02:51:40.716380+0000","last_became_active":"2024-03-19T02:50:57.240373+0000","last_became_peered":"2024-03-19T02:50:57.240373+0000","last_unstale":"2024-03-19T02:51:40.716380+0000","last_undegraded":"2024-03-19T02:51:40.716380+0000","last_fullsized":"2024-03-19T02:51:40.716380+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:22:04.991497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"93'1","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.718520+0000","last_change":"2024-03-19T02:50:51.794071+0000","last_active":"2024-03-19T02:51:40.718520+0000","last_peered":"2024-03-19T02:51:40.718520+0000","last_clean":"2024-03-19T02:51:40.718520+0000","last_became_active":"2024-03-19T02:50:51.793570+0000","last_became_peered":"2024-03-19T02:50:51.793570+0000","last_unstale":"2024-03-19T02:51:40.718520+0000","last_undegraded":"2024-03-19T02:51:40.718520+0000","last_fullsized":"2024-03-19T02:51:40.718520+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:34:20.456515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715731+0000","last_change":"2024-03-19T02:50:52.223341+0000","last_active":"2024-03-19T02:51:40.715731+0000","last_peered":"2024-03-19T02:51:40.715731+0000","last_clean":"2024-03-19T02:51:40.715731+0000","last_became_active":"2024-03-19T02:50:52.220280+0000","last_became_peered":"2024-03-19T02:50:52.220280+0000","last_unstale":"2024-03-19T02:51:40.715731+0000","last_undegraded":"2024-03-19T02:51:40.715731+0000","last_fullsized":"2024-03-19T02:51:40.715731+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:04:15.830811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715695+0000","last_change":"2024-03-19T02:50:57.651293+0000","last_active":"2024-03-19T02:51:40.715695+0000","last_peered":"2024-03-19T02:51:40.715695+0000","last_clean":"2024-03-19T02:51:40.715695+0000","last_became_active":"2024-03-19T02:50:57.650893+0000","last_became_peered":"2024-03-19T02:50:57.650893+0000","last_unstale":"2024-03-19T02:51:40.715695+0000","last_undegraded":"2024-03-19T02:51:40.715695+0000","last_fullsized":"2024-03-19T02:51:40.715695+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:33:35.984728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"93'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715348+0000","last_change":"2024-03-19T02:50:51.798297+0000","last_active":"2024-03-19T02:51:40.715348+0000","last_peered":"2024-03-19T02:51:40.715348+0000","last_clean":"2024-03-19T02:51:40.715348+0000","last_became_active":"2024-03-19T02:50:51.797516+0000","last_became_peered":"2024-03-19T02:50:51.797516+0000","last_unstale":"2024-03-19T02:51:40.715348+0000","last_undegraded":"2024-03-19T02:51:40.715348+0000","last_fullsized":"2024-03-19T02:51:40.715348+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:09:42.295094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.718736+0000","last_change":"2024-03-19T02:50:52.224011+0000","last_active":"2024-03-19T02:51:40.718736+0000","last_peered":"2024-03-19T02:51:40.718736+0000","last_clean":"2024-03-19T02:51:40.718736+0000","last_became_active":"2024-03-19T02:50:52.223708+0000","last_became_peered":"2024-03-19T02:50:52.223708+0000","last_unstale":"2024-03-19T02:51:40.718736+0000","last_undegraded":"2024-03-19T02:51:40.718736+0000","last_fullsized":"2024-03-19T02:51:40.718736+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:17:49.305587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719502+0000","last_change":"2024-03-19T02:50:57.251134+0000","last_active":"2024-03-19T02:51:40.719502+0000","last_peered":"2024-03-19T02:51:40.719502+0000","last_clean":"2024-03-19T02:51:40.719502+0000","last_became_active":"2024-03-19T02:50:57.250761+0000","last_became_peered":"2024-03-19T02:50:57.250761+0000","last_unstale":"2024-03-19T02:51:40.719502+0000","last_undegraded":"2024-03-19T02:51:40.719502+0000","last_fullsized":"2024-03-19T02:51:40.719502+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:09:26.860235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"93'3","reported_seq":17,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.728077+0000","last_change":"2024-03-19T02:51:40.728077+0000","last_active":"2024-03-19T02:51:40.728077+0000","last_peered":"2024-03-19T02:51:40.728077+0000","last_clean":"2024-03-19T02:51:40.728077+0000","last_became_active":"2024-03-19T02:51:40.720767+0000","last_became_peered":"2024-03-19T02:51:40.720767+0000","last_unstale":"2024-03-19T02:51:40.728077+0000","last_undegraded":"2024-03-19T02:51:40.728077+0000","last_fullsized":"2024-03-19T02:51:40.728077+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:22:34.366373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.720568+0000","last_change":"2024-03-19T02:50:52.224776+0000","last_active":"2024-03-19T02:51:40.720568+0000","last_peered":"2024-03-19T02:51:40.720568+0000","last_clean":"2024-03-19T02:51:40.720568+0000","last_became_active":"2024-03-19T02:50:52.224442+0000","last_became_peered":"2024-03-19T02:50:52.224442+0000","last_unstale":"2024-03-19T02:51:40.720568+0000","last_undegraded":"2024-03-19T02:51:40.720568+0000","last_fullsized":"2024-03-19T02:51:40.720568+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:11:13.722005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.722732+0000","last_change":"2024-03-19T02:51:40.722732+0000","last_active":"2024-03-19T02:51:40.722732+0000","last_peered":"2024-03-19T02:51:40.722732+0000","last_clean":"2024-03-19T02:51:40.722732+0000","last_became_active":"2024-03-19T02:51:40.722260+0000","last_became_peered":"2024-03-19T02:51:40.722260+0000","last_unstale":"2024-03-19T02:51:40.722732+0000","last_undegraded":"2024-03-19T02:51:40.722732+0000","last_fullsized":"2024-03-19T02:51:40.722732+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:31:06.383570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.722310+0000","last_change":"2024-03-19T02:50:51.794898+0000","last_active":"2024-03-19T02:51:40.722310+0000","last_peered":"2024-03-19T02:51:40.722310+0000","last_clean":"2024-03-19T02:51:40.722310+0000","last_became_active":"2024-03-19T02:50:51.794562+0000","last_became_peered":"2024-03-19T02:50:51.794562+0000","last_unstale":"2024-03-19T02:51:40.722310+0000","last_undegraded":"2024-03-19T02:51:40.722310+0000","last_fullsized":"2024-03-19T02:51:40.722310+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:17:45.239395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719705+0000","last_change":"2024-03-19T02:50:52.227427+0000","last_active":"2024-03-19T02:51:40.719705+0000","last_peered":"2024-03-19T02:51:40.719705+0000","last_clean":"2024-03-19T02:51:40.719705+0000","last_became_active":"2024-03-19T02:50:52.227111+0000","last_became_peered":"2024-03-19T02:50:52.227111+0000","last_unstale":"2024-03-19T02:51:40.719705+0000","last_undegraded":"2024-03-19T02:51:40.719705+0000","last_fullsized":"2024-03-19T02:51:40.719705+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:40:53.295454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.722973+0000","last_change":"2024-03-19T02:50:57.244449+0000","last_active":"2024-03-19T02:51:40.722973+0000","last_peered":"2024-03-19T02:51:40.722973+0000","last_clean":"2024-03-19T02:51:40.722973+0000","last_became_active":"2024-03-19T02:50:57.243991+0000","last_became_peered":"2024-03-19T02:50:57.243991+0000","last_unstale":"2024-03-19T02:51:40.722973+0000","last_undegraded":"2024-03-19T02:51:40.722973+0000","last_fullsized":"2024-03-19T02:51:40.722973+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:50:45.265773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719481+0000","last_change":"2024-03-19T02:50:51.799834+0000","last_active":"2024-03-19T02:51:40.719481+0000","last_peered":"2024-03-19T02:51:40.719481+0000","last_clean":"2024-03-19T02:51:40.719481+0000","last_became_active":"2024-03-19T02:50:51.799563+0000","last_became_peered":"2024-03-19T02:50:51.799563+0000","last_unstale":"2024-03-19T02:51:40.719481+0000","last_undegraded":"2024-03-19T02:51:40.719481+0000","last_fullsized":"2024-03-19T02:51:40.719481+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:51:00.284454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719916+0000","last_change":"2024-03-19T02:50:52.222710+0000","last_active":"2024-03-19T02:51:40.719916+0000","last_peered":"2024-03-19T02:51:40.719916+0000","last_clean":"2024-03-19T02:51:40.719916+0000","last_became_active":"2024-03-19T02:50:52.222321+0000","last_became_peered":"2024-03-19T02:50:52.222321+0000","last_unstale":"2024-03-19T02:51:40.719916+0000","last_undegraded":"2024-03-19T02:51:40.719916+0000","last_fullsized":"2024-03-19T02:51:40.719916+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:05:27.527696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.717431+0000","last_change":"2024-03-19T02:50:52.226631+0000","last_active":"2024-03-19T02:51:40.717431+0000","last_peered":"2024-03-19T02:51:40.717431+0000","last_clean":"2024-03-19T02:51:40.717431+0000","last_became_active":"2024-03-19T02:50:52.226294+0000","last_became_peered":"2024-03-19T02:50:52.226294+0000","last_unstale":"2024-03-19T02:51:40.717431+0000","last_undegraded":"2024-03-19T02:51:40.717431+0000","last_fullsized":"2024-03-19T02:51:40.717431+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:01:31.350063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.718930+0000","last_change":"2024-03-19T02:50:52.221138+0000","last_active":"2024-03-19T02:51:40.718930+0000","last_peered":"2024-03-19T02:51:40.718930+0000","last_clean":"2024-03-19T02:51:40.718930+0000","last_became_active":"2024-03-19T02:50:52.220730+0000","last_became_peered":"2024-03-19T02:50:52.220730+0000","last_unstale":"2024-03-19T02:51:40.718930+0000","last_undegraded":"2024-03-19T02:51:40.718930+0000","last_fullsized":"2024-03-19T02:51:40.718930+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:26:13.059986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.718416+0000","last_change":"2024-03-19T02:50:52.223567+0000","last_active":"2024-03-19T02:51:40.718416+0000","last_peered":"2024-03-19T02:51:40.718416+0000","last_clean":"2024-03-19T02:51:40.718416+0000","last_became_active":"2024-03-19T02:50:52.223174+0000","last_became_peered":"2024-03-19T02:50:52.223174+0000","last_unstale":"2024-03-19T02:51:40.718416+0000","last_undegraded":"2024-03-19T02:51:40.718416+0000","last_fullsized":"2024-03-19T02:51:40.718416+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:23:06.383584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.720173+0000","last_change":"2024-03-19T02:50:52.229916+0000","last_active":"2024-03-19T02:51:40.720173+0000","last_peered":"2024-03-19T02:51:40.720173+0000","last_clean":"2024-03-19T02:51:40.720173+0000","last_became_active":"2024-03-19T02:50:52.229597+0000","last_became_peered":"2024-03-19T02:50:52.229597+0000","last_unstale":"2024-03-19T02:51:40.720173+0000","last_undegraded":"2024-03-19T02:51:40.720173+0000","last_fullsized":"2024-03-19T02:51:40.720173+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:19.331893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.720033+0000","last_change":"2024-03-19T02:50:52.223331+0000","last_active":"2024-03-19T02:51:40.720033+0000","last_peered":"2024-03-19T02:51:40.720033+0000","last_clean":"2024-03-19T02:51:40.720033+0000","last_became_active":"2024-03-19T02:50:52.222975+0000","last_became_peered":"2024-03-19T02:50:52.222975+0000","last_unstale":"2024-03-19T02:51:40.720033+0000","last_undegraded":"2024-03-19T02:51:40.720033+0000","last_fullsized":"2024-03-19T02:51:40.720033+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:26:23.023749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.717303+0000","last_change":"2024-03-19T02:50:52.218073+0000","last_active":"2024-03-19T02:51:40.717303+0000","last_peered":"2024-03-19T02:51:40.717303+0000","last_clean":"2024-03-19T02:51:40.717303+0000","last_became_active":"2024-03-19T02:50:52.217742+0000","last_became_peered":"2024-03-19T02:50:52.217742+0000","last_unstale":"2024-03-19T02:51:40.717303+0000","last_undegraded":"2024-03-19T02:51:40.717303+0000","last_fullsized":"2024-03-19T02:51:40.717303+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:08:51.053153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719458+0000","last_change":"2024-03-19T02:50:52.222108+0000","last_active":"2024-03-19T02:51:40.719458+0000","last_peered":"2024-03-19T02:51:40.719458+0000","last_clean":"2024-03-19T02:51:40.719458+0000","last_became_active":"2024-03-19T02:50:52.221748+0000","last_became_peered":"2024-03-19T02:50:52.221748+0000","last_unstale":"2024-03-19T02:51:40.719458+0000","last_undegraded":"2024-03-19T02:51:40.719458+0000","last_fullsized":"2024-03-19T02:51:40.719458+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:33:31.401939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.716494+0000","last_change":"2024-03-19T02:50:52.221799+0000","last_active":"2024-03-19T02:51:40.716494+0000","last_peered":"2024-03-19T02:51:40.716494+0000","last_clean":"2024-03-19T02:51:40.716494+0000","last_became_active":"2024-03-19T02:50:52.220196+0000","last_became_peered":"2024-03-19T02:50:52.220196+0000","last_unstale":"2024-03-19T02:51:40.716494+0000","last_undegraded":"2024-03-19T02:51:40.716494+0000","last_fullsized":"2024-03-19T02:51:40.716494+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:24:01.541716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.721422+0000","last_change":"2024-03-19T02:51:40.721422+0000","last_active":"2024-03-19T02:51:40.721422+0000","last_peered":"2024-03-19T02:51:40.721422+0000","last_clean":"2024-03-19T02:51:40.721422+0000","last_became_active":"2024-03-19T02:51:40.720982+0000","last_became_peered":"2024-03-19T02:51:40.720982+0000","last_unstale":"2024-03-19T02:51:40.721422+0000","last_undegraded":"2024-03-19T02:51:40.721422+0000","last_fullsized":"2024-03-19T02:51:40.721422+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:07.391810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,3],"acting":[6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.721832+0000","last_change":"2024-03-19T02:51:40.721188+0000","last_active":"2024-03-19T02:51:40.721832+0000","last_peered":"2024-03-19T02:51:40.721832+0000","last_clean":"2024-03-19T02:51:40.721832+0000","last_became_active":"2024-03-19T02:51:40.720811+0000","last_became_peered":"2024-03-19T02:51:40.720811+0000","last_unstale":"2024-03-19T02:51:40.721832+0000","last_undegraded":"2024-03-19T02:51:40.721832+0000","last_fullsized":"2024-03-19T02:51:40.721832+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:23.970149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719742+0000","last_change":"2024-03-19T02:50:52.230847+0000","last_active":"2024-03-19T02:51:40.719742+0000","last_peered":"2024-03-19T02:51:40.719742+0000","last_clean":"2024-03-19T02:51:40.719742+0000","last_became_active":"2024-03-19T02:50:52.230478+0000","last_became_peered":"2024-03-19T02:50:52.230478+0000","last_unstale":"2024-03-19T02:51:40.719742+0000","last_undegraded":"2024-03-19T02:51:40.719742+0000","last_fullsized":"2024-03-19T02:51:40.719742+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:04:09.459446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719284+0000","last_change":"2024-03-19T02:50:52.223508+0000","last_active":"2024-03-19T02:51:40.719284+0000","last_peered":"2024-03-19T02:51:40.719284+0000","last_clean":"2024-03-19T02:51:40.719284+0000","last_became_active":"2024-03-19T02:50:52.223173+0000","last_became_peered":"2024-03-19T02:50:52.223173+0000","last_unstale":"2024-03-19T02:51:40.719284+0000","last_undegraded":"2024-03-19T02:51:40.719284+0000","last_fullsized":"2024-03-19T02:51:40.719284+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:53.320173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.723512+0000","last_change":"2024-03-19T02:51:40.723512+0000","last_active":"2024-03-19T02:51:40.723512+0000","last_peered":"2024-03-19T02:51:40.723512+0000","last_clean":"2024-03-19T02:51:40.723512+0000","last_became_active":"2024-03-19T02:51:40.723149+0000","last_became_peered":"2024-03-19T02:51:40.723149+0000","last_unstale":"2024-03-19T02:51:40.723512+0000","last_undegraded":"2024-03-19T02:51:40.723512+0000","last_fullsized":"2024-03-19T02:51:40.723512+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:42:38.862569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66966,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":438,"num_read_kb":291,"num_write":160,"num_write_kb":115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":694,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":200898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318305,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32240,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712912,"statfs":{"total":95995035648,"available":95962021888,"internally_reserved":0,"allocated":1081344,"data_stored":526938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28744,"kb_used_data":1728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716408,"statfs":{"total":95995035648,"available":95965601792,"internally_reserved":0,"allocated":1769472,"data_stored":1181546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743468,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1089536,"data_stored":527062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005104,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1081344,"data_stored":526870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266742,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32824,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712328,"statfs":{"total":95995035648,"available":95961423872,"internally_reserved":0,"allocated":1683456,"data_stored":1122500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462969,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1069056,"data_stored":522238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36448,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708704,"statfs":{"total":95995035648,"available":95957712896,"internally_reserved":0,"allocated":1138688,"data_stored":585504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1093632,"data_stored":523626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28148,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717004,"statfs":{"total":95995035648,"available":95966212096,"internally_reserved":0,"allocated":1159168,"data_stored":586272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149770,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1081344,"data_stored":522432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888133,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1085440,"data_stored":527470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724608,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28672,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716480,"statfs":{"total":95995035648,"available":95965675520,"internally_reserved":0,"allocated":1691648,"data_stored":1118088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":63356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":64124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":64934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T02:52:31.064 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-19T02:52:31.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:31 smithi033 ceph-mon[31973]: from='client.24962 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:52:31.321 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:52:31.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:31 smithi096 ceph-mon[35946]: from='client.24962 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:52:31.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:31 smithi155 ceph-mon[35804]: from='client.24962 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:52:32.433 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:32 smithi033 ceph-mon[31973]: pgmap v351: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-19T02:52:32.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:32 smithi096 ceph-mon[35946]: pgmap v351: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-19T02:52:32.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:32 smithi155 ceph-mon[35804]: pgmap v351: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-19T02:52:32.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.972+0000 7f22e7549700 1 -- 172.21.15.33:0/4281119146 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22e0106530 msgr2=0x7f22e010e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:32.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.972+0000 7f22e7549700 1 --2- 172.21.15.33:0/4281119146 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22e0106530 0x7f22e010e100 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f22d8009990 tx=0x7f22d8009ca0 comp rx=0 tx=0).stop 2024-03-19T02:52:32.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.972+0000 7f22e7549700 1 -- 172.21.15.33:0/4281119146 shutdown_connections 2024-03-19T02:52:32.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.972+0000 7f22e7549700 1 --2- 172.21.15.33:0/4281119146 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22e0106530 0x7f22e010e100 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:32.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.972+0000 7f22e7549700 1 --2- 172.21.15.33:0/4281119146 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f22e01014a0 0x7f22e0101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:32.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.972+0000 7f22e7549700 1 --2- 172.21.15.33:0/4281119146 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22e0101e50 0x7f22e0105e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:32.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.972+0000 7f22e7549700 1 -- 172.21.15.33:0/4281119146 >> 172.21.15.33:0/4281119146 conn(0x7f22e0077230 msgr2=0x7f22e0077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:32.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.972+0000 7f22e7549700 1 -- 172.21.15.33:0/4281119146 shutdown_connections 2024-03-19T02:52:32.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.972+0000 7f22e7549700 1 -- 172.21.15.33:0/4281119146 wait complete. 2024-03-19T02:52:32.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.972+0000 7f22e7549700 1 Processor -- start 2024-03-19T02:52:32.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.973+0000 7f22e7549700 1 -- start start 2024-03-19T02:52:32.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.973+0000 7f22e7549700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22e01014a0 0x7f22e01a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:32.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.973+0000 7f22e7549700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f22e0101e50 0x7f22e01a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:32.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.973+0000 7f22e7549700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22e0106530 0x7f22e019a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:32.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.973+0000 7f22e7549700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f22e0111410 con 0x7f22e0106530 2024-03-19T02:52:32.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.973+0000 7f22e7549700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f22e0111290 con 0x7f22e01014a0 2024-03-19T02:52:32.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.973+0000 7f22e7549700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f22e0111590 con 0x7f22e0101e50 2024-03-19T02:52:32.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.973+0000 7f22e5ae6700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22e0106530 0x7f22e019a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:32.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.973+0000 7f22e5ae6700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22e0106530 0x7f22e019a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38112/0 (socket says 172.21.15.33:38112) 2024-03-19T02:52:32.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.973+0000 7f22e5ae6700 1 -- 172.21.15.33:0/3390813000 learned_addr learned my addr 172.21.15.33:0/3390813000 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:32.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.973+0000 7f22e4ae4700 1 --2- 172.21.15.33:0/3390813000 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f22e0101e50 0x7f22e01a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:32.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.973+0000 7f22e52e5700 1 --2- 172.21.15.33:0/3390813000 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22e01014a0 0x7f22e01a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:32.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.973+0000 7f22e5ae6700 1 -- 172.21.15.33:0/3390813000 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f22e0101e50 msgr2=0x7f22e01a0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:32.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.973+0000 7f22e5ae6700 1 --2- 172.21.15.33:0/3390813000 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f22e0101e50 0x7f22e01a0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:32.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.973+0000 7f22e5ae6700 1 -- 172.21.15.33:0/3390813000 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22e01014a0 msgr2=0x7f22e01a0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:32.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.973+0000 7f22e5ae6700 1 --2- 172.21.15.33:0/3390813000 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22e01014a0 0x7f22e01a0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:32.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.973+0000 7f22e5ae6700 1 -- 172.21.15.33:0/3390813000 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22e019b150 con 0x7f22e0106530 2024-03-19T02:52:32.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.973+0000 7f22e5ae6700 1 --2- 172.21.15.33:0/3390813000 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22e0106530 0x7f22e019a9f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f22d8005b40 tx=0x7f22d80057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:32.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.974+0000 7f22de7fc700 1 -- 172.21.15.33:0/3390813000 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d8026070 con 0x7f22e0106530 2024-03-19T02:52:32.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.974+0000 7f22de7fc700 1 -- 172.21.15.33:0/3390813000 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22d8021410 con 0x7f22e0106530 2024-03-19T02:52:32.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.974+0000 7f22e7549700 1 -- 172.21.15.33:0/3390813000 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22e019b3e0 con 0x7f22e0106530 2024-03-19T02:52:32.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.974+0000 7f22de7fc700 1 -- 172.21.15.33:0/3390813000 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d800b610 con 0x7f22e0106530 2024-03-19T02:52:32.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.974+0000 7f22e7549700 1 -- 172.21.15.33:0/3390813000 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22e01a7700 con 0x7f22e0106530 2024-03-19T02:52:32.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.975+0000 7f22e7549700 1 -- 172.21.15.33:0/3390813000 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22e0068fd0 con 0x7f22e0106530 2024-03-19T02:52:32.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.976+0000 7f22de7fc700 1 -- 172.21.15.33:0/3390813000 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f22d8029070 con 0x7f22e0106530 2024-03-19T02:52:32.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.976+0000 7f22de7fc700 1 --2- 172.21.15.33:0/3390813000 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f22c80a29a0 0x7f22c80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:32.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.976+0000 7f22e52e5700 1 --2- 172.21.15.33:0/3390813000 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f22c80a29a0 0x7f22c80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:32.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.976+0000 7f22de7fc700 1 -- 172.21.15.33:0/3390813000 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7f22d80cfc80 con 0x7f22e0106530 2024-03-19T02:52:32.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.977+0000 7f22e52e5700 1 --2- 172.21.15.33:0/3390813000 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f22c80a29a0 0x7f22c80a4e60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f22e019bb30 tx=0x7f22d000b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:32.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:32.981+0000 7f22de7fc700 1 -- 172.21.15.33:0/3390813000 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f22d805d740 con 0x7f22e0106530 2024-03-19T02:52:33.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:33.179+0000 7f22e7549700 1 -- 172.21.15.33:0/3390813000 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f22e019bf20 con 0x7f22c80a29a0 2024-03-19T02:52:33.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:33.188+0000 7f22de7fc700 1 -- 172.21.15.33:0/3390813000 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340381 (secure 0 0 0) 0x7f22e019bf20 con 0x7f22c80a29a0 2024-03-19T02:52:33.194 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:52:33.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:33.192+0000 7f22e7549700 1 -- 172.21.15.33:0/3390813000 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f22c80a29a0 msgr2=0x7f22c80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:33.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:33.192+0000 7f22e7549700 1 --2- 172.21.15.33:0/3390813000 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f22c80a29a0 0x7f22c80a4e60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f22e019bb30 tx=0x7f22d000b040 comp rx=0 tx=0).stop 2024-03-19T02:52:33.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:33.192+0000 7f22e7549700 1 -- 172.21.15.33:0/3390813000 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22e0106530 msgr2=0x7f22e019a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:33.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:33.192+0000 7f22e7549700 1 --2- 172.21.15.33:0/3390813000 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22e0106530 0x7f22e019a9f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f22d8005b40 tx=0x7f22d80057c0 comp rx=0 tx=0).stop 2024-03-19T02:52:33.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:33.192+0000 7f22e7549700 1 -- 172.21.15.33:0/3390813000 shutdown_connections 2024-03-19T02:52:33.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:33.192+0000 7f22e7549700 1 --2- 172.21.15.33:0/3390813000 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f22c80a29a0 0x7f22c80a4e60 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:33.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:33.192+0000 7f22e7549700 1 --2- 172.21.15.33:0/3390813000 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22e0106530 0x7f22e019a9f0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:33.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:33.192+0000 7f22e7549700 1 --2- 172.21.15.33:0/3390813000 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22e01014a0 0x7f22e01a0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:33.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:33.192+0000 7f22e7549700 1 --2- 172.21.15.33:0/3390813000 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f22e0101e50 0x7f22e01a0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:33.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:33.192+0000 7f22e7549700 1 -- 172.21.15.33:0/3390813000 >> 172.21.15.33:0/3390813000 conn(0x7f22e0077230 msgr2=0x7f22e00ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:33.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:33.192+0000 7f22e7549700 1 -- 172.21.15.33:0/3390813000 shutdown_connections 2024-03-19T02:52:33.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:33.192+0000 7f22e7549700 1 -- 172.21.15.33:0/3390813000 wait complete. 2024-03-19T02:52:33.197 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-03-19T02:52:33.944 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:33 smithi033 ceph-mon[31973]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:52:33.944 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:33 smithi033 ceph-mon[31973]: pgmap v352: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:52:33.971 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2024-03-19T02:52:31.730745+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657568,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":589,"num_read_kb":415,"num_write":343,"num_write_kb":3055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1181547,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":278,"ondisk_log_size":278,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":368044,"kb_used_data":14672,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124573780,"statfs":{"total":1151940427776,"available":1151563550720,"internally_reserved":0,"allocated":15024128,"data_stored":8270546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":75,"num_read_kb":35,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003135"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.717879+0000","last_change":"2024-03-19T02:50:52.228368+0000","last_active":"2024-03-19T02:51:40.717879+0000","last_peered":"2024-03-19T02:51:40.717879+0000","last_clean":"2024-03-19T02:51:40.717879+0000","last_became_active":"2024-03-19T02:50:52.228069+0000","last_became_peered":"2024-03-19T02:50:52.228069+0000","last_unstale":"2024-03-19T02:51:40.717879+0000","last_undegraded":"2024-03-19T02:51:40.717879+0000","last_fullsized":"2024-03-19T02:51:40.717879+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:59:34.209095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.720688+0000","last_change":"2024-03-19T02:50:52.231358+0000","last_active":"2024-03-19T02:51:40.720688+0000","last_peered":"2024-03-19T02:51:40.720688+0000","last_clean":"2024-03-19T02:51:40.720688+0000","last_became_active":"2024-03-19T02:50:52.231029+0000","last_became_peered":"2024-03-19T02:50:52.231029+0000","last_unstale":"2024-03-19T02:51:40.720688+0000","last_undegraded":"2024-03-19T02:51:40.720688+0000","last_fullsized":"2024-03-19T02:51:40.720688+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:42:26.031392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719930+0000","last_change":"2024-03-19T02:50:52.232685+0000","last_active":"2024-03-19T02:51:40.719930+0000","last_peered":"2024-03-19T02:51:40.719930+0000","last_clean":"2024-03-19T02:51:40.719930+0000","last_became_active":"2024-03-19T02:50:52.232361+0000","last_became_peered":"2024-03-19T02:50:52.232361+0000","last_unstale":"2024-03-19T02:51:40.719930+0000","last_undegraded":"2024-03-19T02:51:40.719930+0000","last_fullsized":"2024-03-19T02:51:40.719930+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:53.136200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.716118+0000","last_change":"2024-03-19T02:50:52.227648+0000","last_active":"2024-03-19T02:51:40.716118+0000","last_peered":"2024-03-19T02:51:40.716118+0000","last_clean":"2024-03-19T02:51:40.716118+0000","last_became_active":"2024-03-19T02:50:52.226943+0000","last_became_peered":"2024-03-19T02:50:52.226943+0000","last_unstale":"2024-03-19T02:51:40.716118+0000","last_undegraded":"2024-03-19T02:51:40.716118+0000","last_fullsized":"2024-03-19T02:51:40.716118+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:03.293794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715486+0000","last_change":"2024-03-19T02:50:52.225329+0000","last_active":"2024-03-19T02:51:40.715486+0000","last_peered":"2024-03-19T02:51:40.715486+0000","last_clean":"2024-03-19T02:51:40.715486+0000","last_became_active":"2024-03-19T02:50:52.225012+0000","last_became_peered":"2024-03-19T02:50:52.225012+0000","last_unstale":"2024-03-19T02:51:40.715486+0000","last_undegraded":"2024-03-19T02:51:40.715486+0000","last_fullsized":"2024-03-19T02:51:40.715486+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:32:52.877704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719154+0000","last_change":"2024-03-19T02:50:52.217399+0000","last_active":"2024-03-19T02:51:40.719154+0000","last_peered":"2024-03-19T02:51:40.719154+0000","last_clean":"2024-03-19T02:51:40.719154+0000","last_became_active":"2024-03-19T02:50:52.216987+0000","last_became_peered":"2024-03-19T02:50:52.216987+0000","last_unstale":"2024-03-19T02:51:40.719154+0000","last_undegraded":"2024-03-19T02:51:40.719154+0000","last_fullsized":"2024-03-19T02:51:40.719154+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:57:15.772546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.718346+0000","last_change":"2024-03-19T02:50:52.226755+0000","last_active":"2024-03-19T02:51:40.718346+0000","last_peered":"2024-03-19T02:51:40.718346+0000","last_clean":"2024-03-19T02:51:40.718346+0000","last_became_active":"2024-03-19T02:50:52.226434+0000","last_became_peered":"2024-03-19T02:50:52.226434+0000","last_unstale":"2024-03-19T02:51:40.718346+0000","last_undegraded":"2024-03-19T02:51:40.718346+0000","last_fullsized":"2024-03-19T02:51:40.718346+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:49:42.134262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715219+0000","last_change":"2024-03-19T02:50:52.215848+0000","last_active":"2024-03-19T02:51:40.715219+0000","last_peered":"2024-03-19T02:51:40.715219+0000","last_clean":"2024-03-19T02:51:40.715219+0000","last_became_active":"2024-03-19T02:50:52.215494+0000","last_became_peered":"2024-03-19T02:50:52.215494+0000","last_unstale":"2024-03-19T02:51:40.715219+0000","last_undegraded":"2024-03-19T02:51:40.715219+0000","last_fullsized":"2024-03-19T02:51:40.715219+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:55:10.182359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719368+0000","last_change":"2024-03-19T02:50:52.233237+0000","last_active":"2024-03-19T02:51:40.719368+0000","last_peered":"2024-03-19T02:51:40.719368+0000","last_clean":"2024-03-19T02:51:40.719368+0000","last_became_active":"2024-03-19T02:50:52.232091+0000","last_became_peered":"2024-03-19T02:50:52.232091+0000","last_unstale":"2024-03-19T02:51:40.719368+0000","last_undegraded":"2024-03-19T02:51:40.719368+0000","last_fullsized":"2024-03-19T02:51:40.719368+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:08:20.361576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.722463+0000","last_change":"2024-03-19T02:51:40.722463+0000","last_active":"2024-03-19T02:51:40.722463+0000","last_peered":"2024-03-19T02:51:40.722463+0000","last_clean":"2024-03-19T02:51:40.722463+0000","last_became_active":"2024-03-19T02:51:40.722120+0000","last_became_peered":"2024-03-19T02:51:40.722120+0000","last_unstale":"2024-03-19T02:51:40.722463+0000","last_undegraded":"2024-03-19T02:51:40.722463+0000","last_fullsized":"2024-03-19T02:51:40.722463+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:49:21.387064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.720478+0000","last_change":"2024-03-19T02:50:52.226314+0000","last_active":"2024-03-19T02:51:40.720478+0000","last_peered":"2024-03-19T02:51:40.720478+0000","last_clean":"2024-03-19T02:51:40.720478+0000","last_became_active":"2024-03-19T02:50:52.225919+0000","last_became_peered":"2024-03-19T02:50:52.225919+0000","last_unstale":"2024-03-19T02:51:40.720478+0000","last_undegraded":"2024-03-19T02:51:40.720478+0000","last_fullsized":"2024-03-19T02:51:40.720478+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:42:51.782592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.716200+0000","last_change":"2024-03-19T02:50:52.227929+0000","last_active":"2024-03-19T02:51:40.716200+0000","last_peered":"2024-03-19T02:51:40.716200+0000","last_clean":"2024-03-19T02:51:40.716200+0000","last_became_active":"2024-03-19T02:50:52.227360+0000","last_became_peered":"2024-03-19T02:50:52.227360+0000","last_unstale":"2024-03-19T02:51:40.716200+0000","last_undegraded":"2024-03-19T02:51:40.716200+0000","last_fullsized":"2024-03-19T02:51:40.716200+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:24:44.386101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719392+0000","last_change":"2024-03-19T02:50:52.228088+0000","last_active":"2024-03-19T02:51:40.719392+0000","last_peered":"2024-03-19T02:51:40.719392+0000","last_clean":"2024-03-19T02:51:40.719392+0000","last_became_active":"2024-03-19T02:50:52.227754+0000","last_became_peered":"2024-03-19T02:50:52.227754+0000","last_unstale":"2024-03-19T02:51:40.719392+0000","last_undegraded":"2024-03-19T02:51:40.719392+0000","last_fullsized":"2024-03-19T02:51:40.719392+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:58:57.404241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719885+0000","last_change":"2024-03-19T02:50:52.226768+0000","last_active":"2024-03-19T02:51:40.719885+0000","last_peered":"2024-03-19T02:51:40.719885+0000","last_clean":"2024-03-19T02:51:40.719885+0000","last_became_active":"2024-03-19T02:50:52.226452+0000","last_became_peered":"2024-03-19T02:50:52.226452+0000","last_unstale":"2024-03-19T02:51:40.719885+0000","last_undegraded":"2024-03-19T02:51:40.719885+0000","last_fullsized":"2024-03-19T02:51:40.719885+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:23:33.250773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719676+0000","last_change":"2024-03-19T02:50:52.231132+0000","last_active":"2024-03-19T02:51:40.719676+0000","last_peered":"2024-03-19T02:51:40.719676+0000","last_clean":"2024-03-19T02:51:40.719676+0000","last_became_active":"2024-03-19T02:50:52.230778+0000","last_became_peered":"2024-03-19T02:50:52.230778+0000","last_unstale":"2024-03-19T02:51:40.719676+0000","last_undegraded":"2024-03-19T02:51:40.719676+0000","last_fullsized":"2024-03-19T02:51:40.719676+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:33:03.675210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.720289+0000","last_change":"2024-03-19T02:50:52.228302+0000","last_active":"2024-03-19T02:51:40.720289+0000","last_peered":"2024-03-19T02:51:40.720289+0000","last_clean":"2024-03-19T02:51:40.720289+0000","last_became_active":"2024-03-19T02:50:52.228007+0000","last_became_peered":"2024-03-19T02:50:52.228007+0000","last_unstale":"2024-03-19T02:51:40.720289+0000","last_undegraded":"2024-03-19T02:51:40.720289+0000","last_fullsized":"2024-03-19T02:51:40.720289+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:59:28.988504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.717631+0000","last_change":"2024-03-19T02:50:52.228640+0000","last_active":"2024-03-19T02:51:40.717631+0000","last_peered":"2024-03-19T02:51:40.717631+0000","last_clean":"2024-03-19T02:51:40.717631+0000","last_became_active":"2024-03-19T02:50:52.228319+0000","last_became_peered":"2024-03-19T02:50:52.228319+0000","last_unstale":"2024-03-19T02:51:40.717631+0000","last_undegraded":"2024-03-19T02:51:40.717631+0000","last_fullsized":"2024-03-19T02:51:40.717631+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:30.356453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.718484+0000","last_change":"2024-03-19T02:50:52.222300+0000","last_active":"2024-03-19T02:51:40.718484+0000","last_peered":"2024-03-19T02:51:40.718484+0000","last_clean":"2024-03-19T02:51:40.718484+0000","last_became_active":"2024-03-19T02:50:52.221978+0000","last_became_peered":"2024-03-19T02:50:52.221978+0000","last_unstale":"2024-03-19T02:51:40.718484+0000","last_undegraded":"2024-03-19T02:51:40.718484+0000","last_fullsized":"2024-03-19T02:51:40.718484+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:38:36.576987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719330+0000","last_change":"2024-03-19T02:50:52.217640+0000","last_active":"2024-03-19T02:51:40.719330+0000","last_peered":"2024-03-19T02:51:40.719330+0000","last_clean":"2024-03-19T02:51:40.719330+0000","last_became_active":"2024-03-19T02:50:52.217325+0000","last_became_peered":"2024-03-19T02:50:52.217325+0000","last_unstale":"2024-03-19T02:51:40.719330+0000","last_undegraded":"2024-03-19T02:51:40.719330+0000","last_fullsized":"2024-03-19T02:51:40.719330+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:40:25.894843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.718707+0000","last_change":"2024-03-19T02:50:57.245323+0000","last_active":"2024-03-19T02:51:40.718707+0000","last_peered":"2024-03-19T02:51:40.718707+0000","last_clean":"2024-03-19T02:51:40.718707+0000","last_became_active":"2024-03-19T02:50:57.244886+0000","last_became_peered":"2024-03-19T02:50:57.244886+0000","last_unstale":"2024-03-19T02:51:40.718707+0000","last_undegraded":"2024-03-19T02:51:40.718707+0000","last_fullsized":"2024-03-19T02:51:40.718707+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:42:09.465738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"93'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719811+0000","last_change":"2024-03-19T02:50:51.797160+0000","last_active":"2024-03-19T02:51:40.719811+0000","last_peered":"2024-03-19T02:51:40.719811+0000","last_clean":"2024-03-19T02:51:40.719811+0000","last_became_active":"2024-03-19T02:50:51.796783+0000","last_became_peered":"2024-03-19T02:50:51.796783+0000","last_unstale":"2024-03-19T02:51:40.719811+0000","last_undegraded":"2024-03-19T02:51:40.719811+0000","last_fullsized":"2024-03-19T02:51:40.719811+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:57:08.185379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.716088+0000","last_change":"2024-03-19T02:50:52.225114+0000","last_active":"2024-03-19T02:51:40.716088+0000","last_peered":"2024-03-19T02:51:40.716088+0000","last_clean":"2024-03-19T02:51:40.716088+0000","last_became_active":"2024-03-19T02:50:52.220529+0000","last_became_peered":"2024-03-19T02:50:52.220529+0000","last_unstale":"2024-03-19T02:51:40.716088+0000","last_undegraded":"2024-03-19T02:51:40.716088+0000","last_fullsized":"2024-03-19T02:51:40.716088+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:12:51.408689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715363+0000","last_change":"2024-03-19T02:50:57.252827+0000","last_active":"2024-03-19T02:51:40.715363+0000","last_peered":"2024-03-19T02:51:40.715363+0000","last_clean":"2024-03-19T02:51:40.715363+0000","last_became_active":"2024-03-19T02:50:57.252502+0000","last_became_peered":"2024-03-19T02:50:57.252502+0000","last_unstale":"2024-03-19T02:51:40.715363+0000","last_undegraded":"2024-03-19T02:51:40.715363+0000","last_fullsized":"2024-03-19T02:51:40.715363+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:29:36.563533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"93'2","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.716120+0000","last_change":"2024-03-19T02:50:51.797380+0000","last_active":"2024-03-19T02:51:40.716120+0000","last_peered":"2024-03-19T02:51:40.716120+0000","last_clean":"2024-03-19T02:51:40.716120+0000","last_became_active":"2024-03-19T02:50:51.797021+0000","last_became_peered":"2024-03-19T02:50:51.797021+0000","last_unstale":"2024-03-19T02:51:40.716120+0000","last_undegraded":"2024-03-19T02:51:40.716120+0000","last_fullsized":"2024-03-19T02:51:40.716120+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:25.400360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.720551+0000","last_change":"2024-03-19T02:50:52.232717+0000","last_active":"2024-03-19T02:51:40.720551+0000","last_peered":"2024-03-19T02:51:40.720551+0000","last_clean":"2024-03-19T02:51:40.720551+0000","last_became_active":"2024-03-19T02:50:52.232368+0000","last_became_peered":"2024-03-19T02:50:52.232368+0000","last_unstale":"2024-03-19T02:51:40.720551+0000","last_undegraded":"2024-03-19T02:51:40.720551+0000","last_fullsized":"2024-03-19T02:51:40.720551+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:55:38.027120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715501+0000","last_change":"2024-03-19T02:50:57.250840+0000","last_active":"2024-03-19T02:51:40.715501+0000","last_peered":"2024-03-19T02:51:40.715501+0000","last_clean":"2024-03-19T02:51:40.715501+0000","last_became_active":"2024-03-19T02:50:57.250333+0000","last_became_peered":"2024-03-19T02:50:57.250333+0000","last_unstale":"2024-03-19T02:51:40.715501+0000","last_undegraded":"2024-03-19T02:51:40.715501+0000","last_fullsized":"2024-03-19T02:51:40.715501+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:57:09.089939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"93'4","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715764+0000","last_change":"2024-03-19T02:50:51.796889+0000","last_active":"2024-03-19T02:51:40.715764+0000","last_peered":"2024-03-19T02:51:40.715764+0000","last_clean":"2024-03-19T02:51:40.715764+0000","last_became_active":"2024-03-19T02:50:51.796570+0000","last_became_peered":"2024-03-19T02:50:51.796570+0000","last_unstale":"2024-03-19T02:51:40.715764+0000","last_undegraded":"2024-03-19T02:51:40.715764+0000","last_fullsized":"2024-03-19T02:51:40.715764+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:18:38.301905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.720023+0000","last_change":"2024-03-19T02:50:52.224533+0000","last_active":"2024-03-19T02:51:40.720023+0000","last_peered":"2024-03-19T02:51:40.720023+0000","last_clean":"2024-03-19T02:51:40.720023+0000","last_became_active":"2024-03-19T02:50:52.224221+0000","last_became_peered":"2024-03-19T02:50:52.224221+0000","last_unstale":"2024-03-19T02:51:40.720023+0000","last_undegraded":"2024-03-19T02:51:40.720023+0000","last_fullsized":"2024-03-19T02:51:40.720023+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:36:43.335318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715388+0000","last_change":"2024-03-19T02:50:57.252949+0000","last_active":"2024-03-19T02:51:40.715388+0000","last_peered":"2024-03-19T02:51:40.715388+0000","last_clean":"2024-03-19T02:51:40.715388+0000","last_became_active":"2024-03-19T02:50:57.252563+0000","last_became_peered":"2024-03-19T02:50:57.252563+0000","last_unstale":"2024-03-19T02:51:40.715388+0000","last_undegraded":"2024-03-19T02:51:40.715388+0000","last_fullsized":"2024-03-19T02:51:40.715388+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:30:38.244158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715620+0000","last_change":"2024-03-19T02:50:51.794594+0000","last_active":"2024-03-19T02:51:40.715620+0000","last_peered":"2024-03-19T02:51:40.715620+0000","last_clean":"2024-03-19T02:51:40.715620+0000","last_became_active":"2024-03-19T02:50:51.794197+0000","last_became_peered":"2024-03-19T02:50:51.794197+0000","last_unstale":"2024-03-19T02:51:40.715620+0000","last_undegraded":"2024-03-19T02:51:40.715620+0000","last_fullsized":"2024-03-19T02:51:40.715620+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:58:44.534106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.722212+0000","last_change":"2024-03-19T02:50:52.214167+0000","last_active":"2024-03-19T02:51:40.722212+0000","last_peered":"2024-03-19T02:51:40.722212+0000","last_clean":"2024-03-19T02:51:40.722212+0000","last_became_active":"2024-03-19T02:50:52.213873+0000","last_became_peered":"2024-03-19T02:50:52.213873+0000","last_unstale":"2024-03-19T02:51:40.722212+0000","last_undegraded":"2024-03-19T02:51:40.722212+0000","last_fullsized":"2024-03-19T02:51:40.722212+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:28:51.035477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719147+0000","last_change":"2024-03-19T02:50:57.245139+0000","last_active":"2024-03-19T02:51:40.719147+0000","last_peered":"2024-03-19T02:51:40.719147+0000","last_clean":"2024-03-19T02:51:40.719147+0000","last_became_active":"2024-03-19T02:50:57.244636+0000","last_became_peered":"2024-03-19T02:50:57.244636+0000","last_unstale":"2024-03-19T02:51:40.719147+0000","last_undegraded":"2024-03-19T02:51:40.719147+0000","last_fullsized":"2024-03-19T02:51:40.719147+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:44:29.660221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'5","reported_seq":110,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:52:29.157853+0000","last_change":"2024-03-19T02:50:51.795969+0000","last_active":"2024-03-19T02:52:29.157853+0000","last_peered":"2024-03-19T02:52:29.157853+0000","last_clean":"2024-03-19T02:52:29.157853+0000","last_became_active":"2024-03-19T02:50:51.795628+0000","last_became_peered":"2024-03-19T02:50:51.795628+0000","last_unstale":"2024-03-19T02:52:29.157853+0000","last_undegraded":"2024-03-19T02:52:29.157853+0000","last_fullsized":"2024-03-19T02:52:29.157853+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:14:48.463282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715506+0000","last_change":"2024-03-19T02:50:52.223278+0000","last_active":"2024-03-19T02:51:40.715506+0000","last_peered":"2024-03-19T02:51:40.715506+0000","last_clean":"2024-03-19T02:51:40.715506+0000","last_became_active":"2024-03-19T02:50:52.222741+0000","last_became_peered":"2024-03-19T02:50:52.222741+0000","last_unstale":"2024-03-19T02:51:40.715506+0000","last_undegraded":"2024-03-19T02:51:40.715506+0000","last_fullsized":"2024-03-19T02:51:40.715506+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:32:09.841825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719135+0000","last_change":"2024-03-19T02:50:57.253130+0000","last_active":"2024-03-19T02:51:40.719135+0000","last_peered":"2024-03-19T02:51:40.719135+0000","last_clean":"2024-03-19T02:51:40.719135+0000","last_became_active":"2024-03-19T02:50:57.252598+0000","last_became_peered":"2024-03-19T02:50:57.252598+0000","last_unstale":"2024-03-19T02:51:40.719135+0000","last_undegraded":"2024-03-19T02:51:40.719135+0000","last_fullsized":"2024-03-19T02:51:40.719135+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:09:38.183430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"93'3","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719095+0000","last_change":"2024-03-19T02:50:51.794075+0000","last_active":"2024-03-19T02:51:40.719095+0000","last_peered":"2024-03-19T02:51:40.719095+0000","last_clean":"2024-03-19T02:51:40.719095+0000","last_became_active":"2024-03-19T02:50:51.793707+0000","last_became_peered":"2024-03-19T02:50:51.793707+0000","last_unstale":"2024-03-19T02:51:40.719095+0000","last_undegraded":"2024-03-19T02:51:40.719095+0000","last_fullsized":"2024-03-19T02:51:40.719095+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:52:07.180162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.722243+0000","last_change":"2024-03-19T02:50:52.211112+0000","last_active":"2024-03-19T02:51:40.722243+0000","last_peered":"2024-03-19T02:51:40.722243+0000","last_clean":"2024-03-19T02:51:40.722243+0000","last_became_active":"2024-03-19T02:50:52.210790+0000","last_became_peered":"2024-03-19T02:50:52.210790+0000","last_unstale":"2024-03-19T02:51:40.722243+0000","last_undegraded":"2024-03-19T02:51:40.722243+0000","last_fullsized":"2024-03-19T02:51:40.722243+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:39:17.651585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":19,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.717585+0000","last_change":"2024-03-19T02:50:57.250397+0000","last_active":"2024-03-19T02:51:40.717585+0000","last_peered":"2024-03-19T02:51:40.717585+0000","last_clean":"2024-03-19T02:51:40.717585+0000","last_became_active":"2024-03-19T02:50:57.249968+0000","last_became_peered":"2024-03-19T02:50:57.249968+0000","last_unstale":"2024-03-19T02:51:40.717585+0000","last_undegraded":"2024-03-19T02:51:40.717585+0000","last_fullsized":"2024-03-19T02:51:40.717585+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:39.796799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"93'2","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.717379+0000","last_change":"2024-03-19T02:50:51.785347+0000","last_active":"2024-03-19T02:51:40.717379+0000","last_peered":"2024-03-19T02:51:40.717379+0000","last_clean":"2024-03-19T02:51:40.717379+0000","last_became_active":"2024-03-19T02:50:51.785004+0000","last_became_peered":"2024-03-19T02:50:51.785004+0000","last_unstale":"2024-03-19T02:51:40.717379+0000","last_undegraded":"2024-03-19T02:51:40.717379+0000","last_fullsized":"2024-03-19T02:51:40.717379+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:47.269503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715338+0000","last_change":"2024-03-19T02:50:52.221330+0000","last_active":"2024-03-19T02:51:40.715338+0000","last_peered":"2024-03-19T02:51:40.715338+0000","last_clean":"2024-03-19T02:51:40.715338+0000","last_became_active":"2024-03-19T02:50:52.219417+0000","last_became_peered":"2024-03-19T02:50:52.219417+0000","last_unstale":"2024-03-19T02:51:40.715338+0000","last_undegraded":"2024-03-19T02:51:40.715338+0000","last_fullsized":"2024-03-19T02:51:40.715338+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:28:37.856260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.720481+0000","last_change":"2024-03-19T02:50:57.258357+0000","last_active":"2024-03-19T02:51:40.720481+0000","last_peered":"2024-03-19T02:51:40.720481+0000","last_clean":"2024-03-19T02:51:40.720481+0000","last_became_active":"2024-03-19T02:50:57.257924+0000","last_became_peered":"2024-03-19T02:50:57.257924+0000","last_unstale":"2024-03-19T02:51:40.720481+0000","last_undegraded":"2024-03-19T02:51:40.720481+0000","last_fullsized":"2024-03-19T02:51:40.720481+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:46:05.340731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"93'1","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.723269+0000","last_change":"2024-03-19T02:50:51.796425+0000","last_active":"2024-03-19T02:51:40.723269+0000","last_peered":"2024-03-19T02:51:40.723269+0000","last_clean":"2024-03-19T02:51:40.723269+0000","last_became_active":"2024-03-19T02:50:51.796139+0000","last_became_peered":"2024-03-19T02:50:51.796139+0000","last_unstale":"2024-03-19T02:51:40.723269+0000","last_undegraded":"2024-03-19T02:51:40.723269+0000","last_fullsized":"2024-03-19T02:51:40.723269+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:12:17.123556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.717799+0000","last_change":"2024-03-19T02:50:52.219375+0000","last_active":"2024-03-19T02:51:40.717799+0000","last_peered":"2024-03-19T02:51:40.717799+0000","last_clean":"2024-03-19T02:51:40.717799+0000","last_became_active":"2024-03-19T02:50:52.219070+0000","last_became_peered":"2024-03-19T02:50:52.219070+0000","last_unstale":"2024-03-19T02:51:40.717799+0000","last_undegraded":"2024-03-19T02:51:40.717799+0000","last_fullsized":"2024-03-19T02:51:40.717799+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:31:58.847700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715282+0000","last_change":"2024-03-19T02:50:57.246502+0000","last_active":"2024-03-19T02:51:40.715282+0000","last_peered":"2024-03-19T02:51:40.715282+0000","last_clean":"2024-03-19T02:51:40.715282+0000","last_became_active":"2024-03-19T02:50:57.246089+0000","last_became_peered":"2024-03-19T02:50:57.246089+0000","last_unstale":"2024-03-19T02:51:40.715282+0000","last_undegraded":"2024-03-19T02:51:40.715282+0000","last_fullsized":"2024-03-19T02:51:40.715282+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:00.162308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'3","reported_seq":105,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:52:28.156316+0000","last_change":"2024-03-19T02:50:51.796448+0000","last_active":"2024-03-19T02:52:28.156316+0000","last_peered":"2024-03-19T02:52:28.156316+0000","last_clean":"2024-03-19T02:52:28.156316+0000","last_became_active":"2024-03-19T02:50:51.796132+0000","last_became_peered":"2024-03-19T02:50:51.796132+0000","last_unstale":"2024-03-19T02:52:28.156316+0000","last_undegraded":"2024-03-19T02:52:28.156316+0000","last_fullsized":"2024-03-19T02:52:28.156316+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:37:39.442772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":71,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.718528+0000","last_change":"2024-03-19T02:50:52.223022+0000","last_active":"2024-03-19T02:51:40.718528+0000","last_peered":"2024-03-19T02:51:40.718528+0000","last_clean":"2024-03-19T02:51:40.718528+0000","last_became_active":"2024-03-19T02:50:52.221655+0000","last_became_peered":"2024-03-19T02:50:52.221655+0000","last_unstale":"2024-03-19T02:51:40.718528+0000","last_undegraded":"2024-03-19T02:51:40.718528+0000","last_fullsized":"2024-03-19T02:51:40.718528+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:34:36.054088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.718116+0000","last_change":"2024-03-19T02:50:57.244003+0000","last_active":"2024-03-19T02:51:40.718116+0000","last_peered":"2024-03-19T02:51:40.718116+0000","last_clean":"2024-03-19T02:51:40.718116+0000","last_became_active":"2024-03-19T02:50:57.243567+0000","last_became_peered":"2024-03-19T02:50:57.243567+0000","last_unstale":"2024-03-19T02:51:40.718116+0000","last_undegraded":"2024-03-19T02:51:40.718116+0000","last_fullsized":"2024-03-19T02:51:40.718116+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:58:52.400237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"93'2","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719750+0000","last_change":"2024-03-19T02:50:51.801259+0000","last_active":"2024-03-19T02:51:40.719750+0000","last_peered":"2024-03-19T02:51:40.719750+0000","last_clean":"2024-03-19T02:51:40.719750+0000","last_became_active":"2024-03-19T02:50:51.800868+0000","last_became_peered":"2024-03-19T02:50:51.800868+0000","last_unstale":"2024-03-19T02:51:40.719750+0000","last_undegraded":"2024-03-19T02:51:40.719750+0000","last_fullsized":"2024-03-19T02:51:40.719750+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:44:06.486687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.724476+0000","last_change":"2024-03-19T02:51:40.724476+0000","last_active":"2024-03-19T02:51:40.724476+0000","last_peered":"2024-03-19T02:51:40.724476+0000","last_clean":"2024-03-19T02:51:40.724476+0000","last_became_active":"2024-03-19T02:51:40.724081+0000","last_became_peered":"2024-03-19T02:51:40.724081+0000","last_unstale":"2024-03-19T02:51:40.724476+0000","last_undegraded":"2024-03-19T02:51:40.724476+0000","last_fullsized":"2024-03-19T02:51:40.724476+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:04:15.684753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715647+0000","last_change":"2024-03-19T02:50:57.238764+0000","last_active":"2024-03-19T02:51:40.715647+0000","last_peered":"2024-03-19T02:51:40.715647+0000","last_clean":"2024-03-19T02:51:40.715647+0000","last_became_active":"2024-03-19T02:50:57.238401+0000","last_became_peered":"2024-03-19T02:50:57.238401+0000","last_unstale":"2024-03-19T02:51:40.715647+0000","last_undegraded":"2024-03-19T02:51:40.715647+0000","last_fullsized":"2024-03-19T02:51:40.715647+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:20:46.131776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"93'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.722102+0000","last_change":"2024-03-19T02:50:51.797633+0000","last_active":"2024-03-19T02:51:40.722102+0000","last_peered":"2024-03-19T02:51:40.722102+0000","last_clean":"2024-03-19T02:51:40.722102+0000","last_became_active":"2024-03-19T02:50:51.797313+0000","last_became_peered":"2024-03-19T02:50:51.797313+0000","last_unstale":"2024-03-19T02:51:40.722102+0000","last_undegraded":"2024-03-19T02:51:40.722102+0000","last_fullsized":"2024-03-19T02:51:40.722102+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:43:04.659301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.720178+0000","last_change":"2024-03-19T02:50:52.231486+0000","last_active":"2024-03-19T02:51:40.720178+0000","last_peered":"2024-03-19T02:51:40.720178+0000","last_clean":"2024-03-19T02:51:40.720178+0000","last_became_active":"2024-03-19T02:50:52.231172+0000","last_became_peered":"2024-03-19T02:50:52.231172+0000","last_unstale":"2024-03-19T02:51:40.720178+0000","last_undegraded":"2024-03-19T02:51:40.720178+0000","last_fullsized":"2024-03-19T02:51:40.720178+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:24:49.912633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.717867+0000","last_change":"2024-03-19T02:50:57.247207+0000","last_active":"2024-03-19T02:51:40.717867+0000","last_peered":"2024-03-19T02:51:40.717867+0000","last_clean":"2024-03-19T02:51:40.717867+0000","last_became_active":"2024-03-19T02:50:57.246776+0000","last_became_peered":"2024-03-19T02:50:57.246776+0000","last_unstale":"2024-03-19T02:51:40.717867+0000","last_undegraded":"2024-03-19T02:51:40.717867+0000","last_fullsized":"2024-03-19T02:51:40.717867+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:02:24.118858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"95'33","reported_seq":214,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:52:29.157221+0000","last_change":"2024-03-19T02:50:51.792398+0000","last_active":"2024-03-19T02:52:29.157221+0000","last_peered":"2024-03-19T02:52:29.157221+0000","last_clean":"2024-03-19T02:52:29.157221+0000","last_became_active":"2024-03-19T02:50:51.792090+0000","last_became_peered":"2024-03-19T02:50:51.792090+0000","last_unstale":"2024-03-19T02:52:29.157221+0000","last_undegraded":"2024-03-19T02:52:29.157221+0000","last_fullsized":"2024-03-19T02:52:29.157221+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:02:58.175210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63356,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":150,"num_read_kb":137,"num_write":34,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715596+0000","last_change":"2024-03-19T02:50:52.220755+0000","last_active":"2024-03-19T02:51:40.715596+0000","last_peered":"2024-03-19T02:51:40.715596+0000","last_clean":"2024-03-19T02:51:40.715596+0000","last_became_active":"2024-03-19T02:50:52.220453+0000","last_became_peered":"2024-03-19T02:50:52.220453+0000","last_unstale":"2024-03-19T02:51:40.715596+0000","last_undegraded":"2024-03-19T02:51:40.715596+0000","last_fullsized":"2024-03-19T02:51:40.715596+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:48:27.866863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.722499+0000","last_change":"2024-03-19T02:50:57.253843+0000","last_active":"2024-03-19T02:51:40.722499+0000","last_peered":"2024-03-19T02:51:40.722499+0000","last_clean":"2024-03-19T02:51:40.722499+0000","last_became_active":"2024-03-19T02:50:57.252633+0000","last_became_peered":"2024-03-19T02:50:57.252633+0000","last_unstale":"2024-03-19T02:51:40.722499+0000","last_undegraded":"2024-03-19T02:51:40.722499+0000","last_fullsized":"2024-03-19T02:51:40.722499+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:03:48.514219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"93'6","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719004+0000","last_change":"2024-03-19T02:50:51.795041+0000","last_active":"2024-03-19T02:51:40.719004+0000","last_peered":"2024-03-19T02:51:40.719004+0000","last_clean":"2024-03-19T02:51:40.719004+0000","last_became_active":"2024-03-19T02:50:51.793485+0000","last_became_peered":"2024-03-19T02:50:51.793485+0000","last_unstale":"2024-03-19T02:51:40.719004+0000","last_undegraded":"2024-03-19T02:51:40.719004+0000","last_fullsized":"2024-03-19T02:51:40.719004+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:11:50.136394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719177+0000","last_change":"2024-03-19T02:50:52.227554+0000","last_active":"2024-03-19T02:51:40.719177+0000","last_peered":"2024-03-19T02:51:40.719177+0000","last_clean":"2024-03-19T02:51:40.719177+0000","last_became_active":"2024-03-19T02:50:52.227222+0000","last_became_peered":"2024-03-19T02:50:52.227222+0000","last_unstale":"2024-03-19T02:51:40.719177+0000","last_undegraded":"2024-03-19T02:51:40.719177+0000","last_fullsized":"2024-03-19T02:51:40.719177+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:56.323749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":19,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.717521+0000","last_change":"2024-03-19T02:50:57.257504+0000","last_active":"2024-03-19T02:51:40.717521+0000","last_peered":"2024-03-19T02:51:40.717521+0000","last_clean":"2024-03-19T02:51:40.717521+0000","last_became_active":"2024-03-19T02:50:57.257120+0000","last_became_peered":"2024-03-19T02:50:57.257120+0000","last_unstale":"2024-03-19T02:51:40.717521+0000","last_undegraded":"2024-03-19T02:51:40.717521+0000","last_fullsized":"2024-03-19T02:51:40.717521+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:19:10.471666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"93'3","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.722290+0000","last_change":"2024-03-19T02:50:51.798049+0000","last_active":"2024-03-19T02:51:40.722290+0000","last_peered":"2024-03-19T02:51:40.722290+0000","last_clean":"2024-03-19T02:51:40.722290+0000","last_became_active":"2024-03-19T02:50:51.797134+0000","last_became_peered":"2024-03-19T02:50:51.797134+0000","last_unstale":"2024-03-19T02:51:40.722290+0000","last_undegraded":"2024-03-19T02:51:40.722290+0000","last_fullsized":"2024-03-19T02:51:40.722290+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:03:46.984240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719601+0000","last_change":"2024-03-19T02:50:52.219535+0000","last_active":"2024-03-19T02:51:40.719601+0000","last_peered":"2024-03-19T02:51:40.719601+0000","last_clean":"2024-03-19T02:51:40.719601+0000","last_became_active":"2024-03-19T02:50:52.219226+0000","last_became_peered":"2024-03-19T02:50:52.219226+0000","last_unstale":"2024-03-19T02:51:40.719601+0000","last_undegraded":"2024-03-19T02:51:40.719601+0000","last_fullsized":"2024-03-19T02:51:40.719601+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:27:26.875533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719542+0000","last_change":"2024-03-19T02:50:57.252873+0000","last_active":"2024-03-19T02:51:40.719542+0000","last_peered":"2024-03-19T02:51:40.719542+0000","last_clean":"2024-03-19T02:51:40.719542+0000","last_became_active":"2024-03-19T02:50:57.249551+0000","last_became_peered":"2024-03-19T02:50:57.249551+0000","last_unstale":"2024-03-19T02:51:40.719542+0000","last_undegraded":"2024-03-19T02:51:40.719542+0000","last_fullsized":"2024-03-19T02:51:40.719542+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:33:59.750606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"93'3","reported_seq":107,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:52:29.170733+0000","last_change":"2024-03-19T02:50:51.796478+0000","last_active":"2024-03-19T02:52:29.170733+0000","last_peered":"2024-03-19T02:52:29.170733+0000","last_clean":"2024-03-19T02:52:29.170733+0000","last_became_active":"2024-03-19T02:50:51.796126+0000","last_became_peered":"2024-03-19T02:50:51.796126+0000","last_unstale":"2024-03-19T02:52:29.170733+0000","last_undegraded":"2024-03-19T02:52:29.170733+0000","last_fullsized":"2024-03-19T02:52:29.170733+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:13:42.690481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719337+0000","last_change":"2024-03-19T02:50:52.228151+0000","last_active":"2024-03-19T02:51:40.719337+0000","last_peered":"2024-03-19T02:51:40.719337+0000","last_clean":"2024-03-19T02:51:40.719337+0000","last_became_active":"2024-03-19T02:50:52.227858+0000","last_became_peered":"2024-03-19T02:50:52.227858+0000","last_unstale":"2024-03-19T02:51:40.719337+0000","last_undegraded":"2024-03-19T02:51:40.719337+0000","last_fullsized":"2024-03-19T02:51:40.719337+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:10.285606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719376+0000","last_change":"2024-03-19T02:50:57.259269+0000","last_active":"2024-03-19T02:51:40.719376+0000","last_peered":"2024-03-19T02:51:40.719376+0000","last_clean":"2024-03-19T02:51:40.719376+0000","last_became_active":"2024-03-19T02:50:57.258778+0000","last_became_peered":"2024-03-19T02:50:57.258778+0000","last_unstale":"2024-03-19T02:51:40.719376+0000","last_undegraded":"2024-03-19T02:51:40.719376+0000","last_fullsized":"2024-03-19T02:51:40.719376+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:02:44.419512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"93'15","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.739965+0000","last_change":"2024-03-19T02:51:40.739522+0000","last_active":"2024-03-19T02:51:40.739965+0000","last_peered":"2024-03-19T02:51:40.739965+0000","last_clean":"2024-03-19T02:51:40.739965+0000","last_became_active":"2024-03-19T02:51:40.720675+0000","last_became_peered":"2024-03-19T02:51:40.720675+0000","last_unstale":"2024-03-19T02:51:40.739965+0000","last_undegraded":"2024-03-19T02:51:40.739965+0000","last_fullsized":"2024-03-19T02:51:40.739965+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:18:19.449850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.723163+0000","last_change":"2024-03-19T02:50:52.215997+0000","last_active":"2024-03-19T02:51:40.723163+0000","last_peered":"2024-03-19T02:51:40.723163+0000","last_clean":"2024-03-19T02:51:40.723163+0000","last_became_active":"2024-03-19T02:50:52.215398+0000","last_became_peered":"2024-03-19T02:50:52.215398+0000","last_unstale":"2024-03-19T02:51:40.723163+0000","last_undegraded":"2024-03-19T02:51:40.723163+0000","last_fullsized":"2024-03-19T02:51:40.723163+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:49:14.010736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.723079+0000","last_change":"2024-03-19T02:50:57.247254+0000","last_active":"2024-03-19T02:51:40.723079+0000","last_peered":"2024-03-19T02:51:40.723079+0000","last_clean":"2024-03-19T02:51:40.723079+0000","last_became_active":"2024-03-19T02:50:57.246861+0000","last_became_peered":"2024-03-19T02:50:57.246861+0000","last_unstale":"2024-03-19T02:51:40.723079+0000","last_undegraded":"2024-03-19T02:51:40.723079+0000","last_fullsized":"2024-03-19T02:51:40.723079+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:47:20.933550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"93'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715688+0000","last_change":"2024-03-19T02:50:51.796246+0000","last_active":"2024-03-19T02:51:40.715688+0000","last_peered":"2024-03-19T02:51:40.715688+0000","last_clean":"2024-03-19T02:51:40.715688+0000","last_became_active":"2024-03-19T02:50:51.795823+0000","last_became_peered":"2024-03-19T02:50:51.795823+0000","last_unstale":"2024-03-19T02:51:40.715688+0000","last_undegraded":"2024-03-19T02:51:40.715688+0000","last_fullsized":"2024-03-19T02:51:40.715688+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:36:02.220891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.717740+0000","last_change":"2024-03-19T02:50:52.224119+0000","last_active":"2024-03-19T02:51:40.717740+0000","last_peered":"2024-03-19T02:51:40.717740+0000","last_clean":"2024-03-19T02:51:40.717740+0000","last_became_active":"2024-03-19T02:50:52.223773+0000","last_became_peered":"2024-03-19T02:50:52.223773+0000","last_unstale":"2024-03-19T02:51:40.717740+0000","last_undegraded":"2024-03-19T02:51:40.717740+0000","last_fullsized":"2024-03-19T02:51:40.717740+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:56:03.098858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715398+0000","last_change":"2024-03-19T02:50:57.652515+0000","last_active":"2024-03-19T02:51:40.715398+0000","last_peered":"2024-03-19T02:51:40.715398+0000","last_clean":"2024-03-19T02:51:40.715398+0000","last_became_active":"2024-03-19T02:50:57.651808+0000","last_became_peered":"2024-03-19T02:50:57.651808+0000","last_unstale":"2024-03-19T02:51:40.715398+0000","last_undegraded":"2024-03-19T02:51:40.715398+0000","last_fullsized":"2024-03-19T02:51:40.715398+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:16:34.550457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.718884+0000","last_change":"2024-03-19T02:50:51.794282+0000","last_active":"2024-03-19T02:51:40.718884+0000","last_peered":"2024-03-19T02:51:40.718884+0000","last_clean":"2024-03-19T02:51:40.718884+0000","last_became_active":"2024-03-19T02:50:51.793947+0000","last_became_peered":"2024-03-19T02:50:51.793947+0000","last_unstale":"2024-03-19T02:51:40.718884+0000","last_undegraded":"2024-03-19T02:51:40.718884+0000","last_fullsized":"2024-03-19T02:51:40.718884+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:08:54.764915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.718639+0000","last_change":"2024-03-19T02:50:52.221707+0000","last_active":"2024-03-19T02:51:40.718639+0000","last_peered":"2024-03-19T02:51:40.718639+0000","last_clean":"2024-03-19T02:51:40.718639+0000","last_became_active":"2024-03-19T02:50:52.221417+0000","last_became_peered":"2024-03-19T02:50:52.221417+0000","last_unstale":"2024-03-19T02:51:40.718639+0000","last_undegraded":"2024-03-19T02:51:40.718639+0000","last_fullsized":"2024-03-19T02:51:40.718639+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:59:27.528346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719877+0000","last_change":"2024-03-19T02:50:57.257552+0000","last_active":"2024-03-19T02:51:40.719877+0000","last_peered":"2024-03-19T02:51:40.719877+0000","last_clean":"2024-03-19T02:51:40.719877+0000","last_became_active":"2024-03-19T02:50:57.257121+0000","last_became_peered":"2024-03-19T02:50:57.257121+0000","last_unstale":"2024-03-19T02:51:40.719877+0000","last_undegraded":"2024-03-19T02:51:40.719877+0000","last_fullsized":"2024-03-19T02:51:40.719877+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:43:47.145344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"93'1","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.718626+0000","last_change":"2024-03-19T02:50:51.804890+0000","last_active":"2024-03-19T02:51:40.718626+0000","last_peered":"2024-03-19T02:51:40.718626+0000","last_clean":"2024-03-19T02:51:40.718626+0000","last_became_active":"2024-03-19T02:50:51.804545+0000","last_became_peered":"2024-03-19T02:50:51.804545+0000","last_unstale":"2024-03-19T02:51:40.718626+0000","last_undegraded":"2024-03-19T02:51:40.718626+0000","last_fullsized":"2024-03-19T02:51:40.718626+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:16:14.945372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.718269+0000","last_change":"2024-03-19T02:50:52.224849+0000","last_active":"2024-03-19T02:51:40.718269+0000","last_peered":"2024-03-19T02:51:40.718269+0000","last_clean":"2024-03-19T02:51:40.718269+0000","last_became_active":"2024-03-19T02:50:52.224550+0000","last_became_peered":"2024-03-19T02:50:52.224550+0000","last_unstale":"2024-03-19T02:51:40.718269+0000","last_undegraded":"2024-03-19T02:51:40.718269+0000","last_fullsized":"2024-03-19T02:51:40.718269+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:28:59.683849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.720381+0000","last_change":"2024-03-19T02:50:57.652425+0000","last_active":"2024-03-19T02:51:40.720381+0000","last_peered":"2024-03-19T02:51:40.720381+0000","last_clean":"2024-03-19T02:51:40.720381+0000","last_became_active":"2024-03-19T02:50:57.651975+0000","last_became_peered":"2024-03-19T02:50:57.651975+0000","last_unstale":"2024-03-19T02:51:40.720381+0000","last_undegraded":"2024-03-19T02:51:40.720381+0000","last_fullsized":"2024-03-19T02:51:40.720381+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:07:20.587918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.723382+0000","last_change":"2024-03-19T02:50:52.218207+0000","last_active":"2024-03-19T02:51:40.723382+0000","last_peered":"2024-03-19T02:51:40.723382+0000","last_clean":"2024-03-19T02:51:40.723382+0000","last_became_active":"2024-03-19T02:50:52.217012+0000","last_became_peered":"2024-03-19T02:50:52.217012+0000","last_unstale":"2024-03-19T02:51:40.723382+0000","last_undegraded":"2024-03-19T02:51:40.723382+0000","last_fullsized":"2024-03-19T02:51:40.723382+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:07:47.366481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'4","reported_seq":106,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:52:29.170395+0000","last_change":"2024-03-19T02:50:51.801405+0000","last_active":"2024-03-19T02:52:29.170395+0000","last_peered":"2024-03-19T02:52:29.170395+0000","last_clean":"2024-03-19T02:52:29.170395+0000","last_became_active":"2024-03-19T02:50:51.801094+0000","last_became_peered":"2024-03-19T02:50:51.801094+0000","last_unstale":"2024-03-19T02:52:29.170395+0000","last_undegraded":"2024-03-19T02:52:29.170395+0000","last_fullsized":"2024-03-19T02:52:29.170395+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:53:39.534977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":72,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"93'1","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.730836+0000","last_change":"2024-03-19T02:51:40.730836+0000","last_active":"2024-03-19T02:51:40.730836+0000","last_peered":"2024-03-19T02:51:40.730836+0000","last_clean":"2024-03-19T02:51:40.730836+0000","last_became_active":"2024-03-19T02:51:40.723984+0000","last_became_peered":"2024-03-19T02:51:40.723984+0000","last_unstale":"2024-03-19T02:51:40.730836+0000","last_undegraded":"2024-03-19T02:51:40.730836+0000","last_fullsized":"2024-03-19T02:51:40.730836+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:29:57.301066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715624+0000","last_change":"2024-03-19T02:50:52.221852+0000","last_active":"2024-03-19T02:51:40.715624+0000","last_peered":"2024-03-19T02:51:40.715624+0000","last_clean":"2024-03-19T02:51:40.715624+0000","last_became_active":"2024-03-19T02:50:52.221535+0000","last_became_peered":"2024-03-19T02:50:52.221535+0000","last_unstale":"2024-03-19T02:51:40.715624+0000","last_undegraded":"2024-03-19T02:51:40.715624+0000","last_fullsized":"2024-03-19T02:51:40.715624+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:37:18.030714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"93'3","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715607+0000","last_change":"2024-03-19T02:50:51.790231+0000","last_active":"2024-03-19T02:51:40.715607+0000","last_peered":"2024-03-19T02:51:40.715607+0000","last_clean":"2024-03-19T02:51:40.715607+0000","last_became_active":"2024-03-19T02:50:51.789867+0000","last_became_peered":"2024-03-19T02:50:51.789867+0000","last_unstale":"2024-03-19T02:51:40.715607+0000","last_undegraded":"2024-03-19T02:51:40.715607+0000","last_fullsized":"2024-03-19T02:51:40.715607+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:46.099191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":366,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719128+0000","last_change":"2024-03-19T02:49:38.697673+0000","last_active":"2024-03-19T02:51:40.719128+0000","last_peered":"2024-03-19T02:51:40.719128+0000","last_clean":"2024-03-19T02:51:40.719128+0000","last_became_active":"2024-03-19T02:49:38.697161+0000","last_became_peered":"2024-03-19T02:49:38.697161+0000","last_unstale":"2024-03-19T02:51:40.719128+0000","last_undegraded":"2024-03-19T02:51:40.719128+0000","last_fullsized":"2024-03-19T02:51:40.719128+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:46:10.013962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:46:10.013962+0000","last_clean_scrub_stamp":"2024-03-19T02:46:10.013962+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:59:35.179938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.722715+0000","last_change":"2024-03-19T02:50:57.652428+0000","last_active":"2024-03-19T02:51:40.722715+0000","last_peered":"2024-03-19T02:51:40.722715+0000","last_clean":"2024-03-19T02:51:40.722715+0000","last_became_active":"2024-03-19T02:50:57.651864+0000","last_became_peered":"2024-03-19T02:50:57.651864+0000","last_unstale":"2024-03-19T02:51:40.722715+0000","last_undegraded":"2024-03-19T02:51:40.722715+0000","last_fullsized":"2024-03-19T02:51:40.722715+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:52:19.007175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.717092+0000","last_change":"2024-03-19T02:50:52.218728+0000","last_active":"2024-03-19T02:51:40.717092+0000","last_peered":"2024-03-19T02:51:40.717092+0000","last_clean":"2024-03-19T02:51:40.717092+0000","last_became_active":"2024-03-19T02:50:52.218407+0000","last_became_peered":"2024-03-19T02:50:52.218407+0000","last_unstale":"2024-03-19T02:51:40.717092+0000","last_undegraded":"2024-03-19T02:51:40.717092+0000","last_fullsized":"2024-03-19T02:51:40.717092+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:30:49.304034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"93'3","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719242+0000","last_change":"2024-03-19T02:50:51.802057+0000","last_active":"2024-03-19T02:51:40.719242+0000","last_peered":"2024-03-19T02:51:40.719242+0000","last_clean":"2024-03-19T02:51:40.719242+0000","last_became_active":"2024-03-19T02:50:51.801754+0000","last_became_peered":"2024-03-19T02:50:51.801754+0000","last_unstale":"2024-03-19T02:51:40.719242+0000","last_undegraded":"2024-03-19T02:51:40.719242+0000","last_fullsized":"2024-03-19T02:51:40.719242+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:23:15.401822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":19,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.717091+0000","last_change":"2024-03-19T02:50:57.260303+0000","last_active":"2024-03-19T02:51:40.717091+0000","last_peered":"2024-03-19T02:51:40.717091+0000","last_clean":"2024-03-19T02:51:40.717091+0000","last_became_active":"2024-03-19T02:50:57.259792+0000","last_became_peered":"2024-03-19T02:50:57.259792+0000","last_unstale":"2024-03-19T02:51:40.717091+0000","last_undegraded":"2024-03-19T02:51:40.717091+0000","last_fullsized":"2024-03-19T02:51:40.717091+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:03.072161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"93'1","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715677+0000","last_change":"2024-03-19T02:50:51.797949+0000","last_active":"2024-03-19T02:51:40.715677+0000","last_peered":"2024-03-19T02:51:40.715677+0000","last_clean":"2024-03-19T02:51:40.715677+0000","last_became_active":"2024-03-19T02:50:51.797522+0000","last_became_peered":"2024-03-19T02:50:51.797522+0000","last_unstale":"2024-03-19T02:51:40.715677+0000","last_undegraded":"2024-03-19T02:51:40.715677+0000","last_fullsized":"2024-03-19T02:51:40.715677+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:34:06.127993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.720131+0000","last_change":"2024-03-19T02:50:52.224095+0000","last_active":"2024-03-19T02:51:40.720131+0000","last_peered":"2024-03-19T02:51:40.720131+0000","last_clean":"2024-03-19T02:51:40.720131+0000","last_became_active":"2024-03-19T02:50:52.223764+0000","last_became_peered":"2024-03-19T02:50:52.223764+0000","last_unstale":"2024-03-19T02:51:40.720131+0000","last_undegraded":"2024-03-19T02:51:40.720131+0000","last_fullsized":"2024-03-19T02:51:40.720131+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:07:41.503162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715305+0000","last_change":"2024-03-19T02:50:57.652274+0000","last_active":"2024-03-19T02:51:40.715305+0000","last_peered":"2024-03-19T02:51:40.715305+0000","last_clean":"2024-03-19T02:51:40.715305+0000","last_became_active":"2024-03-19T02:50:57.651885+0000","last_became_peered":"2024-03-19T02:50:57.651885+0000","last_unstale":"2024-03-19T02:51:40.715305+0000","last_undegraded":"2024-03-19T02:51:40.715305+0000","last_fullsized":"2024-03-19T02:51:40.715305+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:03:02.081143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'8","reported_seq":39,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719329+0000","last_change":"2024-03-19T02:50:51.796308+0000","last_active":"2024-03-19T02:51:40.719329+0000","last_peered":"2024-03-19T02:51:40.719329+0000","last_clean":"2024-03-19T02:51:40.719329+0000","last_became_active":"2024-03-19T02:50:51.795990+0000","last_became_peered":"2024-03-19T02:50:51.795990+0000","last_unstale":"2024-03-19T02:51:40.719329+0000","last_undegraded":"2024-03-19T02:51:40.719329+0000","last_fullsized":"2024-03-19T02:51:40.719329+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:45:53.608817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.716308+0000","last_change":"2024-03-19T02:50:52.221983+0000","last_active":"2024-03-19T02:51:40.716308+0000","last_peered":"2024-03-19T02:51:40.716308+0000","last_clean":"2024-03-19T02:51:40.716308+0000","last_became_active":"2024-03-19T02:50:52.220605+0000","last_became_peered":"2024-03-19T02:50:52.220605+0000","last_unstale":"2024-03-19T02:51:40.716308+0000","last_undegraded":"2024-03-19T02:51:40.716308+0000","last_fullsized":"2024-03-19T02:51:40.716308+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:12.834159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.718148+0000","last_change":"2024-03-19T02:50:57.242428+0000","last_active":"2024-03-19T02:51:40.718148+0000","last_peered":"2024-03-19T02:51:40.718148+0000","last_clean":"2024-03-19T02:51:40.718148+0000","last_became_active":"2024-03-19T02:50:57.242007+0000","last_became_peered":"2024-03-19T02:50:57.242007+0000","last_unstale":"2024-03-19T02:51:40.718148+0000","last_undegraded":"2024-03-19T02:51:40.718148+0000","last_fullsized":"2024-03-19T02:51:40.718148+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:23:30.987761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"93'4","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715977+0000","last_change":"2024-03-19T02:50:51.791342+0000","last_active":"2024-03-19T02:51:40.715977+0000","last_peered":"2024-03-19T02:51:40.715977+0000","last_clean":"2024-03-19T02:51:40.715977+0000","last_became_active":"2024-03-19T02:50:51.791039+0000","last_became_peered":"2024-03-19T02:50:51.791039+0000","last_unstale":"2024-03-19T02:51:40.715977+0000","last_undegraded":"2024-03-19T02:51:40.715977+0000","last_fullsized":"2024-03-19T02:51:40.715977+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:50:38.216623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.720588+0000","last_change":"2024-03-19T02:50:52.224303+0000","last_active":"2024-03-19T02:51:40.720588+0000","last_peered":"2024-03-19T02:51:40.720588+0000","last_clean":"2024-03-19T02:51:40.720588+0000","last_became_active":"2024-03-19T02:50:52.223942+0000","last_became_peered":"2024-03-19T02:50:52.223942+0000","last_unstale":"2024-03-19T02:51:40.720588+0000","last_undegraded":"2024-03-19T02:51:40.720588+0000","last_fullsized":"2024-03-19T02:51:40.720588+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:57:04.175971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715762+0000","last_change":"2024-03-19T02:50:57.652126+0000","last_active":"2024-03-19T02:51:40.715762+0000","last_peered":"2024-03-19T02:51:40.715762+0000","last_clean":"2024-03-19T02:51:40.715762+0000","last_became_active":"2024-03-19T02:50:57.651713+0000","last_became_peered":"2024-03-19T02:50:57.651713+0000","last_unstale":"2024-03-19T02:51:40.715762+0000","last_undegraded":"2024-03-19T02:51:40.715762+0000","last_fullsized":"2024-03-19T02:51:40.715762+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:28:51.077437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"93'2","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.741378+0000","last_change":"2024-03-19T02:51:40.741378+0000","last_active":"2024-03-19T02:51:40.741378+0000","last_peered":"2024-03-19T02:51:40.741378+0000","last_clean":"2024-03-19T02:51:40.741378+0000","last_became_active":"2024-03-19T02:51:40.724510+0000","last_became_peered":"2024-03-19T02:51:40.724510+0000","last_unstale":"2024-03-19T02:51:40.741378+0000","last_undegraded":"2024-03-19T02:51:40.741378+0000","last_fullsized":"2024-03-19T02:51:40.741378+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:30:16.228635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715268+0000","last_change":"2024-03-19T02:50:52.221681+0000","last_active":"2024-03-19T02:51:40.715268+0000","last_peered":"2024-03-19T02:51:40.715268+0000","last_clean":"2024-03-19T02:51:40.715268+0000","last_became_active":"2024-03-19T02:50:52.220987+0000","last_became_peered":"2024-03-19T02:50:52.220987+0000","last_unstale":"2024-03-19T02:51:40.715268+0000","last_undegraded":"2024-03-19T02:51:40.715268+0000","last_fullsized":"2024-03-19T02:51:40.715268+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:16:36.561143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719714+0000","last_change":"2024-03-19T02:50:57.249830+0000","last_active":"2024-03-19T02:51:40.719714+0000","last_peered":"2024-03-19T02:51:40.719714+0000","last_clean":"2024-03-19T02:51:40.719714+0000","last_became_active":"2024-03-19T02:50:57.248879+0000","last_became_peered":"2024-03-19T02:50:57.248879+0000","last_unstale":"2024-03-19T02:51:40.719714+0000","last_undegraded":"2024-03-19T02:51:40.719714+0000","last_fullsized":"2024-03-19T02:51:40.719714+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:58:43.225841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"93'3","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715344+0000","last_change":"2024-03-19T02:50:51.801374+0000","last_active":"2024-03-19T02:51:40.715344+0000","last_peered":"2024-03-19T02:51:40.715344+0000","last_clean":"2024-03-19T02:51:40.715344+0000","last_became_active":"2024-03-19T02:50:51.801041+0000","last_became_peered":"2024-03-19T02:50:51.801041+0000","last_unstale":"2024-03-19T02:51:40.715344+0000","last_undegraded":"2024-03-19T02:51:40.715344+0000","last_fullsized":"2024-03-19T02:51:40.715344+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:56:14.671718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.717377+0000","last_change":"2024-03-19T02:50:52.225915+0000","last_active":"2024-03-19T02:51:40.717377+0000","last_peered":"2024-03-19T02:51:40.717377+0000","last_clean":"2024-03-19T02:51:40.717377+0000","last_became_active":"2024-03-19T02:50:52.225606+0000","last_became_peered":"2024-03-19T02:50:52.225606+0000","last_unstale":"2024-03-19T02:51:40.717377+0000","last_undegraded":"2024-03-19T02:51:40.717377+0000","last_fullsized":"2024-03-19T02:51:40.717377+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:27:29.549255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"93'1","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.716380+0000","last_change":"2024-03-19T02:50:57.240747+0000","last_active":"2024-03-19T02:51:40.716380+0000","last_peered":"2024-03-19T02:51:40.716380+0000","last_clean":"2024-03-19T02:51:40.716380+0000","last_became_active":"2024-03-19T02:50:57.240373+0000","last_became_peered":"2024-03-19T02:50:57.240373+0000","last_unstale":"2024-03-19T02:51:40.716380+0000","last_undegraded":"2024-03-19T02:51:40.716380+0000","last_fullsized":"2024-03-19T02:51:40.716380+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:22:04.991497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"93'1","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.718520+0000","last_change":"2024-03-19T02:50:51.794071+0000","last_active":"2024-03-19T02:51:40.718520+0000","last_peered":"2024-03-19T02:51:40.718520+0000","last_clean":"2024-03-19T02:51:40.718520+0000","last_became_active":"2024-03-19T02:50:51.793570+0000","last_became_peered":"2024-03-19T02:50:51.793570+0000","last_unstale":"2024-03-19T02:51:40.718520+0000","last_undegraded":"2024-03-19T02:51:40.718520+0000","last_fullsized":"2024-03-19T02:51:40.718520+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:34:20.456515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715731+0000","last_change":"2024-03-19T02:50:52.223341+0000","last_active":"2024-03-19T02:51:40.715731+0000","last_peered":"2024-03-19T02:51:40.715731+0000","last_clean":"2024-03-19T02:51:40.715731+0000","last_became_active":"2024-03-19T02:50:52.220280+0000","last_became_peered":"2024-03-19T02:50:52.220280+0000","last_unstale":"2024-03-19T02:51:40.715731+0000","last_undegraded":"2024-03-19T02:51:40.715731+0000","last_fullsized":"2024-03-19T02:51:40.715731+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:04:15.830811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715695+0000","last_change":"2024-03-19T02:50:57.651293+0000","last_active":"2024-03-19T02:51:40.715695+0000","last_peered":"2024-03-19T02:51:40.715695+0000","last_clean":"2024-03-19T02:51:40.715695+0000","last_became_active":"2024-03-19T02:50:57.650893+0000","last_became_peered":"2024-03-19T02:50:57.650893+0000","last_unstale":"2024-03-19T02:51:40.715695+0000","last_undegraded":"2024-03-19T02:51:40.715695+0000","last_fullsized":"2024-03-19T02:51:40.715695+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:33:35.984728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"93'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.715348+0000","last_change":"2024-03-19T02:50:51.798297+0000","last_active":"2024-03-19T02:51:40.715348+0000","last_peered":"2024-03-19T02:51:40.715348+0000","last_clean":"2024-03-19T02:51:40.715348+0000","last_became_active":"2024-03-19T02:50:51.797516+0000","last_became_peered":"2024-03-19T02:50:51.797516+0000","last_unstale":"2024-03-19T02:51:40.715348+0000","last_undegraded":"2024-03-19T02:51:40.715348+0000","last_fullsized":"2024-03-19T02:51:40.715348+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:09:42.295094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.718736+0000","last_change":"2024-03-19T02:50:52.224011+0000","last_active":"2024-03-19T02:51:40.718736+0000","last_peered":"2024-03-19T02:51:40.718736+0000","last_clean":"2024-03-19T02:51:40.718736+0000","last_became_active":"2024-03-19T02:50:52.223708+0000","last_became_peered":"2024-03-19T02:50:52.223708+0000","last_unstale":"2024-03-19T02:51:40.718736+0000","last_undegraded":"2024-03-19T02:51:40.718736+0000","last_fullsized":"2024-03-19T02:51:40.718736+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:17:49.305587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719502+0000","last_change":"2024-03-19T02:50:57.251134+0000","last_active":"2024-03-19T02:51:40.719502+0000","last_peered":"2024-03-19T02:51:40.719502+0000","last_clean":"2024-03-19T02:51:40.719502+0000","last_became_active":"2024-03-19T02:50:57.250761+0000","last_became_peered":"2024-03-19T02:50:57.250761+0000","last_unstale":"2024-03-19T02:51:40.719502+0000","last_undegraded":"2024-03-19T02:51:40.719502+0000","last_fullsized":"2024-03-19T02:51:40.719502+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:09:26.860235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"93'3","reported_seq":17,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.728077+0000","last_change":"2024-03-19T02:51:40.728077+0000","last_active":"2024-03-19T02:51:40.728077+0000","last_peered":"2024-03-19T02:51:40.728077+0000","last_clean":"2024-03-19T02:51:40.728077+0000","last_became_active":"2024-03-19T02:51:40.720767+0000","last_became_peered":"2024-03-19T02:51:40.720767+0000","last_unstale":"2024-03-19T02:51:40.728077+0000","last_undegraded":"2024-03-19T02:51:40.728077+0000","last_fullsized":"2024-03-19T02:51:40.728077+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:22:34.366373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.720568+0000","last_change":"2024-03-19T02:50:52.224776+0000","last_active":"2024-03-19T02:51:40.720568+0000","last_peered":"2024-03-19T02:51:40.720568+0000","last_clean":"2024-03-19T02:51:40.720568+0000","last_became_active":"2024-03-19T02:50:52.224442+0000","last_became_peered":"2024-03-19T02:50:52.224442+0000","last_unstale":"2024-03-19T02:51:40.720568+0000","last_undegraded":"2024-03-19T02:51:40.720568+0000","last_fullsized":"2024-03-19T02:51:40.720568+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:11:13.722005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.722732+0000","last_change":"2024-03-19T02:51:40.722732+0000","last_active":"2024-03-19T02:51:40.722732+0000","last_peered":"2024-03-19T02:51:40.722732+0000","last_clean":"2024-03-19T02:51:40.722732+0000","last_became_active":"2024-03-19T02:51:40.722260+0000","last_became_peered":"2024-03-19T02:51:40.722260+0000","last_unstale":"2024-03-19T02:51:40.722732+0000","last_undegraded":"2024-03-19T02:51:40.722732+0000","last_fullsized":"2024-03-19T02:51:40.722732+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:31:06.383570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.722310+0000","last_change":"2024-03-19T02:50:51.794898+0000","last_active":"2024-03-19T02:51:40.722310+0000","last_peered":"2024-03-19T02:51:40.722310+0000","last_clean":"2024-03-19T02:51:40.722310+0000","last_became_active":"2024-03-19T02:50:51.794562+0000","last_became_peered":"2024-03-19T02:50:51.794562+0000","last_unstale":"2024-03-19T02:51:40.722310+0000","last_undegraded":"2024-03-19T02:51:40.722310+0000","last_fullsized":"2024-03-19T02:51:40.722310+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:17:45.239395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719705+0000","last_change":"2024-03-19T02:50:52.227427+0000","last_active":"2024-03-19T02:51:40.719705+0000","last_peered":"2024-03-19T02:51:40.719705+0000","last_clean":"2024-03-19T02:51:40.719705+0000","last_became_active":"2024-03-19T02:50:52.227111+0000","last_became_peered":"2024-03-19T02:50:52.227111+0000","last_unstale":"2024-03-19T02:51:40.719705+0000","last_undegraded":"2024-03-19T02:51:40.719705+0000","last_fullsized":"2024-03-19T02:51:40.719705+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:40:53.295454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.722973+0000","last_change":"2024-03-19T02:50:57.244449+0000","last_active":"2024-03-19T02:51:40.722973+0000","last_peered":"2024-03-19T02:51:40.722973+0000","last_clean":"2024-03-19T02:51:40.722973+0000","last_became_active":"2024-03-19T02:50:57.243991+0000","last_became_peered":"2024-03-19T02:50:57.243991+0000","last_unstale":"2024-03-19T02:51:40.722973+0000","last_undegraded":"2024-03-19T02:51:40.722973+0000","last_fullsized":"2024-03-19T02:51:40.722973+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:50:45.265773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719481+0000","last_change":"2024-03-19T02:50:51.799834+0000","last_active":"2024-03-19T02:51:40.719481+0000","last_peered":"2024-03-19T02:51:40.719481+0000","last_clean":"2024-03-19T02:51:40.719481+0000","last_became_active":"2024-03-19T02:50:51.799563+0000","last_became_peered":"2024-03-19T02:50:51.799563+0000","last_unstale":"2024-03-19T02:51:40.719481+0000","last_undegraded":"2024-03-19T02:51:40.719481+0000","last_fullsized":"2024-03-19T02:51:40.719481+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:51:00.284454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719916+0000","last_change":"2024-03-19T02:50:52.222710+0000","last_active":"2024-03-19T02:51:40.719916+0000","last_peered":"2024-03-19T02:51:40.719916+0000","last_clean":"2024-03-19T02:51:40.719916+0000","last_became_active":"2024-03-19T02:50:52.222321+0000","last_became_peered":"2024-03-19T02:50:52.222321+0000","last_unstale":"2024-03-19T02:51:40.719916+0000","last_undegraded":"2024-03-19T02:51:40.719916+0000","last_fullsized":"2024-03-19T02:51:40.719916+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:05:27.527696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.717431+0000","last_change":"2024-03-19T02:50:52.226631+0000","last_active":"2024-03-19T02:51:40.717431+0000","last_peered":"2024-03-19T02:51:40.717431+0000","last_clean":"2024-03-19T02:51:40.717431+0000","last_became_active":"2024-03-19T02:50:52.226294+0000","last_became_peered":"2024-03-19T02:50:52.226294+0000","last_unstale":"2024-03-19T02:51:40.717431+0000","last_undegraded":"2024-03-19T02:51:40.717431+0000","last_fullsized":"2024-03-19T02:51:40.717431+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:01:31.350063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.718930+0000","last_change":"2024-03-19T02:50:52.221138+0000","last_active":"2024-03-19T02:51:40.718930+0000","last_peered":"2024-03-19T02:51:40.718930+0000","last_clean":"2024-03-19T02:51:40.718930+0000","last_became_active":"2024-03-19T02:50:52.220730+0000","last_became_peered":"2024-03-19T02:50:52.220730+0000","last_unstale":"2024-03-19T02:51:40.718930+0000","last_undegraded":"2024-03-19T02:51:40.718930+0000","last_fullsized":"2024-03-19T02:51:40.718930+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:26:13.059986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.718416+0000","last_change":"2024-03-19T02:50:52.223567+0000","last_active":"2024-03-19T02:51:40.718416+0000","last_peered":"2024-03-19T02:51:40.718416+0000","last_clean":"2024-03-19T02:51:40.718416+0000","last_became_active":"2024-03-19T02:50:52.223174+0000","last_became_peered":"2024-03-19T02:50:52.223174+0000","last_unstale":"2024-03-19T02:51:40.718416+0000","last_undegraded":"2024-03-19T02:51:40.718416+0000","last_fullsized":"2024-03-19T02:51:40.718416+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:23:06.383584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.720173+0000","last_change":"2024-03-19T02:50:52.229916+0000","last_active":"2024-03-19T02:51:40.720173+0000","last_peered":"2024-03-19T02:51:40.720173+0000","last_clean":"2024-03-19T02:51:40.720173+0000","last_became_active":"2024-03-19T02:50:52.229597+0000","last_became_peered":"2024-03-19T02:50:52.229597+0000","last_unstale":"2024-03-19T02:51:40.720173+0000","last_undegraded":"2024-03-19T02:51:40.720173+0000","last_fullsized":"2024-03-19T02:51:40.720173+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:19.331893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.720033+0000","last_change":"2024-03-19T02:50:52.223331+0000","last_active":"2024-03-19T02:51:40.720033+0000","last_peered":"2024-03-19T02:51:40.720033+0000","last_clean":"2024-03-19T02:51:40.720033+0000","last_became_active":"2024-03-19T02:50:52.222975+0000","last_became_peered":"2024-03-19T02:50:52.222975+0000","last_unstale":"2024-03-19T02:51:40.720033+0000","last_undegraded":"2024-03-19T02:51:40.720033+0000","last_fullsized":"2024-03-19T02:51:40.720033+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:26:23.023749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.717303+0000","last_change":"2024-03-19T02:50:52.218073+0000","last_active":"2024-03-19T02:51:40.717303+0000","last_peered":"2024-03-19T02:51:40.717303+0000","last_clean":"2024-03-19T02:51:40.717303+0000","last_became_active":"2024-03-19T02:50:52.217742+0000","last_became_peered":"2024-03-19T02:50:52.217742+0000","last_unstale":"2024-03-19T02:51:40.717303+0000","last_undegraded":"2024-03-19T02:51:40.717303+0000","last_fullsized":"2024-03-19T02:51:40.717303+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:08:51.053153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719458+0000","last_change":"2024-03-19T02:50:52.222108+0000","last_active":"2024-03-19T02:51:40.719458+0000","last_peered":"2024-03-19T02:51:40.719458+0000","last_clean":"2024-03-19T02:51:40.719458+0000","last_became_active":"2024-03-19T02:50:52.221748+0000","last_became_peered":"2024-03-19T02:50:52.221748+0000","last_unstale":"2024-03-19T02:51:40.719458+0000","last_undegraded":"2024-03-19T02:51:40.719458+0000","last_fullsized":"2024-03-19T02:51:40.719458+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:33:31.401939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.716494+0000","last_change":"2024-03-19T02:50:52.221799+0000","last_active":"2024-03-19T02:51:40.716494+0000","last_peered":"2024-03-19T02:51:40.716494+0000","last_clean":"2024-03-19T02:51:40.716494+0000","last_became_active":"2024-03-19T02:50:52.220196+0000","last_became_peered":"2024-03-19T02:50:52.220196+0000","last_unstale":"2024-03-19T02:51:40.716494+0000","last_undegraded":"2024-03-19T02:51:40.716494+0000","last_fullsized":"2024-03-19T02:51:40.716494+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:24:01.541716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.721422+0000","last_change":"2024-03-19T02:51:40.721422+0000","last_active":"2024-03-19T02:51:40.721422+0000","last_peered":"2024-03-19T02:51:40.721422+0000","last_clean":"2024-03-19T02:51:40.721422+0000","last_became_active":"2024-03-19T02:51:40.720982+0000","last_became_peered":"2024-03-19T02:51:40.720982+0000","last_unstale":"2024-03-19T02:51:40.721422+0000","last_undegraded":"2024-03-19T02:51:40.721422+0000","last_fullsized":"2024-03-19T02:51:40.721422+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:07.391810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,3],"acting":[6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.721832+0000","last_change":"2024-03-19T02:51:40.721188+0000","last_active":"2024-03-19T02:51:40.721832+0000","last_peered":"2024-03-19T02:51:40.721832+0000","last_clean":"2024-03-19T02:51:40.721832+0000","last_became_active":"2024-03-19T02:51:40.720811+0000","last_became_peered":"2024-03-19T02:51:40.720811+0000","last_unstale":"2024-03-19T02:51:40.721832+0000","last_undegraded":"2024-03-19T02:51:40.721832+0000","last_fullsized":"2024-03-19T02:51:40.721832+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:23.970149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719742+0000","last_change":"2024-03-19T02:50:52.230847+0000","last_active":"2024-03-19T02:51:40.719742+0000","last_peered":"2024-03-19T02:51:40.719742+0000","last_clean":"2024-03-19T02:51:40.719742+0000","last_became_active":"2024-03-19T02:50:52.230478+0000","last_became_peered":"2024-03-19T02:50:52.230478+0000","last_unstale":"2024-03-19T02:51:40.719742+0000","last_undegraded":"2024-03-19T02:51:40.719742+0000","last_fullsized":"2024-03-19T02:51:40.719742+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:04:09.459446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":28,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.719284+0000","last_change":"2024-03-19T02:50:52.223508+0000","last_active":"2024-03-19T02:51:40.719284+0000","last_peered":"2024-03-19T02:51:40.719284+0000","last_clean":"2024-03-19T02:51:40.719284+0000","last_became_active":"2024-03-19T02:50:52.223173+0000","last_became_peered":"2024-03-19T02:50:52.223173+0000","last_unstale":"2024-03-19T02:51:40.719284+0000","last_undegraded":"2024-03-19T02:51:40.719284+0000","last_fullsized":"2024-03-19T02:51:40.719284+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:53.320173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-19T02:51:40.723512+0000","last_change":"2024-03-19T02:51:40.723512+0000","last_active":"2024-03-19T02:51:40.723512+0000","last_peered":"2024-03-19T02:51:40.723512+0000","last_clean":"2024-03-19T02:51:40.723512+0000","last_became_active":"2024-03-19T02:51:40.723149+0000","last_became_peered":"2024-03-19T02:51:40.723149+0000","last_unstale":"2024-03-19T02:51:40.723512+0000","last_undegraded":"2024-03-19T02:51:40.723512+0000","last_fullsized":"2024-03-19T02:51:40.723512+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:42:38.862569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66966,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":453,"num_read_kb":296,"num_write":160,"num_write_kb":115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":694,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":200898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318305,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32240,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712912,"statfs":{"total":95995035648,"available":95962021888,"internally_reserved":0,"allocated":1081344,"data_stored":526938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28744,"kb_used_data":1728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716408,"statfs":{"total":95995035648,"available":95965601792,"internally_reserved":0,"allocated":1769472,"data_stored":1181546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743468,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1089536,"data_stored":527062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005105,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1081344,"data_stored":526870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266743,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32824,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712328,"statfs":{"total":95995035648,"available":95961423872,"internally_reserved":0,"allocated":1683456,"data_stored":1122500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462970,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1069056,"data_stored":522238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36448,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708704,"statfs":{"total":95995035648,"available":95957712896,"internally_reserved":0,"allocated":1138688,"data_stored":585504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1093632,"data_stored":523626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28148,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717004,"statfs":{"total":95995035648,"available":95966212096,"internally_reserved":0,"allocated":1159168,"data_stored":586272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149771,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1081344,"data_stored":522432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888134,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1085440,"data_stored":527470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724609,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28672,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716480,"statfs":{"total":95995035648,"available":95965675520,"internally_reserved":0,"allocated":1691648,"data_stored":1118088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":63356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":64124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":64934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T02:52:33.980 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-19T02:52:33.980 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-19T02:52:33.980 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-19T02:52:33.980 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph health --format=json 2024-03-19T02:52:34.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:33 smithi096 ceph-mon[35946]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:52:34.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:33 smithi096 ceph-mon[35946]: pgmap v352: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:52:34.227 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:52:34.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:33 smithi155 ceph-mon[35804]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:52:34.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:33 smithi155 ceph-mon[35804]: pgmap v352: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:52:36.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.025+0000 7f577f1b9700 1 -- 172.21.15.33:0/1749134037 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5778106520 msgr2=0x7f577810e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:36.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.025+0000 7f577f1b9700 1 --2- 172.21.15.33:0/1749134037 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5778106520 0x7f577810e0f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f577000b0d0 tx=0x7f577000b3e0 comp rx=0 tx=0).stop 2024-03-19T02:52:36.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.025+0000 7f577f1b9700 1 -- 172.21.15.33:0/1749134037 shutdown_connections 2024-03-19T02:52:36.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.025+0000 7f577f1b9700 1 --2- 172.21.15.33:0/1749134037 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5778106520 0x7f577810e0f0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:36.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.025+0000 7f577f1b9700 1 --2- 172.21.15.33:0/1749134037 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57781014d0 0x7f57781018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:36.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.025+0000 7f577f1b9700 1 --2- 172.21.15.33:0/1749134037 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5778101e80 0x7f5778105e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:36.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.025+0000 7f577f1b9700 1 -- 172.21.15.33:0/1749134037 >> 172.21.15.33:0/1749134037 conn(0x7f5778077260 msgr2=0x7f5778077670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:36.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.025+0000 7f577f1b9700 1 -- 172.21.15.33:0/1749134037 shutdown_connections 2024-03-19T02:52:36.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.025+0000 7f577f1b9700 1 -- 172.21.15.33:0/1749134037 wait complete. 2024-03-19T02:52:36.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.025+0000 7f577f1b9700 1 Processor -- start 2024-03-19T02:52:36.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.025+0000 7f577f1b9700 1 -- start start 2024-03-19T02:52:36.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.026+0000 7f577f1b9700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f57781014d0 0x7f5778102ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:36.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.026+0000 7f577f1b9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5778101e80 0x7f5778103420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:36.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.026+0000 7f577f1b9700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5778106520 0x7f5778103960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:36.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.026+0000 7f577f1b9700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f57781113e0 con 0x7f57781014d0 2024-03-19T02:52:36.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.026+0000 7f577f1b9700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5778111260 con 0x7f5778101e80 2024-03-19T02:52:36.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.026+0000 7f577f1b9700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5778111560 con 0x7f5778106520 2024-03-19T02:52:36.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.026+0000 7f577cf55700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f57781014d0 0x7f5778102ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:36.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.026+0000 7f577cf55700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f57781014d0 0x7f5778102ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41400/0 (socket says 172.21.15.33:41400) 2024-03-19T02:52:36.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.026+0000 7f577cf55700 1 -- 172.21.15.33:0/597055499 learned_addr learned my addr 172.21.15.33:0/597055499 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:36.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.026+0000 7f5777fff700 1 --2- 172.21.15.33:0/597055499 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5778101e80 0x7f5778103420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:36.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.026+0000 7f577d756700 1 --2- 172.21.15.33:0/597055499 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5778106520 0x7f5778103960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:36.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.026+0000 7f577cf55700 1 -- 172.21.15.33:0/597055499 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5778106520 msgr2=0x7f5778103960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:36.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.026+0000 7f577cf55700 1 --2- 172.21.15.33:0/597055499 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5778106520 0x7f5778103960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:36.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.026+0000 7f577cf55700 1 -- 172.21.15.33:0/597055499 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5778101e80 msgr2=0x7f5778103420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:36.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.026+0000 7f577cf55700 1 --2- 172.21.15.33:0/597055499 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5778101e80 0x7f5778103420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:36.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.026+0000 7f577cf55700 1 -- 172.21.15.33:0/597055499 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57781ab320 con 0x7f57781014d0 2024-03-19T02:52:36.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.026+0000 7f577d756700 1 --2- 172.21.15.33:0/597055499 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5778106520 0x7f5778103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:36.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.026+0000 7f5777fff700 1 --2- 172.21.15.33:0/597055499 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5778101e80 0x7f5778103420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:36.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.026+0000 7f577cf55700 1 --2- 172.21.15.33:0/597055499 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f57781014d0 0x7f5778102ee0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f576c00e9a0 tx=0x7f576c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:36.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.026+0000 7f5775ffb700 1 -- 172.21.15.33:0/597055499 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f576c00ca10 con 0x7f57781014d0 2024-03-19T02:52:36.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.026+0000 7f5775ffb700 1 -- 172.21.15.33:0/597055499 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f576c00cbb0 con 0x7f57781014d0 2024-03-19T02:52:36.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.026+0000 7f577f1b9700 1 -- 172.21.15.33:0/597055499 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57781ab5b0 con 0x7f57781014d0 2024-03-19T02:52:36.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.027+0000 7f5775ffb700 1 -- 172.21.15.33:0/597055499 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f576c018680 con 0x7f57781014d0 2024-03-19T02:52:36.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.027+0000 7f577f1b9700 1 -- 172.21.15.33:0/597055499 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57781abb40 con 0x7f57781014d0 2024-03-19T02:52:36.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.028+0000 7f577f1b9700 1 -- 172.21.15.33:0/597055499 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5778102920 con 0x7f57781014d0 2024-03-19T02:52:36.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.029+0000 7f5775ffb700 1 -- 172.21.15.33:0/597055499 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f576c010430 con 0x7f57781014d0 2024-03-19T02:52:36.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.029+0000 7f5775ffb700 1 --2- 172.21.15.33:0/597055499 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f57680a29a0 0x7f57680a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:36.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.029+0000 7f5777fff700 1 --2- 172.21.15.33:0/597055499 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f57680a29a0 0x7f57680a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:36.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.029+0000 7f5775ffb700 1 -- 172.21.15.33:0/597055499 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7f576c0cf090 con 0x7f57781014d0 2024-03-19T02:52:36.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.030+0000 7f5777fff700 1 --2- 172.21.15.33:0/597055499 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f57680a29a0 0x7f57680a4e60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f57781046a0 tx=0x7f5764009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:36.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.034+0000 7f5775ffb700 1 -- 172.21.15.33:0/597055499 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f576c09bc70 con 0x7f57781014d0 2024-03-19T02:52:36.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:36 smithi033 ceph-mon[31973]: pgmap v353: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:52:36.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.289+0000 7f577f1b9700 1 -- 172.21.15.33:0/597055499 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f5778104410 con 0x7f57781014d0 2024-03-19T02:52:36.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.290+0000 7f5775ffb700 1 -- 172.21.15.33:0/597055499 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f576c09b570 con 0x7f57781014d0 2024-03-19T02:52:36.291 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:52:36.291 INFO:teuthology.orchestra.run.smithi033.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-19T02:52:36.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.292+0000 7f577f1b9700 1 -- 172.21.15.33:0/597055499 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f57680a29a0 msgr2=0x7f57680a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:36.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.292+0000 7f577f1b9700 1 --2- 172.21.15.33:0/597055499 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f57680a29a0 0x7f57680a4e60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f57781046a0 tx=0x7f5764009260 comp rx=0 tx=0).stop 2024-03-19T02:52:36.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.292+0000 7f577f1b9700 1 -- 172.21.15.33:0/597055499 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f57781014d0 msgr2=0x7f5778102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:36.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.292+0000 7f577f1b9700 1 --2- 172.21.15.33:0/597055499 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f57781014d0 0x7f5778102ee0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f576c00e9a0 tx=0x7f576c00ecb0 comp rx=0 tx=0).stop 2024-03-19T02:52:36.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.292+0000 7f577f1b9700 1 -- 172.21.15.33:0/597055499 shutdown_connections 2024-03-19T02:52:36.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.292+0000 7f577f1b9700 1 --2- 172.21.15.33:0/597055499 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f57680a29a0 0x7f57680a4e60 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:36.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.292+0000 7f577f1b9700 1 --2- 172.21.15.33:0/597055499 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5778106520 0x7f5778103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:36.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.292+0000 7f577f1b9700 1 --2- 172.21.15.33:0/597055499 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f57781014d0 0x7f5778102ee0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:36.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.292+0000 7f577f1b9700 1 --2- 172.21.15.33:0/597055499 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5778101e80 0x7f5778103420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:36.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.292+0000 7f577f1b9700 1 -- 172.21.15.33:0/597055499 >> 172.21.15.33:0/597055499 conn(0x7f5778077260 msgr2=0x7f577806acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:36.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.292+0000 7f577f1b9700 1 -- 172.21.15.33:0/597055499 shutdown_connections 2024-03-19T02:52:36.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:36.292+0000 7f577f1b9700 1 -- 172.21.15.33:0/597055499 wait complete. 2024-03-19T02:52:36.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:36 smithi096 ceph-mon[35946]: pgmap v353: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:52:36.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:36 smithi155 ceph-mon[35804]: pgmap v353: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:52:37.045 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-19T02:52:37.046 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:52:37.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.155+0000 7f83a1997640 1 -- 172.21.15.33:0/1048898990 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f839c101410 msgr2=0x7f839c101890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:37.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.155+0000 7f83a1997640 1 --2- 172.21.15.33:0/1048898990 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f839c101410 0x7f839c101890 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f838c009be0 tx=0x7f838c02bef0 comp rx=0 tx=0).stop 2024-03-19T02:52:37.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.155+0000 7f83a1997640 1 -- 172.21.15.33:0/1048898990 shutdown_connections 2024-03-19T02:52:37.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.155+0000 7f83a1997640 1 --2- 172.21.15.33:0/1048898990 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f839c101410 0x7f839c101890 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:37.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.155+0000 7f83a1997640 1 --2- 172.21.15.33:0/1048898990 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f839c100a70 0x7f839c100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:37.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.155+0000 7f83a1997640 1 --2- 172.21.15.33:0/1048898990 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f839c100110 0x7f839c100530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:37.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.155+0000 7f83a1997640 1 -- 172.21.15.33:0/1048898990 >> 172.21.15.33:0/1048898990 conn(0x7f839c0fbcb0 msgr2=0x7f839c0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:37.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.155+0000 7f83a1997640 1 -- 172.21.15.33:0/1048898990 shutdown_connections 2024-03-19T02:52:37.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.155+0000 7f83a1997640 1 -- 172.21.15.33:0/1048898990 wait complete. 2024-03-19T02:52:37.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.155+0000 7f83a1997640 1 Processor -- start 2024-03-19T02:52:37.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.155+0000 7f83a1997640 1 -- start start 2024-03-19T02:52:37.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.156+0000 7f83a1997640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f839c100110 0x7f839c199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:37.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.156+0000 7f83a1997640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f839c100a70 0x7f839c19a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:37.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.156+0000 7f83a1997640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f839c101410 0x7f839c1a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:37.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.156+0000 7f83a1997640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f839c10f560 con 0x7f839c100110 2024-03-19T02:52:37.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.156+0000 7f83a1997640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f839c10f3e0 con 0x7f839c100a70 2024-03-19T02:52:37.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.156+0000 7f83a1997640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f839c10f6e0 con 0x7f839c101410 2024-03-19T02:52:37.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.156+0000 7f839affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f839c100110 0x7f839c199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:37.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.156+0000 7f839affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f839c100110 0x7f839c199c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41424/0 (socket says 172.21.15.33:41424) 2024-03-19T02:52:37.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.156+0000 7f839affd640 1 -- 172.21.15.33:0/3637724067 learned_addr learned my addr 172.21.15.33:0/3637724067 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:37.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.156+0000 7f839b7fe640 1 --2- 172.21.15.33:0/3637724067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f839c101410 0x7f839c1a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:37.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.156+0000 7f839a7fc640 1 --2- 172.21.15.33:0/3637724067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f839c100a70 0x7f839c19a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:37.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.156+0000 7f839affd640 1 -- 172.21.15.33:0/3637724067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f839c101410 msgr2=0x7f839c1a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:37.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.156+0000 7f839affd640 1 --2- 172.21.15.33:0/3637724067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f839c101410 0x7f839c1a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:37.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.156+0000 7f839affd640 1 -- 172.21.15.33:0/3637724067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f839c100a70 msgr2=0x7f839c19a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:37.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.156+0000 7f839affd640 1 --2- 172.21.15.33:0/3637724067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f839c100a70 0x7f839c19a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:37.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.156+0000 7f839affd640 1 -- 172.21.15.33:0/3637724067 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f839c1a1860 con 0x7f839c100110 2024-03-19T02:52:37.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.156+0000 7f839affd640 1 --2- 172.21.15.33:0/3637724067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f839c100110 0x7f839c199c50 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f839400e9f0 tx=0x7f839400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:37.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.156+0000 7f83a0995640 1 -- 172.21.15.33:0/3637724067 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f839400ccc0 con 0x7f839c100110 2024-03-19T02:52:37.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.156+0000 7f83a1997640 1 -- 172.21.15.33:0/3637724067 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f839c1a1b50 con 0x7f839c100110 2024-03-19T02:52:37.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.156+0000 7f83a0995640 1 -- 172.21.15.33:0/3637724067 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f839400ce60 con 0x7f839c100110 2024-03-19T02:52:37.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.156+0000 7f83a0995640 1 -- 172.21.15.33:0/3637724067 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8394010430 con 0x7f839c100110 2024-03-19T02:52:37.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.156+0000 7f83a1997640 1 -- 172.21.15.33:0/3637724067 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f839c1a2050 con 0x7f839c100110 2024-03-19T02:52:37.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.157+0000 7f83a1997640 1 -- 172.21.15.33:0/3637724067 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f839c10b050 con 0x7f839c100110 2024-03-19T02:52:37.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.159+0000 7f83a0995640 1 -- 172.21.15.33:0/3637724067 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f8394005020 con 0x7f839c100110 2024-03-19T02:52:37.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.159+0000 7f83a0995640 1 --2- 172.21.15.33:0/3637724067 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f83880a2890 0x7f83880a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:37.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.159+0000 7f83a0995640 1 -- 172.21.15.33:0/3637724067 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7f83940cf350 con 0x7f839c100110 2024-03-19T02:52:37.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.159+0000 7f839a7fc640 1 --2- 172.21.15.33:0/3637724067 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f83880a2890 0x7f83880a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:37.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.160+0000 7f839a7fc640 1 --2- 172.21.15.33:0/3637724067 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f83880a2890 0x7f83880a4d50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8384005fd0 tx=0x7f8384009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:37.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.164+0000 7f83a0995640 1 -- 172.21.15.33:0/3637724067 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f839409bfd0 con 0x7f839c100110 2024-03-19T02:52:37.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:37 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/597055499' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-19T02:52:37.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.347+0000 7f83a1997640 1 -- 172.21.15.33:0/3637724067 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f839c02d290 con 0x7f839c100110 2024-03-19T02:52:37.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.349+0000 7f83a0995640 1 -- 172.21.15.33:0/3637724067 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+24570 (secure 0 0 0) 0x7f839409b8d0 con 0x7f839c100110 2024-03-19T02:52:37.350 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:52:37.350 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":95,"fsid":"2b7a8938-e59a-11ee-95c9-87774f69a715","created":"2024-03-19T02:42:37.912179+0000","modified":"2024-03-19T02:51:40.709913+0000","last_up_change":"2024-03-19T02:50:01.539392+0000","last_in_change":"2024-03-19T02:49:40.534415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:46:09.963737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:50:50.069641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:50:51.381064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:50:55.476667+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"090a89ab-6b7b-4317-8823-178d4212b2f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6803","nonce":1633124995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6805","nonce":1633124995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6809","nonce":1633124995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6807","nonce":1633124995}]},"public_addr":"172.21.15.33:6803/1633124995","cluster_addr":"172.21.15.33:6805/1633124995","heartbeat_back_addr":"172.21.15.33:6809/1633124995","heartbeat_front_addr":"172.21.15.33:6807/1633124995","state":["exists","up"]},{"osd":1,"uuid":"f50eec55-1956-4486-8593-f2c8fa610e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6801","nonce":340144398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6803","nonce":340144398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6807","nonce":340144398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6805","nonce":340144398}]},"public_addr":"172.21.15.96:6801/340144398","cluster_addr":"172.21.15.96:6803/340144398","heartbeat_back_addr":"172.21.15.96:6807/340144398","heartbeat_front_addr":"172.21.15.96:6805/340144398","state":["exists","up"]},{"osd":2,"uuid":"893a2d62-80d9-4e5e-bc6a-a1c8c681d652","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6801","nonce":3388743876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6803","nonce":3388743876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6807","nonce":3388743876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6805","nonce":3388743876}]},"public_addr":"172.21.15.155:6801/3388743876","cluster_addr":"172.21.15.155:6803/3388743876","heartbeat_back_addr":"172.21.15.155:6807/3388743876","heartbeat_front_addr":"172.21.15.155:6805/3388743876","state":["exists","up"]},{"osd":3,"uuid":"b07e0482-4281-4d32-9dfe-9995d7246175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6811","nonce":677007171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6813","nonce":677007171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6817","nonce":677007171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6815","nonce":677007171}]},"public_addr":"172.21.15.33:6811/677007171","cluster_addr":"172.21.15.33:6813/677007171","heartbeat_back_addr":"172.21.15.33:6817/677007171","heartbeat_front_addr":"172.21.15.33:6815/677007171","state":["exists","up"]},{"osd":4,"uuid":"a988384b-af6b-47c6-8c07-5c9673acee45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6809","nonce":3106158554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6811","nonce":3106158554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6815","nonce":3106158554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6813","nonce":3106158554}]},"public_addr":"172.21.15.96:6809/3106158554","cluster_addr":"172.21.15.96:6811/3106158554","heartbeat_back_addr":"172.21.15.96:6815/3106158554","heartbeat_front_addr":"172.21.15.96:6813/3106158554","state":["exists","up"]},{"osd":5,"uuid":"198dc992-96f8-4c52-96ee-1fd5de148bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6809","nonce":327293324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6811","nonce":327293324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6815","nonce":327293324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6813","nonce":327293324}]},"public_addr":"172.21.15.155:6809/327293324","cluster_addr":"172.21.15.155:6811/327293324","heartbeat_back_addr":"172.21.15.155:6815/327293324","heartbeat_front_addr":"172.21.15.155:6813/327293324","state":["exists","up"]},{"osd":6,"uuid":"a7278ec0-ab2e-49ed-be53-84d8b9e24177","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6819","nonce":873871509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6821","nonce":873871509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6825","nonce":873871509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6823","nonce":873871509}]},"public_addr":"172.21.15.33:6819/873871509","cluster_addr":"172.21.15.33:6821/873871509","heartbeat_back_addr":"172.21.15.33:6825/873871509","heartbeat_front_addr":"172.21.15.33:6823/873871509","state":["exists","up"]},{"osd":7,"uuid":"727e133d-aa24-4ffc-90e2-266eb2aab8c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6817","nonce":2334123891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6819","nonce":2334123891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6823","nonce":2334123891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6821","nonce":2334123891}]},"public_addr":"172.21.15.96:6817/2334123891","cluster_addr":"172.21.15.96:6819/2334123891","heartbeat_back_addr":"172.21.15.96:6823/2334123891","heartbeat_front_addr":"172.21.15.96:6821/2334123891","state":["exists","up"]},{"osd":8,"uuid":"bb20eec0-39e5-4b86-9db5-486b1840cdad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6817","nonce":3498516903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6819","nonce":3498516903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6823","nonce":3498516903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6821","nonce":3498516903}]},"public_addr":"172.21.15.155:6817/3498516903","cluster_addr":"172.21.15.155:6819/3498516903","heartbeat_back_addr":"172.21.15.155:6823/3498516903","heartbeat_front_addr":"172.21.15.155:6821/3498516903","state":["exists","up"]},{"osd":9,"uuid":"179c6228-3e23-4f65-91a6-1ef0005d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6827","nonce":760777213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6829","nonce":760777213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6833","nonce":760777213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6831","nonce":760777213}]},"public_addr":"172.21.15.33:6827/760777213","cluster_addr":"172.21.15.33:6829/760777213","heartbeat_back_addr":"172.21.15.33:6833/760777213","heartbeat_front_addr":"172.21.15.33:6831/760777213","state":["exists","up"]},{"osd":10,"uuid":"be17b478-3c2e-41fb-bede-958333acdf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6825","nonce":1404898482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6827","nonce":1404898482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6831","nonce":1404898482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6829","nonce":1404898482}]},"public_addr":"172.21.15.96:6825/1404898482","cluster_addr":"172.21.15.96:6827/1404898482","heartbeat_back_addr":"172.21.15.96:6831/1404898482","heartbeat_front_addr":"172.21.15.96:6829/1404898482","state":["exists","up"]},{"osd":11,"uuid":"c0903ed2-cc1e-4c18-8731-1f0ee4dd0457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6825","nonce":792352827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6827","nonce":792352827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6831","nonce":792352827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6829","nonce":792352827}]},"public_addr":"172.21.15.155:6825/792352827","cluster_addr":"172.21.15.155:6827/792352827","heartbeat_back_addr":"172.21.15.155:6831/792352827","heartbeat_front_addr":"172.21.15.155:6829/792352827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:17.942801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:42.109449+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:05.981599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:32.069574+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:57.431015+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:22.448592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:48.526515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:14.498352+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:40.175426+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:06.364006+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:32.853161+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:59.073538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:6801/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:6800/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/1067768047":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/729379453":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2586588742":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/2932814165":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6801/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2212564963":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6800/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/3275126058":"2024-03-20T02:43:39.622218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:52:37.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.351+0000 7f83a1997640 1 -- 172.21.15.33:0/3637724067 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f83880a2890 msgr2=0x7f83880a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:37.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.351+0000 7f83a1997640 1 --2- 172.21.15.33:0/3637724067 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f83880a2890 0x7f83880a4d50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8384005fd0 tx=0x7f8384009260 comp rx=0 tx=0).stop 2024-03-19T02:52:37.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.351+0000 7f83a1997640 1 -- 172.21.15.33:0/3637724067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f839c100110 msgr2=0x7f839c199c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:37.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.351+0000 7f83a1997640 1 --2- 172.21.15.33:0/3637724067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f839c100110 0x7f839c199c50 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f839400e9f0 tx=0x7f839400edf0 comp rx=0 tx=0).stop 2024-03-19T02:52:37.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.351+0000 7f83a1997640 1 -- 172.21.15.33:0/3637724067 shutdown_connections 2024-03-19T02:52:37.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.351+0000 7f83a1997640 1 --2- 172.21.15.33:0/3637724067 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f83880a2890 0x7f83880a4d50 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:37.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.351+0000 7f83a1997640 1 --2- 172.21.15.33:0/3637724067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f839c101410 0x7f839c1a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:37.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.351+0000 7f83a1997640 1 --2- 172.21.15.33:0/3637724067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f839c100a70 0x7f839c19a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:37.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.351+0000 7f83a1997640 1 --2- 172.21.15.33:0/3637724067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f839c100110 0x7f839c199c50 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:37.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.351+0000 7f83a1997640 1 -- 172.21.15.33:0/3637724067 >> 172.21.15.33:0/3637724067 conn(0x7f839c0fbcb0 msgr2=0x7f839c0fd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:37.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.351+0000 7f83a1997640 1 -- 172.21.15.33:0/3637724067 shutdown_connections 2024-03-19T02:52:37.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.351+0000 7f83a1997640 1 -- 172.21.15.33:0/3637724067 wait complete. 2024-03-19T02:52:37.363 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:46:09.963737+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T02:50:50.069641+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T02:50:51.381064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T02:50:55.476667+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T02:52:37.364 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T02:52:37.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:37 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/597055499' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-19T02:52:37.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.469+0000 7f5a6b661640 1 -- 172.21.15.33:0/1675766599 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a641022e0 msgr2=0x7f5a64108b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:37.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.469+0000 7f5a6b661640 1 --2- 172.21.15.33:0/1675766599 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a641022e0 0x7f5a64108b70 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c009be0 tx=0x7f5a5c02bef0 comp rx=0 tx=0).stop 2024-03-19T02:52:37.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.469+0000 7f5a6b661640 1 -- 172.21.15.33:0/1675766599 shutdown_connections 2024-03-19T02:52:37.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.469+0000 7f5a6b661640 1 --2- 172.21.15.33:0/1675766599 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a641022e0 0x7f5a64108b70 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:37.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.469+0000 7f5a6b661640 1 --2- 172.21.15.33:0/1675766599 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a64101920 0x7f5a64101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:37.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.469+0000 7f5a6b661640 1 --2- 172.21.15.33:0/1675766599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a64100730 0x7f5a64100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:37.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.469+0000 7f5a6b661640 1 -- 172.21.15.33:0/1675766599 >> 172.21.15.33:0/1675766599 conn(0x7f5a640fbcc0 msgr2=0x7f5a640fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:37.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.469+0000 7f5a6b661640 1 -- 172.21.15.33:0/1675766599 shutdown_connections 2024-03-19T02:52:37.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.469+0000 7f5a6b661640 1 -- 172.21.15.33:0/1675766599 wait complete. 2024-03-19T02:52:37.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.470+0000 7f5a6b661640 1 Processor -- start 2024-03-19T02:52:37.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.470+0000 7f5a6b661640 1 -- start start 2024-03-19T02:52:37.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.470+0000 7f5a6b661640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a64100730 0x7f5a64197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:37.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.470+0000 7f5a6b661640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a64101920 0x7f5a64197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:37.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.470+0000 7f5a6b661640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a641022e0 0x7f5a6419efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:37.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.470+0000 7f5a6b661640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6410c260 con 0x7f5a64101920 2024-03-19T02:52:37.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.470+0000 7f5a6b661640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6410c0e0 con 0x7f5a64100730 2024-03-19T02:52:37.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.470+0000 7f5a6b661640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6410c3e0 con 0x7f5a641022e0 2024-03-19T02:52:37.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.470+0000 7f5a68bfc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a64101920 0x7f5a64197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:37.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.470+0000 7f5a68bfc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a64101920 0x7f5a64197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41452/0 (socket says 172.21.15.33:41452) 2024-03-19T02:52:37.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.470+0000 7f5a68bfc640 1 -- 172.21.15.33:0/821617656 learned_addr learned my addr 172.21.15.33:0/821617656 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:37.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.470+0000 7f5a693fd640 1 --2- 172.21.15.33:0/821617656 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a64100730 0x7f5a64197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:37.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.470+0000 7f5a69bfe640 1 --2- 172.21.15.33:0/821617656 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a641022e0 0x7f5a6419efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:37.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.470+0000 7f5a68bfc640 1 -- 172.21.15.33:0/821617656 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a641022e0 msgr2=0x7f5a6419efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:37.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.470+0000 7f5a68bfc640 1 --2- 172.21.15.33:0/821617656 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a641022e0 0x7f5a6419efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:37.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.470+0000 7f5a68bfc640 1 -- 172.21.15.33:0/821617656 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a64100730 msgr2=0x7f5a64197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:37.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.470+0000 7f5a68bfc640 1 --2- 172.21.15.33:0/821617656 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a64100730 0x7f5a64197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:37.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.470+0000 7f5a68bfc640 1 -- 172.21.15.33:0/821617656 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a6419f640 con 0x7f5a64101920 2024-03-19T02:52:37.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.471+0000 7f5a68bfc640 1 --2- 172.21.15.33:0/821617656 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a64101920 0x7f5a64197f60 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f5a5400e9c0 tx=0x7f5a5400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:37.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.471+0000 7f5a5a7fc640 1 -- 172.21.15.33:0/821617656 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a5400cde0 con 0x7f5a64101920 2024-03-19T02:52:37.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.471+0000 7f5a6b661640 1 -- 172.21.15.33:0/821617656 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a6419f930 con 0x7f5a64101920 2024-03-19T02:52:37.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.471+0000 7f5a5a7fc640 1 -- 172.21.15.33:0/821617656 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5a540035f0 con 0x7f5a64101920 2024-03-19T02:52:37.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.471+0000 7f5a5a7fc640 1 -- 172.21.15.33:0/821617656 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a54010690 con 0x7f5a64101920 2024-03-19T02:52:37.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.471+0000 7f5a6b661640 1 -- 172.21.15.33:0/821617656 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a6419fe80 con 0x7f5a64101920 2024-03-19T02:52:37.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.472+0000 7f5a6b661640 1 -- 172.21.15.33:0/821617656 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a641a0110 con 0x7f5a64101920 2024-03-19T02:52:37.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.473+0000 7f5a5a7fc640 1 -- 172.21.15.33:0/821617656 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f5a540037a0 con 0x7f5a64101920 2024-03-19T02:52:37.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.474+0000 7f5a5a7fc640 1 --2- 172.21.15.33:0/821617656 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5a500a2890 0x7f5a500a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:37.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.474+0000 7f5a693fd640 1 --2- 172.21.15.33:0/821617656 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5a500a2890 0x7f5a500a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:37.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.474+0000 7f5a5a7fc640 1 -- 172.21.15.33:0/821617656 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7f5a540cf330 con 0x7f5a64101920 2024-03-19T02:52:37.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.474+0000 7f5a693fd640 1 --2- 172.21.15.33:0/821617656 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5a500a2890 0x7f5a500a4d50 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5a64101780 tx=0x7f5a60005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:37.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.478+0000 7f5a5a7fc640 1 -- 172.21.15.33:0/821617656 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5a5409bfa0 con 0x7f5a64101920 2024-03-19T02:52:37.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:37 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/597055499' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-19T02:52:37.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.662+0000 7f5a6b661640 1 -- 172.21.15.33:0/821617656 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5a641a02e0 con 0x7f5a64101920 2024-03-19T02:52:37.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.663+0000 7f5a5a7fc640 1 -- 172.21.15.33:0/821617656 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7f5a5409b8a0 con 0x7f5a64101920 2024-03-19T02:52:37.664 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 1 2024-03-19T02:52:37.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.665+0000 7f5a6b661640 1 -- 172.21.15.33:0/821617656 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5a500a2890 msgr2=0x7f5a500a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:37.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.665+0000 7f5a6b661640 1 --2- 172.21.15.33:0/821617656 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5a500a2890 0x7f5a500a4d50 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5a64101780 tx=0x7f5a60005f90 comp rx=0 tx=0).stop 2024-03-19T02:52:37.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.665+0000 7f5a6b661640 1 -- 172.21.15.33:0/821617656 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a64101920 msgr2=0x7f5a64197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:37.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.665+0000 7f5a6b661640 1 --2- 172.21.15.33:0/821617656 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a64101920 0x7f5a64197f60 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f5a5400e9c0 tx=0x7f5a5400ee80 comp rx=0 tx=0).stop 2024-03-19T02:52:37.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.665+0000 7f5a6b661640 1 -- 172.21.15.33:0/821617656 shutdown_connections 2024-03-19T02:52:37.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.665+0000 7f5a6b661640 1 --2- 172.21.15.33:0/821617656 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5a500a2890 0x7f5a500a4d50 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:37.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.665+0000 7f5a6b661640 1 --2- 172.21.15.33:0/821617656 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a641022e0 0x7f5a6419efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:37.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.665+0000 7f5a6b661640 1 --2- 172.21.15.33:0/821617656 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a64101920 0x7f5a64197f60 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:37.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.665+0000 7f5a6b661640 1 --2- 172.21.15.33:0/821617656 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a64100730 0x7f5a64197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:37.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.665+0000 7f5a6b661640 1 -- 172.21.15.33:0/821617656 >> 172.21.15.33:0/821617656 conn(0x7f5a640fbcc0 msgr2=0x7f5a64069df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:37.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.665+0000 7f5a6b661640 1 -- 172.21.15.33:0/821617656 shutdown_connections 2024-03-19T02:52:37.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.665+0000 7f5a6b661640 1 -- 172.21.15.33:0/821617656 wait complete. 2024-03-19T02:52:37.678 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T02:52:37.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.782+0000 7f456d177640 1 -- 172.21.15.33:0/2484583895 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4568102350 msgr2=0x7f4568104740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:37.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.782+0000 7f456d177640 1 --2- 172.21.15.33:0/2484583895 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4568102350 0x7f4568104740 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4560009900 tx=0x7f456002bc30 comp rx=0 tx=0).stop 2024-03-19T02:52:37.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.782+0000 7f456d177640 1 -- 172.21.15.33:0/2484583895 shutdown_connections 2024-03-19T02:52:37.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.782+0000 7f456d177640 1 --2- 172.21.15.33:0/2484583895 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45681075b0 0x7f45681099a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:37.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.782+0000 7f456d177640 1 --2- 172.21.15.33:0/2484583895 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4568104c80 0x7f4568107070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:37.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.782+0000 7f456d177640 1 --2- 172.21.15.33:0/2484583895 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4568102350 0x7f4568104740 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:37.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.782+0000 7f456d177640 1 -- 172.21.15.33:0/2484583895 >> 172.21.15.33:0/2484583895 conn(0x7f45680fbcc0 msgr2=0x7f45680fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:37.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.782+0000 7f456d177640 1 -- 172.21.15.33:0/2484583895 shutdown_connections 2024-03-19T02:52:37.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.782+0000 7f456d177640 1 -- 172.21.15.33:0/2484583895 wait complete. 2024-03-19T02:52:37.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.783+0000 7f456d177640 1 Processor -- start 2024-03-19T02:52:37.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.783+0000 7f456d177640 1 -- start start 2024-03-19T02:52:37.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.783+0000 7f456d177640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4568102350 0x7f4568199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:37.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.783+0000 7f456d177640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4568104c80 0x7f456819a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:37.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.783+0000 7f456d177640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45681075b0 0x7f45681a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:37.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.783+0000 7f456d177640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f456810d3c0 con 0x7f45681075b0 2024-03-19T02:52:37.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.783+0000 7f456d177640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f456810d240 con 0x7f4568104c80 2024-03-19T02:52:37.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.783+0000 7f456d177640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f456810d540 con 0x7f4568102350 2024-03-19T02:52:37.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.783+0000 7f456759e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45681075b0 0x7f45681a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:37.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.783+0000 7f456759e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45681075b0 0x7f45681a1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41460/0 (socket says 172.21.15.33:41460) 2024-03-19T02:52:37.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.783+0000 7f456759e640 1 -- 172.21.15.33:0/2652678405 learned_addr learned my addr 172.21.15.33:0/2652678405 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:37.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.783+0000 7f4566d9d640 1 --2- 172.21.15.33:0/2652678405 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4568102350 0x7f4568199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:37.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.783+0000 7f456659c640 1 --2- 172.21.15.33:0/2652678405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4568104c80 0x7f456819a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:37.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.784+0000 7f456759e640 1 -- 172.21.15.33:0/2652678405 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4568102350 msgr2=0x7f4568199cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:37.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.784+0000 7f456759e640 1 --2- 172.21.15.33:0/2652678405 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4568102350 0x7f4568199cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:37.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.784+0000 7f456759e640 1 -- 172.21.15.33:0/2652678405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4568104c80 msgr2=0x7f456819a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:37.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.784+0000 7f456759e640 1 --2- 172.21.15.33:0/2652678405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4568104c80 0x7f456819a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:37.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.784+0000 7f456759e640 1 -- 172.21.15.33:0/2652678405 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45681a1900 con 0x7f45681075b0 2024-03-19T02:52:37.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.784+0000 7f4566d9d640 1 --2- 172.21.15.33:0/2652678405 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4568102350 0x7f4568199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:37.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.784+0000 7f456659c640 1 --2- 172.21.15.33:0/2652678405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4568104c80 0x7f456819a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:37.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.784+0000 7f456759e640 1 --2- 172.21.15.33:0/2652678405 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45681075b0 0x7f45681a1260 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f455800eb10 tx=0x7f455800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:37.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.784+0000 7f455ffff640 1 -- 172.21.15.33:0/2652678405 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4558004b20 con 0x7f45681075b0 2024-03-19T02:52:37.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.784+0000 7f456d177640 1 -- 172.21.15.33:0/2652678405 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45681a1bf0 con 0x7f45681075b0 2024-03-19T02:52:37.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.784+0000 7f455ffff640 1 -- 172.21.15.33:0/2652678405 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4558004cc0 con 0x7f45681075b0 2024-03-19T02:52:37.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.784+0000 7f455ffff640 1 -- 172.21.15.33:0/2652678405 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4558010690 con 0x7f45681075b0 2024-03-19T02:52:37.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.784+0000 7f456d177640 1 -- 172.21.15.33:0/2652678405 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45681a2160 con 0x7f45681075b0 2024-03-19T02:52:37.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.785+0000 7f456d177640 1 -- 172.21.15.33:0/2652678405 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45681090a0 con 0x7f45681075b0 2024-03-19T02:52:37.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.786+0000 7f455ffff640 1 -- 172.21.15.33:0/2652678405 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f4558005020 con 0x7f45681075b0 2024-03-19T02:52:37.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.787+0000 7f455ffff640 1 --2- 172.21.15.33:0/2652678405 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f45540a2890 0x7f45540a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:37.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.787+0000 7f4566d9d640 1 --2- 172.21.15.33:0/2652678405 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f45540a2890 0x7f45540a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:37.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.787+0000 7f455ffff640 1 -- 172.21.15.33:0/2652678405 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7f45580cf600 con 0x7f45681075b0 2024-03-19T02:52:37.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.787+0000 7f4566d9d640 1 --2- 172.21.15.33:0/2652678405 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f45540a2890 0x7f45540a4d50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4560000c00 tx=0x7f456002c7e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:37.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.791+0000 7f455ffff640 1 -- 172.21.15.33:0/2652678405 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f455809c1e0 con 0x7f45681075b0 2024-03-19T02:52:37.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.974+0000 7f456d177640 1 -- 172.21.15.33:0/2652678405 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f456802d3a0 con 0x7f45681075b0 2024-03-19T02:52:37.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.974+0000 7f455ffff640 1 -- 172.21.15.33:0/2652678405 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) ==== 104+0+11 (secure 0 0 0) 0x7f455809bae0 con 0x7f45681075b0 2024-03-19T02:52:37.975 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 32 2024-03-19T02:52:37.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.976+0000 7f456d177640 1 -- 172.21.15.33:0/2652678405 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f45540a2890 msgr2=0x7f45540a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:37.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.976+0000 7f456d177640 1 --2- 172.21.15.33:0/2652678405 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f45540a2890 0x7f45540a4d50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4560000c00 tx=0x7f456002c7e0 comp rx=0 tx=0).stop 2024-03-19T02:52:37.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.976+0000 7f456d177640 1 -- 172.21.15.33:0/2652678405 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45681075b0 msgr2=0x7f45681a1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:37.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.976+0000 7f456d177640 1 --2- 172.21.15.33:0/2652678405 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45681075b0 0x7f45681a1260 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f455800eb10 tx=0x7f455800c2d0 comp rx=0 tx=0).stop 2024-03-19T02:52:37.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.976+0000 7f456d177640 1 -- 172.21.15.33:0/2652678405 shutdown_connections 2024-03-19T02:52:37.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.976+0000 7f456d177640 1 --2- 172.21.15.33:0/2652678405 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f45540a2890 0x7f45540a4d50 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:37.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.976+0000 7f456d177640 1 --2- 172.21.15.33:0/2652678405 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45681075b0 0x7f45681a1260 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:37.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.976+0000 7f456d177640 1 --2- 172.21.15.33:0/2652678405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4568104c80 0x7f456819a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:37.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.976+0000 7f456d177640 1 --2- 172.21.15.33:0/2652678405 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4568102350 0x7f4568199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:37.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.976+0000 7f456d177640 1 -- 172.21.15.33:0/2652678405 >> 172.21.15.33:0/2652678405 conn(0x7f45680fbcc0 msgr2=0x7f4568101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:37.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.976+0000 7f456d177640 1 -- 172.21.15.33:0/2652678405 shutdown_connections 2024-03-19T02:52:37.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:37.977+0000 7f456d177640 1 -- 172.21.15.33:0/2652678405 wait complete. 2024-03-19T02:52:37.989 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T02:52:38.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.093+0000 7ffa7529a640 1 -- 172.21.15.33:0/4288637011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa701033e0 msgr2=0x7ffa70109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:38.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.093+0000 7ffa7529a640 1 --2- 172.21.15.33:0/4288637011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa701033e0 0x7ffa70109c70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ffa60009ca0 tx=0x7ffa6002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:52:38.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.093+0000 7ffa7529a640 1 -- 172.21.15.33:0/4288637011 shutdown_connections 2024-03-19T02:52:38.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.093+0000 7ffa7529a640 1 --2- 172.21.15.33:0/4288637011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa701033e0 0x7ffa70109c70 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:38.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.093+0000 7ffa7529a640 1 --2- 172.21.15.33:0/4288637011 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa70102a20 0x7ffa70102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:38.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.093+0000 7ffa7529a640 1 --2- 172.21.15.33:0/4288637011 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa70101830 0x7ffa70101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:38.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.093+0000 7ffa7529a640 1 -- 172.21.15.33:0/4288637011 >> 172.21.15.33:0/4288637011 conn(0x7ffa700fcdc0 msgr2=0x7ffa700ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:38.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.094+0000 7ffa7529a640 1 -- 172.21.15.33:0/4288637011 shutdown_connections 2024-03-19T02:52:38.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.094+0000 7ffa7529a640 1 -- 172.21.15.33:0/4288637011 wait complete. 2024-03-19T02:52:38.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.094+0000 7ffa7529a640 1 Processor -- start 2024-03-19T02:52:38.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.094+0000 7ffa7529a640 1 -- start start 2024-03-19T02:52:38.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.094+0000 7ffa7529a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa70101830 0x7ffa70197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:38.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.094+0000 7ffa7529a640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa70102a20 0x7ffa70197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:38.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.094+0000 7ffa7529a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa701033e0 0x7ffa7019efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:38.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.094+0000 7ffa7529a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7010d340 con 0x7ffa70101830 2024-03-19T02:52:38.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.094+0000 7ffa7529a640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7010d1c0 con 0x7ffa701033e0 2024-03-19T02:52:38.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.094+0000 7ffa7529a640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7010d4c0 con 0x7ffa70102a20 2024-03-19T02:52:38.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.095+0000 7ffa6effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa70101830 0x7ffa70197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:38.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.095+0000 7ffa6effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa70101830 0x7ffa70197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41470/0 (socket says 172.21.15.33:41470) 2024-03-19T02:52:38.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.095+0000 7ffa6f7fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa701033e0 0x7ffa7019efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:38.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.095+0000 7ffa6e7fc640 1 --2- 172.21.15.33:0/2544686530 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa70102a20 0x7ffa70197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:38.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.095+0000 7ffa6effd640 1 -- 172.21.15.33:0/2544686530 learned_addr learned my addr 172.21.15.33:0/2544686530 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:38.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.095+0000 7ffa6effd640 1 -- 172.21.15.33:0/2544686530 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa70102a20 msgr2=0x7ffa70197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:38.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.095+0000 7ffa6effd640 1 --2- 172.21.15.33:0/2544686530 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa70102a20 0x7ffa70197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:38.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.095+0000 7ffa6effd640 1 -- 172.21.15.33:0/2544686530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa701033e0 msgr2=0x7ffa7019efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:38.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.095+0000 7ffa6effd640 1 --2- 172.21.15.33:0/2544686530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa701033e0 0x7ffa7019efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:38.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.095+0000 7ffa6effd640 1 -- 172.21.15.33:0/2544686530 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa7019f630 con 0x7ffa70101830 2024-03-19T02:52:38.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.095+0000 7ffa6e7fc640 1 --2- 172.21.15.33:0/2544686530 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa70102a20 0x7ffa70197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:38.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.095+0000 7ffa6f7fe640 1 --2- 172.21.15.33:0/2544686530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa701033e0 0x7ffa7019efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:38.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.095+0000 7ffa6effd640 1 --2- 172.21.15.33:0/2544686530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa70101830 0x7ffa70197a10 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ffa6800e9c0 tx=0x7ffa6800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:38.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.095+0000 7ffa67fff640 1 -- 172.21.15.33:0/2544686530 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa6800cde0 con 0x7ffa70101830 2024-03-19T02:52:38.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.095+0000 7ffa67fff640 1 -- 172.21.15.33:0/2544686530 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffa680108e0 con 0x7ffa70101830 2024-03-19T02:52:38.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.095+0000 7ffa7529a640 1 -- 172.21.15.33:0/2544686530 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa7019f920 con 0x7ffa70101830 2024-03-19T02:52:38.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.095+0000 7ffa67fff640 1 -- 172.21.15.33:0/2544686530 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa68018990 con 0x7ffa70101830 2024-03-19T02:52:38.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.095+0000 7ffa7529a640 1 -- 172.21.15.33:0/2544686530 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa7019fe70 con 0x7ffa70101830 2024-03-19T02:52:38.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.096+0000 7ffa7529a640 1 -- 172.21.15.33:0/2544686530 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa70068a60 con 0x7ffa70101830 2024-03-19T02:52:38.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.097+0000 7ffa67fff640 1 -- 172.21.15.33:0/2544686530 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7ffa6801f020 con 0x7ffa70101830 2024-03-19T02:52:38.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.098+0000 7ffa67fff640 1 --2- 172.21.15.33:0/2544686530 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ffa5c0a2890 0x7ffa5c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:38.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.098+0000 7ffa6e7fc640 1 --2- 172.21.15.33:0/2544686530 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ffa5c0a2890 0x7ffa5c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:38.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.098+0000 7ffa67fff640 1 -- 172.21.15.33:0/2544686530 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7ffa680cf100 con 0x7ffa70101830 2024-03-19T02:52:38.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.098+0000 7ffa6e7fc640 1 --2- 172.21.15.33:0/2544686530 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ffa5c0a2890 0x7ffa5c0a4d50 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ffa580095f0 tx=0x7ffa58005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:38.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.102+0000 7ffa67fff640 1 -- 172.21.15.33:0/2544686530 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ffa6809bce0 con 0x7ffa70101830 2024-03-19T02:52:38.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:38 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3637724067' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:52:38.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:38 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/821617656' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:52:38.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:38 smithi033 ceph-mon[31973]: pgmap v354: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:52:38.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:38 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2652678405' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:52:38.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.286+0000 7ffa7529a640 1 -- 172.21.15.33:0/2544686530 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ffa70108c20 con 0x7ffa70101830 2024-03-19T02:52:38.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.287+0000 7ffa67fff640 1 -- 172.21.15.33:0/2544686530 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) ==== 100+0+11 (secure 0 0 0) 0x7ffa70108c20 con 0x7ffa70101830 2024-03-19T02:52:38.288 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 64 2024-03-19T02:52:38.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.288+0000 7ffa7529a640 1 -- 172.21.15.33:0/2544686530 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ffa5c0a2890 msgr2=0x7ffa5c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:38.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.288+0000 7ffa7529a640 1 --2- 172.21.15.33:0/2544686530 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ffa5c0a2890 0x7ffa5c0a4d50 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ffa580095f0 tx=0x7ffa58005f90 comp rx=0 tx=0).stop 2024-03-19T02:52:38.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.288+0000 7ffa7529a640 1 -- 172.21.15.33:0/2544686530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa70101830 msgr2=0x7ffa70197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:38.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.288+0000 7ffa7529a640 1 --2- 172.21.15.33:0/2544686530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa70101830 0x7ffa70197a10 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ffa6800e9c0 tx=0x7ffa6800ee80 comp rx=0 tx=0).stop 2024-03-19T02:52:38.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.289+0000 7ffa7529a640 1 -- 172.21.15.33:0/2544686530 shutdown_connections 2024-03-19T02:52:38.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.289+0000 7ffa7529a640 1 --2- 172.21.15.33:0/2544686530 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ffa5c0a2890 0x7ffa5c0a4d50 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:38.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.289+0000 7ffa7529a640 1 --2- 172.21.15.33:0/2544686530 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa70102a20 0x7ffa70197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:38.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.289+0000 7ffa7529a640 1 --2- 172.21.15.33:0/2544686530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa701033e0 0x7ffa7019efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:38.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.289+0000 7ffa7529a640 1 --2- 172.21.15.33:0/2544686530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa70101830 0x7ffa70197a10 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:38.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.289+0000 7ffa7529a640 1 -- 172.21.15.33:0/2544686530 >> 172.21.15.33:0/2544686530 conn(0x7ffa700fcdc0 msgr2=0x7ffa70105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:38.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.289+0000 7ffa7529a640 1 -- 172.21.15.33:0/2544686530 shutdown_connections 2024-03-19T02:52:38.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.289+0000 7ffa7529a640 1 -- 172.21.15.33:0/2544686530 wait complete. 2024-03-19T02:52:38.301 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T02:52:38.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.406+0000 7f6e7f60b640 1 -- 172.21.15.33:0/142131620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e78100740 msgr2=0x7f6e78100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:38.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.406+0000 7f6e7f60b640 1 --2- 172.21.15.33:0/142131620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e78100740 0x7f6e78100b60 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f6e74009900 tx=0x7f6e7402bc30 comp rx=0 tx=0).stop 2024-03-19T02:52:38.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.406+0000 7f6e7f60b640 1 -- 172.21.15.33:0/142131620 shutdown_connections 2024-03-19T02:52:38.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.406+0000 7f6e7f60b640 1 --2- 172.21.15.33:0/142131620 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e781022f0 0x7f6e78108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:38.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.406+0000 7f6e7f60b640 1 --2- 172.21.15.33:0/142131620 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e78101930 0x7f6e78101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:38.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.406+0000 7f6e7f60b640 1 --2- 172.21.15.33:0/142131620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e78100740 0x7f6e78100b60 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:38.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.406+0000 7f6e7f60b640 1 -- 172.21.15.33:0/142131620 >> 172.21.15.33:0/142131620 conn(0x7f6e780fbcd0 msgr2=0x7f6e780fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:38.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.406+0000 7f6e7f60b640 1 -- 172.21.15.33:0/142131620 shutdown_connections 2024-03-19T02:52:38.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.406+0000 7f6e7f60b640 1 -- 172.21.15.33:0/142131620 wait complete. 2024-03-19T02:52:38.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.406+0000 7f6e7f60b640 1 Processor -- start 2024-03-19T02:52:38.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.406+0000 7f6e7f60b640 1 -- start start 2024-03-19T02:52:38.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.407+0000 7f6e7f60b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e78100740 0x7f6e78197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:38.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.407+0000 7f6e7f60b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e78101930 0x7f6e78198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:38.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.407+0000 7f6e7f60b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e781022f0 0x7f6e7819f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:38.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.407+0000 7f6e7f60b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6e7810c270 con 0x7f6e78100740 2024-03-19T02:52:38.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.407+0000 7f6e7f60b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6e7810c0f0 con 0x7f6e781022f0 2024-03-19T02:52:38.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.407+0000 7f6e7f60b640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6e7810c3f0 con 0x7f6e78101930 2024-03-19T02:52:38.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.407+0000 7f6e7d3a7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e78100740 0x7f6e78197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:38.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.407+0000 7f6e7d3a7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e78100740 0x7f6e78197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41482/0 (socket says 172.21.15.33:41482) 2024-03-19T02:52:38.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.407+0000 7f6e7d3a7640 1 -- 172.21.15.33:0/504583019 learned_addr learned my addr 172.21.15.33:0/504583019 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:38.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.407+0000 7f6e7cba6640 1 --2- 172.21.15.33:0/504583019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e78101930 0x7f6e78198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:38.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.407+0000 7f6e7dba8640 1 --2- 172.21.15.33:0/504583019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e781022f0 0x7f6e7819f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:38.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.407+0000 7f6e7d3a7640 1 -- 172.21.15.33:0/504583019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e78101930 msgr2=0x7f6e78198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:38.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.407+0000 7f6e7d3a7640 1 --2- 172.21.15.33:0/504583019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e78101930 0x7f6e78198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:38.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.407+0000 7f6e7d3a7640 1 -- 172.21.15.33:0/504583019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e781022f0 msgr2=0x7f6e7819f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:38.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.407+0000 7f6e7d3a7640 1 --2- 172.21.15.33:0/504583019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e781022f0 0x7f6e7819f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:38.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.407+0000 7f6e7d3a7640 1 -- 172.21.15.33:0/504583019 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e7819f6d0 con 0x7f6e78100740 2024-03-19T02:52:38.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.407+0000 7f6e7cba6640 1 --2- 172.21.15.33:0/504583019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e78101930 0x7f6e78198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:52:38.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.407+0000 7f6e7d3a7640 1 --2- 172.21.15.33:0/504583019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e78100740 0x7f6e78197b50 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6e74000c00 tx=0x7f6e74005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:38.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.407+0000 7f6e6e7fc640 1 -- 172.21.15.33:0/504583019 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e74043020 con 0x7f6e78100740 2024-03-19T02:52:38.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.407+0000 7f6e7f60b640 1 -- 172.21.15.33:0/504583019 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e7819f960 con 0x7f6e78100740 2024-03-19T02:52:38.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.407+0000 7f6e6e7fc640 1 -- 172.21.15.33:0/504583019 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6e74004350 con 0x7f6e78100740 2024-03-19T02:52:38.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.407+0000 7f6e6e7fc640 1 -- 172.21.15.33:0/504583019 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e74003b80 con 0x7f6e78100740 2024-03-19T02:52:38.411 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.407+0000 7f6e7f60b640 1 -- 172.21.15.33:0/504583019 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e7819ff00 con 0x7f6e78100740 2024-03-19T02:52:38.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.408+0000 7f6e7f60b640 1 -- 172.21.15.33:0/504583019 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e78068a60 con 0x7f6e78100740 2024-03-19T02:52:38.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.409+0000 7f6e6e7fc640 1 -- 172.21.15.33:0/504583019 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f6e74046020 con 0x7f6e78100740 2024-03-19T02:52:38.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.410+0000 7f6e6e7fc640 1 --2- 172.21.15.33:0/504583019 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f6e640a2890 0x7f6e640a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:38.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.410+0000 7f6e6e7fc640 1 -- 172.21.15.33:0/504583019 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7f6e740f1010 con 0x7f6e78100740 2024-03-19T02:52:38.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.410+0000 7f6e7cba6640 1 --2- 172.21.15.33:0/504583019 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f6e640a2890 0x7f6e640a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:38.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.410+0000 7f6e7cba6640 1 --2- 172.21.15.33:0/504583019 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f6e640a2890 0x7f6e640a4d50 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6e78198fc0 tx=0x7f6e68005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:38.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.414+0000 7f6e6e7fc640 1 -- 172.21.15.33:0/504583019 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6e7404d020 con 0x7f6e78100740 2024-03-19T02:52:38.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:38 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3637724067' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:52:38.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:38 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/821617656' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:52:38.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:38 smithi096 ceph-mon[35946]: pgmap v354: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:52:38.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:38 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2652678405' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:52:38.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:38 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3637724067' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:52:38.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:38 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/821617656' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:52:38.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:38 smithi155 ceph-mon[35804]: pgmap v354: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:52:38.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:38 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2652678405' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:52:38.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.597+0000 7f6e7f60b640 1 -- 172.21.15.33:0/504583019 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6e781a02b0 con 0x7f6e78100740 2024-03-19T02:52:38.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.597+0000 7f6e6e7fc640 1 -- 172.21.15.33:0/504583019 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) ==== 103+0+11 (secure 0 0 0) 0x7f6e740bdd00 con 0x7f6e78100740 2024-03-19T02:52:38.598 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 32 2024-03-19T02:52:38.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.599+0000 7f6e7f60b640 1 -- 172.21.15.33:0/504583019 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f6e640a2890 msgr2=0x7f6e640a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:38.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.599+0000 7f6e7f60b640 1 --2- 172.21.15.33:0/504583019 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f6e640a2890 0x7f6e640a4d50 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6e78198fc0 tx=0x7f6e68005f90 comp rx=0 tx=0).stop 2024-03-19T02:52:38.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.599+0000 7f6e7f60b640 1 -- 172.21.15.33:0/504583019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e78100740 msgr2=0x7f6e78197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:38.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.599+0000 7f6e7f60b640 1 --2- 172.21.15.33:0/504583019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e78100740 0x7f6e78197b50 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6e74000c00 tx=0x7f6e74005dc0 comp rx=0 tx=0).stop 2024-03-19T02:52:38.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.599+0000 7f6e7f60b640 1 -- 172.21.15.33:0/504583019 shutdown_connections 2024-03-19T02:52:38.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.599+0000 7f6e7f60b640 1 --2- 172.21.15.33:0/504583019 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f6e640a2890 0x7f6e640a4d50 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:38.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.599+0000 7f6e7f60b640 1 --2- 172.21.15.33:0/504583019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e78101930 0x7f6e78198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:38.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.599+0000 7f6e7f60b640 1 --2- 172.21.15.33:0/504583019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e781022f0 0x7f6e7819f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:38.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.599+0000 7f6e7f60b640 1 --2- 172.21.15.33:0/504583019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e78100740 0x7f6e78197b50 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:38.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.599+0000 7f6e7f60b640 1 -- 172.21.15.33:0/504583019 >> 172.21.15.33:0/504583019 conn(0x7f6e780fbcd0 msgr2=0x7f6e78069df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:38.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.599+0000 7f6e7f60b640 1 -- 172.21.15.33:0/504583019 shutdown_connections 2024-03-19T02:52:38.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.599+0000 7f6e7f60b640 1 -- 172.21.15.33:0/504583019 wait complete. 2024-03-19T02:52:38.611 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:52:38.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.713+0000 7fe77d9d3640 1 -- 172.21.15.33:0/3219959052 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe778101840 msgr2=0x7fe778101c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:38.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.713+0000 7fe77d9d3640 1 --2- 172.21.15.33:0/3219959052 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe778101840 0x7fe778101c60 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe768009900 tx=0x7fe76802bc30 comp rx=0 tx=0).stop 2024-03-19T02:52:38.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.713+0000 7fe77d9d3640 1 -- 172.21.15.33:0/3219959052 shutdown_connections 2024-03-19T02:52:38.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.713+0000 7fe77d9d3640 1 --2- 172.21.15.33:0/3219959052 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe778102a30 0x7fe778102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:38.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.713+0000 7fe77d9d3640 1 --2- 172.21.15.33:0/3219959052 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7781033f0 0x7fe778109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:38.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.713+0000 7fe77d9d3640 1 --2- 172.21.15.33:0/3219959052 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe778101840 0x7fe778101c60 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:38.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.713+0000 7fe77d9d3640 1 -- 172.21.15.33:0/3219959052 >> 172.21.15.33:0/3219959052 conn(0x7fe7780fcdd0 msgr2=0x7fe7780ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:38.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.713+0000 7fe77d9d3640 1 -- 172.21.15.33:0/3219959052 shutdown_connections 2024-03-19T02:52:38.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.713+0000 7fe77d9d3640 1 -- 172.21.15.33:0/3219959052 wait complete. 2024-03-19T02:52:38.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.714+0000 7fe77d9d3640 1 Processor -- start 2024-03-19T02:52:38.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.714+0000 7fe77d9d3640 1 -- start start 2024-03-19T02:52:38.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.714+0000 7fe77d9d3640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe778101840 0x7fe778197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:38.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.714+0000 7fe77d9d3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe778102a30 0x7fe778197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:38.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.714+0000 7fe77d9d3640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7781033f0 0x7fe77819f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:38.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.714+0000 7fe77d9d3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe77810d360 con 0x7fe778102a30 2024-03-19T02:52:38.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.714+0000 7fe77d9d3640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe77810d1e0 con 0x7fe7781033f0 2024-03-19T02:52:38.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.714+0000 7fe77d9d3640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe77810d4e0 con 0x7fe778101840 2024-03-19T02:52:38.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.714+0000 7fe76ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe778102a30 0x7fe778197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:38.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.714+0000 7fe76ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe778102a30 0x7fe778197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41498/0 (socket says 172.21.15.33:41498) 2024-03-19T02:52:38.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.714+0000 7fe76ffff640 1 -- 172.21.15.33:0/408175535 learned_addr learned my addr 172.21.15.33:0/408175535 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:38.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.714+0000 7fe776ffd640 1 --2- 172.21.15.33:0/408175535 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe778101840 0x7fe778197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:38.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.714+0000 7fe7777fe640 1 --2- 172.21.15.33:0/408175535 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7781033f0 0x7fe77819f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:38.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.714+0000 7fe76ffff640 1 -- 172.21.15.33:0/408175535 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe778101840 msgr2=0x7fe778197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:38.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.714+0000 7fe76ffff640 1 --2- 172.21.15.33:0/408175535 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe778101840 0x7fe778197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:38.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.714+0000 7fe76ffff640 1 -- 172.21.15.33:0/408175535 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7781033f0 msgr2=0x7fe77819f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:38.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.714+0000 7fe76ffff640 1 --2- 172.21.15.33:0/408175535 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7781033f0 0x7fe77819f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:38.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.714+0000 7fe76ffff640 1 -- 172.21.15.33:0/408175535 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe77819f6b0 con 0x7fe778102a30 2024-03-19T02:52:38.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.714+0000 7fe76ffff640 1 --2- 172.21.15.33:0/408175535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe778102a30 0x7fe778197fd0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe76000e9f0 tx=0x7fe76000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:38.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.715+0000 7fe77c9d1640 1 -- 172.21.15.33:0/408175535 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe76000cd80 con 0x7fe778102a30 2024-03-19T02:52:38.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.715+0000 7fe77c9d1640 1 -- 172.21.15.33:0/408175535 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe7600054d0 con 0x7fe778102a30 2024-03-19T02:52:38.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.715+0000 7fe77c9d1640 1 -- 172.21.15.33:0/408175535 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe760010490 con 0x7fe778102a30 2024-03-19T02:52:38.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.715+0000 7fe77d9d3640 1 -- 172.21.15.33:0/408175535 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe77819f9a0 con 0x7fe778102a30 2024-03-19T02:52:38.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.715+0000 7fe77d9d3640 1 -- 172.21.15.33:0/408175535 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe77819fea0 con 0x7fe778102a30 2024-03-19T02:52:38.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.716+0000 7fe77d9d3640 1 -- 172.21.15.33:0/408175535 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe778108c30 con 0x7fe778102a30 2024-03-19T02:52:38.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.717+0000 7fe77c9d1640 1 -- 172.21.15.33:0/408175535 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fe760005000 con 0x7fe778102a30 2024-03-19T02:52:38.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.717+0000 7fe77c9d1640 1 --2- 172.21.15.33:0/408175535 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe7640a2890 0x7fe7640a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:38.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.717+0000 7fe776ffd640 1 --2- 172.21.15.33:0/408175535 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe7640a2890 0x7fe7640a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:38.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.717+0000 7fe77c9d1640 1 -- 172.21.15.33:0/408175535 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7fe7600cf510 con 0x7fe778102a30 2024-03-19T02:52:38.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.718+0000 7fe776ffd640 1 --2- 172.21.15.33:0/408175535 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe7640a2890 0x7fe7640a4d50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe768005ab0 tx=0x7fe768005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:38.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.721+0000 7fe77c9d1640 1 -- 172.21.15.33:0/408175535 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe760053740 con 0x7fe778102a30 2024-03-19T02:52:38.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.946+0000 7fe77d9d3640 1 -- 172.21.15.33:0/408175535 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe77802d440 con 0x7fe778102a30 2024-03-19T02:52:38.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.947+0000 7fe77c9d1640 1 -- 172.21.15.33:0/408175535 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11867 (secure 0 0 0) 0x7fe7600052f0 con 0x7fe778102a30 2024-03-19T02:52:38.948 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:52:38.948 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:50:52.201798+0000","modified":"2024-03-19T02:51:19.076317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24488,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/3731232169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":3731232169},{"type":"v1","addr":"172.21.15.96:6839","nonce":3731232169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24457,24509,24488]},"id":1}]} 2024-03-19T02:52:38.948 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 24 2024-03-19T02:52:38.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.949+0000 7fe77d9d3640 1 -- 172.21.15.33:0/408175535 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe7640a2890 msgr2=0x7fe7640a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:38.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.949+0000 7fe77d9d3640 1 --2- 172.21.15.33:0/408175535 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe7640a2890 0x7fe7640a4d50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe768005ab0 tx=0x7fe768005fd0 comp rx=0 tx=0).stop 2024-03-19T02:52:38.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.949+0000 7fe77d9d3640 1 -- 172.21.15.33:0/408175535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe778102a30 msgr2=0x7fe778197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:38.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.949+0000 7fe77d9d3640 1 --2- 172.21.15.33:0/408175535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe778102a30 0x7fe778197fd0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe76000e9f0 tx=0x7fe76000edf0 comp rx=0 tx=0).stop 2024-03-19T02:52:38.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.949+0000 7fe77d9d3640 1 -- 172.21.15.33:0/408175535 shutdown_connections 2024-03-19T02:52:38.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.949+0000 7fe77d9d3640 1 --2- 172.21.15.33:0/408175535 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe7640a2890 0x7fe7640a4d50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:38.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.949+0000 7fe77d9d3640 1 --2- 172.21.15.33:0/408175535 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7781033f0 0x7fe77819f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:38.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.949+0000 7fe77d9d3640 1 --2- 172.21.15.33:0/408175535 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe778102a30 0x7fe778197fd0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:38.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.949+0000 7fe77d9d3640 1 --2- 172.21.15.33:0/408175535 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe778101840 0x7fe778197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:38.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.949+0000 7fe77d9d3640 1 -- 172.21.15.33:0/408175535 >> 172.21.15.33:0/408175535 conn(0x7fe7780fcdd0 msgr2=0x7fe778105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:38.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.949+0000 7fe77d9d3640 1 -- 172.21.15.33:0/408175535 shutdown_connections 2024-03-19T02:52:38.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:38.949+0000 7fe77d9d3640 1 -- 172.21.15.33:0/408175535 wait complete. 2024-03-19T02:52:38.961 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:52:39.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.063+0000 7f10cc952640 1 -- 172.21.15.33:0/142048623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f10c8102320 msgr2=0x7f10c8104710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:39.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.063+0000 7f10cc952640 1 --2- 172.21.15.33:0/142048623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f10c8102320 0x7f10c8104710 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f10b8009900 tx=0x7f10b802bc30 comp rx=0 tx=0).stop 2024-03-19T02:52:39.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.063+0000 7f10cc952640 1 -- 172.21.15.33:0/142048623 shutdown_connections 2024-03-19T02:52:39.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.063+0000 7f10cc952640 1 --2- 172.21.15.33:0/142048623 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10c8104c50 0x7f10c8107040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:39.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.063+0000 7f10cc952640 1 --2- 172.21.15.33:0/142048623 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10c8107580 0x7f10c8109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:39.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.063+0000 7f10cc952640 1 --2- 172.21.15.33:0/142048623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f10c8102320 0x7f10c8104710 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:39.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.063+0000 7f10cc952640 1 -- 172.21.15.33:0/142048623 >> 172.21.15.33:0/142048623 conn(0x7f10c80fbcb0 msgr2=0x7f10c80fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:39.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.063+0000 7f10cc952640 1 -- 172.21.15.33:0/142048623 shutdown_connections 2024-03-19T02:52:39.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.063+0000 7f10cc952640 1 -- 172.21.15.33:0/142048623 wait complete. 2024-03-19T02:52:39.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.064+0000 7f10cc952640 1 Processor -- start 2024-03-19T02:52:39.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.064+0000 7f10cc952640 1 -- start start 2024-03-19T02:52:39.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.064+0000 7f10cc952640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10c8102320 0x7f10c8197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:39.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.064+0000 7f10cc952640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f10c8104c50 0x7f10c8198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:39.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.064+0000 7f10cc952640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10c8107580 0x7f10c819f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:39.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.064+0000 7f10cc952640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f10c810d390 con 0x7f10c8104c50 2024-03-19T02:52:39.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.064+0000 7f10cc952640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f10c810d210 con 0x7f10c8107580 2024-03-19T02:52:39.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.064+0000 7f10cc952640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f10c810d510 con 0x7f10c8102320 2024-03-19T02:52:39.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.064+0000 7f10c5d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f10c8104c50 0x7f10c8198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:39.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.064+0000 7f10c5d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f10c8104c50 0x7f10c8198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41526/0 (socket says 172.21.15.33:41526) 2024-03-19T02:52:39.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.064+0000 7f10c5d9b640 1 -- 172.21.15.33:0/4125628526 learned_addr learned my addr 172.21.15.33:0/4125628526 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:39.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.064+0000 7f10c6d9d640 1 --2- 172.21.15.33:0/4125628526 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10c8107580 0x7f10c819f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:39.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.064+0000 7f10c659c640 1 --2- 172.21.15.33:0/4125628526 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10c8102320 0x7f10c8197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:39.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.064+0000 7f10c5d9b640 1 -- 172.21.15.33:0/4125628526 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10c8102320 msgr2=0x7f10c8197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:39.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.064+0000 7f10c5d9b640 1 --2- 172.21.15.33:0/4125628526 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10c8102320 0x7f10c8197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:39.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.064+0000 7f10c5d9b640 1 -- 172.21.15.33:0/4125628526 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10c8107580 msgr2=0x7f10c819f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:39.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.064+0000 7f10c5d9b640 1 --2- 172.21.15.33:0/4125628526 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10c8107580 0x7f10c819f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:39.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.064+0000 7f10c5d9b640 1 -- 172.21.15.33:0/4125628526 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10c819f6e0 con 0x7f10c8104c50 2024-03-19T02:52:39.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.064+0000 7f10c6d9d640 1 --2- 172.21.15.33:0/4125628526 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10c8107580 0x7f10c819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:39.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.064+0000 7f10c5d9b640 1 --2- 172.21.15.33:0/4125628526 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f10c8104c50 0x7f10c8198000 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f10bc00e9f0 tx=0x7f10bc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:39.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.065+0000 7f10b77fe640 1 -- 172.21.15.33:0/4125628526 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10bc00ccc0 con 0x7f10c8104c50 2024-03-19T02:52:39.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.065+0000 7f10b77fe640 1 -- 172.21.15.33:0/4125628526 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f10bc0054d0 con 0x7f10c8104c50 2024-03-19T02:52:39.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.065+0000 7f10cc952640 1 -- 172.21.15.33:0/4125628526 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10c819f9d0 con 0x7f10c8104c50 2024-03-19T02:52:39.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.065+0000 7f10b77fe640 1 -- 172.21.15.33:0/4125628526 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10bc010430 con 0x7f10c8104c50 2024-03-19T02:52:39.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.065+0000 7f10cc952640 1 -- 172.21.15.33:0/4125628526 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10c819fed0 con 0x7f10c8104c50 2024-03-19T02:52:39.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.066+0000 7f10cc952640 1 -- 172.21.15.33:0/4125628526 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10c80fd7a0 con 0x7f10c8104c50 2024-03-19T02:52:39.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.067+0000 7f10b77fe640 1 -- 172.21.15.33:0/4125628526 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f10bc005000 con 0x7f10c8104c50 2024-03-19T02:52:39.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.067+0000 7f10b77fe640 1 --2- 172.21.15.33:0/4125628526 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f10b00a2890 0x7f10b00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:39.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.068+0000 7f10c659c640 1 --2- 172.21.15.33:0/4125628526 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f10b00a2890 0x7f10b00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:39.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.068+0000 7f10b77fe640 1 -- 172.21.15.33:0/4125628526 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7f10bc0cf560 con 0x7f10c8104c50 2024-03-19T02:52:39.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.068+0000 7f10c659c640 1 --2- 172.21.15.33:0/4125628526 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f10b00a2890 0x7f10b00a4d50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f10b8009230 tx=0x7f10b8005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:39.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.071+0000 7f10b77fe640 1 -- 172.21.15.33:0/4125628526 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f10bc053740 con 0x7f10c8104c50 2024-03-19T02:52:39.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.255+0000 7f10cc952640 1 -- 172.21.15.33:0/4125628526 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f10c802d440 con 0x7f10c8104c50 2024-03-19T02:52:39.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.257+0000 7f10b77fe640 1 -- 172.21.15.33:0/4125628526 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+24570 (secure 0 0 0) 0x7f10bc0052f0 con 0x7f10c8104c50 2024-03-19T02:52:39.258 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:52:39.258 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":95,"fsid":"2b7a8938-e59a-11ee-95c9-87774f69a715","created":"2024-03-19T02:42:37.912179+0000","modified":"2024-03-19T02:51:40.709913+0000","last_up_change":"2024-03-19T02:50:01.539392+0000","last_in_change":"2024-03-19T02:49:40.534415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:46:09.963737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:50:50.069641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:50:51.381064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:50:55.476667+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"090a89ab-6b7b-4317-8823-178d4212b2f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6803","nonce":1633124995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6805","nonce":1633124995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6809","nonce":1633124995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6807","nonce":1633124995}]},"public_addr":"172.21.15.33:6803/1633124995","cluster_addr":"172.21.15.33:6805/1633124995","heartbeat_back_addr":"172.21.15.33:6809/1633124995","heartbeat_front_addr":"172.21.15.33:6807/1633124995","state":["exists","up"]},{"osd":1,"uuid":"f50eec55-1956-4486-8593-f2c8fa610e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6801","nonce":340144398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6803","nonce":340144398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6807","nonce":340144398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6805","nonce":340144398}]},"public_addr":"172.21.15.96:6801/340144398","cluster_addr":"172.21.15.96:6803/340144398","heartbeat_back_addr":"172.21.15.96:6807/340144398","heartbeat_front_addr":"172.21.15.96:6805/340144398","state":["exists","up"]},{"osd":2,"uuid":"893a2d62-80d9-4e5e-bc6a-a1c8c681d652","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6801","nonce":3388743876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6803","nonce":3388743876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6807","nonce":3388743876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6805","nonce":3388743876}]},"public_addr":"172.21.15.155:6801/3388743876","cluster_addr":"172.21.15.155:6803/3388743876","heartbeat_back_addr":"172.21.15.155:6807/3388743876","heartbeat_front_addr":"172.21.15.155:6805/3388743876","state":["exists","up"]},{"osd":3,"uuid":"b07e0482-4281-4d32-9dfe-9995d7246175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6811","nonce":677007171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6813","nonce":677007171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6817","nonce":677007171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6815","nonce":677007171}]},"public_addr":"172.21.15.33:6811/677007171","cluster_addr":"172.21.15.33:6813/677007171","heartbeat_back_addr":"172.21.15.33:6817/677007171","heartbeat_front_addr":"172.21.15.33:6815/677007171","state":["exists","up"]},{"osd":4,"uuid":"a988384b-af6b-47c6-8c07-5c9673acee45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6809","nonce":3106158554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6811","nonce":3106158554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6815","nonce":3106158554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6813","nonce":3106158554}]},"public_addr":"172.21.15.96:6809/3106158554","cluster_addr":"172.21.15.96:6811/3106158554","heartbeat_back_addr":"172.21.15.96:6815/3106158554","heartbeat_front_addr":"172.21.15.96:6813/3106158554","state":["exists","up"]},{"osd":5,"uuid":"198dc992-96f8-4c52-96ee-1fd5de148bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6809","nonce":327293324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6811","nonce":327293324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6815","nonce":327293324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6813","nonce":327293324}]},"public_addr":"172.21.15.155:6809/327293324","cluster_addr":"172.21.15.155:6811/327293324","heartbeat_back_addr":"172.21.15.155:6815/327293324","heartbeat_front_addr":"172.21.15.155:6813/327293324","state":["exists","up"]},{"osd":6,"uuid":"a7278ec0-ab2e-49ed-be53-84d8b9e24177","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6819","nonce":873871509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6821","nonce":873871509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6825","nonce":873871509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6823","nonce":873871509}]},"public_addr":"172.21.15.33:6819/873871509","cluster_addr":"172.21.15.33:6821/873871509","heartbeat_back_addr":"172.21.15.33:6825/873871509","heartbeat_front_addr":"172.21.15.33:6823/873871509","state":["exists","up"]},{"osd":7,"uuid":"727e133d-aa24-4ffc-90e2-266eb2aab8c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6817","nonce":2334123891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6819","nonce":2334123891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6823","nonce":2334123891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6821","nonce":2334123891}]},"public_addr":"172.21.15.96:6817/2334123891","cluster_addr":"172.21.15.96:6819/2334123891","heartbeat_back_addr":"172.21.15.96:6823/2334123891","heartbeat_front_addr":"172.21.15.96:6821/2334123891","state":["exists","up"]},{"osd":8,"uuid":"bb20eec0-39e5-4b86-9db5-486b1840cdad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6817","nonce":3498516903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6819","nonce":3498516903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6823","nonce":3498516903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6821","nonce":3498516903}]},"public_addr":"172.21.15.155:6817/3498516903","cluster_addr":"172.21.15.155:6819/3498516903","heartbeat_back_addr":"172.21.15.155:6823/3498516903","heartbeat_front_addr":"172.21.15.155:6821/3498516903","state":["exists","up"]},{"osd":9,"uuid":"179c6228-3e23-4f65-91a6-1ef0005d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6827","nonce":760777213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6829","nonce":760777213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6833","nonce":760777213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6831","nonce":760777213}]},"public_addr":"172.21.15.33:6827/760777213","cluster_addr":"172.21.15.33:6829/760777213","heartbeat_back_addr":"172.21.15.33:6833/760777213","heartbeat_front_addr":"172.21.15.33:6831/760777213","state":["exists","up"]},{"osd":10,"uuid":"be17b478-3c2e-41fb-bede-958333acdf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6825","nonce":1404898482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6827","nonce":1404898482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6831","nonce":1404898482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6829","nonce":1404898482}]},"public_addr":"172.21.15.96:6825/1404898482","cluster_addr":"172.21.15.96:6827/1404898482","heartbeat_back_addr":"172.21.15.96:6831/1404898482","heartbeat_front_addr":"172.21.15.96:6829/1404898482","state":["exists","up"]},{"osd":11,"uuid":"c0903ed2-cc1e-4c18-8731-1f0ee4dd0457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6825","nonce":792352827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6827","nonce":792352827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6831","nonce":792352827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6829","nonce":792352827}]},"public_addr":"172.21.15.155:6825/792352827","cluster_addr":"172.21.15.155:6827/792352827","heartbeat_back_addr":"172.21.15.155:6831/792352827","heartbeat_front_addr":"172.21.15.155:6829/792352827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:17.942801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:42.109449+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:05.981599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:32.069574+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:57.431015+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:22.448592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:48.526515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:14.498352+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:40.175426+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:06.364006+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:32.853161+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:59.073538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:6801/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:6800/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/1067768047":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/729379453":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2586588742":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/2932814165":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6801/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2212564963":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6800/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/3275126058":"2024-03-20T02:43:39.622218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:52:39.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.259+0000 7f10cc952640 1 -- 172.21.15.33:0/4125628526 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f10b00a2890 msgr2=0x7f10b00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:39.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.259+0000 7f10cc952640 1 --2- 172.21.15.33:0/4125628526 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f10b00a2890 0x7f10b00a4d50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f10b8009230 tx=0x7f10b8005fd0 comp rx=0 tx=0).stop 2024-03-19T02:52:39.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.259+0000 7f10cc952640 1 -- 172.21.15.33:0/4125628526 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f10c8104c50 msgr2=0x7f10c8198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:39.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.259+0000 7f10cc952640 1 --2- 172.21.15.33:0/4125628526 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f10c8104c50 0x7f10c8198000 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f10bc00e9f0 tx=0x7f10bc00edf0 comp rx=0 tx=0).stop 2024-03-19T02:52:39.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.259+0000 7f10cc952640 1 -- 172.21.15.33:0/4125628526 shutdown_connections 2024-03-19T02:52:39.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.259+0000 7f10cc952640 1 --2- 172.21.15.33:0/4125628526 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f10b00a2890 0x7f10b00a4d50 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:39.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.259+0000 7f10cc952640 1 --2- 172.21.15.33:0/4125628526 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10c8107580 0x7f10c819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:39.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.259+0000 7f10cc952640 1 --2- 172.21.15.33:0/4125628526 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f10c8104c50 0x7f10c8198000 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:39.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.259+0000 7f10cc952640 1 --2- 172.21.15.33:0/4125628526 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10c8102320 0x7f10c8197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:39.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.259+0000 7f10cc952640 1 -- 172.21.15.33:0/4125628526 >> 172.21.15.33:0/4125628526 conn(0x7f10c80fbcb0 msgr2=0x7f10c8102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:39.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.259+0000 7f10cc952640 1 -- 172.21.15.33:0/4125628526 shutdown_connections 2024-03-19T02:52:39.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.259+0000 7f10cc952640 1 -- 172.21.15.33:0/4125628526 wait complete. 2024-03-19T02:52:39.270 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:39 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2544686530' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:52:39.270 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:39 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/504583019' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:52:39.270 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:39 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/408175535' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:52:39.271 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:46:09.963737+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T02:50:50.069641+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T02:50:51.381064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T02:50:55.476667+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T02:52:39.272 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T02:52:39.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.376+0000 7f2d8a710640 1 -- 172.21.15.33:0/1448613936 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d84101920 msgr2=0x7f2d84101da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:39.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.376+0000 7f2d8a710640 1 --2- 172.21.15.33:0/1448613936 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d84101920 0x7f2d84101da0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f2d74009900 tx=0x7f2d7402bc30 comp rx=0 tx=0).stop 2024-03-19T02:52:39.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.376+0000 7f2d8a710640 1 -- 172.21.15.33:0/1448613936 shutdown_connections 2024-03-19T02:52:39.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.376+0000 7f2d8a710640 1 --2- 172.21.15.33:0/1448613936 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d841022e0 0x7f2d84108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:39.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.376+0000 7f2d8a710640 1 --2- 172.21.15.33:0/1448613936 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d84101920 0x7f2d84101da0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:39.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.376+0000 7f2d8a710640 1 --2- 172.21.15.33:0/1448613936 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d84100730 0x7f2d84100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:39.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.376+0000 7f2d8a710640 1 -- 172.21.15.33:0/1448613936 >> 172.21.15.33:0/1448613936 conn(0x7f2d840fbcc0 msgr2=0x7f2d840fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:39.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.376+0000 7f2d8a710640 1 -- 172.21.15.33:0/1448613936 shutdown_connections 2024-03-19T02:52:39.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.376+0000 7f2d8a710640 1 -- 172.21.15.33:0/1448613936 wait complete. 2024-03-19T02:52:39.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.377+0000 7f2d8a710640 1 Processor -- start 2024-03-19T02:52:39.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.377+0000 7f2d8a710640 1 -- start start 2024-03-19T02:52:39.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.377+0000 7f2d8a710640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d84100730 0x7f2d84197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:39.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.377+0000 7f2d8a710640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d84101920 0x7f2d84197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:39.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.377+0000 7f2d8a710640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d841022e0 0x7f2d8419f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:39.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.377+0000 7f2d8a710640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8410c260 con 0x7f2d84100730 2024-03-19T02:52:39.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.377+0000 7f2d8a710640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8410c0e0 con 0x7f2d84101920 2024-03-19T02:52:39.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.377+0000 7f2d8a710640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8410c3e0 con 0x7f2d841022e0 2024-03-19T02:52:39.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.377+0000 7f2d83fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d84100730 0x7f2d84197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:39.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.377+0000 7f2d83fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d84100730 0x7f2d84197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41536/0 (socket says 172.21.15.33:41536) 2024-03-19T02:52:39.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.377+0000 7f2d83fff640 1 -- 172.21.15.33:0/577717634 learned_addr learned my addr 172.21.15.33:0/577717634 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:39.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.377+0000 7f2d837fe640 1 --2- 172.21.15.33:0/577717634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d84101920 0x7f2d84197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:39.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.377+0000 7f2d88cad640 1 --2- 172.21.15.33:0/577717634 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d841022e0 0x7f2d8419f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:39.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.378+0000 7f2d83fff640 1 -- 172.21.15.33:0/577717634 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d841022e0 msgr2=0x7f2d8419f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:39.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.378+0000 7f2d83fff640 1 --2- 172.21.15.33:0/577717634 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d841022e0 0x7f2d8419f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:39.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.378+0000 7f2d83fff640 1 -- 172.21.15.33:0/577717634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d84101920 msgr2=0x7f2d84197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:39.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.378+0000 7f2d83fff640 1 --2- 172.21.15.33:0/577717634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d84101920 0x7f2d84197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:39.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.378+0000 7f2d83fff640 1 -- 172.21.15.33:0/577717634 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d8419f6a0 con 0x7f2d84100730 2024-03-19T02:52:39.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.378+0000 7f2d83fff640 1 --2- 172.21.15.33:0/577717634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d84100730 0x7f2d84197a80 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c00e9c0 tx=0x7f2d6c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:39.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.378+0000 7f2d817fa640 1 -- 172.21.15.33:0/577717634 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c00cde0 con 0x7f2d84100730 2024-03-19T02:52:39.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.378+0000 7f2d817fa640 1 -- 172.21.15.33:0/577717634 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2d6c0108e0 con 0x7f2d84100730 2024-03-19T02:52:39.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.378+0000 7f2d8a710640 1 -- 172.21.15.33:0/577717634 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d8419f990 con 0x7f2d84100730 2024-03-19T02:52:39.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.378+0000 7f2d817fa640 1 -- 172.21.15.33:0/577717634 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c018990 con 0x7f2d84100730 2024-03-19T02:52:39.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.378+0000 7f2d8a710640 1 -- 172.21.15.33:0/577717634 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d8419ff00 con 0x7f2d84100730 2024-03-19T02:52:39.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.379+0000 7f2d8a710640 1 -- 172.21.15.33:0/577717634 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d841a0130 con 0x7f2d84100730 2024-03-19T02:52:39.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.380+0000 7f2d817fa640 1 -- 172.21.15.33:0/577717634 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f2d6c01f020 con 0x7f2d84100730 2024-03-19T02:52:39.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.380+0000 7f2d817fa640 1 --2- 172.21.15.33:0/577717634 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2d700a2910 0x7f2d700a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:39.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.381+0000 7f2d837fe640 1 --2- 172.21.15.33:0/577717634 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2d700a2910 0x7f2d700a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:39.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.381+0000 7f2d817fa640 1 -- 172.21.15.33:0/577717634 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7f2d6c0cf0f0 con 0x7f2d84100730 2024-03-19T02:52:39.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.381+0000 7f2d837fe640 1 --2- 172.21.15.33:0/577717634 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2d700a2910 0x7f2d700a4dd0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2d74009230 tx=0x7f2d74005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:39.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.385+0000 7f2d817fa640 1 -- 172.21.15.33:0/577717634 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2d6c09bc70 con 0x7f2d84100730 2024-03-19T02:52:39.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:39 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2544686530' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:52:39.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:39 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/504583019' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:52:39.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:39 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/408175535' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:52:39.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:39 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2544686530' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:52:39.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:39 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/504583019' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:52:39.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:39 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/408175535' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:52:39.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.567+0000 7f2d8a710640 1 -- 172.21.15.33:0/577717634 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2d841a04c0 con 0x7f2d84100730 2024-03-19T02:52:39.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.567+0000 7f2d817fa640 1 -- 172.21.15.33:0/577717634 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7f2d6c09b570 con 0x7f2d84100730 2024-03-19T02:52:39.568 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 1 2024-03-19T02:52:39.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.569+0000 7f2d8a710640 1 -- 172.21.15.33:0/577717634 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2d700a2910 msgr2=0x7f2d700a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:39.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.569+0000 7f2d8a710640 1 --2- 172.21.15.33:0/577717634 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2d700a2910 0x7f2d700a4dd0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2d74009230 tx=0x7f2d74005fd0 comp rx=0 tx=0).stop 2024-03-19T02:52:39.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.569+0000 7f2d8a710640 1 -- 172.21.15.33:0/577717634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d84100730 msgr2=0x7f2d84197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:39.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.569+0000 7f2d8a710640 1 --2- 172.21.15.33:0/577717634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d84100730 0x7f2d84197a80 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c00e9c0 tx=0x7f2d6c00ee80 comp rx=0 tx=0).stop 2024-03-19T02:52:39.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.569+0000 7f2d8a710640 1 -- 172.21.15.33:0/577717634 shutdown_connections 2024-03-19T02:52:39.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.569+0000 7f2d8a710640 1 --2- 172.21.15.33:0/577717634 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2d700a2910 0x7f2d700a4dd0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:39.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.569+0000 7f2d8a710640 1 --2- 172.21.15.33:0/577717634 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d841022e0 0x7f2d8419f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:39.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.569+0000 7f2d8a710640 1 --2- 172.21.15.33:0/577717634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2d84101920 0x7f2d84197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:39.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.569+0000 7f2d8a710640 1 --2- 172.21.15.33:0/577717634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2d84100730 0x7f2d84197a80 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:39.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.569+0000 7f2d8a710640 1 -- 172.21.15.33:0/577717634 >> 172.21.15.33:0/577717634 conn(0x7f2d840fbcc0 msgr2=0x7f2d84069df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:39.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.569+0000 7f2d8a710640 1 -- 172.21.15.33:0/577717634 shutdown_connections 2024-03-19T02:52:39.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.569+0000 7f2d8a710640 1 -- 172.21.15.33:0/577717634 wait complete. 2024-03-19T02:52:39.581 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T02:52:39.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.685+0000 7fac33d87640 1 -- 172.21.15.33:0/3376375135 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac2c100720 msgr2=0x7fac2c100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:39.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.685+0000 7fac33d87640 1 --2- 172.21.15.33:0/3376375135 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac2c100720 0x7fac2c100b40 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fac24009860 tx=0x7fac2402bb70 comp rx=0 tx=0).stop 2024-03-19T02:52:39.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.685+0000 7fac33d87640 1 -- 172.21.15.33:0/3376375135 shutdown_connections 2024-03-19T02:52:39.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.685+0000 7fac33d87640 1 --2- 172.21.15.33:0/3376375135 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac2c1022d0 0x7fac2c108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:39.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.685+0000 7fac33d87640 1 --2- 172.21.15.33:0/3376375135 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac2c101910 0x7fac2c101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:39.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.685+0000 7fac33d87640 1 --2- 172.21.15.33:0/3376375135 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac2c100720 0x7fac2c100b40 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:39.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.686+0000 7fac33d87640 1 -- 172.21.15.33:0/3376375135 >> 172.21.15.33:0/3376375135 conn(0x7fac2c0fbcb0 msgr2=0x7fac2c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:39.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.686+0000 7fac33d87640 1 -- 172.21.15.33:0/3376375135 shutdown_connections 2024-03-19T02:52:39.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.686+0000 7fac33d87640 1 -- 172.21.15.33:0/3376375135 wait complete. 2024-03-19T02:52:39.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.686+0000 7fac33d87640 1 Processor -- start 2024-03-19T02:52:39.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.686+0000 7fac33d87640 1 -- start start 2024-03-19T02:52:39.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.686+0000 7fac33d87640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac2c100720 0x7fac2c197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:39.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.686+0000 7fac33d87640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac2c101910 0x7fac2c198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:39.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.686+0000 7fac33d87640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac2c1022d0 0x7fac2c19f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:39.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.686+0000 7fac33d87640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fac2c10c220 con 0x7fac2c101910 2024-03-19T02:52:39.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.686+0000 7fac33d87640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fac2c10c0a0 con 0x7fac2c100720 2024-03-19T02:52:39.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.686+0000 7fac33d87640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fac2c10c3a0 con 0x7fac2c1022d0 2024-03-19T02:52:39.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.686+0000 7fac31322640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac2c101910 0x7fac2c198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:39.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.686+0000 7fac31322640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac2c101910 0x7fac2c198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41544/0 (socket says 172.21.15.33:41544) 2024-03-19T02:52:39.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.686+0000 7fac31322640 1 -- 172.21.15.33:0/3625234229 learned_addr learned my addr 172.21.15.33:0/3625234229 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:39.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.686+0000 7fac31b23640 1 --2- 172.21.15.33:0/3625234229 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac2c100720 0x7fac2c197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:39.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.686+0000 7fac32324640 1 --2- 172.21.15.33:0/3625234229 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac2c1022d0 0x7fac2c19f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:39.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.687+0000 7fac31322640 1 -- 172.21.15.33:0/3625234229 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac2c1022d0 msgr2=0x7fac2c19f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:39.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.687+0000 7fac31322640 1 --2- 172.21.15.33:0/3625234229 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac2c1022d0 0x7fac2c19f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:39.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.687+0000 7fac31322640 1 -- 172.21.15.33:0/3625234229 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac2c100720 msgr2=0x7fac2c197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:39.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.687+0000 7fac31322640 1 --2- 172.21.15.33:0/3625234229 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac2c100720 0x7fac2c197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:39.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.687+0000 7fac31322640 1 -- 172.21.15.33:0/3625234229 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac2c19f720 con 0x7fac2c101910 2024-03-19T02:52:39.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.687+0000 7fac31322640 1 --2- 172.21.15.33:0/3625234229 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac2c101910 0x7fac2c198040 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fac2000ea60 tx=0x7fac2000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:39.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.688+0000 7fac2affd640 1 -- 172.21.15.33:0/3625234229 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2000cde0 con 0x7fac2c101910 2024-03-19T02:52:39.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.688+0000 7fac2affd640 1 -- 172.21.15.33:0/3625234229 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac200035f0 con 0x7fac2c101910 2024-03-19T02:52:39.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.688+0000 7fac2affd640 1 -- 172.21.15.33:0/3625234229 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2000cde0 con 0x7fac2c101910 2024-03-19T02:52:39.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.690+0000 7fac33d87640 1 -- 172.21.15.33:0/3625234229 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac2c19fa10 con 0x7fac2c101910 2024-03-19T02:52:39.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.690+0000 7fac33d87640 1 -- 172.21.15.33:0/3625234229 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac2c19fee0 con 0x7fac2c101910 2024-03-19T02:52:39.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.692+0000 7fac28ff9640 1 -- 172.21.15.33:0/3625234229 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac2c062420 con 0x7fac2c101910 2024-03-19T02:52:39.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.695+0000 7fac2affd640 1 -- 172.21.15.33:0/3625234229 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fac200037a0 con 0x7fac2c101910 2024-03-19T02:52:39.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.696+0000 7fac2affd640 1 --2- 172.21.15.33:0/3625234229 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fac140a2d10 0x7fac140a51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:39.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.696+0000 7fac2affd640 1 -- 172.21.15.33:0/3625234229 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7fac2000cbe0 con 0x7fac2c101910 2024-03-19T02:52:39.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.706+0000 7fac2affd640 1 -- 172.21.15.33:0/3625234229 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fac2001d060 con 0x7fac2c101910 2024-03-19T02:52:39.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.708+0000 7fac31b23640 1 --2- 172.21.15.33:0/3625234229 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fac140a2d10 0x7fac140a51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:39.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.716+0000 7fac31b23640 1 --2- 172.21.15.33:0/3625234229 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fac140a2d10 0x7fac140a51d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fac2402c180 tx=0x7fac240349a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:39.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.904+0000 7fac28ff9640 1 -- 172.21.15.33:0/3625234229 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fac2c062930 con 0x7fac2c101910 2024-03-19T02:52:39.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.904+0000 7fac2affd640 1 -- 172.21.15.33:0/3625234229 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) ==== 104+0+11 (secure 0 0 0) 0x7fac2001d060 con 0x7fac2c101910 2024-03-19T02:52:39.905 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 32 2024-03-19T02:52:39.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.906+0000 7fac28ff9640 1 -- 172.21.15.33:0/3625234229 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fac140a2d10 msgr2=0x7fac140a51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:39.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.906+0000 7fac28ff9640 1 --2- 172.21.15.33:0/3625234229 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fac140a2d10 0x7fac140a51d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fac2402c180 tx=0x7fac240349a0 comp rx=0 tx=0).stop 2024-03-19T02:52:39.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.906+0000 7fac28ff9640 1 -- 172.21.15.33:0/3625234229 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac2c101910 msgr2=0x7fac2c198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:39.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.906+0000 7fac28ff9640 1 --2- 172.21.15.33:0/3625234229 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac2c101910 0x7fac2c198040 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fac2000ea60 tx=0x7fac2000ef20 comp rx=0 tx=0).stop 2024-03-19T02:52:39.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.906+0000 7fac28ff9640 1 -- 172.21.15.33:0/3625234229 shutdown_connections 2024-03-19T02:52:39.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.906+0000 7fac28ff9640 1 --2- 172.21.15.33:0/3625234229 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fac140a2d10 0x7fac140a51d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:39.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.906+0000 7fac28ff9640 1 --2- 172.21.15.33:0/3625234229 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac2c1022d0 0x7fac2c19f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:39.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.906+0000 7fac28ff9640 1 --2- 172.21.15.33:0/3625234229 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac2c101910 0x7fac2c198040 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:39.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.906+0000 7fac28ff9640 1 --2- 172.21.15.33:0/3625234229 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac2c100720 0x7fac2c197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:39.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.906+0000 7fac28ff9640 1 -- 172.21.15.33:0/3625234229 >> 172.21.15.33:0/3625234229 conn(0x7fac2c0fbcb0 msgr2=0x7fac2c068a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:39.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.906+0000 7fac28ff9640 1 -- 172.21.15.33:0/3625234229 shutdown_connections 2024-03-19T02:52:39.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:39.906+0000 7fac28ff9640 1 -- 172.21.15.33:0/3625234229 wait complete. 2024-03-19T02:52:39.918 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T02:52:40.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.023+0000 7f8474162640 1 -- 172.21.15.33:0/794615417 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f846c1022f0 msgr2=0x7f846c108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:40.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.023+0000 7f8474162640 1 --2- 172.21.15.33:0/794615417 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f846c1022f0 0x7f846c108b80 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f8464009ca0 tx=0x7f846402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.023+0000 7f8474162640 1 -- 172.21.15.33:0/794615417 shutdown_connections 2024-03-19T02:52:40.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.023+0000 7f8474162640 1 --2- 172.21.15.33:0/794615417 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f846c1022f0 0x7f846c108b80 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.023+0000 7f8474162640 1 --2- 172.21.15.33:0/794615417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f846c101930 0x7f846c101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.023+0000 7f8474162640 1 --2- 172.21.15.33:0/794615417 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f846c100740 0x7f846c100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.023+0000 7f8474162640 1 -- 172.21.15.33:0/794615417 >> 172.21.15.33:0/794615417 conn(0x7f846c0fbcd0 msgr2=0x7f846c0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:40.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.023+0000 7f8474162640 1 -- 172.21.15.33:0/794615417 shutdown_connections 2024-03-19T02:52:40.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.023+0000 7f8474162640 1 -- 172.21.15.33:0/794615417 wait complete. 2024-03-19T02:52:40.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.024+0000 7f8474162640 1 Processor -- start 2024-03-19T02:52:40.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.024+0000 7f8474162640 1 -- start start 2024-03-19T02:52:40.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.024+0000 7f8474162640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f846c100740 0x7f846c199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:40.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.024+0000 7f8474162640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f846c101930 0x7f846c19a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:40.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.024+0000 7f8474162640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f846c1022f0 0x7f846c1a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:40.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.024+0000 7f8474162640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f846c10c270 con 0x7f846c1022f0 2024-03-19T02:52:40.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.024+0000 7f8474162640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f846c10c0f0 con 0x7f846c101930 2024-03-19T02:52:40.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.024+0000 7f8474162640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f846c10c3f0 con 0x7f846c100740 2024-03-19T02:52:40.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.024+0000 7f84726ff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f846c1022f0 0x7f846c1a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:40.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.024+0000 7f84726ff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f846c1022f0 0x7f846c1a11d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41576/0 (socket says 172.21.15.33:41576) 2024-03-19T02:52:40.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.024+0000 7f84726ff640 1 -- 172.21.15.33:0/918737244 learned_addr learned my addr 172.21.15.33:0/918737244 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:40.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.024+0000 7f8471efe640 1 --2- 172.21.15.33:0/918737244 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f846c100740 0x7f846c199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:40.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.024+0000 7f84716fd640 1 --2- 172.21.15.33:0/918737244 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f846c101930 0x7f846c19a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:40.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.024+0000 7f84726ff640 1 -- 172.21.15.33:0/918737244 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f846c100740 msgr2=0x7f846c199c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:40.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.024+0000 7f84726ff640 1 --2- 172.21.15.33:0/918737244 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f846c100740 0x7f846c199c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.024+0000 7f84726ff640 1 -- 172.21.15.33:0/918737244 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f846c101930 msgr2=0x7f846c19a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:40.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.024+0000 7f84726ff640 1 --2- 172.21.15.33:0/918737244 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f846c101930 0x7f846c19a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.024+0000 7f84726ff640 1 -- 172.21.15.33:0/918737244 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f846c1a1870 con 0x7f846c1022f0 2024-03-19T02:52:40.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.024+0000 7f84726ff640 1 --2- 172.21.15.33:0/918737244 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f846c1022f0 0x7f846c1a11d0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f846402c700 tx=0x7f84640328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:40.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.025+0000 7f8462ffd640 1 -- 172.21.15.33:0/918737244 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8464041020 con 0x7f846c1022f0 2024-03-19T02:52:40.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.025+0000 7f8474162640 1 -- 172.21.15.33:0/918737244 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f846c1a1b00 con 0x7f846c1022f0 2024-03-19T02:52:40.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.025+0000 7f8462ffd640 1 -- 172.21.15.33:0/918737244 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8464003e20 con 0x7f846c1022f0 2024-03-19T02:52:40.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.025+0000 7f8462ffd640 1 -- 172.21.15.33:0/918737244 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8464039040 con 0x7f846c1022f0 2024-03-19T02:52:40.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.025+0000 7f8474162640 1 -- 172.21.15.33:0/918737244 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f846c1a1ff0 con 0x7f846c1022f0 2024-03-19T02:52:40.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.026+0000 7f8474162640 1 -- 172.21.15.33:0/918737244 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f846c1a2220 con 0x7f846c1022f0 2024-03-19T02:52:40.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.027+0000 7f8462ffd640 1 -- 172.21.15.33:0/918737244 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f8464044020 con 0x7f846c1022f0 2024-03-19T02:52:40.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.027+0000 7f8462ffd640 1 --2- 172.21.15.33:0/918737244 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f84580a2890 0x7f84580a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:40.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.027+0000 7f8471efe640 1 --2- 172.21.15.33:0/918737244 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f84580a2890 0x7f84580a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:40.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.027+0000 7f8462ffd640 1 -- 172.21.15.33:0/918737244 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9092+0+0 (secure 0 0 0) 0x7f84640f10d0 con 0x7f846c1022f0 2024-03-19T02:52:40.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.028+0000 7f8471efe640 1 --2- 172.21.15.33:0/918737244 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f84580a2890 0x7f84580a4d50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f846c19b130 tx=0x7f8468005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:40.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.032+0000 7f8462ffd640 1 -- 172.21.15.33:0/918737244 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f846404b020 con 0x7f846c1022f0 2024-03-19T02:52:40.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.220+0000 7f8474162640 1 -- 172.21.15.33:0/918737244 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f846c1a23f0 con 0x7f846c1022f0 2024-03-19T02:52:40.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.220+0000 7f8462ffd640 1 -- 172.21.15.33:0/918737244 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) ==== 100+0+11 (secure 0 0 0) 0x7f84640bddc0 con 0x7f846c1022f0 2024-03-19T02:52:40.221 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 64 2024-03-19T02:52:40.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.222+0000 7f8474162640 1 -- 172.21.15.33:0/918737244 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f84580a2890 msgr2=0x7f84580a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:40.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.222+0000 7f8474162640 1 --2- 172.21.15.33:0/918737244 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f84580a2890 0x7f84580a4d50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f846c19b130 tx=0x7f8468005f90 comp rx=0 tx=0).stop 2024-03-19T02:52:40.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.222+0000 7f8474162640 1 -- 172.21.15.33:0/918737244 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f846c1022f0 msgr2=0x7f846c1a11d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:40.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.222+0000 7f8474162640 1 --2- 172.21.15.33:0/918737244 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f846c1022f0 0x7f846c1a11d0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f846402c700 tx=0x7f84640328a0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.222+0000 7f8474162640 1 -- 172.21.15.33:0/918737244 shutdown_connections 2024-03-19T02:52:40.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.222+0000 7f8474162640 1 --2- 172.21.15.33:0/918737244 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f84580a2890 0x7f84580a4d50 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.222+0000 7f8474162640 1 --2- 172.21.15.33:0/918737244 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f846c1022f0 0x7f846c1a11d0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.222+0000 7f8474162640 1 --2- 172.21.15.33:0/918737244 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f846c101930 0x7f846c19a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.222+0000 7f8474162640 1 --2- 172.21.15.33:0/918737244 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f846c100740 0x7f846c199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.222+0000 7f8474162640 1 -- 172.21.15.33:0/918737244 >> 172.21.15.33:0/918737244 conn(0x7f846c0fbcd0 msgr2=0x7f846c0fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:40.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.222+0000 7f8474162640 1 -- 172.21.15.33:0/918737244 shutdown_connections 2024-03-19T02:52:40.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.222+0000 7f8474162640 1 -- 172.21.15.33:0/918737244 wait complete. 2024-03-19T02:52:40.233 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:40 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/4125628526' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:52:40.233 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:40 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/577717634' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:52:40.233 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 8]}]: dispatch 2024-03-19T02:52:40.233 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:40 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-03-19T02:52:40.233 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:40 smithi033 ceph-mon[31973]: pgmap v355: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-19T02:52:40.233 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:40 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3625234229' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:52:40.234 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T02:52:40.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.339+0000 7fe01bc85640 1 -- 172.21.15.33:0/1807035154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe014102a30 msgr2=0x7fe014102eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:40.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.339+0000 7fe01bc85640 1 --2- 172.21.15.33:0/1807035154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe014102a30 0x7fe014102eb0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe004009900 tx=0x7fe00402bc30 comp rx=0 tx=0).stop 2024-03-19T02:52:40.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.339+0000 7fe01bc85640 1 -- 172.21.15.33:0/1807035154 shutdown_connections 2024-03-19T02:52:40.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.339+0000 7fe01bc85640 1 --2- 172.21.15.33:0/1807035154 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0141033f0 0x7fe014109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.339+0000 7fe01bc85640 1 --2- 172.21.15.33:0/1807035154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe014102a30 0x7fe014102eb0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.339+0000 7fe01bc85640 1 --2- 172.21.15.33:0/1807035154 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe014101840 0x7fe014101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.339+0000 7fe01bc85640 1 -- 172.21.15.33:0/1807035154 >> 172.21.15.33:0/1807035154 conn(0x7fe0140fcd90 msgr2=0x7fe0140ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:40.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.339+0000 7fe01bc85640 1 -- 172.21.15.33:0/1807035154 shutdown_connections 2024-03-19T02:52:40.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.339+0000 7fe01bc85640 1 -- 172.21.15.33:0/1807035154 wait complete. 2024-03-19T02:52:40.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.340+0000 7fe01bc85640 1 Processor -- start 2024-03-19T02:52:40.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.340+0000 7fe01bc85640 1 -- start start 2024-03-19T02:52:40.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.340+0000 7fe01bc85640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe014101840 0x7fe014197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:40.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.340+0000 7fe01bc85640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe014102a30 0x7fe014197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:40.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.340+0000 7fe01bc85640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0141033f0 0x7fe01419f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:40.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.340+0000 7fe01bc85640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe01410d370 con 0x7fe014101840 2024-03-19T02:52:40.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.340+0000 7fe01bc85640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe01410d1f0 con 0x7fe0141033f0 2024-03-19T02:52:40.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.340+0000 7fe01bc85640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe01410d4f0 con 0x7fe014102a30 2024-03-19T02:52:40.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.340+0000 7fe019a21640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe014101840 0x7fe014197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:40.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.340+0000 7fe019a21640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe014101840 0x7fe014197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41592/0 (socket says 172.21.15.33:41592) 2024-03-19T02:52:40.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.340+0000 7fe019a21640 1 -- 172.21.15.33:0/3101531060 learned_addr learned my addr 172.21.15.33:0/3101531060 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:40.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.340+0000 7fe01a222640 1 --2- 172.21.15.33:0/3101531060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0141033f0 0x7fe01419f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:40.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.340+0000 7fe019220640 1 --2- 172.21.15.33:0/3101531060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe014102a30 0x7fe014197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:40.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.340+0000 7fe019a21640 1 -- 172.21.15.33:0/3101531060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe014102a30 msgr2=0x7fe014197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:40.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.340+0000 7fe019a21640 1 --2- 172.21.15.33:0/3101531060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe014102a30 0x7fe014197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.340+0000 7fe019a21640 1 -- 172.21.15.33:0/3101531060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0141033f0 msgr2=0x7fe01419f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:40.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.340+0000 7fe019a21640 1 --2- 172.21.15.33:0/3101531060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0141033f0 0x7fe01419f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.340+0000 7fe019a21640 1 -- 172.21.15.33:0/3101531060 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe01419f6c0 con 0x7fe014101840 2024-03-19T02:52:40.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.340+0000 7fe01a222640 1 --2- 172.21.15.33:0/3101531060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0141033f0 0x7fe01419f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:52:40.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.340+0000 7fe019220640 1 --2- 172.21.15.33:0/3101531060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe014102a30 0x7fe014197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:52:40.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.340+0000 7fe019a21640 1 --2- 172.21.15.33:0/3101531060 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe014101840 0x7fe014197aa0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe01000e9c0 tx=0x7fe01000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:40.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.340+0000 7fe00affd640 1 -- 172.21.15.33:0/3101531060 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe01000cd80 con 0x7fe014101840 2024-03-19T02:52:40.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.340+0000 7fe01bc85640 1 -- 172.21.15.33:0/3101531060 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe01419f9b0 con 0x7fe014101840 2024-03-19T02:52:40.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.340+0000 7fe00affd640 1 -- 172.21.15.33:0/3101531060 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe0100035f0 con 0x7fe014101840 2024-03-19T02:52:40.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.340+0000 7fe00affd640 1 -- 172.21.15.33:0/3101531060 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe010010690 con 0x7fe014101840 2024-03-19T02:52:40.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.341+0000 7fe01bc85640 1 -- 172.21.15.33:0/3101531060 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe01419ff50 con 0x7fe014101840 2024-03-19T02:52:40.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.342+0000 7fe01bc85640 1 -- 172.21.15.33:0/3101531060 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe014068a60 con 0x7fe014101840 2024-03-19T02:52:40.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.342+0000 7fe00affd640 1 -- 172.21.15.33:0/3101531060 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fe0100037a0 con 0x7fe014101840 2024-03-19T02:52:40.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.343+0000 7fe00affd640 1 --2- 172.21.15.33:0/3101531060 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe0000a2890 0x7fe0000a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:40.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.343+0000 7fe019220640 1 --2- 172.21.15.33:0/3101531060 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe0000a2890 0x7fe0000a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:40.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.343+0000 7fe00affd640 1 -- 172.21.15.33:0/3101531060 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7fe0100cf460 con 0x7fe014101840 2024-03-19T02:52:40.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.344+0000 7fe019220640 1 --2- 172.21.15.33:0/3101531060 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe0000a2890 0x7fe0000a4d50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe014198fb0 tx=0x7fe00402c7e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:40.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.348+0000 7fe00affd640 1 -- 172.21.15.33:0/3101531060 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe01009c010 con 0x7fe014101840 2024-03-19T02:52:40.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:40 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/4125628526' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:52:40.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:40 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/577717634' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:52:40.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:40 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 8]}]: dispatch 2024-03-19T02:52:40.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:40 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-03-19T02:52:40.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:40 smithi096 ceph-mon[35946]: pgmap v355: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-19T02:52:40.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:40 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3625234229' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:52:40.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:40 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/4125628526' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:52:40.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:40 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/577717634' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:52:40.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 8]}]: dispatch 2024-03-19T02:52:40.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:40 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-03-19T02:52:40.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:40 smithi155 ceph-mon[35804]: pgmap v355: 129 pgs: 129 active+clean; 642 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-19T02:52:40.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:40 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3625234229' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:52:40.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.530+0000 7fe01bc85640 1 -- 172.21.15.33:0/3101531060 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fe01402d1f0 con 0x7fe014101840 2024-03-19T02:52:40.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.530+0000 7fe00affd640 1 -- 172.21.15.33:0/3101531060 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v96) ==== 103+0+11 (secure 0 0 0) 0x7fe01009b910 con 0x7fe014101840 2024-03-19T02:52:40.531 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 32 2024-03-19T02:52:40.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.532+0000 7fe01bc85640 1 -- 172.21.15.33:0/3101531060 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe0000a2890 msgr2=0x7fe0000a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:40.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.532+0000 7fe01bc85640 1 --2- 172.21.15.33:0/3101531060 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe0000a2890 0x7fe0000a4d50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe014198fb0 tx=0x7fe00402c7e0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.532+0000 7fe01bc85640 1 -- 172.21.15.33:0/3101531060 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe014101840 msgr2=0x7fe014197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:40.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.532+0000 7fe01bc85640 1 --2- 172.21.15.33:0/3101531060 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe014101840 0x7fe014197aa0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe01000e9c0 tx=0x7fe01000ee80 comp rx=0 tx=0).stop 2024-03-19T02:52:40.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.532+0000 7fe01bc85640 1 -- 172.21.15.33:0/3101531060 shutdown_connections 2024-03-19T02:52:40.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.532+0000 7fe01bc85640 1 --2- 172.21.15.33:0/3101531060 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe0000a2890 0x7fe0000a4d50 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.532+0000 7fe01bc85640 1 --2- 172.21.15.33:0/3101531060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe014102a30 0x7fe014197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.532+0000 7fe01bc85640 1 --2- 172.21.15.33:0/3101531060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe0141033f0 0x7fe01419f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.532+0000 7fe01bc85640 1 --2- 172.21.15.33:0/3101531060 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe014101840 0x7fe014197aa0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.532+0000 7fe01bc85640 1 -- 172.21.15.33:0/3101531060 >> 172.21.15.33:0/3101531060 conn(0x7fe0140fcd90 msgr2=0x7fe014105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:40.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.532+0000 7fe01bc85640 1 -- 172.21.15.33:0/3101531060 shutdown_connections 2024-03-19T02:52:40.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.532+0000 7fe01bc85640 1 -- 172.21.15.33:0/3101531060 wait complete. 2024-03-19T02:52:40.544 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:52:40.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.646+0000 7f7cd1fe6640 1 -- 172.21.15.33:0/574611410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ccc102a30 msgr2=0x7f7ccc102eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:40.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.646+0000 7f7cd1fe6640 1 --2- 172.21.15.33:0/574611410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ccc102a30 0x7f7ccc102eb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7cb4009900 tx=0x7f7cb402bc30 comp rx=0 tx=0).stop 2024-03-19T02:52:40.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.646+0000 7f7cd1fe6640 1 -- 172.21.15.33:0/574611410 shutdown_connections 2024-03-19T02:52:40.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.646+0000 7f7cd1fe6640 1 --2- 172.21.15.33:0/574611410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ccc102a30 0x7f7ccc102eb0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.646+0000 7f7cd1fe6640 1 --2- 172.21.15.33:0/574611410 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ccc1033f0 0x7f7ccc109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.646+0000 7f7cd1fe6640 1 --2- 172.21.15.33:0/574611410 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ccc101840 0x7f7ccc101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.646+0000 7f7cd1fe6640 1 -- 172.21.15.33:0/574611410 >> 172.21.15.33:0/574611410 conn(0x7f7ccc0fcdd0 msgr2=0x7f7ccc0ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:40.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.646+0000 7f7cd1fe6640 1 -- 172.21.15.33:0/574611410 shutdown_connections 2024-03-19T02:52:40.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.646+0000 7f7cd1fe6640 1 -- 172.21.15.33:0/574611410 wait complete. 2024-03-19T02:52:40.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.647+0000 7f7cd1fe6640 1 Processor -- start 2024-03-19T02:52:40.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.647+0000 7f7cd1fe6640 1 -- start start 2024-03-19T02:52:40.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.647+0000 7f7cd1fe6640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ccc101840 0x7f7ccc197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:40.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.647+0000 7f7cd1fe6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ccc102a30 0x7f7ccc197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:40.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.647+0000 7f7cd1fe6640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ccc1033f0 0x7f7ccc19f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:40.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.647+0000 7f7cd1fe6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc10d350 con 0x7f7ccc102a30 2024-03-19T02:52:40.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.647+0000 7f7cd1fe6640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc10d1d0 con 0x7f7ccc1033f0 2024-03-19T02:52:40.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.647+0000 7f7cd1fe6640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc10d4d0 con 0x7f7ccc101840 2024-03-19T02:52:40.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.647+0000 7f7cc3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ccc102a30 0x7f7ccc197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:40.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.647+0000 7f7cc3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ccc102a30 0x7f7ccc197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41612/0 (socket says 172.21.15.33:41612) 2024-03-19T02:52:40.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.647+0000 7f7cc3fff640 1 -- 172.21.15.33:0/2220367354 learned_addr learned my addr 172.21.15.33:0/2220367354 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:40.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.647+0000 7f7ccb7fe640 1 --2- 172.21.15.33:0/2220367354 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ccc101840 0x7f7ccc197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:40.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.647+0000 7f7ccbfff640 1 --2- 172.21.15.33:0/2220367354 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ccc1033f0 0x7f7ccc19f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:40.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.647+0000 7f7cc3fff640 1 -- 172.21.15.33:0/2220367354 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ccc101840 msgr2=0x7f7ccc197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:40.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.647+0000 7f7cc3fff640 1 --2- 172.21.15.33:0/2220367354 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ccc101840 0x7f7ccc197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.647+0000 7f7cc3fff640 1 -- 172.21.15.33:0/2220367354 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ccc1033f0 msgr2=0x7f7ccc19f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:40.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.647+0000 7f7cc3fff640 1 --2- 172.21.15.33:0/2220367354 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ccc1033f0 0x7f7ccc19f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.647+0000 7f7cc3fff640 1 -- 172.21.15.33:0/2220367354 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ccc19f670 con 0x7f7ccc102a30 2024-03-19T02:52:40.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.648+0000 7f7cc3fff640 1 --2- 172.21.15.33:0/2220367354 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ccc102a30 0x7f7ccc197f90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f7cb4006010 tx=0x7f7cb4005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:40.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.648+0000 7f7cc97fa640 1 -- 172.21.15.33:0/2220367354 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb4043020 con 0x7f7ccc102a30 2024-03-19T02:52:40.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.648+0000 7f7cd1fe6640 1 -- 172.21.15.33:0/2220367354 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ccc19f900 con 0x7f7ccc102a30 2024-03-19T02:52:40.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.648+0000 7f7cc97fa640 1 -- 172.21.15.33:0/2220367354 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7cb40043d0 con 0x7f7ccc102a30 2024-03-19T02:52:40.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.648+0000 7f7cc97fa640 1 -- 172.21.15.33:0/2220367354 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb4003b80 con 0x7f7ccc102a30 2024-03-19T02:52:40.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.648+0000 7f7cd1fe6640 1 -- 172.21.15.33:0/2220367354 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ccc19fea0 con 0x7f7ccc102a30 2024-03-19T02:52:40.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.649+0000 7f7cd1fe6640 1 -- 172.21.15.33:0/2220367354 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ccc108c30 con 0x7f7ccc102a30 2024-03-19T02:52:40.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.650+0000 7f7cc97fa640 1 -- 172.21.15.33:0/2220367354 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f7cb4046020 con 0x7f7ccc102a30 2024-03-19T02:52:40.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.651+0000 7f7cc97fa640 1 --2- 172.21.15.33:0/2220367354 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7cb80a2890 0x7f7cb80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:40.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.651+0000 7f7ccb7fe640 1 --2- 172.21.15.33:0/2220367354 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7cb80a2890 0x7f7cb80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:40.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.651+0000 7f7cc97fa640 1 -- 172.21.15.33:0/2220367354 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f7cb40f1670 con 0x7f7ccc102a30 2024-03-19T02:52:40.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.651+0000 7f7ccb7fe640 1 --2- 172.21.15.33:0/2220367354 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7cb80a2890 0x7f7cb80a4d50 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc102890 tx=0x7f7cbc005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:40.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.655+0000 7f7cc97fa640 1 -- 172.21.15.33:0/2220367354 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7cb404d020 con 0x7f7ccc102a30 2024-03-19T02:52:40.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.877+0000 7f7cd1fe6640 1 -- 172.21.15.33:0/2220367354 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ccc02d1f0 con 0x7f7ccc102a30 2024-03-19T02:52:40.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.877+0000 7f7cc97fa640 1 -- 172.21.15.33:0/2220367354 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11867 (secure 0 0 0) 0x7f7cb40be330 con 0x7f7ccc102a30 2024-03-19T02:52:40.879 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:52:40.879 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:50:52.201798+0000","modified":"2024-03-19T02:51:19.076317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24488,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/3731232169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":3731232169},{"type":"v1","addr":"172.21.15.96:6839","nonce":3731232169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24457,24509,24488]},"id":1}]} 2024-03-19T02:52:40.879 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 24 2024-03-19T02:52:40.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.880+0000 7f7cd1fe6640 1 -- 172.21.15.33:0/2220367354 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7cb80a2890 msgr2=0x7f7cb80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:40.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.880+0000 7f7cd1fe6640 1 --2- 172.21.15.33:0/2220367354 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7cb80a2890 0x7f7cb80a4d50 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc102890 tx=0x7f7cbc005f90 comp rx=0 tx=0).stop 2024-03-19T02:52:40.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.880+0000 7f7cd1fe6640 1 -- 172.21.15.33:0/2220367354 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ccc102a30 msgr2=0x7f7ccc197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:40.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.880+0000 7f7cd1fe6640 1 --2- 172.21.15.33:0/2220367354 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ccc102a30 0x7f7ccc197f90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f7cb4006010 tx=0x7f7cb4005e40 comp rx=0 tx=0).stop 2024-03-19T02:52:40.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.880+0000 7f7cd1fe6640 1 -- 172.21.15.33:0/2220367354 shutdown_connections 2024-03-19T02:52:40.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.880+0000 7f7cd1fe6640 1 --2- 172.21.15.33:0/2220367354 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7cb80a2890 0x7f7cb80a4d50 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.880+0000 7f7cd1fe6640 1 --2- 172.21.15.33:0/2220367354 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7ccc1033f0 0x7f7ccc19f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.880+0000 7f7cd1fe6640 1 --2- 172.21.15.33:0/2220367354 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7ccc102a30 0x7f7ccc197f90 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.880+0000 7f7cd1fe6640 1 --2- 172.21.15.33:0/2220367354 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ccc101840 0x7f7ccc197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.880+0000 7f7cd1fe6640 1 -- 172.21.15.33:0/2220367354 >> 172.21.15.33:0/2220367354 conn(0x7f7ccc0fcdd0 msgr2=0x7f7ccc105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:40.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.880+0000 7f7cd1fe6640 1 -- 172.21.15.33:0/2220367354 shutdown_connections 2024-03-19T02:52:40.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.880+0000 7f7cd1fe6640 1 -- 172.21.15.33:0/2220367354 wait complete. 2024-03-19T02:52:40.892 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:52:40.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.994+0000 7f88b1fa9640 1 -- 172.21.15.33:0/3901012504 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88ac101840 msgr2=0x7f88ac101c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:40.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.994+0000 7f88b1fa9640 1 --2- 172.21.15.33:0/3901012504 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88ac101840 0x7f88ac101c60 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f889c009900 tx=0x7f889c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:52:40.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.994+0000 7f88b1fa9640 1 -- 172.21.15.33:0/3901012504 shutdown_connections 2024-03-19T02:52:40.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.994+0000 7f88b1fa9640 1 --2- 172.21.15.33:0/3901012504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f88ac1033f0 0x7f88ac109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.994+0000 7f88b1fa9640 1 --2- 172.21.15.33:0/3901012504 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88ac102a30 0x7f88ac102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.994+0000 7f88b1fa9640 1 --2- 172.21.15.33:0/3901012504 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88ac101840 0x7f88ac101c60 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.994+0000 7f88b1fa9640 1 -- 172.21.15.33:0/3901012504 >> 172.21.15.33:0/3901012504 conn(0x7f88ac0fcdd0 msgr2=0x7f88ac0ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:40.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.994+0000 7f88b1fa9640 1 -- 172.21.15.33:0/3901012504 shutdown_connections 2024-03-19T02:52:40.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.994+0000 7f88b1fa9640 1 -- 172.21.15.33:0/3901012504 wait complete. 2024-03-19T02:52:40.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.994+0000 7f88b1fa9640 1 Processor -- start 2024-03-19T02:52:40.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.995+0000 7f88b1fa9640 1 -- start start 2024-03-19T02:52:40.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.995+0000 7f88b1fa9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88ac101840 0x7f88ac197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:40.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.995+0000 7f88b1fa9640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f88ac102a30 0x7f88ac197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:40.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.995+0000 7f88b1fa9640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88ac1033f0 0x7f88ac19f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:40.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.995+0000 7f88b1fa9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac10d370 con 0x7f88ac101840 2024-03-19T02:52:40.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.995+0000 7f88b1fa9640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac10d1f0 con 0x7f88ac1033f0 2024-03-19T02:52:40.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.995+0000 7f88b1fa9640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac10d4f0 con 0x7f88ac102a30 2024-03-19T02:52:40.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.995+0000 7f88ab7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88ac101840 0x7f88ac197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:40.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.995+0000 7f88ab7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88ac101840 0x7f88ac197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41618/0 (socket says 172.21.15.33:41618) 2024-03-19T02:52:40.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.995+0000 7f88ab7fe640 1 -- 172.21.15.33:0/1417585748 learned_addr learned my addr 172.21.15.33:0/1417585748 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:40.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.995+0000 7f88abfff640 1 --2- 172.21.15.33:0/1417585748 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88ac1033f0 0x7f88ac19f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:40.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.995+0000 7f88aaffd640 1 --2- 172.21.15.33:0/1417585748 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f88ac102a30 0x7f88ac197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:40.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.995+0000 7f88ab7fe640 1 -- 172.21.15.33:0/1417585748 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f88ac102a30 msgr2=0x7f88ac197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:40.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.995+0000 7f88ab7fe640 1 --2- 172.21.15.33:0/1417585748 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f88ac102a30 0x7f88ac197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.995+0000 7f88ab7fe640 1 -- 172.21.15.33:0/1417585748 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88ac1033f0 msgr2=0x7f88ac19f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:40.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.995+0000 7f88ab7fe640 1 --2- 172.21.15.33:0/1417585748 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88ac1033f0 0x7f88ac19f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:40.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.995+0000 7f88ab7fe640 1 -- 172.21.15.33:0/1417585748 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88ac19f690 con 0x7f88ac101840 2024-03-19T02:52:40.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.995+0000 7f88ab7fe640 1 --2- 172.21.15.33:0/1417585748 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88ac101840 0x7f88ac197a70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f889c000c00 tx=0x7f889c005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:40.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.996+0000 7f88a8ff9640 1 -- 172.21.15.33:0/1417585748 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f889c043020 con 0x7f88ac101840 2024-03-19T02:52:40.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.996+0000 7f88b1fa9640 1 -- 172.21.15.33:0/1417585748 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88ac19f920 con 0x7f88ac101840 2024-03-19T02:52:40.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.996+0000 7f88a8ff9640 1 -- 172.21.15.33:0/1417585748 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f889c004350 con 0x7f88ac101840 2024-03-19T02:52:40.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.996+0000 7f88a8ff9640 1 -- 172.21.15.33:0/1417585748 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f889c003b80 con 0x7f88ac101840 2024-03-19T02:52:40.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.996+0000 7f88b1fa9640 1 -- 172.21.15.33:0/1417585748 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88ac19fe70 con 0x7f88ac101840 2024-03-19T02:52:41.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.997+0000 7f88b1fa9640 1 -- 172.21.15.33:0/1417585748 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88ac068a60 con 0x7f88ac101840 2024-03-19T02:52:41.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.998+0000 7f88a8ff9640 1 -- 172.21.15.33:0/1417585748 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f889c046020 con 0x7f88ac101840 2024-03-19T02:52:41.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.998+0000 7f88a8ff9640 1 --2- 172.21.15.33:0/1417585748 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f88940a2890 0x7f88940a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.998+0000 7f88aaffd640 1 --2- 172.21.15.33:0/1417585748 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f88940a2890 0x7f88940a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.999+0000 7f88a8ff9640 1 -- 172.21.15.33:0/1417585748 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9150+0+0 (secure 0 0 0) 0x7f889c0f0f90 con 0x7f88ac101840 2024-03-19T02:52:41.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:40.999+0000 7f88aaffd640 1 --2- 172.21.15.33:0/1417585748 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f88940a2890 0x7f88940a4d50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f88a00095f0 tx=0x7f88a0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:41.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.003+0000 7f88a8ff9640 1 -- 172.21.15.33:0/1417585748 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f889c04d020 con 0x7f88ac101840 2024-03-19T02:52:41.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.130+0000 7f88a8ff9640 1 -- 172.21.15.33:0/1417585748 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f889c0b9bc0 con 0x7f88ac101840 2024-03-19T02:52:41.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.194+0000 7f88b1fa9640 1 -- 172.21.15.33:0/1417585748 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f88ac108bb0 con 0x7f88ac101840 2024-03-19T02:52:41.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.196+0000 7f88a8ff9640 1 -- 172.21.15.33:0/1417585748 <== mon.0 v2:172.21.15.33:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) ==== 74+0+24663 (secure 0 0 0) 0x7f88ac108bb0 con 0x7f88ac101840 2024-03-19T02:52:41.197 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:52:41.197 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":97,"fsid":"2b7a8938-e59a-11ee-95c9-87774f69a715","created":"2024-03-19T02:42:37.912179+0000","modified":"2024-03-19T02:52:41.116227+0000","last_up_change":"2024-03-19T02:50:01.539392+0000","last_in_change":"2024-03-19T02:49:40.534415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:46:09.963737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:50:50.069641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:50:51.381064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:50:55.476667+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"090a89ab-6b7b-4317-8823-178d4212b2f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6803","nonce":1633124995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6805","nonce":1633124995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6809","nonce":1633124995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6807","nonce":1633124995}]},"public_addr":"172.21.15.33:6803/1633124995","cluster_addr":"172.21.15.33:6805/1633124995","heartbeat_back_addr":"172.21.15.33:6809/1633124995","heartbeat_front_addr":"172.21.15.33:6807/1633124995","state":["exists","up"]},{"osd":1,"uuid":"f50eec55-1956-4486-8593-f2c8fa610e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6801","nonce":340144398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6803","nonce":340144398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6807","nonce":340144398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6805","nonce":340144398}]},"public_addr":"172.21.15.96:6801/340144398","cluster_addr":"172.21.15.96:6803/340144398","heartbeat_back_addr":"172.21.15.96:6807/340144398","heartbeat_front_addr":"172.21.15.96:6805/340144398","state":["exists","up"]},{"osd":2,"uuid":"893a2d62-80d9-4e5e-bc6a-a1c8c681d652","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6801","nonce":3388743876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6803","nonce":3388743876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6807","nonce":3388743876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6805","nonce":3388743876}]},"public_addr":"172.21.15.155:6801/3388743876","cluster_addr":"172.21.15.155:6803/3388743876","heartbeat_back_addr":"172.21.15.155:6807/3388743876","heartbeat_front_addr":"172.21.15.155:6805/3388743876","state":["exists","up"]},{"osd":3,"uuid":"b07e0482-4281-4d32-9dfe-9995d7246175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6811","nonce":677007171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6813","nonce":677007171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6817","nonce":677007171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6815","nonce":677007171}]},"public_addr":"172.21.15.33:6811/677007171","cluster_addr":"172.21.15.33:6813/677007171","heartbeat_back_addr":"172.21.15.33:6817/677007171","heartbeat_front_addr":"172.21.15.33:6815/677007171","state":["exists","up"]},{"osd":4,"uuid":"a988384b-af6b-47c6-8c07-5c9673acee45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6809","nonce":3106158554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6811","nonce":3106158554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6815","nonce":3106158554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6813","nonce":3106158554}]},"public_addr":"172.21.15.96:6809/3106158554","cluster_addr":"172.21.15.96:6811/3106158554","heartbeat_back_addr":"172.21.15.96:6815/3106158554","heartbeat_front_addr":"172.21.15.96:6813/3106158554","state":["exists","up"]},{"osd":5,"uuid":"198dc992-96f8-4c52-96ee-1fd5de148bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6809","nonce":327293324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6811","nonce":327293324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6815","nonce":327293324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6813","nonce":327293324}]},"public_addr":"172.21.15.155:6809/327293324","cluster_addr":"172.21.15.155:6811/327293324","heartbeat_back_addr":"172.21.15.155:6815/327293324","heartbeat_front_addr":"172.21.15.155:6813/327293324","state":["exists","up"]},{"osd":6,"uuid":"a7278ec0-ab2e-49ed-be53-84d8b9e24177","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6819","nonce":873871509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6821","nonce":873871509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6825","nonce":873871509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6823","nonce":873871509}]},"public_addr":"172.21.15.33:6819/873871509","cluster_addr":"172.21.15.33:6821/873871509","heartbeat_back_addr":"172.21.15.33:6825/873871509","heartbeat_front_addr":"172.21.15.33:6823/873871509","state":["exists","up"]},{"osd":7,"uuid":"727e133d-aa24-4ffc-90e2-266eb2aab8c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6817","nonce":2334123891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6819","nonce":2334123891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6823","nonce":2334123891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6821","nonce":2334123891}]},"public_addr":"172.21.15.96:6817/2334123891","cluster_addr":"172.21.15.96:6819/2334123891","heartbeat_back_addr":"172.21.15.96:6823/2334123891","heartbeat_front_addr":"172.21.15.96:6821/2334123891","state":["exists","up"]},{"osd":8,"uuid":"bb20eec0-39e5-4b86-9db5-486b1840cdad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6817","nonce":3498516903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6819","nonce":3498516903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6823","nonce":3498516903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6821","nonce":3498516903}]},"public_addr":"172.21.15.155:6817/3498516903","cluster_addr":"172.21.15.155:6819/3498516903","heartbeat_back_addr":"172.21.15.155:6823/3498516903","heartbeat_front_addr":"172.21.15.155:6821/3498516903","state":["exists","up"]},{"osd":9,"uuid":"179c6228-3e23-4f65-91a6-1ef0005d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6827","nonce":760777213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6829","nonce":760777213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6833","nonce":760777213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6831","nonce":760777213}]},"public_addr":"172.21.15.33:6827/760777213","cluster_addr":"172.21.15.33:6829/760777213","heartbeat_back_addr":"172.21.15.33:6833/760777213","heartbeat_front_addr":"172.21.15.33:6831/760777213","state":["exists","up"]},{"osd":10,"uuid":"be17b478-3c2e-41fb-bede-958333acdf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6825","nonce":1404898482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6827","nonce":1404898482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6831","nonce":1404898482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6829","nonce":1404898482}]},"public_addr":"172.21.15.96:6825/1404898482","cluster_addr":"172.21.15.96:6827/1404898482","heartbeat_back_addr":"172.21.15.96:6831/1404898482","heartbeat_front_addr":"172.21.15.96:6829/1404898482","state":["exists","up"]},{"osd":11,"uuid":"c0903ed2-cc1e-4c18-8731-1f0ee4dd0457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6825","nonce":792352827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6827","nonce":792352827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6831","nonce":792352827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6829","nonce":792352827}]},"public_addr":"172.21.15.155:6825/792352827","cluster_addr":"172.21.15.155:6827/792352827","heartbeat_back_addr":"172.21.15.155:6831/792352827","heartbeat_front_addr":"172.21.15.155:6829/792352827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:17.942801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:42.109449+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:05.981599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:32.069574+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:57.431015+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:22.448592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:48.526515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:14.498352+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:40.175426+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:06.364006+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:32.853161+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:59.073538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.9","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:6801/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:6800/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/1067768047":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/729379453":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2586588742":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/2932814165":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6801/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2212564963":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6800/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/3275126058":"2024-03-20T02:43:39.622218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:52:41.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.198+0000 7f88b1fa9640 1 -- 172.21.15.33:0/1417585748 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f88940a2890 msgr2=0x7f88940a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:41.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.198+0000 7f88b1fa9640 1 --2- 172.21.15.33:0/1417585748 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f88940a2890 0x7f88940a4d50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f88a00095f0 tx=0x7f88a0005f90 comp rx=0 tx=0).stop 2024-03-19T02:52:41.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.198+0000 7f88b1fa9640 1 -- 172.21.15.33:0/1417585748 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88ac101840 msgr2=0x7f88ac197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:41.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.198+0000 7f88b1fa9640 1 --2- 172.21.15.33:0/1417585748 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88ac101840 0x7f88ac197a70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f889c000c00 tx=0x7f889c005dc0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.198+0000 7f88b1fa9640 1 -- 172.21.15.33:0/1417585748 shutdown_connections 2024-03-19T02:52:41.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.198+0000 7f88b1fa9640 1 --2- 172.21.15.33:0/1417585748 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f88940a2890 0x7f88940a4d50 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.198+0000 7f88b1fa9640 1 --2- 172.21.15.33:0/1417585748 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f88ac102a30 0x7f88ac197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.198+0000 7f88b1fa9640 1 --2- 172.21.15.33:0/1417585748 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f88ac1033f0 0x7f88ac19f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.198+0000 7f88b1fa9640 1 --2- 172.21.15.33:0/1417585748 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f88ac101840 0x7f88ac197a70 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.198+0000 7f88b1fa9640 1 -- 172.21.15.33:0/1417585748 >> 172.21.15.33:0/1417585748 conn(0x7f88ac0fcdd0 msgr2=0x7f88ac105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:41.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.198+0000 7f88b1fa9640 1 -- 172.21.15.33:0/1417585748 shutdown_connections 2024-03-19T02:52:41.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.198+0000 7f88b1fa9640 1 -- 172.21.15.33:0/1417585748 wait complete. 2024-03-19T02:52:41.209 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:41 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 8]}]': finished 2024-03-19T02:52:41.209 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:41 smithi033 ceph-mon[31973]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-03-19T02:52:41.209 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:41 smithi033 ceph-mon[31973]: osdmap e96: 12 total, 12 up, 12 in 2024-03-19T02:52:41.209 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:41 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/918737244' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:52:41.209 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:41 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3101531060' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:52:41.209 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:41 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2220367354' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:52:41.211 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:52:41.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.315+0000 7f2dd8a36640 1 -- 172.21.15.33:0/2023305336 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2dd41007c0 msgr2=0x7f2dd4100be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:41.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.315+0000 7f2dd8a36640 1 --2- 172.21.15.33:0/2023305336 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2dd41007c0 0x7f2dd4100be0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc009900 tx=0x7f2dcc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:52:41.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.315+0000 7f2dd8a36640 1 -- 172.21.15.33:0/2023305336 shutdown_connections 2024-03-19T02:52:41.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.315+0000 7f2dd8a36640 1 --2- 172.21.15.33:0/2023305336 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2dd4102370 0x7f2dd4108c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.315+0000 7f2dd8a36640 1 --2- 172.21.15.33:0/2023305336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2dd41019b0 0x7f2dd4101e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.315+0000 7f2dd8a36640 1 --2- 172.21.15.33:0/2023305336 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2dd41007c0 0x7f2dd4100be0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.315+0000 7f2dd8a36640 1 -- 172.21.15.33:0/2023305336 >> 172.21.15.33:0/2023305336 conn(0x7f2dd40fbd50 msgr2=0x7f2dd40fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:41.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.315+0000 7f2dd8a36640 1 -- 172.21.15.33:0/2023305336 shutdown_connections 2024-03-19T02:52:41.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.315+0000 7f2dd8a36640 1 -- 172.21.15.33:0/2023305336 wait complete. 2024-03-19T02:52:41.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.316+0000 7f2dd8a36640 1 Processor -- start 2024-03-19T02:52:41.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.316+0000 7f2dd8a36640 1 -- start start 2024-03-19T02:52:41.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.316+0000 7f2dd8a36640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2dd41007c0 0x7f2dd406a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.316+0000 7f2dd8a36640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2dd41019b0 0x7f2dd419c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.316+0000 7f2dd8a36640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2dd4102370 0x7f2dd419ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.316+0000 7f2dd8a36640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd410c250 con 0x7f2dd41007c0 2024-03-19T02:52:41.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.316+0000 7f2dd8a36640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd410c0d0 con 0x7f2dd4102370 2024-03-19T02:52:41.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.316+0000 7f2dd8a36640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd410c3d0 con 0x7f2dd41019b0 2024-03-19T02:52:41.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.316+0000 7f2dd259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2dd41007c0 0x7f2dd406a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.316+0000 7f2dd259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2dd41007c0 0x7f2dd406a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41648/0 (socket says 172.21.15.33:41648) 2024-03-19T02:52:41.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.316+0000 7f2dd259c640 1 -- 172.21.15.33:0/4239880998 learned_addr learned my addr 172.21.15.33:0/4239880998 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:41.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.316+0000 7f2dd2d9d640 1 --2- 172.21.15.33:0/4239880998 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2dd4102370 0x7f2dd419ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.316+0000 7f2dd1d9b640 1 --2- 172.21.15.33:0/4239880998 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2dd41019b0 0x7f2dd419c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.316+0000 7f2dd259c640 1 -- 172.21.15.33:0/4239880998 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2dd41019b0 msgr2=0x7f2dd419c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:41.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.316+0000 7f2dd259c640 1 --2- 172.21.15.33:0/4239880998 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2dd41019b0 0x7f2dd419c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.316+0000 7f2dd259c640 1 -- 172.21.15.33:0/4239880998 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2dd4102370 msgr2=0x7f2dd419ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:41.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.316+0000 7f2dd259c640 1 --2- 172.21.15.33:0/4239880998 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2dd4102370 0x7f2dd419ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.316+0000 7f2dd259c640 1 -- 172.21.15.33:0/4239880998 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dd419f6c0 con 0x7f2dd41007c0 2024-03-19T02:52:41.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.316+0000 7f2dd259c640 1 --2- 172.21.15.33:0/4239880998 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2dd41007c0 0x7f2dd406a970 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc000c00 tx=0x7f2dcc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:41.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.317+0000 7f2dcb7fe640 1 -- 172.21.15.33:0/4239880998 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc043020 con 0x7f2dd41007c0 2024-03-19T02:52:41.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.317+0000 7f2dcb7fe640 1 -- 172.21.15.33:0/4239880998 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2dcc004aa0 con 0x7f2dd41007c0 2024-03-19T02:52:41.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.317+0000 7f2dd8a36640 1 -- 172.21.15.33:0/4239880998 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dd419f950 con 0x7f2dd41007c0 2024-03-19T02:52:41.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.317+0000 7f2dcb7fe640 1 -- 172.21.15.33:0/4239880998 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc003d00 con 0x7f2dd41007c0 2024-03-19T02:52:41.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.317+0000 7f2dd8a36640 1 -- 172.21.15.33:0/4239880998 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dd419fea0 con 0x7f2dd41007c0 2024-03-19T02:52:41.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.318+0000 7f2dd8a36640 1 -- 172.21.15.33:0/4239880998 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dd4107c50 con 0x7f2dd41007c0 2024-03-19T02:52:41.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.319+0000 7f2dcb7fe640 1 -- 172.21.15.33:0/4239880998 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2dcc046020 con 0x7f2dd41007c0 2024-03-19T02:52:41.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.319+0000 7f2dcb7fe640 1 --2- 172.21.15.33:0/4239880998 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2dc00a2800 0x7f2dc00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.319+0000 7f2dd1d9b640 1 --2- 172.21.15.33:0/4239880998 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2dc00a2800 0x7f2dc00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.319+0000 7f2dcb7fe640 1 -- 172.21.15.33:0/4239880998 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f2dcc0f10f0 con 0x7f2dd41007c0 2024-03-19T02:52:41.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.320+0000 7f2dd1d9b640 1 --2- 172.21.15.33:0/4239880998 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2dc00a2800 0x7f2dc00a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2dd406a6e0 tx=0x7f2dbc00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:41.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.324+0000 7f2dcb7fe640 1 -- 172.21.15.33:0/4239880998 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2dcc04d020 con 0x7f2dd41007c0 2024-03-19T02:52:41.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:41 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 8]}]': finished 2024-03-19T02:52:41.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:41 smithi096 ceph-mon[35946]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-03-19T02:52:41.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:41 smithi096 ceph-mon[35946]: osdmap e96: 12 total, 12 up, 12 in 2024-03-19T02:52:41.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:41 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/918737244' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:52:41.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:41 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3101531060' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:52:41.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:41 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2220367354' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:52:41.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:41 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 8]}]': finished 2024-03-19T02:52:41.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:41 smithi155 ceph-mon[35804]: from='mgr.14150 172.21.15.33:0/3296104730' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-03-19T02:52:41.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:41 smithi155 ceph-mon[35804]: osdmap e96: 12 total, 12 up, 12 in 2024-03-19T02:52:41.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:41 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/918737244' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:52:41.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:41 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3101531060' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:52:41.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:41 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2220367354' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:52:41.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.545+0000 7f2dd8a36640 1 -- 172.21.15.33:0/4239880998 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2dd4062420 con 0x7f2dd41007c0 2024-03-19T02:52:41.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.545+0000 7f2dcb7fe640 1 -- 172.21.15.33:0/4239880998 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11867 (secure 0 0 0) 0x7f2dcc0bde00 con 0x7f2dd41007c0 2024-03-19T02:52:41.550 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:52:41.550 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:50:52.201798+0000","modified":"2024-03-19T02:51:19.076317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24488,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/3731232169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":3731232169},{"type":"v1","addr":"172.21.15.96:6839","nonce":3731232169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24457,24509,24488]},"id":1}]} 2024-03-19T02:52:41.551 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 24 2024-03-19T02:52:41.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.548+0000 7f2dd8a36640 1 -- 172.21.15.33:0/4239880998 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2dc00a2800 msgr2=0x7f2dc00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:41.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.548+0000 7f2dd8a36640 1 --2- 172.21.15.33:0/4239880998 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2dc00a2800 0x7f2dc00a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2dd406a6e0 tx=0x7f2dbc00b410 comp rx=0 tx=0).stop 2024-03-19T02:52:41.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.548+0000 7f2dd8a36640 1 -- 172.21.15.33:0/4239880998 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2dd41007c0 msgr2=0x7f2dd406a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:41.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.548+0000 7f2dd8a36640 1 --2- 172.21.15.33:0/4239880998 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2dd41007c0 0x7f2dd406a970 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc000c00 tx=0x7f2dcc004270 comp rx=0 tx=0).stop 2024-03-19T02:52:41.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.548+0000 7f2dd8a36640 1 -- 172.21.15.33:0/4239880998 shutdown_connections 2024-03-19T02:52:41.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.548+0000 7f2dd8a36640 1 --2- 172.21.15.33:0/4239880998 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2dc00a2800 0x7f2dc00a4cc0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.548+0000 7f2dd8a36640 1 --2- 172.21.15.33:0/4239880998 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2dd41019b0 0x7f2dd419c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.548+0000 7f2dd8a36640 1 --2- 172.21.15.33:0/4239880998 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2dd4102370 0x7f2dd419ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.548+0000 7f2dd8a36640 1 --2- 172.21.15.33:0/4239880998 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2dd41007c0 0x7f2dd406a970 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.548+0000 7f2dd8a36640 1 -- 172.21.15.33:0/4239880998 >> 172.21.15.33:0/4239880998 conn(0x7f2dd40fbd50 msgr2=0x7f2dd4104e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:41.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.548+0000 7f2dd8a36640 1 -- 172.21.15.33:0/4239880998 shutdown_connections 2024-03-19T02:52:41.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.548+0000 7f2dd8a36640 1 -- 172.21.15.33:0/4239880998 wait complete. 2024-03-19T02:52:41.560 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-19T02:50:52.201798+0000', 'modified': '2024-03-19T02:51:19.076317+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24509, 'mds_1': 24488, 'mds_2': 24457}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24427': {'gid': 24427, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.155:6835/1274854977', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6834', 'nonce': 1274854977}, {'type': 'v1', 'addr': '172.21.15.155:6835', 'nonce': 1274854977}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24436': {'gid': 24436, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.155:6837/1429349330', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6836', 'nonce': 1429349330}, {'type': 'v1', 'addr': '172.21.15.155:6837', 'nonce': 1429349330}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24455': {'gid': 24455, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.96:6835/3551193617', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6834', 'nonce': 3551193617}, {'type': 'v1', 'addr': '172.21.15.96:6835', 'nonce': 3551193617}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24457': {'gid': 24457, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.155:6839/1840534530', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6838', 'nonce': 1840534530}, {'type': 'v1', 'addr': '172.21.15.155:6839', 'nonce': 1840534530}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24488': {'gid': 24488, 'name': 'h', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.96:6837/522588113', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6836', 'nonce': 522588113}, {'type': 'v1', 'addr': '172.21.15.96:6837', 'nonce': 522588113}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24509': {'gid': 24509, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.96:6839/3731232169', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6838', 'nonce': 3731232169}, {'type': 'v1', 'addr': '172.21.15.96:6839', 'nonce': 3731232169}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24509, 'qdb_cluster': [24457, 24509, 24488]} 2024-03-19T02:52:41.560 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-19T02:52:41.560 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-19T02:52:41.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.663+0000 7f3ad8953640 1 -- 172.21.15.33:0/1538455633 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ad4100720 msgr2=0x7f3ad4100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:41.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.663+0000 7f3ad8953640 1 --2- 172.21.15.33:0/1538455633 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ad4100720 0x7f3ad4100b40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3acc009900 tx=0x7f3acc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:52:41.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.664+0000 7f3ad8953640 1 -- 172.21.15.33:0/1538455633 shutdown_connections 2024-03-19T02:52:41.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.664+0000 7f3ad8953640 1 --2- 172.21.15.33:0/1538455633 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ad41022d0 0x7f3ad4108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.664+0000 7f3ad8953640 1 --2- 172.21.15.33:0/1538455633 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ad4101910 0x7f3ad4101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.664+0000 7f3ad8953640 1 --2- 172.21.15.33:0/1538455633 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ad4100720 0x7f3ad4100b40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.664+0000 7f3ad8953640 1 -- 172.21.15.33:0/1538455633 >> 172.21.15.33:0/1538455633 conn(0x7f3ad40fbcb0 msgr2=0x7f3ad40fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:41.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.664+0000 7f3ad8953640 1 -- 172.21.15.33:0/1538455633 shutdown_connections 2024-03-19T02:52:41.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.664+0000 7f3ad8953640 1 -- 172.21.15.33:0/1538455633 wait complete. 2024-03-19T02:52:41.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.664+0000 7f3ad8953640 1 Processor -- start 2024-03-19T02:52:41.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.664+0000 7f3ad8953640 1 -- start start 2024-03-19T02:52:41.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.664+0000 7f3ad8953640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ad4100720 0x7f3ad4197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.664+0000 7f3ad8953640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ad4101910 0x7f3ad4198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.664+0000 7f3ad8953640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ad41022d0 0x7f3ad419f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.664+0000 7f3ad8953640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad410d330 con 0x7f3ad4101910 2024-03-19T02:52:41.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.664+0000 7f3ad8953640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad410d1b0 con 0x7f3ad4100720 2024-03-19T02:52:41.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.664+0000 7f3ad8953640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad410d4b0 con 0x7f3ad41022d0 2024-03-19T02:52:41.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.665+0000 7f3ad1d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ad4101910 0x7f3ad4198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.665+0000 7f3ad259c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ad4100720 0x7f3ad4197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.665+0000 7f3ad1d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ad4101910 0x7f3ad4198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41668/0 (socket says 172.21.15.33:41668) 2024-03-19T02:52:41.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.665+0000 7f3ad1d9b640 1 -- 172.21.15.33:0/2075524787 learned_addr learned my addr 172.21.15.33:0/2075524787 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:41.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.665+0000 7f3ad2d9d640 1 --2- 172.21.15.33:0/2075524787 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ad41022d0 0x7f3ad419f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.665+0000 7f3ad259c640 1 -- 172.21.15.33:0/2075524787 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ad41022d0 msgr2=0x7f3ad419f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:41.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.665+0000 7f3ad259c640 1 --2- 172.21.15.33:0/2075524787 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ad41022d0 0x7f3ad419f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.665+0000 7f3ad259c640 1 -- 172.21.15.33:0/2075524787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ad4101910 msgr2=0x7f3ad4198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:41.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.665+0000 7f3ad259c640 1 --2- 172.21.15.33:0/2075524787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ad4101910 0x7f3ad4198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.665+0000 7f3ad259c640 1 -- 172.21.15.33:0/2075524787 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ad419f6e0 con 0x7f3ad4100720 2024-03-19T02:52:41.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.665+0000 7f3ad1d9b640 1 --2- 172.21.15.33:0/2075524787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ad4101910 0x7f3ad4198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:41.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.665+0000 7f3ad259c640 1 --2- 172.21.15.33:0/2075524787 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ad4100720 0x7f3ad4197ac0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3acc036040 tx=0x7f3acc005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:41.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.665+0000 7f3ac37fe640 1 -- 172.21.15.33:0/2075524787 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3acc043020 con 0x7f3ad4100720 2024-03-19T02:52:41.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.665+0000 7f3ad8953640 1 -- 172.21.15.33:0/2075524787 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ad419f970 con 0x7f3ad4100720 2024-03-19T02:52:41.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.665+0000 7f3ac37fe640 1 -- 172.21.15.33:0/2075524787 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3acc004350 con 0x7f3ad4100720 2024-03-19T02:52:41.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.665+0000 7f3ac37fe640 1 -- 172.21.15.33:0/2075524787 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3acc003ab0 con 0x7f3ad4100720 2024-03-19T02:52:41.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.665+0000 7f3ad8953640 1 -- 172.21.15.33:0/2075524787 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ad419fee0 con 0x7f3ad4100720 2024-03-19T02:52:41.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.668+0000 7f3ac37fe640 1 -- 172.21.15.33:0/2075524787 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3acc046020 con 0x7f3ad4100720 2024-03-19T02:52:41.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.668+0000 7f3adb61e740 1 Processor -- start 2024-03-19T02:52:41.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.668+0000 7f3adb61e740 1 -- start start 2024-03-19T02:52:41.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.668+0000 7f3adb61e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0c4f3aff0 0x55c0c4f3b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.668+0000 7f3adb61e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0c4f3b9e0 0x55c0c4f404e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.668+0000 7f3adb61e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0c4f40b50 0x55c0c4f42fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.668+0000 7f3adb61e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c0c4e9fde0 con 0x55c0c4f40b50 2024-03-19T02:52:41.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.668+0000 7f3adb61e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c0c4e48c80 con 0x55c0c4f3aff0 2024-03-19T02:52:41.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.668+0000 7f3adb61e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c0c4e9be80 con 0x55c0c4f3b9e0 2024-03-19T02:52:41.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.668+0000 7f3ad2d9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0c4f3aff0 0x55c0c4f3b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.668+0000 7f3ad1d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0c4f40b50 0x55c0c4f42fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.668+0000 7f3ad2d9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0c4f3aff0 0x55c0c4f3b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:55312/0 (socket says 172.21.15.33:55312) 2024-03-19T02:52:41.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.668+0000 7f3ad2d9d640 1 -- 172.21.15.33:0/1647020490 learned_addr learned my addr 172.21.15.33:0/1647020490 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:41.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.668+0000 7f3ad259c640 1 --2- 172.21.15.33:0/1647020490 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0c4f3b9e0 0x55c0c4f404e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.668+0000 7f3ac37fe640 1 --2- 172.21.15.33:0/2075524787 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f3abc0a2800 0x7f3abc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.668+0000 7f3ad1d9b640 1 -- 172.21.15.33:0/1647020490 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0c4f3b9e0 msgr2=0x55c0c4f404e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:41.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.668+0000 7f3ad1d9b640 1 --2- 172.21.15.33:0/1647020490 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0c4f3b9e0 0x55c0c4f404e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.668+0000 7f3ad1d9b640 1 -- 172.21.15.33:0/1647020490 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0c4f3aff0 msgr2=0x55c0c4f3b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:41.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.669+0000 7f3ad1d9b640 1 --2- 172.21.15.33:0/1647020490 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0c4f3aff0 0x55c0c4f3b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.669+0000 7f3ad1d9b640 1 -- 172.21.15.33:0/1647020490 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0c4e5afb0 con 0x55c0c4f40b50 2024-03-19T02:52:41.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.669+0000 7f3ad259c640 1 --2- 172.21.15.33:0/1647020490 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0c4f3b9e0 0x55c0c4f404e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:52:41.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.669+0000 7f3ad2d9d640 1 --2- 172.21.15.33:0/1647020490 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0c4f3aff0 0x55c0c4f3b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:41.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.669+0000 7f3ac37fe640 1 -- 172.21.15.33:0/2075524787 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f3acc0f0ed0 con 0x7f3ad4100720 2024-03-19T02:52:41.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.669+0000 7f3ad2d9d640 1 --2- 172.21.15.33:0/2075524787 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f3abc0a2800 0x7f3abc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.669+0000 7f3ad1d9b640 1 --2- 172.21.15.33:0/1647020490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0c4f40b50 0x55c0c4f42fe0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3ac80114c0 tx=0x7f3ac8011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b061bc497f9ccbdc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:41.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.669+0000 7f3ac0ff9640 1 -- 172.21.15.33:0/1647020490 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac8004030 con 0x55c0c4f40b50 2024-03-19T02:52:41.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.669+0000 7f3ac0ff9640 1 -- 172.21.15.33:0/1647020490 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3ac80035f0 con 0x55c0c4f40b50 2024-03-19T02:52:41.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.669+0000 7f3ac0ff9640 1 -- 172.21.15.33:0/1647020490 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac800f800 con 0x55c0c4f40b50 2024-03-19T02:52:41.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.670+0000 7f3adb61e740 1 -- 172.21.15.33:0/1647020490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0c4f40b50 msgr2=0x55c0c4f42fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:41.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.670+0000 7f3adb61e740 1 --2- 172.21.15.33:0/1647020490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0c4f40b50 0x55c0c4f42fe0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3ac80114c0 tx=0x7f3ac8011980 comp rx=0 tx=0).stop 2024-03-19T02:52:41.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.670+0000 7f3ad2d9d640 1 --2- 172.21.15.33:0/2075524787 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f3abc0a2800 0x7f3abc0a4cc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x55c0c4f3c240 tx=0x7f3ac4004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:41.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.670+0000 7f3adb61e740 1 -- 172.21.15.33:0/1647020490 shutdown_connections 2024-03-19T02:52:41.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.670+0000 7f3adb61e740 1 --2- 172.21.15.33:0/1647020490 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0c4f3b9e0 0x55c0c4f404e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.670+0000 7f3adb61e740 1 --2- 172.21.15.33:0/1647020490 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0c4f40b50 0x55c0c4f42fe0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.670+0000 7f3adb61e740 1 --2- 172.21.15.33:0/1647020490 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0c4f3aff0 0x55c0c4f3b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.670+0000 7f3adb61e740 1 -- 172.21.15.33:0/1647020490 >> 172.21.15.33:0/1647020490 conn(0x55c0c4f36560 msgr2=0x55c0c4f36960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:41.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.670+0000 7f3adb61e740 1 -- 172.21.15.33:0/1647020490 shutdown_connections 2024-03-19T02:52:41.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.670+0000 7f3adb61e740 1 -- 172.21.15.33:0/1647020490 wait complete. 2024-03-19T02:52:41.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.670+0000 7f3adb61e740 1 -- start start 2024-03-19T02:52:41.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.670+0000 7f3adb61e740 1 Processor -- start 2024-03-19T02:52:41.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.670+0000 7f3adb61e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0c4f3aff0 0x55c0c4f45480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.670+0000 7f3adb61e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0c4f3b9e0 0x55c0c4f459c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.670+0000 7f3adb61e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0c4f40b50 0x55c0c4f4ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.670+0000 7f3adb61e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c0c4f43b00 con 0x55c0c4f40b50 2024-03-19T02:52:41.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.670+0000 7f3adb61e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c0c4e5bfc0 con 0x55c0c4f3b9e0 2024-03-19T02:52:41.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.670+0000 7f3adb61e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c0c4f43c80 con 0x55c0c4f3aff0 2024-03-19T02:52:41.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.670+0000 7f3adb61e740 10 client.0 _ll_register_callbacks cb 0x55c0c4e97a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:41.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.670+0000 7f3ad2d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0c4f40b50 0x55c0c4f4ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.670+0000 7f3ad1d9b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0c4f3b9e0 0x55c0c4f459c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.670+0000 7f3ad2d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0c4f40b50 0x55c0c4f4ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41672/0 (socket says 172.21.15.33:41672) 2024-03-19T02:52:41.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.670+0000 7f3ad2d9d640 1 -- 172.21.15.33:0/971131361 learned_addr learned my addr 172.21.15.33:0/971131361 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:41.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.670+0000 7f3ad259c640 1 --2- 172.21.15.33:0/971131361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0c4f3aff0 0x55c0c4f45480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.671+0000 7f3ad2d9d640 1 -- 172.21.15.33:0/971131361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0c4f3aff0 msgr2=0x55c0c4f45480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:41.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.671+0000 7f3ad2d9d640 1 --2- 172.21.15.33:0/971131361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0c4f3aff0 0x55c0c4f45480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.671+0000 7f3ad2d9d640 1 -- 172.21.15.33:0/971131361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0c4f3b9e0 msgr2=0x55c0c4f459c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:41.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.671+0000 7f3ad2d9d640 1 --2- 172.21.15.33:0/971131361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0c4f3b9e0 0x55c0c4f459c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.671+0000 7f3ad2d9d640 1 -- 172.21.15.33:0/971131361 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0c4bb42d0 con 0x55c0c4f40b50 2024-03-19T02:52:41.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.671+0000 7f3ad1d9b640 1 --2- 172.21.15.33:0/971131361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0c4f3b9e0 0x55c0c4f459c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:41.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.671+0000 7f3ad2d9d640 1 --2- 172.21.15.33:0/971131361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0c4f40b50 0x55c0c4f4ca00 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4011a50 tx=0x7f3ac4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc03d7035026c627 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:41.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.671+0000 7f3ab9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:52:41.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.671+0000 7f3ab9ffb640 1 -- 172.21.15.33:0/971131361 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac401e0d0 con 0x55c0c4f40b50 2024-03-19T02:52:41.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.671+0000 7f3ab9ffb640 1 -- 172.21.15.33:0/971131361 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f3ac401e910 con 0x55c0c4f40b50 2024-03-19T02:52:41.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.671+0000 7f3adb61e740 1 -- 172.21.15.33:0/971131361 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ac80035f0 con 0x55c0c4f40b50 2024-03-19T02:52:41.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.671+0000 7f3ab9ffb640 1 -- 172.21.15.33:0/971131361 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3ac401e270 con 0x55c0c4f40b50 2024-03-19T02:52:41.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.671+0000 7f3adb61e740 10 client.15198 fetch_fsmap learned FSMap version 24 2024-03-19T02:52:41.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.671+0000 7f3adb61e740 1 -- 172.21.15.33:0/971131361 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c0c4c7e6b0 con 0x55c0c4f40b50 2024-03-19T02:52:41.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.671+0000 7f3ab9ffb640 1 -- 172.21.15.33:0/971131361 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3ac80035f0 con 0x55c0c4f40b50 2024-03-19T02:52:41.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.671+0000 7f3ab9ffb640 1 -- 172.21.15.33:0/971131361 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f3ac4028420 con 0x55c0c4f40b50 2024-03-19T02:52:41.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.672+0000 7f3adb61e740 10 client.15198 fetch_fsmap finished waiting for FSMap version 24 2024-03-19T02:52:41.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.672+0000 7f3adb61e740 10 client.15198 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-19T02:52:41.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.672+0000 7f3adb61e740 20 client.15198 populate_metadata read hostname 'smithi033' 2024-03-19T02:52:41.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.672+0000 7f3adb61e740 1 --2- 172.21.15.33:0/971131361 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55c0c4f52360 0x55c0c4f537d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.672+0000 7f3adb61e740 4 client.15198 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:52:41.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.672+0000 7f3adb61e740 1 -- 172.21.15.33:0/971131361 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c0c4ead320 con 0x55c0c4f52360 2024-03-19T02:52:41.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.672+0000 7f3ad259c640 1 --2- 172.21.15.33:0/971131361 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55c0c4f52360 0x55c0c4f537d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.672+0000 7f3ab9ffb640 10 client.15198 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T02:52:41.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.672+0000 7f3ad259c640 1 --2- 172.21.15.33:0/971131361 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55c0c4f52360 0x55c0c4f537d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=14cf74d4acc0d9ec server_cookie=76563680d2d185eb in_seq=0 out_seq=0 2024-03-19T02:52:41.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.673+0000 7f3ab9ffb640 1 -- 172.21.15.33:0/971131361 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c0c4ead320 con 0x55c0c4f52360 2024-03-19T02:52:41.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.673+0000 7f3ab9ffb640 10 client.15198 handle_command_reply: tid=0 2024-03-19T02:52:41.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.673+0000 7f3adb61e740 1 client.15198 shutdown 2024-03-19T02:52:41.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.673+0000 7f3adb61e740 1 -- 172.21.15.33:0/971131361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0c4f40b50 msgr2=0x55c0c4f4ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:41.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.673+0000 7f3adb61e740 1 --2- 172.21.15.33:0/971131361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0c4f40b50 0x55c0c4f4ca00 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4011a50 tx=0x7f3ac4004ab0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.673+0000 7f3adb61e740 1 -- 172.21.15.33:0/971131361 shutdown_connections 2024-03-19T02:52:41.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.673+0000 7f3adb61e740 1 --2- 172.21.15.33:0/971131361 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55c0c4f52360 0x55c0c4f537d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.673+0000 7f3adb61e740 1 --2- 172.21.15.33:0/971131361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0c4f40b50 0x55c0c4f4ca00 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.673+0000 7f3adb61e740 1 --2- 172.21.15.33:0/971131361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0c4f3b9e0 0x55c0c4f459c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.673+0000 7f3ab9ffb640 0 client.15198 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T02:52:41.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.673+0000 7f3adb61e740 1 --2- 172.21.15.33:0/971131361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0c4f3aff0 0x55c0c4f45480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.673+0000 7f3adb61e740 1 -- 172.21.15.33:0/971131361 >> 172.21.15.33:0/971131361 conn(0x55c0c4f36560 msgr2=0x55c0c4f357e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:41.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.673+0000 7f3adb61e740 1 -- 172.21.15.33:0/971131361 shutdown_connections 2024-03-19T02:52:41.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.673+0000 7f3adb61e740 1 -- 172.21.15.33:0/971131361 wait complete. 2024-03-19T02:52:41.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.673+0000 7f3adb61e740 20 client.15198 trim_cache size 0 max 16384 2024-03-19T02:52:41.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.691+0000 7f3adb61e740 1 Processor -- start 2024-03-19T02:52:41.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.692+0000 7f3adb61e740 1 -- start start 2024-03-19T02:52:41.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.692+0000 7f3adb61e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0c4f3aff0 0x55c0c4f45480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.692+0000 7f3adb61e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0c4f3b9e0 0x55c0c4f4c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.692+0000 7f3adb61e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0c4f40b50 0x55c0c4f459c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.692+0000 7f3adb61e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c0c4f51b10 con 0x55c0c4f3b9e0 2024-03-19T02:52:41.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.692+0000 7f3adb61e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c0c4f521e0 con 0x55c0c4f3aff0 2024-03-19T02:52:41.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.692+0000 7f3adb61e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c0c4f528b0 con 0x55c0c4f40b50 2024-03-19T02:52:41.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.692+0000 7f3ad1d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0c4f3b9e0 0x55c0c4f4c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.692+0000 7f3ad1d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0c4f3b9e0 0x55c0c4f4c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41674/0 (socket says 172.21.15.33:41674) 2024-03-19T02:52:41.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.692+0000 7f3ad1d9b640 1 -- 172.21.15.33:0/2505041413 learned_addr learned my addr 172.21.15.33:0/2505041413 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:41.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.692+0000 7f3ad259c640 1 --2- 172.21.15.33:0/2505041413 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0c4f3aff0 0x55c0c4f45480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.692+0000 7f3ad2d9d640 1 --2- 172.21.15.33:0/2505041413 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0c4f40b50 0x55c0c4f459c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.692+0000 7f3ad1d9b640 1 -- 172.21.15.33:0/2505041413 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0c4f40b50 msgr2=0x55c0c4f459c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:41.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.692+0000 7f3ad1d9b640 1 --2- 172.21.15.33:0/2505041413 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0c4f40b50 0x55c0c4f459c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.692+0000 7f3ad1d9b640 1 -- 172.21.15.33:0/2505041413 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0c4f3aff0 msgr2=0x55c0c4f45480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:41.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.692+0000 7f3ad1d9b640 1 --2- 172.21.15.33:0/2505041413 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0c4f3aff0 0x55c0c4f45480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.692+0000 7f3ad1d9b640 1 -- 172.21.15.33:0/2505041413 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0c4f372d0 con 0x55c0c4f3b9e0 2024-03-19T02:52:41.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.692+0000 7f3ad259c640 1 --2- 172.21.15.33:0/2505041413 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0c4f3aff0 0x55c0c4f45480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:41.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.692+0000 7f3ad1d9b640 1 --2- 172.21.15.33:0/2505041413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0c4f3b9e0 0x55c0c4f4c9c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f3ac80114c0 tx=0x7f3ac8005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9987c7f45c945334 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:41.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.692+0000 7f3abbfff640 1 -- 172.21.15.33:0/2505041413 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac8028070 con 0x55c0c4f3b9e0 2024-03-19T02:52:41.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.692+0000 7f3abbfff640 1 -- 172.21.15.33:0/2505041413 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3ac8009c90 con 0x55c0c4f3b9e0 2024-03-19T02:52:41.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.692+0000 7f3abbfff640 1 -- 172.21.15.33:0/2505041413 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac8011320 con 0x55c0c4f3b9e0 2024-03-19T02:52:41.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.693+0000 7f3adb61e740 1 -- 172.21.15.33:0/2505041413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0c4f3b9e0 msgr2=0x55c0c4f4c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:41.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.693+0000 7f3adb61e740 1 --2- 172.21.15.33:0/2505041413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0c4f3b9e0 0x55c0c4f4c9c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f3ac80114c0 tx=0x7f3ac8005c50 comp rx=0 tx=0).stop 2024-03-19T02:52:41.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.693+0000 7f3adb61e740 1 -- 172.21.15.33:0/2505041413 shutdown_connections 2024-03-19T02:52:41.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.693+0000 7f3adb61e740 1 --2- 172.21.15.33:0/2505041413 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0c4f40b50 0x55c0c4f459c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.693+0000 7f3adb61e740 1 --2- 172.21.15.33:0/2505041413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0c4f3b9e0 0x55c0c4f4c9c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.693+0000 7f3adb61e740 1 --2- 172.21.15.33:0/2505041413 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0c4f3aff0 0x55c0c4f45480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.693+0000 7f3adb61e740 1 -- 172.21.15.33:0/2505041413 >> 172.21.15.33:0/2505041413 conn(0x55c0c4f36560 msgr2=0x55c0c4f3cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:41.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.693+0000 7f3adb61e740 1 -- 172.21.15.33:0/2505041413 shutdown_connections 2024-03-19T02:52:41.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.693+0000 7f3adb61e740 1 -- 172.21.15.33:0/2505041413 wait complete. 2024-03-19T02:52:41.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.694+0000 7f3adb61e740 1 -- start start 2024-03-19T02:52:41.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.694+0000 7f3adb61e740 1 Processor -- start 2024-03-19T02:52:41.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.694+0000 7f3adb61e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0c4f3aff0 0x55c0c4f4a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.694+0000 7f3adb61e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0c4f3b9e0 0x55c0c4fb40b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.694+0000 7f3adb61e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0c4f40b50 0x55c0c4fb6600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.694+0000 7f3adb61e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c0c4f53650 con 0x55c0c4f40b50 2024-03-19T02:52:41.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.694+0000 7f3adb61e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c0c4f52f80 con 0x55c0c4f3aff0 2024-03-19T02:52:41.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.694+0000 7f3adb61e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c0c4f53d20 con 0x55c0c4f3b9e0 2024-03-19T02:52:41.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.694+0000 7f3adb61e740 10 client.0 _ll_register_callbacks cb 0x7f3ac4003ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:41.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.694+0000 7f3ad2d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0c4f40b50 0x55c0c4fb6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.694+0000 7f3ad2d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0c4f40b50 0x55c0c4fb6600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41688/0 (socket says 172.21.15.33:41688) 2024-03-19T02:52:41.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.694+0000 7f3ad2d9d640 1 -- 172.21.15.33:0/3013546165 learned_addr learned my addr 172.21.15.33:0/3013546165 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:41.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.694+0000 7f3ad259c640 1 --2- 172.21.15.33:0/3013546165 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0c4f3aff0 0x55c0c4f4a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.694+0000 7f3ad1d9b640 1 --2- 172.21.15.33:0/3013546165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0c4f3b9e0 0x55c0c4fb40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.694+0000 7f3ad2d9d640 1 -- 172.21.15.33:0/3013546165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0c4f3b9e0 msgr2=0x55c0c4fb40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:41.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.694+0000 7f3ad2d9d640 1 --2- 172.21.15.33:0/3013546165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0c4f3b9e0 0x55c0c4fb40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.694+0000 7f3ad2d9d640 1 -- 172.21.15.33:0/3013546165 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0c4f3aff0 msgr2=0x55c0c4f4a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:41.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.694+0000 7f3ad2d9d640 1 --2- 172.21.15.33:0/3013546165 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0c4f3aff0 0x55c0c4f4a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.694+0000 7f3ad2d9d640 1 -- 172.21.15.33:0/3013546165 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0c4fb6d90 con 0x55c0c4f40b50 2024-03-19T02:52:41.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.694+0000 7f3ad1d9b640 1 --2- 172.21.15.33:0/3013546165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0c4f3b9e0 0x55c0c4fb40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:52:41.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.694+0000 7f3ad2d9d640 1 --2- 172.21.15.33:0/3013546165 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0c4f40b50 0x55c0c4fb6600 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4011a20 tx=0x7f3ac4028650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40e7a84d4e17566e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:41.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.694+0000 7f3ab9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:52:41.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.695+0000 7f3ab9ffb640 1 -- 172.21.15.33:0/3013546165 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac403a020 con 0x55c0c4f40b50 2024-03-19T02:52:41.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.695+0000 7f3ab9ffb640 1 -- 172.21.15.33:0/3013546165 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f3ac401ea20 con 0x55c0c4f40b50 2024-03-19T02:52:41.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.695+0000 7f3adb61e740 1 -- 172.21.15.33:0/3013546165 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ac8009c90 con 0x55c0c4f40b50 2024-03-19T02:52:41.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.695+0000 7f3ab9ffb640 1 -- 172.21.15.33:0/3013546165 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3ac4045050 con 0x55c0c4f40b50 2024-03-19T02:52:41.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.695+0000 7f3adb61e740 10 client.15204 fetch_fsmap learned FSMap version 24 2024-03-19T02:52:41.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.695+0000 7f3adb61e740 1 -- 172.21.15.33:0/3013546165 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c0c4fb8100 con 0x55c0c4f40b50 2024-03-19T02:52:41.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.695+0000 7f3ab9ffb640 1 -- 172.21.15.33:0/3013546165 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3ac8009c90 con 0x55c0c4f40b50 2024-03-19T02:52:41.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.695+0000 7f3ab9ffb640 1 -- 172.21.15.33:0/3013546165 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f3ac40368d0 con 0x55c0c4f40b50 2024-03-19T02:52:41.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.695+0000 7f3adb61e740 10 client.15204 fetch_fsmap finished waiting for FSMap version 24 2024-03-19T02:52:41.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.695+0000 7f3adb61e740 10 client.15204 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-19T02:52:41.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.695+0000 7f3adb61e740 20 client.15204 populate_metadata read hostname 'smithi033' 2024-03-19T02:52:41.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.695+0000 7f3adb61e740 1 --2- 172.21.15.33:0/3013546165 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55c0c4fbcda0 0x55c0c4fbbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.695+0000 7f3adb61e740 4 client.15204 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T02:52:41.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.695+0000 7f3adb61e740 1 -- 172.21.15.33:0/3013546165 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "status"}) -- 0x55c0c4ea7670 con 0x55c0c4fbcda0 2024-03-19T02:52:41.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.695+0000 7f3ad259c640 1 --2- 172.21.15.33:0/3013546165 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55c0c4fbcda0 0x55c0c4fbbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.696+0000 7f3ab9ffb640 10 client.15204 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T02:52:41.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.696+0000 7f3ad259c640 1 --2- 172.21.15.33:0/3013546165 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55c0c4fbcda0 0x55c0c4fbbab0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4b109ac2b1ff9751 server_cookie=1d1f87680d67fd3c in_seq=0 out_seq=0 2024-03-19T02:52:41.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.696+0000 7f3ab9ffb640 1 -- 172.21.15.33:0/3013546165 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55c0c4ea7670 con 0x55c0c4fbcda0 2024-03-19T02:52:41.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.696+0000 7f3ab9ffb640 10 client.15204 handle_command_reply: tid=0 2024-03-19T02:52:41.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.696+0000 7f3adb61e740 1 client.15204 shutdown 2024-03-19T02:52:41.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.696+0000 7f3adb61e740 1 -- 172.21.15.33:0/3013546165 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0c4f40b50 msgr2=0x55c0c4fb6600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:41.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.696+0000 7f3adb61e740 1 --2- 172.21.15.33:0/3013546165 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0c4f40b50 0x55c0c4fb6600 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4011a20 tx=0x7f3ac4028650 comp rx=0 tx=0).stop 2024-03-19T02:52:41.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.696+0000 7f3adb61e740 1 -- 172.21.15.33:0/3013546165 shutdown_connections 2024-03-19T02:52:41.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.696+0000 7f3adb61e740 1 --2- 172.21.15.33:0/3013546165 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55c0c4fbcda0 0x55c0c4fbbab0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.696+0000 7f3adb61e740 1 --2- 172.21.15.33:0/3013546165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0c4f3b9e0 0x55c0c4fb40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.696+0000 7f3ab9ffb640 0 client.15204 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T02:52:41.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.696+0000 7f3adb61e740 1 --2- 172.21.15.33:0/3013546165 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0c4f40b50 0x55c0c4fb6600 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.696+0000 7f3adb61e740 1 --2- 172.21.15.33:0/3013546165 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0c4f3aff0 0x55c0c4f4a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.696+0000 7f3adb61e740 1 -- 172.21.15.33:0/3013546165 >> 172.21.15.33:0/3013546165 conn(0x55c0c4f36560 msgr2=0x55c0c4f45480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:41.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.696+0000 7f3adb61e740 1 -- 172.21.15.33:0/3013546165 shutdown_connections 2024-03-19T02:52:41.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.696+0000 7f3adb61e740 1 -- 172.21.15.33:0/3013546165 wait complete. 2024-03-19T02:52:41.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.696+0000 7f3adb61e740 20 client.15204 trim_cache size 0 max 16384 2024-03-19T02:52:41.705 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:52:41.705 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_fsid": "2b7a8938-e59a-11ee-95c9-87774f69a715", 2024-03-19T02:52:41.705 INFO:teuthology.orchestra.run.smithi033.stdout: "whoami": 2, 2024-03-19T02:52:41.705 INFO:teuthology.orchestra.run.smithi033.stdout: "id": 24457, 2024-03-19T02:52:41.706 INFO:teuthology.orchestra.run.smithi033.stdout: "want_state": "up:active", 2024-03-19T02:52:41.706 INFO:teuthology.orchestra.run.smithi033.stdout: "state": "up:active", 2024-03-19T02:52:41.706 INFO:teuthology.orchestra.run.smithi033.stdout: "fs_name": "cephfs", 2024-03-19T02:52:41.706 INFO:teuthology.orchestra.run.smithi033.stdout: "rank_uptime": 83.625058632000005, 2024-03-19T02:52:41.706 INFO:teuthology.orchestra.run.smithi033.stdout: "mdsmap_epoch": 24, 2024-03-19T02:52:41.706 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch": 97, 2024-03-19T02:52:41.706 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch_barrier": 93, 2024-03-19T02:52:41.706 INFO:teuthology.orchestra.run.smithi033.stdout: "uptime": 122.72387865899999 2024-03-19T02:52:41.706 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:52:41.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.697+0000 7f3ad8953640 1 -- 172.21.15.33:0/2075524787 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f3abc0a2800 msgr2=0x7f3abc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:41.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.697+0000 7f3ad8953640 1 --2- 172.21.15.33:0/2075524787 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f3abc0a2800 0x7f3abc0a4cc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x55c0c4f3c240 tx=0x7f3ac4004a40 comp rx=0 tx=0).stop 2024-03-19T02:52:41.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.697+0000 7f3ad8953640 1 -- 172.21.15.33:0/2075524787 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ad4100720 msgr2=0x7f3ad4197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:41.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.697+0000 7f3ad8953640 1 --2- 172.21.15.33:0/2075524787 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ad4100720 0x7f3ad4197ac0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3acc036040 tx=0x7f3acc005dc0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.697+0000 7f3ad8953640 1 -- 172.21.15.33:0/2075524787 shutdown_connections 2024-03-19T02:52:41.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.697+0000 7f3ad8953640 1 --2- 172.21.15.33:0/2075524787 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f3abc0a2800 0x7f3abc0a4cc0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.697+0000 7f3ad8953640 1 --2- 172.21.15.33:0/2075524787 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ad41022d0 0x7f3ad419f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.697+0000 7f3ad8953640 1 --2- 172.21.15.33:0/2075524787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3ad4101910 0x7f3ad4198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.697+0000 7f3ad8953640 1 --2- 172.21.15.33:0/2075524787 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3ad4100720 0x7f3ad4197ac0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.697+0000 7f3ad8953640 1 -- 172.21.15.33:0/2075524787 >> 172.21.15.33:0/2075524787 conn(0x7f3ad40fbcb0 msgr2=0x7f3ad4104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:41.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.697+0000 7f3ad8953640 1 -- 172.21.15.33:0/2075524787 shutdown_connections 2024-03-19T02:52:41.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.697+0000 7f3ad8953640 1 -- 172.21.15.33:0/2075524787 wait complete. 2024-03-19T02:52:41.709 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-19T02:52:41.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.811+0000 7efd87a36640 1 -- 172.21.15.33:0/772081372 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd801022d0 msgr2=0x7efd80108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:41.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.811+0000 7efd87a36640 1 --2- 172.21.15.33:0/772081372 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd801022d0 0x7efd80108b60 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7efd78009ca0 tx=0x7efd7802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.811+0000 7efd87a36640 1 -- 172.21.15.33:0/772081372 shutdown_connections 2024-03-19T02:52:41.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.811+0000 7efd87a36640 1 --2- 172.21.15.33:0/772081372 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd80101910 0x7efd80101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.811+0000 7efd87a36640 1 --2- 172.21.15.33:0/772081372 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd801022d0 0x7efd80108b60 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.811+0000 7efd87a36640 1 --2- 172.21.15.33:0/772081372 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd80100720 0x7efd80100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.811+0000 7efd87a36640 1 -- 172.21.15.33:0/772081372 >> 172.21.15.33:0/772081372 conn(0x7efd800fbcb0 msgr2=0x7efd800fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:41.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.812+0000 7efd87a36640 1 -- 172.21.15.33:0/772081372 shutdown_connections 2024-03-19T02:52:41.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.812+0000 7efd87a36640 1 -- 172.21.15.33:0/772081372 wait complete. 2024-03-19T02:52:41.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.812+0000 7efd87a36640 1 Processor -- start 2024-03-19T02:52:41.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.812+0000 7efd87a36640 1 -- start start 2024-03-19T02:52:41.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.812+0000 7efd87a36640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd80100720 0x7efd80197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.812+0000 7efd87a36640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd80101910 0x7efd80197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.812+0000 7efd87a36640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd801022d0 0x7efd8019efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.812+0000 7efd87a36640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efd8010c250 con 0x7efd80100720 2024-03-19T02:52:41.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.812+0000 7efd87a36640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efd8010c0d0 con 0x7efd80101910 2024-03-19T02:52:41.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.812+0000 7efd87a36640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7efd8010c3d0 con 0x7efd801022d0 2024-03-19T02:52:41.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.812+0000 7efd857d2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd80100720 0x7efd80197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.812+0000 7efd857d2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd80100720 0x7efd80197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41700/0 (socket says 172.21.15.33:41700) 2024-03-19T02:52:41.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.812+0000 7efd857d2640 1 -- 172.21.15.33:0/156372670 learned_addr learned my addr 172.21.15.33:0/156372670 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:41.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.812+0000 7efd84fd1640 1 --2- 172.21.15.33:0/156372670 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd80101910 0x7efd80197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.812+0000 7efd85fd3640 1 --2- 172.21.15.33:0/156372670 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd801022d0 0x7efd8019efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.813+0000 7efd857d2640 1 -- 172.21.15.33:0/156372670 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd801022d0 msgr2=0x7efd8019efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:41.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.813+0000 7efd857d2640 1 --2- 172.21.15.33:0/156372670 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd801022d0 0x7efd8019efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.813+0000 7efd857d2640 1 -- 172.21.15.33:0/156372670 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd80101910 msgr2=0x7efd80197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:41.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.813+0000 7efd857d2640 1 --2- 172.21.15.33:0/156372670 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd80101910 0x7efd80197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.813+0000 7efd857d2640 1 -- 172.21.15.33:0/156372670 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd8019f640 con 0x7efd80100720 2024-03-19T02:52:41.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.813+0000 7efd84fd1640 1 --2- 172.21.15.33:0/156372670 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd80101910 0x7efd80197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:41.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.813+0000 7efd857d2640 1 --2- 172.21.15.33:0/156372670 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd80100720 0x7efd80197a70 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7efd7c00e9c0 tx=0x7efd7c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:41.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.813+0000 7efd767fc640 1 -- 172.21.15.33:0/156372670 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7c00cd80 con 0x7efd80100720 2024-03-19T02:52:41.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.813+0000 7efd767fc640 1 -- 172.21.15.33:0/156372670 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efd7c0035f0 con 0x7efd80100720 2024-03-19T02:52:41.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.813+0000 7efd87a36640 1 -- 172.21.15.33:0/156372670 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd8019f930 con 0x7efd80100720 2024-03-19T02:52:41.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.813+0000 7efd767fc640 1 -- 172.21.15.33:0/156372670 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7c010690 con 0x7efd80100720 2024-03-19T02:52:41.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.813+0000 7efd87a36640 1 -- 172.21.15.33:0/156372670 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd8019fed0 con 0x7efd80100720 2024-03-19T02:52:41.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.815+0000 7efd767fc640 1 -- 172.21.15.33:0/156372670 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7efd7c0037a0 con 0x7efd80100720 2024-03-19T02:52:41.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.816+0000 7efd8a629740 1 Processor -- start 2024-03-19T02:52:41.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.816+0000 7efd8a629740 1 -- start start 2024-03-19T02:52:41.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.816+0000 7efd767fc640 1 --2- 172.21.15.33:0/156372670 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7efd6c0a2880 0x7efd6c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.816+0000 7efd8a629740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564ac3c2eff0 0x564ac3c2f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.816+0000 7efd8a629740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ac3c2f9e0 0x564ac3c344e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.816+0000 7efd8a629740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564ac3c34b50 0x564ac3c36fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.816+0000 7efd8a629740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564ac3b93de0 con 0x564ac3c2f9e0 2024-03-19T02:52:41.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.816+0000 7efd8a629740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x564ac3b3cc80 con 0x564ac3c34b50 2024-03-19T02:52:41.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.816+0000 7efd8a629740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564ac3b8fe80 con 0x564ac3c2eff0 2024-03-19T02:52:41.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.816+0000 7efd84fd1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ac3c2f9e0 0x564ac3c344e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.816+0000 7efd85fd3640 1 --2- 172.21.15.33:0/156372670 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7efd6c0a2880 0x7efd6c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.816+0000 7efd767fc640 1 -- 172.21.15.33:0/156372670 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7efd7c0cf160 con 0x7efd80100720 2024-03-19T02:52:41.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.816+0000 7efd857d2640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564ac3c2eff0 0x564ac3c2f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.816+0000 7efd84fd1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ac3c2f9e0 0x564ac3c344e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41714/0 (socket says 172.21.15.33:41714) 2024-03-19T02:52:41.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.816+0000 7efd84fd1640 1 -- 172.21.15.33:0/2506896568 learned_addr learned my addr 172.21.15.33:0/2506896568 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:41.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.816+0000 7efd857d2640 1 --2- 172.21.15.33:0/2506896568 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564ac3c2eff0 0x564ac3c2f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:51568/0 (socket says 172.21.15.33:51568) 2024-03-19T02:52:41.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.816+0000 7efd85fd3640 1 --2- 172.21.15.33:0/2506896568 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564ac3c34b50 0x564ac3c36fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.816+0000 7efd857d2640 1 -- 172.21.15.33:0/2506896568 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564ac3c34b50 msgr2=0x564ac3c36fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:41.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.816+0000 7efd857d2640 1 --2- 172.21.15.33:0/2506896568 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564ac3c34b50 0x564ac3c36fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.816+0000 7efd857d2640 1 -- 172.21.15.33:0/2506896568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ac3c2f9e0 msgr2=0x564ac3c344e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:41.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.816+0000 7efd857d2640 1 --2- 172.21.15.33:0/2506896568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ac3c2f9e0 0x564ac3c344e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.817+0000 7efd857d2640 1 -- 172.21.15.33:0/2506896568 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ac3b4efb0 con 0x564ac3c2eff0 2024-03-19T02:52:41.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.817+0000 7efd84fd1640 1 --2- 172.21.15.33:0/2506896568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ac3c2f9e0 0x564ac3c344e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:41.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.817+0000 7efd85fd3640 1 --2- 172.21.15.33:0/156372670 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7efd6c0a2880 0x7efd6c0a4d40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efd78006010 tx=0x7efd78009970 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:41.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.817+0000 7efd857d2640 1 --2- 172.21.15.33:0/2506896568 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564ac3c2eff0 0x564ac3c2f410 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efd7c09b6a0 tx=0x7efd7c09b310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38e54ed5c9a18ce server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:41.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.817+0000 7efd6b7fe640 1 -- 172.21.15.33:0/2506896568 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7c003b30 con 0x564ac3c2eff0 2024-03-19T02:52:41.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.817+0000 7efd6b7fe640 1 -- 172.21.15.33:0/2506896568 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efd7c09be10 con 0x564ac3c2eff0 2024-03-19T02:52:41.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.817+0000 7efd6b7fe640 1 -- 172.21.15.33:0/2506896568 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7c0a0b00 con 0x564ac3c2eff0 2024-03-19T02:52:41.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.818+0000 7efd8a629740 1 -- 172.21.15.33:0/2506896568 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564ac3c2eff0 msgr2=0x564ac3c2f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:41.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.818+0000 7efd8a629740 1 --2- 172.21.15.33:0/2506896568 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564ac3c2eff0 0x564ac3c2f410 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efd7c09b6a0 tx=0x7efd7c09b310 comp rx=0 tx=0).stop 2024-03-19T02:52:41.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.818+0000 7efd8a629740 1 -- 172.21.15.33:0/2506896568 shutdown_connections 2024-03-19T02:52:41.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.818+0000 7efd8a629740 1 --2- 172.21.15.33:0/2506896568 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564ac3c34b50 0x564ac3c36fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.818+0000 7efd8a629740 1 --2- 172.21.15.33:0/2506896568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ac3c2f9e0 0x564ac3c344e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.818+0000 7efd8a629740 1 --2- 172.21.15.33:0/2506896568 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564ac3c2eff0 0x564ac3c2f410 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.818+0000 7efd8a629740 1 -- 172.21.15.33:0/2506896568 >> 172.21.15.33:0/2506896568 conn(0x564ac3c2a560 msgr2=0x564ac3c2a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:41.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.818+0000 7efd8a629740 1 -- 172.21.15.33:0/2506896568 shutdown_connections 2024-03-19T02:52:41.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.818+0000 7efd8a629740 1 -- 172.21.15.33:0/2506896568 wait complete. 2024-03-19T02:52:41.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.818+0000 7efd8a629740 1 -- start start 2024-03-19T02:52:41.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.818+0000 7efd8a629740 1 Processor -- start 2024-03-19T02:52:41.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.818+0000 7efd8a629740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564ac3c2eff0 0x564ac3c39480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.818+0000 7efd8a629740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ac3c2f9e0 0x564ac3c399c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.818+0000 7efd8a629740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564ac3c34b50 0x564ac3c40a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.818+0000 7efd8a629740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564ac3c37aa0 con 0x564ac3c2f9e0 2024-03-19T02:52:41.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.818+0000 7efd8a629740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x564ac3b4ffc0 con 0x564ac3c2eff0 2024-03-19T02:52:41.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.818+0000 7efd8a629740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564ac3c37c20 con 0x564ac3c34b50 2024-03-19T02:52:41.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.818+0000 7efd8a629740 10 client.0 _ll_register_callbacks cb 0x564ac3b8ba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:41.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.818+0000 7efd84fd1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ac3c2f9e0 0x564ac3c399c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.818+0000 7efd84fd1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ac3c2f9e0 0x564ac3c399c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41720/0 (socket says 172.21.15.33:41720) 2024-03-19T02:52:41.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.818+0000 7efd84fd1640 1 -- 172.21.15.33:0/1468934155 learned_addr learned my addr 172.21.15.33:0/1468934155 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:41.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.819+0000 7efd857d2640 1 --2- 172.21.15.33:0/1468934155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564ac3c2eff0 0x564ac3c39480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.819+0000 7efd85fd3640 1 --2- 172.21.15.33:0/1468934155 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564ac3c34b50 0x564ac3c40a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.819+0000 7efd84fd1640 1 -- 172.21.15.33:0/1468934155 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564ac3c34b50 msgr2=0x564ac3c40a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:41.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.819+0000 7efd84fd1640 1 --2- 172.21.15.33:0/1468934155 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564ac3c34b50 0x564ac3c40a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.819+0000 7efd84fd1640 1 -- 172.21.15.33:0/1468934155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564ac3c2eff0 msgr2=0x564ac3c39480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:41.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.819+0000 7efd84fd1640 1 --2- 172.21.15.33:0/1468934155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564ac3c2eff0 0x564ac3c39480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.819+0000 7efd84fd1640 1 -- 172.21.15.33:0/1468934155 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ac38a82d0 con 0x564ac3c2f9e0 2024-03-19T02:52:41.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.819+0000 7efd857d2640 1 --2- 172.21.15.33:0/1468934155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564ac3c2eff0 0x564ac3c39480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:41.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.819+0000 7efd85fd3640 1 --2- 172.21.15.33:0/1468934155 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564ac3c34b50 0x564ac3c40a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:41.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.819+0000 7efd84fd1640 1 --2- 172.21.15.33:0/1468934155 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ac3c2f9e0 0x564ac3c399c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7efd700129f0 tx=0x7efd70012eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f3a4a653aed8ea7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:41.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.819+0000 7efd68ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:52:41.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.819+0000 7efd68ff9640 1 -- 172.21.15.33:0/1468934155 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7000f9e0 con 0x564ac3c2f9e0 2024-03-19T02:52:41.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.819+0000 7efd68ff9640 1 -- 172.21.15.33:0/1468934155 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7efd7000fb80 con 0x564ac3c2f9e0 2024-03-19T02:52:41.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.819+0000 7efd8a629740 1 -- 172.21.15.33:0/1468934155 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd7c09be10 con 0x564ac3c2f9e0 2024-03-19T02:52:41.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.819+0000 7efd68ff9640 1 -- 172.21.15.33:0/1468934155 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7efd70025050 con 0x564ac3c2f9e0 2024-03-19T02:52:41.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.819+0000 7efd8a629740 10 client.15216 fetch_fsmap learned FSMap version 24 2024-03-19T02:52:41.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.819+0000 7efd8a629740 1 -- 172.21.15.33:0/1468934155 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564ac39726b0 con 0x564ac3c2f9e0 2024-03-19T02:52:41.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.819+0000 7efd68ff9640 1 -- 172.21.15.33:0/1468934155 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efd7c09be10 con 0x564ac3c2f9e0 2024-03-19T02:52:41.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.820+0000 7efd68ff9640 1 -- 172.21.15.33:0/1468934155 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7efd70003ab0 con 0x564ac3c2f9e0 2024-03-19T02:52:41.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.820+0000 7efd8a629740 10 client.15216 fetch_fsmap finished waiting for FSMap version 24 2024-03-19T02:52:41.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.820+0000 7efd8a629740 10 client.15216 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-19T02:52:41.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.820+0000 7efd8a629740 20 client.15216 populate_metadata read hostname 'smithi033' 2024-03-19T02:52:41.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.820+0000 7efd8a629740 1 --2- 172.21.15.33:0/1468934155 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x564ac3c46310 0x564ac3c47780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.820+0000 7efd8a629740 4 client.15216 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:52:41.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.820+0000 7efd8a629740 1 -- 172.21.15.33:0/1468934155 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ac3ba1320 con 0x564ac3c46310 2024-03-19T02:52:41.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.820+0000 7efd857d2640 1 --2- 172.21.15.33:0/1468934155 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x564ac3c46310 0x564ac3c47780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.820+0000 7efd68ff9640 10 client.15216 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T02:52:41.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.820+0000 7efd857d2640 1 --2- 172.21.15.33:0/1468934155 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x564ac3c46310 0x564ac3c47780 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b3dcebda8a8fc258 server_cookie=4e5cb708987798ef in_seq=0 out_seq=0 2024-03-19T02:52:41.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.821+0000 7efd68ff9640 1 -- 172.21.15.33:0/1468934155 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564ac3ba1320 con 0x564ac3c46310 2024-03-19T02:52:41.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.821+0000 7efd68ff9640 10 client.15216 handle_command_reply: tid=0 2024-03-19T02:52:41.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.821+0000 7efd8a629740 1 client.15216 shutdown 2024-03-19T02:52:41.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.821+0000 7efd8a629740 1 -- 172.21.15.33:0/1468934155 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ac3c2f9e0 msgr2=0x564ac3c399c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:41.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.821+0000 7efd8a629740 1 --2- 172.21.15.33:0/1468934155 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ac3c2f9e0 0x564ac3c399c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7efd700129f0 tx=0x7efd70012eb0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.821+0000 7efd8a629740 1 -- 172.21.15.33:0/1468934155 shutdown_connections 2024-03-19T02:52:41.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.821+0000 7efd8a629740 1 --2- 172.21.15.33:0/1468934155 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x564ac3c46310 0x564ac3c47780 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.821+0000 7efd8a629740 1 --2- 172.21.15.33:0/1468934155 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564ac3c34b50 0x564ac3c40a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.821+0000 7efd8a629740 1 --2- 172.21.15.33:0/1468934155 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ac3c2f9e0 0x564ac3c399c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.821+0000 7efd68ff9640 0 client.15216 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T02:52:41.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.821+0000 7efd8a629740 1 --2- 172.21.15.33:0/1468934155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564ac3c2eff0 0x564ac3c39480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.821+0000 7efd8a629740 1 -- 172.21.15.33:0/1468934155 >> 172.21.15.33:0/1468934155 conn(0x564ac3c2a560 msgr2=0x564ac3c297e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:41.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.821+0000 7efd8a629740 1 -- 172.21.15.33:0/1468934155 shutdown_connections 2024-03-19T02:52:41.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.821+0000 7efd8a629740 1 -- 172.21.15.33:0/1468934155 wait complete. 2024-03-19T02:52:41.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.821+0000 7efd8a629740 20 client.15216 trim_cache size 0 max 16384 2024-03-19T02:52:41.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.839+0000 7efd8a629740 1 Processor -- start 2024-03-19T02:52:41.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.839+0000 7efd8a629740 1 -- start start 2024-03-19T02:52:41.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.840+0000 7efd8a629740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564ac3c2eff0 0x564ac3c34510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.840+0000 7efd8a629740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564ac3c2f9e0 0x564ac3c409c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.840+0000 7efd8a629740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ac3c34b50 0x564ac3c30ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.840+0000 7efd8a629740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564ac3c45ac0 con 0x564ac3c34b50 2024-03-19T02:52:41.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.840+0000 7efd8a629740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x564ac3c46190 con 0x564ac3c2f9e0 2024-03-19T02:52:41.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.840+0000 7efd8a629740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564ac3c46860 con 0x564ac3c2eff0 2024-03-19T02:52:41.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.840+0000 7efd85fd3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ac3c34b50 0x564ac3c30ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.840+0000 7efd85fd3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ac3c34b50 0x564ac3c30ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41726/0 (socket says 172.21.15.33:41726) 2024-03-19T02:52:41.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.840+0000 7efd85fd3640 1 -- 172.21.15.33:0/2588250188 learned_addr learned my addr 172.21.15.33:0/2588250188 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:41.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.840+0000 7efd857d2640 1 --2- 172.21.15.33:0/2588250188 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564ac3c2eff0 0x564ac3c34510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.840+0000 7efd84fd1640 1 --2- 172.21.15.33:0/2588250188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564ac3c2f9e0 0x564ac3c409c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.840+0000 7efd85fd3640 1 -- 172.21.15.33:0/2588250188 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564ac3c2eff0 msgr2=0x564ac3c34510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:41.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.840+0000 7efd85fd3640 1 --2- 172.21.15.33:0/2588250188 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564ac3c2eff0 0x564ac3c34510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.840+0000 7efd85fd3640 1 -- 172.21.15.33:0/2588250188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564ac3c2f9e0 msgr2=0x564ac3c409c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:41.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.840+0000 7efd85fd3640 1 --2- 172.21.15.33:0/2588250188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564ac3c2f9e0 0x564ac3c409c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.840+0000 7efd85fd3640 1 -- 172.21.15.33:0/2588250188 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ac3c2b2d0 con 0x564ac3c34b50 2024-03-19T02:52:41.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.840+0000 7efd85fd3640 1 --2- 172.21.15.33:0/2588250188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ac3c34b50 0x564ac3c30ea0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7efd78009ca0 tx=0x7efd78047000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2d6a74ea9ace089 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:41.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.840+0000 7efd6affd640 1 -- 172.21.15.33:0/2588250188 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd78053070 con 0x564ac3c34b50 2024-03-19T02:52:41.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.840+0000 7efd6affd640 1 -- 172.21.15.33:0/2588250188 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efd780046f0 con 0x564ac3c34b50 2024-03-19T02:52:41.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.840+0000 7efd6affd640 1 -- 172.21.15.33:0/2588250188 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd78039040 con 0x564ac3c34b50 2024-03-19T02:52:41.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.841+0000 7efd8a629740 1 -- 172.21.15.33:0/2588250188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ac3c34b50 msgr2=0x564ac3c30ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:41.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.841+0000 7efd8a629740 1 --2- 172.21.15.33:0/2588250188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ac3c34b50 0x564ac3c30ea0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7efd78009ca0 tx=0x7efd78047000 comp rx=0 tx=0).stop 2024-03-19T02:52:41.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.841+0000 7efd8a629740 1 -- 172.21.15.33:0/2588250188 shutdown_connections 2024-03-19T02:52:41.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.841+0000 7efd8a629740 1 --2- 172.21.15.33:0/2588250188 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ac3c34b50 0x564ac3c30ea0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.841+0000 7efd8a629740 1 --2- 172.21.15.33:0/2588250188 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564ac3c2f9e0 0x564ac3c409c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.841+0000 7efd8a629740 1 --2- 172.21.15.33:0/2588250188 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564ac3c2eff0 0x564ac3c34510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.841+0000 7efd8a629740 1 -- 172.21.15.33:0/2588250188 >> 172.21.15.33:0/2588250188 conn(0x564ac3c2a560 msgr2=0x564ac3c300d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:41.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.841+0000 7efd8a629740 1 -- 172.21.15.33:0/2588250188 shutdown_connections 2024-03-19T02:52:41.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.841+0000 7efd8a629740 1 -- 172.21.15.33:0/2588250188 wait complete. 2024-03-19T02:52:41.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.841+0000 7efd8a629740 1 -- start start 2024-03-19T02:52:41.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.841+0000 7efd8a629740 1 Processor -- start 2024-03-19T02:52:41.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.842+0000 7efd8a629740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564ac3c2eff0 0x564ac3c3bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.842+0000 7efd8a629740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564ac3c2f9e0 0x564ac3c3c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.842+0000 7efd8a629740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ac3c34b50 0x564ac3c3f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.842+0000 7efd8a629740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564ac3c47600 con 0x564ac3c34b50 2024-03-19T02:52:41.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.842+0000 7efd8a629740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x564ac3c46f30 con 0x564ac3c2eff0 2024-03-19T02:52:41.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.842+0000 7efd8a629740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564ac3c47cd0 con 0x564ac3c2f9e0 2024-03-19T02:52:41.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.842+0000 7efd8a629740 10 client.0 _ll_register_callbacks cb 0x7efd70009be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:41.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.842+0000 7efd85fd3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ac3c34b50 0x564ac3c3f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.842+0000 7efd857d2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564ac3c2eff0 0x564ac3c3bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.842+0000 7efd85fd3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ac3c34b50 0x564ac3c3f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41730/0 (socket says 172.21.15.33:41730) 2024-03-19T02:52:41.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.842+0000 7efd85fd3640 1 -- 172.21.15.33:0/1484906324 learned_addr learned my addr 172.21.15.33:0/1484906324 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:41.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.842+0000 7efd84fd1640 1 --2- 172.21.15.33:0/1484906324 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564ac3c2f9e0 0x564ac3c3c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.842+0000 7efd85fd3640 1 -- 172.21.15.33:0/1484906324 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564ac3c2f9e0 msgr2=0x564ac3c3c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:41.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.842+0000 7efd85fd3640 1 --2- 172.21.15.33:0/1484906324 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564ac3c2f9e0 0x564ac3c3c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.842+0000 7efd85fd3640 1 -- 172.21.15.33:0/1484906324 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564ac3c2eff0 msgr2=0x564ac3c3bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:41.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.842+0000 7efd85fd3640 1 --2- 172.21.15.33:0/1484906324 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564ac3c2eff0 0x564ac3c3bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.842+0000 7efd85fd3640 1 -- 172.21.15.33:0/1484906324 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ac3c3faf0 con 0x564ac3c34b50 2024-03-19T02:52:41.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.842+0000 7efd857d2640 1 --2- 172.21.15.33:0/1484906324 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564ac3c2eff0 0x564ac3c3bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:41.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.842+0000 7efd85fd3640 1 --2- 172.21.15.33:0/1484906324 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ac3c34b50 0x564ac3c3f500 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7efd78051040 tx=0x7efd780041f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d48db3f5f41909b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:41.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.842+0000 7efd68ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:52:41.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.842+0000 7efd68ff9640 1 -- 172.21.15.33:0/1484906324 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7805c060 con 0x564ac3c34b50 2024-03-19T02:52:41.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.842+0000 7efd68ff9640 1 -- 172.21.15.33:0/1484906324 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7efd78034640 con 0x564ac3c34b50 2024-03-19T02:52:41.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.842+0000 7efd8a629740 1 -- 172.21.15.33:0/1484906324 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd780046f0 con 0x564ac3c34b50 2024-03-19T02:52:41.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.842+0000 7efd68ff9640 1 -- 172.21.15.33:0/1484906324 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7efd78053050 con 0x564ac3c34b50 2024-03-19T02:52:41.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.842+0000 7efd8a629740 10 client.15222 fetch_fsmap learned FSMap version 24 2024-03-19T02:52:41.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.842+0000 7efd8a629740 1 -- 172.21.15.33:0/1484906324 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564ac3cac080 con 0x564ac3c34b50 2024-03-19T02:52:41.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.843+0000 7efd68ff9640 1 -- 172.21.15.33:0/1484906324 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efd780046f0 con 0x564ac3c34b50 2024-03-19T02:52:41.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.843+0000 7efd68ff9640 1 -- 172.21.15.33:0/1484906324 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7efd7803f040 con 0x564ac3c34b50 2024-03-19T02:52:41.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.843+0000 7efd8a629740 10 client.15222 fetch_fsmap finished waiting for FSMap version 24 2024-03-19T02:52:41.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.843+0000 7efd8a629740 10 client.15222 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-19T02:52:41.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.843+0000 7efd8a629740 20 client.15222 populate_metadata read hostname 'smithi033' 2024-03-19T02:52:41.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.843+0000 7efd8a629740 1 --2- 172.21.15.33:0/1484906324 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x564ac3cb06d0 0x564ac3caf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.843+0000 7efd8a629740 4 client.15222 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T02:52:41.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.843+0000 7efd8a629740 1 -- 172.21.15.33:0/1484906324 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "status"}) -- 0x564ac3b9b670 con 0x564ac3cb06d0 2024-03-19T02:52:41.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.843+0000 7efd857d2640 1 --2- 172.21.15.33:0/1484906324 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x564ac3cb06d0 0x564ac3caf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.843+0000 7efd68ff9640 10 client.15222 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T02:52:41.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.843+0000 7efd857d2640 1 --2- 172.21.15.33:0/1484906324 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x564ac3cb06d0 0x564ac3caf3e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ff813aff7c8fa493 server_cookie=8eafa5e4a0ac3f74 in_seq=0 out_seq=0 2024-03-19T02:52:41.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.843+0000 7efd68ff9640 1 -- 172.21.15.33:0/1484906324 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x564ac3b9b670 con 0x564ac3cb06d0 2024-03-19T02:52:41.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.843+0000 7efd68ff9640 10 client.15222 handle_command_reply: tid=0 2024-03-19T02:52:41.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.843+0000 7efd8a629740 1 client.15222 shutdown 2024-03-19T02:52:41.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.844+0000 7efd8a629740 1 -- 172.21.15.33:0/1484906324 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ac3c34b50 msgr2=0x564ac3c3f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:41.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.844+0000 7efd8a629740 1 --2- 172.21.15.33:0/1484906324 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ac3c34b50 0x564ac3c3f500 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7efd78051040 tx=0x7efd780041f0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.844+0000 7efd8a629740 1 -- 172.21.15.33:0/1484906324 shutdown_connections 2024-03-19T02:52:41.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.844+0000 7efd8a629740 1 --2- 172.21.15.33:0/1484906324 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x564ac3cb06d0 0x564ac3caf3e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.844+0000 7efd8a629740 1 --2- 172.21.15.33:0/1484906324 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564ac3c2f9e0 0x564ac3c3c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.844+0000 7efd68ff9640 0 client.15222 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T02:52:41.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.844+0000 7efd8a629740 1 --2- 172.21.15.33:0/1484906324 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564ac3c34b50 0x564ac3c3f500 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.844+0000 7efd8a629740 1 --2- 172.21.15.33:0/1484906324 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564ac3c2eff0 0x564ac3c3bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.844+0000 7efd8a629740 1 -- 172.21.15.33:0/1484906324 >> 172.21.15.33:0/1484906324 conn(0x564ac3c2a560 msgr2=0x564ac3c300d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:41.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.844+0000 7efd8a629740 1 -- 172.21.15.33:0/1484906324 shutdown_connections 2024-03-19T02:52:41.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.844+0000 7efd8a629740 1 -- 172.21.15.33:0/1484906324 wait complete. 2024-03-19T02:52:41.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.844+0000 7efd8a629740 20 client.15222 trim_cache size 0 max 16384 2024-03-19T02:52:41.852 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:52:41.852 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_fsid": "2b7a8938-e59a-11ee-95c9-87774f69a715", 2024-03-19T02:52:41.852 INFO:teuthology.orchestra.run.smithi033.stdout: "whoami": 1, 2024-03-19T02:52:41.852 INFO:teuthology.orchestra.run.smithi033.stdout: "id": 24488, 2024-03-19T02:52:41.852 INFO:teuthology.orchestra.run.smithi033.stdout: "want_state": "up:active", 2024-03-19T02:52:41.852 INFO:teuthology.orchestra.run.smithi033.stdout: "state": "up:active", 2024-03-19T02:52:41.852 INFO:teuthology.orchestra.run.smithi033.stdout: "fs_name": "cephfs", 2024-03-19T02:52:41.852 INFO:teuthology.orchestra.run.smithi033.stdout: "rank_uptime": 84.786166382999994, 2024-03-19T02:52:41.852 INFO:teuthology.orchestra.run.smithi033.stdout: "mdsmap_epoch": 24, 2024-03-19T02:52:41.852 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch": 97, 2024-03-19T02:52:41.852 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch_barrier": 93, 2024-03-19T02:52:41.852 INFO:teuthology.orchestra.run.smithi033.stdout: "uptime": 125.523511493 2024-03-19T02:52:41.852 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:52:41.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.844+0000 7efd87a36640 1 -- 172.21.15.33:0/156372670 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7efd6c0a2880 msgr2=0x7efd6c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:41.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.844+0000 7efd87a36640 1 --2- 172.21.15.33:0/156372670 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7efd6c0a2880 0x7efd6c0a4d40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efd78006010 tx=0x7efd78009970 comp rx=0 tx=0).stop 2024-03-19T02:52:41.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.844+0000 7efd87a36640 1 -- 172.21.15.33:0/156372670 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd80100720 msgr2=0x7efd80197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:41.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.844+0000 7efd87a36640 1 --2- 172.21.15.33:0/156372670 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd80100720 0x7efd80197a70 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7efd7c00e9c0 tx=0x7efd7c00ee80 comp rx=0 tx=0).stop 2024-03-19T02:52:41.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.844+0000 7efd87a36640 1 -- 172.21.15.33:0/156372670 shutdown_connections 2024-03-19T02:52:41.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.844+0000 7efd87a36640 1 --2- 172.21.15.33:0/156372670 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7efd6c0a2880 0x7efd6c0a4d40 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.844+0000 7efd87a36640 1 --2- 172.21.15.33:0/156372670 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd801022d0 0x7efd8019efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.844+0000 7efd87a36640 1 --2- 172.21.15.33:0/156372670 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd80101910 0x7efd80197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.844+0000 7efd87a36640 1 --2- 172.21.15.33:0/156372670 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efd80100720 0x7efd80197a70 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.844+0000 7efd87a36640 1 -- 172.21.15.33:0/156372670 >> 172.21.15.33:0/156372670 conn(0x7efd800fbcb0 msgr2=0x7efd800fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:41.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.844+0000 7efd87a36640 1 -- 172.21.15.33:0/156372670 shutdown_connections 2024-03-19T02:52:41.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.844+0000 7efd87a36640 1 -- 172.21.15.33:0/156372670 wait complete. 2024-03-19T02:52:41.856 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-19T02:52:41.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.958+0000 7fc259df4640 1 -- 172.21.15.33:0/1113286313 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc254101830 msgr2=0x7fc254101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:41.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.958+0000 7fc259df4640 1 --2- 172.21.15.33:0/1113286313 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc254101830 0x7fc254101c50 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc244009900 tx=0x7fc24402bc30 comp rx=0 tx=0).stop 2024-03-19T02:52:41.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.958+0000 7fc259df4640 1 -- 172.21.15.33:0/1113286313 shutdown_connections 2024-03-19T02:52:41.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.958+0000 7fc259df4640 1 --2- 172.21.15.33:0/1113286313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2541033e0 0x7fc254109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.958+0000 7fc259df4640 1 --2- 172.21.15.33:0/1113286313 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc254102a20 0x7fc254102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.958+0000 7fc259df4640 1 --2- 172.21.15.33:0/1113286313 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc254101830 0x7fc254101c50 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.958+0000 7fc259df4640 1 -- 172.21.15.33:0/1113286313 >> 172.21.15.33:0/1113286313 conn(0x7fc2540fcdc0 msgr2=0x7fc2540ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:41.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.958+0000 7fc259df4640 1 -- 172.21.15.33:0/1113286313 shutdown_connections 2024-03-19T02:52:41.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.958+0000 7fc259df4640 1 -- 172.21.15.33:0/1113286313 wait complete. 2024-03-19T02:52:41.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.959+0000 7fc259df4640 1 Processor -- start 2024-03-19T02:52:41.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.959+0000 7fc259df4640 1 -- start start 2024-03-19T02:52:41.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.959+0000 7fc259df4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc254101830 0x7fc2540755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.959+0000 7fc259df4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc254102a20 0x7fc254073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.959+0000 7fc259df4640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2541033e0 0x7fc254074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.959+0000 7fc259df4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc25410d330 con 0x7fc254102a20 2024-03-19T02:52:41.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.959+0000 7fc259df4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc25410d1b0 con 0x7fc254101830 2024-03-19T02:52:41.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.959+0000 7fc259df4640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc25410d4b0 con 0x7fc2541033e0 2024-03-19T02:52:41.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.959+0000 7fc252ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc254102a20 0x7fc254073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.959+0000 7fc252ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc254102a20 0x7fc254073c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41756/0 (socket says 172.21.15.33:41756) 2024-03-19T02:52:41.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.959+0000 7fc252ffd640 1 -- 172.21.15.33:0/1971448110 learned_addr learned my addr 172.21.15.33:0/1971448110 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:41.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.959+0000 7fc2537fe640 1 --2- 172.21.15.33:0/1971448110 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc254101830 0x7fc2540755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.959+0000 7fc253fff640 1 --2- 172.21.15.33:0/1971448110 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2541033e0 0x7fc254074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.959+0000 7fc252ffd640 1 -- 172.21.15.33:0/1971448110 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2541033e0 msgr2=0x7fc254074150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:41.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.959+0000 7fc252ffd640 1 --2- 172.21.15.33:0/1971448110 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2541033e0 0x7fc254074150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.959+0000 7fc252ffd640 1 -- 172.21.15.33:0/1971448110 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc254101830 msgr2=0x7fc2540755d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:41.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.959+0000 7fc252ffd640 1 --2- 172.21.15.33:0/1971448110 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc254101830 0x7fc2540755d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.959+0000 7fc252ffd640 1 -- 172.21.15.33:0/1971448110 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2540749e0 con 0x7fc254102a20 2024-03-19T02:52:41.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.959+0000 7fc253fff640 1 --2- 172.21.15.33:0/1971448110 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2541033e0 0x7fc254074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:52:41.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.960+0000 7fc252ffd640 1 --2- 172.21.15.33:0/1971448110 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc254102a20 0x7fc254073c10 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc24800ea60 tx=0x7fc24800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:41.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.960+0000 7fc250ff9640 1 -- 172.21.15.33:0/1971448110 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc24800cde0 con 0x7fc254102a20 2024-03-19T02:52:41.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.960+0000 7fc259df4640 1 -- 172.21.15.33:0/1971448110 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2541a1c00 con 0x7fc254102a20 2024-03-19T02:52:41.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.960+0000 7fc250ff9640 1 -- 172.21.15.33:0/1971448110 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc2480035f0 con 0x7fc254102a20 2024-03-19T02:52:41.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.960+0000 7fc250ff9640 1 -- 172.21.15.33:0/1971448110 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc248010690 con 0x7fc254102a20 2024-03-19T02:52:41.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.960+0000 7fc259df4640 1 -- 172.21.15.33:0/1971448110 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2541a21a0 con 0x7fc254102a20 2024-03-19T02:52:41.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.962+0000 7fc250ff9640 1 -- 172.21.15.33:0/1971448110 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc2480037a0 con 0x7fc254102a20 2024-03-19T02:52:41.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.962+0000 7fc25c9e4740 1 Processor -- start 2024-03-19T02:52:41.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.962+0000 7fc25c9e4740 1 -- start start 2024-03-19T02:52:41.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.962+0000 7fc25c9e4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d985812ff0 0x55d985813410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.962+0000 7fc25c9e4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9858139e0 0x55d9858184e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.962+0000 7fc25c9e4740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d985818b50 0x55d98581afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.962+0000 7fc25c9e4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d985777de0 con 0x55d985812ff0 2024-03-19T02:52:41.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.962+0000 7fc25c9e4740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55d985720c80 con 0x55d985818b50 2024-03-19T02:52:41.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.962+0000 7fc25c9e4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d985773e80 con 0x55d9858139e0 2024-03-19T02:52:41.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.962+0000 7fc250ff9640 1 --2- 172.21.15.33:0/1971448110 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc23c0a2800 0x7fc23c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.963+0000 7fc253fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d985812ff0 0x55d985813410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.963+0000 7fc2537fe640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9858139e0 0x55d9858184e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.963+0000 7fc252ffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d985818b50 0x55d98581afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.963+0000 7fc2537fe640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9858139e0 0x55d9858184e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:51624/0 (socket says 172.21.15.33:51624) 2024-03-19T02:52:41.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.963+0000 7fc2537fe640 1 -- 172.21.15.33:0/839869067 learned_addr learned my addr 172.21.15.33:0/839869067 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:41.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.963+0000 7fc253fff640 1 --2- 172.21.15.33:0/1971448110 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc23c0a2800 0x7fc23c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.963+0000 7fc2537fe640 1 -- 172.21.15.33:0/839869067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d985818b50 msgr2=0x55d98581afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:41.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.963+0000 7fc2537fe640 1 --2- 172.21.15.33:0/839869067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d985818b50 0x55d98581afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.963+0000 7fc250ff9640 1 -- 172.21.15.33:0/1971448110 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7fc2480cf140 con 0x7fc254102a20 2024-03-19T02:52:41.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.963+0000 7fc2537fe640 1 -- 172.21.15.33:0/839869067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d985812ff0 msgr2=0x55d985813410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:41.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.963+0000 7fc2537fe640 1 --2- 172.21.15.33:0/839869067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d985812ff0 0x55d985813410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.963+0000 7fc2537fe640 1 -- 172.21.15.33:0/839869067 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d985732fb0 con 0x55d9858139e0 2024-03-19T02:52:41.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.963+0000 7fc253fff640 1 --2- 172.21.15.33:0/1971448110 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc23c0a2800 0x7fc23c0a4cc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc24c0099a0 tx=0x7fc24c004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:41.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.963+0000 7fc2537fe640 1 --2- 172.21.15.33:0/839869067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9858139e0 0x55d9858184e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc244009900 tx=0x7fc244009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfab04adaad7da37 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:41.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.963+0000 7fc239ffb640 1 -- 172.21.15.33:0/839869067 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc244045020 con 0x55d9858139e0 2024-03-19T02:52:41.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.963+0000 7fc239ffb640 1 -- 172.21.15.33:0/839869067 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc244004040 con 0x55d9858139e0 2024-03-19T02:52:41.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.963+0000 7fc239ffb640 1 -- 172.21.15.33:0/839869067 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc244005570 con 0x55d9858139e0 2024-03-19T02:52:41.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.964+0000 7fc25c9e4740 1 -- 172.21.15.33:0/839869067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9858139e0 msgr2=0x55d9858184e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:41.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.964+0000 7fc25c9e4740 1 --2- 172.21.15.33:0/839869067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9858139e0 0x55d9858184e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc244009900 tx=0x7fc244009560 comp rx=0 tx=0).stop 2024-03-19T02:52:41.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.964+0000 7fc25c9e4740 1 -- 172.21.15.33:0/839869067 shutdown_connections 2024-03-19T02:52:41.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.964+0000 7fc25c9e4740 1 --2- 172.21.15.33:0/839869067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9858139e0 0x55d9858184e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.964+0000 7fc25c9e4740 1 --2- 172.21.15.33:0/839869067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d985818b50 0x55d98581afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.964+0000 7fc25c9e4740 1 --2- 172.21.15.33:0/839869067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d985812ff0 0x55d985813410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.964+0000 7fc25c9e4740 1 -- 172.21.15.33:0/839869067 >> 172.21.15.33:0/839869067 conn(0x55d98580e560 msgr2=0x55d98580e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:41.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.964+0000 7fc25c9e4740 1 -- 172.21.15.33:0/839869067 shutdown_connections 2024-03-19T02:52:41.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.964+0000 7fc25c9e4740 1 -- 172.21.15.33:0/839869067 wait complete. 2024-03-19T02:52:41.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.964+0000 7fc25c9e4740 1 -- start start 2024-03-19T02:52:41.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.965+0000 7fc25c9e4740 1 Processor -- start 2024-03-19T02:52:41.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.965+0000 7fc25c9e4740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d985812ff0 0x55d98581d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.965+0000 7fc25c9e4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9858139e0 0x55d98581d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.965+0000 7fc25c9e4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d985818b50 0x55d985824a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.965+0000 7fc25c9e4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d98581baa0 con 0x55d985818b50 2024-03-19T02:52:41.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.965+0000 7fc25c9e4740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55d985733fc0 con 0x55d985812ff0 2024-03-19T02:52:41.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.965+0000 7fc25c9e4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d98581bc20 con 0x55d9858139e0 2024-03-19T02:52:41.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.965+0000 7fc25c9e4740 10 client.0 _ll_register_callbacks cb 0x55d98576fa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:41.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.965+0000 7fc253fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d985818b50 0x55d985824a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.965+0000 7fc253fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d985818b50 0x55d985824a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41768/0 (socket says 172.21.15.33:41768) 2024-03-19T02:52:41.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.965+0000 7fc253fff640 1 -- 172.21.15.33:0/893618912 learned_addr learned my addr 172.21.15.33:0/893618912 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:41.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.965+0000 7fc2537fe640 1 --2- 172.21.15.33:0/893618912 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d985812ff0 0x55d98581d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.965+0000 7fc252ffd640 1 --2- 172.21.15.33:0/893618912 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9858139e0 0x55d98581d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.965+0000 7fc253fff640 1 -- 172.21.15.33:0/893618912 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9858139e0 msgr2=0x55d98581d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:41.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.965+0000 7fc253fff640 1 --2- 172.21.15.33:0/893618912 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9858139e0 0x55d98581d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.965+0000 7fc253fff640 1 -- 172.21.15.33:0/893618912 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d985812ff0 msgr2=0x55d98581d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:41.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.965+0000 7fc253fff640 1 --2- 172.21.15.33:0/893618912 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d985812ff0 0x55d98581d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.965+0000 7fc253fff640 1 -- 172.21.15.33:0/893618912 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d98548c2d0 con 0x55d985818b50 2024-03-19T02:52:41.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.965+0000 7fc2537fe640 1 --2- 172.21.15.33:0/893618912 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d985812ff0 0x55d98581d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:41.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.965+0000 7fc253fff640 1 --2- 172.21.15.33:0/893618912 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d985818b50 0x55d985824a30 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc24c005e60 tx=0x7fc24c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=495c78b606815875 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:41.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.965+0000 7fc232ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:52:41.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.966+0000 7fc232ffd640 1 -- 172.21.15.33:0/893618912 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc24c004740 con 0x55d985818b50 2024-03-19T02:52:41.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.966+0000 7fc232ffd640 1 -- 172.21.15.33:0/893618912 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7fc24c0213f0 con 0x55d985818b50 2024-03-19T02:52:41.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.966+0000 7fc25c9e4740 1 -- 172.21.15.33:0/893618912 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc244004040 con 0x55d985818b50 2024-03-19T02:52:41.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.966+0000 7fc232ffd640 1 -- 172.21.15.33:0/893618912 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc24c0215f0 con 0x55d985818b50 2024-03-19T02:52:41.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.966+0000 7fc25c9e4740 10 client.15234 fetch_fsmap learned FSMap version 24 2024-03-19T02:52:41.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.966+0000 7fc25c9e4740 1 -- 172.21.15.33:0/893618912 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d9855566b0 con 0x55d985818b50 2024-03-19T02:52:41.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.966+0000 7fc232ffd640 1 -- 172.21.15.33:0/893618912 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc244004040 con 0x55d985818b50 2024-03-19T02:52:41.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.966+0000 7fc232ffd640 1 -- 172.21.15.33:0/893618912 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fc24c0295c0 con 0x55d985818b50 2024-03-19T02:52:41.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.966+0000 7fc25c9e4740 10 client.15234 fetch_fsmap finished waiting for FSMap version 24 2024-03-19T02:52:41.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.966+0000 7fc25c9e4740 10 client.15234 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-19T02:52:41.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.966+0000 7fc25c9e4740 20 client.15234 populate_metadata read hostname 'smithi033' 2024-03-19T02:52:41.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.966+0000 7fc25c9e4740 1 --2- 172.21.15.33:0/893618912 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55d98582a370 0x55d98582b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.966+0000 7fc25c9e4740 4 client.15234 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:52:41.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.966+0000 7fc25c9e4740 1 -- 172.21.15.33:0/893618912 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d985785320 con 0x55d98582a370 2024-03-19T02:52:41.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.966+0000 7fc2537fe640 1 --2- 172.21.15.33:0/893618912 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55d98582a370 0x55d98582b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.967+0000 7fc2537fe640 1 --2- 172.21.15.33:0/893618912 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55d98582a370 0x55d98582b7e0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dccf1528cbb38bd6 server_cookie=d74ef721a54d432c in_seq=0 out_seq=0 2024-03-19T02:52:41.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.967+0000 7fc232ffd640 10 client.15234 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:52:41.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.967+0000 7fc232ffd640 1 -- 172.21.15.33:0/893618912 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d985785320 con 0x55d98582a370 2024-03-19T02:52:41.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.967+0000 7fc232ffd640 10 client.15234 handle_command_reply: tid=0 2024-03-19T02:52:41.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.967+0000 7fc25c9e4740 1 client.15234 shutdown 2024-03-19T02:52:41.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.968+0000 7fc25c9e4740 1 -- 172.21.15.33:0/893618912 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d985818b50 msgr2=0x55d985824a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:41.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.968+0000 7fc25c9e4740 1 --2- 172.21.15.33:0/893618912 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d985818b50 0x55d985824a30 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc24c005e60 tx=0x7fc24c004a40 comp rx=0 tx=0).stop 2024-03-19T02:52:41.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.968+0000 7fc25c9e4740 1 -- 172.21.15.33:0/893618912 shutdown_connections 2024-03-19T02:52:41.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.968+0000 7fc25c9e4740 1 --2- 172.21.15.33:0/893618912 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55d98582a370 0x55d98582b7e0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.968+0000 7fc25c9e4740 1 --2- 172.21.15.33:0/893618912 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9858139e0 0x55d98581d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.968+0000 7fc25c9e4740 1 --2- 172.21.15.33:0/893618912 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d985818b50 0x55d985824a30 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.968+0000 7fc232ffd640 0 client.15234 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:52:41.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.968+0000 7fc25c9e4740 1 --2- 172.21.15.33:0/893618912 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d985812ff0 0x55d98581d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.968+0000 7fc25c9e4740 1 -- 172.21.15.33:0/893618912 >> 172.21.15.33:0/893618912 conn(0x55d98580e560 msgr2=0x55d98580d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:41.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.968+0000 7fc25c9e4740 1 -- 172.21.15.33:0/893618912 shutdown_connections 2024-03-19T02:52:41.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.968+0000 7fc25c9e4740 1 -- 172.21.15.33:0/893618912 wait complete. 2024-03-19T02:52:41.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.968+0000 7fc25c9e4740 20 client.15234 trim_cache size 0 max 16384 2024-03-19T02:52:41.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.986+0000 7fc25c9e4740 1 Processor -- start 2024-03-19T02:52:41.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.986+0000 7fc25c9e4740 1 -- start start 2024-03-19T02:52:41.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.986+0000 7fc25c9e4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d985812ff0 0x55d98580e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.986+0000 7fc25c9e4740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d9858139e0 0x55d9858249f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.986+0000 7fc25c9e4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d985818b50 0x55d985814e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.986+0000 7fc25c9e4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d985829b20 con 0x55d985818b50 2024-03-19T02:52:41.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.986+0000 7fc25c9e4740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55d98582a1f0 con 0x55d9858139e0 2024-03-19T02:52:41.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.986+0000 7fc25c9e4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d98582a8c0 con 0x55d985812ff0 2024-03-19T02:52:41.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.986+0000 7fc253fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d985818b50 0x55d985814e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.986+0000 7fc253fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d985818b50 0x55d985814e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41778/0 (socket says 172.21.15.33:41778) 2024-03-19T02:52:41.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.986+0000 7fc253fff640 1 -- 172.21.15.33:0/2883771042 learned_addr learned my addr 172.21.15.33:0/2883771042 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:41.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.986+0000 7fc2537fe640 1 --2- 172.21.15.33:0/2883771042 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d985812ff0 0x55d98580e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.986+0000 7fc252ffd640 1 --2- 172.21.15.33:0/2883771042 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d9858139e0 0x55d9858249f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.986+0000 7fc253fff640 1 -- 172.21.15.33:0/2883771042 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d985812ff0 msgr2=0x55d98580e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:41.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.986+0000 7fc253fff640 1 --2- 172.21.15.33:0/2883771042 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d985812ff0 0x55d98580e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.986+0000 7fc253fff640 1 -- 172.21.15.33:0/2883771042 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d9858139e0 msgr2=0x55d9858249f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:41.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.986+0000 7fc253fff640 1 --2- 172.21.15.33:0/2883771042 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d9858139e0 0x55d9858249f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.986+0000 7fc253fff640 1 -- 172.21.15.33:0/2883771042 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d98580f2d0 con 0x55d985818b50 2024-03-19T02:52:41.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.986+0000 7fc253fff640 1 --2- 172.21.15.33:0/2883771042 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d985818b50 0x55d985814e40 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc24c000f80 tx=0x7fc24c009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8863a4d16a273081 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:41.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.986+0000 7fc2397fa640 1 -- 172.21.15.33:0/2883771042 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc24c0213f0 con 0x55d985818b50 2024-03-19T02:52:41.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.987+0000 7fc2397fa640 1 -- 172.21.15.33:0/2883771042 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc24c004740 con 0x55d985818b50 2024-03-19T02:52:41.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.987+0000 7fc2397fa640 1 -- 172.21.15.33:0/2883771042 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc24c021a80 con 0x55d985818b50 2024-03-19T02:52:41.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.987+0000 7fc25c9e4740 1 -- 172.21.15.33:0/2883771042 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d985818b50 msgr2=0x55d985814e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:41.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.987+0000 7fc25c9e4740 1 --2- 172.21.15.33:0/2883771042 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d985818b50 0x55d985814e40 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc24c000f80 tx=0x7fc24c009c20 comp rx=0 tx=0).stop 2024-03-19T02:52:41.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.987+0000 7fc25c9e4740 1 -- 172.21.15.33:0/2883771042 shutdown_connections 2024-03-19T02:52:41.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.987+0000 7fc25c9e4740 1 --2- 172.21.15.33:0/2883771042 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d985818b50 0x55d985814e40 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.987+0000 7fc25c9e4740 1 --2- 172.21.15.33:0/2883771042 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d9858139e0 0x55d9858249f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.987+0000 7fc25c9e4740 1 --2- 172.21.15.33:0/2883771042 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d985812ff0 0x55d98580e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.987+0000 7fc25c9e4740 1 -- 172.21.15.33:0/2883771042 >> 172.21.15.33:0/2883771042 conn(0x55d98580e560 msgr2=0x55d98581c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:41.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.987+0000 7fc25c9e4740 1 -- 172.21.15.33:0/2883771042 shutdown_connections 2024-03-19T02:52:41.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.987+0000 7fc25c9e4740 1 -- 172.21.15.33:0/2883771042 wait complete. 2024-03-19T02:52:41.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.987+0000 7fc25c9e4740 1 -- start start 2024-03-19T02:52:41.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.988+0000 7fc25c9e4740 1 Processor -- start 2024-03-19T02:52:41.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.988+0000 7fc25c9e4740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d985812ff0 0x55d9858201c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.988+0000 7fc25c9e4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9858139e0 0x55d985820700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.988+0000 7fc25c9e4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d985818b50 0x55d985823720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.988+0000 7fc25c9e4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d98582b660 con 0x55d985818b50 2024-03-19T02:52:41.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.988+0000 7fc25c9e4740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55d98582af90 con 0x55d985812ff0 2024-03-19T02:52:41.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.988+0000 7fc25c9e4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d98582bd30 con 0x55d9858139e0 2024-03-19T02:52:41.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.988+0000 7fc25c9e4740 10 client.0 _ll_register_callbacks cb 0x7fc24c00cee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:52:41.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.988+0000 7fc253fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d985818b50 0x55d985823720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.988+0000 7fc253fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d985818b50 0x55d985823720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41794/0 (socket says 172.21.15.33:41794) 2024-03-19T02:52:41.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.988+0000 7fc2537fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d985812ff0 0x55d9858201c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.988+0000 7fc253fff640 1 -- 172.21.15.33:0/1703995557 learned_addr learned my addr 172.21.15.33:0/1703995557 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:41.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.988+0000 7fc252ffd640 1 --2- 172.21.15.33:0/1703995557 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9858139e0 0x55d985820700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.988+0000 7fc253fff640 1 -- 172.21.15.33:0/1703995557 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9858139e0 msgr2=0x55d985820700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:41.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.988+0000 7fc253fff640 1 --2- 172.21.15.33:0/1703995557 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9858139e0 0x55d985820700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.988+0000 7fc253fff640 1 -- 172.21.15.33:0/1703995557 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d985812ff0 msgr2=0x55d9858201c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:52:41.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.988+0000 7fc253fff640 1 --2- 172.21.15.33:0/1703995557 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d985812ff0 0x55d9858201c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.988+0000 7fc253fff640 1 -- 172.21.15.33:0/1703995557 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d985823d10 con 0x55d985818b50 2024-03-19T02:52:41.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.988+0000 7fc2537fe640 1 --2- 172.21.15.33:0/1703995557 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d985812ff0 0x55d9858201c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:41.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.988+0000 7fc253fff640 1 --2- 172.21.15.33:0/1703995557 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d985818b50 0x55d985823720 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc24c038040 tx=0x7fc24c022a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef114239efe49a20 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:41.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.988+0000 7fc232ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:52:41.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.989+0000 7fc232ffd640 1 -- 172.21.15.33:0/1703995557 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc24c043060 con 0x55d985818b50 2024-03-19T02:52:41.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.989+0000 7fc232ffd640 1 -- 172.21.15.33:0/1703995557 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7fc24c022e40 con 0x55d985818b50 2024-03-19T02:52:41.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.989+0000 7fc25c9e4740 1 -- 172.21.15.33:0/1703995557 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc24c004740 con 0x55d985818b50 2024-03-19T02:52:41.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.989+0000 7fc232ffd640 1 -- 172.21.15.33:0/1703995557 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc24c01b040 con 0x55d985818b50 2024-03-19T02:52:41.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.989+0000 7fc25c9e4740 10 client.15240 fetch_fsmap learned FSMap version 24 2024-03-19T02:52:41.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.989+0000 7fc25c9e4740 1 -- 172.21.15.33:0/1703995557 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d9858901a0 con 0x55d985818b50 2024-03-19T02:52:41.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.989+0000 7fc232ffd640 1 -- 172.21.15.33:0/1703995557 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc24c004740 con 0x55d985818b50 2024-03-19T02:52:41.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.989+0000 7fc232ffd640 1 -- 172.21.15.33:0/1703995557 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fc24c036970 con 0x55d985818b50 2024-03-19T02:52:41.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.989+0000 7fc25c9e4740 10 client.15240 fetch_fsmap finished waiting for FSMap version 24 2024-03-19T02:52:41.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.989+0000 7fc25c9e4740 10 client.15240 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-19T02:52:41.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.989+0000 7fc25c9e4740 20 client.15240 populate_metadata read hostname 'smithi033' 2024-03-19T02:52:41.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.989+0000 7fc25c9e4740 1 --2- 172.21.15.33:0/1703995557 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55d985894a00 0x55d985893710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:41.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.989+0000 7fc25c9e4740 4 client.15240 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T02:52:41.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.989+0000 7fc25c9e4740 1 -- 172.21.15.33:0/1703995557 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "status"}) -- 0x55d98577f670 con 0x55d985894a00 2024-03-19T02:52:41.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.989+0000 7fc2537fe640 1 --2- 172.21.15.33:0/1703995557 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55d985894a00 0x55d985893710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:41.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.990+0000 7fc232ffd640 10 client.15240 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:52:41.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.990+0000 7fc2537fe640 1 --2- 172.21.15.33:0/1703995557 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55d985894a00 0x55d985893710 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc5eee930789814 server_cookie=ff9c37d2d3bcf452 in_seq=0 out_seq=0 2024-03-19T02:52:41.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.990+0000 7fc232ffd640 1 -- 172.21.15.33:0/1703995557 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55d98577f670 con 0x55d985894a00 2024-03-19T02:52:41.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.990+0000 7fc232ffd640 10 client.15240 handle_command_reply: tid=0 2024-03-19T02:52:41.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.990+0000 7fc25c9e4740 1 client.15240 shutdown 2024-03-19T02:52:41.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.990+0000 7fc25c9e4740 1 -- 172.21.15.33:0/1703995557 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d985818b50 msgr2=0x55d985823720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:41.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.990+0000 7fc25c9e4740 1 --2- 172.21.15.33:0/1703995557 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d985818b50 0x55d985823720 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc24c038040 tx=0x7fc24c022a40 comp rx=0 tx=0).stop 2024-03-19T02:52:41.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.990+0000 7fc25c9e4740 1 -- 172.21.15.33:0/1703995557 shutdown_connections 2024-03-19T02:52:41.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.990+0000 7fc25c9e4740 1 --2- 172.21.15.33:0/1703995557 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55d985894a00 0x55d985893710 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.990+0000 7fc25c9e4740 1 --2- 172.21.15.33:0/1703995557 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d9858139e0 0x55d985820700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.990+0000 7fc25c9e4740 1 --2- 172.21.15.33:0/1703995557 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d985818b50 0x55d985823720 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.990+0000 7fc232ffd640 0 client.15240 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:52:41.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.990+0000 7fc25c9e4740 1 --2- 172.21.15.33:0/1703995557 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d985812ff0 0x55d9858201c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:41.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.990+0000 7fc25c9e4740 1 -- 172.21.15.33:0/1703995557 >> 172.21.15.33:0/1703995557 conn(0x55d98580e560 msgr2=0x55d98580e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:41.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.990+0000 7fc25c9e4740 1 -- 172.21.15.33:0/1703995557 shutdown_connections 2024-03-19T02:52:41.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.990+0000 7fc25c9e4740 1 -- 172.21.15.33:0/1703995557 wait complete. 2024-03-19T02:52:41.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.990+0000 7fc25c9e4740 20 client.15240 trim_cache size 0 max 16384 2024-03-19T02:52:41.999 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:52:41.999 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_fsid": "2b7a8938-e59a-11ee-95c9-87774f69a715", 2024-03-19T02:52:41.999 INFO:teuthology.orchestra.run.smithi033.stdout: "whoami": 0, 2024-03-19T02:52:41.999 INFO:teuthology.orchestra.run.smithi033.stdout: "id": 24509, 2024-03-19T02:52:41.999 INFO:teuthology.orchestra.run.smithi033.stdout: "want_state": "up:active", 2024-03-19T02:52:41.999 INFO:teuthology.orchestra.run.smithi033.stdout: "state": "up:active", 2024-03-19T02:52:41.999 INFO:teuthology.orchestra.run.smithi033.stdout: "fs_name": "cephfs", 2024-03-19T02:52:41.999 INFO:teuthology.orchestra.run.smithi033.stdout: "rank_uptime": 108.79114600600001, 2024-03-19T02:52:41.999 INFO:teuthology.orchestra.run.smithi033.stdout: "mdsmap_epoch": 24, 2024-03-19T02:52:42.000 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch": 97, 2024-03-19T02:52:42.000 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch_barrier": 93, 2024-03-19T02:52:42.000 INFO:teuthology.orchestra.run.smithi033.stdout: "uptime": 114.371895972 2024-03-19T02:52:42.000 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:52:42.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.991+0000 7fc259df4640 1 -- 172.21.15.33:0/1971448110 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc23c0a2800 msgr2=0x7fc23c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:42.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.991+0000 7fc259df4640 1 --2- 172.21.15.33:0/1971448110 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc23c0a2800 0x7fc23c0a4cc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc24c0099a0 tx=0x7fc24c004ab0 comp rx=0 tx=0).stop 2024-03-19T02:52:42.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.991+0000 7fc259df4640 1 -- 172.21.15.33:0/1971448110 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc254102a20 msgr2=0x7fc254073c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:42.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.991+0000 7fc259df4640 1 --2- 172.21.15.33:0/1971448110 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc254102a20 0x7fc254073c10 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc24800ea60 tx=0x7fc24800ef20 comp rx=0 tx=0).stop 2024-03-19T02:52:42.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.991+0000 7fc259df4640 1 -- 172.21.15.33:0/1971448110 shutdown_connections 2024-03-19T02:52:42.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.991+0000 7fc259df4640 1 --2- 172.21.15.33:0/1971448110 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc23c0a2800 0x7fc23c0a4cc0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:42.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.991+0000 7fc259df4640 1 --2- 172.21.15.33:0/1971448110 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc2541033e0 0x7fc254074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:42.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.991+0000 7fc259df4640 1 --2- 172.21.15.33:0/1971448110 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc254102a20 0x7fc254073c10 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:42.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.991+0000 7fc259df4640 1 --2- 172.21.15.33:0/1971448110 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc254101830 0x7fc2540755d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:42.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.991+0000 7fc259df4640 1 -- 172.21.15.33:0/1971448110 >> 172.21.15.33:0/1971448110 conn(0x7fc2540fcdc0 msgr2=0x7fc254105ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:42.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.991+0000 7fc259df4640 1 -- 172.21.15.33:0/1971448110 shutdown_connections 2024-03-19T02:52:42.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:41.991+0000 7fc259df4640 1 -- 172.21.15.33:0/1971448110 wait complete. 2024-03-19T02:52:42.003 INFO:tasks.cephadm:Setup complete, yielding 2024-03-19T02:52:42.003 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-19T02:52:42.013 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi033.front.sepia.ceph.com 2024-03-19T02:52:42.013 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- bash -c 'ceph orch status' 2024-03-19T02:52:42.254 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:52:42.376 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:42 smithi033 ceph-mon[31973]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-19T02:52:42.376 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:42 smithi033 ceph-mon[31973]: osdmap e97: 12 total, 12 up, 12 in 2024-03-19T02:52:42.376 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:42 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1417585748' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:52:42.376 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:42 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/4239880998' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:52:42.376 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:42 smithi033 ceph-mon[31973]: pgmap v358: 129 pgs: 129 active+clean; 642 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:52:42.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:42 smithi096 ceph-mon[35946]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-19T02:52:42.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:42 smithi096 ceph-mon[35946]: osdmap e97: 12 total, 12 up, 12 in 2024-03-19T02:52:42.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:42 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1417585748' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:52:42.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:42 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/4239880998' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:52:42.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:42 smithi096 ceph-mon[35946]: pgmap v358: 129 pgs: 129 active+clean; 642 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:52:42.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:42 smithi155 ceph-mon[35804]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-19T02:52:42.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:42 smithi155 ceph-mon[35804]: osdmap e97: 12 total, 12 up, 12 in 2024-03-19T02:52:42.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:42 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1417585748' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:52:42.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:42 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/4239880998' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:52:42.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:42 smithi155 ceph-mon[35804]: pgmap v358: 129 pgs: 129 active+clean; 642 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:52:43.996 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:43 smithi033 ceph-mon[31973]: pgmap v359: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-19T02:52:44.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.015+0000 7f7f79bfe700 1 -- 172.21.15.33:0/467648870 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f74101240 msgr2=0x7f7f74101660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:44.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.015+0000 7f7f79bfe700 1 --2- 172.21.15.33:0/467648870 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f74101240 0x7f7f74101660 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f7f640099c0 tx=0x7f7f64009cd0 comp rx=0 tx=0).stop 2024-03-19T02:52:44.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.015+0000 7f7f79bfe700 1 -- 172.21.15.33:0/467648870 shutdown_connections 2024-03-19T02:52:44.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.015+0000 7f7f79bfe700 1 --2- 172.21.15.33:0/467648870 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f74102de0 0x7f7f74109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:44.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.015+0000 7f7f79bfe700 1 --2- 172.21.15.33:0/467648870 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f74101240 0x7f7f74101660 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:44.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.015+0000 7f7f79bfe700 1 --2- 172.21.15.33:0/467648870 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f74102440 0x7f7f741028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:44.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.015+0000 7f7f79bfe700 1 -- 172.21.15.33:0/467648870 >> 172.21.15.33:0/467648870 conn(0x7f7f740fc800 msgr2=0x7f7f740fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:44.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.015+0000 7f7f79bfe700 1 -- 172.21.15.33:0/467648870 shutdown_connections 2024-03-19T02:52:44.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.015+0000 7f7f79bfe700 1 -- 172.21.15.33:0/467648870 wait complete. 2024-03-19T02:52:44.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.015+0000 7f7f79bfe700 1 Processor -- start 2024-03-19T02:52:44.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.016+0000 7f7f79bfe700 1 -- start start 2024-03-19T02:52:44.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.016+0000 7f7f79bfe700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f74101240 0x7f7f741984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:44.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.016+0000 7f7f79bfe700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f74102440 0x7f7f74198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:44.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.016+0000 7f7f79bfe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f74102de0 0x7f7f7419fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:44.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.016+0000 7f7f79bfe700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7f7410ccb0 con 0x7f7f74101240 2024-03-19T02:52:44.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.016+0000 7f7f79bfe700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7f7410cb30 con 0x7f7f74102440 2024-03-19T02:52:44.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.016+0000 7f7f79bfe700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7f7410ce30 con 0x7f7f74102de0 2024-03-19T02:52:44.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.016+0000 7f7f737fe700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f74101240 0x7f7f741984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:44.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.016+0000 7f7f737fe700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f74101240 0x7f7f741984c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:44522/0 (socket says 172.21.15.33:44522) 2024-03-19T02:52:44.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.016+0000 7f7f737fe700 1 -- 172.21.15.33:0/1195850492 learned_addr learned my addr 172.21.15.33:0/1195850492 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:44.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.016+0000 7f7f72ffd700 1 --2- 172.21.15.33:0/1195850492 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f74102440 0x7f7f74198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:44.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.016+0000 7f7f73fff700 1 --2- 172.21.15.33:0/1195850492 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f74102de0 0x7f7f7419fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:44.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.016+0000 7f7f737fe700 1 -- 172.21.15.33:0/1195850492 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f74102de0 msgr2=0x7f7f7419fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:44.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.016+0000 7f7f737fe700 1 --2- 172.21.15.33:0/1195850492 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f74102de0 0x7f7f7419fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:44.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.016+0000 7f7f737fe700 1 -- 172.21.15.33:0/1195850492 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f74102440 msgr2=0x7f7f74198a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:44.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.016+0000 7f7f737fe700 1 --2- 172.21.15.33:0/1195850492 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f74102440 0x7f7f74198a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:44.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.016+0000 7f7f737fe700 1 -- 172.21.15.33:0/1195850492 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f741a0120 con 0x7f7f74101240 2024-03-19T02:52:44.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.016+0000 7f7f72ffd700 1 --2- 172.21.15.33:0/1195850492 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f74102440 0x7f7f74198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:44.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.016+0000 7f7f737fe700 1 --2- 172.21.15.33:0/1195850492 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f74101240 0x7f7f741984c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f7f6400f710 tx=0x7f7f6400f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:44.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.017+0000 7f7f70ff9700 1 -- 172.21.15.33:0/1195850492 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f64009e60 con 0x7f7f74101240 2024-03-19T02:52:44.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.017+0000 7f7f79bfe700 1 -- 172.21.15.33:0/1195850492 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f741a03b0 con 0x7f7f74101240 2024-03-19T02:52:44.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.017+0000 7f7f70ff9700 1 -- 172.21.15.33:0/1195850492 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7f64005940 con 0x7f7f74101240 2024-03-19T02:52:44.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.017+0000 7f7f70ff9700 1 -- 172.21.15.33:0/1195850492 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f640054f0 con 0x7f7f74101240 2024-03-19T02:52:44.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.017+0000 7f7f79bfe700 1 -- 172.21.15.33:0/1195850492 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f741a0890 con 0x7f7f74101240 2024-03-19T02:52:44.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.018+0000 7f7f79bfe700 1 -- 172.21.15.33:0/1195850492 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f74068fd0 con 0x7f7f74101240 2024-03-19T02:52:44.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.019+0000 7f7f70ff9700 1 -- 172.21.15.33:0/1195850492 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7f6400fb70 con 0x7f7f74101240 2024-03-19T02:52:44.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.020+0000 7f7f70ff9700 1 --2- 172.21.15.33:0/1195850492 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7f5c0a2890 0x7f7f5c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:44.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.020+0000 7f7f72ffd700 1 --2- 172.21.15.33:0/1195850492 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7f5c0a2890 0x7f7f5c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:44.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.020+0000 7f7f70ff9700 1 -- 172.21.15.33:0/1195850492 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f7f640cfb40 con 0x7f7f74101240 2024-03-19T02:52:44.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.020+0000 7f7f72ffd700 1 --2- 172.21.15.33:0/1195850492 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7f5c0a2890 0x7f7f5c0a4d50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7f68011490 tx=0x7f7f68009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:44.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.024+0000 7f7f70ff9700 1 -- 172.21.15.33:0/1195850492 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7f64013070 con 0x7f7f74101240 2024-03-19T02:52:44.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:43 smithi096 ceph-mon[35946]: pgmap v359: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-19T02:52:44.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.250+0000 7f7f79bfe700 1 -- 172.21.15.33:0/1195850492 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f7f7406b190 con 0x7f7f5c0a2890 2024-03-19T02:52:44.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.252+0000 7f7f70ff9700 1 -- 172.21.15.33:0/1195850492 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f7f7406b190 con 0x7f7f5c0a2890 2024-03-19T02:52:44.252 INFO:teuthology.orchestra.run.smithi033.stdout:Backend: cephadm 2024-03-19T02:52:44.253 INFO:teuthology.orchestra.run.smithi033.stdout:Available: Yes 2024-03-19T02:52:44.253 INFO:teuthology.orchestra.run.smithi033.stdout:Paused: No 2024-03-19T02:52:44.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.254+0000 7f7f79bfe700 1 -- 172.21.15.33:0/1195850492 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7f5c0a2890 msgr2=0x7f7f5c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:44.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.254+0000 7f7f79bfe700 1 --2- 172.21.15.33:0/1195850492 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7f5c0a2890 0x7f7f5c0a4d50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7f68011490 tx=0x7f7f68009260 comp rx=0 tx=0).stop 2024-03-19T02:52:44.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.254+0000 7f7f79bfe700 1 -- 172.21.15.33:0/1195850492 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f74101240 msgr2=0x7f7f741984c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:44.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.254+0000 7f7f79bfe700 1 --2- 172.21.15.33:0/1195850492 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f74101240 0x7f7f741984c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f7f6400f710 tx=0x7f7f6400f740 comp rx=0 tx=0).stop 2024-03-19T02:52:44.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.254+0000 7f7f79bfe700 1 -- 172.21.15.33:0/1195850492 shutdown_connections 2024-03-19T02:52:44.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.254+0000 7f7f79bfe700 1 --2- 172.21.15.33:0/1195850492 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7f5c0a2890 0x7f7f5c0a4d50 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:44.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.254+0000 7f7f79bfe700 1 --2- 172.21.15.33:0/1195850492 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f74102de0 0x7f7f7419fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:44.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.254+0000 7f7f79bfe700 1 --2- 172.21.15.33:0/1195850492 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f74101240 0x7f7f741984c0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:44.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.254+0000 7f7f79bfe700 1 --2- 172.21.15.33:0/1195850492 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f74102440 0x7f7f74198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:44.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.254+0000 7f7f79bfe700 1 -- 172.21.15.33:0/1195850492 >> 172.21.15.33:0/1195850492 conn(0x7f7f740fc800 msgr2=0x7f7f7406aa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:44.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.254+0000 7f7f79bfe700 1 -- 172.21.15.33:0/1195850492 shutdown_connections 2024-03-19T02:52:44.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:44.254+0000 7f7f79bfe700 1 -- 172.21.15.33:0/1195850492 wait complete. 2024-03-19T02:52:44.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:43 smithi155 ceph-mon[35804]: pgmap v359: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-19T02:52:44.951 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:44 smithi033 ceph-mon[31973]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:52:44.977 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-19T02:52:45.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:44 smithi096 ceph-mon[35946]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:52:45.221 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:52:45.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:44 smithi155 ceph-mon[35804]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:52:46.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:45 smithi033 ceph-mon[31973]: pgmap v360: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 383 B/s wr, 8 op/s 2024-03-19T02:52:46.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:45 smithi096 ceph-mon[35946]: pgmap v360: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 383 B/s wr, 8 op/s 2024-03-19T02:52:46.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:45 smithi155 ceph-mon[35804]: pgmap v360: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 383 B/s wr, 8 op/s 2024-03-19T02:52:46.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.943+0000 7f5a09c27700 1 -- 172.21.15.33:0/2789194067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a04102de0 msgr2=0x7f5a04109690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:46.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.943+0000 7f5a09c27700 1 --2- 172.21.15.33:0/2789194067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a04102de0 0x7f5a04109690 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f59f40099c0 tx=0x7f59f4009cd0 comp rx=0 tx=0).stop 2024-03-19T02:52:46.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.943+0000 7f5a09c27700 1 -- 172.21.15.33:0/2789194067 shutdown_connections 2024-03-19T02:52:46.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.943+0000 7f5a09c27700 1 --2- 172.21.15.33:0/2789194067 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a04102de0 0x7f5a04109690 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:46.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.943+0000 7f5a09c27700 1 --2- 172.21.15.33:0/2789194067 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a04101240 0x7f5a04101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:46.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.943+0000 7f5a09c27700 1 --2- 172.21.15.33:0/2789194067 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a04102440 0x7f5a041028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:46.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.943+0000 7f5a09c27700 1 -- 172.21.15.33:0/2789194067 >> 172.21.15.33:0/2789194067 conn(0x7f5a040fc7e0 msgr2=0x7f5a040fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:46.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.943+0000 7f5a09c27700 1 -- 172.21.15.33:0/2789194067 shutdown_connections 2024-03-19T02:52:46.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.943+0000 7f5a09c27700 1 -- 172.21.15.33:0/2789194067 wait complete. 2024-03-19T02:52:46.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.944+0000 7f5a09c27700 1 Processor -- start 2024-03-19T02:52:46.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.944+0000 7f5a09c27700 1 -- start start 2024-03-19T02:52:46.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.944+0000 7f5a09c27700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a04101240 0x7f5a04197320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:46.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.944+0000 7f5a09c27700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a04102440 0x7f5a04197860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:46.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.944+0000 7f5a09c27700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a04102de0 0x7f5a0419e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:46.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.944+0000 7f5a09c27700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0410ccd0 con 0x7f5a04101240 2024-03-19T02:52:46.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.944+0000 7f5a09c27700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0410cb50 con 0x7f5a04102de0 2024-03-19T02:52:46.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.944+0000 7f5a09c27700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0410ce50 con 0x7f5a04102440 2024-03-19T02:52:46.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.944+0000 7f5a037fe700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a04101240 0x7f5a04197320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:46.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.944+0000 7f5a037fe700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a04101240 0x7f5a04197320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:44554/0 (socket says 172.21.15.33:44554) 2024-03-19T02:52:46.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.944+0000 7f5a037fe700 1 -- 172.21.15.33:0/796228615 learned_addr learned my addr 172.21.15.33:0/796228615 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:46.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.944+0000 7f5a03fff700 1 --2- 172.21.15.33:0/796228615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a04102de0 0x7f5a0419e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:46.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.944+0000 7f5a02ffd700 1 --2- 172.21.15.33:0/796228615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a04102440 0x7f5a04197860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:46.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.945+0000 7f5a037fe700 1 -- 172.21.15.33:0/796228615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a04102440 msgr2=0x7f5a04197860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:46.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.945+0000 7f5a037fe700 1 --2- 172.21.15.33:0/796228615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a04102440 0x7f5a04197860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:46.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.945+0000 7f5a037fe700 1 -- 172.21.15.33:0/796228615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a04102de0 msgr2=0x7f5a0419e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:46.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.945+0000 7f5a037fe700 1 --2- 172.21.15.33:0/796228615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a04102de0 0x7f5a0419e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:46.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.945+0000 7f5a037fe700 1 -- 172.21.15.33:0/796228615 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a0419ef80 con 0x7f5a04101240 2024-03-19T02:52:46.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.945+0000 7f5a037fe700 1 --2- 172.21.15.33:0/796228615 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a04101240 0x7f5a04197320 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f59fc00e5b0 tx=0x7f59fc00e8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:46.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.945+0000 7f5a00ff9700 1 -- 172.21.15.33:0/796228615 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59fc0092d0 con 0x7f5a04101240 2024-03-19T02:52:46.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.945+0000 7f5a00ff9700 1 -- 172.21.15.33:0/796228615 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59fc004020 con 0x7f5a04101240 2024-03-19T02:52:46.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.945+0000 7f5a09c27700 1 -- 172.21.15.33:0/796228615 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a0406aa10 con 0x7f5a04101240 2024-03-19T02:52:46.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.945+0000 7f5a00ff9700 1 -- 172.21.15.33:0/796228615 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59fc007760 con 0x7f5a04101240 2024-03-19T02:52:46.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.945+0000 7f5a09c27700 1 -- 172.21.15.33:0/796228615 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a0406af50 con 0x7f5a04101240 2024-03-19T02:52:46.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.946+0000 7f5a09c27700 1 -- 172.21.15.33:0/796228615 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a04068fd0 con 0x7f5a04101240 2024-03-19T02:52:46.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.947+0000 7f5a00ff9700 1 -- 172.21.15.33:0/796228615 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f59fc0041d0 con 0x7f5a04101240 2024-03-19T02:52:46.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.948+0000 7f5a00ff9700 1 --2- 172.21.15.33:0/796228615 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f59f00a2890 0x7f59f00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:46.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.948+0000 7f5a02ffd700 1 --2- 172.21.15.33:0/796228615 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f59f00a2890 0x7f59f00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:46.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.948+0000 7f5a00ff9700 1 -- 172.21.15.33:0/796228615 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f59fc0d1a90 con 0x7f5a04101240 2024-03-19T02:52:46.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.948+0000 7f5a02ffd700 1 --2- 172.21.15.33:0/796228615 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f59f00a2890 0x7f59f00a4d50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f59ec009c60 tx=0x7f59ec009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:46.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:46.951+0000 7f5a00ff9700 1 -- 172.21.15.33:0/796228615 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f59fc01f060 con 0x7f5a04101240 2024-03-19T02:52:47.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:47.170+0000 7f5a09c27700 1 -- 172.21.15.33:0/796228615 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f5a0406b270 con 0x7f59f00a2890 2024-03-19T02:52:47.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:47.184+0000 7f5a00ff9700 1 -- 172.21.15.33:0/796228615 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7f5a0406b270 con 0x7f59f00a2890 2024-03-19T02:52:47.185 INFO:teuthology.orchestra.run.smithi033.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-19T02:52:47.185 INFO:teuthology.orchestra.run.smithi033.stdout:mds.a smithi033 running (2m) 19s ago 2m 13.8M - 19.0.0-2206-gb6ae2491 2312c8e42d00 4b3e4c2dbe20 2024-03-19T02:52:47.185 INFO:teuthology.orchestra.run.smithi033.stdout:mds.b smithi096 running (119s) 52s ago 119s 26.0M - 19.0.0-2206-gb6ae2491 2312c8e42d00 a09bdf94d17e 2024-03-19T02:52:47.186 INFO:teuthology.orchestra.run.smithi033.stdout:mds.c smithi155 running (2m) 53s ago 2m 17.4M - 19.0.0-2206-gb6ae2491 2312c8e42d00 b6f010f3fa3f 2024-03-19T02:52:47.186 INFO:teuthology.orchestra.run.smithi033.stdout:mds.d smithi033 running (2m) 19s ago 2m 16.3M - 19.0.0-2206-gb6ae2491 2312c8e42d00 fb33411f4f2d 2024-03-19T02:52:47.186 INFO:teuthology.orchestra.run.smithi033.stdout:mds.e smithi096 running (2m) 52s ago 2m 18.5M - 19.0.0-2206-gb6ae2491 2312c8e42d00 8c5733c6cf79 2024-03-19T02:52:47.186 INFO:teuthology.orchestra.run.smithi033.stdout:mds.f smithi155 running (2m) 53s ago 2m 16.0M - 19.0.0-2206-gb6ae2491 2312c8e42d00 3b39aaf94f1a 2024-03-19T02:52:47.186 INFO:teuthology.orchestra.run.smithi033.stdout:mds.g smithi033 running (2m) 19s ago 2m 12.8M - 19.0.0-2206-gb6ae2491 2312c8e42d00 d62af71f1410 2024-03-19T02:52:47.186 INFO:teuthology.orchestra.run.smithi033.stdout:mds.h smithi096 running (2m) 52s ago 2m 19.7M - 19.0.0-2206-gb6ae2491 2312c8e42d00 d02dd865bd78 2024-03-19T02:52:47.186 INFO:teuthology.orchestra.run.smithi033.stdout:mds.i smithi155 running (2m) 53s ago 2m 19.5M - 19.0.0-2206-gb6ae2491 2312c8e42d00 3558562598b9 2024-03-19T02:52:47.186 INFO:teuthology.orchestra.run.smithi033.stdout:mds.j smithi033 running (2m) 19s ago 2m 13.8M - 19.0.0-2206-gb6ae2491 2312c8e42d00 deeb835197b6 2024-03-19T02:52:47.186 INFO:teuthology.orchestra.run.smithi033.stdout:mds.k smithi096 running (2m) 52s ago 2m 15.3M - 19.0.0-2206-gb6ae2491 2312c8e42d00 b6f149b36ab3 2024-03-19T02:52:47.186 INFO:teuthology.orchestra.run.smithi033.stdout:mds.l smithi155 running (2m) 53s ago 2m 14.0M - 19.0.0-2206-gb6ae2491 2312c8e42d00 0c647f0e6933 2024-03-19T02:52:47.186 INFO:teuthology.orchestra.run.smithi033.stdout:mgr.x smithi033 *:9283,8765 running (9m) 19s ago 9m 515M - 19.0.0-2206-gb6ae2491 2312c8e42d00 ed11015124be 2024-03-19T02:52:47.186 INFO:teuthology.orchestra.run.smithi033.stdout:mgr.y smithi096 *:8443,8765 running (7m) 52s ago 7m 454M - 19.0.0-2206-gb6ae2491 2312c8e42d00 9e45c9a1db35 2024-03-19T02:52:47.186 INFO:teuthology.orchestra.run.smithi033.stdout:mgr.z smithi155 *:8443,8765 running (7m) 53s ago 7m 452M - 19.0.0-2206-gb6ae2491 2312c8e42d00 eabf96d16ca6 2024-03-19T02:52:47.186 INFO:teuthology.orchestra.run.smithi033.stdout:mon.a smithi033 running (9m) 19s ago 10m 50.7M 2048M 19.0.0-2206-gb6ae2491 2312c8e42d00 24ae7c13ccaa 2024-03-19T02:52:47.187 INFO:teuthology.orchestra.run.smithi033.stdout:mon.b smithi096 running (8m) 52s ago 8m 43.1M 2048M 19.0.0-2206-gb6ae2491 2312c8e42d00 787ee6c286fe 2024-03-19T02:52:47.187 INFO:teuthology.orchestra.run.smithi033.stdout:mon.c smithi155 running (8m) 53s ago 8m 40.1M 2048M 19.0.0-2206-gb6ae2491 2312c8e42d00 326f798dcb9b 2024-03-19T02:52:47.187 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0 smithi033 running (7m) 19s ago 7m 48.2M 4273M 19.0.0-2206-gb6ae2491 2312c8e42d00 3595ce771f88 2024-03-19T02:52:47.187 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1 smithi096 running (7m) 52s ago 7m 54.2M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 3c14e4d75f7b 2024-03-19T02:52:47.187 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2 smithi155 running (6m) 53s ago 6m 47.0M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 9abbde5ee255 2024-03-19T02:52:47.187 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3 smithi033 running (6m) 19s ago 6m 44.8M 4273M 19.0.0-2206-gb6ae2491 2312c8e42d00 d6d277fb8018 2024-03-19T02:52:47.187 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4 smithi096 running (5m) 52s ago 5m 42.7M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 51f69ec22f02 2024-03-19T02:52:47.187 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5 smithi155 running (5m) 53s ago 5m 46.4M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 a99921a4cecf 2024-03-19T02:52:47.187 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6 smithi033 running (5m) 19s ago 5m 45.7M 4273M 19.0.0-2206-gb6ae2491 2312c8e42d00 769f7bd71944 2024-03-19T02:52:47.187 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7 smithi096 running (4m) 52s ago 4m 52.7M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 92c3bf153546 2024-03-19T02:52:47.187 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8 smithi155 running (4m) 53s ago 4m 51.1M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 e95fc8bf30fd 2024-03-19T02:52:47.187 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9 smithi033 running (3m) 19s ago 3m 52.9M 4273M 19.0.0-2206-gb6ae2491 2312c8e42d00 b3b0df9d95fc 2024-03-19T02:52:47.187 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10 smithi096 running (3m) 52s ago 3m 46.0M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 1b7668219643 2024-03-19T02:52:47.187 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11 smithi155 running (2m) 53s ago 2m 50.1M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 40ba55b9d047 2024-03-19T02:52:47.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:47.186+0000 7f5a09c27700 1 -- 172.21.15.33:0/796228615 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f59f00a2890 msgr2=0x7f59f00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:47.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:47.186+0000 7f5a09c27700 1 --2- 172.21.15.33:0/796228615 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f59f00a2890 0x7f59f00a4d50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f59ec009c60 tx=0x7f59ec009260 comp rx=0 tx=0).stop 2024-03-19T02:52:47.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:47.186+0000 7f5a09c27700 1 -- 172.21.15.33:0/796228615 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a04101240 msgr2=0x7f5a04197320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:47.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:47.186+0000 7f5a09c27700 1 --2- 172.21.15.33:0/796228615 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a04101240 0x7f5a04197320 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f59fc00e5b0 tx=0x7f59fc00e8c0 comp rx=0 tx=0).stop 2024-03-19T02:52:47.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:47.187+0000 7f5a09c27700 1 -- 172.21.15.33:0/796228615 shutdown_connections 2024-03-19T02:52:47.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:47.187+0000 7f5a09c27700 1 --2- 172.21.15.33:0/796228615 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f59f00a2890 0x7f59f00a4d50 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:47.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:47.187+0000 7f5a09c27700 1 --2- 172.21.15.33:0/796228615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5a04102de0 0x7f5a0419e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:47.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:47.187+0000 7f5a09c27700 1 --2- 172.21.15.33:0/796228615 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5a04101240 0x7f5a04197320 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:47.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:47.187+0000 7f5a09c27700 1 --2- 172.21.15.33:0/796228615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a04102440 0x7f5a04197860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:47.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:47.187+0000 7f5a09c27700 1 -- 172.21.15.33:0/796228615 >> 172.21.15.33:0/796228615 conn(0x7f5a040fc7e0 msgr2=0x7f5a04105740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:47.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:47.187+0000 7f5a09c27700 1 -- 172.21.15.33:0/796228615 shutdown_connections 2024-03-19T02:52:47.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:47.187+0000 7f5a09c27700 1 -- 172.21.15.33:0/796228615 wait complete. 2024-03-19T02:52:47.841 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-19T02:52:48.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:47 smithi033 ceph-mon[31973]: from='client.15252 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:52:48.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:47 smithi033 ceph-mon[31973]: pgmap v361: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 383 B/s wr, 6 op/s 2024-03-19T02:52:48.088 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:52:48.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:47 smithi096 ceph-mon[35946]: from='client.15252 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:52:48.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:47 smithi096 ceph-mon[35946]: pgmap v361: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 383 B/s wr, 6 op/s 2024-03-19T02:52:48.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:47 smithi155 ceph-mon[35804]: from='client.15252 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:52:48.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:47 smithi155 ceph-mon[35804]: pgmap v361: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 383 B/s wr, 6 op/s 2024-03-19T02:52:49.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.745+0000 7fb90b60a700 1 -- 172.21.15.33:0/1542369389 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb904102e80 msgr2=0x7fb904109730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:49.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.745+0000 7fb90b60a700 1 --2- 172.21.15.33:0/1542369389 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb904102e80 0x7fb904109730 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc00b0d0 tx=0x7fb8fc00b3e0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.745+0000 7fb90b60a700 1 -- 172.21.15.33:0/1542369389 shutdown_connections 2024-03-19T02:52:49.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.745+0000 7fb90b60a700 1 --2- 172.21.15.33:0/1542369389 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb904102e80 0x7fb904109730 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.745+0000 7fb90b60a700 1 --2- 172.21.15.33:0/1542369389 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9041012e0 0x7fb904101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.745+0000 7fb90b60a700 1 --2- 172.21.15.33:0/1542369389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb9041024e0 0x7fb904102940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.745+0000 7fb90b60a700 1 -- 172.21.15.33:0/1542369389 >> 172.21.15.33:0/1542369389 conn(0x7fb9040fc8a0 msgr2=0x7fb9040fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:49.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.746+0000 7fb90b60a700 1 -- 172.21.15.33:0/1542369389 shutdown_connections 2024-03-19T02:52:49.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.746+0000 7fb90b60a700 1 -- 172.21.15.33:0/1542369389 wait complete. 2024-03-19T02:52:49.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.746+0000 7fb90b60a700 1 Processor -- start 2024-03-19T02:52:49.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.746+0000 7fb90b60a700 1 -- start start 2024-03-19T02:52:49.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.746+0000 7fb90b60a700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9041012e0 0x7fb90419a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:49.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.746+0000 7fb90b60a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9041024e0 0x7fb90419ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:49.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.746+0000 7fb90b60a700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb904102e80 0x7fb9041a1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:49.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.746+0000 7fb90b60a700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb90410cd00 con 0x7fb9041012e0 2024-03-19T02:52:49.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.746+0000 7fb90b60a700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb90410cb80 con 0x7fb904102e80 2024-03-19T02:52:49.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.746+0000 7fb90b60a700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb90410ce80 con 0x7fb9041024e0 2024-03-19T02:52:49.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.746+0000 7fb9093a6700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9041012e0 0x7fb90419a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:49.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.746+0000 7fb9093a6700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9041012e0 0x7fb90419a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:44580/0 (socket says 172.21.15.33:44580) 2024-03-19T02:52:49.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.747+0000 7fb9093a6700 1 -- 172.21.15.33:0/2831787137 learned_addr learned my addr 172.21.15.33:0/2831787137 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:49.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.747+0000 7fb909ba7700 1 --2- 172.21.15.33:0/2831787137 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb904102e80 0x7fb9041a1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:49.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.747+0000 7fb908ba5700 1 --2- 172.21.15.33:0/2831787137 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9041024e0 0x7fb90419ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:49.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.747+0000 7fb9093a6700 1 -- 172.21.15.33:0/2831787137 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9041024e0 msgr2=0x7fb90419ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:49.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.747+0000 7fb9093a6700 1 --2- 172.21.15.33:0/2831787137 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9041024e0 0x7fb90419ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.747+0000 7fb9093a6700 1 -- 172.21.15.33:0/2831787137 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb904102e80 msgr2=0x7fb9041a1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:49.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.747+0000 7fb9093a6700 1 --2- 172.21.15.33:0/2831787137 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb904102e80 0x7fb9041a1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.747+0000 7fb9093a6700 1 -- 172.21.15.33:0/2831787137 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9041a22a0 con 0x7fb9041012e0 2024-03-19T02:52:49.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.747+0000 7fb909ba7700 1 --2- 172.21.15.33:0/2831787137 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb904102e80 0x7fb9041a1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:49.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.747+0000 7fb9093a6700 1 --2- 172.21.15.33:0/2831787137 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9041012e0 0x7fb90419a640 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb90000e980 tx=0x7fb90000ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:49.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.747+0000 7fb8fa7fc700 1 -- 172.21.15.33:0/2831787137 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb900004230 con 0x7fb9041012e0 2024-03-19T02:52:49.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.747+0000 7fb90b60a700 1 -- 172.21.15.33:0/2831787137 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9041a2590 con 0x7fb9041012e0 2024-03-19T02:52:49.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.747+0000 7fb8fa7fc700 1 -- 172.21.15.33:0/2831787137 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb9000043d0 con 0x7fb9041012e0 2024-03-19T02:52:49.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.747+0000 7fb8fa7fc700 1 -- 172.21.15.33:0/2831787137 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb900003c90 con 0x7fb9041012e0 2024-03-19T02:52:49.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.747+0000 7fb90b60a700 1 -- 172.21.15.33:0/2831787137 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9041a2a80 con 0x7fb9041012e0 2024-03-19T02:52:49.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.748+0000 7fb90b60a700 1 -- 172.21.15.33:0/2831787137 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb904068fd0 con 0x7fb9041012e0 2024-03-19T02:52:49.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.749+0000 7fb8fa7fc700 1 -- 172.21.15.33:0/2831787137 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb90000c810 con 0x7fb9041012e0 2024-03-19T02:52:49.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.750+0000 7fb8fa7fc700 1 --2- 172.21.15.33:0/2831787137 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb8f00a2890 0x7fb8f00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:49.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.750+0000 7fb908ba5700 1 --2- 172.21.15.33:0/2831787137 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb8f00a2890 0x7fb8f00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:49.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.750+0000 7fb8fa7fc700 1 -- 172.21.15.33:0/2831787137 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7fb9000cf060 con 0x7fb9041012e0 2024-03-19T02:52:49.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.751+0000 7fb908ba5700 1 --2- 172.21.15.33:0/2831787137 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb8f00a2890 0x7fb8f00a4d50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4009c60 tx=0x7fb8f4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:49.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.755+0000 7fb8fa7fc700 1 -- 172.21.15.33:0/2831787137 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb90009bd70 con 0x7fb9041012e0 2024-03-19T02:52:49.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.969+0000 7fb90b60a700 1 -- 172.21.15.33:0/2831787137 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fb9040fea10 con 0x7fb8f00a2890 2024-03-19T02:52:49.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.975+0000 7fb8fa7fc700 1 -- 172.21.15.33:0/2831787137 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fb9040fea10 con 0x7fb8f00a2890 2024-03-19T02:52:49.975 INFO:teuthology.orchestra.run.smithi033.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-19T02:52:49.976 INFO:teuthology.orchestra.run.smithi033.stdout:mds.all 12/12 55s ago 2m smithi033=a;smithi033=d;smithi033=g;smithi033=j;smithi096=b;smithi096=e;smithi096=h;smithi096=k;smithi155=c;smithi155=f;smithi155=i;smithi155=l;count:12 2024-03-19T02:52:49.976 INFO:teuthology.orchestra.run.smithi033.stdout:mgr 3/3 55s ago 8m smithi033=x;smithi096=y;smithi155=z;count:3 2024-03-19T02:52:49.976 INFO:teuthology.orchestra.run.smithi033.stdout:mon 3/3 55s ago 8m smithi033:172.21.15.33=a;smithi096:172.21.15.96=b;smithi155:172.21.15.155=c;count:3 2024-03-19T02:52:49.976 INFO:teuthology.orchestra.run.smithi033.stdout:osd 12 55s ago - 2024-03-19T02:52:49.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.978+0000 7fb90b60a700 1 -- 172.21.15.33:0/2831787137 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb8f00a2890 msgr2=0x7fb8f00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:49.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.978+0000 7fb90b60a700 1 --2- 172.21.15.33:0/2831787137 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb8f00a2890 0x7fb8f00a4d50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4009c60 tx=0x7fb8f4009260 comp rx=0 tx=0).stop 2024-03-19T02:52:49.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.978+0000 7fb90b60a700 1 -- 172.21.15.33:0/2831787137 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9041012e0 msgr2=0x7fb90419a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:49.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.978+0000 7fb90b60a700 1 --2- 172.21.15.33:0/2831787137 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9041012e0 0x7fb90419a640 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb90000e980 tx=0x7fb90000ec90 comp rx=0 tx=0).stop 2024-03-19T02:52:49.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.978+0000 7fb90b60a700 1 -- 172.21.15.33:0/2831787137 shutdown_connections 2024-03-19T02:52:49.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.978+0000 7fb90b60a700 1 --2- 172.21.15.33:0/2831787137 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fb8f00a2890 0x7fb8f00a4d50 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.978+0000 7fb90b60a700 1 --2- 172.21.15.33:0/2831787137 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb904102e80 0x7fb9041a1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.978+0000 7fb90b60a700 1 --2- 172.21.15.33:0/2831787137 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb9041012e0 0x7fb90419a640 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.978+0000 7fb90b60a700 1 --2- 172.21.15.33:0/2831787137 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb9041024e0 0x7fb90419ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:49.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.978+0000 7fb90b60a700 1 -- 172.21.15.33:0/2831787137 >> 172.21.15.33:0/2831787137 conn(0x7fb9040fc8a0 msgr2=0x7fb9040fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:49.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.978+0000 7fb90b60a700 1 -- 172.21.15.33:0/2831787137 shutdown_connections 2024-03-19T02:52:49.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:49.978+0000 7fb90b60a700 1 -- 172.21.15.33:0/2831787137 wait complete. 2024-03-19T02:52:50.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:49 smithi033 ceph-mon[31973]: pgmap v362: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 319 B/s wr, 6 op/s 2024-03-19T02:52:50.068 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:49 smithi155 ceph-mon[35804]: pgmap v362: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 319 B/s wr, 6 op/s 2024-03-19T02:52:50.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:49 smithi096 ceph-mon[35946]: pgmap v362: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 319 B/s wr, 6 op/s 2024-03-19T02:52:50.797 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-19T02:52:51.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:50 smithi033 ceph-mon[31973]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:52:51.050 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:52:51.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:50 smithi096 ceph-mon[35946]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:52:51.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:50 smithi155 ceph-mon[35804]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:52:52.145 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:51 smithi033 ceph-mon[31973]: pgmap v363: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 289 B/s wr, 6 op/s 2024-03-19T02:52:52.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:51 smithi096 ceph-mon[35946]: pgmap v363: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 289 B/s wr, 6 op/s 2024-03-19T02:52:52.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:51 smithi155 ceph-mon[35804]: pgmap v363: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 289 B/s wr, 6 op/s 2024-03-19T02:52:52.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.694+0000 7f5b05125700 1 -- 172.21.15.33:0/1856788137 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b00102e50 msgr2=0x7f5b00109700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:52.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.694+0000 7f5b05125700 1 --2- 172.21.15.33:0/1856788137 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b00102e50 0x7f5b00109700 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5af800b0d0 tx=0x7f5af800b3e0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.694+0000 7f5b05125700 1 -- 172.21.15.33:0/1856788137 shutdown_connections 2024-03-19T02:52:52.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.694+0000 7f5b05125700 1 --2- 172.21.15.33:0/1856788137 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b00102e50 0x7f5b00109700 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.694+0000 7f5b05125700 1 --2- 172.21.15.33:0/1856788137 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b001012b0 0x7f5b001016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.694+0000 7f5b05125700 1 --2- 172.21.15.33:0/1856788137 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b001024b0 0x7f5b00102910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.694+0000 7f5b05125700 1 -- 172.21.15.33:0/1856788137 >> 172.21.15.33:0/1856788137 conn(0x7f5b000fc870 msgr2=0x7f5b000fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:52.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.695+0000 7f5b05125700 1 -- 172.21.15.33:0/1856788137 shutdown_connections 2024-03-19T02:52:52.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.695+0000 7f5b05125700 1 -- 172.21.15.33:0/1856788137 wait complete. 2024-03-19T02:52:52.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.695+0000 7f5b05125700 1 Processor -- start 2024-03-19T02:52:52.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.695+0000 7f5b05125700 1 -- start start 2024-03-19T02:52:52.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.696+0000 7f5b05125700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b001012b0 0x7f5b00198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:52.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.696+0000 7f5b05125700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b001024b0 0x7f5b001989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:52.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.696+0000 7f5b05125700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b00102e50 0x7f5b0019fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:52.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.696+0000 7f5b05125700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5b0010ccd0 con 0x7f5b00102e50 2024-03-19T02:52:52.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.696+0000 7f5b05125700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5b0010cb50 con 0x7f5b001024b0 2024-03-19T02:52:52.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.696+0000 7f5b05125700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5b0010ce50 con 0x7f5b001012b0 2024-03-19T02:52:52.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.696+0000 7f5aff59e700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b00102e50 0x7f5b0019fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:52.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.696+0000 7f5aff59e700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b00102e50 0x7f5b0019fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:44596/0 (socket says 172.21.15.33:44596) 2024-03-19T02:52:52.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.696+0000 7f5aff59e700 1 -- 172.21.15.33:0/1991417396 learned_addr learned my addr 172.21.15.33:0/1991417396 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:52.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.696+0000 7f5afe59c700 1 --2- 172.21.15.33:0/1991417396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b001024b0 0x7f5b001989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:52.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.696+0000 7f5afed9d700 1 --2- 172.21.15.33:0/1991417396 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b001012b0 0x7f5b00198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:52.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.696+0000 7f5aff59e700 1 -- 172.21.15.33:0/1991417396 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b001012b0 msgr2=0x7f5b00198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:52.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.696+0000 7f5aff59e700 1 --2- 172.21.15.33:0/1991417396 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b001012b0 0x7f5b00198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.696+0000 7f5aff59e700 1 -- 172.21.15.33:0/1991417396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b001024b0 msgr2=0x7f5b001989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:52.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.696+0000 7f5aff59e700 1 --2- 172.21.15.33:0/1991417396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b001024b0 0x7f5b001989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.696+0000 7f5aff59e700 1 -- 172.21.15.33:0/1991417396 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b001a00e0 con 0x7f5b00102e50 2024-03-19T02:52:52.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.696+0000 7f5aff59e700 1 --2- 172.21.15.33:0/1991417396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b00102e50 0x7f5b0019fa40 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f5af8004ce0 tx=0x7f5af8004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:52.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.696+0000 7f5ae7fff700 1 -- 172.21.15.33:0/1991417396 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af801f030 con 0x7f5b00102e50 2024-03-19T02:52:52.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.696+0000 7f5ae7fff700 1 -- 172.21.15.33:0/1991417396 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5af8005040 con 0x7f5b00102e50 2024-03-19T02:52:52.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.696+0000 7f5ae7fff700 1 -- 172.21.15.33:0/1991417396 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af801b690 con 0x7f5b00102e50 2024-03-19T02:52:52.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.697+0000 7f5b05125700 1 -- 172.21.15.33:0/1991417396 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b001a0370 con 0x7f5b00102e50 2024-03-19T02:52:52.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.697+0000 7f5b05125700 1 -- 172.21.15.33:0/1991417396 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b001a07d0 con 0x7f5b00102e50 2024-03-19T02:52:52.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.697+0000 7f5b05125700 1 -- 172.21.15.33:0/1991417396 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b00068fd0 con 0x7f5b00102e50 2024-03-19T02:52:52.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.699+0000 7f5ae7fff700 1 -- 172.21.15.33:0/1991417396 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5af8022030 con 0x7f5b00102e50 2024-03-19T02:52:52.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.699+0000 7f5ae7fff700 1 --2- 172.21.15.33:0/1991417396 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5ae80a2890 0x7f5ae80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:52.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.699+0000 7f5afed9d700 1 --2- 172.21.15.33:0/1991417396 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5ae80a2890 0x7f5ae80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:52.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.699+0000 7f5ae7fff700 1 -- 172.21.15.33:0/1991417396 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f5af800f020 con 0x7f5b00102e50 2024-03-19T02:52:52.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.700+0000 7f5afed9d700 1 --2- 172.21.15.33:0/1991417396 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5ae80a2890 0x7f5ae80a4d50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5b00102310 tx=0x7f5af0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:52.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.704+0000 7f5ae7fff700 1 -- 172.21.15.33:0/1991417396 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5af8029020 con 0x7f5b00102e50 2024-03-19T02:52:52.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.915+0000 7f5b05125700 1 -- 172.21.15.33:0/1991417396 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f5b000fea00 con 0x7f5ae80a2890 2024-03-19T02:52:52.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.917+0000 7f5ae7fff700 1 -- 172.21.15.33:0/1991417396 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f5b000fea00 con 0x7f5ae80a2890 2024-03-19T02:52:52.918 INFO:teuthology.orchestra.run.smithi033.stdout:HOST ADDR LABELS STATUS 2024-03-19T02:52:52.918 INFO:teuthology.orchestra.run.smithi033.stdout:smithi033 172.21.15.33 2024-03-19T02:52:52.918 INFO:teuthology.orchestra.run.smithi033.stdout:smithi096 172.21.15.96 2024-03-19T02:52:52.918 INFO:teuthology.orchestra.run.smithi033.stdout:smithi155 172.21.15.155 2024-03-19T02:52:52.918 INFO:teuthology.orchestra.run.smithi033.stdout:3 hosts in cluster 2024-03-19T02:52:52.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.919+0000 7f5b05125700 1 -- 172.21.15.33:0/1991417396 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5ae80a2890 msgr2=0x7f5ae80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:52.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.919+0000 7f5b05125700 1 --2- 172.21.15.33:0/1991417396 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5ae80a2890 0x7f5ae80a4d50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5b00102310 tx=0x7f5af0009260 comp rx=0 tx=0).stop 2024-03-19T02:52:52.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.919+0000 7f5b05125700 1 -- 172.21.15.33:0/1991417396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b00102e50 msgr2=0x7f5b0019fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:52.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.919+0000 7f5b05125700 1 --2- 172.21.15.33:0/1991417396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b00102e50 0x7f5b0019fa40 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f5af8004ce0 tx=0x7f5af8004d10 comp rx=0 tx=0).stop 2024-03-19T02:52:52.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.919+0000 7f5b05125700 1 -- 172.21.15.33:0/1991417396 shutdown_connections 2024-03-19T02:52:52.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.919+0000 7f5b05125700 1 --2- 172.21.15.33:0/1991417396 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5ae80a2890 0x7f5ae80a4d50 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.919+0000 7f5b05125700 1 --2- 172.21.15.33:0/1991417396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5b00102e50 0x7f5b0019fa40 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.919+0000 7f5b05125700 1 --2- 172.21.15.33:0/1991417396 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5b001012b0 0x7f5b00198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.919+0000 7f5b05125700 1 --2- 172.21.15.33:0/1991417396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5b001024b0 0x7f5b001989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:52.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.919+0000 7f5b05125700 1 -- 172.21.15.33:0/1991417396 >> 172.21.15.33:0/1991417396 conn(0x7f5b000fc870 msgr2=0x7f5b000fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:52.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.919+0000 7f5b05125700 1 -- 172.21.15.33:0/1991417396 shutdown_connections 2024-03-19T02:52:52.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:52.919+0000 7f5b05125700 1 -- 172.21.15.33:0/1991417396 wait complete. 2024-03-19T02:52:53.686 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-19T02:52:53.937 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:52:54.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:53 smithi033 ceph-mon[31973]: from='client.15264 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:52:54.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:53 smithi033 ceph-mon[31973]: pgmap v364: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-19T02:52:54.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:53 smithi096 ceph-mon[35946]: from='client.15264 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:52:54.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:53 smithi096 ceph-mon[35946]: pgmap v364: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-19T02:52:54.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:53 smithi155 ceph-mon[35804]: from='client.15264 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:52:54.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:53 smithi155 ceph-mon[35804]: pgmap v364: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-19T02:52:55.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.676+0000 7fa5e103e700 1 -- 172.21.15.33:0/3467980104 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa5dc103570 msgr2=0x7fa5dc1039d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:55.679 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.676+0000 7fa5e103e700 1 --2- 172.21.15.33:0/3467980104 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa5dc103570 0x7fa5dc1039d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa5d00099c0 tx=0x7fa5d0009cd0 comp rx=0 tx=0).stop 2024-03-19T02:52:55.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.676+0000 7fa5e103e700 1 -- 172.21.15.33:0/3467980104 shutdown_connections 2024-03-19T02:52:55.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.676+0000 7fa5e103e700 1 --2- 172.21.15.33:0/3467980104 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5dc103f10 0x7fa5dc10a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:55.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.676+0000 7fa5e103e700 1 --2- 172.21.15.33:0/3467980104 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5dc102370 0x7fa5dc102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:55.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.676+0000 7fa5e103e700 1 --2- 172.21.15.33:0/3467980104 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa5dc103570 0x7fa5dc1039d0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:55.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.676+0000 7fa5e103e700 1 -- 172.21.15.33:0/3467980104 >> 172.21.15.33:0/3467980104 conn(0x7fa5dc0fd910 msgr2=0x7fa5dc0ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:55.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.676+0000 7fa5e103e700 1 -- 172.21.15.33:0/3467980104 shutdown_connections 2024-03-19T02:52:55.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.677+0000 7fa5e103e700 1 -- 172.21.15.33:0/3467980104 wait complete. 2024-03-19T02:52:55.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.677+0000 7fa5e103e700 1 Processor -- start 2024-03-19T02:52:55.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.677+0000 7fa5e103e700 1 -- start start 2024-03-19T02:52:55.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.678+0000 7fa5e103e700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa5dc102370 0x7fa5dc198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:55.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.678+0000 7fa5e103e700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5dc103570 0x7fa5dc1989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:55.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.678+0000 7fa5e103e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5dc103f10 0x7fa5dc19fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:55.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.678+0000 7fa5e103e700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa5dc10ddf0 con 0x7fa5dc102370 2024-03-19T02:52:55.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.678+0000 7fa5e103e700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa5dc10dc70 con 0x7fa5dc103570 2024-03-19T02:52:55.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.678+0000 7fa5e103e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa5dc10df70 con 0x7fa5dc103f10 2024-03-19T02:52:55.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.678+0000 7fa5dad9d700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa5dc102370 0x7fa5dc198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:55.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.678+0000 7fa5dad9d700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa5dc102370 0x7fa5dc198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42068/0 (socket says 172.21.15.33:42068) 2024-03-19T02:52:55.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.678+0000 7fa5dad9d700 1 -- 172.21.15.33:0/4060934170 learned_addr learned my addr 172.21.15.33:0/4060934170 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:55.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.678+0000 7fa5db59e700 1 --2- 172.21.15.33:0/4060934170 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5dc103f10 0x7fa5dc19fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:55.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.678+0000 7fa5da59c700 1 --2- 172.21.15.33:0/4060934170 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5dc103570 0x7fa5dc1989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:55.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.678+0000 7fa5dad9d700 1 -- 172.21.15.33:0/4060934170 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5dc103f10 msgr2=0x7fa5dc19fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:55.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.678+0000 7fa5dad9d700 1 --2- 172.21.15.33:0/4060934170 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5dc103f10 0x7fa5dc19fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:55.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.678+0000 7fa5dad9d700 1 -- 172.21.15.33:0/4060934170 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5dc103570 msgr2=0x7fa5dc1989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:55.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.678+0000 7fa5dad9d700 1 --2- 172.21.15.33:0/4060934170 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5dc103570 0x7fa5dc1989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:55.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.678+0000 7fa5dad9d700 1 -- 172.21.15.33:0/4060934170 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5dc1a00c0 con 0x7fa5dc102370 2024-03-19T02:52:55.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.678+0000 7fa5dad9d700 1 --2- 172.21.15.33:0/4060934170 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa5dc102370 0x7fa5dc198460 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc00e9a0 tx=0x7fa5cc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:55.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.678+0000 7fa5c3fff700 1 -- 172.21.15.33:0/4060934170 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc004220 con 0x7fa5dc102370 2024-03-19T02:52:55.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.679+0000 7fa5c3fff700 1 -- 172.21.15.33:0/4060934170 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa5cc0043c0 con 0x7fa5dc102370 2024-03-19T02:52:55.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.679+0000 7fa5e103e700 1 -- 172.21.15.33:0/4060934170 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5dc1a03b0 con 0x7fa5dc102370 2024-03-19T02:52:55.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.679+0000 7fa5c3fff700 1 -- 172.21.15.33:0/4060934170 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc003ce0 con 0x7fa5dc102370 2024-03-19T02:52:55.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.679+0000 7fa5e103e700 1 -- 172.21.15.33:0/4060934170 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5dc1a08f0 con 0x7fa5dc102370 2024-03-19T02:52:55.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.680+0000 7fa5e103e700 1 -- 172.21.15.33:0/4060934170 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5dc068fd0 con 0x7fa5dc102370 2024-03-19T02:52:55.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.681+0000 7fa5c3fff700 1 -- 172.21.15.33:0/4060934170 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa5cc00c810 con 0x7fa5dc102370 2024-03-19T02:52:55.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.681+0000 7fa5c3fff700 1 --2- 172.21.15.33:0/4060934170 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fa5c40a2890 0x7fa5c40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:55.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.681+0000 7fa5da59c700 1 --2- 172.21.15.33:0/4060934170 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fa5c40a2890 0x7fa5c40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:55.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.682+0000 7fa5c3fff700 1 -- 172.21.15.33:0/4060934170 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7fa5cc0cefb0 con 0x7fa5dc102370 2024-03-19T02:52:55.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.682+0000 7fa5da59c700 1 --2- 172.21.15.33:0/4060934170 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fa5c40a2890 0x7fa5c40a4d50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0000c00 tx=0x7fa5d0005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:55.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.686+0000 7fa5c3fff700 1 -- 172.21.15.33:0/4060934170 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa5cc09bcc0 con 0x7fa5dc102370 2024-03-19T02:52:55.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.901+0000 7fa5e103e700 1 -- 172.21.15.33:0/4060934170 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fa5dc062750 con 0x7fa5c40a2890 2024-03-19T02:52:55.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.903+0000 7fa5c3fff700 1 -- 172.21.15.33:0/4060934170 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fa5dc062750 con 0x7fa5c40a2890 2024-03-19T02:52:55.904 INFO:teuthology.orchestra.run.smithi033.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-19T02:52:55.904 INFO:teuthology.orchestra.run.smithi033.stdout:smithi033 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5330008Q400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-19T02:52:55.904 INFO:teuthology.orchestra.run.smithi033.stdout:smithi096 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400NH400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-19T02:52:55.904 INFO:teuthology.orchestra.run.smithi033.stdout:smithi155 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GW400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-19T02:52:55.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.904+0000 7fa5e103e700 1 -- 172.21.15.33:0/4060934170 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fa5c40a2890 msgr2=0x7fa5c40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:55.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.904+0000 7fa5e103e700 1 --2- 172.21.15.33:0/4060934170 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fa5c40a2890 0x7fa5c40a4d50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0000c00 tx=0x7fa5d0005fb0 comp rx=0 tx=0).stop 2024-03-19T02:52:55.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.905+0000 7fa5e103e700 1 -- 172.21.15.33:0/4060934170 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa5dc102370 msgr2=0x7fa5dc198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:55.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.905+0000 7fa5e103e700 1 --2- 172.21.15.33:0/4060934170 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa5dc102370 0x7fa5dc198460 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc00e9a0 tx=0x7fa5cc00ecb0 comp rx=0 tx=0).stop 2024-03-19T02:52:55.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.905+0000 7fa5e103e700 1 -- 172.21.15.33:0/4060934170 shutdown_connections 2024-03-19T02:52:55.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.905+0000 7fa5e103e700 1 --2- 172.21.15.33:0/4060934170 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fa5c40a2890 0x7fa5c40a4d50 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:55.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.905+0000 7fa5e103e700 1 --2- 172.21.15.33:0/4060934170 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5dc103f10 0x7fa5dc19fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:55.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.905+0000 7fa5e103e700 1 --2- 172.21.15.33:0/4060934170 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa5dc102370 0x7fa5dc198460 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:55.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.905+0000 7fa5e103e700 1 --2- 172.21.15.33:0/4060934170 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5dc103570 0x7fa5dc1989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:55.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.905+0000 7fa5e103e700 1 -- 172.21.15.33:0/4060934170 >> 172.21.15.33:0/4060934170 conn(0x7fa5dc0fd910 msgr2=0x7fa5dc1068f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:55.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.905+0000 7fa5e103e700 1 -- 172.21.15.33:0/4060934170 shutdown_connections 2024-03-19T02:52:55.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:55.905+0000 7fa5e103e700 1 -- 172.21.15.33:0/4060934170 wait complete. 2024-03-19T02:52:56.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:55 smithi033 ceph-mon[31973]: pgmap v365: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-19T02:52:56.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:55 smithi096 ceph-mon[35946]: pgmap v365: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-19T02:52:56.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:55 smithi155 ceph-mon[35804]: pgmap v365: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-19T02:52:56.591 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-19T02:52:56.601 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi033.front.sepia.ceph.com 2024-03-19T02:52:56.601 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-19T02:52:56.848 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:52:56.969 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:56 smithi033 ceph-mon[31973]: from='client.15270 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:52:57.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:56 smithi096 ceph-mon[35946]: from='client.15270 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:52:57.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:56 smithi155 ceph-mon[35804]: from='client.15270 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:52:57.896 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:57 smithi033 ceph-mon[31973]: pgmap v366: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:52:58.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:57 smithi096 ceph-mon[35946]: pgmap v366: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:52:58.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:57 smithi155 ceph-mon[35804]: pgmap v366: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:52:58.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.445+0000 7f39999cb700 1 -- 172.21.15.33:0/1264569487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3994103ee0 msgr2=0x7f399410a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:58.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.445+0000 7f39999cb700 1 --2- 172.21.15.33:0/1264569487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3994103ee0 0x7f399410a790 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f398c00b0d0 tx=0x7f398c00b3e0 comp rx=0 tx=0).stop 2024-03-19T02:52:58.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.445+0000 7f39999cb700 1 -- 172.21.15.33:0/1264569487 shutdown_connections 2024-03-19T02:52:58.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.445+0000 7f39999cb700 1 --2- 172.21.15.33:0/1264569487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3994103ee0 0x7f399410a790 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:58.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.445+0000 7f39999cb700 1 --2- 172.21.15.33:0/1264569487 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3994102340 0x7f3994102760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:58.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.445+0000 7f39999cb700 1 --2- 172.21.15.33:0/1264569487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3994103540 0x7f39941039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:58.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.445+0000 7f39999cb700 1 -- 172.21.15.33:0/1264569487 >> 172.21.15.33:0/1264569487 conn(0x7f39940fd920 msgr2=0x7f39940ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:58.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.445+0000 7f39999cb700 1 -- 172.21.15.33:0/1264569487 shutdown_connections 2024-03-19T02:52:58.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.445+0000 7f39999cb700 1 -- 172.21.15.33:0/1264569487 wait complete. 2024-03-19T02:52:58.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.446+0000 7f39999cb700 1 Processor -- start 2024-03-19T02:52:58.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.446+0000 7f39999cb700 1 -- start start 2024-03-19T02:52:58.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.446+0000 7f39999cb700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3994102340 0x7f39941983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:58.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.446+0000 7f39999cb700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3994103540 0x7f3994198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:58.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.446+0000 7f39999cb700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3994103ee0 0x7f399419f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:58.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.446+0000 7f39999cb700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f399410dde0 con 0x7f3994102340 2024-03-19T02:52:58.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.446+0000 7f39999cb700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f399410dc60 con 0x7f3994103540 2024-03-19T02:52:58.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.446+0000 7f39999cb700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f399410df60 con 0x7f3994103ee0 2024-03-19T02:52:58.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.446+0000 7f3992ffd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3994102340 0x7f39941983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:58.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.446+0000 7f3992ffd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3994102340 0x7f39941983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42082/0 (socket says 172.21.15.33:42082) 2024-03-19T02:52:58.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.446+0000 7f3992ffd700 1 -- 172.21.15.33:0/2328233859 learned_addr learned my addr 172.21.15.33:0/2328233859 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:52:58.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.446+0000 7f39927fc700 1 --2- 172.21.15.33:0/2328233859 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3994103540 0x7f3994198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:58.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.446+0000 7f39937fe700 1 --2- 172.21.15.33:0/2328233859 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3994103ee0 0x7f399419f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:58.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.447+0000 7f3992ffd700 1 -- 172.21.15.33:0/2328233859 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3994103ee0 msgr2=0x7f399419f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:58.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.447+0000 7f3992ffd700 1 --2- 172.21.15.33:0/2328233859 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3994103ee0 0x7f399419f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:58.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.447+0000 7f3992ffd700 1 -- 172.21.15.33:0/2328233859 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3994103540 msgr2=0x7f3994198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:58.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.447+0000 7f3992ffd700 1 --2- 172.21.15.33:0/2328233859 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3994103540 0x7f3994198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:58.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.447+0000 7f3992ffd700 1 -- 172.21.15.33:0/2328233859 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39941a0050 con 0x7f3994102340 2024-03-19T02:52:58.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.447+0000 7f39927fc700 1 --2- 172.21.15.33:0/2328233859 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3994103540 0x7f3994198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:52:58.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.447+0000 7f3992ffd700 1 --2- 172.21.15.33:0/2328233859 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3994102340 0x7f39941983f0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f398800e980 tx=0x7f398800ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:58.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.447+0000 7f39989c9700 1 -- 172.21.15.33:0/2328233859 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3988004230 con 0x7f3994102340 2024-03-19T02:52:58.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.447+0000 7f39989c9700 1 -- 172.21.15.33:0/2328233859 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f39880043d0 con 0x7f3994102340 2024-03-19T02:52:58.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.447+0000 7f39999cb700 1 -- 172.21.15.33:0/2328233859 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39941a0340 con 0x7f3994102340 2024-03-19T02:52:58.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.447+0000 7f39989c9700 1 -- 172.21.15.33:0/2328233859 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3988003c90 con 0x7f3994102340 2024-03-19T02:52:58.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.447+0000 7f39999cb700 1 -- 172.21.15.33:0/2328233859 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39941a0830 con 0x7f3994102340 2024-03-19T02:52:58.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.448+0000 7f39999cb700 1 -- 172.21.15.33:0/2328233859 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3994068fd0 con 0x7f3994102340 2024-03-19T02:52:58.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.449+0000 7f39989c9700 1 -- 172.21.15.33:0/2328233859 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f398800c810 con 0x7f3994102340 2024-03-19T02:52:58.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.450+0000 7f39989c9700 1 --2- 172.21.15.33:0/2328233859 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f397c0a2890 0x7f397c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:52:58.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.450+0000 7f39927fc700 1 --2- 172.21.15.33:0/2328233859 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f397c0a2890 0x7f397c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:52:58.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.450+0000 7f39989c9700 1 -- 172.21.15.33:0/2328233859 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f39880cf140 con 0x7f3994102340 2024-03-19T02:52:58.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.450+0000 7f39927fc700 1 --2- 172.21.15.33:0/2328233859 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f397c0a2890 0x7f397c0a4d50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3984011490 tx=0x7f3984009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:52:58.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.453+0000 7f39989c9700 1 -- 172.21.15.33:0/2328233859 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f398809be50 con 0x7f3994102340 2024-03-19T02:52:58.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.707+0000 7f39999cb700 1 -- 172.21.15.33:0/2328233859 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f399402dc70 con 0x7f3994102340 2024-03-19T02:52:58.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.707+0000 7f39989c9700 1 -- 172.21.15.33:0/2328233859 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) ==== 76+0+3048 (secure 0 0 0) 0x7f398809b750 con 0x7f3994102340 2024-03-19T02:52:58.709 INFO:teuthology.orchestra.run.smithi033.stdout:e24 2024-03-19T02:52:58.709 INFO:teuthology.orchestra.run.smithi033.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-19T02:52:58.709 INFO:teuthology.orchestra.run.smithi033.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-19T02:52:58.709 INFO:teuthology.orchestra.run.smithi033.stdout:legacy client fscid: 1 2024-03-19T02:52:58.709 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:52:58.709 INFO:teuthology.orchestra.run.smithi033.stdout:Filesystem 'cephfs' (1) 2024-03-19T02:52:58.709 INFO:teuthology.orchestra.run.smithi033.stdout:fs_name cephfs 2024-03-19T02:52:58.709 INFO:teuthology.orchestra.run.smithi033.stdout:epoch 24 2024-03-19T02:52:58.709 INFO:teuthology.orchestra.run.smithi033.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-19T02:52:58.709 INFO:teuthology.orchestra.run.smithi033.stdout:created 2024-03-19T02:50:52.201798+0000 2024-03-19T02:52:58.709 INFO:teuthology.orchestra.run.smithi033.stdout:modified 2024-03-19T02:51:19.076317+0000 2024-03-19T02:52:58.710 INFO:teuthology.orchestra.run.smithi033.stdout:tableserver 0 2024-03-19T02:52:58.710 INFO:teuthology.orchestra.run.smithi033.stdout:root 0 2024-03-19T02:52:58.710 INFO:teuthology.orchestra.run.smithi033.stdout:session_timeout 300 2024-03-19T02:52:58.710 INFO:teuthology.orchestra.run.smithi033.stdout:session_autoclose 300 2024-03-19T02:52:58.710 INFO:teuthology.orchestra.run.smithi033.stdout:max_file_size 1099511627776 2024-03-19T02:52:58.710 INFO:teuthology.orchestra.run.smithi033.stdout:max_xattr_size 65536 2024-03-19T02:52:58.710 INFO:teuthology.orchestra.run.smithi033.stdout:required_client_features {} 2024-03-19T02:52:58.710 INFO:teuthology.orchestra.run.smithi033.stdout:last_failure 0 2024-03-19T02:52:58.711 INFO:teuthology.orchestra.run.smithi033.stdout:last_failure_osd_epoch 0 2024-03-19T02:52:58.711 INFO:teuthology.orchestra.run.smithi033.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-19T02:52:58.711 INFO:teuthology.orchestra.run.smithi033.stdout:max_mds 3 2024-03-19T02:52:58.711 INFO:teuthology.orchestra.run.smithi033.stdout:in 0,1,2 2024-03-19T02:52:58.711 INFO:teuthology.orchestra.run.smithi033.stdout:up {0=24509,1=24488,2=24457} 2024-03-19T02:52:58.711 INFO:teuthology.orchestra.run.smithi033.stdout:failed 2024-03-19T02:52:58.711 INFO:teuthology.orchestra.run.smithi033.stdout:damaged 2024-03-19T02:52:58.711 INFO:teuthology.orchestra.run.smithi033.stdout:stopped 2024-03-19T02:52:58.711 INFO:teuthology.orchestra.run.smithi033.stdout:data_pools [3,4] 2024-03-19T02:52:58.711 INFO:teuthology.orchestra.run.smithi033.stdout:metadata_pool 2 2024-03-19T02:52:58.711 INFO:teuthology.orchestra.run.smithi033.stdout:inline_data disabled 2024-03-19T02:52:58.711 INFO:teuthology.orchestra.run.smithi033.stdout:balancer 2024-03-19T02:52:58.711 INFO:teuthology.orchestra.run.smithi033.stdout:bal_rank_mask -1 2024-03-19T02:52:58.712 INFO:teuthology.orchestra.run.smithi033.stdout:standby_count_wanted 0 2024-03-19T02:52:58.712 INFO:teuthology.orchestra.run.smithi033.stdout:qdb_cluster leader: 24509 members: 24457,24509,24488 2024-03-19T02:52:58.712 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.b{0:24509} state up:active seq 3 addr [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T02:52:58.712 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.e{0:24455} state up:standby-replay seq 1 addr [v2:172.21.15.96:6834/3551193617,v1:172.21.15.96:6835/3551193617] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T02:52:58.712 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.h{1:24488} state up:active seq 12 addr [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T02:52:58.712 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.c{1:24436} state up:standby-replay seq 1 addr [v2:172.21.15.155:6836/1429349330,v1:172.21.15.155:6837/1429349330] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T02:52:58.712 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.i{2:24457} state up:active seq 11 addr [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T02:52:58.712 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.f{2:24427} state up:standby-replay seq 1 addr [v2:172.21.15.155:6834/1274854977,v1:172.21.15.155:6835/1274854977] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T02:52:58.712 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:52:58.712 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:52:58.712 INFO:teuthology.orchestra.run.smithi033.stdout:Standby daemons: 2024-03-19T02:52:58.712 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:52:58.712 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.j{-1:14526} state up:standby seq 1 addr [v2:172.21.15.33:6834/973717150,v1:172.21.15.33:6835/973717150] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T02:52:58.712 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.d{-1:14538} state up:standby seq 1 addr [v2:172.21.15.33:6836/3063370040,v1:172.21.15.33:6837/3063370040] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T02:52:58.712 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.g{-1:14601} state up:standby seq 1 addr [v2:172.21.15.33:6838/1789650898,v1:172.21.15.33:6839/1789650898] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T02:52:58.713 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.a{-1:14613} state up:standby seq 1 addr [v2:172.21.15.33:6840/1349324757,v1:172.21.15.33:6841/1349324757] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T02:52:58.713 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.l{-1:24391} state up:standby seq 1 addr [v2:172.21.15.155:6832/268774752,v1:172.21.15.155:6833/268774752] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T02:52:58.713 INFO:teuthology.orchestra.run.smithi033.stdout:[mds.k{-1:24419} state up:standby seq 1 addr [v2:172.21.15.96:6832/2137965374,v1:172.21.15.96:6833/2137965374] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T02:52:58.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.709+0000 7f39999cb700 1 -- 172.21.15.33:0/2328233859 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f397c0a2890 msgr2=0x7f397c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:58.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.709+0000 7f39999cb700 1 --2- 172.21.15.33:0/2328233859 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f397c0a2890 0x7f397c0a4d50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3984011490 tx=0x7f3984009260 comp rx=0 tx=0).stop 2024-03-19T02:52:58.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.710+0000 7f39999cb700 1 -- 172.21.15.33:0/2328233859 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3994102340 msgr2=0x7f39941983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:52:58.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.710+0000 7f39999cb700 1 --2- 172.21.15.33:0/2328233859 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3994102340 0x7f39941983f0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f398800e980 tx=0x7f398800ec90 comp rx=0 tx=0).stop 2024-03-19T02:52:58.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.710+0000 7f39999cb700 1 -- 172.21.15.33:0/2328233859 shutdown_connections 2024-03-19T02:52:58.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.710+0000 7f39999cb700 1 --2- 172.21.15.33:0/2328233859 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f397c0a2890 0x7f397c0a4d50 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:58.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.710+0000 7f39999cb700 1 --2- 172.21.15.33:0/2328233859 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3994103ee0 0x7f399419f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:58.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.710+0000 7f39999cb700 1 --2- 172.21.15.33:0/2328233859 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3994102340 0x7f39941983f0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:58.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.710+0000 7f39999cb700 1 --2- 172.21.15.33:0/2328233859 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3994103540 0x7f3994198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:52:58.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.710+0000 7f39999cb700 1 -- 172.21.15.33:0/2328233859 >> 172.21.15.33:0/2328233859 conn(0x7f39940fd920 msgr2=0x7f3994106890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:52:58.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.710+0000 7f39999cb700 1 -- 172.21.15.33:0/2328233859 shutdown_connections 2024-03-19T02:52:58.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:52:58.710+0000 7f39999cb700 1 -- 172.21.15.33:0/2328233859 wait complete. 2024-03-19T02:52:58.714 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 24 2024-03-19T02:52:59.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:58 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2328233859' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-19T02:52:59.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:58 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2328233859' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-19T02:52:59.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:58 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2328233859' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-19T02:52:59.450 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-19T02:52:59.696 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:53:00.142 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:52:59 smithi033 ceph-mon[31973]: pgmap v367: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-19T02:53:00.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:52:59 smithi096 ceph-mon[35946]: pgmap v367: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-19T02:53:00.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:52:59 smithi155 ceph-mon[35804]: pgmap v367: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-19T02:53:01.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.265+0000 7f6e05df9700 1 -- 172.21.15.33:0/2279287506 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e00103940 msgr2=0x7f6e00103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:01.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.265+0000 7f6e05df9700 1 --2- 172.21.15.33:0/2279287506 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e00103940 0x7f6e00103d60 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f6df00099c0 tx=0x7f6df0009cd0 comp rx=0 tx=0).stop 2024-03-19T02:53:01.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.265+0000 7f6e05df9700 1 -- 172.21.15.33:0/2279287506 shutdown_connections 2024-03-19T02:53:01.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.265+0000 7f6e05df9700 1 --2- 172.21.15.33:0/2279287506 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e00100d80 0x7f6e0010a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:01.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.265+0000 7f6e05df9700 1 --2- 172.21.15.33:0/2279287506 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e00103940 0x7f6e00103d60 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:01.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.265+0000 7f6e05df9700 1 --2- 172.21.15.33:0/2279287506 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e001003c0 0x7f6e00100840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:01.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.265+0000 7f6e05df9700 1 -- 172.21.15.33:0/2279287506 >> 172.21.15.33:0/2279287506 conn(0x7f6e000fbf80 msgr2=0x7f6e000fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:01.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.265+0000 7f6e05df9700 1 -- 172.21.15.33:0/2279287506 shutdown_connections 2024-03-19T02:53:01.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.265+0000 7f6e05df9700 1 -- 172.21.15.33:0/2279287506 wait complete. 2024-03-19T02:53:01.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.266+0000 7f6e05df9700 1 Processor -- start 2024-03-19T02:53:01.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.266+0000 7f6e05df9700 1 -- start start 2024-03-19T02:53:01.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.266+0000 7f6e05df9700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e001003c0 0x7f6e00198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:01.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.266+0000 7f6e05df9700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e00100d80 0x7f6e001989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:01.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.266+0000 7f6e05df9700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e00103940 0x7f6e0019fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:01.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.266+0000 7f6e05df9700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6e0010dde0 con 0x7f6e00100d80 2024-03-19T02:53:01.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.266+0000 7f6e05df9700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6e0010dc60 con 0x7f6e001003c0 2024-03-19T02:53:01.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.266+0000 7f6e05df9700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6e0010df60 con 0x7f6e00103940 2024-03-19T02:53:01.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.266+0000 7f6dfeffd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e00100d80 0x7f6e001989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:01.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.266+0000 7f6dfeffd700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e00100d80 0x7f6e001989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42094/0 (socket says 172.21.15.33:42094) 2024-03-19T02:53:01.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.266+0000 7f6dfeffd700 1 -- 172.21.15.33:0/3844404853 learned_addr learned my addr 172.21.15.33:0/3844404853 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:01.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.266+0000 7f6dff7fe700 1 --2- 172.21.15.33:0/3844404853 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e001003c0 0x7f6e00198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:01.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.266+0000 7f6dfffff700 1 --2- 172.21.15.33:0/3844404853 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e00103940 0x7f6e0019fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:01.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.266+0000 7f6dfeffd700 1 -- 172.21.15.33:0/3844404853 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e00103940 msgr2=0x7f6e0019fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:01.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.266+0000 7f6dfeffd700 1 --2- 172.21.15.33:0/3844404853 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e00103940 0x7f6e0019fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:01.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.266+0000 7f6dfeffd700 1 -- 172.21.15.33:0/3844404853 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e001003c0 msgr2=0x7f6e00198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:01.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.266+0000 7f6dfeffd700 1 --2- 172.21.15.33:0/3844404853 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e001003c0 0x7f6e00198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:01.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.266+0000 7f6dfeffd700 1 -- 172.21.15.33:0/3844404853 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e001a00d0 con 0x7f6e00100d80 2024-03-19T02:53:01.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.266+0000 7f6dff7fe700 1 --2- 172.21.15.33:0/3844404853 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e001003c0 0x7f6e00198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:01.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.267+0000 7f6dfeffd700 1 --2- 172.21.15.33:0/3844404853 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e00100d80 0x7f6e001989b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f6df400d9b0 tx=0x7f6df400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:01.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.267+0000 7f6dfcff9700 1 -- 172.21.15.33:0/3844404853 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df4016070 con 0x7f6e00100d80 2024-03-19T02:53:01.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.267+0000 7f6dfcff9700 1 -- 172.21.15.33:0/3844404853 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6df400be00 con 0x7f6e00100d80 2024-03-19T02:53:01.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.267+0000 7f6e05df9700 1 -- 172.21.15.33:0/3844404853 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e001a03c0 con 0x7f6e00100d80 2024-03-19T02:53:01.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.267+0000 7f6dfcff9700 1 -- 172.21.15.33:0/3844404853 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df4010e30 con 0x7f6e00100d80 2024-03-19T02:53:01.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.267+0000 7f6e05df9700 1 -- 172.21.15.33:0/3844404853 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e001a0950 con 0x7f6e00100d80 2024-03-19T02:53:01.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.268+0000 7f6e05df9700 1 -- 172.21.15.33:0/3844404853 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e00068fd0 con 0x7f6e00100d80 2024-03-19T02:53:01.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.269+0000 7f6dfcff9700 1 -- 172.21.15.33:0/3844404853 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6df4019070 con 0x7f6e00100d80 2024-03-19T02:53:01.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.269+0000 7f6dfcff9700 1 --2- 172.21.15.33:0/3844404853 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f6de80a2910 0x7f6de80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:01.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.270+0000 7f6dff7fe700 1 --2- 172.21.15.33:0/3844404853 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f6de80a2910 0x7f6de80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:01.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.270+0000 7f6dfcff9700 1 -- 172.21.15.33:0/3844404853 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f6df40cf150 con 0x7f6e00100d80 2024-03-19T02:53:01.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.270+0000 7f6dff7fe700 1 --2- 172.21.15.33:0/3844404853 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f6de80a2910 0x7f6de80a4dd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6df00036f0 tx=0x7f6df0005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:01.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.273+0000 7f6dfcff9700 1 -- 172.21.15.33:0/3844404853 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6df4020030 con 0x7f6e00100d80 2024-03-19T02:53:01.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.480+0000 7f6e05df9700 1 -- 172.21.15.33:0/3844404853 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f6e00062f20 con 0x7f6e00100d80 2024-03-19T02:53:01.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.481+0000 7f6dfcff9700 1 -- 172.21.15.33:0/3844404853 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v97) ==== 56+0+5624 (secure 0 0 0) 0x7f6df409be60 con 0x7f6e00100d80 2024-03-19T02:53:01.482 INFO:teuthology.orchestra.run.smithi033.stdout:epoch 97 2024-03-19T02:53:01.483 INFO:teuthology.orchestra.run.smithi033.stdout:fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T02:53:01.483 INFO:teuthology.orchestra.run.smithi033.stdout:created 2024-03-19T02:42:37.912179+0000 2024-03-19T02:53:01.483 INFO:teuthology.orchestra.run.smithi033.stdout:modified 2024-03-19T02:52:41.116227+0000 2024-03-19T02:53:01.483 INFO:teuthology.orchestra.run.smithi033.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-19T02:53:01.483 INFO:teuthology.orchestra.run.smithi033.stdout:crush_version 27 2024-03-19T02:53:01.483 INFO:teuthology.orchestra.run.smithi033.stdout:full_ratio 0.95 2024-03-19T02:53:01.483 INFO:teuthology.orchestra.run.smithi033.stdout:backfillfull_ratio 0.9 2024-03-19T02:53:01.483 INFO:teuthology.orchestra.run.smithi033.stdout:nearfull_ratio 0.85 2024-03-19T02:53:01.483 INFO:teuthology.orchestra.run.smithi033.stdout:require_min_compat_client luminous 2024-03-19T02:53:01.483 INFO:teuthology.orchestra.run.smithi033.stdout:min_compat_client luminous 2024-03-19T02:53:01.483 INFO:teuthology.orchestra.run.smithi033.stdout:require_osd_release squid 2024-03-19T02:53:01.483 INFO:teuthology.orchestra.run.smithi033.stdout:stretch_mode_enabled false 2024-03-19T02:53:01.483 INFO:teuthology.orchestra.run.smithi033.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-19T02:53:01.483 INFO:teuthology.orchestra.run.smithi033.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-19T02:53:01.483 INFO:teuthology.orchestra.run.smithi033.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 87 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-03-19T02:53:01.484 INFO:teuthology.orchestra.run.smithi033.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 93 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-03-19T02:53:01.484 INFO:teuthology.orchestra.run.smithi033.stdout:max_osd 12 2024-03-19T02:53:01.484 INFO:teuthology.orchestra.run.smithi033.stdout:osd.0 up in weight 1 up_from 8 up_thru 94 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6802/1633124995,v1:172.21.15.33:6803/1633124995] [v2:172.21.15.33:6804/1633124995,v1:172.21.15.33:6805/1633124995] exists,up 090a89ab-6b7b-4317-8823-178d4212b2f4 2024-03-19T02:53:01.484 INFO:teuthology.orchestra.run.smithi033.stdout:osd.1 up in weight 1 up_from 13 up_thru 94 down_at 0 last_clean_interval [0,0) [v2:172.21.15.96:6800/340144398,v1:172.21.15.96:6801/340144398] [v2:172.21.15.96:6802/340144398,v1:172.21.15.96:6803/340144398] exists,up f50eec55-1956-4486-8593-f2c8fa610e88 2024-03-19T02:53:01.484 INFO:teuthology.orchestra.run.smithi033.stdout:osd.2 up in weight 1 up_from 18 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.155:6800/3388743876,v1:172.21.15.155:6801/3388743876] [v2:172.21.15.155:6802/3388743876,v1:172.21.15.155:6803/3388743876] exists,up 893a2d62-80d9-4e5e-bc6a-a1c8c681d652 2024-03-19T02:53:01.484 INFO:teuthology.orchestra.run.smithi033.stdout:osd.3 up in weight 1 up_from 26 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6810/677007171,v1:172.21.15.33:6811/677007171] [v2:172.21.15.33:6812/677007171,v1:172.21.15.33:6813/677007171] exists,up b07e0482-4281-4d32-9dfe-9995d7246175 2024-03-19T02:53:01.484 INFO:teuthology.orchestra.run.smithi033.stdout:osd.4 up in weight 1 up_from 34 up_thru 94 down_at 0 last_clean_interval [0,0) [v2:172.21.15.96:6808/3106158554,v1:172.21.15.96:6809/3106158554] [v2:172.21.15.96:6810/3106158554,v1:172.21.15.96:6811/3106158554] exists,up a988384b-af6b-47c6-8c07-5c9673acee45 2024-03-19T02:53:01.484 INFO:teuthology.orchestra.run.smithi033.stdout:osd.5 up in weight 1 up_from 39 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.155:6808/327293324,v1:172.21.15.155:6809/327293324] [v2:172.21.15.155:6810/327293324,v1:172.21.15.155:6811/327293324] exists,up 198dc992-96f8-4c52-96ee-1fd5de148bb8 2024-03-19T02:53:01.484 INFO:teuthology.orchestra.run.smithi033.stdout:osd.6 up in weight 1 up_from 47 up_thru 94 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6818/873871509,v1:172.21.15.33:6819/873871509] [v2:172.21.15.33:6820/873871509,v1:172.21.15.33:6821/873871509] exists,up a7278ec0-ab2e-49ed-be53-84d8b9e24177 2024-03-19T02:53:01.484 INFO:teuthology.orchestra.run.smithi033.stdout:osd.7 up in weight 1 up_from 53 up_thru 94 down_at 0 last_clean_interval [0,0) [v2:172.21.15.96:6816/2334123891,v1:172.21.15.96:6817/2334123891] [v2:172.21.15.96:6818/2334123891,v1:172.21.15.96:6819/2334123891] exists,up 727e133d-aa24-4ffc-90e2-266eb2aab8c5 2024-03-19T02:53:01.484 INFO:teuthology.orchestra.run.smithi033.stdout:osd.8 up in weight 1 up_from 61 up_thru 96 down_at 0 last_clean_interval [0,0) [v2:172.21.15.155:6816/3498516903,v1:172.21.15.155:6817/3498516903] [v2:172.21.15.155:6818/3498516903,v1:172.21.15.155:6819/3498516903] exists,up bb20eec0-39e5-4b86-9db5-486b1840cdad 2024-03-19T02:53:01.484 INFO:teuthology.orchestra.run.smithi033.stdout:osd.9 up in weight 1 up_from 69 up_thru 94 down_at 0 last_clean_interval [0,0) [v2:172.21.15.33:6826/760777213,v1:172.21.15.33:6827/760777213] [v2:172.21.15.33:6828/760777213,v1:172.21.15.33:6829/760777213] exists,up 179c6228-3e23-4f65-91a6-1ef0005d6904 2024-03-19T02:53:01.484 INFO:teuthology.orchestra.run.smithi033.stdout:osd.10 up in weight 1 up_from 74 up_thru 94 down_at 0 last_clean_interval [0,0) [v2:172.21.15.96:6824/1404898482,v1:172.21.15.96:6825/1404898482] [v2:172.21.15.96:6826/1404898482,v1:172.21.15.96:6827/1404898482] exists,up be17b478-3c2e-41fb-bede-958333acdf1d 2024-03-19T02:53:01.484 INFO:teuthology.orchestra.run.smithi033.stdout:osd.11 up in weight 1 up_from 82 up_thru 96 down_at 0 last_clean_interval [0,0) [v2:172.21.15.155:6824/792352827,v1:172.21.15.155:6825/792352827] [v2:172.21.15.155:6826/792352827,v1:172.21.15.155:6827/792352827] exists,up c0903ed2-cc1e-4c18-8731-1f0ee4dd0457 2024-03-19T02:53:01.484 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 2.8 [2,11] 2024-03-19T02:53:01.484 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 2.c [2,4] 2024-03-19T02:53:01.485 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 2.10 [8,11] 2024-03-19T02:53:01.485 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 3.17 [4,8] 2024-03-19T02:53:01.485 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 3.28 [5,6] 2024-03-19T02:53:01.485 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 3.29 [3,9] 2024-03-19T02:53:01.485 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 3.2c [4,9] 2024-03-19T02:53:01.485 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 3.36 [4,9] 2024-03-19T02:53:01.485 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 4.5 [9,10] 2024-03-19T02:53:01.485 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 4.9 [5,8] 2024-03-19T02:53:01.485 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 4.b [9,8] 2024-03-19T02:53:01.485 INFO:teuthology.orchestra.run.smithi033.stdout:pg_upmap_items 4.1f [9,8] 2024-03-19T02:53:01.485 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:6801/906645748 expires 2024-03-20T02:43:39.622218+0000 2024-03-19T02:53:01.485 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:6800/906645748 expires 2024-03-20T02:43:39.622218+0000 2024-03-19T02:53:01.485 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:0/1067768047 expires 2024-03-20T02:43:39.622218+0000 2024-03-19T02:53:01.485 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:0/729379453 expires 2024-03-20T02:43:11.918076+0000 2024-03-19T02:53:01.485 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:0/2586588742 expires 2024-03-20T02:43:39.622218+0000 2024-03-19T02:53:01.486 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:0/2932814165 expires 2024-03-20T02:43:11.918076+0000 2024-03-19T02:53:01.486 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:6801/3342667183 expires 2024-03-20T02:43:11.918076+0000 2024-03-19T02:53:01.486 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:0/2212564963 expires 2024-03-20T02:43:11.918076+0000 2024-03-19T02:53:01.486 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:6800/3342667183 expires 2024-03-20T02:43:11.918076+0000 2024-03-19T02:53:01.486 INFO:teuthology.orchestra.run.smithi033.stdout:blocklist 172.21.15.33:0/3275126058 expires 2024-03-20T02:43:39.622218+0000 2024-03-19T02:53:01.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.484+0000 7f6e05df9700 1 -- 172.21.15.33:0/3844404853 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f6de80a2910 msgr2=0x7f6de80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:01.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.484+0000 7f6e05df9700 1 --2- 172.21.15.33:0/3844404853 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f6de80a2910 0x7f6de80a4dd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6df00036f0 tx=0x7f6df0005fb0 comp rx=0 tx=0).stop 2024-03-19T02:53:01.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.484+0000 7f6e05df9700 1 -- 172.21.15.33:0/3844404853 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e00100d80 msgr2=0x7f6e001989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:01.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.484+0000 7f6e05df9700 1 --2- 172.21.15.33:0/3844404853 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e00100d80 0x7f6e001989b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f6df400d9b0 tx=0x7f6df400dcc0 comp rx=0 tx=0).stop 2024-03-19T02:53:01.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.484+0000 7f6e05df9700 1 -- 172.21.15.33:0/3844404853 shutdown_connections 2024-03-19T02:53:01.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.484+0000 7f6e05df9700 1 --2- 172.21.15.33:0/3844404853 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f6de80a2910 0x7f6de80a4dd0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:01.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.484+0000 7f6e05df9700 1 --2- 172.21.15.33:0/3844404853 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e00103940 0x7f6e0019fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:01.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.484+0000 7f6e05df9700 1 --2- 172.21.15.33:0/3844404853 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e001003c0 0x7f6e00198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:01.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.484+0000 7f6e05df9700 1 --2- 172.21.15.33:0/3844404853 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e00100d80 0x7f6e001989b0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:01.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.484+0000 7f6e05df9700 1 -- 172.21.15.33:0/3844404853 >> 172.21.15.33:0/3844404853 conn(0x7f6e000fbf80 msgr2=0x7f6e00106ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:01.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.484+0000 7f6e05df9700 1 -- 172.21.15.33:0/3844404853 shutdown_connections 2024-03-19T02:53:01.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:01.484+0000 7f6e05df9700 1 -- 172.21.15.33:0/3844404853 wait complete. 2024-03-19T02:53:01.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:01 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3844404853' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-19T02:53:01.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:01 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3844404853' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-19T02:53:02.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:01 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3844404853' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-19T02:53:02.144 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-19T02:53:02.157 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:53:02.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.267+0000 7fc148075640 1 -- 172.21.15.33:0/1927612221 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc140101930 msgr2=0x7fc140101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:02.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.267+0000 7fc148075640 1 --2- 172.21.15.33:0/1927612221 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc140101930 0x7fc140101db0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc130009900 tx=0x7fc13002bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:02.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.267+0000 7fc148075640 1 -- 172.21.15.33:0/1927612221 shutdown_connections 2024-03-19T02:53:02.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.267+0000 7fc148075640 1 --2- 172.21.15.33:0/1927612221 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc140101930 0x7fc140101db0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:02.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.267+0000 7fc148075640 1 --2- 172.21.15.33:0/1927612221 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc1401022f0 0x7fc140108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:02.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.267+0000 7fc148075640 1 --2- 172.21.15.33:0/1927612221 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc140100740 0x7fc140100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:02.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.267+0000 7fc148075640 1 -- 172.21.15.33:0/1927612221 >> 172.21.15.33:0/1927612221 conn(0x7fc1400fbcb0 msgr2=0x7fc1400fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:02.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.267+0000 7fc148075640 1 -- 172.21.15.33:0/1927612221 shutdown_connections 2024-03-19T02:53:02.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.267+0000 7fc148075640 1 -- 172.21.15.33:0/1927612221 wait complete. 2024-03-19T02:53:02.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.268+0000 7fc148075640 1 Processor -- start 2024-03-19T02:53:02.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.268+0000 7fc148075640 1 -- start start 2024-03-19T02:53:02.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.268+0000 7fc148075640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc140100740 0x7fc140197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:02.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.268+0000 7fc148075640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc140101930 0x7fc140197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:02.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.268+0000 7fc148075640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1401022f0 0x7fc14019f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:02.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.268+0000 7fc148075640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc14010d390 con 0x7fc140100740 2024-03-19T02:53:02.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.268+0000 7fc148075640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc14010d210 con 0x7fc140101930 2024-03-19T02:53:02.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.268+0000 7fc148075640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc14010d510 con 0x7fc1401022f0 2024-03-19T02:53:02.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.268+0000 7fc145e11640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc140100740 0x7fc140197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:02.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.268+0000 7fc145e11640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc140100740 0x7fc140197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42118/0 (socket says 172.21.15.33:42118) 2024-03-19T02:53:02.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.268+0000 7fc145e11640 1 -- 172.21.15.33:0/3875301641 learned_addr learned my addr 172.21.15.33:0/3875301641 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:02.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.268+0000 7fc145610640 1 --2- 172.21.15.33:0/3875301641 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc140101930 0x7fc140197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:02.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.268+0000 7fc146612640 1 --2- 172.21.15.33:0/3875301641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1401022f0 0x7fc14019f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:02.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.268+0000 7fc145e11640 1 -- 172.21.15.33:0/3875301641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1401022f0 msgr2=0x7fc14019f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:02.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.268+0000 7fc145e11640 1 --2- 172.21.15.33:0/3875301641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1401022f0 0x7fc14019f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:02.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.268+0000 7fc145e11640 1 -- 172.21.15.33:0/3875301641 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc140101930 msgr2=0x7fc140197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:02.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.268+0000 7fc145e11640 1 --2- 172.21.15.33:0/3875301641 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc140101930 0x7fc140197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:02.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.269+0000 7fc145e11640 1 -- 172.21.15.33:0/3875301641 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc14019f680 con 0x7fc140100740 2024-03-19T02:53:02.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.269+0000 7fc145610640 1 --2- 172.21.15.33:0/3875301641 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc140101930 0x7fc140197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:02.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.269+0000 7fc145e11640 1 --2- 172.21.15.33:0/3875301641 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc140100740 0x7fc140197a60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc13c00e9c0 tx=0x7fc13c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:02.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.269+0000 7fc136ffd640 1 -- 172.21.15.33:0/3875301641 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc13c00cd80 con 0x7fc140100740 2024-03-19T02:53:02.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.269+0000 7fc148075640 1 -- 172.21.15.33:0/3875301641 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc14019f970 con 0x7fc140100740 2024-03-19T02:53:02.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.269+0000 7fc136ffd640 1 -- 172.21.15.33:0/3875301641 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc13c0035f0 con 0x7fc140100740 2024-03-19T02:53:02.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.269+0000 7fc136ffd640 1 -- 172.21.15.33:0/3875301641 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc13c010690 con 0x7fc140100740 2024-03-19T02:53:02.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.269+0000 7fc148075640 1 -- 172.21.15.33:0/3875301641 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc14019fe40 con 0x7fc140100740 2024-03-19T02:53:02.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.270+0000 7fc148075640 1 -- 172.21.15.33:0/3875301641 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc140107b80 con 0x7fc140100740 2024-03-19T02:53:02.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.272+0000 7fc136ffd640 1 -- 172.21.15.33:0/3875301641 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc13c0037a0 con 0x7fc140100740 2024-03-19T02:53:02.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.272+0000 7fc136ffd640 1 --2- 172.21.15.33:0/3875301641 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc12c0a2800 0x7fc12c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:02.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.273+0000 7fc136ffd640 1 -- 172.21.15.33:0/3875301641 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7fc13c0cf4c0 con 0x7fc140100740 2024-03-19T02:53:02.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.273+0000 7fc145610640 1 --2- 172.21.15.33:0/3875301641 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc12c0a2800 0x7fc12c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:02.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.273+0000 7fc145610640 1 --2- 172.21.15.33:0/3875301641 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc12c0a2800 0x7fc12c0a4cc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc130009230 tx=0x7fc130005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:02.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.275+0000 7fc136ffd640 1 -- 172.21.15.33:0/3875301641 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc13c09c170 con 0x7fc140100740 2024-03-19T02:53:02.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.464+0000 7fc148075640 1 -- 172.21.15.33:0/3875301641 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc14002d220 con 0x7fc140100740 2024-03-19T02:53:02.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.466+0000 7fc136ffd640 1 -- 172.21.15.33:0/3875301641 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) ==== 74+0+24663 (secure 0 0 0) 0x7fc13c09ba70 con 0x7fc140100740 2024-03-19T02:53:02.467 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:53:02.467 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":97,"fsid":"2b7a8938-e59a-11ee-95c9-87774f69a715","created":"2024-03-19T02:42:37.912179+0000","modified":"2024-03-19T02:52:41.116227+0000","last_up_change":"2024-03-19T02:50:01.539392+0000","last_in_change":"2024-03-19T02:49:40.534415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:46:09.963737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:50:50.069641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:50:51.381064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:50:55.476667+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"090a89ab-6b7b-4317-8823-178d4212b2f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6803","nonce":1633124995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6805","nonce":1633124995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6809","nonce":1633124995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6807","nonce":1633124995}]},"public_addr":"172.21.15.33:6803/1633124995","cluster_addr":"172.21.15.33:6805/1633124995","heartbeat_back_addr":"172.21.15.33:6809/1633124995","heartbeat_front_addr":"172.21.15.33:6807/1633124995","state":["exists","up"]},{"osd":1,"uuid":"f50eec55-1956-4486-8593-f2c8fa610e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6801","nonce":340144398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6803","nonce":340144398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6807","nonce":340144398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6805","nonce":340144398}]},"public_addr":"172.21.15.96:6801/340144398","cluster_addr":"172.21.15.96:6803/340144398","heartbeat_back_addr":"172.21.15.96:6807/340144398","heartbeat_front_addr":"172.21.15.96:6805/340144398","state":["exists","up"]},{"osd":2,"uuid":"893a2d62-80d9-4e5e-bc6a-a1c8c681d652","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6801","nonce":3388743876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6803","nonce":3388743876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6807","nonce":3388743876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6805","nonce":3388743876}]},"public_addr":"172.21.15.155:6801/3388743876","cluster_addr":"172.21.15.155:6803/3388743876","heartbeat_back_addr":"172.21.15.155:6807/3388743876","heartbeat_front_addr":"172.21.15.155:6805/3388743876","state":["exists","up"]},{"osd":3,"uuid":"b07e0482-4281-4d32-9dfe-9995d7246175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6811","nonce":677007171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6813","nonce":677007171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6817","nonce":677007171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6815","nonce":677007171}]},"public_addr":"172.21.15.33:6811/677007171","cluster_addr":"172.21.15.33:6813/677007171","heartbeat_back_addr":"172.21.15.33:6817/677007171","heartbeat_front_addr":"172.21.15.33:6815/677007171","state":["exists","up"]},{"osd":4,"uuid":"a988384b-af6b-47c6-8c07-5c9673acee45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6809","nonce":3106158554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6811","nonce":3106158554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6815","nonce":3106158554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6813","nonce":3106158554}]},"public_addr":"172.21.15.96:6809/3106158554","cluster_addr":"172.21.15.96:6811/3106158554","heartbeat_back_addr":"172.21.15.96:6815/3106158554","heartbeat_front_addr":"172.21.15.96:6813/3106158554","state":["exists","up"]},{"osd":5,"uuid":"198dc992-96f8-4c52-96ee-1fd5de148bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6809","nonce":327293324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6811","nonce":327293324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6815","nonce":327293324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6813","nonce":327293324}]},"public_addr":"172.21.15.155:6809/327293324","cluster_addr":"172.21.15.155:6811/327293324","heartbeat_back_addr":"172.21.15.155:6815/327293324","heartbeat_front_addr":"172.21.15.155:6813/327293324","state":["exists","up"]},{"osd":6,"uuid":"a7278ec0-ab2e-49ed-be53-84d8b9e24177","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6819","nonce":873871509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6821","nonce":873871509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6825","nonce":873871509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6823","nonce":873871509}]},"public_addr":"172.21.15.33:6819/873871509","cluster_addr":"172.21.15.33:6821/873871509","heartbeat_back_addr":"172.21.15.33:6825/873871509","heartbeat_front_addr":"172.21.15.33:6823/873871509","state":["exists","up"]},{"osd":7,"uuid":"727e133d-aa24-4ffc-90e2-266eb2aab8c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6817","nonce":2334123891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6819","nonce":2334123891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6823","nonce":2334123891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6821","nonce":2334123891}]},"public_addr":"172.21.15.96:6817/2334123891","cluster_addr":"172.21.15.96:6819/2334123891","heartbeat_back_addr":"172.21.15.96:6823/2334123891","heartbeat_front_addr":"172.21.15.96:6821/2334123891","state":["exists","up"]},{"osd":8,"uuid":"bb20eec0-39e5-4b86-9db5-486b1840cdad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6817","nonce":3498516903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6819","nonce":3498516903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6823","nonce":3498516903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6821","nonce":3498516903}]},"public_addr":"172.21.15.155:6817/3498516903","cluster_addr":"172.21.15.155:6819/3498516903","heartbeat_back_addr":"172.21.15.155:6823/3498516903","heartbeat_front_addr":"172.21.15.155:6821/3498516903","state":["exists","up"]},{"osd":9,"uuid":"179c6228-3e23-4f65-91a6-1ef0005d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6827","nonce":760777213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6829","nonce":760777213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6833","nonce":760777213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6831","nonce":760777213}]},"public_addr":"172.21.15.33:6827/760777213","cluster_addr":"172.21.15.33:6829/760777213","heartbeat_back_addr":"172.21.15.33:6833/760777213","heartbeat_front_addr":"172.21.15.33:6831/760777213","state":["exists","up"]},{"osd":10,"uuid":"be17b478-3c2e-41fb-bede-958333acdf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6825","nonce":1404898482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6827","nonce":1404898482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6831","nonce":1404898482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6829","nonce":1404898482}]},"public_addr":"172.21.15.96:6825/1404898482","cluster_addr":"172.21.15.96:6827/1404898482","heartbeat_back_addr":"172.21.15.96:6831/1404898482","heartbeat_front_addr":"172.21.15.96:6829/1404898482","state":["exists","up"]},{"osd":11,"uuid":"c0903ed2-cc1e-4c18-8731-1f0ee4dd0457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6825","nonce":792352827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6827","nonce":792352827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6831","nonce":792352827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6829","nonce":792352827}]},"public_addr":"172.21.15.155:6825/792352827","cluster_addr":"172.21.15.155:6827/792352827","heartbeat_back_addr":"172.21.15.155:6831/792352827","heartbeat_front_addr":"172.21.15.155:6829/792352827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:17.942801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:42.109449+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:05.981599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:32.069574+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:57.431015+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:22.448592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:48.526515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:14.498352+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:40.175426+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:06.364006+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:32.853161+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:59.073538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.9","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:6801/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:6800/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/1067768047":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/729379453":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2586588742":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/2932814165":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6801/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2212564963":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6800/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/3275126058":"2024-03-20T02:43:39.622218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:53:02.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.468+0000 7fc148075640 1 -- 172.21.15.33:0/3875301641 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc12c0a2800 msgr2=0x7fc12c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:02.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.468+0000 7fc148075640 1 --2- 172.21.15.33:0/3875301641 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc12c0a2800 0x7fc12c0a4cc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc130009230 tx=0x7fc130005fd0 comp rx=0 tx=0).stop 2024-03-19T02:53:02.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.468+0000 7fc148075640 1 -- 172.21.15.33:0/3875301641 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc140100740 msgr2=0x7fc140197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:02.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.468+0000 7fc148075640 1 --2- 172.21.15.33:0/3875301641 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc140100740 0x7fc140197a60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc13c00e9c0 tx=0x7fc13c00ee80 comp rx=0 tx=0).stop 2024-03-19T02:53:02.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.468+0000 7fc148075640 1 -- 172.21.15.33:0/3875301641 shutdown_connections 2024-03-19T02:53:02.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.468+0000 7fc148075640 1 --2- 172.21.15.33:0/3875301641 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc12c0a2800 0x7fc12c0a4cc0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:02.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.468+0000 7fc148075640 1 --2- 172.21.15.33:0/3875301641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1401022f0 0x7fc14019f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:02.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.468+0000 7fc148075640 1 --2- 172.21.15.33:0/3875301641 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc140101930 0x7fc140197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:02.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.468+0000 7fc148075640 1 --2- 172.21.15.33:0/3875301641 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc140100740 0x7fc140197a60 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:02.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.468+0000 7fc148075640 1 -- 172.21.15.33:0/3875301641 >> 172.21.15.33:0/3875301641 conn(0x7fc1400fbcb0 msgr2=0x7fc140104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:02.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.468+0000 7fc148075640 1 -- 172.21.15.33:0/3875301641 shutdown_connections 2024-03-19T02:53:02.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.468+0000 7fc148075640 1 -- 172.21.15.33:0/3875301641 wait complete. 2024-03-19T02:53:02.481 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:46:09.963737+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T02:50:50.069641+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T02:50:51.381064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T02:50:55.476667+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T02:53:02.481 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T02:53:02.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.590+0000 7fed3de63640 1 -- 172.21.15.33:0/2318232810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed38101850 msgr2=0x7fed38101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:02.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.590+0000 7fed3de63640 1 --2- 172.21.15.33:0/2318232810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed38101850 0x7fed38101c70 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fed28009900 tx=0x7fed2802bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:02.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.591+0000 7fed3de63640 1 -- 172.21.15.33:0/2318232810 shutdown_connections 2024-03-19T02:53:02.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.591+0000 7fed3de63640 1 --2- 172.21.15.33:0/2318232810 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed38102a40 0x7fed38102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:02.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.591+0000 7fed3de63640 1 --2- 172.21.15.33:0/2318232810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fed38103400 0x7fed38109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:02.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.591+0000 7fed3de63640 1 --2- 172.21.15.33:0/2318232810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed38101850 0x7fed38101c70 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:02.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.591+0000 7fed3de63640 1 -- 172.21.15.33:0/2318232810 >> 172.21.15.33:0/2318232810 conn(0x7fed380fcdc0 msgr2=0x7fed380ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:02.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.591+0000 7fed3de63640 1 -- 172.21.15.33:0/2318232810 shutdown_connections 2024-03-19T02:53:02.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.591+0000 7fed3de63640 1 -- 172.21.15.33:0/2318232810 wait complete. 2024-03-19T02:53:02.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.591+0000 7fed3de63640 1 Processor -- start 2024-03-19T02:53:02.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.591+0000 7fed3de63640 1 -- start start 2024-03-19T02:53:02.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.592+0000 7fed3de63640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed38101850 0x7fed380755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:02.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.592+0000 7fed3de63640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed38102a40 0x7fed38073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:02.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.592+0000 7fed3de63640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fed38103400 0x7fed38074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:02.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.592+0000 7fed3de63640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fed3810d370 con 0x7fed38101850 2024-03-19T02:53:02.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.592+0000 7fed3de63640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fed3810d1f0 con 0x7fed38103400 2024-03-19T02:53:02.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.592+0000 7fed3de63640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fed3810d4f0 con 0x7fed38102a40 2024-03-19T02:53:02.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.592+0000 7fed377fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed38101850 0x7fed380755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:02.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.592+0000 7fed377fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed38101850 0x7fed380755d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42136/0 (socket says 172.21.15.33:42136) 2024-03-19T02:53:02.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.592+0000 7fed377fe640 1 -- 172.21.15.33:0/566180775 learned_addr learned my addr 172.21.15.33:0/566180775 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:02.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.592+0000 7fed37fff640 1 --2- 172.21.15.33:0/566180775 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fed38103400 0x7fed38074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:02.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.592+0000 7fed36ffd640 1 --2- 172.21.15.33:0/566180775 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed38102a40 0x7fed38073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:02.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.592+0000 7fed377fe640 1 -- 172.21.15.33:0/566180775 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed38102a40 msgr2=0x7fed38073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:02.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.592+0000 7fed377fe640 1 --2- 172.21.15.33:0/566180775 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed38102a40 0x7fed38073c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:02.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.592+0000 7fed377fe640 1 -- 172.21.15.33:0/566180775 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fed38103400 msgr2=0x7fed38074150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:02.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.592+0000 7fed377fe640 1 --2- 172.21.15.33:0/566180775 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fed38103400 0x7fed38074150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:02.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.592+0000 7fed377fe640 1 -- 172.21.15.33:0/566180775 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed380749e0 con 0x7fed38101850 2024-03-19T02:53:02.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.592+0000 7fed377fe640 1 --2- 172.21.15.33:0/566180775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed38101850 0x7fed380755d0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fed28000c00 tx=0x7fed28005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:02.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.592+0000 7fed34ff9640 1 -- 172.21.15.33:0/566180775 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed28043020 con 0x7fed38101850 2024-03-19T02:53:02.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.592+0000 7fed34ff9640 1 -- 172.21.15.33:0/566180775 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fed28004350 con 0x7fed38101850 2024-03-19T02:53:02.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.592+0000 7fed3de63640 1 -- 172.21.15.33:0/566180775 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed381a1c10 con 0x7fed38101850 2024-03-19T02:53:02.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.592+0000 7fed34ff9640 1 -- 172.21.15.33:0/566180775 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed28003b80 con 0x7fed38101850 2024-03-19T02:53:02.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.592+0000 7fed3de63640 1 -- 172.21.15.33:0/566180775 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed381a2160 con 0x7fed38101850 2024-03-19T02:53:02.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.594+0000 7fed3de63640 1 -- 172.21.15.33:0/566180775 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed38108c90 con 0x7fed38101850 2024-03-19T02:53:02.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.595+0000 7fed34ff9640 1 -- 172.21.15.33:0/566180775 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fed28046020 con 0x7fed38101850 2024-03-19T02:53:02.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.595+0000 7fed34ff9640 1 --2- 172.21.15.33:0/566180775 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fed200a2800 0x7fed200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:02.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.595+0000 7fed34ff9640 1 -- 172.21.15.33:0/566180775 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7fed280f0f10 con 0x7fed38101850 2024-03-19T02:53:02.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.595+0000 7fed36ffd640 1 --2- 172.21.15.33:0/566180775 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fed200a2800 0x7fed200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:02.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.596+0000 7fed36ffd640 1 --2- 172.21.15.33:0/566180775 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fed200a2800 0x7fed200a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fed38075340 tx=0x7fed2c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:02.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.600+0000 7fed34ff9640 1 -- 172.21.15.33:0/566180775 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fed2804d020 con 0x7fed38101850 2024-03-19T02:53:02.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:02 smithi033 ceph-mon[31973]: pgmap v368: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-19T02:53:02.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:02 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3875301641' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:53:02.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.784+0000 7fed3de63640 1 -- 172.21.15.33:0/566180775 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fed38062420 con 0x7fed38101850 2024-03-19T02:53:02.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.784+0000 7fed34ff9640 1 -- 172.21.15.33:0/566180775 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v97) ==== 93+0+10 (secure 0 0 0) 0x7fed280bdc20 con 0x7fed38101850 2024-03-19T02:53:02.785 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 1 2024-03-19T02:53:02.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.786+0000 7fed3de63640 1 -- 172.21.15.33:0/566180775 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fed200a2800 msgr2=0x7fed200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:02.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.786+0000 7fed3de63640 1 --2- 172.21.15.33:0/566180775 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fed200a2800 0x7fed200a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fed38075340 tx=0x7fed2c005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:02.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.786+0000 7fed3de63640 1 -- 172.21.15.33:0/566180775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed38101850 msgr2=0x7fed380755d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:02.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.786+0000 7fed3de63640 1 --2- 172.21.15.33:0/566180775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed38101850 0x7fed380755d0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fed28000c00 tx=0x7fed28005dc0 comp rx=0 tx=0).stop 2024-03-19T02:53:02.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.786+0000 7fed3de63640 1 -- 172.21.15.33:0/566180775 shutdown_connections 2024-03-19T02:53:02.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.786+0000 7fed3de63640 1 --2- 172.21.15.33:0/566180775 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fed200a2800 0x7fed200a4cc0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:02.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.786+0000 7fed3de63640 1 --2- 172.21.15.33:0/566180775 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed38102a40 0x7fed38073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:02.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.786+0000 7fed3de63640 1 --2- 172.21.15.33:0/566180775 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fed38103400 0x7fed38074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:02.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.786+0000 7fed3de63640 1 --2- 172.21.15.33:0/566180775 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fed38101850 0x7fed380755d0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:02.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.786+0000 7fed3de63640 1 -- 172.21.15.33:0/566180775 >> 172.21.15.33:0/566180775 conn(0x7fed380fcdc0 msgr2=0x7fed38105e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:02.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.786+0000 7fed3de63640 1 -- 172.21.15.33:0/566180775 shutdown_connections 2024-03-19T02:53:02.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.786+0000 7fed3de63640 1 -- 172.21.15.33:0/566180775 wait complete. 2024-03-19T02:53:02.799 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T02:53:02.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.905+0000 7fe8e4244640 1 -- 172.21.15.33:0/2379616557 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe8dc102a40 msgr2=0x7fe8dc102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:02.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.905+0000 7fe8e4244640 1 --2- 172.21.15.33:0/2379616557 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe8dc102a40 0x7fe8dc102ec0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fe8d0009900 tx=0x7fe8d002bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:02.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.905+0000 7fe8e4244640 1 -- 172.21.15.33:0/2379616557 shutdown_connections 2024-03-19T02:53:02.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.905+0000 7fe8e4244640 1 --2- 172.21.15.33:0/2379616557 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe8dc103400 0x7fe8dc109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:02.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.905+0000 7fe8e4244640 1 --2- 172.21.15.33:0/2379616557 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe8dc102a40 0x7fe8dc102ec0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:02.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.905+0000 7fe8e4244640 1 --2- 172.21.15.33:0/2379616557 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe8dc101850 0x7fe8dc101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:02.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.905+0000 7fe8e4244640 1 -- 172.21.15.33:0/2379616557 >> 172.21.15.33:0/2379616557 conn(0x7fe8dc0fcdc0 msgr2=0x7fe8dc0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:02.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.905+0000 7fe8e4244640 1 -- 172.21.15.33:0/2379616557 shutdown_connections 2024-03-19T02:53:02.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.905+0000 7fe8e4244640 1 -- 172.21.15.33:0/2379616557 wait complete. 2024-03-19T02:53:02.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.906+0000 7fe8e4244640 1 Processor -- start 2024-03-19T02:53:02.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.906+0000 7fe8e4244640 1 -- start start 2024-03-19T02:53:02.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.906+0000 7fe8e4244640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe8dc101850 0x7fe8dc197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:02.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.906+0000 7fe8e4244640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe8dc102a40 0x7fe8dc197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:02.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.906+0000 7fe8e4244640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe8dc103400 0x7fe8dc19f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:02.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.906+0000 7fe8e4244640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe8dc10d360 con 0x7fe8dc102a40 2024-03-19T02:53:02.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.906+0000 7fe8e4244640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe8dc10d1e0 con 0x7fe8dc101850 2024-03-19T02:53:02.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.906+0000 7fe8e4244640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe8dc10d4e0 con 0x7fe8dc103400 2024-03-19T02:53:02.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.906+0000 7fe8e17df640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe8dc102a40 0x7fe8dc197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:02.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.906+0000 7fe8e17df640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe8dc102a40 0x7fe8dc197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42162/0 (socket says 172.21.15.33:42162) 2024-03-19T02:53:02.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.906+0000 7fe8e17df640 1 -- 172.21.15.33:0/3691500167 learned_addr learned my addr 172.21.15.33:0/3691500167 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:02.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.906+0000 7fe8e1fe0640 1 --2- 172.21.15.33:0/3691500167 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe8dc101850 0x7fe8dc197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:02.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.906+0000 7fe8e27e1640 1 --2- 172.21.15.33:0/3691500167 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe8dc103400 0x7fe8dc19f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:02.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.907+0000 7fe8e1fe0640 1 -- 172.21.15.33:0/3691500167 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe8dc103400 msgr2=0x7fe8dc19f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:02.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.907+0000 7fe8e1fe0640 1 --2- 172.21.15.33:0/3691500167 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe8dc103400 0x7fe8dc19f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:02.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.907+0000 7fe8e1fe0640 1 -- 172.21.15.33:0/3691500167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe8dc102a40 msgr2=0x7fe8dc197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:02.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.907+0000 7fe8e1fe0640 1 --2- 172.21.15.33:0/3691500167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe8dc102a40 0x7fe8dc197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:02.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.907+0000 7fe8e1fe0640 1 -- 172.21.15.33:0/3691500167 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8dc19f6a0 con 0x7fe8dc101850 2024-03-19T02:53:02.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.907+0000 7fe8e17df640 1 --2- 172.21.15.33:0/3691500167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe8dc102a40 0x7fe8dc197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:02.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.907+0000 7fe8e1fe0640 1 --2- 172.21.15.33:0/3691500167 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe8dc101850 0x7fe8dc197a80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe8d800e9c0 tx=0x7fe8d800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:02.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.907+0000 7fe8ceffd640 1 -- 172.21.15.33:0/3691500167 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d800ccc0 con 0x7fe8dc101850 2024-03-19T02:53:02.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.907+0000 7fe8ceffd640 1 -- 172.21.15.33:0/3691500167 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe8d80035f0 con 0x7fe8dc101850 2024-03-19T02:53:02.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.907+0000 7fe8e4244640 1 -- 172.21.15.33:0/3691500167 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8dc19f990 con 0x7fe8dc101850 2024-03-19T02:53:02.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.907+0000 7fe8ceffd640 1 -- 172.21.15.33:0/3691500167 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d800ce60 con 0x7fe8dc101850 2024-03-19T02:53:02.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.907+0000 7fe8e4244640 1 -- 172.21.15.33:0/3691500167 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8dc19fee0 con 0x7fe8dc101850 2024-03-19T02:53:02.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.908+0000 7fe8e4244640 1 -- 172.21.15.33:0/3691500167 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8dc108c40 con 0x7fe8dc101850 2024-03-19T02:53:02.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.909+0000 7fe8ceffd640 1 -- 172.21.15.33:0/3691500167 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe8d80037a0 con 0x7fe8dc101850 2024-03-19T02:53:02.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.909+0000 7fe8ceffd640 1 --2- 172.21.15.33:0/3691500167 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe8c80a2800 0x7fe8c80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:02.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.910+0000 7fe8e17df640 1 --2- 172.21.15.33:0/3691500167 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe8c80a2800 0x7fe8c80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:02.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.910+0000 7fe8ceffd640 1 -- 172.21.15.33:0/3691500167 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7fe8d80cf230 con 0x7fe8dc101850 2024-03-19T02:53:02.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.910+0000 7fe8e17df640 1 --2- 172.21.15.33:0/3691500167 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe8c80a2800 0x7fe8c80a4cc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc198f90 tx=0x7fe8d002c7e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:02.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:02.914+0000 7fe8ceffd640 1 -- 172.21.15.33:0/3691500167 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe8d809bf40 con 0x7fe8dc101850 2024-03-19T02:53:02.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:02 smithi096 ceph-mon[35946]: pgmap v368: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-19T02:53:02.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:02 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3875301641' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:53:03.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:02 smithi155 ceph-mon[35804]: pgmap v368: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-19T02:53:03.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:02 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3875301641' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:53:03.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.096+0000 7fe8e4244640 1 -- 172.21.15.33:0/3691500167 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe8dc062420 con 0x7fe8dc101850 2024-03-19T02:53:03.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.097+0000 7fe8ceffd640 1 -- 172.21.15.33:0/3691500167 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v97) ==== 104+0+11 (secure 0 0 0) 0x7fe8d809b840 con 0x7fe8dc101850 2024-03-19T02:53:03.098 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 32 2024-03-19T02:53:03.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.099+0000 7fe8e4244640 1 -- 172.21.15.33:0/3691500167 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe8c80a2800 msgr2=0x7fe8c80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:03.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.099+0000 7fe8e4244640 1 --2- 172.21.15.33:0/3691500167 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe8c80a2800 0x7fe8c80a4cc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc198f90 tx=0x7fe8d002c7e0 comp rx=0 tx=0).stop 2024-03-19T02:53:03.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.099+0000 7fe8e4244640 1 -- 172.21.15.33:0/3691500167 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe8dc101850 msgr2=0x7fe8dc197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:03.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.099+0000 7fe8e4244640 1 --2- 172.21.15.33:0/3691500167 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe8dc101850 0x7fe8dc197a80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe8d800e9c0 tx=0x7fe8d800ee80 comp rx=0 tx=0).stop 2024-03-19T02:53:03.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.099+0000 7fe8e4244640 1 -- 172.21.15.33:0/3691500167 shutdown_connections 2024-03-19T02:53:03.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.099+0000 7fe8e4244640 1 --2- 172.21.15.33:0/3691500167 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe8c80a2800 0x7fe8c80a4cc0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:03.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.099+0000 7fe8e4244640 1 --2- 172.21.15.33:0/3691500167 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe8dc103400 0x7fe8dc19f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:03.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.099+0000 7fe8e4244640 1 --2- 172.21.15.33:0/3691500167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe8dc102a40 0x7fe8dc197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:03.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.099+0000 7fe8e4244640 1 --2- 172.21.15.33:0/3691500167 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe8dc101850 0x7fe8dc197a80 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:03.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.099+0000 7fe8e4244640 1 -- 172.21.15.33:0/3691500167 >> 172.21.15.33:0/3691500167 conn(0x7fe8dc0fcdc0 msgr2=0x7fe8dc105e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:03.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.099+0000 7fe8e4244640 1 -- 172.21.15.33:0/3691500167 shutdown_connections 2024-03-19T02:53:03.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.099+0000 7fe8e4244640 1 -- 172.21.15.33:0/3691500167 wait complete. 2024-03-19T02:53:03.111 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T02:53:03.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.217+0000 7f4971d77640 1 -- 172.21.15.33:0/1076652386 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f496c1022d0 msgr2=0x7f496c108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:03.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.217+0000 7f4971d77640 1 --2- 172.21.15.33:0/1076652386 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f496c1022d0 0x7f496c108b60 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4964009ca0 tx=0x7f496402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:53:03.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.217+0000 7f4971d77640 1 -- 172.21.15.33:0/1076652386 shutdown_connections 2024-03-19T02:53:03.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.217+0000 7f4971d77640 1 --2- 172.21.15.33:0/1076652386 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f496c101910 0x7f496c101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:03.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.217+0000 7f4971d77640 1 --2- 172.21.15.33:0/1076652386 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f496c1022d0 0x7f496c108b60 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:03.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.217+0000 7f4971d77640 1 --2- 172.21.15.33:0/1076652386 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f496c100720 0x7f496c100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:03.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.217+0000 7f4971d77640 1 -- 172.21.15.33:0/1076652386 >> 172.21.15.33:0/1076652386 conn(0x7f496c0fbcb0 msgr2=0x7f496c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:03.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.217+0000 7f4971d77640 1 -- 172.21.15.33:0/1076652386 shutdown_connections 2024-03-19T02:53:03.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.217+0000 7f4971d77640 1 -- 172.21.15.33:0/1076652386 wait complete. 2024-03-19T02:53:03.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.218+0000 7f4971d77640 1 Processor -- start 2024-03-19T02:53:03.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.218+0000 7f4971d77640 1 -- start start 2024-03-19T02:53:03.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.218+0000 7f4971d77640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f496c100720 0x7f496c0694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:03.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.218+0000 7f4971d77640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f496c101910 0x7f496c19c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:03.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.218+0000 7f4971d77640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f496c1022d0 0x7f496c19ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:03.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.218+0000 7f4971d77640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f496c10c230 con 0x7f496c1022d0 2024-03-19T02:53:03.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.218+0000 7f4971d77640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f496c10c0b0 con 0x7f496c101910 2024-03-19T02:53:03.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.218+0000 7f4971d77640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f496c10c3b0 con 0x7f496c100720 2024-03-19T02:53:03.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.218+0000 7f496bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f496c1022d0 0x7f496c19ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:03.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.218+0000 7f496bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f496c1022d0 0x7f496c19ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42184/0 (socket says 172.21.15.33:42184) 2024-03-19T02:53:03.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.218+0000 7f496bfff640 1 -- 172.21.15.33:0/313182651 learned_addr learned my addr 172.21.15.33:0/313182651 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:03.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.218+0000 7f496affd640 1 --2- 172.21.15.33:0/313182651 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f496c101910 0x7f496c19c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:03.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.218+0000 7f496b7fe640 1 --2- 172.21.15.33:0/313182651 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f496c100720 0x7f496c0694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:03.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.218+0000 7f496bfff640 1 -- 172.21.15.33:0/313182651 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f496c100720 msgr2=0x7f496c0694f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:03.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.218+0000 7f496bfff640 1 --2- 172.21.15.33:0/313182651 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f496c100720 0x7f496c0694f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:03.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.218+0000 7f496bfff640 1 -- 172.21.15.33:0/313182651 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f496c101910 msgr2=0x7f496c19c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:03.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.218+0000 7f496bfff640 1 --2- 172.21.15.33:0/313182651 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f496c101910 0x7f496c19c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:03.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.218+0000 7f496bfff640 1 -- 172.21.15.33:0/313182651 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f496c19f690 con 0x7f496c1022d0 2024-03-19T02:53:03.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.218+0000 7f496affd640 1 --2- 172.21.15.33:0/313182651 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f496c101910 0x7f496c19c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:03.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.219+0000 7f496bfff640 1 --2- 172.21.15.33:0/313182651 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f496c1022d0 0x7f496c19ed80 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f496402c700 tx=0x7f49640328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:03.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.219+0000 7f4968ff9640 1 -- 172.21.15.33:0/313182651 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4964041020 con 0x7f496c1022d0 2024-03-19T02:53:03.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.219+0000 7f4971d77640 1 -- 172.21.15.33:0/313182651 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f496c19f8c0 con 0x7f496c1022d0 2024-03-19T02:53:03.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.219+0000 7f4968ff9640 1 -- 172.21.15.33:0/313182651 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4964003e20 con 0x7f496c1022d0 2024-03-19T02:53:03.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.219+0000 7f4968ff9640 1 -- 172.21.15.33:0/313182651 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4964039040 con 0x7f496c1022d0 2024-03-19T02:53:03.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.219+0000 7f4971d77640 1 -- 172.21.15.33:0/313182651 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f496c19fe00 con 0x7f496c1022d0 2024-03-19T02:53:03.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.220+0000 7f4971d77640 1 -- 172.21.15.33:0/313182651 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f496c107a60 con 0x7f496c1022d0 2024-03-19T02:53:03.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.221+0000 7f4968ff9640 1 -- 172.21.15.33:0/313182651 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4964044020 con 0x7f496c1022d0 2024-03-19T02:53:03.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.221+0000 7f4968ff9640 1 --2- 172.21.15.33:0/313182651 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f49580a2800 0x7f49580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:03.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.221+0000 7f4968ff9640 1 -- 172.21.15.33:0/313182651 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f49640f0f80 con 0x7f496c1022d0 2024-03-19T02:53:03.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.222+0000 7f496b7fe640 1 --2- 172.21.15.33:0/313182651 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f49580a2800 0x7f49580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:03.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.222+0000 7f496b7fe640 1 --2- 172.21.15.33:0/313182651 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f49580a2800 0x7f49580a4cc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f496c101770 tx=0x7f4954005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:03.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.226+0000 7f4968ff9640 1 -- 172.21.15.33:0/313182651 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f496404b020 con 0x7f496c1022d0 2024-03-19T02:53:03.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.414+0000 7f4971d77640 1 -- 172.21.15.33:0/313182651 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f496c062420 con 0x7f496c1022d0 2024-03-19T02:53:03.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.415+0000 7f4968ff9640 1 -- 172.21.15.33:0/313182651 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v97) ==== 100+0+11 (secure 0 0 0) 0x7f49640bdc90 con 0x7f496c1022d0 2024-03-19T02:53:03.416 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 64 2024-03-19T02:53:03.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.417+0000 7f4971d77640 1 -- 172.21.15.33:0/313182651 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f49580a2800 msgr2=0x7f49580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:03.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.417+0000 7f4971d77640 1 --2- 172.21.15.33:0/313182651 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f49580a2800 0x7f49580a4cc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f496c101770 tx=0x7f4954005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:03.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.417+0000 7f4971d77640 1 -- 172.21.15.33:0/313182651 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f496c1022d0 msgr2=0x7f496c19ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:03.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.417+0000 7f4971d77640 1 --2- 172.21.15.33:0/313182651 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f496c1022d0 0x7f496c19ed80 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f496402c700 tx=0x7f49640328a0 comp rx=0 tx=0).stop 2024-03-19T02:53:03.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.417+0000 7f4971d77640 1 -- 172.21.15.33:0/313182651 shutdown_connections 2024-03-19T02:53:03.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.417+0000 7f4971d77640 1 --2- 172.21.15.33:0/313182651 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f49580a2800 0x7f49580a4cc0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:03.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.417+0000 7f4971d77640 1 --2- 172.21.15.33:0/313182651 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f496c1022d0 0x7f496c19ed80 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:03.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.417+0000 7f4971d77640 1 --2- 172.21.15.33:0/313182651 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f496c101910 0x7f496c19c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:03.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.417+0000 7f4971d77640 1 --2- 172.21.15.33:0/313182651 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f496c100720 0x7f496c0694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:03.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.417+0000 7f4971d77640 1 -- 172.21.15.33:0/313182651 >> 172.21.15.33:0/313182651 conn(0x7f496c0fbcb0 msgr2=0x7f496c104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:03.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.417+0000 7f4971d77640 1 -- 172.21.15.33:0/313182651 shutdown_connections 2024-03-19T02:53:03.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.417+0000 7f4971d77640 1 -- 172.21.15.33:0/313182651 wait complete. 2024-03-19T02:53:03.429 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T02:53:03.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.534+0000 7fe5e6359640 1 -- 172.21.15.33:0/1383055111 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5e01022d0 msgr2=0x7fe5e0108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:03.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.534+0000 7fe5e6359640 1 --2- 172.21.15.33:0/1383055111 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5e01022d0 0x7fe5e0108b60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8009be0 tx=0x7fe5d802bef0 comp rx=0 tx=0).stop 2024-03-19T02:53:03.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.534+0000 7fe5e6359640 1 -- 172.21.15.33:0/1383055111 shutdown_connections 2024-03-19T02:53:03.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.534+0000 7fe5e6359640 1 --2- 172.21.15.33:0/1383055111 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5e0101910 0x7fe5e0101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:03.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.534+0000 7fe5e6359640 1 --2- 172.21.15.33:0/1383055111 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5e01022d0 0x7fe5e0108b60 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:03.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.534+0000 7fe5e6359640 1 --2- 172.21.15.33:0/1383055111 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe5e0100720 0x7fe5e0100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:03.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.534+0000 7fe5e6359640 1 -- 172.21.15.33:0/1383055111 >> 172.21.15.33:0/1383055111 conn(0x7fe5e00fbcb0 msgr2=0x7fe5e00fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:03.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.534+0000 7fe5e6359640 1 -- 172.21.15.33:0/1383055111 shutdown_connections 2024-03-19T02:53:03.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.534+0000 7fe5e6359640 1 -- 172.21.15.33:0/1383055111 wait complete. 2024-03-19T02:53:03.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.534+0000 7fe5e6359640 1 Processor -- start 2024-03-19T02:53:03.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.535+0000 7fe5e6359640 1 -- start start 2024-03-19T02:53:03.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.535+0000 7fe5e6359640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5e0100720 0x7fe5e0197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:03.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.535+0000 7fe5e6359640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe5e0101910 0x7fe5e0197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:03.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.535+0000 7fe5e6359640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5e01022d0 0x7fe5e019f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:03.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.535+0000 7fe5e6359640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e010c230 con 0x7fe5e0101910 2024-03-19T02:53:03.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.535+0000 7fe5e6359640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e010c0b0 con 0x7fe5e01022d0 2024-03-19T02:53:03.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.535+0000 7fe5e6359640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e010c3b0 con 0x7fe5e0100720 2024-03-19T02:53:03.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.535+0000 7fe5df7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe5e0101910 0x7fe5e0197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:03.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.535+0000 7fe5df7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe5e0101910 0x7fe5e0197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46216/0 (socket says 172.21.15.33:46216) 2024-03-19T02:53:03.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.535+0000 7fe5df7fe640 1 -- 172.21.15.33:0/2634985021 learned_addr learned my addr 172.21.15.33:0/2634985021 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:03.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.535+0000 7fe5dffff640 1 --2- 172.21.15.33:0/2634985021 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5e0100720 0x7fe5e0197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:03.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.535+0000 7fe5e48f6640 1 --2- 172.21.15.33:0/2634985021 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5e01022d0 0x7fe5e019f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:03.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.535+0000 7fe5df7fe640 1 -- 172.21.15.33:0/2634985021 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5e0100720 msgr2=0x7fe5e0197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:03.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.535+0000 7fe5df7fe640 1 --2- 172.21.15.33:0/2634985021 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5e0100720 0x7fe5e0197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:03.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.535+0000 7fe5df7fe640 1 -- 172.21.15.33:0/2634985021 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5e01022d0 msgr2=0x7fe5e019f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:03.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.535+0000 7fe5df7fe640 1 --2- 172.21.15.33:0/2634985021 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5e01022d0 0x7fe5e019f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:03.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.535+0000 7fe5df7fe640 1 -- 172.21.15.33:0/2634985021 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5e019f6a0 con 0x7fe5e0101910 2024-03-19T02:53:03.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.536+0000 7fe5df7fe640 1 --2- 172.21.15.33:0/2634985021 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe5e0101910 0x7fe5e0197fc0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe5d400e9f0 tx=0x7fe5d400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:03.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.536+0000 7fe5dd7fa640 1 -- 172.21.15.33:0/2634985021 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d400cd80 con 0x7fe5e0101910 2024-03-19T02:53:03.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.536+0000 7fe5e6359640 1 -- 172.21.15.33:0/2634985021 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5e019f990 con 0x7fe5e0101910 2024-03-19T02:53:03.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.536+0000 7fe5dd7fa640 1 -- 172.21.15.33:0/2634985021 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5d40108e0 con 0x7fe5e0101910 2024-03-19T02:53:03.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.536+0000 7fe5dd7fa640 1 -- 172.21.15.33:0/2634985021 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d4018920 con 0x7fe5e0101910 2024-03-19T02:53:03.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.536+0000 7fe5e6359640 1 -- 172.21.15.33:0/2634985021 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5e019fee0 con 0x7fe5e0101910 2024-03-19T02:53:03.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.537+0000 7fe5e6359640 1 -- 172.21.15.33:0/2634985021 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5e01a0140 con 0x7fe5e0101910 2024-03-19T02:53:03.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.538+0000 7fe5dd7fa640 1 -- 172.21.15.33:0/2634985021 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe5d4010a90 con 0x7fe5e0101910 2024-03-19T02:53:03.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.538+0000 7fe5dd7fa640 1 --2- 172.21.15.33:0/2634985021 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe5c80a2800 0x7fe5c80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:03.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.539+0000 7fe5dffff640 1 --2- 172.21.15.33:0/2634985021 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe5c80a2800 0x7fe5c80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:03.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.539+0000 7fe5dd7fa640 1 -- 172.21.15.33:0/2634985021 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7fe5d40cf0e0 con 0x7fe5e0101910 2024-03-19T02:53:03.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.539+0000 7fe5dffff640 1 --2- 172.21.15.33:0/2634985021 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe5c80a2800 0x7fe5c80a4cc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0009530 tx=0x7fe5d0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:03.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.545+0000 7fe5dd7fa640 1 -- 172.21.15.33:0/2634985021 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe5d409b7a0 con 0x7fe5e0101910 2024-03-19T02:53:03.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.728+0000 7fe5e6359640 1 -- 172.21.15.33:0/2634985021 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fe5e01a0310 con 0x7fe5e0101910 2024-03-19T02:53:03.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.728+0000 7fe5dd7fa640 1 -- 172.21.15.33:0/2634985021 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v97) ==== 103+0+11 (secure 0 0 0) 0x7fe5d409f840 con 0x7fe5e0101910 2024-03-19T02:53:03.729 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 32 2024-03-19T02:53:03.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.730+0000 7fe5e6359640 1 -- 172.21.15.33:0/2634985021 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe5c80a2800 msgr2=0x7fe5c80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:03.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.730+0000 7fe5e6359640 1 --2- 172.21.15.33:0/2634985021 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe5c80a2800 0x7fe5c80a4cc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0009530 tx=0x7fe5d0009260 comp rx=0 tx=0).stop 2024-03-19T02:53:03.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.730+0000 7fe5e6359640 1 -- 172.21.15.33:0/2634985021 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe5e0101910 msgr2=0x7fe5e0197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:03.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.730+0000 7fe5e6359640 1 --2- 172.21.15.33:0/2634985021 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe5e0101910 0x7fe5e0197fc0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe5d400e9f0 tx=0x7fe5d400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:53:03.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.730+0000 7fe5e6359640 1 -- 172.21.15.33:0/2634985021 shutdown_connections 2024-03-19T02:53:03.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.730+0000 7fe5e6359640 1 --2- 172.21.15.33:0/2634985021 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fe5c80a2800 0x7fe5c80a4cc0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:03.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.730+0000 7fe5e6359640 1 --2- 172.21.15.33:0/2634985021 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5e01022d0 0x7fe5e019f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:03.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.730+0000 7fe5e6359640 1 --2- 172.21.15.33:0/2634985021 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe5e0101910 0x7fe5e0197fc0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:03.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.730+0000 7fe5e6359640 1 --2- 172.21.15.33:0/2634985021 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5e0100720 0x7fe5e0197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:03.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.730+0000 7fe5e6359640 1 -- 172.21.15.33:0/2634985021 >> 172.21.15.33:0/2634985021 conn(0x7fe5e00fbcb0 msgr2=0x7fe5e00fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:03.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.730+0000 7fe5e6359640 1 -- 172.21.15.33:0/2634985021 shutdown_connections 2024-03-19T02:53:03.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.730+0000 7fe5e6359640 1 -- 172.21.15.33:0/2634985021 wait complete. 2024-03-19T02:53:03.741 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:03 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/566180775' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:53:03.741 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:03 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3691500167' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:53:03.741 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:03 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/313182651' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:53:03.742 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:53:03.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.843+0000 7f7b7f61f640 1 -- 172.21.15.33:0/1936025081 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b78101930 msgr2=0x7f7b78101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:03.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.844+0000 7f7b7f61f640 1 --2- 172.21.15.33:0/1936025081 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b78101930 0x7f7b78101db0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7b68009900 tx=0x7f7b6802bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:03.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.844+0000 7f7b7f61f640 1 -- 172.21.15.33:0/1936025081 shutdown_connections 2024-03-19T02:53:03.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.844+0000 7f7b7f61f640 1 --2- 172.21.15.33:0/1936025081 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b781022f0 0x7f7b78108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:03.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.844+0000 7f7b7f61f640 1 --2- 172.21.15.33:0/1936025081 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b78101930 0x7f7b78101db0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:03.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.844+0000 7f7b7f61f640 1 --2- 172.21.15.33:0/1936025081 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b78100740 0x7f7b78100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:03.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.844+0000 7f7b7f61f640 1 -- 172.21.15.33:0/1936025081 >> 172.21.15.33:0/1936025081 conn(0x7f7b780fbcb0 msgr2=0x7f7b780fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:03.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.844+0000 7f7b7f61f640 1 -- 172.21.15.33:0/1936025081 shutdown_connections 2024-03-19T02:53:03.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.844+0000 7f7b7f61f640 1 -- 172.21.15.33:0/1936025081 wait complete. 2024-03-19T02:53:03.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.844+0000 7f7b7f61f640 1 Processor -- start 2024-03-19T02:53:03.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.844+0000 7f7b7f61f640 1 -- start start 2024-03-19T02:53:03.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.844+0000 7f7b7f61f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b78100740 0x7f7b781969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:03.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.844+0000 7f7b7f61f640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b78101930 0x7f7b78196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:03.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.844+0000 7f7b7f61f640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b781022f0 0x7f7b7819df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:03.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.844+0000 7f7b7f61f640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7810c270 con 0x7f7b78100740 2024-03-19T02:53:03.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.844+0000 7f7b7f61f640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7810c0f0 con 0x7f7b781022f0 2024-03-19T02:53:03.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.844+0000 7f7b7f61f640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7810c3f0 con 0x7f7b78101930 2024-03-19T02:53:03.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.844+0000 7f7b7d3bb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b78100740 0x7f7b781969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:03.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.845+0000 7f7b7d3bb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b78100740 0x7f7b781969a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46240/0 (socket says 172.21.15.33:46240) 2024-03-19T02:53:03.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.845+0000 7f7b7d3bb640 1 -- 172.21.15.33:0/1278739882 learned_addr learned my addr 172.21.15.33:0/1278739882 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:03.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.845+0000 7f7b7dbbc640 1 --2- 172.21.15.33:0/1278739882 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b781022f0 0x7f7b7819df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:03.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.845+0000 7f7b7cbba640 1 --2- 172.21.15.33:0/1278739882 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b78101930 0x7f7b78196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:03.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.845+0000 7f7b7d3bb640 1 -- 172.21.15.33:0/1278739882 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b78101930 msgr2=0x7f7b78196ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:03.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.845+0000 7f7b7d3bb640 1 --2- 172.21.15.33:0/1278739882 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b78101930 0x7f7b78196ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:03.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.845+0000 7f7b7d3bb640 1 -- 172.21.15.33:0/1278739882 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b781022f0 msgr2=0x7f7b7819df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:03.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.845+0000 7f7b7d3bb640 1 --2- 172.21.15.33:0/1278739882 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b781022f0 0x7f7b7819df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:03.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.845+0000 7f7b7d3bb640 1 -- 172.21.15.33:0/1278739882 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b7819e5c0 con 0x7f7b78100740 2024-03-19T02:53:03.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.845+0000 7f7b7dbbc640 1 --2- 172.21.15.33:0/1278739882 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b781022f0 0x7f7b7819df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:03.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.845+0000 7f7b7d3bb640 1 --2- 172.21.15.33:0/1278739882 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b78100740 0x7f7b781969a0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7b7400ead0 tx=0x7f7b7400ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:03.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.845+0000 7f7b6e7fc640 1 -- 172.21.15.33:0/1278739882 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7401c070 con 0x7f7b78100740 2024-03-19T02:53:03.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.845+0000 7f7b6e7fc640 1 -- 172.21.15.33:0/1278739882 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7b7400bbd0 con 0x7f7b78100740 2024-03-19T02:53:03.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.845+0000 7f7b7f61f640 1 -- 172.21.15.33:0/1278739882 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b780689e0 con 0x7f7b78100740 2024-03-19T02:53:03.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.845+0000 7f7b6e7fc640 1 -- 172.21.15.33:0/1278739882 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b74003d00 con 0x7f7b78100740 2024-03-19T02:53:03.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.845+0000 7f7b7f61f640 1 -- 172.21.15.33:0/1278739882 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b78068eb0 con 0x7f7b78100740 2024-03-19T02:53:03.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.846+0000 7f7b7f61f640 1 -- 172.21.15.33:0/1278739882 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b78068610 con 0x7f7b78100740 2024-03-19T02:53:03.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.847+0000 7f7b6e7fc640 1 -- 172.21.15.33:0/1278739882 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7b7401f070 con 0x7f7b78100740 2024-03-19T02:53:03.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.848+0000 7f7b6e7fc640 1 --2- 172.21.15.33:0/1278739882 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7b640a2800 0x7f7b640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:03.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.848+0000 7f7b7cbba640 1 --2- 172.21.15.33:0/1278739882 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7b640a2800 0x7f7b640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:03.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.848+0000 7f7b6e7fc640 1 -- 172.21.15.33:0/1278739882 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f7b740d04e0 con 0x7f7b78100740 2024-03-19T02:53:03.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.848+0000 7f7b7cbba640 1 --2- 172.21.15.33:0/1278739882 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7b640a2800 0x7f7b640a4cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7b68009230 tx=0x7f7b68005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:03.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:03.852+0000 7f7b6e7fc640 1 -- 172.21.15.33:0/1278739882 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7b7409cb40 con 0x7f7b78100740 2024-03-19T02:53:03.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:03 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/566180775' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:53:03.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:03 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3691500167' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:53:03.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:03 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/313182651' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:53:04.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:03 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/566180775' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:53:04.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:03 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3691500167' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:53:04.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:03 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/313182651' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:53:04.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.076+0000 7f7b7f61f640 1 -- 172.21.15.33:0/1278739882 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7b78107550 con 0x7f7b78100740 2024-03-19T02:53:04.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.076+0000 7f7b6e7fc640 1 -- 172.21.15.33:0/1278739882 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11867 (secure 0 0 0) 0x7f7b740a0be0 con 0x7f7b78100740 2024-03-19T02:53:04.077 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:53:04.078 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:50:52.201798+0000","modified":"2024-03-19T02:51:19.076317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24488,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/3731232169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":3731232169},{"type":"v1","addr":"172.21.15.96:6839","nonce":3731232169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24457,24509,24488]},"id":1}]} 2024-03-19T02:53:04.078 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 24 2024-03-19T02:53:04.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.078+0000 7f7b7f61f640 1 -- 172.21.15.33:0/1278739882 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7b640a2800 msgr2=0x7f7b640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:04.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.078+0000 7f7b7f61f640 1 --2- 172.21.15.33:0/1278739882 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7b640a2800 0x7f7b640a4cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7b68009230 tx=0x7f7b68005fd0 comp rx=0 tx=0).stop 2024-03-19T02:53:04.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.078+0000 7f7b7f61f640 1 -- 172.21.15.33:0/1278739882 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b78100740 msgr2=0x7f7b781969a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:04.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.078+0000 7f7b7f61f640 1 --2- 172.21.15.33:0/1278739882 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b78100740 0x7f7b781969a0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7b7400ead0 tx=0x7f7b7400ef90 comp rx=0 tx=0).stop 2024-03-19T02:53:04.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.078+0000 7f7b7f61f640 1 -- 172.21.15.33:0/1278739882 shutdown_connections 2024-03-19T02:53:04.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.078+0000 7f7b7f61f640 1 --2- 172.21.15.33:0/1278739882 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7b640a2800 0x7f7b640a4cc0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:04.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.078+0000 7f7b7f61f640 1 --2- 172.21.15.33:0/1278739882 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b78101930 0x7f7b78196ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:04.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.078+0000 7f7b7f61f640 1 --2- 172.21.15.33:0/1278739882 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b781022f0 0x7f7b7819df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:04.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.078+0000 7f7b7f61f640 1 --2- 172.21.15.33:0/1278739882 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7b78100740 0x7f7b781969a0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:04.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.078+0000 7f7b7f61f640 1 -- 172.21.15.33:0/1278739882 >> 172.21.15.33:0/1278739882 conn(0x7f7b780fbcb0 msgr2=0x7f7b78104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:04.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.078+0000 7f7b7f61f640 1 -- 172.21.15.33:0/1278739882 shutdown_connections 2024-03-19T02:53:04.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.078+0000 7f7b7f61f640 1 -- 172.21.15.33:0/1278739882 wait complete. 2024-03-19T02:53:04.090 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:53:04.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.192+0000 7f9f9a47e640 1 -- 172.21.15.33:0/1484712686 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f94102310 msgr2=0x7f9f94108ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:04.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.192+0000 7f9f9a47e640 1 --2- 172.21.15.33:0/1484712686 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f94102310 0x7f9f94108ba0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c009ca0 tx=0x7f9f8c02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:53:04.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.192+0000 7f9f9a47e640 1 -- 172.21.15.33:0/1484712686 shutdown_connections 2024-03-19T02:53:04.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.192+0000 7f9f9a47e640 1 --2- 172.21.15.33:0/1484712686 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f94102310 0x7f9f94108ba0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:04.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.192+0000 7f9f9a47e640 1 --2- 172.21.15.33:0/1484712686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f94101950 0x7f9f94101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:04.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.192+0000 7f9f9a47e640 1 --2- 172.21.15.33:0/1484712686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f94100760 0x7f9f94100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:04.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.192+0000 7f9f9a47e640 1 -- 172.21.15.33:0/1484712686 >> 172.21.15.33:0/1484712686 conn(0x7f9f940fbcd0 msgr2=0x7f9f940fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:04.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.192+0000 7f9f9a47e640 1 -- 172.21.15.33:0/1484712686 shutdown_connections 2024-03-19T02:53:04.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.192+0000 7f9f9a47e640 1 -- 172.21.15.33:0/1484712686 wait complete. 2024-03-19T02:53:04.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.193+0000 7f9f9a47e640 1 Processor -- start 2024-03-19T02:53:04.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.193+0000 7f9f9a47e640 1 -- start start 2024-03-19T02:53:04.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.193+0000 7f9f9a47e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f94100760 0x7f9f9406a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:04.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.193+0000 7f9f9a47e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f94101950 0x7f9f9419c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:04.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.193+0000 7f9f9a47e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f94102310 0x7f9f9419ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:04.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.193+0000 7f9f9a47e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9410c270 con 0x7f9f94100760 2024-03-19T02:53:04.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.193+0000 7f9f9a47e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9410c0f0 con 0x7f9f94101950 2024-03-19T02:53:04.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.193+0000 7f9f9a47e640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9410c3f0 con 0x7f9f94102310 2024-03-19T02:53:04.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.193+0000 7f9f93fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f94100760 0x7f9f9406a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:04.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.193+0000 7f9f93fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f94100760 0x7f9f9406a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46258/0 (socket says 172.21.15.33:46258) 2024-03-19T02:53:04.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.193+0000 7f9f93fff640 1 -- 172.21.15.33:0/2588291678 learned_addr learned my addr 172.21.15.33:0/2588291678 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:04.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.193+0000 7f9f937fe640 1 --2- 172.21.15.33:0/2588291678 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f94101950 0x7f9f9419c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:04.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.193+0000 7f9f98a1b640 1 --2- 172.21.15.33:0/2588291678 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f94102310 0x7f9f9419ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:04.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.193+0000 7f9f93fff640 1 -- 172.21.15.33:0/2588291678 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f94102310 msgr2=0x7f9f9419ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:04.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.193+0000 7f9f93fff640 1 --2- 172.21.15.33:0/2588291678 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f94102310 0x7f9f9419ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:04.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.193+0000 7f9f93fff640 1 -- 172.21.15.33:0/2588291678 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f94101950 msgr2=0x7f9f9419c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:04.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.193+0000 7f9f93fff640 1 --2- 172.21.15.33:0/2588291678 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f94101950 0x7f9f9419c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:04.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.193+0000 7f9f93fff640 1 -- 172.21.15.33:0/2588291678 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f9419f690 con 0x7f9f94100760 2024-03-19T02:53:04.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.193+0000 7f9f937fe640 1 --2- 172.21.15.33:0/2588291678 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f94101950 0x7f9f9419c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:04.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.194+0000 7f9f93fff640 1 --2- 172.21.15.33:0/2588291678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f94100760 0x7f9f9406a9c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f9f8400e9c0 tx=0x7f9f8400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:04.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.194+0000 7f9f917fa640 1 -- 172.21.15.33:0/2588291678 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8400cd80 con 0x7f9f94100760 2024-03-19T02:53:04.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.194+0000 7f9f917fa640 1 -- 172.21.15.33:0/2588291678 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9f840035f0 con 0x7f9f94100760 2024-03-19T02:53:04.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.194+0000 7f9f9a47e640 1 -- 172.21.15.33:0/2588291678 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f9419f920 con 0x7f9f94100760 2024-03-19T02:53:04.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.194+0000 7f9f917fa640 1 -- 172.21.15.33:0/2588291678 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f84010690 con 0x7f9f94100760 2024-03-19T02:53:04.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.194+0000 7f9f9a47e640 1 -- 172.21.15.33:0/2588291678 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f9419fe70 con 0x7f9f94100760 2024-03-19T02:53:04.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.195+0000 7f9f9a47e640 1 -- 172.21.15.33:0/2588291678 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f94068a60 con 0x7f9f94100760 2024-03-19T02:53:04.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.196+0000 7f9f917fa640 1 -- 172.21.15.33:0/2588291678 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9f840037a0 con 0x7f9f94100760 2024-03-19T02:53:04.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.196+0000 7f9f917fa640 1 --2- 172.21.15.33:0/2588291678 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9f7c0a2800 0x7f9f7c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:04.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.197+0000 7f9f937fe640 1 --2- 172.21.15.33:0/2588291678 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9f7c0a2800 0x7f9f7c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:04.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.197+0000 7f9f917fa640 1 -- 172.21.15.33:0/2588291678 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f9f840cf290 con 0x7f9f94100760 2024-03-19T02:53:04.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.197+0000 7f9f937fe640 1 --2- 172.21.15.33:0/2588291678 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9f7c0a2800 0x7f9f7c0a4cc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9f9406a730 tx=0x7f9f88005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:04.201 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.200+0000 7f9f917fa640 1 -- 172.21.15.33:0/2588291678 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9f8409bfa0 con 0x7f9f94100760 2024-03-19T02:53:04.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.385+0000 7f9f9a47e640 1 -- 172.21.15.33:0/2588291678 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9f9402d2b0 con 0x7f9f94100760 2024-03-19T02:53:04.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.386+0000 7f9f917fa640 1 -- 172.21.15.33:0/2588291678 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) ==== 74+0+24663 (secure 0 0 0) 0x7f9f8409b8a0 con 0x7f9f94100760 2024-03-19T02:53:04.387 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:53:04.388 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":97,"fsid":"2b7a8938-e59a-11ee-95c9-87774f69a715","created":"2024-03-19T02:42:37.912179+0000","modified":"2024-03-19T02:52:41.116227+0000","last_up_change":"2024-03-19T02:50:01.539392+0000","last_in_change":"2024-03-19T02:49:40.534415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:46:09.963737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:50:50.069641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:50:51.381064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:50:55.476667+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"090a89ab-6b7b-4317-8823-178d4212b2f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6803","nonce":1633124995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6805","nonce":1633124995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6809","nonce":1633124995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6807","nonce":1633124995}]},"public_addr":"172.21.15.33:6803/1633124995","cluster_addr":"172.21.15.33:6805/1633124995","heartbeat_back_addr":"172.21.15.33:6809/1633124995","heartbeat_front_addr":"172.21.15.33:6807/1633124995","state":["exists","up"]},{"osd":1,"uuid":"f50eec55-1956-4486-8593-f2c8fa610e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6801","nonce":340144398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6803","nonce":340144398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6807","nonce":340144398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6805","nonce":340144398}]},"public_addr":"172.21.15.96:6801/340144398","cluster_addr":"172.21.15.96:6803/340144398","heartbeat_back_addr":"172.21.15.96:6807/340144398","heartbeat_front_addr":"172.21.15.96:6805/340144398","state":["exists","up"]},{"osd":2,"uuid":"893a2d62-80d9-4e5e-bc6a-a1c8c681d652","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6801","nonce":3388743876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6803","nonce":3388743876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6807","nonce":3388743876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6805","nonce":3388743876}]},"public_addr":"172.21.15.155:6801/3388743876","cluster_addr":"172.21.15.155:6803/3388743876","heartbeat_back_addr":"172.21.15.155:6807/3388743876","heartbeat_front_addr":"172.21.15.155:6805/3388743876","state":["exists","up"]},{"osd":3,"uuid":"b07e0482-4281-4d32-9dfe-9995d7246175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6811","nonce":677007171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6813","nonce":677007171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6817","nonce":677007171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6815","nonce":677007171}]},"public_addr":"172.21.15.33:6811/677007171","cluster_addr":"172.21.15.33:6813/677007171","heartbeat_back_addr":"172.21.15.33:6817/677007171","heartbeat_front_addr":"172.21.15.33:6815/677007171","state":["exists","up"]},{"osd":4,"uuid":"a988384b-af6b-47c6-8c07-5c9673acee45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6809","nonce":3106158554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6811","nonce":3106158554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6815","nonce":3106158554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6813","nonce":3106158554}]},"public_addr":"172.21.15.96:6809/3106158554","cluster_addr":"172.21.15.96:6811/3106158554","heartbeat_back_addr":"172.21.15.96:6815/3106158554","heartbeat_front_addr":"172.21.15.96:6813/3106158554","state":["exists","up"]},{"osd":5,"uuid":"198dc992-96f8-4c52-96ee-1fd5de148bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6809","nonce":327293324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6811","nonce":327293324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6815","nonce":327293324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6813","nonce":327293324}]},"public_addr":"172.21.15.155:6809/327293324","cluster_addr":"172.21.15.155:6811/327293324","heartbeat_back_addr":"172.21.15.155:6815/327293324","heartbeat_front_addr":"172.21.15.155:6813/327293324","state":["exists","up"]},{"osd":6,"uuid":"a7278ec0-ab2e-49ed-be53-84d8b9e24177","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6819","nonce":873871509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6821","nonce":873871509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6825","nonce":873871509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6823","nonce":873871509}]},"public_addr":"172.21.15.33:6819/873871509","cluster_addr":"172.21.15.33:6821/873871509","heartbeat_back_addr":"172.21.15.33:6825/873871509","heartbeat_front_addr":"172.21.15.33:6823/873871509","state":["exists","up"]},{"osd":7,"uuid":"727e133d-aa24-4ffc-90e2-266eb2aab8c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6817","nonce":2334123891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6819","nonce":2334123891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6823","nonce":2334123891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6821","nonce":2334123891}]},"public_addr":"172.21.15.96:6817/2334123891","cluster_addr":"172.21.15.96:6819/2334123891","heartbeat_back_addr":"172.21.15.96:6823/2334123891","heartbeat_front_addr":"172.21.15.96:6821/2334123891","state":["exists","up"]},{"osd":8,"uuid":"bb20eec0-39e5-4b86-9db5-486b1840cdad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6817","nonce":3498516903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6819","nonce":3498516903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6823","nonce":3498516903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6821","nonce":3498516903}]},"public_addr":"172.21.15.155:6817/3498516903","cluster_addr":"172.21.15.155:6819/3498516903","heartbeat_back_addr":"172.21.15.155:6823/3498516903","heartbeat_front_addr":"172.21.15.155:6821/3498516903","state":["exists","up"]},{"osd":9,"uuid":"179c6228-3e23-4f65-91a6-1ef0005d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6827","nonce":760777213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6829","nonce":760777213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6833","nonce":760777213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6831","nonce":760777213}]},"public_addr":"172.21.15.33:6827/760777213","cluster_addr":"172.21.15.33:6829/760777213","heartbeat_back_addr":"172.21.15.33:6833/760777213","heartbeat_front_addr":"172.21.15.33:6831/760777213","state":["exists","up"]},{"osd":10,"uuid":"be17b478-3c2e-41fb-bede-958333acdf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6825","nonce":1404898482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6827","nonce":1404898482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6831","nonce":1404898482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6829","nonce":1404898482}]},"public_addr":"172.21.15.96:6825/1404898482","cluster_addr":"172.21.15.96:6827/1404898482","heartbeat_back_addr":"172.21.15.96:6831/1404898482","heartbeat_front_addr":"172.21.15.96:6829/1404898482","state":["exists","up"]},{"osd":11,"uuid":"c0903ed2-cc1e-4c18-8731-1f0ee4dd0457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6825","nonce":792352827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6827","nonce":792352827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6831","nonce":792352827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6829","nonce":792352827}]},"public_addr":"172.21.15.155:6825/792352827","cluster_addr":"172.21.15.155:6827/792352827","heartbeat_back_addr":"172.21.15.155:6831/792352827","heartbeat_front_addr":"172.21.15.155:6829/792352827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:17.942801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:42.109449+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:05.981599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:32.069574+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:57.431015+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:22.448592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:48.526515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:14.498352+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:40.175426+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:06.364006+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:32.853161+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:59.073538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.9","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:6801/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:6800/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/1067768047":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/729379453":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2586588742":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/2932814165":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6801/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2212564963":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6800/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/3275126058":"2024-03-20T02:43:39.622218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:53:04.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.388+0000 7f9f9a47e640 1 -- 172.21.15.33:0/2588291678 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9f7c0a2800 msgr2=0x7f9f7c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:04.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.388+0000 7f9f9a47e640 1 --2- 172.21.15.33:0/2588291678 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9f7c0a2800 0x7f9f7c0a4cc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9f9406a730 tx=0x7f9f88005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:04.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.388+0000 7f9f9a47e640 1 -- 172.21.15.33:0/2588291678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f94100760 msgr2=0x7f9f9406a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:04.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.388+0000 7f9f9a47e640 1 --2- 172.21.15.33:0/2588291678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f94100760 0x7f9f9406a9c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f9f8400e9c0 tx=0x7f9f8400ee80 comp rx=0 tx=0).stop 2024-03-19T02:53:04.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.388+0000 7f9f9a47e640 1 -- 172.21.15.33:0/2588291678 shutdown_connections 2024-03-19T02:53:04.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.388+0000 7f9f9a47e640 1 --2- 172.21.15.33:0/2588291678 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9f7c0a2800 0x7f9f7c0a4cc0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:04.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.388+0000 7f9f9a47e640 1 --2- 172.21.15.33:0/2588291678 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f94102310 0x7f9f9419ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:04.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.388+0000 7f9f9a47e640 1 --2- 172.21.15.33:0/2588291678 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f94101950 0x7f9f9419c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:04.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.388+0000 7f9f9a47e640 1 --2- 172.21.15.33:0/2588291678 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f94100760 0x7f9f9406a9c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:04.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.388+0000 7f9f9a47e640 1 -- 172.21.15.33:0/2588291678 >> 172.21.15.33:0/2588291678 conn(0x7f9f940fbcd0 msgr2=0x7f9f94104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:04.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.388+0000 7f9f9a47e640 1 -- 172.21.15.33:0/2588291678 shutdown_connections 2024-03-19T02:53:04.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.388+0000 7f9f9a47e640 1 -- 172.21.15.33:0/2588291678 wait complete. 2024-03-19T02:53:04.400 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:46:09.963737+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T02:50:50.069641+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T02:50:51.381064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T02:50:55.476667+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T02:53:04.401 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T02:53:04.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.505+0000 7f10090f8640 1 -- 172.21.15.33:0/855761374 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1004100720 msgr2=0x7f1004100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:04.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.505+0000 7f10090f8640 1 --2- 172.21.15.33:0/855761374 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1004100720 0x7f1004100b40 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc009900 tx=0x7f0ffc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:04.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.505+0000 7f10090f8640 1 -- 172.21.15.33:0/855761374 shutdown_connections 2024-03-19T02:53:04.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.505+0000 7f10090f8640 1 --2- 172.21.15.33:0/855761374 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1004101910 0x7f1004101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:04.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.505+0000 7f10090f8640 1 --2- 172.21.15.33:0/855761374 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10041022d0 0x7f1004108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:04.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.505+0000 7f10090f8640 1 --2- 172.21.15.33:0/855761374 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1004100720 0x7f1004100b40 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:04.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.505+0000 7f10090f8640 1 -- 172.21.15.33:0/855761374 >> 172.21.15.33:0/855761374 conn(0x7f10040fbcb0 msgr2=0x7f10040fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:04.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.505+0000 7f10090f8640 1 -- 172.21.15.33:0/855761374 shutdown_connections 2024-03-19T02:53:04.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.505+0000 7f10090f8640 1 -- 172.21.15.33:0/855761374 wait complete. 2024-03-19T02:53:04.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.506+0000 7f10090f8640 1 Processor -- start 2024-03-19T02:53:04.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.506+0000 7f10090f8640 1 -- start start 2024-03-19T02:53:04.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.506+0000 7f10090f8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1004100720 0x7f1004197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:04.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.506+0000 7f10090f8640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1004101910 0x7f1004198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:04.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.506+0000 7f10090f8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10041022d0 0x7f100419f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:04.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.506+0000 7f10090f8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f100410c240 con 0x7f1004100720 2024-03-19T02:53:04.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.506+0000 7f10090f8640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f100410c0c0 con 0x7f10041022d0 2024-03-19T02:53:04.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.506+0000 7f10090f8640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f100410c3c0 con 0x7f1004101910 2024-03-19T02:53:04.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.506+0000 7f1002d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1004100720 0x7f1004197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:04.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.506+0000 7f1002d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1004100720 0x7f1004197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46284/0 (socket says 172.21.15.33:46284) 2024-03-19T02:53:04.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.506+0000 7f1002d9d640 1 -- 172.21.15.33:0/2506717103 learned_addr learned my addr 172.21.15.33:0/2506717103 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:04.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.506+0000 7f100359e640 1 --2- 172.21.15.33:0/2506717103 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10041022d0 0x7f100419f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:04.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.507+0000 7f100259c640 1 --2- 172.21.15.33:0/2506717103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1004101910 0x7f1004198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:04.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.507+0000 7f1002d9d640 1 -- 172.21.15.33:0/2506717103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1004101910 msgr2=0x7f1004198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:04.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.507+0000 7f1002d9d640 1 --2- 172.21.15.33:0/2506717103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1004101910 0x7f1004198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:04.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.507+0000 7f1002d9d640 1 -- 172.21.15.33:0/2506717103 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10041022d0 msgr2=0x7f100419f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:04.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.507+0000 7f1002d9d640 1 --2- 172.21.15.33:0/2506717103 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10041022d0 0x7f100419f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:04.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.507+0000 7f1002d9d640 1 -- 172.21.15.33:0/2506717103 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f100419f710 con 0x7f1004100720 2024-03-19T02:53:04.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.507+0000 7f1002d9d640 1 --2- 172.21.15.33:0/2506717103 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1004100720 0x7f1004197af0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc000c00 tx=0x7f0ffc005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:04.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.507+0000 7f0ffbfff640 1 -- 172.21.15.33:0/2506717103 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc043020 con 0x7f1004100720 2024-03-19T02:53:04.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.507+0000 7f10090f8640 1 -- 172.21.15.33:0/2506717103 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f100419f9a0 con 0x7f1004100720 2024-03-19T02:53:04.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.507+0000 7f0ffbfff640 1 -- 172.21.15.33:0/2506717103 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0ffc004350 con 0x7f1004100720 2024-03-19T02:53:04.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.507+0000 7f0ffbfff640 1 -- 172.21.15.33:0/2506717103 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc003b80 con 0x7f1004100720 2024-03-19T02:53:04.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.507+0000 7f10090f8640 1 -- 172.21.15.33:0/2506717103 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f100419fef0 con 0x7f1004100720 2024-03-19T02:53:04.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.508+0000 7f10090f8640 1 -- 172.21.15.33:0/2506717103 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1004068610 con 0x7f1004100720 2024-03-19T02:53:04.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.509+0000 7f0ffbfff640 1 -- 172.21.15.33:0/2506717103 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0ffc046020 con 0x7f1004100720 2024-03-19T02:53:04.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.510+0000 7f0ffbfff640 1 --2- 172.21.15.33:0/2506717103 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f0ff00a2800 0x7f0ff00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:04.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.510+0000 7f100259c640 1 --2- 172.21.15.33:0/2506717103 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f0ff00a2800 0x7f0ff00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:04.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.510+0000 7f0ffbfff640 1 -- 172.21.15.33:0/2506717103 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f0ffc0f0f70 con 0x7f1004100720 2024-03-19T02:53:04.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.510+0000 7f100259c640 1 --2- 172.21.15.33:0/2506717103 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f0ff00a2800 0x7f0ff00a4cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1004199000 tx=0x7f0fec005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:04.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.514+0000 7f0ffbfff640 1 -- 172.21.15.33:0/2506717103 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0ffc04d020 con 0x7f1004100720 2024-03-19T02:53:04.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.696+0000 7f10090f8640 1 -- 172.21.15.33:0/2506717103 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f10041a00c0 con 0x7f1004100720 2024-03-19T02:53:04.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.696+0000 7f0ffbfff640 1 -- 172.21.15.33:0/2506717103 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v97) ==== 93+0+10 (secure 0 0 0) 0x7f0ffc0bdc20 con 0x7f1004100720 2024-03-19T02:53:04.697 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 1 2024-03-19T02:53:04.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.698+0000 7f10090f8640 1 -- 172.21.15.33:0/2506717103 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f0ff00a2800 msgr2=0x7f0ff00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:04.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.698+0000 7f10090f8640 1 --2- 172.21.15.33:0/2506717103 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f0ff00a2800 0x7f0ff00a4cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1004199000 tx=0x7f0fec005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:04.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.698+0000 7f10090f8640 1 -- 172.21.15.33:0/2506717103 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1004100720 msgr2=0x7f1004197af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:04.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.698+0000 7f10090f8640 1 --2- 172.21.15.33:0/2506717103 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1004100720 0x7f1004197af0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc000c00 tx=0x7f0ffc005dc0 comp rx=0 tx=0).stop 2024-03-19T02:53:04.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.698+0000 7f10090f8640 1 -- 172.21.15.33:0/2506717103 shutdown_connections 2024-03-19T02:53:04.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.698+0000 7f10090f8640 1 --2- 172.21.15.33:0/2506717103 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f0ff00a2800 0x7f0ff00a4cc0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:04.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.698+0000 7f10090f8640 1 --2- 172.21.15.33:0/2506717103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1004101910 0x7f1004198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:04.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.698+0000 7f10090f8640 1 --2- 172.21.15.33:0/2506717103 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10041022d0 0x7f100419f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:04.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.698+0000 7f10090f8640 1 --2- 172.21.15.33:0/2506717103 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1004100720 0x7f1004197af0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:04.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.698+0000 7f10090f8640 1 -- 172.21.15.33:0/2506717103 >> 172.21.15.33:0/2506717103 conn(0x7f10040fbcb0 msgr2=0x7f10040689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:04.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.698+0000 7f10090f8640 1 -- 172.21.15.33:0/2506717103 shutdown_connections 2024-03-19T02:53:04.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.698+0000 7f10090f8640 1 -- 172.21.15.33:0/2506717103 wait complete. 2024-03-19T02:53:04.709 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:04 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2634985021' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:53:04.709 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:04 smithi033 ceph-mon[31973]: pgmap v369: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:53:04.709 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:04 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1278739882' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:53:04.709 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:04 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2588291678' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:53:04.710 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T02:53:04.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.814+0000 7feadd5f4640 1 -- 172.21.15.33:0/210779423 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fead8102ac0 msgr2=0x7fead8102f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:04.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.814+0000 7feadd5f4640 1 --2- 172.21.15.33:0/210779423 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fead8102ac0 0x7fead8102f40 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7feac0009930 tx=0x7feac002bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:04.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.814+0000 7feadd5f4640 1 -- 172.21.15.33:0/210779423 shutdown_connections 2024-03-19T02:53:04.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.814+0000 7feadd5f4640 1 --2- 172.21.15.33:0/210779423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fead8103480 0x7fead8109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:04.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.814+0000 7feadd5f4640 1 --2- 172.21.15.33:0/210779423 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fead8102ac0 0x7fead8102f40 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:04.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.814+0000 7feadd5f4640 1 --2- 172.21.15.33:0/210779423 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fead80ffaf0 0x7fead80fff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:04.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.814+0000 7feadd5f4640 1 -- 172.21.15.33:0/210779423 >> 172.21.15.33:0/210779423 conn(0x7fead80fb440 msgr2=0x7fead80fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:04.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.815+0000 7feadd5f4640 1 -- 172.21.15.33:0/210779423 shutdown_connections 2024-03-19T02:53:04.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.815+0000 7feadd5f4640 1 -- 172.21.15.33:0/210779423 wait complete. 2024-03-19T02:53:04.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.815+0000 7feadd5f4640 1 Processor -- start 2024-03-19T02:53:04.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.815+0000 7feadd5f4640 1 -- start start 2024-03-19T02:53:04.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.815+0000 7feadd5f4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fead80ffaf0 0x7fead8102430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:04.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.815+0000 7feadd5f4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fead8102ac0 0x7fead8100a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:04.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.815+0000 7feadd5f4640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fead8103480 0x7fead8100fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:04.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.815+0000 7feadd5f4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fead810d380 con 0x7fead80ffaf0 2024-03-19T02:53:04.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.815+0000 7feadd5f4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fead810d200 con 0x7fead8102ac0 2024-03-19T02:53:04.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.815+0000 7feadd5f4640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fead810d500 con 0x7fead8103480 2024-03-19T02:53:04.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.816+0000 7fead6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fead80ffaf0 0x7fead8102430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:04.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.816+0000 7fead6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fead80ffaf0 0x7fead8102430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46298/0 (socket says 172.21.15.33:46298) 2024-03-19T02:53:04.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.816+0000 7fead6ffd640 1 -- 172.21.15.33:0/2254796822 learned_addr learned my addr 172.21.15.33:0/2254796822 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:04.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.816+0000 7fead67fc640 1 --2- 172.21.15.33:0/2254796822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fead8102ac0 0x7fead8100a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:04.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.816+0000 7fead77fe640 1 --2- 172.21.15.33:0/2254796822 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fead8103480 0x7fead8100fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:04.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.816+0000 7fead6ffd640 1 -- 172.21.15.33:0/2254796822 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fead8103480 msgr2=0x7fead8100fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:04.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.816+0000 7fead6ffd640 1 --2- 172.21.15.33:0/2254796822 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fead8103480 0x7fead8100fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:04.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.816+0000 7fead6ffd640 1 -- 172.21.15.33:0/2254796822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fead8102ac0 msgr2=0x7fead8100a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:04.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.816+0000 7fead6ffd640 1 --2- 172.21.15.33:0/2254796822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fead8102ac0 0x7fead8100a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:04.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.816+0000 7fead6ffd640 1 -- 172.21.15.33:0/2254796822 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fead8101840 con 0x7fead80ffaf0 2024-03-19T02:53:04.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.816+0000 7fead67fc640 1 --2- 172.21.15.33:0/2254796822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fead8102ac0 0x7fead8100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:04.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.816+0000 7fead6ffd640 1 --2- 172.21.15.33:0/2254796822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fead80ffaf0 0x7fead8102430 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fead000e9c0 tx=0x7fead000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:04.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.816+0000 7feacffff640 1 -- 172.21.15.33:0/2254796822 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead000cd80 con 0x7fead80ffaf0 2024-03-19T02:53:04.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.816+0000 7feacffff640 1 -- 172.21.15.33:0/2254796822 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fead00035f0 con 0x7fead80ffaf0 2024-03-19T02:53:04.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.816+0000 7feadd5f4640 1 -- 172.21.15.33:0/2254796822 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fead81a1c10 con 0x7fead80ffaf0 2024-03-19T02:53:04.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.816+0000 7feacffff640 1 -- 172.21.15.33:0/2254796822 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead0010690 con 0x7fead80ffaf0 2024-03-19T02:53:04.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.816+0000 7feadd5f4640 1 -- 172.21.15.33:0/2254796822 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fead81a2130 con 0x7fead80ffaf0 2024-03-19T02:53:04.821 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.817+0000 7feadd5f4640 1 -- 172.21.15.33:0/2254796822 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fead8068a60 con 0x7fead80ffaf0 2024-03-19T02:53:04.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.818+0000 7feacffff640 1 -- 172.21.15.33:0/2254796822 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fead00037a0 con 0x7fead80ffaf0 2024-03-19T02:53:04.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.819+0000 7feacffff640 1 --2- 172.21.15.33:0/2254796822 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7feac40a2800 0x7feac40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:04.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.819+0000 7fead67fc640 1 --2- 172.21.15.33:0/2254796822 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7feac40a2800 0x7feac40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:04.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.819+0000 7feacffff640 1 -- 172.21.15.33:0/2254796822 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7fead00cf1c0 con 0x7fead80ffaf0 2024-03-19T02:53:04.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.819+0000 7fead67fc640 1 --2- 172.21.15.33:0/2254796822 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7feac40a2800 0x7feac40a4cc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7feac0009900 tx=0x7feac0005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:04.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:04.823+0000 7feacffff640 1 -- 172.21.15.33:0/2254796822 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fead009bed0 con 0x7fead80ffaf0 2024-03-19T02:53:04.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:04 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2634985021' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:53:04.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:04 smithi096 ceph-mon[35946]: pgmap v369: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:53:04.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:04 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1278739882' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:53:04.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:04 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2588291678' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:53:05.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.006+0000 7feadd5f4640 1 -- 172.21.15.33:0/2254796822 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fead8062450 con 0x7fead80ffaf0 2024-03-19T02:53:05.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.007+0000 7feacffff640 1 -- 172.21.15.33:0/2254796822 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v97) ==== 104+0+11 (secure 0 0 0) 0x7fead009b7d0 con 0x7fead80ffaf0 2024-03-19T02:53:05.008 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 32 2024-03-19T02:53:05.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.009+0000 7feadd5f4640 1 -- 172.21.15.33:0/2254796822 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7feac40a2800 msgr2=0x7feac40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:05.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.009+0000 7feadd5f4640 1 --2- 172.21.15.33:0/2254796822 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7feac40a2800 0x7feac40a4cc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7feac0009900 tx=0x7feac0005fd0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.009+0000 7feadd5f4640 1 -- 172.21.15.33:0/2254796822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fead80ffaf0 msgr2=0x7fead8102430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:05.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.009+0000 7feadd5f4640 1 --2- 172.21.15.33:0/2254796822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fead80ffaf0 0x7fead8102430 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fead000e9c0 tx=0x7fead000ee80 comp rx=0 tx=0).stop 2024-03-19T02:53:05.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.009+0000 7feadd5f4640 1 -- 172.21.15.33:0/2254796822 shutdown_connections 2024-03-19T02:53:05.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.009+0000 7feadd5f4640 1 --2- 172.21.15.33:0/2254796822 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7feac40a2800 0x7feac40a4cc0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.009+0000 7feadd5f4640 1 --2- 172.21.15.33:0/2254796822 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fead8103480 0x7fead8100fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.009+0000 7feadd5f4640 1 --2- 172.21.15.33:0/2254796822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fead8102ac0 0x7fead8100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.009+0000 7feadd5f4640 1 --2- 172.21.15.33:0/2254796822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fead80ffaf0 0x7fead8102430 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.009+0000 7feadd5f4640 1 -- 172.21.15.33:0/2254796822 >> 172.21.15.33:0/2254796822 conn(0x7fead80fb440 msgr2=0x7fead8105f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:05.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.009+0000 7feadd5f4640 1 -- 172.21.15.33:0/2254796822 shutdown_connections 2024-03-19T02:53:05.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.009+0000 7feadd5f4640 1 -- 172.21.15.33:0/2254796822 wait complete. 2024-03-19T02:53:05.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:04 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2634985021' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:53:05.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:04 smithi155 ceph-mon[35804]: pgmap v369: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:53:05.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:04 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1278739882' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:53:05.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:04 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2588291678' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:53:05.021 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T02:53:05.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.126+0000 7f5e35826640 1 -- 172.21.15.33:0/1759083186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e30101830 msgr2=0x7f5e30101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:05.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.126+0000 7f5e35826640 1 --2- 172.21.15.33:0/1759083186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e30101830 0x7f5e30101c50 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f5e28009900 tx=0x7f5e2802bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:05.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.126+0000 7f5e35826640 1 -- 172.21.15.33:0/1759083186 shutdown_connections 2024-03-19T02:53:05.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.126+0000 7f5e35826640 1 --2- 172.21.15.33:0/1759083186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e30102a20 0x7f5e30102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.126+0000 7f5e35826640 1 --2- 172.21.15.33:0/1759083186 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e301033e0 0x7f5e30109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.126+0000 7f5e35826640 1 --2- 172.21.15.33:0/1759083186 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e30101830 0x7f5e30101c50 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.126+0000 7f5e35826640 1 -- 172.21.15.33:0/1759083186 >> 172.21.15.33:0/1759083186 conn(0x7f5e300fcdc0 msgr2=0x7f5e300ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:05.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.126+0000 7f5e35826640 1 -- 172.21.15.33:0/1759083186 shutdown_connections 2024-03-19T02:53:05.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.126+0000 7f5e35826640 1 -- 172.21.15.33:0/1759083186 wait complete. 2024-03-19T02:53:05.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.126+0000 7f5e35826640 1 Processor -- start 2024-03-19T02:53:05.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.126+0000 7f5e35826640 1 -- start start 2024-03-19T02:53:05.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.127+0000 7f5e35826640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e30101830 0x7f5e30197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:05.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.127+0000 7f5e35826640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e30102a20 0x7f5e30198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:05.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.127+0000 7f5e35826640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e301033e0 0x7f5e3019f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:05.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.127+0000 7f5e35826640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5e3010d350 con 0x7f5e30101830 2024-03-19T02:53:05.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.127+0000 7f5e35826640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5e3010d1d0 con 0x7f5e301033e0 2024-03-19T02:53:05.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.127+0000 7f5e35826640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5e3010d4d0 con 0x7f5e30102a20 2024-03-19T02:53:05.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.127+0000 7f5e2effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e30101830 0x7f5e30197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:05.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.127+0000 7f5e2effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e30101830 0x7f5e30197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46320/0 (socket says 172.21.15.33:46320) 2024-03-19T02:53:05.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.127+0000 7f5e2e7fc640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e30102a20 0x7f5e30198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:05.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.127+0000 7f5e2effd640 1 -- 172.21.15.33:0/4180981133 learned_addr learned my addr 172.21.15.33:0/4180981133 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:05.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.127+0000 7f5e2f7fe640 1 --2- 172.21.15.33:0/4180981133 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e301033e0 0x7f5e3019f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:05.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.127+0000 7f5e2f7fe640 1 -- 172.21.15.33:0/4180981133 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e30102a20 msgr2=0x7f5e30198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:05.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.127+0000 7f5e2f7fe640 1 --2- 172.21.15.33:0/4180981133 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e30102a20 0x7f5e30198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.127+0000 7f5e2f7fe640 1 -- 172.21.15.33:0/4180981133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e30101830 msgr2=0x7f5e30197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:05.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.127+0000 7f5e2f7fe640 1 --2- 172.21.15.33:0/4180981133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e30101830 0x7f5e30197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.127+0000 7f5e2f7fe640 1 -- 172.21.15.33:0/4180981133 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e3019f710 con 0x7f5e301033e0 2024-03-19T02:53:05.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.127+0000 7f5e2e7fc640 1 --2- 172.21.15.33:0/4180981133 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e30102a20 0x7f5e30198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:05.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.127+0000 7f5e2effd640 1 --2- 172.21.15.33:0/4180981133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e30101830 0x7f5e30197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:05.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.127+0000 7f5e2f7fe640 1 --2- 172.21.15.33:0/4180981133 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e301033e0 0x7f5e3019f0a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5e2000ec50 tx=0x7f5e2000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:05.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.127+0000 7f5e34824640 1 -- 172.21.15.33:0/4180981133 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e20003b70 con 0x7f5e301033e0 2024-03-19T02:53:05.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.128+0000 7f5e35826640 1 -- 172.21.15.33:0/4180981133 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e3019fa00 con 0x7f5e301033e0 2024-03-19T02:53:05.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.128+0000 7f5e34824640 1 -- 172.21.15.33:0/4180981133 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e20003d10 con 0x7f5e301033e0 2024-03-19T02:53:05.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.128+0000 7f5e34824640 1 -- 172.21.15.33:0/4180981133 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e20010760 con 0x7f5e301033e0 2024-03-19T02:53:05.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.128+0000 7f5e35826640 1 -- 172.21.15.33:0/4180981133 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e3019ff50 con 0x7f5e301033e0 2024-03-19T02:53:05.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.128+0000 7f5e35826640 1 -- 172.21.15.33:0/4180981133 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e30108c90 con 0x7f5e301033e0 2024-03-19T02:53:05.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.130+0000 7f5e34824640 1 -- 172.21.15.33:0/4180981133 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5e20004b40 con 0x7f5e301033e0 2024-03-19T02:53:05.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.130+0000 7f5e34824640 1 --2- 172.21.15.33:0/4180981133 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5e1c0a2800 0x7f5e1c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:05.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.130+0000 7f5e2effd640 1 --2- 172.21.15.33:0/4180981133 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5e1c0a2800 0x7f5e1c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:05.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.130+0000 7f5e34824640 1 -- 172.21.15.33:0/4180981133 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f5e200cf490 con 0x7f5e301033e0 2024-03-19T02:53:05.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.131+0000 7f5e2effd640 1 --2- 172.21.15.33:0/4180981133 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5e1c0a2800 0x7f5e1c0a4cc0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5e2802c840 tx=0x7f5e280095e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:05.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.138+0000 7f5e34824640 1 -- 172.21.15.33:0/4180981133 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5e2009c230 con 0x7f5e301033e0 2024-03-19T02:53:05.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.324+0000 7f5e35826640 1 -- 172.21.15.33:0/4180981133 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5e3002d3a0 con 0x7f5e301033e0 2024-03-19T02:53:05.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.324+0000 7f5e34824640 1 -- 172.21.15.33:0/4180981133 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v97) ==== 100+0+11 (secure 0 0 0) 0x7f5e2009bb30 con 0x7f5e301033e0 2024-03-19T02:53:05.325 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 64 2024-03-19T02:53:05.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.326+0000 7f5e35826640 1 -- 172.21.15.33:0/4180981133 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5e1c0a2800 msgr2=0x7f5e1c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:05.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.326+0000 7f5e35826640 1 --2- 172.21.15.33:0/4180981133 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5e1c0a2800 0x7f5e1c0a4cc0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5e2802c840 tx=0x7f5e280095e0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.326+0000 7f5e35826640 1 -- 172.21.15.33:0/4180981133 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e301033e0 msgr2=0x7f5e3019f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:05.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.326+0000 7f5e35826640 1 --2- 172.21.15.33:0/4180981133 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e301033e0 0x7f5e3019f0a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5e2000ec50 tx=0x7f5e2000c6a0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.326+0000 7f5e35826640 1 -- 172.21.15.33:0/4180981133 shutdown_connections 2024-03-19T02:53:05.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.326+0000 7f5e35826640 1 --2- 172.21.15.33:0/4180981133 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f5e1c0a2800 0x7f5e1c0a4cc0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.326+0000 7f5e35826640 1 --2- 172.21.15.33:0/4180981133 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e30102a20 0x7f5e30198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.326+0000 7f5e35826640 1 --2- 172.21.15.33:0/4180981133 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e301033e0 0x7f5e3019f0a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.326+0000 7f5e35826640 1 --2- 172.21.15.33:0/4180981133 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e30101830 0x7f5e30197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.326+0000 7f5e35826640 1 -- 172.21.15.33:0/4180981133 >> 172.21.15.33:0/4180981133 conn(0x7f5e300fcdc0 msgr2=0x7f5e30105e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:05.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.326+0000 7f5e35826640 1 -- 172.21.15.33:0/4180981133 shutdown_connections 2024-03-19T02:53:05.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.326+0000 7f5e35826640 1 -- 172.21.15.33:0/4180981133 wait complete. 2024-03-19T02:53:05.338 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T02:53:05.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.442+0000 7f52a6e4d640 1 -- 172.21.15.33:0/3620131260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52a0102a40 msgr2=0x7f52a0102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:05.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.442+0000 7f52a6e4d640 1 --2- 172.21.15.33:0/3620131260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52a0102a40 0x7f52a0102ec0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f528c009900 tx=0x7f528c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:05.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.442+0000 7f52a6e4d640 1 -- 172.21.15.33:0/3620131260 shutdown_connections 2024-03-19T02:53:05.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.442+0000 7f52a6e4d640 1 --2- 172.21.15.33:0/3620131260 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52a0103400 0x7f52a0109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.442+0000 7f52a6e4d640 1 --2- 172.21.15.33:0/3620131260 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52a0102a40 0x7f52a0102ec0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.442+0000 7f52a6e4d640 1 --2- 172.21.15.33:0/3620131260 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f52a0101850 0x7f52a0101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.442+0000 7f52a6e4d640 1 -- 172.21.15.33:0/3620131260 >> 172.21.15.33:0/3620131260 conn(0x7f52a00fcdc0 msgr2=0x7f52a00ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:05.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.443+0000 7f52a6e4d640 1 -- 172.21.15.33:0/3620131260 shutdown_connections 2024-03-19T02:53:05.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.443+0000 7f52a6e4d640 1 -- 172.21.15.33:0/3620131260 wait complete. 2024-03-19T02:53:05.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.443+0000 7f52a6e4d640 1 Processor -- start 2024-03-19T02:53:05.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.443+0000 7f52a6e4d640 1 -- start start 2024-03-19T02:53:05.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.443+0000 7f52a6e4d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52a0101850 0x7f52a0197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:05.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.443+0000 7f52a6e4d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52a0102a40 0x7f52a0197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:05.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.443+0000 7f52a6e4d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f52a0103400 0x7f52a019f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:05.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.443+0000 7f52a6e4d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f52a010d380 con 0x7f52a0101850 2024-03-19T02:53:05.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.443+0000 7f52a6e4d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f52a010d200 con 0x7f52a0103400 2024-03-19T02:53:05.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.443+0000 7f52a6e4d640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f52a010d500 con 0x7f52a0102a40 2024-03-19T02:53:05.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.443+0000 7f52a4be9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52a0101850 0x7f52a0197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:05.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.443+0000 7f52a4be9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52a0101850 0x7f52a0197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46340/0 (socket says 172.21.15.33:46340) 2024-03-19T02:53:05.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.443+0000 7f52a4be9640 1 -- 172.21.15.33:0/3793598099 learned_addr learned my addr 172.21.15.33:0/3793598099 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:05.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.443+0000 7f529ffff640 1 --2- 172.21.15.33:0/3793598099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52a0102a40 0x7f52a0197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:05.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.443+0000 7f52a53ea640 1 --2- 172.21.15.33:0/3793598099 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f52a0103400 0x7f52a019f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:05.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.444+0000 7f52a4be9640 1 -- 172.21.15.33:0/3793598099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52a0102a40 msgr2=0x7f52a0197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:05.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.444+0000 7f52a4be9640 1 --2- 172.21.15.33:0/3793598099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52a0102a40 0x7f52a0197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.444+0000 7f52a4be9640 1 -- 172.21.15.33:0/3793598099 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f52a0103400 msgr2=0x7f52a019f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:05.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.444+0000 7f52a4be9640 1 --2- 172.21.15.33:0/3793598099 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f52a0103400 0x7f52a019f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.444+0000 7f52a4be9640 1 -- 172.21.15.33:0/3793598099 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52a019f6a0 con 0x7f52a0101850 2024-03-19T02:53:05.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.444+0000 7f52a4be9640 1 --2- 172.21.15.33:0/3793598099 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52a0101850 0x7f52a0197a80 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f529400e9c0 tx=0x7f529400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:05.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.444+0000 7f529dffb640 1 -- 172.21.15.33:0/3793598099 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529400cd80 con 0x7f52a0101850 2024-03-19T02:53:05.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.444+0000 7f529dffb640 1 -- 172.21.15.33:0/3793598099 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f52940035f0 con 0x7f52a0101850 2024-03-19T02:53:05.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.444+0000 7f52a6e4d640 1 -- 172.21.15.33:0/3793598099 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52a019f990 con 0x7f52a0101850 2024-03-19T02:53:05.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.444+0000 7f529dffb640 1 -- 172.21.15.33:0/3793598099 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5294010690 con 0x7f52a0101850 2024-03-19T02:53:05.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.444+0000 7f52a6e4d640 1 -- 172.21.15.33:0/3793598099 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52a019ff00 con 0x7f52a0101850 2024-03-19T02:53:05.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.445+0000 7f52a6e4d640 1 -- 172.21.15.33:0/3793598099 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52a0108c10 con 0x7f52a0101850 2024-03-19T02:53:05.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.446+0000 7f529dffb640 1 -- 172.21.15.33:0/3793598099 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f52940037a0 con 0x7f52a0101850 2024-03-19T02:53:05.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.447+0000 7f529dffb640 1 --2- 172.21.15.33:0/3793598099 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f52900a2880 0x7f52900a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:05.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.447+0000 7f529ffff640 1 --2- 172.21.15.33:0/3793598099 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f52900a2880 0x7f52900a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:05.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.447+0000 7f529dffb640 1 -- 172.21.15.33:0/3793598099 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f52940cf100 con 0x7f52a0101850 2024-03-19T02:53:05.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.447+0000 7f529ffff640 1 --2- 172.21.15.33:0/3793598099 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f52900a2880 0x7f52900a4d40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f528c009230 tx=0x7f528c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:05.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.451+0000 7f529dffb640 1 -- 172.21.15.33:0/3793598099 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f529409be10 con 0x7f52a0101850 2024-03-19T02:53:05.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.639+0000 7f52a6e4d640 1 -- 172.21.15.33:0/3793598099 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f52a0062420 con 0x7f52a0101850 2024-03-19T02:53:05.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.640+0000 7f529dffb640 1 -- 172.21.15.33:0/3793598099 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v97) ==== 103+0+11 (secure 0 0 0) 0x7f529409b710 con 0x7f52a0101850 2024-03-19T02:53:05.641 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 32 2024-03-19T02:53:05.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.641+0000 7f52a6e4d640 1 -- 172.21.15.33:0/3793598099 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f52900a2880 msgr2=0x7f52900a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:05.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.641+0000 7f52a6e4d640 1 --2- 172.21.15.33:0/3793598099 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f52900a2880 0x7f52900a4d40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f528c009230 tx=0x7f528c005fd0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.641+0000 7f52a6e4d640 1 -- 172.21.15.33:0/3793598099 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52a0101850 msgr2=0x7f52a0197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:05.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.641+0000 7f52a6e4d640 1 --2- 172.21.15.33:0/3793598099 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52a0101850 0x7f52a0197a80 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f529400e9c0 tx=0x7f529400ee80 comp rx=0 tx=0).stop 2024-03-19T02:53:05.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.641+0000 7f52a6e4d640 1 -- 172.21.15.33:0/3793598099 shutdown_connections 2024-03-19T02:53:05.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.641+0000 7f52a6e4d640 1 --2- 172.21.15.33:0/3793598099 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f52900a2880 0x7f52900a4d40 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.641+0000 7f52a6e4d640 1 --2- 172.21.15.33:0/3793598099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f52a0102a40 0x7f52a0197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.641+0000 7f52a6e4d640 1 --2- 172.21.15.33:0/3793598099 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f52a0103400 0x7f52a019f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.641+0000 7f52a6e4d640 1 --2- 172.21.15.33:0/3793598099 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f52a0101850 0x7f52a0197a80 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.641+0000 7f52a6e4d640 1 -- 172.21.15.33:0/3793598099 >> 172.21.15.33:0/3793598099 conn(0x7f52a00fcdc0 msgr2=0x7f52a0105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:05.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.642+0000 7f52a6e4d640 1 -- 172.21.15.33:0/3793598099 shutdown_connections 2024-03-19T02:53:05.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.642+0000 7f52a6e4d640 1 -- 172.21.15.33:0/3793598099 wait complete. 2024-03-19T02:53:05.652 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:05 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2506717103' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:53:05.652 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:05 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2254796822' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:53:05.652 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:05 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/4180981133' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:53:05.654 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:53:05.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.755+0000 7f33156b2640 1 -- 172.21.15.33:0/1006305537 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3310104c10 msgr2=0x7f3310107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:05.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.755+0000 7f33156b2640 1 --2- 172.21.15.33:0/1006305537 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3310104c10 0x7f3310107000 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f32f8009900 tx=0x7f32f802bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:05.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.755+0000 7f33156b2640 1 -- 172.21.15.33:0/1006305537 shutdown_connections 2024-03-19T02:53:05.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.755+0000 7f33156b2640 1 --2- 172.21.15.33:0/1006305537 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3310107540 0x7f3310109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.755+0000 7f33156b2640 1 --2- 172.21.15.33:0/1006305537 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3310104c10 0x7f3310107000 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.755+0000 7f33156b2640 1 --2- 172.21.15.33:0/1006305537 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33101022e0 0x7f33101046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.755+0000 7f33156b2640 1 -- 172.21.15.33:0/1006305537 >> 172.21.15.33:0/1006305537 conn(0x7f33100fbcb0 msgr2=0x7f33100fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:05.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.755+0000 7f33156b2640 1 -- 172.21.15.33:0/1006305537 shutdown_connections 2024-03-19T02:53:05.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.755+0000 7f33156b2640 1 -- 172.21.15.33:0/1006305537 wait complete. 2024-03-19T02:53:05.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.756+0000 7f33156b2640 1 Processor -- start 2024-03-19T02:53:05.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.756+0000 7f33156b2640 1 -- start start 2024-03-19T02:53:05.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.756+0000 7f33156b2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f33101022e0 0x7f3310197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:05.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.756+0000 7f33156b2640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3310104c10 0x7f3310198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:05.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.756+0000 7f33156b2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3310107540 0x7f331019f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:05.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.756+0000 7f33156b2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f331010d380 con 0x7f33101022e0 2024-03-19T02:53:05.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.756+0000 7f33156b2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f331010d200 con 0x7f3310107540 2024-03-19T02:53:05.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.756+0000 7f33156b2640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f331010d500 con 0x7f3310104c10 2024-03-19T02:53:05.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.756+0000 7f330effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f33101022e0 0x7f3310197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:05.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.756+0000 7f330effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f33101022e0 0x7f3310197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46362/0 (socket says 172.21.15.33:46362) 2024-03-19T02:53:05.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.756+0000 7f330effd640 1 -- 172.21.15.33:0/3722672193 learned_addr learned my addr 172.21.15.33:0/3722672193 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:05.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.756+0000 7f330f7fe640 1 --2- 172.21.15.33:0/3722672193 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3310107540 0x7f331019f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:05.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.756+0000 7f330e7fc640 1 --2- 172.21.15.33:0/3722672193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3310104c10 0x7f3310198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:05.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.757+0000 7f330effd640 1 -- 172.21.15.33:0/3722672193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3310104c10 msgr2=0x7f3310198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:05.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.757+0000 7f330effd640 1 --2- 172.21.15.33:0/3722672193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3310104c10 0x7f3310198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.757+0000 7f330effd640 1 -- 172.21.15.33:0/3722672193 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3310107540 msgr2=0x7f331019f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:05.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.757+0000 7f330effd640 1 --2- 172.21.15.33:0/3722672193 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3310107540 0x7f331019f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.757+0000 7f330effd640 1 -- 172.21.15.33:0/3722672193 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f331019f720 con 0x7f33101022e0 2024-03-19T02:53:05.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.757+0000 7f330effd640 1 --2- 172.21.15.33:0/3722672193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f33101022e0 0x7f3310197b00 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f330800e9f0 tx=0x7f330800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:05.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.757+0000 7f3307fff640 1 -- 172.21.15.33:0/3722672193 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f330800ccc0 con 0x7f33101022e0 2024-03-19T02:53:05.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.757+0000 7f33156b2640 1 -- 172.21.15.33:0/3722672193 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f331019fa10 con 0x7f33101022e0 2024-03-19T02:53:05.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.757+0000 7f3307fff640 1 -- 172.21.15.33:0/3722672193 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f330800ce60 con 0x7f33101022e0 2024-03-19T02:53:05.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.757+0000 7f3307fff640 1 -- 172.21.15.33:0/3722672193 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3308010430 con 0x7f33101022e0 2024-03-19T02:53:05.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.757+0000 7f33156b2640 1 -- 172.21.15.33:0/3722672193 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f331019fee0 con 0x7f33101022e0 2024-03-19T02:53:05.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.758+0000 7f33156b2640 1 -- 172.21.15.33:0/3722672193 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33100fd7a0 con 0x7f33101022e0 2024-03-19T02:53:05.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.759+0000 7f3307fff640 1 -- 172.21.15.33:0/3722672193 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3308005020 con 0x7f33101022e0 2024-03-19T02:53:05.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.760+0000 7f3307fff640 1 --2- 172.21.15.33:0/3722672193 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f32fc0a2800 0x7f32fc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:05.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.760+0000 7f330e7fc640 1 --2- 172.21.15.33:0/3722672193 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f32fc0a2800 0x7f32fc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:05.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.760+0000 7f3307fff640 1 -- 172.21.15.33:0/3722672193 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f33080cf240 con 0x7f33101022e0 2024-03-19T02:53:05.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.760+0000 7f330e7fc640 1 --2- 172.21.15.33:0/3722672193 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f32fc0a2800 0x7f32fc0a4cc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f32f8005ab0 tx=0x7f32f8005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:05.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.764+0000 7f3307fff640 1 -- 172.21.15.33:0/3722672193 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f330809bef0 con 0x7f33101022e0 2024-03-19T02:53:05.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:05 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2506717103' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:53:05.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:05 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2254796822' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:53:05.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:05 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/4180981133' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:53:05.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.985+0000 7f33156b2640 1 -- 172.21.15.33:0/3722672193 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f331002d1f0 con 0x7f33101022e0 2024-03-19T02:53:05.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.986+0000 7f3307fff640 1 -- 172.21.15.33:0/3722672193 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11867 (secure 0 0 0) 0x7f330809b7f0 con 0x7f33101022e0 2024-03-19T02:53:05.987 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:53:05.987 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:50:52.201798+0000","modified":"2024-03-19T02:51:19.076317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24488,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/3731232169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":3731232169},{"type":"v1","addr":"172.21.15.96:6839","nonce":3731232169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24457,24509,24488]},"id":1}]} 2024-03-19T02:53:05.987 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 24 2024-03-19T02:53:05.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.988+0000 7f33156b2640 1 -- 172.21.15.33:0/3722672193 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f32fc0a2800 msgr2=0x7f32fc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:05.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.988+0000 7f33156b2640 1 --2- 172.21.15.33:0/3722672193 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f32fc0a2800 0x7f32fc0a4cc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f32f8005ab0 tx=0x7f32f8005fd0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.988+0000 7f33156b2640 1 -- 172.21.15.33:0/3722672193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f33101022e0 msgr2=0x7f3310197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:05.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.988+0000 7f33156b2640 1 --2- 172.21.15.33:0/3722672193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f33101022e0 0x7f3310197b00 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f330800e9f0 tx=0x7f330800edf0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.988+0000 7f33156b2640 1 -- 172.21.15.33:0/3722672193 shutdown_connections 2024-03-19T02:53:05.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.988+0000 7f33156b2640 1 --2- 172.21.15.33:0/3722672193 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f32fc0a2800 0x7f32fc0a4cc0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.988+0000 7f33156b2640 1 --2- 172.21.15.33:0/3722672193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3310104c10 0x7f3310198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.988+0000 7f33156b2640 1 --2- 172.21.15.33:0/3722672193 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3310107540 0x7f331019f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.988+0000 7f33156b2640 1 --2- 172.21.15.33:0/3722672193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f33101022e0 0x7f3310197b00 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:05.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.988+0000 7f33156b2640 1 -- 172.21.15.33:0/3722672193 >> 172.21.15.33:0/3722672193 conn(0x7f33100fbcb0 msgr2=0x7f3310102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:05.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.988+0000 7f33156b2640 1 -- 172.21.15.33:0/3722672193 shutdown_connections 2024-03-19T02:53:05.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:05.988+0000 7f33156b2640 1 -- 172.21.15.33:0/3722672193 wait complete. 2024-03-19T02:53:06.000 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:53:06.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:05 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2506717103' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:53:06.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:05 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2254796822' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:53:06.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:05 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/4180981133' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:53:06.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.101+0000 7f497ec68640 1 -- 172.21.15.33:0/3390823178 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4978101930 msgr2=0x7f4978101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:06.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.101+0000 7f497ec68640 1 --2- 172.21.15.33:0/3390823178 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4978101930 0x7f4978101db0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f496c009900 tx=0x7f496c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:06.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.101+0000 7f497ec68640 1 -- 172.21.15.33:0/3390823178 shutdown_connections 2024-03-19T02:53:06.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.101+0000 7f497ec68640 1 --2- 172.21.15.33:0/3390823178 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49781022f0 0x7f4978108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.101+0000 7f497ec68640 1 --2- 172.21.15.33:0/3390823178 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4978101930 0x7f4978101db0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.101+0000 7f497ec68640 1 --2- 172.21.15.33:0/3390823178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4978100740 0x7f4978100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.101+0000 7f497ec68640 1 -- 172.21.15.33:0/3390823178 >> 172.21.15.33:0/3390823178 conn(0x7f49780fbcb0 msgr2=0x7f49780fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:06.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.102+0000 7f497ec68640 1 -- 172.21.15.33:0/3390823178 shutdown_connections 2024-03-19T02:53:06.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.102+0000 7f497ec68640 1 -- 172.21.15.33:0/3390823178 wait complete. 2024-03-19T02:53:06.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.102+0000 7f497ec68640 1 Processor -- start 2024-03-19T02:53:06.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.102+0000 7f497ec68640 1 -- start start 2024-03-19T02:53:06.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.102+0000 7f497ec68640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4978100740 0x7f4978197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.102+0000 7f497ec68640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4978101930 0x7f4978198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.102+0000 7f497ec68640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49781022f0 0x7f497819f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.102+0000 7f497ec68640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f497810c270 con 0x7f4978101930 2024-03-19T02:53:06.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.102+0000 7f497ec68640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f497810c0f0 con 0x7f4978100740 2024-03-19T02:53:06.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.102+0000 7f497ec68640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f497810c3f0 con 0x7f49781022f0 2024-03-19T02:53:06.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.102+0000 7f4977fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4978101930 0x7f4978198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.102+0000 7f497d205640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49781022f0 0x7f497819f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.102+0000 7f4977fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4978101930 0x7f4978198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46372/0 (socket says 172.21.15.33:46372) 2024-03-19T02:53:06.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.102+0000 7f4977fff640 1 -- 172.21.15.33:0/3500053136 learned_addr learned my addr 172.21.15.33:0/3500053136 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:06.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.102+0000 7f497ca04640 1 --2- 172.21.15.33:0/3500053136 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4978100740 0x7f4978197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.103+0000 7f4977fff640 1 -- 172.21.15.33:0/3500053136 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49781022f0 msgr2=0x7f497819f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:06.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.103+0000 7f4977fff640 1 --2- 172.21.15.33:0/3500053136 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49781022f0 0x7f497819f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.103+0000 7f4977fff640 1 -- 172.21.15.33:0/3500053136 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4978100740 msgr2=0x7f4978197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:06.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.103+0000 7f4977fff640 1 --2- 172.21.15.33:0/3500053136 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4978100740 0x7f4978197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.103+0000 7f4977fff640 1 -- 172.21.15.33:0/3500053136 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f497819f710 con 0x7f4978101930 2024-03-19T02:53:06.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.103+0000 7f497ca04640 1 --2- 172.21.15.33:0/3500053136 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4978100740 0x7f4978197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:06.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.103+0000 7f497d205640 1 --2- 172.21.15.33:0/3500053136 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49781022f0 0x7f497819f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:06.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.103+0000 7f4977fff640 1 --2- 172.21.15.33:0/3500053136 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4978101930 0x7f4978198030 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f496c006010 tx=0x7f496c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:06.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.103+0000 7f4975ffb640 1 -- 172.21.15.33:0/3500053136 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496c043020 con 0x7f4978101930 2024-03-19T02:53:06.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.103+0000 7f497ec68640 1 -- 172.21.15.33:0/3500053136 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f497819f9a0 con 0x7f4978101930 2024-03-19T02:53:06.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.103+0000 7f4975ffb640 1 -- 172.21.15.33:0/3500053136 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f496c004440 con 0x7f4978101930 2024-03-19T02:53:06.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.103+0000 7f4975ffb640 1 -- 172.21.15.33:0/3500053136 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496c003bf0 con 0x7f4978101930 2024-03-19T02:53:06.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.103+0000 7f497ec68640 1 -- 172.21.15.33:0/3500053136 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f497819fef0 con 0x7f4978101930 2024-03-19T02:53:06.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.104+0000 7f497ec68640 1 -- 172.21.15.33:0/3500053136 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4978068610 con 0x7f4978101930 2024-03-19T02:53:06.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.105+0000 7f4975ffb640 1 -- 172.21.15.33:0/3500053136 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f496c046020 con 0x7f4978101930 2024-03-19T02:53:06.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.106+0000 7f4975ffb640 1 --2- 172.21.15.33:0/3500053136 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f49600a2800 0x7f49600a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.106+0000 7f4975ffb640 1 -- 172.21.15.33:0/3500053136 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f496c0f1100 con 0x7f4978101930 2024-03-19T02:53:06.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.106+0000 7f497ca04640 1 --2- 172.21.15.33:0/3500053136 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f49600a2800 0x7f49600a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.106+0000 7f497ca04640 1 --2- 172.21.15.33:0/3500053136 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f49600a2800 0x7f49600a4cc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f4978101790 tx=0x7f4970009720 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:06.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.110+0000 7f4975ffb640 1 -- 172.21.15.33:0/3500053136 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f496c04d020 con 0x7f4978101930 2024-03-19T02:53:06.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.295+0000 7f497ec68640 1 -- 172.21.15.33:0/3500053136 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f49781a0810 con 0x7f4978101930 2024-03-19T02:53:06.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.296+0000 7f4975ffb640 1 -- 172.21.15.33:0/3500053136 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) ==== 74+0+24663 (secure 0 0 0) 0x7f496c0bde10 con 0x7f4978101930 2024-03-19T02:53:06.298 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:53:06.298 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":97,"fsid":"2b7a8938-e59a-11ee-95c9-87774f69a715","created":"2024-03-19T02:42:37.912179+0000","modified":"2024-03-19T02:52:41.116227+0000","last_up_change":"2024-03-19T02:50:01.539392+0000","last_in_change":"2024-03-19T02:49:40.534415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:46:09.963737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:50:50.069641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:50:51.381064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:50:55.476667+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"090a89ab-6b7b-4317-8823-178d4212b2f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6803","nonce":1633124995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6805","nonce":1633124995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6809","nonce":1633124995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6807","nonce":1633124995}]},"public_addr":"172.21.15.33:6803/1633124995","cluster_addr":"172.21.15.33:6805/1633124995","heartbeat_back_addr":"172.21.15.33:6809/1633124995","heartbeat_front_addr":"172.21.15.33:6807/1633124995","state":["exists","up"]},{"osd":1,"uuid":"f50eec55-1956-4486-8593-f2c8fa610e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6801","nonce":340144398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6803","nonce":340144398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6807","nonce":340144398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6805","nonce":340144398}]},"public_addr":"172.21.15.96:6801/340144398","cluster_addr":"172.21.15.96:6803/340144398","heartbeat_back_addr":"172.21.15.96:6807/340144398","heartbeat_front_addr":"172.21.15.96:6805/340144398","state":["exists","up"]},{"osd":2,"uuid":"893a2d62-80d9-4e5e-bc6a-a1c8c681d652","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6801","nonce":3388743876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6803","nonce":3388743876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6807","nonce":3388743876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6805","nonce":3388743876}]},"public_addr":"172.21.15.155:6801/3388743876","cluster_addr":"172.21.15.155:6803/3388743876","heartbeat_back_addr":"172.21.15.155:6807/3388743876","heartbeat_front_addr":"172.21.15.155:6805/3388743876","state":["exists","up"]},{"osd":3,"uuid":"b07e0482-4281-4d32-9dfe-9995d7246175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6811","nonce":677007171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6813","nonce":677007171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6817","nonce":677007171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6815","nonce":677007171}]},"public_addr":"172.21.15.33:6811/677007171","cluster_addr":"172.21.15.33:6813/677007171","heartbeat_back_addr":"172.21.15.33:6817/677007171","heartbeat_front_addr":"172.21.15.33:6815/677007171","state":["exists","up"]},{"osd":4,"uuid":"a988384b-af6b-47c6-8c07-5c9673acee45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6809","nonce":3106158554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6811","nonce":3106158554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6815","nonce":3106158554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6813","nonce":3106158554}]},"public_addr":"172.21.15.96:6809/3106158554","cluster_addr":"172.21.15.96:6811/3106158554","heartbeat_back_addr":"172.21.15.96:6815/3106158554","heartbeat_front_addr":"172.21.15.96:6813/3106158554","state":["exists","up"]},{"osd":5,"uuid":"198dc992-96f8-4c52-96ee-1fd5de148bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6809","nonce":327293324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6811","nonce":327293324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6815","nonce":327293324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6813","nonce":327293324}]},"public_addr":"172.21.15.155:6809/327293324","cluster_addr":"172.21.15.155:6811/327293324","heartbeat_back_addr":"172.21.15.155:6815/327293324","heartbeat_front_addr":"172.21.15.155:6813/327293324","state":["exists","up"]},{"osd":6,"uuid":"a7278ec0-ab2e-49ed-be53-84d8b9e24177","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6819","nonce":873871509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6821","nonce":873871509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6825","nonce":873871509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6823","nonce":873871509}]},"public_addr":"172.21.15.33:6819/873871509","cluster_addr":"172.21.15.33:6821/873871509","heartbeat_back_addr":"172.21.15.33:6825/873871509","heartbeat_front_addr":"172.21.15.33:6823/873871509","state":["exists","up"]},{"osd":7,"uuid":"727e133d-aa24-4ffc-90e2-266eb2aab8c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6817","nonce":2334123891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6819","nonce":2334123891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6823","nonce":2334123891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6821","nonce":2334123891}]},"public_addr":"172.21.15.96:6817/2334123891","cluster_addr":"172.21.15.96:6819/2334123891","heartbeat_back_addr":"172.21.15.96:6823/2334123891","heartbeat_front_addr":"172.21.15.96:6821/2334123891","state":["exists","up"]},{"osd":8,"uuid":"bb20eec0-39e5-4b86-9db5-486b1840cdad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6817","nonce":3498516903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6819","nonce":3498516903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6823","nonce":3498516903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6821","nonce":3498516903}]},"public_addr":"172.21.15.155:6817/3498516903","cluster_addr":"172.21.15.155:6819/3498516903","heartbeat_back_addr":"172.21.15.155:6823/3498516903","heartbeat_front_addr":"172.21.15.155:6821/3498516903","state":["exists","up"]},{"osd":9,"uuid":"179c6228-3e23-4f65-91a6-1ef0005d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6827","nonce":760777213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6829","nonce":760777213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6833","nonce":760777213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6831","nonce":760777213}]},"public_addr":"172.21.15.33:6827/760777213","cluster_addr":"172.21.15.33:6829/760777213","heartbeat_back_addr":"172.21.15.33:6833/760777213","heartbeat_front_addr":"172.21.15.33:6831/760777213","state":["exists","up"]},{"osd":10,"uuid":"be17b478-3c2e-41fb-bede-958333acdf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6825","nonce":1404898482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6827","nonce":1404898482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6831","nonce":1404898482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6829","nonce":1404898482}]},"public_addr":"172.21.15.96:6825/1404898482","cluster_addr":"172.21.15.96:6827/1404898482","heartbeat_back_addr":"172.21.15.96:6831/1404898482","heartbeat_front_addr":"172.21.15.96:6829/1404898482","state":["exists","up"]},{"osd":11,"uuid":"c0903ed2-cc1e-4c18-8731-1f0ee4dd0457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6825","nonce":792352827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6827","nonce":792352827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6831","nonce":792352827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6829","nonce":792352827}]},"public_addr":"172.21.15.155:6825/792352827","cluster_addr":"172.21.15.155:6827/792352827","heartbeat_back_addr":"172.21.15.155:6831/792352827","heartbeat_front_addr":"172.21.15.155:6829/792352827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:17.942801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:42.109449+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:05.981599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:32.069574+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:57.431015+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:22.448592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:48.526515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:14.498352+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:40.175426+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:06.364006+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:32.853161+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:59.073538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.9","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:6801/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:6800/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/1067768047":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/729379453":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2586588742":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/2932814165":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6801/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2212564963":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6800/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/3275126058":"2024-03-20T02:43:39.622218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:53:06.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.299+0000 7f497ec68640 1 -- 172.21.15.33:0/3500053136 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f49600a2800 msgr2=0x7f49600a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:06.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.299+0000 7f497ec68640 1 --2- 172.21.15.33:0/3500053136 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f49600a2800 0x7f49600a4cc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f4978101790 tx=0x7f4970009720 comp rx=0 tx=0).stop 2024-03-19T02:53:06.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.299+0000 7f497ec68640 1 -- 172.21.15.33:0/3500053136 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4978101930 msgr2=0x7f4978198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:06.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.299+0000 7f497ec68640 1 --2- 172.21.15.33:0/3500053136 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4978101930 0x7f4978198030 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f496c006010 tx=0x7f496c004040 comp rx=0 tx=0).stop 2024-03-19T02:53:06.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.299+0000 7f497ec68640 1 -- 172.21.15.33:0/3500053136 shutdown_connections 2024-03-19T02:53:06.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.299+0000 7f497ec68640 1 --2- 172.21.15.33:0/3500053136 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f49600a2800 0x7f49600a4cc0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.299+0000 7f497ec68640 1 --2- 172.21.15.33:0/3500053136 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49781022f0 0x7f497819f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.299+0000 7f497ec68640 1 --2- 172.21.15.33:0/3500053136 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4978101930 0x7f4978198030 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.299+0000 7f497ec68640 1 --2- 172.21.15.33:0/3500053136 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4978100740 0x7f4978197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.299+0000 7f497ec68640 1 -- 172.21.15.33:0/3500053136 >> 172.21.15.33:0/3500053136 conn(0x7f49780fbcb0 msgr2=0x7f4978068a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:06.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.299+0000 7f497ec68640 1 -- 172.21.15.33:0/3500053136 shutdown_connections 2024-03-19T02:53:06.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.299+0000 7f497ec68640 1 -- 172.21.15.33:0/3500053136 wait complete. 2024-03-19T02:53:06.312 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-19T02:50:52.201798+0000', 'modified': '2024-03-19T02:51:19.076317+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24509, 'mds_1': 24488, 'mds_2': 24457}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24427': {'gid': 24427, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.155:6835/1274854977', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6834', 'nonce': 1274854977}, {'type': 'v1', 'addr': '172.21.15.155:6835', 'nonce': 1274854977}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24436': {'gid': 24436, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.155:6837/1429349330', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6836', 'nonce': 1429349330}, {'type': 'v1', 'addr': '172.21.15.155:6837', 'nonce': 1429349330}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24455': {'gid': 24455, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.96:6835/3551193617', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6834', 'nonce': 3551193617}, {'type': 'v1', 'addr': '172.21.15.96:6835', 'nonce': 3551193617}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24457': {'gid': 24457, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.155:6839/1840534530', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.155:6838', 'nonce': 1840534530}, {'type': 'v1', 'addr': '172.21.15.155:6839', 'nonce': 1840534530}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24488': {'gid': 24488, 'name': 'h', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.96:6837/522588113', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6836', 'nonce': 522588113}, {'type': 'v1', 'addr': '172.21.15.96:6837', 'nonce': 522588113}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24509': {'gid': 24509, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.96:6839/3731232169', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6838', 'nonce': 3731232169}, {'type': 'v1', 'addr': '172.21.15.96:6839', 'nonce': 3731232169}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24509, 'qdb_cluster': [24457, 24509, 24488]} 2024-03-19T02:53:06.312 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-19T02:53:06.312 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-19T02:53:06.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.415+0000 7f6532377640 1 -- 172.21.15.33:0/1132232588 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f652c101830 msgr2=0x7f652c101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:06.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.415+0000 7f6532377640 1 --2- 172.21.15.33:0/1132232588 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f652c101830 0x7f652c101c50 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6514009900 tx=0x7f651402bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:06.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.415+0000 7f6532377640 1 -- 172.21.15.33:0/1132232588 shutdown_connections 2024-03-19T02:53:06.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.415+0000 7f6532377640 1 --2- 172.21.15.33:0/1132232588 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f652c1033e0 0x7f652c109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.415+0000 7f6532377640 1 --2- 172.21.15.33:0/1132232588 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f652c102a20 0x7f652c102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.415+0000 7f6532377640 1 --2- 172.21.15.33:0/1132232588 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f652c101830 0x7f652c101c50 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.415+0000 7f6532377640 1 -- 172.21.15.33:0/1132232588 >> 172.21.15.33:0/1132232588 conn(0x7f652c0fcdc0 msgr2=0x7f652c0ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:06.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.415+0000 7f6532377640 1 -- 172.21.15.33:0/1132232588 shutdown_connections 2024-03-19T02:53:06.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.415+0000 7f6532377640 1 -- 172.21.15.33:0/1132232588 wait complete. 2024-03-19T02:53:06.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.416+0000 7f6532377640 1 Processor -- start 2024-03-19T02:53:06.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.416+0000 7f6532377640 1 -- start start 2024-03-19T02:53:06.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.416+0000 7f6532377640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f652c101830 0x7f652c0755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.416+0000 7f6532377640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f652c102a20 0x7f652c073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.416+0000 7f6532377640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f652c1033e0 0x7f652c074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.416+0000 7f6532377640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f652c10d360 con 0x7f652c102a20 2024-03-19T02:53:06.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.416+0000 7f6532377640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f652c10d1e0 con 0x7f652c101830 2024-03-19T02:53:06.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.416+0000 7f6532377640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f652c10d4e0 con 0x7f652c1033e0 2024-03-19T02:53:06.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.416+0000 7f652b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f652c102a20 0x7f652c073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.416+0000 7f652b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f652c102a20 0x7f652c073c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46388/0 (socket says 172.21.15.33:46388) 2024-03-19T02:53:06.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.416+0000 7f652b7fe640 1 -- 172.21.15.33:0/4030787734 learned_addr learned my addr 172.21.15.33:0/4030787734 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:06.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.416+0000 7f6530914640 1 --2- 172.21.15.33:0/4030787734 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f652c1033e0 0x7f652c074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.416+0000 7f652bfff640 1 --2- 172.21.15.33:0/4030787734 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f652c101830 0x7f652c0755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.417+0000 7f652b7fe640 1 -- 172.21.15.33:0/4030787734 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f652c1033e0 msgr2=0x7f652c074150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:06.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.417+0000 7f652b7fe640 1 --2- 172.21.15.33:0/4030787734 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f652c1033e0 0x7f652c074150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.417+0000 7f652b7fe640 1 -- 172.21.15.33:0/4030787734 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f652c101830 msgr2=0x7f652c0755d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:06.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.417+0000 7f652b7fe640 1 --2- 172.21.15.33:0/4030787734 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f652c101830 0x7f652c0755d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.417+0000 7f652b7fe640 1 -- 172.21.15.33:0/4030787734 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f652c0749e0 con 0x7f652c102a20 2024-03-19T02:53:06.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.417+0000 7f652b7fe640 1 --2- 172.21.15.33:0/4030787734 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f652c102a20 0x7f652c073c10 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f651c00ea60 tx=0x7f651c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:06.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.417+0000 7f65297fa640 1 -- 172.21.15.33:0/4030787734 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f651c00cde0 con 0x7f652c102a20 2024-03-19T02:53:06.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.417+0000 7f65297fa640 1 -- 172.21.15.33:0/4030787734 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f651c0035f0 con 0x7f652c102a20 2024-03-19T02:53:06.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.417+0000 7f6532377640 1 -- 172.21.15.33:0/4030787734 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f652c1a1bc0 con 0x7f652c102a20 2024-03-19T02:53:06.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.417+0000 7f65297fa640 1 -- 172.21.15.33:0/4030787734 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f651c010690 con 0x7f652c102a20 2024-03-19T02:53:06.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.417+0000 7f6532377640 1 -- 172.21.15.33:0/4030787734 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f652c1a2110 con 0x7f652c102a20 2024-03-19T02:53:06.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.419+0000 7f65297fa640 1 -- 172.21.15.33:0/4030787734 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f651c0037a0 con 0x7f652c102a20 2024-03-19T02:53:06.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.419+0000 7f653503a740 1 Processor -- start 2024-03-19T02:53:06.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.419+0000 7f653503a740 1 -- start start 2024-03-19T02:53:06.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.419+0000 7f653503a740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ec9d339ff0 0x55ec9d33a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.420+0000 7f653503a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ec9d33a9e0 0x55ec9d33f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.420+0000 7f653503a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ec9d33fb50 0x55ec9d341fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.420+0000 7f65297fa640 1 --2- 172.21.15.33:0/4030787734 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f65180a2800 0x7f65180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.420+0000 7f653503a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ec9d29ede0 con 0x55ec9d33a9e0 2024-03-19T02:53:06.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.420+0000 7f653503a740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55ec9d247c80 con 0x55ec9d339ff0 2024-03-19T02:53:06.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.420+0000 7f653503a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ec9d29ae80 con 0x55ec9d33fb50 2024-03-19T02:53:06.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.420+0000 7f652bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ec9d33a9e0 0x55ec9d33f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.420+0000 7f6530914640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ec9d339ff0 0x55ec9d33a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.420+0000 7f652bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ec9d33a9e0 0x55ec9d33f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46404/0 (socket says 172.21.15.33:46404) 2024-03-19T02:53:06.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.420+0000 7f652bfff640 1 -- 172.21.15.33:0/1883743642 learned_addr learned my addr 172.21.15.33:0/1883743642 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:06.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.420+0000 7f652b7fe640 1 --2- 172.21.15.33:0/4030787734 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f65180a2800 0x7f65180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.420+0000 7f6530914640 1 --2- 172.21.15.33:0/1883743642 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ec9d33fb50 0x55ec9d341fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.420+0000 7f65297fa640 1 -- 172.21.15.33:0/4030787734 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f651c0cf0f0 con 0x7f652c102a20 2024-03-19T02:53:06.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.420+0000 7f6530914640 1 -- 172.21.15.33:0/1883743642 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ec9d33fb50 msgr2=0x55ec9d341fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:06.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.420+0000 7f6530914640 1 --2- 172.21.15.33:0/1883743642 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ec9d33fb50 0x55ec9d341fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.420+0000 7f6530914640 1 -- 172.21.15.33:0/1883743642 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ec9d33a9e0 msgr2=0x55ec9d33f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:06.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.420+0000 7f6530914640 1 --2- 172.21.15.33:0/1883743642 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ec9d33a9e0 0x55ec9d33f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.420+0000 7f6530914640 1 -- 172.21.15.33:0/1883743642 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec9d259fb0 con 0x55ec9d339ff0 2024-03-19T02:53:06.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.420+0000 7f6530914640 1 --2- 172.21.15.33:0/1883743642 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ec9d339ff0 0x55ec9d33a410 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6524011a70 tx=0x7f6524004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94a16bea90e49549 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:06.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.420+0000 7f652b7fe640 1 --2- 172.21.15.33:0/4030787734 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f65180a2800 0x7f65180a4cc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f651c00d010 tx=0x7f651c090b80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:06.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.420+0000 7f65227fc640 1 -- 172.21.15.33:0/1883743642 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6524004b20 con 0x55ec9d339ff0 2024-03-19T02:53:06.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.420+0000 7f65227fc640 1 -- 172.21.15.33:0/1883743642 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6524004cc0 con 0x55ec9d339ff0 2024-03-19T02:53:06.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.420+0000 7f65227fc640 1 -- 172.21.15.33:0/1883743642 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65240054f0 con 0x55ec9d339ff0 2024-03-19T02:53:06.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.421+0000 7f653503a740 1 -- 172.21.15.33:0/1883743642 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ec9d339ff0 msgr2=0x55ec9d33a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:06.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.421+0000 7f653503a740 1 --2- 172.21.15.33:0/1883743642 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ec9d339ff0 0x55ec9d33a410 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6524011a70 tx=0x7f6524004ab0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.421+0000 7f653503a740 1 -- 172.21.15.33:0/1883743642 shutdown_connections 2024-03-19T02:53:06.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.421+0000 7f653503a740 1 --2- 172.21.15.33:0/1883743642 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ec9d33fb50 0x55ec9d341fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.421+0000 7f653503a740 1 --2- 172.21.15.33:0/1883743642 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ec9d33a9e0 0x55ec9d33f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.421+0000 7f653503a740 1 --2- 172.21.15.33:0/1883743642 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ec9d339ff0 0x55ec9d33a410 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.421+0000 7f653503a740 1 -- 172.21.15.33:0/1883743642 >> 172.21.15.33:0/1883743642 conn(0x55ec9d335560 msgr2=0x55ec9d335960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:06.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.421+0000 7f653503a740 1 -- 172.21.15.33:0/1883743642 shutdown_connections 2024-03-19T02:53:06.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.421+0000 7f653503a740 1 -- 172.21.15.33:0/1883743642 wait complete. 2024-03-19T02:53:06.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.421+0000 7f653503a740 1 -- start start 2024-03-19T02:53:06.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.422+0000 7f653503a740 1 Processor -- start 2024-03-19T02:53:06.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.422+0000 7f653503a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ec9d339ff0 0x55ec9d344570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.422+0000 7f653503a740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ec9d33a9e0 0x55ec9d344ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.422+0000 7f653503a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ec9d33fb50 0x55ec9d34baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.422+0000 7f653503a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ec9d342b00 con 0x55ec9d33fb50 2024-03-19T02:53:06.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.422+0000 7f653503a740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55ec9d25afc0 con 0x55ec9d33a9e0 2024-03-19T02:53:06.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.422+0000 7f653503a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ec9d342c80 con 0x55ec9d339ff0 2024-03-19T02:53:06.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.422+0000 7f653503a740 10 client.0 _ll_register_callbacks cb 0x55ec9d296a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:53:06.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.422+0000 7f6530914640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ec9d33fb50 0x55ec9d34baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.422+0000 7f6530914640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ec9d33fb50 0x55ec9d34baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46420/0 (socket says 172.21.15.33:46420) 2024-03-19T02:53:06.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.422+0000 7f6530914640 1 -- 172.21.15.33:0/2126442839 learned_addr learned my addr 172.21.15.33:0/2126442839 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:06.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.422+0000 7f652bfff640 1 --2- 172.21.15.33:0/2126442839 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ec9d33a9e0 0x55ec9d344ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.422+0000 7f6530914640 1 --2- 172.21.15.33:0/2126442839 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ec9d339ff0 0x55ec9d344570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.422+0000 7f6530914640 1 -- 172.21.15.33:0/2126442839 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ec9d339ff0 msgr2=0x55ec9d344570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:06.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.422+0000 7f6530914640 1 --2- 172.21.15.33:0/2126442839 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ec9d339ff0 0x55ec9d344570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.422+0000 7f6530914640 1 -- 172.21.15.33:0/2126442839 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ec9d33a9e0 msgr2=0x55ec9d344ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:06.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.422+0000 7f6530914640 1 --2- 172.21.15.33:0/2126442839 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ec9d33a9e0 0x55ec9d344ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.422+0000 7f6530914640 1 -- 172.21.15.33:0/2126442839 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec9cfb32d0 con 0x55ec9d33fb50 2024-03-19T02:53:06.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.422+0000 7f6530914640 1 --2- 172.21.15.33:0/2126442839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ec9d33fb50 0x55ec9d34baf0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6524009560 tx=0x7f6524005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14f0c721c3804cf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:06.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.422+0000 7f650b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:53:06.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.422+0000 7f650b7fe640 1 -- 172.21.15.33:0/2126442839 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652400f7c0 con 0x55ec9d33fb50 2024-03-19T02:53:06.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.423+0000 7f650b7fe640 1 -- 172.21.15.33:0/2126442839 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f6524027040 con 0x55ec9d33fb50 2024-03-19T02:53:06.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.423+0000 7f653503a740 1 -- 172.21.15.33:0/2126442839 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6524004cc0 con 0x55ec9d33fb50 2024-03-19T02:53:06.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.423+0000 7f650b7fe640 1 -- 172.21.15.33:0/2126442839 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6524011020 con 0x55ec9d33fb50 2024-03-19T02:53:06.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.423+0000 7f653503a740 10 client.15372 fetch_fsmap learned FSMap version 24 2024-03-19T02:53:06.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.423+0000 7f653503a740 1 -- 172.21.15.33:0/2126442839 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ec9d07d6b0 con 0x55ec9d33fb50 2024-03-19T02:53:06.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.423+0000 7f650b7fe640 1 -- 172.21.15.33:0/2126442839 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6524004cc0 con 0x55ec9d33fb50 2024-03-19T02:53:06.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.423+0000 7f650b7fe640 1 -- 172.21.15.33:0/2126442839 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f652402a020 con 0x55ec9d33fb50 2024-03-19T02:53:06.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.423+0000 7f653503a740 10 client.15372 fetch_fsmap finished waiting for FSMap version 24 2024-03-19T02:53:06.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.423+0000 7f653503a740 10 client.15372 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-19T02:53:06.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.423+0000 7f653503a740 20 client.15372 populate_metadata read hostname 'smithi033' 2024-03-19T02:53:06.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.423+0000 7f653503a740 1 --2- 172.21.15.33:0/2126442839 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55ec9d350f50 0x55ec9d3523c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.423+0000 7f653503a740 4 client.15372 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:53:06.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.423+0000 7f653503a740 1 -- 172.21.15.33:0/2126442839 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ec9d2ac320 con 0x55ec9d350f50 2024-03-19T02:53:06.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.423+0000 7f652bfff640 1 --2- 172.21.15.33:0/2126442839 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55ec9d350f50 0x55ec9d3523c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.424+0000 7f652bfff640 1 --2- 172.21.15.33:0/2126442839 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55ec9d350f50 0x55ec9d3523c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d8c52e618fbebb86 server_cookie=60c4c3f429282c09 in_seq=0 out_seq=0 2024-03-19T02:53:06.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.424+0000 7f650b7fe640 10 client.15372 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T02:53:06.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.424+0000 7f650b7fe640 1 -- 172.21.15.33:0/2126442839 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ec9d2ac320 con 0x55ec9d350f50 2024-03-19T02:53:06.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.424+0000 7f650b7fe640 10 client.15372 handle_command_reply: tid=0 2024-03-19T02:53:06.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.425+0000 7f653503a740 1 client.15372 shutdown 2024-03-19T02:53:06.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.425+0000 7f653503a740 1 -- 172.21.15.33:0/2126442839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ec9d33fb50 msgr2=0x55ec9d34baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:06.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.425+0000 7f653503a740 1 --2- 172.21.15.33:0/2126442839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ec9d33fb50 0x55ec9d34baf0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6524009560 tx=0x7f6524005020 comp rx=0 tx=0).stop 2024-03-19T02:53:06.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.425+0000 7f653503a740 1 -- 172.21.15.33:0/2126442839 shutdown_connections 2024-03-19T02:53:06.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.425+0000 7f653503a740 1 --2- 172.21.15.33:0/2126442839 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55ec9d350f50 0x55ec9d3523c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.425+0000 7f653503a740 1 --2- 172.21.15.33:0/2126442839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ec9d33fb50 0x55ec9d34baf0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.425+0000 7f653503a740 1 --2- 172.21.15.33:0/2126442839 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ec9d33a9e0 0x55ec9d344ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.425+0000 7f650b7fe640 0 client.15372 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T02:53:06.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.425+0000 7f653503a740 1 --2- 172.21.15.33:0/2126442839 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ec9d339ff0 0x55ec9d344570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.425+0000 7f653503a740 1 -- 172.21.15.33:0/2126442839 >> 172.21.15.33:0/2126442839 conn(0x55ec9d335560 msgr2=0x55ec9d3347e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:06.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.425+0000 7f653503a740 1 -- 172.21.15.33:0/2126442839 shutdown_connections 2024-03-19T02:53:06.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.425+0000 7f653503a740 1 -- 172.21.15.33:0/2126442839 wait complete. 2024-03-19T02:53:06.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.425+0000 7f653503a740 20 client.15372 trim_cache size 0 max 16384 2024-03-19T02:53:06.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.443+0000 7f653503a740 1 Processor -- start 2024-03-19T02:53:06.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.443+0000 7f653503a740 1 -- start start 2024-03-19T02:53:06.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.443+0000 7f653503a740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ec9d339ff0 0x55ec9d33bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.443+0000 7f653503a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ec9d33a9e0 0x55ec9d34ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.443+0000 7f653503a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ec9d33fb50 0x55ec9d344180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.443+0000 7f653503a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ec9d350700 con 0x55ec9d33a9e0 2024-03-19T02:53:06.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.443+0000 7f653503a740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55ec9d350dd0 con 0x55ec9d339ff0 2024-03-19T02:53:06.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.443+0000 7f653503a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ec9d3514a0 con 0x55ec9d33fb50 2024-03-19T02:53:06.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.443+0000 7f652bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ec9d33a9e0 0x55ec9d34ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.444+0000 7f6530914640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ec9d339ff0 0x55ec9d33bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.444+0000 7f6530914640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ec9d339ff0 0x55ec9d33bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:36582/0 (socket says 172.21.15.33:36582) 2024-03-19T02:53:06.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.444+0000 7f652bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ec9d33a9e0 0x55ec9d34ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46434/0 (socket says 172.21.15.33:46434) 2024-03-19T02:53:06.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.444+0000 7f6530914640 1 -- 172.21.15.33:0/1961601034 learned_addr learned my addr 172.21.15.33:0/1961601034 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:06.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.444+0000 7f6530914640 1 --2- 172.21.15.33:0/1961601034 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ec9d33fb50 0x55ec9d344180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.444+0000 7f6530914640 1 -- 172.21.15.33:0/1961601034 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ec9d33fb50 msgr2=0x55ec9d344180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:06.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.444+0000 7f6530914640 1 --2- 172.21.15.33:0/1961601034 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ec9d33fb50 0x55ec9d344180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.444+0000 7f6530914640 1 -- 172.21.15.33:0/1961601034 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ec9d33a9e0 msgr2=0x55ec9d34ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:06.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.444+0000 7f6530914640 1 --2- 172.21.15.33:0/1961601034 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ec9d33a9e0 0x55ec9d34ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.444+0000 7f6530914640 1 -- 172.21.15.33:0/1961601034 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec9d33adc0 con 0x55ec9d339ff0 2024-03-19T02:53:06.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.444+0000 7f6530914640 1 --2- 172.21.15.33:0/1961601034 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ec9d339ff0 0x55ec9d33bea0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6524004f90 tx=0x7f6524009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a8c8227933832cd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:06.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.444+0000 7f6521ffb640 1 -- 172.21.15.33:0/1961601034 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65240111c0 con 0x55ec9d339ff0 2024-03-19T02:53:06.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.444+0000 7f6521ffb640 1 -- 172.21.15.33:0/1961601034 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6524005540 con 0x55ec9d339ff0 2024-03-19T02:53:06.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.444+0000 7f6521ffb640 1 -- 172.21.15.33:0/1961601034 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652400f790 con 0x55ec9d339ff0 2024-03-19T02:53:06.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.445+0000 7f653503a740 1 -- 172.21.15.33:0/1961601034 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ec9d339ff0 msgr2=0x55ec9d33bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:06.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.445+0000 7f653503a740 1 --2- 172.21.15.33:0/1961601034 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ec9d339ff0 0x55ec9d33bea0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6524004f90 tx=0x7f6524009f90 comp rx=0 tx=0).stop 2024-03-19T02:53:06.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.445+0000 7f653503a740 1 -- 172.21.15.33:0/1961601034 shutdown_connections 2024-03-19T02:53:06.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.445+0000 7f653503a740 1 --2- 172.21.15.33:0/1961601034 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ec9d33fb50 0x55ec9d344180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.445+0000 7f653503a740 1 --2- 172.21.15.33:0/1961601034 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ec9d33a9e0 0x55ec9d34ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.445+0000 7f653503a740 1 --2- 172.21.15.33:0/1961601034 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ec9d339ff0 0x55ec9d33bea0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.445+0000 7f653503a740 1 -- 172.21.15.33:0/1961601034 >> 172.21.15.33:0/1961601034 conn(0x55ec9d335560 msgr2=0x55ec9d335940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:06.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.445+0000 7f653503a740 1 -- 172.21.15.33:0/1961601034 shutdown_connections 2024-03-19T02:53:06.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.445+0000 7f653503a740 1 -- 172.21.15.33:0/1961601034 wait complete. 2024-03-19T02:53:06.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.445+0000 7f653503a740 1 -- start start 2024-03-19T02:53:06.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.445+0000 7f653503a740 1 Processor -- start 2024-03-19T02:53:06.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.445+0000 7f653503a740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ec9d339ff0 0x55ec9d347660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.445+0000 7f653503a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ec9d33a9e0 0x55ec9d347ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.445+0000 7f653503a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ec9d33fb50 0x55ec9d34abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.445+0000 7f653503a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ec9d352240 con 0x55ec9d33a9e0 2024-03-19T02:53:06.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.445+0000 7f653503a740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55ec9d351b70 con 0x55ec9d339ff0 2024-03-19T02:53:06.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.445+0000 7f653503a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ec9d352910 con 0x55ec9d33fb50 2024-03-19T02:53:06.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.446+0000 7f653503a740 10 client.0 _ll_register_callbacks cb 0x7f65240093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:53:06.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.446+0000 7f652bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ec9d33a9e0 0x55ec9d347ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.446+0000 7f6530914640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ec9d339ff0 0x55ec9d347660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.446+0000 7f652bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ec9d33a9e0 0x55ec9d347ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46448/0 (socket says 172.21.15.33:46448) 2024-03-19T02:53:06.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.446+0000 7f652bfff640 1 -- 172.21.15.33:0/3726326120 learned_addr learned my addr 172.21.15.33:0/3726326120 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:06.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.446+0000 7f6530914640 1 --2- 172.21.15.33:0/3726326120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ec9d33fb50 0x55ec9d34abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.446+0000 7f652bfff640 1 -- 172.21.15.33:0/3726326120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ec9d33fb50 msgr2=0x55ec9d34abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:06.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.446+0000 7f652bfff640 1 --2- 172.21.15.33:0/3726326120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ec9d33fb50 0x55ec9d34abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.446+0000 7f652bfff640 1 -- 172.21.15.33:0/3726326120 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ec9d339ff0 msgr2=0x55ec9d347660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:06.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.446+0000 7f652bfff640 1 --2- 172.21.15.33:0/3726326120 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ec9d339ff0 0x55ec9d347660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.446+0000 7f652bfff640 1 -- 172.21.15.33:0/3726326120 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec9d34b1b0 con 0x55ec9d33a9e0 2024-03-19T02:53:06.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.446+0000 7f652bfff640 1 --2- 172.21.15.33:0/3726326120 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ec9d33a9e0 0x55ec9d347ba0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6514005ab0 tx=0x7f651402cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cabc7dff0f02848 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:06.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.446+0000 7f650b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:53:06.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.446+0000 7f650b7fe640 1 -- 172.21.15.33:0/3726326120 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f651404bd80 con 0x55ec9d33a9e0 2024-03-19T02:53:06.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.446+0000 7f650b7fe640 1 -- 172.21.15.33:0/3726326120 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f6514004040 con 0x55ec9d33a9e0 2024-03-19T02:53:06.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.446+0000 7f653503a740 1 -- 172.21.15.33:0/3726326120 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6524005540 con 0x55ec9d33a9e0 2024-03-19T02:53:06.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.446+0000 7f650b7fe640 1 -- 172.21.15.33:0/3726326120 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6514059050 con 0x55ec9d33a9e0 2024-03-19T02:53:06.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.446+0000 7f653503a740 10 client.15378 fetch_fsmap learned FSMap version 24 2024-03-19T02:53:06.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.446+0000 7f653503a740 1 -- 172.21.15.33:0/3726326120 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ec9d3b72a0 con 0x55ec9d33a9e0 2024-03-19T02:53:06.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.447+0000 7f650b7fe640 1 -- 172.21.15.33:0/3726326120 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6524005540 con 0x55ec9d33a9e0 2024-03-19T02:53:06.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.447+0000 7f650b7fe640 1 -- 172.21.15.33:0/3726326120 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f6514005200 con 0x55ec9d33a9e0 2024-03-19T02:53:06.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.447+0000 7f653503a740 10 client.15378 fetch_fsmap finished waiting for FSMap version 24 2024-03-19T02:53:06.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.447+0000 7f653503a740 10 client.15378 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-19T02:53:06.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.447+0000 7f653503a740 20 client.15378 populate_metadata read hostname 'smithi033' 2024-03-19T02:53:06.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.447+0000 7f653503a740 1 --2- 172.21.15.33:0/3726326120 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55ec9d3bbef0 0x55ec9d3bac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.447+0000 7f653503a740 4 client.15378 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T02:53:06.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.447+0000 7f653503a740 1 -- 172.21.15.33:0/3726326120 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "status"}) -- 0x55ec9d2a6670 con 0x55ec9d3bbef0 2024-03-19T02:53:06.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.447+0000 7f6530914640 1 --2- 172.21.15.33:0/3726326120 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55ec9d3bbef0 0x55ec9d3bac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.447+0000 7f650b7fe640 10 client.15378 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T02:53:06.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.447+0000 7f6530914640 1 --2- 172.21.15.33:0/3726326120 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55ec9d3bbef0 0x55ec9d3bac00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1253815f8757988e server_cookie=c8c7a8e23525d5c4 in_seq=0 out_seq=0 2024-03-19T02:53:06.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.447+0000 7f650b7fe640 1 -- 172.21.15.33:0/3726326120 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55ec9d2a6670 con 0x55ec9d3bbef0 2024-03-19T02:53:06.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.447+0000 7f650b7fe640 10 client.15378 handle_command_reply: tid=0 2024-03-19T02:53:06.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.447+0000 7f653503a740 1 client.15378 shutdown 2024-03-19T02:53:06.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.448+0000 7f653503a740 1 -- 172.21.15.33:0/3726326120 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ec9d33a9e0 msgr2=0x55ec9d347ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:06.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.448+0000 7f653503a740 1 --2- 172.21.15.33:0/3726326120 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ec9d33a9e0 0x55ec9d347ba0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6514005ab0 tx=0x7f651402cdd0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.448+0000 7f653503a740 1 -- 172.21.15.33:0/3726326120 shutdown_connections 2024-03-19T02:53:06.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.448+0000 7f653503a740 1 --2- 172.21.15.33:0/3726326120 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55ec9d3bbef0 0x55ec9d3bac00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.448+0000 7f653503a740 1 --2- 172.21.15.33:0/3726326120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ec9d33fb50 0x55ec9d34abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.448+0000 7f653503a740 1 --2- 172.21.15.33:0/3726326120 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ec9d33a9e0 0x55ec9d347ba0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.448+0000 7f653503a740 1 --2- 172.21.15.33:0/3726326120 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ec9d339ff0 0x55ec9d347660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.448+0000 7f650b7fe640 0 client.15378 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T02:53:06.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.448+0000 7f653503a740 1 -- 172.21.15.33:0/3726326120 >> 172.21.15.33:0/3726326120 conn(0x55ec9d335560 msgr2=0x55ec9d330670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:06.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.448+0000 7f653503a740 1 -- 172.21.15.33:0/3726326120 shutdown_connections 2024-03-19T02:53:06.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.448+0000 7f653503a740 1 -- 172.21.15.33:0/3726326120 wait complete. 2024-03-19T02:53:06.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.448+0000 7f653503a740 20 client.15378 trim_cache size 0 max 16384 2024-03-19T02:53:06.457 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:53:06.457 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_fsid": "2b7a8938-e59a-11ee-95c9-87774f69a715", 2024-03-19T02:53:06.457 INFO:teuthology.orchestra.run.smithi033.stdout: "whoami": 2, 2024-03-19T02:53:06.457 INFO:teuthology.orchestra.run.smithi033.stdout: "id": 24457, 2024-03-19T02:53:06.457 INFO:teuthology.orchestra.run.smithi033.stdout: "want_state": "up:active", 2024-03-19T02:53:06.457 INFO:teuthology.orchestra.run.smithi033.stdout: "state": "up:active", 2024-03-19T02:53:06.457 INFO:teuthology.orchestra.run.smithi033.stdout: "fs_name": "cephfs", 2024-03-19T02:53:06.457 INFO:teuthology.orchestra.run.smithi033.stdout: "rank_uptime": 108.376667488, 2024-03-19T02:53:06.457 INFO:teuthology.orchestra.run.smithi033.stdout: "mdsmap_epoch": 24, 2024-03-19T02:53:06.457 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch": 97, 2024-03-19T02:53:06.457 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch_barrier": 93, 2024-03-19T02:53:06.458 INFO:teuthology.orchestra.run.smithi033.stdout: "uptime": 147.47548718900001 2024-03-19T02:53:06.458 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:53:06.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.448+0000 7f6532377640 1 -- 172.21.15.33:0/4030787734 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f65180a2800 msgr2=0x7f65180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:06.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.448+0000 7f6532377640 1 --2- 172.21.15.33:0/4030787734 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f65180a2800 0x7f65180a4cc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f651c00d010 tx=0x7f651c090b80 comp rx=0 tx=0).stop 2024-03-19T02:53:06.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.448+0000 7f6532377640 1 -- 172.21.15.33:0/4030787734 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f652c102a20 msgr2=0x7f652c073c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:06.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.448+0000 7f6532377640 1 --2- 172.21.15.33:0/4030787734 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f652c102a20 0x7f652c073c10 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f651c00ea60 tx=0x7f651c00ef20 comp rx=0 tx=0).stop 2024-03-19T02:53:06.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.448+0000 7f6532377640 1 -- 172.21.15.33:0/4030787734 shutdown_connections 2024-03-19T02:53:06.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.448+0000 7f6532377640 1 --2- 172.21.15.33:0/4030787734 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f65180a2800 0x7f65180a4cc0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.448+0000 7f6532377640 1 --2- 172.21.15.33:0/4030787734 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f652c1033e0 0x7f652c074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.448+0000 7f6532377640 1 --2- 172.21.15.33:0/4030787734 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f652c102a20 0x7f652c073c10 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.448+0000 7f6532377640 1 --2- 172.21.15.33:0/4030787734 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f652c101830 0x7f652c0755d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.448+0000 7f6532377640 1 -- 172.21.15.33:0/4030787734 >> 172.21.15.33:0/4030787734 conn(0x7f652c0fcdc0 msgr2=0x7f652c105e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:06.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.448+0000 7f6532377640 1 -- 172.21.15.33:0/4030787734 shutdown_connections 2024-03-19T02:53:06.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.448+0000 7f6532377640 1 -- 172.21.15.33:0/4030787734 wait complete. 2024-03-19T02:53:06.460 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-19T02:53:06.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.563+0000 7ff026c80640 1 -- 172.21.15.33:0/137616585 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff0201022e0 msgr2=0x7ff020108b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:06.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.563+0000 7ff026c80640 1 --2- 172.21.15.33:0/137616585 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff0201022e0 0x7ff020108b70 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff018009ca0 tx=0x7ff01802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.563+0000 7ff026c80640 1 -- 172.21.15.33:0/137616585 shutdown_connections 2024-03-19T02:53:06.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.563+0000 7ff026c80640 1 --2- 172.21.15.33:0/137616585 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff0201022e0 0x7ff020108b70 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.563+0000 7ff026c80640 1 --2- 172.21.15.33:0/137616585 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff020101920 0x7ff020101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.563+0000 7ff026c80640 1 --2- 172.21.15.33:0/137616585 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff020100730 0x7ff020100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.563+0000 7ff026c80640 1 -- 172.21.15.33:0/137616585 >> 172.21.15.33:0/137616585 conn(0x7ff0200fbcc0 msgr2=0x7ff0200fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:06.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.563+0000 7ff026c80640 1 -- 172.21.15.33:0/137616585 shutdown_connections 2024-03-19T02:53:06.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.563+0000 7ff026c80640 1 -- 172.21.15.33:0/137616585 wait complete. 2024-03-19T02:53:06.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.564+0000 7ff026c80640 1 Processor -- start 2024-03-19T02:53:06.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.564+0000 7ff026c80640 1 -- start start 2024-03-19T02:53:06.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.564+0000 7ff026c80640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff020100730 0x7ff0201046c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.564+0000 7ff026c80640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff020101920 0x7ff020102d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.564+0000 7ff026c80640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0201022e0 0x7ff020103240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.564+0000 7ff026c80640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff02010c240 con 0x7ff020100730 2024-03-19T02:53:06.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.564+0000 7ff026c80640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff02010c0c0 con 0x7ff020101920 2024-03-19T02:53:06.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.564+0000 7ff026c80640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff02010c3c0 con 0x7ff0201022e0 2024-03-19T02:53:06.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.564+0000 7ff024a1c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff020100730 0x7ff0201046c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.564+0000 7ff024a1c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff020100730 0x7ff0201046c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46474/0 (socket says 172.21.15.33:46474) 2024-03-19T02:53:06.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.564+0000 7ff024a1c640 1 -- 172.21.15.33:0/1604197114 learned_addr learned my addr 172.21.15.33:0/1604197114 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:06.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.564+0000 7ff01ffff640 1 --2- 172.21.15.33:0/1604197114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff020101920 0x7ff020102d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.564+0000 7ff02521d640 1 --2- 172.21.15.33:0/1604197114 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0201022e0 0x7ff020103240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.564+0000 7ff01ffff640 1 -- 172.21.15.33:0/1604197114 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0201022e0 msgr2=0x7ff020103240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:06.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.564+0000 7ff01ffff640 1 --2- 172.21.15.33:0/1604197114 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0201022e0 0x7ff020103240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.564+0000 7ff01ffff640 1 -- 172.21.15.33:0/1604197114 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff020100730 msgr2=0x7ff0201046c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:06.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.564+0000 7ff01ffff640 1 --2- 172.21.15.33:0/1604197114 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff020100730 0x7ff0201046c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.564+0000 7ff01ffff640 1 -- 172.21.15.33:0/1604197114 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff020103ad0 con 0x7ff020101920 2024-03-19T02:53:06.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.564+0000 7ff024a1c640 1 --2- 172.21.15.33:0/1604197114 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff020100730 0x7ff0201046c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:06.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.565+0000 7ff01ffff640 1 --2- 172.21.15.33:0/1604197114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff020101920 0x7ff020102d00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff00c00e9f0 tx=0x7ff00c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:06.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.565+0000 7ff01dffb640 1 -- 172.21.15.33:0/1604197114 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff00c00cd80 con 0x7ff020101920 2024-03-19T02:53:06.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.565+0000 7ff01dffb640 1 -- 172.21.15.33:0/1604197114 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff00c0035f0 con 0x7ff020101920 2024-03-19T02:53:06.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.565+0000 7ff026c80640 1 -- 172.21.15.33:0/1604197114 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0201a1b30 con 0x7ff020101920 2024-03-19T02:53:06.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.565+0000 7ff01dffb640 1 -- 172.21.15.33:0/1604197114 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff00c010670 con 0x7ff020101920 2024-03-19T02:53:06.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.565+0000 7ff026c80640 1 -- 172.21.15.33:0/1604197114 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0201a2080 con 0x7ff020101920 2024-03-19T02:53:06.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.566+0000 7ff01dffb640 1 -- 172.21.15.33:0/1604197114 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff00c0037a0 con 0x7ff020101920 2024-03-19T02:53:06.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.567+0000 7ff01dffb640 1 --2- 172.21.15.33:0/1604197114 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff0100a2800 0x7ff0100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.567+0000 7ff024a1c640 1 --2- 172.21.15.33:0/1604197114 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff0100a2800 0x7ff0100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.567+0000 7ff01dffb640 1 -- 172.21.15.33:0/1604197114 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7ff00c0cf0f0 con 0x7ff020101920 2024-03-19T02:53:06.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.567+0000 7ff02985f740 1 Processor -- start 2024-03-19T02:53:06.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.567+0000 7ff02985f740 1 -- start start 2024-03-19T02:53:06.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.567+0000 7ff02985f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778244bff0 0x55778244c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.567+0000 7ff02985f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55778244c9e0 0x5577824514e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.568+0000 7ff02985f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557782451b50 0x557782453fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.568+0000 7ff02985f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5577823b0de0 con 0x55778244c9e0 2024-03-19T02:53:06.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.568+0000 7ff02985f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557782359c80 con 0x557782451b50 2024-03-19T02:53:06.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.568+0000 7ff02985f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5577823ace80 con 0x55778244bff0 2024-03-19T02:53:06.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.568+0000 7ff024a1c640 1 --2- 172.21.15.33:0/1604197114 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff0100a2800 0x7ff0100a4cc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff0140058c0 tx=0x7ff014005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:06.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.568+0000 7ff01ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55778244c9e0 0x5577824514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.568+0000 7ff024a1c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778244bff0 0x55778244c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.568+0000 7ff01ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55778244c9e0 0x5577824514e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46476/0 (socket says 172.21.15.33:46476) 2024-03-19T02:53:06.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.568+0000 7ff024a1c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778244bff0 0x55778244c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:58470/0 (socket says 172.21.15.33:58470) 2024-03-19T02:53:06.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.568+0000 7ff01ffff640 1 -- 172.21.15.33:0/903385068 learned_addr learned my addr 172.21.15.33:0/903385068 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:06.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.568+0000 7ff02521d640 1 --2- 172.21.15.33:0/903385068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557782451b50 0x557782453fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.568+0000 7ff01ffff640 1 -- 172.21.15.33:0/903385068 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778244bff0 msgr2=0x55778244c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:06.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.568+0000 7ff01ffff640 1 --2- 172.21.15.33:0/903385068 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778244bff0 0x55778244c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.568+0000 7ff01ffff640 1 -- 172.21.15.33:0/903385068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557782451b50 msgr2=0x557782453fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:06.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.568+0000 7ff01ffff640 1 --2- 172.21.15.33:0/903385068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557782451b50 0x557782453fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.568+0000 7ff01ffff640 1 -- 172.21.15.33:0/903385068 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55778236bfb0 con 0x55778244c9e0 2024-03-19T02:53:06.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.568+0000 7ff024a1c640 1 --2- 172.21.15.33:0/903385068 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778244bff0 0x55778244c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:06.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.568+0000 7ff01ffff640 1 --2- 172.21.15.33:0/903385068 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55778244c9e0 0x5577824514e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff00c09b630 tx=0x7ff00c003b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8335b43c68c31f79 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:06.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.568+0000 7ff00affd640 1 -- 172.21.15.33:0/903385068 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff00c0b4070 con 0x55778244c9e0 2024-03-19T02:53:06.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.568+0000 7ff00affd640 1 -- 172.21.15.33:0/903385068 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff00c09b6a0 con 0x55778244c9e0 2024-03-19T02:53:06.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.568+0000 7ff00affd640 1 -- 172.21.15.33:0/903385068 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff00c009430 con 0x55778244c9e0 2024-03-19T02:53:06.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.569+0000 7ff02985f740 1 -- 172.21.15.33:0/903385068 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55778244c9e0 msgr2=0x5577824514e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:06.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.569+0000 7ff02985f740 1 --2- 172.21.15.33:0/903385068 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55778244c9e0 0x5577824514e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff00c09b630 tx=0x7ff00c003b30 comp rx=0 tx=0).stop 2024-03-19T02:53:06.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.569+0000 7ff02985f740 1 -- 172.21.15.33:0/903385068 shutdown_connections 2024-03-19T02:53:06.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.569+0000 7ff02985f740 1 --2- 172.21.15.33:0/903385068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557782451b50 0x557782453fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.569+0000 7ff02985f740 1 --2- 172.21.15.33:0/903385068 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55778244c9e0 0x5577824514e0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.569+0000 7ff02985f740 1 --2- 172.21.15.33:0/903385068 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778244bff0 0x55778244c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.569+0000 7ff02985f740 1 -- 172.21.15.33:0/903385068 >> 172.21.15.33:0/903385068 conn(0x557782447560 msgr2=0x557782447960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:06.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.569+0000 7ff02985f740 1 -- 172.21.15.33:0/903385068 shutdown_connections 2024-03-19T02:53:06.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.569+0000 7ff02985f740 1 -- 172.21.15.33:0/903385068 wait complete. 2024-03-19T02:53:06.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.569+0000 7ff02985f740 1 -- start start 2024-03-19T02:53:06.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.569+0000 7ff02985f740 1 Processor -- start 2024-03-19T02:53:06.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.570+0000 7ff02985f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55778244bff0 0x5577824564e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.570+0000 7ff02985f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778244c9e0 0x557782456a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.570+0000 7ff02985f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557782451b50 0x55778245da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.570+0000 7ff02985f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557782454b00 con 0x557782451b50 2024-03-19T02:53:06.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.570+0000 7ff02985f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55778236cfc0 con 0x55778244bff0 2024-03-19T02:53:06.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.570+0000 7ff02985f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x557782454c80 con 0x55778244c9e0 2024-03-19T02:53:06.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.570+0000 7ff02985f740 10 client.0 _ll_register_callbacks cb 0x5577823a8a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:53:06.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.570+0000 7ff02521d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557782451b50 0x55778245da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.570+0000 7ff02521d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557782451b50 0x55778245da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46482/0 (socket says 172.21.15.33:46482) 2024-03-19T02:53:06.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.570+0000 7ff02521d640 1 -- 172.21.15.33:0/3375573944 learned_addr learned my addr 172.21.15.33:0/3375573944 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:06.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.570+0000 7ff024a1c640 1 --2- 172.21.15.33:0/3375573944 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55778244bff0 0x5577824564e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.570+0000 7ff01ffff640 1 --2- 172.21.15.33:0/3375573944 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778244c9e0 0x557782456a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.570+0000 7ff02521d640 1 -- 172.21.15.33:0/3375573944 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778244c9e0 msgr2=0x557782456a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:06.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.570+0000 7ff02521d640 1 --2- 172.21.15.33:0/3375573944 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778244c9e0 0x557782456a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.570+0000 7ff02521d640 1 -- 172.21.15.33:0/3375573944 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55778244bff0 msgr2=0x5577824564e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:06.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.570+0000 7ff02521d640 1 --2- 172.21.15.33:0/3375573944 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55778244bff0 0x5577824564e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.570+0000 7ff02521d640 1 -- 172.21.15.33:0/3375573944 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577820c52d0 con 0x557782451b50 2024-03-19T02:53:06.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.570+0000 7ff02521d640 1 --2- 172.21.15.33:0/3375573944 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557782451b50 0x55778245da60 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff018032380 tx=0x7ff018009960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33cd181feca211af server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:06.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.570+0000 7feffffff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:53:06.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.570+0000 7feffffff640 1 -- 172.21.15.33:0/3375573944 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff018044070 con 0x557782451b50 2024-03-19T02:53:06.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.570+0000 7feffffff640 1 -- 172.21.15.33:0/3375573944 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7ff018003b70 con 0x557782451b50 2024-03-19T02:53:06.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.570+0000 7ff02985f740 1 -- 172.21.15.33:0/3375573944 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff00c09b6a0 con 0x557782451b50 2024-03-19T02:53:06.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.571+0000 7feffffff640 1 -- 172.21.15.33:0/3375573944 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff018051050 con 0x557782451b50 2024-03-19T02:53:06.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.571+0000 7ff02985f740 10 client.15390 fetch_fsmap learned FSMap version 24 2024-03-19T02:53:06.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.571+0000 7ff02985f740 1 -- 172.21.15.33:0/3375573944 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55778218f6b0 con 0x557782451b50 2024-03-19T02:53:06.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.571+0000 7feffffff640 1 -- 172.21.15.33:0/3375573944 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff00c09b6a0 con 0x557782451b50 2024-03-19T02:53:06.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.571+0000 7feffffff640 1 -- 172.21.15.33:0/3375573944 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff018034440 con 0x557782451b50 2024-03-19T02:53:06.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.571+0000 7ff02985f740 10 client.15390 fetch_fsmap finished waiting for FSMap version 24 2024-03-19T02:53:06.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.571+0000 7ff02985f740 10 client.15390 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-19T02:53:06.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.571+0000 7ff02985f740 20 client.15390 populate_metadata read hostname 'smithi033' 2024-03-19T02:53:06.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.571+0000 7ff02985f740 1 --2- 172.21.15.33:0/3375573944 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x557782463360 0x5577824647d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.581 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.571+0000 7ff02985f740 4 client.15390 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:53:06.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.571+0000 7ff02985f740 1 -- 172.21.15.33:0/3375573944 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5577823be320 con 0x557782463360 2024-03-19T02:53:06.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.571+0000 7ff024a1c640 1 --2- 172.21.15.33:0/3375573944 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x557782463360 0x5577824647d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.571+0000 7feffffff640 10 client.15390 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T02:53:06.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.571+0000 7ff024a1c640 1 --2- 172.21.15.33:0/3375573944 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x557782463360 0x5577824647d0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=20177e2c4c5ba0ef server_cookie=bb03fdde1a130ad7 in_seq=0 out_seq=0 2024-03-19T02:53:06.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.572+0000 7feffffff640 1 -- 172.21.15.33:0/3375573944 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5577823be320 con 0x557782463360 2024-03-19T02:53:06.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.572+0000 7feffffff640 10 client.15390 handle_command_reply: tid=0 2024-03-19T02:53:06.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.572+0000 7ff02985f740 1 client.15390 shutdown 2024-03-19T02:53:06.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.573+0000 7ff02985f740 1 -- 172.21.15.33:0/3375573944 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557782451b50 msgr2=0x55778245da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:06.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.573+0000 7ff02985f740 1 --2- 172.21.15.33:0/3375573944 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557782451b50 0x55778245da60 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff018032380 tx=0x7ff018009960 comp rx=0 tx=0).stop 2024-03-19T02:53:06.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.573+0000 7ff02985f740 1 -- 172.21.15.33:0/3375573944 shutdown_connections 2024-03-19T02:53:06.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.573+0000 7ff02985f740 1 --2- 172.21.15.33:0/3375573944 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x557782463360 0x5577824647d0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.573+0000 7ff02985f740 1 --2- 172.21.15.33:0/3375573944 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778244c9e0 0x557782456a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.573+0000 7ff02985f740 1 --2- 172.21.15.33:0/3375573944 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557782451b50 0x55778245da60 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.573+0000 7feffffff640 0 client.15390 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T02:53:06.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.573+0000 7ff02985f740 1 --2- 172.21.15.33:0/3375573944 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55778244bff0 0x5577824564e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.573+0000 7ff02985f740 1 -- 172.21.15.33:0/3375573944 >> 172.21.15.33:0/3375573944 conn(0x557782447560 msgr2=0x5577824467e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:06.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.573+0000 7ff02985f740 1 -- 172.21.15.33:0/3375573944 shutdown_connections 2024-03-19T02:53:06.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.573+0000 7ff02985f740 1 -- 172.21.15.33:0/3375573944 wait complete. 2024-03-19T02:53:06.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.573+0000 7ff02985f740 20 client.15390 trim_cache size 0 max 16384 2024-03-19T02:53:06.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.591+0000 7ff02985f740 1 Processor -- start 2024-03-19T02:53:06.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.591+0000 7ff02985f740 1 -- start start 2024-03-19T02:53:06.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.591+0000 7ff02985f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778244bff0 0x5577824550b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.591+0000 7ff02985f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55778244c9e0 0x55778245da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.591+0000 7ff02985f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557782451b50 0x557782462130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.591+0000 7ff02985f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557782462b10 con 0x55778244c9e0 2024-03-19T02:53:06.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.591+0000 7ff02985f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5577824631e0 con 0x557782451b50 2024-03-19T02:53:06.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.591+0000 7ff02985f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5577824638b0 con 0x55778244bff0 2024-03-19T02:53:06.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.591+0000 7ff01ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55778244c9e0 0x55778245da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.591+0000 7ff01ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55778244c9e0 0x55778245da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46486/0 (socket says 172.21.15.33:46486) 2024-03-19T02:53:06.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.591+0000 7ff01ffff640 1 -- 172.21.15.33:0/3134454024 learned_addr learned my addr 172.21.15.33:0/3134454024 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:06.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.591+0000 7ff02521d640 1 --2- 172.21.15.33:0/3134454024 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557782451b50 0x557782462130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.591+0000 7ff024a1c640 1 --2- 172.21.15.33:0/3134454024 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778244bff0 0x5577824550b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.591+0000 7ff01ffff640 1 -- 172.21.15.33:0/3134454024 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778244bff0 msgr2=0x5577824550b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:06.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.591+0000 7ff01ffff640 1 --2- 172.21.15.33:0/3134454024 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778244bff0 0x5577824550b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.591+0000 7ff01ffff640 1 -- 172.21.15.33:0/3134454024 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557782451b50 msgr2=0x557782462130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:06.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.591+0000 7ff01ffff640 1 --2- 172.21.15.33:0/3134454024 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557782451b50 0x557782462130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.591+0000 7ff01ffff640 1 -- 172.21.15.33:0/3134454024 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557782462fc0 con 0x55778244c9e0 2024-03-19T02:53:06.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.591+0000 7ff02521d640 1 --2- 172.21.15.33:0/3134454024 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557782451b50 0x557782462130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:53:06.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.591+0000 7ff01ffff640 1 --2- 172.21.15.33:0/3134454024 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55778244c9e0 0x55778245da20 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ff00c09b3a0 tx=0x7ff00c086050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=222f725a293bc261 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:06.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.592+0000 7ff00a7fc640 1 -- 172.21.15.33:0/3134454024 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff00c0b9070 con 0x55778244c9e0 2024-03-19T02:53:06.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.592+0000 7ff00a7fc640 1 -- 172.21.15.33:0/3134454024 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff00c0a3e40 con 0x55778244c9e0 2024-03-19T02:53:06.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.592+0000 7ff00a7fc640 1 -- 172.21.15.33:0/3134454024 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff00c0b4040 con 0x55778244c9e0 2024-03-19T02:53:06.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.592+0000 7ff02985f740 1 -- 172.21.15.33:0/3134454024 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55778244c9e0 msgr2=0x55778245da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:06.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.592+0000 7ff02985f740 1 --2- 172.21.15.33:0/3134454024 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55778244c9e0 0x55778245da20 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ff00c09b3a0 tx=0x7ff00c086050 comp rx=0 tx=0).stop 2024-03-19T02:53:06.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.592+0000 7ff02985f740 1 -- 172.21.15.33:0/3134454024 shutdown_connections 2024-03-19T02:53:06.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.592+0000 7ff02985f740 1 --2- 172.21.15.33:0/3134454024 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557782451b50 0x557782462130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.592+0000 7ff02985f740 1 --2- 172.21.15.33:0/3134454024 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55778244c9e0 0x55778245da20 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.592+0000 7ff02985f740 1 --2- 172.21.15.33:0/3134454024 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778244bff0 0x5577824550b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.592+0000 7ff02985f740 1 -- 172.21.15.33:0/3134454024 >> 172.21.15.33:0/3134454024 conn(0x557782447560 msgr2=0x557782447940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:06.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.592+0000 7ff02985f740 1 -- 172.21.15.33:0/3134454024 shutdown_connections 2024-03-19T02:53:06.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.592+0000 7ff02985f740 1 -- 172.21.15.33:0/3134454024 wait complete. 2024-03-19T02:53:06.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.592+0000 7ff02985f740 1 -- start start 2024-03-19T02:53:06.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.593+0000 7ff02985f740 1 Processor -- start 2024-03-19T02:53:06.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.593+0000 7ff02985f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55778244bff0 0x557782458fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.593+0000 7ff02985f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778244c9e0 0x557782459520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.593+0000 7ff02985f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557782451b50 0x55778245c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.593+0000 7ff02985f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557782464650 con 0x557782451b50 2024-03-19T02:53:06.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.593+0000 7ff02985f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557782463f80 con 0x55778244bff0 2024-03-19T02:53:06.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.593+0000 7ff02985f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x557782464d20 con 0x55778244c9e0 2024-03-19T02:53:06.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.593+0000 7ff02985f740 10 client.0 _ll_register_callbacks cb 0x7ff014005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:53:06.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.593+0000 7ff02521d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557782451b50 0x55778245c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.593+0000 7ff02521d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557782451b50 0x55778245c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46490/0 (socket says 172.21.15.33:46490) 2024-03-19T02:53:06.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.593+0000 7ff02521d640 1 -- 172.21.15.33:0/2629969019 learned_addr learned my addr 172.21.15.33:0/2629969019 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:06.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.593+0000 7ff024a1c640 1 --2- 172.21.15.33:0/2629969019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55778244bff0 0x557782458fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.593+0000 7ff01ffff640 1 --2- 172.21.15.33:0/2629969019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778244c9e0 0x557782459520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.593+0000 7ff02521d640 1 -- 172.21.15.33:0/2629969019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778244c9e0 msgr2=0x557782459520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:06.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.593+0000 7ff02521d640 1 --2- 172.21.15.33:0/2629969019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778244c9e0 0x557782459520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.593+0000 7ff02521d640 1 -- 172.21.15.33:0/2629969019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55778244bff0 msgr2=0x557782458fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:06.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.593+0000 7ff02521d640 1 --2- 172.21.15.33:0/2629969019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55778244bff0 0x557782458fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.593+0000 7ff02521d640 1 -- 172.21.15.33:0/2629969019 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55778245cb30 con 0x557782451b50 2024-03-19T02:53:06.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.593+0000 7ff02521d640 1 --2- 172.21.15.33:0/2629969019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557782451b50 0x55778245c540 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ff01802c130 tx=0x7ff018032f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=459fd2de691bdc2f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:06.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.593+0000 7feffffff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:53:06.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.594+0000 7feffffff640 1 -- 172.21.15.33:0/2629969019 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff018044070 con 0x557782451b50 2024-03-19T02:53:06.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.594+0000 7feffffff640 1 -- 172.21.15.33:0/2629969019 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7ff018040840 con 0x557782451b50 2024-03-19T02:53:06.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.594+0000 7ff02985f740 1 -- 172.21.15.33:0/2629969019 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff00c0a3e40 con 0x557782451b50 2024-03-19T02:53:06.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.594+0000 7feffffff640 1 -- 172.21.15.33:0/2629969019 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff01805d050 con 0x557782451b50 2024-03-19T02:53:06.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.594+0000 7ff02985f740 10 client.15396 fetch_fsmap learned FSMap version 24 2024-03-19T02:53:06.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.594+0000 7ff02985f740 1 -- 172.21.15.33:0/2629969019 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5577824c90d0 con 0x557782451b50 2024-03-19T02:53:06.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.594+0000 7feffffff640 1 -- 172.21.15.33:0/2629969019 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff00c0a3e40 con 0x557782451b50 2024-03-19T02:53:06.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.594+0000 7feffffff640 1 -- 172.21.15.33:0/2629969019 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7ff018054070 con 0x557782451b50 2024-03-19T02:53:06.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.594+0000 7ff02985f740 10 client.15396 fetch_fsmap finished waiting for FSMap version 24 2024-03-19T02:53:06.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.594+0000 7ff02985f740 10 client.15396 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-19T02:53:06.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.594+0000 7ff02985f740 20 client.15396 populate_metadata read hostname 'smithi033' 2024-03-19T02:53:06.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.594+0000 7ff02985f740 1 --2- 172.21.15.33:0/2629969019 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x5577824cdc80 0x5577824cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.594+0000 7ff02985f740 4 client.15396 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T02:53:06.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.594+0000 7ff02985f740 1 -- 172.21.15.33:0/2629969019 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "status"}) -- 0x5577823b8670 con 0x5577824cdc80 2024-03-19T02:53:06.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.594+0000 7ff024a1c640 1 --2- 172.21.15.33:0/2629969019 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x5577824cdc80 0x5577824cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.595+0000 7feffffff640 10 client.15396 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T02:53:06.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.595+0000 7ff024a1c640 1 --2- 172.21.15.33:0/2629969019 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x5577824cdc80 0x5577824cc990 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3be1c0b454ded266 server_cookie=ef2f3da150353caa in_seq=0 out_seq=0 2024-03-19T02:53:06.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.595+0000 7feffffff640 1 -- 172.21.15.33:0/2629969019 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5577823b8670 con 0x5577824cdc80 2024-03-19T02:53:06.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.595+0000 7feffffff640 10 client.15396 handle_command_reply: tid=0 2024-03-19T02:53:06.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.595+0000 7ff02985f740 1 client.15396 shutdown 2024-03-19T02:53:06.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.595+0000 7ff02985f740 1 -- 172.21.15.33:0/2629969019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557782451b50 msgr2=0x55778245c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:06.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.595+0000 7ff02985f740 1 --2- 172.21.15.33:0/2629969019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557782451b50 0x55778245c540 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ff01802c130 tx=0x7ff018032f50 comp rx=0 tx=0).stop 2024-03-19T02:53:06.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.595+0000 7ff02985f740 1 -- 172.21.15.33:0/2629969019 shutdown_connections 2024-03-19T02:53:06.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.595+0000 7ff02985f740 1 --2- 172.21.15.33:0/2629969019 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x5577824cdc80 0x5577824cc990 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.595+0000 7ff02985f740 1 --2- 172.21.15.33:0/2629969019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55778244c9e0 0x557782459520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.595+0000 7feffffff640 0 client.15396 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T02:53:06.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.595+0000 7ff02985f740 1 --2- 172.21.15.33:0/2629969019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557782451b50 0x55778245c540 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.595+0000 7ff02985f740 1 --2- 172.21.15.33:0/2629969019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55778244bff0 0x557782458fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.595+0000 7ff02985f740 1 -- 172.21.15.33:0/2629969019 >> 172.21.15.33:0/2629969019 conn(0x557782447560 msgr2=0x55778244de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:06.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.595+0000 7ff02985f740 1 -- 172.21.15.33:0/2629969019 shutdown_connections 2024-03-19T02:53:06.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.595+0000 7ff02985f740 1 -- 172.21.15.33:0/2629969019 wait complete. 2024-03-19T02:53:06.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.595+0000 7ff02985f740 20 client.15396 trim_cache size 0 max 16384 2024-03-19T02:53:06.604 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:53:06.604 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_fsid": "2b7a8938-e59a-11ee-95c9-87774f69a715", 2024-03-19T02:53:06.604 INFO:teuthology.orchestra.run.smithi033.stdout: "whoami": 1, 2024-03-19T02:53:06.604 INFO:teuthology.orchestra.run.smithi033.stdout: "id": 24488, 2024-03-19T02:53:06.604 INFO:teuthology.orchestra.run.smithi033.stdout: "want_state": "up:active", 2024-03-19T02:53:06.604 INFO:teuthology.orchestra.run.smithi033.stdout: "state": "up:active", 2024-03-19T02:53:06.604 INFO:teuthology.orchestra.run.smithi033.stdout: "fs_name": "cephfs", 2024-03-19T02:53:06.604 INFO:teuthology.orchestra.run.smithi033.stdout: "rank_uptime": 109.53754292000001, 2024-03-19T02:53:06.604 INFO:teuthology.orchestra.run.smithi033.stdout: "mdsmap_epoch": 24, 2024-03-19T02:53:06.604 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch": 97, 2024-03-19T02:53:06.604 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch_barrier": 93, 2024-03-19T02:53:06.604 INFO:teuthology.orchestra.run.smithi033.stdout: "uptime": 150.274889742 2024-03-19T02:53:06.604 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:53:06.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.596+0000 7ff026c80640 1 -- 172.21.15.33:0/1604197114 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff0100a2800 msgr2=0x7ff0100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:06.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.596+0000 7ff026c80640 1 --2- 172.21.15.33:0/1604197114 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff0100a2800 0x7ff0100a4cc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff0140058c0 tx=0x7ff014005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:06.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.596+0000 7ff026c80640 1 -- 172.21.15.33:0/1604197114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff020101920 msgr2=0x7ff020102d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:06.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.596+0000 7ff026c80640 1 --2- 172.21.15.33:0/1604197114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff020101920 0x7ff020102d00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff00c00e9f0 tx=0x7ff00c00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.596+0000 7ff026c80640 1 -- 172.21.15.33:0/1604197114 shutdown_connections 2024-03-19T02:53:06.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.596+0000 7ff026c80640 1 --2- 172.21.15.33:0/1604197114 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff0100a2800 0x7ff0100a4cc0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.596+0000 7ff026c80640 1 --2- 172.21.15.33:0/1604197114 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0201022e0 0x7ff020103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.596+0000 7ff026c80640 1 --2- 172.21.15.33:0/1604197114 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff020101920 0x7ff020102d00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.596+0000 7ff026c80640 1 --2- 172.21.15.33:0/1604197114 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff020100730 0x7ff0201046c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.596+0000 7ff026c80640 1 -- 172.21.15.33:0/1604197114 >> 172.21.15.33:0/1604197114 conn(0x7ff0200fbcc0 msgr2=0x7ff0200fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:06.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.596+0000 7ff026c80640 1 -- 172.21.15.33:0/1604197114 shutdown_connections 2024-03-19T02:53:06.606 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.596+0000 7ff026c80640 1 -- 172.21.15.33:0/1604197114 wait complete. 2024-03-19T02:53:06.606 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:06 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3793598099' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:53:06.606 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:06 smithi033 ceph-mon[31973]: pgmap v370: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:53:06.607 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:06 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3722672193' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:53:06.607 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:06 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3500053136' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:53:06.608 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-19T02:53:06.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.710+0000 7f9bafcc6640 1 -- 172.21.15.33:0/1441215285 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ba81096f0 msgr2=0x7f9ba810bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:06.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.710+0000 7f9bafcc6640 1 --2- 172.21.15.33:0/1441215285 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ba81096f0 0x7f9ba810bb80 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0009ca0 tx=0x7f9ba002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.710+0000 7f9bafcc6640 1 -- 172.21.15.33:0/1441215285 shutdown_connections 2024-03-19T02:53:06.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.710+0000 7f9bafcc6640 1 --2- 172.21.15.33:0/1441215285 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ba8074280 0x7f9ba8074700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.710+0000 7f9bafcc6640 1 --2- 172.21.15.33:0/1441215285 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ba81096f0 0x7f9ba810bb80 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.710+0000 7f9bafcc6640 1 --2- 172.21.15.33:0/1441215285 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ba8075860 0x7f9ba8073cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.710+0000 7f9bafcc6640 1 -- 172.21.15.33:0/1441215285 >> 172.21.15.33:0/1441215285 conn(0x7f9ba80fcdc0 msgr2=0x7f9ba80ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:06.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.710+0000 7f9bafcc6640 1 -- 172.21.15.33:0/1441215285 shutdown_connections 2024-03-19T02:53:06.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.710+0000 7f9bafcc6640 1 -- 172.21.15.33:0/1441215285 wait complete. 2024-03-19T02:53:06.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.711+0000 7f9bafcc6640 1 Processor -- start 2024-03-19T02:53:06.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.711+0000 7f9bafcc6640 1 -- start start 2024-03-19T02:53:06.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.711+0000 7f9bafcc6640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ba8074280 0x7f9ba8199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.711+0000 7f9bafcc6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ba8075860 0x7f9ba819a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.711+0000 7f9bafcc6640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ba81096f0 0x7f9ba81a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.711+0000 7f9bafcc6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba810f540 con 0x7f9ba8075860 2024-03-19T02:53:06.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.711+0000 7f9bafcc6640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba810f3c0 con 0x7f9ba81096f0 2024-03-19T02:53:06.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.711+0000 7f9bafcc6640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba810f6c0 con 0x7f9ba8074280 2024-03-19T02:53:06.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.711+0000 7f9bad261640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ba8075860 0x7f9ba819a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.711+0000 7f9bada62640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ba8074280 0x7f9ba8199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.711+0000 7f9bad261640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ba8075860 0x7f9ba819a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46498/0 (socket says 172.21.15.33:46498) 2024-03-19T02:53:06.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.711+0000 7f9bad261640 1 -- 172.21.15.33:0/2713605423 learned_addr learned my addr 172.21.15.33:0/2713605423 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:06.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.711+0000 7f9bae263640 1 --2- 172.21.15.33:0/2713605423 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ba81096f0 0x7f9ba81a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.711+0000 7f9bada62640 1 -- 172.21.15.33:0/2713605423 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ba81096f0 msgr2=0x7f9ba81a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:06.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.711+0000 7f9bada62640 1 --2- 172.21.15.33:0/2713605423 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ba81096f0 0x7f9ba81a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.711+0000 7f9bada62640 1 -- 172.21.15.33:0/2713605423 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ba8075860 msgr2=0x7f9ba819a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:06.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.711+0000 7f9bada62640 1 --2- 172.21.15.33:0/2713605423 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ba8075860 0x7f9ba819a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.711+0000 7f9bada62640 1 -- 172.21.15.33:0/2713605423 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ba81a1840 con 0x7f9ba8074280 2024-03-19T02:53:06.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.711+0000 7f9bad261640 1 --2- 172.21.15.33:0/2713605423 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ba8075860 0x7f9ba819a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:06.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.712+0000 7f9bada62640 1 --2- 172.21.15.33:0/2713605423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ba8074280 0x7f9ba8199c30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9ba400e9f0 tx=0x7f9ba400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:06.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.712+0000 7f9b9effd640 1 -- 172.21.15.33:0/2713605423 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba400ccc0 con 0x7f9ba8074280 2024-03-19T02:53:06.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.712+0000 7f9bafcc6640 1 -- 172.21.15.33:0/2713605423 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ba81a1b30 con 0x7f9ba8074280 2024-03-19T02:53:06.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.712+0000 7f9b9effd640 1 -- 172.21.15.33:0/2713605423 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9ba40108e0 con 0x7f9ba8074280 2024-03-19T02:53:06.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.712+0000 7f9b9effd640 1 -- 172.21.15.33:0/2713605423 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba4018880 con 0x7f9ba8074280 2024-03-19T02:53:06.716 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.712+0000 7f9bafcc6640 1 -- 172.21.15.33:0/2713605423 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ba81a20d0 con 0x7f9ba8074280 2024-03-19T02:53:06.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.714+0000 7f9b9effd640 1 -- 172.21.15.33:0/2713605423 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9ba4010a90 con 0x7f9ba8074280 2024-03-19T02:53:06.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.715+0000 7f9bb298d740 1 Processor -- start 2024-03-19T02:53:06.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.715+0000 7f9bb298d740 1 -- start start 2024-03-19T02:53:06.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.715+0000 7f9bb298d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f2c534ff0 0x555f2c535410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.715+0000 7f9bb298d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555f2c5359e0 0x555f2c53a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.715+0000 7f9b9effd640 1 --2- 172.21.15.33:0/2713605423 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9b940a2800 0x7f9b940a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.715+0000 7f9bb298d740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555f2c53ab50 0x555f2c53cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.715+0000 7f9bb298d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555f2c499de0 con 0x555f2c534ff0 2024-03-19T02:53:06.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.715+0000 7f9bb298d740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x555f2c442c80 con 0x555f2c53ab50 2024-03-19T02:53:06.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.715+0000 7f9bb298d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x555f2c495e80 con 0x555f2c5359e0 2024-03-19T02:53:06.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.715+0000 7f9bae263640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f2c534ff0 0x555f2c535410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.715+0000 7f9bad261640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555f2c5359e0 0x555f2c53a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.715+0000 7f9bae263640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f2c534ff0 0x555f2c535410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46508/0 (socket says 172.21.15.33:46508) 2024-03-19T02:53:06.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.715+0000 7f9bae263640 1 -- 172.21.15.33:0/3302407701 learned_addr learned my addr 172.21.15.33:0/3302407701 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:06.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.715+0000 7f9bada62640 1 --2- 172.21.15.33:0/2713605423 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9b940a2800 0x7f9b940a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.715+0000 7f9b9effd640 1 -- 172.21.15.33:0/2713605423 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f9ba40cf1c0 con 0x7f9ba8074280 2024-03-19T02:53:06.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.716+0000 7f9bae263640 1 --2- 172.21.15.33:0/3302407701 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555f2c53ab50 0x555f2c53cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.716+0000 7f9bad261640 1 -- 172.21.15.33:0/3302407701 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555f2c53ab50 msgr2=0x555f2c53cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:06.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.716+0000 7f9bad261640 1 --2- 172.21.15.33:0/3302407701 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555f2c53ab50 0x555f2c53cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.716+0000 7f9bad261640 1 -- 172.21.15.33:0/3302407701 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f2c534ff0 msgr2=0x555f2c535410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:06.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.716+0000 7f9bad261640 1 --2- 172.21.15.33:0/3302407701 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f2c534ff0 0x555f2c535410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.716+0000 7f9bad261640 1 -- 172.21.15.33:0/3302407701 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f2c454fb0 con 0x555f2c5359e0 2024-03-19T02:53:06.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.716+0000 7f9bada62640 1 --2- 172.21.15.33:0/2713605423 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9b940a2800 0x7f9b940a4cc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9ba409b580 tx=0x7f9ba4025900 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:06.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.716+0000 7f9bad261640 1 --2- 172.21.15.33:0/3302407701 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555f2c5359e0 0x555f2c53a4e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9b98011510 tx=0x7f9b980119d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d2e448cfc76541e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:06.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.716+0000 7f9b93fff640 1 -- 172.21.15.33:0/3302407701 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9801e070 con 0x555f2c5359e0 2024-03-19T02:53:06.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.716+0000 7f9b93fff640 1 -- 172.21.15.33:0/3302407701 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9b9800f9f0 con 0x555f2c5359e0 2024-03-19T02:53:06.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.716+0000 7f9b93fff640 1 -- 172.21.15.33:0/3302407701 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b98019a80 con 0x555f2c5359e0 2024-03-19T02:53:06.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.717+0000 7f9bb298d740 1 -- 172.21.15.33:0/3302407701 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555f2c5359e0 msgr2=0x555f2c53a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:06.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.717+0000 7f9bb298d740 1 --2- 172.21.15.33:0/3302407701 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555f2c5359e0 0x555f2c53a4e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9b98011510 tx=0x7f9b980119d0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.717+0000 7f9bb298d740 1 -- 172.21.15.33:0/3302407701 shutdown_connections 2024-03-19T02:53:06.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.717+0000 7f9bb298d740 1 --2- 172.21.15.33:0/3302407701 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555f2c5359e0 0x555f2c53a4e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.717+0000 7f9bb298d740 1 --2- 172.21.15.33:0/3302407701 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555f2c53ab50 0x555f2c53cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.717+0000 7f9bb298d740 1 --2- 172.21.15.33:0/3302407701 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f2c534ff0 0x555f2c535410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.717+0000 7f9bb298d740 1 -- 172.21.15.33:0/3302407701 >> 172.21.15.33:0/3302407701 conn(0x555f2c530560 msgr2=0x555f2c530960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:06.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.717+0000 7f9bb298d740 1 -- 172.21.15.33:0/3302407701 shutdown_connections 2024-03-19T02:53:06.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.717+0000 7f9bb298d740 1 -- 172.21.15.33:0/3302407701 wait complete. 2024-03-19T02:53:06.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.717+0000 7f9bb298d740 1 -- start start 2024-03-19T02:53:06.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.717+0000 7f9bb298d740 1 Processor -- start 2024-03-19T02:53:06.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.718+0000 7f9bb298d740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555f2c534ff0 0x555f2c53f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.718+0000 7f9bb298d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f2c5359e0 0x555f2c53fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.718+0000 7f9bb298d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555f2c53ab50 0x555f2c546aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.718+0000 7f9bb298d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555f2c53db00 con 0x555f2c5359e0 2024-03-19T02:53:06.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.718+0000 7f9bb298d740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x555f2c455fc0 con 0x555f2c534ff0 2024-03-19T02:53:06.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.718+0000 7f9bb298d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x555f2c53dc80 con 0x555f2c53ab50 2024-03-19T02:53:06.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.718+0000 7f9bb298d740 10 client.0 _ll_register_callbacks cb 0x555f2c491a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:53:06.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.718+0000 7f9bad261640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f2c5359e0 0x555f2c53fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.718+0000 7f9bad261640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f2c5359e0 0x555f2c53fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46510/0 (socket says 172.21.15.33:46510) 2024-03-19T02:53:06.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.718+0000 7f9bad261640 1 -- 172.21.15.33:0/4168420993 learned_addr learned my addr 172.21.15.33:0/4168420993 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:06.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.718+0000 7f9bae263640 1 --2- 172.21.15.33:0/4168420993 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555f2c534ff0 0x555f2c53f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.718+0000 7f9bae263640 1 --2- 172.21.15.33:0/4168420993 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555f2c53ab50 0x555f2c546aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.718+0000 7f9bad261640 1 -- 172.21.15.33:0/4168420993 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555f2c53ab50 msgr2=0x555f2c546aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:06.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.718+0000 7f9bad261640 1 --2- 172.21.15.33:0/4168420993 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555f2c53ab50 0x555f2c546aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.718+0000 7f9bad261640 1 -- 172.21.15.33:0/4168420993 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555f2c534ff0 msgr2=0x555f2c53f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:06.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.718+0000 7f9bad261640 1 --2- 172.21.15.33:0/4168420993 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555f2c534ff0 0x555f2c53f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.718+0000 7f9bad261640 1 -- 172.21.15.33:0/4168420993 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f2c1ae2d0 con 0x555f2c5359e0 2024-03-19T02:53:06.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.718+0000 7f9bae263640 1 --2- 172.21.15.33:0/4168420993 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555f2c534ff0 0x555f2c53f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:06.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.718+0000 7f9bad261640 1 --2- 172.21.15.33:0/4168420993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f2c5359e0 0x555f2c53fad0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f9b9801b000 tx=0x7f9b9800ffd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ad54c5aa7bc698c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:06.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.718+0000 7f9b917fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:53:06.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.718+0000 7f9b917fa640 1 -- 172.21.15.33:0/4168420993 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9801e040 con 0x555f2c5359e0 2024-03-19T02:53:06.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.718+0000 7f9b917fa640 1 -- 172.21.15.33:0/4168420993 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f9b98005040 con 0x555f2c5359e0 2024-03-19T02:53:06.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.719+0000 7f9bb298d740 1 -- 172.21.15.33:0/4168420993 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b9800f9f0 con 0x555f2c5359e0 2024-03-19T02:53:06.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.719+0000 7f9b917fa640 1 -- 172.21.15.33:0/4168420993 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9b98032050 con 0x555f2c5359e0 2024-03-19T02:53:06.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.719+0000 7f9bb298d740 10 client.15408 fetch_fsmap learned FSMap version 24 2024-03-19T02:53:06.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.719+0000 7f9bb298d740 1 -- 172.21.15.33:0/4168420993 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x555f2c2786b0 con 0x555f2c5359e0 2024-03-19T02:53:06.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.719+0000 7f9b917fa640 1 -- 172.21.15.33:0/4168420993 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9b9800f9f0 con 0x555f2c5359e0 2024-03-19T02:53:06.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.719+0000 7f9b917fa640 1 -- 172.21.15.33:0/4168420993 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f9b98021030 con 0x555f2c5359e0 2024-03-19T02:53:06.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.719+0000 7f9bb298d740 10 client.15408 fetch_fsmap finished waiting for FSMap version 24 2024-03-19T02:53:06.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.719+0000 7f9bb298d740 10 client.15408 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-19T02:53:06.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.719+0000 7f9bb298d740 20 client.15408 populate_metadata read hostname 'smithi033' 2024-03-19T02:53:06.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.719+0000 7f9bb298d740 1 --2- 172.21.15.33:0/4168420993 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x555f2c54c0f0 0x555f2c54d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.719+0000 7f9bb298d740 4 client.15408 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:53:06.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.719+0000 7f9bb298d740 1 -- 172.21.15.33:0/4168420993 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f2c4a7320 con 0x555f2c54c0f0 2024-03-19T02:53:06.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.719+0000 7f9bae263640 1 --2- 172.21.15.33:0/4168420993 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x555f2c54c0f0 0x555f2c54d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.720+0000 7f9bae263640 1 --2- 172.21.15.33:0/4168420993 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x555f2c54c0f0 0x555f2c54d560 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fa7c2c01f273fae server_cookie=bdd840e589bb4469 in_seq=0 out_seq=0 2024-03-19T02:53:06.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.720+0000 7f9b917fa640 10 client.15408 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:53:06.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.721+0000 7f9b917fa640 1 -- 172.21.15.33:0/4168420993 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555f2c4a7320 con 0x555f2c54c0f0 2024-03-19T02:53:06.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.721+0000 7f9b917fa640 10 client.15408 handle_command_reply: tid=0 2024-03-19T02:53:06.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.721+0000 7f9bb298d740 1 client.15408 shutdown 2024-03-19T02:53:06.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.721+0000 7f9bb298d740 1 -- 172.21.15.33:0/4168420993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f2c5359e0 msgr2=0x555f2c53fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:06.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.721+0000 7f9bb298d740 1 --2- 172.21.15.33:0/4168420993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f2c5359e0 0x555f2c53fad0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f9b9801b000 tx=0x7f9b9800ffd0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.721+0000 7f9bb298d740 1 -- 172.21.15.33:0/4168420993 shutdown_connections 2024-03-19T02:53:06.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.721+0000 7f9bb298d740 1 --2- 172.21.15.33:0/4168420993 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x555f2c54c0f0 0x555f2c54d560 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.721+0000 7f9bb298d740 1 --2- 172.21.15.33:0/4168420993 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555f2c53ab50 0x555f2c546aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.721+0000 7f9bb298d740 1 --2- 172.21.15.33:0/4168420993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f2c5359e0 0x555f2c53fad0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.721+0000 7f9bb298d740 1 --2- 172.21.15.33:0/4168420993 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555f2c534ff0 0x555f2c53f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.721+0000 7f9b917fa640 0 client.15408 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:53:06.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.721+0000 7f9bb298d740 1 -- 172.21.15.33:0/4168420993 >> 172.21.15.33:0/4168420993 conn(0x555f2c530560 msgr2=0x555f2c52f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:06.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.721+0000 7f9bb298d740 1 -- 172.21.15.33:0/4168420993 shutdown_connections 2024-03-19T02:53:06.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.721+0000 7f9bb298d740 1 -- 172.21.15.33:0/4168420993 wait complete. 2024-03-19T02:53:06.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.721+0000 7f9bb298d740 20 client.15408 trim_cache size 0 max 16384 2024-03-19T02:53:06.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.739+0000 7f9bb298d740 1 Processor -- start 2024-03-19T02:53:06.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.739+0000 7f9bb298d740 1 -- start start 2024-03-19T02:53:06.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.739+0000 7f9bb298d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f2c534ff0 0x555f2c54b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.739+0000 7f9bb298d740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555f2c5359e0 0x555f2c546a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.739+0000 7f9bb298d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555f2c53ab50 0x555f2c530940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.739+0000 7f9bb298d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555f2c54b8a0 con 0x555f2c534ff0 2024-03-19T02:53:06.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.739+0000 7f9bb298d740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x555f2c54bf70 con 0x555f2c5359e0 2024-03-19T02:53:06.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.739+0000 7f9bb298d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x555f2c54c640 con 0x555f2c53ab50 2024-03-19T02:53:06.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.739+0000 7f9bae263640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f2c534ff0 0x555f2c54b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.739+0000 7f9bae263640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f2c534ff0 0x555f2c54b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46518/0 (socket says 172.21.15.33:46518) 2024-03-19T02:53:06.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.739+0000 7f9bae263640 1 -- 172.21.15.33:0/2514849586 learned_addr learned my addr 172.21.15.33:0/2514849586 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:06.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.739+0000 7f9bad261640 1 --2- 172.21.15.33:0/2514849586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555f2c5359e0 0x555f2c546a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.740+0000 7f9bae263640 1 --2- 172.21.15.33:0/2514849586 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555f2c53ab50 0x555f2c530940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.740+0000 7f9bae263640 1 -- 172.21.15.33:0/2514849586 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555f2c53ab50 msgr2=0x555f2c530940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:06.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.740+0000 7f9bae263640 1 --2- 172.21.15.33:0/2514849586 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555f2c53ab50 0x555f2c530940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.740+0000 7f9bae263640 1 -- 172.21.15.33:0/2514849586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555f2c5359e0 msgr2=0x555f2c546a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:06.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.740+0000 7f9bae263640 1 --2- 172.21.15.33:0/2514849586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555f2c5359e0 0x555f2c546a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.740+0000 7f9bae263640 1 -- 172.21.15.33:0/2514849586 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f2c53a1e0 con 0x555f2c534ff0 2024-03-19T02:53:06.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.740+0000 7f9bad261640 1 --2- 172.21.15.33:0/2514849586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555f2c5359e0 0x555f2c546a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:06.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.740+0000 7f9bae263640 1 --2- 172.21.15.33:0/2514849586 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f2c534ff0 0x555f2c54b090 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0032ae0 tx=0x7f9ba00095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dddc0b2fbb59c9e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:06.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.740+0000 7f9b937fe640 1 -- 172.21.15.33:0/2514849586 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba0039040 con 0x555f2c534ff0 2024-03-19T02:53:06.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.740+0000 7f9b937fe640 1 -- 172.21.15.33:0/2514849586 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9ba0034520 con 0x555f2c534ff0 2024-03-19T02:53:06.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.740+0000 7f9b937fe640 1 -- 172.21.15.33:0/2514849586 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba004f820 con 0x555f2c534ff0 2024-03-19T02:53:06.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.741+0000 7f9bb298d740 1 -- 172.21.15.33:0/2514849586 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f2c534ff0 msgr2=0x555f2c54b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:06.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.741+0000 7f9bb298d740 1 --2- 172.21.15.33:0/2514849586 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f2c534ff0 0x555f2c54b090 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0032ae0 tx=0x7f9ba00095c0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.741+0000 7f9bb298d740 1 -- 172.21.15.33:0/2514849586 shutdown_connections 2024-03-19T02:53:06.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.741+0000 7f9bb298d740 1 --2- 172.21.15.33:0/2514849586 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555f2c53ab50 0x555f2c530940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.741+0000 7f9bb298d740 1 --2- 172.21.15.33:0/2514849586 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555f2c5359e0 0x555f2c546a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.741+0000 7f9bb298d740 1 --2- 172.21.15.33:0/2514849586 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f2c534ff0 0x555f2c54b090 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.741+0000 7f9bb298d740 1 -- 172.21.15.33:0/2514849586 >> 172.21.15.33:0/2514849586 conn(0x555f2c530560 msgr2=0x555f2c536450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:06.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.741+0000 7f9bb298d740 1 -- 172.21.15.33:0/2514849586 shutdown_connections 2024-03-19T02:53:06.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.741+0000 7f9bb298d740 1 -- 172.21.15.33:0/2514849586 wait complete. 2024-03-19T02:53:06.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.741+0000 7f9bb298d740 1 -- start start 2024-03-19T02:53:06.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.741+0000 7f9bb298d740 1 Processor -- start 2024-03-19T02:53:06.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.741+0000 7f9bb298d740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555f2c534ff0 0x555f2c541ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.741+0000 7f9bb298d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555f2c5359e0 0x555f2c542010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.741+0000 7f9bb298d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f2c53ab50 0x555f2c5afe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.741+0000 7f9bb298d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x555f2c54d3e0 con 0x555f2c53ab50 2024-03-19T02:53:06.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.741+0000 7f9bb298d740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x555f2c54cd10 con 0x555f2c534ff0 2024-03-19T02:53:06.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.741+0000 7f9bb298d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x555f2c54dab0 con 0x555f2c5359e0 2024-03-19T02:53:06.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.741+0000 7f9bb298d740 10 client.0 _ll_register_callbacks cb 0x7f9b9801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:53:06.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.742+0000 7f9bae263640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555f2c534ff0 0x555f2c541ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.742+0000 7f9bae263640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f2c53ab50 0x555f2c5afe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.742+0000 7f9bad261640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555f2c5359e0 0x555f2c542010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.742+0000 7f9bae263640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f2c53ab50 0x555f2c5afe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46524/0 (socket says 172.21.15.33:46524) 2024-03-19T02:53:06.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.742+0000 7f9bae263640 1 -- 172.21.15.33:0/2936675500 learned_addr learned my addr 172.21.15.33:0/2936675500 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:06.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.742+0000 7f9bae263640 1 -- 172.21.15.33:0/2936675500 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555f2c5359e0 msgr2=0x555f2c542010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:06.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.742+0000 7f9bae263640 1 --2- 172.21.15.33:0/2936675500 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555f2c5359e0 0x555f2c542010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.742+0000 7f9bae263640 1 -- 172.21.15.33:0/2936675500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555f2c534ff0 msgr2=0x555f2c541ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:06.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.742+0000 7f9bae263640 1 --2- 172.21.15.33:0/2936675500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555f2c534ff0 0x555f2c541ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.749 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.742+0000 7f9bae263640 1 -- 172.21.15.33:0/2936675500 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f2c5b0440 con 0x555f2c53ab50 2024-03-19T02:53:06.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.742+0000 7f9bad261640 1 --2- 172.21.15.33:0/2936675500 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555f2c5359e0 0x555f2c542010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:06.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.742+0000 7f9bae263640 1 --2- 172.21.15.33:0/2936675500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f2c53ab50 0x555f2c5afe50 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f9ba002f020 tx=0x7f9ba002c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1838390cc5727f41 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:06.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.742+0000 7f9b917fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:53:06.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.742+0000 7f9b917fa640 1 -- 172.21.15.33:0/2936675500 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba0060060 con 0x555f2c53ab50 2024-03-19T02:53:06.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.742+0000 7f9bb298d740 1 -- 172.21.15.33:0/2936675500 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ba0034520 con 0x555f2c53ab50 2024-03-19T02:53:06.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.742+0000 7f9b917fa640 1 -- 172.21.15.33:0/2936675500 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f9ba004f370 con 0x555f2c53ab50 2024-03-19T02:53:06.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.742+0000 7f9b917fa640 1 -- 172.21.15.33:0/2936675500 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9ba0039040 con 0x555f2c53ab50 2024-03-19T02:53:06.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.742+0000 7f9bb298d740 10 client.15414 fetch_fsmap learned FSMap version 24 2024-03-19T02:53:06.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.742+0000 7f9bb298d740 1 -- 172.21.15.33:0/2936675500 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x555f2c5b1740 con 0x555f2c53ab50 2024-03-19T02:53:06.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.743+0000 7f9b917fa640 1 -- 172.21.15.33:0/2936675500 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9ba0034520 con 0x555f2c53ab50 2024-03-19T02:53:06.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.743+0000 7f9b917fa640 1 -- 172.21.15.33:0/2936675500 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f9ba003f040 con 0x555f2c53ab50 2024-03-19T02:53:06.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.743+0000 7f9bb298d740 10 client.15414 fetch_fsmap finished waiting for FSMap version 24 2024-03-19T02:53:06.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.743+0000 7f9bb298d740 10 client.15414 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-19T02:53:06.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.743+0000 7f9bb298d740 20 client.15414 populate_metadata read hostname 'smithi033' 2024-03-19T02:53:06.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.743+0000 7f9bb298d740 1 --2- 172.21.15.33:0/2936675500 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x555f2c5b5e50 0x555f2c5b2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:06.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.743+0000 7f9bb298d740 4 client.15414 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T02:53:06.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.743+0000 7f9bb298d740 1 -- 172.21.15.33:0/2936675500 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "status"}) -- 0x555f2c4a1670 con 0x555f2c5b5e50 2024-03-19T02:53:06.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.743+0000 7f9bad261640 1 --2- 172.21.15.33:0/2936675500 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x555f2c5b5e50 0x555f2c5b2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:06.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.743+0000 7f9b917fa640 10 client.15414 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:53:06.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.743+0000 7f9bad261640 1 --2- 172.21.15.33:0/2936675500 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x555f2c5b5e50 0x555f2c5b2960 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8a131d2db1a2a91 server_cookie=3826dc449df37632 in_seq=0 out_seq=0 2024-03-19T02:53:06.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.744+0000 7f9b917fa640 1 -- 172.21.15.33:0/2936675500 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x555f2c4a1670 con 0x555f2c5b5e50 2024-03-19T02:53:06.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.744+0000 7f9b917fa640 10 client.15414 handle_command_reply: tid=0 2024-03-19T02:53:06.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.744+0000 7f9bb298d740 1 client.15414 shutdown 2024-03-19T02:53:06.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.744+0000 7f9bb298d740 1 -- 172.21.15.33:0/2936675500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f2c53ab50 msgr2=0x555f2c5afe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:06.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.744+0000 7f9bb298d740 1 --2- 172.21.15.33:0/2936675500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f2c53ab50 0x555f2c5afe50 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f9ba002f020 tx=0x7f9ba002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.744+0000 7f9bb298d740 1 -- 172.21.15.33:0/2936675500 shutdown_connections 2024-03-19T02:53:06.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.744+0000 7f9bb298d740 1 --2- 172.21.15.33:0/2936675500 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x555f2c5b5e50 0x555f2c5b2960 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.744+0000 7f9bb298d740 1 --2- 172.21.15.33:0/2936675500 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x555f2c5359e0 0x555f2c542010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.744+0000 7f9bb298d740 1 --2- 172.21.15.33:0/2936675500 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x555f2c53ab50 0x555f2c5afe50 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.744+0000 7f9bb298d740 1 --2- 172.21.15.33:0/2936675500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555f2c534ff0 0x555f2c541ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.744+0000 7f9b917fa640 0 client.15414 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:53:06.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.744+0000 7f9bb298d740 1 -- 172.21.15.33:0/2936675500 >> 172.21.15.33:0/2936675500 conn(0x555f2c530560 msgr2=0x555f2c530940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:06.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.744+0000 7f9bb298d740 1 -- 172.21.15.33:0/2936675500 shutdown_connections 2024-03-19T02:53:06.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.744+0000 7f9bb298d740 1 -- 172.21.15.33:0/2936675500 wait complete. 2024-03-19T02:53:06.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.744+0000 7f9bb298d740 20 client.15414 trim_cache size 0 max 16384 2024-03-19T02:53:06.752 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:53:06.752 INFO:teuthology.orchestra.run.smithi033.stdout: "cluster_fsid": "2b7a8938-e59a-11ee-95c9-87774f69a715", 2024-03-19T02:53:06.753 INFO:teuthology.orchestra.run.smithi033.stdout: "whoami": 0, 2024-03-19T02:53:06.753 INFO:teuthology.orchestra.run.smithi033.stdout: "id": 24509, 2024-03-19T02:53:06.753 INFO:teuthology.orchestra.run.smithi033.stdout: "want_state": "up:active", 2024-03-19T02:53:06.753 INFO:teuthology.orchestra.run.smithi033.stdout: "state": "up:active", 2024-03-19T02:53:06.753 INFO:teuthology.orchestra.run.smithi033.stdout: "fs_name": "cephfs", 2024-03-19T02:53:06.753 INFO:teuthology.orchestra.run.smithi033.stdout: "rank_uptime": 133.54484589800001, 2024-03-19T02:53:06.753 INFO:teuthology.orchestra.run.smithi033.stdout: "mdsmap_epoch": 24, 2024-03-19T02:53:06.753 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch": 97, 2024-03-19T02:53:06.753 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_epoch_barrier": 93, 2024-03-19T02:53:06.753 INFO:teuthology.orchestra.run.smithi033.stdout: "uptime": 139.12560106500001 2024-03-19T02:53:06.753 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:53:06.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.744+0000 7f9bafcc6640 1 -- 172.21.15.33:0/2713605423 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9b940a2800 msgr2=0x7f9b940a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:06.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.744+0000 7f9bafcc6640 1 --2- 172.21.15.33:0/2713605423 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9b940a2800 0x7f9b940a4cc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9ba409b580 tx=0x7f9ba4025900 comp rx=0 tx=0).stop 2024-03-19T02:53:06.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.744+0000 7f9bafcc6640 1 -- 172.21.15.33:0/2713605423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ba8074280 msgr2=0x7f9ba8199c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:06.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.744+0000 7f9bafcc6640 1 --2- 172.21.15.33:0/2713605423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ba8074280 0x7f9ba8199c30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9ba400e9f0 tx=0x7f9ba400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.744+0000 7f9bafcc6640 1 -- 172.21.15.33:0/2713605423 shutdown_connections 2024-03-19T02:53:06.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.744+0000 7f9bafcc6640 1 --2- 172.21.15.33:0/2713605423 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9b940a2800 0x7f9b940a4cc0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.744+0000 7f9bafcc6640 1 --2- 172.21.15.33:0/2713605423 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ba81096f0 0x7f9ba81a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.744+0000 7f9bafcc6640 1 --2- 172.21.15.33:0/2713605423 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ba8075860 0x7f9ba819a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.744+0000 7f9bafcc6640 1 --2- 172.21.15.33:0/2713605423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ba8074280 0x7f9ba8199c30 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:06.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.744+0000 7f9bafcc6640 1 -- 172.21.15.33:0/2713605423 >> 172.21.15.33:0/2713605423 conn(0x7f9ba80fcdc0 msgr2=0x7f9ba81055f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:06.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.745+0000 7f9bafcc6640 1 -- 172.21.15.33:0/2713605423 shutdown_connections 2024-03-19T02:53:06.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:06.745+0000 7f9bafcc6640 1 -- 172.21.15.33:0/2713605423 wait complete. 2024-03-19T02:53:06.756 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-19T02:53:06.766 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-19T02:53:06.766 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2024-03-19T02:53:06.767 INFO:tasks.ceph_fuse:client.0 config is {'mount_subvol_num': 0} 2024-03-19T02:53:06.767 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-19T02:53:06.767 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-19T02:53:06.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:53:06.767 DEBUG:teuthology.orchestra.run.smithi033:> ip netns list 2024-03-19T02:53:06.782 INFO:teuthology.orchestra.run.smithi033.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-19T02:53:06.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:53:06.783 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-19T02:53:06.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:53:06.849 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip link delete ceph-brx 2024-03-19T02:53:06.946 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-19T02:53:06.947 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-19T02:53:06.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:06 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3793598099' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:53:06.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:06 smithi096 ceph-mon[35946]: pgmap v370: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:53:06.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:06 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3722672193' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:53:06.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:06 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3500053136' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:53:07.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:06 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3793598099' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:53:07.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:06 smithi155 ceph-mon[35804]: pgmap v370: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:53:07.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:06 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3722672193' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:53:07.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:06 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3500053136' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:53:07.194 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T02:53:08.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:07 smithi033 ceph-mon[31973]: pgmap v371: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:53:08.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:07 smithi096 ceph-mon[35946]: pgmap v371: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:53:08.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:07 smithi155 ceph-mon[35804]: pgmap v371: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:53:08.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.863+0000 7f9619715700 1 -- 172.21.15.33:0/575021578 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9614101e50 msgr2=0x7f9614105e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:08.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.863+0000 7f9619715700 1 --2- 172.21.15.33:0/575021578 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9614101e50 0x7f9614105e40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f9608009990 tx=0x7f9608009ca0 comp rx=0 tx=0).stop 2024-03-19T02:53:08.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.863+0000 7f9619715700 1 -- 172.21.15.33:0/575021578 shutdown_connections 2024-03-19T02:53:08.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.863+0000 7f9619715700 1 --2- 172.21.15.33:0/575021578 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9614106520 0x7f961410e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:08.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.863+0000 7f9619715700 1 --2- 172.21.15.33:0/575021578 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96141014a0 0x7f9614101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:08.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.863+0000 7f9619715700 1 --2- 172.21.15.33:0/575021578 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9614101e50 0x7f9614105e40 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:08.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.863+0000 7f9619715700 1 -- 172.21.15.33:0/575021578 >> 172.21.15.33:0/575021578 conn(0x7f9614077250 msgr2=0x7f9614077660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:08.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.863+0000 7f9619715700 1 -- 172.21.15.33:0/575021578 shutdown_connections 2024-03-19T02:53:08.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.863+0000 7f9619715700 1 -- 172.21.15.33:0/575021578 wait complete. 2024-03-19T02:53:08.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.864+0000 7f9619715700 1 Processor -- start 2024-03-19T02:53:08.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.864+0000 7f9619715700 1 -- start start 2024-03-19T02:53:08.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.864+0000 7f9619715700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96141014a0 0x7f961419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:08.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.864+0000 7f9619715700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9614101e50 0x7f961419ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:08.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.864+0000 7f9619715700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9614106520 0x7f9614198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:08.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.864+0000 7f9619715700 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9614111390 con 0x7f9614101e50 2024-03-19T02:53:08.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.864+0000 7f9619715700 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9614111210 con 0x7f96141014a0 2024-03-19T02:53:08.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.864+0000 7f9619715700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9614111510 con 0x7f9614106520 2024-03-19T02:53:08.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.864+0000 7f9612ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96141014a0 0x7f961419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:08.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.864+0000 7f96127fc700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9614101e50 0x7f961419ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:08.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.864+0000 7f96127fc700 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9614101e50 0x7f961419ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46542/0 (socket says 172.21.15.33:46542) 2024-03-19T02:53:08.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.864+0000 7f9612ffd700 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96141014a0 0x7f961419e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:36726/0 (socket says 172.21.15.33:36726) 2024-03-19T02:53:08.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.865+0000 7f96127fc700 1 -- 172.21.15.33:0/3017243696 learned_addr learned my addr 172.21.15.33:0/3017243696 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:08.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.865+0000 7f96137fe700 1 --2- 172.21.15.33:0/3017243696 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9614106520 0x7f9614198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:08.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.865+0000 7f96127fc700 1 -- 172.21.15.33:0/3017243696 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9614106520 msgr2=0x7f9614198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:08.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.865+0000 7f96127fc700 1 --2- 172.21.15.33:0/3017243696 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9614106520 0x7f9614198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:08.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.865+0000 7f96127fc700 1 -- 172.21.15.33:0/3017243696 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96141014a0 msgr2=0x7f961419e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:08.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.865+0000 7f96127fc700 1 --2- 172.21.15.33:0/3017243696 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96141014a0 0x7f961419e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:08.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.865+0000 7f96127fc700 1 -- 172.21.15.33:0/3017243696 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9614199000 con 0x7f9614101e50 2024-03-19T02:53:08.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.865+0000 7f96127fc700 1 --2- 172.21.15.33:0/3017243696 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9614101e50 0x7f961419ebe0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f9608000c00 tx=0x7f9608004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:08.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.865+0000 7f95fbfff700 1 -- 172.21.15.33:0/3017243696 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9608027070 con 0x7f9614101e50 2024-03-19T02:53:08.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.865+0000 7f95fbfff700 1 -- 172.21.15.33:0/3017243696 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9608022470 con 0x7f9614101e50 2024-03-19T02:53:08.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.865+0000 7f9619715700 1 -- 172.21.15.33:0/3017243696 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9614199290 con 0x7f9614101e50 2024-03-19T02:53:08.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.865+0000 7f95fbfff700 1 -- 172.21.15.33:0/3017243696 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f960800b610 con 0x7f9614101e50 2024-03-19T02:53:08.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.865+0000 7f9619715700 1 -- 172.21.15.33:0/3017243696 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96141a5500 con 0x7f9614101e50 2024-03-19T02:53:08.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.866+0000 7f9619715700 1 -- 172.21.15.33:0/3017243696 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9614102920 con 0x7f9614101e50 2024-03-19T02:53:08.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.867+0000 7f95fbfff700 1 -- 172.21.15.33:0/3017243696 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f960802a070 con 0x7f9614101e50 2024-03-19T02:53:08.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.868+0000 7f95fbfff700 1 --2- 172.21.15.33:0/3017243696 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f95fc0a2910 0x7f95fc0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:08.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.868+0000 7f9612ffd700 1 --2- 172.21.15.33:0/3017243696 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f95fc0a2910 0x7f95fc0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:08.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.868+0000 7f95fbfff700 1 -- 172.21.15.33:0/3017243696 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f96080cfde0 con 0x7f9614101e50 2024-03-19T02:53:08.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.869+0000 7f9612ffd700 1 --2- 172.21.15.33:0/3017243696 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f95fc0a2910 0x7f95fc0a4dd0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9604009c60 tx=0x7f9604009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:08.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:08.872+0000 7f95fbfff700 1 -- 172.21.15.33:0/3017243696 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9608013070 con 0x7f9614101e50 2024-03-19T02:53:09.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:09.123+0000 7f9619715700 1 -- 172.21.15.33:0/3017243696 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f9614199d50 con 0x7f9614101e50 2024-03-19T02:53:09.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:09.123+0000 7f95fbfff700 1 -- 172.21.15.33:0/3017243696 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) ==== 53+0+88 (secure 0 0 0) 0x7f960809cc00 con 0x7f9614101e50 2024-03-19T02:53:09.125 INFO:teuthology.orchestra.run.smithi033.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-19T02:53:09.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:09.125+0000 7f9619715700 1 -- 172.21.15.33:0/3017243696 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f95fc0a2910 msgr2=0x7f95fc0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:09.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:09.125+0000 7f9619715700 1 --2- 172.21.15.33:0/3017243696 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f95fc0a2910 0x7f95fc0a4dd0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9604009c60 tx=0x7f9604009260 comp rx=0 tx=0).stop 2024-03-19T02:53:09.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:09.125+0000 7f9619715700 1 -- 172.21.15.33:0/3017243696 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9614101e50 msgr2=0x7f961419ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:09.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:09.125+0000 7f9619715700 1 --2- 172.21.15.33:0/3017243696 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9614101e50 0x7f961419ebe0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f9608000c00 tx=0x7f9608004790 comp rx=0 tx=0).stop 2024-03-19T02:53:09.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:09.126+0000 7f9619715700 1 -- 172.21.15.33:0/3017243696 shutdown_connections 2024-03-19T02:53:09.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:09.126+0000 7f9619715700 1 --2- 172.21.15.33:0/3017243696 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f95fc0a2910 0x7f95fc0a4dd0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:09.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:09.126+0000 7f9619715700 1 --2- 172.21.15.33:0/3017243696 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9614106520 0x7f9614198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:09.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:09.126+0000 7f9619715700 1 --2- 172.21.15.33:0/3017243696 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96141014a0 0x7f961419e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:09.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:09.126+0000 7f9619715700 1 --2- 172.21.15.33:0/3017243696 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9614101e50 0x7f961419ebe0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:09.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:09.126+0000 7f9619715700 1 -- 172.21.15.33:0/3017243696 >> 172.21.15.33:0/3017243696 conn(0x7f9614077250 msgr2=0x7f961406aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:09.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:09.126+0000 7f9619715700 1 -- 172.21.15.33:0/3017243696 shutdown_connections 2024-03-19T02:53:09.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:09.126+0000 7f9619715700 1 -- 172.21.15.33:0/3017243696 wait complete. 2024-03-19T02:53:09.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:09 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3017243696' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-19T02:53:09.482 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:09 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3017243696' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-19T02:53:09.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:09 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3017243696' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-19T02:53:09.827 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-19T02:53:09.827 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-19T02:53:09.827 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi033.front.sepia.ceph.com 2024-03-19T02:53:09.828 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-19T02:53:09.828 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-19T02:53:09.828 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-19T02:53:09.828 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-19T02:53:09.828 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-19T02:53:09.828 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-19T02:53:09.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:53:09.828 DEBUG:teuthology.orchestra.run.smithi033:> ip addr 2024-03-19T02:53:09.845 INFO:teuthology.orchestra.run.smithi033.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-19T02:53:09.845 INFO:teuthology.orchestra.run.smithi033.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-19T02:53:09.845 INFO:teuthology.orchestra.run.smithi033.stdout: inet 127.0.0.1/8 scope host lo 2024-03-19T02:53:09.845 INFO:teuthology.orchestra.run.smithi033.stdout: valid_lft forever preferred_lft forever 2024-03-19T02:53:09.845 INFO:teuthology.orchestra.run.smithi033.stdout: inet6 ::1/128 scope host 2024-03-19T02:53:09.845 INFO:teuthology.orchestra.run.smithi033.stdout: valid_lft forever preferred_lft forever 2024-03-19T02:53:09.845 INFO:teuthology.orchestra.run.smithi033.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-19T02:53:09.845 INFO:teuthology.orchestra.run.smithi033.stdout: link/ether 0c:c4:7a:6b:fd:14 brd ff:ff:ff:ff:ff:ff 2024-03-19T02:53:09.845 INFO:teuthology.orchestra.run.smithi033.stdout: altname enp7s0f0 2024-03-19T02:53:09.845 INFO:teuthology.orchestra.run.smithi033.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-19T02:53:09.846 INFO:teuthology.orchestra.run.smithi033.stdout: link/ether 0c:c4:7a:6b:fd:15 brd ff:ff:ff:ff:ff:ff 2024-03-19T02:53:09.846 INFO:teuthology.orchestra.run.smithi033.stdout: altname enp7s0f1 2024-03-19T02:53:09.846 INFO:teuthology.orchestra.run.smithi033.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-19T02:53:09.846 INFO:teuthology.orchestra.run.smithi033.stdout: link/ether ac:1f:6b:a5:d2:94 brd ff:ff:ff:ff:ff:ff 2024-03-19T02:53:09.846 INFO:teuthology.orchestra.run.smithi033.stdout: altname enp1s0f0 2024-03-19T02:53:09.846 INFO:teuthology.orchestra.run.smithi033.stdout: inet 172.21.15.33/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-19T02:53:09.846 INFO:teuthology.orchestra.run.smithi033.stdout: valid_lft 41438sec preferred_lft 41438sec 2024-03-19T02:53:09.846 INFO:teuthology.orchestra.run.smithi033.stdout: inet6 fe80::ae1f:6bff:fea5:d294/64 scope link noprefixroute 2024-03-19T02:53:09.846 INFO:teuthology.orchestra.run.smithi033.stdout: valid_lft forever preferred_lft forever 2024-03-19T02:53:09.846 INFO:teuthology.orchestra.run.smithi033.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-19T02:53:09.846 INFO:teuthology.orchestra.run.smithi033.stdout: link/ether ac:1f:6b:a5:d2:95 brd ff:ff:ff:ff:ff:ff 2024-03-19T02:53:09.846 INFO:teuthology.orchestra.run.smithi033.stdout: altname enp1s0f1 2024-03-19T02:53:09.847 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-19T02:53:09.847 DEBUG:teuthology.orchestra.run.smithi033:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T02:53:09.847 DEBUG:teuthology.orchestra.run.smithi033:> set -e 2024-03-19T02:53:09.847 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip link add name ceph-brx type bridge 2024-03-19T02:53:09.847 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip addr flush dev ceph-brx 2024-03-19T02:53:09.847 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip link set ceph-brx up 2024-03-19T02:53:09.847 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-19T02:53:09.847 DEBUG:teuthology.orchestra.run.smithi033:> ') 2024-03-19T02:53:09.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:09 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T02:53:09.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:53:09.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:53:09.997 DEBUG:teuthology.orchestra.run.smithi033:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-19T02:53:10.023 INFO:teuthology.orchestra.run.smithi033.stdout:1 2024-03-19T02:53:10.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:53:10.025 DEBUG:teuthology.orchestra.run.smithi033:> route 2024-03-19T02:53:10.081 INFO:teuthology.orchestra.run.smithi033.stdout:Kernel IP routing table 2024-03-19T02:53:10.081 INFO:teuthology.orchestra.run.smithi033.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-19T02:53:10.082 INFO:teuthology.orchestra.run.smithi033.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-19T02:53:10.082 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-19T02:53:10.082 INFO:teuthology.orchestra.run.smithi033.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-19T02:53:10.082 DEBUG:teuthology.orchestra.run.smithi033:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T02:53:10.082 DEBUG:teuthology.orchestra.run.smithi033:> set -e 2024-03-19T02:53:10.082 DEBUG:teuthology.orchestra.run.smithi033:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-19T02:53:10.082 DEBUG:teuthology.orchestra.run.smithi033:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-19T02:53:10.082 DEBUG:teuthology.orchestra.run.smithi033:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-19T02:53:10.082 DEBUG:teuthology.orchestra.run.smithi033:> ') 2024-03-19T02:53:10.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T02:53:10.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:53:10.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:53:10.218 DEBUG:teuthology.orchestra.run.smithi033:> ip netns list 2024-03-19T02:53:10.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:53:10.233 DEBUG:teuthology.orchestra.run.smithi033:> ip netns list-id 2024-03-19T02:53:10.287 DEBUG:teuthology.orchestra.run.smithi033:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T02:53:10.288 DEBUG:teuthology.orchestra.run.smithi033:> set -e 2024-03-19T02:53:10.288 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T02:53:10.288 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-19T02:53:10.288 DEBUG:teuthology.orchestra.run.smithi033:> ') 2024-03-19T02:53:10.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T02:53:10.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:53:10.410 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-19T02:53:10.410 DEBUG:teuthology.orchestra.run.smithi033:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T02:53:10.410 DEBUG:teuthology.orchestra.run.smithi033:> set -e 2024-03-19T02:53:10.411 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-19T02:53:10.411 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-19T02:53:10.411 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-19T02:53:10.411 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-19T02:53:10.411 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-19T02:53:10.411 DEBUG:teuthology.orchestra.run.smithi033:> ') 2024-03-19T02:53:10.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T02:53:10.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:10 smithi096 ceph-mon[35946]: pgmap v372: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-19T02:53:10.469 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:10 smithi033 ceph-mon[31973]: pgmap v372: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-19T02:53:10.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:10 smithi155 ceph-mon[35804]: pgmap v372: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-19T02:53:10.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:53:10.537 DEBUG:teuthology.orchestra.run.smithi033:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T02:53:10.537 DEBUG:teuthology.orchestra.run.smithi033:> set -e 2024-03-19T02:53:10.537 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip link set brx.0 up 2024-03-19T02:53:10.537 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip link set dev brx.0 master ceph-brx 2024-03-19T02:53:10.537 DEBUG:teuthology.orchestra.run.smithi033:> ') 2024-03-19T02:53:10.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T02:53:10.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:53:10.619 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'mount_subvol_num': 0} 2024-03-19T02:53:10.619 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-19T02:53:10.619 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-19T02:53:10.633 INFO:teuthology.orchestra.run.smithi033.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-19T02:53:10.633 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-19T02:53:10.633 DEBUG:teuthology.orchestra.run.smithi033:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-19T02:53:10.687 DEBUG:teuthology.orchestra.run.smithi033:> sudo modprobe fuse 2024-03-19T02:53:10.755 DEBUG:teuthology.orchestra.run.smithi033:> cat /proc/self/mounts | awk '{print $2}' 2024-03-19T02:53:10.810 INFO:teuthology.orchestra.run.smithi033.stdout:/proc 2024-03-19T02:53:10.810 INFO:teuthology.orchestra.run.smithi033.stdout:/sys 2024-03-19T02:53:10.810 INFO:teuthology.orchestra.run.smithi033.stdout:/dev 2024-03-19T02:53:10.810 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/kernel/security 2024-03-19T02:53:10.810 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/shm 2024-03-19T02:53:10.810 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/pts 2024-03-19T02:53:10.810 INFO:teuthology.orchestra.run.smithi033.stdout:/run 2024-03-19T02:53:10.810 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/fs/cgroup 2024-03-19T02:53:10.811 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/fs/pstore 2024-03-19T02:53:10.811 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/fs/bpf 2024-03-19T02:53:10.811 INFO:teuthology.orchestra.run.smithi033.stdout:/ 2024-03-19T02:53:10.811 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/fs/selinux 2024-03-19T02:53:10.811 INFO:teuthology.orchestra.run.smithi033.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T02:53:10.811 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/mqueue 2024-03-19T02:53:10.811 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/hugepages 2024-03-19T02:53:10.811 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/kernel/debug 2024-03-19T02:53:10.811 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/kernel/tracing 2024-03-19T02:53:10.811 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/kernel/config 2024-03-19T02:53:10.811 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/fs/fuse/connections 2024-03-19T02:53:10.811 INFO:teuthology.orchestra.run.smithi033.stdout:/run/credentials/systemd-sysctl.service 2024-03-19T02:53:10.811 INFO:teuthology.orchestra.run.smithi033.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-19T02:53:10.811 INFO:teuthology.orchestra.run.smithi033.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-19T02:53:10.812 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/nfs/rpc_pipefs 2024-03-19T02:53:10.812 INFO:teuthology.orchestra.run.smithi033.stdout:/run/user/1000 2024-03-19T02:53:10.812 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/ceph 2024-03-19T02:53:10.812 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay 2024-03-19T02:53:10.812 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/d40815713b6d8abf13551e5b03e4726efec7d920162aacc3167eb822ccd4f341/merged 2024-03-19T02:53:10.812 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/f26473e65fe17f5cc9a064aa0a0705b7cf6cdfb5557e691edb195c421fe9311e/merged 2024-03-19T02:53:10.812 INFO:teuthology.orchestra.run.smithi033.stdout:/run/user/0 2024-03-19T02:53:10.812 INFO:teuthology.orchestra.run.smithi033.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T02:53:10.812 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/458cde7783a31ad935025ecce83257fe525e84dfc943bc5ddf3711c44dc9cb47/merged 2024-03-19T02:53:10.812 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/4781736ffa71eef74ba07d8e57fa30a2a7dc9c2ea614310d94ac0a0ba14c808c/merged 2024-03-19T02:53:10.812 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/a6b836646400b0531dc3e7adb38a6372d800021d4cfe7dcdc74b23fb4bff575d/merged 2024-03-19T02:53:10.812 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/a0b05b7ceab9ec0d57bb302cc867219b12bfe23a430eb417e4c02683d743e203/merged 2024-03-19T02:53:10.812 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/264ba1fe315f2d024742741205d4d32695fa53dc0074ea52b5dac59a93dcc6b7/merged 2024-03-19T02:53:10.813 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/a592735a9ef96a00521c4d49ff5be4085daca00927f58bcae50da2dd3c634979/merged 2024-03-19T02:53:10.813 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/547c2dfe92bf1ab04750857a84566c04e128f7d6f517b355478a8dfa63b23f88/merged 2024-03-19T02:53:10.813 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/4613b7ef0ee9acd7aadf0117330f3af918303ea30b265b69578eb1c58954e1b9/merged 2024-03-19T02:53:10.813 INFO:teuthology.orchestra.run.smithi033.stdout:/run/netns 2024-03-19T02:53:10.813 INFO:teuthology.orchestra.run.smithi033.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T02:53:10.813 INFO:teuthology.orchestra.run.smithi033.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T02:53:10.813 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:53:10.813 DEBUG:teuthology.orchestra.run.smithi033:> ls /sys/fs/fuse/connections 2024-03-19T02:53:10.865 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-19T02:53:10.865 DEBUG:teuthology.orchestra.run.smithi033:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/volumes/qa/sv_0/14730696-c69f-4c28-ba34-e177c7403ad4 --client_fs=cephfs) 2024-03-19T02:53:10.907 DEBUG:teuthology.orchestra.run.smithi033:> sudo modprobe fuse 2024-03-19T02:53:10.934 DEBUG:teuthology.orchestra.run.smithi033:> cat /proc/self/mounts | awk '{print $2}' 2024-03-19T02:53:10.989 INFO:teuthology.orchestra.run.smithi033.stdout:/proc 2024-03-19T02:53:10.989 INFO:teuthology.orchestra.run.smithi033.stdout:/sys 2024-03-19T02:53:10.989 INFO:teuthology.orchestra.run.smithi033.stdout:/dev 2024-03-19T02:53:10.989 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/kernel/security 2024-03-19T02:53:10.989 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/shm 2024-03-19T02:53:10.989 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/pts 2024-03-19T02:53:10.989 INFO:teuthology.orchestra.run.smithi033.stdout:/run 2024-03-19T02:53:10.989 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/fs/cgroup 2024-03-19T02:53:10.989 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/fs/pstore 2024-03-19T02:53:10.989 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/fs/bpf 2024-03-19T02:53:10.990 INFO:teuthology.orchestra.run.smithi033.stdout:/ 2024-03-19T02:53:10.990 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/fs/selinux 2024-03-19T02:53:10.990 INFO:teuthology.orchestra.run.smithi033.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T02:53:10.990 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/mqueue 2024-03-19T02:53:10.990 INFO:teuthology.orchestra.run.smithi033.stdout:/dev/hugepages 2024-03-19T02:53:10.990 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/kernel/debug 2024-03-19T02:53:10.990 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/kernel/tracing 2024-03-19T02:53:10.990 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/kernel/config 2024-03-19T02:53:10.990 INFO:teuthology.orchestra.run.smithi033.stdout:/sys/fs/fuse/connections 2024-03-19T02:53:10.990 INFO:teuthology.orchestra.run.smithi033.stdout:/run/credentials/systemd-sysctl.service 2024-03-19T02:53:10.990 INFO:teuthology.orchestra.run.smithi033.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-19T02:53:10.990 INFO:teuthology.orchestra.run.smithi033.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-19T02:53:10.990 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/nfs/rpc_pipefs 2024-03-19T02:53:10.990 INFO:teuthology.orchestra.run.smithi033.stdout:/run/user/1000 2024-03-19T02:53:10.991 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/ceph 2024-03-19T02:53:10.991 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay 2024-03-19T02:53:10.991 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/d40815713b6d8abf13551e5b03e4726efec7d920162aacc3167eb822ccd4f341/merged 2024-03-19T02:53:10.991 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/f26473e65fe17f5cc9a064aa0a0705b7cf6cdfb5557e691edb195c421fe9311e/merged 2024-03-19T02:53:10.991 INFO:teuthology.orchestra.run.smithi033.stdout:/run/user/0 2024-03-19T02:53:10.991 INFO:teuthology.orchestra.run.smithi033.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T02:53:10.991 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/458cde7783a31ad935025ecce83257fe525e84dfc943bc5ddf3711c44dc9cb47/merged 2024-03-19T02:53:10.991 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/4781736ffa71eef74ba07d8e57fa30a2a7dc9c2ea614310d94ac0a0ba14c808c/merged 2024-03-19T02:53:10.991 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/a6b836646400b0531dc3e7adb38a6372d800021d4cfe7dcdc74b23fb4bff575d/merged 2024-03-19T02:53:10.991 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/a0b05b7ceab9ec0d57bb302cc867219b12bfe23a430eb417e4c02683d743e203/merged 2024-03-19T02:53:10.991 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/264ba1fe315f2d024742741205d4d32695fa53dc0074ea52b5dac59a93dcc6b7/merged 2024-03-19T02:53:10.991 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/a592735a9ef96a00521c4d49ff5be4085daca00927f58bcae50da2dd3c634979/merged 2024-03-19T02:53:10.991 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/547c2dfe92bf1ab04750857a84566c04e128f7d6f517b355478a8dfa63b23f88/merged 2024-03-19T02:53:10.991 INFO:teuthology.orchestra.run.smithi033.stdout:/var/lib/containers/storage/overlay/4613b7ef0ee9acd7aadf0117330f3af918303ea30b265b69578eb1c58954e1b9/merged 2024-03-19T02:53:10.991 INFO:teuthology.orchestra.run.smithi033.stdout:/run/netns 2024-03-19T02:53:10.992 INFO:teuthology.orchestra.run.smithi033.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T02:53:10.992 INFO:teuthology.orchestra.run.smithi033.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T02:53:10.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:53:10.992 DEBUG:teuthology.orchestra.run.smithi033:> ls /sys/fs/fuse/connections 2024-03-19T02:53:10.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi033.stderr:2024-03-19T02:53:10.993+0000 7f329e741480 -1 init, newargv = 0x55d6fd925260 newargc=15 2024-03-19T02:53:10.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi033.stderr:ceph-fuse[85913]: starting ceph client 2024-03-19T02:53:11.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi033.stderr:ceph-fuse[85913]: starting fuse 2024-03-19T02:53:11.009 INFO:teuthology.orchestra.run.smithi033.stdout:119 2024-03-19T02:53:11.009 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-03-19T02:53:11.009 DEBUG:teuthology.orchestra.run.smithi033:> sudo stdin-killer -- python3 -c ' 2024-03-19T02:53:11.009 DEBUG:teuthology.orchestra.run.smithi033:> import glob 2024-03-19T02:53:11.009 DEBUG:teuthology.orchestra.run.smithi033:> import re 2024-03-19T02:53:11.009 DEBUG:teuthology.orchestra.run.smithi033:> import os 2024-03-19T02:53:11.010 DEBUG:teuthology.orchestra.run.smithi033:> import subprocess 2024-03-19T02:53:11.010 DEBUG:teuthology.orchestra.run.smithi033:> 2024-03-19T02:53:11.010 DEBUG:teuthology.orchestra.run.smithi033:> def _find_admin_socket(client_name): 2024-03-19T02:53:11.010 DEBUG:teuthology.orchestra.run.smithi033:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-19T02:53:11.010 DEBUG:teuthology.orchestra.run.smithi033:> files = glob.glob(asok_path) 2024-03-19T02:53:11.010 DEBUG:teuthology.orchestra.run.smithi033:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-19T02:53:11.010 DEBUG:teuthology.orchestra.run.smithi033:> 2024-03-19T02:53:11.010 DEBUG:teuthology.orchestra.run.smithi033:> # Given a non-glob path, it better be there 2024-03-19T02:53:11.010 DEBUG:teuthology.orchestra.run.smithi033:> if "*" not in asok_path: 2024-03-19T02:53:11.010 DEBUG:teuthology.orchestra.run.smithi033:> assert(len(files) == 1) 2024-03-19T02:53:11.010 DEBUG:teuthology.orchestra.run.smithi033:> return files[0] 2024-03-19T02:53:11.010 DEBUG:teuthology.orchestra.run.smithi033:> 2024-03-19T02:53:11.010 DEBUG:teuthology.orchestra.run.smithi033:> for f in files: 2024-03-19T02:53:11.010 DEBUG:teuthology.orchestra.run.smithi033:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-19T02:53:11.010 DEBUG:teuthology.orchestra.run.smithi033:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-19T02:53:11.011 DEBUG:teuthology.orchestra.run.smithi033:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-19T02:53:11.011 DEBUG:teuthology.orchestra.run.smithi033:> contents = proc_f.read() 2024-03-19T02:53:11.011 DEBUG:teuthology.orchestra.run.smithi033:> if mountpoint in contents: 2024-03-19T02:53:11.011 DEBUG:teuthology.orchestra.run.smithi033:> return f 2024-03-19T02:53:11.011 DEBUG:teuthology.orchestra.run.smithi033:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-19T02:53:11.011 DEBUG:teuthology.orchestra.run.smithi033:> 2024-03-19T02:53:11.011 DEBUG:teuthology.orchestra.run.smithi033:> print(_find_admin_socket("client.0")) 2024-03-19T02:53:11.011 DEBUG:teuthology.orchestra.run.smithi033:> ' 2024-03-19T02:53:11.131 INFO:teuthology.orchestra.run.smithi033.stdout:/var/run/ceph/ceph-client.0.85913.asok 2024-03-19T02:53:11.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T02:53:11.141 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.85913.asok 2024-03-19T02:53:11.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:53:11.142 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.85913.asok status 2024-03-19T02:53:11.279 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:53:11.280 INFO:teuthology.orchestra.run.smithi033.stdout: "metadata": { 2024-03-19T02:53:11.280 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_sha1": "b6ae24918b03b1b8a19c9263a50655bba80bb810", 2024-03-19T02:53:11.280 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_version": "ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev)", 2024-03-19T02:53:11.280 INFO:teuthology.orchestra.run.smithi033.stdout: "entity_id": "0", 2024-03-19T02:53:11.280 INFO:teuthology.orchestra.run.smithi033.stdout: "hostname": "smithi033", 2024-03-19T02:53:11.280 INFO:teuthology.orchestra.run.smithi033.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-19T02:53:11.280 INFO:teuthology.orchestra.run.smithi033.stdout: "pid": "85913", 2024-03-19T02:53:11.280 INFO:teuthology.orchestra.run.smithi033.stdout: "root": "/volumes/qa/sv_0/14730696-c69f-4c28-ba34-e177c7403ad4" 2024-03-19T02:53:11.280 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T02:53:11.280 INFO:teuthology.orchestra.run.smithi033.stdout: "dentry_count": 0, 2024-03-19T02:53:11.280 INFO:teuthology.orchestra.run.smithi033.stdout: "dentry_pinned_count": 0, 2024-03-19T02:53:11.281 INFO:teuthology.orchestra.run.smithi033.stdout: "id": 15426, 2024-03-19T02:53:11.281 INFO:teuthology.orchestra.run.smithi033.stdout: "inst": { 2024-03-19T02:53:11.281 INFO:teuthology.orchestra.run.smithi033.stdout: "name": { 2024-03-19T02:53:11.281 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "client", 2024-03-19T02:53:11.281 INFO:teuthology.orchestra.run.smithi033.stdout: "num": 15426 2024-03-19T02:53:11.281 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T02:53:11.281 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": { 2024-03-19T02:53:11.281 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-03-19T02:53:11.281 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "192.168.0.1:0", 2024-03-19T02:53:11.281 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 975155070 2024-03-19T02:53:11.281 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T02:53:11.281 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T02:53:11.281 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": { 2024-03-19T02:53:11.281 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-03-19T02:53:11.281 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "192.168.0.1:0", 2024-03-19T02:53:11.282 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 975155070 2024-03-19T02:53:11.282 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T02:53:11.282 INFO:teuthology.orchestra.run.smithi033.stdout: "inst_str": "client.15426 192.168.0.1:0/975155070", 2024-03-19T02:53:11.282 INFO:teuthology.orchestra.run.smithi033.stdout: "addr_str": "192.168.0.1:0/975155070", 2024-03-19T02:53:11.282 INFO:teuthology.orchestra.run.smithi033.stdout: "inode_count": 5, 2024-03-19T02:53:11.282 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_epoch": 24, 2024-03-19T02:53:11.282 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_epoch": 97, 2024-03-19T02:53:11.282 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_epoch_barrier": 0, 2024-03-19T02:53:11.282 INFO:teuthology.orchestra.run.smithi033.stdout: "blocklisted": false, 2024-03-19T02:53:11.282 INFO:teuthology.orchestra.run.smithi033.stdout: "fs_name": "cephfs" 2024-03-19T02:53:11.282 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:53:11.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:53:11.288 DEBUG:teuthology.orchestra.run.smithi033:> stat --file-system '--printf=%T 2024-03-19T02:53:11.288 DEBUG:teuthology.orchestra.run.smithi033:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-19T02:53:11.303 INFO:teuthology.orchestra.run.smithi033.stdout:fuseblk 2024-03-19T02:53:11.303 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-19T02:53:11.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:53:11.303 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-19T02:53:11.371 INFO:teuthology.run_tasks:Running task exec... 2024-03-19T02:53:11.383 INFO:teuthology.task.exec:Executing custom commands... 2024-03-19T02:53:11.383 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi033.front.sepia.ceph.com 2024-03-19T02:53:11.383 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-03-19T02:53:11.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.513+0000 7f9fffde6640 1 -- 172.21.15.33:0/4164054157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ff8100790 msgr2=0x7f9ff8100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:11.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.513+0000 7f9fffde6640 1 --2- 172.21.15.33:0/4164054157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ff8100790 0x7f9ff8100bb0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f9ff4009900 tx=0x7f9ff402bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:11.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.513+0000 7f9fffde6640 1 -- 172.21.15.33:0/4164054157 shutdown_connections 2024-03-19T02:53:11.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.513+0000 7f9fffde6640 1 --2- 172.21.15.33:0/4164054157 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ff8101980 0x7f9ff8101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:11.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.513+0000 7f9fffde6640 1 --2- 172.21.15.33:0/4164054157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ff8102340 0x7f9ff8108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:11.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.513+0000 7f9fffde6640 1 --2- 172.21.15.33:0/4164054157 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ff8100790 0x7f9ff8100bb0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:11.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.513+0000 7f9fffde6640 1 -- 172.21.15.33:0/4164054157 >> 172.21.15.33:0/4164054157 conn(0x7f9ff80fbd00 msgr2=0x7f9ff80fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:11.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.513+0000 7f9fffde6640 1 -- 172.21.15.33:0/4164054157 shutdown_connections 2024-03-19T02:53:11.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.513+0000 7f9fffde6640 1 -- 172.21.15.33:0/4164054157 wait complete. 2024-03-19T02:53:11.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.514+0000 7f9fffde6640 1 Processor -- start 2024-03-19T02:53:11.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.514+0000 7f9fffde6640 1 -- start start 2024-03-19T02:53:11.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.514+0000 7f9fffde6640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ff8100790 0x7f9ff8197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:11.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.514+0000 7f9fffde6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ff8101980 0x7f9ff8197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:11.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.514+0000 7f9fffde6640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ff8102340 0x7f9ff819efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:11.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.514+0000 7f9fffde6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff810c2b0 con 0x7f9ff8101980 2024-03-19T02:53:11.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.514+0000 7f9fffde6640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff810c130 con 0x7f9ff8100790 2024-03-19T02:53:11.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.514+0000 7f9fffde6640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff810c430 con 0x7f9ff8102340 2024-03-19T02:53:11.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.514+0000 7f9ffd381640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ff8101980 0x7f9ff8197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:11.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.514+0000 7f9ffd381640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ff8101980 0x7f9ff8197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46560/0 (socket says 172.21.15.33:46560) 2024-03-19T02:53:11.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.514+0000 7f9ffd381640 1 -- 172.21.15.33:0/1392246336 learned_addr learned my addr 172.21.15.33:0/1392246336 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:11.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.514+0000 7f9ffdb82640 1 --2- 172.21.15.33:0/1392246336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ff8100790 0x7f9ff8197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:11.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.514+0000 7f9ffe383640 1 --2- 172.21.15.33:0/1392246336 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ff8102340 0x7f9ff819efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:11.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.514+0000 7f9ffd381640 1 -- 172.21.15.33:0/1392246336 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ff8102340 msgr2=0x7f9ff819efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:11.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.514+0000 7f9ffd381640 1 --2- 172.21.15.33:0/1392246336 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ff8102340 0x7f9ff819efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:11.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.514+0000 7f9ffd381640 1 -- 172.21.15.33:0/1392246336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ff8100790 msgr2=0x7f9ff8197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:11.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.514+0000 7f9ffd381640 1 --2- 172.21.15.33:0/1392246336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ff8100790 0x7f9ff8197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:11.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.514+0000 7f9ffd381640 1 -- 172.21.15.33:0/1392246336 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ff819f640 con 0x7f9ff8101980 2024-03-19T02:53:11.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.514+0000 7f9ffdb82640 1 --2- 172.21.15.33:0/1392246336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ff8100790 0x7f9ff8197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:11.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.514+0000 7f9ffd381640 1 --2- 172.21.15.33:0/1392246336 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ff8101980 0x7f9ff8197f60 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f9fe800e9f0 tx=0x7f9fe800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:11.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.515+0000 7f9feeffd640 1 -- 172.21.15.33:0/1392246336 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe800cd80 con 0x7f9ff8101980 2024-03-19T02:53:11.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.515+0000 7f9feeffd640 1 -- 172.21.15.33:0/1392246336 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9fe80054d0 con 0x7f9ff8101980 2024-03-19T02:53:11.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.515+0000 7f9feeffd640 1 -- 172.21.15.33:0/1392246336 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe8010490 con 0x7f9ff8101980 2024-03-19T02:53:11.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.515+0000 7f9fffde6640 1 -- 172.21.15.33:0/1392246336 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ff819f930 con 0x7f9ff8101980 2024-03-19T02:53:11.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.515+0000 7f9fffde6640 1 -- 172.21.15.33:0/1392246336 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ff819fe30 con 0x7f9ff8101980 2024-03-19T02:53:11.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.516+0000 7f9fffde6640 1 -- 172.21.15.33:0/1392246336 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ff81a00c0 con 0x7f9ff8101980 2024-03-19T02:53:11.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.517+0000 7f9feeffd640 1 -- 172.21.15.33:0/1392246336 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9fe8005020 con 0x7f9ff8101980 2024-03-19T02:53:11.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.517+0000 7f9feeffd640 1 --2- 172.21.15.33:0/1392246336 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9fe40a2920 0x7f9fe40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:11.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.517+0000 7f9ffdb82640 1 --2- 172.21.15.33:0/1392246336 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9fe40a2920 0x7f9fe40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:11.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.517+0000 7f9feeffd640 1 -- 172.21.15.33:0/1392246336 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f9fe80cf0f0 con 0x7f9ff8101980 2024-03-19T02:53:11.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.518+0000 7f9ffdb82640 1 --2- 172.21.15.33:0/1392246336 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9fe40a2920 0x7f9fe40a4de0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9ff4005ab0 tx=0x7f9ff4005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:11.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.522+0000 7f9feeffd640 1 -- 172.21.15.33:0/1392246336 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9fe809be00 con 0x7f9ff8101980 2024-03-19T02:53:11.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.747+0000 7f9fffde6640 1 -- 172.21.15.33:0/1392246336 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7f9ff81a0290 con 0x7f9ff8101980 2024-03-19T02:53:11.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.798+0000 7f9feeffd640 1 -- 172.21.15.33:0/1392246336 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v25) ==== 117+0+0 (secure 0 0 0) 0x7f9fe809b700 con 0x7f9ff8101980 2024-03-19T02:53:11.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.801+0000 7f9fffde6640 1 -- 172.21.15.33:0/1392246336 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9fe40a2920 msgr2=0x7f9fe40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:11.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.801+0000 7f9fffde6640 1 --2- 172.21.15.33:0/1392246336 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9fe40a2920 0x7f9fe40a4de0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9ff4005ab0 tx=0x7f9ff4005fd0 comp rx=0 tx=0).stop 2024-03-19T02:53:11.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.801+0000 7f9fffde6640 1 -- 172.21.15.33:0/1392246336 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ff8101980 msgr2=0x7f9ff8197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:11.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.801+0000 7f9fffde6640 1 --2- 172.21.15.33:0/1392246336 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ff8101980 0x7f9ff8197f60 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f9fe800e9f0 tx=0x7f9fe800edf0 comp rx=0 tx=0).stop 2024-03-19T02:53:11.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.801+0000 7f9fffde6640 1 -- 172.21.15.33:0/1392246336 shutdown_connections 2024-03-19T02:53:11.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.801+0000 7f9fffde6640 1 --2- 172.21.15.33:0/1392246336 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f9fe40a2920 0x7f9fe40a4de0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:11.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.801+0000 7f9fffde6640 1 --2- 172.21.15.33:0/1392246336 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ff8102340 0x7f9ff819efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:11.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.801+0000 7f9fffde6640 1 --2- 172.21.15.33:0/1392246336 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9ff8101980 0x7f9ff8197f60 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:11.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.801+0000 7f9fffde6640 1 --2- 172.21.15.33:0/1392246336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ff8100790 0x7f9ff8197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:11.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.801+0000 7f9fffde6640 1 -- 172.21.15.33:0/1392246336 >> 172.21.15.33:0/1392246336 conn(0x7f9ff80fbd00 msgr2=0x7f9ff80fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:11.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.801+0000 7f9fffde6640 1 -- 172.21.15.33:0/1392246336 shutdown_connections 2024-03-19T02:53:11.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.801+0000 7f9fffde6640 1 -- 172.21.15.33:0/1392246336 wait complete. 2024-03-19T02:53:11.813 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-19T02:53:11.824 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-19T02:53:11.837 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:53:11.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.939+0000 7f2c9d544640 1 -- 172.21.15.33:0/866213403 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c98100740 msgr2=0x7f2c98100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:11.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.939+0000 7f2c9d544640 1 --2- 172.21.15.33:0/866213403 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c98100740 0x7f2c98100b60 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f2c90009900 tx=0x7f2c9002bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:11.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.939+0000 7f2c9d544640 1 -- 172.21.15.33:0/866213403 shutdown_connections 2024-03-19T02:53:11.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.939+0000 7f2c9d544640 1 --2- 172.21.15.33:0/866213403 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c981022f0 0x7f2c98108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:11.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.939+0000 7f2c9d544640 1 --2- 172.21.15.33:0/866213403 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c98101930 0x7f2c98101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:11.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.939+0000 7f2c9d544640 1 --2- 172.21.15.33:0/866213403 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c98100740 0x7f2c98100b60 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:11.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.939+0000 7f2c9d544640 1 -- 172.21.15.33:0/866213403 >> 172.21.15.33:0/866213403 conn(0x7f2c980fbcb0 msgr2=0x7f2c980fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:11.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.939+0000 7f2c9d544640 1 -- 172.21.15.33:0/866213403 shutdown_connections 2024-03-19T02:53:11.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.939+0000 7f2c9d544640 1 -- 172.21.15.33:0/866213403 wait complete. 2024-03-19T02:53:11.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.939+0000 7f2c9d544640 1 Processor -- start 2024-03-19T02:53:11.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.940+0000 7f2c9d544640 1 -- start start 2024-03-19T02:53:11.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.940+0000 7f2c9d544640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c98100740 0x7f2c98197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:11.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.940+0000 7f2c9d544640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c98101930 0x7f2c98197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:11.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.940+0000 7f2c9d544640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c981022f0 0x7f2c9819f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:11.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.940+0000 7f2c9d544640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9810d390 con 0x7f2c98101930 2024-03-19T02:53:11.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.940+0000 7f2c9d544640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9810d210 con 0x7f2c98100740 2024-03-19T02:53:11.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.940+0000 7f2c9d544640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9810d510 con 0x7f2c981022f0 2024-03-19T02:53:11.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.940+0000 7f2c967fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c98101930 0x7f2c98197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:11.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.940+0000 7f2c967fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c98101930 0x7f2c98197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46576/0 (socket says 172.21.15.33:46576) 2024-03-19T02:53:11.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.940+0000 7f2c967fc640 1 -- 172.21.15.33:0/4047999486 learned_addr learned my addr 172.21.15.33:0/4047999486 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:11.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.940+0000 7f2c96ffd640 1 --2- 172.21.15.33:0/4047999486 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c98100740 0x7f2c98197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:11.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.940+0000 7f2c977fe640 1 --2- 172.21.15.33:0/4047999486 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c981022f0 0x7f2c9819f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:11.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.940+0000 7f2c967fc640 1 -- 172.21.15.33:0/4047999486 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c981022f0 msgr2=0x7f2c9819f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:11.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.940+0000 7f2c967fc640 1 --2- 172.21.15.33:0/4047999486 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c981022f0 0x7f2c9819f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:11.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.940+0000 7f2c967fc640 1 -- 172.21.15.33:0/4047999486 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c98100740 msgr2=0x7f2c98197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:11.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.940+0000 7f2c967fc640 1 --2- 172.21.15.33:0/4047999486 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c98100740 0x7f2c98197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:11.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.940+0000 7f2c967fc640 1 -- 172.21.15.33:0/4047999486 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c9819f6b0 con 0x7f2c98101930 2024-03-19T02:53:11.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.940+0000 7f2c967fc640 1 --2- 172.21.15.33:0/4047999486 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c98101930 0x7f2c98197fd0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2c8000e9f0 tx=0x7f2c8000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:11.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.941+0000 7f2c8ffff640 1 -- 172.21.15.33:0/4047999486 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c8000cd80 con 0x7f2c98101930 2024-03-19T02:53:11.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.941+0000 7f2c9d544640 1 -- 172.21.15.33:0/4047999486 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c9819f9a0 con 0x7f2c98101930 2024-03-19T02:53:11.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.941+0000 7f2c8ffff640 1 -- 172.21.15.33:0/4047999486 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2c800108e0 con 0x7f2c98101930 2024-03-19T02:53:11.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.941+0000 7f2c8ffff640 1 -- 172.21.15.33:0/4047999486 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c80018920 con 0x7f2c98101930 2024-03-19T02:53:11.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.941+0000 7f2c9d544640 1 -- 172.21.15.33:0/4047999486 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c9819ff40 con 0x7f2c98101930 2024-03-19T02:53:11.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.942+0000 7f2c9d544640 1 -- 172.21.15.33:0/4047999486 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c98107b00 con 0x7f2c98101930 2024-03-19T02:53:11.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.943+0000 7f2c8ffff640 1 -- 172.21.15.33:0/4047999486 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2c80010a90 con 0x7f2c98101930 2024-03-19T02:53:11.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.943+0000 7f2c8ffff640 1 --2- 172.21.15.33:0/4047999486 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2c840a2800 0x7f2c840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:11.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.943+0000 7f2c96ffd640 1 --2- 172.21.15.33:0/4047999486 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2c840a2800 0x7f2c840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:11.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.944+0000 7f2c8ffff640 1 -- 172.21.15.33:0/4047999486 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f2c800cf970 con 0x7f2c98101930 2024-03-19T02:53:11.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.944+0000 7f2c96ffd640 1 --2- 172.21.15.33:0/4047999486 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2c840a2800 0x7f2c840a4cc0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2c90005ab0 tx=0x7f2c90005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:11.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:11.947+0000 7f2c8ffff640 1 -- 172.21.15.33:0/4047999486 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2c80026070 con 0x7f2c98101930 2024-03-19T02:53:12.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.130+0000 7f2c9d544640 1 -- 172.21.15.33:0/4047999486 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2c9802d370 con 0x7f2c98101930 2024-03-19T02:53:12.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.132+0000 7f2c8ffff640 1 -- 172.21.15.33:0/4047999486 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) ==== 74+0+24663 (secure 0 0 0) 0x7f2c8009c680 con 0x7f2c98101930 2024-03-19T02:53:12.134 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:53:12.134 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":97,"fsid":"2b7a8938-e59a-11ee-95c9-87774f69a715","created":"2024-03-19T02:42:37.912179+0000","modified":"2024-03-19T02:52:41.116227+0000","last_up_change":"2024-03-19T02:50:01.539392+0000","last_in_change":"2024-03-19T02:49:40.534415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:46:09.963737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:50:50.069641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:50:51.381064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:50:55.476667+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"090a89ab-6b7b-4317-8823-178d4212b2f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6803","nonce":1633124995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6805","nonce":1633124995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6809","nonce":1633124995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6807","nonce":1633124995}]},"public_addr":"172.21.15.33:6803/1633124995","cluster_addr":"172.21.15.33:6805/1633124995","heartbeat_back_addr":"172.21.15.33:6809/1633124995","heartbeat_front_addr":"172.21.15.33:6807/1633124995","state":["exists","up"]},{"osd":1,"uuid":"f50eec55-1956-4486-8593-f2c8fa610e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6801","nonce":340144398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6803","nonce":340144398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6807","nonce":340144398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6805","nonce":340144398}]},"public_addr":"172.21.15.96:6801/340144398","cluster_addr":"172.21.15.96:6803/340144398","heartbeat_back_addr":"172.21.15.96:6807/340144398","heartbeat_front_addr":"172.21.15.96:6805/340144398","state":["exists","up"]},{"osd":2,"uuid":"893a2d62-80d9-4e5e-bc6a-a1c8c681d652","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6801","nonce":3388743876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6803","nonce":3388743876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6807","nonce":3388743876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6805","nonce":3388743876}]},"public_addr":"172.21.15.155:6801/3388743876","cluster_addr":"172.21.15.155:6803/3388743876","heartbeat_back_addr":"172.21.15.155:6807/3388743876","heartbeat_front_addr":"172.21.15.155:6805/3388743876","state":["exists","up"]},{"osd":3,"uuid":"b07e0482-4281-4d32-9dfe-9995d7246175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6811","nonce":677007171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6813","nonce":677007171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6817","nonce":677007171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6815","nonce":677007171}]},"public_addr":"172.21.15.33:6811/677007171","cluster_addr":"172.21.15.33:6813/677007171","heartbeat_back_addr":"172.21.15.33:6817/677007171","heartbeat_front_addr":"172.21.15.33:6815/677007171","state":["exists","up"]},{"osd":4,"uuid":"a988384b-af6b-47c6-8c07-5c9673acee45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6809","nonce":3106158554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6811","nonce":3106158554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6815","nonce":3106158554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6813","nonce":3106158554}]},"public_addr":"172.21.15.96:6809/3106158554","cluster_addr":"172.21.15.96:6811/3106158554","heartbeat_back_addr":"172.21.15.96:6815/3106158554","heartbeat_front_addr":"172.21.15.96:6813/3106158554","state":["exists","up"]},{"osd":5,"uuid":"198dc992-96f8-4c52-96ee-1fd5de148bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6809","nonce":327293324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6811","nonce":327293324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6815","nonce":327293324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6813","nonce":327293324}]},"public_addr":"172.21.15.155:6809/327293324","cluster_addr":"172.21.15.155:6811/327293324","heartbeat_back_addr":"172.21.15.155:6815/327293324","heartbeat_front_addr":"172.21.15.155:6813/327293324","state":["exists","up"]},{"osd":6,"uuid":"a7278ec0-ab2e-49ed-be53-84d8b9e24177","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6819","nonce":873871509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6821","nonce":873871509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6825","nonce":873871509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6823","nonce":873871509}]},"public_addr":"172.21.15.33:6819/873871509","cluster_addr":"172.21.15.33:6821/873871509","heartbeat_back_addr":"172.21.15.33:6825/873871509","heartbeat_front_addr":"172.21.15.33:6823/873871509","state":["exists","up"]},{"osd":7,"uuid":"727e133d-aa24-4ffc-90e2-266eb2aab8c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6817","nonce":2334123891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6819","nonce":2334123891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6823","nonce":2334123891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6821","nonce":2334123891}]},"public_addr":"172.21.15.96:6817/2334123891","cluster_addr":"172.21.15.96:6819/2334123891","heartbeat_back_addr":"172.21.15.96:6823/2334123891","heartbeat_front_addr":"172.21.15.96:6821/2334123891","state":["exists","up"]},{"osd":8,"uuid":"bb20eec0-39e5-4b86-9db5-486b1840cdad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6817","nonce":3498516903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6819","nonce":3498516903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6823","nonce":3498516903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6821","nonce":3498516903}]},"public_addr":"172.21.15.155:6817/3498516903","cluster_addr":"172.21.15.155:6819/3498516903","heartbeat_back_addr":"172.21.15.155:6823/3498516903","heartbeat_front_addr":"172.21.15.155:6821/3498516903","state":["exists","up"]},{"osd":9,"uuid":"179c6228-3e23-4f65-91a6-1ef0005d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6827","nonce":760777213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6829","nonce":760777213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6833","nonce":760777213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6831","nonce":760777213}]},"public_addr":"172.21.15.33:6827/760777213","cluster_addr":"172.21.15.33:6829/760777213","heartbeat_back_addr":"172.21.15.33:6833/760777213","heartbeat_front_addr":"172.21.15.33:6831/760777213","state":["exists","up"]},{"osd":10,"uuid":"be17b478-3c2e-41fb-bede-958333acdf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6825","nonce":1404898482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6827","nonce":1404898482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6831","nonce":1404898482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6829","nonce":1404898482}]},"public_addr":"172.21.15.96:6825/1404898482","cluster_addr":"172.21.15.96:6827/1404898482","heartbeat_back_addr":"172.21.15.96:6831/1404898482","heartbeat_front_addr":"172.21.15.96:6829/1404898482","state":["exists","up"]},{"osd":11,"uuid":"c0903ed2-cc1e-4c18-8731-1f0ee4dd0457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6825","nonce":792352827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6827","nonce":792352827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6831","nonce":792352827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6829","nonce":792352827}]},"public_addr":"172.21.15.155:6825/792352827","cluster_addr":"172.21.15.155:6827/792352827","heartbeat_back_addr":"172.21.15.155:6831/792352827","heartbeat_front_addr":"172.21.15.155:6829/792352827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:17.942801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:42.109449+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:05.981599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:32.069574+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:57.431015+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:22.448592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:48.526515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:14.498352+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:40.175426+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:06.364006+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:32.853161+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:59.073538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.9","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:6801/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:6800/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/1067768047":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/729379453":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2586588742":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/2932814165":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6801/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2212564963":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6800/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/3275126058":"2024-03-20T02:43:39.622218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:53:12.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.134+0000 7f2c9d544640 1 -- 172.21.15.33:0/4047999486 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2c840a2800 msgr2=0x7f2c840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:12.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.134+0000 7f2c9d544640 1 --2- 172.21.15.33:0/4047999486 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2c840a2800 0x7f2c840a4cc0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2c90005ab0 tx=0x7f2c90005fd0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.134+0000 7f2c9d544640 1 -- 172.21.15.33:0/4047999486 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c98101930 msgr2=0x7f2c98197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:12.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.134+0000 7f2c9d544640 1 --2- 172.21.15.33:0/4047999486 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c98101930 0x7f2c98197fd0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2c8000e9f0 tx=0x7f2c8000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.134+0000 7f2c9d544640 1 -- 172.21.15.33:0/4047999486 shutdown_connections 2024-03-19T02:53:12.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.134+0000 7f2c9d544640 1 --2- 172.21.15.33:0/4047999486 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f2c840a2800 0x7f2c840a4cc0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.134+0000 7f2c9d544640 1 --2- 172.21.15.33:0/4047999486 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2c981022f0 0x7f2c9819f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.134+0000 7f2c9d544640 1 --2- 172.21.15.33:0/4047999486 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2c98101930 0x7f2c98197fd0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.134+0000 7f2c9d544640 1 --2- 172.21.15.33:0/4047999486 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2c98100740 0x7f2c98197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.134+0000 7f2c9d544640 1 -- 172.21.15.33:0/4047999486 >> 172.21.15.33:0/4047999486 conn(0x7f2c980fbcb0 msgr2=0x7f2c98104cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:12.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.135+0000 7f2c9d544640 1 -- 172.21.15.33:0/4047999486 shutdown_connections 2024-03-19T02:53:12.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.135+0000 7f2c9d544640 1 -- 172.21.15.33:0/4047999486 wait complete. 2024-03-19T02:53:12.145 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:11 smithi033 ceph-mon[31973]: pgmap v373: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-19T02:53:12.145 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:11 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1392246336' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-19T02:53:12.147 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:46:09.963737+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T02:50:50.069641+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T02:50:51.381064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T02:50:55.476667+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T02:53:12.147 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T02:53:12.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:11 smithi096 ceph-mon[35946]: pgmap v373: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-19T02:53:12.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:11 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1392246336' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-19T02:53:12.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.252+0000 7fa11e353640 1 -- 172.21.15.33:0/881616587 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa118104be0 msgr2=0x7fa118106fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:12.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.252+0000 7fa11e353640 1 --2- 172.21.15.33:0/881616587 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa118104be0 0x7fa118106fd0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa10c009900 tx=0x7fa10c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:12.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.252+0000 7fa11e353640 1 -- 172.21.15.33:0/881616587 shutdown_connections 2024-03-19T02:53:12.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.252+0000 7fa11e353640 1 --2- 172.21.15.33:0/881616587 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa118107510 0x7fa118109960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.252+0000 7fa11e353640 1 --2- 172.21.15.33:0/881616587 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa118104be0 0x7fa118106fd0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.252+0000 7fa11e353640 1 --2- 172.21.15.33:0/881616587 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1181022b0 0x7fa1181046a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.252+0000 7fa11e353640 1 -- 172.21.15.33:0/881616587 >> 172.21.15.33:0/881616587 conn(0x7fa1180fbcc0 msgr2=0x7fa1180fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:12.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.252+0000 7fa11e353640 1 -- 172.21.15.33:0/881616587 shutdown_connections 2024-03-19T02:53:12.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.252+0000 7fa11e353640 1 -- 172.21.15.33:0/881616587 wait complete. 2024-03-19T02:53:12.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.252+0000 7fa11e353640 1 Processor -- start 2024-03-19T02:53:12.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.252+0000 7fa11e353640 1 -- start start 2024-03-19T02:53:12.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.253+0000 7fa11e353640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1181022b0 0x7fa118197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:12.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.253+0000 7fa11e353640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa118104be0 0x7fa118197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:12.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.253+0000 7fa11e353640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa118107510 0x7fa11819f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:12.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.253+0000 7fa11e353640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa11810d360 con 0x7fa1181022b0 2024-03-19T02:53:12.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.253+0000 7fa11e353640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa11810d1e0 con 0x7fa118104be0 2024-03-19T02:53:12.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.253+0000 7fa11e353640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa11810d4e0 con 0x7fa118107510 2024-03-19T02:53:12.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.253+0000 7fa117fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1181022b0 0x7fa118197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:12.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.253+0000 7fa117fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1181022b0 0x7fa118197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46588/0 (socket says 172.21.15.33:46588) 2024-03-19T02:53:12.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.253+0000 7fa117fff640 1 -- 172.21.15.33:0/1538816612 learned_addr learned my addr 172.21.15.33:0/1538816612 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:12.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.253+0000 7fa1177fe640 1 --2- 172.21.15.33:0/1538816612 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa118104be0 0x7fa118197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:12.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.253+0000 7fa11c8f0640 1 --2- 172.21.15.33:0/1538816612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa118107510 0x7fa11819f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:12.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.253+0000 7fa117fff640 1 -- 172.21.15.33:0/1538816612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa118107510 msgr2=0x7fa11819f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:12.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.253+0000 7fa117fff640 1 --2- 172.21.15.33:0/1538816612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa118107510 0x7fa11819f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.253+0000 7fa117fff640 1 -- 172.21.15.33:0/1538816612 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa118104be0 msgr2=0x7fa118197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:12.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.253+0000 7fa117fff640 1 --2- 172.21.15.33:0/1538816612 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa118104be0 0x7fa118197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.253+0000 7fa117fff640 1 -- 172.21.15.33:0/1538816612 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa11819f690 con 0x7fa1181022b0 2024-03-19T02:53:12.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.253+0000 7fa117fff640 1 --2- 172.21.15.33:0/1538816612 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1181022b0 0x7fa118197a70 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa10800e9f0 tx=0x7fa10800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:12.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.254+0000 7fa1157fa640 1 -- 172.21.15.33:0/1538816612 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa10800cd80 con 0x7fa1181022b0 2024-03-19T02:53:12.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.254+0000 7fa1157fa640 1 -- 172.21.15.33:0/1538816612 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa1080108e0 con 0x7fa1181022b0 2024-03-19T02:53:12.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.254+0000 7fa11e353640 1 -- 172.21.15.33:0/1538816612 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa11819f980 con 0x7fa1181022b0 2024-03-19T02:53:12.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.254+0000 7fa1157fa640 1 -- 172.21.15.33:0/1538816612 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa108018920 con 0x7fa1181022b0 2024-03-19T02:53:12.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.254+0000 7fa11e353640 1 -- 172.21.15.33:0/1538816612 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa11819fef0 con 0x7fa1181022b0 2024-03-19T02:53:12.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.255+0000 7fa11e353640 1 -- 172.21.15.33:0/1538816612 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1180fd7b0 con 0x7fa1181022b0 2024-03-19T02:53:12.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.256+0000 7fa1157fa640 1 -- 172.21.15.33:0/1538816612 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa108010a90 con 0x7fa1181022b0 2024-03-19T02:53:12.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.256+0000 7fa1157fa640 1 --2- 172.21.15.33:0/1538816612 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fa1000a2880 0x7fa1000a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:12.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.256+0000 7fa1177fe640 1 --2- 172.21.15.33:0/1538816612 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fa1000a2880 0x7fa1000a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:12.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.256+0000 7fa1157fa640 1 -- 172.21.15.33:0/1538816612 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7fa1080cf080 con 0x7fa1181022b0 2024-03-19T02:53:12.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.257+0000 7fa1177fe640 1 --2- 172.21.15.33:0/1538816612 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fa1000a2880 0x7fa1000a4d40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa10c005ab0 tx=0x7fa10c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:12.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.261+0000 7fa1157fa640 1 -- 172.21.15.33:0/1538816612 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa10809b740 con 0x7fa1181022b0 2024-03-19T02:53:12.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:11 smithi155 ceph-mon[35804]: pgmap v373: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-19T02:53:12.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:11 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1392246336' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-19T02:53:12.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.441+0000 7fa11e353640 1 -- 172.21.15.33:0/1538816612 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa118062420 con 0x7fa1181022b0 2024-03-19T02:53:12.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.441+0000 7fa1157fa640 1 -- 172.21.15.33:0/1538816612 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v97) ==== 93+0+10 (secure 0 0 0) 0x7fa10809f7e0 con 0x7fa1181022b0 2024-03-19T02:53:12.443 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 1 2024-03-19T02:53:12.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.444+0000 7fa11e353640 1 -- 172.21.15.33:0/1538816612 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fa1000a2880 msgr2=0x7fa1000a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:12.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.444+0000 7fa11e353640 1 --2- 172.21.15.33:0/1538816612 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fa1000a2880 0x7fa1000a4d40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa10c005ab0 tx=0x7fa10c005fd0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.444+0000 7fa11e353640 1 -- 172.21.15.33:0/1538816612 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1181022b0 msgr2=0x7fa118197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:12.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.444+0000 7fa11e353640 1 --2- 172.21.15.33:0/1538816612 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1181022b0 0x7fa118197a70 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa10800e9f0 tx=0x7fa10800eeb0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.444+0000 7fa11e353640 1 -- 172.21.15.33:0/1538816612 shutdown_connections 2024-03-19T02:53:12.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.444+0000 7fa11e353640 1 --2- 172.21.15.33:0/1538816612 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fa1000a2880 0x7fa1000a4d40 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.444+0000 7fa11e353640 1 --2- 172.21.15.33:0/1538816612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa118107510 0x7fa11819f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.444+0000 7fa11e353640 1 --2- 172.21.15.33:0/1538816612 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa118104be0 0x7fa118197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.444+0000 7fa11e353640 1 --2- 172.21.15.33:0/1538816612 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa1181022b0 0x7fa118197a70 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.444+0000 7fa11e353640 1 -- 172.21.15.33:0/1538816612 >> 172.21.15.33:0/1538816612 conn(0x7fa1180fbcc0 msgr2=0x7fa118104100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:12.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.444+0000 7fa11e353640 1 -- 172.21.15.33:0/1538816612 shutdown_connections 2024-03-19T02:53:12.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.444+0000 7fa11e353640 1 -- 172.21.15.33:0/1538816612 wait complete. 2024-03-19T02:53:12.456 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T02:53:12.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.560+0000 7fc18c516640 1 -- 172.21.15.33:0/3359606722 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc184103400 msgr2=0x7fc184109c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:12.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.560+0000 7fc18c516640 1 --2- 172.21.15.33:0/3359606722 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc184103400 0x7fc184109c90 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fc17c009be0 tx=0x7fc17c02bef0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.561+0000 7fc18c516640 1 -- 172.21.15.33:0/3359606722 shutdown_connections 2024-03-19T02:53:12.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.561+0000 7fc18c516640 1 --2- 172.21.15.33:0/3359606722 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc184102a40 0x7fc184102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.561+0000 7fc18c516640 1 --2- 172.21.15.33:0/3359606722 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc184103400 0x7fc184109c90 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.561+0000 7fc18c516640 1 --2- 172.21.15.33:0/3359606722 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc184101850 0x7fc184101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.561+0000 7fc18c516640 1 -- 172.21.15.33:0/3359606722 >> 172.21.15.33:0/3359606722 conn(0x7fc1840fcdc0 msgr2=0x7fc1840ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:12.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.561+0000 7fc18c516640 1 -- 172.21.15.33:0/3359606722 shutdown_connections 2024-03-19T02:53:12.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.561+0000 7fc18c516640 1 -- 172.21.15.33:0/3359606722 wait complete. 2024-03-19T02:53:12.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.561+0000 7fc18c516640 1 Processor -- start 2024-03-19T02:53:12.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.561+0000 7fc18c516640 1 -- start start 2024-03-19T02:53:12.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.561+0000 7fc18c516640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc184101850 0x7fc184197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:12.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.561+0000 7fc18c516640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc184102a40 0x7fc184197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:12.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.561+0000 7fc18c516640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc184103400 0x7fc18419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:12.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.561+0000 7fc18c516640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc18410d360 con 0x7fc184103400 2024-03-19T02:53:12.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.561+0000 7fc18c516640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc18410d1e0 con 0x7fc184101850 2024-03-19T02:53:12.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.561+0000 7fc18c516640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc18410d4e0 con 0x7fc184102a40 2024-03-19T02:53:12.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.561+0000 7fc18a2b2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc184101850 0x7fc184197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:12.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.561+0000 7fc18aab3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc184103400 0x7fc18419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:12.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.561+0000 7fc18a2b2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc184101850 0x7fc184197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:36780/0 (socket says 172.21.15.33:36780) 2024-03-19T02:53:12.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.561+0000 7fc18a2b2640 1 -- 172.21.15.33:0/3809654363 learned_addr learned my addr 172.21.15.33:0/3809654363 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:12.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.561+0000 7fc189ab1640 1 --2- 172.21.15.33:0/3809654363 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc184102a40 0x7fc184197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:12.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.562+0000 7fc18a2b2640 1 -- 172.21.15.33:0/3809654363 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc184102a40 msgr2=0x7fc184197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:12.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.562+0000 7fc18a2b2640 1 --2- 172.21.15.33:0/3809654363 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc184102a40 0x7fc184197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.562+0000 7fc18a2b2640 1 -- 172.21.15.33:0/3809654363 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc184103400 msgr2=0x7fc18419efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:12.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.562+0000 7fc18a2b2640 1 --2- 172.21.15.33:0/3809654363 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc184103400 0x7fc18419efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.562+0000 7fc18a2b2640 1 -- 172.21.15.33:0/3809654363 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc18419f620 con 0x7fc184101850 2024-03-19T02:53:12.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.562+0000 7fc18aab3640 1 --2- 172.21.15.33:0/3809654363 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc184103400 0x7fc18419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:12.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.562+0000 7fc18a2b2640 1 --2- 172.21.15.33:0/3809654363 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc184101850 0x7fc184197a00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc18000e9f0 tx=0x7fc18000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:12.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.562+0000 7fc1777fe640 1 -- 172.21.15.33:0/3809654363 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc18000ccc0 con 0x7fc184101850 2024-03-19T02:53:12.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.562+0000 7fc18c516640 1 -- 172.21.15.33:0/3809654363 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc18419f910 con 0x7fc184101850 2024-03-19T02:53:12.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.562+0000 7fc1777fe640 1 -- 172.21.15.33:0/3809654363 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc1800035f0 con 0x7fc184101850 2024-03-19T02:53:12.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.562+0000 7fc1777fe640 1 -- 172.21.15.33:0/3809654363 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc18000ce60 con 0x7fc184101850 2024-03-19T02:53:12.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.562+0000 7fc18c516640 1 -- 172.21.15.33:0/3809654363 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc18419feb0 con 0x7fc184101850 2024-03-19T02:53:12.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.563+0000 7fc18c516640 1 -- 172.21.15.33:0/3809654363 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc184068610 con 0x7fc184101850 2024-03-19T02:53:12.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.564+0000 7fc1777fe640 1 -- 172.21.15.33:0/3809654363 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc1800037a0 con 0x7fc184101850 2024-03-19T02:53:12.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.565+0000 7fc1777fe640 1 --2- 172.21.15.33:0/3809654363 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc1700a2800 0x7fc1700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:12.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.565+0000 7fc189ab1640 1 --2- 172.21.15.33:0/3809654363 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc1700a2800 0x7fc1700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:12.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.565+0000 7fc1777fe640 1 -- 172.21.15.33:0/3809654363 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7fc1800cf1a0 con 0x7fc184101850 2024-03-19T02:53:12.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.565+0000 7fc189ab1640 1 --2- 172.21.15.33:0/3809654363 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc1700a2800 0x7fc1700a4cc0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc184198f10 tx=0x7fc178005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:12.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.573+0000 7fc1777fe640 1 -- 172.21.15.33:0/3809654363 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc18009beb0 con 0x7fc184101850 2024-03-19T02:53:12.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.756+0000 7fc18c516640 1 -- 172.21.15.33:0/3809654363 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc18402d1f0 con 0x7fc184101850 2024-03-19T02:53:12.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.756+0000 7fc1777fe640 1 -- 172.21.15.33:0/3809654363 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v97) ==== 104+0+11 (secure 0 0 0) 0x7fc18009b7b0 con 0x7fc184101850 2024-03-19T02:53:12.758 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 32 2024-03-19T02:53:12.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.759+0000 7fc18c516640 1 -- 172.21.15.33:0/3809654363 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc1700a2800 msgr2=0x7fc1700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:12.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.759+0000 7fc18c516640 1 --2- 172.21.15.33:0/3809654363 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc1700a2800 0x7fc1700a4cc0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc184198f10 tx=0x7fc178005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:12.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.759+0000 7fc18c516640 1 -- 172.21.15.33:0/3809654363 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc184101850 msgr2=0x7fc184197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:12.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.759+0000 7fc18c516640 1 --2- 172.21.15.33:0/3809654363 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc184101850 0x7fc184197a00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc18000e9f0 tx=0x7fc18000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.759+0000 7fc18c516640 1 -- 172.21.15.33:0/3809654363 shutdown_connections 2024-03-19T02:53:12.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.759+0000 7fc18c516640 1 --2- 172.21.15.33:0/3809654363 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc1700a2800 0x7fc1700a4cc0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.759+0000 7fc18c516640 1 --2- 172.21.15.33:0/3809654363 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc184102a40 0x7fc184197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.759+0000 7fc18c516640 1 --2- 172.21.15.33:0/3809654363 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc184103400 0x7fc18419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.759+0000 7fc18c516640 1 --2- 172.21.15.33:0/3809654363 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc184101850 0x7fc184197a00 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.759+0000 7fc18c516640 1 -- 172.21.15.33:0/3809654363 >> 172.21.15.33:0/3809654363 conn(0x7fc1840fcdc0 msgr2=0x7fc184105dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:12.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.759+0000 7fc18c516640 1 -- 172.21.15.33:0/3809654363 shutdown_connections 2024-03-19T02:53:12.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.759+0000 7fc18c516640 1 -- 172.21.15.33:0/3809654363 wait complete. 2024-03-19T02:53:12.771 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T02:53:12.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.880+0000 7f6132bf1640 1 -- 172.21.15.33:0/3000321524 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f612c101930 msgr2=0x7f612c101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:12.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.880+0000 7f6132bf1640 1 --2- 172.21.15.33:0/3000321524 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f612c101930 0x7f612c101db0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6118009900 tx=0x7f611802bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:12.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.880+0000 7f6132bf1640 1 -- 172.21.15.33:0/3000321524 shutdown_connections 2024-03-19T02:53:12.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.880+0000 7f6132bf1640 1 --2- 172.21.15.33:0/3000321524 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f612c1022f0 0x7f612c108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.880+0000 7f6132bf1640 1 --2- 172.21.15.33:0/3000321524 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f612c101930 0x7f612c101db0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.880+0000 7f6132bf1640 1 --2- 172.21.15.33:0/3000321524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f612c100740 0x7f612c100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.880+0000 7f6132bf1640 1 -- 172.21.15.33:0/3000321524 >> 172.21.15.33:0/3000321524 conn(0x7f612c0fbcd0 msgr2=0x7f612c0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:12.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.880+0000 7f6132bf1640 1 -- 172.21.15.33:0/3000321524 shutdown_connections 2024-03-19T02:53:12.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.880+0000 7f6132bf1640 1 -- 172.21.15.33:0/3000321524 wait complete. 2024-03-19T02:53:12.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.881+0000 7f6132bf1640 1 Processor -- start 2024-03-19T02:53:12.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.881+0000 7f6132bf1640 1 -- start start 2024-03-19T02:53:12.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.881+0000 7f6132bf1640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f612c100740 0x7f612c06a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:12.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.881+0000 7f6132bf1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f612c101930 0x7f612c19c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:12.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.881+0000 7f6132bf1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f612c1022f0 0x7f612c19ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:12.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.881+0000 7f6132bf1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f612c10c270 con 0x7f612c1022f0 2024-03-19T02:53:12.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.881+0000 7f6132bf1640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f612c10c0f0 con 0x7f612c101930 2024-03-19T02:53:12.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.881+0000 7f6132bf1640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f612c10c3f0 con 0x7f612c100740 2024-03-19T02:53:12.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.881+0000 7f613118e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f612c1022f0 0x7f612c19ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:12.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.881+0000 7f613118e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f612c1022f0 0x7f612c19ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46618/0 (socket says 172.21.15.33:46618) 2024-03-19T02:53:12.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.881+0000 7f613118e640 1 -- 172.21.15.33:0/1591345445 learned_addr learned my addr 172.21.15.33:0/1591345445 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:12.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.881+0000 7f613098d640 1 --2- 172.21.15.33:0/1591345445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f612c100740 0x7f612c06a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:12.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.881+0000 7f612bfff640 1 --2- 172.21.15.33:0/1591345445 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f612c101930 0x7f612c19c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:12.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.881+0000 7f613118e640 1 -- 172.21.15.33:0/1591345445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f612c100740 msgr2=0x7f612c06a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:12.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.881+0000 7f613118e640 1 --2- 172.21.15.33:0/1591345445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f612c100740 0x7f612c06a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.881+0000 7f613118e640 1 -- 172.21.15.33:0/1591345445 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f612c101930 msgr2=0x7f612c19c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:12.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.881+0000 7f613118e640 1 --2- 172.21.15.33:0/1591345445 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f612c101930 0x7f612c19c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:12.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.881+0000 7f613118e640 1 -- 172.21.15.33:0/1591345445 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f612c19f750 con 0x7f612c1022f0 2024-03-19T02:53:12.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.881+0000 7f613118e640 1 --2- 172.21.15.33:0/1591345445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f612c1022f0 0x7f612c19ee40 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f612400eb10 tx=0x7f612400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:12.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.882+0000 7f6129ffb640 1 -- 172.21.15.33:0/1591345445 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6124004b20 con 0x7f612c1022f0 2024-03-19T02:53:12.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.882+0000 7f6132bf1640 1 -- 172.21.15.33:0/1591345445 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f612c19f9e0 con 0x7f612c1022f0 2024-03-19T02:53:12.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.882+0000 7f6129ffb640 1 -- 172.21.15.33:0/1591345445 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6124004cc0 con 0x7f612c1022f0 2024-03-19T02:53:12.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.882+0000 7f6129ffb640 1 -- 172.21.15.33:0/1591345445 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6124010690 con 0x7f612c1022f0 2024-03-19T02:53:12.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.882+0000 7f6132bf1640 1 -- 172.21.15.33:0/1591345445 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f612c19feb0 con 0x7f612c1022f0 2024-03-19T02:53:12.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.883+0000 7f6132bf1640 1 -- 172.21.15.33:0/1591345445 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f612c107b20 con 0x7f612c1022f0 2024-03-19T02:53:12.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.884+0000 7f6129ffb640 1 -- 172.21.15.33:0/1591345445 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6124005020 con 0x7f612c1022f0 2024-03-19T02:53:12.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.884+0000 7f6129ffb640 1 --2- 172.21.15.33:0/1591345445 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f611c0a2800 0x7f611c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:12.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.884+0000 7f613098d640 1 --2- 172.21.15.33:0/1591345445 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f611c0a2800 0x7f611c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:12.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.884+0000 7f6129ffb640 1 -- 172.21.15.33:0/1591345445 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f61240cf360 con 0x7f612c1022f0 2024-03-19T02:53:12.887 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.885+0000 7f613098d640 1 --2- 172.21.15.33:0/1591345445 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f611c0a2800 0x7f611c0a4cc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f612c101790 tx=0x7f6120005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:12.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:12.889+0000 7f6129ffb640 1 -- 172.21.15.33:0/1591345445 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f612409c070 con 0x7f612c1022f0 2024-03-19T02:53:13.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:12 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1392246336' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-19T02:53:13.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:12 smithi033 ceph-mon[31973]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-19T02:53:13.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:12 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/4047999486' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:53:13.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:12 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1538816612' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:53:13.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:12 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3809654363' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:53:13.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.074+0000 7f6132bf1640 1 -- 172.21.15.33:0/1591345445 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f612c0624c0 con 0x7f612c1022f0 2024-03-19T02:53:13.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.074+0000 7f6129ffb640 1 -- 172.21.15.33:0/1591345445 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v97) ==== 100+0+11 (secure 0 0 0) 0x7f612409b970 con 0x7f612c1022f0 2024-03-19T02:53:13.075 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 64 2024-03-19T02:53:13.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.076+0000 7f6132bf1640 1 -- 172.21.15.33:0/1591345445 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f611c0a2800 msgr2=0x7f611c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:13.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.076+0000 7f6132bf1640 1 --2- 172.21.15.33:0/1591345445 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f611c0a2800 0x7f611c0a4cc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f612c101790 tx=0x7f6120005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:13.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.076+0000 7f6132bf1640 1 -- 172.21.15.33:0/1591345445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f612c1022f0 msgr2=0x7f612c19ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:13.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.076+0000 7f6132bf1640 1 --2- 172.21.15.33:0/1591345445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f612c1022f0 0x7f612c19ee40 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f612400eb10 tx=0x7f612400c2d0 comp rx=0 tx=0).stop 2024-03-19T02:53:13.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.076+0000 7f6132bf1640 1 -- 172.21.15.33:0/1591345445 shutdown_connections 2024-03-19T02:53:13.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.076+0000 7f6132bf1640 1 --2- 172.21.15.33:0/1591345445 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f611c0a2800 0x7f611c0a4cc0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:13.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.076+0000 7f6132bf1640 1 --2- 172.21.15.33:0/1591345445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f612c1022f0 0x7f612c19ee40 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:13.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.076+0000 7f6132bf1640 1 --2- 172.21.15.33:0/1591345445 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f612c101930 0x7f612c19c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:13.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.076+0000 7f6132bf1640 1 --2- 172.21.15.33:0/1591345445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f612c100740 0x7f612c06a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:13.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.076+0000 7f6132bf1640 1 -- 172.21.15.33:0/1591345445 >> 172.21.15.33:0/1591345445 conn(0x7f612c0fbcd0 msgr2=0x7f612c104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:13.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.076+0000 7f6132bf1640 1 -- 172.21.15.33:0/1591345445 shutdown_connections 2024-03-19T02:53:13.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.076+0000 7f6132bf1640 1 -- 172.21.15.33:0/1591345445 wait complete. 2024-03-19T02:53:13.088 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T02:53:13.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.193+0000 7fbc6166e640 1 -- 172.21.15.33:0/2373192915 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc5c102a20 msgr2=0x7fbc5c102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:13.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.193+0000 7fbc6166e640 1 --2- 172.21.15.33:0/2373192915 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc5c102a20 0x7fbc5c102ea0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fbc50009900 tx=0x7fbc5002bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:13.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.193+0000 7fbc6166e640 1 -- 172.21.15.33:0/2373192915 shutdown_connections 2024-03-19T02:53:13.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.193+0000 7fbc6166e640 1 --2- 172.21.15.33:0/2373192915 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc5c1033e0 0x7fbc5c109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:13.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.193+0000 7fbc6166e640 1 --2- 172.21.15.33:0/2373192915 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc5c102a20 0x7fbc5c102ea0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:13.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.193+0000 7fbc6166e640 1 --2- 172.21.15.33:0/2373192915 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc5c101830 0x7fbc5c101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:13.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.193+0000 7fbc6166e640 1 -- 172.21.15.33:0/2373192915 >> 172.21.15.33:0/2373192915 conn(0x7fbc5c0fcdc0 msgr2=0x7fbc5c0ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:13.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.193+0000 7fbc6166e640 1 -- 172.21.15.33:0/2373192915 shutdown_connections 2024-03-19T02:53:13.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.193+0000 7fbc6166e640 1 -- 172.21.15.33:0/2373192915 wait complete. 2024-03-19T02:53:13.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.193+0000 7fbc6166e640 1 Processor -- start 2024-03-19T02:53:13.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.194+0000 7fbc6166e640 1 -- start start 2024-03-19T02:53:13.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.194+0000 7fbc6166e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc5c101830 0x7fbc5c197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:13.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.194+0000 7fbc6166e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc5c102a20 0x7fbc5c198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:13.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.194+0000 7fbc6166e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc5c1033e0 0x7fbc5c19f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:13.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.194+0000 7fbc6166e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5c10d340 con 0x7fbc5c1033e0 2024-03-19T02:53:13.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.194+0000 7fbc6166e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5c10d1c0 con 0x7fbc5c102a20 2024-03-19T02:53:13.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.194+0000 7fbc6166e640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5c10d4c0 con 0x7fbc5c101830 2024-03-19T02:53:13.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.194+0000 7fbc5b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc5c1033e0 0x7fbc5c19f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:13.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.194+0000 7fbc5b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc5c1033e0 0x7fbc5c19f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46632/0 (socket says 172.21.15.33:46632) 2024-03-19T02:53:13.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.194+0000 7fbc5b7fe640 1 -- 172.21.15.33:0/1890446709 learned_addr learned my addr 172.21.15.33:0/1890446709 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:13.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.194+0000 7fbc5a7fc640 1 --2- 172.21.15.33:0/1890446709 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc5c102a20 0x7fbc5c198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:13.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.194+0000 7fbc5affd640 1 --2- 172.21.15.33:0/1890446709 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc5c101830 0x7fbc5c197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:13.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.194+0000 7fbc5b7fe640 1 -- 172.21.15.33:0/1890446709 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc5c101830 msgr2=0x7fbc5c197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:13.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.194+0000 7fbc5b7fe640 1 --2- 172.21.15.33:0/1890446709 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc5c101830 0x7fbc5c197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:13.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.194+0000 7fbc5b7fe640 1 -- 172.21.15.33:0/1890446709 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc5c102a20 msgr2=0x7fbc5c198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:13.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.194+0000 7fbc5b7fe640 1 --2- 172.21.15.33:0/1890446709 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc5c102a20 0x7fbc5c198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:13.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.194+0000 7fbc5b7fe640 1 -- 172.21.15.33:0/1890446709 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc5c19f6e0 con 0x7fbc5c1033e0 2024-03-19T02:53:13.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.194+0000 7fbc5b7fe640 1 --2- 172.21.15.33:0/1890446709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc5c1033e0 0x7fbc5c19f070 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fbc5400e9b0 tx=0x7fbc5400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:13.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.194+0000 7fbc43fff640 1 -- 172.21.15.33:0/1890446709 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc5400cd80 con 0x7fbc5c1033e0 2024-03-19T02:53:13.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.194+0000 7fbc6166e640 1 -- 172.21.15.33:0/1890446709 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc5c19f9d0 con 0x7fbc5c1033e0 2024-03-19T02:53:13.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.194+0000 7fbc43fff640 1 -- 172.21.15.33:0/1890446709 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbc540035f0 con 0x7fbc5c1033e0 2024-03-19T02:53:13.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.194+0000 7fbc43fff640 1 -- 172.21.15.33:0/1890446709 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc54010690 con 0x7fbc5c1033e0 2024-03-19T02:53:13.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.194+0000 7fbc6166e640 1 -- 172.21.15.33:0/1890446709 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc5c19fef0 con 0x7fbc5c1033e0 2024-03-19T02:53:13.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.195+0000 7fbc6166e640 1 -- 172.21.15.33:0/1890446709 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc5c108c10 con 0x7fbc5c1033e0 2024-03-19T02:53:13.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.196+0000 7fbc43fff640 1 -- 172.21.15.33:0/1890446709 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fbc540037a0 con 0x7fbc5c1033e0 2024-03-19T02:53:13.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.197+0000 7fbc43fff640 1 --2- 172.21.15.33:0/1890446709 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbc440a2800 0x7fbc440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:13.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.197+0000 7fbc5affd640 1 --2- 172.21.15.33:0/1890446709 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbc440a2800 0x7fbc440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:13.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.197+0000 7fbc43fff640 1 -- 172.21.15.33:0/1890446709 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7fbc540cf140 con 0x7fbc5c1033e0 2024-03-19T02:53:13.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.197+0000 7fbc5affd640 1 --2- 172.21.15.33:0/1890446709 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbc440a2800 0x7fbc440a4cc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c102880 tx=0x7fbc4c009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:13.202 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.201+0000 7fbc43fff640 1 -- 172.21.15.33:0/1890446709 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbc5409be50 con 0x7fbc5c1033e0 2024-03-19T02:53:13.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:12 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1392246336' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-19T02:53:13.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:12 smithi096 ceph-mon[35946]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-19T02:53:13.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:12 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/4047999486' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:53:13.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:12 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1538816612' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:53:13.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:12 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3809654363' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:53:13.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:12 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1392246336' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-19T02:53:13.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:12 smithi155 ceph-mon[35804]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-19T02:53:13.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:12 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/4047999486' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:53:13.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:12 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1538816612' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:53:13.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:12 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3809654363' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:53:13.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.387+0000 7fbc6166e640 1 -- 172.21.15.33:0/1890446709 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fbc5c02d1f0 con 0x7fbc5c1033e0 2024-03-19T02:53:13.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.387+0000 7fbc43fff640 1 -- 172.21.15.33:0/1890446709 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v97) ==== 103+0+11 (secure 0 0 0) 0x7fbc5409b750 con 0x7fbc5c1033e0 2024-03-19T02:53:13.389 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 32 2024-03-19T02:53:13.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.389+0000 7fbc6166e640 1 -- 172.21.15.33:0/1890446709 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbc440a2800 msgr2=0x7fbc440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:13.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.389+0000 7fbc6166e640 1 --2- 172.21.15.33:0/1890446709 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbc440a2800 0x7fbc440a4cc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c102880 tx=0x7fbc4c009500 comp rx=0 tx=0).stop 2024-03-19T02:53:13.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.389+0000 7fbc6166e640 1 -- 172.21.15.33:0/1890446709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc5c1033e0 msgr2=0x7fbc5c19f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:13.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.389+0000 7fbc6166e640 1 --2- 172.21.15.33:0/1890446709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc5c1033e0 0x7fbc5c19f070 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fbc5400e9b0 tx=0x7fbc5400ee70 comp rx=0 tx=0).stop 2024-03-19T02:53:13.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.389+0000 7fbc6166e640 1 -- 172.21.15.33:0/1890446709 shutdown_connections 2024-03-19T02:53:13.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.390+0000 7fbc6166e640 1 --2- 172.21.15.33:0/1890446709 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbc440a2800 0x7fbc440a4cc0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:13.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.390+0000 7fbc6166e640 1 --2- 172.21.15.33:0/1890446709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbc5c1033e0 0x7fbc5c19f070 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:13.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.390+0000 7fbc6166e640 1 --2- 172.21.15.33:0/1890446709 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc5c102a20 0x7fbc5c198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:13.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.390+0000 7fbc6166e640 1 --2- 172.21.15.33:0/1890446709 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbc5c101830 0x7fbc5c197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:13.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.390+0000 7fbc6166e640 1 -- 172.21.15.33:0/1890446709 >> 172.21.15.33:0/1890446709 conn(0x7fbc5c0fcdc0 msgr2=0x7fbc5c105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:13.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.390+0000 7fbc6166e640 1 -- 172.21.15.33:0/1890446709 shutdown_connections 2024-03-19T02:53:13.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.390+0000 7fbc6166e640 1 -- 172.21.15.33:0/1890446709 wait complete. 2024-03-19T02:53:13.402 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:53:13.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.508+0000 7f8662a27640 1 -- 172.21.15.33:0/256774898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f865c102a40 msgr2=0x7f865c102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:13.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.508+0000 7f8662a27640 1 --2- 172.21.15.33:0/256774898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f865c102a40 0x7f865c102ec0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f864c009900 tx=0x7f864c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:13.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.508+0000 7f8662a27640 1 -- 172.21.15.33:0/256774898 shutdown_connections 2024-03-19T02:53:13.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.508+0000 7f8662a27640 1 --2- 172.21.15.33:0/256774898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f865c103400 0x7f865c109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:13.511 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.508+0000 7f8662a27640 1 --2- 172.21.15.33:0/256774898 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f865c102a40 0x7f865c102ec0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:13.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.508+0000 7f8662a27640 1 --2- 172.21.15.33:0/256774898 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f865c101850 0x7f865c101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:13.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.508+0000 7f8662a27640 1 -- 172.21.15.33:0/256774898 >> 172.21.15.33:0/256774898 conn(0x7f865c0fcdc0 msgr2=0x7f865c0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:13.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.508+0000 7f8662a27640 1 -- 172.21.15.33:0/256774898 shutdown_connections 2024-03-19T02:53:13.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.508+0000 7f8662a27640 1 -- 172.21.15.33:0/256774898 wait complete. 2024-03-19T02:53:13.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.508+0000 7f8662a27640 1 Processor -- start 2024-03-19T02:53:13.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.508+0000 7f8662a27640 1 -- start start 2024-03-19T02:53:13.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.509+0000 7f8662a27640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f865c101850 0x7f865c197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:13.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.509+0000 7f8662a27640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f865c102a40 0x7f865c197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:13.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.509+0000 7f8662a27640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f865c103400 0x7f865c19f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:13.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.509+0000 7f8662a27640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f865c10d360 con 0x7f865c103400 2024-03-19T02:53:13.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.509+0000 7f8662a27640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f865c10d1e0 con 0x7f865c102a40 2024-03-19T02:53:13.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.509+0000 7f8662a27640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f865c10d4e0 con 0x7f865c101850 2024-03-19T02:53:13.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.509+0000 7f8660fc4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f865c103400 0x7f865c19f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:13.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.509+0000 7f8660fc4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f865c103400 0x7f865c19f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46664/0 (socket says 172.21.15.33:46664) 2024-03-19T02:53:13.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.509+0000 7f865bfff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f865c101850 0x7f865c197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:13.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.509+0000 7f8660fc4640 1 -- 172.21.15.33:0/3245887824 learned_addr learned my addr 172.21.15.33:0/3245887824 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:13.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.509+0000 7f865b7fe640 1 --2- 172.21.15.33:0/3245887824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f865c102a40 0x7f865c197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:13.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.509+0000 7f8660fc4640 1 -- 172.21.15.33:0/3245887824 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f865c101850 msgr2=0x7f865c197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:13.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.509+0000 7f8660fc4640 1 --2- 172.21.15.33:0/3245887824 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f865c101850 0x7f865c197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:13.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.509+0000 7f8660fc4640 1 -- 172.21.15.33:0/3245887824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f865c102a40 msgr2=0x7f865c197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:13.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.509+0000 7f8660fc4640 1 --2- 172.21.15.33:0/3245887824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f865c102a40 0x7f865c197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:13.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.509+0000 7f8660fc4640 1 -- 172.21.15.33:0/3245887824 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f865c19f6b0 con 0x7f865c103400 2024-03-19T02:53:13.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.509+0000 7f865b7fe640 1 --2- 172.21.15.33:0/3245887824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f865c102a40 0x7f865c197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:13.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.509+0000 7f8660fc4640 1 --2- 172.21.15.33:0/3245887824 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f865c103400 0x7f865c19f040 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f865400eb10 tx=0x7f865400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:13.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.509+0000 7f86597fa640 1 -- 172.21.15.33:0/3245887824 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8654004b20 con 0x7f865c103400 2024-03-19T02:53:13.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.509+0000 7f8662a27640 1 -- 172.21.15.33:0/3245887824 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f865c19f9a0 con 0x7f865c103400 2024-03-19T02:53:13.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.509+0000 7f86597fa640 1 -- 172.21.15.33:0/3245887824 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8654004cc0 con 0x7f865c103400 2024-03-19T02:53:13.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.509+0000 7f86597fa640 1 -- 172.21.15.33:0/3245887824 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8654010690 con 0x7f865c103400 2024-03-19T02:53:13.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.510+0000 7f8662a27640 1 -- 172.21.15.33:0/3245887824 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f865c19fe70 con 0x7f865c103400 2024-03-19T02:53:13.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.511+0000 7f8662a27640 1 -- 172.21.15.33:0/3245887824 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f865c108ce0 con 0x7f865c103400 2024-03-19T02:53:13.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.512+0000 7f86597fa640 1 -- 172.21.15.33:0/3245887824 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8654005000 con 0x7f865c103400 2024-03-19T02:53:13.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.512+0000 7f86597fa640 1 --2- 172.21.15.33:0/3245887824 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f86440a2800 0x7f86440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:13.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.512+0000 7f865bfff640 1 --2- 172.21.15.33:0/3245887824 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f86440a2800 0x7f86440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:13.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.512+0000 7f86597fa640 1 -- 172.21.15.33:0/3245887824 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f86540cf580 con 0x7f865c103400 2024-03-19T02:53:13.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.513+0000 7f865bfff640 1 --2- 172.21.15.33:0/3245887824 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f86440a2800 0x7f86440a4cc0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f865c1028a0 tx=0x7f8650005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:13.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.516+0000 7f86597fa640 1 -- 172.21.15.33:0/3245887824 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8654053740 con 0x7f865c103400 2024-03-19T02:53:13.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.702+0000 7f8662a27640 1 -- 172.21.15.33:0/3245887824 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f865c02d1f0 con 0x7f865c103400 2024-03-19T02:53:13.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.704+0000 7f86597fa640 1 -- 172.21.15.33:0/3245887824 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) ==== 74+0+24663 (secure 0 0 0) 0x7f865409c290 con 0x7f865c103400 2024-03-19T02:53:13.705 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:53:13.705 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":97,"fsid":"2b7a8938-e59a-11ee-95c9-87774f69a715","created":"2024-03-19T02:42:37.912179+0000","modified":"2024-03-19T02:52:41.116227+0000","last_up_change":"2024-03-19T02:50:01.539392+0000","last_in_change":"2024-03-19T02:49:40.534415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:46:09.963737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:50:50.069641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:50:51.381064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:50:55.476667+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"090a89ab-6b7b-4317-8823-178d4212b2f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6803","nonce":1633124995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6805","nonce":1633124995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6809","nonce":1633124995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6807","nonce":1633124995}]},"public_addr":"172.21.15.33:6803/1633124995","cluster_addr":"172.21.15.33:6805/1633124995","heartbeat_back_addr":"172.21.15.33:6809/1633124995","heartbeat_front_addr":"172.21.15.33:6807/1633124995","state":["exists","up"]},{"osd":1,"uuid":"f50eec55-1956-4486-8593-f2c8fa610e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6801","nonce":340144398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6803","nonce":340144398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6807","nonce":340144398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6805","nonce":340144398}]},"public_addr":"172.21.15.96:6801/340144398","cluster_addr":"172.21.15.96:6803/340144398","heartbeat_back_addr":"172.21.15.96:6807/340144398","heartbeat_front_addr":"172.21.15.96:6805/340144398","state":["exists","up"]},{"osd":2,"uuid":"893a2d62-80d9-4e5e-bc6a-a1c8c681d652","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6801","nonce":3388743876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6803","nonce":3388743876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6807","nonce":3388743876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6805","nonce":3388743876}]},"public_addr":"172.21.15.155:6801/3388743876","cluster_addr":"172.21.15.155:6803/3388743876","heartbeat_back_addr":"172.21.15.155:6807/3388743876","heartbeat_front_addr":"172.21.15.155:6805/3388743876","state":["exists","up"]},{"osd":3,"uuid":"b07e0482-4281-4d32-9dfe-9995d7246175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6811","nonce":677007171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6813","nonce":677007171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6817","nonce":677007171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6815","nonce":677007171}]},"public_addr":"172.21.15.33:6811/677007171","cluster_addr":"172.21.15.33:6813/677007171","heartbeat_back_addr":"172.21.15.33:6817/677007171","heartbeat_front_addr":"172.21.15.33:6815/677007171","state":["exists","up"]},{"osd":4,"uuid":"a988384b-af6b-47c6-8c07-5c9673acee45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6809","nonce":3106158554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6811","nonce":3106158554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6815","nonce":3106158554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6813","nonce":3106158554}]},"public_addr":"172.21.15.96:6809/3106158554","cluster_addr":"172.21.15.96:6811/3106158554","heartbeat_back_addr":"172.21.15.96:6815/3106158554","heartbeat_front_addr":"172.21.15.96:6813/3106158554","state":["exists","up"]},{"osd":5,"uuid":"198dc992-96f8-4c52-96ee-1fd5de148bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6809","nonce":327293324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6811","nonce":327293324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6815","nonce":327293324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6813","nonce":327293324}]},"public_addr":"172.21.15.155:6809/327293324","cluster_addr":"172.21.15.155:6811/327293324","heartbeat_back_addr":"172.21.15.155:6815/327293324","heartbeat_front_addr":"172.21.15.155:6813/327293324","state":["exists","up"]},{"osd":6,"uuid":"a7278ec0-ab2e-49ed-be53-84d8b9e24177","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6819","nonce":873871509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6821","nonce":873871509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6825","nonce":873871509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6823","nonce":873871509}]},"public_addr":"172.21.15.33:6819/873871509","cluster_addr":"172.21.15.33:6821/873871509","heartbeat_back_addr":"172.21.15.33:6825/873871509","heartbeat_front_addr":"172.21.15.33:6823/873871509","state":["exists","up"]},{"osd":7,"uuid":"727e133d-aa24-4ffc-90e2-266eb2aab8c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6817","nonce":2334123891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6819","nonce":2334123891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6823","nonce":2334123891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6821","nonce":2334123891}]},"public_addr":"172.21.15.96:6817/2334123891","cluster_addr":"172.21.15.96:6819/2334123891","heartbeat_back_addr":"172.21.15.96:6823/2334123891","heartbeat_front_addr":"172.21.15.96:6821/2334123891","state":["exists","up"]},{"osd":8,"uuid":"bb20eec0-39e5-4b86-9db5-486b1840cdad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6817","nonce":3498516903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6819","nonce":3498516903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6823","nonce":3498516903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6821","nonce":3498516903}]},"public_addr":"172.21.15.155:6817/3498516903","cluster_addr":"172.21.15.155:6819/3498516903","heartbeat_back_addr":"172.21.15.155:6823/3498516903","heartbeat_front_addr":"172.21.15.155:6821/3498516903","state":["exists","up"]},{"osd":9,"uuid":"179c6228-3e23-4f65-91a6-1ef0005d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6827","nonce":760777213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6829","nonce":760777213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6833","nonce":760777213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6831","nonce":760777213}]},"public_addr":"172.21.15.33:6827/760777213","cluster_addr":"172.21.15.33:6829/760777213","heartbeat_back_addr":"172.21.15.33:6833/760777213","heartbeat_front_addr":"172.21.15.33:6831/760777213","state":["exists","up"]},{"osd":10,"uuid":"be17b478-3c2e-41fb-bede-958333acdf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6825","nonce":1404898482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6827","nonce":1404898482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6831","nonce":1404898482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6829","nonce":1404898482}]},"public_addr":"172.21.15.96:6825/1404898482","cluster_addr":"172.21.15.96:6827/1404898482","heartbeat_back_addr":"172.21.15.96:6831/1404898482","heartbeat_front_addr":"172.21.15.96:6829/1404898482","state":["exists","up"]},{"osd":11,"uuid":"c0903ed2-cc1e-4c18-8731-1f0ee4dd0457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6825","nonce":792352827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6827","nonce":792352827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6831","nonce":792352827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6829","nonce":792352827}]},"public_addr":"172.21.15.155:6825/792352827","cluster_addr":"172.21.15.155:6827/792352827","heartbeat_back_addr":"172.21.15.155:6831/792352827","heartbeat_front_addr":"172.21.15.155:6829/792352827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:17.942801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:42.109449+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:05.981599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:32.069574+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:57.431015+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:22.448592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:48.526515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:14.498352+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:40.175426+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:06.364006+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:32.853161+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:59.073538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.9","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:6801/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:6800/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/1067768047":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/729379453":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2586588742":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/2932814165":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6801/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2212564963":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6800/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/3275126058":"2024-03-20T02:43:39.622218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:53:13.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.706+0000 7f8662a27640 1 -- 172.21.15.33:0/3245887824 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f86440a2800 msgr2=0x7f86440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:13.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.706+0000 7f8662a27640 1 --2- 172.21.15.33:0/3245887824 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f86440a2800 0x7f86440a4cc0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f865c1028a0 tx=0x7f8650005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:13.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.706+0000 7f8662a27640 1 -- 172.21.15.33:0/3245887824 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f865c103400 msgr2=0x7f865c19f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:13.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.706+0000 7f8662a27640 1 --2- 172.21.15.33:0/3245887824 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f865c103400 0x7f865c19f040 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f865400eb10 tx=0x7f865400c2d0 comp rx=0 tx=0).stop 2024-03-19T02:53:13.707 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.706+0000 7f8662a27640 1 -- 172.21.15.33:0/3245887824 shutdown_connections 2024-03-19T02:53:13.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.706+0000 7f8662a27640 1 --2- 172.21.15.33:0/3245887824 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f86440a2800 0x7f86440a4cc0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:13.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.706+0000 7f8662a27640 1 --2- 172.21.15.33:0/3245887824 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f865c103400 0x7f865c19f040 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:13.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.706+0000 7f8662a27640 1 --2- 172.21.15.33:0/3245887824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f865c102a40 0x7f865c197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:13.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.706+0000 7f8662a27640 1 --2- 172.21.15.33:0/3245887824 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f865c101850 0x7f865c197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:13.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.706+0000 7f8662a27640 1 -- 172.21.15.33:0/3245887824 >> 172.21.15.33:0/3245887824 conn(0x7f865c0fcdc0 msgr2=0x7f865c105eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:13.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.706+0000 7f8662a27640 1 -- 172.21.15.33:0/3245887824 shutdown_connections 2024-03-19T02:53:13.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.706+0000 7f8662a27640 1 -- 172.21.15.33:0/3245887824 wait complete. 2024-03-19T02:53:13.719 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:46:09.963737+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T02:50:50.069641+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T02:50:51.381064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T02:50:55.476667+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T02:53:13.719 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T02:53:13.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.824+0000 7f966ab60640 1 -- 172.21.15.33:0/2433390287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9664101840 msgr2=0x7f9664101c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:13.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.824+0000 7f966ab60640 1 --2- 172.21.15.33:0/2433390287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9664101840 0x7f9664101c60 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f965c009ca0 tx=0x7f965c02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:53:13.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.824+0000 7f966ab60640 1 -- 172.21.15.33:0/2433390287 shutdown_connections 2024-03-19T02:53:13.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.824+0000 7f966ab60640 1 --2- 172.21.15.33:0/2433390287 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96641033f0 0x7f9664109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:13.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.824+0000 7f966ab60640 1 --2- 172.21.15.33:0/2433390287 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9664102a30 0x7f9664102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:13.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.824+0000 7f966ab60640 1 --2- 172.21.15.33:0/2433390287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9664101840 0x7f9664101c60 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:13.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.824+0000 7f966ab60640 1 -- 172.21.15.33:0/2433390287 >> 172.21.15.33:0/2433390287 conn(0x7f96640fcd90 msgr2=0x7f96640ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:13.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.824+0000 7f966ab60640 1 -- 172.21.15.33:0/2433390287 shutdown_connections 2024-03-19T02:53:13.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.824+0000 7f966ab60640 1 -- 172.21.15.33:0/2433390287 wait complete. 2024-03-19T02:53:13.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.825+0000 7f966ab60640 1 Processor -- start 2024-03-19T02:53:13.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.825+0000 7f966ab60640 1 -- start start 2024-03-19T02:53:13.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.825+0000 7f966ab60640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9664101840 0x7f96640755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:13.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.825+0000 7f966ab60640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9664102a30 0x7f9664073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:13.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.825+0000 7f966ab60640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96641033f0 0x7f9664074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:13.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.825+0000 7f966ab60640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f966410d370 con 0x7f9664102a30 2024-03-19T02:53:13.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.825+0000 7f966ab60640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f966410d1f0 con 0x7f96641033f0 2024-03-19T02:53:13.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.825+0000 7f966ab60640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f966410d4f0 con 0x7f9664101840 2024-03-19T02:53:13.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.825+0000 7f9663fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9664102a30 0x7f9664073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:13.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.825+0000 7f9663fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9664102a30 0x7f9664073c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46684/0 (socket says 172.21.15.33:46684) 2024-03-19T02:53:13.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.825+0000 7f9663fff640 1 -- 172.21.15.33:0/3222985435 learned_addr learned my addr 172.21.15.33:0/3222985435 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:13.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.825+0000 7f96690fd640 1 --2- 172.21.15.33:0/3222985435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96641033f0 0x7f9664074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:13.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.825+0000 7f96688fc640 1 --2- 172.21.15.33:0/3222985435 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9664101840 0x7f96640755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:13.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.825+0000 7f9663fff640 1 -- 172.21.15.33:0/3222985435 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9664101840 msgr2=0x7f96640755d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:13.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.825+0000 7f9663fff640 1 --2- 172.21.15.33:0/3222985435 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9664101840 0x7f96640755d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:13.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.825+0000 7f9663fff640 1 -- 172.21.15.33:0/3222985435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96641033f0 msgr2=0x7f9664074150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:13.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.825+0000 7f9663fff640 1 --2- 172.21.15.33:0/3222985435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96641033f0 0x7f9664074150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:13.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.825+0000 7f9663fff640 1 -- 172.21.15.33:0/3222985435 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96640749e0 con 0x7f9664102a30 2024-03-19T02:53:13.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.825+0000 7f96688fc640 1 --2- 172.21.15.33:0/3222985435 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9664101840 0x7f96640755d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:53:13.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.826+0000 7f9663fff640 1 --2- 172.21.15.33:0/3222985435 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9664102a30 0x7f9664073c10 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f965800e9f0 tx=0x7f965800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:13.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.826+0000 7f9661ffb640 1 -- 172.21.15.33:0/3222985435 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f965800cd80 con 0x7f9664102a30 2024-03-19T02:53:13.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.826+0000 7f9661ffb640 1 -- 172.21.15.33:0/3222985435 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f96580108e0 con 0x7f9664102a30 2024-03-19T02:53:13.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.826+0000 7f966ab60640 1 -- 172.21.15.33:0/3222985435 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96641a1bb0 con 0x7f9664102a30 2024-03-19T02:53:13.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.826+0000 7f9661ffb640 1 -- 172.21.15.33:0/3222985435 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9658018920 con 0x7f9664102a30 2024-03-19T02:53:13.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.826+0000 7f966ab60640 1 -- 172.21.15.33:0/3222985435 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96641a2150 con 0x7f9664102a30 2024-03-19T02:53:13.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.827+0000 7f966ab60640 1 -- 172.21.15.33:0/3222985435 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9664108c80 con 0x7f9664102a30 2024-03-19T02:53:13.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.828+0000 7f9661ffb640 1 -- 172.21.15.33:0/3222985435 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9658010a90 con 0x7f9664102a30 2024-03-19T02:53:13.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.829+0000 7f9661ffb640 1 --2- 172.21.15.33:0/3222985435 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f964c0a2880 0x7f964c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:13.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.829+0000 7f96688fc640 1 --2- 172.21.15.33:0/3222985435 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f964c0a2880 0x7f964c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:13.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.829+0000 7f9661ffb640 1 -- 172.21.15.33:0/3222985435 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f96580cf580 con 0x7f9664102a30 2024-03-19T02:53:13.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.829+0000 7f96688fc640 1 --2- 172.21.15.33:0/3222985435 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f964c0a2880 0x7f964c0a4d40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f965c02c700 tx=0x7f965c0324f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:13.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:13.832+0000 7f9661ffb640 1 -- 172.21.15.33:0/3222985435 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f965809c290 con 0x7f9664102a30 2024-03-19T02:53:14.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.018+0000 7f966ab60640 1 -- 172.21.15.33:0/3222985435 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f966402d3f0 con 0x7f9664102a30 2024-03-19T02:53:14.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.018+0000 7f9661ffb640 1 -- 172.21.15.33:0/3222985435 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v97) ==== 93+0+10 (secure 0 0 0) 0x7f96580ce020 con 0x7f9664102a30 2024-03-19T02:53:14.019 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 1 2024-03-19T02:53:14.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.021+0000 7f966ab60640 1 -- 172.21.15.33:0/3222985435 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f964c0a2880 msgr2=0x7f964c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:14.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.021+0000 7f966ab60640 1 --2- 172.21.15.33:0/3222985435 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f964c0a2880 0x7f964c0a4d40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f965c02c700 tx=0x7f965c0324f0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.021+0000 7f966ab60640 1 -- 172.21.15.33:0/3222985435 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9664102a30 msgr2=0x7f9664073c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:14.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.021+0000 7f966ab60640 1 --2- 172.21.15.33:0/3222985435 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9664102a30 0x7f9664073c10 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f965800e9f0 tx=0x7f965800eeb0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.021+0000 7f966ab60640 1 -- 172.21.15.33:0/3222985435 shutdown_connections 2024-03-19T02:53:14.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.021+0000 7f966ab60640 1 --2- 172.21.15.33:0/3222985435 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f964c0a2880 0x7f964c0a4d40 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.021+0000 7f966ab60640 1 --2- 172.21.15.33:0/3222985435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96641033f0 0x7f9664074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.021+0000 7f966ab60640 1 --2- 172.21.15.33:0/3222985435 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9664102a30 0x7f9664073c10 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.021+0000 7f966ab60640 1 --2- 172.21.15.33:0/3222985435 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9664101840 0x7f96640755d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.021+0000 7f966ab60640 1 -- 172.21.15.33:0/3222985435 >> 172.21.15.33:0/3222985435 conn(0x7f96640fcd90 msgr2=0x7f9664105e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:14.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.021+0000 7f966ab60640 1 -- 172.21.15.33:0/3222985435 shutdown_connections 2024-03-19T02:53:14.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.021+0000 7f966ab60640 1 -- 172.21.15.33:0/3222985435 wait complete. 2024-03-19T02:53:14.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:13 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1591345445' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:53:14.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:13 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1890446709' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:53:14.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:13 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3245887824' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:53:14.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:13 smithi033 ceph-mon[31973]: pgmap v374: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-19T02:53:14.033 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T02:53:14.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.138+0000 7f0ef4132640 1 -- 172.21.15.33:0/116102652 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0eec101840 msgr2=0x7f0eec101c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:14.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.138+0000 7f0ef4132640 1 --2- 172.21.15.33:0/116102652 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0eec101840 0x7f0eec101c60 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009900 tx=0x7f0ee802bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:14.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.138+0000 7f0ef4132640 1 -- 172.21.15.33:0/116102652 shutdown_connections 2024-03-19T02:53:14.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.138+0000 7f0ef4132640 1 --2- 172.21.15.33:0/116102652 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0eec102a30 0x7f0eec102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.138+0000 7f0ef4132640 1 --2- 172.21.15.33:0/116102652 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0eec1033f0 0x7f0eec109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.138+0000 7f0ef4132640 1 --2- 172.21.15.33:0/116102652 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0eec101840 0x7f0eec101c60 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.138+0000 7f0ef4132640 1 -- 172.21.15.33:0/116102652 >> 172.21.15.33:0/116102652 conn(0x7f0eec0fcdb0 msgr2=0x7f0eec0ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:14.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.138+0000 7f0ef4132640 1 -- 172.21.15.33:0/116102652 shutdown_connections 2024-03-19T02:53:14.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.138+0000 7f0ef4132640 1 -- 172.21.15.33:0/116102652 wait complete. 2024-03-19T02:53:14.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.138+0000 7f0ef4132640 1 Processor -- start 2024-03-19T02:53:14.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.139+0000 7f0ef4132640 1 -- start start 2024-03-19T02:53:14.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.139+0000 7f0ef4132640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0eec101840 0x7f0eec197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:14.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.139+0000 7f0ef4132640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0eec102a30 0x7f0eec198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:14.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.139+0000 7f0ef4132640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0eec1033f0 0x7f0eec19f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:14.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.139+0000 7f0ef4132640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0eec10d360 con 0x7f0eec1033f0 2024-03-19T02:53:14.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.139+0000 7f0ef4132640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0eec10d1e0 con 0x7f0eec102a30 2024-03-19T02:53:14.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.139+0000 7f0ef4132640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f0eec10d4e0 con 0x7f0eec101840 2024-03-19T02:53:14.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.139+0000 7f0ef26cf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0eec1033f0 0x7f0eec19f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:14.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.139+0000 7f0ef16cd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0eec102a30 0x7f0eec198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:14.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.139+0000 7f0ef26cf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0eec1033f0 0x7f0eec19f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46704/0 (socket says 172.21.15.33:46704) 2024-03-19T02:53:14.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.139+0000 7f0ef16cd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0eec102a30 0x7f0eec198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:35294/0 (socket says 172.21.15.33:35294) 2024-03-19T02:53:14.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.139+0000 7f0ef26cf640 1 -- 172.21.15.33:0/4282364167 learned_addr learned my addr 172.21.15.33:0/4282364167 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:14.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.139+0000 7f0ef1ece640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0eec101840 0x7f0eec197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:14.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.139+0000 7f0ef26cf640 1 -- 172.21.15.33:0/4282364167 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0eec101840 msgr2=0x7f0eec197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:14.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.139+0000 7f0ef26cf640 1 --2- 172.21.15.33:0/4282364167 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0eec101840 0x7f0eec197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.139+0000 7f0ef26cf640 1 -- 172.21.15.33:0/4282364167 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0eec102a30 msgr2=0x7f0eec198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.139+0000 7f0ef26cf640 1 --2- 172.21.15.33:0/4282364167 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0eec102a30 0x7f0eec198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.139+0000 7f0ef26cf640 1 -- 172.21.15.33:0/4282364167 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eec19f700 con 0x7f0eec1033f0 2024-03-19T02:53:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.139+0000 7f0ef16cd640 1 --2- 172.21.15.33:0/4282364167 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0eec102a30 0x7f0eec198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.139+0000 7f0ef26cf640 1 --2- 172.21.15.33:0/4282364167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0eec1033f0 0x7f0eec19f090 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f0ee400efd0 tx=0x7f0ee400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.139+0000 7f0ee2ffd640 1 -- 172.21.15.33:0/4282364167 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4003b70 con 0x7f0eec1033f0 2024-03-19T02:53:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.139+0000 7f0ee2ffd640 1 -- 172.21.15.33:0/4282364167 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0ee4003d10 con 0x7f0eec1033f0 2024-03-19T02:53:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.139+0000 7f0ef4132640 1 -- 172.21.15.33:0/4282364167 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0eec19f9f0 con 0x7f0eec1033f0 2024-03-19T02:53:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.139+0000 7f0ee2ffd640 1 -- 172.21.15.33:0/4282364167 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4010980 con 0x7f0eec1033f0 2024-03-19T02:53:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.140+0000 7f0ef4132640 1 -- 172.21.15.33:0/4282364167 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0eec19fef0 con 0x7f0eec1033f0 2024-03-19T02:53:14.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.140+0000 7f0ef4132640 1 -- 172.21.15.33:0/4282364167 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0eec108c80 con 0x7f0eec1033f0 2024-03-19T02:53:14.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.141+0000 7f0ee2ffd640 1 -- 172.21.15.33:0/4282364167 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0ee4004020 con 0x7f0eec1033f0 2024-03-19T02:53:14.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.142+0000 7f0ee2ffd640 1 --2- 172.21.15.33:0/4282364167 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f0ed80a2800 0x7f0ed80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:14.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.142+0000 7f0ef1ece640 1 --2- 172.21.15.33:0/4282364167 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f0ed80a2800 0x7f0ed80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:14.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.142+0000 7f0ee2ffd640 1 -- 172.21.15.33:0/4282364167 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f0ee40cf250 con 0x7f0eec1033f0 2024-03-19T02:53:14.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.142+0000 7f0ef1ece640 1 --2- 172.21.15.33:0/4282364167 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f0ed80a2800 0x7f0ed80a4cc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8005ab0 tx=0x7f0ee8005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:14.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.146+0000 7f0ee2ffd640 1 -- 172.21.15.33:0/4282364167 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0ee409bf60 con 0x7f0eec1033f0 2024-03-19T02:53:14.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:13 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1591345445' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:53:14.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:13 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1890446709' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:53:14.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:13 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3245887824' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:53:14.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:13 smithi096 ceph-mon[35946]: pgmap v374: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-19T02:53:14.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:13 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1591345445' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:53:14.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:13 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1890446709' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:53:14.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:13 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3245887824' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:53:14.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:13 smithi155 ceph-mon[35804]: pgmap v374: 129 pgs: 129 active+clean; 642 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-19T02:53:14.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.330+0000 7f0ef4132640 1 -- 172.21.15.33:0/4282364167 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0eec062420 con 0x7f0eec1033f0 2024-03-19T02:53:14.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.330+0000 7f0ee2ffd640 1 -- 172.21.15.33:0/4282364167 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v97) ==== 104+0+11 (secure 0 0 0) 0x7f0ee409b860 con 0x7f0eec1033f0 2024-03-19T02:53:14.332 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 32 2024-03-19T02:53:14.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.332+0000 7f0ef4132640 1 -- 172.21.15.33:0/4282364167 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f0ed80a2800 msgr2=0x7f0ed80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:14.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.332+0000 7f0ef4132640 1 --2- 172.21.15.33:0/4282364167 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f0ed80a2800 0x7f0ed80a4cc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8005ab0 tx=0x7f0ee8005fd0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.332+0000 7f0ef4132640 1 -- 172.21.15.33:0/4282364167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0eec1033f0 msgr2=0x7f0eec19f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:14.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.332+0000 7f0ef4132640 1 --2- 172.21.15.33:0/4282364167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0eec1033f0 0x7f0eec19f090 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f0ee400efd0 tx=0x7f0ee400c6a0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.333+0000 7f0ef4132640 1 -- 172.21.15.33:0/4282364167 shutdown_connections 2024-03-19T02:53:14.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.333+0000 7f0ef4132640 1 --2- 172.21.15.33:0/4282364167 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f0ed80a2800 0x7f0ed80a4cc0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.333+0000 7f0ef4132640 1 --2- 172.21.15.33:0/4282364167 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0eec1033f0 0x7f0eec19f090 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.333+0000 7f0ef4132640 1 --2- 172.21.15.33:0/4282364167 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0eec102a30 0x7f0eec198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.333+0000 7f0ef4132640 1 --2- 172.21.15.33:0/4282364167 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0eec101840 0x7f0eec197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.333+0000 7f0ef4132640 1 -- 172.21.15.33:0/4282364167 >> 172.21.15.33:0/4282364167 conn(0x7f0eec0fcdb0 msgr2=0x7f0eec105e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:14.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.333+0000 7f0ef4132640 1 -- 172.21.15.33:0/4282364167 shutdown_connections 2024-03-19T02:53:14.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.333+0000 7f0ef4132640 1 -- 172.21.15.33:0/4282364167 wait complete. 2024-03-19T02:53:14.345 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T02:53:14.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.450+0000 7f06e4812640 1 -- 172.21.15.33:0/584989031 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f06e01033f0 msgr2=0x7f06e0109c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:14.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.450+0000 7f06e4812640 1 --2- 172.21.15.33:0/584989031 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f06e01033f0 0x7f06e0109c80 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f06d8009ca0 tx=0x7f06d802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.450+0000 7f06e4812640 1 -- 172.21.15.33:0/584989031 shutdown_connections 2024-03-19T02:53:14.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.450+0000 7f06e4812640 1 --2- 172.21.15.33:0/584989031 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f06e01033f0 0x7f06e0109c80 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.450+0000 7f06e4812640 1 --2- 172.21.15.33:0/584989031 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06e0102a30 0x7f06e0102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.450+0000 7f06e4812640 1 --2- 172.21.15.33:0/584989031 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06e0101840 0x7f06e0101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.450+0000 7f06e4812640 1 -- 172.21.15.33:0/584989031 >> 172.21.15.33:0/584989031 conn(0x7f06e00fcdb0 msgr2=0x7f06e00ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:14.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.450+0000 7f06e4812640 1 -- 172.21.15.33:0/584989031 shutdown_connections 2024-03-19T02:53:14.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.450+0000 7f06e4812640 1 -- 172.21.15.33:0/584989031 wait complete. 2024-03-19T02:53:14.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.450+0000 7f06e4812640 1 Processor -- start 2024-03-19T02:53:14.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.450+0000 7f06e4812640 1 -- start start 2024-03-19T02:53:14.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.450+0000 7f06e4812640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f06e0101840 0x7f06e0197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:14.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.450+0000 7f06e4812640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06e0102a30 0x7f06e0197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:14.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.451+0000 7f06e4812640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06e01033f0 0x7f06e019efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:14.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.451+0000 7f06e4812640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f06e010d370 con 0x7f06e0101840 2024-03-19T02:53:14.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.451+0000 7f06e4812640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f06e010d1f0 con 0x7f06e0102a30 2024-03-19T02:53:14.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.451+0000 7f06e4812640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f06e010d4f0 con 0x7f06e01033f0 2024-03-19T02:53:14.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.451+0000 7f06de59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f06e0101840 0x7f06e0197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:14.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.451+0000 7f06de59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f06e0101840 0x7f06e0197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46722/0 (socket says 172.21.15.33:46722) 2024-03-19T02:53:14.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.451+0000 7f06de59c640 1 -- 172.21.15.33:0/3684401948 learned_addr learned my addr 172.21.15.33:0/3684401948 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:14.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.451+0000 7f06d7fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06e0102a30 0x7f06e0197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:14.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.451+0000 7f06ded9d640 1 --2- 172.21.15.33:0/3684401948 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06e01033f0 0x7f06e019efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:14.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.451+0000 7f06de59c640 1 -- 172.21.15.33:0/3684401948 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06e01033f0 msgr2=0x7f06e019efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:14.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.451+0000 7f06de59c640 1 --2- 172.21.15.33:0/3684401948 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06e01033f0 0x7f06e019efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.451+0000 7f06de59c640 1 -- 172.21.15.33:0/3684401948 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06e0102a30 msgr2=0x7f06e0197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:14.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.451+0000 7f06de59c640 1 --2- 172.21.15.33:0/3684401948 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06e0102a30 0x7f06e0197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.451+0000 7f06de59c640 1 -- 172.21.15.33:0/3684401948 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06e019f630 con 0x7f06e0101840 2024-03-19T02:53:14.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.451+0000 7f06de59c640 1 --2- 172.21.15.33:0/3684401948 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f06e0101840 0x7f06e0197a10 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f06d000e9c0 tx=0x7f06d000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:14.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.451+0000 7f06d77fe640 1 -- 172.21.15.33:0/3684401948 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d000cd80 con 0x7f06e0101840 2024-03-19T02:53:14.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.451+0000 7f06d77fe640 1 -- 172.21.15.33:0/3684401948 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f06d00035f0 con 0x7f06e0101840 2024-03-19T02:53:14.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.451+0000 7f06e4812640 1 -- 172.21.15.33:0/3684401948 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06e019f920 con 0x7f06e0101840 2024-03-19T02:53:14.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.451+0000 7f06d77fe640 1 -- 172.21.15.33:0/3684401948 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d0010690 con 0x7f06e0101840 2024-03-19T02:53:14.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.451+0000 7f06e4812640 1 -- 172.21.15.33:0/3684401948 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06e019fe70 con 0x7f06e0101840 2024-03-19T02:53:14.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.452+0000 7f06e4812640 1 -- 172.21.15.33:0/3684401948 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06e0108b60 con 0x7f06e0101840 2024-03-19T02:53:14.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.453+0000 7f06d77fe640 1 -- 172.21.15.33:0/3684401948 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f06d00037a0 con 0x7f06e0101840 2024-03-19T02:53:14.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.454+0000 7f06d77fe640 1 --2- 172.21.15.33:0/3684401948 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f06cc0a2880 0x7f06cc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:14.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.454+0000 7f06d7fff640 1 --2- 172.21.15.33:0/3684401948 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f06cc0a2880 0x7f06cc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:14.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.454+0000 7f06d77fe640 1 -- 172.21.15.33:0/3684401948 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f06d00cf380 con 0x7f06e0101840 2024-03-19T02:53:14.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.455+0000 7f06d7fff640 1 --2- 172.21.15.33:0/3684401948 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f06cc0a2880 0x7f06cc0a4d40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f06e0198f20 tx=0x7f06c8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:14.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.458+0000 7f06d77fe640 1 -- 172.21.15.33:0/3684401948 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f06d009c130 con 0x7f06e0101840 2024-03-19T02:53:14.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.641+0000 7f06e4812640 1 -- 172.21.15.33:0/3684401948 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f06e002d3a0 con 0x7f06e0101840 2024-03-19T02:53:14.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.641+0000 7f06d77fe640 1 -- 172.21.15.33:0/3684401948 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v97) ==== 100+0+11 (secure 0 0 0) 0x7f06d009ba30 con 0x7f06e0101840 2024-03-19T02:53:14.643 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 64 2024-03-19T02:53:14.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.643+0000 7f06e4812640 1 -- 172.21.15.33:0/3684401948 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f06cc0a2880 msgr2=0x7f06cc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:14.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.643+0000 7f06e4812640 1 --2- 172.21.15.33:0/3684401948 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f06cc0a2880 0x7f06cc0a4d40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f06e0198f20 tx=0x7f06c8005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:14.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.643+0000 7f06e4812640 1 -- 172.21.15.33:0/3684401948 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f06e0101840 msgr2=0x7f06e0197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:14.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.643+0000 7f06e4812640 1 --2- 172.21.15.33:0/3684401948 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f06e0101840 0x7f06e0197a10 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f06d000e9c0 tx=0x7f06d000ee80 comp rx=0 tx=0).stop 2024-03-19T02:53:14.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.643+0000 7f06e4812640 1 -- 172.21.15.33:0/3684401948 shutdown_connections 2024-03-19T02:53:14.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.643+0000 7f06e4812640 1 --2- 172.21.15.33:0/3684401948 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f06cc0a2880 0x7f06cc0a4d40 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.643+0000 7f06e4812640 1 --2- 172.21.15.33:0/3684401948 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06e01033f0 0x7f06e019efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.643+0000 7f06e4812640 1 --2- 172.21.15.33:0/3684401948 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f06e0102a30 0x7f06e0197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.643+0000 7f06e4812640 1 --2- 172.21.15.33:0/3684401948 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f06e0101840 0x7f06e0197a10 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.643+0000 7f06e4812640 1 -- 172.21.15.33:0/3684401948 >> 172.21.15.33:0/3684401948 conn(0x7f06e00fcdb0 msgr2=0x7f06e0105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:14.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.643+0000 7f06e4812640 1 -- 172.21.15.33:0/3684401948 shutdown_connections 2024-03-19T02:53:14.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.643+0000 7f06e4812640 1 -- 172.21.15.33:0/3684401948 wait complete. 2024-03-19T02:53:14.656 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T02:53:14.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.760+0000 7fafd2e85640 1 -- 172.21.15.33:0/3433618817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafcc1022f0 msgr2=0x7fafcc108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:14.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.760+0000 7fafd2e85640 1 --2- 172.21.15.33:0/3433618817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafcc1022f0 0x7fafcc108b80 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fafc4009ca0 tx=0x7fafc402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.760+0000 7fafd2e85640 1 -- 172.21.15.33:0/3433618817 shutdown_connections 2024-03-19T02:53:14.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.760+0000 7fafd2e85640 1 --2- 172.21.15.33:0/3433618817 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fafcc101930 0x7fafcc101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.760+0000 7fafd2e85640 1 --2- 172.21.15.33:0/3433618817 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafcc1022f0 0x7fafcc108b80 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.760+0000 7fafd2e85640 1 --2- 172.21.15.33:0/3433618817 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafcc100740 0x7fafcc100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.760+0000 7fafd2e85640 1 -- 172.21.15.33:0/3433618817 >> 172.21.15.33:0/3433618817 conn(0x7fafcc0fbcd0 msgr2=0x7fafcc0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:14.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.760+0000 7fafd2e85640 1 -- 172.21.15.33:0/3433618817 shutdown_connections 2024-03-19T02:53:14.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.760+0000 7fafd2e85640 1 -- 172.21.15.33:0/3433618817 wait complete. 2024-03-19T02:53:14.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.761+0000 7fafd2e85640 1 Processor -- start 2024-03-19T02:53:14.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.761+0000 7fafd2e85640 1 -- start start 2024-03-19T02:53:14.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.761+0000 7fafd2e85640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fafcc100740 0x7fafcc1979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:14.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.761+0000 7fafd2e85640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafcc101930 0x7fafcc197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:14.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.761+0000 7fafd2e85640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafcc1022f0 0x7fafcc19efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:14.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.761+0000 7fafd2e85640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fafcc10d390 con 0x7fafcc1022f0 2024-03-19T02:53:14.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.761+0000 7fafd2e85640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fafcc10d210 con 0x7fafcc101930 2024-03-19T02:53:14.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.761+0000 7fafd2e85640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fafcc10d510 con 0x7fafcc100740 2024-03-19T02:53:14.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.761+0000 7fafd1422640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafcc1022f0 0x7fafcc19efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:14.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.761+0000 7fafd1422640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafcc1022f0 0x7fafcc19efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46736/0 (socket says 172.21.15.33:46736) 2024-03-19T02:53:14.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.761+0000 7fafd1422640 1 -- 172.21.15.33:0/2008444438 learned_addr learned my addr 172.21.15.33:0/2008444438 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:14.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.761+0000 7fafd0c21640 1 --2- 172.21.15.33:0/2008444438 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fafcc100740 0x7fafcc1979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:14.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.761+0000 7fafcbfff640 1 --2- 172.21.15.33:0/2008444438 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafcc101930 0x7fafcc197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:14.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.762+0000 7fafd1422640 1 -- 172.21.15.33:0/2008444438 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fafcc100740 msgr2=0x7fafcc1979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:14.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.762+0000 7fafd1422640 1 --2- 172.21.15.33:0/2008444438 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fafcc100740 0x7fafcc1979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.762+0000 7fafd1422640 1 -- 172.21.15.33:0/2008444438 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafcc101930 msgr2=0x7fafcc197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:14.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.762+0000 7fafd1422640 1 --2- 172.21.15.33:0/2008444438 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafcc101930 0x7fafcc197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.762+0000 7fafd1422640 1 -- 172.21.15.33:0/2008444438 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafcc19f610 con 0x7fafcc1022f0 2024-03-19T02:53:14.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.762+0000 7fafd1422640 1 --2- 172.21.15.33:0/2008444438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafcc1022f0 0x7fafcc19efa0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fafc402c700 tx=0x7fafc40328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:14.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.762+0000 7fafc9ffb640 1 -- 172.21.15.33:0/2008444438 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc4041020 con 0x7fafcc1022f0 2024-03-19T02:53:14.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.762+0000 7fafc9ffb640 1 -- 172.21.15.33:0/2008444438 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fafc4003e20 con 0x7fafcc1022f0 2024-03-19T02:53:14.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.762+0000 7fafc9ffb640 1 -- 172.21.15.33:0/2008444438 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc4039040 con 0x7fafcc1022f0 2024-03-19T02:53:14.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.762+0000 7fafd2e85640 1 -- 172.21.15.33:0/2008444438 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafcc19f8a0 con 0x7fafcc1022f0 2024-03-19T02:53:14.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.762+0000 7fafd2e85640 1 -- 172.21.15.33:0/2008444438 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafcc19fde0 con 0x7fafcc1022f0 2024-03-19T02:53:14.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.763+0000 7fafd2e85640 1 -- 172.21.15.33:0/2008444438 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafcc107ae0 con 0x7fafcc1022f0 2024-03-19T02:53:14.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.764+0000 7fafc9ffb640 1 -- 172.21.15.33:0/2008444438 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fafc4044020 con 0x7fafcc1022f0 2024-03-19T02:53:14.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.765+0000 7fafc9ffb640 1 --2- 172.21.15.33:0/2008444438 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fafbc0a2880 0x7fafbc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:14.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.765+0000 7fafd0c21640 1 --2- 172.21.15.33:0/2008444438 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fafbc0a2880 0x7fafbc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:14.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.765+0000 7fafc9ffb640 1 -- 172.21.15.33:0/2008444438 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7fafc40f13a0 con 0x7fafcc1022f0 2024-03-19T02:53:14.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.765+0000 7fafd0c21640 1 --2- 172.21.15.33:0/2008444438 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fafbc0a2880 0x7fafbc0a4d40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fafcc101790 tx=0x7fafc0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:14.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.768+0000 7fafc9ffb640 1 -- 172.21.15.33:0/2008444438 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fafc404b020 con 0x7fafcc1022f0 2024-03-19T02:53:14.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.950+0000 7fafd2e85640 1 -- 172.21.15.33:0/2008444438 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fafcc02d380 con 0x7fafcc1022f0 2024-03-19T02:53:14.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.950+0000 7fafc9ffb640 1 -- 172.21.15.33:0/2008444438 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v97) ==== 103+0+11 (secure 0 0 0) 0x7fafc40be050 con 0x7fafcc1022f0 2024-03-19T02:53:14.952 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 32 2024-03-19T02:53:14.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.953+0000 7fafd2e85640 1 -- 172.21.15.33:0/2008444438 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fafbc0a2880 msgr2=0x7fafbc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:14.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.953+0000 7fafd2e85640 1 --2- 172.21.15.33:0/2008444438 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fafbc0a2880 0x7fafbc0a4d40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fafcc101790 tx=0x7fafc0005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:14.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.953+0000 7fafd2e85640 1 -- 172.21.15.33:0/2008444438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafcc1022f0 msgr2=0x7fafcc19efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:14.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.953+0000 7fafd2e85640 1 --2- 172.21.15.33:0/2008444438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafcc1022f0 0x7fafcc19efa0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fafc402c700 tx=0x7fafc40328a0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.953+0000 7fafd2e85640 1 -- 172.21.15.33:0/2008444438 shutdown_connections 2024-03-19T02:53:14.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.953+0000 7fafd2e85640 1 --2- 172.21.15.33:0/2008444438 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fafbc0a2880 0x7fafbc0a4d40 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.953+0000 7fafd2e85640 1 --2- 172.21.15.33:0/2008444438 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fafcc1022f0 0x7fafcc19efa0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.953+0000 7fafd2e85640 1 --2- 172.21.15.33:0/2008444438 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fafcc101930 0x7fafcc197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.953+0000 7fafd2e85640 1 --2- 172.21.15.33:0/2008444438 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fafcc100740 0x7fafcc1979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:14.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.953+0000 7fafd2e85640 1 -- 172.21.15.33:0/2008444438 >> 172.21.15.33:0/2008444438 conn(0x7fafcc0fbcd0 msgr2=0x7fafcc104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:14.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.953+0000 7fafd2e85640 1 -- 172.21.15.33:0/2008444438 shutdown_connections 2024-03-19T02:53:14.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:14.953+0000 7fafd2e85640 1 -- 172.21.15.33:0/2008444438 wait complete. 2024-03-19T02:53:14.964 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:14 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3222985435' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:53:14.964 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:14 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/4282364167' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:53:14.964 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:14 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3684401948' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:53:14.965 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-19T02:53:14.965 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:53:15.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.067+0000 7f27ab081640 1 -- 172.21.15.33:0/3547664027 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27a41075a0 msgr2=0x7f27a41099f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:15.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.067+0000 7f27ab081640 1 --2- 172.21.15.33:0/3547664027 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27a41075a0 0x7f27a41099f0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f279c009ca0 tx=0x7f279c02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:53:15.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.067+0000 7f27ab081640 1 -- 172.21.15.33:0/3547664027 shutdown_connections 2024-03-19T02:53:15.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.067+0000 7f27ab081640 1 --2- 172.21.15.33:0/3547664027 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27a4104c70 0x7f27a4107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:15.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.067+0000 7f27ab081640 1 --2- 172.21.15.33:0/3547664027 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27a41075a0 0x7f27a41099f0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:15.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.067+0000 7f27ab081640 1 --2- 172.21.15.33:0/3547664027 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27a4102340 0x7f27a4104730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:15.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.067+0000 7f27ab081640 1 -- 172.21.15.33:0/3547664027 >> 172.21.15.33:0/3547664027 conn(0x7f27a40fbd50 msgr2=0x7f27a40fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:15.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.067+0000 7f27ab081640 1 -- 172.21.15.33:0/3547664027 shutdown_connections 2024-03-19T02:53:15.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.067+0000 7f27ab081640 1 -- 172.21.15.33:0/3547664027 wait complete. 2024-03-19T02:53:15.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.067+0000 7f27ab081640 1 Processor -- start 2024-03-19T02:53:15.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.067+0000 7f27ab081640 1 -- start start 2024-03-19T02:53:15.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.067+0000 7f27ab081640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27a4102340 0x7f27a4197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:15.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.067+0000 7f27ab081640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27a4104c70 0x7f27a4197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:15.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.067+0000 7f27ab081640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27a41075a0 0x7f27a419efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:15.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.067+0000 7f27ab081640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f27a410d3a0 con 0x7f27a4102340 2024-03-19T02:53:15.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.067+0000 7f27ab081640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f27a410d220 con 0x7f27a41075a0 2024-03-19T02:53:15.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.067+0000 7f27ab081640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f27a410d520 con 0x7f27a4104c70 2024-03-19T02:53:15.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.067+0000 7f27a8e1d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27a4102340 0x7f27a4197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:15.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.068+0000 7f27a8e1d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27a4102340 0x7f27a4197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46750/0 (socket says 172.21.15.33:46750) 2024-03-19T02:53:15.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.068+0000 7f27a8e1d640 1 -- 172.21.15.33:0/297733093 learned_addr learned my addr 172.21.15.33:0/297733093 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:15.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.068+0000 7f27a961e640 1 --2- 172.21.15.33:0/297733093 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27a41075a0 0x7f27a419efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:15.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.068+0000 7f27a3fff640 1 --2- 172.21.15.33:0/297733093 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27a4104c70 0x7f27a4197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:15.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.068+0000 7f27a8e1d640 1 -- 172.21.15.33:0/297733093 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27a4104c70 msgr2=0x7f27a4197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:15.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.068+0000 7f27a8e1d640 1 --2- 172.21.15.33:0/297733093 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27a4104c70 0x7f27a4197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:15.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.068+0000 7f27a8e1d640 1 -- 172.21.15.33:0/297733093 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27a41075a0 msgr2=0x7f27a419efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:15.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.068+0000 7f27a8e1d640 1 --2- 172.21.15.33:0/297733093 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27a41075a0 0x7f27a419efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:15.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.068+0000 7f27a8e1d640 1 -- 172.21.15.33:0/297733093 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27a419f640 con 0x7f27a4102340 2024-03-19T02:53:15.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.068+0000 7f27a961e640 1 --2- 172.21.15.33:0/297733093 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27a41075a0 0x7f27a419efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:15.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.068+0000 7f27a8e1d640 1 --2- 172.21.15.33:0/297733093 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27a4102340 0x7f27a4197a20 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f279400ea60 tx=0x7f279400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:15.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.068+0000 7f27a1ffb640 1 -- 172.21.15.33:0/297733093 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f279400cde0 con 0x7f27a4102340 2024-03-19T02:53:15.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.068+0000 7f27a1ffb640 1 -- 172.21.15.33:0/297733093 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f27940035f0 con 0x7f27a4102340 2024-03-19T02:53:15.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.068+0000 7f27ab081640 1 -- 172.21.15.33:0/297733093 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27a419f930 con 0x7f27a4102340 2024-03-19T02:53:15.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.068+0000 7f27a1ffb640 1 -- 172.21.15.33:0/297733093 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2794010690 con 0x7f27a4102340 2024-03-19T02:53:15.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.068+0000 7f27ab081640 1 -- 172.21.15.33:0/297733093 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27a419fe80 con 0x7f27a4102340 2024-03-19T02:53:15.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.069+0000 7f27ab081640 1 -- 172.21.15.33:0/297733093 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27a40fd840 con 0x7f27a4102340 2024-03-19T02:53:15.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.070+0000 7f27a1ffb640 1 -- 172.21.15.33:0/297733093 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f27940037a0 con 0x7f27a4102340 2024-03-19T02:53:15.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.071+0000 7f27a1ffb640 1 --2- 172.21.15.33:0/297733093 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f278c0a2800 0x7f278c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:15.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.071+0000 7f27a1ffb640 1 -- 172.21.15.33:0/297733093 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f27940cf0e0 con 0x7f27a4102340 2024-03-19T02:53:15.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.071+0000 7f27a3fff640 1 --2- 172.21.15.33:0/297733093 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f278c0a2800 0x7f278c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:15.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.071+0000 7f27a3fff640 1 --2- 172.21.15.33:0/297733093 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f278c0a2800 0x7f278c0a4cc0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f27a4198f30 tx=0x7f2798005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:15.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.075+0000 7f27a1ffb640 1 -- 172.21.15.33:0/297733093 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f279409bd90 con 0x7f27a4102340 2024-03-19T02:53:15.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:14 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3222985435' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:53:15.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:14 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/4282364167' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:53:15.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:14 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3684401948' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:53:15.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:14 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3222985435' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:53:15.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:14 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/4282364167' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:53:15.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:14 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3684401948' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:53:15.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.300+0000 7f27ab081640 1 -- 172.21.15.33:0/297733093 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f27a4062420 con 0x7f27a4102340 2024-03-19T02:53:15.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.301+0000 7f27a1ffb640 1 -- 172.21.15.33:0/297733093 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11867 (secure 0 0 0) 0x7f279409b690 con 0x7f27a4102340 2024-03-19T02:53:15.302 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:53:15.302 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:50:52.201798+0000","modified":"2024-03-19T02:53:11.748697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24488,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/3731232169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":3731232169},{"type":"v1","addr":"172.21.15.96:6839","nonce":3731232169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24457,24488,24509]},"id":1}]} 2024-03-19T02:53:15.302 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-03-19T02:53:15.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.303+0000 7f27ab081640 1 -- 172.21.15.33:0/297733093 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f278c0a2800 msgr2=0x7f278c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:15.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.303+0000 7f27ab081640 1 --2- 172.21.15.33:0/297733093 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f278c0a2800 0x7f278c0a4cc0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f27a4198f30 tx=0x7f2798005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:15.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.303+0000 7f27ab081640 1 -- 172.21.15.33:0/297733093 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27a4102340 msgr2=0x7f27a4197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:15.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.303+0000 7f27ab081640 1 --2- 172.21.15.33:0/297733093 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27a4102340 0x7f27a4197a20 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f279400ea60 tx=0x7f279400ef20 comp rx=0 tx=0).stop 2024-03-19T02:53:15.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.303+0000 7f27ab081640 1 -- 172.21.15.33:0/297733093 shutdown_connections 2024-03-19T02:53:15.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.303+0000 7f27ab081640 1 --2- 172.21.15.33:0/297733093 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f278c0a2800 0x7f278c0a4cc0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:15.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.303+0000 7f27ab081640 1 --2- 172.21.15.33:0/297733093 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27a4104c70 0x7f27a4197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:15.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.303+0000 7f27ab081640 1 --2- 172.21.15.33:0/297733093 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27a41075a0 0x7f27a419efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:15.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.303+0000 7f27ab081640 1 --2- 172.21.15.33:0/297733093 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27a4102340 0x7f27a4197a20 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:15.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.303+0000 7f27ab081640 1 -- 172.21.15.33:0/297733093 >> 172.21.15.33:0/297733093 conn(0x7f27a40fbd50 msgr2=0x7f27a4102c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:15.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.303+0000 7f27ab081640 1 -- 172.21.15.33:0/297733093 shutdown_connections 2024-03-19T02:53:15.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.303+0000 7f27ab081640 1 -- 172.21.15.33:0/297733093 wait complete. 2024-03-19T02:53:15.315 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-19T02:53:15.316 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-19T02:53:15.316 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T02:53:15.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.418+0000 7f7c221a9640 1 -- 172.21.15.33:0/1868610553 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c1c102340 msgr2=0x7f7c1c104730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:15.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.418+0000 7f7c221a9640 1 --2- 172.21.15.33:0/1868610553 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c1c102340 0x7f7c1c104730 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c009860 tx=0x7f7c0c02bb70 comp rx=0 tx=0).stop 2024-03-19T02:53:15.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.418+0000 7f7c221a9640 1 -- 172.21.15.33:0/1868610553 shutdown_connections 2024-03-19T02:53:15.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.418+0000 7f7c221a9640 1 --2- 172.21.15.33:0/1868610553 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c1c1075a0 0x7f7c1c1099f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:15.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.418+0000 7f7c221a9640 1 --2- 172.21.15.33:0/1868610553 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c1c104c70 0x7f7c1c107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:15.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.418+0000 7f7c221a9640 1 --2- 172.21.15.33:0/1868610553 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c1c102340 0x7f7c1c104730 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:15.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.418+0000 7f7c221a9640 1 -- 172.21.15.33:0/1868610553 >> 172.21.15.33:0/1868610553 conn(0x7f7c1c0fbd50 msgr2=0x7f7c1c0fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:15.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.418+0000 7f7c221a9640 1 -- 172.21.15.33:0/1868610553 shutdown_connections 2024-03-19T02:53:15.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.418+0000 7f7c221a9640 1 -- 172.21.15.33:0/1868610553 wait complete. 2024-03-19T02:53:15.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.418+0000 7f7c221a9640 1 Processor -- start 2024-03-19T02:53:15.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.419+0000 7f7c221a9640 1 -- start start 2024-03-19T02:53:15.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.419+0000 7f7c221a9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c1c102340 0x7f7c1c197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:15.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.419+0000 7f7c221a9640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c1c104c70 0x7f7c1c197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:15.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.419+0000 7f7c221a9640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c1c1075a0 0x7f7c1c19f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:15.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.419+0000 7f7c221a9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c10d380 con 0x7f7c1c102340 2024-03-19T02:53:15.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.419+0000 7f7c221a9640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c10d200 con 0x7f7c1c1075a0 2024-03-19T02:53:15.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.419+0000 7f7c221a9640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c10d500 con 0x7f7c1c104c70 2024-03-19T02:53:15.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.419+0000 7f7c1b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c1c102340 0x7f7c1c197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:15.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.419+0000 7f7c1b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c1c102340 0x7f7c1c197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46770/0 (socket says 172.21.15.33:46770) 2024-03-19T02:53:15.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.419+0000 7f7c1b7fe640 1 -- 172.21.15.33:0/3532493939 learned_addr learned my addr 172.21.15.33:0/3532493939 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:15.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.419+0000 7f7c1bfff640 1 --2- 172.21.15.33:0/3532493939 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c1c1075a0 0x7f7c1c19f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:15.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.419+0000 7f7c1affd640 1 --2- 172.21.15.33:0/3532493939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c1c104c70 0x7f7c1c197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:15.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.419+0000 7f7c1b7fe640 1 -- 172.21.15.33:0/3532493939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c1c104c70 msgr2=0x7f7c1c197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:15.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.419+0000 7f7c1b7fe640 1 --2- 172.21.15.33:0/3532493939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c1c104c70 0x7f7c1c197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:15.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.419+0000 7f7c1b7fe640 1 -- 172.21.15.33:0/3532493939 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c1c1075a0 msgr2=0x7f7c1c19f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:15.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.419+0000 7f7c1b7fe640 1 --2- 172.21.15.33:0/3532493939 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c1c1075a0 0x7f7c1c19f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:15.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.419+0000 7f7c1b7fe640 1 -- 172.21.15.33:0/3532493939 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c1c19f6a0 con 0x7f7c1c102340 2024-03-19T02:53:15.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.419+0000 7f7c1b7fe640 1 --2- 172.21.15.33:0/3532493939 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c1c102340 0x7f7c1c197a80 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c005940 tx=0x7f7c0c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:15.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.420+0000 7f7c18ff9640 1 -- 172.21.15.33:0/3532493939 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c041020 con 0x7f7c1c102340 2024-03-19T02:53:15.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.420+0000 7f7c221a9640 1 -- 172.21.15.33:0/3532493939 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c1c19f930 con 0x7f7c1c102340 2024-03-19T02:53:15.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.420+0000 7f7c18ff9640 1 -- 172.21.15.33:0/3532493939 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c0c02c9c0 con 0x7f7c1c102340 2024-03-19T02:53:15.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.420+0000 7f7c18ff9640 1 -- 172.21.15.33:0/3532493939 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c039040 con 0x7f7c1c102340 2024-03-19T02:53:15.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.420+0000 7f7c221a9640 1 -- 172.21.15.33:0/3532493939 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c1c19fe20 con 0x7f7c1c102340 2024-03-19T02:53:15.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.421+0000 7f7c221a9640 1 -- 172.21.15.33:0/3532493939 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c1c0fd840 con 0x7f7c1c102340 2024-03-19T02:53:15.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.422+0000 7f7c18ff9640 1 -- 172.21.15.33:0/3532493939 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7c0c044020 con 0x7f7c1c102340 2024-03-19T02:53:15.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.422+0000 7f7c18ff9640 1 --2- 172.21.15.33:0/3532493939 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7c040a2800 0x7f7c040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:15.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.422+0000 7f7c18ff9640 1 -- 172.21.15.33:0/3532493939 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f7c0c0f0770 con 0x7f7c1c102340 2024-03-19T02:53:15.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.422+0000 7f7c1affd640 1 --2- 172.21.15.33:0/3532493939 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7c040a2800 0x7f7c040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:15.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.423+0000 7f7c1affd640 1 --2- 172.21.15.33:0/3532493939 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7c040a2800 0x7f7c040a4cc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c198f90 tx=0x7f7c10009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:15.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.427+0000 7f7c18ff9640 1 -- 172.21.15.33:0/3532493939 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7c0c04b020 con 0x7f7c1c102340 2024-03-19T02:53:15.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.607+0000 7f7c221a9640 1 -- 172.21.15.33:0/3532493939 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7c1c061c60 con 0x7f7c040a2800 2024-03-19T02:53:15.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.617+0000 7f7c18ff9640 1 -- 172.21.15.33:0/3532493939 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340497 (secure 0 0 0) 0x7f7c1c061c60 con 0x7f7c040a2800 2024-03-19T02:53:15.622 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:53:15.624 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-03-19T02:53:15.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.620+0000 7f7c221a9640 1 -- 172.21.15.33:0/3532493939 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7c040a2800 msgr2=0x7f7c040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:15.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.620+0000 7f7c221a9640 1 --2- 172.21.15.33:0/3532493939 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7c040a2800 0x7f7c040a4cc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c198f90 tx=0x7f7c10009260 comp rx=0 tx=0).stop 2024-03-19T02:53:15.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.620+0000 7f7c221a9640 1 -- 172.21.15.33:0/3532493939 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c1c102340 msgr2=0x7f7c1c197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:15.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.620+0000 7f7c221a9640 1 --2- 172.21.15.33:0/3532493939 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c1c102340 0x7f7c1c197a80 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c005940 tx=0x7f7c0c004270 comp rx=0 tx=0).stop 2024-03-19T02:53:15.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.620+0000 7f7c221a9640 1 -- 172.21.15.33:0/3532493939 shutdown_connections 2024-03-19T02:53:15.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.620+0000 7f7c221a9640 1 --2- 172.21.15.33:0/3532493939 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7c040a2800 0x7f7c040a4cc0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:15.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.620+0000 7f7c221a9640 1 --2- 172.21.15.33:0/3532493939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c1c104c70 0x7f7c1c197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:15.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.620+0000 7f7c221a9640 1 --2- 172.21.15.33:0/3532493939 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c1c1075a0 0x7f7c1c19f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:15.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.620+0000 7f7c221a9640 1 --2- 172.21.15.33:0/3532493939 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c1c102340 0x7f7c1c197a80 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:15.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.620+0000 7f7c221a9640 1 -- 172.21.15.33:0/3532493939 >> 172.21.15.33:0/3532493939 conn(0x7f7c1c0fbd50 msgr2=0x7f7c1c102e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:15.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.620+0000 7f7c221a9640 1 -- 172.21.15.33:0/3532493939 shutdown_connections 2024-03-19T02:53:15.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.620+0000 7f7c221a9640 1 -- 172.21.15.33:0/3532493939 wait complete. 2024-03-19T02:53:15.633 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":374,"stamp":"2024-03-19T02:53:13.741946+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657817,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":830,"num_read_kb":536,"num_write":350,"num_write_kb":3059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1181547,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":783,"ondisk_log_size":783,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":380920,"kb_used_data":15056,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124560904,"statfs":{"total":1151940427776,"available":1151550365696,"internally_reserved":0,"allocated":15417344,"data_stored":8497567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":30,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003027"},"pg_stats":[{"pgid":"3.3f","version":"95'8","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.126776+0000","last_change":"2024-03-19T02:50:52.228368+0000","last_active":"2024-03-19T02:52:41.126776+0000","last_peered":"2024-03-19T02:52:41.126776+0000","last_clean":"2024-03-19T02:52:41.126776+0000","last_became_active":"2024-03-19T02:50:52.228069+0000","last_became_peered":"2024-03-19T02:50:52.228069+0000","last_unstale":"2024-03-19T02:52:41.126776+0000","last_undegraded":"2024-03-19T02:52:41.126776+0000","last_fullsized":"2024-03-19T02:52:41.126776+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:59:34.209095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"95'4","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.230176+0000","last_change":"2024-03-19T02:50:52.231358+0000","last_active":"2024-03-19T02:52:41.230176+0000","last_peered":"2024-03-19T02:52:41.230176+0000","last_clean":"2024-03-19T02:52:41.230176+0000","last_became_active":"2024-03-19T02:50:52.231029+0000","last_became_peered":"2024-03-19T02:50:52.231029+0000","last_unstale":"2024-03-19T02:52:41.230176+0000","last_undegraded":"2024-03-19T02:52:41.230176+0000","last_fullsized":"2024-03-19T02:52:41.230176+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:42:26.031392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"95'7","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.230300+0000","last_change":"2024-03-19T02:50:52.232685+0000","last_active":"2024-03-19T02:52:41.230300+0000","last_peered":"2024-03-19T02:52:41.230300+0000","last_clean":"2024-03-19T02:52:41.230300+0000","last_became_active":"2024-03-19T02:50:52.232361+0000","last_became_peered":"2024-03-19T02:50:52.232361+0000","last_unstale":"2024-03-19T02:52:41.230300+0000","last_undegraded":"2024-03-19T02:52:41.230300+0000","last_fullsized":"2024-03-19T02:52:41.230300+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:53.136200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"95'8","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124441+0000","last_change":"2024-03-19T02:50:52.227648+0000","last_active":"2024-03-19T02:52:41.124441+0000","last_peered":"2024-03-19T02:52:41.124441+0000","last_clean":"2024-03-19T02:52:41.124441+0000","last_became_active":"2024-03-19T02:50:52.226943+0000","last_became_peered":"2024-03-19T02:50:52.226943+0000","last_unstale":"2024-03-19T02:52:41.124441+0000","last_undegraded":"2024-03-19T02:52:41.124441+0000","last_fullsized":"2024-03-19T02:52:41.124441+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:03.293794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"95'4","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.134179+0000","last_change":"2024-03-19T02:50:52.225329+0000","last_active":"2024-03-19T02:52:41.134179+0000","last_peered":"2024-03-19T02:52:41.134179+0000","last_clean":"2024-03-19T02:52:41.134179+0000","last_became_active":"2024-03-19T02:50:52.225012+0000","last_became_peered":"2024-03-19T02:50:52.225012+0000","last_unstale":"2024-03-19T02:52:41.134179+0000","last_undegraded":"2024-03-19T02:52:41.134179+0000","last_fullsized":"2024-03-19T02:52:41.134179+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:32:52.877704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"95'6","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.172669+0000","last_change":"2024-03-19T02:50:52.217399+0000","last_active":"2024-03-19T02:52:41.172669+0000","last_peered":"2024-03-19T02:52:41.172669+0000","last_clean":"2024-03-19T02:52:41.172669+0000","last_became_active":"2024-03-19T02:50:52.216987+0000","last_became_peered":"2024-03-19T02:50:52.216987+0000","last_unstale":"2024-03-19T02:52:41.172669+0000","last_undegraded":"2024-03-19T02:52:41.172669+0000","last_fullsized":"2024-03-19T02:52:41.172669+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:57:15.772546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"95'5","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497233+0000","last_change":"2024-03-19T02:50:52.226755+0000","last_active":"2024-03-19T02:52:41.497233+0000","last_peered":"2024-03-19T02:52:41.497233+0000","last_clean":"2024-03-19T02:52:41.497233+0000","last_became_active":"2024-03-19T02:50:52.226434+0000","last_became_peered":"2024-03-19T02:50:52.226434+0000","last_unstale":"2024-03-19T02:52:41.497233+0000","last_undegraded":"2024-03-19T02:52:41.497233+0000","last_fullsized":"2024-03-19T02:52:41.497233+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:49:42.134262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"95'9","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228846+0000","last_change":"2024-03-19T02:50:52.215848+0000","last_active":"2024-03-19T02:52:41.228846+0000","last_peered":"2024-03-19T02:52:41.228846+0000","last_clean":"2024-03-19T02:52:41.228846+0000","last_became_active":"2024-03-19T02:50:52.215494+0000","last_became_peered":"2024-03-19T02:50:52.215494+0000","last_unstale":"2024-03-19T02:52:41.228846+0000","last_undegraded":"2024-03-19T02:52:41.228846+0000","last_fullsized":"2024-03-19T02:52:41.228846+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:55:10.182359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"95'11","reported_seq":43,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228832+0000","last_change":"2024-03-19T02:50:52.233237+0000","last_active":"2024-03-19T02:52:41.228832+0000","last_peered":"2024-03-19T02:52:41.228832+0000","last_clean":"2024-03-19T02:52:41.228832+0000","last_became_active":"2024-03-19T02:50:52.232091+0000","last_became_peered":"2024-03-19T02:50:52.232091+0000","last_unstale":"2024-03-19T02:52:41.228832+0000","last_undegraded":"2024-03-19T02:52:41.228832+0000","last_fullsized":"2024-03-19T02:52:41.228832+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:08:20.361576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"95'16","reported_seq":55,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.172997+0000","last_change":"2024-03-19T02:51:40.722463+0000","last_active":"2024-03-19T02:52:41.172997+0000","last_peered":"2024-03-19T02:52:41.172997+0000","last_clean":"2024-03-19T02:52:41.172997+0000","last_became_active":"2024-03-19T02:51:40.722120+0000","last_became_peered":"2024-03-19T02:51:40.722120+0000","last_unstale":"2024-03-19T02:52:41.172997+0000","last_undegraded":"2024-03-19T02:52:41.172997+0000","last_fullsized":"2024-03-19T02:52:41.172997+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:49:21.387064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"95'6","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497798+0000","last_change":"2024-03-19T02:50:52.226314+0000","last_active":"2024-03-19T02:52:41.497798+0000","last_peered":"2024-03-19T02:52:41.497798+0000","last_clean":"2024-03-19T02:52:41.497798+0000","last_became_active":"2024-03-19T02:50:52.225919+0000","last_became_peered":"2024-03-19T02:50:52.225919+0000","last_unstale":"2024-03-19T02:52:41.497798+0000","last_undegraded":"2024-03-19T02:52:41.497798+0000","last_fullsized":"2024-03-19T02:52:41.497798+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:42:51.782592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"95'7","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124495+0000","last_change":"2024-03-19T02:50:52.227929+0000","last_active":"2024-03-19T02:52:41.124495+0000","last_peered":"2024-03-19T02:52:41.124495+0000","last_clean":"2024-03-19T02:52:41.124495+0000","last_became_active":"2024-03-19T02:50:52.227360+0000","last_became_peered":"2024-03-19T02:50:52.227360+0000","last_unstale":"2024-03-19T02:52:41.124495+0000","last_undegraded":"2024-03-19T02:52:41.124495+0000","last_fullsized":"2024-03-19T02:52:41.124495+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:24:44.386101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"95'10","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.185886+0000","last_change":"2024-03-19T02:50:52.228088+0000","last_active":"2024-03-19T02:52:41.185886+0000","last_peered":"2024-03-19T02:52:41.185886+0000","last_clean":"2024-03-19T02:52:41.185886+0000","last_became_active":"2024-03-19T02:50:52.227754+0000","last_became_peered":"2024-03-19T02:50:52.227754+0000","last_unstale":"2024-03-19T02:52:41.185886+0000","last_undegraded":"2024-03-19T02:52:41.185886+0000","last_fullsized":"2024-03-19T02:52:41.185886+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:58:57.404241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"95'7","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.229992+0000","last_change":"2024-03-19T02:50:52.226768+0000","last_active":"2024-03-19T02:52:41.229992+0000","last_peered":"2024-03-19T02:52:41.229992+0000","last_clean":"2024-03-19T02:52:41.229992+0000","last_became_active":"2024-03-19T02:50:52.226452+0000","last_became_peered":"2024-03-19T02:50:52.226452+0000","last_unstale":"2024-03-19T02:52:41.229992+0000","last_undegraded":"2024-03-19T02:52:41.229992+0000","last_fullsized":"2024-03-19T02:52:41.229992+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:23:33.250773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"95'7","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.229864+0000","last_change":"2024-03-19T02:50:52.231132+0000","last_active":"2024-03-19T02:52:41.229864+0000","last_peered":"2024-03-19T02:52:41.229864+0000","last_clean":"2024-03-19T02:52:41.229864+0000","last_became_active":"2024-03-19T02:50:52.230778+0000","last_became_peered":"2024-03-19T02:50:52.230778+0000","last_unstale":"2024-03-19T02:52:41.229864+0000","last_undegraded":"2024-03-19T02:52:41.229864+0000","last_fullsized":"2024-03-19T02:52:41.229864+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:33:03.675210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"95'9","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228217+0000","last_change":"2024-03-19T02:50:52.228302+0000","last_active":"2024-03-19T02:52:41.228217+0000","last_peered":"2024-03-19T02:52:41.228217+0000","last_clean":"2024-03-19T02:52:41.228217+0000","last_became_active":"2024-03-19T02:50:52.228007+0000","last_became_peered":"2024-03-19T02:50:52.228007+0000","last_unstale":"2024-03-19T02:52:41.228217+0000","last_undegraded":"2024-03-19T02:52:41.228217+0000","last_fullsized":"2024-03-19T02:52:41.228217+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:59:28.988504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"95'8","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.126552+0000","last_change":"2024-03-19T02:50:52.228640+0000","last_active":"2024-03-19T02:52:41.126552+0000","last_peered":"2024-03-19T02:52:41.126552+0000","last_clean":"2024-03-19T02:52:41.126552+0000","last_became_active":"2024-03-19T02:50:52.228319+0000","last_became_peered":"2024-03-19T02:50:52.228319+0000","last_unstale":"2024-03-19T02:52:41.126552+0000","last_undegraded":"2024-03-19T02:52:41.126552+0000","last_fullsized":"2024-03-19T02:52:41.126552+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:30.356453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"95'10","reported_seq":44,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123572+0000","last_change":"2024-03-19T02:50:52.222300+0000","last_active":"2024-03-19T02:52:41.123572+0000","last_peered":"2024-03-19T02:52:41.123572+0000","last_clean":"2024-03-19T02:52:41.123572+0000","last_became_active":"2024-03-19T02:50:52.221978+0000","last_became_peered":"2024-03-19T02:50:52.221978+0000","last_unstale":"2024-03-19T02:52:41.123572+0000","last_undegraded":"2024-03-19T02:52:41.123572+0000","last_fullsized":"2024-03-19T02:52:41.123572+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:38:36.576987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"95'7","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124386+0000","last_change":"2024-03-19T02:50:52.217640+0000","last_active":"2024-03-19T02:52:41.124386+0000","last_peered":"2024-03-19T02:52:41.124386+0000","last_clean":"2024-03-19T02:52:41.124386+0000","last_became_active":"2024-03-19T02:50:52.217325+0000","last_became_peered":"2024-03-19T02:50:52.217325+0000","last_unstale":"2024-03-19T02:52:41.124386+0000","last_undegraded":"2024-03-19T02:52:41.124386+0000","last_fullsized":"2024-03-19T02:52:41.124386+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:40:25.894843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123733+0000","last_change":"2024-03-19T02:50:57.245323+0000","last_active":"2024-03-19T02:52:41.123733+0000","last_peered":"2024-03-19T02:52:41.123733+0000","last_clean":"2024-03-19T02:52:41.123733+0000","last_became_active":"2024-03-19T02:50:57.244886+0000","last_became_peered":"2024-03-19T02:50:57.244886+0000","last_unstale":"2024-03-19T02:52:41.123733+0000","last_undegraded":"2024-03-19T02:52:41.123733+0000","last_fullsized":"2024-03-19T02:52:41.123733+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:42:09.465738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"93'9","reported_seq":46,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123728+0000","last_change":"2024-03-19T02:50:51.797160+0000","last_active":"2024-03-19T02:52:41.123728+0000","last_peered":"2024-03-19T02:52:41.123728+0000","last_clean":"2024-03-19T02:52:41.123728+0000","last_became_active":"2024-03-19T02:50:51.796783+0000","last_became_peered":"2024-03-19T02:50:51.796783+0000","last_unstale":"2024-03-19T02:52:41.123728+0000","last_undegraded":"2024-03-19T02:52:41.123728+0000","last_fullsized":"2024-03-19T02:52:41.123728+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:57:08.185379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"95'5","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.229622+0000","last_change":"2024-03-19T02:50:52.225114+0000","last_active":"2024-03-19T02:52:41.229622+0000","last_peered":"2024-03-19T02:52:41.229622+0000","last_clean":"2024-03-19T02:52:41.229622+0000","last_became_active":"2024-03-19T02:50:52.220529+0000","last_became_peered":"2024-03-19T02:50:52.220529+0000","last_unstale":"2024-03-19T02:52:41.229622+0000","last_undegraded":"2024-03-19T02:52:41.229622+0000","last_fullsized":"2024-03-19T02:52:41.229622+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:12:51.408689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124051+0000","last_change":"2024-03-19T02:50:57.252827+0000","last_active":"2024-03-19T02:52:41.124051+0000","last_peered":"2024-03-19T02:52:41.124051+0000","last_clean":"2024-03-19T02:52:41.124051+0000","last_became_active":"2024-03-19T02:50:57.252502+0000","last_became_peered":"2024-03-19T02:50:57.252502+0000","last_unstale":"2024-03-19T02:52:41.124051+0000","last_undegraded":"2024-03-19T02:52:41.124051+0000","last_fullsized":"2024-03-19T02:52:41.124051+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:29:36.563533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"93'2","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.229689+0000","last_change":"2024-03-19T02:50:51.797380+0000","last_active":"2024-03-19T02:52:41.229689+0000","last_peered":"2024-03-19T02:52:41.229689+0000","last_clean":"2024-03-19T02:52:41.229689+0000","last_became_active":"2024-03-19T02:50:51.797021+0000","last_became_peered":"2024-03-19T02:50:51.797021+0000","last_unstale":"2024-03-19T02:52:41.229689+0000","last_undegraded":"2024-03-19T02:52:41.229689+0000","last_fullsized":"2024-03-19T02:52:41.229689+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:25.400360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"95'6","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.230538+0000","last_change":"2024-03-19T02:50:52.232717+0000","last_active":"2024-03-19T02:52:41.230538+0000","last_peered":"2024-03-19T02:52:41.230538+0000","last_clean":"2024-03-19T02:52:41.230538+0000","last_became_active":"2024-03-19T02:50:52.232368+0000","last_became_peered":"2024-03-19T02:50:52.232368+0000","last_unstale":"2024-03-19T02:52:41.230538+0000","last_undegraded":"2024-03-19T02:52:41.230538+0000","last_fullsized":"2024-03-19T02:52:41.230538+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:55:38.027120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124226+0000","last_change":"2024-03-19T02:50:57.250840+0000","last_active":"2024-03-19T02:52:41.124226+0000","last_peered":"2024-03-19T02:52:41.124226+0000","last_clean":"2024-03-19T02:52:41.124226+0000","last_became_active":"2024-03-19T02:50:57.250333+0000","last_became_peered":"2024-03-19T02:50:57.250333+0000","last_unstale":"2024-03-19T02:52:41.124226+0000","last_undegraded":"2024-03-19T02:52:41.124226+0000","last_fullsized":"2024-03-19T02:52:41.124226+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:57:09.089939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"93'4","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.134546+0000","last_change":"2024-03-19T02:50:51.796889+0000","last_active":"2024-03-19T02:52:41.134546+0000","last_peered":"2024-03-19T02:52:41.134546+0000","last_clean":"2024-03-19T02:52:41.134546+0000","last_became_active":"2024-03-19T02:50:51.796570+0000","last_became_peered":"2024-03-19T02:50:51.796570+0000","last_unstale":"2024-03-19T02:52:41.134546+0000","last_undegraded":"2024-03-19T02:52:41.134546+0000","last_fullsized":"2024-03-19T02:52:41.134546+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:18:38.301905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"95'9","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228471+0000","last_change":"2024-03-19T02:50:52.224533+0000","last_active":"2024-03-19T02:52:41.228471+0000","last_peered":"2024-03-19T02:52:41.228471+0000","last_clean":"2024-03-19T02:52:41.228471+0000","last_became_active":"2024-03-19T02:50:52.224221+0000","last_became_peered":"2024-03-19T02:50:52.224221+0000","last_unstale":"2024-03-19T02:52:41.228471+0000","last_undegraded":"2024-03-19T02:52:41.228471+0000","last_fullsized":"2024-03-19T02:52:41.228471+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:36:43.335318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124145+0000","last_change":"2024-03-19T02:50:57.252949+0000","last_active":"2024-03-19T02:52:41.124145+0000","last_peered":"2024-03-19T02:52:41.124145+0000","last_clean":"2024-03-19T02:52:41.124145+0000","last_became_active":"2024-03-19T02:50:57.252563+0000","last_became_peered":"2024-03-19T02:50:57.252563+0000","last_unstale":"2024-03-19T02:52:41.124145+0000","last_undegraded":"2024-03-19T02:52:41.124145+0000","last_fullsized":"2024-03-19T02:52:41.124145+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:30:38.244158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.134456+0000","last_change":"2024-03-19T02:50:51.794594+0000","last_active":"2024-03-19T02:52:41.134456+0000","last_peered":"2024-03-19T02:52:41.134456+0000","last_clean":"2024-03-19T02:52:41.134456+0000","last_became_active":"2024-03-19T02:50:51.794197+0000","last_became_peered":"2024-03-19T02:50:51.794197+0000","last_unstale":"2024-03-19T02:52:41.134456+0000","last_undegraded":"2024-03-19T02:52:41.134456+0000","last_fullsized":"2024-03-19T02:52:41.134456+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:58:44.534106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"95'15","reported_seq":48,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.175238+0000","last_change":"2024-03-19T02:50:52.214167+0000","last_active":"2024-03-19T02:52:41.175238+0000","last_peered":"2024-03-19T02:52:41.175238+0000","last_clean":"2024-03-19T02:52:41.175238+0000","last_became_active":"2024-03-19T02:50:52.213873+0000","last_became_peered":"2024-03-19T02:50:52.213873+0000","last_unstale":"2024-03-19T02:52:41.175238+0000","last_undegraded":"2024-03-19T02:52:41.175238+0000","last_fullsized":"2024-03-19T02:52:41.175238+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:28:51.035477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124222+0000","last_change":"2024-03-19T02:50:57.245139+0000","last_active":"2024-03-19T02:52:41.124222+0000","last_peered":"2024-03-19T02:52:41.124222+0000","last_clean":"2024-03-19T02:52:41.124222+0000","last_became_active":"2024-03-19T02:50:57.244636+0000","last_became_peered":"2024-03-19T02:50:57.244636+0000","last_unstale":"2024-03-19T02:52:41.124222+0000","last_undegraded":"2024-03-19T02:52:41.124222+0000","last_fullsized":"2024-03-19T02:52:41.124222+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:44:29.660221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'5","reported_seq":154,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:53:09.201822+0000","last_change":"2024-03-19T02:50:51.795969+0000","last_active":"2024-03-19T02:53:09.201822+0000","last_peered":"2024-03-19T02:53:09.201822+0000","last_clean":"2024-03-19T02:53:09.201822+0000","last_became_active":"2024-03-19T02:50:51.795628+0000","last_became_peered":"2024-03-19T02:50:51.795628+0000","last_unstale":"2024-03-19T02:53:09.201822+0000","last_undegraded":"2024-03-19T02:53:09.201822+0000","last_fullsized":"2024-03-19T02:53:09.201822+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:14:48.463282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":114,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"95'7","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.134019+0000","last_change":"2024-03-19T02:50:52.223278+0000","last_active":"2024-03-19T02:52:41.134019+0000","last_peered":"2024-03-19T02:52:41.134019+0000","last_clean":"2024-03-19T02:52:41.134019+0000","last_became_active":"2024-03-19T02:50:52.222741+0000","last_became_peered":"2024-03-19T02:50:52.222741+0000","last_unstale":"2024-03-19T02:52:41.134019+0000","last_undegraded":"2024-03-19T02:52:41.134019+0000","last_fullsized":"2024-03-19T02:52:41.134019+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:32:09.841825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124151+0000","last_change":"2024-03-19T02:50:57.253130+0000","last_active":"2024-03-19T02:52:41.124151+0000","last_peered":"2024-03-19T02:52:41.124151+0000","last_clean":"2024-03-19T02:52:41.124151+0000","last_became_active":"2024-03-19T02:50:57.252598+0000","last_became_peered":"2024-03-19T02:50:57.252598+0000","last_unstale":"2024-03-19T02:52:41.124151+0000","last_undegraded":"2024-03-19T02:52:41.124151+0000","last_fullsized":"2024-03-19T02:52:41.124151+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:09:38.183430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"93'3","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.172613+0000","last_change":"2024-03-19T02:50:51.794075+0000","last_active":"2024-03-19T02:52:41.172613+0000","last_peered":"2024-03-19T02:52:41.172613+0000","last_clean":"2024-03-19T02:52:41.172613+0000","last_became_active":"2024-03-19T02:50:51.793707+0000","last_became_peered":"2024-03-19T02:50:51.793707+0000","last_unstale":"2024-03-19T02:52:41.172613+0000","last_undegraded":"2024-03-19T02:52:41.172613+0000","last_fullsized":"2024-03-19T02:52:41.172613+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:52:07.180162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"95'8","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.175113+0000","last_change":"2024-03-19T02:50:52.211112+0000","last_active":"2024-03-19T02:52:41.175113+0000","last_peered":"2024-03-19T02:52:41.175113+0000","last_clean":"2024-03-19T02:52:41.175113+0000","last_became_active":"2024-03-19T02:50:52.210790+0000","last_became_peered":"2024-03-19T02:50:52.210790+0000","last_unstale":"2024-03-19T02:52:41.175113+0000","last_undegraded":"2024-03-19T02:52:41.175113+0000","last_fullsized":"2024-03-19T02:52:41.175113+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:39:17.651585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.232069+0000","last_change":"2024-03-19T02:52:41.231645+0000","last_active":"2024-03-19T02:52:41.232069+0000","last_peered":"2024-03-19T02:52:41.232069+0000","last_clean":"2024-03-19T02:52:41.232069+0000","last_became_active":"2024-03-19T02:52:41.231088+0000","last_became_peered":"2024-03-19T02:52:41.231088+0000","last_unstale":"2024-03-19T02:52:41.232069+0000","last_undegraded":"2024-03-19T02:52:41.232069+0000","last_fullsized":"2024-03-19T02:52:41.232069+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:25:29.072005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"93'2","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.185647+0000","last_change":"2024-03-19T02:50:51.785347+0000","last_active":"2024-03-19T02:52:41.185647+0000","last_peered":"2024-03-19T02:52:41.185647+0000","last_clean":"2024-03-19T02:52:41.185647+0000","last_became_active":"2024-03-19T02:50:51.785004+0000","last_became_peered":"2024-03-19T02:50:51.785004+0000","last_unstale":"2024-03-19T02:52:41.185647+0000","last_undegraded":"2024-03-19T02:52:41.185647+0000","last_fullsized":"2024-03-19T02:52:41.185647+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:47.269503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"95'8","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228791+0000","last_change":"2024-03-19T02:50:52.221330+0000","last_active":"2024-03-19T02:52:41.228791+0000","last_peered":"2024-03-19T02:52:41.228791+0000","last_clean":"2024-03-19T02:52:41.228791+0000","last_became_active":"2024-03-19T02:50:52.219417+0000","last_became_peered":"2024-03-19T02:50:52.219417+0000","last_unstale":"2024-03-19T02:52:41.228791+0000","last_undegraded":"2024-03-19T02:52:41.228791+0000","last_fullsized":"2024-03-19T02:52:41.228791+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:28:37.856260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.229931+0000","last_change":"2024-03-19T02:50:57.258357+0000","last_active":"2024-03-19T02:52:41.229931+0000","last_peered":"2024-03-19T02:52:41.229931+0000","last_clean":"2024-03-19T02:52:41.229931+0000","last_became_active":"2024-03-19T02:50:57.257924+0000","last_became_peered":"2024-03-19T02:50:57.257924+0000","last_unstale":"2024-03-19T02:52:41.229931+0000","last_undegraded":"2024-03-19T02:52:41.229931+0000","last_fullsized":"2024-03-19T02:52:41.229931+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:46:05.340731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"93'1","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.174691+0000","last_change":"2024-03-19T02:50:51.796425+0000","last_active":"2024-03-19T02:52:41.174691+0000","last_peered":"2024-03-19T02:52:41.174691+0000","last_clean":"2024-03-19T02:52:41.174691+0000","last_became_active":"2024-03-19T02:50:51.796139+0000","last_became_peered":"2024-03-19T02:50:51.796139+0000","last_unstale":"2024-03-19T02:52:41.174691+0000","last_undegraded":"2024-03-19T02:52:41.174691+0000","last_fullsized":"2024-03-19T02:52:41.174691+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:12:17.123556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"95'11","reported_seq":44,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123343+0000","last_change":"2024-03-19T02:50:52.219375+0000","last_active":"2024-03-19T02:52:41.123343+0000","last_peered":"2024-03-19T02:52:41.123343+0000","last_clean":"2024-03-19T02:52:41.123343+0000","last_became_active":"2024-03-19T02:50:52.219070+0000","last_became_peered":"2024-03-19T02:50:52.219070+0000","last_unstale":"2024-03-19T02:52:41.123343+0000","last_undegraded":"2024-03-19T02:52:41.123343+0000","last_fullsized":"2024-03-19T02:52:41.123343+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:31:58.847700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228989+0000","last_change":"2024-03-19T02:50:57.246502+0000","last_active":"2024-03-19T02:52:41.228989+0000","last_peered":"2024-03-19T02:52:41.228989+0000","last_clean":"2024-03-19T02:52:41.228989+0000","last_became_active":"2024-03-19T02:50:57.246089+0000","last_became_peered":"2024-03-19T02:50:57.246089+0000","last_unstale":"2024-03-19T02:52:41.228989+0000","last_undegraded":"2024-03-19T02:52:41.228989+0000","last_fullsized":"2024-03-19T02:52:41.228989+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:00.162308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'3","reported_seq":149,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:53:08.200323+0000","last_change":"2024-03-19T02:50:51.796448+0000","last_active":"2024-03-19T02:53:08.200323+0000","last_peered":"2024-03-19T02:53:08.200323+0000","last_clean":"2024-03-19T02:53:08.200323+0000","last_became_active":"2024-03-19T02:50:51.796132+0000","last_became_peered":"2024-03-19T02:50:51.796132+0000","last_unstale":"2024-03-19T02:53:08.200323+0000","last_undegraded":"2024-03-19T02:53:08.200323+0000","last_fullsized":"2024-03-19T02:53:08.200323+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:37:39.442772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":111,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"95'10","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228710+0000","last_change":"2024-03-19T02:50:52.223022+0000","last_active":"2024-03-19T02:52:41.228710+0000","last_peered":"2024-03-19T02:52:41.228710+0000","last_clean":"2024-03-19T02:52:41.228710+0000","last_became_active":"2024-03-19T02:50:52.221655+0000","last_became_peered":"2024-03-19T02:50:52.221655+0000","last_unstale":"2024-03-19T02:52:41.228710+0000","last_undegraded":"2024-03-19T02:52:41.228710+0000","last_fullsized":"2024-03-19T02:52:41.228710+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:34:36.054088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123149+0000","last_change":"2024-03-19T02:50:57.244003+0000","last_active":"2024-03-19T02:52:41.123149+0000","last_peered":"2024-03-19T02:52:41.123149+0000","last_clean":"2024-03-19T02:52:41.123149+0000","last_became_active":"2024-03-19T02:50:57.243567+0000","last_became_peered":"2024-03-19T02:50:57.243567+0000","last_unstale":"2024-03-19T02:52:41.123149+0000","last_undegraded":"2024-03-19T02:52:41.123149+0000","last_fullsized":"2024-03-19T02:52:41.123149+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:58:52.400237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"93'2","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497747+0000","last_change":"2024-03-19T02:50:51.801259+0000","last_active":"2024-03-19T02:52:41.497747+0000","last_peered":"2024-03-19T02:52:41.497747+0000","last_clean":"2024-03-19T02:52:41.497747+0000","last_became_active":"2024-03-19T02:50:51.800868+0000","last_became_peered":"2024-03-19T02:50:51.800868+0000","last_unstale":"2024-03-19T02:52:41.497747+0000","last_undegraded":"2024-03-19T02:52:41.497747+0000","last_fullsized":"2024-03-19T02:52:41.497747+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:44:06.486687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"95'9","reported_seq":48,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.230496+0000","last_change":"2024-03-19T02:51:40.724476+0000","last_active":"2024-03-19T02:52:41.230496+0000","last_peered":"2024-03-19T02:52:41.230496+0000","last_clean":"2024-03-19T02:52:41.230496+0000","last_became_active":"2024-03-19T02:51:40.724081+0000","last_became_peered":"2024-03-19T02:51:40.724081+0000","last_unstale":"2024-03-19T02:52:41.230496+0000","last_undegraded":"2024-03-19T02:52:41.230496+0000","last_fullsized":"2024-03-19T02:52:41.230496+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:04:15.684753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124330+0000","last_change":"2024-03-19T02:50:57.238764+0000","last_active":"2024-03-19T02:52:41.124330+0000","last_peered":"2024-03-19T02:52:41.124330+0000","last_clean":"2024-03-19T02:52:41.124330+0000","last_became_active":"2024-03-19T02:50:57.238401+0000","last_became_peered":"2024-03-19T02:50:57.238401+0000","last_unstale":"2024-03-19T02:52:41.124330+0000","last_undegraded":"2024-03-19T02:52:41.124330+0000","last_fullsized":"2024-03-19T02:52:41.124330+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:20:46.131776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"93'4","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.175182+0000","last_change":"2024-03-19T02:50:51.797633+0000","last_active":"2024-03-19T02:52:41.175182+0000","last_peered":"2024-03-19T02:52:41.175182+0000","last_clean":"2024-03-19T02:52:41.175182+0000","last_became_active":"2024-03-19T02:50:51.797313+0000","last_became_peered":"2024-03-19T02:50:51.797313+0000","last_unstale":"2024-03-19T02:52:41.175182+0000","last_undegraded":"2024-03-19T02:52:41.175182+0000","last_fullsized":"2024-03-19T02:52:41.175182+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:43:04.659301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"95'11","reported_seq":43,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228577+0000","last_change":"2024-03-19T02:50:52.231486+0000","last_active":"2024-03-19T02:52:41.228577+0000","last_peered":"2024-03-19T02:52:41.228577+0000","last_clean":"2024-03-19T02:52:41.228577+0000","last_became_active":"2024-03-19T02:50:52.231172+0000","last_became_peered":"2024-03-19T02:50:52.231172+0000","last_unstale":"2024-03-19T02:52:41.228577+0000","last_undegraded":"2024-03-19T02:52:41.228577+0000","last_fullsized":"2024-03-19T02:52:41.228577+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:24:49.912633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123242+0000","last_change":"2024-03-19T02:50:57.247207+0000","last_active":"2024-03-19T02:52:41.123242+0000","last_peered":"2024-03-19T02:52:41.123242+0000","last_clean":"2024-03-19T02:52:41.123242+0000","last_became_active":"2024-03-19T02:50:57.246776+0000","last_became_peered":"2024-03-19T02:50:57.246776+0000","last_unstale":"2024-03-19T02:52:41.123242+0000","last_undegraded":"2024-03-19T02:52:41.123242+0000","last_fullsized":"2024-03-19T02:52:41.123242+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:02:24.118858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"95'36","reported_seq":302,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:53:09.201163+0000","last_change":"2024-03-19T02:50:51.792398+0000","last_active":"2024-03-19T02:53:09.201163+0000","last_peered":"2024-03-19T02:53:09.201163+0000","last_clean":"2024-03-19T02:53:09.201163+0000","last_became_active":"2024-03-19T02:50:51.792090+0000","last_became_peered":"2024-03-19T02:50:51.792090+0000","last_unstale":"2024-03-19T02:53:09.201163+0000","last_undegraded":"2024-03-19T02:53:09.201163+0000","last_fullsized":"2024-03-19T02:53:09.201163+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:02:58.175210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63605,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":231,"num_read_kb":178,"num_write":37,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"95'6","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.134354+0000","last_change":"2024-03-19T02:50:52.220755+0000","last_active":"2024-03-19T02:52:41.134354+0000","last_peered":"2024-03-19T02:52:41.134354+0000","last_clean":"2024-03-19T02:52:41.134354+0000","last_became_active":"2024-03-19T02:50:52.220453+0000","last_became_peered":"2024-03-19T02:50:52.220453+0000","last_unstale":"2024-03-19T02:52:41.134354+0000","last_undegraded":"2024-03-19T02:52:41.134354+0000","last_fullsized":"2024-03-19T02:52:41.134354+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:48:27.866863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.175441+0000","last_change":"2024-03-19T02:50:57.253843+0000","last_active":"2024-03-19T02:52:41.175441+0000","last_peered":"2024-03-19T02:52:41.175441+0000","last_clean":"2024-03-19T02:52:41.175441+0000","last_became_active":"2024-03-19T02:50:57.252633+0000","last_became_peered":"2024-03-19T02:50:57.252633+0000","last_unstale":"2024-03-19T02:52:41.175441+0000","last_undegraded":"2024-03-19T02:52:41.175441+0000","last_fullsized":"2024-03-19T02:52:41.175441+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:03:48.514219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"93'6","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124047+0000","last_change":"2024-03-19T02:50:51.795041+0000","last_active":"2024-03-19T02:52:41.124047+0000","last_peered":"2024-03-19T02:52:41.124047+0000","last_clean":"2024-03-19T02:52:41.124047+0000","last_became_active":"2024-03-19T02:50:51.793485+0000","last_became_peered":"2024-03-19T02:50:51.793485+0000","last_unstale":"2024-03-19T02:52:41.124047+0000","last_undegraded":"2024-03-19T02:52:41.124047+0000","last_fullsized":"2024-03-19T02:52:41.124047+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:11:50.136394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"95'9","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.185789+0000","last_change":"2024-03-19T02:50:52.227554+0000","last_active":"2024-03-19T02:52:41.185789+0000","last_peered":"2024-03-19T02:52:41.185789+0000","last_clean":"2024-03-19T02:52:41.185789+0000","last_became_active":"2024-03-19T02:50:52.227222+0000","last_became_peered":"2024-03-19T02:50:52.227222+0000","last_unstale":"2024-03-19T02:52:41.185789+0000","last_undegraded":"2024-03-19T02:52:41.185789+0000","last_fullsized":"2024-03-19T02:52:41.185789+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:56.323749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":23,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.125538+0000","last_change":"2024-03-19T02:50:57.257504+0000","last_active":"2024-03-19T02:52:41.125538+0000","last_peered":"2024-03-19T02:52:41.125538+0000","last_clean":"2024-03-19T02:52:41.125538+0000","last_became_active":"2024-03-19T02:50:57.257120+0000","last_became_peered":"2024-03-19T02:50:57.257120+0000","last_unstale":"2024-03-19T02:52:41.125538+0000","last_undegraded":"2024-03-19T02:52:41.125538+0000","last_fullsized":"2024-03-19T02:52:41.125538+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:19:10.471666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"93'3","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.174890+0000","last_change":"2024-03-19T02:50:51.798049+0000","last_active":"2024-03-19T02:52:41.174890+0000","last_peered":"2024-03-19T02:52:41.174890+0000","last_clean":"2024-03-19T02:52:41.174890+0000","last_became_active":"2024-03-19T02:50:51.797134+0000","last_became_peered":"2024-03-19T02:50:51.797134+0000","last_unstale":"2024-03-19T02:52:41.174890+0000","last_undegraded":"2024-03-19T02:52:41.174890+0000","last_fullsized":"2024-03-19T02:52:41.174890+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:03:46.984240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"95'15","reported_seq":48,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497542+0000","last_change":"2024-03-19T02:50:52.219535+0000","last_active":"2024-03-19T02:52:41.497542+0000","last_peered":"2024-03-19T02:52:41.497542+0000","last_clean":"2024-03-19T02:52:41.497542+0000","last_became_active":"2024-03-19T02:50:52.219226+0000","last_became_peered":"2024-03-19T02:50:52.219226+0000","last_unstale":"2024-03-19T02:52:41.497542+0000","last_undegraded":"2024-03-19T02:52:41.497542+0000","last_fullsized":"2024-03-19T02:52:41.497542+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:27:26.875533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123493+0000","last_change":"2024-03-19T02:50:57.252873+0000","last_active":"2024-03-19T02:52:41.123493+0000","last_peered":"2024-03-19T02:52:41.123493+0000","last_clean":"2024-03-19T02:52:41.123493+0000","last_became_active":"2024-03-19T02:50:57.249551+0000","last_became_peered":"2024-03-19T02:50:57.249551+0000","last_unstale":"2024-03-19T02:52:41.123493+0000","last_undegraded":"2024-03-19T02:52:41.123493+0000","last_fullsized":"2024-03-19T02:52:41.123493+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:33:59.750606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"93'3","reported_seq":151,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:53:09.218235+0000","last_change":"2024-03-19T02:50:51.796478+0000","last_active":"2024-03-19T02:53:09.218235+0000","last_peered":"2024-03-19T02:53:09.218235+0000","last_clean":"2024-03-19T02:53:09.218235+0000","last_became_active":"2024-03-19T02:50:51.796126+0000","last_became_peered":"2024-03-19T02:50:51.796126+0000","last_unstale":"2024-03-19T02:53:09.218235+0000","last_undegraded":"2024-03-19T02:53:09.218235+0000","last_fullsized":"2024-03-19T02:53:09.218235+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:13:42.690481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":115,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"95'6","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497479+0000","last_change":"2024-03-19T02:50:52.228151+0000","last_active":"2024-03-19T02:52:41.497479+0000","last_peered":"2024-03-19T02:52:41.497479+0000","last_clean":"2024-03-19T02:52:41.497479+0000","last_became_active":"2024-03-19T02:50:52.227858+0000","last_became_peered":"2024-03-19T02:50:52.227858+0000","last_unstale":"2024-03-19T02:52:41.497479+0000","last_undegraded":"2024-03-19T02:52:41.497479+0000","last_fullsized":"2024-03-19T02:52:41.497479+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:10.285606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.186019+0000","last_change":"2024-03-19T02:50:57.259269+0000","last_active":"2024-03-19T02:52:41.186019+0000","last_peered":"2024-03-19T02:52:41.186019+0000","last_clean":"2024-03-19T02:52:41.186019+0000","last_became_active":"2024-03-19T02:50:57.258778+0000","last_became_peered":"2024-03-19T02:50:57.258778+0000","last_unstale":"2024-03-19T02:52:41.186019+0000","last_undegraded":"2024-03-19T02:52:41.186019+0000","last_fullsized":"2024-03-19T02:52:41.186019+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:02:44.419512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"93'15","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.125166+0000","last_change":"2024-03-19T02:51:40.739522+0000","last_active":"2024-03-19T02:52:41.125166+0000","last_peered":"2024-03-19T02:52:41.125166+0000","last_clean":"2024-03-19T02:52:41.125166+0000","last_became_active":"2024-03-19T02:51:40.720675+0000","last_became_peered":"2024-03-19T02:51:40.720675+0000","last_unstale":"2024-03-19T02:52:41.125166+0000","last_undegraded":"2024-03-19T02:52:41.125166+0000","last_fullsized":"2024-03-19T02:52:41.125166+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:18:19.449850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"95'6","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.174766+0000","last_change":"2024-03-19T02:50:52.215997+0000","last_active":"2024-03-19T02:52:41.174766+0000","last_peered":"2024-03-19T02:52:41.174766+0000","last_clean":"2024-03-19T02:52:41.174766+0000","last_became_active":"2024-03-19T02:50:52.215398+0000","last_became_peered":"2024-03-19T02:50:52.215398+0000","last_unstale":"2024-03-19T02:52:41.174766+0000","last_undegraded":"2024-03-19T02:52:41.174766+0000","last_fullsized":"2024-03-19T02:52:41.174766+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:49:14.010736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.230993+0000","last_change":"2024-03-19T02:52:41.230604+0000","last_active":"2024-03-19T02:52:41.230993+0000","last_peered":"2024-03-19T02:52:41.230993+0000","last_clean":"2024-03-19T02:52:41.230993+0000","last_became_active":"2024-03-19T02:52:41.230292+0000","last_became_peered":"2024-03-19T02:52:41.230292+0000","last_unstale":"2024-03-19T02:52:41.230993+0000","last_undegraded":"2024-03-19T02:52:41.230993+0000","last_fullsized":"2024-03-19T02:52:41.230993+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:55:52.552974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,3,1],"acting":[8,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.f","version":"93'2","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.134663+0000","last_change":"2024-03-19T02:50:51.796246+0000","last_active":"2024-03-19T02:52:41.134663+0000","last_peered":"2024-03-19T02:52:41.134663+0000","last_clean":"2024-03-19T02:52:41.134663+0000","last_became_active":"2024-03-19T02:50:51.795823+0000","last_became_peered":"2024-03-19T02:50:51.795823+0000","last_unstale":"2024-03-19T02:52:41.134663+0000","last_undegraded":"2024-03-19T02:52:41.134663+0000","last_fullsized":"2024-03-19T02:52:41.134663+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:36:02.220891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"95'5","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.125358+0000","last_change":"2024-03-19T02:50:52.224119+0000","last_active":"2024-03-19T02:52:41.125358+0000","last_peered":"2024-03-19T02:52:41.125358+0000","last_clean":"2024-03-19T02:52:41.125358+0000","last_became_active":"2024-03-19T02:50:52.223773+0000","last_became_peered":"2024-03-19T02:50:52.223773+0000","last_unstale":"2024-03-19T02:52:41.125358+0000","last_undegraded":"2024-03-19T02:52:41.125358+0000","last_fullsized":"2024-03-19T02:52:41.125358+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:56:03.098858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.134063+0000","last_change":"2024-03-19T02:50:57.652515+0000","last_active":"2024-03-19T02:52:41.134063+0000","last_peered":"2024-03-19T02:52:41.134063+0000","last_clean":"2024-03-19T02:52:41.134063+0000","last_became_active":"2024-03-19T02:50:57.651808+0000","last_became_peered":"2024-03-19T02:50:57.651808+0000","last_unstale":"2024-03-19T02:52:41.134063+0000","last_undegraded":"2024-03-19T02:52:41.134063+0000","last_fullsized":"2024-03-19T02:52:41.134063+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:16:34.550457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123896+0000","last_change":"2024-03-19T02:50:51.794282+0000","last_active":"2024-03-19T02:52:41.123896+0000","last_peered":"2024-03-19T02:52:41.123896+0000","last_clean":"2024-03-19T02:52:41.123896+0000","last_became_active":"2024-03-19T02:50:51.793947+0000","last_became_peered":"2024-03-19T02:50:51.793947+0000","last_unstale":"2024-03-19T02:52:41.123896+0000","last_undegraded":"2024-03-19T02:52:41.123896+0000","last_fullsized":"2024-03-19T02:52:41.123896+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:08:54.764915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"95'4","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.173032+0000","last_change":"2024-03-19T02:50:52.221707+0000","last_active":"2024-03-19T02:52:41.173032+0000","last_peered":"2024-03-19T02:52:41.173032+0000","last_clean":"2024-03-19T02:52:41.173032+0000","last_became_active":"2024-03-19T02:50:52.221417+0000","last_became_peered":"2024-03-19T02:50:52.221417+0000","last_unstale":"2024-03-19T02:52:41.173032+0000","last_undegraded":"2024-03-19T02:52:41.173032+0000","last_fullsized":"2024-03-19T02:52:41.173032+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:59:27.528346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.185950+0000","last_change":"2024-03-19T02:50:57.257552+0000","last_active":"2024-03-19T02:52:41.185950+0000","last_peered":"2024-03-19T02:52:41.185950+0000","last_clean":"2024-03-19T02:52:41.185950+0000","last_became_active":"2024-03-19T02:50:57.257121+0000","last_became_peered":"2024-03-19T02:50:57.257121+0000","last_unstale":"2024-03-19T02:52:41.185950+0000","last_undegraded":"2024-03-19T02:52:41.185950+0000","last_fullsized":"2024-03-19T02:52:41.185950+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:43:47.145344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"93'1","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497345+0000","last_change":"2024-03-19T02:50:51.804890+0000","last_active":"2024-03-19T02:52:41.497345+0000","last_peered":"2024-03-19T02:52:41.497345+0000","last_clean":"2024-03-19T02:52:41.497345+0000","last_became_active":"2024-03-19T02:50:51.804545+0000","last_became_peered":"2024-03-19T02:50:51.804545+0000","last_unstale":"2024-03-19T02:52:41.497345+0000","last_undegraded":"2024-03-19T02:52:41.497345+0000","last_fullsized":"2024-03-19T02:52:41.497345+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:16:14.945372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"95'7","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.172788+0000","last_change":"2024-03-19T02:50:52.224849+0000","last_active":"2024-03-19T02:52:41.172788+0000","last_peered":"2024-03-19T02:52:41.172788+0000","last_clean":"2024-03-19T02:52:41.172788+0000","last_became_active":"2024-03-19T02:50:52.224550+0000","last_became_peered":"2024-03-19T02:50:52.224550+0000","last_unstale":"2024-03-19T02:52:41.172788+0000","last_undegraded":"2024-03-19T02:52:41.172788+0000","last_fullsized":"2024-03-19T02:52:41.172788+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:28:59.683849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.186106+0000","last_change":"2024-03-19T02:50:57.652425+0000","last_active":"2024-03-19T02:52:41.186106+0000","last_peered":"2024-03-19T02:52:41.186106+0000","last_clean":"2024-03-19T02:52:41.186106+0000","last_became_active":"2024-03-19T02:50:57.651975+0000","last_became_peered":"2024-03-19T02:50:57.651975+0000","last_unstale":"2024-03-19T02:52:41.186106+0000","last_undegraded":"2024-03-19T02:52:41.186106+0000","last_fullsized":"2024-03-19T02:52:41.186106+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:07:20.587918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"95'8","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.174802+0000","last_change":"2024-03-19T02:50:52.218207+0000","last_active":"2024-03-19T02:52:41.174802+0000","last_peered":"2024-03-19T02:52:41.174802+0000","last_clean":"2024-03-19T02:52:41.174802+0000","last_became_active":"2024-03-19T02:50:52.217012+0000","last_became_peered":"2024-03-19T02:50:52.217012+0000","last_unstale":"2024-03-19T02:52:41.174802+0000","last_undegraded":"2024-03-19T02:52:41.174802+0000","last_fullsized":"2024-03-19T02:52:41.174802+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:07:47.366481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'4","reported_seq":151,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:53:09.217863+0000","last_change":"2024-03-19T02:50:51.801405+0000","last_active":"2024-03-19T02:53:09.217863+0000","last_peered":"2024-03-19T02:53:09.217863+0000","last_clean":"2024-03-19T02:53:09.217863+0000","last_became_active":"2024-03-19T02:50:51.801094+0000","last_became_peered":"2024-03-19T02:50:51.801094+0000","last_unstale":"2024-03-19T02:53:09.217863+0000","last_undegraded":"2024-03-19T02:53:09.217863+0000","last_fullsized":"2024-03-19T02:53:09.217863+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:53:39.534977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":112,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"93'1","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497740+0000","last_change":"2024-03-19T02:51:40.730836+0000","last_active":"2024-03-19T02:52:41.497740+0000","last_peered":"2024-03-19T02:52:41.497740+0000","last_clean":"2024-03-19T02:52:41.497740+0000","last_became_active":"2024-03-19T02:51:40.723984+0000","last_became_peered":"2024-03-19T02:51:40.723984+0000","last_unstale":"2024-03-19T02:52:41.497740+0000","last_undegraded":"2024-03-19T02:52:41.497740+0000","last_fullsized":"2024-03-19T02:52:41.497740+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:29:57.301066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"95'8","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228900+0000","last_change":"2024-03-19T02:50:52.221852+0000","last_active":"2024-03-19T02:52:41.228900+0000","last_peered":"2024-03-19T02:52:41.228900+0000","last_clean":"2024-03-19T02:52:41.228900+0000","last_became_active":"2024-03-19T02:50:52.221535+0000","last_became_peered":"2024-03-19T02:50:52.221535+0000","last_unstale":"2024-03-19T02:52:41.228900+0000","last_undegraded":"2024-03-19T02:52:41.228900+0000","last_fullsized":"2024-03-19T02:52:41.228900+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:37:18.030714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"93'3","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124331+0000","last_change":"2024-03-19T02:50:51.790231+0000","last_active":"2024-03-19T02:52:41.124331+0000","last_peered":"2024-03-19T02:52:41.124331+0000","last_clean":"2024-03-19T02:52:41.124331+0000","last_became_active":"2024-03-19T02:50:51.789867+0000","last_became_peered":"2024-03-19T02:50:51.789867+0000","last_unstale":"2024-03-19T02:52:41.124331+0000","last_undegraded":"2024-03-19T02:52:41.124331+0000","last_fullsized":"2024-03-19T02:52:41.124331+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:46.099191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":370,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497476+0000","last_change":"2024-03-19T02:49:38.697673+0000","last_active":"2024-03-19T02:52:41.497476+0000","last_peered":"2024-03-19T02:52:41.497476+0000","last_clean":"2024-03-19T02:52:41.497476+0000","last_became_active":"2024-03-19T02:49:38.697161+0000","last_became_peered":"2024-03-19T02:49:38.697161+0000","last_unstale":"2024-03-19T02:52:41.497476+0000","last_undegraded":"2024-03-19T02:52:41.497476+0000","last_fullsized":"2024-03-19T02:52:41.497476+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:46:10.013962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:46:10.013962+0000","last_clean_scrub_stamp":"2024-03-19T02:46:10.013962+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:59:35.179938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.175278+0000","last_change":"2024-03-19T02:50:57.652428+0000","last_active":"2024-03-19T02:52:41.175278+0000","last_peered":"2024-03-19T02:52:41.175278+0000","last_clean":"2024-03-19T02:52:41.175278+0000","last_became_active":"2024-03-19T02:50:57.651864+0000","last_became_peered":"2024-03-19T02:50:57.651864+0000","last_unstale":"2024-03-19T02:52:41.175278+0000","last_undegraded":"2024-03-19T02:52:41.175278+0000","last_fullsized":"2024-03-19T02:52:41.175278+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:52:19.007175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"95'5","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.185550+0000","last_change":"2024-03-19T02:50:52.218728+0000","last_active":"2024-03-19T02:52:41.185550+0000","last_peered":"2024-03-19T02:52:41.185550+0000","last_clean":"2024-03-19T02:52:41.185550+0000","last_became_active":"2024-03-19T02:50:52.218407+0000","last_became_peered":"2024-03-19T02:50:52.218407+0000","last_unstale":"2024-03-19T02:52:41.185550+0000","last_undegraded":"2024-03-19T02:52:41.185550+0000","last_fullsized":"2024-03-19T02:52:41.185550+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:30:49.304034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"93'3","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497416+0000","last_change":"2024-03-19T02:50:51.802057+0000","last_active":"2024-03-19T02:52:41.497416+0000","last_peered":"2024-03-19T02:52:41.497416+0000","last_clean":"2024-03-19T02:52:41.497416+0000","last_became_active":"2024-03-19T02:50:51.801754+0000","last_became_peered":"2024-03-19T02:50:51.801754+0000","last_unstale":"2024-03-19T02:52:41.497416+0000","last_undegraded":"2024-03-19T02:52:41.497416+0000","last_fullsized":"2024-03-19T02:52:41.497416+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:23:15.401822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.125249+0000","last_change":"2024-03-19T02:50:57.260303+0000","last_active":"2024-03-19T02:52:41.125249+0000","last_peered":"2024-03-19T02:52:41.125249+0000","last_clean":"2024-03-19T02:52:41.125249+0000","last_became_active":"2024-03-19T02:50:57.259792+0000","last_became_peered":"2024-03-19T02:50:57.259792+0000","last_unstale":"2024-03-19T02:52:41.125249+0000","last_undegraded":"2024-03-19T02:52:41.125249+0000","last_fullsized":"2024-03-19T02:52:41.125249+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:03.072161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"93'1","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.134478+0000","last_change":"2024-03-19T02:50:51.797949+0000","last_active":"2024-03-19T02:52:41.134478+0000","last_peered":"2024-03-19T02:52:41.134478+0000","last_clean":"2024-03-19T02:52:41.134478+0000","last_became_active":"2024-03-19T02:50:51.797522+0000","last_became_peered":"2024-03-19T02:50:51.797522+0000","last_unstale":"2024-03-19T02:52:41.134478+0000","last_undegraded":"2024-03-19T02:52:41.134478+0000","last_fullsized":"2024-03-19T02:52:41.134478+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:34:06.127993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"95'9","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.186125+0000","last_change":"2024-03-19T02:50:52.224095+0000","last_active":"2024-03-19T02:52:41.186125+0000","last_peered":"2024-03-19T02:52:41.186125+0000","last_clean":"2024-03-19T02:52:41.186125+0000","last_became_active":"2024-03-19T02:50:52.223764+0000","last_became_peered":"2024-03-19T02:50:52.223764+0000","last_unstale":"2024-03-19T02:52:41.186125+0000","last_undegraded":"2024-03-19T02:52:41.186125+0000","last_fullsized":"2024-03-19T02:52:41.186125+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:07:41.503162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228732+0000","last_change":"2024-03-19T02:50:57.652274+0000","last_active":"2024-03-19T02:52:41.228732+0000","last_peered":"2024-03-19T02:52:41.228732+0000","last_clean":"2024-03-19T02:52:41.228732+0000","last_became_active":"2024-03-19T02:50:57.651885+0000","last_became_peered":"2024-03-19T02:50:57.651885+0000","last_unstale":"2024-03-19T02:52:41.228732+0000","last_undegraded":"2024-03-19T02:52:41.228732+0000","last_fullsized":"2024-03-19T02:52:41.228732+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:03:02.081143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"97'10","reported_seq":46,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:53:03.631782+0000","last_change":"2024-03-19T02:50:51.796308+0000","last_active":"2024-03-19T02:53:03.631782+0000","last_peered":"2024-03-19T02:53:03.631782+0000","last_clean":"2024-03-19T02:53:03.631782+0000","last_became_active":"2024-03-19T02:50:51.795990+0000","last_became_peered":"2024-03-19T02:50:51.795990+0000","last_unstale":"2024-03-19T02:53:03.631782+0000","last_undegraded":"2024-03-19T02:53:03.631782+0000","last_fullsized":"2024-03-19T02:53:03.631782+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:45:53.608817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"95'5","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.229132+0000","last_change":"2024-03-19T02:50:52.221983+0000","last_active":"2024-03-19T02:52:41.229132+0000","last_peered":"2024-03-19T02:52:41.229132+0000","last_clean":"2024-03-19T02:52:41.229132+0000","last_became_active":"2024-03-19T02:50:52.220605+0000","last_became_peered":"2024-03-19T02:50:52.220605+0000","last_unstale":"2024-03-19T02:52:41.229132+0000","last_undegraded":"2024-03-19T02:52:41.229132+0000","last_fullsized":"2024-03-19T02:52:41.229132+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:12.834159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123697+0000","last_change":"2024-03-19T02:50:57.242428+0000","last_active":"2024-03-19T02:52:41.123697+0000","last_peered":"2024-03-19T02:52:41.123697+0000","last_clean":"2024-03-19T02:52:41.123697+0000","last_became_active":"2024-03-19T02:50:57.242007+0000","last_became_peered":"2024-03-19T02:50:57.242007+0000","last_unstale":"2024-03-19T02:52:41.123697+0000","last_undegraded":"2024-03-19T02:52:41.123697+0000","last_fullsized":"2024-03-19T02:52:41.123697+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:23:30.987761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"93'4","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124922+0000","last_change":"2024-03-19T02:50:51.791342+0000","last_active":"2024-03-19T02:52:41.124922+0000","last_peered":"2024-03-19T02:52:41.124922+0000","last_clean":"2024-03-19T02:52:41.124922+0000","last_became_active":"2024-03-19T02:50:51.791039+0000","last_became_peered":"2024-03-19T02:50:51.791039+0000","last_unstale":"2024-03-19T02:52:41.124922+0000","last_undegraded":"2024-03-19T02:52:41.124922+0000","last_fullsized":"2024-03-19T02:52:41.124922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:50:38.216623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"95'7","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.230040+0000","last_change":"2024-03-19T02:50:52.224303+0000","last_active":"2024-03-19T02:52:41.230040+0000","last_peered":"2024-03-19T02:52:41.230040+0000","last_clean":"2024-03-19T02:52:41.230040+0000","last_became_active":"2024-03-19T02:50:52.223942+0000","last_became_peered":"2024-03-19T02:50:52.223942+0000","last_unstale":"2024-03-19T02:52:41.230040+0000","last_undegraded":"2024-03-19T02:52:41.230040+0000","last_fullsized":"2024-03-19T02:52:41.230040+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:57:04.175971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124644+0000","last_change":"2024-03-19T02:50:57.652126+0000","last_active":"2024-03-19T02:52:41.124644+0000","last_peered":"2024-03-19T02:52:41.124644+0000","last_clean":"2024-03-19T02:52:41.124644+0000","last_became_active":"2024-03-19T02:50:57.651713+0000","last_became_peered":"2024-03-19T02:50:57.651713+0000","last_unstale":"2024-03-19T02:52:41.124644+0000","last_undegraded":"2024-03-19T02:52:41.124644+0000","last_fullsized":"2024-03-19T02:52:41.124644+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:28:51.077437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"93'2","reported_seq":47,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228284+0000","last_change":"2024-03-19T02:51:40.741378+0000","last_active":"2024-03-19T02:52:41.228284+0000","last_peered":"2024-03-19T02:52:41.228284+0000","last_clean":"2024-03-19T02:52:41.228284+0000","last_became_active":"2024-03-19T02:51:40.724510+0000","last_became_peered":"2024-03-19T02:51:40.724510+0000","last_unstale":"2024-03-19T02:52:41.228284+0000","last_undegraded":"2024-03-19T02:52:41.228284+0000","last_fullsized":"2024-03-19T02:52:41.228284+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:30:16.228635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"95'2","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.133963+0000","last_change":"2024-03-19T02:50:52.221681+0000","last_active":"2024-03-19T02:52:41.133963+0000","last_peered":"2024-03-19T02:52:41.133963+0000","last_clean":"2024-03-19T02:52:41.133963+0000","last_became_active":"2024-03-19T02:50:52.220987+0000","last_became_peered":"2024-03-19T02:50:52.220987+0000","last_unstale":"2024-03-19T02:52:41.133963+0000","last_undegraded":"2024-03-19T02:52:41.133963+0000","last_fullsized":"2024-03-19T02:52:41.133963+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:16:36.561143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123630+0000","last_change":"2024-03-19T02:50:57.249830+0000","last_active":"2024-03-19T02:52:41.123630+0000","last_peered":"2024-03-19T02:52:41.123630+0000","last_clean":"2024-03-19T02:52:41.123630+0000","last_became_active":"2024-03-19T02:50:57.248879+0000","last_became_peered":"2024-03-19T02:50:57.248879+0000","last_unstale":"2024-03-19T02:52:41.123630+0000","last_undegraded":"2024-03-19T02:52:41.123630+0000","last_fullsized":"2024-03-19T02:52:41.123630+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:58:43.225841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"93'3","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.229197+0000","last_change":"2024-03-19T02:50:51.801374+0000","last_active":"2024-03-19T02:52:41.229197+0000","last_peered":"2024-03-19T02:52:41.229197+0000","last_clean":"2024-03-19T02:52:41.229197+0000","last_became_active":"2024-03-19T02:50:51.801041+0000","last_became_peered":"2024-03-19T02:50:51.801041+0000","last_unstale":"2024-03-19T02:52:41.229197+0000","last_undegraded":"2024-03-19T02:52:41.229197+0000","last_fullsized":"2024-03-19T02:52:41.229197+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:56:14.671718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"95'9","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.125323+0000","last_change":"2024-03-19T02:50:52.225915+0000","last_active":"2024-03-19T02:52:41.125323+0000","last_peered":"2024-03-19T02:52:41.125323+0000","last_clean":"2024-03-19T02:52:41.125323+0000","last_became_active":"2024-03-19T02:50:52.225606+0000","last_became_peered":"2024-03-19T02:50:52.225606+0000","last_unstale":"2024-03-19T02:52:41.125323+0000","last_undegraded":"2024-03-19T02:52:41.125323+0000","last_fullsized":"2024-03-19T02:52:41.125323+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:27:29.549255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"93'1","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124600+0000","last_change":"2024-03-19T02:50:57.240747+0000","last_active":"2024-03-19T02:52:41.124600+0000","last_peered":"2024-03-19T02:52:41.124600+0000","last_clean":"2024-03-19T02:52:41.124600+0000","last_became_active":"2024-03-19T02:50:57.240373+0000","last_became_peered":"2024-03-19T02:50:57.240373+0000","last_unstale":"2024-03-19T02:52:41.124600+0000","last_undegraded":"2024-03-19T02:52:41.124600+0000","last_fullsized":"2024-03-19T02:52:41.124600+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:22:04.991497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"93'1","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123868+0000","last_change":"2024-03-19T02:50:51.794071+0000","last_active":"2024-03-19T02:52:41.123868+0000","last_peered":"2024-03-19T02:52:41.123868+0000","last_clean":"2024-03-19T02:52:41.123868+0000","last_became_active":"2024-03-19T02:50:51.793570+0000","last_became_peered":"2024-03-19T02:50:51.793570+0000","last_unstale":"2024-03-19T02:52:41.123868+0000","last_undegraded":"2024-03-19T02:52:41.123868+0000","last_fullsized":"2024-03-19T02:52:41.123868+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:34:20.456515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"95'8","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.229465+0000","last_change":"2024-03-19T02:50:52.223341+0000","last_active":"2024-03-19T02:52:41.229465+0000","last_peered":"2024-03-19T02:52:41.229465+0000","last_clean":"2024-03-19T02:52:41.229465+0000","last_became_active":"2024-03-19T02:50:52.220280+0000","last_became_peered":"2024-03-19T02:50:52.220280+0000","last_unstale":"2024-03-19T02:52:41.229465+0000","last_undegraded":"2024-03-19T02:52:41.229465+0000","last_fullsized":"2024-03-19T02:52:41.229465+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:04:15.830811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124148+0000","last_change":"2024-03-19T02:50:57.651293+0000","last_active":"2024-03-19T02:52:41.124148+0000","last_peered":"2024-03-19T02:52:41.124148+0000","last_clean":"2024-03-19T02:52:41.124148+0000","last_became_active":"2024-03-19T02:50:57.650893+0000","last_became_peered":"2024-03-19T02:50:57.650893+0000","last_unstale":"2024-03-19T02:52:41.124148+0000","last_undegraded":"2024-03-19T02:52:41.124148+0000","last_fullsized":"2024-03-19T02:52:41.124148+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:33:35.984728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"93'4","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.134128+0000","last_change":"2024-03-19T02:50:51.798297+0000","last_active":"2024-03-19T02:52:41.134128+0000","last_peered":"2024-03-19T02:52:41.134128+0000","last_clean":"2024-03-19T02:52:41.134128+0000","last_became_active":"2024-03-19T02:50:51.797516+0000","last_became_peered":"2024-03-19T02:50:51.797516+0000","last_unstale":"2024-03-19T02:52:41.134128+0000","last_undegraded":"2024-03-19T02:52:41.134128+0000","last_fullsized":"2024-03-19T02:52:41.134128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:09:42.295094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"95'5","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497375+0000","last_change":"2024-03-19T02:50:52.224011+0000","last_active":"2024-03-19T02:52:41.497375+0000","last_peered":"2024-03-19T02:52:41.497375+0000","last_clean":"2024-03-19T02:52:41.497375+0000","last_became_active":"2024-03-19T02:50:52.223708+0000","last_became_peered":"2024-03-19T02:50:52.223708+0000","last_unstale":"2024-03-19T02:52:41.497375+0000","last_undegraded":"2024-03-19T02:52:41.497375+0000","last_fullsized":"2024-03-19T02:52:41.497375+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:17:49.305587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228248+0000","last_change":"2024-03-19T02:50:57.251134+0000","last_active":"2024-03-19T02:52:41.228248+0000","last_peered":"2024-03-19T02:52:41.228248+0000","last_clean":"2024-03-19T02:52:41.228248+0000","last_became_active":"2024-03-19T02:50:57.250761+0000","last_became_peered":"2024-03-19T02:50:57.250761+0000","last_unstale":"2024-03-19T02:52:41.228248+0000","last_undegraded":"2024-03-19T02:52:41.228248+0000","last_fullsized":"2024-03-19T02:52:41.228248+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:09:26.860235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"93'3","reported_seq":21,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124386+0000","last_change":"2024-03-19T02:51:40.728077+0000","last_active":"2024-03-19T02:52:41.124386+0000","last_peered":"2024-03-19T02:52:41.124386+0000","last_clean":"2024-03-19T02:52:41.124386+0000","last_became_active":"2024-03-19T02:51:40.720767+0000","last_became_peered":"2024-03-19T02:51:40.720767+0000","last_unstale":"2024-03-19T02:52:41.124386+0000","last_undegraded":"2024-03-19T02:52:41.124386+0000","last_fullsized":"2024-03-19T02:52:41.124386+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:22:34.366373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"95'7","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497892+0000","last_change":"2024-03-19T02:50:52.224776+0000","last_active":"2024-03-19T02:52:41.497892+0000","last_peered":"2024-03-19T02:52:41.497892+0000","last_clean":"2024-03-19T02:52:41.497892+0000","last_became_active":"2024-03-19T02:50:52.224442+0000","last_became_peered":"2024-03-19T02:50:52.224442+0000","last_unstale":"2024-03-19T02:52:41.497892+0000","last_undegraded":"2024-03-19T02:52:41.497892+0000","last_fullsized":"2024-03-19T02:52:41.497892+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:11:13.722005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.230172+0000","last_change":"2024-03-19T02:51:40.722732+0000","last_active":"2024-03-19T02:52:41.230172+0000","last_peered":"2024-03-19T02:52:41.230172+0000","last_clean":"2024-03-19T02:52:41.230172+0000","last_became_active":"2024-03-19T02:51:40.722260+0000","last_became_peered":"2024-03-19T02:51:40.722260+0000","last_unstale":"2024-03-19T02:52:41.230172+0000","last_undegraded":"2024-03-19T02:52:41.230172+0000","last_fullsized":"2024-03-19T02:52:41.230172+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:31:06.383570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.175385+0000","last_change":"2024-03-19T02:50:51.794898+0000","last_active":"2024-03-19T02:52:41.175385+0000","last_peered":"2024-03-19T02:52:41.175385+0000","last_clean":"2024-03-19T02:52:41.175385+0000","last_became_active":"2024-03-19T02:50:51.794562+0000","last_became_peered":"2024-03-19T02:50:51.794562+0000","last_unstale":"2024-03-19T02:52:41.175385+0000","last_undegraded":"2024-03-19T02:52:41.175385+0000","last_fullsized":"2024-03-19T02:52:41.175385+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:17:45.239395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"95'6","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497460+0000","last_change":"2024-03-19T02:50:52.227427+0000","last_active":"2024-03-19T02:52:41.497460+0000","last_peered":"2024-03-19T02:52:41.497460+0000","last_clean":"2024-03-19T02:52:41.497460+0000","last_became_active":"2024-03-19T02:50:52.227111+0000","last_became_peered":"2024-03-19T02:50:52.227111+0000","last_unstale":"2024-03-19T02:52:41.497460+0000","last_undegraded":"2024-03-19T02:52:41.497460+0000","last_fullsized":"2024-03-19T02:52:41.497460+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:40:53.295454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.174601+0000","last_change":"2024-03-19T02:50:57.244449+0000","last_active":"2024-03-19T02:52:41.174601+0000","last_peered":"2024-03-19T02:52:41.174601+0000","last_clean":"2024-03-19T02:52:41.174601+0000","last_became_active":"2024-03-19T02:50:57.243991+0000","last_became_peered":"2024-03-19T02:50:57.243991+0000","last_unstale":"2024-03-19T02:52:41.174601+0000","last_undegraded":"2024-03-19T02:52:41.174601+0000","last_fullsized":"2024-03-19T02:52:41.174601+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:50:45.265773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.186101+0000","last_change":"2024-03-19T02:50:51.799834+0000","last_active":"2024-03-19T02:52:41.186101+0000","last_peered":"2024-03-19T02:52:41.186101+0000","last_clean":"2024-03-19T02:52:41.186101+0000","last_became_active":"2024-03-19T02:50:51.799563+0000","last_became_peered":"2024-03-19T02:50:51.799563+0000","last_unstale":"2024-03-19T02:52:41.186101+0000","last_undegraded":"2024-03-19T02:52:41.186101+0000","last_fullsized":"2024-03-19T02:52:41.186101+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:51:00.284454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"95'9","reported_seq":43,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497913+0000","last_change":"2024-03-19T02:50:52.222710+0000","last_active":"2024-03-19T02:52:41.497913+0000","last_peered":"2024-03-19T02:52:41.497913+0000","last_clean":"2024-03-19T02:52:41.497913+0000","last_became_active":"2024-03-19T02:50:52.222321+0000","last_became_peered":"2024-03-19T02:50:52.222321+0000","last_unstale":"2024-03-19T02:52:41.497913+0000","last_undegraded":"2024-03-19T02:52:41.497913+0000","last_fullsized":"2024-03-19T02:52:41.497913+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:05:27.527696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"95'9","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.125423+0000","last_change":"2024-03-19T02:50:52.226631+0000","last_active":"2024-03-19T02:52:41.125423+0000","last_peered":"2024-03-19T02:52:41.125423+0000","last_clean":"2024-03-19T02:52:41.125423+0000","last_became_active":"2024-03-19T02:50:52.226294+0000","last_became_peered":"2024-03-19T02:50:52.226294+0000","last_unstale":"2024-03-19T02:52:41.125423+0000","last_undegraded":"2024-03-19T02:52:41.125423+0000","last_fullsized":"2024-03-19T02:52:41.125423+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:01:31.350063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"95'12","reported_seq":45,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123985+0000","last_change":"2024-03-19T02:50:52.221138+0000","last_active":"2024-03-19T02:52:41.123985+0000","last_peered":"2024-03-19T02:52:41.123985+0000","last_clean":"2024-03-19T02:52:41.123985+0000","last_became_active":"2024-03-19T02:50:52.220730+0000","last_became_peered":"2024-03-19T02:50:52.220730+0000","last_unstale":"2024-03-19T02:52:41.123985+0000","last_undegraded":"2024-03-19T02:52:41.123985+0000","last_fullsized":"2024-03-19T02:52:41.123985+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:26:13.059986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"95'8","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123763+0000","last_change":"2024-03-19T02:50:52.223567+0000","last_active":"2024-03-19T02:52:41.123763+0000","last_peered":"2024-03-19T02:52:41.123763+0000","last_clean":"2024-03-19T02:52:41.123763+0000","last_became_active":"2024-03-19T02:50:52.223174+0000","last_became_peered":"2024-03-19T02:50:52.223174+0000","last_unstale":"2024-03-19T02:52:41.123763+0000","last_undegraded":"2024-03-19T02:52:41.123763+0000","last_fullsized":"2024-03-19T02:52:41.123763+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:23:06.383584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"95'7","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.230567+0000","last_change":"2024-03-19T02:50:52.229916+0000","last_active":"2024-03-19T02:52:41.230567+0000","last_peered":"2024-03-19T02:52:41.230567+0000","last_clean":"2024-03-19T02:52:41.230567+0000","last_became_active":"2024-03-19T02:50:52.229597+0000","last_became_peered":"2024-03-19T02:50:52.229597+0000","last_unstale":"2024-03-19T02:52:41.230567+0000","last_undegraded":"2024-03-19T02:52:41.230567+0000","last_fullsized":"2024-03-19T02:52:41.230567+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:19.331893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"95'4","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.186073+0000","last_change":"2024-03-19T02:50:52.223331+0000","last_active":"2024-03-19T02:52:41.186073+0000","last_peered":"2024-03-19T02:52:41.186073+0000","last_clean":"2024-03-19T02:52:41.186073+0000","last_became_active":"2024-03-19T02:50:52.222975+0000","last_became_peered":"2024-03-19T02:50:52.222975+0000","last_unstale":"2024-03-19T02:52:41.186073+0000","last_undegraded":"2024-03-19T02:52:41.186073+0000","last_fullsized":"2024-03-19T02:52:41.186073+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:26:23.023749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"95'6","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.125760+0000","last_change":"2024-03-19T02:50:52.218073+0000","last_active":"2024-03-19T02:52:41.125760+0000","last_peered":"2024-03-19T02:52:41.125760+0000","last_clean":"2024-03-19T02:52:41.125760+0000","last_became_active":"2024-03-19T02:50:52.217742+0000","last_became_peered":"2024-03-19T02:50:52.217742+0000","last_unstale":"2024-03-19T02:52:41.125760+0000","last_undegraded":"2024-03-19T02:52:41.125760+0000","last_fullsized":"2024-03-19T02:52:41.125760+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:08:51.053153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"95'15","reported_seq":48,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124597+0000","last_change":"2024-03-19T02:50:52.222108+0000","last_active":"2024-03-19T02:52:41.124597+0000","last_peered":"2024-03-19T02:52:41.124597+0000","last_clean":"2024-03-19T02:52:41.124597+0000","last_became_active":"2024-03-19T02:50:52.221748+0000","last_became_peered":"2024-03-19T02:50:52.221748+0000","last_unstale":"2024-03-19T02:52:41.124597+0000","last_undegraded":"2024-03-19T02:52:41.124597+0000","last_fullsized":"2024-03-19T02:52:41.124597+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:33:31.401939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"95'2","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.229311+0000","last_change":"2024-03-19T02:50:52.221799+0000","last_active":"2024-03-19T02:52:41.229311+0000","last_peered":"2024-03-19T02:52:41.229311+0000","last_clean":"2024-03-19T02:52:41.229311+0000","last_became_active":"2024-03-19T02:50:52.220196+0000","last_became_peered":"2024-03-19T02:50:52.220196+0000","last_unstale":"2024-03-19T02:52:41.229311+0000","last_undegraded":"2024-03-19T02:52:41.229311+0000","last_fullsized":"2024-03-19T02:52:41.229311+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:24:01.541716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"95'6","reported_seq":21,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.185475+0000","last_change":"2024-03-19T02:51:40.721422+0000","last_active":"2024-03-19T02:52:41.185475+0000","last_peered":"2024-03-19T02:52:41.185475+0000","last_clean":"2024-03-19T02:52:41.185475+0000","last_became_active":"2024-03-19T02:51:40.720982+0000","last_became_peered":"2024-03-19T02:51:40.720982+0000","last_unstale":"2024-03-19T02:52:41.185475+0000","last_undegraded":"2024-03-19T02:52:41.185475+0000","last_fullsized":"2024-03-19T02:52:41.185475+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:07.391810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,3],"acting":[6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.29","version":"95'8","reported_seq":23,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123111+0000","last_change":"2024-03-19T02:51:40.721188+0000","last_active":"2024-03-19T02:52:41.123111+0000","last_peered":"2024-03-19T02:52:41.123111+0000","last_clean":"2024-03-19T02:52:41.123111+0000","last_became_active":"2024-03-19T02:51:40.720811+0000","last_became_peered":"2024-03-19T02:51:40.720811+0000","last_unstale":"2024-03-19T02:52:41.123111+0000","last_undegraded":"2024-03-19T02:52:41.123111+0000","last_fullsized":"2024-03-19T02:52:41.123111+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:23.970149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2a","version":"95'10","reported_seq":43,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228360+0000","last_change":"2024-03-19T02:50:52.230847+0000","last_active":"2024-03-19T02:52:41.228360+0000","last_peered":"2024-03-19T02:52:41.228360+0000","last_clean":"2024-03-19T02:52:41.228360+0000","last_became_active":"2024-03-19T02:50:52.230478+0000","last_became_peered":"2024-03-19T02:50:52.230478+0000","last_unstale":"2024-03-19T02:52:41.228360+0000","last_undegraded":"2024-03-19T02:52:41.228360+0000","last_fullsized":"2024-03-19T02:52:41.228360+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:04:09.459446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"95'11","reported_seq":44,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.186018+0000","last_change":"2024-03-19T02:50:52.223508+0000","last_active":"2024-03-19T02:52:41.186018+0000","last_peered":"2024-03-19T02:52:41.186018+0000","last_clean":"2024-03-19T02:52:41.186018+0000","last_became_active":"2024-03-19T02:50:52.223173+0000","last_became_peered":"2024-03-19T02:50:52.223173+0000","last_unstale":"2024-03-19T02:52:41.186018+0000","last_undegraded":"2024-03-19T02:52:41.186018+0000","last_fullsized":"2024-03-19T02:52:41.186018+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:53.320173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"95'8","reported_seq":47,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.125738+0000","last_change":"2024-03-19T02:51:40.723512+0000","last_active":"2024-03-19T02:52:41.125738+0000","last_peered":"2024-03-19T02:52:41.125738+0000","last_clean":"2024-03-19T02:52:41.125738+0000","last_became_active":"2024-03-19T02:51:40.723149+0000","last_became_peered":"2024-03-19T02:51:40.723149+0000","last_unstale":"2024-03-19T02:52:41.125738+0000","last_undegraded":"2024-03-19T02:52:41.125738+0000","last_fullsized":"2024-03-19T02:52:41.125738+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:42:38.862569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67215,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":694,"num_read_kb":417,"num_write":167,"num_write_kb":119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":694,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":202631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318314,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36432,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708720,"statfs":{"total":95995035648,"available":95957729280,"internally_reserved":0,"allocated":1114112,"data_stored":545712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579950,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28784,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716368,"statfs":{"total":95995035648,"available":95965560832,"internally_reserved":0,"allocated":1802240,"data_stored":1200569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743477,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36432,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708720,"statfs":{"total":95995035648,"available":95957729280,"internally_reserved":0,"allocated":1122304,"data_stored":545836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005113,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36432,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708720,"statfs":{"total":95995035648,"available":95957729280,"internally_reserved":0,"allocated":1114112,"data_stored":545644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266751,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37012,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708140,"statfs":{"total":95995035648,"available":95957135360,"internally_reserved":0,"allocated":1716224,"data_stored":1142260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462978,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1101824,"data_stored":541012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36488,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708664,"statfs":{"total":95995035648,"available":95957671936,"internally_reserved":0,"allocated":1171456,"data_stored":604527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1126400,"data_stored":542400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28188,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716964,"statfs":{"total":95995035648,"available":95966171136,"internally_reserved":0,"allocated":1191936,"data_stored":605295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149779,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1114112,"data_stored":541206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888142,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1118208,"data_stored":546244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724617,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28704,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716448,"statfs":{"total":95995035648,"available":95965642752,"internally_reserved":0,"allocated":1724416,"data_stored":1136862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":63605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":64373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":65183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T02:53:15.639 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T02:53:15.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.775+0000 7fc93de73640 1 -- 172.21.15.33:0/28908437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc9381033f0 msgr2=0x7fc938109c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:15.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.775+0000 7fc93de73640 1 --2- 172.21.15.33:0/28908437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc9381033f0 0x7fc938109c80 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fc930009ca0 tx=0x7fc93002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:53:15.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.775+0000 7fc93de73640 1 -- 172.21.15.33:0/28908437 shutdown_connections 2024-03-19T02:53:15.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.775+0000 7fc93de73640 1 --2- 172.21.15.33:0/28908437 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc9381033f0 0x7fc938109c80 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:15.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.775+0000 7fc93de73640 1 --2- 172.21.15.33:0/28908437 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc938102a30 0x7fc938102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:15.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.775+0000 7fc93de73640 1 --2- 172.21.15.33:0/28908437 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc938101840 0x7fc938101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:15.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.775+0000 7fc93de73640 1 -- 172.21.15.33:0/28908437 >> 172.21.15.33:0/28908437 conn(0x7fc9380fcdd0 msgr2=0x7fc9380ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:15.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.775+0000 7fc93de73640 1 -- 172.21.15.33:0/28908437 shutdown_connections 2024-03-19T02:53:15.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.775+0000 7fc93de73640 1 -- 172.21.15.33:0/28908437 wait complete. 2024-03-19T02:53:15.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.775+0000 7fc93de73640 1 Processor -- start 2024-03-19T02:53:15.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.776+0000 7fc93de73640 1 -- start start 2024-03-19T02:53:15.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.776+0000 7fc93de73640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc938101840 0x7fc9380755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:15.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.776+0000 7fc93de73640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc938102a30 0x7fc938073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:15.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.776+0000 7fc93de73640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc9381033f0 0x7fc938074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:15.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.776+0000 7fc93de73640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc93810d350 con 0x7fc9381033f0 2024-03-19T02:53:15.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.776+0000 7fc93de73640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc93810d1d0 con 0x7fc938101840 2024-03-19T02:53:15.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.776+0000 7fc93de73640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc93810d4d0 con 0x7fc938102a30 2024-03-19T02:53:15.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.776+0000 7fc937fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc9381033f0 0x7fc938074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:15.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.776+0000 7fc937fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc9381033f0 0x7fc938074150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46788/0 (socket says 172.21.15.33:46788) 2024-03-19T02:53:15.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.776+0000 7fc937fff640 1 -- 172.21.15.33:0/2738181910 learned_addr learned my addr 172.21.15.33:0/2738181910 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:15.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.776+0000 7fc9377fe640 1 --2- 172.21.15.33:0/2738181910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc938101840 0x7fc9380755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:15.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.776+0000 7fc936ffd640 1 --2- 172.21.15.33:0/2738181910 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc938102a30 0x7fc938073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:15.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.776+0000 7fc937fff640 1 -- 172.21.15.33:0/2738181910 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc938102a30 msgr2=0x7fc938073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:15.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.776+0000 7fc937fff640 1 --2- 172.21.15.33:0/2738181910 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc938102a30 0x7fc938073c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:15.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.776+0000 7fc937fff640 1 -- 172.21.15.33:0/2738181910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc938101840 msgr2=0x7fc9380755d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:15.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.776+0000 7fc937fff640 1 --2- 172.21.15.33:0/2738181910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc938101840 0x7fc9380755d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:15.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.776+0000 7fc937fff640 1 -- 172.21.15.33:0/2738181910 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9380749e0 con 0x7fc9381033f0 2024-03-19T02:53:15.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.776+0000 7fc9377fe640 1 --2- 172.21.15.33:0/2738181910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc938101840 0x7fc9380755d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:15.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.776+0000 7fc937fff640 1 --2- 172.21.15.33:0/2738181910 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc9381033f0 0x7fc938074150 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc930032380 tx=0x7fc930032820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:15.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.776+0000 7fc934ff9640 1 -- 172.21.15.33:0/2738181910 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc930041020 con 0x7fc9381033f0 2024-03-19T02:53:15.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.777+0000 7fc934ff9640 1 -- 172.21.15.33:0/2738181910 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc930003e20 con 0x7fc9381033f0 2024-03-19T02:53:15.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.777+0000 7fc93de73640 1 -- 172.21.15.33:0/2738181910 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9381a1ae0 con 0x7fc9381033f0 2024-03-19T02:53:15.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.777+0000 7fc934ff9640 1 -- 172.21.15.33:0/2738181910 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc930039040 con 0x7fc9381033f0 2024-03-19T02:53:15.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.777+0000 7fc93de73640 1 -- 172.21.15.33:0/2738181910 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9381a2030 con 0x7fc9381033f0 2024-03-19T02:53:15.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.777+0000 7fc93de73640 1 -- 172.21.15.33:0/2738181910 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc938108bb0 con 0x7fc9381033f0 2024-03-19T02:53:15.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.779+0000 7fc934ff9640 1 -- 172.21.15.33:0/2738181910 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc930044020 con 0x7fc9381033f0 2024-03-19T02:53:15.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.779+0000 7fc934ff9640 1 --2- 172.21.15.33:0/2738181910 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc9200a2800 0x7fc9200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:15.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.779+0000 7fc9377fe640 1 --2- 172.21.15.33:0/2738181910 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc9200a2800 0x7fc9200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:15.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.779+0000 7fc934ff9640 1 -- 172.21.15.33:0/2738181910 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7fc9300f1240 con 0x7fc9381033f0 2024-03-19T02:53:15.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.780+0000 7fc9377fe640 1 --2- 172.21.15.33:0/2738181910 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc9200a2800 0x7fc9200a4cc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc938102890 tx=0x7fc928005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:15.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.783+0000 7fc934ff9640 1 -- 172.21.15.33:0/2738181910 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc93004b020 con 0x7fc9381033f0 2024-03-19T02:53:15.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.963+0000 7fc93de73640 1 -- 172.21.15.33:0/2738181910 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc938061c60 con 0x7fc9200a2800 2024-03-19T02:53:15.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.973+0000 7fc934ff9640 1 -- 172.21.15.33:0/2738181910 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340504 (secure 0 0 0) 0x7fc938061c60 con 0x7fc9200a2800 2024-03-19T02:53:15.978 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:53:15.980 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-03-19T02:53:15.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.976+0000 7fc93de73640 1 -- 172.21.15.33:0/2738181910 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc9200a2800 msgr2=0x7fc9200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:15.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.976+0000 7fc93de73640 1 --2- 172.21.15.33:0/2738181910 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc9200a2800 0x7fc9200a4cc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc938102890 tx=0x7fc928005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:15.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.976+0000 7fc93de73640 1 -- 172.21.15.33:0/2738181910 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc9381033f0 msgr2=0x7fc938074150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:15.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.976+0000 7fc93de73640 1 --2- 172.21.15.33:0/2738181910 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc9381033f0 0x7fc938074150 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc930032380 tx=0x7fc930032820 comp rx=0 tx=0).stop 2024-03-19T02:53:15.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.976+0000 7fc93de73640 1 -- 172.21.15.33:0/2738181910 shutdown_connections 2024-03-19T02:53:15.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.976+0000 7fc93de73640 1 --2- 172.21.15.33:0/2738181910 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fc9200a2800 0x7fc9200a4cc0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:15.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.976+0000 7fc93de73640 1 --2- 172.21.15.33:0/2738181910 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc938102a30 0x7fc938073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:15.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.976+0000 7fc93de73640 1 --2- 172.21.15.33:0/2738181910 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc9381033f0 0x7fc938074150 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:15.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.976+0000 7fc93de73640 1 --2- 172.21.15.33:0/2738181910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc938101840 0x7fc9380755d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:15.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.976+0000 7fc93de73640 1 -- 172.21.15.33:0/2738181910 >> 172.21.15.33:0/2738181910 conn(0x7fc9380fcdd0 msgr2=0x7fc938105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:15.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.976+0000 7fc93de73640 1 -- 172.21.15.33:0/2738181910 shutdown_connections 2024-03-19T02:53:15.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:15.976+0000 7fc93de73640 1 -- 172.21.15.33:0/2738181910 wait complete. 2024-03-19T02:53:15.987 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:15 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2008444438' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:53:15.987 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:15 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/297733093' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:53:15.988 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:15 smithi033 ceph-mon[31973]: from='client.15504 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:53:15.988 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:15 smithi033 ceph-mon[31973]: pgmap v375: 129 pgs: 129 active+clean; 648 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-03-19T02:53:15.989 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":375,"stamp":"2024-03-19T02:53:15.742366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":663610,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":864,"num_read_kb":559,"num_write":359,"num_write_kb":3071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1181547,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":791,"ondisk_log_size":791,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":380900,"kb_used_data":15068,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124560924,"statfs":{"total":1151940427776,"available":1151550386176,"internally_reserved":0,"allocated":15429632,"data_stored":8513960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5793,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":94,"num_read_kb":53,"num_write":11,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002895"},"pg_stats":[{"pgid":"3.3f","version":"95'8","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.126776+0000","last_change":"2024-03-19T02:50:52.228368+0000","last_active":"2024-03-19T02:52:41.126776+0000","last_peered":"2024-03-19T02:52:41.126776+0000","last_clean":"2024-03-19T02:52:41.126776+0000","last_became_active":"2024-03-19T02:50:52.228069+0000","last_became_peered":"2024-03-19T02:50:52.228069+0000","last_unstale":"2024-03-19T02:52:41.126776+0000","last_undegraded":"2024-03-19T02:52:41.126776+0000","last_fullsized":"2024-03-19T02:52:41.126776+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:59:34.209095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"95'4","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.230176+0000","last_change":"2024-03-19T02:50:52.231358+0000","last_active":"2024-03-19T02:52:41.230176+0000","last_peered":"2024-03-19T02:52:41.230176+0000","last_clean":"2024-03-19T02:52:41.230176+0000","last_became_active":"2024-03-19T02:50:52.231029+0000","last_became_peered":"2024-03-19T02:50:52.231029+0000","last_unstale":"2024-03-19T02:52:41.230176+0000","last_undegraded":"2024-03-19T02:52:41.230176+0000","last_fullsized":"2024-03-19T02:52:41.230176+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:42:26.031392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"95'7","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.230300+0000","last_change":"2024-03-19T02:50:52.232685+0000","last_active":"2024-03-19T02:52:41.230300+0000","last_peered":"2024-03-19T02:52:41.230300+0000","last_clean":"2024-03-19T02:52:41.230300+0000","last_became_active":"2024-03-19T02:50:52.232361+0000","last_became_peered":"2024-03-19T02:50:52.232361+0000","last_unstale":"2024-03-19T02:52:41.230300+0000","last_undegraded":"2024-03-19T02:52:41.230300+0000","last_fullsized":"2024-03-19T02:52:41.230300+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:53.136200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"95'8","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124441+0000","last_change":"2024-03-19T02:50:52.227648+0000","last_active":"2024-03-19T02:52:41.124441+0000","last_peered":"2024-03-19T02:52:41.124441+0000","last_clean":"2024-03-19T02:52:41.124441+0000","last_became_active":"2024-03-19T02:50:52.226943+0000","last_became_peered":"2024-03-19T02:50:52.226943+0000","last_unstale":"2024-03-19T02:52:41.124441+0000","last_undegraded":"2024-03-19T02:52:41.124441+0000","last_fullsized":"2024-03-19T02:52:41.124441+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:03.293794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"95'4","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.134179+0000","last_change":"2024-03-19T02:50:52.225329+0000","last_active":"2024-03-19T02:52:41.134179+0000","last_peered":"2024-03-19T02:52:41.134179+0000","last_clean":"2024-03-19T02:52:41.134179+0000","last_became_active":"2024-03-19T02:50:52.225012+0000","last_became_peered":"2024-03-19T02:50:52.225012+0000","last_unstale":"2024-03-19T02:52:41.134179+0000","last_undegraded":"2024-03-19T02:52:41.134179+0000","last_fullsized":"2024-03-19T02:52:41.134179+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:32:52.877704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"95'6","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.172669+0000","last_change":"2024-03-19T02:50:52.217399+0000","last_active":"2024-03-19T02:52:41.172669+0000","last_peered":"2024-03-19T02:52:41.172669+0000","last_clean":"2024-03-19T02:52:41.172669+0000","last_became_active":"2024-03-19T02:50:52.216987+0000","last_became_peered":"2024-03-19T02:50:52.216987+0000","last_unstale":"2024-03-19T02:52:41.172669+0000","last_undegraded":"2024-03-19T02:52:41.172669+0000","last_fullsized":"2024-03-19T02:52:41.172669+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:57:15.772546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"95'5","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497233+0000","last_change":"2024-03-19T02:50:52.226755+0000","last_active":"2024-03-19T02:52:41.497233+0000","last_peered":"2024-03-19T02:52:41.497233+0000","last_clean":"2024-03-19T02:52:41.497233+0000","last_became_active":"2024-03-19T02:50:52.226434+0000","last_became_peered":"2024-03-19T02:50:52.226434+0000","last_unstale":"2024-03-19T02:52:41.497233+0000","last_undegraded":"2024-03-19T02:52:41.497233+0000","last_fullsized":"2024-03-19T02:52:41.497233+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:49:42.134262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"95'9","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228846+0000","last_change":"2024-03-19T02:50:52.215848+0000","last_active":"2024-03-19T02:52:41.228846+0000","last_peered":"2024-03-19T02:52:41.228846+0000","last_clean":"2024-03-19T02:52:41.228846+0000","last_became_active":"2024-03-19T02:50:52.215494+0000","last_became_peered":"2024-03-19T02:50:52.215494+0000","last_unstale":"2024-03-19T02:52:41.228846+0000","last_undegraded":"2024-03-19T02:52:41.228846+0000","last_fullsized":"2024-03-19T02:52:41.228846+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:55:10.182359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"95'11","reported_seq":43,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228832+0000","last_change":"2024-03-19T02:50:52.233237+0000","last_active":"2024-03-19T02:52:41.228832+0000","last_peered":"2024-03-19T02:52:41.228832+0000","last_clean":"2024-03-19T02:52:41.228832+0000","last_became_active":"2024-03-19T02:50:52.232091+0000","last_became_peered":"2024-03-19T02:50:52.232091+0000","last_unstale":"2024-03-19T02:52:41.228832+0000","last_undegraded":"2024-03-19T02:52:41.228832+0000","last_fullsized":"2024-03-19T02:52:41.228832+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:08:20.361576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"95'16","reported_seq":55,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.172997+0000","last_change":"2024-03-19T02:51:40.722463+0000","last_active":"2024-03-19T02:52:41.172997+0000","last_peered":"2024-03-19T02:52:41.172997+0000","last_clean":"2024-03-19T02:52:41.172997+0000","last_became_active":"2024-03-19T02:51:40.722120+0000","last_became_peered":"2024-03-19T02:51:40.722120+0000","last_unstale":"2024-03-19T02:52:41.172997+0000","last_undegraded":"2024-03-19T02:52:41.172997+0000","last_fullsized":"2024-03-19T02:52:41.172997+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:49:21.387064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"95'6","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497798+0000","last_change":"2024-03-19T02:50:52.226314+0000","last_active":"2024-03-19T02:52:41.497798+0000","last_peered":"2024-03-19T02:52:41.497798+0000","last_clean":"2024-03-19T02:52:41.497798+0000","last_became_active":"2024-03-19T02:50:52.225919+0000","last_became_peered":"2024-03-19T02:50:52.225919+0000","last_unstale":"2024-03-19T02:52:41.497798+0000","last_undegraded":"2024-03-19T02:52:41.497798+0000","last_fullsized":"2024-03-19T02:52:41.497798+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:42:51.782592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"95'7","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124495+0000","last_change":"2024-03-19T02:50:52.227929+0000","last_active":"2024-03-19T02:52:41.124495+0000","last_peered":"2024-03-19T02:52:41.124495+0000","last_clean":"2024-03-19T02:52:41.124495+0000","last_became_active":"2024-03-19T02:50:52.227360+0000","last_became_peered":"2024-03-19T02:50:52.227360+0000","last_unstale":"2024-03-19T02:52:41.124495+0000","last_undegraded":"2024-03-19T02:52:41.124495+0000","last_fullsized":"2024-03-19T02:52:41.124495+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:24:44.386101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"95'10","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.185886+0000","last_change":"2024-03-19T02:50:52.228088+0000","last_active":"2024-03-19T02:52:41.185886+0000","last_peered":"2024-03-19T02:52:41.185886+0000","last_clean":"2024-03-19T02:52:41.185886+0000","last_became_active":"2024-03-19T02:50:52.227754+0000","last_became_peered":"2024-03-19T02:50:52.227754+0000","last_unstale":"2024-03-19T02:52:41.185886+0000","last_undegraded":"2024-03-19T02:52:41.185886+0000","last_fullsized":"2024-03-19T02:52:41.185886+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:58:57.404241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"95'7","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.230047+0000","last_change":"2024-03-19T02:50:52.226768+0000","last_active":"2024-03-19T02:52:41.230047+0000","last_peered":"2024-03-19T02:52:41.230047+0000","last_clean":"2024-03-19T02:52:41.230047+0000","last_became_active":"2024-03-19T02:50:52.226452+0000","last_became_peered":"2024-03-19T02:50:52.226452+0000","last_unstale":"2024-03-19T02:52:41.230047+0000","last_undegraded":"2024-03-19T02:52:41.230047+0000","last_fullsized":"2024-03-19T02:52:41.230047+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:23:33.250773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"95'7","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.229921+0000","last_change":"2024-03-19T02:50:52.231132+0000","last_active":"2024-03-19T02:52:41.229921+0000","last_peered":"2024-03-19T02:52:41.229921+0000","last_clean":"2024-03-19T02:52:41.229921+0000","last_became_active":"2024-03-19T02:50:52.230778+0000","last_became_peered":"2024-03-19T02:50:52.230778+0000","last_unstale":"2024-03-19T02:52:41.229921+0000","last_undegraded":"2024-03-19T02:52:41.229921+0000","last_fullsized":"2024-03-19T02:52:41.229921+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:33:03.675210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"95'9","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228334+0000","last_change":"2024-03-19T02:50:52.228302+0000","last_active":"2024-03-19T02:52:41.228334+0000","last_peered":"2024-03-19T02:52:41.228334+0000","last_clean":"2024-03-19T02:52:41.228334+0000","last_became_active":"2024-03-19T02:50:52.228007+0000","last_became_peered":"2024-03-19T02:50:52.228007+0000","last_unstale":"2024-03-19T02:52:41.228334+0000","last_undegraded":"2024-03-19T02:52:41.228334+0000","last_fullsized":"2024-03-19T02:52:41.228334+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:59:28.988504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"95'8","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.126552+0000","last_change":"2024-03-19T02:50:52.228640+0000","last_active":"2024-03-19T02:52:41.126552+0000","last_peered":"2024-03-19T02:52:41.126552+0000","last_clean":"2024-03-19T02:52:41.126552+0000","last_became_active":"2024-03-19T02:50:52.228319+0000","last_became_peered":"2024-03-19T02:50:52.228319+0000","last_unstale":"2024-03-19T02:52:41.126552+0000","last_undegraded":"2024-03-19T02:52:41.126552+0000","last_fullsized":"2024-03-19T02:52:41.126552+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:30.356453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"95'10","reported_seq":44,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123572+0000","last_change":"2024-03-19T02:50:52.222300+0000","last_active":"2024-03-19T02:52:41.123572+0000","last_peered":"2024-03-19T02:52:41.123572+0000","last_clean":"2024-03-19T02:52:41.123572+0000","last_became_active":"2024-03-19T02:50:52.221978+0000","last_became_peered":"2024-03-19T02:50:52.221978+0000","last_unstale":"2024-03-19T02:52:41.123572+0000","last_undegraded":"2024-03-19T02:52:41.123572+0000","last_fullsized":"2024-03-19T02:52:41.123572+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:38:36.576987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"95'7","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124386+0000","last_change":"2024-03-19T02:50:52.217640+0000","last_active":"2024-03-19T02:52:41.124386+0000","last_peered":"2024-03-19T02:52:41.124386+0000","last_clean":"2024-03-19T02:52:41.124386+0000","last_became_active":"2024-03-19T02:50:52.217325+0000","last_became_peered":"2024-03-19T02:50:52.217325+0000","last_unstale":"2024-03-19T02:52:41.124386+0000","last_undegraded":"2024-03-19T02:52:41.124386+0000","last_fullsized":"2024-03-19T02:52:41.124386+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:40:25.894843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123733+0000","last_change":"2024-03-19T02:50:57.245323+0000","last_active":"2024-03-19T02:52:41.123733+0000","last_peered":"2024-03-19T02:52:41.123733+0000","last_clean":"2024-03-19T02:52:41.123733+0000","last_became_active":"2024-03-19T02:50:57.244886+0000","last_became_peered":"2024-03-19T02:50:57.244886+0000","last_unstale":"2024-03-19T02:52:41.123733+0000","last_undegraded":"2024-03-19T02:52:41.123733+0000","last_fullsized":"2024-03-19T02:52:41.123733+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:42:09.465738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"93'9","reported_seq":46,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123728+0000","last_change":"2024-03-19T02:50:51.797160+0000","last_active":"2024-03-19T02:52:41.123728+0000","last_peered":"2024-03-19T02:52:41.123728+0000","last_clean":"2024-03-19T02:52:41.123728+0000","last_became_active":"2024-03-19T02:50:51.796783+0000","last_became_peered":"2024-03-19T02:50:51.796783+0000","last_unstale":"2024-03-19T02:52:41.123728+0000","last_undegraded":"2024-03-19T02:52:41.123728+0000","last_fullsized":"2024-03-19T02:52:41.123728+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:57:08.185379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"95'5","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.229622+0000","last_change":"2024-03-19T02:50:52.225114+0000","last_active":"2024-03-19T02:52:41.229622+0000","last_peered":"2024-03-19T02:52:41.229622+0000","last_clean":"2024-03-19T02:52:41.229622+0000","last_became_active":"2024-03-19T02:50:52.220529+0000","last_became_peered":"2024-03-19T02:50:52.220529+0000","last_unstale":"2024-03-19T02:52:41.229622+0000","last_undegraded":"2024-03-19T02:52:41.229622+0000","last_fullsized":"2024-03-19T02:52:41.229622+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:12:51.408689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124051+0000","last_change":"2024-03-19T02:50:57.252827+0000","last_active":"2024-03-19T02:52:41.124051+0000","last_peered":"2024-03-19T02:52:41.124051+0000","last_clean":"2024-03-19T02:52:41.124051+0000","last_became_active":"2024-03-19T02:50:57.252502+0000","last_became_peered":"2024-03-19T02:50:57.252502+0000","last_unstale":"2024-03-19T02:52:41.124051+0000","last_undegraded":"2024-03-19T02:52:41.124051+0000","last_fullsized":"2024-03-19T02:52:41.124051+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:29:36.563533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"93'2","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.229689+0000","last_change":"2024-03-19T02:50:51.797380+0000","last_active":"2024-03-19T02:52:41.229689+0000","last_peered":"2024-03-19T02:52:41.229689+0000","last_clean":"2024-03-19T02:52:41.229689+0000","last_became_active":"2024-03-19T02:50:51.797021+0000","last_became_peered":"2024-03-19T02:50:51.797021+0000","last_unstale":"2024-03-19T02:52:41.229689+0000","last_undegraded":"2024-03-19T02:52:41.229689+0000","last_fullsized":"2024-03-19T02:52:41.229689+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:25.400360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"95'6","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.230538+0000","last_change":"2024-03-19T02:50:52.232717+0000","last_active":"2024-03-19T02:52:41.230538+0000","last_peered":"2024-03-19T02:52:41.230538+0000","last_clean":"2024-03-19T02:52:41.230538+0000","last_became_active":"2024-03-19T02:50:52.232368+0000","last_became_peered":"2024-03-19T02:50:52.232368+0000","last_unstale":"2024-03-19T02:52:41.230538+0000","last_undegraded":"2024-03-19T02:52:41.230538+0000","last_fullsized":"2024-03-19T02:52:41.230538+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:55:38.027120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124226+0000","last_change":"2024-03-19T02:50:57.250840+0000","last_active":"2024-03-19T02:52:41.124226+0000","last_peered":"2024-03-19T02:52:41.124226+0000","last_clean":"2024-03-19T02:52:41.124226+0000","last_became_active":"2024-03-19T02:50:57.250333+0000","last_became_peered":"2024-03-19T02:50:57.250333+0000","last_unstale":"2024-03-19T02:52:41.124226+0000","last_undegraded":"2024-03-19T02:52:41.124226+0000","last_fullsized":"2024-03-19T02:52:41.124226+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:57:09.089939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"93'4","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.134546+0000","last_change":"2024-03-19T02:50:51.796889+0000","last_active":"2024-03-19T02:52:41.134546+0000","last_peered":"2024-03-19T02:52:41.134546+0000","last_clean":"2024-03-19T02:52:41.134546+0000","last_became_active":"2024-03-19T02:50:51.796570+0000","last_became_peered":"2024-03-19T02:50:51.796570+0000","last_unstale":"2024-03-19T02:52:41.134546+0000","last_undegraded":"2024-03-19T02:52:41.134546+0000","last_fullsized":"2024-03-19T02:52:41.134546+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:18:38.301905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"95'9","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228471+0000","last_change":"2024-03-19T02:50:52.224533+0000","last_active":"2024-03-19T02:52:41.228471+0000","last_peered":"2024-03-19T02:52:41.228471+0000","last_clean":"2024-03-19T02:52:41.228471+0000","last_became_active":"2024-03-19T02:50:52.224221+0000","last_became_peered":"2024-03-19T02:50:52.224221+0000","last_unstale":"2024-03-19T02:52:41.228471+0000","last_undegraded":"2024-03-19T02:52:41.228471+0000","last_fullsized":"2024-03-19T02:52:41.228471+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:36:43.335318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124145+0000","last_change":"2024-03-19T02:50:57.252949+0000","last_active":"2024-03-19T02:52:41.124145+0000","last_peered":"2024-03-19T02:52:41.124145+0000","last_clean":"2024-03-19T02:52:41.124145+0000","last_became_active":"2024-03-19T02:50:57.252563+0000","last_became_peered":"2024-03-19T02:50:57.252563+0000","last_unstale":"2024-03-19T02:52:41.124145+0000","last_undegraded":"2024-03-19T02:52:41.124145+0000","last_fullsized":"2024-03-19T02:52:41.124145+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:30:38.244158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.134456+0000","last_change":"2024-03-19T02:50:51.794594+0000","last_active":"2024-03-19T02:52:41.134456+0000","last_peered":"2024-03-19T02:52:41.134456+0000","last_clean":"2024-03-19T02:52:41.134456+0000","last_became_active":"2024-03-19T02:50:51.794197+0000","last_became_peered":"2024-03-19T02:50:51.794197+0000","last_unstale":"2024-03-19T02:52:41.134456+0000","last_undegraded":"2024-03-19T02:52:41.134456+0000","last_fullsized":"2024-03-19T02:52:41.134456+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:58:44.534106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"95'15","reported_seq":48,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.175238+0000","last_change":"2024-03-19T02:50:52.214167+0000","last_active":"2024-03-19T02:52:41.175238+0000","last_peered":"2024-03-19T02:52:41.175238+0000","last_clean":"2024-03-19T02:52:41.175238+0000","last_became_active":"2024-03-19T02:50:52.213873+0000","last_became_peered":"2024-03-19T02:50:52.213873+0000","last_unstale":"2024-03-19T02:52:41.175238+0000","last_undegraded":"2024-03-19T02:52:41.175238+0000","last_fullsized":"2024-03-19T02:52:41.175238+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:28:51.035477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124222+0000","last_change":"2024-03-19T02:50:57.245139+0000","last_active":"2024-03-19T02:52:41.124222+0000","last_peered":"2024-03-19T02:52:41.124222+0000","last_clean":"2024-03-19T02:52:41.124222+0000","last_became_active":"2024-03-19T02:50:57.244636+0000","last_became_peered":"2024-03-19T02:50:57.244636+0000","last_unstale":"2024-03-19T02:52:41.124222+0000","last_undegraded":"2024-03-19T02:52:41.124222+0000","last_fullsized":"2024-03-19T02:52:41.124222+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:44:29.660221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"97'6","reported_seq":161,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:53:14.207172+0000","last_change":"2024-03-19T02:50:51.795969+0000","last_active":"2024-03-19T02:53:14.207172+0000","last_peered":"2024-03-19T02:53:14.207172+0000","last_clean":"2024-03-19T02:53:14.207172+0000","last_became_active":"2024-03-19T02:50:51.795628+0000","last_became_peered":"2024-03-19T02:50:51.795628+0000","last_unstale":"2024-03-19T02:53:14.207172+0000","last_undegraded":"2024-03-19T02:53:14.207172+0000","last_fullsized":"2024-03-19T02:53:14.207172+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:14:48.463282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1115,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":120,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"95'7","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.134019+0000","last_change":"2024-03-19T02:50:52.223278+0000","last_active":"2024-03-19T02:52:41.134019+0000","last_peered":"2024-03-19T02:52:41.134019+0000","last_clean":"2024-03-19T02:52:41.134019+0000","last_became_active":"2024-03-19T02:50:52.222741+0000","last_became_peered":"2024-03-19T02:50:52.222741+0000","last_unstale":"2024-03-19T02:52:41.134019+0000","last_undegraded":"2024-03-19T02:52:41.134019+0000","last_fullsized":"2024-03-19T02:52:41.134019+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:32:09.841825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124151+0000","last_change":"2024-03-19T02:50:57.253130+0000","last_active":"2024-03-19T02:52:41.124151+0000","last_peered":"2024-03-19T02:52:41.124151+0000","last_clean":"2024-03-19T02:52:41.124151+0000","last_became_active":"2024-03-19T02:50:57.252598+0000","last_became_peered":"2024-03-19T02:50:57.252598+0000","last_unstale":"2024-03-19T02:52:41.124151+0000","last_undegraded":"2024-03-19T02:52:41.124151+0000","last_fullsized":"2024-03-19T02:52:41.124151+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:09:38.183430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"93'3","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.172613+0000","last_change":"2024-03-19T02:50:51.794075+0000","last_active":"2024-03-19T02:52:41.172613+0000","last_peered":"2024-03-19T02:52:41.172613+0000","last_clean":"2024-03-19T02:52:41.172613+0000","last_became_active":"2024-03-19T02:50:51.793707+0000","last_became_peered":"2024-03-19T02:50:51.793707+0000","last_unstale":"2024-03-19T02:52:41.172613+0000","last_undegraded":"2024-03-19T02:52:41.172613+0000","last_fullsized":"2024-03-19T02:52:41.172613+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:52:07.180162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"95'8","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.175113+0000","last_change":"2024-03-19T02:50:52.211112+0000","last_active":"2024-03-19T02:52:41.175113+0000","last_peered":"2024-03-19T02:52:41.175113+0000","last_clean":"2024-03-19T02:52:41.175113+0000","last_became_active":"2024-03-19T02:50:52.210790+0000","last_became_peered":"2024-03-19T02:50:52.210790+0000","last_unstale":"2024-03-19T02:52:41.175113+0000","last_undegraded":"2024-03-19T02:52:41.175113+0000","last_fullsized":"2024-03-19T02:52:41.175113+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:39:17.651585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.232069+0000","last_change":"2024-03-19T02:52:41.231645+0000","last_active":"2024-03-19T02:52:41.232069+0000","last_peered":"2024-03-19T02:52:41.232069+0000","last_clean":"2024-03-19T02:52:41.232069+0000","last_became_active":"2024-03-19T02:52:41.231088+0000","last_became_peered":"2024-03-19T02:52:41.231088+0000","last_unstale":"2024-03-19T02:52:41.232069+0000","last_undegraded":"2024-03-19T02:52:41.232069+0000","last_fullsized":"2024-03-19T02:52:41.232069+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:25:29.072005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"93'2","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.185647+0000","last_change":"2024-03-19T02:50:51.785347+0000","last_active":"2024-03-19T02:52:41.185647+0000","last_peered":"2024-03-19T02:52:41.185647+0000","last_clean":"2024-03-19T02:52:41.185647+0000","last_became_active":"2024-03-19T02:50:51.785004+0000","last_became_peered":"2024-03-19T02:50:51.785004+0000","last_unstale":"2024-03-19T02:52:41.185647+0000","last_undegraded":"2024-03-19T02:52:41.185647+0000","last_fullsized":"2024-03-19T02:52:41.185647+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:47.269503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"95'8","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228791+0000","last_change":"2024-03-19T02:50:52.221330+0000","last_active":"2024-03-19T02:52:41.228791+0000","last_peered":"2024-03-19T02:52:41.228791+0000","last_clean":"2024-03-19T02:52:41.228791+0000","last_became_active":"2024-03-19T02:50:52.219417+0000","last_became_peered":"2024-03-19T02:50:52.219417+0000","last_unstale":"2024-03-19T02:52:41.228791+0000","last_undegraded":"2024-03-19T02:52:41.228791+0000","last_fullsized":"2024-03-19T02:52:41.228791+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:28:37.856260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.229931+0000","last_change":"2024-03-19T02:50:57.258357+0000","last_active":"2024-03-19T02:52:41.229931+0000","last_peered":"2024-03-19T02:52:41.229931+0000","last_clean":"2024-03-19T02:52:41.229931+0000","last_became_active":"2024-03-19T02:50:57.257924+0000","last_became_peered":"2024-03-19T02:50:57.257924+0000","last_unstale":"2024-03-19T02:52:41.229931+0000","last_undegraded":"2024-03-19T02:52:41.229931+0000","last_fullsized":"2024-03-19T02:52:41.229931+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:46:05.340731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"93'1","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.174691+0000","last_change":"2024-03-19T02:50:51.796425+0000","last_active":"2024-03-19T02:52:41.174691+0000","last_peered":"2024-03-19T02:52:41.174691+0000","last_clean":"2024-03-19T02:52:41.174691+0000","last_became_active":"2024-03-19T02:50:51.796139+0000","last_became_peered":"2024-03-19T02:50:51.796139+0000","last_unstale":"2024-03-19T02:52:41.174691+0000","last_undegraded":"2024-03-19T02:52:41.174691+0000","last_fullsized":"2024-03-19T02:52:41.174691+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:12:17.123556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"95'11","reported_seq":44,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123343+0000","last_change":"2024-03-19T02:50:52.219375+0000","last_active":"2024-03-19T02:52:41.123343+0000","last_peered":"2024-03-19T02:52:41.123343+0000","last_clean":"2024-03-19T02:52:41.123343+0000","last_became_active":"2024-03-19T02:50:52.219070+0000","last_became_peered":"2024-03-19T02:50:52.219070+0000","last_unstale":"2024-03-19T02:52:41.123343+0000","last_undegraded":"2024-03-19T02:52:41.123343+0000","last_fullsized":"2024-03-19T02:52:41.123343+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:31:58.847700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228989+0000","last_change":"2024-03-19T02:50:57.246502+0000","last_active":"2024-03-19T02:52:41.228989+0000","last_peered":"2024-03-19T02:52:41.228989+0000","last_clean":"2024-03-19T02:52:41.228989+0000","last_became_active":"2024-03-19T02:50:57.246089+0000","last_became_peered":"2024-03-19T02:50:57.246089+0000","last_unstale":"2024-03-19T02:52:41.228989+0000","last_undegraded":"2024-03-19T02:52:41.228989+0000","last_fullsized":"2024-03-19T02:52:41.228989+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:00.162308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"97'4","reported_seq":155,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:53:13.205745+0000","last_change":"2024-03-19T02:50:51.796448+0000","last_active":"2024-03-19T02:53:13.205745+0000","last_peered":"2024-03-19T02:53:13.205745+0000","last_clean":"2024-03-19T02:53:13.205745+0000","last_became_active":"2024-03-19T02:50:51.796132+0000","last_became_peered":"2024-03-19T02:50:51.796132+0000","last_unstale":"2024-03-19T02:53:13.205745+0000","last_undegraded":"2024-03-19T02:53:13.205745+0000","last_fullsized":"2024-03-19T02:53:13.205745+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:37:39.442772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":116,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"95'10","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228710+0000","last_change":"2024-03-19T02:50:52.223022+0000","last_active":"2024-03-19T02:52:41.228710+0000","last_peered":"2024-03-19T02:52:41.228710+0000","last_clean":"2024-03-19T02:52:41.228710+0000","last_became_active":"2024-03-19T02:50:52.221655+0000","last_became_peered":"2024-03-19T02:50:52.221655+0000","last_unstale":"2024-03-19T02:52:41.228710+0000","last_undegraded":"2024-03-19T02:52:41.228710+0000","last_fullsized":"2024-03-19T02:52:41.228710+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:34:36.054088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123149+0000","last_change":"2024-03-19T02:50:57.244003+0000","last_active":"2024-03-19T02:52:41.123149+0000","last_peered":"2024-03-19T02:52:41.123149+0000","last_clean":"2024-03-19T02:52:41.123149+0000","last_became_active":"2024-03-19T02:50:57.243567+0000","last_became_peered":"2024-03-19T02:50:57.243567+0000","last_unstale":"2024-03-19T02:52:41.123149+0000","last_undegraded":"2024-03-19T02:52:41.123149+0000","last_fullsized":"2024-03-19T02:52:41.123149+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:58:52.400237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"93'2","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497747+0000","last_change":"2024-03-19T02:50:51.801259+0000","last_active":"2024-03-19T02:52:41.497747+0000","last_peered":"2024-03-19T02:52:41.497747+0000","last_clean":"2024-03-19T02:52:41.497747+0000","last_became_active":"2024-03-19T02:50:51.800868+0000","last_became_peered":"2024-03-19T02:50:51.800868+0000","last_unstale":"2024-03-19T02:52:41.497747+0000","last_undegraded":"2024-03-19T02:52:41.497747+0000","last_fullsized":"2024-03-19T02:52:41.497747+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:44:06.486687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"95'9","reported_seq":48,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.230496+0000","last_change":"2024-03-19T02:51:40.724476+0000","last_active":"2024-03-19T02:52:41.230496+0000","last_peered":"2024-03-19T02:52:41.230496+0000","last_clean":"2024-03-19T02:52:41.230496+0000","last_became_active":"2024-03-19T02:51:40.724081+0000","last_became_peered":"2024-03-19T02:51:40.724081+0000","last_unstale":"2024-03-19T02:52:41.230496+0000","last_undegraded":"2024-03-19T02:52:41.230496+0000","last_fullsized":"2024-03-19T02:52:41.230496+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:04:15.684753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124330+0000","last_change":"2024-03-19T02:50:57.238764+0000","last_active":"2024-03-19T02:52:41.124330+0000","last_peered":"2024-03-19T02:52:41.124330+0000","last_clean":"2024-03-19T02:52:41.124330+0000","last_became_active":"2024-03-19T02:50:57.238401+0000","last_became_peered":"2024-03-19T02:50:57.238401+0000","last_unstale":"2024-03-19T02:52:41.124330+0000","last_undegraded":"2024-03-19T02:52:41.124330+0000","last_fullsized":"2024-03-19T02:52:41.124330+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:20:46.131776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"93'4","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.175182+0000","last_change":"2024-03-19T02:50:51.797633+0000","last_active":"2024-03-19T02:52:41.175182+0000","last_peered":"2024-03-19T02:52:41.175182+0000","last_clean":"2024-03-19T02:52:41.175182+0000","last_became_active":"2024-03-19T02:50:51.797313+0000","last_became_peered":"2024-03-19T02:50:51.797313+0000","last_unstale":"2024-03-19T02:52:41.175182+0000","last_undegraded":"2024-03-19T02:52:41.175182+0000","last_fullsized":"2024-03-19T02:52:41.175182+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:43:04.659301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"95'11","reported_seq":44,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228621+0000","last_change":"2024-03-19T02:50:52.231486+0000","last_active":"2024-03-19T02:52:41.228621+0000","last_peered":"2024-03-19T02:52:41.228621+0000","last_clean":"2024-03-19T02:52:41.228621+0000","last_became_active":"2024-03-19T02:50:52.231172+0000","last_became_peered":"2024-03-19T02:50:52.231172+0000","last_unstale":"2024-03-19T02:52:41.228621+0000","last_undegraded":"2024-03-19T02:52:41.228621+0000","last_fullsized":"2024-03-19T02:52:41.228621+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:24:49.912633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123242+0000","last_change":"2024-03-19T02:50:57.247207+0000","last_active":"2024-03-19T02:52:41.123242+0000","last_peered":"2024-03-19T02:52:41.123242+0000","last_clean":"2024-03-19T02:52:41.123242+0000","last_became_active":"2024-03-19T02:50:57.246776+0000","last_became_peered":"2024-03-19T02:50:57.246776+0000","last_unstale":"2024-03-19T02:52:41.123242+0000","last_undegraded":"2024-03-19T02:52:41.123242+0000","last_fullsized":"2024-03-19T02:52:41.123242+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:02:24.118858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"97'39","reported_seq":318,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:53:14.207075+0000","last_change":"2024-03-19T02:50:51.792398+0000","last_active":"2024-03-19T02:53:14.207075+0000","last_peered":"2024-03-19T02:53:14.207075+0000","last_clean":"2024-03-19T02:53:14.207075+0000","last_became_active":"2024-03-19T02:50:51.792090+0000","last_became_peered":"2024-03-19T02:50:51.792090+0000","last_unstale":"2024-03-19T02:53:14.207075+0000","last_undegraded":"2024-03-19T02:53:14.207075+0000","last_fullsized":"2024-03-19T02:53:14.207075+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:02:58.175210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68412,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":243,"num_read_kb":189,"num_write":40,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"95'6","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.134354+0000","last_change":"2024-03-19T02:50:52.220755+0000","last_active":"2024-03-19T02:52:41.134354+0000","last_peered":"2024-03-19T02:52:41.134354+0000","last_clean":"2024-03-19T02:52:41.134354+0000","last_became_active":"2024-03-19T02:50:52.220453+0000","last_became_peered":"2024-03-19T02:50:52.220453+0000","last_unstale":"2024-03-19T02:52:41.134354+0000","last_undegraded":"2024-03-19T02:52:41.134354+0000","last_fullsized":"2024-03-19T02:52:41.134354+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:48:27.866863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.175441+0000","last_change":"2024-03-19T02:50:57.253843+0000","last_active":"2024-03-19T02:52:41.175441+0000","last_peered":"2024-03-19T02:52:41.175441+0000","last_clean":"2024-03-19T02:52:41.175441+0000","last_became_active":"2024-03-19T02:50:57.252633+0000","last_became_peered":"2024-03-19T02:50:57.252633+0000","last_unstale":"2024-03-19T02:52:41.175441+0000","last_undegraded":"2024-03-19T02:52:41.175441+0000","last_fullsized":"2024-03-19T02:52:41.175441+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:03:48.514219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"93'6","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124095+0000","last_change":"2024-03-19T02:50:51.795041+0000","last_active":"2024-03-19T02:52:41.124095+0000","last_peered":"2024-03-19T02:52:41.124095+0000","last_clean":"2024-03-19T02:52:41.124095+0000","last_became_active":"2024-03-19T02:50:51.793485+0000","last_became_peered":"2024-03-19T02:50:51.793485+0000","last_unstale":"2024-03-19T02:52:41.124095+0000","last_undegraded":"2024-03-19T02:52:41.124095+0000","last_fullsized":"2024-03-19T02:52:41.124095+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:11:50.136394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"95'9","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.185843+0000","last_change":"2024-03-19T02:50:52.227554+0000","last_active":"2024-03-19T02:52:41.185843+0000","last_peered":"2024-03-19T02:52:41.185843+0000","last_clean":"2024-03-19T02:52:41.185843+0000","last_became_active":"2024-03-19T02:50:52.227222+0000","last_became_peered":"2024-03-19T02:50:52.227222+0000","last_unstale":"2024-03-19T02:52:41.185843+0000","last_undegraded":"2024-03-19T02:52:41.185843+0000","last_fullsized":"2024-03-19T02:52:41.185843+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:56.323749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":23,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.125538+0000","last_change":"2024-03-19T02:50:57.257504+0000","last_active":"2024-03-19T02:52:41.125538+0000","last_peered":"2024-03-19T02:52:41.125538+0000","last_clean":"2024-03-19T02:52:41.125538+0000","last_became_active":"2024-03-19T02:50:57.257120+0000","last_became_peered":"2024-03-19T02:50:57.257120+0000","last_unstale":"2024-03-19T02:52:41.125538+0000","last_undegraded":"2024-03-19T02:52:41.125538+0000","last_fullsized":"2024-03-19T02:52:41.125538+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:19:10.471666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"93'3","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.174956+0000","last_change":"2024-03-19T02:50:51.798049+0000","last_active":"2024-03-19T02:52:41.174956+0000","last_peered":"2024-03-19T02:52:41.174956+0000","last_clean":"2024-03-19T02:52:41.174956+0000","last_became_active":"2024-03-19T02:50:51.797134+0000","last_became_peered":"2024-03-19T02:50:51.797134+0000","last_unstale":"2024-03-19T02:52:41.174956+0000","last_undegraded":"2024-03-19T02:52:41.174956+0000","last_fullsized":"2024-03-19T02:52:41.174956+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:03:46.984240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"95'15","reported_seq":48,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497542+0000","last_change":"2024-03-19T02:50:52.219535+0000","last_active":"2024-03-19T02:52:41.497542+0000","last_peered":"2024-03-19T02:52:41.497542+0000","last_clean":"2024-03-19T02:52:41.497542+0000","last_became_active":"2024-03-19T02:50:52.219226+0000","last_became_peered":"2024-03-19T02:50:52.219226+0000","last_unstale":"2024-03-19T02:52:41.497542+0000","last_undegraded":"2024-03-19T02:52:41.497542+0000","last_fullsized":"2024-03-19T02:52:41.497542+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:27:26.875533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123493+0000","last_change":"2024-03-19T02:50:57.252873+0000","last_active":"2024-03-19T02:52:41.123493+0000","last_peered":"2024-03-19T02:52:41.123493+0000","last_clean":"2024-03-19T02:52:41.123493+0000","last_became_active":"2024-03-19T02:50:57.249551+0000","last_became_peered":"2024-03-19T02:50:57.249551+0000","last_unstale":"2024-03-19T02:52:41.123493+0000","last_undegraded":"2024-03-19T02:52:41.123493+0000","last_fullsized":"2024-03-19T02:52:41.123493+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:33:59.750606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"97'4","reported_seq":159,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:53:15.226100+0000","last_change":"2024-03-19T02:50:51.796478+0000","last_active":"2024-03-19T02:53:15.226100+0000","last_peered":"2024-03-19T02:53:15.226100+0000","last_clean":"2024-03-19T02:53:15.226100+0000","last_became_active":"2024-03-19T02:50:51.796126+0000","last_became_peered":"2024-03-19T02:50:51.796126+0000","last_unstale":"2024-03-19T02:53:15.226100+0000","last_undegraded":"2024-03-19T02:53:15.226100+0000","last_fullsized":"2024-03-19T02:53:15.226100+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:13:42.690481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"95'6","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497479+0000","last_change":"2024-03-19T02:50:52.228151+0000","last_active":"2024-03-19T02:52:41.497479+0000","last_peered":"2024-03-19T02:52:41.497479+0000","last_clean":"2024-03-19T02:52:41.497479+0000","last_became_active":"2024-03-19T02:50:52.227858+0000","last_became_peered":"2024-03-19T02:50:52.227858+0000","last_unstale":"2024-03-19T02:52:41.497479+0000","last_undegraded":"2024-03-19T02:52:41.497479+0000","last_fullsized":"2024-03-19T02:52:41.497479+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:10.285606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.186019+0000","last_change":"2024-03-19T02:50:57.259269+0000","last_active":"2024-03-19T02:52:41.186019+0000","last_peered":"2024-03-19T02:52:41.186019+0000","last_clean":"2024-03-19T02:52:41.186019+0000","last_became_active":"2024-03-19T02:50:57.258778+0000","last_became_peered":"2024-03-19T02:50:57.258778+0000","last_unstale":"2024-03-19T02:52:41.186019+0000","last_undegraded":"2024-03-19T02:52:41.186019+0000","last_fullsized":"2024-03-19T02:52:41.186019+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:02:44.419512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"93'15","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.125166+0000","last_change":"2024-03-19T02:51:40.739522+0000","last_active":"2024-03-19T02:52:41.125166+0000","last_peered":"2024-03-19T02:52:41.125166+0000","last_clean":"2024-03-19T02:52:41.125166+0000","last_became_active":"2024-03-19T02:51:40.720675+0000","last_became_peered":"2024-03-19T02:51:40.720675+0000","last_unstale":"2024-03-19T02:52:41.125166+0000","last_undegraded":"2024-03-19T02:52:41.125166+0000","last_fullsized":"2024-03-19T02:52:41.125166+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:18:19.449850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"95'6","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.174816+0000","last_change":"2024-03-19T02:50:52.215997+0000","last_active":"2024-03-19T02:52:41.174816+0000","last_peered":"2024-03-19T02:52:41.174816+0000","last_clean":"2024-03-19T02:52:41.174816+0000","last_became_active":"2024-03-19T02:50:52.215398+0000","last_became_peered":"2024-03-19T02:50:52.215398+0000","last_unstale":"2024-03-19T02:52:41.174816+0000","last_undegraded":"2024-03-19T02:52:41.174816+0000","last_fullsized":"2024-03-19T02:52:41.174816+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:49:14.010736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.230993+0000","last_change":"2024-03-19T02:52:41.230604+0000","last_active":"2024-03-19T02:52:41.230993+0000","last_peered":"2024-03-19T02:52:41.230993+0000","last_clean":"2024-03-19T02:52:41.230993+0000","last_became_active":"2024-03-19T02:52:41.230292+0000","last_became_peered":"2024-03-19T02:52:41.230292+0000","last_unstale":"2024-03-19T02:52:41.230993+0000","last_undegraded":"2024-03-19T02:52:41.230993+0000","last_fullsized":"2024-03-19T02:52:41.230993+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:55:52.552974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,3,1],"acting":[8,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.f","version":"93'2","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.134663+0000","last_change":"2024-03-19T02:50:51.796246+0000","last_active":"2024-03-19T02:52:41.134663+0000","last_peered":"2024-03-19T02:52:41.134663+0000","last_clean":"2024-03-19T02:52:41.134663+0000","last_became_active":"2024-03-19T02:50:51.795823+0000","last_became_peered":"2024-03-19T02:50:51.795823+0000","last_unstale":"2024-03-19T02:52:41.134663+0000","last_undegraded":"2024-03-19T02:52:41.134663+0000","last_fullsized":"2024-03-19T02:52:41.134663+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:36:02.220891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"95'5","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.125358+0000","last_change":"2024-03-19T02:50:52.224119+0000","last_active":"2024-03-19T02:52:41.125358+0000","last_peered":"2024-03-19T02:52:41.125358+0000","last_clean":"2024-03-19T02:52:41.125358+0000","last_became_active":"2024-03-19T02:50:52.223773+0000","last_became_peered":"2024-03-19T02:50:52.223773+0000","last_unstale":"2024-03-19T02:52:41.125358+0000","last_undegraded":"2024-03-19T02:52:41.125358+0000","last_fullsized":"2024-03-19T02:52:41.125358+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:56:03.098858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.134063+0000","last_change":"2024-03-19T02:50:57.652515+0000","last_active":"2024-03-19T02:52:41.134063+0000","last_peered":"2024-03-19T02:52:41.134063+0000","last_clean":"2024-03-19T02:52:41.134063+0000","last_became_active":"2024-03-19T02:50:57.651808+0000","last_became_peered":"2024-03-19T02:50:57.651808+0000","last_unstale":"2024-03-19T02:52:41.134063+0000","last_undegraded":"2024-03-19T02:52:41.134063+0000","last_fullsized":"2024-03-19T02:52:41.134063+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:16:34.550457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123896+0000","last_change":"2024-03-19T02:50:51.794282+0000","last_active":"2024-03-19T02:52:41.123896+0000","last_peered":"2024-03-19T02:52:41.123896+0000","last_clean":"2024-03-19T02:52:41.123896+0000","last_became_active":"2024-03-19T02:50:51.793947+0000","last_became_peered":"2024-03-19T02:50:51.793947+0000","last_unstale":"2024-03-19T02:52:41.123896+0000","last_undegraded":"2024-03-19T02:52:41.123896+0000","last_fullsized":"2024-03-19T02:52:41.123896+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:08:54.764915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"95'4","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.173032+0000","last_change":"2024-03-19T02:50:52.221707+0000","last_active":"2024-03-19T02:52:41.173032+0000","last_peered":"2024-03-19T02:52:41.173032+0000","last_clean":"2024-03-19T02:52:41.173032+0000","last_became_active":"2024-03-19T02:50:52.221417+0000","last_became_peered":"2024-03-19T02:50:52.221417+0000","last_unstale":"2024-03-19T02:52:41.173032+0000","last_undegraded":"2024-03-19T02:52:41.173032+0000","last_fullsized":"2024-03-19T02:52:41.173032+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:59:27.528346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.185950+0000","last_change":"2024-03-19T02:50:57.257552+0000","last_active":"2024-03-19T02:52:41.185950+0000","last_peered":"2024-03-19T02:52:41.185950+0000","last_clean":"2024-03-19T02:52:41.185950+0000","last_became_active":"2024-03-19T02:50:57.257121+0000","last_became_peered":"2024-03-19T02:50:57.257121+0000","last_unstale":"2024-03-19T02:52:41.185950+0000","last_undegraded":"2024-03-19T02:52:41.185950+0000","last_fullsized":"2024-03-19T02:52:41.185950+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:43:47.145344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"93'1","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497345+0000","last_change":"2024-03-19T02:50:51.804890+0000","last_active":"2024-03-19T02:52:41.497345+0000","last_peered":"2024-03-19T02:52:41.497345+0000","last_clean":"2024-03-19T02:52:41.497345+0000","last_became_active":"2024-03-19T02:50:51.804545+0000","last_became_peered":"2024-03-19T02:50:51.804545+0000","last_unstale":"2024-03-19T02:52:41.497345+0000","last_undegraded":"2024-03-19T02:52:41.497345+0000","last_fullsized":"2024-03-19T02:52:41.497345+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:16:14.945372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"95'7","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.172788+0000","last_change":"2024-03-19T02:50:52.224849+0000","last_active":"2024-03-19T02:52:41.172788+0000","last_peered":"2024-03-19T02:52:41.172788+0000","last_clean":"2024-03-19T02:52:41.172788+0000","last_became_active":"2024-03-19T02:50:52.224550+0000","last_became_peered":"2024-03-19T02:50:52.224550+0000","last_unstale":"2024-03-19T02:52:41.172788+0000","last_undegraded":"2024-03-19T02:52:41.172788+0000","last_fullsized":"2024-03-19T02:52:41.172788+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:28:59.683849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.186106+0000","last_change":"2024-03-19T02:50:57.652425+0000","last_active":"2024-03-19T02:52:41.186106+0000","last_peered":"2024-03-19T02:52:41.186106+0000","last_clean":"2024-03-19T02:52:41.186106+0000","last_became_active":"2024-03-19T02:50:57.651975+0000","last_became_peered":"2024-03-19T02:50:57.651975+0000","last_unstale":"2024-03-19T02:52:41.186106+0000","last_undegraded":"2024-03-19T02:52:41.186106+0000","last_fullsized":"2024-03-19T02:52:41.186106+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:07:20.587918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"95'8","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.174802+0000","last_change":"2024-03-19T02:50:52.218207+0000","last_active":"2024-03-19T02:52:41.174802+0000","last_peered":"2024-03-19T02:52:41.174802+0000","last_clean":"2024-03-19T02:52:41.174802+0000","last_became_active":"2024-03-19T02:50:52.217012+0000","last_became_peered":"2024-03-19T02:50:52.217012+0000","last_unstale":"2024-03-19T02:52:41.174802+0000","last_undegraded":"2024-03-19T02:52:41.174802+0000","last_fullsized":"2024-03-19T02:52:41.174802+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:07:47.366481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"97'5","reported_seq":157,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:53:14.224642+0000","last_change":"2024-03-19T02:50:51.801405+0000","last_active":"2024-03-19T02:53:14.224642+0000","last_peered":"2024-03-19T02:53:14.224642+0000","last_clean":"2024-03-19T02:53:14.224642+0000","last_became_active":"2024-03-19T02:50:51.801094+0000","last_became_peered":"2024-03-19T02:50:51.801094+0000","last_unstale":"2024-03-19T02:53:14.224642+0000","last_undegraded":"2024-03-19T02:53:14.224642+0000","last_fullsized":"2024-03-19T02:53:14.224642+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:53:39.534977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":117,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"93'1","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497740+0000","last_change":"2024-03-19T02:51:40.730836+0000","last_active":"2024-03-19T02:52:41.497740+0000","last_peered":"2024-03-19T02:52:41.497740+0000","last_clean":"2024-03-19T02:52:41.497740+0000","last_became_active":"2024-03-19T02:51:40.723984+0000","last_became_peered":"2024-03-19T02:51:40.723984+0000","last_unstale":"2024-03-19T02:52:41.497740+0000","last_undegraded":"2024-03-19T02:52:41.497740+0000","last_fullsized":"2024-03-19T02:52:41.497740+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:29:57.301066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"95'8","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228900+0000","last_change":"2024-03-19T02:50:52.221852+0000","last_active":"2024-03-19T02:52:41.228900+0000","last_peered":"2024-03-19T02:52:41.228900+0000","last_clean":"2024-03-19T02:52:41.228900+0000","last_became_active":"2024-03-19T02:50:52.221535+0000","last_became_peered":"2024-03-19T02:50:52.221535+0000","last_unstale":"2024-03-19T02:52:41.228900+0000","last_undegraded":"2024-03-19T02:52:41.228900+0000","last_fullsized":"2024-03-19T02:52:41.228900+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:37:18.030714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"93'3","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124331+0000","last_change":"2024-03-19T02:50:51.790231+0000","last_active":"2024-03-19T02:52:41.124331+0000","last_peered":"2024-03-19T02:52:41.124331+0000","last_clean":"2024-03-19T02:52:41.124331+0000","last_became_active":"2024-03-19T02:50:51.789867+0000","last_became_peered":"2024-03-19T02:50:51.789867+0000","last_unstale":"2024-03-19T02:52:41.124331+0000","last_undegraded":"2024-03-19T02:52:41.124331+0000","last_fullsized":"2024-03-19T02:52:41.124331+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:46.099191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":370,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497476+0000","last_change":"2024-03-19T02:49:38.697673+0000","last_active":"2024-03-19T02:52:41.497476+0000","last_peered":"2024-03-19T02:52:41.497476+0000","last_clean":"2024-03-19T02:52:41.497476+0000","last_became_active":"2024-03-19T02:49:38.697161+0000","last_became_peered":"2024-03-19T02:49:38.697161+0000","last_unstale":"2024-03-19T02:52:41.497476+0000","last_undegraded":"2024-03-19T02:52:41.497476+0000","last_fullsized":"2024-03-19T02:52:41.497476+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:46:10.013962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:46:10.013962+0000","last_clean_scrub_stamp":"2024-03-19T02:46:10.013962+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:59:35.179938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.175278+0000","last_change":"2024-03-19T02:50:57.652428+0000","last_active":"2024-03-19T02:52:41.175278+0000","last_peered":"2024-03-19T02:52:41.175278+0000","last_clean":"2024-03-19T02:52:41.175278+0000","last_became_active":"2024-03-19T02:50:57.651864+0000","last_became_peered":"2024-03-19T02:50:57.651864+0000","last_unstale":"2024-03-19T02:52:41.175278+0000","last_undegraded":"2024-03-19T02:52:41.175278+0000","last_fullsized":"2024-03-19T02:52:41.175278+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:52:19.007175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"95'5","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.185550+0000","last_change":"2024-03-19T02:50:52.218728+0000","last_active":"2024-03-19T02:52:41.185550+0000","last_peered":"2024-03-19T02:52:41.185550+0000","last_clean":"2024-03-19T02:52:41.185550+0000","last_became_active":"2024-03-19T02:50:52.218407+0000","last_became_peered":"2024-03-19T02:50:52.218407+0000","last_unstale":"2024-03-19T02:52:41.185550+0000","last_undegraded":"2024-03-19T02:52:41.185550+0000","last_fullsized":"2024-03-19T02:52:41.185550+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:30:49.304034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"93'3","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497416+0000","last_change":"2024-03-19T02:50:51.802057+0000","last_active":"2024-03-19T02:52:41.497416+0000","last_peered":"2024-03-19T02:52:41.497416+0000","last_clean":"2024-03-19T02:52:41.497416+0000","last_became_active":"2024-03-19T02:50:51.801754+0000","last_became_peered":"2024-03-19T02:50:51.801754+0000","last_unstale":"2024-03-19T02:52:41.497416+0000","last_undegraded":"2024-03-19T02:52:41.497416+0000","last_fullsized":"2024-03-19T02:52:41.497416+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:23:15.401822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.125249+0000","last_change":"2024-03-19T02:50:57.260303+0000","last_active":"2024-03-19T02:52:41.125249+0000","last_peered":"2024-03-19T02:52:41.125249+0000","last_clean":"2024-03-19T02:52:41.125249+0000","last_became_active":"2024-03-19T02:50:57.259792+0000","last_became_peered":"2024-03-19T02:50:57.259792+0000","last_unstale":"2024-03-19T02:52:41.125249+0000","last_undegraded":"2024-03-19T02:52:41.125249+0000","last_fullsized":"2024-03-19T02:52:41.125249+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:03.072161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"93'1","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.134478+0000","last_change":"2024-03-19T02:50:51.797949+0000","last_active":"2024-03-19T02:52:41.134478+0000","last_peered":"2024-03-19T02:52:41.134478+0000","last_clean":"2024-03-19T02:52:41.134478+0000","last_became_active":"2024-03-19T02:50:51.797522+0000","last_became_peered":"2024-03-19T02:50:51.797522+0000","last_unstale":"2024-03-19T02:52:41.134478+0000","last_undegraded":"2024-03-19T02:52:41.134478+0000","last_fullsized":"2024-03-19T02:52:41.134478+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:34:06.127993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"95'9","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.186125+0000","last_change":"2024-03-19T02:50:52.224095+0000","last_active":"2024-03-19T02:52:41.186125+0000","last_peered":"2024-03-19T02:52:41.186125+0000","last_clean":"2024-03-19T02:52:41.186125+0000","last_became_active":"2024-03-19T02:50:52.223764+0000","last_became_peered":"2024-03-19T02:50:52.223764+0000","last_unstale":"2024-03-19T02:52:41.186125+0000","last_undegraded":"2024-03-19T02:52:41.186125+0000","last_fullsized":"2024-03-19T02:52:41.186125+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:07:41.503162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228732+0000","last_change":"2024-03-19T02:50:57.652274+0000","last_active":"2024-03-19T02:52:41.228732+0000","last_peered":"2024-03-19T02:52:41.228732+0000","last_clean":"2024-03-19T02:52:41.228732+0000","last_became_active":"2024-03-19T02:50:57.651885+0000","last_became_peered":"2024-03-19T02:50:57.651885+0000","last_unstale":"2024-03-19T02:52:41.228732+0000","last_undegraded":"2024-03-19T02:52:41.228732+0000","last_fullsized":"2024-03-19T02:52:41.228732+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:03:02.081143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"97'10","reported_seq":46,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:53:03.631782+0000","last_change":"2024-03-19T02:50:51.796308+0000","last_active":"2024-03-19T02:53:03.631782+0000","last_peered":"2024-03-19T02:53:03.631782+0000","last_clean":"2024-03-19T02:53:03.631782+0000","last_became_active":"2024-03-19T02:50:51.795990+0000","last_became_peered":"2024-03-19T02:50:51.795990+0000","last_unstale":"2024-03-19T02:53:03.631782+0000","last_undegraded":"2024-03-19T02:53:03.631782+0000","last_fullsized":"2024-03-19T02:53:03.631782+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:45:53.608817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"95'5","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.229132+0000","last_change":"2024-03-19T02:50:52.221983+0000","last_active":"2024-03-19T02:52:41.229132+0000","last_peered":"2024-03-19T02:52:41.229132+0000","last_clean":"2024-03-19T02:52:41.229132+0000","last_became_active":"2024-03-19T02:50:52.220605+0000","last_became_peered":"2024-03-19T02:50:52.220605+0000","last_unstale":"2024-03-19T02:52:41.229132+0000","last_undegraded":"2024-03-19T02:52:41.229132+0000","last_fullsized":"2024-03-19T02:52:41.229132+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:12.834159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123697+0000","last_change":"2024-03-19T02:50:57.242428+0000","last_active":"2024-03-19T02:52:41.123697+0000","last_peered":"2024-03-19T02:52:41.123697+0000","last_clean":"2024-03-19T02:52:41.123697+0000","last_became_active":"2024-03-19T02:50:57.242007+0000","last_became_peered":"2024-03-19T02:50:57.242007+0000","last_unstale":"2024-03-19T02:52:41.123697+0000","last_undegraded":"2024-03-19T02:52:41.123697+0000","last_fullsized":"2024-03-19T02:52:41.123697+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:23:30.987761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"93'4","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124922+0000","last_change":"2024-03-19T02:50:51.791342+0000","last_active":"2024-03-19T02:52:41.124922+0000","last_peered":"2024-03-19T02:52:41.124922+0000","last_clean":"2024-03-19T02:52:41.124922+0000","last_became_active":"2024-03-19T02:50:51.791039+0000","last_became_peered":"2024-03-19T02:50:51.791039+0000","last_unstale":"2024-03-19T02:52:41.124922+0000","last_undegraded":"2024-03-19T02:52:41.124922+0000","last_fullsized":"2024-03-19T02:52:41.124922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:50:38.216623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"95'7","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.230040+0000","last_change":"2024-03-19T02:50:52.224303+0000","last_active":"2024-03-19T02:52:41.230040+0000","last_peered":"2024-03-19T02:52:41.230040+0000","last_clean":"2024-03-19T02:52:41.230040+0000","last_became_active":"2024-03-19T02:50:52.223942+0000","last_became_peered":"2024-03-19T02:50:52.223942+0000","last_unstale":"2024-03-19T02:52:41.230040+0000","last_undegraded":"2024-03-19T02:52:41.230040+0000","last_fullsized":"2024-03-19T02:52:41.230040+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:57:04.175971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124685+0000","last_change":"2024-03-19T02:50:57.652126+0000","last_active":"2024-03-19T02:52:41.124685+0000","last_peered":"2024-03-19T02:52:41.124685+0000","last_clean":"2024-03-19T02:52:41.124685+0000","last_became_active":"2024-03-19T02:50:57.651713+0000","last_became_peered":"2024-03-19T02:50:57.651713+0000","last_unstale":"2024-03-19T02:52:41.124685+0000","last_undegraded":"2024-03-19T02:52:41.124685+0000","last_fullsized":"2024-03-19T02:52:41.124685+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:28:51.077437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"93'2","reported_seq":47,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228284+0000","last_change":"2024-03-19T02:51:40.741378+0000","last_active":"2024-03-19T02:52:41.228284+0000","last_peered":"2024-03-19T02:52:41.228284+0000","last_clean":"2024-03-19T02:52:41.228284+0000","last_became_active":"2024-03-19T02:51:40.724510+0000","last_became_peered":"2024-03-19T02:51:40.724510+0000","last_unstale":"2024-03-19T02:52:41.228284+0000","last_undegraded":"2024-03-19T02:52:41.228284+0000","last_fullsized":"2024-03-19T02:52:41.228284+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:30:16.228635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"95'2","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.133963+0000","last_change":"2024-03-19T02:50:52.221681+0000","last_active":"2024-03-19T02:52:41.133963+0000","last_peered":"2024-03-19T02:52:41.133963+0000","last_clean":"2024-03-19T02:52:41.133963+0000","last_became_active":"2024-03-19T02:50:52.220987+0000","last_became_peered":"2024-03-19T02:50:52.220987+0000","last_unstale":"2024-03-19T02:52:41.133963+0000","last_undegraded":"2024-03-19T02:52:41.133963+0000","last_fullsized":"2024-03-19T02:52:41.133963+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:16:36.561143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123630+0000","last_change":"2024-03-19T02:50:57.249830+0000","last_active":"2024-03-19T02:52:41.123630+0000","last_peered":"2024-03-19T02:52:41.123630+0000","last_clean":"2024-03-19T02:52:41.123630+0000","last_became_active":"2024-03-19T02:50:57.248879+0000","last_became_peered":"2024-03-19T02:50:57.248879+0000","last_unstale":"2024-03-19T02:52:41.123630+0000","last_undegraded":"2024-03-19T02:52:41.123630+0000","last_fullsized":"2024-03-19T02:52:41.123630+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:58:43.225841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"93'3","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.229197+0000","last_change":"2024-03-19T02:50:51.801374+0000","last_active":"2024-03-19T02:52:41.229197+0000","last_peered":"2024-03-19T02:52:41.229197+0000","last_clean":"2024-03-19T02:52:41.229197+0000","last_became_active":"2024-03-19T02:50:51.801041+0000","last_became_peered":"2024-03-19T02:50:51.801041+0000","last_unstale":"2024-03-19T02:52:41.229197+0000","last_undegraded":"2024-03-19T02:52:41.229197+0000","last_fullsized":"2024-03-19T02:52:41.229197+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:56:14.671718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"95'9","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.125323+0000","last_change":"2024-03-19T02:50:52.225915+0000","last_active":"2024-03-19T02:52:41.125323+0000","last_peered":"2024-03-19T02:52:41.125323+0000","last_clean":"2024-03-19T02:52:41.125323+0000","last_became_active":"2024-03-19T02:50:52.225606+0000","last_became_peered":"2024-03-19T02:50:52.225606+0000","last_unstale":"2024-03-19T02:52:41.125323+0000","last_undegraded":"2024-03-19T02:52:41.125323+0000","last_fullsized":"2024-03-19T02:52:41.125323+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:27:29.549255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"93'1","reported_seq":27,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124644+0000","last_change":"2024-03-19T02:50:57.240747+0000","last_active":"2024-03-19T02:52:41.124644+0000","last_peered":"2024-03-19T02:52:41.124644+0000","last_clean":"2024-03-19T02:52:41.124644+0000","last_became_active":"2024-03-19T02:50:57.240373+0000","last_became_peered":"2024-03-19T02:50:57.240373+0000","last_unstale":"2024-03-19T02:52:41.124644+0000","last_undegraded":"2024-03-19T02:52:41.124644+0000","last_fullsized":"2024-03-19T02:52:41.124644+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:22:04.991497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"97'2","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:53:03.898093+0000","last_change":"2024-03-19T02:50:51.794071+0000","last_active":"2024-03-19T02:53:03.898093+0000","last_peered":"2024-03-19T02:53:03.898093+0000","last_clean":"2024-03-19T02:53:03.898093+0000","last_became_active":"2024-03-19T02:50:51.793570+0000","last_became_peered":"2024-03-19T02:50:51.793570+0000","last_unstale":"2024-03-19T02:53:03.898093+0000","last_undegraded":"2024-03-19T02:53:03.898093+0000","last_fullsized":"2024-03-19T02:53:03.898093+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:34:20.456515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"95'8","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.229465+0000","last_change":"2024-03-19T02:50:52.223341+0000","last_active":"2024-03-19T02:52:41.229465+0000","last_peered":"2024-03-19T02:52:41.229465+0000","last_clean":"2024-03-19T02:52:41.229465+0000","last_became_active":"2024-03-19T02:50:52.220280+0000","last_became_peered":"2024-03-19T02:50:52.220280+0000","last_unstale":"2024-03-19T02:52:41.229465+0000","last_undegraded":"2024-03-19T02:52:41.229465+0000","last_fullsized":"2024-03-19T02:52:41.229465+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:04:15.830811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124197+0000","last_change":"2024-03-19T02:50:57.651293+0000","last_active":"2024-03-19T02:52:41.124197+0000","last_peered":"2024-03-19T02:52:41.124197+0000","last_clean":"2024-03-19T02:52:41.124197+0000","last_became_active":"2024-03-19T02:50:57.650893+0000","last_became_peered":"2024-03-19T02:50:57.650893+0000","last_unstale":"2024-03-19T02:52:41.124197+0000","last_undegraded":"2024-03-19T02:52:41.124197+0000","last_fullsized":"2024-03-19T02:52:41.124197+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:33:35.984728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"93'4","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.134128+0000","last_change":"2024-03-19T02:50:51.798297+0000","last_active":"2024-03-19T02:52:41.134128+0000","last_peered":"2024-03-19T02:52:41.134128+0000","last_clean":"2024-03-19T02:52:41.134128+0000","last_became_active":"2024-03-19T02:50:51.797516+0000","last_became_peered":"2024-03-19T02:50:51.797516+0000","last_unstale":"2024-03-19T02:52:41.134128+0000","last_undegraded":"2024-03-19T02:52:41.134128+0000","last_fullsized":"2024-03-19T02:52:41.134128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:09:42.295094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"95'5","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497375+0000","last_change":"2024-03-19T02:50:52.224011+0000","last_active":"2024-03-19T02:52:41.497375+0000","last_peered":"2024-03-19T02:52:41.497375+0000","last_clean":"2024-03-19T02:52:41.497375+0000","last_became_active":"2024-03-19T02:50:52.223708+0000","last_became_peered":"2024-03-19T02:50:52.223708+0000","last_unstale":"2024-03-19T02:52:41.497375+0000","last_undegraded":"2024-03-19T02:52:41.497375+0000","last_fullsized":"2024-03-19T02:52:41.497375+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:17:49.305587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228248+0000","last_change":"2024-03-19T02:50:57.251134+0000","last_active":"2024-03-19T02:52:41.228248+0000","last_peered":"2024-03-19T02:52:41.228248+0000","last_clean":"2024-03-19T02:52:41.228248+0000","last_became_active":"2024-03-19T02:50:57.250761+0000","last_became_peered":"2024-03-19T02:50:57.250761+0000","last_unstale":"2024-03-19T02:52:41.228248+0000","last_undegraded":"2024-03-19T02:52:41.228248+0000","last_fullsized":"2024-03-19T02:52:41.228248+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:09:26.860235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"93'3","reported_seq":21,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124386+0000","last_change":"2024-03-19T02:51:40.728077+0000","last_active":"2024-03-19T02:52:41.124386+0000","last_peered":"2024-03-19T02:52:41.124386+0000","last_clean":"2024-03-19T02:52:41.124386+0000","last_became_active":"2024-03-19T02:51:40.720767+0000","last_became_peered":"2024-03-19T02:51:40.720767+0000","last_unstale":"2024-03-19T02:52:41.124386+0000","last_undegraded":"2024-03-19T02:52:41.124386+0000","last_fullsized":"2024-03-19T02:52:41.124386+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:22:34.366373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"95'7","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497892+0000","last_change":"2024-03-19T02:50:52.224776+0000","last_active":"2024-03-19T02:52:41.497892+0000","last_peered":"2024-03-19T02:52:41.497892+0000","last_clean":"2024-03-19T02:52:41.497892+0000","last_became_active":"2024-03-19T02:50:52.224442+0000","last_became_peered":"2024-03-19T02:50:52.224442+0000","last_unstale":"2024-03-19T02:52:41.497892+0000","last_undegraded":"2024-03-19T02:52:41.497892+0000","last_fullsized":"2024-03-19T02:52:41.497892+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:11:13.722005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.230221+0000","last_change":"2024-03-19T02:51:40.722732+0000","last_active":"2024-03-19T02:52:41.230221+0000","last_peered":"2024-03-19T02:52:41.230221+0000","last_clean":"2024-03-19T02:52:41.230221+0000","last_became_active":"2024-03-19T02:51:40.722260+0000","last_became_peered":"2024-03-19T02:51:40.722260+0000","last_unstale":"2024-03-19T02:52:41.230221+0000","last_undegraded":"2024-03-19T02:52:41.230221+0000","last_fullsized":"2024-03-19T02:52:41.230221+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:31:06.383570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.175385+0000","last_change":"2024-03-19T02:50:51.794898+0000","last_active":"2024-03-19T02:52:41.175385+0000","last_peered":"2024-03-19T02:52:41.175385+0000","last_clean":"2024-03-19T02:52:41.175385+0000","last_became_active":"2024-03-19T02:50:51.794562+0000","last_became_peered":"2024-03-19T02:50:51.794562+0000","last_unstale":"2024-03-19T02:52:41.175385+0000","last_undegraded":"2024-03-19T02:52:41.175385+0000","last_fullsized":"2024-03-19T02:52:41.175385+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:17:45.239395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"95'6","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497460+0000","last_change":"2024-03-19T02:50:52.227427+0000","last_active":"2024-03-19T02:52:41.497460+0000","last_peered":"2024-03-19T02:52:41.497460+0000","last_clean":"2024-03-19T02:52:41.497460+0000","last_became_active":"2024-03-19T02:50:52.227111+0000","last_became_peered":"2024-03-19T02:50:52.227111+0000","last_unstale":"2024-03-19T02:52:41.497460+0000","last_undegraded":"2024-03-19T02:52:41.497460+0000","last_fullsized":"2024-03-19T02:52:41.497460+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:40:53.295454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.174601+0000","last_change":"2024-03-19T02:50:57.244449+0000","last_active":"2024-03-19T02:52:41.174601+0000","last_peered":"2024-03-19T02:52:41.174601+0000","last_clean":"2024-03-19T02:52:41.174601+0000","last_became_active":"2024-03-19T02:50:57.243991+0000","last_became_peered":"2024-03-19T02:50:57.243991+0000","last_unstale":"2024-03-19T02:52:41.174601+0000","last_undegraded":"2024-03-19T02:52:41.174601+0000","last_fullsized":"2024-03-19T02:52:41.174601+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:50:45.265773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.186101+0000","last_change":"2024-03-19T02:50:51.799834+0000","last_active":"2024-03-19T02:52:41.186101+0000","last_peered":"2024-03-19T02:52:41.186101+0000","last_clean":"2024-03-19T02:52:41.186101+0000","last_became_active":"2024-03-19T02:50:51.799563+0000","last_became_peered":"2024-03-19T02:50:51.799563+0000","last_unstale":"2024-03-19T02:52:41.186101+0000","last_undegraded":"2024-03-19T02:52:41.186101+0000","last_fullsized":"2024-03-19T02:52:41.186101+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:51:00.284454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"95'9","reported_seq":43,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497913+0000","last_change":"2024-03-19T02:50:52.222710+0000","last_active":"2024-03-19T02:52:41.497913+0000","last_peered":"2024-03-19T02:52:41.497913+0000","last_clean":"2024-03-19T02:52:41.497913+0000","last_became_active":"2024-03-19T02:50:52.222321+0000","last_became_peered":"2024-03-19T02:50:52.222321+0000","last_unstale":"2024-03-19T02:52:41.497913+0000","last_undegraded":"2024-03-19T02:52:41.497913+0000","last_fullsized":"2024-03-19T02:52:41.497913+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:05:27.527696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"95'9","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.125423+0000","last_change":"2024-03-19T02:50:52.226631+0000","last_active":"2024-03-19T02:52:41.125423+0000","last_peered":"2024-03-19T02:52:41.125423+0000","last_clean":"2024-03-19T02:52:41.125423+0000","last_became_active":"2024-03-19T02:50:52.226294+0000","last_became_peered":"2024-03-19T02:50:52.226294+0000","last_unstale":"2024-03-19T02:52:41.125423+0000","last_undegraded":"2024-03-19T02:52:41.125423+0000","last_fullsized":"2024-03-19T02:52:41.125423+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:01:31.350063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"95'12","reported_seq":45,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123985+0000","last_change":"2024-03-19T02:50:52.221138+0000","last_active":"2024-03-19T02:52:41.123985+0000","last_peered":"2024-03-19T02:52:41.123985+0000","last_clean":"2024-03-19T02:52:41.123985+0000","last_became_active":"2024-03-19T02:50:52.220730+0000","last_became_peered":"2024-03-19T02:50:52.220730+0000","last_unstale":"2024-03-19T02:52:41.123985+0000","last_undegraded":"2024-03-19T02:52:41.123985+0000","last_fullsized":"2024-03-19T02:52:41.123985+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:26:13.059986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"95'8","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123763+0000","last_change":"2024-03-19T02:50:52.223567+0000","last_active":"2024-03-19T02:52:41.123763+0000","last_peered":"2024-03-19T02:52:41.123763+0000","last_clean":"2024-03-19T02:52:41.123763+0000","last_became_active":"2024-03-19T02:50:52.223174+0000","last_became_peered":"2024-03-19T02:50:52.223174+0000","last_unstale":"2024-03-19T02:52:41.123763+0000","last_undegraded":"2024-03-19T02:52:41.123763+0000","last_fullsized":"2024-03-19T02:52:41.123763+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:23:06.383584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"95'7","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.230567+0000","last_change":"2024-03-19T02:50:52.229916+0000","last_active":"2024-03-19T02:52:41.230567+0000","last_peered":"2024-03-19T02:52:41.230567+0000","last_clean":"2024-03-19T02:52:41.230567+0000","last_became_active":"2024-03-19T02:50:52.229597+0000","last_became_peered":"2024-03-19T02:50:52.229597+0000","last_unstale":"2024-03-19T02:52:41.230567+0000","last_undegraded":"2024-03-19T02:52:41.230567+0000","last_fullsized":"2024-03-19T02:52:41.230567+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:19.331893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"95'4","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.186073+0000","last_change":"2024-03-19T02:50:52.223331+0000","last_active":"2024-03-19T02:52:41.186073+0000","last_peered":"2024-03-19T02:52:41.186073+0000","last_clean":"2024-03-19T02:52:41.186073+0000","last_became_active":"2024-03-19T02:50:52.222975+0000","last_became_peered":"2024-03-19T02:50:52.222975+0000","last_unstale":"2024-03-19T02:52:41.186073+0000","last_undegraded":"2024-03-19T02:52:41.186073+0000","last_fullsized":"2024-03-19T02:52:41.186073+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:26:23.023749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"95'6","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.125760+0000","last_change":"2024-03-19T02:50:52.218073+0000","last_active":"2024-03-19T02:52:41.125760+0000","last_peered":"2024-03-19T02:52:41.125760+0000","last_clean":"2024-03-19T02:52:41.125760+0000","last_became_active":"2024-03-19T02:50:52.217742+0000","last_became_peered":"2024-03-19T02:50:52.217742+0000","last_unstale":"2024-03-19T02:52:41.125760+0000","last_undegraded":"2024-03-19T02:52:41.125760+0000","last_fullsized":"2024-03-19T02:52:41.125760+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:08:51.053153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"95'15","reported_seq":48,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124597+0000","last_change":"2024-03-19T02:50:52.222108+0000","last_active":"2024-03-19T02:52:41.124597+0000","last_peered":"2024-03-19T02:52:41.124597+0000","last_clean":"2024-03-19T02:52:41.124597+0000","last_became_active":"2024-03-19T02:50:52.221748+0000","last_became_peered":"2024-03-19T02:50:52.221748+0000","last_unstale":"2024-03-19T02:52:41.124597+0000","last_undegraded":"2024-03-19T02:52:41.124597+0000","last_fullsized":"2024-03-19T02:52:41.124597+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:33:31.401939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"95'2","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.229311+0000","last_change":"2024-03-19T02:50:52.221799+0000","last_active":"2024-03-19T02:52:41.229311+0000","last_peered":"2024-03-19T02:52:41.229311+0000","last_clean":"2024-03-19T02:52:41.229311+0000","last_became_active":"2024-03-19T02:50:52.220196+0000","last_became_peered":"2024-03-19T02:50:52.220196+0000","last_unstale":"2024-03-19T02:52:41.229311+0000","last_undegraded":"2024-03-19T02:52:41.229311+0000","last_fullsized":"2024-03-19T02:52:41.229311+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:24:01.541716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"95'6","reported_seq":21,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.185475+0000","last_change":"2024-03-19T02:51:40.721422+0000","last_active":"2024-03-19T02:52:41.185475+0000","last_peered":"2024-03-19T02:52:41.185475+0000","last_clean":"2024-03-19T02:52:41.185475+0000","last_became_active":"2024-03-19T02:51:40.720982+0000","last_became_peered":"2024-03-19T02:51:40.720982+0000","last_unstale":"2024-03-19T02:52:41.185475+0000","last_undegraded":"2024-03-19T02:52:41.185475+0000","last_fullsized":"2024-03-19T02:52:41.185475+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:07.391810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,3],"acting":[6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.29","version":"95'8","reported_seq":23,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123111+0000","last_change":"2024-03-19T02:51:40.721188+0000","last_active":"2024-03-19T02:52:41.123111+0000","last_peered":"2024-03-19T02:52:41.123111+0000","last_clean":"2024-03-19T02:52:41.123111+0000","last_became_active":"2024-03-19T02:51:40.720811+0000","last_became_peered":"2024-03-19T02:51:40.720811+0000","last_unstale":"2024-03-19T02:52:41.123111+0000","last_undegraded":"2024-03-19T02:52:41.123111+0000","last_fullsized":"2024-03-19T02:52:41.123111+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:23.970149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2a","version":"95'10","reported_seq":43,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228360+0000","last_change":"2024-03-19T02:50:52.230847+0000","last_active":"2024-03-19T02:52:41.228360+0000","last_peered":"2024-03-19T02:52:41.228360+0000","last_clean":"2024-03-19T02:52:41.228360+0000","last_became_active":"2024-03-19T02:50:52.230478+0000","last_became_peered":"2024-03-19T02:50:52.230478+0000","last_unstale":"2024-03-19T02:52:41.228360+0000","last_undegraded":"2024-03-19T02:52:41.228360+0000","last_fullsized":"2024-03-19T02:52:41.228360+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:04:09.459446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"95'11","reported_seq":44,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.186018+0000","last_change":"2024-03-19T02:50:52.223508+0000","last_active":"2024-03-19T02:52:41.186018+0000","last_peered":"2024-03-19T02:52:41.186018+0000","last_clean":"2024-03-19T02:52:41.186018+0000","last_became_active":"2024-03-19T02:50:52.223173+0000","last_became_peered":"2024-03-19T02:50:52.223173+0000","last_unstale":"2024-03-19T02:52:41.186018+0000","last_undegraded":"2024-03-19T02:52:41.186018+0000","last_fullsized":"2024-03-19T02:52:41.186018+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:53.320173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"95'8","reported_seq":47,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.125738+0000","last_change":"2024-03-19T02:51:40.723512+0000","last_active":"2024-03-19T02:52:41.125738+0000","last_peered":"2024-03-19T02:52:41.125738+0000","last_clean":"2024-03-19T02:52:41.125738+0000","last_became_active":"2024-03-19T02:51:40.723149+0000","last_became_peered":"2024-03-19T02:51:40.723149+0000","last_unstale":"2024-03-19T02:52:41.125738+0000","last_undegraded":"2024-03-19T02:52:41.125738+0000","last_fullsized":"2024-03-19T02:52:41.125738+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:42:38.862569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":73008,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":728,"num_read_kb":440,"num_write":176,"num_write_kb":131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":694,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":219024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152,"ondisk_log_size":152,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318314,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36432,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708720,"statfs":{"total":95995035648,"available":95957729280,"internally_reserved":0,"allocated":1114112,"data_stored":545712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579951,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28780,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716372,"statfs":{"total":95995035648,"available":95965564928,"internally_reserved":0,"allocated":1806336,"data_stored":1205869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743477,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36432,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708720,"statfs":{"total":95995035648,"available":95957729280,"internally_reserved":0,"allocated":1122304,"data_stored":545836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005113,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36432,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708720,"statfs":{"total":95995035648,"available":95957729280,"internally_reserved":0,"allocated":1114112,"data_stored":545644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266751,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37012,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708140,"statfs":{"total":95995035648,"available":95957135360,"internally_reserved":0,"allocated":1716224,"data_stored":1142260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462979,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1101824,"data_stored":541012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36484,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708668,"statfs":{"total":95995035648,"available":95957676032,"internally_reserved":0,"allocated":1175552,"data_stored":609334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1126400,"data_stored":542400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28184,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716968,"statfs":{"total":95995035648,"available":95966175232,"internally_reserved":0,"allocated":1196032,"data_stored":610595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149780,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1114112,"data_stored":541206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888143,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1118208,"data_stored":546737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724618,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28700,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716452,"statfs":{"total":95995035648,"available":95965646848,"internally_reserved":0,"allocated":1724416,"data_stored":1137355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":68412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":69673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":70483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T02:53:15.994 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-19T02:53:15.994 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T02:53:16.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.131+0000 7f958428c640 1 -- 172.21.15.33:0/3111745294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f957c102a30 msgr2=0x7f957c102eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:16.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.131+0000 7f958428c640 1 --2- 172.21.15.33:0/3111745294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f957c102a30 0x7f957c102eb0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f956c009900 tx=0x7f956c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:16.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.131+0000 7f958428c640 1 -- 172.21.15.33:0/3111745294 shutdown_connections 2024-03-19T02:53:16.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.131+0000 7f958428c640 1 --2- 172.21.15.33:0/3111745294 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f957c1033f0 0x7f957c109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:16.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.131+0000 7f958428c640 1 --2- 172.21.15.33:0/3111745294 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f957c102a30 0x7f957c102eb0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:16.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.131+0000 7f958428c640 1 --2- 172.21.15.33:0/3111745294 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f957c101840 0x7f957c101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:16.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.131+0000 7f958428c640 1 -- 172.21.15.33:0/3111745294 >> 172.21.15.33:0/3111745294 conn(0x7f957c0fcdd0 msgr2=0x7f957c0ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:16.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.131+0000 7f958428c640 1 -- 172.21.15.33:0/3111745294 shutdown_connections 2024-03-19T02:53:16.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.131+0000 7f958428c640 1 -- 172.21.15.33:0/3111745294 wait complete. 2024-03-19T02:53:16.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.132+0000 7f958428c640 1 Processor -- start 2024-03-19T02:53:16.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.132+0000 7f958428c640 1 -- start start 2024-03-19T02:53:16.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.132+0000 7f958428c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f957c101840 0x7f957c197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:16.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.132+0000 7f958428c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f957c102a30 0x7f957c197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:16.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.132+0000 7f958428c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f957c1033f0 0x7f957c19f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:16.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.132+0000 7f958428c640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f957c10d370 con 0x7f957c1033f0 2024-03-19T02:53:16.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.132+0000 7f958428c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f957c10d1f0 con 0x7f957c101840 2024-03-19T02:53:16.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.132+0000 7f958428c640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f957c10d4f0 con 0x7f957c102a30 2024-03-19T02:53:16.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.132+0000 7f9582829640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f957c1033f0 0x7f957c19f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:16.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.132+0000 7f9582829640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f957c1033f0 0x7f957c19f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46804/0 (socket says 172.21.15.33:46804) 2024-03-19T02:53:16.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.132+0000 7f9582829640 1 -- 172.21.15.33:0/418614212 learned_addr learned my addr 172.21.15.33:0/418614212 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:16.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.132+0000 7f9581827640 1 --2- 172.21.15.33:0/418614212 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f957c102a30 0x7f957c197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:16.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.132+0000 7f9582028640 1 --2- 172.21.15.33:0/418614212 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f957c101840 0x7f957c197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:16.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.132+0000 7f9582829640 1 -- 172.21.15.33:0/418614212 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f957c102a30 msgr2=0x7f957c197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:16.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.132+0000 7f9582829640 1 --2- 172.21.15.33:0/418614212 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f957c102a30 0x7f957c197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:16.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.132+0000 7f9582829640 1 -- 172.21.15.33:0/418614212 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f957c101840 msgr2=0x7f957c197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:16.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.132+0000 7f9582829640 1 --2- 172.21.15.33:0/418614212 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f957c101840 0x7f957c197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:16.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.132+0000 7f9582829640 1 -- 172.21.15.33:0/418614212 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f957c19f6b0 con 0x7f957c1033f0 2024-03-19T02:53:16.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.132+0000 7f9582829640 1 --2- 172.21.15.33:0/418614212 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f957c1033f0 0x7f957c19f040 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f957400eb10 tx=0x7f957400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:16.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.133+0000 7f95737fe640 1 -- 172.21.15.33:0/418614212 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9574004b20 con 0x7f957c1033f0 2024-03-19T02:53:16.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.133+0000 7f95737fe640 1 -- 172.21.15.33:0/418614212 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9574004cc0 con 0x7f957c1033f0 2024-03-19T02:53:16.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.133+0000 7f958428c640 1 -- 172.21.15.33:0/418614212 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f957c19f9a0 con 0x7f957c1033f0 2024-03-19T02:53:16.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.133+0000 7f95737fe640 1 -- 172.21.15.33:0/418614212 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9574010690 con 0x7f957c1033f0 2024-03-19T02:53:16.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.133+0000 7f958428c640 1 -- 172.21.15.33:0/418614212 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f957c19fec0 con 0x7f957c1033f0 2024-03-19T02:53:16.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.134+0000 7f958428c640 1 -- 172.21.15.33:0/418614212 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f957c108c30 con 0x7f957c1033f0 2024-03-19T02:53:16.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.135+0000 7f95737fe640 1 -- 172.21.15.33:0/418614212 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9574005020 con 0x7f957c1033f0 2024-03-19T02:53:16.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.135+0000 7f95737fe640 1 --2- 172.21.15.33:0/418614212 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f95680a2880 0x7f95680a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:16.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.135+0000 7f9582028640 1 --2- 172.21.15.33:0/418614212 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f95680a2880 0x7f95680a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:16.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.135+0000 7f95737fe640 1 -- 172.21.15.33:0/418614212 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f95740cf2e0 con 0x7f957c1033f0 2024-03-19T02:53:16.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.136+0000 7f9582028640 1 --2- 172.21.15.33:0/418614212 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f95680a2880 0x7f95680a4d40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f957c102890 tx=0x7f9578005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:16.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.140+0000 7f95737fe640 1 -- 172.21.15.33:0/418614212 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f957409bff0 con 0x7f957c1033f0 2024-03-19T02:53:16.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:15 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2008444438' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:53:16.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:15 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/297733093' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:53:16.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:15 smithi096 ceph-mon[35946]: from='client.15504 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:53:16.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:15 smithi096 ceph-mon[35946]: pgmap v375: 129 pgs: 129 active+clean; 648 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-03-19T02:53:16.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:15 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2008444438' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:53:16.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:15 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/297733093' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:53:16.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:15 smithi155 ceph-mon[35804]: from='client.15504 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:53:16.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:15 smithi155 ceph-mon[35804]: pgmap v375: 129 pgs: 129 active+clean; 648 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-03-19T02:53:16.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.322+0000 7f958428c640 1 -- 172.21.15.33:0/418614212 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f957c061c60 con 0x7f95680a2880 2024-03-19T02:53:16.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.331+0000 7f95737fe640 1 -- 172.21.15.33:0/418614212 <== mgr.14150 v2:172.21.15.33:6800/1453067701 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340504 (secure 0 0 0) 0x7f957c061c60 con 0x7f95680a2880 2024-03-19T02:53:16.336 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:53:16.337 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-03-19T02:53:16.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.334+0000 7f958428c640 1 -- 172.21.15.33:0/418614212 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f95680a2880 msgr2=0x7f95680a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:16.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.334+0000 7f958428c640 1 --2- 172.21.15.33:0/418614212 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f95680a2880 0x7f95680a4d40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f957c102890 tx=0x7f9578005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:16.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.334+0000 7f958428c640 1 -- 172.21.15.33:0/418614212 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f957c1033f0 msgr2=0x7f957c19f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:16.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.334+0000 7f958428c640 1 --2- 172.21.15.33:0/418614212 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f957c1033f0 0x7f957c19f040 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f957400eb10 tx=0x7f957400c2d0 comp rx=0 tx=0).stop 2024-03-19T02:53:16.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.334+0000 7f958428c640 1 -- 172.21.15.33:0/418614212 shutdown_connections 2024-03-19T02:53:16.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.335+0000 7f958428c640 1 --2- 172.21.15.33:0/418614212 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f95680a2880 0x7f95680a4d40 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:16.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.335+0000 7f958428c640 1 --2- 172.21.15.33:0/418614212 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f957c102a30 0x7f957c197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:16.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.335+0000 7f958428c640 1 --2- 172.21.15.33:0/418614212 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f957c1033f0 0x7f957c19f040 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:16.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.335+0000 7f958428c640 1 --2- 172.21.15.33:0/418614212 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f957c101840 0x7f957c197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:16.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.335+0000 7f958428c640 1 -- 172.21.15.33:0/418614212 >> 172.21.15.33:0/418614212 conn(0x7f957c0fcdd0 msgr2=0x7f957c105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:16.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.335+0000 7f958428c640 1 -- 172.21.15.33:0/418614212 shutdown_connections 2024-03-19T02:53:16.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.335+0000 7f958428c640 1 -- 172.21.15.33:0/418614212 wait complete. 2024-03-19T02:53:16.347 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":375,"stamp":"2024-03-19T02:53:15.742366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":663610,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":864,"num_read_kb":559,"num_write":359,"num_write_kb":3071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1181547,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":791,"ondisk_log_size":791,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":380900,"kb_used_data":15068,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124560924,"statfs":{"total":1151940427776,"available":1151550386176,"internally_reserved":0,"allocated":15429632,"data_stored":8513960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5793,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":94,"num_read_kb":53,"num_write":11,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002895"},"pg_stats":[{"pgid":"3.3f","version":"95'8","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.126776+0000","last_change":"2024-03-19T02:50:52.228368+0000","last_active":"2024-03-19T02:52:41.126776+0000","last_peered":"2024-03-19T02:52:41.126776+0000","last_clean":"2024-03-19T02:52:41.126776+0000","last_became_active":"2024-03-19T02:50:52.228069+0000","last_became_peered":"2024-03-19T02:50:52.228069+0000","last_unstale":"2024-03-19T02:52:41.126776+0000","last_undegraded":"2024-03-19T02:52:41.126776+0000","last_fullsized":"2024-03-19T02:52:41.126776+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:59:34.209095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"95'4","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.230176+0000","last_change":"2024-03-19T02:50:52.231358+0000","last_active":"2024-03-19T02:52:41.230176+0000","last_peered":"2024-03-19T02:52:41.230176+0000","last_clean":"2024-03-19T02:52:41.230176+0000","last_became_active":"2024-03-19T02:50:52.231029+0000","last_became_peered":"2024-03-19T02:50:52.231029+0000","last_unstale":"2024-03-19T02:52:41.230176+0000","last_undegraded":"2024-03-19T02:52:41.230176+0000","last_fullsized":"2024-03-19T02:52:41.230176+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:42:26.031392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"95'7","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.230300+0000","last_change":"2024-03-19T02:50:52.232685+0000","last_active":"2024-03-19T02:52:41.230300+0000","last_peered":"2024-03-19T02:52:41.230300+0000","last_clean":"2024-03-19T02:52:41.230300+0000","last_became_active":"2024-03-19T02:50:52.232361+0000","last_became_peered":"2024-03-19T02:50:52.232361+0000","last_unstale":"2024-03-19T02:52:41.230300+0000","last_undegraded":"2024-03-19T02:52:41.230300+0000","last_fullsized":"2024-03-19T02:52:41.230300+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:26:53.136200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"95'8","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124441+0000","last_change":"2024-03-19T02:50:52.227648+0000","last_active":"2024-03-19T02:52:41.124441+0000","last_peered":"2024-03-19T02:52:41.124441+0000","last_clean":"2024-03-19T02:52:41.124441+0000","last_became_active":"2024-03-19T02:50:52.226943+0000","last_became_peered":"2024-03-19T02:50:52.226943+0000","last_unstale":"2024-03-19T02:52:41.124441+0000","last_undegraded":"2024-03-19T02:52:41.124441+0000","last_fullsized":"2024-03-19T02:52:41.124441+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:03.293794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"95'4","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.134179+0000","last_change":"2024-03-19T02:50:52.225329+0000","last_active":"2024-03-19T02:52:41.134179+0000","last_peered":"2024-03-19T02:52:41.134179+0000","last_clean":"2024-03-19T02:52:41.134179+0000","last_became_active":"2024-03-19T02:50:52.225012+0000","last_became_peered":"2024-03-19T02:50:52.225012+0000","last_unstale":"2024-03-19T02:52:41.134179+0000","last_undegraded":"2024-03-19T02:52:41.134179+0000","last_fullsized":"2024-03-19T02:52:41.134179+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:32:52.877704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"95'6","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.172669+0000","last_change":"2024-03-19T02:50:52.217399+0000","last_active":"2024-03-19T02:52:41.172669+0000","last_peered":"2024-03-19T02:52:41.172669+0000","last_clean":"2024-03-19T02:52:41.172669+0000","last_became_active":"2024-03-19T02:50:52.216987+0000","last_became_peered":"2024-03-19T02:50:52.216987+0000","last_unstale":"2024-03-19T02:52:41.172669+0000","last_undegraded":"2024-03-19T02:52:41.172669+0000","last_fullsized":"2024-03-19T02:52:41.172669+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:57:15.772546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"95'5","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497233+0000","last_change":"2024-03-19T02:50:52.226755+0000","last_active":"2024-03-19T02:52:41.497233+0000","last_peered":"2024-03-19T02:52:41.497233+0000","last_clean":"2024-03-19T02:52:41.497233+0000","last_became_active":"2024-03-19T02:50:52.226434+0000","last_became_peered":"2024-03-19T02:50:52.226434+0000","last_unstale":"2024-03-19T02:52:41.497233+0000","last_undegraded":"2024-03-19T02:52:41.497233+0000","last_fullsized":"2024-03-19T02:52:41.497233+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:49:42.134262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"95'9","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228846+0000","last_change":"2024-03-19T02:50:52.215848+0000","last_active":"2024-03-19T02:52:41.228846+0000","last_peered":"2024-03-19T02:52:41.228846+0000","last_clean":"2024-03-19T02:52:41.228846+0000","last_became_active":"2024-03-19T02:50:52.215494+0000","last_became_peered":"2024-03-19T02:50:52.215494+0000","last_unstale":"2024-03-19T02:52:41.228846+0000","last_undegraded":"2024-03-19T02:52:41.228846+0000","last_fullsized":"2024-03-19T02:52:41.228846+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:55:10.182359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"95'11","reported_seq":43,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228832+0000","last_change":"2024-03-19T02:50:52.233237+0000","last_active":"2024-03-19T02:52:41.228832+0000","last_peered":"2024-03-19T02:52:41.228832+0000","last_clean":"2024-03-19T02:52:41.228832+0000","last_became_active":"2024-03-19T02:50:52.232091+0000","last_became_peered":"2024-03-19T02:50:52.232091+0000","last_unstale":"2024-03-19T02:52:41.228832+0000","last_undegraded":"2024-03-19T02:52:41.228832+0000","last_fullsized":"2024-03-19T02:52:41.228832+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:08:20.361576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"95'16","reported_seq":55,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.172997+0000","last_change":"2024-03-19T02:51:40.722463+0000","last_active":"2024-03-19T02:52:41.172997+0000","last_peered":"2024-03-19T02:52:41.172997+0000","last_clean":"2024-03-19T02:52:41.172997+0000","last_became_active":"2024-03-19T02:51:40.722120+0000","last_became_peered":"2024-03-19T02:51:40.722120+0000","last_unstale":"2024-03-19T02:52:41.172997+0000","last_undegraded":"2024-03-19T02:52:41.172997+0000","last_fullsized":"2024-03-19T02:52:41.172997+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:49:21.387064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"95'6","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497798+0000","last_change":"2024-03-19T02:50:52.226314+0000","last_active":"2024-03-19T02:52:41.497798+0000","last_peered":"2024-03-19T02:52:41.497798+0000","last_clean":"2024-03-19T02:52:41.497798+0000","last_became_active":"2024-03-19T02:50:52.225919+0000","last_became_peered":"2024-03-19T02:50:52.225919+0000","last_unstale":"2024-03-19T02:52:41.497798+0000","last_undegraded":"2024-03-19T02:52:41.497798+0000","last_fullsized":"2024-03-19T02:52:41.497798+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:42:51.782592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"95'7","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124495+0000","last_change":"2024-03-19T02:50:52.227929+0000","last_active":"2024-03-19T02:52:41.124495+0000","last_peered":"2024-03-19T02:52:41.124495+0000","last_clean":"2024-03-19T02:52:41.124495+0000","last_became_active":"2024-03-19T02:50:52.227360+0000","last_became_peered":"2024-03-19T02:50:52.227360+0000","last_unstale":"2024-03-19T02:52:41.124495+0000","last_undegraded":"2024-03-19T02:52:41.124495+0000","last_fullsized":"2024-03-19T02:52:41.124495+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:24:44.386101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"95'10","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.185886+0000","last_change":"2024-03-19T02:50:52.228088+0000","last_active":"2024-03-19T02:52:41.185886+0000","last_peered":"2024-03-19T02:52:41.185886+0000","last_clean":"2024-03-19T02:52:41.185886+0000","last_became_active":"2024-03-19T02:50:52.227754+0000","last_became_peered":"2024-03-19T02:50:52.227754+0000","last_unstale":"2024-03-19T02:52:41.185886+0000","last_undegraded":"2024-03-19T02:52:41.185886+0000","last_fullsized":"2024-03-19T02:52:41.185886+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:58:57.404241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"95'7","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.230047+0000","last_change":"2024-03-19T02:50:52.226768+0000","last_active":"2024-03-19T02:52:41.230047+0000","last_peered":"2024-03-19T02:52:41.230047+0000","last_clean":"2024-03-19T02:52:41.230047+0000","last_became_active":"2024-03-19T02:50:52.226452+0000","last_became_peered":"2024-03-19T02:50:52.226452+0000","last_unstale":"2024-03-19T02:52:41.230047+0000","last_undegraded":"2024-03-19T02:52:41.230047+0000","last_fullsized":"2024-03-19T02:52:41.230047+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:23:33.250773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"95'7","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.229921+0000","last_change":"2024-03-19T02:50:52.231132+0000","last_active":"2024-03-19T02:52:41.229921+0000","last_peered":"2024-03-19T02:52:41.229921+0000","last_clean":"2024-03-19T02:52:41.229921+0000","last_became_active":"2024-03-19T02:50:52.230778+0000","last_became_peered":"2024-03-19T02:50:52.230778+0000","last_unstale":"2024-03-19T02:52:41.229921+0000","last_undegraded":"2024-03-19T02:52:41.229921+0000","last_fullsized":"2024-03-19T02:52:41.229921+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:33:03.675210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"95'9","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228334+0000","last_change":"2024-03-19T02:50:52.228302+0000","last_active":"2024-03-19T02:52:41.228334+0000","last_peered":"2024-03-19T02:52:41.228334+0000","last_clean":"2024-03-19T02:52:41.228334+0000","last_became_active":"2024-03-19T02:50:52.228007+0000","last_became_peered":"2024-03-19T02:50:52.228007+0000","last_unstale":"2024-03-19T02:52:41.228334+0000","last_undegraded":"2024-03-19T02:52:41.228334+0000","last_fullsized":"2024-03-19T02:52:41.228334+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:59:28.988504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"95'8","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.126552+0000","last_change":"2024-03-19T02:50:52.228640+0000","last_active":"2024-03-19T02:52:41.126552+0000","last_peered":"2024-03-19T02:52:41.126552+0000","last_clean":"2024-03-19T02:52:41.126552+0000","last_became_active":"2024-03-19T02:50:52.228319+0000","last_became_peered":"2024-03-19T02:50:52.228319+0000","last_unstale":"2024-03-19T02:52:41.126552+0000","last_undegraded":"2024-03-19T02:52:41.126552+0000","last_fullsized":"2024-03-19T02:52:41.126552+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:30.356453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"95'10","reported_seq":44,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123572+0000","last_change":"2024-03-19T02:50:52.222300+0000","last_active":"2024-03-19T02:52:41.123572+0000","last_peered":"2024-03-19T02:52:41.123572+0000","last_clean":"2024-03-19T02:52:41.123572+0000","last_became_active":"2024-03-19T02:50:52.221978+0000","last_became_peered":"2024-03-19T02:50:52.221978+0000","last_unstale":"2024-03-19T02:52:41.123572+0000","last_undegraded":"2024-03-19T02:52:41.123572+0000","last_fullsized":"2024-03-19T02:52:41.123572+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:38:36.576987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"95'7","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124386+0000","last_change":"2024-03-19T02:50:52.217640+0000","last_active":"2024-03-19T02:52:41.124386+0000","last_peered":"2024-03-19T02:52:41.124386+0000","last_clean":"2024-03-19T02:52:41.124386+0000","last_became_active":"2024-03-19T02:50:52.217325+0000","last_became_peered":"2024-03-19T02:50:52.217325+0000","last_unstale":"2024-03-19T02:52:41.124386+0000","last_undegraded":"2024-03-19T02:52:41.124386+0000","last_fullsized":"2024-03-19T02:52:41.124386+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:40:25.894843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123733+0000","last_change":"2024-03-19T02:50:57.245323+0000","last_active":"2024-03-19T02:52:41.123733+0000","last_peered":"2024-03-19T02:52:41.123733+0000","last_clean":"2024-03-19T02:52:41.123733+0000","last_became_active":"2024-03-19T02:50:57.244886+0000","last_became_peered":"2024-03-19T02:50:57.244886+0000","last_unstale":"2024-03-19T02:52:41.123733+0000","last_undegraded":"2024-03-19T02:52:41.123733+0000","last_fullsized":"2024-03-19T02:52:41.123733+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:42:09.465738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"93'9","reported_seq":46,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123728+0000","last_change":"2024-03-19T02:50:51.797160+0000","last_active":"2024-03-19T02:52:41.123728+0000","last_peered":"2024-03-19T02:52:41.123728+0000","last_clean":"2024-03-19T02:52:41.123728+0000","last_became_active":"2024-03-19T02:50:51.796783+0000","last_became_peered":"2024-03-19T02:50:51.796783+0000","last_unstale":"2024-03-19T02:52:41.123728+0000","last_undegraded":"2024-03-19T02:52:41.123728+0000","last_fullsized":"2024-03-19T02:52:41.123728+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:57:08.185379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"95'5","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.229622+0000","last_change":"2024-03-19T02:50:52.225114+0000","last_active":"2024-03-19T02:52:41.229622+0000","last_peered":"2024-03-19T02:52:41.229622+0000","last_clean":"2024-03-19T02:52:41.229622+0000","last_became_active":"2024-03-19T02:50:52.220529+0000","last_became_peered":"2024-03-19T02:50:52.220529+0000","last_unstale":"2024-03-19T02:52:41.229622+0000","last_undegraded":"2024-03-19T02:52:41.229622+0000","last_fullsized":"2024-03-19T02:52:41.229622+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:12:51.408689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124051+0000","last_change":"2024-03-19T02:50:57.252827+0000","last_active":"2024-03-19T02:52:41.124051+0000","last_peered":"2024-03-19T02:52:41.124051+0000","last_clean":"2024-03-19T02:52:41.124051+0000","last_became_active":"2024-03-19T02:50:57.252502+0000","last_became_peered":"2024-03-19T02:50:57.252502+0000","last_unstale":"2024-03-19T02:52:41.124051+0000","last_undegraded":"2024-03-19T02:52:41.124051+0000","last_fullsized":"2024-03-19T02:52:41.124051+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:29:36.563533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"93'2","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.229689+0000","last_change":"2024-03-19T02:50:51.797380+0000","last_active":"2024-03-19T02:52:41.229689+0000","last_peered":"2024-03-19T02:52:41.229689+0000","last_clean":"2024-03-19T02:52:41.229689+0000","last_became_active":"2024-03-19T02:50:51.797021+0000","last_became_peered":"2024-03-19T02:50:51.797021+0000","last_unstale":"2024-03-19T02:52:41.229689+0000","last_undegraded":"2024-03-19T02:52:41.229689+0000","last_fullsized":"2024-03-19T02:52:41.229689+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:25.400360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"95'6","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.230538+0000","last_change":"2024-03-19T02:50:52.232717+0000","last_active":"2024-03-19T02:52:41.230538+0000","last_peered":"2024-03-19T02:52:41.230538+0000","last_clean":"2024-03-19T02:52:41.230538+0000","last_became_active":"2024-03-19T02:50:52.232368+0000","last_became_peered":"2024-03-19T02:50:52.232368+0000","last_unstale":"2024-03-19T02:52:41.230538+0000","last_undegraded":"2024-03-19T02:52:41.230538+0000","last_fullsized":"2024-03-19T02:52:41.230538+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:55:38.027120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124226+0000","last_change":"2024-03-19T02:50:57.250840+0000","last_active":"2024-03-19T02:52:41.124226+0000","last_peered":"2024-03-19T02:52:41.124226+0000","last_clean":"2024-03-19T02:52:41.124226+0000","last_became_active":"2024-03-19T02:50:57.250333+0000","last_became_peered":"2024-03-19T02:50:57.250333+0000","last_unstale":"2024-03-19T02:52:41.124226+0000","last_undegraded":"2024-03-19T02:52:41.124226+0000","last_fullsized":"2024-03-19T02:52:41.124226+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:57:09.089939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"93'4","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.134546+0000","last_change":"2024-03-19T02:50:51.796889+0000","last_active":"2024-03-19T02:52:41.134546+0000","last_peered":"2024-03-19T02:52:41.134546+0000","last_clean":"2024-03-19T02:52:41.134546+0000","last_became_active":"2024-03-19T02:50:51.796570+0000","last_became_peered":"2024-03-19T02:50:51.796570+0000","last_unstale":"2024-03-19T02:52:41.134546+0000","last_undegraded":"2024-03-19T02:52:41.134546+0000","last_fullsized":"2024-03-19T02:52:41.134546+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:18:38.301905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"95'9","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228471+0000","last_change":"2024-03-19T02:50:52.224533+0000","last_active":"2024-03-19T02:52:41.228471+0000","last_peered":"2024-03-19T02:52:41.228471+0000","last_clean":"2024-03-19T02:52:41.228471+0000","last_became_active":"2024-03-19T02:50:52.224221+0000","last_became_peered":"2024-03-19T02:50:52.224221+0000","last_unstale":"2024-03-19T02:52:41.228471+0000","last_undegraded":"2024-03-19T02:52:41.228471+0000","last_fullsized":"2024-03-19T02:52:41.228471+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:36:43.335318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124145+0000","last_change":"2024-03-19T02:50:57.252949+0000","last_active":"2024-03-19T02:52:41.124145+0000","last_peered":"2024-03-19T02:52:41.124145+0000","last_clean":"2024-03-19T02:52:41.124145+0000","last_became_active":"2024-03-19T02:50:57.252563+0000","last_became_peered":"2024-03-19T02:50:57.252563+0000","last_unstale":"2024-03-19T02:52:41.124145+0000","last_undegraded":"2024-03-19T02:52:41.124145+0000","last_fullsized":"2024-03-19T02:52:41.124145+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:30:38.244158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.134456+0000","last_change":"2024-03-19T02:50:51.794594+0000","last_active":"2024-03-19T02:52:41.134456+0000","last_peered":"2024-03-19T02:52:41.134456+0000","last_clean":"2024-03-19T02:52:41.134456+0000","last_became_active":"2024-03-19T02:50:51.794197+0000","last_became_peered":"2024-03-19T02:50:51.794197+0000","last_unstale":"2024-03-19T02:52:41.134456+0000","last_undegraded":"2024-03-19T02:52:41.134456+0000","last_fullsized":"2024-03-19T02:52:41.134456+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:58:44.534106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"95'15","reported_seq":48,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.175238+0000","last_change":"2024-03-19T02:50:52.214167+0000","last_active":"2024-03-19T02:52:41.175238+0000","last_peered":"2024-03-19T02:52:41.175238+0000","last_clean":"2024-03-19T02:52:41.175238+0000","last_became_active":"2024-03-19T02:50:52.213873+0000","last_became_peered":"2024-03-19T02:50:52.213873+0000","last_unstale":"2024-03-19T02:52:41.175238+0000","last_undegraded":"2024-03-19T02:52:41.175238+0000","last_fullsized":"2024-03-19T02:52:41.175238+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:28:51.035477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124222+0000","last_change":"2024-03-19T02:50:57.245139+0000","last_active":"2024-03-19T02:52:41.124222+0000","last_peered":"2024-03-19T02:52:41.124222+0000","last_clean":"2024-03-19T02:52:41.124222+0000","last_became_active":"2024-03-19T02:50:57.244636+0000","last_became_peered":"2024-03-19T02:50:57.244636+0000","last_unstale":"2024-03-19T02:52:41.124222+0000","last_undegraded":"2024-03-19T02:52:41.124222+0000","last_fullsized":"2024-03-19T02:52:41.124222+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:44:29.660221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"97'6","reported_seq":161,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:53:14.207172+0000","last_change":"2024-03-19T02:50:51.795969+0000","last_active":"2024-03-19T02:53:14.207172+0000","last_peered":"2024-03-19T02:53:14.207172+0000","last_clean":"2024-03-19T02:53:14.207172+0000","last_became_active":"2024-03-19T02:50:51.795628+0000","last_became_peered":"2024-03-19T02:50:51.795628+0000","last_unstale":"2024-03-19T02:53:14.207172+0000","last_undegraded":"2024-03-19T02:53:14.207172+0000","last_fullsized":"2024-03-19T02:53:14.207172+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:14:48.463282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1115,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":120,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"95'7","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.134019+0000","last_change":"2024-03-19T02:50:52.223278+0000","last_active":"2024-03-19T02:52:41.134019+0000","last_peered":"2024-03-19T02:52:41.134019+0000","last_clean":"2024-03-19T02:52:41.134019+0000","last_became_active":"2024-03-19T02:50:52.222741+0000","last_became_peered":"2024-03-19T02:50:52.222741+0000","last_unstale":"2024-03-19T02:52:41.134019+0000","last_undegraded":"2024-03-19T02:52:41.134019+0000","last_fullsized":"2024-03-19T02:52:41.134019+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:32:09.841825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124151+0000","last_change":"2024-03-19T02:50:57.253130+0000","last_active":"2024-03-19T02:52:41.124151+0000","last_peered":"2024-03-19T02:52:41.124151+0000","last_clean":"2024-03-19T02:52:41.124151+0000","last_became_active":"2024-03-19T02:50:57.252598+0000","last_became_peered":"2024-03-19T02:50:57.252598+0000","last_unstale":"2024-03-19T02:52:41.124151+0000","last_undegraded":"2024-03-19T02:52:41.124151+0000","last_fullsized":"2024-03-19T02:52:41.124151+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:09:38.183430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"93'3","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.172613+0000","last_change":"2024-03-19T02:50:51.794075+0000","last_active":"2024-03-19T02:52:41.172613+0000","last_peered":"2024-03-19T02:52:41.172613+0000","last_clean":"2024-03-19T02:52:41.172613+0000","last_became_active":"2024-03-19T02:50:51.793707+0000","last_became_peered":"2024-03-19T02:50:51.793707+0000","last_unstale":"2024-03-19T02:52:41.172613+0000","last_undegraded":"2024-03-19T02:52:41.172613+0000","last_fullsized":"2024-03-19T02:52:41.172613+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:52:07.180162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"95'8","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.175113+0000","last_change":"2024-03-19T02:50:52.211112+0000","last_active":"2024-03-19T02:52:41.175113+0000","last_peered":"2024-03-19T02:52:41.175113+0000","last_clean":"2024-03-19T02:52:41.175113+0000","last_became_active":"2024-03-19T02:50:52.210790+0000","last_became_peered":"2024-03-19T02:50:52.210790+0000","last_unstale":"2024-03-19T02:52:41.175113+0000","last_undegraded":"2024-03-19T02:52:41.175113+0000","last_fullsized":"2024-03-19T02:52:41.175113+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:39:17.651585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.232069+0000","last_change":"2024-03-19T02:52:41.231645+0000","last_active":"2024-03-19T02:52:41.232069+0000","last_peered":"2024-03-19T02:52:41.232069+0000","last_clean":"2024-03-19T02:52:41.232069+0000","last_became_active":"2024-03-19T02:52:41.231088+0000","last_became_peered":"2024-03-19T02:52:41.231088+0000","last_unstale":"2024-03-19T02:52:41.232069+0000","last_undegraded":"2024-03-19T02:52:41.232069+0000","last_fullsized":"2024-03-19T02:52:41.232069+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:25:29.072005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"93'2","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.185647+0000","last_change":"2024-03-19T02:50:51.785347+0000","last_active":"2024-03-19T02:52:41.185647+0000","last_peered":"2024-03-19T02:52:41.185647+0000","last_clean":"2024-03-19T02:52:41.185647+0000","last_became_active":"2024-03-19T02:50:51.785004+0000","last_became_peered":"2024-03-19T02:50:51.785004+0000","last_unstale":"2024-03-19T02:52:41.185647+0000","last_undegraded":"2024-03-19T02:52:41.185647+0000","last_fullsized":"2024-03-19T02:52:41.185647+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:49:47.269503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"95'8","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228791+0000","last_change":"2024-03-19T02:50:52.221330+0000","last_active":"2024-03-19T02:52:41.228791+0000","last_peered":"2024-03-19T02:52:41.228791+0000","last_clean":"2024-03-19T02:52:41.228791+0000","last_became_active":"2024-03-19T02:50:52.219417+0000","last_became_peered":"2024-03-19T02:50:52.219417+0000","last_unstale":"2024-03-19T02:52:41.228791+0000","last_undegraded":"2024-03-19T02:52:41.228791+0000","last_fullsized":"2024-03-19T02:52:41.228791+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:28:37.856260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.229931+0000","last_change":"2024-03-19T02:50:57.258357+0000","last_active":"2024-03-19T02:52:41.229931+0000","last_peered":"2024-03-19T02:52:41.229931+0000","last_clean":"2024-03-19T02:52:41.229931+0000","last_became_active":"2024-03-19T02:50:57.257924+0000","last_became_peered":"2024-03-19T02:50:57.257924+0000","last_unstale":"2024-03-19T02:52:41.229931+0000","last_undegraded":"2024-03-19T02:52:41.229931+0000","last_fullsized":"2024-03-19T02:52:41.229931+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:46:05.340731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"93'1","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.174691+0000","last_change":"2024-03-19T02:50:51.796425+0000","last_active":"2024-03-19T02:52:41.174691+0000","last_peered":"2024-03-19T02:52:41.174691+0000","last_clean":"2024-03-19T02:52:41.174691+0000","last_became_active":"2024-03-19T02:50:51.796139+0000","last_became_peered":"2024-03-19T02:50:51.796139+0000","last_unstale":"2024-03-19T02:52:41.174691+0000","last_undegraded":"2024-03-19T02:52:41.174691+0000","last_fullsized":"2024-03-19T02:52:41.174691+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:12:17.123556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"95'11","reported_seq":44,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123343+0000","last_change":"2024-03-19T02:50:52.219375+0000","last_active":"2024-03-19T02:52:41.123343+0000","last_peered":"2024-03-19T02:52:41.123343+0000","last_clean":"2024-03-19T02:52:41.123343+0000","last_became_active":"2024-03-19T02:50:52.219070+0000","last_became_peered":"2024-03-19T02:50:52.219070+0000","last_unstale":"2024-03-19T02:52:41.123343+0000","last_undegraded":"2024-03-19T02:52:41.123343+0000","last_fullsized":"2024-03-19T02:52:41.123343+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:31:58.847700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228989+0000","last_change":"2024-03-19T02:50:57.246502+0000","last_active":"2024-03-19T02:52:41.228989+0000","last_peered":"2024-03-19T02:52:41.228989+0000","last_clean":"2024-03-19T02:52:41.228989+0000","last_became_active":"2024-03-19T02:50:57.246089+0000","last_became_peered":"2024-03-19T02:50:57.246089+0000","last_unstale":"2024-03-19T02:52:41.228989+0000","last_undegraded":"2024-03-19T02:52:41.228989+0000","last_fullsized":"2024-03-19T02:52:41.228989+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:26:00.162308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"97'4","reported_seq":155,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:53:13.205745+0000","last_change":"2024-03-19T02:50:51.796448+0000","last_active":"2024-03-19T02:53:13.205745+0000","last_peered":"2024-03-19T02:53:13.205745+0000","last_clean":"2024-03-19T02:53:13.205745+0000","last_became_active":"2024-03-19T02:50:51.796132+0000","last_became_peered":"2024-03-19T02:50:51.796132+0000","last_unstale":"2024-03-19T02:53:13.205745+0000","last_undegraded":"2024-03-19T02:53:13.205745+0000","last_fullsized":"2024-03-19T02:53:13.205745+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:37:39.442772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":116,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"95'10","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228710+0000","last_change":"2024-03-19T02:50:52.223022+0000","last_active":"2024-03-19T02:52:41.228710+0000","last_peered":"2024-03-19T02:52:41.228710+0000","last_clean":"2024-03-19T02:52:41.228710+0000","last_became_active":"2024-03-19T02:50:52.221655+0000","last_became_peered":"2024-03-19T02:50:52.221655+0000","last_unstale":"2024-03-19T02:52:41.228710+0000","last_undegraded":"2024-03-19T02:52:41.228710+0000","last_fullsized":"2024-03-19T02:52:41.228710+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:34:36.054088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123149+0000","last_change":"2024-03-19T02:50:57.244003+0000","last_active":"2024-03-19T02:52:41.123149+0000","last_peered":"2024-03-19T02:52:41.123149+0000","last_clean":"2024-03-19T02:52:41.123149+0000","last_became_active":"2024-03-19T02:50:57.243567+0000","last_became_peered":"2024-03-19T02:50:57.243567+0000","last_unstale":"2024-03-19T02:52:41.123149+0000","last_undegraded":"2024-03-19T02:52:41.123149+0000","last_fullsized":"2024-03-19T02:52:41.123149+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:58:52.400237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"93'2","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497747+0000","last_change":"2024-03-19T02:50:51.801259+0000","last_active":"2024-03-19T02:52:41.497747+0000","last_peered":"2024-03-19T02:52:41.497747+0000","last_clean":"2024-03-19T02:52:41.497747+0000","last_became_active":"2024-03-19T02:50:51.800868+0000","last_became_peered":"2024-03-19T02:50:51.800868+0000","last_unstale":"2024-03-19T02:52:41.497747+0000","last_undegraded":"2024-03-19T02:52:41.497747+0000","last_fullsized":"2024-03-19T02:52:41.497747+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:44:06.486687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"95'9","reported_seq":48,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.230496+0000","last_change":"2024-03-19T02:51:40.724476+0000","last_active":"2024-03-19T02:52:41.230496+0000","last_peered":"2024-03-19T02:52:41.230496+0000","last_clean":"2024-03-19T02:52:41.230496+0000","last_became_active":"2024-03-19T02:51:40.724081+0000","last_became_peered":"2024-03-19T02:51:40.724081+0000","last_unstale":"2024-03-19T02:52:41.230496+0000","last_undegraded":"2024-03-19T02:52:41.230496+0000","last_fullsized":"2024-03-19T02:52:41.230496+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:04:15.684753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124330+0000","last_change":"2024-03-19T02:50:57.238764+0000","last_active":"2024-03-19T02:52:41.124330+0000","last_peered":"2024-03-19T02:52:41.124330+0000","last_clean":"2024-03-19T02:52:41.124330+0000","last_became_active":"2024-03-19T02:50:57.238401+0000","last_became_peered":"2024-03-19T02:50:57.238401+0000","last_unstale":"2024-03-19T02:52:41.124330+0000","last_undegraded":"2024-03-19T02:52:41.124330+0000","last_fullsized":"2024-03-19T02:52:41.124330+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:20:46.131776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"93'4","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.175182+0000","last_change":"2024-03-19T02:50:51.797633+0000","last_active":"2024-03-19T02:52:41.175182+0000","last_peered":"2024-03-19T02:52:41.175182+0000","last_clean":"2024-03-19T02:52:41.175182+0000","last_became_active":"2024-03-19T02:50:51.797313+0000","last_became_peered":"2024-03-19T02:50:51.797313+0000","last_unstale":"2024-03-19T02:52:41.175182+0000","last_undegraded":"2024-03-19T02:52:41.175182+0000","last_fullsized":"2024-03-19T02:52:41.175182+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:43:04.659301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"95'11","reported_seq":44,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228621+0000","last_change":"2024-03-19T02:50:52.231486+0000","last_active":"2024-03-19T02:52:41.228621+0000","last_peered":"2024-03-19T02:52:41.228621+0000","last_clean":"2024-03-19T02:52:41.228621+0000","last_became_active":"2024-03-19T02:50:52.231172+0000","last_became_peered":"2024-03-19T02:50:52.231172+0000","last_unstale":"2024-03-19T02:52:41.228621+0000","last_undegraded":"2024-03-19T02:52:41.228621+0000","last_fullsized":"2024-03-19T02:52:41.228621+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:24:49.912633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123242+0000","last_change":"2024-03-19T02:50:57.247207+0000","last_active":"2024-03-19T02:52:41.123242+0000","last_peered":"2024-03-19T02:52:41.123242+0000","last_clean":"2024-03-19T02:52:41.123242+0000","last_became_active":"2024-03-19T02:50:57.246776+0000","last_became_peered":"2024-03-19T02:50:57.246776+0000","last_unstale":"2024-03-19T02:52:41.123242+0000","last_undegraded":"2024-03-19T02:52:41.123242+0000","last_fullsized":"2024-03-19T02:52:41.123242+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:02:24.118858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"97'39","reported_seq":318,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:53:14.207075+0000","last_change":"2024-03-19T02:50:51.792398+0000","last_active":"2024-03-19T02:53:14.207075+0000","last_peered":"2024-03-19T02:53:14.207075+0000","last_clean":"2024-03-19T02:53:14.207075+0000","last_became_active":"2024-03-19T02:50:51.792090+0000","last_became_peered":"2024-03-19T02:50:51.792090+0000","last_unstale":"2024-03-19T02:53:14.207075+0000","last_undegraded":"2024-03-19T02:53:14.207075+0000","last_fullsized":"2024-03-19T02:53:14.207075+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:02:58.175210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68412,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":243,"num_read_kb":189,"num_write":40,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"95'6","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.134354+0000","last_change":"2024-03-19T02:50:52.220755+0000","last_active":"2024-03-19T02:52:41.134354+0000","last_peered":"2024-03-19T02:52:41.134354+0000","last_clean":"2024-03-19T02:52:41.134354+0000","last_became_active":"2024-03-19T02:50:52.220453+0000","last_became_peered":"2024-03-19T02:50:52.220453+0000","last_unstale":"2024-03-19T02:52:41.134354+0000","last_undegraded":"2024-03-19T02:52:41.134354+0000","last_fullsized":"2024-03-19T02:52:41.134354+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:48:27.866863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.175441+0000","last_change":"2024-03-19T02:50:57.253843+0000","last_active":"2024-03-19T02:52:41.175441+0000","last_peered":"2024-03-19T02:52:41.175441+0000","last_clean":"2024-03-19T02:52:41.175441+0000","last_became_active":"2024-03-19T02:50:57.252633+0000","last_became_peered":"2024-03-19T02:50:57.252633+0000","last_unstale":"2024-03-19T02:52:41.175441+0000","last_undegraded":"2024-03-19T02:52:41.175441+0000","last_fullsized":"2024-03-19T02:52:41.175441+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:03:48.514219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"93'6","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124095+0000","last_change":"2024-03-19T02:50:51.795041+0000","last_active":"2024-03-19T02:52:41.124095+0000","last_peered":"2024-03-19T02:52:41.124095+0000","last_clean":"2024-03-19T02:52:41.124095+0000","last_became_active":"2024-03-19T02:50:51.793485+0000","last_became_peered":"2024-03-19T02:50:51.793485+0000","last_unstale":"2024-03-19T02:52:41.124095+0000","last_undegraded":"2024-03-19T02:52:41.124095+0000","last_fullsized":"2024-03-19T02:52:41.124095+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:11:50.136394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"95'9","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.185843+0000","last_change":"2024-03-19T02:50:52.227554+0000","last_active":"2024-03-19T02:52:41.185843+0000","last_peered":"2024-03-19T02:52:41.185843+0000","last_clean":"2024-03-19T02:52:41.185843+0000","last_became_active":"2024-03-19T02:50:52.227222+0000","last_became_peered":"2024-03-19T02:50:52.227222+0000","last_unstale":"2024-03-19T02:52:41.185843+0000","last_undegraded":"2024-03-19T02:52:41.185843+0000","last_fullsized":"2024-03-19T02:52:41.185843+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:56.323749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":23,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.125538+0000","last_change":"2024-03-19T02:50:57.257504+0000","last_active":"2024-03-19T02:52:41.125538+0000","last_peered":"2024-03-19T02:52:41.125538+0000","last_clean":"2024-03-19T02:52:41.125538+0000","last_became_active":"2024-03-19T02:50:57.257120+0000","last_became_peered":"2024-03-19T02:50:57.257120+0000","last_unstale":"2024-03-19T02:52:41.125538+0000","last_undegraded":"2024-03-19T02:52:41.125538+0000","last_fullsized":"2024-03-19T02:52:41.125538+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:19:10.471666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"93'3","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.174956+0000","last_change":"2024-03-19T02:50:51.798049+0000","last_active":"2024-03-19T02:52:41.174956+0000","last_peered":"2024-03-19T02:52:41.174956+0000","last_clean":"2024-03-19T02:52:41.174956+0000","last_became_active":"2024-03-19T02:50:51.797134+0000","last_became_peered":"2024-03-19T02:50:51.797134+0000","last_unstale":"2024-03-19T02:52:41.174956+0000","last_undegraded":"2024-03-19T02:52:41.174956+0000","last_fullsized":"2024-03-19T02:52:41.174956+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:03:46.984240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"95'15","reported_seq":48,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497542+0000","last_change":"2024-03-19T02:50:52.219535+0000","last_active":"2024-03-19T02:52:41.497542+0000","last_peered":"2024-03-19T02:52:41.497542+0000","last_clean":"2024-03-19T02:52:41.497542+0000","last_became_active":"2024-03-19T02:50:52.219226+0000","last_became_peered":"2024-03-19T02:50:52.219226+0000","last_unstale":"2024-03-19T02:52:41.497542+0000","last_undegraded":"2024-03-19T02:52:41.497542+0000","last_fullsized":"2024-03-19T02:52:41.497542+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:27:26.875533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123493+0000","last_change":"2024-03-19T02:50:57.252873+0000","last_active":"2024-03-19T02:52:41.123493+0000","last_peered":"2024-03-19T02:52:41.123493+0000","last_clean":"2024-03-19T02:52:41.123493+0000","last_became_active":"2024-03-19T02:50:57.249551+0000","last_became_peered":"2024-03-19T02:50:57.249551+0000","last_unstale":"2024-03-19T02:52:41.123493+0000","last_undegraded":"2024-03-19T02:52:41.123493+0000","last_fullsized":"2024-03-19T02:52:41.123493+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:33:59.750606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"97'4","reported_seq":159,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:53:15.226100+0000","last_change":"2024-03-19T02:50:51.796478+0000","last_active":"2024-03-19T02:53:15.226100+0000","last_peered":"2024-03-19T02:53:15.226100+0000","last_clean":"2024-03-19T02:53:15.226100+0000","last_became_active":"2024-03-19T02:50:51.796126+0000","last_became_peered":"2024-03-19T02:50:51.796126+0000","last_unstale":"2024-03-19T02:53:15.226100+0000","last_undegraded":"2024-03-19T02:53:15.226100+0000","last_fullsized":"2024-03-19T02:53:15.226100+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:13:42.690481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"95'6","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497479+0000","last_change":"2024-03-19T02:50:52.228151+0000","last_active":"2024-03-19T02:52:41.497479+0000","last_peered":"2024-03-19T02:52:41.497479+0000","last_clean":"2024-03-19T02:52:41.497479+0000","last_became_active":"2024-03-19T02:50:52.227858+0000","last_became_peered":"2024-03-19T02:50:52.227858+0000","last_unstale":"2024-03-19T02:52:41.497479+0000","last_undegraded":"2024-03-19T02:52:41.497479+0000","last_fullsized":"2024-03-19T02:52:41.497479+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:10:10.285606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.186019+0000","last_change":"2024-03-19T02:50:57.259269+0000","last_active":"2024-03-19T02:52:41.186019+0000","last_peered":"2024-03-19T02:52:41.186019+0000","last_clean":"2024-03-19T02:52:41.186019+0000","last_became_active":"2024-03-19T02:50:57.258778+0000","last_became_peered":"2024-03-19T02:50:57.258778+0000","last_unstale":"2024-03-19T02:52:41.186019+0000","last_undegraded":"2024-03-19T02:52:41.186019+0000","last_fullsized":"2024-03-19T02:52:41.186019+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:02:44.419512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"93'15","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.125166+0000","last_change":"2024-03-19T02:51:40.739522+0000","last_active":"2024-03-19T02:52:41.125166+0000","last_peered":"2024-03-19T02:52:41.125166+0000","last_clean":"2024-03-19T02:52:41.125166+0000","last_became_active":"2024-03-19T02:51:40.720675+0000","last_became_peered":"2024-03-19T02:51:40.720675+0000","last_unstale":"2024-03-19T02:52:41.125166+0000","last_undegraded":"2024-03-19T02:52:41.125166+0000","last_fullsized":"2024-03-19T02:52:41.125166+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:18:19.449850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"95'6","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.174816+0000","last_change":"2024-03-19T02:50:52.215997+0000","last_active":"2024-03-19T02:52:41.174816+0000","last_peered":"2024-03-19T02:52:41.174816+0000","last_clean":"2024-03-19T02:52:41.174816+0000","last_became_active":"2024-03-19T02:50:52.215398+0000","last_became_peered":"2024-03-19T02:50:52.215398+0000","last_unstale":"2024-03-19T02:52:41.174816+0000","last_undegraded":"2024-03-19T02:52:41.174816+0000","last_fullsized":"2024-03-19T02:52:41.174816+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:49:14.010736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.230993+0000","last_change":"2024-03-19T02:52:41.230604+0000","last_active":"2024-03-19T02:52:41.230993+0000","last_peered":"2024-03-19T02:52:41.230993+0000","last_clean":"2024-03-19T02:52:41.230993+0000","last_became_active":"2024-03-19T02:52:41.230292+0000","last_became_peered":"2024-03-19T02:52:41.230292+0000","last_unstale":"2024-03-19T02:52:41.230993+0000","last_undegraded":"2024-03-19T02:52:41.230993+0000","last_fullsized":"2024-03-19T02:52:41.230993+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:55:52.552974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,3,1],"acting":[8,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.f","version":"93'2","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.134663+0000","last_change":"2024-03-19T02:50:51.796246+0000","last_active":"2024-03-19T02:52:41.134663+0000","last_peered":"2024-03-19T02:52:41.134663+0000","last_clean":"2024-03-19T02:52:41.134663+0000","last_became_active":"2024-03-19T02:50:51.795823+0000","last_became_peered":"2024-03-19T02:50:51.795823+0000","last_unstale":"2024-03-19T02:52:41.134663+0000","last_undegraded":"2024-03-19T02:52:41.134663+0000","last_fullsized":"2024-03-19T02:52:41.134663+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:36:02.220891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"95'5","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.125358+0000","last_change":"2024-03-19T02:50:52.224119+0000","last_active":"2024-03-19T02:52:41.125358+0000","last_peered":"2024-03-19T02:52:41.125358+0000","last_clean":"2024-03-19T02:52:41.125358+0000","last_became_active":"2024-03-19T02:50:52.223773+0000","last_became_peered":"2024-03-19T02:50:52.223773+0000","last_unstale":"2024-03-19T02:52:41.125358+0000","last_undegraded":"2024-03-19T02:52:41.125358+0000","last_fullsized":"2024-03-19T02:52:41.125358+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:56:03.098858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.134063+0000","last_change":"2024-03-19T02:50:57.652515+0000","last_active":"2024-03-19T02:52:41.134063+0000","last_peered":"2024-03-19T02:52:41.134063+0000","last_clean":"2024-03-19T02:52:41.134063+0000","last_became_active":"2024-03-19T02:50:57.651808+0000","last_became_peered":"2024-03-19T02:50:57.651808+0000","last_unstale":"2024-03-19T02:52:41.134063+0000","last_undegraded":"2024-03-19T02:52:41.134063+0000","last_fullsized":"2024-03-19T02:52:41.134063+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:16:34.550457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123896+0000","last_change":"2024-03-19T02:50:51.794282+0000","last_active":"2024-03-19T02:52:41.123896+0000","last_peered":"2024-03-19T02:52:41.123896+0000","last_clean":"2024-03-19T02:52:41.123896+0000","last_became_active":"2024-03-19T02:50:51.793947+0000","last_became_peered":"2024-03-19T02:50:51.793947+0000","last_unstale":"2024-03-19T02:52:41.123896+0000","last_undegraded":"2024-03-19T02:52:41.123896+0000","last_fullsized":"2024-03-19T02:52:41.123896+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:08:54.764915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"95'4","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.173032+0000","last_change":"2024-03-19T02:50:52.221707+0000","last_active":"2024-03-19T02:52:41.173032+0000","last_peered":"2024-03-19T02:52:41.173032+0000","last_clean":"2024-03-19T02:52:41.173032+0000","last_became_active":"2024-03-19T02:50:52.221417+0000","last_became_peered":"2024-03-19T02:50:52.221417+0000","last_unstale":"2024-03-19T02:52:41.173032+0000","last_undegraded":"2024-03-19T02:52:41.173032+0000","last_fullsized":"2024-03-19T02:52:41.173032+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:59:27.528346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.185950+0000","last_change":"2024-03-19T02:50:57.257552+0000","last_active":"2024-03-19T02:52:41.185950+0000","last_peered":"2024-03-19T02:52:41.185950+0000","last_clean":"2024-03-19T02:52:41.185950+0000","last_became_active":"2024-03-19T02:50:57.257121+0000","last_became_peered":"2024-03-19T02:50:57.257121+0000","last_unstale":"2024-03-19T02:52:41.185950+0000","last_undegraded":"2024-03-19T02:52:41.185950+0000","last_fullsized":"2024-03-19T02:52:41.185950+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:43:47.145344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"93'1","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497345+0000","last_change":"2024-03-19T02:50:51.804890+0000","last_active":"2024-03-19T02:52:41.497345+0000","last_peered":"2024-03-19T02:52:41.497345+0000","last_clean":"2024-03-19T02:52:41.497345+0000","last_became_active":"2024-03-19T02:50:51.804545+0000","last_became_peered":"2024-03-19T02:50:51.804545+0000","last_unstale":"2024-03-19T02:52:41.497345+0000","last_undegraded":"2024-03-19T02:52:41.497345+0000","last_fullsized":"2024-03-19T02:52:41.497345+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:16:14.945372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"95'7","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.172788+0000","last_change":"2024-03-19T02:50:52.224849+0000","last_active":"2024-03-19T02:52:41.172788+0000","last_peered":"2024-03-19T02:52:41.172788+0000","last_clean":"2024-03-19T02:52:41.172788+0000","last_became_active":"2024-03-19T02:50:52.224550+0000","last_became_peered":"2024-03-19T02:50:52.224550+0000","last_unstale":"2024-03-19T02:52:41.172788+0000","last_undegraded":"2024-03-19T02:52:41.172788+0000","last_fullsized":"2024-03-19T02:52:41.172788+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:28:59.683849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.186106+0000","last_change":"2024-03-19T02:50:57.652425+0000","last_active":"2024-03-19T02:52:41.186106+0000","last_peered":"2024-03-19T02:52:41.186106+0000","last_clean":"2024-03-19T02:52:41.186106+0000","last_became_active":"2024-03-19T02:50:57.651975+0000","last_became_peered":"2024-03-19T02:50:57.651975+0000","last_unstale":"2024-03-19T02:52:41.186106+0000","last_undegraded":"2024-03-19T02:52:41.186106+0000","last_fullsized":"2024-03-19T02:52:41.186106+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:07:20.587918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"95'8","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.174802+0000","last_change":"2024-03-19T02:50:52.218207+0000","last_active":"2024-03-19T02:52:41.174802+0000","last_peered":"2024-03-19T02:52:41.174802+0000","last_clean":"2024-03-19T02:52:41.174802+0000","last_became_active":"2024-03-19T02:50:52.217012+0000","last_became_peered":"2024-03-19T02:50:52.217012+0000","last_unstale":"2024-03-19T02:52:41.174802+0000","last_undegraded":"2024-03-19T02:52:41.174802+0000","last_fullsized":"2024-03-19T02:52:41.174802+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:07:47.366481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"97'5","reported_seq":157,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:53:14.224642+0000","last_change":"2024-03-19T02:50:51.801405+0000","last_active":"2024-03-19T02:53:14.224642+0000","last_peered":"2024-03-19T02:53:14.224642+0000","last_clean":"2024-03-19T02:53:14.224642+0000","last_became_active":"2024-03-19T02:50:51.801094+0000","last_became_peered":"2024-03-19T02:50:51.801094+0000","last_unstale":"2024-03-19T02:53:14.224642+0000","last_undegraded":"2024-03-19T02:53:14.224642+0000","last_fullsized":"2024-03-19T02:53:14.224642+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:53:39.534977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":117,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"93'1","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497740+0000","last_change":"2024-03-19T02:51:40.730836+0000","last_active":"2024-03-19T02:52:41.497740+0000","last_peered":"2024-03-19T02:52:41.497740+0000","last_clean":"2024-03-19T02:52:41.497740+0000","last_became_active":"2024-03-19T02:51:40.723984+0000","last_became_peered":"2024-03-19T02:51:40.723984+0000","last_unstale":"2024-03-19T02:52:41.497740+0000","last_undegraded":"2024-03-19T02:52:41.497740+0000","last_fullsized":"2024-03-19T02:52:41.497740+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:29:57.301066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"95'8","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228900+0000","last_change":"2024-03-19T02:50:52.221852+0000","last_active":"2024-03-19T02:52:41.228900+0000","last_peered":"2024-03-19T02:52:41.228900+0000","last_clean":"2024-03-19T02:52:41.228900+0000","last_became_active":"2024-03-19T02:50:52.221535+0000","last_became_peered":"2024-03-19T02:50:52.221535+0000","last_unstale":"2024-03-19T02:52:41.228900+0000","last_undegraded":"2024-03-19T02:52:41.228900+0000","last_fullsized":"2024-03-19T02:52:41.228900+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:37:18.030714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"93'3","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124331+0000","last_change":"2024-03-19T02:50:51.790231+0000","last_active":"2024-03-19T02:52:41.124331+0000","last_peered":"2024-03-19T02:52:41.124331+0000","last_clean":"2024-03-19T02:52:41.124331+0000","last_became_active":"2024-03-19T02:50:51.789867+0000","last_became_peered":"2024-03-19T02:50:51.789867+0000","last_unstale":"2024-03-19T02:52:41.124331+0000","last_undegraded":"2024-03-19T02:52:41.124331+0000","last_fullsized":"2024-03-19T02:52:41.124331+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:46.099191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":370,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497476+0000","last_change":"2024-03-19T02:49:38.697673+0000","last_active":"2024-03-19T02:52:41.497476+0000","last_peered":"2024-03-19T02:52:41.497476+0000","last_clean":"2024-03-19T02:52:41.497476+0000","last_became_active":"2024-03-19T02:49:38.697161+0000","last_became_peered":"2024-03-19T02:49:38.697161+0000","last_unstale":"2024-03-19T02:52:41.497476+0000","last_undegraded":"2024-03-19T02:52:41.497476+0000","last_fullsized":"2024-03-19T02:52:41.497476+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:46:10.013962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:46:10.013962+0000","last_clean_scrub_stamp":"2024-03-19T02:46:10.013962+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:59:35.179938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.175278+0000","last_change":"2024-03-19T02:50:57.652428+0000","last_active":"2024-03-19T02:52:41.175278+0000","last_peered":"2024-03-19T02:52:41.175278+0000","last_clean":"2024-03-19T02:52:41.175278+0000","last_became_active":"2024-03-19T02:50:57.651864+0000","last_became_peered":"2024-03-19T02:50:57.651864+0000","last_unstale":"2024-03-19T02:52:41.175278+0000","last_undegraded":"2024-03-19T02:52:41.175278+0000","last_fullsized":"2024-03-19T02:52:41.175278+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:52:19.007175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"95'5","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.185550+0000","last_change":"2024-03-19T02:50:52.218728+0000","last_active":"2024-03-19T02:52:41.185550+0000","last_peered":"2024-03-19T02:52:41.185550+0000","last_clean":"2024-03-19T02:52:41.185550+0000","last_became_active":"2024-03-19T02:50:52.218407+0000","last_became_peered":"2024-03-19T02:50:52.218407+0000","last_unstale":"2024-03-19T02:52:41.185550+0000","last_undegraded":"2024-03-19T02:52:41.185550+0000","last_fullsized":"2024-03-19T02:52:41.185550+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:30:49.304034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"93'3","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497416+0000","last_change":"2024-03-19T02:50:51.802057+0000","last_active":"2024-03-19T02:52:41.497416+0000","last_peered":"2024-03-19T02:52:41.497416+0000","last_clean":"2024-03-19T02:52:41.497416+0000","last_became_active":"2024-03-19T02:50:51.801754+0000","last_became_peered":"2024-03-19T02:50:51.801754+0000","last_unstale":"2024-03-19T02:52:41.497416+0000","last_undegraded":"2024-03-19T02:52:41.497416+0000","last_fullsized":"2024-03-19T02:52:41.497416+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:23:15.401822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":24,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.125249+0000","last_change":"2024-03-19T02:50:57.260303+0000","last_active":"2024-03-19T02:52:41.125249+0000","last_peered":"2024-03-19T02:52:41.125249+0000","last_clean":"2024-03-19T02:52:41.125249+0000","last_became_active":"2024-03-19T02:50:57.259792+0000","last_became_peered":"2024-03-19T02:50:57.259792+0000","last_unstale":"2024-03-19T02:52:41.125249+0000","last_undegraded":"2024-03-19T02:52:41.125249+0000","last_fullsized":"2024-03-19T02:52:41.125249+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:03.072161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"93'1","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.134478+0000","last_change":"2024-03-19T02:50:51.797949+0000","last_active":"2024-03-19T02:52:41.134478+0000","last_peered":"2024-03-19T02:52:41.134478+0000","last_clean":"2024-03-19T02:52:41.134478+0000","last_became_active":"2024-03-19T02:50:51.797522+0000","last_became_peered":"2024-03-19T02:50:51.797522+0000","last_unstale":"2024-03-19T02:52:41.134478+0000","last_undegraded":"2024-03-19T02:52:41.134478+0000","last_fullsized":"2024-03-19T02:52:41.134478+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:34:06.127993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"95'9","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.186125+0000","last_change":"2024-03-19T02:50:52.224095+0000","last_active":"2024-03-19T02:52:41.186125+0000","last_peered":"2024-03-19T02:52:41.186125+0000","last_clean":"2024-03-19T02:52:41.186125+0000","last_became_active":"2024-03-19T02:50:52.223764+0000","last_became_peered":"2024-03-19T02:50:52.223764+0000","last_unstale":"2024-03-19T02:52:41.186125+0000","last_undegraded":"2024-03-19T02:52:41.186125+0000","last_fullsized":"2024-03-19T02:52:41.186125+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:07:41.503162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228732+0000","last_change":"2024-03-19T02:50:57.652274+0000","last_active":"2024-03-19T02:52:41.228732+0000","last_peered":"2024-03-19T02:52:41.228732+0000","last_clean":"2024-03-19T02:52:41.228732+0000","last_became_active":"2024-03-19T02:50:57.651885+0000","last_became_peered":"2024-03-19T02:50:57.651885+0000","last_unstale":"2024-03-19T02:52:41.228732+0000","last_undegraded":"2024-03-19T02:52:41.228732+0000","last_fullsized":"2024-03-19T02:52:41.228732+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:03:02.081143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"97'10","reported_seq":46,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:53:03.631782+0000","last_change":"2024-03-19T02:50:51.796308+0000","last_active":"2024-03-19T02:53:03.631782+0000","last_peered":"2024-03-19T02:53:03.631782+0000","last_clean":"2024-03-19T02:53:03.631782+0000","last_became_active":"2024-03-19T02:50:51.795990+0000","last_became_peered":"2024-03-19T02:50:51.795990+0000","last_unstale":"2024-03-19T02:53:03.631782+0000","last_undegraded":"2024-03-19T02:53:03.631782+0000","last_fullsized":"2024-03-19T02:53:03.631782+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:45:53.608817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"95'5","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.229132+0000","last_change":"2024-03-19T02:50:52.221983+0000","last_active":"2024-03-19T02:52:41.229132+0000","last_peered":"2024-03-19T02:52:41.229132+0000","last_clean":"2024-03-19T02:52:41.229132+0000","last_became_active":"2024-03-19T02:50:52.220605+0000","last_became_peered":"2024-03-19T02:50:52.220605+0000","last_unstale":"2024-03-19T02:52:41.229132+0000","last_undegraded":"2024-03-19T02:52:41.229132+0000","last_fullsized":"2024-03-19T02:52:41.229132+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:12.834159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123697+0000","last_change":"2024-03-19T02:50:57.242428+0000","last_active":"2024-03-19T02:52:41.123697+0000","last_peered":"2024-03-19T02:52:41.123697+0000","last_clean":"2024-03-19T02:52:41.123697+0000","last_became_active":"2024-03-19T02:50:57.242007+0000","last_became_peered":"2024-03-19T02:50:57.242007+0000","last_unstale":"2024-03-19T02:52:41.123697+0000","last_undegraded":"2024-03-19T02:52:41.123697+0000","last_fullsized":"2024-03-19T02:52:41.123697+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:23:30.987761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"93'4","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124922+0000","last_change":"2024-03-19T02:50:51.791342+0000","last_active":"2024-03-19T02:52:41.124922+0000","last_peered":"2024-03-19T02:52:41.124922+0000","last_clean":"2024-03-19T02:52:41.124922+0000","last_became_active":"2024-03-19T02:50:51.791039+0000","last_became_peered":"2024-03-19T02:50:51.791039+0000","last_unstale":"2024-03-19T02:52:41.124922+0000","last_undegraded":"2024-03-19T02:52:41.124922+0000","last_fullsized":"2024-03-19T02:52:41.124922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:50:38.216623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"95'7","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.230040+0000","last_change":"2024-03-19T02:50:52.224303+0000","last_active":"2024-03-19T02:52:41.230040+0000","last_peered":"2024-03-19T02:52:41.230040+0000","last_clean":"2024-03-19T02:52:41.230040+0000","last_became_active":"2024-03-19T02:50:52.223942+0000","last_became_peered":"2024-03-19T02:50:52.223942+0000","last_unstale":"2024-03-19T02:52:41.230040+0000","last_undegraded":"2024-03-19T02:52:41.230040+0000","last_fullsized":"2024-03-19T02:52:41.230040+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:57:04.175971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124685+0000","last_change":"2024-03-19T02:50:57.652126+0000","last_active":"2024-03-19T02:52:41.124685+0000","last_peered":"2024-03-19T02:52:41.124685+0000","last_clean":"2024-03-19T02:52:41.124685+0000","last_became_active":"2024-03-19T02:50:57.651713+0000","last_became_peered":"2024-03-19T02:50:57.651713+0000","last_unstale":"2024-03-19T02:52:41.124685+0000","last_undegraded":"2024-03-19T02:52:41.124685+0000","last_fullsized":"2024-03-19T02:52:41.124685+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:28:51.077437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"93'2","reported_seq":47,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228284+0000","last_change":"2024-03-19T02:51:40.741378+0000","last_active":"2024-03-19T02:52:41.228284+0000","last_peered":"2024-03-19T02:52:41.228284+0000","last_clean":"2024-03-19T02:52:41.228284+0000","last_became_active":"2024-03-19T02:51:40.724510+0000","last_became_peered":"2024-03-19T02:51:40.724510+0000","last_unstale":"2024-03-19T02:52:41.228284+0000","last_undegraded":"2024-03-19T02:52:41.228284+0000","last_fullsized":"2024-03-19T02:52:41.228284+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:30:16.228635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"95'2","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.133963+0000","last_change":"2024-03-19T02:50:52.221681+0000","last_active":"2024-03-19T02:52:41.133963+0000","last_peered":"2024-03-19T02:52:41.133963+0000","last_clean":"2024-03-19T02:52:41.133963+0000","last_became_active":"2024-03-19T02:50:52.220987+0000","last_became_peered":"2024-03-19T02:50:52.220987+0000","last_unstale":"2024-03-19T02:52:41.133963+0000","last_undegraded":"2024-03-19T02:52:41.133963+0000","last_fullsized":"2024-03-19T02:52:41.133963+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:16:36.561143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123630+0000","last_change":"2024-03-19T02:50:57.249830+0000","last_active":"2024-03-19T02:52:41.123630+0000","last_peered":"2024-03-19T02:52:41.123630+0000","last_clean":"2024-03-19T02:52:41.123630+0000","last_became_active":"2024-03-19T02:50:57.248879+0000","last_became_peered":"2024-03-19T02:50:57.248879+0000","last_unstale":"2024-03-19T02:52:41.123630+0000","last_undegraded":"2024-03-19T02:52:41.123630+0000","last_fullsized":"2024-03-19T02:52:41.123630+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:58:43.225841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"93'3","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.229197+0000","last_change":"2024-03-19T02:50:51.801374+0000","last_active":"2024-03-19T02:52:41.229197+0000","last_peered":"2024-03-19T02:52:41.229197+0000","last_clean":"2024-03-19T02:52:41.229197+0000","last_became_active":"2024-03-19T02:50:51.801041+0000","last_became_peered":"2024-03-19T02:50:51.801041+0000","last_unstale":"2024-03-19T02:52:41.229197+0000","last_undegraded":"2024-03-19T02:52:41.229197+0000","last_fullsized":"2024-03-19T02:52:41.229197+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:56:14.671718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"95'9","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.125323+0000","last_change":"2024-03-19T02:50:52.225915+0000","last_active":"2024-03-19T02:52:41.125323+0000","last_peered":"2024-03-19T02:52:41.125323+0000","last_clean":"2024-03-19T02:52:41.125323+0000","last_became_active":"2024-03-19T02:50:52.225606+0000","last_became_peered":"2024-03-19T02:50:52.225606+0000","last_unstale":"2024-03-19T02:52:41.125323+0000","last_undegraded":"2024-03-19T02:52:41.125323+0000","last_fullsized":"2024-03-19T02:52:41.125323+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:27:29.549255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"93'1","reported_seq":27,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124644+0000","last_change":"2024-03-19T02:50:57.240747+0000","last_active":"2024-03-19T02:52:41.124644+0000","last_peered":"2024-03-19T02:52:41.124644+0000","last_clean":"2024-03-19T02:52:41.124644+0000","last_became_active":"2024-03-19T02:50:57.240373+0000","last_became_peered":"2024-03-19T02:50:57.240373+0000","last_unstale":"2024-03-19T02:52:41.124644+0000","last_undegraded":"2024-03-19T02:52:41.124644+0000","last_fullsized":"2024-03-19T02:52:41.124644+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:22:04.991497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"97'2","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:53:03.898093+0000","last_change":"2024-03-19T02:50:51.794071+0000","last_active":"2024-03-19T02:53:03.898093+0000","last_peered":"2024-03-19T02:53:03.898093+0000","last_clean":"2024-03-19T02:53:03.898093+0000","last_became_active":"2024-03-19T02:50:51.793570+0000","last_became_peered":"2024-03-19T02:50:51.793570+0000","last_unstale":"2024-03-19T02:53:03.898093+0000","last_undegraded":"2024-03-19T02:53:03.898093+0000","last_fullsized":"2024-03-19T02:53:03.898093+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:34:20.456515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"95'8","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.229465+0000","last_change":"2024-03-19T02:50:52.223341+0000","last_active":"2024-03-19T02:52:41.229465+0000","last_peered":"2024-03-19T02:52:41.229465+0000","last_clean":"2024-03-19T02:52:41.229465+0000","last_became_active":"2024-03-19T02:50:52.220280+0000","last_became_peered":"2024-03-19T02:50:52.220280+0000","last_unstale":"2024-03-19T02:52:41.229465+0000","last_undegraded":"2024-03-19T02:52:41.229465+0000","last_fullsized":"2024-03-19T02:52:41.229465+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:04:15.830811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124197+0000","last_change":"2024-03-19T02:50:57.651293+0000","last_active":"2024-03-19T02:52:41.124197+0000","last_peered":"2024-03-19T02:52:41.124197+0000","last_clean":"2024-03-19T02:52:41.124197+0000","last_became_active":"2024-03-19T02:50:57.650893+0000","last_became_peered":"2024-03-19T02:50:57.650893+0000","last_unstale":"2024-03-19T02:52:41.124197+0000","last_undegraded":"2024-03-19T02:52:41.124197+0000","last_fullsized":"2024-03-19T02:52:41.124197+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:33:35.984728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"93'4","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.134128+0000","last_change":"2024-03-19T02:50:51.798297+0000","last_active":"2024-03-19T02:52:41.134128+0000","last_peered":"2024-03-19T02:52:41.134128+0000","last_clean":"2024-03-19T02:52:41.134128+0000","last_became_active":"2024-03-19T02:50:51.797516+0000","last_became_peered":"2024-03-19T02:50:51.797516+0000","last_unstale":"2024-03-19T02:52:41.134128+0000","last_undegraded":"2024-03-19T02:52:41.134128+0000","last_fullsized":"2024-03-19T02:52:41.134128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:09:42.295094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"95'5","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497375+0000","last_change":"2024-03-19T02:50:52.224011+0000","last_active":"2024-03-19T02:52:41.497375+0000","last_peered":"2024-03-19T02:52:41.497375+0000","last_clean":"2024-03-19T02:52:41.497375+0000","last_became_active":"2024-03-19T02:50:52.223708+0000","last_became_peered":"2024-03-19T02:50:52.223708+0000","last_unstale":"2024-03-19T02:52:41.497375+0000","last_undegraded":"2024-03-19T02:52:41.497375+0000","last_fullsized":"2024-03-19T02:52:41.497375+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:17:49.305587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228248+0000","last_change":"2024-03-19T02:50:57.251134+0000","last_active":"2024-03-19T02:52:41.228248+0000","last_peered":"2024-03-19T02:52:41.228248+0000","last_clean":"2024-03-19T02:52:41.228248+0000","last_became_active":"2024-03-19T02:50:57.250761+0000","last_became_peered":"2024-03-19T02:50:57.250761+0000","last_unstale":"2024-03-19T02:52:41.228248+0000","last_undegraded":"2024-03-19T02:52:41.228248+0000","last_fullsized":"2024-03-19T02:52:41.228248+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:09:26.860235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"93'3","reported_seq":21,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124386+0000","last_change":"2024-03-19T02:51:40.728077+0000","last_active":"2024-03-19T02:52:41.124386+0000","last_peered":"2024-03-19T02:52:41.124386+0000","last_clean":"2024-03-19T02:52:41.124386+0000","last_became_active":"2024-03-19T02:51:40.720767+0000","last_became_peered":"2024-03-19T02:51:40.720767+0000","last_unstale":"2024-03-19T02:52:41.124386+0000","last_undegraded":"2024-03-19T02:52:41.124386+0000","last_fullsized":"2024-03-19T02:52:41.124386+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:22:34.366373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"95'7","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497892+0000","last_change":"2024-03-19T02:50:52.224776+0000","last_active":"2024-03-19T02:52:41.497892+0000","last_peered":"2024-03-19T02:52:41.497892+0000","last_clean":"2024-03-19T02:52:41.497892+0000","last_became_active":"2024-03-19T02:50:52.224442+0000","last_became_peered":"2024-03-19T02:50:52.224442+0000","last_unstale":"2024-03-19T02:52:41.497892+0000","last_undegraded":"2024-03-19T02:52:41.497892+0000","last_fullsized":"2024-03-19T02:52:41.497892+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:11:13.722005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.230221+0000","last_change":"2024-03-19T02:51:40.722732+0000","last_active":"2024-03-19T02:52:41.230221+0000","last_peered":"2024-03-19T02:52:41.230221+0000","last_clean":"2024-03-19T02:52:41.230221+0000","last_became_active":"2024-03-19T02:51:40.722260+0000","last_became_peered":"2024-03-19T02:51:40.722260+0000","last_unstale":"2024-03-19T02:52:41.230221+0000","last_undegraded":"2024-03-19T02:52:41.230221+0000","last_fullsized":"2024-03-19T02:52:41.230221+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:31:06.383570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.175385+0000","last_change":"2024-03-19T02:50:51.794898+0000","last_active":"2024-03-19T02:52:41.175385+0000","last_peered":"2024-03-19T02:52:41.175385+0000","last_clean":"2024-03-19T02:52:41.175385+0000","last_became_active":"2024-03-19T02:50:51.794562+0000","last_became_peered":"2024-03-19T02:50:51.794562+0000","last_unstale":"2024-03-19T02:52:41.175385+0000","last_undegraded":"2024-03-19T02:52:41.175385+0000","last_fullsized":"2024-03-19T02:52:41.175385+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:17:45.239395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"95'6","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497460+0000","last_change":"2024-03-19T02:50:52.227427+0000","last_active":"2024-03-19T02:52:41.497460+0000","last_peered":"2024-03-19T02:52:41.497460+0000","last_clean":"2024-03-19T02:52:41.497460+0000","last_became_active":"2024-03-19T02:50:52.227111+0000","last_became_peered":"2024-03-19T02:50:52.227111+0000","last_unstale":"2024-03-19T02:52:41.497460+0000","last_undegraded":"2024-03-19T02:52:41.497460+0000","last_fullsized":"2024-03-19T02:52:41.497460+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:40:53.295454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":26,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.174601+0000","last_change":"2024-03-19T02:50:57.244449+0000","last_active":"2024-03-19T02:52:41.174601+0000","last_peered":"2024-03-19T02:52:41.174601+0000","last_clean":"2024-03-19T02:52:41.174601+0000","last_became_active":"2024-03-19T02:50:57.243991+0000","last_became_peered":"2024-03-19T02:50:57.243991+0000","last_unstale":"2024-03-19T02:52:41.174601+0000","last_undegraded":"2024-03-19T02:52:41.174601+0000","last_fullsized":"2024-03-19T02:52:41.174601+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:56.221427+0000","last_clean_scrub_stamp":"2024-03-19T02:50:56.221427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:50:45.265773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.186101+0000","last_change":"2024-03-19T02:50:51.799834+0000","last_active":"2024-03-19T02:52:41.186101+0000","last_peered":"2024-03-19T02:52:41.186101+0000","last_clean":"2024-03-19T02:52:41.186101+0000","last_became_active":"2024-03-19T02:50:51.799563+0000","last_became_peered":"2024-03-19T02:50:51.799563+0000","last_unstale":"2024-03-19T02:52:41.186101+0000","last_undegraded":"2024-03-19T02:52:41.186101+0000","last_fullsized":"2024-03-19T02:52:41.186101+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:50.769676+0000","last_clean_scrub_stamp":"2024-03-19T02:50:50.769676+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:51:00.284454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"95'9","reported_seq":43,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.497913+0000","last_change":"2024-03-19T02:50:52.222710+0000","last_active":"2024-03-19T02:52:41.497913+0000","last_peered":"2024-03-19T02:52:41.497913+0000","last_clean":"2024-03-19T02:52:41.497913+0000","last_became_active":"2024-03-19T02:50:52.222321+0000","last_became_peered":"2024-03-19T02:50:52.222321+0000","last_unstale":"2024-03-19T02:52:41.497913+0000","last_undegraded":"2024-03-19T02:52:41.497913+0000","last_fullsized":"2024-03-19T02:52:41.497913+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:05:27.527696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"95'9","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.125423+0000","last_change":"2024-03-19T02:50:52.226631+0000","last_active":"2024-03-19T02:52:41.125423+0000","last_peered":"2024-03-19T02:52:41.125423+0000","last_clean":"2024-03-19T02:52:41.125423+0000","last_became_active":"2024-03-19T02:50:52.226294+0000","last_became_peered":"2024-03-19T02:50:52.226294+0000","last_unstale":"2024-03-19T02:52:41.125423+0000","last_undegraded":"2024-03-19T02:52:41.125423+0000","last_fullsized":"2024-03-19T02:52:41.125423+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:01:31.350063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"95'12","reported_seq":45,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123985+0000","last_change":"2024-03-19T02:50:52.221138+0000","last_active":"2024-03-19T02:52:41.123985+0000","last_peered":"2024-03-19T02:52:41.123985+0000","last_clean":"2024-03-19T02:52:41.123985+0000","last_became_active":"2024-03-19T02:50:52.220730+0000","last_became_peered":"2024-03-19T02:50:52.220730+0000","last_unstale":"2024-03-19T02:52:41.123985+0000","last_undegraded":"2024-03-19T02:52:41.123985+0000","last_fullsized":"2024-03-19T02:52:41.123985+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:26:13.059986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"95'8","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123763+0000","last_change":"2024-03-19T02:50:52.223567+0000","last_active":"2024-03-19T02:52:41.123763+0000","last_peered":"2024-03-19T02:52:41.123763+0000","last_clean":"2024-03-19T02:52:41.123763+0000","last_became_active":"2024-03-19T02:50:52.223174+0000","last_became_peered":"2024-03-19T02:50:52.223174+0000","last_unstale":"2024-03-19T02:52:41.123763+0000","last_undegraded":"2024-03-19T02:52:41.123763+0000","last_fullsized":"2024-03-19T02:52:41.123763+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:23:06.383584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"95'7","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.230567+0000","last_change":"2024-03-19T02:50:52.229916+0000","last_active":"2024-03-19T02:52:41.230567+0000","last_peered":"2024-03-19T02:52:41.230567+0000","last_clean":"2024-03-19T02:52:41.230567+0000","last_became_active":"2024-03-19T02:50:52.229597+0000","last_became_peered":"2024-03-19T02:50:52.229597+0000","last_unstale":"2024-03-19T02:52:41.230567+0000","last_undegraded":"2024-03-19T02:52:41.230567+0000","last_fullsized":"2024-03-19T02:52:41.230567+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:39:19.331893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"95'4","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.186073+0000","last_change":"2024-03-19T02:50:52.223331+0000","last_active":"2024-03-19T02:52:41.186073+0000","last_peered":"2024-03-19T02:52:41.186073+0000","last_clean":"2024-03-19T02:52:41.186073+0000","last_became_active":"2024-03-19T02:50:52.222975+0000","last_became_peered":"2024-03-19T02:50:52.222975+0000","last_unstale":"2024-03-19T02:52:41.186073+0000","last_undegraded":"2024-03-19T02:52:41.186073+0000","last_fullsized":"2024-03-19T02:52:41.186073+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:26:23.023749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"95'6","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.125760+0000","last_change":"2024-03-19T02:50:52.218073+0000","last_active":"2024-03-19T02:52:41.125760+0000","last_peered":"2024-03-19T02:52:41.125760+0000","last_clean":"2024-03-19T02:52:41.125760+0000","last_became_active":"2024-03-19T02:50:52.217742+0000","last_became_peered":"2024-03-19T02:50:52.217742+0000","last_unstale":"2024-03-19T02:52:41.125760+0000","last_undegraded":"2024-03-19T02:52:41.125760+0000","last_fullsized":"2024-03-19T02:52:41.125760+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:08:51.053153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"95'15","reported_seq":48,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.124597+0000","last_change":"2024-03-19T02:50:52.222108+0000","last_active":"2024-03-19T02:52:41.124597+0000","last_peered":"2024-03-19T02:52:41.124597+0000","last_clean":"2024-03-19T02:52:41.124597+0000","last_became_active":"2024-03-19T02:50:52.221748+0000","last_became_peered":"2024-03-19T02:50:52.221748+0000","last_unstale":"2024-03-19T02:52:41.124597+0000","last_undegraded":"2024-03-19T02:52:41.124597+0000","last_fullsized":"2024-03-19T02:52:41.124597+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:33:31.401939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"95'2","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.229311+0000","last_change":"2024-03-19T02:50:52.221799+0000","last_active":"2024-03-19T02:52:41.229311+0000","last_peered":"2024-03-19T02:52:41.229311+0000","last_clean":"2024-03-19T02:52:41.229311+0000","last_became_active":"2024-03-19T02:50:52.220196+0000","last_became_peered":"2024-03-19T02:50:52.220196+0000","last_unstale":"2024-03-19T02:52:41.229311+0000","last_undegraded":"2024-03-19T02:52:41.229311+0000","last_fullsized":"2024-03-19T02:52:41.229311+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:24:01.541716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"95'6","reported_seq":21,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.185475+0000","last_change":"2024-03-19T02:51:40.721422+0000","last_active":"2024-03-19T02:52:41.185475+0000","last_peered":"2024-03-19T02:52:41.185475+0000","last_clean":"2024-03-19T02:52:41.185475+0000","last_became_active":"2024-03-19T02:51:40.720982+0000","last_became_peered":"2024-03-19T02:51:40.720982+0000","last_unstale":"2024-03-19T02:52:41.185475+0000","last_undegraded":"2024-03-19T02:52:41.185475+0000","last_fullsized":"2024-03-19T02:52:41.185475+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:07.391810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,3],"acting":[6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.29","version":"95'8","reported_seq":23,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.123111+0000","last_change":"2024-03-19T02:51:40.721188+0000","last_active":"2024-03-19T02:52:41.123111+0000","last_peered":"2024-03-19T02:52:41.123111+0000","last_clean":"2024-03-19T02:52:41.123111+0000","last_became_active":"2024-03-19T02:51:40.720811+0000","last_became_peered":"2024-03-19T02:51:40.720811+0000","last_unstale":"2024-03-19T02:52:41.123111+0000","last_undegraded":"2024-03-19T02:52:41.123111+0000","last_fullsized":"2024-03-19T02:52:41.123111+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:28:23.970149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2a","version":"95'10","reported_seq":43,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.228360+0000","last_change":"2024-03-19T02:50:52.230847+0000","last_active":"2024-03-19T02:52:41.228360+0000","last_peered":"2024-03-19T02:52:41.228360+0000","last_clean":"2024-03-19T02:52:41.228360+0000","last_became_active":"2024-03-19T02:50:52.230478+0000","last_became_peered":"2024-03-19T02:50:52.230478+0000","last_unstale":"2024-03-19T02:52:41.228360+0000","last_undegraded":"2024-03-19T02:52:41.228360+0000","last_fullsized":"2024-03-19T02:52:41.228360+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:04:09.459446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"95'11","reported_seq":44,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.186018+0000","last_change":"2024-03-19T02:50:52.223508+0000","last_active":"2024-03-19T02:52:41.186018+0000","last_peered":"2024-03-19T02:52:41.186018+0000","last_clean":"2024-03-19T02:52:41.186018+0000","last_became_active":"2024-03-19T02:50:52.223173+0000","last_became_peered":"2024-03-19T02:50:52.223173+0000","last_unstale":"2024-03-19T02:52:41.186018+0000","last_undegraded":"2024-03-19T02:52:41.186018+0000","last_fullsized":"2024-03-19T02:52:41.186018+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:53.320173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"95'8","reported_seq":47,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-19T02:52:41.125738+0000","last_change":"2024-03-19T02:51:40.723512+0000","last_active":"2024-03-19T02:52:41.125738+0000","last_peered":"2024-03-19T02:52:41.125738+0000","last_clean":"2024-03-19T02:52:41.125738+0000","last_became_active":"2024-03-19T02:51:40.723149+0000","last_became_peered":"2024-03-19T02:51:40.723149+0000","last_unstale":"2024-03-19T02:52:41.125738+0000","last_undegraded":"2024-03-19T02:52:41.125738+0000","last_fullsized":"2024-03-19T02:52:41.125738+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T02:50:51.771494+0000","last_clean_scrub_stamp":"2024-03-19T02:50:51.771494+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:42:38.862569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":73008,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":728,"num_read_kb":440,"num_write":176,"num_write_kb":131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":694,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":219024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152,"ondisk_log_size":152,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318314,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36432,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708720,"statfs":{"total":95995035648,"available":95957729280,"internally_reserved":0,"allocated":1114112,"data_stored":545712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579951,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28780,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716372,"statfs":{"total":95995035648,"available":95965564928,"internally_reserved":0,"allocated":1806336,"data_stored":1205869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743477,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36432,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708720,"statfs":{"total":95995035648,"available":95957729280,"internally_reserved":0,"allocated":1122304,"data_stored":545836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005113,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36432,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708720,"statfs":{"total":95995035648,"available":95957729280,"internally_reserved":0,"allocated":1114112,"data_stored":545644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266751,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37012,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708140,"statfs":{"total":95995035648,"available":95957135360,"internally_reserved":0,"allocated":1716224,"data_stored":1142260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462979,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1101824,"data_stored":541012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36484,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708668,"statfs":{"total":95995035648,"available":95957676032,"internally_reserved":0,"allocated":1175552,"data_stored":609334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1126400,"data_stored":542400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28184,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716968,"statfs":{"total":95995035648,"available":95966175232,"internally_reserved":0,"allocated":1196032,"data_stored":610595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149780,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1114112,"data_stored":541206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888143,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1118208,"data_stored":546737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724618,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28700,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716452,"statfs":{"total":95995035648,"available":95965646848,"internally_reserved":0,"allocated":1724416,"data_stored":1137355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":68412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":69673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":70483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T02:53:16.352 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:53:16.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.489+0000 7f7cd5a19640 1 -- 172.21.15.33:0/2084644527 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7cd0107580 msgr2=0x7f7cd0109970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:16.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.489+0000 7f7cd5a19640 1 --2- 172.21.15.33:0/2084644527 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7cd0107580 0x7f7cd0109970 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8009ca0 tx=0x7f7cc802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:53:16.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.489+0000 7f7cd5a19640 1 -- 172.21.15.33:0/2084644527 shutdown_connections 2024-03-19T02:53:16.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.489+0000 7f7cd5a19640 1 --2- 172.21.15.33:0/2084644527 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7cd0107580 0x7f7cd0109970 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:16.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.489+0000 7f7cd5a19640 1 --2- 172.21.15.33:0/2084644527 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7cd0104c50 0x7f7cd0107040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:16.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.489+0000 7f7cd5a19640 1 --2- 172.21.15.33:0/2084644527 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7cd0102320 0x7f7cd0104710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:16.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.489+0000 7f7cd5a19640 1 -- 172.21.15.33:0/2084644527 >> 172.21.15.33:0/2084644527 conn(0x7f7cd00fbcb0 msgr2=0x7f7cd00fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:16.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.489+0000 7f7cd5a19640 1 -- 172.21.15.33:0/2084644527 shutdown_connections 2024-03-19T02:53:16.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.489+0000 7f7cd5a19640 1 -- 172.21.15.33:0/2084644527 wait complete. 2024-03-19T02:53:16.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.490+0000 7f7cd5a19640 1 Processor -- start 2024-03-19T02:53:16.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.490+0000 7f7cd5a19640 1 -- start start 2024-03-19T02:53:16.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.490+0000 7f7cd5a19640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7cd0102320 0x7f7cd0197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:16.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.490+0000 7f7cd5a19640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7cd0104c50 0x7f7cd0197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:16.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.490+0000 7f7cd5a19640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7cd0107580 0x7f7cd019f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:16.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.490+0000 7f7cd5a19640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd010d390 con 0x7f7cd0104c50 2024-03-19T02:53:16.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.490+0000 7f7cd5a19640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd010d210 con 0x7f7cd0107580 2024-03-19T02:53:16.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.490+0000 7f7cd5a19640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd010d510 con 0x7f7cd0102320 2024-03-19T02:53:16.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.490+0000 7f7cc7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7cd0104c50 0x7f7cd0197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:16.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.490+0000 7f7cc7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7cd0104c50 0x7f7cd0197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46826/0 (socket says 172.21.15.33:46826) 2024-03-19T02:53:16.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.490+0000 7f7cc7fff640 1 -- 172.21.15.33:0/374282996 learned_addr learned my addr 172.21.15.33:0/374282996 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:16.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.490+0000 7f7cceffd640 1 --2- 172.21.15.33:0/374282996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7cd0102320 0x7f7cd0197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:16.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.490+0000 7f7ccf7fe640 1 --2- 172.21.15.33:0/374282996 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7cd0107580 0x7f7cd019f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:16.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.490+0000 7f7cc7fff640 1 -- 172.21.15.33:0/374282996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7cd0102320 msgr2=0x7f7cd0197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:16.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.490+0000 7f7cc7fff640 1 --2- 172.21.15.33:0/374282996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7cd0102320 0x7f7cd0197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:16.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.490+0000 7f7cc7fff640 1 -- 172.21.15.33:0/374282996 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7cd0107580 msgr2=0x7f7cd019f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:16.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.490+0000 7f7cc7fff640 1 --2- 172.21.15.33:0/374282996 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7cd0107580 0x7f7cd019f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:16.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.490+0000 7f7cc7fff640 1 -- 172.21.15.33:0/374282996 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cd019f670 con 0x7f7cd0104c50 2024-03-19T02:53:16.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.490+0000 7f7cc7fff640 1 --2- 172.21.15.33:0/374282996 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7cd0104c50 0x7f7cd0197fe0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f7cb800e9a0 tx=0x7f7cb800ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:16.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.491+0000 7f7cd4a17640 1 -- 172.21.15.33:0/374282996 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb800cd80 con 0x7f7cd0104c50 2024-03-19T02:53:16.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.491+0000 7f7cd4a17640 1 -- 172.21.15.33:0/374282996 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7cb80035f0 con 0x7f7cd0104c50 2024-03-19T02:53:16.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.491+0000 7f7cd5a19640 1 -- 172.21.15.33:0/374282996 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cd019f960 con 0x7f7cd0104c50 2024-03-19T02:53:16.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.491+0000 7f7cd4a17640 1 -- 172.21.15.33:0/374282996 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb8010690 con 0x7f7cd0104c50 2024-03-19T02:53:16.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.491+0000 7f7cd5a19640 1 -- 172.21.15.33:0/374282996 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cd019ff00 con 0x7f7cd0104c50 2024-03-19T02:53:16.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.492+0000 7f7cd5a19640 1 -- 172.21.15.33:0/374282996 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cd00fd7a0 con 0x7f7cd0104c50 2024-03-19T02:53:16.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.493+0000 7f7cd4a17640 1 -- 172.21.15.33:0/374282996 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7cb80037a0 con 0x7f7cd0104c50 2024-03-19T02:53:16.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.493+0000 7f7cd4a17640 1 --2- 172.21.15.33:0/374282996 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7cbc0a2880 0x7f7cbc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:16.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.493+0000 7f7cd4a17640 1 -- 172.21.15.33:0/374282996 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f7cb80cefb0 con 0x7f7cd0104c50 2024-03-19T02:53:16.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.493+0000 7f7cceffd640 1 --2- 172.21.15.33:0/374282996 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7cbc0a2880 0x7f7cbc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:16.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.494+0000 7f7cceffd640 1 --2- 172.21.15.33:0/374282996 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7cbc0a2880 0x7f7cbc0a4d40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0009bd0 tx=0x7f7cc0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:16.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.498+0000 7f7cd4a17640 1 -- 172.21.15.33:0/374282996 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7cb809bcc0 con 0x7f7cd0104c50 2024-03-19T02:53:16.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.683+0000 7f7cd5a19640 1 -- 172.21.15.33:0/374282996 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7cd0062420 con 0x7f7cd0104c50 2024-03-19T02:53:16.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.685+0000 7f7cd4a17640 1 -- 172.21.15.33:0/374282996 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) ==== 74+0+24663 (secure 0 0 0) 0x7f7cb809b5c0 con 0x7f7cd0104c50 2024-03-19T02:53:16.686 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:53:16.686 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":97,"fsid":"2b7a8938-e59a-11ee-95c9-87774f69a715","created":"2024-03-19T02:42:37.912179+0000","modified":"2024-03-19T02:52:41.116227+0000","last_up_change":"2024-03-19T02:50:01.539392+0000","last_in_change":"2024-03-19T02:49:40.534415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:46:09.963737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:50:50.069641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:50:51.381064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:50:55.476667+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"090a89ab-6b7b-4317-8823-178d4212b2f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6803","nonce":1633124995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6805","nonce":1633124995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6809","nonce":1633124995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6807","nonce":1633124995}]},"public_addr":"172.21.15.33:6803/1633124995","cluster_addr":"172.21.15.33:6805/1633124995","heartbeat_back_addr":"172.21.15.33:6809/1633124995","heartbeat_front_addr":"172.21.15.33:6807/1633124995","state":["exists","up"]},{"osd":1,"uuid":"f50eec55-1956-4486-8593-f2c8fa610e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6801","nonce":340144398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6803","nonce":340144398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6807","nonce":340144398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6805","nonce":340144398}]},"public_addr":"172.21.15.96:6801/340144398","cluster_addr":"172.21.15.96:6803/340144398","heartbeat_back_addr":"172.21.15.96:6807/340144398","heartbeat_front_addr":"172.21.15.96:6805/340144398","state":["exists","up"]},{"osd":2,"uuid":"893a2d62-80d9-4e5e-bc6a-a1c8c681d652","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6801","nonce":3388743876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6803","nonce":3388743876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6807","nonce":3388743876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6805","nonce":3388743876}]},"public_addr":"172.21.15.155:6801/3388743876","cluster_addr":"172.21.15.155:6803/3388743876","heartbeat_back_addr":"172.21.15.155:6807/3388743876","heartbeat_front_addr":"172.21.15.155:6805/3388743876","state":["exists","up"]},{"osd":3,"uuid":"b07e0482-4281-4d32-9dfe-9995d7246175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6811","nonce":677007171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6813","nonce":677007171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6817","nonce":677007171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6815","nonce":677007171}]},"public_addr":"172.21.15.33:6811/677007171","cluster_addr":"172.21.15.33:6813/677007171","heartbeat_back_addr":"172.21.15.33:6817/677007171","heartbeat_front_addr":"172.21.15.33:6815/677007171","state":["exists","up"]},{"osd":4,"uuid":"a988384b-af6b-47c6-8c07-5c9673acee45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6809","nonce":3106158554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6811","nonce":3106158554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6815","nonce":3106158554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6813","nonce":3106158554}]},"public_addr":"172.21.15.96:6809/3106158554","cluster_addr":"172.21.15.96:6811/3106158554","heartbeat_back_addr":"172.21.15.96:6815/3106158554","heartbeat_front_addr":"172.21.15.96:6813/3106158554","state":["exists","up"]},{"osd":5,"uuid":"198dc992-96f8-4c52-96ee-1fd5de148bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6809","nonce":327293324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6811","nonce":327293324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6815","nonce":327293324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6813","nonce":327293324}]},"public_addr":"172.21.15.155:6809/327293324","cluster_addr":"172.21.15.155:6811/327293324","heartbeat_back_addr":"172.21.15.155:6815/327293324","heartbeat_front_addr":"172.21.15.155:6813/327293324","state":["exists","up"]},{"osd":6,"uuid":"a7278ec0-ab2e-49ed-be53-84d8b9e24177","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6819","nonce":873871509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6821","nonce":873871509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6825","nonce":873871509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6823","nonce":873871509}]},"public_addr":"172.21.15.33:6819/873871509","cluster_addr":"172.21.15.33:6821/873871509","heartbeat_back_addr":"172.21.15.33:6825/873871509","heartbeat_front_addr":"172.21.15.33:6823/873871509","state":["exists","up"]},{"osd":7,"uuid":"727e133d-aa24-4ffc-90e2-266eb2aab8c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6817","nonce":2334123891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6819","nonce":2334123891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6823","nonce":2334123891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6821","nonce":2334123891}]},"public_addr":"172.21.15.96:6817/2334123891","cluster_addr":"172.21.15.96:6819/2334123891","heartbeat_back_addr":"172.21.15.96:6823/2334123891","heartbeat_front_addr":"172.21.15.96:6821/2334123891","state":["exists","up"]},{"osd":8,"uuid":"bb20eec0-39e5-4b86-9db5-486b1840cdad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6817","nonce":3498516903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6819","nonce":3498516903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6823","nonce":3498516903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6821","nonce":3498516903}]},"public_addr":"172.21.15.155:6817/3498516903","cluster_addr":"172.21.15.155:6819/3498516903","heartbeat_back_addr":"172.21.15.155:6823/3498516903","heartbeat_front_addr":"172.21.15.155:6821/3498516903","state":["exists","up"]},{"osd":9,"uuid":"179c6228-3e23-4f65-91a6-1ef0005d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6827","nonce":760777213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6829","nonce":760777213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6833","nonce":760777213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6831","nonce":760777213}]},"public_addr":"172.21.15.33:6827/760777213","cluster_addr":"172.21.15.33:6829/760777213","heartbeat_back_addr":"172.21.15.33:6833/760777213","heartbeat_front_addr":"172.21.15.33:6831/760777213","state":["exists","up"]},{"osd":10,"uuid":"be17b478-3c2e-41fb-bede-958333acdf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6825","nonce":1404898482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6827","nonce":1404898482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6831","nonce":1404898482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6829","nonce":1404898482}]},"public_addr":"172.21.15.96:6825/1404898482","cluster_addr":"172.21.15.96:6827/1404898482","heartbeat_back_addr":"172.21.15.96:6831/1404898482","heartbeat_front_addr":"172.21.15.96:6829/1404898482","state":["exists","up"]},{"osd":11,"uuid":"c0903ed2-cc1e-4c18-8731-1f0ee4dd0457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6825","nonce":792352827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6827","nonce":792352827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6831","nonce":792352827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6829","nonce":792352827}]},"public_addr":"172.21.15.155:6825/792352827","cluster_addr":"172.21.15.155:6827/792352827","heartbeat_back_addr":"172.21.15.155:6831/792352827","heartbeat_front_addr":"172.21.15.155:6829/792352827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:17.942801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:42.109449+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:05.981599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:32.069574+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:57.431015+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:22.448592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:48.526515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:14.498352+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:40.175426+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:06.364006+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:32.853161+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:59.073538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.9","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:6801/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:6800/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/1067768047":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/729379453":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2586588742":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/2932814165":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6801/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2212564963":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6800/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/3275126058":"2024-03-20T02:43:39.622218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:53:16.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.687+0000 7f7cd5a19640 1 -- 172.21.15.33:0/374282996 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7cbc0a2880 msgr2=0x7f7cbc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:16.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.687+0000 7f7cd5a19640 1 --2- 172.21.15.33:0/374282996 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7cbc0a2880 0x7f7cbc0a4d40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0009bd0 tx=0x7f7cc0005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:16.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.687+0000 7f7cd5a19640 1 -- 172.21.15.33:0/374282996 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7cd0104c50 msgr2=0x7f7cd0197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:16.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.687+0000 7f7cd5a19640 1 --2- 172.21.15.33:0/374282996 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7cd0104c50 0x7f7cd0197fe0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f7cb800e9a0 tx=0x7f7cb800ee60 comp rx=0 tx=0).stop 2024-03-19T02:53:16.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.687+0000 7f7cd5a19640 1 -- 172.21.15.33:0/374282996 shutdown_connections 2024-03-19T02:53:16.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.687+0000 7f7cd5a19640 1 --2- 172.21.15.33:0/374282996 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f7cbc0a2880 0x7f7cbc0a4d40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:16.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.687+0000 7f7cd5a19640 1 --2- 172.21.15.33:0/374282996 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7cd0107580 0x7f7cd019f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:16.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.687+0000 7f7cd5a19640 1 --2- 172.21.15.33:0/374282996 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7cd0104c50 0x7f7cd0197fe0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:16.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.687+0000 7f7cd5a19640 1 --2- 172.21.15.33:0/374282996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7cd0102320 0x7f7cd0197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:16.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.687+0000 7f7cd5a19640 1 -- 172.21.15.33:0/374282996 >> 172.21.15.33:0/374282996 conn(0x7f7cd00fbcb0 msgr2=0x7f7cd0102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:16.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.687+0000 7f7cd5a19640 1 -- 172.21.15.33:0/374282996 shutdown_connections 2024-03-19T02:53:16.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.687+0000 7f7cd5a19640 1 -- 172.21.15.33:0/374282996 wait complete. 2024-03-19T02:53:16.699 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:46:09.963737+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T02:50:50.069641+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T02:50:51.381064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T02:50:55.476667+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T02:53:16.700 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T02:53:16.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.805+0000 7febcfecc640 1 -- 172.21.15.33:0/4116204016 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febc8100720 msgr2=0x7febc8100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:16.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.805+0000 7febcfecc640 1 --2- 172.21.15.33:0/4116204016 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febc8100720 0x7febc8100b40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7febc0009860 tx=0x7febc002bb70 comp rx=0 tx=0).stop 2024-03-19T02:53:16.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.806+0000 7febcfecc640 1 -- 172.21.15.33:0/4116204016 shutdown_connections 2024-03-19T02:53:16.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.806+0000 7febcfecc640 1 --2- 172.21.15.33:0/4116204016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febc8101910 0x7febc8101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:16.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.806+0000 7febcfecc640 1 --2- 172.21.15.33:0/4116204016 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7febc81022d0 0x7febc8108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:16.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.806+0000 7febcfecc640 1 --2- 172.21.15.33:0/4116204016 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febc8100720 0x7febc8100b40 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:16.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.806+0000 7febcfecc640 1 -- 172.21.15.33:0/4116204016 >> 172.21.15.33:0/4116204016 conn(0x7febc80fbcb0 msgr2=0x7febc80fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:16.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.806+0000 7febcfecc640 1 -- 172.21.15.33:0/4116204016 shutdown_connections 2024-03-19T02:53:16.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.806+0000 7febcfecc640 1 -- 172.21.15.33:0/4116204016 wait complete. 2024-03-19T02:53:16.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.806+0000 7febcfecc640 1 Processor -- start 2024-03-19T02:53:16.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.806+0000 7febcfecc640 1 -- start start 2024-03-19T02:53:16.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.806+0000 7febcfecc640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febc8100720 0x7febc8197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:16.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.806+0000 7febcfecc640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febc8101910 0x7febc8197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:16.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.806+0000 7febcfecc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7febc81022d0 0x7febc819f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:16.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.806+0000 7febcfecc640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7febc810c220 con 0x7febc81022d0 2024-03-19T02:53:16.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.806+0000 7febcfecc640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7febc810c0a0 con 0x7febc8100720 2024-03-19T02:53:16.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.806+0000 7febcfecc640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7febc810c3a0 con 0x7febc8101910 2024-03-19T02:53:16.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.806+0000 7febce469640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7febc81022d0 0x7febc819f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:16.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.807+0000 7febcdc68640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febc8100720 0x7febc8197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:16.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.807+0000 7febce469640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7febc81022d0 0x7febc819f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46842/0 (socket says 172.21.15.33:46842) 2024-03-19T02:53:16.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.807+0000 7febce469640 1 -- 172.21.15.33:0/1354836013 learned_addr learned my addr 172.21.15.33:0/1354836013 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:16.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.807+0000 7febcd467640 1 --2- 172.21.15.33:0/1354836013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febc8101910 0x7febc8197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:16.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.807+0000 7febce469640 1 -- 172.21.15.33:0/1354836013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febc8101910 msgr2=0x7febc8197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:16.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.807+0000 7febce469640 1 --2- 172.21.15.33:0/1354836013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febc8101910 0x7febc8197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:16.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.807+0000 7febce469640 1 -- 172.21.15.33:0/1354836013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febc8100720 msgr2=0x7febc8197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:16.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.807+0000 7febce469640 1 --2- 172.21.15.33:0/1354836013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febc8100720 0x7febc8197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:16.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.807+0000 7febce469640 1 -- 172.21.15.33:0/1354836013 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febc819f6c0 con 0x7febc81022d0 2024-03-19T02:53:16.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.807+0000 7febcdc68640 1 --2- 172.21.15.33:0/1354836013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febc8100720 0x7febc8197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:16.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.807+0000 7febce469640 1 --2- 172.21.15.33:0/1354836013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7febc81022d0 0x7febc819f050 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7febb800eb10 tx=0x7febb800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:16.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.807+0000 7febc6ffd640 1 -- 172.21.15.33:0/1354836013 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb8004b20 con 0x7febc81022d0 2024-03-19T02:53:16.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.807+0000 7febcfecc640 1 -- 172.21.15.33:0/1354836013 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febc819f9b0 con 0x7febc81022d0 2024-03-19T02:53:16.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.807+0000 7febc6ffd640 1 -- 172.21.15.33:0/1354836013 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7febb8004cc0 con 0x7febc81022d0 2024-03-19T02:53:16.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.807+0000 7febc6ffd640 1 -- 172.21.15.33:0/1354836013 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb8010690 con 0x7febc81022d0 2024-03-19T02:53:16.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.807+0000 7febcfecc640 1 -- 172.21.15.33:0/1354836013 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febc819ff00 con 0x7febc81022d0 2024-03-19T02:53:16.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.808+0000 7febcfecc640 1 -- 172.21.15.33:0/1354836013 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febc81a0130 con 0x7febc81022d0 2024-03-19T02:53:16.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.809+0000 7febc6ffd640 1 -- 172.21.15.33:0/1354836013 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7febb8005000 con 0x7febc81022d0 2024-03-19T02:53:16.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.810+0000 7febc6ffd640 1 --2- 172.21.15.33:0/1354836013 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7febb00a2800 0x7febb00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:16.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.810+0000 7febcdc68640 1 --2- 172.21.15.33:0/1354836013 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7febb00a2800 0x7febb00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:16.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.810+0000 7febc6ffd640 1 -- 172.21.15.33:0/1354836013 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7febb80cf580 con 0x7febc81022d0 2024-03-19T02:53:16.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.811+0000 7febcdc68640 1 --2- 172.21.15.33:0/1354836013 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7febb00a2800 0x7febb00a4cc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7febc002c180 tx=0x7febc00349a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:16.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.814+0000 7febc6ffd640 1 -- 172.21.15.33:0/1354836013 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7febb8053740 con 0x7febc81022d0 2024-03-19T02:53:16.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.995+0000 7febcfecc640 1 -- 172.21.15.33:0/1354836013 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7febc81a0300 con 0x7febc81022d0 2024-03-19T02:53:16.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.996+0000 7febc6ffd640 1 -- 172.21.15.33:0/1354836013 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v97) ==== 93+0+10 (secure 0 0 0) 0x7febb809c290 con 0x7febc81022d0 2024-03-19T02:53:16.997 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 1 2024-03-19T02:53:16.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.998+0000 7febcfecc640 1 -- 172.21.15.33:0/1354836013 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7febb00a2800 msgr2=0x7febb00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:16.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.998+0000 7febcfecc640 1 --2- 172.21.15.33:0/1354836013 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7febb00a2800 0x7febb00a4cc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7febc002c180 tx=0x7febc00349a0 comp rx=0 tx=0).stop 2024-03-19T02:53:16.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.998+0000 7febcfecc640 1 -- 172.21.15.33:0/1354836013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7febc81022d0 msgr2=0x7febc819f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:16.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.998+0000 7febcfecc640 1 --2- 172.21.15.33:0/1354836013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7febc81022d0 0x7febc819f050 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7febb800eb10 tx=0x7febb800c2d0 comp rx=0 tx=0).stop 2024-03-19T02:53:16.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.998+0000 7febcfecc640 1 -- 172.21.15.33:0/1354836013 shutdown_connections 2024-03-19T02:53:16.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.998+0000 7febcfecc640 1 --2- 172.21.15.33:0/1354836013 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7febb00a2800 0x7febb00a4cc0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:16.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.998+0000 7febcfecc640 1 --2- 172.21.15.33:0/1354836013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7febc8101910 0x7febc8197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:16.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.998+0000 7febcfecc640 1 --2- 172.21.15.33:0/1354836013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7febc81022d0 0x7febc819f050 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:16.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.998+0000 7febcfecc640 1 --2- 172.21.15.33:0/1354836013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febc8100720 0x7febc8197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:16.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.998+0000 7febcfecc640 1 -- 172.21.15.33:0/1354836013 >> 172.21.15.33:0/1354836013 conn(0x7febc80fbcb0 msgr2=0x7febc80689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:17.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.998+0000 7febcfecc640 1 -- 172.21.15.33:0/1354836013 shutdown_connections 2024-03-19T02:53:17.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:16.998+0000 7febcfecc640 1 -- 172.21.15.33:0/1354836013 wait complete. 2024-03-19T02:53:17.009 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:16 smithi033 ceph-mon[31973]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:53:17.009 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:16 smithi033 ceph-mon[31973]: from='client.15516 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:53:17.009 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:16 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/374282996' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:53:17.010 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T02:53:17.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.115+0000 7ffa23332640 1 -- 172.21.15.33:0/3347767531 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa1c100730 msgr2=0x7ffa1c100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:17.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.115+0000 7ffa23332640 1 --2- 172.21.15.33:0/3347767531 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa1c100730 0x7ffa1c100b50 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ffa18009900 tx=0x7ffa1802bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:17.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.115+0000 7ffa23332640 1 -- 172.21.15.33:0/3347767531 shutdown_connections 2024-03-19T02:53:17.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.115+0000 7ffa23332640 1 --2- 172.21.15.33:0/3347767531 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa1c1022e0 0x7ffa1c108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:17.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.115+0000 7ffa23332640 1 --2- 172.21.15.33:0/3347767531 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa1c101920 0x7ffa1c101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:17.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.115+0000 7ffa23332640 1 --2- 172.21.15.33:0/3347767531 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa1c100730 0x7ffa1c100b50 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:17.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.115+0000 7ffa23332640 1 -- 172.21.15.33:0/3347767531 >> 172.21.15.33:0/3347767531 conn(0x7ffa1c0fbcc0 msgr2=0x7ffa1c0fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:17.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.115+0000 7ffa23332640 1 -- 172.21.15.33:0/3347767531 shutdown_connections 2024-03-19T02:53:17.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.115+0000 7ffa23332640 1 -- 172.21.15.33:0/3347767531 wait complete. 2024-03-19T02:53:17.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.116+0000 7ffa23332640 1 Processor -- start 2024-03-19T02:53:17.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.116+0000 7ffa23332640 1 -- start start 2024-03-19T02:53:17.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.116+0000 7ffa23332640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa1c100730 0x7ffa1c197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:17.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.116+0000 7ffa23332640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa1c101920 0x7ffa1c197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:17.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.116+0000 7ffa23332640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa1c1022e0 0x7ffa1c19f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:17.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.116+0000 7ffa23332640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1c10d380 con 0x7ffa1c100730 2024-03-19T02:53:17.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.116+0000 7ffa23332640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1c10d200 con 0x7ffa1c101920 2024-03-19T02:53:17.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.116+0000 7ffa23332640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1c10d500 con 0x7ffa1c1022e0 2024-03-19T02:53:17.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.116+0000 7ffa210ce640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa1c100730 0x7ffa1c197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:17.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.116+0000 7ffa210ce640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa1c100730 0x7ffa1c197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46870/0 (socket says 172.21.15.33:46870) 2024-03-19T02:53:17.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.116+0000 7ffa210ce640 1 -- 172.21.15.33:0/1826399002 learned_addr learned my addr 172.21.15.33:0/1826399002 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:17.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.116+0000 7ffa208cd640 1 --2- 172.21.15.33:0/1826399002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa1c101920 0x7ffa1c197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:17.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.116+0000 7ffa218cf640 1 --2- 172.21.15.33:0/1826399002 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa1c1022e0 0x7ffa1c19f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:17.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.116+0000 7ffa210ce640 1 -- 172.21.15.33:0/1826399002 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa1c1022e0 msgr2=0x7ffa1c19f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:17.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.116+0000 7ffa210ce640 1 --2- 172.21.15.33:0/1826399002 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa1c1022e0 0x7ffa1c19f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:17.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.116+0000 7ffa210ce640 1 -- 172.21.15.33:0/1826399002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa1c101920 msgr2=0x7ffa1c197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:17.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.116+0000 7ffa210ce640 1 --2- 172.21.15.33:0/1826399002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa1c101920 0x7ffa1c197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:17.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.116+0000 7ffa210ce640 1 -- 172.21.15.33:0/1826399002 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa1c19f6d0 con 0x7ffa1c100730 2024-03-19T02:53:17.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.116+0000 7ffa210ce640 1 --2- 172.21.15.33:0/1826399002 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa1c100730 0x7ffa1c197ab0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7ffa18000c00 tx=0x7ffa18005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:17.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.116+0000 7ffa127fc640 1 -- 172.21.15.33:0/1826399002 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa18043020 con 0x7ffa1c100730 2024-03-19T02:53:17.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.117+0000 7ffa127fc640 1 -- 172.21.15.33:0/1826399002 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffa18004350 con 0x7ffa1c100730 2024-03-19T02:53:17.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.117+0000 7ffa23332640 1 -- 172.21.15.33:0/1826399002 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa1c19f960 con 0x7ffa1c100730 2024-03-19T02:53:17.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.117+0000 7ffa127fc640 1 -- 172.21.15.33:0/1826399002 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa18003b80 con 0x7ffa1c100730 2024-03-19T02:53:17.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.117+0000 7ffa23332640 1 -- 172.21.15.33:0/1826399002 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa1c19feb0 con 0x7ffa1c100730 2024-03-19T02:53:17.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.118+0000 7ffa23332640 1 -- 172.21.15.33:0/1826399002 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa1c107bc0 con 0x7ffa1c100730 2024-03-19T02:53:17.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.119+0000 7ffa127fc640 1 -- 172.21.15.33:0/1826399002 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ffa18046020 con 0x7ffa1c100730 2024-03-19T02:53:17.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.119+0000 7ffa127fc640 1 --2- 172.21.15.33:0/1826399002 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ffa080a2800 0x7ffa080a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:17.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.119+0000 7ffa127fc640 1 -- 172.21.15.33:0/1826399002 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7ffa180f0f10 con 0x7ffa1c100730 2024-03-19T02:53:17.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.119+0000 7ffa208cd640 1 --2- 172.21.15.33:0/1826399002 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ffa080a2800 0x7ffa080a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:17.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.120+0000 7ffa208cd640 1 --2- 172.21.15.33:0/1826399002 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ffa080a2800 0x7ffa080a4cc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c198fc0 tx=0x7ffa0c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:17.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.124+0000 7ffa127fc640 1 -- 172.21.15.33:0/1826399002 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ffa1804d020 con 0x7ffa1c100730 2024-03-19T02:53:17.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:16 smithi096 ceph-mon[35946]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:53:17.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:16 smithi096 ceph-mon[35946]: from='client.15516 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:53:17.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:16 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/374282996' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:53:17.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:16 smithi155 ceph-mon[35804]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:53:17.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:16 smithi155 ceph-mon[35804]: from='client.15516 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T02:53:17.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:16 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/374282996' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:53:17.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.306+0000 7ffa23332640 1 -- 172.21.15.33:0/1826399002 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ffa1c062420 con 0x7ffa1c100730 2024-03-19T02:53:17.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.306+0000 7ffa127fc640 1 -- 172.21.15.33:0/1826399002 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v97) ==== 104+0+11 (secure 0 0 0) 0x7ffa180bdc20 con 0x7ffa1c100730 2024-03-19T02:53:17.307 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 32 2024-03-19T02:53:17.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.308+0000 7ffa23332640 1 -- 172.21.15.33:0/1826399002 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ffa080a2800 msgr2=0x7ffa080a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:17.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.308+0000 7ffa23332640 1 --2- 172.21.15.33:0/1826399002 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ffa080a2800 0x7ffa080a4cc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c198fc0 tx=0x7ffa0c005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:17.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.308+0000 7ffa23332640 1 -- 172.21.15.33:0/1826399002 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa1c100730 msgr2=0x7ffa1c197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:17.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.308+0000 7ffa23332640 1 --2- 172.21.15.33:0/1826399002 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa1c100730 0x7ffa1c197ab0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7ffa18000c00 tx=0x7ffa18005dc0 comp rx=0 tx=0).stop 2024-03-19T02:53:17.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.308+0000 7ffa23332640 1 -- 172.21.15.33:0/1826399002 shutdown_connections 2024-03-19T02:53:17.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.308+0000 7ffa23332640 1 --2- 172.21.15.33:0/1826399002 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ffa080a2800 0x7ffa080a4cc0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:17.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.308+0000 7ffa23332640 1 --2- 172.21.15.33:0/1826399002 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa1c1022e0 0x7ffa1c19f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:17.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.308+0000 7ffa23332640 1 --2- 172.21.15.33:0/1826399002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa1c101920 0x7ffa1c197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:17.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.308+0000 7ffa23332640 1 --2- 172.21.15.33:0/1826399002 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa1c100730 0x7ffa1c197ab0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:17.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.308+0000 7ffa23332640 1 -- 172.21.15.33:0/1826399002 >> 172.21.15.33:0/1826399002 conn(0x7ffa1c0fbcc0 msgr2=0x7ffa1c104d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:17.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.308+0000 7ffa23332640 1 -- 172.21.15.33:0/1826399002 shutdown_connections 2024-03-19T02:53:17.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.308+0000 7ffa23332640 1 -- 172.21.15.33:0/1826399002 wait complete. 2024-03-19T02:53:17.320 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T02:53:17.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.425+0000 7f051a6f1640 1 -- 172.21.15.33:0/4233559537 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0514100740 msgr2=0x7f0514100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:17.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.425+0000 7f051a6f1640 1 --2- 172.21.15.33:0/4233559537 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0514100740 0x7f0514100b60 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f04fc009900 tx=0x7f04fc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:17.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.425+0000 7f051a6f1640 1 -- 172.21.15.33:0/4233559537 shutdown_connections 2024-03-19T02:53:17.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.425+0000 7f051a6f1640 1 --2- 172.21.15.33:0/4233559537 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0514101930 0x7f0514101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:17.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.425+0000 7f051a6f1640 1 --2- 172.21.15.33:0/4233559537 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05141022f0 0x7f0514108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:17.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.425+0000 7f051a6f1640 1 --2- 172.21.15.33:0/4233559537 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0514100740 0x7f0514100b60 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:17.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.425+0000 7f051a6f1640 1 -- 172.21.15.33:0/4233559537 >> 172.21.15.33:0/4233559537 conn(0x7f05140fbcd0 msgr2=0x7f05140fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:17.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.425+0000 7f051a6f1640 1 -- 172.21.15.33:0/4233559537 shutdown_connections 2024-03-19T02:53:17.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.425+0000 7f051a6f1640 1 -- 172.21.15.33:0/4233559537 wait complete. 2024-03-19T02:53:17.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.426+0000 7f051a6f1640 1 Processor -- start 2024-03-19T02:53:17.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.426+0000 7f051a6f1640 1 -- start start 2024-03-19T02:53:17.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.426+0000 7f051a6f1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0514100740 0x7f051406a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:17.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.426+0000 7f051a6f1640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0514101930 0x7f051419c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:17.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.426+0000 7f051a6f1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05141022f0 0x7f051419ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:17.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.426+0000 7f051a6f1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f051410c260 con 0x7f0514100740 2024-03-19T02:53:17.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.426+0000 7f051a6f1640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f051410c0e0 con 0x7f05141022f0 2024-03-19T02:53:17.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.426+0000 7f051a6f1640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f051410c3e0 con 0x7f0514101930 2024-03-19T02:53:17.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.426+0000 7f0513fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0514100740 0x7f051406a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:17.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.426+0000 7f0513fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0514100740 0x7f051406a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46894/0 (socket says 172.21.15.33:46894) 2024-03-19T02:53:17.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.426+0000 7f0513fff640 1 -- 172.21.15.33:0/2615767252 learned_addr learned my addr 172.21.15.33:0/2615767252 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:17.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.426+0000 7f05137fe640 1 --2- 172.21.15.33:0/2615767252 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0514101930 0x7f051419c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:17.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.426+0000 7f0518c8e640 1 --2- 172.21.15.33:0/2615767252 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05141022f0 0x7f051419ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:17.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.426+0000 7f0513fff640 1 -- 172.21.15.33:0/2615767252 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0514101930 msgr2=0x7f051419c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:17.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.426+0000 7f0513fff640 1 --2- 172.21.15.33:0/2615767252 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0514101930 0x7f051419c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:17.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.426+0000 7f0513fff640 1 -- 172.21.15.33:0/2615767252 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05141022f0 msgr2=0x7f051419ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:17.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.426+0000 7f0513fff640 1 --2- 172.21.15.33:0/2615767252 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05141022f0 0x7f051419ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:17.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.426+0000 7f0513fff640 1 -- 172.21.15.33:0/2615767252 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f051419f730 con 0x7f0514100740 2024-03-19T02:53:17.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.427+0000 7f0513fff640 1 --2- 172.21.15.33:0/2615767252 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0514100740 0x7f051406a960 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f04fc000c00 tx=0x7f04fc005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:17.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.427+0000 7f05117fa640 1 -- 172.21.15.33:0/2615767252 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04fc043020 con 0x7f0514100740 2024-03-19T02:53:17.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.427+0000 7f051a6f1640 1 -- 172.21.15.33:0/2615767252 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f051419f960 con 0x7f0514100740 2024-03-19T02:53:17.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.427+0000 7f05117fa640 1 -- 172.21.15.33:0/2615767252 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f04fc004350 con 0x7f0514100740 2024-03-19T02:53:17.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.427+0000 7f05117fa640 1 -- 172.21.15.33:0/2615767252 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04fc003b80 con 0x7f0514100740 2024-03-19T02:53:17.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.427+0000 7f051a6f1640 1 -- 172.21.15.33:0/2615767252 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f051419ff00 con 0x7f0514100740 2024-03-19T02:53:17.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.428+0000 7f051a6f1640 1 -- 172.21.15.33:0/2615767252 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0514068a60 con 0x7f0514100740 2024-03-19T02:53:17.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.429+0000 7f05117fa640 1 -- 172.21.15.33:0/2615767252 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f04fc046020 con 0x7f0514100740 2024-03-19T02:53:17.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.429+0000 7f05117fa640 1 --2- 172.21.15.33:0/2615767252 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f05000a2800 0x7f05000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:17.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.429+0000 7f05137fe640 1 --2- 172.21.15.33:0/2615767252 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f05000a2800 0x7f05000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:17.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.430+0000 7f05117fa640 1 -- 172.21.15.33:0/2615767252 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f04fc0f0f70 con 0x7f0514100740 2024-03-19T02:53:17.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.430+0000 7f05137fe640 1 --2- 172.21.15.33:0/2615767252 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f05000a2800 0x7f05000a4cc0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f051406a6d0 tx=0x7f0504005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:17.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.434+0000 7f05117fa640 1 -- 172.21.15.33:0/2615767252 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f04fc04d020 con 0x7f0514100740 2024-03-19T02:53:17.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.616+0000 7f051a6f1640 1 -- 172.21.15.33:0/2615767252 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0514062420 con 0x7f0514100740 2024-03-19T02:53:17.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.616+0000 7f05117fa640 1 -- 172.21.15.33:0/2615767252 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v97) ==== 100+0+11 (secure 0 0 0) 0x7f04fc0bdc20 con 0x7f0514100740 2024-03-19T02:53:17.617 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 64 2024-03-19T02:53:17.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.618+0000 7f051a6f1640 1 -- 172.21.15.33:0/2615767252 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f05000a2800 msgr2=0x7f05000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:17.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.618+0000 7f051a6f1640 1 --2- 172.21.15.33:0/2615767252 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f05000a2800 0x7f05000a4cc0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f051406a6d0 tx=0x7f0504005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:17.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.618+0000 7f051a6f1640 1 -- 172.21.15.33:0/2615767252 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0514100740 msgr2=0x7f051406a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:17.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.618+0000 7f051a6f1640 1 --2- 172.21.15.33:0/2615767252 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0514100740 0x7f051406a960 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f04fc000c00 tx=0x7f04fc005dc0 comp rx=0 tx=0).stop 2024-03-19T02:53:17.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.618+0000 7f051a6f1640 1 -- 172.21.15.33:0/2615767252 shutdown_connections 2024-03-19T02:53:17.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.618+0000 7f051a6f1640 1 --2- 172.21.15.33:0/2615767252 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f05000a2800 0x7f05000a4cc0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:17.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.618+0000 7f051a6f1640 1 --2- 172.21.15.33:0/2615767252 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0514101930 0x7f051419c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:17.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.618+0000 7f051a6f1640 1 --2- 172.21.15.33:0/2615767252 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05141022f0 0x7f051419ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:17.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.618+0000 7f051a6f1640 1 --2- 172.21.15.33:0/2615767252 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0514100740 0x7f051406a960 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:17.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.618+0000 7f051a6f1640 1 -- 172.21.15.33:0/2615767252 >> 172.21.15.33:0/2615767252 conn(0x7f05140fbcd0 msgr2=0x7f0514104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:17.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.618+0000 7f051a6f1640 1 -- 172.21.15.33:0/2615767252 shutdown_connections 2024-03-19T02:53:17.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.618+0000 7f051a6f1640 1 -- 172.21.15.33:0/2615767252 wait complete. 2024-03-19T02:53:17.630 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T02:53:17.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.734+0000 7ff667f9d640 1 -- 172.21.15.33:0/355017546 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff660102a20 msgr2=0x7ff660102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:17.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.734+0000 7ff667f9d640 1 --2- 172.21.15.33:0/355017546 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff660102a20 0x7ff660102ea0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff654009860 tx=0x7ff65402bb70 comp rx=0 tx=0).stop 2024-03-19T02:53:17.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.734+0000 7ff667f9d640 1 -- 172.21.15.33:0/355017546 shutdown_connections 2024-03-19T02:53:17.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.734+0000 7ff667f9d640 1 --2- 172.21.15.33:0/355017546 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff660102a20 0x7ff660102ea0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:17.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.734+0000 7ff667f9d640 1 --2- 172.21.15.33:0/355017546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff6601033e0 0x7ff660109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:17.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.734+0000 7ff667f9d640 1 --2- 172.21.15.33:0/355017546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff660101830 0x7ff660101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:17.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.734+0000 7ff667f9d640 1 -- 172.21.15.33:0/355017546 >> 172.21.15.33:0/355017546 conn(0x7ff6600fcdc0 msgr2=0x7ff6600ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:17.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.735+0000 7ff667f9d640 1 -- 172.21.15.33:0/355017546 shutdown_connections 2024-03-19T02:53:17.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.735+0000 7ff667f9d640 1 -- 172.21.15.33:0/355017546 wait complete. 2024-03-19T02:53:17.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.735+0000 7ff667f9d640 1 Processor -- start 2024-03-19T02:53:17.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.735+0000 7ff667f9d640 1 -- start start 2024-03-19T02:53:17.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.735+0000 7ff667f9d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff660101830 0x7ff6600755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:17.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.735+0000 7ff667f9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff660102a20 0x7ff660073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:17.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.735+0000 7ff667f9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6601033e0 0x7ff660074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:17.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.735+0000 7ff667f9d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff66010d340 con 0x7ff660102a20 2024-03-19T02:53:17.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.735+0000 7ff667f9d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff66010d1c0 con 0x7ff6601033e0 2024-03-19T02:53:17.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.735+0000 7ff667f9d640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff66010d4c0 con 0x7ff660101830 2024-03-19T02:53:17.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.735+0000 7ff665538640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff660102a20 0x7ff660073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:17.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.735+0000 7ff665d39640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff660101830 0x7ff6600755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:17.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.735+0000 7ff665538640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff660102a20 0x7ff660073c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46910/0 (socket says 172.21.15.33:46910) 2024-03-19T02:53:17.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.735+0000 7ff66653a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6601033e0 0x7ff660074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:17.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.735+0000 7ff665d39640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff660101830 0x7ff6600755d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:48824/0 (socket says 172.21.15.33:48824) 2024-03-19T02:53:17.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.735+0000 7ff665538640 1 -- 172.21.15.33:0/3643395201 learned_addr learned my addr 172.21.15.33:0/3643395201 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:17.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.736+0000 7ff66653a640 1 -- 172.21.15.33:0/3643395201 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff660101830 msgr2=0x7ff6600755d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:17.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.736+0000 7ff66653a640 1 --2- 172.21.15.33:0/3643395201 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff660101830 0x7ff6600755d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:17.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.736+0000 7ff66653a640 1 -- 172.21.15.33:0/3643395201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff660102a20 msgr2=0x7ff660073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:17.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.736+0000 7ff66653a640 1 --2- 172.21.15.33:0/3643395201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff660102a20 0x7ff660073c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:17.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.736+0000 7ff66653a640 1 -- 172.21.15.33:0/3643395201 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6600749e0 con 0x7ff6601033e0 2024-03-19T02:53:17.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.736+0000 7ff665538640 1 --2- 172.21.15.33:0/3643395201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff660102a20 0x7ff660073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:17.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.736+0000 7ff66653a640 1 --2- 172.21.15.33:0/3643395201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6601033e0 0x7ff660074150 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff65000efc0 tx=0x7ff65000c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:17.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.736+0000 7ff65effd640 1 -- 172.21.15.33:0/3643395201 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff650003970 con 0x7ff6601033e0 2024-03-19T02:53:17.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.736+0000 7ff667f9d640 1 -- 172.21.15.33:0/3643395201 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6601a1be0 con 0x7ff6601033e0 2024-03-19T02:53:17.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.736+0000 7ff65effd640 1 -- 172.21.15.33:0/3643395201 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff650004020 con 0x7ff6601033e0 2024-03-19T02:53:17.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.736+0000 7ff65effd640 1 -- 172.21.15.33:0/3643395201 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff65000b940 con 0x7ff6601033e0 2024-03-19T02:53:17.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.736+0000 7ff667f9d640 1 -- 172.21.15.33:0/3643395201 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6601a20b0 con 0x7ff6601033e0 2024-03-19T02:53:17.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.737+0000 7ff667f9d640 1 -- 172.21.15.33:0/3643395201 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff660108c10 con 0x7ff6601033e0 2024-03-19T02:53:17.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.738+0000 7ff65effd640 1 -- 172.21.15.33:0/3643395201 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff6500041d0 con 0x7ff6601033e0 2024-03-19T02:53:17.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.738+0000 7ff65effd640 1 --2- 172.21.15.33:0/3643395201 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff6480a2800 0x7ff6480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:17.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.739+0000 7ff665d39640 1 --2- 172.21.15.33:0/3643395201 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff6480a2800 0x7ff6480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:17.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.739+0000 7ff65effd640 1 -- 172.21.15.33:0/3643395201 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7ff6500cf160 con 0x7ff6601033e0 2024-03-19T02:53:17.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.739+0000 7ff665d39640 1 --2- 172.21.15.33:0/3643395201 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff6480a2800 0x7ff6480a4cc0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff660102880 tx=0x7ff658005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:17.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.742+0000 7ff65effd640 1 -- 172.21.15.33:0/3643395201 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff65009be70 con 0x7ff6601033e0 2024-03-19T02:53:17.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.927+0000 7ff667f9d640 1 -- 172.21.15.33:0/3643395201 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7ff66002d380 con 0x7ff6601033e0 2024-03-19T02:53:17.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.928+0000 7ff65effd640 1 -- 172.21.15.33:0/3643395201 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v97) ==== 103+0+11 (secure 0 0 0) 0x7ff65009b770 con 0x7ff6601033e0 2024-03-19T02:53:17.929 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 32 2024-03-19T02:53:17.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.930+0000 7ff667f9d640 1 -- 172.21.15.33:0/3643395201 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff6480a2800 msgr2=0x7ff6480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:17.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.930+0000 7ff667f9d640 1 --2- 172.21.15.33:0/3643395201 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff6480a2800 0x7ff6480a4cc0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff660102880 tx=0x7ff658005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:17.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.930+0000 7ff667f9d640 1 -- 172.21.15.33:0/3643395201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6601033e0 msgr2=0x7ff660074150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:17.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.930+0000 7ff667f9d640 1 --2- 172.21.15.33:0/3643395201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6601033e0 0x7ff660074150 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff65000efc0 tx=0x7ff65000c800 comp rx=0 tx=0).stop 2024-03-19T02:53:17.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.930+0000 7ff667f9d640 1 -- 172.21.15.33:0/3643395201 shutdown_connections 2024-03-19T02:53:17.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.930+0000 7ff667f9d640 1 --2- 172.21.15.33:0/3643395201 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7ff6480a2800 0x7ff6480a4cc0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:17.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.930+0000 7ff667f9d640 1 --2- 172.21.15.33:0/3643395201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6601033e0 0x7ff660074150 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:17.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.930+0000 7ff667f9d640 1 --2- 172.21.15.33:0/3643395201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff660102a20 0x7ff660073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:17.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.930+0000 7ff667f9d640 1 --2- 172.21.15.33:0/3643395201 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff660101830 0x7ff6600755d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:17.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.930+0000 7ff667f9d640 1 -- 172.21.15.33:0/3643395201 >> 172.21.15.33:0/3643395201 conn(0x7ff6600fcdc0 msgr2=0x7ff660105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:17.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.930+0000 7ff667f9d640 1 -- 172.21.15.33:0/3643395201 shutdown_connections 2024-03-19T02:53:17.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:17.930+0000 7ff667f9d640 1 -- 172.21.15.33:0/3643395201 wait complete. 2024-03-19T02:53:17.941 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:17 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1354836013' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:53:17.941 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:17 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1826399002' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:53:17.941 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:17 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2615767252' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:53:17.941 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:17 smithi033 ceph-mon[31973]: pgmap v376: 129 pgs: 129 active+clean; 648 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-19T02:53:17.942 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:53:18.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.044+0000 7f350317e640 1 -- 172.21.15.33:0/2605708279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34fc102310 msgr2=0x7f34fc108ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:18.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.044+0000 7f350317e640 1 --2- 172.21.15.33:0/2605708279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34fc102310 0x7f34fc108ba0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f34f4009ca0 tx=0x7f34f402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.044+0000 7f350317e640 1 -- 172.21.15.33:0/2605708279 shutdown_connections 2024-03-19T02:53:18.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.044+0000 7f350317e640 1 --2- 172.21.15.33:0/2605708279 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34fc101950 0x7f34fc101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.044+0000 7f350317e640 1 --2- 172.21.15.33:0/2605708279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34fc102310 0x7f34fc108ba0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.044+0000 7f350317e640 1 --2- 172.21.15.33:0/2605708279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34fc100760 0x7f34fc100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.044+0000 7f350317e640 1 -- 172.21.15.33:0/2605708279 >> 172.21.15.33:0/2605708279 conn(0x7f34fc0fbcd0 msgr2=0x7f34fc0fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.044+0000 7f350317e640 1 -- 172.21.15.33:0/2605708279 shutdown_connections 2024-03-19T02:53:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.044+0000 7f350317e640 1 -- 172.21.15.33:0/2605708279 wait complete. 2024-03-19T02:53:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.045+0000 7f350317e640 1 Processor -- start 2024-03-19T02:53:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.045+0000 7f350317e640 1 -- start start 2024-03-19T02:53:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.045+0000 7f350317e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34fc100760 0x7f34fc06a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.045+0000 7f350317e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34fc101950 0x7f34fc19c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.045+0000 7f350317e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34fc102310 0x7f34fc19edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.045+0000 7f350317e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f34fc10c290 con 0x7f34fc102310 2024-03-19T02:53:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.045+0000 7f350317e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f34fc10c110 con 0x7f34fc100760 2024-03-19T02:53:18.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.045+0000 7f350317e640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f34fc10c410 con 0x7f34fc101950 2024-03-19T02:53:18.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.045+0000 7f350171b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34fc102310 0x7f34fc19edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.045+0000 7f350171b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34fc102310 0x7f34fc19edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46926/0 (socket says 172.21.15.33:46926) 2024-03-19T02:53:18.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.045+0000 7f350171b640 1 -- 172.21.15.33:0/3964483632 learned_addr learned my addr 172.21.15.33:0/3964483632 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:18.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.045+0000 7f3500f1a640 1 --2- 172.21.15.33:0/3964483632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34fc100760 0x7f34fc06a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.045+0000 7f34fbfff640 1 --2- 172.21.15.33:0/3964483632 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34fc101950 0x7f34fc19c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.045+0000 7f350171b640 1 -- 172.21.15.33:0/3964483632 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34fc101950 msgr2=0x7f34fc19c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:18.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.045+0000 7f350171b640 1 --2- 172.21.15.33:0/3964483632 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34fc101950 0x7f34fc19c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.045+0000 7f350171b640 1 -- 172.21.15.33:0/3964483632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34fc100760 msgr2=0x7f34fc06a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:18.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.045+0000 7f350171b640 1 --2- 172.21.15.33:0/3964483632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34fc100760 0x7f34fc06a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.045+0000 7f350171b640 1 -- 172.21.15.33:0/3964483632 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34fc19f6e0 con 0x7f34fc102310 2024-03-19T02:53:18.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.045+0000 7f350171b640 1 --2- 172.21.15.33:0/3964483632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34fc102310 0x7f34fc19edd0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f34f4032380 tx=0x7f34f4032820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:18.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.046+0000 7f34f9ffb640 1 -- 172.21.15.33:0/3964483632 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f4041020 con 0x7f34fc102310 2024-03-19T02:53:18.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.046+0000 7f34f9ffb640 1 -- 172.21.15.33:0/3964483632 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f34f4003e20 con 0x7f34fc102310 2024-03-19T02:53:18.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.046+0000 7f350317e640 1 -- 172.21.15.33:0/3964483632 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34fc19f910 con 0x7f34fc102310 2024-03-19T02:53:18.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.046+0000 7f34f9ffb640 1 -- 172.21.15.33:0/3964483632 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f4039040 con 0x7f34fc102310 2024-03-19T02:53:18.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.046+0000 7f350317e640 1 -- 172.21.15.33:0/3964483632 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34fc19fe00 con 0x7f34fc102310 2024-03-19T02:53:18.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.047+0000 7f350317e640 1 -- 172.21.15.33:0/3964483632 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34fc107ad0 con 0x7f34fc102310 2024-03-19T02:53:18.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.048+0000 7f34f9ffb640 1 -- 172.21.15.33:0/3964483632 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f34f4044020 con 0x7f34fc102310 2024-03-19T02:53:18.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.048+0000 7f34f9ffb640 1 --2- 172.21.15.33:0/3964483632 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f34e80a2800 0x7f34e80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.048+0000 7f34f9ffb640 1 -- 172.21.15.33:0/3964483632 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f34f40f0e80 con 0x7f34fc102310 2024-03-19T02:53:18.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.048+0000 7f3500f1a640 1 --2- 172.21.15.33:0/3964483632 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f34e80a2800 0x7f34e80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.049+0000 7f3500f1a640 1 --2- 172.21.15.33:0/3964483632 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f34e80a2800 0x7f34e80a4cc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f34fc1017b0 tx=0x7f34ec005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:18.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.053+0000 7f34f9ffb640 1 -- 172.21.15.33:0/3964483632 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f34f404b020 con 0x7f34fc102310 2024-03-19T02:53:18.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:17 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1354836013' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:53:18.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:17 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1826399002' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:53:18.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:17 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2615767252' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:53:18.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:17 smithi096 ceph-mon[35946]: pgmap v376: 129 pgs: 129 active+clean; 648 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-19T02:53:18.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:17 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1354836013' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T02:53:18.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:17 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1826399002' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T02:53:18.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:17 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2615767252' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T02:53:18.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:17 smithi155 ceph-mon[35804]: pgmap v376: 129 pgs: 129 active+clean; 648 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-19T02:53:18.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.277+0000 7f350317e640 1 -- 172.21.15.33:0/3964483632 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f34fc062420 con 0x7f34fc102310 2024-03-19T02:53:18.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.278+0000 7f34f9ffb640 1 -- 172.21.15.33:0/3964483632 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11867 (secure 0 0 0) 0x7f34f40bdb90 con 0x7f34fc102310 2024-03-19T02:53:18.279 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:53:18.279 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:50:52.201798+0000","modified":"2024-03-19T02:53:11.748697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24488,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/3731232169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":3731232169},{"type":"v1","addr":"172.21.15.96:6839","nonce":3731232169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24457,24488,24509]},"id":1}]} 2024-03-19T02:53:18.279 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-03-19T02:53:18.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.280+0000 7f350317e640 1 -- 172.21.15.33:0/3964483632 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f34e80a2800 msgr2=0x7f34e80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:18.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.280+0000 7f350317e640 1 --2- 172.21.15.33:0/3964483632 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f34e80a2800 0x7f34e80a4cc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f34fc1017b0 tx=0x7f34ec005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:18.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.280+0000 7f350317e640 1 -- 172.21.15.33:0/3964483632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34fc102310 msgr2=0x7f34fc19edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:18.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.280+0000 7f350317e640 1 --2- 172.21.15.33:0/3964483632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34fc102310 0x7f34fc19edd0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f34f4032380 tx=0x7f34f4032820 comp rx=0 tx=0).stop 2024-03-19T02:53:18.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.280+0000 7f350317e640 1 -- 172.21.15.33:0/3964483632 shutdown_connections 2024-03-19T02:53:18.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.280+0000 7f350317e640 1 --2- 172.21.15.33:0/3964483632 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f34e80a2800 0x7f34e80a4cc0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.280+0000 7f350317e640 1 --2- 172.21.15.33:0/3964483632 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34fc101950 0x7f34fc19c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.280+0000 7f350317e640 1 --2- 172.21.15.33:0/3964483632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34fc102310 0x7f34fc19edd0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.280+0000 7f350317e640 1 --2- 172.21.15.33:0/3964483632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34fc100760 0x7f34fc06a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.280+0000 7f350317e640 1 -- 172.21.15.33:0/3964483632 >> 172.21.15.33:0/3964483632 conn(0x7f34fc0fbcd0 msgr2=0x7f34fc104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:18.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.280+0000 7f350317e640 1 -- 172.21.15.33:0/3964483632 shutdown_connections 2024-03-19T02:53:18.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.280+0000 7f350317e640 1 -- 172.21.15.33:0/3964483632 wait complete. 2024-03-19T02:53:18.292 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T02:53:18.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.395+0000 7fbf86b80640 1 -- 172.21.15.33:0/3207104911 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbf801022f0 msgr2=0x7fbf80108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:18.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.395+0000 7fbf86b80640 1 --2- 172.21.15.33:0/3207104911 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbf801022f0 0x7fbf80108b80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fbf78009ca0 tx=0x7fbf7802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.395+0000 7fbf86b80640 1 -- 172.21.15.33:0/3207104911 shutdown_connections 2024-03-19T02:53:18.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.395+0000 7fbf86b80640 1 --2- 172.21.15.33:0/3207104911 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf80101930 0x7fbf80101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.395+0000 7fbf86b80640 1 --2- 172.21.15.33:0/3207104911 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbf801022f0 0x7fbf80108b80 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.395+0000 7fbf86b80640 1 --2- 172.21.15.33:0/3207104911 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf80100740 0x7fbf80100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.395+0000 7fbf86b80640 1 -- 172.21.15.33:0/3207104911 >> 172.21.15.33:0/3207104911 conn(0x7fbf800fbcb0 msgr2=0x7fbf800fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:18.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.395+0000 7fbf86b80640 1 -- 172.21.15.33:0/3207104911 shutdown_connections 2024-03-19T02:53:18.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.395+0000 7fbf86b80640 1 -- 172.21.15.33:0/3207104911 wait complete. 2024-03-19T02:53:18.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.395+0000 7fbf86b80640 1 Processor -- start 2024-03-19T02:53:18.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.395+0000 7fbf86b80640 1 -- start start 2024-03-19T02:53:18.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.395+0000 7fbf86b80640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbf80100740 0x7fbf800694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.395+0000 7fbf86b80640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf80101930 0x7fbf8019c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.395+0000 7fbf86b80640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf801022f0 0x7fbf8019ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.395+0000 7fbf86b80640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8010c250 con 0x7fbf80100740 2024-03-19T02:53:18.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.395+0000 7fbf86b80640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8010c0d0 con 0x7fbf80101930 2024-03-19T02:53:18.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.395+0000 7fbf86b80640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8010c3d0 con 0x7fbf801022f0 2024-03-19T02:53:18.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.395+0000 7fbf8491c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbf80100740 0x7fbf800694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.395+0000 7fbf8491c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbf80100740 0x7fbf800694f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46934/0 (socket says 172.21.15.33:46934) 2024-03-19T02:53:18.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.396+0000 7fbf8491c640 1 -- 172.21.15.33:0/596506148 learned_addr learned my addr 172.21.15.33:0/596506148 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:18.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.396+0000 7fbf7ffff640 1 --2- 172.21.15.33:0/596506148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf80101930 0x7fbf8019c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.396+0000 7fbf8511d640 1 --2- 172.21.15.33:0/596506148 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf801022f0 0x7fbf8019ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.396+0000 7fbf7ffff640 1 -- 172.21.15.33:0/596506148 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf801022f0 msgr2=0x7fbf8019ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:18.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.396+0000 7fbf7ffff640 1 --2- 172.21.15.33:0/596506148 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf801022f0 0x7fbf8019ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.396+0000 7fbf7ffff640 1 -- 172.21.15.33:0/596506148 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbf80100740 msgr2=0x7fbf800694f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:18.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.396+0000 7fbf7ffff640 1 --2- 172.21.15.33:0/596506148 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbf80100740 0x7fbf800694f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.396+0000 7fbf7ffff640 1 -- 172.21.15.33:0/596506148 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf8019f670 con 0x7fbf80101930 2024-03-19T02:53:18.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.396+0000 7fbf7ffff640 1 --2- 172.21.15.33:0/596506148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf80101930 0x7fbf8019c740 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c00e9f0 tx=0x7fbf6c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:18.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.396+0000 7fbf7dffb640 1 -- 172.21.15.33:0/596506148 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c00cd80 con 0x7fbf80101930 2024-03-19T02:53:18.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.396+0000 7fbf86b80640 1 -- 172.21.15.33:0/596506148 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf8019f900 con 0x7fbf80101930 2024-03-19T02:53:18.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.396+0000 7fbf7dffb640 1 -- 172.21.15.33:0/596506148 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbf6c0035f0 con 0x7fbf80101930 2024-03-19T02:53:18.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.396+0000 7fbf7dffb640 1 -- 172.21.15.33:0/596506148 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c010690 con 0x7fbf80101930 2024-03-19T02:53:18.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.396+0000 7fbf86b80640 1 -- 172.21.15.33:0/596506148 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf8019fea0 con 0x7fbf80101930 2024-03-19T02:53:18.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.397+0000 7fbf86b80640 1 -- 172.21.15.33:0/596506148 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf80107a60 con 0x7fbf80101930 2024-03-19T02:53:18.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.398+0000 7fbf7dffb640 1 -- 172.21.15.33:0/596506148 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fbf6c0037a0 con 0x7fbf80101930 2024-03-19T02:53:18.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.399+0000 7fbf7dffb640 1 --2- 172.21.15.33:0/596506148 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbf700a2800 0x7fbf700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.399+0000 7fbf8491c640 1 --2- 172.21.15.33:0/596506148 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbf700a2800 0x7fbf700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.399+0000 7fbf7dffb640 1 -- 172.21.15.33:0/596506148 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7fbf6c0cf0b0 con 0x7fbf80101930 2024-03-19T02:53:18.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.399+0000 7fbf8491c640 1 --2- 172.21.15.33:0/596506148 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbf700a2800 0x7fbf700a4cc0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fbf740058c0 tx=0x7fbf74005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:18.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.407+0000 7fbf7dffb640 1 -- 172.21.15.33:0/596506148 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbf6c09bd60 con 0x7fbf80101930 2024-03-19T02:53:18.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.590+0000 7fbf86b80640 1 -- 172.21.15.33:0/596506148 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbf8002d3b0 con 0x7fbf80101930 2024-03-19T02:53:18.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.593+0000 7fbf7dffb640 1 -- 172.21.15.33:0/596506148 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) ==== 74+0+24663 (secure 0 0 0) 0x7fbf6c09b660 con 0x7fbf80101930 2024-03-19T02:53:18.594 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:53:18.594 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":97,"fsid":"2b7a8938-e59a-11ee-95c9-87774f69a715","created":"2024-03-19T02:42:37.912179+0000","modified":"2024-03-19T02:52:41.116227+0000","last_up_change":"2024-03-19T02:50:01.539392+0000","last_in_change":"2024-03-19T02:49:40.534415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:46:09.963737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:50:50.069641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:50:51.381064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:50:55.476667+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"090a89ab-6b7b-4317-8823-178d4212b2f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6803","nonce":1633124995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6805","nonce":1633124995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6809","nonce":1633124995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6807","nonce":1633124995}]},"public_addr":"172.21.15.33:6803/1633124995","cluster_addr":"172.21.15.33:6805/1633124995","heartbeat_back_addr":"172.21.15.33:6809/1633124995","heartbeat_front_addr":"172.21.15.33:6807/1633124995","state":["exists","up"]},{"osd":1,"uuid":"f50eec55-1956-4486-8593-f2c8fa610e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6801","nonce":340144398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6803","nonce":340144398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6807","nonce":340144398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6805","nonce":340144398}]},"public_addr":"172.21.15.96:6801/340144398","cluster_addr":"172.21.15.96:6803/340144398","heartbeat_back_addr":"172.21.15.96:6807/340144398","heartbeat_front_addr":"172.21.15.96:6805/340144398","state":["exists","up"]},{"osd":2,"uuid":"893a2d62-80d9-4e5e-bc6a-a1c8c681d652","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6801","nonce":3388743876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6803","nonce":3388743876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6807","nonce":3388743876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6805","nonce":3388743876}]},"public_addr":"172.21.15.155:6801/3388743876","cluster_addr":"172.21.15.155:6803/3388743876","heartbeat_back_addr":"172.21.15.155:6807/3388743876","heartbeat_front_addr":"172.21.15.155:6805/3388743876","state":["exists","up"]},{"osd":3,"uuid":"b07e0482-4281-4d32-9dfe-9995d7246175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6811","nonce":677007171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6813","nonce":677007171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6817","nonce":677007171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6815","nonce":677007171}]},"public_addr":"172.21.15.33:6811/677007171","cluster_addr":"172.21.15.33:6813/677007171","heartbeat_back_addr":"172.21.15.33:6817/677007171","heartbeat_front_addr":"172.21.15.33:6815/677007171","state":["exists","up"]},{"osd":4,"uuid":"a988384b-af6b-47c6-8c07-5c9673acee45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6809","nonce":3106158554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6811","nonce":3106158554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6815","nonce":3106158554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6813","nonce":3106158554}]},"public_addr":"172.21.15.96:6809/3106158554","cluster_addr":"172.21.15.96:6811/3106158554","heartbeat_back_addr":"172.21.15.96:6815/3106158554","heartbeat_front_addr":"172.21.15.96:6813/3106158554","state":["exists","up"]},{"osd":5,"uuid":"198dc992-96f8-4c52-96ee-1fd5de148bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6809","nonce":327293324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6811","nonce":327293324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6815","nonce":327293324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6813","nonce":327293324}]},"public_addr":"172.21.15.155:6809/327293324","cluster_addr":"172.21.15.155:6811/327293324","heartbeat_back_addr":"172.21.15.155:6815/327293324","heartbeat_front_addr":"172.21.15.155:6813/327293324","state":["exists","up"]},{"osd":6,"uuid":"a7278ec0-ab2e-49ed-be53-84d8b9e24177","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6819","nonce":873871509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6821","nonce":873871509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6825","nonce":873871509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6823","nonce":873871509}]},"public_addr":"172.21.15.33:6819/873871509","cluster_addr":"172.21.15.33:6821/873871509","heartbeat_back_addr":"172.21.15.33:6825/873871509","heartbeat_front_addr":"172.21.15.33:6823/873871509","state":["exists","up"]},{"osd":7,"uuid":"727e133d-aa24-4ffc-90e2-266eb2aab8c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6817","nonce":2334123891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6819","nonce":2334123891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6823","nonce":2334123891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6821","nonce":2334123891}]},"public_addr":"172.21.15.96:6817/2334123891","cluster_addr":"172.21.15.96:6819/2334123891","heartbeat_back_addr":"172.21.15.96:6823/2334123891","heartbeat_front_addr":"172.21.15.96:6821/2334123891","state":["exists","up"]},{"osd":8,"uuid":"bb20eec0-39e5-4b86-9db5-486b1840cdad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6817","nonce":3498516903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6819","nonce":3498516903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6823","nonce":3498516903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6821","nonce":3498516903}]},"public_addr":"172.21.15.155:6817/3498516903","cluster_addr":"172.21.15.155:6819/3498516903","heartbeat_back_addr":"172.21.15.155:6823/3498516903","heartbeat_front_addr":"172.21.15.155:6821/3498516903","state":["exists","up"]},{"osd":9,"uuid":"179c6228-3e23-4f65-91a6-1ef0005d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6827","nonce":760777213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6829","nonce":760777213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6833","nonce":760777213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6831","nonce":760777213}]},"public_addr":"172.21.15.33:6827/760777213","cluster_addr":"172.21.15.33:6829/760777213","heartbeat_back_addr":"172.21.15.33:6833/760777213","heartbeat_front_addr":"172.21.15.33:6831/760777213","state":["exists","up"]},{"osd":10,"uuid":"be17b478-3c2e-41fb-bede-958333acdf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6825","nonce":1404898482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6827","nonce":1404898482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6831","nonce":1404898482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6829","nonce":1404898482}]},"public_addr":"172.21.15.96:6825/1404898482","cluster_addr":"172.21.15.96:6827/1404898482","heartbeat_back_addr":"172.21.15.96:6831/1404898482","heartbeat_front_addr":"172.21.15.96:6829/1404898482","state":["exists","up"]},{"osd":11,"uuid":"c0903ed2-cc1e-4c18-8731-1f0ee4dd0457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6825","nonce":792352827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6827","nonce":792352827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6831","nonce":792352827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6829","nonce":792352827}]},"public_addr":"172.21.15.155:6825/792352827","cluster_addr":"172.21.15.155:6827/792352827","heartbeat_back_addr":"172.21.15.155:6831/792352827","heartbeat_front_addr":"172.21.15.155:6829/792352827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:17.942801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:42.109449+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:05.981599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:32.069574+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:57.431015+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:22.448592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:48.526515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:14.498352+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:40.175426+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:06.364006+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:32.853161+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:59.073538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.9","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:6801/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:6800/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/1067768047":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/729379453":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2586588742":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/2932814165":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6801/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2212564963":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6800/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/3275126058":"2024-03-20T02:43:39.622218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T02:53:18.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.595+0000 7fbf86b80640 1 -- 172.21.15.33:0/596506148 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbf700a2800 msgr2=0x7fbf700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:18.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.595+0000 7fbf86b80640 1 --2- 172.21.15.33:0/596506148 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbf700a2800 0x7fbf700a4cc0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fbf740058c0 tx=0x7fbf74005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:18.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.595+0000 7fbf86b80640 1 -- 172.21.15.33:0/596506148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf80101930 msgr2=0x7fbf8019c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:18.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.595+0000 7fbf86b80640 1 --2- 172.21.15.33:0/596506148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf80101930 0x7fbf8019c740 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c00e9f0 tx=0x7fbf6c00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.595+0000 7fbf86b80640 1 -- 172.21.15.33:0/596506148 shutdown_connections 2024-03-19T02:53:18.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.595+0000 7fbf86b80640 1 --2- 172.21.15.33:0/596506148 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fbf700a2800 0x7fbf700a4cc0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.595+0000 7fbf86b80640 1 --2- 172.21.15.33:0/596506148 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbf801022f0 0x7fbf8019ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.595+0000 7fbf86b80640 1 --2- 172.21.15.33:0/596506148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf80101930 0x7fbf8019c740 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.595+0000 7fbf86b80640 1 --2- 172.21.15.33:0/596506148 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbf80100740 0x7fbf800694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.595+0000 7fbf86b80640 1 -- 172.21.15.33:0/596506148 >> 172.21.15.33:0/596506148 conn(0x7fbf800fbcb0 msgr2=0x7fbf80104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:18.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.595+0000 7fbf86b80640 1 -- 172.21.15.33:0/596506148 shutdown_connections 2024-03-19T02:53:18.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.595+0000 7fbf86b80640 1 -- 172.21.15.33:0/596506148 wait complete. 2024-03-19T02:53:18.608 DEBUG:tasks.fwd_scrub:Yielding 2024-03-19T02:53:18.608 INFO:teuthology.run_tasks:Running task exec... 2024-03-19T02:53:18.618 INFO:teuthology.task.exec:Executing custom commands... 2024-03-19T02:53:18.618 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi033.front.sepia.ceph.com 2024-03-19T02:53:18.619 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-03-19T02:53:18.619 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-19T02:53:18.619 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:53:18.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:53:18.619 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:53:18.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.723+0000 7f273a9f4640 1 -- 172.21.15.33:0/124024002 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2734100760 msgr2=0x7f2734100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:18.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.723+0000 7f273a9f4640 1 --2- 172.21.15.33:0/124024002 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2734100760 0x7f2734100b80 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f2724009900 tx=0x7f272402bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:18.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.723+0000 7f273a9f4640 1 -- 172.21.15.33:0/124024002 shutdown_connections 2024-03-19T02:53:18.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.723+0000 7f273a9f4640 1 --2- 172.21.15.33:0/124024002 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2734102310 0x7f2734108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.723+0000 7f273a9f4640 1 --2- 172.21.15.33:0/124024002 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2734101950 0x7f2734101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.723+0000 7f273a9f4640 1 --2- 172.21.15.33:0/124024002 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2734100760 0x7f2734100b80 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.723+0000 7f273a9f4640 1 -- 172.21.15.33:0/124024002 >> 172.21.15.33:0/124024002 conn(0x7f27340fbcd0 msgr2=0x7f27340fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:18.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.723+0000 7f273a9f4640 1 -- 172.21.15.33:0/124024002 shutdown_connections 2024-03-19T02:53:18.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.723+0000 7f273a9f4640 1 -- 172.21.15.33:0/124024002 wait complete. 2024-03-19T02:53:18.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.724+0000 7f273a9f4640 1 Processor -- start 2024-03-19T02:53:18.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.724+0000 7f273a9f4640 1 -- start start 2024-03-19T02:53:18.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.724+0000 7f273a9f4640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2734100760 0x7f27341046f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.724+0000 7f273a9f4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2734101950 0x7f2734102d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.724+0000 7f273a9f4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2734102310 0x7f2734103270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.724+0000 7f273a9f4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f273410c290 con 0x7f2734102310 2024-03-19T02:53:18.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.724+0000 7f273a9f4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f273410c110 con 0x7f2734101950 2024-03-19T02:53:18.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.724+0000 7f273a9f4640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f273410c410 con 0x7f2734100760 2024-03-19T02:53:18.727 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.724+0000 7f27337fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2734101950 0x7f2734102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.724+0000 7f2738f91640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2734102310 0x7f2734103270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.724+0000 7f2733fff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2734100760 0x7f27341046f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.724+0000 7f27337fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2734101950 0x7f2734102d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:35558/0 (socket says 172.21.15.33:35558) 2024-03-19T02:53:18.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.724+0000 7f2738f91640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2734102310 0x7f2734103270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46942/0 (socket says 172.21.15.33:46942) 2024-03-19T02:53:18.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.724+0000 7f27337fe640 1 -- 172.21.15.33:0/546545691 learned_addr learned my addr 172.21.15.33:0/546545691 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:18.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.725+0000 7f27337fe640 1 -- 172.21.15.33:0/546545691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2734100760 msgr2=0x7f27341046f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:18.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.725+0000 7f27337fe640 1 --2- 172.21.15.33:0/546545691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2734100760 0x7f27341046f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.725+0000 7f27337fe640 1 -- 172.21.15.33:0/546545691 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2734102310 msgr2=0x7f2734103270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:18.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.725+0000 7f27337fe640 1 --2- 172.21.15.33:0/546545691 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2734102310 0x7f2734103270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.725+0000 7f27337fe640 1 -- 172.21.15.33:0/546545691 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2734103b00 con 0x7f2734101950 2024-03-19T02:53:18.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.725+0000 7f27337fe640 1 --2- 172.21.15.33:0/546545691 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2734101950 0x7f2734102d30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f272800e9f0 tx=0x7f272800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:18.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.725+0000 7f27317fa640 1 -- 172.21.15.33:0/546545691 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f272800cd80 con 0x7f2734101950 2024-03-19T02:53:18.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.725+0000 7f273a9f4640 1 -- 172.21.15.33:0/546545691 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27341a1b60 con 0x7f2734101950 2024-03-19T02:53:18.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.725+0000 7f27317fa640 1 -- 172.21.15.33:0/546545691 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f27280108e0 con 0x7f2734101950 2024-03-19T02:53:18.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.725+0000 7f27317fa640 1 -- 172.21.15.33:0/546545691 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2728018920 con 0x7f2734101950 2024-03-19T02:53:18.728 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.725+0000 7f273a9f4640 1 -- 172.21.15.33:0/546545691 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27341a20b0 con 0x7f2734101950 2024-03-19T02:53:18.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.725+0000 7fd6c9c9a640 1 -- 172.21.15.33:0/2653027708 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6c4100120 msgr2=0x7fd6c4100540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:18.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.725+0000 7fd6c9c9a640 1 --2- 172.21.15.33:0/2653027708 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6c4100120 0x7fd6c4100540 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4009900 tx=0x7fd6b402bcd0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.729 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.725+0000 7fd6c9c9a640 1 -- 172.21.15.33:0/2653027708 shutdown_connections 2024-03-19T02:53:18.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.725+0000 7fd6c9c9a640 1 --2- 172.21.15.33:0/2653027708 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6c4101420 0x7fd6c41018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.725+0000 7fd6c9c9a640 1 --2- 172.21.15.33:0/2653027708 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6c4100a80 0x7fd6c4100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.725+0000 7fd6c9c9a640 1 --2- 172.21.15.33:0/2653027708 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6c4100120 0x7fd6c4100540 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.726+0000 7fd6c9c9a640 1 -- 172.21.15.33:0/2653027708 >> 172.21.15.33:0/2653027708 conn(0x7fd6c40fbcc0 msgr2=0x7fd6c40fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:18.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.726+0000 7fd6c9c9a640 1 -- 172.21.15.33:0/2653027708 shutdown_connections 2024-03-19T02:53:18.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.726+0000 7fd6c9c9a640 1 -- 172.21.15.33:0/2653027708 wait complete. 2024-03-19T02:53:18.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.726+0000 7f273a9f4640 1 -- 172.21.15.33:0/546545691 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27341a2280 con 0x7f2734101950 2024-03-19T02:53:18.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.726+0000 7fd6c9c9a640 1 Processor -- start 2024-03-19T02:53:18.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.726+0000 7fd6c9c9a640 1 -- start start 2024-03-19T02:53:18.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.726+0000 7fd6c9c9a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6c4100120 0x7fd6c4199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.726+0000 7fd6c9c9a640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6c4100a80 0x7fd6c419a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.726+0000 7fd6c9c9a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6c4101420 0x7fd6c41a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.727+0000 7fd6c9c9a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c410f5b0 con 0x7fd6c4100120 2024-03-19T02:53:18.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.727+0000 7fd6c9c9a640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c410f430 con 0x7fd6c4101420 2024-03-19T02:53:18.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.727+0000 7fd6c9c9a640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c410f730 con 0x7fd6c4100a80 2024-03-19T02:53:18.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.727+0000 7fd6c37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6c4100120 0x7fd6c4199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.727+0000 7fd6c37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6c4100120 0x7fd6c4199cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46956/0 (socket says 172.21.15.33:46956) 2024-03-19T02:53:18.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.727+0000 7fd6c37fe640 1 -- 172.21.15.33:0/3831179530 learned_addr learned my addr 172.21.15.33:0/3831179530 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:18.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.727+0000 7fd6c3fff640 1 --2- 172.21.15.33:0/3831179530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6c4101420 0x7fd6c41a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.727+0000 7fd6c2ffd640 1 --2- 172.21.15.33:0/3831179530 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6c4100a80 0x7fd6c419a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.727+0000 7fd6c37fe640 1 -- 172.21.15.33:0/3831179530 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6c4100a80 msgr2=0x7fd6c419a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:18.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.727+0000 7fd6c37fe640 1 --2- 172.21.15.33:0/3831179530 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6c4100a80 0x7fd6c419a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.727+0000 7fd6c37fe640 1 -- 172.21.15.33:0/3831179530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6c4101420 msgr2=0x7fd6c41a1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:18.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.727+0000 7fd6c37fe640 1 --2- 172.21.15.33:0/3831179530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6c4101420 0x7fd6c41a1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.727+0000 7fd6c37fe640 1 -- 172.21.15.33:0/3831179530 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6c41a18f0 con 0x7fd6c4100120 2024-03-19T02:53:18.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.727+0000 7fd6c37fe640 1 --2- 172.21.15.33:0/3831179530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6c4100120 0x7fd6c4199cb0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4000c00 tx=0x7fd6b4005e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:18.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.727+0000 7f27317fa640 1 -- 172.21.15.33:0/546545691 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2728010a90 con 0x7f2734101950 2024-03-19T02:53:18.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.727+0000 7fd6c0ff9640 1 -- 172.21.15.33:0/3831179530 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b4043020 con 0x7fd6c4100120 2024-03-19T02:53:18.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.727+0000 7fd6c0ff9640 1 -- 172.21.15.33:0/3831179530 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd6b402dd20 con 0x7fd6c4100120 2024-03-19T02:53:18.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.727+0000 7fd6c9c9a640 1 -- 172.21.15.33:0/3831179530 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6c41a1b80 con 0x7fd6c4100120 2024-03-19T02:53:18.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.727+0000 7fd6c0ff9640 1 -- 172.21.15.33:0/3831179530 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b4003b40 con 0x7fd6c4100120 2024-03-19T02:53:18.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.727+0000 7fd6c9c9a640 1 -- 172.21.15.33:0/3831179530 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6c41a20d0 con 0x7fd6c4100120 2024-03-19T02:53:18.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.728+0000 7f27317fa640 1 --2- 172.21.15.33:0/546545691 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f271c0a28a0 0x7f271c0a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.728+0000 7f2733fff640 1 --2- 172.21.15.33:0/546545691 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f271c0a28a0 0x7f271c0a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.728+0000 7f27317fa640 1 -- 172.21.15.33:0/546545691 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f27280cf0a0 con 0x7f2734101950 2024-03-19T02:53:18.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.728+0000 7f2733fff640 1 --2- 172.21.15.33:0/546545691 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f271c0a28a0 0x7f271c0a4d60 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2724005ab0 tx=0x7f2724005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:18.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.730+0000 7fd6c0ff9640 1 -- 172.21.15.33:0/3831179530 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd6b4046020 con 0x7fd6c4100120 2024-03-19T02:53:18.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.730+0000 7fd6cc965740 1 Processor -- start 2024-03-19T02:53:18.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.730+0000 7fd6c0ff9640 1 --2- 172.21.15.33:0/3831179530 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fd6ac0a2800 0x7fd6ac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.730+0000 7fd6cc965740 1 -- start start 2024-03-19T02:53:18.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.730+0000 7fd6c3fff640 1 --2- 172.21.15.33:0/3831179530 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fd6ac0a2800 0x7fd6ac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.730+0000 7fd6cc965740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562999d63a0 0x5562999d67c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.730+0000 7fd6cc965740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562999d6d90 0x5562999db890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.730+0000 7fd6cc965740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562999dbf00 0x5562999de390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.730+0000 7fd6c0ff9640 1 -- 172.21.15.33:0/3831179530 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7fd6b40f0eb0 con 0x7fd6c4100120 2024-03-19T02:53:18.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.730+0000 7fd6cc965740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55629993b160 con 0x5562999d6d90 2024-03-19T02:53:18.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.730+0000 7fd6cc965740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5562998e3e10 con 0x5562999dbf00 2024-03-19T02:53:18.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.730+0000 7fd6cc965740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556299937200 con 0x5562999d63a0 2024-03-19T02:53:18.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.730+0000 7fd6c2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562999d6d90 0x5562999db890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.730+0000 7fd6c2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562999d6d90 0x5562999db890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46964/0 (socket says 172.21.15.33:46964) 2024-03-19T02:53:18.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.730+0000 7fd6c2ffd640 1 -- 172.21.15.33:0/1862700573 learned_addr learned my addr 172.21.15.33:0/1862700573 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:18.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.730+0000 7fd6c3fff640 1 --2- 172.21.15.33:0/1862700573 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562999dbf00 0x5562999de390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.730+0000 7fd6c37fe640 1 --2- 172.21.15.33:0/1862700573 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562999d63a0 0x5562999d67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.731+0000 7fd6c2ffd640 1 -- 172.21.15.33:0/1862700573 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562999d63a0 msgr2=0x5562999d67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:18.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.731+0000 7fd6c2ffd640 1 --2- 172.21.15.33:0/1862700573 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562999d63a0 0x5562999d67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.731+0000 7fd6c2ffd640 1 -- 172.21.15.33:0/1862700573 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562999dbf00 msgr2=0x5562999de390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:18.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.731+0000 7fd6c2ffd640 1 --2- 172.21.15.33:0/1862700573 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562999dbf00 0x5562999de390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.731+0000 7fd6c2ffd640 1 -- 172.21.15.33:0/1862700573 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562997177e0 con 0x5562999d6d90 2024-03-19T02:53:18.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.731+0000 7fd6c3fff640 1 --2- 172.21.15.33:0/3831179530 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fd6ac0a2800 0x7fd6ac0a4cc0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc005fd0 tx=0x7fd6bc011bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:18.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.731+0000 7fd6c2ffd640 1 --2- 172.21.15.33:0/1862700573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562999d6d90 0x5562999db890 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8009ca0 tx=0x7fd6b8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8940c937d3eddd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:18.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.731+0000 7fd6a9ffb640 1 -- 172.21.15.33:0/1862700573 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b801e070 con 0x5562999d6d90 2024-03-19T02:53:18.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.731+0000 7f27317fa640 1 -- 172.21.15.33:0/546545691 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f272809b760 con 0x7f2734101950 2024-03-19T02:53:18.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.731+0000 7fd6a9ffb640 1 -- 172.21.15.33:0/1862700573 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd6b8009e50 con 0x5562999d6d90 2024-03-19T02:53:18.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.731+0000 7fd6a9ffb640 1 -- 172.21.15.33:0/1862700573 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b8018730 con 0x5562999d6d90 2024-03-19T02:53:18.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.732+0000 7fd6cc965740 1 -- 172.21.15.33:0/1862700573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562999d6d90 msgr2=0x5562999db890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:18.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.732+0000 7fd6cc965740 1 --2- 172.21.15.33:0/1862700573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562999d6d90 0x5562999db890 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8009ca0 tx=0x7fd6b8011810 comp rx=0 tx=0).stop 2024-03-19T02:53:18.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.732+0000 7fd6cc965740 1 -- 172.21.15.33:0/1862700573 shutdown_connections 2024-03-19T02:53:18.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.732+0000 7fd6cc965740 1 --2- 172.21.15.33:0/1862700573 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562999dbf00 0x5562999de390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.732+0000 7fd6cc965740 1 --2- 172.21.15.33:0/1862700573 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562999d6d90 0x5562999db890 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.732+0000 7fd6cc965740 1 --2- 172.21.15.33:0/1862700573 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562999d63a0 0x5562999d67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.732+0000 7fd6cc965740 1 -- 172.21.15.33:0/1862700573 >> 172.21.15.33:0/1862700573 conn(0x5562999d1910 msgr2=0x5562999d1d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:18.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.732+0000 7fd6cc965740 1 -- 172.21.15.33:0/1862700573 shutdown_connections 2024-03-19T02:53:18.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.732+0000 7fd6cc965740 1 -- 172.21.15.33:0/1862700573 wait complete. 2024-03-19T02:53:18.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.732+0000 7fd6cc965740 1 -- start start 2024-03-19T02:53:18.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.732+0000 7fd6cc965740 1 Processor -- start 2024-03-19T02:53:18.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.733+0000 7fd6cc965740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562999d63a0 0x5562999e0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.733+0000 7fd6cc965740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562999d6d90 0x5562999e0d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.733+0000 7fd6cc965740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562999dbf00 0x5562999e7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.733+0000 7fd6cc965740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5562999deeb0 con 0x5562999dbf00 2024-03-19T02:53:18.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.733+0000 7fd6cc965740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5562998f71a0 con 0x5562999d63a0 2024-03-19T02:53:18.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.733+0000 7fd6cc965740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5562999df030 con 0x5562999d6d90 2024-03-19T02:53:18.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.733+0000 7fd6cc965740 10 client.0 _ll_register_callbacks cb 0x55629964ec60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:53:18.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.733+0000 7fd6c37fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562999d63a0 0x5562999e0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.733+0000 7fd6c3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562999dbf00 0x5562999e7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.733+0000 7fd6c37fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562999d63a0 0x5562999e0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:35588/0 (socket says 172.21.15.33:35588) 2024-03-19T02:53:18.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.733+0000 7fd6c37fe640 1 -- 172.21.15.33:0/2273078783 learned_addr learned my addr 172.21.15.33:0/2273078783 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:18.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.733+0000 7fd6c3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562999dbf00 0x5562999e7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46970/0 (socket says 172.21.15.33:46970) 2024-03-19T02:53:18.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.733+0000 7fd6c2ffd640 1 --2- 172.21.15.33:0/2273078783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562999d6d90 0x5562999e0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.733+0000 7fd6c37fe640 1 -- 172.21.15.33:0/2273078783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562999d6d90 msgr2=0x5562999e0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:18.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.733+0000 7fd6c37fe640 1 --2- 172.21.15.33:0/2273078783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562999d6d90 0x5562999e0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.733+0000 7fd6c37fe640 1 -- 172.21.15.33:0/2273078783 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562999dbf00 msgr2=0x5562999e7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:18.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.733+0000 7fd6c37fe640 1 --2- 172.21.15.33:0/2273078783 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562999dbf00 0x5562999e7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.733+0000 7fd6c37fe640 1 -- 172.21.15.33:0/2273078783 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556299777f70 con 0x5562999d63a0 2024-03-19T02:53:18.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.733+0000 7fd6c3fff640 1 --2- 172.21.15.33:0/2273078783 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562999dbf00 0x5562999e7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:18.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.733+0000 7fd6c37fe640 1 --2- 172.21.15.33:0/2273078783 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562999d63a0 0x5562999e0830 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd6b40bd1c0 tx=0x7fd6b4009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dac7d4f17fbe9336 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:18.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.733+0000 7fd6a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T02:53:18.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.733+0000 7fd6a2ffd640 1 -- 172.21.15.33:0/2273078783 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b40d5020 con 0x5562999d63a0 2024-03-19T02:53:18.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.733+0000 7fd6a2ffd640 1 -- 172.21.15.33:0/2273078783 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7fd6b40b9b30 con 0x5562999d63a0 2024-03-19T02:53:18.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.733+0000 7fd6cc965740 1 -- 172.21.15.33:0/2273078783 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6b8009e50 con 0x5562999d63a0 2024-03-19T02:53:18.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.734+0000 7fd6a2ffd640 1 -- 172.21.15.33:0/2273078783 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd6b40c1610 con 0x5562999d63a0 2024-03-19T02:53:18.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.734+0000 7fd6cc965740 10 client.25448 fetch_fsmap learned FSMap version 25 2024-03-19T02:53:18.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.734+0000 7fd6cc965740 1 -- 172.21.15.33:0/2273078783 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55629960f170 con 0x5562999d63a0 2024-03-19T02:53:18.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.734+0000 7fd6a2ffd640 1 -- 172.21.15.33:0/2273078783 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd6b8009e50 con 0x5562999d63a0 2024-03-19T02:53:18.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.734+0000 7fd6a2ffd640 1 -- 172.21.15.33:0/2273078783 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fd6b40c1170 con 0x5562999d63a0 2024-03-19T02:53:18.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.734+0000 7fd6cc965740 10 client.25448 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:53:18.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.734+0000 7fd6cc965740 10 client.25448 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:53:18.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.734+0000 7fd6cc965740 20 client.25448 populate_metadata read hostname 'smithi033' 2024-03-19T02:53:18.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.734+0000 7fd6cc965740 1 --2- 172.21.15.33:0/2273078783 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5562999ec5c0 0x5562999eea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.734+0000 7fd6cc965740 4 client.25448 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:53:18.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.734+0000 7fd6cc965740 1 -- 172.21.15.33:0/2273078783 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5562999486a0 con 0x5562999ec5c0 2024-03-19T02:53:18.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.734+0000 7fd6c2ffd640 1 --2- 172.21.15.33:0/2273078783 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5562999ec5c0 0x5562999eea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.735+0000 7fd6c2ffd640 1 --2- 172.21.15.33:0/2273078783 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5562999ec5c0 0x5562999eea30 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f21d73d30ecbfce server_cookie=5e665cfaab2cb8fc in_seq=0 out_seq=0 2024-03-19T02:53:18.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.735+0000 7fd6a2ffd640 10 client.25448 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:53:18.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.735+0000 7fd6a2ffd640 1 -- 172.21.15.33:0/2273078783 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5562999486a0 con 0x5562999ec5c0 2024-03-19T02:53:18.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.735+0000 7fd6a2ffd640 10 client.25448 handle_command_reply: tid=0 2024-03-19T02:53:18.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.735+0000 7fd6cc965740 1 client.25448 shutdown 2024-03-19T02:53:18.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.735+0000 7fd6cc965740 1 -- 172.21.15.33:0/2273078783 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562999d63a0 msgr2=0x5562999e0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:18.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.735+0000 7fd6cc965740 1 --2- 172.21.15.33:0/2273078783 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562999d63a0 0x5562999e0830 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd6b40bd1c0 tx=0x7fd6b4009260 comp rx=0 tx=0).stop 2024-03-19T02:53:18.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.735+0000 7fd6cc965740 1 -- 172.21.15.33:0/2273078783 shutdown_connections 2024-03-19T02:53:18.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.735+0000 7fd6cc965740 1 --2- 172.21.15.33:0/2273078783 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5562999ec5c0 0x5562999eea30 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.735+0000 7fd6cc965740 1 --2- 172.21.15.33:0/2273078783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562999d6d90 0x5562999e0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.735+0000 7fd6cc965740 1 --2- 172.21.15.33:0/2273078783 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562999dbf00 0x5562999e7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.735+0000 7fd6a2ffd640 0 client.25448 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:53:18.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.735+0000 7fd6cc965740 1 --2- 172.21.15.33:0/2273078783 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562999d63a0 0x5562999e0830 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.735+0000 7fd6cc965740 1 -- 172.21.15.33:0/2273078783 >> 172.21.15.33:0/2273078783 conn(0x5562999d1910 msgr2=0x5562999d0b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:18.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.736+0000 7fd6cc965740 1 -- 172.21.15.33:0/2273078783 shutdown_connections 2024-03-19T02:53:18.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.736+0000 7fd6cc965740 1 -- 172.21.15.33:0/2273078783 wait complete. 2024-03-19T02:53:18.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.736+0000 7fd6cc965740 20 client.25448 trim_cache size 0 max 16384 2024-03-19T02:53:18.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.753+0000 7fd6cc965740 1 Processor -- start 2024-03-19T02:53:18.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.753+0000 7fd6cc965740 1 -- start start 2024-03-19T02:53:18.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.753+0000 7fd6cc965740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562999d63a0 0x5562999e0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.753+0000 7fd6cc965740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562999d6d90 0x5562999e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.753+0000 7fd6cc965740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562999dbf00 0x5562999d5280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.753+0000 7fd6cc965740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c41aed80 con 0x5562999d63a0 2024-03-19T02:53:18.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.753+0000 7fd6cc965740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c41ac650 con 0x5562999dbf00 2024-03-19T02:53:18.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.753+0000 7fd6cc965740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c41aa4a0 con 0x5562999d6d90 2024-03-19T02:53:18.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.754+0000 7fd6c37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562999d63a0 0x5562999e0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.754+0000 7fd6c2ffd640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562999d6d90 0x5562999e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.754+0000 7fd6c37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562999d63a0 0x5562999e0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46974/0 (socket says 172.21.15.33:46974) 2024-03-19T02:53:18.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.754+0000 7fd6c3fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562999dbf00 0x5562999d5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.754+0000 7fd6c37fe640 1 -- 172.21.15.33:0/3537669640 learned_addr learned my addr 172.21.15.33:0/3537669640 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:18.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.754+0000 7fd6c37fe640 1 -- 172.21.15.33:0/3537669640 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562999d6d90 msgr2=0x5562999e7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:18.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.754+0000 7fd6c37fe640 1 --2- 172.21.15.33:0/3537669640 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562999d6d90 0x5562999e7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.754+0000 7fd6c37fe640 1 -- 172.21.15.33:0/3537669640 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562999dbf00 msgr2=0x5562999d5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:18.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.754+0000 7fd6c37fe640 1 --2- 172.21.15.33:0/3537669640 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562999dbf00 0x5562999d5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.754+0000 7fd6c37fe640 1 -- 172.21.15.33:0/3537669640 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562999d14b0 con 0x5562999d63a0 2024-03-19T02:53:18.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.754+0000 7fd6c2ffd640 1 --2- 172.21.15.33:0/3537669640 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562999d6d90 0x5562999e7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:18.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.754+0000 7fd6c3fff640 1 --2- 172.21.15.33:0/3537669640 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562999dbf00 0x5562999d5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:18.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.754+0000 7fd6c37fe640 1 --2- 172.21.15.33:0/3537669640 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562999d63a0 0x5562999e0cd0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd6b40c1580 tx=0x7fd6b40bb9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb26b774e914b3fb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:18.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.754+0000 7fd6a97fa640 1 -- 172.21.15.33:0/3537669640 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b40d5070 con 0x5562999d63a0 2024-03-19T02:53:18.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.754+0000 7fd6a97fa640 1 -- 172.21.15.33:0/3537669640 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd6b40d6220 con 0x5562999d63a0 2024-03-19T02:53:18.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.754+0000 7fd6a97fa640 1 -- 172.21.15.33:0/3537669640 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b40c1170 con 0x5562999d63a0 2024-03-19T02:53:18.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.755+0000 7fd6cc965740 1 -- 172.21.15.33:0/3537669640 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562999d63a0 msgr2=0x5562999e0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:18.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.755+0000 7fd6cc965740 1 --2- 172.21.15.33:0/3537669640 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562999d63a0 0x5562999e0cd0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd6b40c1580 tx=0x7fd6b40bb9d0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.755+0000 7fd6cc965740 1 -- 172.21.15.33:0/3537669640 shutdown_connections 2024-03-19T02:53:18.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.755+0000 7fd6cc965740 1 --2- 172.21.15.33:0/3537669640 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562999d6d90 0x5562999e7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.755+0000 7fd6cc965740 1 --2- 172.21.15.33:0/3537669640 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562999dbf00 0x5562999d5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.755+0000 7fd6cc965740 1 --2- 172.21.15.33:0/3537669640 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562999d63a0 0x5562999e0cd0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.755+0000 7fd6cc965740 1 -- 172.21.15.33:0/3537669640 >> 172.21.15.33:0/3537669640 conn(0x5562999d1910 msgr2=0x5562999d4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:18.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.755+0000 7fd6cc965740 1 -- 172.21.15.33:0/3537669640 shutdown_connections 2024-03-19T02:53:18.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.755+0000 7fd6cc965740 1 -- 172.21.15.33:0/3537669640 wait complete. 2024-03-19T02:53:18.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.755+0000 7fd6cc965740 1 -- start start 2024-03-19T02:53:18.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.756+0000 7fd6cc965740 1 Processor -- start 2024-03-19T02:53:18.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.756+0000 7fd6cc965740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562999d63a0 0x5562999e5d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.756+0000 7fd6cc965740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562999d6d90 0x5562999e6260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.756+0000 7fd6cc965740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562999dbf00 0x5562999e7270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.756+0000 7fd6cc965740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c41aab10 con 0x5562999dbf00 2024-03-19T02:53:18.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.756+0000 7fd6cc965740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c41abf80 con 0x5562999d63a0 2024-03-19T02:53:18.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.756+0000 7fd6cc965740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c41ab1e0 con 0x5562999d6d90 2024-03-19T02:53:18.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.756+0000 7fd6cc965740 10 client.0 _ll_register_callbacks cb 0x7fd6b40b9760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:53:18.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.756+0000 7fd6c3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562999dbf00 0x5562999e7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.756+0000 7fd6c37fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562999d63a0 0x5562999e5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.756+0000 7fd6c3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562999dbf00 0x5562999e7270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46978/0 (socket says 172.21.15.33:46978) 2024-03-19T02:53:18.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.756+0000 7fd6c37fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562999d63a0 0x5562999e5d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:35600/0 (socket says 172.21.15.33:35600) 2024-03-19T02:53:18.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.756+0000 7fd6c3fff640 1 -- 172.21.15.33:0/1400225590 learned_addr learned my addr 172.21.15.33:0/1400225590 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:18.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.756+0000 7fd6c2ffd640 1 --2- 172.21.15.33:0/1400225590 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562999d6d90 0x5562999e6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.756+0000 7fd6c3fff640 1 -- 172.21.15.33:0/1400225590 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562999d6d90 msgr2=0x5562999e6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:18.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.756+0000 7fd6c3fff640 1 --2- 172.21.15.33:0/1400225590 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562999d6d90 0x5562999e6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.756+0000 7fd6c3fff640 1 -- 172.21.15.33:0/1400225590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562999d63a0 msgr2=0x5562999e5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:18.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.756+0000 7fd6c3fff640 1 --2- 172.21.15.33:0/1400225590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562999d63a0 0x5562999e5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.756+0000 7fd6c3fff640 1 -- 172.21.15.33:0/1400225590 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562999e7860 con 0x5562999dbf00 2024-03-19T02:53:18.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.756+0000 7fd6c37fe640 1 --2- 172.21.15.33:0/1400225590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562999d63a0 0x5562999e5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:18.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.756+0000 7fd6c2ffd640 1 --2- 172.21.15.33:0/1400225590 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562999d6d90 0x5562999e6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:18.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.756+0000 7fd6c3fff640 1 --2- 172.21.15.33:0/1400225590 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562999dbf00 0x5562999e7270 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc020920 tx=0x7fd6bc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64e3d3bfbf37e396 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:18.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.756+0000 7fd6a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:53:18.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.757+0000 7fd6a2ffd640 1 -- 172.21.15.33:0/1400225590 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc02f060 con 0x5562999dbf00 2024-03-19T02:53:18.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.757+0000 7fd6a2ffd640 1 -- 172.21.15.33:0/1400225590 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7fd6bc003970 con 0x5562999dbf00 2024-03-19T02:53:18.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.757+0000 7fd6cc965740 1 -- 172.21.15.33:0/1400225590 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6b40d6220 con 0x5562999dbf00 2024-03-19T02:53:18.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.757+0000 7fd6a2ffd640 1 -- 172.21.15.33:0/1400225590 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd6bc036050 con 0x5562999dbf00 2024-03-19T02:53:18.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.757+0000 7fd6cc965740 10 client.15582 fetch_fsmap learned FSMap version 25 2024-03-19T02:53:18.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.757+0000 7fd6cc965740 1 -- 172.21.15.33:0/1400225590 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x556299a53260 con 0x5562999dbf00 2024-03-19T02:53:18.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.757+0000 7fd6a2ffd640 1 -- 172.21.15.33:0/1400225590 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd6b40d6220 con 0x5562999dbf00 2024-03-19T02:53:18.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.757+0000 7fd6a2ffd640 1 -- 172.21.15.33:0/1400225590 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fd6bc01d030 con 0x5562999dbf00 2024-03-19T02:53:18.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.757+0000 7fd6cc965740 10 client.15582 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:53:18.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.757+0000 7fd6cc965740 10 client.15582 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:53:18.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.757+0000 7fd6cc965740 20 client.15582 populate_metadata read hostname 'smithi033' 2024-03-19T02:53:18.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.757+0000 7fd6cc965740 1 --2- 172.21.15.33:0/1400225590 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x556299a53cc0 0x556299a560e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.757+0000 7fd6cc965740 4 client.15582 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:53:18.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.757+0000 7fd6cc965740 1 -- 172.21.15.33:0/1400225590 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5562999429f0 con 0x556299a53cc0 2024-03-19T02:53:18.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.757+0000 7fd6c37fe640 1 --2- 172.21.15.33:0/1400225590 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x556299a53cc0 0x556299a560e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.758+0000 7fd6a2ffd640 10 client.15582 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:53:18.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.758+0000 7fd6c37fe640 1 --2- 172.21.15.33:0/1400225590 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x556299a53cc0 0x556299a560e0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cf4e1d8ae5dc045 server_cookie=eb6e09fe5cefe2b2 in_seq=0 out_seq=0 2024-03-19T02:53:18.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.759+0000 7fd6a2ffd640 1 -- 172.21.15.33:0/1400225590 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5562999429f0 con 0x556299a53cc0 2024-03-19T02:53:18.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.759+0000 7fd6a2ffd640 10 client.15582 handle_command_reply: tid=0 2024-03-19T02:53:18.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.759+0000 7fd6cc965740 1 client.15582 shutdown 2024-03-19T02:53:18.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.759+0000 7fd6cc965740 1 -- 172.21.15.33:0/1400225590 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562999dbf00 msgr2=0x5562999e7270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:18.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.759+0000 7fd6cc965740 1 --2- 172.21.15.33:0/1400225590 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562999dbf00 0x5562999e7270 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc020920 tx=0x7fd6bc004ab0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.759+0000 7fd6cc965740 1 -- 172.21.15.33:0/1400225590 shutdown_connections 2024-03-19T02:53:18.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.759+0000 7fd6cc965740 1 --2- 172.21.15.33:0/1400225590 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x556299a53cc0 0x556299a560e0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.759+0000 7fd6cc965740 1 --2- 172.21.15.33:0/1400225590 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562999d6d90 0x5562999e6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.759+0000 7fd6cc965740 1 --2- 172.21.15.33:0/1400225590 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562999dbf00 0x5562999e7270 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.759+0000 7fd6cc965740 1 --2- 172.21.15.33:0/1400225590 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562999d63a0 0x5562999e5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.759+0000 7fd6cc965740 1 -- 172.21.15.33:0/1400225590 >> 172.21.15.33:0/1400225590 conn(0x5562999d1910 msgr2=0x5562999d5400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:18.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.759+0000 7fd6a2ffd640 0 client.15582 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:53:18.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.759+0000 7fd6cc965740 1 -- 172.21.15.33:0/1400225590 shutdown_connections 2024-03-19T02:53:18.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.759+0000 7fd6cc965740 1 -- 172.21.15.33:0/1400225590 wait complete. 2024-03-19T02:53:18.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.759+0000 7fd6cc965740 20 client.15582 trim_cache size 0 max 16384 2024-03-19T02:53:18.766 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:53:18.766 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-03-19T02:53:18.766 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "9b3ffdd5-6bb0-4e46-a0ab-38c3bf3f5c3d", 2024-03-19T02:53:18.766 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-03-19T02:53:18.766 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:53:18.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.760+0000 7fd6c9c9a640 1 -- 172.21.15.33:0/3831179530 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fd6ac0a2800 msgr2=0x7fd6ac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:18.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.760+0000 7fd6c9c9a640 1 --2- 172.21.15.33:0/3831179530 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fd6ac0a2800 0x7fd6ac0a4cc0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc005fd0 tx=0x7fd6bc011bc0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.760+0000 7fd6c9c9a640 1 -- 172.21.15.33:0/3831179530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6c4100120 msgr2=0x7fd6c4199cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:18.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.760+0000 7fd6c9c9a640 1 --2- 172.21.15.33:0/3831179530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6c4100120 0x7fd6c4199cb0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4000c00 tx=0x7fd6b4005e30 comp rx=0 tx=0).stop 2024-03-19T02:53:18.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.760+0000 7fd6c9c9a640 1 -- 172.21.15.33:0/3831179530 shutdown_connections 2024-03-19T02:53:18.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.760+0000 7fd6c9c9a640 1 --2- 172.21.15.33:0/3831179530 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fd6ac0a2800 0x7fd6ac0a4cc0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.760+0000 7fd6c9c9a640 1 --2- 172.21.15.33:0/3831179530 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6c4100a80 0x7fd6c419a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.760+0000 7fd6c9c9a640 1 --2- 172.21.15.33:0/3831179530 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd6c4101420 0x7fd6c41a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.760+0000 7fd6c9c9a640 1 --2- 172.21.15.33:0/3831179530 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd6c4100120 0x7fd6c4199cb0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.760+0000 7fd6c9c9a640 1 -- 172.21.15.33:0/3831179530 >> 172.21.15.33:0/3831179530 conn(0x7fd6c40fbcc0 msgr2=0x7fd6c40fd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:18.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.760+0000 7fd6c9c9a640 1 -- 172.21.15.33:0/3831179530 shutdown_connections 2024-03-19T02:53:18.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.760+0000 7fd6c9c9a640 1 -- 172.21.15.33:0/3831179530 wait complete. 2024-03-19T02:53:18.772 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:53:18.772 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:53:18.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.878+0000 7fd593c97640 1 -- 172.21.15.33:0/397757394 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd58c100720 msgr2=0x7fd58c100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:18.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.878+0000 7fd593c97640 1 --2- 172.21.15.33:0/397757394 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd58c100720 0x7fd58c100b40 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd588009900 tx=0x7fd58802bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:18.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.878+0000 7fd593c97640 1 -- 172.21.15.33:0/397757394 shutdown_connections 2024-03-19T02:53:18.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.878+0000 7fd593c97640 1 --2- 172.21.15.33:0/397757394 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd58c1022d0 0x7fd58c108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.878+0000 7fd593c97640 1 --2- 172.21.15.33:0/397757394 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd58c101910 0x7fd58c101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.878+0000 7fd593c97640 1 --2- 172.21.15.33:0/397757394 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd58c100720 0x7fd58c100b40 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.878+0000 7fd593c97640 1 -- 172.21.15.33:0/397757394 >> 172.21.15.33:0/397757394 conn(0x7fd58c0fbcb0 msgr2=0x7fd58c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:18.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.878+0000 7fd593c97640 1 -- 172.21.15.33:0/397757394 shutdown_connections 2024-03-19T02:53:18.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.878+0000 7fd593c97640 1 -- 172.21.15.33:0/397757394 wait complete. 2024-03-19T02:53:18.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.879+0000 7fd593c97640 1 Processor -- start 2024-03-19T02:53:18.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.879+0000 7fd593c97640 1 -- start start 2024-03-19T02:53:18.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.879+0000 7fd593c97640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd58c100720 0x7fd58c1969d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.879+0000 7fd593c97640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd58c101910 0x7fd58c196f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.879+0000 7fd593c97640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd58c1022d0 0x7fd58c19df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.879+0000 7fd593c97640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c10c220 con 0x7fd58c100720 2024-03-19T02:53:18.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.879+0000 7fd593c97640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c10c0a0 con 0x7fd58c101910 2024-03-19T02:53:18.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.879+0000 7fd593c97640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c10c3a0 con 0x7fd58c1022d0 2024-03-19T02:53:18.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.879+0000 7fd591a33640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd58c100720 0x7fd58c1969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.879+0000 7fd591a33640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd58c100720 0x7fd58c1969d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46988/0 (socket says 172.21.15.33:46988) 2024-03-19T02:53:18.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.879+0000 7fd591a33640 1 -- 172.21.15.33:0/2203227149 learned_addr learned my addr 172.21.15.33:0/2203227149 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:18.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.879+0000 7fd591232640 1 --2- 172.21.15.33:0/2203227149 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd58c101910 0x7fd58c196f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.879+0000 7fd592234640 1 --2- 172.21.15.33:0/2203227149 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd58c1022d0 0x7fd58c19df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.879+0000 7fd591a33640 1 -- 172.21.15.33:0/2203227149 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd58c1022d0 msgr2=0x7fd58c19df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:18.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.879+0000 7fd591a33640 1 --2- 172.21.15.33:0/2203227149 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd58c1022d0 0x7fd58c19df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.879+0000 7fd591a33640 1 -- 172.21.15.33:0/2203227149 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd58c101910 msgr2=0x7fd58c196f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:18.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.879+0000 7fd591a33640 1 --2- 172.21.15.33:0/2203227149 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd58c101910 0x7fd58c196f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.879+0000 7fd591a33640 1 -- 172.21.15.33:0/2203227149 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd58c19e5f0 con 0x7fd58c100720 2024-03-19T02:53:18.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.879+0000 7fd591232640 1 --2- 172.21.15.33:0/2203227149 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd58c101910 0x7fd58c196f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:18.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.880+0000 7fd591a33640 1 --2- 172.21.15.33:0/2203227149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd58c100720 0x7fd58c1969d0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fd588000c00 tx=0x7fd588005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:18.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.880+0000 7fd582ffd640 1 -- 172.21.15.33:0/2203227149 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd588043020 con 0x7fd58c100720 2024-03-19T02:53:18.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.880+0000 7fd582ffd640 1 -- 172.21.15.33:0/2203227149 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd588004350 con 0x7fd58c100720 2024-03-19T02:53:18.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.880+0000 7fd593c97640 1 -- 172.21.15.33:0/2203227149 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd58c19e880 con 0x7fd58c100720 2024-03-19T02:53:18.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.880+0000 7fd582ffd640 1 -- 172.21.15.33:0/2203227149 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd588003b10 con 0x7fd58c100720 2024-03-19T02:53:18.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.880+0000 7fd593c97640 1 -- 172.21.15.33:0/2203227149 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd58c19edd0 con 0x7fd58c100720 2024-03-19T02:53:18.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.882+0000 7fd582ffd640 1 -- 172.21.15.33:0/2203227149 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd588046020 con 0x7fd58c100720 2024-03-19T02:53:18.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.882+0000 7fd596880740 1 Processor -- start 2024-03-19T02:53:18.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.883+0000 7fd596880740 1 -- start start 2024-03-19T02:53:18.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.883+0000 7fd582ffd640 1 --2- 172.21.15.33:0/2203227149 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fd5780a2800 0x7fd5780a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.883+0000 7fd596880740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6252a70d0 0x55f6252a5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.883+0000 7fd596880740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6252a5d60 0x55f6252a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.883+0000 7fd596880740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6252afd30 0x55f6252b2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.883+0000 7fd596880740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f62520a500 con 0x55f6252a5d60 2024-03-19T02:53:18.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.883+0000 7fd596880740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55f6251b2be0 con 0x55f6252a70d0 2024-03-19T02:53:18.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.883+0000 7fd596880740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f6252065a0 con 0x55f6252afd30 2024-03-19T02:53:18.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.883+0000 7fd592234640 1 --2- 172.21.15.33:0/2203227149 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fd5780a2800 0x7fd5780a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.883+0000 7fd591a33640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6252a70d0 0x55f6252a5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.883+0000 7fd591232640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6252a5d60 0x55f6252a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.883+0000 7fd591232640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6252a5d60 0x55f6252a61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46994/0 (socket says 172.21.15.33:46994) 2024-03-19T02:53:18.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.883+0000 7fd591a33640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6252a70d0 0x55f6252a5820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:35632/0 (socket says 172.21.15.33:35632) 2024-03-19T02:53:18.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.883+0000 7fd591232640 1 -- 172.21.15.33:0/2584380179 learned_addr learned my addr 172.21.15.33:0/2584380179 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:18.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.883+0000 7fd592234640 1 --2- 172.21.15.33:0/2584380179 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6252afd30 0x55f6252b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.883+0000 7fd582ffd640 1 -- 172.21.15.33:0/2203227149 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7fd5880f0f50 con 0x7fd58c100720 2024-03-19T02:53:18.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.883+0000 7fd591a33640 1 -- 172.21.15.33:0/2584380179 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6252afd30 msgr2=0x55f6252b2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:18.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.883+0000 7fd591a33640 1 --2- 172.21.15.33:0/2584380179 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6252afd30 0x55f6252b2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.883+0000 7fd591a33640 1 -- 172.21.15.33:0/2584380179 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6252a5d60 msgr2=0x55f6252a61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:18.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.883+0000 7fd591a33640 1 --2- 172.21.15.33:0/2584380179 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6252a5d60 0x55f6252a61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.883+0000 7fd591a33640 1 -- 172.21.15.33:0/2584380179 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6251d4000 con 0x55f6252a70d0 2024-03-19T02:53:18.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.884+0000 7fd592234640 1 --2- 172.21.15.33:0/2203227149 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fd5780a2800 0x7fd5780a4cc0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd584005fd0 tx=0x7fd584009e80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:18.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.884+0000 7fd591a33640 1 --2- 172.21.15.33:0/2584380179 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6252a70d0 0x55f6252a5820 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd5880ee710 tx=0x7fd58803ac10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6cc7bd9aa7c495fc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:18.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.884+0000 7fd577fff640 1 -- 172.21.15.33:0/2584380179 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5880bd0a0 con 0x55f6252a70d0 2024-03-19T02:53:18.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.884+0000 7fd577fff640 1 -- 172.21.15.33:0/2584380179 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5880bdc60 con 0x55f6252a70d0 2024-03-19T02:53:18.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.884+0000 7fd577fff640 1 -- 172.21.15.33:0/2584380179 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5880be690 con 0x55f6252a70d0 2024-03-19T02:53:18.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.884+0000 7fd596880740 1 -- 172.21.15.33:0/2584380179 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6252a70d0 msgr2=0x55f6252a5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:18.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.885+0000 7fd596880740 1 --2- 172.21.15.33:0/2584380179 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6252a70d0 0x55f6252a5820 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd5880ee710 tx=0x7fd58803ac10 comp rx=0 tx=0).stop 2024-03-19T02:53:18.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.885+0000 7fd596880740 1 -- 172.21.15.33:0/2584380179 shutdown_connections 2024-03-19T02:53:18.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.885+0000 7fd596880740 1 --2- 172.21.15.33:0/2584380179 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6252afd30 0x55f6252b2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.885+0000 7fd596880740 1 --2- 172.21.15.33:0/2584380179 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6252a5d60 0x55f6252a61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.885+0000 7fd596880740 1 --2- 172.21.15.33:0/2584380179 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6252a70d0 0x55f6252a5820 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.885+0000 7fd596880740 1 -- 172.21.15.33:0/2584380179 >> 172.21.15.33:0/2584380179 conn(0x55f6252a0fc0 msgr2=0x55f6252a13c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:18.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.885+0000 7fd596880740 1 -- 172.21.15.33:0/2584380179 shutdown_connections 2024-03-19T02:53:18.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.885+0000 7fd596880740 1 -- 172.21.15.33:0/2584380179 wait complete. 2024-03-19T02:53:18.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.885+0000 7fd596880740 1 -- start start 2024-03-19T02:53:18.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.885+0000 7fd596880740 1 Processor -- start 2024-03-19T02:53:18.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.885+0000 7fd596880740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6252a5d60 0x55f6252b40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.885+0000 7fd596880740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6252a70d0 0x55f6252b4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.885+0000 7fd596880740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6252afd30 0x55f6252bb580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.885+0000 7fd596880740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f6252b2720 con 0x55f6252a5d60 2024-03-19T02:53:18.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.885+0000 7fd596880740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55f6251c6160 con 0x55f6252afd30 2024-03-19T02:53:18.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.885+0000 7fd596880740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f6252b28a0 con 0x55f6252a70d0 2024-03-19T02:53:18.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.886+0000 7fd596880740 10 client.0 _ll_register_callbacks cb 0x55f625202a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:53:18.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.886+0000 7fd591a33640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6252a5d60 0x55f6252b40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.886+0000 7fd591a33640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6252a5d60 0x55f6252b40e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:46998/0 (socket says 172.21.15.33:46998) 2024-03-19T02:53:18.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.886+0000 7fd591232640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6252a70d0 0x55f6252b4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.886+0000 7fd591a33640 1 -- 172.21.15.33:0/3303936655 learned_addr learned my addr 172.21.15.33:0/3303936655 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:18.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.886+0000 7fd592234640 1 --2- 172.21.15.33:0/3303936655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6252afd30 0x55f6252bb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.886+0000 7fd591a33640 1 -- 172.21.15.33:0/3303936655 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6252a70d0 msgr2=0x55f6252b4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:18.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.886+0000 7fd591a33640 1 --2- 172.21.15.33:0/3303936655 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6252a70d0 0x55f6252b4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.886+0000 7fd591a33640 1 -- 172.21.15.33:0/3303936655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6252afd30 msgr2=0x55f6252bb580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:18.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.886+0000 7fd591a33640 1 --2- 172.21.15.33:0/3303936655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6252afd30 0x55f6252bb580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.886+0000 7fd591a33640 1 -- 172.21.15.33:0/3303936655 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f624fe87d0 con 0x55f6252a5d60 2024-03-19T02:53:18.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.886+0000 7fd591a33640 1 --2- 172.21.15.33:0/3303936655 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6252a5d60 0x55f6252b40e0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fd5880bd260 tx=0x7fd58802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f04b920d2198a760 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:18.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.886+0000 7fd5757fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:53:18.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.886+0000 7fd5757fa640 1 -- 172.21.15.33:0/3303936655 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5880c25d0 con 0x55f6252a5d60 2024-03-19T02:53:18.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.886+0000 7fd5757fa640 1 -- 172.21.15.33:0/3303936655 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7fd588005440 con 0x55f6252a5d60 2024-03-19T02:53:18.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.886+0000 7fd596880740 1 -- 172.21.15.33:0/3303936655 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5880bdc60 con 0x55f6252a5d60 2024-03-19T02:53:18.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.886+0000 7fd5757fa640 1 -- 172.21.15.33:0/3303936655 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd5880c2bd0 con 0x55f6252a5d60 2024-03-19T02:53:18.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.886+0000 7fd596880740 10 client.15594 fetch_fsmap learned FSMap version 25 2024-03-19T02:53:18.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.886+0000 7fd596880740 1 -- 172.21.15.33:0/3303936655 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f624f1e3f0 con 0x55f6252a5d60 2024-03-19T02:53:18.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.887+0000 7fd5757fa640 1 -- 172.21.15.33:0/3303936655 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5880bdc60 con 0x55f6252a5d60 2024-03-19T02:53:18.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.887+0000 7fd5757fa640 1 -- 172.21.15.33:0/3303936655 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fd5880b9bd0 con 0x55f6252a5d60 2024-03-19T02:53:18.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.887+0000 7fd596880740 10 client.15594 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:53:18.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.887+0000 7fd596880740 10 client.15594 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:53:18.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.887+0000 7fd596880740 20 client.15594 populate_metadata read hostname 'smithi033' 2024-03-19T02:53:18.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.887+0000 7fd596880740 1 --2- 172.21.15.33:0/3303936655 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f6252bfbf0 0x55f6252c2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.887+0000 7fd596880740 4 client.15594 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:53:18.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.887+0000 7fd596880740 1 -- 172.21.15.33:0/3303936655 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f625217a40 con 0x55f6252bfbf0 2024-03-19T02:53:18.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.887+0000 7fd591232640 1 --2- 172.21.15.33:0/3303936655 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f6252bfbf0 0x55f6252c2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.887+0000 7fd5757fa640 10 client.15594 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:53:18.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.887+0000 7fd591232640 1 --2- 172.21.15.33:0/3303936655 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f6252bfbf0 0x55f6252c2000 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1dba5d4ef59f07b server_cookie=59c18a22fba339c8 in_seq=0 out_seq=0 2024-03-19T02:53:18.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.888+0000 7fd5757fa640 1 -- 172.21.15.33:0/3303936655 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f625217a40 con 0x55f6252bfbf0 2024-03-19T02:53:18.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.888+0000 7fd5757fa640 10 client.15594 handle_command_reply: tid=0 2024-03-19T02:53:18.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.888+0000 7fd596880740 1 client.15594 shutdown 2024-03-19T02:53:18.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.888+0000 7fd596880740 1 -- 172.21.15.33:0/3303936655 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6252a5d60 msgr2=0x55f6252b40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:18.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.888+0000 7fd596880740 1 --2- 172.21.15.33:0/3303936655 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6252a5d60 0x55f6252b40e0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fd5880bd260 tx=0x7fd58802bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:18.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.888+0000 7fd596880740 1 -- 172.21.15.33:0/3303936655 shutdown_connections 2024-03-19T02:53:18.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.888+0000 7fd596880740 1 --2- 172.21.15.33:0/3303936655 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f6252bfbf0 0x55f6252c2000 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.888+0000 7fd596880740 1 --2- 172.21.15.33:0/3303936655 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6252a70d0 0x55f6252b4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.888+0000 7fd596880740 1 --2- 172.21.15.33:0/3303936655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6252afd30 0x55f6252bb580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.888+0000 7fd596880740 1 --2- 172.21.15.33:0/3303936655 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6252a5d60 0x55f6252b40e0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.888+0000 7fd5757fa640 0 client.15594 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:53:18.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.888+0000 7fd596880740 1 -- 172.21.15.33:0/3303936655 >> 172.21.15.33:0/3303936655 conn(0x55f6252a0fc0 msgr2=0x55f6252a01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:18.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.888+0000 7fd596880740 1 -- 172.21.15.33:0/3303936655 shutdown_connections 2024-03-19T02:53:18.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.888+0000 7fd596880740 1 -- 172.21.15.33:0/3303936655 wait complete. 2024-03-19T02:53:18.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.888+0000 7fd596880740 20 client.15594 trim_cache size 0 max 16384 2024-03-19T02:53:18.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.906+0000 7fd596880740 1 Processor -- start 2024-03-19T02:53:18.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.906+0000 7fd596880740 1 -- start start 2024-03-19T02:53:18.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.906+0000 7fd596880740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6252a5d60 0x55f6252b4580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.906+0000 7fd596880740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6252a70d0 0x55f6252bb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.906+0000 7fd596880740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6252afd30 0x55f6252a4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.906+0000 7fd596880740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c1aa9d0 con 0x55f6252afd30 2024-03-19T02:53:18.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.906+0000 7fd596880740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c1a82a0 con 0x55f6252a70d0 2024-03-19T02:53:18.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.906+0000 7fd596880740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c1a60f0 con 0x55f6252a5d60 2024-03-19T02:53:18.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.906+0000 7fd592234640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6252afd30 0x55f6252a4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.906+0000 7fd592234640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6252afd30 0x55f6252a4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47006/0 (socket says 172.21.15.33:47006) 2024-03-19T02:53:18.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.906+0000 7fd592234640 1 -- 172.21.15.33:0/1339106776 learned_addr learned my addr 172.21.15.33:0/1339106776 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:18.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.906+0000 7fd591232640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6252a70d0 0x55f6252bb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.906+0000 7fd591a33640 1 --2- 172.21.15.33:0/1339106776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6252a5d60 0x55f6252b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.906+0000 7fd592234640 1 -- 172.21.15.33:0/1339106776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6252a5d60 msgr2=0x55f6252b4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:18.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.906+0000 7fd592234640 1 --2- 172.21.15.33:0/1339106776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6252a5d60 0x55f6252b4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.906+0000 7fd592234640 1 -- 172.21.15.33:0/1339106776 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6252a70d0 msgr2=0x55f6252bb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:18.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.906+0000 7fd592234640 1 --2- 172.21.15.33:0/1339106776 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6252a70d0 0x55f6252bb540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.906+0000 7fd592234640 1 -- 172.21.15.33:0/1339106776 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f624ede0f0 con 0x55f6252afd30 2024-03-19T02:53:18.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.906+0000 7fd591232640 1 --2- 172.21.15.33:0/1339106776 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6252a70d0 0x55f6252bb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:18.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.906+0000 7fd592234640 1 --2- 172.21.15.33:0/1339106776 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6252afd30 0x55f6252a4930 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd584013800 tx=0x7fd584021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b293fcd3592a1401 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:18.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.907+0000 7fd5777fe640 1 -- 172.21.15.33:0/1339106776 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd584004430 con 0x55f6252afd30 2024-03-19T02:53:18.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.907+0000 7fd5777fe640 1 -- 172.21.15.33:0/1339106776 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5840045d0 con 0x55f6252afd30 2024-03-19T02:53:18.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.907+0000 7fd5777fe640 1 -- 172.21.15.33:0/1339106776 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd58402a400 con 0x55f6252afd30 2024-03-19T02:53:18.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.907+0000 7fd596880740 1 -- 172.21.15.33:0/1339106776 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6252afd30 msgr2=0x55f6252a4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:18.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.907+0000 7fd596880740 1 --2- 172.21.15.33:0/1339106776 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6252afd30 0x55f6252a4930 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd584013800 tx=0x7fd584021000 comp rx=0 tx=0).stop 2024-03-19T02:53:18.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.907+0000 7fd596880740 1 -- 172.21.15.33:0/1339106776 shutdown_connections 2024-03-19T02:53:18.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.907+0000 7fd596880740 1 --2- 172.21.15.33:0/1339106776 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6252afd30 0x55f6252a4930 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.907+0000 7fd596880740 1 --2- 172.21.15.33:0/1339106776 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6252a70d0 0x55f6252bb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.907+0000 7fd596880740 1 --2- 172.21.15.33:0/1339106776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6252a5d60 0x55f6252b4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.907+0000 7fd596880740 1 -- 172.21.15.33:0/1339106776 >> 172.21.15.33:0/1339106776 conn(0x55f6252a0fc0 msgr2=0x55f6252a4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:18.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.907+0000 7fd596880740 1 -- 172.21.15.33:0/1339106776 shutdown_connections 2024-03-19T02:53:18.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.907+0000 7fd596880740 1 -- 172.21.15.33:0/1339106776 wait complete. 2024-03-19T02:53:18.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.908+0000 7fd596880740 1 -- start start 2024-03-19T02:53:18.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.908+0000 7fd596880740 1 Processor -- start 2024-03-19T02:53:18.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.908+0000 7fd596880740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6252a5d60 0x55f6252b8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.908+0000 7fd596880740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6252a70d0 0x55f6252b9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.908+0000 7fd596880740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6252afd30 0x55f6252ba010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.908+0000 7fd596880740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c1a6760 con 0x55f6252a70d0 2024-03-19T02:53:18.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.908+0000 7fd596880740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c1a7bd0 con 0x55f6252afd30 2024-03-19T02:53:18.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.908+0000 7fd596880740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c1a6e30 con 0x55f6252a5d60 2024-03-19T02:53:18.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.908+0000 7fd596880740 10 client.0 _ll_register_callbacks cb 0x7fd57c009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:53:18.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.908+0000 7fd591232640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6252a70d0 0x55f6252b9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.908+0000 7fd591232640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6252a70d0 0x55f6252b9000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47012/0 (socket says 172.21.15.33:47012) 2024-03-19T02:53:18.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.908+0000 7fd591a33640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6252a5d60 0x55f6252b8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.908+0000 7fd591232640 1 -- 172.21.15.33:0/2116560582 learned_addr learned my addr 172.21.15.33:0/2116560582 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:18.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.908+0000 7fd592234640 1 --2- 172.21.15.33:0/2116560582 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6252afd30 0x55f6252ba010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.908+0000 7fd591232640 1 -- 172.21.15.33:0/2116560582 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6252a5d60 msgr2=0x55f6252b8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:18.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.908+0000 7fd591232640 1 --2- 172.21.15.33:0/2116560582 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6252a5d60 0x55f6252b8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.909+0000 7fd591232640 1 -- 172.21.15.33:0/2116560582 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6252afd30 msgr2=0x55f6252ba010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:18.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.909+0000 7fd591232640 1 --2- 172.21.15.33:0/2116560582 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6252afd30 0x55f6252ba010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.909+0000 7fd591232640 1 -- 172.21.15.33:0/2116560582 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6252ba600 con 0x55f6252a70d0 2024-03-19T02:53:18.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.909+0000 7fd591a33640 1 --2- 172.21.15.33:0/2116560582 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6252a5d60 0x55f6252b8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:18.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.909+0000 7fd592234640 1 --2- 172.21.15.33:0/2116560582 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6252afd30 0x55f6252ba010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:18.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.909+0000 7fd591232640 1 --2- 172.21.15.33:0/2116560582 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6252a70d0 0x55f6252b9000 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fd57c009fd0 tx=0x7fd57c012e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67b3e9a88a9476d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:18.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.909+0000 7fd5757fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:53:18.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.909+0000 7fd5757fa640 1 -- 172.21.15.33:0/2116560582 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd57c011e20 con 0x55f6252a70d0 2024-03-19T02:53:18.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.909+0000 7fd5757fa640 1 -- 172.21.15.33:0/2116560582 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7fd57c005020 con 0x55f6252a70d0 2024-03-19T02:53:18.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.909+0000 7fd596880740 1 -- 172.21.15.33:0/2116560582 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5840045d0 con 0x55f6252a70d0 2024-03-19T02:53:18.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.909+0000 7fd5757fa640 1 -- 172.21.15.33:0/2116560582 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd57c031050 con 0x55f6252a70d0 2024-03-19T02:53:18.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.909+0000 7fd596880740 10 client.15600 fetch_fsmap learned FSMap version 25 2024-03-19T02:53:18.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.909+0000 7fd596880740 1 -- 172.21.15.33:0/2116560582 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f625325f00 con 0x55f6252a70d0 2024-03-19T02:53:18.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.909+0000 7fd5757fa640 1 -- 172.21.15.33:0/2116560582 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5840045d0 con 0x55f6252a70d0 2024-03-19T02:53:18.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.909+0000 7fd5757fa640 1 -- 172.21.15.33:0/2116560582 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fd5880d62a0 con 0x55f6252a70d0 2024-03-19T02:53:18.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.909+0000 7fd596880740 10 client.15600 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:53:18.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.909+0000 7fd596880740 10 client.15600 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:53:18.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.909+0000 7fd596880740 20 client.15600 populate_metadata read hostname 'smithi033' 2024-03-19T02:53:18.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.910+0000 7fd596880740 1 --2- 172.21.15.33:0/2116560582 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f625326840 0x55f625328c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:18.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.910+0000 7fd596880740 4 client.15600 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:53:18.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.910+0000 7fd596880740 1 -- 172.21.15.33:0/2116560582 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f625211d90 con 0x55f625326840 2024-03-19T02:53:18.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.910+0000 7fd591a33640 1 --2- 172.21.15.33:0/2116560582 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f625326840 0x55f625328c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:18.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.910+0000 7fd5757fa640 10 client.15600 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:53:18.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.910+0000 7fd591a33640 1 --2- 172.21.15.33:0/2116560582 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f625326840 0x55f625328c60 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3afefdcf09970aa6 server_cookie=c223e8da7e8c764a in_seq=0 out_seq=0 2024-03-19T02:53:18.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.910+0000 7fd5757fa640 1 -- 172.21.15.33:0/2116560582 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f625211d90 con 0x55f625326840 2024-03-19T02:53:18.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.910+0000 7fd5757fa640 10 client.15600 handle_command_reply: tid=0 2024-03-19T02:53:18.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.910+0000 7fd596880740 1 client.15600 shutdown 2024-03-19T02:53:18.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.910+0000 7fd596880740 1 -- 172.21.15.33:0/2116560582 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6252a70d0 msgr2=0x55f6252b9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:18.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.910+0000 7fd596880740 1 --2- 172.21.15.33:0/2116560582 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6252a70d0 0x55f6252b9000 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fd57c009fd0 tx=0x7fd57c012e80 comp rx=0 tx=0).stop 2024-03-19T02:53:18.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.910+0000 7fd596880740 1 -- 172.21.15.33:0/2116560582 shutdown_connections 2024-03-19T02:53:18.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.910+0000 7fd596880740 1 --2- 172.21.15.33:0/2116560582 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f625326840 0x55f625328c60 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.910+0000 7fd596880740 1 --2- 172.21.15.33:0/2116560582 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6252afd30 0x55f6252ba010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.910+0000 7fd596880740 1 --2- 172.21.15.33:0/2116560582 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6252a70d0 0x55f6252b9000 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.910+0000 7fd5757fa640 0 client.15600 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:53:18.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.910+0000 7fd596880740 1 --2- 172.21.15.33:0/2116560582 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6252a5d60 0x55f6252b8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.910+0000 7fd596880740 1 -- 172.21.15.33:0/2116560582 >> 172.21.15.33:0/2116560582 conn(0x55f6252a0fc0 msgr2=0x55f6252b4400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:18.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.910+0000 7fd596880740 1 -- 172.21.15.33:0/2116560582 shutdown_connections 2024-03-19T02:53:18.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.911+0000 7fd596880740 1 -- 172.21.15.33:0/2116560582 wait complete. 2024-03-19T02:53:18.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.911+0000 7fd596880740 20 client.15600 trim_cache size 0 max 16384 2024-03-19T02:53:18.918 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:53:18.919 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:53:18.919 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-03-19T02:53:18.919 INFO:teuthology.orchestra.run.smithi033.stdout: "9b3ffdd5-6bb0-4e46-a0ab-38c3bf3f5c3d": { 2024-03-19T02:53:18.919 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-03-19T02:53:18.919 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "9b3ffdd5-6bb0-4e46-a0ab-38c3bf3f5c3d", 2024-03-19T02:53:18.919 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-03-19T02:53:18.919 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T02:53:18.919 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T02:53:18.919 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:53:18.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.911+0000 7fd593c97640 1 -- 172.21.15.33:0/2203227149 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fd5780a2800 msgr2=0x7fd5780a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:18.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.911+0000 7fd593c97640 1 --2- 172.21.15.33:0/2203227149 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fd5780a2800 0x7fd5780a4cc0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd584005fd0 tx=0x7fd584009e80 comp rx=0 tx=0).stop 2024-03-19T02:53:18.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.911+0000 7fd593c97640 1 -- 172.21.15.33:0/2203227149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd58c100720 msgr2=0x7fd58c1969d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:18.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.911+0000 7fd593c97640 1 --2- 172.21.15.33:0/2203227149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd58c100720 0x7fd58c1969d0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fd588000c00 tx=0x7fd588005dc0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.911+0000 7fd593c97640 1 -- 172.21.15.33:0/2203227149 shutdown_connections 2024-03-19T02:53:18.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.911+0000 7fd593c97640 1 --2- 172.21.15.33:0/2203227149 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7fd5780a2800 0x7fd5780a4cc0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.911+0000 7fd593c97640 1 --2- 172.21.15.33:0/2203227149 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd58c1022d0 0x7fd58c19df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.911+0000 7fd593c97640 1 --2- 172.21.15.33:0/2203227149 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd58c101910 0x7fd58c196f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.911+0000 7fd593c97640 1 --2- 172.21.15.33:0/2203227149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd58c100720 0x7fd58c1969d0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:18.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.911+0000 7fd593c97640 1 -- 172.21.15.33:0/2203227149 >> 172.21.15.33:0/2203227149 conn(0x7fd58c0fbcb0 msgr2=0x7fd58c104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:18.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.911+0000 7fd593c97640 1 -- 172.21.15.33:0/2203227149 shutdown_connections 2024-03-19T02:53:18.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.911+0000 7fd593c97640 1 -- 172.21.15.33:0/2203227149 wait complete. 2024-03-19T02:53:18.922 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:18 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3643395201' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:53:18.922 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:18 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3964483632' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:53:18.922 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:18 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/596506148' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:53:18.923 INFO:tasks.cephfs.filesystem:scrub status for tag:9b3ffdd5-6bb0-4e46-a0ab-38c3bf3f5c3d - {'path': '/', 'tag': '9b3ffdd5-6bb0-4e46-a0ab-38c3bf3f5c3d', 'options': 'recursive,force'} 2024-03-19T02:53:18.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:18.973+0000 7f273a9f4640 1 -- 172.21.15.33:0/546545691 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f27341a2570 con 0x7f2734101950 2024-03-19T02:53:19.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:18 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3643395201' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:53:19.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:18 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3964483632' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:53:19.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:18 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/596506148' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:53:19.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:18 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3643395201' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T02:53:19.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:18 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3964483632' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:53:19.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:18 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/596506148' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T02:53:19.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.838+0000 7f27317fa640 1 -- 172.21.15.33:0/546545691 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mgrmap(e 19) ==== 137307+0+0 (secure 0 0 0) 0x7f2728097dd0 con 0x7f2734101950 2024-03-19T02:53:19.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.839+0000 7f27317fa640 1 -- 172.21.15.33:0/546545691 <== mon.2 v2:172.21.15.96:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f27280bb020 con 0x7f2734101950 2024-03-19T02:53:19.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.842+0000 7f273a9f4640 1 -- 172.21.15.33:0/546545691 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f271c0a28a0 msgr2=0x7f271c0a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:19.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.842+0000 7f273a9f4640 1 --2- 172.21.15.33:0/546545691 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f271c0a28a0 0x7f271c0a4d60 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2724005ab0 tx=0x7f2724005fd0 comp rx=0 tx=0).stop 2024-03-19T02:53:19.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.842+0000 7f273a9f4640 1 -- 172.21.15.33:0/546545691 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2734101950 msgr2=0x7f2734102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:19.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.842+0000 7f273a9f4640 1 --2- 172.21.15.33:0/546545691 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2734101950 0x7f2734102d30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f272800e9f0 tx=0x7f272800eeb0 comp rx=0 tx=0).stop 2024-03-19T02:53:19.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.842+0000 7f273a9f4640 1 -- 172.21.15.33:0/546545691 shutdown_connections 2024-03-19T02:53:19.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.842+0000 7f273a9f4640 1 --2- 172.21.15.33:0/546545691 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f271c0a28a0 0x7f271c0a4d60 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:19.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.842+0000 7f273a9f4640 1 --2- 172.21.15.33:0/546545691 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2734102310 0x7f2734103270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:19.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.842+0000 7f273a9f4640 1 --2- 172.21.15.33:0/546545691 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2734101950 0x7f2734102d30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:19.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.842+0000 7f273a9f4640 1 --2- 172.21.15.33:0/546545691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2734100760 0x7f27341046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:19.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.842+0000 7f273a9f4640 1 -- 172.21.15.33:0/546545691 >> 172.21.15.33:0/546545691 conn(0x7f27340fbcd0 msgr2=0x7f27340fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:19.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.842+0000 7f273a9f4640 1 -- 172.21.15.33:0/546545691 shutdown_connections 2024-03-19T02:53:19.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.842+0000 7f273a9f4640 1 -- 172.21.15.33:0/546545691 wait complete. 2024-03-19T02:53:19.858 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-03-19T02:53:19.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.987+0000 7f3980532640 1 -- 172.21.15.33:0/3342630670 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3978100760 msgr2=0x7f3978100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:19.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.987+0000 7f3980532640 1 --2- 172.21.15.33:0/3342630670 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3978100760 0x7f3978100b80 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f3970009ca0 tx=0x7f397002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:53:19.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.988+0000 7f3980532640 1 -- 172.21.15.33:0/3342630670 shutdown_connections 2024-03-19T02:53:19.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.988+0000 7f3980532640 1 --2- 172.21.15.33:0/3342630670 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3978101950 0x7f3978101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:19.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.988+0000 7f3980532640 1 --2- 172.21.15.33:0/3342630670 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3978102310 0x7f3978108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:19.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.988+0000 7f3980532640 1 --2- 172.21.15.33:0/3342630670 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3978100760 0x7f3978100b80 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:19.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.988+0000 7f3980532640 1 -- 172.21.15.33:0/3342630670 >> 172.21.15.33:0/3342630670 conn(0x7f39780fbcd0 msgr2=0x7f39780fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:19.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.988+0000 7f3980532640 1 -- 172.21.15.33:0/3342630670 shutdown_connections 2024-03-19T02:53:19.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.988+0000 7f3980532640 1 -- 172.21.15.33:0/3342630670 wait complete. 2024-03-19T02:53:19.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.988+0000 7f3980532640 1 Processor -- start 2024-03-19T02:53:19.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.988+0000 7f3980532640 1 -- start start 2024-03-19T02:53:19.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.988+0000 7f3980532640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3978100760 0x7f39781046f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:19.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.988+0000 7f3980532640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3978101950 0x7f3978102d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:19.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.988+0000 7f3980532640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3978102310 0x7f3978103270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:19.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.988+0000 7f3980532640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f397810c260 con 0x7f3978101950 2024-03-19T02:53:19.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.988+0000 7f3980532640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f397810c0e0 con 0x7f3978100760 2024-03-19T02:53:19.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.988+0000 7f3980532640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f397810c3e0 con 0x7f3978102310 2024-03-19T02:53:19.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.988+0000 7f397dacd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3978101950 0x7f3978102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:19.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.989+0000 7f397dacd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3978101950 0x7f3978102d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47050/0 (socket says 172.21.15.33:47050) 2024-03-19T02:53:19.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.989+0000 7f397dacd640 1 -- 172.21.15.33:0/252401970 learned_addr learned my addr 172.21.15.33:0/252401970 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:19.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.989+0000 7f397e2ce640 1 --2- 172.21.15.33:0/252401970 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3978100760 0x7f39781046f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:19.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.989+0000 7f397eacf640 1 --2- 172.21.15.33:0/252401970 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3978102310 0x7f3978103270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:19.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.989+0000 7f397dacd640 1 -- 172.21.15.33:0/252401970 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3978102310 msgr2=0x7f3978103270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:19.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.989+0000 7f397dacd640 1 --2- 172.21.15.33:0/252401970 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3978102310 0x7f3978103270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:19.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.989+0000 7f397dacd640 1 -- 172.21.15.33:0/252401970 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3978100760 msgr2=0x7f39781046f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:19.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.989+0000 7f397dacd640 1 --2- 172.21.15.33:0/252401970 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3978100760 0x7f39781046f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:19.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.989+0000 7f397dacd640 1 -- 172.21.15.33:0/252401970 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3978103b00 con 0x7f3978101950 2024-03-19T02:53:19.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.989+0000 7f397dacd640 1 --2- 172.21.15.33:0/252401970 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3978101950 0x7f3978102d30 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f396c00d950 tx=0x7f396c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:19.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.989+0000 7f39777fe640 1 -- 172.21.15.33:0/252401970 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f396c017070 con 0x7f3978101950 2024-03-19T02:53:19.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.989+0000 7f3980532640 1 -- 172.21.15.33:0/252401970 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39781a1b20 con 0x7f3978101950 2024-03-19T02:53:19.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.989+0000 7f39777fe640 1 -- 172.21.15.33:0/252401970 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f396c009c30 con 0x7f3978101950 2024-03-19T02:53:19.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.989+0000 7f39777fe640 1 -- 172.21.15.33:0/252401970 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f396c00f620 con 0x7f3978101950 2024-03-19T02:53:19.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.989+0000 7f3980532640 1 -- 172.21.15.33:0/252401970 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39781a20c0 con 0x7f3978101950 2024-03-19T02:53:19.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.990+0000 7f3980532640 1 -- 172.21.15.33:0/252401970 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39781a2290 con 0x7f3978101950 2024-03-19T02:53:19.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.991+0000 7f39777fe640 1 -- 172.21.15.33:0/252401970 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 137307+0+0 (secure 0 0 0) 0x7f396c0109a0 con 0x7f3978101950 2024-03-19T02:53:19.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.992+0000 7f39777fe640 1 --2- 172.21.15.33:0/252401970 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f39600a28f0 0x7f39600a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:19.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.992+0000 7f397e2ce640 1 -- 172.21.15.33:0/252401970 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f39600a28f0 msgr2=0x7f39600a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/1453067701 2024-03-19T02:53:19.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.992+0000 7f39777fe640 1 -- 172.21.15.33:0/252401970 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f396c0cea50 con 0x7f3978101950 2024-03-19T02:53:19.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.992+0000 7f397e2ce640 1 --2- 172.21.15.33:0/252401970 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f39600a28f0 0x7f39600a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T02:53:19.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:19.997+0000 7f39777fe640 1 -- 172.21.15.33:0/252401970 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f396c009de0 con 0x7f3978101950 2024-03-19T02:53:20.168 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:19 smithi155 ceph-mon[35804]: scrub queued for path: / 2024-03-19T02:53:20.168 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:19 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T02:53:20.168 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:19 smithi155 ceph-mon[35804]: scrub summary: active paths [/] 2024-03-19T02:53:20.168 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:19 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T02:53:20.168 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:19 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/546545691' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-19T02:53:20.168 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:19 smithi155 ceph-mon[35804]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-19T02:53:20.168 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:19 smithi155 ceph-mon[35804]: pgmap v377: 129 pgs: 129 active+clean; 648 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-03-19T02:53:20.169 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:53:19 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: ignoring --setuser ceph since I am not root 2024-03-19T02:53:20.169 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:53:19 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: ignoring --setgroup ceph since I am not root 2024-03-19T02:53:20.169 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:53:19 smithi155 ceph-mgr[37692]: -- 172.21.15.155:0/1810753371 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556a71a805a0 con 0x556a71a54800 2024-03-19T02:53:20.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.182+0000 7f3980532640 1 -- 172.21.15.33:0/252401970 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f39781a2460 con 0x7f3978101950 2024-03-19T02:53:20.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.186+0000 7f39777fe640 1 -- 172.21.15.33:0/252401970 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v28)=0 v28) ==== 138+0+0 (secure 0 0 0) 0x7f396c09b0b0 con 0x7f3978101950 2024-03-19T02:53:20.189 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.188+0000 7f3980532640 1 -- 172.21.15.33:0/252401970 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f39600a28f0 msgr2=0x7f39600a4db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T02:53:20.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.188+0000 7f3980532640 1 --2- 172.21.15.33:0/252401970 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f39600a28f0 0x7f39600a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:20.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.188+0000 7f3980532640 1 -- 172.21.15.33:0/252401970 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3978101950 msgr2=0x7f3978102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:20.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.188+0000 7f3980532640 1 --2- 172.21.15.33:0/252401970 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3978101950 0x7f3978102d30 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f396c00d950 tx=0x7f396c00de10 comp rx=0 tx=0).stop 2024-03-19T02:53:20.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.188+0000 7f3980532640 1 -- 172.21.15.33:0/252401970 shutdown_connections 2024-03-19T02:53:20.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.188+0000 7f3980532640 1 --2- 172.21.15.33:0/252401970 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f39600a28f0 0x7f39600a4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:20.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.188+0000 7f3980532640 1 --2- 172.21.15.33:0/252401970 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3978102310 0x7f3978103270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:20.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.188+0000 7f3980532640 1 --2- 172.21.15.33:0/252401970 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3978101950 0x7f3978102d30 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:20.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.188+0000 7f3980532640 1 --2- 172.21.15.33:0/252401970 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3978100760 0x7f39781046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:20.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.188+0000 7f3980532640 1 -- 172.21.15.33:0/252401970 >> 172.21.15.33:0/252401970 conn(0x7f39780fbcd0 msgr2=0x7f39780fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:20.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.188+0000 7f3980532640 1 -- 172.21.15.33:0/252401970 shutdown_connections 2024-03-19T02:53:20.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.188+0000 7f3980532640 1 -- 172.21.15.33:0/252401970 wait complete. 2024-03-19T02:53:20.199 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:19 smithi033 ceph-mon[31973]: scrub queued for path: / 2024-03-19T02:53:20.200 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:19 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T02:53:20.200 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:19 smithi033 ceph-mon[31973]: scrub summary: active paths [/] 2024-03-19T02:53:20.200 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:19 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T02:53:20.200 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:19 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/546545691' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-19T02:53:20.200 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:19 smithi033 ceph-mon[31973]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-19T02:53:20.200 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:19 smithi033 ceph-mon[31973]: pgmap v377: 129 pgs: 129 active+clean; 648 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-03-19T02:53:20.200 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:19 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: ignoring --setuser ceph since I am not root 2024-03-19T02:53:20.200 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:19 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: ignoring --setgroup ceph since I am not root 2024-03-19T02:53:20.201 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-03-19T02:53:20.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:19 smithi096 ceph-mon[35946]: scrub queued for path: / 2024-03-19T02:53:20.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:19 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T02:53:20.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:19 smithi096 ceph-mon[35946]: scrub summary: active paths [/] 2024-03-19T02:53:20.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:19 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T02:53:20.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:19 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/546545691' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-19T02:53:20.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:19 smithi096 ceph-mon[35946]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-19T02:53:20.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:19 smithi096 ceph-mon[35946]: pgmap v377: 129 pgs: 129 active+clean; 648 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-03-19T02:53:20.216 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:53:19 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: ignoring --setuser ceph since I am not root 2024-03-19T02:53:20.216 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:53:19 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: ignoring --setgroup ceph since I am not root 2024-03-19T02:53:20.216 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:53:19 smithi096 ceph-mgr[37370]: -- 172.21.15.96:0/1523788606 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5629d3e425a0 con 0x5629d3e16800 2024-03-19T02:53:20.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.305+0000 7f8c3bfff640 1 -- 172.21.15.33:0/3503003120 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c3c103250 msgr2=0x7f8c3c109ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:20.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.305+0000 7f8c3bfff640 1 --2- 172.21.15.33:0/3503003120 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c3c103250 0x7f8c3c109ae0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f8c34009ca0 tx=0x7f8c3402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:53:20.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.306+0000 7f8c3bfff640 1 -- 172.21.15.33:0/3503003120 shutdown_connections 2024-03-19T02:53:20.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.306+0000 7f8c3bfff640 1 --2- 172.21.15.33:0/3503003120 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c3c103250 0x7f8c3c109ae0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:20.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.306+0000 7f8c3bfff640 1 --2- 172.21.15.33:0/3503003120 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8c3c102890 0x7f8c3c102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:20.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.306+0000 7f8c3bfff640 1 --2- 172.21.15.33:0/3503003120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c3c1016a0 0x7f8c3c101ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:20.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.306+0000 7f8c3bfff640 1 -- 172.21.15.33:0/3503003120 >> 172.21.15.33:0/3503003120 conn(0x7f8c3c0fcc30 msgr2=0x7f8c3c0ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:20.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.306+0000 7f8c3bfff640 1 -- 172.21.15.33:0/3503003120 shutdown_connections 2024-03-19T02:53:20.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.306+0000 7f8c3bfff640 1 -- 172.21.15.33:0/3503003120 wait complete. 2024-03-19T02:53:20.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.306+0000 7f8c3bfff640 1 Processor -- start 2024-03-19T02:53:20.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.306+0000 7f8c3bfff640 1 -- start start 2024-03-19T02:53:20.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.307+0000 7f8c3bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c3c1016a0 0x7f8c3c197830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:20.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.307+0000 7f8c3bfff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8c3c102890 0x7f8c3c197d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:20.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.307+0000 7f8c3bfff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c3c103250 0x7f8c3c19ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:20.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.307+0000 7f8c3bfff640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8c3c10d1b0 con 0x7f8c3c1016a0 2024-03-19T02:53:20.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.307+0000 7f8c3bfff640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8c3c10d030 con 0x7f8c3c102890 2024-03-19T02:53:20.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.307+0000 7f8c3bfff640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8c3c10d330 con 0x7f8c3c103250 2024-03-19T02:53:20.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.307+0000 7f8c3affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c3c1016a0 0x7f8c3c197830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:20.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.307+0000 7f8c3affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c3c1016a0 0x7f8c3c197830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47068/0 (socket says 172.21.15.33:47068) 2024-03-19T02:53:20.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.307+0000 7f8c3affd640 1 -- 172.21.15.33:0/1725990192 learned_addr learned my addr 172.21.15.33:0/1725990192 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:20.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.307+0000 7f8c327fc640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8c3c102890 0x7f8c3c197d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:20.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.307+0000 7f8c3b7fe640 1 --2- 172.21.15.33:0/1725990192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c3c103250 0x7f8c3c19ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:20.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.307+0000 7f8c327fc640 1 -- 172.21.15.33:0/1725990192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c3c103250 msgr2=0x7f8c3c19ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:20.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.307+0000 7f8c327fc640 1 --2- 172.21.15.33:0/1725990192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c3c103250 0x7f8c3c19ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:20.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.307+0000 7f8c327fc640 1 -- 172.21.15.33:0/1725990192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c3c1016a0 msgr2=0x7f8c3c197830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:20.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.307+0000 7f8c327fc640 1 --2- 172.21.15.33:0/1725990192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c3c1016a0 0x7f8c3c197830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:20.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.307+0000 7f8c327fc640 1 -- 172.21.15.33:0/1725990192 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c3c19f450 con 0x7f8c3c102890 2024-03-19T02:53:20.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.307+0000 7f8c3affd640 1 --2- 172.21.15.33:0/1725990192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c3c1016a0 0x7f8c3c197830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:20.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.307+0000 7f8c327fc640 1 --2- 172.21.15.33:0/1725990192 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8c3c102890 0x7f8c3c197d70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8c2400e9f0 tx=0x7f8c2400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:20.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.308+0000 7f8c409fd640 1 -- 172.21.15.33:0/1725990192 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c2400ccc0 con 0x7f8c3c102890 2024-03-19T02:53:20.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.308+0000 7f8c409fd640 1 -- 172.21.15.33:0/1725990192 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8c240108e0 con 0x7f8c3c102890 2024-03-19T02:53:20.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.308+0000 7f8c3bfff640 1 -- 172.21.15.33:0/1725990192 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c3c19f740 con 0x7f8c3c102890 2024-03-19T02:53:20.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.308+0000 7f8c409fd640 1 -- 172.21.15.33:0/1725990192 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c24018880 con 0x7f8c3c102890 2024-03-19T02:53:20.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.308+0000 7f8c3bfff640 1 -- 172.21.15.33:0/1725990192 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c3c19fc90 con 0x7f8c3c102890 2024-03-19T02:53:20.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.309+0000 7f8c3bfff640 1 -- 172.21.15.33:0/1725990192 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c3c108a40 con 0x7f8c3c102890 2024-03-19T02:53:20.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.310+0000 7f8c409fd640 1 -- 172.21.15.33:0/1725990192 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 19) ==== 137307+0+0 (secure 0 0 0) 0x7f8c24010a90 con 0x7f8c3c102890 2024-03-19T02:53:20.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.310+0000 7f8c409fd640 1 --2- 172.21.15.33:0/1725990192 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8c280a2840 0x7f8c280a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:20.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.310+0000 7f8c3affd640 1 -- 172.21.15.33:0/1725990192 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8c280a2840 msgr2=0x7f8c280a4d00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/1453067701 2024-03-19T02:53:20.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.310+0000 7f8c409fd640 1 -- 172.21.15.33:0/1725990192 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f8c240cf110 con 0x7f8c3c102890 2024-03-19T02:53:20.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.310+0000 7f8c3affd640 1 --2- 172.21.15.33:0/1725990192 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8c280a2840 0x7f8c280a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T02:53:20.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.318+0000 7f8c409fd640 1 -- 172.21.15.33:0/1725990192 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8c2409b7d0 con 0x7f8c3c102890 2024-03-19T02:53:20.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.504+0000 7f8c3bfff640 1 -- 172.21.15.33:0/1725990192 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f8c3c02d1f0 con 0x7f8c3c102890 2024-03-19T02:53:20.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.511+0000 7f8c409fd640 1 -- 172.21.15.33:0/1725990192 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v29)=0 v29) ==== 133+0+0 (secure 0 0 0) 0x7f8c2409f870 con 0x7f8c3c102890 2024-03-19T02:53:20.512 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.512+0000 7f8c3affd640 1 -- 172.21.15.33:0/1725990192 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8c280a2840 msgr2=0x7f8c280a4d00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/1453067701 2024-03-19T02:53:20.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.512+0000 7f8c3affd640 1 --2- 172.21.15.33:0/1725990192 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8c280a2840 0x7f8c280a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T02:53:20.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.513+0000 7f8c3bfff640 1 -- 172.21.15.33:0/1725990192 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8c280a2840 msgr2=0x7f8c280a4d00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T02:53:20.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.513+0000 7f8c3bfff640 1 --2- 172.21.15.33:0/1725990192 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8c280a2840 0x7f8c280a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:20.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.513+0000 7f8c3bfff640 1 -- 172.21.15.33:0/1725990192 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8c3c102890 msgr2=0x7f8c3c197d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:20.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.513+0000 7f8c3bfff640 1 --2- 172.21.15.33:0/1725990192 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8c3c102890 0x7f8c3c197d70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8c2400e9f0 tx=0x7f8c2400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:53:20.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.513+0000 7f8c3bfff640 1 -- 172.21.15.33:0/1725990192 shutdown_connections 2024-03-19T02:53:20.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.513+0000 7f8c3bfff640 1 --2- 172.21.15.33:0/1725990192 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f8c280a2840 0x7f8c280a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:20.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.513+0000 7f8c3bfff640 1 --2- 172.21.15.33:0/1725990192 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c3c103250 0x7f8c3c19ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:20.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.513+0000 7f8c3bfff640 1 --2- 172.21.15.33:0/1725990192 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8c3c102890 0x7f8c3c197d70 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:20.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.513+0000 7f8c3bfff640 1 --2- 172.21.15.33:0/1725990192 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8c3c1016a0 0x7f8c3c197830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:20.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.513+0000 7f8c3bfff640 1 -- 172.21.15.33:0/1725990192 >> 172.21.15.33:0/1725990192 conn(0x7f8c3c0fcc30 msgr2=0x7f8c3c105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:20.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.513+0000 7f8c3bfff640 1 -- 172.21.15.33:0/1725990192 shutdown_connections 2024-03-19T02:53:20.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.513+0000 7f8c3bfff640 1 -- 172.21.15.33:0/1725990192 wait complete. 2024-03-19T02:53:20.525 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-03-19T02:53:20.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.630+0000 7f4326abf640 1 -- 172.21.15.33:0/4153604542 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f43201022f0 msgr2=0x7f4320108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:20.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.630+0000 7f4326abf640 1 --2- 172.21.15.33:0/4153604542 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f43201022f0 0x7f4320108b80 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f4318009ca0 tx=0x7f431802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:53:20.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.631+0000 7f4326abf640 1 -- 172.21.15.33:0/4153604542 shutdown_connections 2024-03-19T02:53:20.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.631+0000 7f4326abf640 1 --2- 172.21.15.33:0/4153604542 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f43201022f0 0x7f4320108b80 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:20.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.631+0000 7f4326abf640 1 --2- 172.21.15.33:0/4153604542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4320101930 0x7f4320101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:20.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.631+0000 7f4326abf640 1 --2- 172.21.15.33:0/4153604542 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4320100740 0x7f4320100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:20.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.631+0000 7f4326abf640 1 -- 172.21.15.33:0/4153604542 >> 172.21.15.33:0/4153604542 conn(0x7f43200fbcb0 msgr2=0x7f43200fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:20.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.631+0000 7f4326abf640 1 -- 172.21.15.33:0/4153604542 shutdown_connections 2024-03-19T02:53:20.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.631+0000 7f4326abf640 1 -- 172.21.15.33:0/4153604542 wait complete. 2024-03-19T02:53:20.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.631+0000 7f4326abf640 1 Processor -- start 2024-03-19T02:53:20.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.631+0000 7f4326abf640 1 -- start start 2024-03-19T02:53:20.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.632+0000 7f4326abf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4320100740 0x7f4320196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:20.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.632+0000 7f4326abf640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4320101930 0x7f4320196db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:20.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.632+0000 7f4326abf640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f43201022f0 0x7f432019de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:20.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.632+0000 7f4326abf640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f432010c250 con 0x7f4320100740 2024-03-19T02:53:20.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.632+0000 7f4326abf640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f432010c0d0 con 0x7f43201022f0 2024-03-19T02:53:20.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.632+0000 7f4326abf640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f432010c3d0 con 0x7f4320101930 2024-03-19T02:53:20.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.632+0000 7f432485b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4320100740 0x7f4320196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:20.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.632+0000 7f432485b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4320100740 0x7f4320196870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:47092/0 (socket says 172.21.15.33:47092) 2024-03-19T02:53:20.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.632+0000 7f432485b640 1 -- 172.21.15.33:0/1969252986 learned_addr learned my addr 172.21.15.33:0/1969252986 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:20.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.632+0000 7f432505c640 1 --2- 172.21.15.33:0/1969252986 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f43201022f0 0x7f432019de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:20.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.632+0000 7f431ffff640 1 --2- 172.21.15.33:0/1969252986 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4320101930 0x7f4320196db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:20.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.632+0000 7f432485b640 1 -- 172.21.15.33:0/1969252986 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4320101930 msgr2=0x7f4320196db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:20.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.632+0000 7f432485b640 1 --2- 172.21.15.33:0/1969252986 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4320101930 0x7f4320196db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:20.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.632+0000 7f432485b640 1 -- 172.21.15.33:0/1969252986 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f43201022f0 msgr2=0x7f432019de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:20.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.632+0000 7f432485b640 1 --2- 172.21.15.33:0/1969252986 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f43201022f0 0x7f432019de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:20.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.632+0000 7f432485b640 1 -- 172.21.15.33:0/1969252986 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f432019e490 con 0x7f4320100740 2024-03-19T02:53:20.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.632+0000 7f431ffff640 1 --2- 172.21.15.33:0/1969252986 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4320101930 0x7f4320196db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:53:20.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.632+0000 7f432485b640 1 --2- 172.21.15.33:0/1969252986 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4320100740 0x7f4320196870 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f431400e9c0 tx=0x7f431400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:20.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.632+0000 7f431dffb640 1 -- 172.21.15.33:0/1969252986 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f431400cd80 con 0x7f4320100740 2024-03-19T02:53:20.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.632+0000 7f4326abf640 1 -- 172.21.15.33:0/1969252986 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f432019e780 con 0x7f4320100740 2024-03-19T02:53:20.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.632+0000 7f431dffb640 1 -- 172.21.15.33:0/1969252986 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f43140035f0 con 0x7f4320100740 2024-03-19T02:53:20.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.632+0000 7f431dffb640 1 -- 172.21.15.33:0/1969252986 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4314010690 con 0x7f4320100740 2024-03-19T02:53:20.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.633+0000 7f4326abf640 1 -- 172.21.15.33:0/1969252986 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f432019ecd0 con 0x7f4320100740 2024-03-19T02:53:20.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.634+0000 7f4326abf640 1 -- 172.21.15.33:0/1969252986 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4320107a60 con 0x7f4320100740 2024-03-19T02:53:20.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.634+0000 7f431dffb640 1 -- 172.21.15.33:0/1969252986 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 19) ==== 137307+0+0 (secure 0 0 0) 0x7f43140037a0 con 0x7f4320100740 2024-03-19T02:53:20.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.635+0000 7f431dffb640 1 --2- 172.21.15.33:0/1969252986 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f43100a28f0 0x7f43100a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:20.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.635+0000 7f431ffff640 1 -- 172.21.15.33:0/1969252986 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f43100a28f0 msgr2=0x7f43100a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/1453067701 2024-03-19T02:53:20.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.635+0000 7f431dffb640 1 -- 172.21.15.33:0/1969252986 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9150+0+0 (secure 0 0 0) 0x7f43140cf290 con 0x7f4320100740 2024-03-19T02:53:20.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.635+0000 7f431ffff640 1 --2- 172.21.15.33:0/1969252986 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f43100a28f0 0x7f43100a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T02:53:20.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.640+0000 7f431dffb640 1 -- 172.21.15.33:0/1969252986 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f431409bfa0 con 0x7f4320100740 2024-03-19T02:53:20.766 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:53:20 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:53:20.445+0000 7f1d04c4d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T02:53:20.766 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:53:20 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:53:20.542+0000 7f1d04c4d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T02:53:20.778 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:20 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:20.450+0000 7f1ce1898200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T02:53:20.778 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:20 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:20.550+0000 7f1ce1898200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T02:53:20.830 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:53:20 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:53:20.520+0000 7efc370e9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T02:53:20.830 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:53:20 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:53:20.614+0000 7efc370e9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T02:53:20.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.836+0000 7f431ffff640 1 -- 172.21.15.33:0/1969252986 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f43100a28f0 msgr2=0x7f43100a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/1453067701 2024-03-19T02:53:20.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.836+0000 7f431ffff640 1 --2- 172.21.15.33:0/1969252986 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f43100a28f0 0x7f43100a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T02:53:20.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:20.846+0000 7f4326abf640 1 -- 172.21.15.33:0/1969252986 --> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f4320061c60 con 0x7f43100a28f0 2024-03-19T02:53:21.198 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:20 smithi155 ceph-mon[35804]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-19T02:53:21.198 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:20 smithi155 ceph-mon[35804]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-19T02:53:21.198 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:20 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/252401970' entity='client.admin' 2024-03-19T02:53:21.198 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:20 smithi155 ceph-mon[35804]: from='client.? ' entity='client.admin' 2024-03-19T02:53:21.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:20 smithi096 ceph-mon[35946]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-19T02:53:21.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:20 smithi096 ceph-mon[35946]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-19T02:53:21.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:20 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/252401970' entity='client.admin' 2024-03-19T02:53:21.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:20 smithi096 ceph-mon[35946]: from='client.? ' entity='client.admin' 2024-03-19T02:53:21.219 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:20 smithi033 ceph-mon[31973]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-19T02:53:21.220 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:20 smithi033 ceph-mon[31973]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-19T02:53:21.220 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:20 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/252401970' entity='client.admin' 2024-03-19T02:53:21.220 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:20 smithi033 ceph-mon[31973]: from='client.? ' entity='client.admin' 2024-03-19T02:53:21.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:21.237+0000 7f431ffff640 1 -- 172.21.15.33:0/1969252986 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f43100a28f0 msgr2=0x7f43100a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/1453067701 2024-03-19T02:53:21.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:21.237+0000 7f431ffff640 1 --2- 172.21.15.33:0/1969252986 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f43100a28f0 0x7f43100a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-19T02:53:21.516 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:53:21 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:53:21.197+0000 7f1d04c4d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T02:53:21.516 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:53:21 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:53:21.294+0000 7f1d04c4d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T02:53:21.516 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:53:21 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:53:21.439+0000 7f1d04c4d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T02:53:21.528 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:21 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:21.218+0000 7f1ce1898200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T02:53:21.528 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:21 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:21.317+0000 7f1ce1898200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T02:53:21.528 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:21 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:21.461+0000 7f1ce1898200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T02:53:21.709 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:53:21 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:53:21.424+0000 7efc370e9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T02:53:21.710 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:53:21 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:53:21.541+0000 7efc370e9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T02:53:21.905 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:53:21 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:53:21.537+0000 7f1d04c4d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T02:53:21.905 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:53:21 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:53:21.621+0000 7f1d04c4d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T02:53:21.906 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:53:21 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:53:21.904+0000 7f1d04c4d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-19T02:53:21.917 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:21 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:21.559+0000 7f1ce1898200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T02:53:21.917 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:21 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:21.645+0000 7f1ce1898200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T02:53:21.918 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:21 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:21.916+0000 7f1ce1898200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-19T02:53:21.965 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:53:21 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:53:21.708+0000 7efc370e9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T02:53:21.965 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:53:21 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:53:21.805+0000 7efc370e9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T02:53:21.965 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:53:21 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:53:21.890+0000 7efc370e9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T02:53:22.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:22.038+0000 7f431ffff640 1 -- 172.21.15.33:0/1969252986 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f43100a28f0 msgr2=0x7f43100a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/1453067701 2024-03-19T02:53:22.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:22.038+0000 7f431ffff640 1 --2- 172.21.15.33:0/1969252986 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f43100a28f0 0x7f43100a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-19T02:53:22.196 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:53:22 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:53:22.065+0000 7f1d04c4d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T02:53:22.216 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:22 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:22.028+0000 7f1ce1898200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T02:53:22.216 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:22 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:22.117+0000 7f1ce1898200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T02:53:22.217 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:22 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:22.215+0000 7f1ce1898200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T02:53:22.465 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:53:22 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:53:22.181+0000 7efc370e9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-19T02:53:22.465 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:53:22 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:53:22.295+0000 7efc370e9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T02:53:22.465 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:53:22 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:53:22.384+0000 7efc370e9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T02:53:22.481 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:53:22 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:53:22.195+0000 7f1d04c4d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T02:53:22.481 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:53:22 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:53:22.299+0000 7f1d04c4d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T02:53:22.481 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:53:22 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:53:22.480+0000 7f1d04c4d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T02:53:22.528 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:22 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:22.394+0000 7f1ce1898200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T02:53:22.965 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:53:22 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:53:22.480+0000 7efc370e9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T02:53:22.965 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:53:22 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:53:22.681+0000 7efc370e9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T02:53:23.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:23.639+0000 7f431ffff640 1 -- 172.21.15.33:0/1969252986 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f43100a28f0 msgr2=0x7f43100a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/1453067701 2024-03-19T02:53:23.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:23.639+0000 7f431ffff640 1 --2- 172.21.15.33:0/1969252986 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f43100a28f0 0x7f43100a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-19T02:53:23.766 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:53:23 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:53:23.510+0000 7f1d04c4d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-19T02:53:23.766 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:53:23 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:53:23.682+0000 7f1d04c4d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T02:53:23.778 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:23 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:23.386+0000 7f1ce1898200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-19T02:53:23.778 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:23 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:23.520+0000 7f1ce1898200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T02:53:24.159 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:23 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:23.863+0000 7f1ce1898200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T02:53:24.160 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:23 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:23.955+0000 7f1ce1898200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T02:53:24.160 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:24 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:24.158+0000 7f1ce1898200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:53:24.215 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:53:23 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:53:23.870+0000 7efc370e9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-19T02:53:24.215 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:53:24 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:53:24.002+0000 7efc370e9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T02:53:24.336 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:53:24 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:53:24.033+0000 7f1d04c4d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T02:53:24.336 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:53:24 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:53:24.124+0000 7f1d04c4d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T02:53:24.336 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:53:24 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:53:24.335+0000 7f1d04c4d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:53:24.473 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:53:24 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:53:24.369+0000 7efc370e9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T02:53:24.474 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:53:24 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:53:24.472+0000 7efc370e9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T02:53:24.528 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:24 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:24.353+0000 7f1ce1898200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T02:53:24.766 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:53:24 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:53:24.551+0000 7f1d04c4d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T02:53:24.861 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:53:24 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:53:24.671+0000 7efc370e9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:53:24.861 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:53:24 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:53:24.860+0000 7efc370e9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T02:53:24.915 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:24 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:24.650+0000 7f1ce1898200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:53:24.915 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:24 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:24.914+0000 7f1ce1898200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T02:53:25.149 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:53:24 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:53:24.846+0000 7f1d04c4d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:53:25.149 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:53:25 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:53:25.147+0000 7f1d04c4d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T02:53:25.278 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:25 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:25.162+0000 7f1ce1898200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T02:53:25.415 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:53:25 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:53:25.154+0000 7efc370e9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T02:53:25.415 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:53:25 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:53:25.414+0000 7efc370e9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T02:53:25.516 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:53:25 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:53:25.386+0000 7f1d04c4d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T02:53:25.715 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:53:25 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:53:25.655+0000 7efc370e9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T02:53:26.278 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:26.003+0000 7f1ce1898200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T02:53:26.516 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:53:26.245+0000 7f1d04c4d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T02:53:26.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:26.842+0000 7f431ffff640 1 -- 172.21.15.33:0/1969252986 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f43100a28f0 msgr2=0x7f43100a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.33:6800/1453067701 2024-03-19T02:53:26.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:26.842+0000 7f431ffff640 1 --2- 172.21.15.33:0/1969252986 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f43100a28f0 0x7f43100a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-19T02:53:26.924 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:26.590+0000 7f1ce1898200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T02:53:26.924 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:26.694+0000 7f1ce1898200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T02:53:26.924 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:26.922+0000 7f1ce1898200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T02:53:26.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:26.948+0000 7f431dffb640 1 -- 172.21.15.33:0/1969252986 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mgrmap(e 20) ==== 136975+0+0 (secure 0 0 0) 0x7f4314097f10 con 0x7f4320100740 2024-03-19T02:53:26.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:26.949+0000 7f431dffb640 1 -- 172.21.15.33:0/1969252986 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f43100a28f0 msgr2=0x7f43100a4db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T02:53:26.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:26.949+0000 7f431dffb640 1 --2- 172.21.15.33:0/1969252986 >> [v2:172.21.15.33:6800/1453067701,v1:172.21.15.33:6801/1453067701] conn(0x7f43100a28f0 0x7f43100a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:26.965 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:53:26.506+0000 7efc370e9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T02:53:27.111 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:53:26.829+0000 7f1d04c4d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T02:53:27.111 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:53:26.930+0000 7f1d04c4d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T02:53:27.112 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: Active manager daemon x restarted 2024-03-19T02:53:27.112 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: Activating manager daemon x 2024-03-19T02:53:27.112 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: osdmap e98: 12 total, 12 up, 12 in 2024-03-19T02:53:27.112 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: mgrmap e20: x(active, starting, since 0.0127228s), standbys: z, y 2024-03-19T02:53:27.112 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:53:27.112 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:53:27.112 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:53:27.113 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-19T02:53:27.113 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-19T02:53:27.113 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-19T02:53:27.113 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-19T02:53:27.113 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-19T02:53:27.113 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-19T02:53:27.113 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-19T02:53:27.113 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-19T02:53:27.113 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-19T02:53:27.113 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-19T02:53:27.113 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-19T02:53:27.113 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-19T02:53:27.113 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-19T02:53:27.113 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-19T02:53:27.113 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-19T02:53:27.114 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:53:27.114 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:53:27.114 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:53:27.114 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:53:27.114 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:53:27.114 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:53:27.114 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:53:27.114 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:53:27.114 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:53:27.114 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:53:27.114 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:53:27.114 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:53:27.114 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-19T02:53:27.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-19T02:53:27.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-19T02:53:27.115 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:26 smithi155 ceph-mon[35804]: Manager daemon x is now available 2024-03-19T02:53:27.192 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: Active manager daemon x restarted 2024-03-19T02:53:27.192 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: Activating manager daemon x 2024-03-19T02:53:27.192 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: osdmap e98: 12 total, 12 up, 12 in 2024-03-19T02:53:27.192 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: mgrmap e20: x(active, starting, since 0.0127228s), standbys: z, y 2024-03-19T02:53:27.192 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:53:27.192 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:53:27.192 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:53:27.192 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-19T02:53:27.192 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-19T02:53:27.193 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-19T02:53:27.193 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-19T02:53:27.193 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-19T02:53:27.193 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-19T02:53:27.193 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-19T02:53:27.193 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-19T02:53:27.193 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-19T02:53:27.193 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-19T02:53:27.193 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-19T02:53:27.193 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-19T02:53:27.193 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-19T02:53:27.193 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-19T02:53:27.193 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-19T02:53:27.193 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:53:27.193 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:53:27.193 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:53:27.193 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:53:27.194 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:53:27.194 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:53:27.194 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:53:27.194 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:53:27.194 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:53:27.194 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:53:27.194 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:53:27.194 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:53:27.194 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-19T02:53:27.194 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-19T02:53:27.194 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-19T02:53:27.194 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:26 smithi033 ceph-mon[31973]: Manager daemon x is now available 2024-03-19T02:53:27.465 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:53:27 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:53:27.107+0000 7efc370e9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T02:53:27.465 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:53:27 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:53:27.227+0000 7efc370e9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T02:53:27.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: Active manager daemon x restarted 2024-03-19T02:53:27.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: Activating manager daemon x 2024-03-19T02:53:27.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: osdmap e98: 12 total, 12 up, 12 in 2024-03-19T02:53:27.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: mgrmap e20: x(active, starting, since 0.0127228s), standbys: z, y 2024-03-19T02:53:27.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T02:53:27.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T02:53:27.467 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T02:53:27.467 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-19T02:53:27.467 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-19T02:53:27.467 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-19T02:53:27.467 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-19T02:53:27.467 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-19T02:53:27.467 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-19T02:53:27.467 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-19T02:53:27.467 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-19T02:53:27.467 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-19T02:53:27.467 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-19T02:53:27.467 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-19T02:53:27.467 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-19T02:53:27.467 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-19T02:53:27.467 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-19T02:53:27.467 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-19T02:53:27.468 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T02:53:27.468 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T02:53:27.468 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T02:53:27.468 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T02:53:27.468 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T02:53:27.468 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T02:53:27.468 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T02:53:27.468 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T02:53:27.468 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T02:53:27.468 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T02:53:27.468 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T02:53:27.468 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T02:53:27.468 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-19T02:53:27.468 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-19T02:53:27.468 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-19T02:53:27.468 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:26 smithi096 ceph-mon[35946]: Manager daemon x is now available 2024-03-19T02:53:27.516 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 02:53:27 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z[37670]: 2024-03-19T02:53:27.166+0000 7f1d04c4d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T02:53:27.529 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:27 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:27.200+0000 7f1c41227700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:53:27.529 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:27 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:27.200+0000 7f1c41227700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:53:27.529 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:27 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:27.200+0000 7f1c41227700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:53:27.529 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:27 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:27.200+0000 7f1c41227700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:53:27.529 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:27 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:27.200+0000 7f1c41227700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:53:27.529 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:27 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:27.201+0000 7f1c4422d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:53:27.529 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:27 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:27.201+0000 7f1c4422d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:53:27.529 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:27 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:27.201+0000 7f1c4422d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:53:27.529 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:27 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:27.201+0000 7f1c4422d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:53:27.529 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:53:27 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:53:27.201+0000 7f1c4422d700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:53:27.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:27.960+0000 7f431dffb640 1 -- 172.21.15.33:0/1969252986 <== mon.0 v2:172.21.15.33:3300/0 8 ==== mgrmap(e 21) ==== 137102+0+0 (secure 0 0 0) 0x7f4314099560 con 0x7f4320100740 2024-03-19T02:53:27.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:27.960+0000 7f431dffb640 1 --2- 172.21.15.33:0/1969252986 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f43100511d0 0x7f43100515b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:27.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:27.960+0000 7f431dffb640 1 -- 172.21.15.33:0/1969252986 --> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f4314001d30 con 0x7f43100511d0 2024-03-19T02:53:27.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:27.963+0000 7f431ffff640 1 --2- 172.21.15.33:0/1969252986 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f43100511d0 0x7f43100515b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:27.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:27.964+0000 7f431ffff640 1 --2- 172.21.15.33:0/1969252986 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f43100511d0 0x7f43100515b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4320197d80 tx=0x7f430c005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:27.965 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 02:53:27 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y[37366]: 2024-03-19T02:53:27.530+0000 7efc370e9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T02:53:28.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.029+0000 7f431dffb640 1 -- 172.21.15.33:0/1969252986 <== mgr.15618 v2:172.21.15.33:6800/2362023661 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f4314001d30 con 0x7f43100511d0 2024-03-19T02:53:28.030 INFO:teuthology.orchestra.run.smithi033.stdout:Schedule set for path / 2024-03-19T02:53:28.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.031+0000 7f4326abf640 1 -- 172.21.15.33:0/1969252986 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f43100511d0 msgr2=0x7f43100515b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:28.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.031+0000 7f4326abf640 1 --2- 172.21.15.33:0/1969252986 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f43100511d0 0x7f43100515b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4320197d80 tx=0x7f430c005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:28.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.031+0000 7f4326abf640 1 -- 172.21.15.33:0/1969252986 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4320100740 msgr2=0x7f4320196870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:28.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.031+0000 7f4326abf640 1 --2- 172.21.15.33:0/1969252986 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4320100740 0x7f4320196870 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f431400e9c0 tx=0x7f431400ee80 comp rx=0 tx=0).stop 2024-03-19T02:53:28.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.031+0000 7f432505c640 1 -- 172.21.15.33:0/1969252986 reap_dead start 2024-03-19T02:53:28.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.031+0000 7f4326abf640 1 -- 172.21.15.33:0/1969252986 shutdown_connections 2024-03-19T02:53:28.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.031+0000 7f4326abf640 1 -- 172.21.15.33:0/1969252986 >> 172.21.15.33:0/1969252986 conn(0x7f43200fbcb0 msgr2=0x7f4320104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:28.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.031+0000 7f4326abf640 1 -- 172.21.15.33:0/1969252986 shutdown_connections 2024-03-19T02:53:28.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.031+0000 7f4326abf640 1 -- 172.21.15.33:0/1969252986 wait complete. 2024-03-19T02:53:28.043 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-03-19T02:53:28.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.149+0000 7fcbca9f4640 1 -- 172.21.15.33:0/757767249 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbc41022f0 msgr2=0x7fcbc4108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:28.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.149+0000 7fcbca9f4640 1 --2- 172.21.15.33:0/757767249 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbc41022f0 0x7fcbc4108b80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc009be0 tx=0x7fcbbc02bef0 comp rx=0 tx=0).stop 2024-03-19T02:53:28.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.149+0000 7fcbca9f4640 1 -- 172.21.15.33:0/757767249 shutdown_connections 2024-03-19T02:53:28.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.149+0000 7fcbca9f4640 1 --2- 172.21.15.33:0/757767249 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbc41022f0 0x7fcbc4108b80 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:28.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.149+0000 7fcbca9f4640 1 --2- 172.21.15.33:0/757767249 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbc4101930 0x7fcbc4101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:28.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.149+0000 7fcbca9f4640 1 --2- 172.21.15.33:0/757767249 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcbc4100740 0x7fcbc4100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:28.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.149+0000 7fcbca9f4640 1 -- 172.21.15.33:0/757767249 >> 172.21.15.33:0/757767249 conn(0x7fcbc40fbcb0 msgr2=0x7fcbc40fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:28.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.149+0000 7fcbca9f4640 1 -- 172.21.15.33:0/757767249 shutdown_connections 2024-03-19T02:53:28.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.149+0000 7fcbca9f4640 1 -- 172.21.15.33:0/757767249 wait complete. 2024-03-19T02:53:28.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.150+0000 7fcbca9f4640 1 Processor -- start 2024-03-19T02:53:28.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.150+0000 7fcbca9f4640 1 -- start start 2024-03-19T02:53:28.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.150+0000 7fcbca9f4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbc4100740 0x7fcbc4104750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:28.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.150+0000 7fcbca9f4640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcbc4101930 0x7fcbc4102e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:28.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.150+0000 7fcbca9f4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbc41022f0 0x7fcbc4103370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:28.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.150+0000 7fcbca9f4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc410c250 con 0x7fcbc4100740 2024-03-19T02:53:28.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.150+0000 7fcbca9f4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc410c0d0 con 0x7fcbc41022f0 2024-03-19T02:53:28.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.150+0000 7fcbca9f4640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc410c3d0 con 0x7fcbc4101930 2024-03-19T02:53:28.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.150+0000 7fcbc3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbc4100740 0x7fcbc4104750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:28.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.150+0000 7fcbc3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbc4100740 0x7fcbc4104750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59708/0 (socket says 172.21.15.33:59708) 2024-03-19T02:53:28.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.150+0000 7fcbc3fff640 1 -- 172.21.15.33:0/1709089981 learned_addr learned my addr 172.21.15.33:0/1709089981 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:28.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.150+0000 7fcbc37fe640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcbc4101930 0x7fcbc4102e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:28.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.150+0000 7fcbc8f91640 1 --2- 172.21.15.33:0/1709089981 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbc41022f0 0x7fcbc4103370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:28.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.151+0000 7fcbc3fff640 1 -- 172.21.15.33:0/1709089981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcbc4101930 msgr2=0x7fcbc4102e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:28.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.151+0000 7fcbc3fff640 1 --2- 172.21.15.33:0/1709089981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcbc4101930 0x7fcbc4102e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:28.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.151+0000 7fcbc3fff640 1 -- 172.21.15.33:0/1709089981 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbc41022f0 msgr2=0x7fcbc4103370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:28.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.151+0000 7fcbc3fff640 1 --2- 172.21.15.33:0/1709089981 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbc41022f0 0x7fcbc4103370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:28.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.151+0000 7fcbc3fff640 1 -- 172.21.15.33:0/1709089981 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbc4103b70 con 0x7fcbc4100740 2024-03-19T02:53:28.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.151+0000 7fcbc3fff640 1 --2- 172.21.15.33:0/1709089981 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbc4100740 0x7fcbc4104750 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fcbb400e9c0 tx=0x7fcbb400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:28.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.151+0000 7fcbc17fa640 1 -- 172.21.15.33:0/1709089981 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb400ccc0 con 0x7fcbc4100740 2024-03-19T02:53:28.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.151+0000 7fcbc17fa640 1 -- 172.21.15.33:0/1709089981 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcbb40035f0 con 0x7fcbc4100740 2024-03-19T02:53:28.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.151+0000 7fcbca9f4640 1 -- 172.21.15.33:0/1709089981 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbc41a1af0 con 0x7fcbc4100740 2024-03-19T02:53:28.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.151+0000 7fcbc17fa640 1 -- 172.21.15.33:0/1709089981 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb400ce60 con 0x7fcbc4100740 2024-03-19T02:53:28.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.151+0000 7fcbca9f4640 1 -- 172.21.15.33:0/1709089981 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbc41a2010 con 0x7fcbc4100740 2024-03-19T02:53:28.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.152+0000 7fcbca9f4640 1 -- 172.21.15.33:0/1709089981 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbc41a21e0 con 0x7fcbc4100740 2024-03-19T02:53:28.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.153+0000 7fcbc17fa640 1 -- 172.21.15.33:0/1709089981 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 21) ==== 137102+0+0 (secure 0 0 0) 0x7fcbb40037a0 con 0x7fcbc4100740 2024-03-19T02:53:28.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.154+0000 7fcbc17fa640 1 --2- 172.21.15.33:0/1709089981 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fcbac0a26b0 0x7fcbac0a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:28.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.154+0000 7fcbc17fa640 1 -- 172.21.15.33:0/1709089981 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fcbb40cef90 con 0x7fcbc4100740 2024-03-19T02:53:28.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.154+0000 7fcbc37fe640 1 --2- 172.21.15.33:0/1709089981 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fcbac0a26b0 0x7fcbac0a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:28.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.154+0000 7fcbc37fe640 1 --2- 172.21.15.33:0/1709089981 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fcbac0a26b0 0x7fcbac0a4b70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcbc41044c0 tx=0x7fcbb8009260 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:28.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.161+0000 7fcbc17fa640 1 -- 172.21.15.33:0/1709089981 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcbb409bf20 con 0x7fcbc4100740 2024-03-19T02:53:28.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:27 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:53:28.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:27 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:53:28.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:27 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-19T02:53:28.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:27 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-19T02:53:28.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:27 smithi155 ceph-mon[35804]: Standby manager daemon z restarted 2024-03-19T02:53:28.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:27 smithi155 ceph-mon[35804]: Standby manager daemon z started 2024-03-19T02:53:28.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:27 smithi155 ceph-mon[35804]: from='mgr.? 172.21.15.155:0/3766996235' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-19T02:53:28.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:27 smithi155 ceph-mon[35804]: from='mgr.? 172.21.15.155:0/3766996235' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T02:53:28.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:27 smithi155 ceph-mon[35804]: from='mgr.? 172.21.15.155:0/3766996235' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-19T02:53:28.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:27 smithi155 ceph-mon[35804]: from='mgr.? 172.21.15.155:0/3766996235' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T02:53:28.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:27 smithi155 ceph-mon[35804]: Standby manager daemon y restarted 2024-03-19T02:53:28.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:27 smithi155 ceph-mon[35804]: Standby manager daemon y started 2024-03-19T02:53:28.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:27 smithi155 ceph-mon[35804]: from='mgr.? 172.21.15.96:0/80954390' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-19T02:53:28.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:27 smithi155 ceph-mon[35804]: from='mgr.? 172.21.15.96:0/80954390' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T02:53:28.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:27 smithi155 ceph-mon[35804]: from='mgr.? 172.21.15.96:0/80954390' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-19T02:53:28.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:27 smithi155 ceph-mon[35804]: from='mgr.? 172.21.15.96:0/80954390' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T02:53:28.268 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:27 smithi155 ceph-mon[35804]: mgrmap e21: x(active, since 1.01896s), standbys: z, y 2024-03-19T02:53:28.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:27 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:53:28.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:27 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:53:28.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:27 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-19T02:53:28.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:27 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-19T02:53:28.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:27 smithi033 ceph-mon[31973]: Standby manager daemon z restarted 2024-03-19T02:53:28.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:27 smithi033 ceph-mon[31973]: Standby manager daemon z started 2024-03-19T02:53:28.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:27 smithi033 ceph-mon[31973]: from='mgr.? 172.21.15.155:0/3766996235' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-19T02:53:28.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:27 smithi033 ceph-mon[31973]: from='mgr.? 172.21.15.155:0/3766996235' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T02:53:28.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:27 smithi033 ceph-mon[31973]: from='mgr.? 172.21.15.155:0/3766996235' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-19T02:53:28.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:27 smithi033 ceph-mon[31973]: from='mgr.? 172.21.15.155:0/3766996235' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T02:53:28.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:27 smithi033 ceph-mon[31973]: Standby manager daemon y restarted 2024-03-19T02:53:28.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:27 smithi033 ceph-mon[31973]: Standby manager daemon y started 2024-03-19T02:53:28.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:27 smithi033 ceph-mon[31973]: from='mgr.? 172.21.15.96:0/80954390' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-19T02:53:28.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:27 smithi033 ceph-mon[31973]: from='mgr.? 172.21.15.96:0/80954390' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T02:53:28.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:27 smithi033 ceph-mon[31973]: from='mgr.? 172.21.15.96:0/80954390' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-19T02:53:28.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:27 smithi033 ceph-mon[31973]: from='mgr.? 172.21.15.96:0/80954390' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T02:53:28.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:27 smithi033 ceph-mon[31973]: mgrmap e21: x(active, since 1.01896s), standbys: z, y 2024-03-19T02:53:28.346 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:27 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:53:28.346 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:27 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:53:28.346 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:27 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-19T02:53:28.346 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:27 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-19T02:53:28.347 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:27 smithi096 ceph-mon[35946]: Standby manager daemon z restarted 2024-03-19T02:53:28.347 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:27 smithi096 ceph-mon[35946]: Standby manager daemon z started 2024-03-19T02:53:28.347 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:27 smithi096 ceph-mon[35946]: from='mgr.? 172.21.15.155:0/3766996235' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-19T02:53:28.347 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:27 smithi096 ceph-mon[35946]: from='mgr.? 172.21.15.155:0/3766996235' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T02:53:28.347 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:27 smithi096 ceph-mon[35946]: from='mgr.? 172.21.15.155:0/3766996235' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-19T02:53:28.347 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:27 smithi096 ceph-mon[35946]: from='mgr.? 172.21.15.155:0/3766996235' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T02:53:28.347 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:27 smithi096 ceph-mon[35946]: Standby manager daemon y restarted 2024-03-19T02:53:28.347 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:27 smithi096 ceph-mon[35946]: Standby manager daemon y started 2024-03-19T02:53:28.347 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:27 smithi096 ceph-mon[35946]: from='mgr.? 172.21.15.96:0/80954390' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-19T02:53:28.347 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:27 smithi096 ceph-mon[35946]: from='mgr.? 172.21.15.96:0/80954390' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T02:53:28.347 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:27 smithi096 ceph-mon[35946]: from='mgr.? 172.21.15.96:0/80954390' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-19T02:53:28.347 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:27 smithi096 ceph-mon[35946]: from='mgr.? 172.21.15.96:0/80954390' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T02:53:28.347 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:27 smithi096 ceph-mon[35946]: mgrmap e21: x(active, since 1.01896s), standbys: z, y 2024-03-19T02:53:28.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.374+0000 7fcbca9f4640 1 -- 172.21.15.33:0/1709089981 --> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7fcbc40695b0 con 0x7fcbac0a26b0 2024-03-19T02:53:28.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.425+0000 7fcbc17fa640 1 -- 172.21.15.33:0/1709089981 <== mgr.15618 v2:172.21.15.33:6800/2362023661 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7fcbc40695b0 con 0x7fcbac0a26b0 2024-03-19T02:53:28.427 INFO:teuthology.orchestra.run.smithi033.stdout:Retention added to path / 2024-03-19T02:53:28.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.427+0000 7fcbca9f4640 1 -- 172.21.15.33:0/1709089981 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fcbac0a26b0 msgr2=0x7fcbac0a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:28.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.427+0000 7fcbca9f4640 1 --2- 172.21.15.33:0/1709089981 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fcbac0a26b0 0x7fcbac0a4b70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcbc41044c0 tx=0x7fcbb8009260 comp rx=0 tx=0).stop 2024-03-19T02:53:28.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.427+0000 7fcbca9f4640 1 -- 172.21.15.33:0/1709089981 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbc4100740 msgr2=0x7fcbc4104750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:28.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.427+0000 7fcbca9f4640 1 --2- 172.21.15.33:0/1709089981 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbc4100740 0x7fcbc4104750 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fcbb400e9c0 tx=0x7fcbb400edc0 comp rx=0 tx=0).stop 2024-03-19T02:53:28.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.428+0000 7fcbca9f4640 1 -- 172.21.15.33:0/1709089981 shutdown_connections 2024-03-19T02:53:28.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.428+0000 7fcbca9f4640 1 --2- 172.21.15.33:0/1709089981 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fcbac0a26b0 0x7fcbac0a4b70 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:28.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.428+0000 7fcbca9f4640 1 --2- 172.21.15.33:0/1709089981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcbc4101930 0x7fcbc4102e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:28.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.428+0000 7fcbca9f4640 1 --2- 172.21.15.33:0/1709089981 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcbc41022f0 0x7fcbc4103370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:28.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.428+0000 7fcbca9f4640 1 --2- 172.21.15.33:0/1709089981 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcbc4100740 0x7fcbc4104750 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:28.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.428+0000 7fcbca9f4640 1 -- 172.21.15.33:0/1709089981 >> 172.21.15.33:0/1709089981 conn(0x7fcbc40fbcb0 msgr2=0x7fcbc4068ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:28.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.428+0000 7fcbca9f4640 1 -- 172.21.15.33:0/1709089981 shutdown_connections 2024-03-19T02:53:28.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.428+0000 7fcbca9f4640 1 -- 172.21.15.33:0/1709089981 wait complete. 2024-03-19T02:53:28.440 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-19T02:53:28.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.544+0000 7f46d9337640 1 -- 172.21.15.33:0/1122867481 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46d4101410 msgr2=0x7f46d4101890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:28.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.544+0000 7f46d9337640 1 --2- 172.21.15.33:0/1122867481 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46d4101410 0x7f46d4101890 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f46c4009ca0 tx=0x7f46c402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:53:28.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.544+0000 7f46d9337640 1 -- 172.21.15.33:0/1122867481 shutdown_connections 2024-03-19T02:53:28.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.544+0000 7f46d9337640 1 --2- 172.21.15.33:0/1122867481 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46d4101410 0x7f46d4101890 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:28.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.544+0000 7f46d9337640 1 --2- 172.21.15.33:0/1122867481 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46d4100a70 0x7f46d4100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:28.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.544+0000 7f46d9337640 1 --2- 172.21.15.33:0/1122867481 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46d4100110 0x7f46d4100530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:28.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.544+0000 7f46d9337640 1 -- 172.21.15.33:0/1122867481 >> 172.21.15.33:0/1122867481 conn(0x7f46d40fbcb0 msgr2=0x7f46d40fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:28.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.544+0000 7f46d9337640 1 -- 172.21.15.33:0/1122867481 shutdown_connections 2024-03-19T02:53:28.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.545+0000 7f46d9337640 1 -- 172.21.15.33:0/1122867481 wait complete. 2024-03-19T02:53:28.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.545+0000 7f46d9337640 1 Processor -- start 2024-03-19T02:53:28.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.545+0000 7f46d9337640 1 -- start start 2024-03-19T02:53:28.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.546+0000 7f46d9337640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46d4100110 0x7f46d4199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:28.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.546+0000 7f46d9337640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46d4100a70 0x7f46d419a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:28.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.546+0000 7f46d9337640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46d4101410 0x7f46d41a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:28.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.546+0000 7f46d9337640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f46d410f560 con 0x7f46d4100a70 2024-03-19T02:53:28.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.546+0000 7f46d9337640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f46d410f3e0 con 0x7f46d4101410 2024-03-19T02:53:28.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.546+0000 7f46d9337640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f46d410f6e0 con 0x7f46d4100110 2024-03-19T02:53:28.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.546+0000 7f46d27fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46d4100a70 0x7f46d419a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:28.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.546+0000 7f46d2ffd640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46d4100110 0x7f46d4199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:28.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.546+0000 7f46d37fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46d4101410 0x7f46d41a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:28.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.546+0000 7f46d27fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46d4100a70 0x7f46d419a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59728/0 (socket says 172.21.15.33:59728) 2024-03-19T02:53:28.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.546+0000 7f46d27fc640 1 -- 172.21.15.33:0/3069292125 learned_addr learned my addr 172.21.15.33:0/3069292125 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:28.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.546+0000 7f46d37fe640 1 -- 172.21.15.33:0/3069292125 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46d4100110 msgr2=0x7f46d4199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:28.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.546+0000 7f46d37fe640 1 --2- 172.21.15.33:0/3069292125 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46d4100110 0x7f46d4199ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:28.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.546+0000 7f46d37fe640 1 -- 172.21.15.33:0/3069292125 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46d4100a70 msgr2=0x7f46d419a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:28.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.546+0000 7f46d37fe640 1 --2- 172.21.15.33:0/3069292125 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46d4100a70 0x7f46d419a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:28.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.546+0000 7f46d37fe640 1 -- 172.21.15.33:0/3069292125 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46d41a17c0 con 0x7f46d4101410 2024-03-19T02:53:28.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.546+0000 7f46d2ffd640 1 --2- 172.21.15.33:0/3069292125 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46d4100110 0x7f46d4199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:28.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.546+0000 7f46d27fc640 1 --2- 172.21.15.33:0/3069292125 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46d4100a70 0x7f46d419a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:28.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.546+0000 7f46d37fe640 1 --2- 172.21.15.33:0/3069292125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46d4101410 0x7f46d41a1150 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f46c402c630 tx=0x7f46c40328a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:28.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.546+0000 7f46bbfff640 1 -- 172.21.15.33:0/3069292125 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c4041020 con 0x7f46d4101410 2024-03-19T02:53:28.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.547+0000 7f46d9337640 1 -- 172.21.15.33:0/3069292125 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46d41a1a50 con 0x7f46d4101410 2024-03-19T02:53:28.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.547+0000 7f46bbfff640 1 -- 172.21.15.33:0/3069292125 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f46c4003e20 con 0x7f46d4101410 2024-03-19T02:53:28.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.547+0000 7f46bbfff640 1 -- 172.21.15.33:0/3069292125 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c4039040 con 0x7f46d4101410 2024-03-19T02:53:28.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.547+0000 7f46d9337640 1 -- 172.21.15.33:0/3069292125 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46d41a1f90 con 0x7f46d4101410 2024-03-19T02:53:28.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.547+0000 7f46d9337640 1 -- 172.21.15.33:0/3069292125 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46d410ae60 con 0x7f46d4101410 2024-03-19T02:53:28.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.549+0000 7f46bbfff640 1 -- 172.21.15.33:0/3069292125 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 21) ==== 137102+0+0 (secure 0 0 0) 0x7f46c4044020 con 0x7f46d4101410 2024-03-19T02:53:28.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.549+0000 7f46bbfff640 1 --2- 172.21.15.33:0/3069292125 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f46bc0a26b0 0x7f46bc0a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:28.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.549+0000 7f46d2ffd640 1 --2- 172.21.15.33:0/3069292125 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f46bc0a26b0 0x7f46bc0a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:28.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.549+0000 7f46bbfff640 1 -- 172.21.15.33:0/3069292125 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f46c40f0c00 con 0x7f46d4101410 2024-03-19T02:53:28.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.549+0000 7f46d2ffd640 1 --2- 172.21.15.33:0/3069292125 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f46bc0a26b0 0x7f46bc0a4b70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f46cc009fc0 tx=0x7f46cc005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:28.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.553+0000 7f46bbfff640 1 -- 172.21.15.33:0/3069292125 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f46c404b020 con 0x7f46d4101410 2024-03-19T02:53:28.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.763+0000 7f46d9337640 1 -- 172.21.15.33:0/3069292125 --> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f46d4061c60 con 0x7f46bc0a26b0 2024-03-19T02:53:28.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.772+0000 7f46bbfff640 1 -- 172.21.15.33:0/3069292125 <== mgr.15618 v2:172.21.15.33:6800/2362023661 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f46d4061c60 con 0x7f46bc0a26b0 2024-03-19T02:53:28.774 INFO:teuthology.orchestra.run.smithi033.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-19T00:00:00", "created": "2024-03-19T02:53:27", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-03-19T02:53:28.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.774+0000 7f46d9337640 1 -- 172.21.15.33:0/3069292125 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f46bc0a26b0 msgr2=0x7f46bc0a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:28.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.774+0000 7f46d9337640 1 --2- 172.21.15.33:0/3069292125 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f46bc0a26b0 0x7f46bc0a4b70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f46cc009fc0 tx=0x7f46cc005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:28.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.774+0000 7f46d9337640 1 -- 172.21.15.33:0/3069292125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46d4101410 msgr2=0x7f46d41a1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:28.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.774+0000 7f46d9337640 1 --2- 172.21.15.33:0/3069292125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46d4101410 0x7f46d41a1150 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f46c402c630 tx=0x7f46c40328a0 comp rx=0 tx=0).stop 2024-03-19T02:53:28.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.774+0000 7f46d9337640 1 -- 172.21.15.33:0/3069292125 shutdown_connections 2024-03-19T02:53:28.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.774+0000 7f46d9337640 1 --2- 172.21.15.33:0/3069292125 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f46bc0a26b0 0x7f46bc0a4b70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:28.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.774+0000 7f46d9337640 1 --2- 172.21.15.33:0/3069292125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46d4101410 0x7f46d41a1150 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:28.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.774+0000 7f46d9337640 1 --2- 172.21.15.33:0/3069292125 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f46d4100a70 0x7f46d419a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:28.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.774+0000 7f46d9337640 1 --2- 172.21.15.33:0/3069292125 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f46d4100110 0x7f46d4199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:28.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.774+0000 7f46d9337640 1 -- 172.21.15.33:0/3069292125 >> 172.21.15.33:0/3069292125 conn(0x7f46d40fbcb0 msgr2=0x7f46d40fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:28.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.775+0000 7f46d9337640 1 -- 172.21.15.33:0/3069292125 shutdown_connections 2024-03-19T02:53:28.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.775+0000 7f46d9337640 1 -- 172.21.15.33:0/3069292125 wait complete. 2024-03-19T02:53:28.788 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-19T02:53:28.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.902+0000 7fa5f63f3640 1 -- 172.21.15.33:0/1601679912 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa5f0101850 msgr2=0x7fa5f0101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:28.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.902+0000 7fa5f63f3640 1 --2- 172.21.15.33:0/1601679912 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa5f0101850 0x7fa5f0101c70 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0009900 tx=0x7fa5e002bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:28.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.903+0000 7fa5f63f3640 1 -- 172.21.15.33:0/1601679912 shutdown_connections 2024-03-19T02:53:28.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.903+0000 7fa5f63f3640 1 --2- 172.21.15.33:0/1601679912 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5f0103400 0x7fa5f0109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:28.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.903+0000 7fa5f63f3640 1 --2- 172.21.15.33:0/1601679912 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5f0102a40 0x7fa5f0102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:28.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.903+0000 7fa5f63f3640 1 --2- 172.21.15.33:0/1601679912 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa5f0101850 0x7fa5f0101c70 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:28.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.903+0000 7fa5f63f3640 1 -- 172.21.15.33:0/1601679912 >> 172.21.15.33:0/1601679912 conn(0x7fa5f00fcde0 msgr2=0x7fa5f00ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:28.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.903+0000 7fa5f63f3640 1 -- 172.21.15.33:0/1601679912 shutdown_connections 2024-03-19T02:53:28.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.903+0000 7fa5f63f3640 1 -- 172.21.15.33:0/1601679912 wait complete. 2024-03-19T02:53:28.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.903+0000 7fa5f63f3640 1 Processor -- start 2024-03-19T02:53:28.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.903+0000 7fa5f63f3640 1 -- start start 2024-03-19T02:53:28.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.903+0000 7fa5f63f3640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5f0101850 0x7fa5f01979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:28.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.903+0000 7fa5f63f3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa5f0102a40 0x7fa5f0197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:28.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.904+0000 7fa5f63f3640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5f0103400 0x7fa5f019ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:28.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.904+0000 7fa5f63f3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f010d350 con 0x7fa5f0102a40 2024-03-19T02:53:28.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.904+0000 7fa5f63f3640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f010d1d0 con 0x7fa5f0103400 2024-03-19T02:53:28.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.904+0000 7fa5f63f3640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f010d4d0 con 0x7fa5f0101850 2024-03-19T02:53:28.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.904+0000 7fa5ef7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa5f0102a40 0x7fa5f0197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:28.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.904+0000 7fa5ef7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa5f0102a40 0x7fa5f0197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59742/0 (socket says 172.21.15.33:59742) 2024-03-19T02:53:28.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.904+0000 7fa5ef7fe640 1 -- 172.21.15.33:0/1344308945 learned_addr learned my addr 172.21.15.33:0/1344308945 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:28.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.904+0000 7fa5effff640 1 --2- 172.21.15.33:0/1344308945 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5f0101850 0x7fa5f01979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:28.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.904+0000 7fa5f4990640 1 --2- 172.21.15.33:0/1344308945 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5f0103400 0x7fa5f019ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:28.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.904+0000 7fa5ef7fe640 1 -- 172.21.15.33:0/1344308945 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5f0101850 msgr2=0x7fa5f01979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:28.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.904+0000 7fa5ef7fe640 1 --2- 172.21.15.33:0/1344308945 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5f0101850 0x7fa5f01979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:28.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.904+0000 7fa5ef7fe640 1 -- 172.21.15.33:0/1344308945 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5f0103400 msgr2=0x7fa5f019ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:28.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.904+0000 7fa5ef7fe640 1 --2- 172.21.15.33:0/1344308945 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5f0103400 0x7fa5f019ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:28.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.904+0000 7fa5ef7fe640 1 -- 172.21.15.33:0/1344308945 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5f019f5c0 con 0x7fa5f0102a40 2024-03-19T02:53:28.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.904+0000 7fa5ef7fe640 1 --2- 172.21.15.33:0/1344308945 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa5f0102a40 0x7fa5f0197ee0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa5e400ea60 tx=0x7fa5e400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:28.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.904+0000 7fa5ed7fa640 1 -- 172.21.15.33:0/1344308945 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e400cde0 con 0x7fa5f0102a40 2024-03-19T02:53:28.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.904+0000 7fa5f63f3640 1 -- 172.21.15.33:0/1344308945 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5f019f8b0 con 0x7fa5f0102a40 2024-03-19T02:53:28.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.904+0000 7fa5ed7fa640 1 -- 172.21.15.33:0/1344308945 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa5e40035f0 con 0x7fa5f0102a40 2024-03-19T02:53:28.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.904+0000 7fa5f63f3640 1 -- 172.21.15.33:0/1344308945 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5f019fe50 con 0x7fa5f0102a40 2024-03-19T02:53:28.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.904+0000 7fa5ed7fa640 1 -- 172.21.15.33:0/1344308945 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e4010690 con 0x7fa5f0102a40 2024-03-19T02:53:28.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.905+0000 7fa5f63f3640 1 -- 172.21.15.33:0/1344308945 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5f0108b70 con 0x7fa5f0102a40 2024-03-19T02:53:28.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.907+0000 7fa5ed7fa640 1 -- 172.21.15.33:0/1344308945 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 21) ==== 137102+0+0 (secure 0 0 0) 0x7fa5e40037a0 con 0x7fa5f0102a40 2024-03-19T02:53:28.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.907+0000 7fa5ed7fa640 1 --2- 172.21.15.33:0/1344308945 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa5d80a26b0 0x7fa5d80a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:28.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.907+0000 7fa5effff640 1 --2- 172.21.15.33:0/1344308945 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa5d80a26b0 0x7fa5d80a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:28.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.907+0000 7fa5ed7fa640 1 -- 172.21.15.33:0/1344308945 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fa5e40cedf0 con 0x7fa5f0102a40 2024-03-19T02:53:28.909 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.908+0000 7fa5effff640 1 --2- 172.21.15.33:0/1344308945 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa5d80a26b0 0x7fa5d80a4b70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0005ab0 tx=0x7fa5e0005fd0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:28.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:28.913+0000 7fa5ed7fa640 1 -- 172.21.15.33:0/1344308945 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa5e409bcf0 con 0x7fa5f0102a40 2024-03-19T02:53:29.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:29.128+0000 7fa5f63f3640 1 -- 172.21.15.33:0/1344308945 --> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fa5f0061c60 con 0x7fa5d80a26b0 2024-03-19T02:53:29.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:29.136+0000 7fa5ed7fa640 1 -- 172.21.15.33:0/1344308945 <== mgr.15618 v2:172.21.15.33:6800/2362023661 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fa5f0061c60 con 0x7fa5d80a26b0 2024-03-19T02:53:29.137 INFO:teuthology.orchestra.run.smithi033.stdout:/ 1m 6m3h 2024-03-19T02:53:29.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:29.138+0000 7fa5f63f3640 1 -- 172.21.15.33:0/1344308945 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa5d80a26b0 msgr2=0x7fa5d80a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:29.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:29.138+0000 7fa5f63f3640 1 --2- 172.21.15.33:0/1344308945 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa5d80a26b0 0x7fa5d80a4b70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0005ab0 tx=0x7fa5e0005fd0 comp rx=0 tx=0).stop 2024-03-19T02:53:29.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:29.138+0000 7fa5f63f3640 1 -- 172.21.15.33:0/1344308945 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa5f0102a40 msgr2=0x7fa5f0197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:29.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:29.138+0000 7fa5f63f3640 1 --2- 172.21.15.33:0/1344308945 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa5f0102a40 0x7fa5f0197ee0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa5e400ea60 tx=0x7fa5e400ef20 comp rx=0 tx=0).stop 2024-03-19T02:53:29.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:29.138+0000 7fa5f63f3640 1 -- 172.21.15.33:0/1344308945 shutdown_connections 2024-03-19T02:53:29.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:29.138+0000 7fa5f63f3640 1 --2- 172.21.15.33:0/1344308945 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa5d80a26b0 0x7fa5d80a4b70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:29.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:29.138+0000 7fa5f63f3640 1 --2- 172.21.15.33:0/1344308945 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5f0103400 0x7fa5f019ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:29.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:29.138+0000 7fa5f63f3640 1 --2- 172.21.15.33:0/1344308945 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa5f0102a40 0x7fa5f0197ee0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:29.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:29.138+0000 7fa5f63f3640 1 --2- 172.21.15.33:0/1344308945 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa5f0101850 0x7fa5f01979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:29.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:29.138+0000 7fa5f63f3640 1 -- 172.21.15.33:0/1344308945 >> 172.21.15.33:0/1344308945 conn(0x7fa5f00fcde0 msgr2=0x7fa5f0105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:29.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:29.138+0000 7fa5f63f3640 1 -- 172.21.15.33:0/1344308945 shutdown_connections 2024-03-19T02:53:29.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:29.138+0000 7fa5f63f3640 1 -- 172.21.15.33:0/1344308945 wait complete. 2024-03-19T02:53:29.150 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:28 smithi033 ceph-mon[31973]: [19/Mar/2024:02:53:28] ENGINE Bus STARTING 2024-03-19T02:53:29.151 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:28 smithi033 ceph-mon[31973]: [19/Mar/2024:02:53:28] ENGINE Serving on http://172.21.15.33:8765 2024-03-19T02:53:29.151 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:28 smithi033 ceph-mon[31973]: from='client.15678 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:53:29.151 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:28 smithi033 ceph-mon[31973]: [19/Mar/2024:02:53:28] ENGINE Serving on https://172.21.15.33:7150 2024-03-19T02:53:29.151 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:28 smithi033 ceph-mon[31973]: [19/Mar/2024:02:53:28] ENGINE Bus STARTED 2024-03-19T02:53:29.151 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:28 smithi033 ceph-mon[31973]: from='client.25553 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:53:29.152 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-03-19T02:53:29.181 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-03-19T02:53:29.195 INFO:teuthology.run_tasks:Running task workunit... 2024-03-19T02:53:29.247 INFO:tasks.workunit:Pulling workunits from ref e48b1f6c0608b2615e1890cc46286c3877d6e910 2024-03-19T02:53:29.247 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-19T02:53:29.247 INFO:tasks.workunit:timeout=3h 2024-03-19T02:53:29.247 INFO:tasks.workunit:cleanup=True 2024-03-19T02:53:29.248 DEBUG:teuthology.orchestra.run.smithi033:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-19T02:53:29.264 INFO:teuthology.orchestra.run.smithi033.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-19T02:53:29.264 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-03-19T02:53:29.264 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 77h/119d Inode: 1 Links: 2 2024-03-19T02:53:29.264 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T02:53:29.264 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-19T02:53:29.264 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-03-19 02:51:19.783610885 +0000 2024-03-19T02:53:29.264 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-03-19 02:51:19.783610885 +0000 2024-03-19T02:53:29.264 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-03-19 02:53:11.368424706 +0000 2024-03-19T02:53:29.264 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: - 2024-03-19T02:53:29.264 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-19T02:53:29.265 DEBUG:teuthology.orchestra.run.smithi033:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-19T02:53:29.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:28 smithi155 ceph-mon[35804]: [19/Mar/2024:02:53:28] ENGINE Bus STARTING 2024-03-19T02:53:29.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:28 smithi155 ceph-mon[35804]: [19/Mar/2024:02:53:28] ENGINE Serving on http://172.21.15.33:8765 2024-03-19T02:53:29.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:28 smithi155 ceph-mon[35804]: from='client.15678 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:53:29.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:28 smithi155 ceph-mon[35804]: [19/Mar/2024:02:53:28] ENGINE Serving on https://172.21.15.33:7150 2024-03-19T02:53:29.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:28 smithi155 ceph-mon[35804]: [19/Mar/2024:02:53:28] ENGINE Bus STARTED 2024-03-19T02:53:29.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:28 smithi155 ceph-mon[35804]: from='client.25553 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:53:29.338 DEBUG:teuthology.orchestra.run.smithi033:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/batrick/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout e48b1f6c0608b2615e1890cc46286c3877d6e910 2024-03-19T02:53:29.393 INFO:tasks.workunit.client.0.smithi033.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-19T02:53:29.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:28 smithi096 ceph-mon[35946]: [19/Mar/2024:02:53:28] ENGINE Bus STARTING 2024-03-19T02:53:29.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:28 smithi096 ceph-mon[35946]: [19/Mar/2024:02:53:28] ENGINE Serving on http://172.21.15.33:8765 2024-03-19T02:53:29.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:28 smithi096 ceph-mon[35946]: from='client.15678 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:53:29.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:28 smithi096 ceph-mon[35946]: [19/Mar/2024:02:53:28] ENGINE Serving on https://172.21.15.33:7150 2024-03-19T02:53:29.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:28 smithi096 ceph-mon[35946]: [19/Mar/2024:02:53:28] ENGINE Bus STARTED 2024-03-19T02:53:29.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:28 smithi096 ceph-mon[35946]: from='client.25553 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:53:30.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:30 smithi096 ceph-mon[35946]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:53:30.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:30 smithi096 ceph-mon[35946]: from='client.15690 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:53:30.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:30 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:30.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:30 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:30.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:30 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:30.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:30 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:30.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:30 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:30.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:30 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:30.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:30 smithi096 ceph-mon[35946]: mgrmap e22: x(active, since 3s), standbys: z, y 2024-03-19T02:53:30.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:30 smithi155 ceph-mon[35804]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:53:30.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:30 smithi155 ceph-mon[35804]: from='client.15690 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:53:30.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:30 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:30.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:30 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:30.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:30 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:30.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:30 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:30.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:30 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:30.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:30 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:30.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:30 smithi155 ceph-mon[35804]: mgrmap e22: x(active, since 3s), standbys: z, y 2024-03-19T02:53:30.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:30 smithi033 ceph-mon[31973]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:53:30.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:30 smithi033 ceph-mon[31973]: from='client.15690 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T02:53:30.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:30 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:30.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:30 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:30.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:30 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:30.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:30 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:30.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:30 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:30.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:30 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:30.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:30 smithi033 ceph-mon[31973]: mgrmap e22: x(active, since 3s), standbys: z, y 2024-03-19T02:53:31.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:31 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:31.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:31 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:31.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:31 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:31.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:31 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:31 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:31.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:31 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:31 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:31 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:31 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:31 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:31 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:31 smithi096 ceph-mon[35946]: Adjusting osd_memory_target on smithi033 to 177.2M 2024-03-19T02:53:31.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:31 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:31 smithi096 ceph-mon[35946]: Unable to set osd_memory_target on smithi033 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-03-19T02:53:31.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:31 smithi096 ceph-mon[35946]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-03-19T02:53:31.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:31 smithi096 ceph-mon[35946]: Unable to set osd_memory_target on smithi155 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-03-19T02:53:31.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:31 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:31.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:31 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:31.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:31 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:31 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.718 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:31 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.718 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:31 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.718 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:31 smithi096 ceph-mon[35946]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-03-19T02:53:31.718 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:31 smithi096 ceph-mon[35946]: Unable to set osd_memory_target on smithi096 to 184783872: error parsing value: Value '184783872' is below minimum 939524096 2024-03-19T02:53:31.718 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:31 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:53:31.718 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:31 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:53:31.718 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:31 smithi096 ceph-mon[35946]: Updating smithi033:/etc/ceph/ceph.conf 2024-03-19T02:53:31.718 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:31 smithi096 ceph-mon[35946]: Updating smithi096:/etc/ceph/ceph.conf 2024-03-19T02:53:31.718 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:31 smithi096 ceph-mon[35946]: Updating smithi155:/etc/ceph/ceph.conf 2024-03-19T02:53:31.718 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:31 smithi096 ceph-mon[35946]: Updating smithi096:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:53:31.719 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:31 smithi096 ceph-mon[35946]: Updating smithi033:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:53:31.719 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:31 smithi096 ceph-mon[35946]: Updating smithi155:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:53:31.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:31 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:31.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:31 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:31.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:31 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:31.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:31 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:31 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:31.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:31 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:31 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:31 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:31 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:31 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:31 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:31 smithi155 ceph-mon[35804]: Adjusting osd_memory_target on smithi033 to 177.2M 2024-03-19T02:53:31.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:31 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:31 smithi155 ceph-mon[35804]: Unable to set osd_memory_target on smithi033 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-03-19T02:53:31.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:31 smithi155 ceph-mon[35804]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-03-19T02:53:31.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:31 smithi155 ceph-mon[35804]: Unable to set osd_memory_target on smithi155 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-03-19T02:53:31.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:31 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:31.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:31 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:31.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:31 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:31 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:31 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:31 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:31 smithi155 ceph-mon[35804]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-03-19T02:53:31.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:31 smithi155 ceph-mon[35804]: Unable to set osd_memory_target on smithi096 to 184783872: error parsing value: Value '184783872' is below minimum 939524096 2024-03-19T02:53:31.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:31 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:53:31.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:31 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:53:31.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:31 smithi155 ceph-mon[35804]: Updating smithi033:/etc/ceph/ceph.conf 2024-03-19T02:53:31.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:31 smithi155 ceph-mon[35804]: Updating smithi096:/etc/ceph/ceph.conf 2024-03-19T02:53:31.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:31 smithi155 ceph-mon[35804]: Updating smithi155:/etc/ceph/ceph.conf 2024-03-19T02:53:31.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:31 smithi155 ceph-mon[35804]: Updating smithi096:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:53:31.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:31 smithi155 ceph-mon[35804]: Updating smithi033:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:53:31.770 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:31 smithi155 ceph-mon[35804]: Updating smithi155:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:53:31.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:31 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:31.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:31 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:31.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:31 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:31.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:31 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:31 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:31.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:31 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:31 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:31 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:31 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:31 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:31 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:31 smithi033 ceph-mon[31973]: Adjusting osd_memory_target on smithi033 to 177.2M 2024-03-19T02:53:31.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:31 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:31 smithi033 ceph-mon[31973]: Unable to set osd_memory_target on smithi033 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-03-19T02:53:31.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:31 smithi033 ceph-mon[31973]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-03-19T02:53:31.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:31 smithi033 ceph-mon[31973]: Unable to set osd_memory_target on smithi155 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-03-19T02:53:31.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:31 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:31.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:31 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:31.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:31 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:31 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:31 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:31 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T02:53:31.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:31 smithi033 ceph-mon[31973]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-03-19T02:53:31.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:31 smithi033 ceph-mon[31973]: Unable to set osd_memory_target on smithi096 to 184783872: error parsing value: Value '184783872' is below minimum 939524096 2024-03-19T02:53:31.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:31 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:53:31.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:31 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:53:31.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:31 smithi033 ceph-mon[31973]: Updating smithi033:/etc/ceph/ceph.conf 2024-03-19T02:53:31.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:31 smithi033 ceph-mon[31973]: Updating smithi096:/etc/ceph/ceph.conf 2024-03-19T02:53:31.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:31 smithi033 ceph-mon[31973]: Updating smithi155:/etc/ceph/ceph.conf 2024-03-19T02:53:31.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:31 smithi033 ceph-mon[31973]: Updating smithi096:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:53:31.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:31 smithi033 ceph-mon[31973]: Updating smithi033:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:53:31.781 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:31 smithi033 ceph-mon[31973]: Updating smithi155:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T02:53:32.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:32 smithi096 ceph-mon[35946]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:53:32.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:32 smithi096 ceph-mon[35946]: Updating smithi096:/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:53:32.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:32 smithi096 ceph-mon[35946]: Updating smithi155:/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:53:32.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:32 smithi096 ceph-mon[35946]: Updating smithi033:/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:53:32.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:32 smithi096 ceph-mon[35946]: Updating smithi155:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T02:53:32.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:32 smithi096 ceph-mon[35946]: Updating smithi033:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T02:53:32.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:32 smithi096 ceph-mon[35946]: Updating smithi096:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T02:53:32.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:32 smithi096 ceph-mon[35946]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-03-19T02:53:32.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:32 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:32.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:32 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:32.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:32 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:32.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:32 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:32.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:32 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:32.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:32 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:32.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:32 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:32.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:32 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:53:32.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:32 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:32.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:32 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:32.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:32 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:53:32.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:32 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:53:32.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:32 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:33.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:32 smithi155 ceph-mon[35804]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:53:33.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:32 smithi155 ceph-mon[35804]: Updating smithi096:/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:53:33.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:32 smithi155 ceph-mon[35804]: Updating smithi155:/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:53:33.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:32 smithi155 ceph-mon[35804]: Updating smithi033:/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:53:33.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:32 smithi155 ceph-mon[35804]: Updating smithi155:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T02:53:33.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:32 smithi155 ceph-mon[35804]: Updating smithi033:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T02:53:33.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:32 smithi155 ceph-mon[35804]: Updating smithi096:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T02:53:33.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:32 smithi155 ceph-mon[35804]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-03-19T02:53:33.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:32 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:33.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:32 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:33.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:32 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:33.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:32 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:33.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:32 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:33.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:32 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:33.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:32 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:33.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:32 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:53:33.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:32 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:33.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:32 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:33.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:32 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:53:33.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:32 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:53:33.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:32 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:33.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:32 smithi033 ceph-mon[31973]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:53:33.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:32 smithi033 ceph-mon[31973]: Updating smithi096:/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:53:33.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:32 smithi033 ceph-mon[31973]: Updating smithi155:/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:53:33.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:32 smithi033 ceph-mon[31973]: Updating smithi033:/etc/ceph/ceph.client.admin.keyring 2024-03-19T02:53:33.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:32 smithi033 ceph-mon[31973]: Updating smithi155:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T02:53:33.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:32 smithi033 ceph-mon[31973]: Updating smithi033:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T02:53:33.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:32 smithi033 ceph-mon[31973]: Updating smithi096:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T02:53:33.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:32 smithi033 ceph-mon[31973]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-03-19T02:53:33.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:32 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:33.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:32 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:33.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:32 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:33.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:32 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:33.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:32 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:33.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:32 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:33.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:32 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:33.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:32 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:53:33.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:32 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:33.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:32 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:33.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:32 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:53:33.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:32 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:53:33.030 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:32 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:53:33.929 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:33 smithi033 ceph-mon[31973]: scrub summary: idle 2024-03-19T02:53:33.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:33 smithi096 ceph-mon[35946]: scrub summary: idle 2024-03-19T02:53:34.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:33 smithi155 ceph-mon[35804]: scrub summary: idle 2024-03-19T02:53:34.941 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:34 smithi033 ceph-mon[31973]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:53:34.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:34 smithi096 ceph-mon[35946]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:53:35.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:34 smithi155 ceph-mon[35804]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T02:53:36.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:36 smithi096 ceph-mon[35946]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 76 op/s 2024-03-19T02:53:37.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:36 smithi155 ceph-mon[35804]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 76 op/s 2024-03-19T02:53:37.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:36 smithi033 ceph-mon[31973]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 76 op/s 2024-03-19T02:53:38.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:38 smithi096 ceph-mon[35946]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 103 KiB/s wr, 59 op/s 2024-03-19T02:53:39.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:38 smithi155 ceph-mon[35804]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 103 KiB/s wr, 59 op/s 2024-03-19T02:53:39.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:38 smithi033 ceph-mon[31973]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 103 KiB/s wr, 59 op/s 2024-03-19T02:53:40.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:40 smithi096 ceph-mon[35946]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 85 KiB/s wr, 51 op/s 2024-03-19T02:53:41.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:40 smithi155 ceph-mon[35804]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 85 KiB/s wr, 51 op/s 2024-03-19T02:53:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:40 smithi033 ceph-mon[31973]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 85 KiB/s wr, 51 op/s 2024-03-19T02:53:42.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:42 smithi096 ceph-mon[35946]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-19T02:53:43.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:42 smithi155 ceph-mon[35804]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-19T02:53:43.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:42 smithi033 ceph-mon[31973]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-19T02:53:44.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:44 smithi096 ceph-mon[35946]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-19T02:53:45.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:44 smithi155 ceph-mon[35804]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-19T02:53:45.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:44 smithi033 ceph-mon[31973]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-19T02:53:46.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:46 smithi096 ceph-mon[35946]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 49 op/s 2024-03-19T02:53:47.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:46 smithi155 ceph-mon[35804]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 49 op/s 2024-03-19T02:53:47.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:46 smithi033 ceph-mon[31973]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 49 op/s 2024-03-19T02:53:48.924 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:53:48.924 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:53:48.952 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:48 smithi033 ceph-mon[31973]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 597 B/s wr, 5 op/s 2024-03-19T02:53:48.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:48 smithi096 ceph-mon[35946]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 597 B/s wr, 5 op/s 2024-03-19T02:53:49.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:48 smithi155 ceph-mon[35804]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 597 B/s wr, 5 op/s 2024-03-19T02:53:49.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.037+0000 7fd9bff76640 1 -- 172.21.15.33:0/2207019560 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9b8100690 msgr2=0x7fd9b8100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.037+0000 7fd9bff76640 1 --2- 172.21.15.33:0/2207019560 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9b8100690 0x7fd9b8100b10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd9a8009930 tx=0x7fd9a802bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:49.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.037+0000 7fd9bff76640 1 -- 172.21.15.33:0/2207019560 shutdown_connections 2024-03-19T02:53:49.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.037+0000 7fd9bff76640 1 --2- 172.21.15.33:0/2207019560 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9b8100690 0x7fd9b8100b10 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.037+0000 7fd9bff76640 1 --2- 172.21.15.33:0/2207019560 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd9b8101050 0x7fd9b810d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.037+0000 7fd9bff76640 1 --2- 172.21.15.33:0/2207019560 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9b81066a0 0x7fd9b8106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.037+0000 7fd9bff76640 1 -- 172.21.15.33:0/2207019560 >> 172.21.15.33:0/2207019560 conn(0x7fd9b8077820 msgr2=0x7fd9b8077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:49.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.037+0000 7fd9bff76640 1 -- 172.21.15.33:0/2207019560 shutdown_connections 2024-03-19T02:53:49.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.037+0000 7fd9bff76640 1 -- 172.21.15.33:0/2207019560 wait complete. 2024-03-19T02:53:49.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.038+0000 7fd9bff76640 1 Processor -- start 2024-03-19T02:53:49.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.038+0000 7fd9bff76640 1 -- start start 2024-03-19T02:53:49.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.038+0000 7fd9bff76640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9b8100690 0x7fd9b8197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.038+0000 7fd9bff76640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd9b8101050 0x7fd9b8198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.038+0000 7fd9bff76640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9b81066a0 0x7fd9b819c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.038+0000 7fd9bff76640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b8110960 con 0x7fd9b8101050 2024-03-19T02:53:49.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.038+0000 7fd9bff76640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b81107e0 con 0x7fd9b8100690 2024-03-19T02:53:49.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.038+0000 7fd9bff76640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b8110ae0 con 0x7fd9b81066a0 2024-03-19T02:53:49.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.038+0000 7fd9bd511640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd9b8101050 0x7fd9b8198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.038+0000 7fd9bdd12640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9b8100690 0x7fd9b8197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.038+0000 7fd9bd511640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd9b8101050 0x7fd9b8198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34238/0 (socket says 172.21.15.33:34238) 2024-03-19T02:53:49.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.038+0000 7fd9bd511640 1 -- 172.21.15.33:0/3034140839 learned_addr learned my addr 172.21.15.33:0/3034140839 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:49.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.038+0000 7fd9be513640 1 --2- 172.21.15.33:0/3034140839 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9b81066a0 0x7fd9b819c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.038+0000 7fd9bd511640 1 -- 172.21.15.33:0/3034140839 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9b81066a0 msgr2=0x7fd9b819c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.038+0000 7fd9bd511640 1 --2- 172.21.15.33:0/3034140839 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9b81066a0 0x7fd9b819c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.038+0000 7fd9bd511640 1 -- 172.21.15.33:0/3034140839 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9b8100690 msgr2=0x7fd9b8197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.038+0000 7fd9bd511640 1 --2- 172.21.15.33:0/3034140839 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9b8100690 0x7fd9b8197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.038+0000 7fd9bd511640 1 -- 172.21.15.33:0/3034140839 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9b819cfc0 con 0x7fd9b8101050 2024-03-19T02:53:49.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.038+0000 7fd9bdd12640 1 --2- 172.21.15.33:0/3034140839 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9b8100690 0x7fd9b8197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:49.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.039+0000 7fd9bd511640 1 --2- 172.21.15.33:0/3034140839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd9b8101050 0x7fd9b8198510 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fd9a8009230 tx=0x7fd9a802c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:49.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.039+0000 7fd9aeffd640 1 -- 172.21.15.33:0/3034140839 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a8043020 con 0x7fd9b8101050 2024-03-19T02:53:49.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.039+0000 7fd9bff76640 1 -- 172.21.15.33:0/3034140839 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9b819d250 con 0x7fd9b8101050 2024-03-19T02:53:49.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.039+0000 7fd9aeffd640 1 -- 172.21.15.33:0/3034140839 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9a802ca70 con 0x7fd9b8101050 2024-03-19T02:53:49.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.039+0000 7fd9aeffd640 1 -- 172.21.15.33:0/3034140839 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a8003a50 con 0x7fd9b8101050 2024-03-19T02:53:49.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.039+0000 7fd9bff76640 1 -- 172.21.15.33:0/3034140839 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9b81a4b40 con 0x7fd9b8101050 2024-03-19T02:53:49.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.041+0000 7fd9aeffd640 1 -- 172.21.15.33:0/3034140839 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 23) ==== 137466+0+0 (secure 0 0 0) 0x7fd9a8046020 con 0x7fd9b8101050 2024-03-19T02:53:49.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.041+0000 7fd9c2c35740 1 Processor -- start 2024-03-19T02:53:49.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.042+0000 7fd9c2c35740 1 -- start start 2024-03-19T02:53:49.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.042+0000 7fd9aeffd640 1 --2- 172.21.15.33:0/3034140839 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd9a40a2a00 0x7fd9a40a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.042+0000 7fd9c2c35740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f7ad949a30 0x55f7ad949e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.042+0000 7fd9c2c35740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7ad94a350 0x55f7ad9529f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.042+0000 7fd9be513640 1 --2- 172.21.15.33:0/3034140839 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd9a40a2a00 0x7fd9a40a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.042+0000 7fd9c2c35740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f7ad952f30 0x55f7ad955320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.042+0000 7fd9aeffd640 1 -- 172.21.15.33:0/3034140839 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fd9a80f1250 con 0x7fd9b8101050 2024-03-19T02:53:49.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.042+0000 7fd9c2c35740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f7ad8ad500 con 0x55f7ad94a350 2024-03-19T02:53:49.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.042+0000 7fd9c2c35740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55f7ad855be0 con 0x55f7ad952f30 2024-03-19T02:53:49.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.042+0000 7fd9c2c35740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f7ad8a95a0 con 0x55f7ad949a30 2024-03-19T02:53:49.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.042+0000 7fd9bd511640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7ad94a350 0x55f7ad9529f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.042+0000 7fd9bd511640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7ad94a350 0x55f7ad9529f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34252/0 (socket says 172.21.15.33:34252) 2024-03-19T02:53:49.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.042+0000 7fd9bd511640 1 -- 172.21.15.33:0/3193703406 learned_addr learned my addr 172.21.15.33:0/3193703406 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:49.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.042+0000 7fd9be513640 1 --2- 172.21.15.33:0/3193703406 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f7ad952f30 0x55f7ad955320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.042+0000 7fd9bdd12640 1 --2- 172.21.15.33:0/3193703406 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f7ad949a30 0x55f7ad949e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.043+0000 7fd9bd511640 1 -- 172.21.15.33:0/3193703406 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f7ad949a30 msgr2=0x55f7ad949e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:49.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.043+0000 7fd9bd511640 1 --2- 172.21.15.33:0/3193703406 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f7ad949a30 0x55f7ad949e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.043+0000 7fd9bd511640 1 -- 172.21.15.33:0/3193703406 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f7ad952f30 msgr2=0x55f7ad955320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:49.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.043+0000 7fd9bd511640 1 --2- 172.21.15.33:0/3193703406 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f7ad952f30 0x55f7ad955320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.043+0000 7fd9bd511640 1 -- 172.21.15.33:0/3193703406 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7ad877000 con 0x55f7ad94a350 2024-03-19T02:53:49.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.043+0000 7fd9be513640 1 --2- 172.21.15.33:0/3034140839 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd9a40a2a00 0x7fd9a40a4ec0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0005fd0 tx=0x7fd9b0011bc0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:49.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.043+0000 7fd9bd511640 1 --2- 172.21.15.33:0/3193703406 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7ad94a350 0x55f7ad9529f0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fd9a80ee710 tx=0x7fd9a80bd500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ae202272c709c36 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:49.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.043+0000 7fd9a3fff640 1 -- 172.21.15.33:0/3193703406 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a80b9040 con 0x55f7ad94a350 2024-03-19T02:53:49.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.043+0000 7fd9a3fff640 1 -- 172.21.15.33:0/3193703406 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9a80bd600 con 0x55f7ad94a350 2024-03-19T02:53:49.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.043+0000 7fd9a3fff640 1 -- 172.21.15.33:0/3193703406 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a80053d0 con 0x55f7ad94a350 2024-03-19T02:53:49.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.044+0000 7fd9c2c35740 1 -- 172.21.15.33:0/3193703406 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7ad94a350 msgr2=0x55f7ad9529f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.044+0000 7fd9c2c35740 1 --2- 172.21.15.33:0/3193703406 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7ad94a350 0x55f7ad9529f0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fd9a80ee710 tx=0x7fd9a80bd500 comp rx=0 tx=0).stop 2024-03-19T02:53:49.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.044+0000 7fd9c2c35740 1 -- 172.21.15.33:0/3193703406 shutdown_connections 2024-03-19T02:53:49.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.044+0000 7fd9c2c35740 1 --2- 172.21.15.33:0/3193703406 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f7ad952f30 0x55f7ad955320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.044+0000 7fd9c2c35740 1 --2- 172.21.15.33:0/3193703406 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7ad94a350 0x55f7ad9529f0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.044+0000 7fd9c2c35740 1 --2- 172.21.15.33:0/3193703406 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f7ad949a30 0x55f7ad949e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.044+0000 7fd9c2c35740 1 -- 172.21.15.33:0/3193703406 >> 172.21.15.33:0/3193703406 conn(0x55f7ad944160 msgr2=0x55f7ad944540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:49.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.044+0000 7fd9c2c35740 1 -- 172.21.15.33:0/3193703406 shutdown_connections 2024-03-19T02:53:49.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.044+0000 7fd9c2c35740 1 -- 172.21.15.33:0/3193703406 wait complete. 2024-03-19T02:53:49.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.044+0000 7fd9c2c35740 1 -- start start 2024-03-19T02:53:49.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.044+0000 7fd9c2c35740 1 Processor -- start 2024-03-19T02:53:49.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.045+0000 7fd9c2c35740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7ad949a30 0x55f7ad94ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.045+0000 7fd9c2c35740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f7ad94a350 0x55f7ad9504a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.045+0000 7fd9c2c35740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f7ad952f30 0x55f7ad9509e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.045+0000 7fd9c2c35740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f7ad955ae0 con 0x55f7ad949a30 2024-03-19T02:53:49.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.045+0000 7fd9c2c35740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55f7ad869160 con 0x55f7ad952f30 2024-03-19T02:53:49.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.045+0000 7fd9c2c35740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f7ad955c60 con 0x55f7ad94a350 2024-03-19T02:53:49.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.045+0000 7fd9c2c35740 10 client.0 _ll_register_callbacks cb 0x55f7ad8a5a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:53:49.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.045+0000 7fd9bdd12640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7ad949a30 0x55f7ad94ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.045+0000 7fd9bdd12640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7ad949a30 0x55f7ad94ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34268/0 (socket says 172.21.15.33:34268) 2024-03-19T02:53:49.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.045+0000 7fd9bdd12640 1 -- 172.21.15.33:0/2187729970 learned_addr learned my addr 172.21.15.33:0/2187729970 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:49.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.045+0000 7fd9bd511640 1 --2- 172.21.15.33:0/2187729970 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f7ad94a350 0x55f7ad9504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.045+0000 7fd9be513640 1 --2- 172.21.15.33:0/2187729970 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f7ad952f30 0x55f7ad9509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.045+0000 7fd9bdd12640 1 -- 172.21.15.33:0/2187729970 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f7ad94a350 msgr2=0x55f7ad9504a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:49.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.045+0000 7fd9bdd12640 1 --2- 172.21.15.33:0/2187729970 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f7ad94a350 0x55f7ad9504a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.045+0000 7fd9bdd12640 1 -- 172.21.15.33:0/2187729970 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f7ad952f30 msgr2=0x55f7ad9509e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:49.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.045+0000 7fd9bdd12640 1 --2- 172.21.15.33:0/2187729970 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f7ad952f30 0x55f7ad9509e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.045+0000 7fd9bdd12640 1 -- 172.21.15.33:0/2187729970 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7ad68b7d0 con 0x55f7ad949a30 2024-03-19T02:53:49.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.045+0000 7fd9be513640 1 --2- 172.21.15.33:0/2187729970 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f7ad952f30 0x55f7ad9509e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:49.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.045+0000 7fd9bdd12640 1 --2- 172.21.15.33:0/2187729970 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7ad949a30 0x55f7ad94ff60 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4009e90 tx=0x7fd9b400e780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=159e3535b4f65b70 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:49.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.045+0000 7fd9a17fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:53:49.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.046+0000 7fd9a17fa640 1 -- 172.21.15.33:0/2187729970 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b4023060 con 0x55f7ad949a30 2024-03-19T02:53:49.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.046+0000 7fd9a17fa640 1 -- 172.21.15.33:0/2187729970 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fd9b400d750 con 0x55f7ad949a30 2024-03-19T02:53:49.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.046+0000 7fd9c2c35740 1 -- 172.21.15.33:0/2187729970 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9a80bd600 con 0x55f7ad949a30 2024-03-19T02:53:49.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.046+0000 7fd9a17fa640 1 -- 172.21.15.33:0/2187729970 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd9b402a050 con 0x55f7ad949a30 2024-03-19T02:53:49.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.046+0000 7fd9c2c35740 10 client.15708 fetch_fsmap learned FSMap version 25 2024-03-19T02:53:49.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.046+0000 7fd9c2c35740 1 -- 172.21.15.33:0/2187729970 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f7ad5c13f0 con 0x55f7ad949a30 2024-03-19T02:53:49.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.046+0000 7fd9a17fa640 1 -- 172.21.15.33:0/2187729970 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9a80bd600 con 0x55f7ad949a30 2024-03-19T02:53:49.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.046+0000 7fd9a17fa640 1 -- 172.21.15.33:0/2187729970 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fd9b401e8e0 con 0x55f7ad949a30 2024-03-19T02:53:49.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.046+0000 7fd9c2c35740 10 client.15708 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:53:49.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.046+0000 7fd9c2c35740 10 client.15708 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:53:49.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.046+0000 7fd9c2c35740 20 client.15708 populate_metadata read hostname 'smithi033' 2024-03-19T02:53:49.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.046+0000 7fd9c2c35740 1 --2- 172.21.15.33:0/2187729970 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f7ad963040 0x55f7ad965490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.046+0000 7fd9c2c35740 4 client.15708 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:53:49.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.046+0000 7fd9c2c35740 1 -- 172.21.15.33:0/2187729970 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f7ad8baa40 con 0x55f7ad963040 2024-03-19T02:53:49.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.046+0000 7fd9bd511640 1 --2- 172.21.15.33:0/2187729970 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f7ad963040 0x55f7ad965490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.047+0000 7fd9a17fa640 10 client.15708 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:53:49.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.047+0000 7fd9bd511640 1 --2- 172.21.15.33:0/2187729970 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f7ad963040 0x55f7ad965490 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=364d79e8d7c8335a server_cookie=19265ff5e833bf9 in_seq=0 out_seq=0 2024-03-19T02:53:49.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.048+0000 7fd9a17fa640 1 -- 172.21.15.33:0/2187729970 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f7ad8baa40 con 0x55f7ad963040 2024-03-19T02:53:49.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.048+0000 7fd9a17fa640 10 client.15708 handle_command_reply: tid=0 2024-03-19T02:53:49.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.048+0000 7fd9c2c35740 1 client.15708 shutdown 2024-03-19T02:53:49.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.048+0000 7fd9c2c35740 1 -- 172.21.15.33:0/2187729970 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7ad949a30 msgr2=0x55f7ad94ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.048+0000 7fd9c2c35740 1 --2- 172.21.15.33:0/2187729970 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7ad949a30 0x55f7ad94ff60 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4009e90 tx=0x7fd9b400e780 comp rx=0 tx=0).stop 2024-03-19T02:53:49.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.048+0000 7fd9c2c35740 1 -- 172.21.15.33:0/2187729970 shutdown_connections 2024-03-19T02:53:49.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.048+0000 7fd9c2c35740 1 --2- 172.21.15.33:0/2187729970 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f7ad963040 0x55f7ad965490 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.048+0000 7fd9c2c35740 1 --2- 172.21.15.33:0/2187729970 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f7ad94a350 0x55f7ad9504a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.048+0000 7fd9a17fa640 0 client.15708 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:53:49.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.048+0000 7fd9c2c35740 1 --2- 172.21.15.33:0/2187729970 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f7ad952f30 0x55f7ad9509e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.048+0000 7fd9c2c35740 1 --2- 172.21.15.33:0/2187729970 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7ad949a30 0x55f7ad94ff60 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.048+0000 7fd9c2c35740 1 -- 172.21.15.33:0/2187729970 >> 172.21.15.33:0/2187729970 conn(0x55f7ad944160 msgr2=0x55f7ad945eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:49.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.048+0000 7fd9c2c35740 1 -- 172.21.15.33:0/2187729970 shutdown_connections 2024-03-19T02:53:49.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.048+0000 7fd9c2c35740 1 -- 172.21.15.33:0/2187729970 wait complete. 2024-03-19T02:53:49.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.048+0000 7fd9c2c35740 20 client.15708 trim_cache size 0 max 16384 2024-03-19T02:53:49.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.066+0000 7fd9c2c35740 1 Processor -- start 2024-03-19T02:53:49.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.066+0000 7fd9c2c35740 1 -- start start 2024-03-19T02:53:49.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.066+0000 7fd9c2c35740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f7ad949a30 0x55f7ad952690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.066+0000 7fd9c2c35740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7ad94a350 0x55f7ad94f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.066+0000 7fd9c2c35740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f7ad952f30 0x55f7ad95e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.066+0000 7fd9c2c35740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b81af270 con 0x55f7ad94a350 2024-03-19T02:53:49.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.066+0000 7fd9c2c35740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b81acb40 con 0x55f7ad952f30 2024-03-19T02:53:49.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.066+0000 7fd9c2c35740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b81aa970 con 0x55f7ad949a30 2024-03-19T02:53:49.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.066+0000 7fd9bd511640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7ad94a350 0x55f7ad94f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.066+0000 7fd9bd511640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7ad94a350 0x55f7ad94f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34284/0 (socket says 172.21.15.33:34284) 2024-03-19T02:53:49.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.066+0000 7fd9bd511640 1 -- 172.21.15.33:0/3271692514 learned_addr learned my addr 172.21.15.33:0/3271692514 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:49.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.066+0000 7fd9be513640 1 --2- 172.21.15.33:0/3271692514 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f7ad952f30 0x55f7ad95e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.066+0000 7fd9bdd12640 1 --2- 172.21.15.33:0/3271692514 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f7ad949a30 0x55f7ad952690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.066+0000 7fd9bd511640 1 -- 172.21.15.33:0/3271692514 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f7ad949a30 msgr2=0x55f7ad952690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:49.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.066+0000 7fd9bd511640 1 --2- 172.21.15.33:0/3271692514 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f7ad949a30 0x55f7ad952690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.066+0000 7fd9bd511640 1 -- 172.21.15.33:0/3271692514 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f7ad952f30 msgr2=0x55f7ad95e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:49.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.066+0000 7fd9bd511640 1 --2- 172.21.15.33:0/3271692514 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f7ad952f30 0x55f7ad95e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.066+0000 7fd9bd511640 1 -- 172.21.15.33:0/3271692514 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7ad5810f0 con 0x55f7ad94a350 2024-03-19T02:53:49.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.066+0000 7fd9be513640 1 --2- 172.21.15.33:0/3271692514 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f7ad952f30 0x55f7ad95e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:49.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.067+0000 7fd9bd511640 1 --2- 172.21.15.33:0/3271692514 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7ad94a350 0x55f7ad94f0c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fd9a80b9a10 tx=0x7fd9a8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e95768e7adf68ae7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:49.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.067+0000 7fd9a37fe640 1 -- 172.21.15.33:0/3271692514 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a80b9040 con 0x55f7ad94a350 2024-03-19T02:53:49.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.067+0000 7fd9a37fe640 1 -- 172.21.15.33:0/3271692514 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9a80bd600 con 0x55f7ad94a350 2024-03-19T02:53:49.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.067+0000 7fd9a37fe640 1 -- 172.21.15.33:0/3271692514 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a80053d0 con 0x55f7ad94a350 2024-03-19T02:53:49.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.067+0000 7fd9c2c35740 1 -- 172.21.15.33:0/3271692514 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7ad94a350 msgr2=0x55f7ad94f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.067+0000 7fd9c2c35740 1 --2- 172.21.15.33:0/3271692514 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7ad94a350 0x55f7ad94f0c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fd9a80b9a10 tx=0x7fd9a8009260 comp rx=0 tx=0).stop 2024-03-19T02:53:49.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.068+0000 7fd9c2c35740 1 -- 172.21.15.33:0/3271692514 shutdown_connections 2024-03-19T02:53:49.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.068+0000 7fd9c2c35740 1 --2- 172.21.15.33:0/3271692514 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f7ad952f30 0x55f7ad95e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.068+0000 7fd9c2c35740 1 --2- 172.21.15.33:0/3271692514 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7ad94a350 0x55f7ad94f0c0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.068+0000 7fd9c2c35740 1 --2- 172.21.15.33:0/3271692514 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f7ad949a30 0x55f7ad952690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.068+0000 7fd9c2c35740 1 -- 172.21.15.33:0/3271692514 >> 172.21.15.33:0/3271692514 conn(0x55f7ad944160 msgr2=0x55f7ad942950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:49.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.068+0000 7fd9c2c35740 1 -- 172.21.15.33:0/3271692514 shutdown_connections 2024-03-19T02:53:49.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.068+0000 7fd9c2c35740 1 -- 172.21.15.33:0/3271692514 wait complete. 2024-03-19T02:53:49.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.068+0000 7fd9c2c35740 1 -- start start 2024-03-19T02:53:49.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.068+0000 7fd9c2c35740 1 Processor -- start 2024-03-19T02:53:49.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.068+0000 7fd9c2c35740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f7ad949a30 0x55f7ad95acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.068+0000 7fd9c2c35740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7ad94a350 0x55f7ad95b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.068+0000 7fd9c2c35740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f7ad952f30 0x55f7ad956260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.068+0000 7fd9c2c35740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b81ab000 con 0x55f7ad94a350 2024-03-19T02:53:49.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.068+0000 7fd9c2c35740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b81ac470 con 0x55f7ad949a30 2024-03-19T02:53:49.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.068+0000 7fd9c2c35740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b81ab6d0 con 0x55f7ad952f30 2024-03-19T02:53:49.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.068+0000 7fd9c2c35740 10 client.0 _ll_register_callbacks cb 0x7fd9b4009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:53:49.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.069+0000 7fd9bd511640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7ad94a350 0x55f7ad95b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.069+0000 7fd9bdd12640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f7ad949a30 0x55f7ad95acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.069+0000 7fd9bd511640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7ad94a350 0x55f7ad95b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34292/0 (socket says 172.21.15.33:34292) 2024-03-19T02:53:49.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.069+0000 7fd9bdd12640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f7ad949a30 0x55f7ad95acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:58018/0 (socket says 172.21.15.33:58018) 2024-03-19T02:53:49.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.069+0000 7fd9bd511640 1 -- 172.21.15.33:0/3189904773 learned_addr learned my addr 172.21.15.33:0/3189904773 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:49.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.069+0000 7fd9be513640 1 --2- 172.21.15.33:0/3189904773 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f7ad952f30 0x55f7ad956260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.069+0000 7fd9bd511640 1 -- 172.21.15.33:0/3189904773 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f7ad952f30 msgr2=0x55f7ad956260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:49.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.069+0000 7fd9bd511640 1 --2- 172.21.15.33:0/3189904773 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f7ad952f30 0x55f7ad956260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.069+0000 7fd9bd511640 1 -- 172.21.15.33:0/3189904773 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f7ad949a30 msgr2=0x55f7ad95acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:49.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.069+0000 7fd9bd511640 1 --2- 172.21.15.33:0/3189904773 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f7ad949a30 0x55f7ad95acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.069+0000 7fd9bd511640 1 -- 172.21.15.33:0/3189904773 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7ad956a30 con 0x55f7ad94a350 2024-03-19T02:53:49.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.069+0000 7fd9bdd12640 1 --2- 172.21.15.33:0/3189904773 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f7ad949a30 0x55f7ad95acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:49.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.069+0000 7fd9bd511640 1 --2- 172.21.15.33:0/3189904773 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7ad94a350 0x55f7ad95b230 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fd9a80bd300 tx=0x7fd9a80098a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10496909c5f8cd40 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:49.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.069+0000 7fd9a17fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:53:49.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.069+0000 7fd9a17fa640 1 -- 172.21.15.33:0/3189904773 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a80b9040 con 0x55f7ad94a350 2024-03-19T02:53:49.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.069+0000 7fd9a17fa640 1 -- 172.21.15.33:0/3189904773 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fd9a80053d0 con 0x55f7ad94a350 2024-03-19T02:53:49.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.069+0000 7fd9c2c35740 1 -- 172.21.15.33:0/3189904773 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9a80bd600 con 0x55f7ad94a350 2024-03-19T02:53:49.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.069+0000 7fd9a17fa640 1 -- 172.21.15.33:0/3189904773 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd9a80c5da0 con 0x55f7ad94a350 2024-03-19T02:53:49.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.069+0000 7fd9c2c35740 10 client.15714 fetch_fsmap learned FSMap version 25 2024-03-19T02:53:49.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.069+0000 7fd9c2c35740 1 -- 172.21.15.33:0/3189904773 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f7ad9c9000 con 0x55f7ad94a350 2024-03-19T02:53:49.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.070+0000 7fd9a17fa640 1 -- 172.21.15.33:0/3189904773 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9a80bd600 con 0x55f7ad94a350 2024-03-19T02:53:49.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.070+0000 7fd9a17fa640 1 -- 172.21.15.33:0/3189904773 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fd9a80c5910 con 0x55f7ad94a350 2024-03-19T02:53:49.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.070+0000 7fd9c2c35740 10 client.15714 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:53:49.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.070+0000 7fd9c2c35740 10 client.15714 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:53:49.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.070+0000 7fd9c2c35740 20 client.15714 populate_metadata read hostname 'smithi033' 2024-03-19T02:53:49.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.070+0000 7fd9c2c35740 1 --2- 172.21.15.33:0/3189904773 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f7ad9c9470 0x55f7ad9cb890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.070+0000 7fd9c2c35740 4 client.15714 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:53:49.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.070+0000 7fd9c2c35740 1 -- 172.21.15.33:0/3189904773 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f7ad8b4d90 con 0x55f7ad9c9470 2024-03-19T02:53:49.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.070+0000 7fd9bdd12640 1 --2- 172.21.15.33:0/3189904773 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f7ad9c9470 0x55f7ad9cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.070+0000 7fd9a17fa640 10 client.15714 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:53:49.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.070+0000 7fd9bdd12640 1 --2- 172.21.15.33:0/3189904773 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f7ad9c9470 0x55f7ad9cb890 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa062aeecdc33008 server_cookie=ac74ad7b0e82d4a4 in_seq=0 out_seq=0 2024-03-19T02:53:49.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.071+0000 7fd9a17fa640 1 -- 172.21.15.33:0/3189904773 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f7ad8b4d90 con 0x55f7ad9c9470 2024-03-19T02:53:49.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.071+0000 7fd9a17fa640 10 client.15714 handle_command_reply: tid=0 2024-03-19T02:53:49.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.071+0000 7fd9c2c35740 1 client.15714 shutdown 2024-03-19T02:53:49.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.071+0000 7fd9c2c35740 1 -- 172.21.15.33:0/3189904773 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7ad94a350 msgr2=0x55f7ad95b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.071+0000 7fd9c2c35740 1 --2- 172.21.15.33:0/3189904773 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7ad94a350 0x55f7ad95b230 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fd9a80bd300 tx=0x7fd9a80098a0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.071+0000 7fd9c2c35740 1 -- 172.21.15.33:0/3189904773 shutdown_connections 2024-03-19T02:53:49.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.071+0000 7fd9c2c35740 1 --2- 172.21.15.33:0/3189904773 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f7ad9c9470 0x55f7ad9cb890 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.071+0000 7fd9c2c35740 1 --2- 172.21.15.33:0/3189904773 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f7ad952f30 0x55f7ad956260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.071+0000 7fd9c2c35740 1 --2- 172.21.15.33:0/3189904773 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f7ad94a350 0x55f7ad95b230 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.071+0000 7fd9a17fa640 0 client.15714 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:53:49.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.071+0000 7fd9c2c35740 1 --2- 172.21.15.33:0/3189904773 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f7ad949a30 0x55f7ad95acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.071+0000 7fd9c2c35740 1 -- 172.21.15.33:0/3189904773 >> 172.21.15.33:0/3189904773 conn(0x55f7ad944160 msgr2=0x55f7ad94ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:49.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.071+0000 7fd9c2c35740 1 -- 172.21.15.33:0/3189904773 shutdown_connections 2024-03-19T02:53:49.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.071+0000 7fd9c2c35740 1 -- 172.21.15.33:0/3189904773 wait complete. 2024-03-19T02:53:49.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.071+0000 7fd9c2c35740 20 client.15714 trim_cache size 0 max 16384 2024-03-19T02:53:49.077 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:53:49.077 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-03-19T02:53:49.078 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-03-19T02:53:49.078 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:53:49.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.072+0000 7fd9bff76640 1 -- 172.21.15.33:0/3034140839 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd9a40a2a00 msgr2=0x7fd9a40a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.072+0000 7fd9bff76640 1 --2- 172.21.15.33:0/3034140839 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd9a40a2a00 0x7fd9a40a4ec0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0005fd0 tx=0x7fd9b0011bc0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.072+0000 7fd9bff76640 1 -- 172.21.15.33:0/3034140839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd9b8101050 msgr2=0x7fd9b8198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.072+0000 7fd9bff76640 1 --2- 172.21.15.33:0/3034140839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd9b8101050 0x7fd9b8198510 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fd9a8009230 tx=0x7fd9a802c5d0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.072+0000 7fd9bff76640 1 -- 172.21.15.33:0/3034140839 shutdown_connections 2024-03-19T02:53:49.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.072+0000 7fd9bff76640 1 --2- 172.21.15.33:0/3034140839 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd9a40a2a00 0x7fd9a40a4ec0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.072+0000 7fd9bff76640 1 --2- 172.21.15.33:0/3034140839 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9b81066a0 0x7fd9b819c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.072+0000 7fd9bff76640 1 --2- 172.21.15.33:0/3034140839 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd9b8101050 0x7fd9b8198510 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.072+0000 7fd9bff76640 1 --2- 172.21.15.33:0/3034140839 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9b8100690 0x7fd9b8197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.072+0000 7fd9bff76640 1 -- 172.21.15.33:0/3034140839 >> 172.21.15.33:0/3034140839 conn(0x7fd9b8077820 msgr2=0x7fd9b80ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:49.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.072+0000 7fd9bff76640 1 -- 172.21.15.33:0/3034140839 shutdown_connections 2024-03-19T02:53:49.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.072+0000 7fd9bff76640 1 -- 172.21.15.33:0/3034140839 wait complete. 2024-03-19T02:53:49.083 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:53:49.083 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:53:49.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.228+0000 7fb92c2f1640 1 -- 172.21.15.33:0/2746108062 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb924100420 msgr2=0x7fb924100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.228+0000 7fb92c2f1640 1 --2- 172.21.15.33:0/2746108062 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb924100420 0x7fb924100800 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fb920009900 tx=0x7fb92002bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:49.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.228+0000 7fb92c2f1640 1 -- 172.21.15.33:0/2746108062 shutdown_connections 2024-03-19T02:53:49.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.228+0000 7fb92c2f1640 1 --2- 172.21.15.33:0/2746108062 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb924106be0 0x7fb92410e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.228+0000 7fb92c2f1640 1 --2- 172.21.15.33:0/2746108062 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb924100dd0 0x7fb924106500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.228+0000 7fb92c2f1640 1 --2- 172.21.15.33:0/2746108062 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb924100420 0x7fb924100800 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.228+0000 7fb92c2f1640 1 -- 172.21.15.33:0/2746108062 >> 172.21.15.33:0/2746108062 conn(0x7fb924075a30 msgr2=0x7fb924075e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:49.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.228+0000 7fb92c2f1640 1 -- 172.21.15.33:0/2746108062 shutdown_connections 2024-03-19T02:53:49.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.228+0000 7fb92c2f1640 1 -- 172.21.15.33:0/2746108062 wait complete. 2024-03-19T02:53:49.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.229+0000 7fb92c2f1640 1 Processor -- start 2024-03-19T02:53:49.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.229+0000 7fb92c2f1640 1 -- start start 2024-03-19T02:53:49.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.229+0000 7fb92c2f1640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb924100420 0x7fb92419a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.229+0000 7fb92c2f1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb924106be0 0x7fb92419a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.229+0000 7fb92c2f1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb92419ebd0 0x7fb92419f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.229+0000 7fb92c2f1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb924111bd0 con 0x7fb92419ebd0 2024-03-19T02:53:49.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.229+0000 7fb92c2f1640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb924111a50 con 0x7fb924106be0 2024-03-19T02:53:49.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.229+0000 7fb92c2f1640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb924111d50 con 0x7fb924100420 2024-03-19T02:53:49.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.229+0000 7fb92a88e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb92419ebd0 0x7fb92419f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.229+0000 7fb92988c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb924106be0 0x7fb92419a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.229+0000 7fb92a88e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb92419ebd0 0x7fb92419f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34310/0 (socket says 172.21.15.33:34310) 2024-03-19T02:53:49.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.229+0000 7fb92a08d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb924100420 0x7fb92419a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.229+0000 7fb92a88e640 1 -- 172.21.15.33:0/3609170606 learned_addr learned my addr 172.21.15.33:0/3609170606 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:49.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.230+0000 7fb92a88e640 1 -- 172.21.15.33:0/3609170606 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb924100420 msgr2=0x7fb92419a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.230+0000 7fb92a88e640 1 --2- 172.21.15.33:0/3609170606 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb924100420 0x7fb92419a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.230+0000 7fb92a88e640 1 -- 172.21.15.33:0/3609170606 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb924106be0 msgr2=0x7fb92419a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.230+0000 7fb92a88e640 1 --2- 172.21.15.33:0/3609170606 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb924106be0 0x7fb92419a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.230+0000 7fb92a88e640 1 -- 172.21.15.33:0/3609170606 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb92419f6c0 con 0x7fb92419ebd0 2024-03-19T02:53:49.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.230+0000 7fb92988c640 1 --2- 172.21.15.33:0/3609170606 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb924106be0 0x7fb92419a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:49.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.230+0000 7fb92a88e640 1 --2- 172.21.15.33:0/3609170606 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb92419ebd0 0x7fb92419f080 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fb91c00eb10 tx=0x7fb91c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:49.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.230+0000 7fb91b7fe640 1 -- 172.21.15.33:0/3609170606 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb91c004b20 con 0x7fb92419ebd0 2024-03-19T02:53:49.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.230+0000 7fb91b7fe640 1 -- 172.21.15.33:0/3609170606 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb91c004cc0 con 0x7fb92419ebd0 2024-03-19T02:53:49.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.230+0000 7fb92c2f1640 1 -- 172.21.15.33:0/3609170606 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb92419f9b0 con 0x7fb92419ebd0 2024-03-19T02:53:49.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.230+0000 7fb91b7fe640 1 -- 172.21.15.33:0/3609170606 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb91c010690 con 0x7fb92419ebd0 2024-03-19T02:53:49.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.230+0000 7fb92c2f1640 1 -- 172.21.15.33:0/3609170606 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9241a7150 con 0x7fb92419ebd0 2024-03-19T02:53:49.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.231+0000 7fb92c2f1640 1 -- 172.21.15.33:0/3609170606 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb924069950 con 0x7fb92419ebd0 2024-03-19T02:53:49.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.232+0000 7fb91b7fe640 1 -- 172.21.15.33:0/3609170606 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 23) ==== 137466+0+0 (secure 0 0 0) 0x7fb91c005000 con 0x7fb92419ebd0 2024-03-19T02:53:49.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.233+0000 7fb91b7fe640 1 --2- 172.21.15.33:0/3609170606 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fb9100a2a00 0x7fb9100a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.233+0000 7fb92a08d640 1 --2- 172.21.15.33:0/3609170606 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fb9100a2a00 0x7fb9100a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.233+0000 7fb91b7fe640 1 -- 172.21.15.33:0/3609170606 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fb91c0cfa90 con 0x7fb92419ebd0 2024-03-19T02:53:49.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.234+0000 7fb92a08d640 1 --2- 172.21.15.33:0/3609170606 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fb9100a2a00 0x7fb9100a4ec0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb920005ab0 tx=0x7fb920005a20 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:49.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.237+0000 7fb91b7fe640 1 -- 172.21.15.33:0/3609170606 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb91c053740 con 0x7fb92419ebd0 2024-03-19T02:53:49.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.489+0000 7fb92c2f1640 1 -- 172.21.15.33:0/3609170606 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb924100e30 con 0x7fb92419ebd0 2024-03-19T02:53:49.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.490+0000 7fb91b7fe640 1 -- 172.21.15.33:0/3609170606 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11867 (secure 0 0 0) 0x7fb91c09c310 con 0x7fb92419ebd0 2024-03-19T02:53:49.490 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:53:49.490 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:50:52.201798+0000","modified":"2024-03-19T02:53:11.748697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24488,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/3731232169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":3731232169},{"type":"v1","addr":"172.21.15.96:6839","nonce":3731232169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24457,24488,24509]},"id":1}]} 2024-03-19T02:53:49.490 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-03-19T02:53:49.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.492+0000 7fb92c2f1640 1 -- 172.21.15.33:0/3609170606 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fb9100a2a00 msgr2=0x7fb9100a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.492+0000 7fb92c2f1640 1 --2- 172.21.15.33:0/3609170606 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fb9100a2a00 0x7fb9100a4ec0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb920005ab0 tx=0x7fb920005a20 comp rx=0 tx=0).stop 2024-03-19T02:53:49.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.492+0000 7fb92c2f1640 1 -- 172.21.15.33:0/3609170606 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb92419ebd0 msgr2=0x7fb92419f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.492+0000 7fb92c2f1640 1 --2- 172.21.15.33:0/3609170606 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb92419ebd0 0x7fb92419f080 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fb91c00eb10 tx=0x7fb91c00c2d0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.492+0000 7fb92c2f1640 1 -- 172.21.15.33:0/3609170606 shutdown_connections 2024-03-19T02:53:49.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.492+0000 7fb92c2f1640 1 --2- 172.21.15.33:0/3609170606 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fb9100a2a00 0x7fb9100a4ec0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.492+0000 7fb92c2f1640 1 --2- 172.21.15.33:0/3609170606 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb92419ebd0 0x7fb92419f080 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.492+0000 7fb92c2f1640 1 --2- 172.21.15.33:0/3609170606 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb924106be0 0x7fb92419a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.492+0000 7fb92c2f1640 1 --2- 172.21.15.33:0/3609170606 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb924100420 0x7fb92419a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.492+0000 7fb92c2f1640 1 -- 172.21.15.33:0/3609170606 >> 172.21.15.33:0/3609170606 conn(0x7fb924075a30 msgr2=0x7fb9240fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:49.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.492+0000 7fb92c2f1640 1 -- 172.21.15.33:0/3609170606 shutdown_connections 2024-03-19T02:53:49.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.492+0000 7fb92c2f1640 1 -- 172.21.15.33:0/3609170606 wait complete. 2024-03-19T02:53:49.504 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:53:49.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.626+0000 7ff0a5f13640 1 -- 172.21.15.33:0/4103085803 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0a01077d0 msgr2=0x7ff0a0107bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.626+0000 7ff0a5f13640 1 --2- 172.21.15.33:0/4103085803 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0a01077d0 0x7ff0a0107bb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff090009900 tx=0x7ff09002bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:49.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.626+0000 7ff0a5f13640 1 -- 172.21.15.33:0/4103085803 shutdown_connections 2024-03-19T02:53:49.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.626+0000 7ff0a5f13640 1 --2- 172.21.15.33:0/4103085803 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0a00ffcb0 0x7ff0a0100130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.626+0000 7ff0a5f13640 1 --2- 172.21.15.33:0/4103085803 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff0a0100670 0x7ff0a010e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.626+0000 7ff0a5f13640 1 --2- 172.21.15.33:0/4103085803 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0a01077d0 0x7ff0a0107bb0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.626+0000 7ff0a5f13640 1 -- 172.21.15.33:0/4103085803 >> 172.21.15.33:0/4103085803 conn(0x7ff0a0076c30 msgr2=0x7ff0a00fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:49.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.626+0000 7ff0a5f13640 1 -- 172.21.15.33:0/4103085803 shutdown_connections 2024-03-19T02:53:49.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.626+0000 7ff0a5f13640 1 -- 172.21.15.33:0/4103085803 wait complete. 2024-03-19T02:53:49.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.627+0000 7ff0a5f13640 1 Processor -- start 2024-03-19T02:53:49.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.627+0000 7ff0a5f13640 1 -- start start 2024-03-19T02:53:49.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.627+0000 7ff0a5f13640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0a00ffcb0 0x7ff0a0197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.627+0000 7ff0a5f13640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff0a0100670 0x7ff0a01984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.627+0000 7ff0a5f13640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0a01077d0 0x7ff0a019c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.627+0000 7ff0a5f13640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a0111a50 con 0x7ff0a0100670 2024-03-19T02:53:49.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.627+0000 7ff0a5f13640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a01118d0 con 0x7ff0a00ffcb0 2024-03-19T02:53:49.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.627+0000 7ff0a5f13640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a0111bd0 con 0x7ff0a01077d0 2024-03-19T02:53:49.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.627+0000 7ff09effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff0a0100670 0x7ff0a01984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.627+0000 7ff09effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff0a0100670 0x7ff0a01984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34320/0 (socket says 172.21.15.33:34320) 2024-03-19T02:53:49.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.627+0000 7ff09f7fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0a00ffcb0 0x7ff0a0197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.627+0000 7ff09effd640 1 -- 172.21.15.33:0/3941924855 learned_addr learned my addr 172.21.15.33:0/3941924855 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:49.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.627+0000 7ff09ffff640 1 --2- 172.21.15.33:0/3941924855 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0a01077d0 0x7ff0a019c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.628+0000 7ff09f7fe640 1 -- 172.21.15.33:0/3941924855 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0a01077d0 msgr2=0x7ff0a019c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.628+0000 7ff09f7fe640 1 --2- 172.21.15.33:0/3941924855 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0a01077d0 0x7ff0a019c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.628+0000 7ff09f7fe640 1 -- 172.21.15.33:0/3941924855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff0a0100670 msgr2=0x7ff0a01984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.628+0000 7ff09f7fe640 1 --2- 172.21.15.33:0/3941924855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff0a0100670 0x7ff0a01984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.628+0000 7ff09f7fe640 1 -- 172.21.15.33:0/3941924855 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0a019cf80 con 0x7ff0a00ffcb0 2024-03-19T02:53:49.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.628+0000 7ff09effd640 1 --2- 172.21.15.33:0/3941924855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff0a0100670 0x7ff0a01984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:49.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.628+0000 7ff09f7fe640 1 --2- 172.21.15.33:0/3941924855 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0a00ffcb0 0x7ff0a0197f90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff090036040 tx=0x7ff09002cd20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:49.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.628+0000 7ff09cff9640 1 -- 172.21.15.33:0/3941924855 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff090043020 con 0x7ff0a00ffcb0 2024-03-19T02:53:49.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.628+0000 7ff09cff9640 1 -- 172.21.15.33:0/3941924855 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff090004410 con 0x7ff0a00ffcb0 2024-03-19T02:53:49.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.628+0000 7ff09cff9640 1 -- 172.21.15.33:0/3941924855 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff090003b20 con 0x7ff0a00ffcb0 2024-03-19T02:53:49.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.628+0000 7ff0a5f13640 1 -- 172.21.15.33:0/3941924855 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0a019d210 con 0x7ff0a00ffcb0 2024-03-19T02:53:49.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.628+0000 7ff0a5f13640 1 -- 172.21.15.33:0/3941924855 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0a01a4b00 con 0x7ff0a00ffcb0 2024-03-19T02:53:49.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.630+0000 7ff09cff9640 1 -- 172.21.15.33:0/3941924855 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 23) ==== 137466+0+0 (secure 0 0 0) 0x7ff090046020 con 0x7ff0a00ffcb0 2024-03-19T02:53:49.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.631+0000 7ff09cff9640 1 --2- 172.21.15.33:0/3941924855 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7ff0880a2a80 0x7ff0880a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.631+0000 7ff09effd640 1 --2- 172.21.15.33:0/3941924855 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7ff0880a2a80 0x7ff0880a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.631+0000 7ff09cff9640 1 -- 172.21.15.33:0/3941924855 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7ff0900f0e20 con 0x7ff0a00ffcb0 2024-03-19T02:53:49.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.631+0000 7ff0a8bd0740 1 Processor -- start 2024-03-19T02:53:49.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.631+0000 7ff0a8bd0740 1 -- start start 2024-03-19T02:53:49.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.631+0000 7ff09effd640 1 --2- 172.21.15.33:0/3941924855 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7ff0880a2a80 0x7ff0880a4f40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff0940095f0 tx=0x7ff094005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:49.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.631+0000 7ff0a8bd0740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55682c4d6960 0x55682c4d6d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.631+0000 7ff0a8bd0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55682c4c44d0 0x55682c4c4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.632+0000 7ff0a8bd0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55682c4c4e90 0x55682c4de830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.632+0000 7ff0a8bd0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55682c42d4b0 con 0x55682c4c44d0 2024-03-19T02:53:49.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.632+0000 7ff0a8bd0740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55682c3d5d70 con 0x55682c4d6960 2024-03-19T02:53:49.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.632+0000 7ff0a8bd0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55682c429550 con 0x55682c4c4e90 2024-03-19T02:53:49.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.632+0000 7ff09effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55682c4c44d0 0x55682c4c4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.632+0000 7ff09f7fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55682c4d6960 0x55682c4d6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.632+0000 7ff09f7fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55682c4d6960 0x55682c4d6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:58046/0 (socket says 172.21.15.33:58046) 2024-03-19T02:53:49.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.632+0000 7ff09f7fe640 1 -- 172.21.15.33:0/1865249507 learned_addr learned my addr 172.21.15.33:0/1865249507 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:49.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.632+0000 7ff09effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55682c4c44d0 0x55682c4c4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34324/0 (socket says 172.21.15.33:34324) 2024-03-19T02:53:49.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.632+0000 7ff09ffff640 1 --2- 172.21.15.33:0/1865249507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55682c4c4e90 0x55682c4de830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.632+0000 7ff09f7fe640 1 -- 172.21.15.33:0/1865249507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55682c4c4e90 msgr2=0x55682c4de830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:49.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.632+0000 7ff09f7fe640 1 --2- 172.21.15.33:0/1865249507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55682c4c4e90 0x55682c4de830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.632+0000 7ff09f7fe640 1 -- 172.21.15.33:0/1865249507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55682c4c44d0 msgr2=0x55682c4c4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:49.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.632+0000 7ff09f7fe640 1 --2- 172.21.15.33:0/1865249507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55682c4c44d0 0x55682c4c4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.632+0000 7ff09f7fe640 1 -- 172.21.15.33:0/1865249507 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55682c3f8210 con 0x55682c4d6960 2024-03-19T02:53:49.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.632+0000 7ff09f7fe640 1 --2- 172.21.15.33:0/1865249507 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55682c4d6960 0x55682c4d6d40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff0900bcdb0 tx=0x7ff0900035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdc8a61589011ad1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:49.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.632+0000 7ff085ffb640 1 -- 172.21.15.33:0/1865249507 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0900bd0b0 con 0x55682c4d6960 2024-03-19T02:53:49.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.632+0000 7ff085ffb640 1 -- 172.21.15.33:0/1865249507 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff0900c1200 con 0x55682c4d6960 2024-03-19T02:53:49.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.632+0000 7ff085ffb640 1 -- 172.21.15.33:0/1865249507 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0900b9870 con 0x55682c4d6960 2024-03-19T02:53:49.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.633+0000 7ff0a8bd0740 1 -- 172.21.15.33:0/1865249507 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55682c4d6960 msgr2=0x55682c4d6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.633+0000 7ff0a8bd0740 1 --2- 172.21.15.33:0/1865249507 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55682c4d6960 0x55682c4d6d40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff0900bcdb0 tx=0x7ff0900035f0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.633+0000 7ff0a8bd0740 1 -- 172.21.15.33:0/1865249507 shutdown_connections 2024-03-19T02:53:49.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.633+0000 7ff0a8bd0740 1 --2- 172.21.15.33:0/1865249507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55682c4c4e90 0x55682c4de830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.633+0000 7ff0a8bd0740 1 --2- 172.21.15.33:0/1865249507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55682c4c44d0 0x55682c4c4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.633+0000 7ff0a8bd0740 1 --2- 172.21.15.33:0/1865249507 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55682c4d6960 0x55682c4d6d40 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.633+0000 7ff0a8bd0740 1 -- 172.21.15.33:0/1865249507 >> 172.21.15.33:0/1865249507 conn(0x55682c4c6f90 msgr2=0x55682c4c7370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:49.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.633+0000 7ff0a8bd0740 1 -- 172.21.15.33:0/1865249507 shutdown_connections 2024-03-19T02:53:49.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.633+0000 7ff0a8bd0740 1 -- 172.21.15.33:0/1865249507 wait complete. 2024-03-19T02:53:49.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.634+0000 7ff0a8bd0740 1 -- start start 2024-03-19T02:53:49.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.634+0000 7ff0a8bd0740 1 Processor -- start 2024-03-19T02:53:49.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.634+0000 7ff0a8bd0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55682c4c44d0 0x55682c4dbd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.634+0000 7ff0a8bd0740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55682c4c4e90 0x55682c4dc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.634+0000 7ff0a8bd0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55682c4d6960 0x55682c4dc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.634+0000 7ff0a8bd0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55682c42d4b0 con 0x55682c4c44d0 2024-03-19T02:53:49.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.634+0000 7ff0a8bd0740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55682c3e9310 con 0x55682c4c4e90 2024-03-19T02:53:49.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.634+0000 7ff0a8bd0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55682c3d5d70 con 0x55682c4d6960 2024-03-19T02:53:49.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.634+0000 7ff0a8bd0740 10 client.0 _ll_register_callbacks cb 0x55682c3dd2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:53:49.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.634+0000 7ff09f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55682c4c44d0 0x55682c4dbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.634+0000 7ff09effd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55682c4c4e90 0x55682c4dc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.634+0000 7ff09f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55682c4c44d0 0x55682c4dbd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34334/0 (socket says 172.21.15.33:34334) 2024-03-19T02:53:49.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.634+0000 7ff09effd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55682c4c4e90 0x55682c4dc260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:58054/0 (socket says 172.21.15.33:58054) 2024-03-19T02:53:49.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.634+0000 7ff09f7fe640 1 -- 172.21.15.33:0/809133297 learned_addr learned my addr 172.21.15.33:0/809133297 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:49.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.634+0000 7ff09ffff640 1 --2- 172.21.15.33:0/809133297 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55682c4d6960 0x55682c4dc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.635+0000 7ff09f7fe640 1 -- 172.21.15.33:0/809133297 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55682c4d6960 msgr2=0x55682c4dc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:49.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.635+0000 7ff09f7fe640 1 --2- 172.21.15.33:0/809133297 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55682c4d6960 0x55682c4dc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.635+0000 7ff09f7fe640 1 -- 172.21.15.33:0/809133297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55682c4c4e90 msgr2=0x55682c4dc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:49.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.635+0000 7ff09f7fe640 1 --2- 172.21.15.33:0/809133297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55682c4c4e90 0x55682c4dc260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.635+0000 7ff09f7fe640 1 -- 172.21.15.33:0/809133297 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55682c1413f0 con 0x55682c4c44d0 2024-03-19T02:53:49.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.635+0000 7ff09effd640 1 --2- 172.21.15.33:0/809133297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55682c4c4e90 0x55682c4dc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:49.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.635+0000 7ff09f7fe640 1 --2- 172.21.15.33:0/809133297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55682c4c44d0 0x55682c4dbd20 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ff090004650 tx=0x7ff090005700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c06babfa3823fa21 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:49.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.635+0000 7ff07effd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:53:49.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.635+0000 7ff07effd640 1 -- 172.21.15.33:0/809133297 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0900d7070 con 0x55682c4c44d0 2024-03-19T02:53:49.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.635+0000 7ff07effd640 1 -- 172.21.15.33:0/809133297 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7ff0900db7c0 con 0x55682c4c44d0 2024-03-19T02:53:49.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.635+0000 7ff0a8bd0740 1 -- 172.21.15.33:0/809133297 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0900c1200 con 0x55682c4c44d0 2024-03-19T02:53:49.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.635+0000 7ff07effd640 1 -- 172.21.15.33:0/809133297 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff0900d5070 con 0x55682c4c44d0 2024-03-19T02:53:49.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.635+0000 7ff0a8bd0740 10 client.15732 fetch_fsmap learned FSMap version 25 2024-03-19T02:53:49.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.635+0000 7ff0a8bd0740 1 -- 172.21.15.33:0/809133297 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55682c20b7d0 con 0x55682c4c44d0 2024-03-19T02:53:49.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.635+0000 7ff07effd640 1 -- 172.21.15.33:0/809133297 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff0900c1200 con 0x55682c4c44d0 2024-03-19T02:53:49.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.636+0000 7ff07effd640 1 -- 172.21.15.33:0/809133297 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7ff0900da030 con 0x55682c4c44d0 2024-03-19T02:53:49.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.636+0000 7ff0a8bd0740 10 client.15732 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:53:49.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.636+0000 7ff0a8bd0740 10 client.15732 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T02:53:49.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.636+0000 7ff0a8bd0740 20 client.15732 populate_metadata read hostname 'smithi033' 2024-03-19T02:53:49.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.636+0000 7ff0a8bd0740 1 --2- 172.21.15.33:0/809133297 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55682c4ebe10 0x55682c4ee200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.636+0000 7ff0a8bd0740 4 client.15732 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:53:49.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.636+0000 7ff0a8bd0740 1 -- 172.21.15.33:0/809133297 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55682c48b3e0 con 0x55682c4ebe10 2024-03-19T02:53:49.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.636+0000 7ff09effd640 1 --2- 172.21.15.33:0/809133297 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55682c4ebe10 0x55682c4ee200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.636+0000 7ff09effd640 1 --2- 172.21.15.33:0/809133297 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55682c4ebe10 0x55682c4ee200 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b837f130609a53ac server_cookie=f3fa1e8783eff610 in_seq=0 out_seq=0 2024-03-19T02:53:49.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.636+0000 7ff07effd640 10 client.15732 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T02:53:49.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.637+0000 7ff07effd640 1 -- 172.21.15.33:0/809133297 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55682c48b3e0 con 0x55682c4ebe10 2024-03-19T02:53:49.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.637+0000 7ff07effd640 10 client.15732 handle_command_reply: tid=0 2024-03-19T02:53:49.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.637+0000 7ff0a8bd0740 1 client.15732 shutdown 2024-03-19T02:53:49.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.637+0000 7ff0a8bd0740 1 -- 172.21.15.33:0/809133297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55682c4c44d0 msgr2=0x55682c4dbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.637+0000 7ff0a8bd0740 1 --2- 172.21.15.33:0/809133297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55682c4c44d0 0x55682c4dbd20 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ff090004650 tx=0x7ff090005700 comp rx=0 tx=0).stop 2024-03-19T02:53:49.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.637+0000 7ff0a8bd0740 1 -- 172.21.15.33:0/809133297 shutdown_connections 2024-03-19T02:53:49.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.637+0000 7ff0a8bd0740 1 --2- 172.21.15.33:0/809133297 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55682c4ebe10 0x55682c4ee200 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.637+0000 7ff0a8bd0740 1 --2- 172.21.15.33:0/809133297 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55682c4d6960 0x55682c4dc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.637+0000 7ff0a8bd0740 1 --2- 172.21.15.33:0/809133297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55682c4c4e90 0x55682c4dc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.637+0000 7ff0a8bd0740 1 --2- 172.21.15.33:0/809133297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55682c4c44d0 0x55682c4dbd20 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.637+0000 7ff0a8bd0740 1 -- 172.21.15.33:0/809133297 >> 172.21.15.33:0/809133297 conn(0x55682c4c6f90 msgr2=0x55682c4d8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:49.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.638+0000 7ff07effd640 0 client.15732 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T02:53:49.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.638+0000 7ff0a8bd0740 1 -- 172.21.15.33:0/809133297 shutdown_connections 2024-03-19T02:53:49.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.638+0000 7ff0a8bd0740 1 -- 172.21.15.33:0/809133297 wait complete. 2024-03-19T02:53:49.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.638+0000 7ff0a8bd0740 20 client.15732 trim_cache size 0 max 16384 2024-03-19T02:53:49.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.653+0000 7ff0a8bd0740 1 Processor -- start 2024-03-19T02:53:49.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.653+0000 7ff0a8bd0740 1 -- start start 2024-03-19T02:53:49.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.653+0000 7ff0a8bd0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55682c4c44d0 0x55682c4de450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.653+0000 7ff0a8bd0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55682c4c4e90 0x55682c4d44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.654+0000 7ff0a8bd0740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55682c4d6960 0x55682c4e7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.654+0000 7ff0a8bd0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a01af1d0 con 0x55682c4c44d0 2024-03-19T02:53:49.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.654+0000 7ff0a8bd0740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a01acaa0 con 0x55682c4d6960 2024-03-19T02:53:49.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.654+0000 7ff0a8bd0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a01aa8d0 con 0x55682c4c4e90 2024-03-19T02:53:49.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.654+0000 7ff09f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55682c4c44d0 0x55682c4de450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.654+0000 7ff09effd640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55682c4c4e90 0x55682c4d44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.654+0000 7ff09f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55682c4c44d0 0x55682c4de450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34350/0 (socket says 172.21.15.33:34350) 2024-03-19T02:53:49.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.654+0000 7ff09ffff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55682c4d6960 0x55682c4e7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.654+0000 7ff09f7fe640 1 -- 172.21.15.33:0/144319212 learned_addr learned my addr 172.21.15.33:0/144319212 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:49.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.654+0000 7ff09f7fe640 1 -- 172.21.15.33:0/144319212 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55682c4c4e90 msgr2=0x55682c4d44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:49.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.654+0000 7ff09f7fe640 1 --2- 172.21.15.33:0/144319212 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55682c4c4e90 0x55682c4d44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.654+0000 7ff09f7fe640 1 -- 172.21.15.33:0/144319212 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55682c4d6960 msgr2=0x55682c4e7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:49.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.654+0000 7ff09f7fe640 1 --2- 172.21.15.33:0/144319212 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55682c4d6960 0x55682c4e7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.654+0000 7ff09f7fe640 1 -- 172.21.15.33:0/144319212 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55682c4d2f20 con 0x55682c4c44d0 2024-03-19T02:53:49.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.654+0000 7ff09ffff640 1 --2- 172.21.15.33:0/144319212 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55682c4d6960 0x55682c4e7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:49.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.654+0000 7ff09f7fe640 1 --2- 172.21.15.33:0/144319212 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55682c4c44d0 0x55682c4de450 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff090003fd0 tx=0x7ff0900045c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b496203305fe638 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:49.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.657+0000 7ff0857fa640 1 -- 172.21.15.33:0/144319212 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0900d7070 con 0x55682c4c44d0 2024-03-19T02:53:49.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.657+0000 7ff0857fa640 1 -- 172.21.15.33:0/144319212 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff0900bd7b0 con 0x55682c4c44d0 2024-03-19T02:53:49.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.657+0000 7ff0857fa640 1 -- 172.21.15.33:0/144319212 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0900db760 con 0x55682c4c44d0 2024-03-19T02:53:49.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.657+0000 7ff0a8bd0740 1 -- 172.21.15.33:0/144319212 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55682c4c44d0 msgr2=0x55682c4de450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.657+0000 7ff0a8bd0740 1 --2- 172.21.15.33:0/144319212 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55682c4c44d0 0x55682c4de450 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff090003fd0 tx=0x7ff0900045c0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.657+0000 7ff0a8bd0740 1 -- 172.21.15.33:0/144319212 shutdown_connections 2024-03-19T02:53:49.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.657+0000 7ff0a8bd0740 1 --2- 172.21.15.33:0/144319212 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55682c4c4e90 0x55682c4d44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.657+0000 7ff0a8bd0740 1 --2- 172.21.15.33:0/144319212 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55682c4d6960 0x55682c4e7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.657+0000 7ff0a8bd0740 1 --2- 172.21.15.33:0/144319212 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55682c4c44d0 0x55682c4de450 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.657+0000 7ff0a8bd0740 1 -- 172.21.15.33:0/144319212 >> 172.21.15.33:0/144319212 conn(0x55682c4c6f90 msgr2=0x55682c4da610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:49.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.658+0000 7ff0a8bd0740 1 -- 172.21.15.33:0/144319212 shutdown_connections 2024-03-19T02:53:49.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.658+0000 7ff0a8bd0740 1 -- 172.21.15.33:0/144319212 wait complete. 2024-03-19T02:53:49.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.658+0000 7ff0a8bd0740 1 -- start start 2024-03-19T02:53:49.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.658+0000 7ff0a8bd0740 1 Processor -- start 2024-03-19T02:53:49.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.658+0000 7ff0a8bd0740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55682c4c44d0 0x55682c4f2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.658+0000 7ff0a8bd0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55682c4c4e90 0x55682c4f3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.658+0000 7ff0a8bd0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55682c4d6960 0x55682c4f3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.658+0000 7ff0a8bd0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a01aaf60 con 0x55682c4d6960 2024-03-19T02:53:49.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.658+0000 7ff0a8bd0740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a01ac3d0 con 0x55682c4c44d0 2024-03-19T02:53:49.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.658+0000 7ff0a8bd0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a01ab630 con 0x55682c4c4e90 2024-03-19T02:53:49.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.658+0000 7ff0a8bd0740 10 client.0 _ll_register_callbacks cb 0x7ff094009470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:53:49.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.658+0000 7ff09ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55682c4d6960 0x55682c4f3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.659+0000 7ff09f7fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55682c4c44d0 0x55682c4f2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.659+0000 7ff09ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55682c4d6960 0x55682c4f3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34354/0 (socket says 172.21.15.33:34354) 2024-03-19T02:53:49.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.659+0000 7ff09f7fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55682c4c44d0 0x55682c4f2ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:58068/0 (socket says 172.21.15.33:58068) 2024-03-19T02:53:49.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.659+0000 7ff09ffff640 1 -- 172.21.15.33:0/4089166849 learned_addr learned my addr 172.21.15.33:0/4089166849 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:49.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.659+0000 7ff09effd640 1 --2- 172.21.15.33:0/4089166849 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55682c4c4e90 0x55682c4f3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.659+0000 7ff09f7fe640 1 -- 172.21.15.33:0/4089166849 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55682c4c4e90 msgr2=0x55682c4f3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:49.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.659+0000 7ff09f7fe640 1 --2- 172.21.15.33:0/4089166849 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55682c4c4e90 0x55682c4f3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.659+0000 7ff09f7fe640 1 -- 172.21.15.33:0/4089166849 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55682c4d6960 msgr2=0x55682c4f3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:49.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.659+0000 7ff09f7fe640 1 --2- 172.21.15.33:0/4089166849 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55682c4d6960 0x55682c4f3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.659+0000 7ff09f7fe640 1 -- 172.21.15.33:0/4089166849 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55682c5510f0 con 0x55682c4c44d0 2024-03-19T02:53:49.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.659+0000 7ff09f7fe640 1 --2- 172.21.15.33:0/4089166849 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55682c4c44d0 0x55682c4f2ec0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x55682c4d4a70 tx=0x7ff090000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddf05844a4a84717 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:49.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.659+0000 7ff07effd640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T02:53:49.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.659+0000 7ff07effd640 1 -- 172.21.15.33:0/4089166849 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0900d7070 con 0x55682c4c44d0 2024-03-19T02:53:49.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.659+0000 7ff07effd640 1 -- 172.21.15.33:0/4089166849 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7ff0900dbdc0 con 0x55682c4c44d0 2024-03-19T02:53:49.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.659+0000 7ff0a8bd0740 1 -- 172.21.15.33:0/4089166849 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0900bd7b0 con 0x55682c4c44d0 2024-03-19T02:53:49.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.659+0000 7ff07effd640 1 -- 172.21.15.33:0/4089166849 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff0900bca10 con 0x55682c4c44d0 2024-03-19T02:53:49.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.659+0000 7ff0a8bd0740 10 client.25607 fetch_fsmap learned FSMap version 25 2024-03-19T02:53:49.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.659+0000 7ff0a8bd0740 1 -- 172.21.15.33:0/4089166849 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55682c5510f0 con 0x55682c4c44d0 2024-03-19T02:53:49.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.660+0000 7ff07effd640 1 -- 172.21.15.33:0/4089166849 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff0900bd7b0 con 0x55682c4c44d0 2024-03-19T02:53:49.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.660+0000 7ff07effd640 1 -- 172.21.15.33:0/4089166849 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7ff0900da020 con 0x55682c4c44d0 2024-03-19T02:53:49.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.660+0000 7ff0a8bd0740 10 client.25607 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:53:49.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.660+0000 7ff0a8bd0740 10 client.25607 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T02:53:49.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.660+0000 7ff0a8bd0740 20 client.25607 populate_metadata read hostname 'smithi033' 2024-03-19T02:53:49.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.660+0000 7ff0a8bd0740 1 --2- 172.21.15.33:0/4089166849 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55682c5526f0 0x55682c554ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.660+0000 7ff0a8bd0740 4 client.25607 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:53:49.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.660+0000 7ff0a8bd0740 1 -- 172.21.15.33:0/4089166849 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55682c43a9f0 con 0x55682c5526f0 2024-03-19T02:53:49.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.660+0000 7ff09effd640 1 --2- 172.21.15.33:0/4089166849 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55682c5526f0 0x55682c554ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.660+0000 7ff09effd640 1 --2- 172.21.15.33:0/4089166849 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55682c5526f0 0x55682c554ae0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=90e5d882003d7bc0 server_cookie=4aaf517b4d60d1c2 in_seq=0 out_seq=0 2024-03-19T02:53:49.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.660+0000 7ff07effd640 10 client.25607 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T02:53:49.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.661+0000 7ff07effd640 1 -- 172.21.15.33:0/4089166849 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55682c43a9f0 con 0x55682c5526f0 2024-03-19T02:53:49.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.661+0000 7ff07effd640 10 client.25607 handle_command_reply: tid=0 2024-03-19T02:53:49.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.661+0000 7ff0a8bd0740 1 client.25607 shutdown 2024-03-19T02:53:49.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.661+0000 7ff0a8bd0740 1 -- 172.21.15.33:0/4089166849 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55682c4c44d0 msgr2=0x55682c4f2ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.661+0000 7ff0a8bd0740 1 --2- 172.21.15.33:0/4089166849 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55682c4c44d0 0x55682c4f2ec0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x55682c4d4a70 tx=0x7ff090000bc0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.661+0000 7ff0a8bd0740 1 -- 172.21.15.33:0/4089166849 shutdown_connections 2024-03-19T02:53:49.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.661+0000 7ff0a8bd0740 1 --2- 172.21.15.33:0/4089166849 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55682c5526f0 0x55682c554ae0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.661+0000 7ff0a8bd0740 1 --2- 172.21.15.33:0/4089166849 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55682c4c4e90 0x55682c4f3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.661+0000 7ff0a8bd0740 1 --2- 172.21.15.33:0/4089166849 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55682c4d6960 0x55682c4f3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.661+0000 7ff07effd640 0 client.25607 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T02:53:49.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.661+0000 7ff0a8bd0740 1 --2- 172.21.15.33:0/4089166849 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55682c4c44d0 0x55682c4f2ec0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.661+0000 7ff0a8bd0740 1 -- 172.21.15.33:0/4089166849 >> 172.21.15.33:0/4089166849 conn(0x55682c4c6f90 msgr2=0x55682c4dbcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:49.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.661+0000 7ff0a8bd0740 1 -- 172.21.15.33:0/4089166849 shutdown_connections 2024-03-19T02:53:49.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.661+0000 7ff0a8bd0740 1 -- 172.21.15.33:0/4089166849 wait complete. 2024-03-19T02:53:49.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.661+0000 7ff0a8bd0740 20 client.25607 trim_cache size 0 max 16384 2024-03-19T02:53:49.666 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:53:49.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.661+0000 7ff0a5f13640 1 -- 172.21.15.33:0/3941924855 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7ff0880a2a80 msgr2=0x7ff0880a4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.661+0000 7ff0a5f13640 1 --2- 172.21.15.33:0/3941924855 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7ff0880a2a80 0x7ff0880a4f40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff0940095f0 tx=0x7ff094005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:49.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.661+0000 7ff0a5f13640 1 -- 172.21.15.33:0/3941924855 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0a00ffcb0 msgr2=0x7ff0a0197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.662+0000 7ff0a5f13640 1 --2- 172.21.15.33:0/3941924855 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0a00ffcb0 0x7ff0a0197f90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff090036040 tx=0x7ff09002cd20 comp rx=0 tx=0).stop 2024-03-19T02:53:49.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.662+0000 7ff0a5f13640 1 -- 172.21.15.33:0/3941924855 shutdown_connections 2024-03-19T02:53:49.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.662+0000 7ff0a5f13640 1 --2- 172.21.15.33:0/3941924855 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7ff0880a2a80 0x7ff0880a4f40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.662+0000 7ff0a5f13640 1 --2- 172.21.15.33:0/3941924855 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0a01077d0 0x7ff0a019c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.662+0000 7ff0a5f13640 1 --2- 172.21.15.33:0/3941924855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff0a0100670 0x7ff0a01984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.662+0000 7ff0a5f13640 1 --2- 172.21.15.33:0/3941924855 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0a00ffcb0 0x7ff0a0197f90 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.662+0000 7ff0a5f13640 1 -- 172.21.15.33:0/3941924855 >> 172.21.15.33:0/3941924855 conn(0x7ff0a0076c30 msgr2=0x7ff0a0104720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:49.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.662+0000 7ff0a5f13640 1 -- 172.21.15.33:0/3941924855 shutdown_connections 2024-03-19T02:53:49.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.662+0000 7ff0a5f13640 1 -- 172.21.15.33:0/3941924855 wait complete. 2024-03-19T02:53:49.674 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:53:49.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:49 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3609170606' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:53:49.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.796+0000 7fd01e55b640 1 -- 172.21.15.33:0/648745804 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd018107800 msgr2=0x7fd018107be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.796+0000 7fd01e55b640 1 --2- 172.21.15.33:0/648745804 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd018107800 0x7fd018107be0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd008009890 tx=0x7fd00802bb70 comp rx=0 tx=0).stop 2024-03-19T02:53:49.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.796+0000 7fd01e55b640 1 -- 172.21.15.33:0/648745804 shutdown_connections 2024-03-19T02:53:49.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.796+0000 7fd01e55b640 1 --2- 172.21.15.33:0/648745804 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0181021b0 0x7fd01810e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.796+0000 7fd01e55b640 1 --2- 172.21.15.33:0/648745804 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0181017f0 0x7fd018101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.796+0000 7fd01e55b640 1 --2- 172.21.15.33:0/648745804 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd018107800 0x7fd018107be0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.796+0000 7fd01e55b640 1 -- 172.21.15.33:0/648745804 >> 172.21.15.33:0/648745804 conn(0x7fd0180fd2c0 msgr2=0x7fd0180ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:49.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.796+0000 7fd01e55b640 1 -- 172.21.15.33:0/648745804 shutdown_connections 2024-03-19T02:53:49.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.796+0000 7fd01e55b640 1 -- 172.21.15.33:0/648745804 wait complete. 2024-03-19T02:53:49.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.797+0000 7fd01e55b640 1 Processor -- start 2024-03-19T02:53:49.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.797+0000 7fd01e55b640 1 -- start start 2024-03-19T02:53:49.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.797+0000 7fd01e55b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0181017f0 0x7fd018073b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.797+0000 7fd01e55b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0181021b0 0x7fd018074060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.797+0000 7fd01e55b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd018107800 0x7fd0180745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.797+0000 7fd01e55b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd018111a80 con 0x7fd0181017f0 2024-03-19T02:53:49.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.797+0000 7fd01e55b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd018111900 con 0x7fd0181021b0 2024-03-19T02:53:49.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.797+0000 7fd01e55b640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd018111c00 con 0x7fd018107800 2024-03-19T02:53:49.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.797+0000 7fd017fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0181017f0 0x7fd018073b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.797+0000 7fd017fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0181017f0 0x7fd018073b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34370/0 (socket says 172.21.15.33:34370) 2024-03-19T02:53:49.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.797+0000 7fd017fff640 1 -- 172.21.15.33:0/3646518834 learned_addr learned my addr 172.21.15.33:0/3646518834 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:49.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.797+0000 7fd0177fe640 1 --2- 172.21.15.33:0/3646518834 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0181021b0 0x7fd018074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.797+0000 7fd01caf8640 1 --2- 172.21.15.33:0/3646518834 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd018107800 0x7fd0180745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.798+0000 7fd0177fe640 1 -- 172.21.15.33:0/3646518834 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd018107800 msgr2=0x7fd0180745a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.798+0000 7fd0177fe640 1 --2- 172.21.15.33:0/3646518834 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd018107800 0x7fd0180745a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.798+0000 7fd0177fe640 1 -- 172.21.15.33:0/3646518834 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0181017f0 msgr2=0x7fd018073b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.798+0000 7fd0177fe640 1 --2- 172.21.15.33:0/3646518834 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0181017f0 0x7fd018073b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.798+0000 7fd0177fe640 1 -- 172.21.15.33:0/3646518834 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0181a6500 con 0x7fd0181021b0 2024-03-19T02:53:49.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.798+0000 7fd017fff640 1 --2- 172.21.15.33:0/3646518834 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0181017f0 0x7fd018073b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:49.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.798+0000 7fd0177fe640 1 --2- 172.21.15.33:0/3646518834 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0181021b0 0x7fd018074060 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd00c005fd0 tx=0x7fd00c00ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:49.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.798+0000 7fd0157fa640 1 -- 172.21.15.33:0/3646518834 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd00c00cd00 con 0x7fd0181021b0 2024-03-19T02:53:49.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.798+0000 7fd0157fa640 1 -- 172.21.15.33:0/3646518834 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd00c0035f0 con 0x7fd0181021b0 2024-03-19T02:53:49.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.798+0000 7fd0157fa640 1 -- 172.21.15.33:0/3646518834 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd00c010690 con 0x7fd0181021b0 2024-03-19T02:53:49.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.798+0000 7fd01e55b640 1 -- 172.21.15.33:0/3646518834 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0181a67f0 con 0x7fd0181021b0 2024-03-19T02:53:49.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.798+0000 7fd01e55b640 1 -- 172.21.15.33:0/3646518834 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0181a6d40 con 0x7fd0181021b0 2024-03-19T02:53:49.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.801+0000 7fd0157fa640 1 -- 172.21.15.33:0/3646518834 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 23) ==== 137466+0+0 (secure 0 0 0) 0x7fd00c0037a0 con 0x7fd0181021b0 2024-03-19T02:53:49.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.801+0000 7fd021224740 1 Processor -- start 2024-03-19T02:53:49.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.801+0000 7fd021224740 1 -- start start 2024-03-19T02:53:49.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.801+0000 7fd0157fa640 1 --2- 172.21.15.33:0/3646518834 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd0000a2a00 0x7fd0000a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.801+0000 7fd021224740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab7411a2d0 0x55ab7411a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.801+0000 7fd021224740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab7411abf0 0x55ab74123290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.801+0000 7fd021224740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab741237d0 0x55ab74125bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.801+0000 7fd021224740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ab7407d4b0 con 0x55ab7411a2d0 2024-03-19T02:53:49.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.801+0000 7fd021224740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55ab74025d70 con 0x55ab741237d0 2024-03-19T02:53:49.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.801+0000 7fd021224740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ab74079550 con 0x55ab7411abf0 2024-03-19T02:53:49.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.801+0000 7fd017fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab7411a2d0 0x55ab7411a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.801+0000 7fd01caf8640 1 --2- 172.21.15.33:0/3646518834 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd0000a2a00 0x7fd0000a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.801+0000 7fd017fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab7411a2d0 0x55ab7411a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34386/0 (socket says 172.21.15.33:34386) 2024-03-19T02:53:49.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.802+0000 7fd017fff640 1 -- 172.21.15.33:0/1830111340 learned_addr learned my addr 172.21.15.33:0/1830111340 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:49.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.802+0000 7fd01caf8640 1 --2- 172.21.15.33:0/1830111340 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab741237d0 0x55ab74125bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.802+0000 7fd0157fa640 1 -- 172.21.15.33:0/3646518834 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fd00c0cf550 con 0x7fd0181021b0 2024-03-19T02:53:49.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.802+0000 7fd0177fe640 1 --2- 172.21.15.33:0/1830111340 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab7411abf0 0x55ab74123290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.802+0000 7fd017fff640 1 -- 172.21.15.33:0/1830111340 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab7411abf0 msgr2=0x55ab74123290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:49.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.802+0000 7fd017fff640 1 --2- 172.21.15.33:0/1830111340 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab7411abf0 0x55ab74123290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.802+0000 7fd017fff640 1 -- 172.21.15.33:0/1830111340 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab741237d0 msgr2=0x55ab74125bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:49.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.802+0000 7fd017fff640 1 --2- 172.21.15.33:0/1830111340 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab741237d0 0x55ab74125bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.802+0000 7fd017fff640 1 -- 172.21.15.33:0/1830111340 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab74048210 con 0x55ab7411a2d0 2024-03-19T02:53:49.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.802+0000 7fd0177fe640 1 --2- 172.21.15.33:0/1830111340 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab7411abf0 0x55ab74123290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:53:49.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.802+0000 7fd01caf8640 1 --2- 172.21.15.33:0/3646518834 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd0000a2a00 0x7fd0000a4ec0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd010005fd0 tx=0x7fd010009e90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:49.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.802+0000 7fd017fff640 1 --2- 172.21.15.33:0/1830111340 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab7411a2d0 0x55ab7411a6b0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fd008000c00 tx=0x7fd008009490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a28b5b82cc1eef24 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:49.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.802+0000 7fcffe7fc640 1 -- 172.21.15.33:0/1830111340 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd008043020 con 0x55ab7411a2d0 2024-03-19T02:53:49.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.802+0000 7fcffe7fc640 1 -- 172.21.15.33:0/1830111340 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd008034e30 con 0x55ab7411a2d0 2024-03-19T02:53:49.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.802+0000 7fcffe7fc640 1 -- 172.21.15.33:0/1830111340 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd008039040 con 0x55ab7411a2d0 2024-03-19T02:53:49.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.803+0000 7fd021224740 1 -- 172.21.15.33:0/1830111340 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab7411a2d0 msgr2=0x55ab7411a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.803+0000 7fd021224740 1 --2- 172.21.15.33:0/1830111340 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab7411a2d0 0x55ab7411a6b0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fd008000c00 tx=0x7fd008009490 comp rx=0 tx=0).stop 2024-03-19T02:53:49.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.803+0000 7fd021224740 1 -- 172.21.15.33:0/1830111340 shutdown_connections 2024-03-19T02:53:49.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.803+0000 7fd021224740 1 --2- 172.21.15.33:0/1830111340 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab7411abf0 0x55ab74123290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.803+0000 7fd021224740 1 --2- 172.21.15.33:0/1830111340 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab741237d0 0x55ab74125bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.803+0000 7fd021224740 1 --2- 172.21.15.33:0/1830111340 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab7411a2d0 0x55ab7411a6b0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.803+0000 7fd021224740 1 -- 172.21.15.33:0/1830111340 >> 172.21.15.33:0/1830111340 conn(0x55ab74114a00 msgr2=0x55ab74114de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:49.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.803+0000 7fd021224740 1 -- 172.21.15.33:0/1830111340 shutdown_connections 2024-03-19T02:53:49.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.803+0000 7fd021224740 1 -- 172.21.15.33:0/1830111340 wait complete. 2024-03-19T02:53:49.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.803+0000 7fd021224740 1 -- start start 2024-03-19T02:53:49.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.804+0000 7fd021224740 1 Processor -- start 2024-03-19T02:53:49.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.804+0000 7fd021224740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab7411a2d0 0x55ab74120840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.804+0000 7fd021224740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab7411abf0 0x55ab74120d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.804+0000 7fd021224740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab741237d0 0x55ab741212c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.804+0000 7fd021224740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ab74126560 con 0x55ab741237d0 2024-03-19T02:53:49.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.804+0000 7fd021224740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55ab74039310 con 0x55ab7411a2d0 2024-03-19T02:53:49.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.804+0000 7fd021224740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ab741266e0 con 0x55ab7411abf0 2024-03-19T02:53:49.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.804+0000 7fd021224740 10 client.0 _ll_register_callbacks cb 0x55ab7402d2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:53:49.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.804+0000 7fd01caf8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab741237d0 0x55ab741212c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.804+0000 7fd017fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab7411a2d0 0x55ab74120840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.804+0000 7fd017fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab7411a2d0 0x55ab74120840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:58108/0 (socket says 172.21.15.33:58108) 2024-03-19T02:53:49.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.804+0000 7fd017fff640 1 -- 172.21.15.33:0/434378735 learned_addr learned my addr 172.21.15.33:0/434378735 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:49.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.804+0000 7fd0177fe640 1 --2- 172.21.15.33:0/434378735 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab7411abf0 0x55ab74120d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.804+0000 7fd0177fe640 1 -- 172.21.15.33:0/434378735 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab7411a2d0 msgr2=0x55ab74120840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:49.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.804+0000 7fd0177fe640 1 --2- 172.21.15.33:0/434378735 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab7411a2d0 0x55ab74120840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.804+0000 7fd0177fe640 1 -- 172.21.15.33:0/434378735 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab741237d0 msgr2=0x55ab741212c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:49.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.804+0000 7fd0177fe640 1 --2- 172.21.15.33:0/434378735 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab741237d0 0x55ab741212c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.804+0000 7fd0177fe640 1 -- 172.21.15.33:0/434378735 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab73d913f0 con 0x55ab7411abf0 2024-03-19T02:53:49.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.804+0000 7fd017fff640 1 --2- 172.21.15.33:0/434378735 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab7411a2d0 0x55ab74120840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:49.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.804+0000 7fd01caf8640 1 --2- 172.21.15.33:0/434378735 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab741237d0 0x55ab741212c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:49.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.805+0000 7fd0177fe640 1 --2- 172.21.15.33:0/434378735 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab7411abf0 0x55ab74120d80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd00c0cc840 tx=0x7fd00c009650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b36b88ac64088c98 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:49.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.805+0000 7fcff77fe640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-19T02:53:49.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.805+0000 7fcff77fe640 1 -- 172.21.15.33:0/434378735 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd00c0a3e70 con 0x55ab7411abf0 2024-03-19T02:53:49.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.805+0000 7fcff77fe640 1 -- 172.21.15.33:0/434378735 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fd00c003af0 con 0x55ab7411abf0 2024-03-19T02:53:49.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.805+0000 7fd021224740 1 -- 172.21.15.33:0/434378735 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd008034e30 con 0x55ab7411abf0 2024-03-19T02:53:49.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.805+0000 7fcff77fe640 1 -- 172.21.15.33:0/434378735 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd00c052190 con 0x55ab7411abf0 2024-03-19T02:53:49.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.805+0000 7fd021224740 10 client.25501 fetch_fsmap learned FSMap version 25 2024-03-19T02:53:49.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.805+0000 7fd021224740 1 -- 172.21.15.33:0/434378735 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ab73e5b7d0 con 0x55ab7411abf0 2024-03-19T02:53:49.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.805+0000 7fcff77fe640 1 -- 172.21.15.33:0/434378735 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd008034e30 con 0x55ab7411abf0 2024-03-19T02:53:49.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.805+0000 7fcff77fe640 1 -- 172.21.15.33:0/434378735 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fd00c0ccb20 con 0x55ab7411abf0 2024-03-19T02:53:49.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.805+0000 7fd021224740 10 client.25501 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:53:49.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.805+0000 7fd021224740 10 client.25501 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T02:53:49.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.805+0000 7fd021224740 20 client.25501 populate_metadata read hostname 'smithi033' 2024-03-19T02:53:49.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.805+0000 7fd021224740 1 --2- 172.21.15.33:0/434378735 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55ab74133a50 0x55ab74135e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.805+0000 7fd021224740 4 client.25501 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:53:49.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.805+0000 7fd021224740 1 -- 172.21.15.33:0/434378735 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ab740db3e0 con 0x55ab74133a50 2024-03-19T02:53:49.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.806+0000 7fd017fff640 1 --2- 172.21.15.33:0/434378735 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55ab74133a50 0x55ab74135e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.806+0000 7fcff77fe640 10 client.25501 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T02:53:49.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.806+0000 7fd017fff640 1 --2- 172.21.15.33:0/434378735 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55ab74133a50 0x55ab74135e60 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e925856d182319cb server_cookie=eaf0478cde186b1 in_seq=0 out_seq=0 2024-03-19T02:53:49.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.807+0000 7fcff77fe640 1 -- 172.21.15.33:0/434378735 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ab740db3e0 con 0x55ab74133a50 2024-03-19T02:53:49.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.807+0000 7fcff77fe640 10 client.25501 handle_command_reply: tid=0 2024-03-19T02:53:49.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.807+0000 7fd021224740 1 client.25501 shutdown 2024-03-19T02:53:49.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.807+0000 7fd021224740 1 -- 172.21.15.33:0/434378735 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab7411abf0 msgr2=0x55ab74120d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.807+0000 7fd021224740 1 --2- 172.21.15.33:0/434378735 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab7411abf0 0x55ab74120d80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd00c0cc840 tx=0x7fd00c009650 comp rx=0 tx=0).stop 2024-03-19T02:53:49.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.807+0000 7fd021224740 1 -- 172.21.15.33:0/434378735 shutdown_connections 2024-03-19T02:53:49.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.807+0000 7fd021224740 1 --2- 172.21.15.33:0/434378735 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55ab74133a50 0x55ab74135e60 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.807+0000 7fd021224740 1 --2- 172.21.15.33:0/434378735 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab7411abf0 0x55ab74120d80 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.807+0000 7fd021224740 1 --2- 172.21.15.33:0/434378735 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab741237d0 0x55ab741212c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.807+0000 7fd021224740 1 --2- 172.21.15.33:0/434378735 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab7411a2d0 0x55ab74120840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.807+0000 7fcff77fe640 0 client.25501 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T02:53:49.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.807+0000 7fd021224740 1 -- 172.21.15.33:0/434378735 >> 172.21.15.33:0/434378735 conn(0x55ab74114a00 msgr2=0x55ab74116750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:49.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.807+0000 7fd021224740 1 -- 172.21.15.33:0/434378735 shutdown_connections 2024-03-19T02:53:49.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.807+0000 7fd021224740 1 -- 172.21.15.33:0/434378735 wait complete. 2024-03-19T02:53:49.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.807+0000 7fd021224740 20 client.25501 trim_cache size 0 max 16384 2024-03-19T02:53:49.822 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.821+0000 7fd021224740 1 Processor -- start 2024-03-19T02:53:49.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.821+0000 7fd021224740 1 -- start start 2024-03-19T02:53:49.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.821+0000 7fd021224740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab7411a2d0 0x55ab74122f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.821+0000 7fd021224740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab7411abf0 0x55ab7411f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.821+0000 7fd021224740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab741237d0 0x55ab7412f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.821+0000 7fd021224740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd0181b36a0 con 0x55ab7411abf0 2024-03-19T02:53:49.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.821+0000 7fd021224740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd0181b0f70 con 0x55ab7411a2d0 2024-03-19T02:53:49.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.821+0000 7fd021224740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd0181aeda0 con 0x55ab741237d0 2024-03-19T02:53:49.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.821+0000 7fd0177fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab7411abf0 0x55ab7411f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.822+0000 7fd017fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab7411a2d0 0x55ab74122f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.822+0000 7fd0177fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab7411abf0 0x55ab7411f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34408/0 (socket says 172.21.15.33:34408) 2024-03-19T02:53:49.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.822+0000 7fd0177fe640 1 -- 172.21.15.33:0/984176159 learned_addr learned my addr 172.21.15.33:0/984176159 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:49.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.822+0000 7fd01caf8640 1 --2- 172.21.15.33:0/984176159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab741237d0 0x55ab7412f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.822+0000 7fd0177fe640 1 -- 172.21.15.33:0/984176159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab741237d0 msgr2=0x55ab7412f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:49.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.822+0000 7fd0177fe640 1 --2- 172.21.15.33:0/984176159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab741237d0 0x55ab7412f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.822+0000 7fd0177fe640 1 -- 172.21.15.33:0/984176159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab7411a2d0 msgr2=0x55ab74122f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:49.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.822+0000 7fd0177fe640 1 --2- 172.21.15.33:0/984176159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab7411a2d0 0x55ab74122f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.822+0000 7fd0177fe640 1 -- 172.21.15.33:0/984176159 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab7412f840 con 0x55ab7411abf0 2024-03-19T02:53:49.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.822+0000 7fd017fff640 1 --2- 172.21.15.33:0/984176159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab7411a2d0 0x55ab74122f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:49.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.822+0000 7fd0177fe640 1 --2- 172.21.15.33:0/984176159 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab7411abf0 0x55ab7411f9a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fd00c0096c0 tx=0x7fd00c099be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7d538799f8c3f8f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:49.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.822+0000 7fcffdffb640 1 -- 172.21.15.33:0/984176159 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd00c0a3e70 con 0x55ab7411abf0 2024-03-19T02:53:49.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.822+0000 7fcffdffb640 1 -- 172.21.15.33:0/984176159 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd00c051070 con 0x55ab7411abf0 2024-03-19T02:53:49.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.822+0000 7fcffdffb640 1 -- 172.21.15.33:0/984176159 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd00c0ccb20 con 0x55ab7411abf0 2024-03-19T02:53:49.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.823+0000 7fd021224740 1 -- 172.21.15.33:0/984176159 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab7411abf0 msgr2=0x55ab7411f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.823+0000 7fd021224740 1 --2- 172.21.15.33:0/984176159 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab7411abf0 0x55ab7411f9a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fd00c0096c0 tx=0x7fd00c099be0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.823+0000 7fd021224740 1 -- 172.21.15.33:0/984176159 shutdown_connections 2024-03-19T02:53:49.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.823+0000 7fd021224740 1 --2- 172.21.15.33:0/984176159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab741237d0 0x55ab7412f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.823+0000 7fd021224740 1 --2- 172.21.15.33:0/984176159 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab7411abf0 0x55ab7411f9a0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.823+0000 7fd021224740 1 --2- 172.21.15.33:0/984176159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab7411a2d0 0x55ab74122f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.823+0000 7fd021224740 1 -- 172.21.15.33:0/984176159 >> 172.21.15.33:0/984176159 conn(0x55ab74114a00 msgr2=0x55ab74113310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:49.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.823+0000 7fd021224740 1 -- 172.21.15.33:0/984176159 shutdown_connections 2024-03-19T02:53:49.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.823+0000 7fd021224740 1 -- 172.21.15.33:0/984176159 wait complete. 2024-03-19T02:53:49.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.823+0000 7fd021224740 1 -- start start 2024-03-19T02:53:49.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.824+0000 7fd021224740 1 Processor -- start 2024-03-19T02:53:49.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.824+0000 7fd021224740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab7411a2d0 0x55ab7412b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.824+0000 7fd021224740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab7411abf0 0x55ab74126cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.824+0000 7fd021224740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab741237d0 0x55ab74127200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.824+0000 7fd021224740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd0181af430 con 0x55ab7411abf0 2024-03-19T02:53:49.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.824+0000 7fd021224740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd0181b08a0 con 0x55ab7411a2d0 2024-03-19T02:53:49.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.824+0000 7fd021224740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd0181afb00 con 0x55ab741237d0 2024-03-19T02:53:49.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.824+0000 7fd021224740 10 client.0 _ll_register_callbacks cb 0x7fd00c0cc700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:53:49.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.824+0000 7fd0177fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab7411abf0 0x55ab74126cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.824+0000 7fd0177fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab7411abf0 0x55ab74126cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34414/0 (socket says 172.21.15.33:34414) 2024-03-19T02:53:49.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.824+0000 7fd0177fe640 1 -- 172.21.15.33:0/395638050 learned_addr learned my addr 172.21.15.33:0/395638050 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:49.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.824+0000 7fd017fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab7411a2d0 0x55ab7412b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.824+0000 7fd01caf8640 1 --2- 172.21.15.33:0/395638050 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab741237d0 0x55ab74127200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.825+0000 7fd0177fe640 1 -- 172.21.15.33:0/395638050 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab741237d0 msgr2=0x55ab74127200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:49.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.825+0000 7fd0177fe640 1 --2- 172.21.15.33:0/395638050 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab741237d0 0x55ab74127200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.825+0000 7fd0177fe640 1 -- 172.21.15.33:0/395638050 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab7411a2d0 msgr2=0x55ab7412b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:49.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.825+0000 7fd0177fe640 1 --2- 172.21.15.33:0/395638050 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab7411a2d0 0x55ab7412b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.825+0000 7fd0177fe640 1 -- 172.21.15.33:0/395638050 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab74127740 con 0x55ab7411abf0 2024-03-19T02:53:49.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.825+0000 7fd01caf8640 1 --2- 172.21.15.33:0/395638050 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab741237d0 0x55ab74127200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:49.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.825+0000 7fd017fff640 1 --2- 172.21.15.33:0/395638050 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab7411a2d0 0x55ab7412b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:49.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.825+0000 7fd0177fe640 1 --2- 172.21.15.33:0/395638050 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab7411abf0 0x55ab74126cc0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fd00c0a3940 tx=0x7fd00c003d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da6d57cd178cfa72 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:49.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.825+0000 7fcff77fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:53:49.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.825+0000 7fcff77fe640 1 -- 172.21.15.33:0/395638050 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd00c0b8030 con 0x55ab7411abf0 2024-03-19T02:53:49.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.825+0000 7fcff77fe640 1 -- 172.21.15.33:0/395638050 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fd00c0bb050 con 0x55ab7411abf0 2024-03-19T02:53:49.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.825+0000 7fd021224740 1 -- 172.21.15.33:0/395638050 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd00c051070 con 0x55ab7411abf0 2024-03-19T02:53:49.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.825+0000 7fcff77fe640 1 -- 172.21.15.33:0/395638050 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd00c083050 con 0x55ab7411abf0 2024-03-19T02:53:49.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.825+0000 7fd021224740 10 client.15756 fetch_fsmap learned FSMap version 25 2024-03-19T02:53:49.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.825+0000 7fd021224740 1 -- 172.21.15.33:0/395638050 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ab74199a90 con 0x55ab7411abf0 2024-03-19T02:53:49.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.825+0000 7fcff77fe640 1 -- 172.21.15.33:0/395638050 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd00c051070 con 0x55ab7411abf0 2024-03-19T02:53:49.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.825+0000 7fcff77fe640 1 -- 172.21.15.33:0/395638050 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fd00c0b42e0 con 0x55ab7411abf0 2024-03-19T02:53:49.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.826+0000 7fd021224740 10 client.15756 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:53:49.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.826+0000 7fd021224740 10 client.15756 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T02:53:49.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.826+0000 7fd021224740 20 client.15756 populate_metadata read hostname 'smithi033' 2024-03-19T02:53:49.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.826+0000 7fd021224740 1 --2- 172.21.15.33:0/395638050 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55ab7419a070 0x55ab7419c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.826+0000 7fd021224740 4 client.15756 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:53:49.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.826+0000 7fd021224740 1 -- 172.21.15.33:0/395638050 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ab7408a9f0 con 0x55ab7419a070 2024-03-19T02:53:49.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.826+0000 7fd017fff640 1 --2- 172.21.15.33:0/395638050 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55ab7419a070 0x55ab7419c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.826+0000 7fcff77fe640 10 client.15756 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T02:53:49.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.826+0000 7fd017fff640 1 --2- 172.21.15.33:0/395638050 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55ab7419a070 0x55ab7419c490 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a9619b6d68c2b116 server_cookie=574f42624be2f821 in_seq=0 out_seq=0 2024-03-19T02:53:49.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.826+0000 7fcff77fe640 1 -- 172.21.15.33:0/395638050 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ab7408a9f0 con 0x55ab7419a070 2024-03-19T02:53:49.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.826+0000 7fcff77fe640 10 client.15756 handle_command_reply: tid=0 2024-03-19T02:53:49.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.826+0000 7fd021224740 1 client.15756 shutdown 2024-03-19T02:53:49.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.826+0000 7fd021224740 1 -- 172.21.15.33:0/395638050 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab7411abf0 msgr2=0x55ab74126cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.826+0000 7fd021224740 1 --2- 172.21.15.33:0/395638050 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab7411abf0 0x55ab74126cc0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fd00c0a3940 tx=0x7fd00c003d20 comp rx=0 tx=0).stop 2024-03-19T02:53:49.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.827+0000 7fd021224740 1 -- 172.21.15.33:0/395638050 shutdown_connections 2024-03-19T02:53:49.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.827+0000 7fd021224740 1 --2- 172.21.15.33:0/395638050 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55ab7419a070 0x55ab7419c490 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.827+0000 7fd021224740 1 --2- 172.21.15.33:0/395638050 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ab741237d0 0x55ab74127200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.827+0000 7fd021224740 1 --2- 172.21.15.33:0/395638050 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ab7411abf0 0x55ab74126cc0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.827+0000 7fd021224740 1 --2- 172.21.15.33:0/395638050 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ab7411a2d0 0x55ab7412b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.827+0000 7fd021224740 1 -- 172.21.15.33:0/395638050 >> 172.21.15.33:0/395638050 conn(0x55ab74114a00 msgr2=0x55ab74113310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:49.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.827+0000 7fcff77fe640 0 client.15756 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T02:53:49.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.827+0000 7fd021224740 1 -- 172.21.15.33:0/395638050 shutdown_connections 2024-03-19T02:53:49.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.827+0000 7fd021224740 1 -- 172.21.15.33:0/395638050 wait complete. 2024-03-19T02:53:49.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.827+0000 7fd021224740 20 client.15756 trim_cache size 0 max 16384 2024-03-19T02:53:49.834 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:53:49.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.827+0000 7fd01e55b640 1 -- 172.21.15.33:0/3646518834 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd0000a2a00 msgr2=0x7fd0000a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.827+0000 7fd01e55b640 1 --2- 172.21.15.33:0/3646518834 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd0000a2a00 0x7fd0000a4ec0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd010005fd0 tx=0x7fd010009e90 comp rx=0 tx=0).stop 2024-03-19T02:53:49.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.827+0000 7fd01e55b640 1 -- 172.21.15.33:0/3646518834 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0181021b0 msgr2=0x7fd018074060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.827+0000 7fd01e55b640 1 --2- 172.21.15.33:0/3646518834 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0181021b0 0x7fd018074060 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd00c005fd0 tx=0x7fd00c00ee50 comp rx=0 tx=0).stop 2024-03-19T02:53:49.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.827+0000 7fd01e55b640 1 -- 172.21.15.33:0/3646518834 shutdown_connections 2024-03-19T02:53:49.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.827+0000 7fd01e55b640 1 --2- 172.21.15.33:0/3646518834 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd0000a2a00 0x7fd0000a4ec0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.827+0000 7fd01e55b640 1 --2- 172.21.15.33:0/3646518834 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd018107800 0x7fd0180745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.827+0000 7fd01e55b640 1 --2- 172.21.15.33:0/3646518834 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0181021b0 0x7fd018074060 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.827+0000 7fd01e55b640 1 --2- 172.21.15.33:0/3646518834 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0181017f0 0x7fd018073b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.827+0000 7fd01e55b640 1 -- 172.21.15.33:0/3646518834 >> 172.21.15.33:0/3646518834 conn(0x7fd0180fd2c0 msgr2=0x7fd0180fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:49.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.828+0000 7fd01e55b640 1 -- 172.21.15.33:0/3646518834 shutdown_connections 2024-03-19T02:53:49.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.828+0000 7fd01e55b640 1 -- 172.21.15.33:0/3646518834 wait complete. 2024-03-19T02:53:49.838 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:53:49.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.947+0000 7f3a1acde640 1 -- 172.21.15.33:0/745630202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a14101050 msgr2=0x7f3a1410d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.947+0000 7f3a1acde640 1 --2- 172.21.15.33:0/745630202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a14101050 0x7f3a1410d560 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c009cd0 tx=0x7f3a0c02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.947+0000 7f3a1acde640 1 -- 172.21.15.33:0/745630202 shutdown_connections 2024-03-19T02:53:49.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.947+0000 7f3a1acde640 1 --2- 172.21.15.33:0/745630202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3a14100690 0x7f3a14100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.947+0000 7f3a1acde640 1 --2- 172.21.15.33:0/745630202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a14101050 0x7f3a1410d560 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.947+0000 7f3a1acde640 1 --2- 172.21.15.33:0/745630202 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a141066a0 0x7f3a14106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.947+0000 7f3a1acde640 1 -- 172.21.15.33:0/745630202 >> 172.21.15.33:0/745630202 conn(0x7f3a14077850 msgr2=0x7f3a14077c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:49.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.947+0000 7f3a1acde640 1 -- 172.21.15.33:0/745630202 shutdown_connections 2024-03-19T02:53:49.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.947+0000 7f3a1acde640 1 -- 172.21.15.33:0/745630202 wait complete. 2024-03-19T02:53:49.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.948+0000 7f3a1acde640 1 Processor -- start 2024-03-19T02:53:49.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.948+0000 7f3a1acde640 1 -- start start 2024-03-19T02:53:49.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.948+0000 7f3a1acde640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a14100690 0x7f3a14196f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.948+0000 7f3a1acde640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3a14101050 0x7f3a14197450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.948+0000 7f3a1acde640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a141066a0 0x7f3a1419b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.948+0000 7f3a1acde640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3a14110960 con 0x7f3a14100690 2024-03-19T02:53:49.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.948+0000 7f3a1acde640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3a141107e0 con 0x7f3a141066a0 2024-03-19T02:53:49.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.948+0000 7f3a1acde640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3a14110ae0 con 0x7f3a14101050 2024-03-19T02:53:49.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.948+0000 7f3a18a7a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a14100690 0x7f3a14196f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.948+0000 7f3a18a7a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a14100690 0x7f3a14196f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34434/0 (socket says 172.21.15.33:34434) 2024-03-19T02:53:49.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.948+0000 7f3a18a7a640 1 -- 172.21.15.33:0/1045665154 learned_addr learned my addr 172.21.15.33:0/1045665154 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:49.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.948+0000 7f3a13fff640 1 --2- 172.21.15.33:0/1045665154 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3a14101050 0x7f3a14197450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.948+0000 7f3a1927b640 1 --2- 172.21.15.33:0/1045665154 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a141066a0 0x7f3a1419b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.948+0000 7f3a18a7a640 1 -- 172.21.15.33:0/1045665154 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3a14101050 msgr2=0x7f3a14197450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.948+0000 7f3a18a7a640 1 --2- 172.21.15.33:0/1045665154 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3a14101050 0x7f3a14197450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.948+0000 7f3a18a7a640 1 -- 172.21.15.33:0/1045665154 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a141066a0 msgr2=0x7f3a1419b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.948+0000 7f3a18a7a640 1 --2- 172.21.15.33:0/1045665154 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a141066a0 0x7f3a1419b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.948+0000 7f3a18a7a640 1 -- 172.21.15.33:0/1045665154 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a1419bf00 con 0x7f3a14100690 2024-03-19T02:53:49.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.949+0000 7f3a18a7a640 1 --2- 172.21.15.33:0/1045665154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a14100690 0x7f3a14196f10 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f3a0800ea60 tx=0x7f3a0800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:49.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.949+0000 7f3a11ffb640 1 -- 172.21.15.33:0/1045665154 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0800cde0 con 0x7f3a14100690 2024-03-19T02:53:49.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.949+0000 7f3a11ffb640 1 -- 172.21.15.33:0/1045665154 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a080035f0 con 0x7f3a14100690 2024-03-19T02:53:49.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.949+0000 7f3a11ffb640 1 -- 172.21.15.33:0/1045665154 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a08010690 con 0x7f3a14100690 2024-03-19T02:53:49.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.949+0000 7f3a1acde640 1 -- 172.21.15.33:0/1045665154 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a1419c1f0 con 0x7f3a14100690 2024-03-19T02:53:49.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.949+0000 7f3a1acde640 1 -- 172.21.15.33:0/1045665154 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a141a3a80 con 0x7f3a14100690 2024-03-19T02:53:49.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.951+0000 7f3a11ffb640 1 -- 172.21.15.33:0/1045665154 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 23) ==== 137466+0+0 (secure 0 0 0) 0x7f3a080037a0 con 0x7f3a14100690 2024-03-19T02:53:49.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.952+0000 7f3a1d99b740 1 Processor -- start 2024-03-19T02:53:49.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.952+0000 7f3a11ffb640 1 --2- 172.21.15.33:0/1045665154 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f3a040a2a00 0x7f3a040a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.952+0000 7f3a1d99b740 1 -- start start 2024-03-19T02:53:49.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.952+0000 7f3a1927b640 1 --2- 172.21.15.33:0/1045665154 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f3a040a2a00 0x7f3a040a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.952+0000 7f3a11ffb640 1 -- 172.21.15.33:0/1045665154 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f3a080cf5a0 con 0x7f3a14100690 2024-03-19T02:53:49.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.952+0000 7f3a1d99b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b24e445fc0 0x55b24e4463a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.952+0000 7f3a1d99b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b24e4468e0 0x55b24e4514a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.952+0000 7f3a1d99b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b24e4519e0 0x55b24e453dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.952+0000 7f3a1d99b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b24e3a94b0 con 0x55b24e4519e0 2024-03-19T02:53:49.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.952+0000 7f3a1d99b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55b24e351d70 con 0x55b24e4468e0 2024-03-19T02:53:49.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.952+0000 7f3a1d99b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b24e3a5550 con 0x55b24e445fc0 2024-03-19T02:53:49.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.952+0000 7f3a1927b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b24e4519e0 0x55b24e453dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.952+0000 7f3a13fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b24e4468e0 0x55b24e4514a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.952+0000 7f3a18a7a640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b24e445fc0 0x55b24e4463a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.952+0000 7f3a13fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b24e4468e0 0x55b24e4514a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:58162/0 (socket says 172.21.15.33:58162) 2024-03-19T02:53:49.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.952+0000 7f3a18a7a640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b24e445fc0 0x55b24e4463a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:35542/0 (socket says 172.21.15.33:35542) 2024-03-19T02:53:49.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.952+0000 7f3a1927b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b24e4519e0 0x55b24e453dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34448/0 (socket says 172.21.15.33:34448) 2024-03-19T02:53:49.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.952+0000 7f3a13fff640 1 -- 172.21.15.33:0/1261348115 learned_addr learned my addr 172.21.15.33:0/1261348115 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:49.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.953+0000 7f3a13fff640 1 -- 172.21.15.33:0/1261348115 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b24e445fc0 msgr2=0x55b24e4463a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:49.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.953+0000 7f3a13fff640 1 --2- 172.21.15.33:0/1261348115 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b24e445fc0 0x55b24e4463a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.953+0000 7f3a13fff640 1 -- 172.21.15.33:0/1261348115 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b24e4519e0 msgr2=0x55b24e453dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:49.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.953+0000 7f3a13fff640 1 --2- 172.21.15.33:0/1261348115 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b24e4519e0 0x55b24e453dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.953+0000 7f3a13fff640 1 -- 172.21.15.33:0/1261348115 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b24e374210 con 0x55b24e4468e0 2024-03-19T02:53:49.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.953+0000 7f3a1927b640 1 --2- 172.21.15.33:0/1045665154 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f3a040a2a00 0x7f3a040a4ec0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c009ca0 tx=0x7f3a0c032d80 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:49.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.953+0000 7f3a13fff640 1 --2- 172.21.15.33:0/1261348115 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b24e4468e0 0x55b24e4514a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3a00009ca0 tx=0x7f3a00011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48ffe30c0b2816f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:49.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.953+0000 7f39feffd640 1 -- 172.21.15.33:0/1261348115 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0001e070 con 0x55b24e4468e0 2024-03-19T02:53:49.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.953+0000 7f39feffd640 1 -- 172.21.15.33:0/1261348115 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a00019ae0 con 0x55b24e4468e0 2024-03-19T02:53:49.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.953+0000 7f39feffd640 1 -- 172.21.15.33:0/1261348115 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0000e8e0 con 0x55b24e4468e0 2024-03-19T02:53:49.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.954+0000 7f3a1d99b740 1 -- 172.21.15.33:0/1261348115 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b24e4468e0 msgr2=0x55b24e4514a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.954+0000 7f3a1d99b740 1 --2- 172.21.15.33:0/1261348115 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b24e4468e0 0x55b24e4514a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3a00009ca0 tx=0x7f3a00011810 comp rx=0 tx=0).stop 2024-03-19T02:53:49.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.954+0000 7f3a1d99b740 1 -- 172.21.15.33:0/1261348115 shutdown_connections 2024-03-19T02:53:49.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.954+0000 7f3a1d99b740 1 --2- 172.21.15.33:0/1261348115 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b24e4519e0 0x55b24e453dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.954+0000 7f3a1d99b740 1 --2- 172.21.15.33:0/1261348115 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b24e4468e0 0x55b24e4514a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.954+0000 7f3a1d99b740 1 --2- 172.21.15.33:0/1261348115 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b24e445fc0 0x55b24e4463a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.954+0000 7f3a1d99b740 1 -- 172.21.15.33:0/1261348115 >> 172.21.15.33:0/1261348115 conn(0x55b24e440a00 msgr2=0x55b24e440de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:49.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.954+0000 7f3a1d99b740 1 -- 172.21.15.33:0/1261348115 shutdown_connections 2024-03-19T02:53:49.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.954+0000 7f3a1d99b740 1 -- 172.21.15.33:0/1261348115 wait complete. 2024-03-19T02:53:49.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.954+0000 7f3a1d99b740 1 -- start start 2024-03-19T02:53:49.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.954+0000 7f3a1d99b740 1 Processor -- start 2024-03-19T02:53:49.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.955+0000 7f3a1d99b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b24e445fc0 0x55b24e44d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.955+0000 7f3a1d99b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b24e4468e0 0x55b24e44d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.955+0000 7f3a1d99b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b24e4519e0 0x55b24e4598e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.955+0000 7f3a1d99b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b24e4475b0 con 0x55b24e445fc0 2024-03-19T02:53:49.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.955+0000 7f3a1d99b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55b24e365310 con 0x55b24e4519e0 2024-03-19T02:53:49.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.955+0000 7f3a1d99b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b24e447730 con 0x55b24e4468e0 2024-03-19T02:53:49.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.955+0000 7f3a18a7a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b24e445fc0 0x55b24e44d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.955+0000 7f3a1d99b740 10 client.0 _ll_register_callbacks cb 0x55b24e3592d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:53:49.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.955+0000 7f3a18a7a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b24e445fc0 0x55b24e44d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34464/0 (socket says 172.21.15.33:34464) 2024-03-19T02:53:49.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.955+0000 7f3a18a7a640 1 -- 172.21.15.33:0/25348127 learned_addr learned my addr 172.21.15.33:0/25348127 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:49.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.955+0000 7f3a13fff640 1 --2- 172.21.15.33:0/25348127 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b24e4468e0 0x55b24e44d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.955+0000 7f3a1927b640 1 --2- 172.21.15.33:0/25348127 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b24e4519e0 0x55b24e4598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.955+0000 7f3a18a7a640 1 -- 172.21.15.33:0/25348127 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b24e4468e0 msgr2=0x55b24e44d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:49.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.955+0000 7f3a18a7a640 1 --2- 172.21.15.33:0/25348127 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b24e4468e0 0x55b24e44d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.955+0000 7f3a18a7a640 1 -- 172.21.15.33:0/25348127 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b24e4519e0 msgr2=0x55b24e4598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:49.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.955+0000 7f3a18a7a640 1 --2- 172.21.15.33:0/25348127 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b24e4519e0 0x55b24e4598e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.955+0000 7f3a18a7a640 1 -- 172.21.15.33:0/25348127 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b24e0bd3f0 con 0x55b24e445fc0 2024-03-19T02:53:49.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.955+0000 7f3a1927b640 1 --2- 172.21.15.33:0/25348127 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b24e4519e0 0x55b24e4598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:49.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.956+0000 7f3a18a7a640 1 --2- 172.21.15.33:0/25348127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b24e445fc0 0x55b24e44d1b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f3a0800d010 tx=0x7f3a0809b2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=351cdfafa0d03102 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:49.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.956+0000 7f39f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:53:49.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.956+0000 7f39f3fff640 1 -- 172.21.15.33:0/25348127 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a080b3070 con 0x55b24e445fc0 2024-03-19T02:53:49.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.956+0000 7f39f3fff640 1 -- 172.21.15.33:0/25348127 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f3a08097d50 con 0x55b24e445fc0 2024-03-19T02:53:49.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.956+0000 7f3a1d99b740 1 -- 172.21.15.33:0/25348127 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a00019ae0 con 0x55b24e445fc0 2024-03-19T02:53:49.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.956+0000 7f39f3fff640 1 -- 172.21.15.33:0/25348127 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3a0809bde0 con 0x55b24e445fc0 2024-03-19T02:53:49.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.956+0000 7f3a1d99b740 10 client.15768 fetch_fsmap learned FSMap version 25 2024-03-19T02:53:49.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.956+0000 7f3a1d99b740 1 -- 172.21.15.33:0/25348127 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b24e1877d0 con 0x55b24e445fc0 2024-03-19T02:53:49.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.956+0000 7f39f3fff640 1 -- 172.21.15.33:0/25348127 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a00019ae0 con 0x55b24e445fc0 2024-03-19T02:53:49.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.956+0000 7f39f3fff640 1 -- 172.21.15.33:0/25348127 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f3a080ccb30 con 0x55b24e445fc0 2024-03-19T02:53:49.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.956+0000 7f3a1d99b740 10 client.15768 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:53:49.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.956+0000 7f3a1d99b740 10 client.15768 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:53:49.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.956+0000 7f3a1d99b740 20 client.15768 populate_metadata read hostname 'smithi033' 2024-03-19T02:53:49.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.956+0000 7f3a1d99b740 1 --2- 172.21.15.33:0/25348127 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55b24e4640d0 0x55b24e4664e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.956+0000 7f3a1d99b740 4 client.15768 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:53:49.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.956+0000 7f3a1d99b740 1 -- 172.21.15.33:0/25348127 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b24e4073e0 con 0x55b24e4640d0 2024-03-19T02:53:49.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.957+0000 7f3a13fff640 1 --2- 172.21.15.33:0/25348127 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55b24e4640d0 0x55b24e4664e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.957+0000 7f3a13fff640 1 --2- 172.21.15.33:0/25348127 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55b24e4640d0 0x55b24e4664e0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5993315cb6d03e5d server_cookie=7b905e749812f3d0 in_seq=0 out_seq=0 2024-03-19T02:53:49.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.957+0000 7f39f3fff640 10 client.15768 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:53:49.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.958+0000 7f39f3fff640 1 -- 172.21.15.33:0/25348127 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b24e4073e0 con 0x55b24e4640d0 2024-03-19T02:53:49.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.958+0000 7f39f3fff640 10 client.15768 handle_command_reply: tid=0 2024-03-19T02:53:49.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.958+0000 7f3a1d99b740 1 client.15768 shutdown 2024-03-19T02:53:49.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.958+0000 7f3a1d99b740 1 -- 172.21.15.33:0/25348127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b24e445fc0 msgr2=0x55b24e44d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.958+0000 7f3a1d99b740 1 --2- 172.21.15.33:0/25348127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b24e445fc0 0x55b24e44d1b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f3a0800d010 tx=0x7f3a0809b2e0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.958+0000 7f3a1d99b740 1 -- 172.21.15.33:0/25348127 shutdown_connections 2024-03-19T02:53:49.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.958+0000 7f3a1d99b740 1 --2- 172.21.15.33:0/25348127 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55b24e4640d0 0x55b24e4664e0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.958+0000 7f3a1d99b740 1 --2- 172.21.15.33:0/25348127 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b24e4468e0 0x55b24e44d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.958+0000 7f3a1d99b740 1 --2- 172.21.15.33:0/25348127 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b24e4519e0 0x55b24e4598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.958+0000 7f39f3fff640 0 client.15768 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:53:49.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.958+0000 7f3a1d99b740 1 --2- 172.21.15.33:0/25348127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b24e445fc0 0x55b24e44d1b0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.958+0000 7f3a1d99b740 1 -- 172.21.15.33:0/25348127 >> 172.21.15.33:0/25348127 conn(0x55b24e440a00 msgr2=0x55b24e442750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:49.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.958+0000 7f3a1d99b740 1 -- 172.21.15.33:0/25348127 shutdown_connections 2024-03-19T02:53:49.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.958+0000 7f3a1d99b740 1 -- 172.21.15.33:0/25348127 wait complete. 2024-03-19T02:53:49.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.958+0000 7f3a1d99b740 20 client.15768 trim_cache size 0 max 16384 2024-03-19T02:53:49.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:49 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3609170606' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:53:49.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.985+0000 7f3a1d99b740 1 Processor -- start 2024-03-19T02:53:49.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.985+0000 7f3a1d99b740 1 -- start start 2024-03-19T02:53:49.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.986+0000 7f3a1d99b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b24e445fc0 0x55b24e459870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.986+0000 7f3a1d99b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b24e4468e0 0x55b24e460330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.986+0000 7f3a1d99b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b24e4519e0 0x55b24e44bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.986+0000 7f3a1d99b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3a141af210 con 0x55b24e4519e0 2024-03-19T02:53:49.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.986+0000 7f3a1d99b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3a141acae0 con 0x55b24e4468e0 2024-03-19T02:53:49.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.986+0000 7f3a1d99b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3a141aa910 con 0x55b24e445fc0 2024-03-19T02:53:49.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.986+0000 7f3a1927b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b24e4519e0 0x55b24e44bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.986+0000 7f3a1927b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b24e4519e0 0x55b24e44bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34476/0 (socket says 172.21.15.33:34476) 2024-03-19T02:53:49.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.986+0000 7f3a1927b640 1 -- 172.21.15.33:0/194134919 learned_addr learned my addr 172.21.15.33:0/194134919 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:49.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.986+0000 7f3a13fff640 1 --2- 172.21.15.33:0/194134919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b24e4468e0 0x55b24e460330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.986+0000 7f3a18a7a640 1 --2- 172.21.15.33:0/194134919 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b24e445fc0 0x55b24e459870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.986+0000 7f3a1927b640 1 -- 172.21.15.33:0/194134919 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b24e445fc0 msgr2=0x55b24e459870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.986+0000 7f3a1927b640 1 --2- 172.21.15.33:0/194134919 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b24e445fc0 0x55b24e459870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.986+0000 7f3a1927b640 1 -- 172.21.15.33:0/194134919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b24e4468e0 msgr2=0x55b24e460330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.986+0000 7f3a1927b640 1 --2- 172.21.15.33:0/194134919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b24e4468e0 0x55b24e460330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.986+0000 7f3a1927b640 1 -- 172.21.15.33:0/194134919 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b24e416880 con 0x55b24e4519e0 2024-03-19T02:53:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.986+0000 7f3a13fff640 1 --2- 172.21.15.33:0/194134919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b24e4468e0 0x55b24e460330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.986+0000 7f3a1927b640 1 --2- 172.21.15.33:0/194134919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b24e4519e0 0x55b24e44bfa0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c02c700 tx=0x7f3a0c02e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=701aebf02e85be9c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.987+0000 7f39fe7fc640 1 -- 172.21.15.33:0/194134919 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c039040 con 0x55b24e4519e0 2024-03-19T02:53:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.987+0000 7f39fe7fc640 1 -- 172.21.15.33:0/194134919 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a0c0043b0 con 0x55b24e4519e0 2024-03-19T02:53:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.987+0000 7f39fe7fc640 1 -- 172.21.15.33:0/194134919 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c034980 con 0x55b24e4519e0 2024-03-19T02:53:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.987+0000 7f3a1d99b740 1 -- 172.21.15.33:0/194134919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b24e4519e0 msgr2=0x55b24e44bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.987+0000 7f3a1d99b740 1 --2- 172.21.15.33:0/194134919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b24e4519e0 0x55b24e44bfa0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c02c700 tx=0x7f3a0c02e050 comp rx=0 tx=0).stop 2024-03-19T02:53:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.987+0000 7f3a1d99b740 1 -- 172.21.15.33:0/194134919 shutdown_connections 2024-03-19T02:53:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.987+0000 7f3a1d99b740 1 --2- 172.21.15.33:0/194134919 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b24e4519e0 0x55b24e44bfa0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.987+0000 7f3a1d99b740 1 --2- 172.21.15.33:0/194134919 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b24e4468e0 0x55b24e460330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.987+0000 7f3a1d99b740 1 --2- 172.21.15.33:0/194134919 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b24e445fc0 0x55b24e459870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.987+0000 7f3a1d99b740 1 -- 172.21.15.33:0/194134919 >> 172.21.15.33:0/194134919 conn(0x55b24e440a00 msgr2=0x55b24e44b890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:49.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.987+0000 7f3a1d99b740 1 -- 172.21.15.33:0/194134919 shutdown_connections 2024-03-19T02:53:49.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.988+0000 7f3a1d99b740 1 -- 172.21.15.33:0/194134919 wait complete. 2024-03-19T02:53:49.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.988+0000 7f3a1d99b740 1 -- start start 2024-03-19T02:53:49.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.988+0000 7f3a1d99b740 1 Processor -- start 2024-03-19T02:53:49.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.988+0000 7f3a1d99b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b24e445fc0 0x55b24e460350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.988+0000 7f3a1d99b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b24e4468e0 0x55b24e45c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.988+0000 7f3a1d99b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b24e4519e0 0x55b24e45ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.988+0000 7f3a1d99b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3a141aafa0 con 0x55b24e445fc0 2024-03-19T02:53:49.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.988+0000 7f3a1d99b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3a141ac410 con 0x55b24e4468e0 2024-03-19T02:53:49.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.988+0000 7f3a1d99b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3a141ab670 con 0x55b24e4519e0 2024-03-19T02:53:49.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.988+0000 7f3a1d99b740 10 client.0 _ll_register_callbacks cb 0x7f3a080972a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:53:49.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.988+0000 7f3a18a7a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b24e445fc0 0x55b24e460350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.989+0000 7f3a18a7a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b24e445fc0 0x55b24e460350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34486/0 (socket says 172.21.15.33:34486) 2024-03-19T02:53:49.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.989+0000 7f3a18a7a640 1 -- 172.21.15.33:0/423983048 learned_addr learned my addr 172.21.15.33:0/423983048 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:49.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.989+0000 7f3a13fff640 1 --2- 172.21.15.33:0/423983048 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b24e4468e0 0x55b24e45c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.989+0000 7f3a1927b640 1 --2- 172.21.15.33:0/423983048 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b24e4519e0 0x55b24e45ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.989+0000 7f3a18a7a640 1 -- 172.21.15.33:0/423983048 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b24e4519e0 msgr2=0x55b24e45ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:49.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.989+0000 7f3a18a7a640 1 --2- 172.21.15.33:0/423983048 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b24e4519e0 0x55b24e45ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.989+0000 7f3a18a7a640 1 -- 172.21.15.33:0/423983048 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b24e4468e0 msgr2=0x55b24e45c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:49.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.989+0000 7f3a18a7a640 1 --2- 172.21.15.33:0/423983048 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b24e4468e0 0x55b24e45c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.989+0000 7f3a18a7a640 1 -- 172.21.15.33:0/423983048 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b24e453dd0 con 0x55b24e445fc0 2024-03-19T02:53:49.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.989+0000 7f3a18a7a640 1 --2- 172.21.15.33:0/423983048 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b24e445fc0 0x55b24e460350 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3a08086010 tx=0x7f3a0800ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97c5c56c9333fe16 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:49.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.989+0000 7f39f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:53:49.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.989+0000 7f39f3fff640 1 -- 172.21.15.33:0/423983048 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a080b3070 con 0x55b24e445fc0 2024-03-19T02:53:49.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.989+0000 7f39f3fff640 1 -- 172.21.15.33:0/423983048 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f3a0809ac00 con 0x55b24e445fc0 2024-03-19T02:53:49.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.989+0000 7f3a1d99b740 1 -- 172.21.15.33:0/423983048 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a0c0043b0 con 0x55b24e445fc0 2024-03-19T02:53:49.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.989+0000 7f39f3fff640 1 -- 172.21.15.33:0/423983048 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3a080cf3c0 con 0x55b24e445fc0 2024-03-19T02:53:49.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.989+0000 7f3a1d99b740 10 client.15774 fetch_fsmap learned FSMap version 25 2024-03-19T02:53:49.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.989+0000 7f3a1d99b740 1 -- 172.21.15.33:0/423983048 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b24e4ca0e0 con 0x55b24e445fc0 2024-03-19T02:53:49.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.990+0000 7f39f3fff640 1 -- 172.21.15.33:0/423983048 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a0c0043b0 con 0x55b24e445fc0 2024-03-19T02:53:49.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.990+0000 7f39f3fff640 1 -- 172.21.15.33:0/423983048 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f3a08081070 con 0x55b24e445fc0 2024-03-19T02:53:49.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.990+0000 7f3a1d99b740 10 client.15774 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:53:49.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.990+0000 7f3a1d99b740 10 client.15774 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:53:49.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.990+0000 7f3a1d99b740 20 client.15774 populate_metadata read hostname 'smithi033' 2024-03-19T02:53:49.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.990+0000 7f3a1d99b740 1 --2- 172.21.15.33:0/423983048 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55b24e4ca650 0x55b24e4cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:49.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.990+0000 7f3a1d99b740 4 client.15774 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:53:49.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.990+0000 7f3a1d99b740 1 -- 172.21.15.33:0/423983048 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b24e3b69f0 con 0x55b24e4ca650 2024-03-19T02:53:49.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.990+0000 7f3a13fff640 1 --2- 172.21.15.33:0/423983048 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55b24e4ca650 0x55b24e4cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:49.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.990+0000 7f39f3fff640 10 client.15774 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:53:49.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.990+0000 7f3a13fff640 1 --2- 172.21.15.33:0/423983048 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55b24e4ca650 0x55b24e4cca70 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb9c708648d02803 server_cookie=31a27366fa03c6fc in_seq=0 out_seq=0 2024-03-19T02:53:49.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.991+0000 7f39f3fff640 1 -- 172.21.15.33:0/423983048 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b24e3b69f0 con 0x55b24e4ca650 2024-03-19T02:53:49.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.991+0000 7f39f3fff640 10 client.15774 handle_command_reply: tid=0 2024-03-19T02:53:49.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.991+0000 7f3a1d99b740 1 client.15774 shutdown 2024-03-19T02:53:49.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.991+0000 7f3a1d99b740 1 -- 172.21.15.33:0/423983048 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b24e445fc0 msgr2=0x55b24e460350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:49.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.991+0000 7f3a1d99b740 1 --2- 172.21.15.33:0/423983048 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b24e445fc0 0x55b24e460350 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3a08086010 tx=0x7f3a0800ef90 comp rx=0 tx=0).stop 2024-03-19T02:53:49.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.991+0000 7f3a1d99b740 1 -- 172.21.15.33:0/423983048 shutdown_connections 2024-03-19T02:53:49.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.991+0000 7f3a1d99b740 1 --2- 172.21.15.33:0/423983048 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55b24e4ca650 0x55b24e4cca70 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.991+0000 7f3a1d99b740 1 --2- 172.21.15.33:0/423983048 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b24e4519e0 0x55b24e45ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.991+0000 7f3a1d99b740 1 --2- 172.21.15.33:0/423983048 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b24e4468e0 0x55b24e45c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.991+0000 7f3a1d99b740 1 --2- 172.21.15.33:0/423983048 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b24e445fc0 0x55b24e460350 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:49.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.991+0000 7f3a1d99b740 1 -- 172.21.15.33:0/423983048 >> 172.21.15.33:0/423983048 conn(0x55b24e440a00 msgr2=0x55b24e44c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:49.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.991+0000 7f39f3fff640 0 client.15774 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:53:49.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.991+0000 7f3a1d99b740 1 -- 172.21.15.33:0/423983048 shutdown_connections 2024-03-19T02:53:49.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.991+0000 7f3a1d99b740 1 -- 172.21.15.33:0/423983048 wait complete. 2024-03-19T02:53:49.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.991+0000 7f3a1d99b740 20 client.15774 trim_cache size 0 max 16384 2024-03-19T02:53:49.999 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:53:50.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.992+0000 7f3a1acde640 1 -- 172.21.15.33:0/1045665154 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f3a040a2a00 msgr2=0x7f3a040a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:50.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.992+0000 7f3a1acde640 1 --2- 172.21.15.33:0/1045665154 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f3a040a2a00 0x7f3a040a4ec0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c009ca0 tx=0x7f3a0c032d80 comp rx=0 tx=0).stop 2024-03-19T02:53:50.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.992+0000 7f3a1acde640 1 -- 172.21.15.33:0/1045665154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a14100690 msgr2=0x7f3a14196f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:50.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.992+0000 7f3a1acde640 1 --2- 172.21.15.33:0/1045665154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a14100690 0x7f3a14196f10 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f3a0800ea60 tx=0x7f3a0800ef20 comp rx=0 tx=0).stop 2024-03-19T02:53:50.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.992+0000 7f3a1acde640 1 -- 172.21.15.33:0/1045665154 shutdown_connections 2024-03-19T02:53:50.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.992+0000 7f3a1acde640 1 --2- 172.21.15.33:0/1045665154 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f3a040a2a00 0x7f3a040a4ec0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:50.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.992+0000 7f3a1acde640 1 --2- 172.21.15.33:0/1045665154 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3a14101050 0x7f3a14197450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:50.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.992+0000 7f3a1acde640 1 --2- 172.21.15.33:0/1045665154 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a141066a0 0x7f3a1419b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:50.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.992+0000 7f3a1acde640 1 --2- 172.21.15.33:0/1045665154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a14100690 0x7f3a14196f10 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:50.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.992+0000 7f3a1acde640 1 -- 172.21.15.33:0/1045665154 >> 172.21.15.33:0/1045665154 conn(0x7f3a14077850 msgr2=0x7f3a14103360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:50.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.992+0000 7f3a1acde640 1 -- 172.21.15.33:0/1045665154 shutdown_connections 2024-03-19T02:53:50.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:49.992+0000 7f3a1acde640 1 -- 172.21.15.33:0/1045665154 wait complete. 2024-03-19T02:53:50.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:49 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3609170606' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:53:50.917 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:50 smithi155 ceph-mon[35804]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-19T02:53:50.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:50 smithi096 ceph-mon[35946]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-19T02:53:51.004 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:53:51.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:53:51.005 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:53:51.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:50 smithi033 ceph-mon[31973]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-19T02:53:51.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.116+0000 7f332717e640 1 -- 172.21.15.33:0/1480044444 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33201066a0 msgr2=0x7f3320106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:51.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.116+0000 7f332717e640 1 --2- 172.21.15.33:0/1480044444 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33201066a0 0x7f3320106a80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3314009900 tx=0x7f331402bc30 comp rx=0 tx=0).stop 2024-03-19T02:53:51.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.116+0000 7f332717e640 1 -- 172.21.15.33:0/1480044444 shutdown_connections 2024-03-19T02:53:51.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.116+0000 7f332717e640 1 --2- 172.21.15.33:0/1480044444 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3320100690 0x7f3320100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.116+0000 7f332717e640 1 --2- 172.21.15.33:0/1480044444 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3320101050 0x7f332010d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.116+0000 7f332717e640 1 --2- 172.21.15.33:0/1480044444 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33201066a0 0x7f3320106a80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.116+0000 7f332717e640 1 -- 172.21.15.33:0/1480044444 >> 172.21.15.33:0/1480044444 conn(0x7f3320077820 msgr2=0x7f3320077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:51.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.116+0000 7f332717e640 1 -- 172.21.15.33:0/1480044444 shutdown_connections 2024-03-19T02:53:51.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.116+0000 7f332717e640 1 -- 172.21.15.33:0/1480044444 wait complete. 2024-03-19T02:53:51.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.117+0000 7f332717e640 1 Processor -- start 2024-03-19T02:53:51.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.117+0000 7f332717e640 1 -- start start 2024-03-19T02:53:51.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.117+0000 7f332717e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3320100690 0x7f3320198da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.117+0000 7f332717e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3320101050 0x7f33201992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.118+0000 7f3324f1a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3320100690 0x7f3320198da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.118+0000 7f3324f1a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3320100690 0x7f3320198da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:58218/0 (socket says 172.21.15.33:58218) 2024-03-19T02:53:51.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.118+0000 7f331ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3320101050 0x7f33201992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.118+0000 7f331ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3320101050 0x7f33201992e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34506/0 (socket says 172.21.15.33:34506) 2024-03-19T02:53:51.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.118+0000 7f3324f1a640 1 -- 172.21.15.33:0/4182738377 learned_addr learned my addr 172.21.15.33:0/4182738377 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:51.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.118+0000 7f332717e640 1 --2- 172.21.15.33:0/4182738377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33201066a0 0x7f33201a3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.118+0000 7f332717e640 1 -- 172.21.15.33:0/4182738377 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3320110960 con 0x7f3320101050 2024-03-19T02:53:51.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.118+0000 7f332717e640 1 -- 172.21.15.33:0/4182738377 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f33201107e0 con 0x7f3320100690 2024-03-19T02:53:51.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.118+0000 7f332717e640 1 -- 172.21.15.33:0/4182738377 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3320110ae0 con 0x7f33201066a0 2024-03-19T02:53:51.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.118+0000 7f332571b640 1 --2- 172.21.15.33:0/4182738377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33201066a0 0x7f33201a3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.118+0000 7f3324f1a640 1 -- 172.21.15.33:0/4182738377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33201066a0 msgr2=0x7f33201a3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:51.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.118+0000 7f3324f1a640 1 --2- 172.21.15.33:0/4182738377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33201066a0 0x7f33201a3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.118+0000 7f3324f1a640 1 -- 172.21.15.33:0/4182738377 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3320101050 msgr2=0x7f33201992e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:51.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.118+0000 7f3324f1a640 1 --2- 172.21.15.33:0/4182738377 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3320101050 0x7f33201992e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.118+0000 7f3324f1a640 1 -- 172.21.15.33:0/4182738377 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33201a4360 con 0x7f3320100690 2024-03-19T02:53:51.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.118+0000 7f3324f1a640 1 --2- 172.21.15.33:0/4182738377 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3320100690 0x7f3320198da0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3314006010 tx=0x7f3314005d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:51.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.118+0000 7f331dffb640 1 -- 172.21.15.33:0/4182738377 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3314043020 con 0x7f3320100690 2024-03-19T02:53:51.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.119+0000 7f331dffb640 1 -- 172.21.15.33:0/4182738377 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3314004350 con 0x7f3320100690 2024-03-19T02:53:51.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.119+0000 7f331dffb640 1 -- 172.21.15.33:0/4182738377 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3314003a10 con 0x7f3320100690 2024-03-19T02:53:51.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.119+0000 7f332717e640 1 -- 172.21.15.33:0/4182738377 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33201a4530 con 0x7f3320100690 2024-03-19T02:53:51.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.119+0000 7f332717e640 1 -- 172.21.15.33:0/4182738377 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33201a4990 con 0x7f3320100690 2024-03-19T02:53:51.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.121+0000 7f331dffb640 1 -- 172.21.15.33:0/4182738377 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 23) ==== 137466+0+0 (secure 0 0 0) 0x7f3314046020 con 0x7f3320100690 2024-03-19T02:53:51.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.122+0000 7f331dffb640 1 --2- 172.21.15.33:0/4182738377 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f33100a2b00 0x7f33100a4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.122+0000 7f3329d6d740 1 Processor -- start 2024-03-19T02:53:51.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.122+0000 7f331ffff640 1 --2- 172.21.15.33:0/4182738377 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f33100a2b00 0x7f33100a4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.122+0000 7f331dffb640 1 -- 172.21.15.33:0/4182738377 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f33140f0d60 con 0x7f3320100690 2024-03-19T02:53:51.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.122+0000 7f3329d6d740 1 -- start start 2024-03-19T02:53:51.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.122+0000 7f3329d6d740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558827b0f380 0x558827b0f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.122+0000 7f3329d6d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558827b0fca0 0x558827b18340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.122+0000 7f3329d6d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558827b18880 0x558827b1ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.122+0000 7f3329d6d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558827a73160 con 0x558827b0fca0 2024-03-19T02:53:51.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.122+0000 7f3329d6d740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x558827a1be10 con 0x558827b0f380 2024-03-19T02:53:51.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.123+0000 7f3329d6d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558827a6f200 con 0x558827b18880 2024-03-19T02:53:51.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.123+0000 7f3324f1a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558827b0f380 0x558827b0f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.123+0000 7f3324f1a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558827b0f380 0x558827b0f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:58220/0 (socket says 172.21.15.33:58220) 2024-03-19T02:53:51.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.123+0000 7f3324f1a640 1 -- 172.21.15.33:0/1463646536 learned_addr learned my addr 172.21.15.33:0/1463646536 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:51.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.123+0000 7f332571b640 1 --2- 172.21.15.33:0/1463646536 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558827b18880 0x558827b1ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.123+0000 7f331ffff640 1 --2- 172.21.15.33:0/4182738377 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f33100a2b00 0x7f33100a4fc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3320069530 tx=0x7f330c005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:51.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.123+0000 7f3324f1a640 1 -- 172.21.15.33:0/1463646536 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558827b18880 msgr2=0x558827b1ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:51.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.123+0000 7f3324f1a640 1 --2- 172.21.15.33:0/1463646536 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558827b18880 0x558827b1ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.123+0000 7f3324f1a640 1 -- 172.21.15.33:0/1463646536 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558827b0fca0 msgr2=0x558827b18340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:51.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.123+0000 7f3324f1a640 1 --2- 172.21.15.33:0/1463646536 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558827b0fca0 0x558827b18340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.123+0000 7f3324f1a640 1 -- 172.21.15.33:0/1463646536 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55882784f7e0 con 0x558827b0f380 2024-03-19T02:53:51.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.123+0000 7f3324f1a640 1 --2- 172.21.15.33:0/1463646536 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558827b0f380 0x558827b0f760 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f33140b9410 tx=0x7f33140bcbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ff4359cb40738d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:51.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.123+0000 7f330affd640 1 -- 172.21.15.33:0/1463646536 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33140bcfb0 con 0x558827b0f380 2024-03-19T02:53:51.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.123+0000 7f330affd640 1 -- 172.21.15.33:0/1463646536 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f33140c1100 con 0x558827b0f380 2024-03-19T02:53:51.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.123+0000 7f330affd640 1 -- 172.21.15.33:0/1463646536 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33140c5520 con 0x558827b0f380 2024-03-19T02:53:51.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.124+0000 7f3329d6d740 1 -- 172.21.15.33:0/1463646536 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558827b0f380 msgr2=0x558827b0f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:51.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.124+0000 7f3329d6d740 1 --2- 172.21.15.33:0/1463646536 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558827b0f380 0x558827b0f760 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f33140b9410 tx=0x7f33140bcbb0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.124+0000 7f3329d6d740 1 -- 172.21.15.33:0/1463646536 shutdown_connections 2024-03-19T02:53:51.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.124+0000 7f3329d6d740 1 --2- 172.21.15.33:0/1463646536 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558827b18880 0x558827b1ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.124+0000 7f3329d6d740 1 --2- 172.21.15.33:0/1463646536 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558827b0fca0 0x558827b18340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.124+0000 7f3329d6d740 1 --2- 172.21.15.33:0/1463646536 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558827b0f380 0x558827b0f760 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.124+0000 7f3329d6d740 1 -- 172.21.15.33:0/1463646536 >> 172.21.15.33:0/1463646536 conn(0x558827b09ab0 msgr2=0x558827b09e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:51.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.124+0000 7f3329d6d740 1 -- 172.21.15.33:0/1463646536 shutdown_connections 2024-03-19T02:53:51.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.124+0000 7f3329d6d740 1 -- 172.21.15.33:0/1463646536 wait complete. 2024-03-19T02:53:51.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.124+0000 7f3329d6d740 1 -- start start 2024-03-19T02:53:51.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.124+0000 7f3329d6d740 1 Processor -- start 2024-03-19T02:53:51.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.125+0000 7f3329d6d740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558827b0f380 0x558827b15910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.125+0000 7f3329d6d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558827b0fca0 0x558827b15e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.125+0000 7f3329d6d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558827b18880 0x558827b16390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.125+0000 7f3329d6d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558827a1be10 con 0x558827b18880 2024-03-19T02:53:51.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.125+0000 7f3329d6d740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x558827a2f1a0 con 0x558827b0f380 2024-03-19T02:53:51.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.125+0000 7f3329d6d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558827b1b430 con 0x558827b0fca0 2024-03-19T02:53:51.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.125+0000 7f3329d6d740 10 client.0 _ll_register_callbacks cb 0x558827786c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:53:51.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.125+0000 7f332571b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558827b18880 0x558827b16390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.125+0000 7f332571b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558827b18880 0x558827b16390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34512/0 (socket says 172.21.15.33:34512) 2024-03-19T02:53:51.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.125+0000 7f332571b640 1 -- 172.21.15.33:0/4086046475 learned_addr learned my addr 172.21.15.33:0/4086046475 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:51.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.125+0000 7f3324f1a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558827b0f380 0x558827b15910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.125+0000 7f331ffff640 1 --2- 172.21.15.33:0/4086046475 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558827b0fca0 0x558827b15e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.125+0000 7f332571b640 1 -- 172.21.15.33:0/4086046475 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558827b0fca0 msgr2=0x558827b15e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:51.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.125+0000 7f332571b640 1 --2- 172.21.15.33:0/4086046475 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558827b0fca0 0x558827b15e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.125+0000 7f332571b640 1 -- 172.21.15.33:0/4086046475 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558827b0f380 msgr2=0x558827b15910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:51.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.125+0000 7f332571b640 1 --2- 172.21.15.33:0/4086046475 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558827b0f380 0x558827b15910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.125+0000 7f332571b640 1 -- 172.21.15.33:0/4086046475 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588278aff70 con 0x558827b18880 2024-03-19T02:53:51.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.125+0000 7f3324f1a640 1 --2- 172.21.15.33:0/4086046475 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558827b0f380 0x558827b15910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:51.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.126+0000 7f332571b640 1 --2- 172.21.15.33:0/4086046475 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558827b18880 0x558827b16390 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3318009fc0 tx=0x7f33180044d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7168b14f5b2bc10 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:51.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.126+0000 7f32fffff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:53:51.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.126+0000 7f32fffff640 1 -- 172.21.15.33:0/4086046475 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3318011850 con 0x558827b18880 2024-03-19T02:53:51.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.126+0000 7f32fffff640 1 -- 172.21.15.33:0/4086046475 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f33180119f0 con 0x558827b18880 2024-03-19T02:53:51.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.126+0000 7f3329d6d740 1 -- 172.21.15.33:0/4086046475 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33140c1100 con 0x558827b18880 2024-03-19T02:53:51.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.126+0000 7f32fffff640 1 -- 172.21.15.33:0/4086046475 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3318011bf0 con 0x558827b18880 2024-03-19T02:53:51.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.126+0000 7f3329d6d740 10 client.15783 fetch_fsmap learned FSMap version 25 2024-03-19T02:53:51.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.126+0000 7f3329d6d740 1 -- 172.21.15.33:0/4086046475 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558827747170 con 0x558827b18880 2024-03-19T02:53:51.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.126+0000 7f32fffff640 1 -- 172.21.15.33:0/4086046475 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f33140c1100 con 0x558827b18880 2024-03-19T02:53:51.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.127+0000 7f32fffff640 1 -- 172.21.15.33:0/4086046475 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f331801b420 con 0x558827b18880 2024-03-19T02:53:51.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.127+0000 7f3329d6d740 10 client.15783 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:53:51.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.127+0000 7f3329d6d740 10 client.15783 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:53:51.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.127+0000 7f3329d6d740 20 client.15783 populate_metadata read hostname 'smithi033' 2024-03-19T02:53:51.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.127+0000 7f3329d6d740 1 --2- 172.21.15.33:0/4086046475 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x558827b288d0 0x558827b2acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.127+0000 7f3329d6d740 4 client.15783 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:53:51.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.127+0000 7f3329d6d740 1 -- 172.21.15.33:0/4086046475 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558827a806a0 con 0x558827b288d0 2024-03-19T02:53:51.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.127+0000 7f3324f1a640 1 --2- 172.21.15.33:0/4086046475 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x558827b288d0 0x558827b2acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.127+0000 7f3324f1a640 1 --2- 172.21.15.33:0/4086046475 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x558827b288d0 0x558827b2acc0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fec47d29b0dc839b server_cookie=1d16c2fc8e5c33a0 in_seq=0 out_seq=0 2024-03-19T02:53:51.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.127+0000 7f32fffff640 10 client.15783 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:53:51.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.128+0000 7f32fffff640 1 -- 172.21.15.33:0/4086046475 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558827a806a0 con 0x558827b288d0 2024-03-19T02:53:51.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.128+0000 7f32fffff640 10 client.15783 handle_command_reply: tid=0 2024-03-19T02:53:51.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.128+0000 7f3329d6d740 1 client.15783 shutdown 2024-03-19T02:53:51.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.128+0000 7f3329d6d740 1 -- 172.21.15.33:0/4086046475 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558827b18880 msgr2=0x558827b16390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:51.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.128+0000 7f3329d6d740 1 --2- 172.21.15.33:0/4086046475 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558827b18880 0x558827b16390 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3318009fc0 tx=0x7f33180044d0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.128+0000 7f3329d6d740 1 -- 172.21.15.33:0/4086046475 shutdown_connections 2024-03-19T02:53:51.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.128+0000 7f3329d6d740 1 --2- 172.21.15.33:0/4086046475 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x558827b288d0 0x558827b2acc0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.128+0000 7f3329d6d740 1 --2- 172.21.15.33:0/4086046475 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558827b0fca0 0x558827b15e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.128+0000 7f32fffff640 0 client.15783 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:53:51.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.128+0000 7f3329d6d740 1 --2- 172.21.15.33:0/4086046475 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558827b18880 0x558827b16390 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.128+0000 7f3329d6d740 1 --2- 172.21.15.33:0/4086046475 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558827b0f380 0x558827b15910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.128+0000 7f3329d6d740 1 -- 172.21.15.33:0/4086046475 >> 172.21.15.33:0/4086046475 conn(0x558827b09ab0 msgr2=0x558827b0b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:51.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.129+0000 7f3329d6d740 1 -- 172.21.15.33:0/4086046475 shutdown_connections 2024-03-19T02:53:51.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.129+0000 7f3329d6d740 1 -- 172.21.15.33:0/4086046475 wait complete. 2024-03-19T02:53:51.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.129+0000 7f3329d6d740 20 client.15783 trim_cache size 0 max 16384 2024-03-19T02:53:51.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.163+0000 7f3329d6d740 1 Processor -- start 2024-03-19T02:53:51.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.163+0000 7f3329d6d740 1 -- start start 2024-03-19T02:53:51.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.163+0000 7f3329d6d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558827b0f380 0x558827b18040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.163+0000 7f3329d6d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558827b0fca0 0x558827b23f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.163+0000 7f3329d6d740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558827b18880 0x558827b305b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.163+0000 7f3329d6d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f33201b35e0 con 0x558827b0f380 2024-03-19T02:53:51.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.163+0000 7f3329d6d740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f33201b0eb0 con 0x558827b18880 2024-03-19T02:53:51.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.163+0000 7f3329d6d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f33201aece0 con 0x558827b0fca0 2024-03-19T02:53:51.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.163+0000 7f3324f1a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558827b0f380 0x558827b18040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.163+0000 7f3324f1a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558827b0f380 0x558827b18040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:34526/0 (socket says 172.21.15.33:34526) 2024-03-19T02:53:51.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.163+0000 7f3324f1a640 1 -- 172.21.15.33:0/2483714368 learned_addr learned my addr 172.21.15.33:0/2483714368 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:51.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.163+0000 7f332571b640 1 --2- 172.21.15.33:0/2483714368 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558827b18880 0x558827b305b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.163+0000 7f331ffff640 1 --2- 172.21.15.33:0/2483714368 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558827b0fca0 0x558827b23f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.164+0000 7f3324f1a640 1 -- 172.21.15.33:0/2483714368 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558827b0fca0 msgr2=0x558827b23f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:51.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.164+0000 7f3324f1a640 1 --2- 172.21.15.33:0/2483714368 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558827b0fca0 0x558827b23f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.164+0000 7f3324f1a640 1 -- 172.21.15.33:0/2483714368 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558827b18880 msgr2=0x558827b305b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:51.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.164+0000 7f3324f1a640 1 --2- 172.21.15.33:0/2483714368 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558827b18880 0x558827b305b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.164+0000 7f3324f1a640 1 -- 172.21.15.33:0/2483714368 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558827b2b030 con 0x558827b0f380 2024-03-19T02:53:51.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.164+0000 7f331ffff640 1 --2- 172.21.15.33:0/2483714368 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558827b0fca0 0x558827b23f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:53:51.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.164+0000 7f3324f1a640 1 --2- 172.21.15.33:0/2483714368 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558827b0f380 0x558827b18040 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f33140bccb0 tx=0x7f3314005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f501e021c83ffd48 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:51.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.164+0000 7f330a7fc640 1 -- 172.21.15.33:0/2483714368 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33140053d0 con 0x558827b0f380 2024-03-19T02:53:51.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.164+0000 7f330a7fc640 1 -- 172.21.15.33:0/2483714368 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f33140b9620 con 0x558827b0f380 2024-03-19T02:53:51.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.164+0000 7f330a7fc640 1 -- 172.21.15.33:0/2483714368 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33140ee550 con 0x558827b0f380 2024-03-19T02:53:51.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.165+0000 7f3329d6d740 1 -- 172.21.15.33:0/2483714368 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558827b0f380 msgr2=0x558827b18040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:51.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.165+0000 7f3329d6d740 1 --2- 172.21.15.33:0/2483714368 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558827b0f380 0x558827b18040 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f33140bccb0 tx=0x7f3314005c90 comp rx=0 tx=0).stop 2024-03-19T02:53:51.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.165+0000 7f3329d6d740 1 -- 172.21.15.33:0/2483714368 shutdown_connections 2024-03-19T02:53:51.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.165+0000 7f3329d6d740 1 --2- 172.21.15.33:0/2483714368 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558827b0fca0 0x558827b23f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.165+0000 7f3329d6d740 1 --2- 172.21.15.33:0/2483714368 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558827b18880 0x558827b305b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.165+0000 7f3329d6d740 1 --2- 172.21.15.33:0/2483714368 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558827b0f380 0x558827b18040 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.165+0000 7f3329d6d740 1 -- 172.21.15.33:0/2483714368 >> 172.21.15.33:0/2483714368 conn(0x558827b09ab0 msgr2=0x558827b158d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:51.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.165+0000 7f3329d6d740 1 -- 172.21.15.33:0/2483714368 shutdown_connections 2024-03-19T02:53:51.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.165+0000 7f3329d6d740 1 -- 172.21.15.33:0/2483714368 wait complete. 2024-03-19T02:53:51.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.165+0000 7f3329d6d740 1 -- start start 2024-03-19T02:53:51.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.165+0000 7f3329d6d740 1 Processor -- start 2024-03-19T02:53:51.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.166+0000 7f3329d6d740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558827b0f380 0x558827b1c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.166+0000 7f3329d6d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558827b0fca0 0x558827b1c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.166+0000 7f3329d6d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558827b18880 0x558827b1cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.166+0000 7f3329d6d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f33201af370 con 0x558827b0fca0 2024-03-19T02:53:51.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.166+0000 7f3329d6d740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f33201b07e0 con 0x558827b0f380 2024-03-19T02:53:51.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.166+0000 7f3329d6d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f33201afa40 con 0x558827b18880 2024-03-19T02:53:51.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.166+0000 7f3329d6d740 10 client.0 _ll_register_callbacks cb 0x7f3318009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:53:51.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.166+0000 7f3324f1a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558827b0f380 0x558827b1c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.166+0000 7f331ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558827b0fca0 0x558827b1c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.166+0000 7f3324f1a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558827b0f380 0x558827b1c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:58260/0 (socket says 172.21.15.33:58260) 2024-03-19T02:53:51.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.166+0000 7f3324f1a640 1 -- 172.21.15.33:0/2064622521 learned_addr learned my addr 172.21.15.33:0/2064622521 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:51.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.166+0000 7f332571b640 1 --2- 172.21.15.33:0/2064622521 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558827b18880 0x558827b1cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.166+0000 7f3324f1a640 1 -- 172.21.15.33:0/2064622521 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558827b18880 msgr2=0x558827b1cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:51.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.166+0000 7f3324f1a640 1 --2- 172.21.15.33:0/2064622521 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558827b18880 0x558827b1cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.166+0000 7f3324f1a640 1 -- 172.21.15.33:0/2064622521 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558827b0fca0 msgr2=0x558827b1c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:51.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.166+0000 7f3324f1a640 1 --2- 172.21.15.33:0/2064622521 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558827b0fca0 0x558827b1c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.166+0000 7f3324f1a640 1 -- 172.21.15.33:0/2064622521 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558827b8ddb0 con 0x558827b0f380 2024-03-19T02:53:51.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.166+0000 7f331ffff640 1 --2- 172.21.15.33:0/2064622521 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558827b0fca0 0x558827b1c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:51.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.166+0000 7f3324f1a640 1 --2- 172.21.15.33:0/2064622521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558827b0f380 0x558827b1c050 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3314005ec0 tx=0x7f33140c5350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=caeb49bb0ec161b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:51.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.166+0000 7f32fffff640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T02:53:51.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.167+0000 7f32fffff640 1 -- 172.21.15.33:0/2064622521 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33140c5520 con 0x558827b0f380 2024-03-19T02:53:51.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.167+0000 7f32fffff640 1 -- 172.21.15.33:0/2064622521 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f33140ee2f0 con 0x558827b0f380 2024-03-19T02:53:51.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.167+0000 7f3329d6d740 1 -- 172.21.15.33:0/2064622521 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33140b9620 con 0x558827b0f380 2024-03-19T02:53:51.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.167+0000 7f32fffff640 1 -- 172.21.15.33:0/2064622521 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f33140c1100 con 0x558827b0f380 2024-03-19T02:53:51.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.167+0000 7f3329d6d740 10 client.25661 fetch_fsmap learned FSMap version 25 2024-03-19T02:53:51.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.167+0000 7f3329d6d740 1 -- 172.21.15.33:0/2064622521 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558827b8f0b0 con 0x558827b0f380 2024-03-19T02:53:51.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.167+0000 7f32fffff640 1 -- 172.21.15.33:0/2064622521 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f33140ee4f0 con 0x558827b0f380 2024-03-19T02:53:51.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.167+0000 7f32fffff640 1 -- 172.21.15.33:0/2064622521 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f33140b7bb0 con 0x558827b0f380 2024-03-19T02:53:51.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.167+0000 7f3329d6d740 10 client.25661 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:53:51.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.167+0000 7f3329d6d740 10 client.25661 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:53:51.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.167+0000 7f3329d6d740 20 client.25661 populate_metadata read hostname 'smithi033' 2024-03-19T02:53:51.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.167+0000 7f3329d6d740 1 --2- 172.21.15.33:0/2064622521 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x558827b288d0 0x558827b915f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.167+0000 7f3329d6d740 4 client.25661 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:53:51.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.167+0000 7f3329d6d740 1 -- 172.21.15.33:0/2064622521 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f33140b9620 con 0x558827b288d0 2024-03-19T02:53:51.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.167+0000 7f331ffff640 1 --2- 172.21.15.33:0/2064622521 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x558827b288d0 0x558827b915f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.168+0000 7f32fffff640 10 client.25661 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:53:51.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.168+0000 7f331ffff640 1 --2- 172.21.15.33:0/2064622521 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x558827b288d0 0x558827b915f0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a2f0e00e5c54e45 server_cookie=bc7eeaeceee9fe0c in_seq=0 out_seq=0 2024-03-19T02:53:51.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.169+0000 7f32fffff640 1 -- 172.21.15.33:0/2064622521 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f33140b9620 con 0x558827b288d0 2024-03-19T02:53:51.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.169+0000 7f32fffff640 10 client.25661 handle_command_reply: tid=0 2024-03-19T02:53:51.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.169+0000 7f3329d6d740 1 client.25661 shutdown 2024-03-19T02:53:51.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.170+0000 7f3329d6d740 1 -- 172.21.15.33:0/2064622521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558827b0f380 msgr2=0x558827b1c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:51.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.170+0000 7f3329d6d740 1 --2- 172.21.15.33:0/2064622521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558827b0f380 0x558827b1c050 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3314005ec0 tx=0x7f33140c5350 comp rx=0 tx=0).stop 2024-03-19T02:53:51.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.170+0000 7f3329d6d740 1 -- 172.21.15.33:0/2064622521 shutdown_connections 2024-03-19T02:53:51.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.170+0000 7f3329d6d740 1 --2- 172.21.15.33:0/2064622521 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x558827b288d0 0x558827b915f0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.170+0000 7f3329d6d740 1 --2- 172.21.15.33:0/2064622521 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558827b18880 0x558827b1cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.170+0000 7f32fffff640 0 client.25661 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:53:51.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.170+0000 7f3329d6d740 1 --2- 172.21.15.33:0/2064622521 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558827b0fca0 0x558827b1c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.170+0000 7f3329d6d740 1 --2- 172.21.15.33:0/2064622521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558827b0f380 0x558827b1c050 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.170+0000 7f3329d6d740 1 -- 172.21.15.33:0/2064622521 >> 172.21.15.33:0/2064622521 conn(0x558827b09ab0 msgr2=0x558827b160f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:51.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.170+0000 7f3329d6d740 1 -- 172.21.15.33:0/2064622521 shutdown_connections 2024-03-19T02:53:51.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.170+0000 7f3329d6d740 1 -- 172.21.15.33:0/2064622521 wait complete. 2024-03-19T02:53:51.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.170+0000 7f3329d6d740 20 client.25661 trim_cache size 0 max 16384 2024-03-19T02:53:51.175 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:53:51.175 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-03-19T02:53:51.175 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "454919b3-b470-4e13-ab5b-9f2d6e1469d9", 2024-03-19T02:53:51.175 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-03-19T02:53:51.175 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:53:51.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.171+0000 7f332717e640 1 -- 172.21.15.33:0/4182738377 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f33100a2b00 msgr2=0x7f33100a4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:51.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.171+0000 7f332717e640 1 --2- 172.21.15.33:0/4182738377 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f33100a2b00 0x7f33100a4fc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3320069530 tx=0x7f330c005f90 comp rx=0 tx=0).stop 2024-03-19T02:53:51.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.171+0000 7f332717e640 1 -- 172.21.15.33:0/4182738377 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3320100690 msgr2=0x7f3320198da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:51.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.171+0000 7f332717e640 1 --2- 172.21.15.33:0/4182738377 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3320100690 0x7f3320198da0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3314006010 tx=0x7f3314005d90 comp rx=0 tx=0).stop 2024-03-19T02:53:51.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.171+0000 7f332717e640 1 -- 172.21.15.33:0/4182738377 shutdown_connections 2024-03-19T02:53:51.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.171+0000 7f332717e640 1 --2- 172.21.15.33:0/4182738377 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f33100a2b00 0x7f33100a4fc0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.171+0000 7f332717e640 1 --2- 172.21.15.33:0/4182738377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f33201066a0 0x7f33201a3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.171+0000 7f332717e640 1 --2- 172.21.15.33:0/4182738377 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3320101050 0x7f33201992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.171+0000 7f332717e640 1 --2- 172.21.15.33:0/4182738377 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3320100690 0x7f3320198da0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.171+0000 7f332717e640 1 -- 172.21.15.33:0/4182738377 >> 172.21.15.33:0/4182738377 conn(0x7f3320077820 msgr2=0x7f3320102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:51.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.171+0000 7f332717e640 1 -- 172.21.15.33:0/4182738377 shutdown_connections 2024-03-19T02:53:51.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.171+0000 7f332717e640 1 -- 172.21.15.33:0/4182738377 wait complete. 2024-03-19T02:53:51.185 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:53:51.186 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:53:51.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.336+0000 7f7f7843d640 1 -- 172.21.15.33:0/4091479707 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f70068a20 msgr2=0x7f7f70068ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:51.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.336+0000 7f7f7843d640 1 --2- 172.21.15.33:0/4091479707 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f70068a20 0x7f7f70068ea0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7f60009960 tx=0x7f7f6002bc20 comp rx=0 tx=0).stop 2024-03-19T02:53:51.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.336+0000 7f7f7843d640 1 -- 172.21.15.33:0/4091479707 shutdown_connections 2024-03-19T02:53:51.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.336+0000 7f7f7843d640 1 --2- 172.21.15.33:0/4091479707 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f700693e0 0x7f7f7010e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.336+0000 7f7f7843d640 1 --2- 172.21.15.33:0/4091479707 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f70068a20 0x7f7f70068ea0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.336+0000 7f7f7843d640 1 --2- 172.21.15.33:0/4091479707 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f70107870 0x7f7f70107c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.336+0000 7f7f7843d640 1 -- 172.21.15.33:0/4091479707 >> 172.21.15.33:0/4091479707 conn(0x7f7f70077820 msgr2=0x7f7f70077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:51.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.336+0000 7f7f7843d640 1 -- 172.21.15.33:0/4091479707 shutdown_connections 2024-03-19T02:53:51.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.336+0000 7f7f7843d640 1 -- 172.21.15.33:0/4091479707 wait complete. 2024-03-19T02:53:51.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.337+0000 7f7f7843d640 1 Processor -- start 2024-03-19T02:53:51.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.337+0000 7f7f7843d640 1 -- start start 2024-03-19T02:53:51.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.337+0000 7f7f7843d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f70068a20 0x7f7f70198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.337+0000 7f7f7843d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f700693e0 0x7f7f70198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.337+0000 7f7f7843d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f70107870 0x7f7f7019c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.337+0000 7f7f7843d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7f70111a70 con 0x7f7f700693e0 2024-03-19T02:53:51.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.337+0000 7f7f7843d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7f701118f0 con 0x7f7f70107870 2024-03-19T02:53:51.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.337+0000 7f7f7843d640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7f70111bf0 con 0x7f7f70068a20 2024-03-19T02:53:51.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.337+0000 7f7f761d9640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f70068a20 0x7f7f70198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.337+0000 7f7f759d8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f700693e0 0x7f7f70198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.337+0000 7f7f761d9640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f70068a20 0x7f7f70198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:35670/0 (socket says 172.21.15.33:35670) 2024-03-19T02:53:51.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.337+0000 7f7f761d9640 1 -- 172.21.15.33:0/418568790 learned_addr learned my addr 172.21.15.33:0/418568790 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:51.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.338+0000 7f7f769da640 1 --2- 172.21.15.33:0/418568790 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f70107870 0x7f7f7019c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.338+0000 7f7f761d9640 1 -- 172.21.15.33:0/418568790 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f70107870 msgr2=0x7f7f7019c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:51.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.338+0000 7f7f761d9640 1 --2- 172.21.15.33:0/418568790 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f70107870 0x7f7f7019c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.338+0000 7f7f761d9640 1 -- 172.21.15.33:0/418568790 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f700693e0 msgr2=0x7f7f70198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:51.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.338+0000 7f7f761d9640 1 --2- 172.21.15.33:0/418568790 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f700693e0 0x7f7f70198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.338+0000 7f7f761d9640 1 -- 172.21.15.33:0/418568790 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f7019d010 con 0x7f7f70068a20 2024-03-19T02:53:51.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.338+0000 7f7f759d8640 1 --2- 172.21.15.33:0/418568790 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f700693e0 0x7f7f70198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:51.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.338+0000 7f7f761d9640 1 --2- 172.21.15.33:0/418568790 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f70068a20 0x7f7f70198020 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c00e9f0 tx=0x7f7f6c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:51.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.338+0000 7f7f677fe640 1 -- 172.21.15.33:0/418568790 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f6c00cd80 con 0x7f7f70068a20 2024-03-19T02:53:51.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.338+0000 7f7f677fe640 1 -- 172.21.15.33:0/418568790 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7f6c0054d0 con 0x7f7f70068a20 2024-03-19T02:53:51.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.338+0000 7f7f677fe640 1 -- 172.21.15.33:0/418568790 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f6c010490 con 0x7f7f70068a20 2024-03-19T02:53:51.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.338+0000 7f7f7843d640 1 -- 172.21.15.33:0/418568790 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f7019d300 con 0x7f7f70068a20 2024-03-19T02:53:51.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.338+0000 7f7f7843d640 1 -- 172.21.15.33:0/418568790 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f70100610 con 0x7f7f70068a20 2024-03-19T02:53:51.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.342+0000 7f7f7b022740 1 Processor -- start 2024-03-19T02:53:51.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.342+0000 7f7f7b022740 1 -- start start 2024-03-19T02:53:51.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.342+0000 7f7f677fe640 1 -- 172.21.15.33:0/418568790 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 23) ==== 137466+0+0 (secure 0 0 0) 0x7f7f6c005000 con 0x7f7f70068a20 2024-03-19T02:53:51.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.342+0000 7f7f7b022740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4dc831a30 0x55a4dc831e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.342+0000 7f7f7b022740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a4dc832350 0x55a4dc83a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.342+0000 7f7f7b022740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4dc83af30 0x55a4dc83d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.342+0000 7f7f7b022740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a4dc795500 con 0x55a4dc83af30 2024-03-19T02:53:51.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.342+0000 7f7f7b022740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55a4dc73dbe0 con 0x55a4dc831a30 2024-03-19T02:53:51.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.342+0000 7f7f7b022740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a4dc7915a0 con 0x55a4dc832350 2024-03-19T02:53:51.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.342+0000 7f7f769da640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4dc831a30 0x55a4dc831e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.343+0000 7f7f769da640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4dc831a30 0x55a4dc831e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:58288/0 (socket says 172.21.15.33:58288) 2024-03-19T02:53:51.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.343+0000 7f7f769da640 1 -- 172.21.15.33:0/4103625915 learned_addr learned my addr 172.21.15.33:0/4103625915 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:51.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.343+0000 7f7f759d8640 1 --2- 172.21.15.33:0/4103625915 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4dc83af30 0x55a4dc83d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.343+0000 7f7f761d9640 1 --2- 172.21.15.33:0/4103625915 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a4dc832350 0x55a4dc83a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.343+0000 7f7f769da640 1 -- 172.21.15.33:0/4103625915 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a4dc832350 msgr2=0x55a4dc83a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:51.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.343+0000 7f7f769da640 1 --2- 172.21.15.33:0/4103625915 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a4dc832350 0x55a4dc83a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.343+0000 7f7f769da640 1 -- 172.21.15.33:0/4103625915 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4dc83af30 msgr2=0x55a4dc83d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:51.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.343+0000 7f7f769da640 1 --2- 172.21.15.33:0/4103625915 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4dc83af30 0x55a4dc83d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.343+0000 7f7f769da640 1 -- 172.21.15.33:0/4103625915 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4dc75f000 con 0x55a4dc831a30 2024-03-19T02:53:51.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.343+0000 7f7f761d9640 1 --2- 172.21.15.33:0/4103625915 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a4dc832350 0x55a4dc83a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:53:51.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.343+0000 7f7f769da640 1 --2- 172.21.15.33:0/4103625915 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4dc831a30 0x55a4dc831e10 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7f68009fd0 tx=0x7f7f680118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48fca6d462684864 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:51.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.343+0000 7f7f677fe640 1 --2- 172.21.15.33:0/418568790 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f7f5c0a2a00 0x7f7f5c0a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.343+0000 7f7f64bfe640 1 -- 172.21.15.33:0/4103625915 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f68004d20 con 0x55a4dc831a30 2024-03-19T02:53:51.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.343+0000 7f7f64bfe640 1 -- 172.21.15.33:0/4103625915 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7f68004050 con 0x55a4dc831a30 2024-03-19T02:53:51.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.343+0000 7f7f64bfe640 1 -- 172.21.15.33:0/4103625915 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f68005280 con 0x55a4dc831a30 2024-03-19T02:53:51.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.343+0000 7f7f677fe640 1 -- 172.21.15.33:0/418568790 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f7f6c0cf740 con 0x7f7f70068a20 2024-03-19T02:53:51.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.344+0000 7f7f761d9640 1 --2- 172.21.15.33:0/418568790 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f7f5c0a2a00 0x7f7f5c0a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.344+0000 7f7f7b022740 1 -- 172.21.15.33:0/4103625915 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4dc831a30 msgr2=0x55a4dc831e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:51.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.344+0000 7f7f761d9640 1 --2- 172.21.15.33:0/418568790 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f7f5c0a2a00 0x7f7f5c0a4ec0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c0ccac0 tx=0x7f7f6c004990 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:51.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.344+0000 7f7f7b022740 1 --2- 172.21.15.33:0/4103625915 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4dc831a30 0x55a4dc831e10 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7f68009fd0 tx=0x7f7f680118d0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.344+0000 7f7f7b022740 1 -- 172.21.15.33:0/4103625915 shutdown_connections 2024-03-19T02:53:51.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.344+0000 7f7f7b022740 1 --2- 172.21.15.33:0/4103625915 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a4dc832350 0x55a4dc83a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.344+0000 7f7f7b022740 1 --2- 172.21.15.33:0/4103625915 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4dc83af30 0x55a4dc83d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.344+0000 7f7f7b022740 1 --2- 172.21.15.33:0/4103625915 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4dc831a30 0x55a4dc831e10 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.344+0000 7f7f7b022740 1 -- 172.21.15.33:0/4103625915 >> 172.21.15.33:0/4103625915 conn(0x55a4dc82c160 msgr2=0x55a4dc82c540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:51.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.344+0000 7f7f7b022740 1 -- 172.21.15.33:0/4103625915 shutdown_connections 2024-03-19T02:53:51.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.345+0000 7f7f7b022740 1 -- 172.21.15.33:0/4103625915 wait complete. 2024-03-19T02:53:51.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.345+0000 7f7f7b022740 1 -- start start 2024-03-19T02:53:51.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.345+0000 7f7f7b022740 1 Processor -- start 2024-03-19T02:53:51.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.345+0000 7f7f7b022740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4dc831a30 0x55a4dc837eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.345+0000 7f7f7b022740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a4dc832350 0x55a4dc8383f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.345+0000 7f7f7b022740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4dc83af30 0x55a4dc838930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.345+0000 7f7f7b022740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a4dc83dd20 con 0x55a4dc83af30 2024-03-19T02:53:51.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.345+0000 7f7f7b022740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55a4dc751160 con 0x55a4dc831a30 2024-03-19T02:53:51.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.345+0000 7f7f7b022740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a4dc83dea0 con 0x55a4dc832350 2024-03-19T02:53:51.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.345+0000 7f7f7b022740 10 client.0 _ll_register_callbacks cb 0x55a4dc78da40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:53:51.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.345+0000 7f7f769da640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4dc831a30 0x55a4dc837eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.346+0000 7f7f769da640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4dc831a30 0x55a4dc837eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:58296/0 (socket says 172.21.15.33:58296) 2024-03-19T02:53:51.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.346+0000 7f7f769da640 1 -- 172.21.15.33:0/3016456612 learned_addr learned my addr 172.21.15.33:0/3016456612 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:51.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.346+0000 7f7f759d8640 1 --2- 172.21.15.33:0/3016456612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a4dc832350 0x55a4dc8383f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.346+0000 7f7f769da640 1 --2- 172.21.15.33:0/3016456612 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4dc83af30 0x55a4dc838930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.346+0000 7f7f769da640 1 -- 172.21.15.33:0/3016456612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a4dc832350 msgr2=0x55a4dc8383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:51.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.346+0000 7f7f769da640 1 --2- 172.21.15.33:0/3016456612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a4dc832350 0x55a4dc8383f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.346+0000 7f7f769da640 1 -- 172.21.15.33:0/3016456612 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4dc83af30 msgr2=0x55a4dc838930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:51.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.346+0000 7f7f769da640 1 --2- 172.21.15.33:0/3016456612 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4dc83af30 0x55a4dc838930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.346+0000 7f7f769da640 1 -- 172.21.15.33:0/3016456612 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4dc5737d0 con 0x55a4dc831a30 2024-03-19T02:53:51.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.346+0000 7f7f769da640 1 --2- 172.21.15.33:0/3016456612 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4dc831a30 0x55a4dc837eb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7f680094d0 tx=0x7f7f68003f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19ff05a9039d7a2b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:51.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.346+0000 7f7f59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T02:53:51.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.346+0000 7f7f59ffb640 1 -- 172.21.15.33:0/3016456612 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f6802c020 con 0x55a4dc831a30 2024-03-19T02:53:51.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.346+0000 7f7f59ffb640 1 -- 172.21.15.33:0/3016456612 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f7f68004640 con 0x55a4dc831a30 2024-03-19T02:53:51.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.346+0000 7f7f7b022740 1 -- 172.21.15.33:0/3016456612 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f68004050 con 0x55a4dc831a30 2024-03-19T02:53:51.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.346+0000 7f7f59ffb640 1 -- 172.21.15.33:0/3016456612 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7f68037050 con 0x55a4dc831a30 2024-03-19T02:53:51.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.346+0000 7f7f7b022740 10 client.25673 fetch_fsmap learned FSMap version 25 2024-03-19T02:53:51.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.346+0000 7f7f7b022740 1 -- 172.21.15.33:0/3016456612 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a4dc4a93f0 con 0x55a4dc831a30 2024-03-19T02:53:51.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.347+0000 7f7f59ffb640 1 -- 172.21.15.33:0/3016456612 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7f68004050 con 0x55a4dc831a30 2024-03-19T02:53:51.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.347+0000 7f7f59ffb640 1 -- 172.21.15.33:0/3016456612 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f7f68028450 con 0x55a4dc831a30 2024-03-19T02:53:51.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.347+0000 7f7f7b022740 10 client.25673 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:53:51.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.347+0000 7f7f7b022740 10 client.25673 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:53:51.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.347+0000 7f7f7b022740 20 client.25673 populate_metadata read hostname 'smithi033' 2024-03-19T02:53:51.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.347+0000 7f7f7b022740 1 --2- 172.21.15.33:0/3016456612 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55a4dc84b100 0x55a4dc84d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.347+0000 7f7f7b022740 4 client.25673 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:53:51.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.347+0000 7f7f7b022740 1 -- 172.21.15.33:0/3016456612 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a4dc7a2a40 con 0x55a4dc84b100 2024-03-19T02:53:51.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.347+0000 7f7f759d8640 1 --2- 172.21.15.33:0/3016456612 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55a4dc84b100 0x55a4dc84d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.348+0000 7f7f59ffb640 10 client.25673 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:53:51.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.348+0000 7f7f759d8640 1 --2- 172.21.15.33:0/3016456612 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55a4dc84b100 0x55a4dc84d4f0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1b4d1e2e5af1b7 server_cookie=2e23be7ba0cef4d3 in_seq=0 out_seq=0 2024-03-19T02:53:51.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.348+0000 7f7f59ffb640 1 -- 172.21.15.33:0/3016456612 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a4dc7a2a40 con 0x55a4dc84b100 2024-03-19T02:53:51.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.348+0000 7f7f59ffb640 10 client.25673 handle_command_reply: tid=0 2024-03-19T02:53:51.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.348+0000 7f7f7b022740 1 client.25673 shutdown 2024-03-19T02:53:51.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.348+0000 7f7f7b022740 1 -- 172.21.15.33:0/3016456612 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4dc831a30 msgr2=0x55a4dc837eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:51.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.348+0000 7f7f7b022740 1 --2- 172.21.15.33:0/3016456612 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4dc831a30 0x55a4dc837eb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7f680094d0 tx=0x7f7f68003f50 comp rx=0 tx=0).stop 2024-03-19T02:53:51.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.348+0000 7f7f7b022740 1 -- 172.21.15.33:0/3016456612 shutdown_connections 2024-03-19T02:53:51.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.348+0000 7f7f7b022740 1 --2- 172.21.15.33:0/3016456612 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55a4dc84b100 0x55a4dc84d4f0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.348+0000 7f7f59ffb640 0 client.25673 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:53:51.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.348+0000 7f7f7b022740 1 --2- 172.21.15.33:0/3016456612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a4dc832350 0x55a4dc8383f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.349+0000 7f7f7b022740 1 --2- 172.21.15.33:0/3016456612 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4dc83af30 0x55a4dc838930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.349+0000 7f7f7b022740 1 --2- 172.21.15.33:0/3016456612 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4dc831a30 0x55a4dc837eb0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.349+0000 7f7f7b022740 1 -- 172.21.15.33:0/3016456612 >> 172.21.15.33:0/3016456612 conn(0x55a4dc82c160 msgr2=0x55a4dc82deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:51.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.349+0000 7f7f7b022740 1 -- 172.21.15.33:0/3016456612 shutdown_connections 2024-03-19T02:53:51.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.349+0000 7f7f7b022740 1 -- 172.21.15.33:0/3016456612 wait complete. 2024-03-19T02:53:51.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.349+0000 7f7f7b022740 20 client.25673 trim_cache size 0 max 16384 2024-03-19T02:53:51.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.382+0000 7f7f7b022740 1 Processor -- start 2024-03-19T02:53:51.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.383+0000 7f7f7b022740 1 -- start start 2024-03-19T02:53:51.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.383+0000 7f7f7b022740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4dc831a30 0x55a4dc837e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.383+0000 7f7f7b022740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4dc832350 0x55a4dc846ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.383+0000 7f7f7b022740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a4dc83af30 0x55a4dc837660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.383+0000 7f7f7b022740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7f701af340 con 0x55a4dc831a30 2024-03-19T02:53:51.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.383+0000 7f7f7b022740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7f701acc10 con 0x55a4dc832350 2024-03-19T02:53:51.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.383+0000 7f7f7b022740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7f701aaa40 con 0x55a4dc83af30 2024-03-19T02:53:51.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.383+0000 7f7f759d8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4dc832350 0x55a4dc846ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.383+0000 7f7f769da640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4dc831a30 0x55a4dc837e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.383+0000 7f7f759d8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4dc832350 0x55a4dc846ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:58310/0 (socket says 172.21.15.33:58310) 2024-03-19T02:53:51.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.383+0000 7f7f759d8640 1 -- 172.21.15.33:0/3621489992 learned_addr learned my addr 172.21.15.33:0/3621489992 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:51.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.383+0000 7f7f769da640 1 --2- 172.21.15.33:0/3621489992 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a4dc83af30 0x55a4dc837660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.383+0000 7f7f759d8640 1 -- 172.21.15.33:0/3621489992 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a4dc83af30 msgr2=0x55a4dc837660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:51.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.383+0000 7f7f759d8640 1 --2- 172.21.15.33:0/3621489992 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a4dc83af30 0x55a4dc837660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.383+0000 7f7f759d8640 1 -- 172.21.15.33:0/3621489992 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4dc831a30 msgr2=0x55a4dc837e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:51.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.383+0000 7f7f759d8640 1 --2- 172.21.15.33:0/3621489992 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4dc831a30 0x55a4dc837e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.383+0000 7f7f759d8640 1 -- 172.21.15.33:0/3621489992 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4dc832180 con 0x55a4dc832350 2024-03-19T02:53:51.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.383+0000 7f7f769da640 1 --2- 172.21.15.33:0/3621489992 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a4dc83af30 0x55a4dc837660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:53:51.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.384+0000 7f7f759d8640 1 --2- 172.21.15.33:0/3621489992 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4dc832350 0x55a4dc846ac0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7f6002c160 tx=0x7f7f6003f040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1425c9f5a94557c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:51.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.384+0000 7f7f5bfff640 1 -- 172.21.15.33:0/3621489992 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f60004200 con 0x55a4dc832350 2024-03-19T02:53:51.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.384+0000 7f7f5bfff640 1 -- 172.21.15.33:0/3621489992 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7f6002cc80 con 0x55a4dc832350 2024-03-19T02:53:51.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.384+0000 7f7f5bfff640 1 -- 172.21.15.33:0/3621489992 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f600035f0 con 0x55a4dc832350 2024-03-19T02:53:51.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.385+0000 7f7f7b022740 1 -- 172.21.15.33:0/3621489992 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4dc832350 msgr2=0x55a4dc846ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:51.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.385+0000 7f7f7b022740 1 --2- 172.21.15.33:0/3621489992 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4dc832350 0x55a4dc846ac0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7f6002c160 tx=0x7f7f6003f040 comp rx=0 tx=0).stop 2024-03-19T02:53:51.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.385+0000 7f7f7b022740 1 -- 172.21.15.33:0/3621489992 shutdown_connections 2024-03-19T02:53:51.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.385+0000 7f7f7b022740 1 --2- 172.21.15.33:0/3621489992 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a4dc83af30 0x55a4dc837660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.385+0000 7f7f7b022740 1 --2- 172.21.15.33:0/3621489992 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4dc832350 0x55a4dc846ac0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.385+0000 7f7f7b022740 1 --2- 172.21.15.33:0/3621489992 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4dc831a30 0x55a4dc837e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.385+0000 7f7f7b022740 1 -- 172.21.15.33:0/3621489992 >> 172.21.15.33:0/3621489992 conn(0x55a4dc82c160 msgr2=0x55a4dc82dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:51.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.385+0000 7f7f7b022740 1 -- 172.21.15.33:0/3621489992 shutdown_connections 2024-03-19T02:53:51.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.385+0000 7f7f7b022740 1 -- 172.21.15.33:0/3621489992 wait complete. 2024-03-19T02:53:51.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.385+0000 7f7f7b022740 1 -- start start 2024-03-19T02:53:51.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.385+0000 7f7f7b022740 1 Processor -- start 2024-03-19T02:53:51.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.386+0000 7f7f7b022740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a4dc831a30 0x55a4dc83ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.386+0000 7f7f7b022740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4dc832350 0x55a4dc83ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.386+0000 7f7f7b022740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4dc83af30 0x55a4dc83f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.386+0000 7f7f7b022740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7f701ab0d0 con 0x55a4dc832350 2024-03-19T02:53:51.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.386+0000 7f7f769da640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a4dc831a30 0x55a4dc83ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.386+0000 7f7f7b022740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7f701ac540 con 0x55a4dc83af30 2024-03-19T02:53:51.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.386+0000 7f7f7b022740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7f701ab7a0 con 0x55a4dc831a30 2024-03-19T02:53:51.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.386+0000 7f7f769da640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a4dc831a30 0x55a4dc83ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:35718/0 (socket says 172.21.15.33:35718) 2024-03-19T02:53:51.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.386+0000 7f7f759d8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4dc832350 0x55a4dc83ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.386+0000 7f7f769da640 1 -- 172.21.15.33:0/4122287623 learned_addr learned my addr 172.21.15.33:0/4122287623 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:53:51.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.386+0000 7f7f769da640 1 --2- 172.21.15.33:0/4122287623 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4dc83af30 0x55a4dc83f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.386+0000 7f7f7b022740 10 client.0 _ll_register_callbacks cb 0x7f7f6c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:53:51.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.386+0000 7f7f769da640 1 -- 172.21.15.33:0/4122287623 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a4dc831a30 msgr2=0x55a4dc83ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:51.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.386+0000 7f7f769da640 1 --2- 172.21.15.33:0/4122287623 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a4dc831a30 0x55a4dc83ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.386+0000 7f7f769da640 1 -- 172.21.15.33:0/4122287623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4dc832350 msgr2=0x55a4dc83ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:53:51.393 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.386+0000 7f7f769da640 1 --2- 172.21.15.33:0/4122287623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4dc832350 0x55a4dc83ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.393 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.386+0000 7f7f769da640 1 -- 172.21.15.33:0/4122287623 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4dc4690f0 con 0x55a4dc83af30 2024-03-19T02:53:51.393 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.386+0000 7f7f759d8640 1 --2- 172.21.15.33:0/4122287623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4dc832350 0x55a4dc83ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:53:51.393 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.387+0000 7f7f769da640 1 --2- 172.21.15.33:0/4122287623 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4dc83af30 0x55a4dc83f4d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7f68003b60 tx=0x7f7f68003f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=659b557e5ba7e0e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:53:51.393 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.387+0000 7f7f59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T02:53:51.393 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.387+0000 7f7f59ffb640 1 -- 172.21.15.33:0/4122287623 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f6802c040 con 0x55a4dc83af30 2024-03-19T02:53:51.393 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.387+0000 7f7f59ffb640 1 -- 172.21.15.33:0/4122287623 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f7f680049a0 con 0x55a4dc83af30 2024-03-19T02:53:51.393 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.387+0000 7f7f7b022740 1 -- 172.21.15.33:0/4122287623 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f6002cc80 con 0x55a4dc83af30 2024-03-19T02:53:51.393 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.387+0000 7f7f59ffb640 1 -- 172.21.15.33:0/4122287623 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7f68004780 con 0x55a4dc83af30 2024-03-19T02:53:51.393 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.387+0000 7f7f7b022740 10 client.25679 fetch_fsmap learned FSMap version 25 2024-03-19T02:53:51.393 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.387+0000 7f7f7b022740 1 -- 172.21.15.33:0/4122287623 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a4dc8b1370 con 0x55a4dc83af30 2024-03-19T02:53:51.393 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.387+0000 7f7f59ffb640 1 -- 172.21.15.33:0/4122287623 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7f6002cc80 con 0x55a4dc83af30 2024-03-19T02:53:51.393 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.387+0000 7f7f59ffb640 1 -- 172.21.15.33:0/4122287623 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f7f6800f990 con 0x55a4dc83af30 2024-03-19T02:53:51.393 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.387+0000 7f7f7b022740 10 client.25679 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:53:51.393 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.387+0000 7f7f7b022740 10 client.25679 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:53:51.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.388+0000 7f7f7b022740 20 client.25679 populate_metadata read hostname 'smithi033' 2024-03-19T02:53:51.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.388+0000 7f7f7b022740 1 --2- 172.21.15.33:0/4122287623 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55a4dc84b100 0x55a4dc8b3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:53:51.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.388+0000 7f7f7b022740 4 client.25679 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:53:51.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.388+0000 7f7f7b022740 1 -- 172.21.15.33:0/4122287623 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a4dc79cd90 con 0x55a4dc84b100 2024-03-19T02:53:51.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.388+0000 7f7f759d8640 1 --2- 172.21.15.33:0/4122287623 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55a4dc84b100 0x55a4dc8b3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:53:51.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.388+0000 7f7f59ffb640 10 client.25679 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:53:51.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.388+0000 7f7f759d8640 1 --2- 172.21.15.33:0/4122287623 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55a4dc84b100 0x55a4dc8b3aa0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec5378ec07057cc4 server_cookie=5e24a1cbf5639022 in_seq=0 out_seq=0 2024-03-19T02:53:51.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.388+0000 7f7f59ffb640 1 -- 172.21.15.33:0/4122287623 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55a4dc79cd90 con 0x55a4dc84b100 2024-03-19T02:53:51.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.388+0000 7f7f59ffb640 10 client.25679 handle_command_reply: tid=0 2024-03-19T02:53:51.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.388+0000 7f7f7b022740 1 client.25679 shutdown 2024-03-19T02:53:51.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.389+0000 7f7f7b022740 1 -- 172.21.15.33:0/4122287623 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4dc83af30 msgr2=0x55a4dc83f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:51.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.389+0000 7f7f7b022740 1 --2- 172.21.15.33:0/4122287623 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4dc83af30 0x55a4dc83f4d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7f68003b60 tx=0x7f7f68003f50 comp rx=0 tx=0).stop 2024-03-19T02:53:51.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.389+0000 7f7f7b022740 1 -- 172.21.15.33:0/4122287623 shutdown_connections 2024-03-19T02:53:51.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.389+0000 7f7f7b022740 1 --2- 172.21.15.33:0/4122287623 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55a4dc84b100 0x55a4dc8b3aa0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.389+0000 7f7f7b022740 1 --2- 172.21.15.33:0/4122287623 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a4dc83af30 0x55a4dc83f4d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.389+0000 7f7f7b022740 1 --2- 172.21.15.33:0/4122287623 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a4dc832350 0x55a4dc83ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.389+0000 7f7f7b022740 1 --2- 172.21.15.33:0/4122287623 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a4dc831a30 0x55a4dc83ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.389+0000 7f7f7b022740 1 -- 172.21.15.33:0/4122287623 >> 172.21.15.33:0/4122287623 conn(0x55a4dc82c160 msgr2=0x55a4dc82d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:51.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.389+0000 7f7f59ffb640 0 client.25679 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:53:51.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.389+0000 7f7f7b022740 1 -- 172.21.15.33:0/4122287623 shutdown_connections 2024-03-19T02:53:51.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.389+0000 7f7f7b022740 1 -- 172.21.15.33:0/4122287623 wait complete. 2024-03-19T02:53:51.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.389+0000 7f7f7b022740 20 client.25679 trim_cache size 0 max 16384 2024-03-19T02:53:51.395 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:53:51.396 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:53:51.396 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-03-19T02:53:51.396 INFO:teuthology.orchestra.run.smithi033.stdout: "454919b3-b470-4e13-ab5b-9f2d6e1469d9": { 2024-03-19T02:53:51.396 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-03-19T02:53:51.396 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "454919b3-b470-4e13-ab5b-9f2d6e1469d9", 2024-03-19T02:53:51.396 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-03-19T02:53:51.396 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T02:53:51.396 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T02:53:51.396 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:53:51.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.389+0000 7f7f7843d640 1 -- 172.21.15.33:0/418568790 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f7f5c0a2a00 msgr2=0x7f7f5c0a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:51.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.389+0000 7f7f7843d640 1 --2- 172.21.15.33:0/418568790 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f7f5c0a2a00 0x7f7f5c0a4ec0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c0ccac0 tx=0x7f7f6c004990 comp rx=0 tx=0).stop 2024-03-19T02:53:51.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.389+0000 7f7f7843d640 1 -- 172.21.15.33:0/418568790 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f70068a20 msgr2=0x7f7f70198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:53:51.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.389+0000 7f7f7843d640 1 --2- 172.21.15.33:0/418568790 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f70068a20 0x7f7f70198020 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c00e9f0 tx=0x7f7f6c00edf0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.390+0000 7f7f7843d640 1 -- 172.21.15.33:0/418568790 shutdown_connections 2024-03-19T02:53:51.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.390+0000 7f7f7843d640 1 --2- 172.21.15.33:0/418568790 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f7f5c0a2a00 0x7f7f5c0a4ec0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.390+0000 7f7f7843d640 1 --2- 172.21.15.33:0/418568790 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f70107870 0x7f7f7019c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.390+0000 7f7f7843d640 1 --2- 172.21.15.33:0/418568790 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7f700693e0 0x7f7f70198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.390+0000 7f7f7843d640 1 --2- 172.21.15.33:0/418568790 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f70068a20 0x7f7f70198020 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c00e9f0 tx=0x7f7f6c00edf0 comp rx=0 tx=0).stop 2024-03-19T02:53:51.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.390+0000 7f7f7843d640 1 -- 172.21.15.33:0/418568790 >> 172.21.15.33:0/418568790 conn(0x7f7f70077820 msgr2=0x7f7f700fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:53:51.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.390+0000 7f7f7843d640 1 -- 172.21.15.33:0/418568790 shutdown_connections 2024-03-19T02:53:51.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:53:51.390+0000 7f7f7843d640 1 -- 172.21.15.33:0/418568790 wait complete. 2024-03-19T02:53:51.403 INFO:tasks.cephfs.filesystem:scrub status for tag:454919b3-b470-4e13-ab5b-9f2d6e1469d9 - {'path': '/', 'tag': '454919b3-b470-4e13-ab5b-9f2d6e1469d9', 'options': 'recursive,force'} 2024-03-19T02:53:51.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:51 smithi096 ceph-mon[35946]: scrub queued for path: / 2024-03-19T02:53:51.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:51 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T02:53:51.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:51 smithi096 ceph-mon[35946]: scrub summary: active paths [/] 2024-03-19T02:53:51.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:51 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T02:53:52.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:51 smithi155 ceph-mon[35804]: scrub queued for path: / 2024-03-19T02:53:52.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:51 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T02:53:52.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:51 smithi155 ceph-mon[35804]: scrub summary: active paths [/] 2024-03-19T02:53:52.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:51 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T02:53:52.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:51 smithi033 ceph-mon[31973]: scrub queued for path: / 2024-03-19T02:53:52.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:51 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T02:53:52.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:51 smithi033 ceph-mon[31973]: scrub summary: active paths [/] 2024-03-19T02:53:52.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:51 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T02:53:52.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:52 smithi096 ceph-mon[35946]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-19T02:53:52.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:52 smithi096 ceph-mon[35946]: scrub summary: idle 2024-03-19T02:53:53.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:52 smithi155 ceph-mon[35804]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-19T02:53:53.018 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:52 smithi155 ceph-mon[35804]: scrub summary: idle 2024-03-19T02:53:53.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:52 smithi033 ceph-mon[31973]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-19T02:53:53.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:52 smithi033 ceph-mon[31973]: scrub summary: idle 2024-03-19T02:53:54.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:54 smithi096 ceph-mon[35946]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 6 op/s 2024-03-19T02:53:55.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:54 smithi155 ceph-mon[35804]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 6 op/s 2024-03-19T02:53:55.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:54 smithi033 ceph-mon[31973]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 6 op/s 2024-03-19T02:53:56.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:56 smithi096 ceph-mon[35946]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-03-19T02:53:57.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:56 smithi155 ceph-mon[35804]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-03-19T02:53:57.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:56 smithi033 ceph-mon[31973]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-03-19T02:53:58.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:53:58 smithi096 ceph-mon[35946]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:53:59.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:53:58 smithi155 ceph-mon[35804]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:53:59.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:53:58 smithi033 ceph-mon[31973]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:54:00.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:00 smithi033 ceph-mon[31973]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:54:00.779 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:54:00 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:54:00.437+0000 7f1c21e29700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:54:00.779 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:54:00 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:54:00.437+0000 7f1c21e29700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:54:00.779 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:54:00 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:54:00.437+0000 7f1c21e29700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:54:00.779 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:54:00 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:54:00.437+0000 7f1c21e29700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:54:00.779 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 02:54:00 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T02:54:00.437+0000 7f1c21e29700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T02:54:00.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:00 smithi096 ceph-mon[35946]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:54:01.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:00 smithi155 ceph-mon[35804]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:54:02.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:02 smithi096 ceph-mon[35946]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:54:02.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:02 smithi096 ceph-mon[35946]: mgrmap e24: x(active, since 34s), standbys: z, y 2024-03-19T02:54:03.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:02 smithi155 ceph-mon[35804]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:54:03.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:02 smithi155 ceph-mon[35804]: mgrmap e24: x(active, since 34s), standbys: z, y 2024-03-19T02:54:03.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:02 smithi033 ceph-mon[31973]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:54:03.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:02 smithi033 ceph-mon[31973]: mgrmap e24: x(active, since 34s), standbys: z, y 2024-03-19T02:54:04.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:04 smithi096 ceph-mon[35946]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-19T02:54:05.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:04 smithi155 ceph-mon[35804]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-19T02:54:05.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:04 smithi033 ceph-mon[31973]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-19T02:54:06.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:06 smithi096 ceph-mon[35946]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-03-19T02:54:07.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:06 smithi155 ceph-mon[35804]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-03-19T02:54:07.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:06 smithi033 ceph-mon[31973]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-03-19T02:54:08.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:08 smithi096 ceph-mon[35946]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-19T02:54:09.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:08 smithi155 ceph-mon[35804]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-19T02:54:09.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:08 smithi033 ceph-mon[31973]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-19T02:54:10.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:10 smithi096 ceph-mon[35946]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-19T02:54:11.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:10 smithi155 ceph-mon[35804]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-19T02:54:11.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:10 smithi033 ceph-mon[31973]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-19T02:54:12.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:12 smithi096 ceph-mon[35946]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-19T02:54:13.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:12 smithi155 ceph-mon[35804]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-19T02:54:13.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:12 smithi033 ceph-mon[31973]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-19T02:54:14.789 INFO:tasks.workunit.client.0.smithi033.stderr:Note: switching to 'e48b1f6c0608b2615e1890cc46286c3877d6e910'. 2024-03-19T02:54:14.789 INFO:tasks.workunit.client.0.smithi033.stderr: 2024-03-19T02:54:14.789 INFO:tasks.workunit.client.0.smithi033.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-19T02:54:14.789 INFO:tasks.workunit.client.0.smithi033.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-19T02:54:14.789 INFO:tasks.workunit.client.0.smithi033.stderr:state without impacting any branches by switching back to a branch. 2024-03-19T02:54:14.789 INFO:tasks.workunit.client.0.smithi033.stderr: 2024-03-19T02:54:14.789 INFO:tasks.workunit.client.0.smithi033.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-19T02:54:14.789 INFO:tasks.workunit.client.0.smithi033.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-19T02:54:14.790 INFO:tasks.workunit.client.0.smithi033.stderr: 2024-03-19T02:54:14.790 INFO:tasks.workunit.client.0.smithi033.stderr: git switch -c 2024-03-19T02:54:14.790 INFO:tasks.workunit.client.0.smithi033.stderr: 2024-03-19T02:54:14.790 INFO:tasks.workunit.client.0.smithi033.stderr:Or undo this operation with: 2024-03-19T02:54:14.790 INFO:tasks.workunit.client.0.smithi033.stderr: 2024-03-19T02:54:14.790 INFO:tasks.workunit.client.0.smithi033.stderr: git switch - 2024-03-19T02:54:14.790 INFO:tasks.workunit.client.0.smithi033.stderr: 2024-03-19T02:54:14.790 INFO:tasks.workunit.client.0.smithi033.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-19T02:54:14.790 INFO:tasks.workunit.client.0.smithi033.stderr: 2024-03-19T02:54:14.790 INFO:tasks.workunit.client.0.smithi033.stderr:HEAD is now at e48b1f6c060 testfix 2024-03-19T02:54:14.794 DEBUG:teuthology.orchestra.run.smithi033:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-19T02:54:14.817 INFO:tasks.workunit.client.0.smithi033.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-19T02:54:14.819 INFO:tasks.workunit.client.0.smithi033.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-19T02:54:14.819 INFO:tasks.workunit.client.0.smithi033.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-19T02:54:14.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:14 smithi096 ceph-mon[35946]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-03-19T02:54:15.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:14 smithi155 ceph-mon[35804]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-03-19T02:54:15.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:14 smithi033 ceph-mon[31973]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-03-19T02:54:15.533 INFO:tasks.workunit.client.0.smithi033.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-19T02:54:15.608 INFO:tasks.workunit.client.0.smithi033.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-19T02:54:15.647 INFO:tasks.workunit.client.0.smithi033.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-19T02:54:15.649 INFO:tasks.workunit.client.0.smithi033.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-19T02:54:15.649 INFO:tasks.workunit.client.0.smithi033.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-19T02:54:15.687 INFO:tasks.workunit.client.0.smithi033.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-19T02:54:15.690 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-19T02:54:15.690 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-19T02:54:15.747 INFO:tasks.workunit:Running workunits matching suites/iozone.sh on client.0... 2024-03-19T02:54:15.748 INFO:tasks.workunit:Running workunit suites/iozone.sh... 2024-03-19T02:54:15.748 DEBUG:teuthology.orchestra.run.smithi033:workunit test suites/iozone.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e48b1f6c0608b2615e1890cc46286c3877d6e910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iozone.sh 2024-03-19T02:54:15.810 INFO:tasks.workunit.client.0.smithi033.stderr:+ iozone -c -e -s 1024M -r 16K -t 1 -F f1 -i 0 -i 1 2024-03-19T02:54:15.887 INFO:tasks.workunit.client.0.smithi033.stdout: Iozone: Performance Test of File I/O 2024-03-19T02:54:15.887 INFO:tasks.workunit.client.0.smithi033.stdout: Version $Revision: 3.494 $ 2024-03-19T02:54:15.887 INFO:tasks.workunit.client.0.smithi033.stdout: Compiled for 64 bit mode. 2024-03-19T02:54:15.887 INFO:tasks.workunit.client.0.smithi033.stdout: Build: linux-AMD64 2024-03-19T02:54:15.887 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-03-19T02:54:15.887 INFO:tasks.workunit.client.0.smithi033.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-03-19T02:54:15.887 INFO:tasks.workunit.client.0.smithi033.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-03-19T02:54:15.887 INFO:tasks.workunit.client.0.smithi033.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-03-19T02:54:15.887 INFO:tasks.workunit.client.0.smithi033.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-03-19T02:54:15.887 INFO:tasks.workunit.client.0.smithi033.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-03-19T02:54:15.887 INFO:tasks.workunit.client.0.smithi033.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-03-19T02:54:15.887 INFO:tasks.workunit.client.0.smithi033.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-03-19T02:54:15.887 INFO:tasks.workunit.client.0.smithi033.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-03-19T02:54:15.887 INFO:tasks.workunit.client.0.smithi033.stdout: Alexey Skidanov, Sudhir Kumar. 2024-03-19T02:54:15.888 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-03-19T02:54:15.888 INFO:tasks.workunit.client.0.smithi033.stdout: Run began: Tue Mar 19 02:54:15 2024 2024-03-19T02:54:15.888 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-03-19T02:54:15.888 INFO:tasks.workunit.client.0.smithi033.stdout: Include close in write timing 2024-03-19T02:54:15.888 INFO:tasks.workunit.client.0.smithi033.stdout: Include fsync in write timing 2024-03-19T02:54:15.888 INFO:tasks.workunit.client.0.smithi033.stdout: File size set to 1048576 kB 2024-03-19T02:54:15.888 INFO:tasks.workunit.client.0.smithi033.stdout: Record Size 16 kB 2024-03-19T02:54:15.888 INFO:tasks.workunit.client.0.smithi033.stdout: Command line used: iozone -c -e -s 1024M -r 16K -t 1 -F f1 -i 0 -i 1 2024-03-19T02:54:15.888 INFO:tasks.workunit.client.0.smithi033.stdout: Output is in kBytes/sec 2024-03-19T02:54:15.888 INFO:tasks.workunit.client.0.smithi033.stdout: Time Resolution = 0.000001 seconds. 2024-03-19T02:54:15.888 INFO:tasks.workunit.client.0.smithi033.stdout: Processor cache size set to 1024 kBytes. 2024-03-19T02:54:15.888 INFO:tasks.workunit.client.0.smithi033.stdout: Processor cache line size set to 32 bytes. 2024-03-19T02:54:15.888 INFO:tasks.workunit.client.0.smithi033.stdout: File stride size set to 17 * record size. 2024-03-19T02:54:15.888 INFO:tasks.workunit.client.0.smithi033.stdout: Throughput test with 1 process 2024-03-19T02:54:15.888 INFO:tasks.workunit.client.0.smithi033.stdout: Each process writes a 1048576 kByte file in 16 kByte records 2024-03-19T02:54:16.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:16 smithi096 ceph-mon[35946]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-03-19T02:54:17.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:16 smithi155 ceph-mon[35804]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-03-19T02:54:17.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:16 smithi033 ceph-mon[31973]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-03-19T02:54:18.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:18 smithi096 ceph-mon[35946]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-19T02:54:19.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:18 smithi155 ceph-mon[35804]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-19T02:54:19.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:18 smithi033 ceph-mon[31973]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-19T02:54:20.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:20 smithi096 ceph-mon[35946]: pgmap v29: 129 pgs: 129 active+clean; 205 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 17 MiB/s wr, 13 op/s 2024-03-19T02:54:21.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:20 smithi155 ceph-mon[35804]: pgmap v29: 129 pgs: 129 active+clean; 205 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 17 MiB/s wr, 13 op/s 2024-03-19T02:54:21.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:20 smithi033 ceph-mon[31973]: pgmap v29: 129 pgs: 129 active+clean; 205 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 17 MiB/s wr, 13 op/s 2024-03-19T02:54:21.405 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:54:21.405 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:54:21.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.528+0000 7f452ccb2640 1 -- 172.21.15.33:0/719437556 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45281017b0 msgr2=0x7f4528101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:21.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.528+0000 7f452ccb2640 1 --2- 172.21.15.33:0/719437556 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45281017b0 0x7f4528101c30 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f451c009930 tx=0x7f451c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:54:21.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.528+0000 7f452ccb2640 1 -- 172.21.15.33:0/719437556 shutdown_connections 2024-03-19T02:54:21.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.528+0000 7f452ccb2640 1 --2- 172.21.15.33:0/719437556 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4528102170 0x7f452810e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.528+0000 7f452ccb2640 1 --2- 172.21.15.33:0/719437556 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45281017b0 0x7f4528101c30 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.528+0000 7f452ccb2640 1 --2- 172.21.15.33:0/719437556 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45281077c0 0x7f4528107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.528+0000 7f452ccb2640 1 -- 172.21.15.33:0/719437556 >> 172.21.15.33:0/719437556 conn(0x7f45280fd2c0 msgr2=0x7f45280ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:21.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.528+0000 7f452ccb2640 1 -- 172.21.15.33:0/719437556 shutdown_connections 2024-03-19T02:54:21.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.528+0000 7f452ccb2640 1 -- 172.21.15.33:0/719437556 wait complete. 2024-03-19T02:54:21.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.529+0000 7f452ccb2640 1 Processor -- start 2024-03-19T02:54:21.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.529+0000 7f452ccb2640 1 -- start start 2024-03-19T02:54:21.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.529+0000 7f452ccb2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45281017b0 0x7f4528198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:21.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.529+0000 7f452ccb2640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4528102170 0x7f4528198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:21.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.529+0000 7f452ccb2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45281077c0 0x7f452819c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:21.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.529+0000 7f452ccb2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4528111a80 con 0x7f45281077c0 2024-03-19T02:54:21.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.529+0000 7f452ccb2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4528111900 con 0x7f45281017b0 2024-03-19T02:54:21.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.529+0000 7f452ccb2640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4528111c00 con 0x7f4528102170 2024-03-19T02:54:21.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.530+0000 7f4526d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45281077c0 0x7f452819c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:21.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.530+0000 7f452659c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45281017b0 0x7f4528198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:21.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.530+0000 7f4526d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45281077c0 0x7f452819c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37184/0 (socket says 172.21.15.33:37184) 2024-03-19T02:54:21.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.530+0000 7f4526d9d640 1 -- 172.21.15.33:0/1074041103 learned_addr learned my addr 172.21.15.33:0/1074041103 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:21.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.530+0000 7f4525d9b640 1 --2- 172.21.15.33:0/1074041103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4528102170 0x7f4528198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:21.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.530+0000 7f452659c640 1 -- 172.21.15.33:0/1074041103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4528102170 msgr2=0x7f4528198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:21.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.530+0000 7f452659c640 1 --2- 172.21.15.33:0/1074041103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4528102170 0x7f4528198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.530+0000 7f452659c640 1 -- 172.21.15.33:0/1074041103 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45281077c0 msgr2=0x7f452819c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:21.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.530+0000 7f452659c640 1 --2- 172.21.15.33:0/1074041103 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45281077c0 0x7f452819c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.530+0000 7f452659c640 1 -- 172.21.15.33:0/1074041103 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f452819d040 con 0x7f45281017b0 2024-03-19T02:54:21.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.530+0000 7f4526d9d640 1 --2- 172.21.15.33:0/1074041103 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45281077c0 0x7f452819c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:21.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.530+0000 7f452659c640 1 --2- 172.21.15.33:0/1074041103 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45281017b0 0x7f4528198050 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f451800e9f0 tx=0x7f451800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:21.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.530+0000 7f45177fe640 1 -- 172.21.15.33:0/1074041103 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f451800ccc0 con 0x7f45281017b0 2024-03-19T02:54:21.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.530+0000 7f452ccb2640 1 -- 172.21.15.33:0/1074041103 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f452819d330 con 0x7f45281017b0 2024-03-19T02:54:21.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.530+0000 7f45177fe640 1 -- 172.21.15.33:0/1074041103 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4518005020 con 0x7f45281017b0 2024-03-19T02:54:21.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.530+0000 7f45177fe640 1 -- 172.21.15.33:0/1074041103 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4518004a00 con 0x7f45281017b0 2024-03-19T02:54:21.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.530+0000 7f452ccb2640 1 -- 172.21.15.33:0/1074041103 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45281a4bc0 con 0x7f45281017b0 2024-03-19T02:54:21.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.532+0000 7f45177fe640 1 -- 172.21.15.33:0/1074041103 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7f4518004ba0 con 0x7f45281017b0 2024-03-19T02:54:21.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.533+0000 7f452f97d740 1 Processor -- start 2024-03-19T02:54:21.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.533+0000 7f452f97d740 1 -- start start 2024-03-19T02:54:21.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.533+0000 7f45177fe640 1 --2- 172.21.15.33:0/1074041103 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f45100a2a90 0x7f45100a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:21.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.533+0000 7f452f97d740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557403efaa30 0x557403efae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:21.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.533+0000 7f452f97d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557403efb350 0x557403f039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:21.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.533+0000 7f452f97d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557403f03f30 0x557403f06320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:21.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.533+0000 7f452f97d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557403e5e500 con 0x557403efb350 2024-03-19T02:54:21.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.533+0000 7f452f97d740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557403e06be0 con 0x557403efaa30 2024-03-19T02:54:21.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.533+0000 7f452f97d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x557403e5a5a0 con 0x557403f03f30 2024-03-19T02:54:21.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.533+0000 7f45177fe640 1 -- 172.21.15.33:0/1074041103 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f45180cf3b0 con 0x7f45281017b0 2024-03-19T02:54:21.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.533+0000 7f4526d9d640 1 --2- 172.21.15.33:0/1074041103 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f45100a2a90 0x7f45100a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:21.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.533+0000 7f452659c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557403efaa30 0x557403efae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:21.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.533+0000 7f452659c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557403efaa30 0x557403efae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:48624/0 (socket says 172.21.15.33:48624) 2024-03-19T02:54:21.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.533+0000 7f452659c640 1 -- 172.21.15.33:0/2499120579 learned_addr learned my addr 172.21.15.33:0/2499120579 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:21.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.534+0000 7f4526d9d640 1 --2- 172.21.15.33:0/2499120579 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557403f03f30 0x557403f06320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:21.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.534+0000 7f4525d9b640 1 --2- 172.21.15.33:0/2499120579 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557403efb350 0x557403f039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:21.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.534+0000 7f452659c640 1 -- 172.21.15.33:0/2499120579 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557403f03f30 msgr2=0x557403f06320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:21.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.534+0000 7f452659c640 1 --2- 172.21.15.33:0/2499120579 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557403f03f30 0x557403f06320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.534+0000 7f452659c640 1 -- 172.21.15.33:0/2499120579 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557403efb350 msgr2=0x557403f039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:21.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.534+0000 7f452659c640 1 --2- 172.21.15.33:0/2499120579 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557403efb350 0x557403f039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.534+0000 7f452659c640 1 -- 172.21.15.33:0/2499120579 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557403e28000 con 0x557403efaa30 2024-03-19T02:54:21.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.534+0000 7f4526d9d640 1 --2- 172.21.15.33:0/1074041103 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f45100a2a90 0x7f45100a4f50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4520009aa0 tx=0x7f45200116c0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:21.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.534+0000 7f452659c640 1 --2- 172.21.15.33:0/2499120579 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557403efaa30 0x557403efae10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f451809b3e0 tx=0x7f451809b0c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4dc31c05d1be8219 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:21.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.534+0000 7f4514ff9640 1 -- 172.21.15.33:0/2499120579 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4518014930 con 0x557403efaa30 2024-03-19T02:54:21.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.534+0000 7f4514ff9640 1 -- 172.21.15.33:0/2499120579 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f451801b170 con 0x557403efaa30 2024-03-19T02:54:21.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.534+0000 7f4514ff9640 1 -- 172.21.15.33:0/2499120579 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45180051d0 con 0x557403efaa30 2024-03-19T02:54:21.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.535+0000 7f452f97d740 1 -- 172.21.15.33:0/2499120579 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557403efaa30 msgr2=0x557403efae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:21.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.535+0000 7f452f97d740 1 --2- 172.21.15.33:0/2499120579 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557403efaa30 0x557403efae10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f451809b3e0 tx=0x7f451809b0c0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.535+0000 7f452f97d740 1 -- 172.21.15.33:0/2499120579 shutdown_connections 2024-03-19T02:54:21.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.535+0000 7f452f97d740 1 --2- 172.21.15.33:0/2499120579 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557403f03f30 0x557403f06320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.535+0000 7f452f97d740 1 --2- 172.21.15.33:0/2499120579 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557403efb350 0x557403f039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.535+0000 7f452f97d740 1 --2- 172.21.15.33:0/2499120579 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557403efaa30 0x557403efae10 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.535+0000 7f452f97d740 1 -- 172.21.15.33:0/2499120579 >> 172.21.15.33:0/2499120579 conn(0x557403ef5160 msgr2=0x557403ef5540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:21.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.535+0000 7f452f97d740 1 -- 172.21.15.33:0/2499120579 shutdown_connections 2024-03-19T02:54:21.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.535+0000 7f452f97d740 1 -- 172.21.15.33:0/2499120579 wait complete. 2024-03-19T02:54:21.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.536+0000 7f452f97d740 1 -- start start 2024-03-19T02:54:21.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.536+0000 7f452f97d740 1 Processor -- start 2024-03-19T02:54:21.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.536+0000 7f452f97d740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557403efaa30 0x557403f00f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:21.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.536+0000 7f452f97d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557403efb350 0x557403f01470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:21.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.536+0000 7f452f97d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557403f03f30 0x557403f019b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:21.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.536+0000 7f452f97d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557403f06d20 con 0x557403efb350 2024-03-19T02:54:21.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.536+0000 7f452f97d740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557403e1a160 con 0x557403efaa30 2024-03-19T02:54:21.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.536+0000 7f452f97d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x557403f06ea0 con 0x557403f03f30 2024-03-19T02:54:21.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.536+0000 7f452f97d740 10 client.0 _ll_register_callbacks cb 0x557403e56a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:54:21.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.536+0000 7f4525d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557403efb350 0x557403f01470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:21.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.536+0000 7f4525d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557403efb350 0x557403f01470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37190/0 (socket says 172.21.15.33:37190) 2024-03-19T02:54:21.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.536+0000 7f4525d9b640 1 -- 172.21.15.33:0/1649689060 learned_addr learned my addr 172.21.15.33:0/1649689060 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:21.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.536+0000 7f452659c640 1 --2- 172.21.15.33:0/1649689060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557403efaa30 0x557403f00f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:21.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.536+0000 7f4526d9d640 1 --2- 172.21.15.33:0/1649689060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557403f03f30 0x557403f019b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:21.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.536+0000 7f4525d9b640 1 -- 172.21.15.33:0/1649689060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557403f03f30 msgr2=0x557403f019b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:21.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.536+0000 7f4525d9b640 1 --2- 172.21.15.33:0/1649689060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557403f03f30 0x557403f019b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.536+0000 7f4525d9b640 1 -- 172.21.15.33:0/1649689060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557403efaa30 msgr2=0x557403f00f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:21.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.536+0000 7f4525d9b640 1 --2- 172.21.15.33:0/1649689060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557403efaa30 0x557403f00f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.536+0000 7f4525d9b640 1 -- 172.21.15.33:0/1649689060 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557403c3c7d0 con 0x557403efb350 2024-03-19T02:54:21.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.537+0000 7f4525d9b640 1 --2- 172.21.15.33:0/1649689060 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557403efb350 0x557403f01470 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f451c036040 tx=0x7f451c02c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b83bd356de736651 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:21.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.537+0000 7f450dffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:54:21.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.537+0000 7f450dffb640 1 -- 172.21.15.33:0/1649689060 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f451c02c5d0 con 0x557403efb350 2024-03-19T02:54:21.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.537+0000 7f450dffb640 1 -- 172.21.15.33:0/1649689060 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f451c005030 con 0x557403efb350 2024-03-19T02:54:21.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.537+0000 7f452f97d740 1 -- 172.21.15.33:0/1649689060 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f451801b170 con 0x557403efb350 2024-03-19T02:54:21.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.537+0000 7f450dffb640 1 -- 172.21.15.33:0/1649689060 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f451c005230 con 0x557403efb350 2024-03-19T02:54:21.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.537+0000 7f452f97d740 10 client.15825 fetch_fsmap learned FSMap version 25 2024-03-19T02:54:21.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.537+0000 7f452f97d740 1 -- 172.21.15.33:0/1649689060 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x557403b723f0 con 0x557403efb350 2024-03-19T02:54:21.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.537+0000 7f450dffb640 1 -- 172.21.15.33:0/1649689060 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f451801b170 con 0x557403efb350 2024-03-19T02:54:21.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.537+0000 7f450dffb640 1 -- 172.21.15.33:0/1649689060 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f451c0035f0 con 0x557403efb350 2024-03-19T02:54:21.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.537+0000 7f452f97d740 10 client.15825 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:54:21.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.538+0000 7f452f97d740 10 client.15825 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:54:21.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.538+0000 7f452f97d740 20 client.15825 populate_metadata read hostname 'smithi033' 2024-03-19T02:54:21.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.538+0000 7f452f97d740 1 --2- 172.21.15.33:0/1649689060 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557403f14270 0x557403f166a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:21.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.538+0000 7f452f97d740 4 client.15825 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:54:21.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.538+0000 7f452f97d740 1 -- 172.21.15.33:0/1649689060 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557403e6ba40 con 0x557403f14270 2024-03-19T02:54:21.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.538+0000 7f452659c640 1 --2- 172.21.15.33:0/1649689060 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557403f14270 0x557403f166a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:21.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.538+0000 7f452659c640 1 --2- 172.21.15.33:0/1649689060 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557403f14270 0x557403f166a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf838e6b2deaff36 server_cookie=921b820bcf821ce2 in_seq=0 out_seq=0 2024-03-19T02:54:21.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.538+0000 7f450dffb640 10 client.15825 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:54:21.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.539+0000 7f450dffb640 1 -- 172.21.15.33:0/1649689060 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557403e6ba40 con 0x557403f14270 2024-03-19T02:54:21.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.539+0000 7f450dffb640 10 client.15825 handle_command_reply: tid=0 2024-03-19T02:54:21.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.539+0000 7f452f97d740 1 client.15825 shutdown 2024-03-19T02:54:21.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.539+0000 7f452f97d740 1 -- 172.21.15.33:0/1649689060 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557403efb350 msgr2=0x557403f01470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:21.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.539+0000 7f452f97d740 1 --2- 172.21.15.33:0/1649689060 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557403efb350 0x557403f01470 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f451c036040 tx=0x7f451c02c9f0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.539+0000 7f452f97d740 1 -- 172.21.15.33:0/1649689060 shutdown_connections 2024-03-19T02:54:21.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.539+0000 7f452f97d740 1 --2- 172.21.15.33:0/1649689060 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557403f14270 0x557403f166a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.539+0000 7f452f97d740 1 --2- 172.21.15.33:0/1649689060 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557403f03f30 0x557403f019b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.539+0000 7f450dffb640 0 client.15825 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:54:21.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.539+0000 7f452f97d740 1 --2- 172.21.15.33:0/1649689060 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557403efb350 0x557403f01470 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.539+0000 7f452f97d740 1 --2- 172.21.15.33:0/1649689060 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557403efaa30 0x557403f00f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.539+0000 7f452f97d740 1 -- 172.21.15.33:0/1649689060 >> 172.21.15.33:0/1649689060 conn(0x557403ef5160 msgr2=0x557403ef6eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:21.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.539+0000 7f452f97d740 1 -- 172.21.15.33:0/1649689060 shutdown_connections 2024-03-19T02:54:21.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.539+0000 7f452f97d740 1 -- 172.21.15.33:0/1649689060 wait complete. 2024-03-19T02:54:21.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.539+0000 7f452f97d740 20 client.15825 trim_cache size 0 max 16384 2024-03-19T02:54:21.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.565+0000 7f452f97d740 1 Processor -- start 2024-03-19T02:54:21.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.565+0000 7f452f97d740 1 -- start start 2024-03-19T02:54:21.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.565+0000 7f452f97d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557403efaa30 0x557403f03660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:21.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.565+0000 7f452f97d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557403efb350 0x557403effeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:21.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.565+0000 7f452f97d740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557403f03f30 0x557403f0fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:21.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.565+0000 7f452f97d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f45281b1540 con 0x557403efaa30 2024-03-19T02:54:21.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.565+0000 7f452f97d740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f45281aee10 con 0x557403f03f30 2024-03-19T02:54:21.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.565+0000 7f452f97d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f45281acc40 con 0x557403efb350 2024-03-19T02:54:21.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.565+0000 7f452659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557403efaa30 0x557403f03660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:21.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.566+0000 7f452659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557403efaa30 0x557403f03660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37206/0 (socket says 172.21.15.33:37206) 2024-03-19T02:54:21.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.566+0000 7f452659c640 1 -- 172.21.15.33:0/905040001 learned_addr learned my addr 172.21.15.33:0/905040001 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:21.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.566+0000 7f4525d9b640 1 --2- 172.21.15.33:0/905040001 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557403efb350 0x557403effeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:21.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.566+0000 7f4526d9d640 1 --2- 172.21.15.33:0/905040001 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557403f03f30 0x557403f0fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:21.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.566+0000 7f452659c640 1 -- 172.21.15.33:0/905040001 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557403efb350 msgr2=0x557403effeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:21.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.566+0000 7f452659c640 1 --2- 172.21.15.33:0/905040001 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557403efb350 0x557403effeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.566+0000 7f452659c640 1 -- 172.21.15.33:0/905040001 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557403f03f30 msgr2=0x557403f0fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:21.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.566+0000 7f452659c640 1 --2- 172.21.15.33:0/905040001 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557403f03f30 0x557403f0fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.566+0000 7f452659c640 1 -- 172.21.15.33:0/905040001 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557403b320f0 con 0x557403efaa30 2024-03-19T02:54:21.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.566+0000 7f4525d9b640 1 --2- 172.21.15.33:0/905040001 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557403efb350 0x557403effeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:21.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.566+0000 7f452659c640 1 --2- 172.21.15.33:0/905040001 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557403efaa30 0x557403f03660 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f451800e9c0 tx=0x7f4518009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efbb4aed237eb78c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:21.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.566+0000 7f450ffff640 1 -- 172.21.15.33:0/905040001 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f451809f5a0 con 0x557403efaa30 2024-03-19T02:54:21.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.566+0000 7f450ffff640 1 -- 172.21.15.33:0/905040001 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f451809b450 con 0x557403efaa30 2024-03-19T02:54:21.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.566+0000 7f450ffff640 1 -- 172.21.15.33:0/905040001 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4518097ac0 con 0x557403efaa30 2024-03-19T02:54:21.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.567+0000 7f452f97d740 1 -- 172.21.15.33:0/905040001 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557403efaa30 msgr2=0x557403f03660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:21.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.567+0000 7f452f97d740 1 --2- 172.21.15.33:0/905040001 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557403efaa30 0x557403f03660 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f451800e9c0 tx=0x7f4518009f60 comp rx=0 tx=0).stop 2024-03-19T02:54:21.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.567+0000 7f452f97d740 1 -- 172.21.15.33:0/905040001 shutdown_connections 2024-03-19T02:54:21.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.567+0000 7f452f97d740 1 --2- 172.21.15.33:0/905040001 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557403efb350 0x557403effeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.567+0000 7f452f97d740 1 --2- 172.21.15.33:0/905040001 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557403f03f30 0x557403f0fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.567+0000 7f452f97d740 1 --2- 172.21.15.33:0/905040001 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557403efaa30 0x557403f03660 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.567+0000 7f452f97d740 1 -- 172.21.15.33:0/905040001 >> 172.21.15.33:0/905040001 conn(0x557403ef5160 msgr2=0x557403f01780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:21.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.567+0000 7f452f97d740 1 -- 172.21.15.33:0/905040001 shutdown_connections 2024-03-19T02:54:21.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.567+0000 7f452f97d740 1 -- 172.21.15.33:0/905040001 wait complete. 2024-03-19T02:54:21.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.567+0000 7f452f97d740 1 -- start start 2024-03-19T02:54:21.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.568+0000 7f452f97d740 1 Processor -- start 2024-03-19T02:54:21.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.568+0000 7f452f97d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557403efaa30 0x557403efe210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:21.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.568+0000 7f452f97d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557403efb350 0x557403efe750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:21.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.568+0000 7f452f97d740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557403f03f30 0x557403f795e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:21.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.568+0000 7f452f97d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f45281ad2d0 con 0x557403efaa30 2024-03-19T02:54:21.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.568+0000 7f452f97d740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f45281ae740 con 0x557403f03f30 2024-03-19T02:54:21.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.568+0000 7f452f97d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f45281ad9a0 con 0x557403efb350 2024-03-19T02:54:21.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.568+0000 7f452f97d740 10 client.0 _ll_register_callbacks cb 0x7f451c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:54:21.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.568+0000 7f452659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557403efaa30 0x557403efe210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:21.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.568+0000 7f452659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557403efaa30 0x557403efe210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37208/0 (socket says 172.21.15.33:37208) 2024-03-19T02:54:21.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.568+0000 7f452659c640 1 -- 172.21.15.33:0/2670051577 learned_addr learned my addr 172.21.15.33:0/2670051577 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:21.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.568+0000 7f4526d9d640 1 --2- 172.21.15.33:0/2670051577 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557403f03f30 0x557403f795e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:21.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.568+0000 7f4525d9b640 1 --2- 172.21.15.33:0/2670051577 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557403efb350 0x557403efe750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:21.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.568+0000 7f452659c640 1 -- 172.21.15.33:0/2670051577 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557403efb350 msgr2=0x557403efe750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:21.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.568+0000 7f452659c640 1 --2- 172.21.15.33:0/2670051577 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557403efb350 0x557403efe750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.568+0000 7f452659c640 1 -- 172.21.15.33:0/2670051577 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557403f03f30 msgr2=0x557403f795e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:21.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.568+0000 7f452659c640 1 --2- 172.21.15.33:0/2670051577 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557403f03f30 0x557403f795e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.568+0000 7f452659c640 1 -- 172.21.15.33:0/2670051577 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557403f79b20 con 0x557403efaa30 2024-03-19T02:54:21.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.568+0000 7f452659c640 1 --2- 172.21.15.33:0/2670051577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557403efaa30 0x557403efe210 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f45180978b0 tx=0x7f45180143c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8451dd5e23f36a71 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:21.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.568+0000 7f450dffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:54:21.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.569+0000 7f450dffb640 1 -- 172.21.15.33:0/2670051577 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f451801b170 con 0x557403efaa30 2024-03-19T02:54:21.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.569+0000 7f450dffb640 1 -- 172.21.15.33:0/2670051577 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f45180b4220 con 0x557403efaa30 2024-03-19T02:54:21.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.569+0000 7f452f97d740 1 -- 172.21.15.33:0/2670051577 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f451809b450 con 0x557403efaa30 2024-03-19T02:54:21.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.569+0000 7f450dffb640 1 -- 172.21.15.33:0/2670051577 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f45180a3bf0 con 0x557403efaa30 2024-03-19T02:54:21.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.569+0000 7f452f97d740 10 client.15831 fetch_fsmap learned FSMap version 25 2024-03-19T02:54:21.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.569+0000 7f452f97d740 1 -- 172.21.15.33:0/2670051577 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x557403f7ade0 con 0x557403efaa30 2024-03-19T02:54:21.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.569+0000 7f450dffb640 1 -- 172.21.15.33:0/2670051577 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f451809b450 con 0x557403efaa30 2024-03-19T02:54:21.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.569+0000 7f450dffb640 1 -- 172.21.15.33:0/2670051577 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f451809f170 con 0x557403efaa30 2024-03-19T02:54:21.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.569+0000 7f452f97d740 10 client.15831 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:54:21.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.569+0000 7f452f97d740 10 client.15831 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:54:21.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.569+0000 7f452f97d740 20 client.15831 populate_metadata read hostname 'smithi033' 2024-03-19T02:54:21.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.569+0000 7f452f97d740 1 --2- 172.21.15.33:0/2670051577 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557403f14270 0x557403f7d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:21.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.569+0000 7f452f97d740 4 client.15831 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:54:21.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.569+0000 7f452f97d740 1 -- 172.21.15.33:0/2670051577 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557403e65d90 con 0x557403f14270 2024-03-19T02:54:21.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.569+0000 7f4525d9b640 1 --2- 172.21.15.33:0/2670051577 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557403f14270 0x557403f7d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:21.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.570+0000 7f4525d9b640 1 --2- 172.21.15.33:0/2670051577 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557403f14270 0x557403f7d070 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4ba7cefcbda0603 server_cookie=c2e73504a6f9345 in_seq=0 out_seq=0 2024-03-19T02:54:21.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.570+0000 7f450dffb640 10 client.15831 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:54:21.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.570+0000 7f450dffb640 1 -- 172.21.15.33:0/2670051577 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557403e65d90 con 0x557403f14270 2024-03-19T02:54:21.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.570+0000 7f450dffb640 10 client.15831 handle_command_reply: tid=0 2024-03-19T02:54:21.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.570+0000 7f452f97d740 1 client.15831 shutdown 2024-03-19T02:54:21.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.570+0000 7f452f97d740 1 -- 172.21.15.33:0/2670051577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557403efaa30 msgr2=0x557403efe210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:21.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.570+0000 7f452f97d740 1 --2- 172.21.15.33:0/2670051577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557403efaa30 0x557403efe210 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f45180978b0 tx=0x7f45180143c0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.570+0000 7f452f97d740 1 -- 172.21.15.33:0/2670051577 shutdown_connections 2024-03-19T02:54:21.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.570+0000 7f452f97d740 1 --2- 172.21.15.33:0/2670051577 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557403f14270 0x557403f7d070 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.570+0000 7f452f97d740 1 --2- 172.21.15.33:0/2670051577 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557403efb350 0x557403efe750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.570+0000 7f452f97d740 1 --2- 172.21.15.33:0/2670051577 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557403f03f30 0x557403f795e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.570+0000 7f450dffb640 0 client.15831 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:54:21.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.570+0000 7f452f97d740 1 --2- 172.21.15.33:0/2670051577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557403efaa30 0x557403efe210 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.570+0000 7f452f97d740 1 -- 172.21.15.33:0/2670051577 >> 172.21.15.33:0/2670051577 conn(0x557403ef5160 msgr2=0x557403ef6440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:21.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.570+0000 7f452f97d740 1 -- 172.21.15.33:0/2670051577 shutdown_connections 2024-03-19T02:54:21.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.570+0000 7f452f97d740 1 -- 172.21.15.33:0/2670051577 wait complete. 2024-03-19T02:54:21.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.570+0000 7f452f97d740 20 client.15831 trim_cache size 0 max 16384 2024-03-19T02:54:21.578 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:54:21.578 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-03-19T02:54:21.578 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-03-19T02:54:21.578 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:54:21.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.571+0000 7f452ccb2640 1 -- 172.21.15.33:0/1074041103 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f45100a2a90 msgr2=0x7f45100a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:21.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.571+0000 7f452ccb2640 1 --2- 172.21.15.33:0/1074041103 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f45100a2a90 0x7f45100a4f50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4520009aa0 tx=0x7f45200116c0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.571+0000 7f452ccb2640 1 -- 172.21.15.33:0/1074041103 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45281017b0 msgr2=0x7f4528198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:21.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.571+0000 7f452ccb2640 1 --2- 172.21.15.33:0/1074041103 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45281017b0 0x7f4528198050 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f451800e9f0 tx=0x7f451800eeb0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.571+0000 7f452ccb2640 1 -- 172.21.15.33:0/1074041103 shutdown_connections 2024-03-19T02:54:21.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.571+0000 7f452ccb2640 1 --2- 172.21.15.33:0/1074041103 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f45100a2a90 0x7f45100a4f50 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.571+0000 7f452ccb2640 1 --2- 172.21.15.33:0/1074041103 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4528102170 0x7f4528198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.571+0000 7f452ccb2640 1 --2- 172.21.15.33:0/1074041103 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f45281077c0 0x7f452819c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.571+0000 7f452ccb2640 1 --2- 172.21.15.33:0/1074041103 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45281017b0 0x7f4528198050 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.571+0000 7f452ccb2640 1 -- 172.21.15.33:0/1074041103 >> 172.21.15.33:0/1074041103 conn(0x7f45280fd2c0 msgr2=0x7f45280fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:21.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.571+0000 7f452ccb2640 1 -- 172.21.15.33:0/1074041103 shutdown_connections 2024-03-19T02:54:21.580 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.571+0000 7f452ccb2640 1 -- 172.21.15.33:0/1074041103 wait complete. 2024-03-19T02:54:21.584 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:54:21.585 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:54:21.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.713+0000 7f3925a4b640 1 -- 172.21.15.33:0/2126507602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f39201010e0 msgr2=0x7f392010d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:21.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.713+0000 7f3925a4b640 1 --2- 172.21.15.33:0/2126507602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f39201010e0 0x7f392010d5f0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f3918009cd0 tx=0x7f391802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.714+0000 7f3925a4b640 1 -- 172.21.15.33:0/2126507602 shutdown_connections 2024-03-19T02:54:21.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.714+0000 7f3925a4b640 1 --2- 172.21.15.33:0/2126507602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f39201010e0 0x7f392010d5f0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.714+0000 7f3925a4b640 1 --2- 172.21.15.33:0/2126507602 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3920100720 0x7f3920100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.714+0000 7f3925a4b640 1 --2- 172.21.15.33:0/2126507602 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3920106730 0x7f3920106b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.714+0000 7f3925a4b640 1 -- 172.21.15.33:0/2126507602 >> 172.21.15.33:0/2126507602 conn(0x7f3920077820 msgr2=0x7f3920077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:21.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.714+0000 7f3925a4b640 1 -- 172.21.15.33:0/2126507602 shutdown_connections 2024-03-19T02:54:21.715 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.714+0000 7f3925a4b640 1 -- 172.21.15.33:0/2126507602 wait complete. 2024-03-19T02:54:21.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.714+0000 7f3925a4b640 1 Processor -- start 2024-03-19T02:54:21.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.715+0000 7f3925a4b640 1 -- start start 2024-03-19T02:54:21.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.715+0000 7f3925a4b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3920100720 0x7f3920197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:21.717 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.715+0000 7f3925a4b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39201010e0 0x7f39201984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:21.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.715+0000 7f3925a4b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3920106730 0x7f392019c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:21.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.715+0000 7f3925a4b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f39201109f0 con 0x7f3920100720 2024-03-19T02:54:21.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.715+0000 7f3925a4b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3920110870 con 0x7f39201010e0 2024-03-19T02:54:21.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.715+0000 7f3925a4b640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3920110b70 con 0x7f3920106730 2024-03-19T02:54:21.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.715+0000 7f391effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3920100720 0x7f3920197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:21.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.715+0000 7f391effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3920100720 0x7f3920197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37226/0 (socket says 172.21.15.33:37226) 2024-03-19T02:54:21.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.715+0000 7f391effd640 1 -- 172.21.15.33:0/1273623266 learned_addr learned my addr 172.21.15.33:0/1273623266 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:21.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.715+0000 7f391e7fc640 1 --2- 172.21.15.33:0/1273623266 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39201010e0 0x7f39201984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:21.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.715+0000 7f391effd640 1 -- 172.21.15.33:0/1273623266 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3920106730 msgr2=0x7f392019c850 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-19T02:54:21.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.715+0000 7f391effd640 1 --2- 172.21.15.33:0/1273623266 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3920106730 0x7f392019c850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.715+0000 7f391effd640 1 -- 172.21.15.33:0/1273623266 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39201010e0 msgr2=0x7f39201984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:21.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.715+0000 7f391effd640 1 --2- 172.21.15.33:0/1273623266 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39201010e0 0x7f39201984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.718 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.715+0000 7f391effd640 1 -- 172.21.15.33:0/1273623266 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f392019cf80 con 0x7f3920100720 2024-03-19T02:54:21.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.716+0000 7f391effd640 1 --2- 172.21.15.33:0/1273623266 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3920100720 0x7f3920197f90 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f391000e9f0 tx=0x7f391000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:21.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.716+0000 7f3924a49640 1 -- 172.21.15.33:0/1273623266 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f391000cd80 con 0x7f3920100720 2024-03-19T02:54:21.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.716+0000 7f3925a4b640 1 -- 172.21.15.33:0/1273623266 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f392019d270 con 0x7f3920100720 2024-03-19T02:54:21.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.716+0000 7f3924a49640 1 -- 172.21.15.33:0/1273623266 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f39100108e0 con 0x7f3920100720 2024-03-19T02:54:21.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.716+0000 7f3924a49640 1 -- 172.21.15.33:0/1273623266 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3910018920 con 0x7f3920100720 2024-03-19T02:54:21.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.716+0000 7f3925a4b640 1 -- 172.21.15.33:0/1273623266 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39201a4b00 con 0x7f3920100720 2024-03-19T02:54:21.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.717+0000 7f3925a4b640 1 -- 172.21.15.33:0/1273623266 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3920068610 con 0x7f3920100720 2024-03-19T02:54:21.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.718+0000 7f3924a49640 1 -- 172.21.15.33:0/1273623266 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7f3910010a90 con 0x7f3920100720 2024-03-19T02:54:21.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.719+0000 7f3924a49640 1 --2- 172.21.15.33:0/1273623266 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f39080a2a90 0x7f39080a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:21.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.719+0000 7f391e7fc640 1 --2- 172.21.15.33:0/1273623266 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f39080a2a90 0x7f39080a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:21.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.719+0000 7f3924a49640 1 -- 172.21.15.33:0/1273623266 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f39100cf7f0 con 0x7f3920100720 2024-03-19T02:54:21.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.720+0000 7f391e7fc640 1 --2- 172.21.15.33:0/1273623266 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f39080a2a90 0x7f39080a4f50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f39140095f0 tx=0x7f3914005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:21.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.725+0000 7f3924a49640 1 -- 172.21.15.33:0/1273623266 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f391009bf90 con 0x7f3920100720 2024-03-19T02:54:21.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.987+0000 7f3925a4b640 1 -- 172.21.15.33:0/1273623266 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3920109d00 con 0x7f3920100720 2024-03-19T02:54:21.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.988+0000 7f3924a49640 1 -- 172.21.15.33:0/1273623266 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11867 (secure 0 0 0) 0x7f391009b890 con 0x7f3920100720 2024-03-19T02:54:21.989 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:54:21.989 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:50:52.201798+0000","modified":"2024-03-19T02:53:11.748697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24488,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/3731232169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":3731232169},{"type":"v1","addr":"172.21.15.96:6839","nonce":3731232169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24457,24488,24509]},"id":1}]} 2024-03-19T02:54:21.989 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-03-19T02:54:21.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.990+0000 7f3925a4b640 1 -- 172.21.15.33:0/1273623266 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f39080a2a90 msgr2=0x7f39080a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:21.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.990+0000 7f3925a4b640 1 --2- 172.21.15.33:0/1273623266 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f39080a2a90 0x7f39080a4f50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f39140095f0 tx=0x7f3914005f90 comp rx=0 tx=0).stop 2024-03-19T02:54:21.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.990+0000 7f3925a4b640 1 -- 172.21.15.33:0/1273623266 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3920100720 msgr2=0x7f3920197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:21.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.990+0000 7f3925a4b640 1 --2- 172.21.15.33:0/1273623266 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3920100720 0x7f3920197f90 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f391000e9f0 tx=0x7f391000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.990+0000 7f3925a4b640 1 -- 172.21.15.33:0/1273623266 shutdown_connections 2024-03-19T02:54:21.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.990+0000 7f3925a4b640 1 --2- 172.21.15.33:0/1273623266 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f39080a2a90 0x7f39080a4f50 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.990+0000 7f3925a4b640 1 --2- 172.21.15.33:0/1273623266 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3920106730 0x7f392019c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.990+0000 7f3925a4b640 1 --2- 172.21.15.33:0/1273623266 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f39201010e0 0x7f39201984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.990+0000 7f3925a4b640 1 --2- 172.21.15.33:0/1273623266 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3920100720 0x7f3920197f90 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:21.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.990+0000 7f3925a4b640 1 -- 172.21.15.33:0/1273623266 >> 172.21.15.33:0/1273623266 conn(0x7f3920077820 msgr2=0x7f3920068c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:21.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.990+0000 7f3925a4b640 1 -- 172.21.15.33:0/1273623266 shutdown_connections 2024-03-19T02:54:21.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:21.990+0000 7f3925a4b640 1 -- 172.21.15.33:0/1273623266 wait complete. 2024-03-19T02:54:22.002 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:54:22.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.123+0000 7f866f113640 1 -- 172.21.15.33:0/567838733 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8668101060 msgr2=0x7f866810d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:22.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.123+0000 7f866f113640 1 --2- 172.21.15.33:0/567838733 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8668101060 0x7f866810d570 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f8660009cd0 tx=0x7f866002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.123+0000 7f866f113640 1 -- 172.21.15.33:0/567838733 shutdown_connections 2024-03-19T02:54:22.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.123+0000 7f866f113640 1 --2- 172.21.15.33:0/567838733 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f86681006a0 0x7f8668100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.123+0000 7f866f113640 1 --2- 172.21.15.33:0/567838733 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8668101060 0x7f866810d570 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.123+0000 7f866f113640 1 --2- 172.21.15.33:0/567838733 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86681066b0 0x7f8668106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.123+0000 7f866f113640 1 -- 172.21.15.33:0/567838733 >> 172.21.15.33:0/567838733 conn(0x7f86680778d0 msgr2=0x7f8668077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:22.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.123+0000 7f866f113640 1 -- 172.21.15.33:0/567838733 shutdown_connections 2024-03-19T02:54:22.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.123+0000 7f866f113640 1 -- 172.21.15.33:0/567838733 wait complete. 2024-03-19T02:54:22.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.124+0000 7f866f113640 1 Processor -- start 2024-03-19T02:54:22.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.124+0000 7f866f113640 1 -- start start 2024-03-19T02:54:22.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.124+0000 7f866f113640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86681006a0 0x7f8668198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.124+0000 7f866f113640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8668101060 0x7f8668199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.124+0000 7f866f113640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f86681066b0 0x7f86681a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.124+0000 7f866f113640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8668110970 con 0x7f86681066b0 2024-03-19T02:54:22.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.124+0000 7f866f113640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f86681107f0 con 0x7f86681006a0 2024-03-19T02:54:22.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.124+0000 7f866f113640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8668110af0 con 0x7f8668101060 2024-03-19T02:54:22.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.124+0000 7f866d6b0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f86681066b0 0x7f86681a3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.124+0000 7f866d6b0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f86681066b0 0x7f86681a3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37242/0 (socket says 172.21.15.33:37242) 2024-03-19T02:54:22.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.124+0000 7f8667fff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8668101060 0x7f8668199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.124+0000 7f866d6b0640 1 -- 172.21.15.33:0/2809011725 learned_addr learned my addr 172.21.15.33:0/2809011725 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:22.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.124+0000 7f866ceaf640 1 --2- 172.21.15.33:0/2809011725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86681006a0 0x7f8668198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.125+0000 7f866d6b0640 1 -- 172.21.15.33:0/2809011725 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8668101060 msgr2=0x7f8668199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:22.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.125+0000 7f866d6b0640 1 --2- 172.21.15.33:0/2809011725 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8668101060 0x7f8668199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.125+0000 7f866d6b0640 1 -- 172.21.15.33:0/2809011725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86681006a0 msgr2=0x7f8668198df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:22.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.125+0000 7f866d6b0640 1 --2- 172.21.15.33:0/2809011725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86681006a0 0x7f8668198df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.125+0000 7f866d6b0640 1 -- 172.21.15.33:0/2809011725 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86681a43b0 con 0x7f86681066b0 2024-03-19T02:54:22.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.125+0000 7f866ceaf640 1 --2- 172.21.15.33:0/2809011725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86681006a0 0x7f8668198df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:22.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.125+0000 7f866d6b0640 1 --2- 172.21.15.33:0/2809011725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f86681066b0 0x7f86681a3e70 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f8660004af0 tx=0x7f8660035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:22.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.125+0000 7f8665ffb640 1 -- 172.21.15.33:0/2809011725 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8660039040 con 0x7f86681066b0 2024-03-19T02:54:22.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.125+0000 7f866f113640 1 -- 172.21.15.33:0/2809011725 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86681a4580 con 0x7f86681066b0 2024-03-19T02:54:22.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.125+0000 7f8665ffb640 1 -- 172.21.15.33:0/2809011725 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8660044070 con 0x7f86681066b0 2024-03-19T02:54:22.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.125+0000 7f8665ffb640 1 -- 172.21.15.33:0/2809011725 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86600042a0 con 0x7f86681066b0 2024-03-19T02:54:22.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.125+0000 7f866f113640 1 -- 172.21.15.33:0/2809011725 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86681a4a40 con 0x7f86681066b0 2024-03-19T02:54:22.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.127+0000 7f8665ffb640 1 -- 172.21.15.33:0/2809011725 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7f8660003970 con 0x7f86681066b0 2024-03-19T02:54:22.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.128+0000 7f8671dd0740 1 Processor -- start 2024-03-19T02:54:22.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.128+0000 7f8665ffb640 1 --2- 172.21.15.33:0/2809011725 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f86580a2a90 0x7f86580a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.128+0000 7f8671dd0740 1 -- start start 2024-03-19T02:54:22.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.128+0000 7f8667fff640 1 --2- 172.21.15.33:0/2809011725 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f86580a2a90 0x7f86580a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.128+0000 7f8665ffb640 1 -- 172.21.15.33:0/2809011725 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f86600f1790 con 0x7f86681066b0 2024-03-19T02:54:22.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.128+0000 7f8671dd0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642f6aef2d0 0x5642f6aef6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.128+0000 7f8671dd0740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642f6aefbf0 0x5642f6af8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.128+0000 7f8671dd0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642f6af87d0 0x5642f6afabc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.128+0000 7f8671dd0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5642f6a524b0 con 0x5642f6aef2d0 2024-03-19T02:54:22.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.128+0000 7f8671dd0740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5642f69fad70 con 0x5642f6aefbf0 2024-03-19T02:54:22.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.128+0000 7f8671dd0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5642f6a4e550 con 0x5642f6af87d0 2024-03-19T02:54:22.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.129+0000 7f866ceaf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642f6aef2d0 0x5642f6aef6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.129+0000 7f8667fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642f6aefbf0 0x5642f6af8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.129+0000 7f866ceaf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642f6aef2d0 0x5642f6aef6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37254/0 (socket says 172.21.15.33:37254) 2024-03-19T02:54:22.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.129+0000 7f866ceaf640 1 -- 172.21.15.33:0/3418595442 learned_addr learned my addr 172.21.15.33:0/3418595442 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:22.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.129+0000 7f866d6b0640 1 --2- 172.21.15.33:0/3418595442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642f6af87d0 0x5642f6afabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.129+0000 7f866ceaf640 1 -- 172.21.15.33:0/3418595442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642f6af87d0 msgr2=0x5642f6afabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:22.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.129+0000 7f866ceaf640 1 --2- 172.21.15.33:0/3418595442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642f6af87d0 0x5642f6afabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.129+0000 7f866ceaf640 1 -- 172.21.15.33:0/3418595442 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642f6aefbf0 msgr2=0x5642f6af8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:22.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.129+0000 7f866ceaf640 1 --2- 172.21.15.33:0/3418595442 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642f6aefbf0 0x5642f6af8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.129+0000 7f866ceaf640 1 -- 172.21.15.33:0/3418595442 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642f6a1d210 con 0x5642f6aef2d0 2024-03-19T02:54:22.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.129+0000 7f8667fff640 1 --2- 172.21.15.33:0/2809011725 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f86580a2a90 0x7f86580a4f50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f86680694f0 tx=0x7f8654011920 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:22.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.129+0000 7f866ceaf640 1 --2- 172.21.15.33:0/3418595442 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642f6aef2d0 0x5642f6aef6b0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f865c0114f0 tx=0x7f865c0118f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cccb20c0a47d5bb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:22.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.129+0000 7f8652ffd640 1 -- 172.21.15.33:0/3418595442 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865c01e070 con 0x5642f6aef2d0 2024-03-19T02:54:22.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.129+0000 7f8652ffd640 1 -- 172.21.15.33:0/3418595442 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f865c00f930 con 0x5642f6aef2d0 2024-03-19T02:54:22.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.129+0000 7f8652ffd640 1 -- 172.21.15.33:0/3418595442 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865c019980 con 0x5642f6aef2d0 2024-03-19T02:54:22.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.130+0000 7f8671dd0740 1 -- 172.21.15.33:0/3418595442 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642f6aef2d0 msgr2=0x5642f6aef6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:22.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.130+0000 7f8671dd0740 1 --2- 172.21.15.33:0/3418595442 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642f6aef2d0 0x5642f6aef6b0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f865c0114f0 tx=0x7f865c0118f0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.130+0000 7f8671dd0740 1 -- 172.21.15.33:0/3418595442 shutdown_connections 2024-03-19T02:54:22.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.130+0000 7f8671dd0740 1 --2- 172.21.15.33:0/3418595442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642f6af87d0 0x5642f6afabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.130+0000 7f8671dd0740 1 --2- 172.21.15.33:0/3418595442 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642f6aefbf0 0x5642f6af8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.130+0000 7f8671dd0740 1 --2- 172.21.15.33:0/3418595442 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642f6aef2d0 0x5642f6aef6b0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.130+0000 7f8671dd0740 1 -- 172.21.15.33:0/3418595442 >> 172.21.15.33:0/3418595442 conn(0x5642f6ae9a00 msgr2=0x5642f6ae9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:22.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.130+0000 7f8671dd0740 1 -- 172.21.15.33:0/3418595442 shutdown_connections 2024-03-19T02:54:22.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.130+0000 7f8671dd0740 1 -- 172.21.15.33:0/3418595442 wait complete. 2024-03-19T02:54:22.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.131+0000 7f8671dd0740 1 -- start start 2024-03-19T02:54:22.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.131+0000 7f8671dd0740 1 Processor -- start 2024-03-19T02:54:22.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.131+0000 7f8671dd0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642f6aef2d0 0x5642f6af58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.131+0000 7f8671dd0740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642f6aefbf0 0x5642f6af5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.131+0000 7f8671dd0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642f6af87d0 0x5642f6af6320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.131+0000 7f8671dd0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5642f6afb5c0 con 0x5642f6aef2d0 2024-03-19T02:54:22.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.131+0000 7f8671dd0740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5642f6a0e310 con 0x5642f6aefbf0 2024-03-19T02:54:22.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.131+0000 7f8671dd0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5642f6afb740 con 0x5642f6af87d0 2024-03-19T02:54:22.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.131+0000 7f866ceaf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642f6aef2d0 0x5642f6af58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.131+0000 7f866ceaf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642f6aef2d0 0x5642f6af58a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37256/0 (socket says 172.21.15.33:37256) 2024-03-19T02:54:22.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.131+0000 7f866ceaf640 1 -- 172.21.15.33:0/2351974527 learned_addr learned my addr 172.21.15.33:0/2351974527 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:22.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.131+0000 7f8667fff640 1 --2- 172.21.15.33:0/2351974527 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642f6aefbf0 0x5642f6af5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.131+0000 7f8671dd0740 10 client.0 _ll_register_callbacks cb 0x5642f6a022d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:54:22.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.131+0000 7f866d6b0640 1 --2- 172.21.15.33:0/2351974527 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642f6af87d0 0x5642f6af6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.131+0000 7f8667fff640 1 -- 172.21.15.33:0/2351974527 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642f6af87d0 msgr2=0x5642f6af6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:22.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.131+0000 7f8667fff640 1 --2- 172.21.15.33:0/2351974527 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642f6af87d0 0x5642f6af6320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.131+0000 7f8667fff640 1 -- 172.21.15.33:0/2351974527 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642f6aef2d0 msgr2=0x5642f6af58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:22.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.131+0000 7f8667fff640 1 --2- 172.21.15.33:0/2351974527 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642f6aef2d0 0x5642f6af58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.132+0000 7f8667fff640 1 -- 172.21.15.33:0/2351974527 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642f67663f0 con 0x5642f6aefbf0 2024-03-19T02:54:22.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.132+0000 7f866ceaf640 1 --2- 172.21.15.33:0/2351974527 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642f6aef2d0 0x5642f6af58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:22.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.132+0000 7f8667fff640 1 --2- 172.21.15.33:0/2351974527 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642f6aefbf0 0x5642f6af5de0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8654009de0 tx=0x7f8654004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=702daa2d4b4a34f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:22.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.132+0000 7f8647fff640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T02:54:22.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.132+0000 7f8647fff640 1 -- 172.21.15.33:0/2351974527 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865401a040 con 0x5642f6aefbf0 2024-03-19T02:54:22.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.132+0000 7f8671dd0740 1 -- 172.21.15.33:0/2351974527 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f865c00f930 con 0x5642f6aefbf0 2024-03-19T02:54:22.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.132+0000 7f8647fff640 1 -- 172.21.15.33:0/2351974527 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f8654003a00 con 0x5642f6aefbf0 2024-03-19T02:54:22.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.132+0000 7f8647fff640 1 -- 172.21.15.33:0/2351974527 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8654003c00 con 0x5642f6aefbf0 2024-03-19T02:54:22.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.132+0000 7f8671dd0740 10 client.25721 fetch_fsmap learned FSMap version 25 2024-03-19T02:54:22.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.132+0000 7f8671dd0740 1 -- 172.21.15.33:0/2351974527 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5642f68307d0 con 0x5642f6aefbf0 2024-03-19T02:54:22.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.132+0000 7f8647fff640 1 -- 172.21.15.33:0/2351974527 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f865c00f930 con 0x5642f6aefbf0 2024-03-19T02:54:22.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.132+0000 7f8647fff640 1 -- 172.21.15.33:0/2351974527 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f865401d030 con 0x5642f6aefbf0 2024-03-19T02:54:22.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.133+0000 7f8671dd0740 10 client.25721 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:54:22.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.133+0000 7f8671dd0740 10 client.25721 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T02:54:22.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.133+0000 7f8671dd0740 20 client.25721 populate_metadata read hostname 'smithi033' 2024-03-19T02:54:22.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.133+0000 7f8671dd0740 1 --2- 172.21.15.33:0/2351974527 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x5642f6b08b60 0x5642f6b0aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.133+0000 7f8671dd0740 4 client.25721 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:54:22.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.133+0000 7f8671dd0740 1 -- 172.21.15.33:0/2351974527 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5642f6ab03e0 con 0x5642f6b08b60 2024-03-19T02:54:22.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.133+0000 7f866ceaf640 1 --2- 172.21.15.33:0/2351974527 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x5642f6b08b60 0x5642f6b0aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.133+0000 7f8647fff640 10 client.25721 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T02:54:22.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.133+0000 7f866ceaf640 1 --2- 172.21.15.33:0/2351974527 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x5642f6b08b60 0x5642f6b0aff0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=841230027bd57b3d server_cookie=9e3fcd54648605cb in_seq=0 out_seq=0 2024-03-19T02:54:22.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.134+0000 7f8647fff640 1 -- 172.21.15.33:0/2351974527 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5642f6ab03e0 con 0x5642f6b08b60 2024-03-19T02:54:22.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.134+0000 7f8647fff640 10 client.25721 handle_command_reply: tid=0 2024-03-19T02:54:22.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.134+0000 7f8671dd0740 1 client.25721 shutdown 2024-03-19T02:54:22.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.134+0000 7f8671dd0740 1 -- 172.21.15.33:0/2351974527 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642f6aefbf0 msgr2=0x5642f6af5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:22.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.134+0000 7f8671dd0740 1 --2- 172.21.15.33:0/2351974527 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642f6aefbf0 0x5642f6af5de0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8654009de0 tx=0x7f8654004730 comp rx=0 tx=0).stop 2024-03-19T02:54:22.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.134+0000 7f8671dd0740 1 -- 172.21.15.33:0/2351974527 shutdown_connections 2024-03-19T02:54:22.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.134+0000 7f8671dd0740 1 --2- 172.21.15.33:0/2351974527 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x5642f6b08b60 0x5642f6b0aff0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.134+0000 7f8671dd0740 1 --2- 172.21.15.33:0/2351974527 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642f6af87d0 0x5642f6af6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.134+0000 7f8671dd0740 1 --2- 172.21.15.33:0/2351974527 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642f6aefbf0 0x5642f6af5de0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.134+0000 7f8671dd0740 1 --2- 172.21.15.33:0/2351974527 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642f6aef2d0 0x5642f6af58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.134+0000 7f8647fff640 0 client.25721 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T02:54:22.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.134+0000 7f8671dd0740 1 -- 172.21.15.33:0/2351974527 >> 172.21.15.33:0/2351974527 conn(0x5642f6ae9a00 msgr2=0x5642f6aeb750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:22.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.134+0000 7f8671dd0740 1 -- 172.21.15.33:0/2351974527 shutdown_connections 2024-03-19T02:54:22.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.134+0000 7f8671dd0740 1 -- 172.21.15.33:0/2351974527 wait complete. 2024-03-19T02:54:22.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.134+0000 7f8671dd0740 20 client.25721 trim_cache size 0 max 16384 2024-03-19T02:54:22.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.149+0000 7f8671dd0740 1 Processor -- start 2024-03-19T02:54:22.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.150+0000 7f8671dd0740 1 -- start start 2024-03-19T02:54:22.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.150+0000 7f8671dd0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642f6aef2d0 0x5642f6b04360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.150+0000 7f8671dd0740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642f6aefbf0 0x5642f6b108c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.150+0000 7f8671dd0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642f6af87d0 0x5642f6af4170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.150+0000 7f8671dd0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f86681b1420 con 0x5642f6af87d0 2024-03-19T02:54:22.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.150+0000 7f8671dd0740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f86681aecf0 con 0x5642f6aefbf0 2024-03-19T02:54:22.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.150+0000 7f8671dd0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f86681acb20 con 0x5642f6aef2d0 2024-03-19T02:54:22.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.150+0000 7f866d6b0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642f6af87d0 0x5642f6af4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.150+0000 7f866d6b0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642f6af87d0 0x5642f6af4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37270/0 (socket says 172.21.15.33:37270) 2024-03-19T02:54:22.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.150+0000 7f866d6b0640 1 -- 172.21.15.33:0/2905854174 learned_addr learned my addr 172.21.15.33:0/2905854174 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:22.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.150+0000 7f866ceaf640 1 --2- 172.21.15.33:0/2905854174 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642f6aef2d0 0x5642f6b04360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.151+0000 7f866d6b0640 1 -- 172.21.15.33:0/2905854174 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642f6aef2d0 msgr2=0x5642f6b04360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:22.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.151+0000 7f866d6b0640 1 --2- 172.21.15.33:0/2905854174 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642f6aef2d0 0x5642f6b04360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.151+0000 7f866d6b0640 1 -- 172.21.15.33:0/2905854174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642f6aefbf0 msgr2=0x5642f6b108c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:22.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.151+0000 7f8667fff640 1 --2- 172.21.15.33:0/2905854174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642f6aefbf0 0x5642f6b108c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.151+0000 7f866d6b0640 1 --2- 172.21.15.33:0/2905854174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642f6aefbf0 0x5642f6b108c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.151+0000 7f866d6b0640 1 -- 172.21.15.33:0/2905854174 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642f6aea5d0 con 0x5642f6af87d0 2024-03-19T02:54:22.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.151+0000 7f8667fff640 1 --2- 172.21.15.33:0/2905854174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642f6aefbf0 0x5642f6b108c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:54:22.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.151+0000 7f866d6b0640 1 --2- 172.21.15.33:0/2905854174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642f6af87d0 0x5642f6af4170 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f8660006010 tx=0x7f8660035f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a452cca5deaa8e15 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:22.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.151+0000 7f86527fc640 1 -- 172.21.15.33:0/2905854174 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8660034700 con 0x5642f6af87d0 2024-03-19T02:54:22.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.151+0000 7f86527fc640 1 -- 172.21.15.33:0/2905854174 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f86600098d0 con 0x5642f6af87d0 2024-03-19T02:54:22.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.151+0000 7f86527fc640 1 -- 172.21.15.33:0/2905854174 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86600b9fb0 con 0x5642f6af87d0 2024-03-19T02:54:22.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.152+0000 7f8671dd0740 1 -- 172.21.15.33:0/2905854174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642f6af87d0 msgr2=0x5642f6af4170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:22.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.152+0000 7f8671dd0740 1 --2- 172.21.15.33:0/2905854174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642f6af87d0 0x5642f6af4170 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f8660006010 tx=0x7f8660035f50 comp rx=0 tx=0).stop 2024-03-19T02:54:22.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.152+0000 7f8671dd0740 1 -- 172.21.15.33:0/2905854174 shutdown_connections 2024-03-19T02:54:22.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.152+0000 7f8671dd0740 1 --2- 172.21.15.33:0/2905854174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642f6af87d0 0x5642f6af4170 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.152+0000 7f8671dd0740 1 --2- 172.21.15.33:0/2905854174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642f6aefbf0 0x5642f6b108c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.152+0000 7f8671dd0740 1 --2- 172.21.15.33:0/2905854174 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642f6aef2d0 0x5642f6b04360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.152+0000 7f8671dd0740 1 -- 172.21.15.33:0/2905854174 >> 172.21.15.33:0/2905854174 conn(0x5642f6ae9a00 msgr2=0x5642f6aeffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:22.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.153+0000 7f8671dd0740 1 -- 172.21.15.33:0/2905854174 shutdown_connections 2024-03-19T02:54:22.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.153+0000 7f8671dd0740 1 -- 172.21.15.33:0/2905854174 wait complete. 2024-03-19T02:54:22.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.153+0000 7f8671dd0740 1 -- start start 2024-03-19T02:54:22.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.153+0000 7f8671dd0740 1 Processor -- start 2024-03-19T02:54:22.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.153+0000 7f8671dd0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642f6aef2d0 0x5642f6b67260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.153+0000 7f8671dd0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642f6aefbf0 0x5642f6b677a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.153+0000 7f8671dd0740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642f6af87d0 0x5642f6afbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.153+0000 7f8671dd0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f86681ad1b0 con 0x5642f6aefbf0 2024-03-19T02:54:22.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.153+0000 7f8671dd0740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f86681ae620 con 0x5642f6af87d0 2024-03-19T02:54:22.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.153+0000 7f8671dd0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f86681ad880 con 0x5642f6aef2d0 2024-03-19T02:54:22.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.154+0000 7f8671dd0740 10 client.0 _ll_register_callbacks cb 0x7f8654009d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:54:22.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.154+0000 7f8667fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642f6aefbf0 0x5642f6b677a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.154+0000 7f866d6b0640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642f6af87d0 0x5642f6afbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.154+0000 7f8667fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642f6aefbf0 0x5642f6b677a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37276/0 (socket says 172.21.15.33:37276) 2024-03-19T02:54:22.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.154+0000 7f8667fff640 1 -- 172.21.15.33:0/4049813788 learned_addr learned my addr 172.21.15.33:0/4049813788 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:22.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.154+0000 7f866ceaf640 1 --2- 172.21.15.33:0/4049813788 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642f6aef2d0 0x5642f6b67260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.154+0000 7f8667fff640 1 -- 172.21.15.33:0/4049813788 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642f6aef2d0 msgr2=0x5642f6b67260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:22.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.154+0000 7f8667fff640 1 --2- 172.21.15.33:0/4049813788 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642f6aef2d0 0x5642f6b67260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.154+0000 7f8667fff640 1 -- 172.21.15.33:0/4049813788 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642f6af87d0 msgr2=0x5642f6afbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:22.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.154+0000 7f8667fff640 1 --2- 172.21.15.33:0/4049813788 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642f6af87d0 0x5642f6afbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.154+0000 7f8667fff640 1 -- 172.21.15.33:0/4049813788 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642f6afc1a0 con 0x5642f6aefbf0 2024-03-19T02:54:22.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.154+0000 7f866d6b0640 1 --2- 172.21.15.33:0/4049813788 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642f6af87d0 0x5642f6afbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:22.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.154+0000 7f8667fff640 1 --2- 172.21.15.33:0/4049813788 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642f6aefbf0 0x5642f6b677a0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f8654009aa0 tx=0x7f8654003fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc49d494f5232485 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:22.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.154+0000 7f8647fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:54:22.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.154+0000 7f8647fff640 1 -- 172.21.15.33:0/4049813788 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865401a070 con 0x5642f6aefbf0 2024-03-19T02:54:22.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.154+0000 7f8647fff640 1 -- 172.21.15.33:0/4049813788 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f86540288f0 con 0x5642f6aefbf0 2024-03-19T02:54:22.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.154+0000 7f8671dd0740 1 -- 172.21.15.33:0/4049813788 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5642f6a5f9f0 con 0x5642f6aefbf0 2024-03-19T02:54:22.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.155+0000 7f8647fff640 1 -- 172.21.15.33:0/4049813788 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8654004350 con 0x5642f6aefbf0 2024-03-19T02:54:22.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.155+0000 7f8671dd0740 10 client.15855 fetch_fsmap learned FSMap version 25 2024-03-19T02:54:22.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.155+0000 7f8671dd0740 1 -- 172.21.15.33:0/4049813788 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5642f6b6e970 con 0x5642f6aefbf0 2024-03-19T02:54:22.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.155+0000 7f8647fff640 1 -- 172.21.15.33:0/4049813788 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5642f6a5f9f0 con 0x5642f6aefbf0 2024-03-19T02:54:22.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.155+0000 7f8647fff640 1 -- 172.21.15.33:0/4049813788 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f865401d070 con 0x5642f6aefbf0 2024-03-19T02:54:22.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.155+0000 7f8671dd0740 10 client.15855 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:54:22.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.155+0000 7f8671dd0740 10 client.15855 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T02:54:22.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.155+0000 7f8671dd0740 20 client.15855 populate_metadata read hostname 'smithi033' 2024-03-19T02:54:22.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.155+0000 7f8671dd0740 1 --2- 172.21.15.33:0/4049813788 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x5642f6b6f020 0x5642f6b71440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.155+0000 7f8671dd0740 4 client.15855 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:54:22.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.155+0000 7f8671dd0740 1 -- 172.21.15.33:0/4049813788 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5642f6a59d40 con 0x5642f6b6f020 2024-03-19T02:54:22.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.157+0000 7f866ceaf640 1 --2- 172.21.15.33:0/4049813788 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x5642f6b6f020 0x5642f6b71440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.157+0000 7f8647fff640 10 client.15855 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T02:54:22.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.157+0000 7f866ceaf640 1 --2- 172.21.15.33:0/4049813788 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x5642f6b6f020 0x5642f6b71440 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9029adc47ffa27b8 server_cookie=c3871e7e29aca379 in_seq=0 out_seq=0 2024-03-19T02:54:22.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.157+0000 7f8647fff640 1 -- 172.21.15.33:0/4049813788 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5642f6a59d40 con 0x5642f6b6f020 2024-03-19T02:54:22.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.157+0000 7f8647fff640 10 client.15855 handle_command_reply: tid=0 2024-03-19T02:54:22.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.157+0000 7f8671dd0740 1 client.15855 shutdown 2024-03-19T02:54:22.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.157+0000 7f8671dd0740 1 -- 172.21.15.33:0/4049813788 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642f6aefbf0 msgr2=0x5642f6b677a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:22.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.157+0000 7f8671dd0740 1 --2- 172.21.15.33:0/4049813788 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642f6aefbf0 0x5642f6b677a0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f8654009aa0 tx=0x7f8654003fd0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.157+0000 7f8671dd0740 1 -- 172.21.15.33:0/4049813788 shutdown_connections 2024-03-19T02:54:22.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.157+0000 7f8671dd0740 1 --2- 172.21.15.33:0/4049813788 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x5642f6b6f020 0x5642f6b71440 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.157+0000 7f8671dd0740 1 --2- 172.21.15.33:0/4049813788 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642f6af87d0 0x5642f6afbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.157+0000 7f8671dd0740 1 --2- 172.21.15.33:0/4049813788 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642f6aefbf0 0x5642f6b677a0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.157+0000 7f8647fff640 0 client.15855 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T02:54:22.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.157+0000 7f8671dd0740 1 --2- 172.21.15.33:0/4049813788 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642f6aef2d0 0x5642f6b67260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.157+0000 7f8671dd0740 1 -- 172.21.15.33:0/4049813788 >> 172.21.15.33:0/4049813788 conn(0x5642f6ae9a00 msgr2=0x5642f6af40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:22.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.158+0000 7f8671dd0740 1 -- 172.21.15.33:0/4049813788 shutdown_connections 2024-03-19T02:54:22.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.158+0000 7f8671dd0740 1 -- 172.21.15.33:0/4049813788 wait complete. 2024-03-19T02:54:22.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.158+0000 7f8671dd0740 20 client.15855 trim_cache size 0 max 16384 2024-03-19T02:54:22.163 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:54:22.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.158+0000 7f866f113640 1 -- 172.21.15.33:0/2809011725 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f86580a2a90 msgr2=0x7f86580a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:22.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.158+0000 7f866f113640 1 --2- 172.21.15.33:0/2809011725 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f86580a2a90 0x7f86580a4f50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f86680694f0 tx=0x7f8654011920 comp rx=0 tx=0).stop 2024-03-19T02:54:22.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.158+0000 7f866f113640 1 -- 172.21.15.33:0/2809011725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f86681066b0 msgr2=0x7f86681a3e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:22.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.158+0000 7f866f113640 1 --2- 172.21.15.33:0/2809011725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f86681066b0 0x7f86681a3e70 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f8660004af0 tx=0x7f8660035ce0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.158+0000 7f866f113640 1 -- 172.21.15.33:0/2809011725 shutdown_connections 2024-03-19T02:54:22.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.158+0000 7f866f113640 1 --2- 172.21.15.33:0/2809011725 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f86580a2a90 0x7f86580a4f50 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.158+0000 7f866f113640 1 --2- 172.21.15.33:0/2809011725 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8668101060 0x7f8668199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.158+0000 7f866f113640 1 --2- 172.21.15.33:0/2809011725 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f86681066b0 0x7f86681a3e70 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.158+0000 7f866f113640 1 --2- 172.21.15.33:0/2809011725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f86681006a0 0x7f8668198df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.158+0000 7f866f113640 1 -- 172.21.15.33:0/2809011725 >> 172.21.15.33:0/2809011725 conn(0x7f86680778d0 msgr2=0x7f8668102df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:22.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.158+0000 7f866f113640 1 -- 172.21.15.33:0/2809011725 shutdown_connections 2024-03-19T02:54:22.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.159+0000 7f866f113640 1 -- 172.21.15.33:0/2809011725 wait complete. 2024-03-19T02:54:22.171 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:54:22.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.294+0000 7f8355174640 1 -- 172.21.15.33:0/4177767162 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8350101060 msgr2=0x7f835010d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:22.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.294+0000 7f8355174640 1 --2- 172.21.15.33:0/4177767162 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8350101060 0x7f835010d570 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f8348009cd0 tx=0x7f834802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.294+0000 7f8355174640 1 -- 172.21.15.33:0/4177767162 shutdown_connections 2024-03-19T02:54:22.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.294+0000 7f8355174640 1 --2- 172.21.15.33:0/4177767162 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f83501006a0 0x7f8350100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.294+0000 7f8355174640 1 --2- 172.21.15.33:0/4177767162 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8350101060 0x7f835010d570 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.294+0000 7f8355174640 1 --2- 172.21.15.33:0/4177767162 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83501066b0 0x7f8350106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.294+0000 7f8355174640 1 -- 172.21.15.33:0/4177767162 >> 172.21.15.33:0/4177767162 conn(0x7f83500778d0 msgr2=0x7f8350077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:22.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.294+0000 7f8355174640 1 -- 172.21.15.33:0/4177767162 shutdown_connections 2024-03-19T02:54:22.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.294+0000 7f8355174640 1 -- 172.21.15.33:0/4177767162 wait complete. 2024-03-19T02:54:22.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.295+0000 7f8355174640 1 Processor -- start 2024-03-19T02:54:22.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.295+0000 7f8355174640 1 -- start start 2024-03-19T02:54:22.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.295+0000 7f8355174640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83501006a0 0x7f8350197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.295+0000 7f8355174640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8350101060 0x7f8350198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.295+0000 7f8355174640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f83501066b0 0x7f835019c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.295+0000 7f8355174640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8350111a70 con 0x7f8350101060 2024-03-19T02:54:22.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.295+0000 7f8355174640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f83501118f0 con 0x7f83501006a0 2024-03-19T02:54:22.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.295+0000 7f8355174640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8350111bf0 con 0x7f83501066b0 2024-03-19T02:54:22.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.295+0000 7f8347fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8350101060 0x7f8350198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.295+0000 7f834ed9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83501006a0 0x7f8350197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.295+0000 7f834ed9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83501006a0 0x7f8350197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:48742/0 (socket says 172.21.15.33:48742) 2024-03-19T02:54:22.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.295+0000 7f8347fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8350101060 0x7f8350198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37292/0 (socket says 172.21.15.33:37292) 2024-03-19T02:54:22.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.295+0000 7f834ed9d640 1 -- 172.21.15.33:0/403470374 learned_addr learned my addr 172.21.15.33:0/403470374 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:22.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.295+0000 7f834f59e640 1 --2- 172.21.15.33:0/403470374 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f83501066b0 0x7f835019c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.296+0000 7f834ed9d640 1 -- 172.21.15.33:0/403470374 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f83501066b0 msgr2=0x7f835019c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:22.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.296+0000 7f834ed9d640 1 --2- 172.21.15.33:0/403470374 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f83501066b0 0x7f835019c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.296+0000 7f834ed9d640 1 -- 172.21.15.33:0/403470374 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8350101060 msgr2=0x7f8350198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:22.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.296+0000 7f834ed9d640 1 --2- 172.21.15.33:0/403470374 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8350101060 0x7f8350198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.296+0000 7f834ed9d640 1 -- 172.21.15.33:0/403470374 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f835019cf40 con 0x7f83501006a0 2024-03-19T02:54:22.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.296+0000 7f834ed9d640 1 --2- 172.21.15.33:0/403470374 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83501006a0 0x7f8350197f50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f834000e9f0 tx=0x7f834000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:22.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.296+0000 7f834cd99640 1 -- 172.21.15.33:0/403470374 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834000cd80 con 0x7f83501006a0 2024-03-19T02:54:22.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.296+0000 7f8355174640 1 -- 172.21.15.33:0/403470374 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f835019d230 con 0x7f83501006a0 2024-03-19T02:54:22.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.296+0000 7f834cd99640 1 -- 172.21.15.33:0/403470374 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f83400108e0 con 0x7f83501006a0 2024-03-19T02:54:22.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.296+0000 7f834cd99640 1 -- 172.21.15.33:0/403470374 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8340018920 con 0x7f83501006a0 2024-03-19T02:54:22.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.296+0000 7f8355174640 1 -- 172.21.15.33:0/403470374 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83501a4ac0 con 0x7f83501006a0 2024-03-19T02:54:22.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.298+0000 7f834cd99640 1 -- 172.21.15.33:0/403470374 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7f8340010a90 con 0x7f83501006a0 2024-03-19T02:54:22.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.299+0000 7f8357e37740 1 Processor -- start 2024-03-19T02:54:22.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.299+0000 7f8357e37740 1 -- start start 2024-03-19T02:54:22.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.299+0000 7f8357e37740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560b2df982d0 0x560b2df986b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.299+0000 7f834cd99640 1 --2- 172.21.15.33:0/403470374 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f833c0a2a90 0x7f833c0a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.299+0000 7f8357e37740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560b2df98bf0 0x560b2dfa1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.299+0000 7f8357e37740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b2dfa17d0 0x560b2dfa3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.299+0000 7f8357e37740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560b2defb4b0 con 0x560b2df982d0 2024-03-19T02:54:22.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.299+0000 7f8357e37740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x560b2dea3d70 con 0x560b2dfa17d0 2024-03-19T02:54:22.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.299+0000 7f8357e37740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560b2def7550 con 0x560b2df98bf0 2024-03-19T02:54:22.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.299+0000 7f834ed9d640 1 --2- 172.21.15.33:0/403470374 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f833c0a2a90 0x7f833c0a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.299+0000 7f834f59e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560b2df982d0 0x560b2df986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.299+0000 7f8347fff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560b2df98bf0 0x560b2dfa1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.299+0000 7f834f59e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560b2df982d0 0x560b2df986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37304/0 (socket says 172.21.15.33:37304) 2024-03-19T02:54:22.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.299+0000 7f834f59e640 1 -- 172.21.15.33:0/3072548384 learned_addr learned my addr 172.21.15.33:0/3072548384 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:22.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.299+0000 7f834f59e640 1 --2- 172.21.15.33:0/3072548384 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b2dfa17d0 0x560b2dfa3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.299+0000 7f834cd99640 1 -- 172.21.15.33:0/403470374 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f83400cf740 con 0x7f83501006a0 2024-03-19T02:54:22.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.300+0000 7f8347fff640 1 -- 172.21.15.33:0/3072548384 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b2dfa17d0 msgr2=0x560b2dfa3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:22.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.300+0000 7f8347fff640 1 --2- 172.21.15.33:0/3072548384 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b2dfa17d0 0x560b2dfa3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.300+0000 7f8347fff640 1 -- 172.21.15.33:0/3072548384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560b2df982d0 msgr2=0x560b2df986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:22.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.300+0000 7f8347fff640 1 --2- 172.21.15.33:0/3072548384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560b2df982d0 0x560b2df986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.300+0000 7f8347fff640 1 -- 172.21.15.33:0/3072548384 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b2dec6210 con 0x560b2df98bf0 2024-03-19T02:54:22.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.300+0000 7f834f59e640 1 --2- 172.21.15.33:0/3072548384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560b2df982d0 0x560b2df986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:22.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.300+0000 7f834ed9d640 1 --2- 172.21.15.33:0/403470374 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f833c0a2a90 0x7f833c0a4f50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f83400cca00 tx=0x7f83400242e0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:22.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.300+0000 7f8347fff640 1 --2- 172.21.15.33:0/3072548384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560b2df98bf0 0x560b2dfa1290 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8338009bd0 tx=0x7f8338011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2ae5b1cdf09f873 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:22.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.300+0000 7f83457fa640 1 -- 172.21.15.33:0/3072548384 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8338004230 con 0x560b2df98bf0 2024-03-19T02:54:22.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.300+0000 7f83457fa640 1 -- 172.21.15.33:0/3072548384 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8338009e40 con 0x560b2df98bf0 2024-03-19T02:54:22.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.300+0000 7f83457fa640 1 -- 172.21.15.33:0/3072548384 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8338004bf0 con 0x560b2df98bf0 2024-03-19T02:54:22.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.301+0000 7f8357e37740 1 -- 172.21.15.33:0/3072548384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560b2df98bf0 msgr2=0x560b2dfa1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:22.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.301+0000 7f8357e37740 1 --2- 172.21.15.33:0/3072548384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560b2df98bf0 0x560b2dfa1290 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8338009bd0 tx=0x7f8338011810 comp rx=0 tx=0).stop 2024-03-19T02:54:22.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.301+0000 7f8357e37740 1 -- 172.21.15.33:0/3072548384 shutdown_connections 2024-03-19T02:54:22.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.301+0000 7f8357e37740 1 --2- 172.21.15.33:0/3072548384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560b2df98bf0 0x560b2dfa1290 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.301+0000 7f8357e37740 1 --2- 172.21.15.33:0/3072548384 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b2dfa17d0 0x560b2dfa3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.301+0000 7f8357e37740 1 --2- 172.21.15.33:0/3072548384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560b2df982d0 0x560b2df986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.301+0000 7f8357e37740 1 -- 172.21.15.33:0/3072548384 >> 172.21.15.33:0/3072548384 conn(0x560b2df92a00 msgr2=0x560b2df92de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:22.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.301+0000 7f8357e37740 1 -- 172.21.15.33:0/3072548384 shutdown_connections 2024-03-19T02:54:22.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.301+0000 7f8357e37740 1 -- 172.21.15.33:0/3072548384 wait complete. 2024-03-19T02:54:22.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.301+0000 7f8357e37740 1 -- start start 2024-03-19T02:54:22.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.301+0000 7f8357e37740 1 Processor -- start 2024-03-19T02:54:22.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.302+0000 7f8357e37740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b2df982d0 0x560b2df9e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.302+0000 7f8357e37740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560b2df98bf0 0x560b2df9ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.302+0000 7f8357e37740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560b2dfa17d0 0x560b2df9f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.302+0000 7f8357e37740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560b2dfa45c0 con 0x560b2df98bf0 2024-03-19T02:54:22.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.302+0000 7f8357e37740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x560b2deb7310 con 0x560b2df982d0 2024-03-19T02:54:22.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.302+0000 7f8357e37740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560b2dfa4740 con 0x560b2dfa17d0 2024-03-19T02:54:22.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.302+0000 7f8357e37740 10 client.0 _ll_register_callbacks cb 0x560b2deab2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:54:22.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.302+0000 7f8347fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560b2df98bf0 0x560b2df9ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.302+0000 7f834f59e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b2df982d0 0x560b2df9e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.302+0000 7f834f59e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b2df982d0 0x560b2df9e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:48754/0 (socket says 172.21.15.33:48754) 2024-03-19T02:54:22.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.302+0000 7f834f59e640 1 -- 172.21.15.33:0/158214072 learned_addr learned my addr 172.21.15.33:0/158214072 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:22.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.302+0000 7f834f59e640 1 --2- 172.21.15.33:0/158214072 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560b2dfa17d0 0x560b2df9f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.302+0000 7f8347fff640 1 -- 172.21.15.33:0/158214072 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560b2dfa17d0 msgr2=0x560b2df9f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:22.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.302+0000 7f8347fff640 1 --2- 172.21.15.33:0/158214072 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560b2dfa17d0 0x560b2df9f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.302+0000 7f8347fff640 1 -- 172.21.15.33:0/158214072 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b2df982d0 msgr2=0x560b2df9e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:22.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.302+0000 7f8347fff640 1 --2- 172.21.15.33:0/158214072 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b2df982d0 0x560b2df9e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.302+0000 7f8347fff640 1 -- 172.21.15.33:0/158214072 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b2dc0f3f0 con 0x560b2df98bf0 2024-03-19T02:54:22.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.302+0000 7f8347fff640 1 --2- 172.21.15.33:0/158214072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560b2df98bf0 0x560b2df9ecb0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f8338005890 tx=0x7f83380038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abce86d2d8b3af6a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:22.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.302+0000 7f83367fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:54:22.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.303+0000 7f83367fc640 1 -- 172.21.15.33:0/158214072 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8338027020 con 0x560b2df98bf0 2024-03-19T02:54:22.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.303+0000 7f83367fc640 1 -- 172.21.15.33:0/158214072 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f8338003c90 con 0x560b2df98bf0 2024-03-19T02:54:22.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.303+0000 7f8357e37740 1 -- 172.21.15.33:0/158214072 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560b2df593e0 con 0x560b2df98bf0 2024-03-19T02:54:22.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.303+0000 7f83367fc640 1 -- 172.21.15.33:0/158214072 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8338032050 con 0x560b2df98bf0 2024-03-19T02:54:22.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.303+0000 7f8357e37740 10 client.15867 fetch_fsmap learned FSMap version 25 2024-03-19T02:54:22.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.303+0000 7f8357e37740 1 -- 172.21.15.33:0/158214072 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x560b2dcd97d0 con 0x560b2df98bf0 2024-03-19T02:54:22.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.303+0000 7f83367fc640 1 -- 172.21.15.33:0/158214072 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x560b2df593e0 con 0x560b2df98bf0 2024-03-19T02:54:22.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.303+0000 7f83367fc640 1 -- 172.21.15.33:0/158214072 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f8338018070 con 0x560b2df98bf0 2024-03-19T02:54:22.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.303+0000 7f8357e37740 10 client.15867 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:54:22.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.303+0000 7f8357e37740 10 client.15867 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T02:54:22.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.303+0000 7f8357e37740 20 client.15867 populate_metadata read hostname 'smithi033' 2024-03-19T02:54:22.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.303+0000 7f8357e37740 1 --2- 172.21.15.33:0/158214072 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x560b2dfb1a50 0x560b2dfb3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.303+0000 7f8357e37740 4 client.15867 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:54:22.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.303+0000 7f8357e37740 1 -- 172.21.15.33:0/158214072 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560b2df089f0 con 0x560b2dfb1a50 2024-03-19T02:54:22.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.304+0000 7f834f59e640 1 --2- 172.21.15.33:0/158214072 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x560b2dfb1a50 0x560b2dfb3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.304+0000 7f83367fc640 10 client.15867 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T02:54:22.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.304+0000 7f834f59e640 1 --2- 172.21.15.33:0/158214072 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x560b2dfb1a50 0x560b2dfb3e40 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9a54bf05fa45091 server_cookie=d476134b5604a0bf in_seq=0 out_seq=0 2024-03-19T02:54:22.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.305+0000 7f83367fc640 1 -- 172.21.15.33:0/158214072 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560b2df089f0 con 0x560b2dfb1a50 2024-03-19T02:54:22.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.305+0000 7f83367fc640 10 client.15867 handle_command_reply: tid=0 2024-03-19T02:54:22.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.305+0000 7f8357e37740 1 client.15867 shutdown 2024-03-19T02:54:22.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.305+0000 7f8357e37740 1 -- 172.21.15.33:0/158214072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560b2df98bf0 msgr2=0x560b2df9ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:22.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.305+0000 7f8357e37740 1 --2- 172.21.15.33:0/158214072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560b2df98bf0 0x560b2df9ecb0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f8338005890 tx=0x7f83380038a0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.305+0000 7f8357e37740 1 -- 172.21.15.33:0/158214072 shutdown_connections 2024-03-19T02:54:22.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.305+0000 7f8357e37740 1 --2- 172.21.15.33:0/158214072 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x560b2dfb1a50 0x560b2dfb3e40 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.305+0000 7f8357e37740 1 --2- 172.21.15.33:0/158214072 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560b2dfa17d0 0x560b2df9f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.305+0000 7f8357e37740 1 --2- 172.21.15.33:0/158214072 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560b2df98bf0 0x560b2df9ecb0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.305+0000 7f83367fc640 0 client.15867 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T02:54:22.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.305+0000 7f8357e37740 1 --2- 172.21.15.33:0/158214072 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b2df982d0 0x560b2df9e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.305+0000 7f8357e37740 1 -- 172.21.15.33:0/158214072 >> 172.21.15.33:0/158214072 conn(0x560b2df92a00 msgr2=0x560b2df94750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:22.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.305+0000 7f8357e37740 1 -- 172.21.15.33:0/158214072 shutdown_connections 2024-03-19T02:54:22.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.305+0000 7f8357e37740 1 -- 172.21.15.33:0/158214072 wait complete. 2024-03-19T02:54:22.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.305+0000 7f8357e37740 20 client.15867 trim_cache size 0 max 16384 2024-03-19T02:54:22.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.325+0000 7f8357e37740 1 Processor -- start 2024-03-19T02:54:22.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.325+0000 7f8357e37740 1 -- start start 2024-03-19T02:54:22.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.325+0000 7f8357e37740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b2df982d0 0x560b2dfa0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.325+0000 7f8357e37740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560b2df98bf0 0x560b2dfad360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.325+0000 7f8357e37740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560b2dfa17d0 0x560b2df91ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.325+0000 7f8357e37740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f83501b1460 con 0x560b2dfa17d0 2024-03-19T02:54:22.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.325+0000 7f8357e37740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f83501aed30 con 0x560b2df982d0 2024-03-19T02:54:22.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.325+0000 7f8357e37740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f83501acb60 con 0x560b2df98bf0 2024-03-19T02:54:22.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.325+0000 7f8347fff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560b2df98bf0 0x560b2dfad360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.325+0000 7f834f59e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b2df982d0 0x560b2dfa0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.325+0000 7f8347fff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560b2df98bf0 0x560b2dfad360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:39596/0 (socket says 172.21.15.33:39596) 2024-03-19T02:54:22.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.325+0000 7f8347fff640 1 -- 172.21.15.33:0/105012564 learned_addr learned my addr 172.21.15.33:0/105012564 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:22.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.325+0000 7f834f59e640 1 --2- 172.21.15.33:0/105012564 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560b2dfa17d0 0x560b2df91ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.326+0000 7f8347fff640 1 -- 172.21.15.33:0/105012564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b2df982d0 msgr2=0x560b2dfa0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:22.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.326+0000 7f8347fff640 1 --2- 172.21.15.33:0/105012564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b2df982d0 0x560b2dfa0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.326+0000 7f8347fff640 1 -- 172.21.15.33:0/105012564 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560b2dfa17d0 msgr2=0x560b2df91ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:22.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.326+0000 7f8347fff640 1 --2- 172.21.15.33:0/105012564 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560b2dfa17d0 0x560b2df91ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.326+0000 7f8347fff640 1 -- 172.21.15.33:0/105012564 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b2df9d760 con 0x560b2df98bf0 2024-03-19T02:54:22.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.326+0000 7f8347fff640 1 --2- 172.21.15.33:0/105012564 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560b2df98bf0 0x560b2dfad360 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8338023cf0 tx=0x7f83380193f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ead4cec6c7571be4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:22.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.326+0000 7f8344ff9640 1 -- 172.21.15.33:0/105012564 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8338027040 con 0x560b2df98bf0 2024-03-19T02:54:22.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.326+0000 7f8344ff9640 1 -- 172.21.15.33:0/105012564 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f83380238a0 con 0x560b2df98bf0 2024-03-19T02:54:22.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.326+0000 7f8344ff9640 1 -- 172.21.15.33:0/105012564 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833801e040 con 0x560b2df98bf0 2024-03-19T02:54:22.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.327+0000 7f8357e37740 1 -- 172.21.15.33:0/105012564 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560b2df98bf0 msgr2=0x560b2dfad360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:22.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.327+0000 7f8357e37740 1 --2- 172.21.15.33:0/105012564 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560b2df98bf0 0x560b2dfad360 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8338023cf0 tx=0x7f83380193f0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.327+0000 7f8357e37740 1 -- 172.21.15.33:0/105012564 shutdown_connections 2024-03-19T02:54:22.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.327+0000 7f8357e37740 1 --2- 172.21.15.33:0/105012564 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560b2df98bf0 0x560b2dfad360 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.327+0000 7f8357e37740 1 --2- 172.21.15.33:0/105012564 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560b2dfa17d0 0x560b2df91ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.327+0000 7f8357e37740 1 --2- 172.21.15.33:0/105012564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b2df982d0 0x560b2dfa0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.327+0000 7f8357e37740 1 -- 172.21.15.33:0/105012564 >> 172.21.15.33:0/105012564 conn(0x560b2df92a00 msgr2=0x560b2df92de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:22.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.327+0000 7f8357e37740 1 -- 172.21.15.33:0/105012564 shutdown_connections 2024-03-19T02:54:22.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.327+0000 7f8357e37740 1 -- 172.21.15.33:0/105012564 wait complete. 2024-03-19T02:54:22.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.327+0000 7f8357e37740 1 -- start start 2024-03-19T02:54:22.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.327+0000 7f8357e37740 1 Processor -- start 2024-03-19T02:54:22.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.328+0000 7f8357e37740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560b2df982d0 0x560b2df9b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.328+0000 7f8357e37740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560b2df98bf0 0x560b2df9b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.328+0000 7f8357e37740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b2dfa17d0 0x560b2df9be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.328+0000 7f8357e37740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f83501ad1f0 con 0x560b2df982d0 2024-03-19T02:54:22.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.328+0000 7f8357e37740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f83501ae660 con 0x560b2dfa17d0 2024-03-19T02:54:22.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.328+0000 7f8357e37740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f83501ad8c0 con 0x560b2df98bf0 2024-03-19T02:54:22.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.328+0000 7f8357e37740 10 client.0 _ll_register_callbacks cb 0x7f8340005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:54:22.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.328+0000 7f834f59e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560b2df982d0 0x560b2df9b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.328+0000 7f834f59e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560b2df982d0 0x560b2df9b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37324/0 (socket says 172.21.15.33:37324) 2024-03-19T02:54:22.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.328+0000 7f8347fff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560b2df98bf0 0x560b2df9b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.328+0000 7f834f59e640 1 -- 172.21.15.33:0/3221528752 learned_addr learned my addr 172.21.15.33:0/3221528752 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:22.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.328+0000 7f834f59e640 1 --2- 172.21.15.33:0/3221528752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b2dfa17d0 0x560b2df9be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.328+0000 7f8347fff640 1 -- 172.21.15.33:0/3221528752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b2dfa17d0 msgr2=0x560b2df9be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:22.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.328+0000 7f8347fff640 1 --2- 172.21.15.33:0/3221528752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b2dfa17d0 0x560b2df9be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.328+0000 7f8347fff640 1 -- 172.21.15.33:0/3221528752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560b2df982d0 msgr2=0x560b2df9b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:22.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.328+0000 7f8347fff640 1 --2- 172.21.15.33:0/3221528752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560b2df982d0 0x560b2df9b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.328+0000 7f8347fff640 1 -- 172.21.15.33:0/3221528752 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b2df995a0 con 0x560b2df98bf0 2024-03-19T02:54:22.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.328+0000 7f8347fff640 1 --2- 172.21.15.33:0/3221528752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560b2df98bf0 0x560b2df9b8c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x560b2df8daa0 tx=0x7f8338009ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a8e67f36c5e008e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:22.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.328+0000 7f83367fc640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-19T02:54:22.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.328+0000 7f83367fc640 1 -- 172.21.15.33:0/3221528752 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8338027040 con 0x560b2df98bf0 2024-03-19T02:54:22.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.328+0000 7f8357e37740 1 -- 172.21.15.33:0/3221528752 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83380238a0 con 0x560b2df98bf0 2024-03-19T02:54:22.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.328+0000 7f83367fc640 1 -- 172.21.15.33:0/3221528752 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f83380233f0 con 0x560b2df98bf0 2024-03-19T02:54:22.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.329+0000 7f83367fc640 1 -- 172.21.15.33:0/3221528752 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8338045050 con 0x560b2df98bf0 2024-03-19T02:54:22.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.329+0000 7f8357e37740 10 client.25618 fetch_fsmap learned FSMap version 25 2024-03-19T02:54:22.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.329+0000 7f8357e37740 1 -- 172.21.15.33:0/3221528752 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x560b2df98a20 con 0x560b2df98bf0 2024-03-19T02:54:22.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.329+0000 7f83367fc640 1 -- 172.21.15.33:0/3221528752 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f83380238a0 con 0x560b2df98bf0 2024-03-19T02:54:22.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.329+0000 7f83367fc640 1 -- 172.21.15.33:0/3221528752 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f8338018070 con 0x560b2df98bf0 2024-03-19T02:54:22.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.329+0000 7f8357e37740 10 client.25618 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:54:22.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.329+0000 7f8357e37740 10 client.25618 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T02:54:22.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.329+0000 7f8357e37740 20 client.25618 populate_metadata read hostname 'smithi033' 2024-03-19T02:54:22.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.329+0000 7f8357e37740 1 --2- 172.21.15.33:0/3221528752 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x560b2e018280 0x560b2e01a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.329+0000 7f8357e37740 4 client.25618 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:54:22.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.329+0000 7f8357e37740 1 -- 172.21.15.33:0/3221528752 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560b2df02d40 con 0x560b2e018280 2024-03-19T02:54:22.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.329+0000 7f834f59e640 1 --2- 172.21.15.33:0/3221528752 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x560b2e018280 0x560b2e01a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.330+0000 7f83367fc640 10 client.25618 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T02:54:22.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.330+0000 7f834f59e640 1 --2- 172.21.15.33:0/3221528752 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x560b2e018280 0x560b2e01a6a0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=49a3bff994937cc1 server_cookie=c25637024b15c7c in_seq=0 out_seq=0 2024-03-19T02:54:22.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.330+0000 7f83367fc640 1 -- 172.21.15.33:0/3221528752 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560b2df02d40 con 0x560b2e018280 2024-03-19T02:54:22.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.330+0000 7f83367fc640 10 client.25618 handle_command_reply: tid=0 2024-03-19T02:54:22.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.330+0000 7f8357e37740 1 client.25618 shutdown 2024-03-19T02:54:22.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.330+0000 7f8357e37740 1 -- 172.21.15.33:0/3221528752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560b2df98bf0 msgr2=0x560b2df9b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:22.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.330+0000 7f8357e37740 1 --2- 172.21.15.33:0/3221528752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560b2df98bf0 0x560b2df9b8c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x560b2df8daa0 tx=0x7f8338009ac0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.330+0000 7f8357e37740 1 -- 172.21.15.33:0/3221528752 shutdown_connections 2024-03-19T02:54:22.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.330+0000 7f8357e37740 1 --2- 172.21.15.33:0/3221528752 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x560b2e018280 0x560b2e01a6a0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.330+0000 7f8357e37740 1 --2- 172.21.15.33:0/3221528752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560b2df98bf0 0x560b2df9b8c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.330+0000 7f83367fc640 0 client.25618 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T02:54:22.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.330+0000 7f8357e37740 1 --2- 172.21.15.33:0/3221528752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560b2dfa17d0 0x560b2df9be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.330+0000 7f8357e37740 1 --2- 172.21.15.33:0/3221528752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560b2df982d0 0x560b2df9b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.330+0000 7f8357e37740 1 -- 172.21.15.33:0/3221528752 >> 172.21.15.33:0/3221528752 conn(0x560b2df92a00 msgr2=0x560b2df945e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:22.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.330+0000 7f8357e37740 1 -- 172.21.15.33:0/3221528752 shutdown_connections 2024-03-19T02:54:22.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.330+0000 7f8357e37740 1 -- 172.21.15.33:0/3221528752 wait complete. 2024-03-19T02:54:22.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.330+0000 7f8357e37740 20 client.25618 trim_cache size 0 max 16384 2024-03-19T02:54:22.339 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:54:22.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.331+0000 7f8355174640 1 -- 172.21.15.33:0/403470374 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f833c0a2a90 msgr2=0x7f833c0a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:22.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.331+0000 7f8355174640 1 --2- 172.21.15.33:0/403470374 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f833c0a2a90 0x7f833c0a4f50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f83400cca00 tx=0x7f83400242e0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.331+0000 7f8355174640 1 -- 172.21.15.33:0/403470374 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83501006a0 msgr2=0x7f8350197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:22.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.331+0000 7f8355174640 1 --2- 172.21.15.33:0/403470374 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83501006a0 0x7f8350197f50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f834000e9f0 tx=0x7f834000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.331+0000 7f8355174640 1 -- 172.21.15.33:0/403470374 shutdown_connections 2024-03-19T02:54:22.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.331+0000 7f8355174640 1 --2- 172.21.15.33:0/403470374 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f833c0a2a90 0x7f833c0a4f50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.331+0000 7f8355174640 1 --2- 172.21.15.33:0/403470374 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f83501066b0 0x7f835019c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.331+0000 7f8355174640 1 --2- 172.21.15.33:0/403470374 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8350101060 0x7f8350198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.331+0000 7f8355174640 1 --2- 172.21.15.33:0/403470374 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f83501006a0 0x7f8350197f50 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.331+0000 7f8355174640 1 -- 172.21.15.33:0/403470374 >> 172.21.15.33:0/403470374 conn(0x7f83500778d0 msgr2=0x7f8350102ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:22.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.331+0000 7f8355174640 1 -- 172.21.15.33:0/403470374 shutdown_connections 2024-03-19T02:54:22.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.331+0000 7f8355174640 1 -- 172.21.15.33:0/403470374 wait complete. 2024-03-19T02:54:22.343 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:54:22.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.459+0000 7f44be6c6640 1 -- 172.21.15.33:0/3309681658 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44b8103360 msgr2=0x7f44b810b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:22.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.459+0000 7f44be6c6640 1 --2- 172.21.15.33:0/3309681658 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44b8103360 0x7f44b810b840 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f44ac009900 tx=0x7f44ac02bcd0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.459+0000 7f44be6c6640 1 -- 172.21.15.33:0/3309681658 shutdown_connections 2024-03-19T02:54:22.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.459+0000 7f44be6c6640 1 --2- 172.21.15.33:0/3309681658 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44b810bd80 0x7f44b810e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.459+0000 7f44be6c6640 1 --2- 172.21.15.33:0/3309681658 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44b8103360 0x7f44b810b840 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.459+0000 7f44be6c6640 1 --2- 172.21.15.33:0/3309681658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44b8102a40 0x7f44b8102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.459+0000 7f44be6c6640 1 -- 172.21.15.33:0/3309681658 >> 172.21.15.33:0/3309681658 conn(0x7f44b80778d0 msgr2=0x7f44b8077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:22.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.459+0000 7f44be6c6640 1 -- 172.21.15.33:0/3309681658 shutdown_connections 2024-03-19T02:54:22.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.459+0000 7f44be6c6640 1 -- 172.21.15.33:0/3309681658 wait complete. 2024-03-19T02:54:22.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.459+0000 7f44be6c6640 1 Processor -- start 2024-03-19T02:54:22.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.460+0000 7f44be6c6640 1 -- start start 2024-03-19T02:54:22.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.460+0000 7f44be6c6640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44b8102a40 0x7f44b8197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.460+0000 7f44be6c6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44b8103360 0x7f44b8198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.460+0000 7f44be6c6640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44b810bd80 0x7f44b819c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.460+0000 7f44be6c6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f44b8111a30 con 0x7f44b8103360 2024-03-19T02:54:22.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.460+0000 7f44be6c6640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f44b81118b0 con 0x7f44b8102a40 2024-03-19T02:54:22.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.460+0000 7f44be6c6640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f44b8111bb0 con 0x7f44b810bd80 2024-03-19T02:54:22.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.460+0000 7f44b77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44b8103360 0x7f44b8198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.460+0000 7f44b77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44b8103360 0x7f44b8198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37340/0 (socket says 172.21.15.33:37340) 2024-03-19T02:54:22.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.460+0000 7f44b77fe640 1 -- 172.21.15.33:0/2147544958 learned_addr learned my addr 172.21.15.33:0/2147544958 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:22.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.460+0000 7f44b7fff640 1 --2- 172.21.15.33:0/2147544958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44b8102a40 0x7f44b8197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.460+0000 7f44bcc63640 1 --2- 172.21.15.33:0/2147544958 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44b810bd80 0x7f44b819c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.460+0000 7f44b7fff640 1 -- 172.21.15.33:0/2147544958 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44b810bd80 msgr2=0x7f44b819c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:22.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.460+0000 7f44b7fff640 1 --2- 172.21.15.33:0/2147544958 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44b810bd80 0x7f44b819c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.460+0000 7f44b7fff640 1 -- 172.21.15.33:0/2147544958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44b8103360 msgr2=0x7f44b8198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:22.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.460+0000 7f44b7fff640 1 --2- 172.21.15.33:0/2147544958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44b8103360 0x7f44b8198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.460+0000 7f44b7fff640 1 -- 172.21.15.33:0/2147544958 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44b819cfc0 con 0x7f44b8102a40 2024-03-19T02:54:22.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.460+0000 7f44b77fe640 1 --2- 172.21.15.33:0/2147544958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44b8103360 0x7f44b8198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:22.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.461+0000 7f44b7fff640 1 --2- 172.21.15.33:0/2147544958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44b8102a40 0x7f44b8197fd0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f44a800e9f0 tx=0x7f44a800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:22.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.461+0000 7f44b57fa640 1 -- 172.21.15.33:0/2147544958 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a800ccc0 con 0x7f44b8102a40 2024-03-19T02:54:22.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.461+0000 7f44be6c6640 1 -- 172.21.15.33:0/2147544958 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44b819d2b0 con 0x7f44b8102a40 2024-03-19T02:54:22.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.461+0000 7f44b57fa640 1 -- 172.21.15.33:0/2147544958 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f44a8005020 con 0x7f44b8102a40 2024-03-19T02:54:22.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.461+0000 7f44b57fa640 1 -- 172.21.15.33:0/2147544958 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a800ce60 con 0x7f44b8102a40 2024-03-19T02:54:22.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.461+0000 7f44be6c6640 1 -- 172.21.15.33:0/2147544958 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44b81a4b40 con 0x7f44b8102a40 2024-03-19T02:54:22.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.463+0000 7f44b57fa640 1 -- 172.21.15.33:0/2147544958 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7f44a80051d0 con 0x7f44b8102a40 2024-03-19T02:54:22.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.464+0000 7f44c1393740 1 Processor -- start 2024-03-19T02:54:22.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.464+0000 7f44c1393740 1 -- start start 2024-03-19T02:54:22.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.464+0000 7f44c1393740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605b13ea2d0 0x5605b13ea6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.464+0000 7f44c1393740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5605b13eabf0 0x5605b13f3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.464+0000 7f44c1393740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b13f37d0 0x5605b13f5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.464+0000 7f44c1393740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5605b134d4b0 con 0x5605b13f37d0 2024-03-19T02:54:22.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.464+0000 7f44b57fa640 1 --2- 172.21.15.33:0/2147544958 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f44a00a2a90 0x7f44a00a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.464+0000 7f44c1393740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5605b12f5d70 con 0x5605b13eabf0 2024-03-19T02:54:22.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.464+0000 7f44c1393740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5605b1349550 con 0x5605b13ea2d0 2024-03-19T02:54:22.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.464+0000 7f44b77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b13f37d0 0x5605b13f5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.464+0000 7f44bcc63640 1 --2- 172.21.15.33:0/2147544958 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f44a00a2a90 0x7f44a00a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.464+0000 7f44b7fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5605b13eabf0 0x5605b13f3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.464+0000 7f44b77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b13f37d0 0x5605b13f5bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37342/0 (socket says 172.21.15.33:37342) 2024-03-19T02:54:22.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.464+0000 7f44b77fe640 1 -- 172.21.15.33:0/3763314829 learned_addr learned my addr 172.21.15.33:0/3763314829 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:22.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.464+0000 7f44bcc63640 1 --2- 172.21.15.33:0/3763314829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605b13ea2d0 0x5605b13ea6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.464+0000 7f44b57fa640 1 -- 172.21.15.33:0/2147544958 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f44a80cf3a0 con 0x7f44b8102a40 2024-03-19T02:54:22.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.465+0000 7f44b7fff640 1 -- 172.21.15.33:0/3763314829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605b13ea2d0 msgr2=0x5605b13ea6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:22.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.465+0000 7f44b7fff640 1 --2- 172.21.15.33:0/3763314829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605b13ea2d0 0x5605b13ea6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.465+0000 7f44b7fff640 1 -- 172.21.15.33:0/3763314829 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b13f37d0 msgr2=0x5605b13f5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:22.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.465+0000 7f44b7fff640 1 --2- 172.21.15.33:0/3763314829 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b13f37d0 0x5605b13f5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.465+0000 7f44b7fff640 1 -- 172.21.15.33:0/3763314829 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605b1318210 con 0x5605b13eabf0 2024-03-19T02:54:22.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.465+0000 7f44bcc63640 1 --2- 172.21.15.33:0/3763314829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605b13ea2d0 0x5605b13ea6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:22.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.465+0000 7f44b77fe640 1 --2- 172.21.15.33:0/3763314829 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b13f37d0 0x5605b13f5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:22.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.465+0000 7f44bcc63640 1 --2- 172.21.15.33:0/2147544958 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f44a00a2a90 0x7f44a00a4f50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f44b0005fd0 tx=0x7f44b0009ea0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:22.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.465+0000 7f44b7fff640 1 --2- 172.21.15.33:0/3763314829 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5605b13eabf0 0x5605b13f3290 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f44a80cc600 tx=0x7f44a8013330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc0853d51d5250e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:22.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.465+0000 7f449e7fc640 1 -- 172.21.15.33:0/3763314829 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a805de60 con 0x5605b13eabf0 2024-03-19T02:54:22.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.465+0000 7f449e7fc640 1 -- 172.21.15.33:0/3763314829 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f44a805f0d0 con 0x5605b13eabf0 2024-03-19T02:54:22.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.465+0000 7f449e7fc640 1 -- 172.21.15.33:0/3763314829 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a809b440 con 0x5605b13eabf0 2024-03-19T02:54:22.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.466+0000 7f44c1393740 1 -- 172.21.15.33:0/3763314829 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5605b13eabf0 msgr2=0x5605b13f3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:22.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.466+0000 7f44c1393740 1 --2- 172.21.15.33:0/3763314829 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5605b13eabf0 0x5605b13f3290 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f44a80cc600 tx=0x7f44a8013330 comp rx=0 tx=0).stop 2024-03-19T02:54:22.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.466+0000 7f44c1393740 1 -- 172.21.15.33:0/3763314829 shutdown_connections 2024-03-19T02:54:22.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.466+0000 7f44c1393740 1 --2- 172.21.15.33:0/3763314829 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b13f37d0 0x5605b13f5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.466+0000 7f44c1393740 1 --2- 172.21.15.33:0/3763314829 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5605b13eabf0 0x5605b13f3290 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.466+0000 7f44c1393740 1 --2- 172.21.15.33:0/3763314829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605b13ea2d0 0x5605b13ea6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.466+0000 7f44c1393740 1 -- 172.21.15.33:0/3763314829 >> 172.21.15.33:0/3763314829 conn(0x5605b13e4a00 msgr2=0x5605b13e4de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:22.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.466+0000 7f44c1393740 1 -- 172.21.15.33:0/3763314829 shutdown_connections 2024-03-19T02:54:22.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.466+0000 7f44c1393740 1 -- 172.21.15.33:0/3763314829 wait complete. 2024-03-19T02:54:22.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.466+0000 7f44c1393740 1 -- start start 2024-03-19T02:54:22.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.466+0000 7f44c1393740 1 Processor -- start 2024-03-19T02:54:22.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.467+0000 7f44c1393740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b13ea2d0 0x5605b13f0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.467+0000 7f44c1393740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5605b13eabf0 0x5605b13f0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.467+0000 7f44c1393740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605b13f37d0 0x5605b13f1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.467+0000 7f44c1393740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5605b13f6560 con 0x5605b13ea2d0 2024-03-19T02:54:22.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.467+0000 7f44c1393740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5605b1309310 con 0x5605b13eabf0 2024-03-19T02:54:22.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.467+0000 7f44c1393740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5605b13f66e0 con 0x5605b13f37d0 2024-03-19T02:54:22.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.467+0000 7f44c1393740 10 client.0 _ll_register_callbacks cb 0x5605b12fd2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:54:22.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.467+0000 7f44b7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b13ea2d0 0x5605b13f0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.467+0000 7f44b7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b13ea2d0 0x5605b13f0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37348/0 (socket says 172.21.15.33:37348) 2024-03-19T02:54:22.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.467+0000 7f44b7fff640 1 -- 172.21.15.33:0/2921175422 learned_addr learned my addr 172.21.15.33:0/2921175422 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:22.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.467+0000 7f44b77fe640 1 --2- 172.21.15.33:0/2921175422 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5605b13eabf0 0x5605b13f0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.467+0000 7f44bcc63640 1 --2- 172.21.15.33:0/2921175422 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605b13f37d0 0x5605b13f1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.467+0000 7f44b7fff640 1 -- 172.21.15.33:0/2921175422 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605b13f37d0 msgr2=0x5605b13f1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:22.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.467+0000 7f44b7fff640 1 --2- 172.21.15.33:0/2921175422 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605b13f37d0 0x5605b13f1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.467+0000 7f44b7fff640 1 -- 172.21.15.33:0/2921175422 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5605b13eabf0 msgr2=0x5605b13f0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:22.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.467+0000 7f44b7fff640 1 --2- 172.21.15.33:0/2921175422 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5605b13eabf0 0x5605b13f0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.467+0000 7f44b7fff640 1 -- 172.21.15.33:0/2921175422 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605b10613f0 con 0x5605b13ea2d0 2024-03-19T02:54:22.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.467+0000 7f44b77fe640 1 --2- 172.21.15.33:0/2921175422 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5605b13eabf0 0x5605b13f0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:22.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.467+0000 7f44b7fff640 1 --2- 172.21.15.33:0/2921175422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b13ea2d0 0x5605b13f0810 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f44a8004610 tx=0x7f44a80a3980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e03b52952c6bd636 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:22.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.467+0000 7f44977fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:54:22.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.467+0000 7f44977fe640 1 -- 172.21.15.33:0/2921175422 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a8095060 con 0x5605b13ea2d0 2024-03-19T02:54:22.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.467+0000 7f44977fe640 1 -- 172.21.15.33:0/2921175422 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f44a809a090 con 0x5605b13ea2d0 2024-03-19T02:54:22.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.467+0000 7f44c1393740 1 -- 172.21.15.33:0/2921175422 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44a805f0d0 con 0x5605b13ea2d0 2024-03-19T02:54:22.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.468+0000 7f44977fe640 1 -- 172.21.15.33:0/2921175422 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f44a801c040 con 0x5605b13ea2d0 2024-03-19T02:54:22.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.468+0000 7f44c1393740 10 client.15885 fetch_fsmap learned FSMap version 25 2024-03-19T02:54:22.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.468+0000 7f44c1393740 1 -- 172.21.15.33:0/2921175422 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5605b112b7d0 con 0x5605b13ea2d0 2024-03-19T02:54:22.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.468+0000 7f44977fe640 1 -- 172.21.15.33:0/2921175422 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f44a805f0d0 con 0x5605b13ea2d0 2024-03-19T02:54:22.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.468+0000 7f44977fe640 1 -- 172.21.15.33:0/2921175422 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f44a809f170 con 0x5605b13ea2d0 2024-03-19T02:54:22.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.468+0000 7f44c1393740 10 client.15885 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:54:22.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.468+0000 7f44c1393740 10 client.15885 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:54:22.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.468+0000 7f44c1393740 20 client.15885 populate_metadata read hostname 'smithi033' 2024-03-19T02:54:22.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.468+0000 7f44c1393740 1 --2- 172.21.15.33:0/2921175422 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5605b14039a0 0x5605b1405d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.468+0000 7f44c1393740 4 client.15885 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:54:22.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.468+0000 7f44c1393740 1 -- 172.21.15.33:0/2921175422 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5605b13ab3e0 con 0x5605b14039a0 2024-03-19T02:54:22.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.469+0000 7f44b77fe640 1 --2- 172.21.15.33:0/2921175422 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5605b14039a0 0x5605b1405d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.470+0000 7f44b77fe640 1 --2- 172.21.15.33:0/2921175422 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5605b14039a0 0x5605b1405d90 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45aa456144b9ff18 server_cookie=f099a68daca85655 in_seq=0 out_seq=0 2024-03-19T02:54:22.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.470+0000 7f44977fe640 10 client.15885 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:54:22.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.471+0000 7f44977fe640 1 -- 172.21.15.33:0/2921175422 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5605b13ab3e0 con 0x5605b14039a0 2024-03-19T02:54:22.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.471+0000 7f44977fe640 10 client.15885 handle_command_reply: tid=0 2024-03-19T02:54:22.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.471+0000 7f44c1393740 1 client.15885 shutdown 2024-03-19T02:54:22.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.471+0000 7f44c1393740 1 -- 172.21.15.33:0/2921175422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b13ea2d0 msgr2=0x5605b13f0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:22.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.471+0000 7f44c1393740 1 --2- 172.21.15.33:0/2921175422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b13ea2d0 0x5605b13f0810 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f44a8004610 tx=0x7f44a80a3980 comp rx=0 tx=0).stop 2024-03-19T02:54:22.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.471+0000 7f44c1393740 1 -- 172.21.15.33:0/2921175422 shutdown_connections 2024-03-19T02:54:22.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.471+0000 7f44c1393740 1 --2- 172.21.15.33:0/2921175422 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5605b14039a0 0x5605b1405d90 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.471+0000 7f44c1393740 1 --2- 172.21.15.33:0/2921175422 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605b13f37d0 0x5605b13f1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.471+0000 7f44c1393740 1 --2- 172.21.15.33:0/2921175422 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5605b13eabf0 0x5605b13f0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.471+0000 7f44c1393740 1 --2- 172.21.15.33:0/2921175422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b13ea2d0 0x5605b13f0810 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.471+0000 7f44c1393740 1 -- 172.21.15.33:0/2921175422 >> 172.21.15.33:0/2921175422 conn(0x5605b13e4a00 msgr2=0x5605b13e6750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:22.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.471+0000 7f44977fe640 0 client.15885 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:54:22.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.471+0000 7f44c1393740 1 -- 172.21.15.33:0/2921175422 shutdown_connections 2024-03-19T02:54:22.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.471+0000 7f44c1393740 1 -- 172.21.15.33:0/2921175422 wait complete. 2024-03-19T02:54:22.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.471+0000 7f44c1393740 20 client.15885 trim_cache size 0 max 16384 2024-03-19T02:54:22.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.488+0000 7f44c1393740 1 Processor -- start 2024-03-19T02:54:22.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.488+0000 7f44c1393740 1 -- start start 2024-03-19T02:54:22.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.488+0000 7f44c1393740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b13ea2d0 0x5605b13f2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.488+0000 7f44c1393740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5605b13eabf0 0x5605b13ef0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.488+0000 7f44c1393740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605b13f37d0 0x5605b13f1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.488+0000 7f44c1393740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f44b81af1d0 con 0x5605b13ea2d0 2024-03-19T02:54:22.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.488+0000 7f44c1393740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f44b81acaa0 con 0x5605b13eabf0 2024-03-19T02:54:22.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.488+0000 7f44c1393740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f44b81aa8d0 con 0x5605b13f37d0 2024-03-19T02:54:22.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.488+0000 7f44b7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b13ea2d0 0x5605b13f2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.488+0000 7f44bcc63640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605b13f37d0 0x5605b13f1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.488+0000 7f44b77fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5605b13eabf0 0x5605b13ef0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.489+0000 7f44b7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b13ea2d0 0x5605b13f2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37352/0 (socket says 172.21.15.33:37352) 2024-03-19T02:54:22.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.489+0000 7f44b7fff640 1 -- 172.21.15.33:0/2256987740 learned_addr learned my addr 172.21.15.33:0/2256987740 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:22.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.489+0000 7f44b77fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5605b13eabf0 0x5605b13ef0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:48812/0 (socket says 172.21.15.33:48812) 2024-03-19T02:54:22.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.489+0000 7f44bcc63640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605b13f37d0 0x5605b13f1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:39666/0 (socket says 172.21.15.33:39666) 2024-03-19T02:54:22.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.489+0000 7f44b7fff640 1 -- 172.21.15.33:0/2256987740 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605b13f37d0 msgr2=0x5605b13f1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:22.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.489+0000 7f44b7fff640 1 --2- 172.21.15.33:0/2256987740 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605b13f37d0 0x5605b13f1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.489+0000 7f44b7fff640 1 -- 172.21.15.33:0/2256987740 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5605b13eabf0 msgr2=0x5605b13ef0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:22.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.489+0000 7f44b7fff640 1 --2- 172.21.15.33:0/2256987740 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5605b13eabf0 0x5605b13ef0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.489+0000 7f44b7fff640 1 -- 172.21.15.33:0/2256987740 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605b13ef7c0 con 0x5605b13ea2d0 2024-03-19T02:54:22.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.489+0000 7f44b7fff640 1 --2- 172.21.15.33:0/2256987740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b13ea2d0 0x5605b13f2f40 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f44a8097dc0 tx=0x7f44a8005750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=796815cddb407c53 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:22.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.489+0000 7f449dffb640 1 -- 172.21.15.33:0/2256987740 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a8095070 con 0x5605b13ea2d0 2024-03-19T02:54:22.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.489+0000 7f449dffb640 1 -- 172.21.15.33:0/2256987740 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f44a809bb40 con 0x5605b13ea2d0 2024-03-19T02:54:22.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.489+0000 7f449dffb640 1 -- 172.21.15.33:0/2256987740 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a80cf1b0 con 0x5605b13ea2d0 2024-03-19T02:54:22.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.490+0000 7f44c1393740 1 -- 172.21.15.33:0/2256987740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b13ea2d0 msgr2=0x5605b13f2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:22.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.490+0000 7f44c1393740 1 --2- 172.21.15.33:0/2256987740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b13ea2d0 0x5605b13f2f40 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f44a8097dc0 tx=0x7f44a8005750 comp rx=0 tx=0).stop 2024-03-19T02:54:22.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.490+0000 7f44c1393740 1 -- 172.21.15.33:0/2256987740 shutdown_connections 2024-03-19T02:54:22.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.490+0000 7f44c1393740 1 --2- 172.21.15.33:0/2256987740 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605b13f37d0 0x5605b13f1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.490+0000 7f44c1393740 1 --2- 172.21.15.33:0/2256987740 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5605b13eabf0 0x5605b13ef0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.490+0000 7f44c1393740 1 --2- 172.21.15.33:0/2256987740 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b13ea2d0 0x5605b13f2f40 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.490+0000 7f44c1393740 1 -- 172.21.15.33:0/2256987740 >> 172.21.15.33:0/2256987740 conn(0x5605b13e4a00 msgr2=0x5605b13ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:22.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.490+0000 7f44c1393740 1 -- 172.21.15.33:0/2256987740 shutdown_connections 2024-03-19T02:54:22.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.490+0000 7f44c1393740 1 -- 172.21.15.33:0/2256987740 wait complete. 2024-03-19T02:54:22.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.490+0000 7f44c1393740 1 -- start start 2024-03-19T02:54:22.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.491+0000 7f44c1393740 1 Processor -- start 2024-03-19T02:54:22.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.491+0000 7f44c1393740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b13ea2d0 0x5605b1408c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.491+0000 7f44c1393740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605b13eabf0 0x5605b1405fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.491+0000 7f44c1393740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5605b13f37d0 0x5605b14064f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.491+0000 7f44c1393740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f44b81aaf60 con 0x5605b13ea2d0 2024-03-19T02:54:22.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.491+0000 7f44c1393740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f44b81ac3d0 con 0x5605b13f37d0 2024-03-19T02:54:22.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.491+0000 7f44c1393740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f44b81ab630 con 0x5605b13eabf0 2024-03-19T02:54:22.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.491+0000 7f44c1393740 10 client.0 _ll_register_callbacks cb 0x7f44a8097030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:54:22.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.491+0000 7f44b7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b13ea2d0 0x5605b1408c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.491+0000 7f44bcc63640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5605b13f37d0 0x5605b14064f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.491+0000 7f44b7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b13ea2d0 0x5605b1408c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37360/0 (socket says 172.21.15.33:37360) 2024-03-19T02:54:22.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.491+0000 7f44b77fe640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605b13eabf0 0x5605b1405fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.491+0000 7f44b7fff640 1 -- 172.21.15.33:0/1968999116 learned_addr learned my addr 172.21.15.33:0/1968999116 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:22.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.492+0000 7f44bcc63640 1 -- 172.21.15.33:0/1968999116 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605b13eabf0 msgr2=0x5605b1405fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:22.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.492+0000 7f44bcc63640 1 --2- 172.21.15.33:0/1968999116 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605b13eabf0 0x5605b1405fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.492+0000 7f44bcc63640 1 -- 172.21.15.33:0/1968999116 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b13ea2d0 msgr2=0x5605b1408c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:22.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.492+0000 7f44bcc63640 1 --2- 172.21.15.33:0/1968999116 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b13ea2d0 0x5605b1408c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.492+0000 7f44bcc63640 1 -- 172.21.15.33:0/1968999116 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605b1406cc0 con 0x5605b13f37d0 2024-03-19T02:54:22.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.492+0000 7f44b77fe640 1 --2- 172.21.15.33:0/1968999116 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605b13eabf0 0x5605b1405fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:22.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.492+0000 7f44b7fff640 1 --2- 172.21.15.33:0/1968999116 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b13ea2d0 0x5605b1408c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:22.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.492+0000 7f44bcc63640 1 --2- 172.21.15.33:0/1968999116 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5605b13f37d0 0x5605b14064f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x5605b13ff950 tx=0x7f44b0009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22109b7fd85fd798 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:22.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.492+0000 7f44977fe640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T02:54:22.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.492+0000 7f44977fe640 1 -- 172.21.15.33:0/1968999116 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44b002f060 con 0x5605b13f37d0 2024-03-19T02:54:22.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.492+0000 7f44c1393740 1 -- 172.21.15.33:0/1968999116 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44a809bb40 con 0x5605b13f37d0 2024-03-19T02:54:22.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.492+0000 7f44977fe640 1 -- 172.21.15.33:0/1968999116 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f44b001a070 con 0x5605b13f37d0 2024-03-19T02:54:22.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.492+0000 7f44977fe640 1 -- 172.21.15.33:0/1968999116 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f44b0036050 con 0x5605b13f37d0 2024-03-19T02:54:22.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.492+0000 7f44c1393740 10 client.25760 fetch_fsmap learned FSMap version 25 2024-03-19T02:54:22.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.492+0000 7f44c1393740 1 -- 172.21.15.33:0/1968999116 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5605b1469930 con 0x5605b13f37d0 2024-03-19T02:54:22.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.492+0000 7f44977fe640 1 -- 172.21.15.33:0/1968999116 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f44a809bb40 con 0x5605b13f37d0 2024-03-19T02:54:22.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.493+0000 7f44977fe640 1 -- 172.21.15.33:0/1968999116 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f44b001ecc0 con 0x5605b13f37d0 2024-03-19T02:54:22.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.493+0000 7f44c1393740 10 client.25760 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:54:22.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.493+0000 7f44c1393740 10 client.25760 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:54:22.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.493+0000 7f44c1393740 20 client.25760 populate_metadata read hostname 'smithi033' 2024-03-19T02:54:22.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.493+0000 7f44c1393740 1 --2- 172.21.15.33:0/1968999116 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5605b146a060 0x5605b146c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:22.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.493+0000 7f44c1393740 4 client.25760 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:54:22.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.493+0000 7f44c1393740 1 -- 172.21.15.33:0/1968999116 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5605b135a9f0 con 0x5605b146a060 2024-03-19T02:54:22.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.493+0000 7f44b7fff640 1 --2- 172.21.15.33:0/1968999116 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5605b146a060 0x5605b146c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:22.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.493+0000 7f44977fe640 10 client.25760 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:54:22.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.493+0000 7f44b7fff640 1 --2- 172.21.15.33:0/1968999116 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5605b146a060 0x5605b146c480 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ef7a5b76d63916e server_cookie=970a741882707f0e in_seq=0 out_seq=0 2024-03-19T02:54:22.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.493+0000 7f44977fe640 1 -- 172.21.15.33:0/1968999116 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5605b135a9f0 con 0x5605b146a060 2024-03-19T02:54:22.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.493+0000 7f44977fe640 10 client.25760 handle_command_reply: tid=0 2024-03-19T02:54:22.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.493+0000 7f44c1393740 1 client.25760 shutdown 2024-03-19T02:54:22.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.494+0000 7f44c1393740 1 -- 172.21.15.33:0/1968999116 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5605b13f37d0 msgr2=0x5605b14064f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:22.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.494+0000 7f44c1393740 1 --2- 172.21.15.33:0/1968999116 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5605b13f37d0 0x5605b14064f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x5605b13ff950 tx=0x7f44b0009500 comp rx=0 tx=0).stop 2024-03-19T02:54:22.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.494+0000 7f44c1393740 1 -- 172.21.15.33:0/1968999116 shutdown_connections 2024-03-19T02:54:22.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.494+0000 7f44c1393740 1 --2- 172.21.15.33:0/1968999116 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5605b146a060 0x5605b146c480 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.494+0000 7f44c1393740 1 --2- 172.21.15.33:0/1968999116 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5605b13eabf0 0x5605b1405fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.494+0000 7f44c1393740 1 --2- 172.21.15.33:0/1968999116 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5605b13f37d0 0x5605b14064f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.494+0000 7f44c1393740 1 --2- 172.21.15.33:0/1968999116 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5605b13ea2d0 0x5605b1408c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.494+0000 7f44977fe640 0 client.25760 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:54:22.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.494+0000 7f44c1393740 1 -- 172.21.15.33:0/1968999116 >> 172.21.15.33:0/1968999116 conn(0x5605b13e4a00 msgr2=0x5605b13e3ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:22.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.494+0000 7f44c1393740 1 -- 172.21.15.33:0/1968999116 shutdown_connections 2024-03-19T02:54:22.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.494+0000 7f44c1393740 1 -- 172.21.15.33:0/1968999116 wait complete. 2024-03-19T02:54:22.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.494+0000 7f44c1393740 20 client.25760 trim_cache size 0 max 16384 2024-03-19T02:54:22.502 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:54:22.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.494+0000 7f44be6c6640 1 -- 172.21.15.33:0/2147544958 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f44a00a2a90 msgr2=0x7f44a00a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:22.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.494+0000 7f44be6c6640 1 --2- 172.21.15.33:0/2147544958 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f44a00a2a90 0x7f44a00a4f50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f44b0005fd0 tx=0x7f44b0009ea0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.494+0000 7f44be6c6640 1 -- 172.21.15.33:0/2147544958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44b8102a40 msgr2=0x7f44b8197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:22.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.494+0000 7f44be6c6640 1 --2- 172.21.15.33:0/2147544958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44b8102a40 0x7f44b8197fd0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f44a800e9f0 tx=0x7f44a800edf0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.494+0000 7f44be6c6640 1 -- 172.21.15.33:0/2147544958 shutdown_connections 2024-03-19T02:54:22.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.494+0000 7f44be6c6640 1 --2- 172.21.15.33:0/2147544958 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f44a00a2a90 0x7f44a00a4f50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.494+0000 7f44be6c6640 1 --2- 172.21.15.33:0/2147544958 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44b810bd80 0x7f44b819c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.494+0000 7f44be6c6640 1 --2- 172.21.15.33:0/2147544958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f44b8103360 0x7f44b8198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.494+0000 7f44be6c6640 1 --2- 172.21.15.33:0/2147544958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44b8102a40 0x7f44b8197fd0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:22.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.494+0000 7f44be6c6640 1 -- 172.21.15.33:0/2147544958 >> 172.21.15.33:0/2147544958 conn(0x7f44b80778d0 msgr2=0x7f44b80fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:22.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.495+0000 7f44be6c6640 1 -- 172.21.15.33:0/2147544958 shutdown_connections 2024-03-19T02:54:22.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:22.495+0000 7f44be6c6640 1 -- 172.21.15.33:0/2147544958 wait complete. 2024-03-19T02:54:22.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:22 smithi096 ceph-mon[35946]: pgmap v30: 129 pgs: 129 active+clean; 205 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 17 MiB/s wr, 10 op/s 2024-03-19T02:54:22.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:22 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1273623266' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:54:23.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:22 smithi155 ceph-mon[35804]: pgmap v30: 129 pgs: 129 active+clean; 205 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 17 MiB/s wr, 10 op/s 2024-03-19T02:54:23.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:22 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1273623266' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:54:23.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:22 smithi033 ceph-mon[31973]: pgmap v30: 129 pgs: 129 active+clean; 205 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 17 MiB/s wr, 10 op/s 2024-03-19T02:54:23.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:22 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1273623266' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:54:23.507 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:54:23.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:54:23.508 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:54:23.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.624+0000 7f6dce04d640 1 -- 172.21.15.33:0/3947939853 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6dc81066a0 msgr2=0x7f6dc8106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:23.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.624+0000 7f6dce04d640 1 --2- 172.21.15.33:0/3947939853 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6dc81066a0 0x7f6dc8106a80 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f6db8009930 tx=0x7f6db802bc30 comp rx=0 tx=0).stop 2024-03-19T02:54:23.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.624+0000 7f6dce04d640 1 -- 172.21.15.33:0/3947939853 shutdown_connections 2024-03-19T02:54:23.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.624+0000 7f6dce04d640 1 --2- 172.21.15.33:0/3947939853 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6dc8101050 0x7f6dc810d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.624+0000 7f6dce04d640 1 --2- 172.21.15.33:0/3947939853 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6dc8100690 0x7f6dc8100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.624+0000 7f6dce04d640 1 --2- 172.21.15.33:0/3947939853 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6dc81066a0 0x7f6dc8106a80 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.624+0000 7f6dce04d640 1 -- 172.21.15.33:0/3947939853 >> 172.21.15.33:0/3947939853 conn(0x7f6dc8077820 msgr2=0x7f6dc8077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:23.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.624+0000 7f6dce04d640 1 -- 172.21.15.33:0/3947939853 shutdown_connections 2024-03-19T02:54:23.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.624+0000 7f6dce04d640 1 -- 172.21.15.33:0/3947939853 wait complete. 2024-03-19T02:54:23.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.625+0000 7f6dce04d640 1 Processor -- start 2024-03-19T02:54:23.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.625+0000 7f6dce04d640 1 -- start start 2024-03-19T02:54:23.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.626+0000 7f6dce04d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6dc8100690 0x7f6dc8198e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.626+0000 7f6dce04d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6dc8101050 0x7f6dc81993a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.626+0000 7f6dce04d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6dc81066a0 0x7f6dc81a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.626+0000 7f6dce04d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc8110960 con 0x7f6dc81066a0 2024-03-19T02:54:23.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.626+0000 7f6dce04d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc81107e0 con 0x7f6dc8101050 2024-03-19T02:54:23.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.626+0000 7f6dce04d640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc8110ae0 con 0x7f6dc8100690 2024-03-19T02:54:23.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.626+0000 7f6dc7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6dc81066a0 0x7f6dc81a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.626+0000 7f6dc6ffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6dc8101050 0x7f6dc81993a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.626+0000 7f6dc7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6dc81066a0 0x7f6dc81a3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33572/0 (socket says 172.21.15.33:33572) 2024-03-19T02:54:23.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.626+0000 7f6dc6ffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6dc8101050 0x7f6dc81993a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:58522/0 (socket says 172.21.15.33:58522) 2024-03-19T02:54:23.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.626+0000 7f6dc7fff640 1 -- 172.21.15.33:0/760023982 learned_addr learned my addr 172.21.15.33:0/760023982 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:23.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.626+0000 7f6dc77fe640 1 --2- 172.21.15.33:0/760023982 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6dc8100690 0x7f6dc8198e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.626+0000 7f6dc7fff640 1 -- 172.21.15.33:0/760023982 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6dc8100690 msgr2=0x7f6dc8198e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:23.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.626+0000 7f6dc7fff640 1 --2- 172.21.15.33:0/760023982 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6dc8100690 0x7f6dc8198e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.626+0000 7f6dc7fff640 1 -- 172.21.15.33:0/760023982 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6dc8101050 msgr2=0x7f6dc81993a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:23.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.626+0000 7f6dc7fff640 1 --2- 172.21.15.33:0/760023982 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6dc8101050 0x7f6dc81993a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.626+0000 7f6dc7fff640 1 -- 172.21.15.33:0/760023982 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dc81a4420 con 0x7f6dc81066a0 2024-03-19T02:54:23.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.626+0000 7f6dc77fe640 1 --2- 172.21.15.33:0/760023982 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6dc8100690 0x7f6dc8198e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:54:23.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.626+0000 7f6dc6ffd640 1 --2- 172.21.15.33:0/760023982 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6dc8101050 0x7f6dc81993a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:23.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.627+0000 7f6dc7fff640 1 --2- 172.21.15.33:0/760023982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6dc81066a0 0x7f6dc81a3ee0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f6dc000eaa0 tx=0x7f6dc000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:23.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.627+0000 7f6dc4ff9640 1 -- 172.21.15.33:0/760023982 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc000cdb0 con 0x7f6dc81066a0 2024-03-19T02:54:23.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.627+0000 7f6dce04d640 1 -- 172.21.15.33:0/760023982 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dc81a45f0 con 0x7f6dc81066a0 2024-03-19T02:54:23.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.627+0000 7f6dc4ff9640 1 -- 172.21.15.33:0/760023982 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6dc0005020 con 0x7f6dc81066a0 2024-03-19T02:54:23.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.627+0000 7f6dc4ff9640 1 -- 172.21.15.33:0/760023982 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc00116d0 con 0x7f6dc81066a0 2024-03-19T02:54:23.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.627+0000 7f6dce04d640 1 -- 172.21.15.33:0/760023982 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dc81a4b10 con 0x7f6dc81066a0 2024-03-19T02:54:23.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.629+0000 7f6dc4ff9640 1 -- 172.21.15.33:0/760023982 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7f6dc00051d0 con 0x7f6dc81066a0 2024-03-19T02:54:23.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.630+0000 7f6dc4ff9640 1 --2- 172.21.15.33:0/760023982 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6db00a2a90 0x7f6db00a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.630+0000 7f6dd0c30740 1 Processor -- start 2024-03-19T02:54:23.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.630+0000 7f6dc4ff9640 1 -- 172.21.15.33:0/760023982 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f6dc00cf5d0 con 0x7f6dc81066a0 2024-03-19T02:54:23.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.630+0000 7f6dc77fe640 1 --2- 172.21.15.33:0/760023982 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6db00a2a90 0x7f6db00a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.630+0000 7f6dd0c30740 1 -- start start 2024-03-19T02:54:23.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.630+0000 7f6dd0c30740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563ffdd8b380 0x563ffdd8b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.630+0000 7f6dd0c30740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ffdd8bca0 0x563ffdd94340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.630+0000 7f6dd0c30740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ffdd94880 0x563ffdd96c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.630+0000 7f6dd0c30740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563ffdcef160 con 0x563ffdd94880 2024-03-19T02:54:23.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.630+0000 7f6dd0c30740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x563ffdc97e10 con 0x563ffdd8bca0 2024-03-19T02:54:23.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.630+0000 7f6dd0c30740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x563ffdceb200 con 0x563ffdd8b380 2024-03-19T02:54:23.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.631+0000 7f6dc7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ffdd94880 0x563ffdd96c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.631+0000 7f6dc6ffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ffdd8bca0 0x563ffdd94340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.631+0000 7f6dc6ffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ffdd8bca0 0x563ffdd94340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:58528/0 (socket says 172.21.15.33:58528) 2024-03-19T02:54:23.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.631+0000 7f6dc6ffd640 1 -- 172.21.15.33:0/3700506026 learned_addr learned my addr 172.21.15.33:0/3700506026 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:23.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.631+0000 7f6dc7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ffdd94880 0x563ffdd96c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33586/0 (socket says 172.21.15.33:33586) 2024-03-19T02:54:23.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.631+0000 7f6dc77fe640 1 --2- 172.21.15.33:0/760023982 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6db00a2a90 0x7f6db00a4f50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6db8006010 tx=0x7f6db8009610 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:23.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.631+0000 7f6dc77fe640 1 --2- 172.21.15.33:0/3700506026 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563ffdd8b380 0x563ffdd8b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.631+0000 7f6dc6ffd640 1 -- 172.21.15.33:0/3700506026 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563ffdd8b380 msgr2=0x563ffdd8b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:23.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.631+0000 7f6dc6ffd640 1 --2- 172.21.15.33:0/3700506026 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563ffdd8b380 0x563ffdd8b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.631+0000 7f6dc6ffd640 1 -- 172.21.15.33:0/3700506026 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ffdd94880 msgr2=0x563ffdd96c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:23.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.631+0000 7f6dc6ffd640 1 --2- 172.21.15.33:0/3700506026 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ffdd94880 0x563ffdd96c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.631+0000 7f6dc6ffd640 1 -- 172.21.15.33:0/3700506026 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ffdacb7e0 con 0x563ffdd8bca0 2024-03-19T02:54:23.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.631+0000 7f6dc6ffd640 1 --2- 172.21.15.33:0/3700506026 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ffdd8bca0 0x563ffdd94340 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc0114f0 tx=0x7f6dbc0119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5179a9264ebaf2cd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:23.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.631+0000 7f6dadffb640 1 -- 172.21.15.33:0/3700506026 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dbc01e070 con 0x563ffdd8bca0 2024-03-19T02:54:23.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.631+0000 7f6dadffb640 1 -- 172.21.15.33:0/3700506026 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6dbc00fa70 con 0x563ffdd8bca0 2024-03-19T02:54:23.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.631+0000 7f6dadffb640 1 -- 172.21.15.33:0/3700506026 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dbc019b00 con 0x563ffdd8bca0 2024-03-19T02:54:23.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.632+0000 7f6dd0c30740 1 -- 172.21.15.33:0/3700506026 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ffdd8bca0 msgr2=0x563ffdd94340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:23.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.632+0000 7f6dd0c30740 1 --2- 172.21.15.33:0/3700506026 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ffdd8bca0 0x563ffdd94340 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc0114f0 tx=0x7f6dbc0119b0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.632+0000 7f6dd0c30740 1 -- 172.21.15.33:0/3700506026 shutdown_connections 2024-03-19T02:54:23.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.632+0000 7f6dd0c30740 1 --2- 172.21.15.33:0/3700506026 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ffdd94880 0x563ffdd96c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.632+0000 7f6dd0c30740 1 --2- 172.21.15.33:0/3700506026 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ffdd8bca0 0x563ffdd94340 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.632+0000 7f6dd0c30740 1 --2- 172.21.15.33:0/3700506026 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563ffdd8b380 0x563ffdd8b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.632+0000 7f6dd0c30740 1 -- 172.21.15.33:0/3700506026 >> 172.21.15.33:0/3700506026 conn(0x563ffdd85ab0 msgr2=0x563ffdd85e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:23.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.632+0000 7f6dd0c30740 1 -- 172.21.15.33:0/3700506026 shutdown_connections 2024-03-19T02:54:23.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.632+0000 7f6dd0c30740 1 -- 172.21.15.33:0/3700506026 wait complete. 2024-03-19T02:54:23.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.633+0000 7f6dd0c30740 1 -- start start 2024-03-19T02:54:23.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.633+0000 7f6dd0c30740 1 Processor -- start 2024-03-19T02:54:23.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.633+0000 7f6dd0c30740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ffdd8b380 0x563ffdd917c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.633+0000 7f6dd0c30740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ffdd8bca0 0x563ffdd91d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.633+0000 7f6dd0c30740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563ffdd94880 0x563ffdd92240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.633+0000 7f6dd0c30740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x563ffdd97670 con 0x563ffdd8b380 2024-03-19T02:54:23.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.633+0000 7f6dd0c30740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x563ffdcab1a0 con 0x563ffdd8bca0 2024-03-19T02:54:23.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.633+0000 7f6dd0c30740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x563ffdd977f0 con 0x563ffdd94880 2024-03-19T02:54:23.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.633+0000 7f6dd0c30740 10 client.0 _ll_register_callbacks cb 0x563ffda02c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:54:23.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.633+0000 7f6dc77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ffdd8b380 0x563ffdd917c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.633+0000 7f6dc77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ffdd8b380 0x563ffdd917c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33590/0 (socket says 172.21.15.33:33590) 2024-03-19T02:54:23.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.633+0000 7f6dc77fe640 1 -- 172.21.15.33:0/1034909947 learned_addr learned my addr 172.21.15.33:0/1034909947 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:23.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.633+0000 7f6dc7fff640 1 --2- 172.21.15.33:0/1034909947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563ffdd94880 0x563ffdd92240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.633+0000 7f6dc6ffd640 1 --2- 172.21.15.33:0/1034909947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ffdd8bca0 0x563ffdd91d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.634+0000 7f6dc6ffd640 1 -- 172.21.15.33:0/1034909947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563ffdd94880 msgr2=0x563ffdd92240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:23.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.634+0000 7f6dc6ffd640 1 --2- 172.21.15.33:0/1034909947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563ffdd94880 0x563ffdd92240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.634+0000 7f6dc6ffd640 1 -- 172.21.15.33:0/1034909947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ffdd8b380 msgr2=0x563ffdd917c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:23.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.634+0000 7f6dc6ffd640 1 --2- 172.21.15.33:0/1034909947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ffdd8b380 0x563ffdd917c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.634+0000 7f6dc6ffd640 1 -- 172.21.15.33:0/1034909947 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ffdb2bf70 con 0x563ffdd8bca0 2024-03-19T02:54:23.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.634+0000 7f6dc77fe640 1 --2- 172.21.15.33:0/1034909947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ffdd8b380 0x563ffdd917c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:23.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.634+0000 7f6dc6ffd640 1 --2- 172.21.15.33:0/1034909947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ffdd8bca0 0x563ffdd91d00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc000c00 tx=0x7f6dbc009d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bfa77edc1c5d326 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:23.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.634+0000 7f6da6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T02:54:23.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.634+0000 7f6da6ffd640 1 -- 172.21.15.33:0/1034909947 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dbc01e040 con 0x563ffdd8bca0 2024-03-19T02:54:23.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.634+0000 7f6da6ffd640 1 -- 172.21.15.33:0/1034909947 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f6dbc00f6c0 con 0x563ffdd8bca0 2024-03-19T02:54:23.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.634+0000 7f6dd0c30740 1 -- 172.21.15.33:0/1034909947 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6dbc00fa70 con 0x563ffdd8bca0 2024-03-19T02:54:23.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.634+0000 7f6da6ffd640 1 -- 172.21.15.33:0/1034909947 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f6dbc031050 con 0x563ffdd8bca0 2024-03-19T02:54:23.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.634+0000 7f6dd0c30740 10 client.25772 fetch_fsmap learned FSMap version 25 2024-03-19T02:54:23.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.634+0000 7f6dd0c30740 1 -- 172.21.15.33:0/1034909947 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563ffd9c3170 con 0x563ffdd8bca0 2024-03-19T02:54:23.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.634+0000 7f6da6ffd640 1 -- 172.21.15.33:0/1034909947 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6dbc00fa70 con 0x563ffdd8bca0 2024-03-19T02:54:23.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.635+0000 7f6da6ffd640 1 -- 172.21.15.33:0/1034909947 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f6dbc02c030 con 0x563ffdd8bca0 2024-03-19T02:54:23.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.635+0000 7f6dd0c30740 10 client.25772 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:54:23.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.635+0000 7f6dd0c30740 10 client.25772 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:54:23.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.635+0000 7f6dd0c30740 20 client.25772 populate_metadata read hostname 'smithi033' 2024-03-19T02:54:23.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.635+0000 7f6dd0c30740 1 --2- 172.21.15.33:0/1034909947 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x563ffdda4a60 0x563ffdda6e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.635+0000 7f6dd0c30740 4 client.25772 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:54:23.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.635+0000 7f6dd0c30740 1 -- 172.21.15.33:0/1034909947 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ffdcfc6a0 con 0x563ffdda4a60 2024-03-19T02:54:23.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.635+0000 7f6dc77fe640 1 --2- 172.21.15.33:0/1034909947 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x563ffdda4a60 0x563ffdda6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.635+0000 7f6dc77fe640 1 --2- 172.21.15.33:0/1034909947 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x563ffdda4a60 0x563ffdda6e50 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d5dcce1fd04a3 server_cookie=c003cfb3f46639d5 in_seq=0 out_seq=0 2024-03-19T02:54:23.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.635+0000 7f6da6ffd640 10 client.25772 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:54:23.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.636+0000 7f6da6ffd640 1 -- 172.21.15.33:0/1034909947 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563ffdcfc6a0 con 0x563ffdda4a60 2024-03-19T02:54:23.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.636+0000 7f6da6ffd640 10 client.25772 handle_command_reply: tid=0 2024-03-19T02:54:23.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.636+0000 7f6dd0c30740 1 client.25772 shutdown 2024-03-19T02:54:23.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.636+0000 7f6dd0c30740 1 -- 172.21.15.33:0/1034909947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ffdd8bca0 msgr2=0x563ffdd91d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:23.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.636+0000 7f6dd0c30740 1 --2- 172.21.15.33:0/1034909947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ffdd8bca0 0x563ffdd91d00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc000c00 tx=0x7f6dbc009d30 comp rx=0 tx=0).stop 2024-03-19T02:54:23.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.636+0000 7f6dd0c30740 1 -- 172.21.15.33:0/1034909947 shutdown_connections 2024-03-19T02:54:23.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.636+0000 7f6dd0c30740 1 --2- 172.21.15.33:0/1034909947 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x563ffdda4a60 0x563ffdda6e50 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.636+0000 7f6dd0c30740 1 --2- 172.21.15.33:0/1034909947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563ffdd94880 0x563ffdd92240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.636+0000 7f6dd0c30740 1 --2- 172.21.15.33:0/1034909947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ffdd8bca0 0x563ffdd91d00 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.636+0000 7f6da6ffd640 0 client.25772 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:54:23.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.636+0000 7f6dd0c30740 1 --2- 172.21.15.33:0/1034909947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ffdd8b380 0x563ffdd917c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.636+0000 7f6dd0c30740 1 -- 172.21.15.33:0/1034909947 >> 172.21.15.33:0/1034909947 conn(0x563ffdd85ab0 msgr2=0x563ffdd87800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:23.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.636+0000 7f6dd0c30740 1 -- 172.21.15.33:0/1034909947 shutdown_connections 2024-03-19T02:54:23.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.636+0000 7f6dd0c30740 1 -- 172.21.15.33:0/1034909947 wait complete. 2024-03-19T02:54:23.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.636+0000 7f6dd0c30740 20 client.25772 trim_cache size 0 max 16384 2024-03-19T02:54:23.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.654+0000 7f6dd0c30740 1 Processor -- start 2024-03-19T02:54:23.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.654+0000 7f6dd0c30740 1 -- start start 2024-03-19T02:54:23.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.654+0000 7f6dd0c30740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ffdd8b380 0x563ffdd93ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.654+0000 7f6dd0c30740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563ffdd8bca0 0x563ffdda0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.654+0000 7f6dd0c30740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ffdd94880 0x563ffdd875c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.654+0000 7f6dd0c30740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc81b1580 con 0x563ffdd8b380 2024-03-19T02:54:23.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.654+0000 7f6dd0c30740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc81aee50 con 0x563ffdd94880 2024-03-19T02:54:23.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.654+0000 7f6dd0c30740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc81acc80 con 0x563ffdd8bca0 2024-03-19T02:54:23.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.654+0000 7f6dc7fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ffdd94880 0x563ffdd875c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.654+0000 7f6dc77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ffdd8b380 0x563ffdd93ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.654+0000 7f6dc7fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ffdd94880 0x563ffdd875c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:58544/0 (socket says 172.21.15.33:58544) 2024-03-19T02:54:23.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.654+0000 7f6dc7fff640 1 -- 172.21.15.33:0/2984415023 learned_addr learned my addr 172.21.15.33:0/2984415023 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:23.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.655+0000 7f6dc6ffd640 1 --2- 172.21.15.33:0/2984415023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563ffdd8bca0 0x563ffdda0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.655+0000 7f6dc7fff640 1 -- 172.21.15.33:0/2984415023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563ffdd8bca0 msgr2=0x563ffdda0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:23.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.655+0000 7f6dc7fff640 1 --2- 172.21.15.33:0/2984415023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563ffdd8bca0 0x563ffdda0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.655+0000 7f6dc7fff640 1 -- 172.21.15.33:0/2984415023 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ffdd8b380 msgr2=0x563ffdd93ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:23.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.655+0000 7f6dc7fff640 1 --2- 172.21.15.33:0/2984415023 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ffdd8b380 0x563ffdd93ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.655+0000 7f6dc7fff640 1 -- 172.21.15.33:0/2984415023 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ffdd858e0 con 0x563ffdd94880 2024-03-19T02:54:23.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.655+0000 7f6dc7fff640 1 --2- 172.21.15.33:0/2984415023 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ffdd94880 0x563ffdd875c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6dc009b370 tx=0x7f6dc009b090 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2f055ea9572899c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:23.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.655+0000 7f6dad7fa640 1 -- 172.21.15.33:0/2984415023 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc00b3070 con 0x563ffdd94880 2024-03-19T02:54:23.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.655+0000 7f6dad7fa640 1 -- 172.21.15.33:0/2984415023 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6dc009f7c0 con 0x563ffdd94880 2024-03-19T02:54:23.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.655+0000 7f6dad7fa640 1 -- 172.21.15.33:0/2984415023 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc00bb040 con 0x563ffdd94880 2024-03-19T02:54:23.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.656+0000 7f6dd0c30740 1 -- 172.21.15.33:0/2984415023 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ffdd94880 msgr2=0x563ffdd875c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:23.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.656+0000 7f6dd0c30740 1 --2- 172.21.15.33:0/2984415023 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ffdd94880 0x563ffdd875c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6dc009b370 tx=0x7f6dc009b090 comp rx=0 tx=0).stop 2024-03-19T02:54:23.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.656+0000 7f6dd0c30740 1 -- 172.21.15.33:0/2984415023 shutdown_connections 2024-03-19T02:54:23.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.656+0000 7f6dd0c30740 1 --2- 172.21.15.33:0/2984415023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563ffdd8bca0 0x563ffdda0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.656+0000 7f6dd0c30740 1 --2- 172.21.15.33:0/2984415023 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ffdd94880 0x563ffdd875c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.656+0000 7f6dd0c30740 1 --2- 172.21.15.33:0/2984415023 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ffdd8b380 0x563ffdd93ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.656+0000 7f6dd0c30740 1 -- 172.21.15.33:0/2984415023 >> 172.21.15.33:0/2984415023 conn(0x563ffdd85ab0 msgr2=0x563ffdd86eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:23.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.656+0000 7f6dd0c30740 1 -- 172.21.15.33:0/2984415023 shutdown_connections 2024-03-19T02:54:23.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.656+0000 7f6dd0c30740 1 -- 172.21.15.33:0/2984415023 wait complete. 2024-03-19T02:54:23.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.656+0000 7f6dd0c30740 1 -- start start 2024-03-19T02:54:23.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.657+0000 7f6dd0c30740 1 Processor -- start 2024-03-19T02:54:23.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.657+0000 7f6dd0c30740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ffdd8b380 0x563ffdd8e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.657+0000 7f6dd0c30740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563ffdd8bca0 0x563ffdd8ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.657+0000 7f6dd0c30740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ffdd94880 0x563ffdd8ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.657+0000 7f6dd0c30740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc81ad310 con 0x563ffdd8b380 2024-03-19T02:54:23.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.657+0000 7f6dd0c30740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc81ae780 con 0x563ffdd94880 2024-03-19T02:54:23.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.657+0000 7f6dd0c30740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc81ad9e0 con 0x563ffdd8bca0 2024-03-19T02:54:23.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.657+0000 7f6dd0c30740 10 client.0 _ll_register_callbacks cb 0x7f6dbc01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:54:23.663 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.657+0000 7f6dc77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ffdd8b380 0x563ffdd8e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.657+0000 7f6dc77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ffdd8b380 0x563ffdd8e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33608/0 (socket says 172.21.15.33:33608) 2024-03-19T02:54:23.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.657+0000 7f6dc77fe640 1 -- 172.21.15.33:0/2932850213 learned_addr learned my addr 172.21.15.33:0/2932850213 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:23.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.657+0000 7f6dc7fff640 1 --2- 172.21.15.33:0/2932850213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ffdd94880 0x563ffdd8ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.657+0000 7f6dc6ffd640 1 --2- 172.21.15.33:0/2932850213 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563ffdd8bca0 0x563ffdd8ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.658+0000 7f6dc77fe640 1 -- 172.21.15.33:0/2932850213 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563ffdd8bca0 msgr2=0x563ffdd8ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:23.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.658+0000 7f6dc77fe640 1 --2- 172.21.15.33:0/2932850213 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563ffdd8bca0 0x563ffdd8ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.658+0000 7f6dc77fe640 1 -- 172.21.15.33:0/2932850213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ffdd94880 msgr2=0x563ffdd8ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:23.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.658+0000 7f6dc77fe640 1 --2- 172.21.15.33:0/2932850213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ffdd94880 0x563ffdd8ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.658+0000 7f6dc77fe640 1 -- 172.21.15.33:0/2932850213 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ffde09de0 con 0x563ffdd8b380 2024-03-19T02:54:23.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.658+0000 7f6dc77fe640 1 --2- 172.21.15.33:0/2932850213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ffdd8b380 0x563ffdd8e4f0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f6db8000c00 tx=0x7f6db803d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=396e4bac0a9ba1ed server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:23.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.658+0000 7f6da6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:54:23.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.658+0000 7f6da6ffd640 1 -- 172.21.15.33:0/2932850213 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db805d060 con 0x563ffdd8b380 2024-03-19T02:54:23.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.658+0000 7f6da6ffd640 1 -- 172.21.15.33:0/2932850213 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f6db8004ad0 con 0x563ffdd8b380 2024-03-19T02:54:23.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.658+0000 7f6dd0c30740 1 -- 172.21.15.33:0/2932850213 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6dc009f7c0 con 0x563ffdd8b380 2024-03-19T02:54:23.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.658+0000 7f6da6ffd640 1 -- 172.21.15.33:0/2932850213 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f6db8064050 con 0x563ffdd8b380 2024-03-19T02:54:23.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.658+0000 7f6dd0c30740 10 client.15909 fetch_fsmap learned FSMap version 25 2024-03-19T02:54:23.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.658+0000 7f6dd0c30740 1 -- 172.21.15.33:0/2932850213 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563ffde0b140 con 0x563ffdd8b380 2024-03-19T02:54:23.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.659+0000 7f6da6ffd640 1 -- 172.21.15.33:0/2932850213 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6dc009f7c0 con 0x563ffdd8b380 2024-03-19T02:54:23.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.659+0000 7f6da6ffd640 1 -- 172.21.15.33:0/2932850213 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f6db8040450 con 0x563ffdd8b380 2024-03-19T02:54:23.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.659+0000 7f6dd0c30740 10 client.15909 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:54:23.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.659+0000 7f6dd0c30740 10 client.15909 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:54:23.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.659+0000 7f6dd0c30740 20 client.15909 populate_metadata read hostname 'smithi033' 2024-03-19T02:54:23.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.659+0000 7f6dd0c30740 1 --2- 172.21.15.33:0/2932850213 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x563ffde0b810 0x563ffde0dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.659+0000 7f6dd0c30740 4 client.15909 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:54:23.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.659+0000 7f6dd0c30740 1 -- 172.21.15.33:0/2932850213 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x563ffdcf69f0 con 0x563ffde0b810 2024-03-19T02:54:23.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.659+0000 7f6dc6ffd640 1 --2- 172.21.15.33:0/2932850213 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x563ffde0b810 0x563ffde0dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.659+0000 7f6da6ffd640 10 client.15909 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:54:23.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.659+0000 7f6dc6ffd640 1 --2- 172.21.15.33:0/2932850213 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x563ffde0b810 0x563ffde0dc20 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9186151bad918d43 server_cookie=90c4ede91ead6064 in_seq=0 out_seq=0 2024-03-19T02:54:23.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.660+0000 7f6da6ffd640 1 -- 172.21.15.33:0/2932850213 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x563ffdcf69f0 con 0x563ffde0b810 2024-03-19T02:54:23.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.660+0000 7f6da6ffd640 10 client.15909 handle_command_reply: tid=0 2024-03-19T02:54:23.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.660+0000 7f6dd0c30740 1 client.15909 shutdown 2024-03-19T02:54:23.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.661+0000 7f6dd0c30740 1 -- 172.21.15.33:0/2932850213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ffdd8b380 msgr2=0x563ffdd8e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:23.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.661+0000 7f6dd0c30740 1 --2- 172.21.15.33:0/2932850213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ffdd8b380 0x563ffdd8e4f0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f6db8000c00 tx=0x7f6db803d000 comp rx=0 tx=0).stop 2024-03-19T02:54:23.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.661+0000 7f6dd0c30740 1 -- 172.21.15.33:0/2932850213 shutdown_connections 2024-03-19T02:54:23.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.661+0000 7f6dd0c30740 1 --2- 172.21.15.33:0/2932850213 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x563ffde0b810 0x563ffde0dc20 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.661+0000 7f6dd0c30740 1 --2- 172.21.15.33:0/2932850213 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x563ffdd8bca0 0x563ffdd8ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.661+0000 7f6dd0c30740 1 --2- 172.21.15.33:0/2932850213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563ffdd94880 0x563ffdd8ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.661+0000 7f6dd0c30740 1 --2- 172.21.15.33:0/2932850213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x563ffdd8b380 0x563ffdd8e4f0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.661+0000 7f6dd0c30740 1 -- 172.21.15.33:0/2932850213 >> 172.21.15.33:0/2932850213 conn(0x563ffdd85ab0 msgr2=0x563ffdd91ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:23.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.661+0000 7f6da6ffd640 0 client.15909 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:54:23.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.661+0000 7f6dd0c30740 1 -- 172.21.15.33:0/2932850213 shutdown_connections 2024-03-19T02:54:23.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.661+0000 7f6dd0c30740 1 -- 172.21.15.33:0/2932850213 wait complete. 2024-03-19T02:54:23.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.661+0000 7f6dd0c30740 20 client.15909 trim_cache size 0 max 16384 2024-03-19T02:54:23.667 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:54:23.667 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-03-19T02:54:23.667 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "1d0d9f33-87ad-4c9f-8c7f-00ea941ecd6e", 2024-03-19T02:54:23.668 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-03-19T02:54:23.668 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:54:23.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.661+0000 7f6dce04d640 1 -- 172.21.15.33:0/760023982 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6db00a2a90 msgr2=0x7f6db00a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:23.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.661+0000 7f6dce04d640 1 --2- 172.21.15.33:0/760023982 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6db00a2a90 0x7f6db00a4f50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6db8006010 tx=0x7f6db8009610 comp rx=0 tx=0).stop 2024-03-19T02:54:23.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.661+0000 7f6dce04d640 1 -- 172.21.15.33:0/760023982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6dc81066a0 msgr2=0x7f6dc81a3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:23.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.661+0000 7f6dce04d640 1 --2- 172.21.15.33:0/760023982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6dc81066a0 0x7f6dc81a3ee0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f6dc000eaa0 tx=0x7f6dc000ef60 comp rx=0 tx=0).stop 2024-03-19T02:54:23.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.661+0000 7f6dce04d640 1 -- 172.21.15.33:0/760023982 shutdown_connections 2024-03-19T02:54:23.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.661+0000 7f6dce04d640 1 --2- 172.21.15.33:0/760023982 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6db00a2a90 0x7f6db00a4f50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.662+0000 7f6dce04d640 1 --2- 172.21.15.33:0/760023982 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6dc81066a0 0x7f6dc81a3ee0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.662+0000 7f6dce04d640 1 --2- 172.21.15.33:0/760023982 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6dc8101050 0x7f6dc81993a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.662+0000 7f6dce04d640 1 --2- 172.21.15.33:0/760023982 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6dc8100690 0x7f6dc8198e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.662+0000 7f6dce04d640 1 -- 172.21.15.33:0/760023982 >> 172.21.15.33:0/760023982 conn(0x7f6dc8077820 msgr2=0x7f6dc8102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:23.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.662+0000 7f6dce04d640 1 -- 172.21.15.33:0/760023982 shutdown_connections 2024-03-19T02:54:23.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.662+0000 7f6dce04d640 1 -- 172.21.15.33:0/760023982 wait complete. 2024-03-19T02:54:23.677 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:54:23.677 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:54:23.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.798+0000 7f587b960640 1 -- 172.21.15.33:0/3588114377 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5874106750 msgr2=0x7f5874106b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:23.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.798+0000 7f587b960640 1 --2- 172.21.15.33:0/3588114377 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5874106750 0x7f5874106b30 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f5870009930 tx=0x7f587002bc30 comp rx=0 tx=0).stop 2024-03-19T02:54:23.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.798+0000 7f587b960640 1 -- 172.21.15.33:0/3588114377 shutdown_connections 2024-03-19T02:54:23.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.798+0000 7f587b960640 1 --2- 172.21.15.33:0/3588114377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5874101100 0x7f587410d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.798+0000 7f587b960640 1 --2- 172.21.15.33:0/3588114377 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5874100740 0x7f5874100bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.798+0000 7f587b960640 1 --2- 172.21.15.33:0/3588114377 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5874106750 0x7f5874106b30 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.798+0000 7f587b960640 1 -- 172.21.15.33:0/3588114377 >> 172.21.15.33:0/3588114377 conn(0x7f5874077880 msgr2=0x7f5874077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:23.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.798+0000 7f587b960640 1 -- 172.21.15.33:0/3588114377 shutdown_connections 2024-03-19T02:54:23.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.798+0000 7f587b960640 1 -- 172.21.15.33:0/3588114377 wait complete. 2024-03-19T02:54:23.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.798+0000 7f587b960640 1 Processor -- start 2024-03-19T02:54:23.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.798+0000 7f587b960640 1 -- start start 2024-03-19T02:54:23.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.799+0000 7f587b960640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5874100740 0x7f587419a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.799+0000 7f587b960640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5874101100 0x7f587419a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.799+0000 7f587b960640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5874106750 0x7f587419ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.799+0000 7f587b960640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f58741109d0 con 0x7f5874101100 2024-03-19T02:54:23.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.799+0000 7f587b960640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5874110850 con 0x7f5874100740 2024-03-19T02:54:23.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.799+0000 7f587b960640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5874110b50 con 0x7f5874106750 2024-03-19T02:54:23.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.799+0000 7f5878efb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5874101100 0x7f587419a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.799+0000 7f5878efb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5874101100 0x7f587419a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33638/0 (socket says 172.21.15.33:33638) 2024-03-19T02:54:23.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.799+0000 7f5878efb640 1 -- 172.21.15.33:0/3339068516 learned_addr learned my addr 172.21.15.33:0/3339068516 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:23.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.799+0000 7f58796fc640 1 --2- 172.21.15.33:0/3339068516 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5874100740 0x7f587419a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.799+0000 7f5879efd640 1 --2- 172.21.15.33:0/3339068516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5874106750 0x7f587419ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.799+0000 7f5878efb640 1 -- 172.21.15.33:0/3339068516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5874106750 msgr2=0x7f587419ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:23.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.799+0000 7f5878efb640 1 --2- 172.21.15.33:0/3339068516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5874106750 0x7f587419ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.799+0000 7f5878efb640 1 -- 172.21.15.33:0/3339068516 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5874100740 msgr2=0x7f587419a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:23.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.799+0000 7f5878efb640 1 --2- 172.21.15.33:0/3339068516 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5874100740 0x7f587419a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.799+0000 7f5878efb640 1 -- 172.21.15.33:0/3339068516 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f587419f300 con 0x7f5874101100 2024-03-19T02:54:23.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.799+0000 7f58796fc640 1 --2- 172.21.15.33:0/3339068516 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5874100740 0x7f587419a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:23.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.799+0000 7f5878efb640 1 --2- 172.21.15.33:0/3339068516 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5874101100 0x7f587419a850 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f586400e9f0 tx=0x7f586400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:23.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.799+0000 7f586a7fc640 1 -- 172.21.15.33:0/3339068516 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f586400cd80 con 0x7f5874101100 2024-03-19T02:54:23.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.800+0000 7f586a7fc640 1 -- 172.21.15.33:0/3339068516 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f58640108e0 con 0x7f5874101100 2024-03-19T02:54:23.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.800+0000 7f587b960640 1 -- 172.21.15.33:0/3339068516 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f587419f5f0 con 0x7f5874101100 2024-03-19T02:54:23.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.800+0000 7f586a7fc640 1 -- 172.21.15.33:0/3339068516 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5864018920 con 0x7f5874101100 2024-03-19T02:54:23.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.800+0000 7f587b960640 1 -- 172.21.15.33:0/3339068516 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58741a6e80 con 0x7f5874101100 2024-03-19T02:54:23.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.802+0000 7f587e620740 1 Processor -- start 2024-03-19T02:54:23.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.803+0000 7f587e620740 1 -- start start 2024-03-19T02:54:23.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.803+0000 7f586a7fc640 1 -- 172.21.15.33:0/3339068516 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7f5864010a90 con 0x7f5874101100 2024-03-19T02:54:23.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.803+0000 7f587e620740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559de182da30 0x559de182de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.803+0000 7f587e620740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559de182e350 0x559de18369f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.803+0000 7f587e620740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559de1836f30 0x559de1839320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.803+0000 7f587e620740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559de1791500 con 0x559de182da30 2024-03-19T02:54:23.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.803+0000 7f587e620740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x559de1739be0 con 0x559de182e350 2024-03-19T02:54:23.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.803+0000 7f587e620740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x559de178d5a0 con 0x559de1836f30 2024-03-19T02:54:23.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.803+0000 7f5879efd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559de182da30 0x559de182de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.803+0000 7f5879efd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559de182da30 0x559de182de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33646/0 (socket says 172.21.15.33:33646) 2024-03-19T02:54:23.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.803+0000 7f5879efd640 1 -- 172.21.15.33:0/1289225358 learned_addr learned my addr 172.21.15.33:0/1289225358 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:23.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.803+0000 7f58796fc640 1 --2- 172.21.15.33:0/1289225358 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559de182e350 0x559de18369f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.803+0000 7f5878efb640 1 --2- 172.21.15.33:0/1289225358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559de1836f30 0x559de1839320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.803+0000 7f58796fc640 1 -- 172.21.15.33:0/1289225358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559de1836f30 msgr2=0x559de1839320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:23.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.803+0000 7f58796fc640 1 --2- 172.21.15.33:0/1289225358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559de1836f30 0x559de1839320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.803+0000 7f58796fc640 1 -- 172.21.15.33:0/1289225358 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559de182da30 msgr2=0x559de182de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:23.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.803+0000 7f58796fc640 1 --2- 172.21.15.33:0/1289225358 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559de182da30 0x559de182de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.803+0000 7f58796fc640 1 -- 172.21.15.33:0/1289225358 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559de175b000 con 0x559de182e350 2024-03-19T02:54:23.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.803+0000 7f5879efd640 1 --2- 172.21.15.33:0/1289225358 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559de182da30 0x559de182de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:23.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.803+0000 7f586a7fc640 1 --2- 172.21.15.33:0/3339068516 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f58600a2b10 0x7f58600a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.804+0000 7f5879efd640 1 --2- 172.21.15.33:0/3339068516 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f58600a2b10 0x7f58600a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.804+0000 7f586a7fc640 1 -- 172.21.15.33:0/3339068516 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f58640cf710 con 0x7f5874101100 2024-03-19T02:54:23.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.804+0000 7f58796fc640 1 --2- 172.21.15.33:0/1289225358 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559de182e350 0x559de18369f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5870009230 tx=0x7f58700095f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fe4f2f04081aa2a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:23.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.804+0000 7f585f7fe640 1 -- 172.21.15.33:0/1289225358 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5870045070 con 0x559de182e350 2024-03-19T02:54:23.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.804+0000 7f585f7fe640 1 -- 172.21.15.33:0/1289225358 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f587002c7e0 con 0x559de182e350 2024-03-19T02:54:23.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.804+0000 7f585f7fe640 1 -- 172.21.15.33:0/1289225358 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5870003a40 con 0x559de182e350 2024-03-19T02:54:23.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.805+0000 7f587e620740 1 -- 172.21.15.33:0/1289225358 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559de182e350 msgr2=0x559de18369f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:23.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.805+0000 7f5879efd640 1 --2- 172.21.15.33:0/3339068516 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f58600a2b10 0x7f58600a4fd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f586c009fd0 tx=0x7f586c009500 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:23.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.805+0000 7f587e620740 1 --2- 172.21.15.33:0/1289225358 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559de182e350 0x559de18369f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5870009230 tx=0x7f58700095f0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.805+0000 7f587e620740 1 -- 172.21.15.33:0/1289225358 shutdown_connections 2024-03-19T02:54:23.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.805+0000 7f587e620740 1 --2- 172.21.15.33:0/1289225358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559de1836f30 0x559de1839320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.805+0000 7f587e620740 1 --2- 172.21.15.33:0/1289225358 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559de182e350 0x559de18369f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.805+0000 7f587e620740 1 --2- 172.21.15.33:0/1289225358 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559de182da30 0x559de182de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.805+0000 7f587e620740 1 -- 172.21.15.33:0/1289225358 >> 172.21.15.33:0/1289225358 conn(0x559de1828160 msgr2=0x559de1828540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:23.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.805+0000 7f587e620740 1 -- 172.21.15.33:0/1289225358 shutdown_connections 2024-03-19T02:54:23.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.805+0000 7f587e620740 1 -- 172.21.15.33:0/1289225358 wait complete. 2024-03-19T02:54:23.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.805+0000 7f587e620740 1 -- start start 2024-03-19T02:54:23.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.805+0000 7f587e620740 1 Processor -- start 2024-03-19T02:54:23.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.806+0000 7f587e620740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559de182da30 0x559de1833fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.806+0000 7f587e620740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559de182e350 0x559de18344e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.806+0000 7f587e620740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559de1836f30 0x559de1834a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.806+0000 7f587e620740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559de1839cc0 con 0x559de1836f30 2024-03-19T02:54:23.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.806+0000 7f587e620740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x559de174d160 con 0x559de182da30 2024-03-19T02:54:23.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.806+0000 7f587e620740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x559de1839e40 con 0x559de182e350 2024-03-19T02:54:23.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.806+0000 7f587e620740 10 client.0 _ll_register_callbacks cb 0x559de1789a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:54:23.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.806+0000 7f58796fc640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559de182da30 0x559de1833fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.806+0000 7f5879efd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559de1836f30 0x559de1834a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.806+0000 7f58796fc640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559de182da30 0x559de1833fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:58596/0 (socket says 172.21.15.33:58596) 2024-03-19T02:54:23.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.806+0000 7f58796fc640 1 -- 172.21.15.33:0/934418564 learned_addr learned my addr 172.21.15.33:0/934418564 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:23.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.806+0000 7f5878efb640 1 --2- 172.21.15.33:0/934418564 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559de182e350 0x559de18344e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.806+0000 7f58796fc640 1 -- 172.21.15.33:0/934418564 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559de182e350 msgr2=0x559de18344e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:23.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.806+0000 7f58796fc640 1 --2- 172.21.15.33:0/934418564 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559de182e350 0x559de18344e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.806+0000 7f58796fc640 1 -- 172.21.15.33:0/934418564 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559de1836f30 msgr2=0x559de1834a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:23.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.806+0000 7f58796fc640 1 --2- 172.21.15.33:0/934418564 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559de1836f30 0x559de1834a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.806+0000 7f58796fc640 1 -- 172.21.15.33:0/934418564 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559de156f7d0 con 0x559de182da30 2024-03-19T02:54:23.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.806+0000 7f5879efd640 1 --2- 172.21.15.33:0/934418564 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559de1836f30 0x559de1834a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:23.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.806+0000 7f58796fc640 1 --2- 172.21.15.33:0/934418564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559de182da30 0x559de1833fa0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5870005ab0 tx=0x7f5870005e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62fb6070bf051fcb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:23.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.806+0000 7f585cff9640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T02:54:23.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.806+0000 7f585cff9640 1 -- 172.21.15.33:0/934418564 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5870045040 con 0x559de182da30 2024-03-19T02:54:23.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.807+0000 7f585cff9640 1 -- 172.21.15.33:0/934418564 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f587003fc80 con 0x559de182da30 2024-03-19T02:54:23.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.807+0000 7f587e620740 1 -- 172.21.15.33:0/934418564 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f587002c7e0 con 0x559de182da30 2024-03-19T02:54:23.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.807+0000 7f585cff9640 1 -- 172.21.15.33:0/934418564 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f587005b050 con 0x559de182da30 2024-03-19T02:54:23.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.807+0000 7f587e620740 10 client.25790 fetch_fsmap learned FSMap version 25 2024-03-19T02:54:23.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.807+0000 7f587e620740 1 -- 172.21.15.33:0/934418564 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559de14a53f0 con 0x559de182da30 2024-03-19T02:54:23.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.807+0000 7f585cff9640 1 -- 172.21.15.33:0/934418564 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f587002c7e0 con 0x559de182da30 2024-03-19T02:54:23.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.807+0000 7f585cff9640 1 -- 172.21.15.33:0/934418564 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f5870034440 con 0x559de182da30 2024-03-19T02:54:23.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.807+0000 7f587e620740 10 client.25790 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:54:23.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.807+0000 7f587e620740 10 client.25790 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:54:23.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.807+0000 7f587e620740 20 client.25790 populate_metadata read hostname 'smithi033' 2024-03-19T02:54:23.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.807+0000 7f587e620740 1 --2- 172.21.15.33:0/934418564 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x559de1847130 0x559de1849520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.807+0000 7f587e620740 4 client.25790 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:54:23.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.807+0000 7f587e620740 1 -- 172.21.15.33:0/934418564 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559de179ea40 con 0x559de1847130 2024-03-19T02:54:23.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.807+0000 7f5878efb640 1 --2- 172.21.15.33:0/934418564 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x559de1847130 0x559de1849520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.808+0000 7f5878efb640 1 --2- 172.21.15.33:0/934418564 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x559de1847130 0x559de1849520 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f74604ea4edac59 server_cookie=b83b82ef353731e6 in_seq=0 out_seq=0 2024-03-19T02:54:23.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.808+0000 7f585cff9640 10 client.25790 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:54:23.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.808+0000 7f585cff9640 1 -- 172.21.15.33:0/934418564 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559de179ea40 con 0x559de1847130 2024-03-19T02:54:23.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.808+0000 7f585cff9640 10 client.25790 handle_command_reply: tid=0 2024-03-19T02:54:23.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.808+0000 7f587e620740 1 client.25790 shutdown 2024-03-19T02:54:23.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.808+0000 7f587e620740 1 -- 172.21.15.33:0/934418564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559de182da30 msgr2=0x559de1833fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:23.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.808+0000 7f587e620740 1 --2- 172.21.15.33:0/934418564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559de182da30 0x559de1833fa0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5870005ab0 tx=0x7f5870005e50 comp rx=0 tx=0).stop 2024-03-19T02:54:23.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.809+0000 7f587e620740 1 -- 172.21.15.33:0/934418564 shutdown_connections 2024-03-19T02:54:23.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.809+0000 7f587e620740 1 --2- 172.21.15.33:0/934418564 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x559de1847130 0x559de1849520 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.809+0000 7f587e620740 1 --2- 172.21.15.33:0/934418564 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559de182e350 0x559de18344e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.809+0000 7f587e620740 1 --2- 172.21.15.33:0/934418564 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559de1836f30 0x559de1834a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.809+0000 7f587e620740 1 --2- 172.21.15.33:0/934418564 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559de182da30 0x559de1833fa0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.809+0000 7f585cff9640 0 client.25790 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:54:23.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.809+0000 7f587e620740 1 -- 172.21.15.33:0/934418564 >> 172.21.15.33:0/934418564 conn(0x559de1828160 msgr2=0x559de1829eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:23.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.809+0000 7f587e620740 1 -- 172.21.15.33:0/934418564 shutdown_connections 2024-03-19T02:54:23.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.809+0000 7f587e620740 1 -- 172.21.15.33:0/934418564 wait complete. 2024-03-19T02:54:23.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.809+0000 7f587e620740 20 client.25790 trim_cache size 0 max 16384 2024-03-19T02:54:23.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.834+0000 7f587e620740 1 Processor -- start 2024-03-19T02:54:23.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.834+0000 7f587e620740 1 -- start start 2024-03-19T02:54:23.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.834+0000 7f587e620740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559de182da30 0x559de18366d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.834+0000 7f587e620740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559de182e350 0x559de1842a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.834+0000 7f587e620740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559de1836f30 0x559de184eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.834+0000 7f587e620740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f58741b3580 con 0x559de182e350 2024-03-19T02:54:23.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.834+0000 7f587e620740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f58741b0e50 con 0x559de1836f30 2024-03-19T02:54:23.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.834+0000 7f587e620740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f58741aec80 con 0x559de182da30 2024-03-19T02:54:23.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.834+0000 7f5878efb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559de182e350 0x559de1842a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.834+0000 7f5878efb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559de182e350 0x559de1842a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33666/0 (socket says 172.21.15.33:33666) 2024-03-19T02:54:23.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.834+0000 7f5878efb640 1 -- 172.21.15.33:0/2825642625 learned_addr learned my addr 172.21.15.33:0/2825642625 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:23.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.834+0000 7f58796fc640 1 --2- 172.21.15.33:0/2825642625 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559de182da30 0x559de18366d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.834+0000 7f5879efd640 1 --2- 172.21.15.33:0/2825642625 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559de1836f30 0x559de184eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.834+0000 7f5879efd640 1 -- 172.21.15.33:0/2825642625 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559de182da30 msgr2=0x559de18366d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:23.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.834+0000 7f5879efd640 1 --2- 172.21.15.33:0/2825642625 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559de182da30 0x559de18366d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.834+0000 7f5879efd640 1 -- 172.21.15.33:0/2825642625 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559de182e350 msgr2=0x559de1842a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:23.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.834+0000 7f5879efd640 1 --2- 172.21.15.33:0/2825642625 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559de182e350 0x559de1842a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.834+0000 7f5879efd640 1 -- 172.21.15.33:0/2825642625 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559de14650f0 con 0x559de1836f30 2024-03-19T02:54:23.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.834+0000 7f5878efb640 1 --2- 172.21.15.33:0/2825642625 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559de182e350 0x559de1842a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:23.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.835+0000 7f5879efd640 1 --2- 172.21.15.33:0/2825642625 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559de1836f30 0x559de184eb80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f586c00f790 tx=0x7f586c016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9cd3e5e3c412e98 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:23.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.835+0000 7f585effd640 1 -- 172.21.15.33:0/2825642625 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f586c00fd20 con 0x559de1836f30 2024-03-19T02:54:23.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.835+0000 7f585effd640 1 -- 172.21.15.33:0/2825642625 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f586c003970 con 0x559de1836f30 2024-03-19T02:54:23.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.835+0000 7f585effd640 1 -- 172.21.15.33:0/2825642625 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f586c01ce30 con 0x559de1836f30 2024-03-19T02:54:23.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.835+0000 7f587e620740 1 -- 172.21.15.33:0/2825642625 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559de1836f30 msgr2=0x559de184eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:23.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.836+0000 7f587e620740 1 --2- 172.21.15.33:0/2825642625 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559de1836f30 0x559de184eb80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f586c00f790 tx=0x7f586c016000 comp rx=0 tx=0).stop 2024-03-19T02:54:23.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.836+0000 7f587e620740 1 -- 172.21.15.33:0/2825642625 shutdown_connections 2024-03-19T02:54:23.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.836+0000 7f587e620740 1 --2- 172.21.15.33:0/2825642625 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559de1836f30 0x559de184eb80 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.836+0000 7f587e620740 1 --2- 172.21.15.33:0/2825642625 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559de182e350 0x559de1842a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.836+0000 7f587e620740 1 --2- 172.21.15.33:0/2825642625 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559de182da30 0x559de18366d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.836+0000 7f587e620740 1 -- 172.21.15.33:0/2825642625 >> 172.21.15.33:0/2825642625 conn(0x559de1828160 msgr2=0x559de1826950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:23.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.836+0000 7f587e620740 1 -- 172.21.15.33:0/2825642625 shutdown_connections 2024-03-19T02:54:23.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.836+0000 7f587e620740 1 -- 172.21.15.33:0/2825642625 wait complete. 2024-03-19T02:54:23.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.836+0000 7f587e620740 1 -- start start 2024-03-19T02:54:23.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.836+0000 7f587e620740 1 Processor -- start 2024-03-19T02:54:23.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.836+0000 7f587e620740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559de182da30 0x559de1830c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.836+0000 7f587e620740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559de182e350 0x559de18311b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.836+0000 7f587e620740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559de1836f30 0x559de18316f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.836+0000 7f587e620740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f58741af310 con 0x559de182e350 2024-03-19T02:54:23.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.836+0000 7f587e620740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f58741b0780 con 0x559de182da30 2024-03-19T02:54:23.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.836+0000 7f587e620740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f58741af9e0 con 0x559de1836f30 2024-03-19T02:54:23.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.837+0000 7f587e620740 10 client.0 _ll_register_callbacks cb 0x7f586c009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:54:23.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.837+0000 7f58796fc640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559de182da30 0x559de1830c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.837+0000 7f5878efb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559de182e350 0x559de18311b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.837+0000 7f5879efd640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559de1836f30 0x559de18316f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.837+0000 7f58796fc640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559de182da30 0x559de1830c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:58614/0 (socket says 172.21.15.33:58614) 2024-03-19T02:54:23.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.837+0000 7f5878efb640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559de182e350 0x559de18311b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:33670/0 (socket says 172.21.15.33:33670) 2024-03-19T02:54:23.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.837+0000 7f58796fc640 1 -- 172.21.15.33:0/1058000644 learned_addr learned my addr 172.21.15.33:0/1058000644 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:23.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.837+0000 7f58796fc640 1 -- 172.21.15.33:0/1058000644 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559de1836f30 msgr2=0x559de18316f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:23.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.837+0000 7f58796fc640 1 --2- 172.21.15.33:0/1058000644 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559de1836f30 0x559de18316f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.837+0000 7f58796fc640 1 -- 172.21.15.33:0/1058000644 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559de182e350 msgr2=0x559de18311b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:23.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.837+0000 7f58796fc640 1 --2- 172.21.15.33:0/1058000644 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559de182e350 0x559de18311b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.837+0000 7f58796fc640 1 -- 172.21.15.33:0/1058000644 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559de18ac2b0 con 0x559de182da30 2024-03-19T02:54:23.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.837+0000 7f5878efb640 1 --2- 172.21.15.33:0/1058000644 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559de182e350 0x559de18311b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:23.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.837+0000 7f58796fc640 1 --2- 172.21.15.33:0/1058000644 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559de182da30 0x559de1830c70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5870000c00 tx=0x7f5870034670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=795d40654cea5293 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:23.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.837+0000 7f585cff9640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T02:54:23.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.837+0000 7f585cff9640 1 -- 172.21.15.33:0/1058000644 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f587004e040 con 0x559de182da30 2024-03-19T02:54:23.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.837+0000 7f587e620740 1 -- 172.21.15.33:0/1058000644 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f586c003970 con 0x559de182da30 2024-03-19T02:54:23.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.837+0000 7f585cff9640 1 -- 172.21.15.33:0/1058000644 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f587003fc80 con 0x559de182da30 2024-03-19T02:54:23.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.838+0000 7f585cff9640 1 -- 172.21.15.33:0/1058000644 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f5870045040 con 0x559de182da30 2024-03-19T02:54:23.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.838+0000 7f587e620740 10 client.25796 fetch_fsmap learned FSMap version 25 2024-03-19T02:54:23.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.838+0000 7f587e620740 1 -- 172.21.15.33:0/1058000644 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559de18ad580 con 0x559de182da30 2024-03-19T02:54:23.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.838+0000 7f585cff9640 1 -- 172.21.15.33:0/1058000644 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f586c003970 con 0x559de182da30 2024-03-19T02:54:23.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.838+0000 7f585cff9640 1 -- 172.21.15.33:0/1058000644 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f5870003680 con 0x559de182da30 2024-03-19T02:54:23.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.838+0000 7f587e620740 10 client.25796 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:54:23.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.838+0000 7f587e620740 10 client.25796 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:54:23.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.838+0000 7f587e620740 20 client.25796 populate_metadata read hostname 'smithi033' 2024-03-19T02:54:23.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.838+0000 7f587e620740 1 --2- 172.21.15.33:0/1058000644 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x559de18adcc0 0x559de18b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:23.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.838+0000 7f587e620740 4 client.25796 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:54:23.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.838+0000 7f587e620740 1 -- 172.21.15.33:0/1058000644 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559de1798d90 con 0x559de18adcc0 2024-03-19T02:54:23.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.838+0000 7f5878efb640 1 --2- 172.21.15.33:0/1058000644 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x559de18adcc0 0x559de18b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:23.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.838+0000 7f585cff9640 10 client.25796 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:54:23.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.838+0000 7f5878efb640 1 --2- 172.21.15.33:0/1058000644 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x559de18adcc0 0x559de18b00e0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcde9f1003c30024 server_cookie=79973c9e09be24a3 in_seq=0 out_seq=0 2024-03-19T02:54:23.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.839+0000 7f585cff9640 1 -- 172.21.15.33:0/1058000644 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x559de1798d90 con 0x559de18adcc0 2024-03-19T02:54:23.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.839+0000 7f585cff9640 10 client.25796 handle_command_reply: tid=0 2024-03-19T02:54:23.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.839+0000 7f587e620740 1 client.25796 shutdown 2024-03-19T02:54:23.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.839+0000 7f587e620740 1 -- 172.21.15.33:0/1058000644 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559de182da30 msgr2=0x559de1830c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:23.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.839+0000 7f587e620740 1 --2- 172.21.15.33:0/1058000644 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559de182da30 0x559de1830c70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5870000c00 tx=0x7f5870034670 comp rx=0 tx=0).stop 2024-03-19T02:54:23.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.839+0000 7f587e620740 1 -- 172.21.15.33:0/1058000644 shutdown_connections 2024-03-19T02:54:23.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.839+0000 7f587e620740 1 --2- 172.21.15.33:0/1058000644 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x559de18adcc0 0x559de18b00e0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.839+0000 7f587e620740 1 --2- 172.21.15.33:0/1058000644 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559de1836f30 0x559de18316f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.839+0000 7f585cff9640 0 client.25796 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:54:23.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.839+0000 7f587e620740 1 --2- 172.21.15.33:0/1058000644 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559de182e350 0x559de18311b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.839+0000 7f587e620740 1 --2- 172.21.15.33:0/1058000644 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559de182da30 0x559de1830c70 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.839+0000 7f587e620740 1 -- 172.21.15.33:0/1058000644 >> 172.21.15.33:0/1058000644 conn(0x559de1828160 msgr2=0x559de18271e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:23.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.839+0000 7f587e620740 1 -- 172.21.15.33:0/1058000644 shutdown_connections 2024-03-19T02:54:23.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.839+0000 7f587e620740 1 -- 172.21.15.33:0/1058000644 wait complete. 2024-03-19T02:54:23.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.839+0000 7f587e620740 20 client.25796 trim_cache size 0 max 16384 2024-03-19T02:54:23.848 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:54:23.848 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:54:23.848 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-03-19T02:54:23.848 INFO:teuthology.orchestra.run.smithi033.stdout: "1d0d9f33-87ad-4c9f-8c7f-00ea941ecd6e": { 2024-03-19T02:54:23.848 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-03-19T02:54:23.848 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "1d0d9f33-87ad-4c9f-8c7f-00ea941ecd6e", 2024-03-19T02:54:23.848 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-03-19T02:54:23.849 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T02:54:23.849 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T02:54:23.849 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:54:23.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.839+0000 7f587b960640 1 -- 172.21.15.33:0/3339068516 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f58600a2b10 msgr2=0x7f58600a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:23.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.839+0000 7f587b960640 1 --2- 172.21.15.33:0/3339068516 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f58600a2b10 0x7f58600a4fd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f586c009fd0 tx=0x7f586c009500 comp rx=0 tx=0).stop 2024-03-19T02:54:23.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.839+0000 7f587b960640 1 -- 172.21.15.33:0/3339068516 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5874101100 msgr2=0x7f587419a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:23.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.839+0000 7f587b960640 1 --2- 172.21.15.33:0/3339068516 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5874101100 0x7f587419a850 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f586400e9f0 tx=0x7f586400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.840+0000 7f587b960640 1 -- 172.21.15.33:0/3339068516 shutdown_connections 2024-03-19T02:54:23.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.840+0000 7f587b960640 1 --2- 172.21.15.33:0/3339068516 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f58600a2b10 0x7f58600a4fd0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.840+0000 7f587b960640 1 --2- 172.21.15.33:0/3339068516 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5874106750 0x7f587419ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.840+0000 7f587b960640 1 --2- 172.21.15.33:0/3339068516 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5874101100 0x7f587419a850 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.840+0000 7f587b960640 1 --2- 172.21.15.33:0/3339068516 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5874100740 0x7f587419a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:23.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.840+0000 7f587b960640 1 -- 172.21.15.33:0/3339068516 >> 172.21.15.33:0/3339068516 conn(0x7f5874077880 msgr2=0x7f587410bf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:23.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.840+0000 7f587b960640 1 -- 172.21.15.33:0/3339068516 shutdown_connections 2024-03-19T02:54:23.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:23.840+0000 7f587b960640 1 -- 172.21.15.33:0/3339068516 wait complete. 2024-03-19T02:54:23.853 INFO:tasks.cephfs.filesystem:scrub status for tag:1d0d9f33-87ad-4c9f-8c7f-00ea941ecd6e - {'path': '/', 'tag': '1d0d9f33-87ad-4c9f-8c7f-00ea941ecd6e', 'options': 'recursive,force'} 2024-03-19T02:54:24.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:24 smithi096 ceph-mon[35946]: pgmap v31: 129 pgs: 129 active+clean; 681 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 57 MiB/s wr, 23 op/s 2024-03-19T02:54:24.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:24 smithi096 ceph-mon[35946]: scrub queued for path: / 2024-03-19T02:54:24.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:24 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T02:54:24.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:24 smithi096 ceph-mon[35946]: scrub summary: active paths [/] 2024-03-19T02:54:24.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:24 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T02:54:25.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:24 smithi155 ceph-mon[35804]: pgmap v31: 129 pgs: 129 active+clean; 681 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 57 MiB/s wr, 23 op/s 2024-03-19T02:54:25.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:24 smithi155 ceph-mon[35804]: scrub queued for path: / 2024-03-19T02:54:25.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:24 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T02:54:25.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:24 smithi155 ceph-mon[35804]: scrub summary: active paths [/] 2024-03-19T02:54:25.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:24 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T02:54:25.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:24 smithi033 ceph-mon[31973]: pgmap v31: 129 pgs: 129 active+clean; 681 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 57 MiB/s wr, 23 op/s 2024-03-19T02:54:25.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:24 smithi033 ceph-mon[31973]: scrub queued for path: / 2024-03-19T02:54:25.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:24 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T02:54:25.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:24 smithi033 ceph-mon[31973]: scrub summary: active paths [/] 2024-03-19T02:54:25.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:24 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T02:54:25.538 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-03-19T02:54:25.538 INFO:tasks.workunit.client.0.smithi033.stdout: Children see throughput for 1 initial writers = 108673.36 kB/sec 2024-03-19T02:54:25.538 INFO:tasks.workunit.client.0.smithi033.stdout: Parent sees throughput for 1 initial writers = 108671.24 kB/sec 2024-03-19T02:54:25.538 INFO:tasks.workunit.client.0.smithi033.stdout: Min throughput per process = 108673.36 kB/sec 2024-03-19T02:54:25.538 INFO:tasks.workunit.client.0.smithi033.stdout: Max throughput per process = 108673.36 kB/sec 2024-03-19T02:54:25.539 INFO:tasks.workunit.client.0.smithi033.stdout: Avg throughput per process = 108673.36 kB/sec 2024-03-19T02:54:25.539 INFO:tasks.workunit.client.0.smithi033.stdout: Min xfer = 1048576.00 kB 2024-03-19T02:54:26.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:26 smithi096 ceph-mon[35946]: pgmap v32: 129 pgs: 129 active+clean; 737 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 61 MiB/s wr, 23 op/s 2024-03-19T02:54:27.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:26 smithi155 ceph-mon[35804]: pgmap v32: 129 pgs: 129 active+clean; 737 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 61 MiB/s wr, 23 op/s 2024-03-19T02:54:27.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:26 smithi033 ceph-mon[31973]: pgmap v32: 129 pgs: 129 active+clean; 737 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 61 MiB/s wr, 23 op/s 2024-03-19T02:54:28.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:27 smithi155 ceph-mon[35804]: scrub summary: idle 2024-03-19T02:54:28.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:27 smithi033 ceph-mon[31973]: scrub summary: idle 2024-03-19T02:54:28.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:27 smithi096 ceph-mon[35946]: scrub summary: idle 2024-03-19T02:54:29.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:28 smithi155 ceph-mon[35804]: pgmap v33: 129 pgs: 129 active+clean; 737 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 61 MiB/s wr, 22 op/s 2024-03-19T02:54:29.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:28 smithi033 ceph-mon[31973]: pgmap v33: 129 pgs: 129 active+clean; 737 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 61 MiB/s wr, 22 op/s 2024-03-19T02:54:29.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:28 smithi096 ceph-mon[35946]: pgmap v33: 129 pgs: 129 active+clean; 737 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 61 MiB/s wr, 22 op/s 2024-03-19T02:54:31.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:30 smithi155 ceph-mon[35804]: pgmap v34: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 85 MiB/s wr, 31 op/s 2024-03-19T02:54:31.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:30 smithi033 ceph-mon[31973]: pgmap v34: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 85 MiB/s wr, 31 op/s 2024-03-19T02:54:31.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:30 smithi096 ceph-mon[35946]: pgmap v34: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 85 MiB/s wr, 31 op/s 2024-03-19T02:54:33.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:32 smithi155 ceph-mon[35804]: pgmap v35: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 68 MiB/s wr, 23 op/s 2024-03-19T02:54:33.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:32 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:54:33.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:32 smithi033 ceph-mon[31973]: pgmap v35: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 68 MiB/s wr, 23 op/s 2024-03-19T02:54:33.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:32 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:54:33.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:32 smithi096 ceph-mon[35946]: pgmap v35: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 68 MiB/s wr, 23 op/s 2024-03-19T02:54:33.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:32 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:54:34.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:34 smithi096 ceph-mon[35946]: pgmap v36: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 80 MiB/s wr, 29 op/s 2024-03-19T02:54:34.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:34 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:54:34.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:34 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:54:34.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:34 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:54:34.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:34 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:54:34.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:34 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:54:34.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:34 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:54:34.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:34 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:54:34.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:34 smithi155 ceph-mon[35804]: pgmap v36: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 80 MiB/s wr, 29 op/s 2024-03-19T02:54:34.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:34 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:54:34.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:34 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:54:34.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:34 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:54:34.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:34 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:54:34.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:34 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:54:34.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:34 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:54:34.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:34 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:54:34.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:34 smithi033 ceph-mon[31973]: pgmap v36: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 80 MiB/s wr, 29 op/s 2024-03-19T02:54:34.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:34 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:54:34.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:34 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:54:34.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:34 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:54:34.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:34 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:54:34.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:34 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:54:34.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:34 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:54:34.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:34 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:54:36.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:36 smithi096 ceph-mon[35946]: pgmap v37: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 40 MiB/s wr, 16 op/s 2024-03-19T02:54:36.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:36 smithi155 ceph-mon[35804]: pgmap v37: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 40 MiB/s wr, 16 op/s 2024-03-19T02:54:36.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:36 smithi033 ceph-mon[31973]: pgmap v37: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 40 MiB/s wr, 16 op/s 2024-03-19T02:54:38.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:38 smithi096 ceph-mon[35946]: pgmap v38: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 35 MiB/s wr, 15 op/s 2024-03-19T02:54:38.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:38 smithi155 ceph-mon[35804]: pgmap v38: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 35 MiB/s wr, 15 op/s 2024-03-19T02:54:38.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:38 smithi033 ceph-mon[31973]: pgmap v38: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 35 MiB/s wr, 15 op/s 2024-03-19T02:54:40.221 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-03-19T02:54:40.221 INFO:tasks.workunit.client.0.smithi033.stdout: Children see throughput for 1 rewriters = 123875.23 kB/sec 2024-03-19T02:54:40.221 INFO:tasks.workunit.client.0.smithi033.stdout: Parent sees throughput for 1 rewriters = 123871.90 kB/sec 2024-03-19T02:54:40.221 INFO:tasks.workunit.client.0.smithi033.stdout: Min throughput per process = 123875.23 kB/sec 2024-03-19T02:54:40.221 INFO:tasks.workunit.client.0.smithi033.stdout: Max throughput per process = 123875.23 kB/sec 2024-03-19T02:54:40.221 INFO:tasks.workunit.client.0.smithi033.stdout: Avg throughput per process = 123875.23 kB/sec 2024-03-19T02:54:40.221 INFO:tasks.workunit.client.0.smithi033.stdout: Min xfer = 1048576.00 kB 2024-03-19T02:54:40.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:40 smithi096 ceph-mon[35946]: pgmap v39: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 86 MiB/s wr, 30 op/s 2024-03-19T02:54:40.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:40 smithi155 ceph-mon[35804]: pgmap v39: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 86 MiB/s wr, 30 op/s 2024-03-19T02:54:40.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:40 smithi033 ceph-mon[31973]: pgmap v39: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 86 MiB/s wr, 30 op/s 2024-03-19T02:54:42.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:42 smithi096 ceph-mon[35946]: pgmap v40: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 62 MiB/s wr, 21 op/s 2024-03-19T02:54:42.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:42 smithi155 ceph-mon[35804]: pgmap v40: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 62 MiB/s wr, 21 op/s 2024-03-19T02:54:42.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:42 smithi033 ceph-mon[31973]: pgmap v40: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 62 MiB/s wr, 21 op/s 2024-03-19T02:54:44.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:44 smithi096 ceph-mon[35946]: pgmap v41: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 85 MiB/s wr, 29 op/s 2024-03-19T02:54:44.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:44 smithi155 ceph-mon[35804]: pgmap v41: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 85 MiB/s wr, 29 op/s 2024-03-19T02:54:44.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:44 smithi033 ceph-mon[31973]: pgmap v41: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 85 MiB/s wr, 29 op/s 2024-03-19T02:54:44.756 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-03-19T02:54:44.756 INFO:tasks.workunit.client.0.smithi033.stdout: Children see throughput for 1 readers = 7681079.50 kB/sec 2024-03-19T02:54:44.756 INFO:tasks.workunit.client.0.smithi033.stdout: Parent sees throughput for 1 readers = 7670697.32 kB/sec 2024-03-19T02:54:44.756 INFO:tasks.workunit.client.0.smithi033.stdout: Min throughput per process = 7681079.50 kB/sec 2024-03-19T02:54:44.756 INFO:tasks.workunit.client.0.smithi033.stdout: Max throughput per process = 7681079.50 kB/sec 2024-03-19T02:54:44.756 INFO:tasks.workunit.client.0.smithi033.stdout: Avg throughput per process = 7681079.50 kB/sec 2024-03-19T02:54:44.756 INFO:tasks.workunit.client.0.smithi033.stdout: Min xfer = 1048576.00 kB 2024-03-19T02:54:46.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:46 smithi096 ceph-mon[35946]: pgmap v42: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 74 MiB/s wr, 23 op/s 2024-03-19T02:54:46.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:46 smithi155 ceph-mon[35804]: pgmap v42: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 74 MiB/s wr, 23 op/s 2024-03-19T02:54:46.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:46 smithi033 ceph-mon[31973]: pgmap v42: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 74 MiB/s wr, 23 op/s 2024-03-19T02:54:46.967 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-03-19T02:54:46.967 INFO:tasks.workunit.client.0.smithi033.stdout: Children see throughput for 1 re-readers = 7941352.50 kB/sec 2024-03-19T02:54:46.967 INFO:tasks.workunit.client.0.smithi033.stdout: Parent sees throughput for 1 re-readers = 7930183.51 kB/sec 2024-03-19T02:54:46.967 INFO:tasks.workunit.client.0.smithi033.stdout: Min throughput per process = 7941352.50 kB/sec 2024-03-19T02:54:46.967 INFO:tasks.workunit.client.0.smithi033.stdout: Max throughput per process = 7941352.50 kB/sec 2024-03-19T02:54:46.968 INFO:tasks.workunit.client.0.smithi033.stdout: Avg throughput per process = 7941352.50 kB/sec 2024-03-19T02:54:46.968 INFO:tasks.workunit.client.0.smithi033.stdout: Min xfer = 1048576.00 kB 2024-03-19T02:54:48.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:48 smithi096 ceph-mon[35946]: pgmap v43: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 74 MiB/s wr, 23 op/s 2024-03-19T02:54:48.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:48 smithi155 ceph-mon[35804]: pgmap v43: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 74 MiB/s wr, 23 op/s 2024-03-19T02:54:48.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:48 smithi033 ceph-mon[31973]: pgmap v43: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 74 MiB/s wr, 23 op/s 2024-03-19T02:54:50.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:50 smithi096 ceph-mon[35946]: pgmap v44: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 74 MiB/s wr, 26 op/s 2024-03-19T02:54:50.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:50 smithi155 ceph-mon[35804]: pgmap v44: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 74 MiB/s wr, 26 op/s 2024-03-19T02:54:50.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:50 smithi033 ceph-mon[31973]: pgmap v44: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 74 MiB/s wr, 26 op/s 2024-03-19T02:54:52.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:52 smithi096 ceph-mon[35946]: pgmap v45: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 23 MiB/s wr, 11 op/s 2024-03-19T02:54:52.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:52 smithi155 ceph-mon[35804]: pgmap v45: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 23 MiB/s wr, 11 op/s 2024-03-19T02:54:52.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:52 smithi033 ceph-mon[31973]: pgmap v45: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 23 MiB/s wr, 11 op/s 2024-03-19T02:54:53.400 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-03-19T02:54:53.400 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-03-19T02:54:53.400 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-03-19T02:54:53.400 INFO:tasks.workunit.client.0.smithi033.stdout:iozone test complete. 2024-03-19T02:54:53.401 INFO:tasks.workunit.client.0.smithi033.stderr:+ iozone -c -e -s 1024M -r 1M -t 1 -F f2 -i 0 -i 1 2024-03-19T02:54:53.467 INFO:tasks.workunit.client.0.smithi033.stdout: Iozone: Performance Test of File I/O 2024-03-19T02:54:53.468 INFO:tasks.workunit.client.0.smithi033.stdout: Version $Revision: 3.494 $ 2024-03-19T02:54:53.468 INFO:tasks.workunit.client.0.smithi033.stdout: Compiled for 64 bit mode. 2024-03-19T02:54:53.468 INFO:tasks.workunit.client.0.smithi033.stdout: Build: linux-AMD64 2024-03-19T02:54:53.468 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-03-19T02:54:53.468 INFO:tasks.workunit.client.0.smithi033.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-03-19T02:54:53.468 INFO:tasks.workunit.client.0.smithi033.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-03-19T02:54:53.468 INFO:tasks.workunit.client.0.smithi033.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-03-19T02:54:53.468 INFO:tasks.workunit.client.0.smithi033.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-03-19T02:54:53.468 INFO:tasks.workunit.client.0.smithi033.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-03-19T02:54:53.468 INFO:tasks.workunit.client.0.smithi033.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-03-19T02:54:53.468 INFO:tasks.workunit.client.0.smithi033.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-03-19T02:54:53.469 INFO:tasks.workunit.client.0.smithi033.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-03-19T02:54:53.469 INFO:tasks.workunit.client.0.smithi033.stdout: Alexey Skidanov, Sudhir Kumar. 2024-03-19T02:54:53.469 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-03-19T02:54:53.469 INFO:tasks.workunit.client.0.smithi033.stdout: Run began: Tue Mar 19 02:54:53 2024 2024-03-19T02:54:53.469 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-03-19T02:54:53.469 INFO:tasks.workunit.client.0.smithi033.stdout: Include close in write timing 2024-03-19T02:54:53.469 INFO:tasks.workunit.client.0.smithi033.stdout: Include fsync in write timing 2024-03-19T02:54:53.469 INFO:tasks.workunit.client.0.smithi033.stdout: File size set to 1048576 kB 2024-03-19T02:54:53.469 INFO:tasks.workunit.client.0.smithi033.stdout: Record Size 1024 kB 2024-03-19T02:54:53.469 INFO:tasks.workunit.client.0.smithi033.stdout: Command line used: iozone -c -e -s 1024M -r 1M -t 1 -F f2 -i 0 -i 1 2024-03-19T02:54:53.469 INFO:tasks.workunit.client.0.smithi033.stdout: Output is in kBytes/sec 2024-03-19T02:54:53.469 INFO:tasks.workunit.client.0.smithi033.stdout: Time Resolution = 0.000001 seconds. 2024-03-19T02:54:53.469 INFO:tasks.workunit.client.0.smithi033.stdout: Processor cache size set to 1024 kBytes. 2024-03-19T02:54:53.470 INFO:tasks.workunit.client.0.smithi033.stdout: Processor cache line size set to 32 bytes. 2024-03-19T02:54:53.470 INFO:tasks.workunit.client.0.smithi033.stdout: File stride size set to 17 * record size. 2024-03-19T02:54:53.470 INFO:tasks.workunit.client.0.smithi033.stdout: Throughput test with 1 process 2024-03-19T02:54:53.470 INFO:tasks.workunit.client.0.smithi033.stdout: Each process writes a 1048576 kByte file in 1024 kByte records 2024-03-19T02:54:53.855 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:54:53.855 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:54:54.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.031+0000 7fe88f7bc640 1 -- 172.21.15.33:0/3452167069 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe888068a00 msgr2=0x7fe888068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:54.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.032+0000 7fe88f7bc640 1 --2- 172.21.15.33:0/3452167069 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe888068a00 0x7fe888068de0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fe884009900 tx=0x7fe88402bc50 comp rx=0 tx=0).stop 2024-03-19T02:54:54.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.032+0000 7fe88f7bc640 1 -- 172.21.15.33:0/3452167069 shutdown_connections 2024-03-19T02:54:54.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.032+0000 7fe88f7bc640 1 --2- 172.21.15.33:0/3452167069 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe88810e1c0 0x7fe8881105b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.032+0000 7fe88f7bc640 1 --2- 172.21.15.33:0/3452167069 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe888069320 0x7fe88810dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.032+0000 7fe88f7bc640 1 --2- 172.21.15.33:0/3452167069 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe888068a00 0x7fe888068de0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.032+0000 7fe88f7bc640 1 -- 172.21.15.33:0/3452167069 >> 172.21.15.33:0/3452167069 conn(0x7fe888077820 msgr2=0x7fe888077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:54.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.032+0000 7fe88f7bc640 1 -- 172.21.15.33:0/3452167069 shutdown_connections 2024-03-19T02:54:54.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.033+0000 7fe88f7bc640 1 -- 172.21.15.33:0/3452167069 wait complete. 2024-03-19T02:54:54.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.034+0000 7fe88f7bc640 1 Processor -- start 2024-03-19T02:54:54.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.034+0000 7fe88f7bc640 1 -- start start 2024-03-19T02:54:54.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.034+0000 7fe88f7bc640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe888068a00 0x7fe88819a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:54.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.034+0000 7fe88f7bc640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe888069320 0x7fe88819a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:54.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.034+0000 7fe88f7bc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe88810e1c0 0x7fe88819ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:54.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.034+0000 7fe88f7bc640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe888113d30 con 0x7fe88810e1c0 2024-03-19T02:54:54.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.034+0000 7fe88f7bc640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe888113bb0 con 0x7fe888069320 2024-03-19T02:54:54.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.034+0000 7fe88f7bc640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe888113eb0 con 0x7fe888068a00 2024-03-19T02:54:54.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.035+0000 7fe88d558640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe888068a00 0x7fe88819a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:54.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.035+0000 7fe88d558640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe888068a00 0x7fe88819a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:52002/0 (socket says 172.21.15.33:52002) 2024-03-19T02:54:54.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.035+0000 7fe88d558640 1 -- 172.21.15.33:0/806518923 learned_addr learned my addr 172.21.15.33:0/806518923 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:54.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.035+0000 7fe88cd57640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe888069320 0x7fe88819a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:54.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.035+0000 7fe88dd59640 1 --2- 172.21.15.33:0/806518923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe88810e1c0 0x7fe88819ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:54.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.035+0000 7fe88d558640 1 -- 172.21.15.33:0/806518923 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe888069320 msgr2=0x7fe88819a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:54.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.035+0000 7fe88d558640 1 --2- 172.21.15.33:0/806518923 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe888069320 0x7fe88819a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.035+0000 7fe88d558640 1 -- 172.21.15.33:0/806518923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe88810e1c0 msgr2=0x7fe88819ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:54.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.035+0000 7fe88d558640 1 --2- 172.21.15.33:0/806518923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe88810e1c0 0x7fe88819ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.035+0000 7fe88d558640 1 -- 172.21.15.33:0/806518923 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe88819f340 con 0x7fe888068a00 2024-03-19T02:54:54.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.035+0000 7fe88d558640 1 --2- 172.21.15.33:0/806518923 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe888068a00 0x7fe88819a350 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe884000c00 tx=0x7fe884005710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:54.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.036+0000 7fe87e7fc640 1 -- 172.21.15.33:0/806518923 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe884043020 con 0x7fe888068a00 2024-03-19T02:54:54.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.036+0000 7fe88f7bc640 1 -- 172.21.15.33:0/806518923 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe88819f5d0 con 0x7fe888068a00 2024-03-19T02:54:54.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.036+0000 7fe87e7fc640 1 -- 172.21.15.33:0/806518923 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe8840035f0 con 0x7fe888068a00 2024-03-19T02:54:54.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.036+0000 7fe87e7fc640 1 -- 172.21.15.33:0/806518923 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe884005220 con 0x7fe888068a00 2024-03-19T02:54:54.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.036+0000 7fe88f7bc640 1 -- 172.21.15.33:0/806518923 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8881a6ec0 con 0x7fe888068a00 2024-03-19T02:54:54.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.041+0000 7fe87e7fc640 1 -- 172.21.15.33:0/806518923 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7fe884046020 con 0x7fe888068a00 2024-03-19T02:54:54.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.042+0000 7fe87e7fc640 1 --2- 172.21.15.33:0/806518923 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe8740a2b10 0x7fe8740a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:54.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.042+0000 7fe89239c740 1 Processor -- start 2024-03-19T02:54:54.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.042+0000 7fe87e7fc640 1 -- 172.21.15.33:0/806518923 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fe8840f15a0 con 0x7fe888068a00 2024-03-19T02:54:54.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.043+0000 7fe88cd57640 1 --2- 172.21.15.33:0/806518923 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe8740a2b10 0x7fe8740a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:54.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.043+0000 7fe89239c740 1 -- start start 2024-03-19T02:54:54.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.044+0000 7fe88cd57640 1 --2- 172.21.15.33:0/806518923 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe8740a2b10 0x7fe8740a4fd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe88810f140 tx=0x7fe8780090e0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:54.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.044+0000 7fe89239c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a1a428a30 0x562a1a428e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:54.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.044+0000 7fe89239c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a1a429350 0x562a1a4319f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:54.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.044+0000 7fe89239c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a1a431f30 0x562a1a434320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:54.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.044+0000 7fe88dd59640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a1a431f30 0x562a1a434320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:54.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.044+0000 7fe88dd59640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a1a431f30 0x562a1a434320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53190/0 (socket says 172.21.15.33:53190) 2024-03-19T02:54:54.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.044+0000 7fe88dd59640 1 -- 172.21.15.33:0/23693407 learned_addr learned my addr 172.21.15.33:0/23693407 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:54.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.044+0000 7fe89239c740 1 -- 172.21.15.33:0/23693407 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562a1a38c500 con 0x562a1a431f30 2024-03-19T02:54:54.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.044+0000 7fe89239c740 1 -- 172.21.15.33:0/23693407 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x562a1a334be0 con 0x562a1a429350 2024-03-19T02:54:54.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.044+0000 7fe89239c740 1 -- 172.21.15.33:0/23693407 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x562a1a3885a0 con 0x562a1a428a30 2024-03-19T02:54:54.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.044+0000 7fe88d558640 1 --2- 172.21.15.33:0/23693407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a1a428a30 0x562a1a428e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:54.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.044+0000 7fe88cd57640 1 --2- 172.21.15.33:0/23693407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a1a429350 0x562a1a4319f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:54.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.044+0000 7fe88dd59640 1 -- 172.21.15.33:0/23693407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a1a428a30 msgr2=0x562a1a428e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:54.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.044+0000 7fe88dd59640 1 --2- 172.21.15.33:0/23693407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a1a428a30 0x562a1a428e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.044+0000 7fe88dd59640 1 -- 172.21.15.33:0/23693407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a1a429350 msgr2=0x562a1a4319f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:54.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.045+0000 7fe88dd59640 1 --2- 172.21.15.33:0/23693407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a1a429350 0x562a1a4319f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.045+0000 7fe88dd59640 1 -- 172.21.15.33:0/23693407 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a1a356000 con 0x562a1a431f30 2024-03-19T02:54:54.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.045+0000 7fe88dd59640 1 --2- 172.21.15.33:0/23693407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a1a431f30 0x562a1a434320 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fe88000a400 tx=0x7fe880009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e6dd0f6bde3285 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:54.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.045+0000 7fe8737fe640 1 -- 172.21.15.33:0/23693407 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe880013790 con 0x562a1a431f30 2024-03-19T02:54:54.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.046+0000 7fe8737fe640 1 -- 172.21.15.33:0/23693407 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe880019070 con 0x562a1a431f30 2024-03-19T02:54:54.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.046+0000 7fe8737fe640 1 -- 172.21.15.33:0/23693407 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe880014d00 con 0x562a1a431f30 2024-03-19T02:54:54.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.047+0000 7fe89239c740 1 -- 172.21.15.33:0/23693407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a1a431f30 msgr2=0x562a1a434320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:54.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.047+0000 7fe89239c740 1 --2- 172.21.15.33:0/23693407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a1a431f30 0x562a1a434320 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fe88000a400 tx=0x7fe880009650 comp rx=0 tx=0).stop 2024-03-19T02:54:54.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.048+0000 7fe89239c740 1 -- 172.21.15.33:0/23693407 shutdown_connections 2024-03-19T02:54:54.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.048+0000 7fe89239c740 1 --2- 172.21.15.33:0/23693407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a1a431f30 0x562a1a434320 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.048+0000 7fe89239c740 1 --2- 172.21.15.33:0/23693407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a1a429350 0x562a1a4319f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.048+0000 7fe89239c740 1 --2- 172.21.15.33:0/23693407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a1a428a30 0x562a1a428e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.048+0000 7fe89239c740 1 -- 172.21.15.33:0/23693407 >> 172.21.15.33:0/23693407 conn(0x562a1a423160 msgr2=0x562a1a423540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:54.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.048+0000 7fe89239c740 1 -- 172.21.15.33:0/23693407 shutdown_connections 2024-03-19T02:54:54.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.048+0000 7fe89239c740 1 -- 172.21.15.33:0/23693407 wait complete. 2024-03-19T02:54:54.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.049+0000 7fe89239c740 1 -- start start 2024-03-19T02:54:54.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.049+0000 7fe89239c740 1 Processor -- start 2024-03-19T02:54:54.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.049+0000 7fe89239c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a1a428a30 0x562a1a42ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:54.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.049+0000 7fe89239c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a1a429350 0x562a1a42f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:54.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.049+0000 7fe89239c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a1a431f30 0x562a1a42f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:54.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.050+0000 7fe89239c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562a1a434d20 con 0x562a1a429350 2024-03-19T02:54:54.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.050+0000 7fe89239c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x562a1a348160 con 0x562a1a431f30 2024-03-19T02:54:54.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.050+0000 7fe89239c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x562a1a434ea0 con 0x562a1a428a30 2024-03-19T02:54:54.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.050+0000 7fe88cd57640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a1a429350 0x562a1a42f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:54.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.050+0000 7fe89239c740 10 client.0 _ll_register_callbacks cb 0x562a1a384a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:54:54.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.050+0000 7fe88cd57640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a1a429350 0x562a1a42f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53206/0 (socket says 172.21.15.33:53206) 2024-03-19T02:54:54.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.050+0000 7fe88cd57640 1 -- 172.21.15.33:0/4072705154 learned_addr learned my addr 172.21.15.33:0/4072705154 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:54.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.050+0000 7fe88cd57640 1 -- 172.21.15.33:0/4072705154 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a1a428a30 msgr2=0x562a1a42ef20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T02:54:54.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.050+0000 7fe88cd57640 1 --2- 172.21.15.33:0/4072705154 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a1a428a30 0x562a1a42ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.050+0000 7fe88cd57640 1 -- 172.21.15.33:0/4072705154 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a1a431f30 msgr2=0x562a1a42f9a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T02:54:54.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.050+0000 7fe88cd57640 1 --2- 172.21.15.33:0/4072705154 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a1a431f30 0x562a1a42f9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.050+0000 7fe88cd57640 1 -- 172.21.15.33:0/4072705154 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a1a16a7d0 con 0x562a1a429350 2024-03-19T02:54:54.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.050+0000 7fe88cd57640 1 --2- 172.21.15.33:0/4072705154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a1a429350 0x562a1a42f460 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fe878009780 tx=0x7fe878004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfbe4ce96e611d94 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:54.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.050+0000 7fe870ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:54:54.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.051+0000 7fe870ff9640 1 -- 172.21.15.33:0/4072705154 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe878025020 con 0x562a1a429350 2024-03-19T02:54:54.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.051+0000 7fe870ff9640 1 -- 172.21.15.33:0/4072705154 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fe878015d20 con 0x562a1a429350 2024-03-19T02:54:54.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.051+0000 7fe89239c740 1 -- 172.21.15.33:0/4072705154 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe880019070 con 0x562a1a429350 2024-03-19T02:54:54.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.051+0000 7fe870ff9640 1 -- 172.21.15.33:0/4072705154 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe878030050 con 0x562a1a429350 2024-03-19T02:54:54.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.051+0000 7fe89239c740 10 client.15939 fetch_fsmap learned FSMap version 25 2024-03-19T02:54:54.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.051+0000 7fe89239c740 1 -- 172.21.15.33:0/4072705154 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562a1a0a03f0 con 0x562a1a429350 2024-03-19T02:54:54.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.052+0000 7fe870ff9640 1 -- 172.21.15.33:0/4072705154 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe880019070 con 0x562a1a429350 2024-03-19T02:54:54.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.052+0000 7fe870ff9640 1 -- 172.21.15.33:0/4072705154 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fe878021b00 con 0x562a1a429350 2024-03-19T02:54:54.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.052+0000 7fe89239c740 10 client.15939 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:54:54.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.052+0000 7fe89239c740 10 client.15939 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:54:54.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.052+0000 7fe89239c740 20 client.15939 populate_metadata read hostname 'smithi033' 2024-03-19T02:54:54.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.052+0000 7fe89239c740 1 --2- 172.21.15.33:0/4072705154 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x562a1a442360 0x562a1a4447e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:54.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.052+0000 7fe89239c740 4 client.15939 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:54:54.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.052+0000 7fe89239c740 1 -- 172.21.15.33:0/4072705154 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a1a399a40 con 0x562a1a442360 2024-03-19T02:54:54.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.052+0000 7fe88d558640 1 --2- 172.21.15.33:0/4072705154 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x562a1a442360 0x562a1a4447e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:54.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.054+0000 7fe870ff9640 10 client.15939 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:54:54.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.054+0000 7fe88d558640 1 --2- 172.21.15.33:0/4072705154 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x562a1a442360 0x562a1a4447e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e36a8cc542f29e9 server_cookie=1089a1ef35a66eb3 in_seq=0 out_seq=0 2024-03-19T02:54:54.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.055+0000 7fe870ff9640 1 -- 172.21.15.33:0/4072705154 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562a1a399a40 con 0x562a1a442360 2024-03-19T02:54:54.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.055+0000 7fe870ff9640 10 client.15939 handle_command_reply: tid=0 2024-03-19T02:54:54.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.055+0000 7fe89239c740 1 client.15939 shutdown 2024-03-19T02:54:54.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.055+0000 7fe89239c740 1 -- 172.21.15.33:0/4072705154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a1a429350 msgr2=0x562a1a42f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:54.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.055+0000 7fe89239c740 1 --2- 172.21.15.33:0/4072705154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a1a429350 0x562a1a42f460 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fe878009780 tx=0x7fe878004750 comp rx=0 tx=0).stop 2024-03-19T02:54:54.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.055+0000 7fe89239c740 1 -- 172.21.15.33:0/4072705154 shutdown_connections 2024-03-19T02:54:54.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.055+0000 7fe89239c740 1 --2- 172.21.15.33:0/4072705154 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x562a1a442360 0x562a1a4447e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.055+0000 7fe870ff9640 0 client.15939 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:54:54.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.055+0000 7fe89239c740 1 --2- 172.21.15.33:0/4072705154 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a1a431f30 0x562a1a42f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.055+0000 7fe89239c740 1 --2- 172.21.15.33:0/4072705154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a1a429350 0x562a1a42f460 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.055+0000 7fe89239c740 1 --2- 172.21.15.33:0/4072705154 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a1a428a30 0x562a1a42ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.055+0000 7fe89239c740 1 -- 172.21.15.33:0/4072705154 >> 172.21.15.33:0/4072705154 conn(0x562a1a423160 msgr2=0x562a1a424eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:54.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.056+0000 7fe89239c740 1 -- 172.21.15.33:0/4072705154 shutdown_connections 2024-03-19T02:54:54.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.056+0000 7fe89239c740 1 -- 172.21.15.33:0/4072705154 wait complete. 2024-03-19T02:54:54.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.056+0000 7fe89239c740 20 client.15939 trim_cache size 0 max 16384 2024-03-19T02:54:54.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.083+0000 7fe89239c740 1 Processor -- start 2024-03-19T02:54:54.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.084+0000 7fe89239c740 1 -- start start 2024-03-19T02:54:54.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.085+0000 7fe89239c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a1a428a30 0x562a1a431650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:54.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.085+0000 7fe89239c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a1a429350 0x562a1a42de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:54.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.086+0000 7fe89239c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a1a431f30 0x562a1a42a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:54.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.086+0000 7fe89239c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe8881b3660 con 0x562a1a429350 2024-03-19T02:54:54.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.086+0000 7fe89239c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe8881b0f30 con 0x562a1a428a30 2024-03-19T02:54:54.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.086+0000 7fe89239c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe8881aed60 con 0x562a1a431f30 2024-03-19T02:54:54.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.086+0000 7fe88cd57640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a1a429350 0x562a1a42de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:54.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.086+0000 7fe88cd57640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a1a429350 0x562a1a42de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53216/0 (socket says 172.21.15.33:53216) 2024-03-19T02:54:54.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.086+0000 7fe88cd57640 1 -- 172.21.15.33:0/1727232189 learned_addr learned my addr 172.21.15.33:0/1727232189 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:54.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.086+0000 7fe88d558640 1 --2- 172.21.15.33:0/1727232189 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a1a428a30 0x562a1a431650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:54.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.086+0000 7fe88cd57640 1 -- 172.21.15.33:0/1727232189 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a1a431f30 msgr2=0x562a1a42a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:54.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.086+0000 7fe88cd57640 1 --2- 172.21.15.33:0/1727232189 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a1a431f30 0x562a1a42a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.086+0000 7fe88cd57640 1 -- 172.21.15.33:0/1727232189 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a1a428a30 msgr2=0x562a1a431650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:54.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.086+0000 7fe88cd57640 1 --2- 172.21.15.33:0/1727232189 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a1a428a30 0x562a1a431650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.086+0000 7fe88cd57640 1 -- 172.21.15.33:0/1727232189 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a1a0600f0 con 0x562a1a429350 2024-03-19T02:54:54.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.087+0000 7fe88cd57640 1 --2- 172.21.15.33:0/1727232189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a1a429350 0x562a1a42de60 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fe878021ea0 tx=0x7fe878006c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b42f427c44b7997 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:54.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.087+0000 7fe872ffd640 1 -- 172.21.15.33:0/1727232189 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe878039070 con 0x562a1a429350 2024-03-19T02:54:54.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.087+0000 7fe872ffd640 1 -- 172.21.15.33:0/1727232189 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe878021470 con 0x562a1a429350 2024-03-19T02:54:54.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.087+0000 7fe872ffd640 1 -- 172.21.15.33:0/1727232189 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe878025040 con 0x562a1a429350 2024-03-19T02:54:54.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.089+0000 7fe89239c740 1 -- 172.21.15.33:0/1727232189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a1a429350 msgr2=0x562a1a42de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:54.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.089+0000 7fe89239c740 1 --2- 172.21.15.33:0/1727232189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a1a429350 0x562a1a42de60 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fe878021ea0 tx=0x7fe878006c50 comp rx=0 tx=0).stop 2024-03-19T02:54:54.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.090+0000 7fe89239c740 1 -- 172.21.15.33:0/1727232189 shutdown_connections 2024-03-19T02:54:54.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.090+0000 7fe89239c740 1 --2- 172.21.15.33:0/1727232189 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a1a431f30 0x562a1a42a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.090+0000 7fe89239c740 1 --2- 172.21.15.33:0/1727232189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a1a429350 0x562a1a42de60 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.090+0000 7fe89239c740 1 --2- 172.21.15.33:0/1727232189 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a1a428a30 0x562a1a431650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.090+0000 7fe89239c740 1 -- 172.21.15.33:0/1727232189 >> 172.21.15.33:0/1727232189 conn(0x562a1a423160 msgr2=0x562a1a42f770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:54.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.090+0000 7fe89239c740 1 -- 172.21.15.33:0/1727232189 shutdown_connections 2024-03-19T02:54:54.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.090+0000 7fe89239c740 1 -- 172.21.15.33:0/1727232189 wait complete. 2024-03-19T02:54:54.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.091+0000 7fe89239c740 1 -- start start 2024-03-19T02:54:54.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.091+0000 7fe89239c740 1 Processor -- start 2024-03-19T02:54:54.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.092+0000 7fe89239c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a1a428a30 0x562a1a42bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:54.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.092+0000 7fe89239c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a1a429350 0x562a1a42c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:54.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.092+0000 7fe89239c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a1a431f30 0x562a1a42c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:54.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.092+0000 7fe89239c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe8881af3f0 con 0x562a1a429350 2024-03-19T02:54:54.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.092+0000 7fe89239c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe8881b0860 con 0x562a1a428a30 2024-03-19T02:54:54.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.092+0000 7fe89239c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe8881afac0 con 0x562a1a431f30 2024-03-19T02:54:54.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.092+0000 7fe89239c740 10 client.0 _ll_register_callbacks cb 0x7fe8780096d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:54:54.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.092+0000 7fe88cd57640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a1a429350 0x562a1a42c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:54.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.092+0000 7fe88d558640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a1a428a30 0x562a1a42bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:54.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.092+0000 7fe88d558640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a1a428a30 0x562a1a42bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:50802/0 (socket says 172.21.15.33:50802) 2024-03-19T02:54:54.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.092+0000 7fe88d558640 1 -- 172.21.15.33:0/303537358 learned_addr learned my addr 172.21.15.33:0/303537358 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:54.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.092+0000 7fe88dd59640 1 --2- 172.21.15.33:0/303537358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a1a431f30 0x562a1a42c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:54.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.092+0000 7fe88cd57640 1 -- 172.21.15.33:0/303537358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a1a431f30 msgr2=0x562a1a42c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:54.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.092+0000 7fe88cd57640 1 --2- 172.21.15.33:0/303537358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a1a431f30 0x562a1a42c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.092+0000 7fe88cd57640 1 -- 172.21.15.33:0/303537358 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a1a428a30 msgr2=0x562a1a42bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:54.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.092+0000 7fe88cd57640 1 --2- 172.21.15.33:0/303537358 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a1a428a30 0x562a1a42bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.092+0000 7fe88cd57640 1 -- 172.21.15.33:0/303537358 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a1a4a7830 con 0x562a1a429350 2024-03-19T02:54:54.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.092+0000 7fe88d558640 1 --2- 172.21.15.33:0/303537358 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a1a428a30 0x562a1a42bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:54.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.093+0000 7fe88dd59640 1 --2- 172.21.15.33:0/303537358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a1a431f30 0x562a1a42c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:54.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.093+0000 7fe88cd57640 1 --2- 172.21.15.33:0/303537358 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a1a429350 0x562a1a42c160 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x562a1a434ff0 tx=0x7fe878013b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0a5331bb8eb2e8b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:54.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.093+0000 7fe870ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:54:54.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.093+0000 7fe870ff9640 1 -- 172.21.15.33:0/303537358 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe878039070 con 0x562a1a429350 2024-03-19T02:54:54.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.093+0000 7fe870ff9640 1 -- 172.21.15.33:0/303537358 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fe878021a30 con 0x562a1a429350 2024-03-19T02:54:54.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.094+0000 7fe89239c740 1 -- 172.21.15.33:0/303537358 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe878021470 con 0x562a1a429350 2024-03-19T02:54:54.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.094+0000 7fe870ff9640 1 -- 172.21.15.33:0/303537358 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe878047050 con 0x562a1a429350 2024-03-19T02:54:54.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.094+0000 7fe89239c740 10 client.15945 fetch_fsmap learned FSMap version 25 2024-03-19T02:54:54.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.094+0000 7fe89239c740 1 -- 172.21.15.33:0/303537358 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562a1a4a8b20 con 0x562a1a429350 2024-03-19T02:54:54.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.094+0000 7fe870ff9640 1 -- 172.21.15.33:0/303537358 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe878021470 con 0x562a1a429350 2024-03-19T02:54:54.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.095+0000 7fe870ff9640 1 -- 172.21.15.33:0/303537358 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fe878034430 con 0x562a1a429350 2024-03-19T02:54:54.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.095+0000 7fe89239c740 10 client.15945 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:54:54.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.095+0000 7fe89239c740 10 client.15945 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:54:54.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.095+0000 7fe89239c740 20 client.15945 populate_metadata read hostname 'smithi033' 2024-03-19T02:54:54.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.095+0000 7fe89239c740 1 --2- 172.21.15.33:0/303537358 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x562a1a442360 0x562a1a4ab160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:54.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.095+0000 7fe89239c740 4 client.15945 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:54:54.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.095+0000 7fe89239c740 1 -- 172.21.15.33:0/303537358 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562a1a393d90 con 0x562a1a442360 2024-03-19T02:54:54.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.096+0000 7fe88d558640 1 --2- 172.21.15.33:0/303537358 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x562a1a442360 0x562a1a4ab160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:54.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.097+0000 7fe88d558640 1 --2- 172.21.15.33:0/303537358 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x562a1a442360 0x562a1a4ab160 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5b40cea48aad4ec server_cookie=5c2aa28ef5838f79 in_seq=0 out_seq=0 2024-03-19T02:54:54.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.097+0000 7fe870ff9640 10 client.15945 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:54:54.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.098+0000 7fe870ff9640 1 -- 172.21.15.33:0/303537358 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562a1a393d90 con 0x562a1a442360 2024-03-19T02:54:54.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.098+0000 7fe870ff9640 10 client.15945 handle_command_reply: tid=0 2024-03-19T02:54:54.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.098+0000 7fe89239c740 1 client.15945 shutdown 2024-03-19T02:54:54.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.098+0000 7fe89239c740 1 -- 172.21.15.33:0/303537358 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a1a429350 msgr2=0x562a1a42c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:54.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.098+0000 7fe89239c740 1 --2- 172.21.15.33:0/303537358 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a1a429350 0x562a1a42c160 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x562a1a434ff0 tx=0x7fe878013b60 comp rx=0 tx=0).stop 2024-03-19T02:54:54.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.098+0000 7fe89239c740 1 -- 172.21.15.33:0/303537358 shutdown_connections 2024-03-19T02:54:54.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.098+0000 7fe89239c740 1 --2- 172.21.15.33:0/303537358 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x562a1a442360 0x562a1a4ab160 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.098+0000 7fe89239c740 1 --2- 172.21.15.33:0/303537358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a1a431f30 0x562a1a42c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.098+0000 7fe870ff9640 0 client.15945 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:54:54.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.098+0000 7fe89239c740 1 --2- 172.21.15.33:0/303537358 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a1a429350 0x562a1a42c160 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.098+0000 7fe89239c740 1 --2- 172.21.15.33:0/303537358 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a1a428a30 0x562a1a42bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.098+0000 7fe89239c740 1 -- 172.21.15.33:0/303537358 >> 172.21.15.33:0/303537358 conn(0x562a1a423160 msgr2=0x562a1a421950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:54.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.098+0000 7fe89239c740 1 -- 172.21.15.33:0/303537358 shutdown_connections 2024-03-19T02:54:54.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.098+0000 7fe89239c740 1 -- 172.21.15.33:0/303537358 wait complete. 2024-03-19T02:54:54.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.098+0000 7fe89239c740 20 client.15945 trim_cache size 0 max 16384 2024-03-19T02:54:54.101 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:54:54.101 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-03-19T02:54:54.102 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-03-19T02:54:54.102 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:54:54.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.099+0000 7fe88f7bc640 1 -- 172.21.15.33:0/806518923 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe8740a2b10 msgr2=0x7fe8740a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:54.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.099+0000 7fe88f7bc640 1 --2- 172.21.15.33:0/806518923 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe8740a2b10 0x7fe8740a4fd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe88810f140 tx=0x7fe8780090e0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.100+0000 7fe88f7bc640 1 -- 172.21.15.33:0/806518923 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe888068a00 msgr2=0x7fe88819a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:54.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.100+0000 7fe88f7bc640 1 --2- 172.21.15.33:0/806518923 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe888068a00 0x7fe88819a350 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe884000c00 tx=0x7fe884005710 comp rx=0 tx=0).stop 2024-03-19T02:54:54.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.100+0000 7fe88f7bc640 1 -- 172.21.15.33:0/806518923 shutdown_connections 2024-03-19T02:54:54.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.100+0000 7fe88f7bc640 1 --2- 172.21.15.33:0/806518923 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe8740a2b10 0x7fe8740a4fd0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.100+0000 7fe88f7bc640 1 --2- 172.21.15.33:0/806518923 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe88810e1c0 0x7fe88819ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.100+0000 7fe88f7bc640 1 --2- 172.21.15.33:0/806518923 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe888069320 0x7fe88819a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.100+0000 7fe88f7bc640 1 --2- 172.21.15.33:0/806518923 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe888068a00 0x7fe88819a350 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.100+0000 7fe88f7bc640 1 -- 172.21.15.33:0/806518923 >> 172.21.15.33:0/806518923 conn(0x7fe888077820 msgr2=0x7fe88810fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:54.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.100+0000 7fe88f7bc640 1 -- 172.21.15.33:0/806518923 shutdown_connections 2024-03-19T02:54:54.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.100+0000 7fe88f7bc640 1 -- 172.21.15.33:0/806518923 wait complete. 2024-03-19T02:54:54.121 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:54:54.122 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:54:54.159 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:54 smithi033 ceph-mon[31973]: pgmap v46: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 23 MiB/s wr, 13 op/s 2024-03-19T02:54:54.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.313+0000 7fb57bcd1640 1 -- 172.21.15.33:0/3558697014 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb5741114f0 msgr2=0x7fb5741190c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:54.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.313+0000 7fb57bcd1640 1 --2- 172.21.15.33:0/3558697014 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb5741114f0 0x7fb5741190c0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fb570009900 tx=0x7fb57002bc40 comp rx=0 tx=0).stop 2024-03-19T02:54:54.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.314+0000 7fb57bcd1640 1 -- 172.21.15.33:0/3558697014 shutdown_connections 2024-03-19T02:54:54.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.314+0000 7fb57bcd1640 1 --2- 172.21.15.33:0/3558697014 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb57410cdd0 0x7fb574110e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.314+0000 7fb57bcd1640 1 --2- 172.21.15.33:0/3558697014 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb5741114f0 0x7fb5741190c0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.314+0000 7fb57bcd1640 1 --2- 172.21.15.33:0/3558697014 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb57410c420 0x7fb57410c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.314+0000 7fb57bcd1640 1 -- 172.21.15.33:0/3558697014 >> 172.21.15.33:0/3558697014 conn(0x7fb57406c880 msgr2=0x7fb57410a830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:54.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.315+0000 7fb57bcd1640 1 -- 172.21.15.33:0/3558697014 shutdown_connections 2024-03-19T02:54:54.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.315+0000 7fb57bcd1640 1 -- 172.21.15.33:0/3558697014 wait complete. 2024-03-19T02:54:54.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.316+0000 7fb57bcd1640 1 Processor -- start 2024-03-19T02:54:54.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.316+0000 7fb57bcd1640 1 -- start start 2024-03-19T02:54:54.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.316+0000 7fb57bcd1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb57410c420 0x7fb574118fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:54.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.316+0000 7fb57bcd1640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb5741114f0 0x7fb574116060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:54.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.316+0000 7fb57bcd1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5741165a0 0x7fb574116a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:54.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.316+0000 7fb57bcd1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb57411c440 con 0x7fb57410c420 2024-03-19T02:54:54.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.316+0000 7fb57bcd1640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb57411c2c0 con 0x7fb5741165a0 2024-03-19T02:54:54.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.316+0000 7fb57bcd1640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb57411c5c0 con 0x7fb5741114f0 2024-03-19T02:54:54.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.317+0000 7fb57926c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb5741114f0 0x7fb574116060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:54.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.317+0000 7fb57926c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb5741114f0 0x7fb574116060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:52062/0 (socket says 172.21.15.33:52062) 2024-03-19T02:54:54.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.317+0000 7fb57926c640 1 -- 172.21.15.33:0/696516445 learned_addr learned my addr 172.21.15.33:0/696516445 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:54.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.317+0000 7fb57a26e640 1 --2- 172.21.15.33:0/696516445 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5741165a0 0x7fb574116a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:54.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.317+0000 7fb579a6d640 1 --2- 172.21.15.33:0/696516445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb57410c420 0x7fb574118fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:54.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.317+0000 7fb57926c640 1 -- 172.21.15.33:0/696516445 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5741165a0 msgr2=0x7fb574116a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:54.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.317+0000 7fb57926c640 1 --2- 172.21.15.33:0/696516445 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5741165a0 0x7fb574116a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.317+0000 7fb57926c640 1 -- 172.21.15.33:0/696516445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb57410c420 msgr2=0x7fb574118fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:54.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.317+0000 7fb57926c640 1 --2- 172.21.15.33:0/696516445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb57410c420 0x7fb574118fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.317+0000 7fb57926c640 1 -- 172.21.15.33:0/696516445 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5741b33b0 con 0x7fb5741114f0 2024-03-19T02:54:54.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.318+0000 7fb57926c640 1 --2- 172.21.15.33:0/696516445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb5741114f0 0x7fb574116060 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb56400d7a0 tx=0x7fb56400dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:54.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.319+0000 7fb56affd640 1 -- 172.21.15.33:0/696516445 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb564017070 con 0x7fb5741114f0 2024-03-19T02:54:54.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.319+0000 7fb56affd640 1 -- 172.21.15.33:0/696516445 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb564009c30 con 0x7fb5741114f0 2024-03-19T02:54:54.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.319+0000 7fb57bcd1640 1 -- 172.21.15.33:0/696516445 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5741b36a0 con 0x7fb5741114f0 2024-03-19T02:54:54.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.319+0000 7fb57bcd1640 1 -- 172.21.15.33:0/696516445 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5741b3b30 con 0x7fb5741114f0 2024-03-19T02:54:54.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.319+0000 7fb56affd640 1 -- 172.21.15.33:0/696516445 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56400f620 con 0x7fb5741114f0 2024-03-19T02:54:54.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.321+0000 7fb57bcd1640 1 -- 172.21.15.33:0/696516445 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5580052c0 con 0x7fb5741114f0 2024-03-19T02:54:54.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.324+0000 7fb56affd640 1 -- 172.21.15.33:0/696516445 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7fb56401a070 con 0x7fb5741114f0 2024-03-19T02:54:54.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.324+0000 7fb56affd640 1 --2- 172.21.15.33:0/696516445 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fb5600a2ca0 0x7fb5600a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:54.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.325+0000 7fb56affd640 1 -- 172.21.15.33:0/696516445 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fb5640cec40 con 0x7fb5741114f0 2024-03-19T02:54:54.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.328+0000 7fb579a6d640 1 --2- 172.21.15.33:0/696516445 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fb5600a2ca0 0x7fb5600a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:54.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.329+0000 7fb579a6d640 1 --2- 172.21.15.33:0/696516445 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fb5600a2ca0 0x7fb5600a5160 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb56c005fd0 tx=0x7fb56c009870 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:54.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.333+0000 7fb56affd640 1 -- 172.21.15.33:0/696516445 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb56409b4a0 con 0x7fb5741114f0 2024-03-19T02:54:54.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:54 smithi096 ceph-mon[35946]: pgmap v46: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 23 MiB/s wr, 13 op/s 2024-03-19T02:54:54.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:54 smithi155 ceph-mon[35804]: pgmap v46: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 23 MiB/s wr, 13 op/s 2024-03-19T02:54:54.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.766+0000 7fb57bcd1640 1 -- 172.21.15.33:0/696516445 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb5580055b0 con 0x7fb5741114f0 2024-03-19T02:54:54.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.767+0000 7fb56affd640 1 -- 172.21.15.33:0/696516445 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11867 (secure 0 0 0) 0x7fb56409ada0 con 0x7fb5741114f0 2024-03-19T02:54:54.769 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:54:54.769 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:50:52.201798+0000","modified":"2024-03-19T02:53:11.748697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24488,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/3731232169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":3731232169},{"type":"v1","addr":"172.21.15.96:6839","nonce":3731232169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24457,24488,24509]},"id":1}]} 2024-03-19T02:54:54.769 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-03-19T02:54:54.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.770+0000 7fb57bcd1640 1 -- 172.21.15.33:0/696516445 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fb5600a2ca0 msgr2=0x7fb5600a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:54.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.770+0000 7fb57bcd1640 1 --2- 172.21.15.33:0/696516445 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fb5600a2ca0 0x7fb5600a5160 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb56c005fd0 tx=0x7fb56c009870 comp rx=0 tx=0).stop 2024-03-19T02:54:54.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.770+0000 7fb57bcd1640 1 -- 172.21.15.33:0/696516445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb5741114f0 msgr2=0x7fb574116060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:54.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.770+0000 7fb57bcd1640 1 --2- 172.21.15.33:0/696516445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb5741114f0 0x7fb574116060 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb56400d7a0 tx=0x7fb56400dc60 comp rx=0 tx=0).stop 2024-03-19T02:54:54.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.770+0000 7fb57bcd1640 1 -- 172.21.15.33:0/696516445 shutdown_connections 2024-03-19T02:54:54.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.770+0000 7fb57bcd1640 1 --2- 172.21.15.33:0/696516445 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fb5600a2ca0 0x7fb5600a5160 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.770+0000 7fb57bcd1640 1 --2- 172.21.15.33:0/696516445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb5741114f0 0x7fb574116060 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.770+0000 7fb57bcd1640 1 --2- 172.21.15.33:0/696516445 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5741165a0 0x7fb574116a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.770+0000 7fb57bcd1640 1 --2- 172.21.15.33:0/696516445 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb57410c420 0x7fb574118fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:54.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.770+0000 7fb57bcd1640 1 -- 172.21.15.33:0/696516445 >> 172.21.15.33:0/696516445 conn(0x7fb57406c880 msgr2=0x7fb57410f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:54.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.770+0000 7fb57bcd1640 1 -- 172.21.15.33:0/696516445 shutdown_connections 2024-03-19T02:54:54.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:54.771+0000 7fb57bcd1640 1 -- 172.21.15.33:0/696516445 wait complete. 2024-03-19T02:54:54.795 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:54:55.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.006+0000 7f5007fe6640 1 -- 172.21.15.33:0/1573062809 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5000104380 msgr2=0x7f5000110890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.006+0000 7f5007fe6640 1 --2- 172.21.15.33:0/1573062809 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5000104380 0x7f5000110890 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4ff8009be0 tx=0x7f4ff802bef0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.006+0000 7f5007fe6640 1 -- 172.21.15.33:0/1573062809 shutdown_connections 2024-03-19T02:54:55.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.006+0000 7f5007fe6640 1 --2- 172.21.15.33:0/1573062809 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50001039c0 0x7f5000103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.006+0000 7f5007fe6640 1 --2- 172.21.15.33:0/1573062809 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5000104380 0x7f5000110890 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.006+0000 7f5007fe6640 1 --2- 172.21.15.33:0/1573062809 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50001099d0 0x7f5000109db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.006+0000 7f5007fe6640 1 -- 172.21.15.33:0/1573062809 >> 172.21.15.33:0/1573062809 conn(0x7f50000ff4d0 msgr2=0x7f50001018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:55.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.006+0000 7f5007fe6640 1 -- 172.21.15.33:0/1573062809 shutdown_connections 2024-03-19T02:54:55.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.006+0000 7f5007fe6640 1 -- 172.21.15.33:0/1573062809 wait complete. 2024-03-19T02:54:55.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.007+0000 7f5007fe6640 1 Processor -- start 2024-03-19T02:54:55.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.007+0000 7f5007fe6640 1 -- start start 2024-03-19T02:54:55.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.007+0000 7f5007fe6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50001039c0 0x7f500019a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.007+0000 7f5007fe6640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5000104380 0x7f500019a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.007+0000 7f5007fe6640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50001099d0 0x7f500019ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.007+0000 7f5007fe6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5000113c90 con 0x7f50001039c0 2024-03-19T02:54:55.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.007+0000 7f5007fe6640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5000113b10 con 0x7f5000104380 2024-03-19T02:54:55.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.007+0000 7f5007fe6640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5000113e10 con 0x7f50001099d0 2024-03-19T02:54:55.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.007+0000 7f5005d82640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50001039c0 0x7f500019a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.007+0000 7f5005d82640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50001039c0 0x7f500019a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53264/0 (socket says 172.21.15.33:53264) 2024-03-19T02:54:55.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.007+0000 7f5005d82640 1 -- 172.21.15.33:0/2397146602 learned_addr learned my addr 172.21.15.33:0/2397146602 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:55.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.008+0000 7f5005581640 1 --2- 172.21.15.33:0/2397146602 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5000104380 0x7f500019a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.008+0000 7f5005d82640 1 -- 172.21.15.33:0/2397146602 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50001099d0 msgr2=0x7f500019ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.008+0000 7f5005d82640 1 --2- 172.21.15.33:0/2397146602 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50001099d0 0x7f500019ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.008+0000 7f5005d82640 1 -- 172.21.15.33:0/2397146602 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5000104380 msgr2=0x7f500019a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.008+0000 7f5005d82640 1 --2- 172.21.15.33:0/2397146602 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5000104380 0x7f500019a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.008+0000 7f5005d82640 1 -- 172.21.15.33:0/2397146602 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f500019f1a0 con 0x7f50001039c0 2024-03-19T02:54:55.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.008+0000 7f5005581640 1 --2- 172.21.15.33:0/2397146602 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5000104380 0x7f500019a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:54:55.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.008+0000 7f5005d82640 1 --2- 172.21.15.33:0/2397146602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50001039c0 0x7f500019a1b0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc00e9c0 tx=0x7f4ffc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:55.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.008+0000 7f4ff6ffd640 1 -- 172.21.15.33:0/2397146602 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ffc00cde0 con 0x7f50001039c0 2024-03-19T02:54:55.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.008+0000 7f4ff6ffd640 1 -- 172.21.15.33:0/2397146602 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4ffc0035f0 con 0x7f50001039c0 2024-03-19T02:54:55.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.008+0000 7f5007fe6640 1 -- 172.21.15.33:0/2397146602 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f500019f490 con 0x7f50001039c0 2024-03-19T02:54:55.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.008+0000 7f4ff6ffd640 1 -- 172.21.15.33:0/2397146602 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ffc010690 con 0x7f50001039c0 2024-03-19T02:54:55.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.008+0000 7f5007fe6640 1 -- 172.21.15.33:0/2397146602 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50001a6d20 con 0x7f50001039c0 2024-03-19T02:54:55.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.011+0000 7f500abd0740 1 Processor -- start 2024-03-19T02:54:55.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.012+0000 7f500abd0740 1 -- start start 2024-03-19T02:54:55.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.012+0000 7f500abd0740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b203bee2d0 0x55b203bee6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.012+0000 7f500abd0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b203beebf0 0x55b203bf7290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.012+0000 7f500abd0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b203bf77d0 0x55b203bf9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.012+0000 7f500abd0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b203b514b0 con 0x55b203bf77d0 2024-03-19T02:54:55.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.012+0000 7f500abd0740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55b203af9d70 con 0x55b203bee2d0 2024-03-19T02:54:55.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.012+0000 7f500abd0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b203b4d550 con 0x55b203beebf0 2024-03-19T02:54:55.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.012+0000 7f5005581640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b203bf77d0 0x55b203bf9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.012+0000 7f5005581640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b203bf77d0 0x55b203bf9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53276/0 (socket says 172.21.15.33:53276) 2024-03-19T02:54:55.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.012+0000 7f5005581640 1 -- 172.21.15.33:0/2531369479 learned_addr learned my addr 172.21.15.33:0/2531369479 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:55.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.013+0000 7f4ff6ffd640 1 -- 172.21.15.33:0/2397146602 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7f4ffc0037a0 con 0x7f50001039c0 2024-03-19T02:54:55.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.013+0000 7f5005581640 1 -- 172.21.15.33:0/2531369479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b203beebf0 msgr2=0x55b203bf7290 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T02:54:55.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.013+0000 7f5005581640 1 --2- 172.21.15.33:0/2531369479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b203beebf0 0x55b203bf7290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.013+0000 7f5005581640 1 -- 172.21.15.33:0/2531369479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b203bee2d0 msgr2=0x55b203bee6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:55.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.013+0000 7f5005581640 1 --2- 172.21.15.33:0/2531369479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b203bee2d0 0x55b203bee6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.013+0000 7f5005581640 1 -- 172.21.15.33:0/2531369479 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b203b1c210 con 0x55b203bf77d0 2024-03-19T02:54:55.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.013+0000 7f5005581640 1 --2- 172.21.15.33:0/2531369479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b203bf77d0 0x55b203bf9bc0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0007800 tx=0x7f4ff0007cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc0f3343a3a47258 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:55.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.013+0000 7f4febfff640 1 -- 172.21.15.33:0/2531369479 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff000d950 con 0x55b203bf77d0 2024-03-19T02:54:55.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.013+0000 7f4febfff640 1 -- 172.21.15.33:0/2531369479 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4ff0018070 con 0x55b203bf77d0 2024-03-19T02:54:55.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.013+0000 7f4febfff640 1 -- 172.21.15.33:0/2531369479 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff00123e0 con 0x55b203bf77d0 2024-03-19T02:54:55.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.013+0000 7f4ff6ffd640 1 --2- 172.21.15.33:0/2397146602 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4fec0a2a90 0x7f4fec0a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.014+0000 7f5006583640 1 --2- 172.21.15.33:0/2397146602 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4fec0a2a90 0x7f4fec0a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.014+0000 7f4ff6ffd640 1 -- 172.21.15.33:0/2397146602 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f4ffc0cf910 con 0x7f50001039c0 2024-03-19T02:54:55.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.014+0000 7f500abd0740 1 -- 172.21.15.33:0/2531369479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b203bf77d0 msgr2=0x55b203bf9bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.014+0000 7f5006583640 1 --2- 172.21.15.33:0/2397146602 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4fec0a2a90 0x7f4fec0a4f50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4ff8036040 tx=0x7f4ff8009330 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:55.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.014+0000 7f500abd0740 1 --2- 172.21.15.33:0/2531369479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b203bf77d0 0x55b203bf9bc0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0007800 tx=0x7f4ff0007cc0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.014+0000 7f500abd0740 1 -- 172.21.15.33:0/2531369479 shutdown_connections 2024-03-19T02:54:55.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.014+0000 7f500abd0740 1 --2- 172.21.15.33:0/2531369479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b203beebf0 0x55b203bf7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.014+0000 7f500abd0740 1 --2- 172.21.15.33:0/2531369479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b203bf77d0 0x55b203bf9bc0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.014+0000 7f500abd0740 1 --2- 172.21.15.33:0/2531369479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b203bee2d0 0x55b203bee6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.014+0000 7f500abd0740 1 -- 172.21.15.33:0/2531369479 >> 172.21.15.33:0/2531369479 conn(0x55b203be8a00 msgr2=0x55b203be8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:55.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.014+0000 7f500abd0740 1 -- 172.21.15.33:0/2531369479 shutdown_connections 2024-03-19T02:54:55.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.015+0000 7f500abd0740 1 -- 172.21.15.33:0/2531369479 wait complete. 2024-03-19T02:54:55.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.015+0000 7f500abd0740 1 -- start start 2024-03-19T02:54:55.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.015+0000 7f500abd0740 1 Processor -- start 2024-03-19T02:54:55.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.016+0000 7f500abd0740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b203bee2d0 0x55b203bf48f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.016+0000 7f500abd0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b203beebf0 0x55b203bf4e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.016+0000 7f500abd0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b203bf77d0 0x55b203bf5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.016+0000 7f500abd0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b203bfa5c0 con 0x55b203bf77d0 2024-03-19T02:54:55.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.016+0000 7f500abd0740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55b203b0d310 con 0x55b203bee2d0 2024-03-19T02:54:55.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.016+0000 7f500abd0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b203bfa740 con 0x55b203beebf0 2024-03-19T02:54:55.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.016+0000 7f500abd0740 10 client.0 _ll_register_callbacks cb 0x55b203b012d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:54:55.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.016+0000 7f5005d82640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b203bee2d0 0x55b203bf48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.016+0000 7f5006583640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b203bf77d0 0x55b203bf5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.016+0000 7f5006583640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b203bf77d0 0x55b203bf5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53288/0 (socket says 172.21.15.33:53288) 2024-03-19T02:54:55.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.016+0000 7f5006583640 1 -- 172.21.15.33:0/1042025955 learned_addr learned my addr 172.21.15.33:0/1042025955 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:55.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.016+0000 7f5005581640 1 --2- 172.21.15.33:0/1042025955 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b203beebf0 0x55b203bf4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.017+0000 7f5005d82640 1 -- 172.21.15.33:0/1042025955 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b203beebf0 msgr2=0x55b203bf4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:55.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.017+0000 7f5005d82640 1 --2- 172.21.15.33:0/1042025955 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b203beebf0 0x55b203bf4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.017+0000 7f5005d82640 1 -- 172.21.15.33:0/1042025955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b203bf77d0 msgr2=0x55b203bf5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:55.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.017+0000 7f5005d82640 1 --2- 172.21.15.33:0/1042025955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b203bf77d0 0x55b203bf5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.017+0000 7f5005d82640 1 -- 172.21.15.33:0/1042025955 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2038653f0 con 0x55b203bee2d0 2024-03-19T02:54:55.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.017+0000 7f5005d82640 1 --2- 172.21.15.33:0/1042025955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b203bee2d0 0x55b203bf48f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc0ccb70 tx=0x7f4ffc09b3d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=191906378f6a8df0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:55.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.017+0000 7f4fe97fa640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T02:54:55.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.017+0000 7f4fe97fa640 1 -- 172.21.15.33:0/1042025955 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ffc0b4060 con 0x55b203bee2d0 2024-03-19T02:54:55.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.017+0000 7f4fe97fa640 1 -- 172.21.15.33:0/1042025955 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f4ffc0860d0 con 0x55b203bee2d0 2024-03-19T02:54:55.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.017+0000 7f500abd0740 1 -- 172.21.15.33:0/1042025955 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ff0018070 con 0x55b203bee2d0 2024-03-19T02:54:55.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.018+0000 7f4fe97fa640 1 -- 172.21.15.33:0/1042025955 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4ffc09c0b0 con 0x55b203bee2d0 2024-03-19T02:54:55.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.018+0000 7f500abd0740 10 client.25823 fetch_fsmap learned FSMap version 25 2024-03-19T02:54:55.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.018+0000 7f500abd0740 1 -- 172.21.15.33:0/1042025955 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b20392f7d0 con 0x55b203bee2d0 2024-03-19T02:54:55.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.018+0000 7f4fe97fa640 1 -- 172.21.15.33:0/1042025955 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4ff0018070 con 0x55b203bee2d0 2024-03-19T02:54:55.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.020+0000 7f4fe97fa640 1 -- 172.21.15.33:0/1042025955 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f4ffc09cce0 con 0x55b203bee2d0 2024-03-19T02:54:55.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.020+0000 7f500abd0740 10 client.25823 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:54:55.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.020+0000 7f500abd0740 10 client.25823 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T02:54:55.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.020+0000 7f500abd0740 20 client.25823 populate_metadata read hostname 'smithi033' 2024-03-19T02:54:55.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.020+0000 7f500abd0740 1 --2- 172.21.15.33:0/1042025955 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55b203c07c00 0x55b203c09ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.020+0000 7f500abd0740 4 client.25823 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:54:55.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.020+0000 7f500abd0740 1 -- 172.21.15.33:0/1042025955 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b203baf3e0 con 0x55b203c07c00 2024-03-19T02:54:55.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.021+0000 7f5005581640 1 --2- 172.21.15.33:0/1042025955 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55b203c07c00 0x55b203c09ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.022+0000 7f4fe97fa640 10 client.25823 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T02:54:55.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.022+0000 7f5005581640 1 --2- 172.21.15.33:0/1042025955 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55b203c07c00 0x55b203c09ff0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=575773de7f93ba01 server_cookie=2cabddc65c2b6000 in_seq=0 out_seq=0 2024-03-19T02:54:55.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.023+0000 7f4fe97fa640 1 -- 172.21.15.33:0/1042025955 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b203baf3e0 con 0x55b203c07c00 2024-03-19T02:54:55.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.023+0000 7f4fe97fa640 10 client.25823 handle_command_reply: tid=0 2024-03-19T02:54:55.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.023+0000 7f500abd0740 1 client.25823 shutdown 2024-03-19T02:54:55.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.023+0000 7f500abd0740 1 -- 172.21.15.33:0/1042025955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b203bee2d0 msgr2=0x55b203bf48f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.023+0000 7f500abd0740 1 --2- 172.21.15.33:0/1042025955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b203bee2d0 0x55b203bf48f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc0ccb70 tx=0x7f4ffc09b3d0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.024+0000 7f500abd0740 1 -- 172.21.15.33:0/1042025955 shutdown_connections 2024-03-19T02:54:55.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.024+0000 7f500abd0740 1 --2- 172.21.15.33:0/1042025955 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55b203c07c00 0x55b203c09ff0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.024+0000 7f4fe97fa640 0 client.25823 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T02:54:55.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.024+0000 7f500abd0740 1 --2- 172.21.15.33:0/1042025955 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b203beebf0 0x55b203bf4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.024+0000 7f500abd0740 1 --2- 172.21.15.33:0/1042025955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b203bf77d0 0x55b203bf5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.024+0000 7f500abd0740 1 --2- 172.21.15.33:0/1042025955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b203bee2d0 0x55b203bf48f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.024+0000 7f500abd0740 1 -- 172.21.15.33:0/1042025955 >> 172.21.15.33:0/1042025955 conn(0x55b203be8a00 msgr2=0x55b203bea750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:55.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.024+0000 7f500abd0740 1 -- 172.21.15.33:0/1042025955 shutdown_connections 2024-03-19T02:54:55.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.024+0000 7f500abd0740 1 -- 172.21.15.33:0/1042025955 wait complete. 2024-03-19T02:54:55.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.024+0000 7f500abd0740 20 client.25823 trim_cache size 0 max 16384 2024-03-19T02:54:55.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.049+0000 7f500abd0740 1 Processor -- start 2024-03-19T02:54:55.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.049+0000 7f500abd0740 1 -- start start 2024-03-19T02:54:55.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.049+0000 7f500abd0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b203bee2d0 0x55b203bf41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.049+0000 7f500abd0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b203beebf0 0x55b203c03360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.049+0000 7f500abd0740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b203bf77d0 0x55b203c0f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.049+0000 7f500abd0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f50001b3700 con 0x55b203beebf0 2024-03-19T02:54:55.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.049+0000 7f500abd0740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f50001b0fd0 con 0x55b203bf77d0 2024-03-19T02:54:55.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.049+0000 7f500abd0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f50001aee00 con 0x55b203bee2d0 2024-03-19T02:54:55.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.049+0000 7f5005581640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b203beebf0 0x55b203c03360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.049+0000 7f5005581640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b203beebf0 0x55b203c03360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53304/0 (socket says 172.21.15.33:53304) 2024-03-19T02:54:55.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.049+0000 7f5005581640 1 -- 172.21.15.33:0/4047765482 learned_addr learned my addr 172.21.15.33:0/4047765482 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:55.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.049+0000 7f5005d82640 1 --2- 172.21.15.33:0/4047765482 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b203bee2d0 0x55b203bf41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.049+0000 7f5006583640 1 --2- 172.21.15.33:0/4047765482 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b203bf77d0 0x55b203c0f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.050+0000 7f5005581640 1 -- 172.21.15.33:0/4047765482 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b203bee2d0 msgr2=0x55b203bf41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:55.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.050+0000 7f5005581640 1 --2- 172.21.15.33:0/4047765482 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b203bee2d0 0x55b203bf41d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.050+0000 7f5005581640 1 -- 172.21.15.33:0/4047765482 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b203bf77d0 msgr2=0x55b203c0f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:55.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.050+0000 7f5005581640 1 --2- 172.21.15.33:0/4047765482 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b203bf77d0 0x55b203c0f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.050+0000 7f5005581640 1 -- 172.21.15.33:0/4047765482 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b203c0a360 con 0x55b203beebf0 2024-03-19T02:54:55.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.050+0000 7f5005d82640 1 --2- 172.21.15.33:0/4047765482 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b203bee2d0 0x55b203bf41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:55.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.050+0000 7f5005581640 1 --2- 172.21.15.33:0/4047765482 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b203beebf0 0x55b203c03360 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0004810 tx=0x7f4ff000f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c12953affbda8af server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:55.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.050+0000 7f4feb7fe640 1 -- 172.21.15.33:0/4047765482 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff002d070 con 0x55b203beebf0 2024-03-19T02:54:55.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.050+0000 7f4feb7fe640 1 -- 172.21.15.33:0/4047765482 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4ff0018070 con 0x55b203beebf0 2024-03-19T02:54:55.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.050+0000 7f4feb7fe640 1 -- 172.21.15.33:0/4047765482 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff000d950 con 0x55b203beebf0 2024-03-19T02:54:55.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.051+0000 7f500abd0740 1 -- 172.21.15.33:0/4047765482 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b203beebf0 msgr2=0x55b203c03360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.051+0000 7f500abd0740 1 --2- 172.21.15.33:0/4047765482 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b203beebf0 0x55b203c03360 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0004810 tx=0x7f4ff000f040 comp rx=0 tx=0).stop 2024-03-19T02:54:55.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.051+0000 7f500abd0740 1 -- 172.21.15.33:0/4047765482 shutdown_connections 2024-03-19T02:54:55.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.051+0000 7f500abd0740 1 --2- 172.21.15.33:0/4047765482 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b203bf77d0 0x55b203c0f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.051+0000 7f500abd0740 1 --2- 172.21.15.33:0/4047765482 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b203beebf0 0x55b203c03360 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.051+0000 7f500abd0740 1 --2- 172.21.15.33:0/4047765482 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b203bee2d0 0x55b203bf41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.051+0000 7f500abd0740 1 -- 172.21.15.33:0/4047765482 >> 172.21.15.33:0/4047765482 conn(0x55b203be8a00 msgr2=0x55b203bf5140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:55.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.051+0000 7f500abd0740 1 -- 172.21.15.33:0/4047765482 shutdown_connections 2024-03-19T02:54:55.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.051+0000 7f500abd0740 1 -- 172.21.15.33:0/4047765482 wait complete. 2024-03-19T02:54:55.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.052+0000 7f500abd0740 1 -- start start 2024-03-19T02:54:55.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.052+0000 7f500abd0740 1 Processor -- start 2024-03-19T02:54:55.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.053+0000 7f500abd0740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b203bee2d0 0x55b203bffd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.054+0000 7f500abd0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b203beebf0 0x55b203bfb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.054+0000 7f5005581640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b203beebf0 0x55b203bfb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.054+0000 7f5005581640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b203beebf0 0x55b203bfb0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53320/0 (socket says 172.21.15.33:53320) 2024-03-19T02:54:55.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.054+0000 7f5005581640 1 -- 172.21.15.33:0/2521409261 learned_addr learned my addr 172.21.15.33:0/2521409261 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:55.056 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.054+0000 7f500abd0740 1 --2- 172.21.15.33:0/2521409261 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b203bf77d0 0x55b203bfb830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.054+0000 7f500abd0740 1 -- 172.21.15.33:0/2521409261 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f50001af490 con 0x55b203beebf0 2024-03-19T02:54:55.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.054+0000 7f500abd0740 1 -- 172.21.15.33:0/2521409261 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f50001b0900 con 0x55b203bee2d0 2024-03-19T02:54:55.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.054+0000 7f500abd0740 1 -- 172.21.15.33:0/2521409261 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f50001afb60 con 0x55b203bf77d0 2024-03-19T02:54:55.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.054+0000 7f5005d82640 1 --2- 172.21.15.33:0/2521409261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b203bee2d0 0x55b203bffd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.054+0000 7f500abd0740 10 client.0 _ll_register_callbacks cb 0x7f4ffc003d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:54:55.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.055+0000 7f5005581640 1 -- 172.21.15.33:0/2521409261 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b203bf77d0 msgr2=0x55b203bfb830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:55.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.055+0000 7f5005581640 1 --2- 172.21.15.33:0/2521409261 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b203bf77d0 0x55b203bfb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.055+0000 7f5005581640 1 -- 172.21.15.33:0/2521409261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b203bee2d0 msgr2=0x55b203bffd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:55.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.055+0000 7f5005581640 1 --2- 172.21.15.33:0/2521409261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b203bee2d0 0x55b203bffd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.055+0000 7f5005581640 1 -- 172.21.15.33:0/2521409261 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b203c6ced0 con 0x55b203beebf0 2024-03-19T02:54:55.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.055+0000 7f5005581640 1 --2- 172.21.15.33:0/2521409261 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b203beebf0 0x55b203bfb0f0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f4ff00077d0 tx=0x7f4ff0007300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7db0ad2f3db32767 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:55.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.055+0000 7f4fe97fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:54:55.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.056+0000 7f4fe97fa640 1 -- 172.21.15.33:0/2521409261 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff002d040 con 0x55b203beebf0 2024-03-19T02:54:55.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.056+0000 7f500abd0740 1 -- 172.21.15.33:0/2521409261 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ff0018070 con 0x55b203beebf0 2024-03-19T02:54:55.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.056+0000 7f4fe97fa640 1 -- 172.21.15.33:0/2521409261 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f4ff0027cb0 con 0x55b203beebf0 2024-03-19T02:54:55.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.057+0000 7f4fe97fa640 1 -- 172.21.15.33:0/2521409261 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4ff0013a90 con 0x55b203beebf0 2024-03-19T02:54:55.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.057+0000 7f500abd0740 10 client.15969 fetch_fsmap learned FSMap version 25 2024-03-19T02:54:55.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.057+0000 7f500abd0740 1 -- 172.21.15.33:0/2521409261 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b203c6dff0 con 0x55b203beebf0 2024-03-19T02:54:55.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.058+0000 7f4fe97fa640 1 -- 172.21.15.33:0/2521409261 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4ff0018070 con 0x55b203beebf0 2024-03-19T02:54:55.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.058+0000 7f4fe97fa640 1 -- 172.21.15.33:0/2521409261 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f4ff001d400 con 0x55b203beebf0 2024-03-19T02:54:55.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.058+0000 7f500abd0740 10 client.15969 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:54:55.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.058+0000 7f500abd0740 10 client.15969 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T02:54:55.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.058+0000 7f500abd0740 20 client.15969 populate_metadata read hostname 'smithi033' 2024-03-19T02:54:55.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.058+0000 7f500abd0740 1 --2- 172.21.15.33:0/2521409261 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55b203c6e440 0x55b203c70860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.058+0000 7f500abd0740 4 client.15969 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:54:55.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.058+0000 7f500abd0740 1 -- 172.21.15.33:0/2521409261 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b203b5e9f0 con 0x55b203c6e440 2024-03-19T02:54:55.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.059+0000 7f5005d82640 1 --2- 172.21.15.33:0/2521409261 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55b203c6e440 0x55b203c70860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.060+0000 7f5005d82640 1 --2- 172.21.15.33:0/2521409261 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55b203c6e440 0x55b203c70860 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=35a75d2d270fb057 server_cookie=920e8876a80a771d in_seq=0 out_seq=0 2024-03-19T02:54:55.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.060+0000 7f4fe97fa640 10 client.15969 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T02:54:55.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.060+0000 7f4fe97fa640 1 -- 172.21.15.33:0/2521409261 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b203b5e9f0 con 0x55b203c6e440 2024-03-19T02:54:55.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.060+0000 7f4fe97fa640 10 client.15969 handle_command_reply: tid=0 2024-03-19T02:54:55.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.060+0000 7f500abd0740 1 client.15969 shutdown 2024-03-19T02:54:55.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.060+0000 7f500abd0740 1 -- 172.21.15.33:0/2521409261 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b203beebf0 msgr2=0x55b203bfb0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.060+0000 7f500abd0740 1 --2- 172.21.15.33:0/2521409261 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b203beebf0 0x55b203bfb0f0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f4ff00077d0 tx=0x7f4ff0007300 comp rx=0 tx=0).stop 2024-03-19T02:54:55.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.060+0000 7f500abd0740 1 -- 172.21.15.33:0/2521409261 shutdown_connections 2024-03-19T02:54:55.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.060+0000 7f500abd0740 1 --2- 172.21.15.33:0/2521409261 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55b203c6e440 0x55b203c70860 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.061+0000 7f500abd0740 1 --2- 172.21.15.33:0/2521409261 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b203bf77d0 0x55b203bfb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.061+0000 7f4fe97fa640 0 client.15969 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T02:54:55.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.061+0000 7f500abd0740 1 --2- 172.21.15.33:0/2521409261 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b203beebf0 0x55b203bfb0f0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.061+0000 7f500abd0740 1 --2- 172.21.15.33:0/2521409261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b203bee2d0 0x55b203bffd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.061+0000 7f500abd0740 1 -- 172.21.15.33:0/2521409261 >> 172.21.15.33:0/2521409261 conn(0x55b203be8a00 msgr2=0x55b203bf48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:55.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.061+0000 7f500abd0740 1 -- 172.21.15.33:0/2521409261 shutdown_connections 2024-03-19T02:54:55.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.061+0000 7f500abd0740 1 -- 172.21.15.33:0/2521409261 wait complete. 2024-03-19T02:54:55.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.061+0000 7f500abd0740 20 client.15969 trim_cache size 0 max 16384 2024-03-19T02:54:55.063 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:54:55.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.062+0000 7f5007fe6640 1 -- 172.21.15.33:0/2397146602 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4fec0a2a90 msgr2=0x7f4fec0a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.062+0000 7f5007fe6640 1 --2- 172.21.15.33:0/2397146602 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4fec0a2a90 0x7f4fec0a4f50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4ff8036040 tx=0x7f4ff8009330 comp rx=0 tx=0).stop 2024-03-19T02:54:55.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.062+0000 7f5007fe6640 1 -- 172.21.15.33:0/2397146602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50001039c0 msgr2=0x7f500019a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.062+0000 7f5007fe6640 1 --2- 172.21.15.33:0/2397146602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50001039c0 0x7f500019a1b0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc00e9c0 tx=0x7f4ffc00ee80 comp rx=0 tx=0).stop 2024-03-19T02:54:55.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.062+0000 7f5007fe6640 1 -- 172.21.15.33:0/2397146602 shutdown_connections 2024-03-19T02:54:55.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.062+0000 7f5007fe6640 1 --2- 172.21.15.33:0/2397146602 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4fec0a2a90 0x7f4fec0a4f50 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.062+0000 7f5007fe6640 1 --2- 172.21.15.33:0/2397146602 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50001099d0 0x7f500019ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.062+0000 7f5007fe6640 1 --2- 172.21.15.33:0/2397146602 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5000104380 0x7f500019a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.062+0000 7f5007fe6640 1 --2- 172.21.15.33:0/2397146602 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50001039c0 0x7f500019a1b0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.062+0000 7f5007fe6640 1 -- 172.21.15.33:0/2397146602 >> 172.21.15.33:0/2397146602 conn(0x7f50000ff4d0 msgr2=0x7f50000ffb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:55.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.062+0000 7f5007fe6640 1 -- 172.21.15.33:0/2397146602 shutdown_connections 2024-03-19T02:54:55.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.062+0000 7f5007fe6640 1 -- 172.21.15.33:0/2397146602 wait complete. 2024-03-19T02:54:55.084 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:54:55.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.262+0000 7f5e24bcd640 1 -- 172.21.15.33:0/2258967333 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e201066b0 msgr2=0x7f5e20106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.262+0000 7f5e24bcd640 1 --2- 172.21.15.33:0/2258967333 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e201066b0 0x7f5e20106a90 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f5e10009930 tx=0x7f5e1002bc30 comp rx=0 tx=0).stop 2024-03-19T02:54:55.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.262+0000 7f5e24bcd640 1 -- 172.21.15.33:0/2258967333 shutdown_connections 2024-03-19T02:54:55.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.262+0000 7f5e24bcd640 1 --2- 172.21.15.33:0/2258967333 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e20101060 0x7f5e2010d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.262+0000 7f5e24bcd640 1 --2- 172.21.15.33:0/2258967333 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e201006a0 0x7f5e20100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.262+0000 7f5e24bcd640 1 --2- 172.21.15.33:0/2258967333 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e201066b0 0x7f5e20106a90 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.262+0000 7f5e24bcd640 1 -- 172.21.15.33:0/2258967333 >> 172.21.15.33:0/2258967333 conn(0x7f5e200778d0 msgr2=0x7f5e20077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:55.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.262+0000 7f5e24bcd640 1 -- 172.21.15.33:0/2258967333 shutdown_connections 2024-03-19T02:54:55.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.262+0000 7f5e24bcd640 1 -- 172.21.15.33:0/2258967333 wait complete. 2024-03-19T02:54:55.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.263+0000 7f5e24bcd640 1 Processor -- start 2024-03-19T02:54:55.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.263+0000 7f5e24bcd640 1 -- start start 2024-03-19T02:54:55.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.263+0000 7f5e24bcd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e201006a0 0x7f5e20198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.263+0000 7f5e24bcd640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e20101060 0x7f5e201985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.263+0000 7f5e24bcd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e201066b0 0x7f5e2019c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.263+0000 7f5e24bcd640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5e20110970 con 0x7f5e201006a0 2024-03-19T02:54:55.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.263+0000 7f5e24bcd640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5e201107f0 con 0x7f5e201066b0 2024-03-19T02:54:55.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.263+0000 7f5e24bcd640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5e20110af0 con 0x7f5e20101060 2024-03-19T02:54:55.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.264+0000 7f5e1e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e201006a0 0x7f5e20198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.264+0000 7f5e1e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e201006a0 0x7f5e20198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53330/0 (socket says 172.21.15.33:53330) 2024-03-19T02:54:55.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.264+0000 7f5e1e59c640 1 -- 172.21.15.33:0/312330200 learned_addr learned my addr 172.21.15.33:0/312330200 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:55.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.264+0000 7f5e1e59c640 1 -- 172.21.15.33:0/312330200 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e20101060 msgr2=0x7f5e201985c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-19T02:54:55.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.264+0000 7f5e1e59c640 1 --2- 172.21.15.33:0/312330200 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e20101060 0x7f5e201985c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.264+0000 7f5e1e59c640 1 -- 172.21.15.33:0/312330200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e201066b0 msgr2=0x7f5e2019c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.264+0000 7f5e1e59c640 1 --2- 172.21.15.33:0/312330200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e201066b0 0x7f5e2019c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.264+0000 7f5e1e59c640 1 -- 172.21.15.33:0/312330200 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e2019cff0 con 0x7f5e201006a0 2024-03-19T02:54:55.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.264+0000 7f5e1e59c640 1 --2- 172.21.15.33:0/312330200 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e201006a0 0x7f5e20198080 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f5e10009230 tx=0x7f5e100045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:55.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.265+0000 7f5e177fe640 1 -- 172.21.15.33:0/312330200 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e10043020 con 0x7f5e201006a0 2024-03-19T02:54:55.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.265+0000 7f5e177fe640 1 -- 172.21.15.33:0/312330200 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e1002cb70 con 0x7f5e201006a0 2024-03-19T02:54:55.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.265+0000 7f5e177fe640 1 -- 172.21.15.33:0/312330200 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e10003b80 con 0x7f5e201006a0 2024-03-19T02:54:55.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.265+0000 7f5e24bcd640 1 -- 172.21.15.33:0/312330200 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e2019d280 con 0x7f5e201006a0 2024-03-19T02:54:55.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.265+0000 7f5e24bcd640 1 -- 172.21.15.33:0/312330200 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e2006b240 con 0x7f5e201006a0 2024-03-19T02:54:55.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.269+0000 7f5e277b0740 1 Processor -- start 2024-03-19T02:54:55.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.270+0000 7f5e277b0740 1 -- start start 2024-03-19T02:54:55.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.269+0000 7f5e177fe640 1 -- 172.21.15.33:0/312330200 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7f5e10046020 con 0x7f5e201006a0 2024-03-19T02:54:55.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.270+0000 7f5e277b0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6eff692d0 0x55f6eff696b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.270+0000 7f5e277b0740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6eff69bf0 0x55f6eff72290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.270+0000 7f5e277b0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6eff727d0 0x55f6eff74bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.270+0000 7f5e277b0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f6efecc4b0 con 0x55f6eff727d0 2024-03-19T02:54:55.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.270+0000 7f5e277b0740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55f6efe74d70 con 0x55f6eff69bf0 2024-03-19T02:54:55.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.270+0000 7f5e277b0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f6efec8550 con 0x55f6eff692d0 2024-03-19T02:54:55.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.270+0000 7f5e17fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6eff727d0 0x55f6eff74bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.270+0000 7f5e17fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6eff727d0 0x55f6eff74bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53346/0 (socket says 172.21.15.33:53346) 2024-03-19T02:54:55.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.270+0000 7f5e17fff640 1 -- 172.21.15.33:0/3689283864 learned_addr learned my addr 172.21.15.33:0/3689283864 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:55.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.271+0000 7f5e17fff640 1 -- 172.21.15.33:0/3689283864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6eff692d0 msgr2=0x55f6eff696b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:55.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.271+0000 7f5e17fff640 1 --2- 172.21.15.33:0/3689283864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6eff692d0 0x55f6eff696b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.271+0000 7f5e17fff640 1 -- 172.21.15.33:0/3689283864 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6eff69bf0 msgr2=0x55f6eff72290 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T02:54:55.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.271+0000 7f5e17fff640 1 --2- 172.21.15.33:0/3689283864 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6eff69bf0 0x55f6eff72290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.271+0000 7f5e17fff640 1 -- 172.21.15.33:0/3689283864 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6efe97210 con 0x55f6eff727d0 2024-03-19T02:54:55.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.271+0000 7f5e17fff640 1 --2- 172.21.15.33:0/3689283864 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6eff727d0 0x55f6eff74bc0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f5e0800ba50 tx=0x7f5e0800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2da62bad3a62d921 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:55.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.271+0000 7f5e177fe640 1 --2- 172.21.15.33:0/312330200 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f5e000a2b90 0x7f5e000a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.271+0000 7f5e14ff9640 1 -- 172.21.15.33:0/3689283864 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e08016070 con 0x55f6eff727d0 2024-03-19T02:54:55.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.271+0000 7f5e14ff9640 1 -- 172.21.15.33:0/3689283864 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e080054d0 con 0x55f6eff727d0 2024-03-19T02:54:55.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.271+0000 7f5e14ff9640 1 -- 172.21.15.33:0/3689283864 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e08010460 con 0x55f6eff727d0 2024-03-19T02:54:55.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.271+0000 7f5e1ed9d640 1 --2- 172.21.15.33:0/312330200 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f5e000a2b90 0x7f5e000a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.271+0000 7f5e177fe640 1 -- 172.21.15.33:0/312330200 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f5e100f15b0 con 0x7f5e201006a0 2024-03-19T02:54:55.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.272+0000 7f5e1ed9d640 1 --2- 172.21.15.33:0/312330200 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f5e000a2b90 0x7f5e000a5050 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5e1800aa90 tx=0x7f5e18009250 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:55.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.272+0000 7f5e277b0740 1 -- 172.21.15.33:0/3689283864 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6eff727d0 msgr2=0x55f6eff74bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.272+0000 7f5e277b0740 1 --2- 172.21.15.33:0/3689283864 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6eff727d0 0x55f6eff74bc0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f5e0800ba50 tx=0x7f5e0800bf10 comp rx=0 tx=0).stop 2024-03-19T02:54:55.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.272+0000 7f5e277b0740 1 -- 172.21.15.33:0/3689283864 shutdown_connections 2024-03-19T02:54:55.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.272+0000 7f5e277b0740 1 --2- 172.21.15.33:0/3689283864 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6eff727d0 0x55f6eff74bc0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.272+0000 7f5e277b0740 1 --2- 172.21.15.33:0/3689283864 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6eff69bf0 0x55f6eff72290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.272+0000 7f5e277b0740 1 --2- 172.21.15.33:0/3689283864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6eff692d0 0x55f6eff696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.272+0000 7f5e277b0740 1 -- 172.21.15.33:0/3689283864 >> 172.21.15.33:0/3689283864 conn(0x55f6eff63a00 msgr2=0x55f6eff63de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:55.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.272+0000 7f5e277b0740 1 -- 172.21.15.33:0/3689283864 shutdown_connections 2024-03-19T02:54:55.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.273+0000 7f5e277b0740 1 -- 172.21.15.33:0/3689283864 wait complete. 2024-03-19T02:54:55.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.273+0000 7f5e277b0740 1 -- start start 2024-03-19T02:54:55.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.273+0000 7f5e277b0740 1 Processor -- start 2024-03-19T02:54:55.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.274+0000 7f5e277b0740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6eff692d0 0x55f6eff6f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.274+0000 7f5e277b0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6eff69bf0 0x55f6eff6fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.274+0000 7f5e277b0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6eff727d0 0x55f6eff702e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.274+0000 7f5e277b0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f6eff755c0 con 0x55f6eff727d0 2024-03-19T02:54:55.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.274+0000 7f5e277b0740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55f6efe88310 con 0x55f6eff692d0 2024-03-19T02:54:55.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.274+0000 7f5e277b0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f6eff75740 con 0x55f6eff69bf0 2024-03-19T02:54:55.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.274+0000 7f5e277b0740 10 client.0 _ll_register_callbacks cb 0x55f6efe7c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:54:55.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.274+0000 7f5e1ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6eff727d0 0x55f6eff702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.274+0000 7f5e1ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6eff727d0 0x55f6eff702e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53354/0 (socket says 172.21.15.33:53354) 2024-03-19T02:54:55.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.274+0000 7f5e1ed9d640 1 -- 172.21.15.33:0/1512334998 learned_addr learned my addr 172.21.15.33:0/1512334998 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:55.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.274+0000 7f5e1ed9d640 1 -- 172.21.15.33:0/1512334998 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6eff69bf0 msgr2=0x55f6eff6fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:55.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.274+0000 7f5e1ed9d640 1 --2- 172.21.15.33:0/1512334998 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6eff69bf0 0x55f6eff6fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.274+0000 7f5e1ed9d640 1 -- 172.21.15.33:0/1512334998 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6eff692d0 msgr2=0x55f6eff6f860 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T02:54:55.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.274+0000 7f5e1ed9d640 1 --2- 172.21.15.33:0/1512334998 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6eff692d0 0x55f6eff6f860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.274+0000 7f5e1ed9d640 1 -- 172.21.15.33:0/1512334998 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6efbe03f0 con 0x55f6eff727d0 2024-03-19T02:54:55.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.274+0000 7f5e1ed9d640 1 --2- 172.21.15.33:0/1512334998 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6eff727d0 0x55f6eff702e0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f5e18004b90 tx=0x7f5e18004c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63b99009c85a2dc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:55.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.274+0000 7f5e0dffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:54:55.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.274+0000 7f5e0dffb640 1 -- 172.21.15.33:0/1512334998 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e18018d90 con 0x55f6eff727d0 2024-03-19T02:54:55.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.275+0000 7f5e0dffb640 1 -- 172.21.15.33:0/1512334998 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f5e18023480 con 0x55f6eff727d0 2024-03-19T02:54:55.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.275+0000 7f5e277b0740 1 -- 172.21.15.33:0/1512334998 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e080054d0 con 0x55f6eff727d0 2024-03-19T02:54:55.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.275+0000 7f5e0dffb640 1 -- 172.21.15.33:0/1512334998 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f5e18023680 con 0x55f6eff727d0 2024-03-19T02:54:55.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.275+0000 7f5e277b0740 10 client.15981 fetch_fsmap learned FSMap version 25 2024-03-19T02:54:55.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.275+0000 7f5e277b0740 1 -- 172.21.15.33:0/1512334998 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f6efcaa7d0 con 0x55f6eff727d0 2024-03-19T02:54:55.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.275+0000 7f5e0dffb640 1 -- 172.21.15.33:0/1512334998 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e080054d0 con 0x55f6eff727d0 2024-03-19T02:54:55.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.275+0000 7f5e0dffb640 1 -- 172.21.15.33:0/1512334998 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f5e1801a980 con 0x55f6eff727d0 2024-03-19T02:54:55.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.275+0000 7f5e277b0740 10 client.15981 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:54:55.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.275+0000 7f5e277b0740 10 client.15981 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T02:54:55.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.275+0000 7f5e277b0740 20 client.15981 populate_metadata read hostname 'smithi033' 2024-03-19T02:54:55.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.275+0000 7f5e277b0740 1 --2- 172.21.15.33:0/1512334998 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55f6eff82aa0 0x55f6eff84f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.275+0000 7f5e277b0740 4 client.15981 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:54:55.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.275+0000 7f5e277b0740 1 -- 172.21.15.33:0/1512334998 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6eff2a3e0 con 0x55f6eff82aa0 2024-03-19T02:54:55.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.276+0000 7f5e1e59c640 1 --2- 172.21.15.33:0/1512334998 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55f6eff82aa0 0x55f6eff84f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.276+0000 7f5e0dffb640 10 client.15981 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T02:54:55.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.276+0000 7f5e1e59c640 1 --2- 172.21.15.33:0/1512334998 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55f6eff82aa0 0x55f6eff84f10 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5df2b42fda5e0cd7 server_cookie=7919d8c83b6907ec in_seq=0 out_seq=0 2024-03-19T02:54:55.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.277+0000 7f5e0dffb640 1 -- 172.21.15.33:0/1512334998 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f6eff2a3e0 con 0x55f6eff82aa0 2024-03-19T02:54:55.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.277+0000 7f5e0dffb640 10 client.15981 handle_command_reply: tid=0 2024-03-19T02:54:55.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.277+0000 7f5e277b0740 1 client.15981 shutdown 2024-03-19T02:54:55.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.277+0000 7f5e277b0740 1 -- 172.21.15.33:0/1512334998 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6eff727d0 msgr2=0x55f6eff702e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.277+0000 7f5e277b0740 1 --2- 172.21.15.33:0/1512334998 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6eff727d0 0x55f6eff702e0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f5e18004b90 tx=0x7f5e18004c30 comp rx=0 tx=0).stop 2024-03-19T02:54:55.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.277+0000 7f5e277b0740 1 -- 172.21.15.33:0/1512334998 shutdown_connections 2024-03-19T02:54:55.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.277+0000 7f5e277b0740 1 --2- 172.21.15.33:0/1512334998 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55f6eff82aa0 0x55f6eff84f10 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.277+0000 7f5e277b0740 1 --2- 172.21.15.33:0/1512334998 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6eff69bf0 0x55f6eff6fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.277+0000 7f5e0dffb640 0 client.15981 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T02:54:55.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.277+0000 7f5e277b0740 1 --2- 172.21.15.33:0/1512334998 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6eff727d0 0x55f6eff702e0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.277+0000 7f5e277b0740 1 --2- 172.21.15.33:0/1512334998 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6eff692d0 0x55f6eff6f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.277+0000 7f5e277b0740 1 -- 172.21.15.33:0/1512334998 >> 172.21.15.33:0/1512334998 conn(0x55f6eff63a00 msgr2=0x55f6eff65750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:55.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.277+0000 7f5e277b0740 1 -- 172.21.15.33:0/1512334998 shutdown_connections 2024-03-19T02:54:55.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.277+0000 7f5e277b0740 1 -- 172.21.15.33:0/1512334998 wait complete. 2024-03-19T02:54:55.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.278+0000 7f5e277b0740 20 client.15981 trim_cache size 0 max 16384 2024-03-19T02:54:55.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.291+0000 7f5e277b0740 1 Processor -- start 2024-03-19T02:54:55.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.291+0000 7f5e277b0740 1 -- start start 2024-03-19T02:54:55.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.292+0000 7f5e277b0740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6eff692d0 0x55f6eff71f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.292+0000 7f5e277b0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6eff69bf0 0x55f6eff7e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.292+0000 7f5e277b0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6eff727d0 0x55f6eff8a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.292+0000 7f5e277b0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5e201b9d50 con 0x55f6eff69bf0 2024-03-19T02:54:55.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.292+0000 7f5e277b0740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5e201b7620 con 0x55f6eff692d0 2024-03-19T02:54:55.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.292+0000 7f5e277b0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5e20199380 con 0x55f6eff727d0 2024-03-19T02:54:55.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.292+0000 7f5e17fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6eff69bf0 0x55f6eff7e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.292+0000 7f5e17fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6eff69bf0 0x55f6eff7e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53358/0 (socket says 172.21.15.33:53358) 2024-03-19T02:54:55.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.292+0000 7f5e17fff640 1 -- 172.21.15.33:0/109854241 learned_addr learned my addr 172.21.15.33:0/109854241 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:55.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.292+0000 7f5e1e59c640 1 --2- 172.21.15.33:0/109854241 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6eff692d0 0x55f6eff71f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.292+0000 7f5e1ed9d640 1 --2- 172.21.15.33:0/109854241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6eff727d0 0x55f6eff8a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.295 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-03-19T02:54:55.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.292+0000 7f5e1e59c640 1 -- 172.21.15.33:0/109854241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6eff727d0 msgr2=0x55f6eff8a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:55.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.292+0000 7f5e1e59c640 1 --2- 172.21.15.33:0/109854241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6eff727d0 0x55f6eff8a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.292+0000 7f5e1e59c640 1 -- 172.21.15.33:0/109854241 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6eff69bf0 msgr2=0x55f6eff7e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:55.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.292+0000 7f5e1e59c640 1 --2- 172.21.15.33:0/109854241 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6eff69bf0 0x55f6eff7e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.292+0000 7f5e1e59c640 1 -- 172.21.15.33:0/109854241 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6eff39880 con 0x55f6eff692d0 2024-03-19T02:54:55.296 INFO:tasks.workunit.client.0.smithi033.stdout: Children see throughput for 1 initial writers = 585641.88 kB/sec 2024-03-19T02:54:55.296 INFO:tasks.workunit.client.0.smithi033.stdout: Parent sees throughput for 1 initial writers = 585549.99 kB/sec 2024-03-19T02:54:55.296 INFO:tasks.workunit.client.0.smithi033.stdout: Min throughput per process = 585641.88 kB/sec 2024-03-19T02:54:55.296 INFO:tasks.workunit.client.0.smithi033.stdout: Max throughput per process = 585641.88 kB/sec 2024-03-19T02:54:55.296 INFO:tasks.workunit.client.0.smithi033.stdout: Avg throughput per process = 585641.88 kB/sec 2024-03-19T02:54:55.296 INFO:tasks.workunit.client.0.smithi033.stdout: Min xfer = 1048576.00 kB 2024-03-19T02:54:55.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.292+0000 7f5e1e59c640 1 --2- 172.21.15.33:0/109854241 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6eff692d0 0x55f6eff71f90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5e100ee8e0 tx=0x7f5e1002bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bba5b8348b2cea58 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:55.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.293+0000 7f5e0ffff640 1 -- 172.21.15.33:0/109854241 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e100bdf40 con 0x55f6eff692d0 2024-03-19T02:54:55.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.293+0000 7f5e0ffff640 1 -- 172.21.15.33:0/109854241 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e100dc070 con 0x55f6eff692d0 2024-03-19T02:54:55.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.293+0000 7f5e0ffff640 1 -- 172.21.15.33:0/109854241 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e100bbbd0 con 0x55f6eff692d0 2024-03-19T02:54:55.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.293+0000 7f5e277b0740 1 -- 172.21.15.33:0/109854241 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6eff692d0 msgr2=0x55f6eff71f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.293+0000 7f5e277b0740 1 --2- 172.21.15.33:0/109854241 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6eff692d0 0x55f6eff71f90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5e100ee8e0 tx=0x7f5e1002bc30 comp rx=0 tx=0).stop 2024-03-19T02:54:55.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.293+0000 7f5e277b0740 1 -- 172.21.15.33:0/109854241 shutdown_connections 2024-03-19T02:54:55.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.293+0000 7f5e277b0740 1 --2- 172.21.15.33:0/109854241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6eff727d0 0x55f6eff8a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.293+0000 7f5e277b0740 1 --2- 172.21.15.33:0/109854241 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6eff69bf0 0x55f6eff7e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.293+0000 7f5e277b0740 1 --2- 172.21.15.33:0/109854241 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6eff692d0 0x55f6eff71f90 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.293+0000 7f5e277b0740 1 -- 172.21.15.33:0/109854241 >> 172.21.15.33:0/109854241 conn(0x55f6eff63a00 msgr2=0x55f6eff62ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:55.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.293+0000 7f5e277b0740 1 -- 172.21.15.33:0/109854241 shutdown_connections 2024-03-19T02:54:55.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.293+0000 7f5e277b0740 1 -- 172.21.15.33:0/109854241 wait complete. 2024-03-19T02:54:55.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.294+0000 7f5e277b0740 1 -- start start 2024-03-19T02:54:55.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.294+0000 7f5e277b0740 1 Processor -- start 2024-03-19T02:54:55.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.294+0000 7f5e277b0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6eff692d0 0x55f6eff76b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.294+0000 7f5e277b0740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6eff69bf0 0x55f6eff77070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.294+0000 7f5e277b0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6eff727d0 0x55f6effe1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.294+0000 7f5e17fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6eff69bf0 0x55f6eff77070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.294+0000 7f5e1e59c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6eff692d0 0x55f6eff76b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.294+0000 7f5e1e59c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6eff692d0 0x55f6eff76b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:52178/0 (socket says 172.21.15.33:52178) 2024-03-19T02:54:55.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.294+0000 7f5e17fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6eff69bf0 0x55f6eff77070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:50918/0 (socket says 172.21.15.33:50918) 2024-03-19T02:54:55.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.294+0000 7f5e1ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6eff727d0 0x55f6effe1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.294+0000 7f5e1ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6eff727d0 0x55f6effe1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53372/0 (socket says 172.21.15.33:53372) 2024-03-19T02:54:55.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.294+0000 7f5e1e59c640 1 -- 172.21.15.33:0/3503971549 learned_addr learned my addr 172.21.15.33:0/3503971549 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:55.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.294+0000 7f5e277b0740 1 -- 172.21.15.33:0/3503971549 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5e201b5bb0 con 0x55f6eff727d0 2024-03-19T02:54:55.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.295+0000 7f5e277b0740 1 -- 172.21.15.33:0/3503971549 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5e201b6f50 con 0x55f6eff69bf0 2024-03-19T02:54:55.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.295+0000 7f5e277b0740 1 -- 172.21.15.33:0/3503971549 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5e201b61b0 con 0x55f6eff692d0 2024-03-19T02:54:55.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.295+0000 7f5e277b0740 10 client.0 _ll_register_callbacks cb 0x7f5e1800a780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:54:55.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.295+0000 7f5e17fff640 1 -- 172.21.15.33:0/3503971549 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6eff692d0 msgr2=0x55f6eff76b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:55.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.295+0000 7f5e17fff640 1 --2- 172.21.15.33:0/3503971549 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6eff692d0 0x55f6eff76b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.295+0000 7f5e17fff640 1 -- 172.21.15.33:0/3503971549 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6eff727d0 msgr2=0x55f6effe1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:55.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.295+0000 7f5e17fff640 1 --2- 172.21.15.33:0/3503971549 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6eff727d0 0x55f6effe1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.295+0000 7f5e17fff640 1 -- 172.21.15.33:0/3503971549 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6effe2490 con 0x55f6eff69bf0 2024-03-19T02:54:55.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.295+0000 7f5e1e59c640 1 --2- 172.21.15.33:0/3503971549 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6eff692d0 0x55f6eff76b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:55.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.295+0000 7f5e1ed9d640 1 --2- 172.21.15.33:0/3503971549 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6eff727d0 0x55f6effe1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:55.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.295+0000 7f5e17fff640 1 --2- 172.21.15.33:0/3503971549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6eff69bf0 0x55f6eff77070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x55f6eff64860 tx=0x7f5e0800dfc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3daec03340deabf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:55.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.295+0000 7f5e0dffb640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T02:54:55.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.295+0000 7f5e0dffb640 1 -- 172.21.15.33:0/3503971549 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e08028060 con 0x55f6eff69bf0 2024-03-19T02:54:55.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.295+0000 7f5e0dffb640 1 -- 172.21.15.33:0/3503971549 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f5e080052c0 con 0x55f6eff69bf0 2024-03-19T02:54:55.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.295+0000 7f5e277b0740 1 -- 172.21.15.33:0/3503971549 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e100dc070 con 0x55f6eff69bf0 2024-03-19T02:54:55.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.296+0000 7f5e0dffb640 1 -- 172.21.15.33:0/3503971549 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f5e0802f050 con 0x55f6eff69bf0 2024-03-19T02:54:55.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.296+0000 7f5e277b0740 10 client.25838 fetch_fsmap learned FSMap version 25 2024-03-19T02:54:55.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.296+0000 7f5e277b0740 1 -- 172.21.15.33:0/3503971549 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f6effe9870 con 0x55f6eff69bf0 2024-03-19T02:54:55.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.296+0000 7f5e0dffb640 1 -- 172.21.15.33:0/3503971549 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e100dc070 con 0x55f6eff69bf0 2024-03-19T02:54:55.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.296+0000 7f5e0dffb640 1 -- 172.21.15.33:0/3503971549 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f5e08010460 con 0x55f6eff69bf0 2024-03-19T02:54:55.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.296+0000 7f5e277b0740 10 client.25838 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:54:55.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.296+0000 7f5e277b0740 10 client.25838 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T02:54:55.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.296+0000 7f5e277b0740 20 client.25838 populate_metadata read hostname 'smithi033' 2024-03-19T02:54:55.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.296+0000 7f5e277b0740 1 --2- 172.21.15.33:0/3503971549 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55f6effe9d00 0x55f6effec0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.296+0000 7f5e277b0740 4 client.25838 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:54:55.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.296+0000 7f5e277b0740 1 -- 172.21.15.33:0/3503971549 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f6efed99f0 con 0x55f6effe9d00 2024-03-19T02:54:55.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.296+0000 7f5e1e59c640 1 --2- 172.21.15.33:0/3503971549 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55f6effe9d00 0x55f6effec0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.296+0000 7f5e0dffb640 10 client.25838 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T02:54:55.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.296+0000 7f5e1e59c640 1 --2- 172.21.15.33:0/3503971549 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55f6effe9d00 0x55f6effec0f0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e0c155e9e6dd0aac server_cookie=792f51243e34e69a in_seq=0 out_seq=0 2024-03-19T02:54:55.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.297+0000 7f5e0dffb640 1 -- 172.21.15.33:0/3503971549 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f6efed99f0 con 0x55f6effe9d00 2024-03-19T02:54:55.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.297+0000 7f5e0dffb640 10 client.25838 handle_command_reply: tid=0 2024-03-19T02:54:55.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.297+0000 7f5e277b0740 1 client.25838 shutdown 2024-03-19T02:54:55.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.297+0000 7f5e277b0740 1 -- 172.21.15.33:0/3503971549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6eff69bf0 msgr2=0x55f6eff77070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.297+0000 7f5e277b0740 1 --2- 172.21.15.33:0/3503971549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6eff69bf0 0x55f6eff77070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x55f6eff64860 tx=0x7f5e0800dfc0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.297+0000 7f5e277b0740 1 -- 172.21.15.33:0/3503971549 shutdown_connections 2024-03-19T02:54:55.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.297+0000 7f5e277b0740 1 --2- 172.21.15.33:0/3503971549 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55f6effe9d00 0x55f6effec0f0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.297+0000 7f5e277b0740 1 --2- 172.21.15.33:0/3503971549 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f6eff727d0 0x55f6effe1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.297+0000 7f5e277b0740 1 --2- 172.21.15.33:0/3503971549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f6eff69bf0 0x55f6eff77070 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.297+0000 7f5e0dffb640 0 client.25838 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T02:54:55.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.297+0000 7f5e277b0740 1 --2- 172.21.15.33:0/3503971549 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f6eff692d0 0x55f6eff76b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.297+0000 7f5e277b0740 1 -- 172.21.15.33:0/3503971549 >> 172.21.15.33:0/3503971549 conn(0x55f6eff63a00 msgr2=0x55f6eff6f820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:55.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.297+0000 7f5e277b0740 1 -- 172.21.15.33:0/3503971549 shutdown_connections 2024-03-19T02:54:55.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.297+0000 7f5e277b0740 1 -- 172.21.15.33:0/3503971549 wait complete. 2024-03-19T02:54:55.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.297+0000 7f5e277b0740 20 client.25838 trim_cache size 0 max 16384 2024-03-19T02:54:55.307 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:54:55.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.298+0000 7f5e24bcd640 1 -- 172.21.15.33:0/312330200 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f5e000a2b90 msgr2=0x7f5e000a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.298+0000 7f5e24bcd640 1 --2- 172.21.15.33:0/312330200 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f5e000a2b90 0x7f5e000a5050 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5e1800aa90 tx=0x7f5e18009250 comp rx=0 tx=0).stop 2024-03-19T02:54:55.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.298+0000 7f5e24bcd640 1 -- 172.21.15.33:0/312330200 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e201006a0 msgr2=0x7f5e20198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.298+0000 7f5e24bcd640 1 --2- 172.21.15.33:0/312330200 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e201006a0 0x7f5e20198080 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f5e10009230 tx=0x7f5e100045e0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.298+0000 7f5e24bcd640 1 -- 172.21.15.33:0/312330200 shutdown_connections 2024-03-19T02:54:55.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.298+0000 7f5e24bcd640 1 --2- 172.21.15.33:0/312330200 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f5e000a2b90 0x7f5e000a5050 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.298+0000 7f5e24bcd640 1 --2- 172.21.15.33:0/312330200 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e20101060 0x7f5e201985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.298+0000 7f5e24bcd640 1 --2- 172.21.15.33:0/312330200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e201066b0 0x7f5e2019c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.298+0000 7f5e24bcd640 1 --2- 172.21.15.33:0/312330200 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e201006a0 0x7f5e20198080 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.298+0000 7f5e24bcd640 1 -- 172.21.15.33:0/312330200 >> 172.21.15.33:0/312330200 conn(0x7f5e200778d0 msgr2=0x7f5e20103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:55.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.298+0000 7f5e24bcd640 1 -- 172.21.15.33:0/312330200 shutdown_connections 2024-03-19T02:54:55.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.298+0000 7f5e24bcd640 1 -- 172.21.15.33:0/312330200 wait complete. 2024-03-19T02:54:55.310 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:54:55.333 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:55 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/696516445' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:54:55.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.413+0000 7f9761b7e640 1 -- 172.21.15.33:0/1058514391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f975c101060 msgr2=0x7f975c10d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.413+0000 7f9761b7e640 1 --2- 172.21.15.33:0/1058514391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f975c101060 0x7f975c10d570 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9754009860 tx=0x7f975402bb70 comp rx=0 tx=0).stop 2024-03-19T02:54:55.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.413+0000 7f9761b7e640 1 -- 172.21.15.33:0/1058514391 shutdown_connections 2024-03-19T02:54:55.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.413+0000 7f9761b7e640 1 --2- 172.21.15.33:0/1058514391 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f975c1006a0 0x7f975c100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.413+0000 7f9761b7e640 1 --2- 172.21.15.33:0/1058514391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f975c101060 0x7f975c10d570 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.413+0000 7f9761b7e640 1 --2- 172.21.15.33:0/1058514391 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f975c1066b0 0x7f975c106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.413+0000 7f9761b7e640 1 -- 172.21.15.33:0/1058514391 >> 172.21.15.33:0/1058514391 conn(0x7f975c0778d0 msgr2=0x7f975c077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:55.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.413+0000 7f9761b7e640 1 -- 172.21.15.33:0/1058514391 shutdown_connections 2024-03-19T02:54:55.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.413+0000 7f9761b7e640 1 -- 172.21.15.33:0/1058514391 wait complete. 2024-03-19T02:54:55.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.413+0000 7f9761b7e640 1 Processor -- start 2024-03-19T02:54:55.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.414+0000 7f9761b7e640 1 -- start start 2024-03-19T02:54:55.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.414+0000 7f9761b7e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f975c1006a0 0x7f975c198d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.414+0000 7f9761b7e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f975c101060 0x7f975c199270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.414+0000 7f9761b7e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f975c1066b0 0x7f975c1a3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.414+0000 7f9761b7e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f975c110930 con 0x7f975c1066b0 2024-03-19T02:54:55.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.414+0000 7f9761b7e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f975c1107b0 con 0x7f975c1006a0 2024-03-19T02:54:55.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.414+0000 7f9761b7e640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f975c110ab0 con 0x7f975c101060 2024-03-19T02:54:55.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.414+0000 7f975bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f975c1066b0 0x7f975c1a3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.414+0000 7f975bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f975c1066b0 0x7f975c1a3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53384/0 (socket says 172.21.15.33:53384) 2024-03-19T02:54:55.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.414+0000 7f975bfff640 1 -- 172.21.15.33:0/1511890894 learned_addr learned my addr 172.21.15.33:0/1511890894 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:55.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.414+0000 7f975b7fe640 1 --2- 172.21.15.33:0/1511890894 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f975c1006a0 0x7f975c198d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.414+0000 7f975affd640 1 --2- 172.21.15.33:0/1511890894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f975c101060 0x7f975c199270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.414+0000 7f975bfff640 1 -- 172.21.15.33:0/1511890894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f975c101060 msgr2=0x7f975c199270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.414+0000 7f975bfff640 1 --2- 172.21.15.33:0/1511890894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f975c101060 0x7f975c199270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.414+0000 7f975bfff640 1 -- 172.21.15.33:0/1511890894 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f975c1006a0 msgr2=0x7f975c198d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.414+0000 7f975bfff640 1 --2- 172.21.15.33:0/1511890894 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f975c1006a0 0x7f975c198d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.414+0000 7f975bfff640 1 -- 172.21.15.33:0/1511890894 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f975c1a42f0 con 0x7f975c1066b0 2024-03-19T02:54:55.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.414+0000 7f975b7fe640 1 --2- 172.21.15.33:0/1511890894 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f975c1006a0 0x7f975c198d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:55.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.415+0000 7f975bfff640 1 --2- 172.21.15.33:0/1511890894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f975c1066b0 0x7f975c1a3db0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f975402c180 tx=0x7f9754035f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:55.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.415+0000 7f9758ff9640 1 -- 172.21.15.33:0/1511890894 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9754039040 con 0x7f975c1066b0 2024-03-19T02:54:55.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.415+0000 7f9761b7e640 1 -- 172.21.15.33:0/1511890894 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f975c1a44c0 con 0x7f975c1066b0 2024-03-19T02:54:55.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.415+0000 7f9758ff9640 1 -- 172.21.15.33:0/1511890894 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9754042070 con 0x7f975c1066b0 2024-03-19T02:54:55.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.415+0000 7f9758ff9640 1 -- 172.21.15.33:0/1511890894 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97540046c0 con 0x7f975c1066b0 2024-03-19T02:54:55.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.415+0000 7f9761b7e640 1 -- 172.21.15.33:0/1511890894 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f975c1a4980 con 0x7f975c1066b0 2024-03-19T02:54:55.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.417+0000 7f9758ff9640 1 -- 172.21.15.33:0/1511890894 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7f975402c510 con 0x7f975c1066b0 2024-03-19T02:54:55.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.417+0000 7f976476c740 1 Processor -- start 2024-03-19T02:54:55.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.417+0000 7f976476c740 1 -- start start 2024-03-19T02:54:55.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.417+0000 7f9758ff9640 1 --2- 172.21.15.33:0/1511890894 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f97440a2a90 0x7f97440a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.418+0000 7f976476c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df760e62d0 0x55df760e66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.418+0000 7f976476c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55df760e6bf0 0x55df760ef290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.418+0000 7f976476c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df760ef7d0 0x55df760f1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.418+0000 7f976476c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55df760494b0 con 0x55df760ef7d0 2024-03-19T02:54:55.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.418+0000 7f976476c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55df75ff1d70 con 0x55df760e6bf0 2024-03-19T02:54:55.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.418+0000 7f976476c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55df76045550 con 0x55df760e62d0 2024-03-19T02:54:55.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.418+0000 7f975affd640 1 --2- 172.21.15.33:0/1511890894 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f97440a2a90 0x7f97440a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.418+0000 7f9758ff9640 1 -- 172.21.15.33:0/1511890894 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f97540f0990 con 0x7f975c1066b0 2024-03-19T02:54:55.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.418+0000 7f975bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df760ef7d0 0x55df760f1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.418+0000 7f975bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df760ef7d0 0x55df760f1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53386/0 (socket says 172.21.15.33:53386) 2024-03-19T02:54:55.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.418+0000 7f975b7fe640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df760e62d0 0x55df760e66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.418+0000 7f975bfff640 1 -- 172.21.15.33:0/2995681739 learned_addr learned my addr 172.21.15.33:0/2995681739 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:55.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.418+0000 7f975affd640 1 --2- 172.21.15.33:0/2995681739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55df760e6bf0 0x55df760ef290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.418+0000 7f975bfff640 1 -- 172.21.15.33:0/2995681739 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df760e62d0 msgr2=0x55df760e66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:55.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.418+0000 7f975bfff640 1 --2- 172.21.15.33:0/2995681739 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df760e62d0 0x55df760e66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.418+0000 7f975bfff640 1 -- 172.21.15.33:0/2995681739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55df760e6bf0 msgr2=0x55df760ef290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:55.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.418+0000 7f975bfff640 1 --2- 172.21.15.33:0/2995681739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55df760e6bf0 0x55df760ef290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.418+0000 7f975bfff640 1 -- 172.21.15.33:0/2995681739 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df76014210 con 0x55df760ef7d0 2024-03-19T02:54:55.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.418+0000 7f975b7fe640 1 --2- 172.21.15.33:0/2995681739 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df760e62d0 0x55df760e66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:55.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.418+0000 7f975affd640 1 --2- 172.21.15.33:0/2995681739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55df760e6bf0 0x55df760ef290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:54:55.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.418+0000 7f975affd640 1 --2- 172.21.15.33:0/1511890894 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f97440a2a90 0x7f97440a4f50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f975c0694f0 tx=0x7f97500098b0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:55.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.418+0000 7f975bfff640 1 --2- 172.21.15.33:0/2995681739 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df760ef7d0 0x55df760f1bc0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f97540b9510 tx=0x7f9754005d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd88029c22aa4a77 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:55.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.418+0000 7f9741ffb640 1 -- 172.21.15.33:0/2995681739 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97540bd240 con 0x55df760ef7d0 2024-03-19T02:54:55.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.418+0000 7f9741ffb640 1 -- 172.21.15.33:0/2995681739 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f97540bcb40 con 0x55df760ef7d0 2024-03-19T02:54:55.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.418+0000 7f9741ffb640 1 -- 172.21.15.33:0/2995681739 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97540bae40 con 0x55df760ef7d0 2024-03-19T02:54:55.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.419+0000 7f976476c740 1 -- 172.21.15.33:0/2995681739 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df760ef7d0 msgr2=0x55df760f1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.419+0000 7f976476c740 1 --2- 172.21.15.33:0/2995681739 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df760ef7d0 0x55df760f1bc0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f97540b9510 tx=0x7f9754005d60 comp rx=0 tx=0).stop 2024-03-19T02:54:55.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.419+0000 7f976476c740 1 -- 172.21.15.33:0/2995681739 shutdown_connections 2024-03-19T02:54:55.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.419+0000 7f976476c740 1 --2- 172.21.15.33:0/2995681739 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df760ef7d0 0x55df760f1bc0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.419+0000 7f976476c740 1 --2- 172.21.15.33:0/2995681739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55df760e6bf0 0x55df760ef290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.419+0000 7f976476c740 1 --2- 172.21.15.33:0/2995681739 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df760e62d0 0x55df760e66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.419+0000 7f976476c740 1 -- 172.21.15.33:0/2995681739 >> 172.21.15.33:0/2995681739 conn(0x55df760e0a00 msgr2=0x55df760e0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:55.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.419+0000 7f976476c740 1 -- 172.21.15.33:0/2995681739 shutdown_connections 2024-03-19T02:54:55.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.419+0000 7f976476c740 1 -- 172.21.15.33:0/2995681739 wait complete. 2024-03-19T02:54:55.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.419+0000 7f976476c740 1 -- start start 2024-03-19T02:54:55.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.420+0000 7f976476c740 1 Processor -- start 2024-03-19T02:54:55.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.420+0000 7f976476c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df760e62d0 0x55df760ec840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.420+0000 7f976476c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df760e6bf0 0x55df760ecd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.420+0000 7f976476c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55df760ef7d0 0x55df760ed2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.420+0000 7f976476c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55df760f2380 con 0x55df760e62d0 2024-03-19T02:54:55.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.420+0000 7f976476c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55df76005310 con 0x55df760ef7d0 2024-03-19T02:54:55.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.420+0000 7f976476c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55df760f2500 con 0x55df760e6bf0 2024-03-19T02:54:55.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.420+0000 7f976476c740 10 client.0 _ll_register_callbacks cb 0x55df75ff92d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:54:55.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.420+0000 7f975b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df760e62d0 0x55df760ec840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.420+0000 7f975b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df760e62d0 0x55df760ec840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53396/0 (socket says 172.21.15.33:53396) 2024-03-19T02:54:55.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.420+0000 7f975b7fe640 1 -- 172.21.15.33:0/2795240270 learned_addr learned my addr 172.21.15.33:0/2795240270 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:55.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.420+0000 7f975bfff640 1 --2- 172.21.15.33:0/2795240270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55df760ef7d0 0x55df760ed2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.420+0000 7f975affd640 1 --2- 172.21.15.33:0/2795240270 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df760e6bf0 0x55df760ecd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.420+0000 7f975b7fe640 1 -- 172.21.15.33:0/2795240270 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df760e6bf0 msgr2=0x55df760ecd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:55.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.420+0000 7f975b7fe640 1 --2- 172.21.15.33:0/2795240270 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df760e6bf0 0x55df760ecd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.420+0000 7f975b7fe640 1 -- 172.21.15.33:0/2795240270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55df760ef7d0 msgr2=0x55df760ed2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:55.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.420+0000 7f975b7fe640 1 --2- 172.21.15.33:0/2795240270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55df760ef7d0 0x55df760ed2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.420+0000 7f975b7fe640 1 -- 172.21.15.33:0/2795240270 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df75d5d3f0 con 0x55df760e62d0 2024-03-19T02:54:55.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.420+0000 7f975bfff640 1 --2- 172.21.15.33:0/2795240270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55df760ef7d0 0x55df760ed2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:55.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.420+0000 7f975b7fe640 1 --2- 172.21.15.33:0/2795240270 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df760e62d0 0x55df760ec840 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f974c00b390 tx=0x7f974c00b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac73e57c27f187ed server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:55.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.420+0000 7f973affd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:54:55.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.421+0000 7f973affd640 1 -- 172.21.15.33:0/2795240270 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974c0197e0 con 0x55df760e62d0 2024-03-19T02:54:55.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.421+0000 7f973affd640 1 -- 172.21.15.33:0/2795240270 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f974c019980 con 0x55df760e62d0 2024-03-19T02:54:55.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.421+0000 7f976476c740 1 -- 172.21.15.33:0/2795240270 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97540bcb40 con 0x55df760e62d0 2024-03-19T02:54:55.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.421+0000 7f973affd640 1 -- 172.21.15.33:0/2795240270 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f974c029050 con 0x55df760e62d0 2024-03-19T02:54:55.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.421+0000 7f976476c740 10 client.15999 fetch_fsmap learned FSMap version 25 2024-03-19T02:54:55.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.421+0000 7f976476c740 1 -- 172.21.15.33:0/2795240270 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55df75e277d0 con 0x55df760e62d0 2024-03-19T02:54:55.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.421+0000 7f973affd640 1 -- 172.21.15.33:0/2795240270 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f97540bcb40 con 0x55df760e62d0 2024-03-19T02:54:55.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.421+0000 7f973affd640 1 -- 172.21.15.33:0/2795240270 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f974c00b040 con 0x55df760e62d0 2024-03-19T02:54:55.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.421+0000 7f976476c740 10 client.15999 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:54:55.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.421+0000 7f976476c740 10 client.15999 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:54:55.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.421+0000 7f976476c740 20 client.15999 populate_metadata read hostname 'smithi033' 2024-03-19T02:54:55.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.421+0000 7f976476c740 1 --2- 172.21.15.33:0/2795240270 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55df760ff9d0 0x55df76101e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.421+0000 7f976476c740 4 client.15999 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:54:55.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.421+0000 7f976476c740 1 -- 172.21.15.33:0/2795240270 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55df760a73e0 con 0x55df760ff9d0 2024-03-19T02:54:55.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.421+0000 7f975affd640 1 --2- 172.21.15.33:0/2795240270 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55df760ff9d0 0x55df76101e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.422+0000 7f975affd640 1 --2- 172.21.15.33:0/2795240270 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55df760ff9d0 0x55df76101e10 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=feace0e21510c058 server_cookie=e0a87439242e7c0e in_seq=0 out_seq=0 2024-03-19T02:54:55.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.422+0000 7f973affd640 10 client.15999 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:54:55.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.422+0000 7f973affd640 1 -- 172.21.15.33:0/2795240270 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55df760a73e0 con 0x55df760ff9d0 2024-03-19T02:54:55.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.422+0000 7f973affd640 10 client.15999 handle_command_reply: tid=0 2024-03-19T02:54:55.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.422+0000 7f976476c740 1 client.15999 shutdown 2024-03-19T02:54:55.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.423+0000 7f976476c740 1 -- 172.21.15.33:0/2795240270 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df760e62d0 msgr2=0x55df760ec840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.423+0000 7f976476c740 1 --2- 172.21.15.33:0/2795240270 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df760e62d0 0x55df760ec840 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f974c00b390 tx=0x7f974c00b790 comp rx=0 tx=0).stop 2024-03-19T02:54:55.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.423+0000 7f976476c740 1 -- 172.21.15.33:0/2795240270 shutdown_connections 2024-03-19T02:54:55.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.423+0000 7f976476c740 1 --2- 172.21.15.33:0/2795240270 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55df760ff9d0 0x55df76101e10 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.423+0000 7f976476c740 1 --2- 172.21.15.33:0/2795240270 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df760e6bf0 0x55df760ecd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.423+0000 7f976476c740 1 --2- 172.21.15.33:0/2795240270 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55df760ef7d0 0x55df760ed2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.423+0000 7f976476c740 1 --2- 172.21.15.33:0/2795240270 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df760e62d0 0x55df760ec840 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.423+0000 7f973affd640 0 client.15999 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:54:55.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.423+0000 7f976476c740 1 -- 172.21.15.33:0/2795240270 >> 172.21.15.33:0/2795240270 conn(0x55df760e0a00 msgr2=0x55df760e2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:55.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.423+0000 7f976476c740 1 -- 172.21.15.33:0/2795240270 shutdown_connections 2024-03-19T02:54:55.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.423+0000 7f976476c740 1 -- 172.21.15.33:0/2795240270 wait complete. 2024-03-19T02:54:55.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.423+0000 7f976476c740 20 client.15999 trim_cache size 0 max 16384 2024-03-19T02:54:55.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.437+0000 7f976476c740 1 Processor -- start 2024-03-19T02:54:55.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.437+0000 7f976476c740 1 -- start start 2024-03-19T02:54:55.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.437+0000 7f976476c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df760e62d0 0x55df760fb050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.437+0000 7f976476c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df760e6bf0 0x55df760ed0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.437+0000 7f976476c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55df760ef7d0 0x55df761076e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.437+0000 7f976476c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f975c1b13b0 con 0x55df760e62d0 2024-03-19T02:54:55.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.437+0000 7f976476c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f975c1aec80 con 0x55df760ef7d0 2024-03-19T02:54:55.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.437+0000 7f976476c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f975c1acab0 con 0x55df760e6bf0 2024-03-19T02:54:55.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.437+0000 7f975b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df760e62d0 0x55df760fb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.437+0000 7f975b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df760e62d0 0x55df760fb050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53400/0 (socket says 172.21.15.33:53400) 2024-03-19T02:54:55.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.437+0000 7f975b7fe640 1 -- 172.21.15.33:0/3299918764 learned_addr learned my addr 172.21.15.33:0/3299918764 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:55.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.437+0000 7f975bfff640 1 --2- 172.21.15.33:0/3299918764 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55df760ef7d0 0x55df761076e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.437+0000 7f975affd640 1 --2- 172.21.15.33:0/3299918764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df760e6bf0 0x55df760ed0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.437+0000 7f975b7fe640 1 -- 172.21.15.33:0/3299918764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df760e6bf0 msgr2=0x55df760ed0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:55.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.437+0000 7f975b7fe640 1 --2- 172.21.15.33:0/3299918764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df760e6bf0 0x55df760ed0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.437+0000 7f975b7fe640 1 -- 172.21.15.33:0/3299918764 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55df760ef7d0 msgr2=0x55df761076e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:55.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.437+0000 7f975b7fe640 1 --2- 172.21.15.33:0/3299918764 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55df760ef7d0 0x55df761076e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.437+0000 7f975b7fe640 1 -- 172.21.15.33:0/3299918764 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df760dc2c0 con 0x55df760e62d0 2024-03-19T02:54:55.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.437+0000 7f975affd640 1 --2- 172.21.15.33:0/3299918764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df760e6bf0 0x55df760ed0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:54:55.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.437+0000 7f975b7fe640 1 --2- 172.21.15.33:0/3299918764 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df760e62d0 0x55df760fb050 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f974c000f80 tx=0x7f974c024640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ea1a7feb3b384ca server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:55.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.437+0000 7f97417fa640 1 -- 172.21.15.33:0/3299918764 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974c003b70 con 0x55df760e62d0 2024-03-19T02:54:55.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.437+0000 7f97417fa640 1 -- 172.21.15.33:0/3299918764 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f974c020070 con 0x55df760e62d0 2024-03-19T02:54:55.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.437+0000 7f97417fa640 1 -- 172.21.15.33:0/3299918764 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974c0248f0 con 0x55df760e62d0 2024-03-19T02:54:55.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.438+0000 7f976476c740 1 -- 172.21.15.33:0/3299918764 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df760e62d0 msgr2=0x55df760fb050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.438+0000 7f976476c740 1 --2- 172.21.15.33:0/3299918764 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df760e62d0 0x55df760fb050 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f974c000f80 tx=0x7f974c024640 comp rx=0 tx=0).stop 2024-03-19T02:54:55.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.438+0000 7f976476c740 1 -- 172.21.15.33:0/3299918764 shutdown_connections 2024-03-19T02:54:55.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.438+0000 7f976476c740 1 --2- 172.21.15.33:0/3299918764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df760e6bf0 0x55df760ed0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.438+0000 7f976476c740 1 --2- 172.21.15.33:0/3299918764 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55df760ef7d0 0x55df761076e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.438+0000 7f976476c740 1 --2- 172.21.15.33:0/3299918764 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df760e62d0 0x55df760fb050 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.438+0000 7f976476c740 1 -- 172.21.15.33:0/3299918764 >> 172.21.15.33:0/3299918764 conn(0x55df760e0a00 msgr2=0x55df760dfba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:55.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.438+0000 7f976476c740 1 -- 172.21.15.33:0/3299918764 shutdown_connections 2024-03-19T02:54:55.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.438+0000 7f976476c740 1 -- 172.21.15.33:0/3299918764 wait complete. 2024-03-19T02:54:55.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.438+0000 7f976476c740 1 -- start start 2024-03-19T02:54:55.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.439+0000 7f976476c740 1 Processor -- start 2024-03-19T02:54:55.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.439+0000 7f976476c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df760e62d0 0x55df760f78d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.439+0000 7f976476c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df760e6bf0 0x55df760f2c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.439+0000 7f976476c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55df760ef7d0 0x55df760f3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.439+0000 7f976476c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f975c1ad140 con 0x55df760e6bf0 2024-03-19T02:54:55.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.439+0000 7f976476c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f975c1ae5b0 con 0x55df760ef7d0 2024-03-19T02:54:55.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.439+0000 7f976476c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f975c1ad810 con 0x55df760e62d0 2024-03-19T02:54:55.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.439+0000 7f976476c740 10 client.0 _ll_register_callbacks cb 0x7f974c00fea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:54:55.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.439+0000 7f975affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df760e6bf0 0x55df760f2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.439+0000 7f975affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df760e6bf0 0x55df760f2c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53416/0 (socket says 172.21.15.33:53416) 2024-03-19T02:54:55.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.439+0000 7f975affd640 1 -- 172.21.15.33:0/1241239686 learned_addr learned my addr 172.21.15.33:0/1241239686 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:55.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.439+0000 7f975b7fe640 1 --2- 172.21.15.33:0/1241239686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df760e62d0 0x55df760f78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.439+0000 7f975bfff640 1 --2- 172.21.15.33:0/1241239686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55df760ef7d0 0x55df760f3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.439+0000 7f975affd640 1 -- 172.21.15.33:0/1241239686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df760e62d0 msgr2=0x55df760f78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:55.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.439+0000 7f975affd640 1 --2- 172.21.15.33:0/1241239686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df760e62d0 0x55df760f78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.439+0000 7f975affd640 1 -- 172.21.15.33:0/1241239686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55df760ef7d0 msgr2=0x55df760f3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:55.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.439+0000 7f975affd640 1 --2- 172.21.15.33:0/1241239686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55df760ef7d0 0x55df760f3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.439+0000 7f975affd640 1 -- 172.21.15.33:0/1241239686 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df760f3850 con 0x55df760e6bf0 2024-03-19T02:54:55.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.439+0000 7f975affd640 1 --2- 172.21.15.33:0/1241239686 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df760e6bf0 0x55df760f2c20 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x55df760ec0a0 tx=0x7f975001b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bc8b9d59f205324 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:55.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.439+0000 7f973affd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:54:55.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.439+0000 7f973affd640 1 -- 172.21.15.33:0/1241239686 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975002b070 con 0x55df760e6bf0 2024-03-19T02:54:55.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.440+0000 7f973affd640 1 -- 172.21.15.33:0/1241239686 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f975002b210 con 0x55df760e6bf0 2024-03-19T02:54:55.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.440+0000 7f976476c740 1 -- 172.21.15.33:0/1241239686 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f974c020070 con 0x55df760e6bf0 2024-03-19T02:54:55.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.440+0000 7f973affd640 1 -- 172.21.15.33:0/1241239686 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f975003a050 con 0x55df760e6bf0 2024-03-19T02:54:55.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.440+0000 7f976476c740 10 client.16005 fetch_fsmap learned FSMap version 25 2024-03-19T02:54:55.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.440+0000 7f976476c740 1 -- 172.21.15.33:0/1241239686 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55df76165c90 con 0x55df760e6bf0 2024-03-19T02:54:55.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.440+0000 7f973affd640 1 -- 172.21.15.33:0/1241239686 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f974c020070 con 0x55df760e6bf0 2024-03-19T02:54:55.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.440+0000 7f973affd640 1 -- 172.21.15.33:0/1241239686 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f9750029070 con 0x55df760e6bf0 2024-03-19T02:54:55.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.440+0000 7f976476c740 10 client.16005 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:54:55.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.440+0000 7f976476c740 10 client.16005 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:54:55.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.440+0000 7f976476c740 20 client.16005 populate_metadata read hostname 'smithi033' 2024-03-19T02:54:55.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.440+0000 7f976476c740 1 --2- 172.21.15.33:0/1241239686 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55df761663e0 0x55df76168800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:55.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.440+0000 7f976476c740 4 client.16005 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:54:55.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.440+0000 7f976476c740 1 -- 172.21.15.33:0/1241239686 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55df760569f0 con 0x55df761663e0 2024-03-19T02:54:55.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.440+0000 7f975b7fe640 1 --2- 172.21.15.33:0/1241239686 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55df761663e0 0x55df76168800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:55.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.441+0000 7f975b7fe640 1 --2- 172.21.15.33:0/1241239686 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55df761663e0 0x55df76168800 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb9b642ebf31d037 server_cookie=8f52be5b45563782 in_seq=0 out_seq=0 2024-03-19T02:54:55.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.441+0000 7f973affd640 10 client.16005 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:54:55.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.441+0000 7f973affd640 1 -- 172.21.15.33:0/1241239686 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55df760569f0 con 0x55df761663e0 2024-03-19T02:54:55.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.441+0000 7f973affd640 10 client.16005 handle_command_reply: tid=0 2024-03-19T02:54:55.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.441+0000 7f976476c740 1 client.16005 shutdown 2024-03-19T02:54:55.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.441+0000 7f976476c740 1 -- 172.21.15.33:0/1241239686 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df760e6bf0 msgr2=0x55df760f2c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.441+0000 7f976476c740 1 --2- 172.21.15.33:0/1241239686 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df760e6bf0 0x55df760f2c20 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x55df760ec0a0 tx=0x7f975001b000 comp rx=0 tx=0).stop 2024-03-19T02:54:55.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.441+0000 7f976476c740 1 -- 172.21.15.33:0/1241239686 shutdown_connections 2024-03-19T02:54:55.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.441+0000 7f976476c740 1 --2- 172.21.15.33:0/1241239686 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55df761663e0 0x55df76168800 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.441+0000 7f976476c740 1 --2- 172.21.15.33:0/1241239686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55df760ef7d0 0x55df760f3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.441+0000 7f976476c740 1 --2- 172.21.15.33:0/1241239686 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55df760e6bf0 0x55df760f2c20 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.441+0000 7f976476c740 1 --2- 172.21.15.33:0/1241239686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55df760e62d0 0x55df760f78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.441+0000 7f973affd640 0 client.16005 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:54:55.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.441+0000 7f976476c740 1 -- 172.21.15.33:0/1241239686 >> 172.21.15.33:0/1241239686 conn(0x55df760e0a00 msgr2=0x55df760e1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:55.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.441+0000 7f976476c740 1 -- 172.21.15.33:0/1241239686 shutdown_connections 2024-03-19T02:54:55.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.441+0000 7f976476c740 1 -- 172.21.15.33:0/1241239686 wait complete. 2024-03-19T02:54:55.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.441+0000 7f976476c740 20 client.16005 trim_cache size 0 max 16384 2024-03-19T02:54:55.450 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:54:55.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.442+0000 7f9761b7e640 1 -- 172.21.15.33:0/1511890894 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f97440a2a90 msgr2=0x7f97440a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.442+0000 7f9761b7e640 1 --2- 172.21.15.33:0/1511890894 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f97440a2a90 0x7f97440a4f50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f975c0694f0 tx=0x7f97500098b0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.442+0000 7f9761b7e640 1 -- 172.21.15.33:0/1511890894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f975c1066b0 msgr2=0x7f975c1a3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:55.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.442+0000 7f9761b7e640 1 --2- 172.21.15.33:0/1511890894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f975c1066b0 0x7f975c1a3db0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f975402c180 tx=0x7f9754035f40 comp rx=0 tx=0).stop 2024-03-19T02:54:55.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.442+0000 7f9761b7e640 1 -- 172.21.15.33:0/1511890894 shutdown_connections 2024-03-19T02:54:55.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.442+0000 7f9761b7e640 1 --2- 172.21.15.33:0/1511890894 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f97440a2a90 0x7f97440a4f50 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.442+0000 7f9761b7e640 1 --2- 172.21.15.33:0/1511890894 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f975c101060 0x7f975c199270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.442+0000 7f9761b7e640 1 --2- 172.21.15.33:0/1511890894 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f975c1066b0 0x7f975c1a3db0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.442+0000 7f9761b7e640 1 --2- 172.21.15.33:0/1511890894 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f975c1006a0 0x7f975c198d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:55.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.442+0000 7f9761b7e640 1 -- 172.21.15.33:0/1511890894 >> 172.21.15.33:0/1511890894 conn(0x7f975c0778d0 msgr2=0x7f975c102eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:55.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.442+0000 7f9761b7e640 1 -- 172.21.15.33:0/1511890894 shutdown_connections 2024-03-19T02:54:55.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:55.442+0000 7f9761b7e640 1 -- 172.21.15.33:0/1511890894 wait complete. 2024-03-19T02:54:55.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:55 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/696516445' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:54:55.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:55 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/696516445' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:54:56.454 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:54:56.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:54:56.455 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:54:56.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:56 smithi096 ceph-mon[35946]: pgmap v47: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-19T02:54:56.483 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:56 smithi033 ceph-mon[31973]: pgmap v47: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-19T02:54:56.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:56 smithi155 ceph-mon[35804]: pgmap v47: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-19T02:54:56.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.573+0000 7fc513590640 1 -- 172.21.15.33:0/4263537825 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc50c106750 msgr2=0x7fc50c106b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:56.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.573+0000 7fc513590640 1 --2- 172.21.15.33:0/4263537825 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc50c106750 0x7fc50c106b30 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fc508009930 tx=0x7fc50802bc30 comp rx=0 tx=0).stop 2024-03-19T02:54:56.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.573+0000 7fc513590640 1 -- 172.21.15.33:0/4263537825 shutdown_connections 2024-03-19T02:54:56.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.573+0000 7fc513590640 1 --2- 172.21.15.33:0/4263537825 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc50c101100 0x7fc50c10d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.573+0000 7fc513590640 1 --2- 172.21.15.33:0/4263537825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc50c100740 0x7fc50c100bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.573+0000 7fc513590640 1 --2- 172.21.15.33:0/4263537825 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc50c106750 0x7fc50c106b30 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.573+0000 7fc513590640 1 -- 172.21.15.33:0/4263537825 >> 172.21.15.33:0/4263537825 conn(0x7fc50c077880 msgr2=0x7fc50c077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:56.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.573+0000 7fc513590640 1 -- 172.21.15.33:0/4263537825 shutdown_connections 2024-03-19T02:54:56.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.573+0000 7fc513590640 1 -- 172.21.15.33:0/4263537825 wait complete. 2024-03-19T02:54:56.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.574+0000 7fc513590640 1 Processor -- start 2024-03-19T02:54:56.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.574+0000 7fc513590640 1 -- start start 2024-03-19T02:54:56.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.574+0000 7fc513590640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc50c100740 0x7fc50c198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.574+0000 7fc513590640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc50c101100 0x7fc50c198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.574+0000 7fc513590640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc50c106750 0x7fc50c19c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.574+0000 7fc513590640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c110a10 con 0x7fc50c106750 2024-03-19T02:54:56.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.574+0000 7fc513590640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c110890 con 0x7fc50c100740 2024-03-19T02:54:56.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.574+0000 7fc513590640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c110b90 con 0x7fc50c101100 2024-03-19T02:54:56.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.574+0000 7fc51132c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc50c100740 0x7fc50c198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.574+0000 7fc511b2d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc50c106750 0x7fc50c19c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.574+0000 7fc51132c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc50c100740 0x7fc50c198100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:50994/0 (socket says 172.21.15.33:50994) 2024-03-19T02:54:56.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.574+0000 7fc51132c640 1 -- 172.21.15.33:0/1928800943 learned_addr learned my addr 172.21.15.33:0/1928800943 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:56.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.574+0000 7fc510b2b640 1 --2- 172.21.15.33:0/1928800943 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc50c101100 0x7fc50c198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.575+0000 7fc511b2d640 1 -- 172.21.15.33:0/1928800943 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc50c101100 msgr2=0x7fc50c198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:56.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.575+0000 7fc511b2d640 1 --2- 172.21.15.33:0/1928800943 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc50c101100 0x7fc50c198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.575+0000 7fc511b2d640 1 -- 172.21.15.33:0/1928800943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc50c100740 msgr2=0x7fc50c198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:56.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.575+0000 7fc511b2d640 1 --2- 172.21.15.33:0/1928800943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc50c100740 0x7fc50c198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.575+0000 7fc511b2d640 1 -- 172.21.15.33:0/1928800943 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc50c19d0a0 con 0x7fc50c106750 2024-03-19T02:54:56.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.575+0000 7fc51132c640 1 --2- 172.21.15.33:0/1928800943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc50c100740 0x7fc50c198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:56.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.575+0000 7fc510b2b640 1 --2- 172.21.15.33:0/1928800943 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc50c101100 0x7fc50c198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:56.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.575+0000 7fc511b2d640 1 --2- 172.21.15.33:0/1928800943 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc50c106750 0x7fc50c19c970 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc504009fd0 tx=0x7fc50400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:56.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.575+0000 7fc5027fc640 1 -- 172.21.15.33:0/1928800943 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc50400ce20 con 0x7fc50c106750 2024-03-19T02:54:56.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.575+0000 7fc5027fc640 1 -- 172.21.15.33:0/1928800943 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc504005020 con 0x7fc50c106750 2024-03-19T02:54:56.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.575+0000 7fc513590640 1 -- 172.21.15.33:0/1928800943 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc50c19d390 con 0x7fc50c106750 2024-03-19T02:54:56.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.575+0000 7fc5027fc640 1 -- 172.21.15.33:0/1928800943 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc504010690 con 0x7fc50c106750 2024-03-19T02:54:56.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.575+0000 7fc513590640 1 -- 172.21.15.33:0/1928800943 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc50c1a4c20 con 0x7fc50c106750 2024-03-19T02:54:56.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.577+0000 7fc5027fc640 1 -- 172.21.15.33:0/1928800943 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7fc5040051d0 con 0x7fc50c106750 2024-03-19T02:54:56.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.578+0000 7fc516176740 1 Processor -- start 2024-03-19T02:54:56.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.578+0000 7fc516176740 1 -- start start 2024-03-19T02:54:56.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.578+0000 7fc5027fc640 1 --2- 172.21.15.33:0/1928800943 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fc4f80a2a90 0x7fc4f80a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.578+0000 7fc516176740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55895b9bb0b0 0x55895b9bb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.578+0000 7fc516176740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55895b9bb9d0 0x55895b9c8750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.578+0000 7fc516176740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895b9c8c90 0x55895b9cb080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.578+0000 7fc516176740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55895b91f160 con 0x55895b9c8c90 2024-03-19T02:54:56.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.578+0000 7fc516176740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55895b8c7e10 con 0x55895b9bb0b0 2024-03-19T02:54:56.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.578+0000 7fc516176740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55895b91b200 con 0x55895b9bb9d0 2024-03-19T02:54:56.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.578+0000 7fc5027fc640 1 -- 172.21.15.33:0/1928800943 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fc5040cfaa0 con 0x7fc50c106750 2024-03-19T02:54:56.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.578+0000 7fc510b2b640 1 --2- 172.21.15.33:0/1928800943 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fc4f80a2a90 0x7fc4f80a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.578+0000 7fc51132c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55895b9bb0b0 0x55895b9bb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.578+0000 7fc511b2d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895b9c8c90 0x55895b9cb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.578+0000 7fc51132c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55895b9bb0b0 0x55895b9bb490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:50998/0 (socket says 172.21.15.33:50998) 2024-03-19T02:54:56.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.578+0000 7fc51132c640 1 -- 172.21.15.33:0/4159811043 learned_addr learned my addr 172.21.15.33:0/4159811043 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:56.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.578+0000 7fc510b2b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55895b9bb9d0 0x55895b9c8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.578+0000 7fc51132c640 1 -- 172.21.15.33:0/4159811043 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55895b9bb9d0 msgr2=0x55895b9c8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:56.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.578+0000 7fc51132c640 1 --2- 172.21.15.33:0/4159811043 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55895b9bb9d0 0x55895b9c8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.578+0000 7fc51132c640 1 -- 172.21.15.33:0/4159811043 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895b9c8c90 msgr2=0x55895b9cb080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:56.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.578+0000 7fc51132c640 1 --2- 172.21.15.33:0/4159811043 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895b9c8c90 0x55895b9cb080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.578+0000 7fc51132c640 1 -- 172.21.15.33:0/4159811043 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55895b6fb7e0 con 0x55895b9bb0b0 2024-03-19T02:54:56.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.578+0000 7fc511b2d640 1 --2- 172.21.15.33:0/4159811043 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895b9c8c90 0x55895b9cb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:56.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.579+0000 7fc510b2b640 1 --2- 172.21.15.33:0/1928800943 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fc4f80a2a90 0x7fc4f80a4f50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc50c078120 tx=0x7fc4fc009ed0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:56.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.579+0000 7fc51132c640 1 --2- 172.21.15.33:0/4159811043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55895b9bb0b0 0x55895b9bb490 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc508009430 tx=0x7fc508009670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=348f1f30ee55d060 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:56.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.579+0000 7fc4f77fe640 1 -- 172.21.15.33:0/4159811043 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc508045070 con 0x55895b9bb0b0 2024-03-19T02:54:56.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.579+0000 7fc4f77fe640 1 -- 172.21.15.33:0/4159811043 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc50802c7e0 con 0x55895b9bb0b0 2024-03-19T02:54:56.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.579+0000 7fc4f77fe640 1 -- 172.21.15.33:0/4159811043 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc508003a10 con 0x55895b9bb0b0 2024-03-19T02:54:56.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.579+0000 7fc516176740 1 -- 172.21.15.33:0/4159811043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55895b9bb0b0 msgr2=0x55895b9bb490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:56.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.580+0000 7fc516176740 1 --2- 172.21.15.33:0/4159811043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55895b9bb0b0 0x55895b9bb490 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc508009430 tx=0x7fc508009670 comp rx=0 tx=0).stop 2024-03-19T02:54:56.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.580+0000 7fc516176740 1 -- 172.21.15.33:0/4159811043 shutdown_connections 2024-03-19T02:54:56.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.580+0000 7fc516176740 1 --2- 172.21.15.33:0/4159811043 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55895b9bb9d0 0x55895b9c8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.580+0000 7fc516176740 1 --2- 172.21.15.33:0/4159811043 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895b9c8c90 0x55895b9cb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.580+0000 7fc516176740 1 --2- 172.21.15.33:0/4159811043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55895b9bb0b0 0x55895b9bb490 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.580+0000 7fc516176740 1 -- 172.21.15.33:0/4159811043 >> 172.21.15.33:0/4159811043 conn(0x55895b9b5ab0 msgr2=0x55895b9b5e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:56.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.580+0000 7fc516176740 1 -- 172.21.15.33:0/4159811043 shutdown_connections 2024-03-19T02:54:56.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.580+0000 7fc516176740 1 -- 172.21.15.33:0/4159811043 wait complete. 2024-03-19T02:54:56.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.580+0000 7fc516176740 1 -- start start 2024-03-19T02:54:56.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.580+0000 7fc516176740 1 Processor -- start 2024-03-19T02:54:56.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.580+0000 7fc516176740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55895b9bb0b0 0x55895b9c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.580+0000 7fc516176740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55895b9bb9d0 0x55895b9c60a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.580+0000 7fc516176740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895b9c8c90 0x55895b9c65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.580+0000 7fc516176740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55895b9cb9f0 con 0x55895b9c8c90 2024-03-19T02:54:56.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.580+0000 7fc516176740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55895b8db1a0 con 0x55895b9bb9d0 2024-03-19T02:54:56.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.580+0000 7fc516176740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55895b9cbb70 con 0x55895b9bb0b0 2024-03-19T02:54:56.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.580+0000 7fc516176740 10 client.0 _ll_register_callbacks cb 0x55895b632c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:54:56.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.581+0000 7fc511b2d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895b9c8c90 0x55895b9c65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.581+0000 7fc511b2d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895b9c8c90 0x55895b9c65e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53444/0 (socket says 172.21.15.33:53444) 2024-03-19T02:54:56.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.581+0000 7fc51132c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55895b9bb0b0 0x55895b9c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.581+0000 7fc511b2d640 1 -- 172.21.15.33:0/2468868007 learned_addr learned my addr 172.21.15.33:0/2468868007 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:56.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.581+0000 7fc510b2b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55895b9bb9d0 0x55895b9c60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.581+0000 7fc51132c640 1 -- 172.21.15.33:0/2468868007 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55895b9bb9d0 msgr2=0x55895b9c60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:56.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.581+0000 7fc51132c640 1 --2- 172.21.15.33:0/2468868007 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55895b9bb9d0 0x55895b9c60a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.581+0000 7fc51132c640 1 -- 172.21.15.33:0/2468868007 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895b9c8c90 msgr2=0x55895b9c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:56.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.581+0000 7fc51132c640 1 --2- 172.21.15.33:0/2468868007 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895b9c8c90 0x55895b9c65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.581+0000 7fc51132c640 1 -- 172.21.15.33:0/2468868007 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55895b75bf70 con 0x55895b9bb0b0 2024-03-19T02:54:56.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.581+0000 7fc511b2d640 1 --2- 172.21.15.33:0/2468868007 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895b9c8c90 0x55895b9c65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:56.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.581+0000 7fc51132c640 1 --2- 172.21.15.33:0/2468868007 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55895b9bb0b0 0x55895b9c5b60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc508005940 tx=0x7fc508005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af0f1268d0c9ce1c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:56.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.581+0000 7fc4f4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-19T02:54:56.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.581+0000 7fc4f4ff9640 1 -- 172.21.15.33:0/2468868007 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc508045040 con 0x55895b9bb0b0 2024-03-19T02:54:56.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.581+0000 7fc516176740 1 -- 172.21.15.33:0/2468868007 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc50802c7e0 con 0x55895b9bb0b0 2024-03-19T02:54:56.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.581+0000 7fc4f4ff9640 1 -- 172.21.15.33:0/2468868007 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fc50803fc00 con 0x55895b9bb0b0 2024-03-19T02:54:56.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.581+0000 7fc4f4ff9640 1 -- 172.21.15.33:0/2468868007 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc50805a050 con 0x55895b9bb0b0 2024-03-19T02:54:56.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.581+0000 7fc516176740 10 client.25729 fetch_fsmap learned FSMap version 25 2024-03-19T02:54:56.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.581+0000 7fc516176740 1 -- 172.21.15.33:0/2468868007 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55895b5f3170 con 0x55895b9bb0b0 2024-03-19T02:54:56.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.582+0000 7fc4f4ff9640 1 -- 172.21.15.33:0/2468868007 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc50802c7e0 con 0x55895b9bb0b0 2024-03-19T02:54:56.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.582+0000 7fc4f4ff9640 1 -- 172.21.15.33:0/2468868007 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc508034440 con 0x55895b9bb0b0 2024-03-19T02:54:56.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.582+0000 7fc516176740 10 client.25729 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:54:56.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.582+0000 7fc516176740 10 client.25729 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:54:56.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.582+0000 7fc516176740 20 client.25729 populate_metadata read hostname 'smithi033' 2024-03-19T02:54:56.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.582+0000 7fc516176740 1 --2- 172.21.15.33:0/2468868007 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55895b9d8d40 0x55895b9db130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.582+0000 7fc516176740 4 client.25729 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:54:56.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.582+0000 7fc516176740 1 -- 172.21.15.33:0/2468868007 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55895b92c6a0 con 0x55895b9d8d40 2024-03-19T02:54:56.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.582+0000 7fc510b2b640 1 --2- 172.21.15.33:0/2468868007 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55895b9d8d40 0x55895b9db130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.583+0000 7fc4f4ff9640 10 client.25729 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:54:56.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.582+0000 7fc510b2b640 1 --2- 172.21.15.33:0/2468868007 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55895b9d8d40 0x55895b9db130 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17c8723e72675b9d server_cookie=773feb21adce2065 in_seq=0 out_seq=0 2024-03-19T02:54:56.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.583+0000 7fc4f4ff9640 1 -- 172.21.15.33:0/2468868007 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55895b92c6a0 con 0x55895b9d8d40 2024-03-19T02:54:56.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.583+0000 7fc4f4ff9640 10 client.25729 handle_command_reply: tid=0 2024-03-19T02:54:56.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.583+0000 7fc516176740 1 client.25729 shutdown 2024-03-19T02:54:56.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.583+0000 7fc516176740 1 -- 172.21.15.33:0/2468868007 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55895b9bb0b0 msgr2=0x55895b9c5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:56.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.583+0000 7fc516176740 1 --2- 172.21.15.33:0/2468868007 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55895b9bb0b0 0x55895b9c5b60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc508005940 tx=0x7fc508005a20 comp rx=0 tx=0).stop 2024-03-19T02:54:56.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.583+0000 7fc516176740 1 -- 172.21.15.33:0/2468868007 shutdown_connections 2024-03-19T02:54:56.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.583+0000 7fc516176740 1 --2- 172.21.15.33:0/2468868007 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55895b9d8d40 0x55895b9db130 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.583+0000 7fc516176740 1 --2- 172.21.15.33:0/2468868007 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895b9c8c90 0x55895b9c65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.583+0000 7fc516176740 1 --2- 172.21.15.33:0/2468868007 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55895b9bb9d0 0x55895b9c60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.583+0000 7fc4f4ff9640 0 client.25729 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:54:56.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.583+0000 7fc516176740 1 --2- 172.21.15.33:0/2468868007 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55895b9bb0b0 0x55895b9c5b60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.583+0000 7fc516176740 1 -- 172.21.15.33:0/2468868007 >> 172.21.15.33:0/2468868007 conn(0x55895b9b5ab0 msgr2=0x55895b9b7800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:56.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.583+0000 7fc516176740 1 -- 172.21.15.33:0/2468868007 shutdown_connections 2024-03-19T02:54:56.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.583+0000 7fc516176740 1 -- 172.21.15.33:0/2468868007 wait complete. 2024-03-19T02:54:56.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.584+0000 7fc516176740 20 client.25729 trim_cache size 0 max 16384 2024-03-19T02:54:56.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.601+0000 7fc516176740 1 Processor -- start 2024-03-19T02:54:56.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.601+0000 7fc516176740 1 -- start start 2024-03-19T02:54:56.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.601+0000 7fc516176740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55895b9bb0b0 0x55895b9d4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.601+0000 7fc516176740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55895b9bb9d0 0x55895b9c7990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.601+0000 7fc516176740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895b9c8c90 0x55895b9e07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.601+0000 7fc516176740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c1af230 con 0x55895b9c8c90 2024-03-19T02:54:56.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.601+0000 7fc516176740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c1acb00 con 0x55895b9bb9d0 2024-03-19T02:54:56.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.601+0000 7fc516176740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c1aa930 con 0x55895b9bb0b0 2024-03-19T02:54:56.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.601+0000 7fc511b2d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895b9c8c90 0x55895b9e07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.601+0000 7fc511b2d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895b9c8c90 0x55895b9e07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53460/0 (socket says 172.21.15.33:53460) 2024-03-19T02:54:56.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.601+0000 7fc511b2d640 1 -- 172.21.15.33:0/4070141661 learned_addr learned my addr 172.21.15.33:0/4070141661 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:56.607 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.601+0000 7fc51132c640 1 --2- 172.21.15.33:0/4070141661 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55895b9bb0b0 0x55895b9d4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.601+0000 7fc510b2b640 1 --2- 172.21.15.33:0/4070141661 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55895b9bb9d0 0x55895b9c7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.602+0000 7fc511b2d640 1 -- 172.21.15.33:0/4070141661 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55895b9bb0b0 msgr2=0x55895b9d4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:56.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.602+0000 7fc511b2d640 1 --2- 172.21.15.33:0/4070141661 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55895b9bb0b0 0x55895b9d4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.602+0000 7fc511b2d640 1 -- 172.21.15.33:0/4070141661 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55895b9bb9d0 msgr2=0x55895b9c7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:56.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.602+0000 7fc511b2d640 1 --2- 172.21.15.33:0/4070141661 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55895b9bb9d0 0x55895b9c7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.602+0000 7fc511b2d640 1 -- 172.21.15.33:0/4070141661 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55895b9b1380 con 0x55895b9c8c90 2024-03-19T02:54:56.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.602+0000 7fc511b2d640 1 --2- 172.21.15.33:0/4070141661 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895b9c8c90 0x55895b9e07f0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fc504097fa0 tx=0x7fc504016cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c86f94a30953be34 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:56.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.602+0000 7fc4f6ffd640 1 -- 172.21.15.33:0/4070141661 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc50409bb40 con 0x55895b9c8c90 2024-03-19T02:54:56.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.602+0000 7fc4f6ffd640 1 -- 172.21.15.33:0/4070141661 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc5040b4070 con 0x55895b9c8c90 2024-03-19T02:54:56.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.602+0000 7fc4f6ffd640 1 -- 172.21.15.33:0/4070141661 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5040a40b0 con 0x55895b9c8c90 2024-03-19T02:54:56.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.603+0000 7fc516176740 1 -- 172.21.15.33:0/4070141661 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895b9c8c90 msgr2=0x55895b9e07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:56.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.603+0000 7fc516176740 1 --2- 172.21.15.33:0/4070141661 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895b9c8c90 0x55895b9e07f0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fc504097fa0 tx=0x7fc504016cd0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.603+0000 7fc516176740 1 -- 172.21.15.33:0/4070141661 shutdown_connections 2024-03-19T02:54:56.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.603+0000 7fc516176740 1 --2- 172.21.15.33:0/4070141661 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895b9c8c90 0x55895b9e07f0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.603+0000 7fc516176740 1 --2- 172.21.15.33:0/4070141661 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55895b9bb9d0 0x55895b9c7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.603+0000 7fc516176740 1 --2- 172.21.15.33:0/4070141661 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55895b9bb0b0 0x55895b9d4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.603+0000 7fc516176740 1 -- 172.21.15.33:0/4070141661 >> 172.21.15.33:0/4070141661 conn(0x55895b9b5ab0 msgr2=0x55895b9c7280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:56.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.603+0000 7fc516176740 1 -- 172.21.15.33:0/4070141661 shutdown_connections 2024-03-19T02:54:56.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.603+0000 7fc516176740 1 -- 172.21.15.33:0/4070141661 wait complete. 2024-03-19T02:54:56.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.603+0000 7fc516176740 1 -- start start 2024-03-19T02:54:56.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.603+0000 7fc516176740 1 Processor -- start 2024-03-19T02:54:56.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.603+0000 7fc516176740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55895b9bb0b0 0x55895b9c2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.603+0000 7fc516176740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55895b9bb9d0 0x55895b9c32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.603+0000 7fc516176740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895b9c8c90 0x55895ba3de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.603+0000 7fc516176740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c1aafc0 con 0x55895b9c8c90 2024-03-19T02:54:56.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.603+0000 7fc516176740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c1ac430 con 0x55895b9bb9d0 2024-03-19T02:54:56.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.603+0000 7fc516176740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c1ab690 con 0x55895b9bb0b0 2024-03-19T02:54:56.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.604+0000 7fc516176740 10 client.0 _ll_register_callbacks cb 0x7fc504016ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:54:56.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.604+0000 7fc511b2d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895b9c8c90 0x55895ba3de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.604+0000 7fc510b2b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55895b9bb9d0 0x55895b9c32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.604+0000 7fc51132c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55895b9bb0b0 0x55895b9c2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.604+0000 7fc510b2b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55895b9bb9d0 0x55895b9c32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:51024/0 (socket says 172.21.15.33:51024) 2024-03-19T02:54:56.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.604+0000 7fc511b2d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895b9c8c90 0x55895ba3de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53466/0 (socket says 172.21.15.33:53466) 2024-03-19T02:54:56.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.604+0000 7fc510b2b640 1 -- 172.21.15.33:0/1729267378 learned_addr learned my addr 172.21.15.33:0/1729267378 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:56.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.604+0000 7fc51132c640 1 --2- 172.21.15.33:0/1729267378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55895b9bb0b0 0x55895b9c2d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:52282/0 (socket says 172.21.15.33:52282) 2024-03-19T02:54:56.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.604+0000 7fc51132c640 1 -- 172.21.15.33:0/1729267378 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55895b9bb9d0 msgr2=0x55895b9c32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:56.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.604+0000 7fc51132c640 1 --2- 172.21.15.33:0/1729267378 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55895b9bb9d0 0x55895b9c32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.604+0000 7fc51132c640 1 -- 172.21.15.33:0/1729267378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895b9c8c90 msgr2=0x55895ba3de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:56.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.604+0000 7fc51132c640 1 --2- 172.21.15.33:0/1729267378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895b9c8c90 0x55895ba3de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.604+0000 7fc51132c640 1 -- 172.21.15.33:0/1729267378 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55895ba3e360 con 0x55895b9bb0b0 2024-03-19T02:54:56.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.604+0000 7fc511b2d640 1 --2- 172.21.15.33:0/1729267378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895b9c8c90 0x55895ba3de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:56.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.604+0000 7fc51132c640 1 --2- 172.21.15.33:0/1729267378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55895b9bb0b0 0x55895b9c2d70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc508034810 tx=0x7fc5080035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27b58d27e1b6f198 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:56.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.604+0000 7fc4f4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-19T02:54:56.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.604+0000 7fc4f4ff9640 1 -- 172.21.15.33:0/1729267378 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc508045040 con 0x55895b9bb0b0 2024-03-19T02:54:56.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.604+0000 7fc4f4ff9640 1 -- 172.21.15.33:0/1729267378 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fc50803fa00 con 0x55895b9bb0b0 2024-03-19T02:54:56.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.604+0000 7fc516176740 1 -- 172.21.15.33:0/1729267378 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5040b4070 con 0x55895b9bb0b0 2024-03-19T02:54:56.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.604+0000 7fc4f4ff9640 1 -- 172.21.15.33:0/1729267378 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc50806c050 con 0x55895b9bb0b0 2024-03-19T02:54:56.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.604+0000 7fc516176740 10 client.25735 fetch_fsmap learned FSMap version 25 2024-03-19T02:54:56.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.604+0000 7fc516176740 1 -- 172.21.15.33:0/1729267378 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55895ba3f5a0 con 0x55895b9bb0b0 2024-03-19T02:54:56.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.605+0000 7fc4f4ff9640 1 -- 172.21.15.33:0/1729267378 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc5040b4070 con 0x55895b9bb0b0 2024-03-19T02:54:56.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.605+0000 7fc4f4ff9640 1 -- 172.21.15.33:0/1729267378 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc50803fcd0 con 0x55895b9bb0b0 2024-03-19T02:54:56.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.605+0000 7fc516176740 10 client.25735 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:54:56.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.605+0000 7fc516176740 10 client.25735 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:54:56.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.605+0000 7fc516176740 20 client.25735 populate_metadata read hostname 'smithi033' 2024-03-19T02:54:56.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.605+0000 7fc516176740 1 --2- 172.21.15.33:0/1729267378 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55895ba3fc50 0x55895ba42070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.605+0000 7fc516176740 4 client.25735 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:54:56.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.605+0000 7fc516176740 1 -- 172.21.15.33:0/1729267378 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55895b9269f0 con 0x55895ba3fc50 2024-03-19T02:54:56.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.605+0000 7fc510b2b640 1 --2- 172.21.15.33:0/1729267378 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55895ba3fc50 0x55895ba42070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.605+0000 7fc510b2b640 1 --2- 172.21.15.33:0/1729267378 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55895ba3fc50 0x55895ba42070 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7264538ce7c30b57 server_cookie=e85d66af768f3db7 in_seq=0 out_seq=0 2024-03-19T02:54:56.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.605+0000 7fc4f4ff9640 10 client.25735 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:54:56.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.606+0000 7fc4f4ff9640 1 -- 172.21.15.33:0/1729267378 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55895b9269f0 con 0x55895ba3fc50 2024-03-19T02:54:56.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.606+0000 7fc4f4ff9640 10 client.25735 handle_command_reply: tid=0 2024-03-19T02:54:56.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.606+0000 7fc516176740 1 client.25735 shutdown 2024-03-19T02:54:56.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.606+0000 7fc516176740 1 -- 172.21.15.33:0/1729267378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55895b9bb0b0 msgr2=0x55895b9c2d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:56.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.606+0000 7fc516176740 1 --2- 172.21.15.33:0/1729267378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55895b9bb0b0 0x55895b9c2d70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc508034810 tx=0x7fc5080035f0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.606+0000 7fc516176740 1 -- 172.21.15.33:0/1729267378 shutdown_connections 2024-03-19T02:54:56.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.606+0000 7fc516176740 1 --2- 172.21.15.33:0/1729267378 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55895ba3fc50 0x55895ba42070 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.606+0000 7fc516176740 1 --2- 172.21.15.33:0/1729267378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55895b9c8c90 0x55895ba3de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.606+0000 7fc516176740 1 --2- 172.21.15.33:0/1729267378 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55895b9bb9d0 0x55895b9c32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.606+0000 7fc516176740 1 --2- 172.21.15.33:0/1729267378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55895b9bb0b0 0x55895b9c2d70 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.606+0000 7fc4f4ff9640 0 client.25735 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:54:56.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.606+0000 7fc516176740 1 -- 172.21.15.33:0/1729267378 >> 172.21.15.33:0/1729267378 conn(0x55895b9b5ab0 msgr2=0x55895b9c7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:56.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.607+0000 7fc516176740 1 -- 172.21.15.33:0/1729267378 shutdown_connections 2024-03-19T02:54:56.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.607+0000 7fc516176740 1 -- 172.21.15.33:0/1729267378 wait complete. 2024-03-19T02:54:56.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.607+0000 7fc516176740 20 client.25735 trim_cache size 0 max 16384 2024-03-19T02:54:56.614 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:54:56.614 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-03-19T02:54:56.614 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "685d94f0-8322-420c-baa2-c8e0f5a4a289", 2024-03-19T02:54:56.614 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-03-19T02:54:56.614 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:54:56.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.607+0000 7fc513590640 1 -- 172.21.15.33:0/1928800943 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fc4f80a2a90 msgr2=0x7fc4f80a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:56.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.607+0000 7fc513590640 1 --2- 172.21.15.33:0/1928800943 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fc4f80a2a90 0x7fc4f80a4f50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc50c078120 tx=0x7fc4fc009ed0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.607+0000 7fc513590640 1 -- 172.21.15.33:0/1928800943 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc50c106750 msgr2=0x7fc50c19c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:56.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.607+0000 7fc513590640 1 --2- 172.21.15.33:0/1928800943 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc50c106750 0x7fc50c19c970 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc504009fd0 tx=0x7fc50400ef10 comp rx=0 tx=0).stop 2024-03-19T02:54:56.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.607+0000 7fc513590640 1 -- 172.21.15.33:0/1928800943 shutdown_connections 2024-03-19T02:54:56.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.607+0000 7fc513590640 1 --2- 172.21.15.33:0/1928800943 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fc4f80a2a90 0x7fc4f80a4f50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.607+0000 7fc513590640 1 --2- 172.21.15.33:0/1928800943 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc50c101100 0x7fc50c198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.607+0000 7fc513590640 1 --2- 172.21.15.33:0/1928800943 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc50c106750 0x7fc50c19c970 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.607+0000 7fc513590640 1 --2- 172.21.15.33:0/1928800943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc50c100740 0x7fc50c198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.607+0000 7fc513590640 1 -- 172.21.15.33:0/1928800943 >> 172.21.15.33:0/1928800943 conn(0x7fc50c077880 msgr2=0x7fc50c06a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:56.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.607+0000 7fc513590640 1 -- 172.21.15.33:0/1928800943 shutdown_connections 2024-03-19T02:54:56.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.607+0000 7fc513590640 1 -- 172.21.15.33:0/1928800943 wait complete. 2024-03-19T02:54:56.619 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:54:56.619 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:54:56.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.764+0000 7f1d273f4640 1 -- 172.21.15.33:0/618760025 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d201066a0 msgr2=0x7f1d20106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:56.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.764+0000 7f1d273f4640 1 --2- 172.21.15.33:0/618760025 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d201066a0 0x7f1d20106a80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c009860 tx=0x7f1d1c02bb70 comp rx=0 tx=0).stop 2024-03-19T02:54:56.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.764+0000 7f1d273f4640 1 -- 172.21.15.33:0/618760025 shutdown_connections 2024-03-19T02:54:56.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.764+0000 7f1d273f4640 1 --2- 172.21.15.33:0/618760025 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1d20100690 0x7f1d20100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.764+0000 7f1d273f4640 1 --2- 172.21.15.33:0/618760025 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1d20101050 0x7f1d2010d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.764+0000 7f1d273f4640 1 --2- 172.21.15.33:0/618760025 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d201066a0 0x7f1d20106a80 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.764+0000 7f1d273f4640 1 -- 172.21.15.33:0/618760025 >> 172.21.15.33:0/618760025 conn(0x7f1d20077820 msgr2=0x7f1d20077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:56.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.764+0000 7f1d273f4640 1 -- 172.21.15.33:0/618760025 shutdown_connections 2024-03-19T02:54:56.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.764+0000 7f1d273f4640 1 -- 172.21.15.33:0/618760025 wait complete. 2024-03-19T02:54:56.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.764+0000 7f1d273f4640 1 Processor -- start 2024-03-19T02:54:56.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.764+0000 7f1d273f4640 1 -- start start 2024-03-19T02:54:56.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.765+0000 7f1d273f4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1d20100690 0x7f1d20196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.765+0000 7f1d273f4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d20101050 0x7f1d20197420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.765+0000 7f1d273f4640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1d201066a0 0x7f1d2019b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.765+0000 7f1d273f4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1d20110960 con 0x7f1d20100690 2024-03-19T02:54:56.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.765+0000 7f1d273f4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1d201107e0 con 0x7f1d20101050 2024-03-19T02:54:56.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.765+0000 7f1d273f4640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1d20110ae0 con 0x7f1d201066a0 2024-03-19T02:54:56.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.765+0000 7f1d25190640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1d20100690 0x7f1d20196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.765+0000 7f1d25190640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1d20100690 0x7f1d20196ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53482/0 (socket says 172.21.15.33:53482) 2024-03-19T02:54:56.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.765+0000 7f1d25190640 1 -- 172.21.15.33:0/2734831614 learned_addr learned my addr 172.21.15.33:0/2734831614 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:56.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.765+0000 7f1d2498f640 1 --2- 172.21.15.33:0/2734831614 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d20101050 0x7f1d20197420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.765+0000 7f1d25991640 1 --2- 172.21.15.33:0/2734831614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1d201066a0 0x7f1d2019b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.765+0000 7f1d25190640 1 -- 172.21.15.33:0/2734831614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1d201066a0 msgr2=0x7f1d2019b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:56.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.765+0000 7f1d25190640 1 --2- 172.21.15.33:0/2734831614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1d201066a0 0x7f1d2019b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.765+0000 7f1d25190640 1 -- 172.21.15.33:0/2734831614 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d20101050 msgr2=0x7f1d20197420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:56.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.765+0000 7f1d25190640 1 --2- 172.21.15.33:0/2734831614 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d20101050 0x7f1d20197420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.765+0000 7f1d25190640 1 -- 172.21.15.33:0/2734831614 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d2019bed0 con 0x7f1d20100690 2024-03-19T02:54:56.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.765+0000 7f1d25190640 1 --2- 172.21.15.33:0/2734831614 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1d20100690 0x7f1d20196ee0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c037040 tx=0x7f1d1c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:56.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.765+0000 7f1d167fc640 1 -- 172.21.15.33:0/2734831614 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c041020 con 0x7f1d20100690 2024-03-19T02:54:56.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.765+0000 7f1d273f4640 1 -- 172.21.15.33:0/2734831614 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d2019c160 con 0x7f1d20100690 2024-03-19T02:54:56.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.765+0000 7f1d167fc640 1 -- 172.21.15.33:0/2734831614 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d1c02c9c0 con 0x7f1d20100690 2024-03-19T02:54:56.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.765+0000 7f1d167fc640 1 -- 172.21.15.33:0/2734831614 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c039040 con 0x7f1d20100690 2024-03-19T02:54:56.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.766+0000 7f1d273f4640 1 -- 172.21.15.33:0/2734831614 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d201a3a50 con 0x7f1d20100690 2024-03-19T02:54:56.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.768+0000 7f1d167fc640 1 -- 172.21.15.33:0/2734831614 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 24) ==== 137523+0+0 (secure 0 0 0) 0x7f1d1c044020 con 0x7f1d20100690 2024-03-19T02:54:56.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.768+0000 7f1d29fe8740 1 Processor -- start 2024-03-19T02:54:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.768+0000 7f1d29fe8740 1 -- start start 2024-03-19T02:54:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.768+0000 7f1d167fc640 1 --2- 172.21.15.33:0/2734831614 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f1d0c0a2a90 0x7f1d0c0a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.768+0000 7f1d29fe8740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9211f2c40 0x55c9211f3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.768+0000 7f1d29fe8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9211f3560 0x55c9211fbc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.768+0000 7f1d29fe8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9211fc140 0x55c9211fe530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.768+0000 7f1d29fe8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c921154500 con 0x55c9211fc140 2024-03-19T02:54:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.768+0000 7f1d29fe8740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c9210fcbe0 con 0x55c9211f2c40 2024-03-19T02:54:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.768+0000 7f1d29fe8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c9211505a0 con 0x55c9211f3560 2024-03-19T02:54:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.769+0000 7f1d25991640 1 --2- 172.21.15.33:0/2734831614 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f1d0c0a2a90 0x7f1d0c0a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.769+0000 7f1d25190640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9211f2c40 0x55c9211f3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.769+0000 7f1d25190640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9211f2c40 0x55c9211f3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:51056/0 (socket says 172.21.15.33:51056) 2024-03-19T02:54:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.769+0000 7f1d25190640 1 -- 172.21.15.33:0/3194678010 learned_addr learned my addr 172.21.15.33:0/3194678010 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.769+0000 7f1d2498f640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9211f3560 0x55c9211fbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.769+0000 7f1d167fc640 1 -- 172.21.15.33:0/2734831614 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f1d1c0f0b30 con 0x7f1d20100690 2024-03-19T02:54:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.769+0000 7f1d25991640 1 --2- 172.21.15.33:0/3194678010 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9211fc140 0x55c9211fe530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.769+0000 7f1d25190640 1 -- 172.21.15.33:0/3194678010 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9211f3560 msgr2=0x55c9211fbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:56.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.769+0000 7f1d25190640 1 --2- 172.21.15.33:0/3194678010 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9211f3560 0x55c9211fbc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.769+0000 7f1d25190640 1 -- 172.21.15.33:0/3194678010 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9211fc140 msgr2=0x55c9211fe530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:56.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.769+0000 7f1d25190640 1 --2- 172.21.15.33:0/3194678010 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9211fc140 0x55c9211fe530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.769+0000 7f1d25190640 1 -- 172.21.15.33:0/3194678010 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c92111e000 con 0x55c9211f2c40 2024-03-19T02:54:56.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.769+0000 7f1d25991640 1 --2- 172.21.15.33:0/2734831614 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f1d0c0a2a90 0x7f1d0c0a4f50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1d18005fd0 tx=0x7f1d18009db0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:56.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.769+0000 7f1d25190640 1 --2- 172.21.15.33:0/3194678010 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9211f2c40 0x55c9211f3020 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c0c4f50 tx=0x7f1d1c005b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=308d5928dd7c75f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:56.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.769+0000 7f1d0b7fe640 1 -- 172.21.15.33:0/3194678010 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c0bc820 con 0x55c9211f2c40 2024-03-19T02:54:56.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.769+0000 7f1d0b7fe640 1 -- 172.21.15.33:0/3194678010 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d1c0bd3e0 con 0x55c9211f2c40 2024-03-19T02:54:56.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.769+0000 7f1d0b7fe640 1 -- 172.21.15.33:0/3194678010 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c0bc640 con 0x55c9211f2c40 2024-03-19T02:54:56.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.770+0000 7f1d29fe8740 1 -- 172.21.15.33:0/3194678010 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9211f2c40 msgr2=0x55c9211f3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:56.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.770+0000 7f1d29fe8740 1 --2- 172.21.15.33:0/3194678010 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9211f2c40 0x55c9211f3020 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c0c4f50 tx=0x7f1d1c005b20 comp rx=0 tx=0).stop 2024-03-19T02:54:56.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.770+0000 7f1d29fe8740 1 -- 172.21.15.33:0/3194678010 shutdown_connections 2024-03-19T02:54:56.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.770+0000 7f1d29fe8740 1 --2- 172.21.15.33:0/3194678010 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9211f3560 0x55c9211fbc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.770+0000 7f1d29fe8740 1 --2- 172.21.15.33:0/3194678010 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9211fc140 0x55c9211fe530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.770+0000 7f1d29fe8740 1 --2- 172.21.15.33:0/3194678010 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9211f2c40 0x55c9211f3020 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.770+0000 7f1d29fe8740 1 -- 172.21.15.33:0/3194678010 >> 172.21.15.33:0/3194678010 conn(0x55c9211ed370 msgr2=0x55c9211ed750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:56.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.770+0000 7f1d29fe8740 1 -- 172.21.15.33:0/3194678010 shutdown_connections 2024-03-19T02:54:56.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.770+0000 7f1d29fe8740 1 -- 172.21.15.33:0/3194678010 wait complete. 2024-03-19T02:54:56.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.770+0000 7f1d29fe8740 1 -- start start 2024-03-19T02:54:56.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.770+0000 7f1d29fe8740 1 Processor -- start 2024-03-19T02:54:56.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.771+0000 7f1d29fe8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9211f2c40 0x55c9211f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.771+0000 7f1d29fe8740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9211f3560 0x55c9211f7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.771+0000 7f1d29fe8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9211fc140 0x55c9211f7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.771+0000 7f1d29fe8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c9211eaa50 con 0x55c9211f2c40 2024-03-19T02:54:56.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.771+0000 7f1d29fe8740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c921110160 con 0x55c9211f3560 2024-03-19T02:54:56.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.771+0000 7f1d29fe8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c9211eabd0 con 0x55c9211fc140 2024-03-19T02:54:56.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.771+0000 7f1d29fe8740 10 client.0 _ll_register_callbacks cb 0x55c92114ca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:54:56.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.771+0000 7f1d25190640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9211f2c40 0x55c9211f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.771+0000 7f1d25190640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9211f2c40 0x55c9211f72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53500/0 (socket says 172.21.15.33:53500) 2024-03-19T02:54:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.771+0000 7f1d25190640 1 -- 172.21.15.33:0/1645402682 learned_addr learned my addr 172.21.15.33:0/1645402682 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.771+0000 7f1d2498f640 1 --2- 172.21.15.33:0/1645402682 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9211f3560 0x55c9211f7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.771+0000 7f1d25991640 1 --2- 172.21.15.33:0/1645402682 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9211fc140 0x55c9211f7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.771+0000 7f1d25190640 1 -- 172.21.15.33:0/1645402682 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9211fc140 msgr2=0x55c9211f7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.771+0000 7f1d25190640 1 --2- 172.21.15.33:0/1645402682 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9211fc140 0x55c9211f7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.771+0000 7f1d25190640 1 -- 172.21.15.33:0/1645402682 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9211f3560 msgr2=0x55c9211f7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.771+0000 7f1d25190640 1 --2- 172.21.15.33:0/1645402682 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9211f3560 0x55c9211f7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.771+0000 7f1d25190640 1 -- 172.21.15.33:0/1645402682 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c920f327d0 con 0x55c9211f2c40 2024-03-19T02:54:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.771+0000 7f1d25190640 1 --2- 172.21.15.33:0/1645402682 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9211f2c40 0x55c9211f72f0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c02cb90 tx=0x7f1d1c0bc640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f8f1ed9dc2beee server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.771+0000 7f1d08ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:54:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.772+0000 7f1d08ff9640 1 -- 172.21.15.33:0/1645402682 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c0bcce0 con 0x55c9211f2c40 2024-03-19T02:54:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.772+0000 7f1d08ff9640 1 -- 172.21.15.33:0/1645402682 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f1d1c003680 con 0x55c9211f2c40 2024-03-19T02:54:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.772+0000 7f1d29fe8740 1 -- 172.21.15.33:0/1645402682 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d1c0bd3e0 con 0x55c9211f2c40 2024-03-19T02:54:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.772+0000 7f1d08ff9640 1 -- 172.21.15.33:0/1645402682 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1d1c0c0e30 con 0x55c9211f2c40 2024-03-19T02:54:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.772+0000 7f1d29fe8740 10 client.16035 fetch_fsmap learned FSMap version 25 2024-03-19T02:54:56.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.772+0000 7f1d29fe8740 1 -- 172.21.15.33:0/1645402682 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c920e683f0 con 0x55c9211f2c40 2024-03-19T02:54:56.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.772+0000 7f1d08ff9640 1 -- 172.21.15.33:0/1645402682 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d1c0bd3e0 con 0x55c9211f2c40 2024-03-19T02:54:56.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.772+0000 7f1d08ff9640 1 -- 172.21.15.33:0/1645402682 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f1d1c03c030 con 0x55c9211f2c40 2024-03-19T02:54:56.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.772+0000 7f1d29fe8740 10 client.16035 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:54:56.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.772+0000 7f1d29fe8740 10 client.16035 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:54:56.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.772+0000 7f1d29fe8740 20 client.16035 populate_metadata read hostname 'smithi033' 2024-03-19T02:54:56.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.772+0000 7f1d29fe8740 1 --2- 172.21.15.33:0/1645402682 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55c921209e80 0x55c92120c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.772+0000 7f1d29fe8740 4 client.16035 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:54:56.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.772+0000 7f1d29fe8740 1 -- 172.21.15.33:0/1645402682 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c921161a40 con 0x55c921209e80 2024-03-19T02:54:56.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.772+0000 7f1d2498f640 1 --2- 172.21.15.33:0/1645402682 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55c921209e80 0x55c92120c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.773+0000 7f1d2498f640 1 --2- 172.21.15.33:0/1645402682 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55c921209e80 0x55c92120c270 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69633c68be9aa40d server_cookie=f32427bec8ce05c0 in_seq=0 out_seq=0 2024-03-19T02:54:56.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.773+0000 7f1d08ff9640 10 client.16035 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:54:56.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.773+0000 7f1d08ff9640 1 -- 172.21.15.33:0/1645402682 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c921161a40 con 0x55c921209e80 2024-03-19T02:54:56.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.773+0000 7f1d08ff9640 10 client.16035 handle_command_reply: tid=0 2024-03-19T02:54:56.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.773+0000 7f1d29fe8740 1 client.16035 shutdown 2024-03-19T02:54:56.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.773+0000 7f1d29fe8740 1 -- 172.21.15.33:0/1645402682 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9211f2c40 msgr2=0x55c9211f72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:56.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.773+0000 7f1d29fe8740 1 --2- 172.21.15.33:0/1645402682 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9211f2c40 0x55c9211f72f0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c02cb90 tx=0x7f1d1c0bc640 comp rx=0 tx=0).stop 2024-03-19T02:54:56.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.773+0000 7f1d29fe8740 1 -- 172.21.15.33:0/1645402682 shutdown_connections 2024-03-19T02:54:56.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.773+0000 7f1d29fe8740 1 --2- 172.21.15.33:0/1645402682 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55c921209e80 0x55c92120c270 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.773+0000 7f1d29fe8740 1 --2- 172.21.15.33:0/1645402682 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9211fc140 0x55c9211f7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.773+0000 7f1d08ff9640 0 client.16035 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:54:56.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.773+0000 7f1d29fe8740 1 --2- 172.21.15.33:0/1645402682 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9211f3560 0x55c9211f7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.773+0000 7f1d29fe8740 1 --2- 172.21.15.33:0/1645402682 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9211f2c40 0x55c9211f72f0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.773+0000 7f1d29fe8740 1 -- 172.21.15.33:0/1645402682 >> 172.21.15.33:0/1645402682 conn(0x55c9211ed370 msgr2=0x55c9211ef0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:56.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.774+0000 7f1d29fe8740 1 -- 172.21.15.33:0/1645402682 shutdown_connections 2024-03-19T02:54:56.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.774+0000 7f1d29fe8740 1 -- 172.21.15.33:0/1645402682 wait complete. 2024-03-19T02:54:56.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.774+0000 7f1d29fe8740 20 client.16035 trim_cache size 0 max 16384 2024-03-19T02:54:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.791+0000 7f1d29fe8740 1 Processor -- start 2024-03-19T02:54:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.791+0000 7f1d29fe8740 1 -- start start 2024-03-19T02:54:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.791+0000 7f1d29fe8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9211f2c40 0x55c9211fba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.791+0000 7f1d29fe8740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9211f3560 0x55c9211f4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.791+0000 7f1d29fe8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9211fc140 0x55c921211b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.791+0000 7f1d29fe8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1d201af250 con 0x55c9211fc140 2024-03-19T02:54:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.791+0000 7f1d29fe8740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1d201acb20 con 0x55c9211f3560 2024-03-19T02:54:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.791+0000 7f1d29fe8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1d201aa950 con 0x55c9211f2c40 2024-03-19T02:54:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.791+0000 7f1d25991640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9211fc140 0x55c921211b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.791+0000 7f1d25991640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9211fc140 0x55c921211b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53506/0 (socket says 172.21.15.33:53506) 2024-03-19T02:54:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.791+0000 7f1d25991640 1 -- 172.21.15.33:0/251531710 learned_addr learned my addr 172.21.15.33:0/251531710 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.791+0000 7f1d2498f640 1 --2- 172.21.15.33:0/251531710 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9211f3560 0x55c9211f4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.791+0000 7f1d25190640 1 --2- 172.21.15.33:0/251531710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9211f2c40 0x55c9211fba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.791+0000 7f1d25991640 1 -- 172.21.15.33:0/251531710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9211f2c40 msgr2=0x55c9211fba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.791+0000 7f1d25991640 1 --2- 172.21.15.33:0/251531710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9211f2c40 0x55c9211fba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.791+0000 7f1d25991640 1 -- 172.21.15.33:0/251531710 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9211f3560 msgr2=0x55c9211f4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.791+0000 7f1d25991640 1 --2- 172.21.15.33:0/251531710 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9211f3560 0x55c9211f4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.791+0000 7f1d25991640 1 -- 172.21.15.33:0/251531710 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c920e280f0 con 0x55c9211fc140 2024-03-19T02:54:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.792+0000 7f1d25991640 1 --2- 172.21.15.33:0/251531710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9211fc140 0x55c921211b20 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f1d18009a60 tx=0x7f1d18020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91732d50fb7ded27 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.792+0000 7f1d0affd640 1 -- 172.21.15.33:0/251531710 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1802d070 con 0x55c9211fc140 2024-03-19T02:54:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.792+0000 7f1d0affd640 1 -- 172.21.15.33:0/251531710 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d18003d00 con 0x55c9211fc140 2024-03-19T02:54:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.792+0000 7f1d0affd640 1 -- 172.21.15.33:0/251531710 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1801d630 con 0x55c9211fc140 2024-03-19T02:54:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.792+0000 7f1d29fe8740 1 -- 172.21.15.33:0/251531710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9211fc140 msgr2=0x55c921211b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.792+0000 7f1d29fe8740 1 --2- 172.21.15.33:0/251531710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9211fc140 0x55c921211b20 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f1d18009a60 tx=0x7f1d18020000 comp rx=0 tx=0).stop 2024-03-19T02:54:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.792+0000 7f1d29fe8740 1 -- 172.21.15.33:0/251531710 shutdown_connections 2024-03-19T02:54:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.792+0000 7f1d29fe8740 1 --2- 172.21.15.33:0/251531710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9211fc140 0x55c921211b20 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.792+0000 7f1d29fe8740 1 --2- 172.21.15.33:0/251531710 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9211f3560 0x55c9211f4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.792+0000 7f1d29fe8740 1 --2- 172.21.15.33:0/251531710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9211f2c40 0x55c9211fba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.792+0000 7f1d29fe8740 1 -- 172.21.15.33:0/251531710 >> 172.21.15.33:0/251531710 conn(0x55c9211ed370 msgr2=0x55c9211ea1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.792+0000 7f1d29fe8740 1 -- 172.21.15.33:0/251531710 shutdown_connections 2024-03-19T02:54:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.792+0000 7f1d29fe8740 1 -- 172.21.15.33:0/251531710 wait complete. 2024-03-19T02:54:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.793+0000 7f1d29fe8740 1 -- start start 2024-03-19T02:54:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.793+0000 7f1d29fe8740 1 Processor -- start 2024-03-19T02:54:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.793+0000 7f1d29fe8740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9211f2c40 0x55c921205150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.793+0000 7f1d29fe8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9211f3560 0x55c921205690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.793+0000 7f1d29fe8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9211fc140 0x55c9211eb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.793+0000 7f1d29fe8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1d201aafe0 con 0x55c9211fc140 2024-03-19T02:54:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.793+0000 7f1d29fe8740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1d201ac450 con 0x55c9211f2c40 2024-03-19T02:54:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.793+0000 7f1d29fe8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1d201ab6b0 con 0x55c9211f3560 2024-03-19T02:54:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.793+0000 7f1d29fe8740 10 client.0 _ll_register_callbacks cb 0x7f1d1800e6f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:54:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.793+0000 7f1d25991640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9211fc140 0x55c9211eb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.793+0000 7f1d25190640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9211f2c40 0x55c921205150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.793+0000 7f1d25991640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9211fc140 0x55c9211eb500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53508/0 (socket says 172.21.15.33:53508) 2024-03-19T02:54:56.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.793+0000 7f1d25991640 1 -- 172.21.15.33:0/2279887780 learned_addr learned my addr 172.21.15.33:0/2279887780 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:54:56.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.793+0000 7f1d2498f640 1 --2- 172.21.15.33:0/2279887780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9211f3560 0x55c921205690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.794+0000 7f1d25190640 1 -- 172.21.15.33:0/2279887780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9211f3560 msgr2=0x55c921205690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:56.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.794+0000 7f1d25190640 1 --2- 172.21.15.33:0/2279887780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9211f3560 0x55c921205690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.794+0000 7f1d25190640 1 -- 172.21.15.33:0/2279887780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9211fc140 msgr2=0x55c9211eb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:54:56.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.794+0000 7f1d25190640 1 --2- 172.21.15.33:0/2279887780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9211fc140 0x55c9211eb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.794+0000 7f1d25190640 1 -- 172.21.15.33:0/2279887780 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9211ebd00 con 0x55c9211f2c40 2024-03-19T02:54:56.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.794+0000 7f1d25991640 1 --2- 172.21.15.33:0/2279887780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9211fc140 0x55c9211eb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:54:56.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.794+0000 7f1d25190640 1 --2- 172.21.15.33:0/2279887780 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9211f2c40 0x55c921205150 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c0b9140 tx=0x7f1d1c02c630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=618a9ea2a140aec5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:54:56.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.794+0000 7f1d08ff9640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T02:54:56.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.794+0000 7f1d08ff9640 1 -- 172.21.15.33:0/2279887780 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c0e4350 con 0x55c9211f2c40 2024-03-19T02:54:56.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.794+0000 7f1d08ff9640 1 -- 172.21.15.33:0/2279887780 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f1d1c0c4ff0 con 0x55c9211f2c40 2024-03-19T02:54:56.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.794+0000 7f1d29fe8740 1 -- 172.21.15.33:0/2279887780 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d18003d00 con 0x55c9211f2c40 2024-03-19T02:54:56.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.794+0000 7f1d08ff9640 1 -- 172.21.15.33:0/2279887780 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1d1c0b8f80 con 0x55c9211f2c40 2024-03-19T02:54:56.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.794+0000 7f1d29fe8740 10 client.25892 fetch_fsmap learned FSMap version 25 2024-03-19T02:54:56.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.794+0000 7f1d29fe8740 1 -- 172.21.15.33:0/2279887780 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c9211ebd00 con 0x55c9211f2c40 2024-03-19T02:54:56.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.794+0000 7f1d08ff9640 1 -- 172.21.15.33:0/2279887780 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d18003d00 con 0x55c9211f2c40 2024-03-19T02:54:56.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.794+0000 7f1d08ff9640 1 -- 172.21.15.33:0/2279887780 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f1d1c0b8a30 con 0x55c9211f2c40 2024-03-19T02:54:56.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.794+0000 7f1d29fe8740 10 client.25892 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:54:56.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.794+0000 7f1d29fe8740 10 client.25892 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:54:56.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.794+0000 7f1d29fe8740 20 client.25892 populate_metadata read hostname 'smithi033' 2024-03-19T02:54:56.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.794+0000 7f1d29fe8740 1 --2- 172.21.15.33:0/2279887780 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55c921270fc0 0x55c9212733b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:54:56.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.794+0000 7f1d29fe8740 4 client.25892 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:54:56.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.794+0000 7f1d29fe8740 1 -- 172.21.15.33:0/2279887780 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c92115bd90 con 0x55c921270fc0 2024-03-19T02:54:56.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.795+0000 7f1d2498f640 1 --2- 172.21.15.33:0/2279887780 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55c921270fc0 0x55c9212733b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:54:56.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.795+0000 7f1d2498f640 1 --2- 172.21.15.33:0/2279887780 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55c921270fc0 0x55c9212733b0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60385ae4a6cede7e server_cookie=e0efcecf3c296db3 in_seq=0 out_seq=0 2024-03-19T02:54:56.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.795+0000 7f1d08ff9640 10 client.25892 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:54:56.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.795+0000 7f1d08ff9640 1 -- 172.21.15.33:0/2279887780 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55c92115bd90 con 0x55c921270fc0 2024-03-19T02:54:56.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.795+0000 7f1d08ff9640 10 client.25892 handle_command_reply: tid=0 2024-03-19T02:54:56.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.795+0000 7f1d29fe8740 1 client.25892 shutdown 2024-03-19T02:54:56.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.795+0000 7f1d29fe8740 1 -- 172.21.15.33:0/2279887780 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9211f2c40 msgr2=0x55c921205150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:56.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.795+0000 7f1d29fe8740 1 --2- 172.21.15.33:0/2279887780 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9211f2c40 0x55c921205150 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c0b9140 tx=0x7f1d1c02c630 comp rx=0 tx=0).stop 2024-03-19T02:54:56.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.795+0000 7f1d29fe8740 1 -- 172.21.15.33:0/2279887780 shutdown_connections 2024-03-19T02:54:56.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.795+0000 7f1d29fe8740 1 --2- 172.21.15.33:0/2279887780 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55c921270fc0 0x55c9212733b0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.795+0000 7f1d29fe8740 1 --2- 172.21.15.33:0/2279887780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c9211f3560 0x55c921205690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.795+0000 7f1d29fe8740 1 --2- 172.21.15.33:0/2279887780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c9211fc140 0x55c9211eb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.795+0000 7f1d29fe8740 1 --2- 172.21.15.33:0/2279887780 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c9211f2c40 0x55c921205150 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.795+0000 7f1d08ff9640 0 client.25892 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:54:56.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.795+0000 7f1d29fe8740 1 -- 172.21.15.33:0/2279887780 >> 172.21.15.33:0/2279887780 conn(0x55c9211ed370 msgr2=0x55c9211e9950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:56.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.795+0000 7f1d29fe8740 1 -- 172.21.15.33:0/2279887780 shutdown_connections 2024-03-19T02:54:56.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.795+0000 7f1d29fe8740 1 -- 172.21.15.33:0/2279887780 wait complete. 2024-03-19T02:54:56.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.796+0000 7f1d29fe8740 20 client.25892 trim_cache size 0 max 16384 2024-03-19T02:54:56.805 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:54:56.805 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:54:56.805 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-03-19T02:54:56.805 INFO:teuthology.orchestra.run.smithi033.stdout: "685d94f0-8322-420c-baa2-c8e0f5a4a289": { 2024-03-19T02:54:56.805 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-03-19T02:54:56.805 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "685d94f0-8322-420c-baa2-c8e0f5a4a289", 2024-03-19T02:54:56.805 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-03-19T02:54:56.805 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T02:54:56.805 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T02:54:56.805 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:54:56.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.796+0000 7f1d273f4640 1 -- 172.21.15.33:0/2734831614 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f1d0c0a2a90 msgr2=0x7f1d0c0a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:56.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.796+0000 7f1d273f4640 1 --2- 172.21.15.33:0/2734831614 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f1d0c0a2a90 0x7f1d0c0a4f50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1d18005fd0 tx=0x7f1d18009db0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.796+0000 7f1d273f4640 1 -- 172.21.15.33:0/2734831614 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1d20100690 msgr2=0x7f1d20196ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:54:56.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.796+0000 7f1d273f4640 1 --2- 172.21.15.33:0/2734831614 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1d20100690 0x7f1d20196ee0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c037040 tx=0x7f1d1c004040 comp rx=0 tx=0).stop 2024-03-19T02:54:56.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.796+0000 7f1d273f4640 1 -- 172.21.15.33:0/2734831614 shutdown_connections 2024-03-19T02:54:56.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.796+0000 7f1d273f4640 1 --2- 172.21.15.33:0/2734831614 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f1d0c0a2a90 0x7f1d0c0a4f50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.796+0000 7f1d273f4640 1 --2- 172.21.15.33:0/2734831614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1d201066a0 0x7f1d2019b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.796+0000 7f1d273f4640 1 --2- 172.21.15.33:0/2734831614 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1d20101050 0x7f1d20197420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.796+0000 7f1d273f4640 1 --2- 172.21.15.33:0/2734831614 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1d20100690 0x7f1d20196ee0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:54:56.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.796+0000 7f1d273f4640 1 -- 172.21.15.33:0/2734831614 >> 172.21.15.33:0/2734831614 conn(0x7f1d20077820 msgr2=0x7f1d20102ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:54:56.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.796+0000 7f1d273f4640 1 -- 172.21.15.33:0/2734831614 shutdown_connections 2024-03-19T02:54:56.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:54:56.796+0000 7f1d273f4640 1 -- 172.21.15.33:0/2734831614 wait complete. 2024-03-19T02:54:56.808 INFO:tasks.cephfs.filesystem:scrub status for tag:685d94f0-8322-420c-baa2-c8e0f5a4a289 - {'path': '/', 'tag': '685d94f0-8322-420c-baa2-c8e0f5a4a289', 'options': 'recursive,force'} 2024-03-19T02:54:57.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:57 smithi096 ceph-mon[35946]: scrub queued for path: / 2024-03-19T02:54:57.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:57 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T02:54:57.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:57 smithi096 ceph-mon[35946]: scrub summary: active paths [/] 2024-03-19T02:54:57.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:57 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T02:54:57.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:57 smithi155 ceph-mon[35804]: scrub queued for path: / 2024-03-19T02:54:57.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:57 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T02:54:57.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:57 smithi155 ceph-mon[35804]: scrub summary: active paths [/] 2024-03-19T02:54:57.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:57 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T02:54:57.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:57 smithi033 ceph-mon[31973]: scrub queued for path: / 2024-03-19T02:54:57.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:57 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T02:54:57.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:57 smithi033 ceph-mon[31973]: scrub summary: active paths [/] 2024-03-19T02:54:57.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:57 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T02:54:58.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:58 smithi096 ceph-mon[35946]: pgmap v48: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-19T02:54:58.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:58 smithi096 ceph-mon[35946]: scrub summary: idle 2024-03-19T02:54:58.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:58 smithi155 ceph-mon[35804]: pgmap v48: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-19T02:54:58.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:58 smithi155 ceph-mon[35804]: scrub summary: idle 2024-03-19T02:54:58.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:58 smithi033 ceph-mon[31973]: pgmap v48: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-19T02:54:58.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:58 smithi033 ceph-mon[31973]: scrub summary: idle 2024-03-19T02:54:59.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:54:59 smithi096 ceph-mon[35946]: mgrmap e25: x(active, since 91s), standbys: z, y 2024-03-19T02:54:59.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:54:59 smithi155 ceph-mon[35804]: mgrmap e25: x(active, since 91s), standbys: z, y 2024-03-19T02:54:59.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:54:59 smithi033 ceph-mon[31973]: mgrmap e25: x(active, since 91s), standbys: z, y 2024-03-19T02:54:59.763 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-03-19T02:54:59.763 INFO:tasks.workunit.client.0.smithi033.stdout: Children see throughput for 1 rewriters = 694013.00 kB/sec 2024-03-19T02:54:59.763 INFO:tasks.workunit.client.0.smithi033.stdout: Parent sees throughput for 1 rewriters = 693906.93 kB/sec 2024-03-19T02:54:59.763 INFO:tasks.workunit.client.0.smithi033.stdout: Min throughput per process = 694013.00 kB/sec 2024-03-19T02:54:59.763 INFO:tasks.workunit.client.0.smithi033.stdout: Max throughput per process = 694013.00 kB/sec 2024-03-19T02:54:59.763 INFO:tasks.workunit.client.0.smithi033.stdout: Avg throughput per process = 694013.00 kB/sec 2024-03-19T02:54:59.763 INFO:tasks.workunit.client.0.smithi033.stdout: Min xfer = 1048576.00 kB 2024-03-19T02:55:00.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:00 smithi096 ceph-mon[35946]: pgmap v49: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 85 MiB/s wr, 54 op/s 2024-03-19T02:55:00.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:00 smithi155 ceph-mon[35804]: pgmap v49: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 85 MiB/s wr, 54 op/s 2024-03-19T02:55:00.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:00 smithi033 ceph-mon[31973]: pgmap v49: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 85 MiB/s wr, 54 op/s 2024-03-19T02:55:02.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:02 smithi096 ceph-mon[35946]: pgmap v50: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 85 MiB/s wr, 51 op/s 2024-03-19T02:55:02.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:02 smithi155 ceph-mon[35804]: pgmap v50: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 85 MiB/s wr, 51 op/s 2024-03-19T02:55:02.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:02 smithi033 ceph-mon[31973]: pgmap v50: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 85 MiB/s wr, 51 op/s 2024-03-19T02:55:03.041 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-03-19T02:55:03.041 INFO:tasks.workunit.client.0.smithi033.stdout: Children see throughput for 1 readers = 7591283.50 kB/sec 2024-03-19T02:55:03.041 INFO:tasks.workunit.client.0.smithi033.stdout: Parent sees throughput for 1 readers = 7580031.51 kB/sec 2024-03-19T02:55:03.041 INFO:tasks.workunit.client.0.smithi033.stdout: Min throughput per process = 7591283.50 kB/sec 2024-03-19T02:55:03.041 INFO:tasks.workunit.client.0.smithi033.stdout: Max throughput per process = 7591283.50 kB/sec 2024-03-19T02:55:03.042 INFO:tasks.workunit.client.0.smithi033.stdout: Avg throughput per process = 7591283.50 kB/sec 2024-03-19T02:55:03.042 INFO:tasks.workunit.client.0.smithi033.stdout: Min xfer = 1048576.00 kB 2024-03-19T02:55:04.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:04 smithi096 ceph-mon[35946]: pgmap v51: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 171 MiB/s wr, 90 op/s 2024-03-19T02:55:04.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:04 smithi155 ceph-mon[35804]: pgmap v51: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 171 MiB/s wr, 90 op/s 2024-03-19T02:55:04.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:04 smithi033 ceph-mon[31973]: pgmap v51: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 171 MiB/s wr, 90 op/s 2024-03-19T02:55:05.265 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-03-19T02:55:05.266 INFO:tasks.workunit.client.0.smithi033.stdout: Children see throughput for 1 re-readers = 7595675.50 kB/sec 2024-03-19T02:55:05.266 INFO:tasks.workunit.client.0.smithi033.stdout: Parent sees throughput for 1 re-readers = 7584305.91 kB/sec 2024-03-19T02:55:05.266 INFO:tasks.workunit.client.0.smithi033.stdout: Min throughput per process = 7595675.50 kB/sec 2024-03-19T02:55:05.266 INFO:tasks.workunit.client.0.smithi033.stdout: Max throughput per process = 7595675.50 kB/sec 2024-03-19T02:55:05.266 INFO:tasks.workunit.client.0.smithi033.stdout: Avg throughput per process = 7595675.50 kB/sec 2024-03-19T02:55:05.266 INFO:tasks.workunit.client.0.smithi033.stdout: Min xfer = 1048576.00 kB 2024-03-19T02:55:06.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:06 smithi096 ceph-mon[35946]: pgmap v52: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 171 MiB/s wr, 87 op/s 2024-03-19T02:55:06.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:06 smithi155 ceph-mon[35804]: pgmap v52: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 171 MiB/s wr, 87 op/s 2024-03-19T02:55:06.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:06 smithi033 ceph-mon[31973]: pgmap v52: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 171 MiB/s wr, 87 op/s 2024-03-19T02:55:08.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:08 smithi096 ceph-mon[35946]: pgmap v53: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 171 MiB/s wr, 87 op/s 2024-03-19T02:55:08.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:08 smithi155 ceph-mon[35804]: pgmap v53: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 171 MiB/s wr, 87 op/s 2024-03-19T02:55:08.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:08 smithi033 ceph-mon[31973]: pgmap v53: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 171 MiB/s wr, 87 op/s 2024-03-19T02:55:10.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:10 smithi096 ceph-mon[35946]: pgmap v54: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 171 MiB/s wr, 90 op/s 2024-03-19T02:55:10.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:10 smithi155 ceph-mon[35804]: pgmap v54: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 171 MiB/s wr, 90 op/s 2024-03-19T02:55:10.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:10 smithi033 ceph-mon[31973]: pgmap v54: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 171 MiB/s wr, 90 op/s 2024-03-19T02:55:11.701 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-03-19T02:55:11.701 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-03-19T02:55:11.701 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-03-19T02:55:11.701 INFO:tasks.workunit.client.0.smithi033.stdout:iozone test complete. 2024-03-19T02:55:11.702 INFO:tasks.workunit.client.0.smithi033.stderr:+ iozone -c -e -s 10240M -r 1M -t 1 -F f3 -i 0 -i 1 2024-03-19T02:55:11.768 INFO:tasks.workunit.client.0.smithi033.stdout: Iozone: Performance Test of File I/O 2024-03-19T02:55:11.769 INFO:tasks.workunit.client.0.smithi033.stdout: Version $Revision: 3.494 $ 2024-03-19T02:55:11.769 INFO:tasks.workunit.client.0.smithi033.stdout: Compiled for 64 bit mode. 2024-03-19T02:55:11.769 INFO:tasks.workunit.client.0.smithi033.stdout: Build: linux-AMD64 2024-03-19T02:55:11.769 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-03-19T02:55:11.769 INFO:tasks.workunit.client.0.smithi033.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-03-19T02:55:11.769 INFO:tasks.workunit.client.0.smithi033.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-03-19T02:55:11.769 INFO:tasks.workunit.client.0.smithi033.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-03-19T02:55:11.769 INFO:tasks.workunit.client.0.smithi033.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-03-19T02:55:11.769 INFO:tasks.workunit.client.0.smithi033.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-03-19T02:55:11.769 INFO:tasks.workunit.client.0.smithi033.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-03-19T02:55:11.769 INFO:tasks.workunit.client.0.smithi033.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-03-19T02:55:11.769 INFO:tasks.workunit.client.0.smithi033.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-03-19T02:55:11.769 INFO:tasks.workunit.client.0.smithi033.stdout: Alexey Skidanov, Sudhir Kumar. 2024-03-19T02:55:11.769 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-03-19T02:55:11.770 INFO:tasks.workunit.client.0.smithi033.stdout: Run began: Tue Mar 19 02:55:11 2024 2024-03-19T02:55:11.770 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-03-19T02:55:11.770 INFO:tasks.workunit.client.0.smithi033.stdout: Include close in write timing 2024-03-19T02:55:11.770 INFO:tasks.workunit.client.0.smithi033.stdout: Include fsync in write timing 2024-03-19T02:55:11.770 INFO:tasks.workunit.client.0.smithi033.stdout: File size set to 10485760 kB 2024-03-19T02:55:11.770 INFO:tasks.workunit.client.0.smithi033.stdout: Record Size 1024 kB 2024-03-19T02:55:11.770 INFO:tasks.workunit.client.0.smithi033.stdout: Command line used: iozone -c -e -s 10240M -r 1M -t 1 -F f3 -i 0 -i 1 2024-03-19T02:55:11.770 INFO:tasks.workunit.client.0.smithi033.stdout: Output is in kBytes/sec 2024-03-19T02:55:11.770 INFO:tasks.workunit.client.0.smithi033.stdout: Time Resolution = 0.000001 seconds. 2024-03-19T02:55:11.770 INFO:tasks.workunit.client.0.smithi033.stdout: Processor cache size set to 1024 kBytes. 2024-03-19T02:55:11.770 INFO:tasks.workunit.client.0.smithi033.stdout: Processor cache line size set to 32 bytes. 2024-03-19T02:55:11.770 INFO:tasks.workunit.client.0.smithi033.stdout: File stride size set to 17 * record size. 2024-03-19T02:55:11.770 INFO:tasks.workunit.client.0.smithi033.stdout: Throughput test with 1 process 2024-03-19T02:55:11.770 INFO:tasks.workunit.client.0.smithi033.stdout: Each process writes a 10485760 kByte file in 1024 kByte records 2024-03-19T02:55:12.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:12 smithi096 ceph-mon[35946]: pgmap v55: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 85 MiB/s wr, 41 op/s 2024-03-19T02:55:12.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:12 smithi155 ceph-mon[35804]: pgmap v55: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 85 MiB/s wr, 41 op/s 2024-03-19T02:55:12.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:12 smithi033 ceph-mon[31973]: pgmap v55: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 85 MiB/s wr, 41 op/s 2024-03-19T02:55:14.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:14 smithi096 ceph-mon[35946]: pgmap v56: 129 pgs: 129 active+clean; 1.7 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 144 MiB/s wr, 60 op/s 2024-03-19T02:55:14.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:14 smithi155 ceph-mon[35804]: pgmap v56: 129 pgs: 129 active+clean; 1.7 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 144 MiB/s wr, 60 op/s 2024-03-19T02:55:14.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:14 smithi033 ceph-mon[31973]: pgmap v56: 129 pgs: 129 active+clean; 1.7 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 144 MiB/s wr, 60 op/s 2024-03-19T02:55:16.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:16 smithi155 ceph-mon[35804]: pgmap v57: 129 pgs: 129 active+clean; 1.7 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 59 MiB/s wr, 21 op/s 2024-03-19T02:55:16.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:16 smithi033 ceph-mon[31973]: pgmap v57: 129 pgs: 129 active+clean; 1.7 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 59 MiB/s wr, 21 op/s 2024-03-19T02:55:16.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:16 smithi096 ceph-mon[35946]: pgmap v57: 129 pgs: 129 active+clean; 1.7 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 59 MiB/s wr, 21 op/s 2024-03-19T02:55:18.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:18 smithi155 ceph-mon[35804]: pgmap v58: 129 pgs: 129 active+clean; 1.7 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 59 MiB/s wr, 21 op/s 2024-03-19T02:55:18.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:18 smithi033 ceph-mon[31973]: pgmap v58: 129 pgs: 129 active+clean; 1.7 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 59 MiB/s wr, 21 op/s 2024-03-19T02:55:18.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:18 smithi096 ceph-mon[35946]: pgmap v58: 129 pgs: 129 active+clean; 1.7 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 59 MiB/s wr, 21 op/s 2024-03-19T02:55:20.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:20 smithi155 ceph-mon[35804]: pgmap v59: 129 pgs: 129 active+clean; 4.7 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 314 MiB/s wr, 110 op/s 2024-03-19T02:55:20.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:20 smithi033 ceph-mon[31973]: pgmap v59: 129 pgs: 129 active+clean; 4.7 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 314 MiB/s wr, 110 op/s 2024-03-19T02:55:20.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:20 smithi096 ceph-mon[35946]: pgmap v59: 129 pgs: 129 active+clean; 4.7 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 314 MiB/s wr, 110 op/s 2024-03-19T02:55:22.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:22 smithi155 ceph-mon[35804]: pgmap v60: 129 pgs: 129 active+clean; 4.7 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 314 MiB/s wr, 107 op/s 2024-03-19T02:55:22.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:22 smithi033 ceph-mon[31973]: pgmap v60: 129 pgs: 129 active+clean; 4.7 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 314 MiB/s wr, 107 op/s 2024-03-19T02:55:22.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:22 smithi096 ceph-mon[35946]: pgmap v60: 129 pgs: 129 active+clean; 4.7 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 314 MiB/s wr, 107 op/s 2024-03-19T02:55:24.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:24 smithi155 ceph-mon[35804]: pgmap v61: 129 pgs: 129 active+clean; 7.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 571 MiB/s wr, 174 op/s 2024-03-19T02:55:24.517 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:24 smithi096 ceph-mon[35946]: pgmap v61: 129 pgs: 129 active+clean; 7.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 571 MiB/s wr, 174 op/s 2024-03-19T02:55:24.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:24 smithi033 ceph-mon[31973]: pgmap v61: 129 pgs: 129 active+clean; 7.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 571 MiB/s wr, 174 op/s 2024-03-19T02:55:26.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:26 smithi155 ceph-mon[35804]: pgmap v62: 129 pgs: 129 active+clean; 7.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 512 MiB/s wr, 156 op/s 2024-03-19T02:55:26.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:26 smithi033 ceph-mon[31973]: pgmap v62: 129 pgs: 129 active+clean; 7.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 512 MiB/s wr, 156 op/s 2024-03-19T02:55:26.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:26 smithi096 ceph-mon[35946]: pgmap v62: 129 pgs: 129 active+clean; 7.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 512 MiB/s wr, 156 op/s 2024-03-19T02:55:26.809 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:55:26.810 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:55:26.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.990+0000 7fac3d51b640 1 -- 172.21.15.33:0/633617349 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac381066a0 msgr2=0x7fac38106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:26.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.990+0000 7fac3d51b640 1 --2- 172.21.15.33:0/633617349 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac381066a0 0x7fac38106a80 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fac28009930 tx=0x7fac2802bc30 comp rx=0 tx=0).stop 2024-03-19T02:55:26.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.990+0000 7fac3d51b640 1 -- 172.21.15.33:0/633617349 shutdown_connections 2024-03-19T02:55:26.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.990+0000 7fac3d51b640 1 --2- 172.21.15.33:0/633617349 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac38100690 0x7fac38100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:26.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.990+0000 7fac3d51b640 1 --2- 172.21.15.33:0/633617349 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac38101050 0x7fac3810d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:26.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.990+0000 7fac3d51b640 1 --2- 172.21.15.33:0/633617349 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac381066a0 0x7fac38106a80 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:26.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.990+0000 7fac3d51b640 1 -- 172.21.15.33:0/633617349 >> 172.21.15.33:0/633617349 conn(0x7fac38077820 msgr2=0x7fac38077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:26.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.990+0000 7fac3d51b640 1 -- 172.21.15.33:0/633617349 shutdown_connections 2024-03-19T02:55:26.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.990+0000 7fac3d51b640 1 -- 172.21.15.33:0/633617349 wait complete. 2024-03-19T02:55:26.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.991+0000 7fac3d51b640 1 Processor -- start 2024-03-19T02:55:26.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.992+0000 7fac3d51b640 1 -- start start 2024-03-19T02:55:26.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.993+0000 7fac3d51b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac38100690 0x7fac3819a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:26.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.993+0000 7fac3d51b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac38101050 0x7fac3819a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:26.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.993+0000 7fac3d51b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac381066a0 0x7fac3819eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:26.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.993+0000 7fac367fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac38101050 0x7fac3819a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:26.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.993+0000 7fac367fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac38101050 0x7fac3819a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55248/0 (socket says 172.21.15.33:55248) 2024-03-19T02:55:26.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.993+0000 7fac367fc640 1 -- 172.21.15.33:0/1349257905 learned_addr learned my addr 172.21.15.33:0/1349257905 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:26.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.993+0000 7fac3d51b640 1 -- 172.21.15.33:0/1349257905 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fac38111ab0 con 0x7fac38101050 2024-03-19T02:55:26.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.993+0000 7fac3d51b640 1 -- 172.21.15.33:0/1349257905 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fac38111930 con 0x7fac38100690 2024-03-19T02:55:26.995 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.993+0000 7fac3d51b640 1 -- 172.21.15.33:0/1349257905 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fac38111c30 con 0x7fac381066a0 2024-03-19T02:55:26.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.994+0000 7fac367fc640 1 -- 172.21.15.33:0/1349257905 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac381066a0 msgr2=0x7fac3819eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:26.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.994+0000 7fac367fc640 1 --2- 172.21.15.33:0/1349257905 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac381066a0 0x7fac3819eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:26.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.994+0000 7fac367fc640 1 -- 172.21.15.33:0/1349257905 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac38100690 msgr2=0x7fac3819a2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-19T02:55:26.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.994+0000 7fac367fc640 1 --2- 172.21.15.33:0/1349257905 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac38100690 0x7fac3819a2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:26.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.994+0000 7fac367fc640 1 -- 172.21.15.33:0/1349257905 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac3819f220 con 0x7fac38101050 2024-03-19T02:55:26.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.994+0000 7fac367fc640 1 --2- 172.21.15.33:0/1349257905 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac38101050 0x7fac3819a810 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fac2c00e9f0 tx=0x7fac2c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:26.996 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.994+0000 7fac1ffff640 1 -- 172.21.15.33:0/1349257905 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2c00cd80 con 0x7fac38101050 2024-03-19T02:55:26.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.994+0000 7fac1ffff640 1 -- 172.21.15.33:0/1349257905 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac2c0108e0 con 0x7fac38101050 2024-03-19T02:55:26.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.995+0000 7fac3d51b640 1 -- 172.21.15.33:0/1349257905 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac3819f510 con 0x7fac38101050 2024-03-19T02:55:26.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.995+0000 7fac3d51b640 1 -- 172.21.15.33:0/1349257905 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac381a6da0 con 0x7fac38101050 2024-03-19T02:55:26.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.995+0000 7fac1ffff640 1 -- 172.21.15.33:0/1349257905 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2c018920 con 0x7fac38101050 2024-03-19T02:55:26.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.998+0000 7fac401e8740 1 Processor -- start 2024-03-19T02:55:26.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.998+0000 7fac1ffff640 1 -- 172.21.15.33:0/1349257905 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fac2c018ac0 con 0x7fac38101050 2024-03-19T02:55:26.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.998+0000 7fac401e8740 1 -- start start 2024-03-19T02:55:26.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.998+0000 7fac401e8740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d55268ca30 0x55d55268ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.998+0000 7fac401e8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d55268d350 0x55d5526959f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.999+0000 7fac401e8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d552695f30 0x55d552698320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.999+0000 7fac401e8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d5525f0500 con 0x55d55268d350 2024-03-19T02:55:27.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.999+0000 7fac401e8740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55d552598be0 con 0x55d55268ca30 2024-03-19T02:55:27.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.999+0000 7fac401e8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d5525ec5a0 con 0x55d552695f30 2024-03-19T02:55:27.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.999+0000 7fac36ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d55268d350 0x55d5526959f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.999+0000 7fac36ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d55268d350 0x55d5526959f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55250/0 (socket says 172.21.15.33:55250) 2024-03-19T02:55:27.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.999+0000 7fac36ffd640 1 -- 172.21.15.33:0/30355327 learned_addr learned my addr 172.21.15.33:0/30355327 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:27.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.999+0000 7fac377fe640 1 --2- 172.21.15.33:0/30355327 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d55268ca30 0x55d55268ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.001 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.999+0000 7fac1ffff640 1 --2- 172.21.15.33:0/1349257905 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fac200a2ab0 0x7fac200a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.999+0000 7fac367fc640 1 --2- 172.21.15.33:0/30355327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d552695f30 0x55d552698320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.999+0000 7fac377fe640 1 --2- 172.21.15.33:0/1349257905 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fac200a2ab0 0x7fac200a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:26.999+0000 7fac1ffff640 1 -- 172.21.15.33:0/1349257905 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fac2c0cf610 con 0x7fac38101050 2024-03-19T02:55:27.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.000+0000 7fac36ffd640 1 -- 172.21.15.33:0/30355327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d552695f30 msgr2=0x55d552698320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:27.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.000+0000 7fac36ffd640 1 --2- 172.21.15.33:0/30355327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d552695f30 0x55d552698320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.000+0000 7fac36ffd640 1 -- 172.21.15.33:0/30355327 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d55268ca30 msgr2=0x55d55268ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:27.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.000+0000 7fac36ffd640 1 --2- 172.21.15.33:0/30355327 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d55268ca30 0x55d55268ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.000+0000 7fac36ffd640 1 -- 172.21.15.33:0/30355327 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5525ba000 con 0x55d55268d350 2024-03-19T02:55:27.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.000+0000 7fac36ffd640 1 --2- 172.21.15.33:0/30355327 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d55268d350 0x55d5526959f0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fac28000c00 tx=0x7fac28009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b86923229b6199d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:27.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.000+0000 7fac377fe640 1 --2- 172.21.15.33:0/1349257905 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fac200a2ab0 0x7fac200a4f70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fac30009fd0 tx=0x7fac30004bc0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:27.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.000+0000 7fac1d7fa640 1 -- 172.21.15.33:0/30355327 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac28042020 con 0x55d55268d350 2024-03-19T02:55:27.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.000+0000 7fac1d7fa640 1 -- 172.21.15.33:0/30355327 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac2802cc90 con 0x55d55268d350 2024-03-19T02:55:27.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.001+0000 7fac1d7fa640 1 -- 172.21.15.33:0/30355327 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac28003a80 con 0x55d55268d350 2024-03-19T02:55:27.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.001+0000 7fac401e8740 1 -- 172.21.15.33:0/30355327 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d55268d350 msgr2=0x55d5526959f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:27.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.001+0000 7fac401e8740 1 --2- 172.21.15.33:0/30355327 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d55268d350 0x55d5526959f0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fac28000c00 tx=0x7fac28009520 comp rx=0 tx=0).stop 2024-03-19T02:55:27.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.002+0000 7fac401e8740 1 -- 172.21.15.33:0/30355327 shutdown_connections 2024-03-19T02:55:27.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.002+0000 7fac401e8740 1 --2- 172.21.15.33:0/30355327 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d552695f30 0x55d552698320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.002+0000 7fac401e8740 1 --2- 172.21.15.33:0/30355327 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d55268d350 0x55d5526959f0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.002+0000 7fac401e8740 1 --2- 172.21.15.33:0/30355327 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d55268ca30 0x55d55268ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.002+0000 7fac401e8740 1 -- 172.21.15.33:0/30355327 >> 172.21.15.33:0/30355327 conn(0x55d552687160 msgr2=0x55d552687540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:27.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.002+0000 7fac401e8740 1 -- 172.21.15.33:0/30355327 shutdown_connections 2024-03-19T02:55:27.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.002+0000 7fac401e8740 1 -- 172.21.15.33:0/30355327 wait complete. 2024-03-19T02:55:27.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.002+0000 7fac401e8740 1 -- start start 2024-03-19T02:55:27.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.003+0000 7fac401e8740 1 Processor -- start 2024-03-19T02:55:27.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.003+0000 7fac401e8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d55268ca30 0x55d552692fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.003+0000 7fac401e8740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d552695f30 0x55d552693500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.004+0000 7fac401e8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d552693a40 0x55d5526a2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.004+0000 7fac36ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d55268ca30 0x55d552692fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.004+0000 7fac36ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d55268ca30 0x55d552692fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55260/0 (socket says 172.21.15.33:55260) 2024-03-19T02:55:27.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.004+0000 7fac367fc640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d552695f30 0x55d552693500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.004+0000 7fac367fc640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d552695f30 0x55d552693500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:44946/0 (socket says 172.21.15.33:44946) 2024-03-19T02:55:27.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.004+0000 7fac401e8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d552698cc0 con 0x55d55268ca30 2024-03-19T02:55:27.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.004+0000 7fac401e8740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55d5525ac160 con 0x55d552695f30 2024-03-19T02:55:27.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.004+0000 7fac36ffd640 1 -- 172.21.15.33:0/373159201 learned_addr learned my addr 172.21.15.33:0/373159201 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:27.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.004+0000 7fac401e8740 1 -- 172.21.15.33:0/373159201 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d552698e40 con 0x55d552693a40 2024-03-19T02:55:27.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.004+0000 7fac377fe640 1 --2- 172.21.15.33:0/373159201 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d552693a40 0x55d5526a2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.004+0000 7fac401e8740 10 client.0 _ll_register_callbacks cb 0x55d5525e8a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:55:27.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.005+0000 7fac367fc640 1 -- 172.21.15.33:0/373159201 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d552693a40 msgr2=0x55d5526a2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:27.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.005+0000 7fac367fc640 1 --2- 172.21.15.33:0/373159201 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d552693a40 0x55d5526a2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.005+0000 7fac367fc640 1 -- 172.21.15.33:0/373159201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d55268ca30 msgr2=0x55d552692fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:27.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.005+0000 7fac367fc640 1 --2- 172.21.15.33:0/373159201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d55268ca30 0x55d552692fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.005+0000 7fac367fc640 1 -- 172.21.15.33:0/373159201 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5523ce7d0 con 0x55d552695f30 2024-03-19T02:55:27.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.005+0000 7fac36ffd640 1 --2- 172.21.15.33:0/373159201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d55268ca30 0x55d552692fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:27.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.005+0000 7fac367fc640 1 --2- 172.21.15.33:0/373159201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d552695f30 0x55d552693500 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fac2c0cc9b0 tx=0x7fac2c09bfc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccdab40ca15dd3bb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:27.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.005+0000 7fac1aff5640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T02:55:27.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.005+0000 7fac1aff5640 1 -- 172.21.15.33:0/373159201 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2c09fa10 con 0x55d552695f30 2024-03-19T02:55:27.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.005+0000 7fac401e8740 1 -- 172.21.15.33:0/373159201 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac2802cc90 con 0x55d552695f30 2024-03-19T02:55:27.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.005+0000 7fac1aff5640 1 -- 172.21.15.33:0/373159201 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fac2c097f30 con 0x55d552695f30 2024-03-19T02:55:27.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.005+0000 7fac1aff5640 1 -- 172.21.15.33:0/373159201 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fac2c0b3030 con 0x55d552695f30 2024-03-19T02:55:27.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.005+0000 7fac401e8740 10 client.25898 fetch_fsmap learned FSMap version 25 2024-03-19T02:55:27.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.005+0000 7fac401e8740 1 -- 172.21.15.33:0/373159201 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d5523043f0 con 0x55d552695f30 2024-03-19T02:55:27.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.006+0000 7fac1aff5640 1 -- 172.21.15.33:0/373159201 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac2802cc90 con 0x55d552695f30 2024-03-19T02:55:27.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.006+0000 7fac1aff5640 1 -- 172.21.15.33:0/373159201 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fac2c09ad10 con 0x55d552695f30 2024-03-19T02:55:27.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.006+0000 7fac401e8740 10 client.25898 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:55:27.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.006+0000 7fac401e8740 10 client.25898 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:55:27.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.006+0000 7fac401e8740 20 client.25898 populate_metadata read hostname 'smithi033' 2024-03-19T02:55:27.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.006+0000 7fac401e8740 1 --2- 172.21.15.33:0/373159201 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55d5526a6660 0x55d5526a8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.006+0000 7fac401e8740 4 client.25898 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:55:27.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.006+0000 7fac401e8740 1 -- 172.21.15.33:0/373159201 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d5525fda40 con 0x55d5526a6660 2024-03-19T02:55:27.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.007+0000 7fac36ffd640 1 --2- 172.21.15.33:0/373159201 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55d5526a6660 0x55d5526a8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.009+0000 7fac36ffd640 1 --2- 172.21.15.33:0/373159201 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55d5526a6660 0x55d5526a8a70 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b5a1ee0e5dc6a86 server_cookie=be1b39d4e27986c3 in_seq=0 out_seq=0 2024-03-19T02:55:27.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.009+0000 7fac1aff5640 10 client.25898 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:55:27.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.010+0000 7fac1aff5640 1 -- 172.21.15.33:0/373159201 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d5525fda40 con 0x55d5526a6660 2024-03-19T02:55:27.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.010+0000 7fac1aff5640 10 client.25898 handle_command_reply: tid=0 2024-03-19T02:55:27.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.010+0000 7fac401e8740 1 client.25898 shutdown 2024-03-19T02:55:27.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.011+0000 7fac401e8740 1 -- 172.21.15.33:0/373159201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d552695f30 msgr2=0x55d552693500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:27.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.011+0000 7fac401e8740 1 --2- 172.21.15.33:0/373159201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d552695f30 0x55d552693500 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fac2c0cc9b0 tx=0x7fac2c09bfc0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.011+0000 7fac401e8740 1 -- 172.21.15.33:0/373159201 shutdown_connections 2024-03-19T02:55:27.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.011+0000 7fac401e8740 1 --2- 172.21.15.33:0/373159201 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55d5526a6660 0x55d5526a8a70 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.011+0000 7fac401e8740 1 --2- 172.21.15.33:0/373159201 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d552693a40 0x55d5526a2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.011+0000 7fac401e8740 1 --2- 172.21.15.33:0/373159201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d552695f30 0x55d552693500 secure :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fac2c0cc9b0 tx=0x7fac2c09bfc0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.011+0000 7fac401e8740 1 --2- 172.21.15.33:0/373159201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d55268ca30 0x55d552692fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.011+0000 7fac401e8740 1 -- 172.21.15.33:0/373159201 >> 172.21.15.33:0/373159201 conn(0x55d552687160 msgr2=0x55d552688eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:27.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.011+0000 7fac1aff5640 0 client.25898 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:55:27.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.012+0000 7fac401e8740 1 -- 172.21.15.33:0/373159201 shutdown_connections 2024-03-19T02:55:27.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.012+0000 7fac401e8740 1 -- 172.21.15.33:0/373159201 wait complete. 2024-03-19T02:55:27.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.012+0000 7fac401e8740 20 client.25898 trim_cache size 0 max 16384 2024-03-19T02:55:27.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.040+0000 7fac401e8740 1 Processor -- start 2024-03-19T02:55:27.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.040+0000 7fac401e8740 1 -- start start 2024-03-19T02:55:27.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.041+0000 7fac401e8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d55268ca30 0x55d552688ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.041+0000 7fac401e8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d552693a40 0x55d5526ae340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.041+0000 7fac401e8740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d5526ae880 0x55d5526a1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.041+0000 7fac401e8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fac381b3640 con 0x55d55268ca30 2024-03-19T02:55:27.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.041+0000 7fac401e8740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fac381b0f10 con 0x55d5526ae880 2024-03-19T02:55:27.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.041+0000 7fac401e8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fac381aed40 con 0x55d552693a40 2024-03-19T02:55:27.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.041+0000 7fac36ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d55268ca30 0x55d552688ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.041+0000 7fac36ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d55268ca30 0x55d552688ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55272/0 (socket says 172.21.15.33:55272) 2024-03-19T02:55:27.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.041+0000 7fac36ffd640 1 -- 172.21.15.33:0/3844931016 learned_addr learned my addr 172.21.15.33:0/3844931016 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:27.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.041+0000 7fac367fc640 1 --2- 172.21.15.33:0/3844931016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d552693a40 0x55d5526ae340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.041+0000 7fac36ffd640 1 -- 172.21.15.33:0/3844931016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d552693a40 msgr2=0x55d5526ae340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:27.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.041+0000 7fac36ffd640 1 --2- 172.21.15.33:0/3844931016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d552693a40 0x55d5526ae340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.041+0000 7fac36ffd640 1 -- 172.21.15.33:0/3844931016 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d5526ae880 msgr2=0x55d5526a1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:27.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.041+0000 7fac36ffd640 1 --2- 172.21.15.33:0/3844931016 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d5526ae880 0x55d5526a1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.041+0000 7fac36ffd640 1 -- 172.21.15.33:0/3844931016 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5522c40f0 con 0x55d55268ca30 2024-03-19T02:55:27.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.042+0000 7fac36ffd640 1 --2- 172.21.15.33:0/3844931016 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d55268ca30 0x55d552688ca0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fac2802c240 tx=0x7fac2802c150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7294e4ab500f99da server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:27.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.042+0000 7fac1cff9640 1 -- 172.21.15.33:0/3844931016 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac28042070 con 0x55d55268ca30 2024-03-19T02:55:27.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.042+0000 7fac1cff9640 1 -- 172.21.15.33:0/3844931016 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac2802c5d0 con 0x55d55268ca30 2024-03-19T02:55:27.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.042+0000 7fac1cff9640 1 -- 172.21.15.33:0/3844931016 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac28034b30 con 0x55d55268ca30 2024-03-19T02:55:27.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.043+0000 7fac401e8740 1 -- 172.21.15.33:0/3844931016 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d55268ca30 msgr2=0x55d552688ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:27.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.043+0000 7fac401e8740 1 --2- 172.21.15.33:0/3844931016 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d55268ca30 0x55d552688ca0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fac2802c240 tx=0x7fac2802c150 comp rx=0 tx=0).stop 2024-03-19T02:55:27.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.043+0000 7fac401e8740 1 -- 172.21.15.33:0/3844931016 shutdown_connections 2024-03-19T02:55:27.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.043+0000 7fac401e8740 1 --2- 172.21.15.33:0/3844931016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d552693a40 0x55d5526ae340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.043+0000 7fac401e8740 1 --2- 172.21.15.33:0/3844931016 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d5526ae880 0x55d5526a1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.043+0000 7fac401e8740 1 --2- 172.21.15.33:0/3844931016 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d55268ca30 0x55d552688ca0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.043+0000 7fac401e8740 1 -- 172.21.15.33:0/3844931016 >> 172.21.15.33:0/3844931016 conn(0x55d552687160 msgr2=0x55d552688590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:27.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.043+0000 7fac401e8740 1 -- 172.21.15.33:0/3844931016 shutdown_connections 2024-03-19T02:55:27.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.043+0000 7fac401e8740 1 -- 172.21.15.33:0/3844931016 wait complete. 2024-03-19T02:55:27.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.043+0000 7fac401e8740 1 -- start start 2024-03-19T02:55:27.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.044+0000 7fac401e8740 1 Processor -- start 2024-03-19T02:55:27.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.044+0000 7fac401e8740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d552693a40 0x55d55269e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.044+0000 7fac401e8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d5526ae880 0x55d552699670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.044+0000 7fac401e8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d552699bb0 0x55d55269a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.045+0000 7fac401e8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fac381af3d0 con 0x55d552699bb0 2024-03-19T02:55:27.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.045+0000 7fac401e8740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fac381b0840 con 0x55d552693a40 2024-03-19T02:55:27.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.045+0000 7fac401e8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fac381afaa0 con 0x55d5526ae880 2024-03-19T02:55:27.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.045+0000 7fac377fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d552699bb0 0x55d55269a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.045+0000 7fac377fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d552699bb0 0x55d55269a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55278/0 (socket says 172.21.15.33:55278) 2024-03-19T02:55:27.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.045+0000 7fac401e8740 10 client.0 _ll_register_callbacks cb 0x7fac2c09b690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:55:27.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.045+0000 7fac377fe640 1 -- 172.21.15.33:0/1253876940 learned_addr learned my addr 172.21.15.33:0/1253876940 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:27.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.045+0000 7fac367fc640 1 --2- 172.21.15.33:0/1253876940 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d5526ae880 0x55d552699670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.045+0000 7fac36ffd640 1 --2- 172.21.15.33:0/1253876940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d552693a40 0x55d55269e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.045+0000 7fac367fc640 1 -- 172.21.15.33:0/1253876940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d552693a40 msgr2=0x55d55269e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:27.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.045+0000 7fac367fc640 1 --2- 172.21.15.33:0/1253876940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d552693a40 0x55d55269e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.045+0000 7fac367fc640 1 -- 172.21.15.33:0/1253876940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d552699bb0 msgr2=0x55d55269a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:27.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.045+0000 7fac367fc640 1 --2- 172.21.15.33:0/1253876940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d552699bb0 0x55d55269a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.045+0000 7fac367fc640 1 -- 172.21.15.33:0/1253876940 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d55270b9a0 con 0x55d5526ae880 2024-03-19T02:55:27.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.046+0000 7fac367fc640 1 --2- 172.21.15.33:0/1253876940 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d5526ae880 0x55d552699670 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x55d552694080 tx=0x7fac2c000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd10fadf974c9800 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:27.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.046+0000 7fac1aff5640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-19T02:55:27.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.046+0000 7fac1aff5640 1 -- 172.21.15.33:0/1253876940 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2c09fa10 con 0x55d5526ae880 2024-03-19T02:55:27.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.046+0000 7fac1aff5640 1 -- 172.21.15.33:0/1253876940 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fac2c0b4070 con 0x55d5526ae880 2024-03-19T02:55:27.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.046+0000 7fac401e8740 1 -- 172.21.15.33:0/1253876940 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac2802c5d0 con 0x55d5526ae880 2024-03-19T02:55:27.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.046+0000 7fac1aff5640 1 -- 172.21.15.33:0/1253876940 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fac2c09b220 con 0x55d5526ae880 2024-03-19T02:55:27.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.046+0000 7fac401e8740 10 client.25768 fetch_fsmap learned FSMap version 25 2024-03-19T02:55:27.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.046+0000 7fac401e8740 1 -- 172.21.15.33:0/1253876940 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d55270cc50 con 0x55d5526ae880 2024-03-19T02:55:27.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.047+0000 7fac1aff5640 1 -- 172.21.15.33:0/1253876940 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac2c0cf200 con 0x55d5526ae880 2024-03-19T02:55:27.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.047+0000 7fac1aff5640 1 -- 172.21.15.33:0/1253876940 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fac2c097610 con 0x55d5526ae880 2024-03-19T02:55:27.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.047+0000 7fac401e8740 10 client.25768 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:55:27.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.047+0000 7fac401e8740 10 client.25768 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:55:27.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.047+0000 7fac401e8740 20 client.25768 populate_metadata read hostname 'smithi033' 2024-03-19T02:55:27.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.047+0000 7fac401e8740 1 --2- 172.21.15.33:0/1253876940 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55d55270d230 0x55d55270f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.047+0000 7fac401e8740 4 client.25768 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:55:27.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.047+0000 7fac401e8740 1 -- 172.21.15.33:0/1253876940 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fac2802c5d0 con 0x55d55270d230 2024-03-19T02:55:27.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.047+0000 7fac36ffd640 1 --2- 172.21.15.33:0/1253876940 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55d55270d230 0x55d55270f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.048+0000 7fac1aff5640 10 client.25768 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:55:27.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.048+0000 7fac36ffd640 1 --2- 172.21.15.33:0/1253876940 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55d55270d230 0x55d55270f620 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c3988c6f9a0b34f server_cookie=f9b60bf0d6bf98f1 in_seq=0 out_seq=0 2024-03-19T02:55:27.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.048+0000 7fac1aff5640 1 -- 172.21.15.33:0/1253876940 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7fac2802c5d0 con 0x55d55270d230 2024-03-19T02:55:27.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.048+0000 7fac1aff5640 10 client.25768 handle_command_reply: tid=0 2024-03-19T02:55:27.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.048+0000 7fac401e8740 1 client.25768 shutdown 2024-03-19T02:55:27.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.048+0000 7fac401e8740 1 -- 172.21.15.33:0/1253876940 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d5526ae880 msgr2=0x55d552699670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:27.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.048+0000 7fac401e8740 1 --2- 172.21.15.33:0/1253876940 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d5526ae880 0x55d552699670 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x55d552694080 tx=0x7fac2c000bc0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.048+0000 7fac401e8740 1 -- 172.21.15.33:0/1253876940 shutdown_connections 2024-03-19T02:55:27.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.048+0000 7fac401e8740 1 --2- 172.21.15.33:0/1253876940 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55d55270d230 0x55d55270f620 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.048+0000 7fac401e8740 1 --2- 172.21.15.33:0/1253876940 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d5526ae880 0x55d552699670 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.048+0000 7fac401e8740 1 --2- 172.21.15.33:0/1253876940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d552699bb0 0x55d55269a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.048+0000 7fac401e8740 1 --2- 172.21.15.33:0/1253876940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d552693a40 0x55d55269e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.048+0000 7fac401e8740 1 -- 172.21.15.33:0/1253876940 >> 172.21.15.33:0/1253876940 conn(0x55d552687160 msgr2=0x55d552688590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:27.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.048+0000 7fac1aff5640 0 client.25768 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:55:27.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.048+0000 7fac401e8740 1 -- 172.21.15.33:0/1253876940 shutdown_connections 2024-03-19T02:55:27.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.048+0000 7fac401e8740 1 -- 172.21.15.33:0/1253876940 wait complete. 2024-03-19T02:55:27.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.049+0000 7fac401e8740 20 client.25768 trim_cache size 0 max 16384 2024-03-19T02:55:27.053 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:55:27.053 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-03-19T02:55:27.053 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-03-19T02:55:27.053 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:55:27.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.050+0000 7fac3d51b640 1 -- 172.21.15.33:0/1349257905 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fac200a2ab0 msgr2=0x7fac200a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:27.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.050+0000 7fac3d51b640 1 --2- 172.21.15.33:0/1349257905 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fac200a2ab0 0x7fac200a4f70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fac30009fd0 tx=0x7fac30004bc0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.050+0000 7fac3d51b640 1 -- 172.21.15.33:0/1349257905 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac38101050 msgr2=0x7fac3819a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:27.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.050+0000 7fac3d51b640 1 --2- 172.21.15.33:0/1349257905 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac38101050 0x7fac3819a810 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fac2c00e9f0 tx=0x7fac2c00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.050+0000 7fac3d51b640 1 -- 172.21.15.33:0/1349257905 shutdown_connections 2024-03-19T02:55:27.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.050+0000 7fac3d51b640 1 --2- 172.21.15.33:0/1349257905 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fac200a2ab0 0x7fac200a4f70 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.050+0000 7fac3d51b640 1 --2- 172.21.15.33:0/1349257905 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac381066a0 0x7fac3819eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.050+0000 7fac3d51b640 1 --2- 172.21.15.33:0/1349257905 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fac38101050 0x7fac3819a810 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.050+0000 7fac3d51b640 1 --2- 172.21.15.33:0/1349257905 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac38100690 0x7fac3819a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.050+0000 7fac3d51b640 1 -- 172.21.15.33:0/1349257905 >> 172.21.15.33:0/1349257905 conn(0x7fac38077820 msgr2=0x7fac38102cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:27.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.051+0000 7fac3d51b640 1 -- 172.21.15.33:0/1349257905 shutdown_connections 2024-03-19T02:55:27.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.051+0000 7fac3d51b640 1 -- 172.21.15.33:0/1349257905 wait complete. 2024-03-19T02:55:27.074 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:55:27.074 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:55:27.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.235+0000 7f8126634640 1 -- 172.21.15.33:0/2105669381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8120105600 msgr2=0x7f81201059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:27.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.235+0000 7f8126634640 1 --2- 172.21.15.33:0/2105669381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8120105600 0x7f81201059e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f8110009900 tx=0x7f811002bce0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.235+0000 7f8126634640 1 -- 172.21.15.33:0/2105669381 shutdown_connections 2024-03-19T02:55:27.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.235+0000 7f8126634640 1 --2- 172.21.15.33:0/2105669381 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81201010c0 0x7f812010d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.235+0000 7f8126634640 1 --2- 172.21.15.33:0/2105669381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8120100720 0x7f8120100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.235+0000 7f8126634640 1 --2- 172.21.15.33:0/2105669381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8120105600 0x7f81201059e0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.236+0000 7f8126634640 1 -- 172.21.15.33:0/2105669381 >> 172.21.15.33:0/2105669381 conn(0x7f8120077850 msgr2=0x7f8120077c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:27.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.236+0000 7f8126634640 1 -- 172.21.15.33:0/2105669381 shutdown_connections 2024-03-19T02:55:27.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.236+0000 7f8126634640 1 -- 172.21.15.33:0/2105669381 wait complete. 2024-03-19T02:55:27.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.236+0000 7f8126634640 1 Processor -- start 2024-03-19T02:55:27.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.236+0000 7f8126634640 1 -- start start 2024-03-19T02:55:27.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.237+0000 7f8126634640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8120100720 0x7f8120198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.237+0000 7f8126634640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81201010c0 0x7f81201985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.237+0000 7f8126634640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8120105600 0x7f812019c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.237+0000 7f8126634640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8120110a70 con 0x7f8120100720 2024-03-19T02:55:27.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.237+0000 7f8126634640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f81201108f0 con 0x7f8120105600 2024-03-19T02:55:27.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.237+0000 7f8126634640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8120110bf0 con 0x7f81201010c0 2024-03-19T02:55:27.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.237+0000 7f811ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8120100720 0x7f8120198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.237+0000 7f811ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8120100720 0x7f8120198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55308/0 (socket says 172.21.15.33:55308) 2024-03-19T02:55:27.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.237+0000 7f811ffff640 1 -- 172.21.15.33:0/1573335315 learned_addr learned my addr 172.21.15.33:0/1573335315 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:27.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.237+0000 7f811ffff640 1 -- 172.21.15.33:0/1573335315 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81201010c0 msgr2=0x7f81201985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:27.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.237+0000 7f811ffff640 1 --2- 172.21.15.33:0/1573335315 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81201010c0 0x7f81201985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.237+0000 7f811ffff640 1 -- 172.21.15.33:0/1573335315 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8120105600 msgr2=0x7f812019c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:27.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.237+0000 7f811ffff640 1 --2- 172.21.15.33:0/1573335315 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8120105600 0x7f812019c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.237+0000 7f811ffff640 1 -- 172.21.15.33:0/1573335315 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f812019d070 con 0x7f8120100720 2024-03-19T02:55:27.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.238+0000 7f811ffff640 1 --2- 172.21.15.33:0/1573335315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8120100720 0x7f8120198080 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f8110005ab0 tx=0x7f811002d920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:27.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.238+0000 7f811d7fa640 1 -- 172.21.15.33:0/1573335315 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8110043020 con 0x7f8120100720 2024-03-19T02:55:27.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.238+0000 7f8126634640 1 -- 172.21.15.33:0/1573335315 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f812019d300 con 0x7f8120100720 2024-03-19T02:55:27.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.238+0000 7f811d7fa640 1 -- 172.21.15.33:0/1573335315 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f811002dd20 con 0x7f8120100720 2024-03-19T02:55:27.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.238+0000 7f811d7fa640 1 -- 172.21.15.33:0/1573335315 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8110003b10 con 0x7f8120100720 2024-03-19T02:55:27.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.238+0000 7f8126634640 1 -- 172.21.15.33:0/1573335315 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81201a4bf0 con 0x7f8120100720 2024-03-19T02:55:27.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.240+0000 7f8126634640 1 -- 172.21.15.33:0/1573335315 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8120067d80 con 0x7f8120100720 2024-03-19T02:55:27.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.241+0000 7f811d7fa640 1 -- 172.21.15.33:0/1573335315 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f8110046020 con 0x7f8120100720 2024-03-19T02:55:27.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.242+0000 7f811d7fa640 1 --2- 172.21.15.33:0/1573335315 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f81080a2970 0x7f81080a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.242+0000 7f811d7fa640 1 -- 172.21.15.33:0/1573335315 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f81100f1260 con 0x7f8120100720 2024-03-19T02:55:27.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.242+0000 7f811f7fe640 1 --2- 172.21.15.33:0/1573335315 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f81080a2970 0x7f81080a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.243+0000 7f811f7fe640 1 --2- 172.21.15.33:0/1573335315 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f81080a2970 0x7f81080a4e30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8114005fd0 tx=0x7f8114005850 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:27.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.248+0000 7f811d7fa640 1 -- 172.21.15.33:0/1573335315 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f811004d020 con 0x7f8120100720 2024-03-19T02:55:27.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.597+0000 7f8126634640 1 -- 172.21.15.33:0/1573335315 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8120109be0 con 0x7f8120100720 2024-03-19T02:55:27.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.598+0000 7f811d7fa640 1 -- 172.21.15.33:0/1573335315 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11867 (secure 0 0 0) 0x7f81100bdd60 con 0x7f8120100720 2024-03-19T02:55:27.599 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:55:27.599 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:50:52.201798+0000","modified":"2024-03-19T02:53:11.748697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24488,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/3731232169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":3731232169},{"type":"v1","addr":"172.21.15.96:6839","nonce":3731232169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24457,24488,24509]},"id":1}]} 2024-03-19T02:55:27.599 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-03-19T02:55:27.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.600+0000 7f8126634640 1 -- 172.21.15.33:0/1573335315 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f81080a2970 msgr2=0x7f81080a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:27.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.600+0000 7f8126634640 1 --2- 172.21.15.33:0/1573335315 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f81080a2970 0x7f81080a4e30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8114005fd0 tx=0x7f8114005850 comp rx=0 tx=0).stop 2024-03-19T02:55:27.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.601+0000 7f8126634640 1 -- 172.21.15.33:0/1573335315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8120100720 msgr2=0x7f8120198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:27.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.601+0000 7f8126634640 1 --2- 172.21.15.33:0/1573335315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8120100720 0x7f8120198080 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f8110005ab0 tx=0x7f811002d920 comp rx=0 tx=0).stop 2024-03-19T02:55:27.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.601+0000 7f8126634640 1 -- 172.21.15.33:0/1573335315 shutdown_connections 2024-03-19T02:55:27.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.601+0000 7f8126634640 1 --2- 172.21.15.33:0/1573335315 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f81080a2970 0x7f81080a4e30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.601+0000 7f8126634640 1 --2- 172.21.15.33:0/1573335315 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81201010c0 0x7f81201985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.601+0000 7f8126634640 1 --2- 172.21.15.33:0/1573335315 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8120105600 0x7f812019c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.601+0000 7f8126634640 1 --2- 172.21.15.33:0/1573335315 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8120100720 0x7f8120198080 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.601+0000 7f8126634640 1 -- 172.21.15.33:0/1573335315 >> 172.21.15.33:0/1573335315 conn(0x7f8120077850 msgr2=0x7f8120069ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:27.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.601+0000 7f8126634640 1 -- 172.21.15.33:0/1573335315 shutdown_connections 2024-03-19T02:55:27.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.601+0000 7f8126634640 1 -- 172.21.15.33:0/1573335315 wait complete. 2024-03-19T02:55:27.620 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:55:27.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.794+0000 7f18aedcd640 1 -- 172.21.15.33:0/2453370189 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18a8101060 msgr2=0x7f18a810d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:27.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.795+0000 7f18aedcd640 1 --2- 172.21.15.33:0/2453370189 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18a8101060 0x7f18a810d570 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f18a0009cd0 tx=0x7f18a002c100 comp rx=0 tx=0).stop 2024-03-19T02:55:27.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.795+0000 7f18aedcd640 1 -- 172.21.15.33:0/2453370189 shutdown_connections 2024-03-19T02:55:27.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.795+0000 7f18aedcd640 1 --2- 172.21.15.33:0/2453370189 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18a8101060 0x7f18a810d570 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.795+0000 7f18aedcd640 1 --2- 172.21.15.33:0/2453370189 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f18a81006a0 0x7f18a8100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.795+0000 7f18aedcd640 1 --2- 172.21.15.33:0/2453370189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18a81066b0 0x7f18a8106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.795+0000 7f18aedcd640 1 -- 172.21.15.33:0/2453370189 >> 172.21.15.33:0/2453370189 conn(0x7f18a80778d0 msgr2=0x7f18a8077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:27.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.796+0000 7f18aedcd640 1 -- 172.21.15.33:0/2453370189 shutdown_connections 2024-03-19T02:55:27.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.796+0000 7f18aedcd640 1 -- 172.21.15.33:0/2453370189 wait complete. 2024-03-19T02:55:27.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.797+0000 7f18aedcd640 1 Processor -- start 2024-03-19T02:55:27.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.797+0000 7f18aedcd640 1 -- start start 2024-03-19T02:55:27.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.798+0000 7f18aedcd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f18a81006a0 0x7f18a819ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.798+0000 7f18aedcd640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18a8101060 0x7f18a819bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.798+0000 7f18aedcd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18a81066b0 0x7f18a8195b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.798+0000 7f18aedcd640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f18a810f860 con 0x7f18a81066b0 2024-03-19T02:55:27.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.798+0000 7f18aedcd640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f18a810f6e0 con 0x7f18a81006a0 2024-03-19T02:55:27.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.798+0000 7f18acb69640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f18a81006a0 0x7f18a819ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.798+0000 7f18a7fff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18a8101060 0x7f18a819bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.798+0000 7f18a7fff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18a8101060 0x7f18a819bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:60182/0 (socket says 172.21.15.33:60182) 2024-03-19T02:55:27.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.798+0000 7f18a7fff640 1 -- 172.21.15.33:0/1321194428 learned_addr learned my addr 172.21.15.33:0/1321194428 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:27.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.798+0000 7f18aedcd640 1 -- 172.21.15.33:0/1321194428 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f18a810f9e0 con 0x7f18a8101060 2024-03-19T02:55:27.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.798+0000 7f18ad36a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18a81066b0 0x7f18a8195b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.799+0000 7f18a7fff640 1 -- 172.21.15.33:0/1321194428 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f18a81006a0 msgr2=0x7f18a819ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:27.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.799+0000 7f18a7fff640 1 --2- 172.21.15.33:0/1321194428 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f18a81006a0 0x7f18a819ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.799+0000 7f18a7fff640 1 -- 172.21.15.33:0/1321194428 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18a81066b0 msgr2=0x7f18a8195b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:27.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.799+0000 7f18a7fff640 1 --2- 172.21.15.33:0/1321194428 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18a81066b0 0x7f18a8195b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.799+0000 7f18a7fff640 1 -- 172.21.15.33:0/1321194428 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18a81963e0 con 0x7f18a8101060 2024-03-19T02:55:27.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.799+0000 7f18acb69640 1 --2- 172.21.15.33:0/1321194428 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f18a81006a0 0x7f18a819ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:27.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.799+0000 7f18ad36a640 1 --2- 172.21.15.33:0/1321194428 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18a81066b0 0x7f18a8195b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:55:27.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.799+0000 7f18a7fff640 1 --2- 172.21.15.33:0/1321194428 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18a8101060 0x7f18a819bfa0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f189400ea60 tx=0x7f189400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:27.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.799+0000 7f18a5ffb640 1 -- 172.21.15.33:0/1321194428 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f189400cde0 con 0x7f18a8101060 2024-03-19T02:55:27.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.799+0000 7f18a5ffb640 1 -- 172.21.15.33:0/1321194428 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f18940035f0 con 0x7f18a8101060 2024-03-19T02:55:27.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.799+0000 7f18a5ffb640 1 -- 172.21.15.33:0/1321194428 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f189400cde0 con 0x7f18a8101060 2024-03-19T02:55:27.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.799+0000 7f18aedcd640 1 -- 172.21.15.33:0/1321194428 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18a81966d0 con 0x7f18a8101060 2024-03-19T02:55:27.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.800+0000 7f18aedcd640 1 -- 172.21.15.33:0/1321194428 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18a81a29e0 con 0x7f18a8101060 2024-03-19T02:55:27.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.802+0000 7f18a5ffb640 1 -- 172.21.15.33:0/1321194428 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f18940037a0 con 0x7f18a8101060 2024-03-19T02:55:27.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.803+0000 7f18b19bb740 1 Processor -- start 2024-03-19T02:55:27.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.803+0000 7f18b19bb740 1 -- start start 2024-03-19T02:55:27.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.803+0000 7f18b19bb740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5f83a50e0 0x55b5f83a54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.803+0000 7f18b19bb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5f83a5a90 0x55b5f83b6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.803+0000 7f18b19bb740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5f83b6fe0 0x55b5f83b93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.803+0000 7f18b19bb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b5f83084b0 con 0x55b5f83a5a90 2024-03-19T02:55:27.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.803+0000 7f18b19bb740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55b5f82b0d70 con 0x55b5f83b6fe0 2024-03-19T02:55:27.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.803+0000 7f18b19bb740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b5f8304550 con 0x55b5f83a50e0 2024-03-19T02:55:27.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.804+0000 7f18a7fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5f83b6fe0 0x55b5f83b93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.804+0000 7f18ad36a640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5f83a50e0 0x55b5f83a54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.804+0000 7f18ad36a640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5f83a50e0 0x55b5f83a54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:60190/0 (socket says 172.21.15.33:60190) 2024-03-19T02:55:27.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.804+0000 7f18ad36a640 1 -- 172.21.15.33:0/2182888507 learned_addr learned my addr 172.21.15.33:0/2182888507 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:27.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.804+0000 7f18acb69640 1 --2- 172.21.15.33:0/2182888507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5f83a5a90 0x55b5f83b6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.804+0000 7f18a5ffb640 1 --2- 172.21.15.33:0/1321194428 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f18980a2b70 0x7f18980a5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.805+0000 7f18a5ffb640 1 -- 172.21.15.33:0/1321194428 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f18940cf140 con 0x7f18a8101060 2024-03-19T02:55:27.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.805+0000 7f18ad36a640 1 -- 172.21.15.33:0/2182888507 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5f83b6fe0 msgr2=0x55b5f83b93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:27.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.805+0000 7f18ad36a640 1 --2- 172.21.15.33:0/2182888507 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5f83b6fe0 0x55b5f83b93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.805+0000 7f18ad36a640 1 -- 172.21.15.33:0/2182888507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5f83a5a90 msgr2=0x55b5f83b6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:27.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.805+0000 7f18ad36a640 1 --2- 172.21.15.33:0/2182888507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5f83a5a90 0x55b5f83b6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.805+0000 7f18ad36a640 1 -- 172.21.15.33:0/2182888507 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5f82d3210 con 0x55b5f83a50e0 2024-03-19T02:55:27.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.806+0000 7f18ad36a640 1 --2- 172.21.15.33:0/1321194428 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f18980a2b70 0x7f18980a5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.806+0000 7f18ad36a640 1 --2- 172.21.15.33:0/2182888507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5f83a50e0 0x55b5f83a54c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f18a0009cd0 tx=0x7f18a00323d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7f827a6ca09073c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:27.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.807+0000 7f1892ffd640 1 -- 172.21.15.33:0/2182888507 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a0039040 con 0x55b5f83a50e0 2024-03-19T02:55:27.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.807+0000 7f1892ffd640 1 -- 172.21.15.33:0/2182888507 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f18a003f920 con 0x55b5f83a50e0 2024-03-19T02:55:27.808 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.807+0000 7f1892ffd640 1 -- 172.21.15.33:0/2182888507 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a00327a0 con 0x55b5f83a50e0 2024-03-19T02:55:27.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.808+0000 7f18b19bb740 1 -- 172.21.15.33:0/2182888507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5f83a50e0 msgr2=0x55b5f83a54c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:27.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.808+0000 7f18ad36a640 1 --2- 172.21.15.33:0/1321194428 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f18980a2b70 0x7f18980a5030 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f18a0049000 tx=0x7f18a004c000 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:27.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.808+0000 7f18b19bb740 1 --2- 172.21.15.33:0/2182888507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5f83a50e0 0x55b5f83a54c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f18a0009cd0 tx=0x7f18a00323d0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.808+0000 7f18b19bb740 1 -- 172.21.15.33:0/2182888507 shutdown_connections 2024-03-19T02:55:27.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.808+0000 7f18b19bb740 1 --2- 172.21.15.33:0/2182888507 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5f83b6fe0 0x55b5f83b93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.808+0000 7f18b19bb740 1 --2- 172.21.15.33:0/2182888507 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5f83a5a90 0x55b5f83b6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.808+0000 7f18b19bb740 1 --2- 172.21.15.33:0/2182888507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5f83a50e0 0x55b5f83a54c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.808+0000 7f18b19bb740 1 -- 172.21.15.33:0/2182888507 >> 172.21.15.33:0/2182888507 conn(0x55b5f839fa00 msgr2=0x55b5f839fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:27.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.808+0000 7f18b19bb740 1 -- 172.21.15.33:0/2182888507 shutdown_connections 2024-03-19T02:55:27.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.808+0000 7f18b19bb740 1 -- 172.21.15.33:0/2182888507 wait complete. 2024-03-19T02:55:27.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.808+0000 7f18b19bb740 1 -- start start 2024-03-19T02:55:27.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.808+0000 7f18b19bb740 1 Processor -- start 2024-03-19T02:55:27.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.809+0000 7f18b19bb740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5f83a50e0 0x55b5f83b0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.809+0000 7f18b19bb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5f83a5a90 0x55b5f83b69e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.809+0000 7f18b19bb740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5f83b6fe0 0x55b5f83b12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.809+0000 7f18b19bb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b5f83a6630 con 0x55b5f83a5a90 2024-03-19T02:55:27.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.809+0000 7f18b19bb740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55b5f82c4310 con 0x55b5f83b6fe0 2024-03-19T02:55:27.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.809+0000 7f18b19bb740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b5f83a67b0 con 0x55b5f83a50e0 2024-03-19T02:55:27.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.809+0000 7f18ad36a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5f83b6fe0 0x55b5f83b12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.809+0000 7f18ad36a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5f83b6fe0 0x55b5f83b12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:45020/0 (socket says 172.21.15.33:45020) 2024-03-19T02:55:27.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.809+0000 7f18ad36a640 1 -- 172.21.15.33:0/769513818 learned_addr learned my addr 172.21.15.33:0/769513818 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:27.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.809+0000 7f18a7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5f83a5a90 0x55b5f83b69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.809+0000 7f18b19bb740 10 client.0 _ll_register_callbacks cb 0x55b5f82b82d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:55:27.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.809+0000 7f18acb69640 1 --2- 172.21.15.33:0/769513818 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5f83a50e0 0x55b5f83b0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.809+0000 7f18ad36a640 1 -- 172.21.15.33:0/769513818 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5f83a50e0 msgr2=0x55b5f83b0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:27.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.809+0000 7f18ad36a640 1 --2- 172.21.15.33:0/769513818 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5f83a50e0 0x55b5f83b0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.809+0000 7f18ad36a640 1 -- 172.21.15.33:0/769513818 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5f83a5a90 msgr2=0x55b5f83b69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:27.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.809+0000 7f18ad36a640 1 --2- 172.21.15.33:0/769513818 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5f83a5a90 0x55b5f83b69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.809+0000 7f18ad36a640 1 -- 172.21.15.33:0/769513818 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5f801c3f0 con 0x55b5f83b6fe0 2024-03-19T02:55:27.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.809+0000 7f18a7fff640 1 --2- 172.21.15.33:0/769513818 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5f83a5a90 0x55b5f83b69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:27.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.809+0000 7f18ad36a640 1 --2- 172.21.15.33:0/769513818 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5f83b6fe0 0x55b5f83b12f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f18a003ffa0 tx=0x7f18a002c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3898e1c297f23a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:27.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.809+0000 7f1887fff640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T02:55:27.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.810+0000 7f1887fff640 1 -- 172.21.15.33:0/769513818 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a0039070 con 0x55b5f83b6fe0 2024-03-19T02:55:27.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.810+0000 7f18b19bb740 1 -- 172.21.15.33:0/769513818 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18a003f920 con 0x55b5f83b6fe0 2024-03-19T02:55:27.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.810+0000 7f1887fff640 1 -- 172.21.15.33:0/769513818 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f18a0032760 con 0x55b5f83b6fe0 2024-03-19T02:55:27.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.810+0000 7f1887fff640 1 -- 172.21.15.33:0/769513818 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f18a0064050 con 0x55b5f83b6fe0 2024-03-19T02:55:27.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.810+0000 7f18b19bb740 10 client.25916 fetch_fsmap learned FSMap version 25 2024-03-19T02:55:27.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.810+0000 7f18b19bb740 1 -- 172.21.15.33:0/769513818 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b5f80e67d0 con 0x55b5f83b6fe0 2024-03-19T02:55:27.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.810+0000 7f1887fff640 1 -- 172.21.15.33:0/769513818 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f18a0032ac0 con 0x55b5f83b6fe0 2024-03-19T02:55:27.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.811+0000 7f1887fff640 1 -- 172.21.15.33:0/769513818 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f18a0054040 con 0x55b5f83b6fe0 2024-03-19T02:55:27.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.811+0000 7f18b19bb740 10 client.25916 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:55:27.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.811+0000 7f18b19bb740 10 client.25916 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T02:55:27.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.811+0000 7f18b19bb740 20 client.25916 populate_metadata read hostname 'smithi033' 2024-03-19T02:55:27.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.811+0000 7f18b19bb740 1 --2- 172.21.15.33:0/769513818 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55b5f83cfaa0 0x55b5f83d1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.811+0000 7f18b19bb740 4 client.25916 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:55:27.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.811+0000 7f18b19bb740 1 -- 172.21.15.33:0/769513818 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f18a003f920 con 0x55b5f83cfaa0 2024-03-19T02:55:27.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.811+0000 7f18acb69640 1 --2- 172.21.15.33:0/769513818 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55b5f83cfaa0 0x55b5f83d1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.812+0000 7f1887fff640 10 client.25916 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T02:55:27.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.812+0000 7f18acb69640 1 --2- 172.21.15.33:0/769513818 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55b5f83cfaa0 0x55b5f83d1ed0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f19a8907eace7e46 server_cookie=26f60ab52a9a6295 in_seq=0 out_seq=0 2024-03-19T02:55:27.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.814+0000 7f1887fff640 1 -- 172.21.15.33:0/769513818 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f18a003f920 con 0x55b5f83cfaa0 2024-03-19T02:55:27.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.814+0000 7f1887fff640 10 client.25916 handle_command_reply: tid=0 2024-03-19T02:55:27.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.814+0000 7f18b19bb740 1 client.25916 shutdown 2024-03-19T02:55:27.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.814+0000 7f18b19bb740 1 -- 172.21.15.33:0/769513818 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5f83b6fe0 msgr2=0x55b5f83b12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:27.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.814+0000 7f18b19bb740 1 --2- 172.21.15.33:0/769513818 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5f83b6fe0 0x55b5f83b12f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f18a003ffa0 tx=0x7f18a002c800 comp rx=0 tx=0).stop 2024-03-19T02:55:27.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.814+0000 7f18b19bb740 1 -- 172.21.15.33:0/769513818 shutdown_connections 2024-03-19T02:55:27.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.814+0000 7f18b19bb740 1 --2- 172.21.15.33:0/769513818 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55b5f83cfaa0 0x55b5f83d1ed0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.814+0000 7f18b19bb740 1 --2- 172.21.15.33:0/769513818 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5f83b6fe0 0x55b5f83b12f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.814+0000 7f18b19bb740 1 --2- 172.21.15.33:0/769513818 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5f83a5a90 0x55b5f83b69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.814+0000 7f18b19bb740 1 --2- 172.21.15.33:0/769513818 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5f83a50e0 0x55b5f83b0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.814+0000 7f1887fff640 0 client.25916 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T02:55:27.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.814+0000 7f18b19bb740 1 -- 172.21.15.33:0/769513818 >> 172.21.15.33:0/769513818 conn(0x55b5f839fa00 msgr2=0x55b5f83a1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:27.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.815+0000 7f18b19bb740 1 -- 172.21.15.33:0/769513818 shutdown_connections 2024-03-19T02:55:27.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.815+0000 7f18b19bb740 1 -- 172.21.15.33:0/769513818 wait complete. 2024-03-19T02:55:27.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.815+0000 7f18b19bb740 20 client.25916 trim_cache size 0 max 16384 2024-03-19T02:55:27.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.835+0000 7f18b19bb740 1 Processor -- start 2024-03-19T02:55:27.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.835+0000 7f18b19bb740 1 -- start start 2024-03-19T02:55:27.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.835+0000 7f18b19bb740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5f83a50e0 0x55b5f839ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.835+0000 7f18b19bb740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5f83a5a90 0x55b5f83d7770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.835+0000 7f18b19bb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5f83b6fe0 0x55b5f83e2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.835+0000 7f18b19bb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f18a81af260 con 0x55b5f83b6fe0 2024-03-19T02:55:27.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.835+0000 7f18b19bb740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f18a81acb30 con 0x55b5f83a5a90 2024-03-19T02:55:27.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.835+0000 7f18b19bb740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f18a81aa960 con 0x55b5f83a50e0 2024-03-19T02:55:27.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.835+0000 7f18ad36a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5f83b6fe0 0x55b5f83e2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.835+0000 7f18ad36a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5f83b6fe0 0x55b5f83e2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55362/0 (socket says 172.21.15.33:55362) 2024-03-19T02:55:27.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.835+0000 7f18ad36a640 1 -- 172.21.15.33:0/3876091279 learned_addr learned my addr 172.21.15.33:0/3876091279 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:27.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.835+0000 7f18acb69640 1 --2- 172.21.15.33:0/3876091279 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5f83a50e0 0x55b5f839ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.836+0000 7f18a7fff640 1 --2- 172.21.15.33:0/3876091279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5f83a5a90 0x55b5f83d7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.836+0000 7f18ad36a640 1 -- 172.21.15.33:0/3876091279 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5f83a50e0 msgr2=0x55b5f839ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:27.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.836+0000 7f18ad36a640 1 --2- 172.21.15.33:0/3876091279 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5f83a50e0 0x55b5f839ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.836+0000 7f18ad36a640 1 -- 172.21.15.33:0/3876091279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5f83a5a90 msgr2=0x55b5f83d7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:27.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.836+0000 7f18ad36a640 1 --2- 172.21.15.33:0/3876091279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5f83a5a90 0x55b5f83d7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.836+0000 7f18ad36a640 1 -- 172.21.15.33:0/3876091279 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5f8375880 con 0x55b5f83b6fe0 2024-03-19T02:55:27.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.836+0000 7f18a7fff640 1 --2- 172.21.15.33:0/3876091279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5f83a5a90 0x55b5f83d7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:55:27.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.836+0000 7f18ad36a640 1 --2- 172.21.15.33:0/3876091279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5f83b6fe0 0x55b5f83e2b10 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f18a002c140 tx=0x7f18a00095a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97e9441347460f81 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:27.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.836+0000 7f18927fc640 1 -- 172.21.15.33:0/3876091279 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a006d070 con 0x55b5f83b6fe0 2024-03-19T02:55:27.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.836+0000 7f18927fc640 1 -- 172.21.15.33:0/3876091279 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f18a003fde0 con 0x55b5f83b6fe0 2024-03-19T02:55:27.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.836+0000 7f18927fc640 1 -- 172.21.15.33:0/3876091279 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a0039040 con 0x55b5f83b6fe0 2024-03-19T02:55:27.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.837+0000 7f18b19bb740 1 -- 172.21.15.33:0/3876091279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5f83b6fe0 msgr2=0x55b5f83e2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:27.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.837+0000 7f18b19bb740 1 --2- 172.21.15.33:0/3876091279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5f83b6fe0 0x55b5f83e2b10 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f18a002c140 tx=0x7f18a00095a0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.837+0000 7f18b19bb740 1 -- 172.21.15.33:0/3876091279 shutdown_connections 2024-03-19T02:55:27.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.837+0000 7f18b19bb740 1 --2- 172.21.15.33:0/3876091279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5f83b6fe0 0x55b5f83e2b10 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.837+0000 7f18b19bb740 1 --2- 172.21.15.33:0/3876091279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5f83a5a90 0x55b5f83d7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.837+0000 7f18b19bb740 1 --2- 172.21.15.33:0/3876091279 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5f83a50e0 0x55b5f839ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.837+0000 7f18b19bb740 1 -- 172.21.15.33:0/3876091279 >> 172.21.15.33:0/3876091279 conn(0x55b5f839fa00 msgr2=0x55b5f839e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:27.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.838+0000 7f18b19bb740 1 -- 172.21.15.33:0/3876091279 shutdown_connections 2024-03-19T02:55:27.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.838+0000 7f18b19bb740 1 -- 172.21.15.33:0/3876091279 wait complete. 2024-03-19T02:55:27.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.838+0000 7f18b19bb740 1 -- start start 2024-03-19T02:55:27.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.838+0000 7f18b19bb740 1 Processor -- start 2024-03-19T02:55:27.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.839+0000 7f18b19bb740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5f83a50e0 0x55b5f83defd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.839+0000 7f18b19bb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5f83a5a90 0x55b5f83df510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.839+0000 7f18b19bb740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5f83b6fe0 0x55b5f842d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.839+0000 7f18a7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5f83a5a90 0x55b5f83df510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.839+0000 7f18a7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5f83a5a90 0x55b5f83df510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55366/0 (socket says 172.21.15.33:55366) 2024-03-19T02:55:27.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.839+0000 7f18acb69640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5f83a50e0 0x55b5f83defd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.839+0000 7f18a7fff640 1 -- 172.21.15.33:0/841971639 learned_addr learned my addr 172.21.15.33:0/841971639 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:27.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.839+0000 7f18b19bb740 1 -- 172.21.15.33:0/841971639 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f18a81aaff0 con 0x55b5f83a5a90 2024-03-19T02:55:27.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.839+0000 7f18b19bb740 1 -- 172.21.15.33:0/841971639 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f18a81ac460 con 0x55b5f83b6fe0 2024-03-19T02:55:27.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.839+0000 7f18b19bb740 1 -- 172.21.15.33:0/841971639 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f18a81ab6c0 con 0x55b5f83a50e0 2024-03-19T02:55:27.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.839+0000 7f18b19bb740 10 client.0 _ll_register_callbacks cb 0x7f189c009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:55:27.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.839+0000 7f18a7fff640 1 -- 172.21.15.33:0/841971639 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5f83a50e0 msgr2=0x55b5f83defd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:27.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.839+0000 7f18a7fff640 1 --2- 172.21.15.33:0/841971639 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5f83a50e0 0x55b5f83defd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.839+0000 7f18a7fff640 1 -- 172.21.15.33:0/841971639 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5f83b6fe0 msgr2=0x55b5f842d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:27.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.839+0000 7f18a7fff640 1 --2- 172.21.15.33:0/841971639 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5f83b6fe0 0x55b5f842d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.839+0000 7f18a7fff640 1 -- 172.21.15.33:0/841971639 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5f83b93d0 con 0x55b5f83a5a90 2024-03-19T02:55:27.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.839+0000 7f18a7fff640 1 --2- 172.21.15.33:0/841971639 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5f83a5a90 0x55b5f83df510 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f1894097e00 tx=0x7f189409ae50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4572594f96f865e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:27.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.839+0000 7f1887fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:55:27.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.840+0000 7f1887fff640 1 -- 172.21.15.33:0/841971639 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18940cc9b0 con 0x55b5f83a5a90 2024-03-19T02:55:27.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.840+0000 7f1887fff640 1 -- 172.21.15.33:0/841971639 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f18940d1110 con 0x55b5f83a5a90 2024-03-19T02:55:27.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.840+0000 7f18b19bb740 1 -- 172.21.15.33:0/841971639 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b5f83663e0 con 0x55b5f83a5a90 2024-03-19T02:55:27.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.840+0000 7f1887fff640 1 -- 172.21.15.33:0/841971639 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f189409b430 con 0x55b5f83a5a90 2024-03-19T02:55:27.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.840+0000 7f18b19bb740 10 client.16083 fetch_fsmap learned FSMap version 25 2024-03-19T02:55:27.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.840+0000 7f18b19bb740 1 -- 172.21.15.33:0/841971639 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b5f8436030 con 0x55b5f83a5a90 2024-03-19T02:55:27.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.840+0000 7f1887fff640 1 -- 172.21.15.33:0/841971639 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55b5f83663e0 con 0x55b5f83a5a90 2024-03-19T02:55:27.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.840+0000 7f1887fff640 1 -- 172.21.15.33:0/841971639 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f18940bb0b0 con 0x55b5f83a5a90 2024-03-19T02:55:27.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.840+0000 7f18b19bb740 10 client.16083 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:55:27.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.840+0000 7f18b19bb740 10 client.16083 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T02:55:27.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.840+0000 7f18b19bb740 20 client.16083 populate_metadata read hostname 'smithi033' 2024-03-19T02:55:27.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.840+0000 7f18b19bb740 1 --2- 172.21.15.33:0/841971639 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55b5f8436830 0x55b5f8438c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:27.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.840+0000 7f18b19bb740 4 client.16083 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:55:27.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.840+0000 7f18b19bb740 1 -- 172.21.15.33:0/841971639 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b5f83159f0 con 0x55b5f8436830 2024-03-19T02:55:27.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.841+0000 7f18acb69640 1 --2- 172.21.15.33:0/841971639 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55b5f8436830 0x55b5f8438c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:27.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.843+0000 7f18acb69640 1 --2- 172.21.15.33:0/841971639 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55b5f8436830 0x55b5f8438c20 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ac3d1a4b8e20fa30 server_cookie=1b4bccadada691f3 in_seq=0 out_seq=0 2024-03-19T02:55:27.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.843+0000 7f1887fff640 10 client.16083 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T02:55:27.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.845+0000 7f1887fff640 1 -- 172.21.15.33:0/841971639 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b5f83159f0 con 0x55b5f8436830 2024-03-19T02:55:27.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.845+0000 7f1887fff640 10 client.16083 handle_command_reply: tid=0 2024-03-19T02:55:27.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.845+0000 7f18b19bb740 1 client.16083 shutdown 2024-03-19T02:55:27.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.845+0000 7f18b19bb740 1 -- 172.21.15.33:0/841971639 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5f83a5a90 msgr2=0x55b5f83df510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:27.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.845+0000 7f18b19bb740 1 --2- 172.21.15.33:0/841971639 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5f83a5a90 0x55b5f83df510 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f1894097e00 tx=0x7f189409ae50 comp rx=0 tx=0).stop 2024-03-19T02:55:27.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.845+0000 7f18b19bb740 1 -- 172.21.15.33:0/841971639 shutdown_connections 2024-03-19T02:55:27.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.845+0000 7f18b19bb740 1 --2- 172.21.15.33:0/841971639 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55b5f8436830 0x55b5f8438c20 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.845+0000 7f1887fff640 0 client.16083 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T02:55:27.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.845+0000 7f18b19bb740 1 --2- 172.21.15.33:0/841971639 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5f83b6fe0 0x55b5f842d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.845+0000 7f18b19bb740 1 --2- 172.21.15.33:0/841971639 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5f83a5a90 0x55b5f83df510 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.845+0000 7f18b19bb740 1 --2- 172.21.15.33:0/841971639 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5f83a50e0 0x55b5f83defd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.845+0000 7f18b19bb740 1 -- 172.21.15.33:0/841971639 >> 172.21.15.33:0/841971639 conn(0x55b5f839fa00 msgr2=0x55b5f839eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:27.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.845+0000 7f18b19bb740 1 -- 172.21.15.33:0/841971639 shutdown_connections 2024-03-19T02:55:27.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.845+0000 7f18b19bb740 1 -- 172.21.15.33:0/841971639 wait complete. 2024-03-19T02:55:27.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.845+0000 7f18b19bb740 20 client.16083 trim_cache size 0 max 16384 2024-03-19T02:55:27.849 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:55:27.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.846+0000 7f18aedcd640 1 -- 172.21.15.33:0/1321194428 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f18980a2b70 msgr2=0x7f18980a5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:27.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.846+0000 7f18aedcd640 1 --2- 172.21.15.33:0/1321194428 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f18980a2b70 0x7f18980a5030 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f18a0049000 tx=0x7f18a004c000 comp rx=0 tx=0).stop 2024-03-19T02:55:27.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.846+0000 7f18aedcd640 1 -- 172.21.15.33:0/1321194428 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18a8101060 msgr2=0x7f18a819bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:27.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.846+0000 7f18aedcd640 1 --2- 172.21.15.33:0/1321194428 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18a8101060 0x7f18a819bfa0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f189400ea60 tx=0x7f189400ef20 comp rx=0 tx=0).stop 2024-03-19T02:55:27.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.846+0000 7f18aedcd640 1 -- 172.21.15.33:0/1321194428 shutdown_connections 2024-03-19T02:55:27.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.846+0000 7f18aedcd640 1 --2- 172.21.15.33:0/1321194428 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f18980a2b70 0x7f18980a5030 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.846+0000 7f18aedcd640 1 --2- 172.21.15.33:0/1321194428 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18a8101060 0x7f18a819bfa0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.846+0000 7f18aedcd640 1 --2- 172.21.15.33:0/1321194428 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18a81066b0 0x7f18a8195b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.846+0000 7f18aedcd640 1 --2- 172.21.15.33:0/1321194428 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f18a81006a0 0x7f18a819ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:27.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.846+0000 7f18aedcd640 1 -- 172.21.15.33:0/1321194428 >> 172.21.15.33:0/1321194428 conn(0x7f18a80778d0 msgr2=0x7f18a8102ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:27.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.846+0000 7f18aedcd640 1 -- 172.21.15.33:0/1321194428 shutdown_connections 2024-03-19T02:55:27.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:27.846+0000 7f18aedcd640 1 -- 172.21.15.33:0/1321194428 wait complete. 2024-03-19T02:55:27.867 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:55:28.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.043+0000 7fb2753e6640 1 -- 172.21.15.33:0/4174529731 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb270102180 msgr2=0x7fb27010e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:28.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.043+0000 7fb2753e6640 1 --2- 172.21.15.33:0/4174529731 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb270102180 0x7fb27010e690 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fb268009cd0 tx=0x7fb26802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.043+0000 7fb2753e6640 1 -- 172.21.15.33:0/4174529731 shutdown_connections 2024-03-19T02:55:28.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.043+0000 7fb2753e6640 1 --2- 172.21.15.33:0/4174529731 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2701017c0 0x7fb270101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.043+0000 7fb2753e6640 1 --2- 172.21.15.33:0/4174529731 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb270102180 0x7fb27010e690 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.043+0000 7fb2753e6640 1 --2- 172.21.15.33:0/4174529731 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2701077d0 0x7fb270107bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.043+0000 7fb2753e6640 1 -- 172.21.15.33:0/4174529731 >> 172.21.15.33:0/4174529731 conn(0x7fb2700fd2d0 msgr2=0x7fb2700ff6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:28.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.043+0000 7fb2753e6640 1 -- 172.21.15.33:0/4174529731 shutdown_connections 2024-03-19T02:55:28.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.044+0000 7fb2753e6640 1 -- 172.21.15.33:0/4174529731 wait complete. 2024-03-19T02:55:28.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.044+0000 7fb2753e6640 1 Processor -- start 2024-03-19T02:55:28.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.044+0000 7fb2753e6640 1 -- start start 2024-03-19T02:55:28.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.045+0000 7fb2753e6640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2701017c0 0x7fb270197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.045+0000 7fb2753e6640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb270102180 0x7fb2701983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.045+0000 7fb2753e6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb2701077d0 0x7fb27019c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.045+0000 7fb2753e6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb270111a90 con 0x7fb2701077d0 2024-03-19T02:55:28.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.045+0000 7fb2753e6640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb270111910 con 0x7fb2701017c0 2024-03-19T02:55:28.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.045+0000 7fb2753e6640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb270111c10 con 0x7fb270102180 2024-03-19T02:55:28.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.045+0000 7fb26f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb2701077d0 0x7fb27019c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:28.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.045+0000 7fb26effd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2701017c0 0x7fb270197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:28.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.045+0000 7fb26effd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2701017c0 0x7fb270197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:45048/0 (socket says 172.21.15.33:45048) 2024-03-19T02:55:28.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.045+0000 7fb26f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb2701077d0 0x7fb27019c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55390/0 (socket says 172.21.15.33:55390) 2024-03-19T02:55:28.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.045+0000 7fb26effd640 1 -- 172.21.15.33:0/1182115125 learned_addr learned my addr 172.21.15.33:0/1182115125 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:28.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.045+0000 7fb26effd640 1 -- 172.21.15.33:0/1182115125 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb270102180 msgr2=0x7fb2701983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:28.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.045+0000 7fb26effd640 1 --2- 172.21.15.33:0/1182115125 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb270102180 0x7fb2701983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.045+0000 7fb26effd640 1 -- 172.21.15.33:0/1182115125 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb2701077d0 msgr2=0x7fb27019c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:28.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.045+0000 7fb26effd640 1 --2- 172.21.15.33:0/1182115125 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb2701077d0 0x7fb27019c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.045+0000 7fb26effd640 1 -- 172.21.15.33:0/1182115125 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb27019cea0 con 0x7fb2701017c0 2024-03-19T02:55:28.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.046+0000 7fb26effd640 1 --2- 172.21.15.33:0/1182115125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2701017c0 0x7fb270197eb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb26000d950 tx=0x7fb26000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:28.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.046+0000 7fb257fff640 1 -- 172.21.15.33:0/1182115125 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb260017070 con 0x7fb2701017c0 2024-03-19T02:55:28.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.046+0000 7fb257fff640 1 -- 172.21.15.33:0/1182115125 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb260009c30 con 0x7fb2701017c0 2024-03-19T02:55:28.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.046+0000 7fb257fff640 1 -- 172.21.15.33:0/1182115125 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb26000f620 con 0x7fb2701017c0 2024-03-19T02:55:28.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.046+0000 7fb2753e6640 1 -- 172.21.15.33:0/1182115125 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb27019d190 con 0x7fb2701017c0 2024-03-19T02:55:28.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.046+0000 7fb2753e6640 1 -- 172.21.15.33:0/1182115125 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb270073a50 con 0x7fb2701017c0 2024-03-19T02:55:28.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.050+0000 7fb257fff640 1 -- 172.21.15.33:0/1182115125 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fb26000f7c0 con 0x7fb2701017c0 2024-03-19T02:55:28.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.050+0000 7fb277fd4740 1 Processor -- start 2024-03-19T02:55:28.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.051+0000 7fb277fd4740 1 -- start start 2024-03-19T02:55:28.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.051+0000 7fb257fff640 1 --2- 172.21.15.33:0/1182115125 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fb2580a2a70 0x7fb2580a4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.051+0000 7fb257fff640 1 -- 172.21.15.33:0/1182115125 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fb2600ced70 con 0x7fb2701017c0 2024-03-19T02:55:28.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.051+0000 7fb277fd4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c73bed02d0 0x55c73bed06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.051+0000 7fb277fd4740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c73bed0bf0 0x55c73bed9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.051+0000 7fb277fd4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c73bed97d0 0x55c73bedbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.051+0000 7fb277fd4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c73be334b0 con 0x55c73bed97d0 2024-03-19T02:55:28.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.051+0000 7fb277fd4740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c73bddbd70 con 0x55c73bed0bf0 2024-03-19T02:55:28.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.051+0000 7fb277fd4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c73be2f550 con 0x55c73bed02d0 2024-03-19T02:55:28.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.052+0000 7fb26f7fe640 1 --2- 172.21.15.33:0/1182115125 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fb2580a2a70 0x7fb2580a4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:28.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.052+0000 7fb26effd640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c73bed02d0 0x55c73bed06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:28.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.052+0000 7fb26effd640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c73bed02d0 0x55c73bed06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:60244/0 (socket says 172.21.15.33:60244) 2024-03-19T02:55:28.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.052+0000 7fb26effd640 1 -- 172.21.15.33:0/1026949649 learned_addr learned my addr 172.21.15.33:0/1026949649 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:28.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.052+0000 7fb26f7fe640 1 --2- 172.21.15.33:0/1026949649 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c73bed97d0 0x55c73bedbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:28.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.052+0000 7fb26effd640 1 -- 172.21.15.33:0/1026949649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c73bed0bf0 msgr2=0x55c73bed9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:28.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.052+0000 7fb26effd640 1 --2- 172.21.15.33:0/1026949649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c73bed0bf0 0x55c73bed9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.052+0000 7fb26effd640 1 -- 172.21.15.33:0/1026949649 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c73bed97d0 msgr2=0x55c73bedbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:28.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.052+0000 7fb26effd640 1 --2- 172.21.15.33:0/1026949649 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c73bed97d0 0x55c73bedbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.052+0000 7fb26effd640 1 -- 172.21.15.33:0/1026949649 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c73bdfe210 con 0x55c73bed02d0 2024-03-19T02:55:28.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.053+0000 7fb26f7fe640 1 --2- 172.21.15.33:0/1182115125 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fb2580a2a70 0x7fb2580a4f30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb268006010 tx=0x7fb268003c20 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:28.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.053+0000 7fb26effd640 1 --2- 172.21.15.33:0/1026949649 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c73bed02d0 0x55c73bed06b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb2600a3290 tx=0x7fb2600973f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30057be7affdfbf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:28.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.053+0000 7fb2557fa640 1 -- 172.21.15.33:0/1026949649 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb260020040 con 0x55c73bed02d0 2024-03-19T02:55:28.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.053+0000 7fb2557fa640 1 -- 172.21.15.33:0/1026949649 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb26009b020 con 0x55c73bed02d0 2024-03-19T02:55:28.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.053+0000 7fb2557fa640 1 -- 172.21.15.33:0/1026949649 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2600cc390 con 0x55c73bed02d0 2024-03-19T02:55:28.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.054+0000 7fb277fd4740 1 -- 172.21.15.33:0/1026949649 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c73bed02d0 msgr2=0x55c73bed06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:28.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.054+0000 7fb277fd4740 1 --2- 172.21.15.33:0/1026949649 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c73bed02d0 0x55c73bed06b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb2600a3290 tx=0x7fb2600973f0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.054+0000 7fb277fd4740 1 -- 172.21.15.33:0/1026949649 shutdown_connections 2024-03-19T02:55:28.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.054+0000 7fb277fd4740 1 --2- 172.21.15.33:0/1026949649 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c73bed97d0 0x55c73bedbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.054+0000 7fb277fd4740 1 --2- 172.21.15.33:0/1026949649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c73bed0bf0 0x55c73bed9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.054+0000 7fb277fd4740 1 --2- 172.21.15.33:0/1026949649 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c73bed02d0 0x55c73bed06b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.054+0000 7fb277fd4740 1 -- 172.21.15.33:0/1026949649 >> 172.21.15.33:0/1026949649 conn(0x55c73becaa00 msgr2=0x55c73becade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:28.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.054+0000 7fb277fd4740 1 -- 172.21.15.33:0/1026949649 shutdown_connections 2024-03-19T02:55:28.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.054+0000 7fb277fd4740 1 -- 172.21.15.33:0/1026949649 wait complete. 2024-03-19T02:55:28.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.054+0000 7fb277fd4740 1 -- start start 2024-03-19T02:55:28.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.055+0000 7fb277fd4740 1 Processor -- start 2024-03-19T02:55:28.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.055+0000 7fb277fd4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c73bed02d0 0x55c73bed6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.055+0000 7fb277fd4740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c73bed0bf0 0x55c73bed6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.055+0000 7fb277fd4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c73bed97d0 0x55c73bed72f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.055+0000 7fb277fd4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c73bedc5c0 con 0x55c73bed97d0 2024-03-19T02:55:28.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.055+0000 7fb277fd4740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c73bdef310 con 0x55c73bed0bf0 2024-03-19T02:55:28.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.055+0000 7fb277fd4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c73bedc740 con 0x55c73bed02d0 2024-03-19T02:55:28.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.055+0000 7fb26effd640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c73bed02d0 0x55c73bed6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:28.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.055+0000 7fb277fd4740 10 client.0 _ll_register_callbacks cb 0x55c73bde32d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:55:28.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.055+0000 7fb26effd640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c73bed02d0 0x55c73bed6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:60258/0 (socket says 172.21.15.33:60258) 2024-03-19T02:55:28.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.055+0000 7fb26effd640 1 -- 172.21.15.33:0/1931684394 learned_addr learned my addr 172.21.15.33:0/1931684394 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:28.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.055+0000 7fb26e7fc640 1 --2- 172.21.15.33:0/1931684394 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c73bed0bf0 0x55c73bed6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:28.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.055+0000 7fb26f7fe640 1 --2- 172.21.15.33:0/1931684394 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c73bed97d0 0x55c73bed72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:28.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.056+0000 7fb26effd640 1 -- 172.21.15.33:0/1931684394 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c73bed0bf0 msgr2=0x55c73bed6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:28.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.056+0000 7fb26effd640 1 --2- 172.21.15.33:0/1931684394 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c73bed0bf0 0x55c73bed6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.056+0000 7fb26effd640 1 -- 172.21.15.33:0/1931684394 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c73bed97d0 msgr2=0x55c73bed72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:28.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.056+0000 7fb26effd640 1 --2- 172.21.15.33:0/1931684394 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c73bed97d0 0x55c73bed72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.056+0000 7fb26effd640 1 -- 172.21.15.33:0/1931684394 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c73bb473f0 con 0x55c73bed02d0 2024-03-19T02:55:28.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.056+0000 7fb26e7fc640 1 --2- 172.21.15.33:0/1931684394 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c73bed0bf0 0x55c73bed6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:55:28.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.056+0000 7fb26f7fe640 1 --2- 172.21.15.33:0/1931684394 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c73bed97d0 0x55c73bed72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:55:28.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.056+0000 7fb26effd640 1 --2- 172.21.15.33:0/1931684394 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c73bed02d0 0x55c73bed6870 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb2600058c0 tx=0x7fb260010c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fc39010a1427547 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:28.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.056+0000 7fb252ff5640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-19T02:55:28.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.056+0000 7fb252ff5640 1 -- 172.21.15.33:0/1931684394 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb26007e060 con 0x55c73bed02d0 2024-03-19T02:55:28.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.056+0000 7fb252ff5640 1 -- 172.21.15.33:0/1931684394 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fb260097690 con 0x55c73bed02d0 2024-03-19T02:55:28.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.056+0000 7fb277fd4740 1 -- 172.21.15.33:0/1931684394 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb26009b020 con 0x55c73bed02d0 2024-03-19T02:55:28.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.056+0000 7fb252ff5640 1 -- 172.21.15.33:0/1931684394 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb260020070 con 0x55c73bed02d0 2024-03-19T02:55:28.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.056+0000 7fb277fd4740 10 client.25792 fetch_fsmap learned FSMap version 25 2024-03-19T02:55:28.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.056+0000 7fb277fd4740 1 -- 172.21.15.33:0/1931684394 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c73bc117d0 con 0x55c73bed02d0 2024-03-19T02:55:28.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.057+0000 7fb252ff5640 1 -- 172.21.15.33:0/1931684394 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb26009b020 con 0x55c73bed02d0 2024-03-19T02:55:28.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.057+0000 7fb252ff5640 1 -- 172.21.15.33:0/1931684394 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fb260096d70 con 0x55c73bed02d0 2024-03-19T02:55:28.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.058+0000 7fb277fd4740 10 client.25792 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:55:28.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.058+0000 7fb277fd4740 10 client.25792 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T02:55:28.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.058+0000 7fb277fd4740 20 client.25792 populate_metadata read hostname 'smithi033' 2024-03-19T02:55:28.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.058+0000 7fb277fd4740 1 --2- 172.21.15.33:0/1931684394 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55c73bee9a70 0x55c73beebe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.058+0000 7fb277fd4740 4 client.25792 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:55:28.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.058+0000 7fb277fd4740 1 -- 172.21.15.33:0/1931684394 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c73be913e0 con 0x55c73bee9a70 2024-03-19T02:55:28.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.059+0000 7fb26e7fc640 1 --2- 172.21.15.33:0/1931684394 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55c73bee9a70 0x55c73beebe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:28.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.060+0000 7fb252ff5640 10 client.25792 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T02:55:28.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.060+0000 7fb26e7fc640 1 --2- 172.21.15.33:0/1931684394 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55c73bee9a70 0x55c73beebe60 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8f1f1a6ac984c248 server_cookie=8214603387207a5c in_seq=0 out_seq=0 2024-03-19T02:55:28.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.061+0000 7fb252ff5640 1 -- 172.21.15.33:0/1931684394 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c73be913e0 con 0x55c73bee9a70 2024-03-19T02:55:28.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.061+0000 7fb252ff5640 10 client.25792 handle_command_reply: tid=0 2024-03-19T02:55:28.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.061+0000 7fb277fd4740 1 client.25792 shutdown 2024-03-19T02:55:28.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.062+0000 7fb277fd4740 1 -- 172.21.15.33:0/1931684394 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c73bed02d0 msgr2=0x55c73bed6870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:28.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.062+0000 7fb277fd4740 1 --2- 172.21.15.33:0/1931684394 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c73bed02d0 0x55c73bed6870 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb2600058c0 tx=0x7fb260010c50 comp rx=0 tx=0).stop 2024-03-19T02:55:28.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.062+0000 7fb277fd4740 1 -- 172.21.15.33:0/1931684394 shutdown_connections 2024-03-19T02:55:28.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.062+0000 7fb277fd4740 1 --2- 172.21.15.33:0/1931684394 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55c73bee9a70 0x55c73beebe60 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.062+0000 7fb277fd4740 1 --2- 172.21.15.33:0/1931684394 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c73bed97d0 0x55c73bed72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.062+0000 7fb277fd4740 1 --2- 172.21.15.33:0/1931684394 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c73bed0bf0 0x55c73bed6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.062+0000 7fb277fd4740 1 --2- 172.21.15.33:0/1931684394 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c73bed02d0 0x55c73bed6870 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.062+0000 7fb277fd4740 1 -- 172.21.15.33:0/1931684394 >> 172.21.15.33:0/1931684394 conn(0x55c73becaa00 msgr2=0x55c73becc750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:28.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.062+0000 7fb252ff5640 0 client.25792 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T02:55:28.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.062+0000 7fb277fd4740 1 -- 172.21.15.33:0/1931684394 shutdown_connections 2024-03-19T02:55:28.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.062+0000 7fb277fd4740 1 -- 172.21.15.33:0/1931684394 wait complete. 2024-03-19T02:55:28.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.062+0000 7fb277fd4740 20 client.25792 trim_cache size 0 max 16384 2024-03-19T02:55:28.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.085+0000 7fb277fd4740 1 Processor -- start 2024-03-19T02:55:28.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.085+0000 7fb277fd4740 1 -- start start 2024-03-19T02:55:28.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.086+0000 7fb277fd4740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c73bed02d0 0x55c73bee5390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.086+0000 7fb277fd4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c73bed0bf0 0x55c73bed7110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.086+0000 7fb277fd4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c73bed97d0 0x55c73bef1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.086+0000 7fb26f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c73bed97d0 0x55c73bef1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:28.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.086+0000 7fb26effd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c73bed02d0 0x55c73bee5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:28.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.086+0000 7fb277fd4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb2701af200 con 0x55c73bed97d0 2024-03-19T02:55:28.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.086+0000 7fb277fd4740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb2701acad0 con 0x55c73bed02d0 2024-03-19T02:55:28.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.086+0000 7fb277fd4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb2701aa900 con 0x55c73bed0bf0 2024-03-19T02:55:28.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.086+0000 7fb26f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c73bed97d0 0x55c73bef1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55416/0 (socket says 172.21.15.33:55416) 2024-03-19T02:55:28.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.086+0000 7fb26f7fe640 1 -- 172.21.15.33:0/229865813 learned_addr learned my addr 172.21.15.33:0/229865813 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:28.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.087+0000 7fb26e7fc640 1 --2- 172.21.15.33:0/229865813 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c73bed0bf0 0x55c73bed7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:28.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.087+0000 7fb26f7fe640 1 -- 172.21.15.33:0/229865813 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c73bed0bf0 msgr2=0x55c73bed7110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:28.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.087+0000 7fb26f7fe640 1 --2- 172.21.15.33:0/229865813 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c73bed0bf0 0x55c73bed7110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.087+0000 7fb26f7fe640 1 -- 172.21.15.33:0/229865813 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c73bed02d0 msgr2=0x55c73bee5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:28.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.087+0000 7fb26f7fe640 1 --2- 172.21.15.33:0/229865813 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c73bed02d0 0x55c73bee5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.087+0000 7fb26f7fe640 1 -- 172.21.15.33:0/229865813 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c73bedbf30 con 0x55c73bed97d0 2024-03-19T02:55:28.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.087+0000 7fb26f7fe640 1 --2- 172.21.15.33:0/229865813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c73bed97d0 0x55c73bef1480 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fb268036000 tx=0x7fb26804d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b758ee6c01b8f87a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:28.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.088+0000 7fb254ff9640 1 -- 172.21.15.33:0/229865813 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb268039070 con 0x55c73bed97d0 2024-03-19T02:55:28.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.088+0000 7fb254ff9640 1 -- 172.21.15.33:0/229865813 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb268051070 con 0x55c73bed97d0 2024-03-19T02:55:28.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.088+0000 7fb254ff9640 1 -- 172.21.15.33:0/229865813 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb26804d420 con 0x55c73bed97d0 2024-03-19T02:55:28.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.089+0000 7fb277fd4740 1 -- 172.21.15.33:0/229865813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c73bed97d0 msgr2=0x55c73bef1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:28.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.089+0000 7fb277fd4740 1 --2- 172.21.15.33:0/229865813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c73bed97d0 0x55c73bef1480 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fb268036000 tx=0x7fb26804d000 comp rx=0 tx=0).stop 2024-03-19T02:55:28.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.089+0000 7fb277fd4740 1 -- 172.21.15.33:0/229865813 shutdown_connections 2024-03-19T02:55:28.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.089+0000 7fb277fd4740 1 --2- 172.21.15.33:0/229865813 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c73bed0bf0 0x55c73bed7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.089+0000 7fb277fd4740 1 --2- 172.21.15.33:0/229865813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c73bed97d0 0x55c73bef1480 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.089+0000 7fb277fd4740 1 --2- 172.21.15.33:0/229865813 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c73bed02d0 0x55c73bee5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.089+0000 7fb277fd4740 1 -- 172.21.15.33:0/229865813 >> 172.21.15.33:0/229865813 conn(0x55c73becaa00 msgr2=0x55c73bec9ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:28.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.089+0000 7fb277fd4740 1 -- 172.21.15.33:0/229865813 shutdown_connections 2024-03-19T02:55:28.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.089+0000 7fb277fd4740 1 -- 172.21.15.33:0/229865813 wait complete. 2024-03-19T02:55:28.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.090+0000 7fb277fd4740 1 -- start start 2024-03-19T02:55:28.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.090+0000 7fb277fd4740 1 Processor -- start 2024-03-19T02:55:28.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.090+0000 7fb277fd4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c73bed02d0 0x55c73bee1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.090+0000 7fb277fd4740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c73bed0bf0 0x55c73bedd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.090+0000 7fb277fd4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c73bed97d0 0x55c73bedd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.090+0000 7fb26effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c73bed02d0 0x55c73bee1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:28.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.090+0000 7fb26effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c73bed02d0 0x55c73bee1d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55430/0 (socket says 172.21.15.33:55430) 2024-03-19T02:55:28.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.090+0000 7fb26effd640 1 -- 172.21.15.33:0/2538921846 learned_addr learned my addr 172.21.15.33:0/2538921846 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:28.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.090+0000 7fb277fd4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb2701aaf90 con 0x55c73bed02d0 2024-03-19T02:55:28.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.090+0000 7fb277fd4740 1 -- 172.21.15.33:0/2538921846 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb2701ac400 con 0x55c73bed0bf0 2024-03-19T02:55:28.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.090+0000 7fb277fd4740 1 -- 172.21.15.33:0/2538921846 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb2701ab660 con 0x55c73bed97d0 2024-03-19T02:55:28.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.091+0000 7fb26e7fc640 1 --2- 172.21.15.33:0/2538921846 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c73bed0bf0 0x55c73bedd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:28.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.091+0000 7fb277fd4740 10 client.0 _ll_register_callbacks cb 0x7fb2600b2070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:55:28.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.091+0000 7fb26f7fe640 1 --2- 172.21.15.33:0/2538921846 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c73bed97d0 0x55c73bedd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:28.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.091+0000 7fb26e7fc640 1 -- 172.21.15.33:0/2538921846 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c73bed97d0 msgr2=0x55c73bedd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:28.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.091+0000 7fb26e7fc640 1 --2- 172.21.15.33:0/2538921846 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c73bed97d0 0x55c73bedd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.091+0000 7fb26e7fc640 1 -- 172.21.15.33:0/2538921846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c73bed02d0 msgr2=0x55c73bee1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:28.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.091+0000 7fb26e7fc640 1 --2- 172.21.15.33:0/2538921846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c73bed02d0 0x55c73bee1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.091+0000 7fb26e7fc640 1 -- 172.21.15.33:0/2538921846 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c73beddee0 con 0x55c73bed0bf0 2024-03-19T02:55:28.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.091+0000 7fb26effd640 1 --2- 172.21.15.33:0/2538921846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c73bed02d0 0x55c73bee1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:28.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.091+0000 7fb26e7fc640 1 --2- 172.21.15.33:0/2538921846 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c73bed0bf0 0x55c73bedd0d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x55c73beec3a0 tx=0x7fb26400afd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=950b79ad74975914 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:28.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.091+0000 7fb252ff5640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T02:55:28.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.091+0000 7fb252ff5640 1 -- 172.21.15.33:0/2538921846 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb264004220 con 0x55c73bed0bf0 2024-03-19T02:55:28.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.092+0000 7fb252ff5640 1 -- 172.21.15.33:0/2538921846 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fb2640043c0 con 0x55c73bed0bf0 2024-03-19T02:55:28.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.092+0000 7fb277fd4740 1 -- 172.21.15.33:0/2538921846 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb268051070 con 0x55c73bed0bf0 2024-03-19T02:55:28.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.092+0000 7fb252ff5640 1 -- 172.21.15.33:0/2538921846 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb2640109a0 con 0x55c73bed0bf0 2024-03-19T02:55:28.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.092+0000 7fb277fd4740 10 client.25931 fetch_fsmap learned FSMap version 25 2024-03-19T02:55:28.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.092+0000 7fb277fd4740 1 -- 172.21.15.33:0/2538921846 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c73bf58400 con 0x55c73bed0bf0 2024-03-19T02:55:28.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.092+0000 7fb252ff5640 1 -- 172.21.15.33:0/2538921846 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb268051070 con 0x55c73bed0bf0 2024-03-19T02:55:28.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.092+0000 7fb252ff5640 1 -- 172.21.15.33:0/2538921846 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fb2640036f0 con 0x55c73bed0bf0 2024-03-19T02:55:28.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.093+0000 7fb277fd4740 10 client.25931 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:55:28.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.093+0000 7fb277fd4740 10 client.25931 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T02:55:28.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.093+0000 7fb277fd4740 20 client.25931 populate_metadata read hostname 'smithi033' 2024-03-19T02:55:28.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.093+0000 7fb277fd4740 1 --2- 172.21.15.33:0/2538921846 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55c73bf58c00 0x55c73bf5b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.093+0000 7fb26effd640 1 --2- 172.21.15.33:0/2538921846 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55c73bf58c00 0x55c73bf5b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:28.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.093+0000 7fb277fd4740 4 client.25931 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:55:28.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.093+0000 7fb277fd4740 1 -- 172.21.15.33:0/2538921846 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c73be409f0 con 0x55c73bf58c00 2024-03-19T02:55:28.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.094+0000 7fb252ff5640 10 client.25931 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T02:55:28.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.094+0000 7fb26effd640 1 --2- 172.21.15.33:0/2538921846 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55c73bf58c00 0x55c73bf5b020 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7f964bbb4a34a1d1 server_cookie=3d85739092b75d52 in_seq=0 out_seq=0 2024-03-19T02:55:28.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.094+0000 7fb252ff5640 1 -- 172.21.15.33:0/2538921846 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c73be409f0 con 0x55c73bf58c00 2024-03-19T02:55:28.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.094+0000 7fb252ff5640 10 client.25931 handle_command_reply: tid=0 2024-03-19T02:55:28.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.095+0000 7fb277fd4740 1 client.25931 shutdown 2024-03-19T02:55:28.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.095+0000 7fb277fd4740 1 -- 172.21.15.33:0/2538921846 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c73bed0bf0 msgr2=0x55c73bedd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:28.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.095+0000 7fb277fd4740 1 --2- 172.21.15.33:0/2538921846 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c73bed0bf0 0x55c73bedd0d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x55c73beec3a0 tx=0x7fb26400afd0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.095+0000 7fb277fd4740 1 -- 172.21.15.33:0/2538921846 shutdown_connections 2024-03-19T02:55:28.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.095+0000 7fb277fd4740 1 --2- 172.21.15.33:0/2538921846 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55c73bf58c00 0x55c73bf5b020 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.095+0000 7fb252ff5640 0 client.25931 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T02:55:28.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.095+0000 7fb277fd4740 1 --2- 172.21.15.33:0/2538921846 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c73bed97d0 0x55c73bedd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.095+0000 7fb277fd4740 1 --2- 172.21.15.33:0/2538921846 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c73bed0bf0 0x55c73bedd0d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.095+0000 7fb277fd4740 1 --2- 172.21.15.33:0/2538921846 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c73bed02d0 0x55c73bee1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.095+0000 7fb277fd4740 1 -- 172.21.15.33:0/2538921846 >> 172.21.15.33:0/2538921846 conn(0x55c73becaa00 msgr2=0x55c73becbd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:28.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.095+0000 7fb277fd4740 1 -- 172.21.15.33:0/2538921846 shutdown_connections 2024-03-19T02:55:28.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.095+0000 7fb277fd4740 1 -- 172.21.15.33:0/2538921846 wait complete. 2024-03-19T02:55:28.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.096+0000 7fb277fd4740 20 client.25931 trim_cache size 0 max 16384 2024-03-19T02:55:28.099 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:55:28.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.097+0000 7fb2753e6640 1 -- 172.21.15.33:0/1182115125 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fb2580a2a70 msgr2=0x7fb2580a4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:28.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.097+0000 7fb2753e6640 1 --2- 172.21.15.33:0/1182115125 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fb2580a2a70 0x7fb2580a4f30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb268006010 tx=0x7fb268003c20 comp rx=0 tx=0).stop 2024-03-19T02:55:28.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.097+0000 7fb2753e6640 1 -- 172.21.15.33:0/1182115125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2701017c0 msgr2=0x7fb270197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:28.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.097+0000 7fb2753e6640 1 --2- 172.21.15.33:0/1182115125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2701017c0 0x7fb270197eb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb26000d950 tx=0x7fb26000de10 comp rx=0 tx=0).stop 2024-03-19T02:55:28.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.097+0000 7fb2753e6640 1 -- 172.21.15.33:0/1182115125 shutdown_connections 2024-03-19T02:55:28.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.097+0000 7fb2753e6640 1 --2- 172.21.15.33:0/1182115125 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fb2580a2a70 0x7fb2580a4f30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.097+0000 7fb2753e6640 1 --2- 172.21.15.33:0/1182115125 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb270102180 0x7fb2701983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.097+0000 7fb2753e6640 1 --2- 172.21.15.33:0/1182115125 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb2701077d0 0x7fb27019c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.097+0000 7fb2753e6640 1 --2- 172.21.15.33:0/1182115125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2701017c0 0x7fb270197eb0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.097+0000 7fb2753e6640 1 -- 172.21.15.33:0/1182115125 >> 172.21.15.33:0/1182115125 conn(0x7fb2700fd2d0 msgr2=0x7fb2700fd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:28.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.098+0000 7fb2753e6640 1 -- 172.21.15.33:0/1182115125 shutdown_connections 2024-03-19T02:55:28.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.098+0000 7fb2753e6640 1 -- 172.21.15.33:0/1182115125 wait complete. 2024-03-19T02:55:28.115 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:55:28.266 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:28 smithi033 ceph-mon[31973]: pgmap v63: 129 pgs: 129 active+clean; 7.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 513 MiB/s wr, 156 op/s 2024-03-19T02:55:28.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.291+0000 7f6e97697640 1 -- 172.21.15.33:0/2078098901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e901103b0 msgr2=0x7f6e9010e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:28.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.291+0000 7f6e97697640 1 --2- 172.21.15.33:0/2078098901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e901103b0 0x7f6e9010e0a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6e80009900 tx=0x7f6e8002bc50 comp rx=0 tx=0).stop 2024-03-19T02:55:28.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.291+0000 7f6e97697640 1 -- 172.21.15.33:0/2078098901 shutdown_connections 2024-03-19T02:55:28.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.291+0000 7f6e97697640 1 --2- 172.21.15.33:0/2078098901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e901103b0 0x7f6e9010e0a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.291+0000 7f6e97697640 1 --2- 172.21.15.33:0/2078098901 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e9011b000 0x7f6e9011d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.291+0000 7f6e97697640 1 --2- 172.21.15.33:0/2078098901 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e9010ff40 0x7f6e9010db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.291+0000 7f6e97697640 1 -- 172.21.15.33:0/2078098901 >> 172.21.15.33:0/2078098901 conn(0x7f6e9006c910 msgr2=0x7f6e9006cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:28.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.291+0000 7f6e97697640 1 -- 172.21.15.33:0/2078098901 shutdown_connections 2024-03-19T02:55:28.293 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.291+0000 7f6e97697640 1 -- 172.21.15.33:0/2078098901 wait complete. 2024-03-19T02:55:28.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.292+0000 7f6e97697640 1 Processor -- start 2024-03-19T02:55:28.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.292+0000 7f6e97697640 1 -- start start 2024-03-19T02:55:28.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.292+0000 7f6e97697640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e9010ff40 0x7f6e901a6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.292+0000 7f6e97697640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e901103b0 0x7f6e901a7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.292+0000 7f6e97697640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e9011b000 0x7f6e901ab790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.292+0000 7f6e97697640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6e901208a0 con 0x7f6e901103b0 2024-03-19T02:55:28.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.292+0000 7f6e97697640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6e90120720 con 0x7f6e9011b000 2024-03-19T02:55:28.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.292+0000 7f6e97697640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6e90120a20 con 0x7f6e9010ff40 2024-03-19T02:55:28.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.293+0000 7f6e94c32640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e901103b0 0x7f6e901a7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:28.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.293+0000 7f6e94c32640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e901103b0 0x7f6e901a7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55434/0 (socket says 172.21.15.33:55434) 2024-03-19T02:55:28.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.293+0000 7f6e94c32640 1 -- 172.21.15.33:0/283551194 learned_addr learned my addr 172.21.15.33:0/283551194 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:28.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.293+0000 7f6e94c32640 1 -- 172.21.15.33:0/283551194 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e9010ff40 msgr2=0x7f6e901a6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:28.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.293+0000 7f6e94c32640 1 --2- 172.21.15.33:0/283551194 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e9010ff40 0x7f6e901a6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.293+0000 7f6e94c32640 1 -- 172.21.15.33:0/283551194 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e9011b000 msgr2=0x7f6e901ab790 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-19T02:55:28.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.293+0000 7f6e94c32640 1 --2- 172.21.15.33:0/283551194 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e9011b000 0x7f6e901ab790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.293+0000 7f6e94c32640 1 -- 172.21.15.33:0/283551194 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e901abec0 con 0x7f6e901103b0 2024-03-19T02:55:28.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.293+0000 7f6e94c32640 1 --2- 172.21.15.33:0/283551194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e901103b0 0x7f6e901a7410 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f6e8002c190 tx=0x7f6e80005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:28.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.294+0000 7f6e867fc640 1 -- 172.21.15.33:0/283551194 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e80043020 con 0x7f6e901103b0 2024-03-19T02:55:28.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.294+0000 7f6e97697640 1 -- 172.21.15.33:0/283551194 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e901ac150 con 0x7f6e901103b0 2024-03-19T02:55:28.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.294+0000 7f6e867fc640 1 -- 172.21.15.33:0/283551194 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6e8002cb10 con 0x7f6e901103b0 2024-03-19T02:55:28.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.295+0000 7f6e867fc640 1 -- 172.21.15.33:0/283551194 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e80003ac0 con 0x7f6e901103b0 2024-03-19T02:55:28.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.295+0000 7f6e97697640 1 -- 172.21.15.33:0/283551194 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e901b3a40 con 0x7f6e901103b0 2024-03-19T02:55:28.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.298+0000 7f6e867fc640 1 -- 172.21.15.33:0/283551194 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f6e80046020 con 0x7f6e901103b0 2024-03-19T02:55:28.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.299+0000 7f6e9a281740 1 Processor -- start 2024-03-19T02:55:28.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.300+0000 7f6e867fc640 1 --2- 172.21.15.33:0/283551194 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6e7c0a2970 0x7f6e7c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.300+0000 7f6e867fc640 1 -- 172.21.15.33:0/283551194 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f6e800f1330 con 0x7f6e901103b0 2024-03-19T02:55:28.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.300+0000 7f6e95c34640 1 --2- 172.21.15.33:0/283551194 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6e7c0a2970 0x7f6e7c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:28.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.300+0000 7f6e9a281740 1 -- start start 2024-03-19T02:55:28.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.301+0000 7f6e9a281740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b8170932d0 0x55b8170936b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.301+0000 7f6e9a281740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b817093bf0 0x55b81709c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.301+0000 7f6e95c34640 1 --2- 172.21.15.33:0/283551194 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6e7c0a2970 0x7f6e7c0a4e30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6e9006ac50 tx=0x7f6e88009230 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:28.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.301+0000 7f6e9a281740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b81709c7d0 0x55b81709ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.301+0000 7f6e9a281740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b816ff64b0 con 0x55b817093bf0 2024-03-19T02:55:28.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.301+0000 7f6e9a281740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55b816f9ed70 con 0x55b8170932d0 2024-03-19T02:55:28.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.301+0000 7f6e9a281740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b816ff2550 con 0x55b81709c7d0 2024-03-19T02:55:28.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.301+0000 7f6e94c32640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b817093bf0 0x55b81709c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:28.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.301+0000 7f6e94c32640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b817093bf0 0x55b81709c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55448/0 (socket says 172.21.15.33:55448) 2024-03-19T02:55:28.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.301+0000 7f6e94c32640 1 -- 172.21.15.33:0/3043571166 learned_addr learned my addr 172.21.15.33:0/3043571166 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:28.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.301+0000 7f6e95c34640 1 --2- 172.21.15.33:0/3043571166 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b81709c7d0 0x55b81709ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:28.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.301+0000 7f6e94c32640 1 -- 172.21.15.33:0/3043571166 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b81709c7d0 msgr2=0x55b81709ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:28.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.301+0000 7f6e94c32640 1 --2- 172.21.15.33:0/3043571166 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b81709c7d0 0x55b81709ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.301+0000 7f6e94c32640 1 -- 172.21.15.33:0/3043571166 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b8170932d0 msgr2=0x55b8170936b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:28.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.301+0000 7f6e94c32640 1 --2- 172.21.15.33:0/3043571166 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b8170932d0 0x55b8170936b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.301+0000 7f6e94c32640 1 -- 172.21.15.33:0/3043571166 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b816fc1210 con 0x55b817093bf0 2024-03-19T02:55:28.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.301+0000 7f6e94c32640 1 --2- 172.21.15.33:0/3043571166 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b817093bf0 0x55b81709c290 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f6e800ba100 tx=0x7f6e8002ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d54977b5bf8227e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:28.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.302+0000 7f6e7b7fe640 1 -- 172.21.15.33:0/3043571166 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e800d6070 con 0x55b817093bf0 2024-03-19T02:55:28.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.302+0000 7f6e7b7fe640 1 -- 172.21.15.33:0/3043571166 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6e800f1160 con 0x55b817093bf0 2024-03-19T02:55:28.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.302+0000 7f6e7b7fe640 1 -- 172.21.15.33:0/3043571166 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e800eee60 con 0x55b817093bf0 2024-03-19T02:55:28.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.302+0000 7f6e9a281740 1 -- 172.21.15.33:0/3043571166 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b817093bf0 msgr2=0x55b81709c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:28.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.302+0000 7f6e9a281740 1 --2- 172.21.15.33:0/3043571166 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b817093bf0 0x55b81709c290 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f6e800ba100 tx=0x7f6e8002ccc0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.302+0000 7f6e9a281740 1 -- 172.21.15.33:0/3043571166 shutdown_connections 2024-03-19T02:55:28.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.302+0000 7f6e9a281740 1 --2- 172.21.15.33:0/3043571166 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b81709c7d0 0x55b81709ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.302+0000 7f6e9a281740 1 --2- 172.21.15.33:0/3043571166 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b817093bf0 0x55b81709c290 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.302+0000 7f6e9a281740 1 --2- 172.21.15.33:0/3043571166 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b8170932d0 0x55b8170936b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.302+0000 7f6e9a281740 1 -- 172.21.15.33:0/3043571166 >> 172.21.15.33:0/3043571166 conn(0x55b81708da00 msgr2=0x55b81708dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:28.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.303+0000 7f6e9a281740 1 -- 172.21.15.33:0/3043571166 shutdown_connections 2024-03-19T02:55:28.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.303+0000 7f6e9a281740 1 -- 172.21.15.33:0/3043571166 wait complete. 2024-03-19T02:55:28.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.303+0000 7f6e9a281740 1 -- start start 2024-03-19T02:55:28.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.303+0000 7f6e9a281740 1 Processor -- start 2024-03-19T02:55:28.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.304+0000 7f6e9a281740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b8170932d0 0x55b817099890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.304+0000 7f6e9a281740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b817093bf0 0x55b817099dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.304+0000 7f6e9a281740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b81709c7d0 0x55b81709a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.304+0000 7f6e9a281740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b81709f380 con 0x55b81709c7d0 2024-03-19T02:55:28.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.304+0000 7f6e9a281740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55b816fb2310 con 0x55b817093bf0 2024-03-19T02:55:28.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.304+0000 7f6e9a281740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b81709f500 con 0x55b8170932d0 2024-03-19T02:55:28.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.304+0000 7f6e9a281740 10 client.0 _ll_register_callbacks cb 0x55b816fa62d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:55:28.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.304+0000 7f6e95c34640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b81709c7d0 0x55b81709a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:28.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.304+0000 7f6e95c34640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b81709c7d0 0x55b81709a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55450/0 (socket says 172.21.15.33:55450) 2024-03-19T02:55:28.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.304+0000 7f6e95c34640 1 -- 172.21.15.33:0/423665056 learned_addr learned my addr 172.21.15.33:0/423665056 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:28.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.304+0000 7f6e94c32640 1 --2- 172.21.15.33:0/423665056 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b817093bf0 0x55b817099dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:28.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.305+0000 7f6e95433640 1 --2- 172.21.15.33:0/423665056 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b8170932d0 0x55b817099890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:28.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.305+0000 7f6e95c34640 1 -- 172.21.15.33:0/423665056 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b8170932d0 msgr2=0x55b817099890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:28.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.305+0000 7f6e95c34640 1 --2- 172.21.15.33:0/423665056 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b8170932d0 0x55b817099890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.305+0000 7f6e95c34640 1 -- 172.21.15.33:0/423665056 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b817093bf0 msgr2=0x55b817099dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:28.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.305+0000 7f6e95c34640 1 --2- 172.21.15.33:0/423665056 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b817093bf0 0x55b817099dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.305+0000 7f6e95c34640 1 -- 172.21.15.33:0/423665056 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b816d0a3f0 con 0x55b81709c7d0 2024-03-19T02:55:28.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.305+0000 7f6e95c34640 1 --2- 172.21.15.33:0/423665056 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b81709c7d0 0x55b81709a310 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f6e8800ddd0 tx=0x7f6e88004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=771387bb6a1bf116 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:28.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.305+0000 7f6e78ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:55:28.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.306+0000 7f6e78ff9640 1 -- 172.21.15.33:0/423665056 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e8800cea0 con 0x55b81709c7d0 2024-03-19T02:55:28.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.306+0000 7f6e9a281740 1 -- 172.21.15.33:0/423665056 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e800f1160 con 0x55b81709c7d0 2024-03-19T02:55:28.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.306+0000 7f6e78ff9640 1 -- 172.21.15.33:0/423665056 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f6e8801b430 con 0x55b81709c7d0 2024-03-19T02:55:28.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.306+0000 7f6e78ff9640 1 -- 172.21.15.33:0/423665056 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f6e8801ae90 con 0x55b81709c7d0 2024-03-19T02:55:28.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.306+0000 7f6e9a281740 10 client.16110 fetch_fsmap learned FSMap version 25 2024-03-19T02:55:28.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.306+0000 7f6e9a281740 1 -- 172.21.15.33:0/423665056 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b816dd47d0 con 0x55b81709c7d0 2024-03-19T02:55:28.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.307+0000 7f6e78ff9640 1 -- 172.21.15.33:0/423665056 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6e800f1160 con 0x55b81709c7d0 2024-03-19T02:55:28.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.307+0000 7f6e78ff9640 1 -- 172.21.15.33:0/423665056 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f6e880248a0 con 0x55b81709c7d0 2024-03-19T02:55:28.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.307+0000 7f6e9a281740 10 client.16110 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:55:28.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.307+0000 7f6e9a281740 10 client.16110 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:55:28.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.307+0000 7f6e9a281740 20 client.16110 populate_metadata read hostname 'smithi033' 2024-03-19T02:55:28.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.307+0000 7f6e9a281740 1 --2- 172.21.15.33:0/423665056 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55b8170ac840 0x55b8170aec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.307+0000 7f6e9a281740 4 client.16110 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:55:28.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.307+0000 7f6e9a281740 1 -- 172.21.15.33:0/423665056 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b8170543e0 con 0x55b8170ac840 2024-03-19T02:55:28.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.308+0000 7f6e95433640 1 --2- 172.21.15.33:0/423665056 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55b8170ac840 0x55b8170aec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:28.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.309+0000 7f6e78ff9640 10 client.16110 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:55:28.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.309+0000 7f6e95433640 1 --2- 172.21.15.33:0/423665056 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55b8170ac840 0x55b8170aec60 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a45e1c78abc89a1f server_cookie=b53e1e602d82f07f in_seq=0 out_seq=0 2024-03-19T02:55:28.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.310+0000 7f6e78ff9640 1 -- 172.21.15.33:0/423665056 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b8170543e0 con 0x55b8170ac840 2024-03-19T02:55:28.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.310+0000 7f6e78ff9640 10 client.16110 handle_command_reply: tid=0 2024-03-19T02:55:28.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.310+0000 7f6e9a281740 1 client.16110 shutdown 2024-03-19T02:55:28.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.310+0000 7f6e9a281740 1 -- 172.21.15.33:0/423665056 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b81709c7d0 msgr2=0x55b81709a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:28.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.310+0000 7f6e9a281740 1 --2- 172.21.15.33:0/423665056 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b81709c7d0 0x55b81709a310 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f6e8800ddd0 tx=0x7f6e88004af0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.310+0000 7f6e9a281740 1 -- 172.21.15.33:0/423665056 shutdown_connections 2024-03-19T02:55:28.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.310+0000 7f6e9a281740 1 --2- 172.21.15.33:0/423665056 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55b8170ac840 0x55b8170aec60 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.311+0000 7f6e78ff9640 0 client.16110 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:55:28.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.311+0000 7f6e9a281740 1 --2- 172.21.15.33:0/423665056 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b81709c7d0 0x55b81709a310 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.311+0000 7f6e9a281740 1 --2- 172.21.15.33:0/423665056 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b817093bf0 0x55b817099dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.311+0000 7f6e9a281740 1 --2- 172.21.15.33:0/423665056 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b8170932d0 0x55b817099890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.311+0000 7f6e9a281740 1 -- 172.21.15.33:0/423665056 >> 172.21.15.33:0/423665056 conn(0x55b81708da00 msgr2=0x55b81708f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:28.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.311+0000 7f6e9a281740 1 -- 172.21.15.33:0/423665056 shutdown_connections 2024-03-19T02:55:28.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.311+0000 7f6e9a281740 1 -- 172.21.15.33:0/423665056 wait complete. 2024-03-19T02:55:28.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.312+0000 7f6e9a281740 20 client.16110 trim_cache size 0 max 16384 2024-03-19T02:55:28.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.334+0000 7f6e9a281740 1 Processor -- start 2024-03-19T02:55:28.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.334+0000 7f6e9a281740 1 -- start start 2024-03-19T02:55:28.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.335+0000 7f6e9a281740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b8170932d0 0x55b81708ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.335+0000 7f6e9a281740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b817093bf0 0x55b8170a7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.335+0000 7f6e9a281740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b81709c7d0 0x55b8170b44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.335+0000 7f6e9a281740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6e901c02c0 con 0x55b81709c7d0 2024-03-19T02:55:28.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.335+0000 7f6e9a281740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6e901bdb90 con 0x55b8170932d0 2024-03-19T02:55:28.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.335+0000 7f6e9a281740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6e901bb9c0 con 0x55b817093bf0 2024-03-19T02:55:28.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.335+0000 7f6e95c34640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b81709c7d0 0x55b8170b44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:28.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.335+0000 7f6e95c34640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b81709c7d0 0x55b8170b44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55462/0 (socket says 172.21.15.33:55462) 2024-03-19T02:55:28.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.335+0000 7f6e95c34640 1 -- 172.21.15.33:0/3501356457 learned_addr learned my addr 172.21.15.33:0/3501356457 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:28.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.335+0000 7f6e95433640 1 --2- 172.21.15.33:0/3501356457 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b8170932d0 0x55b81708ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:28.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.335+0000 7f6e94c32640 1 --2- 172.21.15.33:0/3501356457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b817093bf0 0x55b8170a7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:28.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.336+0000 7f6e95c34640 1 -- 172.21.15.33:0/3501356457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b817093bf0 msgr2=0x55b8170a7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:28.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.336+0000 7f6e95c34640 1 --2- 172.21.15.33:0/3501356457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b817093bf0 0x55b8170a7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.336+0000 7f6e95c34640 1 -- 172.21.15.33:0/3501356457 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b8170932d0 msgr2=0x55b81708ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:28.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.336+0000 7f6e95c34640 1 --2- 172.21.15.33:0/3501356457 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b8170932d0 0x55b81708ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.336+0000 7f6e95c34640 1 -- 172.21.15.33:0/3501356457 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b817063880 con 0x55b81709c7d0 2024-03-19T02:55:28.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.336+0000 7f6e95c34640 1 --2- 172.21.15.33:0/3501356457 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b81709c7d0 0x55b8170b44f0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f6e8800d820 tx=0x7f6e8800c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9607e8df6aa0a63 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:28.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.337+0000 7f6e7affd640 1 -- 172.21.15.33:0/3501356457 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e88037020 con 0x55b81709c7d0 2024-03-19T02:55:28.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.337+0000 7f6e7affd640 1 -- 172.21.15.33:0/3501356457 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6e8801ac20 con 0x55b81709c7d0 2024-03-19T02:55:28.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.337+0000 7f6e7affd640 1 -- 172.21.15.33:0/3501356457 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e88032b70 con 0x55b81709c7d0 2024-03-19T02:55:28.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.338+0000 7f6e9a281740 1 -- 172.21.15.33:0/3501356457 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b81709c7d0 msgr2=0x55b8170b44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:28.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.338+0000 7f6e9a281740 1 --2- 172.21.15.33:0/3501356457 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b81709c7d0 0x55b8170b44f0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f6e8800d820 tx=0x7f6e8800c930 comp rx=0 tx=0).stop 2024-03-19T02:55:28.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.338+0000 7f6e9a281740 1 -- 172.21.15.33:0/3501356457 shutdown_connections 2024-03-19T02:55:28.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.338+0000 7f6e9a281740 1 --2- 172.21.15.33:0/3501356457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b817093bf0 0x55b8170a7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.338+0000 7f6e9a281740 1 --2- 172.21.15.33:0/3501356457 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b81709c7d0 0x55b8170b44f0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.338+0000 7f6e9a281740 1 --2- 172.21.15.33:0/3501356457 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b8170932d0 0x55b81708ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.338+0000 7f6e9a281740 1 -- 172.21.15.33:0/3501356457 >> 172.21.15.33:0/3501356457 conn(0x55b81708da00 msgr2=0x55b81708c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:28.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.338+0000 7f6e9a281740 1 -- 172.21.15.33:0/3501356457 shutdown_connections 2024-03-19T02:55:28.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.338+0000 7f6e9a281740 1 -- 172.21.15.33:0/3501356457 wait complete. 2024-03-19T02:55:28.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.339+0000 7f6e9a281740 1 -- start start 2024-03-19T02:55:28.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.339+0000 7f6e9a281740 1 Processor -- start 2024-03-19T02:55:28.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.339+0000 7f6e9a281740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b8170932d0 0x55b8170a4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.339+0000 7f6e9a281740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b817093bf0 0x55b81709fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.339+0000 7f6e9a281740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b81709c7d0 0x55b8170a02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.339+0000 7f6e9a281740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6e901bc050 con 0x55b817093bf0 2024-03-19T02:55:28.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.339+0000 7f6e9a281740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6e901bd4c0 con 0x55b8170932d0 2024-03-19T02:55:28.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.339+0000 7f6e9a281740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6e901bc720 con 0x55b81709c7d0 2024-03-19T02:55:28.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.340+0000 7f6e9a281740 10 client.0 _ll_register_callbacks cb 0x7f6e8800cbb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:55:28.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.340+0000 7f6e94c32640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b817093bf0 0x55b81709fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:28.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.340+0000 7f6e94c32640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b817093bf0 0x55b81709fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55466/0 (socket says 172.21.15.33:55466) 2024-03-19T02:55:28.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.340+0000 7f6e94c32640 1 -- 172.21.15.33:0/1998686786 learned_addr learned my addr 172.21.15.33:0/1998686786 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:28.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.340+0000 7f6e95433640 1 --2- 172.21.15.33:0/1998686786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b8170932d0 0x55b8170a4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:28.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.340+0000 7f6e94c32640 1 -- 172.21.15.33:0/1998686786 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b81709c7d0 msgr2=0x55b8170a02a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T02:55:28.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.340+0000 7f6e94c32640 1 --2- 172.21.15.33:0/1998686786 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b81709c7d0 0x55b8170a02a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.340+0000 7f6e94c32640 1 -- 172.21.15.33:0/1998686786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b8170932d0 msgr2=0x55b8170a4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:28.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.340+0000 7f6e94c32640 1 --2- 172.21.15.33:0/1998686786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b8170932d0 0x55b8170a4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.340+0000 7f6e94c32640 1 -- 172.21.15.33:0/1998686786 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8170a0ad0 con 0x55b817093bf0 2024-03-19T02:55:28.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.340+0000 7f6e95433640 1 --2- 172.21.15.33:0/1998686786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b8170932d0 0x55b8170a4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:55:28.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.341+0000 7f6e94c32640 1 --2- 172.21.15.33:0/1998686786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b817093bf0 0x55b81709fc90 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f6e80005ec0 tx=0x7f6e8002bc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8993f9707da1550 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:28.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.341+0000 7f6e78ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:55:28.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.341+0000 7f6e78ff9640 1 -- 172.21.15.33:0/1998686786 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e80087060 con 0x55b817093bf0 2024-03-19T02:55:28.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.341+0000 7f6e78ff9640 1 -- 172.21.15.33:0/1998686786 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f6e800c5a40 con 0x55b817093bf0 2024-03-19T02:55:28.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.341+0000 7f6e9a281740 1 -- 172.21.15.33:0/1998686786 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e8801ac20 con 0x55b817093bf0 2024-03-19T02:55:28.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.341+0000 7f6e78ff9640 1 -- 172.21.15.33:0/1998686786 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f6e800eee60 con 0x55b817093bf0 2024-03-19T02:55:28.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.341+0000 7f6e9a281740 10 client.16116 fetch_fsmap learned FSMap version 25 2024-03-19T02:55:28.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.341+0000 7f6e9a281740 1 -- 172.21.15.33:0/1998686786 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b817112c20 con 0x55b817093bf0 2024-03-19T02:55:28.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.341+0000 7f6e78ff9640 1 -- 172.21.15.33:0/1998686786 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6e8801ac20 con 0x55b817093bf0 2024-03-19T02:55:28.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.341+0000 7f6e78ff9640 1 -- 172.21.15.33:0/1998686786 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f6e800d1b10 con 0x55b817093bf0 2024-03-19T02:55:28.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.342+0000 7f6e9a281740 10 client.16116 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:55:28.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.342+0000 7f6e9a281740 10 client.16116 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:55:28.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.342+0000 7f6e9a281740 20 client.16116 populate_metadata read hostname 'smithi033' 2024-03-19T02:55:28.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.342+0000 7f6e9a281740 1 --2- 172.21.15.33:0/1998686786 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55b8170ac840 0x55b817114fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:28.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.342+0000 7f6e9a281740 4 client.16116 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:55:28.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.342+0000 7f6e9a281740 1 -- 172.21.15.33:0/1998686786 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b8170039f0 con 0x55b8170ac840 2024-03-19T02:55:28.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.342+0000 7f6e95433640 1 --2- 172.21.15.33:0/1998686786 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55b8170ac840 0x55b817114fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:28.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.345+0000 7f6e78ff9640 10 client.16116 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:55:28.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.345+0000 7f6e95433640 1 --2- 172.21.15.33:0/1998686786 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55b8170ac840 0x55b817114fd0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2547c5a2810b3b7 server_cookie=9cc2abe3ee3dc83f in_seq=0 out_seq=0 2024-03-19T02:55:28.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.345+0000 7f6e78ff9640 1 -- 172.21.15.33:0/1998686786 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b8170039f0 con 0x55b8170ac840 2024-03-19T02:55:28.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.345+0000 7f6e78ff9640 10 client.16116 handle_command_reply: tid=0 2024-03-19T02:55:28.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.345+0000 7f6e9a281740 1 client.16116 shutdown 2024-03-19T02:55:28.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.345+0000 7f6e9a281740 1 -- 172.21.15.33:0/1998686786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b817093bf0 msgr2=0x55b81709fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:28.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.345+0000 7f6e9a281740 1 --2- 172.21.15.33:0/1998686786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b817093bf0 0x55b81709fc90 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f6e80005ec0 tx=0x7f6e8002bc50 comp rx=0 tx=0).stop 2024-03-19T02:55:28.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.345+0000 7f6e9a281740 1 -- 172.21.15.33:0/1998686786 shutdown_connections 2024-03-19T02:55:28.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.345+0000 7f6e9a281740 1 --2- 172.21.15.33:0/1998686786 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55b8170ac840 0x55b817114fd0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.345+0000 7f6e9a281740 1 --2- 172.21.15.33:0/1998686786 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b81709c7d0 0x55b8170a02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.345+0000 7f6e9a281740 1 --2- 172.21.15.33:0/1998686786 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b817093bf0 0x55b81709fc90 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.345+0000 7f6e9a281740 1 --2- 172.21.15.33:0/1998686786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b8170932d0 0x55b8170a4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.345+0000 7f6e9a281740 1 -- 172.21.15.33:0/1998686786 >> 172.21.15.33:0/1998686786 conn(0x55b81708da00 msgr2=0x55b81708cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:28.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.345+0000 7f6e78ff9640 0 client.16116 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:55:28.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.346+0000 7f6e9a281740 1 -- 172.21.15.33:0/1998686786 shutdown_connections 2024-03-19T02:55:28.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.346+0000 7f6e9a281740 1 -- 172.21.15.33:0/1998686786 wait complete. 2024-03-19T02:55:28.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.346+0000 7f6e9a281740 20 client.16116 trim_cache size 0 max 16384 2024-03-19T02:55:28.349 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:55:28.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.346+0000 7f6e97697640 1 -- 172.21.15.33:0/283551194 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6e7c0a2970 msgr2=0x7f6e7c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:28.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.346+0000 7f6e97697640 1 --2- 172.21.15.33:0/283551194 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6e7c0a2970 0x7f6e7c0a4e30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6e9006ac50 tx=0x7f6e88009230 comp rx=0 tx=0).stop 2024-03-19T02:55:28.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.346+0000 7f6e97697640 1 -- 172.21.15.33:0/283551194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e901103b0 msgr2=0x7f6e901a7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:28.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.347+0000 7f6e97697640 1 --2- 172.21.15.33:0/283551194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e901103b0 0x7f6e901a7410 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f6e8002c190 tx=0x7f6e80005e50 comp rx=0 tx=0).stop 2024-03-19T02:55:28.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.347+0000 7f6e97697640 1 -- 172.21.15.33:0/283551194 shutdown_connections 2024-03-19T02:55:28.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.347+0000 7f6e97697640 1 --2- 172.21.15.33:0/283551194 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6e7c0a2970 0x7f6e7c0a4e30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.347+0000 7f6e97697640 1 --2- 172.21.15.33:0/283551194 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e9011b000 0x7f6e901ab790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.347+0000 7f6e97697640 1 --2- 172.21.15.33:0/283551194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6e901103b0 0x7f6e901a7410 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.347+0000 7f6e97697640 1 --2- 172.21.15.33:0/283551194 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e9010ff40 0x7f6e901a6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:28.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.347+0000 7f6e97697640 1 -- 172.21.15.33:0/283551194 >> 172.21.15.33:0/283551194 conn(0x7f6e9006c910 msgr2=0x7f6e90119b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:28.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.347+0000 7f6e97697640 1 -- 172.21.15.33:0/283551194 shutdown_connections 2024-03-19T02:55:28.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:28.347+0000 7f6e97697640 1 -- 172.21.15.33:0/283551194 wait complete. 2024-03-19T02:55:28.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:28 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1573335315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:55:28.570 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-03-19T02:55:28.571 INFO:tasks.workunit.client.0.smithi033.stdout: Children see throughput for 1 initial writers = 625389.50 kB/sec 2024-03-19T02:55:28.571 INFO:tasks.workunit.client.0.smithi033.stdout: Parent sees throughput for 1 initial writers = 625378.09 kB/sec 2024-03-19T02:55:28.571 INFO:tasks.workunit.client.0.smithi033.stdout: Min throughput per process = 625389.50 kB/sec 2024-03-19T02:55:28.571 INFO:tasks.workunit.client.0.smithi033.stdout: Max throughput per process = 625389.50 kB/sec 2024-03-19T02:55:28.571 INFO:tasks.workunit.client.0.smithi033.stdout: Avg throughput per process = 625389.50 kB/sec 2024-03-19T02:55:28.571 INFO:tasks.workunit.client.0.smithi033.stdout: Min xfer = 10485760.00 kB 2024-03-19T02:55:28.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:28 smithi096 ceph-mon[35946]: pgmap v63: 129 pgs: 129 active+clean; 7.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 513 MiB/s wr, 156 op/s 2024-03-19T02:55:28.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:28 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1573335315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:55:28.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:28 smithi155 ceph-mon[35804]: pgmap v63: 129 pgs: 129 active+clean; 7.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 513 MiB/s wr, 156 op/s 2024-03-19T02:55:28.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:28 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1573335315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:55:29.367 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:55:29.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:55:29.367 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:55:29.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.486+0000 7fca1e313640 1 -- 172.21.15.33:0/2825731439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fca181033b0 msgr2=0x7fca1810b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:29.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.486+0000 7fca1e313640 1 --2- 172.21.15.33:0/2825731439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fca181033b0 0x7fca1810b890 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fca0c009900 tx=0x7fca0c02bcd0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.486+0000 7fca1e313640 1 -- 172.21.15.33:0/2825731439 shutdown_connections 2024-03-19T02:55:29.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.486+0000 7fca1e313640 1 --2- 172.21.15.33:0/2825731439 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fca1810be60 0x7fca1810e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.486+0000 7fca1e313640 1 --2- 172.21.15.33:0/2825731439 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fca181033b0 0x7fca1810b890 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.486+0000 7fca1e313640 1 --2- 172.21.15.33:0/2825731439 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fca18102a90 0x7fca18102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.486+0000 7fca1e313640 1 -- 172.21.15.33:0/2825731439 >> 172.21.15.33:0/2825731439 conn(0x7fca18077880 msgr2=0x7fca18077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:29.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.487+0000 7fca1e313640 1 -- 172.21.15.33:0/2825731439 shutdown_connections 2024-03-19T02:55:29.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.487+0000 7fca1e313640 1 -- 172.21.15.33:0/2825731439 wait complete. 2024-03-19T02:55:29.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.487+0000 7fca1e313640 1 Processor -- start 2024-03-19T02:55:29.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.487+0000 7fca1e313640 1 -- start start 2024-03-19T02:55:29.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.487+0000 7fca1e313640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fca18102a90 0x7fca18198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.488+0000 7fca1e313640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fca181033b0 0x7fca18198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.488+0000 7fca1e313640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fca1810be60 0x7fca1819c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.488+0000 7fca1e313640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fca18111ae0 con 0x7fca18102a90 2024-03-19T02:55:29.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.488+0000 7fca1e313640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fca18111960 con 0x7fca181033b0 2024-03-19T02:55:29.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.488+0000 7fca1e313640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fca18111c60 con 0x7fca1810be60 2024-03-19T02:55:29.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.488+0000 7fca17fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fca18102a90 0x7fca18198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.488+0000 7fca177fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fca181033b0 0x7fca18198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.488+0000 7fca17fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fca18102a90 0x7fca18198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55486/0 (socket says 172.21.15.33:55486) 2024-03-19T02:55:29.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.488+0000 7fca17fff640 1 -- 172.21.15.33:0/242649822 learned_addr learned my addr 172.21.15.33:0/242649822 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:29.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.488+0000 7fca177fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fca181033b0 0x7fca18198580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:45156/0 (socket says 172.21.15.33:45156) 2024-03-19T02:55:29.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.488+0000 7fca1c8b0640 1 --2- 172.21.15.33:0/242649822 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fca1810be60 0x7fca1819c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.488+0000 7fca177fe640 1 -- 172.21.15.33:0/242649822 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fca1810be60 msgr2=0x7fca1819c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:29.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.488+0000 7fca177fe640 1 --2- 172.21.15.33:0/242649822 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fca1810be60 0x7fca1819c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.488+0000 7fca177fe640 1 -- 172.21.15.33:0/242649822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fca18102a90 msgr2=0x7fca18198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:29.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.488+0000 7fca177fe640 1 --2- 172.21.15.33:0/242649822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fca18102a90 0x7fca18198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.488+0000 7fca177fe640 1 -- 172.21.15.33:0/242649822 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca1819d030 con 0x7fca181033b0 2024-03-19T02:55:29.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.488+0000 7fca17fff640 1 --2- 172.21.15.33:0/242649822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fca18102a90 0x7fca18198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:29.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.488+0000 7fca177fe640 1 --2- 172.21.15.33:0/242649822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fca181033b0 0x7fca18198580 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fca0c036040 tx=0x7fca0c0045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:29.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.488+0000 7fca157fa640 1 -- 172.21.15.33:0/242649822 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0c04a070 con 0x7fca181033b0 2024-03-19T02:55:29.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.488+0000 7fca1e313640 1 -- 172.21.15.33:0/242649822 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca1819d2c0 con 0x7fca181033b0 2024-03-19T02:55:29.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.488+0000 7fca157fa640 1 -- 172.21.15.33:0/242649822 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fca0c004e00 con 0x7fca181033b0 2024-03-19T02:55:29.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.488+0000 7fca157fa640 1 -- 172.21.15.33:0/242649822 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0c0038e0 con 0x7fca181033b0 2024-03-19T02:55:29.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.489+0000 7fca1e313640 1 -- 172.21.15.33:0/242649822 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca18100610 con 0x7fca181033b0 2024-03-19T02:55:29.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.490+0000 7fca157fa640 1 -- 172.21.15.33:0/242649822 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fca0c04d070 con 0x7fca181033b0 2024-03-19T02:55:29.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.491+0000 7fca20fcf740 1 Processor -- start 2024-03-19T02:55:29.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.491+0000 7fca20fcf740 1 -- start start 2024-03-19T02:55:29.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.491+0000 7fca157fa640 1 --2- 172.21.15.33:0/242649822 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fca000a2970 0x7fca000a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.491+0000 7fca20fcf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557be7436380 0x557be7436760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.491+0000 7fca1c8b0640 1 --2- 172.21.15.33:0/242649822 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fca000a2970 0x7fca000a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.491+0000 7fca20fcf740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557be7436ca0 0x557be743f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.491+0000 7fca157fa640 1 -- 172.21.15.33:0/242649822 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fca0c0f40a0 con 0x7fca181033b0 2024-03-19T02:55:29.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.491+0000 7fca20fcf740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557be743f880 0x557be7441c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.491+0000 7fca20fcf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557be739a160 con 0x557be7436380 2024-03-19T02:55:29.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.491+0000 7fca20fcf740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557be7342e10 con 0x557be743f880 2024-03-19T02:55:29.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.491+0000 7fca20fcf740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x557be7396200 con 0x557be7436ca0 2024-03-19T02:55:29.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.491+0000 7fca17fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557be7436380 0x557be7436760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.491+0000 7fca17fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557be7436380 0x557be7436760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55500/0 (socket says 172.21.15.33:55500) 2024-03-19T02:55:29.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.491+0000 7fca17fff640 1 -- 172.21.15.33:0/692246420 learned_addr learned my addr 172.21.15.33:0/692246420 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:29.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.491+0000 7fca1c8b0640 1 --2- 172.21.15.33:0/692246420 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557be743f880 0x557be7441c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.491+0000 7fca177fe640 1 --2- 172.21.15.33:0/692246420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557be7436ca0 0x557be743f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.492+0000 7fca17fff640 1 -- 172.21.15.33:0/692246420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557be7436ca0 msgr2=0x557be743f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:29.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.492+0000 7fca17fff640 1 --2- 172.21.15.33:0/692246420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557be7436ca0 0x557be743f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.492+0000 7fca17fff640 1 -- 172.21.15.33:0/692246420 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557be743f880 msgr2=0x557be7441c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:29.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.492+0000 7fca17fff640 1 --2- 172.21.15.33:0/692246420 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557be743f880 0x557be7441c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.492+0000 7fca17fff640 1 -- 172.21.15.33:0/692246420 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557be71767e0 con 0x557be7436380 2024-03-19T02:55:29.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.492+0000 7fca1c8b0640 1 --2- 172.21.15.33:0/242649822 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fca000a2970 0x7fca000a4e30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fca10005fd0 tx=0x7fca10011b40 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:29.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.492+0000 7fca17fff640 1 --2- 172.21.15.33:0/692246420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557be7436380 0x557be7436760 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fca08011630 tx=0x7fca08011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=955763c1e35180cf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:29.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.492+0000 7fc9fe7fc640 1 -- 172.21.15.33:0/692246420 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0801e070 con 0x557be7436380 2024-03-19T02:55:29.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.492+0000 7fc9fe7fc640 1 -- 172.21.15.33:0/692246420 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fca0800fba0 con 0x557be7436380 2024-03-19T02:55:29.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.492+0000 7fc9fe7fc640 1 -- 172.21.15.33:0/692246420 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca08019c10 con 0x557be7436380 2024-03-19T02:55:29.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.493+0000 7fca20fcf740 1 -- 172.21.15.33:0/692246420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557be7436380 msgr2=0x557be7436760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:29.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.493+0000 7fca20fcf740 1 --2- 172.21.15.33:0/692246420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557be7436380 0x557be7436760 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fca08011630 tx=0x7fca08011af0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.493+0000 7fca20fcf740 1 -- 172.21.15.33:0/692246420 shutdown_connections 2024-03-19T02:55:29.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.493+0000 7fca20fcf740 1 --2- 172.21.15.33:0/692246420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557be7436ca0 0x557be743f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.493+0000 7fca20fcf740 1 --2- 172.21.15.33:0/692246420 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557be743f880 0x557be7441c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.493+0000 7fca20fcf740 1 --2- 172.21.15.33:0/692246420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557be7436380 0x557be7436760 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.493+0000 7fca20fcf740 1 -- 172.21.15.33:0/692246420 >> 172.21.15.33:0/692246420 conn(0x557be7430ab0 msgr2=0x557be7430e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:29.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.493+0000 7fca20fcf740 1 -- 172.21.15.33:0/692246420 shutdown_connections 2024-03-19T02:55:29.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.493+0000 7fca20fcf740 1 -- 172.21.15.33:0/692246420 wait complete. 2024-03-19T02:55:29.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.493+0000 7fca20fcf740 1 -- start start 2024-03-19T02:55:29.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.493+0000 7fca20fcf740 1 Processor -- start 2024-03-19T02:55:29.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.494+0000 7fca20fcf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557be7436380 0x557be743c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.494+0000 7fca20fcf740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557be7436ca0 0x557be743cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.494+0000 7fca20fcf740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557be743f880 0x557be743d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.494+0000 7fca20fcf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557be7442670 con 0x557be7436380 2024-03-19T02:55:29.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.494+0000 7fca20fcf740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557be73561a0 con 0x557be7436ca0 2024-03-19T02:55:29.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.494+0000 7fca20fcf740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x557be74427f0 con 0x557be743f880 2024-03-19T02:55:29.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.494+0000 7fca20fcf740 10 client.0 _ll_register_callbacks cb 0x557be70adc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:55:29.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.494+0000 7fca17fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557be7436380 0x557be743c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.494+0000 7fca17fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557be7436380 0x557be743c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55502/0 (socket says 172.21.15.33:55502) 2024-03-19T02:55:29.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.494+0000 7fca17fff640 1 -- 172.21.15.33:0/2004076929 learned_addr learned my addr 172.21.15.33:0/2004076929 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:29.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.494+0000 7fca177fe640 1 --2- 172.21.15.33:0/2004076929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557be7436ca0 0x557be743cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.494+0000 7fca1c8b0640 1 --2- 172.21.15.33:0/2004076929 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557be743f880 0x557be743d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.494+0000 7fca17fff640 1 -- 172.21.15.33:0/2004076929 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557be743f880 msgr2=0x557be743d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:29.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.494+0000 7fca17fff640 1 --2- 172.21.15.33:0/2004076929 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557be743f880 0x557be743d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.494+0000 7fca17fff640 1 -- 172.21.15.33:0/2004076929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557be7436ca0 msgr2=0x557be743cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:29.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.494+0000 7fca17fff640 1 --2- 172.21.15.33:0/2004076929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557be7436ca0 0x557be743cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.494+0000 7fca17fff640 1 -- 172.21.15.33:0/2004076929 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557be71d6f70 con 0x557be7436380 2024-03-19T02:55:29.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.494+0000 7fca17fff640 1 --2- 172.21.15.33:0/2004076929 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557be7436380 0x557be743c760 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fca080197f0 tx=0x7fca08009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d44607bb3252095 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:29.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.494+0000 7fc9f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:55:29.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.495+0000 7fc9f77fe640 1 -- 172.21.15.33:0/2004076929 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca08026020 con 0x557be7436380 2024-03-19T02:55:29.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.495+0000 7fc9f77fe640 1 -- 172.21.15.33:0/2004076929 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fca0800fd50 con 0x557be7436380 2024-03-19T02:55:29.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.495+0000 7fca20fcf740 1 -- 172.21.15.33:0/2004076929 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca0800fba0 con 0x557be7436380 2024-03-19T02:55:29.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.495+0000 7fc9f77fe640 1 -- 172.21.15.33:0/2004076929 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fca08031050 con 0x557be7436380 2024-03-19T02:55:29.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.495+0000 7fca20fcf740 10 client.16128 fetch_fsmap learned FSMap version 25 2024-03-19T02:55:29.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.495+0000 7fca20fcf740 1 -- 172.21.15.33:0/2004076929 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x557be706e170 con 0x557be7436380 2024-03-19T02:55:29.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.495+0000 7fc9f77fe640 1 -- 172.21.15.33:0/2004076929 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fca0800fba0 con 0x557be7436380 2024-03-19T02:55:29.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.495+0000 7fc9f77fe640 1 -- 172.21.15.33:0/2004076929 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fca08021050 con 0x557be7436380 2024-03-19T02:55:29.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.495+0000 7fca20fcf740 10 client.16128 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:55:29.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.495+0000 7fca20fcf740 10 client.16128 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:55:29.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.495+0000 7fca20fcf740 20 client.16128 populate_metadata read hostname 'smithi033' 2024-03-19T02:55:29.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.495+0000 7fca20fcf740 1 --2- 172.21.15.33:0/2004076929 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557be744fa80 0x557be7451e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.495+0000 7fca20fcf740 4 client.16128 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:55:29.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.495+0000 7fca20fcf740 1 -- 172.21.15.33:0/2004076929 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557be73a76a0 con 0x557be744fa80 2024-03-19T02:55:29.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.495+0000 7fca177fe640 1 --2- 172.21.15.33:0/2004076929 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557be744fa80 0x557be7451e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.496+0000 7fca177fe640 1 --2- 172.21.15.33:0/2004076929 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557be744fa80 0x557be7451e70 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b35264c6ff65bba7 server_cookie=d572b424655a2b7c in_seq=0 out_seq=0 2024-03-19T02:55:29.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.496+0000 7fc9f77fe640 10 client.16128 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:55:29.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.496+0000 7fc9f77fe640 1 -- 172.21.15.33:0/2004076929 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557be73a76a0 con 0x557be744fa80 2024-03-19T02:55:29.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.496+0000 7fc9f77fe640 10 client.16128 handle_command_reply: tid=0 2024-03-19T02:55:29.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.496+0000 7fca20fcf740 1 client.16128 shutdown 2024-03-19T02:55:29.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.496+0000 7fca20fcf740 1 -- 172.21.15.33:0/2004076929 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557be7436380 msgr2=0x557be743c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:29.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.496+0000 7fca20fcf740 1 --2- 172.21.15.33:0/2004076929 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557be7436380 0x557be743c760 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fca080197f0 tx=0x7fca08009f10 comp rx=0 tx=0).stop 2024-03-19T02:55:29.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.497+0000 7fca20fcf740 1 -- 172.21.15.33:0/2004076929 shutdown_connections 2024-03-19T02:55:29.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.497+0000 7fca20fcf740 1 --2- 172.21.15.33:0/2004076929 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557be744fa80 0x557be7451e70 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.497+0000 7fca20fcf740 1 --2- 172.21.15.33:0/2004076929 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557be743f880 0x557be743d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.497+0000 7fca20fcf740 1 --2- 172.21.15.33:0/2004076929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557be7436ca0 0x557be743cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.497+0000 7fc9f77fe640 0 client.16128 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:55:29.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.497+0000 7fca20fcf740 1 --2- 172.21.15.33:0/2004076929 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557be7436380 0x557be743c760 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.497+0000 7fca20fcf740 1 -- 172.21.15.33:0/2004076929 >> 172.21.15.33:0/2004076929 conn(0x557be7430ab0 msgr2=0x557be7432800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:29.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.497+0000 7fca20fcf740 1 -- 172.21.15.33:0/2004076929 shutdown_connections 2024-03-19T02:55:29.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.497+0000 7fca20fcf740 1 -- 172.21.15.33:0/2004076929 wait complete. 2024-03-19T02:55:29.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.497+0000 7fca20fcf740 20 client.16128 trim_cache size 0 max 16384 2024-03-19T02:55:29.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.514+0000 7fca20fcf740 1 Processor -- start 2024-03-19T02:55:29.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.514+0000 7fca20fcf740 1 -- start start 2024-03-19T02:55:29.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.515+0000 7fca20fcf740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557be7436380 0x557be743ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.515+0000 7fca20fcf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557be7436ca0 0x557be744b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.515+0000 7fca20fcf740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557be743f880 0x557be742f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.515+0000 7fca20fcf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fca181af170 con 0x557be7436ca0 2024-03-19T02:55:29.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.515+0000 7fca20fcf740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fca181aca40 con 0x557be7436380 2024-03-19T02:55:29.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.515+0000 7fca20fcf740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fca181aa870 con 0x557be743f880 2024-03-19T02:55:29.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.515+0000 7fca177fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557be7436ca0 0x557be744b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.515+0000 7fca177fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557be7436ca0 0x557be744b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55518/0 (socket says 172.21.15.33:55518) 2024-03-19T02:55:29.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.515+0000 7fca177fe640 1 -- 172.21.15.33:0/352948361 learned_addr learned my addr 172.21.15.33:0/352948361 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:29.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.515+0000 7fca1c8b0640 1 --2- 172.21.15.33:0/352948361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557be743f880 0x557be742f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.515+0000 7fca17fff640 1 --2- 172.21.15.33:0/352948361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557be7436380 0x557be743ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.515+0000 7fca177fe640 1 -- 172.21.15.33:0/352948361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557be743f880 msgr2=0x557be742f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:29.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.515+0000 7fca177fe640 1 --2- 172.21.15.33:0/352948361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557be743f880 0x557be742f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.515+0000 7fca177fe640 1 -- 172.21.15.33:0/352948361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557be7436380 msgr2=0x557be743ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:29.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.515+0000 7fca177fe640 1 --2- 172.21.15.33:0/352948361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557be7436380 0x557be743ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.515+0000 7fca177fe640 1 -- 172.21.15.33:0/352948361 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557be74308e0 con 0x557be7436ca0 2024-03-19T02:55:29.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.515+0000 7fca17fff640 1 --2- 172.21.15.33:0/352948361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557be7436380 0x557be743ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:55:29.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.515+0000 7fca177fe640 1 --2- 172.21.15.33:0/352948361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557be7436ca0 0x557be744b440 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fca0c006010 tx=0x7fca0c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a393d001ec3a2ab server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:29.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.515+0000 7fc9fdffb640 1 -- 172.21.15.33:0/352948361 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0c0a9070 con 0x557be7436ca0 2024-03-19T02:55:29.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.515+0000 7fc9fdffb640 1 -- 172.21.15.33:0/352948361 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fca0c0c8c40 con 0x557be7436ca0 2024-03-19T02:55:29.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.515+0000 7fc9fdffb640 1 -- 172.21.15.33:0/352948361 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0c0052b0 con 0x557be7436ca0 2024-03-19T02:55:29.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.516+0000 7fca20fcf740 1 -- 172.21.15.33:0/352948361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557be7436ca0 msgr2=0x557be744b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:29.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.516+0000 7fca20fcf740 1 --2- 172.21.15.33:0/352948361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557be7436ca0 0x557be744b440 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fca0c006010 tx=0x7fca0c000bc0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.516+0000 7fca20fcf740 1 -- 172.21.15.33:0/352948361 shutdown_connections 2024-03-19T02:55:29.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.516+0000 7fca20fcf740 1 --2- 172.21.15.33:0/352948361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557be743f880 0x557be742f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.516+0000 7fca20fcf740 1 --2- 172.21.15.33:0/352948361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557be7436ca0 0x557be744b440 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.516+0000 7fca20fcf740 1 --2- 172.21.15.33:0/352948361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557be7436380 0x557be743ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.516+0000 7fca20fcf740 1 -- 172.21.15.33:0/352948361 >> 172.21.15.33:0/352948361 conn(0x557be7430ab0 msgr2=0x557be742f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:29.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.516+0000 7fca20fcf740 1 -- 172.21.15.33:0/352948361 shutdown_connections 2024-03-19T02:55:29.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.516+0000 7fca20fcf740 1 -- 172.21.15.33:0/352948361 wait complete. 2024-03-19T02:55:29.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.516+0000 7fca20fcf740 1 -- start start 2024-03-19T02:55:29.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.517+0000 7fca20fcf740 1 Processor -- start 2024-03-19T02:55:29.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.517+0000 7fca20fcf740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557be7436380 0x557be7447d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.517+0000 7fca20fcf740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557be7436ca0 0x557be7438ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.517+0000 7fca20fcf740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557be743f880 0x557be7439530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.517+0000 7fca20fcf740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fca181aaf00 con 0x557be7436ca0 2024-03-19T02:55:29.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.517+0000 7fca20fcf740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fca181ac370 con 0x557be7436380 2024-03-19T02:55:29.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.517+0000 7fca20fcf740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fca181ab5d0 con 0x557be743f880 2024-03-19T02:55:29.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.517+0000 7fca20fcf740 10 client.0 _ll_register_callbacks cb 0x7fca0801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:55:29.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.517+0000 7fca177fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557be7436ca0 0x557be7438ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.517+0000 7fca17fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557be7436380 0x557be7447d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.517+0000 7fca17fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557be7436380 0x557be7447d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:45186/0 (socket says 172.21.15.33:45186) 2024-03-19T02:55:29.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.517+0000 7fca177fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557be7436ca0 0x557be7438ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55524/0 (socket says 172.21.15.33:55524) 2024-03-19T02:55:29.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.517+0000 7fca17fff640 1 -- 172.21.15.33:0/2520262193 learned_addr learned my addr 172.21.15.33:0/2520262193 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:29.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.517+0000 7fca1c8b0640 1 --2- 172.21.15.33:0/2520262193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557be743f880 0x557be7439530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.517+0000 7fca177fe640 1 -- 172.21.15.33:0/2520262193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557be743f880 msgr2=0x557be7439530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:29.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.517+0000 7fca177fe640 1 --2- 172.21.15.33:0/2520262193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557be743f880 0x557be7439530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.517+0000 7fca177fe640 1 -- 172.21.15.33:0/2520262193 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557be7436380 msgr2=0x557be7447d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:29.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.517+0000 7fca177fe640 1 --2- 172.21.15.33:0/2520262193 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557be7436380 0x557be7447d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.517+0000 7fca177fe640 1 -- 172.21.15.33:0/2520262193 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557be7439d60 con 0x557be7436ca0 2024-03-19T02:55:29.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.517+0000 7fca17fff640 1 --2- 172.21.15.33:0/2520262193 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557be7436380 0x557be7447d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:29.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.517+0000 7fca177fe640 1 --2- 172.21.15.33:0/2520262193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557be7436ca0 0x557be7438ff0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x557be7435490 tx=0x7fca0c0f1550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df7393b0bbffb9dc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:29.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.517+0000 7fc9f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:55:29.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.518+0000 7fc9f77fe640 1 -- 172.21.15.33:0/2520262193 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0c0d8020 con 0x557be7436ca0 2024-03-19T02:55:29.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.518+0000 7fc9f77fe640 1 -- 172.21.15.33:0/2520262193 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fca0c0d9220 con 0x557be7436ca0 2024-03-19T02:55:29.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.518+0000 7fca20fcf740 1 -- 172.21.15.33:0/2520262193 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca0c0c8c40 con 0x557be7436ca0 2024-03-19T02:55:29.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.518+0000 7fc9f77fe640 1 -- 172.21.15.33:0/2520262193 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fca0c0a9070 con 0x557be7436ca0 2024-03-19T02:55:29.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.518+0000 7fca20fcf740 10 client.16134 fetch_fsmap learned FSMap version 25 2024-03-19T02:55:29.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.518+0000 7fca20fcf740 1 -- 172.21.15.33:0/2520262193 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x557be74b5d20 con 0x557be7436ca0 2024-03-19T02:55:29.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.518+0000 7fc9f77fe640 1 -- 172.21.15.33:0/2520262193 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fca0c0c8c40 con 0x557be7436ca0 2024-03-19T02:55:29.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.518+0000 7fc9f77fe640 1 -- 172.21.15.33:0/2520262193 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fca0c0052b0 con 0x557be7436ca0 2024-03-19T02:55:29.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.518+0000 7fca20fcf740 10 client.16134 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:55:29.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.518+0000 7fca20fcf740 10 client.16134 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:55:29.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.518+0000 7fca20fcf740 20 client.16134 populate_metadata read hostname 'smithi033' 2024-03-19T02:55:29.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.518+0000 7fca20fcf740 1 --2- 172.21.15.33:0/2520262193 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557be74b64f0 0x557be74b8910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.518+0000 7fca20fcf740 4 client.16134 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:55:29.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.518+0000 7fca20fcf740 1 -- 172.21.15.33:0/2520262193 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557be73a19f0 con 0x557be74b64f0 2024-03-19T02:55:29.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.518+0000 7fca17fff640 1 --2- 172.21.15.33:0/2520262193 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557be74b64f0 0x557be74b8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.519+0000 7fc9f77fe640 10 client.16134 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:55:29.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.519+0000 7fca17fff640 1 --2- 172.21.15.33:0/2520262193 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557be74b64f0 0x557be74b8910 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4bfbc63e4d3a5c75 server_cookie=56b27f32e6b13d45 in_seq=0 out_seq=0 2024-03-19T02:55:29.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.520+0000 7fc9f77fe640 1 -- 172.21.15.33:0/2520262193 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557be73a19f0 con 0x557be74b64f0 2024-03-19T02:55:29.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.520+0000 7fc9f77fe640 10 client.16134 handle_command_reply: tid=0 2024-03-19T02:55:29.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.520+0000 7fca20fcf740 1 client.16134 shutdown 2024-03-19T02:55:29.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.520+0000 7fca20fcf740 1 -- 172.21.15.33:0/2520262193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557be7436ca0 msgr2=0x557be7438ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:29.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.520+0000 7fca20fcf740 1 --2- 172.21.15.33:0/2520262193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557be7436ca0 0x557be7438ff0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x557be7435490 tx=0x7fca0c0f1550 comp rx=0 tx=0).stop 2024-03-19T02:55:29.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.520+0000 7fca20fcf740 1 -- 172.21.15.33:0/2520262193 shutdown_connections 2024-03-19T02:55:29.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.520+0000 7fca20fcf740 1 --2- 172.21.15.33:0/2520262193 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557be74b64f0 0x557be74b8910 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.520+0000 7fca20fcf740 1 --2- 172.21.15.33:0/2520262193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557be743f880 0x557be7439530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.520+0000 7fca20fcf740 1 --2- 172.21.15.33:0/2520262193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557be7436ca0 0x557be7438ff0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.520+0000 7fca20fcf740 1 --2- 172.21.15.33:0/2520262193 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557be7436380 0x557be7447d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.520+0000 7fca20fcf740 1 -- 172.21.15.33:0/2520262193 >> 172.21.15.33:0/2520262193 conn(0x557be7430ab0 msgr2=0x557be742fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:29.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.520+0000 7fc9f77fe640 0 client.16134 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:55:29.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.520+0000 7fca20fcf740 1 -- 172.21.15.33:0/2520262193 shutdown_connections 2024-03-19T02:55:29.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.520+0000 7fca20fcf740 1 -- 172.21.15.33:0/2520262193 wait complete. 2024-03-19T02:55:29.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.520+0000 7fca20fcf740 20 client.16134 trim_cache size 0 max 16384 2024-03-19T02:55:29.529 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:55:29.529 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-03-19T02:55:29.529 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "0a562807-33d8-467d-9eac-bbfb945a786b", 2024-03-19T02:55:29.529 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-03-19T02:55:29.529 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:55:29.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.521+0000 7fca1e313640 1 -- 172.21.15.33:0/242649822 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fca000a2970 msgr2=0x7fca000a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:29.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.521+0000 7fca1e313640 1 --2- 172.21.15.33:0/242649822 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fca000a2970 0x7fca000a4e30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fca10005fd0 tx=0x7fca10011b40 comp rx=0 tx=0).stop 2024-03-19T02:55:29.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.521+0000 7fca1e313640 1 -- 172.21.15.33:0/242649822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fca181033b0 msgr2=0x7fca18198580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:29.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.521+0000 7fca1e313640 1 --2- 172.21.15.33:0/242649822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fca181033b0 0x7fca18198580 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fca0c036040 tx=0x7fca0c0045e0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.521+0000 7fca1e313640 1 -- 172.21.15.33:0/242649822 shutdown_connections 2024-03-19T02:55:29.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.521+0000 7fca1e313640 1 --2- 172.21.15.33:0/242649822 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fca000a2970 0x7fca000a4e30 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.521+0000 7fca1e313640 1 --2- 172.21.15.33:0/242649822 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fca1810be60 0x7fca1819c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.521+0000 7fca1e313640 1 --2- 172.21.15.33:0/242649822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fca181033b0 0x7fca18198580 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.521+0000 7fca1e313640 1 --2- 172.21.15.33:0/242649822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fca18102a90 0x7fca18198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.521+0000 7fca1e313640 1 -- 172.21.15.33:0/242649822 >> 172.21.15.33:0/242649822 conn(0x7fca18077880 msgr2=0x7fca180fd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:29.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.521+0000 7fca1e313640 1 -- 172.21.15.33:0/242649822 shutdown_connections 2024-03-19T02:55:29.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.521+0000 7fca1e313640 1 -- 172.21.15.33:0/242649822 wait complete. 2024-03-19T02:55:29.533 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:55:29.533 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:55:29.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.637+0000 7f13d8261640 1 -- 172.21.15.33:0/1643321546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13d01006d0 msgr2=0x7f13d0100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:29.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.637+0000 7f13d8261640 1 --2- 172.21.15.33:0/1643321546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13d01006d0 0x7f13d0100b50 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f13c0009930 tx=0x7f13c002bc30 comp rx=0 tx=0).stop 2024-03-19T02:55:29.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.638+0000 7f13d8261640 1 -- 172.21.15.33:0/1643321546 shutdown_connections 2024-03-19T02:55:29.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.638+0000 7f13d8261640 1 --2- 172.21.15.33:0/1643321546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13d0101090 0x7f13d010d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.638+0000 7f13d8261640 1 --2- 172.21.15.33:0/1643321546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13d01006d0 0x7f13d0100b50 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.638+0000 7f13d8261640 1 --2- 172.21.15.33:0/1643321546 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f13d01066e0 0x7f13d0106ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.638+0000 7f13d8261640 1 -- 172.21.15.33:0/1643321546 >> 172.21.15.33:0/1643321546 conn(0x7f13d0076aa0 msgr2=0x7f13d0076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:29.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.638+0000 7f13d8261640 1 -- 172.21.15.33:0/1643321546 shutdown_connections 2024-03-19T02:55:29.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.638+0000 7f13d8261640 1 -- 172.21.15.33:0/1643321546 wait complete. 2024-03-19T02:55:29.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.638+0000 7f13d8261640 1 Processor -- start 2024-03-19T02:55:29.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.638+0000 7f13d8261640 1 -- start start 2024-03-19T02:55:29.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.638+0000 7f13d8261640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f13d01006d0 0x7f13d00fdea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.638+0000 7f13d8261640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13d0101090 0x7f13d00fe3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.638+0000 7f13d8261640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13d01066e0 0x7f13d00fe920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.638+0000 7f13d8261640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f13d01109a0 con 0x7f13d0101090 2024-03-19T02:55:29.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.638+0000 7f13d8261640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f13d0110820 con 0x7f13d01066e0 2024-03-19T02:55:29.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.638+0000 7f13d8261640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f13d0110b20 con 0x7f13d01006d0 2024-03-19T02:55:29.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.638+0000 7f13d57fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13d0101090 0x7f13d00fe3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.638+0000 7f13d57fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13d0101090 0x7f13d00fe3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55552/0 (socket says 172.21.15.33:55552) 2024-03-19T02:55:29.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.638+0000 7f13d57fc640 1 -- 172.21.15.33:0/3654454931 learned_addr learned my addr 172.21.15.33:0/3654454931 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:29.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.639+0000 7f13d67fe640 1 --2- 172.21.15.33:0/3654454931 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13d01066e0 0x7f13d00fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.639+0000 7f13d5ffd640 1 --2- 172.21.15.33:0/3654454931 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f13d01006d0 0x7f13d00fdea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.639+0000 7f13d57fc640 1 -- 172.21.15.33:0/3654454931 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f13d01006d0 msgr2=0x7f13d00fdea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:29.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.639+0000 7f13d57fc640 1 --2- 172.21.15.33:0/3654454931 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f13d01006d0 0x7f13d00fdea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.639+0000 7f13d57fc640 1 -- 172.21.15.33:0/3654454931 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13d01066e0 msgr2=0x7f13d00fe920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:29.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.639+0000 7f13d57fc640 1 --2- 172.21.15.33:0/3654454931 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13d01066e0 0x7f13d00fe920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.639+0000 7f13d57fc640 1 -- 172.21.15.33:0/3654454931 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d01a6550 con 0x7f13d0101090 2024-03-19T02:55:29.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.639+0000 7f13d67fe640 1 --2- 172.21.15.33:0/3654454931 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13d01066e0 0x7f13d00fe920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:29.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.639+0000 7f13d57fc640 1 --2- 172.21.15.33:0/3654454931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13d0101090 0x7f13d00fe3e0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f13c0009230 tx=0x7f13c002c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:29.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.639+0000 7f13c6ffd640 1 -- 172.21.15.33:0/3654454931 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c0043020 con 0x7f13d0101090 2024-03-19T02:55:29.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.639+0000 7f13c6ffd640 1 -- 172.21.15.33:0/3654454931 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f13c002ca70 con 0x7f13d0101090 2024-03-19T02:55:29.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.639+0000 7f13d8261640 1 -- 172.21.15.33:0/3654454931 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13d01a67e0 con 0x7f13d0101090 2024-03-19T02:55:29.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.639+0000 7f13c6ffd640 1 -- 172.21.15.33:0/3654454931 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c0003a50 con 0x7f13d0101090 2024-03-19T02:55:29.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.639+0000 7f13d8261640 1 -- 172.21.15.33:0/3654454931 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13d01a6d30 con 0x7f13d0101090 2024-03-19T02:55:29.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.641+0000 7f13c6ffd640 1 -- 172.21.15.33:0/3654454931 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f13c0046020 con 0x7f13d0101090 2024-03-19T02:55:29.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.642+0000 7f13dae52740 1 Processor -- start 2024-03-19T02:55:29.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.642+0000 7f13dae52740 1 -- start start 2024-03-19T02:55:29.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.642+0000 7f13c6ffd640 1 --2- 172.21.15.33:0/3654454931 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f13bc0a2970 0x7f13bc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.642+0000 7f13dae52740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dcf3f2da30 0x55dcf3f2de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.642+0000 7f13dae52740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcf3f2e350 0x55dcf3f369f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.642+0000 7f13dae52740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dcf3f36f30 0x55dcf3f39320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.642+0000 7f13dae52740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55dcf3e91500 con 0x55dcf3f2e350 2024-03-19T02:55:29.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.642+0000 7f13dae52740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55dcf3e39be0 con 0x55dcf3f36f30 2024-03-19T02:55:29.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.642+0000 7f13dae52740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55dcf3e8d5a0 con 0x55dcf3f2da30 2024-03-19T02:55:29.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.642+0000 7f13c6ffd640 1 -- 172.21.15.33:0/3654454931 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f13c00f10a0 con 0x7f13d0101090 2024-03-19T02:55:29.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.642+0000 7f13d67fe640 1 --2- 172.21.15.33:0/3654454931 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f13bc0a2970 0x7f13bc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.642+0000 7f13d57fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcf3f2e350 0x55dcf3f369f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.642+0000 7f13d57fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcf3f2e350 0x55dcf3f369f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55564/0 (socket says 172.21.15.33:55564) 2024-03-19T02:55:29.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.642+0000 7f13d57fc640 1 -- 172.21.15.33:0/3566346595 learned_addr learned my addr 172.21.15.33:0/3566346595 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:29.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.642+0000 7f13d5ffd640 1 --2- 172.21.15.33:0/3566346595 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dcf3f2da30 0x55dcf3f2de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.642+0000 7f13d67fe640 1 --2- 172.21.15.33:0/3566346595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dcf3f36f30 0x55dcf3f39320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.643+0000 7f13d57fc640 1 -- 172.21.15.33:0/3566346595 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dcf3f2da30 msgr2=0x55dcf3f2de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:29.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.643+0000 7f13d57fc640 1 --2- 172.21.15.33:0/3566346595 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dcf3f2da30 0x55dcf3f2de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.643+0000 7f13d57fc640 1 -- 172.21.15.33:0/3566346595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dcf3f36f30 msgr2=0x55dcf3f39320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:29.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.643+0000 7f13d57fc640 1 --2- 172.21.15.33:0/3566346595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dcf3f36f30 0x55dcf3f39320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.643+0000 7f13d57fc640 1 -- 172.21.15.33:0/3566346595 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcf3e5b000 con 0x55dcf3f2e350 2024-03-19T02:55:29.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.643+0000 7f13d67fe640 1 --2- 172.21.15.33:0/3654454931 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f13bc0a2970 0x7f13bc0a4e30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f13c80099e0 tx=0x7f13c8013660 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:29.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.643+0000 7f13d57fc640 1 --2- 172.21.15.33:0/3566346595 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcf3f2e350 0x55dcf3f369f0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f13c00b9e70 tx=0x7f13c0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f08e94697f2cabe2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:29.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.643+0000 7f13bbfff640 1 -- 172.21.15.33:0/3566346595 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c00bd4a0 con 0x55dcf3f2e350 2024-03-19T02:55:29.651 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.643+0000 7f13bbfff640 1 -- 172.21.15.33:0/3566346595 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f13c00c15f0 con 0x55dcf3f2e350 2024-03-19T02:55:29.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.643+0000 7f13bbfff640 1 -- 172.21.15.33:0/3566346595 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c0081f20 con 0x55dcf3f2e350 2024-03-19T02:55:29.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.644+0000 7f13dae52740 1 -- 172.21.15.33:0/3566346595 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcf3f2e350 msgr2=0x55dcf3f369f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:29.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.644+0000 7f13dae52740 1 --2- 172.21.15.33:0/3566346595 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcf3f2e350 0x55dcf3f369f0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f13c00b9e70 tx=0x7f13c0009260 comp rx=0 tx=0).stop 2024-03-19T02:55:29.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.644+0000 7f13dae52740 1 -- 172.21.15.33:0/3566346595 shutdown_connections 2024-03-19T02:55:29.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.644+0000 7f13dae52740 1 --2- 172.21.15.33:0/3566346595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dcf3f36f30 0x55dcf3f39320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.644+0000 7f13dae52740 1 --2- 172.21.15.33:0/3566346595 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcf3f2e350 0x55dcf3f369f0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.644+0000 7f13dae52740 1 --2- 172.21.15.33:0/3566346595 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dcf3f2da30 0x55dcf3f2de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.644+0000 7f13dae52740 1 -- 172.21.15.33:0/3566346595 >> 172.21.15.33:0/3566346595 conn(0x55dcf3f28160 msgr2=0x55dcf3f28540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:29.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.644+0000 7f13dae52740 1 -- 172.21.15.33:0/3566346595 shutdown_connections 2024-03-19T02:55:29.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.644+0000 7f13dae52740 1 -- 172.21.15.33:0/3566346595 wait complete. 2024-03-19T02:55:29.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.644+0000 7f13dae52740 1 -- start start 2024-03-19T02:55:29.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.644+0000 7f13dae52740 1 Processor -- start 2024-03-19T02:55:29.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.645+0000 7f13dae52740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcf3f2da30 0x55dcf3f33ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.652 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.645+0000 7f13dae52740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dcf3f2e350 0x55dcf3f34410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.645+0000 7f13dae52740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dcf3f36f30 0x55dcf3f34950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.645+0000 7f13dae52740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55dcf3f39ae0 con 0x55dcf3f2da30 2024-03-19T02:55:29.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.645+0000 7f13dae52740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55dcf3e4d160 con 0x55dcf3f2e350 2024-03-19T02:55:29.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.645+0000 7f13dae52740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55dcf3f39c60 con 0x55dcf3f36f30 2024-03-19T02:55:29.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.645+0000 7f13dae52740 10 client.0 _ll_register_callbacks cb 0x55dcf3e89a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:55:29.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.645+0000 7f13d5ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcf3f2da30 0x55dcf3f33ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.645+0000 7f13d5ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcf3f2da30 0x55dcf3f33ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55578/0 (socket says 172.21.15.33:55578) 2024-03-19T02:55:29.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.645+0000 7f13d5ffd640 1 -- 172.21.15.33:0/3666439019 learned_addr learned my addr 172.21.15.33:0/3666439019 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:29.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.645+0000 7f13d57fc640 1 --2- 172.21.15.33:0/3666439019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dcf3f2e350 0x55dcf3f34410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.645+0000 7f13d67fe640 1 --2- 172.21.15.33:0/3666439019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dcf3f36f30 0x55dcf3f34950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.645+0000 7f13d5ffd640 1 -- 172.21.15.33:0/3666439019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dcf3f36f30 msgr2=0x55dcf3f34950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:29.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.645+0000 7f13d5ffd640 1 --2- 172.21.15.33:0/3666439019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dcf3f36f30 0x55dcf3f34950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.645+0000 7f13d5ffd640 1 -- 172.21.15.33:0/3666439019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dcf3f2e350 msgr2=0x55dcf3f34410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:29.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.645+0000 7f13d5ffd640 1 --2- 172.21.15.33:0/3666439019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dcf3f2e350 0x55dcf3f34410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.645+0000 7f13d5ffd640 1 -- 172.21.15.33:0/3666439019 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcf3c6f7d0 con 0x55dcf3f2da30 2024-03-19T02:55:29.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.645+0000 7f13d5ffd640 1 --2- 172.21.15.33:0/3666439019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcf3f2da30 0x55dcf3f33ed0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f13cc012740 tx=0x7f13cc012c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9feb5cc92a865e54 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:29.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.645+0000 7f13b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:55:29.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.645+0000 7f13b97fa640 1 -- 172.21.15.33:0/3666439019 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13cc004030 con 0x55dcf3f2da30 2024-03-19T02:55:29.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.645+0000 7f13b97fa640 1 -- 172.21.15.33:0/3666439019 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f13cc0041d0 con 0x55dcf3f2da30 2024-03-19T02:55:29.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.645+0000 7f13dae52740 1 -- 172.21.15.33:0/3666439019 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13c00c15f0 con 0x55dcf3f2da30 2024-03-19T02:55:29.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.646+0000 7f13b97fa640 1 -- 172.21.15.33:0/3666439019 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f13cc004410 con 0x55dcf3f2da30 2024-03-19T02:55:29.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.646+0000 7f13dae52740 10 client.16146 fetch_fsmap learned FSMap version 25 2024-03-19T02:55:29.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.646+0000 7f13dae52740 1 -- 172.21.15.33:0/3666439019 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55dcf3ba53f0 con 0x55dcf3f2da30 2024-03-19T02:55:29.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.646+0000 7f13b97fa640 1 -- 172.21.15.33:0/3666439019 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f13c00c15f0 con 0x55dcf3f2da30 2024-03-19T02:55:29.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.646+0000 7f13b97fa640 1 -- 172.21.15.33:0/3666439019 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f13cc003ab0 con 0x55dcf3f2da30 2024-03-19T02:55:29.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.646+0000 7f13dae52740 10 client.16146 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:55:29.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.646+0000 7f13dae52740 10 client.16146 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:55:29.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.646+0000 7f13dae52740 20 client.16146 populate_metadata read hostname 'smithi033' 2024-03-19T02:55:29.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.646+0000 7f13dae52740 1 --2- 172.21.15.33:0/3666439019 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55dcf3f46fa0 0x55dcf3f49410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.646+0000 7f13dae52740 4 client.16146 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:55:29.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.646+0000 7f13dae52740 1 -- 172.21.15.33:0/3666439019 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dcf3e9ea40 con 0x55dcf3f46fa0 2024-03-19T02:55:29.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.646+0000 7f13d57fc640 1 --2- 172.21.15.33:0/3666439019 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55dcf3f46fa0 0x55dcf3f49410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.647+0000 7f13b97fa640 10 client.16146 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:55:29.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.647+0000 7f13d57fc640 1 --2- 172.21.15.33:0/3666439019 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55dcf3f46fa0 0x55dcf3f49410 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5486ab5d58c6fb1 server_cookie=3b2ab798764cdb92 in_seq=0 out_seq=0 2024-03-19T02:55:29.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.647+0000 7f13b97fa640 1 -- 172.21.15.33:0/3666439019 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dcf3e9ea40 con 0x55dcf3f46fa0 2024-03-19T02:55:29.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.647+0000 7f13b97fa640 10 client.16146 handle_command_reply: tid=0 2024-03-19T02:55:29.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.647+0000 7f13dae52740 1 client.16146 shutdown 2024-03-19T02:55:29.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.647+0000 7f13dae52740 1 -- 172.21.15.33:0/3666439019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcf3f2da30 msgr2=0x55dcf3f33ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:29.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.647+0000 7f13dae52740 1 --2- 172.21.15.33:0/3666439019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcf3f2da30 0x55dcf3f33ed0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f13cc012740 tx=0x7f13cc012c00 comp rx=0 tx=0).stop 2024-03-19T02:55:29.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.647+0000 7f13dae52740 1 -- 172.21.15.33:0/3666439019 shutdown_connections 2024-03-19T02:55:29.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.647+0000 7f13dae52740 1 --2- 172.21.15.33:0/3666439019 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55dcf3f46fa0 0x55dcf3f49410 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.647+0000 7f13dae52740 1 --2- 172.21.15.33:0/3666439019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dcf3f36f30 0x55dcf3f34950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.647+0000 7f13dae52740 1 --2- 172.21.15.33:0/3666439019 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dcf3f2e350 0x55dcf3f34410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.647+0000 7f13dae52740 1 --2- 172.21.15.33:0/3666439019 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcf3f2da30 0x55dcf3f33ed0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.647+0000 7f13b97fa640 0 client.16146 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:55:29.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.647+0000 7f13dae52740 1 -- 172.21.15.33:0/3666439019 >> 172.21.15.33:0/3666439019 conn(0x55dcf3f28160 msgr2=0x55dcf3f29eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:29.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.647+0000 7f13dae52740 1 -- 172.21.15.33:0/3666439019 shutdown_connections 2024-03-19T02:55:29.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.647+0000 7f13dae52740 1 -- 172.21.15.33:0/3666439019 wait complete. 2024-03-19T02:55:29.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.648+0000 7f13dae52740 20 client.16146 trim_cache size 0 max 16384 2024-03-19T02:55:29.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.665+0000 7f13dae52740 1 Processor -- start 2024-03-19T02:55:29.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.665+0000 7f13dae52740 1 -- start start 2024-03-19T02:55:29.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.665+0000 7f13dae52740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcf3f28160 0x55dcf3f36600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.665+0000 7f13dae52740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dcf3f2da30 0x55dcf3f427c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.665+0000 7f13dae52740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dcf3f2e350 0x55dcf3f4ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.665+0000 7f13dae52740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f13d01b3600 con 0x55dcf3f28160 2024-03-19T02:55:29.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.665+0000 7f13dae52740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f13d01b0ed0 con 0x55dcf3f2da30 2024-03-19T02:55:29.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.665+0000 7f13dae52740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f13d01aed00 con 0x55dcf3f2e350 2024-03-19T02:55:29.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.665+0000 7f13d5ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcf3f28160 0x55dcf3f36600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.665+0000 7f13d5ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcf3f28160 0x55dcf3f36600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55580/0 (socket says 172.21.15.33:55580) 2024-03-19T02:55:29.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.665+0000 7f13d5ffd640 1 -- 172.21.15.33:0/4183895178 learned_addr learned my addr 172.21.15.33:0/4183895178 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:29.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.665+0000 7f13d57fc640 1 --2- 172.21.15.33:0/4183895178 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dcf3f2da30 0x55dcf3f427c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.665+0000 7f13d67fe640 1 --2- 172.21.15.33:0/4183895178 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dcf3f2e350 0x55dcf3f4ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.666+0000 7f13d5ffd640 1 -- 172.21.15.33:0/4183895178 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dcf3f2e350 msgr2=0x55dcf3f4ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:29.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.666+0000 7f13d5ffd640 1 --2- 172.21.15.33:0/4183895178 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dcf3f2e350 0x55dcf3f4ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.666+0000 7f13d5ffd640 1 -- 172.21.15.33:0/4183895178 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dcf3f2da30 msgr2=0x55dcf3f427c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:29.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.666+0000 7f13d5ffd640 1 --2- 172.21.15.33:0/4183895178 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dcf3f2da30 0x55dcf3f427c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.666+0000 7f13d5ffd640 1 -- 172.21.15.33:0/4183895178 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcf3b650f0 con 0x55dcf3f28160 2024-03-19T02:55:29.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.666+0000 7f13d5ffd640 1 --2- 172.21.15.33:0/4183895178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcf3f28160 0x55dcf3f36600 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f13cc003f50 tx=0x7f13cc009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18b91ed360497ea6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:29.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.666+0000 7f13bb7fe640 1 -- 172.21.15.33:0/4183895178 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13cc003fd0 con 0x55dcf3f28160 2024-03-19T02:55:29.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.666+0000 7f13bb7fe640 1 -- 172.21.15.33:0/4183895178 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f13cc012e50 con 0x55dcf3f28160 2024-03-19T02:55:29.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.666+0000 7f13bb7fe640 1 -- 172.21.15.33:0/4183895178 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13cc0056a0 con 0x55dcf3f28160 2024-03-19T02:55:29.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.667+0000 7f13dae52740 1 -- 172.21.15.33:0/4183895178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcf3f28160 msgr2=0x55dcf3f36600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:29.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.667+0000 7f13dae52740 1 --2- 172.21.15.33:0/4183895178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcf3f28160 0x55dcf3f36600 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f13cc003f50 tx=0x7f13cc009f80 comp rx=0 tx=0).stop 2024-03-19T02:55:29.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.667+0000 7f13dae52740 1 -- 172.21.15.33:0/4183895178 shutdown_connections 2024-03-19T02:55:29.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.667+0000 7f13dae52740 1 --2- 172.21.15.33:0/4183895178 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dcf3f2e350 0x55dcf3f4ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.667+0000 7f13dae52740 1 --2- 172.21.15.33:0/4183895178 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dcf3f2da30 0x55dcf3f427c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.667+0000 7f13dae52740 1 --2- 172.21.15.33:0/4183895178 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcf3f28160 0x55dcf3f36600 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.667+0000 7f13dae52740 1 -- 172.21.15.33:0/4183895178 >> 172.21.15.33:0/4183895178 conn(0x55dcf3f32b10 msgr2=0x55dcf3f34720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:29.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.667+0000 7f13dae52740 1 -- 172.21.15.33:0/4183895178 shutdown_connections 2024-03-19T02:55:29.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.667+0000 7f13dae52740 1 -- 172.21.15.33:0/4183895178 wait complete. 2024-03-19T02:55:29.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.667+0000 7f13dae52740 1 -- start start 2024-03-19T02:55:29.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.667+0000 7f13dae52740 1 Processor -- start 2024-03-19T02:55:29.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.667+0000 7f13dae52740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dcf3f28160 0x55dcf3f3ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.667+0000 7f13dae52740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dcf3f2da30 0x55dcf3f3b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.667+0000 7f13dae52740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcf3f2e350 0x55dcf3f3b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.667+0000 7f13dae52740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f13d01af390 con 0x55dcf3f2e350 2024-03-19T02:55:29.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.667+0000 7f13dae52740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f13d01b0800 con 0x55dcf3f28160 2024-03-19T02:55:29.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.667+0000 7f13dae52740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f13d01afa60 con 0x55dcf3f2da30 2024-03-19T02:55:29.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.667+0000 7f13dae52740 10 client.0 _ll_register_callbacks cb 0x7f13cc009830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:55:29.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.668+0000 7f13d67fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcf3f2e350 0x55dcf3f3b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.668+0000 7f13d5ffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dcf3f28160 0x55dcf3f3ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.668+0000 7f13d57fc640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dcf3f2da30 0x55dcf3f3b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.668+0000 7f13d67fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcf3f2e350 0x55dcf3f3b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55588/0 (socket says 172.21.15.33:55588) 2024-03-19T02:55:29.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.668+0000 7f13d5ffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dcf3f28160 0x55dcf3f3ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:45240/0 (socket says 172.21.15.33:45240) 2024-03-19T02:55:29.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.668+0000 7f13d57fc640 1 --2- 172.21.15.33:0/1961191035 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dcf3f2da30 0x55dcf3f3b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:60466/0 (socket says 172.21.15.33:60466) 2024-03-19T02:55:29.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.668+0000 7f13d67fe640 1 -- 172.21.15.33:0/1961191035 learned_addr learned my addr 172.21.15.33:0/1961191035 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:29.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.668+0000 7f13d67fe640 1 -- 172.21.15.33:0/1961191035 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dcf3f2da30 msgr2=0x55dcf3f3b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:29.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.668+0000 7f13d67fe640 1 --2- 172.21.15.33:0/1961191035 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dcf3f2da30 0x55dcf3f3b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.668+0000 7f13d67fe640 1 -- 172.21.15.33:0/1961191035 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dcf3f28160 msgr2=0x55dcf3f3ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:29.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.668+0000 7f13d67fe640 1 --2- 172.21.15.33:0/1961191035 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dcf3f28160 0x55dcf3f3ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.668+0000 7f13d67fe640 1 -- 172.21.15.33:0/1961191035 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcf3f3bce0 con 0x55dcf3f2e350 2024-03-19T02:55:29.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.668+0000 7f13d5ffd640 1 --2- 172.21.15.33:0/1961191035 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dcf3f28160 0x55dcf3f3ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:29.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.668+0000 7f13d67fe640 1 --2- 172.21.15.33:0/1961191035 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcf3f2e350 0x55dcf3f3b720 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f13c8003d20 tx=0x7f13c801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a48157743d265976 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:29.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.668+0000 7f13b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:55:29.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.668+0000 7f13b97fa640 1 -- 172.21.15.33:0/1961191035 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c8004b20 con 0x55dcf3f2e350 2024-03-19T02:55:29.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.668+0000 7f13dae52740 1 -- 172.21.15.33:0/1961191035 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13cc012e50 con 0x55dcf3f2e350 2024-03-19T02:55:29.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.668+0000 7f13b97fa640 1 -- 172.21.15.33:0/1961191035 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f13c8004cc0 con 0x55dcf3f2e350 2024-03-19T02:55:29.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.668+0000 7f13b97fa640 1 -- 172.21.15.33:0/1961191035 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f13c8035050 con 0x55dcf3f2e350 2024-03-19T02:55:29.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.668+0000 7f13dae52740 10 client.16152 fetch_fsmap learned FSMap version 25 2024-03-19T02:55:29.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.668+0000 7f13dae52740 1 -- 172.21.15.33:0/1961191035 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55dcf3fae150 con 0x55dcf3f2e350 2024-03-19T02:55:29.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.669+0000 7f13b97fa640 1 -- 172.21.15.33:0/1961191035 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f13cc012e50 con 0x55dcf3f2e350 2024-03-19T02:55:29.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.669+0000 7f13b97fa640 1 -- 172.21.15.33:0/1961191035 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f13c8020050 con 0x55dcf3f2e350 2024-03-19T02:55:29.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.669+0000 7f13dae52740 10 client.16152 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:55:29.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.669+0000 7f13dae52740 10 client.16152 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:55:29.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.669+0000 7f13dae52740 20 client.16152 populate_metadata read hostname 'smithi033' 2024-03-19T02:55:29.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.669+0000 7f13dae52740 1 --2- 172.21.15.33:0/1961191035 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55dcf3f36f30 0x55dcf3fb0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:29.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.669+0000 7f13dae52740 4 client.16152 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:55:29.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.669+0000 7f13dae52740 1 -- 172.21.15.33:0/1961191035 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55dcf3e98d90 con 0x55dcf3f36f30 2024-03-19T02:55:29.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.669+0000 7f13d5ffd640 1 --2- 172.21.15.33:0/1961191035 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55dcf3f36f30 0x55dcf3fb0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:29.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.669+0000 7f13b97fa640 10 client.16152 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:55:29.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.669+0000 7f13d5ffd640 1 --2- 172.21.15.33:0/1961191035 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55dcf3f36f30 0x55dcf3fb0880 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a62ea828698c0cdc server_cookie=ded72323812ec7d4 in_seq=0 out_seq=0 2024-03-19T02:55:29.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.670+0000 7f13b97fa640 1 -- 172.21.15.33:0/1961191035 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55dcf3e98d90 con 0x55dcf3f36f30 2024-03-19T02:55:29.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.670+0000 7f13b97fa640 10 client.16152 handle_command_reply: tid=0 2024-03-19T02:55:29.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.670+0000 7f13dae52740 1 client.16152 shutdown 2024-03-19T02:55:29.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.670+0000 7f13dae52740 1 -- 172.21.15.33:0/1961191035 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcf3f2e350 msgr2=0x55dcf3f3b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:29.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.670+0000 7f13dae52740 1 --2- 172.21.15.33:0/1961191035 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcf3f2e350 0x55dcf3f3b720 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f13c8003d20 tx=0x7f13c801d000 comp rx=0 tx=0).stop 2024-03-19T02:55:29.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.670+0000 7f13dae52740 1 -- 172.21.15.33:0/1961191035 shutdown_connections 2024-03-19T02:55:29.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.670+0000 7f13dae52740 1 --2- 172.21.15.33:0/1961191035 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55dcf3f36f30 0x55dcf3fb0880 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.670+0000 7f13dae52740 1 --2- 172.21.15.33:0/1961191035 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dcf3f2da30 0x55dcf3f3b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.670+0000 7f13b97fa640 0 client.16152 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:55:29.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.670+0000 7f13dae52740 1 --2- 172.21.15.33:0/1961191035 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dcf3f2e350 0x55dcf3f3b720 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.670+0000 7f13dae52740 1 --2- 172.21.15.33:0/1961191035 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dcf3f28160 0x55dcf3f3ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.670+0000 7f13dae52740 1 -- 172.21.15.33:0/1961191035 >> 172.21.15.33:0/1961191035 conn(0x55dcf3f32b10 msgr2=0x55dcf3f34700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:29.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.670+0000 7f13dae52740 1 -- 172.21.15.33:0/1961191035 shutdown_connections 2024-03-19T02:55:29.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.670+0000 7f13dae52740 1 -- 172.21.15.33:0/1961191035 wait complete. 2024-03-19T02:55:29.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.670+0000 7f13dae52740 20 client.16152 trim_cache size 0 max 16384 2024-03-19T02:55:29.678 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:55:29.678 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:55:29.678 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-03-19T02:55:29.679 INFO:teuthology.orchestra.run.smithi033.stdout: "0a562807-33d8-467d-9eac-bbfb945a786b": { 2024-03-19T02:55:29.679 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-03-19T02:55:29.679 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "0a562807-33d8-467d-9eac-bbfb945a786b", 2024-03-19T02:55:29.679 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-03-19T02:55:29.679 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T02:55:29.679 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T02:55:29.679 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:55:29.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.670+0000 7f13d8261640 1 -- 172.21.15.33:0/3654454931 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f13bc0a2970 msgr2=0x7f13bc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:29.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.670+0000 7f13d8261640 1 --2- 172.21.15.33:0/3654454931 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f13bc0a2970 0x7f13bc0a4e30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f13c80099e0 tx=0x7f13c8013660 comp rx=0 tx=0).stop 2024-03-19T02:55:29.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.670+0000 7f13d8261640 1 -- 172.21.15.33:0/3654454931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13d0101090 msgr2=0x7f13d00fe3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:29.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.670+0000 7f13d8261640 1 --2- 172.21.15.33:0/3654454931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13d0101090 0x7f13d00fe3e0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f13c0009230 tx=0x7f13c002c5d0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.670+0000 7f13d8261640 1 -- 172.21.15.33:0/3654454931 shutdown_connections 2024-03-19T02:55:29.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.670+0000 7f13d8261640 1 --2- 172.21.15.33:0/3654454931 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f13bc0a2970 0x7f13bc0a4e30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.670+0000 7f13d8261640 1 --2- 172.21.15.33:0/3654454931 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13d01066e0 0x7f13d00fe920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.670+0000 7f13d8261640 1 --2- 172.21.15.33:0/3654454931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f13d0101090 0x7f13d00fe3e0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.670+0000 7f13d8261640 1 --2- 172.21.15.33:0/3654454931 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f13d01006d0 0x7f13d00fdea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:29.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.670+0000 7f13d8261640 1 -- 172.21.15.33:0/3654454931 >> 172.21.15.33:0/3654454931 conn(0x7f13d0076aa0 msgr2=0x7f13d006a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:29.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.671+0000 7f13d8261640 1 -- 172.21.15.33:0/3654454931 shutdown_connections 2024-03-19T02:55:29.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:29.671+0000 7f13d8261640 1 -- 172.21.15.33:0/3654454931 wait complete. 2024-03-19T02:55:29.682 INFO:tasks.cephfs.filesystem:scrub status for tag:0a562807-33d8-467d-9eac-bbfb945a786b - {'path': '/', 'tag': '0a562807-33d8-467d-9eac-bbfb945a786b', 'options': 'recursive,force'} 2024-03-19T02:55:30.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:30 smithi033 ceph-mon[31973]: pgmap v64: 129 pgs: 129 active+clean; 11 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 765 MiB/s wr, 222 op/s 2024-03-19T02:55:30.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:30 smithi033 ceph-mon[31973]: scrub queued for path: / 2024-03-19T02:55:30.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:30 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T02:55:30.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:30 smithi033 ceph-mon[31973]: scrub summary: active paths [/] 2024-03-19T02:55:30.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:30 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T02:55:30.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:30 smithi096 ceph-mon[35946]: pgmap v64: 129 pgs: 129 active+clean; 11 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 765 MiB/s wr, 222 op/s 2024-03-19T02:55:30.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:30 smithi096 ceph-mon[35946]: scrub queued for path: / 2024-03-19T02:55:30.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:30 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T02:55:30.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:30 smithi096 ceph-mon[35946]: scrub summary: active paths [/] 2024-03-19T02:55:30.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:30 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T02:55:30.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:30 smithi155 ceph-mon[35804]: pgmap v64: 129 pgs: 129 active+clean; 11 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 765 MiB/s wr, 222 op/s 2024-03-19T02:55:30.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:30 smithi155 ceph-mon[35804]: scrub queued for path: / 2024-03-19T02:55:30.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:30 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T02:55:30.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:30 smithi155 ceph-mon[35804]: scrub summary: active paths [/] 2024-03-19T02:55:30.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:30 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T02:55:32.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:32 smithi033 ceph-mon[31973]: pgmap v65: 129 pgs: 129 active+clean; 11 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 511 MiB/s wr, 133 op/s 2024-03-19T02:55:32.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:32 smithi096 ceph-mon[35946]: pgmap v65: 129 pgs: 129 active+clean; 11 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 511 MiB/s wr, 133 op/s 2024-03-19T02:55:32.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:32 smithi155 ceph-mon[35804]: pgmap v65: 129 pgs: 129 active+clean; 11 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 511 MiB/s wr, 133 op/s 2024-03-19T02:55:33.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:33 smithi033 ceph-mon[31973]: scrub summary: idle 2024-03-19T02:55:33.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:33 smithi096 ceph-mon[35946]: scrub summary: idle 2024-03-19T02:55:33.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:33 smithi155 ceph-mon[35804]: scrub summary: idle 2024-03-19T02:55:34.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:34 smithi033 ceph-mon[31973]: pgmap v66: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 540 MiB/s wr, 143 op/s 2024-03-19T02:55:34.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:34 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:55:34.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:34 smithi096 ceph-mon[35946]: pgmap v66: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 540 MiB/s wr, 143 op/s 2024-03-19T02:55:34.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:34 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:55:34.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:34 smithi155 ceph-mon[35804]: pgmap v66: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 540 MiB/s wr, 143 op/s 2024-03-19T02:55:34.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:34 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:55:35.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:35 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:55:35.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:35 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:55:35.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:35 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:55:35.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:35 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:55:35.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:35 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:55:35.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:35 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:55:35.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:35 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:55:35.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:35 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:55:35.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:35 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:55:37.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:36 smithi096 ceph-mon[35946]: pgmap v67: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 282 MiB/s wr, 76 op/s 2024-03-19T02:55:37.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:36 smithi155 ceph-mon[35804]: pgmap v67: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 282 MiB/s wr, 76 op/s 2024-03-19T02:55:37.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:36 smithi033 ceph-mon[31973]: pgmap v67: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 282 MiB/s wr, 76 op/s 2024-03-19T02:55:39.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:38 smithi096 ceph-mon[35946]: pgmap v68: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 282 MiB/s wr, 76 op/s 2024-03-19T02:55:39.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:38 smithi155 ceph-mon[35804]: pgmap v68: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 282 MiB/s wr, 76 op/s 2024-03-19T02:55:39.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:38 smithi033 ceph-mon[31973]: pgmap v68: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 282 MiB/s wr, 76 op/s 2024-03-19T02:55:41.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:40 smithi096 ceph-mon[35946]: pgmap v69: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 454 MiB/s wr, 122 op/s 2024-03-19T02:55:41.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:40 smithi155 ceph-mon[35804]: pgmap v69: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 454 MiB/s wr, 122 op/s 2024-03-19T02:55:41.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:40 smithi033 ceph-mon[31973]: pgmap v69: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 454 MiB/s wr, 122 op/s 2024-03-19T02:55:43.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:42 smithi096 ceph-mon[35946]: pgmap v70: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 201 MiB/s wr, 56 op/s 2024-03-19T02:55:43.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:42 smithi155 ceph-mon[35804]: pgmap v70: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 201 MiB/s wr, 56 op/s 2024-03-19T02:55:43.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:42 smithi033 ceph-mon[31973]: pgmap v70: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 201 MiB/s wr, 56 op/s 2024-03-19T02:55:45.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:44 smithi096 ceph-mon[35946]: pgmap v71: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 484 MiB/s wr, 129 op/s 2024-03-19T02:55:45.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:44 smithi155 ceph-mon[35804]: pgmap v71: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 484 MiB/s wr, 129 op/s 2024-03-19T02:55:45.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:44 smithi033 ceph-mon[31973]: pgmap v71: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 484 MiB/s wr, 129 op/s 2024-03-19T02:55:47.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:46 smithi096 ceph-mon[35946]: pgmap v72: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 455 MiB/s wr, 119 op/s 2024-03-19T02:55:47.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:46 smithi155 ceph-mon[35804]: pgmap v72: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 455 MiB/s wr, 119 op/s 2024-03-19T02:55:47.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:46 smithi033 ceph-mon[31973]: pgmap v72: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 455 MiB/s wr, 119 op/s 2024-03-19T02:55:49.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:48 smithi096 ceph-mon[35946]: pgmap v73: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 455 MiB/s wr, 119 op/s 2024-03-19T02:55:49.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:48 smithi155 ceph-mon[35804]: pgmap v73: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 455 MiB/s wr, 119 op/s 2024-03-19T02:55:49.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:48 smithi033 ceph-mon[31973]: pgmap v73: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 455 MiB/s wr, 119 op/s 2024-03-19T02:55:50.043 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-03-19T02:55:50.044 INFO:tasks.workunit.client.0.smithi033.stdout: Children see throughput for 1 rewriters = 694479.75 kB/sec 2024-03-19T02:55:50.044 INFO:tasks.workunit.client.0.smithi033.stdout: Parent sees throughput for 1 rewriters = 694465.85 kB/sec 2024-03-19T02:55:50.044 INFO:tasks.workunit.client.0.smithi033.stdout: Min throughput per process = 694479.75 kB/sec 2024-03-19T02:55:50.044 INFO:tasks.workunit.client.0.smithi033.stdout: Max throughput per process = 694479.75 kB/sec 2024-03-19T02:55:50.044 INFO:tasks.workunit.client.0.smithi033.stdout: Avg throughput per process = 694479.75 kB/sec 2024-03-19T02:55:50.044 INFO:tasks.workunit.client.0.smithi033.stdout: Min xfer = 10485760.00 kB 2024-03-19T02:55:51.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:50 smithi096 ceph-mon[35946]: pgmap v74: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 736 MiB/s wr, 192 op/s 2024-03-19T02:55:51.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:50 smithi155 ceph-mon[35804]: pgmap v74: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 736 MiB/s wr, 192 op/s 2024-03-19T02:55:51.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:50 smithi033 ceph-mon[31973]: pgmap v74: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 736 MiB/s wr, 192 op/s 2024-03-19T02:55:53.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:52 smithi096 ceph-mon[35946]: pgmap v75: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 564 MiB/s wr, 146 op/s 2024-03-19T02:55:53.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:52 smithi155 ceph-mon[35804]: pgmap v75: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 564 MiB/s wr, 146 op/s 2024-03-19T02:55:53.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:52 smithi033 ceph-mon[31973]: pgmap v75: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 564 MiB/s wr, 146 op/s 2024-03-19T02:55:55.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:54 smithi096 ceph-mon[35946]: pgmap v76: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 681 MiB/s wr, 178 op/s 2024-03-19T02:55:55.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:54 smithi155 ceph-mon[35804]: pgmap v76: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 681 MiB/s wr, 178 op/s 2024-03-19T02:55:55.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:54 smithi033 ceph-mon[31973]: pgmap v76: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 681 MiB/s wr, 178 op/s 2024-03-19T02:55:57.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:56 smithi096 ceph-mon[35946]: pgmap v77: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 399 MiB/s wr, 104 op/s 2024-03-19T02:55:57.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:56 smithi155 ceph-mon[35804]: pgmap v77: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 399 MiB/s wr, 104 op/s 2024-03-19T02:55:57.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:56 smithi033 ceph-mon[31973]: pgmap v77: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 399 MiB/s wr, 104 op/s 2024-03-19T02:55:59.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:55:58 smithi096 ceph-mon[35946]: pgmap v78: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 399 MiB/s wr, 104 op/s 2024-03-19T02:55:59.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:55:58 smithi155 ceph-mon[35804]: pgmap v78: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 399 MiB/s wr, 104 op/s 2024-03-19T02:55:59.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:55:58 smithi033 ceph-mon[31973]: pgmap v78: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 399 MiB/s wr, 104 op/s 2024-03-19T02:55:59.683 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:55:59.683 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:55:59.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.798+0000 7f3dd5d44640 1 -- 172.21.15.33:0/2562117985 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3dd0101090 msgr2=0x7f3dd010d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:59.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.798+0000 7f3dd5d44640 1 --2- 172.21.15.33:0/2562117985 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3dd0101090 0x7f3dd010d5a0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8009cd0 tx=0x7f3dc802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.799+0000 7f3dd5d44640 1 -- 172.21.15.33:0/2562117985 shutdown_connections 2024-03-19T02:55:59.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.799+0000 7f3dd5d44640 1 --2- 172.21.15.33:0/2562117985 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3dd0101090 0x7f3dd010d5a0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.799+0000 7f3dd5d44640 1 --2- 172.21.15.33:0/2562117985 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dd01006d0 0x7f3dd0100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.799+0000 7f3dd5d44640 1 --2- 172.21.15.33:0/2562117985 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3dd01066e0 0x7f3dd0106ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.799+0000 7f3dd5d44640 1 -- 172.21.15.33:0/2562117985 >> 172.21.15.33:0/2562117985 conn(0x7f3dd0076aa0 msgr2=0x7f3dd0076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:59.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.799+0000 7f3dd5d44640 1 -- 172.21.15.33:0/2562117985 shutdown_connections 2024-03-19T02:55:59.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.799+0000 7f3dd5d44640 1 -- 172.21.15.33:0/2562117985 wait complete. 2024-03-19T02:55:59.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.799+0000 7f3dd5d44640 1 Processor -- start 2024-03-19T02:55:59.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.799+0000 7f3dd5d44640 1 -- start start 2024-03-19T02:55:59.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.800+0000 7f3dd5d44640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dd01006d0 0x7f3dd0198e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:59.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.800+0000 7f3dd5d44640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3dd0101090 0x7f3dd0199380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:59.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.800+0000 7f3dd5d44640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3dd01066e0 0x7f3dd01a3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:59.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.800+0000 7f3dd5d44640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd01109a0 con 0x7f3dd01066e0 2024-03-19T02:55:59.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.800+0000 7f3dd5d44640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd0110820 con 0x7f3dd01006d0 2024-03-19T02:55:59.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.800+0000 7f3dd5d44640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd0110b20 con 0x7f3dd0101090 2024-03-19T02:55:59.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.800+0000 7f3dcffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3dd01066e0 0x7f3dd01a3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:59.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.800+0000 7f3dcf7fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dd01006d0 0x7f3dd0198e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:59.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.800+0000 7f3dcffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3dd01066e0 0x7f3dd01a3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41976/0 (socket says 172.21.15.33:41976) 2024-03-19T02:55:59.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.800+0000 7f3dcffff640 1 -- 172.21.15.33:0/3867564493 learned_addr learned my addr 172.21.15.33:0/3867564493 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:59.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.800+0000 7f3dceffd640 1 --2- 172.21.15.33:0/3867564493 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3dd0101090 0x7f3dd0199380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:59.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.800+0000 7f3dcf7fe640 1 -- 172.21.15.33:0/3867564493 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3dd0101090 msgr2=0x7f3dd0199380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:59.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.800+0000 7f3dcf7fe640 1 --2- 172.21.15.33:0/3867564493 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3dd0101090 0x7f3dd0199380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.800+0000 7f3dcf7fe640 1 -- 172.21.15.33:0/3867564493 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3dd01066e0 msgr2=0x7f3dd01a3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:59.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.800+0000 7f3dcf7fe640 1 --2- 172.21.15.33:0/3867564493 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3dd01066e0 0x7f3dd01a3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.800+0000 7f3dcf7fe640 1 -- 172.21.15.33:0/3867564493 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dd01a4400 con 0x7f3dd01006d0 2024-03-19T02:55:59.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.800+0000 7f3dcffff640 1 --2- 172.21.15.33:0/3867564493 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3dd01066e0 0x7f3dd01a3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:59.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.800+0000 7f3dcf7fe640 1 --2- 172.21.15.33:0/3867564493 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dd01006d0 0x7f3dd0198e40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3dc000e9f0 tx=0x7f3dc000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:59.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.800+0000 7f3dccff9640 1 -- 172.21.15.33:0/3867564493 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc000ccc0 con 0x7f3dd01006d0 2024-03-19T02:55:59.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.800+0000 7f3dd5d44640 1 -- 172.21.15.33:0/3867564493 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dd01a45d0 con 0x7f3dd01006d0 2024-03-19T02:55:59.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.801+0000 7f3dccff9640 1 -- 172.21.15.33:0/3867564493 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3dc00108e0 con 0x7f3dd01006d0 2024-03-19T02:55:59.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.801+0000 7f3dccff9640 1 -- 172.21.15.33:0/3867564493 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc0018880 con 0x7f3dd01006d0 2024-03-19T02:55:59.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.801+0000 7f3dd5d44640 1 -- 172.21.15.33:0/3867564493 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dd01a4b40 con 0x7f3dd01006d0 2024-03-19T02:55:59.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.803+0000 7f3dccff9640 1 -- 172.21.15.33:0/3867564493 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f3dc0010a90 con 0x7f3dd01006d0 2024-03-19T02:55:59.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.803+0000 7f3dd8a05740 1 Processor -- start 2024-03-19T02:55:59.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.803+0000 7f3dd8a05740 1 -- start start 2024-03-19T02:55:59.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.803+0000 7f3dd8a05740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f5bea5a30 0x562f5bea5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:59.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.803+0000 7f3dd8a05740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f5bea6350 0x562f5beae9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:59.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.803+0000 7f3dd8a05740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f5beaef30 0x562f5beb1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:59.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.803+0000 7f3dd8a05740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562f5be09500 con 0x562f5bea6350 2024-03-19T02:55:59.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.803+0000 7f3dd8a05740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x562f5bdb1be0 con 0x562f5bea5a30 2024-03-19T02:55:59.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.803+0000 7f3dd8a05740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x562f5be055a0 con 0x562f5beaef30 2024-03-19T02:55:59.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.803+0000 7f3dccff9640 1 --2- 172.21.15.33:0/3867564493 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f3db80a2970 0x7f3db80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:59.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.803+0000 7f3dcf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f5bea6350 0x562f5beae9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:59.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.803+0000 7f3dcf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f5bea6350 0x562f5beae9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41986/0 (socket says 172.21.15.33:41986) 2024-03-19T02:55:59.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.803+0000 7f3dcf7fe640 1 -- 172.21.15.33:0/971798930 learned_addr learned my addr 172.21.15.33:0/971798930 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:59.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.803+0000 7f3dceffd640 1 --2- 172.21.15.33:0/971798930 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f5beaef30 0x562f5beb1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:59.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.803+0000 7f3dcffff640 1 --2- 172.21.15.33:0/971798930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f5bea5a30 0x562f5bea5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:59.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.804+0000 7f3dcffff640 1 --2- 172.21.15.33:0/3867564493 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f3db80a2970 0x7f3db80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:59.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.804+0000 7f3dccff9640 1 -- 172.21.15.33:0/3867564493 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f3dc00cf480 con 0x7f3dd01006d0 2024-03-19T02:55:59.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.804+0000 7f3dceffd640 1 -- 172.21.15.33:0/971798930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f5bea5a30 msgr2=0x562f5bea5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:59.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.804+0000 7f3dceffd640 1 --2- 172.21.15.33:0/971798930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f5bea5a30 0x562f5bea5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.804+0000 7f3dceffd640 1 -- 172.21.15.33:0/971798930 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f5bea6350 msgr2=0x562f5beae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:59.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.804+0000 7f3dceffd640 1 --2- 172.21.15.33:0/971798930 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f5bea6350 0x562f5beae9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.804+0000 7f3dceffd640 1 -- 172.21.15.33:0/971798930 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f5bdd3000 con 0x562f5beaef30 2024-03-19T02:55:59.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.804+0000 7f3dceffd640 1 --2- 172.21.15.33:0/971798930 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f5beaef30 0x562f5beb1320 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4009bf0 tx=0x7f3dc4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd354f3df8f87131 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:59.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.804+0000 7f3dcffff640 1 --2- 172.21.15.33:0/3867564493 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f3db80a2970 0x7f3db80a4e30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8009430 tx=0x7f3dc80095c0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:59.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.804+0000 7f3db5ffb640 1 -- 172.21.15.33:0/971798930 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc4004050 con 0x562f5beaef30 2024-03-19T02:55:59.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.804+0000 7f3db5ffb640 1 -- 172.21.15.33:0/971798930 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3dc40054d0 con 0x562f5beaef30 2024-03-19T02:55:59.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.804+0000 7f3db5ffb640 1 -- 172.21.15.33:0/971798930 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc4004c40 con 0x562f5beaef30 2024-03-19T02:55:59.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.805+0000 7f3dd8a05740 1 -- 172.21.15.33:0/971798930 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f5beaef30 msgr2=0x562f5beb1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:59.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.805+0000 7f3dd8a05740 1 --2- 172.21.15.33:0/971798930 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f5beaef30 0x562f5beb1320 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4009bf0 tx=0x7f3dc4011810 comp rx=0 tx=0).stop 2024-03-19T02:55:59.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.805+0000 7f3dd8a05740 1 -- 172.21.15.33:0/971798930 shutdown_connections 2024-03-19T02:55:59.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.805+0000 7f3dd8a05740 1 --2- 172.21.15.33:0/971798930 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f5beaef30 0x562f5beb1320 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.805+0000 7f3dd8a05740 1 --2- 172.21.15.33:0/971798930 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f5bea6350 0x562f5beae9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.805+0000 7f3dd8a05740 1 --2- 172.21.15.33:0/971798930 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f5bea5a30 0x562f5bea5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.805+0000 7f3dd8a05740 1 -- 172.21.15.33:0/971798930 >> 172.21.15.33:0/971798930 conn(0x562f5bea0160 msgr2=0x562f5bea0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:59.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.805+0000 7f3dd8a05740 1 -- 172.21.15.33:0/971798930 shutdown_connections 2024-03-19T02:55:59.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.805+0000 7f3dd8a05740 1 -- 172.21.15.33:0/971798930 wait complete. 2024-03-19T02:55:59.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.805+0000 7f3dd8a05740 1 -- start start 2024-03-19T02:55:59.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.805+0000 7f3dd8a05740 1 Processor -- start 2024-03-19T02:55:59.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.806+0000 7f3dd8a05740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f5bea5a30 0x562f5beabe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:59.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.806+0000 7f3dd8a05740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f5bea6350 0x562f5beac390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:59.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.806+0000 7f3dd8a05740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f5beaef30 0x562f5beac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:59.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.806+0000 7f3dd8a05740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562f5beb1d20 con 0x562f5bea5a30 2024-03-19T02:55:59.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.806+0000 7f3dd8a05740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x562f5bdc5160 con 0x562f5bea6350 2024-03-19T02:55:59.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.806+0000 7f3dd8a05740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x562f5beb1ea0 con 0x562f5beaef30 2024-03-19T02:55:59.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.806+0000 7f3dd8a05740 10 client.0 _ll_register_callbacks cb 0x562f5be01a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:55:59.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.806+0000 7f3dcf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f5bea5a30 0x562f5beabe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:59.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.806+0000 7f3dcf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f5bea5a30 0x562f5beabe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41996/0 (socket says 172.21.15.33:41996) 2024-03-19T02:55:59.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.806+0000 7f3dcf7fe640 1 -- 172.21.15.33:0/3937222150 learned_addr learned my addr 172.21.15.33:0/3937222150 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:59.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.806+0000 7f3dceffd640 1 --2- 172.21.15.33:0/3937222150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f5bea6350 0x562f5beac390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:59.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.806+0000 7f3dcffff640 1 --2- 172.21.15.33:0/3937222150 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f5beaef30 0x562f5beac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:59.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.806+0000 7f3dcf7fe640 1 -- 172.21.15.33:0/3937222150 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f5beaef30 msgr2=0x562f5beac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:59.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.806+0000 7f3dcf7fe640 1 --2- 172.21.15.33:0/3937222150 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f5beaef30 0x562f5beac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.806+0000 7f3dcf7fe640 1 -- 172.21.15.33:0/3937222150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f5bea6350 msgr2=0x562f5beac390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:59.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.806+0000 7f3dcf7fe640 1 --2- 172.21.15.33:0/3937222150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f5bea6350 0x562f5beac390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.806+0000 7f3dcf7fe640 1 -- 172.21.15.33:0/3937222150 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f5bbe77d0 con 0x562f5bea5a30 2024-03-19T02:55:59.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.806+0000 7f3dcffff640 1 --2- 172.21.15.33:0/3937222150 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f5beaef30 0x562f5beac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:55:59.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.806+0000 7f3dceffd640 1 --2- 172.21.15.33:0/3937222150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f5bea6350 0x562f5beac390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:55:59.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.806+0000 7f3dcf7fe640 1 --2- 172.21.15.33:0/3937222150 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f5bea5a30 0x562f5beabe50 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0000c00 tx=0x7f3dc009b270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bea9087b1c3676f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:59.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.806+0000 7f3daeffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:55:59.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.806+0000 7f3daeffd640 1 -- 172.21.15.33:0/3937222150 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc00a3f10 con 0x562f5bea5a30 2024-03-19T02:55:59.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.806+0000 7f3daeffd640 1 -- 172.21.15.33:0/3937222150 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f3dc00ccae0 con 0x562f5bea5a30 2024-03-19T02:55:59.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.806+0000 7f3dd8a05740 1 -- 172.21.15.33:0/3937222150 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3dc40054d0 con 0x562f5bea5a30 2024-03-19T02:55:59.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.807+0000 7f3daeffd640 1 -- 172.21.15.33:0/3937222150 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3dc00ccce0 con 0x562f5bea5a30 2024-03-19T02:55:59.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.807+0000 7f3dd8a05740 10 client.16164 fetch_fsmap learned FSMap version 25 2024-03-19T02:55:59.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.807+0000 7f3dd8a05740 1 -- 172.21.15.33:0/3937222150 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562f5bb1d3f0 con 0x562f5bea5a30 2024-03-19T02:55:59.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.807+0000 7f3daeffd640 1 -- 172.21.15.33:0/3937222150 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3dc40054d0 con 0x562f5bea5a30 2024-03-19T02:55:59.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.807+0000 7f3daeffd640 1 -- 172.21.15.33:0/3937222150 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f3dc0097570 con 0x562f5bea5a30 2024-03-19T02:55:59.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.807+0000 7f3dd8a05740 10 client.16164 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:55:59.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.807+0000 7f3dd8a05740 10 client.16164 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:55:59.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.807+0000 7f3dd8a05740 20 client.16164 populate_metadata read hostname 'smithi033' 2024-03-19T02:55:59.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.807+0000 7f3dd8a05740 1 --2- 172.21.15.33:0/3937222150 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x562f5bebf1f0 0x562f5bec15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:59.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.807+0000 7f3dd8a05740 4 client.16164 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:55:59.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.807+0000 7f3dd8a05740 1 -- 172.21.15.33:0/3937222150 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562f5be16a40 con 0x562f5bebf1f0 2024-03-19T02:55:59.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.807+0000 7f3dceffd640 1 --2- 172.21.15.33:0/3937222150 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x562f5bebf1f0 0x562f5bec15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:59.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.808+0000 7f3daeffd640 10 client.16164 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:55:59.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.808+0000 7f3dceffd640 1 --2- 172.21.15.33:0/3937222150 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x562f5bebf1f0 0x562f5bec15e0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=341deacb5fccba15 server_cookie=6a50a22ffe4f272c in_seq=0 out_seq=0 2024-03-19T02:55:59.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.808+0000 7f3daeffd640 1 -- 172.21.15.33:0/3937222150 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562f5be16a40 con 0x562f5bebf1f0 2024-03-19T02:55:59.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.808+0000 7f3daeffd640 10 client.16164 handle_command_reply: tid=0 2024-03-19T02:55:59.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.808+0000 7f3dd8a05740 1 client.16164 shutdown 2024-03-19T02:55:59.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.809+0000 7f3dd8a05740 1 -- 172.21.15.33:0/3937222150 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f5bea5a30 msgr2=0x562f5beabe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:59.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.809+0000 7f3dd8a05740 1 --2- 172.21.15.33:0/3937222150 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f5bea5a30 0x562f5beabe50 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0000c00 tx=0x7f3dc009b270 comp rx=0 tx=0).stop 2024-03-19T02:55:59.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.809+0000 7f3dd8a05740 1 -- 172.21.15.33:0/3937222150 shutdown_connections 2024-03-19T02:55:59.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.809+0000 7f3dd8a05740 1 --2- 172.21.15.33:0/3937222150 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x562f5bebf1f0 0x562f5bec15e0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.809+0000 7f3dd8a05740 1 --2- 172.21.15.33:0/3937222150 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f5beaef30 0x562f5beac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.809+0000 7f3dd8a05740 1 --2- 172.21.15.33:0/3937222150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f5bea6350 0x562f5beac390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.809+0000 7f3daeffd640 0 client.16164 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:55:59.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.809+0000 7f3dd8a05740 1 --2- 172.21.15.33:0/3937222150 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f5bea5a30 0x562f5beabe50 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.809+0000 7f3dd8a05740 1 -- 172.21.15.33:0/3937222150 >> 172.21.15.33:0/3937222150 conn(0x562f5bea0160 msgr2=0x562f5bea1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:59.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.809+0000 7f3dd8a05740 1 -- 172.21.15.33:0/3937222150 shutdown_connections 2024-03-19T02:55:59.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.809+0000 7f3dd8a05740 1 -- 172.21.15.33:0/3937222150 wait complete. 2024-03-19T02:55:59.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.809+0000 7f3dd8a05740 20 client.16164 trim_cache size 0 max 16384 2024-03-19T02:55:59.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.826+0000 7f3dd8a05740 1 Processor -- start 2024-03-19T02:55:59.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.826+0000 7f3dd8a05740 1 -- start start 2024-03-19T02:55:59.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.826+0000 7f3dd8a05740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f5bea5a30 0x562f5beae580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:59.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.826+0000 7f3dd8a05740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f5bea6350 0x562f5bea70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:59.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.826+0000 7f3dd8a05740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f5beaef30 0x562f5bebaac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:59.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.826+0000 7f3dd8a05740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd01b1490 con 0x562f5bea5a30 2024-03-19T02:55:59.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.826+0000 7f3dd8a05740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd01aed60 con 0x562f5beaef30 2024-03-19T02:55:59.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.826+0000 7f3dd8a05740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd01acb90 con 0x562f5bea6350 2024-03-19T02:55:59.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.827+0000 7f3dcf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f5bea5a30 0x562f5beae580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:59.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.827+0000 7f3dcf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f5bea5a30 0x562f5beae580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42012/0 (socket says 172.21.15.33:42012) 2024-03-19T02:55:59.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.827+0000 7f3dcf7fe640 1 -- 172.21.15.33:0/4064140504 learned_addr learned my addr 172.21.15.33:0/4064140504 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:59.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.827+0000 7f3dceffd640 1 --2- 172.21.15.33:0/4064140504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f5bea6350 0x562f5bea70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:59.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.827+0000 7f3dcffff640 1 --2- 172.21.15.33:0/4064140504 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f5beaef30 0x562f5bebaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:59.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.827+0000 7f3dcf7fe640 1 -- 172.21.15.33:0/4064140504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f5bea6350 msgr2=0x562f5bea70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:59.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.827+0000 7f3dcf7fe640 1 --2- 172.21.15.33:0/4064140504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f5bea6350 0x562f5bea70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.827+0000 7f3dcf7fe640 1 -- 172.21.15.33:0/4064140504 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f5beaef30 msgr2=0x562f5bebaac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:59.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.827+0000 7f3dcf7fe640 1 --2- 172.21.15.33:0/4064140504 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f5beaef30 0x562f5bebaac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.827+0000 7f3dcf7fe640 1 -- 172.21.15.33:0/4064140504 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f5badd0f0 con 0x562f5bea5a30 2024-03-19T02:55:59.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.827+0000 7f3dcf7fe640 1 --2- 172.21.15.33:0/4064140504 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f5bea5a30 0x562f5beae580 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f3dc009b520 tx=0x7f3dc0098110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d7f250076b149bc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:59.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.827+0000 7f3db57fa640 1 -- 172.21.15.33:0/4064140504 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc009b820 con 0x562f5bea5a30 2024-03-19T02:55:59.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.827+0000 7f3db57fa640 1 -- 172.21.15.33:0/4064140504 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3dc009a910 con 0x562f5bea5a30 2024-03-19T02:55:59.834 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.827+0000 7f3db57fa640 1 -- 172.21.15.33:0/4064140504 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc009ac70 con 0x562f5bea5a30 2024-03-19T02:55:59.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.828+0000 7f3dd8a05740 1 -- 172.21.15.33:0/4064140504 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f5bea5a30 msgr2=0x562f5beae580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:59.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.828+0000 7f3dd8a05740 1 --2- 172.21.15.33:0/4064140504 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f5bea5a30 0x562f5beae580 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f3dc009b520 tx=0x7f3dc0098110 comp rx=0 tx=0).stop 2024-03-19T02:55:59.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.828+0000 7f3dd8a05740 1 -- 172.21.15.33:0/4064140504 shutdown_connections 2024-03-19T02:55:59.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.828+0000 7f3dd8a05740 1 --2- 172.21.15.33:0/4064140504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f5bea6350 0x562f5bea70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.828+0000 7f3dd8a05740 1 --2- 172.21.15.33:0/4064140504 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f5beaef30 0x562f5bebaac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.828+0000 7f3dd8a05740 1 --2- 172.21.15.33:0/4064140504 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f5bea5a30 0x562f5beae580 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.828+0000 7f3dd8a05740 1 -- 172.21.15.33:0/4064140504 >> 172.21.15.33:0/4064140504 conn(0x562f5bea0160 msgr2=0x562f5beaa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:59.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.828+0000 7f3dd8a05740 1 -- 172.21.15.33:0/4064140504 shutdown_connections 2024-03-19T02:55:59.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.828+0000 7f3dd8a05740 1 -- 172.21.15.33:0/4064140504 wait complete. 2024-03-19T02:55:59.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.828+0000 7f3dd8a05740 1 -- start start 2024-03-19T02:55:59.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.828+0000 7f3dd8a05740 1 Processor -- start 2024-03-19T02:55:59.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.829+0000 7f3dd8a05740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f5bea5a30 0x562f5beb72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:59.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.829+0000 7f3dd8a05740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f5bea6350 0x562f5beb2650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:59.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.829+0000 7f3dd8a05740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f5beaef30 0x562f5beb2b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:59.835 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.829+0000 7f3dd8a05740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd01ad220 con 0x562f5bea5a30 2024-03-19T02:55:59.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.829+0000 7f3dd8a05740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd01ae690 con 0x562f5bea6350 2024-03-19T02:55:59.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.829+0000 7f3dd8a05740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd01ad8f0 con 0x562f5beaef30 2024-03-19T02:55:59.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.829+0000 7f3dd8a05740 10 client.0 _ll_register_callbacks cb 0x7f3dc0097b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:55:59.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.829+0000 7f3dcf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f5bea5a30 0x562f5beb72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:59.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.829+0000 7f3dcf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f5bea5a30 0x562f5beb72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42016/0 (socket says 172.21.15.33:42016) 2024-03-19T02:55:59.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.829+0000 7f3dcf7fe640 1 -- 172.21.15.33:0/4184121585 learned_addr learned my addr 172.21.15.33:0/4184121585 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:59.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.829+0000 7f3dceffd640 1 --2- 172.21.15.33:0/4184121585 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f5bea6350 0x562f5beb2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:59.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.829+0000 7f3dcffff640 1 --2- 172.21.15.33:0/4184121585 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f5beaef30 0x562f5beb2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:59.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.829+0000 7f3dcf7fe640 1 -- 172.21.15.33:0/4184121585 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f5beaef30 msgr2=0x562f5beb2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:59.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.829+0000 7f3dcf7fe640 1 --2- 172.21.15.33:0/4184121585 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f5beaef30 0x562f5beb2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.829+0000 7f3dcf7fe640 1 -- 172.21.15.33:0/4184121585 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f5bea6350 msgr2=0x562f5beb2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:55:59.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.829+0000 7f3dcf7fe640 1 --2- 172.21.15.33:0/4184121585 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f5bea6350 0x562f5beb2650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.829+0000 7f3dcf7fe640 1 -- 172.21.15.33:0/4184121585 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f5beb3390 con 0x562f5bea5a30 2024-03-19T02:55:59.836 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.829+0000 7f3dcf7fe640 1 --2- 172.21.15.33:0/4184121585 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f5bea5a30 0x562f5beb72e0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0097c80 tx=0x7f3dc0086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5e00171d380dac4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:59.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.829+0000 7f3daeffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:55:59.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.829+0000 7f3daeffd640 1 -- 172.21.15.33:0/4184121585 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc009b820 con 0x562f5bea5a30 2024-03-19T02:55:59.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.829+0000 7f3daeffd640 1 -- 172.21.15.33:0/4184121585 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f3dc00afb50 con 0x562f5bea5a30 2024-03-19T02:55:59.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.829+0000 7f3dd8a05740 1 -- 172.21.15.33:0/4184121585 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3dc009a910 con 0x562f5bea5a30 2024-03-19T02:55:59.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.829+0000 7f3daeffd640 1 -- 172.21.15.33:0/4184121585 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3dc0082050 con 0x562f5bea5a30 2024-03-19T02:55:59.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.829+0000 7f3dd8a05740 10 client.16170 fetch_fsmap learned FSMap version 25 2024-03-19T02:55:59.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.829+0000 7f3dd8a05740 1 -- 172.21.15.33:0/4184121585 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562f5beb3390 con 0x562f5bea5a30 2024-03-19T02:55:59.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.830+0000 7f3daeffd640 1 -- 172.21.15.33:0/4184121585 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3dc009a910 con 0x562f5bea5a30 2024-03-19T02:55:59.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.830+0000 7f3daeffd640 1 -- 172.21.15.33:0/4184121585 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f3dc00a07e0 con 0x562f5bea5a30 2024-03-19T02:55:59.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.830+0000 7f3dd8a05740 10 client.16170 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:55:59.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.830+0000 7f3dd8a05740 10 client.16170 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:55:59.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.830+0000 7f3dd8a05740 20 client.16170 populate_metadata read hostname 'smithi033' 2024-03-19T02:55:59.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.830+0000 7f3dd8a05740 1 --2- 172.21.15.33:0/4184121585 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x562f5bf25770 0x562f5bf27b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:59.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.830+0000 7f3dd8a05740 4 client.16170 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:55:59.837 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.830+0000 7f3dd8a05740 1 -- 172.21.15.33:0/4184121585 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562f5be10d90 con 0x562f5bf25770 2024-03-19T02:55:59.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.830+0000 7f3dceffd640 1 --2- 172.21.15.33:0/4184121585 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x562f5bf25770 0x562f5bf27b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:59.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.830+0000 7f3dceffd640 1 --2- 172.21.15.33:0/4184121585 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x562f5bf25770 0x562f5bf27b90 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf77f48bbad3260f server_cookie=fdc1f3459990faaa in_seq=0 out_seq=0 2024-03-19T02:55:59.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.830+0000 7f3daeffd640 10 client.16170 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:55:59.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.831+0000 7f3daeffd640 1 -- 172.21.15.33:0/4184121585 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562f5be10d90 con 0x562f5bf25770 2024-03-19T02:55:59.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.831+0000 7f3daeffd640 10 client.16170 handle_command_reply: tid=0 2024-03-19T02:55:59.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.831+0000 7f3dd8a05740 1 client.16170 shutdown 2024-03-19T02:55:59.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.831+0000 7f3dd8a05740 1 -- 172.21.15.33:0/4184121585 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f5bea5a30 msgr2=0x562f5beb72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:59.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.831+0000 7f3dd8a05740 1 --2- 172.21.15.33:0/4184121585 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f5bea5a30 0x562f5beb72e0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0097c80 tx=0x7f3dc0086040 comp rx=0 tx=0).stop 2024-03-19T02:55:59.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.831+0000 7f3dd8a05740 1 -- 172.21.15.33:0/4184121585 shutdown_connections 2024-03-19T02:55:59.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.831+0000 7f3dd8a05740 1 --2- 172.21.15.33:0/4184121585 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x562f5bf25770 0x562f5bf27b90 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.831+0000 7f3dd8a05740 1 --2- 172.21.15.33:0/4184121585 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f5beaef30 0x562f5beb2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.831+0000 7f3dd8a05740 1 --2- 172.21.15.33:0/4184121585 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f5bea6350 0x562f5beb2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.831+0000 7f3daeffd640 0 client.16170 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:55:59.838 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.831+0000 7f3dd8a05740 1 --2- 172.21.15.33:0/4184121585 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f5bea5a30 0x562f5beb72e0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.831+0000 7f3dd8a05740 1 -- 172.21.15.33:0/4184121585 >> 172.21.15.33:0/4184121585 conn(0x562f5bea0160 msgr2=0x562f5beaae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:59.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.831+0000 7f3dd8a05740 1 -- 172.21.15.33:0/4184121585 shutdown_connections 2024-03-19T02:55:59.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.831+0000 7f3dd8a05740 1 -- 172.21.15.33:0/4184121585 wait complete. 2024-03-19T02:55:59.839 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.831+0000 7f3dd8a05740 20 client.16170 trim_cache size 0 max 16384 2024-03-19T02:55:59.839 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:55:59.839 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-03-19T02:55:59.839 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-03-19T02:55:59.839 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:55:59.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.831+0000 7f3dd5d44640 1 -- 172.21.15.33:0/3867564493 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f3db80a2970 msgr2=0x7f3db80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:59.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.831+0000 7f3dd5d44640 1 --2- 172.21.15.33:0/3867564493 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f3db80a2970 0x7f3db80a4e30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8009430 tx=0x7f3dc80095c0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.831+0000 7f3dd5d44640 1 -- 172.21.15.33:0/3867564493 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dd01006d0 msgr2=0x7f3dd0198e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:59.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.831+0000 7f3dd5d44640 1 --2- 172.21.15.33:0/3867564493 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dd01006d0 0x7f3dd0198e40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3dc000e9f0 tx=0x7f3dc000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.831+0000 7f3dd5d44640 1 -- 172.21.15.33:0/3867564493 shutdown_connections 2024-03-19T02:55:59.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.831+0000 7f3dd5d44640 1 --2- 172.21.15.33:0/3867564493 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f3db80a2970 0x7f3db80a4e30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.831+0000 7f3dd5d44640 1 --2- 172.21.15.33:0/3867564493 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3dd0101090 0x7f3dd0199380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.831+0000 7f3dd5d44640 1 --2- 172.21.15.33:0/3867564493 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3dd01066e0 0x7f3dd01a3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.831+0000 7f3dd5d44640 1 --2- 172.21.15.33:0/3867564493 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3dd01006d0 0x7f3dd0198e40 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.831+0000 7f3dd5d44640 1 -- 172.21.15.33:0/3867564493 >> 172.21.15.33:0/3867564493 conn(0x7f3dd0076aa0 msgr2=0x7f3dd00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:59.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.832+0000 7f3dd5d44640 1 -- 172.21.15.33:0/3867564493 shutdown_connections 2024-03-19T02:55:59.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.832+0000 7f3dd5d44640 1 -- 172.21.15.33:0/3867564493 wait complete. 2024-03-19T02:55:59.843 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:55:59.844 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:55:59.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.946+0000 7fdaead77640 1 -- 172.21.15.33:0/2616655013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdae41077c0 msgr2=0x7fdae4107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:59.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.946+0000 7fdaead77640 1 --2- 172.21.15.33:0/2616655013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdae41077c0 0x7fdae4107ba0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fdadc009930 tx=0x7fdadc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:55:59.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.946+0000 7fdaead77640 1 -- 172.21.15.33:0/2616655013 shutdown_connections 2024-03-19T02:55:59.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.946+0000 7fdaead77640 1 --2- 172.21.15.33:0/2616655013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdae4102170 0x7fdae410e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.946+0000 7fdaead77640 1 --2- 172.21.15.33:0/2616655013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdae41017b0 0x7fdae4101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.946+0000 7fdaead77640 1 --2- 172.21.15.33:0/2616655013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdae41077c0 0x7fdae4107ba0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.946+0000 7fdaead77640 1 -- 172.21.15.33:0/2616655013 >> 172.21.15.33:0/2616655013 conn(0x7fdae40fd2c0 msgr2=0x7fdae40ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:55:59.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.946+0000 7fdaead77640 1 -- 172.21.15.33:0/2616655013 shutdown_connections 2024-03-19T02:55:59.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.946+0000 7fdaead77640 1 -- 172.21.15.33:0/2616655013 wait complete. 2024-03-19T02:55:59.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.946+0000 7fdaead77640 1 Processor -- start 2024-03-19T02:55:59.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.946+0000 7fdaead77640 1 -- start start 2024-03-19T02:55:59.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.947+0000 7fdaead77640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdae41017b0 0x7fdae4198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:59.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.947+0000 7fdaead77640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdae4102170 0x7fdae4198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:59.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.947+0000 7fdaead77640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdae41077c0 0x7fdae419c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:59.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.947+0000 7fdaead77640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdae4111a80 con 0x7fdae41017b0 2024-03-19T02:55:59.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.947+0000 7fdaead77640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdae4111900 con 0x7fdae4102170 2024-03-19T02:55:59.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.947+0000 7fdaead77640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fdae4111c00 con 0x7fdae41077c0 2024-03-19T02:55:59.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.947+0000 7fdae8b13640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdae41017b0 0x7fdae4198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:59.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.947+0000 7fdae8b13640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdae41017b0 0x7fdae4198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42038/0 (socket says 172.21.15.33:42038) 2024-03-19T02:55:59.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.947+0000 7fdae8b13640 1 -- 172.21.15.33:0/3516465662 learned_addr learned my addr 172.21.15.33:0/3516465662 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:55:59.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.947+0000 7fdae3fff640 1 --2- 172.21.15.33:0/3516465662 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdae4102170 0x7fdae4198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:59.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.947+0000 7fdae9314640 1 --2- 172.21.15.33:0/3516465662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdae41077c0 0x7fdae419c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:59.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.947+0000 7fdae8b13640 1 -- 172.21.15.33:0/3516465662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdae41077c0 msgr2=0x7fdae419c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:59.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.947+0000 7fdae8b13640 1 --2- 172.21.15.33:0/3516465662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdae41077c0 0x7fdae419c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.947+0000 7fdae8b13640 1 -- 172.21.15.33:0/3516465662 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdae4102170 msgr2=0x7fdae4198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:55:59.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.947+0000 7fdae8b13640 1 --2- 172.21.15.33:0/3516465662 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdae4102170 0x7fdae4198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:55:59.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.947+0000 7fdae8b13640 1 -- 172.21.15.33:0/3516465662 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdae419d040 con 0x7fdae41017b0 2024-03-19T02:55:59.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.947+0000 7fdae8b13640 1 --2- 172.21.15.33:0/3516465662 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdae41017b0 0x7fdae4198050 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fdadc009230 tx=0x7fdadc02c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:59.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.947+0000 7fdae1ffb640 1 -- 172.21.15.33:0/3516465662 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdadc043020 con 0x7fdae41017b0 2024-03-19T02:55:59.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.947+0000 7fdaead77640 1 -- 172.21.15.33:0/3516465662 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdae419d2d0 con 0x7fdae41017b0 2024-03-19T02:55:59.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.947+0000 7fdae1ffb640 1 -- 172.21.15.33:0/3516465662 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdadc02ca70 con 0x7fdae41017b0 2024-03-19T02:55:59.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.947+0000 7fdae1ffb640 1 -- 172.21.15.33:0/3516465662 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdadc003a50 con 0x7fdae41017b0 2024-03-19T02:55:59.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.948+0000 7fdaead77640 1 -- 172.21.15.33:0/3516465662 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdae41a4bc0 con 0x7fdae41017b0 2024-03-19T02:55:59.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.948+0000 7fdaead77640 1 -- 172.21.15.33:0/3516465662 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdae4068a60 con 0x7fdae41017b0 2024-03-19T02:55:59.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.950+0000 7fdae1ffb640 1 -- 172.21.15.33:0/3516465662 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fdadc046020 con 0x7fdae41017b0 2024-03-19T02:55:59.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.950+0000 7fdae1ffb640 1 --2- 172.21.15.33:0/3516465662 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fdad00a2970 0x7fdad00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:55:59.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.950+0000 7fdae3fff640 1 --2- 172.21.15.33:0/3516465662 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fdad00a2970 0x7fdad00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:55:59.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.950+0000 7fdae1ffb640 1 -- 172.21.15.33:0/3516465662 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fdadc0f1450 con 0x7fdae41017b0 2024-03-19T02:55:59.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.951+0000 7fdae3fff640 1 --2- 172.21.15.33:0/3516465662 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fdad00a2970 0x7fdad00a4e30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdae40ffa20 tx=0x7fdad4009500 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:55:59.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:55:59.954+0000 7fdae1ffb640 1 -- 172.21.15.33:0/3516465662 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdadc04d020 con 0x7fdae41017b0 2024-03-19T02:56:00.176 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.175+0000 7fdaead77640 1 -- 172.21.15.33:0/3516465662 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdae41993a0 con 0x7fdae41017b0 2024-03-19T02:56:00.177 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.176+0000 7fdae1ffb640 1 -- 172.21.15.33:0/3516465662 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11867 (secure 0 0 0) 0x7fdae41993a0 con 0x7fdae41017b0 2024-03-19T02:56:00.177 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:56:00.177 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:50:52.201798+0000","modified":"2024-03-19T02:53:11.748697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24488,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/3731232169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":3731232169},{"type":"v1","addr":"172.21.15.96:6839","nonce":3731232169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24457,24488,24509]},"id":1}]} 2024-03-19T02:56:00.178 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-03-19T02:56:00.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.178+0000 7fdaead77640 1 -- 172.21.15.33:0/3516465662 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fdad00a2970 msgr2=0x7fdad00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:00.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.178+0000 7fdaead77640 1 --2- 172.21.15.33:0/3516465662 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fdad00a2970 0x7fdad00a4e30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdae40ffa20 tx=0x7fdad4009500 comp rx=0 tx=0).stop 2024-03-19T02:56:00.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.178+0000 7fdaead77640 1 -- 172.21.15.33:0/3516465662 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdae41017b0 msgr2=0x7fdae4198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:00.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.178+0000 7fdaead77640 1 --2- 172.21.15.33:0/3516465662 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdae41017b0 0x7fdae4198050 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fdadc009230 tx=0x7fdadc02c5d0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.178+0000 7fdaead77640 1 -- 172.21.15.33:0/3516465662 shutdown_connections 2024-03-19T02:56:00.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.178+0000 7fdaead77640 1 --2- 172.21.15.33:0/3516465662 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fdad00a2970 0x7fdad00a4e30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.178+0000 7fdaead77640 1 --2- 172.21.15.33:0/3516465662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdae41077c0 0x7fdae419c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.178+0000 7fdaead77640 1 --2- 172.21.15.33:0/3516465662 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdae4102170 0x7fdae4198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.178+0000 7fdaead77640 1 --2- 172.21.15.33:0/3516465662 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdae41017b0 0x7fdae4198050 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.178+0000 7fdaead77640 1 -- 172.21.15.33:0/3516465662 >> 172.21.15.33:0/3516465662 conn(0x7fdae40fd2c0 msgr2=0x7fdae40fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:00.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.178+0000 7fdaead77640 1 -- 172.21.15.33:0/3516465662 shutdown_connections 2024-03-19T02:56:00.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.178+0000 7fdaead77640 1 -- 172.21.15.33:0/3516465662 wait complete. 2024-03-19T02:56:00.191 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:56:00.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.294+0000 7f1093ce8640 1 -- 172.21.15.33:0/1723503181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f108c06a7f0 msgr2=0x7f108c10e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:00.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.294+0000 7f1093ce8640 1 --2- 172.21.15.33:0/1723503181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f108c06a7f0 0x7f108c10e660 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1084009860 tx=0x7f108402bb70 comp rx=0 tx=0).stop 2024-03-19T02:56:00.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.294+0000 7f1093ce8640 1 -- 172.21.15.33:0/1723503181 shutdown_connections 2024-03-19T02:56:00.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.294+0000 7f1093ce8640 1 --2- 172.21.15.33:0/1723503181 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f108c069e50 0x7f108c06a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.294+0000 7f1093ce8640 1 --2- 172.21.15.33:0/1723503181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f108c06a7f0 0x7f108c10e660 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.294+0000 7f1093ce8640 1 --2- 172.21.15.33:0/1723503181 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f108c107970 0x7f108c107d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.294+0000 7f1093ce8640 1 -- 172.21.15.33:0/1723503181 >> 172.21.15.33:0/1723503181 conn(0x7f108c0774c0 msgr2=0x7f108c0778d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:00.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.295+0000 7f1093ce8640 1 -- 172.21.15.33:0/1723503181 shutdown_connections 2024-03-19T02:56:00.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.295+0000 7f1093ce8640 1 -- 172.21.15.33:0/1723503181 wait complete. 2024-03-19T02:56:00.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.295+0000 7f1093ce8640 1 Processor -- start 2024-03-19T02:56:00.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.295+0000 7f1093ce8640 1 -- start start 2024-03-19T02:56:00.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.295+0000 7f1093ce8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f108c069e50 0x7f108c197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.295+0000 7f1093ce8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f108c06a7f0 0x7f108c1983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.295+0000 7f1093ce8640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f108c107970 0x7f108c19c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.295+0000 7f1093ce8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f108c111a90 con 0x7f108c06a7f0 2024-03-19T02:56:00.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.295+0000 7f1093ce8640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f108c111910 con 0x7f108c069e50 2024-03-19T02:56:00.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.295+0000 7f1093ce8640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f108c111c10 con 0x7f108c107970 2024-03-19T02:56:00.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.295+0000 7f1091283640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f108c06a7f0 0x7f108c1983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.296+0000 7f1091283640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f108c06a7f0 0x7f108c1983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42058/0 (socket says 172.21.15.33:42058) 2024-03-19T02:56:00.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.296+0000 7f1091a84640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f108c069e50 0x7f108c197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.296+0000 7f1091283640 1 -- 172.21.15.33:0/106493369 learned_addr learned my addr 172.21.15.33:0/106493369 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:00.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.296+0000 7f1092285640 1 --2- 172.21.15.33:0/106493369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f108c107970 0x7f108c19c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.296+0000 7f1091283640 1 -- 172.21.15.33:0/106493369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f108c107970 msgr2=0x7f108c19c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:00.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.296+0000 7f1091283640 1 --2- 172.21.15.33:0/106493369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f108c107970 0x7f108c19c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.296+0000 7f1091283640 1 -- 172.21.15.33:0/106493369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f108c069e50 msgr2=0x7f108c197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:00.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.296+0000 7f1091283640 1 --2- 172.21.15.33:0/106493369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f108c069e50 0x7f108c197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.296+0000 7f1091283640 1 -- 172.21.15.33:0/106493369 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f108c19ce80 con 0x7f108c06a7f0 2024-03-19T02:56:00.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.296+0000 7f1091a84640 1 --2- 172.21.15.33:0/106493369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f108c069e50 0x7f108c197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:00.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.296+0000 7f1091283640 1 --2- 172.21.15.33:0/106493369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f108c06a7f0 0x7f108c1983d0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f108000e9f0 tx=0x7f108000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:00.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.296+0000 7f108affd640 1 -- 172.21.15.33:0/106493369 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108000cd80 con 0x7f108c06a7f0 2024-03-19T02:56:00.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.296+0000 7f1093ce8640 1 -- 172.21.15.33:0/106493369 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f108c19d170 con 0x7f108c06a7f0 2024-03-19T02:56:00.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.296+0000 7f108affd640 1 -- 172.21.15.33:0/106493369 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f10800108e0 con 0x7f108c06a7f0 2024-03-19T02:56:00.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.296+0000 7f108affd640 1 -- 172.21.15.33:0/106493369 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1080018920 con 0x7f108c06a7f0 2024-03-19T02:56:00.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.296+0000 7f1093ce8640 1 -- 172.21.15.33:0/106493369 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f108c1a4a00 con 0x7f108c06a7f0 2024-03-19T02:56:00.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.298+0000 7f108affd640 1 -- 172.21.15.33:0/106493369 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f1080010a90 con 0x7f108c06a7f0 2024-03-19T02:56:00.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.299+0000 7f10969aa740 1 Processor -- start 2024-03-19T02:56:00.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.299+0000 7f10969aa740 1 -- start start 2024-03-19T02:56:00.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.299+0000 7f10969aa740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fdb758f80 0x560fdb759360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.299+0000 7f10969aa740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fdb7598a0 0x560fdb7666a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.299+0000 7f10969aa740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fdb766be0 0x560fdb768fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.299+0000 7f108affd640 1 --2- 172.21.15.33:0/106493369 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f10740a2970 0x7f10740a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.299+0000 7f10969aa740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560fdb6bc4b0 con 0x560fdb7598a0 2024-03-19T02:56:00.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.299+0000 7f10969aa740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x560fdb664d70 con 0x560fdb758f80 2024-03-19T02:56:00.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.299+0000 7f10969aa740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560fdb6b8550 con 0x560fdb766be0 2024-03-19T02:56:00.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.299+0000 7f1091a84640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fdb7598a0 0x560fdb7666a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.299+0000 7f1091283640 1 --2- 172.21.15.33:0/106493369 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f10740a2970 0x7f10740a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.299+0000 7f1092285640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fdb758f80 0x560fdb759360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.299+0000 7f1091a84640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fdb7598a0 0x560fdb7666a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42060/0 (socket says 172.21.15.33:42060) 2024-03-19T02:56:00.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.299+0000 7f1091a84640 1 -- 172.21.15.33:0/1090216750 learned_addr learned my addr 172.21.15.33:0/1090216750 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:00.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.299+0000 7f1092285640 1 --2- 172.21.15.33:0/1090216750 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fdb766be0 0x560fdb768fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.299+0000 7f108affd640 1 -- 172.21.15.33:0/106493369 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f10800cf450 con 0x7f108c06a7f0 2024-03-19T02:56:00.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.300+0000 7f1091a84640 1 -- 172.21.15.33:0/1090216750 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fdb766be0 msgr2=0x560fdb768fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:00.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.300+0000 7f1091a84640 1 --2- 172.21.15.33:0/1090216750 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fdb766be0 0x560fdb768fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.300+0000 7f1091a84640 1 -- 172.21.15.33:0/1090216750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fdb758f80 msgr2=0x560fdb759360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:00.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.300+0000 7f1091a84640 1 --2- 172.21.15.33:0/1090216750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fdb758f80 0x560fdb759360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.300+0000 7f1091a84640 1 -- 172.21.15.33:0/1090216750 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fdb687210 con 0x560fdb7598a0 2024-03-19T02:56:00.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.300+0000 7f1092285640 1 --2- 172.21.15.33:0/1090216750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fdb758f80 0x560fdb759360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:00.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.300+0000 7f1091283640 1 --2- 172.21.15.33:0/106493369 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f10740a2970 0x7f10740a4e30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1080097570 tx=0x7f1080097500 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:00.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.300+0000 7f1091a84640 1 --2- 172.21.15.33:0/1090216750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fdb7598a0 0x560fdb7666a0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f107c009fd0 tx=0x7f107c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf147dc1d84a04fb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:00.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.300+0000 7f1073fff640 1 -- 172.21.15.33:0/1090216750 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107c00cd60 con 0x560fdb7598a0 2024-03-19T02:56:00.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.300+0000 7f1073fff640 1 -- 172.21.15.33:0/1090216750 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f107c01d070 con 0x560fdb7598a0 2024-03-19T02:56:00.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.300+0000 7f1073fff640 1 -- 172.21.15.33:0/1090216750 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107c00b5c0 con 0x560fdb7598a0 2024-03-19T02:56:00.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.301+0000 7f10969aa740 1 -- 172.21.15.33:0/1090216750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fdb7598a0 msgr2=0x560fdb7666a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:00.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.301+0000 7f10969aa740 1 --2- 172.21.15.33:0/1090216750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fdb7598a0 0x560fdb7666a0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f107c009fd0 tx=0x7f107c00ec80 comp rx=0 tx=0).stop 2024-03-19T02:56:00.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.301+0000 7f10969aa740 1 -- 172.21.15.33:0/1090216750 shutdown_connections 2024-03-19T02:56:00.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.301+0000 7f10969aa740 1 --2- 172.21.15.33:0/1090216750 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fdb766be0 0x560fdb768fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.301+0000 7f10969aa740 1 --2- 172.21.15.33:0/1090216750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fdb7598a0 0x560fdb7666a0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.301+0000 7f10969aa740 1 --2- 172.21.15.33:0/1090216750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fdb758f80 0x560fdb759360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.301+0000 7f10969aa740 1 -- 172.21.15.33:0/1090216750 >> 172.21.15.33:0/1090216750 conn(0x560fdb753a00 msgr2=0x560fdb753de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:00.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.301+0000 7f10969aa740 1 -- 172.21.15.33:0/1090216750 shutdown_connections 2024-03-19T02:56:00.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.301+0000 7f10969aa740 1 -- 172.21.15.33:0/1090216750 wait complete. 2024-03-19T02:56:00.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.301+0000 7f10969aa740 1 -- start start 2024-03-19T02:56:00.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.301+0000 7f10969aa740 1 Processor -- start 2024-03-19T02:56:00.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.301+0000 7f10969aa740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fdb758f80 0x560fdb763b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.302+0000 7f10969aa740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fdb7598a0 0x560fdb764060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.302+0000 7f10969aa740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fdb766be0 0x560fdb7645a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.302+0000 7f10969aa740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560fdb769a20 con 0x560fdb766be0 2024-03-19T02:56:00.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.302+0000 7f10969aa740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x560fdb678310 con 0x560fdb758f80 2024-03-19T02:56:00.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.302+0000 7f10969aa740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560fdb769ba0 con 0x560fdb7598a0 2024-03-19T02:56:00.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.302+0000 7f10969aa740 10 client.0 _ll_register_callbacks cb 0x560fdb66c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:56:00.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.302+0000 7f1092285640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fdb766be0 0x560fdb7645a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.302+0000 7f1091a84640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fdb7598a0 0x560fdb764060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.302+0000 7f1091a84640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fdb7598a0 0x560fdb764060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:52254/0 (socket says 172.21.15.33:52254) 2024-03-19T02:56:00.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.302+0000 7f1092285640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fdb766be0 0x560fdb7645a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42064/0 (socket says 172.21.15.33:42064) 2024-03-19T02:56:00.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.302+0000 7f1091a84640 1 -- 172.21.15.33:0/1170821628 learned_addr learned my addr 172.21.15.33:0/1170821628 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:00.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.302+0000 7f1092285640 1 --2- 172.21.15.33:0/1170821628 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fdb758f80 0x560fdb763b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.302+0000 7f1091a84640 1 -- 172.21.15.33:0/1170821628 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fdb758f80 msgr2=0x560fdb763b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:00.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.302+0000 7f1091a84640 1 --2- 172.21.15.33:0/1170821628 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fdb758f80 0x560fdb763b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.302+0000 7f1091a84640 1 -- 172.21.15.33:0/1170821628 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fdb766be0 msgr2=0x560fdb7645a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:00.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.302+0000 7f1091a84640 1 --2- 172.21.15.33:0/1170821628 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fdb766be0 0x560fdb7645a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.302+0000 7f1091a84640 1 -- 172.21.15.33:0/1170821628 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fdb3d03f0 con 0x560fdb7598a0 2024-03-19T02:56:00.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.302+0000 7f1092285640 1 --2- 172.21.15.33:0/1170821628 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fdb766be0 0x560fdb7645a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:00.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.302+0000 7f1091a84640 1 --2- 172.21.15.33:0/1170821628 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fdb7598a0 0x560fdb764060 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f107c000f80 tx=0x7f107c00cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df1f4bdd82e1bbe2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:00.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.302+0000 7f10717fa640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-19T02:56:00.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.302+0000 7f10717fa640 1 -- 172.21.15.33:0/1170821628 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107c004e50 con 0x560fdb7598a0 2024-03-19T02:56:00.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.302+0000 7f10969aa740 1 -- 172.21.15.33:0/1170821628 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f107c01d070 con 0x560fdb7598a0 2024-03-19T02:56:00.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.302+0000 7f10717fa640 1 -- 172.21.15.33:0/1170821628 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f107c0183f0 con 0x560fdb7598a0 2024-03-19T02:56:00.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.303+0000 7f10717fa640 1 -- 172.21.15.33:0/1170821628 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f107c033050 con 0x560fdb7598a0 2024-03-19T02:56:00.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.303+0000 7f10969aa740 10 client.25882 fetch_fsmap learned FSMap version 25 2024-03-19T02:56:00.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.303+0000 7f10969aa740 1 -- 172.21.15.33:0/1170821628 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x560fdb49a7d0 con 0x560fdb7598a0 2024-03-19T02:56:00.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.303+0000 7f10717fa640 1 -- 172.21.15.33:0/1170821628 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f107c01d070 con 0x560fdb7598a0 2024-03-19T02:56:00.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.303+0000 7f10717fa640 1 -- 172.21.15.33:0/1170821628 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f107c00b040 con 0x560fdb7598a0 2024-03-19T02:56:00.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.303+0000 7f10969aa740 10 client.25882 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:56:00.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.303+0000 7f10969aa740 10 client.25882 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T02:56:00.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.303+0000 7f10969aa740 20 client.25882 populate_metadata read hostname 'smithi033' 2024-03-19T02:56:00.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.303+0000 7f10969aa740 1 --2- 172.21.15.33:0/1170821628 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x560fdb776f10 0x560fdb779300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.303+0000 7f10969aa740 4 client.25882 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:56:00.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.303+0000 7f10969aa740 1 -- 172.21.15.33:0/1170821628 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560fdb71a3e0 con 0x560fdb776f10 2024-03-19T02:56:00.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.303+0000 7f1092285640 1 --2- 172.21.15.33:0/1170821628 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x560fdb776f10 0x560fdb779300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.304+0000 7f10717fa640 10 client.25882 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T02:56:00.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.304+0000 7f1092285640 1 --2- 172.21.15.33:0/1170821628 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x560fdb776f10 0x560fdb779300 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=15734b8832fe9dc server_cookie=a62342a3114d63b9 in_seq=0 out_seq=0 2024-03-19T02:56:00.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.304+0000 7f10717fa640 1 -- 172.21.15.33:0/1170821628 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560fdb71a3e0 con 0x560fdb776f10 2024-03-19T02:56:00.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.304+0000 7f10717fa640 10 client.25882 handle_command_reply: tid=0 2024-03-19T02:56:00.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.304+0000 7f10969aa740 1 client.25882 shutdown 2024-03-19T02:56:00.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.304+0000 7f10969aa740 1 -- 172.21.15.33:0/1170821628 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fdb7598a0 msgr2=0x560fdb764060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:00.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.304+0000 7f10969aa740 1 --2- 172.21.15.33:0/1170821628 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fdb7598a0 0x560fdb764060 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f107c000f80 tx=0x7f107c00cf90 comp rx=0 tx=0).stop 2024-03-19T02:56:00.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.304+0000 7f10969aa740 1 -- 172.21.15.33:0/1170821628 shutdown_connections 2024-03-19T02:56:00.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.304+0000 7f10969aa740 1 --2- 172.21.15.33:0/1170821628 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x560fdb776f10 0x560fdb779300 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.304+0000 7f10969aa740 1 --2- 172.21.15.33:0/1170821628 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fdb7598a0 0x560fdb764060 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.304+0000 7f10969aa740 1 --2- 172.21.15.33:0/1170821628 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fdb766be0 0x560fdb7645a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.304+0000 7f10717fa640 0 client.25882 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T02:56:00.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.304+0000 7f10969aa740 1 --2- 172.21.15.33:0/1170821628 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fdb758f80 0x560fdb763b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.304+0000 7f10969aa740 1 -- 172.21.15.33:0/1170821628 >> 172.21.15.33:0/1170821628 conn(0x560fdb753a00 msgr2=0x560fdb755750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:00.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.304+0000 7f10969aa740 1 -- 172.21.15.33:0/1170821628 shutdown_connections 2024-03-19T02:56:00.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.305+0000 7f10969aa740 1 -- 172.21.15.33:0/1170821628 wait complete. 2024-03-19T02:56:00.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.305+0000 7f10969aa740 20 client.25882 trim_cache size 0 max 16384 2024-03-19T02:56:00.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.318+0000 7f10969aa740 1 Processor -- start 2024-03-19T02:56:00.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.318+0000 7f10969aa740 1 -- start start 2024-03-19T02:56:00.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.318+0000 7f10969aa740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fdb758f80 0x560fdb766250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.318+0000 7f10969aa740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fdb7598a0 0x560fdb772860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.318+0000 7f10969aa740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fdb766be0 0x560fdb762c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.318+0000 7f10969aa740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f108c1af250 con 0x560fdb7598a0 2024-03-19T02:56:00.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.318+0000 7f10969aa740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f108c1acb20 con 0x560fdb758f80 2024-03-19T02:56:00.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.318+0000 7f10969aa740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f108c1aa950 con 0x560fdb766be0 2024-03-19T02:56:00.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.318+0000 7f1091a84640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fdb7598a0 0x560fdb772860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.318+0000 7f1091a84640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fdb7598a0 0x560fdb772860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42074/0 (socket says 172.21.15.33:42074) 2024-03-19T02:56:00.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.318+0000 7f1091a84640 1 -- 172.21.15.33:0/3511101732 learned_addr learned my addr 172.21.15.33:0/3511101732 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:00.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.318+0000 7f1092285640 1 --2- 172.21.15.33:0/3511101732 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fdb758f80 0x560fdb766250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.318+0000 7f1092285640 1 --2- 172.21.15.33:0/3511101732 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fdb766be0 0x560fdb762c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.318+0000 7f1091a84640 1 -- 172.21.15.33:0/3511101732 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fdb766be0 msgr2=0x560fdb762c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:00.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.318+0000 7f1091a84640 1 --2- 172.21.15.33:0/3511101732 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fdb766be0 0x560fdb762c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.318+0000 7f1091a84640 1 -- 172.21.15.33:0/3511101732 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fdb758f80 msgr2=0x560fdb766250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:00.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.318+0000 7f1091a84640 1 --2- 172.21.15.33:0/3511101732 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fdb758f80 0x560fdb766250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.318+0000 7f1091a84640 1 -- 172.21.15.33:0/3511101732 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fdb7596d0 con 0x560fdb7598a0 2024-03-19T02:56:00.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.318+0000 7f1092285640 1 --2- 172.21.15.33:0/3511101732 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fdb766be0 0x560fdb762c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:56:00.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.318+0000 7f1091a84640 1 --2- 172.21.15.33:0/3511101732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fdb7598a0 0x560fdb772860 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f107c00cf10 tx=0x7f107c00c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=753bf56011e6ce3f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:00.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.318+0000 7f10737fe640 1 -- 172.21.15.33:0/3511101732 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107c009cc0 con 0x560fdb7598a0 2024-03-19T02:56:00.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.319+0000 7f10737fe640 1 -- 172.21.15.33:0/3511101732 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f107c01d040 con 0x560fdb7598a0 2024-03-19T02:56:00.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.319+0000 7f10737fe640 1 -- 172.21.15.33:0/3511101732 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107c004120 con 0x560fdb7598a0 2024-03-19T02:56:00.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.319+0000 7f10969aa740 1 -- 172.21.15.33:0/3511101732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fdb7598a0 msgr2=0x560fdb772860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:00.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.319+0000 7f10969aa740 1 --2- 172.21.15.33:0/3511101732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fdb7598a0 0x560fdb772860 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f107c00cf10 tx=0x7f107c00c740 comp rx=0 tx=0).stop 2024-03-19T02:56:00.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.319+0000 7f10969aa740 1 -- 172.21.15.33:0/3511101732 shutdown_connections 2024-03-19T02:56:00.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.319+0000 7f10969aa740 1 --2- 172.21.15.33:0/3511101732 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fdb766be0 0x560fdb762c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.319+0000 7f10969aa740 1 --2- 172.21.15.33:0/3511101732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fdb7598a0 0x560fdb772860 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.319+0000 7f10969aa740 1 --2- 172.21.15.33:0/3511101732 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fdb758f80 0x560fdb766250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.319+0000 7f10969aa740 1 -- 172.21.15.33:0/3511101732 >> 172.21.15.33:0/3511101732 conn(0x560fdb753a00 msgr2=0x560fdb752310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:00.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.319+0000 7f10969aa740 1 -- 172.21.15.33:0/3511101732 shutdown_connections 2024-03-19T02:56:00.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.319+0000 7f10969aa740 1 -- 172.21.15.33:0/3511101732 wait complete. 2024-03-19T02:56:00.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.319+0000 7f10969aa740 1 -- start start 2024-03-19T02:56:00.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.319+0000 7f10969aa740 1 Processor -- start 2024-03-19T02:56:00.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.320+0000 7f10969aa740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fdb758f80 0x560fdb76a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.320+0000 7f10969aa740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fdb7598a0 0x560fdb76a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.320+0000 7f10969aa740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fdb766be0 0x560fdb7db1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.320+0000 7f10969aa740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f108c1aafe0 con 0x560fdb7598a0 2024-03-19T02:56:00.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.320+0000 7f10969aa740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f108c1ac450 con 0x560fdb758f80 2024-03-19T02:56:00.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.320+0000 7f10969aa740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f108c1ab6b0 con 0x560fdb766be0 2024-03-19T02:56:00.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.320+0000 7f10969aa740 10 client.0 _ll_register_callbacks cb 0x7f107c009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:56:00.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.320+0000 7f1091a84640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fdb7598a0 0x560fdb76a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.320+0000 7f1091a84640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fdb7598a0 0x560fdb76a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42086/0 (socket says 172.21.15.33:42086) 2024-03-19T02:56:00.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.320+0000 7f1091a84640 1 -- 172.21.15.33:0/334450525 learned_addr learned my addr 172.21.15.33:0/334450525 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:00.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.320+0000 7f1092285640 1 --2- 172.21.15.33:0/334450525 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fdb758f80 0x560fdb76a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.320+0000 7f1092285640 1 --2- 172.21.15.33:0/334450525 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fdb766be0 0x560fdb7db1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.320+0000 7f1091a84640 1 -- 172.21.15.33:0/334450525 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fdb766be0 msgr2=0x560fdb7db1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:00.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.320+0000 7f1091a84640 1 --2- 172.21.15.33:0/334450525 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fdb766be0 0x560fdb7db1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.320+0000 7f1091a84640 1 -- 172.21.15.33:0/334450525 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fdb758f80 msgr2=0x560fdb76a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:00.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.320+0000 7f1091a84640 1 --2- 172.21.15.33:0/334450525 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fdb758f80 0x560fdb76a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.320+0000 7f1091a84640 1 -- 172.21.15.33:0/334450525 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fdb7db710 con 0x560fdb7598a0 2024-03-19T02:56:00.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.320+0000 7f1092285640 1 --2- 172.21.15.33:0/334450525 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fdb766be0 0x560fdb7db1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:56:00.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.320+0000 7f1091a84640 1 --2- 172.21.15.33:0/334450525 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fdb7598a0 0x560fdb76a580 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f107c000f80 tx=0x7f107c00e250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c7fad201283e49 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:00.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.320+0000 7f10717fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:56:00.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.321+0000 7f10717fa640 1 -- 172.21.15.33:0/334450525 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107c018a10 con 0x560fdb7598a0 2024-03-19T02:56:00.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.321+0000 7f10717fa640 1 -- 172.21.15.33:0/334450525 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f107c018600 con 0x560fdb7598a0 2024-03-19T02:56:00.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.321+0000 7f10969aa740 1 -- 172.21.15.33:0/334450525 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f107c01d040 con 0x560fdb7598a0 2024-03-19T02:56:00.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.321+0000 7f10717fa640 1 -- 172.21.15.33:0/334450525 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f107c003970 con 0x560fdb7598a0 2024-03-19T02:56:00.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.321+0000 7f10969aa740 10 client.16194 fetch_fsmap learned FSMap version 25 2024-03-19T02:56:00.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.321+0000 7f10969aa740 1 -- 172.21.15.33:0/334450525 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x560fdb7dc760 con 0x560fdb7598a0 2024-03-19T02:56:00.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.321+0000 7f10717fa640 1 -- 172.21.15.33:0/334450525 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f107c01d040 con 0x560fdb7598a0 2024-03-19T02:56:00.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.321+0000 7f10717fa640 1 -- 172.21.15.33:0/334450525 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f107c00b070 con 0x560fdb7598a0 2024-03-19T02:56:00.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.321+0000 7f10969aa740 10 client.16194 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:56:00.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.321+0000 7f10969aa740 10 client.16194 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T02:56:00.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.321+0000 7f10969aa740 20 client.16194 populate_metadata read hostname 'smithi033' 2024-03-19T02:56:00.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.321+0000 7f10969aa740 1 --2- 172.21.15.33:0/334450525 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x560fdb7dcdf0 0x560fdb7df1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.321+0000 7f10969aa740 4 client.16194 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:56:00.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.321+0000 7f10969aa740 1 -- 172.21.15.33:0/334450525 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560fdb6c99f0 con 0x560fdb7dcdf0 2024-03-19T02:56:00.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.321+0000 7f1092285640 1 --2- 172.21.15.33:0/334450525 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x560fdb7dcdf0 0x560fdb7df1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.322+0000 7f1092285640 1 --2- 172.21.15.33:0/334450525 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x560fdb7dcdf0 0x560fdb7df1e0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7a7e0711e60fd2b server_cookie=87958c4b3807454c in_seq=0 out_seq=0 2024-03-19T02:56:00.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.322+0000 7f10717fa640 10 client.16194 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T02:56:00.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.322+0000 7f10717fa640 1 -- 172.21.15.33:0/334450525 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560fdb6c99f0 con 0x560fdb7dcdf0 2024-03-19T02:56:00.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.322+0000 7f10717fa640 10 client.16194 handle_command_reply: tid=0 2024-03-19T02:56:00.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.322+0000 7f10969aa740 1 client.16194 shutdown 2024-03-19T02:56:00.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.322+0000 7f10969aa740 1 -- 172.21.15.33:0/334450525 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fdb7598a0 msgr2=0x560fdb76a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:00.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.322+0000 7f10969aa740 1 --2- 172.21.15.33:0/334450525 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fdb7598a0 0x560fdb76a580 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f107c000f80 tx=0x7f107c00e250 comp rx=0 tx=0).stop 2024-03-19T02:56:00.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.322+0000 7f10969aa740 1 -- 172.21.15.33:0/334450525 shutdown_connections 2024-03-19T02:56:00.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.322+0000 7f10969aa740 1 --2- 172.21.15.33:0/334450525 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x560fdb7dcdf0 0x560fdb7df1e0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.322+0000 7f10969aa740 1 --2- 172.21.15.33:0/334450525 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fdb766be0 0x560fdb7db1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.322+0000 7f10969aa740 1 --2- 172.21.15.33:0/334450525 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fdb7598a0 0x560fdb76a580 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.322+0000 7f10969aa740 1 --2- 172.21.15.33:0/334450525 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fdb758f80 0x560fdb76a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.322+0000 7f10717fa640 0 client.16194 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T02:56:00.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.322+0000 7f10969aa740 1 -- 172.21.15.33:0/334450525 >> 172.21.15.33:0/334450525 conn(0x560fdb753a00 msgr2=0x560fdb763ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:00.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.322+0000 7f10969aa740 1 -- 172.21.15.33:0/334450525 shutdown_connections 2024-03-19T02:56:00.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.322+0000 7f10969aa740 1 -- 172.21.15.33:0/334450525 wait complete. 2024-03-19T02:56:00.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.322+0000 7f10969aa740 20 client.16194 trim_cache size 0 max 16384 2024-03-19T02:56:00.330 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:56:00.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.323+0000 7f1093ce8640 1 -- 172.21.15.33:0/106493369 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f10740a2970 msgr2=0x7f10740a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:00.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.323+0000 7f1093ce8640 1 --2- 172.21.15.33:0/106493369 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f10740a2970 0x7f10740a4e30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1080097570 tx=0x7f1080097500 comp rx=0 tx=0).stop 2024-03-19T02:56:00.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.323+0000 7f1093ce8640 1 -- 172.21.15.33:0/106493369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f108c06a7f0 msgr2=0x7f108c1983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:00.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.323+0000 7f1093ce8640 1 --2- 172.21.15.33:0/106493369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f108c06a7f0 0x7f108c1983d0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f108000e9f0 tx=0x7f108000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.323+0000 7f1093ce8640 1 -- 172.21.15.33:0/106493369 shutdown_connections 2024-03-19T02:56:00.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.323+0000 7f1093ce8640 1 --2- 172.21.15.33:0/106493369 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f10740a2970 0x7f10740a4e30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.323+0000 7f1093ce8640 1 --2- 172.21.15.33:0/106493369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f108c107970 0x7f108c19c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.323+0000 7f1093ce8640 1 --2- 172.21.15.33:0/106493369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f108c06a7f0 0x7f108c1983d0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.323+0000 7f1093ce8640 1 --2- 172.21.15.33:0/106493369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f108c069e50 0x7f108c197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.323+0000 7f1093ce8640 1 -- 172.21.15.33:0/106493369 >> 172.21.15.33:0/106493369 conn(0x7f108c0774c0 msgr2=0x7f108c0ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:00.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.323+0000 7f1093ce8640 1 -- 172.21.15.33:0/106493369 shutdown_connections 2024-03-19T02:56:00.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.323+0000 7f1093ce8640 1 -- 172.21.15.33:0/106493369 wait complete. 2024-03-19T02:56:00.335 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:56:00.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.438+0000 7f60df137640 1 -- 172.21.15.33:0/623851422 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60d8106ba0 msgr2=0x7f60d810e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:00.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.438+0000 7f60df137640 1 --2- 172.21.15.33:0/623851422 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60d8106ba0 0x7f60d810e700 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f60d0009cd0 tx=0x7f60d002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.438+0000 7f60df137640 1 -- 172.21.15.33:0/623851422 shutdown_connections 2024-03-19T02:56:00.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.438+0000 7f60df137640 1 --2- 172.21.15.33:0/623851422 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60d8100940 0x7f60d8106470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.438+0000 7f60df137640 1 --2- 172.21.15.33:0/623851422 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60d8106ba0 0x7f60d810e700 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.438+0000 7f60df137640 1 --2- 172.21.15.33:0/623851422 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f60d80fff90 0x7f60d8100370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.438+0000 7f60df137640 1 -- 172.21.15.33:0/623851422 >> 172.21.15.33:0/623851422 conn(0x7f60d8076c50 msgr2=0x7f60d80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:00.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.438+0000 7f60df137640 1 -- 172.21.15.33:0/623851422 shutdown_connections 2024-03-19T02:56:00.442 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.438+0000 7f60df137640 1 -- 172.21.15.33:0/623851422 wait complete. 2024-03-19T02:56:00.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.439+0000 7f60df137640 1 Processor -- start 2024-03-19T02:56:00.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.439+0000 7f60df137640 1 -- start start 2024-03-19T02:56:00.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.439+0000 7f60df137640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f60d80fff90 0x7f60d8197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.439+0000 7f60df137640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60d8106ba0 0x7f60d8198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.439+0000 7f60df137640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60d819c830 0x7f60d819cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.439+0000 7f60df137640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f60d8111a00 con 0x7f60d80fff90 2024-03-19T02:56:00.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.439+0000 7f60df137640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f60d8111880 con 0x7f60d819c830 2024-03-19T02:56:00.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.439+0000 7f60df137640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f60d8111b80 con 0x7f60d8106ba0 2024-03-19T02:56:00.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.439+0000 7f60dced3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f60d80fff90 0x7f60d8197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.439+0000 7f60dced3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f60d80fff90 0x7f60d8197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42114/0 (socket says 172.21.15.33:42114) 2024-03-19T02:56:00.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.439+0000 7f60dced3640 1 -- 172.21.15.33:0/1931168595 learned_addr learned my addr 172.21.15.33:0/1931168595 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:00.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.439+0000 7f60dd6d4640 1 --2- 172.21.15.33:0/1931168595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60d819c830 0x7f60d819cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.439+0000 7f60d7fff640 1 --2- 172.21.15.33:0/1931168595 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60d8106ba0 0x7f60d8198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.439+0000 7f60dced3640 1 -- 172.21.15.33:0/1931168595 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60d8106ba0 msgr2=0x7f60d8198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:00.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.439+0000 7f60dced3640 1 --2- 172.21.15.33:0/1931168595 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60d8106ba0 0x7f60d8198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.439+0000 7f60dced3640 1 -- 172.21.15.33:0/1931168595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60d819c830 msgr2=0x7f60d819cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:00.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.439+0000 7f60dced3640 1 --2- 172.21.15.33:0/1931168595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60d819c830 0x7f60d819cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.440+0000 7f60dced3640 1 -- 172.21.15.33:0/1931168595 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60d819d320 con 0x7f60d80fff90 2024-03-19T02:56:00.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.440+0000 7f60dd6d4640 1 --2- 172.21.15.33:0/1931168595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60d819c830 0x7f60d819cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:00.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.440+0000 7f60dced3640 1 --2- 172.21.15.33:0/1931168595 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f60d80fff90 0x7f60d8197fc0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f60cc00e9f0 tx=0x7f60cc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:00.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.440+0000 7f60d5ffb640 1 -- 172.21.15.33:0/1931168595 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc00cd80 con 0x7f60d80fff90 2024-03-19T02:56:00.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.440+0000 7f60df137640 1 -- 172.21.15.33:0/1931168595 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60d819d610 con 0x7f60d80fff90 2024-03-19T02:56:00.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.440+0000 7f60d5ffb640 1 -- 172.21.15.33:0/1931168595 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60cc0108e0 con 0x7f60d80fff90 2024-03-19T02:56:00.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.440+0000 7f60d5ffb640 1 -- 172.21.15.33:0/1931168595 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc018920 con 0x7f60d80fff90 2024-03-19T02:56:00.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.440+0000 7f60df137640 1 -- 172.21.15.33:0/1931168595 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60d81a4e00 con 0x7f60d80fff90 2024-03-19T02:56:00.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.442+0000 7f60d5ffb640 1 -- 172.21.15.33:0/1931168595 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f60cc010a90 con 0x7f60d80fff90 2024-03-19T02:56:00.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.442+0000 7f60e1df9740 1 Processor -- start 2024-03-19T02:56:00.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.442+0000 7f60e1df9740 1 -- start start 2024-03-19T02:56:00.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.443+0000 7f60e1df9740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c3a6db2d0 0x556c3a6db6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.443+0000 7f60e1df9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c3a6dbbf0 0x556c3a6e4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.443+0000 7f60e1df9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556c3a6e47d0 0x556c3a6e6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.443+0000 7f60e1df9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556c3a63e4b0 con 0x556c3a6dbbf0 2024-03-19T02:56:00.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.443+0000 7f60e1df9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x556c3a5e6d70 con 0x556c3a6e47d0 2024-03-19T02:56:00.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.443+0000 7f60e1df9740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556c3a63a550 con 0x556c3a6db2d0 2024-03-19T02:56:00.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.443+0000 7f60dced3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c3a6dbbf0 0x556c3a6e4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.443+0000 7f60dced3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c3a6dbbf0 0x556c3a6e4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42128/0 (socket says 172.21.15.33:42128) 2024-03-19T02:56:00.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.443+0000 7f60dced3640 1 -- 172.21.15.33:0/1845447184 learned_addr learned my addr 172.21.15.33:0/1845447184 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:00.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.443+0000 7f60d7fff640 1 --2- 172.21.15.33:0/1845447184 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556c3a6e47d0 0x556c3a6e6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.443+0000 7f60d5ffb640 1 --2- 172.21.15.33:0/1931168595 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f60c80a2970 0x7f60c80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.443+0000 7f60dd6d4640 1 --2- 172.21.15.33:0/1845447184 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c3a6db2d0 0x556c3a6db6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.443+0000 7f60dd6d4640 1 --2- 172.21.15.33:0/1931168595 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f60c80a2970 0x7f60c80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.443+0000 7f60d5ffb640 1 -- 172.21.15.33:0/1931168595 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f60cc0cf4d0 con 0x7f60d80fff90 2024-03-19T02:56:00.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.443+0000 7f60dced3640 1 -- 172.21.15.33:0/1845447184 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c3a6db2d0 msgr2=0x556c3a6db6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:00.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.443+0000 7f60dced3640 1 --2- 172.21.15.33:0/1845447184 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c3a6db2d0 0x556c3a6db6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.443+0000 7f60dced3640 1 -- 172.21.15.33:0/1845447184 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556c3a6e47d0 msgr2=0x556c3a6e6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:00.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.443+0000 7f60dced3640 1 --2- 172.21.15.33:0/1845447184 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556c3a6e47d0 0x556c3a6e6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.443+0000 7f60dced3640 1 -- 172.21.15.33:0/1845447184 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c3a609210 con 0x556c3a6dbbf0 2024-03-19T02:56:00.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.443+0000 7f60d7fff640 1 --2- 172.21.15.33:0/1845447184 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556c3a6e47d0 0x556c3a6e6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:00.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.443+0000 7f60dd6d4640 1 --2- 172.21.15.33:0/1845447184 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c3a6db2d0 0x556c3a6db6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:00.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.443+0000 7f60dced3640 1 --2- 172.21.15.33:0/1845447184 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c3a6dbbf0 0x556c3a6e4290 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f60cc09b570 tx=0x7f60cc09b720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=272c1ef7669a4586 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:00.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.443+0000 7f60dd6d4640 1 --2- 172.21.15.33:0/1931168595 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f60c80a2970 0x7f60c80a4e30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f60d0005e30 tx=0x7f60d00095c0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:00.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.444+0000 7f60c2ffd640 1 -- 172.21.15.33:0/1845447184 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc0d2e10 con 0x556c3a6dbbf0 2024-03-19T02:56:00.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.444+0000 7f60c2ffd640 1 -- 172.21.15.33:0/1845447184 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60cc09b870 con 0x556c3a6dbbf0 2024-03-19T02:56:00.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.444+0000 7f60c2ffd640 1 -- 172.21.15.33:0/1845447184 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc0cf170 con 0x556c3a6dbbf0 2024-03-19T02:56:00.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.444+0000 7f60e1df9740 1 -- 172.21.15.33:0/1845447184 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c3a6dbbf0 msgr2=0x556c3a6e4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:00.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.444+0000 7f60e1df9740 1 --2- 172.21.15.33:0/1845447184 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c3a6dbbf0 0x556c3a6e4290 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f60cc09b570 tx=0x7f60cc09b720 comp rx=0 tx=0).stop 2024-03-19T02:56:00.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.444+0000 7f60e1df9740 1 -- 172.21.15.33:0/1845447184 shutdown_connections 2024-03-19T02:56:00.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.444+0000 7f60e1df9740 1 --2- 172.21.15.33:0/1845447184 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556c3a6e47d0 0x556c3a6e6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.444+0000 7f60e1df9740 1 --2- 172.21.15.33:0/1845447184 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c3a6dbbf0 0x556c3a6e4290 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.444+0000 7f60e1df9740 1 --2- 172.21.15.33:0/1845447184 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c3a6db2d0 0x556c3a6db6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.444+0000 7f60e1df9740 1 -- 172.21.15.33:0/1845447184 >> 172.21.15.33:0/1845447184 conn(0x556c3a6d5a00 msgr2=0x556c3a6d5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:00.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.444+0000 7f60e1df9740 1 -- 172.21.15.33:0/1845447184 shutdown_connections 2024-03-19T02:56:00.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.444+0000 7f60e1df9740 1 -- 172.21.15.33:0/1845447184 wait complete. 2024-03-19T02:56:00.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.445+0000 7f60e1df9740 1 -- start start 2024-03-19T02:56:00.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.445+0000 7f60e1df9740 1 Processor -- start 2024-03-19T02:56:00.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.445+0000 7f60e1df9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556c3a6db2d0 0x556c3a6e1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.445+0000 7f60e1df9740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c3a6dbbf0 0x556c3a6e1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.445+0000 7f60e1df9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c3a6e47d0 0x556c3a6e2280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.445+0000 7f60e1df9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556c3a6e7560 con 0x556c3a6e47d0 2024-03-19T02:56:00.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.445+0000 7f60e1df9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x556c3a5fa310 con 0x556c3a6db2d0 2024-03-19T02:56:00.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.445+0000 7f60e1df9740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x556c3a6e76e0 con 0x556c3a6dbbf0 2024-03-19T02:56:00.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.445+0000 7f60e1df9740 10 client.0 _ll_register_callbacks cb 0x556c3a5ee2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:56:00.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.445+0000 7f60dd6d4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c3a6e47d0 0x556c3a6e2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.445+0000 7f60dced3640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556c3a6db2d0 0x556c3a6e1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.445+0000 7f60dd6d4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c3a6e47d0 0x556c3a6e2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42134/0 (socket says 172.21.15.33:42134) 2024-03-19T02:56:00.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.445+0000 7f60dd6d4640 1 -- 172.21.15.33:0/3278358920 learned_addr learned my addr 172.21.15.33:0/3278358920 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:00.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.445+0000 7f60d7fff640 1 --2- 172.21.15.33:0/3278358920 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c3a6dbbf0 0x556c3a6e1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.446+0000 7f60dd6d4640 1 -- 172.21.15.33:0/3278358920 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c3a6dbbf0 msgr2=0x556c3a6e1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:00.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.446+0000 7f60dd6d4640 1 --2- 172.21.15.33:0/3278358920 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c3a6dbbf0 0x556c3a6e1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.446+0000 7f60dd6d4640 1 -- 172.21.15.33:0/3278358920 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556c3a6db2d0 msgr2=0x556c3a6e1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:00.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.446+0000 7f60dd6d4640 1 --2- 172.21.15.33:0/3278358920 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556c3a6db2d0 0x556c3a6e1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.446+0000 7f60dd6d4640 1 -- 172.21.15.33:0/3278358920 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c3a3523f0 con 0x556c3a6e47d0 2024-03-19T02:56:00.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.446+0000 7f60dced3640 1 --2- 172.21.15.33:0/3278358920 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556c3a6db2d0 0x556c3a6e1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:00.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.446+0000 7f60dd6d4640 1 --2- 172.21.15.33:0/3278358920 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c3a6e47d0 0x556c3a6e2280 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f60d0036000 tx=0x7f60d002c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd0c62267bd1e854 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:00.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.446+0000 7f60b7fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:56:00.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.446+0000 7f60b7fff640 1 -- 172.21.15.33:0/3278358920 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d0054060 con 0x556c3a6e47d0 2024-03-19T02:56:00.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.446+0000 7f60b7fff640 1 -- 172.21.15.33:0/3278358920 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f60d00046d0 con 0x556c3a6e47d0 2024-03-19T02:56:00.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.446+0000 7f60e1df9740 1 -- 172.21.15.33:0/3278358920 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60cc09b870 con 0x556c3a6e47d0 2024-03-19T02:56:00.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.446+0000 7f60b7fff640 1 -- 172.21.15.33:0/3278358920 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f60d0039040 con 0x556c3a6e47d0 2024-03-19T02:56:00.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.446+0000 7f60e1df9740 10 client.16206 fetch_fsmap learned FSMap version 25 2024-03-19T02:56:00.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.446+0000 7f60e1df9740 1 -- 172.21.15.33:0/3278358920 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x556c3a41c7d0 con 0x556c3a6e47d0 2024-03-19T02:56:00.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.446+0000 7f60b7fff640 1 -- 172.21.15.33:0/3278358920 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60cc09b870 con 0x556c3a6e47d0 2024-03-19T02:56:00.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.446+0000 7f60b7fff640 1 -- 172.21.15.33:0/3278358920 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f60d0042420 con 0x556c3a6e47d0 2024-03-19T02:56:00.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.447+0000 7f60e1df9740 10 client.16206 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:56:00.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.447+0000 7f60e1df9740 10 client.16206 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T02:56:00.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.447+0000 7f60e1df9740 20 client.16206 populate_metadata read hostname 'smithi033' 2024-03-19T02:56:00.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.447+0000 7f60e1df9740 1 --2- 172.21.15.33:0/3278358920 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x556c3a6f4b10 0x556c3a6f6f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.447+0000 7f60e1df9740 4 client.16206 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:56:00.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.447+0000 7f60e1df9740 1 -- 172.21.15.33:0/3278358920 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556c3a69c3e0 con 0x556c3a6f4b10 2024-03-19T02:56:00.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.447+0000 7f60dced3640 1 --2- 172.21.15.33:0/3278358920 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x556c3a6f4b10 0x556c3a6f6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.447+0000 7f60dced3640 1 --2- 172.21.15.33:0/3278358920 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x556c3a6f4b10 0x556c3a6f6f00 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dd3e8ee9773a3f6d server_cookie=db9f6d481104e094 in_seq=0 out_seq=0 2024-03-19T02:56:00.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.447+0000 7f60b7fff640 10 client.16206 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T02:56:00.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.448+0000 7f60b7fff640 1 -- 172.21.15.33:0/3278358920 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556c3a69c3e0 con 0x556c3a6f4b10 2024-03-19T02:56:00.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.448+0000 7f60b7fff640 10 client.16206 handle_command_reply: tid=0 2024-03-19T02:56:00.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.448+0000 7f60e1df9740 1 client.16206 shutdown 2024-03-19T02:56:00.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.448+0000 7f60e1df9740 1 -- 172.21.15.33:0/3278358920 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c3a6e47d0 msgr2=0x556c3a6e2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:00.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.448+0000 7f60e1df9740 1 --2- 172.21.15.33:0/3278358920 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c3a6e47d0 0x556c3a6e2280 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f60d0036000 tx=0x7f60d002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.448+0000 7f60e1df9740 1 -- 172.21.15.33:0/3278358920 shutdown_connections 2024-03-19T02:56:00.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.448+0000 7f60e1df9740 1 --2- 172.21.15.33:0/3278358920 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x556c3a6f4b10 0x556c3a6f6f00 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.448+0000 7f60e1df9740 1 --2- 172.21.15.33:0/3278358920 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c3a6dbbf0 0x556c3a6e1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.448+0000 7f60e1df9740 1 --2- 172.21.15.33:0/3278358920 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c3a6e47d0 0x556c3a6e2280 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.448+0000 7f60b7fff640 0 client.16206 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T02:56:00.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.448+0000 7f60e1df9740 1 --2- 172.21.15.33:0/3278358920 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556c3a6db2d0 0x556c3a6e1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.448+0000 7f60e1df9740 1 -- 172.21.15.33:0/3278358920 >> 172.21.15.33:0/3278358920 conn(0x556c3a6d5a00 msgr2=0x556c3a6d7750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:00.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.448+0000 7f60e1df9740 1 -- 172.21.15.33:0/3278358920 shutdown_connections 2024-03-19T02:56:00.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.448+0000 7f60e1df9740 1 -- 172.21.15.33:0/3278358920 wait complete. 2024-03-19T02:56:00.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.448+0000 7f60e1df9740 20 client.16206 trim_cache size 0 max 16384 2024-03-19T02:56:00.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.461+0000 7f60e1df9740 1 Processor -- start 2024-03-19T02:56:00.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.461+0000 7f60e1df9740 1 -- start start 2024-03-19T02:56:00.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.462+0000 7f60e1df9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c3a6db2d0 0x556c3a6e3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.462+0000 7f60e1df9740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c3a6dbbf0 0x556c3a6f0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.462+0000 7f60e1df9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556c3a6e47d0 0x556c3a6fc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.462+0000 7f60e1df9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f60d81b1430 con 0x556c3a6db2d0 2024-03-19T02:56:00.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.462+0000 7f60e1df9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f60d81aed00 con 0x556c3a6e47d0 2024-03-19T02:56:00.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.462+0000 7f60e1df9740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f60d81acb30 con 0x556c3a6dbbf0 2024-03-19T02:56:00.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.462+0000 7f60dced3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c3a6db2d0 0x556c3a6e3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.462+0000 7f60dced3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c3a6db2d0 0x556c3a6e3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42138/0 (socket says 172.21.15.33:42138) 2024-03-19T02:56:00.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.462+0000 7f60dced3640 1 -- 172.21.15.33:0/3234863110 learned_addr learned my addr 172.21.15.33:0/3234863110 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:00.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.462+0000 7f60dd6d4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556c3a6e47d0 0x556c3a6fc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.462+0000 7f60d7fff640 1 --2- 172.21.15.33:0/3234863110 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c3a6dbbf0 0x556c3a6f0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.462+0000 7f60dced3640 1 -- 172.21.15.33:0/3234863110 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c3a6dbbf0 msgr2=0x556c3a6f0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:00.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.462+0000 7f60dced3640 1 --2- 172.21.15.33:0/3234863110 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c3a6dbbf0 0x556c3a6f0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.462+0000 7f60dced3640 1 -- 172.21.15.33:0/3234863110 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556c3a6e47d0 msgr2=0x556c3a6fc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:00.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.462+0000 7f60dced3640 1 --2- 172.21.15.33:0/3234863110 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556c3a6e47d0 0x556c3a6fc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.462+0000 7f60dced3640 1 -- 172.21.15.33:0/3234863110 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c3a6ab880 con 0x556c3a6db2d0 2024-03-19T02:56:00.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.462+0000 7f60dced3640 1 --2- 172.21.15.33:0/3234863110 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c3a6db2d0 0x556c3a6e3f30 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f60cc09b7b0 tx=0x7f60cc01a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab659c668a6d385 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:00.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.462+0000 7f60c27fc640 1 -- 172.21.15.33:0/3234863110 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc0b4070 con 0x556c3a6db2d0 2024-03-19T02:56:00.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.462+0000 7f60c27fc640 1 -- 172.21.15.33:0/3234863110 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60cc0cf170 con 0x556c3a6db2d0 2024-03-19T02:56:00.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.462+0000 7f60c27fc640 1 -- 172.21.15.33:0/3234863110 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc0ccd30 con 0x556c3a6db2d0 2024-03-19T02:56:00.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.463+0000 7f60e1df9740 1 -- 172.21.15.33:0/3234863110 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c3a6db2d0 msgr2=0x556c3a6e3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:00.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.463+0000 7f60e1df9740 1 --2- 172.21.15.33:0/3234863110 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c3a6db2d0 0x556c3a6e3f30 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f60cc09b7b0 tx=0x7f60cc01a040 comp rx=0 tx=0).stop 2024-03-19T02:56:00.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.463+0000 7f60e1df9740 1 -- 172.21.15.33:0/3234863110 shutdown_connections 2024-03-19T02:56:00.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.463+0000 7f60e1df9740 1 --2- 172.21.15.33:0/3234863110 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c3a6dbbf0 0x556c3a6f0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.463+0000 7f60e1df9740 1 --2- 172.21.15.33:0/3234863110 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556c3a6e47d0 0x556c3a6fc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.463+0000 7f60e1df9740 1 --2- 172.21.15.33:0/3234863110 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c3a6db2d0 0x556c3a6e3f30 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.463+0000 7f60e1df9740 1 -- 172.21.15.33:0/3234863110 >> 172.21.15.33:0/3234863110 conn(0x556c3a6d5a00 msgr2=0x556c3a6d4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:00.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.463+0000 7f60e1df9740 1 -- 172.21.15.33:0/3234863110 shutdown_connections 2024-03-19T02:56:00.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.463+0000 7f60e1df9740 1 -- 172.21.15.33:0/3234863110 wait complete. 2024-03-19T02:56:00.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.463+0000 7f60e1df9740 1 -- start start 2024-03-19T02:56:00.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.463+0000 7f60e1df9740 1 Processor -- start 2024-03-19T02:56:00.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.464+0000 7f60e1df9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556c3a6db2d0 0x556c3a6e85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.464+0000 7f60e1df9740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c3a6dbbf0 0x556c3a6e8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.464+0000 7f60e1df9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c3a6e47d0 0x556c3a759dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.464+0000 7f60e1df9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f60d81ad1c0 con 0x556c3a6e47d0 2024-03-19T02:56:00.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.464+0000 7f60e1df9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f60d81ae630 con 0x556c3a6db2d0 2024-03-19T02:56:00.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.464+0000 7f60e1df9740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f60d81ad890 con 0x556c3a6dbbf0 2024-03-19T02:56:00.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.464+0000 7f60e1df9740 10 client.0 _ll_register_callbacks cb 0x7f60c4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:56:00.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.464+0000 7f60dd6d4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c3a6e47d0 0x556c3a759dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.464+0000 7f60dced3640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556c3a6db2d0 0x556c3a6e85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.464+0000 7f60dced3640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556c3a6db2d0 0x556c3a6e85f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:37276/0 (socket says 172.21.15.33:37276) 2024-03-19T02:56:00.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.464+0000 7f60dd6d4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c3a6e47d0 0x556c3a759dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42142/0 (socket says 172.21.15.33:42142) 2024-03-19T02:56:00.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.464+0000 7f60dced3640 1 -- 172.21.15.33:0/2526984111 learned_addr learned my addr 172.21.15.33:0/2526984111 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:00.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.464+0000 7f60d7fff640 1 --2- 172.21.15.33:0/2526984111 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c3a6dbbf0 0x556c3a6e8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.464+0000 7f60dd6d4640 1 -- 172.21.15.33:0/2526984111 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c3a6dbbf0 msgr2=0x556c3a6e8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:00.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.464+0000 7f60dd6d4640 1 --2- 172.21.15.33:0/2526984111 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c3a6dbbf0 0x556c3a6e8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.464+0000 7f60dd6d4640 1 -- 172.21.15.33:0/2526984111 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556c3a6db2d0 msgr2=0x556c3a6e85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:00.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.464+0000 7f60dd6d4640 1 --2- 172.21.15.33:0/2526984111 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556c3a6db2d0 0x556c3a6e85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.464+0000 7f60dd6d4640 1 -- 172.21.15.33:0/2526984111 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c3a6daa70 con 0x556c3a6e47d0 2024-03-19T02:56:00.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.464+0000 7f60dced3640 1 --2- 172.21.15.33:0/2526984111 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556c3a6db2d0 0x556c3a6e85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:00.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.464+0000 7f60dd6d4640 1 --2- 172.21.15.33:0/2526984111 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c3a6e47d0 0x556c3a759dd0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f60d0006010 tx=0x7f60d0032a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5677d37ecfde7f01 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:00.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.464+0000 7f60b7fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:56:00.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.464+0000 7f60b7fff640 1 -- 172.21.15.33:0/2526984111 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d0054070 con 0x556c3a6e47d0 2024-03-19T02:56:00.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.464+0000 7f60b7fff640 1 -- 172.21.15.33:0/2526984111 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f60d0004b40 con 0x556c3a6e47d0 2024-03-19T02:56:00.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.464+0000 7f60e1df9740 1 -- 172.21.15.33:0/2526984111 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60cc0cf170 con 0x556c3a6e47d0 2024-03-19T02:56:00.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.465+0000 7f60b7fff640 1 -- 172.21.15.33:0/2526984111 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f60d0004d40 con 0x556c3a6e47d0 2024-03-19T02:56:00.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.465+0000 7f60e1df9740 10 client.16212 fetch_fsmap learned FSMap version 25 2024-03-19T02:56:00.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.465+0000 7f60e1df9740 1 -- 172.21.15.33:0/2526984111 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x556c3a75b270 con 0x556c3a6e47d0 2024-03-19T02:56:00.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.465+0000 7f60b7fff640 1 -- 172.21.15.33:0/2526984111 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60cc0cf170 con 0x556c3a6e47d0 2024-03-19T02:56:00.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.465+0000 7f60b7fff640 1 -- 172.21.15.33:0/2526984111 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f60d005e060 con 0x556c3a6e47d0 2024-03-19T02:56:00.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.465+0000 7f60e1df9740 10 client.16212 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:56:00.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.465+0000 7f60e1df9740 10 client.16212 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T02:56:00.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.465+0000 7f60e1df9740 20 client.16212 populate_metadata read hostname 'smithi033' 2024-03-19T02:56:00.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.465+0000 7f60e1df9740 1 --2- 172.21.15.33:0/2526984111 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x556c3a75b9e0 0x556c3a75de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.465+0000 7f60e1df9740 4 client.16212 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:56:00.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.465+0000 7f60e1df9740 1 -- 172.21.15.33:0/2526984111 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556c3a64b9f0 con 0x556c3a75b9e0 2024-03-19T02:56:00.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.465+0000 7f60dced3640 1 --2- 172.21.15.33:0/2526984111 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x556c3a75b9e0 0x556c3a75de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.466+0000 7f60dced3640 1 --2- 172.21.15.33:0/2526984111 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x556c3a75b9e0 0x556c3a75de00 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=87ec0ff296680a90 server_cookie=ef911ab980a26dfe in_seq=0 out_seq=0 2024-03-19T02:56:00.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.466+0000 7f60b7fff640 10 client.16212 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T02:56:00.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.466+0000 7f60b7fff640 1 -- 172.21.15.33:0/2526984111 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556c3a64b9f0 con 0x556c3a75b9e0 2024-03-19T02:56:00.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.466+0000 7f60b7fff640 10 client.16212 handle_command_reply: tid=0 2024-03-19T02:56:00.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.466+0000 7f60e1df9740 1 client.16212 shutdown 2024-03-19T02:56:00.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.466+0000 7f60e1df9740 1 -- 172.21.15.33:0/2526984111 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c3a6e47d0 msgr2=0x556c3a759dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:00.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.466+0000 7f60e1df9740 1 --2- 172.21.15.33:0/2526984111 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c3a6e47d0 0x556c3a759dd0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f60d0006010 tx=0x7f60d0032a00 comp rx=0 tx=0).stop 2024-03-19T02:56:00.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.466+0000 7f60e1df9740 1 -- 172.21.15.33:0/2526984111 shutdown_connections 2024-03-19T02:56:00.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.466+0000 7f60e1df9740 1 --2- 172.21.15.33:0/2526984111 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x556c3a75b9e0 0x556c3a75de00 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.466+0000 7f60e1df9740 1 --2- 172.21.15.33:0/2526984111 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x556c3a6dbbf0 0x556c3a6e8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.466+0000 7f60e1df9740 1 --2- 172.21.15.33:0/2526984111 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x556c3a6e47d0 0x556c3a759dd0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.466+0000 7f60b7fff640 0 client.16212 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T02:56:00.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.466+0000 7f60e1df9740 1 --2- 172.21.15.33:0/2526984111 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556c3a6db2d0 0x556c3a6e85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.466+0000 7f60e1df9740 1 -- 172.21.15.33:0/2526984111 >> 172.21.15.33:0/2526984111 conn(0x556c3a6d5a00 msgr2=0x556c3a6e2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:00.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.466+0000 7f60e1df9740 1 -- 172.21.15.33:0/2526984111 shutdown_connections 2024-03-19T02:56:00.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.466+0000 7f60e1df9740 1 -- 172.21.15.33:0/2526984111 wait complete. 2024-03-19T02:56:00.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.466+0000 7f60e1df9740 20 client.16212 trim_cache size 0 max 16384 2024-03-19T02:56:00.474 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:56:00.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.467+0000 7f60df137640 1 -- 172.21.15.33:0/1931168595 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f60c80a2970 msgr2=0x7f60c80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:00.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.467+0000 7f60df137640 1 --2- 172.21.15.33:0/1931168595 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f60c80a2970 0x7f60c80a4e30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f60d0005e30 tx=0x7f60d00095c0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.467+0000 7f60df137640 1 -- 172.21.15.33:0/1931168595 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f60d80fff90 msgr2=0x7f60d8197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:00.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.467+0000 7f60df137640 1 --2- 172.21.15.33:0/1931168595 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f60d80fff90 0x7f60d8197fc0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f60cc00e9f0 tx=0x7f60cc00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.467+0000 7f60df137640 1 -- 172.21.15.33:0/1931168595 shutdown_connections 2024-03-19T02:56:00.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.467+0000 7f60df137640 1 --2- 172.21.15.33:0/1931168595 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f60c80a2970 0x7f60c80a4e30 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.467+0000 7f60df137640 1 --2- 172.21.15.33:0/1931168595 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f60d8106ba0 0x7f60d8198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.467+0000 7f60df137640 1 --2- 172.21.15.33:0/1931168595 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60d819c830 0x7f60d819cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.467+0000 7f60df137640 1 --2- 172.21.15.33:0/1931168595 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f60d80fff90 0x7f60d8197fc0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.467+0000 7f60df137640 1 -- 172.21.15.33:0/1931168595 >> 172.21.15.33:0/1931168595 conn(0x7f60d8076c50 msgr2=0x7f60d80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:00.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.467+0000 7f60df137640 1 -- 172.21.15.33:0/1931168595 shutdown_connections 2024-03-19T02:56:00.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.467+0000 7f60df137640 1 -- 172.21.15.33:0/1931168595 wait complete. 2024-03-19T02:56:00.479 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:56:00.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.582+0000 7efecb569640 1 -- 172.21.15.33:0/1856202205 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efec41066b0 msgr2=0x7efec4106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:00.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.582+0000 7efecb569640 1 --2- 172.21.15.33:0/1856202205 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efec41066b0 0x7efec4106a90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7efec0009900 tx=0x7efec002bc30 comp rx=0 tx=0).stop 2024-03-19T02:56:00.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.583+0000 7efecb569640 1 -- 172.21.15.33:0/1856202205 shutdown_connections 2024-03-19T02:56:00.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.583+0000 7efecb569640 1 --2- 172.21.15.33:0/1856202205 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efec41006a0 0x7efec4100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.583+0000 7efecb569640 1 --2- 172.21.15.33:0/1856202205 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efec4101060 0x7efec410d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.583+0000 7efecb569640 1 --2- 172.21.15.33:0/1856202205 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efec41066b0 0x7efec4106a90 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.583+0000 7efecb569640 1 -- 172.21.15.33:0/1856202205 >> 172.21.15.33:0/1856202205 conn(0x7efec40778d0 msgr2=0x7efec4077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:00.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.583+0000 7efecb569640 1 -- 172.21.15.33:0/1856202205 shutdown_connections 2024-03-19T02:56:00.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.583+0000 7efecb569640 1 -- 172.21.15.33:0/1856202205 wait complete. 2024-03-19T02:56:00.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.583+0000 7efecb569640 1 Processor -- start 2024-03-19T02:56:00.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.583+0000 7efecb569640 1 -- start start 2024-03-19T02:56:00.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.583+0000 7efecb569640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efec41006a0 0x7efec4196e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.583+0000 7efecb569640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efec4101060 0x7efec41973d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.583+0000 7efecb569640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efec41066b0 0x7efec419b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.583+0000 7efecb569640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efec4110930 con 0x7efec41006a0 2024-03-19T02:56:00.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.583+0000 7efecb569640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efec41107b0 con 0x7efec41066b0 2024-03-19T02:56:00.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.583+0000 7efecb569640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7efec4110ab0 con 0x7efec4101060 2024-03-19T02:56:00.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.584+0000 7efec9305640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efec41006a0 0x7efec4196e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.584+0000 7efec9305640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efec41006a0 0x7efec4196e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42156/0 (socket says 172.21.15.33:42156) 2024-03-19T02:56:00.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.584+0000 7efec9305640 1 -- 172.21.15.33:0/1887463891 learned_addr learned my addr 172.21.15.33:0/1887463891 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:00.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.584+0000 7efec8b04640 1 --2- 172.21.15.33:0/1887463891 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efec4101060 0x7efec41973d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.584+0000 7efec9b06640 1 --2- 172.21.15.33:0/1887463891 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efec41066b0 0x7efec419b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.584+0000 7efec9305640 1 -- 172.21.15.33:0/1887463891 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efec4101060 msgr2=0x7efec41973d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:00.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.584+0000 7efec9305640 1 --2- 172.21.15.33:0/1887463891 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efec4101060 0x7efec41973d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.584+0000 7efec9305640 1 -- 172.21.15.33:0/1887463891 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efec41066b0 msgr2=0x7efec419b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:00.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.584+0000 7efec9305640 1 --2- 172.21.15.33:0/1887463891 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efec41066b0 0x7efec419b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.584+0000 7efec9305640 1 -- 172.21.15.33:0/1887463891 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efec419be30 con 0x7efec41006a0 2024-03-19T02:56:00.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.584+0000 7efec9305640 1 --2- 172.21.15.33:0/1887463891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efec41006a0 0x7efec4196e90 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7efec0006010 tx=0x7efec0005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:00.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.584+0000 7efeba7fc640 1 -- 172.21.15.33:0/1887463891 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec0004e00 con 0x7efec41006a0 2024-03-19T02:56:00.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.584+0000 7efecb569640 1 -- 172.21.15.33:0/1887463891 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efec419c0c0 con 0x7efec41006a0 2024-03-19T02:56:00.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.584+0000 7efeba7fc640 1 -- 172.21.15.33:0/1887463891 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efec0005450 con 0x7efec41006a0 2024-03-19T02:56:00.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.584+0000 7efecb569640 1 -- 172.21.15.33:0/1887463891 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efec4068980 con 0x7efec41006a0 2024-03-19T02:56:00.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.584+0000 7efeba7fc640 1 -- 172.21.15.33:0/1887463891 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec0003df0 con 0x7efec41006a0 2024-03-19T02:56:00.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.587+0000 7efece232740 1 Processor -- start 2024-03-19T02:56:00.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.587+0000 7efece232740 1 -- start start 2024-03-19T02:56:00.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.588+0000 7efeba7fc640 1 -- 172.21.15.33:0/1887463891 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7efec00035f0 con 0x7efec41006a0 2024-03-19T02:56:00.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.588+0000 7efece232740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573e98c82d0 0x5573e98c86b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.588+0000 7efece232740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573e98c8bf0 0x5573e98d1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.588+0000 7efece232740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5573e98d17d0 0x5573e98d3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.588+0000 7efece232740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5573e982b4b0 con 0x5573e98c8bf0 2024-03-19T02:56:00.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.588+0000 7efece232740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5573e97d3d70 con 0x5573e98c82d0 2024-03-19T02:56:00.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.588+0000 7efece232740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5573e9827550 con 0x5573e98d17d0 2024-03-19T02:56:00.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.588+0000 7efec9b06640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573e98c82d0 0x5573e98c86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.588+0000 7efec9305640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573e98c8bf0 0x5573e98d1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.588+0000 7efec9b06640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573e98c82d0 0x5573e98c86b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:37304/0 (socket says 172.21.15.33:37304) 2024-03-19T02:56:00.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.588+0000 7efec9b06640 1 -- 172.21.15.33:0/3931320359 learned_addr learned my addr 172.21.15.33:0/3931320359 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:00.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.588+0000 7efec8b04640 1 --2- 172.21.15.33:0/3931320359 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5573e98d17d0 0x5573e98d3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.588+0000 7efec9b06640 1 -- 172.21.15.33:0/3931320359 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5573e98d17d0 msgr2=0x5573e98d3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:00.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.588+0000 7efec9b06640 1 --2- 172.21.15.33:0/3931320359 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5573e98d17d0 0x5573e98d3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.588+0000 7efec9b06640 1 -- 172.21.15.33:0/3931320359 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573e98c8bf0 msgr2=0x5573e98d1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:00.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.588+0000 7efec9b06640 1 --2- 172.21.15.33:0/3931320359 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573e98c8bf0 0x5573e98d1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.588+0000 7efec9b06640 1 -- 172.21.15.33:0/3931320359 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573e97f6210 con 0x5573e98c82d0 2024-03-19T02:56:00.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.588+0000 7efeba7fc640 1 --2- 172.21.15.33:0/1887463891 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7efeb00a2970 0x7efeb00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.589+0000 7efec9305640 1 --2- 172.21.15.33:0/1887463891 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7efeb00a2970 0x7efeb00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.589+0000 7efec9b06640 1 --2- 172.21.15.33:0/3931320359 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573e98c82d0 0x5573e98c86b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7efebc009f50 tx=0x7efebc0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7ad341839bd68c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:00.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.589+0000 7efeba7fc640 1 -- 172.21.15.33:0/1887463891 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7efec00f3090 con 0x7efec41006a0 2024-03-19T02:56:00.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.589+0000 7efeaf7fe640 1 -- 172.21.15.33:0/3931320359 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efebc004d20 con 0x5573e98c82d0 2024-03-19T02:56:00.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.589+0000 7efeaf7fe640 1 -- 172.21.15.33:0/3931320359 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efebc004050 con 0x5573e98c82d0 2024-03-19T02:56:00.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.589+0000 7efeaf7fe640 1 -- 172.21.15.33:0/3931320359 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efebc005280 con 0x5573e98c82d0 2024-03-19T02:56:00.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.590+0000 7efece232740 1 -- 172.21.15.33:0/3931320359 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573e98c82d0 msgr2=0x5573e98c86b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:00.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.590+0000 7efece232740 1 --2- 172.21.15.33:0/3931320359 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573e98c82d0 0x5573e98c86b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7efebc009f50 tx=0x7efebc0118d0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.590+0000 7efec9305640 1 --2- 172.21.15.33:0/1887463891 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7efeb00a2970 0x7efeb00a4e30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x5573e98c7da0 tx=0x7efec00ddf90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:00.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.590+0000 7efece232740 1 -- 172.21.15.33:0/3931320359 shutdown_connections 2024-03-19T02:56:00.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.590+0000 7efece232740 1 --2- 172.21.15.33:0/3931320359 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5573e98d17d0 0x5573e98d3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.590+0000 7efece232740 1 --2- 172.21.15.33:0/3931320359 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573e98c8bf0 0x5573e98d1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.590+0000 7efece232740 1 --2- 172.21.15.33:0/3931320359 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573e98c82d0 0x5573e98c86b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.590+0000 7efece232740 1 -- 172.21.15.33:0/3931320359 >> 172.21.15.33:0/3931320359 conn(0x5573e98c2a00 msgr2=0x5573e98c2de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:00.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.590+0000 7efece232740 1 -- 172.21.15.33:0/3931320359 shutdown_connections 2024-03-19T02:56:00.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.590+0000 7efece232740 1 -- 172.21.15.33:0/3931320359 wait complete. 2024-03-19T02:56:00.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.590+0000 7efece232740 1 -- start start 2024-03-19T02:56:00.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.590+0000 7efece232740 1 Processor -- start 2024-03-19T02:56:00.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.591+0000 7efece232740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573e98c82d0 0x5573e98ce860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.591+0000 7efece232740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5573e98c8bf0 0x5573e98ceda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.591+0000 7efece232740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573e98d17d0 0x5573e98cf2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.591+0000 7efece232740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5573e98d89d0 con 0x5573e98c82d0 2024-03-19T02:56:00.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.591+0000 7efece232740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5573e97e7310 con 0x5573e98d17d0 2024-03-19T02:56:00.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.591+0000 7efece232740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5573e98d8b50 con 0x5573e98c8bf0 2024-03-19T02:56:00.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.591+0000 7efece232740 10 client.0 _ll_register_callbacks cb 0x5573e97db2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:56:00.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.591+0000 7efec9b06640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573e98c82d0 0x5573e98ce860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.591+0000 7efec8b04640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5573e98c8bf0 0x5573e98ceda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.591+0000 7efec9b06640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573e98c82d0 0x5573e98ce860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42168/0 (socket says 172.21.15.33:42168) 2024-03-19T02:56:00.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.591+0000 7efec9b06640 1 -- 172.21.15.33:0/2678497802 learned_addr learned my addr 172.21.15.33:0/2678497802 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:00.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.591+0000 7efec9b06640 1 --2- 172.21.15.33:0/2678497802 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573e98d17d0 0x5573e98cf2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.591+0000 7efec9b06640 1 -- 172.21.15.33:0/2678497802 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5573e98c8bf0 msgr2=0x5573e98ceda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:00.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.591+0000 7efec9b06640 1 --2- 172.21.15.33:0/2678497802 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5573e98c8bf0 0x5573e98ceda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.591+0000 7efec9b06640 1 -- 172.21.15.33:0/2678497802 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573e98d17d0 msgr2=0x5573e98cf2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:00.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.591+0000 7efec9b06640 1 --2- 172.21.15.33:0/2678497802 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573e98d17d0 0x5573e98cf2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.591+0000 7efec9b06640 1 -- 172.21.15.33:0/2678497802 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573e953f3f0 con 0x5573e98c82d0 2024-03-19T02:56:00.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.591+0000 7efec8b04640 1 --2- 172.21.15.33:0/2678497802 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5573e98c8bf0 0x5573e98ceda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:00.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.591+0000 7efec9b06640 1 --2- 172.21.15.33:0/2678497802 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573e98c82d0 0x5573e98ce860 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7efebc005c40 tx=0x7efebc005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd58c8c52a6aef99 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:00.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.591+0000 7efeacff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:56:00.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.592+0000 7efeacff9640 1 -- 172.21.15.33:0/2678497802 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efebc004500 con 0x5573e98c82d0 2024-03-19T02:56:00.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.592+0000 7efeacff9640 1 -- 172.21.15.33:0/2678497802 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7efebc004c10 con 0x5573e98c82d0 2024-03-19T02:56:00.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.592+0000 7efece232740 1 -- 172.21.15.33:0/2678497802 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efebc004050 con 0x5573e98c82d0 2024-03-19T02:56:00.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.592+0000 7efeacff9640 1 -- 172.21.15.33:0/2678497802 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7efebc034050 con 0x5573e98c82d0 2024-03-19T02:56:00.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.592+0000 7efece232740 10 client.16224 fetch_fsmap learned FSMap version 25 2024-03-19T02:56:00.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.592+0000 7efece232740 1 -- 172.21.15.33:0/2678497802 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5573e96097d0 con 0x5573e98c82d0 2024-03-19T02:56:00.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.592+0000 7efeacff9640 1 -- 172.21.15.33:0/2678497802 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efebc004050 con 0x5573e98c82d0 2024-03-19T02:56:00.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.592+0000 7efeacff9640 1 -- 172.21.15.33:0/2678497802 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7efebc028450 con 0x5573e98c82d0 2024-03-19T02:56:00.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.592+0000 7efece232740 10 client.16224 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:56:00.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.592+0000 7efece232740 10 client.16224 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:56:00.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.592+0000 7efece232740 20 client.16224 populate_metadata read hostname 'smithi033' 2024-03-19T02:56:00.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.592+0000 7efece232740 1 --2- 172.21.15.33:0/2678497802 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5573e98e5f00 0x5573e98e82f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.592+0000 7efece232740 4 client.16224 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:56:00.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.592+0000 7efece232740 1 -- 172.21.15.33:0/2678497802 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5573e98893e0 con 0x5573e98e5f00 2024-03-19T02:56:00.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.593+0000 7efec8b04640 1 --2- 172.21.15.33:0/2678497802 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5573e98e5f00 0x5573e98e82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.593+0000 7efec8b04640 1 --2- 172.21.15.33:0/2678497802 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5573e98e5f00 0x5573e98e82f0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49c5d77c1076101d server_cookie=63940d6b0bab5051 in_seq=0 out_seq=0 2024-03-19T02:56:00.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.593+0000 7efeacff9640 10 client.16224 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:56:00.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.594+0000 7efeacff9640 1 -- 172.21.15.33:0/2678497802 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5573e98893e0 con 0x5573e98e5f00 2024-03-19T02:56:00.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.594+0000 7efeacff9640 10 client.16224 handle_command_reply: tid=0 2024-03-19T02:56:00.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.594+0000 7efece232740 1 client.16224 shutdown 2024-03-19T02:56:00.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.594+0000 7efece232740 1 -- 172.21.15.33:0/2678497802 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573e98c82d0 msgr2=0x5573e98ce860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:00.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.594+0000 7efece232740 1 --2- 172.21.15.33:0/2678497802 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573e98c82d0 0x5573e98ce860 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7efebc005c40 tx=0x7efebc005bd0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.594+0000 7efece232740 1 -- 172.21.15.33:0/2678497802 shutdown_connections 2024-03-19T02:56:00.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.594+0000 7efece232740 1 --2- 172.21.15.33:0/2678497802 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5573e98e5f00 0x5573e98e82f0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.594+0000 7efece232740 1 --2- 172.21.15.33:0/2678497802 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5573e98c8bf0 0x5573e98ceda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.594+0000 7efece232740 1 --2- 172.21.15.33:0/2678497802 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573e98d17d0 0x5573e98cf2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.594+0000 7efece232740 1 --2- 172.21.15.33:0/2678497802 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573e98c82d0 0x5573e98ce860 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.594+0000 7efece232740 1 -- 172.21.15.33:0/2678497802 >> 172.21.15.33:0/2678497802 conn(0x5573e98c2a00 msgr2=0x5573e98c4750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:00.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.594+0000 7efeacff9640 0 client.16224 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:56:00.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.594+0000 7efece232740 1 -- 172.21.15.33:0/2678497802 shutdown_connections 2024-03-19T02:56:00.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.594+0000 7efece232740 1 -- 172.21.15.33:0/2678497802 wait complete. 2024-03-19T02:56:00.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.594+0000 7efece232740 20 client.16224 trim_cache size 0 max 16384 2024-03-19T02:56:00.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.610+0000 7efece232740 1 Processor -- start 2024-03-19T02:56:00.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.610+0000 7efece232740 1 -- start start 2024-03-19T02:56:00.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.611+0000 7efece232740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5573e98c82d0 0x5573e98d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.611+0000 7efece232740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573e98c8bf0 0x5573e98e1770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.611+0000 7efece232740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573e98d17d0 0x5573e98cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.611+0000 7efece232740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efec41af1e0 con 0x5573e98c8bf0 2024-03-19T02:56:00.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.611+0000 7efece232740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efec41acab0 con 0x5573e98d17d0 2024-03-19T02:56:00.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.611+0000 7efece232740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7efec41aa8e0 con 0x5573e98c82d0 2024-03-19T02:56:00.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.611+0000 7efec8b04640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573e98c8bf0 0x5573e98e1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.611+0000 7efec9b06640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5573e98c82d0 0x5573e98d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.611+0000 7efec9b06640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5573e98c82d0 0x5573e98d0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:52390/0 (socket says 172.21.15.33:52390) 2024-03-19T02:56:00.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.611+0000 7efec9b06640 1 -- 172.21.15.33:0/2887975700 learned_addr learned my addr 172.21.15.33:0/2887975700 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:00.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.611+0000 7efec9b06640 1 --2- 172.21.15.33:0/2887975700 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573e98d17d0 0x5573e98cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.611+0000 7efec9b06640 1 -- 172.21.15.33:0/2887975700 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573e98d17d0 msgr2=0x5573e98cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:00.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.611+0000 7efec9b06640 1 --2- 172.21.15.33:0/2887975700 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573e98d17d0 0x5573e98cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.611+0000 7efec9b06640 1 -- 172.21.15.33:0/2887975700 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573e98c8bf0 msgr2=0x5573e98e1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:00.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.611+0000 7efec9b06640 1 --2- 172.21.15.33:0/2887975700 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573e98c8bf0 0x5573e98e1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.611+0000 7efec9b06640 1 -- 172.21.15.33:0/2887975700 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573e98c35d0 con 0x5573e98c82d0 2024-03-19T02:56:00.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.612+0000 7efec9b06640 1 --2- 172.21.15.33:0/2887975700 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5573e98c82d0 0x5573e98d0f90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efebc009560 tx=0x7efebc0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=218a76ae0cc1fe00 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:00.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.612+0000 7efeaeffd640 1 -- 172.21.15.33:0/2887975700 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efebc003970 con 0x5573e98c82d0 2024-03-19T02:56:00.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.612+0000 7efeaeffd640 1 -- 172.21.15.33:0/2887975700 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efebc028b70 con 0x5573e98c82d0 2024-03-19T02:56:00.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.612+0000 7efeaeffd640 1 -- 172.21.15.33:0/2887975700 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efebc004130 con 0x5573e98c82d0 2024-03-19T02:56:00.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.613+0000 7efece232740 1 -- 172.21.15.33:0/2887975700 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5573e98c82d0 msgr2=0x5573e98d0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:00.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.613+0000 7efece232740 1 --2- 172.21.15.33:0/2887975700 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5573e98c82d0 0x5573e98d0f90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efebc009560 tx=0x7efebc0118d0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.613+0000 7efece232740 1 -- 172.21.15.33:0/2887975700 shutdown_connections 2024-03-19T02:56:00.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.613+0000 7efece232740 1 --2- 172.21.15.33:0/2887975700 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573e98d17d0 0x5573e98cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.613+0000 7efece232740 1 --2- 172.21.15.33:0/2887975700 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573e98c8bf0 0x5573e98e1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.613+0000 7efece232740 1 --2- 172.21.15.33:0/2887975700 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5573e98c82d0 0x5573e98d0f90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.613+0000 7efece232740 1 -- 172.21.15.33:0/2887975700 >> 172.21.15.33:0/2887975700 conn(0x5573e98c2a00 msgr2=0x5573e98cd130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:00.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.613+0000 7efece232740 1 -- 172.21.15.33:0/2887975700 shutdown_connections 2024-03-19T02:56:00.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.613+0000 7efece232740 1 -- 172.21.15.33:0/2887975700 wait complete. 2024-03-19T02:56:00.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.613+0000 7efece232740 1 -- start start 2024-03-19T02:56:00.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.613+0000 7efece232740 1 Processor -- start 2024-03-19T02:56:00.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.614+0000 7efece232740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5573e98c82d0 0x5573e98d8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.614+0000 7efece232740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573e98c8bf0 0x5573e98d92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.614+0000 7efece232740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573e98d17d0 0x5573e98d97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.614+0000 7efece232740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efec41aaf70 con 0x5573e98c8bf0 2024-03-19T02:56:00.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.614+0000 7efece232740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efec41ac3e0 con 0x5573e98d17d0 2024-03-19T02:56:00.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.614+0000 7efece232740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7efec41ab640 con 0x5573e98c82d0 2024-03-19T02:56:00.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.614+0000 7efece232740 10 client.0 _ll_register_callbacks cb 0x7efec00f4840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:56:00.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.614+0000 7efec8b04640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573e98c8bf0 0x5573e98d92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.614+0000 7efec9b06640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5573e98c82d0 0x5573e98d8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.614+0000 7efec9b06640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5573e98c82d0 0x5573e98d8d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:52398/0 (socket says 172.21.15.33:52398) 2024-03-19T02:56:00.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.614+0000 7efec8b04640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573e98c8bf0 0x5573e98d92a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42190/0 (socket says 172.21.15.33:42190) 2024-03-19T02:56:00.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.614+0000 7efec9b06640 1 -- 172.21.15.33:0/1726950264 learned_addr learned my addr 172.21.15.33:0/1726950264 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:00.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.614+0000 7efec9b06640 1 --2- 172.21.15.33:0/1726950264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573e98d17d0 0x5573e98d97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.614+0000 7efec9b06640 1 -- 172.21.15.33:0/1726950264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573e98d17d0 msgr2=0x5573e98d97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:00.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.614+0000 7efec9b06640 1 --2- 172.21.15.33:0/1726950264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573e98d17d0 0x5573e98d97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.614+0000 7efec9b06640 1 -- 172.21.15.33:0/1726950264 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573e98c8bf0 msgr2=0x5573e98d92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:00.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.614+0000 7efec9b06640 1 --2- 172.21.15.33:0/1726950264 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573e98c8bf0 0x5573e98d92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.614+0000 7efec9b06640 1 -- 172.21.15.33:0/1726950264 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573e994a190 con 0x5573e98c82d0 2024-03-19T02:56:00.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.614+0000 7efec8b04640 1 --2- 172.21.15.33:0/1726950264 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573e98c8bf0 0x5573e98d92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:00.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.614+0000 7efec9b06640 1 --2- 172.21.15.33:0/1726950264 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5573e98c82d0 0x5573e98d8d60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efebc028970 tx=0x7efebc0285f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbecf1c292387feb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:00.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.614+0000 7efeacff9640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-19T02:56:00.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.614+0000 7efeacff9640 1 -- 172.21.15.33:0/1726950264 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efebc03d070 con 0x5573e98c82d0 2024-03-19T02:56:00.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.614+0000 7efeacff9640 1 -- 172.21.15.33:0/1726950264 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7efebc009d40 con 0x5573e98c82d0 2024-03-19T02:56:00.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.614+0000 7efece232740 1 -- 172.21.15.33:0/1726950264 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efebc028b70 con 0x5573e98c82d0 2024-03-19T02:56:00.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.615+0000 7efeacff9640 1 -- 172.21.15.33:0/1726950264 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7efebc004100 con 0x5573e98c82d0 2024-03-19T02:56:00.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.615+0000 7efece232740 10 client.25921 fetch_fsmap learned FSMap version 25 2024-03-19T02:56:00.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.615+0000 7efece232740 1 -- 172.21.15.33:0/1726950264 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5573e994b490 con 0x5573e98c82d0 2024-03-19T02:56:00.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.615+0000 7efeacff9640 1 -- 172.21.15.33:0/1726950264 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efebc028b70 con 0x5573e98c82d0 2024-03-19T02:56:00.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.615+0000 7efeacff9640 1 -- 172.21.15.33:0/1726950264 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7efebc04f070 con 0x5573e98c82d0 2024-03-19T02:56:00.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.615+0000 7efece232740 10 client.25921 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:56:00.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.615+0000 7efece232740 10 client.25921 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:56:00.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.615+0000 7efece232740 20 client.25921 populate_metadata read hostname 'smithi033' 2024-03-19T02:56:00.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.615+0000 7efece232740 1 --2- 172.21.15.33:0/1726950264 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5573e994ba40 0x5573e994de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:00.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.615+0000 7efece232740 4 client.25921 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:56:00.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.615+0000 7efece232740 1 -- 172.21.15.33:0/1726950264 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5573e98389f0 con 0x5573e994ba40 2024-03-19T02:56:00.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.615+0000 7efec8b04640 1 --2- 172.21.15.33:0/1726950264 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5573e994ba40 0x5573e994de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:00.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.616+0000 7efeacff9640 10 client.25921 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:56:00.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.616+0000 7efec8b04640 1 --2- 172.21.15.33:0/1726950264 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5573e994ba40 0x5573e994de30 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f40e9d9006de081a server_cookie=d60962002be001af in_seq=0 out_seq=0 2024-03-19T02:56:00.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.616+0000 7efeacff9640 1 -- 172.21.15.33:0/1726950264 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5573e98389f0 con 0x5573e994ba40 2024-03-19T02:56:00.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.616+0000 7efeacff9640 10 client.25921 handle_command_reply: tid=0 2024-03-19T02:56:00.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.616+0000 7efece232740 1 client.25921 shutdown 2024-03-19T02:56:00.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.616+0000 7efece232740 1 -- 172.21.15.33:0/1726950264 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5573e98c82d0 msgr2=0x5573e98d8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:00.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.616+0000 7efece232740 1 --2- 172.21.15.33:0/1726950264 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5573e98c82d0 0x5573e98d8d60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efebc028970 tx=0x7efebc0285f0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.616+0000 7efece232740 1 -- 172.21.15.33:0/1726950264 shutdown_connections 2024-03-19T02:56:00.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.616+0000 7efece232740 1 --2- 172.21.15.33:0/1726950264 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5573e994ba40 0x5573e994de30 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.616+0000 7efece232740 1 --2- 172.21.15.33:0/1726950264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5573e98d17d0 0x5573e98d97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.616+0000 7efece232740 1 --2- 172.21.15.33:0/1726950264 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5573e98c8bf0 0x5573e98d92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.616+0000 7efeacff9640 0 client.25921 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:56:00.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.616+0000 7efece232740 1 --2- 172.21.15.33:0/1726950264 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5573e98c82d0 0x5573e98d8d60 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.616+0000 7efece232740 1 -- 172.21.15.33:0/1726950264 >> 172.21.15.33:0/1726950264 conn(0x5573e98c2a00 msgr2=0x5573e98ce820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:00.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.616+0000 7efece232740 1 -- 172.21.15.33:0/1726950264 shutdown_connections 2024-03-19T02:56:00.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.616+0000 7efece232740 1 -- 172.21.15.33:0/1726950264 wait complete. 2024-03-19T02:56:00.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.617+0000 7efece232740 20 client.25921 trim_cache size 0 max 16384 2024-03-19T02:56:00.625 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:56:00.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.617+0000 7efecb569640 1 -- 172.21.15.33:0/1887463891 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7efeb00a2970 msgr2=0x7efeb00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:00.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.617+0000 7efecb569640 1 --2- 172.21.15.33:0/1887463891 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7efeb00a2970 0x7efeb00a4e30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x5573e98c7da0 tx=0x7efec00ddf90 comp rx=0 tx=0).stop 2024-03-19T02:56:00.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.617+0000 7efecb569640 1 -- 172.21.15.33:0/1887463891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efec41006a0 msgr2=0x7efec4196e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:00.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.617+0000 7efecb569640 1 --2- 172.21.15.33:0/1887463891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efec41006a0 0x7efec4196e90 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7efec0006010 tx=0x7efec0005d90 comp rx=0 tx=0).stop 2024-03-19T02:56:00.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.617+0000 7efecb569640 1 -- 172.21.15.33:0/1887463891 shutdown_connections 2024-03-19T02:56:00.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.617+0000 7efecb569640 1 --2- 172.21.15.33:0/1887463891 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7efeb00a2970 0x7efeb00a4e30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.617+0000 7efecb569640 1 --2- 172.21.15.33:0/1887463891 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efec4101060 0x7efec41973d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.617+0000 7efecb569640 1 --2- 172.21.15.33:0/1887463891 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efec41066b0 0x7efec419b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.617+0000 7efecb569640 1 --2- 172.21.15.33:0/1887463891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efec41006a0 0x7efec4196e90 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:00.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.617+0000 7efecb569640 1 -- 172.21.15.33:0/1887463891 >> 172.21.15.33:0/1887463891 conn(0x7efec40778d0 msgr2=0x7efec4103000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:00.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.617+0000 7efecb569640 1 -- 172.21.15.33:0/1887463891 shutdown_connections 2024-03-19T02:56:00.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:00.617+0000 7efecb569640 1 -- 172.21.15.33:0/1887463891 wait complete. 2024-03-19T02:56:01.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:00 smithi096 ceph-mon[35946]: pgmap v79: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 399 MiB/s wr, 107 op/s 2024-03-19T02:56:01.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:00 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3516465662' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:56:01.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:00 smithi155 ceph-mon[35804]: pgmap v79: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 399 MiB/s wr, 107 op/s 2024-03-19T02:56:01.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:00 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3516465662' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:56:01.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:00 smithi033 ceph-mon[31973]: pgmap v79: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 399 MiB/s wr, 107 op/s 2024-03-19T02:56:01.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:00 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3516465662' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:56:01.631 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:56:01.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:56:01.632 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:56:01.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.753+0000 7f3a475f3640 1 -- 172.21.15.33:0/501464673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a401017a0 msgr2=0x7f3a40101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:01.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.753+0000 7f3a475f3640 1 --2- 172.21.15.33:0/501464673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a401017a0 0x7f3a40101c20 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c009930 tx=0x7f3a3c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:56:01.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.753+0000 7f3a475f3640 1 -- 172.21.15.33:0/501464673 shutdown_connections 2024-03-19T02:56:01.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.753+0000 7f3a475f3640 1 --2- 172.21.15.33:0/501464673 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a40102160 0x7f3a4010e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.753+0000 7f3a475f3640 1 --2- 172.21.15.33:0/501464673 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a401017a0 0x7f3a40101c20 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.753+0000 7f3a475f3640 1 --2- 172.21.15.33:0/501464673 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3a401077b0 0x7f3a40107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.753+0000 7f3a475f3640 1 -- 172.21.15.33:0/501464673 >> 172.21.15.33:0/501464673 conn(0x7f3a400fd2b0 msgr2=0x7f3a400ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:01.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.753+0000 7f3a475f3640 1 -- 172.21.15.33:0/501464673 shutdown_connections 2024-03-19T02:56:01.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.753+0000 7f3a475f3640 1 -- 172.21.15.33:0/501464673 wait complete. 2024-03-19T02:56:01.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.754+0000 7f3a475f3640 1 Processor -- start 2024-03-19T02:56:01.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.754+0000 7f3a475f3640 1 -- start start 2024-03-19T02:56:01.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.754+0000 7f3a475f3640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a401017a0 0x7f3a4019a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.754+0000 7f3a475f3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a40102160 0x7f3a4019a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.754+0000 7f3a475f3640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3a401077b0 0x7f3a4019eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.754+0000 7f3a475f3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3a40111a70 con 0x7f3a40102160 2024-03-19T02:56:01.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.754+0000 7f3a475f3640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3a401118f0 con 0x7f3a401017a0 2024-03-19T02:56:01.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.754+0000 7f3a475f3640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3a40111bf0 con 0x7f3a401077b0 2024-03-19T02:56:01.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.754+0000 7f3a4538f640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a401017a0 0x7f3a4019a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.754+0000 7f3a4538f640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a401017a0 0x7f3a4019a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:37322/0 (socket says 172.21.15.33:37322) 2024-03-19T02:56:01.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.754+0000 7f3a44b8e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a40102160 0x7f3a4019a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.754+0000 7f3a4538f640 1 -- 172.21.15.33:0/548765313 learned_addr learned my addr 172.21.15.33:0/548765313 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:01.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.754+0000 7f3a45b90640 1 --2- 172.21.15.33:0/548765313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3a401077b0 0x7f3a4019eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.755+0000 7f3a4538f640 1 -- 172.21.15.33:0/548765313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3a401077b0 msgr2=0x7f3a4019eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:01.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.755+0000 7f3a4538f640 1 --2- 172.21.15.33:0/548765313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3a401077b0 0x7f3a4019eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.755+0000 7f3a4538f640 1 -- 172.21.15.33:0/548765313 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a40102160 msgr2=0x7f3a4019a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:01.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.755+0000 7f3a4538f640 1 --2- 172.21.15.33:0/548765313 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a40102160 0x7f3a4019a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.755+0000 7f3a4538f640 1 -- 172.21.15.33:0/548765313 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a4019f230 con 0x7f3a401017a0 2024-03-19T02:56:01.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.755+0000 7f3a4538f640 1 --2- 172.21.15.33:0/548765313 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a401017a0 0x7f3a4019a290 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3a3000e9f0 tx=0x7f3a3000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:01.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.755+0000 7f3a367fc640 1 -- 172.21.15.33:0/548765313 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3000ccc0 con 0x7f3a401017a0 2024-03-19T02:56:01.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.755+0000 7f3a367fc640 1 -- 172.21.15.33:0/548765313 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a3000ce60 con 0x7f3a401017a0 2024-03-19T02:56:01.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.755+0000 7f3a475f3640 1 -- 172.21.15.33:0/548765313 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a4019f520 con 0x7f3a401017a0 2024-03-19T02:56:01.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.755+0000 7f3a367fc640 1 -- 172.21.15.33:0/548765313 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a30010430 con 0x7f3a401017a0 2024-03-19T02:56:01.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.755+0000 7f3a475f3640 1 -- 172.21.15.33:0/548765313 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a401a6db0 con 0x7f3a401017a0 2024-03-19T02:56:01.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.757+0000 7f3a367fc640 1 -- 172.21.15.33:0/548765313 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f3a30005020 con 0x7f3a401017a0 2024-03-19T02:56:01.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.758+0000 7f3a4a1d7740 1 Processor -- start 2024-03-19T02:56:01.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.758+0000 7f3a4a1d7740 1 -- start start 2024-03-19T02:56:01.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.758+0000 7f3a4a1d7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bc8c56a380 0x55bc8c56a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.758+0000 7f3a367fc640 1 --2- 172.21.15.33:0/548765313 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f3a2c0a2970 0x7f3a2c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.758+0000 7f3a4a1d7740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bc8c56aca0 0x55bc8c573340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.758+0000 7f3a4a1d7740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bc8c573880 0x55bc8c575c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.758+0000 7f3a4a1d7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55bc8c4ce160 con 0x55bc8c56a380 2024-03-19T02:56:01.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.758+0000 7f3a4a1d7740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55bc8c476e10 con 0x55bc8c56aca0 2024-03-19T02:56:01.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.758+0000 7f3a4a1d7740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55bc8c4ca200 con 0x55bc8c573880 2024-03-19T02:56:01.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.759+0000 7f3a44b8e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bc8c56aca0 0x55bc8c573340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.759+0000 7f3a4538f640 1 --2- 172.21.15.33:0/548765313 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f3a2c0a2970 0x7f3a2c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.759+0000 7f3a44b8e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bc8c56aca0 0x55bc8c573340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:37336/0 (socket says 172.21.15.33:37336) 2024-03-19T02:56:01.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.759+0000 7f3a367fc640 1 -- 172.21.15.33:0/548765313 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f3a300cf4c0 con 0x7f3a401017a0 2024-03-19T02:56:01.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.759+0000 7f3a44b8e640 1 -- 172.21.15.33:0/2792770455 learned_addr learned my addr 172.21.15.33:0/2792770455 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:01.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.759+0000 7f3a45b90640 1 --2- 172.21.15.33:0/2792770455 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bc8c56a380 0x55bc8c56a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.759+0000 7f3a45b90640 1 --2- 172.21.15.33:0/2792770455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bc8c573880 0x55bc8c575c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.759+0000 7f3a44b8e640 1 -- 172.21.15.33:0/2792770455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bc8c573880 msgr2=0x55bc8c575c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:01.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.759+0000 7f3a44b8e640 1 --2- 172.21.15.33:0/2792770455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bc8c573880 0x55bc8c575c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.759+0000 7f3a44b8e640 1 -- 172.21.15.33:0/2792770455 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bc8c56a380 msgr2=0x55bc8c56a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:01.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.759+0000 7f3a44b8e640 1 --2- 172.21.15.33:0/2792770455 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bc8c56a380 0x55bc8c56a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.759+0000 7f3a44b8e640 1 -- 172.21.15.33:0/2792770455 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc8c2aa7e0 con 0x55bc8c56aca0 2024-03-19T02:56:01.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.759+0000 7f3a4538f640 1 --2- 172.21.15.33:0/548765313 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f3a2c0a2970 0x7f3a2c0a4e30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3a300cc990 tx=0x7f3a3001bb30 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:01.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.759+0000 7f3a44b8e640 1 --2- 172.21.15.33:0/2792770455 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bc8c56aca0 0x55bc8c573340 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c036040 tx=0x7f3a3c009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88fab54d634168f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:01.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.759+0000 7f3a2b7fe640 1 -- 172.21.15.33:0/2792770455 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c045020 con 0x55bc8c56aca0 2024-03-19T02:56:01.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.759+0000 7f3a2b7fe640 1 -- 172.21.15.33:0/2792770455 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a3c02c7e0 con 0x55bc8c56aca0 2024-03-19T02:56:01.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.759+0000 7f3a2b7fe640 1 -- 172.21.15.33:0/2792770455 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c005480 con 0x55bc8c56aca0 2024-03-19T02:56:01.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.760+0000 7f3a4a1d7740 1 -- 172.21.15.33:0/2792770455 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bc8c56aca0 msgr2=0x55bc8c573340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:01.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.760+0000 7f3a4a1d7740 1 --2- 172.21.15.33:0/2792770455 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bc8c56aca0 0x55bc8c573340 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c036040 tx=0x7f3a3c009560 comp rx=0 tx=0).stop 2024-03-19T02:56:01.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.760+0000 7f3a4a1d7740 1 -- 172.21.15.33:0/2792770455 shutdown_connections 2024-03-19T02:56:01.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.760+0000 7f3a4a1d7740 1 --2- 172.21.15.33:0/2792770455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bc8c573880 0x55bc8c575c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.760+0000 7f3a4a1d7740 1 --2- 172.21.15.33:0/2792770455 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bc8c56aca0 0x55bc8c573340 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.760+0000 7f3a4a1d7740 1 --2- 172.21.15.33:0/2792770455 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bc8c56a380 0x55bc8c56a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.760+0000 7f3a4a1d7740 1 -- 172.21.15.33:0/2792770455 >> 172.21.15.33:0/2792770455 conn(0x55bc8c564ab0 msgr2=0x55bc8c564e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:01.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.760+0000 7f3a4a1d7740 1 -- 172.21.15.33:0/2792770455 shutdown_connections 2024-03-19T02:56:01.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.760+0000 7f3a4a1d7740 1 -- 172.21.15.33:0/2792770455 wait complete. 2024-03-19T02:56:01.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.761+0000 7f3a4a1d7740 1 -- start start 2024-03-19T02:56:01.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.761+0000 7f3a4a1d7740 1 Processor -- start 2024-03-19T02:56:01.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.761+0000 7f3a4a1d7740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bc8c56a380 0x55bc8c5707c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.761+0000 7f3a4a1d7740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bc8c56aca0 0x55bc8c570d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.761+0000 7f3a4a1d7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bc8c573880 0x55bc8c571240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.761+0000 7f3a4a1d7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55bc8c576610 con 0x55bc8c573880 2024-03-19T02:56:01.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.761+0000 7f3a4a1d7740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55bc8c48a1a0 con 0x55bc8c56a380 2024-03-19T02:56:01.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.761+0000 7f3a4a1d7740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55bc8c576790 con 0x55bc8c56aca0 2024-03-19T02:56:01.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.761+0000 7f3a4a1d7740 10 client.0 _ll_register_callbacks cb 0x55bc8c1e1c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:56:01.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.761+0000 7f3a45b90640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bc8c56a380 0x55bc8c5707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.761+0000 7f3a45b90640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bc8c56a380 0x55bc8c5707c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:37340/0 (socket says 172.21.15.33:37340) 2024-03-19T02:56:01.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.761+0000 7f3a45b90640 1 -- 172.21.15.33:0/1127346360 learned_addr learned my addr 172.21.15.33:0/1127346360 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:01.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.761+0000 7f3a44b8e640 1 --2- 172.21.15.33:0/1127346360 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bc8c56aca0 0x55bc8c570d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.761+0000 7f3a45b90640 1 --2- 172.21.15.33:0/1127346360 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bc8c573880 0x55bc8c571240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.762+0000 7f3a44b8e640 1 -- 172.21.15.33:0/1127346360 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bc8c56a380 msgr2=0x55bc8c5707c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:01.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.762+0000 7f3a44b8e640 1 --2- 172.21.15.33:0/1127346360 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bc8c56a380 0x55bc8c5707c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.762+0000 7f3a44b8e640 1 -- 172.21.15.33:0/1127346360 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bc8c573880 msgr2=0x55bc8c571240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:01.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.762+0000 7f3a44b8e640 1 --2- 172.21.15.33:0/1127346360 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bc8c573880 0x55bc8c571240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.762+0000 7f3a44b8e640 1 -- 172.21.15.33:0/1127346360 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc8c30af70 con 0x55bc8c56aca0 2024-03-19T02:56:01.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.762+0000 7f3a45b90640 1 --2- 172.21.15.33:0/1127346360 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bc8c56a380 0x55bc8c5707c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:01.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.762+0000 7f3a44b8e640 1 --2- 172.21.15.33:0/1127346360 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bc8c56aca0 0x55bc8c570d00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c009900 tx=0x7f3a3c0053f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd924c652964d57c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:01.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.762+0000 7f3a28ff9640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-19T02:56:01.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.762+0000 7f3a28ff9640 1 -- 172.21.15.33:0/1127346360 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c050070 con 0x55bc8c56aca0 2024-03-19T02:56:01.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.762+0000 7f3a28ff9640 1 -- 172.21.15.33:0/1127346360 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f3a3c004450 con 0x55bc8c56aca0 2024-03-19T02:56:01.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.762+0000 7f3a4a1d7740 1 -- 172.21.15.33:0/1127346360 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a3c02c7e0 con 0x55bc8c56aca0 2024-03-19T02:56:01.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.762+0000 7f3a28ff9640 1 -- 172.21.15.33:0/1127346360 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3a3c059050 con 0x55bc8c56aca0 2024-03-19T02:56:01.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.762+0000 7f3a4a1d7740 10 client.25933 fetch_fsmap learned FSMap version 25 2024-03-19T02:56:01.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.762+0000 7f3a4a1d7740 1 -- 172.21.15.33:0/1127346360 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55bc8c1a2170 con 0x55bc8c56aca0 2024-03-19T02:56:01.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.762+0000 7f3a28ff9640 1 -- 172.21.15.33:0/1127346360 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a3c02c7e0 con 0x55bc8c56aca0 2024-03-19T02:56:01.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.763+0000 7f3a28ff9640 1 -- 172.21.15.33:0/1127346360 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f3a3c053020 con 0x55bc8c56aca0 2024-03-19T02:56:01.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.763+0000 7f3a4a1d7740 10 client.25933 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:56:01.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.763+0000 7f3a4a1d7740 10 client.25933 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:56:01.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.763+0000 7f3a4a1d7740 20 client.25933 populate_metadata read hostname 'smithi033' 2024-03-19T02:56:01.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.763+0000 7f3a4a1d7740 1 --2- 172.21.15.33:0/1127346360 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55bc8c583980 0x55bc8c585d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.763+0000 7f3a4a1d7740 4 client.25933 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:56:01.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.763+0000 7f3a4a1d7740 1 -- 172.21.15.33:0/1127346360 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bc8c4db6a0 con 0x55bc8c583980 2024-03-19T02:56:01.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.763+0000 7f3a45b90640 1 --2- 172.21.15.33:0/1127346360 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55bc8c583980 0x55bc8c585d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.763+0000 7f3a45b90640 1 --2- 172.21.15.33:0/1127346360 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55bc8c583980 0x55bc8c585d70 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7460ee3c3def929c server_cookie=3f78749efc7fd75d in_seq=0 out_seq=0 2024-03-19T02:56:01.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.763+0000 7f3a28ff9640 10 client.25933 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:56:01.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.764+0000 7f3a28ff9640 1 -- 172.21.15.33:0/1127346360 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bc8c4db6a0 con 0x55bc8c583980 2024-03-19T02:56:01.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.764+0000 7f3a28ff9640 10 client.25933 handle_command_reply: tid=0 2024-03-19T02:56:01.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.764+0000 7f3a4a1d7740 1 client.25933 shutdown 2024-03-19T02:56:01.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.764+0000 7f3a4a1d7740 1 -- 172.21.15.33:0/1127346360 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bc8c56aca0 msgr2=0x55bc8c570d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:01.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.764+0000 7f3a4a1d7740 1 --2- 172.21.15.33:0/1127346360 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bc8c56aca0 0x55bc8c570d00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c009900 tx=0x7f3a3c0053f0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.764+0000 7f3a4a1d7740 1 -- 172.21.15.33:0/1127346360 shutdown_connections 2024-03-19T02:56:01.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.764+0000 7f3a4a1d7740 1 --2- 172.21.15.33:0/1127346360 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55bc8c583980 0x55bc8c585d70 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.764+0000 7f3a4a1d7740 1 --2- 172.21.15.33:0/1127346360 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bc8c56aca0 0x55bc8c570d00 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.764+0000 7f3a4a1d7740 1 --2- 172.21.15.33:0/1127346360 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bc8c573880 0x55bc8c571240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.764+0000 7f3a4a1d7740 1 --2- 172.21.15.33:0/1127346360 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bc8c56a380 0x55bc8c5707c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.764+0000 7f3a28ff9640 0 client.25933 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:56:01.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.764+0000 7f3a4a1d7740 1 -- 172.21.15.33:0/1127346360 >> 172.21.15.33:0/1127346360 conn(0x55bc8c564ab0 msgr2=0x55bc8c566800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:01.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.764+0000 7f3a4a1d7740 1 -- 172.21.15.33:0/1127346360 shutdown_connections 2024-03-19T02:56:01.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.764+0000 7f3a4a1d7740 1 -- 172.21.15.33:0/1127346360 wait complete. 2024-03-19T02:56:01.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.764+0000 7f3a4a1d7740 20 client.25933 trim_cache size 0 max 16384 2024-03-19T02:56:01.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.782+0000 7f3a4a1d7740 1 Processor -- start 2024-03-19T02:56:01.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.782+0000 7f3a4a1d7740 1 -- start start 2024-03-19T02:56:01.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.783+0000 7f3a4a1d7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bc8c56a380 0x55bc8c572ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.783+0000 7f3a4a1d7740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bc8c56aca0 0x55bc8c57f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.783+0000 7f3a4a1d7740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bc8c573880 0x55bc8c58b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.783+0000 7f3a4a1d7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3a401b35e0 con 0x55bc8c56a380 2024-03-19T02:56:01.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.783+0000 7f3a4a1d7740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3a401b0eb0 con 0x55bc8c573880 2024-03-19T02:56:01.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.783+0000 7f3a4a1d7740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3a401aece0 con 0x55bc8c56aca0 2024-03-19T02:56:01.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.783+0000 7f3a45b90640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bc8c56a380 0x55bc8c572ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.783+0000 7f3a44b8e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bc8c56aca0 0x55bc8c57f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.783+0000 7f3a45b90640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bc8c56a380 0x55bc8c572ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42224/0 (socket says 172.21.15.33:42224) 2024-03-19T02:56:01.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.783+0000 7f3a45b90640 1 -- 172.21.15.33:0/4216423479 learned_addr learned my addr 172.21.15.33:0/4216423479 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:01.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.783+0000 7f3a45b90640 1 --2- 172.21.15.33:0/4216423479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bc8c573880 0x55bc8c58b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.783+0000 7f3a44b8e640 1 -- 172.21.15.33:0/4216423479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bc8c573880 msgr2=0x55bc8c58b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:01.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.783+0000 7f3a44b8e640 1 --2- 172.21.15.33:0/4216423479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bc8c573880 0x55bc8c58b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.783+0000 7f3a44b8e640 1 -- 172.21.15.33:0/4216423479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bc8c56a380 msgr2=0x55bc8c572ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:01.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.783+0000 7f3a44b8e640 1 --2- 172.21.15.33:0/4216423479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bc8c56a380 0x55bc8c572ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.783+0000 7f3a44b8e640 1 -- 172.21.15.33:0/4216423479 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc8c56ed10 con 0x55bc8c56aca0 2024-03-19T02:56:01.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.783+0000 7f3a44b8e640 1 --2- 172.21.15.33:0/4216423479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bc8c56aca0 0x55bc8c57f3e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c005c40 tx=0x7f3a3c005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad58b0fefe3865b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:01.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.783+0000 7f3a2affd640 1 -- 172.21.15.33:0/4216423479 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c050070 con 0x55bc8c56aca0 2024-03-19T02:56:01.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.783+0000 7f3a2affd640 1 -- 172.21.15.33:0/4216423479 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a3c02c7e0 con 0x55bc8c56aca0 2024-03-19T02:56:01.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.784+0000 7f3a2affd640 1 -- 172.21.15.33:0/4216423479 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c045030 con 0x55bc8c56aca0 2024-03-19T02:56:01.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.784+0000 7f3a4a1d7740 1 -- 172.21.15.33:0/4216423479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bc8c56aca0 msgr2=0x55bc8c57f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:01.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.784+0000 7f3a4a1d7740 1 --2- 172.21.15.33:0/4216423479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bc8c56aca0 0x55bc8c57f3e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c005c40 tx=0x7f3a3c005fd0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.784+0000 7f3a4a1d7740 1 -- 172.21.15.33:0/4216423479 shutdown_connections 2024-03-19T02:56:01.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.784+0000 7f3a4a1d7740 1 --2- 172.21.15.33:0/4216423479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bc8c56aca0 0x55bc8c57f3e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.784+0000 7f3a4a1d7740 1 --2- 172.21.15.33:0/4216423479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bc8c573880 0x55bc8c58b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.784+0000 7f3a4a1d7740 1 --2- 172.21.15.33:0/4216423479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bc8c56a380 0x55bc8c572ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.784+0000 7f3a4a1d7740 1 -- 172.21.15.33:0/4216423479 >> 172.21.15.33:0/4216423479 conn(0x55bc8c564ab0 msgr2=0x55bc8c570780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:01.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.784+0000 7f3a4a1d7740 1 -- 172.21.15.33:0/4216423479 shutdown_connections 2024-03-19T02:56:01.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.784+0000 7f3a4a1d7740 1 -- 172.21.15.33:0/4216423479 wait complete. 2024-03-19T02:56:01.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.785+0000 7f3a4a1d7740 1 -- start start 2024-03-19T02:56:01.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.785+0000 7f3a4a1d7740 1 Processor -- start 2024-03-19T02:56:01.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.785+0000 7f3a4a1d7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bc8c56a380 0x55bc8c5776a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.785+0000 7f3a4a1d7740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bc8c56aca0 0x55bc8c577be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.785+0000 7f3a4a1d7740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bc8c573880 0x55bc8c5e89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.785+0000 7f3a4a1d7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3a401af370 con 0x55bc8c56a380 2024-03-19T02:56:01.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.785+0000 7f3a4a1d7740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3a401b07e0 con 0x55bc8c573880 2024-03-19T02:56:01.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.785+0000 7f3a4a1d7740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3a401afa40 con 0x55bc8c56aca0 2024-03-19T02:56:01.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.785+0000 7f3a4a1d7740 10 client.0 _ll_register_callbacks cb 0x7f3a38009df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:56:01.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.785+0000 7f3a45b90640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bc8c56a380 0x55bc8c5776a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.785+0000 7f3a44b8e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bc8c56aca0 0x55bc8c577be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.785+0000 7f3a44b8e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bc8c56aca0 0x55bc8c577be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:52450/0 (socket says 172.21.15.33:52450) 2024-03-19T02:56:01.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.785+0000 7f3a44b8e640 1 -- 172.21.15.33:0/56732407 learned_addr learned my addr 172.21.15.33:0/56732407 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:01.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.785+0000 7f3a45b90640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bc8c56a380 0x55bc8c5776a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42240/0 (socket says 172.21.15.33:42240) 2024-03-19T02:56:01.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.785+0000 7f3a45b90640 1 --2- 172.21.15.33:0/56732407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bc8c573880 0x55bc8c5e89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.786+0000 7f3a44b8e640 1 -- 172.21.15.33:0/56732407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bc8c573880 msgr2=0x55bc8c5e89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:01.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.786+0000 7f3a44b8e640 1 --2- 172.21.15.33:0/56732407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bc8c573880 0x55bc8c5e89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.786+0000 7f3a44b8e640 1 -- 172.21.15.33:0/56732407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bc8c56a380 msgr2=0x55bc8c5776a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:01.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.786+0000 7f3a44b8e640 1 --2- 172.21.15.33:0/56732407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bc8c56a380 0x55bc8c5776a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.786+0000 7f3a44b8e640 1 -- 172.21.15.33:0/56732407 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc8c5e8ee0 con 0x55bc8c56aca0 2024-03-19T02:56:01.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.786+0000 7f3a45b90640 1 --2- 172.21.15.33:0/56732407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bc8c56a380 0x55bc8c5776a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:01.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.786+0000 7f3a44b8e640 1 --2- 172.21.15.33:0/56732407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bc8c56aca0 0x55bc8c577be0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c02c170 tx=0x7f3a3c005600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b95811c9a4997be server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:01.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.786+0000 7f3a28ff9640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-19T02:56:01.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.786+0000 7f3a28ff9640 1 -- 172.21.15.33:0/56732407 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c061020 con 0x55bc8c56aca0 2024-03-19T02:56:01.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.786+0000 7f3a28ff9640 1 -- 172.21.15.33:0/56732407 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f3a3c0046c0 con 0x55bc8c56aca0 2024-03-19T02:56:01.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.786+0000 7f3a4a1d7740 1 -- 172.21.15.33:0/56732407 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a3c02c7e0 con 0x55bc8c56aca0 2024-03-19T02:56:01.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.786+0000 7f3a28ff9640 1 -- 172.21.15.33:0/56732407 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3a3c06c050 con 0x55bc8c56aca0 2024-03-19T02:56:01.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.786+0000 7f3a4a1d7740 10 client.25939 fetch_fsmap learned FSMap version 25 2024-03-19T02:56:01.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.786+0000 7f3a4a1d7740 1 -- 172.21.15.33:0/56732407 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55bc8c5ea050 con 0x55bc8c56aca0 2024-03-19T02:56:01.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.787+0000 7f3a28ff9640 1 -- 172.21.15.33:0/56732407 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a3c02c7e0 con 0x55bc8c56aca0 2024-03-19T02:56:01.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.787+0000 7f3a28ff9640 1 -- 172.21.15.33:0/56732407 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f3a3c053040 con 0x55bc8c56aca0 2024-03-19T02:56:01.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.787+0000 7f3a4a1d7740 10 client.25939 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:56:01.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.787+0000 7f3a4a1d7740 10 client.25939 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:56:01.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.787+0000 7f3a4a1d7740 20 client.25939 populate_metadata read hostname 'smithi033' 2024-03-19T02:56:01.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.787+0000 7f3a4a1d7740 1 --2- 172.21.15.33:0/56732407 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55bc8c5ea760 0x55bc8c5ecb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.787+0000 7f3a4a1d7740 4 client.25939 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:56:01.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.787+0000 7f3a4a1d7740 1 -- 172.21.15.33:0/56732407 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55bc8c4d59f0 con 0x55bc8c5ea760 2024-03-19T02:56:01.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.787+0000 7f3a45b90640 1 --2- 172.21.15.33:0/56732407 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55bc8c5ea760 0x55bc8c5ecb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.787+0000 7f3a28ff9640 10 client.25939 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:56:01.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.787+0000 7f3a45b90640 1 --2- 172.21.15.33:0/56732407 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55bc8c5ea760 0x55bc8c5ecb80 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c8b222a72b60be7 server_cookie=49cbc7021bbededf in_seq=0 out_seq=0 2024-03-19T02:56:01.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.788+0000 7f3a28ff9640 1 -- 172.21.15.33:0/56732407 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55bc8c4d59f0 con 0x55bc8c5ea760 2024-03-19T02:56:01.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.788+0000 7f3a28ff9640 10 client.25939 handle_command_reply: tid=0 2024-03-19T02:56:01.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.788+0000 7f3a4a1d7740 1 client.25939 shutdown 2024-03-19T02:56:01.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.788+0000 7f3a4a1d7740 1 -- 172.21.15.33:0/56732407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bc8c56aca0 msgr2=0x55bc8c577be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:01.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.788+0000 7f3a4a1d7740 1 --2- 172.21.15.33:0/56732407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bc8c56aca0 0x55bc8c577be0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c02c170 tx=0x7f3a3c005600 comp rx=0 tx=0).stop 2024-03-19T02:56:01.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.789+0000 7f3a4a1d7740 1 -- 172.21.15.33:0/56732407 shutdown_connections 2024-03-19T02:56:01.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.789+0000 7f3a4a1d7740 1 --2- 172.21.15.33:0/56732407 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55bc8c5ea760 0x55bc8c5ecb80 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.789+0000 7f3a4a1d7740 1 --2- 172.21.15.33:0/56732407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bc8c56aca0 0x55bc8c577be0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.789+0000 7f3a4a1d7740 1 --2- 172.21.15.33:0/56732407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bc8c573880 0x55bc8c5e89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.789+0000 7f3a4a1d7740 1 --2- 172.21.15.33:0/56732407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bc8c56a380 0x55bc8c5776a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.789+0000 7f3a28ff9640 0 client.25939 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:56:01.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.789+0000 7f3a4a1d7740 1 -- 172.21.15.33:0/56732407 >> 172.21.15.33:0/56732407 conn(0x55bc8c564ab0 msgr2=0x55bc8c563b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:01.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.789+0000 7f3a4a1d7740 1 -- 172.21.15.33:0/56732407 shutdown_connections 2024-03-19T02:56:01.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.789+0000 7f3a4a1d7740 1 -- 172.21.15.33:0/56732407 wait complete. 2024-03-19T02:56:01.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.789+0000 7f3a4a1d7740 20 client.25939 trim_cache size 0 max 16384 2024-03-19T02:56:01.795 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:56:01.795 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-03-19T02:56:01.795 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "36560140-2f46-4cfb-ade0-2e795cad1d97", 2024-03-19T02:56:01.795 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-03-19T02:56:01.795 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:56:01.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.790+0000 7f3a475f3640 1 -- 172.21.15.33:0/548765313 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f3a2c0a2970 msgr2=0x7f3a2c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:01.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.790+0000 7f3a475f3640 1 --2- 172.21.15.33:0/548765313 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f3a2c0a2970 0x7f3a2c0a4e30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3a300cc990 tx=0x7f3a3001bb30 comp rx=0 tx=0).stop 2024-03-19T02:56:01.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.790+0000 7f3a475f3640 1 -- 172.21.15.33:0/548765313 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a401017a0 msgr2=0x7f3a4019a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:01.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.790+0000 7f3a475f3640 1 --2- 172.21.15.33:0/548765313 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a401017a0 0x7f3a4019a290 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3a3000e9f0 tx=0x7f3a3000edf0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.790+0000 7f3a475f3640 1 -- 172.21.15.33:0/548765313 shutdown_connections 2024-03-19T02:56:01.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.790+0000 7f3a475f3640 1 --2- 172.21.15.33:0/548765313 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f3a2c0a2970 0x7f3a2c0a4e30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.790+0000 7f3a475f3640 1 --2- 172.21.15.33:0/548765313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3a401077b0 0x7f3a4019eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.790+0000 7f3a475f3640 1 --2- 172.21.15.33:0/548765313 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3a40102160 0x7f3a4019a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.790+0000 7f3a475f3640 1 --2- 172.21.15.33:0/548765313 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a401017a0 0x7f3a4019a290 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.790+0000 7f3a475f3640 1 -- 172.21.15.33:0/548765313 >> 172.21.15.33:0/548765313 conn(0x7f3a400fd2b0 msgr2=0x7f3a4010ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:01.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.790+0000 7f3a475f3640 1 -- 172.21.15.33:0/548765313 shutdown_connections 2024-03-19T02:56:01.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.790+0000 7f3a475f3640 1 -- 172.21.15.33:0/548765313 wait complete. 2024-03-19T02:56:01.816 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:56:01.816 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:56:01.936 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.934+0000 7f321e4a1640 1 -- 172.21.15.33:0/1594718656 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3218101050 msgr2=0x7f321810d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:01.936 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.934+0000 7f321e4a1640 1 --2- 172.21.15.33:0/1594718656 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3218101050 0x7f321810d560 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3210009ca0 tx=0x7f321002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.936 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.934+0000 7f321e4a1640 1 -- 172.21.15.33:0/1594718656 shutdown_connections 2024-03-19T02:56:01.936 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.934+0000 7f321e4a1640 1 --2- 172.21.15.33:0/1594718656 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3218101050 0x7f321810d560 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.936 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.934+0000 7f321e4a1640 1 --2- 172.21.15.33:0/1594718656 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3218100690 0x7f3218100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.936 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.934+0000 7f321e4a1640 1 --2- 172.21.15.33:0/1594718656 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32181066a0 0x7f3218106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.934+0000 7f321e4a1640 1 -- 172.21.15.33:0/1594718656 >> 172.21.15.33:0/1594718656 conn(0x7f3218077820 msgr2=0x7f3218077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:01.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.935+0000 7f321e4a1640 1 -- 172.21.15.33:0/1594718656 shutdown_connections 2024-03-19T02:56:01.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.935+0000 7f321e4a1640 1 -- 172.21.15.33:0/1594718656 wait complete. 2024-03-19T02:56:01.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.935+0000 7f321e4a1640 1 Processor -- start 2024-03-19T02:56:01.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.935+0000 7f321e4a1640 1 -- start start 2024-03-19T02:56:01.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.936+0000 7f321e4a1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3218100690 0x7f3218197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.936+0000 7f321e4a1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3218101050 0x7f32181984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.936+0000 7f321e4a1640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32181066a0 0x7f321819c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.936+0000 7f321e4a1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3218110960 con 0x7f3218100690 2024-03-19T02:56:01.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.936+0000 7f321e4a1640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f32181107e0 con 0x7f3218101050 2024-03-19T02:56:01.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.936+0000 7f321e4a1640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3218110ae0 con 0x7f32181066a0 2024-03-19T02:56:01.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.936+0000 7f3217fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3218100690 0x7f3218197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.936+0000 7f321ca3e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32181066a0 0x7f321819c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.936+0000 7f32177fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3218101050 0x7f32181984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.936+0000 7f3217fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3218100690 0x7f3218197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42254/0 (socket says 172.21.15.33:42254) 2024-03-19T02:56:01.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.936+0000 7f321ca3e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32181066a0 0x7f321819c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:52478/0 (socket says 172.21.15.33:52478) 2024-03-19T02:56:01.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.936+0000 7f3217fff640 1 -- 172.21.15.33:0/1166578682 learned_addr learned my addr 172.21.15.33:0/1166578682 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:01.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.936+0000 7f32177fe640 1 -- 172.21.15.33:0/1166578682 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32181066a0 msgr2=0x7f321819c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:01.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.936+0000 7f32177fe640 1 --2- 172.21.15.33:0/1166578682 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32181066a0 0x7f321819c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.936+0000 7f32177fe640 1 -- 172.21.15.33:0/1166578682 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3218100690 msgr2=0x7f3218197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:01.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.936+0000 7f32177fe640 1 --2- 172.21.15.33:0/1166578682 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3218100690 0x7f3218197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.936+0000 7f32177fe640 1 -- 172.21.15.33:0/1166578682 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f321819cf80 con 0x7f3218101050 2024-03-19T02:56:01.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.936+0000 7f321ca3e640 1 --2- 172.21.15.33:0/1166578682 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32181066a0 0x7f321819c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:01.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.936+0000 7f32177fe640 1 --2- 172.21.15.33:0/1166578682 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3218101050 0x7f32181984d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f320c00e9f0 tx=0x7f320c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:01.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.937+0000 7f32157fa640 1 -- 172.21.15.33:0/1166578682 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f320c00ccc0 con 0x7f3218101050 2024-03-19T02:56:01.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.937+0000 7f32157fa640 1 -- 172.21.15.33:0/1166578682 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f320c0108e0 con 0x7f3218101050 2024-03-19T02:56:01.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.937+0000 7f32157fa640 1 -- 172.21.15.33:0/1166578682 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f320c018880 con 0x7f3218101050 2024-03-19T02:56:01.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.937+0000 7f321e4a1640 1 -- 172.21.15.33:0/1166578682 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f321819d270 con 0x7f3218101050 2024-03-19T02:56:01.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.937+0000 7f321e4a1640 1 -- 172.21.15.33:0/1166578682 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32181a4b00 con 0x7f3218101050 2024-03-19T02:56:01.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.939+0000 7f32157fa640 1 -- 172.21.15.33:0/1166578682 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f320c010a90 con 0x7f3218101050 2024-03-19T02:56:01.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.939+0000 7f32157fa640 1 --2- 172.21.15.33:0/1166578682 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f32000a29f0 0x7f32000a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.939+0000 7f322116a740 1 Processor -- start 2024-03-19T02:56:01.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.939+0000 7f32157fa640 1 -- 172.21.15.33:0/1166578682 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f320c0cf480 con 0x7f3218101050 2024-03-19T02:56:01.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.939+0000 7f322116a740 1 -- start start 2024-03-19T02:56:01.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.939+0000 7f3217fff640 1 --2- 172.21.15.33:0/1166578682 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f32000a29f0 0x7f32000a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.940+0000 7f322116a740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eba48c7a30 0x55eba48c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.940+0000 7f322116a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eba48c8350 0x55eba48d09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.940+0000 7f322116a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eba48d0f30 0x55eba48d3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.940+0000 7f322116a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55eba482b500 con 0x55eba48c8350 2024-03-19T02:56:01.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.940+0000 7f322116a740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55eba47d3be0 con 0x55eba48c7a30 2024-03-19T02:56:01.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.940+0000 7f322116a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55eba48275a0 con 0x55eba48d0f30 2024-03-19T02:56:01.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.940+0000 7f32177fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eba48c8350 0x55eba48d09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.940+0000 7f3217fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eba48c7a30 0x55eba48c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.940+0000 7f32177fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eba48c8350 0x55eba48d09f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42258/0 (socket says 172.21.15.33:42258) 2024-03-19T02:56:01.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.940+0000 7f32177fe640 1 -- 172.21.15.33:0/2779636296 learned_addr learned my addr 172.21.15.33:0/2779636296 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:01.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.940+0000 7f321ca3e640 1 --2- 172.21.15.33:0/2779636296 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eba48d0f30 0x55eba48d3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.940+0000 7f3217fff640 1 --2- 172.21.15.33:0/1166578682 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f32000a29f0 0x7f32000a4eb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3208009ca0 tx=0x7f3208005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:01.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.940+0000 7f3217fff640 1 -- 172.21.15.33:0/2779636296 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eba48d0f30 msgr2=0x55eba48d3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:01.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.940+0000 7f3217fff640 1 --2- 172.21.15.33:0/2779636296 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eba48d0f30 0x55eba48d3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.940+0000 7f3217fff640 1 -- 172.21.15.33:0/2779636296 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eba48c8350 msgr2=0x55eba48d09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:01.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.940+0000 7f3217fff640 1 --2- 172.21.15.33:0/2779636296 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eba48c8350 0x55eba48d09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.940+0000 7f3217fff640 1 -- 172.21.15.33:0/2779636296 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eba47f5000 con 0x55eba48c7a30 2024-03-19T02:56:01.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.940+0000 7f32177fe640 1 --2- 172.21.15.33:0/2779636296 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eba48c8350 0x55eba48d09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:01.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.940+0000 7f3217fff640 1 --2- 172.21.15.33:0/2779636296 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eba48c7a30 0x55eba48c7e10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3208009ed0 tx=0x7f320801c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f25f6036ddd491d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:01.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.940+0000 7f31fe7fc640 1 -- 172.21.15.33:0/2779636296 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3208029070 con 0x55eba48c7a30 2024-03-19T02:56:01.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.941+0000 7f31fe7fc640 1 -- 172.21.15.33:0/2779636296 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f320801ce10 con 0x55eba48c7a30 2024-03-19T02:56:01.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.941+0000 7f31fe7fc640 1 -- 172.21.15.33:0/2779636296 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f320800f8f0 con 0x55eba48c7a30 2024-03-19T02:56:01.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.941+0000 7f322116a740 1 -- 172.21.15.33:0/2779636296 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eba48c7a30 msgr2=0x55eba48c7e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:01.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.941+0000 7f322116a740 1 --2- 172.21.15.33:0/2779636296 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eba48c7a30 0x55eba48c7e10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3208009ed0 tx=0x7f320801c000 comp rx=0 tx=0).stop 2024-03-19T02:56:01.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.941+0000 7f322116a740 1 -- 172.21.15.33:0/2779636296 shutdown_connections 2024-03-19T02:56:01.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.941+0000 7f322116a740 1 --2- 172.21.15.33:0/2779636296 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eba48d0f30 0x55eba48d3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.941+0000 7f322116a740 1 --2- 172.21.15.33:0/2779636296 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eba48c8350 0x55eba48d09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.941+0000 7f322116a740 1 --2- 172.21.15.33:0/2779636296 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eba48c7a30 0x55eba48c7e10 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.941+0000 7f322116a740 1 -- 172.21.15.33:0/2779636296 >> 172.21.15.33:0/2779636296 conn(0x55eba48c2160 msgr2=0x55eba48c2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:01.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.941+0000 7f322116a740 1 -- 172.21.15.33:0/2779636296 shutdown_connections 2024-03-19T02:56:01.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.941+0000 7f322116a740 1 -- 172.21.15.33:0/2779636296 wait complete. 2024-03-19T02:56:01.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.942+0000 7f322116a740 1 -- start start 2024-03-19T02:56:01.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.942+0000 7f322116a740 1 Processor -- start 2024-03-19T02:56:01.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.942+0000 7f322116a740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eba48c7a30 0x55eba48cde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.942+0000 7f322116a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eba48c8350 0x55eba48ce3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.942+0000 7f322116a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eba48d0f30 0x55eba48ce900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.942+0000 7f322116a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55eba48d3d20 con 0x55eba48d0f30 2024-03-19T02:56:01.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.942+0000 7f322116a740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55eba47e7160 con 0x55eba48c7a30 2024-03-19T02:56:01.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.942+0000 7f322116a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55eba48d3ea0 con 0x55eba48c8350 2024-03-19T02:56:01.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.942+0000 7f322116a740 10 client.0 _ll_register_callbacks cb 0x55eba4823a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:56:01.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.942+0000 7f3217fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eba48c7a30 0x55eba48cde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.942+0000 7f3217fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eba48c7a30 0x55eba48cde80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:37382/0 (socket says 172.21.15.33:37382) 2024-03-19T02:56:01.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.942+0000 7f321ca3e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eba48d0f30 0x55eba48ce900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.942+0000 7f3217fff640 1 -- 172.21.15.33:0/3677415877 learned_addr learned my addr 172.21.15.33:0/3677415877 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:01.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.942+0000 7f32177fe640 1 --2- 172.21.15.33:0/3677415877 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eba48c8350 0x55eba48ce3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.943+0000 7f3217fff640 1 -- 172.21.15.33:0/3677415877 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eba48c8350 msgr2=0x55eba48ce3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:01.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.943+0000 7f3217fff640 1 --2- 172.21.15.33:0/3677415877 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eba48c8350 0x55eba48ce3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.943+0000 7f3217fff640 1 -- 172.21.15.33:0/3677415877 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eba48d0f30 msgr2=0x55eba48ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:01.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.943+0000 7f3217fff640 1 --2- 172.21.15.33:0/3677415877 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eba48d0f30 0x55eba48ce900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.943+0000 7f3217fff640 1 -- 172.21.15.33:0/3677415877 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eba46097d0 con 0x55eba48c7a30 2024-03-19T02:56:01.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.943+0000 7f3217fff640 1 --2- 172.21.15.33:0/3677415877 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eba48c7a30 0x55eba48cde80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f320800fb30 tx=0x7f320800fb60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e41193eeebcb8a84 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:01.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.943+0000 7f31f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T02:56:01.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.943+0000 7f31f77fe640 1 -- 172.21.15.33:0/3677415877 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3208032060 con 0x55eba48c7a30 2024-03-19T02:56:01.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.943+0000 7f31f77fe640 1 -- 172.21.15.33:0/3677415877 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f320800fd00 con 0x55eba48c7a30 2024-03-19T02:56:01.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.943+0000 7f322116a740 1 -- 172.21.15.33:0/3677415877 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f320801ce10 con 0x55eba48c7a30 2024-03-19T02:56:01.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.943+0000 7f31f77fe640 1 -- 172.21.15.33:0/3677415877 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3208039050 con 0x55eba48c7a30 2024-03-19T02:56:01.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.943+0000 7f322116a740 10 client.26084 fetch_fsmap learned FSMap version 25 2024-03-19T02:56:01.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.943+0000 7f322116a740 1 -- 172.21.15.33:0/3677415877 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55eba453f3f0 con 0x55eba48c7a30 2024-03-19T02:56:01.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.944+0000 7f31f77fe640 1 -- 172.21.15.33:0/3677415877 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f320801ce10 con 0x55eba48c7a30 2024-03-19T02:56:01.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.944+0000 7f31f77fe640 1 -- 172.21.15.33:0/3677415877 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f320802c070 con 0x55eba48c7a30 2024-03-19T02:56:01.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.944+0000 7f322116a740 10 client.26084 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:56:01.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.944+0000 7f322116a740 10 client.26084 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:56:01.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.944+0000 7f322116a740 20 client.26084 populate_metadata read hostname 'smithi033' 2024-03-19T02:56:01.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.944+0000 7f322116a740 1 --2- 172.21.15.33:0/3677415877 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55eba48e1060 0x55eba48e3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.944+0000 7f322116a740 4 client.26084 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:56:01.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.944+0000 7f322116a740 1 -- 172.21.15.33:0/3677415877 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eba4838a40 con 0x55eba48e1060 2024-03-19T02:56:01.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.944+0000 7f32177fe640 1 --2- 172.21.15.33:0/3677415877 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55eba48e1060 0x55eba48e3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.944+0000 7f31f77fe640 10 client.26084 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:56:01.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.944+0000 7f32177fe640 1 --2- 172.21.15.33:0/3677415877 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55eba48e1060 0x55eba48e3450 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d25980ee1adc770d server_cookie=33111ca3849aa971 in_seq=0 out_seq=0 2024-03-19T02:56:01.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.945+0000 7f31f77fe640 1 -- 172.21.15.33:0/3677415877 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55eba4838a40 con 0x55eba48e1060 2024-03-19T02:56:01.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.945+0000 7f31f77fe640 10 client.26084 handle_command_reply: tid=0 2024-03-19T02:56:01.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.945+0000 7f322116a740 1 client.26084 shutdown 2024-03-19T02:56:01.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.945+0000 7f322116a740 1 -- 172.21.15.33:0/3677415877 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eba48c7a30 msgr2=0x55eba48cde80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:01.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.945+0000 7f322116a740 1 --2- 172.21.15.33:0/3677415877 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eba48c7a30 0x55eba48cde80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f320800fb30 tx=0x7f320800fb60 comp rx=0 tx=0).stop 2024-03-19T02:56:01.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.945+0000 7f322116a740 1 -- 172.21.15.33:0/3677415877 shutdown_connections 2024-03-19T02:56:01.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.945+0000 7f322116a740 1 --2- 172.21.15.33:0/3677415877 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55eba48e1060 0x55eba48e3450 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.945+0000 7f322116a740 1 --2- 172.21.15.33:0/3677415877 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eba48c8350 0x55eba48ce3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.945+0000 7f322116a740 1 --2- 172.21.15.33:0/3677415877 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eba48d0f30 0x55eba48ce900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.945+0000 7f322116a740 1 --2- 172.21.15.33:0/3677415877 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eba48c7a30 0x55eba48cde80 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.945+0000 7f31f77fe640 0 client.26084 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:56:01.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.945+0000 7f322116a740 1 -- 172.21.15.33:0/3677415877 >> 172.21.15.33:0/3677415877 conn(0x55eba48c2160 msgr2=0x55eba48c3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:01.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.945+0000 7f322116a740 1 -- 172.21.15.33:0/3677415877 shutdown_connections 2024-03-19T02:56:01.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.945+0000 7f322116a740 1 -- 172.21.15.33:0/3677415877 wait complete. 2024-03-19T02:56:01.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.945+0000 7f322116a740 20 client.26084 trim_cache size 0 max 16384 2024-03-19T02:56:01.962 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-03-19T02:56:01.962 INFO:tasks.workunit.client.0.smithi033.stdout: Children see throughput for 1 readers = 8983641.00 kB/sec 2024-03-19T02:56:01.962 INFO:tasks.workunit.client.0.smithi033.stdout: Parent sees throughput for 1 readers = 8981801.09 kB/sec 2024-03-19T02:56:01.962 INFO:tasks.workunit.client.0.smithi033.stdout: Min throughput per process = 8983641.00 kB/sec 2024-03-19T02:56:01.962 INFO:tasks.workunit.client.0.smithi033.stdout: Max throughput per process = 8983641.00 kB/sec 2024-03-19T02:56:01.962 INFO:tasks.workunit.client.0.smithi033.stdout: Avg throughput per process = 8983641.00 kB/sec 2024-03-19T02:56:01.962 INFO:tasks.workunit.client.0.smithi033.stdout: Min xfer = 10485760.00 kB 2024-03-19T02:56:01.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.964+0000 7f322116a740 1 Processor -- start 2024-03-19T02:56:01.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.964+0000 7f322116a740 1 -- start start 2024-03-19T02:56:01.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.964+0000 7f322116a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eba48c7a30 0x55eba48d05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.964+0000 7f322116a740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eba48c8350 0x55eba48c0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.964+0000 7f322116a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eba48d0f30 0x55eba48dcac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.964+0000 7f322116a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f32181b1470 con 0x55eba48d0f30 2024-03-19T02:56:01.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.964+0000 7f322116a740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f32181aed40 con 0x55eba48c8350 2024-03-19T02:56:01.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.964+0000 7f322116a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f32181acb70 con 0x55eba48c7a30 2024-03-19T02:56:01.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.964+0000 7f321ca3e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eba48d0f30 0x55eba48dcac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.964+0000 7f321ca3e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eba48d0f30 0x55eba48dcac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42286/0 (socket says 172.21.15.33:42286) 2024-03-19T02:56:01.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.964+0000 7f32177fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eba48c8350 0x55eba48c0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.964+0000 7f321ca3e640 1 -- 172.21.15.33:0/2687867518 learned_addr learned my addr 172.21.15.33:0/2687867518 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:01.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.964+0000 7f3217fff640 1 --2- 172.21.15.33:0/2687867518 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eba48c7a30 0x55eba48d05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.964+0000 7f321ca3e640 1 -- 172.21.15.33:0/2687867518 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eba48c7a30 msgr2=0x55eba48d05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:01.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.964+0000 7f321ca3e640 1 --2- 172.21.15.33:0/2687867518 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eba48c7a30 0x55eba48d05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.964+0000 7f321ca3e640 1 -- 172.21.15.33:0/2687867518 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eba48c8350 msgr2=0x55eba48c0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:01.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.964+0000 7f321ca3e640 1 --2- 172.21.15.33:0/2687867518 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eba48c8350 0x55eba48c0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.964+0000 7f321ca3e640 1 -- 172.21.15.33:0/2687867518 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eba48c1800 con 0x55eba48d0f30 2024-03-19T02:56:01.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.964+0000 7f32177fe640 1 --2- 172.21.15.33:0/2687867518 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eba48c8350 0x55eba48c0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:01.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.964+0000 7f321ca3e640 1 --2- 172.21.15.33:0/2687867518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eba48d0f30 0x55eba48dcac0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f3210037040 tx=0x7f3210005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a781cd89d7bbe88 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:01.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.965+0000 7f31fdffb640 1 -- 172.21.15.33:0/2687867518 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3210039040 con 0x55eba48d0f30 2024-03-19T02:56:01.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.965+0000 7f31fdffb640 1 -- 172.21.15.33:0/2687867518 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3210003e20 con 0x55eba48d0f30 2024-03-19T02:56:01.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.965+0000 7f31fdffb640 1 -- 172.21.15.33:0/2687867518 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3210004670 con 0x55eba48d0f30 2024-03-19T02:56:01.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.965+0000 7f322116a740 1 -- 172.21.15.33:0/2687867518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eba48d0f30 msgr2=0x55eba48dcac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:01.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.965+0000 7f322116a740 1 --2- 172.21.15.33:0/2687867518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eba48d0f30 0x55eba48dcac0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f3210037040 tx=0x7f3210005c50 comp rx=0 tx=0).stop 2024-03-19T02:56:01.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.965+0000 7f322116a740 1 -- 172.21.15.33:0/2687867518 shutdown_connections 2024-03-19T02:56:01.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.965+0000 7f322116a740 1 --2- 172.21.15.33:0/2687867518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eba48d0f30 0x55eba48dcac0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.965+0000 7f322116a740 1 --2- 172.21.15.33:0/2687867518 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eba48c8350 0x55eba48c0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.965+0000 7f322116a740 1 --2- 172.21.15.33:0/2687867518 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eba48c7a30 0x55eba48d05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.965+0000 7f322116a740 1 -- 172.21.15.33:0/2687867518 >> 172.21.15.33:0/2687867518 conn(0x55eba48c2160 msgr2=0x55eba48cceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:01.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.966+0000 7f322116a740 1 -- 172.21.15.33:0/2687867518 shutdown_connections 2024-03-19T02:56:01.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.966+0000 7f322116a740 1 -- 172.21.15.33:0/2687867518 wait complete. 2024-03-19T02:56:01.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.966+0000 7f322116a740 1 -- start start 2024-03-19T02:56:01.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.966+0000 7f322116a740 1 Processor -- start 2024-03-19T02:56:01.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.966+0000 7f322116a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eba48c7a30 0x55eba48d4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.966+0000 7f322116a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eba48c8350 0x55eba48d4e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.966+0000 7f322116a740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eba48d0f30 0x55eba48d53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.966+0000 7f322116a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f32181ad200 con 0x55eba48c7a30 2024-03-19T02:56:01.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.966+0000 7f322116a740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f32181ae670 con 0x55eba48d0f30 2024-03-19T02:56:01.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.966+0000 7f322116a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f32181ad8d0 con 0x55eba48c8350 2024-03-19T02:56:01.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.966+0000 7f322116a740 10 client.0 _ll_register_callbacks cb 0x7f3208027070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:56:01.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.966+0000 7f3217fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eba48c7a30 0x55eba48d4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.966+0000 7f3217fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eba48c7a30 0x55eba48d4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42292/0 (socket says 172.21.15.33:42292) 2024-03-19T02:56:01.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.966+0000 7f3217fff640 1 -- 172.21.15.33:0/2004746826 learned_addr learned my addr 172.21.15.33:0/2004746826 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:01.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.966+0000 7f321ca3e640 1 --2- 172.21.15.33:0/2004746826 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eba48d0f30 0x55eba48d53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.966+0000 7f32177fe640 1 --2- 172.21.15.33:0/2004746826 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eba48c8350 0x55eba48d4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.967+0000 7f3217fff640 1 -- 172.21.15.33:0/2004746826 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eba48c8350 msgr2=0x55eba48d4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:01.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.967+0000 7f3217fff640 1 --2- 172.21.15.33:0/2004746826 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eba48c8350 0x55eba48d4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.967+0000 7f3217fff640 1 -- 172.21.15.33:0/2004746826 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eba48d0f30 msgr2=0x55eba48d53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:01.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.967+0000 7f3217fff640 1 --2- 172.21.15.33:0/2004746826 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eba48d0f30 0x55eba48d53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.967+0000 7f3217fff640 1 -- 172.21.15.33:0/2004746826 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eba44ff0f0 con 0x55eba48c7a30 2024-03-19T02:56:01.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.967+0000 7f321ca3e640 1 --2- 172.21.15.33:0/2004746826 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eba48d0f30 0x55eba48d53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:01.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.967+0000 7f3217fff640 1 --2- 172.21.15.33:0/2004746826 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eba48c7a30 0x55eba48d4930 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f3208009ed0 tx=0x7f3208004e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcb7b6664f74a005 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:01.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.967+0000 7f31f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:56:01.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.967+0000 7f31f77fe640 1 -- 172.21.15.33:0/2004746826 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3208042060 con 0x55eba48c7a30 2024-03-19T02:56:01.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.967+0000 7f31f77fe640 1 -- 172.21.15.33:0/2004746826 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f320800fdd0 con 0x55eba48c7a30 2024-03-19T02:56:01.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.967+0000 7f322116a740 1 -- 172.21.15.33:0/2004746826 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3210003e20 con 0x55eba48c7a30 2024-03-19T02:56:01.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.967+0000 7f31f77fe640 1 -- 172.21.15.33:0/2004746826 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3208049050 con 0x55eba48c7a30 2024-03-19T02:56:01.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.967+0000 7f322116a740 10 client.16266 fetch_fsmap learned FSMap version 25 2024-03-19T02:56:01.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.967+0000 7f322116a740 1 -- 172.21.15.33:0/2004746826 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55eba4947220 con 0x55eba48c7a30 2024-03-19T02:56:01.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.967+0000 7f31f77fe640 1 -- 172.21.15.33:0/2004746826 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3210003e20 con 0x55eba48c7a30 2024-03-19T02:56:01.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.968+0000 7f31f77fe640 1 -- 172.21.15.33:0/2004746826 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f320802c040 con 0x55eba48c7a30 2024-03-19T02:56:01.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.968+0000 7f322116a740 10 client.16266 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:56:01.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.968+0000 7f322116a740 10 client.16266 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:56:01.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.968+0000 7f322116a740 20 client.16266 populate_metadata read hostname 'smithi033' 2024-03-19T02:56:01.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.968+0000 7f322116a740 1 --2- 172.21.15.33:0/2004746826 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55eba49478b0 0x55eba4949cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:01.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.968+0000 7f322116a740 4 client.16266 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:56:01.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.968+0000 7f322116a740 1 -- 172.21.15.33:0/2004746826 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55eba48c9a80 con 0x55eba49478b0 2024-03-19T02:56:01.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.968+0000 7f32177fe640 1 --2- 172.21.15.33:0/2004746826 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55eba49478b0 0x55eba4949cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:01.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.968+0000 7f31f77fe640 10 client.16266 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:56:01.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.968+0000 7f32177fe640 1 --2- 172.21.15.33:0/2004746826 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55eba49478b0 0x55eba4949cd0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32423dca6cc532e3 server_cookie=32db90028e299fb0 in_seq=0 out_seq=0 2024-03-19T02:56:01.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.968+0000 7f31f77fe640 1 -- 172.21.15.33:0/2004746826 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55eba48c9a80 con 0x55eba49478b0 2024-03-19T02:56:01.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.968+0000 7f31f77fe640 10 client.16266 handle_command_reply: tid=0 2024-03-19T02:56:01.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.968+0000 7f322116a740 1 client.16266 shutdown 2024-03-19T02:56:01.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.968+0000 7f322116a740 1 -- 172.21.15.33:0/2004746826 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eba48c7a30 msgr2=0x55eba48d4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:01.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.968+0000 7f322116a740 1 --2- 172.21.15.33:0/2004746826 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eba48c7a30 0x55eba48d4930 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f3208009ed0 tx=0x7f3208004e60 comp rx=0 tx=0).stop 2024-03-19T02:56:01.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.968+0000 7f322116a740 1 -- 172.21.15.33:0/2004746826 shutdown_connections 2024-03-19T02:56:01.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.968+0000 7f322116a740 1 --2- 172.21.15.33:0/2004746826 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55eba49478b0 0x55eba4949cd0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.968+0000 7f322116a740 1 --2- 172.21.15.33:0/2004746826 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eba48c8350 0x55eba48d4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.968+0000 7f322116a740 1 --2- 172.21.15.33:0/2004746826 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eba48d0f30 0x55eba48d53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.968+0000 7f31f77fe640 0 client.16266 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:56:01.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.968+0000 7f322116a740 1 --2- 172.21.15.33:0/2004746826 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eba48c7a30 0x55eba48d4930 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.968+0000 7f322116a740 1 -- 172.21.15.33:0/2004746826 >> 172.21.15.33:0/2004746826 conn(0x55eba48c2160 msgr2=0x55eba48c3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:01.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.969+0000 7f322116a740 1 -- 172.21.15.33:0/2004746826 shutdown_connections 2024-03-19T02:56:01.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.969+0000 7f322116a740 1 -- 172.21.15.33:0/2004746826 wait complete. 2024-03-19T02:56:01.976 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.969+0000 7f322116a740 20 client.16266 trim_cache size 0 max 16384 2024-03-19T02:56:01.977 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:56:01.977 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:56:01.977 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-03-19T02:56:01.977 INFO:teuthology.orchestra.run.smithi033.stdout: "36560140-2f46-4cfb-ade0-2e795cad1d97": { 2024-03-19T02:56:01.977 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-03-19T02:56:01.977 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "36560140-2f46-4cfb-ade0-2e795cad1d97", 2024-03-19T02:56:01.977 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-03-19T02:56:01.977 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T02:56:01.977 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T02:56:01.977 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:56:01.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.969+0000 7f321e4a1640 1 -- 172.21.15.33:0/1166578682 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f32000a29f0 msgr2=0x7f32000a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:01.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.969+0000 7f321e4a1640 1 --2- 172.21.15.33:0/1166578682 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f32000a29f0 0x7f32000a4eb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3208009ca0 tx=0x7f3208005f90 comp rx=0 tx=0).stop 2024-03-19T02:56:01.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.969+0000 7f321e4a1640 1 -- 172.21.15.33:0/1166578682 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3218101050 msgr2=0x7f32181984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:01.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.969+0000 7f321e4a1640 1 --2- 172.21.15.33:0/1166578682 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3218101050 0x7f32181984d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f320c00e9f0 tx=0x7f320c00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.969+0000 7f321e4a1640 1 -- 172.21.15.33:0/1166578682 shutdown_connections 2024-03-19T02:56:01.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.969+0000 7f321e4a1640 1 --2- 172.21.15.33:0/1166578682 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f32000a29f0 0x7f32000a4eb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.969+0000 7f321e4a1640 1 --2- 172.21.15.33:0/1166578682 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32181066a0 0x7f321819c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.969+0000 7f321e4a1640 1 --2- 172.21.15.33:0/1166578682 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3218101050 0x7f32181984d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.969+0000 7f321e4a1640 1 --2- 172.21.15.33:0/1166578682 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3218100690 0x7f3218197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:01.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.969+0000 7f321e4a1640 1 -- 172.21.15.33:0/1166578682 >> 172.21.15.33:0/1166578682 conn(0x7f3218077820 msgr2=0x7f32180ff9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:01.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.969+0000 7f321e4a1640 1 -- 172.21.15.33:0/1166578682 shutdown_connections 2024-03-19T02:56:01.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:01.969+0000 7f321e4a1640 1 -- 172.21.15.33:0/1166578682 wait complete. 2024-03-19T02:56:01.981 INFO:tasks.cephfs.filesystem:scrub status for tag:36560140-2f46-4cfb-ade0-2e795cad1d97 - {'path': '/', 'tag': '36560140-2f46-4cfb-ade0-2e795cad1d97', 'options': 'recursive,force'} 2024-03-19T02:56:03.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:02 smithi096 ceph-mon[35946]: pgmap v80: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 117 MiB/s wr, 34 op/s 2024-03-19T02:56:03.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:02 smithi096 ceph-mon[35946]: scrub queued for path: / 2024-03-19T02:56:03.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:02 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T02:56:03.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:02 smithi096 ceph-mon[35946]: scrub summary: active paths [/] 2024-03-19T02:56:03.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:02 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T02:56:03.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:02 smithi096 ceph-mon[35946]: scrub summary: idle 2024-03-19T02:56:03.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:02 smithi155 ceph-mon[35804]: pgmap v80: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 117 MiB/s wr, 34 op/s 2024-03-19T02:56:03.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:02 smithi155 ceph-mon[35804]: scrub queued for path: / 2024-03-19T02:56:03.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:02 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T02:56:03.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:02 smithi155 ceph-mon[35804]: scrub summary: active paths [/] 2024-03-19T02:56:03.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:02 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T02:56:03.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:02 smithi155 ceph-mon[35804]: scrub summary: idle 2024-03-19T02:56:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:02 smithi033 ceph-mon[31973]: pgmap v80: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 117 MiB/s wr, 34 op/s 2024-03-19T02:56:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:02 smithi033 ceph-mon[31973]: scrub queued for path: / 2024-03-19T02:56:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:02 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T02:56:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:02 smithi033 ceph-mon[31973]: scrub summary: active paths [/] 2024-03-19T02:56:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:02 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T02:56:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:02 smithi033 ceph-mon[31973]: scrub summary: idle 2024-03-19T02:56:05.218 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:04 smithi096 ceph-mon[35946]: pgmap v81: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 117 MiB/s wr, 52 op/s 2024-03-19T02:56:05.240 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-03-19T02:56:05.240 INFO:tasks.workunit.client.0.smithi033.stdout: Children see throughput for 1 re-readers = 8966532.00 kB/sec 2024-03-19T02:56:05.240 INFO:tasks.workunit.client.0.smithi033.stdout: Parent sees throughput for 1 re-readers = 8964769.09 kB/sec 2024-03-19T02:56:05.240 INFO:tasks.workunit.client.0.smithi033.stdout: Min throughput per process = 8966532.00 kB/sec 2024-03-19T02:56:05.240 INFO:tasks.workunit.client.0.smithi033.stdout: Max throughput per process = 8966532.00 kB/sec 2024-03-19T02:56:05.240 INFO:tasks.workunit.client.0.smithi033.stdout: Avg throughput per process = 8966532.00 kB/sec 2024-03-19T02:56:05.240 INFO:tasks.workunit.client.0.smithi033.stdout: Min xfer = 10485760.00 kB 2024-03-19T02:56:05.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:04 smithi155 ceph-mon[35804]: pgmap v81: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 117 MiB/s wr, 52 op/s 2024-03-19T02:56:05.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:04 smithi033 ceph-mon[31973]: pgmap v81: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 117 MiB/s wr, 52 op/s 2024-03-19T02:56:07.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:06 smithi096 ceph-mon[35946]: pgmap v82: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-19T02:56:07.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:06 smithi155 ceph-mon[35804]: pgmap v82: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-19T02:56:07.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:06 smithi033 ceph-mon[31973]: pgmap v82: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-19T02:56:09.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:08 smithi096 ceph-mon[35946]: pgmap v83: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-19T02:56:09.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:08 smithi155 ceph-mon[35804]: pgmap v83: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-19T02:56:09.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:08 smithi033 ceph-mon[31973]: pgmap v83: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-19T02:56:11.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:10 smithi096 ceph-mon[35946]: pgmap v84: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-19T02:56:11.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:10 smithi155 ceph-mon[35804]: pgmap v84: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-19T02:56:11.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:10 smithi033 ceph-mon[31973]: pgmap v84: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-19T02:56:12.100 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-03-19T02:56:12.101 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-03-19T02:56:12.101 INFO:tasks.workunit.client.0.smithi033.stdout: 2024-03-19T02:56:12.101 INFO:tasks.workunit.client.0.smithi033.stdout:iozone test complete. 2024-03-19T02:56:12.102 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-19T02:56:12.102 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-19T02:56:12.147 INFO:tasks.workunit:Stopping ['suites/iozone.sh'] on client.0... 2024-03-19T02:56:12.147 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-19T02:56:12.437 DEBUG:teuthology.parallel:result is None 2024-03-19T02:56:12.438 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-19T02:56:12.476 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-19T02:56:12.477 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-03-19T02:56:12.490 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-03-19T02:56:12.490 INFO:teuthology.task.exec:Executing custom commands... 2024-03-19T02:56:12.490 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi033.front.sepia.ceph.com 2024-03-19T02:56:12.491 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-03-19T02:56:12.545 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-03-19T02:56:13.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:12 smithi096 ceph-mon[35946]: pgmap v85: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-19T02:56:13.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:12 smithi155 ceph-mon[35804]: pgmap v85: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-19T02:56:13.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:12 smithi033 ceph-mon[31973]: pgmap v85: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-19T02:56:15.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:14 smithi096 ceph-mon[35946]: pgmap v86: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 27 KiB/s wr, 23 op/s 2024-03-19T02:56:15.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:14 smithi155 ceph-mon[35804]: pgmap v86: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 27 KiB/s wr, 23 op/s 2024-03-19T02:56:15.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:14 smithi033 ceph-mon[31973]: pgmap v86: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 27 KiB/s wr, 23 op/s 2024-03-19T02:56:17.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:16 smithi096 ceph-mon[35946]: pgmap v87: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 4.4 KiB/s wr, 5 op/s 2024-03-19T02:56:17.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:16 smithi155 ceph-mon[35804]: pgmap v87: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 4.4 KiB/s wr, 5 op/s 2024-03-19T02:56:17.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:16 smithi033 ceph-mon[31973]: pgmap v87: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 4.4 KiB/s wr, 5 op/s 2024-03-19T02:56:19.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:18 smithi096 ceph-mon[35946]: pgmap v88: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 4.4 KiB/s wr, 5 op/s 2024-03-19T02:56:19.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:18 smithi155 ceph-mon[35804]: pgmap v88: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 4.4 KiB/s wr, 5 op/s 2024-03-19T02:56:19.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:18 smithi033 ceph-mon[31973]: pgmap v88: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 4.4 KiB/s wr, 5 op/s 2024-03-19T02:56:21.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:20 smithi096 ceph-mon[35946]: pgmap v89: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.9 KiB/s wr, 222 op/s 2024-03-19T02:56:21.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:20 smithi155 ceph-mon[35804]: pgmap v89: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.9 KiB/s wr, 222 op/s 2024-03-19T02:56:21.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:20 smithi033 ceph-mon[31973]: pgmap v89: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.9 KiB/s wr, 222 op/s 2024-03-19T02:56:23.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:22 smithi096 ceph-mon[35946]: pgmap v90: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 5.8 KiB/s wr, 220 op/s 2024-03-19T02:56:23.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:22 smithi155 ceph-mon[35804]: pgmap v90: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 5.8 KiB/s wr, 220 op/s 2024-03-19T02:56:23.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:22 smithi033 ceph-mon[31973]: pgmap v90: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 5.8 KiB/s wr, 220 op/s 2024-03-19T02:56:25.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:24 smithi096 ceph-mon[35946]: pgmap v91: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.8 KiB/s wr, 222 op/s 2024-03-19T02:56:25.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:24 smithi155 ceph-mon[35804]: pgmap v91: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.8 KiB/s wr, 222 op/s 2024-03-19T02:56:25.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:24 smithi033 ceph-mon[31973]: pgmap v91: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.8 KiB/s wr, 222 op/s 2024-03-19T02:56:27.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:26 smithi096 ceph-mon[35946]: pgmap v92: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.5 KiB/s wr, 219 op/s 2024-03-19T02:56:27.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:26 smithi155 ceph-mon[35804]: pgmap v92: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.5 KiB/s wr, 219 op/s 2024-03-19T02:56:27.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:26 smithi033 ceph-mon[31973]: pgmap v92: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.5 KiB/s wr, 219 op/s 2024-03-19T02:56:29.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:28 smithi096 ceph-mon[35946]: pgmap v93: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.5 KiB/s wr, 219 op/s 2024-03-19T02:56:29.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:28 smithi155 ceph-mon[35804]: pgmap v93: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.5 KiB/s wr, 219 op/s 2024-03-19T02:56:29.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:28 smithi033 ceph-mon[31973]: pgmap v93: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.5 KiB/s wr, 219 op/s 2024-03-19T02:56:31.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:30 smithi096 ceph-mon[35946]: pgmap v94: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.5 KiB/s wr, 221 op/s 2024-03-19T02:56:31.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:30 smithi155 ceph-mon[35804]: pgmap v94: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.5 KiB/s wr, 221 op/s 2024-03-19T02:56:31.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:30 smithi033 ceph-mon[31973]: pgmap v94: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.5 KiB/s wr, 221 op/s 2024-03-19T02:56:31.982 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:56:31.983 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:56:32.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.098+0000 7f20350e8640 1 -- 172.21.15.33:0/1694663961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2030101790 msgr2=0x7f2030101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.098+0000 7f20350e8640 1 --2- 172.21.15.33:0/1694663961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2030101790 0x7f2030101c10 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f2018009930 tx=0x7f201802bc30 comp rx=0 tx=0).stop 2024-03-19T02:56:32.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.098+0000 7f20350e8640 1 -- 172.21.15.33:0/1694663961 shutdown_connections 2024-03-19T02:56:32.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.098+0000 7f20350e8640 1 --2- 172.21.15.33:0/1694663961 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2030102150 0x7f203010e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.098+0000 7f20350e8640 1 --2- 172.21.15.33:0/1694663961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2030101790 0x7f2030101c10 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.098+0000 7f20350e8640 1 --2- 172.21.15.33:0/1694663961 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20301077a0 0x7f2030107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.098+0000 7f20350e8640 1 -- 172.21.15.33:0/1694663961 >> 172.21.15.33:0/1694663961 conn(0x7f20300fd280 msgr2=0x7f20300ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:32.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.098+0000 7f20350e8640 1 -- 172.21.15.33:0/1694663961 shutdown_connections 2024-03-19T02:56:32.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.098+0000 7f20350e8640 1 -- 172.21.15.33:0/1694663961 wait complete. 2024-03-19T02:56:32.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.099+0000 7f20350e8640 1 Processor -- start 2024-03-19T02:56:32.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.099+0000 7f20350e8640 1 -- start start 2024-03-19T02:56:32.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.099+0000 7f20350e8640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2030101790 0x7f2030073b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.099+0000 7f20350e8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2030102150 0x7f2030074060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.099+0000 7f20350e8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20301077a0 0x7f20300745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.099+0000 7f20350e8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2030111a60 con 0x7f20301077a0 2024-03-19T02:56:32.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.099+0000 7f20350e8640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f20301118e0 con 0x7f2030102150 2024-03-19T02:56:32.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.099+0000 7f20350e8640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2030111be0 con 0x7f2030101790 2024-03-19T02:56:32.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.099+0000 7f202f59e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20301077a0 0x7f20300745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.099+0000 7f202f59e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20301077a0 0x7f20300745a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42584/0 (socket says 172.21.15.33:42584) 2024-03-19T02:56:32.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.099+0000 7f202f59e640 1 -- 172.21.15.33:0/1471545374 learned_addr learned my addr 172.21.15.33:0/1471545374 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:32.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.099+0000 7f2027fff640 1 --2- 172.21.15.33:0/1471545374 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2030102150 0x7f2030074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.099+0000 7f202ed9d640 1 --2- 172.21.15.33:0/1471545374 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2030101790 0x7f2030073b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.100+0000 7f202f59e640 1 -- 172.21.15.33:0/1471545374 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2030101790 msgr2=0x7f2030073b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.100+0000 7f202f59e640 1 --2- 172.21.15.33:0/1471545374 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2030101790 0x7f2030073b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.100+0000 7f202f59e640 1 -- 172.21.15.33:0/1471545374 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2030102150 msgr2=0x7f2030074060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.100+0000 7f202f59e640 1 --2- 172.21.15.33:0/1471545374 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2030102150 0x7f2030074060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.100+0000 7f202f59e640 1 -- 172.21.15.33:0/1471545374 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20301a6530 con 0x7f20301077a0 2024-03-19T02:56:32.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.100+0000 7f2027fff640 1 --2- 172.21.15.33:0/1471545374 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2030102150 0x7f2030074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:32.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.100+0000 7f202f59e640 1 --2- 172.21.15.33:0/1471545374 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20301077a0 0x7f20300745a0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f202800eb10 tx=0x7f202800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:32.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.100+0000 7f202cd99640 1 -- 172.21.15.33:0/1471545374 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2028004b20 con 0x7f20301077a0 2024-03-19T02:56:32.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.100+0000 7f202cd99640 1 -- 172.21.15.33:0/1471545374 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2028004cc0 con 0x7f20301077a0 2024-03-19T02:56:32.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.100+0000 7f20350e8640 1 -- 172.21.15.33:0/1471545374 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20301a6820 con 0x7f20301077a0 2024-03-19T02:56:32.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.100+0000 7f202cd99640 1 -- 172.21.15.33:0/1471545374 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2028010690 con 0x7f20301077a0 2024-03-19T02:56:32.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.100+0000 7f20350e8640 1 -- 172.21.15.33:0/1471545374 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20301a6d90 con 0x7f20301077a0 2024-03-19T02:56:32.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.102+0000 7f202cd99640 1 -- 172.21.15.33:0/1471545374 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f2028005020 con 0x7f20301077a0 2024-03-19T02:56:32.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.103+0000 7f2037daa740 1 Processor -- start 2024-03-19T02:56:32.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.103+0000 7f2037daa740 1 -- start start 2024-03-19T02:56:32.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.103+0000 7f202cd99640 1 --2- 172.21.15.33:0/1471545374 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f201c0a29f0 0x7f201c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.103+0000 7f2027fff640 1 --2- 172.21.15.33:0/1471545374 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f201c0a29f0 0x7f201c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.103+0000 7f202cd99640 1 -- 172.21.15.33:0/1471545374 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f20280cf820 con 0x7f20301077a0 2024-03-19T02:56:32.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.103+0000 7f2037daa740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642c08a1a30 0x5642c08a1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.103+0000 7f2037daa740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642c08a2350 0x5642c08aa9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.103+0000 7f2037daa740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642c08aaf30 0x5642c08ad320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.103+0000 7f2037daa740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5642c0805500 con 0x5642c08aaf30 2024-03-19T02:56:32.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.103+0000 7f2037daa740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5642c07adbe0 con 0x5642c08a1a30 2024-03-19T02:56:32.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.103+0000 7f2037daa740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5642c08015a0 con 0x5642c08a2350 2024-03-19T02:56:32.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.103+0000 7f202ed9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642c08a1a30 0x5642c08a1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.103+0000 7f202f59e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642c08aaf30 0x5642c08ad320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.103+0000 7f202ed9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642c08a1a30 0x5642c08a1e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:49822/0 (socket says 172.21.15.33:49822) 2024-03-19T02:56:32.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.103+0000 7f202ed9d640 1 -- 172.21.15.33:0/3493113924 learned_addr learned my addr 172.21.15.33:0/3493113924 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:32.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.103+0000 7f2027fff640 1 --2- 172.21.15.33:0/3493113924 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642c08a2350 0x5642c08aa9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.104+0000 7f202ed9d640 1 -- 172.21.15.33:0/3493113924 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642c08a2350 msgr2=0x5642c08aa9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:32.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.104+0000 7f202ed9d640 1 --2- 172.21.15.33:0/3493113924 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642c08a2350 0x5642c08aa9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.104+0000 7f202ed9d640 1 -- 172.21.15.33:0/3493113924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642c08aaf30 msgr2=0x5642c08ad320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:32.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.104+0000 7f202ed9d640 1 --2- 172.21.15.33:0/3493113924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642c08aaf30 0x5642c08ad320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.104+0000 7f202ed9d640 1 -- 172.21.15.33:0/3493113924 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642c07cf000 con 0x5642c08a1a30 2024-03-19T02:56:32.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.104+0000 7f202f59e640 1 --2- 172.21.15.33:0/3493113924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642c08aaf30 0x5642c08ad320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:32.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.104+0000 7f2027fff640 1 --2- 172.21.15.33:0/1471545374 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f201c0a29f0 0x7f201c0a4eb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2018005ab0 tx=0x7f201802c6d0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:32.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.104+0000 7f202ed9d640 1 --2- 172.21.15.33:0/3493113924 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642c08a1a30 0x5642c08a1e10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2020009c70 tx=0x7f2020011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8784c4841d19ecf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:32.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.104+0000 7f20257fa640 1 -- 172.21.15.33:0/3493113924 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2020004230 con 0x5642c08a1a30 2024-03-19T02:56:32.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.104+0000 7f20257fa640 1 -- 172.21.15.33:0/3493113924 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2020009e20 con 0x5642c08a1a30 2024-03-19T02:56:32.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.104+0000 7f20257fa640 1 -- 172.21.15.33:0/3493113924 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20200196d0 con 0x5642c08a1a30 2024-03-19T02:56:32.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.105+0000 7f2037daa740 1 -- 172.21.15.33:0/3493113924 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642c08a1a30 msgr2=0x5642c08a1e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.105+0000 7f2037daa740 1 --2- 172.21.15.33:0/3493113924 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642c08a1a30 0x5642c08a1e10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2020009c70 tx=0x7f2020011810 comp rx=0 tx=0).stop 2024-03-19T02:56:32.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.105+0000 7f2037daa740 1 -- 172.21.15.33:0/3493113924 shutdown_connections 2024-03-19T02:56:32.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.105+0000 7f2037daa740 1 --2- 172.21.15.33:0/3493113924 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642c08a2350 0x5642c08aa9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.105+0000 7f2037daa740 1 --2- 172.21.15.33:0/3493113924 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642c08aaf30 0x5642c08ad320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.105+0000 7f2037daa740 1 --2- 172.21.15.33:0/3493113924 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642c08a1a30 0x5642c08a1e10 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.105+0000 7f2037daa740 1 -- 172.21.15.33:0/3493113924 >> 172.21.15.33:0/3493113924 conn(0x5642c089c160 msgr2=0x5642c089c540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:32.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.105+0000 7f2037daa740 1 -- 172.21.15.33:0/3493113924 shutdown_connections 2024-03-19T02:56:32.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.105+0000 7f2037daa740 1 -- 172.21.15.33:0/3493113924 wait complete. 2024-03-19T02:56:32.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.105+0000 7f2037daa740 1 -- start start 2024-03-19T02:56:32.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.105+0000 7f2037daa740 1 Processor -- start 2024-03-19T02:56:32.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.105+0000 7f2037daa740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642c08a1a30 0x5642c08a7e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.105+0000 7f2037daa740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642c08a2350 0x5642c08a8370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.105+0000 7f2037daa740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642c08aaf30 0x5642c08a88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.106+0000 7f2037daa740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5642c08add20 con 0x5642c08aaf30 2024-03-19T02:56:32.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.106+0000 7f2037daa740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5642c07c1160 con 0x5642c08a2350 2024-03-19T02:56:32.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.106+0000 7f2037daa740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5642c08adea0 con 0x5642c08a1a30 2024-03-19T02:56:32.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.106+0000 7f2037daa740 10 client.0 _ll_register_callbacks cb 0x5642c07fda40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:56:32.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.106+0000 7f2027fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642c08a2350 0x5642c08a8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.106+0000 7f202f59e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642c08aaf30 0x5642c08a88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.106+0000 7f202ed9d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642c08a1a30 0x5642c08a7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.106+0000 7f2027fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642c08a2350 0x5642c08a8370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:49826/0 (socket says 172.21.15.33:49826) 2024-03-19T02:56:32.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.106+0000 7f2027fff640 1 -- 172.21.15.33:0/4263948015 learned_addr learned my addr 172.21.15.33:0/4263948015 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:32.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.106+0000 7f202f59e640 1 -- 172.21.15.33:0/4263948015 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642c08a1a30 msgr2=0x5642c08a7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:32.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.106+0000 7f202f59e640 1 --2- 172.21.15.33:0/4263948015 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642c08a1a30 0x5642c08a7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.122 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.106+0000 7f202f59e640 1 -- 172.21.15.33:0/4263948015 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642c08a2350 msgr2=0x5642c08a8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:32.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.106+0000 7f202f59e640 1 --2- 172.21.15.33:0/4263948015 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642c08a2350 0x5642c08a8370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.106+0000 7f202f59e640 1 -- 172.21.15.33:0/4263948015 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642c05e37d0 con 0x5642c08aaf30 2024-03-19T02:56:32.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.106+0000 7f2027fff640 1 --2- 172.21.15.33:0/4263948015 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642c08a2350 0x5642c08a8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:32.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.106+0000 7f202f59e640 1 --2- 172.21.15.33:0/4263948015 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642c08aaf30 0x5642c08a88b0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f20280ccc00 tx=0x7f202809c210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e98177f23fd1439b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:32.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.106+0000 7f20167fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:56:32.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.106+0000 7f20167fc640 1 -- 172.21.15.33:0/4263948015 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20280b3020 con 0x5642c08aaf30 2024-03-19T02:56:32.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.106+0000 7f20167fc640 1 -- 172.21.15.33:0/4263948015 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f2028005310 con 0x5642c08aaf30 2024-03-19T02:56:32.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.106+0000 7f2037daa740 1 -- 172.21.15.33:0/4263948015 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2020009e20 con 0x5642c08aaf30 2024-03-19T02:56:32.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.106+0000 7f20167fc640 1 -- 172.21.15.33:0/4263948015 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f202809fc60 con 0x5642c08aaf30 2024-03-19T02:56:32.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.106+0000 7f2037daa740 10 client.16278 fetch_fsmap learned FSMap version 25 2024-03-19T02:56:32.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.106+0000 7f2037daa740 1 -- 172.21.15.33:0/4263948015 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5642c05193f0 con 0x5642c08aaf30 2024-03-19T02:56:32.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.107+0000 7f20167fc640 1 -- 172.21.15.33:0/4263948015 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2020009e20 con 0x5642c08aaf30 2024-03-19T02:56:32.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.107+0000 7f20167fc640 1 -- 172.21.15.33:0/4263948015 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f20280b4bd0 con 0x5642c08aaf30 2024-03-19T02:56:32.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.107+0000 7f2037daa740 10 client.16278 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:56:32.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.107+0000 7f2037daa740 10 client.16278 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:56:32.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.107+0000 7f2037daa740 20 client.16278 populate_metadata read hostname 'smithi033' 2024-03-19T02:56:32.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.107+0000 7f2037daa740 1 --2- 172.21.15.33:0/4263948015 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5642c08bb180 0x5642c08bd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.107+0000 7f2037daa740 4 client.16278 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:56:32.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.107+0000 7f2037daa740 1 -- 172.21.15.33:0/4263948015 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5642c0812a40 con 0x5642c08bb180 2024-03-19T02:56:32.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.107+0000 7f202ed9d640 1 --2- 172.21.15.33:0/4263948015 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5642c08bb180 0x5642c08bd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.107+0000 7f20167fc640 10 client.16278 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:56:32.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.107+0000 7f202ed9d640 1 --2- 172.21.15.33:0/4263948015 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5642c08bb180 0x5642c08bd5e0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f87d993a19d095f server_cookie=9cd938f07ef200b in_seq=0 out_seq=0 2024-03-19T02:56:32.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.108+0000 7f20167fc640 1 -- 172.21.15.33:0/4263948015 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5642c0812a40 con 0x5642c08bb180 2024-03-19T02:56:32.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.108+0000 7f20167fc640 10 client.16278 handle_command_reply: tid=0 2024-03-19T02:56:32.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.108+0000 7f2037daa740 1 client.16278 shutdown 2024-03-19T02:56:32.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.108+0000 7f2037daa740 1 -- 172.21.15.33:0/4263948015 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642c08aaf30 msgr2=0x5642c08a88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.108+0000 7f2037daa740 1 --2- 172.21.15.33:0/4263948015 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642c08aaf30 0x5642c08a88b0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f20280ccc00 tx=0x7f202809c210 comp rx=0 tx=0).stop 2024-03-19T02:56:32.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.108+0000 7f2037daa740 1 -- 172.21.15.33:0/4263948015 shutdown_connections 2024-03-19T02:56:32.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.108+0000 7f2037daa740 1 --2- 172.21.15.33:0/4263948015 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5642c08bb180 0x5642c08bd5e0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.108+0000 7f2037daa740 1 --2- 172.21.15.33:0/4263948015 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642c08aaf30 0x5642c08a88b0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.108+0000 7f2037daa740 1 --2- 172.21.15.33:0/4263948015 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642c08a2350 0x5642c08a8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.108+0000 7f20167fc640 0 client.16278 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:56:32.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.108+0000 7f2037daa740 1 --2- 172.21.15.33:0/4263948015 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642c08a1a30 0x5642c08a7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.108+0000 7f2037daa740 1 -- 172.21.15.33:0/4263948015 >> 172.21.15.33:0/4263948015 conn(0x5642c089c160 msgr2=0x5642c089deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:32.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.108+0000 7f2037daa740 1 -- 172.21.15.33:0/4263948015 shutdown_connections 2024-03-19T02:56:32.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.108+0000 7f2037daa740 1 -- 172.21.15.33:0/4263948015 wait complete. 2024-03-19T02:56:32.126 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.109+0000 7f2037daa740 20 client.16278 trim_cache size 0 max 16384 2024-03-19T02:56:32.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.126+0000 7f2037daa740 1 Processor -- start 2024-03-19T02:56:32.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.126+0000 7f2037daa740 1 -- start start 2024-03-19T02:56:32.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.126+0000 7f2037daa740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642c08a1a30 0x5642c08aa560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.126+0000 7f2037daa740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642c08a2350 0x5642c08b6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.126+0000 7f2037daa740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642c08aaf30 0x5642c08a6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.126+0000 7f2037daa740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f20301b3620 con 0x5642c08a1a30 2024-03-19T02:56:32.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.126+0000 7f2037daa740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f20301b0ef0 con 0x5642c08aaf30 2024-03-19T02:56:32.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.126+0000 7f2037daa740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f20301aed20 con 0x5642c08a2350 2024-03-19T02:56:32.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.127+0000 7f202ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642c08a1a30 0x5642c08aa560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.127+0000 7f202ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642c08a1a30 0x5642c08aa560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42604/0 (socket says 172.21.15.33:42604) 2024-03-19T02:56:32.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.127+0000 7f202ed9d640 1 -- 172.21.15.33:0/3596017383 learned_addr learned my addr 172.21.15.33:0/3596017383 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:32.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.127+0000 7f2027fff640 1 --2- 172.21.15.33:0/3596017383 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642c08a2350 0x5642c08b6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.127+0000 7f202f59e640 1 --2- 172.21.15.33:0/3596017383 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642c08aaf30 0x5642c08a6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.127+0000 7f202ed9d640 1 -- 172.21.15.33:0/3596017383 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642c08a2350 msgr2=0x5642c08b6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:32.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.127+0000 7f202ed9d640 1 --2- 172.21.15.33:0/3596017383 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642c08a2350 0x5642c08b6ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.127+0000 7f202ed9d640 1 -- 172.21.15.33:0/3596017383 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642c08aaf30 msgr2=0x5642c08a6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:32.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.127+0000 7f202ed9d640 1 --2- 172.21.15.33:0/3596017383 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642c08aaf30 0x5642c08a6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.127+0000 7f202ed9d640 1 -- 172.21.15.33:0/3596017383 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642c04d90f0 con 0x5642c08a1a30 2024-03-19T02:56:32.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.127+0000 7f202ed9d640 1 --2- 172.21.15.33:0/3596017383 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642c08a1a30 0x5642c08aa560 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f202000a010 tx=0x7f2020003e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67da8e6cdc1063a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:32.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.127+0000 7f2024ff9640 1 -- 172.21.15.33:0/3596017383 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2020009e20 con 0x5642c08a1a30 2024-03-19T02:56:32.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.127+0000 7f2024ff9640 1 -- 172.21.15.33:0/3596017383 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f20200035f0 con 0x5642c08a1a30 2024-03-19T02:56:32.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.127+0000 7f2024ff9640 1 -- 172.21.15.33:0/3596017383 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f202001a680 con 0x5642c08a1a30 2024-03-19T02:56:32.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.128+0000 7f2037daa740 1 -- 172.21.15.33:0/3596017383 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642c08a1a30 msgr2=0x5642c08aa560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.128+0000 7f2037daa740 1 --2- 172.21.15.33:0/3596017383 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642c08a1a30 0x5642c08aa560 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f202000a010 tx=0x7f2020003e70 comp rx=0 tx=0).stop 2024-03-19T02:56:32.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.128+0000 7f2037daa740 1 -- 172.21.15.33:0/3596017383 shutdown_connections 2024-03-19T02:56:32.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.128+0000 7f2037daa740 1 --2- 172.21.15.33:0/3596017383 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642c08a2350 0x5642c08b6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.128+0000 7f2037daa740 1 --2- 172.21.15.33:0/3596017383 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642c08aaf30 0x5642c08a6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.128+0000 7f2037daa740 1 --2- 172.21.15.33:0/3596017383 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642c08a1a30 0x5642c08aa560 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.128+0000 7f2037daa740 1 -- 172.21.15.33:0/3596017383 >> 172.21.15.33:0/3596017383 conn(0x5642c089c160 msgr2=0x5642c08a7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:32.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.128+0000 7f2037daa740 1 -- 172.21.15.33:0/3596017383 shutdown_connections 2024-03-19T02:56:32.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.128+0000 7f2037daa740 1 -- 172.21.15.33:0/3596017383 wait complete. 2024-03-19T02:56:32.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.128+0000 7f2037daa740 1 -- start start 2024-03-19T02:56:32.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.128+0000 7f2037daa740 1 Processor -- start 2024-03-19T02:56:32.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.129+0000 7f2037daa740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642c08a1a30 0x5642c08b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.129+0000 7f2037daa740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642c08a2350 0x5642c08ae4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.129+0000 7f2037daa740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642c08aaf30 0x5642c08aea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.129+0000 7f2037daa740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f20301af3b0 con 0x5642c08a1a30 2024-03-19T02:56:32.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.129+0000 7f2037daa740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f20301b0820 con 0x5642c08a2350 2024-03-19T02:56:32.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.129+0000 7f2037daa740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f20301afa80 con 0x5642c08aaf30 2024-03-19T02:56:32.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.129+0000 7f2037daa740 10 client.0 _ll_register_callbacks cb 0x7f202809b650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:56:32.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.129+0000 7f202ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642c08a1a30 0x5642c08b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.129+0000 7f202ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642c08a1a30 0x5642c08b3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42616/0 (socket says 172.21.15.33:42616) 2024-03-19T02:56:32.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.129+0000 7f202ed9d640 1 -- 172.21.15.33:0/72669209 learned_addr learned my addr 172.21.15.33:0/72669209 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:32.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.129+0000 7f202f59e640 1 --2- 172.21.15.33:0/72669209 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642c08aaf30 0x5642c08aea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.129+0000 7f2027fff640 1 --2- 172.21.15.33:0/72669209 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642c08a2350 0x5642c08ae4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.129+0000 7f202ed9d640 1 -- 172.21.15.33:0/72669209 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642c08aaf30 msgr2=0x5642c08aea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:32.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.129+0000 7f202ed9d640 1 --2- 172.21.15.33:0/72669209 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642c08aaf30 0x5642c08aea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.129+0000 7f202ed9d640 1 -- 172.21.15.33:0/72669209 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642c08a2350 msgr2=0x5642c08ae4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:32.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.129+0000 7f202ed9d640 1 --2- 172.21.15.33:0/72669209 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642c08a2350 0x5642c08ae4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.129+0000 7f202ed9d640 1 -- 172.21.15.33:0/72669209 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642c08aef70 con 0x5642c08a1a30 2024-03-19T02:56:32.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.129+0000 7f202ed9d640 1 --2- 172.21.15.33:0/72669209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642c08a1a30 0x5642c08b3120 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f20200093b0 tx=0x7f2020005af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1612b9abf6a5311 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:32.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.129+0000 7f20167fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:56:32.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.129+0000 7f20167fc640 1 -- 172.21.15.33:0/72669209 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2020009e20 con 0x5642c08a1a30 2024-03-19T02:56:32.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.129+0000 7f20167fc640 1 -- 172.21.15.33:0/72669209 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f20200052c0 con 0x5642c08a1a30 2024-03-19T02:56:32.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.129+0000 7f2037daa740 1 -- 172.21.15.33:0/72669209 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20200035f0 con 0x5642c08a1a30 2024-03-19T02:56:32.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.129+0000 7f20167fc640 1 -- 172.21.15.33:0/72669209 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f202003e050 con 0x5642c08a1a30 2024-03-19T02:56:32.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.129+0000 7f2037daa740 10 client.16284 fetch_fsmap learned FSMap version 25 2024-03-19T02:56:32.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.129+0000 7f2037daa740 1 -- 172.21.15.33:0/72669209 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5642c0921270 con 0x5642c08a1a30 2024-03-19T02:56:32.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.130+0000 7f20167fc640 1 -- 172.21.15.33:0/72669209 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f20200035f0 con 0x5642c08a1a30 2024-03-19T02:56:32.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.130+0000 7f20167fc640 1 -- 172.21.15.33:0/72669209 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f2020018070 con 0x5642c08a1a30 2024-03-19T02:56:32.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.130+0000 7f2037daa740 10 client.16284 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:56:32.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.130+0000 7f2037daa740 10 client.16284 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:56:32.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.130+0000 7f2037daa740 20 client.16284 populate_metadata read hostname 'smithi033' 2024-03-19T02:56:32.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.130+0000 7f2037daa740 1 --2- 172.21.15.33:0/72669209 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5642c0921870 0x5642c0923c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.130+0000 7f2037daa740 4 client.16284 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:56:32.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.130+0000 7f2037daa740 1 -- 172.21.15.33:0/72669209 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5642c080cd90 con 0x5642c0921870 2024-03-19T02:56:32.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.130+0000 7f2027fff640 1 --2- 172.21.15.33:0/72669209 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5642c0921870 0x5642c0923c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.130+0000 7f20167fc640 10 client.16284 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:56:32.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.130+0000 7f2027fff640 1 --2- 172.21.15.33:0/72669209 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5642c0921870 0x5642c0923c90 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fdd1e4d5c6c97e5a server_cookie=81633e70afa5ee48 in_seq=0 out_seq=0 2024-03-19T02:56:32.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.131+0000 7f20167fc640 1 -- 172.21.15.33:0/72669209 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5642c080cd90 con 0x5642c0921870 2024-03-19T02:56:32.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.131+0000 7f20167fc640 10 client.16284 handle_command_reply: tid=0 2024-03-19T02:56:32.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.131+0000 7f2037daa740 1 client.16284 shutdown 2024-03-19T02:56:32.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.131+0000 7f2037daa740 1 -- 172.21.15.33:0/72669209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642c08a1a30 msgr2=0x5642c08b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.131+0000 7f2037daa740 1 --2- 172.21.15.33:0/72669209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642c08a1a30 0x5642c08b3120 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f20200093b0 tx=0x7f2020005af0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.131+0000 7f2037daa740 1 -- 172.21.15.33:0/72669209 shutdown_connections 2024-03-19T02:56:32.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.131+0000 7f2037daa740 1 --2- 172.21.15.33:0/72669209 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5642c0921870 0x5642c0923c90 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.131+0000 7f2037daa740 1 --2- 172.21.15.33:0/72669209 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5642c08aaf30 0x5642c08aea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.131+0000 7f20167fc640 0 client.16284 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:56:32.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.131+0000 7f2037daa740 1 --2- 172.21.15.33:0/72669209 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5642c08a2350 0x5642c08ae4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.131+0000 7f2037daa740 1 --2- 172.21.15.33:0/72669209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5642c08a1a30 0x5642c08b3120 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.131+0000 7f2037daa740 1 -- 172.21.15.33:0/72669209 >> 172.21.15.33:0/72669209 conn(0x5642c089c160 msgr2=0x5642c08a6e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:32.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.131+0000 7f2037daa740 1 -- 172.21.15.33:0/72669209 shutdown_connections 2024-03-19T02:56:32.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.131+0000 7f2037daa740 1 -- 172.21.15.33:0/72669209 wait complete. 2024-03-19T02:56:32.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.131+0000 7f2037daa740 20 client.16284 trim_cache size 0 max 16384 2024-03-19T02:56:32.142 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:56:32.142 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-03-19T02:56:32.142 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-03-19T02:56:32.142 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:56:32.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.131+0000 7f20350e8640 1 -- 172.21.15.33:0/1471545374 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f201c0a29f0 msgr2=0x7f201c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.131+0000 7f20350e8640 1 --2- 172.21.15.33:0/1471545374 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f201c0a29f0 0x7f201c0a4eb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2018005ab0 tx=0x7f201802c6d0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.131+0000 7f20350e8640 1 -- 172.21.15.33:0/1471545374 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20301077a0 msgr2=0x7f20300745a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.131+0000 7f20350e8640 1 --2- 172.21.15.33:0/1471545374 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20301077a0 0x7f20300745a0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f202800eb10 tx=0x7f202800c2d0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.131+0000 7f20350e8640 1 -- 172.21.15.33:0/1471545374 shutdown_connections 2024-03-19T02:56:32.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.131+0000 7f20350e8640 1 --2- 172.21.15.33:0/1471545374 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f201c0a29f0 0x7f201c0a4eb0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.131+0000 7f20350e8640 1 --2- 172.21.15.33:0/1471545374 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f20301077a0 0x7f20300745a0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.131+0000 7f20350e8640 1 --2- 172.21.15.33:0/1471545374 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2030102150 0x7f2030074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.131+0000 7f20350e8640 1 --2- 172.21.15.33:0/1471545374 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2030101790 0x7f2030073b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.131+0000 7f20350e8640 1 -- 172.21.15.33:0/1471545374 >> 172.21.15.33:0/1471545374 conn(0x7f20300fd280 msgr2=0x7f20300fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:32.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.132+0000 7f20350e8640 1 -- 172.21.15.33:0/1471545374 shutdown_connections 2024-03-19T02:56:32.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.132+0000 7f20350e8640 1 -- 172.21.15.33:0/1471545374 wait complete. 2024-03-19T02:56:32.145 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:56:32.145 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:56:32.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.247+0000 7fe40064b640 1 -- 172.21.15.33:0/2206374099 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3f81077f0 msgr2=0x7fe3f8107bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.247+0000 7fe40064b640 1 --2- 172.21.15.33:0/2206374099 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3f81077f0 0x7fe3f8107bd0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4009930 tx=0x7fe3f402bc30 comp rx=0 tx=0).stop 2024-03-19T02:56:32.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.247+0000 7fe40064b640 1 -- 172.21.15.33:0/2206374099 shutdown_connections 2024-03-19T02:56:32.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.247+0000 7fe40064b640 1 --2- 172.21.15.33:0/2206374099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3f81021a0 0x7fe3f810e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.247+0000 7fe40064b640 1 --2- 172.21.15.33:0/2206374099 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3f81017e0 0x7fe3f8101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.247+0000 7fe40064b640 1 --2- 172.21.15.33:0/2206374099 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3f81077f0 0x7fe3f8107bd0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.247+0000 7fe40064b640 1 -- 172.21.15.33:0/2206374099 >> 172.21.15.33:0/2206374099 conn(0x7fe3f80fd290 msgr2=0x7fe3f80ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:32.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.247+0000 7fe40064b640 1 -- 172.21.15.33:0/2206374099 shutdown_connections 2024-03-19T02:56:32.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.247+0000 7fe40064b640 1 -- 172.21.15.33:0/2206374099 wait complete. 2024-03-19T02:56:32.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.248+0000 7fe40064b640 1 Processor -- start 2024-03-19T02:56:32.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.248+0000 7fe40064b640 1 -- start start 2024-03-19T02:56:32.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.248+0000 7fe40064b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3f81017e0 0x7fe3f8198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.248+0000 7fe40064b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3f81021a0 0x7fe3f81985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.248+0000 7fe40064b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3f81077f0 0x7fe3f819c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.248+0000 7fe40064b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f8111a70 con 0x7fe3f81077f0 2024-03-19T02:56:32.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.248+0000 7fe40064b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f81118f0 con 0x7fe3f81021a0 2024-03-19T02:56:32.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.248+0000 7fe40064b640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f8111bf0 con 0x7fe3f81017e0 2024-03-19T02:56:32.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.248+0000 7fe3febe8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3f81077f0 0x7fe3f819c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.248+0000 7fe3febe8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3f81077f0 0x7fe3f819c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42624/0 (socket says 172.21.15.33:42624) 2024-03-19T02:56:32.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.248+0000 7fe3febe8640 1 -- 172.21.15.33:0/1435450837 learned_addr learned my addr 172.21.15.33:0/1435450837 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:32.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.248+0000 7fe3fdbe6640 1 --2- 172.21.15.33:0/1435450837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3f81021a0 0x7fe3f81985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.248+0000 7fe3fe3e7640 1 --2- 172.21.15.33:0/1435450837 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3f81017e0 0x7fe3f8198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.248+0000 7fe3febe8640 1 -- 172.21.15.33:0/1435450837 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3f81017e0 msgr2=0x7fe3f8198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.248+0000 7fe3febe8640 1 --2- 172.21.15.33:0/1435450837 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3f81017e0 0x7fe3f8198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.248+0000 7fe3febe8640 1 -- 172.21.15.33:0/1435450837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3f81021a0 msgr2=0x7fe3f81985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.248+0000 7fe3febe8640 1 --2- 172.21.15.33:0/1435450837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3f81021a0 0x7fe3f81985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.248+0000 7fe3febe8640 1 -- 172.21.15.33:0/1435450837 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3f819d080 con 0x7fe3f81077f0 2024-03-19T02:56:32.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.248+0000 7fe3fdbe6640 1 --2- 172.21.15.33:0/1435450837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3f81021a0 0x7fe3f81985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:32.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.249+0000 7fe3febe8640 1 --2- 172.21.15.33:0/1435450837 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3f81077f0 0x7fe3f819c950 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000efd0 tx=0x7fe3f000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:32.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.249+0000 7fe3ef7fe640 1 -- 172.21.15.33:0/1435450837 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0003b70 con 0x7fe3f81077f0 2024-03-19T02:56:32.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.249+0000 7fe3ef7fe640 1 -- 172.21.15.33:0/1435450837 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe3f0003d10 con 0x7fe3f81077f0 2024-03-19T02:56:32.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.249+0000 7fe40064b640 1 -- 172.21.15.33:0/1435450837 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3f819d370 con 0x7fe3f81077f0 2024-03-19T02:56:32.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.249+0000 7fe3ef7fe640 1 -- 172.21.15.33:0/1435450837 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0010980 con 0x7fe3f81077f0 2024-03-19T02:56:32.253 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.249+0000 7fe40064b640 1 -- 172.21.15.33:0/1435450837 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3f81a4c00 con 0x7fe3f81077f0 2024-03-19T02:56:32.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.250+0000 7fe40064b640 1 -- 172.21.15.33:0/1435450837 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3f8068a60 con 0x7fe3f81077f0 2024-03-19T02:56:32.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.251+0000 7fe3ef7fe640 1 -- 172.21.15.33:0/1435450837 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fe3f0004020 con 0x7fe3f81077f0 2024-03-19T02:56:32.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.251+0000 7fe3ef7fe640 1 --2- 172.21.15.33:0/1435450837 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe3e40a2970 0x7fe3e40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.251+0000 7fe3fe3e7640 1 --2- 172.21.15.33:0/1435450837 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe3e40a2970 0x7fe3e40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.252+0000 7fe3ef7fe640 1 -- 172.21.15.33:0/1435450837 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fe3f00cf5f0 con 0x7fe3f81077f0 2024-03-19T02:56:32.254 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.252+0000 7fe3fe3e7640 1 --2- 172.21.15.33:0/1435450837 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe3e40a2970 0x7fe3e40a4e30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4005ab0 tx=0x7fe3f4005fd0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:32.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.256+0000 7fe3ef7fe640 1 -- 172.21.15.33:0/1435450837 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe3f009bfe0 con 0x7fe3f81077f0 2024-03-19T02:56:32.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:31 smithi155 ceph-mon[35804]: pgmap v95: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-03-19T02:56:32.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:31 smithi033 ceph-mon[31973]: pgmap v95: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-03-19T02:56:32.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:31 smithi096 ceph-mon[35946]: pgmap v95: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 5 op/s 2024-03-19T02:56:32.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.477+0000 7fe40064b640 1 -- 172.21.15.33:0/1435450837 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3f8109db0 con 0x7fe3f81077f0 2024-03-19T02:56:32.479 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.478+0000 7fe3ef7fe640 1 -- 172.21.15.33:0/1435450837 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11867 (secure 0 0 0) 0x7fe3f009b8e0 con 0x7fe3f81077f0 2024-03-19T02:56:32.479 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:56:32.479 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:50:52.201798+0000","modified":"2024-03-19T02:53:11.748697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24488,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/3731232169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":3731232169},{"type":"v1","addr":"172.21.15.96:6839","nonce":3731232169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24457,24488,24509]},"id":1}]} 2024-03-19T02:56:32.480 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-03-19T02:56:32.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.479+0000 7fe40064b640 1 -- 172.21.15.33:0/1435450837 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe3e40a2970 msgr2=0x7fe3e40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.480+0000 7fe40064b640 1 --2- 172.21.15.33:0/1435450837 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe3e40a2970 0x7fe3e40a4e30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4005ab0 tx=0x7fe3f4005fd0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.480+0000 7fe40064b640 1 -- 172.21.15.33:0/1435450837 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3f81077f0 msgr2=0x7fe3f819c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.480+0000 7fe40064b640 1 --2- 172.21.15.33:0/1435450837 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3f81077f0 0x7fe3f819c950 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000efd0 tx=0x7fe3f000c6a0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.480+0000 7fe40064b640 1 -- 172.21.15.33:0/1435450837 shutdown_connections 2024-03-19T02:56:32.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.480+0000 7fe40064b640 1 --2- 172.21.15.33:0/1435450837 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe3e40a2970 0x7fe3e40a4e30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.480+0000 7fe40064b640 1 --2- 172.21.15.33:0/1435450837 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3f81077f0 0x7fe3f819c950 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.480+0000 7fe40064b640 1 --2- 172.21.15.33:0/1435450837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3f81021a0 0x7fe3f81985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.480+0000 7fe40064b640 1 --2- 172.21.15.33:0/1435450837 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3f81017e0 0x7fe3f8198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.481 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.480+0000 7fe40064b640 1 -- 172.21.15.33:0/1435450837 >> 172.21.15.33:0/1435450837 conn(0x7fe3f80fd290 msgr2=0x7fe3f80fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:32.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.480+0000 7fe40064b640 1 -- 172.21.15.33:0/1435450837 shutdown_connections 2024-03-19T02:56:32.482 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.480+0000 7fe40064b640 1 -- 172.21.15.33:0/1435450837 wait complete. 2024-03-19T02:56:32.492 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:56:32.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.596+0000 7efe71836640 1 -- 172.21.15.33:0/58645721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe6c101060 msgr2=0x7efe6c10d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.596+0000 7efe71836640 1 --2- 172.21.15.33:0/58645721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe6c101060 0x7efe6c10d570 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7efe64009cd0 tx=0x7efe6402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.596+0000 7efe71836640 1 -- 172.21.15.33:0/58645721 shutdown_connections 2024-03-19T02:56:32.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.596+0000 7efe71836640 1 --2- 172.21.15.33:0/58645721 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe6c1006a0 0x7efe6c100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.596+0000 7efe71836640 1 --2- 172.21.15.33:0/58645721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe6c101060 0x7efe6c10d570 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.596+0000 7efe71836640 1 --2- 172.21.15.33:0/58645721 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe6c1066b0 0x7efe6c106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.596+0000 7efe71836640 1 -- 172.21.15.33:0/58645721 >> 172.21.15.33:0/58645721 conn(0x7efe6c0778d0 msgr2=0x7efe6c077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:32.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.596+0000 7efe71836640 1 -- 172.21.15.33:0/58645721 shutdown_connections 2024-03-19T02:56:32.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.596+0000 7efe71836640 1 -- 172.21.15.33:0/58645721 wait complete. 2024-03-19T02:56:32.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.596+0000 7efe71836640 1 Processor -- start 2024-03-19T02:56:32.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.597+0000 7efe71836640 1 -- start start 2024-03-19T02:56:32.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.597+0000 7efe71836640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe6c1006a0 0x7efe6c196c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.597+0000 7efe71836640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe6c101060 0x7efe6c197140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.597+0000 7efe71836640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe6c1066b0 0x7efe6c197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.597+0000 7efe71836640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efe6c110970 con 0x7efe6c1006a0 2024-03-19T02:56:32.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.597+0000 7efe71836640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efe6c1107f0 con 0x7efe6c1066b0 2024-03-19T02:56:32.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.597+0000 7efe71836640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7efe6c110af0 con 0x7efe6c101060 2024-03-19T02:56:32.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.597+0000 7efe6affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe6c1006a0 0x7efe6c196c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.597+0000 7efe6affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe6c1006a0 0x7efe6c196c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42652/0 (socket says 172.21.15.33:42652) 2024-03-19T02:56:32.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.597+0000 7efe6affd640 1 -- 172.21.15.33:0/1288289361 learned_addr learned my addr 172.21.15.33:0/1288289361 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:32.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.597+0000 7efe63fff640 1 --2- 172.21.15.33:0/1288289361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe6c101060 0x7efe6c197140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.597+0000 7efe6b7fe640 1 --2- 172.21.15.33:0/1288289361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe6c1066b0 0x7efe6c197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.597+0000 7efe6affd640 1 -- 172.21.15.33:0/1288289361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe6c101060 msgr2=0x7efe6c197140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.597+0000 7efe6affd640 1 --2- 172.21.15.33:0/1288289361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe6c101060 0x7efe6c197140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.597+0000 7efe6affd640 1 -- 172.21.15.33:0/1288289361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe6c1066b0 msgr2=0x7efe6c197680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.597+0000 7efe6affd640 1 --2- 172.21.15.33:0/1288289361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe6c1066b0 0x7efe6c197680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.597+0000 7efe6affd640 1 -- 172.21.15.33:0/1288289361 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe6c1a6470 con 0x7efe6c1006a0 2024-03-19T02:56:32.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.597+0000 7efe63fff640 1 --2- 172.21.15.33:0/1288289361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe6c101060 0x7efe6c197140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:32.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.597+0000 7efe6affd640 1 --2- 172.21.15.33:0/1288289361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe6c1006a0 0x7efe6c196c00 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7efe5c00ea60 tx=0x7efe5c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:32.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.597+0000 7efe70834640 1 -- 172.21.15.33:0/1288289361 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5c00cde0 con 0x7efe6c1006a0 2024-03-19T02:56:32.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.597+0000 7efe70834640 1 -- 172.21.15.33:0/1288289361 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe5c0035f0 con 0x7efe6c1006a0 2024-03-19T02:56:32.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.597+0000 7efe71836640 1 -- 172.21.15.33:0/1288289361 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe6c1a6760 con 0x7efe6c1006a0 2024-03-19T02:56:32.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.598+0000 7efe70834640 1 -- 172.21.15.33:0/1288289361 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5c010690 con 0x7efe6c1006a0 2024-03-19T02:56:32.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.598+0000 7efe71836640 1 -- 172.21.15.33:0/1288289361 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe6c1a6c80 con 0x7efe6c1006a0 2024-03-19T02:56:32.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.600+0000 7efe70834640 1 -- 172.21.15.33:0/1288289361 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7efe5c0037a0 con 0x7efe6c1006a0 2024-03-19T02:56:32.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.600+0000 7efe74428740 1 Processor -- start 2024-03-19T02:56:32.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.600+0000 7efe74428740 1 -- start start 2024-03-19T02:56:32.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.600+0000 7efe74428740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fcceb3d2d0 0x55fcceb3d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.600+0000 7efe74428740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcceb3dbf0 0x55fcceb46290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.600+0000 7efe74428740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fcceb467d0 0x55fcceb48bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.608 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.600+0000 7efe74428740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55fcceaa04b0 con 0x55fcceb467d0 2024-03-19T02:56:32.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.600+0000 7efe74428740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55fccea48d70 con 0x55fcceb3d2d0 2024-03-19T02:56:32.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.600+0000 7efe74428740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55fccea9c550 con 0x55fcceb3dbf0 2024-03-19T02:56:32.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.600+0000 7efe70834640 1 --2- 172.21.15.33:0/1288289361 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7efe580a2970 0x7efe580a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.600+0000 7efe63fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fcceb467d0 0x55fcceb48bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.600+0000 7efe6b7fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fcceb3d2d0 0x55fcceb3d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.600+0000 7efe6b7fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fcceb3d2d0 0x55fcceb3d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:49902/0 (socket says 172.21.15.33:49902) 2024-03-19T02:56:32.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.600+0000 7efe6b7fe640 1 -- 172.21.15.33:0/895759310 learned_addr learned my addr 172.21.15.33:0/895759310 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:32.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.600+0000 7efe63fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fcceb467d0 0x55fcceb48bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42662/0 (socket says 172.21.15.33:42662) 2024-03-19T02:56:32.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.600+0000 7efe6affd640 1 --2- 172.21.15.33:0/895759310 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcceb3dbf0 0x55fcceb46290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.601+0000 7efe6b7fe640 1 --2- 172.21.15.33:0/1288289361 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7efe580a2970 0x7efe580a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.601+0000 7efe63fff640 1 -- 172.21.15.33:0/895759310 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcceb3dbf0 msgr2=0x55fcceb46290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:32.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.601+0000 7efe63fff640 1 --2- 172.21.15.33:0/895759310 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcceb3dbf0 0x55fcceb46290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.601+0000 7efe70834640 1 -- 172.21.15.33:0/1288289361 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7efe5c0cf3e0 con 0x7efe6c1006a0 2024-03-19T02:56:32.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.601+0000 7efe63fff640 1 -- 172.21.15.33:0/895759310 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fcceb3d2d0 msgr2=0x55fcceb3d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:32.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.601+0000 7efe63fff640 1 --2- 172.21.15.33:0/895759310 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fcceb3d2d0 0x55fcceb3d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.601+0000 7efe63fff640 1 -- 172.21.15.33:0/895759310 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fccea6b210 con 0x55fcceb467d0 2024-03-19T02:56:32.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.601+0000 7efe6b7fe640 1 --2- 172.21.15.33:0/895759310 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fcceb3d2d0 0x55fcceb3d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:32.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.601+0000 7efe63fff640 1 --2- 172.21.15.33:0/895759310 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fcceb467d0 0x55fcceb48bc0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7efe54009fa0 tx=0x7efe540118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9d0aa3bbd715942 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:32.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.601+0000 7efe6b7fe640 1 --2- 172.21.15.33:0/1288289361 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7efe580a2970 0x7efe580a4e30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7efe64009a60 tx=0x7efe6402c0f0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:32.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.601+0000 7efe61ffb640 1 -- 172.21.15.33:0/895759310 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe54004310 con 0x55fcceb467d0 2024-03-19T02:56:32.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.601+0000 7efe61ffb640 1 -- 172.21.15.33:0/895759310 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe540044b0 con 0x55fcceb467d0 2024-03-19T02:56:32.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.601+0000 7efe61ffb640 1 -- 172.21.15.33:0/895759310 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5400f9b0 con 0x55fcceb467d0 2024-03-19T02:56:32.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.602+0000 7efe74428740 1 -- 172.21.15.33:0/895759310 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fcceb467d0 msgr2=0x55fcceb48bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.602+0000 7efe74428740 1 --2- 172.21.15.33:0/895759310 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fcceb467d0 0x55fcceb48bc0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7efe54009fa0 tx=0x7efe540118d0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.602+0000 7efe74428740 1 -- 172.21.15.33:0/895759310 shutdown_connections 2024-03-19T02:56:32.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.602+0000 7efe74428740 1 --2- 172.21.15.33:0/895759310 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcceb3dbf0 0x55fcceb46290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.602+0000 7efe74428740 1 --2- 172.21.15.33:0/895759310 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fcceb467d0 0x55fcceb48bc0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.602+0000 7efe74428740 1 --2- 172.21.15.33:0/895759310 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fcceb3d2d0 0x55fcceb3d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.602+0000 7efe74428740 1 -- 172.21.15.33:0/895759310 >> 172.21.15.33:0/895759310 conn(0x55fcceb37a00 msgr2=0x55fcceb37de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:32.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.602+0000 7efe74428740 1 -- 172.21.15.33:0/895759310 shutdown_connections 2024-03-19T02:56:32.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.602+0000 7efe74428740 1 -- 172.21.15.33:0/895759310 wait complete. 2024-03-19T02:56:32.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.602+0000 7efe74428740 1 -- start start 2024-03-19T02:56:32.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.602+0000 7efe74428740 1 Processor -- start 2024-03-19T02:56:32.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.603+0000 7efe74428740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fcceb3d2d0 0x55fcceb43850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.603+0000 7efe74428740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fcceb3dbf0 0x55fcceb43d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.603+0000 7efe74428740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcceb467d0 0x55fcceb442d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.603+0000 7efe74428740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55fcceb495c0 con 0x55fcceb3d2d0 2024-03-19T02:56:32.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.603+0000 7efe74428740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55fccea5c310 con 0x55fcceb3dbf0 2024-03-19T02:56:32.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.603+0000 7efe74428740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55fcceb49740 con 0x55fcceb467d0 2024-03-19T02:56:32.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.603+0000 7efe74428740 10 client.0 _ll_register_callbacks cb 0x55fccea502d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:56:32.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.603+0000 7efe6affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fcceb3d2d0 0x55fcceb43850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.603+0000 7efe6affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fcceb3d2d0 0x55fcceb43850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42668/0 (socket says 172.21.15.33:42668) 2024-03-19T02:56:32.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.603+0000 7efe6affd640 1 -- 172.21.15.33:0/3255992302 learned_addr learned my addr 172.21.15.33:0/3255992302 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:32.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.603+0000 7efe63fff640 1 --2- 172.21.15.33:0/3255992302 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fcceb3dbf0 0x55fcceb43d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.603+0000 7efe6b7fe640 1 --2- 172.21.15.33:0/3255992302 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcceb467d0 0x55fcceb442d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.603+0000 7efe6affd640 1 -- 172.21.15.33:0/3255992302 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcceb467d0 msgr2=0x55fcceb442d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:32.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.603+0000 7efe6affd640 1 --2- 172.21.15.33:0/3255992302 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcceb467d0 0x55fcceb442d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.603+0000 7efe6affd640 1 -- 172.21.15.33:0/3255992302 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fcceb3dbf0 msgr2=0x55fcceb43d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:32.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.603+0000 7efe6affd640 1 --2- 172.21.15.33:0/3255992302 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fcceb3dbf0 0x55fcceb43d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.603+0000 7efe6affd640 1 -- 172.21.15.33:0/3255992302 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcce7b43f0 con 0x55fcceb3d2d0 2024-03-19T02:56:32.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.603+0000 7efe6affd640 1 --2- 172.21.15.33:0/3255992302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fcceb3d2d0 0x55fcceb43850 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7efe5c009920 tx=0x7efe5c09b210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e2b749fb53f5fea server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:32.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.603+0000 7efe4affd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:56:32.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.603+0000 7efe4affd640 1 -- 172.21.15.33:0/3255992302 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5c09bdd0 con 0x55fcceb3d2d0 2024-03-19T02:56:32.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.603+0000 7efe4affd640 1 -- 172.21.15.33:0/3255992302 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7efe5c086070 con 0x55fcceb3d2d0 2024-03-19T02:56:32.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.603+0000 7efe74428740 1 -- 172.21.15.33:0/3255992302 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe540044b0 con 0x55fcceb3d2d0 2024-03-19T02:56:32.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.604+0000 7efe4affd640 1 -- 172.21.15.33:0/3255992302 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7efe5c09b6d0 con 0x55fcceb3d2d0 2024-03-19T02:56:32.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.604+0000 7efe74428740 10 client.16302 fetch_fsmap learned FSMap version 25 2024-03-19T02:56:32.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.604+0000 7efe74428740 1 -- 172.21.15.33:0/3255992302 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55fcce87e7d0 con 0x55fcceb3d2d0 2024-03-19T02:56:32.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.604+0000 7efe4affd640 1 -- 172.21.15.33:0/3255992302 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe540044b0 con 0x55fcceb3d2d0 2024-03-19T02:56:32.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.604+0000 7efe4affd640 1 -- 172.21.15.33:0/3255992302 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7efe5c097d40 con 0x55fcceb3d2d0 2024-03-19T02:56:32.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.604+0000 7efe74428740 10 client.16302 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:56:32.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.604+0000 7efe74428740 10 client.16302 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T02:56:32.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.604+0000 7efe74428740 20 client.16302 populate_metadata read hostname 'smithi033' 2024-03-19T02:56:32.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.604+0000 7efe74428740 1 --2- 172.21.15.33:0/3255992302 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55fcceb56c00 0x55fcceb59090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.604+0000 7efe74428740 4 client.16302 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:56:32.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.604+0000 7efe74428740 1 -- 172.21.15.33:0/3255992302 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fcceafe3e0 con 0x55fcceb56c00 2024-03-19T02:56:32.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.604+0000 7efe63fff640 1 --2- 172.21.15.33:0/3255992302 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55fcceb56c00 0x55fcceb59090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.605+0000 7efe63fff640 1 --2- 172.21.15.33:0/3255992302 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55fcceb56c00 0x55fcceb59090 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6537ac173f2933b7 server_cookie=a98e4ab4d35e6f3 in_seq=0 out_seq=0 2024-03-19T02:56:32.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.605+0000 7efe4affd640 10 client.16302 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T02:56:32.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.606+0000 7efe4affd640 1 -- 172.21.15.33:0/3255992302 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fcceafe3e0 con 0x55fcceb56c00 2024-03-19T02:56:32.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.606+0000 7efe4affd640 10 client.16302 handle_command_reply: tid=0 2024-03-19T02:56:32.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.606+0000 7efe74428740 1 client.16302 shutdown 2024-03-19T02:56:32.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.606+0000 7efe74428740 1 -- 172.21.15.33:0/3255992302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fcceb3d2d0 msgr2=0x55fcceb43850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.606+0000 7efe74428740 1 --2- 172.21.15.33:0/3255992302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fcceb3d2d0 0x55fcceb43850 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7efe5c009920 tx=0x7efe5c09b210 comp rx=0 tx=0).stop 2024-03-19T02:56:32.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.606+0000 7efe74428740 1 -- 172.21.15.33:0/3255992302 shutdown_connections 2024-03-19T02:56:32.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.606+0000 7efe74428740 1 --2- 172.21.15.33:0/3255992302 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55fcceb56c00 0x55fcceb59090 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.606+0000 7efe74428740 1 --2- 172.21.15.33:0/3255992302 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcceb467d0 0x55fcceb442d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.606+0000 7efe74428740 1 --2- 172.21.15.33:0/3255992302 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fcceb3dbf0 0x55fcceb43d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.606+0000 7efe74428740 1 --2- 172.21.15.33:0/3255992302 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fcceb3d2d0 0x55fcceb43850 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.606+0000 7efe4affd640 0 client.16302 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T02:56:32.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.606+0000 7efe74428740 1 -- 172.21.15.33:0/3255992302 >> 172.21.15.33:0/3255992302 conn(0x55fcceb37a00 msgr2=0x55fcceb39750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:32.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.606+0000 7efe74428740 1 -- 172.21.15.33:0/3255992302 shutdown_connections 2024-03-19T02:56:32.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.606+0000 7efe74428740 1 -- 172.21.15.33:0/3255992302 wait complete. 2024-03-19T02:56:32.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.606+0000 7efe74428740 20 client.16302 trim_cache size 0 max 16384 2024-03-19T02:56:32.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.619+0000 7efe74428740 1 Processor -- start 2024-03-19T02:56:32.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.619+0000 7efe74428740 1 -- start start 2024-03-19T02:56:32.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.619+0000 7efe74428740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcceb3d2d0 0x55fcceb52360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.619+0000 7efe74428740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fcceb3dbf0 0x55fcceb440f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.619+0000 7efe74428740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fcceb467d0 0x55fcceb5e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.619+0000 7efe74428740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efe6c1b35d0 con 0x55fcceb3dbf0 2024-03-19T02:56:32.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.619+0000 7efe74428740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efe6c1b0ea0 con 0x55fcceb467d0 2024-03-19T02:56:32.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.619+0000 7efe74428740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7efe6c1aecd0 con 0x55fcceb3d2d0 2024-03-19T02:56:32.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.619+0000 7efe63fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fcceb3dbf0 0x55fcceb440f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.619+0000 7efe63fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fcceb3dbf0 0x55fcceb440f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42672/0 (socket says 172.21.15.33:42672) 2024-03-19T02:56:32.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.619+0000 7efe63fff640 1 -- 172.21.15.33:0/3996906686 learned_addr learned my addr 172.21.15.33:0/3996906686 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:32.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.619+0000 7efe6affd640 1 --2- 172.21.15.33:0/3996906686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcceb3d2d0 0x55fcceb52360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.619+0000 7efe6b7fe640 1 --2- 172.21.15.33:0/3996906686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fcceb467d0 0x55fcceb5e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.620+0000 7efe63fff640 1 -- 172.21.15.33:0/3996906686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcceb3d2d0 msgr2=0x55fcceb52360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:32.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.620+0000 7efe63fff640 1 --2- 172.21.15.33:0/3996906686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcceb3d2d0 0x55fcceb52360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.620+0000 7efe63fff640 1 -- 172.21.15.33:0/3996906686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fcceb467d0 msgr2=0x55fcceb5e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:32.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.620+0000 7efe63fff640 1 --2- 172.21.15.33:0/3996906686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fcceb467d0 0x55fcceb5e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.620+0000 7efe63fff640 1 -- 172.21.15.33:0/3996906686 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcceb49660 con 0x55fcceb3dbf0 2024-03-19T02:56:32.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.620+0000 7efe63fff640 1 --2- 172.21.15.33:0/3996906686 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fcceb3dbf0 0x55fcceb440f0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7efe54009250 tx=0x7efe54004200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4cec4297228a6bd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:32.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.620+0000 7efe617fa640 1 -- 172.21.15.33:0/3996906686 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5401d040 con 0x55fcceb3dbf0 2024-03-19T02:56:32.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.620+0000 7efe617fa640 1 -- 172.21.15.33:0/3996906686 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe5400f680 con 0x55fcceb3dbf0 2024-03-19T02:56:32.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.620+0000 7efe617fa640 1 -- 172.21.15.33:0/3996906686 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe54003850 con 0x55fcceb3dbf0 2024-03-19T02:56:32.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.621+0000 7efe74428740 1 -- 172.21.15.33:0/3996906686 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fcceb3dbf0 msgr2=0x55fcceb440f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.621+0000 7efe74428740 1 --2- 172.21.15.33:0/3996906686 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fcceb3dbf0 0x55fcceb440f0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7efe54009250 tx=0x7efe54004200 comp rx=0 tx=0).stop 2024-03-19T02:56:32.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.621+0000 7efe74428740 1 -- 172.21.15.33:0/3996906686 shutdown_connections 2024-03-19T02:56:32.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.621+0000 7efe74428740 1 --2- 172.21.15.33:0/3996906686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fcceb467d0 0x55fcceb5e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.621+0000 7efe74428740 1 --2- 172.21.15.33:0/3996906686 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fcceb3dbf0 0x55fcceb440f0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.621+0000 7efe74428740 1 --2- 172.21.15.33:0/3996906686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcceb3d2d0 0x55fcceb52360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.621+0000 7efe74428740 1 -- 172.21.15.33:0/3996906686 >> 172.21.15.33:0/3996906686 conn(0x55fcceb37a00 msgr2=0x55fcceb36ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:32.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.621+0000 7efe74428740 1 -- 172.21.15.33:0/3996906686 shutdown_connections 2024-03-19T02:56:32.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.621+0000 7efe74428740 1 -- 172.21.15.33:0/3996906686 wait complete. 2024-03-19T02:56:32.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.621+0000 7efe74428740 1 -- start start 2024-03-19T02:56:32.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.621+0000 7efe74428740 1 Processor -- start 2024-03-19T02:56:32.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.621+0000 7efe74428740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fcceb3d2d0 0x55fcceb4a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.621+0000 7efe74428740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fcceb3dbf0 0x55fcceb4a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.621+0000 7efe74428740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcceb467d0 0x55fcceb4adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.621+0000 7efe74428740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efe6c1af360 con 0x55fcceb3dbf0 2024-03-19T02:56:32.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.621+0000 7efe74428740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efe6c1b07d0 con 0x55fcceb3d2d0 2024-03-19T02:56:32.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.621+0000 7efe74428740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7efe6c1afa30 con 0x55fcceb467d0 2024-03-19T02:56:32.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.621+0000 7efe74428740 10 client.0 _ll_register_callbacks cb 0x7efe5c003d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:56:32.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.621+0000 7efe63fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fcceb3dbf0 0x55fcceb4a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.621+0000 7efe63fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fcceb3dbf0 0x55fcceb4a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42676/0 (socket says 172.21.15.33:42676) 2024-03-19T02:56:32.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.621+0000 7efe63fff640 1 -- 172.21.15.33:0/4180837041 learned_addr learned my addr 172.21.15.33:0/4180837041 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:32.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.621+0000 7efe6affd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fcceb3d2d0 0x55fcceb4a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.622+0000 7efe6b7fe640 1 --2- 172.21.15.33:0/4180837041 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcceb467d0 0x55fcceb4adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.622+0000 7efe63fff640 1 -- 172.21.15.33:0/4180837041 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcceb467d0 msgr2=0x55fcceb4adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:32.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.622+0000 7efe63fff640 1 --2- 172.21.15.33:0/4180837041 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcceb467d0 0x55fcceb4adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.622+0000 7efe63fff640 1 -- 172.21.15.33:0/4180837041 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fcceb3d2d0 msgr2=0x55fcceb4a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:32.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.622+0000 7efe63fff640 1 --2- 172.21.15.33:0/4180837041 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fcceb3d2d0 0x55fcceb4a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.622+0000 7efe63fff640 1 -- 172.21.15.33:0/4180837041 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fccebbc090 con 0x55fcceb3dbf0 2024-03-19T02:56:32.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.622+0000 7efe6affd640 1 --2- 172.21.15.33:0/4180837041 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fcceb3d2d0 0x55fcceb4a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:32.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.622+0000 7efe63fff640 1 --2- 172.21.15.33:0/4180837041 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fcceb3dbf0 0x55fcceb4a870 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7efe5401b040 tx=0x7efe54009420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffa1913512063251 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:32.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.622+0000 7efe4affd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:56:32.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.622+0000 7efe4affd640 1 -- 172.21.15.33:0/4180837041 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5401d030 con 0x55fcceb3dbf0 2024-03-19T02:56:32.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.622+0000 7efe4affd640 1 -- 172.21.15.33:0/4180837041 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7efe54005240 con 0x55fcceb3dbf0 2024-03-19T02:56:32.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.622+0000 7efe74428740 1 -- 172.21.15.33:0/4180837041 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe5400f680 con 0x55fcceb3dbf0 2024-03-19T02:56:32.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.622+0000 7efe4affd640 1 -- 172.21.15.33:0/4180837041 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7efe54004050 con 0x55fcceb3dbf0 2024-03-19T02:56:32.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.622+0000 7efe74428740 10 client.16308 fetch_fsmap learned FSMap version 25 2024-03-19T02:56:32.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.622+0000 7efe74428740 1 -- 172.21.15.33:0/4180837041 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55fccebbd370 con 0x55fcceb3dbf0 2024-03-19T02:56:32.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.622+0000 7efe4affd640 1 -- 172.21.15.33:0/4180837041 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe5400f680 con 0x55fcceb3dbf0 2024-03-19T02:56:32.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.623+0000 7efe4affd640 1 -- 172.21.15.33:0/4180837041 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7efe54003680 con 0x55fcceb3dbf0 2024-03-19T02:56:32.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.623+0000 7efe74428740 10 client.16308 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:56:32.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.623+0000 7efe74428740 10 client.16308 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T02:56:32.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.623+0000 7efe74428740 20 client.16308 populate_metadata read hostname 'smithi033' 2024-03-19T02:56:32.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.623+0000 7efe74428740 1 --2- 172.21.15.33:0/4180837041 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55fccebbd840 0x55fccebbfc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.623+0000 7efe74428740 4 client.16308 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:56:32.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.623+0000 7efe74428740 1 -- 172.21.15.33:0/4180837041 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fcceaad9f0 con 0x55fccebbd840 2024-03-19T02:56:32.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.623+0000 7efe6affd640 1 --2- 172.21.15.33:0/4180837041 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55fccebbd840 0x55fccebbfc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.623+0000 7efe4affd640 10 client.16308 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T02:56:32.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.623+0000 7efe6affd640 1 --2- 172.21.15.33:0/4180837041 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55fccebbd840 0x55fccebbfc60 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=47e7a501d266c749 server_cookie=bcb13232b7eb6a7c in_seq=0 out_seq=0 2024-03-19T02:56:32.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.623+0000 7efe4affd640 1 -- 172.21.15.33:0/4180837041 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fcceaad9f0 con 0x55fccebbd840 2024-03-19T02:56:32.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.623+0000 7efe4affd640 10 client.16308 handle_command_reply: tid=0 2024-03-19T02:56:32.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.623+0000 7efe74428740 1 client.16308 shutdown 2024-03-19T02:56:32.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.623+0000 7efe74428740 1 -- 172.21.15.33:0/4180837041 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fcceb3dbf0 msgr2=0x55fcceb4a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.623+0000 7efe74428740 1 --2- 172.21.15.33:0/4180837041 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fcceb3dbf0 0x55fcceb4a870 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7efe5401b040 tx=0x7efe54009420 comp rx=0 tx=0).stop 2024-03-19T02:56:32.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.623+0000 7efe74428740 1 -- 172.21.15.33:0/4180837041 shutdown_connections 2024-03-19T02:56:32.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.623+0000 7efe74428740 1 --2- 172.21.15.33:0/4180837041 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55fccebbd840 0x55fccebbfc60 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.623+0000 7efe74428740 1 --2- 172.21.15.33:0/4180837041 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fcceb467d0 0x55fcceb4adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.623+0000 7efe4affd640 0 client.16308 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T02:56:32.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.623+0000 7efe74428740 1 --2- 172.21.15.33:0/4180837041 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fcceb3dbf0 0x55fcceb4a870 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.623+0000 7efe74428740 1 --2- 172.21.15.33:0/4180837041 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fcceb3d2d0 0x55fcceb4a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.623+0000 7efe74428740 1 -- 172.21.15.33:0/4180837041 >> 172.21.15.33:0/4180837041 conn(0x55fcceb37a00 msgr2=0x55fcceb38bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:32.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.624+0000 7efe74428740 1 -- 172.21.15.33:0/4180837041 shutdown_connections 2024-03-19T02:56:32.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.624+0000 7efe74428740 1 -- 172.21.15.33:0/4180837041 wait complete. 2024-03-19T02:56:32.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.624+0000 7efe74428740 20 client.16308 trim_cache size 0 max 16384 2024-03-19T02:56:32.632 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:56:32.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.624+0000 7efe71836640 1 -- 172.21.15.33:0/1288289361 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7efe580a2970 msgr2=0x7efe580a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.624+0000 7efe71836640 1 --2- 172.21.15.33:0/1288289361 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7efe580a2970 0x7efe580a4e30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7efe64009a60 tx=0x7efe6402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.624+0000 7efe71836640 1 -- 172.21.15.33:0/1288289361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe6c1006a0 msgr2=0x7efe6c196c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.624+0000 7efe71836640 1 --2- 172.21.15.33:0/1288289361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe6c1006a0 0x7efe6c196c00 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7efe5c00ea60 tx=0x7efe5c00ef20 comp rx=0 tx=0).stop 2024-03-19T02:56:32.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.624+0000 7efe71836640 1 -- 172.21.15.33:0/1288289361 shutdown_connections 2024-03-19T02:56:32.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.624+0000 7efe71836640 1 --2- 172.21.15.33:0/1288289361 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7efe580a2970 0x7efe580a4e30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.624+0000 7efe71836640 1 --2- 172.21.15.33:0/1288289361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe6c101060 0x7efe6c197140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.624+0000 7efe71836640 1 --2- 172.21.15.33:0/1288289361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe6c1066b0 0x7efe6c197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.624+0000 7efe71836640 1 --2- 172.21.15.33:0/1288289361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efe6c1006a0 0x7efe6c196c00 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.624+0000 7efe71836640 1 -- 172.21.15.33:0/1288289361 >> 172.21.15.33:0/1288289361 conn(0x7efe6c0778d0 msgr2=0x7efe6c102df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:32.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.624+0000 7efe71836640 1 -- 172.21.15.33:0/1288289361 shutdown_connections 2024-03-19T02:56:32.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.624+0000 7efe71836640 1 -- 172.21.15.33:0/1288289361 wait complete. 2024-03-19T02:56:32.636 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:56:32.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.740+0000 7f7535b62640 1 -- 172.21.15.33:0/2402620897 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7530102160 msgr2=0x7f753010e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.740+0000 7f7535b62640 1 --2- 172.21.15.33:0/2402620897 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7530102160 0x7f753010e670 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7528009cd0 tx=0x7f752802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.740+0000 7f7535b62640 1 -- 172.21.15.33:0/2402620897 shutdown_connections 2024-03-19T02:56:32.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.740+0000 7f7535b62640 1 --2- 172.21.15.33:0/2402620897 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7530102160 0x7f753010e670 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.740+0000 7f7535b62640 1 --2- 172.21.15.33:0/2402620897 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75301017a0 0x7f7530101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.740+0000 7f7535b62640 1 --2- 172.21.15.33:0/2402620897 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75301077b0 0x7f7530107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.740+0000 7f7535b62640 1 -- 172.21.15.33:0/2402620897 >> 172.21.15.33:0/2402620897 conn(0x7f75300fd2d0 msgr2=0x7f75300ff6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:32.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.741+0000 7f7535b62640 1 -- 172.21.15.33:0/2402620897 shutdown_connections 2024-03-19T02:56:32.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.741+0000 7f7535b62640 1 -- 172.21.15.33:0/2402620897 wait complete. 2024-03-19T02:56:32.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.741+0000 7f7535b62640 1 Processor -- start 2024-03-19T02:56:32.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.741+0000 7f7535b62640 1 -- start start 2024-03-19T02:56:32.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.741+0000 7f7535b62640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75301017a0 0x7f7530197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.741+0000 7f7535b62640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7530102160 0x7f7530198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.741+0000 7f7535b62640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75301077b0 0x7f753019c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.741+0000 7f7535b62640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7530111a70 con 0x7f75301077b0 2024-03-19T02:56:32.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.741+0000 7f7535b62640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f75301118f0 con 0x7f7530102160 2024-03-19T02:56:32.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.741+0000 7f7535b62640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7530111bf0 con 0x7f75301017a0 2024-03-19T02:56:32.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.742+0000 7f752ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75301077b0 0x7f753019c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.742+0000 7f752f7fe640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75301017a0 0x7f7530197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.742+0000 7f752effd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7530102160 0x7f7530198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.742+0000 7f752f7fe640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75301017a0 0x7f7530197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:49130/0 (socket says 172.21.15.33:49130) 2024-03-19T02:56:32.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.742+0000 7f752f7fe640 1 -- 172.21.15.33:0/1457166869 learned_addr learned my addr 172.21.15.33:0/1457166869 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:32.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.742+0000 7f752ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75301077b0 0x7f753019c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42684/0 (socket says 172.21.15.33:42684) 2024-03-19T02:56:32.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.742+0000 7f752ffff640 1 -- 172.21.15.33:0/1457166869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75301017a0 msgr2=0x7f7530197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.742+0000 7f752ffff640 1 --2- 172.21.15.33:0/1457166869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75301017a0 0x7f7530197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.745 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.742+0000 7f752ffff640 1 -- 172.21.15.33:0/1457166869 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7530102160 msgr2=0x7f7530198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.742+0000 7f752ffff640 1 --2- 172.21.15.33:0/1457166869 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7530102160 0x7f7530198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.742+0000 7f752ffff640 1 -- 172.21.15.33:0/1457166869 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f753019cf90 con 0x7f75301077b0 2024-03-19T02:56:32.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.742+0000 7f752effd640 1 --2- 172.21.15.33:0/1457166869 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7530102160 0x7f7530198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:32.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.742+0000 7f752ffff640 1 --2- 172.21.15.33:0/1457166869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75301077b0 0x7f753019c860 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f752802c630 tx=0x7f7528035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:32.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.742+0000 7f752cff9640 1 -- 172.21.15.33:0/1457166869 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7528039040 con 0x7f75301077b0 2024-03-19T02:56:32.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.742+0000 7f7535b62640 1 -- 172.21.15.33:0/1457166869 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f753019d220 con 0x7f75301077b0 2024-03-19T02:56:32.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.742+0000 7f752cff9640 1 -- 172.21.15.33:0/1457166869 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7528044070 con 0x7f75301077b0 2024-03-19T02:56:32.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.742+0000 7f752cff9640 1 -- 172.21.15.33:0/1457166869 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75280042a0 con 0x7f75301077b0 2024-03-19T02:56:32.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.742+0000 7f7535b62640 1 -- 172.21.15.33:0/1457166869 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75301a4b10 con 0x7f75301077b0 2024-03-19T02:56:32.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.744+0000 7f752cff9640 1 -- 172.21.15.33:0/1457166869 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f7528004440 con 0x7f75301077b0 2024-03-19T02:56:32.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.745+0000 7f753882f740 1 Processor -- start 2024-03-19T02:56:32.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.745+0000 7f753882f740 1 -- start start 2024-03-19T02:56:32.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.745+0000 7f752cff9640 1 --2- 172.21.15.33:0/1457166869 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f75180a2970 0x7f75180a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.745+0000 7f753882f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d771c162d0 0x55d771c166b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.745+0000 7f753882f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d771c16bf0 0x55d771c1f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.745+0000 7f753882f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d771c1f7d0 0x55d771c21bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.745+0000 7f753882f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d771b794b0 con 0x55d771c1f7d0 2024-03-19T02:56:32.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.745+0000 7f753882f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55d771b21d70 con 0x55d771c16bf0 2024-03-19T02:56:32.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.745+0000 7f753882f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d771b75550 con 0x55d771c162d0 2024-03-19T02:56:32.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.745+0000 7f752effd640 1 --2- 172.21.15.33:0/1457166869 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f75180a2970 0x7f75180a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.745+0000 7f752ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d771c1f7d0 0x55d771c21bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.745+0000 7f752f7fe640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d771c162d0 0x55d771c166b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.745+0000 7f752ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d771c1f7d0 0x55d771c21bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42688/0 (socket says 172.21.15.33:42688) 2024-03-19T02:56:32.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.745+0000 7f752f7fe640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d771c162d0 0x55d771c166b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:49140/0 (socket says 172.21.15.33:49140) 2024-03-19T02:56:32.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.745+0000 7f752ffff640 1 -- 172.21.15.33:0/3734086979 learned_addr learned my addr 172.21.15.33:0/3734086979 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:32.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.745+0000 7f752cff9640 1 -- 172.21.15.33:0/1457166869 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f75280f14e0 con 0x7f75301077b0 2024-03-19T02:56:32.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.745+0000 7f752effd640 1 --2- 172.21.15.33:0/3734086979 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d771c16bf0 0x55d771c1f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.745+0000 7f752ffff640 1 -- 172.21.15.33:0/3734086979 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d771c162d0 msgr2=0x55d771c166b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:32.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.745+0000 7f752ffff640 1 --2- 172.21.15.33:0/3734086979 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d771c162d0 0x55d771c166b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.745+0000 7f752ffff640 1 -- 172.21.15.33:0/3734086979 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d771c16bf0 msgr2=0x55d771c1f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:32.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.745+0000 7f752ffff640 1 --2- 172.21.15.33:0/3734086979 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d771c16bf0 0x55d771c1f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.745+0000 7f752ffff640 1 -- 172.21.15.33:0/3734086979 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d771b44210 con 0x55d771c1f7d0 2024-03-19T02:56:32.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.745+0000 7f752f7fe640 1 --2- 172.21.15.33:0/3734086979 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d771c162d0 0x55d771c166b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:32.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.746+0000 7f752effd640 1 --2- 172.21.15.33:0/1457166869 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f75180a2970 0x7f75180a4e30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7524009fd0 tx=0x7f7524009ee0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:32.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.746+0000 7f752ffff640 1 --2- 172.21.15.33:0/3734086979 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d771c1f7d0 0x55d771c21bc0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f7528006010 tx=0x7f752802c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14e0d5151607ae23 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:32.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.746+0000 7f7515ffb640 1 -- 172.21.15.33:0/3734086979 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75280bd200 con 0x55d771c1f7d0 2024-03-19T02:56:32.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.746+0000 7f7515ffb640 1 -- 172.21.15.33:0/3734086979 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7528034700 con 0x55d771c1f7d0 2024-03-19T02:56:32.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.746+0000 7f7515ffb640 1 -- 172.21.15.33:0/3734086979 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75280c6040 con 0x55d771c1f7d0 2024-03-19T02:56:32.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.746+0000 7f753882f740 1 -- 172.21.15.33:0/3734086979 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d771c1f7d0 msgr2=0x55d771c21bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.746+0000 7f753882f740 1 --2- 172.21.15.33:0/3734086979 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d771c1f7d0 0x55d771c21bc0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f7528006010 tx=0x7f752802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.746+0000 7f753882f740 1 -- 172.21.15.33:0/3734086979 shutdown_connections 2024-03-19T02:56:32.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.746+0000 7f753882f740 1 --2- 172.21.15.33:0/3734086979 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d771c1f7d0 0x55d771c21bc0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.746+0000 7f753882f740 1 --2- 172.21.15.33:0/3734086979 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d771c16bf0 0x55d771c1f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.746+0000 7f753882f740 1 --2- 172.21.15.33:0/3734086979 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d771c162d0 0x55d771c166b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.746+0000 7f753882f740 1 -- 172.21.15.33:0/3734086979 >> 172.21.15.33:0/3734086979 conn(0x55d771c10a00 msgr2=0x55d771c10de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:32.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.747+0000 7f753882f740 1 -- 172.21.15.33:0/3734086979 shutdown_connections 2024-03-19T02:56:32.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.747+0000 7f753882f740 1 -- 172.21.15.33:0/3734086979 wait complete. 2024-03-19T02:56:32.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.747+0000 7f753882f740 1 -- start start 2024-03-19T02:56:32.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.747+0000 7f753882f740 1 Processor -- start 2024-03-19T02:56:32.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.747+0000 7f753882f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d771c162d0 0x55d771c1c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.747+0000 7f753882f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d771c16bf0 0x55d771c1cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.747+0000 7f753882f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d771c1f7d0 0x55d771c1d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.747+0000 7f753882f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d771c22380 con 0x55d771c16bf0 2024-03-19T02:56:32.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.747+0000 7f753882f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55d771b35310 con 0x55d771c162d0 2024-03-19T02:56:32.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.747+0000 7f753882f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d771c22500 con 0x55d771c1f7d0 2024-03-19T02:56:32.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.747+0000 7f753882f740 10 client.0 _ll_register_callbacks cb 0x55d771b292d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:56:32.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.747+0000 7f752effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d771c16bf0 0x55d771c1cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.747+0000 7f752f7fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d771c162d0 0x55d771c1c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.747+0000 7f752effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d771c16bf0 0x55d771c1cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42700/0 (socket says 172.21.15.33:42700) 2024-03-19T02:56:32.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.747+0000 7f752effd640 1 -- 172.21.15.33:0/1569618075 learned_addr learned my addr 172.21.15.33:0/1569618075 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:32.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.748+0000 7f752ffff640 1 --2- 172.21.15.33:0/1569618075 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d771c1f7d0 0x55d771c1d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.748+0000 7f752f7fe640 1 -- 172.21.15.33:0/1569618075 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d771c1f7d0 msgr2=0x55d771c1d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:32.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.748+0000 7f752f7fe640 1 --2- 172.21.15.33:0/1569618075 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d771c1f7d0 0x55d771c1d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.748+0000 7f752f7fe640 1 -- 172.21.15.33:0/1569618075 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d771c16bf0 msgr2=0x55d771c1cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:32.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.748+0000 7f752f7fe640 1 --2- 172.21.15.33:0/1569618075 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d771c16bf0 0x55d771c1cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.748+0000 7f752f7fe640 1 -- 172.21.15.33:0/1569618075 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d77188d3f0 con 0x55d771c162d0 2024-03-19T02:56:32.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.748+0000 7f752effd640 1 --2- 172.21.15.33:0/1569618075 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d771c16bf0 0x55d771c1cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:32.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.748+0000 7f752f7fe640 1 --2- 172.21.15.33:0/1569618075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d771c162d0 0x55d771c1c8a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7520012720 tx=0x7f7520012be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70f3bb11df30054f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:32.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.748+0000 7f750effd640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T02:56:32.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.748+0000 7f750effd640 1 -- 172.21.15.33:0/1569618075 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7520004020 con 0x55d771c162d0 2024-03-19T02:56:32.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.748+0000 7f750effd640 1 -- 172.21.15.33:0/1569618075 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f75200041c0 con 0x55d771c162d0 2024-03-19T02:56:32.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.748+0000 7f753882f740 1 -- 172.21.15.33:0/1569618075 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d771bd73e0 con 0x55d771c162d0 2024-03-19T02:56:32.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.748+0000 7f750effd640 1 -- 172.21.15.33:0/1569618075 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f75200043c0 con 0x55d771c162d0 2024-03-19T02:56:32.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.748+0000 7f753882f740 10 client.26144 fetch_fsmap learned FSMap version 25 2024-03-19T02:56:32.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.748+0000 7f753882f740 1 -- 172.21.15.33:0/1569618075 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d7719577d0 con 0x55d771c162d0 2024-03-19T02:56:32.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.749+0000 7f750effd640 1 -- 172.21.15.33:0/1569618075 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55d771bd73e0 con 0x55d771c162d0 2024-03-19T02:56:32.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.749+0000 7f750effd640 1 -- 172.21.15.33:0/1569618075 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f7520003a70 con 0x55d771c162d0 2024-03-19T02:56:32.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.749+0000 7f753882f740 10 client.26144 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:56:32.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.749+0000 7f753882f740 10 client.26144 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T02:56:32.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.749+0000 7f753882f740 20 client.26144 populate_metadata read hostname 'smithi033' 2024-03-19T02:56:32.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.749+0000 7f753882f740 1 --2- 172.21.15.33:0/1569618075 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55d771c2f9c0 0x55d771c31e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.749+0000 7f753882f740 4 client.26144 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:56:32.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.749+0000 7f753882f740 1 -- 172.21.15.33:0/1569618075 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d771b869f0 con 0x55d771c2f9c0 2024-03-19T02:56:32.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.749+0000 7f752effd640 1 --2- 172.21.15.33:0/1569618075 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55d771c2f9c0 0x55d771c31e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.749+0000 7f752effd640 1 --2- 172.21.15.33:0/1569618075 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55d771c2f9c0 0x55d771c31e10 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b60e595ff7e4bdaf server_cookie=438598eae1df7f24 in_seq=0 out_seq=0 2024-03-19T02:56:32.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.749+0000 7f750effd640 10 client.26144 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T02:56:32.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.750+0000 7f750effd640 1 -- 172.21.15.33:0/1569618075 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d771b869f0 con 0x55d771c2f9c0 2024-03-19T02:56:32.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.750+0000 7f750effd640 10 client.26144 handle_command_reply: tid=0 2024-03-19T02:56:32.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.750+0000 7f753882f740 1 client.26144 shutdown 2024-03-19T02:56:32.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.750+0000 7f753882f740 1 -- 172.21.15.33:0/1569618075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d771c162d0 msgr2=0x55d771c1c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.750+0000 7f753882f740 1 --2- 172.21.15.33:0/1569618075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d771c162d0 0x55d771c1c8a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7520012720 tx=0x7f7520012be0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.750+0000 7f753882f740 1 -- 172.21.15.33:0/1569618075 shutdown_connections 2024-03-19T02:56:32.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.750+0000 7f753882f740 1 --2- 172.21.15.33:0/1569618075 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55d771c2f9c0 0x55d771c31e10 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.750+0000 7f753882f740 1 --2- 172.21.15.33:0/1569618075 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d771c1f7d0 0x55d771c1d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.750+0000 7f750effd640 0 client.26144 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T02:56:32.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.750+0000 7f753882f740 1 --2- 172.21.15.33:0/1569618075 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d771c16bf0 0x55d771c1cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.750+0000 7f753882f740 1 --2- 172.21.15.33:0/1569618075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d771c162d0 0x55d771c1c8a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.750+0000 7f753882f740 1 -- 172.21.15.33:0/1569618075 >> 172.21.15.33:0/1569618075 conn(0x55d771c10a00 msgr2=0x55d771c12750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:32.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.750+0000 7f753882f740 1 -- 172.21.15.33:0/1569618075 shutdown_connections 2024-03-19T02:56:32.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.750+0000 7f753882f740 1 -- 172.21.15.33:0/1569618075 wait complete. 2024-03-19T02:56:32.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.750+0000 7f753882f740 20 client.26144 trim_cache size 0 max 16384 2024-03-19T02:56:32.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.763+0000 7f753882f740 1 Processor -- start 2024-03-19T02:56:32.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.763+0000 7f753882f740 1 -- start start 2024-03-19T02:56:32.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.764+0000 7f753882f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d771c162d0 0x55d771c1efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.764+0000 7f753882f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d771c16bf0 0x55d771c1d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.764+0000 7f753882f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d771c1f7d0 0x55d771c37720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.770 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.764+0000 7f753882f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f75301b14c0 con 0x55d771c16bf0 2024-03-19T02:56:32.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.764+0000 7f753882f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f75301aed90 con 0x55d771c162d0 2024-03-19T02:56:32.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.764+0000 7f753882f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f75301acbc0 con 0x55d771c1f7d0 2024-03-19T02:56:32.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.764+0000 7f752effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d771c16bf0 0x55d771c1d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.764+0000 7f752effd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d771c16bf0 0x55d771c1d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42716/0 (socket says 172.21.15.33:42716) 2024-03-19T02:56:32.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.764+0000 7f752effd640 1 -- 172.21.15.33:0/2087062660 learned_addr learned my addr 172.21.15.33:0/2087062660 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:32.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.764+0000 7f752ffff640 1 --2- 172.21.15.33:0/2087062660 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d771c1f7d0 0x55d771c37720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.764+0000 7f752f7fe640 1 --2- 172.21.15.33:0/2087062660 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d771c162d0 0x55d771c1efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.764+0000 7f752effd640 1 -- 172.21.15.33:0/2087062660 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d771c1f7d0 msgr2=0x55d771c37720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:32.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.764+0000 7f752effd640 1 --2- 172.21.15.33:0/2087062660 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d771c1f7d0 0x55d771c37720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.764+0000 7f752effd640 1 -- 172.21.15.33:0/2087062660 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d771c162d0 msgr2=0x55d771c1efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:32.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.764+0000 7f752effd640 1 --2- 172.21.15.33:0/2087062660 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d771c162d0 0x55d771c1efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.764+0000 7f752effd640 1 -- 172.21.15.33:0/2087062660 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d771c32490 con 0x55d771c16bf0 2024-03-19T02:56:32.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.764+0000 7f752f7fe640 1 --2- 172.21.15.33:0/2087062660 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d771c162d0 0x55d771c1efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:56:32.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.764+0000 7f752ffff640 1 --2- 172.21.15.33:0/2087062660 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d771c1f7d0 0x55d771c37720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:56:32.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.764+0000 7f752effd640 1 --2- 172.21.15.33:0/2087062660 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d771c16bf0 0x55d771c1d140 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f7524013870 tx=0x7f7524021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9ebdec1712f7ca server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:32.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.764+0000 7f75157fa640 1 -- 172.21.15.33:0/2087062660 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7524003a40 con 0x55d771c16bf0 2024-03-19T02:56:32.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.764+0000 7f75157fa640 1 -- 172.21.15.33:0/2087062660 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7524003be0 con 0x55d771c16bf0 2024-03-19T02:56:32.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.764+0000 7f75157fa640 1 -- 172.21.15.33:0/2087062660 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f752402c630 con 0x55d771c16bf0 2024-03-19T02:56:32.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.765+0000 7f753882f740 1 -- 172.21.15.33:0/2087062660 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d771c16bf0 msgr2=0x55d771c1d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.765+0000 7f753882f740 1 --2- 172.21.15.33:0/2087062660 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d771c16bf0 0x55d771c1d140 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f7524013870 tx=0x7f7524021000 comp rx=0 tx=0).stop 2024-03-19T02:56:32.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.765+0000 7f753882f740 1 -- 172.21.15.33:0/2087062660 shutdown_connections 2024-03-19T02:56:32.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.765+0000 7f753882f740 1 --2- 172.21.15.33:0/2087062660 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d771c1f7d0 0x55d771c37720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.765+0000 7f753882f740 1 --2- 172.21.15.33:0/2087062660 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d771c16bf0 0x55d771c1d140 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.765+0000 7f753882f740 1 --2- 172.21.15.33:0/2087062660 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d771c162d0 0x55d771c1efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.765+0000 7f753882f740 1 -- 172.21.15.33:0/2087062660 >> 172.21.15.33:0/2087062660 conn(0x55d771c10a00 msgr2=0x55d771c11c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:32.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.765+0000 7f753882f740 1 -- 172.21.15.33:0/2087062660 shutdown_connections 2024-03-19T02:56:32.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.765+0000 7f753882f740 1 -- 172.21.15.33:0/2087062660 wait complete. 2024-03-19T02:56:32.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.765+0000 7f753882f740 1 -- start start 2024-03-19T02:56:32.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.765+0000 7f753882f740 1 Processor -- start 2024-03-19T02:56:32.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.766+0000 7f753882f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d771c162d0 0x55d771c22f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.766+0000 7f753882f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d771c16bf0 0x55d771c234d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.766+0000 7f753882f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d771c1f7d0 0x55d771c23a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.766+0000 7f753882f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f75301ad250 con 0x55d771c162d0 2024-03-19T02:56:32.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.766+0000 7f753882f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f75301ae6c0 con 0x55d771c1f7d0 2024-03-19T02:56:32.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.766+0000 7f753882f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f75301ad920 con 0x55d771c16bf0 2024-03-19T02:56:32.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.766+0000 7f753882f740 10 client.0 _ll_register_callbacks cb 0x7f7520009950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:56:32.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.766+0000 7f752f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d771c162d0 0x55d771c22f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.766+0000 7f752f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d771c162d0 0x55d771c22f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42724/0 (socket says 172.21.15.33:42724) 2024-03-19T02:56:32.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.766+0000 7f752f7fe640 1 -- 172.21.15.33:0/299171203 learned_addr learned my addr 172.21.15.33:0/299171203 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:32.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.766+0000 7f752effd640 1 --2- 172.21.15.33:0/299171203 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d771c16bf0 0x55d771c234d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.766+0000 7f752ffff640 1 --2- 172.21.15.33:0/299171203 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d771c1f7d0 0x55d771c23a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.766+0000 7f752f7fe640 1 -- 172.21.15.33:0/299171203 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d771c16bf0 msgr2=0x55d771c234d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:32.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.766+0000 7f752f7fe640 1 --2- 172.21.15.33:0/299171203 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d771c16bf0 0x55d771c234d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.773 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.766+0000 7f752f7fe640 1 -- 172.21.15.33:0/299171203 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d771c1f7d0 msgr2=0x55d771c23a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:32.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.766+0000 7f752f7fe640 1 --2- 172.21.15.33:0/299171203 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d771c1f7d0 0x55d771c23a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.766+0000 7f752f7fe640 1 -- 172.21.15.33:0/299171203 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d771c94da0 con 0x55d771c162d0 2024-03-19T02:56:32.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.766+0000 7f752ffff640 1 --2- 172.21.15.33:0/299171203 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d771c1f7d0 0x55d771c23a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:56:32.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.766+0000 7f752f7fe640 1 --2- 172.21.15.33:0/299171203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d771c162d0 0x55d771c22f90 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f752001c610 tx=0x7f752001c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44effa6aba842f98 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:32.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.766+0000 7f750effd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:56:32.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.766+0000 7f750effd640 1 -- 172.21.15.33:0/299171203 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75200041f0 con 0x55d771c162d0 2024-03-19T02:56:32.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.766+0000 7f750effd640 1 -- 172.21.15.33:0/299171203 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f752001c920 con 0x55d771c162d0 2024-03-19T02:56:32.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.766+0000 7f753882f740 1 -- 172.21.15.33:0/299171203 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7524003be0 con 0x55d771c162d0 2024-03-19T02:56:32.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.767+0000 7f750effd640 1 -- 172.21.15.33:0/299171203 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f752001cb20 con 0x55d771c162d0 2024-03-19T02:56:32.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.767+0000 7f753882f740 10 client.16326 fetch_fsmap learned FSMap version 25 2024-03-19T02:56:32.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.767+0000 7f753882f740 1 -- 172.21.15.33:0/299171203 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d771c96080 con 0x55d771c162d0 2024-03-19T02:56:32.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.767+0000 7f750effd640 1 -- 172.21.15.33:0/299171203 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7524003be0 con 0x55d771c162d0 2024-03-19T02:56:32.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.767+0000 7f750effd640 1 -- 172.21.15.33:0/299171203 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f7520003820 con 0x55d771c162d0 2024-03-19T02:56:32.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.767+0000 7f753882f740 10 client.16326 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:56:32.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.767+0000 7f753882f740 10 client.16326 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T02:56:32.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.767+0000 7f753882f740 20 client.16326 populate_metadata read hostname 'smithi033' 2024-03-19T02:56:32.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.767+0000 7f753882f740 1 --2- 172.21.15.33:0/299171203 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55d771c96850 0x55d771c98c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.767+0000 7f753882f740 4 client.16326 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:56:32.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.767+0000 7f753882f740 1 -- 172.21.15.33:0/299171203 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d771b80d40 con 0x55d771c96850 2024-03-19T02:56:32.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.767+0000 7f752effd640 1 --2- 172.21.15.33:0/299171203 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55d771c96850 0x55d771c98c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.767+0000 7f752effd640 1 --2- 172.21.15.33:0/299171203 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55d771c96850 0x55d771c98c70 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=df954a748644daa7 server_cookie=c5666ac2783fbeb2 in_seq=0 out_seq=0 2024-03-19T02:56:32.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.767+0000 7f750effd640 10 client.16326 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T02:56:32.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.768+0000 7f750effd640 1 -- 172.21.15.33:0/299171203 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d771b80d40 con 0x55d771c96850 2024-03-19T02:56:32.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.768+0000 7f750effd640 10 client.16326 handle_command_reply: tid=0 2024-03-19T02:56:32.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.768+0000 7f753882f740 1 client.16326 shutdown 2024-03-19T02:56:32.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.768+0000 7f753882f740 1 -- 172.21.15.33:0/299171203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d771c162d0 msgr2=0x55d771c22f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.768+0000 7f753882f740 1 --2- 172.21.15.33:0/299171203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d771c162d0 0x55d771c22f90 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f752001c610 tx=0x7f752001c640 comp rx=0 tx=0).stop 2024-03-19T02:56:32.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.768+0000 7f753882f740 1 -- 172.21.15.33:0/299171203 shutdown_connections 2024-03-19T02:56:32.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.768+0000 7f753882f740 1 --2- 172.21.15.33:0/299171203 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55d771c96850 0x55d771c98c70 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.768+0000 7f753882f740 1 --2- 172.21.15.33:0/299171203 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d771c16bf0 0x55d771c234d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.768+0000 7f750effd640 0 client.16326 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T02:56:32.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.768+0000 7f753882f740 1 --2- 172.21.15.33:0/299171203 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d771c1f7d0 0x55d771c23a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.768+0000 7f753882f740 1 --2- 172.21.15.33:0/299171203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d771c162d0 0x55d771c22f90 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.768+0000 7f753882f740 1 -- 172.21.15.33:0/299171203 >> 172.21.15.33:0/299171203 conn(0x55d771c10a00 msgr2=0x55d771c12450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:32.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.768+0000 7f753882f740 1 -- 172.21.15.33:0/299171203 shutdown_connections 2024-03-19T02:56:32.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.768+0000 7f753882f740 1 -- 172.21.15.33:0/299171203 wait complete. 2024-03-19T02:56:32.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.768+0000 7f753882f740 20 client.16326 trim_cache size 0 max 16384 2024-03-19T02:56:32.776 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:56:32.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.768+0000 7f7535b62640 1 -- 172.21.15.33:0/1457166869 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f75180a2970 msgr2=0x7f75180a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.768+0000 7f7535b62640 1 --2- 172.21.15.33:0/1457166869 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f75180a2970 0x7f75180a4e30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7524009fd0 tx=0x7f7524009ee0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.768+0000 7f7535b62640 1 -- 172.21.15.33:0/1457166869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75301077b0 msgr2=0x7f753019c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.768+0000 7f7535b62640 1 --2- 172.21.15.33:0/1457166869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75301077b0 0x7f753019c860 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f752802c630 tx=0x7f7528035d60 comp rx=0 tx=0).stop 2024-03-19T02:56:32.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.768+0000 7f7535b62640 1 -- 172.21.15.33:0/1457166869 shutdown_connections 2024-03-19T02:56:32.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.768+0000 7f7535b62640 1 --2- 172.21.15.33:0/1457166869 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f75180a2970 0x7f75180a4e30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.768+0000 7f7535b62640 1 --2- 172.21.15.33:0/1457166869 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f75301077b0 0x7f753019c860 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.768+0000 7f7535b62640 1 --2- 172.21.15.33:0/1457166869 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7530102160 0x7f7530198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.768+0000 7f7535b62640 1 --2- 172.21.15.33:0/1457166869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75301017a0 0x7f7530197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.769+0000 7f7535b62640 1 -- 172.21.15.33:0/1457166869 >> 172.21.15.33:0/1457166869 conn(0x7f75300fd2d0 msgr2=0x7f75300fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:32.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.769+0000 7f7535b62640 1 -- 172.21.15.33:0/1457166869 shutdown_connections 2024-03-19T02:56:32.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.769+0000 7f7535b62640 1 -- 172.21.15.33:0/1457166869 wait complete. 2024-03-19T02:56:32.781 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:56:32.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.885+0000 7f9ac104d640 1 -- 172.21.15.33:0/1557144227 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9abc1066b0 msgr2=0x7f9abc106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.885+0000 7f9ac104d640 1 --2- 172.21.15.33:0/1557144227 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9abc1066b0 0x7f9abc106a90 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f9aac009930 tx=0x7f9aac02bc30 comp rx=0 tx=0).stop 2024-03-19T02:56:32.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.885+0000 7f9ac104d640 1 -- 172.21.15.33:0/1557144227 shutdown_connections 2024-03-19T02:56:32.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.885+0000 7f9ac104d640 1 --2- 172.21.15.33:0/1557144227 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9abc1006a0 0x7f9abc100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.885+0000 7f9ac104d640 1 --2- 172.21.15.33:0/1557144227 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9abc101060 0x7f9abc10d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.885+0000 7f9ac104d640 1 --2- 172.21.15.33:0/1557144227 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9abc1066b0 0x7f9abc106a90 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.885+0000 7f9ac104d640 1 -- 172.21.15.33:0/1557144227 >> 172.21.15.33:0/1557144227 conn(0x7f9abc0778d0 msgr2=0x7f9abc077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:32.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.885+0000 7f9ac104d640 1 -- 172.21.15.33:0/1557144227 shutdown_connections 2024-03-19T02:56:32.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.885+0000 7f9ac104d640 1 -- 172.21.15.33:0/1557144227 wait complete. 2024-03-19T02:56:32.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.885+0000 7f9ac104d640 1 Processor -- start 2024-03-19T02:56:32.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.885+0000 7f9ac104d640 1 -- start start 2024-03-19T02:56:32.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.886+0000 7f9ac104d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9abc1006a0 0x7f9abc198e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.886+0000 7f9ac104d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9abc101060 0x7f9abc199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.886+0000 7f9ac104d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9abc1066b0 0x7f9abc1a3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.886+0000 7f9ac104d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9abc110930 con 0x7f9abc1006a0 2024-03-19T02:56:32.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.886+0000 7f9ac104d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9abc1107b0 con 0x7f9abc101060 2024-03-19T02:56:32.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.886+0000 7f9ac104d640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9abc110ab0 con 0x7f9abc1066b0 2024-03-19T02:56:32.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.886+0000 7f9abad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9abc1006a0 0x7f9abc198e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.886+0000 7f9abad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9abc1006a0 0x7f9abc198e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42746/0 (socket says 172.21.15.33:42746) 2024-03-19T02:56:32.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.886+0000 7f9abad9d640 1 -- 172.21.15.33:0/3796743462 learned_addr learned my addr 172.21.15.33:0/3796743462 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:32.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.886+0000 7f9abb59e640 1 --2- 172.21.15.33:0/3796743462 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9abc1066b0 0x7f9abc1a3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.886+0000 7f9ab3fff640 1 --2- 172.21.15.33:0/3796743462 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9abc101060 0x7f9abc199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.886+0000 7f9abad9d640 1 -- 172.21.15.33:0/3796743462 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9abc1066b0 msgr2=0x7f9abc1a3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.886+0000 7f9abad9d640 1 --2- 172.21.15.33:0/3796743462 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9abc1066b0 0x7f9abc1a3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.886+0000 7f9abad9d640 1 -- 172.21.15.33:0/3796743462 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9abc101060 msgr2=0x7f9abc199350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.886+0000 7f9abad9d640 1 --2- 172.21.15.33:0/3796743462 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9abc101060 0x7f9abc199350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.886+0000 7f9abad9d640 1 -- 172.21.15.33:0/3796743462 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9abc1a43d0 con 0x7f9abc1006a0 2024-03-19T02:56:32.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.886+0000 7f9abad9d640 1 --2- 172.21.15.33:0/3796743462 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9abc1006a0 0x7f9abc198e10 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f9aac009230 tx=0x7f9aac02c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:32.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.886+0000 7f9ab8d99640 1 -- 172.21.15.33:0/3796743462 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aac043020 con 0x7f9abc1006a0 2024-03-19T02:56:32.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.886+0000 7f9ab8d99640 1 -- 172.21.15.33:0/3796743462 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9aac02ca70 con 0x7f9abc1006a0 2024-03-19T02:56:32.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.886+0000 7f9ab8d99640 1 -- 172.21.15.33:0/3796743462 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aac003970 con 0x7f9abc1006a0 2024-03-19T02:56:32.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.886+0000 7f9ac104d640 1 -- 172.21.15.33:0/3796743462 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9abc1a45a0 con 0x7f9abc1006a0 2024-03-19T02:56:32.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.886+0000 7f9ac104d640 1 -- 172.21.15.33:0/3796743462 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9abc1a4a90 con 0x7f9abc1006a0 2024-03-19T02:56:32.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.889+0000 7f9ab8d99640 1 -- 172.21.15.33:0/3796743462 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f9aac046020 con 0x7f9abc1006a0 2024-03-19T02:56:32.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.889+0000 7f9ac3c30740 1 Processor -- start 2024-03-19T02:56:32.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.889+0000 7f9ac3c30740 1 -- start start 2024-03-19T02:56:32.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.889+0000 7f9ac3c30740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a5b1122d0 0x562a5b1126b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.889+0000 7f9ac3c30740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a5b112bf0 0x562a5b11b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.889+0000 7f9ac3c30740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a5b11b7d0 0x562a5b11dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.889+0000 7f9ac3c30740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562a5b0754b0 con 0x562a5b1122d0 2024-03-19T02:56:32.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.889+0000 7f9ac3c30740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x562a5b01dd70 con 0x562a5b112bf0 2024-03-19T02:56:32.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.889+0000 7f9ac3c30740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x562a5b071550 con 0x562a5b11b7d0 2024-03-19T02:56:32.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.889+0000 7f9ab8d99640 1 --2- 172.21.15.33:0/3796743462 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f9aa80a2970 0x7f9aa80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.889+0000 7f9abb59e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a5b1122d0 0x562a5b1126b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.890+0000 7f9ab8d99640 1 -- 172.21.15.33:0/3796743462 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f9aac0f10e0 con 0x7f9abc1006a0 2024-03-19T02:56:32.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.890+0000 7f9abad9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a5b112bf0 0x562a5b11b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.890+0000 7f9abb59e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a5b1122d0 0x562a5b1126b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42754/0 (socket says 172.21.15.33:42754) 2024-03-19T02:56:32.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.890+0000 7f9abb59e640 1 -- 172.21.15.33:0/2969215926 learned_addr learned my addr 172.21.15.33:0/2969215926 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:32.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.890+0000 7f9ab3fff640 1 --2- 172.21.15.33:0/2969215926 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a5b11b7d0 0x562a5b11dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.890+0000 7f9abb59e640 1 --2- 172.21.15.33:0/3796743462 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f9aa80a2970 0x7f9aa80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.890+0000 7f9abad9d640 1 -- 172.21.15.33:0/2969215926 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a5b11b7d0 msgr2=0x562a5b11dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:32.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.890+0000 7f9abad9d640 1 --2- 172.21.15.33:0/2969215926 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a5b11b7d0 0x562a5b11dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.890+0000 7f9abad9d640 1 -- 172.21.15.33:0/2969215926 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a5b1122d0 msgr2=0x562a5b1126b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:32.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.890+0000 7f9abad9d640 1 --2- 172.21.15.33:0/2969215926 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a5b1122d0 0x562a5b1126b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.890+0000 7f9abad9d640 1 -- 172.21.15.33:0/2969215926 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a5b040210 con 0x562a5b112bf0 2024-03-19T02:56:32.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.890+0000 7f9ab3fff640 1 --2- 172.21.15.33:0/2969215926 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a5b11b7d0 0x562a5b11dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:32.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.890+0000 7f9abad9d640 1 --2- 172.21.15.33:0/2969215926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a5b112bf0 0x562a5b11b290 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9aac0ee5d0 tx=0x7f9aac0b9870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47096824d9041783 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:32.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.890+0000 7f9abb59e640 1 --2- 172.21.15.33:0/3796743462 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f9aa80a2970 0x7f9aa80a4e30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4005c40 tx=0x7f9ab4013040 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:32.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.890+0000 7f9ab17fa640 1 -- 172.21.15.33:0/2969215926 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aac0d6070 con 0x562a5b112bf0 2024-03-19T02:56:32.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.890+0000 7f9ab17fa640 1 -- 172.21.15.33:0/2969215926 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9aac0bdbe0 con 0x562a5b112bf0 2024-03-19T02:56:32.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.890+0000 7f9ab17fa640 1 -- 172.21.15.33:0/2969215926 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aac0bb8c0 con 0x562a5b112bf0 2024-03-19T02:56:32.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.891+0000 7f9ac3c30740 1 -- 172.21.15.33:0/2969215926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a5b112bf0 msgr2=0x562a5b11b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.891+0000 7f9ac3c30740 1 --2- 172.21.15.33:0/2969215926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a5b112bf0 0x562a5b11b290 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9aac0ee5d0 tx=0x7f9aac0b9870 comp rx=0 tx=0).stop 2024-03-19T02:56:32.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.891+0000 7f9ac3c30740 1 -- 172.21.15.33:0/2969215926 shutdown_connections 2024-03-19T02:56:32.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.891+0000 7f9ac3c30740 1 --2- 172.21.15.33:0/2969215926 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a5b11b7d0 0x562a5b11dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.891+0000 7f9ac3c30740 1 --2- 172.21.15.33:0/2969215926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a5b112bf0 0x562a5b11b290 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.891+0000 7f9ac3c30740 1 --2- 172.21.15.33:0/2969215926 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a5b1122d0 0x562a5b1126b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.891+0000 7f9ac3c30740 1 -- 172.21.15.33:0/2969215926 >> 172.21.15.33:0/2969215926 conn(0x562a5b10ca00 msgr2=0x562a5b10cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:32.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.891+0000 7f9ac3c30740 1 -- 172.21.15.33:0/2969215926 shutdown_connections 2024-03-19T02:56:32.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.891+0000 7f9ac3c30740 1 -- 172.21.15.33:0/2969215926 wait complete. 2024-03-19T02:56:32.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.891+0000 7f9ac3c30740 1 -- start start 2024-03-19T02:56:32.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.891+0000 7f9ac3c30740 1 Processor -- start 2024-03-19T02:56:32.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.892+0000 7f9ac3c30740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a5b1122d0 0x562a5b1188c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.892+0000 7f9ac3c30740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a5b112bf0 0x562a5b118e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.892+0000 7f9ac3c30740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a5b11b7d0 0x562a5b119340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.892+0000 7f9ac3c30740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562a5b11e380 con 0x562a5b11b7d0 2024-03-19T02:56:32.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.892+0000 7f9ac3c30740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x562a5b031310 con 0x562a5b112bf0 2024-03-19T02:56:32.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.892+0000 7f9ac3c30740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x562a5b11e500 con 0x562a5b1122d0 2024-03-19T02:56:32.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.892+0000 7f9ac3c30740 10 client.0 _ll_register_callbacks cb 0x562a5b0252d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:56:32.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.892+0000 7f9abb59e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a5b11b7d0 0x562a5b119340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.892+0000 7f9abad9d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a5b1122d0 0x562a5b1188c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.892+0000 7f9ab3fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a5b112bf0 0x562a5b118e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.892+0000 7f9abb59e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a5b11b7d0 0x562a5b119340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42762/0 (socket says 172.21.15.33:42762) 2024-03-19T02:56:32.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.892+0000 7f9abb59e640 1 -- 172.21.15.33:0/4124550064 learned_addr learned my addr 172.21.15.33:0/4124550064 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:32.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.892+0000 7f9abad9d640 1 --2- 172.21.15.33:0/4124550064 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a5b1122d0 0x562a5b1188c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:49188/0 (socket says 172.21.15.33:49188) 2024-03-19T02:56:32.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.892+0000 7f9ab3fff640 1 -- 172.21.15.33:0/4124550064 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a5b1122d0 msgr2=0x562a5b1188c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:32.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.892+0000 7f9ab3fff640 1 --2- 172.21.15.33:0/4124550064 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a5b1122d0 0x562a5b1188c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.892+0000 7f9ab3fff640 1 -- 172.21.15.33:0/4124550064 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a5b11b7d0 msgr2=0x562a5b119340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:32.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.892+0000 7f9ab3fff640 1 --2- 172.21.15.33:0/4124550064 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a5b11b7d0 0x562a5b119340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.892+0000 7f9ab3fff640 1 -- 172.21.15.33:0/4124550064 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a5ad893f0 con 0x562a5b112bf0 2024-03-19T02:56:32.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.892+0000 7f9abad9d640 1 --2- 172.21.15.33:0/4124550064 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a5b1122d0 0x562a5b1188c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:32.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.892+0000 7f9abb59e640 1 --2- 172.21.15.33:0/4124550064 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a5b11b7d0 0x562a5b119340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:32.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.892+0000 7f9ab3fff640 1 --2- 172.21.15.33:0/4124550064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a5b112bf0 0x562a5b118e00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9abc0694f0 tx=0x7f9aa4012b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b1f920b3edde05d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:32.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.892+0000 7f9aa27fc640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T02:56:32.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.892+0000 7f9aa27fc640 1 -- 172.21.15.33:0/4124550064 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa4004030 con 0x562a5b112bf0 2024-03-19T02:56:32.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.892+0000 7f9ac3c30740 1 -- 172.21.15.33:0/4124550064 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9aac0bdbe0 con 0x562a5b112bf0 2024-03-19T02:56:32.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.892+0000 7f9aa27fc640 1 -- 172.21.15.33:0/4124550064 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f9aa40041d0 con 0x562a5b112bf0 2024-03-19T02:56:32.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.893+0000 7f9aa27fc640 1 -- 172.21.15.33:0/4124550064 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9aa4025050 con 0x562a5b112bf0 2024-03-19T02:56:32.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.893+0000 7f9ac3c30740 10 client.26162 fetch_fsmap learned FSMap version 25 2024-03-19T02:56:32.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.893+0000 7f9ac3c30740 1 -- 172.21.15.33:0/4124550064 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562a5ae537d0 con 0x562a5b112bf0 2024-03-19T02:56:32.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.893+0000 7f9aa27fc640 1 -- 172.21.15.33:0/4124550064 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9aac0bdbe0 con 0x562a5b112bf0 2024-03-19T02:56:32.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.893+0000 7f9aa27fc640 1 -- 172.21.15.33:0/4124550064 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f9aa4003950 con 0x562a5b112bf0 2024-03-19T02:56:32.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.893+0000 7f9ac3c30740 10 client.26162 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:56:32.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.893+0000 7f9ac3c30740 10 client.26162 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:56:32.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.893+0000 7f9ac3c30740 20 client.26162 populate_metadata read hostname 'smithi033' 2024-03-19T02:56:32.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.893+0000 7f9ac3c30740 1 --2- 172.21.15.33:0/4124550064 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x562a5b12b9f0 0x562a5b12de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.893+0000 7f9ac3c30740 4 client.26162 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:56:32.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.893+0000 7f9ac3c30740 1 -- 172.21.15.33:0/4124550064 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a5b0d33e0 con 0x562a5b12b9f0 2024-03-19T02:56:32.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.893+0000 7f9abad9d640 1 --2- 172.21.15.33:0/4124550064 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x562a5b12b9f0 0x562a5b12de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.893+0000 7f9aa27fc640 10 client.26162 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:56:32.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.893+0000 7f9abad9d640 1 --2- 172.21.15.33:0/4124550064 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x562a5b12b9f0 0x562a5b12de10 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b87f03b2d39e752e server_cookie=61c5f9c4920b6cd6 in_seq=0 out_seq=0 2024-03-19T02:56:32.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.894+0000 7f9aa27fc640 1 -- 172.21.15.33:0/4124550064 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562a5b0d33e0 con 0x562a5b12b9f0 2024-03-19T02:56:32.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.894+0000 7f9aa27fc640 10 client.26162 handle_command_reply: tid=0 2024-03-19T02:56:32.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.894+0000 7f9ac3c30740 1 client.26162 shutdown 2024-03-19T02:56:32.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.894+0000 7f9ac3c30740 1 -- 172.21.15.33:0/4124550064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a5b112bf0 msgr2=0x562a5b118e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.894+0000 7f9ac3c30740 1 --2- 172.21.15.33:0/4124550064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a5b112bf0 0x562a5b118e00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9abc0694f0 tx=0x7f9aa4012b60 comp rx=0 tx=0).stop 2024-03-19T02:56:32.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.894+0000 7f9ac3c30740 1 -- 172.21.15.33:0/4124550064 shutdown_connections 2024-03-19T02:56:32.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.894+0000 7f9ac3c30740 1 --2- 172.21.15.33:0/4124550064 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x562a5b12b9f0 0x562a5b12de10 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.894+0000 7f9ac3c30740 1 --2- 172.21.15.33:0/4124550064 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a5b11b7d0 0x562a5b119340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.894+0000 7f9ac3c30740 1 --2- 172.21.15.33:0/4124550064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a5b112bf0 0x562a5b118e00 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.894+0000 7f9ac3c30740 1 --2- 172.21.15.33:0/4124550064 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a5b1122d0 0x562a5b1188c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.894+0000 7f9aa27fc640 0 client.26162 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:56:32.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.894+0000 7f9ac3c30740 1 -- 172.21.15.33:0/4124550064 >> 172.21.15.33:0/4124550064 conn(0x562a5b10ca00 msgr2=0x562a5b10e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:32.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.894+0000 7f9ac3c30740 1 -- 172.21.15.33:0/4124550064 shutdown_connections 2024-03-19T02:56:32.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.894+0000 7f9ac3c30740 1 -- 172.21.15.33:0/4124550064 wait complete. 2024-03-19T02:56:32.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.894+0000 7f9ac3c30740 20 client.26162 trim_cache size 0 max 16384 2024-03-19T02:56:32.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.908+0000 7f9ac3c30740 1 Processor -- start 2024-03-19T02:56:32.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.908+0000 7f9ac3c30740 1 -- start start 2024-03-19T02:56:32.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.908+0000 7f9ac3c30740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a5b1122d0 0x562a5b11aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.908+0000 7f9ac3c30740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a5b112bf0 0x562a5b117a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.908+0000 7f9ac3c30740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a5b11b7d0 0x562a5b1336e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.908+0000 7f9ac3c30740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9abc1b14a0 con 0x562a5b11b7d0 2024-03-19T02:56:32.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.908+0000 7f9ac3c30740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9abc1aed70 con 0x562a5b1122d0 2024-03-19T02:56:32.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.908+0000 7f9ac3c30740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9abc1acba0 con 0x562a5b112bf0 2024-03-19T02:56:32.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.908+0000 7f9abb59e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a5b11b7d0 0x562a5b1336e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.908+0000 7f9abb59e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a5b11b7d0 0x562a5b1336e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42772/0 (socket says 172.21.15.33:42772) 2024-03-19T02:56:32.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.908+0000 7f9abb59e640 1 -- 172.21.15.33:0/3761207984 learned_addr learned my addr 172.21.15.33:0/3761207984 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:32.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.908+0000 7f9abad9d640 1 --2- 172.21.15.33:0/3761207984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a5b1122d0 0x562a5b11aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.908+0000 7f9ab3fff640 1 --2- 172.21.15.33:0/3761207984 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a5b112bf0 0x562a5b117a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.908+0000 7f9abb59e640 1 -- 172.21.15.33:0/3761207984 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a5b112bf0 msgr2=0x562a5b117a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:32.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.908+0000 7f9abb59e640 1 --2- 172.21.15.33:0/3761207984 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a5b112bf0 0x562a5b117a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.908+0000 7f9abb59e640 1 -- 172.21.15.33:0/3761207984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a5b1122d0 msgr2=0x562a5b11aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:32.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.908+0000 7f9abb59e640 1 --2- 172.21.15.33:0/3761207984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a5b1122d0 0x562a5b11aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.908+0000 7f9abb59e640 1 -- 172.21.15.33:0/3761207984 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a5b0e2880 con 0x562a5b11b7d0 2024-03-19T02:56:32.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.908+0000 7f9abad9d640 1 --2- 172.21.15.33:0/3761207984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a5b1122d0 0x562a5b11aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:32.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.909+0000 7f9abb59e640 1 --2- 172.21.15.33:0/3761207984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a5b11b7d0 0x562a5b1336e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4003f10 tx=0x7f9ab4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4aeafb3de6b50ae server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:32.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.909+0000 7f9ab0ff9640 1 -- 172.21.15.33:0/3761207984 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4011050 con 0x562a5b11b7d0 2024-03-19T02:56:32.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.909+0000 7f9ab0ff9640 1 -- 172.21.15.33:0/3761207984 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9ab40111f0 con 0x562a5b11b7d0 2024-03-19T02:56:32.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.909+0000 7f9ab0ff9640 1 -- 172.21.15.33:0/3761207984 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab40105b0 con 0x562a5b11b7d0 2024-03-19T02:56:32.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.909+0000 7f9ac3c30740 1 -- 172.21.15.33:0/3761207984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a5b11b7d0 msgr2=0x562a5b1336e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.909+0000 7f9ac3c30740 1 --2- 172.21.15.33:0/3761207984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a5b11b7d0 0x562a5b1336e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4003f10 tx=0x7f9ab4004ab0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.909+0000 7f9ac3c30740 1 -- 172.21.15.33:0/3761207984 shutdown_connections 2024-03-19T02:56:32.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.909+0000 7f9ac3c30740 1 --2- 172.21.15.33:0/3761207984 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a5b112bf0 0x562a5b117a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.909+0000 7f9ac3c30740 1 --2- 172.21.15.33:0/3761207984 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a5b11b7d0 0x562a5b1336e0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.909+0000 7f9ac3c30740 1 --2- 172.21.15.33:0/3761207984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a5b1122d0 0x562a5b11aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.909+0000 7f9ac3c30740 1 -- 172.21.15.33:0/3761207984 >> 172.21.15.33:0/3761207984 conn(0x562a5b10ca00 msgr2=0x562a5b10cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:32.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.910+0000 7f9ac3c30740 1 -- 172.21.15.33:0/3761207984 shutdown_connections 2024-03-19T02:56:32.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.910+0000 7f9ac3c30740 1 -- 172.21.15.33:0/3761207984 wait complete. 2024-03-19T02:56:32.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.910+0000 7f9ac3c30740 1 -- start start 2024-03-19T02:56:32.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.910+0000 7f9ac3c30740 1 Processor -- start 2024-03-19T02:56:32.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.910+0000 7f9ac3c30740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a5b1122d0 0x562a5b123590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.910+0000 7f9ac3c30740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a5b112bf0 0x562a5b114b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.910+0000 7f9ac3c30740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a5b11b7d0 0x562a5b115070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.910+0000 7f9ac3c30740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9abc1ad230 con 0x562a5b11b7d0 2024-03-19T02:56:32.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.910+0000 7f9ac3c30740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9abc1ae6a0 con 0x562a5b1122d0 2024-03-19T02:56:32.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.910+0000 7f9ac3c30740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9abc1ad900 con 0x562a5b112bf0 2024-03-19T02:56:32.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.910+0000 7f9ac3c30740 10 client.0 _ll_register_callbacks cb 0x7f9aa4009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:56:32.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.910+0000 7f9abad9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a5b1122d0 0x562a5b123590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.910+0000 7f9abb59e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a5b11b7d0 0x562a5b115070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.910+0000 7f9abad9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a5b1122d0 0x562a5b123590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:50030/0 (socket says 172.21.15.33:50030) 2024-03-19T02:56:32.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.910+0000 7f9abad9d640 1 -- 172.21.15.33:0/700155731 learned_addr learned my addr 172.21.15.33:0/700155731 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:32.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.910+0000 7f9ab3fff640 1 --2- 172.21.15.33:0/700155731 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a5b112bf0 0x562a5b114b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.910+0000 7f9abad9d640 1 -- 172.21.15.33:0/700155731 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a5b112bf0 msgr2=0x562a5b114b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:32.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.910+0000 7f9abad9d640 1 --2- 172.21.15.33:0/700155731 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a5b112bf0 0x562a5b114b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.910+0000 7f9abad9d640 1 -- 172.21.15.33:0/700155731 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a5b11b7d0 msgr2=0x562a5b115070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:32.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.910+0000 7f9abad9d640 1 --2- 172.21.15.33:0/700155731 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a5b11b7d0 0x562a5b115070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.910+0000 7f9abad9d640 1 -- 172.21.15.33:0/700155731 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a5b123b00 con 0x562a5b1122d0 2024-03-19T02:56:32.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.910+0000 7f9abb59e640 1 --2- 172.21.15.33:0/700155731 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a5b11b7d0 0x562a5b115070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:32.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.911+0000 7f9abad9d640 1 --2- 172.21.15.33:0/700155731 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a5b1122d0 0x562a5b123590 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9aac0b9940 tx=0x7f9aac009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58f56da24e0d3a52 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:32.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.911+0000 7f9aa27fc640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T02:56:32.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.911+0000 7f9aa27fc640 1 -- 172.21.15.33:0/700155731 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aac0d6040 con 0x562a5b1122d0 2024-03-19T02:56:32.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.911+0000 7f9ac3c30740 1 -- 172.21.15.33:0/700155731 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ab40111f0 con 0x562a5b1122d0 2024-03-19T02:56:32.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.911+0000 7f9aa27fc640 1 -- 172.21.15.33:0/700155731 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f9aac0bd740 con 0x562a5b1122d0 2024-03-19T02:56:32.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.911+0000 7f9aa27fc640 1 -- 172.21.15.33:0/700155731 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9aac0bdbe0 con 0x562a5b1122d0 2024-03-19T02:56:32.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.911+0000 7f9ac3c30740 10 client.26168 fetch_fsmap learned FSMap version 25 2024-03-19T02:56:32.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.911+0000 7f9ac3c30740 1 -- 172.21.15.33:0/700155731 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562a5b191650 con 0x562a5b1122d0 2024-03-19T02:56:32.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.911+0000 7f9aa27fc640 1 -- 172.21.15.33:0/700155731 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9ab40111f0 con 0x562a5b1122d0 2024-03-19T02:56:32.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.911+0000 7f9aa27fc640 1 -- 172.21.15.33:0/700155731 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f9aac0c57f0 con 0x562a5b1122d0 2024-03-19T02:56:32.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.911+0000 7f9ac3c30740 10 client.26168 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:56:32.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.911+0000 7f9ac3c30740 10 client.26168 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:56:32.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.911+0000 7f9ac3c30740 20 client.26168 populate_metadata read hostname 'smithi033' 2024-03-19T02:56:32.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.911+0000 7f9ac3c30740 1 --2- 172.21.15.33:0/700155731 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x562a5b191b10 0x562a5b193f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:32.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.911+0000 7f9ac3c30740 4 client.26168 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:56:32.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.911+0000 7f9ac3c30740 1 -- 172.21.15.33:0/700155731 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562a5b0829f0 con 0x562a5b191b10 2024-03-19T02:56:32.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.911+0000 7f9ab3fff640 1 --2- 172.21.15.33:0/700155731 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x562a5b191b10 0x562a5b193f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:32.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.912+0000 7f9aa27fc640 10 client.26168 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:56:32.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.912+0000 7f9ab3fff640 1 --2- 172.21.15.33:0/700155731 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x562a5b191b10 0x562a5b193f00 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43e2da6c3559b373 server_cookie=dbc43610ee65c5d4 in_seq=0 out_seq=0 2024-03-19T02:56:32.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.912+0000 7f9aa27fc640 1 -- 172.21.15.33:0/700155731 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562a5b0829f0 con 0x562a5b191b10 2024-03-19T02:56:32.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.912+0000 7f9aa27fc640 10 client.26168 handle_command_reply: tid=0 2024-03-19T02:56:32.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.912+0000 7f9ac3c30740 1 client.26168 shutdown 2024-03-19T02:56:32.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.912+0000 7f9ac3c30740 1 -- 172.21.15.33:0/700155731 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a5b1122d0 msgr2=0x562a5b123590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.912+0000 7f9ac3c30740 1 --2- 172.21.15.33:0/700155731 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a5b1122d0 0x562a5b123590 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9aac0b9940 tx=0x7f9aac009260 comp rx=0 tx=0).stop 2024-03-19T02:56:32.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.912+0000 7f9ac3c30740 1 -- 172.21.15.33:0/700155731 shutdown_connections 2024-03-19T02:56:32.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.912+0000 7f9ac3c30740 1 --2- 172.21.15.33:0/700155731 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x562a5b191b10 0x562a5b193f00 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.912+0000 7f9ac3c30740 1 --2- 172.21.15.33:0/700155731 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562a5b112bf0 0x562a5b114b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.912+0000 7f9ac3c30740 1 --2- 172.21.15.33:0/700155731 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562a5b11b7d0 0x562a5b115070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.912+0000 7f9ac3c30740 1 --2- 172.21.15.33:0/700155731 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562a5b1122d0 0x562a5b123590 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.912+0000 7f9aa27fc640 0 client.26168 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:56:32.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.912+0000 7f9ac3c30740 1 -- 172.21.15.33:0/700155731 >> 172.21.15.33:0/700155731 conn(0x562a5b10ca00 msgr2=0x562a5b10de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:32.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.912+0000 7f9ac3c30740 1 -- 172.21.15.33:0/700155731 shutdown_connections 2024-03-19T02:56:32.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.912+0000 7f9ac3c30740 1 -- 172.21.15.33:0/700155731 wait complete. 2024-03-19T02:56:32.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.912+0000 7f9ac3c30740 20 client.26168 trim_cache size 0 max 16384 2024-03-19T02:56:32.920 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:56:32.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.913+0000 7f9ac104d640 1 -- 172.21.15.33:0/3796743462 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f9aa80a2970 msgr2=0x7f9aa80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.913+0000 7f9ac104d640 1 --2- 172.21.15.33:0/3796743462 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f9aa80a2970 0x7f9aa80a4e30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4005c40 tx=0x7f9ab4013040 comp rx=0 tx=0).stop 2024-03-19T02:56:32.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.913+0000 7f9ac104d640 1 -- 172.21.15.33:0/3796743462 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9abc1006a0 msgr2=0x7f9abc198e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:32.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.913+0000 7f9ac104d640 1 --2- 172.21.15.33:0/3796743462 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9abc1006a0 0x7f9abc198e10 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f9aac009230 tx=0x7f9aac02c5d0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.913+0000 7f9ac104d640 1 -- 172.21.15.33:0/3796743462 shutdown_connections 2024-03-19T02:56:32.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.913+0000 7f9ac104d640 1 --2- 172.21.15.33:0/3796743462 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f9aa80a2970 0x7f9aa80a4e30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.913+0000 7f9ac104d640 1 --2- 172.21.15.33:0/3796743462 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9abc1066b0 0x7f9abc1a3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.913+0000 7f9ac104d640 1 --2- 172.21.15.33:0/3796743462 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9abc101060 0x7f9abc199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.913+0000 7f9ac104d640 1 --2- 172.21.15.33:0/3796743462 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9abc1006a0 0x7f9abc198e10 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:32.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.913+0000 7f9ac104d640 1 -- 172.21.15.33:0/3796743462 >> 172.21.15.33:0/3796743462 conn(0x7f9abc0778d0 msgr2=0x7f9abc1030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:32.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.913+0000 7f9ac104d640 1 -- 172.21.15.33:0/3796743462 shutdown_connections 2024-03-19T02:56:32.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:32.913+0000 7f9ac104d640 1 -- 172.21.15.33:0/3796743462 wait complete. 2024-03-19T02:56:33.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:32 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1435450837' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:56:33.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:32 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1435450837' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:56:33.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:32 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1435450837' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:56:33.925 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:56:33.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:56:33.926 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:56:34.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.045+0000 7f7621dd4640 1 -- 172.21.15.33:0/3281788149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f761c102a90 msgr2=0x7f761c102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:34.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.045+0000 7f7621dd4640 1 --2- 172.21.15.33:0/3281788149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f761c102a90 0x7f761c102e70 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f760c009900 tx=0x7f760c02bcd0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.045+0000 7f7621dd4640 1 -- 172.21.15.33:0/3281788149 shutdown_connections 2024-03-19T02:56:34.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.045+0000 7f7621dd4640 1 --2- 172.21.15.33:0/3281788149 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f761c10be60 0x7f761c10e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.045+0000 7f7621dd4640 1 --2- 172.21.15.33:0/3281788149 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f761c1033b0 0x7f761c10b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.045+0000 7f7621dd4640 1 --2- 172.21.15.33:0/3281788149 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f761c102a90 0x7f761c102e70 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.045+0000 7f7621dd4640 1 -- 172.21.15.33:0/3281788149 >> 172.21.15.33:0/3281788149 conn(0x7f761c077880 msgr2=0x7f761c077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:34.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.045+0000 7f7621dd4640 1 -- 172.21.15.33:0/3281788149 shutdown_connections 2024-03-19T02:56:34.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.045+0000 7f7621dd4640 1 -- 172.21.15.33:0/3281788149 wait complete. 2024-03-19T02:56:34.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.046+0000 7f7621dd4640 1 Processor -- start 2024-03-19T02:56:34.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.046+0000 7f7621dd4640 1 -- start start 2024-03-19T02:56:34.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.046+0000 7f7621dd4640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f761c102a90 0x7f761c198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.046+0000 7f7621dd4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f761c1033b0 0x7f761c198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.046+0000 7f7621dd4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f761c10be60 0x7f761c19c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.046+0000 7f7621dd4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f761c111ae0 con 0x7f761c10be60 2024-03-19T02:56:34.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.046+0000 7f7621dd4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f761c111960 con 0x7f761c1033b0 2024-03-19T02:56:34.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.046+0000 7f7621dd4640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f761c111c60 con 0x7f761c102a90 2024-03-19T02:56:34.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.046+0000 7f761bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f761c10be60 0x7f761c19c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.046+0000 7f761affd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f761c1033b0 0x7f761c198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.046+0000 7f761bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f761c10be60 0x7f761c19c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48318/0 (socket says 172.21.15.33:48318) 2024-03-19T02:56:34.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.046+0000 7f761bfff640 1 -- 172.21.15.33:0/2555024655 learned_addr learned my addr 172.21.15.33:0/2555024655 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:34.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.046+0000 7f761affd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f761c1033b0 0x7f761c198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:48674/0 (socket says 172.21.15.33:48674) 2024-03-19T02:56:34.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.046+0000 7f761b7fe640 1 --2- 172.21.15.33:0/2555024655 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f761c102a90 0x7f761c198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.046+0000 7f761bfff640 1 -- 172.21.15.33:0/2555024655 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f761c102a90 msgr2=0x7f761c198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:34.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.046+0000 7f761bfff640 1 --2- 172.21.15.33:0/2555024655 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f761c102a90 0x7f761c198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.046+0000 7f761bfff640 1 -- 172.21.15.33:0/2555024655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f761c1033b0 msgr2=0x7f761c198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:34.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.046+0000 7f761bfff640 1 --2- 172.21.15.33:0/2555024655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f761c1033b0 0x7f761c198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.047+0000 7f761bfff640 1 -- 172.21.15.33:0/2555024655 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f761c19d010 con 0x7f761c10be60 2024-03-19T02:56:34.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.047+0000 7f761affd640 1 --2- 172.21.15.33:0/2555024655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f761c1033b0 0x7f761c198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:34.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.047+0000 7f761bfff640 1 --2- 172.21.15.33:0/2555024655 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f761c10be60 0x7f761c19c8e0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f761400eb10 tx=0x7f761400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:34.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.047+0000 7f7618ff9640 1 -- 172.21.15.33:0/2555024655 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7614004b20 con 0x7f761c10be60 2024-03-19T02:56:34.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.047+0000 7f7618ff9640 1 -- 172.21.15.33:0/2555024655 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7614004cc0 con 0x7f761c10be60 2024-03-19T02:56:34.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.047+0000 7f7621dd4640 1 -- 172.21.15.33:0/2555024655 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f761c19d300 con 0x7f761c10be60 2024-03-19T02:56:34.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.047+0000 7f7618ff9640 1 -- 172.21.15.33:0/2555024655 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7614010690 con 0x7f761c10be60 2024-03-19T02:56:34.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.047+0000 7f7621dd4640 1 -- 172.21.15.33:0/2555024655 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f761c1a4b90 con 0x7f761c10be60 2024-03-19T02:56:34.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.049+0000 7f7618ff9640 1 -- 172.21.15.33:0/2555024655 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f7614005020 con 0x7f761c10be60 2024-03-19T02:56:34.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.049+0000 7f76249c4740 1 Processor -- start 2024-03-19T02:56:34.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.049+0000 7f76249c4740 1 -- start start 2024-03-19T02:56:34.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.050+0000 7f76249c4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5583cbe50380 0x5583cbe50760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.050+0000 7f76249c4740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5583cbe50ca0 0x5583cbe59340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.050+0000 7f76249c4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5583cbe59880 0x5583cbe5bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.050+0000 7f76249c4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5583cbdb4160 con 0x5583cbe50380 2024-03-19T02:56:34.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.050+0000 7f76249c4740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5583cbd5ce10 con 0x5583cbe50ca0 2024-03-19T02:56:34.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.050+0000 7f76249c4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5583cbdb0200 con 0x5583cbe59880 2024-03-19T02:56:34.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.050+0000 7f761affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5583cbe50380 0x5583cbe50760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.050+0000 7f761affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5583cbe50380 0x5583cbe50760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48334/0 (socket says 172.21.15.33:48334) 2024-03-19T02:56:34.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.050+0000 7f761affd640 1 -- 172.21.15.33:0/1701299377 learned_addr learned my addr 172.21.15.33:0/1701299377 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:34.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.050+0000 7f761b7fe640 1 --2- 172.21.15.33:0/1701299377 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5583cbe50ca0 0x5583cbe59340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.050+0000 7f761affd640 1 --2- 172.21.15.33:0/1701299377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5583cbe59880 0x5583cbe5bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.050+0000 7f7618ff9640 1 --2- 172.21.15.33:0/2555024655 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f76040a2970 0x7f76040a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.050+0000 7f761affd640 1 -- 172.21.15.33:0/1701299377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5583cbe59880 msgr2=0x5583cbe5bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:34.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.050+0000 7f761affd640 1 --2- 172.21.15.33:0/1701299377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5583cbe59880 0x5583cbe5bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.050+0000 7f761affd640 1 -- 172.21.15.33:0/1701299377 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5583cbe50ca0 msgr2=0x5583cbe59340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:34.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.050+0000 7f761affd640 1 --2- 172.21.15.33:0/1701299377 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5583cbe50ca0 0x5583cbe59340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.050+0000 7f761affd640 1 -- 172.21.15.33:0/1701299377 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583cbb907e0 con 0x5583cbe50380 2024-03-19T02:56:34.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.050+0000 7f761bfff640 1 --2- 172.21.15.33:0/2555024655 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f76040a2970 0x7f76040a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.050+0000 7f7618ff9640 1 -- 172.21.15.33:0/2555024655 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f76140cf800 con 0x7f761c10be60 2024-03-19T02:56:34.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.050+0000 7f761affd640 1 --2- 172.21.15.33:0/1701299377 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5583cbe50380 0x5583cbe50760 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f76100139a0 tx=0x7f7610009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25a1773b80b09399 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:34.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.051+0000 7f7601ffb640 1 -- 172.21.15.33:0/1701299377 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f761000f940 con 0x5583cbe50380 2024-03-19T02:56:34.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.051+0000 7f7601ffb640 1 -- 172.21.15.33:0/1701299377 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7610010430 con 0x5583cbe50380 2024-03-19T02:56:34.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.051+0000 7f7601ffb640 1 -- 172.21.15.33:0/1701299377 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f761000e610 con 0x5583cbe50380 2024-03-19T02:56:34.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.051+0000 7f76249c4740 1 -- 172.21.15.33:0/1701299377 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5583cbe50380 msgr2=0x5583cbe50760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:34.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.051+0000 7f76249c4740 1 --2- 172.21.15.33:0/1701299377 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5583cbe50380 0x5583cbe50760 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f76100139a0 tx=0x7f7610009c00 comp rx=0 tx=0).stop 2024-03-19T02:56:34.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.051+0000 7f761bfff640 1 --2- 172.21.15.33:0/2555024655 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f76040a2970 0x7f76040a4e30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7614097f50 tx=0x7f761401b8f0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:34.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.051+0000 7f76249c4740 1 -- 172.21.15.33:0/1701299377 shutdown_connections 2024-03-19T02:56:34.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.051+0000 7f76249c4740 1 --2- 172.21.15.33:0/1701299377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5583cbe59880 0x5583cbe5bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.051+0000 7f76249c4740 1 --2- 172.21.15.33:0/1701299377 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5583cbe50ca0 0x5583cbe59340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.051+0000 7f76249c4740 1 --2- 172.21.15.33:0/1701299377 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5583cbe50380 0x5583cbe50760 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.051+0000 7f76249c4740 1 -- 172.21.15.33:0/1701299377 >> 172.21.15.33:0/1701299377 conn(0x5583cbe4aab0 msgr2=0x5583cbe4ae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:34.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.051+0000 7f76249c4740 1 -- 172.21.15.33:0/1701299377 shutdown_connections 2024-03-19T02:56:34.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.051+0000 7f76249c4740 1 -- 172.21.15.33:0/1701299377 wait complete. 2024-03-19T02:56:34.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.052+0000 7f76249c4740 1 -- start start 2024-03-19T02:56:34.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.052+0000 7f76249c4740 1 Processor -- start 2024-03-19T02:56:34.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.052+0000 7f76249c4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5583cbe50380 0x5583cbe56820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.052+0000 7f76249c4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5583cbe50ca0 0x5583cbe56d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.052+0000 7f76249c4740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5583cbe59880 0x5583cbe572a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.052+0000 7f76249c4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5583cbe5c670 con 0x5583cbe50380 2024-03-19T02:56:34.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.052+0000 7f76249c4740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5583cbd701a0 con 0x5583cbe59880 2024-03-19T02:56:34.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.052+0000 7f76249c4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5583cbe5c7f0 con 0x5583cbe50ca0 2024-03-19T02:56:34.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.052+0000 7f76249c4740 10 client.0 _ll_register_callbacks cb 0x5583cbac7c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:56:34.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.052+0000 7f761affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5583cbe50380 0x5583cbe56820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.052+0000 7f761affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5583cbe50380 0x5583cbe56820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48340/0 (socket says 172.21.15.33:48340) 2024-03-19T02:56:34.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.052+0000 7f761affd640 1 -- 172.21.15.33:0/2211478912 learned_addr learned my addr 172.21.15.33:0/2211478912 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:34.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.052+0000 7f761b7fe640 1 --2- 172.21.15.33:0/2211478912 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5583cbe50ca0 0x5583cbe56d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.052+0000 7f761affd640 1 --2- 172.21.15.33:0/2211478912 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5583cbe59880 0x5583cbe572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.053+0000 7f761affd640 1 -- 172.21.15.33:0/2211478912 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5583cbe50ca0 msgr2=0x5583cbe56d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:34.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.053+0000 7f761affd640 1 --2- 172.21.15.33:0/2211478912 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5583cbe50ca0 0x5583cbe56d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.053+0000 7f761affd640 1 -- 172.21.15.33:0/2211478912 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5583cbe59880 msgr2=0x5583cbe572a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:34.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.053+0000 7f761affd640 1 --2- 172.21.15.33:0/2211478912 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5583cbe59880 0x5583cbe572a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.053+0000 7f761affd640 1 -- 172.21.15.33:0/2211478912 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583cbbf0f70 con 0x5583cbe50380 2024-03-19T02:56:34.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.053+0000 7f761affd640 1 --2- 172.21.15.33:0/2211478912 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5583cbe50380 0x5583cbe56820 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f76100093e0 tx=0x7f7610005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bef09978501fca8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:34.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.053+0000 7f75faffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:56:34.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.053+0000 7f75faffd640 1 -- 172.21.15.33:0/2211478912 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7610024070 con 0x5583cbe50380 2024-03-19T02:56:34.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.053+0000 7f75faffd640 1 -- 172.21.15.33:0/2211478912 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f761000fd50 con 0x5583cbe50380 2024-03-19T02:56:34.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.053+0000 7f76249c4740 1 -- 172.21.15.33:0/2211478912 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7610010430 con 0x5583cbe50380 2024-03-19T02:56:34.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.053+0000 7f75faffd640 1 -- 172.21.15.33:0/2211478912 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7610038050 con 0x5583cbe50380 2024-03-19T02:56:34.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.053+0000 7f76249c4740 10 client.16356 fetch_fsmap learned FSMap version 25 2024-03-19T02:56:34.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.053+0000 7f76249c4740 1 -- 172.21.15.33:0/2211478912 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5583cba88170 con 0x5583cbe50380 2024-03-19T02:56:34.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.053+0000 7f75faffd640 1 -- 172.21.15.33:0/2211478912 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7610010430 con 0x5583cbe50380 2024-03-19T02:56:34.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.053+0000 7f75faffd640 1 -- 172.21.15.33:0/2211478912 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f7610004980 con 0x5583cbe50380 2024-03-19T02:56:34.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.054+0000 7f76249c4740 10 client.16356 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:56:34.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.054+0000 7f76249c4740 10 client.16356 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:56:34.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.054+0000 7f76249c4740 20 client.16356 populate_metadata read hostname 'smithi033' 2024-03-19T02:56:34.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.054+0000 7f76249c4740 1 --2- 172.21.15.33:0/2211478912 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5583cbe69ab0 0x5583cbe6bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.054+0000 7f76249c4740 4 client.16356 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:56:34.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.054+0000 7f76249c4740 1 -- 172.21.15.33:0/2211478912 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5583cbdc16a0 con 0x5583cbe69ab0 2024-03-19T02:56:34.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.054+0000 7f761b7fe640 1 --2- 172.21.15.33:0/2211478912 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5583cbe69ab0 0x5583cbe6bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.054+0000 7f75faffd640 10 client.16356 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:56:34.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.054+0000 7f761b7fe640 1 --2- 172.21.15.33:0/2211478912 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5583cbe69ab0 0x5583cbe6bea0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf27defd48a864f server_cookie=2d7124800862bee9 in_seq=0 out_seq=0 2024-03-19T02:56:34.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.055+0000 7f75faffd640 1 -- 172.21.15.33:0/2211478912 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5583cbdc16a0 con 0x5583cbe69ab0 2024-03-19T02:56:34.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.055+0000 7f75faffd640 10 client.16356 handle_command_reply: tid=0 2024-03-19T02:56:34.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.055+0000 7f76249c4740 1 client.16356 shutdown 2024-03-19T02:56:34.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.055+0000 7f76249c4740 1 -- 172.21.15.33:0/2211478912 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5583cbe50380 msgr2=0x5583cbe56820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:34.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.055+0000 7f76249c4740 1 --2- 172.21.15.33:0/2211478912 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5583cbe50380 0x5583cbe56820 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f76100093e0 tx=0x7f7610005f90 comp rx=0 tx=0).stop 2024-03-19T02:56:34.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.055+0000 7f76249c4740 1 -- 172.21.15.33:0/2211478912 shutdown_connections 2024-03-19T02:56:34.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.055+0000 7f76249c4740 1 --2- 172.21.15.33:0/2211478912 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5583cbe69ab0 0x5583cbe6bea0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.055+0000 7f76249c4740 1 --2- 172.21.15.33:0/2211478912 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5583cbe50ca0 0x5583cbe56d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.055+0000 7f76249c4740 1 --2- 172.21.15.33:0/2211478912 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5583cbe59880 0x5583cbe572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.055+0000 7f75faffd640 0 client.16356 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:56:34.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.055+0000 7f76249c4740 1 --2- 172.21.15.33:0/2211478912 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5583cbe50380 0x5583cbe56820 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.055+0000 7f76249c4740 1 -- 172.21.15.33:0/2211478912 >> 172.21.15.33:0/2211478912 conn(0x5583cbe4aab0 msgr2=0x5583cbe4c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:34.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.055+0000 7f76249c4740 1 -- 172.21.15.33:0/2211478912 shutdown_connections 2024-03-19T02:56:34.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.055+0000 7f76249c4740 1 -- 172.21.15.33:0/2211478912 wait complete. 2024-03-19T02:56:34.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.055+0000 7f76249c4740 20 client.16356 trim_cache size 0 max 16384 2024-03-19T02:56:34.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.073+0000 7f76249c4740 1 Processor -- start 2024-03-19T02:56:34.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.073+0000 7f76249c4740 1 -- start start 2024-03-19T02:56:34.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.074+0000 7f76249c4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5583cbe50380 0x5583cbe58f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.074+0000 7f76249c4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5583cbe50ca0 0x5583cbe55fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.074+0000 7f76249c4740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5583cbe59880 0x5583cbe71630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.074+0000 7f76249c4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f761c1b1490 con 0x5583cbe50ca0 2024-03-19T02:56:34.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.074+0000 7f76249c4740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f761c1aed60 con 0x5583cbe59880 2024-03-19T02:56:34.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.074+0000 7f76249c4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f761c1acb90 con 0x5583cbe50380 2024-03-19T02:56:34.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.074+0000 7f761b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5583cbe50ca0 0x5583cbe55fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.074+0000 7f761b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5583cbe50ca0 0x5583cbe55fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48346/0 (socket says 172.21.15.33:48346) 2024-03-19T02:56:34.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.074+0000 7f761b7fe640 1 -- 172.21.15.33:0/1111586910 learned_addr learned my addr 172.21.15.33:0/1111586910 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:34.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.074+0000 7f761affd640 1 --2- 172.21.15.33:0/1111586910 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5583cbe50380 0x5583cbe58f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.074+0000 7f761affd640 1 --2- 172.21.15.33:0/1111586910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5583cbe59880 0x5583cbe71630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.074+0000 7f761b7fe640 1 -- 172.21.15.33:0/1111586910 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5583cbe50380 msgr2=0x5583cbe58f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:34.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.074+0000 7f761b7fe640 1 --2- 172.21.15.33:0/1111586910 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5583cbe50380 0x5583cbe58f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.074+0000 7f761b7fe640 1 -- 172.21.15.33:0/1111586910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5583cbe59880 msgr2=0x5583cbe71630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:34.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.074+0000 7f761b7fe640 1 --2- 172.21.15.33:0/1111586910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5583cbe59880 0x5583cbe71630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.074+0000 7f761b7fe640 1 -- 172.21.15.33:0/1111586910 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583cbe50ad0 con 0x5583cbe50ca0 2024-03-19T02:56:34.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.074+0000 7f761b7fe640 1 --2- 172.21.15.33:0/1111586910 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5583cbe50ca0 0x5583cbe55fd0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f760c02c210 tx=0x7f760c02c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e129ede88ae077ce server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:34.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.074+0000 7f76017fa640 1 -- 172.21.15.33:0/1111586910 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f760c0042a0 con 0x5583cbe50ca0 2024-03-19T02:56:34.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.074+0000 7f76017fa640 1 -- 172.21.15.33:0/1111586910 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f760c004440 con 0x5583cbe50ca0 2024-03-19T02:56:34.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.075+0000 7f76017fa640 1 -- 172.21.15.33:0/1111586910 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f760c0055c0 con 0x5583cbe50ca0 2024-03-19T02:56:34.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.075+0000 7f76249c4740 1 -- 172.21.15.33:0/1111586910 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5583cbe50ca0 msgr2=0x5583cbe55fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:34.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.075+0000 7f76249c4740 1 --2- 172.21.15.33:0/1111586910 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5583cbe50ca0 0x5583cbe55fd0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f760c02c210 tx=0x7f760c02c480 comp rx=0 tx=0).stop 2024-03-19T02:56:34.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.075+0000 7f76249c4740 1 -- 172.21.15.33:0/1111586910 shutdown_connections 2024-03-19T02:56:34.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.075+0000 7f76249c4740 1 --2- 172.21.15.33:0/1111586910 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5583cbe59880 0x5583cbe71630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.075+0000 7f76249c4740 1 --2- 172.21.15.33:0/1111586910 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5583cbe50ca0 0x5583cbe55fd0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.075+0000 7f76249c4740 1 --2- 172.21.15.33:0/1111586910 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5583cbe50380 0x5583cbe58f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.075+0000 7f76249c4740 1 -- 172.21.15.33:0/1111586910 >> 172.21.15.33:0/1111586910 conn(0x5583cbe4aab0 msgr2=0x5583cbe4c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:34.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.075+0000 7f76249c4740 1 -- 172.21.15.33:0/1111586910 shutdown_connections 2024-03-19T02:56:34.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.075+0000 7f76249c4740 1 -- 172.21.15.33:0/1111586910 wait complete. 2024-03-19T02:56:34.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.076+0000 7f76249c4740 1 -- start start 2024-03-19T02:56:34.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.076+0000 7f76249c4740 1 Processor -- start 2024-03-19T02:56:34.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.076+0000 7f76249c4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5583cbe50380 0x5583cbe619b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.076+0000 7f76249c4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5583cbe50ca0 0x5583cbe61ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.076+0000 7f76249c4740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5583cbe59880 0x5583cbe5dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.076+0000 7f76249c4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f761c1ad220 con 0x5583cbe50380 2024-03-19T02:56:34.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.076+0000 7f76249c4740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f761c1ae690 con 0x5583cbe59880 2024-03-19T02:56:34.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.076+0000 7f76249c4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f761c1ad8f0 con 0x5583cbe50ca0 2024-03-19T02:56:34.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.076+0000 7f76249c4740 10 client.0 _ll_register_callbacks cb 0x7f76140093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:56:34.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.076+0000 7f761affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5583cbe50380 0x5583cbe619b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.076+0000 7f761affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5583cbe50380 0x5583cbe619b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48356/0 (socket says 172.21.15.33:48356) 2024-03-19T02:56:34.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.076+0000 7f761affd640 1 -- 172.21.15.33:0/1599229943 learned_addr learned my addr 172.21.15.33:0/1599229943 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:34.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.076+0000 7f761b7fe640 1 --2- 172.21.15.33:0/1599229943 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5583cbe50ca0 0x5583cbe61ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.076+0000 7f761affd640 1 --2- 172.21.15.33:0/1599229943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5583cbe59880 0x5583cbe5dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.076+0000 7f761affd640 1 -- 172.21.15.33:0/1599229943 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5583cbe50ca0 msgr2=0x5583cbe61ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:34.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.076+0000 7f761affd640 1 --2- 172.21.15.33:0/1599229943 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5583cbe50ca0 0x5583cbe61ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.076+0000 7f761affd640 1 -- 172.21.15.33:0/1599229943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5583cbe59880 msgr2=0x5583cbe5dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:34.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.076+0000 7f761affd640 1 --2- 172.21.15.33:0/1599229943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5583cbe59880 0x5583cbe5dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.076+0000 7f761affd640 1 -- 172.21.15.33:0/1599229943 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583cbe5e6f0 con 0x5583cbe50380 2024-03-19T02:56:34.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.077+0000 7f761affd640 1 --2- 172.21.15.33:0/1599229943 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5583cbe50380 0x5583cbe619b0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f76100095f0 tx=0x7f7610005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a4dd2d88d65715e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:34.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.077+0000 7f75faffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:56:34.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.077+0000 7f75faffd640 1 -- 172.21.15.33:0/1599229943 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7610024050 con 0x5583cbe50380 2024-03-19T02:56:34.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.077+0000 7f75faffd640 1 -- 172.21.15.33:0/1599229943 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f7610010db0 con 0x5583cbe50380 2024-03-19T02:56:34.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.077+0000 7f76249c4740 1 -- 172.21.15.33:0/1599229943 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f760c004440 con 0x5583cbe50380 2024-03-19T02:56:34.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.077+0000 7f75faffd640 1 -- 172.21.15.33:0/1599229943 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7610010970 con 0x5583cbe50380 2024-03-19T02:56:34.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.077+0000 7f76249c4740 10 client.16362 fetch_fsmap learned FSMap version 25 2024-03-19T02:56:34.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.077+0000 7f76249c4740 1 -- 172.21.15.33:0/1599229943 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5583cbed0910 con 0x5583cbe50380 2024-03-19T02:56:34.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.077+0000 7f75faffd640 1 -- 172.21.15.33:0/1599229943 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f760c004440 con 0x5583cbe50380 2024-03-19T02:56:34.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.077+0000 7f75faffd640 1 -- 172.21.15.33:0/1599229943 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f7610005570 con 0x5583cbe50380 2024-03-19T02:56:34.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.077+0000 7f76249c4740 10 client.16362 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:56:34.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.077+0000 7f76249c4740 10 client.16362 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:56:34.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.077+0000 7f76249c4740 20 client.16362 populate_metadata read hostname 'smithi033' 2024-03-19T02:56:34.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.077+0000 7f76249c4740 1 --2- 172.21.15.33:0/1599229943 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5583cbed0fa0 0x5583cbed33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.077+0000 7f76249c4740 4 client.16362 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:56:34.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.077+0000 7f76249c4740 1 -- 172.21.15.33:0/1599229943 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5583cbdbb9f0 con 0x5583cbed0fa0 2024-03-19T02:56:34.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.078+0000 7f761b7fe640 1 --2- 172.21.15.33:0/1599229943 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5583cbed0fa0 0x5583cbed33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.078+0000 7f761b7fe640 1 --2- 172.21.15.33:0/1599229943 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5583cbed0fa0 0x5583cbed33c0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a3fc8cd5a31e28a server_cookie=23fbadd830b670a5 in_seq=0 out_seq=0 2024-03-19T02:56:34.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.078+0000 7f75faffd640 10 client.16362 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:56:34.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.079+0000 7f75faffd640 1 -- 172.21.15.33:0/1599229943 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5583cbdbb9f0 con 0x5583cbed0fa0 2024-03-19T02:56:34.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.079+0000 7f75faffd640 10 client.16362 handle_command_reply: tid=0 2024-03-19T02:56:34.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.079+0000 7f76249c4740 1 client.16362 shutdown 2024-03-19T02:56:34.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.079+0000 7f76249c4740 1 -- 172.21.15.33:0/1599229943 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5583cbe50380 msgr2=0x5583cbe619b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:34.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.079+0000 7f76249c4740 1 --2- 172.21.15.33:0/1599229943 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5583cbe50380 0x5583cbe619b0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f76100095f0 tx=0x7f7610005f90 comp rx=0 tx=0).stop 2024-03-19T02:56:34.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.079+0000 7f76249c4740 1 -- 172.21.15.33:0/1599229943 shutdown_connections 2024-03-19T02:56:34.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.079+0000 7f76249c4740 1 --2- 172.21.15.33:0/1599229943 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5583cbed0fa0 0x5583cbed33c0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.079+0000 7f76249c4740 1 --2- 172.21.15.33:0/1599229943 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5583cbe50ca0 0x5583cbe61ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.079+0000 7f76249c4740 1 --2- 172.21.15.33:0/1599229943 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5583cbe59880 0x5583cbe5dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.079+0000 7f75faffd640 0 client.16362 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:56:34.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.079+0000 7f76249c4740 1 --2- 172.21.15.33:0/1599229943 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5583cbe50380 0x5583cbe619b0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.079+0000 7f76249c4740 1 -- 172.21.15.33:0/1599229943 >> 172.21.15.33:0/1599229943 conn(0x5583cbe4aab0 msgr2=0x5583cbe57ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:34.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.079+0000 7f76249c4740 1 -- 172.21.15.33:0/1599229943 shutdown_connections 2024-03-19T02:56:34.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.079+0000 7f76249c4740 1 -- 172.21.15.33:0/1599229943 wait complete. 2024-03-19T02:56:34.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.079+0000 7f76249c4740 20 client.16362 trim_cache size 0 max 16384 2024-03-19T02:56:34.091 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:56:34.091 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-03-19T02:56:34.091 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "75433454-729c-4fb8-ae84-9b56d7f302cc", 2024-03-19T02:56:34.092 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-03-19T02:56:34.092 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:56:34.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.080+0000 7f7621dd4640 1 -- 172.21.15.33:0/2555024655 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f76040a2970 msgr2=0x7f76040a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:34.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.080+0000 7f7621dd4640 1 --2- 172.21.15.33:0/2555024655 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f76040a2970 0x7f76040a4e30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7614097f50 tx=0x7f761401b8f0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.080+0000 7f7621dd4640 1 -- 172.21.15.33:0/2555024655 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f761c10be60 msgr2=0x7f761c19c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:34.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.080+0000 7f7621dd4640 1 --2- 172.21.15.33:0/2555024655 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f761c10be60 0x7f761c19c8e0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f761400eb10 tx=0x7f761400c2d0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.080+0000 7f7621dd4640 1 -- 172.21.15.33:0/2555024655 shutdown_connections 2024-03-19T02:56:34.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.080+0000 7f7621dd4640 1 --2- 172.21.15.33:0/2555024655 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f76040a2970 0x7f76040a4e30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.080+0000 7f7621dd4640 1 --2- 172.21.15.33:0/2555024655 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f761c10be60 0x7f761c19c8e0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.080+0000 7f7621dd4640 1 --2- 172.21.15.33:0/2555024655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f761c1033b0 0x7f761c198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.080+0000 7f7621dd4640 1 --2- 172.21.15.33:0/2555024655 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f761c102a90 0x7f761c198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.080+0000 7f7621dd4640 1 -- 172.21.15.33:0/2555024655 >> 172.21.15.33:0/2555024655 conn(0x7f761c077880 msgr2=0x7f761c0fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:34.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.080+0000 7f7621dd4640 1 -- 172.21.15.33:0/2555024655 shutdown_connections 2024-03-19T02:56:34.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.080+0000 7f7621dd4640 1 -- 172.21.15.33:0/2555024655 wait complete. 2024-03-19T02:56:34.096 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:56:34.096 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:56:34.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.200+0000 7fbe640ff640 1 -- 172.21.15.33:0/1105829460 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe5c101090 msgr2=0x7fbe5c10d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:34.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.200+0000 7fbe640ff640 1 --2- 172.21.15.33:0/1105829460 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe5c101090 0x7fbe5c10d5a0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fbe54009cd0 tx=0x7fbe5402c0f0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.200+0000 7fbe640ff640 1 -- 172.21.15.33:0/1105829460 shutdown_connections 2024-03-19T02:56:34.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.200+0000 7fbe640ff640 1 --2- 172.21.15.33:0/1105829460 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe5c101090 0x7fbe5c10d5a0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.200+0000 7fbe640ff640 1 --2- 172.21.15.33:0/1105829460 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe5c1006d0 0x7fbe5c100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.200+0000 7fbe640ff640 1 --2- 172.21.15.33:0/1105829460 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe5c1066e0 0x7fbe5c106ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.200+0000 7fbe640ff640 1 -- 172.21.15.33:0/1105829460 >> 172.21.15.33:0/1105829460 conn(0x7fbe5c076aa0 msgr2=0x7fbe5c076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:34.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.200+0000 7fbe640ff640 1 -- 172.21.15.33:0/1105829460 shutdown_connections 2024-03-19T02:56:34.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.200+0000 7fbe640ff640 1 -- 172.21.15.33:0/1105829460 wait complete. 2024-03-19T02:56:34.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.201+0000 7fbe640ff640 1 Processor -- start 2024-03-19T02:56:34.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.201+0000 7fbe640ff640 1 -- start start 2024-03-19T02:56:34.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.201+0000 7fbe640ff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe5c1006d0 0x7fbe5c198de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.201+0000 7fbe640ff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe5c101090 0x7fbe5c199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.201+0000 7fbe640ff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe5c1066e0 0x7fbe5c1a3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.201+0000 7fbe640ff640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbe5c1109a0 con 0x7fbe5c101090 2024-03-19T02:56:34.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.201+0000 7fbe640ff640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbe5c110820 con 0x7fbe5c1006d0 2024-03-19T02:56:34.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.201+0000 7fbe640ff640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbe5c110b20 con 0x7fbe5c1066e0 2024-03-19T02:56:34.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.201+0000 7fbe6169a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe5c101090 0x7fbe5c199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.201+0000 7fbe6169a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe5c101090 0x7fbe5c199320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48374/0 (socket says 172.21.15.33:48374) 2024-03-19T02:56:34.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.201+0000 7fbe6169a640 1 -- 172.21.15.33:0/3994333313 learned_addr learned my addr 172.21.15.33:0/3994333313 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:34.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.201+0000 7fbe61e9b640 1 --2- 172.21.15.33:0/3994333313 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe5c1006d0 0x7fbe5c198de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.201+0000 7fbe6269c640 1 --2- 172.21.15.33:0/3994333313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe5c1066e0 0x7fbe5c1a3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.201+0000 7fbe6169a640 1 -- 172.21.15.33:0/3994333313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe5c1066e0 msgr2=0x7fbe5c1a3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:34.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.201+0000 7fbe6169a640 1 --2- 172.21.15.33:0/3994333313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe5c1066e0 0x7fbe5c1a3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.201+0000 7fbe6169a640 1 -- 172.21.15.33:0/3994333313 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe5c1006d0 msgr2=0x7fbe5c198de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:34.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.201+0000 7fbe6169a640 1 --2- 172.21.15.33:0/3994333313 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe5c1006d0 0x7fbe5c198de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.202+0000 7fbe6169a640 1 -- 172.21.15.33:0/3994333313 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe5c1a43a0 con 0x7fbe5c101090 2024-03-19T02:56:34.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.202+0000 7fbe6169a640 1 --2- 172.21.15.33:0/3994333313 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe5c101090 0x7fbe5c199320 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c00ea60 tx=0x7fbe4c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:34.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.202+0000 7fbe52ffd640 1 -- 172.21.15.33:0/3994333313 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c00cde0 con 0x7fbe5c101090 2024-03-19T02:56:34.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.202+0000 7fbe52ffd640 1 -- 172.21.15.33:0/3994333313 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe4c0035f0 con 0x7fbe5c101090 2024-03-19T02:56:34.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.202+0000 7fbe640ff640 1 -- 172.21.15.33:0/3994333313 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe5c1a45d0 con 0x7fbe5c101090 2024-03-19T02:56:34.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.202+0000 7fbe52ffd640 1 -- 172.21.15.33:0/3994333313 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c010690 con 0x7fbe5c101090 2024-03-19T02:56:34.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.202+0000 7fbe640ff640 1 -- 172.21.15.33:0/3994333313 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe5c1a4b40 con 0x7fbe5c101090 2024-03-19T02:56:34.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.204+0000 7fbe52ffd640 1 -- 172.21.15.33:0/3994333313 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fbe4c0037a0 con 0x7fbe5c101090 2024-03-19T02:56:34.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.204+0000 7fbe66dc7740 1 Processor -- start 2024-03-19T02:56:34.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.205+0000 7fbe66dc7740 1 -- start start 2024-03-19T02:56:34.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.205+0000 7fbe66dc7740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bd097a2a30 0x55bd097a2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.205+0000 7fbe66dc7740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bd097a3350 0x55bd097ab9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.205+0000 7fbe66dc7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd097abf30 0x55bd097ae320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.205+0000 7fbe66dc7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55bd09706500 con 0x55bd097abf30 2024-03-19T02:56:34.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.205+0000 7fbe66dc7740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55bd096aebe0 con 0x55bd097a3350 2024-03-19T02:56:34.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.205+0000 7fbe66dc7740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55bd097025a0 con 0x55bd097a2a30 2024-03-19T02:56:34.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.205+0000 7fbe52ffd640 1 --2- 172.21.15.33:0/3994333313 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbe480a29f0 0x7fbe480a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.205+0000 7fbe6169a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd097abf30 0x55bd097ae320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.205+0000 7fbe6269c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bd097a2a30 0x55bd097a2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.205+0000 7fbe6269c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bd097a2a30 0x55bd097a2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:40128/0 (socket says 172.21.15.33:40128) 2024-03-19T02:56:34.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.205+0000 7fbe6269c640 1 -- 172.21.15.33:0/1288923189 learned_addr learned my addr 172.21.15.33:0/1288923189 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:34.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.205+0000 7fbe61e9b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bd097a3350 0x55bd097ab9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.205+0000 7fbe6269c640 1 --2- 172.21.15.33:0/3994333313 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbe480a29f0 0x7fbe480a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.205+0000 7fbe52ffd640 1 -- 172.21.15.33:0/3994333313 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fbe4c0cf3b0 con 0x7fbe5c101090 2024-03-19T02:56:34.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.205+0000 7fbe61e9b640 1 -- 172.21.15.33:0/1288923189 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bd097a2a30 msgr2=0x55bd097a2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:34.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.205+0000 7fbe61e9b640 1 --2- 172.21.15.33:0/1288923189 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bd097a2a30 0x55bd097a2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.205+0000 7fbe61e9b640 1 -- 172.21.15.33:0/1288923189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd097abf30 msgr2=0x55bd097ae320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:34.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.205+0000 7fbe61e9b640 1 --2- 172.21.15.33:0/1288923189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd097abf30 0x55bd097ae320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.205+0000 7fbe61e9b640 1 -- 172.21.15.33:0/1288923189 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd096d0000 con 0x55bd097a3350 2024-03-19T02:56:34.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.205+0000 7fbe61e9b640 1 --2- 172.21.15.33:0/1288923189 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bd097a3350 0x55bd097ab9f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fbe58009c20 tx=0x7fbe58011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4eff52867d0a0602 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:34.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.205+0000 7fbe6269c640 1 --2- 172.21.15.33:0/3994333313 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbe480a29f0 0x7fbe480a4eb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbe54006010 tx=0x7fbe54005c50 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:34.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.206+0000 7fbe47fff640 1 -- 172.21.15.33:0/1288923189 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe58004050 con 0x55bd097a3350 2024-03-19T02:56:34.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.206+0000 7fbe47fff640 1 -- 172.21.15.33:0/1288923189 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe580054d0 con 0x55bd097a3350 2024-03-19T02:56:34.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.206+0000 7fbe47fff640 1 -- 172.21.15.33:0/1288923189 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe58004c20 con 0x55bd097a3350 2024-03-19T02:56:34.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.206+0000 7fbe66dc7740 1 -- 172.21.15.33:0/1288923189 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bd097a3350 msgr2=0x55bd097ab9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:34.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.206+0000 7fbe66dc7740 1 --2- 172.21.15.33:0/1288923189 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bd097a3350 0x55bd097ab9f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fbe58009c20 tx=0x7fbe58011810 comp rx=0 tx=0).stop 2024-03-19T02:56:34.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.206+0000 7fbe66dc7740 1 -- 172.21.15.33:0/1288923189 shutdown_connections 2024-03-19T02:56:34.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.206+0000 7fbe66dc7740 1 --2- 172.21.15.33:0/1288923189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd097abf30 0x55bd097ae320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.206+0000 7fbe66dc7740 1 --2- 172.21.15.33:0/1288923189 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bd097a3350 0x55bd097ab9f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.206+0000 7fbe66dc7740 1 --2- 172.21.15.33:0/1288923189 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bd097a2a30 0x55bd097a2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.206+0000 7fbe66dc7740 1 -- 172.21.15.33:0/1288923189 >> 172.21.15.33:0/1288923189 conn(0x55bd0979d160 msgr2=0x55bd0979d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:34.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.207+0000 7fbe66dc7740 1 -- 172.21.15.33:0/1288923189 shutdown_connections 2024-03-19T02:56:34.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.207+0000 7fbe66dc7740 1 -- 172.21.15.33:0/1288923189 wait complete. 2024-03-19T02:56:34.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.207+0000 7fbe66dc7740 1 -- start start 2024-03-19T02:56:34.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.207+0000 7fbe66dc7740 1 Processor -- start 2024-03-19T02:56:34.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.207+0000 7fbe66dc7740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bd097a2a30 0x55bd097a8e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.207+0000 7fbe66dc7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd097a3350 0x55bd097a9390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.207+0000 7fbe66dc7740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bd097abf30 0x55bd097a98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.207+0000 7fbe66dc7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55bd097aed20 con 0x55bd097a3350 2024-03-19T02:56:34.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.207+0000 7fbe66dc7740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55bd096c2160 con 0x55bd097a2a30 2024-03-19T02:56:34.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.207+0000 7fbe66dc7740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55bd097aeea0 con 0x55bd097abf30 2024-03-19T02:56:34.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.207+0000 7fbe66dc7740 10 client.0 _ll_register_callbacks cb 0x55bd096fea40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:56:34.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.207+0000 7fbe6169a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd097a3350 0x55bd097a9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.207+0000 7fbe61e9b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bd097a2a30 0x55bd097a8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.207+0000 7fbe6169a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd097a3350 0x55bd097a9390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48398/0 (socket says 172.21.15.33:48398) 2024-03-19T02:56:34.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.207+0000 7fbe6169a640 1 -- 172.21.15.33:0/4145760563 learned_addr learned my addr 172.21.15.33:0/4145760563 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:34.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.208+0000 7fbe6269c640 1 --2- 172.21.15.33:0/4145760563 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bd097abf30 0x55bd097a98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.208+0000 7fbe61e9b640 1 -- 172.21.15.33:0/4145760563 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bd097abf30 msgr2=0x55bd097a98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:34.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.208+0000 7fbe61e9b640 1 --2- 172.21.15.33:0/4145760563 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bd097abf30 0x55bd097a98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.208+0000 7fbe61e9b640 1 -- 172.21.15.33:0/4145760563 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd097a3350 msgr2=0x55bd097a9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:34.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.208+0000 7fbe61e9b640 1 --2- 172.21.15.33:0/4145760563 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd097a3350 0x55bd097a9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.208+0000 7fbe61e9b640 1 -- 172.21.15.33:0/4145760563 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd094e47d0 con 0x55bd097a2a30 2024-03-19T02:56:34.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.208+0000 7fbe6169a640 1 --2- 172.21.15.33:0/4145760563 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd097a3350 0x55bd097a9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:34.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.208+0000 7fbe61e9b640 1 --2- 172.21.15.33:0/4145760563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bd097a2a30 0x55bd097a8e50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbe58000c00 tx=0x7fbe580046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=923fccab6c2f4da2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:34.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.208+0000 7fbe457fa640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T02:56:34.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.208+0000 7fbe457fa640 1 -- 172.21.15.33:0/4145760563 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe58003a00 con 0x55bd097a2a30 2024-03-19T02:56:34.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.208+0000 7fbe457fa640 1 -- 172.21.15.33:0/4145760563 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fbe58003ba0 con 0x55bd097a2a30 2024-03-19T02:56:34.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.208+0000 7fbe66dc7740 1 -- 172.21.15.33:0/4145760563 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe580054d0 con 0x55bd097a2a30 2024-03-19T02:56:34.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.208+0000 7fbe457fa640 1 -- 172.21.15.33:0/4145760563 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fbe58033050 con 0x55bd097a2a30 2024-03-19T02:56:34.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.208+0000 7fbe66dc7740 10 client.26198 fetch_fsmap learned FSMap version 25 2024-03-19T02:56:34.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.208+0000 7fbe66dc7740 1 -- 172.21.15.33:0/4145760563 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55bd0941a3f0 con 0x55bd097a2a30 2024-03-19T02:56:34.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.208+0000 7fbe457fa640 1 -- 172.21.15.33:0/4145760563 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe580054d0 con 0x55bd097a2a30 2024-03-19T02:56:34.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.209+0000 7fbe457fa640 1 -- 172.21.15.33:0/4145760563 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fbe58023430 con 0x55bd097a2a30 2024-03-19T02:56:34.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.209+0000 7fbe66dc7740 10 client.26198 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:56:34.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.209+0000 7fbe66dc7740 10 client.26198 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:56:34.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.209+0000 7fbe66dc7740 20 client.26198 populate_metadata read hostname 'smithi033' 2024-03-19T02:56:34.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.209+0000 7fbe66dc7740 1 --2- 172.21.15.33:0/4145760563 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55bd097bc140 0x55bd097be530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.209+0000 7fbe66dc7740 4 client.26198 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:56:34.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.209+0000 7fbe66dc7740 1 -- 172.21.15.33:0/4145760563 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd09713a40 con 0x55bd097bc140 2024-03-19T02:56:34.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.209+0000 7fbe6169a640 1 --2- 172.21.15.33:0/4145760563 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55bd097bc140 0x55bd097be530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.209+0000 7fbe6169a640 1 --2- 172.21.15.33:0/4145760563 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55bd097bc140 0x55bd097be530 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=daed140699cad014 server_cookie=5a738dbac6ef0c6f in_seq=0 out_seq=0 2024-03-19T02:56:34.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.209+0000 7fbe457fa640 10 client.26198 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:56:34.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.210+0000 7fbe457fa640 1 -- 172.21.15.33:0/4145760563 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bd09713a40 con 0x55bd097bc140 2024-03-19T02:56:34.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.210+0000 7fbe457fa640 10 client.26198 handle_command_reply: tid=0 2024-03-19T02:56:34.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.210+0000 7fbe66dc7740 1 client.26198 shutdown 2024-03-19T02:56:34.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.210+0000 7fbe66dc7740 1 -- 172.21.15.33:0/4145760563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bd097a2a30 msgr2=0x55bd097a8e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:34.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.210+0000 7fbe66dc7740 1 --2- 172.21.15.33:0/4145760563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bd097a2a30 0x55bd097a8e50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbe58000c00 tx=0x7fbe580046c0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.210+0000 7fbe66dc7740 1 -- 172.21.15.33:0/4145760563 shutdown_connections 2024-03-19T02:56:34.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.210+0000 7fbe66dc7740 1 --2- 172.21.15.33:0/4145760563 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55bd097bc140 0x55bd097be530 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.210+0000 7fbe66dc7740 1 --2- 172.21.15.33:0/4145760563 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bd097abf30 0x55bd097a98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.210+0000 7fbe66dc7740 1 --2- 172.21.15.33:0/4145760563 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd097a3350 0x55bd097a9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.210+0000 7fbe457fa640 0 client.26198 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:56:34.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.210+0000 7fbe66dc7740 1 --2- 172.21.15.33:0/4145760563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bd097a2a30 0x55bd097a8e50 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.210+0000 7fbe66dc7740 1 -- 172.21.15.33:0/4145760563 >> 172.21.15.33:0/4145760563 conn(0x55bd0979d160 msgr2=0x55bd0979eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:34.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.210+0000 7fbe66dc7740 1 -- 172.21.15.33:0/4145760563 shutdown_connections 2024-03-19T02:56:34.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.210+0000 7fbe66dc7740 1 -- 172.21.15.33:0/4145760563 wait complete. 2024-03-19T02:56:34.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.210+0000 7fbe66dc7740 20 client.26198 trim_cache size 0 max 16384 2024-03-19T02:56:34.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.227+0000 7fbe66dc7740 1 Processor -- start 2024-03-19T02:56:34.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.227+0000 7fbe66dc7740 1 -- start start 2024-03-19T02:56:34.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.227+0000 7fbe66dc7740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bd097a2a30 0x55bd097ab580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.228+0000 7fbe66dc7740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bd097a3350 0x55bd097b7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.228+0000 7fbe66dc7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd097abf30 0x55bd097c3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.228+0000 7fbe66dc7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbe5c1b14f0 con 0x55bd097abf30 2024-03-19T02:56:34.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.228+0000 7fbe66dc7740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbe5c1aedc0 con 0x55bd097a2a30 2024-03-19T02:56:34.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.228+0000 7fbe66dc7740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbe5c1acbf0 con 0x55bd097a3350 2024-03-19T02:56:34.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.228+0000 7fbe6269c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd097abf30 0x55bd097c3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.228+0000 7fbe6269c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd097abf30 0x55bd097c3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48410/0 (socket says 172.21.15.33:48410) 2024-03-19T02:56:34.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.228+0000 7fbe6269c640 1 -- 172.21.15.33:0/1368846467 learned_addr learned my addr 172.21.15.33:0/1368846467 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:34.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.228+0000 7fbe61e9b640 1 --2- 172.21.15.33:0/1368846467 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bd097a2a30 0x55bd097ab580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.228+0000 7fbe6169a640 1 --2- 172.21.15.33:0/1368846467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bd097a3350 0x55bd097b7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.228+0000 7fbe6269c640 1 -- 172.21.15.33:0/1368846467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bd097a3350 msgr2=0x55bd097b7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:34.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.228+0000 7fbe6269c640 1 --2- 172.21.15.33:0/1368846467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bd097a3350 0x55bd097b7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.228+0000 7fbe6269c640 1 -- 172.21.15.33:0/1368846467 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bd097a2a30 msgr2=0x55bd097ab580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:34.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.228+0000 7fbe6269c640 1 --2- 172.21.15.33:0/1368846467 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bd097a2a30 0x55bd097ab580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.228+0000 7fbe6269c640 1 -- 172.21.15.33:0/1368846467 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd093da0f0 con 0x55bd097abf30 2024-03-19T02:56:34.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.228+0000 7fbe61e9b640 1 --2- 172.21.15.33:0/1368846467 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bd097a2a30 0x55bd097ab580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:56:34.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.228+0000 7fbe6269c640 1 --2- 172.21.15.33:0/1368846467 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd097abf30 0x55bd097c3c70 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fbe54009cd0 tx=0x7fbe5402e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cdde3140357911a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:34.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.228+0000 7fbe477fe640 1 -- 172.21.15.33:0/1368846467 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe54039040 con 0x55bd097abf30 2024-03-19T02:56:34.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.228+0000 7fbe477fe640 1 -- 172.21.15.33:0/1368846467 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe54054070 con 0x55bd097abf30 2024-03-19T02:56:34.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.228+0000 7fbe477fe640 1 -- 172.21.15.33:0/1368846467 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe540346a0 con 0x55bd097abf30 2024-03-19T02:56:34.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.229+0000 7fbe66dc7740 1 -- 172.21.15.33:0/1368846467 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd097abf30 msgr2=0x55bd097c3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:34.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.229+0000 7fbe66dc7740 1 --2- 172.21.15.33:0/1368846467 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd097abf30 0x55bd097c3c70 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fbe54009cd0 tx=0x7fbe5402e040 comp rx=0 tx=0).stop 2024-03-19T02:56:34.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.229+0000 7fbe66dc7740 1 -- 172.21.15.33:0/1368846467 shutdown_connections 2024-03-19T02:56:34.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.229+0000 7fbe66dc7740 1 --2- 172.21.15.33:0/1368846467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bd097a3350 0x55bd097b7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.229+0000 7fbe66dc7740 1 --2- 172.21.15.33:0/1368846467 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd097abf30 0x55bd097c3c70 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.229+0000 7fbe66dc7740 1 --2- 172.21.15.33:0/1368846467 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bd097a2a30 0x55bd097ab580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.229+0000 7fbe66dc7740 1 -- 172.21.15.33:0/1368846467 >> 172.21.15.33:0/1368846467 conn(0x55bd0979d160 msgr2=0x55bd0979b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:34.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.229+0000 7fbe66dc7740 1 -- 172.21.15.33:0/1368846467 shutdown_connections 2024-03-19T02:56:34.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.229+0000 7fbe66dc7740 1 -- 172.21.15.33:0/1368846467 wait complete. 2024-03-19T02:56:34.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.229+0000 7fbe66dc7740 1 -- start start 2024-03-19T02:56:34.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.229+0000 7fbe66dc7740 1 Processor -- start 2024-03-19T02:56:34.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.230+0000 7fbe66dc7740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bd097a2a30 0x55bd097a5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.230+0000 7fbe66dc7740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd097a3350 0x55bd097a6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.230+0000 7fbe66dc7740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bd097abf30 0x55bd097a6690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.230+0000 7fbe66dc7740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbe5c1ad280 con 0x55bd097a3350 2024-03-19T02:56:34.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.230+0000 7fbe66dc7740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbe5c1ae6f0 con 0x55bd097a2a30 2024-03-19T02:56:34.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.230+0000 7fbe66dc7740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbe5c1ad950 con 0x55bd097abf30 2024-03-19T02:56:34.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.230+0000 7fbe66dc7740 10 client.0 _ll_register_callbacks cb 0x7fbe4c003d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:56:34.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.230+0000 7fbe6169a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd097a3350 0x55bd097a6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.230+0000 7fbe6169a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd097a3350 0x55bd097a6150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:48416/0 (socket says 172.21.15.33:48416) 2024-03-19T02:56:34.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.230+0000 7fbe6169a640 1 -- 172.21.15.33:0/380136174 learned_addr learned my addr 172.21.15.33:0/380136174 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:56:34.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.230+0000 7fbe61e9b640 1 --2- 172.21.15.33:0/380136174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bd097a2a30 0x55bd097a5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.230+0000 7fbe6269c640 1 --2- 172.21.15.33:0/380136174 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bd097abf30 0x55bd097a6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.230+0000 7fbe6169a640 1 -- 172.21.15.33:0/380136174 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bd097abf30 msgr2=0x55bd097a6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:34.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.230+0000 7fbe6169a640 1 --2- 172.21.15.33:0/380136174 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bd097abf30 0x55bd097a6690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.230+0000 7fbe6169a640 1 -- 172.21.15.33:0/380136174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bd097a2a30 msgr2=0x55bd097a5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:56:34.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.230+0000 7fbe6169a640 1 --2- 172.21.15.33:0/380136174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bd097a2a30 0x55bd097a5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.230+0000 7fbe6169a640 1 -- 172.21.15.33:0/380136174 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd098214f0 con 0x55bd097a3350 2024-03-19T02:56:34.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.230+0000 7fbe6169a640 1 --2- 172.21.15.33:0/380136174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd097a3350 0x55bd097a6150 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c00c760 tx=0x7fbe4c06b170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76d5ab833089a0a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:56:34.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.230+0000 7fbe457fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:56:34.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.230+0000 7fbe457fa640 1 -- 172.21.15.33:0/380136174 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c09b000 con 0x55bd097a3350 2024-03-19T02:56:34.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.230+0000 7fbe457fa640 1 -- 172.21.15.33:0/380136174 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fbe4c06a220 con 0x55bd097a3350 2024-03-19T02:56:34.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.230+0000 7fbe66dc7740 1 -- 172.21.15.33:0/380136174 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe54054070 con 0x55bd097a3350 2024-03-19T02:56:34.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.231+0000 7fbe457fa640 1 -- 172.21.15.33:0/380136174 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fbe4c003b30 con 0x55bd097a3350 2024-03-19T02:56:34.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.231+0000 7fbe66dc7740 10 client.16380 fetch_fsmap learned FSMap version 25 2024-03-19T02:56:34.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.231+0000 7fbe66dc7740 1 -- 172.21.15.33:0/380136174 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55bd098227e0 con 0x55bd097a3350 2024-03-19T02:56:34.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.231+0000 7fbe457fa640 1 -- 172.21.15.33:0/380136174 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe54054070 con 0x55bd097a3350 2024-03-19T02:56:34.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.231+0000 7fbe457fa640 1 -- 172.21.15.33:0/380136174 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fbe4c004000 con 0x55bd097a3350 2024-03-19T02:56:34.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.231+0000 7fbe66dc7740 10 client.16380 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:56:34.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.231+0000 7fbe66dc7740 10 client.16380 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:56:34.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.231+0000 7fbe66dc7740 20 client.16380 populate_metadata read hostname 'smithi033' 2024-03-19T02:56:34.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.231+0000 7fbe66dc7740 1 --2- 172.21.15.33:0/380136174 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55bd09822f30 0x55bd09825350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:56:34.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.231+0000 7fbe66dc7740 4 client.16380 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:56:34.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.231+0000 7fbe66dc7740 1 -- 172.21.15.33:0/380136174 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bd0970dd90 con 0x55bd09822f30 2024-03-19T02:56:34.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.231+0000 7fbe61e9b640 1 --2- 172.21.15.33:0/380136174 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55bd09822f30 0x55bd09825350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:56:34.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.231+0000 7fbe457fa640 10 client.16380 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:56:34.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.231+0000 7fbe61e9b640 1 --2- 172.21.15.33:0/380136174 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55bd09822f30 0x55bd09825350 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42626abf0d13e2e7 server_cookie=c5a196d650e87711 in_seq=0 out_seq=0 2024-03-19T02:56:34.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.232+0000 7fbe457fa640 1 -- 172.21.15.33:0/380136174 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55bd0970dd90 con 0x55bd09822f30 2024-03-19T02:56:34.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.232+0000 7fbe457fa640 10 client.16380 handle_command_reply: tid=0 2024-03-19T02:56:34.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.232+0000 7fbe66dc7740 1 client.16380 shutdown 2024-03-19T02:56:34.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.232+0000 7fbe66dc7740 1 -- 172.21.15.33:0/380136174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd097a3350 msgr2=0x55bd097a6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:34.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.232+0000 7fbe66dc7740 1 --2- 172.21.15.33:0/380136174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd097a3350 0x55bd097a6150 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c00c760 tx=0x7fbe4c06b170 comp rx=0 tx=0).stop 2024-03-19T02:56:34.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.232+0000 7fbe66dc7740 1 -- 172.21.15.33:0/380136174 shutdown_connections 2024-03-19T02:56:34.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.232+0000 7fbe66dc7740 1 --2- 172.21.15.33:0/380136174 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55bd09822f30 0x55bd09825350 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.232+0000 7fbe66dc7740 1 --2- 172.21.15.33:0/380136174 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bd097abf30 0x55bd097a6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.232+0000 7fbe66dc7740 1 --2- 172.21.15.33:0/380136174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bd097a3350 0x55bd097a6150 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.232+0000 7fbe66dc7740 1 --2- 172.21.15.33:0/380136174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bd097a2a30 0x55bd097a5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.232+0000 7fbe457fa640 0 client.16380 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:56:34.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.232+0000 7fbe66dc7740 1 -- 172.21.15.33:0/380136174 >> 172.21.15.33:0/380136174 conn(0x55bd0979d160 msgr2=0x55bd0979c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:34.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.232+0000 7fbe66dc7740 1 -- 172.21.15.33:0/380136174 shutdown_connections 2024-03-19T02:56:34.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.232+0000 7fbe66dc7740 1 -- 172.21.15.33:0/380136174 wait complete. 2024-03-19T02:56:34.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.232+0000 7fbe66dc7740 20 client.16380 trim_cache size 0 max 16384 2024-03-19T02:56:34.240 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:56:34.240 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:56:34.240 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-03-19T02:56:34.240 INFO:teuthology.orchestra.run.smithi033.stdout: "75433454-729c-4fb8-ae84-9b56d7f302cc": { 2024-03-19T02:56:34.240 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-03-19T02:56:34.240 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "75433454-729c-4fb8-ae84-9b56d7f302cc", 2024-03-19T02:56:34.241 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-03-19T02:56:34.241 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T02:56:34.241 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T02:56:34.241 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:56:34.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.232+0000 7fbe640ff640 1 -- 172.21.15.33:0/3994333313 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbe480a29f0 msgr2=0x7fbe480a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:34.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.232+0000 7fbe640ff640 1 --2- 172.21.15.33:0/3994333313 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbe480a29f0 0x7fbe480a4eb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbe54006010 tx=0x7fbe54005c50 comp rx=0 tx=0).stop 2024-03-19T02:56:34.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.232+0000 7fbe640ff640 1 -- 172.21.15.33:0/3994333313 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe5c101090 msgr2=0x7fbe5c199320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:56:34.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.232+0000 7fbe640ff640 1 --2- 172.21.15.33:0/3994333313 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe5c101090 0x7fbe5c199320 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c00ea60 tx=0x7fbe4c00ef20 comp rx=0 tx=0).stop 2024-03-19T02:56:34.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.232+0000 7fbe640ff640 1 -- 172.21.15.33:0/3994333313 shutdown_connections 2024-03-19T02:56:34.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.232+0000 7fbe640ff640 1 --2- 172.21.15.33:0/3994333313 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbe480a29f0 0x7fbe480a4eb0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.232+0000 7fbe640ff640 1 --2- 172.21.15.33:0/3994333313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe5c1066e0 0x7fbe5c1a3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.232+0000 7fbe640ff640 1 --2- 172.21.15.33:0/3994333313 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe5c101090 0x7fbe5c199320 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.232+0000 7fbe640ff640 1 --2- 172.21.15.33:0/3994333313 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe5c1006d0 0x7fbe5c198de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:56:34.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.233+0000 7fbe640ff640 1 -- 172.21.15.33:0/3994333313 >> 172.21.15.33:0/3994333313 conn(0x7fbe5c076aa0 msgr2=0x7fbe5c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:56:34.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.233+0000 7fbe640ff640 1 -- 172.21.15.33:0/3994333313 shutdown_connections 2024-03-19T02:56:34.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:56:34.233+0000 7fbe640ff640 1 -- 172.21.15.33:0/3994333313 wait complete. 2024-03-19T02:56:34.243 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:33 smithi033 ceph-mon[31973]: pgmap v96: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 7 op/s 2024-03-19T02:56:34.245 INFO:tasks.cephfs.filesystem:scrub status for tag:75433454-729c-4fb8-ae84-9b56d7f302cc - {'path': '/', 'tag': '75433454-729c-4fb8-ae84-9b56d7f302cc', 'options': 'recursive,force'} 2024-03-19T02:56:34.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:33 smithi155 ceph-mon[35804]: pgmap v96: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 7 op/s 2024-03-19T02:56:34.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:33 smithi096 ceph-mon[35946]: pgmap v96: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 7 op/s 2024-03-19T02:56:35.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:34 smithi155 ceph-mon[35804]: scrub queued for path: / 2024-03-19T02:56:35.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:34 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T02:56:35.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:34 smithi155 ceph-mon[35804]: scrub summary: active paths [/] 2024-03-19T02:56:35.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:34 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T02:56:35.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:34 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:56:35.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:34 smithi033 ceph-mon[31973]: scrub queued for path: / 2024-03-19T02:56:35.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:34 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T02:56:35.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:34 smithi033 ceph-mon[31973]: scrub summary: active paths [/] 2024-03-19T02:56:35.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:34 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T02:56:35.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:34 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:56:35.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:34 smithi096 ceph-mon[35946]: scrub queued for path: / 2024-03-19T02:56:35.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:34 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T02:56:35.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:34 smithi096 ceph-mon[35946]: scrub summary: active paths [/] 2024-03-19T02:56:35.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:34 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T02:56:35.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:34 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:56:36.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:35 smithi155 ceph-mon[35804]: pgmap v97: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:56:36.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:35 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:56:36.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:35 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:56:36.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:35 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:56:36.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:35 smithi033 ceph-mon[31973]: pgmap v97: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:56:36.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:35 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:56:36.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:35 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:56:36.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:35 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:56:36.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:35 smithi096 ceph-mon[35946]: pgmap v97: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:56:36.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:35 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:56:36.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:35 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:56:36.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:35 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:56:38.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:38 smithi096 ceph-mon[35946]: pgmap v98: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:56:38.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:38 smithi096 ceph-mon[35946]: scrub summary: idle 2024-03-19T02:56:38.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:38 smithi155 ceph-mon[35804]: pgmap v98: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:56:38.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:38 smithi155 ceph-mon[35804]: scrub summary: idle 2024-03-19T02:56:38.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:38 smithi033 ceph-mon[31973]: pgmap v98: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:56:38.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:38 smithi033 ceph-mon[31973]: scrub summary: idle 2024-03-19T02:56:40.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:40 smithi096 ceph-mon[35946]: pgmap v99: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:56:40.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:40 smithi155 ceph-mon[35804]: pgmap v99: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:56:40.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:40 smithi033 ceph-mon[31973]: pgmap v99: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:56:42.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:42 smithi096 ceph-mon[35946]: pgmap v100: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:56:42.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:42 smithi155 ceph-mon[35804]: pgmap v100: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:56:42.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:42 smithi033 ceph-mon[31973]: pgmap v100: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:56:44.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:44 smithi096 ceph-mon[35946]: pgmap v101: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:56:44.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:44 smithi155 ceph-mon[35804]: pgmap v101: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:56:44.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:44 smithi033 ceph-mon[31973]: pgmap v101: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:56:46.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:46 smithi096 ceph-mon[35946]: pgmap v102: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:56:46.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:46 smithi155 ceph-mon[35804]: pgmap v102: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:56:46.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:46 smithi033 ceph-mon[31973]: pgmap v102: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:56:48.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:48 smithi096 ceph-mon[35946]: pgmap v103: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:56:48.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:48 smithi155 ceph-mon[35804]: pgmap v103: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:56:48.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:48 smithi033 ceph-mon[31973]: pgmap v103: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:56:50.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:50 smithi096 ceph-mon[35946]: pgmap v104: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:56:50.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:50 smithi155 ceph-mon[35804]: pgmap v104: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:56:50.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:50 smithi033 ceph-mon[31973]: pgmap v104: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:56:52.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:52 smithi096 ceph-mon[35946]: pgmap v105: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:56:52.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:52 smithi155 ceph-mon[35804]: pgmap v105: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:56:52.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:52 smithi033 ceph-mon[31973]: pgmap v105: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:56:54.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:54 smithi096 ceph-mon[35946]: pgmap v106: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:56:54.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:54 smithi155 ceph-mon[35804]: pgmap v106: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:56:54.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:54 smithi033 ceph-mon[31973]: pgmap v106: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:56:56.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:56 smithi096 ceph-mon[35946]: pgmap v107: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:56:56.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:56 smithi155 ceph-mon[35804]: pgmap v107: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:56:56.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:56 smithi033 ceph-mon[31973]: pgmap v107: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:56:58.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:56:58 smithi096 ceph-mon[35946]: pgmap v108: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:56:58.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:56:58 smithi155 ceph-mon[35804]: pgmap v108: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:56:58.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:56:58 smithi033 ceph-mon[31973]: pgmap v108: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:57:00.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:00 smithi096 ceph-mon[35946]: pgmap v109: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:57:00.518 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:00 smithi155 ceph-mon[35804]: pgmap v109: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:57:00.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:00 smithi033 ceph-mon[31973]: pgmap v109: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:57:02.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:02 smithi096 ceph-mon[35946]: pgmap v110: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:57:02.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:02 smithi155 ceph-mon[35804]: pgmap v110: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:57:02.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:02 smithi033 ceph-mon[31973]: pgmap v110: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:57:04.245 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:57:04.246 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:57:04.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.362+0000 7f84f49fe640 1 -- 172.21.15.33:0/48088972 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84f0100f60 msgr2=0x7f84f010d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:04.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.362+0000 7f84f49fe640 1 --2- 172.21.15.33:0/48088972 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84f0100f60 0x7f84f010d420 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f84e8009ca0 tx=0x7f84e802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.362+0000 7f84f49fe640 1 -- 172.21.15.33:0/48088972 shutdown_connections 2024-03-19T02:57:04.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.362+0000 7f84f49fe640 1 --2- 172.21.15.33:0/48088972 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84f0100f60 0x7f84f010d420 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.362+0000 7f84f49fe640 1 --2- 172.21.15.33:0/48088972 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84f01005a0 0x7f84f0100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.364 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.362+0000 7f84f49fe640 1 --2- 172.21.15.33:0/48088972 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84f0106660 0x7f84f0106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.362+0000 7f84f49fe640 1 -- 172.21.15.33:0/48088972 >> 172.21.15.33:0/48088972 conn(0x7f84f00fc070 msgr2=0x7f84f00fe490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:04.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.362+0000 7f84f49fe640 1 -- 172.21.15.33:0/48088972 shutdown_connections 2024-03-19T02:57:04.365 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.362+0000 7f84f49fe640 1 -- 172.21.15.33:0/48088972 wait complete. 2024-03-19T02:57:04.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.363+0000 7f84f49fe640 1 Processor -- start 2024-03-19T02:57:04.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.363+0000 7f84f49fe640 1 -- start start 2024-03-19T02:57:04.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.363+0000 7f84f49fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84f01005a0 0x7f84f0198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.364+0000 7f84f49fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84f0100f60 0x7f84f0199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.364+0000 7f84f49fe640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84f0106660 0x7f84f01a3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.364+0000 7f84f49fe640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f84f0110810 con 0x7f84f0100f60 2024-03-19T02:57:04.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.364+0000 7f84f49fe640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f84f0110690 con 0x7f84f01005a0 2024-03-19T02:57:04.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.364+0000 7f84f49fe640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f84f0110990 con 0x7f84f0106660 2024-03-19T02:57:04.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.364+0000 7f84e7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84f0100f60 0x7f84f0199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.364+0000 7f84eed9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84f01005a0 0x7f84f0198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.364+0000 7f84e7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84f0100f60 0x7f84f0199110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59392/0 (socket says 172.21.15.33:59392) 2024-03-19T02:57:04.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.364+0000 7f84e7fff640 1 -- 172.21.15.33:0/915864435 learned_addr learned my addr 172.21.15.33:0/915864435 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:04.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.364+0000 7f84ef59e640 1 --2- 172.21.15.33:0/915864435 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84f0106660 0x7f84f01a3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.364+0000 7f84e7fff640 1 -- 172.21.15.33:0/915864435 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84f0106660 msgr2=0x7f84f01a3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:04.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.364+0000 7f84e7fff640 1 --2- 172.21.15.33:0/915864435 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84f0106660 0x7f84f01a3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.364+0000 7f84e7fff640 1 -- 172.21.15.33:0/915864435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84f01005a0 msgr2=0x7f84f0198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:04.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.364+0000 7f84e7fff640 1 --2- 172.21.15.33:0/915864435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84f01005a0 0x7f84f0198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.364+0000 7f84e7fff640 1 -- 172.21.15.33:0/915864435 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84f01a4190 con 0x7f84f0100f60 2024-03-19T02:57:04.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.364+0000 7f84eed9d640 1 --2- 172.21.15.33:0/915864435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84f01005a0 0x7f84f0198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:04.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.364+0000 7f84e7fff640 1 --2- 172.21.15.33:0/915864435 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84f0100f60 0x7f84f0199110 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f84d800e9f0 tx=0x7f84d800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:04.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.364+0000 7f84ecd99640 1 -- 172.21.15.33:0/915864435 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d800cd80 con 0x7f84f0100f60 2024-03-19T02:57:04.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.364+0000 7f84ecd99640 1 -- 172.21.15.33:0/915864435 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f84d80108e0 con 0x7f84f0100f60 2024-03-19T02:57:04.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.364+0000 7f84f49fe640 1 -- 172.21.15.33:0/915864435 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84f01a4360 con 0x7f84f0100f60 2024-03-19T02:57:04.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.364+0000 7f84ecd99640 1 -- 172.21.15.33:0/915864435 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d8018920 con 0x7f84f0100f60 2024-03-19T02:57:04.369 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.364+0000 7f84f49fe640 1 -- 172.21.15.33:0/915864435 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84f01a48d0 con 0x7f84f0100f60 2024-03-19T02:57:04.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.366+0000 7f84ecd99640 1 -- 172.21.15.33:0/915864435 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f84d8010a90 con 0x7f84f0100f60 2024-03-19T02:57:04.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.367+0000 7f84ecd99640 1 --2- 172.21.15.33:0/915864435 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f84dc0a29f0 0x7f84dc0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.367+0000 7f84f7def740 1 Processor -- start 2024-03-19T02:57:04.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.367+0000 7f84f7def740 1 -- start start 2024-03-19T02:57:04.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.367+0000 7f84eed9d640 1 --2- 172.21.15.33:0/915864435 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f84dc0a29f0 0x7f84dc0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.367+0000 7f84ecd99640 1 -- 172.21.15.33:0/915864435 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f84d80cf470 con 0x7f84f0100f60 2024-03-19T02:57:04.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.368+0000 7f84f7def740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f478617bf0 0x55f478617fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.368+0000 7f84f7def740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f478618510 0x55f478620bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.368+0000 7f84f7def740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f4786210f0 0x55f4786234e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.368+0000 7f84f7def740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f47857b500 con 0x55f4786210f0 2024-03-19T02:57:04.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.368+0000 7f84f7def740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55f478523be0 con 0x55f478617bf0 2024-03-19T02:57:04.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.368+0000 7f84f7def740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f4785775a0 con 0x55f478618510 2024-03-19T02:57:04.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.368+0000 7f84eed9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f478617bf0 0x55f478617fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.368+0000 7f84ef59e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f4786210f0 0x55f4786234e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.368+0000 7f84e7fff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f478618510 0x55f478620bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.368+0000 7f84ef59e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f4786210f0 0x55f4786234e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59396/0 (socket says 172.21.15.33:59396) 2024-03-19T02:57:04.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.368+0000 7f84ef59e640 1 -- 172.21.15.33:0/597823349 learned_addr learned my addr 172.21.15.33:0/597823349 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:04.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.368+0000 7f84eed9d640 1 --2- 172.21.15.33:0/915864435 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f84dc0a29f0 0x7f84dc0a4eb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f84e0009f90 tx=0x7f84e0005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:04.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.368+0000 7f84e7fff640 1 -- 172.21.15.33:0/597823349 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f478617bf0 msgr2=0x55f478617fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:04.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.368+0000 7f84e7fff640 1 --2- 172.21.15.33:0/597823349 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f478617bf0 0x55f478617fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.368+0000 7f84e7fff640 1 -- 172.21.15.33:0/597823349 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f4786210f0 msgr2=0x55f4786234e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:04.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.368+0000 7f84e7fff640 1 --2- 172.21.15.33:0/597823349 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f4786210f0 0x55f4786234e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.368+0000 7f84e7fff640 1 -- 172.21.15.33:0/597823349 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f478545000 con 0x55f478618510 2024-03-19T02:57:04.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.368+0000 7f84ef59e640 1 --2- 172.21.15.33:0/597823349 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f4786210f0 0x55f4786234e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:04.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.368+0000 7f84e7fff640 1 --2- 172.21.15.33:0/597823349 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f478618510 0x55f478620bb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f84d80cc850 tx=0x7f84d809b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7875bf2412da7036 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:04.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.368+0000 7f84e57fa640 1 -- 172.21.15.33:0/597823349 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d80a3f00 con 0x55f478618510 2024-03-19T02:57:04.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.368+0000 7f84e57fa640 1 -- 172.21.15.33:0/597823349 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f84d80cf150 con 0x55f478618510 2024-03-19T02:57:04.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.368+0000 7f84e57fa640 1 -- 172.21.15.33:0/597823349 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d809ac60 con 0x55f478618510 2024-03-19T02:57:04.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.369+0000 7f84f7def740 1 -- 172.21.15.33:0/597823349 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f478618510 msgr2=0x55f478620bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:04.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.369+0000 7f84f7def740 1 --2- 172.21.15.33:0/597823349 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f478618510 0x55f478620bb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f84d80cc850 tx=0x7f84d809b710 comp rx=0 tx=0).stop 2024-03-19T02:57:04.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.369+0000 7f84f7def740 1 -- 172.21.15.33:0/597823349 shutdown_connections 2024-03-19T02:57:04.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.369+0000 7f84f7def740 1 --2- 172.21.15.33:0/597823349 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f478618510 0x55f478620bb0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.369+0000 7f84f7def740 1 --2- 172.21.15.33:0/597823349 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f4786210f0 0x55f4786234e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.369+0000 7f84f7def740 1 --2- 172.21.15.33:0/597823349 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f478617bf0 0x55f478617fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.369+0000 7f84f7def740 1 -- 172.21.15.33:0/597823349 >> 172.21.15.33:0/597823349 conn(0x55f478612320 msgr2=0x55f478612700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:04.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.369+0000 7f84f7def740 1 -- 172.21.15.33:0/597823349 shutdown_connections 2024-03-19T02:57:04.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.369+0000 7f84f7def740 1 -- 172.21.15.33:0/597823349 wait complete. 2024-03-19T02:57:04.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.370+0000 7f84f7def740 1 -- start start 2024-03-19T02:57:04.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.370+0000 7f84f7def740 1 Processor -- start 2024-03-19T02:57:04.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.370+0000 7f84f7def740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f478617bf0 0x55f47861e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.370+0000 7f84f7def740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f478618510 0x55f47861e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.370+0000 7f84f7def740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f4786210f0 0x55f47861ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.370+0000 7f84f7def740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f478623e80 con 0x55f478618510 2024-03-19T02:57:04.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.370+0000 7f84f7def740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55f478537160 con 0x55f478617bf0 2024-03-19T02:57:04.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.370+0000 7f84f7def740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f478624000 con 0x55f4786210f0 2024-03-19T02:57:04.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.370+0000 7f84f7def740 10 client.0 _ll_register_callbacks cb 0x55f478573a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:57:04.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.370+0000 7f84e7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f478618510 0x55f47861e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.370+0000 7f84e7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f478618510 0x55f47861e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59410/0 (socket says 172.21.15.33:59410) 2024-03-19T02:57:04.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.370+0000 7f84e7fff640 1 -- 172.21.15.33:0/3609502496 learned_addr learned my addr 172.21.15.33:0/3609502496 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:04.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.370+0000 7f84ef59e640 1 --2- 172.21.15.33:0/3609502496 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f4786210f0 0x55f47861ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.370+0000 7f84eed9d640 1 --2- 172.21.15.33:0/3609502496 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f478617bf0 0x55f47861e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.370+0000 7f84e7fff640 1 -- 172.21.15.33:0/3609502496 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f4786210f0 msgr2=0x55f47861ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:04.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.370+0000 7f84e7fff640 1 --2- 172.21.15.33:0/3609502496 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f4786210f0 0x55f47861ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.370+0000 7f84e7fff640 1 -- 172.21.15.33:0/3609502496 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f478617bf0 msgr2=0x55f47861e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:04.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.370+0000 7f84e7fff640 1 --2- 172.21.15.33:0/3609502496 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f478617bf0 0x55f47861e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.370+0000 7f84e7fff640 1 -- 172.21.15.33:0/3609502496 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4783597d0 con 0x55f478618510 2024-03-19T02:57:04.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.371+0000 7f84e7fff640 1 --2- 172.21.15.33:0/3609502496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f478618510 0x55f47861e680 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f84d8097c70 tx=0x7f84d809b5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec0b5447076a75c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:04.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.371+0000 7f84d67fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:57:04.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.371+0000 7f84d67fc640 1 -- 172.21.15.33:0/3609502496 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d80b9060 con 0x55f478618510 2024-03-19T02:57:04.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.371+0000 7f84d67fc640 1 -- 172.21.15.33:0/3609502496 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f84d8099b10 con 0x55f478618510 2024-03-19T02:57:04.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.371+0000 7f84f7def740 1 -- 172.21.15.33:0/3609502496 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84d80cf150 con 0x55f478618510 2024-03-19T02:57:04.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.371+0000 7f84d67fc640 1 -- 172.21.15.33:0/3609502496 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f84d80ab050 con 0x55f478618510 2024-03-19T02:57:04.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.371+0000 7f84f7def740 10 client.16392 fetch_fsmap learned FSMap version 25 2024-03-19T02:57:04.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.371+0000 7f84f7def740 1 -- 172.21.15.33:0/3609502496 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f47828f3f0 con 0x55f478618510 2024-03-19T02:57:04.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.371+0000 7f84d67fc640 1 -- 172.21.15.33:0/3609502496 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f84d80cf150 con 0x55f478618510 2024-03-19T02:57:04.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.371+0000 7f84d67fc640 1 -- 172.21.15.33:0/3609502496 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f84d8097560 con 0x55f478618510 2024-03-19T02:57:04.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.371+0000 7f84f7def740 10 client.16392 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:57:04.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.371+0000 7f84f7def740 10 client.16392 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:57:04.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.371+0000 7f84f7def740 20 client.16392 populate_metadata read hostname 'smithi033' 2024-03-19T02:57:04.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.371+0000 7f84f7def740 1 --2- 172.21.15.33:0/3609502496 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f4786312c0 0x55f4786336b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.371+0000 7f84f7def740 4 client.16392 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:57:04.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.371+0000 7f84f7def740 1 -- 172.21.15.33:0/3609502496 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f478588a40 con 0x55f4786312c0 2024-03-19T02:57:04.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.371+0000 7f84eed9d640 1 --2- 172.21.15.33:0/3609502496 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f4786312c0 0x55f4786336b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.372+0000 7f84d67fc640 10 client.16392 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:57:04.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.372+0000 7f84eed9d640 1 --2- 172.21.15.33:0/3609502496 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f4786312c0 0x55f4786336b0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35dc7b5e64ddbf5d server_cookie=6335ccab354a5870 in_seq=0 out_seq=0 2024-03-19T02:57:04.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.372+0000 7f84d67fc640 1 -- 172.21.15.33:0/3609502496 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f478588a40 con 0x55f4786312c0 2024-03-19T02:57:04.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.372+0000 7f84d67fc640 10 client.16392 handle_command_reply: tid=0 2024-03-19T02:57:04.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.372+0000 7f84f7def740 1 client.16392 shutdown 2024-03-19T02:57:04.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.373+0000 7f84f7def740 1 -- 172.21.15.33:0/3609502496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f478618510 msgr2=0x55f47861e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:04.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.373+0000 7f84f7def740 1 --2- 172.21.15.33:0/3609502496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f478618510 0x55f47861e680 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f84d8097c70 tx=0x7f84d809b5e0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.373+0000 7f84f7def740 1 -- 172.21.15.33:0/3609502496 shutdown_connections 2024-03-19T02:57:04.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.373+0000 7f84f7def740 1 --2- 172.21.15.33:0/3609502496 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f4786312c0 0x55f4786336b0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.373+0000 7f84f7def740 1 --2- 172.21.15.33:0/3609502496 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f4786210f0 0x55f47861ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.373+0000 7f84f7def740 1 --2- 172.21.15.33:0/3609502496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f478618510 0x55f47861e680 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.373+0000 7f84d67fc640 0 client.16392 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:57:04.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.373+0000 7f84f7def740 1 --2- 172.21.15.33:0/3609502496 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f478617bf0 0x55f47861e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.373+0000 7f84f7def740 1 -- 172.21.15.33:0/3609502496 >> 172.21.15.33:0/3609502496 conn(0x55f478612320 msgr2=0x55f478614070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:04.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.373+0000 7f84f7def740 1 -- 172.21.15.33:0/3609502496 shutdown_connections 2024-03-19T02:57:04.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.373+0000 7f84f7def740 1 -- 172.21.15.33:0/3609502496 wait complete. 2024-03-19T02:57:04.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.373+0000 7f84f7def740 20 client.16392 trim_cache size 0 max 16384 2024-03-19T02:57:04.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.390+0000 7f84f7def740 1 Processor -- start 2024-03-19T02:57:04.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.390+0000 7f84f7def740 1 -- start start 2024-03-19T02:57:04.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.390+0000 7f84f7def740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f478617bf0 0x55f478620870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.390+0000 7f84f7def740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f478618510 0x55f47861e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.391+0000 7f84f7def740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f4786210f0 0x55f4786366d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.391+0000 7f84f7def740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f84f01b12e0 con 0x55f478617bf0 2024-03-19T02:57:04.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.391+0000 7f84f7def740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f84f01aebb0 con 0x55f4786210f0 2024-03-19T02:57:04.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.391+0000 7f84f7def740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f84f01ac9e0 con 0x55f478618510 2024-03-19T02:57:04.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.391+0000 7f84eed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f478617bf0 0x55f478620870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.391+0000 7f84eed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f478617bf0 0x55f478620870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59426/0 (socket says 172.21.15.33:59426) 2024-03-19T02:57:04.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.391+0000 7f84eed9d640 1 -- 172.21.15.33:0/1879868805 learned_addr learned my addr 172.21.15.33:0/1879868805 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:04.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.391+0000 7f84e7fff640 1 --2- 172.21.15.33:0/1879868805 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f478618510 0x55f47861e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.391+0000 7f84ef59e640 1 --2- 172.21.15.33:0/1879868805 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f4786210f0 0x55f4786366d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.391+0000 7f84eed9d640 1 -- 172.21.15.33:0/1879868805 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f478618510 msgr2=0x55f47861e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:04.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.391+0000 7f84eed9d640 1 --2- 172.21.15.33:0/1879868805 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f478618510 0x55f47861e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.391+0000 7f84eed9d640 1 -- 172.21.15.33:0/1879868805 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f4786210f0 msgr2=0x55f4786366d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:04.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.391+0000 7f84eed9d640 1 --2- 172.21.15.33:0/1879868805 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f4786210f0 0x55f4786366d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.391+0000 7f84eed9d640 1 -- 172.21.15.33:0/1879868805 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f47824f0f0 con 0x55f478617bf0 2024-03-19T02:57:04.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.391+0000 7f84eed9d640 1 --2- 172.21.15.33:0/1879868805 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f478617bf0 0x55f478620870 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f84e0013010 tx=0x7f84e0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7757d2faf05b3037 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:04.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.391+0000 7f84e4ff9640 1 -- 172.21.15.33:0/1879868805 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e002f070 con 0x55f478617bf0 2024-03-19T02:57:04.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.391+0000 7f84e4ff9640 1 -- 172.21.15.33:0/1879868805 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f84e002a520 con 0x55f478617bf0 2024-03-19T02:57:04.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.391+0000 7f84e4ff9640 1 -- 172.21.15.33:0/1879868805 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e000fb30 con 0x55f478617bf0 2024-03-19T02:57:04.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.392+0000 7f84f7def740 1 -- 172.21.15.33:0/1879868805 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f478617bf0 msgr2=0x55f478620870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:04.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.392+0000 7f84f7def740 1 --2- 172.21.15.33:0/1879868805 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f478617bf0 0x55f478620870 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f84e0013010 tx=0x7f84e0019000 comp rx=0 tx=0).stop 2024-03-19T02:57:04.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.392+0000 7f84f7def740 1 -- 172.21.15.33:0/1879868805 shutdown_connections 2024-03-19T02:57:04.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.392+0000 7f84f7def740 1 --2- 172.21.15.33:0/1879868805 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f478618510 0x55f47861e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.392+0000 7f84f7def740 1 --2- 172.21.15.33:0/1879868805 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f4786210f0 0x55f4786366d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.392+0000 7f84f7def740 1 --2- 172.21.15.33:0/1879868805 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f478617bf0 0x55f478620870 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.392+0000 7f84f7def740 1 -- 172.21.15.33:0/1879868805 >> 172.21.15.33:0/1879868805 conn(0x55f478612320 msgr2=0x55f4786114a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:04.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.392+0000 7f84f7def740 1 -- 172.21.15.33:0/1879868805 shutdown_connections 2024-03-19T02:57:04.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.392+0000 7f84f7def740 1 -- 172.21.15.33:0/1879868805 wait complete. 2024-03-19T02:57:04.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.392+0000 7f84f7def740 1 -- start start 2024-03-19T02:57:04.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.392+0000 7f84f7def740 1 Processor -- start 2024-03-19T02:57:04.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.393+0000 7f84f7def740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f478617bf0 0x55f478633fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.393+0000 7f84f7def740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f478618510 0x55f4786344f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.393+0000 7f84f7def740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f4786210f0 0x55f478634a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.393+0000 7f84f7def740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f84f01ad070 con 0x55f478617bf0 2024-03-19T02:57:04.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.393+0000 7f84f7def740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f84f01ae4e0 con 0x55f4786210f0 2024-03-19T02:57:04.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.393+0000 7f84f7def740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f84f01ad740 con 0x55f478618510 2024-03-19T02:57:04.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.393+0000 7f84f7def740 10 client.0 _ll_register_callbacks cb 0x7f84d8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:57:04.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.393+0000 7f84eed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f478617bf0 0x55f478633fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.393+0000 7f84eed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f478617bf0 0x55f478633fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59440/0 (socket says 172.21.15.33:59440) 2024-03-19T02:57:04.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.393+0000 7f84eed9d640 1 -- 172.21.15.33:0/1090798388 learned_addr learned my addr 172.21.15.33:0/1090798388 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:04.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.393+0000 7f84e7fff640 1 --2- 172.21.15.33:0/1090798388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f478618510 0x55f4786344f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.393+0000 7f84ef59e640 1 --2- 172.21.15.33:0/1090798388 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f4786210f0 0x55f478634a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.393+0000 7f84eed9d640 1 -- 172.21.15.33:0/1090798388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f478618510 msgr2=0x55f4786344f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:04.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.393+0000 7f84eed9d640 1 --2- 172.21.15.33:0/1090798388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f478618510 0x55f4786344f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.393+0000 7f84eed9d640 1 -- 172.21.15.33:0/1090798388 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f4786210f0 msgr2=0x55f478634a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:04.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.393+0000 7f84eed9d640 1 --2- 172.21.15.33:0/1090798388 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f4786210f0 0x55f478634a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.393+0000 7f84eed9d640 1 -- 172.21.15.33:0/1090798388 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f478696260 con 0x55f478617bf0 2024-03-19T02:57:04.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.393+0000 7f84eed9d640 1 --2- 172.21.15.33:0/1090798388 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f478617bf0 0x55f478633fb0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f84e0004260 tx=0x7f84e00056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c5b4f494bf9a19d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:04.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.393+0000 7f84d67fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:57:04.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.393+0000 7f84d67fc640 1 -- 172.21.15.33:0/1090798388 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e002f070 con 0x55f478617bf0 2024-03-19T02:57:04.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.393+0000 7f84d67fc640 1 -- 172.21.15.33:0/1090798388 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f84e002a6d0 con 0x55f478617bf0 2024-03-19T02:57:04.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.393+0000 7f84f7def740 1 -- 172.21.15.33:0/1090798388 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84e002a520 con 0x55f478617bf0 2024-03-19T02:57:04.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.393+0000 7f84d67fc640 1 -- 172.21.15.33:0/1090798388 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f84e003d050 con 0x55f478617bf0 2024-03-19T02:57:04.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.393+0000 7f84f7def740 10 client.16398 fetch_fsmap learned FSMap version 25 2024-03-19T02:57:04.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.393+0000 7f84f7def740 1 -- 172.21.15.33:0/1090798388 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f478697510 con 0x55f478617bf0 2024-03-19T02:57:04.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.394+0000 7f84d67fc640 1 -- 172.21.15.33:0/1090798388 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f84e002a520 con 0x55f478617bf0 2024-03-19T02:57:04.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.394+0000 7f84d67fc640 1 -- 172.21.15.33:0/1090798388 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f84e0034460 con 0x55f478617bf0 2024-03-19T02:57:04.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.394+0000 7f84f7def740 10 client.16398 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:57:04.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.394+0000 7f84f7def740 10 client.16398 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:57:04.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.394+0000 7f84f7def740 20 client.16398 populate_metadata read hostname 'smithi033' 2024-03-19T02:57:04.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.394+0000 7f84f7def740 1 --2- 172.21.15.33:0/1090798388 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f478697ba0 0x55f478699fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.394+0000 7f84f7def740 4 client.16398 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:57:04.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.394+0000 7f84f7def740 1 -- 172.21.15.33:0/1090798388 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f478582d90 con 0x55f478697ba0 2024-03-19T02:57:04.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.394+0000 7f84e7fff640 1 --2- 172.21.15.33:0/1090798388 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f478697ba0 0x55f478699fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.394+0000 7f84d67fc640 10 client.16398 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:57:04.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.394+0000 7f84e7fff640 1 --2- 172.21.15.33:0/1090798388 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f478697ba0 0x55f478699fc0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7366acd0661e6463 server_cookie=691c47745abeb797 in_seq=0 out_seq=0 2024-03-19T02:57:04.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.395+0000 7f84d67fc640 1 -- 172.21.15.33:0/1090798388 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f478582d90 con 0x55f478697ba0 2024-03-19T02:57:04.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.395+0000 7f84d67fc640 10 client.16398 handle_command_reply: tid=0 2024-03-19T02:57:04.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.395+0000 7f84f7def740 1 client.16398 shutdown 2024-03-19T02:57:04.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.395+0000 7f84f7def740 1 -- 172.21.15.33:0/1090798388 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f478617bf0 msgr2=0x55f478633fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:04.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.395+0000 7f84f7def740 1 --2- 172.21.15.33:0/1090798388 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f478617bf0 0x55f478633fb0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f84e0004260 tx=0x7f84e00056d0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.395+0000 7f84f7def740 1 -- 172.21.15.33:0/1090798388 shutdown_connections 2024-03-19T02:57:04.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.395+0000 7f84f7def740 1 --2- 172.21.15.33:0/1090798388 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f478697ba0 0x55f478699fc0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.395+0000 7f84f7def740 1 --2- 172.21.15.33:0/1090798388 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f478618510 0x55f4786344f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.395+0000 7f84d67fc640 0 client.16398 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:57:04.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.395+0000 7f84f7def740 1 --2- 172.21.15.33:0/1090798388 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f4786210f0 0x55f478634a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.395+0000 7f84f7def740 1 --2- 172.21.15.33:0/1090798388 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f478617bf0 0x55f478633fb0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.395+0000 7f84f7def740 1 -- 172.21.15.33:0/1090798388 >> 172.21.15.33:0/1090798388 conn(0x55f478612320 msgr2=0x55f47862c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:04.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.395+0000 7f84f7def740 1 -- 172.21.15.33:0/1090798388 shutdown_connections 2024-03-19T02:57:04.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.395+0000 7f84f7def740 1 -- 172.21.15.33:0/1090798388 wait complete. 2024-03-19T02:57:04.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.395+0000 7f84f7def740 20 client.16398 trim_cache size 0 max 16384 2024-03-19T02:57:04.403 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:57:04.403 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-03-19T02:57:04.404 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-03-19T02:57:04.404 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:57:04.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.395+0000 7f84f49fe640 1 -- 172.21.15.33:0/915864435 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f84dc0a29f0 msgr2=0x7f84dc0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:04.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.395+0000 7f84f49fe640 1 --2- 172.21.15.33:0/915864435 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f84dc0a29f0 0x7f84dc0a4eb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f84e0009f90 tx=0x7f84e0005f90 comp rx=0 tx=0).stop 2024-03-19T02:57:04.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.395+0000 7f84f49fe640 1 -- 172.21.15.33:0/915864435 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84f0100f60 msgr2=0x7f84f0199110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:04.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.395+0000 7f84f49fe640 1 --2- 172.21.15.33:0/915864435 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84f0100f60 0x7f84f0199110 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f84d800e9f0 tx=0x7f84d800eeb0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.395+0000 7f84f49fe640 1 -- 172.21.15.33:0/915864435 shutdown_connections 2024-03-19T02:57:04.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.395+0000 7f84f49fe640 1 --2- 172.21.15.33:0/915864435 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f84dc0a29f0 0x7f84dc0a4eb0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.395+0000 7f84f49fe640 1 --2- 172.21.15.33:0/915864435 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84f0106660 0x7f84f01a3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.395+0000 7f84f49fe640 1 --2- 172.21.15.33:0/915864435 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84f0100f60 0x7f84f0199110 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.395+0000 7f84f49fe640 1 --2- 172.21.15.33:0/915864435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f84f01005a0 0x7f84f0198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.395+0000 7f84f49fe640 1 -- 172.21.15.33:0/915864435 >> 172.21.15.33:0/915864435 conn(0x7f84f00fc070 msgr2=0x7f84f00fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:04.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.396+0000 7f84f49fe640 1 -- 172.21.15.33:0/915864435 shutdown_connections 2024-03-19T02:57:04.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.396+0000 7f84f49fe640 1 -- 172.21.15.33:0/915864435 wait complete. 2024-03-19T02:57:04.406 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:04 smithi033 ceph-mon[31973]: pgmap v111: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-19T02:57:04.408 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:57:04.408 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:57:04.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:04 smithi096 ceph-mon[35946]: pgmap v111: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-19T02:57:04.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.510+0000 7f0a1d463640 1 -- 172.21.15.33:0/2843921879 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a181006c0 msgr2=0x7f0a18100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:04.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.510+0000 7f0a1d463640 1 --2- 172.21.15.33:0/2843921879 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a181006c0 0x7f0a18100b40 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f0a00009930 tx=0x7f0a0002bc30 comp rx=0 tx=0).stop 2024-03-19T02:57:04.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.510+0000 7f0a1d463640 1 -- 172.21.15.33:0/2843921879 shutdown_connections 2024-03-19T02:57:04.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.510+0000 7f0a1d463640 1 --2- 172.21.15.33:0/2843921879 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a18101080 0x7f0a1810d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.513 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.510+0000 7f0a1d463640 1 --2- 172.21.15.33:0/2843921879 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a181006c0 0x7f0a18100b40 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.510+0000 7f0a1d463640 1 --2- 172.21.15.33:0/2843921879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a181066d0 0x7f0a18106ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.510+0000 7f0a1d463640 1 -- 172.21.15.33:0/2843921879 >> 172.21.15.33:0/2843921879 conn(0x7f0a18076aa0 msgr2=0x7f0a18076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:04.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.510+0000 7f0a1d463640 1 -- 172.21.15.33:0/2843921879 shutdown_connections 2024-03-19T02:57:04.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.510+0000 7f0a1d463640 1 -- 172.21.15.33:0/2843921879 wait complete. 2024-03-19T02:57:04.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.510+0000 7f0a1d463640 1 Processor -- start 2024-03-19T02:57:04.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.511+0000 7f0a1d463640 1 -- start start 2024-03-19T02:57:04.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.511+0000 7f0a1d463640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a181006c0 0x7f0a18196cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.511+0000 7f0a1d463640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a18101080 0x7f0a181971f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.511+0000 7f0a1d463640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a181066d0 0x7f0a18197730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.511+0000 7f0a1d463640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0a18110990 con 0x7f0a181006c0 2024-03-19T02:57:04.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.511+0000 7f0a1d463640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0a18110810 con 0x7f0a181066d0 2024-03-19T02:57:04.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.511+0000 7f0a1d463640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f0a18110b10 con 0x7f0a18101080 2024-03-19T02:57:04.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.511+0000 7f0a16ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a181006c0 0x7f0a18196cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.511+0000 7f0a16ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a181006c0 0x7f0a18196cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59472/0 (socket says 172.21.15.33:59472) 2024-03-19T02:57:04.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.511+0000 7f0a16ffd640 1 -- 172.21.15.33:0/3638378399 learned_addr learned my addr 172.21.15.33:0/3638378399 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:04.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.511+0000 7f0a177fe640 1 --2- 172.21.15.33:0/3638378399 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a181066d0 0x7f0a18197730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.511+0000 7f0a167fc640 1 --2- 172.21.15.33:0/3638378399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a18101080 0x7f0a181971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.511+0000 7f0a177fe640 1 -- 172.21.15.33:0/3638378399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a18101080 msgr2=0x7f0a181971f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:04.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.511+0000 7f0a177fe640 1 --2- 172.21.15.33:0/3638378399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a18101080 0x7f0a181971f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.511+0000 7f0a177fe640 1 -- 172.21.15.33:0/3638378399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a181006c0 msgr2=0x7f0a18196cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:04.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.511+0000 7f0a177fe640 1 --2- 172.21.15.33:0/3638378399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a181006c0 0x7f0a18196cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.511+0000 7f0a177fe640 1 -- 172.21.15.33:0/3638378399 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a181a6520 con 0x7f0a181066d0 2024-03-19T02:57:04.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.511+0000 7f0a16ffd640 1 --2- 172.21.15.33:0/3638378399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a181006c0 0x7f0a18196cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:04.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.511+0000 7f0a177fe640 1 --2- 172.21.15.33:0/3638378399 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a181066d0 0x7f0a18197730 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0a0800e970 tx=0x7f0a0800ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:04.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.512+0000 7f0a0ffff640 1 -- 172.21.15.33:0/3638378399 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a0800ccc0 con 0x7f0a181066d0 2024-03-19T02:57:04.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.512+0000 7f0a1d463640 1 -- 172.21.15.33:0/3638378399 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a181a6810 con 0x7f0a181066d0 2024-03-19T02:57:04.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.512+0000 7f0a0ffff640 1 -- 172.21.15.33:0/3638378399 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0a080035f0 con 0x7f0a181066d0 2024-03-19T02:57:04.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.512+0000 7f0a0ffff640 1 -- 172.21.15.33:0/3638378399 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a0800ce60 con 0x7f0a181066d0 2024-03-19T02:57:04.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.512+0000 7f0a1d463640 1 -- 172.21.15.33:0/3638378399 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a181a6d30 con 0x7f0a181066d0 2024-03-19T02:57:04.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:04 smithi155 ceph-mon[35804]: pgmap v111: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-19T02:57:04.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.513+0000 7f0a1d463640 1 -- 172.21.15.33:0/3638378399 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a18101e00 con 0x7f0a181066d0 2024-03-19T02:57:04.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.514+0000 7f0a0ffff640 1 -- 172.21.15.33:0/3638378399 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f0a080037a0 con 0x7f0a181066d0 2024-03-19T02:57:04.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.514+0000 7f0a0ffff640 1 --2- 172.21.15.33:0/3638378399 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f0a040a2970 0x7f0a040a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.514+0000 7f0a16ffd640 1 --2- 172.21.15.33:0/3638378399 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f0a040a2970 0x7f0a040a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.514+0000 7f0a0ffff640 1 -- 172.21.15.33:0/3638378399 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f0a080cf530 con 0x7f0a181066d0 2024-03-19T02:57:04.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.515+0000 7f0a16ffd640 1 --2- 172.21.15.33:0/3638378399 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f0a040a2970 0x7f0a040a4e30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0a100049d0 tx=0x7f0a10009500 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:04.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.520+0000 7f0a0ffff640 1 -- 172.21.15.33:0/3638378399 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0a0809bf70 con 0x7f0a181066d0 2024-03-19T02:57:04.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.742+0000 7f0a1d463640 1 -- 172.21.15.33:0/3638378399 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a1806a440 con 0x7f0a181066d0 2024-03-19T02:57:04.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.743+0000 7f0a0ffff640 1 -- 172.21.15.33:0/3638378399 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11867 (secure 0 0 0) 0x7f0a0809b870 con 0x7f0a181066d0 2024-03-19T02:57:04.744 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:57:04.744 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:50:52.201798+0000","modified":"2024-03-19T02:53:11.748697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24488,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/3731232169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":3731232169},{"type":"v1","addr":"172.21.15.96:6839","nonce":3731232169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24457,24488,24509]},"id":1}]} 2024-03-19T02:57:04.744 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-03-19T02:57:04.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.745+0000 7f0a1d463640 1 -- 172.21.15.33:0/3638378399 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f0a040a2970 msgr2=0x7f0a040a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:04.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.745+0000 7f0a1d463640 1 --2- 172.21.15.33:0/3638378399 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f0a040a2970 0x7f0a040a4e30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0a100049d0 tx=0x7f0a10009500 comp rx=0 tx=0).stop 2024-03-19T02:57:04.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.745+0000 7f0a1d463640 1 -- 172.21.15.33:0/3638378399 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a181066d0 msgr2=0x7f0a18197730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:04.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.745+0000 7f0a1d463640 1 --2- 172.21.15.33:0/3638378399 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a181066d0 0x7f0a18197730 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0a0800e970 tx=0x7f0a0800ee30 comp rx=0 tx=0).stop 2024-03-19T02:57:04.746 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.745+0000 7f0a1d463640 1 -- 172.21.15.33:0/3638378399 shutdown_connections 2024-03-19T02:57:04.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.745+0000 7f0a1d463640 1 --2- 172.21.15.33:0/3638378399 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f0a040a2970 0x7f0a040a4e30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.745+0000 7f0a1d463640 1 --2- 172.21.15.33:0/3638378399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a18101080 0x7f0a181971f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.745+0000 7f0a1d463640 1 --2- 172.21.15.33:0/3638378399 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a181066d0 0x7f0a18197730 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.745+0000 7f0a1d463640 1 --2- 172.21.15.33:0/3638378399 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0a181006c0 0x7f0a18196cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.745+0000 7f0a1d463640 1 -- 172.21.15.33:0/3638378399 >> 172.21.15.33:0/3638378399 conn(0x7f0a18076aa0 msgr2=0x7f0a180fe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:04.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.745+0000 7f0a1d463640 1 -- 172.21.15.33:0/3638378399 shutdown_connections 2024-03-19T02:57:04.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.745+0000 7f0a1d463640 1 -- 172.21.15.33:0/3638378399 wait complete. 2024-03-19T02:57:04.757 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:57:04.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.861+0000 7fd307ade640 1 -- 172.21.15.33:0/2539576958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd3001077a0 msgr2=0x7fd300107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:04.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.861+0000 7fd307ade640 1 --2- 172.21.15.33:0/2539576958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd3001077a0 0x7fd300107b80 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc009930 tx=0x7fd2fc02bc30 comp rx=0 tx=0).stop 2024-03-19T02:57:04.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.861+0000 7fd307ade640 1 -- 172.21.15.33:0/2539576958 shutdown_connections 2024-03-19T02:57:04.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.861+0000 7fd307ade640 1 --2- 172.21.15.33:0/2539576958 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd300102150 0x7fd30010e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.861+0000 7fd307ade640 1 --2- 172.21.15.33:0/2539576958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd300101790 0x7fd300101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.861+0000 7fd307ade640 1 --2- 172.21.15.33:0/2539576958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd3001077a0 0x7fd300107b80 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.861+0000 7fd307ade640 1 -- 172.21.15.33:0/2539576958 >> 172.21.15.33:0/2539576958 conn(0x7fd3000fd2c0 msgr2=0x7fd3000ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:04.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.861+0000 7fd307ade640 1 -- 172.21.15.33:0/2539576958 shutdown_connections 2024-03-19T02:57:04.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.861+0000 7fd307ade640 1 -- 172.21.15.33:0/2539576958 wait complete. 2024-03-19T02:57:04.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.862+0000 7fd307ade640 1 Processor -- start 2024-03-19T02:57:04.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.862+0000 7fd307ade640 1 -- start start 2024-03-19T02:57:04.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.862+0000 7fd307ade640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd300101790 0x7fd300198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.862+0000 7fd307ade640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd300102150 0x7fd300198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.862+0000 7fd307ade640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3001077a0 0x7fd30019c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.862+0000 7fd307ade640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd300111a60 con 0x7fd300102150 2024-03-19T02:57:04.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.862+0000 7fd307ade640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd3001118e0 con 0x7fd300101790 2024-03-19T02:57:04.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.862+0000 7fd307ade640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd300111be0 con 0x7fd3001077a0 2024-03-19T02:57:04.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.862+0000 7fd305079640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd300102150 0x7fd300198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.862+0000 7fd305079640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd300102150 0x7fd300198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59482/0 (socket says 172.21.15.33:59482) 2024-03-19T02:57:04.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.862+0000 7fd305079640 1 -- 172.21.15.33:0/118198430 learned_addr learned my addr 172.21.15.33:0/118198430 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:04.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.862+0000 7fd30587a640 1 --2- 172.21.15.33:0/118198430 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd300101790 0x7fd300198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.862+0000 7fd30607b640 1 --2- 172.21.15.33:0/118198430 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3001077a0 0x7fd30019c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.862+0000 7fd305079640 1 -- 172.21.15.33:0/118198430 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3001077a0 msgr2=0x7fd30019c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:04.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.862+0000 7fd305079640 1 --2- 172.21.15.33:0/118198430 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3001077a0 0x7fd30019c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.862+0000 7fd305079640 1 -- 172.21.15.33:0/118198430 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd300101790 msgr2=0x7fd300198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:04.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.862+0000 7fd305079640 1 --2- 172.21.15.33:0/118198430 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd300101790 0x7fd300198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.862+0000 7fd305079640 1 -- 172.21.15.33:0/118198430 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd30019cff0 con 0x7fd300102150 2024-03-19T02:57:04.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.862+0000 7fd305079640 1 --2- 172.21.15.33:0/118198430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd300102150 0x7fd300198540 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fd2f000e9f0 tx=0x7fd2f000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:04.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.863+0000 7fd2f6ffd640 1 -- 172.21.15.33:0/118198430 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f000cd80 con 0x7fd300102150 2024-03-19T02:57:04.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.863+0000 7fd307ade640 1 -- 172.21.15.33:0/118198430 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd30019d2e0 con 0x7fd300102150 2024-03-19T02:57:04.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.863+0000 7fd2f6ffd640 1 -- 172.21.15.33:0/118198430 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd2f00108e0 con 0x7fd300102150 2024-03-19T02:57:04.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.863+0000 7fd2f6ffd640 1 -- 172.21.15.33:0/118198430 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0018920 con 0x7fd300102150 2024-03-19T02:57:04.866 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.863+0000 7fd307ade640 1 -- 172.21.15.33:0/118198430 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3001a4b70 con 0x7fd300102150 2024-03-19T02:57:04.873 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.865+0000 7fd2f6ffd640 1 -- 172.21.15.33:0/118198430 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fd2f0010a90 con 0x7fd300102150 2024-03-19T02:57:04.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.865+0000 7fd30a79c740 1 Processor -- start 2024-03-19T02:57:04.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.865+0000 7fd30a79c740 1 -- start start 2024-03-19T02:57:04.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.865+0000 7fd30a79c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fea551cb0 0x561fea552090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.865+0000 7fd2f6ffd640 1 --2- 172.21.15.33:0/118198430 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd2ec0a2970 0x7fd2ec0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.865+0000 7fd30a79c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fea552660 0x561fea55f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.865+0000 7fd30a79c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fea55fbe0 0x561fea561fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.866+0000 7fd30a79c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561fea4b54b0 con 0x561fea55fbe0 2024-03-19T02:57:04.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.866+0000 7fd30a79c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x561fea45dd70 con 0x561fea552660 2024-03-19T02:57:04.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.866+0000 7fd30a79c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x561fea4b1550 con 0x561fea551cb0 2024-03-19T02:57:04.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.866+0000 7fd305079640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fea552660 0x561fea55f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.866+0000 7fd30587a640 1 --2- 172.21.15.33:0/118198430 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd2ec0a2970 0x7fd2ec0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.866+0000 7fd30607b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fea551cb0 0x561fea552090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.874 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.866+0000 7fd30607b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fea551cb0 0x561fea552090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:54858/0 (socket says 172.21.15.33:54858) 2024-03-19T02:57:04.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.866+0000 7fd30607b640 1 -- 172.21.15.33:0/1737212926 learned_addr learned my addr 172.21.15.33:0/1737212926 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:04.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.866+0000 7fd305079640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fea552660 0x561fea55f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:33938/0 (socket says 172.21.15.33:33938) 2024-03-19T02:57:04.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.866+0000 7fd30607b640 1 --2- 172.21.15.33:0/1737212926 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fea55fbe0 0x561fea561fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.866+0000 7fd2f6ffd640 1 -- 172.21.15.33:0/118198430 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fd2f00cf430 con 0x7fd300102150 2024-03-19T02:57:04.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.866+0000 7fd30607b640 1 -- 172.21.15.33:0/1737212926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fea552660 msgr2=0x561fea55f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:04.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.866+0000 7fd30607b640 1 --2- 172.21.15.33:0/1737212926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fea552660 0x561fea55f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.866+0000 7fd30607b640 1 -- 172.21.15.33:0/1737212926 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fea55fbe0 msgr2=0x561fea561fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:04.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.866+0000 7fd30607b640 1 --2- 172.21.15.33:0/1737212926 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fea55fbe0 0x561fea561fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.866+0000 7fd30607b640 1 -- 172.21.15.33:0/1737212926 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fea480210 con 0x561fea551cb0 2024-03-19T02:57:04.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.866+0000 7fd30607b640 1 --2- 172.21.15.33:0/1737212926 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fea551cb0 0x561fea552090 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8005f40 tx=0x7fd2f8005bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f475fadf098039f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:04.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.866+0000 7fd30587a640 1 --2- 172.21.15.33:0/118198430 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd2ec0a2970 0x7fd2ec0a4e30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc005ab0 tx=0x7fd2fc005fd0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:04.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.866+0000 7fd2ebfff640 1 -- 172.21.15.33:0/1737212926 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f8011130 con 0x561fea551cb0 2024-03-19T02:57:04.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.866+0000 7fd2ebfff640 1 -- 172.21.15.33:0/1737212926 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd2f80112d0 con 0x561fea551cb0 2024-03-19T02:57:04.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.866+0000 7fd2ebfff640 1 -- 172.21.15.33:0/1737212926 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f800fb10 con 0x561fea551cb0 2024-03-19T02:57:04.875 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.867+0000 7fd30a79c740 1 -- 172.21.15.33:0/1737212926 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fea551cb0 msgr2=0x561fea552090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:04.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.867+0000 7fd30a79c740 1 --2- 172.21.15.33:0/1737212926 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fea551cb0 0x561fea552090 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8005f40 tx=0x7fd2f8005bd0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.867+0000 7fd30a79c740 1 -- 172.21.15.33:0/1737212926 shutdown_connections 2024-03-19T02:57:04.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.867+0000 7fd30a79c740 1 --2- 172.21.15.33:0/1737212926 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fea55fbe0 0x561fea561fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.867+0000 7fd30a79c740 1 --2- 172.21.15.33:0/1737212926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fea552660 0x561fea55f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.867+0000 7fd30a79c740 1 --2- 172.21.15.33:0/1737212926 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fea551cb0 0x561fea552090 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.867+0000 7fd30a79c740 1 -- 172.21.15.33:0/1737212926 >> 172.21.15.33:0/1737212926 conn(0x561fea54ca00 msgr2=0x561fea54cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:04.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.867+0000 7fd30a79c740 1 -- 172.21.15.33:0/1737212926 shutdown_connections 2024-03-19T02:57:04.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.867+0000 7fd30a79c740 1 -- 172.21.15.33:0/1737212926 wait complete. 2024-03-19T02:57:04.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.867+0000 7fd30a79c740 1 -- start start 2024-03-19T02:57:04.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.868+0000 7fd30a79c740 1 Processor -- start 2024-03-19T02:57:04.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.868+0000 7fd30a79c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fea551cb0 0x561fea55cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.868+0000 7fd30a79c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fea552660 0x561fea55d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.868+0000 7fd30a79c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fea55fbe0 0x561fea55d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.868+0000 7fd30a79c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561fea5629a0 con 0x561fea551cb0 2024-03-19T02:57:04.876 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.868+0000 7fd30a79c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x561fea471310 con 0x561fea55fbe0 2024-03-19T02:57:04.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.868+0000 7fd30a79c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x561fea562b20 con 0x561fea552660 2024-03-19T02:57:04.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.868+0000 7fd30a79c740 10 client.0 _ll_register_callbacks cb 0x561fea4652d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:57:04.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.868+0000 7fd30587a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fea551cb0 0x561fea55cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.868+0000 7fd30587a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fea551cb0 0x561fea55cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59488/0 (socket says 172.21.15.33:59488) 2024-03-19T02:57:04.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.868+0000 7fd30587a640 1 -- 172.21.15.33:0/3548417152 learned_addr learned my addr 172.21.15.33:0/3548417152 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:04.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.868+0000 7fd30607b640 1 --2- 172.21.15.33:0/3548417152 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fea55fbe0 0x561fea55d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.868+0000 7fd305079640 1 --2- 172.21.15.33:0/3548417152 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fea552660 0x561fea55d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.868+0000 7fd30587a640 1 -- 172.21.15.33:0/3548417152 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fea552660 msgr2=0x561fea55d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:04.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.868+0000 7fd30587a640 1 --2- 172.21.15.33:0/3548417152 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fea552660 0x561fea55d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.868+0000 7fd30587a640 1 -- 172.21.15.33:0/3548417152 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fea55fbe0 msgr2=0x561fea55d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:04.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.868+0000 7fd30587a640 1 --2- 172.21.15.33:0/3548417152 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fea55fbe0 0x561fea55d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.868+0000 7fd30587a640 1 -- 172.21.15.33:0/3548417152 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fea1c93f0 con 0x561fea551cb0 2024-03-19T02:57:04.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.869+0000 7fd30587a640 1 --2- 172.21.15.33:0/3548417152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fea551cb0 0x561fea55cba0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc035000 tx=0x7fd2fc009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4679447c6d1ad060 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:04.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.869+0000 7fd2e97fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:57:04.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.869+0000 7fd2e97fa640 1 -- 172.21.15.33:0/3548417152 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc04f020 con 0x561fea551cb0 2024-03-19T02:57:04.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.869+0000 7fd2e97fa640 1 -- 172.21.15.33:0/3548417152 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fd2fc02cb20 con 0x561fea551cb0 2024-03-19T02:57:04.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.869+0000 7fd30a79c740 1 -- 172.21.15.33:0/3548417152 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2f80112d0 con 0x561fea551cb0 2024-03-19T02:57:04.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.869+0000 7fd2e97fa640 1 -- 172.21.15.33:0/3548417152 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd2fc05a050 con 0x561fea551cb0 2024-03-19T02:57:04.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.869+0000 7fd30a79c740 10 client.16416 fetch_fsmap learned FSMap version 25 2024-03-19T02:57:04.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.869+0000 7fd30a79c740 1 -- 172.21.15.33:0/3548417152 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x561fea2937d0 con 0x561fea551cb0 2024-03-19T02:57:04.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.869+0000 7fd2e97fa640 1 -- 172.21.15.33:0/3548417152 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd2f80112d0 con 0x561fea551cb0 2024-03-19T02:57:04.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.869+0000 7fd2e97fa640 1 -- 172.21.15.33:0/3548417152 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fd2fc003820 con 0x561fea551cb0 2024-03-19T02:57:04.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.869+0000 7fd30a79c740 10 client.16416 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:57:04.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.869+0000 7fd30a79c740 10 client.16416 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T02:57:04.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.869+0000 7fd30a79c740 20 client.16416 populate_metadata read hostname 'smithi033' 2024-03-19T02:57:04.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.869+0000 7fd30a79c740 1 --2- 172.21.15.33:0/3548417152 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x561fea570030 0x561fea572460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.869+0000 7fd30a79c740 4 client.16416 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:57:04.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.869+0000 7fd30a79c740 1 -- 172.21.15.33:0/3548417152 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561fea5133e0 con 0x561fea570030 2024-03-19T02:57:04.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.870+0000 7fd305079640 1 --2- 172.21.15.33:0/3548417152 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x561fea570030 0x561fea572460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.870+0000 7fd2e97fa640 10 client.16416 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T02:57:04.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.870+0000 7fd305079640 1 --2- 172.21.15.33:0/3548417152 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x561fea570030 0x561fea572460 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=185cfe6aa112a693 server_cookie=13d52e6c526146dc in_seq=0 out_seq=0 2024-03-19T02:57:04.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.870+0000 7fd2e97fa640 1 -- 172.21.15.33:0/3548417152 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561fea5133e0 con 0x561fea570030 2024-03-19T02:57:04.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.870+0000 7fd2e97fa640 10 client.16416 handle_command_reply: tid=0 2024-03-19T02:57:04.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.871+0000 7fd30a79c740 1 client.16416 shutdown 2024-03-19T02:57:04.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.871+0000 7fd30a79c740 1 -- 172.21.15.33:0/3548417152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fea551cb0 msgr2=0x561fea55cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:04.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.871+0000 7fd30a79c740 1 --2- 172.21.15.33:0/3548417152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fea551cb0 0x561fea55cba0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc035000 tx=0x7fd2fc009260 comp rx=0 tx=0).stop 2024-03-19T02:57:04.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.871+0000 7fd30a79c740 1 -- 172.21.15.33:0/3548417152 shutdown_connections 2024-03-19T02:57:04.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.871+0000 7fd30a79c740 1 --2- 172.21.15.33:0/3548417152 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x561fea570030 0x561fea572460 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.871+0000 7fd30a79c740 1 --2- 172.21.15.33:0/3548417152 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fea552660 0x561fea55d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.871+0000 7fd2e97fa640 0 client.16416 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T02:57:04.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.871+0000 7fd30a79c740 1 --2- 172.21.15.33:0/3548417152 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fea55fbe0 0x561fea55d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.871+0000 7fd30a79c740 1 --2- 172.21.15.33:0/3548417152 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fea551cb0 0x561fea55cba0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.871+0000 7fd30a79c740 1 -- 172.21.15.33:0/3548417152 >> 172.21.15.33:0/3548417152 conn(0x561fea54ca00 msgr2=0x561fea54e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:04.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.871+0000 7fd30a79c740 1 -- 172.21.15.33:0/3548417152 shutdown_connections 2024-03-19T02:57:04.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.871+0000 7fd30a79c740 1 -- 172.21.15.33:0/3548417152 wait complete. 2024-03-19T02:57:04.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.871+0000 7fd30a79c740 20 client.16416 trim_cache size 0 max 16384 2024-03-19T02:57:04.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.884+0000 7fd30a79c740 1 Processor -- start 2024-03-19T02:57:04.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.884+0000 7fd30a79c740 1 -- start start 2024-03-19T02:57:04.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.884+0000 7fd30a79c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fea551cb0 0x561fea55c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.884+0000 7fd30a79c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fea552660 0x561fea54dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.884+0000 7fd30a79c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fea55fbe0 0x561fea56b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.884+0000 7fd30a79c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd3001af210 con 0x561fea55fbe0 2024-03-19T02:57:04.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.884+0000 7fd30a79c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd3001acae0 con 0x561fea552660 2024-03-19T02:57:04.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.884+0000 7fd30a79c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd3001aa910 con 0x561fea551cb0 2024-03-19T02:57:04.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.885+0000 7fd30607b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fea55fbe0 0x561fea56b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.885+0000 7fd30607b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fea55fbe0 0x561fea56b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59504/0 (socket says 172.21.15.33:59504) 2024-03-19T02:57:04.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.885+0000 7fd305079640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fea552660 0x561fea54dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.885+0000 7fd30607b640 1 -- 172.21.15.33:0/852381164 learned_addr learned my addr 172.21.15.33:0/852381164 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:04.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.885+0000 7fd30587a640 1 --2- 172.21.15.33:0/852381164 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fea551cb0 0x561fea55c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.885+0000 7fd30607b640 1 -- 172.21.15.33:0/852381164 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fea551cb0 msgr2=0x561fea55c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:04.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.885+0000 7fd30607b640 1 --2- 172.21.15.33:0/852381164 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fea551cb0 0x561fea55c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.885+0000 7fd30607b640 1 -- 172.21.15.33:0/852381164 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fea552660 msgr2=0x561fea54dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:04.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.885+0000 7fd30607b640 1 --2- 172.21.15.33:0/852381164 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fea552660 0x561fea54dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.885+0000 7fd30607b640 1 -- 172.21.15.33:0/852381164 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fea5513c0 con 0x561fea55fbe0 2024-03-19T02:57:04.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.885+0000 7fd30607b640 1 --2- 172.21.15.33:0/852381164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fea55fbe0 0x561fea56b7b0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8005f40 tx=0x7fd2f8005c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d88b70b172fcd66b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:04.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.885+0000 7fd2eb7fe640 1 -- 172.21.15.33:0/852381164 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f8011300 con 0x561fea55fbe0 2024-03-19T02:57:04.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.885+0000 7fd2eb7fe640 1 -- 172.21.15.33:0/852381164 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd2f800f850 con 0x561fea55fbe0 2024-03-19T02:57:04.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.885+0000 7fd2eb7fe640 1 -- 172.21.15.33:0/852381164 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f80052b0 con 0x561fea55fbe0 2024-03-19T02:57:04.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.886+0000 7fd30a79c740 1 -- 172.21.15.33:0/852381164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fea55fbe0 msgr2=0x561fea56b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:04.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.886+0000 7fd30a79c740 1 --2- 172.21.15.33:0/852381164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fea55fbe0 0x561fea56b7b0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8005f40 tx=0x7fd2f8005c60 comp rx=0 tx=0).stop 2024-03-19T02:57:04.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.886+0000 7fd30a79c740 1 -- 172.21.15.33:0/852381164 shutdown_connections 2024-03-19T02:57:04.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.886+0000 7fd30a79c740 1 --2- 172.21.15.33:0/852381164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fea55fbe0 0x561fea56b7b0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.886+0000 7fd30a79c740 1 --2- 172.21.15.33:0/852381164 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fea552660 0x561fea54dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.886+0000 7fd30a79c740 1 --2- 172.21.15.33:0/852381164 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fea551cb0 0x561fea55c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.886+0000 7fd30a79c740 1 -- 172.21.15.33:0/852381164 >> 172.21.15.33:0/852381164 conn(0x561fea54ca00 msgr2=0x561fea54bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:04.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.886+0000 7fd30a79c740 1 -- 172.21.15.33:0/852381164 shutdown_connections 2024-03-19T02:57:04.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.886+0000 7fd30a79c740 1 -- 172.21.15.33:0/852381164 wait complete. 2024-03-19T02:57:04.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.886+0000 7fd30a79c740 1 -- start start 2024-03-19T02:57:04.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.886+0000 7fd30a79c740 1 Processor -- start 2024-03-19T02:57:04.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.886+0000 7fd30a79c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fea551cb0 0x561fea5636d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.886+0000 7fd30a79c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fea552660 0x561fea563c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.886+0000 7fd30a79c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fea55fbe0 0x561fea564150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.886+0000 7fd30a79c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd3001aafa0 con 0x561fea55fbe0 2024-03-19T02:57:04.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.886+0000 7fd30a79c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd3001ac410 con 0x561fea551cb0 2024-03-19T02:57:04.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.886+0000 7fd30a79c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd3001ab670 con 0x561fea552660 2024-03-19T02:57:04.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.886+0000 7fd30a79c740 10 client.0 _ll_register_callbacks cb 0x7fd2f80093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:57:04.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.887+0000 7fd30607b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fea55fbe0 0x561fea564150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.887+0000 7fd30607b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fea55fbe0 0x561fea564150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59518/0 (socket says 172.21.15.33:59518) 2024-03-19T02:57:04.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.887+0000 7fd30587a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fea551cb0 0x561fea5636d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.887+0000 7fd30607b640 1 -- 172.21.15.33:0/3175533356 learned_addr learned my addr 172.21.15.33:0/3175533356 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:04.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.887+0000 7fd305079640 1 --2- 172.21.15.33:0/3175533356 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fea552660 0x561fea563c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.887+0000 7fd30607b640 1 -- 172.21.15.33:0/3175533356 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fea552660 msgr2=0x561fea563c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:04.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.887+0000 7fd30607b640 1 --2- 172.21.15.33:0/3175533356 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fea552660 0x561fea563c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.887+0000 7fd30607b640 1 -- 172.21.15.33:0/3175533356 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fea551cb0 msgr2=0x561fea5636d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:04.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.887+0000 7fd30607b640 1 --2- 172.21.15.33:0/3175533356 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fea551cb0 0x561fea5636d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.887+0000 7fd30607b640 1 -- 172.21.15.33:0/3175533356 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fea522880 con 0x561fea55fbe0 2024-03-19T02:57:04.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.887+0000 7fd30587a640 1 --2- 172.21.15.33:0/3175533356 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fea551cb0 0x561fea5636d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:04.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.887+0000 7fd30607b640 1 --2- 172.21.15.33:0/3175533356 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fea55fbe0 0x561fea564150 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8011470 tx=0x7fd2f80114a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b49c90dfd3674734 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:04.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.887+0000 7fd2e97fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:57:04.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.887+0000 7fd2e97fa640 1 -- 172.21.15.33:0/3175533356 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f8003970 con 0x561fea55fbe0 2024-03-19T02:57:04.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.887+0000 7fd30a79c740 1 -- 172.21.15.33:0/3175533356 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2f800f850 con 0x561fea55fbe0 2024-03-19T02:57:04.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.887+0000 7fd2e97fa640 1 -- 172.21.15.33:0/3175533356 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fd2f8003b10 con 0x561fea55fbe0 2024-03-19T02:57:04.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.887+0000 7fd2e97fa640 1 -- 172.21.15.33:0/3175533356 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd2f8003d10 con 0x561fea55fbe0 2024-03-19T02:57:04.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.887+0000 7fd30a79c740 10 client.16422 fetch_fsmap learned FSMap version 25 2024-03-19T02:57:04.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.887+0000 7fd30a79c740 1 -- 172.21.15.33:0/3175533356 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x561fea5d6470 con 0x561fea55fbe0 2024-03-19T02:57:04.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.888+0000 7fd2e97fa640 1 -- 172.21.15.33:0/3175533356 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd2f800f850 con 0x561fea55fbe0 2024-03-19T02:57:04.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.888+0000 7fd2e97fa640 1 -- 172.21.15.33:0/3175533356 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fd2f801d020 con 0x561fea55fbe0 2024-03-19T02:57:04.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.888+0000 7fd30a79c740 10 client.16422 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:57:04.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.888+0000 7fd30a79c740 10 client.16422 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T02:57:04.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.888+0000 7fd30a79c740 20 client.16422 populate_metadata read hostname 'smithi033' 2024-03-19T02:57:04.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.888+0000 7fd30a79c740 1 --2- 172.21.15.33:0/3175533356 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x561fea5d68f0 0x561fea5d8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:04.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.888+0000 7fd30a79c740 4 client.16422 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:57:04.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.888+0000 7fd30a79c740 1 -- 172.21.15.33:0/3175533356 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561fea4c29f0 con 0x561fea5d68f0 2024-03-19T02:57:04.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.888+0000 7fd30587a640 1 --2- 172.21.15.33:0/3175533356 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x561fea5d68f0 0x561fea5d8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:04.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.888+0000 7fd2e97fa640 10 client.16422 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T02:57:04.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.888+0000 7fd30587a640 1 --2- 172.21.15.33:0/3175533356 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x561fea5d68f0 0x561fea5d8d10 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a84304d8eedd2b83 server_cookie=a9cceec1f36d0571 in_seq=0 out_seq=0 2024-03-19T02:57:04.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.888+0000 7fd2e97fa640 1 -- 172.21.15.33:0/3175533356 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561fea4c29f0 con 0x561fea5d68f0 2024-03-19T02:57:04.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.888+0000 7fd2e97fa640 10 client.16422 handle_command_reply: tid=0 2024-03-19T02:57:04.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.888+0000 7fd30a79c740 1 client.16422 shutdown 2024-03-19T02:57:04.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.889+0000 7fd30a79c740 1 -- 172.21.15.33:0/3175533356 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fea55fbe0 msgr2=0x561fea564150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:04.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.889+0000 7fd30a79c740 1 --2- 172.21.15.33:0/3175533356 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fea55fbe0 0x561fea564150 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8011470 tx=0x7fd2f80114a0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.889+0000 7fd30a79c740 1 -- 172.21.15.33:0/3175533356 shutdown_connections 2024-03-19T02:57:04.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.889+0000 7fd30a79c740 1 --2- 172.21.15.33:0/3175533356 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x561fea5d68f0 0x561fea5d8d10 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.889+0000 7fd30a79c740 1 --2- 172.21.15.33:0/3175533356 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fea552660 0x561fea563c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.889+0000 7fd30a79c740 1 --2- 172.21.15.33:0/3175533356 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fea55fbe0 0x561fea564150 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.889+0000 7fd2e97fa640 0 client.16422 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T02:57:04.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.889+0000 7fd30a79c740 1 --2- 172.21.15.33:0/3175533356 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fea551cb0 0x561fea5636d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.889+0000 7fd30a79c740 1 -- 172.21.15.33:0/3175533356 >> 172.21.15.33:0/3175533356 conn(0x561fea54ca00 msgr2=0x561fea55d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:04.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.889+0000 7fd30a79c740 1 -- 172.21.15.33:0/3175533356 shutdown_connections 2024-03-19T02:57:04.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.889+0000 7fd30a79c740 1 -- 172.21.15.33:0/3175533356 wait complete. 2024-03-19T02:57:04.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.889+0000 7fd30a79c740 20 client.16422 trim_cache size 0 max 16384 2024-03-19T02:57:04.897 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:57:04.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.889+0000 7fd307ade640 1 -- 172.21.15.33:0/118198430 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd2ec0a2970 msgr2=0x7fd2ec0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:04.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.889+0000 7fd307ade640 1 --2- 172.21.15.33:0/118198430 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd2ec0a2970 0x7fd2ec0a4e30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc005ab0 tx=0x7fd2fc005fd0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.889+0000 7fd307ade640 1 -- 172.21.15.33:0/118198430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd300102150 msgr2=0x7fd300198540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:04.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.889+0000 7fd307ade640 1 --2- 172.21.15.33:0/118198430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd300102150 0x7fd300198540 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fd2f000e9f0 tx=0x7fd2f000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.889+0000 7fd307ade640 1 -- 172.21.15.33:0/118198430 shutdown_connections 2024-03-19T02:57:04.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.889+0000 7fd307ade640 1 --2- 172.21.15.33:0/118198430 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd2ec0a2970 0x7fd2ec0a4e30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.889+0000 7fd307ade640 1 --2- 172.21.15.33:0/118198430 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd3001077a0 0x7fd30019c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.889+0000 7fd307ade640 1 --2- 172.21.15.33:0/118198430 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd300102150 0x7fd300198540 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.889+0000 7fd307ade640 1 --2- 172.21.15.33:0/118198430 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd300101790 0x7fd300198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:04.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.889+0000 7fd307ade640 1 -- 172.21.15.33:0/118198430 >> 172.21.15.33:0/118198430 conn(0x7fd3000fd2c0 msgr2=0x7fd3000fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:04.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.889+0000 7fd307ade640 1 -- 172.21.15.33:0/118198430 shutdown_connections 2024-03-19T02:57:04.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:04.889+0000 7fd307ade640 1 -- 172.21.15.33:0/118198430 wait complete. 2024-03-19T02:57:04.901 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:57:05.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.004+0000 7f9f4ea12640 1 -- 172.21.15.33:0/2387864638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f4806a7f0 msgr2=0x7f9f4810e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:05.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.004+0000 7f9f4ea12640 1 --2- 172.21.15.33:0/2387864638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f4806a7f0 0x7f9f4810e670 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9f40009ca0 tx=0x7f9f4002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.004+0000 7f9f4ea12640 1 -- 172.21.15.33:0/2387864638 shutdown_connections 2024-03-19T02:57:05.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.004+0000 7f9f4ea12640 1 --2- 172.21.15.33:0/2387864638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f4806a7f0 0x7f9f4810e670 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.004+0000 7f9f4ea12640 1 --2- 172.21.15.33:0/2387864638 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f48069e50 0x7f9f4806a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.004+0000 7f9f4ea12640 1 --2- 172.21.15.33:0/2387864638 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f48107890 0x7f9f48107c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.004+0000 7f9f4ea12640 1 -- 172.21.15.33:0/2387864638 >> 172.21.15.33:0/2387864638 conn(0x7f9f480774c0 msgr2=0x7f9f480778d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:05.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.004+0000 7f9f4ea12640 1 -- 172.21.15.33:0/2387864638 shutdown_connections 2024-03-19T02:57:05.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.004+0000 7f9f4ea12640 1 -- 172.21.15.33:0/2387864638 wait complete. 2024-03-19T02:57:05.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.005+0000 7f9f4ea12640 1 Processor -- start 2024-03-19T02:57:05.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.005+0000 7f9f4ea12640 1 -- start start 2024-03-19T02:57:05.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.005+0000 7f9f4ea12640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f48069e50 0x7f9f48197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.005+0000 7f9f4ea12640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f4806a7f0 0x7f9f481983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.005+0000 7f9f4ea12640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f48107890 0x7f9f4819c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.005+0000 7f9f4ea12640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9f48111a60 con 0x7f9f48069e50 2024-03-19T02:57:05.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.005+0000 7f9f4ea12640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9f481118e0 con 0x7f9f48107890 2024-03-19T02:57:05.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.005+0000 7f9f4ea12640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9f48111be0 con 0x7f9f4806a7f0 2024-03-19T02:57:05.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.005+0000 7f9f47fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f48069e50 0x7f9f48197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.005+0000 7f9f47fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f48069e50 0x7f9f48197ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59538/0 (socket says 172.21.15.33:59538) 2024-03-19T02:57:05.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.005+0000 7f9f47fff640 1 -- 172.21.15.33:0/373226518 learned_addr learned my addr 172.21.15.33:0/373226518 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:05.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.005+0000 7f9f4cfaf640 1 --2- 172.21.15.33:0/373226518 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f48107890 0x7f9f4819c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.005+0000 7f9f477fe640 1 --2- 172.21.15.33:0/373226518 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f4806a7f0 0x7f9f481983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.005+0000 7f9f47fff640 1 -- 172.21.15.33:0/373226518 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f4806a7f0 msgr2=0x7f9f481983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:05.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.005+0000 7f9f47fff640 1 --2- 172.21.15.33:0/373226518 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f4806a7f0 0x7f9f481983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.005+0000 7f9f47fff640 1 -- 172.21.15.33:0/373226518 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f48107890 msgr2=0x7f9f4819c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:05.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.005+0000 7f9f47fff640 1 --2- 172.21.15.33:0/373226518 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f48107890 0x7f9f4819c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.005+0000 7f9f47fff640 1 -- 172.21.15.33:0/373226518 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f4819ce90 con 0x7f9f48069e50 2024-03-19T02:57:05.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.006+0000 7f9f47fff640 1 --2- 172.21.15.33:0/373226518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f48069e50 0x7f9f48197ea0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f9f3000e9f0 tx=0x7f9f3000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:05.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.006+0000 7f9f457fa640 1 -- 172.21.15.33:0/373226518 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f3000cd80 con 0x7f9f48069e50 2024-03-19T02:57:05.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.006+0000 7f9f4ea12640 1 -- 172.21.15.33:0/373226518 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f4819d180 con 0x7f9f48069e50 2024-03-19T02:57:05.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.006+0000 7f9f4ea12640 1 -- 172.21.15.33:0/373226518 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f481a4a10 con 0x7f9f48069e50 2024-03-19T02:57:05.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.006+0000 7f9f457fa640 1 -- 172.21.15.33:0/373226518 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9f300108e0 con 0x7f9f48069e50 2024-03-19T02:57:05.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.006+0000 7f9f457fa640 1 -- 172.21.15.33:0/373226518 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f300188e0 con 0x7f9f48069e50 2024-03-19T02:57:05.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.008+0000 7f9f457fa640 1 -- 172.21.15.33:0/373226518 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f9f30010a90 con 0x7f9f48069e50 2024-03-19T02:57:05.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.008+0000 7f9f51603740 1 Processor -- start 2024-03-19T02:57:05.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.009+0000 7f9f51603740 1 -- start start 2024-03-19T02:57:05.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.009+0000 7f9f457fa640 1 --2- 172.21.15.33:0/373226518 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f9f340a2970 0x7f9f340a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.009+0000 7f9f51603740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564e08ff72d0 0x564e08ff76b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.009+0000 7f9f51603740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e08ff7bf0 0x564e09000290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.009+0000 7f9f51603740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564e090007d0 0x564e09002bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.009+0000 7f9f51603740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564e08f5a4b0 con 0x564e08ff7bf0 2024-03-19T02:57:05.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.009+0000 7f9f51603740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x564e08f02d70 con 0x564e090007d0 2024-03-19T02:57:05.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.009+0000 7f9f51603740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564e08f56550 con 0x564e08ff72d0 2024-03-19T02:57:05.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.009+0000 7f9f477fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e08ff7bf0 0x564e09000290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.009+0000 7f9f4cfaf640 1 --2- 172.21.15.33:0/373226518 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f9f340a2970 0x7f9f340a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.009+0000 7f9f477fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e08ff7bf0 0x564e09000290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59542/0 (socket says 172.21.15.33:59542) 2024-03-19T02:57:05.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.009+0000 7f9f457fa640 1 -- 172.21.15.33:0/373226518 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f9f300cf4c0 con 0x7f9f48069e50 2024-03-19T02:57:05.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.009+0000 7f9f477fe640 1 -- 172.21.15.33:0/557466698 learned_addr learned my addr 172.21.15.33:0/557466698 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:05.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.009+0000 7f9f47fff640 1 --2- 172.21.15.33:0/557466698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564e08ff72d0 0x564e08ff76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.009+0000 7f9f4cfaf640 1 --2- 172.21.15.33:0/557466698 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564e090007d0 0x564e09002bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.009+0000 7f9f477fe640 1 -- 172.21.15.33:0/557466698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564e08ff72d0 msgr2=0x564e08ff76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:05.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.009+0000 7f9f477fe640 1 --2- 172.21.15.33:0/557466698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564e08ff72d0 0x564e08ff76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.009+0000 7f9f477fe640 1 -- 172.21.15.33:0/557466698 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564e090007d0 msgr2=0x564e09002bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:05.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.009+0000 7f9f477fe640 1 --2- 172.21.15.33:0/557466698 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564e090007d0 0x564e09002bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.009+0000 7f9f477fe640 1 -- 172.21.15.33:0/557466698 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e08f25210 con 0x564e08ff7bf0 2024-03-19T02:57:05.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.009+0000 7f9f4cfaf640 1 --2- 172.21.15.33:0/373226518 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f9f340a2970 0x7f9f340a4e30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9f40004af0 tx=0x7f9f40009960 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:05.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.009+0000 7f9f477fe640 1 --2- 172.21.15.33:0/557466698 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e08ff7bf0 0x564e09000290 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f9f38009c90 tx=0x7f9f38011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cdf52ecfffe94f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:05.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.010+0000 7f9f3e7fc640 1 -- 172.21.15.33:0/557466698 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f3801e070 con 0x564e08ff7bf0 2024-03-19T02:57:05.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.010+0000 7f9f3e7fc640 1 -- 172.21.15.33:0/557466698 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9f38019ae0 con 0x564e08ff7bf0 2024-03-19T02:57:05.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.010+0000 7f9f3e7fc640 1 -- 172.21.15.33:0/557466698 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f3800e7f0 con 0x564e08ff7bf0 2024-03-19T02:57:05.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.010+0000 7f9f51603740 1 -- 172.21.15.33:0/557466698 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e08ff7bf0 msgr2=0x564e09000290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:05.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.010+0000 7f9f51603740 1 --2- 172.21.15.33:0/557466698 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e08ff7bf0 0x564e09000290 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f9f38009c90 tx=0x7f9f38011810 comp rx=0 tx=0).stop 2024-03-19T02:57:05.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.010+0000 7f9f51603740 1 -- 172.21.15.33:0/557466698 shutdown_connections 2024-03-19T02:57:05.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.010+0000 7f9f51603740 1 --2- 172.21.15.33:0/557466698 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564e090007d0 0x564e09002bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.010+0000 7f9f51603740 1 --2- 172.21.15.33:0/557466698 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e08ff7bf0 0x564e09000290 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.010+0000 7f9f51603740 1 --2- 172.21.15.33:0/557466698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564e08ff72d0 0x564e08ff76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.010+0000 7f9f51603740 1 -- 172.21.15.33:0/557466698 >> 172.21.15.33:0/557466698 conn(0x564e08ff1a00 msgr2=0x564e08ff1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:05.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.010+0000 7f9f51603740 1 -- 172.21.15.33:0/557466698 shutdown_connections 2024-03-19T02:57:05.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.010+0000 7f9f51603740 1 -- 172.21.15.33:0/557466698 wait complete. 2024-03-19T02:57:05.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.011+0000 7f9f51603740 1 -- start start 2024-03-19T02:57:05.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.011+0000 7f9f51603740 1 Processor -- start 2024-03-19T02:57:05.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.011+0000 7f9f51603740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e08ff72d0 0x564e08ffd820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.011+0000 7f9f51603740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564e08ff7bf0 0x564e08ffdd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.011+0000 7f9f51603740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564e090007d0 0x564e08ffe2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.011+0000 7f9f51603740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564e090035c0 con 0x564e08ff72d0 2024-03-19T02:57:05.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.011+0000 7f9f51603740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x564e08f16310 con 0x564e08ff7bf0 2024-03-19T02:57:05.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.011+0000 7f9f51603740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564e09003740 con 0x564e090007d0 2024-03-19T02:57:05.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.011+0000 7f9f47fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e08ff72d0 0x564e08ffd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.011+0000 7f9f51603740 10 client.0 _ll_register_callbacks cb 0x564e08f0a2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:57:05.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.011+0000 7f9f47fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e08ff72d0 0x564e08ffd820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59544/0 (socket says 172.21.15.33:59544) 2024-03-19T02:57:05.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.011+0000 7f9f47fff640 1 -- 172.21.15.33:0/1382190792 learned_addr learned my addr 172.21.15.33:0/1382190792 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:05.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.011+0000 7f9f477fe640 1 --2- 172.21.15.33:0/1382190792 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564e08ff7bf0 0x564e08ffdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.011+0000 7f9f4cfaf640 1 --2- 172.21.15.33:0/1382190792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564e090007d0 0x564e08ffe2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.011+0000 7f9f47fff640 1 -- 172.21.15.33:0/1382190792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564e090007d0 msgr2=0x564e08ffe2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:05.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.011+0000 7f9f47fff640 1 --2- 172.21.15.33:0/1382190792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564e090007d0 0x564e08ffe2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.011+0000 7f9f47fff640 1 -- 172.21.15.33:0/1382190792 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564e08ff7bf0 msgr2=0x564e08ffdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:05.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.011+0000 7f9f47fff640 1 --2- 172.21.15.33:0/1382190792 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564e08ff7bf0 0x564e08ffdd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.012+0000 7f9f47fff640 1 -- 172.21.15.33:0/1382190792 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e08c6e3f0 con 0x564e08ff72d0 2024-03-19T02:57:05.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.012+0000 7f9f47fff640 1 --2- 172.21.15.33:0/1382190792 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e08ff72d0 0x564e08ffd820 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f9f3000e9c0 tx=0x7f9f3009b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe96f239d5998a0c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:05.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.012+0000 7f9f277fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:57:05.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.012+0000 7f9f277fe640 1 -- 172.21.15.33:0/1382190792 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f300a3f50 con 0x564e08ff72d0 2024-03-19T02:57:05.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.012+0000 7f9f277fe640 1 -- 172.21.15.33:0/1382190792 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f9f300cf130 con 0x564e08ff72d0 2024-03-19T02:57:05.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.012+0000 7f9f51603740 1 -- 172.21.15.33:0/1382190792 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f38019ae0 con 0x564e08ff72d0 2024-03-19T02:57:05.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.012+0000 7f9f277fe640 1 -- 172.21.15.33:0/1382190792 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9f300ccb20 con 0x564e08ff72d0 2024-03-19T02:57:05.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.012+0000 7f9f51603740 10 client.16434 fetch_fsmap learned FSMap version 25 2024-03-19T02:57:05.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.012+0000 7f9f51603740 1 -- 172.21.15.33:0/1382190792 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564e08d387d0 con 0x564e08ff72d0 2024-03-19T02:57:05.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.012+0000 7f9f277fe640 1 -- 172.21.15.33:0/1382190792 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9f38019ae0 con 0x564e08ff72d0 2024-03-19T02:57:05.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.012+0000 7f9f277fe640 1 -- 172.21.15.33:0/1382190792 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f9f3009cae0 con 0x564e08ff72d0 2024-03-19T02:57:05.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.012+0000 7f9f51603740 10 client.16434 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:57:05.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.012+0000 7f9f51603740 10 client.16434 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T02:57:05.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.012+0000 7f9f51603740 20 client.16434 populate_metadata read hostname 'smithi033' 2024-03-19T02:57:05.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.012+0000 7f9f51603740 1 --2- 172.21.15.33:0/1382190792 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x564e09010b10 0x564e09012fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.012+0000 7f9f51603740 4 client.16434 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:57:05.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.012+0000 7f9f51603740 1 -- 172.21.15.33:0/1382190792 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e08fb83e0 con 0x564e09010b10 2024-03-19T02:57:05.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.013+0000 7f9f477fe640 1 --2- 172.21.15.33:0/1382190792 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x564e09010b10 0x564e09012fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.013+0000 7f9f477fe640 1 --2- 172.21.15.33:0/1382190792 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x564e09010b10 0x564e09012fa0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e18748b1eb519147 server_cookie=9d53202a837f78b4 in_seq=0 out_seq=0 2024-03-19T02:57:05.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.013+0000 7f9f277fe640 10 client.16434 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T02:57:05.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.014+0000 7f9f277fe640 1 -- 172.21.15.33:0/1382190792 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564e08fb83e0 con 0x564e09010b10 2024-03-19T02:57:05.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.014+0000 7f9f277fe640 10 client.16434 handle_command_reply: tid=0 2024-03-19T02:57:05.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.014+0000 7f9f51603740 1 client.16434 shutdown 2024-03-19T02:57:05.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.014+0000 7f9f51603740 1 -- 172.21.15.33:0/1382190792 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e08ff72d0 msgr2=0x564e08ffd820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:05.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.014+0000 7f9f51603740 1 --2- 172.21.15.33:0/1382190792 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e08ff72d0 0x564e08ffd820 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f9f3000e9c0 tx=0x7f9f3009b860 comp rx=0 tx=0).stop 2024-03-19T02:57:05.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.014+0000 7f9f51603740 1 -- 172.21.15.33:0/1382190792 shutdown_connections 2024-03-19T02:57:05.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.014+0000 7f9f51603740 1 --2- 172.21.15.33:0/1382190792 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x564e09010b10 0x564e09012fa0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.014+0000 7f9f51603740 1 --2- 172.21.15.33:0/1382190792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564e090007d0 0x564e08ffe2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.014+0000 7f9f51603740 1 --2- 172.21.15.33:0/1382190792 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564e08ff7bf0 0x564e08ffdd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.014+0000 7f9f51603740 1 --2- 172.21.15.33:0/1382190792 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e08ff72d0 0x564e08ffd820 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.014+0000 7f9f277fe640 0 client.16434 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T02:57:05.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.014+0000 7f9f51603740 1 -- 172.21.15.33:0/1382190792 >> 172.21.15.33:0/1382190792 conn(0x564e08ff1a00 msgr2=0x564e08ff3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:05.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.014+0000 7f9f51603740 1 -- 172.21.15.33:0/1382190792 shutdown_connections 2024-03-19T02:57:05.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.014+0000 7f9f51603740 1 -- 172.21.15.33:0/1382190792 wait complete. 2024-03-19T02:57:05.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.014+0000 7f9f51603740 20 client.16434 trim_cache size 0 max 16384 2024-03-19T02:57:05.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.027+0000 7f9f51603740 1 Processor -- start 2024-03-19T02:57:05.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.027+0000 7f9f51603740 1 -- start start 2024-03-19T02:57:05.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.027+0000 7f9f51603740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564e08ff72d0 0x564e08ffff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.027+0000 7f9f51603740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e08ff7bf0 0x564e08ff3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.027+0000 7f9f51603740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564e090007d0 0x564e09018870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.028+0000 7f9f51603740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9f481b13a0 con 0x564e08ff7bf0 2024-03-19T02:57:05.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.028+0000 7f9f51603740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9f481aec70 con 0x564e090007d0 2024-03-19T02:57:05.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.028+0000 7f9f51603740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9f481acaa0 con 0x564e08ff72d0 2024-03-19T02:57:05.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.028+0000 7f9f477fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e08ff7bf0 0x564e08ff3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.028+0000 7f9f477fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e08ff7bf0 0x564e08ff3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59560/0 (socket says 172.21.15.33:59560) 2024-03-19T02:57:05.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.028+0000 7f9f477fe640 1 -- 172.21.15.33:0/2556657078 learned_addr learned my addr 172.21.15.33:0/2556657078 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:05.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.028+0000 7f9f47fff640 1 --2- 172.21.15.33:0/2556657078 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564e08ff72d0 0x564e08ffff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.028+0000 7f9f4cfaf640 1 --2- 172.21.15.33:0/2556657078 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564e090007d0 0x564e09018870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.028+0000 7f9f477fe640 1 -- 172.21.15.33:0/2556657078 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564e08ff72d0 msgr2=0x564e08ffff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:05.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.028+0000 7f9f477fe640 1 --2- 172.21.15.33:0/2556657078 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564e08ff72d0 0x564e08ffff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.028+0000 7f9f477fe640 1 -- 172.21.15.33:0/2556657078 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564e090007d0 msgr2=0x564e09018870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:05.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.028+0000 7f9f477fe640 1 --2- 172.21.15.33:0/2556657078 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564e090007d0 0x564e09018870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.028+0000 7f9f477fe640 1 -- 172.21.15.33:0/2556657078 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e08fc7880 con 0x564e08ff7bf0 2024-03-19T02:57:05.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.028+0000 7f9f477fe640 1 --2- 172.21.15.33:0/2556657078 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e08ff7bf0 0x564e08ff3540 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f9f38009c90 tx=0x7f9f38009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6496070e58fbe377 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:05.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.028+0000 7f9f3dffb640 1 -- 172.21.15.33:0/2556657078 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f3801e040 con 0x564e08ff7bf0 2024-03-19T02:57:05.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.028+0000 7f9f3dffb640 1 -- 172.21.15.33:0/2556657078 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9f3801f040 con 0x564e08ff7bf0 2024-03-19T02:57:05.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.028+0000 7f9f3dffb640 1 -- 172.21.15.33:0/2556657078 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f380052a0 con 0x564e08ff7bf0 2024-03-19T02:57:05.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.029+0000 7f9f51603740 1 -- 172.21.15.33:0/2556657078 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e08ff7bf0 msgr2=0x564e08ff3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:05.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.029+0000 7f9f51603740 1 --2- 172.21.15.33:0/2556657078 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e08ff7bf0 0x564e08ff3540 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f9f38009c90 tx=0x7f9f38009c00 comp rx=0 tx=0).stop 2024-03-19T02:57:05.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.029+0000 7f9f51603740 1 -- 172.21.15.33:0/2556657078 shutdown_connections 2024-03-19T02:57:05.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.029+0000 7f9f51603740 1 --2- 172.21.15.33:0/2556657078 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564e090007d0 0x564e09018870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.029+0000 7f9f51603740 1 --2- 172.21.15.33:0/2556657078 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e08ff7bf0 0x564e08ff3540 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.029+0000 7f9f51603740 1 --2- 172.21.15.33:0/2556657078 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564e08ff72d0 0x564e08ffff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.029+0000 7f9f51603740 1 -- 172.21.15.33:0/2556657078 >> 172.21.15.33:0/2556657078 conn(0x564e08ff1a00 msgr2=0x564e08ff2e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:05.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.029+0000 7f9f51603740 1 -- 172.21.15.33:0/2556657078 shutdown_connections 2024-03-19T02:57:05.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.029+0000 7f9f51603740 1 -- 172.21.15.33:0/2556657078 wait complete. 2024-03-19T02:57:05.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.029+0000 7f9f51603740 1 -- start start 2024-03-19T02:57:05.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.029+0000 7f9f51603740 1 Processor -- start 2024-03-19T02:57:05.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.030+0000 7f9f51603740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564e08ff72d0 0x564e09008d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.030+0000 7f9f51603740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564e08ff7bf0 0x564e090721c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.030+0000 7f9f51603740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e090007d0 0x564e09072720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.030+0000 7f9f51603740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9f481ad130 con 0x564e090007d0 2024-03-19T02:57:05.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.030+0000 7f9f51603740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9f481ae5a0 con 0x564e08ff72d0 2024-03-19T02:57:05.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.030+0000 7f9f51603740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9f481ad800 con 0x564e08ff7bf0 2024-03-19T02:57:05.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.030+0000 7f9f51603740 10 client.0 _ll_register_callbacks cb 0x7f9f3009b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:57:05.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.030+0000 7f9f4cfaf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e090007d0 0x564e09072720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.030+0000 7f9f4cfaf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e090007d0 0x564e09072720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59574/0 (socket says 172.21.15.33:59574) 2024-03-19T02:57:05.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.030+0000 7f9f4cfaf640 1 -- 172.21.15.33:0/3978736703 learned_addr learned my addr 172.21.15.33:0/3978736703 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:05.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.030+0000 7f9f47fff640 1 --2- 172.21.15.33:0/3978736703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564e08ff72d0 0x564e09008d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.030+0000 7f9f477fe640 1 --2- 172.21.15.33:0/3978736703 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564e08ff7bf0 0x564e090721c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.030+0000 7f9f4cfaf640 1 -- 172.21.15.33:0/3978736703 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564e08ff7bf0 msgr2=0x564e090721c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:05.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.030+0000 7f9f4cfaf640 1 --2- 172.21.15.33:0/3978736703 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564e08ff7bf0 0x564e090721c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.030+0000 7f9f4cfaf640 1 -- 172.21.15.33:0/3978736703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564e08ff72d0 msgr2=0x564e09008d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:05.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.030+0000 7f9f4cfaf640 1 --2- 172.21.15.33:0/3978736703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564e08ff72d0 0x564e09008d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.030+0000 7f9f4cfaf640 1 -- 172.21.15.33:0/3978736703 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e09072f50 con 0x564e090007d0 2024-03-19T02:57:05.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.030+0000 7f9f4cfaf640 1 --2- 172.21.15.33:0/3978736703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e090007d0 0x564e09072720 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f9f40037040 tx=0x7f9f40005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6d3f95c31e2e0e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:05.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.030+0000 7f9f277fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:57:05.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.030+0000 7f9f277fe640 1 -- 172.21.15.33:0/3978736703 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f40052070 con 0x564e090007d0 2024-03-19T02:57:05.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.030+0000 7f9f277fe640 1 -- 172.21.15.33:0/3978736703 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f9f40004450 con 0x564e090007d0 2024-03-19T02:57:05.037 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.030+0000 7f9f51603740 1 -- 172.21.15.33:0/3978736703 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f3801f040 con 0x564e090007d0 2024-03-19T02:57:05.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.030+0000 7f9f277fe640 1 -- 172.21.15.33:0/3978736703 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9f4005e050 con 0x564e090007d0 2024-03-19T02:57:05.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.030+0000 7f9f51603740 10 client.16440 fetch_fsmap learned FSMap version 25 2024-03-19T02:57:05.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.031+0000 7f9f51603740 1 -- 172.21.15.33:0/3978736703 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564e0907fb60 con 0x564e090007d0 2024-03-19T02:57:05.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.031+0000 7f9f277fe640 1 -- 172.21.15.33:0/3978736703 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9f3801f040 con 0x564e090007d0 2024-03-19T02:57:05.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.031+0000 7f9f277fe640 1 -- 172.21.15.33:0/3978736703 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f9f4003f050 con 0x564e090007d0 2024-03-19T02:57:05.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.031+0000 7f9f51603740 10 client.16440 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:57:05.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.031+0000 7f9f51603740 10 client.16440 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T02:57:05.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.031+0000 7f9f51603740 20 client.16440 populate_metadata read hostname 'smithi033' 2024-03-19T02:57:05.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.031+0000 7f9f51603740 1 --2- 172.21.15.33:0/3978736703 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x564e09080310 0x564e09082730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.031+0000 7f9f51603740 4 client.16440 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:57:05.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.031+0000 7f9f51603740 1 -- 172.21.15.33:0/3978736703 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564e08f679f0 con 0x564e09080310 2024-03-19T02:57:05.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.031+0000 7f9f47fff640 1 --2- 172.21.15.33:0/3978736703 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x564e09080310 0x564e09082730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.031+0000 7f9f47fff640 1 --2- 172.21.15.33:0/3978736703 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x564e09080310 0x564e09082730 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b73490e25b589216 server_cookie=2cd7486aa04a1f94 in_seq=0 out_seq=0 2024-03-19T02:57:05.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.031+0000 7f9f277fe640 10 client.16440 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T02:57:05.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.031+0000 7f9f277fe640 1 -- 172.21.15.33:0/3978736703 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564e08f679f0 con 0x564e09080310 2024-03-19T02:57:05.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.031+0000 7f9f277fe640 10 client.16440 handle_command_reply: tid=0 2024-03-19T02:57:05.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.032+0000 7f9f51603740 1 client.16440 shutdown 2024-03-19T02:57:05.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.032+0000 7f9f51603740 1 -- 172.21.15.33:0/3978736703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e090007d0 msgr2=0x564e09072720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:05.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.032+0000 7f9f51603740 1 --2- 172.21.15.33:0/3978736703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e090007d0 0x564e09072720 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f9f40037040 tx=0x7f9f40005c50 comp rx=0 tx=0).stop 2024-03-19T02:57:05.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.032+0000 7f9f51603740 1 -- 172.21.15.33:0/3978736703 shutdown_connections 2024-03-19T02:57:05.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.032+0000 7f9f51603740 1 --2- 172.21.15.33:0/3978736703 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x564e09080310 0x564e09082730 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.032+0000 7f9f51603740 1 --2- 172.21.15.33:0/3978736703 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564e08ff7bf0 0x564e090721c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.032+0000 7f9f51603740 1 --2- 172.21.15.33:0/3978736703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564e090007d0 0x564e09072720 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.032+0000 7f9f51603740 1 --2- 172.21.15.33:0/3978736703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564e08ff72d0 0x564e09008d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.032+0000 7f9f277fe640 0 client.16440 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T02:57:05.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.032+0000 7f9f51603740 1 -- 172.21.15.33:0/3978736703 >> 172.21.15.33:0/3978736703 conn(0x564e08ff1a00 msgr2=0x564e08ff36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:05.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.032+0000 7f9f51603740 1 -- 172.21.15.33:0/3978736703 shutdown_connections 2024-03-19T02:57:05.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.032+0000 7f9f51603740 1 -- 172.21.15.33:0/3978736703 wait complete. 2024-03-19T02:57:05.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.032+0000 7f9f51603740 20 client.16440 trim_cache size 0 max 16384 2024-03-19T02:57:05.040 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:57:05.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.032+0000 7f9f4ea12640 1 -- 172.21.15.33:0/373226518 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f9f340a2970 msgr2=0x7f9f340a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:05.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.032+0000 7f9f4ea12640 1 --2- 172.21.15.33:0/373226518 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f9f340a2970 0x7f9f340a4e30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9f40004af0 tx=0x7f9f40009960 comp rx=0 tx=0).stop 2024-03-19T02:57:05.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.032+0000 7f9f4ea12640 1 -- 172.21.15.33:0/373226518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f48069e50 msgr2=0x7f9f48197ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:05.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.032+0000 7f9f4ea12640 1 --2- 172.21.15.33:0/373226518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f48069e50 0x7f9f48197ea0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f9f3000e9f0 tx=0x7f9f3000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.032+0000 7f9f4ea12640 1 -- 172.21.15.33:0/373226518 shutdown_connections 2024-03-19T02:57:05.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.032+0000 7f9f4ea12640 1 --2- 172.21.15.33:0/373226518 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f9f340a2970 0x7f9f340a4e30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.032+0000 7f9f4ea12640 1 --2- 172.21.15.33:0/373226518 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9f4806a7f0 0x7f9f481983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.032+0000 7f9f4ea12640 1 --2- 172.21.15.33:0/373226518 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9f48107890 0x7f9f4819c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.032+0000 7f9f4ea12640 1 --2- 172.21.15.33:0/373226518 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9f48069e50 0x7f9f48197ea0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.032+0000 7f9f4ea12640 1 -- 172.21.15.33:0/373226518 >> 172.21.15.33:0/373226518 conn(0x7f9f480774c0 msgr2=0x7f9f480ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:05.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.033+0000 7f9f4ea12640 1 -- 172.21.15.33:0/373226518 shutdown_connections 2024-03-19T02:57:05.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.033+0000 7f9f4ea12640 1 -- 172.21.15.33:0/373226518 wait complete. 2024-03-19T02:57:05.044 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:57:05.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.148+0000 7fc0e6ba9640 1 -- 172.21.15.33:0/3954391719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc0e01006a0 msgr2=0x7fc0e0100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:05.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.148+0000 7fc0e6ba9640 1 --2- 172.21.15.33:0/3954391719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc0e01006a0 0x7fc0e0100b20 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0009cd0 tx=0x7fc0d002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.149+0000 7fc0e6ba9640 1 -- 172.21.15.33:0/3954391719 shutdown_connections 2024-03-19T02:57:05.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.149+0000 7fc0e6ba9640 1 --2- 172.21.15.33:0/3954391719 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0e0101060 0x7fc0e010d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.149+0000 7fc0e6ba9640 1 --2- 172.21.15.33:0/3954391719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc0e01006a0 0x7fc0e0100b20 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.149+0000 7fc0e6ba9640 1 --2- 172.21.15.33:0/3954391719 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0e01066b0 0x7fc0e0106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.149+0000 7fc0e6ba9640 1 -- 172.21.15.33:0/3954391719 >> 172.21.15.33:0/3954391719 conn(0x7fc0e00778d0 msgr2=0x7fc0e0077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:05.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.149+0000 7fc0e6ba9640 1 -- 172.21.15.33:0/3954391719 shutdown_connections 2024-03-19T02:57:05.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.149+0000 7fc0e6ba9640 1 -- 172.21.15.33:0/3954391719 wait complete. 2024-03-19T02:57:05.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.149+0000 7fc0e6ba9640 1 Processor -- start 2024-03-19T02:57:05.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.149+0000 7fc0e6ba9640 1 -- start start 2024-03-19T02:57:05.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.149+0000 7fc0e6ba9640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0e01006a0 0x7fc0e0196ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.149+0000 7fc0e6ba9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc0e0101060 0x7fc0e0197410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.149+0000 7fc0e6ba9640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0e01066b0 0x7fc0e019b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.149+0000 7fc0e6ba9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e0110970 con 0x7fc0e0101060 2024-03-19T02:57:05.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.149+0000 7fc0e6ba9640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e01107f0 con 0x7fc0e01066b0 2024-03-19T02:57:05.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.149+0000 7fc0e6ba9640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e0110af0 con 0x7fc0e01006a0 2024-03-19T02:57:05.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.150+0000 7fc0dffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc0e0101060 0x7fc0e0197410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.150+0000 7fc0dffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc0e0101060 0x7fc0e0197410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59582/0 (socket says 172.21.15.33:59582) 2024-03-19T02:57:05.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.150+0000 7fc0e5146640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0e01066b0 0x7fc0e019b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.150+0000 7fc0dffff640 1 -- 172.21.15.33:0/3454673605 learned_addr learned my addr 172.21.15.33:0/3454673605 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:05.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.150+0000 7fc0e4945640 1 --2- 172.21.15.33:0/3454673605 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0e01006a0 0x7fc0e0196ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.150+0000 7fc0dffff640 1 -- 172.21.15.33:0/3454673605 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0e01006a0 msgr2=0x7fc0e0196ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:05.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.150+0000 7fc0dffff640 1 --2- 172.21.15.33:0/3454673605 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0e01006a0 0x7fc0e0196ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.150+0000 7fc0dffff640 1 -- 172.21.15.33:0/3454673605 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0e01066b0 msgr2=0x7fc0e019b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:05.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.150+0000 7fc0dffff640 1 --2- 172.21.15.33:0/3454673605 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0e01066b0 0x7fc0e019b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.150+0000 7fc0dffff640 1 -- 172.21.15.33:0/3454673605 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e019bec0 con 0x7fc0e0101060 2024-03-19T02:57:05.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.150+0000 7fc0dffff640 1 --2- 172.21.15.33:0/3454673605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc0e0101060 0x7fc0e0197410 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fc0d002c700 tx=0x7fc0d0035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:05.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.150+0000 7fc0ddffb640 1 -- 172.21.15.33:0/3454673605 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d0039040 con 0x7fc0e0101060 2024-03-19T02:57:05.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.150+0000 7fc0ddffb640 1 -- 172.21.15.33:0/3454673605 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc0d0044070 con 0x7fc0e0101060 2024-03-19T02:57:05.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.150+0000 7fc0e6ba9640 1 -- 172.21.15.33:0/3454673605 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0e019c150 con 0x7fc0e0101060 2024-03-19T02:57:05.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.150+0000 7fc0ddffb640 1 -- 172.21.15.33:0/3454673605 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d00042a0 con 0x7fc0e0101060 2024-03-19T02:57:05.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.150+0000 7fc0e6ba9640 1 -- 172.21.15.33:0/3454673605 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0e0068980 con 0x7fc0e0101060 2024-03-19T02:57:05.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.152+0000 7fc0ddffb640 1 -- 172.21.15.33:0/3454673605 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fc0d0004440 con 0x7fc0e0101060 2024-03-19T02:57:05.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.153+0000 7fc0e978c740 1 Processor -- start 2024-03-19T02:57:05.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.153+0000 7fc0e978c740 1 -- start start 2024-03-19T02:57:05.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.153+0000 7fc0e978c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f050132d0 0x562f050136b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.153+0000 7fc0e978c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f05013bf0 0x562f0501c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.153+0000 7fc0e978c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f0501c7d0 0x562f0501ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.153+0000 7fc0e978c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562f04f764b0 con 0x562f0501c7d0 2024-03-19T02:57:05.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.153+0000 7fc0e978c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x562f04f1ed70 con 0x562f05013bf0 2024-03-19T02:57:05.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.153+0000 7fc0e978c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x562f04f72550 con 0x562f050132d0 2024-03-19T02:57:05.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.153+0000 7fc0ddffb640 1 --2- 172.21.15.33:0/3454673605 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fc0cc0a2970 0x7fc0cc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.153+0000 7fc0dffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f0501c7d0 0x562f0501ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.153+0000 7fc0dffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f0501c7d0 0x562f0501ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59588/0 (socket says 172.21.15.33:59588) 2024-03-19T02:57:05.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.153+0000 7fc0e4945640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f05013bf0 0x562f0501c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.153+0000 7fc0dffff640 1 -- 172.21.15.33:0/2750373391 learned_addr learned my addr 172.21.15.33:0/2750373391 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:05.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.153+0000 7fc0e5146640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f050132d0 0x562f050136b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.153+0000 7fc0e5146640 1 --2- 172.21.15.33:0/3454673605 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fc0cc0a2970 0x7fc0cc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.153+0000 7fc0ddffb640 1 -- 172.21.15.33:0/3454673605 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fc0d00f14a0 con 0x7fc0e0101060 2024-03-19T02:57:05.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.153+0000 7fc0e4945640 1 -- 172.21.15.33:0/2750373391 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f050132d0 msgr2=0x562f050136b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:05.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.153+0000 7fc0e4945640 1 --2- 172.21.15.33:0/2750373391 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f050132d0 0x562f050136b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.153+0000 7fc0e4945640 1 -- 172.21.15.33:0/2750373391 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f0501c7d0 msgr2=0x562f0501ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:05.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.153+0000 7fc0e4945640 1 --2- 172.21.15.33:0/2750373391 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f0501c7d0 0x562f0501ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.154+0000 7fc0e4945640 1 -- 172.21.15.33:0/2750373391 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f04f41210 con 0x562f05013bf0 2024-03-19T02:57:05.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.154+0000 7fc0e5146640 1 --2- 172.21.15.33:0/3454673605 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fc0cc0a2970 0x7fc0cc0a4e30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc0c8011940 tx=0x7fc0c8005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:05.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.154+0000 7fc0e4945640 1 --2- 172.21.15.33:0/2750373391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f05013bf0 0x562f0501c290 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc0d8009c20 tx=0x7fc0d8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41246c3bf7eb051d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:05.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.154+0000 7fc0d6ffd640 1 -- 172.21.15.33:0/2750373391 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d8004050 con 0x562f05013bf0 2024-03-19T02:57:05.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.154+0000 7fc0d6ffd640 1 -- 172.21.15.33:0/2750373391 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc0d80054d0 con 0x562f05013bf0 2024-03-19T02:57:05.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.154+0000 7fc0d6ffd640 1 -- 172.21.15.33:0/2750373391 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d8004c30 con 0x562f05013bf0 2024-03-19T02:57:05.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.155+0000 7fc0e978c740 1 -- 172.21.15.33:0/2750373391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f05013bf0 msgr2=0x562f0501c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:05.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.155+0000 7fc0e978c740 1 --2- 172.21.15.33:0/2750373391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f05013bf0 0x562f0501c290 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc0d8009c20 tx=0x7fc0d8011810 comp rx=0 tx=0).stop 2024-03-19T02:57:05.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.155+0000 7fc0e978c740 1 -- 172.21.15.33:0/2750373391 shutdown_connections 2024-03-19T02:57:05.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.155+0000 7fc0e978c740 1 --2- 172.21.15.33:0/2750373391 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f0501c7d0 0x562f0501ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.155+0000 7fc0e978c740 1 --2- 172.21.15.33:0/2750373391 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f05013bf0 0x562f0501c290 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.155+0000 7fc0e978c740 1 --2- 172.21.15.33:0/2750373391 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f050132d0 0x562f050136b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.155+0000 7fc0e978c740 1 -- 172.21.15.33:0/2750373391 >> 172.21.15.33:0/2750373391 conn(0x562f0500da00 msgr2=0x562f0500dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:05.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.155+0000 7fc0e978c740 1 -- 172.21.15.33:0/2750373391 shutdown_connections 2024-03-19T02:57:05.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.155+0000 7fc0e978c740 1 -- 172.21.15.33:0/2750373391 wait complete. 2024-03-19T02:57:05.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.155+0000 7fc0e978c740 1 -- start start 2024-03-19T02:57:05.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.155+0000 7fc0e978c740 1 Processor -- start 2024-03-19T02:57:05.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.155+0000 7fc0e978c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f050132d0 0x562f05019790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.155+0000 7fc0e978c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f05013bf0 0x562f05019cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.155+0000 7fc0e978c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f0501c7d0 0x562f0501a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.155+0000 7fc0e978c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x562f0501f5c0 con 0x562f05013bf0 2024-03-19T02:57:05.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.155+0000 7fc0e978c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x562f04f32310 con 0x562f050132d0 2024-03-19T02:57:05.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.155+0000 7fc0e978c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x562f0501f740 con 0x562f0501c7d0 2024-03-19T02:57:05.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.156+0000 7fc0dffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f05013bf0 0x562f05019cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.155+0000 7fc0e978c740 10 client.0 _ll_register_callbacks cb 0x562f04f262d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:57:05.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.156+0000 7fc0dffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f05013bf0 0x562f05019cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59594/0 (socket says 172.21.15.33:59594) 2024-03-19T02:57:05.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.156+0000 7fc0e4945640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f050132d0 0x562f05019790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.156+0000 7fc0dffff640 1 -- 172.21.15.33:0/1105031337 learned_addr learned my addr 172.21.15.33:0/1105031337 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:05.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.156+0000 7fc0e5146640 1 --2- 172.21.15.33:0/1105031337 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f0501c7d0 0x562f0501a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.156+0000 7fc0dffff640 1 -- 172.21.15.33:0/1105031337 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f0501c7d0 msgr2=0x562f0501a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:05.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.156+0000 7fc0dffff640 1 --2- 172.21.15.33:0/1105031337 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f0501c7d0 0x562f0501a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.156+0000 7fc0dffff640 1 -- 172.21.15.33:0/1105031337 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f050132d0 msgr2=0x562f05019790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:05.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.156+0000 7fc0dffff640 1 --2- 172.21.15.33:0/1105031337 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f050132d0 0x562f05019790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.156+0000 7fc0dffff640 1 -- 172.21.15.33:0/1105031337 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f04c8a3f0 con 0x562f05013bf0 2024-03-19T02:57:05.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.156+0000 7fc0dffff640 1 --2- 172.21.15.33:0/1105031337 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f05013bf0 0x562f05019cd0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fc0d002c6d0 tx=0x7fc0d00eedd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b82fb338268281f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:05.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.156+0000 7fc0bffff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:57:05.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.156+0000 7fc0bffff640 1 -- 172.21.15.33:0/1105031337 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d00bd1c0 con 0x562f05013bf0 2024-03-19T02:57:05.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.156+0000 7fc0bffff640 1 -- 172.21.15.33:0/1105031337 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fc0d00f1150 con 0x562f05013bf0 2024-03-19T02:57:05.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.156+0000 7fc0e978c740 1 -- 172.21.15.33:0/1105031337 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0d80054d0 con 0x562f05013bf0 2024-03-19T02:57:05.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.156+0000 7fc0bffff640 1 -- 172.21.15.33:0/1105031337 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc0d0034700 con 0x562f05013bf0 2024-03-19T02:57:05.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.156+0000 7fc0e978c740 10 client.16452 fetch_fsmap learned FSMap version 25 2024-03-19T02:57:05.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.156+0000 7fc0e978c740 1 -- 172.21.15.33:0/1105031337 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562f04d547d0 con 0x562f05013bf0 2024-03-19T02:57:05.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.156+0000 7fc0bffff640 1 -- 172.21.15.33:0/1105031337 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc0d80054d0 con 0x562f05013bf0 2024-03-19T02:57:05.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.157+0000 7fc0bffff640 1 -- 172.21.15.33:0/1105031337 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc0d003c030 con 0x562f05013bf0 2024-03-19T02:57:05.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.157+0000 7fc0e978c740 10 client.16452 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:57:05.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.157+0000 7fc0e978c740 10 client.16452 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:57:05.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.157+0000 7fc0e978c740 20 client.16452 populate_metadata read hostname 'smithi033' 2024-03-19T02:57:05.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.157+0000 7fc0e978c740 1 --2- 172.21.15.33:0/1105031337 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x562f0502ca20 0x562f0502eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.157+0000 7fc0e978c740 4 client.16452 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:57:05.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.157+0000 7fc0e978c740 1 -- 172.21.15.33:0/1105031337 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562f04fd43e0 con 0x562f0502ca20 2024-03-19T02:57:05.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.157+0000 7fc0e4945640 1 --2- 172.21.15.33:0/1105031337 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x562f0502ca20 0x562f0502eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.157+0000 7fc0bffff640 10 client.16452 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:57:05.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.157+0000 7fc0e4945640 1 --2- 172.21.15.33:0/1105031337 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x562f0502ca20 0x562f0502eea0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a5563c30ebd6518 server_cookie=6e1dbb670b7f99f8 in_seq=0 out_seq=0 2024-03-19T02:57:05.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.158+0000 7fc0bffff640 1 -- 172.21.15.33:0/1105031337 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562f04fd43e0 con 0x562f0502ca20 2024-03-19T02:57:05.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.158+0000 7fc0bffff640 10 client.16452 handle_command_reply: tid=0 2024-03-19T02:57:05.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.158+0000 7fc0e978c740 1 client.16452 shutdown 2024-03-19T02:57:05.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.158+0000 7fc0e978c740 1 -- 172.21.15.33:0/1105031337 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f05013bf0 msgr2=0x562f05019cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:05.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.158+0000 7fc0e978c740 1 --2- 172.21.15.33:0/1105031337 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f05013bf0 0x562f05019cd0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fc0d002c6d0 tx=0x7fc0d00eedd0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.158+0000 7fc0e978c740 1 -- 172.21.15.33:0/1105031337 shutdown_connections 2024-03-19T02:57:05.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.158+0000 7fc0e978c740 1 --2- 172.21.15.33:0/1105031337 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x562f0502ca20 0x562f0502eea0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.158+0000 7fc0e978c740 1 --2- 172.21.15.33:0/1105031337 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f0501c7d0 0x562f0501a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.158+0000 7fc0e978c740 1 --2- 172.21.15.33:0/1105031337 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f05013bf0 0x562f05019cd0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.158+0000 7fc0bffff640 0 client.16452 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:57:05.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.158+0000 7fc0e978c740 1 --2- 172.21.15.33:0/1105031337 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f050132d0 0x562f05019790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.158+0000 7fc0e978c740 1 -- 172.21.15.33:0/1105031337 >> 172.21.15.33:0/1105031337 conn(0x562f0500da00 msgr2=0x562f0500f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:05.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.159+0000 7fc0e978c740 1 -- 172.21.15.33:0/1105031337 shutdown_connections 2024-03-19T02:57:05.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.159+0000 7fc0e978c740 1 -- 172.21.15.33:0/1105031337 wait complete. 2024-03-19T02:57:05.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.159+0000 7fc0e978c740 20 client.16452 trim_cache size 0 max 16384 2024-03-19T02:57:05.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.172+0000 7fc0e978c740 1 Processor -- start 2024-03-19T02:57:05.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.172+0000 7fc0e978c740 1 -- start start 2024-03-19T02:57:05.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.172+0000 7fc0e978c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f050132d0 0x562f05028360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.172+0000 7fc0e978c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f05013bf0 0x562f05019e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.172+0000 7fc0e978c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f0501c7d0 0x562f050347b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.172+0000 7fc0e978c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e01b1480 con 0x562f0501c7d0 2024-03-19T02:57:05.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.172+0000 7fc0e978c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e01aed50 con 0x562f050132d0 2024-03-19T02:57:05.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.172+0000 7fc0e978c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e01acb80 con 0x562f05013bf0 2024-03-19T02:57:05.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.172+0000 7fc0e5146640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f0501c7d0 0x562f050347b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.172+0000 7fc0e4945640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f050132d0 0x562f05028360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.172+0000 7fc0e5146640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f0501c7d0 0x562f050347b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59596/0 (socket says 172.21.15.33:59596) 2024-03-19T02:57:05.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.172+0000 7fc0e5146640 1 -- 172.21.15.33:0/4071331384 learned_addr learned my addr 172.21.15.33:0/4071331384 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:05.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.172+0000 7fc0dffff640 1 --2- 172.21.15.33:0/4071331384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f05013bf0 0x562f05019e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.173+0000 7fc0e5146640 1 -- 172.21.15.33:0/4071331384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f05013bf0 msgr2=0x562f05019e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:05.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.173+0000 7fc0e5146640 1 --2- 172.21.15.33:0/4071331384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f05013bf0 0x562f05019e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.173+0000 7fc0e5146640 1 -- 172.21.15.33:0/4071331384 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f050132d0 msgr2=0x562f05028360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:05.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.173+0000 7fc0e5146640 1 --2- 172.21.15.33:0/4071331384 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f050132d0 0x562f05028360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.173+0000 7fc0e5146640 1 -- 172.21.15.33:0/4071331384 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f0501ebc0 con 0x562f0501c7d0 2024-03-19T02:57:05.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.173+0000 7fc0e4945640 1 --2- 172.21.15.33:0/4071331384 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f050132d0 0x562f05028360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:05.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.173+0000 7fc0e5146640 1 --2- 172.21.15.33:0/4071331384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f0501c7d0 0x562f050347b0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc0c8011fc0 tx=0x7fc0c800e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0d1a18fc247d421 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:05.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.173+0000 7fc0d67fc640 1 -- 172.21.15.33:0/4071331384 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c80044e0 con 0x562f0501c7d0 2024-03-19T02:57:05.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.173+0000 7fc0d67fc640 1 -- 172.21.15.33:0/4071331384 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc0c8003aa0 con 0x562f0501c7d0 2024-03-19T02:57:05.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.173+0000 7fc0d67fc640 1 -- 172.21.15.33:0/4071331384 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c802cbf0 con 0x562f0501c7d0 2024-03-19T02:57:05.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.173+0000 7fc0e978c740 1 -- 172.21.15.33:0/4071331384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f0501c7d0 msgr2=0x562f050347b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:05.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.173+0000 7fc0e978c740 1 --2- 172.21.15.33:0/4071331384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f0501c7d0 0x562f050347b0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc0c8011fc0 tx=0x7fc0c800e000 comp rx=0 tx=0).stop 2024-03-19T02:57:05.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.173+0000 7fc0e978c740 1 -- 172.21.15.33:0/4071331384 shutdown_connections 2024-03-19T02:57:05.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.173+0000 7fc0e978c740 1 --2- 172.21.15.33:0/4071331384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f05013bf0 0x562f05019e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.173+0000 7fc0e978c740 1 --2- 172.21.15.33:0/4071331384 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f0501c7d0 0x562f050347b0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.173+0000 7fc0e978c740 1 --2- 172.21.15.33:0/4071331384 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f050132d0 0x562f05028360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.173+0000 7fc0e978c740 1 -- 172.21.15.33:0/4071331384 >> 172.21.15.33:0/4071331384 conn(0x562f0500da00 msgr2=0x562f05019750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:05.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.173+0000 7fc0e978c740 1 -- 172.21.15.33:0/4071331384 shutdown_connections 2024-03-19T02:57:05.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.174+0000 7fc0e978c740 1 -- 172.21.15.33:0/4071331384 wait complete. 2024-03-19T02:57:05.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.174+0000 7fc0e978c740 1 -- start start 2024-03-19T02:57:05.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.174+0000 7fc0e978c740 1 Processor -- start 2024-03-19T02:57:05.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.174+0000 7fc0e978c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f050132d0 0x562f05024b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.174+0000 7fc0e978c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f05013bf0 0x562f05021960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.174+0000 7fc0e978c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f0501c7d0 0x562f05021ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.174+0000 7fc0e978c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e01ad210 con 0x562f050132d0 2024-03-19T02:57:05.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.174+0000 7fc0e978c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e01ae680 con 0x562f0501c7d0 2024-03-19T02:57:05.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.174+0000 7fc0e978c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e01ad8e0 con 0x562f05013bf0 2024-03-19T02:57:05.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.174+0000 7fc0e978c740 10 client.0 _ll_register_callbacks cb 0x7fc0d0009c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:57:05.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.174+0000 7fc0e4945640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f050132d0 0x562f05024b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.174+0000 7fc0e4945640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f050132d0 0x562f05024b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59610/0 (socket says 172.21.15.33:59610) 2024-03-19T02:57:05.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.174+0000 7fc0e4945640 1 -- 172.21.15.33:0/2368259319 learned_addr learned my addr 172.21.15.33:0/2368259319 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:05.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.174+0000 7fc0e5146640 1 --2- 172.21.15.33:0/2368259319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f0501c7d0 0x562f05021ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.174+0000 7fc0dffff640 1 --2- 172.21.15.33:0/2368259319 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f05013bf0 0x562f05021960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.175+0000 7fc0e4945640 1 -- 172.21.15.33:0/2368259319 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f05013bf0 msgr2=0x562f05021960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:05.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.175+0000 7fc0e4945640 1 --2- 172.21.15.33:0/2368259319 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f05013bf0 0x562f05021960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.175+0000 7fc0e4945640 1 -- 172.21.15.33:0/2368259319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f0501c7d0 msgr2=0x562f05021ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:05.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.175+0000 7fc0e4945640 1 --2- 172.21.15.33:0/2368259319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f0501c7d0 0x562f05021ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.175+0000 7fc0e4945640 1 -- 172.21.15.33:0/2368259319 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f050224e0 con 0x562f050132d0 2024-03-19T02:57:05.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.175+0000 7fc0e5146640 1 --2- 172.21.15.33:0/2368259319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f0501c7d0 0x562f05021ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:05.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.175+0000 7fc0e4945640 1 --2- 172.21.15.33:0/2368259319 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f050132d0 0x562f05024b80 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fc0d80099b0 tx=0x7fc0d8009d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2553c9f8aa9334eb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:05.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.175+0000 7fc0bffff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:57:05.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.175+0000 7fc0bffff640 1 -- 172.21.15.33:0/2368259319 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d8011d10 con 0x562f050132d0 2024-03-19T02:57:05.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.175+0000 7fc0bffff640 1 -- 172.21.15.33:0/2368259319 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fc0d802e5c0 con 0x562f050132d0 2024-03-19T02:57:05.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.175+0000 7fc0e978c740 1 -- 172.21.15.33:0/2368259319 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0c8003aa0 con 0x562f050132d0 2024-03-19T02:57:05.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.175+0000 7fc0bffff640 1 -- 172.21.15.33:0/2368259319 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc0d803b050 con 0x562f050132d0 2024-03-19T02:57:05.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.175+0000 7fc0e978c740 10 client.16458 fetch_fsmap learned FSMap version 25 2024-03-19T02:57:05.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.175+0000 7fc0e978c740 1 -- 172.21.15.33:0/2368259319 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562f05093c60 con 0x562f050132d0 2024-03-19T02:57:05.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.175+0000 7fc0bffff640 1 -- 172.21.15.33:0/2368259319 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc0c8003aa0 con 0x562f050132d0 2024-03-19T02:57:05.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.175+0000 7fc0bffff640 1 -- 172.21.15.33:0/2368259319 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc0d80035f0 con 0x562f050132d0 2024-03-19T02:57:05.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.175+0000 7fc0e978c740 10 client.16458 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:57:05.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.175+0000 7fc0e978c740 10 client.16458 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:57:05.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.175+0000 7fc0e978c740 20 client.16458 populate_metadata read hostname 'smithi033' 2024-03-19T02:57:05.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.175+0000 7fc0e978c740 1 --2- 172.21.15.33:0/2368259319 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x562f05094210 0x562f05096630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:05.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.175+0000 7fc0e978c740 4 client.16458 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:57:05.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.175+0000 7fc0e978c740 1 -- 172.21.15.33:0/2368259319 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562f04f839f0 con 0x562f05094210 2024-03-19T02:57:05.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.176+0000 7fc0dffff640 1 --2- 172.21.15.33:0/2368259319 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x562f05094210 0x562f05096630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:05.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.176+0000 7fc0dffff640 1 --2- 172.21.15.33:0/2368259319 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x562f05094210 0x562f05096630 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68ef7f4fe3627102 server_cookie=4beda6e10f18b485 in_seq=0 out_seq=0 2024-03-19T02:57:05.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.176+0000 7fc0bffff640 10 client.16458 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:57:05.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.176+0000 7fc0bffff640 1 -- 172.21.15.33:0/2368259319 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562f04f839f0 con 0x562f05094210 2024-03-19T02:57:05.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.176+0000 7fc0bffff640 10 client.16458 handle_command_reply: tid=0 2024-03-19T02:57:05.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.176+0000 7fc0e978c740 1 client.16458 shutdown 2024-03-19T02:57:05.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.176+0000 7fc0e978c740 1 -- 172.21.15.33:0/2368259319 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f050132d0 msgr2=0x562f05024b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:05.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.176+0000 7fc0e978c740 1 --2- 172.21.15.33:0/2368259319 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f050132d0 0x562f05024b80 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fc0d80099b0 tx=0x7fc0d8009d10 comp rx=0 tx=0).stop 2024-03-19T02:57:05.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.176+0000 7fc0e978c740 1 -- 172.21.15.33:0/2368259319 shutdown_connections 2024-03-19T02:57:05.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.176+0000 7fc0e978c740 1 --2- 172.21.15.33:0/2368259319 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x562f05094210 0x562f05096630 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.176+0000 7fc0e978c740 1 --2- 172.21.15.33:0/2368259319 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562f05013bf0 0x562f05021960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.176+0000 7fc0e978c740 1 --2- 172.21.15.33:0/2368259319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562f0501c7d0 0x562f05021ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.176+0000 7fc0bffff640 0 client.16458 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:57:05.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.176+0000 7fc0e978c740 1 --2- 172.21.15.33:0/2368259319 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x562f050132d0 0x562f05024b80 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.176+0000 7fc0e978c740 1 -- 172.21.15.33:0/2368259319 >> 172.21.15.33:0/2368259319 conn(0x562f0500da00 msgr2=0x562f05019fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:05.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.176+0000 7fc0e978c740 1 -- 172.21.15.33:0/2368259319 shutdown_connections 2024-03-19T02:57:05.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.176+0000 7fc0e978c740 1 -- 172.21.15.33:0/2368259319 wait complete. 2024-03-19T02:57:05.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.176+0000 7fc0e978c740 20 client.16458 trim_cache size 0 max 16384 2024-03-19T02:57:05.185 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:57:05.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.177+0000 7fc0e6ba9640 1 -- 172.21.15.33:0/3454673605 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fc0cc0a2970 msgr2=0x7fc0cc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:05.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.177+0000 7fc0e6ba9640 1 --2- 172.21.15.33:0/3454673605 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fc0cc0a2970 0x7fc0cc0a4e30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc0c8011940 tx=0x7fc0c8005f90 comp rx=0 tx=0).stop 2024-03-19T02:57:05.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.177+0000 7fc0e6ba9640 1 -- 172.21.15.33:0/3454673605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc0e0101060 msgr2=0x7fc0e0197410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:05.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.177+0000 7fc0e6ba9640 1 --2- 172.21.15.33:0/3454673605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc0e0101060 0x7fc0e0197410 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fc0d002c700 tx=0x7fc0d0035ce0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.177+0000 7fc0e6ba9640 1 -- 172.21.15.33:0/3454673605 shutdown_connections 2024-03-19T02:57:05.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.177+0000 7fc0e6ba9640 1 --2- 172.21.15.33:0/3454673605 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fc0cc0a2970 0x7fc0cc0a4e30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.177+0000 7fc0e6ba9640 1 --2- 172.21.15.33:0/3454673605 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc0e01066b0 0x7fc0e019b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.177+0000 7fc0e6ba9640 1 --2- 172.21.15.33:0/3454673605 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc0e0101060 0x7fc0e0197410 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.177+0000 7fc0e6ba9640 1 --2- 172.21.15.33:0/3454673605 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc0e01006a0 0x7fc0e0196ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:05.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.177+0000 7fc0e6ba9640 1 -- 172.21.15.33:0/3454673605 >> 172.21.15.33:0/3454673605 conn(0x7fc0e00778d0 msgr2=0x7fc0e0102ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:05.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.177+0000 7fc0e6ba9640 1 -- 172.21.15.33:0/3454673605 shutdown_connections 2024-03-19T02:57:05.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:05.177+0000 7fc0e6ba9640 1 -- 172.21.15.33:0/3454673605 wait complete. 2024-03-19T02:57:05.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:05 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3638378399' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:57:05.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:05 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3638378399' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:57:05.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:05 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3638378399' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:57:06.189 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:57:06.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:57:06.190 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:57:06.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.308+0000 7efce72a1640 1 -- 172.21.15.33:0/3964973663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efce01077c0 msgr2=0x7efce0107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:06.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.308+0000 7efce72a1640 1 --2- 172.21.15.33:0/3964973663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efce01077c0 0x7efce0107ba0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7efcd8009930 tx=0x7efcd802bc30 comp rx=0 tx=0).stop 2024-03-19T02:57:06.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.308+0000 7efce72a1640 1 -- 172.21.15.33:0/3964973663 shutdown_connections 2024-03-19T02:57:06.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.308+0000 7efce72a1640 1 --2- 172.21.15.33:0/3964973663 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efce0102170 0x7efce010e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.308+0000 7efce72a1640 1 --2- 172.21.15.33:0/3964973663 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efce01017b0 0x7efce0101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.308+0000 7efce72a1640 1 --2- 172.21.15.33:0/3964973663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efce01077c0 0x7efce0107ba0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.308+0000 7efce72a1640 1 -- 172.21.15.33:0/3964973663 >> 172.21.15.33:0/3964973663 conn(0x7efce0078c20 msgr2=0x7efce0079030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:06.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.308+0000 7efce72a1640 1 -- 172.21.15.33:0/3964973663 shutdown_connections 2024-03-19T02:57:06.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.308+0000 7efce72a1640 1 -- 172.21.15.33:0/3964973663 wait complete. 2024-03-19T02:57:06.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.309+0000 7efce72a1640 1 Processor -- start 2024-03-19T02:57:06.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.309+0000 7efce72a1640 1 -- start start 2024-03-19T02:57:06.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.309+0000 7efce72a1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efce01017b0 0x7efce0197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.309+0000 7efce72a1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efce0102170 0x7efce0198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.309+0000 7efce72a1640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efce01077c0 0x7efce019c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.309+0000 7efce72a1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efce0111a40 con 0x7efce01017b0 2024-03-19T02:57:06.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.309+0000 7efce72a1640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efce01118c0 con 0x7efce0102170 2024-03-19T02:57:06.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.309+0000 7efce72a1640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7efce0111bc0 con 0x7efce01077c0 2024-03-19T02:57:06.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.309+0000 7efce503d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efce01017b0 0x7efce0197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.309+0000 7efce503d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efce01017b0 0x7efce0197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59640/0 (socket says 172.21.15.33:59640) 2024-03-19T02:57:06.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.309+0000 7efce503d640 1 -- 172.21.15.33:0/3944280727 learned_addr learned my addr 172.21.15.33:0/3944280727 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:06.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.309+0000 7efce483c640 1 --2- 172.21.15.33:0/3944280727 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efce0102170 0x7efce0198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.309+0000 7efce583e640 1 --2- 172.21.15.33:0/3944280727 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efce01077c0 0x7efce019c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.309+0000 7efce503d640 1 -- 172.21.15.33:0/3944280727 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efce01077c0 msgr2=0x7efce019c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:06.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.309+0000 7efce503d640 1 --2- 172.21.15.33:0/3944280727 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efce01077c0 0x7efce019c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.309+0000 7efce503d640 1 -- 172.21.15.33:0/3944280727 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efce0102170 msgr2=0x7efce0198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:06.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.309+0000 7efce503d640 1 --2- 172.21.15.33:0/3944280727 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efce0102170 0x7efce0198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.310+0000 7efce503d640 1 -- 172.21.15.33:0/3944280727 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efce019cfb0 con 0x7efce01017b0 2024-03-19T02:57:06.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.310+0000 7efce503d640 1 --2- 172.21.15.33:0/3944280727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efce01017b0 0x7efce0197fc0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7efcd8009230 tx=0x7efcd802c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:06.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.310+0000 7efcde7fc640 1 -- 172.21.15.33:0/3944280727 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd8043020 con 0x7efce01017b0 2024-03-19T02:57:06.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.310+0000 7efce72a1640 1 -- 172.21.15.33:0/3944280727 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efce019d240 con 0x7efce01017b0 2024-03-19T02:57:06.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.310+0000 7efcde7fc640 1 -- 172.21.15.33:0/3944280727 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efcd802caf0 con 0x7efce01017b0 2024-03-19T02:57:06.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.310+0000 7efcde7fc640 1 -- 172.21.15.33:0/3944280727 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd8003ae0 con 0x7efce01017b0 2024-03-19T02:57:06.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.310+0000 7efce72a1640 1 -- 172.21.15.33:0/3944280727 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efce01a4b30 con 0x7efce01017b0 2024-03-19T02:57:06.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.312+0000 7efcde7fc640 1 -- 172.21.15.33:0/3944280727 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7efcd8046020 con 0x7efce01017b0 2024-03-19T02:57:06.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.313+0000 7efce9f6b740 1 Processor -- start 2024-03-19T02:57:06.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.313+0000 7efce9f6b740 1 -- start start 2024-03-19T02:57:06.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.313+0000 7efcde7fc640 1 --2- 172.21.15.33:0/3944280727 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7efcc80a2970 0x7efcc80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.313+0000 7efce9f6b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612645c3d60 0x5612645c4140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.313+0000 7efce9f6b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612645c4710 0x5612645d1750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.313+0000 7efce9f6b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612645d1c90 0x5612645d4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.313+0000 7efce9f6b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561264528160 con 0x5612645c3d60 2024-03-19T02:57:06.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.313+0000 7efce9f6b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5612644d0e10 con 0x5612645d1c90 2024-03-19T02:57:06.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.313+0000 7efce9f6b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x561264524200 con 0x5612645c4710 2024-03-19T02:57:06.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.313+0000 7efce503d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612645c3d60 0x5612645c4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.313+0000 7efce583e640 1 --2- 172.21.15.33:0/3944280727 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7efcc80a2970 0x7efcc80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.313+0000 7efce503d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612645c3d60 0x5612645c4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59654/0 (socket says 172.21.15.33:59654) 2024-03-19T02:57:06.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.313+0000 7efce503d640 1 -- 172.21.15.33:0/1595511936 learned_addr learned my addr 172.21.15.33:0/1595511936 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:06.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.313+0000 7efce483c640 1 --2- 172.21.15.33:0/1595511936 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612645c4710 0x5612645d1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.313+0000 7efce583e640 1 --2- 172.21.15.33:0/1595511936 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612645d1c90 0x5612645d4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.313+0000 7efcde7fc640 1 -- 172.21.15.33:0/3944280727 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7efcd80f1120 con 0x7efce01017b0 2024-03-19T02:57:06.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.314+0000 7efce483c640 1 -- 172.21.15.33:0/1595511936 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612645d1c90 msgr2=0x5612645d4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:06.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.314+0000 7efce483c640 1 --2- 172.21.15.33:0/1595511936 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612645d1c90 0x5612645d4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.314+0000 7efce483c640 1 -- 172.21.15.33:0/1595511936 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612645c3d60 msgr2=0x5612645c4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:06.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.314+0000 7efce483c640 1 --2- 172.21.15.33:0/1595511936 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612645c3d60 0x5612645c4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.314+0000 7efce483c640 1 -- 172.21.15.33:0/1595511936 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612643047e0 con 0x5612645c4710 2024-03-19T02:57:06.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.314+0000 7efce503d640 1 --2- 172.21.15.33:0/1595511936 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612645c3d60 0x5612645c4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:06.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.314+0000 7efce583e640 1 --2- 172.21.15.33:0/1595511936 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612645d1c90 0x5612645d4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:06.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.314+0000 7efce583e640 1 --2- 172.21.15.33:0/3944280727 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7efcc80a2970 0x7efcc80a4e30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7efcd0009990 tx=0x7efcd0009de0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:06.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.314+0000 7efce483c640 1 --2- 172.21.15.33:0/1595511936 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612645c4710 0x5612645d1750 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7efcd4009bf0 tx=0x7efcd4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8fcd6073a9f3425 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:06.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.314+0000 7efcc77fe640 1 -- 172.21.15.33:0/1595511936 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd4004050 con 0x5612645c4710 2024-03-19T02:57:06.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.314+0000 7efcc77fe640 1 -- 172.21.15.33:0/1595511936 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efcd40041f0 con 0x5612645c4710 2024-03-19T02:57:06.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.314+0000 7efcc77fe640 1 -- 172.21.15.33:0/1595511936 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd4019680 con 0x5612645c4710 2024-03-19T02:57:06.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.315+0000 7efce9f6b740 1 -- 172.21.15.33:0/1595511936 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612645c4710 msgr2=0x5612645d1750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:06.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.315+0000 7efce9f6b740 1 --2- 172.21.15.33:0/1595511936 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612645c4710 0x5612645d1750 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7efcd4009bf0 tx=0x7efcd4011810 comp rx=0 tx=0).stop 2024-03-19T02:57:06.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.315+0000 7efce9f6b740 1 -- 172.21.15.33:0/1595511936 shutdown_connections 2024-03-19T02:57:06.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.315+0000 7efce9f6b740 1 --2- 172.21.15.33:0/1595511936 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612645c4710 0x5612645d1750 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.315+0000 7efce9f6b740 1 --2- 172.21.15.33:0/1595511936 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612645d1c90 0x5612645d4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.315+0000 7efce9f6b740 1 --2- 172.21.15.33:0/1595511936 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612645c3d60 0x5612645c4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.315+0000 7efce9f6b740 1 -- 172.21.15.33:0/1595511936 >> 172.21.15.33:0/1595511936 conn(0x5612645beab0 msgr2=0x5612645bee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:06.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.315+0000 7efce9f6b740 1 -- 172.21.15.33:0/1595511936 shutdown_connections 2024-03-19T02:57:06.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.315+0000 7efce9f6b740 1 -- 172.21.15.33:0/1595511936 wait complete. 2024-03-19T02:57:06.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.315+0000 7efce9f6b740 1 -- start start 2024-03-19T02:57:06.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.315+0000 7efce9f6b740 1 Processor -- start 2024-03-19T02:57:06.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.316+0000 7efce9f6b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612645c3d60 0x5612645ceb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.316+0000 7efce9f6b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612645c4710 0x5612645cf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.316+0000 7efce9f6b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612645d1c90 0x5612645cf5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.316+0000 7efce9f6b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5612645d4a50 con 0x5612645c3d60 2024-03-19T02:57:06.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.316+0000 7efce9f6b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5612644e41a0 con 0x5612645d1c90 2024-03-19T02:57:06.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.316+0000 7efce9f6b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5612645d4bd0 con 0x5612645c4710 2024-03-19T02:57:06.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.316+0000 7efce9f6b740 10 client.0 _ll_register_callbacks cb 0x56126423bc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:57:06.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.316+0000 7efce503d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612645c3d60 0x5612645ceb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.316+0000 7efce583e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612645d1c90 0x5612645cf5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.316+0000 7efce483c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612645c4710 0x5612645cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.316+0000 7efce503d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612645c3d60 0x5612645ceb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59664/0 (socket says 172.21.15.33:59664) 2024-03-19T02:57:06.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.316+0000 7efce583e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612645d1c90 0x5612645cf5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:34086/0 (socket says 172.21.15.33:34086) 2024-03-19T02:57:06.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.316+0000 7efce503d640 1 -- 172.21.15.33:0/1956890206 learned_addr learned my addr 172.21.15.33:0/1956890206 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:06.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.316+0000 7efce483c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612645c4710 0x5612645cf080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:54996/0 (socket says 172.21.15.33:54996) 2024-03-19T02:57:06.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.316+0000 7efce503d640 1 -- 172.21.15.33:0/1956890206 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612645c4710 msgr2=0x5612645cf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:06.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.316+0000 7efce503d640 1 --2- 172.21.15.33:0/1956890206 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612645c4710 0x5612645cf080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.316+0000 7efce503d640 1 -- 172.21.15.33:0/1956890206 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612645d1c90 msgr2=0x5612645cf5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:06.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.316+0000 7efce503d640 1 --2- 172.21.15.33:0/1956890206 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612645d1c90 0x5612645cf5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.316+0000 7efce503d640 1 -- 172.21.15.33:0/1956890206 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561264364f70 con 0x5612645c3d60 2024-03-19T02:57:06.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.316+0000 7efce583e640 1 --2- 172.21.15.33:0/1956890206 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612645d1c90 0x5612645cf5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:06.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.316+0000 7efce483c640 1 --2- 172.21.15.33:0/1956890206 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612645c4710 0x5612645cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:06.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.316+0000 7efce503d640 1 --2- 172.21.15.33:0/1956890206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612645c3d60 0x5612645ceb40 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7efcd8005c20 tx=0x7efcd80bd360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff9c8382fa8de532 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:06.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.316+0000 7efcc4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:57:06.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.316+0000 7efcc4ff9640 1 -- 172.21.15.33:0/1956890206 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd80bdc20 con 0x5612645c3d60 2024-03-19T02:57:06.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.316+0000 7efcc4ff9640 1 -- 172.21.15.33:0/1956890206 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7efcd8003c80 con 0x5612645c3d60 2024-03-19T02:57:06.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.316+0000 7efce9f6b740 1 -- 172.21.15.33:0/1956890206 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efcd40041f0 con 0x5612645c3d60 2024-03-19T02:57:06.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.317+0000 7efcc4ff9640 1 -- 172.21.15.33:0/1956890206 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7efcd80bd520 con 0x5612645c3d60 2024-03-19T02:57:06.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.317+0000 7efce9f6b740 10 client.16470 fetch_fsmap learned FSMap version 25 2024-03-19T02:57:06.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.317+0000 7efce9f6b740 1 -- 172.21.15.33:0/1956890206 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5612641fc170 con 0x5612645c3d60 2024-03-19T02:57:06.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.317+0000 7efcc4ff9640 1 -- 172.21.15.33:0/1956890206 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efcd40041f0 con 0x5612645c3d60 2024-03-19T02:57:06.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.317+0000 7efcc4ff9640 1 -- 172.21.15.33:0/1956890206 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7efcd8005490 con 0x5612645c3d60 2024-03-19T02:57:06.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.317+0000 7efce9f6b740 10 client.16470 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:57:06.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.317+0000 7efce9f6b740 10 client.16470 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:57:06.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.317+0000 7efce9f6b740 20 client.16470 populate_metadata read hostname 'smithi033' 2024-03-19T02:57:06.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.317+0000 7efce9f6b740 1 --2- 172.21.15.33:0/1956890206 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5612645e1f30 0x5612645e43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.317+0000 7efce9f6b740 4 client.16470 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:57:06.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.317+0000 7efce9f6b740 1 -- 172.21.15.33:0/1956890206 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612645356a0 con 0x5612645e1f30 2024-03-19T02:57:06.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.317+0000 7efce483c640 1 --2- 172.21.15.33:0/1956890206 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5612645e1f30 0x5612645e43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.318+0000 7efcc4ff9640 10 client.16470 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:57:06.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.318+0000 7efce483c640 1 --2- 172.21.15.33:0/1956890206 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5612645e1f30 0x5612645e43b0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58bc173ce6445040 server_cookie=6e372bdf176784c2 in_seq=0 out_seq=0 2024-03-19T02:57:06.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.318+0000 7efcc4ff9640 1 -- 172.21.15.33:0/1956890206 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5612645356a0 con 0x5612645e1f30 2024-03-19T02:57:06.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.318+0000 7efcc4ff9640 10 client.16470 handle_command_reply: tid=0 2024-03-19T02:57:06.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.318+0000 7efce9f6b740 1 client.16470 shutdown 2024-03-19T02:57:06.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.318+0000 7efce9f6b740 1 -- 172.21.15.33:0/1956890206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612645c3d60 msgr2=0x5612645ceb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:06.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.318+0000 7efce9f6b740 1 --2- 172.21.15.33:0/1956890206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612645c3d60 0x5612645ceb40 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7efcd8005c20 tx=0x7efcd80bd360 comp rx=0 tx=0).stop 2024-03-19T02:57:06.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.318+0000 7efce9f6b740 1 -- 172.21.15.33:0/1956890206 shutdown_connections 2024-03-19T02:57:06.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.318+0000 7efce9f6b740 1 --2- 172.21.15.33:0/1956890206 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5612645e1f30 0x5612645e43b0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.318+0000 7efce9f6b740 1 --2- 172.21.15.33:0/1956890206 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612645c4710 0x5612645cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.318+0000 7efce9f6b740 1 --2- 172.21.15.33:0/1956890206 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612645d1c90 0x5612645cf5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.318+0000 7efcc4ff9640 0 client.16470 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:57:06.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.318+0000 7efce9f6b740 1 --2- 172.21.15.33:0/1956890206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612645c3d60 0x5612645ceb40 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.318+0000 7efce9f6b740 1 -- 172.21.15.33:0/1956890206 >> 172.21.15.33:0/1956890206 conn(0x5612645beab0 msgr2=0x5612645c0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:06.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.318+0000 7efce9f6b740 1 -- 172.21.15.33:0/1956890206 shutdown_connections 2024-03-19T02:57:06.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.318+0000 7efce9f6b740 1 -- 172.21.15.33:0/1956890206 wait complete. 2024-03-19T02:57:06.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.319+0000 7efce9f6b740 20 client.16470 trim_cache size 0 max 16384 2024-03-19T02:57:06.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.336+0000 7efce9f6b740 1 Processor -- start 2024-03-19T02:57:06.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.336+0000 7efce9f6b740 1 -- start start 2024-03-19T02:57:06.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.336+0000 7efce9f6b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612645c3d60 0x5612645d1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.337+0000 7efce9f6b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612645c4710 0x5612645dd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.337+0000 7efce9f6b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612645d1c90 0x5612645c0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.337+0000 7efce9f6b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efce01af1a0 con 0x5612645c4710 2024-03-19T02:57:06.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.337+0000 7efce9f6b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efce01aca70 con 0x5612645c3d60 2024-03-19T02:57:06.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.337+0000 7efce9f6b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7efce01aa8a0 con 0x5612645d1c90 2024-03-19T02:57:06.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.337+0000 7efce483c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612645c4710 0x5612645dd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.337+0000 7efce503d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612645c3d60 0x5612645d1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.337+0000 7efce483c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612645c4710 0x5612645dd8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59668/0 (socket says 172.21.15.33:59668) 2024-03-19T02:57:06.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.337+0000 7efce483c640 1 -- 172.21.15.33:0/2348755059 learned_addr learned my addr 172.21.15.33:0/2348755059 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:06.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.337+0000 7efce503d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612645c3d60 0x5612645d1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:34102/0 (socket says 172.21.15.33:34102) 2024-03-19T02:57:06.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.337+0000 7efce583e640 1 --2- 172.21.15.33:0/2348755059 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612645d1c90 0x5612645c0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.337+0000 7efce483c640 1 -- 172.21.15.33:0/2348755059 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612645d1c90 msgr2=0x5612645c0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:06.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.337+0000 7efce483c640 1 --2- 172.21.15.33:0/2348755059 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612645d1c90 0x5612645c0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.337+0000 7efce483c640 1 -- 172.21.15.33:0/2348755059 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612645c3d60 msgr2=0x5612645d1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:06.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.337+0000 7efce483c640 1 --2- 172.21.15.33:0/2348755059 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612645c3d60 0x5612645d1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.337+0000 7efce483c640 1 -- 172.21.15.33:0/2348755059 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612645be8e0 con 0x5612645c4710 2024-03-19T02:57:06.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.337+0000 7efce503d640 1 --2- 172.21.15.33:0/2348755059 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612645c3d60 0x5612645d1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:06.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.337+0000 7efce483c640 1 --2- 172.21.15.33:0/2348755059 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612645c4710 0x5612645dd8c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7efcd4014040 tx=0x7efcd40096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81a5b4c5c569a21e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:06.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.337+0000 7efcc6ffd640 1 -- 172.21.15.33:0/2348755059 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd4009da0 con 0x5612645c4710 2024-03-19T02:57:06.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.337+0000 7efcc6ffd640 1 -- 172.21.15.33:0/2348755059 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efcd4003ad0 con 0x5612645c4710 2024-03-19T02:57:06.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.337+0000 7efcc6ffd640 1 -- 172.21.15.33:0/2348755059 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd4019920 con 0x5612645c4710 2024-03-19T02:57:06.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.338+0000 7efce9f6b740 1 -- 172.21.15.33:0/2348755059 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612645c4710 msgr2=0x5612645dd8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:06.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.338+0000 7efce9f6b740 1 --2- 172.21.15.33:0/2348755059 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612645c4710 0x5612645dd8c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7efcd4014040 tx=0x7efcd40096a0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.338+0000 7efce9f6b740 1 -- 172.21.15.33:0/2348755059 shutdown_connections 2024-03-19T02:57:06.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.338+0000 7efce9f6b740 1 --2- 172.21.15.33:0/2348755059 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612645d1c90 0x5612645c0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.338+0000 7efce9f6b740 1 --2- 172.21.15.33:0/2348755059 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612645c4710 0x5612645dd8c0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.338+0000 7efce9f6b740 1 --2- 172.21.15.33:0/2348755059 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612645c3d60 0x5612645d1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.338+0000 7efce9f6b740 1 -- 172.21.15.33:0/2348755059 >> 172.21.15.33:0/2348755059 conn(0x5612645beab0 msgr2=0x5612645bff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:06.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.338+0000 7efce9f6b740 1 -- 172.21.15.33:0/2348755059 shutdown_connections 2024-03-19T02:57:06.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.338+0000 7efce9f6b740 1 -- 172.21.15.33:0/2348755059 wait complete. 2024-03-19T02:57:06.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.338+0000 7efce9f6b740 1 -- start start 2024-03-19T02:57:06.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.338+0000 7efce9f6b740 1 Processor -- start 2024-03-19T02:57:06.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.339+0000 7efce9f6b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612645c3d60 0x5612645da360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.339+0000 7efce9f6b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612645c4710 0x5612645d56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.339+0000 7efce9f6b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612645d1c90 0x5612645d5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.339+0000 7efce9f6b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7efce01aaf30 con 0x5612645c4710 2024-03-19T02:57:06.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.339+0000 7efce9f6b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efce01ac3a0 con 0x5612645d1c90 2024-03-19T02:57:06.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.339+0000 7efce9f6b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7efce01ab600 con 0x5612645c3d60 2024-03-19T02:57:06.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.339+0000 7efce9f6b740 10 client.0 _ll_register_callbacks cb 0x7efcd80b98b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:57:06.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.339+0000 7efce483c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612645c4710 0x5612645d56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.339+0000 7efce483c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612645c4710 0x5612645d56d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59672/0 (socket says 172.21.15.33:59672) 2024-03-19T02:57:06.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.339+0000 7efce483c640 1 -- 172.21.15.33:0/1720672852 learned_addr learned my addr 172.21.15.33:0/1720672852 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:06.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.339+0000 7efce583e640 1 --2- 172.21.15.33:0/1720672852 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612645d1c90 0x5612645d5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.339+0000 7efce503d640 1 --2- 172.21.15.33:0/1720672852 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612645c3d60 0x5612645da360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.339+0000 7efce483c640 1 -- 172.21.15.33:0/1720672852 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612645c3d60 msgr2=0x5612645da360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:06.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.339+0000 7efce483c640 1 --2- 172.21.15.33:0/1720672852 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612645c3d60 0x5612645da360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.339+0000 7efce483c640 1 -- 172.21.15.33:0/1720672852 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612645d1c90 msgr2=0x5612645d5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:06.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.339+0000 7efce483c640 1 --2- 172.21.15.33:0/1720672852 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612645d1c90 0x5612645d5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.339+0000 7efce483c640 1 -- 172.21.15.33:0/1720672852 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612645c3470 con 0x5612645c4710 2024-03-19T02:57:06.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.339+0000 7efce503d640 1 --2- 172.21.15.33:0/1720672852 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612645c3d60 0x5612645da360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:57:06.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.339+0000 7efce583e640 1 --2- 172.21.15.33:0/1720672852 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612645d1c90 0x5612645d5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:06.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.339+0000 7efce483c640 1 --2- 172.21.15.33:0/1720672852 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612645c4710 0x5612645d56d0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7efcd4009820 tx=0x7efcd4009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a8197edba367ab4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:06.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.339+0000 7efcc4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:57:06.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.339+0000 7efcc4ff9640 1 -- 172.21.15.33:0/1720672852 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd4039020 con 0x5612645c4710 2024-03-19T02:57:06.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.339+0000 7efcc4ff9640 1 -- 172.21.15.33:0/1720672852 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7efcd4005040 con 0x5612645c4710 2024-03-19T02:57:06.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.339+0000 7efce9f6b740 1 -- 172.21.15.33:0/1720672852 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efcd4003ad0 con 0x5612645c4710 2024-03-19T02:57:06.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.340+0000 7efcc4ff9640 1 -- 172.21.15.33:0/1720672852 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7efcd4003e90 con 0x5612645c4710 2024-03-19T02:57:06.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.340+0000 7efce9f6b740 10 client.16476 fetch_fsmap learned FSMap version 25 2024-03-19T02:57:06.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.340+0000 7efce9f6b740 1 -- 172.21.15.33:0/1720672852 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x561264648290 con 0x5612645c4710 2024-03-19T02:57:06.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.340+0000 7efcc4ff9640 1 -- 172.21.15.33:0/1720672852 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efcd4003ad0 con 0x5612645c4710 2024-03-19T02:57:06.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.340+0000 7efcc4ff9640 1 -- 172.21.15.33:0/1720672852 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7efcd4019ca0 con 0x5612645c4710 2024-03-19T02:57:06.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.340+0000 7efce9f6b740 10 client.16476 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:57:06.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.340+0000 7efce9f6b740 10 client.16476 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:57:06.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.340+0000 7efce9f6b740 20 client.16476 populate_metadata read hostname 'smithi033' 2024-03-19T02:57:06.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.340+0000 7efce9f6b740 1 --2- 172.21.15.33:0/1720672852 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x561264648800 0x56126464ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.340+0000 7efce9f6b740 4 client.16476 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:57:06.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.340+0000 7efce9f6b740 1 -- 172.21.15.33:0/1720672852 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56126452f9f0 con 0x561264648800 2024-03-19T02:57:06.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.340+0000 7efce503d640 1 --2- 172.21.15.33:0/1720672852 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x561264648800 0x56126464ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.340+0000 7efcc4ff9640 10 client.16476 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:57:06.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.341+0000 7efce503d640 1 --2- 172.21.15.33:0/1720672852 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x561264648800 0x56126464ac20 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f8b5844cc742b23 server_cookie=7fb9bcc177495fb1 in_seq=0 out_seq=0 2024-03-19T02:57:06.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.342+0000 7efcc4ff9640 1 -- 172.21.15.33:0/1720672852 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56126452f9f0 con 0x561264648800 2024-03-19T02:57:06.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.342+0000 7efcc4ff9640 10 client.16476 handle_command_reply: tid=0 2024-03-19T02:57:06.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.342+0000 7efce9f6b740 1 client.16476 shutdown 2024-03-19T02:57:06.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.342+0000 7efce9f6b740 1 -- 172.21.15.33:0/1720672852 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612645c4710 msgr2=0x5612645d56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:06.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.342+0000 7efce9f6b740 1 --2- 172.21.15.33:0/1720672852 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612645c4710 0x5612645d56d0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7efcd4009820 tx=0x7efcd4009310 comp rx=0 tx=0).stop 2024-03-19T02:57:06.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.342+0000 7efce9f6b740 1 -- 172.21.15.33:0/1720672852 shutdown_connections 2024-03-19T02:57:06.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.342+0000 7efce9f6b740 1 --2- 172.21.15.33:0/1720672852 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x561264648800 0x56126464ac20 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.342+0000 7efce9f6b740 1 --2- 172.21.15.33:0/1720672852 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5612645d1c90 0x5612645d5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.342+0000 7efce9f6b740 1 --2- 172.21.15.33:0/1720672852 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5612645c4710 0x5612645d56d0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.342+0000 7efcc4ff9640 0 client.16476 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:57:06.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.342+0000 7efce9f6b740 1 --2- 172.21.15.33:0/1720672852 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5612645c3d60 0x5612645da360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.342+0000 7efce9f6b740 1 -- 172.21.15.33:0/1720672852 >> 172.21.15.33:0/1720672852 conn(0x5612645beab0 msgr2=0x5612645c0790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:06.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.342+0000 7efce9f6b740 1 -- 172.21.15.33:0/1720672852 shutdown_connections 2024-03-19T02:57:06.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.342+0000 7efce9f6b740 1 -- 172.21.15.33:0/1720672852 wait complete. 2024-03-19T02:57:06.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.342+0000 7efce9f6b740 20 client.16476 trim_cache size 0 max 16384 2024-03-19T02:57:06.350 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:57:06.350 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-03-19T02:57:06.350 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "f199ae72-870d-4e4a-b991-7afcf16af4ff", 2024-03-19T02:57:06.350 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-03-19T02:57:06.350 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:57:06.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.343+0000 7efce72a1640 1 -- 172.21.15.33:0/3944280727 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7efcc80a2970 msgr2=0x7efcc80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:06.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.343+0000 7efce72a1640 1 --2- 172.21.15.33:0/3944280727 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7efcc80a2970 0x7efcc80a4e30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7efcd0009990 tx=0x7efcd0009de0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.343+0000 7efce72a1640 1 -- 172.21.15.33:0/3944280727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efce01017b0 msgr2=0x7efce0197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:06.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.343+0000 7efce72a1640 1 --2- 172.21.15.33:0/3944280727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efce01017b0 0x7efce0197fc0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7efcd8009230 tx=0x7efcd802c710 comp rx=0 tx=0).stop 2024-03-19T02:57:06.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.343+0000 7efce72a1640 1 -- 172.21.15.33:0/3944280727 shutdown_connections 2024-03-19T02:57:06.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.343+0000 7efce72a1640 1 --2- 172.21.15.33:0/3944280727 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7efcc80a2970 0x7efcc80a4e30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.343+0000 7efce72a1640 1 --2- 172.21.15.33:0/3944280727 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efce01077c0 0x7efce019c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.343+0000 7efce72a1640 1 --2- 172.21.15.33:0/3944280727 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efce0102170 0x7efce0198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.343+0000 7efce72a1640 1 --2- 172.21.15.33:0/3944280727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7efce01017b0 0x7efce0197fc0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.343+0000 7efce72a1640 1 -- 172.21.15.33:0/3944280727 >> 172.21.15.33:0/3944280727 conn(0x7efce0078c20 msgr2=0x7efce01042c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:06.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.343+0000 7efce72a1640 1 -- 172.21.15.33:0/3944280727 shutdown_connections 2024-03-19T02:57:06.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.343+0000 7efce72a1640 1 -- 172.21.15.33:0/3944280727 wait complete. 2024-03-19T02:57:06.355 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:57:06.355 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:57:06.378 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:06 smithi033 ceph-mon[31973]: pgmap v112: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-19T02:57:06.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.459+0000 7fe326f3b640 1 -- 172.21.15.33:0/2462902958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe320100740 msgr2=0x7fe320100bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:06.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.459+0000 7fe326f3b640 1 --2- 172.21.15.33:0/2462902958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe320100740 0x7fe320100bc0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fe30c009930 tx=0x7fe30c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:57:06.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.459+0000 7fe326f3b640 1 -- 172.21.15.33:0/2462902958 shutdown_connections 2024-03-19T02:57:06.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.459+0000 7fe326f3b640 1 --2- 172.21.15.33:0/2462902958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe320101100 0x7fe32010d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.459+0000 7fe326f3b640 1 --2- 172.21.15.33:0/2462902958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe320100740 0x7fe320100bc0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.459+0000 7fe326f3b640 1 --2- 172.21.15.33:0/2462902958 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe320106750 0x7fe320106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.459+0000 7fe326f3b640 1 -- 172.21.15.33:0/2462902958 >> 172.21.15.33:0/2462902958 conn(0x7fe320077880 msgr2=0x7fe320077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:06.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.459+0000 7fe326f3b640 1 -- 172.21.15.33:0/2462902958 shutdown_connections 2024-03-19T02:57:06.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.459+0000 7fe326f3b640 1 -- 172.21.15.33:0/2462902958 wait complete. 2024-03-19T02:57:06.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.460+0000 7fe326f3b640 1 Processor -- start 2024-03-19T02:57:06.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.460+0000 7fe326f3b640 1 -- start start 2024-03-19T02:57:06.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.460+0000 7fe326f3b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe320100740 0x7fe320196f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.460+0000 7fe326f3b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe320101100 0x7fe320197470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.460+0000 7fe326f3b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe320106750 0x7fe32019b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.460+0000 7fe326f3b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe320110a10 con 0x7fe320100740 2024-03-19T02:57:06.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.460+0000 7fe326f3b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe320110890 con 0x7fe320101100 2024-03-19T02:57:06.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.460+0000 7fe326f3b640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe320110b90 con 0x7fe320106750 2024-03-19T02:57:06.463 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.460+0000 7fe324cd7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe320100740 0x7fe320196f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.460+0000 7fe324cd7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe320100740 0x7fe320196f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59692/0 (socket says 172.21.15.33:59692) 2024-03-19T02:57:06.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.460+0000 7fe324cd7640 1 -- 172.21.15.33:0/3871981407 learned_addr learned my addr 172.21.15.33:0/3871981407 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:06.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.460+0000 7fe31ffff640 1 --2- 172.21.15.33:0/3871981407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe320101100 0x7fe320197470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.460+0000 7fe3254d8640 1 --2- 172.21.15.33:0/3871981407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe320106750 0x7fe32019b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.460+0000 7fe324cd7640 1 -- 172.21.15.33:0/3871981407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe320106750 msgr2=0x7fe32019b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:06.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.460+0000 7fe324cd7640 1 --2- 172.21.15.33:0/3871981407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe320106750 0x7fe32019b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.460+0000 7fe324cd7640 1 -- 172.21.15.33:0/3871981407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe320101100 msgr2=0x7fe320197470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:06.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.460+0000 7fe324cd7640 1 --2- 172.21.15.33:0/3871981407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe320101100 0x7fe320197470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.460+0000 7fe324cd7640 1 -- 172.21.15.33:0/3871981407 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe32019bf20 con 0x7fe320100740 2024-03-19T02:57:06.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.460+0000 7fe324cd7640 1 --2- 172.21.15.33:0/3871981407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe320100740 0x7fe320196f30 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fe31400eb00 tx=0x7fe31400ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:06.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.461+0000 7fe31dffb640 1 -- 172.21.15.33:0/3871981407 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31401c070 con 0x7fe320100740 2024-03-19T02:57:06.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.461+0000 7fe31dffb640 1 -- 172.21.15.33:0/3871981407 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe31400bb70 con 0x7fe320100740 2024-03-19T02:57:06.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.461+0000 7fe31dffb640 1 -- 172.21.15.33:0/3871981407 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe314016610 con 0x7fe320100740 2024-03-19T02:57:06.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.461+0000 7fe326f3b640 1 -- 172.21.15.33:0/3871981407 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe32019c210 con 0x7fe320100740 2024-03-19T02:57:06.464 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.461+0000 7fe326f3b640 1 -- 172.21.15.33:0/3871981407 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe320069df0 con 0x7fe320100740 2024-03-19T02:57:06.470 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:06 smithi096 ceph-mon[35946]: pgmap v112: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-19T02:57:06.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.463+0000 7fe31dffb640 1 -- 172.21.15.33:0/3871981407 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fe31401f070 con 0x7fe320100740 2024-03-19T02:57:06.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.464+0000 7fe329c03740 1 Processor -- start 2024-03-19T02:57:06.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.464+0000 7fe329c03740 1 -- start start 2024-03-19T02:57:06.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.464+0000 7fe329c03740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56243e639a30 0x56243e639e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.464+0000 7fe329c03740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56243e63a350 0x56243e6429f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.464+0000 7fe329c03740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56243e642f30 0x56243e645320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.464+0000 7fe329c03740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56243e59d500 con 0x56243e642f30 2024-03-19T02:57:06.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.464+0000 7fe329c03740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x56243e545be0 con 0x56243e639a30 2024-03-19T02:57:06.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.464+0000 7fe329c03740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56243e5995a0 con 0x56243e63a350 2024-03-19T02:57:06.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.464+0000 7fe31dffb640 1 --2- 172.21.15.33:0/3871981407 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe3100a29f0 0x7fe3100a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.464+0000 7fe31ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56243e642f30 0x56243e645320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.464+0000 7fe324cd7640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56243e63a350 0x56243e6429f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.464+0000 7fe31ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56243e642f30 0x56243e645320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59696/0 (socket says 172.21.15.33:59696) 2024-03-19T02:57:06.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.464+0000 7fe3254d8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56243e639a30 0x56243e639e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.464+0000 7fe31ffff640 1 -- 172.21.15.33:0/2811951304 learned_addr learned my addr 172.21.15.33:0/2811951304 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:06.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.464+0000 7fe3254d8640 1 --2- 172.21.15.33:0/3871981407 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe3100a29f0 0x7fe3100a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.464+0000 7fe31dffb640 1 -- 172.21.15.33:0/3871981407 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fe3140d0620 con 0x7fe320100740 2024-03-19T02:57:06.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.464+0000 7fe31ffff640 1 -- 172.21.15.33:0/2811951304 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56243e63a350 msgr2=0x56243e6429f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:06.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.464+0000 7fe31ffff640 1 --2- 172.21.15.33:0/2811951304 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56243e63a350 0x56243e6429f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.464+0000 7fe31ffff640 1 -- 172.21.15.33:0/2811951304 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56243e639a30 msgr2=0x56243e639e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:06.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.464+0000 7fe31ffff640 1 --2- 172.21.15.33:0/2811951304 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56243e639a30 0x56243e639e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.464+0000 7fe31ffff640 1 -- 172.21.15.33:0/2811951304 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56243e567000 con 0x56243e642f30 2024-03-19T02:57:06.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.465+0000 7fe31ffff640 1 --2- 172.21.15.33:0/2811951304 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56243e642f30 0x56243e645320 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fe30c036040 tx=0x7fe30c009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ef029d116d30eeb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:06.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.465+0000 7fe3254d8640 1 --2- 172.21.15.33:0/3871981407 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe3100a29f0 0x7fe3100a4eb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe318009b80 tx=0x7fe318009500 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:06.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.465+0000 7fe30affd640 1 -- 172.21.15.33:0/2811951304 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30c045020 con 0x56243e642f30 2024-03-19T02:57:06.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.465+0000 7fe30affd640 1 -- 172.21.15.33:0/2811951304 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe30c02c7e0 con 0x56243e642f30 2024-03-19T02:57:06.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.465+0000 7fe30affd640 1 -- 172.21.15.33:0/2811951304 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30c005430 con 0x56243e642f30 2024-03-19T02:57:06.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.465+0000 7fe329c03740 1 -- 172.21.15.33:0/2811951304 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56243e642f30 msgr2=0x56243e645320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:06.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.465+0000 7fe329c03740 1 --2- 172.21.15.33:0/2811951304 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56243e642f30 0x56243e645320 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fe30c036040 tx=0x7fe30c009560 comp rx=0 tx=0).stop 2024-03-19T02:57:06.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.466+0000 7fe329c03740 1 -- 172.21.15.33:0/2811951304 shutdown_connections 2024-03-19T02:57:06.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.466+0000 7fe329c03740 1 --2- 172.21.15.33:0/2811951304 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56243e63a350 0x56243e6429f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.466+0000 7fe329c03740 1 --2- 172.21.15.33:0/2811951304 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56243e642f30 0x56243e645320 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.466+0000 7fe329c03740 1 --2- 172.21.15.33:0/2811951304 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56243e639a30 0x56243e639e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.466+0000 7fe329c03740 1 -- 172.21.15.33:0/2811951304 >> 172.21.15.33:0/2811951304 conn(0x56243e634160 msgr2=0x56243e634540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:06.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.466+0000 7fe329c03740 1 -- 172.21.15.33:0/2811951304 shutdown_connections 2024-03-19T02:57:06.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.466+0000 7fe329c03740 1 -- 172.21.15.33:0/2811951304 wait complete. 2024-03-19T02:57:06.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.466+0000 7fe329c03740 1 -- start start 2024-03-19T02:57:06.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.466+0000 7fe329c03740 1 Processor -- start 2024-03-19T02:57:06.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.466+0000 7fe329c03740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56243e639a30 0x56243e63fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.466+0000 7fe329c03740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56243e63a350 0x56243e640350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.466+0000 7fe329c03740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56243e642f30 0x56243e640890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.466+0000 7fe329c03740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56243e645cc0 con 0x56243e639a30 2024-03-19T02:57:06.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.466+0000 7fe329c03740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x56243e559160 con 0x56243e63a350 2024-03-19T02:57:06.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.466+0000 7fe329c03740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56243e645e40 con 0x56243e642f30 2024-03-19T02:57:06.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.466+0000 7fe329c03740 10 client.0 _ll_register_callbacks cb 0x56243e595a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:57:06.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.466+0000 7fe324cd7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56243e639a30 0x56243e63fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.466+0000 7fe324cd7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56243e639a30 0x56243e63fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59710/0 (socket says 172.21.15.33:59710) 2024-03-19T02:57:06.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.466+0000 7fe324cd7640 1 -- 172.21.15.33:0/1024756257 learned_addr learned my addr 172.21.15.33:0/1024756257 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:06.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.466+0000 7fe31ffff640 1 --2- 172.21.15.33:0/1024756257 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56243e63a350 0x56243e640350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.466+0000 7fe3254d8640 1 --2- 172.21.15.33:0/1024756257 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56243e642f30 0x56243e640890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.467+0000 7fe324cd7640 1 -- 172.21.15.33:0/1024756257 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56243e642f30 msgr2=0x56243e640890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:06.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.467+0000 7fe324cd7640 1 --2- 172.21.15.33:0/1024756257 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56243e642f30 0x56243e640890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.467+0000 7fe324cd7640 1 -- 172.21.15.33:0/1024756257 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56243e63a350 msgr2=0x56243e640350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:06.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.467+0000 7fe324cd7640 1 --2- 172.21.15.33:0/1024756257 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56243e63a350 0x56243e640350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.467+0000 7fe324cd7640 1 -- 172.21.15.33:0/1024756257 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56243e37b7d0 con 0x56243e639a30 2024-03-19T02:57:06.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.467+0000 7fe324cd7640 1 --2- 172.21.15.33:0/1024756257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56243e639a30 0x56243e63fe10 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fe31409c950 tx=0x7fe314009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=220fc656eb0e3ebd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:06.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.467+0000 7fe2fffff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:57:06.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.467+0000 7fe2fffff640 1 -- 172.21.15.33:0/1024756257 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe314026040 con 0x56243e639a30 2024-03-19T02:57:06.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.467+0000 7fe2fffff640 1 -- 172.21.15.33:0/1024756257 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fe3140cdc80 con 0x56243e639a30 2024-03-19T02:57:06.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.467+0000 7fe329c03740 1 -- 172.21.15.33:0/1024756257 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe30c02c7e0 con 0x56243e639a30 2024-03-19T02:57:06.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.467+0000 7fe2fffff640 1 -- 172.21.15.33:0/1024756257 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe3140cde80 con 0x56243e639a30 2024-03-19T02:57:06.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.467+0000 7fe329c03740 10 client.16488 fetch_fsmap learned FSMap version 25 2024-03-19T02:57:06.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.467+0000 7fe329c03740 1 -- 172.21.15.33:0/1024756257 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56243e2b13f0 con 0x56243e639a30 2024-03-19T02:57:06.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.467+0000 7fe2fffff640 1 -- 172.21.15.33:0/1024756257 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe30c02c7e0 con 0x56243e639a30 2024-03-19T02:57:06.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.467+0000 7fe2fffff640 1 -- 172.21.15.33:0/1024756257 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fe31409bfa0 con 0x56243e639a30 2024-03-19T02:57:06.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.467+0000 7fe329c03740 10 client.16488 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:57:06.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.467+0000 7fe329c03740 10 client.16488 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:57:06.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.467+0000 7fe329c03740 20 client.16488 populate_metadata read hostname 'smithi033' 2024-03-19T02:57:06.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.467+0000 7fe329c03740 1 --2- 172.21.15.33:0/1024756257 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x56243e653100 0x56243e655510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.467+0000 7fe329c03740 4 client.16488 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:57:06.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.467+0000 7fe329c03740 1 -- 172.21.15.33:0/1024756257 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56243e5aaa40 con 0x56243e653100 2024-03-19T02:57:06.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.468+0000 7fe31ffff640 1 --2- 172.21.15.33:0/1024756257 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x56243e653100 0x56243e655510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.468+0000 7fe2fffff640 10 client.16488 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:57:06.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.468+0000 7fe31ffff640 1 --2- 172.21.15.33:0/1024756257 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x56243e653100 0x56243e655510 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fdf4d567b1fdbeb2 server_cookie=dfe347b68568d60b in_seq=0 out_seq=0 2024-03-19T02:57:06.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.469+0000 7fe2fffff640 1 -- 172.21.15.33:0/1024756257 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56243e5aaa40 con 0x56243e653100 2024-03-19T02:57:06.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.469+0000 7fe2fffff640 10 client.16488 handle_command_reply: tid=0 2024-03-19T02:57:06.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.469+0000 7fe329c03740 1 client.16488 shutdown 2024-03-19T02:57:06.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.469+0000 7fe329c03740 1 -- 172.21.15.33:0/1024756257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56243e639a30 msgr2=0x56243e63fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:06.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.469+0000 7fe329c03740 1 --2- 172.21.15.33:0/1024756257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56243e639a30 0x56243e63fe10 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fe31409c950 tx=0x7fe314009700 comp rx=0 tx=0).stop 2024-03-19T02:57:06.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.469+0000 7fe329c03740 1 -- 172.21.15.33:0/1024756257 shutdown_connections 2024-03-19T02:57:06.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.469+0000 7fe329c03740 1 --2- 172.21.15.33:0/1024756257 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x56243e653100 0x56243e655510 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.469+0000 7fe329c03740 1 --2- 172.21.15.33:0/1024756257 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56243e642f30 0x56243e640890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.469+0000 7fe2fffff640 0 client.16488 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:57:06.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.469+0000 7fe329c03740 1 --2- 172.21.15.33:0/1024756257 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56243e63a350 0x56243e640350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.469+0000 7fe329c03740 1 --2- 172.21.15.33:0/1024756257 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56243e639a30 0x56243e63fe10 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.469+0000 7fe329c03740 1 -- 172.21.15.33:0/1024756257 >> 172.21.15.33:0/1024756257 conn(0x56243e634160 msgr2=0x56243e635eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:06.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.469+0000 7fe329c03740 1 -- 172.21.15.33:0/1024756257 shutdown_connections 2024-03-19T02:57:06.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.469+0000 7fe329c03740 1 -- 172.21.15.33:0/1024756257 wait complete. 2024-03-19T02:57:06.478 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.469+0000 7fe329c03740 20 client.16488 trim_cache size 0 max 16384 2024-03-19T02:57:06.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.487+0000 7fe329c03740 1 Processor -- start 2024-03-19T02:57:06.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.487+0000 7fe329c03740 1 -- start start 2024-03-19T02:57:06.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.487+0000 7fe329c03740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56243e639a30 0x56243e642540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.487+0000 7fe329c03740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56243e63a350 0x56243e64ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.487+0000 7fe329c03740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56243e642f30 0x56243e65ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.487+0000 7fe329c03740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe3201af130 con 0x56243e639a30 2024-03-19T02:57:06.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.487+0000 7fe329c03740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe3201aca00 con 0x56243e63a350 2024-03-19T02:57:06.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.487+0000 7fe329c03740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe3201aa830 con 0x56243e642f30 2024-03-19T02:57:06.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.487+0000 7fe324cd7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56243e639a30 0x56243e642540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.487+0000 7fe324cd7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56243e639a30 0x56243e642540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59718/0 (socket says 172.21.15.33:59718) 2024-03-19T02:57:06.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.487+0000 7fe324cd7640 1 -- 172.21.15.33:0/972002683 learned_addr learned my addr 172.21.15.33:0/972002683 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:06.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.487+0000 7fe31ffff640 1 --2- 172.21.15.33:0/972002683 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56243e63a350 0x56243e64ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.487+0000 7fe3254d8640 1 --2- 172.21.15.33:0/972002683 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56243e642f30 0x56243e65ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.487+0000 7fe324cd7640 1 -- 172.21.15.33:0/972002683 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56243e642f30 msgr2=0x56243e65ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:06.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.487+0000 7fe324cd7640 1 --2- 172.21.15.33:0/972002683 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56243e642f30 0x56243e65ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.487+0000 7fe324cd7640 1 -- 172.21.15.33:0/972002683 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56243e63a350 msgr2=0x56243e64ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:06.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.487+0000 7fe324cd7640 1 --2- 172.21.15.33:0/972002683 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56243e63a350 0x56243e64ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.487+0000 7fe324cd7640 1 -- 172.21.15.33:0/972002683 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56243e2710f0 con 0x56243e639a30 2024-03-19T02:57:06.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.487+0000 7fe324cd7640 1 --2- 172.21.15.33:0/972002683 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56243e639a30 0x56243e642540 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fe314098e20 tx=0x7fe31409aed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8abbbdb02c583eb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:06.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.488+0000 7fe30a7fc640 1 -- 172.21.15.33:0/972002683 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31409c500 con 0x56243e639a30 2024-03-19T02:57:06.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.488+0000 7fe30a7fc640 1 -- 172.21.15.33:0/972002683 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe3140d0460 con 0x56243e639a30 2024-03-19T02:57:06.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.488+0000 7fe30a7fc640 1 -- 172.21.15.33:0/972002683 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3140872d0 con 0x56243e639a30 2024-03-19T02:57:06.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.488+0000 7fe329c03740 1 -- 172.21.15.33:0/972002683 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56243e639a30 msgr2=0x56243e642540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:06.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.488+0000 7fe329c03740 1 --2- 172.21.15.33:0/972002683 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56243e639a30 0x56243e642540 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fe314098e20 tx=0x7fe31409aed0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.488+0000 7fe329c03740 1 -- 172.21.15.33:0/972002683 shutdown_connections 2024-03-19T02:57:06.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.488+0000 7fe329c03740 1 --2- 172.21.15.33:0/972002683 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56243e642f30 0x56243e65ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.488+0000 7fe329c03740 1 --2- 172.21.15.33:0/972002683 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56243e63a350 0x56243e64ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.488+0000 7fe329c03740 1 --2- 172.21.15.33:0/972002683 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56243e639a30 0x56243e642540 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.488+0000 7fe329c03740 1 -- 172.21.15.33:0/972002683 >> 172.21.15.33:0/972002683 conn(0x56243e634160 msgr2=0x56243e63fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:06.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.488+0000 7fe329c03740 1 -- 172.21.15.33:0/972002683 shutdown_connections 2024-03-19T02:57:06.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.488+0000 7fe329c03740 1 -- 172.21.15.33:0/972002683 wait complete. 2024-03-19T02:57:06.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.488+0000 7fe329c03740 1 -- start start 2024-03-19T02:57:06.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.489+0000 7fe329c03740 1 Processor -- start 2024-03-19T02:57:06.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.489+0000 7fe329c03740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56243e639a30 0x56243e64b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.489+0000 7fe329c03740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56243e63a350 0x56243e63c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.489+0000 7fe329c03740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56243e642f30 0x56243e63cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.489+0000 7fe329c03740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe3201aaec0 con 0x56243e642f30 2024-03-19T02:57:06.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.489+0000 7fe329c03740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe3201ac330 con 0x56243e63a350 2024-03-19T02:57:06.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.489+0000 7fe329c03740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe3201ab590 con 0x56243e639a30 2024-03-19T02:57:06.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.489+0000 7fe329c03740 10 client.0 _ll_register_callbacks cb 0x7fe314017a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:57:06.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.489+0000 7fe3254d8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56243e642f30 0x56243e63cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.489+0000 7fe3254d8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56243e642f30 0x56243e63cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:59730/0 (socket says 172.21.15.33:59730) 2024-03-19T02:57:06.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.489+0000 7fe3254d8640 1 -- 172.21.15.33:0/3617704940 learned_addr learned my addr 172.21.15.33:0/3617704940 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:06.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.489+0000 7fe324cd7640 1 --2- 172.21.15.33:0/3617704940 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56243e639a30 0x56243e64b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.489+0000 7fe31ffff640 1 --2- 172.21.15.33:0/3617704940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56243e63a350 0x56243e63c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.489+0000 7fe3254d8640 1 -- 172.21.15.33:0/3617704940 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56243e639a30 msgr2=0x56243e64b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:06.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.489+0000 7fe3254d8640 1 --2- 172.21.15.33:0/3617704940 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56243e639a30 0x56243e64b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.489+0000 7fe3254d8640 1 -- 172.21.15.33:0/3617704940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56243e63a350 msgr2=0x56243e63c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:06.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.489+0000 7fe3254d8640 1 --2- 172.21.15.33:0/3617704940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56243e63a350 0x56243e63c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.489+0000 7fe3254d8640 1 -- 172.21.15.33:0/3617704940 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56243e63d3c0 con 0x56243e642f30 2024-03-19T02:57:06.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.490+0000 7fe3254d8640 1 --2- 172.21.15.33:0/3617704940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56243e642f30 0x56243e63cbc0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fe318011b80 tx=0x7fe318004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11555c1db0d00c75 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:06.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.490+0000 7fe2fffff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:57:06.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.490+0000 7fe2fffff640 1 -- 172.21.15.33:0/3617704940 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31802e060 con 0x56243e642f30 2024-03-19T02:57:06.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.490+0000 7fe2fffff640 1 -- 172.21.15.33:0/3617704940 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fe318020450 con 0x56243e642f30 2024-03-19T02:57:06.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.490+0000 7fe329c03740 1 -- 172.21.15.33:0/3617704940 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56243e5a4d90 con 0x56243e642f30 2024-03-19T02:57:06.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.490+0000 7fe2fffff640 1 -- 172.21.15.33:0/3617704940 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe318035050 con 0x56243e642f30 2024-03-19T02:57:06.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.490+0000 7fe329c03740 10 client.16494 fetch_fsmap learned FSMap version 25 2024-03-19T02:57:06.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.490+0000 7fe329c03740 1 -- 172.21.15.33:0/3617704940 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56243e6bd860 con 0x56243e642f30 2024-03-19T02:57:06.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.490+0000 7fe2fffff640 1 -- 172.21.15.33:0/3617704940 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x56243e5a4d90 con 0x56243e642f30 2024-03-19T02:57:06.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.490+0000 7fe2fffff640 1 -- 172.21.15.33:0/3617704940 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fe31801d040 con 0x56243e642f30 2024-03-19T02:57:06.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.490+0000 7fe329c03740 10 client.16494 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:57:06.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.490+0000 7fe329c03740 10 client.16494 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:57:06.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.490+0000 7fe329c03740 20 client.16494 populate_metadata read hostname 'smithi033' 2024-03-19T02:57:06.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.490+0000 7fe329c03740 1 --2- 172.21.15.33:0/3617704940 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x56243e6be010 0x56243e6c0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:06.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.490+0000 7fe329c03740 4 client.16494 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:57:06.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.490+0000 7fe329c03740 1 -- 172.21.15.33:0/3617704940 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56243e522540 con 0x56243e6be010 2024-03-19T02:57:06.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.490+0000 7fe324cd7640 1 --2- 172.21.15.33:0/3617704940 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x56243e6be010 0x56243e6c0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:06.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.491+0000 7fe324cd7640 1 --2- 172.21.15.33:0/3617704940 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x56243e6be010 0x56243e6c0430 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4cc8ea17021ac6c server_cookie=6c69d96c80dd1650 in_seq=0 out_seq=0 2024-03-19T02:57:06.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.491+0000 7fe2fffff640 10 client.16494 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:57:06.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.491+0000 7fe2fffff640 1 -- 172.21.15.33:0/3617704940 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56243e522540 con 0x56243e6be010 2024-03-19T02:57:06.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.491+0000 7fe2fffff640 10 client.16494 handle_command_reply: tid=0 2024-03-19T02:57:06.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.491+0000 7fe329c03740 1 client.16494 shutdown 2024-03-19T02:57:06.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.491+0000 7fe329c03740 1 -- 172.21.15.33:0/3617704940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56243e642f30 msgr2=0x56243e63cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:06.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.491+0000 7fe329c03740 1 --2- 172.21.15.33:0/3617704940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56243e642f30 0x56243e63cbc0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fe318011b80 tx=0x7fe318004a40 comp rx=0 tx=0).stop 2024-03-19T02:57:06.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.491+0000 7fe329c03740 1 -- 172.21.15.33:0/3617704940 shutdown_connections 2024-03-19T02:57:06.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.491+0000 7fe329c03740 1 --2- 172.21.15.33:0/3617704940 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x56243e6be010 0x56243e6c0430 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.491+0000 7fe329c03740 1 --2- 172.21.15.33:0/3617704940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56243e642f30 0x56243e63cbc0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.491+0000 7fe329c03740 1 --2- 172.21.15.33:0/3617704940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56243e63a350 0x56243e63c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.491+0000 7fe2fffff640 0 client.16494 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:57:06.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.491+0000 7fe329c03740 1 --2- 172.21.15.33:0/3617704940 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56243e639a30 0x56243e64b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.491+0000 7fe329c03740 1 -- 172.21.15.33:0/3617704940 >> 172.21.15.33:0/3617704940 conn(0x56243e634160 msgr2=0x56243e6331e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:06.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.491+0000 7fe329c03740 1 -- 172.21.15.33:0/3617704940 shutdown_connections 2024-03-19T02:57:06.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.491+0000 7fe329c03740 1 -- 172.21.15.33:0/3617704940 wait complete. 2024-03-19T02:57:06.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.491+0000 7fe329c03740 20 client.16494 trim_cache size 0 max 16384 2024-03-19T02:57:06.500 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:57:06.500 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:57:06.500 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-03-19T02:57:06.500 INFO:teuthology.orchestra.run.smithi033.stdout: "f199ae72-870d-4e4a-b991-7afcf16af4ff": { 2024-03-19T02:57:06.500 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-03-19T02:57:06.500 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "f199ae72-870d-4e4a-b991-7afcf16af4ff", 2024-03-19T02:57:06.500 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-03-19T02:57:06.500 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T02:57:06.501 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T02:57:06.501 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:57:06.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.492+0000 7fe326f3b640 1 -- 172.21.15.33:0/3871981407 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe3100a29f0 msgr2=0x7fe3100a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:06.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.492+0000 7fe326f3b640 1 --2- 172.21.15.33:0/3871981407 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe3100a29f0 0x7fe3100a4eb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe318009b80 tx=0x7fe318009500 comp rx=0 tx=0).stop 2024-03-19T02:57:06.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.492+0000 7fe326f3b640 1 -- 172.21.15.33:0/3871981407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe320100740 msgr2=0x7fe320196f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:06.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.492+0000 7fe326f3b640 1 --2- 172.21.15.33:0/3871981407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe320100740 0x7fe320196f30 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fe31400eb00 tx=0x7fe31400ef00 comp rx=0 tx=0).stop 2024-03-19T02:57:06.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.492+0000 7fe326f3b640 1 -- 172.21.15.33:0/3871981407 shutdown_connections 2024-03-19T02:57:06.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.492+0000 7fe326f3b640 1 --2- 172.21.15.33:0/3871981407 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe3100a29f0 0x7fe3100a4eb0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.492+0000 7fe326f3b640 1 --2- 172.21.15.33:0/3871981407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe320106750 0x7fe32019b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.492+0000 7fe326f3b640 1 --2- 172.21.15.33:0/3871981407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe320101100 0x7fe320197470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.492+0000 7fe326f3b640 1 --2- 172.21.15.33:0/3871981407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe320100740 0x7fe320196f30 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:06.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.492+0000 7fe326f3b640 1 -- 172.21.15.33:0/3871981407 >> 172.21.15.33:0/3871981407 conn(0x7fe320077880 msgr2=0x7fe3201033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:06.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.492+0000 7fe326f3b640 1 -- 172.21.15.33:0/3871981407 shutdown_connections 2024-03-19T02:57:06.502 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:06.492+0000 7fe326f3b640 1 -- 172.21.15.33:0/3871981407 wait complete. 2024-03-19T02:57:06.504 INFO:tasks.cephfs.filesystem:scrub status for tag:f199ae72-870d-4e4a-b991-7afcf16af4ff - {'path': '/', 'tag': 'f199ae72-870d-4e4a-b991-7afcf16af4ff', 'options': 'recursive,force'} 2024-03-19T02:57:06.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:06 smithi155 ceph-mon[35804]: pgmap v112: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-19T02:57:07.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:07 smithi096 ceph-mon[35946]: scrub queued for path: / 2024-03-19T02:57:07.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:07 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T02:57:07.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:07 smithi096 ceph-mon[35946]: scrub summary: active paths [/] 2024-03-19T02:57:07.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:07 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T02:57:07.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:07 smithi155 ceph-mon[35804]: scrub queued for path: / 2024-03-19T02:57:07.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:07 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T02:57:07.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:07 smithi155 ceph-mon[35804]: scrub summary: active paths [/] 2024-03-19T02:57:07.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:07 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T02:57:07.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:07 smithi033 ceph-mon[31973]: scrub queued for path: / 2024-03-19T02:57:07.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:07 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T02:57:07.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:07 smithi033 ceph-mon[31973]: scrub summary: active paths [/] 2024-03-19T02:57:07.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:07 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T02:57:08.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:08 smithi096 ceph-mon[35946]: pgmap v113: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-19T02:57:08.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:08 smithi096 ceph-mon[35946]: scrub summary: idle 2024-03-19T02:57:08.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:08 smithi155 ceph-mon[35804]: pgmap v113: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-19T02:57:08.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:08 smithi155 ceph-mon[35804]: scrub summary: idle 2024-03-19T02:57:08.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:08 smithi033 ceph-mon[31973]: pgmap v113: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-19T02:57:08.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:08 smithi033 ceph-mon[31973]: scrub summary: idle 2024-03-19T02:57:10.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:10 smithi096 ceph-mon[35946]: pgmap v114: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-19T02:57:10.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:10 smithi155 ceph-mon[35804]: pgmap v114: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-19T02:57:10.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:10 smithi033 ceph-mon[31973]: pgmap v114: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-19T02:57:12.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:12 smithi096 ceph-mon[35946]: pgmap v115: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-19T02:57:12.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:12 smithi155 ceph-mon[35804]: pgmap v115: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-19T02:57:12.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:12 smithi033 ceph-mon[31973]: pgmap v115: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-19T02:57:14.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:14 smithi096 ceph-mon[35946]: pgmap v116: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-19T02:57:14.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:14 smithi155 ceph-mon[35804]: pgmap v116: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-19T02:57:14.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:14 smithi033 ceph-mon[31973]: pgmap v116: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-19T02:57:16.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:16 smithi096 ceph-mon[35946]: pgmap v117: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-19T02:57:16.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:16 smithi155 ceph-mon[35804]: pgmap v117: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-19T02:57:16.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:16 smithi033 ceph-mon[31973]: pgmap v117: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-19T02:57:18.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:18 smithi096 ceph-mon[35946]: pgmap v118: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-19T02:57:18.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:18 smithi155 ceph-mon[35804]: pgmap v118: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-19T02:57:18.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:18 smithi033 ceph-mon[31973]: pgmap v118: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-19T02:57:20.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:20 smithi096 ceph-mon[35946]: pgmap v119: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-19T02:57:20.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:20 smithi155 ceph-mon[35804]: pgmap v119: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-19T02:57:20.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:20 smithi033 ceph-mon[31973]: pgmap v119: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-19T02:57:22.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:22 smithi096 ceph-mon[35946]: pgmap v120: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-19T02:57:22.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:22 smithi155 ceph-mon[35804]: pgmap v120: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-19T02:57:22.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:22 smithi033 ceph-mon[31973]: pgmap v120: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-19T02:57:24.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:24 smithi096 ceph-mon[35946]: pgmap v121: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-03-19T02:57:24.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:24 smithi155 ceph-mon[35804]: pgmap v121: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-03-19T02:57:24.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:24 smithi033 ceph-mon[31973]: pgmap v121: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-03-19T02:57:26.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:26 smithi096 ceph-mon[35946]: pgmap v122: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:57:26.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:26 smithi155 ceph-mon[35804]: pgmap v122: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:57:26.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:26 smithi033 ceph-mon[31973]: pgmap v122: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:57:28.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:28 smithi096 ceph-mon[35946]: pgmap v123: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:57:28.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:28 smithi155 ceph-mon[35804]: pgmap v123: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:57:28.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:28 smithi033 ceph-mon[31973]: pgmap v123: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:57:30.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:30 smithi096 ceph-mon[35946]: pgmap v124: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:57:30.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:30 smithi155 ceph-mon[35804]: pgmap v124: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:57:30.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:30 smithi033 ceph-mon[31973]: pgmap v124: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:57:32.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:32 smithi096 ceph-mon[35946]: pgmap v125: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:57:32.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:32 smithi155 ceph-mon[35804]: pgmap v125: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:57:32.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:32 smithi033 ceph-mon[31973]: pgmap v125: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:57:34.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:34 smithi096 ceph-mon[35946]: pgmap v126: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:57:34.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:34 smithi155 ceph-mon[35804]: pgmap v126: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:57:34.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:34 smithi033 ceph-mon[31973]: pgmap v126: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:57:36.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:36 smithi096 ceph-mon[35946]: pgmap v127: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:57:36.465 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:36 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:57:36.505 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:57:36.505 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:57:36.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:36 smithi155 ceph-mon[35804]: pgmap v127: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:57:36.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:36 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:57:36.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:36 smithi033 ceph-mon[31973]: pgmap v127: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:57:36.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:36 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:57:36.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.609+0000 7fc74eeb2640 1 -- 172.21.15.33:0/1375602555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7480693e0 msgr2=0x7fc74810e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:36.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.609+0000 7fc74eeb2640 1 --2- 172.21.15.33:0/1375602555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7480693e0 0x7fc74810e680 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fc740009cd0 tx=0x7fc74002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.609+0000 7fc74eeb2640 1 -- 172.21.15.33:0/1375602555 shutdown_connections 2024-03-19T02:57:36.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.609+0000 7fc74eeb2640 1 --2- 172.21.15.33:0/1375602555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7480693e0 0x7fc74810e680 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.609+0000 7fc74eeb2640 1 --2- 172.21.15.33:0/1375602555 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc748068a20 0x7fc748068ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.609+0000 7fc74eeb2640 1 --2- 172.21.15.33:0/1375602555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc748107870 0x7fc748107c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.609+0000 7fc74eeb2640 1 -- 172.21.15.33:0/1375602555 >> 172.21.15.33:0/1375602555 conn(0x7fc748077820 msgr2=0x7fc748077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:36.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.609+0000 7fc74eeb2640 1 -- 172.21.15.33:0/1375602555 shutdown_connections 2024-03-19T02:57:36.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.609+0000 7fc74eeb2640 1 -- 172.21.15.33:0/1375602555 wait complete. 2024-03-19T02:57:36.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.610+0000 7fc74eeb2640 1 Processor -- start 2024-03-19T02:57:36.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.610+0000 7fc74eeb2640 1 -- start start 2024-03-19T02:57:36.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.610+0000 7fc74eeb2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc748068a20 0x7fc748197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:36.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.610+0000 7fc74eeb2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7480693e0 0x7fc748198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:36.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.610+0000 7fc74eeb2640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc748107870 0x7fc74819c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:36.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.610+0000 7fc74eeb2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc748111a70 con 0x7fc7480693e0 2024-03-19T02:57:36.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.610+0000 7fc74eeb2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc7481118f0 con 0x7fc748068a20 2024-03-19T02:57:36.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.610+0000 7fc74eeb2640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc748111bf0 con 0x7fc748107870 2024-03-19T02:57:36.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.610+0000 7fc747fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7480693e0 0x7fc748198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:36.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.610+0000 7fc747fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7480693e0 0x7fc748198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55926/0 (socket says 172.21.15.33:55926) 2024-03-19T02:57:36.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.610+0000 7fc747fff640 1 -- 172.21.15.33:0/1453415202 learned_addr learned my addr 172.21.15.33:0/1453415202 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:36.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.610+0000 7fc74d44f640 1 --2- 172.21.15.33:0/1453415202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc748107870 0x7fc74819c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:36.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.610+0000 7fc74cc4e640 1 --2- 172.21.15.33:0/1453415202 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc748068a20 0x7fc748197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:36.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.611+0000 7fc747fff640 1 -- 172.21.15.33:0/1453415202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc748107870 msgr2=0x7fc74819c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:36.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.611+0000 7fc747fff640 1 --2- 172.21.15.33:0/1453415202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc748107870 0x7fc74819c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.611+0000 7fc747fff640 1 -- 172.21.15.33:0/1453415202 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc748068a20 msgr2=0x7fc748197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:36.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.611+0000 7fc747fff640 1 --2- 172.21.15.33:0/1453415202 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc748068a20 0x7fc748197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.611+0000 7fc747fff640 1 -- 172.21.15.33:0/1453415202 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc74819cf40 con 0x7fc7480693e0 2024-03-19T02:57:36.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.611+0000 7fc747fff640 1 --2- 172.21.15.33:0/1453415202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7480693e0 0x7fc748198490 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fc73c00ea60 tx=0x7fc73c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:36.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.611+0000 7fc745ffb640 1 -- 172.21.15.33:0/1453415202 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73c00cde0 con 0x7fc7480693e0 2024-03-19T02:57:36.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.611+0000 7fc745ffb640 1 -- 172.21.15.33:0/1453415202 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc73c0035f0 con 0x7fc7480693e0 2024-03-19T02:57:36.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.611+0000 7fc74eeb2640 1 -- 172.21.15.33:0/1453415202 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc74819d230 con 0x7fc7480693e0 2024-03-19T02:57:36.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.611+0000 7fc745ffb640 1 -- 172.21.15.33:0/1453415202 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73c010690 con 0x7fc7480693e0 2024-03-19T02:57:36.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.611+0000 7fc74eeb2640 1 -- 172.21.15.33:0/1453415202 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7481a4ac0 con 0x7fc7480693e0 2024-03-19T02:57:36.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.613+0000 7fc745ffb640 1 -- 172.21.15.33:0/1453415202 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fc73c0037a0 con 0x7fc7480693e0 2024-03-19T02:57:36.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.614+0000 7fc751b7e740 1 Processor -- start 2024-03-19T02:57:36.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.614+0000 7fc745ffb640 1 --2- 172.21.15.33:0/1453415202 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fc7300a2970 0x7fc7300a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:36.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.614+0000 7fc751b7e740 1 -- start start 2024-03-19T02:57:36.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.614+0000 7fc745ffb640 1 -- 172.21.15.33:0/1453415202 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fc73c0cf450 con 0x7fc7480693e0 2024-03-19T02:57:36.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.614+0000 7fc74d44f640 1 --2- 172.21.15.33:0/1453415202 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fc7300a2970 0x7fc7300a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:36.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.614+0000 7fc751b7e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557787f47a30 0x557787f47e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:36.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.614+0000 7fc751b7e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557787f48350 0x557787f509f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:36.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.614+0000 7fc751b7e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557787f50f30 0x557787f53320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:36.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.614+0000 7fc751b7e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557787eab500 con 0x557787f47a30 2024-03-19T02:57:36.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.614+0000 7fc751b7e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557787e53be0 con 0x557787f50f30 2024-03-19T02:57:36.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.614+0000 7fc751b7e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x557787ea75a0 con 0x557787f48350 2024-03-19T02:57:36.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.614+0000 7fc74cc4e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557787f47a30 0x557787f47e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:36.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.614+0000 7fc74cc4e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557787f47a30 0x557787f47e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55928/0 (socket says 172.21.15.33:55928) 2024-03-19T02:57:36.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.614+0000 7fc74cc4e640 1 -- 172.21.15.33:0/1772453087 learned_addr learned my addr 172.21.15.33:0/1772453087 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:36.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.614+0000 7fc747fff640 1 --2- 172.21.15.33:0/1772453087 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557787f48350 0x557787f509f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:36.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.614+0000 7fc74d44f640 1 --2- 172.21.15.33:0/1772453087 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557787f50f30 0x557787f53320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:36.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.614+0000 7fc74cc4e640 1 -- 172.21.15.33:0/1772453087 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557787f48350 msgr2=0x557787f509f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:36.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.614+0000 7fc74cc4e640 1 --2- 172.21.15.33:0/1772453087 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557787f48350 0x557787f509f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.614+0000 7fc74cc4e640 1 -- 172.21.15.33:0/1772453087 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557787f50f30 msgr2=0x557787f53320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:36.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.614+0000 7fc74cc4e640 1 --2- 172.21.15.33:0/1772453087 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557787f50f30 0x557787f53320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.614+0000 7fc74cc4e640 1 -- 172.21.15.33:0/1772453087 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557787e75000 con 0x557787f47a30 2024-03-19T02:57:36.623 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.614+0000 7fc747fff640 1 --2- 172.21.15.33:0/1772453087 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557787f48350 0x557787f509f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:36.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.614+0000 7fc74d44f640 1 --2- 172.21.15.33:0/1453415202 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fc7300a2970 0x7fc7300a4e30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc740005cc0 tx=0x7fc740009900 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:36.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.615+0000 7fc74cc4e640 1 --2- 172.21.15.33:0/1772453087 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557787f47a30 0x557787f47e10 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fc738009d20 tx=0x7fc738011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ede9fa3980bd532 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:36.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.615+0000 7fc72effd640 1 -- 172.21.15.33:0/1772453087 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73801e070 con 0x557787f47a30 2024-03-19T02:57:36.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.615+0000 7fc72effd640 1 -- 172.21.15.33:0/1772453087 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc738019ae0 con 0x557787f47a30 2024-03-19T02:57:36.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.615+0000 7fc72effd640 1 -- 172.21.15.33:0/1772453087 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73800ea80 con 0x557787f47a30 2024-03-19T02:57:36.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.615+0000 7fc751b7e740 1 -- 172.21.15.33:0/1772453087 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557787f47a30 msgr2=0x557787f47e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:36.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.615+0000 7fc751b7e740 1 --2- 172.21.15.33:0/1772453087 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557787f47a30 0x557787f47e10 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fc738009d20 tx=0x7fc738011810 comp rx=0 tx=0).stop 2024-03-19T02:57:36.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.615+0000 7fc751b7e740 1 -- 172.21.15.33:0/1772453087 shutdown_connections 2024-03-19T02:57:36.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.615+0000 7fc751b7e740 1 --2- 172.21.15.33:0/1772453087 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557787f48350 0x557787f509f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.615+0000 7fc751b7e740 1 --2- 172.21.15.33:0/1772453087 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557787f50f30 0x557787f53320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.615+0000 7fc751b7e740 1 --2- 172.21.15.33:0/1772453087 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557787f47a30 0x557787f47e10 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.615+0000 7fc751b7e740 1 -- 172.21.15.33:0/1772453087 >> 172.21.15.33:0/1772453087 conn(0x557787f42160 msgr2=0x557787f42540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:36.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.615+0000 7fc751b7e740 1 -- 172.21.15.33:0/1772453087 shutdown_connections 2024-03-19T02:57:36.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.615+0000 7fc751b7e740 1 -- 172.21.15.33:0/1772453087 wait complete. 2024-03-19T02:57:36.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.616+0000 7fc751b7e740 1 -- start start 2024-03-19T02:57:36.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.616+0000 7fc751b7e740 1 Processor -- start 2024-03-19T02:57:36.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.616+0000 7fc751b7e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557787f47a30 0x557787f4de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:36.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.616+0000 7fc751b7e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557787f48350 0x557787f4e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:36.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.616+0000 7fc751b7e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557787f50f30 0x557787f4e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:36.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.616+0000 7fc751b7e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557787f53d20 con 0x557787f50f30 2024-03-19T02:57:36.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.616+0000 7fc751b7e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557787e67160 con 0x557787f48350 2024-03-19T02:57:36.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.616+0000 7fc751b7e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x557787f53ea0 con 0x557787f47a30 2024-03-19T02:57:36.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.616+0000 7fc751b7e740 10 client.0 _ll_register_callbacks cb 0x557787ea3a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:57:36.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.616+0000 7fc74d44f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557787f50f30 0x557787f4e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:36.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.616+0000 7fc74d44f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557787f50f30 0x557787f4e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55934/0 (socket says 172.21.15.33:55934) 2024-03-19T02:57:36.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.616+0000 7fc74d44f640 1 -- 172.21.15.33:0/788138567 learned_addr learned my addr 172.21.15.33:0/788138567 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:36.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.616+0000 7fc747fff640 1 --2- 172.21.15.33:0/788138567 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557787f48350 0x557787f4e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:36.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.616+0000 7fc74cc4e640 1 --2- 172.21.15.33:0/788138567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557787f47a30 0x557787f4de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:36.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.617+0000 7fc74d44f640 1 -- 172.21.15.33:0/788138567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557787f47a30 msgr2=0x557787f4de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:36.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.617+0000 7fc74d44f640 1 --2- 172.21.15.33:0/788138567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557787f47a30 0x557787f4de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.617+0000 7fc74d44f640 1 -- 172.21.15.33:0/788138567 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557787f48350 msgr2=0x557787f4e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:36.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.617+0000 7fc74d44f640 1 --2- 172.21.15.33:0/788138567 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557787f48350 0x557787f4e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.617+0000 7fc74d44f640 1 -- 172.21.15.33:0/788138567 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557787c897d0 con 0x557787f50f30 2024-03-19T02:57:36.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.617+0000 7fc747fff640 1 --2- 172.21.15.33:0/788138567 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557787f48350 0x557787f4e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:36.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.617+0000 7fc74d44f640 1 --2- 172.21.15.33:0/788138567 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557787f50f30 0x557787f4e8a0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fc740036000 tx=0x7fc74002c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edfc17154ab272f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:36.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.617+0000 7fc727fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:57:36.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.617+0000 7fc727fff640 1 -- 172.21.15.33:0/788138567 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc740039040 con 0x557787f50f30 2024-03-19T02:57:36.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.617+0000 7fc727fff640 1 -- 172.21.15.33:0/788138567 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fc740003d80 con 0x557787f50f30 2024-03-19T02:57:36.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.617+0000 7fc751b7e740 1 -- 172.21.15.33:0/788138567 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc738019ae0 con 0x557787f50f30 2024-03-19T02:57:36.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.617+0000 7fc727fff640 1 -- 172.21.15.33:0/788138567 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc740059050 con 0x557787f50f30 2024-03-19T02:57:36.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.617+0000 7fc751b7e740 10 client.16506 fetch_fsmap learned FSMap version 25 2024-03-19T02:57:36.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.617+0000 7fc751b7e740 1 -- 172.21.15.33:0/788138567 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x557787bbf3f0 con 0x557787f50f30 2024-03-19T02:57:36.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.617+0000 7fc727fff640 1 -- 172.21.15.33:0/788138567 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc738019ae0 con 0x557787f50f30 2024-03-19T02:57:36.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.617+0000 7fc727fff640 1 -- 172.21.15.33:0/788138567 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc74003f040 con 0x557787f50f30 2024-03-19T02:57:36.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.618+0000 7fc751b7e740 10 client.16506 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:57:36.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.618+0000 7fc751b7e740 10 client.16506 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:57:36.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.618+0000 7fc751b7e740 20 client.16506 populate_metadata read hostname 'smithi033' 2024-03-19T02:57:36.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.618+0000 7fc751b7e740 1 --2- 172.21.15.33:0/788138567 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557787f611b0 0x557787f635c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:36.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.618+0000 7fc751b7e740 4 client.16506 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:57:36.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.618+0000 7fc751b7e740 1 -- 172.21.15.33:0/788138567 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557787eb8a40 con 0x557787f611b0 2024-03-19T02:57:36.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.618+0000 7fc74cc4e640 1 --2- 172.21.15.33:0/788138567 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557787f611b0 0x557787f635c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:36.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.618+0000 7fc74cc4e640 1 --2- 172.21.15.33:0/788138567 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557787f611b0 0x557787f635c0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49814bb6691f9840 server_cookie=2413d7e49536fddb in_seq=0 out_seq=0 2024-03-19T02:57:36.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.618+0000 7fc727fff640 10 client.16506 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:57:36.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.619+0000 7fc727fff640 1 -- 172.21.15.33:0/788138567 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557787eb8a40 con 0x557787f611b0 2024-03-19T02:57:36.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.619+0000 7fc727fff640 10 client.16506 handle_command_reply: tid=0 2024-03-19T02:57:36.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.619+0000 7fc751b7e740 1 client.16506 shutdown 2024-03-19T02:57:36.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.619+0000 7fc751b7e740 1 -- 172.21.15.33:0/788138567 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557787f50f30 msgr2=0x557787f4e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:36.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.619+0000 7fc751b7e740 1 --2- 172.21.15.33:0/788138567 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557787f50f30 0x557787f4e8a0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fc740036000 tx=0x7fc74002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.619+0000 7fc751b7e740 1 -- 172.21.15.33:0/788138567 shutdown_connections 2024-03-19T02:57:36.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.619+0000 7fc751b7e740 1 --2- 172.21.15.33:0/788138567 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557787f611b0 0x557787f635c0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.619+0000 7fc751b7e740 1 --2- 172.21.15.33:0/788138567 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557787f50f30 0x557787f4e8a0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.619+0000 7fc727fff640 0 client.16506 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:57:36.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.619+0000 7fc751b7e740 1 --2- 172.21.15.33:0/788138567 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557787f48350 0x557787f4e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.619+0000 7fc751b7e740 1 --2- 172.21.15.33:0/788138567 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557787f47a30 0x557787f4de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.619+0000 7fc751b7e740 1 -- 172.21.15.33:0/788138567 >> 172.21.15.33:0/788138567 conn(0x557787f42160 msgr2=0x557787f43eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:36.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.619+0000 7fc751b7e740 1 -- 172.21.15.33:0/788138567 shutdown_connections 2024-03-19T02:57:36.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.619+0000 7fc751b7e740 1 -- 172.21.15.33:0/788138567 wait complete. 2024-03-19T02:57:36.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.619+0000 7fc751b7e740 20 client.16506 trim_cache size 0 max 16384 2024-03-19T02:57:36.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.636+0000 7fc751b7e740 1 Processor -- start 2024-03-19T02:57:36.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.636+0000 7fc751b7e740 1 -- start start 2024-03-19T02:57:36.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.636+0000 7fc751b7e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557787f47a30 0x557787f50550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:36.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.636+0000 7fc751b7e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557787f48350 0x557787f4cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:36.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.636+0000 7fc751b7e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557787f50f30 0x557787f49170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:36.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.636+0000 7fc751b7e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc7481b1470 con 0x557787f47a30 2024-03-19T02:57:36.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.636+0000 7fc751b7e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc7481aed40 con 0x557787f48350 2024-03-19T02:57:36.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.636+0000 7fc751b7e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc7481acb70 con 0x557787f50f30 2024-03-19T02:57:36.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.637+0000 7fc74cc4e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557787f47a30 0x557787f50550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:36.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.637+0000 7fc74cc4e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557787f47a30 0x557787f50550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55936/0 (socket says 172.21.15.33:55936) 2024-03-19T02:57:36.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.637+0000 7fc74cc4e640 1 -- 172.21.15.33:0/636328532 learned_addr learned my addr 172.21.15.33:0/636328532 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:36.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.637+0000 7fc747fff640 1 --2- 172.21.15.33:0/636328532 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557787f48350 0x557787f4cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:36.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.637+0000 7fc74d44f640 1 --2- 172.21.15.33:0/636328532 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557787f50f30 0x557787f49170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:36.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.637+0000 7fc74cc4e640 1 -- 172.21.15.33:0/636328532 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557787f50f30 msgr2=0x557787f49170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:36.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.637+0000 7fc74cc4e640 1 --2- 172.21.15.33:0/636328532 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557787f50f30 0x557787f49170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.637+0000 7fc74cc4e640 1 -- 172.21.15.33:0/636328532 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557787f48350 msgr2=0x557787f4cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:36.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.637+0000 7fc74cc4e640 1 --2- 172.21.15.33:0/636328532 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557787f48350 0x557787f4cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.637+0000 7fc74cc4e640 1 -- 172.21.15.33:0/636328532 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557787b7f0f0 con 0x557787f47a30 2024-03-19T02:57:36.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.637+0000 7fc74cc4e640 1 --2- 172.21.15.33:0/636328532 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557787f47a30 0x557787f50550 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fc738011d10 tx=0x7fc7380097c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbe97d606caa010c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:36.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.637+0000 7fc72e7fc640 1 -- 172.21.15.33:0/636328532 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73802f070 con 0x557787f47a30 2024-03-19T02:57:36.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.637+0000 7fc72e7fc640 1 -- 172.21.15.33:0/636328532 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc73801f040 con 0x557787f47a30 2024-03-19T02:57:36.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.637+0000 7fc72e7fc640 1 -- 172.21.15.33:0/636328532 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73801e040 con 0x557787f47a30 2024-03-19T02:57:36.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.638+0000 7fc751b7e740 1 -- 172.21.15.33:0/636328532 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557787f47a30 msgr2=0x557787f50550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:36.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.638+0000 7fc751b7e740 1 --2- 172.21.15.33:0/636328532 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557787f47a30 0x557787f50550 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fc738011d10 tx=0x7fc7380097c0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.638+0000 7fc751b7e740 1 -- 172.21.15.33:0/636328532 shutdown_connections 2024-03-19T02:57:36.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.638+0000 7fc751b7e740 1 --2- 172.21.15.33:0/636328532 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557787f50f30 0x557787f49170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.638+0000 7fc751b7e740 1 --2- 172.21.15.33:0/636328532 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557787f48350 0x557787f4cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.638+0000 7fc751b7e740 1 --2- 172.21.15.33:0/636328532 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557787f47a30 0x557787f50550 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.638+0000 7fc751b7e740 1 -- 172.21.15.33:0/636328532 >> 172.21.15.33:0/636328532 conn(0x557787f42160 msgr2=0x557787f4e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:36.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.638+0000 7fc751b7e740 1 -- 172.21.15.33:0/636328532 shutdown_connections 2024-03-19T02:57:36.644 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.638+0000 7fc751b7e740 1 -- 172.21.15.33:0/636328532 wait complete. 2024-03-19T02:57:36.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.638+0000 7fc751b7e740 1 -- start start 2024-03-19T02:57:36.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.638+0000 7fc751b7e740 1 Processor -- start 2024-03-19T02:57:36.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.638+0000 7fc751b7e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557787f47a30 0x557787f4a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:36.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.638+0000 7fc751b7e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557787f48350 0x557787f4ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:36.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.638+0000 7fc751b7e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557787f50f30 0x557787f4b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:36.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.639+0000 7fc751b7e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc7481ad200 con 0x557787f47a30 2024-03-19T02:57:36.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.639+0000 7fc751b7e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc7481ae670 con 0x557787f48350 2024-03-19T02:57:36.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.639+0000 7fc751b7e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc7481ad8d0 con 0x557787f50f30 2024-03-19T02:57:36.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.639+0000 7fc751b7e740 10 client.0 _ll_register_callbacks cb 0x7fc740045030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:57:36.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.639+0000 7fc74cc4e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557787f47a30 0x557787f4a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:36.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.639+0000 7fc74cc4e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557787f47a30 0x557787f4a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55952/0 (socket says 172.21.15.33:55952) 2024-03-19T02:57:36.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.639+0000 7fc74cc4e640 1 -- 172.21.15.33:0/331818995 learned_addr learned my addr 172.21.15.33:0/331818995 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:36.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.639+0000 7fc747fff640 1 --2- 172.21.15.33:0/331818995 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557787f48350 0x557787f4ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:36.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.639+0000 7fc74d44f640 1 --2- 172.21.15.33:0/331818995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557787f50f30 0x557787f4b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:36.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.639+0000 7fc74cc4e640 1 -- 172.21.15.33:0/331818995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557787f50f30 msgr2=0x557787f4b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:36.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.639+0000 7fc74cc4e640 1 --2- 172.21.15.33:0/331818995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557787f50f30 0x557787f4b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.639+0000 7fc74cc4e640 1 -- 172.21.15.33:0/331818995 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557787f48350 msgr2=0x557787f4ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:36.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.639+0000 7fc74cc4e640 1 --2- 172.21.15.33:0/331818995 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557787f48350 0x557787f4ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.639+0000 7fc74cc4e640 1 -- 172.21.15.33:0/331818995 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557787fc64c0 con 0x557787f47a30 2024-03-19T02:57:36.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.639+0000 7fc74cc4e640 1 --2- 172.21.15.33:0/331818995 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557787f47a30 0x557787f4a910 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fc73800a010 tx=0x7fc738005db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a396531f3613999 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:36.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.639+0000 7fc727fff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:57:36.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.639+0000 7fc727fff640 1 -- 172.21.15.33:0/331818995 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc738037020 con 0x557787f47a30 2024-03-19T02:57:36.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.639+0000 7fc727fff640 1 -- 172.21.15.33:0/331818995 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fc7380054a0 con 0x557787f47a30 2024-03-19T02:57:36.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.639+0000 7fc751b7e740 1 -- 172.21.15.33:0/331818995 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc73801f040 con 0x557787f47a30 2024-03-19T02:57:36.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.639+0000 7fc727fff640 1 -- 172.21.15.33:0/331818995 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc73801e040 con 0x557787f47a30 2024-03-19T02:57:36.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.639+0000 7fc751b7e740 10 client.16512 fetch_fsmap learned FSMap version 25 2024-03-19T02:57:36.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.639+0000 7fc751b7e740 1 -- 172.21.15.33:0/331818995 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x557787fc76d0 con 0x557787f47a30 2024-03-19T02:57:36.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.640+0000 7fc727fff640 1 -- 172.21.15.33:0/331818995 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc73801f040 con 0x557787f47a30 2024-03-19T02:57:36.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.640+0000 7fc727fff640 1 -- 172.21.15.33:0/331818995 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc738025040 con 0x557787f47a30 2024-03-19T02:57:36.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.640+0000 7fc751b7e740 10 client.16512 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:57:36.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.640+0000 7fc751b7e740 10 client.16512 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:57:36.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.640+0000 7fc751b7e740 20 client.16512 populate_metadata read hostname 'smithi033' 2024-03-19T02:57:36.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.640+0000 7fc751b7e740 1 --2- 172.21.15.33:0/331818995 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557787fc7cd0 0x557787fca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:36.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.640+0000 7fc751b7e740 4 client.16512 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:57:36.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.640+0000 7fc751b7e740 1 -- 172.21.15.33:0/331818995 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557787eb2d90 con 0x557787fc7cd0 2024-03-19T02:57:36.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.640+0000 7fc747fff640 1 --2- 172.21.15.33:0/331818995 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557787fc7cd0 0x557787fca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:36.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.640+0000 7fc747fff640 1 --2- 172.21.15.33:0/331818995 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557787fc7cd0 0x557787fca0f0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cc1a22185f1b9fc server_cookie=2a315a8af399fbc2 in_seq=0 out_seq=0 2024-03-19T02:57:36.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.640+0000 7fc727fff640 10 client.16512 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:57:36.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.640+0000 7fc727fff640 1 -- 172.21.15.33:0/331818995 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557787eb2d90 con 0x557787fc7cd0 2024-03-19T02:57:36.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.640+0000 7fc727fff640 10 client.16512 handle_command_reply: tid=0 2024-03-19T02:57:36.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.640+0000 7fc751b7e740 1 client.16512 shutdown 2024-03-19T02:57:36.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.640+0000 7fc751b7e740 1 -- 172.21.15.33:0/331818995 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557787f47a30 msgr2=0x557787f4a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:36.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.640+0000 7fc751b7e740 1 --2- 172.21.15.33:0/331818995 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557787f47a30 0x557787f4a910 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fc73800a010 tx=0x7fc738005db0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.640+0000 7fc751b7e740 1 -- 172.21.15.33:0/331818995 shutdown_connections 2024-03-19T02:57:36.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.640+0000 7fc751b7e740 1 --2- 172.21.15.33:0/331818995 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557787fc7cd0 0x557787fca0f0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.640+0000 7fc751b7e740 1 --2- 172.21.15.33:0/331818995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557787f50f30 0x557787f4b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.640+0000 7fc727fff640 0 client.16512 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:57:36.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.640+0000 7fc751b7e740 1 --2- 172.21.15.33:0/331818995 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557787f48350 0x557787f4ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.640+0000 7fc751b7e740 1 --2- 172.21.15.33:0/331818995 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557787f47a30 0x557787f4a910 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.640+0000 7fc751b7e740 1 -- 172.21.15.33:0/331818995 >> 172.21.15.33:0/331818995 conn(0x557787f42160 msgr2=0x557787f40950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:36.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.641+0000 7fc751b7e740 1 -- 172.21.15.33:0/331818995 shutdown_connections 2024-03-19T02:57:36.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.641+0000 7fc751b7e740 1 -- 172.21.15.33:0/331818995 wait complete. 2024-03-19T02:57:36.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.641+0000 7fc751b7e740 20 client.16512 trim_cache size 0 max 16384 2024-03-19T02:57:36.648 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:57:36.648 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-03-19T02:57:36.649 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-03-19T02:57:36.649 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:57:36.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.641+0000 7fc74eeb2640 1 -- 172.21.15.33:0/1453415202 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fc7300a2970 msgr2=0x7fc7300a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:36.649 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.641+0000 7fc74eeb2640 1 --2- 172.21.15.33:0/1453415202 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fc7300a2970 0x7fc7300a4e30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc740005cc0 tx=0x7fc740009900 comp rx=0 tx=0).stop 2024-03-19T02:57:36.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.641+0000 7fc74eeb2640 1 -- 172.21.15.33:0/1453415202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7480693e0 msgr2=0x7fc748198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:36.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.641+0000 7fc74eeb2640 1 --2- 172.21.15.33:0/1453415202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7480693e0 0x7fc748198490 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fc73c00ea60 tx=0x7fc73c00ef20 comp rx=0 tx=0).stop 2024-03-19T02:57:36.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.641+0000 7fc74eeb2640 1 -- 172.21.15.33:0/1453415202 shutdown_connections 2024-03-19T02:57:36.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.641+0000 7fc74eeb2640 1 --2- 172.21.15.33:0/1453415202 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fc7300a2970 0x7fc7300a4e30 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.641+0000 7fc74eeb2640 1 --2- 172.21.15.33:0/1453415202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc748107870 0x7fc74819c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.641+0000 7fc74eeb2640 1 --2- 172.21.15.33:0/1453415202 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc7480693e0 0x7fc748198490 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.641+0000 7fc74eeb2640 1 --2- 172.21.15.33:0/1453415202 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc748068a20 0x7fc748197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.641+0000 7fc74eeb2640 1 -- 172.21.15.33:0/1453415202 >> 172.21.15.33:0/1453415202 conn(0x7fc748077820 msgr2=0x7fc7480fe2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:36.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.641+0000 7fc74eeb2640 1 -- 172.21.15.33:0/1453415202 shutdown_connections 2024-03-19T02:57:36.650 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.641+0000 7fc74eeb2640 1 -- 172.21.15.33:0/1453415202 wait complete. 2024-03-19T02:57:36.653 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:57:36.653 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:57:36.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.796+0000 7ff95d31d640 1 -- 172.21.15.33:0/1769857841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff9581033d0 msgr2=0x7ff95810b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:36.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.796+0000 7ff95d31d640 1 --2- 172.21.15.33:0/1769857841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff9581033d0 0x7ff95810b8b0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7ff94c009900 tx=0x7ff94c02bcd0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.796+0000 7ff95d31d640 1 -- 172.21.15.33:0/1769857841 shutdown_connections 2024-03-19T02:57:36.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.796+0000 7ff95d31d640 1 --2- 172.21.15.33:0/1769857841 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff95810be80 0x7ff95810e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.796+0000 7ff95d31d640 1 --2- 172.21.15.33:0/1769857841 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff9581033d0 0x7ff95810b8b0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.796+0000 7ff95d31d640 1 --2- 172.21.15.33:0/1769857841 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff958102ab0 0x7ff958102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.796+0000 7ff95d31d640 1 -- 172.21.15.33:0/1769857841 >> 172.21.15.33:0/1769857841 conn(0x7ff958077880 msgr2=0x7ff958077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:36.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.796+0000 7ff95d31d640 1 -- 172.21.15.33:0/1769857841 shutdown_connections 2024-03-19T02:57:36.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.796+0000 7ff95d31d640 1 -- 172.21.15.33:0/1769857841 wait complete. 2024-03-19T02:57:36.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.797+0000 7ff95d31d640 1 Processor -- start 2024-03-19T02:57:36.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.797+0000 7ff95d31d640 1 -- start start 2024-03-19T02:57:36.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.797+0000 7ff95d31d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff958102ab0 0x7ff958100700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:36.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.797+0000 7ff95d31d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff9581033d0 0x7ff958100c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:36.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.797+0000 7ff95d31d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff95810be80 0x7ff958101180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:36.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.797+0000 7ff95d31d640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff958111aa0 con 0x7ff95810be80 2024-03-19T02:57:36.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.797+0000 7ff95d31d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff958111920 con 0x7ff958102ab0 2024-03-19T02:57:36.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.797+0000 7ff95d31d640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff958111c20 con 0x7ff9581033d0 2024-03-19T02:57:36.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.797+0000 7ff9577fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff95810be80 0x7ff958101180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:36.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.797+0000 7ff956ffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff958102ab0 0x7ff958100700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:36.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.797+0000 7ff9577fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff95810be80 0x7ff958101180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55974/0 (socket says 172.21.15.33:55974) 2024-03-19T02:57:36.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.797+0000 7ff956ffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff958102ab0 0x7ff958100700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:33302/0 (socket says 172.21.15.33:33302) 2024-03-19T02:57:36.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.797+0000 7ff9577fe640 1 -- 172.21.15.33:0/3505031901 learned_addr learned my addr 172.21.15.33:0/3505031901 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:36.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.797+0000 7ff9567fc640 1 --2- 172.21.15.33:0/3505031901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff9581033d0 0x7ff958100c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:36.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.798+0000 7ff9577fe640 1 -- 172.21.15.33:0/3505031901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff9581033d0 msgr2=0x7ff958100c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:36.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.798+0000 7ff9577fe640 1 --2- 172.21.15.33:0/3505031901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff9581033d0 0x7ff958100c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.798+0000 7ff9577fe640 1 -- 172.21.15.33:0/3505031901 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff958102ab0 msgr2=0x7ff958100700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:36.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.798+0000 7ff9577fe640 1 --2- 172.21.15.33:0/3505031901 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff958102ab0 0x7ff958100700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:36.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.798+0000 7ff9577fe640 1 -- 172.21.15.33:0/3505031901 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9581a6540 con 0x7ff95810be80 2024-03-19T02:57:36.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.798+0000 7ff9577fe640 1 --2- 172.21.15.33:0/3505031901 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff95810be80 0x7ff958101180 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7ff95000eb10 tx=0x7ff95000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:36.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.798+0000 7ff93ffff640 1 -- 172.21.15.33:0/3505031901 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff950004b20 con 0x7ff95810be80 2024-03-19T02:57:36.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.798+0000 7ff95d31d640 1 -- 172.21.15.33:0/3505031901 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9581a6830 con 0x7ff95810be80 2024-03-19T02:57:36.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.798+0000 7ff93ffff640 1 -- 172.21.15.33:0/3505031901 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff950004cc0 con 0x7ff95810be80 2024-03-19T02:57:36.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.798+0000 7ff93ffff640 1 -- 172.21.15.33:0/3505031901 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff950010690 con 0x7ff95810be80 2024-03-19T02:57:36.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.798+0000 7ff95d31d640 1 -- 172.21.15.33:0/3505031901 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9581a6d50 con 0x7ff95810be80 2024-03-19T02:57:36.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.799+0000 7ff95d31d640 1 -- 172.21.15.33:0/3505031901 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff958068a60 con 0x7ff95810be80 2024-03-19T02:57:36.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.800+0000 7ff93ffff640 1 -- 172.21.15.33:0/3505031901 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7ff950005000 con 0x7ff95810be80 2024-03-19T02:57:36.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.801+0000 7ff93ffff640 1 --2- 172.21.15.33:0/3505031901 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7ff9400a2970 0x7ff9400a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:36.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.801+0000 7ff956ffd640 1 --2- 172.21.15.33:0/3505031901 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7ff9400a2970 0x7ff9400a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:36.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.801+0000 7ff93ffff640 1 -- 172.21.15.33:0/3505031901 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7ff9500cf930 con 0x7ff95810be80 2024-03-19T02:57:36.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.801+0000 7ff956ffd640 1 --2- 172.21.15.33:0/3505031901 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7ff9400a2970 0x7ff9400a4e30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff948009bd0 tx=0x7ff948005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:36.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:36.805+0000 7ff93ffff640 1 -- 172.21.15.33:0/3505031901 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff950053740 con 0x7ff95810be80 2024-03-19T02:57:37.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.026+0000 7ff95d31d640 1 -- 172.21.15.33:0/3505031901 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff958106f70 con 0x7ff95810be80 2024-03-19T02:57:37.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.026+0000 7ff93ffff640 1 -- 172.21.15.33:0/3505031901 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11867 (secure 0 0 0) 0x7ff95009c2e0 con 0x7ff95810be80 2024-03-19T02:57:37.028 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:57:37.028 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:50:52.201798+0000","modified":"2024-03-19T02:53:11.748697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24488,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/3731232169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":3731232169},{"type":"v1","addr":"172.21.15.96:6839","nonce":3731232169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24457,24488,24509]},"id":1}]} 2024-03-19T02:57:37.028 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-03-19T02:57:37.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.028+0000 7ff95d31d640 1 -- 172.21.15.33:0/3505031901 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7ff9400a2970 msgr2=0x7ff9400a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:37.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.028+0000 7ff95d31d640 1 --2- 172.21.15.33:0/3505031901 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7ff9400a2970 0x7ff9400a4e30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff948009bd0 tx=0x7ff948005f90 comp rx=0 tx=0).stop 2024-03-19T02:57:37.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.028+0000 7ff95d31d640 1 -- 172.21.15.33:0/3505031901 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff95810be80 msgr2=0x7ff958101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:37.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.028+0000 7ff95d31d640 1 --2- 172.21.15.33:0/3505031901 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff95810be80 0x7ff958101180 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7ff95000eb10 tx=0x7ff95000c2d0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.028+0000 7ff95d31d640 1 -- 172.21.15.33:0/3505031901 shutdown_connections 2024-03-19T02:57:37.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.028+0000 7ff95d31d640 1 --2- 172.21.15.33:0/3505031901 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7ff9400a2970 0x7ff9400a4e30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.028+0000 7ff95d31d640 1 --2- 172.21.15.33:0/3505031901 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff9581033d0 0x7ff958100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.028+0000 7ff95d31d640 1 --2- 172.21.15.33:0/3505031901 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff95810be80 0x7ff958101180 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.029+0000 7ff95d31d640 1 --2- 172.21.15.33:0/3505031901 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff958102ab0 0x7ff958100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.029+0000 7ff95d31d640 1 -- 172.21.15.33:0/3505031901 >> 172.21.15.33:0/3505031901 conn(0x7ff958077880 msgr2=0x7ff9580fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:37.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.029+0000 7ff95d31d640 1 -- 172.21.15.33:0/3505031901 shutdown_connections 2024-03-19T02:57:37.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.029+0000 7ff95d31d640 1 -- 172.21.15.33:0/3505031901 wait complete. 2024-03-19T02:57:37.041 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:57:37.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.145+0000 7fcd548ae640 1 -- 172.21.15.33:0/1409995086 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd50102150 msgr2=0x7fcd5010e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:37.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.145+0000 7fcd548ae640 1 --2- 172.21.15.33:0/1409995086 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd50102150 0x7fcd5010e660 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fcd48009930 tx=0x7fcd4802bc30 comp rx=0 tx=0).stop 2024-03-19T02:57:37.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.145+0000 7fcd548ae640 1 -- 172.21.15.33:0/1409995086 shutdown_connections 2024-03-19T02:57:37.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.145+0000 7fcd548ae640 1 --2- 172.21.15.33:0/1409995086 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd50102150 0x7fcd5010e660 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.145+0000 7fcd548ae640 1 --2- 172.21.15.33:0/1409995086 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd50101790 0x7fcd50101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.145+0000 7fcd548ae640 1 --2- 172.21.15.33:0/1409995086 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd501077a0 0x7fcd50107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.145+0000 7fcd548ae640 1 -- 172.21.15.33:0/1409995086 >> 172.21.15.33:0/1409995086 conn(0x7fcd500fd2c0 msgr2=0x7fcd500ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:37.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.145+0000 7fcd548ae640 1 -- 172.21.15.33:0/1409995086 shutdown_connections 2024-03-19T02:57:37.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.145+0000 7fcd548ae640 1 -- 172.21.15.33:0/1409995086 wait complete. 2024-03-19T02:57:37.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.146+0000 7fcd548ae640 1 Processor -- start 2024-03-19T02:57:37.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.146+0000 7fcd548ae640 1 -- start start 2024-03-19T02:57:37.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.146+0000 7fcd548ae640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd50101790 0x7fcd50197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.146+0000 7fcd548ae640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd50102150 0x7fcd501984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.146+0000 7fcd548ae640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd501077a0 0x7fcd5019c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.146+0000 7fcd548ae640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcd50111a60 con 0x7fcd50101790 2024-03-19T02:57:37.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.146+0000 7fcd548ae640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcd501118e0 con 0x7fcd501077a0 2024-03-19T02:57:37.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.146+0000 7fcd548ae640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fcd50111be0 con 0x7fcd50102150 2024-03-19T02:57:37.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.146+0000 7fcd4e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd50101790 0x7fcd50197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.146+0000 7fcd4e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd50101790 0x7fcd50197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55996/0 (socket says 172.21.15.33:55996) 2024-03-19T02:57:37.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.146+0000 7fcd4e59c640 1 -- 172.21.15.33:0/3416844335 learned_addr learned my addr 172.21.15.33:0/3416844335 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:37.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.146+0000 7fcd4dd9b640 1 --2- 172.21.15.33:0/3416844335 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd50102150 0x7fcd501984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.146+0000 7fcd4ed9d640 1 --2- 172.21.15.33:0/3416844335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd501077a0 0x7fcd5019c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.146+0000 7fcd4e59c640 1 -- 172.21.15.33:0/3416844335 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd50102150 msgr2=0x7fcd501984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:37.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.146+0000 7fcd4e59c640 1 --2- 172.21.15.33:0/3416844335 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd50102150 0x7fcd501984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.146+0000 7fcd4e59c640 1 -- 172.21.15.33:0/3416844335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd501077a0 msgr2=0x7fcd5019c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:37.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.146+0000 7fcd4e59c640 1 --2- 172.21.15.33:0/3416844335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd501077a0 0x7fcd5019c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.146+0000 7fcd4e59c640 1 -- 172.21.15.33:0/3416844335 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd5019cf70 con 0x7fcd50101790 2024-03-19T02:57:37.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.147+0000 7fcd4e59c640 1 --2- 172.21.15.33:0/3416844335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd50101790 0x7fcd50197f80 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fcd4000eb90 tx=0x7fcd4000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:37.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.147+0000 7fcd477fe640 1 -- 172.21.15.33:0/3416844335 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd40004d20 con 0x7fcd50101790 2024-03-19T02:57:37.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.147+0000 7fcd548ae640 1 -- 172.21.15.33:0/3416844335 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd5019d260 con 0x7fcd50101790 2024-03-19T02:57:37.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.147+0000 7fcd477fe640 1 -- 172.21.15.33:0/3416844335 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcd4000ee90 con 0x7fcd50101790 2024-03-19T02:57:37.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.147+0000 7fcd477fe640 1 -- 172.21.15.33:0/3416844335 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd40010670 con 0x7fcd50101790 2024-03-19T02:57:37.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.147+0000 7fcd548ae640 1 -- 172.21.15.33:0/3416844335 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd501a4af0 con 0x7fcd50101790 2024-03-19T02:57:37.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.149+0000 7fcd477fe640 1 -- 172.21.15.33:0/3416844335 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fcd40005020 con 0x7fcd50101790 2024-03-19T02:57:37.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.149+0000 7fcd57570740 1 Processor -- start 2024-03-19T02:57:37.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.150+0000 7fcd57570740 1 -- start start 2024-03-19T02:57:37.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.150+0000 7fcd57570740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6b89592d0 0x55b6b89596b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.150+0000 7fcd57570740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b6b8959bf0 0x55b6b8962290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.150+0000 7fcd57570740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6b89627d0 0x55b6b8964bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.150+0000 7fcd57570740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b6b88bc4b0 con 0x55b6b89592d0 2024-03-19T02:57:37.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.150+0000 7fcd57570740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55b6b8864d70 con 0x55b6b89627d0 2024-03-19T02:57:37.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.150+0000 7fcd57570740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b6b88b8550 con 0x55b6b8959bf0 2024-03-19T02:57:37.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.150+0000 7fcd477fe640 1 --2- 172.21.15.33:0/3416844335 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fcd3c0a2970 0x7fcd3c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.150+0000 7fcd4ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6b89592d0 0x55b6b89596b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.150+0000 7fcd4ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6b89592d0 0x55b6b89596b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56002/0 (socket says 172.21.15.33:56002) 2024-03-19T02:57:37.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.150+0000 7fcd4ed9d640 1 -- 172.21.15.33:0/1189004263 learned_addr learned my addr 172.21.15.33:0/1189004263 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:37.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.150+0000 7fcd4dd9b640 1 --2- 172.21.15.33:0/1189004263 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6b89627d0 0x55b6b8964bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.150+0000 7fcd4e59c640 1 --2- 172.21.15.33:0/1189004263 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b6b8959bf0 0x55b6b8962290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.150+0000 7fcd4ed9d640 1 --2- 172.21.15.33:0/3416844335 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fcd3c0a2970 0x7fcd3c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.150+0000 7fcd477fe640 1 -- 172.21.15.33:0/3416844335 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fcd400cf750 con 0x7fcd50101790 2024-03-19T02:57:37.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.150+0000 7fcd4dd9b640 1 -- 172.21.15.33:0/1189004263 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b6b8959bf0 msgr2=0x55b6b8962290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:37.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.150+0000 7fcd4dd9b640 1 --2- 172.21.15.33:0/1189004263 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b6b8959bf0 0x55b6b8962290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.150+0000 7fcd4dd9b640 1 -- 172.21.15.33:0/1189004263 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6b89592d0 msgr2=0x55b6b89596b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:37.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.150+0000 7fcd4dd9b640 1 --2- 172.21.15.33:0/1189004263 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6b89592d0 0x55b6b89596b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.150+0000 7fcd4dd9b640 1 -- 172.21.15.33:0/1189004263 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6b8887210 con 0x55b6b89627d0 2024-03-19T02:57:37.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.150+0000 7fcd4ed9d640 1 --2- 172.21.15.33:0/3416844335 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fcd3c0a2970 0x7fcd3c0a4e30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcd48005ab0 tx=0x7fcd480058d0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:37.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.151+0000 7fcd4dd9b640 1 --2- 172.21.15.33:0/1189004263 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6b89627d0 0x55b6b8964bc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcd38009bf0 tx=0x7fcd38011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffccb000af3f3bd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:37.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.151+0000 7fcd44ff9640 1 -- 172.21.15.33:0/1189004263 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd38004310 con 0x55b6b89627d0 2024-03-19T02:57:37.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.151+0000 7fcd44ff9640 1 -- 172.21.15.33:0/1189004263 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcd380044b0 con 0x55b6b89627d0 2024-03-19T02:57:37.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.151+0000 7fcd44ff9640 1 -- 172.21.15.33:0/1189004263 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd380035f0 con 0x55b6b89627d0 2024-03-19T02:57:37.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.151+0000 7fcd57570740 1 -- 172.21.15.33:0/1189004263 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6b89627d0 msgr2=0x55b6b8964bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:37.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.151+0000 7fcd57570740 1 --2- 172.21.15.33:0/1189004263 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6b89627d0 0x55b6b8964bc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcd38009bf0 tx=0x7fcd38011440 comp rx=0 tx=0).stop 2024-03-19T02:57:37.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.151+0000 7fcd57570740 1 -- 172.21.15.33:0/1189004263 shutdown_connections 2024-03-19T02:57:37.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.152+0000 7fcd57570740 1 --2- 172.21.15.33:0/1189004263 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b6b8959bf0 0x55b6b8962290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.152+0000 7fcd57570740 1 --2- 172.21.15.33:0/1189004263 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6b89627d0 0x55b6b8964bc0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.152+0000 7fcd57570740 1 --2- 172.21.15.33:0/1189004263 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6b89592d0 0x55b6b89596b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.152+0000 7fcd57570740 1 -- 172.21.15.33:0/1189004263 >> 172.21.15.33:0/1189004263 conn(0x55b6b8953a00 msgr2=0x55b6b8953de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:37.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.152+0000 7fcd57570740 1 -- 172.21.15.33:0/1189004263 shutdown_connections 2024-03-19T02:57:37.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.152+0000 7fcd57570740 1 -- 172.21.15.33:0/1189004263 wait complete. 2024-03-19T02:57:37.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.152+0000 7fcd57570740 1 -- start start 2024-03-19T02:57:37.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.152+0000 7fcd57570740 1 Processor -- start 2024-03-19T02:57:37.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.152+0000 7fcd57570740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b6b89592d0 0x55b6b895f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.152+0000 7fcd57570740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6b8959bf0 0x55b6b895fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.152+0000 7fcd57570740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6b89627d0 0x55b6b8960200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.152+0000 7fcd57570740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b6b89655c0 con 0x55b6b89627d0 2024-03-19T02:57:37.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.152+0000 7fcd57570740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55b6b8878310 con 0x55b6b8959bf0 2024-03-19T02:57:37.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.152+0000 7fcd57570740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b6b8965740 con 0x55b6b89592d0 2024-03-19T02:57:37.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.152+0000 7fcd57570740 10 client.0 _ll_register_callbacks cb 0x55b6b886c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:57:37.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.152+0000 7fcd4ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6b89627d0 0x55b6b8960200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.152+0000 7fcd4dd9b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6b8959bf0 0x55b6b895fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.152+0000 7fcd4e59c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b6b89592d0 0x55b6b895f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.152+0000 7fcd4ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6b89627d0 0x55b6b8960200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56008/0 (socket says 172.21.15.33:56008) 2024-03-19T02:57:37.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.152+0000 7fcd4e59c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b6b89592d0 0x55b6b895f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:39034/0 (socket says 172.21.15.33:39034) 2024-03-19T02:57:37.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.152+0000 7fcd4ed9d640 1 -- 172.21.15.33:0/3606333293 learned_addr learned my addr 172.21.15.33:0/3606333293 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:37.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.152+0000 7fcd4dd9b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6b8959bf0 0x55b6b895fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:33338/0 (socket says 172.21.15.33:33338) 2024-03-19T02:57:37.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.153+0000 7fcd4ed9d640 1 -- 172.21.15.33:0/3606333293 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b6b89592d0 msgr2=0x55b6b895f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:37.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.153+0000 7fcd4ed9d640 1 --2- 172.21.15.33:0/3606333293 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b6b89592d0 0x55b6b895f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.153+0000 7fcd4ed9d640 1 -- 172.21.15.33:0/3606333293 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6b8959bf0 msgr2=0x55b6b895fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:37.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.153+0000 7fcd4ed9d640 1 --2- 172.21.15.33:0/3606333293 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6b8959bf0 0x55b6b895fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.153+0000 7fcd4ed9d640 1 -- 172.21.15.33:0/3606333293 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6b85d03f0 con 0x55b6b89627d0 2024-03-19T02:57:37.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.153+0000 7fcd4dd9b640 1 --2- 172.21.15.33:0/3606333293 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6b8959bf0 0x55b6b895fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:37.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.153+0000 7fcd4ed9d640 1 --2- 172.21.15.33:0/3606333293 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6b89627d0 0x55b6b8960200 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fcd48036040 tx=0x7fcd48009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a6cdb34b7a48d9d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:37.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.153+0000 7fcd35ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:57:37.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.157+0000 7fcd35ffb640 1 -- 172.21.15.33:0/3606333293 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd48053020 con 0x55b6b89627d0 2024-03-19T02:57:37.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.157+0000 7fcd35ffb640 1 -- 172.21.15.33:0/3606333293 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fcd480048d0 con 0x55b6b89627d0 2024-03-19T02:57:37.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.157+0000 7fcd57570740 1 -- 172.21.15.33:0/3606333293 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd380044b0 con 0x55b6b89627d0 2024-03-19T02:57:37.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.157+0000 7fcd35ffb640 1 -- 172.21.15.33:0/3606333293 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fcd4805e050 con 0x55b6b89627d0 2024-03-19T02:57:37.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.157+0000 7fcd57570740 10 client.16530 fetch_fsmap learned FSMap version 25 2024-03-19T02:57:37.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.157+0000 7fcd57570740 1 -- 172.21.15.33:0/3606333293 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b6b869a7d0 con 0x55b6b89627d0 2024-03-19T02:57:37.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.157+0000 7fcd35ffb640 1 -- 172.21.15.33:0/3606333293 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcd380044b0 con 0x55b6b89627d0 2024-03-19T02:57:37.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.157+0000 7fcd35ffb640 1 -- 172.21.15.33:0/3606333293 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fcd48003c20 con 0x55b6b89627d0 2024-03-19T02:57:37.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.157+0000 7fcd57570740 10 client.16530 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:57:37.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.157+0000 7fcd57570740 10 client.16530 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T02:57:37.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.157+0000 7fcd57570740 20 client.16530 populate_metadata read hostname 'smithi033' 2024-03-19T02:57:37.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.157+0000 7fcd57570740 1 --2- 172.21.15.33:0/3606333293 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55b6b8972b20 0x55b6b8974fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.157+0000 7fcd57570740 4 client.16530 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:57:37.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.157+0000 7fcd57570740 1 -- 172.21.15.33:0/3606333293 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6b891a3e0 con 0x55b6b8972b20 2024-03-19T02:57:37.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.158+0000 7fcd4e59c640 1 --2- 172.21.15.33:0/3606333293 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55b6b8972b20 0x55b6b8974fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.158+0000 7fcd4e59c640 1 --2- 172.21.15.33:0/3606333293 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55b6b8972b20 0x55b6b8974fa0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f62c01f0adbab150 server_cookie=4efac931a973d46b in_seq=0 out_seq=0 2024-03-19T02:57:37.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.158+0000 7fcd35ffb640 10 client.16530 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T02:57:37.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.159+0000 7fcd35ffb640 1 -- 172.21.15.33:0/3606333293 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b6b891a3e0 con 0x55b6b8972b20 2024-03-19T02:57:37.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.159+0000 7fcd35ffb640 10 client.16530 handle_command_reply: tid=0 2024-03-19T02:57:37.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.159+0000 7fcd57570740 1 client.16530 shutdown 2024-03-19T02:57:37.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.159+0000 7fcd57570740 1 -- 172.21.15.33:0/3606333293 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6b89627d0 msgr2=0x55b6b8960200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:37.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.159+0000 7fcd57570740 1 --2- 172.21.15.33:0/3606333293 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6b89627d0 0x55b6b8960200 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fcd48036040 tx=0x7fcd48009260 comp rx=0 tx=0).stop 2024-03-19T02:57:37.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.159+0000 7fcd57570740 1 -- 172.21.15.33:0/3606333293 shutdown_connections 2024-03-19T02:57:37.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.159+0000 7fcd57570740 1 --2- 172.21.15.33:0/3606333293 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55b6b8972b20 0x55b6b8974fa0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.159+0000 7fcd57570740 1 --2- 172.21.15.33:0/3606333293 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6b89627d0 0x55b6b8960200 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.159+0000 7fcd57570740 1 --2- 172.21.15.33:0/3606333293 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6b8959bf0 0x55b6b895fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.159+0000 7fcd35ffb640 0 client.16530 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T02:57:37.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.159+0000 7fcd57570740 1 --2- 172.21.15.33:0/3606333293 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b6b89592d0 0x55b6b895f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.159+0000 7fcd57570740 1 -- 172.21.15.33:0/3606333293 >> 172.21.15.33:0/3606333293 conn(0x55b6b8953a00 msgr2=0x55b6b8955750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:37.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.159+0000 7fcd57570740 1 -- 172.21.15.33:0/3606333293 shutdown_connections 2024-03-19T02:57:37.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.159+0000 7fcd57570740 1 -- 172.21.15.33:0/3606333293 wait complete. 2024-03-19T02:57:37.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.159+0000 7fcd57570740 20 client.16530 trim_cache size 0 max 16384 2024-03-19T02:57:37.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.172+0000 7fcd57570740 1 Processor -- start 2024-03-19T02:57:37.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.172+0000 7fcd57570740 1 -- start start 2024-03-19T02:57:37.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.172+0000 7fcd57570740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b6b89592d0 0x55b6b8961eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.172+0000 7fcd57570740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6b8959bf0 0x55b6b896e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.172+0000 7fcd57570740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6b89627d0 0x55b6b8952a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.172+0000 7fcd57570740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcd501b1450 con 0x55b6b89627d0 2024-03-19T02:57:37.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.172+0000 7fcd57570740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcd501aed20 con 0x55b6b8959bf0 2024-03-19T02:57:37.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.172+0000 7fcd57570740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fcd501acb50 con 0x55b6b89592d0 2024-03-19T02:57:37.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.172+0000 7fcd4ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6b89627d0 0x55b6b8952a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.179 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.173+0000 7fcd4ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6b89627d0 0x55b6b8952a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56012/0 (socket says 172.21.15.33:56012) 2024-03-19T02:57:37.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.173+0000 7fcd4ed9d640 1 -- 172.21.15.33:0/3817192729 learned_addr learned my addr 172.21.15.33:0/3817192729 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:37.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.173+0000 7fcd4dd9b640 1 --2- 172.21.15.33:0/3817192729 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6b8959bf0 0x55b6b896e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.173+0000 7fcd4e59c640 1 --2- 172.21.15.33:0/3817192729 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b6b89592d0 0x55b6b8961eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.173+0000 7fcd4ed9d640 1 -- 172.21.15.33:0/3817192729 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b6b89592d0 msgr2=0x55b6b8961eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:37.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.173+0000 7fcd4ed9d640 1 --2- 172.21.15.33:0/3817192729 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b6b89592d0 0x55b6b8961eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.173+0000 7fcd4ed9d640 1 -- 172.21.15.33:0/3817192729 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6b8959bf0 msgr2=0x55b6b896e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:37.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.173+0000 7fcd4ed9d640 1 --2- 172.21.15.33:0/3817192729 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6b8959bf0 0x55b6b896e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.173+0000 7fcd4ed9d640 1 -- 172.21.15.33:0/3817192729 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6b8953830 con 0x55b6b89627d0 2024-03-19T02:57:37.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.173+0000 7fcd4ed9d640 1 --2- 172.21.15.33:0/3817192729 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6b89627d0 0x55b6b8952a20 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fcd48009660 tx=0x7fcd4802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c14a77d17ada2dd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:37.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.173+0000 7fcd37fff640 1 -- 172.21.15.33:0/3817192729 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd48053040 con 0x55b6b89627d0 2024-03-19T02:57:37.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.173+0000 7fcd37fff640 1 -- 172.21.15.33:0/3817192729 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcd48067070 con 0x55b6b89627d0 2024-03-19T02:57:37.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.173+0000 7fcd37fff640 1 -- 172.21.15.33:0/3817192729 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd48004bc0 con 0x55b6b89627d0 2024-03-19T02:57:37.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.174+0000 7fcd57570740 1 -- 172.21.15.33:0/3817192729 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6b89627d0 msgr2=0x55b6b8952a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:37.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.174+0000 7fcd57570740 1 --2- 172.21.15.33:0/3817192729 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6b89627d0 0x55b6b8952a20 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fcd48009660 tx=0x7fcd4802bc30 comp rx=0 tx=0).stop 2024-03-19T02:57:37.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.174+0000 7fcd57570740 1 -- 172.21.15.33:0/3817192729 shutdown_connections 2024-03-19T02:57:37.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.174+0000 7fcd57570740 1 --2- 172.21.15.33:0/3817192729 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6b89627d0 0x55b6b8952a20 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.174+0000 7fcd57570740 1 --2- 172.21.15.33:0/3817192729 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6b8959bf0 0x55b6b896e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.174+0000 7fcd57570740 1 --2- 172.21.15.33:0/3817192729 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b6b89592d0 0x55b6b8961eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.174+0000 7fcd57570740 1 -- 172.21.15.33:0/3817192729 >> 172.21.15.33:0/3817192729 conn(0x55b6b8953a00 msgr2=0x55b6b8952310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:37.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.174+0000 7fcd57570740 1 -- 172.21.15.33:0/3817192729 shutdown_connections 2024-03-19T02:57:37.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.174+0000 7fcd57570740 1 -- 172.21.15.33:0/3817192729 wait complete. 2024-03-19T02:57:37.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.174+0000 7fcd57570740 1 -- start start 2024-03-19T02:57:37.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.174+0000 7fcd57570740 1 Processor -- start 2024-03-19T02:57:37.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.174+0000 7fcd57570740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b6b89592d0 0x55b6b895c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.174+0000 7fcd57570740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6b8959bf0 0x55b6b895c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.174+0000 7fcd57570740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6b89627d0 0x55b6b895cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.174+0000 7fcd57570740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcd501ad1e0 con 0x55b6b8959bf0 2024-03-19T02:57:37.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.174+0000 7fcd57570740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcd501ae650 con 0x55b6b89627d0 2024-03-19T02:57:37.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.174+0000 7fcd57570740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fcd501ad8b0 con 0x55b6b89592d0 2024-03-19T02:57:37.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.174+0000 7fcd57570740 10 client.0 _ll_register_callbacks cb 0x7fcd48009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:57:37.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.175+0000 7fcd4dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6b8959bf0 0x55b6b895c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.175+0000 7fcd4dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6b8959bf0 0x55b6b895c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56026/0 (socket says 172.21.15.33:56026) 2024-03-19T02:57:37.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.175+0000 7fcd4dd9b640 1 -- 172.21.15.33:0/1261315724 learned_addr learned my addr 172.21.15.33:0/1261315724 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:37.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.175+0000 7fcd4ed9d640 1 --2- 172.21.15.33:0/1261315724 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6b89627d0 0x55b6b895cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.175+0000 7fcd4e59c640 1 --2- 172.21.15.33:0/1261315724 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b6b89592d0 0x55b6b895c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.175+0000 7fcd4dd9b640 1 -- 172.21.15.33:0/1261315724 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b6b89592d0 msgr2=0x55b6b895c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:37.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.175+0000 7fcd4dd9b640 1 --2- 172.21.15.33:0/1261315724 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b6b89592d0 0x55b6b895c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.175+0000 7fcd4dd9b640 1 -- 172.21.15.33:0/1261315724 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6b89627d0 msgr2=0x55b6b895cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:37.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.175+0000 7fcd4dd9b640 1 --2- 172.21.15.33:0/1261315724 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6b89627d0 0x55b6b895cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.175+0000 7fcd4dd9b640 1 -- 172.21.15.33:0/1261315724 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6b89d7ea0 con 0x55b6b8959bf0 2024-03-19T02:57:37.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.175+0000 7fcd4dd9b640 1 --2- 172.21.15.33:0/1261315724 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6b8959bf0 0x55b6b895c730 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fcd38019000 tx=0x7fcd38004b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd3bc9c20b8a4d25 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:37.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.175+0000 7fcd35ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:57:37.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.175+0000 7fcd35ffb640 1 -- 172.21.15.33:0/1261315724 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd38028020 con 0x55b6b8959bf0 2024-03-19T02:57:37.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.175+0000 7fcd35ffb640 1 -- 172.21.15.33:0/1261315724 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fcd38003d00 con 0x55b6b8959bf0 2024-03-19T02:57:37.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.175+0000 7fcd57570740 1 -- 172.21.15.33:0/1261315724 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd48067070 con 0x55b6b8959bf0 2024-03-19T02:57:37.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.175+0000 7fcd35ffb640 1 -- 172.21.15.33:0/1261315724 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fcd38033050 con 0x55b6b8959bf0 2024-03-19T02:57:37.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.175+0000 7fcd57570740 10 client.16536 fetch_fsmap learned FSMap version 25 2024-03-19T02:57:37.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.175+0000 7fcd57570740 1 -- 172.21.15.33:0/1261315724 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b6b89d9080 con 0x55b6b8959bf0 2024-03-19T02:57:37.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.176+0000 7fcd35ffb640 1 -- 172.21.15.33:0/1261315724 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcd48067070 con 0x55b6b8959bf0 2024-03-19T02:57:37.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.176+0000 7fcd35ffb640 1 -- 172.21.15.33:0/1261315724 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fcd38018070 con 0x55b6b8959bf0 2024-03-19T02:57:37.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.176+0000 7fcd57570740 10 client.16536 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:57:37.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.176+0000 7fcd57570740 10 client.16536 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T02:57:37.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.176+0000 7fcd57570740 20 client.16536 populate_metadata read hostname 'smithi033' 2024-03-19T02:57:37.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.176+0000 7fcd57570740 1 --2- 172.21.15.33:0/1261315724 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55b6b89d96e0 0x55b6b89dbb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.176+0000 7fcd57570740 4 client.16536 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:57:37.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.176+0000 7fcd57570740 1 -- 172.21.15.33:0/1261315724 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b6b88c99f0 con 0x55b6b89d96e0 2024-03-19T02:57:37.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.176+0000 7fcd4e59c640 1 --2- 172.21.15.33:0/1261315724 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55b6b89d96e0 0x55b6b89dbb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.176+0000 7fcd35ffb640 10 client.16536 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T02:57:37.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.176+0000 7fcd4e59c640 1 --2- 172.21.15.33:0/1261315724 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55b6b89d96e0 0x55b6b89dbb00 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=151588c69d377428 server_cookie=914112bc49684087 in_seq=0 out_seq=0 2024-03-19T02:57:37.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.176+0000 7fcd35ffb640 1 -- 172.21.15.33:0/1261315724 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b6b88c99f0 con 0x55b6b89d96e0 2024-03-19T02:57:37.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.176+0000 7fcd35ffb640 10 client.16536 handle_command_reply: tid=0 2024-03-19T02:57:37.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.176+0000 7fcd57570740 1 client.16536 shutdown 2024-03-19T02:57:37.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.177+0000 7fcd57570740 1 -- 172.21.15.33:0/1261315724 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6b8959bf0 msgr2=0x55b6b895c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:37.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.177+0000 7fcd57570740 1 --2- 172.21.15.33:0/1261315724 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6b8959bf0 0x55b6b895c730 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fcd38019000 tx=0x7fcd38004b00 comp rx=0 tx=0).stop 2024-03-19T02:57:37.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.177+0000 7fcd57570740 1 -- 172.21.15.33:0/1261315724 shutdown_connections 2024-03-19T02:57:37.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.177+0000 7fcd57570740 1 --2- 172.21.15.33:0/1261315724 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55b6b89d96e0 0x55b6b89dbb00 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.177+0000 7fcd57570740 1 --2- 172.21.15.33:0/1261315724 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b6b89627d0 0x55b6b895cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.177+0000 7fcd57570740 1 --2- 172.21.15.33:0/1261315724 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b6b8959bf0 0x55b6b895c730 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.177+0000 7fcd57570740 1 --2- 172.21.15.33:0/1261315724 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b6b89592d0 0x55b6b895c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.177+0000 7fcd35ffb640 0 client.16536 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T02:57:37.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.177+0000 7fcd57570740 1 -- 172.21.15.33:0/1261315724 >> 172.21.15.33:0/1261315724 conn(0x55b6b8953a00 msgr2=0x55b6b8955650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:37.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.177+0000 7fcd57570740 1 -- 172.21.15.33:0/1261315724 shutdown_connections 2024-03-19T02:57:37.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.177+0000 7fcd57570740 1 -- 172.21.15.33:0/1261315724 wait complete. 2024-03-19T02:57:37.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.177+0000 7fcd57570740 20 client.16536 trim_cache size 0 max 16384 2024-03-19T02:57:37.185 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:57:37.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.177+0000 7fcd548ae640 1 -- 172.21.15.33:0/3416844335 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fcd3c0a2970 msgr2=0x7fcd3c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:37.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.177+0000 7fcd548ae640 1 --2- 172.21.15.33:0/3416844335 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fcd3c0a2970 0x7fcd3c0a4e30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcd48005ab0 tx=0x7fcd480058d0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.177+0000 7fcd548ae640 1 -- 172.21.15.33:0/3416844335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd50101790 msgr2=0x7fcd50197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:37.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.177+0000 7fcd548ae640 1 --2- 172.21.15.33:0/3416844335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd50101790 0x7fcd50197f80 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fcd4000eb90 tx=0x7fcd4000c6a0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.177+0000 7fcd548ae640 1 -- 172.21.15.33:0/3416844335 shutdown_connections 2024-03-19T02:57:37.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.177+0000 7fcd548ae640 1 --2- 172.21.15.33:0/3416844335 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fcd3c0a2970 0x7fcd3c0a4e30 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.177+0000 7fcd548ae640 1 --2- 172.21.15.33:0/3416844335 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcd50102150 0x7fcd501984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.177+0000 7fcd548ae640 1 --2- 172.21.15.33:0/3416844335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd501077a0 0x7fcd5019c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.177+0000 7fcd548ae640 1 --2- 172.21.15.33:0/3416844335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcd50101790 0x7fcd50197f80 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.177+0000 7fcd548ae640 1 -- 172.21.15.33:0/3416844335 >> 172.21.15.33:0/3416844335 conn(0x7fcd500fd2c0 msgr2=0x7fcd500fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:37.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.177+0000 7fcd548ae640 1 -- 172.21.15.33:0/3416844335 shutdown_connections 2024-03-19T02:57:37.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.177+0000 7fcd548ae640 1 -- 172.21.15.33:0/3416844335 wait complete. 2024-03-19T02:57:37.189 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:57:37.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.294+0000 7f4ad9f3b640 1 -- 172.21.15.33:0/2407125479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ad4105690 msgr2=0x7f4ad4105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:37.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.294+0000 7f4ad9f3b640 1 --2- 172.21.15.33:0/2407125479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ad4105690 0x7f4ad4105a70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4ac4009890 tx=0x7f4ac402bb70 comp rx=0 tx=0).stop 2024-03-19T02:57:37.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.294+0000 7f4ad9f3b640 1 -- 172.21.15.33:0/2407125479 shutdown_connections 2024-03-19T02:57:37.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.294+0000 7f4ad9f3b640 1 --2- 172.21.15.33:0/2407125479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ad4100970 0x7f4ad4100dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.294+0000 7f4ad9f3b640 1 --2- 172.21.15.33:0/2407125479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ad4101310 0x7f4ad410d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.294+0000 7f4ad9f3b640 1 --2- 172.21.15.33:0/2407125479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ad4105690 0x7f4ad4105a70 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.294+0000 7f4ad9f3b640 1 -- 172.21.15.33:0/2407125479 >> 172.21.15.33:0/2407125479 conn(0x7f4ad4077500 msgr2=0x7f4ad4077910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:37.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.294+0000 7f4ad9f3b640 1 -- 172.21.15.33:0/2407125479 shutdown_connections 2024-03-19T02:57:37.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.294+0000 7f4ad9f3b640 1 -- 172.21.15.33:0/2407125479 wait complete. 2024-03-19T02:57:37.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.295+0000 7f4ad9f3b640 1 Processor -- start 2024-03-19T02:57:37.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.295+0000 7f4ad9f3b640 1 -- start start 2024-03-19T02:57:37.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.295+0000 7f4ad9f3b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ad4100970 0x7f4ad41980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.295+0000 7f4ad9f3b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ad4101310 0x7f4ad4198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.295+0000 7f4ad9f3b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ad4105690 0x7f4ad419c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.295+0000 7f4ad9f3b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4ad4111b30 con 0x7f4ad4100970 2024-03-19T02:57:37.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.295+0000 7f4ad9f3b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4ad41119b0 con 0x7f4ad4101310 2024-03-19T02:57:37.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.295+0000 7f4ad9f3b640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4ad4111cb0 con 0x7f4ad4105690 2024-03-19T02:57:37.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.295+0000 7f4ad37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ad4100970 0x7f4ad41980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.295+0000 7f4ad37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ad4100970 0x7f4ad41980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56040/0 (socket says 172.21.15.33:56040) 2024-03-19T02:57:37.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.295+0000 7f4ad37fe640 1 -- 172.21.15.33:0/1202088325 learned_addr learned my addr 172.21.15.33:0/1202088325 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:37.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.295+0000 7f4ad2ffd640 1 --2- 172.21.15.33:0/1202088325 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ad4101310 0x7f4ad4198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.295+0000 7f4ad3fff640 1 --2- 172.21.15.33:0/1202088325 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ad4105690 0x7f4ad419c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.296+0000 7f4ad37fe640 1 -- 172.21.15.33:0/1202088325 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ad4105690 msgr2=0x7f4ad419c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:37.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.296+0000 7f4ad37fe640 1 --2- 172.21.15.33:0/1202088325 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ad4105690 0x7f4ad419c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.296+0000 7f4ad37fe640 1 -- 172.21.15.33:0/1202088325 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ad4101310 msgr2=0x7f4ad4198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:37.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.296+0000 7f4ad37fe640 1 --2- 172.21.15.33:0/1202088325 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ad4101310 0x7f4ad4198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.296+0000 7f4ad37fe640 1 -- 172.21.15.33:0/1202088325 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ad419d070 con 0x7f4ad4100970 2024-03-19T02:57:37.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.296+0000 7f4ad37fe640 1 --2- 172.21.15.33:0/1202088325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ad4100970 0x7f4ad41980d0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f4ac4009860 tx=0x7f4ac4035990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:37.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.296+0000 7f4ad0ff9640 1 -- 172.21.15.33:0/1202088325 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac4041020 con 0x7f4ad4100970 2024-03-19T02:57:37.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.296+0000 7f4ad9f3b640 1 -- 172.21.15.33:0/1202088325 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ad419d300 con 0x7f4ad4100970 2024-03-19T02:57:37.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.296+0000 7f4ad0ff9640 1 -- 172.21.15.33:0/1202088325 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4ac402c9c0 con 0x7f4ad4100970 2024-03-19T02:57:37.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.296+0000 7f4ad0ff9640 1 -- 172.21.15.33:0/1202088325 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac4039040 con 0x7f4ad4100970 2024-03-19T02:57:37.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.296+0000 7f4ad9f3b640 1 -- 172.21.15.33:0/1202088325 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ad41a4bf0 con 0x7f4ad4100970 2024-03-19T02:57:37.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.298+0000 7f4ad0ff9640 1 -- 172.21.15.33:0/1202088325 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f4ac4044020 con 0x7f4ad4100970 2024-03-19T02:57:37.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.298+0000 7f4adcc07740 1 Processor -- start 2024-03-19T02:57:37.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.298+0000 7f4adcc07740 1 -- start start 2024-03-19T02:57:37.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.299+0000 7f4adcc07740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ad697bc000 0x55ad697bc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.299+0000 7f4adcc07740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ad697bc920 0x55ad697c96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.299+0000 7f4adcc07740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad697c9be0 0x55ad697cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.299+0000 7f4adcc07740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ad6971f4b0 con 0x55ad697c9be0 2024-03-19T02:57:37.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.299+0000 7f4adcc07740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55ad696c7d70 con 0x55ad697bc920 2024-03-19T02:57:37.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.299+0000 7f4adcc07740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ad6971b550 con 0x55ad697bc000 2024-03-19T02:57:37.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.299+0000 7f4ad0ff9640 1 --2- 172.21.15.33:0/1202088325 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4abc0a2970 0x7f4abc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.299+0000 7f4ad2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad697c9be0 0x55ad697cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.299+0000 7f4ad2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad697c9be0 0x55ad697cbfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56046/0 (socket says 172.21.15.33:56046) 2024-03-19T02:57:37.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.299+0000 7f4ad2ffd640 1 -- 172.21.15.33:0/1071784013 learned_addr learned my addr 172.21.15.33:0/1071784013 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:37.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.299+0000 7f4ad37fe640 1 --2- 172.21.15.33:0/1071784013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ad697bc920 0x55ad697c96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.299+0000 7f4ad3fff640 1 --2- 172.21.15.33:0/1071784013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ad697bc000 0x55ad697bc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.299+0000 7f4ad3fff640 1 --2- 172.21.15.33:0/1202088325 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4abc0a2970 0x7f4abc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.299+0000 7f4ad0ff9640 1 -- 172.21.15.33:0/1202088325 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f4ac40f0880 con 0x7f4ad4100970 2024-03-19T02:57:37.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.299+0000 7f4ad2ffd640 1 -- 172.21.15.33:0/1071784013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ad697bc000 msgr2=0x55ad697bc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:37.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.299+0000 7f4ad2ffd640 1 --2- 172.21.15.33:0/1071784013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ad697bc000 0x55ad697bc3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.299+0000 7f4ad2ffd640 1 -- 172.21.15.33:0/1071784013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ad697bc920 msgr2=0x55ad697c96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:37.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.299+0000 7f4ad2ffd640 1 --2- 172.21.15.33:0/1071784013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ad697bc920 0x55ad697c96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.299+0000 7f4ad2ffd640 1 -- 172.21.15.33:0/1071784013 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad696ea210 con 0x55ad697c9be0 2024-03-19T02:57:37.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.299+0000 7f4ad3fff640 1 --2- 172.21.15.33:0/1202088325 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4abc0a2970 0x7f4abc0a4e30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4acc005fd0 tx=0x7f4acc0095c0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:37.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.299+0000 7f4ad2ffd640 1 --2- 172.21.15.33:0/1071784013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad697c9be0 0x55ad697cbfd0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8009ca0 tx=0x7f4ac8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d4a5711420ef59 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:37.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.299+0000 7f4ab9ffb640 1 -- 172.21.15.33:0/1071784013 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac801e070 con 0x55ad697c9be0 2024-03-19T02:57:37.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.300+0000 7f4ab9ffb640 1 -- 172.21.15.33:0/1071784013 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4ac8009e50 con 0x55ad697c9be0 2024-03-19T02:57:37.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.300+0000 7f4ab9ffb640 1 -- 172.21.15.33:0/1071784013 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac800e870 con 0x55ad697c9be0 2024-03-19T02:57:37.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.300+0000 7f4adcc07740 1 -- 172.21.15.33:0/1071784013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad697c9be0 msgr2=0x55ad697cbfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:37.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.300+0000 7f4adcc07740 1 --2- 172.21.15.33:0/1071784013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad697c9be0 0x55ad697cbfd0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8009ca0 tx=0x7f4ac8011810 comp rx=0 tx=0).stop 2024-03-19T02:57:37.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.300+0000 7f4adcc07740 1 -- 172.21.15.33:0/1071784013 shutdown_connections 2024-03-19T02:57:37.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.300+0000 7f4adcc07740 1 --2- 172.21.15.33:0/1071784013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad697c9be0 0x55ad697cbfd0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.300+0000 7f4adcc07740 1 --2- 172.21.15.33:0/1071784013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ad697bc920 0x55ad697c96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.300+0000 7f4adcc07740 1 --2- 172.21.15.33:0/1071784013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ad697bc000 0x55ad697bc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.300+0000 7f4adcc07740 1 -- 172.21.15.33:0/1071784013 >> 172.21.15.33:0/1071784013 conn(0x55ad697b6a00 msgr2=0x55ad697b6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:37.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.300+0000 7f4adcc07740 1 -- 172.21.15.33:0/1071784013 shutdown_connections 2024-03-19T02:57:37.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.300+0000 7f4adcc07740 1 -- 172.21.15.33:0/1071784013 wait complete. 2024-03-19T02:57:37.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.300+0000 7f4adcc07740 1 -- start start 2024-03-19T02:57:37.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.301+0000 7f4adcc07740 1 Processor -- start 2024-03-19T02:57:37.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.301+0000 7f4adcc07740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ad697bc000 0x55ad697c6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.301+0000 7f4adcc07740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ad697bc920 0x55ad697c7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.301+0000 7f4adcc07740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad697c9be0 0x55ad697c7660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.301+0000 7f4adcc07740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ad697cc9a0 con 0x55ad697c9be0 2024-03-19T02:57:37.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.301+0000 7f4adcc07740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55ad696db310 con 0x55ad697bc000 2024-03-19T02:57:37.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.301+0000 7f4adcc07740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ad697ccb20 con 0x55ad697bc920 2024-03-19T02:57:37.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.301+0000 7f4adcc07740 10 client.0 _ll_register_callbacks cb 0x55ad696cf2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:57:37.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.301+0000 7f4ad3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad697c9be0 0x55ad697c7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.301+0000 7f4ad3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad697c9be0 0x55ad697c7660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56052/0 (socket says 172.21.15.33:56052) 2024-03-19T02:57:37.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.301+0000 7f4ad3fff640 1 -- 172.21.15.33:0/380348199 learned_addr learned my addr 172.21.15.33:0/380348199 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:37.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.301+0000 7f4ad37fe640 1 --2- 172.21.15.33:0/380348199 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ad697bc000 0x55ad697c6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.301+0000 7f4ad2ffd640 1 --2- 172.21.15.33:0/380348199 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ad697bc920 0x55ad697c7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.301+0000 7f4ad3fff640 1 -- 172.21.15.33:0/380348199 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ad697bc920 msgr2=0x55ad697c7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:37.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.301+0000 7f4ad3fff640 1 --2- 172.21.15.33:0/380348199 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ad697bc920 0x55ad697c7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.301+0000 7f4ad3fff640 1 -- 172.21.15.33:0/380348199 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ad697bc000 msgr2=0x55ad697c6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:37.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.301+0000 7f4ad3fff640 1 --2- 172.21.15.33:0/380348199 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ad697bc000 0x55ad697c6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.301+0000 7f4ad3fff640 1 -- 172.21.15.33:0/380348199 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad694333f0 con 0x55ad697c9be0 2024-03-19T02:57:37.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.302+0000 7f4ad3fff640 1 --2- 172.21.15.33:0/380348199 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad697c9be0 0x55ad697c7660 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f4acc003f20 tx=0x7f4acc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8df3027862adc54 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:37.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.302+0000 7f4ab2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:57:37.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.302+0000 7f4ab2ffd640 1 -- 172.21.15.33:0/380348199 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4acc004b20 con 0x55ad697c9be0 2024-03-19T02:57:37.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.302+0000 7f4ab2ffd640 1 -- 172.21.15.33:0/380348199 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f4acc017040 con 0x55ad697c9be0 2024-03-19T02:57:37.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.302+0000 7f4adcc07740 1 -- 172.21.15.33:0/380348199 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ac8009e50 con 0x55ad697c9be0 2024-03-19T02:57:37.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.302+0000 7f4ab2ffd640 1 -- 172.21.15.33:0/380348199 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4acc032050 con 0x55ad697c9be0 2024-03-19T02:57:37.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.302+0000 7f4adcc07740 10 client.16548 fetch_fsmap learned FSMap version 25 2024-03-19T02:57:37.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.302+0000 7f4adcc07740 1 -- 172.21.15.33:0/380348199 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ad694fd7d0 con 0x55ad697c9be0 2024-03-19T02:57:37.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.302+0000 7f4ab2ffd640 1 -- 172.21.15.33:0/380348199 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4ac8009e50 con 0x55ad697c9be0 2024-03-19T02:57:37.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.302+0000 7f4ab2ffd640 1 -- 172.21.15.33:0/380348199 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f4acc00b040 con 0x55ad697c9be0 2024-03-19T02:57:37.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.302+0000 7f4adcc07740 10 client.16548 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:57:37.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.302+0000 7f4adcc07740 10 client.16548 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T02:57:37.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.302+0000 7f4adcc07740 20 client.16548 populate_metadata read hostname 'smithi033' 2024-03-19T02:57:37.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.302+0000 7f4adcc07740 1 --2- 172.21.15.33:0/380348199 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55ad697d9f30 0x55ad697dc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.302+0000 7f4adcc07740 4 client.16548 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:57:37.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.302+0000 7f4adcc07740 1 -- 172.21.15.33:0/380348199 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad6977d3e0 con 0x55ad697d9f30 2024-03-19T02:57:37.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.303+0000 7f4ad37fe640 1 --2- 172.21.15.33:0/380348199 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55ad697d9f30 0x55ad697dc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.303+0000 7f4ad37fe640 1 --2- 172.21.15.33:0/380348199 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55ad697d9f30 0x55ad697dc3b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6625dab8e5df53da server_cookie=1ddeebc635171b2c in_seq=0 out_seq=0 2024-03-19T02:57:37.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.303+0000 7f4ab2ffd640 10 client.16548 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T02:57:37.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.303+0000 7f4ab2ffd640 1 -- 172.21.15.33:0/380348199 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ad6977d3e0 con 0x55ad697d9f30 2024-03-19T02:57:37.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.303+0000 7f4ab2ffd640 10 client.16548 handle_command_reply: tid=0 2024-03-19T02:57:37.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.303+0000 7f4adcc07740 1 client.16548 shutdown 2024-03-19T02:57:37.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.304+0000 7f4adcc07740 1 -- 172.21.15.33:0/380348199 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad697c9be0 msgr2=0x55ad697c7660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:37.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.304+0000 7f4adcc07740 1 --2- 172.21.15.33:0/380348199 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad697c9be0 0x55ad697c7660 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f4acc003f20 tx=0x7f4acc004a40 comp rx=0 tx=0).stop 2024-03-19T02:57:37.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.304+0000 7f4adcc07740 1 -- 172.21.15.33:0/380348199 shutdown_connections 2024-03-19T02:57:37.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.304+0000 7f4adcc07740 1 --2- 172.21.15.33:0/380348199 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55ad697d9f30 0x55ad697dc3b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.304+0000 7f4adcc07740 1 --2- 172.21.15.33:0/380348199 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ad697bc920 0x55ad697c7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.304+0000 7f4adcc07740 1 --2- 172.21.15.33:0/380348199 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad697c9be0 0x55ad697c7660 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.304+0000 7f4ab2ffd640 0 client.16548 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T02:57:37.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.304+0000 7f4adcc07740 1 --2- 172.21.15.33:0/380348199 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ad697bc000 0x55ad697c6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.304+0000 7f4adcc07740 1 -- 172.21.15.33:0/380348199 >> 172.21.15.33:0/380348199 conn(0x55ad697b6a00 msgr2=0x55ad697b8750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:37.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.304+0000 7f4adcc07740 1 -- 172.21.15.33:0/380348199 shutdown_connections 2024-03-19T02:57:37.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.304+0000 7f4adcc07740 1 -- 172.21.15.33:0/380348199 wait complete. 2024-03-19T02:57:37.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.304+0000 7f4adcc07740 20 client.16548 trim_cache size 0 max 16384 2024-03-19T02:57:37.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.317+0000 7f4adcc07740 1 Processor -- start 2024-03-19T02:57:37.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.317+0000 7f4adcc07740 1 -- start start 2024-03-19T02:57:37.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.317+0000 7f4adcc07740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ad697bc000 0x55ad697c9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.317+0000 7f4adcc07740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad697bc920 0x55ad697d5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.317+0000 7f4adcc07740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ad697c9be0 0x55ad697b5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.317+0000 7f4adcc07740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4ad41af210 con 0x55ad697bc920 2024-03-19T02:57:37.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.317+0000 7f4adcc07740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4ad41acae0 con 0x55ad697bc000 2024-03-19T02:57:37.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.317+0000 7f4adcc07740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4ad41aa910 con 0x55ad697c9be0 2024-03-19T02:57:37.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.317+0000 7f4ad2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad697bc920 0x55ad697d5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.317+0000 7f4ad2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad697bc920 0x55ad697d5780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56068/0 (socket says 172.21.15.33:56068) 2024-03-19T02:57:37.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.317+0000 7f4ad2ffd640 1 -- 172.21.15.33:0/1875228047 learned_addr learned my addr 172.21.15.33:0/1875228047 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:37.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.317+0000 7f4ad37fe640 1 --2- 172.21.15.33:0/1875228047 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ad697bc000 0x55ad697c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.317+0000 7f4ad3fff640 1 --2- 172.21.15.33:0/1875228047 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ad697c9be0 0x55ad697b5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.317+0000 7f4ad2ffd640 1 -- 172.21.15.33:0/1875228047 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ad697c9be0 msgr2=0x55ad697b5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:37.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.317+0000 7f4ad2ffd640 1 --2- 172.21.15.33:0/1875228047 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ad697c9be0 0x55ad697b5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.317+0000 7f4ad2ffd640 1 -- 172.21.15.33:0/1875228047 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ad697bc000 msgr2=0x55ad697c9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:37.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.317+0000 7f4ad2ffd640 1 --2- 172.21.15.33:0/1875228047 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ad697bc000 0x55ad697c9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.317+0000 7f4ad2ffd640 1 -- 172.21.15.33:0/1875228047 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad697bbc30 con 0x55ad697bc920 2024-03-19T02:57:37.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.317+0000 7f4ad3fff640 1 --2- 172.21.15.33:0/1875228047 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ad697c9be0 0x55ad697b5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:57:37.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.318+0000 7f4ad2ffd640 1 --2- 172.21.15.33:0/1875228047 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad697bc920 0x55ad697d5780 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8009bc0 tx=0x7f4ac8009f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=881562eec7d0fb92 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:37.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.318+0000 7f4ab97fa640 1 -- 172.21.15.33:0/1875228047 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac8027070 con 0x55ad697bc920 2024-03-19T02:57:37.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.318+0000 7f4ab97fa640 1 -- 172.21.15.33:0/1875228047 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4ac801f040 con 0x55ad697bc920 2024-03-19T02:57:37.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.318+0000 7f4ab97fa640 1 -- 172.21.15.33:0/1875228047 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac801e040 con 0x55ad697bc920 2024-03-19T02:57:37.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.318+0000 7f4adcc07740 1 -- 172.21.15.33:0/1875228047 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad697bc920 msgr2=0x55ad697d5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:37.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.318+0000 7f4adcc07740 1 --2- 172.21.15.33:0/1875228047 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad697bc920 0x55ad697d5780 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8009bc0 tx=0x7f4ac8009f20 comp rx=0 tx=0).stop 2024-03-19T02:57:37.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.318+0000 7f4adcc07740 1 -- 172.21.15.33:0/1875228047 shutdown_connections 2024-03-19T02:57:37.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.318+0000 7f4adcc07740 1 --2- 172.21.15.33:0/1875228047 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ad697c9be0 0x55ad697b5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.318+0000 7f4adcc07740 1 --2- 172.21.15.33:0/1875228047 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad697bc920 0x55ad697d5780 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.318+0000 7f4adcc07740 1 --2- 172.21.15.33:0/1875228047 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ad697bc000 0x55ad697c9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.318+0000 7f4adcc07740 1 -- 172.21.15.33:0/1875228047 >> 172.21.15.33:0/1875228047 conn(0x55ad697b6a00 msgr2=0x55ad697b5310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:37.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.318+0000 7f4adcc07740 1 -- 172.21.15.33:0/1875228047 shutdown_connections 2024-03-19T02:57:37.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.318+0000 7f4adcc07740 1 -- 172.21.15.33:0/1875228047 wait complete. 2024-03-19T02:57:37.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.319+0000 7f4adcc07740 1 -- start start 2024-03-19T02:57:37.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.319+0000 7f4adcc07740 1 Processor -- start 2024-03-19T02:57:37.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.319+0000 7f4adcc07740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ad697bc000 0x55ad697cda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.319+0000 7f4adcc07740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad697bc920 0x55ad697cdf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.319+0000 7f4adcc07740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ad697c9be0 0x55ad6983f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.319+0000 7f4adcc07740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4ad41aafa0 con 0x55ad697bc920 2024-03-19T02:57:37.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.319+0000 7f4adcc07740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4ad41ac410 con 0x55ad697c9be0 2024-03-19T02:57:37.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.319+0000 7f4adcc07740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4ad41ab670 con 0x55ad697bc000 2024-03-19T02:57:37.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.319+0000 7f4adcc07740 10 client.0 _ll_register_callbacks cb 0x7f4acc009e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:57:37.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.319+0000 7f4ad2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad697bc920 0x55ad697cdf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.319+0000 7f4ad2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad697bc920 0x55ad697cdf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56076/0 (socket says 172.21.15.33:56076) 2024-03-19T02:57:37.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.319+0000 7f4ad2ffd640 1 -- 172.21.15.33:0/2381486310 learned_addr learned my addr 172.21.15.33:0/2381486310 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:37.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.319+0000 7f4ad37fe640 1 --2- 172.21.15.33:0/2381486310 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ad697bc000 0x55ad697cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.319+0000 7f4ad3fff640 1 --2- 172.21.15.33:0/2381486310 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ad697c9be0 0x55ad6983f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.320+0000 7f4ad2ffd640 1 -- 172.21.15.33:0/2381486310 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ad697bc000 msgr2=0x55ad697cda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:37.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.320+0000 7f4ad2ffd640 1 --2- 172.21.15.33:0/2381486310 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ad697bc000 0x55ad697cda20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.320+0000 7f4ad2ffd640 1 -- 172.21.15.33:0/2381486310 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ad697c9be0 msgr2=0x55ad6983f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:37.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.320+0000 7f4ad2ffd640 1 --2- 172.21.15.33:0/2381486310 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ad697c9be0 0x55ad6983f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.320+0000 7f4ad2ffd640 1 -- 172.21.15.33:0/2381486310 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad697ce860 con 0x55ad697bc920 2024-03-19T02:57:37.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.320+0000 7f4ad2ffd640 1 --2- 172.21.15.33:0/2381486310 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad697bc920 0x55ad697cdf60 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8009c70 tx=0x7f4ac8003b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae6f5849665fe1dd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:37.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.320+0000 7f4ab2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:57:37.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.320+0000 7f4ab2ffd640 1 -- 172.21.15.33:0/2381486310 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac801e070 con 0x55ad697bc920 2024-03-19T02:57:37.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.320+0000 7f4ab2ffd640 1 -- 172.21.15.33:0/2381486310 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f4ac8005030 con 0x55ad697bc920 2024-03-19T02:57:37.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.320+0000 7f4adcc07740 1 -- 172.21.15.33:0/2381486310 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ac801f040 con 0x55ad697bc920 2024-03-19T02:57:37.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.320+0000 7f4ab2ffd640 1 -- 172.21.15.33:0/2381486310 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4ac803e050 con 0x55ad697bc920 2024-03-19T02:57:37.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.320+0000 7f4adcc07740 10 client.16554 fetch_fsmap learned FSMap version 25 2024-03-19T02:57:37.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.320+0000 7f4adcc07740 1 -- 172.21.15.33:0/2381486310 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ad69840970 con 0x55ad697bc920 2024-03-19T02:57:37.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.320+0000 7f4ab2ffd640 1 -- 172.21.15.33:0/2381486310 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4ac801f040 con 0x55ad697bc920 2024-03-19T02:57:37.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.320+0000 7f4ab2ffd640 1 -- 172.21.15.33:0/2381486310 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f4ac800e8e0 con 0x55ad697bc920 2024-03-19T02:57:37.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.320+0000 7f4adcc07740 10 client.16554 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:57:37.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.320+0000 7f4adcc07740 10 client.16554 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T02:57:37.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.320+0000 7f4adcc07740 20 client.16554 populate_metadata read hostname 'smithi033' 2024-03-19T02:57:37.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.320+0000 7f4adcc07740 1 --2- 172.21.15.33:0/2381486310 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55ad69840df0 0x55ad69843210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.320+0000 7f4adcc07740 4 client.16554 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:57:37.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.320+0000 7f4adcc07740 1 -- 172.21.15.33:0/2381486310 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ad6972c9f0 con 0x55ad69840df0 2024-03-19T02:57:37.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.321+0000 7f4ad37fe640 1 --2- 172.21.15.33:0/2381486310 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55ad69840df0 0x55ad69843210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.321+0000 7f4ab2ffd640 10 client.16554 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T02:57:37.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.321+0000 7f4ad37fe640 1 --2- 172.21.15.33:0/2381486310 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55ad69840df0 0x55ad69843210 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e5f51474e2534530 server_cookie=da329d88f94214e8 in_seq=0 out_seq=0 2024-03-19T02:57:37.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.321+0000 7f4ab2ffd640 1 -- 172.21.15.33:0/2381486310 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ad6972c9f0 con 0x55ad69840df0 2024-03-19T02:57:37.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.321+0000 7f4ab2ffd640 10 client.16554 handle_command_reply: tid=0 2024-03-19T02:57:37.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.321+0000 7f4adcc07740 1 client.16554 shutdown 2024-03-19T02:57:37.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.321+0000 7f4adcc07740 1 -- 172.21.15.33:0/2381486310 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad697bc920 msgr2=0x55ad697cdf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:37.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.321+0000 7f4adcc07740 1 --2- 172.21.15.33:0/2381486310 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad697bc920 0x55ad697cdf60 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8009c70 tx=0x7f4ac8003b60 comp rx=0 tx=0).stop 2024-03-19T02:57:37.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.321+0000 7f4adcc07740 1 -- 172.21.15.33:0/2381486310 shutdown_connections 2024-03-19T02:57:37.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.321+0000 7f4adcc07740 1 --2- 172.21.15.33:0/2381486310 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55ad69840df0 0x55ad69843210 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.321+0000 7f4adcc07740 1 --2- 172.21.15.33:0/2381486310 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ad697c9be0 0x55ad6983f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.321+0000 7f4adcc07740 1 --2- 172.21.15.33:0/2381486310 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ad697bc920 0x55ad697cdf60 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.321+0000 7f4ab2ffd640 0 client.16554 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T02:57:37.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.321+0000 7f4adcc07740 1 --2- 172.21.15.33:0/2381486310 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ad697bc000 0x55ad697cda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.321+0000 7f4adcc07740 1 -- 172.21.15.33:0/2381486310 >> 172.21.15.33:0/2381486310 conn(0x55ad697b6a00 msgr2=0x55ad697c7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:37.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.321+0000 7f4adcc07740 1 -- 172.21.15.33:0/2381486310 shutdown_connections 2024-03-19T02:57:37.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.321+0000 7f4adcc07740 1 -- 172.21.15.33:0/2381486310 wait complete. 2024-03-19T02:57:37.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.322+0000 7f4adcc07740 20 client.16554 trim_cache size 0 max 16384 2024-03-19T02:57:37.330 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:57:37.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.322+0000 7f4ad9f3b640 1 -- 172.21.15.33:0/1202088325 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4abc0a2970 msgr2=0x7f4abc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:37.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.322+0000 7f4ad9f3b640 1 --2- 172.21.15.33:0/1202088325 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4abc0a2970 0x7f4abc0a4e30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4acc005fd0 tx=0x7f4acc0095c0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.322+0000 7f4ad9f3b640 1 -- 172.21.15.33:0/1202088325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ad4100970 msgr2=0x7f4ad41980d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:37.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.322+0000 7f4ad9f3b640 1 --2- 172.21.15.33:0/1202088325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ad4100970 0x7f4ad41980d0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f4ac4009860 tx=0x7f4ac4035990 comp rx=0 tx=0).stop 2024-03-19T02:57:37.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.322+0000 7f4ad9f3b640 1 -- 172.21.15.33:0/1202088325 shutdown_connections 2024-03-19T02:57:37.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.322+0000 7f4ad9f3b640 1 --2- 172.21.15.33:0/1202088325 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4abc0a2970 0x7f4abc0a4e30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.322+0000 7f4ad9f3b640 1 --2- 172.21.15.33:0/1202088325 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ad4105690 0x7f4ad419c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.322+0000 7f4ad9f3b640 1 --2- 172.21.15.33:0/1202088325 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ad4101310 0x7f4ad4198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.322+0000 7f4ad9f3b640 1 --2- 172.21.15.33:0/1202088325 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4ad4100970 0x7f4ad41980d0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.322+0000 7f4ad9f3b640 1 -- 172.21.15.33:0/1202088325 >> 172.21.15.33:0/1202088325 conn(0x7f4ad4077500 msgr2=0x7f4ad40fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:37.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.322+0000 7f4ad9f3b640 1 -- 172.21.15.33:0/1202088325 shutdown_connections 2024-03-19T02:57:37.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.322+0000 7f4ad9f3b640 1 -- 172.21.15.33:0/1202088325 wait complete. 2024-03-19T02:57:37.333 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:37 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:57:37.333 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:37 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:57:37.333 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:37 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:57:37.333 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:37 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3505031901' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:57:37.334 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:57:37.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.438+0000 7fe384c0b640 1 -- 172.21.15.33:0/1531585663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe380102160 msgr2=0x7fe38010e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:37.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.438+0000 7fe384c0b640 1 --2- 172.21.15.33:0/1531585663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe380102160 0x7fe38010e670 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fe378009cd0 tx=0x7fe37802c0f0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.439+0000 7fe384c0b640 1 -- 172.21.15.33:0/1531585663 shutdown_connections 2024-03-19T02:57:37.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.439+0000 7fe384c0b640 1 --2- 172.21.15.33:0/1531585663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe380102160 0x7fe38010e670 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.439+0000 7fe384c0b640 1 --2- 172.21.15.33:0/1531585663 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3801017a0 0x7fe380101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.439+0000 7fe384c0b640 1 --2- 172.21.15.33:0/1531585663 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3801077b0 0x7fe380107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.439+0000 7fe384c0b640 1 -- 172.21.15.33:0/1531585663 >> 172.21.15.33:0/1531585663 conn(0x7fe3800fd2d0 msgr2=0x7fe3800ff6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:37.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.439+0000 7fe384c0b640 1 -- 172.21.15.33:0/1531585663 shutdown_connections 2024-03-19T02:57:37.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.439+0000 7fe384c0b640 1 -- 172.21.15.33:0/1531585663 wait complete. 2024-03-19T02:57:37.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.439+0000 7fe384c0b640 1 Processor -- start 2024-03-19T02:57:37.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.439+0000 7fe384c0b640 1 -- start start 2024-03-19T02:57:37.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.440+0000 7fe384c0b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3801017a0 0x7fe380197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.440+0000 7fe384c0b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe380102160 0x7fe380198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.440+0000 7fe384c0b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3801077b0 0x7fe38019c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.440+0000 7fe384c0b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe380111a70 con 0x7fe3801077b0 2024-03-19T02:57:37.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.440+0000 7fe384c0b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe3801118f0 con 0x7fe380102160 2024-03-19T02:57:37.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.440+0000 7fe384c0b640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe380111bf0 con 0x7fe3801017a0 2024-03-19T02:57:37.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.440+0000 7fe37ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3801077b0 0x7fe38019c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.440+0000 7fe37e59c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3801017a0 0x7fe380197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.440+0000 7fe37ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3801077b0 0x7fe38019c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56092/0 (socket says 172.21.15.33:56092) 2024-03-19T02:57:37.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.440+0000 7fe37ed9d640 1 -- 172.21.15.33:0/448659224 learned_addr learned my addr 172.21.15.33:0/448659224 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:37.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.440+0000 7fe37dd9b640 1 --2- 172.21.15.33:0/448659224 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe380102160 0x7fe380198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.440+0000 7fe37e59c640 1 -- 172.21.15.33:0/448659224 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe380102160 msgr2=0x7fe380198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:37.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.440+0000 7fe37e59c640 1 --2- 172.21.15.33:0/448659224 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe380102160 0x7fe380198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.440+0000 7fe37e59c640 1 -- 172.21.15.33:0/448659224 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3801077b0 msgr2=0x7fe38019c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:37.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.440+0000 7fe37e59c640 1 --2- 172.21.15.33:0/448659224 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3801077b0 0x7fe38019c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.440+0000 7fe37e59c640 1 -- 172.21.15.33:0/448659224 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe38019cf40 con 0x7fe3801017a0 2024-03-19T02:57:37.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.440+0000 7fe37ed9d640 1 --2- 172.21.15.33:0/448659224 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3801077b0 0x7fe38019c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:37.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.440+0000 7fe37e59c640 1 --2- 172.21.15.33:0/448659224 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3801017a0 0x7fe380197f50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe37000e9f0 tx=0x7fe37000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:37.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.440+0000 7fe36f7fe640 1 -- 172.21.15.33:0/448659224 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe37000ccc0 con 0x7fe3801017a0 2024-03-19T02:57:37.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.440+0000 7fe384c0b640 1 -- 172.21.15.33:0/448659224 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe38019d230 con 0x7fe3801017a0 2024-03-19T02:57:37.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.440+0000 7fe36f7fe640 1 -- 172.21.15.33:0/448659224 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe3700108e0 con 0x7fe3801017a0 2024-03-19T02:57:37.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.440+0000 7fe36f7fe640 1 -- 172.21.15.33:0/448659224 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe370018880 con 0x7fe3801017a0 2024-03-19T02:57:37.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.440+0000 7fe384c0b640 1 -- 172.21.15.33:0/448659224 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3801a4ac0 con 0x7fe3801017a0 2024-03-19T02:57:37.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.442+0000 7fe36f7fe640 1 -- 172.21.15.33:0/448659224 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fe370010a90 con 0x7fe3801017a0 2024-03-19T02:57:37.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.443+0000 7fe3877f6740 1 Processor -- start 2024-03-19T02:57:37.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.443+0000 7fe36f7fe640 1 --2- 172.21.15.33:0/448659224 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe3680a29f0 0x7fe3680a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.443+0000 7fe37ed9d640 1 --2- 172.21.15.33:0/448659224 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe3680a29f0 0x7fe3680a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.443+0000 7fe3877f6740 1 -- start start 2024-03-19T02:57:37.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.443+0000 7fe36f7fe640 1 -- 172.21.15.33:0/448659224 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fe3700cf490 con 0x7fe3801017a0 2024-03-19T02:57:37.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.443+0000 7fe3877f6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621ae5562d0 0x5621ae5566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.443+0000 7fe3877f6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5621ae556bf0 0x5621ae55f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.443+0000 7fe3877f6740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5621ae55f7d0 0x5621ae561bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.443+0000 7fe3877f6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5621ae4b94b0 con 0x5621ae5562d0 2024-03-19T02:57:37.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.443+0000 7fe3877f6740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5621ae461d70 con 0x5621ae55f7d0 2024-03-19T02:57:37.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.443+0000 7fe3877f6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5621ae4b5550 con 0x5621ae556bf0 2024-03-19T02:57:37.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.444+0000 7fe37ed9d640 1 --2- 172.21.15.33:0/448659224 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe3680a29f0 0x7fe3680a4eb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe378033000 tx=0x7fe378005f10 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:37.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.444+0000 7fe37e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621ae5562d0 0x5621ae5566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.444+0000 7fe37e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621ae5562d0 0x5621ae5566b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56100/0 (socket says 172.21.15.33:56100) 2024-03-19T02:57:37.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.444+0000 7fe37e59c640 1 -- 172.21.15.33:0/1208903247 learned_addr learned my addr 172.21.15.33:0/1208903247 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:37.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.444+0000 7fe37ed9d640 1 --2- 172.21.15.33:0/1208903247 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5621ae55f7d0 0x5621ae561bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.444+0000 7fe37dd9b640 1 --2- 172.21.15.33:0/1208903247 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5621ae556bf0 0x5621ae55f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.444+0000 7fe37e59c640 1 -- 172.21.15.33:0/1208903247 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5621ae556bf0 msgr2=0x5621ae55f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:37.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.444+0000 7fe37e59c640 1 --2- 172.21.15.33:0/1208903247 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5621ae556bf0 0x5621ae55f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.444+0000 7fe37e59c640 1 -- 172.21.15.33:0/1208903247 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5621ae55f7d0 msgr2=0x5621ae561bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:37.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.444+0000 7fe37e59c640 1 --2- 172.21.15.33:0/1208903247 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5621ae55f7d0 0x5621ae561bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.444+0000 7fe37e59c640 1 -- 172.21.15.33:0/1208903247 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621ae484210 con 0x5621ae5562d0 2024-03-19T02:57:37.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.444+0000 7fe37e59c640 1 --2- 172.21.15.33:0/1208903247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621ae5562d0 0x5621ae5566b0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fe3700981c0 tx=0x7fe370010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14818203b2f7f7d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:37.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.444+0000 7fe36cff9640 1 -- 172.21.15.33:0/1208903247 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3700a3ee0 con 0x5621ae5562d0 2024-03-19T02:57:37.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.444+0000 7fe36cff9640 1 -- 172.21.15.33:0/1208903247 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe3700860c0 con 0x5621ae5562d0 2024-03-19T02:57:37.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.444+0000 7fe36cff9640 1 -- 172.21.15.33:0/1208903247 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe37009ac40 con 0x5621ae5562d0 2024-03-19T02:57:37.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.445+0000 7fe3877f6740 1 -- 172.21.15.33:0/1208903247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621ae5562d0 msgr2=0x5621ae5566b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:37.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.445+0000 7fe3877f6740 1 --2- 172.21.15.33:0/1208903247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621ae5562d0 0x5621ae5566b0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fe3700981c0 tx=0x7fe370010f40 comp rx=0 tx=0).stop 2024-03-19T02:57:37.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.445+0000 7fe3877f6740 1 -- 172.21.15.33:0/1208903247 shutdown_connections 2024-03-19T02:57:37.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.445+0000 7fe3877f6740 1 --2- 172.21.15.33:0/1208903247 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5621ae556bf0 0x5621ae55f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.445+0000 7fe3877f6740 1 --2- 172.21.15.33:0/1208903247 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5621ae55f7d0 0x5621ae561bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.445+0000 7fe3877f6740 1 --2- 172.21.15.33:0/1208903247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621ae5562d0 0x5621ae5566b0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.445+0000 7fe3877f6740 1 -- 172.21.15.33:0/1208903247 >> 172.21.15.33:0/1208903247 conn(0x5621ae550a00 msgr2=0x5621ae550de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:37.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.445+0000 7fe3877f6740 1 -- 172.21.15.33:0/1208903247 shutdown_connections 2024-03-19T02:57:37.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.445+0000 7fe3877f6740 1 -- 172.21.15.33:0/1208903247 wait complete. 2024-03-19T02:57:37.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.445+0000 7fe3877f6740 1 -- start start 2024-03-19T02:57:37.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.445+0000 7fe3877f6740 1 Processor -- start 2024-03-19T02:57:37.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.446+0000 7fe3877f6740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5621ae5562d0 0x5621ae55c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.446+0000 7fe3877f6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621ae556bf0 0x5621ae55cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.446+0000 7fe3877f6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5621ae55f7d0 0x5621ae55d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.446+0000 7fe3877f6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5621ae562500 con 0x5621ae556bf0 2024-03-19T02:57:37.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.446+0000 7fe3877f6740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5621ae475310 con 0x5621ae5562d0 2024-03-19T02:57:37.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.446+0000 7fe3877f6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5621ae562680 con 0x5621ae55f7d0 2024-03-19T02:57:37.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.446+0000 7fe3877f6740 10 client.0 _ll_register_callbacks cb 0x5621ae4692d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:57:37.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.446+0000 7fe37dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621ae556bf0 0x5621ae55cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.446+0000 7fe37dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621ae556bf0 0x5621ae55cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56114/0 (socket says 172.21.15.33:56114) 2024-03-19T02:57:37.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.446+0000 7fe37dd9b640 1 -- 172.21.15.33:0/1972302469 learned_addr learned my addr 172.21.15.33:0/1972302469 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:37.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.446+0000 7fe37e59c640 1 --2- 172.21.15.33:0/1972302469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5621ae5562d0 0x5621ae55c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.446+0000 7fe37ed9d640 1 --2- 172.21.15.33:0/1972302469 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5621ae55f7d0 0x5621ae55d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.446+0000 7fe37dd9b640 1 -- 172.21.15.33:0/1972302469 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5621ae55f7d0 msgr2=0x5621ae55d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:37.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.446+0000 7fe37dd9b640 1 --2- 172.21.15.33:0/1972302469 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5621ae55f7d0 0x5621ae55d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.446+0000 7fe37dd9b640 1 -- 172.21.15.33:0/1972302469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5621ae5562d0 msgr2=0x5621ae55c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:37.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.446+0000 7fe37dd9b640 1 --2- 172.21.15.33:0/1972302469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5621ae5562d0 0x5621ae55c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.446+0000 7fe37dd9b640 1 -- 172.21.15.33:0/1972302469 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621ae1cd3f0 con 0x5621ae556bf0 2024-03-19T02:57:37.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.446+0000 7fe37dd9b640 1 --2- 172.21.15.33:0/1972302469 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621ae556bf0 0x5621ae55cd80 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fe374012720 tx=0x7fe374012be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f16f1321a0e975f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:37.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.446+0000 7fe365ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:57:37.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.446+0000 7fe365ffb640 1 -- 172.21.15.33:0/1972302469 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe37400f920 con 0x5621ae556bf0 2024-03-19T02:57:37.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.446+0000 7fe365ffb640 1 -- 172.21.15.33:0/1972302469 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fe37400fac0 con 0x5621ae556bf0 2024-03-19T02:57:37.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.446+0000 7fe3877f6740 1 -- 172.21.15.33:0/1972302469 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3700860c0 con 0x5621ae556bf0 2024-03-19T02:57:37.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.446+0000 7fe365ffb640 1 -- 172.21.15.33:0/1972302469 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe37400fcc0 con 0x5621ae556bf0 2024-03-19T02:57:37.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.446+0000 7fe3877f6740 10 client.16566 fetch_fsmap learned FSMap version 25 2024-03-19T02:57:37.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.446+0000 7fe3877f6740 1 -- 172.21.15.33:0/1972302469 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5621ae2977d0 con 0x5621ae556bf0 2024-03-19T02:57:37.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.447+0000 7fe365ffb640 1 -- 172.21.15.33:0/1972302469 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe3700860c0 con 0x5621ae556bf0 2024-03-19T02:57:37.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.447+0000 7fe365ffb640 1 -- 172.21.15.33:0/1972302469 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fe3740052e0 con 0x5621ae556bf0 2024-03-19T02:57:37.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.447+0000 7fe3877f6740 10 client.16566 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:57:37.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.447+0000 7fe3877f6740 10 client.16566 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:57:37.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.447+0000 7fe3877f6740 20 client.16566 populate_metadata read hostname 'smithi033' 2024-03-19T02:57:37.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.447+0000 7fe3877f6740 1 --2- 172.21.15.33:0/1972302469 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5621ae56fb90 0x5621ae571fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.447+0000 7fe3877f6740 4 client.16566 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:57:37.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.447+0000 7fe3877f6740 1 -- 172.21.15.33:0/1972302469 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5621ae5173e0 con 0x5621ae56fb90 2024-03-19T02:57:37.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.447+0000 7fe37e59c640 1 --2- 172.21.15.33:0/1972302469 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5621ae56fb90 0x5621ae571fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.447+0000 7fe365ffb640 10 client.16566 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:57:37.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.447+0000 7fe37e59c640 1 --2- 172.21.15.33:0/1972302469 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5621ae56fb90 0x5621ae571fa0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c7dd60ea0ebf967 server_cookie=86a16019fed56b28 in_seq=0 out_seq=0 2024-03-19T02:57:37.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.448+0000 7fe365ffb640 1 -- 172.21.15.33:0/1972302469 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5621ae5173e0 con 0x5621ae56fb90 2024-03-19T02:57:37.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.448+0000 7fe365ffb640 10 client.16566 handle_command_reply: tid=0 2024-03-19T02:57:37.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.448+0000 7fe3877f6740 1 client.16566 shutdown 2024-03-19T02:57:37.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.448+0000 7fe3877f6740 1 -- 172.21.15.33:0/1972302469 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621ae556bf0 msgr2=0x5621ae55cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:37.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.448+0000 7fe3877f6740 1 --2- 172.21.15.33:0/1972302469 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621ae556bf0 0x5621ae55cd80 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fe374012720 tx=0x7fe374012be0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.448+0000 7fe3877f6740 1 -- 172.21.15.33:0/1972302469 shutdown_connections 2024-03-19T02:57:37.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.448+0000 7fe3877f6740 1 --2- 172.21.15.33:0/1972302469 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5621ae56fb90 0x5621ae571fa0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.448+0000 7fe3877f6740 1 --2- 172.21.15.33:0/1972302469 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5621ae55f7d0 0x5621ae55d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.448+0000 7fe3877f6740 1 --2- 172.21.15.33:0/1972302469 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621ae556bf0 0x5621ae55cd80 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.448+0000 7fe3877f6740 1 --2- 172.21.15.33:0/1972302469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5621ae5562d0 0x5621ae55c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.448+0000 7fe365ffb640 0 client.16566 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:57:37.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.448+0000 7fe3877f6740 1 -- 172.21.15.33:0/1972302469 >> 172.21.15.33:0/1972302469 conn(0x5621ae550a00 msgr2=0x5621ae552750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:37.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.448+0000 7fe3877f6740 1 -- 172.21.15.33:0/1972302469 shutdown_connections 2024-03-19T02:57:37.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.448+0000 7fe3877f6740 1 -- 172.21.15.33:0/1972302469 wait complete. 2024-03-19T02:57:37.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.448+0000 7fe3877f6740 20 client.16566 trim_cache size 0 max 16384 2024-03-19T02:57:37.467 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:37 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:57:37.468 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:37 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:57:37.468 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:37 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:57:37.468 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:37 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3505031901' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:57:37.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.461+0000 7fe3877f6740 1 Processor -- start 2024-03-19T02:57:37.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.461+0000 7fe3877f6740 1 -- start start 2024-03-19T02:57:37.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.462+0000 7fe3877f6740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5621ae5562d0 0x5621ae55ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.462+0000 7fe3877f6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5621ae556bf0 0x5621ae56b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.462+0000 7fe3877f6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621ae55f7d0 0x5621ae5778b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.462+0000 7fe3877f6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe3801b14e0 con 0x5621ae55f7d0 2024-03-19T02:57:37.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.462+0000 7fe3877f6740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe3801aedb0 con 0x5621ae5562d0 2024-03-19T02:57:37.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.462+0000 7fe3877f6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe3801acbe0 con 0x5621ae556bf0 2024-03-19T02:57:37.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.462+0000 7fe37ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621ae55f7d0 0x5621ae5778b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.462+0000 7fe37ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621ae55f7d0 0x5621ae5778b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56124/0 (socket says 172.21.15.33:56124) 2024-03-19T02:57:37.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.462+0000 7fe37ed9d640 1 -- 172.21.15.33:0/907651477 learned_addr learned my addr 172.21.15.33:0/907651477 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:37.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.462+0000 7fe37e59c640 1 --2- 172.21.15.33:0/907651477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5621ae5562d0 0x5621ae55ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.462+0000 7fe37dd9b640 1 --2- 172.21.15.33:0/907651477 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5621ae556bf0 0x5621ae56b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.462+0000 7fe37ed9d640 1 -- 172.21.15.33:0/907651477 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5621ae556bf0 msgr2=0x5621ae56b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:37.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.462+0000 7fe37ed9d640 1 --2- 172.21.15.33:0/907651477 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5621ae556bf0 0x5621ae56b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.462+0000 7fe37ed9d640 1 -- 172.21.15.33:0/907651477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5621ae5562d0 msgr2=0x5621ae55ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:37.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.462+0000 7fe37ed9d640 1 --2- 172.21.15.33:0/907651477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5621ae5562d0 0x5621ae55ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.462+0000 7fe37ed9d640 1 -- 172.21.15.33:0/907651477 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621ae526880 con 0x5621ae55f7d0 2024-03-19T02:57:37.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.462+0000 7fe37ed9d640 1 --2- 172.21.15.33:0/907651477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621ae55f7d0 0x5621ae5778b0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fe378037040 tx=0x7fe37803e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95f43cddd3c19788 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:37.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.462+0000 7fe367fff640 1 -- 172.21.15.33:0/907651477 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe378039040 con 0x5621ae55f7d0 2024-03-19T02:57:37.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.462+0000 7fe367fff640 1 -- 172.21.15.33:0/907651477 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe378051070 con 0x5621ae55f7d0 2024-03-19T02:57:37.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.462+0000 7fe367fff640 1 -- 172.21.15.33:0/907651477 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe378034970 con 0x5621ae55f7d0 2024-03-19T02:57:37.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.463+0000 7fe3877f6740 1 -- 172.21.15.33:0/907651477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621ae55f7d0 msgr2=0x5621ae5778b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:37.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.463+0000 7fe3877f6740 1 --2- 172.21.15.33:0/907651477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621ae55f7d0 0x5621ae5778b0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fe378037040 tx=0x7fe37803e000 comp rx=0 tx=0).stop 2024-03-19T02:57:37.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.463+0000 7fe3877f6740 1 -- 172.21.15.33:0/907651477 shutdown_connections 2024-03-19T02:57:37.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.463+0000 7fe3877f6740 1 --2- 172.21.15.33:0/907651477 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5621ae556bf0 0x5621ae56b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.463+0000 7fe3877f6740 1 --2- 172.21.15.33:0/907651477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621ae55f7d0 0x5621ae5778b0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.463+0000 7fe3877f6740 1 --2- 172.21.15.33:0/907651477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5621ae5562d0 0x5621ae55ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.463+0000 7fe3877f6740 1 -- 172.21.15.33:0/907651477 >> 172.21.15.33:0/907651477 conn(0x5621ae550a00 msgr2=0x5621ae54f310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:37.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.463+0000 7fe3877f6740 1 -- 172.21.15.33:0/907651477 shutdown_connections 2024-03-19T02:57:37.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.463+0000 7fe3877f6740 1 -- 172.21.15.33:0/907651477 wait complete. 2024-03-19T02:57:37.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.463+0000 7fe3877f6740 1 -- start start 2024-03-19T02:57:37.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.463+0000 7fe3877f6740 1 Processor -- start 2024-03-19T02:57:37.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.464+0000 7fe3877f6740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5621ae5562d0 0x5621ae5678f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.464+0000 7fe3877f6740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5621ae556bf0 0x5621ae563c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.464+0000 7fe3877f6740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621ae55f7d0 0x5621ae5641b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.464+0000 7fe3877f6740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe3801ad270 con 0x5621ae55f7d0 2024-03-19T02:57:37.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.464+0000 7fe3877f6740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe3801ae6e0 con 0x5621ae556bf0 2024-03-19T02:57:37.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.464+0000 7fe3877f6740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe3801ad940 con 0x5621ae5562d0 2024-03-19T02:57:37.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.464+0000 7fe3877f6740 10 client.0 _ll_register_callbacks cb 0x7fe374009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:57:37.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.464+0000 7fe37ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621ae55f7d0 0x5621ae5641b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.464+0000 7fe37ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621ae55f7d0 0x5621ae5641b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56132/0 (socket says 172.21.15.33:56132) 2024-03-19T02:57:37.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.464+0000 7fe37dd9b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5621ae556bf0 0x5621ae563c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.464+0000 7fe37ed9d640 1 -- 172.21.15.33:0/875581154 learned_addr learned my addr 172.21.15.33:0/875581154 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:37.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.464+0000 7fe37e59c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5621ae5562d0 0x5621ae5678f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.464+0000 7fe37ed9d640 1 -- 172.21.15.33:0/875581154 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5621ae5562d0 msgr2=0x5621ae5678f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:37.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.464+0000 7fe37ed9d640 1 --2- 172.21.15.33:0/875581154 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5621ae5562d0 0x5621ae5678f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.464+0000 7fe37ed9d640 1 -- 172.21.15.33:0/875581154 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5621ae556bf0 msgr2=0x5621ae563c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:37.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.464+0000 7fe37ed9d640 1 --2- 172.21.15.33:0/875581154 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5621ae556bf0 0x5621ae563c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.464+0000 7fe37ed9d640 1 -- 172.21.15.33:0/875581154 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621ae564720 con 0x5621ae55f7d0 2024-03-19T02:57:37.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.464+0000 7fe37dd9b640 1 --2- 172.21.15.33:0/875581154 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5621ae556bf0 0x5621ae563c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:37.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.464+0000 7fe37ed9d640 1 --2- 172.21.15.33:0/875581154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621ae55f7d0 0x5621ae5641b0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fe378034a50 tx=0x7fe378034a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ac60c399583a112 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:37.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.464+0000 7fe365ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:57:37.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.464+0000 7fe365ffb640 1 -- 172.21.15.33:0/875581154 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe378003fb0 con 0x5621ae55f7d0 2024-03-19T02:57:37.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.464+0000 7fe365ffb640 1 -- 172.21.15.33:0/875581154 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fe378004150 con 0x5621ae55f7d0 2024-03-19T02:57:37.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.464+0000 7fe3877f6740 1 -- 172.21.15.33:0/875581154 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe378051070 con 0x5621ae55f7d0 2024-03-19T02:57:37.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.464+0000 7fe365ffb640 1 -- 172.21.15.33:0/875581154 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe378065050 con 0x5621ae55f7d0 2024-03-19T02:57:37.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.465+0000 7fe3877f6740 10 client.16572 fetch_fsmap learned FSMap version 25 2024-03-19T02:57:37.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.465+0000 7fe3877f6740 1 -- 172.21.15.33:0/875581154 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5621ae5d6c70 con 0x5621ae55f7d0 2024-03-19T02:57:37.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.465+0000 7fe365ffb640 1 -- 172.21.15.33:0/875581154 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe378051070 con 0x5621ae55f7d0 2024-03-19T02:57:37.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.465+0000 7fe365ffb640 1 -- 172.21.15.33:0/875581154 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fe37803f7b0 con 0x5621ae55f7d0 2024-03-19T02:57:37.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.465+0000 7fe3877f6740 10 client.16572 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:57:37.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.465+0000 7fe3877f6740 10 client.16572 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:57:37.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.465+0000 7fe3877f6740 20 client.16572 populate_metadata read hostname 'smithi033' 2024-03-19T02:57:37.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.465+0000 7fe3877f6740 1 --2- 172.21.15.33:0/875581154 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5621ae5d7310 0x5621ae5d9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:37.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.465+0000 7fe3877f6740 4 client.16572 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:57:37.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.465+0000 7fe3877f6740 1 -- 172.21.15.33:0/875581154 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5621ae4c69f0 con 0x5621ae5d7310 2024-03-19T02:57:37.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.465+0000 7fe37e59c640 1 --2- 172.21.15.33:0/875581154 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5621ae5d7310 0x5621ae5d9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:37.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.465+0000 7fe37e59c640 1 --2- 172.21.15.33:0/875581154 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5621ae5d7310 0x5621ae5d9730 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6419c8d6213b3ec8 server_cookie=762f039938705858 in_seq=0 out_seq=0 2024-03-19T02:57:37.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.465+0000 7fe365ffb640 10 client.16572 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:57:37.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.466+0000 7fe365ffb640 1 -- 172.21.15.33:0/875581154 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5621ae4c69f0 con 0x5621ae5d7310 2024-03-19T02:57:37.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.466+0000 7fe365ffb640 10 client.16572 handle_command_reply: tid=0 2024-03-19T02:57:37.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.466+0000 7fe3877f6740 1 client.16572 shutdown 2024-03-19T02:57:37.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.466+0000 7fe3877f6740 1 -- 172.21.15.33:0/875581154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621ae55f7d0 msgr2=0x5621ae5641b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:37.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.466+0000 7fe3877f6740 1 --2- 172.21.15.33:0/875581154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621ae55f7d0 0x5621ae5641b0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fe378034a50 tx=0x7fe378034a80 comp rx=0 tx=0).stop 2024-03-19T02:57:37.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.466+0000 7fe3877f6740 1 -- 172.21.15.33:0/875581154 shutdown_connections 2024-03-19T02:57:37.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.466+0000 7fe3877f6740 1 --2- 172.21.15.33:0/875581154 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5621ae5d7310 0x5621ae5d9730 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.466+0000 7fe3877f6740 1 --2- 172.21.15.33:0/875581154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5621ae55f7d0 0x5621ae5641b0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.466+0000 7fe3877f6740 1 --2- 172.21.15.33:0/875581154 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5621ae556bf0 0x5621ae563c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.466+0000 7fe3877f6740 1 --2- 172.21.15.33:0/875581154 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5621ae5562d0 0x5621ae5678f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.466+0000 7fe3877f6740 1 -- 172.21.15.33:0/875581154 >> 172.21.15.33:0/875581154 conn(0x5621ae550a00 msgr2=0x5621ae55c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:37.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.466+0000 7fe365ffb640 0 client.16572 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:57:37.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.466+0000 7fe3877f6740 1 -- 172.21.15.33:0/875581154 shutdown_connections 2024-03-19T02:57:37.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.466+0000 7fe3877f6740 1 -- 172.21.15.33:0/875581154 wait complete. 2024-03-19T02:57:37.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.466+0000 7fe3877f6740 20 client.16572 trim_cache size 0 max 16384 2024-03-19T02:57:37.474 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:57:37.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.466+0000 7fe384c0b640 1 -- 172.21.15.33:0/448659224 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe3680a29f0 msgr2=0x7fe3680a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:37.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.466+0000 7fe384c0b640 1 --2- 172.21.15.33:0/448659224 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe3680a29f0 0x7fe3680a4eb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe378033000 tx=0x7fe378005f10 comp rx=0 tx=0).stop 2024-03-19T02:57:37.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.466+0000 7fe384c0b640 1 -- 172.21.15.33:0/448659224 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3801017a0 msgr2=0x7fe380197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:37.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.466+0000 7fe384c0b640 1 --2- 172.21.15.33:0/448659224 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3801017a0 0x7fe380197f50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe37000e9f0 tx=0x7fe37000eeb0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.466+0000 7fe384c0b640 1 -- 172.21.15.33:0/448659224 shutdown_connections 2024-03-19T02:57:37.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.466+0000 7fe384c0b640 1 --2- 172.21.15.33:0/448659224 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe3680a29f0 0x7fe3680a4eb0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.466+0000 7fe384c0b640 1 --2- 172.21.15.33:0/448659224 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe3801077b0 0x7fe38019c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.466+0000 7fe384c0b640 1 --2- 172.21.15.33:0/448659224 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe380102160 0x7fe380198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.466+0000 7fe384c0b640 1 --2- 172.21.15.33:0/448659224 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3801017a0 0x7fe380197f50 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:37.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.466+0000 7fe384c0b640 1 -- 172.21.15.33:0/448659224 >> 172.21.15.33:0/448659224 conn(0x7fe3800fd2d0 msgr2=0x7fe3800fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:37.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.466+0000 7fe384c0b640 1 -- 172.21.15.33:0/448659224 shutdown_connections 2024-03-19T02:57:37.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:37.466+0000 7fe384c0b640 1 -- 172.21.15.33:0/448659224 wait complete. 2024-03-19T02:57:37.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:37 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:57:37.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:37 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:57:37.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:37 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:57:37.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:37 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3505031901' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:57:38.479 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:57:38.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:57:38.479 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:57:38.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.598+0000 7f1a3b59e640 1 -- 172.21.15.33:0/116822731 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a3c103200 msgr2=0x7f1a3c10b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:38.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.598+0000 7f1a3b59e640 1 --2- 172.21.15.33:0/116822731 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a3c103200 0x7f1a3c10b6e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1a24009860 tx=0x7f1a2402bc00 comp rx=0 tx=0).stop 2024-03-19T02:57:38.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.598+0000 7f1a3b59e640 1 -- 172.21.15.33:0/116822731 shutdown_connections 2024-03-19T02:57:38.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.598+0000 7f1a3b59e640 1 --2- 172.21.15.33:0/116822731 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a3c10bd00 0x7f1a3c10e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.598+0000 7f1a3b59e640 1 --2- 172.21.15.33:0/116822731 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a3c103200 0x7f1a3c10b6e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.598+0000 7f1a3b59e640 1 --2- 172.21.15.33:0/116822731 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a3c1028e0 0x7f1a3c102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.598+0000 7f1a3b59e640 1 -- 172.21.15.33:0/116822731 >> 172.21.15.33:0/116822731 conn(0x7f1a3c0fc120 msgr2=0x7f1a3c0fe540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:38.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.598+0000 7f1a3b59e640 1 -- 172.21.15.33:0/116822731 shutdown_connections 2024-03-19T02:57:38.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.598+0000 7f1a3b59e640 1 -- 172.21.15.33:0/116822731 wait complete. 2024-03-19T02:57:38.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.599+0000 7f1a3b59e640 1 Processor -- start 2024-03-19T02:57:38.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.599+0000 7f1a3b59e640 1 -- start start 2024-03-19T02:57:38.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.599+0000 7f1a3b59e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a3c1028e0 0x7f1a3c197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.599+0000 7f1a3b59e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a3c103200 0x7f1a3c198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.599+0000 7f1a3b59e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a3c10bd00 0x7f1a3c19c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.599+0000 7f1a3b59e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3c111960 con 0x7f1a3c103200 2024-03-19T02:57:38.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.599+0000 7f1a3b59e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3c1117e0 con 0x7f1a3c10bd00 2024-03-19T02:57:38.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.600+0000 7f1a3b59e640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3c111ae0 con 0x7f1a3c1028e0 2024-03-19T02:57:38.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.600+0000 7f1a33fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a3c103200 0x7f1a3c198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.600+0000 7f1a3a59c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a3c1028e0 0x7f1a3c197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.600+0000 7f1a33fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a3c103200 0x7f1a3c198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56156/0 (socket says 172.21.15.33:56156) 2024-03-19T02:57:38.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.600+0000 7f1a33fff640 1 -- 172.21.15.33:0/1018099474 learned_addr learned my addr 172.21.15.33:0/1018099474 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:38.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.600+0000 7f1a3ad9d640 1 --2- 172.21.15.33:0/1018099474 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a3c10bd00 0x7f1a3c19c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.600+0000 7f1a33fff640 1 -- 172.21.15.33:0/1018099474 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a3c1028e0 msgr2=0x7f1a3c197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:38.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.600+0000 7f1a33fff640 1 --2- 172.21.15.33:0/1018099474 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a3c1028e0 0x7f1a3c197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.600+0000 7f1a33fff640 1 -- 172.21.15.33:0/1018099474 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a3c10bd00 msgr2=0x7f1a3c19c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:38.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.600+0000 7f1a33fff640 1 --2- 172.21.15.33:0/1018099474 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a3c10bd00 0x7f1a3c19c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.600+0000 7f1a33fff640 1 -- 172.21.15.33:0/1018099474 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a3c19cf20 con 0x7f1a3c103200 2024-03-19T02:57:38.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.600+0000 7f1a3a59c640 1 --2- 172.21.15.33:0/1018099474 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a3c1028e0 0x7f1a3c197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:38.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.600+0000 7f1a33fff640 1 --2- 172.21.15.33:0/1018099474 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a3c103200 0x7f1a3c198470 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f1a24000c00 tx=0x7f1a24004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:38.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.600+0000 7f1a337fe640 1 -- 172.21.15.33:0/1018099474 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a24039050 con 0x7f1a3c103200 2024-03-19T02:57:38.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.600+0000 7f1a3b59e640 1 -- 172.21.15.33:0/1018099474 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a3c19d1b0 con 0x7f1a3c103200 2024-03-19T02:57:38.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.600+0000 7f1a337fe640 1 -- 172.21.15.33:0/1018099474 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a24045070 con 0x7f1a3c103200 2024-03-19T02:57:38.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.600+0000 7f1a337fe640 1 -- 172.21.15.33:0/1018099474 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a24040450 con 0x7f1a3c103200 2024-03-19T02:57:38.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.600+0000 7f1a3b59e640 1 -- 172.21.15.33:0/1018099474 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a3c1a4aa0 con 0x7f1a3c103200 2024-03-19T02:57:38.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.603+0000 7f1a337fe640 1 -- 172.21.15.33:0/1018099474 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f1a24003680 con 0x7f1a3c103200 2024-03-19T02:57:38.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.603+0000 7f1a43bec740 1 Processor -- start 2024-03-19T02:57:38.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.603+0000 7f1a43bec740 1 -- start start 2024-03-19T02:57:38.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.604+0000 7f1a43bec740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558023712560 0x558023712940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.604+0000 7f1a43bec740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558023712e80 0x55802371b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.604+0000 7f1a43bec740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55802371ba60 0x55802371de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.604+0000 7f1a43bec740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558023676160 con 0x558023712560 2024-03-19T02:57:38.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.604+0000 7f1a43bec740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55802361ee10 con 0x55802371ba60 2024-03-19T02:57:38.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.604+0000 7f1a43bec740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558023672200 con 0x558023712e80 2024-03-19T02:57:38.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.604+0000 7f1a337fe640 1 --2- 172.21.15.33:0/1018099474 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f1a280a2970 0x7f1a280a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.604+0000 7f1a3ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558023712560 0x558023712940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.604+0000 7f1a3a59c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558023712e80 0x55802371b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.604+0000 7f1a3ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558023712560 0x558023712940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56164/0 (socket says 172.21.15.33:56164) 2024-03-19T02:57:38.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.604+0000 7f1a3ad9d640 1 -- 172.21.15.33:0/395189483 learned_addr learned my addr 172.21.15.33:0/395189483 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:38.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.604+0000 7f1a33fff640 1 --2- 172.21.15.33:0/395189483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55802371ba60 0x55802371de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.604+0000 7f1a3ad9d640 1 --2- 172.21.15.33:0/1018099474 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f1a280a2970 0x7f1a280a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.604+0000 7f1a337fe640 1 -- 172.21.15.33:0/1018099474 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f1a240f0b50 con 0x7f1a3c103200 2024-03-19T02:57:38.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.604+0000 7f1a3a59c640 1 -- 172.21.15.33:0/395189483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55802371ba60 msgr2=0x55802371de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:38.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.604+0000 7f1a3a59c640 1 --2- 172.21.15.33:0/395189483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55802371ba60 0x55802371de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.604+0000 7f1a3a59c640 1 -- 172.21.15.33:0/395189483 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558023712560 msgr2=0x558023712940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:38.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.604+0000 7f1a3a59c640 1 --2- 172.21.15.33:0/395189483 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558023712560 0x558023712940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.604+0000 7f1a3a59c640 1 -- 172.21.15.33:0/395189483 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580234527e0 con 0x558023712e80 2024-03-19T02:57:38.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.604+0000 7f1a33fff640 1 --2- 172.21.15.33:0/395189483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55802371ba60 0x55802371de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:57:38.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.604+0000 7f1a3a59c640 1 --2- 172.21.15.33:0/395189483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558023712e80 0x55802371b520 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c009fd0 tx=0x7f1a2c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=327d3b2b2d7ad284 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:38.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.605+0000 7f1a3ad9d640 1 --2- 172.21.15.33:0/1018099474 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f1a280a2970 0x7f1a280a4e30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1a34009ba0 tx=0x7f1a34004ab0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:38.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.605+0000 7f1a30ff9640 1 -- 172.21.15.33:0/395189483 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a2c004030 con 0x558023712e80 2024-03-19T02:57:38.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.605+0000 7f1a30ff9640 1 -- 172.21.15.33:0/395189483 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a2c0054d0 con 0x558023712e80 2024-03-19T02:57:38.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.605+0000 7f1a30ff9640 1 -- 172.21.15.33:0/395189483 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a2c004ce0 con 0x558023712e80 2024-03-19T02:57:38.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.605+0000 7f1a43bec740 1 -- 172.21.15.33:0/395189483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558023712e80 msgr2=0x55802371b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:38.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.605+0000 7f1a43bec740 1 --2- 172.21.15.33:0/395189483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558023712e80 0x55802371b520 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c009fd0 tx=0x7f1a2c0118d0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.605+0000 7f1a43bec740 1 -- 172.21.15.33:0/395189483 shutdown_connections 2024-03-19T02:57:38.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.605+0000 7f1a43bec740 1 --2- 172.21.15.33:0/395189483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558023712e80 0x55802371b520 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.605+0000 7f1a43bec740 1 --2- 172.21.15.33:0/395189483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55802371ba60 0x55802371de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.605+0000 7f1a43bec740 1 --2- 172.21.15.33:0/395189483 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558023712560 0x558023712940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.605+0000 7f1a43bec740 1 -- 172.21.15.33:0/395189483 >> 172.21.15.33:0/395189483 conn(0x55802370cc40 msgr2=0x55802370d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:38.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.605+0000 7f1a43bec740 1 -- 172.21.15.33:0/395189483 shutdown_connections 2024-03-19T02:57:38.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.605+0000 7f1a43bec740 1 -- 172.21.15.33:0/395189483 wait complete. 2024-03-19T02:57:38.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.606+0000 7f1a43bec740 1 -- start start 2024-03-19T02:57:38.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.606+0000 7f1a43bec740 1 Processor -- start 2024-03-19T02:57:38.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.606+0000 7f1a43bec740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558023712560 0x558023718a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.606+0000 7f1a43bec740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558023712e80 0x558023718f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.606+0000 7f1a43bec740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55802371ba60 0x5580237194d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.606+0000 7f1a43bec740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55802371e850 con 0x55802371ba60 2024-03-19T02:57:38.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.606+0000 7f1a43bec740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5580236321a0 con 0x558023712560 2024-03-19T02:57:38.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.606+0000 7f1a43bec740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55802371e9d0 con 0x558023712e80 2024-03-19T02:57:38.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.606+0000 7f1a43bec740 10 client.0 _ll_register_callbacks cb 0x558023389c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:57:38.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.606+0000 7f1a3ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55802371ba60 0x5580237194d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.606+0000 7f1a3a59c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558023712560 0x558023718a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.606+0000 7f1a3a59c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558023712560 0x558023718a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:33486/0 (socket says 172.21.15.33:33486) 2024-03-19T02:57:38.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.606+0000 7f1a3ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55802371ba60 0x5580237194d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56180/0 (socket says 172.21.15.33:56180) 2024-03-19T02:57:38.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.606+0000 7f1a3a59c640 1 -- 172.21.15.33:0/1733952222 learned_addr learned my addr 172.21.15.33:0/1733952222 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:38.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.606+0000 7f1a33fff640 1 --2- 172.21.15.33:0/1733952222 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558023712e80 0x558023718f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.606+0000 7f1a3ad9d640 1 -- 172.21.15.33:0/1733952222 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558023712e80 msgr2=0x558023718f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:38.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.606+0000 7f1a3ad9d640 1 --2- 172.21.15.33:0/1733952222 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558023712e80 0x558023718f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.606+0000 7f1a3ad9d640 1 -- 172.21.15.33:0/1733952222 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558023712560 msgr2=0x558023718a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:38.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.606+0000 7f1a3ad9d640 1 --2- 172.21.15.33:0/1733952222 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558023712560 0x558023718a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.607+0000 7f1a3ad9d640 1 -- 172.21.15.33:0/1733952222 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580234b2f70 con 0x55802371ba60 2024-03-19T02:57:38.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.607+0000 7f1a3a59c640 1 --2- 172.21.15.33:0/1733952222 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558023712560 0x558023718a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:38.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.607+0000 7f1a3ad9d640 1 --2- 172.21.15.33:0/1733952222 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55802371ba60 0x5580237194d0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f1a340049c0 tx=0x7f1a34004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bd0cddfa3b6ea00 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:38.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.607+0000 7f1a21ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:57:38.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.607+0000 7f1a21ffb640 1 -- 172.21.15.33:0/1733952222 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a3401c050 con 0x55802371ba60 2024-03-19T02:57:38.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.607+0000 7f1a21ffb640 1 -- 172.21.15.33:0/1733952222 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f1a34004d20 con 0x55802371ba60 2024-03-19T02:57:38.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.607+0000 7f1a43bec740 1 -- 172.21.15.33:0/1733952222 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a2c0054d0 con 0x55802371ba60 2024-03-19T02:57:38.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.607+0000 7f1a21ffb640 1 -- 172.21.15.33:0/1733952222 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1a34022620 con 0x55802371ba60 2024-03-19T02:57:38.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.607+0000 7f1a43bec740 10 client.16584 fetch_fsmap learned FSMap version 25 2024-03-19T02:57:38.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.607+0000 7f1a43bec740 1 -- 172.21.15.33:0/1733952222 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55802334a170 con 0x55802371ba60 2024-03-19T02:57:38.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.607+0000 7f1a21ffb640 1 -- 172.21.15.33:0/1733952222 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a2c0054d0 con 0x55802371ba60 2024-03-19T02:57:38.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.607+0000 7f1a21ffb640 1 -- 172.21.15.33:0/1733952222 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f1a3402c070 con 0x55802371ba60 2024-03-19T02:57:38.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.607+0000 7f1a43bec740 10 client.16584 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:57:38.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.607+0000 7f1a43bec740 10 client.16584 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:57:38.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.608+0000 7f1a43bec740 20 client.16584 populate_metadata read hostname 'smithi033' 2024-03-19T02:57:38.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.608+0000 7f1a43bec740 1 --2- 172.21.15.33:0/1733952222 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55802372be70 0x55802372e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.608+0000 7f1a43bec740 4 client.16584 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:57:38.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.608+0000 7f1a43bec740 1 -- 172.21.15.33:0/1733952222 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5580236836a0 con 0x55802372be70 2024-03-19T02:57:38.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.608+0000 7f1a3a59c640 1 --2- 172.21.15.33:0/1733952222 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55802372be70 0x55802372e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.608+0000 7f1a21ffb640 10 client.16584 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:57:38.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.608+0000 7f1a3a59c640 1 --2- 172.21.15.33:0/1733952222 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55802372be70 0x55802372e280 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=144fcb7bb4319c55 server_cookie=8b46e9808bfe9b0d in_seq=0 out_seq=0 2024-03-19T02:57:38.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.609+0000 7f1a21ffb640 1 -- 172.21.15.33:0/1733952222 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5580236836a0 con 0x55802372be70 2024-03-19T02:57:38.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.609+0000 7f1a21ffb640 10 client.16584 handle_command_reply: tid=0 2024-03-19T02:57:38.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.609+0000 7f1a43bec740 1 client.16584 shutdown 2024-03-19T02:57:38.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.609+0000 7f1a43bec740 1 -- 172.21.15.33:0/1733952222 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55802371ba60 msgr2=0x5580237194d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:38.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.609+0000 7f1a43bec740 1 --2- 172.21.15.33:0/1733952222 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55802371ba60 0x5580237194d0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f1a340049c0 tx=0x7f1a34004a40 comp rx=0 tx=0).stop 2024-03-19T02:57:38.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.609+0000 7f1a43bec740 1 -- 172.21.15.33:0/1733952222 shutdown_connections 2024-03-19T02:57:38.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.609+0000 7f1a43bec740 1 --2- 172.21.15.33:0/1733952222 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55802372be70 0x55802372e280 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.609+0000 7f1a43bec740 1 --2- 172.21.15.33:0/1733952222 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558023712e80 0x558023718f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.609+0000 7f1a43bec740 1 --2- 172.21.15.33:0/1733952222 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55802371ba60 0x5580237194d0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.609+0000 7f1a21ffb640 0 client.16584 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:57:38.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.609+0000 7f1a43bec740 1 --2- 172.21.15.33:0/1733952222 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558023712560 0x558023718a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.609+0000 7f1a43bec740 1 -- 172.21.15.33:0/1733952222 >> 172.21.15.33:0/1733952222 conn(0x55802370cc40 msgr2=0x55802370e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:38.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.609+0000 7f1a43bec740 1 -- 172.21.15.33:0/1733952222 shutdown_connections 2024-03-19T02:57:38.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.609+0000 7f1a43bec740 1 -- 172.21.15.33:0/1733952222 wait complete. 2024-03-19T02:57:38.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.609+0000 7f1a43bec740 20 client.16584 trim_cache size 0 max 16384 2024-03-19T02:57:38.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.627+0000 7f1a43bec740 1 Processor -- start 2024-03-19T02:57:38.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.627+0000 7f1a43bec740 1 -- start start 2024-03-19T02:57:38.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.627+0000 7f1a43bec740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558023712560 0x5580237192a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.627+0000 7f1a43bec740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558023712e80 0x558023727620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.627+0000 7f1a43bec740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55802371ba60 0x558023733b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.627+0000 7f1a43bec740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3c1af090 con 0x558023712e80 2024-03-19T02:57:38.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.627+0000 7f1a43bec740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3c1ac960 con 0x558023712560 2024-03-19T02:57:38.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.627+0000 7f1a43bec740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3c1aa790 con 0x55802371ba60 2024-03-19T02:57:38.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.627+0000 7f1a33fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558023712e80 0x558023727620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.627+0000 7f1a33fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558023712e80 0x558023727620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56196/0 (socket says 172.21.15.33:56196) 2024-03-19T02:57:38.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.627+0000 7f1a33fff640 1 -- 172.21.15.33:0/3858231990 learned_addr learned my addr 172.21.15.33:0/3858231990 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:38.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.627+0000 7f1a3a59c640 1 --2- 172.21.15.33:0/3858231990 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558023712560 0x5580237192a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.627+0000 7f1a3ad9d640 1 --2- 172.21.15.33:0/3858231990 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55802371ba60 0x558023733b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.627+0000 7f1a33fff640 1 -- 172.21.15.33:0/3858231990 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55802371ba60 msgr2=0x558023733b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:38.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.627+0000 7f1a33fff640 1 --2- 172.21.15.33:0/3858231990 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55802371ba60 0x558023733b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.627+0000 7f1a33fff640 1 -- 172.21.15.33:0/3858231990 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558023712560 msgr2=0x5580237192a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:38.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.627+0000 7f1a33fff640 1 --2- 172.21.15.33:0/3858231990 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558023712560 0x5580237192a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.627+0000 7f1a33fff640 1 -- 172.21.15.33:0/3858231990 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558023714630 con 0x558023712e80 2024-03-19T02:57:38.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.627+0000 7f1a3ad9d640 1 --2- 172.21.15.33:0/3858231990 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55802371ba60 0x558023733b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:57:38.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.628+0000 7f1a33fff640 1 --2- 172.21.15.33:0/3858231990 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558023712e80 0x558023727620 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f1a240b98f0 tx=0x7f1a240e78c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9862fc56b3c2b03 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:38.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.628+0000 7f1a23fff640 1 -- 172.21.15.33:0/3858231990 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a240bc880 con 0x558023712e80 2024-03-19T02:57:38.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.628+0000 7f1a23fff640 1 -- 172.21.15.33:0/3858231990 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a2403a040 con 0x558023712e80 2024-03-19T02:57:38.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.628+0000 7f1a23fff640 1 -- 172.21.15.33:0/3858231990 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a240ee170 con 0x558023712e80 2024-03-19T02:57:38.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.628+0000 7f1a43bec740 1 -- 172.21.15.33:0/3858231990 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558023712e80 msgr2=0x558023727620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:38.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.628+0000 7f1a43bec740 1 --2- 172.21.15.33:0/3858231990 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558023712e80 0x558023727620 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f1a240b98f0 tx=0x7f1a240e78c0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.628+0000 7f1a43bec740 1 -- 172.21.15.33:0/3858231990 shutdown_connections 2024-03-19T02:57:38.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.628+0000 7f1a43bec740 1 --2- 172.21.15.33:0/3858231990 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55802371ba60 0x558023733b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.628+0000 7f1a43bec740 1 --2- 172.21.15.33:0/3858231990 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558023712e80 0x558023727620 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.628+0000 7f1a43bec740 1 --2- 172.21.15.33:0/3858231990 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558023712560 0x5580237192a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.628+0000 7f1a43bec740 1 -- 172.21.15.33:0/3858231990 >> 172.21.15.33:0/3858231990 conn(0x55802370cc40 msgr2=0x55802370b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:38.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.628+0000 7f1a43bec740 1 -- 172.21.15.33:0/3858231990 shutdown_connections 2024-03-19T02:57:38.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.628+0000 7f1a43bec740 1 -- 172.21.15.33:0/3858231990 wait complete. 2024-03-19T02:57:38.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.629+0000 7f1a43bec740 1 -- start start 2024-03-19T02:57:38.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.629+0000 7f1a43bec740 1 Processor -- start 2024-03-19T02:57:38.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.629+0000 7f1a43bec740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558023712560 0x558023720e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.629+0000 7f1a43bec740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558023712e80 0x558023721350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.629+0000 7f1a43bec740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55802371ba60 0x558023792190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.629+0000 7f1a43bec740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3c1aae20 con 0x558023712560 2024-03-19T02:57:38.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.629+0000 7f1a43bec740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3c1ac290 con 0x558023712e80 2024-03-19T02:57:38.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.629+0000 7f1a43bec740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3c1ab4f0 con 0x55802371ba60 2024-03-19T02:57:38.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.629+0000 7f1a43bec740 10 client.0 _ll_register_callbacks cb 0x7f1a34013530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:57:38.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.629+0000 7f1a3a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558023712560 0x558023720e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.629+0000 7f1a3a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558023712560 0x558023720e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56210/0 (socket says 172.21.15.33:56210) 2024-03-19T02:57:38.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.629+0000 7f1a3a59c640 1 -- 172.21.15.33:0/3426257931 learned_addr learned my addr 172.21.15.33:0/3426257931 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:38.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.629+0000 7f1a33fff640 1 --2- 172.21.15.33:0/3426257931 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558023712e80 0x558023721350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.629+0000 7f1a3ad9d640 1 --2- 172.21.15.33:0/3426257931 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55802371ba60 0x558023792190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.629+0000 7f1a3a59c640 1 -- 172.21.15.33:0/3426257931 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55802371ba60 msgr2=0x558023792190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:38.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.629+0000 7f1a3a59c640 1 --2- 172.21.15.33:0/3426257931 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55802371ba60 0x558023792190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.629+0000 7f1a3a59c640 1 -- 172.21.15.33:0/3426257931 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558023712e80 msgr2=0x558023721350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:38.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.629+0000 7f1a3a59c640 1 --2- 172.21.15.33:0/3426257931 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558023712e80 0x558023721350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.629+0000 7f1a3a59c640 1 -- 172.21.15.33:0/3426257931 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558023792720 con 0x558023712560 2024-03-19T02:57:38.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.629+0000 7f1a3ad9d640 1 --2- 172.21.15.33:0/3426257931 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55802371ba60 0x558023792190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:57:38.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.630+0000 7f1a3a59c640 1 --2- 172.21.15.33:0/3426257931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558023712560 0x558023720e10 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c0058c0 tx=0x7f1a2c009d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf37d0c3d5f975fd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:38.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.630+0000 7f1a21ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:57:38.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.630+0000 7f1a21ffb640 1 -- 172.21.15.33:0/3426257931 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a2c0035f0 con 0x558023712560 2024-03-19T02:57:38.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.630+0000 7f1a21ffb640 1 -- 172.21.15.33:0/3426257931 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f1a2c016040 con 0x558023712560 2024-03-19T02:57:38.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.630+0000 7f1a43bec740 1 -- 172.21.15.33:0/3426257931 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a2403a040 con 0x558023712560 2024-03-19T02:57:38.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.630+0000 7f1a21ffb640 1 -- 172.21.15.33:0/3426257931 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1a2c03a050 con 0x558023712560 2024-03-19T02:57:38.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.630+0000 7f1a43bec740 10 client.16590 fetch_fsmap learned FSMap version 25 2024-03-19T02:57:38.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.630+0000 7f1a43bec740 1 -- 172.21.15.33:0/3426257931 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558023793a60 con 0x558023712560 2024-03-19T02:57:38.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.630+0000 7f1a21ffb640 1 -- 172.21.15.33:0/3426257931 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a2403a040 con 0x558023712560 2024-03-19T02:57:38.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.630+0000 7f1a21ffb640 1 -- 172.21.15.33:0/3426257931 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f1a2c00f860 con 0x558023712560 2024-03-19T02:57:38.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.630+0000 7f1a43bec740 10 client.16590 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:57:38.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.630+0000 7f1a43bec740 10 client.16590 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:57:38.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.630+0000 7f1a43bec740 20 client.16590 populate_metadata read hostname 'smithi033' 2024-03-19T02:57:38.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.630+0000 7f1a43bec740 1 --2- 172.21.15.33:0/3426257931 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x558023794050 0x558023796470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.630+0000 7f1a43bec740 4 client.16590 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:57:38.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.630+0000 7f1a43bec740 1 -- 172.21.15.33:0/3426257931 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55802367d9f0 con 0x558023794050 2024-03-19T02:57:38.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.631+0000 7f1a33fff640 1 --2- 172.21.15.33:0/3426257931 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x558023794050 0x558023796470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.631+0000 7f1a21ffb640 10 client.16590 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:57:38.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.631+0000 7f1a33fff640 1 --2- 172.21.15.33:0/3426257931 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x558023794050 0x558023796470 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e3dcbb31d1a7f88 server_cookie=f767b5e19610c798 in_seq=0 out_seq=0 2024-03-19T02:57:38.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.632+0000 7f1a21ffb640 1 -- 172.21.15.33:0/3426257931 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55802367d9f0 con 0x558023794050 2024-03-19T02:57:38.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.632+0000 7f1a21ffb640 10 client.16590 handle_command_reply: tid=0 2024-03-19T02:57:38.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.632+0000 7f1a43bec740 1 client.16590 shutdown 2024-03-19T02:57:38.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.632+0000 7f1a43bec740 1 -- 172.21.15.33:0/3426257931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558023712560 msgr2=0x558023720e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:38.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.632+0000 7f1a43bec740 1 --2- 172.21.15.33:0/3426257931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558023712560 0x558023720e10 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c0058c0 tx=0x7f1a2c009d10 comp rx=0 tx=0).stop 2024-03-19T02:57:38.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.632+0000 7f1a43bec740 1 -- 172.21.15.33:0/3426257931 shutdown_connections 2024-03-19T02:57:38.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.632+0000 7f1a43bec740 1 --2- 172.21.15.33:0/3426257931 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x558023794050 0x558023796470 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.632+0000 7f1a43bec740 1 --2- 172.21.15.33:0/3426257931 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55802371ba60 0x558023792190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.632+0000 7f1a43bec740 1 --2- 172.21.15.33:0/3426257931 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558023712e80 0x558023721350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.632+0000 7f1a43bec740 1 --2- 172.21.15.33:0/3426257931 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558023712560 0x558023720e10 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.632+0000 7f1a43bec740 1 -- 172.21.15.33:0/3426257931 >> 172.21.15.33:0/3426257931 conn(0x55802370cc40 msgr2=0x558023719120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:38.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.632+0000 7f1a21ffb640 0 client.16590 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:57:38.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.632+0000 7f1a43bec740 1 -- 172.21.15.33:0/3426257931 shutdown_connections 2024-03-19T02:57:38.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.632+0000 7f1a43bec740 1 -- 172.21.15.33:0/3426257931 wait complete. 2024-03-19T02:57:38.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.633+0000 7f1a43bec740 20 client.16590 trim_cache size 0 max 16384 2024-03-19T02:57:38.640 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:57:38.640 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-03-19T02:57:38.640 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "e5fb108a-4d36-4b08-af7c-bdce0977670a", 2024-03-19T02:57:38.640 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-03-19T02:57:38.640 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:57:38.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.633+0000 7f1a3b59e640 1 -- 172.21.15.33:0/1018099474 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f1a280a2970 msgr2=0x7f1a280a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:38.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.633+0000 7f1a3b59e640 1 --2- 172.21.15.33:0/1018099474 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f1a280a2970 0x7f1a280a4e30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1a34009ba0 tx=0x7f1a34004ab0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.633+0000 7f1a3b59e640 1 -- 172.21.15.33:0/1018099474 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a3c103200 msgr2=0x7f1a3c198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:38.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.633+0000 7f1a3b59e640 1 --2- 172.21.15.33:0/1018099474 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a3c103200 0x7f1a3c198470 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f1a24000c00 tx=0x7f1a24004040 comp rx=0 tx=0).stop 2024-03-19T02:57:38.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.633+0000 7f1a3b59e640 1 -- 172.21.15.33:0/1018099474 shutdown_connections 2024-03-19T02:57:38.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.633+0000 7f1a3b59e640 1 --2- 172.21.15.33:0/1018099474 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f1a280a2970 0x7f1a280a4e30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.633+0000 7f1a3b59e640 1 --2- 172.21.15.33:0/1018099474 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a3c10bd00 0x7f1a3c19c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.633+0000 7f1a3b59e640 1 --2- 172.21.15.33:0/1018099474 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1a3c103200 0x7f1a3c198470 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.633+0000 7f1a3b59e640 1 --2- 172.21.15.33:0/1018099474 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1a3c1028e0 0x7f1a3c197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.633+0000 7f1a3b59e640 1 -- 172.21.15.33:0/1018099474 >> 172.21.15.33:0/1018099474 conn(0x7f1a3c0fc120 msgr2=0x7f1a3c0fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:38.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.633+0000 7f1a3b59e640 1 -- 172.21.15.33:0/1018099474 shutdown_connections 2024-03-19T02:57:38.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.633+0000 7f1a3b59e640 1 -- 172.21.15.33:0/1018099474 wait complete. 2024-03-19T02:57:38.645 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:57:38.645 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:57:38.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:38 smithi096 ceph-mon[35946]: pgmap v128: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:57:38.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.749+0000 7f05370b5640 1 -- 172.21.15.33:0/4163306640 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05301017c0 msgr2=0x7f0530101c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:38.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.749+0000 7f05370b5640 1 --2- 172.21.15.33:0/4163306640 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05301017c0 0x7f0530101c40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0524009900 tx=0x7f052402bc30 comp rx=0 tx=0).stop 2024-03-19T02:57:38.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.749+0000 7f05370b5640 1 -- 172.21.15.33:0/4163306640 shutdown_connections 2024-03-19T02:57:38.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.749+0000 7f05370b5640 1 --2- 172.21.15.33:0/4163306640 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0530102180 0x7f053010e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.749+0000 7f05370b5640 1 --2- 172.21.15.33:0/4163306640 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05301017c0 0x7f0530101c40 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.749+0000 7f05370b5640 1 --2- 172.21.15.33:0/4163306640 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f05301077d0 0x7f0530107bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.749+0000 7f05370b5640 1 -- 172.21.15.33:0/4163306640 >> 172.21.15.33:0/4163306640 conn(0x7f05300fd2b0 msgr2=0x7f05300ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:38.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.750+0000 7f05370b5640 1 -- 172.21.15.33:0/4163306640 shutdown_connections 2024-03-19T02:57:38.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.750+0000 7f05370b5640 1 -- 172.21.15.33:0/4163306640 wait complete. 2024-03-19T02:57:38.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.750+0000 7f05370b5640 1 Processor -- start 2024-03-19T02:57:38.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.750+0000 7f05370b5640 1 -- start start 2024-03-19T02:57:38.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.750+0000 7f05370b5640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05301017c0 0x7f0530073b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.750+0000 7f05370b5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0530102180 0x7f0530074060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.750+0000 7f05370b5640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05301077d0 0x7f05300745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.750+0000 7f05370b5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0530111a90 con 0x7f0530102180 2024-03-19T02:57:38.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.750+0000 7f05370b5640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0530111910 con 0x7f05301077d0 2024-03-19T02:57:38.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.750+0000 7f05370b5640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f0530111c10 con 0x7f05301017c0 2024-03-19T02:57:38.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.750+0000 7f052ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0530102180 0x7f0530074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.751+0000 7f052ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0530102180 0x7f0530074060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56228/0 (socket says 172.21.15.33:56228) 2024-03-19T02:57:38.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.751+0000 7f052ffff640 1 -- 172.21.15.33:0/3208853440 learned_addr learned my addr 172.21.15.33:0/3208853440 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:38.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.751+0000 7f0535652640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05301077d0 0x7f05300745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.751+0000 7f0534e51640 1 --2- 172.21.15.33:0/3208853440 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05301017c0 0x7f0530073b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.751+0000 7f052ffff640 1 -- 172.21.15.33:0/3208853440 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05301017c0 msgr2=0x7f0530073b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:38.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.751+0000 7f052ffff640 1 --2- 172.21.15.33:0/3208853440 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05301017c0 0x7f0530073b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.751+0000 7f052ffff640 1 -- 172.21.15.33:0/3208853440 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05301077d0 msgr2=0x7f05300745a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:38.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.751+0000 7f052ffff640 1 --2- 172.21.15.33:0/3208853440 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05301077d0 0x7f05300745a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.751+0000 7f052ffff640 1 -- 172.21.15.33:0/3208853440 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05301a64d0 con 0x7f0530102180 2024-03-19T02:57:38.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.751+0000 7f0535652640 1 --2- 172.21.15.33:0/3208853440 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05301077d0 0x7f05300745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:38.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.751+0000 7f052ffff640 1 --2- 172.21.15.33:0/3208853440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0530102180 0x7f0530074060 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f0524006010 tx=0x7f0524005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:38.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.751+0000 7f052dffb640 1 -- 172.21.15.33:0/3208853440 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0524043020 con 0x7f0530102180 2024-03-19T02:57:38.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.751+0000 7f05370b5640 1 -- 172.21.15.33:0/3208853440 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05301a6760 con 0x7f0530102180 2024-03-19T02:57:38.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.751+0000 7f052dffb640 1 -- 172.21.15.33:0/3208853440 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0524004350 con 0x7f0530102180 2024-03-19T02:57:38.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.751+0000 7f052dffb640 1 -- 172.21.15.33:0/3208853440 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0524003a10 con 0x7f0530102180 2024-03-19T02:57:38.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.751+0000 7f05370b5640 1 -- 172.21.15.33:0/3208853440 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05301a6cb0 con 0x7f0530102180 2024-03-19T02:57:38.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.753+0000 7f052dffb640 1 -- 172.21.15.33:0/3208853440 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f0524046020 con 0x7f0530102180 2024-03-19T02:57:38.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.754+0000 7f0539d76740 1 Processor -- start 2024-03-19T02:57:38.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.754+0000 7f0539d76740 1 -- start start 2024-03-19T02:57:38.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.754+0000 7f0539d76740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfbedbea30 0x55dfbedbee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.754+0000 7f052dffb640 1 --2- 172.21.15.33:0/3208853440 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f05180a2970 0x7f05180a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.754+0000 7f0539d76740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dfbedbf350 0x55dfbedc79f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.754+0000 7f0539d76740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfbedc7f30 0x55dfbedca320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.754+0000 7f0539d76740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55dfbed22500 con 0x55dfbedc7f30 2024-03-19T02:57:38.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.754+0000 7f0539d76740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55dfbeccabe0 con 0x55dfbedbf350 2024-03-19T02:57:38.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.754+0000 7f0539d76740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55dfbed1e5a0 con 0x55dfbedbea30 2024-03-19T02:57:38.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.754+0000 7f0534e51640 1 --2- 172.21.15.33:0/3208853440 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f05180a2970 0x7f05180a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.754+0000 7f0535652640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfbedc7f30 0x55dfbedca320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.754+0000 7f052ffff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dfbedbf350 0x55dfbedc79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.754+0000 7f0535652640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfbedbea30 0x55dfbedbee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.754+0000 7f052ffff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dfbedbf350 0x55dfbedc79f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:33536/0 (socket says 172.21.15.33:33536) 2024-03-19T02:57:38.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.754+0000 7f052ffff640 1 -- 172.21.15.33:0/1288657090 learned_addr learned my addr 172.21.15.33:0/1288657090 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:38.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.754+0000 7f052dffb640 1 -- 172.21.15.33:0/3208853440 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f05240f0bb0 con 0x7f0530102180 2024-03-19T02:57:38.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.754+0000 7f0535652640 1 -- 172.21.15.33:0/1288657090 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dfbedbf350 msgr2=0x55dfbedc79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:38.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.754+0000 7f0535652640 1 --2- 172.21.15.33:0/1288657090 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dfbedbf350 0x55dfbedc79f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.754+0000 7f0535652640 1 -- 172.21.15.33:0/1288657090 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfbedc7f30 msgr2=0x55dfbedca320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:38.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.754+0000 7f0535652640 1 --2- 172.21.15.33:0/1288657090 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfbedc7f30 0x55dfbedca320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.754+0000 7f0535652640 1 -- 172.21.15.33:0/1288657090 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfbecec000 con 0x55dfbedbea30 2024-03-19T02:57:38.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.754+0000 7f052ffff640 1 --2- 172.21.15.33:0/1288657090 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dfbedbf350 0x55dfbedc79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:38.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.755+0000 7f0534e51640 1 --2- 172.21.15.33:0/3208853440 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f05180a2970 0x7f05180a4e30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0528005fd0 tx=0x7f0528009500 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:38.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.755+0000 7f0535652640 1 --2- 172.21.15.33:0/1288657090 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfbedbea30 0x55dfbedbee10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0520010c20 tx=0x7f0520005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2dcf65915eed82d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:38.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.755+0000 7f0516ffd640 1 -- 172.21.15.33:0/1288657090 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0520016070 con 0x55dfbedbea30 2024-03-19T02:57:38.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.755+0000 7f0516ffd640 1 -- 172.21.15.33:0/1288657090 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f052000fe50 con 0x55dfbedbea30 2024-03-19T02:57:38.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.755+0000 7f0516ffd640 1 -- 172.21.15.33:0/1288657090 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0520019040 con 0x55dfbedbea30 2024-03-19T02:57:38.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.755+0000 7f0539d76740 1 -- 172.21.15.33:0/1288657090 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfbedbea30 msgr2=0x55dfbedbee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:38.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.755+0000 7f0539d76740 1 --2- 172.21.15.33:0/1288657090 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfbedbea30 0x55dfbedbee10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0520010c20 tx=0x7f0520005f90 comp rx=0 tx=0).stop 2024-03-19T02:57:38.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.756+0000 7f0539d76740 1 -- 172.21.15.33:0/1288657090 shutdown_connections 2024-03-19T02:57:38.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.756+0000 7f0539d76740 1 --2- 172.21.15.33:0/1288657090 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfbedc7f30 0x55dfbedca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.756+0000 7f0539d76740 1 --2- 172.21.15.33:0/1288657090 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dfbedbf350 0x55dfbedc79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.756+0000 7f0539d76740 1 --2- 172.21.15.33:0/1288657090 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfbedbea30 0x55dfbedbee10 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.756+0000 7f0539d76740 1 -- 172.21.15.33:0/1288657090 >> 172.21.15.33:0/1288657090 conn(0x55dfbedb9160 msgr2=0x55dfbedb9540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:38.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.756+0000 7f0539d76740 1 -- 172.21.15.33:0/1288657090 shutdown_connections 2024-03-19T02:57:38.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.756+0000 7f0539d76740 1 -- 172.21.15.33:0/1288657090 wait complete. 2024-03-19T02:57:38.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.756+0000 7f0539d76740 1 -- start start 2024-03-19T02:57:38.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.756+0000 7f0539d76740 1 Processor -- start 2024-03-19T02:57:38.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.756+0000 7f0539d76740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dfbedbea30 0x55dfbedc4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.756+0000 7f0539d76740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfbedbf350 0x55dfbedc5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.756+0000 7f0539d76740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfbedc7f30 0x55dfbedc5970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.756+0000 7f0539d76740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55dfbedcad20 con 0x55dfbedbf350 2024-03-19T02:57:38.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.756+0000 7f0539d76740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55dfbecde160 con 0x55dfbedbea30 2024-03-19T02:57:38.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.756+0000 7f0539d76740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55dfbedcaea0 con 0x55dfbedc7f30 2024-03-19T02:57:38.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.756+0000 7f0539d76740 10 client.0 _ll_register_callbacks cb 0x55dfbed1aa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:57:38.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.756+0000 7f052ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfbedbf350 0x55dfbedc5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.757+0000 7f0534e51640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dfbedbea30 0x55dfbedc4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.757+0000 7f052ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfbedbf350 0x55dfbedc5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56248/0 (socket says 172.21.15.33:56248) 2024-03-19T02:57:38.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.757+0000 7f052ffff640 1 -- 172.21.15.33:0/2908302962 learned_addr learned my addr 172.21.15.33:0/2908302962 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:38.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.757+0000 7f0535652640 1 --2- 172.21.15.33:0/2908302962 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfbedc7f30 0x55dfbedc5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.757+0000 7f052ffff640 1 -- 172.21.15.33:0/2908302962 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfbedc7f30 msgr2=0x55dfbedc5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:38.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.757+0000 7f052ffff640 1 --2- 172.21.15.33:0/2908302962 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfbedc7f30 0x55dfbedc5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.757+0000 7f052ffff640 1 -- 172.21.15.33:0/2908302962 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dfbedbea30 msgr2=0x55dfbedc4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:38.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.757+0000 7f052ffff640 1 --2- 172.21.15.33:0/2908302962 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dfbedbea30 0x55dfbedc4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.757+0000 7f052ffff640 1 -- 172.21.15.33:0/2908302962 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfbeb007d0 con 0x55dfbedbf350 2024-03-19T02:57:38.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.757+0000 7f0534e51640 1 --2- 172.21.15.33:0/2908302962 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dfbedbea30 0x55dfbedc4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:38.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.757+0000 7f0535652640 1 --2- 172.21.15.33:0/2908302962 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfbedc7f30 0x55dfbedc5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:57:38.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.757+0000 7f052ffff640 1 --2- 172.21.15.33:0/2908302962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfbedbf350 0x55dfbedc5430 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f05240bcc70 tx=0x7f0524005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfcdff529b36bd0d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:38.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.757+0000 7f050ffff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:57:38.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.757+0000 7f050ffff640 1 -- 172.21.15.33:0/2908302962 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05240c10c0 con 0x55dfbedbf350 2024-03-19T02:57:38.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.757+0000 7f050ffff640 1 -- 172.21.15.33:0/2908302962 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f05240b95e0 con 0x55dfbedbf350 2024-03-19T02:57:38.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.757+0000 7f0539d76740 1 -- 172.21.15.33:0/2908302962 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f052000fe50 con 0x55dfbedbf350 2024-03-19T02:57:38.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.757+0000 7f050ffff640 1 -- 172.21.15.33:0/2908302962 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f05240c31c0 con 0x55dfbedbf350 2024-03-19T02:57:38.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.757+0000 7f0539d76740 10 client.16602 fetch_fsmap learned FSMap version 25 2024-03-19T02:57:38.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.757+0000 7f0539d76740 1 -- 172.21.15.33:0/2908302962 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55dfbea363f0 con 0x55dfbedbf350 2024-03-19T02:57:38.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.757+0000 7f050ffff640 1 -- 172.21.15.33:0/2908302962 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f052000fe50 con 0x55dfbedbf350 2024-03-19T02:57:38.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.758+0000 7f050ffff640 1 -- 172.21.15.33:0/2908302962 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f05240c2b90 con 0x55dfbedbf350 2024-03-19T02:57:38.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.758+0000 7f0539d76740 10 client.16602 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:57:38.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.758+0000 7f0539d76740 10 client.16602 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:57:38.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.758+0000 7f0539d76740 20 client.16602 populate_metadata read hostname 'smithi033' 2024-03-19T02:57:38.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.758+0000 7f0539d76740 1 --2- 172.21.15.33:0/2908302962 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55dfbedd8290 0x55dfbedda730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.758+0000 7f0539d76740 4 client.16602 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:57:38.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.758+0000 7f0539d76740 1 -- 172.21.15.33:0/2908302962 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dfbed2fa40 con 0x55dfbedd8290 2024-03-19T02:57:38.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.758+0000 7f0534e51640 1 --2- 172.21.15.33:0/2908302962 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55dfbedd8290 0x55dfbedda730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.758+0000 7f050ffff640 10 client.16602 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:57:38.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.758+0000 7f0534e51640 1 --2- 172.21.15.33:0/2908302962 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55dfbedd8290 0x55dfbedda730 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26cc7c9bdae0428b server_cookie=832319a0de46e93 in_seq=0 out_seq=0 2024-03-19T02:57:38.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.759+0000 7f050ffff640 1 -- 172.21.15.33:0/2908302962 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dfbed2fa40 con 0x55dfbedd8290 2024-03-19T02:57:38.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.759+0000 7f050ffff640 10 client.16602 handle_command_reply: tid=0 2024-03-19T02:57:38.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.759+0000 7f0539d76740 1 client.16602 shutdown 2024-03-19T02:57:38.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.759+0000 7f0539d76740 1 -- 172.21.15.33:0/2908302962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfbedbf350 msgr2=0x55dfbedc5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:38.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.759+0000 7f0539d76740 1 --2- 172.21.15.33:0/2908302962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfbedbf350 0x55dfbedc5430 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f05240bcc70 tx=0x7f0524005c00 comp rx=0 tx=0).stop 2024-03-19T02:57:38.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.759+0000 7f0539d76740 1 -- 172.21.15.33:0/2908302962 shutdown_connections 2024-03-19T02:57:38.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.759+0000 7f0539d76740 1 --2- 172.21.15.33:0/2908302962 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55dfbedd8290 0x55dfbedda730 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.759+0000 7f0539d76740 1 --2- 172.21.15.33:0/2908302962 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfbedc7f30 0x55dfbedc5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.759+0000 7f0539d76740 1 --2- 172.21.15.33:0/2908302962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfbedbf350 0x55dfbedc5430 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.759+0000 7f0539d76740 1 --2- 172.21.15.33:0/2908302962 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dfbedbea30 0x55dfbedc4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.759+0000 7f050ffff640 0 client.16602 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:57:38.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.759+0000 7f0539d76740 1 -- 172.21.15.33:0/2908302962 >> 172.21.15.33:0/2908302962 conn(0x55dfbedb9160 msgr2=0x55dfbedbaeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:38.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.759+0000 7f0539d76740 1 -- 172.21.15.33:0/2908302962 shutdown_connections 2024-03-19T02:57:38.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.759+0000 7f0539d76740 1 -- 172.21.15.33:0/2908302962 wait complete. 2024-03-19T02:57:38.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.759+0000 7f0539d76740 20 client.16602 trim_cache size 0 max 16384 2024-03-19T02:57:38.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:38 smithi155 ceph-mon[35804]: pgmap v128: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:57:38.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.777+0000 7f0539d76740 1 Processor -- start 2024-03-19T02:57:38.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.777+0000 7f0539d76740 1 -- start start 2024-03-19T02:57:38.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.777+0000 7f0539d76740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfbedbea30 0x55dfbedc7620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.777+0000 7f0539d76740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfbedbf350 0x55dfbedc3e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.777+0000 7f0539d76740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dfbedc7f30 0x55dfbedd3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.777+0000 7f0539d76740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f05301b35d0 con 0x55dfbedbea30 2024-03-19T02:57:38.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.777+0000 7f0539d76740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f05301b0ea0 con 0x55dfbedc7f30 2024-03-19T02:57:38.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.777+0000 7f0539d76740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f05301aecd0 con 0x55dfbedbf350 2024-03-19T02:57:38.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.777+0000 7f0534e51640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfbedbea30 0x55dfbedc7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.777+0000 7f0534e51640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfbedbea30 0x55dfbedc7620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56262/0 (socket says 172.21.15.33:56262) 2024-03-19T02:57:38.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.777+0000 7f0534e51640 1 -- 172.21.15.33:0/3431543371 learned_addr learned my addr 172.21.15.33:0/3431543371 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:38.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.777+0000 7f0535652640 1 --2- 172.21.15.33:0/3431543371 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dfbedc7f30 0x55dfbedd3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.777+0000 7f052ffff640 1 --2- 172.21.15.33:0/3431543371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfbedbf350 0x55dfbedc3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.777+0000 7f0534e51640 1 -- 172.21.15.33:0/3431543371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfbedbf350 msgr2=0x55dfbedc3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:38.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.777+0000 7f0534e51640 1 --2- 172.21.15.33:0/3431543371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfbedbf350 0x55dfbedc3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.777+0000 7f0534e51640 1 -- 172.21.15.33:0/3431543371 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dfbedc7f30 msgr2=0x55dfbedd3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:38.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.777+0000 7f0534e51640 1 --2- 172.21.15.33:0/3431543371 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dfbedc7f30 0x55dfbedd3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.777+0000 7f0534e51640 1 -- 172.21.15.33:0/3431543371 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfbe9f60f0 con 0x55dfbedbea30 2024-03-19T02:57:38.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.777+0000 7f052ffff640 1 --2- 172.21.15.33:0/3431543371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfbedbf350 0x55dfbedc3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:57:38.780 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:38 smithi033 ceph-mon[31973]: pgmap v128: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:57:38.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.777+0000 7f0534e51640 1 --2- 172.21.15.33:0/3431543371 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfbedbea30 0x55dfbedc7620 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x55dfbeddafb0 tx=0x7f0528014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84086e687af1b6af server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:38.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.777+0000 7f05167fc640 1 -- 172.21.15.33:0/3431543371 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0528004840 con 0x55dfbedbea30 2024-03-19T02:57:38.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.778+0000 7f05167fc640 1 -- 172.21.15.33:0/3431543371 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0528029500 con 0x55dfbedbea30 2024-03-19T02:57:38.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.778+0000 7f05167fc640 1 -- 172.21.15.33:0/3431543371 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f052801dbe0 con 0x55dfbedbea30 2024-03-19T02:57:38.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.778+0000 7f0539d76740 1 -- 172.21.15.33:0/3431543371 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfbedbea30 msgr2=0x55dfbedc7620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:38.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.778+0000 7f0539d76740 1 --2- 172.21.15.33:0/3431543371 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfbedbea30 0x55dfbedc7620 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x55dfbeddafb0 tx=0x7f0528014000 comp rx=0 tx=0).stop 2024-03-19T02:57:38.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.778+0000 7f0539d76740 1 -- 172.21.15.33:0/3431543371 shutdown_connections 2024-03-19T02:57:38.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.778+0000 7f0539d76740 1 --2- 172.21.15.33:0/3431543371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfbedbf350 0x55dfbedc3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.778+0000 7f0539d76740 1 --2- 172.21.15.33:0/3431543371 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dfbedc7f30 0x55dfbedd3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.778+0000 7f0539d76740 1 --2- 172.21.15.33:0/3431543371 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfbedbea30 0x55dfbedc7620 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.778+0000 7f0539d76740 1 -- 172.21.15.33:0/3431543371 >> 172.21.15.33:0/3431543371 conn(0x55dfbedb9160 msgr2=0x55dfbedb81e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:38.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.778+0000 7f0539d76740 1 -- 172.21.15.33:0/3431543371 shutdown_connections 2024-03-19T02:57:38.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.778+0000 7f0539d76740 1 -- 172.21.15.33:0/3431543371 wait complete. 2024-03-19T02:57:38.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.778+0000 7f0539d76740 1 -- start start 2024-03-19T02:57:38.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.779+0000 7f0539d76740 1 Processor -- start 2024-03-19T02:57:38.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.779+0000 7f0539d76740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfbedbea30 0x55dfbedd03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.779+0000 7f0539d76740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dfbedbf350 0x55dfbedc17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.779+0000 7f0539d76740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfbedc7f30 0x55dfbedc1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.779+0000 7f0539d76740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f05301af360 con 0x55dfbedbea30 2024-03-19T02:57:38.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.779+0000 7f0539d76740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f05301b07d0 con 0x55dfbedbf350 2024-03-19T02:57:38.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.779+0000 7f0539d76740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f05301afa30 con 0x55dfbedc7f30 2024-03-19T02:57:38.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.779+0000 7f0539d76740 10 client.0 _ll_register_callbacks cb 0x7f05240bcab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:57:38.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.779+0000 7f0534e51640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfbedbea30 0x55dfbedd03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.779+0000 7f0534e51640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfbedbea30 0x55dfbedd03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56268/0 (socket says 172.21.15.33:56268) 2024-03-19T02:57:38.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.779+0000 7f0534e51640 1 -- 172.21.15.33:0/866341247 learned_addr learned my addr 172.21.15.33:0/866341247 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:57:38.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.779+0000 7f052ffff640 1 --2- 172.21.15.33:0/866341247 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dfbedbf350 0x55dfbedc17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.779+0000 7f0535652640 1 --2- 172.21.15.33:0/866341247 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfbedc7f30 0x55dfbedc1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.779+0000 7f0534e51640 1 -- 172.21.15.33:0/866341247 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfbedc7f30 msgr2=0x55dfbedc1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:38.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.779+0000 7f0534e51640 1 --2- 172.21.15.33:0/866341247 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfbedc7f30 0x55dfbedc1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.779+0000 7f0534e51640 1 -- 172.21.15.33:0/866341247 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dfbedbf350 msgr2=0x55dfbedc17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:57:38.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.779+0000 7f0534e51640 1 --2- 172.21.15.33:0/866341247 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dfbedbf350 0x55dfbedc17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.779+0000 7f0534e51640 1 -- 172.21.15.33:0/866341247 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfbedc2510 con 0x55dfbedbea30 2024-03-19T02:57:38.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.780+0000 7f0534e51640 1 --2- 172.21.15.33:0/866341247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfbedbea30 0x55dfbedd03c0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f0528005f40 tx=0x7f05280296b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d80f2692b521cd9b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:57:38.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.780+0000 7f050ffff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:57:38.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.780+0000 7f050ffff640 1 -- 172.21.15.33:0/866341247 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0528004840 con 0x55dfbedbea30 2024-03-19T02:57:38.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.780+0000 7f050ffff640 1 -- 172.21.15.33:0/866341247 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f0528029050 con 0x55dfbedbea30 2024-03-19T02:57:38.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.780+0000 7f0539d76740 1 -- 172.21.15.33:0/866341247 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0528029500 con 0x55dfbedbea30 2024-03-19T02:57:38.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.780+0000 7f050ffff640 1 -- 172.21.15.33:0/866341247 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0528044050 con 0x55dfbedbea30 2024-03-19T02:57:38.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.780+0000 7f0539d76740 10 client.16608 fetch_fsmap learned FSMap version 25 2024-03-19T02:57:38.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.780+0000 7f0539d76740 1 -- 172.21.15.33:0/866341247 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55dfbee3e660 con 0x55dfbedbea30 2024-03-19T02:57:38.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.780+0000 7f050ffff640 1 -- 172.21.15.33:0/866341247 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0528029500 con 0x55dfbedbea30 2024-03-19T02:57:38.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.780+0000 7f050ffff640 1 -- 172.21.15.33:0/866341247 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f0528005040 con 0x55dfbedbea30 2024-03-19T02:57:38.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.780+0000 7f0539d76740 10 client.16608 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:57:38.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.780+0000 7f0539d76740 10 client.16608 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:57:38.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.780+0000 7f0539d76740 20 client.16608 populate_metadata read hostname 'smithi033' 2024-03-19T02:57:38.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.780+0000 7f0539d76740 1 --2- 172.21.15.33:0/866341247 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55dfbee3ed20 0x55dfbee41110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:57:38.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.780+0000 7f0539d76740 4 client.16608 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:57:38.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.780+0000 7f0539d76740 1 -- 172.21.15.33:0/866341247 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55dfbed29d90 con 0x55dfbee3ed20 2024-03-19T02:57:38.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.780+0000 7f052ffff640 1 --2- 172.21.15.33:0/866341247 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55dfbee3ed20 0x55dfbee41110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:57:38.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.781+0000 7f050ffff640 10 client.16608 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:57:38.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.781+0000 7f052ffff640 1 --2- 172.21.15.33:0/866341247 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55dfbee3ed20 0x55dfbee41110 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=917178c79b9d60e6 server_cookie=17e2fb458e318cbe in_seq=0 out_seq=0 2024-03-19T02:57:38.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.781+0000 7f050ffff640 1 -- 172.21.15.33:0/866341247 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55dfbed29d90 con 0x55dfbee3ed20 2024-03-19T02:57:38.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.781+0000 7f050ffff640 10 client.16608 handle_command_reply: tid=0 2024-03-19T02:57:38.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.781+0000 7f0539d76740 1 client.16608 shutdown 2024-03-19T02:57:38.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.781+0000 7f0539d76740 1 -- 172.21.15.33:0/866341247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfbedbea30 msgr2=0x55dfbedd03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:38.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.781+0000 7f0539d76740 1 --2- 172.21.15.33:0/866341247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfbedbea30 0x55dfbedd03c0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f0528005f40 tx=0x7f05280296b0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.781+0000 7f0539d76740 1 -- 172.21.15.33:0/866341247 shutdown_connections 2024-03-19T02:57:38.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.781+0000 7f0539d76740 1 --2- 172.21.15.33:0/866341247 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55dfbee3ed20 0x55dfbee41110 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.781+0000 7f0539d76740 1 --2- 172.21.15.33:0/866341247 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55dfbedc7f30 0x55dfbedc1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.781+0000 7f0539d76740 1 --2- 172.21.15.33:0/866341247 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55dfbedbf350 0x55dfbedc17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.781+0000 7f0539d76740 1 --2- 172.21.15.33:0/866341247 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55dfbedbea30 0x55dfbedd03c0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.781+0000 7f0539d76740 1 -- 172.21.15.33:0/866341247 >> 172.21.15.33:0/866341247 conn(0x55dfbedb9160 msgr2=0x55dfbedba460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:38.789 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.781+0000 7f050ffff640 0 client.16608 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:57:38.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.781+0000 7f0539d76740 1 -- 172.21.15.33:0/866341247 shutdown_connections 2024-03-19T02:57:38.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.781+0000 7f0539d76740 1 -- 172.21.15.33:0/866341247 wait complete. 2024-03-19T02:57:38.790 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.781+0000 7f0539d76740 20 client.16608 trim_cache size 0 max 16384 2024-03-19T02:57:38.790 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:57:38.790 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:57:38.790 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-03-19T02:57:38.790 INFO:teuthology.orchestra.run.smithi033.stdout: "e5fb108a-4d36-4b08-af7c-bdce0977670a": { 2024-03-19T02:57:38.790 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-03-19T02:57:38.790 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "e5fb108a-4d36-4b08-af7c-bdce0977670a", 2024-03-19T02:57:38.790 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-03-19T02:57:38.791 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T02:57:38.791 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T02:57:38.791 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:57:38.791 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.782+0000 7f05370b5640 1 -- 172.21.15.33:0/3208853440 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f05180a2970 msgr2=0x7f05180a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:38.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.782+0000 7f05370b5640 1 --2- 172.21.15.33:0/3208853440 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f05180a2970 0x7f05180a4e30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0528005fd0 tx=0x7f0528009500 comp rx=0 tx=0).stop 2024-03-19T02:57:38.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.782+0000 7f05370b5640 1 -- 172.21.15.33:0/3208853440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0530102180 msgr2=0x7f0530074060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:57:38.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.782+0000 7f05370b5640 1 --2- 172.21.15.33:0/3208853440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0530102180 0x7f0530074060 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f0524006010 tx=0x7f0524005d90 comp rx=0 tx=0).stop 2024-03-19T02:57:38.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.782+0000 7f05370b5640 1 -- 172.21.15.33:0/3208853440 shutdown_connections 2024-03-19T02:57:38.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.782+0000 7f05370b5640 1 --2- 172.21.15.33:0/3208853440 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f05180a2970 0x7f05180a4e30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.782+0000 7f05370b5640 1 --2- 172.21.15.33:0/3208853440 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05301077d0 0x7f05300745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.782+0000 7f05370b5640 1 --2- 172.21.15.33:0/3208853440 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0530102180 0x7f0530074060 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.782+0000 7f05370b5640 1 --2- 172.21.15.33:0/3208853440 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05301017c0 0x7f0530073b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:57:38.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.782+0000 7f05370b5640 1 -- 172.21.15.33:0/3208853440 >> 172.21.15.33:0/3208853440 conn(0x7f05300fd2b0 msgr2=0x7f05300fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:57:38.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.782+0000 7f05370b5640 1 -- 172.21.15.33:0/3208853440 shutdown_connections 2024-03-19T02:57:38.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:57:38.782+0000 7f05370b5640 1 -- 172.21.15.33:0/3208853440 wait complete. 2024-03-19T02:57:38.794 INFO:tasks.cephfs.filesystem:scrub status for tag:e5fb108a-4d36-4b08-af7c-bdce0977670a - {'path': '/', 'tag': 'e5fb108a-4d36-4b08-af7c-bdce0977670a', 'options': 'recursive,force'} 2024-03-19T02:57:39.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:39 smithi096 ceph-mon[35946]: scrub queued for path: / 2024-03-19T02:57:39.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:39 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T02:57:39.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:39 smithi096 ceph-mon[35946]: scrub summary: active paths [/] 2024-03-19T02:57:39.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:39 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T02:57:39.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:39 smithi155 ceph-mon[35804]: scrub queued for path: / 2024-03-19T02:57:39.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:39 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T02:57:39.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:39 smithi155 ceph-mon[35804]: scrub summary: active paths [/] 2024-03-19T02:57:39.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:39 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T02:57:39.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:39 smithi033 ceph-mon[31973]: scrub queued for path: / 2024-03-19T02:57:39.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:39 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T02:57:39.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:39 smithi033 ceph-mon[31973]: scrub summary: active paths [/] 2024-03-19T02:57:39.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:39 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T02:57:40.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:40 smithi096 ceph-mon[35946]: pgmap v129: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:57:40.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:40 smithi155 ceph-mon[35804]: pgmap v129: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:57:40.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:40 smithi033 ceph-mon[31973]: pgmap v129: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:57:42.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:42 smithi096 ceph-mon[35946]: pgmap v130: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:57:42.769 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:42 smithi155 ceph-mon[35804]: pgmap v130: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:57:42.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:42 smithi033 ceph-mon[31973]: pgmap v130: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:57:43.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:43 smithi096 ceph-mon[35946]: scrub summary: idle 2024-03-19T02:57:43.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:43 smithi155 ceph-mon[35804]: scrub summary: idle 2024-03-19T02:57:43.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:43 smithi033 ceph-mon[31973]: scrub summary: idle 2024-03-19T02:57:44.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:44 smithi096 ceph-mon[35946]: pgmap v131: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:57:44.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:44 smithi155 ceph-mon[35804]: pgmap v131: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:57:44.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:44 smithi033 ceph-mon[31973]: pgmap v131: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:57:46.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:46 smithi096 ceph-mon[35946]: pgmap v132: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:57:46.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:46 smithi155 ceph-mon[35804]: pgmap v132: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:57:46.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:46 smithi033 ceph-mon[31973]: pgmap v132: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:57:48.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:48 smithi096 ceph-mon[35946]: pgmap v133: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:57:48.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:48 smithi155 ceph-mon[35804]: pgmap v133: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:57:48.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:48 smithi033 ceph-mon[31973]: pgmap v133: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:57:50.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:50 smithi096 ceph-mon[35946]: pgmap v134: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:57:50.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:50 smithi155 ceph-mon[35804]: pgmap v134: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:57:50.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:50 smithi033 ceph-mon[31973]: pgmap v134: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:57:52.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:52 smithi096 ceph-mon[35946]: pgmap v135: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:57:52.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:52 smithi155 ceph-mon[35804]: pgmap v135: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:57:52.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:52 smithi033 ceph-mon[31973]: pgmap v135: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:57:54.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:54 smithi096 ceph-mon[35946]: pgmap v136: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:57:54.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:54 smithi155 ceph-mon[35804]: pgmap v136: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:57:54.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:54 smithi033 ceph-mon[31973]: pgmap v136: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:57:56.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:56 smithi096 ceph-mon[35946]: pgmap v137: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:57:56.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:56 smithi155 ceph-mon[35804]: pgmap v137: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:57:56.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:56 smithi033 ceph-mon[31973]: pgmap v137: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:57:58.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:57:58 smithi096 ceph-mon[35946]: pgmap v138: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:57:58.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:57:58 smithi155 ceph-mon[35804]: pgmap v138: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:57:58.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:57:58 smithi033 ceph-mon[31973]: pgmap v138: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:58:00.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:00 smithi155 ceph-mon[35804]: pgmap v139: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:58:00.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:00 smithi033 ceph-mon[31973]: pgmap v139: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:58:00.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:00 smithi096 ceph-mon[35946]: pgmap v139: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:58:02.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:02 smithi155 ceph-mon[35804]: pgmap v140: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:58:02.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:02 smithi033 ceph-mon[31973]: pgmap v140: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:58:02.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:02 smithi096 ceph-mon[35946]: pgmap v140: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:58:04.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:04 smithi155 ceph-mon[35804]: pgmap v141: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-19T02:58:04.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:04 smithi033 ceph-mon[31973]: pgmap v141: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-19T02:58:04.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:04 smithi096 ceph-mon[35946]: pgmap v141: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-19T02:58:06.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:06 smithi155 ceph-mon[35804]: pgmap v142: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-19T02:58:06.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:06 smithi033 ceph-mon[31973]: pgmap v142: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-19T02:58:06.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:06 smithi096 ceph-mon[35946]: pgmap v142: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-19T02:58:08.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:08 smithi155 ceph-mon[35804]: pgmap v143: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-19T02:58:08.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:08 smithi033 ceph-mon[31973]: pgmap v143: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-19T02:58:08.795 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:58:08.796 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:58:08.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.912+0000 7f66ad043640 1 -- 172.21.15.33:0/317595354 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66a81077c0 msgr2=0x7f66a8107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:08.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.912+0000 7f66ad043640 1 --2- 172.21.15.33:0/317595354 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66a81077c0 0x7f66a8107ba0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f66a0009930 tx=0x7f66a002bc30 comp rx=0 tx=0).stop 2024-03-19T02:58:08.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.912+0000 7f66ad043640 1 -- 172.21.15.33:0/317595354 shutdown_connections 2024-03-19T02:58:08.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.912+0000 7f66ad043640 1 --2- 172.21.15.33:0/317595354 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66a8102170 0x7f66a810e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:08.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.912+0000 7f66ad043640 1 --2- 172.21.15.33:0/317595354 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66a81017b0 0x7f66a8101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:08.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.912+0000 7f66ad043640 1 --2- 172.21.15.33:0/317595354 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66a81077c0 0x7f66a8107ba0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:08.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.912+0000 7f66ad043640 1 -- 172.21.15.33:0/317595354 >> 172.21.15.33:0/317595354 conn(0x7f66a8078c20 msgr2=0x7f66a8079030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:08.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.912+0000 7f66ad043640 1 -- 172.21.15.33:0/317595354 shutdown_connections 2024-03-19T02:58:08.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.912+0000 7f66ad043640 1 -- 172.21.15.33:0/317595354 wait complete. 2024-03-19T02:58:08.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.913+0000 7f66ad043640 1 Processor -- start 2024-03-19T02:58:08.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.913+0000 7f66ad043640 1 -- start start 2024-03-19T02:58:08.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.913+0000 7f66ad043640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66a81017b0 0x7f66a8197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:08.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.913+0000 7f66ad043640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66a8102170 0x7f66a8198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:08.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.913+0000 7f66ad043640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66a81077c0 0x7f66a819c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:08.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.913+0000 7f66ad043640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f66a8111a40 con 0x7f66a8102170 2024-03-19T02:58:08.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.913+0000 7f66ad043640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f66a81118c0 con 0x7f66a81017b0 2024-03-19T02:58:08.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.913+0000 7f66ad043640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f66a8111bc0 con 0x7f66a81077c0 2024-03-19T02:58:08.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.913+0000 7f66a659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66a8102170 0x7f66a8198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:08.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.913+0000 7f66a6d9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66a81017b0 0x7f66a8197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:08.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.913+0000 7f66a6d9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66a81017b0 0x7f66a8197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:51224/0 (socket says 172.21.15.33:51224) 2024-03-19T02:58:08.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.913+0000 7f66a659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66a8102170 0x7f66a8198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49662/0 (socket says 172.21.15.33:49662) 2024-03-19T02:58:08.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.913+0000 7f66a6d9d640 1 -- 172.21.15.33:0/591857219 learned_addr learned my addr 172.21.15.33:0/591857219 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:08.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.913+0000 7f66a759e640 1 --2- 172.21.15.33:0/591857219 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66a81077c0 0x7f66a819c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:08.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.914+0000 7f66a6d9d640 1 -- 172.21.15.33:0/591857219 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66a81077c0 msgr2=0x7f66a819c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:08.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.914+0000 7f66a6d9d640 1 --2- 172.21.15.33:0/591857219 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66a81077c0 0x7f66a819c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:08.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.914+0000 7f66a6d9d640 1 -- 172.21.15.33:0/591857219 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66a8102170 msgr2=0x7f66a8198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:08.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.914+0000 7f66a6d9d640 1 --2- 172.21.15.33:0/591857219 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66a8102170 0x7f66a8198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:08.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.914+0000 7f66a6d9d640 1 -- 172.21.15.33:0/591857219 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66a819cfb0 con 0x7f66a81017b0 2024-03-19T02:58:08.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.914+0000 7f66a6d9d640 1 --2- 172.21.15.33:0/591857219 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66a81017b0 0x7f66a8197fc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f66a0009230 tx=0x7f66a002c5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:08.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.914+0000 7f669ffff640 1 -- 172.21.15.33:0/591857219 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a0043020 con 0x7f66a81017b0 2024-03-19T02:58:08.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.914+0000 7f66ad043640 1 -- 172.21.15.33:0/591857219 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66a819d240 con 0x7f66a81017b0 2024-03-19T02:58:08.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.914+0000 7f669ffff640 1 -- 172.21.15.33:0/591857219 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f66a002ca70 con 0x7f66a81017b0 2024-03-19T02:58:08.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.914+0000 7f669ffff640 1 -- 172.21.15.33:0/591857219 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a0003a50 con 0x7f66a81017b0 2024-03-19T02:58:08.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.914+0000 7f66ad043640 1 -- 172.21.15.33:0/591857219 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66a8073a50 con 0x7f66a81017b0 2024-03-19T02:58:08.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.916+0000 7f669ffff640 1 -- 172.21.15.33:0/591857219 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f66a0046020 con 0x7f66a81017b0 2024-03-19T02:58:08.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.916+0000 7f66afc2a740 1 Processor -- start 2024-03-19T02:58:08.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.917+0000 7f66afc2a740 1 -- start start 2024-03-19T02:58:08.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.917+0000 7f669ffff640 1 --2- 172.21.15.33:0/591857219 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f66940a2970 0x7f66940a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:08.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.917+0000 7f66a759e640 1 --2- 172.21.15.33:0/591857219 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f66940a2970 0x7f66940a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:08.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.917+0000 7f669ffff640 1 -- 172.21.15.33:0/591857219 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f66a00f10e0 con 0x7f66a81017b0 2024-03-19T02:58:08.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.917+0000 7f66afc2a740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579592ffa30 0x5579592ffe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:08.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.917+0000 7f66afc2a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557959300350 0x5579593089f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:08.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.917+0000 7f66afc2a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557959308f30 0x55795930b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:08.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.917+0000 7f66afc2a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557959263500 con 0x557959308f30 2024-03-19T02:58:08.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.917+0000 7f66afc2a740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55795920bbe0 con 0x5579592ffa30 2024-03-19T02:58:08.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.917+0000 7f66afc2a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55795925f5a0 con 0x557959300350 2024-03-19T02:58:08.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.917+0000 7f66a6d9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579592ffa30 0x5579592ffe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:08.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.917+0000 7f66a6d9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579592ffa30 0x5579592ffe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:51232/0 (socket says 172.21.15.33:51232) 2024-03-19T02:58:08.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.917+0000 7f66a6d9d640 1 -- 172.21.15.33:0/349848675 learned_addr learned my addr 172.21.15.33:0/349848675 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:08.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.917+0000 7f66a659c640 1 --2- 172.21.15.33:0/349848675 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557959300350 0x5579593089f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:08.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.917+0000 7f66a759e640 1 --2- 172.21.15.33:0/349848675 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557959308f30 0x55795930b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:08.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.917+0000 7f66a6d9d640 1 -- 172.21.15.33:0/349848675 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557959300350 msgr2=0x5579593089f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:08.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.917+0000 7f66a6d9d640 1 --2- 172.21.15.33:0/349848675 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557959300350 0x5579593089f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:08.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.917+0000 7f66a6d9d640 1 -- 172.21.15.33:0/349848675 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557959308f30 msgr2=0x55795930b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:08.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.917+0000 7f66a6d9d640 1 --2- 172.21.15.33:0/349848675 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557959308f30 0x55795930b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:08.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.917+0000 7f66a6d9d640 1 -- 172.21.15.33:0/349848675 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55795922d000 con 0x5579592ffa30 2024-03-19T02:58:08.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.917+0000 7f66a759e640 1 --2- 172.21.15.33:0/349848675 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557959308f30 0x55795930b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:58:08.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.917+0000 7f66a759e640 1 --2- 172.21.15.33:0/591857219 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f66940a2970 0x7f66940a4e30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6698005fd0 tx=0x7f6698011810 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:08.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.918+0000 7f66a6d9d640 1 --2- 172.21.15.33:0/349848675 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579592ffa30 0x5579592ffe10 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f66a00b9900 tx=0x7f66a002c740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1046b9aae5747ec7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:08.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.918+0000 7f669d7fa640 1 -- 172.21.15.33:0/349848675 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a00bd4a0 con 0x5579592ffa30 2024-03-19T02:58:08.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.918+0000 7f669d7fa640 1 -- 172.21.15.33:0/349848675 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f66a00c15f0 con 0x5579592ffa30 2024-03-19T02:58:08.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.918+0000 7f669d7fa640 1 -- 172.21.15.33:0/349848675 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a00053d0 con 0x5579592ffa30 2024-03-19T02:58:08.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.918+0000 7f66afc2a740 1 -- 172.21.15.33:0/349848675 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579592ffa30 msgr2=0x5579592ffe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:08.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.918+0000 7f66afc2a740 1 --2- 172.21.15.33:0/349848675 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579592ffa30 0x5579592ffe10 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f66a00b9900 tx=0x7f66a002c740 comp rx=0 tx=0).stop 2024-03-19T02:58:08.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.919+0000 7f66afc2a740 1 -- 172.21.15.33:0/349848675 shutdown_connections 2024-03-19T02:58:08.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.919+0000 7f66afc2a740 1 --2- 172.21.15.33:0/349848675 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557959300350 0x5579593089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:08.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.919+0000 7f66afc2a740 1 --2- 172.21.15.33:0/349848675 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557959308f30 0x55795930b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:08.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.919+0000 7f66afc2a740 1 --2- 172.21.15.33:0/349848675 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579592ffa30 0x5579592ffe10 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:08.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.919+0000 7f66afc2a740 1 -- 172.21.15.33:0/349848675 >> 172.21.15.33:0/349848675 conn(0x5579592fa160 msgr2=0x5579592fa540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:08.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.919+0000 7f66afc2a740 1 -- 172.21.15.33:0/349848675 shutdown_connections 2024-03-19T02:58:08.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.919+0000 7f66afc2a740 1 -- 172.21.15.33:0/349848675 wait complete. 2024-03-19T02:58:08.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.919+0000 7f66afc2a740 1 -- start start 2024-03-19T02:58:08.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.919+0000 7f66afc2a740 1 Processor -- start 2024-03-19T02:58:08.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.919+0000 7f66afc2a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579592ffa30 0x557959305e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:08.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.919+0000 7f66afc2a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557959300350 0x5579593063a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:08.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.919+0000 7f66afc2a740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557959308f30 0x5579593068e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:08.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.919+0000 7f66afc2a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55795930bae0 con 0x5579592ffa30 2024-03-19T02:58:08.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.919+0000 7f66afc2a740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55795921f160 con 0x557959308f30 2024-03-19T02:58:08.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.919+0000 7f66afc2a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55795930bc60 con 0x557959300350 2024-03-19T02:58:08.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.919+0000 7f66afc2a740 10 client.0 _ll_register_callbacks cb 0x55795925ba40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:58:08.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.919+0000 7f66a6d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579592ffa30 0x557959305e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:08.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.919+0000 7f66a759e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557959308f30 0x5579593068e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:08.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.919+0000 7f66a6d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579592ffa30 0x557959305e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49686/0 (socket says 172.21.15.33:49686) 2024-03-19T02:58:08.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.919+0000 7f66a6d9d640 1 -- 172.21.15.33:0/604881349 learned_addr learned my addr 172.21.15.33:0/604881349 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:08.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.919+0000 7f66a659c640 1 --2- 172.21.15.33:0/604881349 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557959300350 0x5579593063a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:08.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.920+0000 7f66a6d9d640 1 -- 172.21.15.33:0/604881349 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557959300350 msgr2=0x5579593063a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:08.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.920+0000 7f66a6d9d640 1 --2- 172.21.15.33:0/604881349 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557959300350 0x5579593063a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:08.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.920+0000 7f66a6d9d640 1 -- 172.21.15.33:0/604881349 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557959308f30 msgr2=0x5579593068e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:08.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.920+0000 7f66a6d9d640 1 --2- 172.21.15.33:0/604881349 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557959308f30 0x5579593068e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:08.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.920+0000 7f66a6d9d640 1 -- 172.21.15.33:0/604881349 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579590417d0 con 0x5579592ffa30 2024-03-19T02:58:08.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.920+0000 7f66a759e640 1 --2- 172.21.15.33:0/604881349 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557959308f30 0x5579593068e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:58:08.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.920+0000 7f66a6d9d640 1 --2- 172.21.15.33:0/604881349 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579592ffa30 0x557959305e60 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f66a0006010 tx=0x7f66a00c5c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0f270b575c4dbc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:08.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.920+0000 7f668e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:58:08.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.920+0000 7f668e7fc640 1 -- 172.21.15.33:0/604881349 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a00053d0 con 0x5579592ffa30 2024-03-19T02:58:08.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.920+0000 7f668e7fc640 1 -- 172.21.15.33:0/604881349 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f66a00c1170 con 0x5579592ffa30 2024-03-19T02:58:08.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.920+0000 7f66afc2a740 1 -- 172.21.15.33:0/604881349 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66a00c15f0 con 0x5579592ffa30 2024-03-19T02:58:08.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.920+0000 7f668e7fc640 1 -- 172.21.15.33:0/604881349 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f66a0005570 con 0x5579592ffa30 2024-03-19T02:58:08.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.920+0000 7f66afc2a740 10 client.16620 fetch_fsmap learned FSMap version 25 2024-03-19T02:58:08.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.920+0000 7f66afc2a740 1 -- 172.21.15.33:0/604881349 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x557958f773f0 con 0x5579592ffa30 2024-03-19T02:58:08.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.920+0000 7f668e7fc640 1 -- 172.21.15.33:0/604881349 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f66a00c15f0 con 0x5579592ffa30 2024-03-19T02:58:08.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.921+0000 7f668e7fc640 1 -- 172.21.15.33:0/604881349 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f66a00eec60 con 0x5579592ffa30 2024-03-19T02:58:08.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.921+0000 7f66afc2a740 10 client.16620 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:58:08.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.921+0000 7f66afc2a740 10 client.16620 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:58:08.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.921+0000 7f66afc2a740 20 client.16620 populate_metadata read hostname 'smithi033' 2024-03-19T02:58:08.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.921+0000 7f66afc2a740 1 --2- 172.21.15.33:0/604881349 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557959318e10 0x55795931b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:08.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.921+0000 7f66afc2a740 4 client.16620 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:58:08.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.921+0000 7f66afc2a740 1 -- 172.21.15.33:0/604881349 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557959270a40 con 0x557959318e10 2024-03-19T02:58:08.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.921+0000 7f66a659c640 1 --2- 172.21.15.33:0/604881349 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557959318e10 0x55795931b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:08.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.921+0000 7f66a659c640 1 --2- 172.21.15.33:0/604881349 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557959318e10 0x55795931b200 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=476d4b22930baf56 server_cookie=7150898bf7ac22f2 in_seq=0 out_seq=0 2024-03-19T02:58:08.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.921+0000 7f668e7fc640 10 client.16620 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:58:08.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.922+0000 7f668e7fc640 1 -- 172.21.15.33:0/604881349 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557959270a40 con 0x557959318e10 2024-03-19T02:58:08.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.922+0000 7f668e7fc640 10 client.16620 handle_command_reply: tid=0 2024-03-19T02:58:08.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.922+0000 7f66afc2a740 1 client.16620 shutdown 2024-03-19T02:58:08.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.922+0000 7f66afc2a740 1 -- 172.21.15.33:0/604881349 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579592ffa30 msgr2=0x557959305e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:08.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.922+0000 7f66afc2a740 1 --2- 172.21.15.33:0/604881349 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579592ffa30 0x557959305e60 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f66a0006010 tx=0x7f66a00c5c40 comp rx=0 tx=0).stop 2024-03-19T02:58:08.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.922+0000 7f66afc2a740 1 -- 172.21.15.33:0/604881349 shutdown_connections 2024-03-19T02:58:08.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.922+0000 7f66afc2a740 1 --2- 172.21.15.33:0/604881349 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557959318e10 0x55795931b200 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:08.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.922+0000 7f66afc2a740 1 --2- 172.21.15.33:0/604881349 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557959300350 0x5579593063a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:08.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.922+0000 7f668e7fc640 0 client.16620 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:58:08.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.922+0000 7f66afc2a740 1 --2- 172.21.15.33:0/604881349 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557959308f30 0x5579593068e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:08.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.922+0000 7f66afc2a740 1 --2- 172.21.15.33:0/604881349 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579592ffa30 0x557959305e60 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:08.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.922+0000 7f66afc2a740 1 -- 172.21.15.33:0/604881349 >> 172.21.15.33:0/604881349 conn(0x5579592fa160 msgr2=0x5579592fbeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:08.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.923+0000 7f66afc2a740 1 -- 172.21.15.33:0/604881349 shutdown_connections 2024-03-19T02:58:08.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.923+0000 7f66afc2a740 1 -- 172.21.15.33:0/604881349 wait complete. 2024-03-19T02:58:08.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.923+0000 7f66afc2a740 20 client.16620 trim_cache size 0 max 16384 2024-03-19T02:58:08.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.940+0000 7f66afc2a740 1 Processor -- start 2024-03-19T02:58:08.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.940+0000 7f66afc2a740 1 -- start start 2024-03-19T02:58:08.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.940+0000 7f66afc2a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579592ffa30 0x557959304fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:08.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.940+0000 7f66afc2a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557959300350 0x5579593147b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:08.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.940+0000 7f66afc2a740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557959308f30 0x5579592fbc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:08.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.940+0000 7f66afc2a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f66a81af290 con 0x5579592ffa30 2024-03-19T02:58:08.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.940+0000 7f66afc2a740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f66a81acb60 con 0x557959308f30 2024-03-19T02:58:08.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.940+0000 7f66afc2a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f66a81aa990 con 0x557959300350 2024-03-19T02:58:08.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.940+0000 7f66a6d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579592ffa30 0x557959304fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:08.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.941+0000 7f66a759e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557959308f30 0x5579592fbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:08.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.941+0000 7f66a659c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557959300350 0x5579593147b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:08.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.941+0000 7f66a6d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579592ffa30 0x557959304fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49702/0 (socket says 172.21.15.33:49702) 2024-03-19T02:58:08.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.941+0000 7f66a659c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557959300350 0x5579593147b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:37692/0 (socket says 172.21.15.33:37692) 2024-03-19T02:58:08.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.941+0000 7f66a6d9d640 1 -- 172.21.15.33:0/1670832810 learned_addr learned my addr 172.21.15.33:0/1670832810 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:08.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.941+0000 7f66a759e640 1 --2- 172.21.15.33:0/1670832810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557959308f30 0x5579592fbc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:51246/0 (socket says 172.21.15.33:51246) 2024-03-19T02:58:08.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.941+0000 7f66a659c640 1 -- 172.21.15.33:0/1670832810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557959308f30 msgr2=0x5579592fbc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:08.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.941+0000 7f66a659c640 1 --2- 172.21.15.33:0/1670832810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557959308f30 0x5579592fbc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:08.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.941+0000 7f66a659c640 1 -- 172.21.15.33:0/1670832810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579592ffa30 msgr2=0x557959304fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:08.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.941+0000 7f66a659c640 1 --2- 172.21.15.33:0/1670832810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579592ffa30 0x557959304fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:08.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.941+0000 7f66a659c640 1 -- 172.21.15.33:0/1670832810 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557958f370f0 con 0x557959300350 2024-03-19T02:58:08.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.941+0000 7f66a6d9d640 1 --2- 172.21.15.33:0/1670832810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579592ffa30 0x557959304fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:58:08.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.941+0000 7f66a659c640 1 --2- 172.21.15.33:0/1670832810 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557959300350 0x5579593147b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x55795931b8e0 tx=0x7f6690012040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afde85243476a9fe server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:08.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.941+0000 7f669cff9640 1 -- 172.21.15.33:0/1670832810 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6690005220 con 0x557959300350 2024-03-19T02:58:08.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.941+0000 7f669cff9640 1 -- 172.21.15.33:0/1670832810 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f669001ccd0 con 0x557959300350 2024-03-19T02:58:08.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.941+0000 7f669cff9640 1 -- 172.21.15.33:0/1670832810 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6690004c90 con 0x557959300350 2024-03-19T02:58:08.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.942+0000 7f66afc2a740 1 -- 172.21.15.33:0/1670832810 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557959300350 msgr2=0x5579593147b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:08.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.942+0000 7f66afc2a740 1 --2- 172.21.15.33:0/1670832810 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557959300350 0x5579593147b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x55795931b8e0 tx=0x7f6690012040 comp rx=0 tx=0).stop 2024-03-19T02:58:08.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.942+0000 7f66afc2a740 1 -- 172.21.15.33:0/1670832810 shutdown_connections 2024-03-19T02:58:08.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.942+0000 7f66afc2a740 1 --2- 172.21.15.33:0/1670832810 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557959300350 0x5579593147b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:08.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.942+0000 7f66afc2a740 1 --2- 172.21.15.33:0/1670832810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557959308f30 0x5579592fbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:08.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.942+0000 7f66afc2a740 1 --2- 172.21.15.33:0/1670832810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579592ffa30 0x557959304fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:08.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.942+0000 7f66afc2a740 1 -- 172.21.15.33:0/1670832810 >> 172.21.15.33:0/1670832810 conn(0x5579592fa160 msgr2=0x5579592fb570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:08.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.942+0000 7f66afc2a740 1 -- 172.21.15.33:0/1670832810 shutdown_connections 2024-03-19T02:58:08.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.942+0000 7f66afc2a740 1 -- 172.21.15.33:0/1670832810 wait complete. 2024-03-19T02:58:08.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.942+0000 7f66afc2a740 1 -- start start 2024-03-19T02:58:08.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.942+0000 7f66afc2a740 1 Processor -- start 2024-03-19T02:58:08.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.942+0000 7f66afc2a740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579592ffa30 0x55795930c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:08.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.943+0000 7f66afc2a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557959300350 0x55795930cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:08.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.943+0000 7f66afc2a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557959308f30 0x55795930d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:08.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.943+0000 7f66afc2a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f66a81ab020 con 0x557959300350 2024-03-19T02:58:08.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.943+0000 7f66afc2a740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f66a81ac490 con 0x5579592ffa30 2024-03-19T02:58:08.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.943+0000 7f66afc2a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f66a81ab6f0 con 0x557959308f30 2024-03-19T02:58:08.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.943+0000 7f66afc2a740 10 client.0 _ll_register_callbacks cb 0x7f6690009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:58:08.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.943+0000 7f66a659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557959300350 0x55795930cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:08.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.943+0000 7f66a659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557959300350 0x55795930cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49718/0 (socket says 172.21.15.33:49718) 2024-03-19T02:58:08.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.943+0000 7f66a6d9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579592ffa30 0x55795930c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:08.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.943+0000 7f66a659c640 1 -- 172.21.15.33:0/334166126 learned_addr learned my addr 172.21.15.33:0/334166126 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:08.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.943+0000 7f66a759e640 1 --2- 172.21.15.33:0/334166126 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557959308f30 0x55795930d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:08.950 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.943+0000 7f66a659c640 1 -- 172.21.15.33:0/334166126 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557959308f30 msgr2=0x55795930d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:08.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.943+0000 7f66a659c640 1 --2- 172.21.15.33:0/334166126 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557959308f30 0x55795930d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:08.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.943+0000 7f66a659c640 1 -- 172.21.15.33:0/334166126 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579592ffa30 msgr2=0x55795930c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:08.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.943+0000 7f66a659c640 1 --2- 172.21.15.33:0/334166126 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579592ffa30 0x55795930c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:08.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.943+0000 7f66a659c640 1 -- 172.21.15.33:0/334166126 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55795937e170 con 0x557959300350 2024-03-19T02:58:08.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.943+0000 7f66a6d9d640 1 --2- 172.21.15.33:0/334166126 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579592ffa30 0x55795930c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:58:08.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.943+0000 7f66a659c640 1 --2- 172.21.15.33:0/334166126 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557959300350 0x55795930cdc0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x557959308e50 tx=0x7f6690010c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88ae2f64a49c784b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:08.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.943+0000 7f668e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:58:08.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.943+0000 7f668e7fc640 1 -- 172.21.15.33:0/334166126 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f669001c820 con 0x557959300350 2024-03-19T02:58:08.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.943+0000 7f668e7fc640 1 -- 172.21.15.33:0/334166126 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f669001c9c0 con 0x557959300350 2024-03-19T02:58:08.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.943+0000 7f66afc2a740 1 -- 172.21.15.33:0/334166126 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f669001ccd0 con 0x557959300350 2024-03-19T02:58:08.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.943+0000 7f668e7fc640 1 -- 172.21.15.33:0/334166126 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f6690004050 con 0x557959300350 2024-03-19T02:58:08.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.943+0000 7f66afc2a740 10 client.16626 fetch_fsmap learned FSMap version 25 2024-03-19T02:58:08.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.943+0000 7f66afc2a740 1 -- 172.21.15.33:0/334166126 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55795937f470 con 0x557959300350 2024-03-19T02:58:08.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.944+0000 7f668e7fc640 1 -- 172.21.15.33:0/334166126 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f669001ccd0 con 0x557959300350 2024-03-19T02:58:08.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.944+0000 7f668e7fc640 1 -- 172.21.15.33:0/334166126 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f6690034020 con 0x557959300350 2024-03-19T02:58:08.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.944+0000 7f66afc2a740 10 client.16626 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:58:08.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.944+0000 7f66afc2a740 10 client.16626 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:58:08.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.944+0000 7f66afc2a740 20 client.16626 populate_metadata read hostname 'smithi033' 2024-03-19T02:58:08.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.944+0000 7f66afc2a740 1 --2- 172.21.15.33:0/334166126 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55795937fbc0 0x557959381fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:08.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.944+0000 7f66afc2a740 4 client.16626 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:58:08.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.944+0000 7f66afc2a740 1 -- 172.21.15.33:0/334166126 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55795926ad90 con 0x55795937fbc0 2024-03-19T02:58:08.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.944+0000 7f66a6d9d640 1 --2- 172.21.15.33:0/334166126 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55795937fbc0 0x557959381fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:08.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.944+0000 7f668e7fc640 10 client.16626 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:58:08.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.944+0000 7f66a6d9d640 1 --2- 172.21.15.33:0/334166126 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55795937fbc0 0x557959381fe0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=758a5ca3c7687438 server_cookie=842cd91aa811a448 in_seq=0 out_seq=0 2024-03-19T02:58:08.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.944+0000 7f668e7fc640 1 -- 172.21.15.33:0/334166126 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55795926ad90 con 0x55795937fbc0 2024-03-19T02:58:08.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.944+0000 7f668e7fc640 10 client.16626 handle_command_reply: tid=0 2024-03-19T02:58:08.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.945+0000 7f66afc2a740 1 client.16626 shutdown 2024-03-19T02:58:08.952 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.945+0000 7f66afc2a740 1 -- 172.21.15.33:0/334166126 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557959300350 msgr2=0x55795930cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:08.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.945+0000 7f66afc2a740 1 --2- 172.21.15.33:0/334166126 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557959300350 0x55795930cdc0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x557959308e50 tx=0x7f6690010c40 comp rx=0 tx=0).stop 2024-03-19T02:58:08.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.945+0000 7f66afc2a740 1 -- 172.21.15.33:0/334166126 shutdown_connections 2024-03-19T02:58:08.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.945+0000 7f66afc2a740 1 --2- 172.21.15.33:0/334166126 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55795937fbc0 0x557959381fe0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:08.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.945+0000 7f66afc2a740 1 --2- 172.21.15.33:0/334166126 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557959308f30 0x55795930d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:08.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.945+0000 7f668e7fc640 0 client.16626 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:58:08.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.945+0000 7f66afc2a740 1 --2- 172.21.15.33:0/334166126 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557959300350 0x55795930cdc0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:08.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.945+0000 7f66afc2a740 1 --2- 172.21.15.33:0/334166126 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579592ffa30 0x55795930c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:08.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.945+0000 7f66afc2a740 1 -- 172.21.15.33:0/334166126 >> 172.21.15.33:0/334166126 conn(0x5579592fa160 msgr2=0x5579592fbde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:08.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.945+0000 7f66afc2a740 1 -- 172.21.15.33:0/334166126 shutdown_connections 2024-03-19T02:58:08.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.945+0000 7f66afc2a740 1 -- 172.21.15.33:0/334166126 wait complete. 2024-03-19T02:58:08.953 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.945+0000 7f66afc2a740 20 client.16626 trim_cache size 0 max 16384 2024-03-19T02:58:08.954 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:58:08.954 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-03-19T02:58:08.954 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-03-19T02:58:08.954 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:58:08.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.945+0000 7f66ad043640 1 -- 172.21.15.33:0/591857219 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f66940a2970 msgr2=0x7f66940a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:08.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.945+0000 7f66ad043640 1 --2- 172.21.15.33:0/591857219 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f66940a2970 0x7f66940a4e30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6698005fd0 tx=0x7f6698011810 comp rx=0 tx=0).stop 2024-03-19T02:58:08.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.945+0000 7f66ad043640 1 -- 172.21.15.33:0/591857219 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66a81017b0 msgr2=0x7f66a8197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:08.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.945+0000 7f66ad043640 1 --2- 172.21.15.33:0/591857219 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66a81017b0 0x7f66a8197fc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f66a0009230 tx=0x7f66a002c5d0 comp rx=0 tx=0).stop 2024-03-19T02:58:08.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.945+0000 7f66ad043640 1 -- 172.21.15.33:0/591857219 shutdown_connections 2024-03-19T02:58:08.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.945+0000 7f66ad043640 1 --2- 172.21.15.33:0/591857219 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f66940a2970 0x7f66940a4e30 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:08.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.945+0000 7f66ad043640 1 --2- 172.21.15.33:0/591857219 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f66a81077c0 0x7f66a819c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:08.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.945+0000 7f66ad043640 1 --2- 172.21.15.33:0/591857219 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f66a8102170 0x7f66a8198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:08.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.945+0000 7f66ad043640 1 --2- 172.21.15.33:0/591857219 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66a81017b0 0x7f66a8197fc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:08.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.945+0000 7f66ad043640 1 -- 172.21.15.33:0/591857219 >> 172.21.15.33:0/591857219 conn(0x7f66a8078c20 msgr2=0x7f66a81042e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:08.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.945+0000 7f66ad043640 1 -- 172.21.15.33:0/591857219 shutdown_connections 2024-03-19T02:58:08.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:08.946+0000 7f66ad043640 1 -- 172.21.15.33:0/591857219 wait complete. 2024-03-19T02:58:08.958 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:58:08.958 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:58:08.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:08 smithi096 ceph-mon[35946]: pgmap v143: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-19T02:58:09.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.061+0000 7f6c0d310640 1 -- 172.21.15.33:0/1290260554 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c08102ab0 msgr2=0x7f6c08102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:09.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.061+0000 7f6c0d310640 1 --2- 172.21.15.33:0/1290260554 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c08102ab0 0x7f6c08102e90 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f6bf8009900 tx=0x7f6bf802bcd0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.061+0000 7f6c0d310640 1 -- 172.21.15.33:0/1290260554 shutdown_connections 2024-03-19T02:58:09.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.061+0000 7f6c0d310640 1 --2- 172.21.15.33:0/1290260554 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6c0810be80 0x7f6c0810e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.061+0000 7f6c0d310640 1 --2- 172.21.15.33:0/1290260554 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c081033d0 0x7f6c0810b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.061+0000 7f6c0d310640 1 --2- 172.21.15.33:0/1290260554 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c08102ab0 0x7f6c08102e90 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.061+0000 7f6c0d310640 1 -- 172.21.15.33:0/1290260554 >> 172.21.15.33:0/1290260554 conn(0x7f6c08077880 msgr2=0x7f6c08077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:09.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.061+0000 7f6c0d310640 1 -- 172.21.15.33:0/1290260554 shutdown_connections 2024-03-19T02:58:09.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.061+0000 7f6c0d310640 1 -- 172.21.15.33:0/1290260554 wait complete. 2024-03-19T02:58:09.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.062+0000 7f6c0d310640 1 Processor -- start 2024-03-19T02:58:09.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.062+0000 7f6c0d310640 1 -- start start 2024-03-19T02:58:09.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.062+0000 7f6c0d310640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c08102ab0 0x7f6c08100700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.062+0000 7f6c0d310640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6c081033d0 0x7f6c08100c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.062+0000 7f6c0d310640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c0810be80 0x7f6c08101180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.062+0000 7f6c0d310640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6c08111aa0 con 0x7f6c08102ab0 2024-03-19T02:58:09.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.062+0000 7f6c0d310640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6c08111920 con 0x7f6c0810be80 2024-03-19T02:58:09.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.062+0000 7f6c0d310640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6c08111c20 con 0x7f6c081033d0 2024-03-19T02:58:09.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.062+0000 7f6c06ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c08102ab0 0x7f6c08100700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.062+0000 7f6c06ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c08102ab0 0x7f6c08100700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49742/0 (socket says 172.21.15.33:49742) 2024-03-19T02:58:09.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.062+0000 7f6c077fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c0810be80 0x7f6c08101180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.062+0000 7f6c06ffd640 1 -- 172.21.15.33:0/2859320389 learned_addr learned my addr 172.21.15.33:0/2859320389 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:09.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.062+0000 7f6c067fc640 1 --2- 172.21.15.33:0/2859320389 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6c081033d0 0x7f6c08100c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.063+0000 7f6c06ffd640 1 -- 172.21.15.33:0/2859320389 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6c081033d0 msgr2=0x7f6c08100c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:09.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.063+0000 7f6c06ffd640 1 --2- 172.21.15.33:0/2859320389 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6c081033d0 0x7f6c08100c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.063+0000 7f6c06ffd640 1 -- 172.21.15.33:0/2859320389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c0810be80 msgr2=0x7f6c08101180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:09.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.063+0000 7f6c06ffd640 1 --2- 172.21.15.33:0/2859320389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c0810be80 0x7f6c08101180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.063+0000 7f6c06ffd640 1 -- 172.21.15.33:0/2859320389 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c081a6570 con 0x7f6c08102ab0 2024-03-19T02:58:09.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.063+0000 7f6c077fe640 1 --2- 172.21.15.33:0/2859320389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c0810be80 0x7f6c08101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:58:09.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.063+0000 7f6c06ffd640 1 --2- 172.21.15.33:0/2859320389 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c08102ab0 0x7f6c08100700 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f6bf8005940 tx=0x7f6bf802c690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:09.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.063+0000 7f6beffff640 1 -- 172.21.15.33:0/2859320389 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf8043020 con 0x7f6c08102ab0 2024-03-19T02:58:09.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.063+0000 7f6beffff640 1 -- 172.21.15.33:0/2859320389 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6bf8004e00 con 0x7f6c08102ab0 2024-03-19T02:58:09.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.063+0000 7f6c0d310640 1 -- 172.21.15.33:0/2859320389 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c081a6800 con 0x7f6c08102ab0 2024-03-19T02:58:09.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.063+0000 7f6beffff640 1 -- 172.21.15.33:0/2859320389 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf8003970 con 0x7f6c08102ab0 2024-03-19T02:58:09.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.063+0000 7f6c0d310640 1 -- 172.21.15.33:0/2859320389 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c081a6d50 con 0x7f6c08102ab0 2024-03-19T02:58:09.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.064+0000 7f6c0d310640 1 -- 172.21.15.33:0/2859320389 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c08101c30 con 0x7f6c08102ab0 2024-03-19T02:58:09.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.065+0000 7f6beffff640 1 -- 172.21.15.33:0/2859320389 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f6bf8046020 con 0x7f6c08102ab0 2024-03-19T02:58:09.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.065+0000 7f6beffff640 1 --2- 172.21.15.33:0/2859320389 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6bf00a2970 0x7f6bf00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.066+0000 7f6c067fc640 1 --2- 172.21.15.33:0/2859320389 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6bf00a2970 0x7f6bf00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.066+0000 7f6beffff640 1 -- 172.21.15.33:0/2859320389 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f6bf80f1060 con 0x7f6c08102ab0 2024-03-19T02:58:09.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.066+0000 7f6c067fc640 1 --2- 172.21.15.33:0/2859320389 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6bf00a2970 0x7f6bf00a4e30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6c08102190 tx=0x7f6bfc005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:09.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.070+0000 7f6beffff640 1 -- 172.21.15.33:0/2859320389 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6bf804d020 con 0x7f6c08102ab0 2024-03-19T02:58:09.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.297+0000 7f6c0d310640 1 -- 172.21.15.33:0/2859320389 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6c08106f70 con 0x7f6c08102ab0 2024-03-19T02:58:09.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.298+0000 7f6beffff640 1 -- 172.21.15.33:0/2859320389 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11867 (secure 0 0 0) 0x7f6bf80bdb20 con 0x7f6c08102ab0 2024-03-19T02:58:09.299 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:58:09.299 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:50:52.201798+0000","modified":"2024-03-19T02:53:11.748697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24488,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/3731232169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":3731232169},{"type":"v1","addr":"172.21.15.96:6839","nonce":3731232169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24457,24488,24509]},"id":1}]} 2024-03-19T02:58:09.299 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-03-19T02:58:09.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.300+0000 7f6c0d310640 1 -- 172.21.15.33:0/2859320389 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6bf00a2970 msgr2=0x7f6bf00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:09.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.300+0000 7f6c0d310640 1 --2- 172.21.15.33:0/2859320389 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6bf00a2970 0x7f6bf00a4e30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6c08102190 tx=0x7f6bfc005f90 comp rx=0 tx=0).stop 2024-03-19T02:58:09.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.300+0000 7f6c0d310640 1 -- 172.21.15.33:0/2859320389 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c08102ab0 msgr2=0x7f6c08100700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:09.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.300+0000 7f6c0d310640 1 --2- 172.21.15.33:0/2859320389 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c08102ab0 0x7f6c08100700 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f6bf8005940 tx=0x7f6bf802c690 comp rx=0 tx=0).stop 2024-03-19T02:58:09.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.300+0000 7f6c0d310640 1 -- 172.21.15.33:0/2859320389 shutdown_connections 2024-03-19T02:58:09.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.300+0000 7f6c0d310640 1 --2- 172.21.15.33:0/2859320389 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6bf00a2970 0x7f6bf00a4e30 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.300+0000 7f6c0d310640 1 --2- 172.21.15.33:0/2859320389 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6c081033d0 0x7f6c08100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.300+0000 7f6c0d310640 1 --2- 172.21.15.33:0/2859320389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c0810be80 0x7f6c08101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.300+0000 7f6c0d310640 1 --2- 172.21.15.33:0/2859320389 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c08102ab0 0x7f6c08100700 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.300+0000 7f6c0d310640 1 -- 172.21.15.33:0/2859320389 >> 172.21.15.33:0/2859320389 conn(0x7f6c08077880 msgr2=0x7f6c080fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:09.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.300+0000 7f6c0d310640 1 -- 172.21.15.33:0/2859320389 shutdown_connections 2024-03-19T02:58:09.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.300+0000 7f6c0d310640 1 -- 172.21.15.33:0/2859320389 wait complete. 2024-03-19T02:58:09.312 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:58:09.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.416+0000 7f0e7cede640 1 -- 172.21.15.33:0/3213663168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e7810bd80 msgr2=0x7f0e7810e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:09.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.416+0000 7f0e7cede640 1 --2- 172.21.15.33:0/3213663168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e7810bd80 0x7f0e7810e220 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f0e70009ca0 tx=0x7f0e7002c180 comp rx=0 tx=0).stop 2024-03-19T02:58:09.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.417+0000 7f0e7cede640 1 -- 172.21.15.33:0/3213663168 shutdown_connections 2024-03-19T02:58:09.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.417+0000 7f0e7cede640 1 --2- 172.21.15.33:0/3213663168 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e78103360 0x7f0e7810b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.417+0000 7f0e7cede640 1 --2- 172.21.15.33:0/3213663168 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e7810bd80 0x7f0e7810e220 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.417+0000 7f0e7cede640 1 --2- 172.21.15.33:0/3213663168 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e78102a40 0x7f0e78102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.417+0000 7f0e7cede640 1 -- 172.21.15.33:0/3213663168 >> 172.21.15.33:0/3213663168 conn(0x7f0e780778d0 msgr2=0x7f0e78077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:09.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.417+0000 7f0e7cede640 1 -- 172.21.15.33:0/3213663168 shutdown_connections 2024-03-19T02:58:09.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.417+0000 7f0e7cede640 1 -- 172.21.15.33:0/3213663168 wait complete. 2024-03-19T02:58:09.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.417+0000 7f0e7cede640 1 Processor -- start 2024-03-19T02:58:09.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.417+0000 7f0e7cede640 1 -- start start 2024-03-19T02:58:09.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.417+0000 7f0e7cede640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e78102a40 0x7f0e78197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.417+0000 7f0e7cede640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e78103360 0x7f0e78198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.418+0000 7f0e7cede640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e7810bd80 0x7f0e7819c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.418+0000 7f0e7cede640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0e78111a30 con 0x7f0e7810bd80 2024-03-19T02:58:09.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.418+0000 7f0e7cede640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0e781118b0 con 0x7f0e78103360 2024-03-19T02:58:09.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.418+0000 7f0e7cede640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f0e78111bb0 con 0x7f0e78102a40 2024-03-19T02:58:09.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.418+0000 7f0e76d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e7810bd80 0x7f0e7819c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.418+0000 7f0e76d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e7810bd80 0x7f0e7819c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49754/0 (socket says 172.21.15.33:49754) 2024-03-19T02:58:09.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.418+0000 7f0e76d9d640 1 -- 172.21.15.33:0/2422705694 learned_addr learned my addr 172.21.15.33:0/2422705694 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:09.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.418+0000 7f0e7659c640 1 --2- 172.21.15.33:0/2422705694 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e78102a40 0x7f0e78197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.418+0000 7f0e6ffff640 1 --2- 172.21.15.33:0/2422705694 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e78103360 0x7f0e78198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.418+0000 7f0e76d9d640 1 -- 172.21.15.33:0/2422705694 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e78102a40 msgr2=0x7f0e78197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:09.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.418+0000 7f0e76d9d640 1 --2- 172.21.15.33:0/2422705694 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e78102a40 0x7f0e78197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.418+0000 7f0e76d9d640 1 -- 172.21.15.33:0/2422705694 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e78103360 msgr2=0x7f0e78198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:09.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.418+0000 7f0e76d9d640 1 --2- 172.21.15.33:0/2422705694 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e78103360 0x7f0e78198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.418+0000 7f0e76d9d640 1 -- 172.21.15.33:0/2422705694 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e7819cee0 con 0x7f0e7810bd80 2024-03-19T02:58:09.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.418+0000 7f0e76d9d640 1 --2- 172.21.15.33:0/2422705694 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e7810bd80 0x7f0e7819c7b0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f0e7002c790 tx=0x7f0e70035db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:09.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.418+0000 7f0e6f7fe640 1 -- 172.21.15.33:0/2422705694 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e70039040 con 0x7f0e7810bd80 2024-03-19T02:58:09.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.418+0000 7f0e6f7fe640 1 -- 172.21.15.33:0/2422705694 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0e70044070 con 0x7f0e7810bd80 2024-03-19T02:58:09.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.418+0000 7f0e6f7fe640 1 -- 172.21.15.33:0/2422705694 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e70032850 con 0x7f0e7810bd80 2024-03-19T02:58:09.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.418+0000 7f0e7cede640 1 -- 172.21.15.33:0/2422705694 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e7819d170 con 0x7f0e7810bd80 2024-03-19T02:58:09.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.418+0000 7f0e7cede640 1 -- 172.21.15.33:0/2422705694 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e781a4a60 con 0x7f0e7810bd80 2024-03-19T02:58:09.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.420+0000 7f0e6f7fe640 1 -- 172.21.15.33:0/2422705694 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f0e700329f0 con 0x7f0e7810bd80 2024-03-19T02:58:09.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.421+0000 7f0e7fb9c740 1 Processor -- start 2024-03-19T02:58:09.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.421+0000 7f0e7fb9c740 1 -- start start 2024-03-19T02:58:09.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.421+0000 7f0e6f7fe640 1 --2- 172.21.15.33:0/2422705694 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f0e640a2970 0x7f0e640a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.421+0000 7f0e7fb9c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5572740382d0 0x5572740386b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.421+0000 7f0e7fb9c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557274038bf0 0x557274041290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.421+0000 7f0e7fb9c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5572740417d0 0x557274043bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.421+0000 7f0e7fb9c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557273f9b4b0 con 0x557274038bf0 2024-03-19T02:58:09.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.421+0000 7f0e7fb9c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557273f43d70 con 0x5572740382d0 2024-03-19T02:58:09.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.421+0000 7f0e7fb9c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x557273f97550 con 0x5572740417d0 2024-03-19T02:58:09.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.421+0000 7f0e6f7fe640 1 -- 172.21.15.33:0/2422705694 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f0e700f1480 con 0x7f0e7810bd80 2024-03-19T02:58:09.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.421+0000 7f0e6ffff640 1 --2- 172.21.15.33:0/2422705694 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f0e640a2970 0x7f0e640a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.421+0000 7f0e6ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557274038bf0 0x557274041290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.421+0000 7f0e6ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557274038bf0 0x557274041290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49770/0 (socket says 172.21.15.33:49770) 2024-03-19T02:58:09.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.421+0000 7f0e7659c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5572740382d0 0x5572740386b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.421+0000 7f0e6ffff640 1 -- 172.21.15.33:0/2153338942 learned_addr learned my addr 172.21.15.33:0/2153338942 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:09.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.421+0000 7f0e76d9d640 1 --2- 172.21.15.33:0/2153338942 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5572740417d0 0x557274043bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.422+0000 7f0e7659c640 1 -- 172.21.15.33:0/2153338942 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5572740417d0 msgr2=0x557274043bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:09.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.422+0000 7f0e7659c640 1 --2- 172.21.15.33:0/2153338942 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5572740417d0 0x557274043bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.422+0000 7f0e7659c640 1 -- 172.21.15.33:0/2153338942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557274038bf0 msgr2=0x557274041290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:09.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.422+0000 7f0e7659c640 1 --2- 172.21.15.33:0/2153338942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557274038bf0 0x557274041290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.422+0000 7f0e7659c640 1 -- 172.21.15.33:0/2153338942 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557273f66210 con 0x5572740382d0 2024-03-19T02:58:09.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.422+0000 7f0e6ffff640 1 --2- 172.21.15.33:0/2153338942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557274038bf0 0x557274041290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:58:09.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.422+0000 7f0e6ffff640 1 --2- 172.21.15.33:0/2422705694 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f0e640a2970 0x7f0e640a4e30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0e60011950 tx=0x7f0e60009b90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:09.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.422+0000 7f0e7659c640 1 --2- 172.21.15.33:0/2153338942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5572740382d0 0x5572740386b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0e68009c20 tx=0x7f0e68011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73d2c1c8d4179e27 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:09.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.422+0000 7f0e6cff9640 1 -- 172.21.15.33:0/2153338942 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e68004050 con 0x5572740382d0 2024-03-19T02:58:09.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.422+0000 7f0e6cff9640 1 -- 172.21.15.33:0/2153338942 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0e680054d0 con 0x5572740382d0 2024-03-19T02:58:09.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.422+0000 7f0e6cff9640 1 -- 172.21.15.33:0/2153338942 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e68004bc0 con 0x5572740382d0 2024-03-19T02:58:09.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.423+0000 7f0e7fb9c740 1 -- 172.21.15.33:0/2153338942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5572740382d0 msgr2=0x5572740386b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:09.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.423+0000 7f0e7fb9c740 1 --2- 172.21.15.33:0/2153338942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5572740382d0 0x5572740386b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0e68009c20 tx=0x7f0e68011810 comp rx=0 tx=0).stop 2024-03-19T02:58:09.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.423+0000 7f0e7fb9c740 1 -- 172.21.15.33:0/2153338942 shutdown_connections 2024-03-19T02:58:09.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.423+0000 7f0e7fb9c740 1 --2- 172.21.15.33:0/2153338942 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5572740417d0 0x557274043bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.423+0000 7f0e7fb9c740 1 --2- 172.21.15.33:0/2153338942 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557274038bf0 0x557274041290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.423+0000 7f0e7fb9c740 1 --2- 172.21.15.33:0/2153338942 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5572740382d0 0x5572740386b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.423+0000 7f0e7fb9c740 1 -- 172.21.15.33:0/2153338942 >> 172.21.15.33:0/2153338942 conn(0x557274032a00 msgr2=0x557274032de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:09.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.423+0000 7f0e7fb9c740 1 -- 172.21.15.33:0/2153338942 shutdown_connections 2024-03-19T02:58:09.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.423+0000 7f0e7fb9c740 1 -- 172.21.15.33:0/2153338942 wait complete. 2024-03-19T02:58:09.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.423+0000 7f0e7fb9c740 1 -- start start 2024-03-19T02:58:09.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.423+0000 7f0e7fb9c740 1 Processor -- start 2024-03-19T02:58:09.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.423+0000 7f0e7fb9c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5572740382d0 0x55727403e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.424+0000 7f0e7fb9c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557274038bf0 0x55727403ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.424+0000 7f0e7fb9c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5572740417d0 0x55727403f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.431 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.424+0000 7f0e7fb9c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5572740489d0 con 0x5572740382d0 2024-03-19T02:58:09.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.424+0000 7f0e7fb9c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557273f57310 con 0x5572740417d0 2024-03-19T02:58:09.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.424+0000 7f0e7fb9c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x557274048b50 con 0x557274038bf0 2024-03-19T02:58:09.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.424+0000 7f0e7fb9c740 10 client.0 _ll_register_callbacks cb 0x557273f4b2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:58:09.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.424+0000 7f0e7659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5572740382d0 0x55727403e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.424+0000 7f0e7659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5572740382d0 0x55727403e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49784/0 (socket says 172.21.15.33:49784) 2024-03-19T02:58:09.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.424+0000 7f0e7659c640 1 -- 172.21.15.33:0/3180302340 learned_addr learned my addr 172.21.15.33:0/3180302340 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:09.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.424+0000 7f0e6ffff640 1 --2- 172.21.15.33:0/3180302340 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557274038bf0 0x55727403ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.424+0000 7f0e76d9d640 1 --2- 172.21.15.33:0/3180302340 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5572740417d0 0x55727403f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.424+0000 7f0e7659c640 1 -- 172.21.15.33:0/3180302340 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557274038bf0 msgr2=0x55727403ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:09.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.424+0000 7f0e7659c640 1 --2- 172.21.15.33:0/3180302340 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557274038bf0 0x55727403ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.424+0000 7f0e7659c640 1 -- 172.21.15.33:0/3180302340 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5572740417d0 msgr2=0x55727403f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:09.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.424+0000 7f0e7659c640 1 --2- 172.21.15.33:0/3180302340 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5572740417d0 0x55727403f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.424+0000 7f0e7659c640 1 -- 172.21.15.33:0/3180302340 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557273caf3f0 con 0x5572740382d0 2024-03-19T02:58:09.432 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.424+0000 7f0e7659c640 1 --2- 172.21.15.33:0/3180302340 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5572740382d0 0x55727403e7c0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f0e680057c0 tx=0x7f0e68004300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bd00034fa3d99b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:09.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.424+0000 7f0e5dffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:58:09.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.424+0000 7f0e5dffb640 1 -- 172.21.15.33:0/3180302340 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e68004730 con 0x5572740382d0 2024-03-19T02:58:09.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.424+0000 7f0e7fb9c740 1 -- 172.21.15.33:0/3180302340 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e680054d0 con 0x5572740382d0 2024-03-19T02:58:09.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.424+0000 7f0e5dffb640 1 -- 172.21.15.33:0/3180302340 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f0e68003a00 con 0x5572740382d0 2024-03-19T02:58:09.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.424+0000 7f0e5dffb640 1 -- 172.21.15.33:0/3180302340 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0e68033050 con 0x5572740382d0 2024-03-19T02:58:09.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.424+0000 7f0e7fb9c740 10 client.16644 fetch_fsmap learned FSMap version 25 2024-03-19T02:58:09.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.424+0000 7f0e7fb9c740 1 -- 172.21.15.33:0/3180302340 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x557273d797d0 con 0x5572740382d0 2024-03-19T02:58:09.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.425+0000 7f0e5dffb640 1 -- 172.21.15.33:0/3180302340 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0e680054d0 con 0x5572740382d0 2024-03-19T02:58:09.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.425+0000 7f0e5dffb640 1 -- 172.21.15.33:0/3180302340 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f0e680050f0 con 0x5572740382d0 2024-03-19T02:58:09.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.425+0000 7f0e7fb9c740 10 client.16644 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:58:09.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.425+0000 7f0e7fb9c740 10 client.16644 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T02:58:09.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.425+0000 7f0e7fb9c740 20 client.16644 populate_metadata read hostname 'smithi033' 2024-03-19T02:58:09.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.425+0000 7f0e7fb9c740 1 --2- 172.21.15.33:0/3180302340 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x557274055ee0 0x5572740582d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.425+0000 7f0e7fb9c740 4 client.16644 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:58:09.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.425+0000 7f0e7fb9c740 1 -- 172.21.15.33:0/3180302340 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557273ff93e0 con 0x557274055ee0 2024-03-19T02:58:09.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.425+0000 7f0e6ffff640 1 --2- 172.21.15.33:0/3180302340 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x557274055ee0 0x5572740582d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.426+0000 7f0e6ffff640 1 --2- 172.21.15.33:0/3180302340 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x557274055ee0 0x5572740582d0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4ea8dc40a586036d server_cookie=ab95c2993be1486b in_seq=0 out_seq=0 2024-03-19T02:58:09.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.426+0000 7f0e5dffb640 10 client.16644 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T02:58:09.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.426+0000 7f0e5dffb640 1 -- 172.21.15.33:0/3180302340 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557273ff93e0 con 0x557274055ee0 2024-03-19T02:58:09.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.426+0000 7f0e5dffb640 10 client.16644 handle_command_reply: tid=0 2024-03-19T02:58:09.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.426+0000 7f0e7fb9c740 1 client.16644 shutdown 2024-03-19T02:58:09.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.426+0000 7f0e7fb9c740 1 -- 172.21.15.33:0/3180302340 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5572740382d0 msgr2=0x55727403e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:09.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.426+0000 7f0e7fb9c740 1 --2- 172.21.15.33:0/3180302340 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5572740382d0 0x55727403e7c0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f0e680057c0 tx=0x7f0e68004300 comp rx=0 tx=0).stop 2024-03-19T02:58:09.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.426+0000 7f0e7fb9c740 1 -- 172.21.15.33:0/3180302340 shutdown_connections 2024-03-19T02:58:09.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.426+0000 7f0e7fb9c740 1 --2- 172.21.15.33:0/3180302340 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x557274055ee0 0x5572740582d0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.426+0000 7f0e7fb9c740 1 --2- 172.21.15.33:0/3180302340 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557274038bf0 0x55727403ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.426+0000 7f0e7fb9c740 1 --2- 172.21.15.33:0/3180302340 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5572740417d0 0x55727403f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.426+0000 7f0e7fb9c740 1 --2- 172.21.15.33:0/3180302340 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5572740382d0 0x55727403e7c0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.426+0000 7f0e5dffb640 0 client.16644 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T02:58:09.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.426+0000 7f0e7fb9c740 1 -- 172.21.15.33:0/3180302340 >> 172.21.15.33:0/3180302340 conn(0x557274032a00 msgr2=0x557274034750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:09.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.426+0000 7f0e7fb9c740 1 -- 172.21.15.33:0/3180302340 shutdown_connections 2024-03-19T02:58:09.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.426+0000 7f0e7fb9c740 1 -- 172.21.15.33:0/3180302340 wait complete. 2024-03-19T02:58:09.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.427+0000 7f0e7fb9c740 20 client.16644 trim_cache size 0 max 16384 2024-03-19T02:58:09.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.440+0000 7f0e7fb9c740 1 Processor -- start 2024-03-19T02:58:09.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.440+0000 7f0e7fb9c740 1 -- start start 2024-03-19T02:58:09.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.440+0000 7f0e7fb9c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5572740382d0 0x557274040ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.440+0000 7f0e7fb9c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557274038bf0 0x5572740517a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.440+0000 7f0e7fb9c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5572740417d0 0x55727405dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.440+0000 7f0e7fb9c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0e781af1e0 con 0x5572740382d0 2024-03-19T02:58:09.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.440+0000 7f0e7fb9c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0e781acab0 con 0x557274038bf0 2024-03-19T02:58:09.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.440+0000 7f0e7fb9c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f0e781aa8e0 con 0x5572740417d0 2024-03-19T02:58:09.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.440+0000 7f0e7659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5572740382d0 0x557274040ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.440+0000 7f0e7659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5572740382d0 0x557274040ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49786/0 (socket says 172.21.15.33:49786) 2024-03-19T02:58:09.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.440+0000 7f0e7659c640 1 -- 172.21.15.33:0/1535114940 learned_addr learned my addr 172.21.15.33:0/1535114940 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:09.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.440+0000 7f0e6ffff640 1 --2- 172.21.15.33:0/1535114940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557274038bf0 0x5572740517a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.440+0000 7f0e76d9d640 1 --2- 172.21.15.33:0/1535114940 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5572740417d0 0x55727405dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.440+0000 7f0e7659c640 1 -- 172.21.15.33:0/1535114940 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5572740417d0 msgr2=0x55727405dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:09.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.440+0000 7f0e7659c640 1 --2- 172.21.15.33:0/1535114940 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5572740417d0 0x55727405dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.440+0000 7f0e7659c640 1 -- 172.21.15.33:0/1535114940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557274038bf0 msgr2=0x5572740517a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:09.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.440+0000 7f0e7659c640 1 --2- 172.21.15.33:0/1535114940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557274038bf0 0x5572740517a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.440+0000 7f0e7659c640 1 -- 172.21.15.33:0/1535114940 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557274038a20 con 0x5572740382d0 2024-03-19T02:58:09.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.440+0000 7f0e7659c640 1 --2- 172.21.15.33:0/1535114940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5572740382d0 0x557274040ef0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f0e6802e510 tx=0x7f0e68009640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec21213b35989066 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:09.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.440+0000 7f0e5ffff640 1 -- 172.21.15.33:0/1535114940 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e68009b70 con 0x5572740382d0 2024-03-19T02:58:09.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.441+0000 7f0e5ffff640 1 -- 172.21.15.33:0/1535114940 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0e68003cb0 con 0x5572740382d0 2024-03-19T02:58:09.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.441+0000 7f0e5ffff640 1 -- 172.21.15.33:0/1535114940 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e68019430 con 0x5572740382d0 2024-03-19T02:58:09.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.441+0000 7f0e7fb9c740 1 -- 172.21.15.33:0/1535114940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5572740382d0 msgr2=0x557274040ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:09.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.441+0000 7f0e7fb9c740 1 --2- 172.21.15.33:0/1535114940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5572740382d0 0x557274040ef0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f0e6802e510 tx=0x7f0e68009640 comp rx=0 tx=0).stop 2024-03-19T02:58:09.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.441+0000 7f0e7fb9c740 1 -- 172.21.15.33:0/1535114940 shutdown_connections 2024-03-19T02:58:09.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.441+0000 7f0e7fb9c740 1 --2- 172.21.15.33:0/1535114940 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5572740417d0 0x55727405dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.441+0000 7f0e7fb9c740 1 --2- 172.21.15.33:0/1535114940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557274038bf0 0x5572740517a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.441+0000 7f0e7fb9c740 1 --2- 172.21.15.33:0/1535114940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5572740382d0 0x557274040ef0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.441+0000 7f0e7fb9c740 1 -- 172.21.15.33:0/1535114940 >> 172.21.15.33:0/1535114940 conn(0x557274032a00 msgr2=0x557274031ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:09.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.441+0000 7f0e7fb9c740 1 -- 172.21.15.33:0/1535114940 shutdown_connections 2024-03-19T02:58:09.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.441+0000 7f0e7fb9c740 1 -- 172.21.15.33:0/1535114940 wait complete. 2024-03-19T02:58:09.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.441+0000 7f0e7fb9c740 1 -- start start 2024-03-19T02:58:09.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.441+0000 7f0e7fb9c740 1 Processor -- start 2024-03-19T02:58:09.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.442+0000 7f0e7fb9c740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5572740382d0 0x5572740498c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.442+0000 7f0e7fb9c740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557274038bf0 0x557274049e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.442+0000 7f0e7fb9c740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5572740417d0 0x55727404a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.442+0000 7f0e7fb9c740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0e781aaf70 con 0x5572740382d0 2024-03-19T02:58:09.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.442+0000 7f0e7fb9c740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0e781ac3e0 con 0x5572740417d0 2024-03-19T02:58:09.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.442+0000 7f0e7fb9c740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f0e781ab640 con 0x557274038bf0 2024-03-19T02:58:09.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.442+0000 7f0e7fb9c740 10 client.0 _ll_register_callbacks cb 0x7f0e600114c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:58:09.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.442+0000 7f0e7659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5572740382d0 0x5572740498c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.442+0000 7f0e7659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5572740382d0 0x5572740498c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49792/0 (socket says 172.21.15.33:49792) 2024-03-19T02:58:09.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.442+0000 7f0e7659c640 1 -- 172.21.15.33:0/3389358707 learned_addr learned my addr 172.21.15.33:0/3389358707 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:09.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.442+0000 7f0e76d9d640 1 --2- 172.21.15.33:0/3389358707 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5572740417d0 0x55727404a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.442+0000 7f0e6ffff640 1 --2- 172.21.15.33:0/3389358707 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557274038bf0 0x557274049e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.442+0000 7f0e7659c640 1 -- 172.21.15.33:0/3389358707 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557274038bf0 msgr2=0x557274049e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:09.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.442+0000 7f0e7659c640 1 --2- 172.21.15.33:0/3389358707 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557274038bf0 0x557274049e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.442+0000 7f0e7659c640 1 -- 172.21.15.33:0/3389358707 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5572740417d0 msgr2=0x55727404a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:09.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.442+0000 7f0e7659c640 1 --2- 172.21.15.33:0/3389358707 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5572740417d0 0x55727404a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.442+0000 7f0e7659c640 1 -- 172.21.15.33:0/3389358707 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572740bb3d0 con 0x5572740382d0 2024-03-19T02:58:09.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.442+0000 7f0e6ffff640 1 --2- 172.21.15.33:0/3389358707 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557274038bf0 0x557274049e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:58:09.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.442+0000 7f0e7659c640 1 --2- 172.21.15.33:0/3389358707 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5572740382d0 0x5572740498c0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f0e680057c0 tx=0x7f0e680056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d098a8886a7ad75 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:09.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.442+0000 7f0e5dffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:58:09.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.442+0000 7f0e5dffb640 1 -- 172.21.15.33:0/3389358707 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e68046060 con 0x5572740382d0 2024-03-19T02:58:09.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.442+0000 7f0e5dffb640 1 -- 172.21.15.33:0/3389358707 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f0e68003a00 con 0x5572740382d0 2024-03-19T02:58:09.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.442+0000 7f0e7fb9c740 1 -- 172.21.15.33:0/3389358707 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e68003cb0 con 0x5572740382d0 2024-03-19T02:58:09.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.443+0000 7f0e5dffb640 1 -- 172.21.15.33:0/3389358707 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0e6804d050 con 0x5572740382d0 2024-03-19T02:58:09.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.443+0000 7f0e7fb9c740 10 client.16650 fetch_fsmap learned FSMap version 25 2024-03-19T02:58:09.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.443+0000 7f0e7fb9c740 1 -- 172.21.15.33:0/3389358707 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5572740bc6b0 con 0x5572740382d0 2024-03-19T02:58:09.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.443+0000 7f0e5dffb640 1 -- 172.21.15.33:0/3389358707 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0e68003cb0 con 0x5572740382d0 2024-03-19T02:58:09.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.443+0000 7f0e5dffb640 1 -- 172.21.15.33:0/3389358707 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f0e68023c90 con 0x5572740382d0 2024-03-19T02:58:09.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.443+0000 7f0e7fb9c740 10 client.16650 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:58:09.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.443+0000 7f0e7fb9c740 10 client.16650 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T02:58:09.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.443+0000 7f0e7fb9c740 20 client.16650 populate_metadata read hostname 'smithi033' 2024-03-19T02:58:09.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.443+0000 7f0e7fb9c740 1 --2- 172.21.15.33:0/3389358707 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x5572740bccc0 0x5572740bf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.443+0000 7f0e7fb9c740 4 client.16650 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:58:09.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.443+0000 7f0e7fb9c740 1 -- 172.21.15.33:0/3389358707 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557273fa89f0 con 0x5572740bccc0 2024-03-19T02:58:09.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.443+0000 7f0e6ffff640 1 --2- 172.21.15.33:0/3389358707 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x5572740bccc0 0x5572740bf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.444+0000 7f0e6ffff640 1 --2- 172.21.15.33:0/3389358707 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x5572740bccc0 0x5572740bf0b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8044db7b5039b1ab server_cookie=1094fcdb8875bc9c in_seq=0 out_seq=0 2024-03-19T02:58:09.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.444+0000 7f0e5dffb640 10 client.16650 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T02:58:09.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.444+0000 7f0e5dffb640 1 -- 172.21.15.33:0/3389358707 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557273fa89f0 con 0x5572740bccc0 2024-03-19T02:58:09.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.444+0000 7f0e5dffb640 10 client.16650 handle_command_reply: tid=0 2024-03-19T02:58:09.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.444+0000 7f0e7fb9c740 1 client.16650 shutdown 2024-03-19T02:58:09.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.444+0000 7f0e7fb9c740 1 -- 172.21.15.33:0/3389358707 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5572740382d0 msgr2=0x5572740498c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:09.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.444+0000 7f0e7fb9c740 1 --2- 172.21.15.33:0/3389358707 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5572740382d0 0x5572740498c0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f0e680057c0 tx=0x7f0e680056a0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.444+0000 7f0e7fb9c740 1 -- 172.21.15.33:0/3389358707 shutdown_connections 2024-03-19T02:58:09.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.444+0000 7f0e7fb9c740 1 --2- 172.21.15.33:0/3389358707 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x5572740bccc0 0x5572740bf0b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.444+0000 7f0e7fb9c740 1 --2- 172.21.15.33:0/3389358707 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557274038bf0 0x557274049e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.444+0000 7f0e7fb9c740 1 --2- 172.21.15.33:0/3389358707 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5572740417d0 0x55727404a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.444+0000 7f0e7fb9c740 1 --2- 172.21.15.33:0/3389358707 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5572740382d0 0x5572740498c0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.444+0000 7f0e5dffb640 0 client.16650 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T02:58:09.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.444+0000 7f0e7fb9c740 1 -- 172.21.15.33:0/3389358707 >> 172.21.15.33:0/3389358707 conn(0x557274032a00 msgr2=0x55727403e760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:09.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.444+0000 7f0e7fb9c740 1 -- 172.21.15.33:0/3389358707 shutdown_connections 2024-03-19T02:58:09.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.444+0000 7f0e7fb9c740 1 -- 172.21.15.33:0/3389358707 wait complete. 2024-03-19T02:58:09.452 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.444+0000 7f0e7fb9c740 20 client.16650 trim_cache size 0 max 16384 2024-03-19T02:58:09.452 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:58:09.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.444+0000 7f0e7cede640 1 -- 172.21.15.33:0/2422705694 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f0e640a2970 msgr2=0x7f0e640a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:09.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.444+0000 7f0e7cede640 1 --2- 172.21.15.33:0/2422705694 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f0e640a2970 0x7f0e640a4e30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0e60011950 tx=0x7f0e60009b90 comp rx=0 tx=0).stop 2024-03-19T02:58:09.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.444+0000 7f0e7cede640 1 -- 172.21.15.33:0/2422705694 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e7810bd80 msgr2=0x7f0e7819c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:09.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.444+0000 7f0e7cede640 1 --2- 172.21.15.33:0/2422705694 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e7810bd80 0x7f0e7819c7b0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f0e7002c790 tx=0x7f0e70035db0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.444+0000 7f0e7cede640 1 -- 172.21.15.33:0/2422705694 shutdown_connections 2024-03-19T02:58:09.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.444+0000 7f0e7cede640 1 --2- 172.21.15.33:0/2422705694 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f0e640a2970 0x7f0e640a4e30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.444+0000 7f0e7cede640 1 --2- 172.21.15.33:0/2422705694 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0e7810bd80 0x7f0e7819c7b0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.444+0000 7f0e7cede640 1 --2- 172.21.15.33:0/2422705694 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e78103360 0x7f0e78198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.445+0000 7f0e7cede640 1 --2- 172.21.15.33:0/2422705694 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e78102a40 0x7f0e78197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.453 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.445+0000 7f0e7cede640 1 -- 172.21.15.33:0/2422705694 >> 172.21.15.33:0/2422705694 conn(0x7f0e780778d0 msgr2=0x7f0e780fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:09.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.445+0000 7f0e7cede640 1 -- 172.21.15.33:0/2422705694 shutdown_connections 2024-03-19T02:58:09.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.445+0000 7f0e7cede640 1 -- 172.21.15.33:0/2422705694 wait complete. 2024-03-19T02:58:09.456 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:58:09.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.560+0000 7ff60badc640 1 -- 172.21.15.33:0/441065452 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff604105660 msgr2=0x7ff604105a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:09.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.560+0000 7ff60badc640 1 --2- 172.21.15.33:0/441065452 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff604105660 0x7ff604105a40 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7ff600009900 tx=0x7ff60002bc30 comp rx=0 tx=0).stop 2024-03-19T02:58:09.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.560+0000 7ff60badc640 1 -- 172.21.15.33:0/441065452 shutdown_connections 2024-03-19T02:58:09.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.560+0000 7ff60badc640 1 --2- 172.21.15.33:0/441065452 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff604100970 0x7ff604100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.560+0000 7ff60badc640 1 --2- 172.21.15.33:0/441065452 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff604101330 0x7ff60410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.560+0000 7ff60badc640 1 --2- 172.21.15.33:0/441065452 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff604105660 0x7ff604105a40 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.560+0000 7ff60badc640 1 -- 172.21.15.33:0/441065452 >> 172.21.15.33:0/441065452 conn(0x7ff6040774c0 msgr2=0x7ff6040778d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:09.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.560+0000 7ff60badc640 1 -- 172.21.15.33:0/441065452 shutdown_connections 2024-03-19T02:58:09.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.560+0000 7ff60badc640 1 -- 172.21.15.33:0/441065452 wait complete. 2024-03-19T02:58:09.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.561+0000 7ff60badc640 1 Processor -- start 2024-03-19T02:58:09.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.561+0000 7ff60badc640 1 -- start start 2024-03-19T02:58:09.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.561+0000 7ff60badc640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff604100970 0x7ff604197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.561+0000 7ff60badc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff604101330 0x7ff604198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.561+0000 7ff60badc640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff604105660 0x7ff60419c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.561+0000 7ff60badc640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff604110930 con 0x7ff604101330 2024-03-19T02:58:09.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.561+0000 7ff60badc640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff6041107b0 con 0x7ff604105660 2024-03-19T02:58:09.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.561+0000 7ff60badc640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff604110ab0 con 0x7ff604100970 2024-03-19T02:58:09.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.561+0000 7ff609077640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff604101330 0x7ff604198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.561+0000 7ff609077640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff604101330 0x7ff604198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49800/0 (socket says 172.21.15.33:49800) 2024-03-19T02:58:09.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.561+0000 7ff609077640 1 -- 172.21.15.33:0/3127482066 learned_addr learned my addr 172.21.15.33:0/3127482066 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:09.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.561+0000 7ff609878640 1 --2- 172.21.15.33:0/3127482066 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff604100970 0x7ff604197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.561+0000 7ff60a079640 1 --2- 172.21.15.33:0/3127482066 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff604105660 0x7ff60419c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.561+0000 7ff609077640 1 -- 172.21.15.33:0/3127482066 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff604100970 msgr2=0x7ff604197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:09.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.561+0000 7ff609077640 1 --2- 172.21.15.33:0/3127482066 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff604100970 0x7ff604197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.561+0000 7ff609077640 1 -- 172.21.15.33:0/3127482066 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff604105660 msgr2=0x7ff60419c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:09.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.561+0000 7ff609077640 1 --2- 172.21.15.33:0/3127482066 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff604105660 0x7ff60419c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.561+0000 7ff609077640 1 -- 172.21.15.33:0/3127482066 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff60419cfe0 con 0x7ff604101330 2024-03-19T02:58:09.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.562+0000 7ff609077640 1 --2- 172.21.15.33:0/3127482066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff604101330 0x7ff604198530 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7ff5f400e9f0 tx=0x7ff5f400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:09.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.562+0000 7ff5faffd640 1 -- 172.21.15.33:0/3127482066 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f400ccc0 con 0x7ff604101330 2024-03-19T02:58:09.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.562+0000 7ff5faffd640 1 -- 172.21.15.33:0/3127482066 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff5f400ce60 con 0x7ff604101330 2024-03-19T02:58:09.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.562+0000 7ff60badc640 1 -- 172.21.15.33:0/3127482066 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff60419d2d0 con 0x7ff604101330 2024-03-19T02:58:09.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.562+0000 7ff5faffd640 1 -- 172.21.15.33:0/3127482066 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f4010430 con 0x7ff604101330 2024-03-19T02:58:09.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.562+0000 7ff60badc640 1 -- 172.21.15.33:0/3127482066 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6041a4b60 con 0x7ff604101330 2024-03-19T02:58:09.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.564+0000 7ff5faffd640 1 -- 172.21.15.33:0/3127482066 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7ff5f4005020 con 0x7ff604101330 2024-03-19T02:58:09.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.564+0000 7ff60e79f740 1 Processor -- start 2024-03-19T02:58:09.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.565+0000 7ff60e79f740 1 -- start start 2024-03-19T02:58:09.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.565+0000 7ff60e79f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5b7cf82d0 0x55b5b7cf86b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.565+0000 7ff60e79f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5b7cf8bf0 0x55b5b7d01290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.565+0000 7ff60e79f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5b7d017d0 0x55b5b7d03bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.565+0000 7ff60e79f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b5b7c5b4b0 con 0x55b5b7cf82d0 2024-03-19T02:58:09.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.565+0000 7ff60e79f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55b5b7c03d70 con 0x55b5b7d017d0 2024-03-19T02:58:09.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.565+0000 7ff60e79f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b5b7c57550 con 0x55b5b7cf8bf0 2024-03-19T02:58:09.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.565+0000 7ff5faffd640 1 --2- 172.21.15.33:0/3127482066 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7ff5f00a2970 0x7ff5f00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.565+0000 7ff609878640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5b7cf8bf0 0x55b5b7d01290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.565+0000 7ff609878640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5b7cf8bf0 0x55b5b7d01290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:37810/0 (socket says 172.21.15.33:37810) 2024-03-19T02:58:09.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.565+0000 7ff609878640 1 -- 172.21.15.33:0/1111922886 learned_addr learned my addr 172.21.15.33:0/1111922886 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:09.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.565+0000 7ff60a079640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5b7cf82d0 0x55b5b7cf86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.565+0000 7ff609077640 1 --2- 172.21.15.33:0/1111922886 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5b7d017d0 0x55b5b7d03bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.565+0000 7ff60a079640 1 --2- 172.21.15.33:0/3127482066 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7ff5f00a2970 0x7ff5f00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.565+0000 7ff5faffd640 1 -- 172.21.15.33:0/3127482066 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7ff5f40cf610 con 0x7ff604101330 2024-03-19T02:58:09.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.565+0000 7ff609878640 1 -- 172.21.15.33:0/1111922886 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5b7d017d0 msgr2=0x55b5b7d03bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:09.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.565+0000 7ff609878640 1 --2- 172.21.15.33:0/1111922886 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5b7d017d0 0x55b5b7d03bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.565+0000 7ff609878640 1 -- 172.21.15.33:0/1111922886 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5b7cf82d0 msgr2=0x55b5b7cf86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:09.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.565+0000 7ff609878640 1 --2- 172.21.15.33:0/1111922886 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5b7cf82d0 0x55b5b7cf86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.565+0000 7ff609878640 1 -- 172.21.15.33:0/1111922886 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5b7c26210 con 0x55b5b7cf8bf0 2024-03-19T02:58:09.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.565+0000 7ff60a079640 1 --2- 172.21.15.33:0/1111922886 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5b7cf82d0 0x55b5b7cf86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:58:09.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.566+0000 7ff60a079640 1 --2- 172.21.15.33:0/3127482066 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7ff5f00a2970 0x7ff5f00a4e30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc005fd0 tx=0x7ff5fc009500 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:09.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.566+0000 7ff609878640 1 --2- 172.21.15.33:0/1111922886 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5b7cf8bf0 0x55b5b7d01290 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff600009900 tx=0x7ff6000095f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f6d8649e16173c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:09.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.566+0000 7ff5effff640 1 -- 172.21.15.33:0/1111922886 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff600045020 con 0x55b5b7cf8bf0 2024-03-19T02:58:09.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.566+0000 7ff5effff640 1 -- 172.21.15.33:0/1111922886 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff60002c7e0 con 0x55b5b7cf8bf0 2024-03-19T02:58:09.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.566+0000 7ff5effff640 1 -- 172.21.15.33:0/1111922886 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff600005460 con 0x55b5b7cf8bf0 2024-03-19T02:58:09.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.566+0000 7ff60e79f740 1 -- 172.21.15.33:0/1111922886 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5b7cf8bf0 msgr2=0x55b5b7d01290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:09.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.566+0000 7ff60e79f740 1 --2- 172.21.15.33:0/1111922886 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5b7cf8bf0 0x55b5b7d01290 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff600009900 tx=0x7ff6000095f0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.567+0000 7ff60e79f740 1 -- 172.21.15.33:0/1111922886 shutdown_connections 2024-03-19T02:58:09.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.567+0000 7ff60e79f740 1 --2- 172.21.15.33:0/1111922886 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5b7cf8bf0 0x55b5b7d01290 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.567+0000 7ff60e79f740 1 --2- 172.21.15.33:0/1111922886 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5b7d017d0 0x55b5b7d03bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.567+0000 7ff60e79f740 1 --2- 172.21.15.33:0/1111922886 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5b7cf82d0 0x55b5b7cf86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.567+0000 7ff60e79f740 1 -- 172.21.15.33:0/1111922886 >> 172.21.15.33:0/1111922886 conn(0x55b5b7cf2a00 msgr2=0x55b5b7cf2de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:09.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.567+0000 7ff60e79f740 1 -- 172.21.15.33:0/1111922886 shutdown_connections 2024-03-19T02:58:09.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.567+0000 7ff60e79f740 1 -- 172.21.15.33:0/1111922886 wait complete. 2024-03-19T02:58:09.574 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.567+0000 7ff60e79f740 1 -- start start 2024-03-19T02:58:09.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.567+0000 7ff60e79f740 1 Processor -- start 2024-03-19T02:58:09.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.567+0000 7ff60e79f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5b7cf82d0 0x55b5b7cfe7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.567+0000 7ff60e79f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5b7cf8bf0 0x55b5b7cfed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.567+0000 7ff60e79f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5b7d017d0 0x55b5b7cff270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.567+0000 7ff60e79f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b5b7d04560 con 0x55b5b7cf8bf0 2024-03-19T02:58:09.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.567+0000 7ff60e79f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55b5b7c17310 con 0x55b5b7cf82d0 2024-03-19T02:58:09.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.567+0000 7ff60e79f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b5b7d046e0 con 0x55b5b7d017d0 2024-03-19T02:58:09.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.567+0000 7ff60e79f740 10 client.0 _ll_register_callbacks cb 0x55b5b7c0b2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:58:09.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.568+0000 7ff609077640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5b7cf8bf0 0x55b5b7cfed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.568+0000 7ff609878640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5b7cf82d0 0x55b5b7cfe7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.568+0000 7ff609077640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5b7cf8bf0 0x55b5b7cfed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49814/0 (socket says 172.21.15.33:49814) 2024-03-19T02:58:09.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.568+0000 7ff609077640 1 -- 172.21.15.33:0/1569477791 learned_addr learned my addr 172.21.15.33:0/1569477791 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:09.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.568+0000 7ff60a079640 1 --2- 172.21.15.33:0/1569477791 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5b7d017d0 0x55b5b7cff270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.568+0000 7ff609878640 1 -- 172.21.15.33:0/1569477791 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5b7d017d0 msgr2=0x55b5b7cff270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:09.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.568+0000 7ff609878640 1 --2- 172.21.15.33:0/1569477791 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5b7d017d0 0x55b5b7cff270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.568+0000 7ff609878640 1 -- 172.21.15.33:0/1569477791 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5b7cf8bf0 msgr2=0x55b5b7cfed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:09.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.568+0000 7ff609878640 1 --2- 172.21.15.33:0/1569477791 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5b7cf8bf0 0x55b5b7cfed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.568+0000 7ff609878640 1 -- 172.21.15.33:0/1569477791 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5b796f3f0 con 0x55b5b7cf82d0 2024-03-19T02:58:09.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.568+0000 7ff609077640 1 --2- 172.21.15.33:0/1569477791 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5b7cf8bf0 0x55b5b7cfed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:58:09.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.568+0000 7ff609878640 1 --2- 172.21.15.33:0/1569477791 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5b7cf82d0 0x55b5b7cfe7f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff60002c170 tx=0x7ff600000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39922382ec6b0b38 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:09.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.568+0000 7ff5ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T02:58:09.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.568+0000 7ff5ed7fa640 1 -- 172.21.15.33:0/1569477791 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff600045070 con 0x55b5b7cf82d0 2024-03-19T02:58:09.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.568+0000 7ff60e79f740 1 -- 172.21.15.33:0/1569477791 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff60002c7e0 con 0x55b5b7cf82d0 2024-03-19T02:58:09.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.568+0000 7ff5ed7fa640 1 -- 172.21.15.33:0/1569477791 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7ff600004ad0 con 0x55b5b7cf82d0 2024-03-19T02:58:09.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.568+0000 7ff5ed7fa640 1 -- 172.21.15.33:0/1569477791 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff60005a050 con 0x55b5b7cf82d0 2024-03-19T02:58:09.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.568+0000 7ff60e79f740 10 client.26486 fetch_fsmap learned FSMap version 25 2024-03-19T02:58:09.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.568+0000 7ff60e79f740 1 -- 172.21.15.33:0/1569477791 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b5b7a397d0 con 0x55b5b7cf82d0 2024-03-19T02:58:09.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.569+0000 7ff5ed7fa640 1 -- 172.21.15.33:0/1569477791 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff60002c7e0 con 0x55b5b7cf82d0 2024-03-19T02:58:09.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.569+0000 7ff5ed7fa640 1 -- 172.21.15.33:0/1569477791 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7ff600041af0 con 0x55b5b7cf82d0 2024-03-19T02:58:09.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.569+0000 7ff60e79f740 10 client.26486 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:58:09.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.569+0000 7ff60e79f740 10 client.26486 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T02:58:09.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.569+0000 7ff60e79f740 20 client.26486 populate_metadata read hostname 'smithi033' 2024-03-19T02:58:09.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.569+0000 7ff60e79f740 1 --2- 172.21.15.33:0/1569477791 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55b5b7d11960 0x55b5b7d13d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.569+0000 7ff60e79f740 4 client.26486 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:58:09.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.569+0000 7ff60e79f740 1 -- 172.21.15.33:0/1569477791 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5b7cb93e0 con 0x55b5b7d11960 2024-03-19T02:58:09.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.569+0000 7ff609077640 1 --2- 172.21.15.33:0/1569477791 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55b5b7d11960 0x55b5b7d13d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.569+0000 7ff5ed7fa640 10 client.26486 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T02:58:09.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.569+0000 7ff609077640 1 --2- 172.21.15.33:0/1569477791 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55b5b7d11960 0x55b5b7d13d50 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e0a9c406d065b5f6 server_cookie=7c24068951c407d4 in_seq=0 out_seq=0 2024-03-19T02:58:09.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.570+0000 7ff5ed7fa640 1 -- 172.21.15.33:0/1569477791 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b5b7cb93e0 con 0x55b5b7d11960 2024-03-19T02:58:09.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.570+0000 7ff5ed7fa640 10 client.26486 handle_command_reply: tid=0 2024-03-19T02:58:09.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.570+0000 7ff60e79f740 1 client.26486 shutdown 2024-03-19T02:58:09.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.570+0000 7ff60e79f740 1 -- 172.21.15.33:0/1569477791 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5b7cf82d0 msgr2=0x55b5b7cfe7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:09.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.570+0000 7ff60e79f740 1 --2- 172.21.15.33:0/1569477791 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5b7cf82d0 0x55b5b7cfe7f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff60002c170 tx=0x7ff600000bc0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.570+0000 7ff60e79f740 1 -- 172.21.15.33:0/1569477791 shutdown_connections 2024-03-19T02:58:09.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.570+0000 7ff60e79f740 1 --2- 172.21.15.33:0/1569477791 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55b5b7d11960 0x55b5b7d13d50 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.570+0000 7ff60e79f740 1 --2- 172.21.15.33:0/1569477791 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5b7d017d0 0x55b5b7cff270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.570+0000 7ff60e79f740 1 --2- 172.21.15.33:0/1569477791 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5b7cf8bf0 0x55b5b7cfed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.570+0000 7ff5ed7fa640 0 client.26486 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T02:58:09.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.570+0000 7ff60e79f740 1 --2- 172.21.15.33:0/1569477791 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5b7cf82d0 0x55b5b7cfe7f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.570+0000 7ff60e79f740 1 -- 172.21.15.33:0/1569477791 >> 172.21.15.33:0/1569477791 conn(0x55b5b7cf2a00 msgr2=0x55b5b7cf4750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:09.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.571+0000 7ff60e79f740 1 -- 172.21.15.33:0/1569477791 shutdown_connections 2024-03-19T02:58:09.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.571+0000 7ff60e79f740 1 -- 172.21.15.33:0/1569477791 wait complete. 2024-03-19T02:58:09.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.571+0000 7ff60e79f740 20 client.26486 trim_cache size 0 max 16384 2024-03-19T02:58:09.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.584+0000 7ff60e79f740 1 Processor -- start 2024-03-19T02:58:09.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.584+0000 7ff60e79f740 1 -- start start 2024-03-19T02:58:09.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.584+0000 7ff60e79f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5b7cf82d0 0x55b5b7d0d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.584+0000 7ff60e79f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5b7cf8bf0 0x55b5b7cff090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.591 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.584+0000 7ff60e79f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5b7d017d0 0x55b5b7d19380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.584+0000 7ff60e79f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff6041af150 con 0x55b5b7cf82d0 2024-03-19T02:58:09.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.584+0000 7ff60e79f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff6041aca20 con 0x55b5b7cf8bf0 2024-03-19T02:58:09.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.584+0000 7ff60e79f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff6041aa850 con 0x55b5b7d017d0 2024-03-19T02:58:09.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.584+0000 7ff609878640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5b7cf82d0 0x55b5b7d0d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.584+0000 7ff609878640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5b7cf82d0 0x55b5b7d0d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49820/0 (socket says 172.21.15.33:49820) 2024-03-19T02:58:09.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.584+0000 7ff609878640 1 -- 172.21.15.33:0/1459618995 learned_addr learned my addr 172.21.15.33:0/1459618995 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:09.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.584+0000 7ff609077640 1 --2- 172.21.15.33:0/1459618995 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5b7cf8bf0 0x55b5b7cff090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.584+0000 7ff60a079640 1 --2- 172.21.15.33:0/1459618995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5b7d017d0 0x55b5b7d19380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.584+0000 7ff609878640 1 -- 172.21.15.33:0/1459618995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5b7d017d0 msgr2=0x55b5b7d19380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:09.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.584+0000 7ff609878640 1 --2- 172.21.15.33:0/1459618995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5b7d017d0 0x55b5b7d19380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.584+0000 7ff609878640 1 -- 172.21.15.33:0/1459618995 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5b7cf8bf0 msgr2=0x55b5b7cff090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:09.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.584+0000 7ff609878640 1 --2- 172.21.15.33:0/1459618995 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5b7cf8bf0 0x55b5b7cff090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.592 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.584+0000 7ff609878640 1 -- 172.21.15.33:0/1459618995 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5b7cf2830 con 0x55b5b7cf82d0 2024-03-19T02:58:09.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.584+0000 7ff60a079640 1 --2- 172.21.15.33:0/1459618995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5b7d017d0 0x55b5b7d19380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:58:09.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.585+0000 7ff609878640 1 --2- 172.21.15.33:0/1459618995 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5b7cf82d0 0x55b5b7d0d300 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7ff600009430 tx=0x7ff60002c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe1d341de6204177 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:09.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.585+0000 7ff5ef7fe640 1 -- 172.21.15.33:0/1459618995 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff600057030 con 0x55b5b7cf82d0 2024-03-19T02:58:09.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.585+0000 7ff5ef7fe640 1 -- 172.21.15.33:0/1459618995 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff60002c7e0 con 0x55b5b7cf82d0 2024-03-19T02:58:09.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.585+0000 7ff5ef7fe640 1 -- 172.21.15.33:0/1459618995 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff600045040 con 0x55b5b7cf82d0 2024-03-19T02:58:09.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.585+0000 7ff60e79f740 1 -- 172.21.15.33:0/1459618995 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5b7cf82d0 msgr2=0x55b5b7d0d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:09.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.585+0000 7ff60e79f740 1 --2- 172.21.15.33:0/1459618995 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5b7cf82d0 0x55b5b7d0d300 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7ff600009430 tx=0x7ff60002c330 comp rx=0 tx=0).stop 2024-03-19T02:58:09.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.585+0000 7ff60e79f740 1 -- 172.21.15.33:0/1459618995 shutdown_connections 2024-03-19T02:58:09.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.585+0000 7ff60e79f740 1 --2- 172.21.15.33:0/1459618995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5b7d017d0 0x55b5b7d19380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.585+0000 7ff60e79f740 1 --2- 172.21.15.33:0/1459618995 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5b7cf8bf0 0x55b5b7cff090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.585+0000 7ff60e79f740 1 --2- 172.21.15.33:0/1459618995 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5b7cf82d0 0x55b5b7d0d300 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.585+0000 7ff60e79f740 1 -- 172.21.15.33:0/1459618995 >> 172.21.15.33:0/1459618995 conn(0x55b5b7cf2a00 msgr2=0x55b5b7cf4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:09.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.585+0000 7ff60e79f740 1 -- 172.21.15.33:0/1459618995 shutdown_connections 2024-03-19T02:58:09.593 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.585+0000 7ff60e79f740 1 -- 172.21.15.33:0/1459618995 wait complete. 2024-03-19T02:58:09.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.586+0000 7ff60e79f740 1 -- start start 2024-03-19T02:58:09.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.586+0000 7ff60e79f740 1 Processor -- start 2024-03-19T02:58:09.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.586+0000 7ff60e79f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5b7cf82d0 0x55b5b7d09990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.586+0000 7ff60e79f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5b7cf8bf0 0x55b5b7d04d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.586+0000 7ff60e79f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5b7d017d0 0x55b5b7d05240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.586+0000 7ff60e79f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ff6041aaee0 con 0x55b5b7cf8bf0 2024-03-19T02:58:09.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.586+0000 7ff60e79f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff6041ac350 con 0x55b5b7d017d0 2024-03-19T02:58:09.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.586+0000 7ff60e79f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ff6041ab5b0 con 0x55b5b7cf82d0 2024-03-19T02:58:09.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.586+0000 7ff60e79f740 10 client.0 _ll_register_callbacks cb 0x7ff5fc009ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:58:09.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.586+0000 7ff609077640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5b7cf8bf0 0x55b5b7d04d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.586+0000 7ff609077640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5b7cf8bf0 0x55b5b7d04d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49832/0 (socket says 172.21.15.33:49832) 2024-03-19T02:58:09.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.586+0000 7ff609077640 1 -- 172.21.15.33:0/3050481373 learned_addr learned my addr 172.21.15.33:0/3050481373 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:09.594 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.586+0000 7ff60a079640 1 --2- 172.21.15.33:0/3050481373 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5b7d017d0 0x55b5b7d05240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.586+0000 7ff609878640 1 --2- 172.21.15.33:0/3050481373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5b7cf82d0 0x55b5b7d09990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.586+0000 7ff609077640 1 -- 172.21.15.33:0/3050481373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5b7cf82d0 msgr2=0x55b5b7d09990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:09.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.586+0000 7ff609077640 1 --2- 172.21.15.33:0/3050481373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5b7cf82d0 0x55b5b7d09990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.586+0000 7ff609077640 1 -- 172.21.15.33:0/3050481373 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5b7d017d0 msgr2=0x55b5b7d05240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:09.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.586+0000 7ff609077640 1 --2- 172.21.15.33:0/3050481373 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5b7d017d0 0x55b5b7d05240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.586+0000 7ff609077640 1 -- 172.21.15.33:0/3050481373 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5b7cf7cd0 con 0x55b5b7cf8bf0 2024-03-19T02:58:09.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.586+0000 7ff609878640 1 --2- 172.21.15.33:0/3050481373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5b7cf82d0 0x55b5b7d09990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:58:09.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.587+0000 7ff609077640 1 --2- 172.21.15.33:0/3050481373 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5b7cf8bf0 0x55b5b7d04d00 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4098240 tx=0x7ff5f4007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7ac00cc1c23688f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:09.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.587+0000 7ff5ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:58:09.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.587+0000 7ff5ed7fa640 1 -- 172.21.15.33:0/3050481373 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f40b3030 con 0x55b5b7cf8bf0 2024-03-19T02:58:09.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.587+0000 7ff5ed7fa640 1 -- 172.21.15.33:0/3050481373 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7ff5f4084e00 con 0x55b5b7cf8bf0 2024-03-19T02:58:09.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.587+0000 7ff60e79f740 1 -- 172.21.15.33:0/3050481373 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff60002c7e0 con 0x55b5b7cf8bf0 2024-03-19T02:58:09.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.587+0000 7ff5ed7fa640 1 -- 172.21.15.33:0/3050481373 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff5f40a4060 con 0x55b5b7cf8bf0 2024-03-19T02:58:09.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.587+0000 7ff60e79f740 10 client.16668 fetch_fsmap learned FSMap version 25 2024-03-19T02:58:09.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.587+0000 7ff60e79f740 1 -- 172.21.15.33:0/3050481373 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b5b7cc8880 con 0x55b5b7cf8bf0 2024-03-19T02:58:09.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.587+0000 7ff5ed7fa640 1 -- 172.21.15.33:0/3050481373 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff60002c7e0 con 0x55b5b7cf8bf0 2024-03-19T02:58:09.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.587+0000 7ff5ed7fa640 1 -- 172.21.15.33:0/3050481373 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7ff5f4005380 con 0x55b5b7cf8bf0 2024-03-19T02:58:09.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.587+0000 7ff60e79f740 10 client.16668 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:58:09.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.587+0000 7ff60e79f740 10 client.16668 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T02:58:09.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.587+0000 7ff60e79f740 20 client.16668 populate_metadata read hostname 'smithi033' 2024-03-19T02:58:09.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.587+0000 7ff60e79f740 1 --2- 172.21.15.33:0/3050481373 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55b5b7d77b70 0x55b5b7d79f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.587+0000 7ff60e79f740 4 client.16668 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:58:09.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.587+0000 7ff60e79f740 1 -- 172.21.15.33:0/3050481373 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b5b7c689f0 con 0x55b5b7d77b70 2024-03-19T02:58:09.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.587+0000 7ff609878640 1 --2- 172.21.15.33:0/3050481373 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55b5b7d77b70 0x55b5b7d79f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.588+0000 7ff5ed7fa640 10 client.16668 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T02:58:09.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.588+0000 7ff609878640 1 --2- 172.21.15.33:0/3050481373 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55b5b7d77b70 0x55b5b7d79f90 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1b2c18196b146165 server_cookie=7ad0c4b65504067a in_seq=0 out_seq=0 2024-03-19T02:58:09.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.588+0000 7ff5ed7fa640 1 -- 172.21.15.33:0/3050481373 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b5b7c689f0 con 0x55b5b7d77b70 2024-03-19T02:58:09.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.588+0000 7ff5ed7fa640 10 client.16668 handle_command_reply: tid=0 2024-03-19T02:58:09.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.588+0000 7ff60e79f740 1 client.16668 shutdown 2024-03-19T02:58:09.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.588+0000 7ff60e79f740 1 -- 172.21.15.33:0/3050481373 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5b7cf8bf0 msgr2=0x55b5b7d04d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:09.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.588+0000 7ff60e79f740 1 --2- 172.21.15.33:0/3050481373 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5b7cf8bf0 0x55b5b7d04d00 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4098240 tx=0x7ff5f4007620 comp rx=0 tx=0).stop 2024-03-19T02:58:09.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.588+0000 7ff60e79f740 1 -- 172.21.15.33:0/3050481373 shutdown_connections 2024-03-19T02:58:09.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.588+0000 7ff60e79f740 1 --2- 172.21.15.33:0/3050481373 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55b5b7d77b70 0x55b5b7d79f90 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.588+0000 7ff60e79f740 1 --2- 172.21.15.33:0/3050481373 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b5b7d017d0 0x55b5b7d05240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.588+0000 7ff60e79f740 1 --2- 172.21.15.33:0/3050481373 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b5b7cf8bf0 0x55b5b7d04d00 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.588+0000 7ff60e79f740 1 --2- 172.21.15.33:0/3050481373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b5b7cf82d0 0x55b5b7d09990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.588+0000 7ff5ed7fa640 0 client.16668 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T02:58:09.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.588+0000 7ff60e79f740 1 -- 172.21.15.33:0/3050481373 >> 172.21.15.33:0/3050481373 conn(0x55b5b7cf2a00 msgr2=0x55b5b7cf1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:09.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.588+0000 7ff60e79f740 1 -- 172.21.15.33:0/3050481373 shutdown_connections 2024-03-19T02:58:09.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.588+0000 7ff60e79f740 1 -- 172.21.15.33:0/3050481373 wait complete. 2024-03-19T02:58:09.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.588+0000 7ff60e79f740 20 client.16668 trim_cache size 0 max 16384 2024-03-19T02:58:09.598 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:58:09.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.589+0000 7ff60badc640 1 -- 172.21.15.33:0/3127482066 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7ff5f00a2970 msgr2=0x7ff5f00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:09.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.589+0000 7ff60badc640 1 --2- 172.21.15.33:0/3127482066 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7ff5f00a2970 0x7ff5f00a4e30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc005fd0 tx=0x7ff5fc009500 comp rx=0 tx=0).stop 2024-03-19T02:58:09.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.589+0000 7ff60badc640 1 -- 172.21.15.33:0/3127482066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff604101330 msgr2=0x7ff604198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:09.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.589+0000 7ff60badc640 1 --2- 172.21.15.33:0/3127482066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff604101330 0x7ff604198530 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7ff5f400e9f0 tx=0x7ff5f400edf0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.589+0000 7ff60badc640 1 -- 172.21.15.33:0/3127482066 shutdown_connections 2024-03-19T02:58:09.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.589+0000 7ff60badc640 1 --2- 172.21.15.33:0/3127482066 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7ff5f00a2970 0x7ff5f00a4e30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.589+0000 7ff60badc640 1 --2- 172.21.15.33:0/3127482066 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff604105660 0x7ff60419c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.589+0000 7ff60badc640 1 --2- 172.21.15.33:0/3127482066 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ff604101330 0x7ff604198530 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.589+0000 7ff60badc640 1 --2- 172.21.15.33:0/3127482066 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff604100970 0x7ff604197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.589+0000 7ff60badc640 1 -- 172.21.15.33:0/3127482066 >> 172.21.15.33:0/3127482066 conn(0x7ff6040774c0 msgr2=0x7ff60406a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:09.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.589+0000 7ff60badc640 1 -- 172.21.15.33:0/3127482066 shutdown_connections 2024-03-19T02:58:09.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.589+0000 7ff60badc640 1 -- 172.21.15.33:0/3127482066 wait complete. 2024-03-19T02:58:09.600 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:09 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2859320389' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:58:09.601 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:58:09.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.705+0000 7f649f73b640 1 -- 172.21.15.33:0/516702206 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64981010a0 msgr2=0x7f6498101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:09.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.705+0000 7f649f73b640 1 --2- 172.21.15.33:0/516702206 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64981010a0 0x7f6498101480 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6494009860 tx=0x7f649402bbd0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.705+0000 7f649f73b640 1 -- 172.21.15.33:0/516702206 shutdown_connections 2024-03-19T02:58:09.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.705+0000 7f649f73b640 1 --2- 172.21.15.33:0/516702206 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f64981059f0 0x7f649810d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.705+0000 7f649f73b640 1 --2- 172.21.15.33:0/516702206 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6498101a50 0x7f64981052c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.705+0000 7f649f73b640 1 --2- 172.21.15.33:0/516702206 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64981010a0 0x7f6498101480 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.705+0000 7f649f73b640 1 -- 172.21.15.33:0/516702206 >> 172.21.15.33:0/516702206 conn(0x7f6498076c50 msgr2=0x7f64980fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:09.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.705+0000 7f649f73b640 1 -- 172.21.15.33:0/516702206 shutdown_connections 2024-03-19T02:58:09.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.705+0000 7f649f73b640 1 -- 172.21.15.33:0/516702206 wait complete. 2024-03-19T02:58:09.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.706+0000 7f649f73b640 1 Processor -- start 2024-03-19T02:58:09.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.706+0000 7f649f73b640 1 -- start start 2024-03-19T02:58:09.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.706+0000 7f649f73b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f64981010a0 0x7f6498198e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.706+0000 7f649f73b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64981059f0 0x7f6498199380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.706+0000 7f649f73b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64981998c0 0x7f64981a3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.706+0000 7f649f73b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6498110960 con 0x7f64981010a0 2024-03-19T02:58:09.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.706+0000 7f649f73b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f64981107e0 con 0x7f64981059f0 2024-03-19T02:58:09.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.706+0000 7f649f73b640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6498110ae0 con 0x7f64981998c0 2024-03-19T02:58:09.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.706+0000 7f649d4d7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f64981010a0 0x7f6498198e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.706+0000 7f649d4d7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f64981010a0 0x7f6498198e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49854/0 (socket says 172.21.15.33:49854) 2024-03-19T02:58:09.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.706+0000 7f649d4d7640 1 -- 172.21.15.33:0/692834783 learned_addr learned my addr 172.21.15.33:0/692834783 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:09.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.706+0000 7f649ccd6640 1 --2- 172.21.15.33:0/692834783 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64981059f0 0x7f6498199380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.706+0000 7f649dcd8640 1 --2- 172.21.15.33:0/692834783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64981998c0 0x7f64981a3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.706+0000 7f649d4d7640 1 -- 172.21.15.33:0/692834783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64981998c0 msgr2=0x7f64981a3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:09.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.707+0000 7f649d4d7640 1 --2- 172.21.15.33:0/692834783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64981998c0 0x7f64981a3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.707+0000 7f649d4d7640 1 -- 172.21.15.33:0/692834783 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64981059f0 msgr2=0x7f6498199380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:09.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.707+0000 7f649d4d7640 1 --2- 172.21.15.33:0/692834783 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64981059f0 0x7f6498199380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.707+0000 7f649d4d7640 1 -- 172.21.15.33:0/692834783 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64981a4640 con 0x7f64981010a0 2024-03-19T02:58:09.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.707+0000 7f649dcd8640 1 --2- 172.21.15.33:0/692834783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64981998c0 0x7f64981a3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:58:09.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.707+0000 7f649ccd6640 1 --2- 172.21.15.33:0/692834783 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64981059f0 0x7f6498199380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:58:09.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.707+0000 7f649d4d7640 1 --2- 172.21.15.33:0/692834783 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f64981010a0 0x7f6498198e40 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f6494037040 tx=0x7f6494004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:09.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.707+0000 7f648e7fc640 1 -- 172.21.15.33:0/692834783 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6494041020 con 0x7f64981010a0 2024-03-19T02:58:09.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.707+0000 7f649f73b640 1 -- 172.21.15.33:0/692834783 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64981a48d0 con 0x7f64981010a0 2024-03-19T02:58:09.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.707+0000 7f648e7fc640 1 -- 172.21.15.33:0/692834783 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f649402c9c0 con 0x7f64981010a0 2024-03-19T02:58:09.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.707+0000 7f648e7fc640 1 -- 172.21.15.33:0/692834783 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6494039040 con 0x7f64981010a0 2024-03-19T02:58:09.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.707+0000 7f649f73b640 1 -- 172.21.15.33:0/692834783 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64981a4e10 con 0x7f64981010a0 2024-03-19T02:58:09.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.709+0000 7f648e7fc640 1 -- 172.21.15.33:0/692834783 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f6494044020 con 0x7f64981010a0 2024-03-19T02:58:09.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.710+0000 7f64a2408740 1 Processor -- start 2024-03-19T02:58:09.719 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.710+0000 7f64a2408740 1 -- start start 2024-03-19T02:58:09.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.710+0000 7f64a2408740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559edbc83cb0 0x559edbc84090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.710+0000 7f64a2408740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559edbc84660 0x559edbc916a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.710+0000 7f64a2408740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559edbc91be0 0x559edbc93fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.710+0000 7f64a2408740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559edbbe74b0 con 0x559edbc91be0 2024-03-19T02:58:09.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.710+0000 7f64a2408740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x559edbb8fd70 con 0x559edbc83cb0 2024-03-19T02:58:09.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.710+0000 7f64a2408740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x559edbbe3550 con 0x559edbc84660 2024-03-19T02:58:09.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.710+0000 7f648e7fc640 1 --2- 172.21.15.33:0/692834783 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f64840a2970 0x7f64840a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.710+0000 7f649ccd6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559edbc91be0 0x559edbc93fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.710+0000 7f649ccd6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559edbc91be0 0x559edbc93fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49862/0 (socket says 172.21.15.33:49862) 2024-03-19T02:58:09.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.710+0000 7f649ccd6640 1 -- 172.21.15.33:0/347547808 learned_addr learned my addr 172.21.15.33:0/347547808 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:09.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.710+0000 7f649dcd8640 1 --2- 172.21.15.33:0/347547808 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559edbc83cb0 0x559edbc84090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.710+0000 7f649d4d7640 1 --2- 172.21.15.33:0/347547808 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559edbc84660 0x559edbc916a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.710+0000 7f649dcd8640 1 --2- 172.21.15.33:0/692834783 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f64840a2970 0x7f64840a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.720 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.710+0000 7f648e7fc640 1 -- 172.21.15.33:0/692834783 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f64940f0970 con 0x7f64981010a0 2024-03-19T02:58:09.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.710+0000 7f649dcd8640 1 -- 172.21.15.33:0/347547808 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559edbc84660 msgr2=0x559edbc916a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:09.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.710+0000 7f649dcd8640 1 --2- 172.21.15.33:0/347547808 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559edbc84660 0x559edbc916a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.710+0000 7f649dcd8640 1 -- 172.21.15.33:0/347547808 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559edbc91be0 msgr2=0x559edbc93fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:09.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.710+0000 7f649dcd8640 1 --2- 172.21.15.33:0/347547808 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559edbc91be0 0x559edbc93fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.710+0000 7f649dcd8640 1 -- 172.21.15.33:0/347547808 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559edbbb2210 con 0x559edbc83cb0 2024-03-19T02:58:09.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.711+0000 7f649ccd6640 1 --2- 172.21.15.33:0/347547808 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559edbc91be0 0x559edbc93fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:58:09.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.711+0000 7f649dcd8640 1 --2- 172.21.15.33:0/692834783 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f64840a2970 0x7f64840a4e30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f649000dcb0 tx=0x7f6490008000 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:09.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.711+0000 7f649dcd8640 1 --2- 172.21.15.33:0/347547808 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559edbc83cb0 0x559edbc84090 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6490010d00 tx=0x7f649000d7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8809f23c69d75213 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:09.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.711+0000 7f64837fe640 1 -- 172.21.15.33:0/347547808 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6490021070 con 0x559edbc83cb0 2024-03-19T02:58:09.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.711+0000 7f64837fe640 1 -- 172.21.15.33:0/347547808 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f64900100e0 con 0x559edbc83cb0 2024-03-19T02:58:09.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.711+0000 7f64837fe640 1 -- 172.21.15.33:0/347547808 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6490003b70 con 0x559edbc83cb0 2024-03-19T02:58:09.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.712+0000 7f64a2408740 1 -- 172.21.15.33:0/347547808 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559edbc83cb0 msgr2=0x559edbc84090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:09.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.712+0000 7f64a2408740 1 --2- 172.21.15.33:0/347547808 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559edbc83cb0 0x559edbc84090 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6490010d00 tx=0x7f649000d7b0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.712+0000 7f64a2408740 1 -- 172.21.15.33:0/347547808 shutdown_connections 2024-03-19T02:58:09.721 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.712+0000 7f64a2408740 1 --2- 172.21.15.33:0/347547808 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559edbc84660 0x559edbc916a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.712+0000 7f64a2408740 1 --2- 172.21.15.33:0/347547808 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559edbc91be0 0x559edbc93fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.712+0000 7f64a2408740 1 --2- 172.21.15.33:0/347547808 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559edbc83cb0 0x559edbc84090 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.712+0000 7f64a2408740 1 -- 172.21.15.33:0/347547808 >> 172.21.15.33:0/347547808 conn(0x559edbc7ea00 msgr2=0x559edbc7ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:09.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.712+0000 7f64a2408740 1 -- 172.21.15.33:0/347547808 shutdown_connections 2024-03-19T02:58:09.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.712+0000 7f64a2408740 1 -- 172.21.15.33:0/347547808 wait complete. 2024-03-19T02:58:09.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.712+0000 7f64a2408740 1 -- start start 2024-03-19T02:58:09.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.712+0000 7f64a2408740 1 Processor -- start 2024-03-19T02:58:09.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.712+0000 7f64a2408740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559edbc83cb0 0x559edbc8ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.712+0000 7f64a2408740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559edbc84660 0x559edbc8f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.712+0000 7f64a2408740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559edbc91be0 0x559edbc8f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.712+0000 7f64a2408740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559edbc94940 con 0x559edbc84660 2024-03-19T02:58:09.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.712+0000 7f64a2408740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x559edbba3310 con 0x559edbc83cb0 2024-03-19T02:58:09.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.712+0000 7f64a2408740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x559edbc94ac0 con 0x559edbc91be0 2024-03-19T02:58:09.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.713+0000 7f64a2408740 10 client.0 _ll_register_callbacks cb 0x559edbb972d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:58:09.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.713+0000 7f649d4d7640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559edbc83cb0 0x559edbc8ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.713+0000 7f649ccd6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559edbc84660 0x559edbc8f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.713+0000 7f649d4d7640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559edbc83cb0 0x559edbc8ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:51376/0 (socket says 172.21.15.33:51376) 2024-03-19T02:58:09.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.713+0000 7f649d4d7640 1 -- 172.21.15.33:0/1425909691 learned_addr learned my addr 172.21.15.33:0/1425909691 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:09.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.713+0000 7f649dcd8640 1 --2- 172.21.15.33:0/1425909691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559edbc91be0 0x559edbc8f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.713+0000 7f649d4d7640 1 -- 172.21.15.33:0/1425909691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559edbc91be0 msgr2=0x559edbc8f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:09.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.713+0000 7f649d4d7640 1 --2- 172.21.15.33:0/1425909691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559edbc91be0 0x559edbc8f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.713+0000 7f649d4d7640 1 -- 172.21.15.33:0/1425909691 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559edbc84660 msgr2=0x559edbc8f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:09.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.713+0000 7f649d4d7640 1 --2- 172.21.15.33:0/1425909691 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559edbc84660 0x559edbc8f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.713+0000 7f649d4d7640 1 -- 172.21.15.33:0/1425909691 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559edb8fb3f0 con 0x559edbc83cb0 2024-03-19T02:58:09.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.713+0000 7f649ccd6640 1 --2- 172.21.15.33:0/1425909691 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559edbc84660 0x559edbc8f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:58:09.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.713+0000 7f649d4d7640 1 --2- 172.21.15.33:0/1425909691 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559edbc83cb0 0x559edbc8ec10 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f649402cb90 tx=0x7f649408bb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34b998e9b6e6ab06 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:09.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.713+0000 7f6480ff9640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T02:58:09.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.713+0000 7f6480ff9640 1 -- 172.21.15.33:0/1425909691 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64940bcd70 con 0x559edbc83cb0 2024-03-19T02:58:09.723 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.713+0000 7f64a2408740 1 -- 172.21.15.33:0/1425909691 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64900100e0 con 0x559edbc83cb0 2024-03-19T02:58:09.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.713+0000 7f6480ff9640 1 -- 172.21.15.33:0/1425909691 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f64940edfd0 con 0x559edbc83cb0 2024-03-19T02:58:09.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.713+0000 7f6480ff9640 1 -- 172.21.15.33:0/1425909691 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f64940ee1d0 con 0x559edbc83cb0 2024-03-19T02:58:09.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.713+0000 7f64a2408740 10 client.26504 fetch_fsmap learned FSMap version 25 2024-03-19T02:58:09.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.713+0000 7f64a2408740 1 -- 172.21.15.33:0/1425909691 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559edb9c57d0 con 0x559edbc83cb0 2024-03-19T02:58:09.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.713+0000 7f6480ff9640 1 -- 172.21.15.33:0/1425909691 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f64900100e0 con 0x559edbc83cb0 2024-03-19T02:58:09.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.714+0000 7f6480ff9640 1 -- 172.21.15.33:0/1425909691 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f649403c030 con 0x559edbc83cb0 2024-03-19T02:58:09.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.714+0000 7f64a2408740 10 client.26504 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:58:09.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.714+0000 7f64a2408740 10 client.26504 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:58:09.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.714+0000 7f64a2408740 20 client.26504 populate_metadata read hostname 'smithi033' 2024-03-19T02:58:09.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.714+0000 7f64a2408740 1 --2- 172.21.15.33:0/1425909691 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x559edbca1f00 0x559edbca4380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.714+0000 7f64a2408740 4 client.26504 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:58:09.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.714+0000 7f64a2408740 1 -- 172.21.15.33:0/1425909691 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559edbc453e0 con 0x559edbca1f00 2024-03-19T02:58:09.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.714+0000 7f649ccd6640 1 --2- 172.21.15.33:0/1425909691 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x559edbca1f00 0x559edbca4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.714+0000 7f6480ff9640 10 client.26504 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:58:09.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.714+0000 7f649ccd6640 1 --2- 172.21.15.33:0/1425909691 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x559edbca1f00 0x559edbca4380 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a50b596f040e5d92 server_cookie=4357fdf97e69b13 in_seq=0 out_seq=0 2024-03-19T02:58:09.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.715+0000 7f6480ff9640 1 -- 172.21.15.33:0/1425909691 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559edbc453e0 con 0x559edbca1f00 2024-03-19T02:58:09.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.715+0000 7f6480ff9640 10 client.26504 handle_command_reply: tid=0 2024-03-19T02:58:09.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.715+0000 7f64a2408740 1 client.26504 shutdown 2024-03-19T02:58:09.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.715+0000 7f64a2408740 1 -- 172.21.15.33:0/1425909691 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559edbc83cb0 msgr2=0x559edbc8ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:09.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.715+0000 7f64a2408740 1 --2- 172.21.15.33:0/1425909691 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559edbc83cb0 0x559edbc8ec10 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f649402cb90 tx=0x7f649408bb80 comp rx=0 tx=0).stop 2024-03-19T02:58:09.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.715+0000 7f64a2408740 1 -- 172.21.15.33:0/1425909691 shutdown_connections 2024-03-19T02:58:09.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.715+0000 7f64a2408740 1 --2- 172.21.15.33:0/1425909691 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x559edbca1f00 0x559edbca4380 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.715+0000 7f64a2408740 1 --2- 172.21.15.33:0/1425909691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559edbc91be0 0x559edbc8f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.715+0000 7f64a2408740 1 --2- 172.21.15.33:0/1425909691 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559edbc84660 0x559edbc8f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.715+0000 7f6480ff9640 0 client.26504 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:58:09.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.715+0000 7f64a2408740 1 --2- 172.21.15.33:0/1425909691 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559edbc83cb0 0x559edbc8ec10 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.715+0000 7f64a2408740 1 -- 172.21.15.33:0/1425909691 >> 172.21.15.33:0/1425909691 conn(0x559edbc7ea00 msgr2=0x559edbc80750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:09.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.715+0000 7f64a2408740 1 -- 172.21.15.33:0/1425909691 shutdown_connections 2024-03-19T02:58:09.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.715+0000 7f64a2408740 1 -- 172.21.15.33:0/1425909691 wait complete. 2024-03-19T02:58:09.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.715+0000 7f64a2408740 20 client.26504 trim_cache size 0 max 16384 2024-03-19T02:58:09.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.728+0000 7f64a2408740 1 Processor -- start 2024-03-19T02:58:09.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.728+0000 7f64a2408740 1 -- start start 2024-03-19T02:58:09.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.729+0000 7f64a2408740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559edbc83cb0 0x559edbc9d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.729+0000 7f64a2408740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559edbc84660 0x559edbc79a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.729+0000 7f64a2408740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559edbc91be0 0x559edbca9c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.729+0000 7f64a2408740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f64981af1c0 con 0x559edbc84660 2024-03-19T02:58:09.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.729+0000 7f64a2408740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f64981aca90 con 0x559edbc91be0 2024-03-19T02:58:09.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.729+0000 7f64a2408740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f64981aa8c0 con 0x559edbc83cb0 2024-03-19T02:58:09.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.729+0000 7f649ccd6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559edbc84660 0x559edbc79a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.729+0000 7f649ccd6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559edbc84660 0x559edbc79a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49868/0 (socket says 172.21.15.33:49868) 2024-03-19T02:58:09.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.729+0000 7f649ccd6640 1 -- 172.21.15.33:0/332850169 learned_addr learned my addr 172.21.15.33:0/332850169 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:09.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.729+0000 7f649dcd8640 1 --2- 172.21.15.33:0/332850169 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559edbc91be0 0x559edbca9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.729+0000 7f649d4d7640 1 --2- 172.21.15.33:0/332850169 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559edbc83cb0 0x559edbc9d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.729+0000 7f649ccd6640 1 -- 172.21.15.33:0/332850169 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559edbc83cb0 msgr2=0x559edbc9d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:09.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.729+0000 7f649ccd6640 1 --2- 172.21.15.33:0/332850169 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559edbc83cb0 0x559edbc9d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.729+0000 7f649ccd6640 1 -- 172.21.15.33:0/332850169 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559edbc91be0 msgr2=0x559edbca9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:09.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.729+0000 7f649ccd6640 1 --2- 172.21.15.33:0/332850169 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559edbc91be0 0x559edbca9c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.729+0000 7f649ccd6640 1 -- 172.21.15.33:0/332850169 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559edbc8dbc0 con 0x559edbc84660 2024-03-19T02:58:09.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.729+0000 7f649dcd8640 1 --2- 172.21.15.33:0/332850169 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559edbc91be0 0x559edbca9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:58:09.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.729+0000 7f649ccd6640 1 --2- 172.21.15.33:0/332850169 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559edbc84660 0x559edbc79a30 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x559edbc901e0 tx=0x7f6488005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ebfbcd6210ecb1c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:09.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.729+0000 7f6482ffd640 1 -- 172.21.15.33:0/332850169 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6488028070 con 0x559edbc84660 2024-03-19T02:58:09.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.729+0000 7f6482ffd640 1 -- 172.21.15.33:0/332850169 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6488022620 con 0x559edbc84660 2024-03-19T02:58:09.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.729+0000 7f6482ffd640 1 -- 172.21.15.33:0/332850169 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6488010800 con 0x559edbc84660 2024-03-19T02:58:09.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.730+0000 7f64a2408740 1 -- 172.21.15.33:0/332850169 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559edbc84660 msgr2=0x559edbc79a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:09.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.730+0000 7f64a2408740 1 --2- 172.21.15.33:0/332850169 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559edbc84660 0x559edbc79a30 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x559edbc901e0 tx=0x7f6488005f90 comp rx=0 tx=0).stop 2024-03-19T02:58:09.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.730+0000 7f64a2408740 1 -- 172.21.15.33:0/332850169 shutdown_connections 2024-03-19T02:58:09.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.730+0000 7f64a2408740 1 --2- 172.21.15.33:0/332850169 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559edbc91be0 0x559edbca9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.730+0000 7f64a2408740 1 --2- 172.21.15.33:0/332850169 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559edbc84660 0x559edbc79a30 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.730+0000 7f64a2408740 1 --2- 172.21.15.33:0/332850169 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559edbc83cb0 0x559edbc9d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.730+0000 7f64a2408740 1 -- 172.21.15.33:0/332850169 >> 172.21.15.33:0/332850169 conn(0x559edbc7ea00 msgr2=0x559edbc90cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:09.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.730+0000 7f64a2408740 1 -- 172.21.15.33:0/332850169 shutdown_connections 2024-03-19T02:58:09.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.730+0000 7f64a2408740 1 -- 172.21.15.33:0/332850169 wait complete. 2024-03-19T02:58:09.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.730+0000 7f64a2408740 1 -- start start 2024-03-19T02:58:09.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.730+0000 7f64a2408740 1 Processor -- start 2024-03-19T02:58:09.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.731+0000 7f64a2408740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559edbc83cb0 0x559edbc95a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.731+0000 7f64a2408740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559edbc84660 0x559edbc95fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.731+0000 7f64a2408740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559edbc91be0 0x559edbd07280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.731+0000 7f64a2408740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f64981aaf50 con 0x559edbc83cb0 2024-03-19T02:58:09.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.731+0000 7f64a2408740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f64981ac3c0 con 0x559edbc84660 2024-03-19T02:58:09.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.731+0000 7f64a2408740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f64981ab620 con 0x559edbc91be0 2024-03-19T02:58:09.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.731+0000 7f64a2408740 10 client.0 _ll_register_callbacks cb 0x7f64940bcbd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:58:09.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.731+0000 7f649d4d7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559edbc83cb0 0x559edbc95a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.731+0000 7f649d4d7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559edbc83cb0 0x559edbc95a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49870/0 (socket says 172.21.15.33:49870) 2024-03-19T02:58:09.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.731+0000 7f649d4d7640 1 -- 172.21.15.33:0/3080159292 learned_addr learned my addr 172.21.15.33:0/3080159292 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:09.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.731+0000 7f649ccd6640 1 --2- 172.21.15.33:0/3080159292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559edbc84660 0x559edbc95fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.731+0000 7f649dcd8640 1 --2- 172.21.15.33:0/3080159292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559edbc91be0 0x559edbd07280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.731+0000 7f649d4d7640 1 -- 172.21.15.33:0/3080159292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559edbc91be0 msgr2=0x559edbd07280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:09.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.731+0000 7f649d4d7640 1 --2- 172.21.15.33:0/3080159292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559edbc91be0 0x559edbd07280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.731+0000 7f649d4d7640 1 -- 172.21.15.33:0/3080159292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559edbc84660 msgr2=0x559edbc95fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:09.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.731+0000 7f649d4d7640 1 --2- 172.21.15.33:0/3080159292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559edbc84660 0x559edbc95fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.731+0000 7f649d4d7640 1 -- 172.21.15.33:0/3080159292 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559edbd077c0 con 0x559edbc83cb0 2024-03-19T02:58:09.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.731+0000 7f649d4d7640 1 --2- 172.21.15.33:0/3080159292 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559edbc83cb0 0x559edbc95a60 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f649402c1e0 tx=0x7f6494005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6880709deeff5e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:09.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.731+0000 7f6480ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:58:09.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.731+0000 7f6480ff9640 1 -- 172.21.15.33:0/3080159292 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64940bc8b0 con 0x559edbc83cb0 2024-03-19T02:58:09.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.731+0000 7f6480ff9640 1 -- 172.21.15.33:0/3080159292 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f64940ee150 con 0x559edbc83cb0 2024-03-19T02:58:09.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.731+0000 7f64a2408740 1 -- 172.21.15.33:0/3080159292 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6488022620 con 0x559edbc83cb0 2024-03-19T02:58:09.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.732+0000 7f6480ff9640 1 -- 172.21.15.33:0/3080159292 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f64940b9010 con 0x559edbc83cb0 2024-03-19T02:58:09.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.732+0000 7f64a2408740 10 client.16686 fetch_fsmap learned FSMap version 25 2024-03-19T02:58:09.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.732+0000 7f64a2408740 1 -- 172.21.15.33:0/3080159292 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559edbd077c0 con 0x559edbc83cb0 2024-03-19T02:58:09.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.732+0000 7f6480ff9640 1 -- 172.21.15.33:0/3080159292 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6488022620 con 0x559edbc83cb0 2024-03-19T02:58:09.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.732+0000 7f6480ff9640 1 -- 172.21.15.33:0/3080159292 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f64940bbc30 con 0x559edbc83cb0 2024-03-19T02:58:09.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.732+0000 7f64a2408740 10 client.16686 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:58:09.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.732+0000 7f64a2408740 10 client.16686 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:58:09.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.732+0000 7f64a2408740 20 client.16686 populate_metadata read hostname 'smithi033' 2024-03-19T02:58:09.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.732+0000 7f64a2408740 1 --2- 172.21.15.33:0/3080159292 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x559edbd08cc0 0x559edbd0b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:09.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.732+0000 7f64a2408740 4 client.16686 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:58:09.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.732+0000 7f64a2408740 1 -- 172.21.15.33:0/3080159292 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559edbbf49f0 con 0x559edbd08cc0 2024-03-19T02:58:09.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.732+0000 7f649ccd6640 1 --2- 172.21.15.33:0/3080159292 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x559edbd08cc0 0x559edbd0b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:09.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.733+0000 7f6480ff9640 10 client.16686 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:58:09.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.733+0000 7f649ccd6640 1 --2- 172.21.15.33:0/3080159292 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x559edbd08cc0 0x559edbd0b0b0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7e11024f254f736 server_cookie=9db21e2dbaf0b099 in_seq=0 out_seq=0 2024-03-19T02:58:09.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.733+0000 7f6480ff9640 1 -- 172.21.15.33:0/3080159292 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559edbbf49f0 con 0x559edbd08cc0 2024-03-19T02:58:09.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.733+0000 7f6480ff9640 10 client.16686 handle_command_reply: tid=0 2024-03-19T02:58:09.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.733+0000 7f64a2408740 1 client.16686 shutdown 2024-03-19T02:58:09.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.733+0000 7f64a2408740 1 -- 172.21.15.33:0/3080159292 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559edbc83cb0 msgr2=0x559edbc95a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:09.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.733+0000 7f64a2408740 1 --2- 172.21.15.33:0/3080159292 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559edbc83cb0 0x559edbc95a60 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f649402c1e0 tx=0x7f6494005bd0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.733+0000 7f64a2408740 1 -- 172.21.15.33:0/3080159292 shutdown_connections 2024-03-19T02:58:09.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.733+0000 7f64a2408740 1 --2- 172.21.15.33:0/3080159292 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x559edbd08cc0 0x559edbd0b0b0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.733+0000 7f64a2408740 1 --2- 172.21.15.33:0/3080159292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559edbc91be0 0x559edbd07280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.733+0000 7f64a2408740 1 --2- 172.21.15.33:0/3080159292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559edbc84660 0x559edbc95fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.733+0000 7f6480ff9640 0 client.16686 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:58:09.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.733+0000 7f64a2408740 1 --2- 172.21.15.33:0/3080159292 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559edbc83cb0 0x559edbc95a60 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.733+0000 7f64a2408740 1 -- 172.21.15.33:0/3080159292 >> 172.21.15.33:0/3080159292 conn(0x559edbc7ea00 msgr2=0x559edbc7d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:09.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.733+0000 7f64a2408740 1 -- 172.21.15.33:0/3080159292 shutdown_connections 2024-03-19T02:58:09.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.733+0000 7f64a2408740 1 -- 172.21.15.33:0/3080159292 wait complete. 2024-03-19T02:58:09.742 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.733+0000 7f64a2408740 20 client.16686 trim_cache size 0 max 16384 2024-03-19T02:58:09.743 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:58:09.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.733+0000 7f649f73b640 1 -- 172.21.15.33:0/692834783 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f64840a2970 msgr2=0x7f64840a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:09.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.733+0000 7f649f73b640 1 --2- 172.21.15.33:0/692834783 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f64840a2970 0x7f64840a4e30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f649000dcb0 tx=0x7f6490008000 comp rx=0 tx=0).stop 2024-03-19T02:58:09.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.733+0000 7f649f73b640 1 -- 172.21.15.33:0/692834783 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f64981010a0 msgr2=0x7f6498198e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:09.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.733+0000 7f649f73b640 1 --2- 172.21.15.33:0/692834783 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f64981010a0 0x7f6498198e40 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f6494037040 tx=0x7f6494004040 comp rx=0 tx=0).stop 2024-03-19T02:58:09.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.734+0000 7f649f73b640 1 -- 172.21.15.33:0/692834783 shutdown_connections 2024-03-19T02:58:09.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.734+0000 7f649f73b640 1 --2- 172.21.15.33:0/692834783 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f64840a2970 0x7f64840a4e30 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.734+0000 7f649f73b640 1 --2- 172.21.15.33:0/692834783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64981998c0 0x7f64981a3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.734+0000 7f649f73b640 1 --2- 172.21.15.33:0/692834783 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64981059f0 0x7f6498199380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.734+0000 7f649f73b640 1 --2- 172.21.15.33:0/692834783 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f64981010a0 0x7f6498198e40 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:09.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.734+0000 7f649f73b640 1 -- 172.21.15.33:0/692834783 >> 172.21.15.33:0/692834783 conn(0x7f6498076c50 msgr2=0x7f64980fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:09.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.734+0000 7f649f73b640 1 -- 172.21.15.33:0/692834783 shutdown_connections 2024-03-19T02:58:09.744 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:09.734+0000 7f649f73b640 1 -- 172.21.15.33:0/692834783 wait complete. 2024-03-19T02:58:09.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:09 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2859320389' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:58:09.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:09 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2859320389' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:58:10.746 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:58:10.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:58:10.747 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:58:10.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:10 smithi155 ceph-mon[35804]: pgmap v144: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-19T02:58:10.775 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:10 smithi033 ceph-mon[31973]: pgmap v144: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-19T02:58:10.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.865+0000 7f139a07e640 1 -- 172.21.15.33:0/1250102213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13941017c0 msgr2=0x7f1394101c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:10.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.865+0000 7f139a07e640 1 --2- 172.21.15.33:0/1250102213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13941017c0 0x7f1394101c40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1388009860 tx=0x7f138802bb70 comp rx=0 tx=0).stop 2024-03-19T02:58:10.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.865+0000 7f139a07e640 1 -- 172.21.15.33:0/1250102213 shutdown_connections 2024-03-19T02:58:10.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.865+0000 7f139a07e640 1 --2- 172.21.15.33:0/1250102213 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1394102180 0x7f139410e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:10.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.865+0000 7f139a07e640 1 --2- 172.21.15.33:0/1250102213 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13941017c0 0x7f1394101c40 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:10.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.865+0000 7f139a07e640 1 --2- 172.21.15.33:0/1250102213 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f13941077d0 0x7f1394107bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:10.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.865+0000 7f139a07e640 1 -- 172.21.15.33:0/1250102213 >> 172.21.15.33:0/1250102213 conn(0x7f13940fd250 msgr2=0x7f13940ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:10.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.865+0000 7f139a07e640 1 -- 172.21.15.33:0/1250102213 shutdown_connections 2024-03-19T02:58:10.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.865+0000 7f139a07e640 1 -- 172.21.15.33:0/1250102213 wait complete. 2024-03-19T02:58:10.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.865+0000 7f139a07e640 1 Processor -- start 2024-03-19T02:58:10.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.866+0000 7f139a07e640 1 -- start start 2024-03-19T02:58:10.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.866+0000 7f139a07e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f13941017c0 0x7f1394197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:10.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.866+0000 7f139a07e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1394102180 0x7f1394198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:10.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.866+0000 7f139a07e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13941077d0 0x7f139419c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:10.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.866+0000 7f139a07e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1394111a90 con 0x7f1394102180 2024-03-19T02:58:10.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.866+0000 7f139a07e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1394111910 con 0x7f13941077d0 2024-03-19T02:58:10.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.866+0000 7f139a07e640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1394111c10 con 0x7f13941017c0 2024-03-19T02:58:10.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.866+0000 7f1392ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1394102180 0x7f1394198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:10.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.866+0000 7f1392ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1394102180 0x7f1394198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49902/0 (socket says 172.21.15.33:49902) 2024-03-19T02:58:10.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.866+0000 7f1392ffd640 1 -- 172.21.15.33:0/1305068477 learned_addr learned my addr 172.21.15.33:0/1305068477 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:10.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.866+0000 7f1393fff640 1 --2- 172.21.15.33:0/1305068477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13941077d0 0x7f139419c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:10.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.866+0000 7f13937fe640 1 --2- 172.21.15.33:0/1305068477 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f13941017c0 0x7f1394197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:10.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.866+0000 7f1392ffd640 1 -- 172.21.15.33:0/1305068477 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f13941017c0 msgr2=0x7f1394197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:10.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.866+0000 7f1392ffd640 1 --2- 172.21.15.33:0/1305068477 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f13941017c0 0x7f1394197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:10.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.866+0000 7f1392ffd640 1 -- 172.21.15.33:0/1305068477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13941077d0 msgr2=0x7f139419c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:10.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.866+0000 7f1392ffd640 1 --2- 172.21.15.33:0/1305068477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13941077d0 0x7f139419c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:10.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.866+0000 7f1392ffd640 1 -- 172.21.15.33:0/1305068477 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f139419cfe0 con 0x7f1394102180 2024-03-19T02:58:10.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.867+0000 7f1392ffd640 1 --2- 172.21.15.33:0/1305068477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1394102180 0x7f1394198530 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f1388037040 tx=0x7f1388004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:10.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.867+0000 7f1390ff9640 1 -- 172.21.15.33:0/1305068477 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1388041020 con 0x7f1394102180 2024-03-19T02:58:10.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.867+0000 7f139a07e640 1 -- 172.21.15.33:0/1305068477 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f139419d270 con 0x7f1394102180 2024-03-19T02:58:10.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.867+0000 7f1390ff9640 1 -- 172.21.15.33:0/1305068477 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f138802c9c0 con 0x7f1394102180 2024-03-19T02:58:10.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.867+0000 7f139a07e640 1 -- 172.21.15.33:0/1305068477 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13941a4b60 con 0x7f1394102180 2024-03-19T02:58:10.871 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.867+0000 7f1390ff9640 1 -- 172.21.15.33:0/1305068477 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1388039040 con 0x7f1394102180 2024-03-19T02:58:10.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.869+0000 7f1390ff9640 1 -- 172.21.15.33:0/1305068477 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f1388044020 con 0x7f1394102180 2024-03-19T02:58:10.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.869+0000 7f139cc69740 1 Processor -- start 2024-03-19T02:58:10.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.869+0000 7f139cc69740 1 -- start start 2024-03-19T02:58:10.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.870+0000 7f1390ff9640 1 --2- 172.21.15.33:0/1305068477 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f137c0a2970 0x7f137c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:10.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.870+0000 7f139cc69740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fb8ab68380 0x55fb8ab68760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:10.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.870+0000 7f139cc69740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb8ab68ca0 0x55fb8ab71340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:10.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.870+0000 7f139cc69740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb8ab71880 0x55fb8ab73c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:10.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.870+0000 7f139cc69740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55fb8aacc160 con 0x55fb8ab68ca0 2024-03-19T02:58:10.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.870+0000 7f139cc69740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55fb8aa74e10 con 0x55fb8ab68380 2024-03-19T02:58:10.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.870+0000 7f139cc69740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55fb8aac8200 con 0x55fb8ab71880 2024-03-19T02:58:10.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.870+0000 7f1393fff640 1 --2- 172.21.15.33:0/1305068477 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f137c0a2970 0x7f137c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:10.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.870+0000 7f13937fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fb8ab68380 0x55fb8ab68760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:10.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.870+0000 7f1392ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb8ab68ca0 0x55fb8ab71340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:10.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.870+0000 7f13937fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fb8ab68380 0x55fb8ab68760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:51432/0 (socket says 172.21.15.33:51432) 2024-03-19T02:58:10.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.870+0000 7f13937fe640 1 -- 172.21.15.33:0/218440068 learned_addr learned my addr 172.21.15.33:0/218440068 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:10.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.870+0000 7f1393fff640 1 --2- 172.21.15.33:0/218440068 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb8ab71880 0x55fb8ab73c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:10.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.870+0000 7f1390ff9640 1 -- 172.21.15.33:0/1305068477 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f13880f0870 con 0x7f1394102180 2024-03-19T02:58:10.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.870+0000 7f13937fe640 1 -- 172.21.15.33:0/218440068 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb8ab71880 msgr2=0x55fb8ab73c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:10.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.870+0000 7f13937fe640 1 --2- 172.21.15.33:0/218440068 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb8ab71880 0x55fb8ab73c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:10.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.870+0000 7f13937fe640 1 -- 172.21.15.33:0/218440068 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb8ab68ca0 msgr2=0x55fb8ab71340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:10.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.870+0000 7f13937fe640 1 --2- 172.21.15.33:0/218440068 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb8ab68ca0 0x55fb8ab71340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:10.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.870+0000 7f13937fe640 1 -- 172.21.15.33:0/218440068 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb8a8a87e0 con 0x55fb8ab68380 2024-03-19T02:58:10.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.870+0000 7f1393fff640 1 --2- 172.21.15.33:0/1305068477 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f137c0a2970 0x7f137c0a4e30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f138c009fd0 tx=0x7f138c00eba0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:10.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.870+0000 7f13937fe640 1 --2- 172.21.15.33:0/218440068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fb8ab68380 0x55fb8ab68760 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1384009c70 tx=0x7f1384011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ed9791fb3d9cb24 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:10.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.871+0000 7f1379ffb640 1 -- 172.21.15.33:0/218440068 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138401e070 con 0x55fb8ab68380 2024-03-19T02:58:10.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.871+0000 7f1379ffb640 1 -- 172.21.15.33:0/218440068 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1384009e20 con 0x55fb8ab68380 2024-03-19T02:58:10.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.871+0000 7f1379ffb640 1 -- 172.21.15.33:0/218440068 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1384019690 con 0x55fb8ab68380 2024-03-19T02:58:10.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.871+0000 7f139cc69740 1 -- 172.21.15.33:0/218440068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fb8ab68380 msgr2=0x55fb8ab68760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:10.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.871+0000 7f139cc69740 1 --2- 172.21.15.33:0/218440068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fb8ab68380 0x55fb8ab68760 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1384009c70 tx=0x7f1384011810 comp rx=0 tx=0).stop 2024-03-19T02:58:10.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.871+0000 7f139cc69740 1 -- 172.21.15.33:0/218440068 shutdown_connections 2024-03-19T02:58:10.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.871+0000 7f139cc69740 1 --2- 172.21.15.33:0/218440068 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb8ab71880 0x55fb8ab73c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:10.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.871+0000 7f139cc69740 1 --2- 172.21.15.33:0/218440068 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb8ab68ca0 0x55fb8ab71340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:10.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.871+0000 7f139cc69740 1 --2- 172.21.15.33:0/218440068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fb8ab68380 0x55fb8ab68760 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:10.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.871+0000 7f139cc69740 1 -- 172.21.15.33:0/218440068 >> 172.21.15.33:0/218440068 conn(0x55fb8ab62ab0 msgr2=0x55fb8ab62e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:10.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.871+0000 7f139cc69740 1 -- 172.21.15.33:0/218440068 shutdown_connections 2024-03-19T02:58:10.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.871+0000 7f139cc69740 1 -- 172.21.15.33:0/218440068 wait complete. 2024-03-19T02:58:10.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.872+0000 7f139cc69740 1 -- start start 2024-03-19T02:58:10.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.872+0000 7f139cc69740 1 Processor -- start 2024-03-19T02:58:10.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.872+0000 7f139cc69740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb8ab68380 0x55fb8ab6e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:10.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.872+0000 7f139cc69740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fb8ab68ca0 0x55fb8ab6ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:10.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.872+0000 7f139cc69740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb8ab71880 0x55fb8ab6f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:10.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.872+0000 7f139cc69740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55fb8ab74670 con 0x55fb8ab71880 2024-03-19T02:58:10.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.872+0000 7f139cc69740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55fb8aa881a0 con 0x55fb8ab68ca0 2024-03-19T02:58:10.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.872+0000 7f139cc69740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55fb8ab747f0 con 0x55fb8ab68380 2024-03-19T02:58:10.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.872+0000 7f139cc69740 10 client.0 _ll_register_callbacks cb 0x55fb8a7dfc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:58:10.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.872+0000 7f1393fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb8ab71880 0x55fb8ab6f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:10.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.872+0000 7f13937fe640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb8ab68380 0x55fb8ab6e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:10.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.872+0000 7f13937fe640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb8ab68380 0x55fb8ab6e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:37904/0 (socket says 172.21.15.33:37904) 2024-03-19T02:58:10.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.872+0000 7f1392ffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fb8ab68ca0 0x55fb8ab6ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:10.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.872+0000 7f1393fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb8ab71880 0x55fb8ab6f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49924/0 (socket says 172.21.15.33:49924) 2024-03-19T02:58:10.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.872+0000 7f13937fe640 1 -- 172.21.15.33:0/393887934 learned_addr learned my addr 172.21.15.33:0/393887934 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:10.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.873+0000 7f13937fe640 1 -- 172.21.15.33:0/393887934 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fb8ab68ca0 msgr2=0x55fb8ab6ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:10.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.873+0000 7f13937fe640 1 --2- 172.21.15.33:0/393887934 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fb8ab68ca0 0x55fb8ab6ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:10.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.873+0000 7f13937fe640 1 -- 172.21.15.33:0/393887934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb8ab71880 msgr2=0x55fb8ab6f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:10.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.873+0000 7f13937fe640 1 --2- 172.21.15.33:0/393887934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb8ab71880 0x55fb8ab6f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:10.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.873+0000 7f13937fe640 1 -- 172.21.15.33:0/393887934 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb8a908f70 con 0x55fb8ab68380 2024-03-19T02:58:10.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.873+0000 7f13937fe640 1 --2- 172.21.15.33:0/393887934 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb8ab68380 0x55fb8ab6e740 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1384000c00 tx=0x7f1384003820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48d637972c5fa7f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:10.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.873+0000 7f1372ffd640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-19T02:58:10.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.873+0000 7f1372ffd640 1 -- 172.21.15.33:0/393887934 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138401e040 con 0x55fb8ab68380 2024-03-19T02:58:10.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.873+0000 7f139cc69740 1 -- 172.21.15.33:0/393887934 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1384009e20 con 0x55fb8ab68380 2024-03-19T02:58:10.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.873+0000 7f1372ffd640 1 -- 172.21.15.33:0/393887934 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f1384005040 con 0x55fb8ab68380 2024-03-19T02:58:10.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.873+0000 7f1372ffd640 1 -- 172.21.15.33:0/393887934 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1384031050 con 0x55fb8ab68380 2024-03-19T02:58:10.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.873+0000 7f139cc69740 10 client.26389 fetch_fsmap learned FSMap version 25 2024-03-19T02:58:10.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.873+0000 7f139cc69740 1 -- 172.21.15.33:0/393887934 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55fb8a7a0170 con 0x55fb8ab68380 2024-03-19T02:58:10.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.873+0000 7f1372ffd640 1 -- 172.21.15.33:0/393887934 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1384009e20 con 0x55fb8ab68380 2024-03-19T02:58:10.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.874+0000 7f1372ffd640 1 -- 172.21.15.33:0/393887934 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f138402c030 con 0x55fb8ab68380 2024-03-19T02:58:10.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.874+0000 7f139cc69740 10 client.26389 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:58:10.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.874+0000 7f139cc69740 10 client.26389 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:58:10.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.874+0000 7f139cc69740 20 client.26389 populate_metadata read hostname 'smithi033' 2024-03-19T02:58:10.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.874+0000 7f139cc69740 1 --2- 172.21.15.33:0/393887934 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55fb8ab81ae0 0x55fb8ab83ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:10.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.874+0000 7f139cc69740 4 client.26389 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:58:10.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.874+0000 7f139cc69740 1 -- 172.21.15.33:0/393887934 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb8aad96a0 con 0x55fb8ab81ae0 2024-03-19T02:58:10.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.874+0000 7f1392ffd640 1 --2- 172.21.15.33:0/393887934 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55fb8ab81ae0 0x55fb8ab83ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:10.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.874+0000 7f1372ffd640 10 client.26389 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:58:10.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.874+0000 7f1392ffd640 1 --2- 172.21.15.33:0/393887934 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55fb8ab81ae0 0x55fb8ab83ed0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f3130622ffac59f server_cookie=698617ef1f03ec92 in_seq=0 out_seq=0 2024-03-19T02:58:10.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.875+0000 7f1372ffd640 1 -- 172.21.15.33:0/393887934 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fb8aad96a0 con 0x55fb8ab81ae0 2024-03-19T02:58:10.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.875+0000 7f1372ffd640 10 client.26389 handle_command_reply: tid=0 2024-03-19T02:58:10.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.875+0000 7f139cc69740 1 client.26389 shutdown 2024-03-19T02:58:10.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.875+0000 7f139cc69740 1 -- 172.21.15.33:0/393887934 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb8ab68380 msgr2=0x55fb8ab6e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:10.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.875+0000 7f139cc69740 1 --2- 172.21.15.33:0/393887934 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb8ab68380 0x55fb8ab6e740 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1384000c00 tx=0x7f1384003820 comp rx=0 tx=0).stop 2024-03-19T02:58:10.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.875+0000 7f139cc69740 1 -- 172.21.15.33:0/393887934 shutdown_connections 2024-03-19T02:58:10.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.875+0000 7f139cc69740 1 --2- 172.21.15.33:0/393887934 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55fb8ab81ae0 0x55fb8ab83ed0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:10.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.875+0000 7f139cc69740 1 --2- 172.21.15.33:0/393887934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb8ab71880 0x55fb8ab6f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:10.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.875+0000 7f139cc69740 1 --2- 172.21.15.33:0/393887934 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fb8ab68ca0 0x55fb8ab6ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:10.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.875+0000 7f139cc69740 1 --2- 172.21.15.33:0/393887934 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb8ab68380 0x55fb8ab6e740 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:10.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.875+0000 7f1372ffd640 0 client.26389 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:58:10.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.875+0000 7f139cc69740 1 -- 172.21.15.33:0/393887934 >> 172.21.15.33:0/393887934 conn(0x55fb8ab62ab0 msgr2=0x55fb8ab64800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:10.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.875+0000 7f139cc69740 1 -- 172.21.15.33:0/393887934 shutdown_connections 2024-03-19T02:58:10.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.875+0000 7f139cc69740 1 -- 172.21.15.33:0/393887934 wait complete. 2024-03-19T02:58:10.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.875+0000 7f139cc69740 20 client.26389 trim_cache size 0 max 16384 2024-03-19T02:58:10.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.893+0000 7f139cc69740 1 Processor -- start 2024-03-19T02:58:10.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.893+0000 7f139cc69740 1 -- start start 2024-03-19T02:58:10.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.893+0000 7f139cc69740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb8ab68380 0x55fb8ab70e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:10.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.893+0000 7f139cc69740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fb8ab68ca0 0x55fb8ab7d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:10.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.893+0000 7f139cc69740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb8ab71880 0x55fb8ab6d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:10.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.893+0000 7f139cc69740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f13941b1420 con 0x55fb8ab71880 2024-03-19T02:58:10.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.893+0000 7f139cc69740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f13941aecf0 con 0x55fb8ab68ca0 2024-03-19T02:58:10.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.893+0000 7f139cc69740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f13941acb20 con 0x55fb8ab68380 2024-03-19T02:58:10.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.893+0000 7f1393fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb8ab71880 0x55fb8ab6d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:10.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.893+0000 7f1393fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb8ab71880 0x55fb8ab6d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49926/0 (socket says 172.21.15.33:49926) 2024-03-19T02:58:10.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.893+0000 7f1393fff640 1 -- 172.21.15.33:0/979116811 learned_addr learned my addr 172.21.15.33:0/979116811 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:10.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.893+0000 7f1392ffd640 1 --2- 172.21.15.33:0/979116811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fb8ab68ca0 0x55fb8ab7d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:10.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.893+0000 7f13937fe640 1 --2- 172.21.15.33:0/979116811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb8ab68380 0x55fb8ab70e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:10.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.893+0000 7f1393fff640 1 -- 172.21.15.33:0/979116811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb8ab68380 msgr2=0x55fb8ab70e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:10.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.893+0000 7f1393fff640 1 --2- 172.21.15.33:0/979116811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb8ab68380 0x55fb8ab70e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:10.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.893+0000 7f1393fff640 1 -- 172.21.15.33:0/979116811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fb8ab68ca0 msgr2=0x55fb8ab7d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:10.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.893+0000 7f1393fff640 1 --2- 172.21.15.33:0/979116811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fb8ab68ca0 0x55fb8ab7d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:10.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.893+0000 7f1393fff640 1 -- 172.21.15.33:0/979116811 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb8ab628e0 con 0x55fb8ab71880 2024-03-19T02:58:10.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.893+0000 7f1392ffd640 1 --2- 172.21.15.33:0/979116811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fb8ab68ca0 0x55fb8ab7d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:58:10.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.894+0000 7f1393fff640 1 --2- 172.21.15.33:0/979116811 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb8ab71880 0x55fb8ab6d5f0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f138c00ee70 tx=0x7f138c01f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d70f4c507e7caddb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:10.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.894+0000 7f13797fa640 1 -- 172.21.15.33:0/979116811 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138c024040 con 0x55fb8ab71880 2024-03-19T02:58:10.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.894+0000 7f13797fa640 1 -- 172.21.15.33:0/979116811 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f138c01ce40 con 0x55fb8ab71880 2024-03-19T02:58:10.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.894+0000 7f13797fa640 1 -- 172.21.15.33:0/979116811 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138c01ea70 con 0x55fb8ab71880 2024-03-19T02:58:10.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.894+0000 7f139cc69740 1 -- 172.21.15.33:0/979116811 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb8ab71880 msgr2=0x55fb8ab6d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:10.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.894+0000 7f139cc69740 1 --2- 172.21.15.33:0/979116811 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb8ab71880 0x55fb8ab6d5f0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f138c00ee70 tx=0x7f138c01f000 comp rx=0 tx=0).stop 2024-03-19T02:58:10.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.894+0000 7f139cc69740 1 -- 172.21.15.33:0/979116811 shutdown_connections 2024-03-19T02:58:10.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.894+0000 7f139cc69740 1 --2- 172.21.15.33:0/979116811 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb8ab71880 0x55fb8ab6d5f0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:10.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.894+0000 7f139cc69740 1 --2- 172.21.15.33:0/979116811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fb8ab68ca0 0x55fb8ab7d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:10.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.894+0000 7f139cc69740 1 --2- 172.21.15.33:0/979116811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb8ab68380 0x55fb8ab70e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:10.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.894+0000 7f139cc69740 1 -- 172.21.15.33:0/979116811 >> 172.21.15.33:0/979116811 conn(0x55fb8ab62ab0 msgr2=0x55fb8ab6cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:10.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.894+0000 7f139cc69740 1 -- 172.21.15.33:0/979116811 shutdown_connections 2024-03-19T02:58:10.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.894+0000 7f139cc69740 1 -- 172.21.15.33:0/979116811 wait complete. 2024-03-19T02:58:10.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.895+0000 7f139cc69740 1 -- start start 2024-03-19T02:58:10.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.895+0000 7f139cc69740 1 Processor -- start 2024-03-19T02:58:10.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.895+0000 7f139cc69740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fb8ab68380 0x55fb8ab755e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:10.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.895+0000 7f139cc69740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb8ab68ca0 0x55fb8ab75b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:10.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.895+0000 7f139cc69740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb8ab71880 0x55fb8ab76060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:10.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.895+0000 7f139cc69740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f13941ad1b0 con 0x55fb8ab68ca0 2024-03-19T02:58:10.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.895+0000 7f139cc69740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f13941ae620 con 0x55fb8ab68380 2024-03-19T02:58:10.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.895+0000 7f139cc69740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f13941ad880 con 0x55fb8ab71880 2024-03-19T02:58:10.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.895+0000 7f139cc69740 10 client.0 _ll_register_callbacks cb 0x7f138401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:58:10.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.895+0000 7f1392ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb8ab68ca0 0x55fb8ab75b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:10.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.895+0000 7f1392ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb8ab68ca0 0x55fb8ab75b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49934/0 (socket says 172.21.15.33:49934) 2024-03-19T02:58:10.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.895+0000 7f1392ffd640 1 -- 172.21.15.33:0/2077724336 learned_addr learned my addr 172.21.15.33:0/2077724336 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:10.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.895+0000 7f13937fe640 1 --2- 172.21.15.33:0/2077724336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fb8ab68380 0x55fb8ab755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:10.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.895+0000 7f1393fff640 1 --2- 172.21.15.33:0/2077724336 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb8ab71880 0x55fb8ab76060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:10.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.896+0000 7f1392ffd640 1 -- 172.21.15.33:0/2077724336 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb8ab71880 msgr2=0x55fb8ab76060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:10.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.896+0000 7f1392ffd640 1 --2- 172.21.15.33:0/2077724336 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb8ab71880 0x55fb8ab76060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:10.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.896+0000 7f1392ffd640 1 -- 172.21.15.33:0/2077724336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fb8ab68380 msgr2=0x55fb8ab755e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:10.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.896+0000 7f1392ffd640 1 --2- 172.21.15.33:0/2077724336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fb8ab68380 0x55fb8ab755e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:10.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.896+0000 7f1392ffd640 1 -- 172.21.15.33:0/2077724336 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb8abe6f60 con 0x55fb8ab68ca0 2024-03-19T02:58:10.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.896+0000 7f1392ffd640 1 --2- 172.21.15.33:0/2077724336 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb8ab68ca0 0x55fb8ab75b20 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f13880bc930 tx=0x7f1388048000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76eb264e3495006b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:10.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.896+0000 7f1372ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:58:10.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.896+0000 7f1372ffd640 1 -- 172.21.15.33:0/2077724336 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13880a7040 con 0x55fb8ab68ca0 2024-03-19T02:58:10.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.896+0000 7f1372ffd640 1 -- 172.21.15.33:0/2077724336 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f13880b92a0 con 0x55fb8ab68ca0 2024-03-19T02:58:10.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.896+0000 7f139cc69740 1 -- 172.21.15.33:0/2077724336 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fb8aad39f0 con 0x55fb8ab68ca0 2024-03-19T02:58:10.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.896+0000 7f1372ffd640 1 -- 172.21.15.33:0/2077724336 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f13880bcc30 con 0x55fb8ab68ca0 2024-03-19T02:58:10.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.896+0000 7f139cc69740 10 client.16704 fetch_fsmap learned FSMap version 25 2024-03-19T02:58:10.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.896+0000 7f139cc69740 1 -- 172.21.15.33:0/2077724336 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55fb8abe82e0 con 0x55fb8ab68ca0 2024-03-19T02:58:10.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.896+0000 7f1372ffd640 1 -- 172.21.15.33:0/2077724336 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55fb8aad39f0 con 0x55fb8ab68ca0 2024-03-19T02:58:10.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.896+0000 7f1372ffd640 1 -- 172.21.15.33:0/2077724336 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f138803c030 con 0x55fb8ab68ca0 2024-03-19T02:58:10.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.896+0000 7f139cc69740 10 client.16704 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:58:10.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.896+0000 7f139cc69740 10 client.16704 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:58:10.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.896+0000 7f139cc69740 20 client.16704 populate_metadata read hostname 'smithi033' 2024-03-19T02:58:10.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.896+0000 7f139cc69740 1 --2- 172.21.15.33:0/2077724336 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55fb8abe8a10 0x55fb8abeae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:10.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.896+0000 7f139cc69740 4 client.16704 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:58:10.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.896+0000 7f139cc69740 1 -- 172.21.15.33:0/2077724336 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55fb8aa51be0 con 0x55fb8abe8a10 2024-03-19T02:58:10.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.897+0000 7f13937fe640 1 --2- 172.21.15.33:0/2077724336 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55fb8abe8a10 0x55fb8abeae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:10.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.897+0000 7f1372ffd640 10 client.16704 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:58:10.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.897+0000 7f13937fe640 1 --2- 172.21.15.33:0/2077724336 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55fb8abe8a10 0x55fb8abeae30 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81faf2f3fcdf6767 server_cookie=881f0bdc3c0d58c3 in_seq=0 out_seq=0 2024-03-19T02:58:10.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.898+0000 7f1372ffd640 1 -- 172.21.15.33:0/2077724336 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55fb8aa51be0 con 0x55fb8abe8a10 2024-03-19T02:58:10.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.898+0000 7f1372ffd640 10 client.16704 handle_command_reply: tid=0 2024-03-19T02:58:10.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.898+0000 7f139cc69740 1 client.16704 shutdown 2024-03-19T02:58:10.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.898+0000 7f139cc69740 1 -- 172.21.15.33:0/2077724336 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb8ab68ca0 msgr2=0x55fb8ab75b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:10.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.898+0000 7f139cc69740 1 --2- 172.21.15.33:0/2077724336 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb8ab68ca0 0x55fb8ab75b20 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f13880bc930 tx=0x7f1388048000 comp rx=0 tx=0).stop 2024-03-19T02:58:10.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.898+0000 7f139cc69740 1 -- 172.21.15.33:0/2077724336 shutdown_connections 2024-03-19T02:58:10.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.898+0000 7f139cc69740 1 --2- 172.21.15.33:0/2077724336 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55fb8abe8a10 0x55fb8abeae30 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:10.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.898+0000 7f139cc69740 1 --2- 172.21.15.33:0/2077724336 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55fb8ab71880 0x55fb8ab76060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:10.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.899+0000 7f1372ffd640 0 client.16704 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:58:10.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.899+0000 7f139cc69740 1 --2- 172.21.15.33:0/2077724336 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55fb8ab68ca0 0x55fb8ab75b20 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:10.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.899+0000 7f139cc69740 1 --2- 172.21.15.33:0/2077724336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55fb8ab68380 0x55fb8ab755e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:10.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.899+0000 7f139cc69740 1 -- 172.21.15.33:0/2077724336 >> 172.21.15.33:0/2077724336 conn(0x55fb8ab62ab0 msgr2=0x55fb8ab6d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:10.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.899+0000 7f139cc69740 1 -- 172.21.15.33:0/2077724336 shutdown_connections 2024-03-19T02:58:10.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.899+0000 7f139cc69740 1 -- 172.21.15.33:0/2077724336 wait complete. 2024-03-19T02:58:10.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.899+0000 7f139cc69740 20 client.16704 trim_cache size 0 max 16384 2024-03-19T02:58:10.906 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:58:10.906 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-03-19T02:58:10.906 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "79f16656-534d-45ee-ac8f-cc742298ea88", 2024-03-19T02:58:10.906 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-03-19T02:58:10.906 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:58:10.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.899+0000 7f139a07e640 1 -- 172.21.15.33:0/1305068477 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f137c0a2970 msgr2=0x7f137c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:10.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.899+0000 7f139a07e640 1 --2- 172.21.15.33:0/1305068477 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f137c0a2970 0x7f137c0a4e30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f138c009fd0 tx=0x7f138c00eba0 comp rx=0 tx=0).stop 2024-03-19T02:58:10.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.899+0000 7f139a07e640 1 -- 172.21.15.33:0/1305068477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1394102180 msgr2=0x7f1394198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:10.907 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.899+0000 7f139a07e640 1 --2- 172.21.15.33:0/1305068477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1394102180 0x7f1394198530 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f1388037040 tx=0x7f1388004040 comp rx=0 tx=0).stop 2024-03-19T02:58:10.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.899+0000 7f139a07e640 1 -- 172.21.15.33:0/1305068477 shutdown_connections 2024-03-19T02:58:10.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.899+0000 7f139a07e640 1 --2- 172.21.15.33:0/1305068477 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f137c0a2970 0x7f137c0a4e30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:10.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.899+0000 7f139a07e640 1 --2- 172.21.15.33:0/1305068477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13941077d0 0x7f139419c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:10.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.899+0000 7f139a07e640 1 --2- 172.21.15.33:0/1305068477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1394102180 0x7f1394198530 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:10.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.899+0000 7f139a07e640 1 --2- 172.21.15.33:0/1305068477 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f13941017c0 0x7f1394197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:10.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.899+0000 7f139a07e640 1 -- 172.21.15.33:0/1305068477 >> 172.21.15.33:0/1305068477 conn(0x7f13940fd250 msgr2=0x7f13940fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:10.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.899+0000 7f139a07e640 1 -- 172.21.15.33:0/1305068477 shutdown_connections 2024-03-19T02:58:10.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:10.899+0000 7f139a07e640 1 -- 172.21.15.33:0/1305068477 wait complete. 2024-03-19T02:58:10.911 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:58:10.912 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:58:10.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:10 smithi096 ceph-mon[35946]: pgmap v144: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-19T02:58:11.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.057+0000 7f6d59661640 1 -- 172.21.15.33:0/2844350709 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d541055c0 msgr2=0x7f6d541059a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:11.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.057+0000 7f6d59661640 1 --2- 172.21.15.33:0/2844350709 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d541055c0 0x7f6d541059a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c009860 tx=0x7f6d4c02bc20 comp rx=0 tx=0).stop 2024-03-19T02:58:11.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.057+0000 7f6d59661640 1 -- 172.21.15.33:0/2844350709 shutdown_connections 2024-03-19T02:58:11.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.057+0000 7f6d59661640 1 --2- 172.21.15.33:0/2844350709 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d54101080 0x7f6d5410d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.057+0000 7f6d59661640 1 --2- 172.21.15.33:0/2844350709 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6d541006e0 0x7f6d54100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.057+0000 7f6d59661640 1 --2- 172.21.15.33:0/2844350709 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d541055c0 0x7f6d541059a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.057+0000 7f6d59661640 1 -- 172.21.15.33:0/2844350709 >> 172.21.15.33:0/2844350709 conn(0x7f6d540777b0 msgr2=0x7f6d54077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:11.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.058+0000 7f6d59661640 1 -- 172.21.15.33:0/2844350709 shutdown_connections 2024-03-19T02:58:11.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.058+0000 7f6d59661640 1 -- 172.21.15.33:0/2844350709 wait complete. 2024-03-19T02:58:11.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.058+0000 7f6d59661640 1 Processor -- start 2024-03-19T02:58:11.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.058+0000 7f6d59661640 1 -- start start 2024-03-19T02:58:11.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.058+0000 7f6d59661640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d541006e0 0x7f6d54102260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:11.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.058+0000 7f6d59661640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d54101080 0x7f6d541027a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:11.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.058+0000 7f6d59661640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6d541055c0 0x7f6d54102ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:11.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.058+0000 7f6d59661640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6d54110960 con 0x7f6d541055c0 2024-03-19T02:58:11.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.058+0000 7f6d59661640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6d541107e0 con 0x7f6d54101080 2024-03-19T02:58:11.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.058+0000 7f6d59661640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6d54110ae0 con 0x7f6d541006e0 2024-03-19T02:58:11.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.059+0000 7f6d537fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6d541055c0 0x7f6d54102ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:11.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.059+0000 7f6d537fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6d541055c0 0x7f6d54102ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49952/0 (socket says 172.21.15.33:49952) 2024-03-19T02:58:11.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.059+0000 7f6d537fe640 1 -- 172.21.15.33:0/1298873780 learned_addr learned my addr 172.21.15.33:0/1298873780 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:11.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.059+0000 7f6d52ffd640 1 --2- 172.21.15.33:0/1298873780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d541006e0 0x7f6d54102260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:11.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.059+0000 7f6d527fc640 1 --2- 172.21.15.33:0/1298873780 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d54101080 0x7f6d541027a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:11.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.059+0000 7f6d537fe640 1 -- 172.21.15.33:0/1298873780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d541006e0 msgr2=0x7f6d54102260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:11.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.059+0000 7f6d537fe640 1 --2- 172.21.15.33:0/1298873780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d541006e0 0x7f6d54102260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.059+0000 7f6d537fe640 1 -- 172.21.15.33:0/1298873780 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d54101080 msgr2=0x7f6d541027a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:11.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.059+0000 7f6d537fe640 1 --2- 172.21.15.33:0/1298873780 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d54101080 0x7f6d541027a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.059+0000 7f6d537fe640 1 -- 172.21.15.33:0/1298873780 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d541a6470 con 0x7f6d541055c0 2024-03-19T02:58:11.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.059+0000 7f6d52ffd640 1 --2- 172.21.15.33:0/1298873780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d541006e0 0x7f6d54102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:58:11.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.059+0000 7f6d527fc640 1 --2- 172.21.15.33:0/1298873780 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d54101080 0x7f6d541027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:58:11.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.059+0000 7f6d537fe640 1 --2- 172.21.15.33:0/1298873780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6d541055c0 0x7f6d54102ce0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f6d4400eb10 tx=0x7f6d4400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:11.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.059+0000 7f6d4bfff640 1 -- 172.21.15.33:0/1298873780 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d44004b20 con 0x7f6d541055c0 2024-03-19T02:58:11.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.059+0000 7f6d59661640 1 -- 172.21.15.33:0/1298873780 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d541a6760 con 0x7f6d541055c0 2024-03-19T02:58:11.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.059+0000 7f6d4bfff640 1 -- 172.21.15.33:0/1298873780 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6d44004cc0 con 0x7f6d541055c0 2024-03-19T02:58:11.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.059+0000 7f6d4bfff640 1 -- 172.21.15.33:0/1298873780 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d44010690 con 0x7f6d541055c0 2024-03-19T02:58:11.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.059+0000 7f6d59661640 1 -- 172.21.15.33:0/1298873780 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d541a6cb0 con 0x7f6d541055c0 2024-03-19T02:58:11.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.061+0000 7f6d4bfff640 1 -- 172.21.15.33:0/1298873780 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f6d44004fe0 con 0x7f6d541055c0 2024-03-19T02:58:11.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.062+0000 7f6d5c255740 1 Processor -- start 2024-03-19T02:58:11.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.062+0000 7f6d5c255740 1 -- start start 2024-03-19T02:58:11.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.062+0000 7f6d4bfff640 1 --2- 172.21.15.33:0/1298873780 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6d400a29a0 0x7f6d400a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:11.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.062+0000 7f6d527fc640 1 --2- 172.21.15.33:0/1298873780 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6d400a29a0 0x7f6d400a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:11.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.062+0000 7f6d5c255740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565802cca30 0x5565802cce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:11.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.062+0000 7f6d5c255740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5565802cd350 0x5565802d59f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:11.070 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.062+0000 7f6d4bfff640 1 -- 172.21.15.33:0/1298873780 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f6d440cf750 con 0x7f6d541055c0 2024-03-19T02:58:11.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.062+0000 7f6d5c255740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565802d5f30 0x5565802d8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:11.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.062+0000 7f6d5c255740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x556580230500 con 0x5565802d5f30 2024-03-19T02:58:11.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.062+0000 7f6d5c255740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5565801d8be0 con 0x5565802cca30 2024-03-19T02:58:11.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.062+0000 7f6d5c255740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55658022c5a0 con 0x5565802cd350 2024-03-19T02:58:11.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.062+0000 7f6d537fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565802d5f30 0x5565802d8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:11.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.062+0000 7f6d537fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565802d5f30 0x5565802d8320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49960/0 (socket says 172.21.15.33:49960) 2024-03-19T02:58:11.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.062+0000 7f6d537fe640 1 -- 172.21.15.33:0/2960834163 learned_addr learned my addr 172.21.15.33:0/2960834163 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:11.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.062+0000 7f6d52ffd640 1 --2- 172.21.15.33:0/2960834163 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565802cca30 0x5565802cce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:11.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.062+0000 7f6d527fc640 1 --2- 172.21.15.33:0/2960834163 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5565802cd350 0x5565802d59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:11.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.063+0000 7f6d537fe640 1 -- 172.21.15.33:0/2960834163 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5565802cd350 msgr2=0x5565802d59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:11.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.063+0000 7f6d537fe640 1 --2- 172.21.15.33:0/2960834163 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5565802cd350 0x5565802d59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.063+0000 7f6d537fe640 1 -- 172.21.15.33:0/2960834163 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565802cca30 msgr2=0x5565802cce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:11.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.063+0000 7f6d537fe640 1 --2- 172.21.15.33:0/2960834163 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565802cca30 0x5565802cce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.063+0000 7f6d537fe640 1 -- 172.21.15.33:0/2960834163 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565801fa000 con 0x5565802d5f30 2024-03-19T02:58:11.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.063+0000 7f6d52ffd640 1 --2- 172.21.15.33:0/2960834163 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565802cca30 0x5565802cce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:58:11.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.063+0000 7f6d527fc640 1 --2- 172.21.15.33:0/2960834163 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5565802cd350 0x5565802d59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:58:11.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.063+0000 7f6d527fc640 1 --2- 172.21.15.33:0/1298873780 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6d400a29a0 0x7f6d400a4e60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6d54103ca0 tx=0x7f6d3c011810 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:11.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.063+0000 7f6d537fe640 1 --2- 172.21.15.33:0/2960834163 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565802d5f30 0x5565802d8320 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f6d44005ba0 tx=0x7f6d44004f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aa458554cb7b362 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:11.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.063+0000 7f6d497fa640 1 -- 172.21.15.33:0/2960834163 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d4409c100 con 0x5565802d5f30 2024-03-19T02:58:11.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.063+0000 7f6d497fa640 1 -- 172.21.15.33:0/2960834163 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6d4409ba00 con 0x5565802d5f30 2024-03-19T02:58:11.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.063+0000 7f6d497fa640 1 -- 172.21.15.33:0/2960834163 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d440a3d10 con 0x5565802d5f30 2024-03-19T02:58:11.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.064+0000 7f6d5c255740 1 -- 172.21.15.33:0/2960834163 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565802d5f30 msgr2=0x5565802d8320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:11.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.064+0000 7f6d5c255740 1 --2- 172.21.15.33:0/2960834163 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565802d5f30 0x5565802d8320 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f6d44005ba0 tx=0x7f6d44004f50 comp rx=0 tx=0).stop 2024-03-19T02:58:11.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.064+0000 7f6d5c255740 1 -- 172.21.15.33:0/2960834163 shutdown_connections 2024-03-19T02:58:11.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.064+0000 7f6d5c255740 1 --2- 172.21.15.33:0/2960834163 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5565802cd350 0x5565802d59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.064+0000 7f6d5c255740 1 --2- 172.21.15.33:0/2960834163 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565802d5f30 0x5565802d8320 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.064+0000 7f6d5c255740 1 --2- 172.21.15.33:0/2960834163 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565802cca30 0x5565802cce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.064+0000 7f6d5c255740 1 -- 172.21.15.33:0/2960834163 >> 172.21.15.33:0/2960834163 conn(0x5565802c7160 msgr2=0x5565802c7540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:11.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.064+0000 7f6d5c255740 1 -- 172.21.15.33:0/2960834163 shutdown_connections 2024-03-19T02:58:11.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.064+0000 7f6d5c255740 1 -- 172.21.15.33:0/2960834163 wait complete. 2024-03-19T02:58:11.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.064+0000 7f6d5c255740 1 -- start start 2024-03-19T02:58:11.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.064+0000 7f6d5c255740 1 Processor -- start 2024-03-19T02:58:11.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.064+0000 7f6d5c255740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5565802cca30 0x5565802d2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:11.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.064+0000 7f6d5c255740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565802cd350 0x5565802d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:11.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.064+0000 7f6d5c255740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565802d5f30 0x5565802d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:11.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.064+0000 7f6d5c255740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5565802d8d20 con 0x5565802cd350 2024-03-19T02:58:11.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.064+0000 7f6d5c255740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5565801ec160 con 0x5565802d5f30 2024-03-19T02:58:11.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.064+0000 7f6d5c255740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5565802d8ea0 con 0x5565802cca30 2024-03-19T02:58:11.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.065+0000 7f6d5c255740 10 client.0 _ll_register_callbacks cb 0x556580228a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:58:11.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.065+0000 7f6d527fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565802cd350 0x5565802d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:11.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.065+0000 7f6d527fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565802cd350 0x5565802d33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49964/0 (socket says 172.21.15.33:49964) 2024-03-19T02:58:11.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.065+0000 7f6d527fc640 1 -- 172.21.15.33:0/2407763009 learned_addr learned my addr 172.21.15.33:0/2407763009 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:11.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.065+0000 7f6d52ffd640 1 --2- 172.21.15.33:0/2407763009 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5565802cca30 0x5565802d2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:11.073 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.065+0000 7f6d537fe640 1 --2- 172.21.15.33:0/2407763009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565802d5f30 0x5565802d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:11.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.065+0000 7f6d527fc640 1 -- 172.21.15.33:0/2407763009 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5565802cca30 msgr2=0x5565802d2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:11.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.065+0000 7f6d527fc640 1 --2- 172.21.15.33:0/2407763009 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5565802cca30 0x5565802d2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.065+0000 7f6d527fc640 1 -- 172.21.15.33:0/2407763009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565802d5f30 msgr2=0x5565802d3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:11.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.065+0000 7f6d527fc640 1 --2- 172.21.15.33:0/2407763009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565802d5f30 0x5565802d3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.065+0000 7f6d527fc640 1 -- 172.21.15.33:0/2407763009 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55658000e7d0 con 0x5565802cd350 2024-03-19T02:58:11.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.065+0000 7f6d527fc640 1 --2- 172.21.15.33:0/2407763009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565802cd350 0x5565802d33d0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c00fd00 tx=0x7f6d3c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff7b85f20f5a38e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:11.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.065+0000 7f6d3a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:58:11.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.065+0000 7f6d3a7fc640 1 -- 172.21.15.33:0/2407763009 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d3c017040 con 0x5565802cd350 2024-03-19T02:58:11.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.065+0000 7f6d3a7fc640 1 -- 172.21.15.33:0/2407763009 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f6d3c01ede0 con 0x5565802cd350 2024-03-19T02:58:11.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.065+0000 7f6d5c255740 1 -- 172.21.15.33:0/2407763009 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d4409ba00 con 0x5565802cd350 2024-03-19T02:58:11.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.065+0000 7f6d3a7fc640 1 -- 172.21.15.33:0/2407763009 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f6d3c005020 con 0x5565802cd350 2024-03-19T02:58:11.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.065+0000 7f6d5c255740 10 client.16716 fetch_fsmap learned FSMap version 25 2024-03-19T02:58:11.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.065+0000 7f6d5c255740 1 -- 172.21.15.33:0/2407763009 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55657ff443f0 con 0x5565802cd350 2024-03-19T02:58:11.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.066+0000 7f6d3a7fc640 1 -- 172.21.15.33:0/2407763009 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6d4409ba00 con 0x5565802cd350 2024-03-19T02:58:11.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.066+0000 7f6d3a7fc640 1 -- 172.21.15.33:0/2407763009 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f6d3c025070 con 0x5565802cd350 2024-03-19T02:58:11.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.066+0000 7f6d5c255740 10 client.16716 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:58:11.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.066+0000 7f6d5c255740 10 client.16716 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:58:11.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.066+0000 7f6d5c255740 20 client.16716 populate_metadata read hostname 'smithi033' 2024-03-19T02:58:11.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.066+0000 7f6d5c255740 1 --2- 172.21.15.33:0/2407763009 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5565802e62e0 0x5565802e8700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:11.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.066+0000 7f6d5c255740 4 client.16716 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:58:11.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.066+0000 7f6d5c255740 1 -- 172.21.15.33:0/2407763009 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55658023da40 con 0x5565802e62e0 2024-03-19T02:58:11.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.066+0000 7f6d52ffd640 1 --2- 172.21.15.33:0/2407763009 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5565802e62e0 0x5565802e8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:11.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.066+0000 7f6d52ffd640 1 --2- 172.21.15.33:0/2407763009 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5565802e62e0 0x5565802e8700 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=913543283ac5420d server_cookie=b1ea5789d88baf94 in_seq=0 out_seq=0 2024-03-19T02:58:11.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.066+0000 7f6d3a7fc640 10 client.16716 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:58:11.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.067+0000 7f6d3a7fc640 1 -- 172.21.15.33:0/2407763009 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55658023da40 con 0x5565802e62e0 2024-03-19T02:58:11.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.067+0000 7f6d3a7fc640 10 client.16716 handle_command_reply: tid=0 2024-03-19T02:58:11.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.067+0000 7f6d5c255740 1 client.16716 shutdown 2024-03-19T02:58:11.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.067+0000 7f6d5c255740 1 -- 172.21.15.33:0/2407763009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565802cd350 msgr2=0x5565802d33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:11.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.067+0000 7f6d5c255740 1 --2- 172.21.15.33:0/2407763009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565802cd350 0x5565802d33d0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c00fd00 tx=0x7f6d3c016000 comp rx=0 tx=0).stop 2024-03-19T02:58:11.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.067+0000 7f6d5c255740 1 -- 172.21.15.33:0/2407763009 shutdown_connections 2024-03-19T02:58:11.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.067+0000 7f6d5c255740 1 --2- 172.21.15.33:0/2407763009 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5565802e62e0 0x5565802e8700 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.067+0000 7f6d5c255740 1 --2- 172.21.15.33:0/2407763009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565802d5f30 0x5565802d3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.067+0000 7f6d5c255740 1 --2- 172.21.15.33:0/2407763009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565802cd350 0x5565802d33d0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.067+0000 7f6d5c255740 1 --2- 172.21.15.33:0/2407763009 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5565802cca30 0x5565802d2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.067+0000 7f6d3a7fc640 0 client.16716 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:58:11.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.067+0000 7f6d5c255740 1 -- 172.21.15.33:0/2407763009 >> 172.21.15.33:0/2407763009 conn(0x5565802c7160 msgr2=0x5565802c8eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:11.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.067+0000 7f6d5c255740 1 -- 172.21.15.33:0/2407763009 shutdown_connections 2024-03-19T02:58:11.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.067+0000 7f6d5c255740 1 -- 172.21.15.33:0/2407763009 wait complete. 2024-03-19T02:58:11.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.067+0000 7f6d5c255740 20 client.16716 trim_cache size 0 max 16384 2024-03-19T02:58:11.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.085+0000 7f6d5c255740 1 Processor -- start 2024-03-19T02:58:11.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.085+0000 7f6d5c255740 1 -- start start 2024-03-19T02:58:11.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.085+0000 7f6d5c255740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5565802cca30 0x5565802d55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:11.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.085+0000 7f6d5c255740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565802cd350 0x5565802d1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:11.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.085+0000 7f6d5c255740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565802d5f30 0x5565802ce3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:11.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.085+0000 7f6d5c255740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6d541b3670 con 0x5565802d5f30 2024-03-19T02:58:11.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.085+0000 7f6d5c255740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6d541b0f40 con 0x5565802cd350 2024-03-19T02:58:11.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.085+0000 7f6d5c255740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6d541aed70 con 0x5565802cca30 2024-03-19T02:58:11.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.085+0000 7f6d537fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565802d5f30 0x5565802ce3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:11.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.085+0000 7f6d537fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565802d5f30 0x5565802ce3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49966/0 (socket says 172.21.15.33:49966) 2024-03-19T02:58:11.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.085+0000 7f6d537fe640 1 -- 172.21.15.33:0/2399735289 learned_addr learned my addr 172.21.15.33:0/2399735289 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:11.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.085+0000 7f6d52ffd640 1 --2- 172.21.15.33:0/2399735289 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5565802cca30 0x5565802d55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:11.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.085+0000 7f6d527fc640 1 --2- 172.21.15.33:0/2399735289 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565802cd350 0x5565802d1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:11.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.085+0000 7f6d537fe640 1 -- 172.21.15.33:0/2399735289 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5565802cca30 msgr2=0x5565802d55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:11.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.085+0000 7f6d537fe640 1 --2- 172.21.15.33:0/2399735289 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5565802cca30 0x5565802d55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.085+0000 7f6d537fe640 1 -- 172.21.15.33:0/2399735289 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565802cd350 msgr2=0x5565802d1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:11.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.085+0000 7f6d537fe640 1 --2- 172.21.15.33:0/2399735289 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565802cd350 0x5565802d1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.085+0000 7f6d537fe640 1 -- 172.21.15.33:0/2399735289 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55657ff040f0 con 0x5565802d5f30 2024-03-19T02:58:11.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.086+0000 7f6d537fe640 1 --2- 172.21.15.33:0/2399735289 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565802d5f30 0x5565802ce3a0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f6d44004af0 tx=0x7f6d44009a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10c2aebff7b31658 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:11.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.086+0000 7f6d48ff9640 1 -- 172.21.15.33:0/2399735289 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d4409c100 con 0x5565802d5f30 2024-03-19T02:58:11.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.086+0000 7f6d48ff9640 1 -- 172.21.15.33:0/2399735289 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6d440a41a0 con 0x5565802d5f30 2024-03-19T02:58:11.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.086+0000 7f6d48ff9640 1 -- 172.21.15.33:0/2399735289 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d440cf360 con 0x5565802d5f30 2024-03-19T02:58:11.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.086+0000 7f6d5c255740 1 -- 172.21.15.33:0/2399735289 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565802d5f30 msgr2=0x5565802ce3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:11.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.086+0000 7f6d5c255740 1 --2- 172.21.15.33:0/2399735289 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565802d5f30 0x5565802ce3a0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f6d44004af0 tx=0x7f6d44009a70 comp rx=0 tx=0).stop 2024-03-19T02:58:11.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.086+0000 7f6d5c255740 1 -- 172.21.15.33:0/2399735289 shutdown_connections 2024-03-19T02:58:11.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.086+0000 7f6d5c255740 1 --2- 172.21.15.33:0/2399735289 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565802d5f30 0x5565802ce3a0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.086+0000 7f6d5c255740 1 --2- 172.21.15.33:0/2399735289 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565802cd350 0x5565802d1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.086+0000 7f6d5c255740 1 --2- 172.21.15.33:0/2399735289 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5565802cca30 0x5565802d55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.086+0000 7f6d5c255740 1 -- 172.21.15.33:0/2399735289 >> 172.21.15.33:0/2399735289 conn(0x5565802c7160 msgr2=0x5565802c8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:11.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.087+0000 7f6d5c255740 1 -- 172.21.15.33:0/2399735289 shutdown_connections 2024-03-19T02:58:11.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.087+0000 7f6d5c255740 1 -- 172.21.15.33:0/2399735289 wait complete. 2024-03-19T02:58:11.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.087+0000 7f6d5c255740 1 -- start start 2024-03-19T02:58:11.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.087+0000 7f6d5c255740 1 Processor -- start 2024-03-19T02:58:11.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.087+0000 7f6d5c255740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565802cca30 0x5565802da560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:11.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.087+0000 7f6d5c255740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565802cd350 0x5565802daaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:11.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.087+0000 7f6d5c255740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5565802d5f30 0x5565803445c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:11.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.087+0000 7f6d5c255740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6d541af400 con 0x5565802cd350 2024-03-19T02:58:11.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.087+0000 7f6d5c255740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6d541b0870 con 0x5565802cca30 2024-03-19T02:58:11.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.087+0000 7f6d5c255740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6d541afad0 con 0x5565802d5f30 2024-03-19T02:58:11.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.087+0000 7f6d5c255740 10 client.0 _ll_register_callbacks cb 0x7f6d3c00f5e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:58:11.094 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.087+0000 7f6d527fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565802cd350 0x5565802daaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:11.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.087+0000 7f6d527fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565802cd350 0x5565802daaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:49970/0 (socket says 172.21.15.33:49970) 2024-03-19T02:58:11.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.087+0000 7f6d527fc640 1 -- 172.21.15.33:0/1490362764 learned_addr learned my addr 172.21.15.33:0/1490362764 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:11.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.087+0000 7f6d537fe640 1 --2- 172.21.15.33:0/1490362764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5565802d5f30 0x5565803445c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:11.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.087+0000 7f6d52ffd640 1 --2- 172.21.15.33:0/1490362764 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565802cca30 0x5565802da560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:11.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.087+0000 7f6d527fc640 1 -- 172.21.15.33:0/1490362764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5565802d5f30 msgr2=0x5565803445c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:11.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.087+0000 7f6d527fc640 1 --2- 172.21.15.33:0/1490362764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5565802d5f30 0x5565803445c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.087+0000 7f6d527fc640 1 -- 172.21.15.33:0/1490362764 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565802cca30 msgr2=0x5565802da560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:11.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.087+0000 7f6d527fc640 1 --2- 172.21.15.33:0/1490362764 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565802cca30 0x5565802da560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.087+0000 7f6d527fc640 1 -- 172.21.15.33:0/1490362764 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556580344c80 con 0x5565802cd350 2024-03-19T02:58:11.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.088+0000 7f6d527fc640 1 --2- 172.21.15.33:0/1490362764 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565802cd350 0x5565802daaa0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x5565802d26f0 tx=0x7f6d3c00feb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a4ad18562d83db6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:11.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.088+0000 7f6d3a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:58:11.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.088+0000 7f6d3a7fc640 1 -- 172.21.15.33:0/1490362764 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d3c017040 con 0x5565802cd350 2024-03-19T02:58:11.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.088+0000 7f6d3a7fc640 1 -- 172.21.15.33:0/1490362764 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f6d3c004660 con 0x5565802cd350 2024-03-19T02:58:11.095 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.088+0000 7f6d5c255740 1 -- 172.21.15.33:0/1490362764 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d440a41a0 con 0x5565802cd350 2024-03-19T02:58:11.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.088+0000 7f6d3a7fc640 1 -- 172.21.15.33:0/1490362764 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f6d3c01e820 con 0x5565802cd350 2024-03-19T02:58:11.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.088+0000 7f6d5c255740 10 client.16722 fetch_fsmap learned FSMap version 25 2024-03-19T02:58:11.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.088+0000 7f6d5c255740 1 -- 172.21.15.33:0/1490362764 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55658034d130 con 0x5565802cd350 2024-03-19T02:58:11.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.088+0000 7f6d3a7fc640 1 -- 172.21.15.33:0/1490362764 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6d440a41a0 con 0x5565802cd350 2024-03-19T02:58:11.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.088+0000 7f6d3a7fc640 1 -- 172.21.15.33:0/1490362764 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f6d3c025040 con 0x5565802cd350 2024-03-19T02:58:11.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.088+0000 7f6d5c255740 10 client.16722 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:58:11.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.088+0000 7f6d5c255740 10 client.16722 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:58:11.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.088+0000 7f6d5c255740 20 client.16722 populate_metadata read hostname 'smithi033' 2024-03-19T02:58:11.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.088+0000 7f6d5c255740 1 --2- 172.21.15.33:0/1490362764 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55658034d900 0x55658034fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:11.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.088+0000 7f6d5c255740 4 client.16722 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:58:11.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.088+0000 7f6d5c255740 1 -- 172.21.15.33:0/1490362764 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556580237d90 con 0x55658034d900 2024-03-19T02:58:11.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.088+0000 7f6d52ffd640 1 --2- 172.21.15.33:0/1490362764 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55658034d900 0x55658034fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:11.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.089+0000 7f6d3a7fc640 10 client.16722 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:58:11.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.089+0000 7f6d52ffd640 1 --2- 172.21.15.33:0/1490362764 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55658034d900 0x55658034fd20 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=286ebbcd5a1cdd07 server_cookie=768e9163b99df4c3 in_seq=0 out_seq=0 2024-03-19T02:58:11.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.089+0000 7f6d3a7fc640 1 -- 172.21.15.33:0/1490362764 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x556580237d90 con 0x55658034d900 2024-03-19T02:58:11.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.089+0000 7f6d3a7fc640 10 client.16722 handle_command_reply: tid=0 2024-03-19T02:58:11.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.089+0000 7f6d5c255740 1 client.16722 shutdown 2024-03-19T02:58:11.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.089+0000 7f6d5c255740 1 -- 172.21.15.33:0/1490362764 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565802cd350 msgr2=0x5565802daaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:11.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.089+0000 7f6d5c255740 1 --2- 172.21.15.33:0/1490362764 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565802cd350 0x5565802daaa0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x5565802d26f0 tx=0x7f6d3c00feb0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.089+0000 7f6d5c255740 1 -- 172.21.15.33:0/1490362764 shutdown_connections 2024-03-19T02:58:11.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.089+0000 7f6d5c255740 1 --2- 172.21.15.33:0/1490362764 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55658034d900 0x55658034fd20 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.089+0000 7f6d5c255740 1 --2- 172.21.15.33:0/1490362764 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5565802d5f30 0x5565803445c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.089+0000 7f6d5c255740 1 --2- 172.21.15.33:0/1490362764 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5565802cd350 0x5565802daaa0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.089+0000 7f6d3a7fc640 0 client.16722 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:58:11.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.089+0000 7f6d5c255740 1 --2- 172.21.15.33:0/1490362764 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5565802cca30 0x5565802da560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.089+0000 7f6d5c255740 1 -- 172.21.15.33:0/1490362764 >> 172.21.15.33:0/1490362764 conn(0x5565802c7160 msgr2=0x5565802d2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:11.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.089+0000 7f6d5c255740 1 -- 172.21.15.33:0/1490362764 shutdown_connections 2024-03-19T02:58:11.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.089+0000 7f6d5c255740 1 -- 172.21.15.33:0/1490362764 wait complete. 2024-03-19T02:58:11.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.089+0000 7f6d5c255740 20 client.16722 trim_cache size 0 max 16384 2024-03-19T02:58:11.098 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:58:11.098 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:58:11.098 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-03-19T02:58:11.098 INFO:teuthology.orchestra.run.smithi033.stdout: "79f16656-534d-45ee-ac8f-cc742298ea88": { 2024-03-19T02:58:11.098 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-03-19T02:58:11.098 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "79f16656-534d-45ee-ac8f-cc742298ea88", 2024-03-19T02:58:11.098 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-03-19T02:58:11.098 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T02:58:11.098 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T02:58:11.098 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:58:11.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.090+0000 7f6d59661640 1 -- 172.21.15.33:0/1298873780 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6d400a29a0 msgr2=0x7f6d400a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:11.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.090+0000 7f6d59661640 1 --2- 172.21.15.33:0/1298873780 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6d400a29a0 0x7f6d400a4e60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6d54103ca0 tx=0x7f6d3c011810 comp rx=0 tx=0).stop 2024-03-19T02:58:11.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.090+0000 7f6d59661640 1 -- 172.21.15.33:0/1298873780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6d541055c0 msgr2=0x7f6d54102ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:11.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.090+0000 7f6d59661640 1 --2- 172.21.15.33:0/1298873780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6d541055c0 0x7f6d54102ce0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f6d4400eb10 tx=0x7f6d4400c2d0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.090+0000 7f6d59661640 1 -- 172.21.15.33:0/1298873780 shutdown_connections 2024-03-19T02:58:11.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.090+0000 7f6d59661640 1 --2- 172.21.15.33:0/1298873780 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6d400a29a0 0x7f6d400a4e60 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.090+0000 7f6d59661640 1 --2- 172.21.15.33:0/1298873780 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6d541055c0 0x7f6d54102ce0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.090+0000 7f6d59661640 1 --2- 172.21.15.33:0/1298873780 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d54101080 0x7f6d541027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.090+0000 7f6d59661640 1 --2- 172.21.15.33:0/1298873780 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d541006e0 0x7f6d54102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:11.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.090+0000 7f6d59661640 1 -- 172.21.15.33:0/1298873780 >> 172.21.15.33:0/1298873780 conn(0x7f6d540777b0 msgr2=0x7f6d54069ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:11.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.090+0000 7f6d59661640 1 -- 172.21.15.33:0/1298873780 shutdown_connections 2024-03-19T02:58:11.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:11.090+0000 7f6d59661640 1 -- 172.21.15.33:0/1298873780 wait complete. 2024-03-19T02:58:11.102 INFO:tasks.cephfs.filesystem:scrub status for tag:79f16656-534d-45ee-ac8f-cc742298ea88 - {'path': '/', 'tag': '79f16656-534d-45ee-ac8f-cc742298ea88', 'options': 'recursive,force'} 2024-03-19T02:58:12.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:12 smithi155 ceph-mon[35804]: scrub queued for path: / 2024-03-19T02:58:12.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:12 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T02:58:12.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:12 smithi155 ceph-mon[35804]: scrub summary: active paths [/] 2024-03-19T02:58:12.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:12 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T02:58:12.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:12 smithi155 ceph-mon[35804]: pgmap v145: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-19T02:58:12.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:12 smithi033 ceph-mon[31973]: scrub queued for path: / 2024-03-19T02:58:12.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:12 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T02:58:12.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:12 smithi033 ceph-mon[31973]: scrub summary: active paths [/] 2024-03-19T02:58:12.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:12 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T02:58:12.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:12 smithi033 ceph-mon[31973]: pgmap v145: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-19T02:58:12.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:12 smithi096 ceph-mon[35946]: scrub queued for path: / 2024-03-19T02:58:12.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:12 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T02:58:12.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:12 smithi096 ceph-mon[35946]: scrub summary: active paths [/] 2024-03-19T02:58:12.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:12 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T02:58:12.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:12 smithi096 ceph-mon[35946]: pgmap v145: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-19T02:58:13.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:13 smithi155 ceph-mon[35804]: scrub summary: idle 2024-03-19T02:58:13.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:13 smithi033 ceph-mon[31973]: scrub summary: idle 2024-03-19T02:58:13.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:13 smithi096 ceph-mon[35946]: scrub summary: idle 2024-03-19T02:58:14.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:14 smithi155 ceph-mon[35804]: pgmap v146: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-19T02:58:14.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:14 smithi033 ceph-mon[31973]: pgmap v146: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-19T02:58:14.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:14 smithi096 ceph-mon[35946]: pgmap v146: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-19T02:58:16.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:16 smithi155 ceph-mon[35804]: pgmap v147: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-19T02:58:16.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:16 smithi033 ceph-mon[31973]: pgmap v147: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-19T02:58:16.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:16 smithi096 ceph-mon[35946]: pgmap v147: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-19T02:58:18.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:18 smithi155 ceph-mon[35804]: pgmap v148: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-19T02:58:18.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:18 smithi033 ceph-mon[31973]: pgmap v148: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-19T02:58:18.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:18 smithi096 ceph-mon[35946]: pgmap v148: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-19T02:58:20.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:20 smithi155 ceph-mon[35804]: pgmap v149: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-19T02:58:20.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:20 smithi033 ceph-mon[31973]: pgmap v149: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-19T02:58:20.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:20 smithi096 ceph-mon[35946]: pgmap v149: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-19T02:58:22.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:22 smithi155 ceph-mon[35804]: pgmap v150: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:58:22.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:22 smithi033 ceph-mon[31973]: pgmap v150: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:58:22.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:22 smithi096 ceph-mon[35946]: pgmap v150: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:58:24.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:24 smithi155 ceph-mon[35804]: pgmap v151: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-03-19T02:58:24.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:24 smithi033 ceph-mon[31973]: pgmap v151: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-03-19T02:58:24.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:24 smithi096 ceph-mon[35946]: pgmap v151: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-03-19T02:58:26.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:26 smithi155 ceph-mon[35804]: pgmap v152: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:58:26.767 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:26 smithi096 ceph-mon[35946]: pgmap v152: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:58:26.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:26 smithi033 ceph-mon[31973]: pgmap v152: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:58:28.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:28 smithi155 ceph-mon[35804]: pgmap v153: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:58:28.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:28 smithi033 ceph-mon[31973]: pgmap v153: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:58:28.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:28 smithi096 ceph-mon[35946]: pgmap v153: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:58:30.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:30 smithi155 ceph-mon[35804]: pgmap v154: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:58:30.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:30 smithi033 ceph-mon[31973]: pgmap v154: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:58:30.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:30 smithi096 ceph-mon[35946]: pgmap v154: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:58:32.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:32 smithi033 ceph-mon[31973]: pgmap v155: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:58:32.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:32 smithi096 ceph-mon[35946]: pgmap v155: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:58:33.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:32 smithi155 ceph-mon[35804]: pgmap v155: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:58:34.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:34 smithi033 ceph-mon[31973]: pgmap v156: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:58:34.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:34 smithi096 ceph-mon[35946]: pgmap v156: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:58:35.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:34 smithi155 ceph-mon[35804]: pgmap v156: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:58:36.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:36 smithi033 ceph-mon[31973]: pgmap v157: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:58:36.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:36 smithi096 ceph-mon[35946]: pgmap v157: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:58:37.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:36 smithi155 ceph-mon[35804]: pgmap v157: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:58:37.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:37 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:58:37.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:37 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:58:37.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:37 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:58:37.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:37 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:58:37.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:37 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:58:37.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:37 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:58:37.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:37 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:58:37.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:37 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:58:38.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:37 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:58:38.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:37 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:58:38.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:37 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:58:38.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:37 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:58:38.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:38 smithi033 ceph-mon[31973]: pgmap v158: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:58:38.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:38 smithi096 ceph-mon[35946]: pgmap v158: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:58:39.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:38 smithi155 ceph-mon[35804]: pgmap v158: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:58:40.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:40 smithi096 ceph-mon[35946]: pgmap v159: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:58:41.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:40 smithi155 ceph-mon[35804]: pgmap v159: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:58:41.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:40 smithi033 ceph-mon[31973]: pgmap v159: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:58:41.102 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:58:41.103 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:58:41.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.219+0000 7f480ff3b640 1 -- 172.21.15.33:0/3845219220 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48081033b0 msgr2=0x7f480810b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:41.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.219+0000 7f480ff3b640 1 --2- 172.21.15.33:0/3845219220 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48081033b0 0x7f480810b890 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f47f8009900 tx=0x7f47f802bcd0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.219+0000 7f480ff3b640 1 -- 172.21.15.33:0/3845219220 shutdown_connections 2024-03-19T02:58:41.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.219+0000 7f480ff3b640 1 --2- 172.21.15.33:0/3845219220 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f480810be60 0x7f480810e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.219+0000 7f480ff3b640 1 --2- 172.21.15.33:0/3845219220 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48081033b0 0x7f480810b890 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.219+0000 7f480ff3b640 1 --2- 172.21.15.33:0/3845219220 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4808102a90 0x7f4808102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.219+0000 7f480ff3b640 1 -- 172.21.15.33:0/3845219220 >> 172.21.15.33:0/3845219220 conn(0x7f4808077880 msgr2=0x7f4808077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:41.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.219+0000 7f480ff3b640 1 -- 172.21.15.33:0/3845219220 shutdown_connections 2024-03-19T02:58:41.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.219+0000 7f480ff3b640 1 -- 172.21.15.33:0/3845219220 wait complete. 2024-03-19T02:58:41.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.220+0000 7f480ff3b640 1 Processor -- start 2024-03-19T02:58:41.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.220+0000 7f480ff3b640 1 -- start start 2024-03-19T02:58:41.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.220+0000 7f480ff3b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4808102a90 0x7f4808198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.220+0000 7f480ff3b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48081033b0 0x7f4808198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.220+0000 7f480ff3b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f480810be60 0x7f480819c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.220+0000 7f480ff3b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4808111ae0 con 0x7f48081033b0 2024-03-19T02:58:41.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.220+0000 7f480ff3b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4808111960 con 0x7f4808102a90 2024-03-19T02:58:41.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.220+0000 7f480ff3b640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4808111c60 con 0x7f480810be60 2024-03-19T02:58:41.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.220+0000 7f480d4d6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48081033b0 0x7f4808198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.220+0000 7f480d4d6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48081033b0 0x7f4808198580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53648/0 (socket says 172.21.15.33:53648) 2024-03-19T02:58:41.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.220+0000 7f480d4d6640 1 -- 172.21.15.33:0/129192280 learned_addr learned my addr 172.21.15.33:0/129192280 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:41.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.220+0000 7f480dcd7640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4808102a90 0x7f4808198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.220+0000 7f480e4d8640 1 --2- 172.21.15.33:0/129192280 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f480810be60 0x7f480819c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.220+0000 7f480d4d6640 1 -- 172.21.15.33:0/129192280 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f480810be60 msgr2=0x7f480819c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:41.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.220+0000 7f480d4d6640 1 --2- 172.21.15.33:0/129192280 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f480810be60 0x7f480819c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.220+0000 7f480d4d6640 1 -- 172.21.15.33:0/129192280 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4808102a90 msgr2=0x7f4808198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:41.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.220+0000 7f480d4d6640 1 --2- 172.21.15.33:0/129192280 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4808102a90 0x7f4808198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.220+0000 7f480d4d6640 1 -- 172.21.15.33:0/129192280 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f480819d030 con 0x7f48081033b0 2024-03-19T02:58:41.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.220+0000 7f480dcd7640 1 --2- 172.21.15.33:0/129192280 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4808102a90 0x7f4808198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:58:41.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.221+0000 7f480d4d6640 1 --2- 172.21.15.33:0/129192280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48081033b0 0x7f4808198580 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f47f8005940 tx=0x7f47f802c690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:41.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.221+0000 7f47feffd640 1 -- 172.21.15.33:0/129192280 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f8043020 con 0x7f48081033b0 2024-03-19T02:58:41.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.221+0000 7f480ff3b640 1 -- 172.21.15.33:0/129192280 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f480819d2c0 con 0x7f48081033b0 2024-03-19T02:58:41.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.221+0000 7f47feffd640 1 -- 172.21.15.33:0/129192280 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47f8004e00 con 0x7f48081033b0 2024-03-19T02:58:41.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.221+0000 7f47feffd640 1 -- 172.21.15.33:0/129192280 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f8003970 con 0x7f48081033b0 2024-03-19T02:58:41.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.221+0000 7f480ff3b640 1 -- 172.21.15.33:0/129192280 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48081a4bb0 con 0x7f48081033b0 2024-03-19T02:58:41.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.223+0000 7f47feffd640 1 -- 172.21.15.33:0/129192280 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f47f8046020 con 0x7f48081033b0 2024-03-19T02:58:41.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.224+0000 7f47feffd640 1 --2- 172.21.15.33:0/129192280 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f47f40a2970 0x7f47f40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.224+0000 7f4812c01740 1 Processor -- start 2024-03-19T02:58:41.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.224+0000 7f480dcd7640 1 --2- 172.21.15.33:0/129192280 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f47f40a2970 0x7f47f40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.224+0000 7f47feffd640 1 -- 172.21.15.33:0/129192280 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f47f80f1080 con 0x7f48081033b0 2024-03-19T02:58:41.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.224+0000 7f4812c01740 1 -- start start 2024-03-19T02:58:41.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.224+0000 7f4812c01740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f682d8a30 0x557f682d8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.224+0000 7f4812c01740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f682d9350 0x557f682e19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.224+0000 7f4812c01740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f682e1f30 0x557f682e4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.224+0000 7f4812c01740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557f6823c500 con 0x557f682e1f30 2024-03-19T02:58:41.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.224+0000 7f4812c01740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557f681e4be0 con 0x557f682d8a30 2024-03-19T02:58:41.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.224+0000 7f4812c01740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x557f682385a0 con 0x557f682d9350 2024-03-19T02:58:41.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.224+0000 7f480dcd7640 1 --2- 172.21.15.33:0/129192280 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f47f40a2970 0x7f47f40a4e30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f48040095f0 tx=0x7f4804005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:41.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.225+0000 7f480dcd7640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f682d8a30 0x557f682d8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.225+0000 7f480e4d8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f682e1f30 0x557f682e4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.225+0000 7f480d4d6640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f682d9350 0x557f682e19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.225+0000 7f480e4d8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f682e1f30 0x557f682e4320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53656/0 (socket says 172.21.15.33:53656) 2024-03-19T02:58:41.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.225+0000 7f480e4d8640 1 -- 172.21.15.33:0/3887014279 learned_addr learned my addr 172.21.15.33:0/3887014279 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:41.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.225+0000 7f480e4d8640 1 -- 172.21.15.33:0/3887014279 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f682d9350 msgr2=0x557f682e19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:41.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.225+0000 7f480e4d8640 1 --2- 172.21.15.33:0/3887014279 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f682d9350 0x557f682e19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.225+0000 7f480e4d8640 1 -- 172.21.15.33:0/3887014279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f682d8a30 msgr2=0x557f682d8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:41.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.225+0000 7f480e4d8640 1 --2- 172.21.15.33:0/3887014279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f682d8a30 0x557f682d8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.225+0000 7f480e4d8640 1 -- 172.21.15.33:0/3887014279 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f68206000 con 0x557f682e1f30 2024-03-19T02:58:41.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.225+0000 7f480dcd7640 1 --2- 172.21.15.33:0/3887014279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f682d8a30 0x557f682d8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:58:41.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.225+0000 7f480e4d8640 1 --2- 172.21.15.33:0/3887014279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f682e1f30 0x557f682e4320 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f4800009f50 tx=0x7f48000118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7028c164c6c0517d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:41.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.225+0000 7f47f3fff640 1 -- 172.21.15.33:0/3887014279 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4800004d20 con 0x557f682e1f30 2024-03-19T02:58:41.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.225+0000 7f47f3fff640 1 -- 172.21.15.33:0/3887014279 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4800004050 con 0x557f682e1f30 2024-03-19T02:58:41.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.225+0000 7f47f3fff640 1 -- 172.21.15.33:0/3887014279 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4800005280 con 0x557f682e1f30 2024-03-19T02:58:41.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.226+0000 7f4812c01740 1 -- 172.21.15.33:0/3887014279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f682e1f30 msgr2=0x557f682e4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:41.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.226+0000 7f4812c01740 1 --2- 172.21.15.33:0/3887014279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f682e1f30 0x557f682e4320 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f4800009f50 tx=0x7f48000118d0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.226+0000 7f4812c01740 1 -- 172.21.15.33:0/3887014279 shutdown_connections 2024-03-19T02:58:41.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.226+0000 7f4812c01740 1 --2- 172.21.15.33:0/3887014279 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f682d9350 0x557f682e19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.226+0000 7f4812c01740 1 --2- 172.21.15.33:0/3887014279 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f682e1f30 0x557f682e4320 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.226+0000 7f4812c01740 1 --2- 172.21.15.33:0/3887014279 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f682d8a30 0x557f682d8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.226+0000 7f4812c01740 1 -- 172.21.15.33:0/3887014279 >> 172.21.15.33:0/3887014279 conn(0x557f682d3160 msgr2=0x557f682d3540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:41.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.226+0000 7f4812c01740 1 -- 172.21.15.33:0/3887014279 shutdown_connections 2024-03-19T02:58:41.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.226+0000 7f4812c01740 1 -- 172.21.15.33:0/3887014279 wait complete. 2024-03-19T02:58:41.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.226+0000 7f4812c01740 1 -- start start 2024-03-19T02:58:41.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.226+0000 7f4812c01740 1 Processor -- start 2024-03-19T02:58:41.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.227+0000 7f4812c01740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f682d8a30 0x557f682dee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.227+0000 7f4812c01740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f682d9350 0x557f682df3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.227+0000 7f4812c01740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f682e1f30 0x557f682df910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.227+0000 7f4812c01740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557f682e4d20 con 0x557f682e1f30 2024-03-19T02:58:41.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.227+0000 7f4812c01740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557f681f8160 con 0x557f682d9350 2024-03-19T02:58:41.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.227+0000 7f4812c01740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x557f682e4ea0 con 0x557f682d8a30 2024-03-19T02:58:41.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.227+0000 7f4812c01740 10 client.0 _ll_register_callbacks cb 0x557f68234a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:58:41.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.227+0000 7f480e4d8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f682e1f30 0x557f682df910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.227+0000 7f480d4d6640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f682d9350 0x557f682df3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.227+0000 7f480e4d8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f682e1f30 0x557f682df910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53666/0 (socket says 172.21.15.33:53666) 2024-03-19T02:58:41.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.227+0000 7f480e4d8640 1 -- 172.21.15.33:0/749109382 learned_addr learned my addr 172.21.15.33:0/749109382 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:41.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.227+0000 7f480d4d6640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f682d9350 0x557f682df3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:56990/0 (socket says 172.21.15.33:56990) 2024-03-19T02:58:41.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.227+0000 7f480dcd7640 1 --2- 172.21.15.33:0/749109382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f682d8a30 0x557f682dee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.227+0000 7f480e4d8640 1 -- 172.21.15.33:0/749109382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f682d8a30 msgr2=0x557f682dee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:41.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.227+0000 7f480e4d8640 1 --2- 172.21.15.33:0/749109382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f682d8a30 0x557f682dee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.227+0000 7f480e4d8640 1 -- 172.21.15.33:0/749109382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f682d9350 msgr2=0x557f682df3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:41.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.227+0000 7f480e4d8640 1 --2- 172.21.15.33:0/749109382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f682d9350 0x557f682df3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.227+0000 7f480e4d8640 1 -- 172.21.15.33:0/749109382 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f6801a7d0 con 0x557f682e1f30 2024-03-19T02:58:41.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.227+0000 7f480d4d6640 1 --2- 172.21.15.33:0/749109382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f682d9350 0x557f682df3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:58:41.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.228+0000 7f480e4d8640 1 --2- 172.21.15.33:0/749109382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f682e1f30 0x557f682df910 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f4800012fd0 tx=0x7f4800003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=478527c64fd92131 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:41.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.228+0000 7f47f17fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:58:41.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.228+0000 7f47f17fa640 1 -- 172.21.15.33:0/749109382 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4800028020 con 0x557f682e1f30 2024-03-19T02:58:41.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.228+0000 7f47f17fa640 1 -- 172.21.15.33:0/749109382 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f4800005050 con 0x557f682e1f30 2024-03-19T02:58:41.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.228+0000 7f4812c01740 1 -- 172.21.15.33:0/749109382 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4800004050 con 0x557f682e1f30 2024-03-19T02:58:41.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.228+0000 7f47f17fa640 1 -- 172.21.15.33:0/749109382 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4800009be0 con 0x557f682e1f30 2024-03-19T02:58:41.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.228+0000 7f4812c01740 10 client.16734 fetch_fsmap learned FSMap version 25 2024-03-19T02:58:41.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.228+0000 7f4812c01740 1 -- 172.21.15.33:0/749109382 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x557f67f503f0 con 0x557f682e1f30 2024-03-19T02:58:41.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.228+0000 7f47f17fa640 1 -- 172.21.15.33:0/749109382 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4800004050 con 0x557f682e1f30 2024-03-19T02:58:41.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.228+0000 7f47f17fa640 1 -- 172.21.15.33:0/749109382 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f48000244f0 con 0x557f682e1f30 2024-03-19T02:58:41.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.228+0000 7f4812c01740 10 client.16734 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:58:41.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.228+0000 7f4812c01740 10 client.16734 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:58:41.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.228+0000 7f4812c01740 20 client.16734 populate_metadata read hostname 'smithi033' 2024-03-19T02:58:41.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.228+0000 7f4812c01740 1 --2- 172.21.15.33:0/749109382 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557f682f2180 0x557f682f4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.228+0000 7f4812c01740 4 client.16734 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:58:41.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.228+0000 7f4812c01740 1 -- 172.21.15.33:0/749109382 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f68249a40 con 0x557f682f2180 2024-03-19T02:58:41.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.229+0000 7f480dcd7640 1 --2- 172.21.15.33:0/749109382 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557f682f2180 0x557f682f4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.229+0000 7f480dcd7640 1 --2- 172.21.15.33:0/749109382 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557f682f2180 0x557f682f4570 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c29cccc27e34e29 server_cookie=c264aad634883bbf in_seq=0 out_seq=0 2024-03-19T02:58:41.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.229+0000 7f47f17fa640 10 client.16734 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:58:41.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.230+0000 7f47f17fa640 1 -- 172.21.15.33:0/749109382 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557f68249a40 con 0x557f682f2180 2024-03-19T02:58:41.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.230+0000 7f47f17fa640 10 client.16734 handle_command_reply: tid=0 2024-03-19T02:58:41.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.230+0000 7f4812c01740 1 client.16734 shutdown 2024-03-19T02:58:41.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.230+0000 7f4812c01740 1 -- 172.21.15.33:0/749109382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f682e1f30 msgr2=0x557f682df910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:41.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.230+0000 7f4812c01740 1 --2- 172.21.15.33:0/749109382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f682e1f30 0x557f682df910 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f4800012fd0 tx=0x7f4800003ce0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.230+0000 7f4812c01740 1 -- 172.21.15.33:0/749109382 shutdown_connections 2024-03-19T02:58:41.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.230+0000 7f4812c01740 1 --2- 172.21.15.33:0/749109382 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557f682f2180 0x557f682f4570 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.230+0000 7f4812c01740 1 --2- 172.21.15.33:0/749109382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f682e1f30 0x557f682df910 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.230+0000 7f4812c01740 1 --2- 172.21.15.33:0/749109382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f682d9350 0x557f682df3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.230+0000 7f47f17fa640 0 client.16734 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:58:41.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.230+0000 7f4812c01740 1 --2- 172.21.15.33:0/749109382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f682d8a30 0x557f682dee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.230+0000 7f4812c01740 1 -- 172.21.15.33:0/749109382 >> 172.21.15.33:0/749109382 conn(0x557f682d3160 msgr2=0x557f682d4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:41.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.230+0000 7f4812c01740 1 -- 172.21.15.33:0/749109382 shutdown_connections 2024-03-19T02:58:41.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.230+0000 7f4812c01740 1 -- 172.21.15.33:0/749109382 wait complete. 2024-03-19T02:58:41.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.230+0000 7f4812c01740 20 client.16734 trim_cache size 0 max 16384 2024-03-19T02:58:41.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.248+0000 7f4812c01740 1 Processor -- start 2024-03-19T02:58:41.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.248+0000 7f4812c01740 1 -- start start 2024-03-19T02:58:41.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.248+0000 7f4812c01740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f682d8a30 0x557f682edac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.248+0000 7f4812c01740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f682d9350 0x557f682d21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.248+0000 7f4812c01740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f682e1f30 0x557f682df730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.248+0000 7f4812c01740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f48081b14c0 con 0x557f682d9350 2024-03-19T02:58:41.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.248+0000 7f4812c01740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f48081aed90 con 0x557f682d8a30 2024-03-19T02:58:41.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.248+0000 7f4812c01740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f48081acbc0 con 0x557f682e1f30 2024-03-19T02:58:41.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.248+0000 7f480d4d6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f682d9350 0x557f682d21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.248+0000 7f480d4d6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f682d9350 0x557f682d21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53682/0 (socket says 172.21.15.33:53682) 2024-03-19T02:58:41.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.248+0000 7f480d4d6640 1 -- 172.21.15.33:0/1546695584 learned_addr learned my addr 172.21.15.33:0/1546695584 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:41.255 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.248+0000 7f480dcd7640 1 --2- 172.21.15.33:0/1546695584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f682d8a30 0x557f682edac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.248+0000 7f480e4d8640 1 --2- 172.21.15.33:0/1546695584 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f682e1f30 0x557f682df730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.248+0000 7f480d4d6640 1 -- 172.21.15.33:0/1546695584 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f682e1f30 msgr2=0x557f682df730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:41.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.248+0000 7f480d4d6640 1 --2- 172.21.15.33:0/1546695584 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f682e1f30 0x557f682df730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.248+0000 7f480d4d6640 1 -- 172.21.15.33:0/1546695584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f682d8a30 msgr2=0x557f682edac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:41.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.248+0000 7f480d4d6640 1 --2- 172.21.15.33:0/1546695584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f682d8a30 0x557f682edac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.248+0000 7f480d4d6640 1 -- 172.21.15.33:0/1546695584 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f682d9180 con 0x557f682d9350 2024-03-19T02:58:41.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.249+0000 7f480d4d6640 1 --2- 172.21.15.33:0/1546695584 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f682d9350 0x557f682d21e0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f47f80bd140 tx=0x7f47f8041040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e11722b46775dd0d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:41.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.249+0000 7f47f37fe640 1 -- 172.21.15.33:0/1546695584 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f80bd440 con 0x557f682d9350 2024-03-19T02:58:41.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.249+0000 7f47f37fe640 1 -- 172.21.15.33:0/1546695584 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47f80c1590 con 0x557f682d9350 2024-03-19T02:58:41.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.249+0000 7f47f37fe640 1 -- 172.21.15.33:0/1546695584 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f80ee6a0 con 0x557f682d9350 2024-03-19T02:58:41.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.249+0000 7f4812c01740 1 -- 172.21.15.33:0/1546695584 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f682d9350 msgr2=0x557f682d21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:41.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.249+0000 7f4812c01740 1 --2- 172.21.15.33:0/1546695584 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f682d9350 0x557f682d21e0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f47f80bd140 tx=0x7f47f8041040 comp rx=0 tx=0).stop 2024-03-19T02:58:41.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.249+0000 7f4812c01740 1 -- 172.21.15.33:0/1546695584 shutdown_connections 2024-03-19T02:58:41.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.250+0000 7f4812c01740 1 --2- 172.21.15.33:0/1546695584 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f682e1f30 0x557f682df730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.250+0000 7f4812c01740 1 --2- 172.21.15.33:0/1546695584 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f682d9350 0x557f682d21e0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.250+0000 7f4812c01740 1 --2- 172.21.15.33:0/1546695584 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f682d8a30 0x557f682edac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.250+0000 7f4812c01740 1 -- 172.21.15.33:0/1546695584 >> 172.21.15.33:0/1546695584 conn(0x557f682d3160 msgr2=0x557f682dddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:41.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.250+0000 7f4812c01740 1 -- 172.21.15.33:0/1546695584 shutdown_connections 2024-03-19T02:58:41.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.250+0000 7f4812c01740 1 -- 172.21.15.33:0/1546695584 wait complete. 2024-03-19T02:58:41.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.250+0000 7f4812c01740 1 -- start start 2024-03-19T02:58:41.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.250+0000 7f4812c01740 1 Processor -- start 2024-03-19T02:58:41.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.250+0000 7f4812c01740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f682d8a30 0x557f682ea240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.250+0000 7f4812c01740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f682d9350 0x557f682e5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.250+0000 7f4812c01740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f682e1f30 0x557f682e5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.250+0000 7f4812c01740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f48081ad250 con 0x557f682d8a30 2024-03-19T02:58:41.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.250+0000 7f4812c01740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f48081ae6c0 con 0x557f682e1f30 2024-03-19T02:58:41.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.250+0000 7f4812c01740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f48081ad920 con 0x557f682d9350 2024-03-19T02:58:41.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.250+0000 7f4812c01740 10 client.0 _ll_register_callbacks cb 0x7f4804009490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:58:41.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.250+0000 7f480dcd7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f682d8a30 0x557f682ea240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.250+0000 7f480dcd7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f682d8a30 0x557f682ea240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53684/0 (socket says 172.21.15.33:53684) 2024-03-19T02:58:41.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.250+0000 7f480dcd7640 1 -- 172.21.15.33:0/1863270193 learned_addr learned my addr 172.21.15.33:0/1863270193 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:41.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.251+0000 7f480e4d8640 1 --2- 172.21.15.33:0/1863270193 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f682e1f30 0x557f682e5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.251+0000 7f480d4d6640 1 --2- 172.21.15.33:0/1863270193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f682d9350 0x557f682e5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.251+0000 7f480dcd7640 1 -- 172.21.15.33:0/1863270193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f682d9350 msgr2=0x557f682e5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:41.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.251+0000 7f480dcd7640 1 --2- 172.21.15.33:0/1863270193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f682d9350 0x557f682e5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.251+0000 7f480dcd7640 1 -- 172.21.15.33:0/1863270193 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f682e1f30 msgr2=0x557f682e5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:41.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.251+0000 7f480dcd7640 1 --2- 172.21.15.33:0/1863270193 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f682e1f30 0x557f682e5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.251+0000 7f480dcd7640 1 -- 172.21.15.33:0/1863270193 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f67f100f0 con 0x557f682d8a30 2024-03-19T02:58:41.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.251+0000 7f480dcd7640 1 --2- 172.21.15.33:0/1863270193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f682d8a30 0x557f682ea240 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x557f682f4b60 tx=0x7f4804004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9928498fe34c5c45 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:41.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.251+0000 7f47f17fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:58:41.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.251+0000 7f47f17fa640 1 -- 172.21.15.33:0/1863270193 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f480402aa60 con 0x557f682d8a30 2024-03-19T02:58:41.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.251+0000 7f47f17fa640 1 -- 172.21.15.33:0/1863270193 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f480402ac00 con 0x557f682d8a30 2024-03-19T02:58:41.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.251+0000 7f4812c01740 1 -- 172.21.15.33:0/1863270193 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47f80c1590 con 0x557f682d8a30 2024-03-19T02:58:41.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.251+0000 7f47f17fa640 1 -- 172.21.15.33:0/1863270193 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f480403e050 con 0x557f682d8a30 2024-03-19T02:58:41.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.251+0000 7f4812c01740 10 client.16740 fetch_fsmap learned FSMap version 25 2024-03-19T02:58:41.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.251+0000 7f4812c01740 1 -- 172.21.15.33:0/1863270193 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x557f68358010 con 0x557f682d8a30 2024-03-19T02:58:41.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.251+0000 7f47f17fa640 1 -- 172.21.15.33:0/1863270193 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47f80c1590 con 0x557f682d8a30 2024-03-19T02:58:41.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.252+0000 7f47f17fa640 1 -- 172.21.15.33:0/1863270193 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f48040354b0 con 0x557f682d8a30 2024-03-19T02:58:41.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.252+0000 7f4812c01740 10 client.16740 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:58:41.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.252+0000 7f4812c01740 10 client.16740 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:58:41.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.252+0000 7f4812c01740 20 client.16740 populate_metadata read hostname 'smithi033' 2024-03-19T02:58:41.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.252+0000 7f4812c01740 1 --2- 172.21.15.33:0/1863270193 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557f68358770 0x557f6835ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.252+0000 7f4812c01740 4 client.16740 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:58:41.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.252+0000 7f4812c01740 1 -- 172.21.15.33:0/1863270193 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557f68243d90 con 0x557f68358770 2024-03-19T02:58:41.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.252+0000 7f480d4d6640 1 --2- 172.21.15.33:0/1863270193 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557f68358770 0x557f6835ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.252+0000 7f480d4d6640 1 --2- 172.21.15.33:0/1863270193 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557f68358770 0x557f6835ab90 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5eeac9d615288377 server_cookie=ff8f6a2dc2ce1393 in_seq=0 out_seq=0 2024-03-19T02:58:41.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.252+0000 7f47f17fa640 10 client.16740 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:58:41.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.252+0000 7f47f17fa640 1 -- 172.21.15.33:0/1863270193 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557f68243d90 con 0x557f68358770 2024-03-19T02:58:41.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.252+0000 7f47f17fa640 10 client.16740 handle_command_reply: tid=0 2024-03-19T02:58:41.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.252+0000 7f4812c01740 1 client.16740 shutdown 2024-03-19T02:58:41.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.252+0000 7f4812c01740 1 -- 172.21.15.33:0/1863270193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f682d8a30 msgr2=0x557f682ea240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:41.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.252+0000 7f4812c01740 1 --2- 172.21.15.33:0/1863270193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f682d8a30 0x557f682ea240 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x557f682f4b60 tx=0x7f4804004730 comp rx=0 tx=0).stop 2024-03-19T02:58:41.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.253+0000 7f4812c01740 1 -- 172.21.15.33:0/1863270193 shutdown_connections 2024-03-19T02:58:41.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.253+0000 7f4812c01740 1 --2- 172.21.15.33:0/1863270193 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557f68358770 0x557f6835ab90 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.253+0000 7f4812c01740 1 --2- 172.21.15.33:0/1863270193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f682d9350 0x557f682e5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.253+0000 7f4812c01740 1 --2- 172.21.15.33:0/1863270193 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f682e1f30 0x557f682e5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.253+0000 7f4812c01740 1 --2- 172.21.15.33:0/1863270193 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f682d8a30 0x557f682ea240 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.253+0000 7f47f17fa640 0 client.16740 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:58:41.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.253+0000 7f4812c01740 1 -- 172.21.15.33:0/1863270193 >> 172.21.15.33:0/1863270193 conn(0x557f682d3160 msgr2=0x557f682d21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:41.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.253+0000 7f4812c01740 1 -- 172.21.15.33:0/1863270193 shutdown_connections 2024-03-19T02:58:41.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.253+0000 7f4812c01740 1 -- 172.21.15.33:0/1863270193 wait complete. 2024-03-19T02:58:41.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.253+0000 7f4812c01740 20 client.16740 trim_cache size 0 max 16384 2024-03-19T02:58:41.261 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:58:41.261 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-03-19T02:58:41.261 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-03-19T02:58:41.261 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:58:41.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.253+0000 7f480ff3b640 1 -- 172.21.15.33:0/129192280 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f47f40a2970 msgr2=0x7f47f40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:41.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.253+0000 7f480ff3b640 1 --2- 172.21.15.33:0/129192280 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f47f40a2970 0x7f47f40a4e30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f48040095f0 tx=0x7f4804005f90 comp rx=0 tx=0).stop 2024-03-19T02:58:41.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.253+0000 7f480ff3b640 1 -- 172.21.15.33:0/129192280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48081033b0 msgr2=0x7f4808198580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:41.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.253+0000 7f480ff3b640 1 --2- 172.21.15.33:0/129192280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48081033b0 0x7f4808198580 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f47f8005940 tx=0x7f47f802c690 comp rx=0 tx=0).stop 2024-03-19T02:58:41.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.253+0000 7f480ff3b640 1 -- 172.21.15.33:0/129192280 shutdown_connections 2024-03-19T02:58:41.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.253+0000 7f480ff3b640 1 --2- 172.21.15.33:0/129192280 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f47f40a2970 0x7f47f40a4e30 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.253+0000 7f480ff3b640 1 --2- 172.21.15.33:0/129192280 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f480810be60 0x7f480819c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.253+0000 7f480ff3b640 1 --2- 172.21.15.33:0/129192280 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f48081033b0 0x7f4808198580 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.253+0000 7f480ff3b640 1 --2- 172.21.15.33:0/129192280 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4808102a90 0x7f4808198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.253+0000 7f480ff3b640 1 -- 172.21.15.33:0/129192280 >> 172.21.15.33:0/129192280 conn(0x7f4808077880 msgr2=0x7f48080fd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:41.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.253+0000 7f480ff3b640 1 -- 172.21.15.33:0/129192280 shutdown_connections 2024-03-19T02:58:41.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.253+0000 7f480ff3b640 1 -- 172.21.15.33:0/129192280 wait complete. 2024-03-19T02:58:41.267 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:58:41.267 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:58:41.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.369+0000 7fa950ca1640 1 -- 172.21.15.33:0/987045899 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa94c100720 msgr2=0x7fa94c100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:41.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.369+0000 7fa950ca1640 1 --2- 172.21.15.33:0/987045899 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa94c100720 0x7fa94c100ba0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa934009860 tx=0x7fa93402bb70 comp rx=0 tx=0).stop 2024-03-19T02:58:41.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.369+0000 7fa950ca1640 1 -- 172.21.15.33:0/987045899 shutdown_connections 2024-03-19T02:58:41.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.369+0000 7fa950ca1640 1 --2- 172.21.15.33:0/987045899 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa94c1010e0 0x7fa94c10d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.369+0000 7fa950ca1640 1 --2- 172.21.15.33:0/987045899 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa94c100720 0x7fa94c100ba0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.369+0000 7fa950ca1640 1 --2- 172.21.15.33:0/987045899 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa94c106730 0x7fa94c106b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.369+0000 7fa950ca1640 1 -- 172.21.15.33:0/987045899 >> 172.21.15.33:0/987045899 conn(0x7fa94c077820 msgr2=0x7fa94c077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:41.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.369+0000 7fa950ca1640 1 -- 172.21.15.33:0/987045899 shutdown_connections 2024-03-19T02:58:41.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.369+0000 7fa950ca1640 1 -- 172.21.15.33:0/987045899 wait complete. 2024-03-19T02:58:41.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.369+0000 7fa950ca1640 1 Processor -- start 2024-03-19T02:58:41.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.370+0000 7fa950ca1640 1 -- start start 2024-03-19T02:58:41.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.370+0000 7fa950ca1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa94c100720 0x7fa94c102d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.370+0000 7fa950ca1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa94c1010e0 0x7fa94c1032d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.370+0000 7fa950ca1640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa94c106730 0x7fa94c103810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.370+0000 7fa950ca1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c1109b0 con 0x7fa94c1010e0 2024-03-19T02:58:41.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.370+0000 7fa950ca1640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c110830 con 0x7fa94c100720 2024-03-19T02:58:41.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.370+0000 7fa950ca1640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c110b30 con 0x7fa94c106730 2024-03-19T02:58:41.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.370+0000 7fa943fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa94c1010e0 0x7fa94c1032d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.370+0000 7fa943fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa94c1010e0 0x7fa94c1032d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53704/0 (socket says 172.21.15.33:53704) 2024-03-19T02:58:41.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.370+0000 7fa943fff640 1 -- 172.21.15.33:0/1529678464 learned_addr learned my addr 172.21.15.33:0/1529678464 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:41.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.370+0000 7fa94a59c640 1 --2- 172.21.15.33:0/1529678464 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa94c100720 0x7fa94c102d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.370+0000 7fa94ad9d640 1 --2- 172.21.15.33:0/1529678464 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa94c106730 0x7fa94c103810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.370+0000 7fa943fff640 1 -- 172.21.15.33:0/1529678464 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa94c106730 msgr2=0x7fa94c103810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:41.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.370+0000 7fa943fff640 1 --2- 172.21.15.33:0/1529678464 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa94c106730 0x7fa94c103810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.370+0000 7fa943fff640 1 -- 172.21.15.33:0/1529678464 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa94c100720 msgr2=0x7fa94c102d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:41.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.370+0000 7fa943fff640 1 --2- 172.21.15.33:0/1529678464 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa94c100720 0x7fa94c102d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.370+0000 7fa943fff640 1 -- 172.21.15.33:0/1529678464 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa94c1a6500 con 0x7fa94c1010e0 2024-03-19T02:58:41.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.370+0000 7fa943fff640 1 --2- 172.21.15.33:0/1529678464 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa94c1010e0 0x7fa94c1032d0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fa934037040 tx=0x7fa934004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:41.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.370+0000 7fa9437fe640 1 -- 172.21.15.33:0/1529678464 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa934041020 con 0x7fa94c1010e0 2024-03-19T02:58:41.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.371+0000 7fa950ca1640 1 -- 172.21.15.33:0/1529678464 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa94c1a6790 con 0x7fa94c1010e0 2024-03-19T02:58:41.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.371+0000 7fa9437fe640 1 -- 172.21.15.33:0/1529678464 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa93402c9c0 con 0x7fa94c1010e0 2024-03-19T02:58:41.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.371+0000 7fa9437fe640 1 -- 172.21.15.33:0/1529678464 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa934039040 con 0x7fa94c1010e0 2024-03-19T02:58:41.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.371+0000 7fa950ca1640 1 -- 172.21.15.33:0/1529678464 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa94c1a6cd0 con 0x7fa94c1010e0 2024-03-19T02:58:41.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.372+0000 7fa950ca1640 1 -- 172.21.15.33:0/1529678464 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa94c068610 con 0x7fa94c1010e0 2024-03-19T02:58:41.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.373+0000 7fa9437fe640 1 -- 172.21.15.33:0/1529678464 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fa934044020 con 0x7fa94c1010e0 2024-03-19T02:58:41.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.373+0000 7fa9437fe640 1 --2- 172.21.15.33:0/1529678464 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa9380a2970 0x7fa9380a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.373+0000 7fa9437fe640 1 -- 172.21.15.33:0/1529678464 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fa9340f08e0 con 0x7fa94c1010e0 2024-03-19T02:58:41.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.373+0000 7fa94a59c640 1 --2- 172.21.15.33:0/1529678464 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa9380a2970 0x7fa9380a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.374+0000 7fa94a59c640 1 --2- 172.21.15.33:0/1529678464 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa9380a2970 0x7fa9380a4e30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa93c0096c0 tx=0x7fa93c009260 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:41.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.378+0000 7fa9437fe640 1 -- 172.21.15.33:0/1529678464 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa93404b020 con 0x7fa94c1010e0 2024-03-19T02:58:41.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.600+0000 7fa950ca1640 1 -- 172.21.15.33:0/1529678464 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa94c109cc0 con 0x7fa94c1010e0 2024-03-19T02:58:41.602 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.601+0000 7fa9437fe640 1 -- 172.21.15.33:0/1529678464 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11867 (secure 0 0 0) 0x7fa9340bd3e0 con 0x7fa94c1010e0 2024-03-19T02:58:41.602 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:58:41.602 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:50:52.201798+0000","modified":"2024-03-19T02:53:11.748697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24488,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/3731232169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":3731232169},{"type":"v1","addr":"172.21.15.96:6839","nonce":3731232169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24457,24488,24509]},"id":1}]} 2024-03-19T02:58:41.602 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-03-19T02:58:41.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.603+0000 7fa950ca1640 1 -- 172.21.15.33:0/1529678464 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa9380a2970 msgr2=0x7fa9380a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:41.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.603+0000 7fa950ca1640 1 --2- 172.21.15.33:0/1529678464 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa9380a2970 0x7fa9380a4e30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa93c0096c0 tx=0x7fa93c009260 comp rx=0 tx=0).stop 2024-03-19T02:58:41.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.603+0000 7fa950ca1640 1 -- 172.21.15.33:0/1529678464 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa94c1010e0 msgr2=0x7fa94c1032d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:41.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.603+0000 7fa950ca1640 1 --2- 172.21.15.33:0/1529678464 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa94c1010e0 0x7fa94c1032d0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fa934037040 tx=0x7fa934004040 comp rx=0 tx=0).stop 2024-03-19T02:58:41.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.603+0000 7fa950ca1640 1 -- 172.21.15.33:0/1529678464 shutdown_connections 2024-03-19T02:58:41.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.603+0000 7fa950ca1640 1 --2- 172.21.15.33:0/1529678464 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa9380a2970 0x7fa9380a4e30 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.603+0000 7fa950ca1640 1 --2- 172.21.15.33:0/1529678464 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa94c106730 0x7fa94c103810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.603+0000 7fa950ca1640 1 --2- 172.21.15.33:0/1529678464 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa94c1010e0 0x7fa94c1032d0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.603+0000 7fa950ca1640 1 --2- 172.21.15.33:0/1529678464 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa94c100720 0x7fa94c102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.603+0000 7fa950ca1640 1 -- 172.21.15.33:0/1529678464 >> 172.21.15.33:0/1529678464 conn(0x7fa94c077820 msgr2=0x7fa94c068cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:41.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.603+0000 7fa950ca1640 1 -- 172.21.15.33:0/1529678464 shutdown_connections 2024-03-19T02:58:41.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.603+0000 7fa950ca1640 1 -- 172.21.15.33:0/1529678464 wait complete. 2024-03-19T02:58:41.615 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:58:41.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.720+0000 7f5be01a9640 1 -- 172.21.15.33:0/13140020 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bd81077c0 msgr2=0x7f5bd8107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:41.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.720+0000 7f5be01a9640 1 --2- 172.21.15.33:0/13140020 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bd81077c0 0x7f5bd8107ba0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0009930 tx=0x7f5bd002bc30 comp rx=0 tx=0).stop 2024-03-19T02:58:41.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.720+0000 7f5be01a9640 1 -- 172.21.15.33:0/13140020 shutdown_connections 2024-03-19T02:58:41.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.720+0000 7f5be01a9640 1 --2- 172.21.15.33:0/13140020 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bd8102170 0x7f5bd810e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.720+0000 7f5be01a9640 1 --2- 172.21.15.33:0/13140020 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bd81017b0 0x7f5bd8101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.720+0000 7f5be01a9640 1 --2- 172.21.15.33:0/13140020 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bd81077c0 0x7f5bd8107ba0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.720+0000 7f5be01a9640 1 -- 172.21.15.33:0/13140020 >> 172.21.15.33:0/13140020 conn(0x7f5bd80fd2c0 msgr2=0x7f5bd80ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:41.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.720+0000 7f5be01a9640 1 -- 172.21.15.33:0/13140020 shutdown_connections 2024-03-19T02:58:41.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.720+0000 7f5be01a9640 1 -- 172.21.15.33:0/13140020 wait complete. 2024-03-19T02:58:41.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.720+0000 7f5be01a9640 1 Processor -- start 2024-03-19T02:58:41.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.720+0000 7f5be01a9640 1 -- start start 2024-03-19T02:58:41.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.720+0000 7f5be01a9640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bd81017b0 0x7f5bd819a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.720+0000 7f5be01a9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bd8102170 0x7f5bd819a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.720+0000 7f5be01a9640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bd81077c0 0x7f5bd819eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.720+0000 7f5be01a9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd8111a80 con 0x7f5bd8102170 2024-03-19T02:58:41.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.720+0000 7f5be01a9640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd8111900 con 0x7f5bd81077c0 2024-03-19T02:58:41.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.720+0000 7f5be01a9640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd8111c00 con 0x7f5bd81017b0 2024-03-19T02:58:41.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.721+0000 7f5bdd744640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bd8102170 0x7f5bd819a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.721+0000 7f5bddf45640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bd81017b0 0x7f5bd819a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.721+0000 7f5bdd744640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bd8102170 0x7f5bd819a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53712/0 (socket says 172.21.15.33:53712) 2024-03-19T02:58:41.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.721+0000 7f5bde746640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bd81077c0 0x7f5bd819eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.721+0000 7f5bdd744640 1 -- 172.21.15.33:0/2674264143 learned_addr learned my addr 172.21.15.33:0/2674264143 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:41.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.721+0000 7f5bdd744640 1 -- 172.21.15.33:0/2674264143 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bd81017b0 msgr2=0x7f5bd819a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:41.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.721+0000 7f5bdd744640 1 --2- 172.21.15.33:0/2674264143 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bd81017b0 0x7f5bd819a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.721+0000 7f5bdd744640 1 -- 172.21.15.33:0/2674264143 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bd81077c0 msgr2=0x7f5bd819eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:41.725 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.721+0000 7f5bdd744640 1 --2- 172.21.15.33:0/2674264143 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bd81077c0 0x7f5bd819eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.721+0000 7f5bdd744640 1 -- 172.21.15.33:0/2674264143 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bd819f1f0 con 0x7f5bd8102170 2024-03-19T02:58:41.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.721+0000 7f5bddf45640 1 --2- 172.21.15.33:0/2674264143 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bd81017b0 0x7f5bd819a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:58:41.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.721+0000 7f5bde746640 1 --2- 172.21.15.33:0/2674264143 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bd81077c0 0x7f5bd819eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:58:41.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.721+0000 7f5bdd744640 1 --2- 172.21.15.33:0/2674264143 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bd8102170 0x7f5bd819a740 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc00e9f0 tx=0x7f5bcc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:41.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.721+0000 7f5bd6ffd640 1 -- 172.21.15.33:0/2674264143 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc00cd80 con 0x7f5bd8102170 2024-03-19T02:58:41.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.721+0000 7f5be01a9640 1 -- 172.21.15.33:0/2674264143 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bd819f4e0 con 0x7f5bd8102170 2024-03-19T02:58:41.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.721+0000 7f5bd6ffd640 1 -- 172.21.15.33:0/2674264143 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5bcc0054d0 con 0x7f5bd8102170 2024-03-19T02:58:41.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.721+0000 7f5bd6ffd640 1 -- 172.21.15.33:0/2674264143 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc010490 con 0x7f5bd8102170 2024-03-19T02:58:41.726 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.721+0000 7f5be01a9640 1 -- 172.21.15.33:0/2674264143 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bd81a6d70 con 0x7f5bd8102170 2024-03-19T02:58:41.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.724+0000 7f5bd6ffd640 1 -- 172.21.15.33:0/2674264143 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f5bcc005020 con 0x7f5bd8102170 2024-03-19T02:58:41.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.724+0000 7f5be2d8f740 1 Processor -- start 2024-03-19T02:58:41.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.724+0000 7f5be2d8f740 1 -- start start 2024-03-19T02:58:41.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.724+0000 7f5be2d8f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557e64a9e2d0 0x557e64a9e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.724+0000 7f5be2d8f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557e64a9ebf0 0x557e64aa7290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.724+0000 7f5be2d8f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557e64aa77d0 0x557e64aa9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.724+0000 7f5be2d8f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557e64a014b0 con 0x557e64a9e2d0 2024-03-19T02:58:41.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.724+0000 7f5be2d8f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557e649a9d70 con 0x557e64aa77d0 2024-03-19T02:58:41.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.724+0000 7f5be2d8f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x557e649fd550 con 0x557e64a9ebf0 2024-03-19T02:58:41.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.724+0000 7f5bde746640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557e64a9e2d0 0x557e64a9e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.724+0000 7f5bde746640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557e64a9e2d0 0x557e64a9e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53728/0 (socket says 172.21.15.33:53728) 2024-03-19T02:58:41.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.725+0000 7f5bde746640 1 -- 172.21.15.33:0/3194428751 learned_addr learned my addr 172.21.15.33:0/3194428751 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:41.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.725+0000 7f5bdd744640 1 --2- 172.21.15.33:0/3194428751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557e64aa77d0 0x557e64aa9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.725+0000 7f5bddf45640 1 --2- 172.21.15.33:0/3194428751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557e64a9ebf0 0x557e64aa7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.725+0000 7f5bd6ffd640 1 --2- 172.21.15.33:0/2674264143 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f5bc00a2970 0x7f5bc00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.725+0000 7f5bde746640 1 -- 172.21.15.33:0/3194428751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557e64a9ebf0 msgr2=0x557e64aa7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:41.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.725+0000 7f5bde746640 1 --2- 172.21.15.33:0/3194428751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557e64a9ebf0 0x557e64aa7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.725+0000 7f5bde746640 1 -- 172.21.15.33:0/3194428751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557e64aa77d0 msgr2=0x557e64aa9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:41.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.725+0000 7f5bde746640 1 --2- 172.21.15.33:0/3194428751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557e64aa77d0 0x557e64aa9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.725+0000 7f5bde746640 1 -- 172.21.15.33:0/3194428751 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e649cc210 con 0x557e64a9e2d0 2024-03-19T02:58:41.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.725+0000 7f5bd6ffd640 1 -- 172.21.15.33:0/2674264143 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f5bcc0cf540 con 0x7f5bd8102170 2024-03-19T02:58:41.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.725+0000 7f5bde746640 1 --2- 172.21.15.33:0/2674264143 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f5bc00a2970 0x7f5bc00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.725+0000 7f5bde746640 1 --2- 172.21.15.33:0/3194428751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557e64a9e2d0 0x557e64a9e6b0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f5bc80118d0 tx=0x7f5bc8011d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7792360a621f87e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:41.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.725+0000 7f5bbffff640 1 -- 172.21.15.33:0/3194428751 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc8017050 con 0x557e64a9e2d0 2024-03-19T02:58:41.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.725+0000 7f5bbffff640 1 -- 172.21.15.33:0/3194428751 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5bc800e3b0 con 0x557e64a9e2d0 2024-03-19T02:58:41.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.725+0000 7f5bbffff640 1 -- 172.21.15.33:0/3194428751 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc801a5c0 con 0x557e64a9e2d0 2024-03-19T02:58:41.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.726+0000 7f5be2d8f740 1 -- 172.21.15.33:0/3194428751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557e64a9e2d0 msgr2=0x557e64a9e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:41.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.726+0000 7f5be2d8f740 1 --2- 172.21.15.33:0/3194428751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557e64a9e2d0 0x557e64a9e6b0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f5bc80118d0 tx=0x7f5bc8011d90 comp rx=0 tx=0).stop 2024-03-19T02:58:41.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.726+0000 7f5bde746640 1 --2- 172.21.15.33:0/2674264143 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f5bc00a2970 0x7f5bc00a4e30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8023000 tx=0x7f5bc8024000 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:41.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.726+0000 7f5be2d8f740 1 -- 172.21.15.33:0/3194428751 shutdown_connections 2024-03-19T02:58:41.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.726+0000 7f5be2d8f740 1 --2- 172.21.15.33:0/3194428751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557e64a9ebf0 0x557e64aa7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.726+0000 7f5be2d8f740 1 --2- 172.21.15.33:0/3194428751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557e64aa77d0 0x557e64aa9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.726+0000 7f5be2d8f740 1 --2- 172.21.15.33:0/3194428751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557e64a9e2d0 0x557e64a9e6b0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.726+0000 7f5be2d8f740 1 -- 172.21.15.33:0/3194428751 >> 172.21.15.33:0/3194428751 conn(0x557e64a98a00 msgr2=0x557e64a98de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:41.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.726+0000 7f5be2d8f740 1 -- 172.21.15.33:0/3194428751 shutdown_connections 2024-03-19T02:58:41.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.726+0000 7f5be2d8f740 1 -- 172.21.15.33:0/3194428751 wait complete. 2024-03-19T02:58:41.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.726+0000 7f5be2d8f740 1 -- start start 2024-03-19T02:58:41.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.726+0000 7f5be2d8f740 1 Processor -- start 2024-03-19T02:58:41.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.727+0000 7f5be2d8f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557e64a9e2d0 0x557e64aa4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.727+0000 7f5be2d8f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557e64a9ebf0 0x557e64aa4db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.727+0000 7f5be2d8f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557e64aa77d0 0x557e64aa52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.727+0000 7f5be2d8f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557e64aaa5c0 con 0x557e64a9ebf0 2024-03-19T02:58:41.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.727+0000 7f5be2d8f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557e649bd310 con 0x557e64a9e2d0 2024-03-19T02:58:41.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.727+0000 7f5be2d8f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x557e64aaa740 con 0x557e64aa77d0 2024-03-19T02:58:41.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.727+0000 7f5be2d8f740 10 client.0 _ll_register_callbacks cb 0x557e649b12d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:58:41.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.727+0000 7f5bdd744640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557e64a9ebf0 0x557e64aa4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.727+0000 7f5bddf45640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557e64a9e2d0 0x557e64aa4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.727+0000 7f5bdd744640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557e64a9ebf0 0x557e64aa4db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53732/0 (socket says 172.21.15.33:53732) 2024-03-19T02:58:41.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.727+0000 7f5bddf45640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557e64a9e2d0 0x557e64aa4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:57068/0 (socket says 172.21.15.33:57068) 2024-03-19T02:58:41.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.727+0000 7f5bdd744640 1 -- 172.21.15.33:0/632117695 learned_addr learned my addr 172.21.15.33:0/632117695 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:41.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.727+0000 7f5bde746640 1 --2- 172.21.15.33:0/632117695 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557e64aa77d0 0x557e64aa52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.727+0000 7f5bdd744640 1 -- 172.21.15.33:0/632117695 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557e64aa77d0 msgr2=0x557e64aa52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:41.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.727+0000 7f5bdd744640 1 --2- 172.21.15.33:0/632117695 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557e64aa77d0 0x557e64aa52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.727+0000 7f5bdd744640 1 -- 172.21.15.33:0/632117695 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557e64a9e2d0 msgr2=0x557e64aa4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:41.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.727+0000 7f5bdd744640 1 --2- 172.21.15.33:0/632117695 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557e64a9e2d0 0x557e64aa4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.727+0000 7f5bdd744640 1 -- 172.21.15.33:0/632117695 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e647153f0 con 0x557e64a9ebf0 2024-03-19T02:58:41.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.727+0000 7f5bde746640 1 --2- 172.21.15.33:0/632117695 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557e64aa77d0 0x557e64aa52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:58:41.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.727+0000 7f5bdd744640 1 --2- 172.21.15.33:0/632117695 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557e64a9ebf0 0x557e64aa4db0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc0cc980 tx=0x7f5bcc004aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44d84ffe977766b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:41.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.727+0000 7f5bbd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:58:41.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.728+0000 7f5bbd7fa640 1 -- 172.21.15.33:0/632117695 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc09b190 con 0x557e64a9ebf0 2024-03-19T02:58:41.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.728+0000 7f5bbd7fa640 1 -- 172.21.15.33:0/632117695 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f5bcc0cbde0 con 0x557e64a9ebf0 2024-03-19T02:58:41.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.728+0000 7f5be2d8f740 1 -- 172.21.15.33:0/632117695 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bc800e3b0 con 0x557e64a9ebf0 2024-03-19T02:58:41.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.728+0000 7f5bbd7fa640 1 -- 172.21.15.33:0/632117695 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f5bcc005310 con 0x557e64a9ebf0 2024-03-19T02:58:41.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.728+0000 7f5be2d8f740 10 client.16758 fetch_fsmap learned FSMap version 25 2024-03-19T02:58:41.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.728+0000 7f5be2d8f740 1 -- 172.21.15.33:0/632117695 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x557e647df7d0 con 0x557e64a9ebf0 2024-03-19T02:58:41.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.728+0000 7f5bbd7fa640 1 -- 172.21.15.33:0/632117695 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5bc800e3b0 con 0x557e64a9ebf0 2024-03-19T02:58:41.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.728+0000 7f5bbd7fa640 1 -- 172.21.15.33:0/632117695 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f5bcc0cc160 con 0x557e64a9ebf0 2024-03-19T02:58:41.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.728+0000 7f5be2d8f740 10 client.16758 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:58:41.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.728+0000 7f5be2d8f740 10 client.16758 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T02:58:41.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.728+0000 7f5be2d8f740 20 client.16758 populate_metadata read hostname 'smithi033' 2024-03-19T02:58:41.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.728+0000 7f5be2d8f740 1 --2- 172.21.15.33:0/632117695 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x557e64ab7c00 0x557e64ab9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.728+0000 7f5be2d8f740 4 client.16758 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:58:41.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.728+0000 7f5be2d8f740 1 -- 172.21.15.33:0/632117695 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557e64a5f3e0 con 0x557e64ab7c00 2024-03-19T02:58:41.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.728+0000 7f5bddf45640 1 --2- 172.21.15.33:0/632117695 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x557e64ab7c00 0x557e64ab9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.729+0000 7f5bbd7fa640 10 client.16758 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T02:58:41.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.729+0000 7f5bddf45640 1 --2- 172.21.15.33:0/632117695 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x557e64ab7c00 0x557e64ab9ff0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f0a26a552b600489 server_cookie=8b5b42d6fb97b1a3 in_seq=0 out_seq=0 2024-03-19T02:58:41.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.729+0000 7f5bbd7fa640 1 -- 172.21.15.33:0/632117695 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557e64a5f3e0 con 0x557e64ab7c00 2024-03-19T02:58:41.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.729+0000 7f5bbd7fa640 10 client.16758 handle_command_reply: tid=0 2024-03-19T02:58:41.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.729+0000 7f5be2d8f740 1 client.16758 shutdown 2024-03-19T02:58:41.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.730+0000 7f5be2d8f740 1 -- 172.21.15.33:0/632117695 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557e64a9ebf0 msgr2=0x557e64aa4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:41.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.730+0000 7f5be2d8f740 1 --2- 172.21.15.33:0/632117695 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557e64a9ebf0 0x557e64aa4db0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc0cc980 tx=0x7f5bcc004aa0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.730+0000 7f5be2d8f740 1 -- 172.21.15.33:0/632117695 shutdown_connections 2024-03-19T02:58:41.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.730+0000 7f5be2d8f740 1 --2- 172.21.15.33:0/632117695 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x557e64ab7c00 0x557e64ab9ff0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.730+0000 7f5be2d8f740 1 --2- 172.21.15.33:0/632117695 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557e64aa77d0 0x557e64aa52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.730+0000 7f5be2d8f740 1 --2- 172.21.15.33:0/632117695 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557e64a9ebf0 0x557e64aa4db0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.730+0000 7f5bbd7fa640 0 client.16758 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T02:58:41.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.730+0000 7f5be2d8f740 1 --2- 172.21.15.33:0/632117695 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557e64a9e2d0 0x557e64aa4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.730+0000 7f5be2d8f740 1 -- 172.21.15.33:0/632117695 >> 172.21.15.33:0/632117695 conn(0x557e64a98a00 msgr2=0x557e64a9a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:41.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.730+0000 7f5be2d8f740 1 -- 172.21.15.33:0/632117695 shutdown_connections 2024-03-19T02:58:41.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.730+0000 7f5be2d8f740 1 -- 172.21.15.33:0/632117695 wait complete. 2024-03-19T02:58:41.741 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.730+0000 7f5be2d8f740 20 client.16758 trim_cache size 0 max 16384 2024-03-19T02:58:41.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.743+0000 7f5be2d8f740 1 Processor -- start 2024-03-19T02:58:41.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.743+0000 7f5be2d8f740 1 -- start start 2024-03-19T02:58:41.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.743+0000 7f5be2d8f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557e64a9e2d0 0x557e64aa6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.743+0000 7f5be2d8f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557e64a9ebf0 0x557e64ab3360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.743+0000 7f5be2d8f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557e64aa77d0 0x557e64aa5110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.743+0000 7f5be2d8f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd81b3670 con 0x557e64a9e2d0 2024-03-19T02:58:41.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.743+0000 7f5be2d8f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd81b0f40 con 0x557e64a9ebf0 2024-03-19T02:58:41.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.743+0000 7f5be2d8f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd81aed70 con 0x557e64aa77d0 2024-03-19T02:58:41.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.743+0000 7f5bddf45640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557e64a9e2d0 0x557e64aa6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.743+0000 7f5bddf45640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557e64a9e2d0 0x557e64aa6fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53742/0 (socket says 172.21.15.33:53742) 2024-03-19T02:58:41.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.743+0000 7f5bddf45640 1 -- 172.21.15.33:0/3903330990 learned_addr learned my addr 172.21.15.33:0/3903330990 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:41.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.743+0000 7f5bde746640 1 --2- 172.21.15.33:0/3903330990 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557e64aa77d0 0x557e64aa5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.743+0000 7f5bdd744640 1 --2- 172.21.15.33:0/3903330990 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557e64a9ebf0 0x557e64ab3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.744+0000 7f5bddf45640 1 -- 172.21.15.33:0/3903330990 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557e64aa77d0 msgr2=0x557e64aa5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:41.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.744+0000 7f5bddf45640 1 --2- 172.21.15.33:0/3903330990 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557e64aa77d0 0x557e64aa5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.744+0000 7f5bddf45640 1 -- 172.21.15.33:0/3903330990 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557e64a9ebf0 msgr2=0x557e64ab3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:41.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.744+0000 7f5bddf45640 1 --2- 172.21.15.33:0/3903330990 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557e64a9ebf0 0x557e64ab3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.744+0000 7f5bddf45640 1 -- 172.21.15.33:0/3903330990 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e64aba360 con 0x557e64a9e2d0 2024-03-19T02:58:41.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.744+0000 7f5bdd744640 1 --2- 172.21.15.33:0/3903330990 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557e64a9ebf0 0x557e64ab3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:58:41.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.744+0000 7f5bddf45640 1 --2- 172.21.15.33:0/3903330990 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557e64a9e2d0 0x557e64aa6fa0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f5bd00096c0 tx=0x7f5bd0005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e64391daa32eb6fd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:41.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.744+0000 7f5bbf7fe640 1 -- 172.21.15.33:0/3903330990 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd004dce0 con 0x557e64a9e2d0 2024-03-19T02:58:41.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.744+0000 7f5bbf7fe640 1 -- 172.21.15.33:0/3903330990 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5bd0051030 con 0x557e64a9e2d0 2024-03-19T02:58:41.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.744+0000 7f5bbf7fe640 1 -- 172.21.15.33:0/3903330990 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd002cb10 con 0x557e64a9e2d0 2024-03-19T02:58:41.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.744+0000 7f5be2d8f740 1 -- 172.21.15.33:0/3903330990 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557e64a9e2d0 msgr2=0x557e64aa6fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:41.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.744+0000 7f5be2d8f740 1 --2- 172.21.15.33:0/3903330990 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557e64a9e2d0 0x557e64aa6fa0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f5bd00096c0 tx=0x7f5bd0005d90 comp rx=0 tx=0).stop 2024-03-19T02:58:41.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.745+0000 7f5be2d8f740 1 -- 172.21.15.33:0/3903330990 shutdown_connections 2024-03-19T02:58:41.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.745+0000 7f5be2d8f740 1 --2- 172.21.15.33:0/3903330990 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557e64aa77d0 0x557e64aa5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.745+0000 7f5be2d8f740 1 --2- 172.21.15.33:0/3903330990 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557e64a9ebf0 0x557e64ab3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.745+0000 7f5be2d8f740 1 --2- 172.21.15.33:0/3903330990 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557e64a9e2d0 0x557e64aa6fa0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.745+0000 7f5be2d8f740 1 -- 172.21.15.33:0/3903330990 >> 172.21.15.33:0/3903330990 conn(0x557e64a98a00 msgr2=0x557e64a98de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:41.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.745+0000 7f5be2d8f740 1 -- 172.21.15.33:0/3903330990 shutdown_connections 2024-03-19T02:58:41.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.745+0000 7f5be2d8f740 1 -- 172.21.15.33:0/3903330990 wait complete. 2024-03-19T02:58:41.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.745+0000 7f5be2d8f740 1 -- start start 2024-03-19T02:58:41.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.745+0000 7f5be2d8f740 1 Processor -- start 2024-03-19T02:58:41.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.745+0000 7f5be2d8f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557e64a9e2d0 0x557e64aafd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.745+0000 7f5be2d8f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557e64a9ebf0 0x557e64aa1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.745+0000 7f5be2d8f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557e64aa77d0 0x557e64aa19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.745+0000 7f5be2d8f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd81af400 con 0x557e64aa77d0 2024-03-19T02:58:41.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.745+0000 7f5be2d8f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd81b0870 con 0x557e64a9e2d0 2024-03-19T02:58:41.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.745+0000 7f5be2d8f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd81afad0 con 0x557e64a9ebf0 2024-03-19T02:58:41.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.745+0000 7f5be2d8f740 10 client.0 _ll_register_callbacks cb 0x7f5bcc097bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:58:41.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.745+0000 7f5bde746640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557e64aa77d0 0x557e64aa19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.745+0000 7f5bde746640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557e64aa77d0 0x557e64aa19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53756/0 (socket says 172.21.15.33:53756) 2024-03-19T02:58:41.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.745+0000 7f5bde746640 1 -- 172.21.15.33:0/2289200505 learned_addr learned my addr 172.21.15.33:0/2289200505 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:41.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.746+0000 7f5bddf45640 1 --2- 172.21.15.33:0/2289200505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557e64a9e2d0 0x557e64aafd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.746+0000 7f5bdd744640 1 --2- 172.21.15.33:0/2289200505 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557e64a9ebf0 0x557e64aa1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.746+0000 7f5bde746640 1 -- 172.21.15.33:0/2289200505 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557e64a9ebf0 msgr2=0x557e64aa1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:41.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.746+0000 7f5bde746640 1 --2- 172.21.15.33:0/2289200505 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557e64a9ebf0 0x557e64aa1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.746+0000 7f5bde746640 1 -- 172.21.15.33:0/2289200505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557e64a9e2d0 msgr2=0x557e64aafd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:41.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.746+0000 7f5bde746640 1 --2- 172.21.15.33:0/2289200505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557e64a9e2d0 0x557e64aafd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.746+0000 7f5bde746640 1 -- 172.21.15.33:0/2289200505 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e64b1ced0 con 0x557e64aa77d0 2024-03-19T02:58:41.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.746+0000 7f5bde746640 1 --2- 172.21.15.33:0/2289200505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557e64aa77d0 0x557e64aa19c0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8009fd0 tx=0x7f5bc8004f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eba4564eb5aee209 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:41.753 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.746+0000 7f5bbd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:58:41.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.746+0000 7f5bbd7fa640 1 -- 172.21.15.33:0/2289200505 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc802b040 con 0x557e64aa77d0 2024-03-19T02:58:41.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.746+0000 7f5bbd7fa640 1 -- 172.21.15.33:0/2289200505 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f5bc801a960 con 0x557e64aa77d0 2024-03-19T02:58:41.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.746+0000 7f5be2d8f740 1 -- 172.21.15.33:0/2289200505 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bd0051030 con 0x557e64aa77d0 2024-03-19T02:58:41.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.746+0000 7f5bbd7fa640 1 -- 172.21.15.33:0/2289200505 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f5bc8017020 con 0x557e64aa77d0 2024-03-19T02:58:41.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.746+0000 7f5be2d8f740 10 client.16764 fetch_fsmap learned FSMap version 25 2024-03-19T02:58:41.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.746+0000 7f5be2d8f740 1 -- 172.21.15.33:0/2289200505 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x557e64b1e050 con 0x557e64aa77d0 2024-03-19T02:58:41.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.746+0000 7f5bbd7fa640 1 -- 172.21.15.33:0/2289200505 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5bd0051030 con 0x557e64aa77d0 2024-03-19T02:58:41.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.747+0000 7f5bbd7fa640 1 -- 172.21.15.33:0/2289200505 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f5bc801bbb0 con 0x557e64aa77d0 2024-03-19T02:58:41.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.747+0000 7f5be2d8f740 10 client.16764 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:58:41.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.747+0000 7f5be2d8f740 10 client.16764 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T02:58:41.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.747+0000 7f5be2d8f740 20 client.16764 populate_metadata read hostname 'smithi033' 2024-03-19T02:58:41.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.747+0000 7f5be2d8f740 1 --2- 172.21.15.33:0/2289200505 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x557e64b1e660 0x557e64b20a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.747+0000 7f5be2d8f740 4 client.16764 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:58:41.754 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.747+0000 7f5be2d8f740 1 -- 172.21.15.33:0/2289200505 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557e64a0e9f0 con 0x557e64b1e660 2024-03-19T02:58:41.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.747+0000 7f5bddf45640 1 --2- 172.21.15.33:0/2289200505 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x557e64b1e660 0x557e64b20a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.747+0000 7f5bddf45640 1 --2- 172.21.15.33:0/2289200505 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x557e64b1e660 0x557e64b20a80 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8cadd3a29d8db051 server_cookie=e13340ac52ec6eda in_seq=0 out_seq=0 2024-03-19T02:58:41.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.747+0000 7f5bbd7fa640 10 client.16764 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T02:58:41.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.747+0000 7f5bbd7fa640 1 -- 172.21.15.33:0/2289200505 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557e64a0e9f0 con 0x557e64b1e660 2024-03-19T02:58:41.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.747+0000 7f5bbd7fa640 10 client.16764 handle_command_reply: tid=0 2024-03-19T02:58:41.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.747+0000 7f5be2d8f740 1 client.16764 shutdown 2024-03-19T02:58:41.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.747+0000 7f5be2d8f740 1 -- 172.21.15.33:0/2289200505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557e64aa77d0 msgr2=0x557e64aa19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:41.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.747+0000 7f5be2d8f740 1 --2- 172.21.15.33:0/2289200505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557e64aa77d0 0x557e64aa19c0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8009fd0 tx=0x7f5bc8004f40 comp rx=0 tx=0).stop 2024-03-19T02:58:41.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.748+0000 7f5be2d8f740 1 -- 172.21.15.33:0/2289200505 shutdown_connections 2024-03-19T02:58:41.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.748+0000 7f5be2d8f740 1 --2- 172.21.15.33:0/2289200505 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x557e64b1e660 0x557e64b20a80 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.748+0000 7f5be2d8f740 1 --2- 172.21.15.33:0/2289200505 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557e64a9ebf0 0x557e64aa1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.748+0000 7f5be2d8f740 1 --2- 172.21.15.33:0/2289200505 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557e64aa77d0 0x557e64aa19c0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.748+0000 7f5be2d8f740 1 --2- 172.21.15.33:0/2289200505 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557e64a9e2d0 0x557e64aafd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.748+0000 7f5bbd7fa640 0 client.16764 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T02:58:41.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.748+0000 7f5be2d8f740 1 -- 172.21.15.33:0/2289200505 >> 172.21.15.33:0/2289200505 conn(0x557e64a98a00 msgr2=0x557e64a97310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:41.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.748+0000 7f5be2d8f740 1 -- 172.21.15.33:0/2289200505 shutdown_connections 2024-03-19T02:58:41.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.748+0000 7f5be2d8f740 1 -- 172.21.15.33:0/2289200505 wait complete. 2024-03-19T02:58:41.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.748+0000 7f5be2d8f740 20 client.16764 trim_cache size 0 max 16384 2024-03-19T02:58:41.756 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:58:41.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.748+0000 7f5be01a9640 1 -- 172.21.15.33:0/2674264143 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f5bc00a2970 msgr2=0x7f5bc00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:41.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.748+0000 7f5be01a9640 1 --2- 172.21.15.33:0/2674264143 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f5bc00a2970 0x7f5bc00a4e30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8023000 tx=0x7f5bc8024000 comp rx=0 tx=0).stop 2024-03-19T02:58:41.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.748+0000 7f5be01a9640 1 -- 172.21.15.33:0/2674264143 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bd8102170 msgr2=0x7f5bd819a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:41.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.748+0000 7f5be01a9640 1 --2- 172.21.15.33:0/2674264143 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bd8102170 0x7f5bd819a740 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc00e9f0 tx=0x7f5bcc00edf0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.748+0000 7f5be01a9640 1 -- 172.21.15.33:0/2674264143 shutdown_connections 2024-03-19T02:58:41.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.748+0000 7f5be01a9640 1 --2- 172.21.15.33:0/2674264143 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f5bc00a2970 0x7f5bc00a4e30 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.748+0000 7f5be01a9640 1 --2- 172.21.15.33:0/2674264143 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5bd81077c0 0x7f5bd819eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.748+0000 7f5be01a9640 1 --2- 172.21.15.33:0/2674264143 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5bd8102170 0x7f5bd819a740 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.748+0000 7f5be01a9640 1 --2- 172.21.15.33:0/2674264143 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bd81017b0 0x7f5bd819a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.748+0000 7f5be01a9640 1 -- 172.21.15.33:0/2674264143 >> 172.21.15.33:0/2674264143 conn(0x7f5bd80fd2c0 msgr2=0x7f5bd810cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:41.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.748+0000 7f5be01a9640 1 -- 172.21.15.33:0/2674264143 shutdown_connections 2024-03-19T02:58:41.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.748+0000 7f5be01a9640 1 -- 172.21.15.33:0/2674264143 wait complete. 2024-03-19T02:58:41.760 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:58:41.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.864+0000 7fe03de9a640 1 -- 172.21.15.33:0/1276341721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe038101310 msgr2=0x7fe03810d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:41.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.864+0000 7fe03de9a640 1 --2- 172.21.15.33:0/1276341721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe038101310 0x7fe03810d4f0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fe030009ca0 tx=0x7fe03002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.864+0000 7fe03de9a640 1 -- 172.21.15.33:0/1276341721 shutdown_connections 2024-03-19T02:58:41.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.864+0000 7fe03de9a640 1 --2- 172.21.15.33:0/1276341721 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe038101310 0x7fe03810d4f0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.864+0000 7fe03de9a640 1 --2- 172.21.15.33:0/1276341721 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe038100970 0x7fe038100dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.864+0000 7fe03de9a640 1 --2- 172.21.15.33:0/1276341721 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe038106750 0x7fe038106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.864+0000 7fe03de9a640 1 -- 172.21.15.33:0/1276341721 >> 172.21.15.33:0/1276341721 conn(0x7fe0380774c0 msgr2=0x7fe0380778d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:41.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.864+0000 7fe03de9a640 1 -- 172.21.15.33:0/1276341721 shutdown_connections 2024-03-19T02:58:41.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.864+0000 7fe03de9a640 1 -- 172.21.15.33:0/1276341721 wait complete. 2024-03-19T02:58:41.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.865+0000 7fe03de9a640 1 Processor -- start 2024-03-19T02:58:41.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.865+0000 7fe03de9a640 1 -- start start 2024-03-19T02:58:41.868 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.865+0000 7fe03de9a640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe038100970 0x7fe038102390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.865+0000 7fe03de9a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe038101310 0x7fe0381028d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.865+0000 7fe03de9a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe038106750 0x7fe038102e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.865+0000 7fe03de9a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe038110980 con 0x7fe038106750 2024-03-19T02:58:41.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.865+0000 7fe03de9a640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe038110800 con 0x7fe038101310 2024-03-19T02:58:41.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.865+0000 7fe03de9a640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe038110b00 con 0x7fe038100970 2024-03-19T02:58:41.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.865+0000 7fe037fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe038106750 0x7fe038102e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.865+0000 7fe037fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe038106750 0x7fe038102e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53774/0 (socket says 172.21.15.33:53774) 2024-03-19T02:58:41.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.865+0000 7fe037fff640 1 -- 172.21.15.33:0/3374905609 learned_addr learned my addr 172.21.15.33:0/3374905609 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:41.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.865+0000 7fe036ffd640 1 --2- 172.21.15.33:0/3374905609 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe038101310 0x7fe0381028d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.865+0000 7fe0377fe640 1 --2- 172.21.15.33:0/3374905609 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe038100970 0x7fe038102390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.866+0000 7fe037fff640 1 -- 172.21.15.33:0/3374905609 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe038100970 msgr2=0x7fe038102390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:41.869 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.866+0000 7fe037fff640 1 --2- 172.21.15.33:0/3374905609 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe038100970 0x7fe038102390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.866+0000 7fe037fff640 1 -- 172.21.15.33:0/3374905609 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe038101310 msgr2=0x7fe0381028d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:41.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.866+0000 7fe037fff640 1 --2- 172.21.15.33:0/3374905609 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe038101310 0x7fe0381028d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.866+0000 7fe037fff640 1 -- 172.21.15.33:0/3374905609 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0381a63d0 con 0x7fe038106750 2024-03-19T02:58:41.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.866+0000 7fe036ffd640 1 --2- 172.21.15.33:0/3374905609 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe038101310 0x7fe0381028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:58:41.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.866+0000 7fe0377fe640 1 --2- 172.21.15.33:0/3374905609 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe038100970 0x7fe038102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:58:41.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.866+0000 7fe037fff640 1 --2- 172.21.15.33:0/3374905609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe038106750 0x7fe038102e10 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fe03002c700 tx=0x7fe030035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:41.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.866+0000 7fe034ff9640 1 -- 172.21.15.33:0/3374905609 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe030039040 con 0x7fe038106750 2024-03-19T02:58:41.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.866+0000 7fe034ff9640 1 -- 172.21.15.33:0/3374905609 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe030044070 con 0x7fe038106750 2024-03-19T02:58:41.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.866+0000 7fe03de9a640 1 -- 172.21.15.33:0/3374905609 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0381a6660 con 0x7fe038106750 2024-03-19T02:58:41.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.866+0000 7fe034ff9640 1 -- 172.21.15.33:0/3374905609 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0300042a0 con 0x7fe038106750 2024-03-19T02:58:41.870 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.866+0000 7fe03de9a640 1 -- 172.21.15.33:0/3374905609 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0381a6b50 con 0x7fe038106750 2024-03-19T02:58:41.877 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.868+0000 7fe034ff9640 1 -- 172.21.15.33:0/3374905609 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fe030003970 con 0x7fe038106750 2024-03-19T02:58:41.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.868+0000 7fe040b66740 1 Processor -- start 2024-03-19T02:58:41.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.869+0000 7fe040b66740 1 -- start start 2024-03-19T02:58:41.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.869+0000 7fe034ff9640 1 --2- 172.21.15.33:0/3374905609 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe0200a2970 0x7fe0200a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.869+0000 7fe034ff9640 1 -- 172.21.15.33:0/3374905609 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fe0300f1550 con 0x7fe038106750 2024-03-19T02:58:41.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.869+0000 7fe040b66740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fa82742d0 0x561fa82746b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.869+0000 7fe040b66740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fa8274bf0 0x561fa827d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.869+0000 7fe036ffd640 1 --2- 172.21.15.33:0/3374905609 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe0200a2970 0x7fe0200a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.869+0000 7fe040b66740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa827d7d0 0x561fa827fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.869+0000 7fe040b66740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561fa81d74b0 con 0x561fa82742d0 2024-03-19T02:58:41.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.869+0000 7fe040b66740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x561fa817fd70 con 0x561fa8274bf0 2024-03-19T02:58:41.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.869+0000 7fe040b66740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x561fa81d3550 con 0x561fa827d7d0 2024-03-19T02:58:41.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.869+0000 7fe0377fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fa82742d0 0x561fa82746b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.878 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.869+0000 7fe0377fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fa82742d0 0x561fa82746b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53782/0 (socket says 172.21.15.33:53782) 2024-03-19T02:58:41.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.869+0000 7fe0377fe640 1 -- 172.21.15.33:0/2559835317 learned_addr learned my addr 172.21.15.33:0/2559835317 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:41.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.869+0000 7fe036ffd640 1 --2- 172.21.15.33:0/2559835317 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fa8274bf0 0x561fa827d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.869+0000 7fe037fff640 1 --2- 172.21.15.33:0/2559835317 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa827d7d0 0x561fa827fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.869+0000 7fe0377fe640 1 -- 172.21.15.33:0/2559835317 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa827d7d0 msgr2=0x561fa827fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:41.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.869+0000 7fe0377fe640 1 --2- 172.21.15.33:0/2559835317 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa827d7d0 0x561fa827fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.869+0000 7fe0377fe640 1 -- 172.21.15.33:0/2559835317 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fa8274bf0 msgr2=0x561fa827d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:41.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.869+0000 7fe0377fe640 1 --2- 172.21.15.33:0/2559835317 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fa8274bf0 0x561fa827d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.869+0000 7fe0377fe640 1 -- 172.21.15.33:0/2559835317 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fa81a2210 con 0x561fa82742d0 2024-03-19T02:58:41.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.869+0000 7fe037fff640 1 --2- 172.21.15.33:0/2559835317 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa827d7d0 0x561fa827fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:58:41.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.869+0000 7fe036ffd640 1 --2- 172.21.15.33:0/2559835317 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fa8274bf0 0x561fa827d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:58:41.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.869+0000 7fe036ffd640 1 --2- 172.21.15.33:0/3374905609 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe0200a2970 0x7fe0200a4e30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe038103e20 tx=0x7fe02c011c80 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:41.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.869+0000 7fe0377fe640 1 --2- 172.21.15.33:0/2559835317 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fa82742d0 0x561fa82746b0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fe028009cd0 tx=0x7fe0280117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3837085e696822f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:41.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.870+0000 7fe01dffb640 1 -- 172.21.15.33:0/2559835317 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe028012450 con 0x561fa82742d0 2024-03-19T02:58:41.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.870+0000 7fe01dffb640 1 -- 172.21.15.33:0/2559835317 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe028012aa0 con 0x561fa82742d0 2024-03-19T02:58:41.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.870+0000 7fe01dffb640 1 -- 172.21.15.33:0/2559835317 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe02800fae0 con 0x561fa82742d0 2024-03-19T02:58:41.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.870+0000 7fe040b66740 1 -- 172.21.15.33:0/2559835317 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fa82742d0 msgr2=0x561fa82746b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:41.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.870+0000 7fe040b66740 1 --2- 172.21.15.33:0/2559835317 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fa82742d0 0x561fa82746b0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fe028009cd0 tx=0x7fe0280117b0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.870+0000 7fe040b66740 1 -- 172.21.15.33:0/2559835317 shutdown_connections 2024-03-19T02:58:41.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.870+0000 7fe040b66740 1 --2- 172.21.15.33:0/2559835317 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa827d7d0 0x561fa827fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.870+0000 7fe040b66740 1 --2- 172.21.15.33:0/2559835317 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fa8274bf0 0x561fa827d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.870+0000 7fe040b66740 1 --2- 172.21.15.33:0/2559835317 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fa82742d0 0x561fa82746b0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.870+0000 7fe040b66740 1 -- 172.21.15.33:0/2559835317 >> 172.21.15.33:0/2559835317 conn(0x561fa826ea00 msgr2=0x561fa826ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:41.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.870+0000 7fe040b66740 1 -- 172.21.15.33:0/2559835317 shutdown_connections 2024-03-19T02:58:41.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.870+0000 7fe040b66740 1 -- 172.21.15.33:0/2559835317 wait complete. 2024-03-19T02:58:41.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.871+0000 7fe040b66740 1 -- start start 2024-03-19T02:58:41.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.871+0000 7fe040b66740 1 Processor -- start 2024-03-19T02:58:41.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.871+0000 7fe040b66740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fa82742d0 0x561fa827a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.871+0000 7fe040b66740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa8274bf0 0x561fa827aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.871+0000 7fe040b66740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fa827d7d0 0x561fa827b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.871+0000 7fe040b66740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561fa82805c0 con 0x561fa827d7d0 2024-03-19T02:58:41.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.871+0000 7fe040b66740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x561fa8193310 con 0x561fa82742d0 2024-03-19T02:58:41.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.871+0000 7fe040b66740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x561fa8280740 con 0x561fa8274bf0 2024-03-19T02:58:41.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.871+0000 7fe040b66740 10 client.0 _ll_register_callbacks cb 0x561fa81872d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:58:41.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.871+0000 7fe037fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fa827d7d0 0x561fa827b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.871+0000 7fe0377fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fa82742d0 0x561fa827a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.871+0000 7fe0377fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fa82742d0 0x561fa827a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:57122/0 (socket says 172.21.15.33:57122) 2024-03-19T02:58:41.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.871+0000 7fe0377fe640 1 -- 172.21.15.33:0/2006561658 learned_addr learned my addr 172.21.15.33:0/2006561658 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:41.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.871+0000 7fe037fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fa827d7d0 0x561fa827b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53796/0 (socket says 172.21.15.33:53796) 2024-03-19T02:58:41.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.871+0000 7fe036ffd640 1 --2- 172.21.15.33:0/2006561658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa8274bf0 0x561fa827aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.871+0000 7fe0377fe640 1 -- 172.21.15.33:0/2006561658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa8274bf0 msgr2=0x561fa827aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:41.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.871+0000 7fe0377fe640 1 --2- 172.21.15.33:0/2006561658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa8274bf0 0x561fa827aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.871+0000 7fe0377fe640 1 -- 172.21.15.33:0/2006561658 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fa827d7d0 msgr2=0x561fa827b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:41.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.871+0000 7fe0377fe640 1 --2- 172.21.15.33:0/2006561658 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fa827d7d0 0x561fa827b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.871+0000 7fe0377fe640 1 -- 172.21.15.33:0/2006561658 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fa7eeb3f0 con 0x561fa82742d0 2024-03-19T02:58:41.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.872+0000 7fe0377fe640 1 --2- 172.21.15.33:0/2006561658 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fa82742d0 0x561fa827a760 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe02800f650 tx=0x7fe0280097c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32185ee4f39b53e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:41.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.872+0000 7fe016ffd640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T02:58:41.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.872+0000 7fe016ffd640 1 -- 172.21.15.33:0/2006561658 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe028027030 con 0x561fa82742d0 2024-03-19T02:58:41.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.872+0000 7fe016ffd640 1 -- 172.21.15.33:0/2006561658 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fe028004250 con 0x561fa82742d0 2024-03-19T02:58:41.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.872+0000 7fe040b66740 1 -- 172.21.15.33:0/2006561658 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe028012aa0 con 0x561fa82742d0 2024-03-19T02:58:41.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.872+0000 7fe016ffd640 1 -- 172.21.15.33:0/2006561658 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe028031050 con 0x561fa82742d0 2024-03-19T02:58:41.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.872+0000 7fe040b66740 10 client.26600 fetch_fsmap learned FSMap version 25 2024-03-19T02:58:41.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.872+0000 7fe040b66740 1 -- 172.21.15.33:0/2006561658 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x561fa7fb57d0 con 0x561fa82742d0 2024-03-19T02:58:41.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.872+0000 7fe016ffd640 1 -- 172.21.15.33:0/2006561658 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe028012aa0 con 0x561fa82742d0 2024-03-19T02:58:41.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.872+0000 7fe016ffd640 1 -- 172.21.15.33:0/2006561658 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fe02800f860 con 0x561fa82742d0 2024-03-19T02:58:41.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.872+0000 7fe040b66740 10 client.26600 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:58:41.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.872+0000 7fe040b66740 10 client.26600 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T02:58:41.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.872+0000 7fe040b66740 20 client.26600 populate_metadata read hostname 'smithi033' 2024-03-19T02:58:41.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.872+0000 7fe040b66740 1 --2- 172.21.15.33:0/2006561658 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x561fa828da20 0x561fa828fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.872+0000 7fe040b66740 4 client.26600 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:58:41.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.872+0000 7fe040b66740 1 -- 172.21.15.33:0/2006561658 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561fa82353e0 con 0x561fa828da20 2024-03-19T02:58:41.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.873+0000 7fe036ffd640 1 --2- 172.21.15.33:0/2006561658 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x561fa828da20 0x561fa828fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.873+0000 7fe016ffd640 10 client.26600 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T02:58:41.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.873+0000 7fe036ffd640 1 --2- 172.21.15.33:0/2006561658 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x561fa828da20 0x561fa828fe10 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=65278356f5a32c27 server_cookie=e77bc16c3dc3eccc in_seq=0 out_seq=0 2024-03-19T02:58:41.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.874+0000 7fe016ffd640 1 -- 172.21.15.33:0/2006561658 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561fa82353e0 con 0x561fa828da20 2024-03-19T02:58:41.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.874+0000 7fe016ffd640 10 client.26600 handle_command_reply: tid=0 2024-03-19T02:58:41.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.874+0000 7fe040b66740 1 client.26600 shutdown 2024-03-19T02:58:41.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.874+0000 7fe040b66740 1 -- 172.21.15.33:0/2006561658 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fa82742d0 msgr2=0x561fa827a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:41.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.874+0000 7fe040b66740 1 --2- 172.21.15.33:0/2006561658 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fa82742d0 0x561fa827a760 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe02800f650 tx=0x7fe0280097c0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.874+0000 7fe040b66740 1 -- 172.21.15.33:0/2006561658 shutdown_connections 2024-03-19T02:58:41.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.874+0000 7fe040b66740 1 --2- 172.21.15.33:0/2006561658 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x561fa828da20 0x561fa828fe10 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.874+0000 7fe040b66740 1 --2- 172.21.15.33:0/2006561658 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa8274bf0 0x561fa827aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.874+0000 7fe040b66740 1 --2- 172.21.15.33:0/2006561658 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fa827d7d0 0x561fa827b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.874+0000 7fe016ffd640 0 client.26600 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T02:58:41.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.874+0000 7fe040b66740 1 --2- 172.21.15.33:0/2006561658 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fa82742d0 0x561fa827a760 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.874+0000 7fe040b66740 1 -- 172.21.15.33:0/2006561658 >> 172.21.15.33:0/2006561658 conn(0x561fa826ea00 msgr2=0x561fa8270750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:41.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.874+0000 7fe040b66740 1 -- 172.21.15.33:0/2006561658 shutdown_connections 2024-03-19T02:58:41.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.874+0000 7fe040b66740 1 -- 172.21.15.33:0/2006561658 wait complete. 2024-03-19T02:58:41.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.874+0000 7fe040b66740 20 client.26600 trim_cache size 0 max 16384 2024-03-19T02:58:41.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.887+0000 7fe040b66740 1 Processor -- start 2024-03-19T02:58:41.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.887+0000 7fe040b66740 1 -- start start 2024-03-19T02:58:41.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.887+0000 7fe040b66740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fa82742d0 0x561fa8289360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.887+0000 7fe040b66740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa8274bf0 0x561fa827b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.887+0000 7fe040b66740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fa827d7d0 0x561fa826da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.887+0000 7fe040b66740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe0381b3580 con 0x561fa82742d0 2024-03-19T02:58:41.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.887+0000 7fe040b66740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe0381b0e50 con 0x561fa827d7d0 2024-03-19T02:58:41.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.887+0000 7fe040b66740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe0381aec80 con 0x561fa8274bf0 2024-03-19T02:58:41.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.887+0000 7fe0377fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fa82742d0 0x561fa8289360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.887+0000 7fe0377fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fa82742d0 0x561fa8289360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53802/0 (socket says 172.21.15.33:53802) 2024-03-19T02:58:41.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.887+0000 7fe0377fe640 1 -- 172.21.15.33:0/3930913214 learned_addr learned my addr 172.21.15.33:0/3930913214 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:41.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.888+0000 7fe037fff640 1 --2- 172.21.15.33:0/3930913214 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fa827d7d0 0x561fa826da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.888+0000 7fe036ffd640 1 --2- 172.21.15.33:0/3930913214 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa8274bf0 0x561fa827b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.888+0000 7fe0377fe640 1 -- 172.21.15.33:0/3930913214 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa8274bf0 msgr2=0x561fa827b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:41.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.888+0000 7fe0377fe640 1 --2- 172.21.15.33:0/3930913214 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa8274bf0 0x561fa827b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.888+0000 7fe0377fe640 1 -- 172.21.15.33:0/3930913214 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fa827d7d0 msgr2=0x561fa826da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:41.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.888+0000 7fe0377fe640 1 --2- 172.21.15.33:0/3930913214 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fa827d7d0 0x561fa826da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.888+0000 7fe0377fe640 1 -- 172.21.15.33:0/3930913214 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fa826a2c0 con 0x561fa82742d0 2024-03-19T02:58:41.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.888+0000 7fe0377fe640 1 --2- 172.21.15.33:0/3930913214 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fa82742d0 0x561fa8289360 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fe028001eb0 tx=0x7fe028005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19444178bb3c338c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:41.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.888+0000 7fe01d7fa640 1 -- 172.21.15.33:0/3930913214 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe028039020 con 0x561fa82742d0 2024-03-19T02:58:41.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.888+0000 7fe01d7fa640 1 -- 172.21.15.33:0/3930913214 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe028011cb0 con 0x561fa82742d0 2024-03-19T02:58:41.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.888+0000 7fe01d7fa640 1 -- 172.21.15.33:0/3930913214 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe028027070 con 0x561fa82742d0 2024-03-19T02:58:41.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.889+0000 7fe040b66740 1 -- 172.21.15.33:0/3930913214 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fa82742d0 msgr2=0x561fa8289360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:41.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.889+0000 7fe040b66740 1 --2- 172.21.15.33:0/3930913214 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fa82742d0 0x561fa8289360 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fe028001eb0 tx=0x7fe028005fd0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.889+0000 7fe040b66740 1 -- 172.21.15.33:0/3930913214 shutdown_connections 2024-03-19T02:58:41.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.889+0000 7fe040b66740 1 --2- 172.21.15.33:0/3930913214 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa8274bf0 0x561fa827b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.889+0000 7fe040b66740 1 --2- 172.21.15.33:0/3930913214 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fa827d7d0 0x561fa826da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.889+0000 7fe040b66740 1 --2- 172.21.15.33:0/3930913214 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fa82742d0 0x561fa8289360 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.889+0000 7fe040b66740 1 -- 172.21.15.33:0/3930913214 >> 172.21.15.33:0/3930913214 conn(0x561fa826ea00 msgr2=0x561fa826d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:41.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.889+0000 7fe040b66740 1 -- 172.21.15.33:0/3930913214 shutdown_connections 2024-03-19T02:58:41.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.889+0000 7fe040b66740 1 -- 172.21.15.33:0/3930913214 wait complete. 2024-03-19T02:58:41.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.889+0000 7fe040b66740 1 -- start start 2024-03-19T02:58:41.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.889+0000 7fe040b66740 1 Processor -- start 2024-03-19T02:58:41.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.889+0000 7fe040b66740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa82742d0 0x561fa8285b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.890+0000 7fe040b66740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fa8274bf0 0x561fa8276ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.890+0000 7fe040b66740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fa827d7d0 0x561fa8277530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.890+0000 7fe040b66740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe0381af310 con 0x561fa827d7d0 2024-03-19T02:58:41.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.890+0000 7fe040b66740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe0381b0780 con 0x561fa8274bf0 2024-03-19T02:58:41.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.890+0000 7fe040b66740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe0381af9e0 con 0x561fa82742d0 2024-03-19T02:58:41.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.890+0000 7fe040b66740 10 client.0 _ll_register_callbacks cb 0x7fe028009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:58:41.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.890+0000 7fe037fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fa827d7d0 0x561fa8277530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.890+0000 7fe037fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fa827d7d0 0x561fa8277530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53804/0 (socket says 172.21.15.33:53804) 2024-03-19T02:58:41.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.890+0000 7fe037fff640 1 -- 172.21.15.33:0/635539944 learned_addr learned my addr 172.21.15.33:0/635539944 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:41.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.890+0000 7fe036ffd640 1 --2- 172.21.15.33:0/635539944 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fa8274bf0 0x561fa8276ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.890+0000 7fe0377fe640 1 --2- 172.21.15.33:0/635539944 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa82742d0 0x561fa8285b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.890+0000 7fe037fff640 1 -- 172.21.15.33:0/635539944 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa82742d0 msgr2=0x561fa8285b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:41.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.890+0000 7fe037fff640 1 --2- 172.21.15.33:0/635539944 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa82742d0 0x561fa8285b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.890+0000 7fe037fff640 1 -- 172.21.15.33:0/635539944 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fa8274bf0 msgr2=0x561fa8276ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:41.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.890+0000 7fe037fff640 1 --2- 172.21.15.33:0/635539944 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fa8274bf0 0x561fa8276ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.890+0000 7fe037fff640 1 -- 172.21.15.33:0/635539944 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fa8277bd0 con 0x561fa827d7d0 2024-03-19T02:58:41.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.890+0000 7fe037fff640 1 --2- 172.21.15.33:0/635539944 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fa827d7d0 0x561fa8277530 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fe030009430 tx=0x7fe0300bd330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cfc040596ea042f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:41.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.890+0000 7fe016ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:58:41.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.890+0000 7fe016ffd640 1 -- 172.21.15.33:0/635539944 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0300be050 con 0x561fa827d7d0 2024-03-19T02:58:41.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.890+0000 7fe016ffd640 1 -- 172.21.15.33:0/635539944 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fe0300f1160 con 0x561fa827d7d0 2024-03-19T02:58:41.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.890+0000 7fe040b66740 1 -- 172.21.15.33:0/635539944 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe028011cb0 con 0x561fa827d7d0 2024-03-19T02:58:41.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.890+0000 7fe016ffd640 1 -- 172.21.15.33:0/635539944 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe0300bd950 con 0x561fa827d7d0 2024-03-19T02:58:41.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.890+0000 7fe040b66740 10 client.16782 fetch_fsmap learned FSMap version 25 2024-03-19T02:58:41.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.890+0000 7fe040b66740 1 -- 172.21.15.33:0/635539944 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x561fa82f3a90 con 0x561fa827d7d0 2024-03-19T02:58:41.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.891+0000 7fe016ffd640 1 -- 172.21.15.33:0/635539944 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe028011cb0 con 0x561fa827d7d0 2024-03-19T02:58:41.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.891+0000 7fe016ffd640 1 -- 172.21.15.33:0/635539944 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fe03003c030 con 0x561fa827d7d0 2024-03-19T02:58:41.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.891+0000 7fe040b66740 10 client.16782 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:58:41.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.891+0000 7fe040b66740 10 client.16782 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T02:58:41.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.891+0000 7fe040b66740 20 client.16782 populate_metadata read hostname 'smithi033' 2024-03-19T02:58:41.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.891+0000 7fe040b66740 1 --2- 172.21.15.33:0/635539944 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x561fa82f4160 0x561fa82f6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:41.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.891+0000 7fe040b66740 4 client.16782 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:58:41.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.891+0000 7fe040b66740 1 -- 172.21.15.33:0/635539944 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561fa81e49f0 con 0x561fa82f4160 2024-03-19T02:58:41.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.891+0000 7fe0377fe640 1 --2- 172.21.15.33:0/635539944 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x561fa82f4160 0x561fa82f6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:41.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.891+0000 7fe016ffd640 10 client.16782 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T02:58:41.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.891+0000 7fe0377fe640 1 --2- 172.21.15.33:0/635539944 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x561fa82f4160 0x561fa82f6580 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6bdcce1dab4e63e9 server_cookie=a4eab7cc47e93b52 in_seq=0 out_seq=0 2024-03-19T02:58:41.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.892+0000 7fe016ffd640 1 -- 172.21.15.33:0/635539944 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561fa81e49f0 con 0x561fa82f4160 2024-03-19T02:58:41.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.892+0000 7fe016ffd640 10 client.16782 handle_command_reply: tid=0 2024-03-19T02:58:41.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.892+0000 7fe040b66740 1 client.16782 shutdown 2024-03-19T02:58:41.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.892+0000 7fe040b66740 1 -- 172.21.15.33:0/635539944 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fa827d7d0 msgr2=0x561fa8277530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:41.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.892+0000 7fe040b66740 1 --2- 172.21.15.33:0/635539944 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fa827d7d0 0x561fa8277530 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fe030009430 tx=0x7fe0300bd330 comp rx=0 tx=0).stop 2024-03-19T02:58:41.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.892+0000 7fe040b66740 1 -- 172.21.15.33:0/635539944 shutdown_connections 2024-03-19T02:58:41.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.892+0000 7fe040b66740 1 --2- 172.21.15.33:0/635539944 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x561fa82f4160 0x561fa82f6580 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.892+0000 7fe040b66740 1 --2- 172.21.15.33:0/635539944 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561fa827d7d0 0x561fa8277530 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.892+0000 7fe040b66740 1 --2- 172.21.15.33:0/635539944 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561fa8274bf0 0x561fa8276ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.892+0000 7fe016ffd640 0 client.16782 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T02:58:41.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.892+0000 7fe040b66740 1 --2- 172.21.15.33:0/635539944 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561fa82742d0 0x561fa8285b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.892+0000 7fe040b66740 1 -- 172.21.15.33:0/635539944 >> 172.21.15.33:0/635539944 conn(0x561fa826ea00 msgr2=0x561fa826dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:41.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.892+0000 7fe040b66740 1 -- 172.21.15.33:0/635539944 shutdown_connections 2024-03-19T02:58:41.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.892+0000 7fe040b66740 1 -- 172.21.15.33:0/635539944 wait complete. 2024-03-19T02:58:41.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.892+0000 7fe040b66740 20 client.16782 trim_cache size 0 max 16384 2024-03-19T02:58:41.900 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:58:41.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.892+0000 7fe03de9a640 1 -- 172.21.15.33:0/3374905609 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe0200a2970 msgr2=0x7fe0200a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:41.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.892+0000 7fe03de9a640 1 --2- 172.21.15.33:0/3374905609 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe0200a2970 0x7fe0200a4e30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe038103e20 tx=0x7fe02c011c80 comp rx=0 tx=0).stop 2024-03-19T02:58:41.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.892+0000 7fe03de9a640 1 -- 172.21.15.33:0/3374905609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe038106750 msgr2=0x7fe038102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:41.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.892+0000 7fe03de9a640 1 --2- 172.21.15.33:0/3374905609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe038106750 0x7fe038102e10 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fe03002c700 tx=0x7fe030035d60 comp rx=0 tx=0).stop 2024-03-19T02:58:41.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.892+0000 7fe03de9a640 1 -- 172.21.15.33:0/3374905609 shutdown_connections 2024-03-19T02:58:41.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.892+0000 7fe03de9a640 1 --2- 172.21.15.33:0/3374905609 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe0200a2970 0x7fe0200a4e30 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.892+0000 7fe03de9a640 1 --2- 172.21.15.33:0/3374905609 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe038106750 0x7fe038102e10 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.892+0000 7fe03de9a640 1 --2- 172.21.15.33:0/3374905609 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe038101310 0x7fe0381028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.892+0000 7fe03de9a640 1 --2- 172.21.15.33:0/3374905609 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe038100970 0x7fe038102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:41.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.892+0000 7fe03de9a640 1 -- 172.21.15.33:0/3374905609 >> 172.21.15.33:0/3374905609 conn(0x7fe0380774c0 msgr2=0x7fe0380ffc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:41.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.893+0000 7fe03de9a640 1 -- 172.21.15.33:0/3374905609 shutdown_connections 2024-03-19T02:58:41.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:41.893+0000 7fe03de9a640 1 -- 172.21.15.33:0/3374905609 wait complete. 2024-03-19T02:58:41.905 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:58:42.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.008+0000 7f4d5497e640 1 -- 172.21.15.33:0/1567023314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d50106750 msgr2=0x7f4d50106b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:42.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.008+0000 7f4d5497e640 1 --2- 172.21.15.33:0/1567023314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d50106750 0x7f4d50106b30 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f4d40009900 tx=0x7f4d4002bc30 comp rx=0 tx=0).stop 2024-03-19T02:58:42.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.009+0000 7f4d5497e640 1 -- 172.21.15.33:0/1567023314 shutdown_connections 2024-03-19T02:58:42.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.009+0000 7f4d5497e640 1 --2- 172.21.15.33:0/1567023314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d50101310 0x7f4d5010d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.009+0000 7f4d5497e640 1 --2- 172.21.15.33:0/1567023314 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d50100970 0x7f4d50100dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.009+0000 7f4d5497e640 1 --2- 172.21.15.33:0/1567023314 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d50106750 0x7f4d50106b30 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.009+0000 7f4d5497e640 1 -- 172.21.15.33:0/1567023314 >> 172.21.15.33:0/1567023314 conn(0x7f4d500774c0 msgr2=0x7f4d500778d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:42.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.009+0000 7f4d5497e640 1 -- 172.21.15.33:0/1567023314 shutdown_connections 2024-03-19T02:58:42.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.009+0000 7f4d5497e640 1 -- 172.21.15.33:0/1567023314 wait complete. 2024-03-19T02:58:42.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.009+0000 7f4d5497e640 1 Processor -- start 2024-03-19T02:58:42.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.009+0000 7f4d5497e640 1 -- start start 2024-03-19T02:58:42.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.010+0000 7f4d5497e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d50100970 0x7f4d50102390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:42.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.010+0000 7f4d5497e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d50101310 0x7f4d501028d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:42.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.010+0000 7f4d5497e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d50106750 0x7f4d50102e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:42.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.010+0000 7f4d5497e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4d50110940 con 0x7f4d50106750 2024-03-19T02:58:42.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.010+0000 7f4d5497e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4d501107c0 con 0x7f4d50101310 2024-03-19T02:58:42.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.010+0000 7f4d5497e640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4d50110ac0 con 0x7f4d50100970 2024-03-19T02:58:42.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.010+0000 7f4d4ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d50106750 0x7f4d50102e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:42.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.010+0000 7f4d4ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d50106750 0x7f4d50102e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53824/0 (socket says 172.21.15.33:53824) 2024-03-19T02:58:42.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.010+0000 7f4d4ed9d640 1 -- 172.21.15.33:0/53719929 learned_addr learned my addr 172.21.15.33:0/53719929 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:42.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.010+0000 7f4d4e59c640 1 --2- 172.21.15.33:0/53719929 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d50100970 0x7f4d50102390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:42.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.010+0000 7f4d47fff640 1 --2- 172.21.15.33:0/53719929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d50101310 0x7f4d501028d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:42.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.010+0000 7f4d4ed9d640 1 -- 172.21.15.33:0/53719929 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d50100970 msgr2=0x7f4d50102390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:42.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.010+0000 7f4d4ed9d640 1 --2- 172.21.15.33:0/53719929 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d50100970 0x7f4d50102390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.010+0000 7f4d4ed9d640 1 -- 172.21.15.33:0/53719929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d50101310 msgr2=0x7f4d501028d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:42.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.010+0000 7f4d4ed9d640 1 --2- 172.21.15.33:0/53719929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d50101310 0x7f4d501028d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.010+0000 7f4d4ed9d640 1 -- 172.21.15.33:0/53719929 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d501a6430 con 0x7f4d50106750 2024-03-19T02:58:42.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.010+0000 7f4d4ed9d640 1 --2- 172.21.15.33:0/53719929 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d50106750 0x7f4d50102e10 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f4d4800efd0 tx=0x7f4d4800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:42.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.010+0000 7f4d477fe640 1 -- 172.21.15.33:0/53719929 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d48003b70 con 0x7f4d50106750 2024-03-19T02:58:42.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.010+0000 7f4d5497e640 1 -- 172.21.15.33:0/53719929 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d501a6720 con 0x7f4d50106750 2024-03-19T02:58:42.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.010+0000 7f4d477fe640 1 -- 172.21.15.33:0/53719929 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4d48003d10 con 0x7f4d50106750 2024-03-19T02:58:42.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.010+0000 7f4d477fe640 1 -- 172.21.15.33:0/53719929 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d48010980 con 0x7f4d50106750 2024-03-19T02:58:42.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.010+0000 7f4d5497e640 1 -- 172.21.15.33:0/53719929 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d501a6c20 con 0x7f4d50106750 2024-03-19T02:58:42.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.012+0000 7f4d477fe640 1 -- 172.21.15.33:0/53719929 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f4d48004020 con 0x7f4d50106750 2024-03-19T02:58:42.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.013+0000 7f4d57569740 1 Processor -- start 2024-03-19T02:58:42.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.013+0000 7f4d57569740 1 -- start start 2024-03-19T02:58:42.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.013+0000 7f4d57569740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b374782d0 0x558b374786b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:42.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.013+0000 7f4d477fe640 1 --2- 172.21.15.33:0/53719929 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4d3c0a29a0 0x7f4d3c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:42.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.013+0000 7f4d57569740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558b37478bf0 0x558b37481290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:42.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.013+0000 7f4d57569740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b374817d0 0x558b37483bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:42.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.013+0000 7f4d57569740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558b373db4b0 con 0x558b374817d0 2024-03-19T02:58:42.021 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.013+0000 7f4d57569740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x558b37383d70 con 0x558b37478bf0 2024-03-19T02:58:42.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.013+0000 7f4d57569740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558b373d7550 con 0x558b374782d0 2024-03-19T02:58:42.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.013+0000 7f4d4ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b374817d0 0x558b37483bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:42.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.013+0000 7f4d4e59c640 1 --2- 172.21.15.33:0/53719929 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4d3c0a29a0 0x7f4d3c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:42.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.013+0000 7f4d47fff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b374782d0 0x558b374786b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:42.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.013+0000 7f4d4ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b374817d0 0x558b37483bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53836/0 (socket says 172.21.15.33:53836) 2024-03-19T02:58:42.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.013+0000 7f4d4ed9d640 1 -- 172.21.15.33:0/1028054427 learned_addr learned my addr 172.21.15.33:0/1028054427 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:42.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.013+0000 7f4d477fe640 1 -- 172.21.15.33:0/53719929 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f4d480cf5a0 con 0x7f4d50106750 2024-03-19T02:58:42.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.013+0000 7f4d47fff640 1 --2- 172.21.15.33:0/1028054427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558b37478bf0 0x558b37481290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:42.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.014+0000 7f4d4ed9d640 1 -- 172.21.15.33:0/1028054427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b374782d0 msgr2=0x558b374786b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:42.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.014+0000 7f4d4ed9d640 1 --2- 172.21.15.33:0/1028054427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b374782d0 0x558b374786b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.014+0000 7f4d4ed9d640 1 -- 172.21.15.33:0/1028054427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558b37478bf0 msgr2=0x558b37481290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:42.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.014+0000 7f4d4ed9d640 1 --2- 172.21.15.33:0/1028054427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558b37478bf0 0x558b37481290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.014+0000 7f4d4ed9d640 1 -- 172.21.15.33:0/1028054427 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b373a6210 con 0x558b374817d0 2024-03-19T02:58:42.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.014+0000 7f4d4e59c640 1 --2- 172.21.15.33:0/53719929 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4d3c0a29a0 0x7f4d3c0a4e60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4d40006010 tx=0x7f4d40005e40 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:42.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.014+0000 7f4d4ed9d640 1 --2- 172.21.15.33:0/1028054427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b374817d0 0x558b37483bc0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f4d4809b7e0 tx=0x7f4d48010430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=982492aab1a24e09 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:42.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.014+0000 7f4d44ff9640 1 -- 172.21.15.33:0/1028054427 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d480b4910 con 0x558b374817d0 2024-03-19T02:58:42.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.014+0000 7f4d44ff9640 1 -- 172.21.15.33:0/1028054427 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4d48009820 con 0x558b374817d0 2024-03-19T02:58:42.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.014+0000 7f4d44ff9640 1 -- 172.21.15.33:0/1028054427 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d48004310 con 0x558b374817d0 2024-03-19T02:58:42.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.015+0000 7f4d57569740 1 -- 172.21.15.33:0/1028054427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b374817d0 msgr2=0x558b37483bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:42.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.015+0000 7f4d57569740 1 --2- 172.21.15.33:0/1028054427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b374817d0 0x558b37483bc0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f4d4809b7e0 tx=0x7f4d48010430 comp rx=0 tx=0).stop 2024-03-19T02:58:42.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.015+0000 7f4d57569740 1 -- 172.21.15.33:0/1028054427 shutdown_connections 2024-03-19T02:58:42.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.015+0000 7f4d57569740 1 --2- 172.21.15.33:0/1028054427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b374817d0 0x558b37483bc0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.015+0000 7f4d57569740 1 --2- 172.21.15.33:0/1028054427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558b37478bf0 0x558b37481290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.015+0000 7f4d57569740 1 --2- 172.21.15.33:0/1028054427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b374782d0 0x558b374786b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.015+0000 7f4d57569740 1 -- 172.21.15.33:0/1028054427 >> 172.21.15.33:0/1028054427 conn(0x558b37472a00 msgr2=0x558b37472de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:42.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.015+0000 7f4d57569740 1 -- 172.21.15.33:0/1028054427 shutdown_connections 2024-03-19T02:58:42.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.015+0000 7f4d57569740 1 -- 172.21.15.33:0/1028054427 wait complete. 2024-03-19T02:58:42.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.015+0000 7f4d57569740 1 -- start start 2024-03-19T02:58:42.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.015+0000 7f4d57569740 1 Processor -- start 2024-03-19T02:58:42.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.015+0000 7f4d57569740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b374782d0 0x558b3747e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:42.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.015+0000 7f4d57569740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558b37478bf0 0x558b3747ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:42.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.015+0000 7f4d57569740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b374817d0 0x558b3747f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:42.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.015+0000 7f4d57569740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558b37484560 con 0x558b374782d0 2024-03-19T02:58:42.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.015+0000 7f4d57569740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x558b37397310 con 0x558b37478bf0 2024-03-19T02:58:42.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.015+0000 7f4d57569740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558b374846e0 con 0x558b374817d0 2024-03-19T02:58:42.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.015+0000 7f4d57569740 10 client.0 _ll_register_callbacks cb 0x558b3738b2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:58:42.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.016+0000 7f4d4e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b374782d0 0x558b3747e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:42.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.016+0000 7f4d4e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b374782d0 0x558b3747e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53838/0 (socket says 172.21.15.33:53838) 2024-03-19T02:58:42.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.016+0000 7f4d4e59c640 1 -- 172.21.15.33:0/2807177350 learned_addr learned my addr 172.21.15.33:0/2807177350 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:42.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.016+0000 7f4d47fff640 1 --2- 172.21.15.33:0/2807177350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558b37478bf0 0x558b3747ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:42.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.016+0000 7f4d4ed9d640 1 --2- 172.21.15.33:0/2807177350 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b374817d0 0x558b3747f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:42.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.016+0000 7f4d47fff640 1 -- 172.21.15.33:0/2807177350 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b374817d0 msgr2=0x558b3747f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:42.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.016+0000 7f4d47fff640 1 --2- 172.21.15.33:0/2807177350 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b374817d0 0x558b3747f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.016+0000 7f4d47fff640 1 -- 172.21.15.33:0/2807177350 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b374782d0 msgr2=0x558b3747e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:42.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.016+0000 7f4d47fff640 1 --2- 172.21.15.33:0/2807177350 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b374782d0 0x558b3747e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.016+0000 7f4d47fff640 1 -- 172.21.15.33:0/2807177350 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b370ef3f0 con 0x558b37478bf0 2024-03-19T02:58:42.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.016+0000 7f4d4e59c640 1 --2- 172.21.15.33:0/2807177350 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b374782d0 0x558b3747e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:58:42.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.016+0000 7f4d47fff640 1 --2- 172.21.15.33:0/2807177350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558b37478bf0 0x558b3747ee20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x558b37477da0 tx=0x7f4d380039c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5bf53ef51454de2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:42.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.016+0000 7f4d35ffb640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T02:58:42.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.016+0000 7f4d35ffb640 1 -- 172.21.15.33:0/2807177350 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d38023060 con 0x558b37478bf0 2024-03-19T02:58:42.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.016+0000 7f4d57569740 1 -- 172.21.15.33:0/2807177350 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d48009820 con 0x558b37478bf0 2024-03-19T02:58:42.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.016+0000 7f4d35ffb640 1 -- 172.21.15.33:0/2807177350 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f4d380118c0 con 0x558b37478bf0 2024-03-19T02:58:42.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.016+0000 7f4d35ffb640 1 -- 172.21.15.33:0/2807177350 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4d3802a050 con 0x558b37478bf0 2024-03-19T02:58:42.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.016+0000 7f4d57569740 10 client.26618 fetch_fsmap learned FSMap version 25 2024-03-19T02:58:42.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.016+0000 7f4d57569740 1 -- 172.21.15.33:0/2807177350 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558b371b97d0 con 0x558b37478bf0 2024-03-19T02:58:42.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.017+0000 7f4d35ffb640 1 -- 172.21.15.33:0/2807177350 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4d48009820 con 0x558b37478bf0 2024-03-19T02:58:42.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.017+0000 7f4d35ffb640 1 -- 172.21.15.33:0/2807177350 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f4d3801d070 con 0x558b37478bf0 2024-03-19T02:58:42.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.017+0000 7f4d57569740 10 client.26618 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:58:42.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.017+0000 7f4d57569740 10 client.26618 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:58:42.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.017+0000 7f4d57569740 20 client.26618 populate_metadata read hostname 'smithi033' 2024-03-19T02:58:42.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.017+0000 7f4d57569740 1 --2- 172.21.15.33:0/2807177350 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x558b37491bd0 0x558b37493fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:42.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.017+0000 7f4d57569740 4 client.26618 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:58:42.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.017+0000 7f4d57569740 1 -- 172.21.15.33:0/2807177350 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b374393e0 con 0x558b37491bd0 2024-03-19T02:58:42.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.017+0000 7f4d4e59c640 1 --2- 172.21.15.33:0/2807177350 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x558b37491bd0 0x558b37493fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:42.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.017+0000 7f4d35ffb640 10 client.26618 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:58:42.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.017+0000 7f4d4e59c640 1 --2- 172.21.15.33:0/2807177350 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x558b37491bd0 0x558b37493fc0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88984159e57649ac server_cookie=95c1b257e842fa98 in_seq=0 out_seq=0 2024-03-19T02:58:42.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.018+0000 7f4d35ffb640 1 -- 172.21.15.33:0/2807177350 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558b374393e0 con 0x558b37491bd0 2024-03-19T02:58:42.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.018+0000 7f4d35ffb640 10 client.26618 handle_command_reply: tid=0 2024-03-19T02:58:42.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.018+0000 7f4d57569740 1 client.26618 shutdown 2024-03-19T02:58:42.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.018+0000 7f4d57569740 1 -- 172.21.15.33:0/2807177350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558b37478bf0 msgr2=0x558b3747ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:42.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.018+0000 7f4d57569740 1 --2- 172.21.15.33:0/2807177350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558b37478bf0 0x558b3747ee20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x558b37477da0 tx=0x7f4d380039c0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.018+0000 7f4d57569740 1 -- 172.21.15.33:0/2807177350 shutdown_connections 2024-03-19T02:58:42.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.018+0000 7f4d57569740 1 --2- 172.21.15.33:0/2807177350 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x558b37491bd0 0x558b37493fc0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.018+0000 7f4d57569740 1 --2- 172.21.15.33:0/2807177350 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b374817d0 0x558b3747f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.018+0000 7f4d35ffb640 0 client.26618 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:58:42.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.018+0000 7f4d57569740 1 --2- 172.21.15.33:0/2807177350 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558b37478bf0 0x558b3747ee20 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.018+0000 7f4d57569740 1 --2- 172.21.15.33:0/2807177350 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b374782d0 0x558b3747e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.018+0000 7f4d57569740 1 -- 172.21.15.33:0/2807177350 >> 172.21.15.33:0/2807177350 conn(0x558b37472a00 msgr2=0x558b37474750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:42.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.018+0000 7f4d57569740 1 -- 172.21.15.33:0/2807177350 shutdown_connections 2024-03-19T02:58:42.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.018+0000 7f4d57569740 1 -- 172.21.15.33:0/2807177350 wait complete. 2024-03-19T02:58:42.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.018+0000 7f4d57569740 20 client.26618 trim_cache size 0 max 16384 2024-03-19T02:58:42.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.031+0000 7f4d57569740 1 Processor -- start 2024-03-19T02:58:42.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.031+0000 7f4d57569740 1 -- start start 2024-03-19T02:58:42.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.032+0000 7f4d57569740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b374782d0 0x558b37481010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:42.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.032+0000 7f4d57569740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558b37478bf0 0x558b3748d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:42.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.032+0000 7f4d57569740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b374817d0 0x558b37499850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:42.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.032+0000 7f4d57569740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4d501b3600 con 0x558b374782d0 2024-03-19T02:58:42.038 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.032+0000 7f4d57569740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4d501b0ed0 con 0x558b37478bf0 2024-03-19T02:58:42.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.032+0000 7f4d57569740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4d501aed00 con 0x558b374817d0 2024-03-19T02:58:42.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.032+0000 7f4d4e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b374782d0 0x558b37481010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:42.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.032+0000 7f4d4e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b374782d0 0x558b37481010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53844/0 (socket says 172.21.15.33:53844) 2024-03-19T02:58:42.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.032+0000 7f4d4e59c640 1 -- 172.21.15.33:0/3160489546 learned_addr learned my addr 172.21.15.33:0/3160489546 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:42.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.032+0000 7f4d4ed9d640 1 --2- 172.21.15.33:0/3160489546 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b374817d0 0x558b37499850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:42.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.032+0000 7f4d47fff640 1 --2- 172.21.15.33:0/3160489546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558b37478bf0 0x558b3748d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:42.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.032+0000 7f4d4e59c640 1 -- 172.21.15.33:0/3160489546 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b374817d0 msgr2=0x558b37499850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:42.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.032+0000 7f4d4e59c640 1 --2- 172.21.15.33:0/3160489546 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b374817d0 0x558b37499850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.032+0000 7f4d4e59c640 1 -- 172.21.15.33:0/3160489546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558b37478bf0 msgr2=0x558b3748d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:42.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.032+0000 7f4d4e59c640 1 --2- 172.21.15.33:0/3160489546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558b37478bf0 0x558b3748d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.032+0000 7f4d4e59c640 1 -- 172.21.15.33:0/3160489546 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b37490820 con 0x558b374782d0 2024-03-19T02:58:42.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.032+0000 7f4d4e59c640 1 --2- 172.21.15.33:0/3160489546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b374782d0 0x558b37481010 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f4d40009230 tx=0x7f4d4003e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eb64ef0eb19eee2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:42.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.032+0000 7f4d37fff640 1 -- 172.21.15.33:0/3160489546 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d40059070 con 0x558b374782d0 2024-03-19T02:58:42.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.032+0000 7f4d37fff640 1 -- 172.21.15.33:0/3160489546 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4d400046c0 con 0x558b374782d0 2024-03-19T02:58:42.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.032+0000 7f4d37fff640 1 -- 172.21.15.33:0/3160489546 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d40003810 con 0x558b374782d0 2024-03-19T02:58:42.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.033+0000 7f4d57569740 1 -- 172.21.15.33:0/3160489546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b374782d0 msgr2=0x558b37481010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:42.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.033+0000 7f4d57569740 1 --2- 172.21.15.33:0/3160489546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b374782d0 0x558b37481010 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f4d40009230 tx=0x7f4d4003e000 comp rx=0 tx=0).stop 2024-03-19T02:58:42.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.033+0000 7f4d57569740 1 -- 172.21.15.33:0/3160489546 shutdown_connections 2024-03-19T02:58:42.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.033+0000 7f4d57569740 1 --2- 172.21.15.33:0/3160489546 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b374817d0 0x558b37499850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.033+0000 7f4d57569740 1 --2- 172.21.15.33:0/3160489546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558b37478bf0 0x558b3748d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.033+0000 7f4d57569740 1 --2- 172.21.15.33:0/3160489546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b374782d0 0x558b37481010 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.033+0000 7f4d57569740 1 -- 172.21.15.33:0/3160489546 >> 172.21.15.33:0/3160489546 conn(0x558b37472a00 msgr2=0x558b374746e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:42.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.033+0000 7f4d57569740 1 -- 172.21.15.33:0/3160489546 shutdown_connections 2024-03-19T02:58:42.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.033+0000 7f4d57569740 1 -- 172.21.15.33:0/3160489546 wait complete. 2024-03-19T02:58:42.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.033+0000 7f4d57569740 1 -- start start 2024-03-19T02:58:42.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.033+0000 7f4d57569740 1 Processor -- start 2024-03-19T02:58:42.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.034+0000 7f4d57569740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b374782d0 0x558b3747b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:42.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.034+0000 7f4d57569740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b37478bf0 0x558b3747bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:42.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.034+0000 7f4d57569740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558b374817d0 0x558b374f6e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:42.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.034+0000 7f4d57569740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4d501af390 con 0x558b37478bf0 2024-03-19T02:58:42.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.034+0000 7f4d57569740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4d501b0800 con 0x558b374817d0 2024-03-19T02:58:42.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.034+0000 7f4d57569740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4d501afa60 con 0x558b374782d0 2024-03-19T02:58:42.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.034+0000 7f4d57569740 10 client.0 _ll_register_callbacks cb 0x7f4d38004420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:58:42.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.034+0000 7f4d47fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b37478bf0 0x558b3747bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:42.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.034+0000 7f4d47fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b37478bf0 0x558b3747bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53846/0 (socket says 172.21.15.33:53846) 2024-03-19T02:58:42.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.034+0000 7f4d47fff640 1 -- 172.21.15.33:0/259608824 learned_addr learned my addr 172.21.15.33:0/259608824 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:42.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.034+0000 7f4d4e59c640 1 --2- 172.21.15.33:0/259608824 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b374782d0 0x558b3747b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:42.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.034+0000 7f4d4ed9d640 1 --2- 172.21.15.33:0/259608824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558b374817d0 0x558b374f6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:42.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.034+0000 7f4d47fff640 1 -- 172.21.15.33:0/259608824 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b374782d0 msgr2=0x558b3747b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:42.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.034+0000 7f4d47fff640 1 --2- 172.21.15.33:0/259608824 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b374782d0 0x558b3747b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.034+0000 7f4d47fff640 1 -- 172.21.15.33:0/259608824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558b374817d0 msgr2=0x558b374f6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:42.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.034+0000 7f4d47fff640 1 --2- 172.21.15.33:0/259608824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558b374817d0 0x558b374f6e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.034+0000 7f4d47fff640 1 -- 172.21.15.33:0/259608824 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b3747c680 con 0x558b37478bf0 2024-03-19T02:58:42.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.034+0000 7f4d47fff640 1 --2- 172.21.15.33:0/259608824 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b37478bf0 0x558b3747bd80 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x558b3747feb0 tx=0x7f4d3800eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f11d21199a85dea server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:42.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.034+0000 7f4d35ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:58:42.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.034+0000 7f4d35ffb640 1 -- 172.21.15.33:0/259608824 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d38011240 con 0x558b37478bf0 2024-03-19T02:58:42.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.034+0000 7f4d35ffb640 1 -- 172.21.15.33:0/259608824 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f4d3800e3c0 con 0x558b37478bf0 2024-03-19T02:58:42.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.034+0000 7f4d57569740 1 -- 172.21.15.33:0/259608824 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d400046c0 con 0x558b37478bf0 2024-03-19T02:58:42.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.034+0000 7f4d35ffb640 1 -- 172.21.15.33:0/259608824 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4d3800e5c0 con 0x558b37478bf0 2024-03-19T02:58:42.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.034+0000 7f4d57569740 10 client.16800 fetch_fsmap learned FSMap version 25 2024-03-19T02:58:42.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.034+0000 7f4d57569740 1 -- 172.21.15.33:0/259608824 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558b374f8520 con 0x558b37478bf0 2024-03-19T02:58:42.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.035+0000 7f4d35ffb640 1 -- 172.21.15.33:0/259608824 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4d400046c0 con 0x558b37478bf0 2024-03-19T02:58:42.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.035+0000 7f4d35ffb640 1 -- 172.21.15.33:0/259608824 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f4d3801d040 con 0x558b37478bf0 2024-03-19T02:58:42.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.035+0000 7f4d57569740 10 client.16800 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:58:42.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.035+0000 7f4d57569740 10 client.16800 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:58:42.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.035+0000 7f4d57569740 20 client.16800 populate_metadata read hostname 'smithi033' 2024-03-19T02:58:42.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.035+0000 7f4d57569740 1 --2- 172.21.15.33:0/259608824 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x558b374f8cd0 0x558b374fb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:42.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.035+0000 7f4d57569740 4 client.16800 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:58:42.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.035+0000 7f4d57569740 1 -- 172.21.15.33:0/259608824 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558b373e89f0 con 0x558b374f8cd0 2024-03-19T02:58:42.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.035+0000 7f4d4e59c640 1 --2- 172.21.15.33:0/259608824 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x558b374f8cd0 0x558b374fb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:42.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.035+0000 7f4d35ffb640 10 client.16800 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:58:42.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.035+0000 7f4d4e59c640 1 --2- 172.21.15.33:0/259608824 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x558b374f8cd0 0x558b374fb0f0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=647f9e2fdad7b78f server_cookie=1596547b2aaff5ff in_seq=0 out_seq=0 2024-03-19T02:58:42.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.036+0000 7f4d35ffb640 1 -- 172.21.15.33:0/259608824 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558b373e89f0 con 0x558b374f8cd0 2024-03-19T02:58:42.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.036+0000 7f4d35ffb640 10 client.16800 handle_command_reply: tid=0 2024-03-19T02:58:42.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.036+0000 7f4d57569740 1 client.16800 shutdown 2024-03-19T02:58:42.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.036+0000 7f4d57569740 1 -- 172.21.15.33:0/259608824 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b37478bf0 msgr2=0x558b3747bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:42.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.036+0000 7f4d57569740 1 --2- 172.21.15.33:0/259608824 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b37478bf0 0x558b3747bd80 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x558b3747feb0 tx=0x7f4d3800eee0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.036+0000 7f4d57569740 1 -- 172.21.15.33:0/259608824 shutdown_connections 2024-03-19T02:58:42.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.036+0000 7f4d57569740 1 --2- 172.21.15.33:0/259608824 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x558b374f8cd0 0x558b374fb0f0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.036+0000 7f4d57569740 1 --2- 172.21.15.33:0/259608824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558b374817d0 0x558b374f6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.036+0000 7f4d57569740 1 --2- 172.21.15.33:0/259608824 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558b37478bf0 0x558b3747bd80 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.036+0000 7f4d35ffb640 0 client.16800 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:58:42.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.036+0000 7f4d57569740 1 --2- 172.21.15.33:0/259608824 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558b374782d0 0x558b3747b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.036+0000 7f4d57569740 1 -- 172.21.15.33:0/259608824 >> 172.21.15.33:0/259608824 conn(0x558b37472a00 msgr2=0x558b3747e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:42.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.036+0000 7f4d57569740 1 -- 172.21.15.33:0/259608824 shutdown_connections 2024-03-19T02:58:42.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.036+0000 7f4d57569740 1 -- 172.21.15.33:0/259608824 wait complete. 2024-03-19T02:58:42.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.036+0000 7f4d57569740 20 client.16800 trim_cache size 0 max 16384 2024-03-19T02:58:42.044 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:58:42.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.036+0000 7f4d5497e640 1 -- 172.21.15.33:0/53719929 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4d3c0a29a0 msgr2=0x7f4d3c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:42.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.036+0000 7f4d5497e640 1 --2- 172.21.15.33:0/53719929 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4d3c0a29a0 0x7f4d3c0a4e60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4d40006010 tx=0x7f4d40005e40 comp rx=0 tx=0).stop 2024-03-19T02:58:42.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.036+0000 7f4d5497e640 1 -- 172.21.15.33:0/53719929 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d50106750 msgr2=0x7f4d50102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:42.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.036+0000 7f4d5497e640 1 --2- 172.21.15.33:0/53719929 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d50106750 0x7f4d50102e10 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f4d4800efd0 tx=0x7f4d4800c6a0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.036+0000 7f4d5497e640 1 -- 172.21.15.33:0/53719929 shutdown_connections 2024-03-19T02:58:42.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.036+0000 7f4d5497e640 1 --2- 172.21.15.33:0/53719929 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4d3c0a29a0 0x7f4d3c0a4e60 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.036+0000 7f4d5497e640 1 --2- 172.21.15.33:0/53719929 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d50106750 0x7f4d50102e10 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.036+0000 7f4d5497e640 1 --2- 172.21.15.33:0/53719929 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d50101310 0x7f4d501028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.036+0000 7f4d5497e640 1 --2- 172.21.15.33:0/53719929 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d50100970 0x7f4d50102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:42.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.036+0000 7f4d5497e640 1 -- 172.21.15.33:0/53719929 >> 172.21.15.33:0/53719929 conn(0x7f4d500774c0 msgr2=0x7f4d5006a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:42.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.036+0000 7f4d5497e640 1 -- 172.21.15.33:0/53719929 shutdown_connections 2024-03-19T02:58:42.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:42.036+0000 7f4d5497e640 1 -- 172.21.15.33:0/53719929 wait complete. 2024-03-19T02:58:42.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:42 smithi096 ceph-mon[35946]: pgmap v160: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:58:42.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:42 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1529678464' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:58:43.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:42 smithi155 ceph-mon[35804]: pgmap v160: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:58:43.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:42 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1529678464' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:58:43.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:42 smithi033 ceph-mon[31973]: pgmap v160: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:58:43.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:42 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1529678464' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:58:43.049 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:58:43.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:58:43.049 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:58:43.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.168+0000 7f1839205640 1 -- 172.21.15.33:0/3391289494 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18341077a0 msgr2=0x7f1834107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:43.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.168+0000 7f1839205640 1 --2- 172.21.15.33:0/3391289494 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18341077a0 0x7f1834107b80 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f182c009930 tx=0x7f182c02bc30 comp rx=0 tx=0).stop 2024-03-19T02:58:43.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.169+0000 7f1839205640 1 -- 172.21.15.33:0/3391289494 shutdown_connections 2024-03-19T02:58:43.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.169+0000 7f1839205640 1 --2- 172.21.15.33:0/3391289494 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1834068a00 0x7f1834068e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.169+0000 7f1839205640 1 --2- 172.21.15.33:0/3391289494 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f18340693c0 0x7f183410e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.169+0000 7f1839205640 1 --2- 172.21.15.33:0/3391289494 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18341077a0 0x7f1834107b80 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.169+0000 7f1839205640 1 -- 172.21.15.33:0/3391289494 >> 172.21.15.33:0/3391289494 conn(0x7f1834077820 msgr2=0x7f1834077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:43.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.169+0000 7f1839205640 1 -- 172.21.15.33:0/3391289494 shutdown_connections 2024-03-19T02:58:43.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.169+0000 7f1839205640 1 -- 172.21.15.33:0/3391289494 wait complete. 2024-03-19T02:58:43.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.169+0000 7f1839205640 1 Processor -- start 2024-03-19T02:58:43.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.170+0000 7f1839205640 1 -- start start 2024-03-19T02:58:43.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.170+0000 7f1839205640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1834068a00 0x7f18341006e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.170+0000 7f1839205640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18340693c0 0x7f1834100c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.170+0000 7f1839205640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18341077a0 0x7f1834101160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.170+0000 7f1839205640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1834111a60 con 0x7f18341077a0 2024-03-19T02:58:43.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.170+0000 7f1839205640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f18341118e0 con 0x7f1834068a00 2024-03-19T02:58:43.173 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.170+0000 7f1839205640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1834111be0 con 0x7f18340693c0 2024-03-19T02:58:43.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.170+0000 7f1832d9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1834068a00 0x7f18341006e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.170+0000 7f183359e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18341077a0 0x7f1834101160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.170+0000 7f1832d9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1834068a00 0x7f18341006e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:57206/0 (socket says 172.21.15.33:57206) 2024-03-19T02:58:43.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.170+0000 7f183359e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18341077a0 0x7f1834101160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53860/0 (socket says 172.21.15.33:53860) 2024-03-19T02:58:43.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.170+0000 7f1832d9d640 1 -- 172.21.15.33:0/3555418711 learned_addr learned my addr 172.21.15.33:0/3555418711 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:43.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.170+0000 7f183259c640 1 --2- 172.21.15.33:0/3555418711 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18340693c0 0x7f1834100c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.170+0000 7f1832d9d640 1 -- 172.21.15.33:0/3555418711 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18340693c0 msgr2=0x7f1834100c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:43.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.170+0000 7f1832d9d640 1 --2- 172.21.15.33:0/3555418711 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18340693c0 0x7f1834100c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.170+0000 7f1832d9d640 1 -- 172.21.15.33:0/3555418711 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18341077a0 msgr2=0x7f1834101160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:43.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.170+0000 7f1832d9d640 1 --2- 172.21.15.33:0/3555418711 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18341077a0 0x7f1834101160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.170+0000 7f1832d9d640 1 -- 172.21.15.33:0/3555418711 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18341a6530 con 0x7f1834068a00 2024-03-19T02:58:43.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.170+0000 7f183259c640 1 --2- 172.21.15.33:0/3555418711 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18340693c0 0x7f1834100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:58:43.174 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.171+0000 7f1832d9d640 1 --2- 172.21.15.33:0/3555418711 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1834068a00 0x7f18341006e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f182c009230 tx=0x7f182c02c5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:43.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.171+0000 7f182bfff640 1 -- 172.21.15.33:0/3555418711 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c043020 con 0x7f1834068a00 2024-03-19T02:58:43.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.171+0000 7f1839205640 1 -- 172.21.15.33:0/3555418711 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18341a67c0 con 0x7f1834068a00 2024-03-19T02:58:43.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.171+0000 7f182bfff640 1 -- 172.21.15.33:0/3555418711 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f182c02ca70 con 0x7f1834068a00 2024-03-19T02:58:43.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.171+0000 7f182bfff640 1 -- 172.21.15.33:0/3555418711 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c003a50 con 0x7f1834068a00 2024-03-19T02:58:43.175 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.171+0000 7f1839205640 1 -- 172.21.15.33:0/3555418711 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18341a6d60 con 0x7f1834068a00 2024-03-19T02:58:43.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.173+0000 7f182bfff640 1 -- 172.21.15.33:0/3555418711 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f182c046020 con 0x7f1834068a00 2024-03-19T02:58:43.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.173+0000 7f183bdf2740 1 Processor -- start 2024-03-19T02:58:43.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.173+0000 7f183bdf2740 1 -- start start 2024-03-19T02:58:43.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.174+0000 7f182bfff640 1 --2- 172.21.15.33:0/3555418711 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f18200a2970 0x7f18200a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.174+0000 7f183bdf2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cbc6758380 0x55cbc6758760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.174+0000 7f183bdf2740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cbc6758ca0 0x55cbc6761340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.174+0000 7f183bdf2740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cbc6761880 0x55cbc6763c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.174+0000 7f183bdf2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55cbc66bc160 con 0x55cbc6758380 2024-03-19T02:58:43.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.174+0000 7f183bdf2740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55cbc6664e10 con 0x55cbc6758ca0 2024-03-19T02:58:43.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.174+0000 7f183bdf2740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55cbc66b8200 con 0x55cbc6761880 2024-03-19T02:58:43.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.174+0000 7f1832d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cbc6758380 0x55cbc6758760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.174+0000 7f183359e640 1 --2- 172.21.15.33:0/3555418711 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f18200a2970 0x7f18200a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.174+0000 7f1832d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cbc6758380 0x55cbc6758760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53870/0 (socket says 172.21.15.33:53870) 2024-03-19T02:58:43.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.174+0000 7f1832d9d640 1 -- 172.21.15.33:0/3037469662 learned_addr learned my addr 172.21.15.33:0/3037469662 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:43.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.174+0000 7f183259c640 1 --2- 172.21.15.33:0/3037469662 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cbc6758ca0 0x55cbc6761340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.174+0000 7f182bfff640 1 -- 172.21.15.33:0/3555418711 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f182c0f10e0 con 0x7f1834068a00 2024-03-19T02:58:43.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.174+0000 7f183359e640 1 --2- 172.21.15.33:0/3037469662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cbc6761880 0x55cbc6763c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.174+0000 7f183259c640 1 -- 172.21.15.33:0/3037469662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cbc6761880 msgr2=0x55cbc6763c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:43.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.174+0000 7f183259c640 1 --2- 172.21.15.33:0/3037469662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cbc6761880 0x55cbc6763c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.174+0000 7f183259c640 1 -- 172.21.15.33:0/3037469662 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cbc6758380 msgr2=0x55cbc6758760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:43.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.174+0000 7f183259c640 1 --2- 172.21.15.33:0/3037469662 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cbc6758380 0x55cbc6758760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.174+0000 7f183259c640 1 -- 172.21.15.33:0/3037469662 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbc64987e0 con 0x55cbc6758ca0 2024-03-19T02:58:43.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.174+0000 7f1832d9d640 1 --2- 172.21.15.33:0/3037469662 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cbc6758380 0x55cbc6758760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:58:43.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.174+0000 7f183359e640 1 --2- 172.21.15.33:0/3555418711 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f18200a2970 0x7f18200a4e30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1824009fd0 tx=0x7f1824013b00 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:43.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.174+0000 7f183259c640 1 --2- 172.21.15.33:0/3037469662 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cbc6758ca0 0x55cbc6761340 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f181c009fa0 tx=0x7f181c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a27468488705f249 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:43.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.175+0000 7f18297fa640 1 -- 172.21.15.33:0/3037469662 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c004310 con 0x55cbc6758ca0 2024-03-19T02:58:43.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.175+0000 7f18297fa640 1 -- 172.21.15.33:0/3037469662 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f181c0044b0 con 0x55cbc6758ca0 2024-03-19T02:58:43.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.175+0000 7f18297fa640 1 -- 172.21.15.33:0/3037469662 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c0035f0 con 0x55cbc6758ca0 2024-03-19T02:58:43.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.175+0000 7f183bdf2740 1 -- 172.21.15.33:0/3037469662 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cbc6758ca0 msgr2=0x55cbc6761340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:43.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.175+0000 7f183bdf2740 1 --2- 172.21.15.33:0/3037469662 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cbc6758ca0 0x55cbc6761340 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f181c009fa0 tx=0x7f181c0118d0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.175+0000 7f183bdf2740 1 -- 172.21.15.33:0/3037469662 shutdown_connections 2024-03-19T02:58:43.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.175+0000 7f183bdf2740 1 --2- 172.21.15.33:0/3037469662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cbc6761880 0x55cbc6763c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.175+0000 7f183bdf2740 1 --2- 172.21.15.33:0/3037469662 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cbc6758ca0 0x55cbc6761340 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.175+0000 7f183bdf2740 1 --2- 172.21.15.33:0/3037469662 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cbc6758380 0x55cbc6758760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.175+0000 7f183bdf2740 1 -- 172.21.15.33:0/3037469662 >> 172.21.15.33:0/3037469662 conn(0x55cbc6752ab0 msgr2=0x55cbc6752e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:43.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.176+0000 7f183bdf2740 1 -- 172.21.15.33:0/3037469662 shutdown_connections 2024-03-19T02:58:43.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.176+0000 7f183bdf2740 1 -- 172.21.15.33:0/3037469662 wait complete. 2024-03-19T02:58:43.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.176+0000 7f183bdf2740 1 -- start start 2024-03-19T02:58:43.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.176+0000 7f183bdf2740 1 Processor -- start 2024-03-19T02:58:43.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.176+0000 7f183bdf2740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cbc6758380 0x55cbc675e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.176+0000 7f183bdf2740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cbc6758ca0 0x55cbc675edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.176+0000 7f183bdf2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cbc6761880 0x55cbc675f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.176+0000 7f183bdf2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55cbc6764670 con 0x55cbc6761880 2024-03-19T02:58:43.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.176+0000 7f183bdf2740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55cbc66781a0 con 0x55cbc6758ca0 2024-03-19T02:58:43.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.176+0000 7f183bdf2740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55cbc67647f0 con 0x55cbc6758380 2024-03-19T02:58:43.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.176+0000 7f183bdf2740 10 client.0 _ll_register_callbacks cb 0x55cbc63cfc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:58:43.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.176+0000 7f183359e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cbc6761880 0x55cbc675f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.176+0000 7f183359e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cbc6761880 0x55cbc675f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53874/0 (socket says 172.21.15.33:53874) 2024-03-19T02:58:43.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.176+0000 7f183359e640 1 -- 172.21.15.33:0/3271613009 learned_addr learned my addr 172.21.15.33:0/3271613009 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:43.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.176+0000 7f183259c640 1 --2- 172.21.15.33:0/3271613009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cbc6758ca0 0x55cbc675edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.176+0000 7f1832d9d640 1 --2- 172.21.15.33:0/3271613009 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cbc6758380 0x55cbc675e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.177+0000 7f183359e640 1 -- 172.21.15.33:0/3271613009 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cbc6758380 msgr2=0x55cbc675e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:43.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.177+0000 7f183359e640 1 --2- 172.21.15.33:0/3271613009 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cbc6758380 0x55cbc675e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.177+0000 7f183359e640 1 -- 172.21.15.33:0/3271613009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cbc6758ca0 msgr2=0x55cbc675edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:43.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.177+0000 7f183359e640 1 --2- 172.21.15.33:0/3271613009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cbc6758ca0 0x55cbc675edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.177+0000 7f183359e640 1 -- 172.21.15.33:0/3271613009 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbc64f8f70 con 0x55cbc6761880 2024-03-19T02:58:43.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.177+0000 7f183359e640 1 --2- 172.21.15.33:0/3271613009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cbc6761880 0x55cbc675f300 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f1824003d90 tx=0x7f1824004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=490f4abd71abe4d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:43.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.177+0000 7f181a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:58:43.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.177+0000 7f181a7fc640 1 -- 172.21.15.33:0/3271613009 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1824024020 con 0x55cbc6761880 2024-03-19T02:58:43.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.177+0000 7f181a7fc640 1 -- 172.21.15.33:0/3271613009 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f1824004240 con 0x55cbc6761880 2024-03-19T02:58:43.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.177+0000 7f183bdf2740 1 -- 172.21.15.33:0/3271613009 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f181c0044b0 con 0x55cbc6761880 2024-03-19T02:58:43.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.177+0000 7f181a7fc640 1 -- 172.21.15.33:0/3271613009 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f182402f050 con 0x55cbc6761880 2024-03-19T02:58:43.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.177+0000 7f183bdf2740 10 client.16812 fetch_fsmap learned FSMap version 25 2024-03-19T02:58:43.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.177+0000 7f183bdf2740 1 -- 172.21.15.33:0/3271613009 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55cbc6390170 con 0x55cbc6761880 2024-03-19T02:58:43.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.177+0000 7f181a7fc640 1 -- 172.21.15.33:0/3271613009 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f181c0044b0 con 0x55cbc6761880 2024-03-19T02:58:43.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.178+0000 7f181a7fc640 1 -- 172.21.15.33:0/3271613009 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f182401f070 con 0x55cbc6761880 2024-03-19T02:58:43.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.178+0000 7f183bdf2740 10 client.16812 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:58:43.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.178+0000 7f183bdf2740 10 client.16812 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:58:43.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.178+0000 7f183bdf2740 20 client.16812 populate_metadata read hostname 'smithi033' 2024-03-19T02:58:43.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.178+0000 7f183bdf2740 1 --2- 172.21.15.33:0/3271613009 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55cbc6771c10 0x55cbc6774040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.178+0000 7f183bdf2740 4 client.16812 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:58:43.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.178+0000 7f183bdf2740 1 -- 172.21.15.33:0/3271613009 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cbc66c96a0 con 0x55cbc6771c10 2024-03-19T02:58:43.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.178+0000 7f1832d9d640 1 --2- 172.21.15.33:0/3271613009 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55cbc6771c10 0x55cbc6774040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.178+0000 7f1832d9d640 1 --2- 172.21.15.33:0/3271613009 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55cbc6771c10 0x55cbc6774040 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f54a93949117742b server_cookie=224a68c5fd20c51 in_seq=0 out_seq=0 2024-03-19T02:58:43.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.178+0000 7f181a7fc640 10 client.16812 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:58:43.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.179+0000 7f181a7fc640 1 -- 172.21.15.33:0/3271613009 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cbc66c96a0 con 0x55cbc6771c10 2024-03-19T02:58:43.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.179+0000 7f181a7fc640 10 client.16812 handle_command_reply: tid=0 2024-03-19T02:58:43.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.179+0000 7f183bdf2740 1 client.16812 shutdown 2024-03-19T02:58:43.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.179+0000 7f183bdf2740 1 -- 172.21.15.33:0/3271613009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cbc6761880 msgr2=0x55cbc675f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:43.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.179+0000 7f183bdf2740 1 --2- 172.21.15.33:0/3271613009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cbc6761880 0x55cbc675f300 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f1824003d90 tx=0x7f1824004ab0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.179+0000 7f183bdf2740 1 -- 172.21.15.33:0/3271613009 shutdown_connections 2024-03-19T02:58:43.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.179+0000 7f183bdf2740 1 --2- 172.21.15.33:0/3271613009 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55cbc6771c10 0x55cbc6774040 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.179+0000 7f183bdf2740 1 --2- 172.21.15.33:0/3271613009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cbc6761880 0x55cbc675f300 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.179+0000 7f183bdf2740 1 --2- 172.21.15.33:0/3271613009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cbc6758ca0 0x55cbc675edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.179+0000 7f181a7fc640 0 client.16812 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:58:43.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.179+0000 7f183bdf2740 1 --2- 172.21.15.33:0/3271613009 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cbc6758380 0x55cbc675e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.179+0000 7f183bdf2740 1 -- 172.21.15.33:0/3271613009 >> 172.21.15.33:0/3271613009 conn(0x55cbc6752ab0 msgr2=0x55cbc6754800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:43.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.179+0000 7f183bdf2740 1 -- 172.21.15.33:0/3271613009 shutdown_connections 2024-03-19T02:58:43.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.179+0000 7f183bdf2740 1 -- 172.21.15.33:0/3271613009 wait complete. 2024-03-19T02:58:43.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.179+0000 7f183bdf2740 20 client.16812 trim_cache size 0 max 16384 2024-03-19T02:58:43.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.197+0000 7f183bdf2740 1 Processor -- start 2024-03-19T02:58:43.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.197+0000 7f183bdf2740 1 -- start start 2024-03-19T02:58:43.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.198+0000 7f183bdf2740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cbc6758380 0x55cbc6760fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.198+0000 7f183bdf2740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cbc6758ca0 0x55cbc676d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.198+0000 7f183bdf2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cbc6761880 0x55cbc6779910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.198+0000 7f183bdf2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f18341b3630 con 0x55cbc6761880 2024-03-19T02:58:43.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.198+0000 7f183bdf2740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f18341b0f00 con 0x55cbc6758ca0 2024-03-19T02:58:43.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.198+0000 7f183bdf2740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f18341aed30 con 0x55cbc6758380 2024-03-19T02:58:43.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.198+0000 7f183359e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cbc6761880 0x55cbc6779910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.198+0000 7f183359e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cbc6761880 0x55cbc6779910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53876/0 (socket says 172.21.15.33:53876) 2024-03-19T02:58:43.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.198+0000 7f183359e640 1 -- 172.21.15.33:0/489417961 learned_addr learned my addr 172.21.15.33:0/489417961 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:43.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.198+0000 7f1832d9d640 1 --2- 172.21.15.33:0/489417961 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cbc6758380 0x55cbc6760fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.198+0000 7f183259c640 1 --2- 172.21.15.33:0/489417961 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cbc6758ca0 0x55cbc676d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.198+0000 7f183359e640 1 -- 172.21.15.33:0/489417961 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cbc6758380 msgr2=0x55cbc6760fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:43.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.198+0000 7f183359e640 1 --2- 172.21.15.33:0/489417961 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cbc6758380 0x55cbc6760fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.198+0000 7f183359e640 1 -- 172.21.15.33:0/489417961 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cbc6758ca0 msgr2=0x55cbc676d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:43.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.198+0000 7f183359e640 1 --2- 172.21.15.33:0/489417961 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cbc6758ca0 0x55cbc676d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.198+0000 7f183359e640 1 -- 172.21.15.33:0/489417961 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbc6758ad0 con 0x55cbc6761880 2024-03-19T02:58:43.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.198+0000 7f1832d9d640 1 --2- 172.21.15.33:0/489417961 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cbc6758380 0x55cbc6760fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:58:43.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.198+0000 7f183259c640 1 --2- 172.21.15.33:0/489417961 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cbc6758ca0 0x55cbc676d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:58:43.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.198+0000 7f183359e640 1 --2- 172.21.15.33:0/489417961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cbc6761880 0x55cbc6779910 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f1824010c60 tx=0x7f182400f560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ab366c6a574fe92 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:43.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.199+0000 7f1828ff9640 1 -- 172.21.15.33:0/489417961 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1824024070 con 0x55cbc6761880 2024-03-19T02:58:43.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.199+0000 7f1828ff9640 1 -- 172.21.15.33:0/489417961 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1824010e50 con 0x55cbc6761880 2024-03-19T02:58:43.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.199+0000 7f1828ff9640 1 -- 172.21.15.33:0/489417961 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1824004780 con 0x55cbc6761880 2024-03-19T02:58:43.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.199+0000 7f183bdf2740 1 -- 172.21.15.33:0/489417961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cbc6761880 msgr2=0x55cbc6779910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:43.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.199+0000 7f183bdf2740 1 --2- 172.21.15.33:0/489417961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cbc6761880 0x55cbc6779910 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f1824010c60 tx=0x7f182400f560 comp rx=0 tx=0).stop 2024-03-19T02:58:43.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.199+0000 7f183bdf2740 1 -- 172.21.15.33:0/489417961 shutdown_connections 2024-03-19T02:58:43.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.199+0000 7f183bdf2740 1 --2- 172.21.15.33:0/489417961 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cbc6761880 0x55cbc6779910 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.199+0000 7f183bdf2740 1 --2- 172.21.15.33:0/489417961 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cbc6758ca0 0x55cbc676d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.199+0000 7f183bdf2740 1 --2- 172.21.15.33:0/489417961 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cbc6758380 0x55cbc6760fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.199+0000 7f183bdf2740 1 -- 172.21.15.33:0/489417961 >> 172.21.15.33:0/489417961 conn(0x55cbc6752ab0 msgr2=0x55cbc6754770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:43.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.199+0000 7f183bdf2740 1 -- 172.21.15.33:0/489417961 shutdown_connections 2024-03-19T02:58:43.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.199+0000 7f183bdf2740 1 -- 172.21.15.33:0/489417961 wait complete. 2024-03-19T02:58:43.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.200+0000 7f183bdf2740 1 -- start start 2024-03-19T02:58:43.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.200+0000 7f183bdf2740 1 Processor -- start 2024-03-19T02:58:43.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.200+0000 7f183bdf2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cbc6758380 0x55cbc6765b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.200+0000 7f183bdf2740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cbc6758ca0 0x55cbc6766040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.200+0000 7f183bdf2740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cbc6761880 0x55cbc67d6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.200+0000 7f183bdf2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f18341af3c0 con 0x55cbc6758380 2024-03-19T02:58:43.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.200+0000 7f183bdf2740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f18341b0830 con 0x55cbc6761880 2024-03-19T02:58:43.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.200+0000 7f183bdf2740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f18341afa90 con 0x55cbc6758ca0 2024-03-19T02:58:43.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.200+0000 7f183bdf2740 10 client.0 _ll_register_callbacks cb 0x7f1824012070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:58:43.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.200+0000 7f1832d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cbc6758380 0x55cbc6765b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.200+0000 7f1832d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cbc6758380 0x55cbc6765b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53880/0 (socket says 172.21.15.33:53880) 2024-03-19T02:58:43.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.200+0000 7f1832d9d640 1 -- 172.21.15.33:0/1188391653 learned_addr learned my addr 172.21.15.33:0/1188391653 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:43.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.200+0000 7f183259c640 1 --2- 172.21.15.33:0/1188391653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cbc6758ca0 0x55cbc6766040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.200+0000 7f183359e640 1 --2- 172.21.15.33:0/1188391653 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cbc6761880 0x55cbc67d6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.200+0000 7f1832d9d640 1 -- 172.21.15.33:0/1188391653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cbc6758ca0 msgr2=0x55cbc6766040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:43.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.200+0000 7f1832d9d640 1 --2- 172.21.15.33:0/1188391653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cbc6758ca0 0x55cbc6766040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.200+0000 7f1832d9d640 1 -- 172.21.15.33:0/1188391653 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cbc6761880 msgr2=0x55cbc67d6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:43.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.200+0000 7f1832d9d640 1 --2- 172.21.15.33:0/1188391653 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cbc6761880 0x55cbc67d6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.200+0000 7f1832d9d640 1 -- 172.21.15.33:0/1188391653 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbc67d74d0 con 0x55cbc6758380 2024-03-19T02:58:43.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.200+0000 7f183359e640 1 --2- 172.21.15.33:0/1188391653 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cbc6761880 0x55cbc67d6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:58:43.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.201+0000 7f1832d9d640 1 --2- 172.21.15.33:0/1188391653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cbc6758380 0x55cbc6765b00 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f182c0bd430 tx=0x7f182c02c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91efaa2a535d5106 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:43.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.201+0000 7f181a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:58:43.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.201+0000 7f181a7fc640 1 -- 172.21.15.33:0/1188391653 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c0bd4a0 con 0x55cbc6758380 2024-03-19T02:58:43.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.201+0000 7f181a7fc640 1 -- 172.21.15.33:0/1188391653 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f182c0b9b10 con 0x55cbc6758380 2024-03-19T02:58:43.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.201+0000 7f183bdf2740 1 -- 172.21.15.33:0/1188391653 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1824010e50 con 0x55cbc6758380 2024-03-19T02:58:43.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.201+0000 7f181a7fc640 1 -- 172.21.15.33:0/1188391653 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f182c0c15f0 con 0x55cbc6758380 2024-03-19T02:58:43.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.201+0000 7f183bdf2740 10 client.16818 fetch_fsmap learned FSMap version 25 2024-03-19T02:58:43.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.201+0000 7f183bdf2740 1 -- 172.21.15.33:0/1188391653 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55cbc67d8870 con 0x55cbc6758380 2024-03-19T02:58:43.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.201+0000 7f181a7fc640 1 -- 172.21.15.33:0/1188391653 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1824010e50 con 0x55cbc6758380 2024-03-19T02:58:43.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.201+0000 7f181a7fc640 1 -- 172.21.15.33:0/1188391653 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f182c0ee700 con 0x55cbc6758380 2024-03-19T02:58:43.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.201+0000 7f183bdf2740 10 client.16818 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:58:43.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.201+0000 7f183bdf2740 10 client.16818 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:58:43.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.201+0000 7f183bdf2740 20 client.16818 populate_metadata read hostname 'smithi033' 2024-03-19T02:58:43.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.201+0000 7f183bdf2740 1 --2- 172.21.15.33:0/1188391653 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55cbc67d8f20 0x55cbc67db340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.201+0000 7f183bdf2740 4 client.16818 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:58:43.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.201+0000 7f183bdf2740 1 -- 172.21.15.33:0/1188391653 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55cbc66c39f0 con 0x55cbc67d8f20 2024-03-19T02:58:43.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.201+0000 7f183259c640 1 --2- 172.21.15.33:0/1188391653 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55cbc67d8f20 0x55cbc67db340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.202+0000 7f181a7fc640 10 client.16818 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:58:43.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.202+0000 7f183259c640 1 --2- 172.21.15.33:0/1188391653 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55cbc67d8f20 0x55cbc67db340 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7e1518ff3e10d6e server_cookie=7105a3498e318bec in_seq=0 out_seq=0 2024-03-19T02:58:43.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.203+0000 7f181a7fc640 1 -- 172.21.15.33:0/1188391653 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55cbc66c39f0 con 0x55cbc67d8f20 2024-03-19T02:58:43.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.203+0000 7f181a7fc640 10 client.16818 handle_command_reply: tid=0 2024-03-19T02:58:43.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.203+0000 7f183bdf2740 1 client.16818 shutdown 2024-03-19T02:58:43.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.203+0000 7f183bdf2740 1 -- 172.21.15.33:0/1188391653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cbc6758380 msgr2=0x55cbc6765b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:43.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.203+0000 7f183bdf2740 1 --2- 172.21.15.33:0/1188391653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cbc6758380 0x55cbc6765b00 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f182c0bd430 tx=0x7f182c02c740 comp rx=0 tx=0).stop 2024-03-19T02:58:43.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.203+0000 7f183bdf2740 1 -- 172.21.15.33:0/1188391653 shutdown_connections 2024-03-19T02:58:43.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.203+0000 7f183bdf2740 1 --2- 172.21.15.33:0/1188391653 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55cbc67d8f20 0x55cbc67db340 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.203+0000 7f183bdf2740 1 --2- 172.21.15.33:0/1188391653 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cbc6758ca0 0x55cbc6766040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.203+0000 7f183bdf2740 1 --2- 172.21.15.33:0/1188391653 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cbc6761880 0x55cbc67d6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.203+0000 7f183bdf2740 1 --2- 172.21.15.33:0/1188391653 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cbc6758380 0x55cbc6765b00 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.203+0000 7f183bdf2740 1 -- 172.21.15.33:0/1188391653 >> 172.21.15.33:0/1188391653 conn(0x55cbc6752ab0 msgr2=0x55cbc67512a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:43.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.203+0000 7f181a7fc640 0 client.16818 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:58:43.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.203+0000 7f183bdf2740 1 -- 172.21.15.33:0/1188391653 shutdown_connections 2024-03-19T02:58:43.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.203+0000 7f183bdf2740 1 -- 172.21.15.33:0/1188391653 wait complete. 2024-03-19T02:58:43.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.203+0000 7f183bdf2740 20 client.16818 trim_cache size 0 max 16384 2024-03-19T02:58:43.210 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:58:43.211 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-03-19T02:58:43.211 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "1e86cc18-be47-4033-a4e7-1074918c2dd7", 2024-03-19T02:58:43.211 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-03-19T02:58:43.211 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:58:43.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.203+0000 7f1839205640 1 -- 172.21.15.33:0/3555418711 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f18200a2970 msgr2=0x7f18200a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:43.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.203+0000 7f1839205640 1 --2- 172.21.15.33:0/3555418711 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f18200a2970 0x7f18200a4e30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1824009fd0 tx=0x7f1824013b00 comp rx=0 tx=0).stop 2024-03-19T02:58:43.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.203+0000 7f1839205640 1 -- 172.21.15.33:0/3555418711 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1834068a00 msgr2=0x7f18341006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:43.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.203+0000 7f1839205640 1 --2- 172.21.15.33:0/3555418711 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1834068a00 0x7f18341006e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f182c009230 tx=0x7f182c02c5d0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.204+0000 7f1839205640 1 -- 172.21.15.33:0/3555418711 shutdown_connections 2024-03-19T02:58:43.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.204+0000 7f1839205640 1 --2- 172.21.15.33:0/3555418711 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f18200a2970 0x7f18200a4e30 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.204+0000 7f1839205640 1 --2- 172.21.15.33:0/3555418711 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18340693c0 0x7f1834100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.204+0000 7f1839205640 1 --2- 172.21.15.33:0/3555418711 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f18341077a0 0x7f1834101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.204+0000 7f1839205640 1 --2- 172.21.15.33:0/3555418711 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1834068a00 0x7f18341006e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.204+0000 7f1839205640 1 -- 172.21.15.33:0/3555418711 >> 172.21.15.33:0/3555418711 conn(0x7f1834077820 msgr2=0x7f18340fe400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:43.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.204+0000 7f1839205640 1 -- 172.21.15.33:0/3555418711 shutdown_connections 2024-03-19T02:58:43.212 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.204+0000 7f1839205640 1 -- 172.21.15.33:0/3555418711 wait complete. 2024-03-19T02:58:43.216 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:58:43.216 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:58:43.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.320+0000 7fb67d80b640 1 -- 172.21.15.33:0/3276146234 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb678106750 msgr2=0x7fb678106b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:43.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.320+0000 7fb67d80b640 1 --2- 172.21.15.33:0/3276146234 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb678106750 0x7fb678106b30 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb668009930 tx=0x7fb66802bc30 comp rx=0 tx=0).stop 2024-03-19T02:58:43.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.320+0000 7fb67d80b640 1 -- 172.21.15.33:0/3276146234 shutdown_connections 2024-03-19T02:58:43.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.320+0000 7fb67d80b640 1 --2- 172.21.15.33:0/3276146234 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb678101100 0x7fb67810d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.320+0000 7fb67d80b640 1 --2- 172.21.15.33:0/3276146234 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb678100740 0x7fb678100bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.320+0000 7fb67d80b640 1 --2- 172.21.15.33:0/3276146234 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb678106750 0x7fb678106b30 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.320+0000 7fb67d80b640 1 -- 172.21.15.33:0/3276146234 >> 172.21.15.33:0/3276146234 conn(0x7fb678077880 msgr2=0x7fb678077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:43.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.320+0000 7fb67d80b640 1 -- 172.21.15.33:0/3276146234 shutdown_connections 2024-03-19T02:58:43.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.320+0000 7fb67d80b640 1 -- 172.21.15.33:0/3276146234 wait complete. 2024-03-19T02:58:43.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.320+0000 7fb67d80b640 1 Processor -- start 2024-03-19T02:58:43.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.320+0000 7fb67d80b640 1 -- start start 2024-03-19T02:58:43.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.321+0000 7fb67d80b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb678100740 0x7fb678198e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.321+0000 7fb67d80b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb678101100 0x7fb6781993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.321+0000 7fb67d80b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb678106750 0x7fb6781a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.321+0000 7fb67d80b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb6781109d0 con 0x7fb678101100 2024-03-19T02:58:43.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.321+0000 7fb67d80b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb678110850 con 0x7fb678100740 2024-03-19T02:58:43.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.321+0000 7fb67d80b640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb678110b50 con 0x7fb678106750 2024-03-19T02:58:43.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.321+0000 7fb66ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb678101100 0x7fb6781993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.321+0000 7fb676ffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb678100740 0x7fb678198e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.321+0000 7fb676ffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb678100740 0x7fb678198e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:57256/0 (socket says 172.21.15.33:57256) 2024-03-19T02:58:43.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.321+0000 7fb676ffd640 1 -- 172.21.15.33:0/784980196 learned_addr learned my addr 172.21.15.33:0/784980196 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:43.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.321+0000 7fb66ffff640 1 --2- 172.21.15.33:0/784980196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb678101100 0x7fb6781993b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53902/0 (socket says 172.21.15.33:53902) 2024-03-19T02:58:43.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.321+0000 7fb6777fe640 1 --2- 172.21.15.33:0/784980196 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb678106750 0x7fb6781a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.321+0000 7fb676ffd640 1 -- 172.21.15.33:0/784980196 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb678106750 msgr2=0x7fb6781a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:43.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.321+0000 7fb676ffd640 1 --2- 172.21.15.33:0/784980196 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb678106750 0x7fb6781a3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.321+0000 7fb676ffd640 1 -- 172.21.15.33:0/784980196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb678101100 msgr2=0x7fb6781993b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:43.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.321+0000 7fb676ffd640 1 --2- 172.21.15.33:0/784980196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb678101100 0x7fb6781993b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.321+0000 7fb676ffd640 1 -- 172.21.15.33:0/784980196 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6781a4430 con 0x7fb678100740 2024-03-19T02:58:43.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.321+0000 7fb66ffff640 1 --2- 172.21.15.33:0/784980196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb678101100 0x7fb6781993b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:58:43.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.321+0000 7fb676ffd640 1 --2- 172.21.15.33:0/784980196 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb678100740 0x7fb678198e70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb668009230 tx=0x7fb66802c5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:43.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.321+0000 7fb67c809640 1 -- 172.21.15.33:0/784980196 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb668043020 con 0x7fb678100740 2024-03-19T02:58:43.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.321+0000 7fb67d80b640 1 -- 172.21.15.33:0/784980196 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6781a4600 con 0x7fb678100740 2024-03-19T02:58:43.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.321+0000 7fb67c809640 1 -- 172.21.15.33:0/784980196 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb66802ca70 con 0x7fb678100740 2024-03-19T02:58:43.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.321+0000 7fb67d80b640 1 -- 172.21.15.33:0/784980196 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6781a4b40 con 0x7fb678100740 2024-03-19T02:58:43.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.322+0000 7fb67c809640 1 -- 172.21.15.33:0/784980196 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb668003910 con 0x7fb678100740 2024-03-19T02:58:43.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.324+0000 7fb67c809640 1 -- 172.21.15.33:0/784980196 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fb668046020 con 0x7fb678100740 2024-03-19T02:58:43.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.324+0000 7fb6803f9740 1 Processor -- start 2024-03-19T02:58:43.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.324+0000 7fb6803f9740 1 -- start start 2024-03-19T02:58:43.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.324+0000 7fb6803f9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a70710760 0x561a70710b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.324+0000 7fb6803f9740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561a70711080 0x561a7071de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.324+0000 7fb6803f9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561a7071e340 0x561a70720730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.324+0000 7fb6803f9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561a70674500 con 0x561a70710760 2024-03-19T02:58:43.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.324+0000 7fb6803f9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x561a7061cbe0 con 0x561a7071e340 2024-03-19T02:58:43.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.324+0000 7fb6803f9740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x561a706705a0 con 0x561a70711080 2024-03-19T02:58:43.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.324+0000 7fb6777fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a70710760 0x561a70710b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.324+0000 7fb6777fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a70710760 0x561a70710b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53916/0 (socket says 172.21.15.33:53916) 2024-03-19T02:58:43.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.324+0000 7fb6777fe640 1 -- 172.21.15.33:0/3487024753 learned_addr learned my addr 172.21.15.33:0/3487024753 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:43.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.324+0000 7fb67c809640 1 --2- 172.21.15.33:0/784980196 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fb6640a2970 0x7fb6640a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.324+0000 7fb676ffd640 1 --2- 172.21.15.33:0/3487024753 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561a70711080 0x561a7071de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.324+0000 7fb66ffff640 1 --2- 172.21.15.33:0/3487024753 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561a7071e340 0x561a70720730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.324+0000 7fb6777fe640 1 --2- 172.21.15.33:0/784980196 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fb6640a2970 0x7fb6640a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.324+0000 7fb6777fe640 1 -- 172.21.15.33:0/3487024753 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561a70711080 msgr2=0x561a7071de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:43.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.324+0000 7fb6777fe640 1 --2- 172.21.15.33:0/3487024753 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561a70711080 0x561a7071de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.324+0000 7fb6777fe640 1 -- 172.21.15.33:0/3487024753 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561a7071e340 msgr2=0x561a70720730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:43.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.324+0000 7fb6777fe640 1 --2- 172.21.15.33:0/3487024753 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561a7071e340 0x561a70720730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.324+0000 7fb6777fe640 1 -- 172.21.15.33:0/3487024753 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a7063e000 con 0x561a70710760 2024-03-19T02:58:43.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.324+0000 7fb67c809640 1 -- 172.21.15.33:0/784980196 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fb6680f1080 con 0x7fb678100740 2024-03-19T02:58:43.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.325+0000 7fb66ffff640 1 --2- 172.21.15.33:0/3487024753 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561a7071e340 0x561a70720730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:58:43.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.325+0000 7fb6777fe640 1 --2- 172.21.15.33:0/3487024753 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a70710760 0x561a70710b40 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fb6700049d0 tx=0x7fb670013f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=581f7991944ff98c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:43.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.325+0000 7fb66dffb640 1 -- 172.21.15.33:0/3487024753 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb670022020 con 0x561a70710760 2024-03-19T02:58:43.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.325+0000 7fb66dffb640 1 -- 172.21.15.33:0/3487024753 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb6700134b0 con 0x561a70710760 2024-03-19T02:58:43.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.325+0000 7fb66dffb640 1 -- 172.21.15.33:0/3487024753 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb670010960 con 0x561a70710760 2024-03-19T02:58:43.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.325+0000 7fb6803f9740 1 -- 172.21.15.33:0/3487024753 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a70710760 msgr2=0x561a70710b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:43.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.325+0000 7fb6777fe640 1 --2- 172.21.15.33:0/784980196 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fb6640a2970 0x7fb6640a4e30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb670023000 tx=0x7fb670021000 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:43.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.325+0000 7fb6803f9740 1 --2- 172.21.15.33:0/3487024753 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a70710760 0x561a70710b40 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fb6700049d0 tx=0x7fb670013f70 comp rx=0 tx=0).stop 2024-03-19T02:58:43.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.325+0000 7fb6803f9740 1 -- 172.21.15.33:0/3487024753 shutdown_connections 2024-03-19T02:58:43.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.326+0000 7fb6803f9740 1 --2- 172.21.15.33:0/3487024753 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561a70711080 0x561a7071de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.326+0000 7fb6803f9740 1 --2- 172.21.15.33:0/3487024753 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561a7071e340 0x561a70720730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.326+0000 7fb6803f9740 1 --2- 172.21.15.33:0/3487024753 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a70710760 0x561a70710b40 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.326+0000 7fb6803f9740 1 -- 172.21.15.33:0/3487024753 >> 172.21.15.33:0/3487024753 conn(0x561a7070b160 msgr2=0x561a7070b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:43.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.326+0000 7fb6803f9740 1 -- 172.21.15.33:0/3487024753 shutdown_connections 2024-03-19T02:58:43.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.326+0000 7fb6803f9740 1 -- 172.21.15.33:0/3487024753 wait complete. 2024-03-19T02:58:43.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.326+0000 7fb6803f9740 1 -- start start 2024-03-19T02:58:43.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.326+0000 7fb6803f9740 1 Processor -- start 2024-03-19T02:58:43.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.326+0000 7fb6803f9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a70710760 0x561a7071b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.326+0000 7fb6803f9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561a70711080 0x561a7071b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.326+0000 7fb6803f9740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561a7071e340 0x561a7071bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.326+0000 7fb6803f9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561a70721100 con 0x561a70710760 2024-03-19T02:58:43.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.326+0000 7fb6803f9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x561a70630160 con 0x561a70711080 2024-03-19T02:58:43.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.326+0000 7fb6803f9740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x561a70721280 con 0x561a7071e340 2024-03-19T02:58:43.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.326+0000 7fb6803f9740 10 client.0 _ll_register_callbacks cb 0x561a7066ca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:58:43.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.326+0000 7fb676ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a70710760 0x561a7071b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.326+0000 7fb676ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a70710760 0x561a7071b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53924/0 (socket says 172.21.15.33:53924) 2024-03-19T02:58:43.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.326+0000 7fb676ffd640 1 -- 172.21.15.33:0/3133155502 learned_addr learned my addr 172.21.15.33:0/3133155502 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:43.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.326+0000 7fb66ffff640 1 --2- 172.21.15.33:0/3133155502 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561a70711080 0x561a7071b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.327+0000 7fb6777fe640 1 --2- 172.21.15.33:0/3133155502 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561a7071e340 0x561a7071bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.327+0000 7fb676ffd640 1 -- 172.21.15.33:0/3133155502 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561a7071e340 msgr2=0x561a7071bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:43.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.327+0000 7fb676ffd640 1 --2- 172.21.15.33:0/3133155502 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561a7071e340 0x561a7071bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.327+0000 7fb676ffd640 1 -- 172.21.15.33:0/3133155502 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561a70711080 msgr2=0x561a7071b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:43.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.327+0000 7fb676ffd640 1 --2- 172.21.15.33:0/3133155502 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561a70711080 0x561a7071b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.327+0000 7fb676ffd640 1 -- 172.21.15.33:0/3133155502 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a704527d0 con 0x561a70710760 2024-03-19T02:58:43.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.327+0000 7fb676ffd640 1 --2- 172.21.15.33:0/3133155502 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a70710760 0x561a7071b2e0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fb668000c00 tx=0x7fb66802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53ff61227688183b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:43.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.327+0000 7fb656ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:58:43.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.327+0000 7fb656ffd640 1 -- 172.21.15.33:0/3133155502 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6680bd480 con 0x561a70710760 2024-03-19T02:58:43.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.327+0000 7fb656ffd640 1 -- 172.21.15.33:0/3133155502 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fb6680ee6e0 con 0x561a70710760 2024-03-19T02:58:43.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.327+0000 7fb6803f9740 1 -- 172.21.15.33:0/3133155502 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6700134b0 con 0x561a70710760 2024-03-19T02:58:43.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.327+0000 7fb656ffd640 1 -- 172.21.15.33:0/3133155502 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb6680ee8e0 con 0x561a70710760 2024-03-19T02:58:43.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.327+0000 7fb6803f9740 10 client.16830 fetch_fsmap learned FSMap version 25 2024-03-19T02:58:43.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.327+0000 7fb6803f9740 1 -- 172.21.15.33:0/3133155502 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x561a703883f0 con 0x561a70710760 2024-03-19T02:58:43.335 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.327+0000 7fb656ffd640 1 -- 172.21.15.33:0/3133155502 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb6700134b0 con 0x561a70710760 2024-03-19T02:58:43.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.327+0000 7fb656ffd640 1 -- 172.21.15.33:0/3133155502 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fb6680c1160 con 0x561a70710760 2024-03-19T02:58:43.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.327+0000 7fb6803f9740 10 client.16830 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:58:43.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.327+0000 7fb6803f9740 10 client.16830 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:58:43.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.327+0000 7fb6803f9740 20 client.16830 populate_metadata read hostname 'smithi033' 2024-03-19T02:58:43.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.327+0000 7fb6803f9740 1 --2- 172.21.15.33:0/3133155502 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x561a7072e560 0x561a70730990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.328+0000 7fb6803f9740 4 client.16830 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:58:43.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.328+0000 7fb6803f9740 1 -- 172.21.15.33:0/3133155502 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561a70681a40 con 0x561a7072e560 2024-03-19T02:58:43.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.328+0000 7fb66ffff640 1 --2- 172.21.15.33:0/3133155502 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x561a7072e560 0x561a70730990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.337 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.328+0000 7fb66ffff640 1 --2- 172.21.15.33:0/3133155502 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x561a7072e560 0x561a70730990 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d79cb561842ccdd server_cookie=da5b2583428e92fb in_seq=0 out_seq=0 2024-03-19T02:58:43.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.328+0000 7fb656ffd640 10 client.16830 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:58:43.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.329+0000 7fb656ffd640 1 -- 172.21.15.33:0/3133155502 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561a70681a40 con 0x561a7072e560 2024-03-19T02:58:43.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.329+0000 7fb656ffd640 10 client.16830 handle_command_reply: tid=0 2024-03-19T02:58:43.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.329+0000 7fb6803f9740 1 client.16830 shutdown 2024-03-19T02:58:43.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.329+0000 7fb6803f9740 1 -- 172.21.15.33:0/3133155502 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a70710760 msgr2=0x561a7071b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:43.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.329+0000 7fb6803f9740 1 --2- 172.21.15.33:0/3133155502 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a70710760 0x561a7071b2e0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fb668000c00 tx=0x7fb66802bc30 comp rx=0 tx=0).stop 2024-03-19T02:58:43.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.329+0000 7fb6803f9740 1 -- 172.21.15.33:0/3133155502 shutdown_connections 2024-03-19T02:58:43.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.329+0000 7fb6803f9740 1 --2- 172.21.15.33:0/3133155502 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x561a7072e560 0x561a70730990 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.329+0000 7fb6803f9740 1 --2- 172.21.15.33:0/3133155502 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561a7071e340 0x561a7071bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.329+0000 7fb6803f9740 1 --2- 172.21.15.33:0/3133155502 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561a70711080 0x561a7071b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.329+0000 7fb6803f9740 1 --2- 172.21.15.33:0/3133155502 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a70710760 0x561a7071b2e0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.329+0000 7fb6803f9740 1 -- 172.21.15.33:0/3133155502 >> 172.21.15.33:0/3133155502 conn(0x561a7070b160 msgr2=0x561a7070ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:43.338 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.329+0000 7fb656ffd640 0 client.16830 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:58:43.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.329+0000 7fb6803f9740 1 -- 172.21.15.33:0/3133155502 shutdown_connections 2024-03-19T02:58:43.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.329+0000 7fb6803f9740 1 -- 172.21.15.33:0/3133155502 wait complete. 2024-03-19T02:58:43.339 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.329+0000 7fb6803f9740 20 client.16830 trim_cache size 0 max 16384 2024-03-19T02:58:43.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.346+0000 7fb6803f9740 1 Processor -- start 2024-03-19T02:58:43.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.346+0000 7fb6803f9740 1 -- start start 2024-03-19T02:58:43.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.346+0000 7fb6803f9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561a70710760 0x561a7071da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.346+0000 7fb6803f9740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561a70711080 0x561a7071a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.346+0000 7fb6803f9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a7071e340 0x561a707165c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.346+0000 7fb6803f9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb6781af220 con 0x561a7071e340 2024-03-19T02:58:43.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.346+0000 7fb6803f9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb6781acaf0 con 0x561a70710760 2024-03-19T02:58:43.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.346+0000 7fb6803f9740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb6781aa920 con 0x561a70711080 2024-03-19T02:58:43.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.347+0000 7fb6777fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a7071e340 0x561a707165c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.347+0000 7fb676ffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561a70710760 0x561a7071da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.347+0000 7fb6777fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a7071e340 0x561a707165c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53936/0 (socket says 172.21.15.33:53936) 2024-03-19T02:58:43.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.347+0000 7fb6777fe640 1 -- 172.21.15.33:0/663647847 learned_addr learned my addr 172.21.15.33:0/663647847 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:43.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.347+0000 7fb66ffff640 1 --2- 172.21.15.33:0/663647847 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561a70711080 0x561a7071a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.347+0000 7fb6777fe640 1 -- 172.21.15.33:0/663647847 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561a70711080 msgr2=0x561a7071a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:43.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.347+0000 7fb6777fe640 1 --2- 172.21.15.33:0/663647847 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561a70711080 0x561a7071a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.347+0000 7fb6777fe640 1 -- 172.21.15.33:0/663647847 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561a70710760 msgr2=0x561a7071da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:43.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.347+0000 7fb6777fe640 1 --2- 172.21.15.33:0/663647847 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561a70710760 0x561a7071da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.347+0000 7fb6777fe640 1 -- 172.21.15.33:0/663647847 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a703480f0 con 0x561a7071e340 2024-03-19T02:58:43.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.347+0000 7fb676ffd640 1 --2- 172.21.15.33:0/663647847 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561a70710760 0x561a7071da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:58:43.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.347+0000 7fb6777fe640 1 --2- 172.21.15.33:0/663647847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a7071e340 0x561a707165c0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fb670013a10 tx=0x7fb670009480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4346b920a32e7080 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:43.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.347+0000 7fb66d7fa640 1 -- 172.21.15.33:0/663647847 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb670022030 con 0x561a7071e340 2024-03-19T02:58:43.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.347+0000 7fb66d7fa640 1 -- 172.21.15.33:0/663647847 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb670012070 con 0x561a7071e340 2024-03-19T02:58:43.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.347+0000 7fb66d7fa640 1 -- 172.21.15.33:0/663647847 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb670004c60 con 0x561a7071e340 2024-03-19T02:58:43.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.348+0000 7fb6803f9740 1 -- 172.21.15.33:0/663647847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a7071e340 msgr2=0x561a707165c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:43.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.348+0000 7fb6803f9740 1 --2- 172.21.15.33:0/663647847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a7071e340 0x561a707165c0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fb670013a10 tx=0x7fb670009480 comp rx=0 tx=0).stop 2024-03-19T02:58:43.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.348+0000 7fb6803f9740 1 -- 172.21.15.33:0/663647847 shutdown_connections 2024-03-19T02:58:43.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.348+0000 7fb6803f9740 1 --2- 172.21.15.33:0/663647847 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561a70711080 0x561a7071a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.348+0000 7fb6803f9740 1 --2- 172.21.15.33:0/663647847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a7071e340 0x561a707165c0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.348+0000 7fb6803f9740 1 --2- 172.21.15.33:0/663647847 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561a70710760 0x561a7071da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.348+0000 7fb6803f9740 1 -- 172.21.15.33:0/663647847 >> 172.21.15.33:0/663647847 conn(0x561a7070b160 msgr2=0x561a7070a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:43.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.348+0000 7fb6803f9740 1 -- 172.21.15.33:0/663647847 shutdown_connections 2024-03-19T02:58:43.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.348+0000 7fb6803f9740 1 -- 172.21.15.33:0/663647847 wait complete. 2024-03-19T02:58:43.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.348+0000 7fb6803f9740 1 -- start start 2024-03-19T02:58:43.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.348+0000 7fb6803f9740 1 Processor -- start 2024-03-19T02:58:43.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.348+0000 7fb6803f9740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561a70710760 0x561a70721da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.348+0000 7fb6803f9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561a70711080 0x561a707222e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.349+0000 7fb6803f9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a7071e340 0x561a70722820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.349+0000 7fb6803f9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb6781aafb0 con 0x561a7071e340 2024-03-19T02:58:43.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.349+0000 7fb6803f9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb6781ac420 con 0x561a70711080 2024-03-19T02:58:43.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.349+0000 7fb6803f9740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb6781ab680 con 0x561a70710760 2024-03-19T02:58:43.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.349+0000 7fb6803f9740 10 client.0 _ll_register_callbacks cb 0x7fb6680057a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:58:43.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.349+0000 7fb6777fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a7071e340 0x561a70722820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.349+0000 7fb6777fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a7071e340 0x561a70722820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:53944/0 (socket says 172.21.15.33:53944) 2024-03-19T02:58:43.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.349+0000 7fb6777fe640 1 -- 172.21.15.33:0/1504144568 learned_addr learned my addr 172.21.15.33:0/1504144568 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:58:43.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.349+0000 7fb676ffd640 1 --2- 172.21.15.33:0/1504144568 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561a70710760 0x561a70721da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.349+0000 7fb66ffff640 1 --2- 172.21.15.33:0/1504144568 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561a70711080 0x561a707222e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.349+0000 7fb6777fe640 1 -- 172.21.15.33:0/1504144568 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561a70710760 msgr2=0x561a70721da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:43.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.349+0000 7fb6777fe640 1 --2- 172.21.15.33:0/1504144568 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561a70710760 0x561a70721da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.349+0000 7fb6777fe640 1 -- 172.21.15.33:0/1504144568 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561a70711080 msgr2=0x561a707222e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:58:43.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.349+0000 7fb6777fe640 1 --2- 172.21.15.33:0/1504144568 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561a70711080 0x561a707222e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.349+0000 7fb6777fe640 1 -- 172.21.15.33:0/1504144568 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a70793880 con 0x561a7071e340 2024-03-19T02:58:43.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.349+0000 7fb6777fe640 1 --2- 172.21.15.33:0/1504144568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a7071e340 0x561a70722820 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fb670009e50 tx=0x7fb670009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd364cdf3b0c0a81 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:58:43.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.349+0000 7fb656ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:58:43.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.349+0000 7fb656ffd640 1 -- 172.21.15.33:0/1504144568 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb670022040 con 0x561a7071e340 2024-03-19T02:58:43.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.349+0000 7fb656ffd640 1 -- 172.21.15.33:0/1504144568 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fb6700049d0 con 0x561a7071e340 2024-03-19T02:58:43.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.349+0000 7fb6803f9740 1 -- 172.21.15.33:0/1504144568 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb670012070 con 0x561a7071e340 2024-03-19T02:58:43.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.349+0000 7fb656ffd640 1 -- 172.21.15.33:0/1504144568 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb670004bd0 con 0x561a7071e340 2024-03-19T02:58:43.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.349+0000 7fb6803f9740 10 client.16836 fetch_fsmap learned FSMap version 25 2024-03-19T02:58:43.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.349+0000 7fb6803f9740 1 -- 172.21.15.33:0/1504144568 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x561a70794b10 con 0x561a7071e340 2024-03-19T02:58:43.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.350+0000 7fb656ffd640 1 -- 172.21.15.33:0/1504144568 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb670012070 con 0x561a7071e340 2024-03-19T02:58:43.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.350+0000 7fb656ffd640 1 -- 172.21.15.33:0/1504144568 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fb67003c4d0 con 0x561a7071e340 2024-03-19T02:58:43.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.350+0000 7fb6803f9740 10 client.16836 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:58:43.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.350+0000 7fb6803f9740 10 client.16836 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:58:43.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.350+0000 7fb6803f9740 20 client.16836 populate_metadata read hostname 'smithi033' 2024-03-19T02:58:43.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.350+0000 7fb6803f9740 1 --2- 172.21.15.33:0/1504144568 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x561a70795000 0x561a70797420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:58:43.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.350+0000 7fb6803f9740 4 client.16836 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:58:43.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.350+0000 7fb6803f9740 1 -- 172.21.15.33:0/1504144568 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561a7067bd90 con 0x561a70795000 2024-03-19T02:58:43.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.350+0000 7fb676ffd640 1 --2- 172.21.15.33:0/1504144568 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x561a70795000 0x561a70797420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:58:43.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.350+0000 7fb656ffd640 10 client.16836 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:58:43.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.350+0000 7fb676ffd640 1 --2- 172.21.15.33:0/1504144568 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x561a70795000 0x561a70797420 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3728f0e2651a44da server_cookie=b1a47916038405c4 in_seq=0 out_seq=0 2024-03-19T02:58:43.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.350+0000 7fb656ffd640 1 -- 172.21.15.33:0/1504144568 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x561a7067bd90 con 0x561a70795000 2024-03-19T02:58:43.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.350+0000 7fb656ffd640 10 client.16836 handle_command_reply: tid=0 2024-03-19T02:58:43.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.350+0000 7fb6803f9740 1 client.16836 shutdown 2024-03-19T02:58:43.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.351+0000 7fb6803f9740 1 -- 172.21.15.33:0/1504144568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a7071e340 msgr2=0x561a70722820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:43.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.351+0000 7fb6803f9740 1 --2- 172.21.15.33:0/1504144568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a7071e340 0x561a70722820 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fb670009e50 tx=0x7fb670009900 comp rx=0 tx=0).stop 2024-03-19T02:58:43.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.351+0000 7fb6803f9740 1 -- 172.21.15.33:0/1504144568 shutdown_connections 2024-03-19T02:58:43.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.351+0000 7fb6803f9740 1 --2- 172.21.15.33:0/1504144568 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x561a70795000 0x561a70797420 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.351+0000 7fb6803f9740 1 --2- 172.21.15.33:0/1504144568 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561a7071e340 0x561a70722820 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.351+0000 7fb6803f9740 1 --2- 172.21.15.33:0/1504144568 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561a70711080 0x561a707222e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.351+0000 7fb656ffd640 0 client.16836 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:58:43.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.351+0000 7fb6803f9740 1 --2- 172.21.15.33:0/1504144568 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561a70710760 0x561a70721da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.351+0000 7fb6803f9740 1 -- 172.21.15.33:0/1504144568 >> 172.21.15.33:0/1504144568 conn(0x561a7070b160 msgr2=0x561a7071b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:43.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.351+0000 7fb6803f9740 1 -- 172.21.15.33:0/1504144568 shutdown_connections 2024-03-19T02:58:43.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.351+0000 7fb6803f9740 1 -- 172.21.15.33:0/1504144568 wait complete. 2024-03-19T02:58:43.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.351+0000 7fb6803f9740 20 client.16836 trim_cache size 0 max 16384 2024-03-19T02:58:43.359 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:58:43.359 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:58:43.359 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-03-19T02:58:43.359 INFO:teuthology.orchestra.run.smithi033.stdout: "1e86cc18-be47-4033-a4e7-1074918c2dd7": { 2024-03-19T02:58:43.359 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-03-19T02:58:43.359 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "1e86cc18-be47-4033-a4e7-1074918c2dd7", 2024-03-19T02:58:43.359 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-03-19T02:58:43.360 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T02:58:43.360 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T02:58:43.360 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:58:43.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.351+0000 7fb67d80b640 1 -- 172.21.15.33:0/784980196 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fb6640a2970 msgr2=0x7fb6640a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:43.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.351+0000 7fb67d80b640 1 --2- 172.21.15.33:0/784980196 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fb6640a2970 0x7fb6640a4e30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb670023000 tx=0x7fb670021000 comp rx=0 tx=0).stop 2024-03-19T02:58:43.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.351+0000 7fb67d80b640 1 -- 172.21.15.33:0/784980196 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb678100740 msgr2=0x7fb678198e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:58:43.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.351+0000 7fb67d80b640 1 --2- 172.21.15.33:0/784980196 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb678100740 0x7fb678198e70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb668009230 tx=0x7fb66802c5d0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.351+0000 7fb67d80b640 1 -- 172.21.15.33:0/784980196 shutdown_connections 2024-03-19T02:58:43.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.351+0000 7fb67d80b640 1 --2- 172.21.15.33:0/784980196 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fb6640a2970 0x7fb6640a4e30 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.351+0000 7fb67d80b640 1 --2- 172.21.15.33:0/784980196 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb678106750 0x7fb6781a3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.351+0000 7fb67d80b640 1 --2- 172.21.15.33:0/784980196 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb678101100 0x7fb6781993b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.351+0000 7fb67d80b640 1 --2- 172.21.15.33:0/784980196 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb678100740 0x7fb678198e70 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:58:43.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.351+0000 7fb67d80b640 1 -- 172.21.15.33:0/784980196 >> 172.21.15.33:0/784980196 conn(0x7fb678077880 msgr2=0x7fb6781033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:58:43.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.351+0000 7fb67d80b640 1 -- 172.21.15.33:0/784980196 shutdown_connections 2024-03-19T02:58:43.361 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:58:43.351+0000 7fb67d80b640 1 -- 172.21.15.33:0/784980196 wait complete. 2024-03-19T02:58:43.363 INFO:tasks.cephfs.filesystem:scrub status for tag:1e86cc18-be47-4033-a4e7-1074918c2dd7 - {'path': '/', 'tag': '1e86cc18-be47-4033-a4e7-1074918c2dd7', 'options': 'recursive,force'} 2024-03-19T02:58:44.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:44 smithi096 ceph-mon[35946]: pgmap v161: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:58:44.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:44 smithi096 ceph-mon[35946]: scrub queued for path: / 2024-03-19T02:58:44.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:44 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T02:58:44.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:44 smithi096 ceph-mon[35946]: scrub summary: active paths [/] 2024-03-19T02:58:44.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:44 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T02:58:45.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:44 smithi155 ceph-mon[35804]: pgmap v161: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:58:45.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:44 smithi155 ceph-mon[35804]: scrub queued for path: / 2024-03-19T02:58:45.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:44 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T02:58:45.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:44 smithi155 ceph-mon[35804]: scrub summary: active paths [/] 2024-03-19T02:58:45.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:44 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T02:58:45.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:44 smithi033 ceph-mon[31973]: pgmap v161: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:58:45.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:44 smithi033 ceph-mon[31973]: scrub queued for path: / 2024-03-19T02:58:45.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:44 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T02:58:45.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:44 smithi033 ceph-mon[31973]: scrub summary: active paths [/] 2024-03-19T02:58:45.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:44 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T02:58:46.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:46 smithi096 ceph-mon[35946]: pgmap v162: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:58:47.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:46 smithi155 ceph-mon[35804]: pgmap v162: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:58:47.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:46 smithi033 ceph-mon[31973]: pgmap v162: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:58:48.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:48 smithi096 ceph-mon[35946]: pgmap v163: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:58:48.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:48 smithi096 ceph-mon[35946]: scrub summary: idle 2024-03-19T02:58:49.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:48 smithi155 ceph-mon[35804]: pgmap v163: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:58:49.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:48 smithi155 ceph-mon[35804]: scrub summary: idle 2024-03-19T02:58:49.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:48 smithi033 ceph-mon[31973]: pgmap v163: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:58:49.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:48 smithi033 ceph-mon[31973]: scrub summary: idle 2024-03-19T02:58:50.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:50 smithi096 ceph-mon[35946]: pgmap v164: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:58:51.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:50 smithi155 ceph-mon[35804]: pgmap v164: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:58:51.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:50 smithi033 ceph-mon[31973]: pgmap v164: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:58:52.819 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:52 smithi155 ceph-mon[35804]: pgmap v165: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:58:52.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:52 smithi096 ceph-mon[35946]: pgmap v165: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:58:53.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:52 smithi033 ceph-mon[31973]: pgmap v165: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:58:54.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:54 smithi096 ceph-mon[35946]: pgmap v166: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:58:55.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:54 smithi155 ceph-mon[35804]: pgmap v166: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:58:55.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:54 smithi033 ceph-mon[31973]: pgmap v166: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:58:56.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:56 smithi096 ceph-mon[35946]: pgmap v167: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:58:57.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:56 smithi155 ceph-mon[35804]: pgmap v167: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:58:57.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:56 smithi033 ceph-mon[31973]: pgmap v167: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:58:58.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:58:58 smithi096 ceph-mon[35946]: pgmap v168: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:58:59.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:58:58 smithi155 ceph-mon[35804]: pgmap v168: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:58:59.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:58:58 smithi033 ceph-mon[31973]: pgmap v168: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:59:00.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:00 smithi096 ceph-mon[35946]: pgmap v169: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:59:01.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:00 smithi155 ceph-mon[35804]: pgmap v169: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:59:01.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:00 smithi033 ceph-mon[31973]: pgmap v169: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:59:02.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:02 smithi096 ceph-mon[35946]: pgmap v170: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:59:03.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:02 smithi155 ceph-mon[35804]: pgmap v170: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:59:03.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:02 smithi033 ceph-mon[31973]: pgmap v170: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:59:04.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:04 smithi096 ceph-mon[35946]: pgmap v171: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-19T02:59:05.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:04 smithi155 ceph-mon[35804]: pgmap v171: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-19T02:59:05.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:04 smithi033 ceph-mon[31973]: pgmap v171: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-19T02:59:06.830 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:06 smithi033 ceph-mon[31973]: pgmap v172: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-19T02:59:06.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:06 smithi096 ceph-mon[35946]: pgmap v172: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-19T02:59:07.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:06 smithi155 ceph-mon[35804]: pgmap v172: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-19T02:59:08.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:08 smithi096 ceph-mon[35946]: pgmap v173: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-19T02:59:09.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:08 smithi155 ceph-mon[35804]: pgmap v173: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-19T02:59:09.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:08 smithi033 ceph-mon[31973]: pgmap v173: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-19T02:59:10.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:10 smithi096 ceph-mon[35946]: pgmap v174: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-19T02:59:11.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:10 smithi155 ceph-mon[35804]: pgmap v174: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-19T02:59:11.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:10 smithi033 ceph-mon[31973]: pgmap v174: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-19T02:59:12.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:12 smithi096 ceph-mon[35946]: pgmap v175: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-19T02:59:13.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:12 smithi155 ceph-mon[35804]: pgmap v175: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-19T02:59:13.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:12 smithi033 ceph-mon[31973]: pgmap v175: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-19T02:59:13.364 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:59:13.365 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:59:13.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.482+0000 7fabe527e640 1 -- 172.21.15.33:0/360653854 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabe0102a90 msgr2=0x7fabe0102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:13.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.482+0000 7fabe527e640 1 --2- 172.21.15.33:0/360653854 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabe0102a90 0x7fabe0102e70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fabd0009be0 tx=0x7fabd002bf80 comp rx=0 tx=0).stop 2024-03-19T02:59:13.483 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.482+0000 7fabe527e640 1 -- 172.21.15.33:0/360653854 shutdown_connections 2024-03-19T02:59:13.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.482+0000 7fabe527e640 1 --2- 172.21.15.33:0/360653854 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fabe010be60 0x7fabe010e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.482+0000 7fabe527e640 1 --2- 172.21.15.33:0/360653854 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fabe01033b0 0x7fabe010b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.482+0000 7fabe527e640 1 --2- 172.21.15.33:0/360653854 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabe0102a90 0x7fabe0102e70 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.482+0000 7fabe527e640 1 -- 172.21.15.33:0/360653854 >> 172.21.15.33:0/360653854 conn(0x7fabe0077880 msgr2=0x7fabe0077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:13.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.482+0000 7fabe527e640 1 -- 172.21.15.33:0/360653854 shutdown_connections 2024-03-19T02:59:13.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.482+0000 7fabe527e640 1 -- 172.21.15.33:0/360653854 wait complete. 2024-03-19T02:59:13.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.482+0000 7fabe527e640 1 Processor -- start 2024-03-19T02:59:13.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.483+0000 7fabe527e640 1 -- start start 2024-03-19T02:59:13.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.483+0000 7fabe527e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabe0102a90 0x7fabe01980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.483+0000 7fabe527e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fabe01033b0 0x7fabe01985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.483+0000 7fabe527e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fabe010be60 0x7fabe019c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.483+0000 7fabe527e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fabe0111ae0 con 0x7fabe010be60 2024-03-19T02:59:13.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.483+0000 7fabe527e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fabe0111960 con 0x7fabe0102a90 2024-03-19T02:59:13.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.483+0000 7fabe527e640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fabe0111c60 con 0x7fabe01033b0 2024-03-19T02:59:13.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.483+0000 7fabdf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fabe010be60 0x7fabe019c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.483+0000 7fabdf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fabe010be60 0x7fabe019c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51264/0 (socket says 172.21.15.33:51264) 2024-03-19T02:59:13.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.483+0000 7fabdf7fe640 1 -- 172.21.15.33:0/517311491 learned_addr learned my addr 172.21.15.33:0/517311491 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:13.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.483+0000 7fabdeffd640 1 --2- 172.21.15.33:0/517311491 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabe0102a90 0x7fabe01980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.483+0000 7fabde7fc640 1 --2- 172.21.15.33:0/517311491 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fabe01033b0 0x7fabe01985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.483+0000 7fabdeffd640 1 -- 172.21.15.33:0/517311491 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fabe01033b0 msgr2=0x7fabe01985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:13.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.483+0000 7fabdeffd640 1 --2- 172.21.15.33:0/517311491 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fabe01033b0 0x7fabe01985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.483+0000 7fabdeffd640 1 -- 172.21.15.33:0/517311491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fabe010be60 msgr2=0x7fabe019c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:13.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.483+0000 7fabdeffd640 1 --2- 172.21.15.33:0/517311491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fabe010be60 0x7fabe019c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.483+0000 7fabdeffd640 1 -- 172.21.15.33:0/517311491 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabe019d090 con 0x7fabe0102a90 2024-03-19T02:59:13.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.483+0000 7fabdf7fe640 1 --2- 172.21.15.33:0/517311491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fabe010be60 0x7fabe019c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:13.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.483+0000 7fabdeffd640 1 --2- 172.21.15.33:0/517311491 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabe0102a90 0x7fabe01980a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fabd002c4c0 tx=0x7fabd0033610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:13.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.484+0000 7fabc7fff640 1 -- 172.21.15.33:0/517311491 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd0043020 con 0x7fabe0102a90 2024-03-19T02:59:13.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.484+0000 7fabc7fff640 1 -- 172.21.15.33:0/517311491 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fabd0003b80 con 0x7fabe0102a90 2024-03-19T02:59:13.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.484+0000 7fabe527e640 1 -- 172.21.15.33:0/517311491 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabe019d320 con 0x7fabe0102a90 2024-03-19T02:59:13.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.484+0000 7fabc7fff640 1 -- 172.21.15.33:0/517311491 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd0034670 con 0x7fabe0102a90 2024-03-19T02:59:13.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.484+0000 7fabe527e640 1 -- 172.21.15.33:0/517311491 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabe01a4c10 con 0x7fabe0102a90 2024-03-19T02:59:13.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.486+0000 7fabc7fff640 1 -- 172.21.15.33:0/517311491 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fabd0046020 con 0x7fabe0102a90 2024-03-19T02:59:13.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.486+0000 7fabe7e65740 1 Processor -- start 2024-03-19T02:59:13.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.486+0000 7fabe7e65740 1 -- start start 2024-03-19T02:59:13.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.486+0000 7fabe7e65740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f3d3b83a30 0x55f3d3b83e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.486+0000 7fabe7e65740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3b84350 0x55f3d3b8c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.487+0000 7fabe7e65740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3d3b8cf30 0x55f3d3b8f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.487+0000 7fabe7e65740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f3d3ae7500 con 0x55f3d3b84350 2024-03-19T02:59:13.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.487+0000 7fabe7e65740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55f3d3a8fbe0 con 0x55f3d3b8cf30 2024-03-19T02:59:13.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.487+0000 7fabe7e65740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f3d3ae35a0 con 0x55f3d3b83a30 2024-03-19T02:59:13.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.487+0000 7fabc7fff640 1 --2- 172.21.15.33:0/517311491 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fabc80a2970 0x7fabc80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.487+0000 7fabdeffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3b84350 0x55f3d3b8c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.487+0000 7fabdf7fe640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f3d3b83a30 0x55f3d3b83e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.487+0000 7fabdeffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3b84350 0x55f3d3b8c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51266/0 (socket says 172.21.15.33:51266) 2024-03-19T02:59:13.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.487+0000 7fabdeffd640 1 -- 172.21.15.33:0/3391645926 learned_addr learned my addr 172.21.15.33:0/3391645926 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:13.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.487+0000 7fabdf7fe640 1 --2- 172.21.15.33:0/517311491 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fabc80a2970 0x7fabc80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.487+0000 7fabde7fc640 1 --2- 172.21.15.33:0/3391645926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3d3b8cf30 0x55f3d3b8f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.487+0000 7fabc7fff640 1 -- 172.21.15.33:0/517311491 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fabd00f1760 con 0x7fabe0102a90 2024-03-19T02:59:13.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.487+0000 7fabdf7fe640 1 -- 172.21.15.33:0/3391645926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3d3b8cf30 msgr2=0x55f3d3b8f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:13.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.487+0000 7fabdf7fe640 1 --2- 172.21.15.33:0/3391645926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3d3b8cf30 0x55f3d3b8f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.487+0000 7fabdf7fe640 1 -- 172.21.15.33:0/3391645926 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3b84350 msgr2=0x55f3d3b8c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:13.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.487+0000 7fabdf7fe640 1 --2- 172.21.15.33:0/3391645926 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3b84350 0x55f3d3b8c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.487+0000 7fabdf7fe640 1 -- 172.21.15.33:0/3391645926 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3d3ab1000 con 0x55f3d3b83a30 2024-03-19T02:59:13.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.487+0000 7fabdf7fe640 1 --2- 172.21.15.33:0/517311491 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fabc80a2970 0x7fabc80a4e30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fabd800fc90 tx=0x7fabd8018000 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:13.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.487+0000 7fabdf7fe640 1 --2- 172.21.15.33:0/3391645926 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f3d3b83a30 0x55f3d3b83e10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fabd8010e70 tx=0x7fabd800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e934a6235a3fb076 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:13.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.487+0000 7fabc57fa640 1 -- 172.21.15.33:0/3391645926 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd8022070 con 0x55f3d3b83a30 2024-03-19T02:59:13.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.487+0000 7fabc57fa640 1 -- 172.21.15.33:0/3391645926 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fabd8009df0 con 0x55f3d3b83a30 2024-03-19T02:59:13.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.488+0000 7fabc57fa640 1 -- 172.21.15.33:0/3391645926 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd8019040 con 0x55f3d3b83a30 2024-03-19T02:59:13.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.488+0000 7fabe7e65740 1 -- 172.21.15.33:0/3391645926 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f3d3b83a30 msgr2=0x55f3d3b83e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:13.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.488+0000 7fabe7e65740 1 --2- 172.21.15.33:0/3391645926 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f3d3b83a30 0x55f3d3b83e10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fabd8010e70 tx=0x7fabd800f790 comp rx=0 tx=0).stop 2024-03-19T02:59:13.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.488+0000 7fabe7e65740 1 -- 172.21.15.33:0/3391645926 shutdown_connections 2024-03-19T02:59:13.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.488+0000 7fabe7e65740 1 --2- 172.21.15.33:0/3391645926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3d3b8cf30 0x55f3d3b8f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.488+0000 7fabe7e65740 1 --2- 172.21.15.33:0/3391645926 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3b84350 0x55f3d3b8c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.488+0000 7fabe7e65740 1 --2- 172.21.15.33:0/3391645926 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f3d3b83a30 0x55f3d3b83e10 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.488+0000 7fabe7e65740 1 -- 172.21.15.33:0/3391645926 >> 172.21.15.33:0/3391645926 conn(0x55f3d3b7e160 msgr2=0x55f3d3b7e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:13.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.488+0000 7fabe7e65740 1 -- 172.21.15.33:0/3391645926 shutdown_connections 2024-03-19T02:59:13.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.488+0000 7fabe7e65740 1 -- 172.21.15.33:0/3391645926 wait complete. 2024-03-19T02:59:13.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.488+0000 7fabe7e65740 1 -- start start 2024-03-19T02:59:13.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.489+0000 7fabe7e65740 1 Processor -- start 2024-03-19T02:59:13.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.489+0000 7fabe7e65740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3b83a30 0x55f3d3b89f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.489+0000 7fabe7e65740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3d3b84350 0x55f3d3b8a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.489+0000 7fabe7e65740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f3d3b8cf30 0x55f3d3b8a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.489+0000 7fabe7e65740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f3d3b8fcc0 con 0x55f3d3b83a30 2024-03-19T02:59:13.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.489+0000 7fabe7e65740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55f3d3aa3160 con 0x55f3d3b84350 2024-03-19T02:59:13.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.489+0000 7fabe7e65740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f3d3b8fe40 con 0x55f3d3b8cf30 2024-03-19T02:59:13.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.489+0000 7fabe7e65740 10 client.0 _ll_register_callbacks cb 0x55f3d3adfa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:59:13.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.489+0000 7fabdeffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3b83a30 0x55f3d3b89f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.489+0000 7fabdeffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3b83a30 0x55f3d3b89f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51274/0 (socket says 172.21.15.33:51274) 2024-03-19T02:59:13.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.489+0000 7fabdeffd640 1 -- 172.21.15.33:0/853548668 learned_addr learned my addr 172.21.15.33:0/853548668 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:13.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.489+0000 7fabde7fc640 1 --2- 172.21.15.33:0/853548668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3d3b84350 0x55f3d3b8a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.489+0000 7fabdf7fe640 1 --2- 172.21.15.33:0/853548668 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f3d3b8cf30 0x55f3d3b8a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.489+0000 7fabdeffd640 1 -- 172.21.15.33:0/853548668 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f3d3b8cf30 msgr2=0x55f3d3b8a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:13.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.489+0000 7fabdeffd640 1 --2- 172.21.15.33:0/853548668 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f3d3b8cf30 0x55f3d3b8a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.489+0000 7fabdeffd640 1 -- 172.21.15.33:0/853548668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3d3b84350 msgr2=0x55f3d3b8a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:13.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.489+0000 7fabdeffd640 1 --2- 172.21.15.33:0/853548668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3d3b84350 0x55f3d3b8a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.489+0000 7fabdeffd640 1 -- 172.21.15.33:0/853548668 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3d38c57d0 con 0x55f3d3b83a30 2024-03-19T02:59:13.498 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.489+0000 7fabdeffd640 1 --2- 172.21.15.33:0/853548668 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3b83a30 0x55f3d3b89f00 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fabd0000f80 tx=0x7fabd0009910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eb1455ab5bd1572 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:13.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.489+0000 7fabc2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:59:13.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.490+0000 7fabc2ff5640 1 -- 172.21.15.33:0/853548668 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd00eef90 con 0x55f3d3b83a30 2024-03-19T02:59:13.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.490+0000 7fabc2ff5640 1 -- 172.21.15.33:0/853548668 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fabd00f1540 con 0x55f3d3b83a30 2024-03-19T02:59:13.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.490+0000 7fabe7e65740 1 -- 172.21.15.33:0/853548668 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fabd8009df0 con 0x55f3d3b83a30 2024-03-19T02:59:13.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.490+0000 7fabc2ff5640 1 -- 172.21.15.33:0/853548668 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fabd00be220 con 0x55f3d3b83a30 2024-03-19T02:59:13.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.490+0000 7fabe7e65740 10 client.16848 fetch_fsmap learned FSMap version 25 2024-03-19T02:59:13.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.490+0000 7fabe7e65740 1 -- 172.21.15.33:0/853548668 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f3d37fb3f0 con 0x55f3d3b83a30 2024-03-19T02:59:13.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.490+0000 7fabc2ff5640 1 -- 172.21.15.33:0/853548668 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fabd8009df0 con 0x55f3d3b83a30 2024-03-19T02:59:13.499 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.490+0000 7fabc2ff5640 1 -- 172.21.15.33:0/853548668 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fabd00f1020 con 0x55f3d3b83a30 2024-03-19T02:59:13.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.490+0000 7fabe7e65740 10 client.16848 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:59:13.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.490+0000 7fabe7e65740 10 client.16848 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:59:13.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.490+0000 7fabe7e65740 20 client.16848 populate_metadata read hostname 'smithi033' 2024-03-19T02:59:13.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.490+0000 7fabe7e65740 1 --2- 172.21.15.33:0/853548668 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f3d3b9d210 0x55f3d3b9f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.490+0000 7fabe7e65740 4 client.16848 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:59:13.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.490+0000 7fabe7e65740 1 -- 172.21.15.33:0/853548668 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f3d3af4a40 con 0x55f3d3b9d210 2024-03-19T02:59:13.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.490+0000 7fabde7fc640 1 --2- 172.21.15.33:0/853548668 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f3d3b9d210 0x55f3d3b9f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.491+0000 7fabc2ff5640 10 client.16848 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:59:13.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.491+0000 7fabde7fc640 1 --2- 172.21.15.33:0/853548668 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f3d3b9d210 0x55f3d3b9f6b0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9718425d16fb2c69 server_cookie=ed9862dea8a835aa in_seq=0 out_seq=0 2024-03-19T02:59:13.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.491+0000 7fabc2ff5640 1 -- 172.21.15.33:0/853548668 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f3d3af4a40 con 0x55f3d3b9d210 2024-03-19T02:59:13.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.491+0000 7fabc2ff5640 10 client.16848 handle_command_reply: tid=0 2024-03-19T02:59:13.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.491+0000 7fabe7e65740 1 client.16848 shutdown 2024-03-19T02:59:13.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.491+0000 7fabe7e65740 1 -- 172.21.15.33:0/853548668 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3b83a30 msgr2=0x55f3d3b89f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:13.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.492+0000 7fabe7e65740 1 --2- 172.21.15.33:0/853548668 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3b83a30 0x55f3d3b89f00 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fabd0000f80 tx=0x7fabd0009910 comp rx=0 tx=0).stop 2024-03-19T02:59:13.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.492+0000 7fabe7e65740 1 -- 172.21.15.33:0/853548668 shutdown_connections 2024-03-19T02:59:13.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.492+0000 7fabe7e65740 1 --2- 172.21.15.33:0/853548668 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f3d3b9d210 0x55f3d3b9f6b0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.492+0000 7fabe7e65740 1 --2- 172.21.15.33:0/853548668 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f3d3b8cf30 0x55f3d3b8a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.492+0000 7fabe7e65740 1 --2- 172.21.15.33:0/853548668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3d3b84350 0x55f3d3b8a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.492+0000 7fabc2ff5640 0 client.16848 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:59:13.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.492+0000 7fabe7e65740 1 --2- 172.21.15.33:0/853548668 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3b83a30 0x55f3d3b89f00 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.492+0000 7fabe7e65740 1 -- 172.21.15.33:0/853548668 >> 172.21.15.33:0/853548668 conn(0x55f3d3b7e160 msgr2=0x55f3d3b7feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:13.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.492+0000 7fabe7e65740 1 -- 172.21.15.33:0/853548668 shutdown_connections 2024-03-19T02:59:13.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.492+0000 7fabe7e65740 1 -- 172.21.15.33:0/853548668 wait complete. 2024-03-19T02:59:13.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.492+0000 7fabe7e65740 20 client.16848 trim_cache size 0 max 16384 2024-03-19T02:59:13.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.509+0000 7fabe7e65740 1 Processor -- start 2024-03-19T02:59:13.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.509+0000 7fabe7e65740 1 -- start start 2024-03-19T02:59:13.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.509+0000 7fabe7e65740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f3d3b83a30 0x55f3d3b8c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.509+0000 7fabe7e65740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3d3b84350 0x55f3d3b88e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.509+0000 7fabe7e65740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3b8cf30 0x55f3d3b78f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.509+0000 7fabe7e65740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fabe01b13f0 con 0x55f3d3b8cf30 2024-03-19T02:59:13.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.509+0000 7fabe7e65740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fabe01aecc0 con 0x55f3d3b84350 2024-03-19T02:59:13.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.509+0000 7fabe7e65740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fabe01acaf0 con 0x55f3d3b83a30 2024-03-19T02:59:13.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.509+0000 7fabdf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3b8cf30 0x55f3d3b78f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.509+0000 7fabdf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3b8cf30 0x55f3d3b78f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51288/0 (socket says 172.21.15.33:51288) 2024-03-19T02:59:13.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.509+0000 7fabdf7fe640 1 -- 172.21.15.33:0/3730669216 learned_addr learned my addr 172.21.15.33:0/3730669216 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:13.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.509+0000 7fabdeffd640 1 --2- 172.21.15.33:0/3730669216 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f3d3b83a30 0x55f3d3b8c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.510+0000 7fabde7fc640 1 --2- 172.21.15.33:0/3730669216 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3d3b84350 0x55f3d3b88e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.510+0000 7fabdf7fe640 1 -- 172.21.15.33:0/3730669216 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f3d3b83a30 msgr2=0x55f3d3b8c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:13.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.510+0000 7fabdf7fe640 1 --2- 172.21.15.33:0/3730669216 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f3d3b83a30 0x55f3d3b8c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.510+0000 7fabdf7fe640 1 -- 172.21.15.33:0/3730669216 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3d3b84350 msgr2=0x55f3d3b88e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:13.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.510+0000 7fabdf7fe640 1 --2- 172.21.15.33:0/3730669216 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3d3b84350 0x55f3d3b88e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.510+0000 7fabdf7fe640 1 -- 172.21.15.33:0/3730669216 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3d37bb0f0 con 0x55f3d3b8cf30 2024-03-19T02:59:13.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.510+0000 7fabdf7fe640 1 --2- 172.21.15.33:0/3730669216 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3b8cf30 0x55f3d3b78f80 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fabd800ffd0 tx=0x7fabd800fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6524dbc9992154bd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:13.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.510+0000 7fabc4ff9640 1 -- 172.21.15.33:0/3730669216 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd8034070 con 0x55f3d3b8cf30 2024-03-19T02:59:13.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.510+0000 7fabc4ff9640 1 -- 172.21.15.33:0/3730669216 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fabd8009df0 con 0x55f3d3b8cf30 2024-03-19T02:59:13.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.510+0000 7fabc4ff9640 1 -- 172.21.15.33:0/3730669216 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd8022040 con 0x55f3d3b8cf30 2024-03-19T02:59:13.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.511+0000 7fabe7e65740 1 -- 172.21.15.33:0/3730669216 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3b8cf30 msgr2=0x55f3d3b78f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:13.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.511+0000 7fabe7e65740 1 --2- 172.21.15.33:0/3730669216 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3b8cf30 0x55f3d3b78f80 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fabd800ffd0 tx=0x7fabd800fd40 comp rx=0 tx=0).stop 2024-03-19T02:59:13.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.511+0000 7fabe7e65740 1 -- 172.21.15.33:0/3730669216 shutdown_connections 2024-03-19T02:59:13.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.511+0000 7fabe7e65740 1 --2- 172.21.15.33:0/3730669216 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3b8cf30 0x55f3d3b78f80 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.511+0000 7fabe7e65740 1 --2- 172.21.15.33:0/3730669216 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3d3b84350 0x55f3d3b88e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.511+0000 7fabe7e65740 1 --2- 172.21.15.33:0/3730669216 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f3d3b83a30 0x55f3d3b8c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.511+0000 7fabe7e65740 1 -- 172.21.15.33:0/3730669216 >> 172.21.15.33:0/3730669216 conn(0x55f3d3b7e160 msgr2=0x55f3d3b7fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:13.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.511+0000 7fabe7e65740 1 -- 172.21.15.33:0/3730669216 shutdown_connections 2024-03-19T02:59:13.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.511+0000 7fabe7e65740 1 -- 172.21.15.33:0/3730669216 wait complete. 2024-03-19T02:59:13.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.511+0000 7fabe7e65740 1 -- start start 2024-03-19T02:59:13.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.511+0000 7fabe7e65740 1 Processor -- start 2024-03-19T02:59:13.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.511+0000 7fabe7e65740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3b83a30 0x55f3d3b90d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.511+0000 7fabe7e65740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f3d3b84350 0x55f3d3b91250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.511+0000 7fabe7e65740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3d3b8cf30 0x55f3d3c025f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.511+0000 7fabe7e65740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fabe01ad180 con 0x55f3d3b83a30 2024-03-19T02:59:13.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.511+0000 7fabe7e65740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fabe01ae5f0 con 0x55f3d3b8cf30 2024-03-19T02:59:13.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.511+0000 7fabe7e65740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fabe01ad850 con 0x55f3d3b84350 2024-03-19T02:59:13.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.511+0000 7fabe7e65740 10 client.0 _ll_register_callbacks cb 0x7fabd0009840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:59:13.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.511+0000 7fabdeffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3b83a30 0x55f3d3b90d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.511+0000 7fabdeffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3b83a30 0x55f3d3b90d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51296/0 (socket says 172.21.15.33:51296) 2024-03-19T02:59:13.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.511+0000 7fabdeffd640 1 -- 172.21.15.33:0/2247349385 learned_addr learned my addr 172.21.15.33:0/2247349385 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:13.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.511+0000 7fabdf7fe640 1 --2- 172.21.15.33:0/2247349385 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3d3b8cf30 0x55f3d3c025f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.511+0000 7fabde7fc640 1 --2- 172.21.15.33:0/2247349385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f3d3b84350 0x55f3d3b91250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.512+0000 7fabdeffd640 1 -- 172.21.15.33:0/2247349385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f3d3b84350 msgr2=0x55f3d3b91250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:13.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.512+0000 7fabdeffd640 1 --2- 172.21.15.33:0/2247349385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f3d3b84350 0x55f3d3b91250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.512+0000 7fabdeffd640 1 -- 172.21.15.33:0/2247349385 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3d3b8cf30 msgr2=0x55f3d3c025f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:13.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.512+0000 7fabdeffd640 1 --2- 172.21.15.33:0/2247349385 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3d3b8cf30 0x55f3d3c025f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.512+0000 7fabdeffd640 1 -- 172.21.15.33:0/2247349385 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3d3b91b20 con 0x55f3d3b83a30 2024-03-19T02:59:13.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.512+0000 7fabde7fc640 1 --2- 172.21.15.33:0/2247349385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f3d3b84350 0x55f3d3b91250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:59:13.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.512+0000 7fabdf7fe640 1 --2- 172.21.15.33:0/2247349385 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3d3b8cf30 0x55f3d3c025f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:59:13.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.512+0000 7fabdeffd640 1 --2- 172.21.15.33:0/2247349385 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3b83a30 0x55f3d3b90d10 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fabd00b9f80 tx=0x7fabd0003df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7e1ce2e93ca1e56 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:13.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.512+0000 7fabc2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:59:13.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.512+0000 7fabc2ff5640 1 -- 172.21.15.33:0/2247349385 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd00da070 con 0x55f3d3b83a30 2024-03-19T02:59:13.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.512+0000 7fabc2ff5640 1 -- 172.21.15.33:0/2247349385 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fabd00dc2f0 con 0x55f3d3b83a30 2024-03-19T02:59:13.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.512+0000 7fabe7e65740 1 -- 172.21.15.33:0/2247349385 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fabd8009df0 con 0x55f3d3b83a30 2024-03-19T02:59:13.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.512+0000 7fabc2ff5640 1 -- 172.21.15.33:0/2247349385 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fabd00eef90 con 0x55f3d3b83a30 2024-03-19T02:59:13.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.512+0000 7fabe7e65740 10 client.16854 fetch_fsmap learned FSMap version 25 2024-03-19T02:59:13.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.512+0000 7fabe7e65740 1 -- 172.21.15.33:0/2247349385 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f3d3c03c40 con 0x55f3d3b83a30 2024-03-19T02:59:13.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.512+0000 7fabc2ff5640 1 -- 172.21.15.33:0/2247349385 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fabd8009df0 con 0x55f3d3b83a30 2024-03-19T02:59:13.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.512+0000 7fabc2ff5640 1 -- 172.21.15.33:0/2247349385 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fabd00f1040 con 0x55f3d3b83a30 2024-03-19T02:59:13.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.512+0000 7fabe7e65740 10 client.16854 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:59:13.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.512+0000 7fabe7e65740 10 client.16854 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:59:13.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.512+0000 7fabe7e65740 20 client.16854 populate_metadata read hostname 'smithi033' 2024-03-19T02:59:13.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.512+0000 7fabe7e65740 1 --2- 172.21.15.33:0/2247349385 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f3d3c04170 0x55f3d3c06560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.512+0000 7fabe7e65740 4 client.16854 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:59:13.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.512+0000 7fabe7e65740 1 -- 172.21.15.33:0/2247349385 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f3d3aeed90 con 0x55f3d3c04170 2024-03-19T02:59:13.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.513+0000 7fabde7fc640 1 --2- 172.21.15.33:0/2247349385 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f3d3c04170 0x55f3d3c06560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.513+0000 7fabde7fc640 1 --2- 172.21.15.33:0/2247349385 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f3d3c04170 0x55f3d3c06560 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=364c6a094c5a3e3e server_cookie=924c714468112ce9 in_seq=0 out_seq=0 2024-03-19T02:59:13.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.513+0000 7fabc2ff5640 10 client.16854 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:59:13.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.513+0000 7fabc2ff5640 1 -- 172.21.15.33:0/2247349385 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f3d3aeed90 con 0x55f3d3c04170 2024-03-19T02:59:13.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.513+0000 7fabc2ff5640 10 client.16854 handle_command_reply: tid=0 2024-03-19T02:59:13.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.513+0000 7fabe7e65740 1 client.16854 shutdown 2024-03-19T02:59:13.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.513+0000 7fabe7e65740 1 -- 172.21.15.33:0/2247349385 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3b83a30 msgr2=0x55f3d3b90d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:13.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.513+0000 7fabe7e65740 1 --2- 172.21.15.33:0/2247349385 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3b83a30 0x55f3d3b90d10 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fabd00b9f80 tx=0x7fabd0003df0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.513+0000 7fabe7e65740 1 -- 172.21.15.33:0/2247349385 shutdown_connections 2024-03-19T02:59:13.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.513+0000 7fabe7e65740 1 --2- 172.21.15.33:0/2247349385 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f3d3c04170 0x55f3d3c06560 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.513+0000 7fabe7e65740 1 --2- 172.21.15.33:0/2247349385 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f3d3b84350 0x55f3d3b91250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.513+0000 7fabe7e65740 1 --2- 172.21.15.33:0/2247349385 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f3d3b8cf30 0x55f3d3c025f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.513+0000 7fabc2ff5640 0 client.16854 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:59:13.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.513+0000 7fabe7e65740 1 --2- 172.21.15.33:0/2247349385 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f3d3b83a30 0x55f3d3b90d10 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.522 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.513+0000 7fabe7e65740 1 -- 172.21.15.33:0/2247349385 >> 172.21.15.33:0/2247349385 conn(0x55f3d3b7e160 msgr2=0x55f3d3b7f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:13.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.513+0000 7fabe7e65740 1 -- 172.21.15.33:0/2247349385 shutdown_connections 2024-03-19T02:59:13.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.513+0000 7fabe7e65740 1 -- 172.21.15.33:0/2247349385 wait complete. 2024-03-19T02:59:13.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.513+0000 7fabe7e65740 20 client.16854 trim_cache size 0 max 16384 2024-03-19T02:59:13.523 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:59:13.523 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-03-19T02:59:13.523 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-03-19T02:59:13.523 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:59:13.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.514+0000 7fabe527e640 1 -- 172.21.15.33:0/517311491 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fabc80a2970 msgr2=0x7fabc80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:13.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.514+0000 7fabe527e640 1 --2- 172.21.15.33:0/517311491 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fabc80a2970 0x7fabc80a4e30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fabd800fc90 tx=0x7fabd8018000 comp rx=0 tx=0).stop 2024-03-19T02:59:13.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.514+0000 7fabe527e640 1 -- 172.21.15.33:0/517311491 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabe0102a90 msgr2=0x7fabe01980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:13.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.514+0000 7fabe527e640 1 --2- 172.21.15.33:0/517311491 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabe0102a90 0x7fabe01980a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fabd002c4c0 tx=0x7fabd0033610 comp rx=0 tx=0).stop 2024-03-19T02:59:13.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.514+0000 7fabe527e640 1 -- 172.21.15.33:0/517311491 shutdown_connections 2024-03-19T02:59:13.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.514+0000 7fabe527e640 1 --2- 172.21.15.33:0/517311491 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fabc80a2970 0x7fabc80a4e30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.514+0000 7fabe527e640 1 --2- 172.21.15.33:0/517311491 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fabe01033b0 0x7fabe01985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.514+0000 7fabe527e640 1 --2- 172.21.15.33:0/517311491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fabe010be60 0x7fabe019c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.514+0000 7fabe527e640 1 --2- 172.21.15.33:0/517311491 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabe0102a90 0x7fabe01980a0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.514+0000 7fabe527e640 1 -- 172.21.15.33:0/517311491 >> 172.21.15.33:0/517311491 conn(0x7fabe0077880 msgr2=0x7fabe00fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:13.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.514+0000 7fabe527e640 1 -- 172.21.15.33:0/517311491 shutdown_connections 2024-03-19T02:59:13.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.514+0000 7fabe527e640 1 -- 172.21.15.33:0/517311491 wait complete. 2024-03-19T02:59:13.526 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:59:13.526 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:59:13.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.629+0000 7f41c2522640 1 -- 172.21.15.33:0/980319120 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41bc107820 msgr2=0x7f41bc107c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:13.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.629+0000 7f41c2522640 1 --2- 172.21.15.33:0/980319120 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41bc107820 0x7f41bc107c00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f41a4009900 tx=0x7f41a402bcd0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.629+0000 7f41c2522640 1 -- 172.21.15.33:0/980319120 shutdown_connections 2024-03-19T02:59:13.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.629+0000 7f41c2522640 1 --2- 172.21.15.33:0/980319120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41bc069ed0 0x7f41bc06a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.629+0000 7f41c2522640 1 --2- 172.21.15.33:0/980319120 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f41bc06a870 0x7f41bc10e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.629+0000 7f41c2522640 1 --2- 172.21.15.33:0/980319120 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41bc107820 0x7f41bc107c00 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.629+0000 7f41c2522640 1 -- 172.21.15.33:0/980319120 >> 172.21.15.33:0/980319120 conn(0x7f41bc0777c0 msgr2=0x7f41bc077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:13.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.629+0000 7f41c2522640 1 -- 172.21.15.33:0/980319120 shutdown_connections 2024-03-19T02:59:13.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.629+0000 7f41c2522640 1 -- 172.21.15.33:0/980319120 wait complete. 2024-03-19T02:59:13.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.630+0000 7f41c2522640 1 Processor -- start 2024-03-19T02:59:13.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.630+0000 7f41c2522640 1 -- start start 2024-03-19T02:59:13.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.630+0000 7f41c2522640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f41bc069ed0 0x7f41bc197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.630+0000 7f41c2522640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41bc06a870 0x7f41bc198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.630+0000 7f41c2522640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41bc107820 0x7f41bc19c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.630+0000 7f41c2522640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f41bc111a90 con 0x7f41bc069ed0 2024-03-19T02:59:13.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.630+0000 7f41c2522640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f41bc111910 con 0x7f41bc107820 2024-03-19T02:59:13.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.630+0000 7f41c2522640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f41bc111c10 con 0x7f41bc06a870 2024-03-19T02:59:13.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.630+0000 7f41bbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f41bc069ed0 0x7f41bc197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.631+0000 7f41bbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f41bc069ed0 0x7f41bc197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51318/0 (socket says 172.21.15.33:51318) 2024-03-19T02:59:13.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.631+0000 7f41bbfff640 1 -- 172.21.15.33:0/2694523917 learned_addr learned my addr 172.21.15.33:0/2694523917 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:13.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.631+0000 7f41c0abf640 1 --2- 172.21.15.33:0/2694523917 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41bc107820 0x7f41bc19c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.631+0000 7f41bb7fe640 1 --2- 172.21.15.33:0/2694523917 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41bc06a870 0x7f41bc198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.631+0000 7f41bbfff640 1 -- 172.21.15.33:0/2694523917 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41bc06a870 msgr2=0x7f41bc198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:13.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.631+0000 7f41bbfff640 1 --2- 172.21.15.33:0/2694523917 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41bc06a870 0x7f41bc198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.631+0000 7f41bbfff640 1 -- 172.21.15.33:0/2694523917 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41bc107820 msgr2=0x7f41bc19c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:13.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.631+0000 7f41bbfff640 1 --2- 172.21.15.33:0/2694523917 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41bc107820 0x7f41bc19c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.631+0000 7f41bbfff640 1 -- 172.21.15.33:0/2694523917 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41bc19cfe0 con 0x7f41bc069ed0 2024-03-19T02:59:13.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.631+0000 7f41bbfff640 1 --2- 172.21.15.33:0/2694523917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f41bc069ed0 0x7f41bc197ff0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f41a4005940 tx=0x7f41a402c690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:13.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.631+0000 7f41b97fa640 1 -- 172.21.15.33:0/2694523917 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a4043020 con 0x7f41bc069ed0 2024-03-19T02:59:13.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.631+0000 7f41c2522640 1 -- 172.21.15.33:0/2694523917 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41bc19d270 con 0x7f41bc069ed0 2024-03-19T02:59:13.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.631+0000 7f41b97fa640 1 -- 172.21.15.33:0/2694523917 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41a4004e00 con 0x7f41bc069ed0 2024-03-19T02:59:13.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.631+0000 7f41b97fa640 1 -- 172.21.15.33:0/2694523917 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a4003970 con 0x7f41bc069ed0 2024-03-19T02:59:13.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.631+0000 7f41c2522640 1 -- 172.21.15.33:0/2694523917 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41bc1a4b60 con 0x7f41bc069ed0 2024-03-19T02:59:13.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.632+0000 7f41c2522640 1 -- 172.21.15.33:0/2694523917 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41bc102f40 con 0x7f41bc069ed0 2024-03-19T02:59:13.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.633+0000 7f41b97fa640 1 -- 172.21.15.33:0/2694523917 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f41a4046020 con 0x7f41bc069ed0 2024-03-19T02:59:13.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.634+0000 7f41b97fa640 1 --2- 172.21.15.33:0/2694523917 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f41a80a2970 0x7f41a80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.634+0000 7f41b97fa640 1 -- 172.21.15.33:0/2694523917 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f41a40f1690 con 0x7f41bc069ed0 2024-03-19T02:59:13.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.634+0000 7f41bb7fe640 1 --2- 172.21.15.33:0/2694523917 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f41a80a2970 0x7f41a80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.634+0000 7f41bb7fe640 1 --2- 172.21.15.33:0/2694523917 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f41a80a2970 0x7f41a80a4e30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f41bc077ea0 tx=0x7f41ac005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:13.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.638+0000 7f41b97fa640 1 -- 172.21.15.33:0/2694523917 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f41a404d020 con 0x7f41bc069ed0 2024-03-19T02:59:13.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.862+0000 7f41c2522640 1 -- 172.21.15.33:0/2694523917 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f41bc068a60 con 0x7f41bc069ed0 2024-03-19T02:59:13.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.863+0000 7f41b97fa640 1 -- 172.21.15.33:0/2694523917 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11867 (secure 0 0 0) 0x7f41bc068a60 con 0x7f41bc069ed0 2024-03-19T02:59:13.864 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:59:13.865 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:50:52.201798+0000","modified":"2024-03-19T02:53:11.748697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24488,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/3731232169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":3731232169},{"type":"v1","addr":"172.21.15.96:6839","nonce":3731232169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24457,24488,24509]},"id":1}]} 2024-03-19T02:59:13.865 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-03-19T02:59:13.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.865+0000 7f41c2522640 1 -- 172.21.15.33:0/2694523917 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f41a80a2970 msgr2=0x7f41a80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:13.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.865+0000 7f41c2522640 1 --2- 172.21.15.33:0/2694523917 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f41a80a2970 0x7f41a80a4e30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f41bc077ea0 tx=0x7f41ac005f90 comp rx=0 tx=0).stop 2024-03-19T02:59:13.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.865+0000 7f41c2522640 1 -- 172.21.15.33:0/2694523917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f41bc069ed0 msgr2=0x7f41bc197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:13.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.865+0000 7f41c2522640 1 --2- 172.21.15.33:0/2694523917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f41bc069ed0 0x7f41bc197ff0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f41a4005940 tx=0x7f41a402c690 comp rx=0 tx=0).stop 2024-03-19T02:59:13.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.865+0000 7f41c2522640 1 -- 172.21.15.33:0/2694523917 shutdown_connections 2024-03-19T02:59:13.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.865+0000 7f41c2522640 1 --2- 172.21.15.33:0/2694523917 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f41a80a2970 0x7f41a80a4e30 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.865+0000 7f41c2522640 1 --2- 172.21.15.33:0/2694523917 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41bc06a870 0x7f41bc198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.865+0000 7f41c2522640 1 --2- 172.21.15.33:0/2694523917 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41bc107820 0x7f41bc19c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.865+0000 7f41c2522640 1 --2- 172.21.15.33:0/2694523917 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f41bc069ed0 0x7f41bc197ff0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.865+0000 7f41c2522640 1 -- 172.21.15.33:0/2694523917 >> 172.21.15.33:0/2694523917 conn(0x7f41bc0777c0 msgr2=0x7f41bc0ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:13.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.865+0000 7f41c2522640 1 -- 172.21.15.33:0/2694523917 shutdown_connections 2024-03-19T02:59:13.867 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.865+0000 7f41c2522640 1 -- 172.21.15.33:0/2694523917 wait complete. 2024-03-19T02:59:13.878 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:59:13.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.981+0000 7f98882ff640 1 -- 172.21.15.33:0/3913808933 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9880103360 msgr2=0x7f988010b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:13.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.981+0000 7f98882ff640 1 --2- 172.21.15.33:0/3913808933 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9880103360 0x7f988010b840 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f9870009900 tx=0x7f987002bcd0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.982+0000 7f98882ff640 1 -- 172.21.15.33:0/3913808933 shutdown_connections 2024-03-19T02:59:13.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.982+0000 7f98882ff640 1 --2- 172.21.15.33:0/3913808933 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f988010bd80 0x7f988010e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.982+0000 7f98882ff640 1 --2- 172.21.15.33:0/3913808933 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9880103360 0x7f988010b840 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.982+0000 7f98882ff640 1 --2- 172.21.15.33:0/3913808933 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9880102a40 0x7f9880102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.982+0000 7f98882ff640 1 -- 172.21.15.33:0/3913808933 >> 172.21.15.33:0/3913808933 conn(0x7f98800778d0 msgr2=0x7f9880077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:13.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.982+0000 7f98882ff640 1 -- 172.21.15.33:0/3913808933 shutdown_connections 2024-03-19T02:59:13.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.982+0000 7f98882ff640 1 -- 172.21.15.33:0/3913808933 wait complete. 2024-03-19T02:59:13.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.982+0000 7f98882ff640 1 Processor -- start 2024-03-19T02:59:13.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.982+0000 7f98882ff640 1 -- start start 2024-03-19T02:59:13.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.982+0000 7f98882ff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9880102a40 0x7f9880197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.982+0000 7f98882ff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9880103360 0x7f98801984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.982+0000 7f98882ff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f988010bd80 0x7f988019c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.982+0000 7f98882ff640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9880111a30 con 0x7f9880103360 2024-03-19T02:59:13.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.982+0000 7f98882ff640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f98801118b0 con 0x7f9880102a40 2024-03-19T02:59:13.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.982+0000 7f98882ff640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9880111bb0 con 0x7f988010bd80 2024-03-19T02:59:13.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.982+0000 7f988589a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9880103360 0x7f98801984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.982+0000 7f988609b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9880102a40 0x7f9880197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.982+0000 7f988589a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9880103360 0x7f98801984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51338/0 (socket says 172.21.15.33:51338) 2024-03-19T02:59:13.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.982+0000 7f988589a640 1 -- 172.21.15.33:0/1063400367 learned_addr learned my addr 172.21.15.33:0/1063400367 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:13.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.983+0000 7f988689c640 1 --2- 172.21.15.33:0/1063400367 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f988010bd80 0x7f988019c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.983+0000 7f988589a640 1 -- 172.21.15.33:0/1063400367 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f988010bd80 msgr2=0x7f988019c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:13.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.983+0000 7f988589a640 1 --2- 172.21.15.33:0/1063400367 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f988010bd80 0x7f988019c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.983+0000 7f988589a640 1 -- 172.21.15.33:0/1063400367 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9880102a40 msgr2=0x7f9880197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:13.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.983+0000 7f988589a640 1 --2- 172.21.15.33:0/1063400367 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9880102a40 0x7f9880197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.983+0000 7f988589a640 1 -- 172.21.15.33:0/1063400367 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f988019cf60 con 0x7f9880103360 2024-03-19T02:59:13.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.983+0000 7f988589a640 1 --2- 172.21.15.33:0/1063400367 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9880103360 0x7f98801984b0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f9870005940 tx=0x7f987002c690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:13.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.983+0000 7f98777fe640 1 -- 172.21.15.33:0/1063400367 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9870043020 con 0x7f9880103360 2024-03-19T02:59:13.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.983+0000 7f98777fe640 1 -- 172.21.15.33:0/1063400367 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9870004e00 con 0x7f9880103360 2024-03-19T02:59:13.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.983+0000 7f98882ff640 1 -- 172.21.15.33:0/1063400367 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f988019d1f0 con 0x7f9880103360 2024-03-19T02:59:13.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.983+0000 7f98777fe640 1 -- 172.21.15.33:0/1063400367 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9870003970 con 0x7f9880103360 2024-03-19T02:59:13.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.983+0000 7f98882ff640 1 -- 172.21.15.33:0/1063400367 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98801a4ae0 con 0x7f9880103360 2024-03-19T02:59:13.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.985+0000 7f98777fe640 1 -- 172.21.15.33:0/1063400367 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f9870046020 con 0x7f9880103360 2024-03-19T02:59:13.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.986+0000 7f988afc8740 1 Processor -- start 2024-03-19T02:59:13.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.986+0000 7f98777fe640 1 --2- 172.21.15.33:0/1063400367 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f986c0a29f0 0x7f986c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.986+0000 7f988afc8740 1 -- start start 2024-03-19T02:59:13.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.986+0000 7f988afc8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c65c8a42d0 0x55c65c8a46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.986+0000 7f988afc8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65c8a4bf0 0x55c65c8ad290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.986+0000 7f988afc8740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c65c8ad7d0 0x55c65c8afbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.986+0000 7f988afc8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c65c8074b0 con 0x55c65c8a4bf0 2024-03-19T02:59:13.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.986+0000 7f988689c640 1 --2- 172.21.15.33:0/1063400367 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f986c0a29f0 0x7f986c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.986+0000 7f988afc8740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c65c7afd70 con 0x55c65c8ad7d0 2024-03-19T02:59:13.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.986+0000 7f988afc8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c65c803550 con 0x55c65c8a42d0 2024-03-19T02:59:13.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.986+0000 7f98777fe640 1 -- 172.21.15.33:0/1063400367 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f98700f0f90 con 0x7f9880103360 2024-03-19T02:59:13.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.986+0000 7f988609b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c65c8a42d0 0x55c65c8a46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.986+0000 7f988609b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c65c8a42d0 0x55c65c8a46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:41856/0 (socket says 172.21.15.33:41856) 2024-03-19T02:59:13.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.986+0000 7f988609b640 1 -- 172.21.15.33:0/1619202419 learned_addr learned my addr 172.21.15.33:0/1619202419 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:13.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.986+0000 7f988589a640 1 --2- 172.21.15.33:0/1619202419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65c8a4bf0 0x55c65c8ad290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.986+0000 7f988689c640 1 --2- 172.21.15.33:0/1619202419 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c65c8ad7d0 0x55c65c8afbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.987+0000 7f988609b640 1 -- 172.21.15.33:0/1619202419 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c65c8ad7d0 msgr2=0x55c65c8afbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:13.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.987+0000 7f988609b640 1 --2- 172.21.15.33:0/1619202419 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c65c8ad7d0 0x55c65c8afbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.987+0000 7f988609b640 1 -- 172.21.15.33:0/1619202419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65c8a4bf0 msgr2=0x55c65c8ad290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:13.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.987+0000 7f988609b640 1 --2- 172.21.15.33:0/1619202419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65c8a4bf0 0x55c65c8ad290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.987+0000 7f988609b640 1 -- 172.21.15.33:0/1619202419 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c65c7d2210 con 0x55c65c8a42d0 2024-03-19T02:59:13.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.987+0000 7f988689c640 1 --2- 172.21.15.33:0/1619202419 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c65c8ad7d0 0x55c65c8afbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:59:13.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.987+0000 7f988689c640 1 --2- 172.21.15.33:0/1063400367 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f986c0a29f0 0x7f986c0a4eb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f987c005fd0 tx=0x7f987c011b40 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:13.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.987+0000 7f988609b640 1 --2- 172.21.15.33:0/1619202419 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c65c8a42d0 0x55c65c8a46b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9878009cf0 tx=0x7f9878011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a279689d17c8b8f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:13.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.987+0000 7f9874bfe640 1 -- 172.21.15.33:0/1619202419 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f987801e070 con 0x55c65c8a42d0 2024-03-19T02:59:13.992 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.987+0000 7f9874bfe640 1 -- 172.21.15.33:0/1619202419 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9878009de0 con 0x55c65c8a42d0 2024-03-19T02:59:13.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.987+0000 7f9874bfe640 1 -- 172.21.15.33:0/1619202419 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9878019450 con 0x55c65c8a42d0 2024-03-19T02:59:13.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.988+0000 7f988afc8740 1 -- 172.21.15.33:0/1619202419 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c65c8a42d0 msgr2=0x55c65c8a46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:13.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.988+0000 7f988afc8740 1 --2- 172.21.15.33:0/1619202419 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c65c8a42d0 0x55c65c8a46b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9878009cf0 tx=0x7f9878011810 comp rx=0 tx=0).stop 2024-03-19T02:59:13.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.988+0000 7f988afc8740 1 -- 172.21.15.33:0/1619202419 shutdown_connections 2024-03-19T02:59:13.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.988+0000 7f988afc8740 1 --2- 172.21.15.33:0/1619202419 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c65c8ad7d0 0x55c65c8afbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.988+0000 7f988afc8740 1 --2- 172.21.15.33:0/1619202419 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65c8a4bf0 0x55c65c8ad290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.988+0000 7f988afc8740 1 --2- 172.21.15.33:0/1619202419 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c65c8a42d0 0x55c65c8a46b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.988+0000 7f988afc8740 1 -- 172.21.15.33:0/1619202419 >> 172.21.15.33:0/1619202419 conn(0x55c65c89ea00 msgr2=0x55c65c89ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:13.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.988+0000 7f988afc8740 1 -- 172.21.15.33:0/1619202419 shutdown_connections 2024-03-19T02:59:13.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.988+0000 7f988afc8740 1 -- 172.21.15.33:0/1619202419 wait complete. 2024-03-19T02:59:13.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.988+0000 7f988afc8740 1 -- start start 2024-03-19T02:59:13.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.988+0000 7f988afc8740 1 Processor -- start 2024-03-19T02:59:13.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.988+0000 7f988afc8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65c8a42d0 0x55c65c8aa7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.988+0000 7f988afc8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c65c8a4bf0 0x55c65c8aad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.989+0000 7f988afc8740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c65c8ad7d0 0x55c65c8ab240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.989+0000 7f988afc8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c65c8b05c0 con 0x55c65c8a42d0 2024-03-19T02:59:13.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.989+0000 7f988afc8740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c65c7c3310 con 0x55c65c8ad7d0 2024-03-19T02:59:13.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.989+0000 7f988afc8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c65c8b0740 con 0x55c65c8a4bf0 2024-03-19T02:59:13.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.989+0000 7f988afc8740 10 client.0 _ll_register_callbacks cb 0x55c65c7b72d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:59:13.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.989+0000 7f988609b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65c8a42d0 0x55c65c8aa7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.989+0000 7f988609b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65c8a42d0 0x55c65c8aa7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51366/0 (socket says 172.21.15.33:51366) 2024-03-19T02:59:13.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.989+0000 7f988609b640 1 -- 172.21.15.33:0/1566472457 learned_addr learned my addr 172.21.15.33:0/1566472457 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:13.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.989+0000 7f988689c640 1 --2- 172.21.15.33:0/1566472457 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c65c8ad7d0 0x55c65c8ab240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.989+0000 7f988589a640 1 --2- 172.21.15.33:0/1566472457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c65c8a4bf0 0x55c65c8aad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.989+0000 7f988609b640 1 -- 172.21.15.33:0/1566472457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c65c8a4bf0 msgr2=0x55c65c8aad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:13.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.989+0000 7f988609b640 1 --2- 172.21.15.33:0/1566472457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c65c8a4bf0 0x55c65c8aad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.989+0000 7f988609b640 1 -- 172.21.15.33:0/1566472457 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c65c8ad7d0 msgr2=0x55c65c8ab240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:13.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.989+0000 7f988609b640 1 --2- 172.21.15.33:0/1566472457 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c65c8ad7d0 0x55c65c8ab240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.997 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.989+0000 7f988609b640 1 -- 172.21.15.33:0/1566472457 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c65c51b3f0 con 0x55c65c8a42d0 2024-03-19T02:59:13.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.989+0000 7f988609b640 1 --2- 172.21.15.33:0/1566472457 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65c8a42d0 0x55c65c8aa7c0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f98780052f0 tx=0x7f9878005320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=483407cfff0b02c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:13.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.989+0000 7f9869ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:59:13.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.989+0000 7f9869ffb640 1 -- 172.21.15.33:0/1566472457 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f987801e040 con 0x55c65c8a42d0 2024-03-19T02:59:13.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.989+0000 7f9869ffb640 1 -- 172.21.15.33:0/1566472457 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f9878005540 con 0x55c65c8a42d0 2024-03-19T02:59:13.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.989+0000 7f988afc8740 1 -- 172.21.15.33:0/1566472457 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9878009de0 con 0x55c65c8a42d0 2024-03-19T02:59:13.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.989+0000 7f9869ffb640 1 -- 172.21.15.33:0/1566472457 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9878030050 con 0x55c65c8a42d0 2024-03-19T02:59:13.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.989+0000 7f988afc8740 10 client.16872 fetch_fsmap learned FSMap version 25 2024-03-19T02:59:13.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.989+0000 7f988afc8740 1 -- 172.21.15.33:0/1566472457 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c65c5e57d0 con 0x55c65c8a42d0 2024-03-19T02:59:13.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.990+0000 7f9869ffb640 1 -- 172.21.15.33:0/1566472457 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9878009de0 con 0x55c65c8a42d0 2024-03-19T02:59:13.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.990+0000 7f9869ffb640 1 -- 172.21.15.33:0/1566472457 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f9878021070 con 0x55c65c8a42d0 2024-03-19T02:59:13.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.990+0000 7f988afc8740 10 client.16872 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:59:13.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.990+0000 7f988afc8740 10 client.16872 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T02:59:13.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.990+0000 7f988afc8740 20 client.16872 populate_metadata read hostname 'smithi033' 2024-03-19T02:59:13.998 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.990+0000 7f988afc8740 1 --2- 172.21.15.33:0/1566472457 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55c65c8bdb20 0x55c65c8bff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:13.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.990+0000 7f988afc8740 4 client.16872 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:59:13.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.990+0000 7f988afc8740 1 -- 172.21.15.33:0/1566472457 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c65c8653e0 con 0x55c65c8bdb20 2024-03-19T02:59:13.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.990+0000 7f988589a640 1 --2- 172.21.15.33:0/1566472457 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55c65c8bdb20 0x55c65c8bff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:13.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.990+0000 7f9869ffb640 10 client.16872 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T02:59:13.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.990+0000 7f988589a640 1 --2- 172.21.15.33:0/1566472457 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55c65c8bdb20 0x55c65c8bff10 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cdc0e997314f2e30 server_cookie=ccf53056b69f9f3a in_seq=0 out_seq=0 2024-03-19T02:59:13.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.991+0000 7f9869ffb640 1 -- 172.21.15.33:0/1566472457 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c65c8653e0 con 0x55c65c8bdb20 2024-03-19T02:59:13.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.991+0000 7f9869ffb640 10 client.16872 handle_command_reply: tid=0 2024-03-19T02:59:13.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.991+0000 7f988afc8740 1 client.16872 shutdown 2024-03-19T02:59:13.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.991+0000 7f988afc8740 1 -- 172.21.15.33:0/1566472457 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65c8a42d0 msgr2=0x55c65c8aa7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:13.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.992+0000 7f988afc8740 1 --2- 172.21.15.33:0/1566472457 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65c8a42d0 0x55c65c8aa7c0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f98780052f0 tx=0x7f9878005320 comp rx=0 tx=0).stop 2024-03-19T02:59:13.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.992+0000 7f988afc8740 1 -- 172.21.15.33:0/1566472457 shutdown_connections 2024-03-19T02:59:13.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.992+0000 7f988afc8740 1 --2- 172.21.15.33:0/1566472457 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55c65c8bdb20 0x55c65c8bff10 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.992+0000 7f988afc8740 1 --2- 172.21.15.33:0/1566472457 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c65c8a4bf0 0x55c65c8aad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:13.999 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.992+0000 7f988afc8740 1 --2- 172.21.15.33:0/1566472457 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c65c8ad7d0 0x55c65c8ab240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.992+0000 7f988afc8740 1 --2- 172.21.15.33:0/1566472457 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65c8a42d0 0x55c65c8aa7c0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.992+0000 7f9869ffb640 0 client.16872 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T02:59:14.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.992+0000 7f988afc8740 1 -- 172.21.15.33:0/1566472457 >> 172.21.15.33:0/1566472457 conn(0x55c65c89ea00 msgr2=0x55c65c8a0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:14.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.992+0000 7f988afc8740 1 -- 172.21.15.33:0/1566472457 shutdown_connections 2024-03-19T02:59:14.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.992+0000 7f988afc8740 1 -- 172.21.15.33:0/1566472457 wait complete. 2024-03-19T02:59:14.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:13.992+0000 7f988afc8740 20 client.16872 trim_cache size 0 max 16384 2024-03-19T02:59:14.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.005+0000 7f988afc8740 1 Processor -- start 2024-03-19T02:59:14.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.005+0000 7f988afc8740 1 -- start start 2024-03-19T02:59:14.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.005+0000 7f988afc8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c65c8a42d0 0x55c65c89dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.005+0000 7f988afc8740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c65c8a4bf0 0x55c65c8b9390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.005+0000 7f988afc8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65c8ad7d0 0x55c65c8ab060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.005+0000 7f988afc8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f98801b1390 con 0x55c65c8ad7d0 2024-03-19T02:59:14.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.005+0000 7f988afc8740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f98801aec60 con 0x55c65c8a4bf0 2024-03-19T02:59:14.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.005+0000 7f988afc8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f98801aca90 con 0x55c65c8a42d0 2024-03-19T02:59:14.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.005+0000 7f988689c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65c8ad7d0 0x55c65c8ab060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.005+0000 7f988689c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65c8ad7d0 0x55c65c8ab060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51372/0 (socket says 172.21.15.33:51372) 2024-03-19T02:59:14.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.005+0000 7f988689c640 1 -- 172.21.15.33:0/150151401 learned_addr learned my addr 172.21.15.33:0/150151401 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:14.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.005+0000 7f988589a640 1 --2- 172.21.15.33:0/150151401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c65c8a4bf0 0x55c65c8b9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.005+0000 7f988609b640 1 --2- 172.21.15.33:0/150151401 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c65c8a42d0 0x55c65c89dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.005+0000 7f988689c640 1 -- 172.21.15.33:0/150151401 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c65c8a42d0 msgr2=0x55c65c89dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:14.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.005+0000 7f988689c640 1 --2- 172.21.15.33:0/150151401 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c65c8a42d0 0x55c65c89dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.005+0000 7f988689c640 1 -- 172.21.15.33:0/150151401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c65c8a4bf0 msgr2=0x55c65c8b9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:14.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.005+0000 7f988689c640 1 --2- 172.21.15.33:0/150151401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c65c8a4bf0 0x55c65c8b9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.005+0000 7f988689c640 1 -- 172.21.15.33:0/150151401 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c65c89e830 con 0x55c65c8ad7d0 2024-03-19T02:59:14.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.006+0000 7f988689c640 1 --2- 172.21.15.33:0/150151401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65c8ad7d0 0x55c65c8ab060 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f987c00fe10 tx=0x7f987c01e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4efef20c4b8fb4be server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:14.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.006+0000 7f986bfff640 1 -- 172.21.15.33:0/150151401 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f987c01a040 con 0x55c65c8ad7d0 2024-03-19T02:59:14.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.006+0000 7f986bfff640 1 -- 172.21.15.33:0/150151401 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f987c025070 con 0x55c65c8ad7d0 2024-03-19T02:59:14.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.006+0000 7f986bfff640 1 -- 172.21.15.33:0/150151401 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f987c021d50 con 0x55c65c8ad7d0 2024-03-19T02:59:14.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.006+0000 7f988afc8740 1 -- 172.21.15.33:0/150151401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65c8ad7d0 msgr2=0x55c65c8ab060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:14.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.006+0000 7f988afc8740 1 --2- 172.21.15.33:0/150151401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65c8ad7d0 0x55c65c8ab060 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f987c00fe10 tx=0x7f987c01e000 comp rx=0 tx=0).stop 2024-03-19T02:59:14.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.006+0000 7f988afc8740 1 -- 172.21.15.33:0/150151401 shutdown_connections 2024-03-19T02:59:14.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.006+0000 7f988afc8740 1 --2- 172.21.15.33:0/150151401 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65c8ad7d0 0x55c65c8ab060 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.006+0000 7f988afc8740 1 --2- 172.21.15.33:0/150151401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c65c8a4bf0 0x55c65c8b9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.006+0000 7f988afc8740 1 --2- 172.21.15.33:0/150151401 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c65c8a42d0 0x55c65c89dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.006+0000 7f988afc8740 1 -- 172.21.15.33:0/150151401 >> 172.21.15.33:0/150151401 conn(0x55c65c89ea00 msgr2=0x55c65c8a97f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:14.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.006+0000 7f988afc8740 1 -- 172.21.15.33:0/150151401 shutdown_connections 2024-03-19T02:59:14.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.006+0000 7f988afc8740 1 -- 172.21.15.33:0/150151401 wait complete. 2024-03-19T02:59:14.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.007+0000 7f988afc8740 1 -- start start 2024-03-19T02:59:14.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.007+0000 7f988afc8740 1 Processor -- start 2024-03-19T02:59:14.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.007+0000 7f988afc8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65c8a42d0 0x55c65c8b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.007+0000 7f988afc8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c65c8a4bf0 0x55c65c8b0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.007+0000 7f988afc8740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c65c8ad7d0 0x55c65c8b14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.007+0000 7f988afc8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f98801ad120 con 0x55c65c8a42d0 2024-03-19T02:59:14.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.007+0000 7f988afc8740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f98801ae590 con 0x55c65c8ad7d0 2024-03-19T02:59:14.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.007+0000 7f988afc8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f98801ad7f0 con 0x55c65c8a4bf0 2024-03-19T02:59:14.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.007+0000 7f988afc8740 10 client.0 _ll_register_callbacks cb 0x7f987801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:59:14.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.007+0000 7f988609b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65c8a42d0 0x55c65c8b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.007+0000 7f988609b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65c8a42d0 0x55c65c8b5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51380/0 (socket says 172.21.15.33:51380) 2024-03-19T02:59:14.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.007+0000 7f988609b640 1 -- 172.21.15.33:0/326774264 learned_addr learned my addr 172.21.15.33:0/326774264 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:14.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.007+0000 7f988589a640 1 --2- 172.21.15.33:0/326774264 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c65c8a4bf0 0x55c65c8b0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.007+0000 7f988689c640 1 --2- 172.21.15.33:0/326774264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c65c8ad7d0 0x55c65c8b14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.007+0000 7f988609b640 1 -- 172.21.15.33:0/326774264 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c65c8a4bf0 msgr2=0x55c65c8b0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:14.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.007+0000 7f988609b640 1 --2- 172.21.15.33:0/326774264 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c65c8a4bf0 0x55c65c8b0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.007+0000 7f988609b640 1 -- 172.21.15.33:0/326774264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c65c8ad7d0 msgr2=0x55c65c8b14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:14.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.007+0000 7f988609b640 1 --2- 172.21.15.33:0/326774264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c65c8ad7d0 0x55c65c8b14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.007+0000 7f988609b640 1 -- 172.21.15.33:0/326774264 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c65c8b1a80 con 0x55c65c8a42d0 2024-03-19T02:59:14.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.007+0000 7f988689c640 1 --2- 172.21.15.33:0/326774264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c65c8ad7d0 0x55c65c8b14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:59:14.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.007+0000 7f988609b640 1 --2- 172.21.15.33:0/326774264 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65c8a42d0 0x55c65c8b5ab0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f98780049a0 tx=0x7f9878004f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd45feac079177a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:14.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.007+0000 7f9869ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:59:14.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.008+0000 7f9869ffb640 1 -- 172.21.15.33:0/326774264 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f987801e040 con 0x55c65c8a42d0 2024-03-19T02:59:14.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.008+0000 7f9869ffb640 1 -- 172.21.15.33:0/326774264 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f98780097a0 con 0x55c65c8a42d0 2024-03-19T02:59:14.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.008+0000 7f988afc8740 1 -- 172.21.15.33:0/326774264 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f987c025070 con 0x55c65c8a42d0 2024-03-19T02:59:14.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.008+0000 7f9869ffb640 1 -- 172.21.15.33:0/326774264 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9878005390 con 0x55c65c8a42d0 2024-03-19T02:59:14.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.008+0000 7f988afc8740 10 client.16878 fetch_fsmap learned FSMap version 25 2024-03-19T02:59:14.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.008+0000 7f988afc8740 1 -- 172.21.15.33:0/326774264 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c65c923b20 con 0x55c65c8a42d0 2024-03-19T02:59:14.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.008+0000 7f9869ffb640 1 -- 172.21.15.33:0/326774264 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f987c025070 con 0x55c65c8a42d0 2024-03-19T02:59:14.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.008+0000 7f9869ffb640 1 -- 172.21.15.33:0/326774264 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f9878021040 con 0x55c65c8a42d0 2024-03-19T02:59:14.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.008+0000 7f988afc8740 10 client.16878 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:59:14.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.008+0000 7f988afc8740 10 client.16878 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T02:59:14.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.008+0000 7f988afc8740 20 client.16878 populate_metadata read hostname 'smithi033' 2024-03-19T02:59:14.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.008+0000 7f988afc8740 1 --2- 172.21.15.33:0/326774264 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55c65c9242c0 0x55c65c9266e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.008+0000 7f988afc8740 4 client.16878 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:59:14.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.008+0000 7f988afc8740 1 -- 172.21.15.33:0/326774264 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c65c8149f0 con 0x55c65c9242c0 2024-03-19T02:59:14.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.008+0000 7f988589a640 1 --2- 172.21.15.33:0/326774264 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55c65c9242c0 0x55c65c9266e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.009+0000 7f9869ffb640 10 client.16878 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T02:59:14.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.009+0000 7f988589a640 1 --2- 172.21.15.33:0/326774264 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55c65c9242c0 0x55c65c9266e0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=738f807c563c9845 server_cookie=d6c954551c5d4c60 in_seq=0 out_seq=0 2024-03-19T02:59:14.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.009+0000 7f9869ffb640 1 -- 172.21.15.33:0/326774264 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c65c8149f0 con 0x55c65c9242c0 2024-03-19T02:59:14.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.009+0000 7f9869ffb640 10 client.16878 handle_command_reply: tid=0 2024-03-19T02:59:14.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.009+0000 7f988afc8740 1 client.16878 shutdown 2024-03-19T02:59:14.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.009+0000 7f988afc8740 1 -- 172.21.15.33:0/326774264 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65c8a42d0 msgr2=0x55c65c8b5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:14.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.009+0000 7f988afc8740 1 --2- 172.21.15.33:0/326774264 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65c8a42d0 0x55c65c8b5ab0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f98780049a0 tx=0x7f9878004f50 comp rx=0 tx=0).stop 2024-03-19T02:59:14.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.009+0000 7f988afc8740 1 -- 172.21.15.33:0/326774264 shutdown_connections 2024-03-19T02:59:14.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.009+0000 7f988afc8740 1 --2- 172.21.15.33:0/326774264 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55c65c9242c0 0x55c65c9266e0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.009+0000 7f988afc8740 1 --2- 172.21.15.33:0/326774264 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c65c8a4bf0 0x55c65c8b0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.009+0000 7f9869ffb640 0 client.16878 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T02:59:14.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.009+0000 7f988afc8740 1 --2- 172.21.15.33:0/326774264 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c65c8ad7d0 0x55c65c8b14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.009+0000 7f988afc8740 1 --2- 172.21.15.33:0/326774264 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c65c8a42d0 0x55c65c8b5ab0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.009+0000 7f988afc8740 1 -- 172.21.15.33:0/326774264 >> 172.21.15.33:0/326774264 conn(0x55c65c89ea00 msgr2=0x55c65c89fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:14.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.009+0000 7f988afc8740 1 -- 172.21.15.33:0/326774264 shutdown_connections 2024-03-19T02:59:14.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.009+0000 7f988afc8740 1 -- 172.21.15.33:0/326774264 wait complete. 2024-03-19T02:59:14.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.009+0000 7f988afc8740 20 client.16878 trim_cache size 0 max 16384 2024-03-19T02:59:14.018 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:59:14.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.010+0000 7f98882ff640 1 -- 172.21.15.33:0/1063400367 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f986c0a29f0 msgr2=0x7f986c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:14.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.010+0000 7f98882ff640 1 --2- 172.21.15.33:0/1063400367 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f986c0a29f0 0x7f986c0a4eb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f987c005fd0 tx=0x7f987c011b40 comp rx=0 tx=0).stop 2024-03-19T02:59:14.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.010+0000 7f98882ff640 1 -- 172.21.15.33:0/1063400367 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9880103360 msgr2=0x7f98801984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:14.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.010+0000 7f98882ff640 1 --2- 172.21.15.33:0/1063400367 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9880103360 0x7f98801984b0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f9870005940 tx=0x7f987002c690 comp rx=0 tx=0).stop 2024-03-19T02:59:14.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.010+0000 7f98882ff640 1 -- 172.21.15.33:0/1063400367 shutdown_connections 2024-03-19T02:59:14.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.010+0000 7f98882ff640 1 --2- 172.21.15.33:0/1063400367 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f986c0a29f0 0x7f986c0a4eb0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.010+0000 7f98882ff640 1 --2- 172.21.15.33:0/1063400367 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f988010bd80 0x7f988019c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.010+0000 7f98882ff640 1 --2- 172.21.15.33:0/1063400367 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9880103360 0x7f98801984b0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.010+0000 7f98882ff640 1 --2- 172.21.15.33:0/1063400367 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9880102a40 0x7f9880197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.010+0000 7f98882ff640 1 -- 172.21.15.33:0/1063400367 >> 172.21.15.33:0/1063400367 conn(0x7f98800778d0 msgr2=0x7f98800fd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:14.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.010+0000 7f98882ff640 1 -- 172.21.15.33:0/1063400367 shutdown_connections 2024-03-19T02:59:14.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.010+0000 7f98882ff640 1 -- 172.21.15.33:0/1063400367 wait complete. 2024-03-19T02:59:14.022 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:59:14.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.126+0000 7f6882012640 1 -- 172.21.15.33:0/454235916 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f687c10bd80 msgr2=0x7f687c10e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:14.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.126+0000 7f6882012640 1 --2- 172.21.15.33:0/454235916 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f687c10bd80 0x7f687c10e220 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f6874009ca0 tx=0x7f687402c180 comp rx=0 tx=0).stop 2024-03-19T02:59:14.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.126+0000 7f6882012640 1 -- 172.21.15.33:0/454235916 shutdown_connections 2024-03-19T02:59:14.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.126+0000 7f6882012640 1 --2- 172.21.15.33:0/454235916 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f687c103360 0x7f687c10b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.126+0000 7f6882012640 1 --2- 172.21.15.33:0/454235916 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f687c10bd80 0x7f687c10e220 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.126+0000 7f6882012640 1 --2- 172.21.15.33:0/454235916 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f687c102a40 0x7f687c102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.126+0000 7f6882012640 1 -- 172.21.15.33:0/454235916 >> 172.21.15.33:0/454235916 conn(0x7f687c0778d0 msgr2=0x7f687c077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:14.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.126+0000 7f6882012640 1 -- 172.21.15.33:0/454235916 shutdown_connections 2024-03-19T02:59:14.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.127+0000 7f6882012640 1 -- 172.21.15.33:0/454235916 wait complete. 2024-03-19T02:59:14.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.127+0000 7f6882012640 1 Processor -- start 2024-03-19T02:59:14.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.127+0000 7f6882012640 1 -- start start 2024-03-19T02:59:14.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.127+0000 7f6882012640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f687c102a40 0x7f687c197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.127+0000 7f6882012640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f687c103360 0x7f687c198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.127+0000 7f6882012640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f687c10bd80 0x7f687c19c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.127+0000 7f6882012640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f687c111a30 con 0x7f687c103360 2024-03-19T02:59:14.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.127+0000 7f6882012640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f687c1118b0 con 0x7f687c10bd80 2024-03-19T02:59:14.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.127+0000 7f6882012640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f687c111bb0 con 0x7f687c102a40 2024-03-19T02:59:14.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.128+0000 7f687b7fe640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f687c102a40 0x7f687c197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.128+0000 7f687affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f687c103360 0x7f687c198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.128+0000 7f687b7fe640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f687c102a40 0x7f687c197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:41890/0 (socket says 172.21.15.33:41890) 2024-03-19T02:59:14.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.128+0000 7f687affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f687c103360 0x7f687c198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51396/0 (socket says 172.21.15.33:51396) 2024-03-19T02:59:14.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.128+0000 7f687b7fe640 1 -- 172.21.15.33:0/2470734497 learned_addr learned my addr 172.21.15.33:0/2470734497 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:14.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.128+0000 7f687bfff640 1 --2- 172.21.15.33:0/2470734497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f687c10bd80 0x7f687c19c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.128+0000 7f687b7fe640 1 -- 172.21.15.33:0/2470734497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f687c10bd80 msgr2=0x7f687c19c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:14.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.128+0000 7f687b7fe640 1 --2- 172.21.15.33:0/2470734497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f687c10bd80 0x7f687c19c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.128+0000 7f687b7fe640 1 -- 172.21.15.33:0/2470734497 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f687c103360 msgr2=0x7f687c198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:14.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.128+0000 7f687b7fe640 1 --2- 172.21.15.33:0/2470734497 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f687c103360 0x7f687c198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.128+0000 7f687b7fe640 1 -- 172.21.15.33:0/2470734497 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f687c19cf20 con 0x7f687c102a40 2024-03-19T02:59:14.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.128+0000 7f687affd640 1 --2- 172.21.15.33:0/2470734497 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f687c103360 0x7f687c198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:14.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.128+0000 7f687bfff640 1 --2- 172.21.15.33:0/2470734497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f687c10bd80 0x7f687c19c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:14.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.128+0000 7f687b7fe640 1 --2- 172.21.15.33:0/2470734497 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f687c102a40 0x7f687c197f30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f686c00e9f0 tx=0x7f686c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:14.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.128+0000 7f6878ff9640 1 -- 172.21.15.33:0/2470734497 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f686c00cd80 con 0x7f687c102a40 2024-03-19T02:59:14.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.128+0000 7f6882012640 1 -- 172.21.15.33:0/2470734497 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f687c19d210 con 0x7f687c102a40 2024-03-19T02:59:14.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.128+0000 7f6878ff9640 1 -- 172.21.15.33:0/2470734497 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f686c0108e0 con 0x7f687c102a40 2024-03-19T02:59:14.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.128+0000 7f6878ff9640 1 -- 172.21.15.33:0/2470734497 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f686c018920 con 0x7f687c102a40 2024-03-19T02:59:14.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.128+0000 7f6882012640 1 -- 172.21.15.33:0/2470734497 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f687c1a4aa0 con 0x7f687c102a40 2024-03-19T02:59:14.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.130+0000 7f6878ff9640 1 -- 172.21.15.33:0/2470734497 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f686c010a90 con 0x7f687c102a40 2024-03-19T02:59:14.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.131+0000 7f6884c04740 1 Processor -- start 2024-03-19T02:59:14.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.131+0000 7f6884c04740 1 -- start start 2024-03-19T02:59:14.140 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.131+0000 7f6878ff9640 1 --2- 172.21.15.33:0/2470734497 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f68640a2970 0x7f68640a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.131+0000 7f6884c04740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559f0fec42d0 0x559f0fec46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.131+0000 7f6884c04740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559f0fec4bf0 0x559f0fecd290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.131+0000 7f687bfff640 1 --2- 172.21.15.33:0/2470734497 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f68640a2970 0x7f68640a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.131+0000 7f6884c04740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559f0fecd7d0 0x559f0fecfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.131+0000 7f6884c04740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559f0fe274b0 con 0x559f0fec42d0 2024-03-19T02:59:14.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.131+0000 7f6884c04740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x559f0fdcfd70 con 0x559f0fecd7d0 2024-03-19T02:59:14.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.131+0000 7f6884c04740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x559f0fe23550 con 0x559f0fec4bf0 2024-03-19T02:59:14.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.131+0000 7f6878ff9640 1 -- 172.21.15.33:0/2470734497 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f686c0cf4d0 con 0x7f687c102a40 2024-03-19T02:59:14.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.131+0000 7f687b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559f0fec42d0 0x559f0fec46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.131+0000 7f687affd640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559f0fec4bf0 0x559f0fecd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.131+0000 7f687bfff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559f0fecd7d0 0x559f0fecfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.131+0000 7f687affd640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559f0fec4bf0 0x559f0fecd290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:41898/0 (socket says 172.21.15.33:41898) 2024-03-19T02:59:14.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.131+0000 7f687affd640 1 -- 172.21.15.33:0/517696933 learned_addr learned my addr 172.21.15.33:0/517696933 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:14.141 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.131+0000 7f687b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559f0fec42d0 0x559f0fec46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51406/0 (socket says 172.21.15.33:51406) 2024-03-19T02:59:14.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.132+0000 7f687affd640 1 -- 172.21.15.33:0/517696933 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559f0fecd7d0 msgr2=0x559f0fecfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:14.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.132+0000 7f687affd640 1 --2- 172.21.15.33:0/517696933 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559f0fecd7d0 0x559f0fecfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.132+0000 7f687affd640 1 -- 172.21.15.33:0/517696933 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559f0fec42d0 msgr2=0x559f0fec46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:14.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.132+0000 7f687affd640 1 --2- 172.21.15.33:0/517696933 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559f0fec42d0 0x559f0fec46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.132+0000 7f687affd640 1 -- 172.21.15.33:0/517696933 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f0fdf2210 con 0x559f0fec4bf0 2024-03-19T02:59:14.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.132+0000 7f687b7fe640 1 --2- 172.21.15.33:0/517696933 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559f0fec42d0 0x559f0fec46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:14.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.132+0000 7f687bfff640 1 --2- 172.21.15.33:0/2470734497 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f68640a2970 0x7f68640a4e30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6874004af0 tx=0x7f6874003b30 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:14.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.132+0000 7f687affd640 1 --2- 172.21.15.33:0/517696933 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559f0fec4bf0 0x559f0fecd290 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6870011470 tx=0x7f6870011930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d4ed37c0f0299ad server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:14.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.132+0000 7f6861ffb640 1 -- 172.21.15.33:0/517696933 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6870004310 con 0x559f0fec4bf0 2024-03-19T02:59:14.142 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.132+0000 7f6861ffb640 1 -- 172.21.15.33:0/517696933 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f68700044b0 con 0x559f0fec4bf0 2024-03-19T02:59:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.132+0000 7f6861ffb640 1 -- 172.21.15.33:0/517696933 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f687000fae0 con 0x559f0fec4bf0 2024-03-19T02:59:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.133+0000 7f6884c04740 1 -- 172.21.15.33:0/517696933 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559f0fec4bf0 msgr2=0x559f0fecd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.133+0000 7f6884c04740 1 --2- 172.21.15.33:0/517696933 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559f0fec4bf0 0x559f0fecd290 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6870011470 tx=0x7f6870011930 comp rx=0 tx=0).stop 2024-03-19T02:59:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.133+0000 7f6884c04740 1 -- 172.21.15.33:0/517696933 shutdown_connections 2024-03-19T02:59:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.133+0000 7f6884c04740 1 --2- 172.21.15.33:0/517696933 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559f0fec4bf0 0x559f0fecd290 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.133+0000 7f6884c04740 1 --2- 172.21.15.33:0/517696933 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559f0fecd7d0 0x559f0fecfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.133+0000 7f6884c04740 1 --2- 172.21.15.33:0/517696933 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559f0fec42d0 0x559f0fec46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.133+0000 7f6884c04740 1 -- 172.21.15.33:0/517696933 >> 172.21.15.33:0/517696933 conn(0x559f0febea00 msgr2=0x559f0febede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.133+0000 7f6884c04740 1 -- 172.21.15.33:0/517696933 shutdown_connections 2024-03-19T02:59:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.133+0000 7f6884c04740 1 -- 172.21.15.33:0/517696933 wait complete. 2024-03-19T02:59:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.133+0000 7f6884c04740 1 -- start start 2024-03-19T02:59:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.133+0000 7f6884c04740 1 Processor -- start 2024-03-19T02:59:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.133+0000 7f6884c04740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559f0fec42d0 0x559f0feca8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.143 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.133+0000 7f6884c04740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559f0fec4bf0 0x559f0fecae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.133+0000 7f6884c04740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559f0fecd7d0 0x559f0fecb350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.133+0000 7f6884c04740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x559f0fed05c0 con 0x559f0fecd7d0 2024-03-19T02:59:14.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.133+0000 7f6884c04740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x559f0fde3310 con 0x559f0fec4bf0 2024-03-19T02:59:14.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.133+0000 7f6884c04740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x559f0fed0740 con 0x559f0fec42d0 2024-03-19T02:59:14.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.133+0000 7f6884c04740 10 client.0 _ll_register_callbacks cb 0x559f0fdd72d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:59:14.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.134+0000 7f687bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559f0fecd7d0 0x559f0fecb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.134+0000 7f687affd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559f0fec4bf0 0x559f0fecae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.134+0000 7f687b7fe640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559f0fec42d0 0x559f0feca8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.134+0000 7f687bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559f0fecd7d0 0x559f0fecb350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51410/0 (socket says 172.21.15.33:51410) 2024-03-19T02:59:14.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.134+0000 7f687affd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559f0fec4bf0 0x559f0fecae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:57904/0 (socket says 172.21.15.33:57904) 2024-03-19T02:59:14.144 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.134+0000 7f687bfff640 1 -- 172.21.15.33:0/746540240 learned_addr learned my addr 172.21.15.33:0/746540240 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:14.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.134+0000 7f687bfff640 1 -- 172.21.15.33:0/746540240 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559f0fec42d0 msgr2=0x559f0feca8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:14.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.134+0000 7f687bfff640 1 --2- 172.21.15.33:0/746540240 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559f0fec42d0 0x559f0feca8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.134+0000 7f687bfff640 1 -- 172.21.15.33:0/746540240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559f0fec4bf0 msgr2=0x559f0fecae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:14.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.134+0000 7f687bfff640 1 --2- 172.21.15.33:0/746540240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559f0fec4bf0 0x559f0fecae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.134+0000 7f687bfff640 1 -- 172.21.15.33:0/746540240 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f0fb3b3f0 con 0x559f0fecd7d0 2024-03-19T02:59:14.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.134+0000 7f687b7fe640 1 --2- 172.21.15.33:0/746540240 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559f0fec42d0 0x559f0feca8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:14.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.134+0000 7f687affd640 1 --2- 172.21.15.33:0/746540240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559f0fec4bf0 0x559f0fecae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:14.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.134+0000 7f687bfff640 1 --2- 172.21.15.33:0/746540240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559f0fecd7d0 0x559f0fecb350 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f6874009ca0 tx=0x7f687402c180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3be15e53a298a67 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:14.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.134+0000 7f685affd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:59:14.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.134+0000 7f685affd640 1 -- 172.21.15.33:0/746540240 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6874039040 con 0x559f0fecd7d0 2024-03-19T02:59:14.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.134+0000 7f685affd640 1 -- 172.21.15.33:0/746540240 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f6874032c00 con 0x559f0fecd7d0 2024-03-19T02:59:14.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.134+0000 7f6884c04740 1 -- 172.21.15.33:0/746540240 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f68700044b0 con 0x559f0fecd7d0 2024-03-19T02:59:14.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.134+0000 7f685affd640 1 -- 172.21.15.33:0/746540240 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f687405d050 con 0x559f0fecd7d0 2024-03-19T02:59:14.145 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.134+0000 7f6884c04740 10 client.16890 fetch_fsmap learned FSMap version 25 2024-03-19T02:59:14.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.134+0000 7f6884c04740 1 -- 172.21.15.33:0/746540240 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559f0fc057d0 con 0x559f0fecd7d0 2024-03-19T02:59:14.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.135+0000 7f685affd640 1 -- 172.21.15.33:0/746540240 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f68700044b0 con 0x559f0fecd7d0 2024-03-19T02:59:14.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.135+0000 7f685affd640 1 -- 172.21.15.33:0/746540240 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f687403f040 con 0x559f0fecd7d0 2024-03-19T02:59:14.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.135+0000 7f6884c04740 10 client.16890 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:59:14.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.135+0000 7f6884c04740 10 client.16890 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T02:59:14.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.135+0000 7f6884c04740 20 client.16890 populate_metadata read hostname 'smithi033' 2024-03-19T02:59:14.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.135+0000 7f6884c04740 1 --2- 172.21.15.33:0/746540240 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x559f0feddc80 0x559f0fee0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.135+0000 7f6884c04740 4 client.16890 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:59:14.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.135+0000 7f6884c04740 1 -- 172.21.15.33:0/746540240 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f0fe853e0 con 0x559f0feddc80 2024-03-19T02:59:14.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.135+0000 7f687b7fe640 1 --2- 172.21.15.33:0/746540240 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x559f0feddc80 0x559f0fee0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.135+0000 7f685affd640 10 client.16890 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T02:59:14.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.135+0000 7f687b7fe640 1 --2- 172.21.15.33:0/746540240 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x559f0feddc80 0x559f0fee0090 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c543e33bc2ea7c88 server_cookie=e0c23f6bde9e1041 in_seq=0 out_seq=0 2024-03-19T02:59:14.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.136+0000 7f685affd640 1 -- 172.21.15.33:0/746540240 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559f0fe853e0 con 0x559f0feddc80 2024-03-19T02:59:14.146 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.136+0000 7f685affd640 10 client.16890 handle_command_reply: tid=0 2024-03-19T02:59:14.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.136+0000 7f6884c04740 1 client.16890 shutdown 2024-03-19T02:59:14.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.136+0000 7f6884c04740 1 -- 172.21.15.33:0/746540240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559f0fecd7d0 msgr2=0x559f0fecb350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:14.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.136+0000 7f6884c04740 1 --2- 172.21.15.33:0/746540240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559f0fecd7d0 0x559f0fecb350 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f6874009ca0 tx=0x7f687402c180 comp rx=0 tx=0).stop 2024-03-19T02:59:14.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.136+0000 7f6884c04740 1 -- 172.21.15.33:0/746540240 shutdown_connections 2024-03-19T02:59:14.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.136+0000 7f6884c04740 1 --2- 172.21.15.33:0/746540240 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x559f0feddc80 0x559f0fee0090 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.136+0000 7f6884c04740 1 --2- 172.21.15.33:0/746540240 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559f0fecd7d0 0x559f0fecb350 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.136+0000 7f6884c04740 1 --2- 172.21.15.33:0/746540240 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559f0fec4bf0 0x559f0fecae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.136+0000 7f6884c04740 1 --2- 172.21.15.33:0/746540240 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559f0fec42d0 0x559f0feca8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.136+0000 7f685affd640 0 client.16890 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T02:59:14.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.136+0000 7f6884c04740 1 -- 172.21.15.33:0/746540240 >> 172.21.15.33:0/746540240 conn(0x559f0febea00 msgr2=0x559f0fec0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:14.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.136+0000 7f6884c04740 1 -- 172.21.15.33:0/746540240 shutdown_connections 2024-03-19T02:59:14.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.136+0000 7f6884c04740 1 -- 172.21.15.33:0/746540240 wait complete. 2024-03-19T02:59:14.147 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.136+0000 7f6884c04740 20 client.16890 trim_cache size 0 max 16384 2024-03-19T02:59:14.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.150+0000 7f6884c04740 1 Processor -- start 2024-03-19T02:59:14.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.150+0000 7f6884c04740 1 -- start start 2024-03-19T02:59:14.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.150+0000 7f6884c04740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559f0fec42d0 0x559f0fecd000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.150+0000 7f6884c04740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559f0fec4bf0 0x559f0fed9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.150+0000 7f6884c04740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559f0fecd7d0 0x559f0fee5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.150+0000 7f6884c04740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f687c1b1430 con 0x559f0fec4bf0 2024-03-19T02:59:14.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.150+0000 7f6884c04740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f687c1aed00 con 0x559f0fec42d0 2024-03-19T02:59:14.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.150+0000 7f6884c04740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f687c1acb30 con 0x559f0fecd7d0 2024-03-19T02:59:14.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.150+0000 7f687affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559f0fec4bf0 0x559f0fed9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.150+0000 7f687b7fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559f0fec42d0 0x559f0fecd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.150+0000 7f687b7fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559f0fec42d0 0x559f0fecd000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:57912/0 (socket says 172.21.15.33:57912) 2024-03-19T02:59:14.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.150+0000 7f687b7fe640 1 -- 172.21.15.33:0/1667112036 learned_addr learned my addr 172.21.15.33:0/1667112036 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:14.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.150+0000 7f687bfff640 1 --2- 172.21.15.33:0/1667112036 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559f0fecd7d0 0x559f0fee5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.150+0000 7f687b7fe640 1 -- 172.21.15.33:0/1667112036 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559f0fecd7d0 msgr2=0x559f0fee5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:14.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.150+0000 7f687b7fe640 1 --2- 172.21.15.33:0/1667112036 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559f0fecd7d0 0x559f0fee5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.150+0000 7f687b7fe640 1 -- 172.21.15.33:0/1667112036 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559f0fec4bf0 msgr2=0x559f0fed9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:14.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.150+0000 7f687b7fe640 1 --2- 172.21.15.33:0/1667112036 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559f0fec4bf0 0x559f0fed9360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.150+0000 7f687b7fe640 1 -- 172.21.15.33:0/1667112036 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f0feba0a0 con 0x559f0fec42d0 2024-03-19T02:59:14.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.150+0000 7f687bfff640 1 --2- 172.21.15.33:0/1667112036 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559f0fecd7d0 0x559f0fee5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:59:14.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.151+0000 7f687b7fe640 1 --2- 172.21.15.33:0/1667112036 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559f0fec42d0 0x559f0fecd000 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f686c09b570 tx=0x7f686c009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55f5d607a5971370 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:14.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.151+0000 7f68617fa640 1 -- 172.21.15.33:0/1667112036 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f686c097ee0 con 0x559f0fec42d0 2024-03-19T02:59:14.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.151+0000 7f68617fa640 1 -- 172.21.15.33:0/1667112036 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f686c0860b0 con 0x559f0fec42d0 2024-03-19T02:59:14.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.151+0000 7f68617fa640 1 -- 172.21.15.33:0/1667112036 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f686c0afe00 con 0x559f0fec42d0 2024-03-19T02:59:14.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.151+0000 7f6884c04740 1 -- 172.21.15.33:0/1667112036 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559f0fec42d0 msgr2=0x559f0fecd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:14.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.151+0000 7f6884c04740 1 --2- 172.21.15.33:0/1667112036 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559f0fec42d0 0x559f0fecd000 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f686c09b570 tx=0x7f686c009310 comp rx=0 tx=0).stop 2024-03-19T02:59:14.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.151+0000 7f6884c04740 1 -- 172.21.15.33:0/1667112036 shutdown_connections 2024-03-19T02:59:14.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.151+0000 7f6884c04740 1 --2- 172.21.15.33:0/1667112036 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559f0fecd7d0 0x559f0fee5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.151+0000 7f6884c04740 1 --2- 172.21.15.33:0/1667112036 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559f0fec4bf0 0x559f0fed9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.151+0000 7f6884c04740 1 --2- 172.21.15.33:0/1667112036 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559f0fec42d0 0x559f0fecd000 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.151+0000 7f6884c04740 1 -- 172.21.15.33:0/1667112036 >> 172.21.15.33:0/1667112036 conn(0x559f0febea00 msgr2=0x559f0fec91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:14.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.151+0000 7f6884c04740 1 -- 172.21.15.33:0/1667112036 shutdown_connections 2024-03-19T02:59:14.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.151+0000 7f6884c04740 1 -- 172.21.15.33:0/1667112036 wait complete. 2024-03-19T02:59:14.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.152+0000 7f6884c04740 1 -- start start 2024-03-19T02:59:14.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.152+0000 7f6884c04740 1 Processor -- start 2024-03-19T02:59:14.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.152+0000 7f6884c04740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559f0fec42d0 0x559f0fec8020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.152+0000 7f6884c04740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559f0fec4bf0 0x559f0fec8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.152+0000 7f6884c04740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559f0fecd7d0 0x559f0fec6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.152+0000 7f6884c04740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f687c1ad1c0 con 0x559f0fecd7d0 2024-03-19T02:59:14.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.152+0000 7f6884c04740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f687c1ae630 con 0x559f0fec42d0 2024-03-19T02:59:14.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.152+0000 7f6884c04740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f687c1ad890 con 0x559f0fec4bf0 2024-03-19T02:59:14.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.152+0000 7f6884c04740 10 client.0 _ll_register_callbacks cb 0x7f6870005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:59:14.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.152+0000 7f687b7fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559f0fec42d0 0x559f0fec8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.152+0000 7f687bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559f0fecd7d0 0x559f0fec6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.152+0000 7f687b7fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559f0fec42d0 0x559f0fec8020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:57928/0 (socket says 172.21.15.33:57928) 2024-03-19T02:59:14.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.152+0000 7f687b7fe640 1 -- 172.21.15.33:0/1161669710 learned_addr learned my addr 172.21.15.33:0/1161669710 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:14.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.152+0000 7f687affd640 1 --2- 172.21.15.33:0/1161669710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559f0fec4bf0 0x559f0fec8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.153+0000 7f687b7fe640 1 -- 172.21.15.33:0/1161669710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559f0fec4bf0 msgr2=0x559f0fec8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:14.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.153+0000 7f687b7fe640 1 --2- 172.21.15.33:0/1161669710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559f0fec4bf0 0x559f0fec8560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.153+0000 7f687b7fe640 1 -- 172.21.15.33:0/1161669710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559f0fecd7d0 msgr2=0x559f0fec6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:14.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.153+0000 7f687b7fe640 1 --2- 172.21.15.33:0/1161669710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559f0fecd7d0 0x559f0fec6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.153+0000 7f687b7fe640 1 -- 172.21.15.33:0/1161669710 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f0fec70a0 con 0x559f0fec42d0 2024-03-19T02:59:14.160 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.153+0000 7f687affd640 1 --2- 172.21.15.33:0/1161669710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559f0fec4bf0 0x559f0fec8560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:59:14.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.153+0000 7f687bfff640 1 --2- 172.21.15.33:0/1161669710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559f0fecd7d0 0x559f0fec6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:14.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.153+0000 7f687b7fe640 1 --2- 172.21.15.33:0/1161669710 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559f0fec42d0 0x559f0fec8020 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f686c00c6a0 tx=0x7f686c097c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=615eafae19467e6b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:14.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.153+0000 7f685affd640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T02:59:14.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.153+0000 7f685affd640 1 -- 172.21.15.33:0/1161669710 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f686c0ae070 con 0x559f0fec42d0 2024-03-19T02:59:14.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.153+0000 7f6884c04740 1 -- 172.21.15.33:0/1161669710 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f686c0860b0 con 0x559f0fec42d0 2024-03-19T02:59:14.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.153+0000 7f685affd640 1 -- 172.21.15.33:0/1161669710 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f686c0aaba0 con 0x559f0fec42d0 2024-03-19T02:59:14.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.153+0000 7f685affd640 1 -- 172.21.15.33:0/1161669710 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f686c0aada0 con 0x559f0fec42d0 2024-03-19T02:59:14.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.153+0000 7f6884c04740 10 client.26720 fetch_fsmap learned FSMap version 25 2024-03-19T02:59:14.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.153+0000 7f6884c04740 1 -- 172.21.15.33:0/1161669710 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559f0ff43690 con 0x559f0fec42d0 2024-03-19T02:59:14.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.153+0000 7f685affd640 1 -- 172.21.15.33:0/1161669710 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f686c0860b0 con 0x559f0fec42d0 2024-03-19T02:59:14.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.153+0000 7f685affd640 1 -- 172.21.15.33:0/1161669710 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f686c09acc0 con 0x559f0fec42d0 2024-03-19T02:59:14.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.153+0000 7f6884c04740 10 client.26720 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:59:14.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.153+0000 7f6884c04740 10 client.26720 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T02:59:14.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.153+0000 7f6884c04740 20 client.26720 populate_metadata read hostname 'smithi033' 2024-03-19T02:59:14.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.153+0000 7f6884c04740 1 --2- 172.21.15.33:0/1161669710 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x559f0ff43cb0 0x559f0ff460a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.153+0000 7f6884c04740 4 client.26720 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:59:14.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.153+0000 7f6884c04740 1 -- 172.21.15.33:0/1161669710 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559f0fe349f0 con 0x559f0ff43cb0 2024-03-19T02:59:14.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.154+0000 7f687affd640 1 --2- 172.21.15.33:0/1161669710 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x559f0ff43cb0 0x559f0ff460a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.154+0000 7f687affd640 1 --2- 172.21.15.33:0/1161669710 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x559f0ff43cb0 0x559f0ff460a0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1083d6edbd17fbbe server_cookie=6ae2e7948219a44f in_seq=0 out_seq=0 2024-03-19T02:59:14.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.154+0000 7f685affd640 10 client.26720 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T02:59:14.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.154+0000 7f685affd640 1 -- 172.21.15.33:0/1161669710 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559f0fe349f0 con 0x559f0ff43cb0 2024-03-19T02:59:14.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.154+0000 7f685affd640 10 client.26720 handle_command_reply: tid=0 2024-03-19T02:59:14.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.154+0000 7f6884c04740 1 client.26720 shutdown 2024-03-19T02:59:14.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.154+0000 7f6884c04740 1 -- 172.21.15.33:0/1161669710 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559f0fec42d0 msgr2=0x559f0fec8020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:14.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.154+0000 7f6884c04740 1 --2- 172.21.15.33:0/1161669710 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559f0fec42d0 0x559f0fec8020 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f686c00c6a0 tx=0x7f686c097c40 comp rx=0 tx=0).stop 2024-03-19T02:59:14.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.154+0000 7f6884c04740 1 -- 172.21.15.33:0/1161669710 shutdown_connections 2024-03-19T02:59:14.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.154+0000 7f6884c04740 1 --2- 172.21.15.33:0/1161669710 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x559f0ff43cb0 0x559f0ff460a0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.154+0000 7f6884c04740 1 --2- 172.21.15.33:0/1161669710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x559f0fec4bf0 0x559f0fec8560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.154+0000 7f6884c04740 1 --2- 172.21.15.33:0/1161669710 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x559f0fecd7d0 0x559f0fec6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.154+0000 7f685affd640 0 client.26720 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T02:59:14.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.154+0000 7f6884c04740 1 --2- 172.21.15.33:0/1161669710 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559f0fec42d0 0x559f0fec8020 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.154+0000 7f6884c04740 1 -- 172.21.15.33:0/1161669710 >> 172.21.15.33:0/1161669710 conn(0x559f0febea00 msgr2=0x559f0feca890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:14.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.154+0000 7f6884c04740 1 -- 172.21.15.33:0/1161669710 shutdown_connections 2024-03-19T02:59:14.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.154+0000 7f6884c04740 1 -- 172.21.15.33:0/1161669710 wait complete. 2024-03-19T02:59:14.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.154+0000 7f6884c04740 20 client.26720 trim_cache size 0 max 16384 2024-03-19T02:59:14.163 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:59:14.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.155+0000 7f6882012640 1 -- 172.21.15.33:0/2470734497 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f68640a2970 msgr2=0x7f68640a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:14.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.155+0000 7f6882012640 1 --2- 172.21.15.33:0/2470734497 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f68640a2970 0x7f68640a4e30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6874004af0 tx=0x7f6874003b30 comp rx=0 tx=0).stop 2024-03-19T02:59:14.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.155+0000 7f6882012640 1 -- 172.21.15.33:0/2470734497 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f687c102a40 msgr2=0x7f687c197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:14.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.155+0000 7f6882012640 1 --2- 172.21.15.33:0/2470734497 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f687c102a40 0x7f687c197f30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f686c00e9f0 tx=0x7f686c00eeb0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.155+0000 7f6882012640 1 -- 172.21.15.33:0/2470734497 shutdown_connections 2024-03-19T02:59:14.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.155+0000 7f6882012640 1 --2- 172.21.15.33:0/2470734497 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f68640a2970 0x7f68640a4e30 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.155+0000 7f6882012640 1 --2- 172.21.15.33:0/2470734497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f687c10bd80 0x7f687c19c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.155+0000 7f6882012640 1 --2- 172.21.15.33:0/2470734497 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f687c103360 0x7f687c198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.155+0000 7f6882012640 1 --2- 172.21.15.33:0/2470734497 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f687c102a40 0x7f687c197f30 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.155+0000 7f6882012640 1 -- 172.21.15.33:0/2470734497 >> 172.21.15.33:0/2470734497 conn(0x7f687c0778d0 msgr2=0x7f687c0fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:14.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.155+0000 7f6882012640 1 -- 172.21.15.33:0/2470734497 shutdown_connections 2024-03-19T02:59:14.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.155+0000 7f6882012640 1 -- 172.21.15.33:0/2470734497 wait complete. 2024-03-19T02:59:14.167 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:59:14.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.272+0000 7f035c6f1640 1 -- 172.21.15.33:0/2442735292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0354101060 msgr2=0x7f035410d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:14.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.272+0000 7f035c6f1640 1 --2- 172.21.15.33:0/2442735292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0354101060 0x7f035410d570 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f034c009cd0 tx=0x7f034c02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.272+0000 7f035c6f1640 1 -- 172.21.15.33:0/2442735292 shutdown_connections 2024-03-19T02:59:14.276 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.272+0000 7f035c6f1640 1 --2- 172.21.15.33:0/2442735292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0354101060 0x7f035410d570 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.272+0000 7f035c6f1640 1 --2- 172.21.15.33:0/2442735292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03541006a0 0x7f0354100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.272+0000 7f035c6f1640 1 --2- 172.21.15.33:0/2442735292 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f03541066b0 0x7f0354106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.272+0000 7f035c6f1640 1 -- 172.21.15.33:0/2442735292 >> 172.21.15.33:0/2442735292 conn(0x7f03540778d0 msgr2=0x7f0354077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:14.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.272+0000 7f035c6f1640 1 -- 172.21.15.33:0/2442735292 shutdown_connections 2024-03-19T02:59:14.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.272+0000 7f035c6f1640 1 -- 172.21.15.33:0/2442735292 wait complete. 2024-03-19T02:59:14.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.272+0000 7f035c6f1640 1 Processor -- start 2024-03-19T02:59:14.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.273+0000 7f035c6f1640 1 -- start start 2024-03-19T02:59:14.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.273+0000 7f035c6f1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03541006a0 0x7f0354197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.273+0000 7f035c6f1640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0354101060 0x7f0354198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.273+0000 7f035c6f1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f03541066b0 0x7f035419c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.273+0000 7f035c6f1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0354111a70 con 0x7f03541066b0 2024-03-19T02:59:14.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.273+0000 7f035c6f1640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f03541118f0 con 0x7f03541006a0 2024-03-19T02:59:14.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.273+0000 7f035c6f1640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f0354111bf0 con 0x7f0354101060 2024-03-19T02:59:14.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.273+0000 7f035ac8e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f03541066b0 0x7f035419c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.273+0000 7f035ac8e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f03541066b0 0x7f035419c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51450/0 (socket says 172.21.15.33:51450) 2024-03-19T02:59:14.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.273+0000 7f035a48d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03541006a0 0x7f0354197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.273+0000 7f035ac8e640 1 -- 172.21.15.33:0/2275476501 learned_addr learned my addr 172.21.15.33:0/2275476501 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:14.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.273+0000 7f0359c8c640 1 --2- 172.21.15.33:0/2275476501 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0354101060 0x7f0354198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.273+0000 7f035ac8e640 1 -- 172.21.15.33:0/2275476501 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0354101060 msgr2=0x7f0354198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:14.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.273+0000 7f035ac8e640 1 --2- 172.21.15.33:0/2275476501 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0354101060 0x7f0354198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.273+0000 7f035ac8e640 1 -- 172.21.15.33:0/2275476501 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03541006a0 msgr2=0x7f0354197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:14.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.273+0000 7f035ac8e640 1 --2- 172.21.15.33:0/2275476501 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03541006a0 0x7f0354197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.273+0000 7f035ac8e640 1 -- 172.21.15.33:0/2275476501 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f035419cf20 con 0x7f03541066b0 2024-03-19T02:59:14.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.273+0000 7f035a48d640 1 --2- 172.21.15.33:0/2275476501 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03541006a0 0x7f0354197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:14.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.273+0000 7f035ac8e640 1 --2- 172.21.15.33:0/2275476501 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f03541066b0 0x7f035419c7f0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f034c004af0 tx=0x7f034c035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:14.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.273+0000 7f034b7fe640 1 -- 172.21.15.33:0/2275476501 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034c039040 con 0x7f03541066b0 2024-03-19T02:59:14.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.274+0000 7f035c6f1640 1 -- 172.21.15.33:0/2275476501 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f035419d1b0 con 0x7f03541066b0 2024-03-19T02:59:14.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.274+0000 7f034b7fe640 1 -- 172.21.15.33:0/2275476501 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f034c044070 con 0x7f03541066b0 2024-03-19T02:59:14.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.274+0000 7f034b7fe640 1 -- 172.21.15.33:0/2275476501 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034c0042a0 con 0x7f03541066b0 2024-03-19T02:59:14.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.274+0000 7f035c6f1640 1 -- 172.21.15.33:0/2275476501 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03541a4aa0 con 0x7f03541066b0 2024-03-19T02:59:14.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.276+0000 7f034b7fe640 1 -- 172.21.15.33:0/2275476501 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f034c003970 con 0x7f03541066b0 2024-03-19T02:59:14.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.276+0000 7f035f3bc740 1 Processor -- start 2024-03-19T02:59:14.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.276+0000 7f035f3bc740 1 -- start start 2024-03-19T02:59:14.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.276+0000 7f035f3bc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e876f0a2d0 0x55e876f0a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.276+0000 7f035f3bc740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e876f0abf0 0x55e876f13290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.276+0000 7f035f3bc740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e876f137d0 0x55e876f15bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.276+0000 7f035f3bc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e876e6d4b0 con 0x55e876f0a2d0 2024-03-19T02:59:14.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.276+0000 7f035f3bc740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55e876e15d70 con 0x55e876f137d0 2024-03-19T02:59:14.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.276+0000 7f035f3bc740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e876e69550 con 0x55e876f0abf0 2024-03-19T02:59:14.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.276+0000 7f034b7fe640 1 --2- 172.21.15.33:0/2275476501 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f03400a2970 0x7f03400a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.276+0000 7f0359c8c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e876f0a2d0 0x55e876f0a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.276+0000 7f0359c8c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e876f0a2d0 0x55e876f0a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51462/0 (socket says 172.21.15.33:51462) 2024-03-19T02:59:14.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.276+0000 7f0359c8c640 1 -- 172.21.15.33:0/1486973245 learned_addr learned my addr 172.21.15.33:0/1486973245 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:14.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.277+0000 7f035a48d640 1 --2- 172.21.15.33:0/1486973245 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e876f0abf0 0x55e876f13290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.277+0000 7f0359c8c640 1 --2- 172.21.15.33:0/1486973245 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e876f137d0 0x55e876f15bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.277+0000 7f035ac8e640 1 --2- 172.21.15.33:0/2275476501 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f03400a2970 0x7f03400a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.277+0000 7f034b7fe640 1 -- 172.21.15.33:0/2275476501 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f034c0f14b0 con 0x7f03541066b0 2024-03-19T02:59:14.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.277+0000 7f0359c8c640 1 -- 172.21.15.33:0/1486973245 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e876f0abf0 msgr2=0x55e876f13290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:14.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.277+0000 7f0359c8c640 1 --2- 172.21.15.33:0/1486973245 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e876f0abf0 0x55e876f13290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.277+0000 7f0359c8c640 1 -- 172.21.15.33:0/1486973245 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e876f137d0 msgr2=0x55e876f15bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:14.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.277+0000 7f0359c8c640 1 --2- 172.21.15.33:0/1486973245 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e876f137d0 0x55e876f15bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.277+0000 7f0359c8c640 1 -- 172.21.15.33:0/1486973245 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e876e38210 con 0x55e876f0a2d0 2024-03-19T02:59:14.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.277+0000 7f0359c8c640 1 --2- 172.21.15.33:0/1486973245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e876f0a2d0 0x55e876f0a6b0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f03440095f0 tx=0x7f0344005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ca73a8b7b4cffed server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:14.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.277+0000 7f035ac8e640 1 --2- 172.21.15.33:0/2275476501 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f03400a2970 0x7f03400a4e30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f034c0bd5b0 tx=0x7f034c08c6c0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:14.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.277+0000 7f0348ff9640 1 -- 172.21.15.33:0/1486973245 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0344013dd0 con 0x55e876f0a2d0 2024-03-19T02:59:14.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.277+0000 7f0348ff9640 1 -- 172.21.15.33:0/1486973245 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0344013170 con 0x55e876f0a2d0 2024-03-19T02:59:14.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.277+0000 7f0348ff9640 1 -- 172.21.15.33:0/1486973245 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0344010950 con 0x55e876f0a2d0 2024-03-19T02:59:14.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.278+0000 7f035f3bc740 1 -- 172.21.15.33:0/1486973245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e876f0a2d0 msgr2=0x55e876f0a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:14.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.278+0000 7f035f3bc740 1 --2- 172.21.15.33:0/1486973245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e876f0a2d0 0x55e876f0a6b0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f03440095f0 tx=0x7f0344005f90 comp rx=0 tx=0).stop 2024-03-19T02:59:14.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.278+0000 7f035f3bc740 1 -- 172.21.15.33:0/1486973245 shutdown_connections 2024-03-19T02:59:14.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.278+0000 7f035f3bc740 1 --2- 172.21.15.33:0/1486973245 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e876f0abf0 0x55e876f13290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.278+0000 7f035f3bc740 1 --2- 172.21.15.33:0/1486973245 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e876f137d0 0x55e876f15bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.278+0000 7f035f3bc740 1 --2- 172.21.15.33:0/1486973245 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e876f0a2d0 0x55e876f0a6b0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.278+0000 7f035f3bc740 1 -- 172.21.15.33:0/1486973245 >> 172.21.15.33:0/1486973245 conn(0x55e876f04a00 msgr2=0x55e876f04de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:14.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.278+0000 7f035f3bc740 1 -- 172.21.15.33:0/1486973245 shutdown_connections 2024-03-19T02:59:14.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.278+0000 7f035f3bc740 1 -- 172.21.15.33:0/1486973245 wait complete. 2024-03-19T02:59:14.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.278+0000 7f035f3bc740 1 -- start start 2024-03-19T02:59:14.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.278+0000 7f035f3bc740 1 Processor -- start 2024-03-19T02:59:14.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.279+0000 7f035f3bc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e876f0a2d0 0x55e876f107d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.279+0000 7f035f3bc740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e876f0abf0 0x55e876f10d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.279+0000 7f035f3bc740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e876f137d0 0x55e876f11250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.279+0000 7f035f3bc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e876f16560 con 0x55e876f0a2d0 2024-03-19T02:59:14.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.279+0000 7f035f3bc740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55e876e29310 con 0x55e876f0abf0 2024-03-19T02:59:14.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.279+0000 7f035f3bc740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e876f166e0 con 0x55e876f137d0 2024-03-19T02:59:14.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.279+0000 7f035f3bc740 10 client.0 _ll_register_callbacks cb 0x55e876e1d2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:59:14.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.279+0000 7f0359c8c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e876f0a2d0 0x55e876f107d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.279+0000 7f0359c8c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e876f0a2d0 0x55e876f107d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51472/0 (socket says 172.21.15.33:51472) 2024-03-19T02:59:14.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.279+0000 7f0359c8c640 1 -- 172.21.15.33:0/2422391106 learned_addr learned my addr 172.21.15.33:0/2422391106 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:14.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.279+0000 7f035a48d640 1 --2- 172.21.15.33:0/2422391106 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e876f0abf0 0x55e876f10d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.279+0000 7f0359c8c640 1 --2- 172.21.15.33:0/2422391106 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e876f137d0 0x55e876f11250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.279+0000 7f0359c8c640 1 -- 172.21.15.33:0/2422391106 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e876f137d0 msgr2=0x55e876f11250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:14.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.279+0000 7f0359c8c640 1 --2- 172.21.15.33:0/2422391106 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e876f137d0 0x55e876f11250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.279+0000 7f0359c8c640 1 -- 172.21.15.33:0/2422391106 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e876f0abf0 msgr2=0x55e876f10d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:14.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.279+0000 7f0359c8c640 1 --2- 172.21.15.33:0/2422391106 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e876f0abf0 0x55e876f10d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.279+0000 7f0359c8c640 1 -- 172.21.15.33:0/2422391106 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e876b813f0 con 0x55e876f0a2d0 2024-03-19T02:59:14.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.279+0000 7f035a48d640 1 --2- 172.21.15.33:0/2422391106 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e876f0abf0 0x55e876f10d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:14.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.279+0000 7f0359c8c640 1 --2- 172.21.15.33:0/2422391106 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e876f0a2d0 0x55e876f107d0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f0344009c00 tx=0x7f03440105d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51935bb97c89ca65 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:14.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.279+0000 7f033dffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:59:14.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.279+0000 7f033dffb640 1 -- 172.21.15.33:0/2422391106 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034402e060 con 0x55e876f0a2d0 2024-03-19T02:59:14.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.279+0000 7f033dffb640 1 -- 172.21.15.33:0/2422391106 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f034400fc20 con 0x55e876f0a2d0 2024-03-19T02:59:14.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.279+0000 7f035f3bc740 1 -- 172.21.15.33:0/2422391106 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0344013170 con 0x55e876f0a2d0 2024-03-19T02:59:14.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.280+0000 7f033dffb640 1 -- 172.21.15.33:0/2422391106 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f034400fe20 con 0x55e876f0a2d0 2024-03-19T02:59:14.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.280+0000 7f035f3bc740 10 client.16908 fetch_fsmap learned FSMap version 25 2024-03-19T02:59:14.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.280+0000 7f035f3bc740 1 -- 172.21.15.33:0/2422391106 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55e876c4b7d0 con 0x55e876f0a2d0 2024-03-19T02:59:14.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.280+0000 7f033dffb640 1 -- 172.21.15.33:0/2422391106 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0344013170 con 0x55e876f0a2d0 2024-03-19T02:59:14.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.280+0000 7f033dffb640 1 -- 172.21.15.33:0/2422391106 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f03440048f0 con 0x55e876f0a2d0 2024-03-19T02:59:14.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.280+0000 7f035f3bc740 10 client.16908 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:59:14.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.280+0000 7f035f3bc740 10 client.16908 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:59:14.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.280+0000 7f035f3bc740 20 client.16908 populate_metadata read hostname 'smithi033' 2024-03-19T02:59:14.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.280+0000 7f035f3bc740 1 --2- 172.21.15.33:0/2422391106 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55e876f23a80 0x55e876f25e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.280+0000 7f035f3bc740 4 client.16908 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:59:14.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.280+0000 7f035f3bc740 1 -- 172.21.15.33:0/2422391106 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e876ecb3e0 con 0x55e876f23a80 2024-03-19T02:59:14.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.280+0000 7f035a48d640 1 --2- 172.21.15.33:0/2422391106 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55e876f23a80 0x55e876f25e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.281+0000 7f033dffb640 10 client.16908 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:59:14.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.281+0000 7f035a48d640 1 --2- 172.21.15.33:0/2422391106 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55e876f23a80 0x55e876f25e70 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a00f76222e05d4e server_cookie=47eba9d4e20866f1 in_seq=0 out_seq=0 2024-03-19T02:59:14.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.281+0000 7f033dffb640 1 -- 172.21.15.33:0/2422391106 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e876ecb3e0 con 0x55e876f23a80 2024-03-19T02:59:14.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.281+0000 7f033dffb640 10 client.16908 handle_command_reply: tid=0 2024-03-19T02:59:14.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.281+0000 7f035f3bc740 1 client.16908 shutdown 2024-03-19T02:59:14.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.281+0000 7f035f3bc740 1 -- 172.21.15.33:0/2422391106 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e876f0a2d0 msgr2=0x55e876f107d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:14.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.281+0000 7f035f3bc740 1 --2- 172.21.15.33:0/2422391106 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e876f0a2d0 0x55e876f107d0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f0344009c00 tx=0x7f03440105d0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.281+0000 7f035f3bc740 1 -- 172.21.15.33:0/2422391106 shutdown_connections 2024-03-19T02:59:14.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.281+0000 7f035f3bc740 1 --2- 172.21.15.33:0/2422391106 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55e876f23a80 0x55e876f25e70 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.281+0000 7f035f3bc740 1 --2- 172.21.15.33:0/2422391106 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e876f137d0 0x55e876f11250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.281+0000 7f035f3bc740 1 --2- 172.21.15.33:0/2422391106 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e876f0abf0 0x55e876f10d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.281+0000 7f033dffb640 0 client.16908 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:59:14.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.281+0000 7f035f3bc740 1 --2- 172.21.15.33:0/2422391106 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e876f0a2d0 0x55e876f107d0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.281+0000 7f035f3bc740 1 -- 172.21.15.33:0/2422391106 >> 172.21.15.33:0/2422391106 conn(0x55e876f04a00 msgr2=0x55e876f06750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:14.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.281+0000 7f035f3bc740 1 -- 172.21.15.33:0/2422391106 shutdown_connections 2024-03-19T02:59:14.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.281+0000 7f035f3bc740 1 -- 172.21.15.33:0/2422391106 wait complete. 2024-03-19T02:59:14.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.282+0000 7f035f3bc740 20 client.16908 trim_cache size 0 max 16384 2024-03-19T02:59:14.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.295+0000 7f035f3bc740 1 Processor -- start 2024-03-19T02:59:14.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.295+0000 7f035f3bc740 1 -- start start 2024-03-19T02:59:14.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.295+0000 7f035f3bc740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e876f0a2d0 0x55e876effa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.295+0000 7f035f3bc740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e876f0abf0 0x55e876f1f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.295+0000 7f035f3bc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e876f137d0 0x55e876f11070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.295+0000 7f035f3bc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f03541b1460 con 0x55e876f137d0 2024-03-19T02:59:14.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.295+0000 7f035f3bc740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f03541aed30 con 0x55e876f0abf0 2024-03-19T02:59:14.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.295+0000 7f035f3bc740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f03541acb60 con 0x55e876f0a2d0 2024-03-19T02:59:14.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.295+0000 7f0359c8c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e876f137d0 0x55e876f11070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.295+0000 7f0359c8c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e876f137d0 0x55e876f11070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51484/0 (socket says 172.21.15.33:51484) 2024-03-19T02:59:14.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.295+0000 7f0359c8c640 1 -- 172.21.15.33:0/1283939578 learned_addr learned my addr 172.21.15.33:0/1283939578 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:14.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.295+0000 7f035a48d640 1 --2- 172.21.15.33:0/1283939578 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e876f0abf0 0x55e876f1f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.295+0000 7f0359c8c640 1 --2- 172.21.15.33:0/1283939578 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e876f0a2d0 0x55e876effa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.295+0000 7f0359c8c640 1 -- 172.21.15.33:0/1283939578 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e876f0a2d0 msgr2=0x55e876effa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:14.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.295+0000 7f0359c8c640 1 --2- 172.21.15.33:0/1283939578 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e876f0a2d0 0x55e876effa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.295+0000 7f0359c8c640 1 -- 172.21.15.33:0/1283939578 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e876f0abf0 msgr2=0x55e876f1f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:14.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.295+0000 7f0359c8c640 1 --2- 172.21.15.33:0/1283939578 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e876f0abf0 0x55e876f1f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.295+0000 7f0359c8c640 1 -- 172.21.15.33:0/1283939578 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e876f0f780 con 0x55e876f137d0 2024-03-19T02:59:14.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.295+0000 7f0359c8c640 1 --2- 172.21.15.33:0/1283939578 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e876f137d0 0x55e876f11070 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f0344004fb0 tx=0x7f0344005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeccec0e640a682f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:14.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.295+0000 7f033ffff640 1 -- 172.21.15.33:0/1283939578 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034402e070 con 0x55e876f137d0 2024-03-19T02:59:14.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.295+0000 7f033ffff640 1 -- 172.21.15.33:0/1283939578 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f034400fa20 con 0x55e876f137d0 2024-03-19T02:59:14.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.295+0000 7f033ffff640 1 -- 172.21.15.33:0/1283939578 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0344010de0 con 0x55e876f137d0 2024-03-19T02:59:14.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.296+0000 7f035f3bc740 1 -- 172.21.15.33:0/1283939578 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e876f137d0 msgr2=0x55e876f11070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:14.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.296+0000 7f035f3bc740 1 --2- 172.21.15.33:0/1283939578 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e876f137d0 0x55e876f11070 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f0344004fb0 tx=0x7f0344005970 comp rx=0 tx=0).stop 2024-03-19T02:59:14.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.296+0000 7f035f3bc740 1 -- 172.21.15.33:0/1283939578 shutdown_connections 2024-03-19T02:59:14.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.296+0000 7f035f3bc740 1 --2- 172.21.15.33:0/1283939578 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e876f137d0 0x55e876f11070 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.296+0000 7f035f3bc740 1 --2- 172.21.15.33:0/1283939578 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e876f0abf0 0x55e876f1f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.296+0000 7f035f3bc740 1 --2- 172.21.15.33:0/1283939578 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e876f0a2d0 0x55e876effa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.296+0000 7f035f3bc740 1 -- 172.21.15.33:0/1283939578 >> 172.21.15.33:0/1283939578 conn(0x55e876f04a00 msgr2=0x55e876f03ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:14.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.296+0000 7f035f3bc740 1 -- 172.21.15.33:0/1283939578 shutdown_connections 2024-03-19T02:59:14.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.296+0000 7f035f3bc740 1 -- 172.21.15.33:0/1283939578 wait complete. 2024-03-19T02:59:14.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.296+0000 7f035f3bc740 1 -- start start 2024-03-19T02:59:14.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.296+0000 7f035f3bc740 1 Processor -- start 2024-03-19T02:59:14.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.297+0000 7f035f3bc740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e876f0a2d0 0x55e876f1b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.297+0000 7f035f3bc740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e876f0abf0 0x55e876f16f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.297+0000 7f035f3bc740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e876f137d0 0x55e876f17480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.297+0000 7f035f3bc740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f03541ad1f0 con 0x55e876f0a2d0 2024-03-19T02:59:14.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.297+0000 7f035f3bc740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f03541ae660 con 0x55e876f137d0 2024-03-19T02:59:14.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.297+0000 7f035f3bc740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f03541ad8c0 con 0x55e876f0abf0 2024-03-19T02:59:14.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.297+0000 7f035f3bc740 10 client.0 _ll_register_callbacks cb 0x7f0344009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:59:14.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.297+0000 7f0359c8c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e876f0a2d0 0x55e876f1b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.297+0000 7f0359c8c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e876f0a2d0 0x55e876f1b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51490/0 (socket says 172.21.15.33:51490) 2024-03-19T02:59:14.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.297+0000 7f0359c8c640 1 -- 172.21.15.33:0/2629395614 learned_addr learned my addr 172.21.15.33:0/2629395614 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:14.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.297+0000 7f035a48d640 1 --2- 172.21.15.33:0/2629395614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e876f0abf0 0x55e876f16f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.297+0000 7f0359c8c640 1 --2- 172.21.15.33:0/2629395614 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e876f137d0 0x55e876f17480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.297+0000 7f0359c8c640 1 -- 172.21.15.33:0/2629395614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e876f0abf0 msgr2=0x55e876f16f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:14.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.297+0000 7f0359c8c640 1 --2- 172.21.15.33:0/2629395614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e876f0abf0 0x55e876f16f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.297+0000 7f0359c8c640 1 -- 172.21.15.33:0/2629395614 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e876f137d0 msgr2=0x55e876f17480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:14.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.297+0000 7f0359c8c640 1 --2- 172.21.15.33:0/2629395614 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e876f137d0 0x55e876f17480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.297+0000 7f0359c8c640 1 -- 172.21.15.33:0/2629395614 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e876f17a70 con 0x55e876f0a2d0 2024-03-19T02:59:14.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.297+0000 7f0359c8c640 1 --2- 172.21.15.33:0/2629395614 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e876f0a2d0 0x55e876f1b9e0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f0344009da0 tx=0x7f0344009850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=800fcb7499afbc14 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:14.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.297+0000 7f033dffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:59:14.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.297+0000 7f033dffb640 1 -- 172.21.15.33:0/2629395614 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0344045060 con 0x55e876f0a2d0 2024-03-19T02:59:14.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.297+0000 7f033dffb640 1 -- 172.21.15.33:0/2629395614 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f0344003a00 con 0x55e876f0a2d0 2024-03-19T02:59:14.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.297+0000 7f035f3bc740 1 -- 172.21.15.33:0/2629395614 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f034400fa20 con 0x55e876f0a2d0 2024-03-19T02:59:14.305 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.298+0000 7f033dffb640 1 -- 172.21.15.33:0/2629395614 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f034402e040 con 0x55e876f0a2d0 2024-03-19T02:59:14.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.298+0000 7f035f3bc740 10 client.16914 fetch_fsmap learned FSMap version 25 2024-03-19T02:59:14.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.298+0000 7f035f3bc740 1 -- 172.21.15.33:0/2629395614 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55e876f89990 con 0x55e876f0a2d0 2024-03-19T02:59:14.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.298+0000 7f033dffb640 1 -- 172.21.15.33:0/2629395614 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f034400fa20 con 0x55e876f0a2d0 2024-03-19T02:59:14.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.298+0000 7f033dffb640 1 -- 172.21.15.33:0/2629395614 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f0344026070 con 0x55e876f0a2d0 2024-03-19T02:59:14.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.298+0000 7f035f3bc740 10 client.16914 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:59:14.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.298+0000 7f035f3bc740 10 client.16914 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:59:14.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.298+0000 7f035f3bc740 20 client.16914 populate_metadata read hostname 'smithi033' 2024-03-19T02:59:14.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.298+0000 7f035f3bc740 1 --2- 172.21.15.33:0/2629395614 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55e876f89e40 0x55e876f8c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:14.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.298+0000 7f035f3bc740 4 client.16914 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:59:14.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.298+0000 7f035f3bc740 1 -- 172.21.15.33:0/2629395614 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e876e7a9f0 con 0x55e876f89e40 2024-03-19T02:59:14.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.298+0000 7f035a48d640 1 --2- 172.21.15.33:0/2629395614 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55e876f89e40 0x55e876f8c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:14.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.298+0000 7f033dffb640 10 client.16914 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:59:14.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.298+0000 7f035a48d640 1 --2- 172.21.15.33:0/2629395614 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55e876f89e40 0x55e876f8c260 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b13bdfccdee7418 server_cookie=be85eab599d6df1 in_seq=0 out_seq=0 2024-03-19T02:59:14.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.299+0000 7f033dffb640 1 -- 172.21.15.33:0/2629395614 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e876e7a9f0 con 0x55e876f89e40 2024-03-19T02:59:14.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.299+0000 7f033dffb640 10 client.16914 handle_command_reply: tid=0 2024-03-19T02:59:14.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.299+0000 7f035f3bc740 1 client.16914 shutdown 2024-03-19T02:59:14.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.299+0000 7f035f3bc740 1 -- 172.21.15.33:0/2629395614 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e876f0a2d0 msgr2=0x55e876f1b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:14.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.299+0000 7f035f3bc740 1 --2- 172.21.15.33:0/2629395614 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e876f0a2d0 0x55e876f1b9e0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f0344009da0 tx=0x7f0344009850 comp rx=0 tx=0).stop 2024-03-19T02:59:14.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.299+0000 7f035f3bc740 1 -- 172.21.15.33:0/2629395614 shutdown_connections 2024-03-19T02:59:14.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.299+0000 7f035f3bc740 1 --2- 172.21.15.33:0/2629395614 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55e876f89e40 0x55e876f8c260 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.299+0000 7f035f3bc740 1 --2- 172.21.15.33:0/2629395614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e876f0abf0 0x55e876f16f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.299+0000 7f035f3bc740 1 --2- 172.21.15.33:0/2629395614 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e876f137d0 0x55e876f17480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.299+0000 7f033dffb640 0 client.16914 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:59:14.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.299+0000 7f035f3bc740 1 --2- 172.21.15.33:0/2629395614 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e876f0a2d0 0x55e876f1b9e0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.299+0000 7f035f3bc740 1 -- 172.21.15.33:0/2629395614 >> 172.21.15.33:0/2629395614 conn(0x55e876f04a00 msgr2=0x55e876f05c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:14.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.299+0000 7f035f3bc740 1 -- 172.21.15.33:0/2629395614 shutdown_connections 2024-03-19T02:59:14.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.299+0000 7f035f3bc740 1 -- 172.21.15.33:0/2629395614 wait complete. 2024-03-19T02:59:14.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.299+0000 7f035f3bc740 20 client.16914 trim_cache size 0 max 16384 2024-03-19T02:59:14.308 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:59:14.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.299+0000 7f035c6f1640 1 -- 172.21.15.33:0/2275476501 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f03400a2970 msgr2=0x7f03400a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:14.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.299+0000 7f035c6f1640 1 --2- 172.21.15.33:0/2275476501 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f03400a2970 0x7f03400a4e30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f034c0bd5b0 tx=0x7f034c08c6c0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.299+0000 7f035c6f1640 1 -- 172.21.15.33:0/2275476501 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f03541066b0 msgr2=0x7f035419c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:14.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.299+0000 7f035c6f1640 1 --2- 172.21.15.33:0/2275476501 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f03541066b0 0x7f035419c7f0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f034c004af0 tx=0x7f034c035ce0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.299+0000 7f035c6f1640 1 -- 172.21.15.33:0/2275476501 shutdown_connections 2024-03-19T02:59:14.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.299+0000 7f035c6f1640 1 --2- 172.21.15.33:0/2275476501 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f03400a2970 0x7f03400a4e30 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.299+0000 7f035c6f1640 1 --2- 172.21.15.33:0/2275476501 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0354101060 0x7f0354198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.299+0000 7f035c6f1640 1 --2- 172.21.15.33:0/2275476501 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f03541066b0 0x7f035419c7f0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.299+0000 7f035c6f1640 1 --2- 172.21.15.33:0/2275476501 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03541006a0 0x7f0354197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:14.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.299+0000 7f035c6f1640 1 -- 172.21.15.33:0/2275476501 >> 172.21.15.33:0/2275476501 conn(0x7f03540778d0 msgr2=0x7f0354102fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:14.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.299+0000 7f035c6f1640 1 -- 172.21.15.33:0/2275476501 shutdown_connections 2024-03-19T02:59:14.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:14.299+0000 7f035c6f1640 1 -- 172.21.15.33:0/2275476501 wait complete. 2024-03-19T02:59:14.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:14 smithi096 ceph-mon[35946]: pgmap v176: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-19T02:59:14.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:14 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2694523917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:59:15.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:14 smithi155 ceph-mon[35804]: pgmap v176: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-19T02:59:15.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:14 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2694523917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:59:15.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:14 smithi033 ceph-mon[31973]: pgmap v176: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-19T02:59:15.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:14 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2694523917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:59:15.312 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:59:15.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:59:15.313 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:59:15.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.430+0000 7f09fc2a1640 1 -- 172.21.15.33:0/3310296746 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09f41033b0 msgr2=0x7f09f410b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:15.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.430+0000 7f09fc2a1640 1 --2- 172.21.15.33:0/3310296746 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09f41033b0 0x7f09f410b890 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f09e8009860 tx=0x7f09e802bc00 comp rx=0 tx=0).stop 2024-03-19T02:59:15.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.430+0000 7f09fc2a1640 1 -- 172.21.15.33:0/3310296746 shutdown_connections 2024-03-19T02:59:15.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.430+0000 7f09fc2a1640 1 --2- 172.21.15.33:0/3310296746 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09f410be60 0x7f09f410e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.430+0000 7f09fc2a1640 1 --2- 172.21.15.33:0/3310296746 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09f41033b0 0x7f09f410b890 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.430+0000 7f09fc2a1640 1 --2- 172.21.15.33:0/3310296746 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f09f4102a90 0x7f09f4102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.430+0000 7f09fc2a1640 1 -- 172.21.15.33:0/3310296746 >> 172.21.15.33:0/3310296746 conn(0x7f09f4077880 msgr2=0x7f09f4077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:15.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.431+0000 7f09fc2a1640 1 -- 172.21.15.33:0/3310296746 shutdown_connections 2024-03-19T02:59:15.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.431+0000 7f09fc2a1640 1 -- 172.21.15.33:0/3310296746 wait complete. 2024-03-19T02:59:15.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.431+0000 7f09fc2a1640 1 Processor -- start 2024-03-19T02:59:15.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.431+0000 7f09fc2a1640 1 -- start start 2024-03-19T02:59:15.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.431+0000 7f09fc2a1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f09f4102a90 0x7f09f4198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.431+0000 7f09fc2a1640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09f41033b0 0x7f09f41985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.431+0000 7f09fc2a1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09f410be60 0x7f09f419c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.431+0000 7f09fc2a1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f09f4111a90 con 0x7f09f4102a90 2024-03-19T02:59:15.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.431+0000 7f09fc2a1640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f09f4111910 con 0x7f09f410be60 2024-03-19T02:59:15.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.431+0000 7f09fc2a1640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f09f4111c10 con 0x7f09f41033b0 2024-03-19T02:59:15.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.431+0000 7f09fa03d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f09f4102a90 0x7f09f4198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.431+0000 7f09fa03d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f09f4102a90 0x7f09f4198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51508/0 (socket says 172.21.15.33:51508) 2024-03-19T02:59:15.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.431+0000 7f09fa03d640 1 -- 172.21.15.33:0/23282008 learned_addr learned my addr 172.21.15.33:0/23282008 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:15.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.431+0000 7f09fa83e640 1 --2- 172.21.15.33:0/23282008 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09f410be60 0x7f09f419c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.432+0000 7f09f983c640 1 --2- 172.21.15.33:0/23282008 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09f41033b0 0x7f09f41985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.432+0000 7f09fa03d640 1 -- 172.21.15.33:0/23282008 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09f41033b0 msgr2=0x7f09f41985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:15.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.432+0000 7f09fa03d640 1 --2- 172.21.15.33:0/23282008 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09f41033b0 0x7f09f41985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.432+0000 7f09fa03d640 1 -- 172.21.15.33:0/23282008 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09f410be60 msgr2=0x7f09f419c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:15.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.432+0000 7f09fa03d640 1 --2- 172.21.15.33:0/23282008 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09f410be60 0x7f09f419c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.432+0000 7f09fa03d640 1 -- 172.21.15.33:0/23282008 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f419d070 con 0x7f09f4102a90 2024-03-19T02:59:15.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.432+0000 7f09fa83e640 1 --2- 172.21.15.33:0/23282008 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09f410be60 0x7f09f419c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:15.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.432+0000 7f09fa03d640 1 --2- 172.21.15.33:0/23282008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f09f4102a90 0x7f09f4198080 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f09f000e9c0 tx=0x7f09f000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:15.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.432+0000 7f09e77fe640 1 -- 172.21.15.33:0/23282008 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f000ccc0 con 0x7f09f4102a90 2024-03-19T02:59:15.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.432+0000 7f09fc2a1640 1 -- 172.21.15.33:0/23282008 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09f419d360 con 0x7f09f4102a90 2024-03-19T02:59:15.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.432+0000 7f09e77fe640 1 -- 172.21.15.33:0/23282008 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09f00054d0 con 0x7f09f4102a90 2024-03-19T02:59:15.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.432+0000 7f09e77fe640 1 -- 172.21.15.33:0/23282008 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f0010430 con 0x7f09f4102a90 2024-03-19T02:59:15.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.432+0000 7f09fc2a1640 1 -- 172.21.15.33:0/23282008 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09f41a4bf0 con 0x7f09f4102a90 2024-03-19T02:59:15.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.434+0000 7f09e77fe640 1 -- 172.21.15.33:0/23282008 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f09f00046c0 con 0x7f09f4102a90 2024-03-19T02:59:15.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.435+0000 7f09fef6e740 1 Processor -- start 2024-03-19T02:59:15.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.435+0000 7f09fef6e740 1 -- start start 2024-03-19T02:59:15.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.435+0000 7f09fef6e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5627b672c380 0x5627b672c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.435+0000 7f09fef6e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627b672cca0 0x5627b6735340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.435+0000 7f09e77fe640 1 --2- 172.21.15.33:0/23282008 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f09e00a2970 0x7f09e00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.435+0000 7f09fef6e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5627b6735880 0x5627b6737c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.435+0000 7f09fef6e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5627b6690160 con 0x5627b6735880 2024-03-19T02:59:15.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.435+0000 7f09fef6e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5627b6638e10 con 0x5627b672c380 2024-03-19T02:59:15.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.435+0000 7f09fef6e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5627b668c200 con 0x5627b672cca0 2024-03-19T02:59:15.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.435+0000 7f09fa83e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5627b6735880 0x5627b6737c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.435+0000 7f09fa03d640 1 --2- 172.21.15.33:0/23282008 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f09e00a2970 0x7f09e00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.435+0000 7f09fa83e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5627b6735880 0x5627b6737c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51524/0 (socket says 172.21.15.33:51524) 2024-03-19T02:59:15.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.435+0000 7f09fa83e640 1 -- 172.21.15.33:0/2230943522 learned_addr learned my addr 172.21.15.33:0/2230943522 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:15.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.435+0000 7f09f983c640 1 --2- 172.21.15.33:0/2230943522 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627b672cca0 0x5627b6735340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.436+0000 7f09fa83e640 1 --2- 172.21.15.33:0/2230943522 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5627b672c380 0x5627b672c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.436+0000 7f09e77fe640 1 -- 172.21.15.33:0/23282008 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f09f00cf480 con 0x7f09f4102a90 2024-03-19T02:59:15.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.436+0000 7f09fa83e640 1 -- 172.21.15.33:0/2230943522 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627b672cca0 msgr2=0x5627b6735340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:15.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.436+0000 7f09fa83e640 1 --2- 172.21.15.33:0/2230943522 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627b672cca0 0x5627b6735340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.436+0000 7f09fa83e640 1 -- 172.21.15.33:0/2230943522 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5627b672c380 msgr2=0x5627b672c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:15.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.436+0000 7f09fa83e640 1 --2- 172.21.15.33:0/2230943522 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5627b672c380 0x5627b672c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.436+0000 7f09fa83e640 1 -- 172.21.15.33:0/2230943522 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627b646c7e0 con 0x5627b6735880 2024-03-19T02:59:15.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.436+0000 7f09f983c640 1 --2- 172.21.15.33:0/2230943522 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627b672cca0 0x5627b6735340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:15.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.436+0000 7f09fa03d640 1 --2- 172.21.15.33:0/23282008 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f09e00a2970 0x7f09e00a4e30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f09f0097b90 tx=0x7f09f0005880 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:15.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.436+0000 7f09fa83e640 1 --2- 172.21.15.33:0/2230943522 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5627b6735880 0x5627b6737c70 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f09ec009fd0 tx=0x7f09ec005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cb0d5b9e87dad80 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:15.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.436+0000 7f09e4bfe640 1 -- 172.21.15.33:0/2230943522 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09ec00fc20 con 0x5627b6735880 2024-03-19T02:59:15.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.436+0000 7f09e4bfe640 1 -- 172.21.15.33:0/2230943522 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09ec00fdc0 con 0x5627b6735880 2024-03-19T02:59:15.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.436+0000 7f09e4bfe640 1 -- 172.21.15.33:0/2230943522 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09ec00e610 con 0x5627b6735880 2024-03-19T02:59:15.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.437+0000 7f09fef6e740 1 -- 172.21.15.33:0/2230943522 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5627b6735880 msgr2=0x5627b6737c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:15.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.437+0000 7f09fef6e740 1 --2- 172.21.15.33:0/2230943522 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5627b6735880 0x5627b6737c70 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f09ec009fd0 tx=0x7f09ec005bd0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.437+0000 7f09fef6e740 1 -- 172.21.15.33:0/2230943522 shutdown_connections 2024-03-19T02:59:15.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.437+0000 7f09fef6e740 1 --2- 172.21.15.33:0/2230943522 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627b672cca0 0x5627b6735340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.437+0000 7f09fef6e740 1 --2- 172.21.15.33:0/2230943522 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5627b6735880 0x5627b6737c70 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.437+0000 7f09fef6e740 1 --2- 172.21.15.33:0/2230943522 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5627b672c380 0x5627b672c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.437+0000 7f09fef6e740 1 -- 172.21.15.33:0/2230943522 >> 172.21.15.33:0/2230943522 conn(0x5627b6726ab0 msgr2=0x5627b6726e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:15.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.437+0000 7f09fef6e740 1 -- 172.21.15.33:0/2230943522 shutdown_connections 2024-03-19T02:59:15.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.437+0000 7f09fef6e740 1 -- 172.21.15.33:0/2230943522 wait complete. 2024-03-19T02:59:15.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.437+0000 7f09fef6e740 1 -- start start 2024-03-19T02:59:15.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.438+0000 7f09fef6e740 1 Processor -- start 2024-03-19T02:59:15.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.438+0000 7f09fef6e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5627b672c380 0x5627b6732750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.438+0000 7f09fef6e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5627b672cca0 0x5627b6732c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.438+0000 7f09fef6e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627b6735880 0x5627b67331d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.438+0000 7f09fef6e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5627b6738610 con 0x5627b672c380 2024-03-19T02:59:15.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.438+0000 7f09fef6e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5627b664c1a0 con 0x5627b672cca0 2024-03-19T02:59:15.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.438+0000 7f09fef6e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5627b6738790 con 0x5627b6735880 2024-03-19T02:59:15.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.438+0000 7f09fef6e740 10 client.0 _ll_register_callbacks cb 0x5627b63a3c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:59:15.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.438+0000 7f09fa83e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5627b672c380 0x5627b6732750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.438+0000 7f09fa83e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5627b672c380 0x5627b6732750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51526/0 (socket says 172.21.15.33:51526) 2024-03-19T02:59:15.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.438+0000 7f09fa83e640 1 -- 172.21.15.33:0/2638229963 learned_addr learned my addr 172.21.15.33:0/2638229963 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:15.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.438+0000 7f09f983c640 1 --2- 172.21.15.33:0/2638229963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5627b672cca0 0x5627b6732c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.438+0000 7f09fa83e640 1 --2- 172.21.15.33:0/2638229963 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627b6735880 0x5627b67331d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.438+0000 7f09fa83e640 1 -- 172.21.15.33:0/2638229963 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627b6735880 msgr2=0x5627b67331d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:15.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.438+0000 7f09fa83e640 1 --2- 172.21.15.33:0/2638229963 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627b6735880 0x5627b67331d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.438+0000 7f09fa83e640 1 -- 172.21.15.33:0/2638229963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5627b672cca0 msgr2=0x5627b6732c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:15.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.438+0000 7f09fa83e640 1 --2- 172.21.15.33:0/2638229963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5627b672cca0 0x5627b6732c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.438+0000 7f09fa83e640 1 -- 172.21.15.33:0/2638229963 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627b64ccf70 con 0x5627b672c380 2024-03-19T02:59:15.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.438+0000 7f09fa83e640 1 --2- 172.21.15.33:0/2638229963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5627b672c380 0x5627b6732750 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f09ec01ef20 tx=0x7f09ec01ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78610112ef98cc37 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:15.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.438+0000 7f09ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:59:15.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.439+0000 7f09ddffb640 1 -- 172.21.15.33:0/2638229963 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09ec00fc20 con 0x5627b672c380 2024-03-19T02:59:15.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.439+0000 7f09ddffb640 1 -- 172.21.15.33:0/2638229963 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f09ec004c40 con 0x5627b672c380 2024-03-19T02:59:15.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.439+0000 7f09fef6e740 1 -- 172.21.15.33:0/2638229963 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09ec00fdc0 con 0x5627b672c380 2024-03-19T02:59:15.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.439+0000 7f09ddffb640 1 -- 172.21.15.33:0/2638229963 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f09ec034050 con 0x5627b672c380 2024-03-19T02:59:15.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.439+0000 7f09fef6e740 10 client.16926 fetch_fsmap learned FSMap version 25 2024-03-19T02:59:15.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.439+0000 7f09fef6e740 1 -- 172.21.15.33:0/2638229963 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5627b6364170 con 0x5627b672c380 2024-03-19T02:59:15.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.439+0000 7f09ddffb640 1 -- 172.21.15.33:0/2638229963 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09ec00fdc0 con 0x5627b672c380 2024-03-19T02:59:15.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.439+0000 7f09ddffb640 1 -- 172.21.15.33:0/2638229963 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f09ec02c020 con 0x5627b672c380 2024-03-19T02:59:15.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.439+0000 7f09fef6e740 10 client.16926 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:59:15.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.439+0000 7f09fef6e740 10 client.16926 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:59:15.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.439+0000 7f09fef6e740 20 client.16926 populate_metadata read hostname 'smithi033' 2024-03-19T02:59:15.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.439+0000 7f09fef6e740 1 --2- 172.21.15.33:0/2638229963 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5627b6745920 0x5627b6747d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.439+0000 7f09fef6e740 4 client.16926 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:59:15.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.439+0000 7f09fef6e740 1 -- 172.21.15.33:0/2638229963 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627b669d6a0 con 0x5627b6745920 2024-03-19T02:59:15.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.439+0000 7f09f983c640 1 --2- 172.21.15.33:0/2638229963 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5627b6745920 0x5627b6747d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.440+0000 7f09ddffb640 10 client.16926 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:59:15.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.440+0000 7f09f983c640 1 --2- 172.21.15.33:0/2638229963 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5627b6745920 0x5627b6747d10 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef73ef120add18f5 server_cookie=2d872d175526c14b in_seq=0 out_seq=0 2024-03-19T02:59:15.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.441+0000 7f09ddffb640 1 -- 172.21.15.33:0/2638229963 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5627b669d6a0 con 0x5627b6745920 2024-03-19T02:59:15.448 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.441+0000 7f09ddffb640 10 client.16926 handle_command_reply: tid=0 2024-03-19T02:59:15.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.441+0000 7f09fef6e740 1 client.16926 shutdown 2024-03-19T02:59:15.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.441+0000 7f09fef6e740 1 -- 172.21.15.33:0/2638229963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5627b672c380 msgr2=0x5627b6732750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:15.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.441+0000 7f09fef6e740 1 --2- 172.21.15.33:0/2638229963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5627b672c380 0x5627b6732750 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f09ec01ef20 tx=0x7f09ec01ee20 comp rx=0 tx=0).stop 2024-03-19T02:59:15.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.441+0000 7f09fef6e740 1 -- 172.21.15.33:0/2638229963 shutdown_connections 2024-03-19T02:59:15.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.441+0000 7f09fef6e740 1 --2- 172.21.15.33:0/2638229963 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5627b6745920 0x5627b6747d10 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.441+0000 7f09fef6e740 1 --2- 172.21.15.33:0/2638229963 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627b6735880 0x5627b67331d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.441+0000 7f09fef6e740 1 --2- 172.21.15.33:0/2638229963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5627b672cca0 0x5627b6732c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.441+0000 7f09ddffb640 0 client.16926 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:59:15.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.441+0000 7f09fef6e740 1 --2- 172.21.15.33:0/2638229963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5627b672c380 0x5627b6732750 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.441+0000 7f09fef6e740 1 -- 172.21.15.33:0/2638229963 >> 172.21.15.33:0/2638229963 conn(0x5627b6726ab0 msgr2=0x5627b6728800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:15.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.441+0000 7f09fef6e740 1 -- 172.21.15.33:0/2638229963 shutdown_connections 2024-03-19T02:59:15.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.441+0000 7f09fef6e740 1 -- 172.21.15.33:0/2638229963 wait complete. 2024-03-19T02:59:15.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.441+0000 7f09fef6e740 20 client.16926 trim_cache size 0 max 16384 2024-03-19T02:59:15.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.459+0000 7f09fef6e740 1 Processor -- start 2024-03-19T02:59:15.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.459+0000 7f09fef6e740 1 -- start start 2024-03-19T02:59:15.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.459+0000 7f09fef6e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627b672c380 0x5627b6734e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.459+0000 7f09fef6e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5627b672cca0 0x5627b6741410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.459+0000 7f09fef6e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5627b6735880 0x5627b6731600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.459+0000 7f09fef6e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f09f41b15a0 con 0x5627b6735880 2024-03-19T02:59:15.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.459+0000 7f09fef6e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f09f41aee70 con 0x5627b672cca0 2024-03-19T02:59:15.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.459+0000 7f09fef6e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f09f41acca0 con 0x5627b672c380 2024-03-19T02:59:15.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.459+0000 7f09fa83e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5627b6735880 0x5627b6731600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.465 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.459+0000 7f09f983c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5627b672cca0 0x5627b6741410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.459+0000 7f09fa83e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5627b6735880 0x5627b6731600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51534/0 (socket says 172.21.15.33:51534) 2024-03-19T02:59:15.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.459+0000 7f09fa83e640 1 -- 172.21.15.33:0/2263471086 learned_addr learned my addr 172.21.15.33:0/2263471086 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:15.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.459+0000 7f09fa83e640 1 --2- 172.21.15.33:0/2263471086 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627b672c380 0x5627b6734e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.459+0000 7f09f983c640 1 -- 172.21.15.33:0/2263471086 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627b672c380 msgr2=0x5627b6734e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:15.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.459+0000 7f09f983c640 1 --2- 172.21.15.33:0/2263471086 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627b672c380 0x5627b6734e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.459+0000 7f09f983c640 1 -- 172.21.15.33:0/2263471086 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5627b6735880 msgr2=0x5627b6731600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:15.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.459+0000 7f09f983c640 1 --2- 172.21.15.33:0/2263471086 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5627b6735880 0x5627b6731600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.459+0000 7f09f983c640 1 -- 172.21.15.33:0/2263471086 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627b6748370 con 0x5627b672cca0 2024-03-19T02:59:15.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.460+0000 7f09f983c640 1 --2- 172.21.15.33:0/2263471086 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5627b672cca0 0x5627b6741410 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f09e8005940 tx=0x7f09e8034580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9002e98c9e5cc760 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:15.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.460+0000 7f09dffff640 1 -- 172.21.15.33:0/2263471086 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e8047070 con 0x5627b672cca0 2024-03-19T02:59:15.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.460+0000 7f09dffff640 1 -- 172.21.15.33:0/2263471086 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09e8045070 con 0x5627b672cca0 2024-03-19T02:59:15.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.460+0000 7f09dffff640 1 -- 172.21.15.33:0/2263471086 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e8039040 con 0x5627b672cca0 2024-03-19T02:59:15.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.460+0000 7f09fef6e740 1 -- 172.21.15.33:0/2263471086 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5627b672cca0 msgr2=0x5627b6741410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:15.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.460+0000 7f09fef6e740 1 --2- 172.21.15.33:0/2263471086 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5627b672cca0 0x5627b6741410 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f09e8005940 tx=0x7f09e8034580 comp rx=0 tx=0).stop 2024-03-19T02:59:15.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.460+0000 7f09fef6e740 1 -- 172.21.15.33:0/2263471086 shutdown_connections 2024-03-19T02:59:15.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.460+0000 7f09fef6e740 1 --2- 172.21.15.33:0/2263471086 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5627b6735880 0x5627b6731600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.460+0000 7f09fef6e740 1 --2- 172.21.15.33:0/2263471086 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5627b672cca0 0x5627b6741410 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.460+0000 7f09fef6e740 1 --2- 172.21.15.33:0/2263471086 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627b672c380 0x5627b6734e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.460+0000 7f09fef6e740 1 -- 172.21.15.33:0/2263471086 >> 172.21.15.33:0/2263471086 conn(0x5627b6726ab0 msgr2=0x5627b6730ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:15.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.460+0000 7f09fef6e740 1 -- 172.21.15.33:0/2263471086 shutdown_connections 2024-03-19T02:59:15.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.461+0000 7f09fef6e740 1 -- 172.21.15.33:0/2263471086 wait complete. 2024-03-19T02:59:15.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.461+0000 7f09fef6e740 1 -- start start 2024-03-19T02:59:15.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.461+0000 7f09fef6e740 1 Processor -- start 2024-03-19T02:59:15.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.461+0000 7f09fef6e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5627b672c380 0x5627b672f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.461+0000 7f09fef6e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627b672cca0 0x5627b672fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.461+0000 7f09fef6e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5627b6735880 0x5627b67aabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.461+0000 7f09fef6e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f09f41ad330 con 0x5627b6735880 2024-03-19T02:59:15.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.461+0000 7f09fef6e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f09f41ae7a0 con 0x5627b672c380 2024-03-19T02:59:15.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.461+0000 7f09fef6e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f09f41ada00 con 0x5627b672cca0 2024-03-19T02:59:15.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.461+0000 7f09fef6e740 10 client.0 _ll_register_callbacks cb 0x7f09ec0119d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:59:15.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.461+0000 7f09fa83e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5627b6735880 0x5627b67aabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.461+0000 7f09fa83e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5627b6735880 0x5627b67aabd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51536/0 (socket says 172.21.15.33:51536) 2024-03-19T02:59:15.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.461+0000 7f09fa83e640 1 -- 172.21.15.33:0/3613226785 learned_addr learned my addr 172.21.15.33:0/3613226785 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:15.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.461+0000 7f09f983c640 1 --2- 172.21.15.33:0/3613226785 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627b672cca0 0x5627b672fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.461+0000 7f09fa83e640 1 --2- 172.21.15.33:0/3613226785 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5627b672c380 0x5627b672f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.462+0000 7f09fa83e640 1 -- 172.21.15.33:0/3613226785 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627b672cca0 msgr2=0x5627b672fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:15.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.462+0000 7f09fa83e640 1 --2- 172.21.15.33:0/3613226785 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627b672cca0 0x5627b672fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.462+0000 7f09fa83e640 1 -- 172.21.15.33:0/3613226785 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5627b672c380 msgr2=0x5627b672f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:15.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.462+0000 7f09fa83e640 1 --2- 172.21.15.33:0/3613226785 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5627b672c380 0x5627b672f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.462+0000 7f09fa83e640 1 -- 172.21.15.33:0/3613226785 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627b67268e0 con 0x5627b6735880 2024-03-19T02:59:15.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.462+0000 7f09fa83e640 1 --2- 172.21.15.33:0/3613226785 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5627b6735880 0x5627b67aabd0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f09ec000f80 tx=0x7f09ec024040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdac3dfff787c257 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:15.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.462+0000 7f09ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:59:15.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.462+0000 7f09ddffb640 1 -- 172.21.15.33:0/3613226785 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09ec003b70 con 0x5627b6735880 2024-03-19T02:59:15.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.462+0000 7f09ddffb640 1 -- 172.21.15.33:0/3613226785 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f09ec003d10 con 0x5627b6735880 2024-03-19T02:59:15.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.462+0000 7f09fef6e740 1 -- 172.21.15.33:0/3613226785 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09e8045070 con 0x5627b6735880 2024-03-19T02:59:15.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.462+0000 7f09ddffb640 1 -- 172.21.15.33:0/3613226785 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f09ec02d7f0 con 0x5627b6735880 2024-03-19T02:59:15.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.462+0000 7f09fef6e740 10 client.16932 fetch_fsmap learned FSMap version 25 2024-03-19T02:59:15.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.462+0000 7f09fef6e740 1 -- 172.21.15.33:0/3613226785 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5627b67ac0c0 con 0x5627b6735880 2024-03-19T02:59:15.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.462+0000 7f09ddffb640 1 -- 172.21.15.33:0/3613226785 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09e8045070 con 0x5627b6735880 2024-03-19T02:59:15.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.462+0000 7f09ddffb640 1 -- 172.21.15.33:0/3613226785 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f09ec02c070 con 0x5627b6735880 2024-03-19T02:59:15.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.462+0000 7f09fef6e740 10 client.16932 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:59:15.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.462+0000 7f09fef6e740 10 client.16932 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:59:15.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.462+0000 7f09fef6e740 20 client.16932 populate_metadata read hostname 'smithi033' 2024-03-19T02:59:15.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.462+0000 7f09fef6e740 1 --2- 172.21.15.33:0/3613226785 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5627b67ac730 0x5627b67aeb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.462+0000 7f09fef6e740 4 client.16932 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:59:15.469 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.462+0000 7f09fef6e740 1 -- 172.21.15.33:0/3613226785 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5627b66979f0 con 0x5627b67ac730 2024-03-19T02:59:15.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.463+0000 7f09f983c640 1 --2- 172.21.15.33:0/3613226785 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5627b67ac730 0x5627b67aeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.463+0000 7f09f983c640 1 --2- 172.21.15.33:0/3613226785 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5627b67ac730 0x5627b67aeb20 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cad723e2b9a77a4 server_cookie=7bd368e779cc90ad in_seq=0 out_seq=0 2024-03-19T02:59:15.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.463+0000 7f09ddffb640 10 client.16932 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:59:15.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.464+0000 7f09ddffb640 1 -- 172.21.15.33:0/3613226785 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5627b66979f0 con 0x5627b67ac730 2024-03-19T02:59:15.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.464+0000 7f09ddffb640 10 client.16932 handle_command_reply: tid=0 2024-03-19T02:59:15.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.464+0000 7f09fef6e740 1 client.16932 shutdown 2024-03-19T02:59:15.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.464+0000 7f09fef6e740 1 -- 172.21.15.33:0/3613226785 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5627b6735880 msgr2=0x5627b67aabd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:15.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.464+0000 7f09fef6e740 1 --2- 172.21.15.33:0/3613226785 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5627b6735880 0x5627b67aabd0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f09ec000f80 tx=0x7f09ec024040 comp rx=0 tx=0).stop 2024-03-19T02:59:15.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.464+0000 7f09fef6e740 1 -- 172.21.15.33:0/3613226785 shutdown_connections 2024-03-19T02:59:15.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.464+0000 7f09fef6e740 1 --2- 172.21.15.33:0/3613226785 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5627b67ac730 0x5627b67aeb20 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.464+0000 7f09fef6e740 1 --2- 172.21.15.33:0/3613226785 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5627b672cca0 0x5627b672fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.464+0000 7f09fef6e740 1 --2- 172.21.15.33:0/3613226785 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5627b6735880 0x5627b67aabd0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.464+0000 7f09fef6e740 1 --2- 172.21.15.33:0/3613226785 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5627b672c380 0x5627b672f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.464+0000 7f09ddffb640 0 client.16932 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:59:15.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.464+0000 7f09fef6e740 1 -- 172.21.15.33:0/3613226785 >> 172.21.15.33:0/3613226785 conn(0x5627b6726ab0 msgr2=0x5627b6731780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:15.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.464+0000 7f09fef6e740 1 -- 172.21.15.33:0/3613226785 shutdown_connections 2024-03-19T02:59:15.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.464+0000 7f09fef6e740 1 -- 172.21.15.33:0/3613226785 wait complete. 2024-03-19T02:59:15.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.465+0000 7f09fef6e740 20 client.16932 trim_cache size 0 max 16384 2024-03-19T02:59:15.472 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:59:15.472 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-03-19T02:59:15.472 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "83e8f064-2453-4a5d-a6bf-7156dd1b6686", 2024-03-19T02:59:15.472 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-03-19T02:59:15.472 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:59:15.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.465+0000 7f09fc2a1640 1 -- 172.21.15.33:0/23282008 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f09e00a2970 msgr2=0x7f09e00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:15.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.465+0000 7f09fc2a1640 1 --2- 172.21.15.33:0/23282008 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f09e00a2970 0x7f09e00a4e30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f09f0097b90 tx=0x7f09f0005880 comp rx=0 tx=0).stop 2024-03-19T02:59:15.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.465+0000 7f09fc2a1640 1 -- 172.21.15.33:0/23282008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f09f4102a90 msgr2=0x7f09f4198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:15.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.465+0000 7f09fc2a1640 1 --2- 172.21.15.33:0/23282008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f09f4102a90 0x7f09f4198080 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f09f000e9c0 tx=0x7f09f000ee80 comp rx=0 tx=0).stop 2024-03-19T02:59:15.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.465+0000 7f09fc2a1640 1 -- 172.21.15.33:0/23282008 shutdown_connections 2024-03-19T02:59:15.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.465+0000 7f09fc2a1640 1 --2- 172.21.15.33:0/23282008 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f09e00a2970 0x7f09e00a4e30 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.465+0000 7f09fc2a1640 1 --2- 172.21.15.33:0/23282008 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09f41033b0 0x7f09f41985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.465+0000 7f09fc2a1640 1 --2- 172.21.15.33:0/23282008 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f09f410be60 0x7f09f419c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.465+0000 7f09fc2a1640 1 --2- 172.21.15.33:0/23282008 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f09f4102a90 0x7f09f4198080 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.465+0000 7f09fc2a1640 1 -- 172.21.15.33:0/23282008 >> 172.21.15.33:0/23282008 conn(0x7f09f4077880 msgr2=0x7f09f40fd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:15.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.465+0000 7f09fc2a1640 1 -- 172.21.15.33:0/23282008 shutdown_connections 2024-03-19T02:59:15.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.465+0000 7f09fc2a1640 1 -- 172.21.15.33:0/23282008 wait complete. 2024-03-19T02:59:15.477 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:59:15.477 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:59:15.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.582+0000 7f151c1e6640 1 -- 172.21.15.33:0/406016858 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f15141033b0 msgr2=0x7f151410b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:15.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.582+0000 7f151c1e6640 1 --2- 172.21.15.33:0/406016858 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f15141033b0 0x7f151410b890 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f1504009900 tx=0x7f150402bcd0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.582+0000 7f151c1e6640 1 -- 172.21.15.33:0/406016858 shutdown_connections 2024-03-19T02:59:15.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.582+0000 7f151c1e6640 1 --2- 172.21.15.33:0/406016858 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f151410be60 0x7f151410e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.582+0000 7f151c1e6640 1 --2- 172.21.15.33:0/406016858 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f15141033b0 0x7f151410b890 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.582+0000 7f151c1e6640 1 --2- 172.21.15.33:0/406016858 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1514102a90 0x7f1514102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.582+0000 7f151c1e6640 1 -- 172.21.15.33:0/406016858 >> 172.21.15.33:0/406016858 conn(0x7f1514077880 msgr2=0x7f1514077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:15.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.582+0000 7f151c1e6640 1 -- 172.21.15.33:0/406016858 shutdown_connections 2024-03-19T02:59:15.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.582+0000 7f151c1e6640 1 -- 172.21.15.33:0/406016858 wait complete. 2024-03-19T02:59:15.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.583+0000 7f151c1e6640 1 Processor -- start 2024-03-19T02:59:15.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.583+0000 7f151c1e6640 1 -- start start 2024-03-19T02:59:15.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.583+0000 7f151c1e6640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1514102a90 0x7f15141980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.583+0000 7f151c1e6640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15141033b0 0x7f15141985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.583+0000 7f151c1e6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f151410be60 0x7f151419c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.583+0000 7f151c1e6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1514111ae0 con 0x7f151410be60 2024-03-19T02:59:15.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.583+0000 7f151c1e6640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1514111960 con 0x7f1514102a90 2024-03-19T02:59:15.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.583+0000 7f151c1e6640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1514111c60 con 0x7f15141033b0 2024-03-19T02:59:15.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.583+0000 7f151a783640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f151410be60 0x7f151419c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.583+0000 7f151a783640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f151410be60 0x7f151419c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51566/0 (socket says 172.21.15.33:51566) 2024-03-19T02:59:15.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.583+0000 7f151a783640 1 -- 172.21.15.33:0/4261732555 learned_addr learned my addr 172.21.15.33:0/4261732555 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:15.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.583+0000 7f1519f82640 1 --2- 172.21.15.33:0/4261732555 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1514102a90 0x7f15141980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.583+0000 7f1519781640 1 --2- 172.21.15.33:0/4261732555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15141033b0 0x7f15141985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.583+0000 7f151a783640 1 -- 172.21.15.33:0/4261732555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15141033b0 msgr2=0x7f15141985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:15.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.583+0000 7f151a783640 1 --2- 172.21.15.33:0/4261732555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15141033b0 0x7f15141985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.583+0000 7f151a783640 1 -- 172.21.15.33:0/4261732555 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1514102a90 msgr2=0x7f15141980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:15.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.583+0000 7f151a783640 1 --2- 172.21.15.33:0/4261732555 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1514102a90 0x7f15141980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.583+0000 7f151a783640 1 -- 172.21.15.33:0/4261732555 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f151419d040 con 0x7f151410be60 2024-03-19T02:59:15.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.583+0000 7f1519781640 1 --2- 172.21.15.33:0/4261732555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15141033b0 0x7f15141985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:59:15.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.584+0000 7f151a783640 1 --2- 172.21.15.33:0/4261732555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f151410be60 0x7f151419c910 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f150c00eb10 tx=0x7f150c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:15.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.584+0000 7f150affd640 1 -- 172.21.15.33:0/4261732555 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f150c004b20 con 0x7f151410be60 2024-03-19T02:59:15.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.584+0000 7f151c1e6640 1 -- 172.21.15.33:0/4261732555 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f151419d330 con 0x7f151410be60 2024-03-19T02:59:15.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.584+0000 7f150affd640 1 -- 172.21.15.33:0/4261732555 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f150c004cc0 con 0x7f151410be60 2024-03-19T02:59:15.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.584+0000 7f150affd640 1 -- 172.21.15.33:0/4261732555 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f150c010690 con 0x7f151410be60 2024-03-19T02:59:15.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.584+0000 7f151c1e6640 1 -- 172.21.15.33:0/4261732555 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15141a4bc0 con 0x7f151410be60 2024-03-19T02:59:15.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.586+0000 7f150affd640 1 -- 172.21.15.33:0/4261732555 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f150c005020 con 0x7f151410be60 2024-03-19T02:59:15.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.586+0000 7f151edd4740 1 Processor -- start 2024-03-19T02:59:15.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.587+0000 7f151edd4740 1 -- start start 2024-03-19T02:59:15.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.587+0000 7f151edd4740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564598154a30 0x564598154e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.587+0000 7f151edd4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564598155350 0x56459815d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.587+0000 7f151edd4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56459815df30 0x564598160320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.587+0000 7f151edd4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5645980b8500 con 0x56459815df30 2024-03-19T02:59:15.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.587+0000 7f151edd4740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x564598060be0 con 0x564598154a30 2024-03-19T02:59:15.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.587+0000 7f151edd4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5645980b45a0 con 0x564598155350 2024-03-19T02:59:15.595 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.587+0000 7f1519781640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56459815df30 0x564598160320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.587+0000 7f150affd640 1 --2- 172.21.15.33:0/4261732555 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f15000a2970 0x7f15000a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.587+0000 7f1519781640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56459815df30 0x564598160320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51568/0 (socket says 172.21.15.33:51568) 2024-03-19T02:59:15.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.587+0000 7f1519781640 1 -- 172.21.15.33:0/2274366559 learned_addr learned my addr 172.21.15.33:0/2274366559 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:15.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.587+0000 7f1519781640 1 --2- 172.21.15.33:0/2274366559 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564598154a30 0x564598154e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.587+0000 7f1519f82640 1 --2- 172.21.15.33:0/2274366559 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564598155350 0x56459815d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.587+0000 7f151a783640 1 --2- 172.21.15.33:0/4261732555 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f15000a2970 0x7f15000a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.587+0000 7f1519781640 1 -- 172.21.15.33:0/2274366559 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564598155350 msgr2=0x56459815d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:15.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.587+0000 7f1519781640 1 --2- 172.21.15.33:0/2274366559 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564598155350 0x56459815d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.587+0000 7f1519781640 1 -- 172.21.15.33:0/2274366559 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564598154a30 msgr2=0x564598154e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:15.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.587+0000 7f1519781640 1 --2- 172.21.15.33:0/2274366559 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564598154a30 0x564598154e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.587+0000 7f1519781640 1 -- 172.21.15.33:0/2274366559 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564598082000 con 0x56459815df30 2024-03-19T02:59:15.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.587+0000 7f150affd640 1 -- 172.21.15.33:0/4261732555 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f150c0cf8b0 con 0x7f151410be60 2024-03-19T02:59:15.596 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.587+0000 7f1519f82640 1 --2- 172.21.15.33:0/2274366559 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564598155350 0x56459815d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:59:15.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.588+0000 7f1519781640 1 --2- 172.21.15.33:0/2274366559 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56459815df30 0x564598160320 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f150402c210 tx=0x7f15040056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=567a8c02b513612c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:15.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.588+0000 7f151a783640 1 --2- 172.21.15.33:0/4261732555 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f15000a2970 0x7f15000a4e30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f150c098570 tx=0x7f150c06d7d0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:15.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.588+0000 7f14fffff640 1 -- 172.21.15.33:0/2274366559 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1504049070 con 0x56459815df30 2024-03-19T02:59:15.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.588+0000 7f14fffff640 1 -- 172.21.15.33:0/2274366559 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f150402cd30 con 0x56459815df30 2024-03-19T02:59:15.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.588+0000 7f14fffff640 1 -- 172.21.15.33:0/2274366559 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1504003d40 con 0x56459815df30 2024-03-19T02:59:15.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.588+0000 7f151edd4740 1 -- 172.21.15.33:0/2274366559 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56459815df30 msgr2=0x564598160320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:15.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.588+0000 7f151edd4740 1 --2- 172.21.15.33:0/2274366559 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56459815df30 0x564598160320 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f150402c210 tx=0x7f15040056d0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.588+0000 7f151edd4740 1 -- 172.21.15.33:0/2274366559 shutdown_connections 2024-03-19T02:59:15.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.588+0000 7f151edd4740 1 --2- 172.21.15.33:0/2274366559 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564598155350 0x56459815d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.588+0000 7f151edd4740 1 --2- 172.21.15.33:0/2274366559 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56459815df30 0x564598160320 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.588+0000 7f151edd4740 1 --2- 172.21.15.33:0/2274366559 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564598154a30 0x564598154e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.589+0000 7f151edd4740 1 -- 172.21.15.33:0/2274366559 >> 172.21.15.33:0/2274366559 conn(0x56459814f160 msgr2=0x56459814f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:15.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.589+0000 7f151edd4740 1 -- 172.21.15.33:0/2274366559 shutdown_connections 2024-03-19T02:59:15.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.589+0000 7f151edd4740 1 -- 172.21.15.33:0/2274366559 wait complete. 2024-03-19T02:59:15.597 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.589+0000 7f151edd4740 1 -- start start 2024-03-19T02:59:15.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.589+0000 7f151edd4740 1 Processor -- start 2024-03-19T02:59:15.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.589+0000 7f151edd4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564598154a30 0x56459815af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.589+0000 7f151edd4740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564598155350 0x56459815b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.589+0000 7f151edd4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56459815df30 0x56459815b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.589+0000 7f151edd4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564598160cc0 con 0x56459815df30 2024-03-19T02:59:15.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.589+0000 7f151edd4740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x564598074160 con 0x564598155350 2024-03-19T02:59:15.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.589+0000 7f151edd4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564598160e40 con 0x564598154a30 2024-03-19T02:59:15.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.589+0000 7f151edd4740 10 client.0 _ll_register_callbacks cb 0x5645980b0a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:59:15.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.589+0000 7f1519781640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56459815df30 0x56459815b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.590+0000 7f1519f82640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564598155350 0x56459815b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.590+0000 7f1519781640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56459815df30 0x56459815b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51580/0 (socket says 172.21.15.33:51580) 2024-03-19T02:59:15.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.590+0000 7f1519781640 1 -- 172.21.15.33:0/2820863212 learned_addr learned my addr 172.21.15.33:0/2820863212 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:15.598 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.590+0000 7f1519781640 1 --2- 172.21.15.33:0/2820863212 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564598154a30 0x56459815af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.590+0000 7f1519f82640 1 -- 172.21.15.33:0/2820863212 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564598154a30 msgr2=0x56459815af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:15.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.590+0000 7f1519f82640 1 --2- 172.21.15.33:0/2820863212 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564598154a30 0x56459815af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.590+0000 7f1519f82640 1 -- 172.21.15.33:0/2820863212 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56459815df30 msgr2=0x56459815b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:15.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.590+0000 7f1519f82640 1 --2- 172.21.15.33:0/2820863212 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56459815df30 0x56459815b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.590+0000 7f1519f82640 1 -- 172.21.15.33:0/2820863212 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564597e967d0 con 0x564598155350 2024-03-19T02:59:15.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.590+0000 7f1519781640 1 --2- 172.21.15.33:0/2820863212 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56459815df30 0x56459815b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:15.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.590+0000 7f1519f82640 1 --2- 172.21.15.33:0/2820863212 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564598155350 0x56459815b490 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f15100126d0 tx=0x7f1510012b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b4d7144a5597b04 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:15.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.590+0000 7f14fd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T02:59:15.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.590+0000 7f14fd7fa640 1 -- 172.21.15.33:0/2820863212 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1510004050 con 0x564598155350 2024-03-19T02:59:15.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.590+0000 7f14fd7fa640 1 -- 172.21.15.33:0/2820863212 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f15100041f0 con 0x564598155350 2024-03-19T02:59:15.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.590+0000 7f151edd4740 1 -- 172.21.15.33:0/2820863212 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f150402cd30 con 0x564598155350 2024-03-19T02:59:15.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.590+0000 7f14fd7fa640 1 -- 172.21.15.33:0/2820863212 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1510024050 con 0x564598155350 2024-03-19T02:59:15.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.590+0000 7f151edd4740 10 client.26768 fetch_fsmap learned FSMap version 25 2024-03-19T02:59:15.599 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.590+0000 7f151edd4740 1 -- 172.21.15.33:0/2820863212 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564597dcc3f0 con 0x564598155350 2024-03-19T02:59:15.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.591+0000 7f14fd7fa640 1 -- 172.21.15.33:0/2820863212 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f150402cd30 con 0x564598155350 2024-03-19T02:59:15.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.591+0000 7f14fd7fa640 1 -- 172.21.15.33:0/2820863212 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f1510003930 con 0x564598155350 2024-03-19T02:59:15.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.591+0000 7f151edd4740 10 client.26768 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:59:15.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.591+0000 7f151edd4740 10 client.26768 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:59:15.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.591+0000 7f151edd4740 20 client.26768 populate_metadata read hostname 'smithi033' 2024-03-19T02:59:15.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.591+0000 7f151edd4740 1 --2- 172.21.15.33:0/2820863212 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x56459816e360 0x564598170780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.591+0000 7f151edd4740 4 client.26768 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:59:15.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.591+0000 7f151edd4740 1 -- 172.21.15.33:0/2820863212 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5645980c5a40 con 0x56459816e360 2024-03-19T02:59:15.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.591+0000 7f1519781640 1 --2- 172.21.15.33:0/2820863212 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x56459816e360 0x564598170780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.591+0000 7f14fd7fa640 10 client.26768 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:59:15.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.591+0000 7f1519781640 1 --2- 172.21.15.33:0/2820863212 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x56459816e360 0x564598170780 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=334532f53b15135b server_cookie=9d87647b5190ab54 in_seq=0 out_seq=0 2024-03-19T02:59:15.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.592+0000 7f14fd7fa640 1 -- 172.21.15.33:0/2820863212 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5645980c5a40 con 0x56459816e360 2024-03-19T02:59:15.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.592+0000 7f14fd7fa640 10 client.26768 handle_command_reply: tid=0 2024-03-19T02:59:15.600 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.592+0000 7f151edd4740 1 client.26768 shutdown 2024-03-19T02:59:15.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.592+0000 7f151edd4740 1 -- 172.21.15.33:0/2820863212 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564598155350 msgr2=0x56459815b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:15.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.592+0000 7f151edd4740 1 --2- 172.21.15.33:0/2820863212 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564598155350 0x56459815b490 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f15100126d0 tx=0x7f1510012b90 comp rx=0 tx=0).stop 2024-03-19T02:59:15.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.592+0000 7f151edd4740 1 -- 172.21.15.33:0/2820863212 shutdown_connections 2024-03-19T02:59:15.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.592+0000 7f151edd4740 1 --2- 172.21.15.33:0/2820863212 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x56459816e360 0x564598170780 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.592+0000 7f151edd4740 1 --2- 172.21.15.33:0/2820863212 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56459815df30 0x56459815b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.592+0000 7f151edd4740 1 --2- 172.21.15.33:0/2820863212 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564598155350 0x56459815b490 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.592+0000 7f151edd4740 1 --2- 172.21.15.33:0/2820863212 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564598154a30 0x56459815af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.592+0000 7f14fd7fa640 0 client.26768 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:59:15.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.592+0000 7f151edd4740 1 -- 172.21.15.33:0/2820863212 >> 172.21.15.33:0/2820863212 conn(0x56459814f160 msgr2=0x564598150eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:15.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.592+0000 7f151edd4740 1 -- 172.21.15.33:0/2820863212 shutdown_connections 2024-03-19T02:59:15.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.592+0000 7f151edd4740 1 -- 172.21.15.33:0/2820863212 wait complete. 2024-03-19T02:59:15.601 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.592+0000 7f151edd4740 20 client.26768 trim_cache size 0 max 16384 2024-03-19T02:59:15.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.609+0000 7f151edd4740 1 Processor -- start 2024-03-19T02:59:15.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.609+0000 7f151edd4740 1 -- start start 2024-03-19T02:59:15.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.609+0000 7f151edd4740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564598154a30 0x56459815d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.610+0000 7f151edd4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564598155350 0x564598159e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.610+0000 7f151edd4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56459815df30 0x564598156250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.610+0000 7f151edd4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f15141b14e0 con 0x56459815df30 2024-03-19T02:59:15.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.610+0000 7f151edd4740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f15141aedb0 con 0x564598154a30 2024-03-19T02:59:15.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.610+0000 7f151edd4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f15141acbe0 con 0x564598155350 2024-03-19T02:59:15.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.610+0000 7f1519781640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56459815df30 0x564598156250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.610+0000 7f1519781640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56459815df30 0x564598156250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51584/0 (socket says 172.21.15.33:51584) 2024-03-19T02:59:15.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.610+0000 7f1519781640 1 -- 172.21.15.33:0/3517029295 learned_addr learned my addr 172.21.15.33:0/3517029295 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:15.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.610+0000 7f1519781640 1 --2- 172.21.15.33:0/3517029295 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564598154a30 0x56459815d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.610+0000 7f1519f82640 1 --2- 172.21.15.33:0/3517029295 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564598155350 0x564598159e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.610+0000 7f1519781640 1 -- 172.21.15.33:0/3517029295 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564598155350 msgr2=0x564598159e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:15.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.610+0000 7f1519781640 1 --2- 172.21.15.33:0/3517029295 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564598155350 0x564598159e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.610+0000 7f1519781640 1 -- 172.21.15.33:0/3517029295 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564598154a30 msgr2=0x56459815d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:15.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.610+0000 7f1519781640 1 --2- 172.21.15.33:0/3517029295 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564598154a30 0x56459815d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.610+0000 7f1519781640 1 -- 172.21.15.33:0/3517029295 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564597d8c0f0 con 0x56459815df30 2024-03-19T02:59:15.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.610+0000 7f1519781640 1 --2- 172.21.15.33:0/3517029295 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56459815df30 0x564598156250 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f1504004650 tx=0x7f15040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eab128daff21a32d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:15.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.610+0000 7f14ff7fe640 1 -- 172.21.15.33:0/3517029295 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1504049040 con 0x56459815df30 2024-03-19T02:59:15.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.610+0000 7f14ff7fe640 1 -- 172.21.15.33:0/3517029295 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f150402cc20 con 0x56459815df30 2024-03-19T02:59:15.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.610+0000 7f14ff7fe640 1 -- 172.21.15.33:0/3517029295 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1504043b50 con 0x56459815df30 2024-03-19T02:59:15.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.611+0000 7f151edd4740 1 -- 172.21.15.33:0/3517029295 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56459815df30 msgr2=0x564598156250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:15.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.611+0000 7f151edd4740 1 --2- 172.21.15.33:0/3517029295 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56459815df30 0x564598156250 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f1504004650 tx=0x7f15040057f0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.611+0000 7f151edd4740 1 -- 172.21.15.33:0/3517029295 shutdown_connections 2024-03-19T02:59:15.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.611+0000 7f151edd4740 1 --2- 172.21.15.33:0/3517029295 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564598155350 0x564598159e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.611+0000 7f151edd4740 1 --2- 172.21.15.33:0/3517029295 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56459815df30 0x564598156250 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.611+0000 7f151edd4740 1 --2- 172.21.15.33:0/3517029295 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564598154a30 0x56459815d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.611+0000 7f151edd4740 1 -- 172.21.15.33:0/3517029295 >> 172.21.15.33:0/3517029295 conn(0x56459814f160 msgr2=0x56459814e1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:15.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.611+0000 7f151edd4740 1 -- 172.21.15.33:0/3517029295 shutdown_connections 2024-03-19T02:59:15.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.611+0000 7f151edd4740 1 -- 172.21.15.33:0/3517029295 wait complete. 2024-03-19T02:59:15.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.611+0000 7f151edd4740 1 -- start start 2024-03-19T02:59:15.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.611+0000 7f151edd4740 1 Processor -- start 2024-03-19T02:59:15.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.612+0000 7f151edd4740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564598154a30 0x5645981664b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.612+0000 7f151edd4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564598155350 0x564598157960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.612+0000 7f151edd4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56459815df30 0x5645981580a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.612+0000 7f151edd4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f15141ad270 con 0x56459815df30 2024-03-19T02:59:15.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.612+0000 7f151edd4740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f15141ae6e0 con 0x564598154a30 2024-03-19T02:59:15.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.612+0000 7f151edd4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f15141ad940 con 0x564598155350 2024-03-19T02:59:15.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.612+0000 7f151edd4740 10 client.0 _ll_register_callbacks cb 0x7f1510009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:59:15.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.612+0000 7f1519781640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56459815df30 0x5645981580a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.612+0000 7f1519781640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56459815df30 0x5645981580a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51590/0 (socket says 172.21.15.33:51590) 2024-03-19T02:59:15.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.612+0000 7f1519781640 1 -- 172.21.15.33:0/2853164618 learned_addr learned my addr 172.21.15.33:0/2853164618 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:15.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.612+0000 7f1519781640 1 --2- 172.21.15.33:0/2853164618 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564598154a30 0x5645981664b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.619 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.612+0000 7f1519f82640 1 --2- 172.21.15.33:0/2853164618 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564598155350 0x564598157960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.612+0000 7f1519781640 1 -- 172.21.15.33:0/2853164618 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564598155350 msgr2=0x564598157960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:15.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.612+0000 7f1519781640 1 --2- 172.21.15.33:0/2853164618 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564598155350 0x564598157960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.612+0000 7f1519781640 1 -- 172.21.15.33:0/2853164618 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564598154a30 msgr2=0x5645981664b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:15.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.612+0000 7f1519781640 1 --2- 172.21.15.33:0/2853164618 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564598154a30 0x5645981664b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.612+0000 7f1519781640 1 -- 172.21.15.33:0/2853164618 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645981d3680 con 0x56459815df30 2024-03-19T02:59:15.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.612+0000 7f1519781640 1 --2- 172.21.15.33:0/2853164618 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56459815df30 0x5645981580a0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f1504005860 tx=0x7f1504046040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca6554ae5ba3260a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:15.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.612+0000 7f14fd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:59:15.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.612+0000 7f14fd7fa640 1 -- 172.21.15.33:0/2853164618 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1504049070 con 0x56459815df30 2024-03-19T02:59:15.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.612+0000 7f14fd7fa640 1 -- 172.21.15.33:0/2853164618 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f1504043650 con 0x56459815df30 2024-03-19T02:59:15.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.612+0000 7f151edd4740 1 -- 172.21.15.33:0/2853164618 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f150402cc20 con 0x56459815df30 2024-03-19T02:59:15.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.613+0000 7f14fd7fa640 1 -- 172.21.15.33:0/2853164618 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f150405b040 con 0x56459815df30 2024-03-19T02:59:15.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.613+0000 7f151edd4740 10 client.16950 fetch_fsmap learned FSMap version 25 2024-03-19T02:59:15.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.613+0000 7f151edd4740 1 -- 172.21.15.33:0/2853164618 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5645981d4810 con 0x56459815df30 2024-03-19T02:59:15.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.613+0000 7f14fd7fa640 1 -- 172.21.15.33:0/2853164618 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f150402cc20 con 0x56459815df30 2024-03-19T02:59:15.620 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.613+0000 7f14fd7fa640 1 -- 172.21.15.33:0/2853164618 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f150404f040 con 0x56459815df30 2024-03-19T02:59:15.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.613+0000 7f151edd4740 10 client.16950 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:59:15.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.613+0000 7f151edd4740 10 client.16950 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:59:15.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.613+0000 7f151edd4740 20 client.16950 populate_metadata read hostname 'smithi033' 2024-03-19T02:59:15.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.613+0000 7f151edd4740 1 --2- 172.21.15.33:0/2853164618 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5645981d4aa0 0x5645981d6e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:15.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.613+0000 7f151edd4740 4 client.16950 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:59:15.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.613+0000 7f151edd4740 1 -- 172.21.15.33:0/2853164618 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5645980bfd90 con 0x5645981d4aa0 2024-03-19T02:59:15.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.613+0000 7f1519f82640 1 --2- 172.21.15.33:0/2853164618 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5645981d4aa0 0x5645981d6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:15.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.614+0000 7f1519f82640 1 --2- 172.21.15.33:0/2853164618 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5645981d4aa0 0x5645981d6e90 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b50cf8f1f927998 server_cookie=2d1a1b506586c855 in_seq=0 out_seq=0 2024-03-19T02:59:15.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.614+0000 7f14fd7fa640 10 client.16950 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:59:15.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.614+0000 7f14fd7fa640 1 -- 172.21.15.33:0/2853164618 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5645980bfd90 con 0x5645981d4aa0 2024-03-19T02:59:15.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.614+0000 7f14fd7fa640 10 client.16950 handle_command_reply: tid=0 2024-03-19T02:59:15.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.614+0000 7f151edd4740 1 client.16950 shutdown 2024-03-19T02:59:15.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.614+0000 7f151edd4740 1 -- 172.21.15.33:0/2853164618 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56459815df30 msgr2=0x5645981580a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:15.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.614+0000 7f151edd4740 1 --2- 172.21.15.33:0/2853164618 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56459815df30 0x5645981580a0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f1504005860 tx=0x7f1504046040 comp rx=0 tx=0).stop 2024-03-19T02:59:15.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.614+0000 7f151edd4740 1 -- 172.21.15.33:0/2853164618 shutdown_connections 2024-03-19T02:59:15.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.614+0000 7f151edd4740 1 --2- 172.21.15.33:0/2853164618 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5645981d4aa0 0x5645981d6e90 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.614+0000 7f151edd4740 1 --2- 172.21.15.33:0/2853164618 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564598155350 0x564598157960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.614+0000 7f151edd4740 1 --2- 172.21.15.33:0/2853164618 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56459815df30 0x5645981580a0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.614+0000 7f14fd7fa640 0 client.16950 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:59:15.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.614+0000 7f151edd4740 1 --2- 172.21.15.33:0/2853164618 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564598154a30 0x5645981664b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.614+0000 7f151edd4740 1 -- 172.21.15.33:0/2853164618 >> 172.21.15.33:0/2853164618 conn(0x56459814f160 msgr2=0x56459815af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:15.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.614+0000 7f151edd4740 1 -- 172.21.15.33:0/2853164618 shutdown_connections 2024-03-19T02:59:15.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.614+0000 7f151edd4740 1 -- 172.21.15.33:0/2853164618 wait complete. 2024-03-19T02:59:15.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.614+0000 7f151edd4740 20 client.16950 trim_cache size 0 max 16384 2024-03-19T02:59:15.623 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:59:15.623 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:59:15.623 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-03-19T02:59:15.623 INFO:teuthology.orchestra.run.smithi033.stdout: "83e8f064-2453-4a5d-a6bf-7156dd1b6686": { 2024-03-19T02:59:15.623 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-03-19T02:59:15.623 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "83e8f064-2453-4a5d-a6bf-7156dd1b6686", 2024-03-19T02:59:15.623 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-03-19T02:59:15.623 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T02:59:15.623 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T02:59:15.623 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:59:15.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.615+0000 7f151c1e6640 1 -- 172.21.15.33:0/4261732555 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f15000a2970 msgr2=0x7f15000a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:15.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.615+0000 7f151c1e6640 1 --2- 172.21.15.33:0/4261732555 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f15000a2970 0x7f15000a4e30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f150c098570 tx=0x7f150c06d7d0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.615+0000 7f151c1e6640 1 -- 172.21.15.33:0/4261732555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f151410be60 msgr2=0x7f151419c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:15.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.615+0000 7f151c1e6640 1 --2- 172.21.15.33:0/4261732555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f151410be60 0x7f151419c910 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f150c00eb10 tx=0x7f150c00c2d0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.615+0000 7f151c1e6640 1 -- 172.21.15.33:0/4261732555 shutdown_connections 2024-03-19T02:59:15.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.615+0000 7f151c1e6640 1 --2- 172.21.15.33:0/4261732555 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f15000a2970 0x7f15000a4e30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.624 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.615+0000 7f151c1e6640 1 --2- 172.21.15.33:0/4261732555 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f15141033b0 0x7f15141985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.615+0000 7f151c1e6640 1 --2- 172.21.15.33:0/4261732555 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f151410be60 0x7f151419c910 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.615+0000 7f151c1e6640 1 --2- 172.21.15.33:0/4261732555 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1514102a90 0x7f15141980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:15.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.615+0000 7f151c1e6640 1 -- 172.21.15.33:0/4261732555 >> 172.21.15.33:0/4261732555 conn(0x7f1514077880 msgr2=0x7f15140fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:15.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.615+0000 7f151c1e6640 1 -- 172.21.15.33:0/4261732555 shutdown_connections 2024-03-19T02:59:15.625 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:15.615+0000 7f151c1e6640 1 -- 172.21.15.33:0/4261732555 wait complete. 2024-03-19T02:59:15.627 INFO:tasks.cephfs.filesystem:scrub status for tag:83e8f064-2453-4a5d-a6bf-7156dd1b6686 - {'path': '/', 'tag': '83e8f064-2453-4a5d-a6bf-7156dd1b6686', 'options': 'recursive,force'} 2024-03-19T02:59:16.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:16 smithi096 ceph-mon[35946]: pgmap v177: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-19T02:59:16.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:16 smithi096 ceph-mon[35946]: scrub queued for path: / 2024-03-19T02:59:16.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:16 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T02:59:16.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:16 smithi096 ceph-mon[35946]: scrub summary: active paths [/] 2024-03-19T02:59:16.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:16 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T02:59:17.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:16 smithi155 ceph-mon[35804]: pgmap v177: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-19T02:59:17.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:16 smithi155 ceph-mon[35804]: scrub queued for path: / 2024-03-19T02:59:17.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:16 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T02:59:17.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:16 smithi155 ceph-mon[35804]: scrub summary: active paths [/] 2024-03-19T02:59:17.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:16 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T02:59:17.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:16 smithi033 ceph-mon[31973]: pgmap v177: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-19T02:59:17.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:16 smithi033 ceph-mon[31973]: scrub queued for path: / 2024-03-19T02:59:17.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:16 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T02:59:17.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:16 smithi033 ceph-mon[31973]: scrub summary: active paths [/] 2024-03-19T02:59:17.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:16 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T02:59:18.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:18 smithi096 ceph-mon[35946]: pgmap v178: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-19T02:59:18.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:18 smithi096 ceph-mon[35946]: scrub summary: idle 2024-03-19T02:59:19.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:18 smithi155 ceph-mon[35804]: pgmap v178: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-19T02:59:19.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:18 smithi155 ceph-mon[35804]: scrub summary: idle 2024-03-19T02:59:19.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:18 smithi033 ceph-mon[31973]: pgmap v178: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-19T02:59:19.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:18 smithi033 ceph-mon[31973]: scrub summary: idle 2024-03-19T02:59:20.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:20 smithi096 ceph-mon[35946]: pgmap v179: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-19T02:59:21.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:20 smithi155 ceph-mon[35804]: pgmap v179: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-19T02:59:21.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:20 smithi033 ceph-mon[31973]: pgmap v179: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-19T02:59:22.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:22 smithi096 ceph-mon[35946]: pgmap v180: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:59:23.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:22 smithi155 ceph-mon[35804]: pgmap v180: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:59:23.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:22 smithi033 ceph-mon[31973]: pgmap v180: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:59:24.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:24 smithi096 ceph-mon[35946]: pgmap v181: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-03-19T02:59:25.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:24 smithi155 ceph-mon[35804]: pgmap v181: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-03-19T02:59:25.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:24 smithi033 ceph-mon[31973]: pgmap v181: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-03-19T02:59:26.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:26 smithi096 ceph-mon[35946]: pgmap v182: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:59:27.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:26 smithi155 ceph-mon[35804]: pgmap v182: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:59:27.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:26 smithi033 ceph-mon[31973]: pgmap v182: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:59:28.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:28 smithi096 ceph-mon[35946]: pgmap v183: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:59:29.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:28 smithi155 ceph-mon[35804]: pgmap v183: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:59:29.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:28 smithi033 ceph-mon[31973]: pgmap v183: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:59:30.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:30 smithi096 ceph-mon[35946]: pgmap v184: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:59:31.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:30 smithi155 ceph-mon[35804]: pgmap v184: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:59:31.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:30 smithi033 ceph-mon[31973]: pgmap v184: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:59:32.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:32 smithi096 ceph-mon[35946]: pgmap v185: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:59:33.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:32 smithi155 ceph-mon[35804]: pgmap v185: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:59:33.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:32 smithi033 ceph-mon[31973]: pgmap v185: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:59:34.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:34 smithi096 ceph-mon[35946]: pgmap v186: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:59:35.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:34 smithi155 ceph-mon[35804]: pgmap v186: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:59:35.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:34 smithi033 ceph-mon[31973]: pgmap v186: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:59:36.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:36 smithi096 ceph-mon[35946]: pgmap v187: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:59:37.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:36 smithi155 ceph-mon[35804]: pgmap v187: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:59:37.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:36 smithi033 ceph-mon[31973]: pgmap v187: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:59:37.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:37 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:59:38.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:37 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:59:38.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:37 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T02:59:38.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:38 smithi096 ceph-mon[35946]: pgmap v188: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:59:38.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:38 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:59:38.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:38 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:59:38.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:38 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:59:39.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:38 smithi155 ceph-mon[35804]: pgmap v188: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:59:39.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:38 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:59:39.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:38 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:59:39.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:38 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:59:39.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:38 smithi033 ceph-mon[31973]: pgmap v188: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:59:39.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:38 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T02:59:39.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:38 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T02:59:39.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:38 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T02:59:40.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:40 smithi096 ceph-mon[35946]: pgmap v189: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:59:41.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:40 smithi155 ceph-mon[35804]: pgmap v189: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:59:41.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:40 smithi033 ceph-mon[31973]: pgmap v189: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:59:42.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:42 smithi096 ceph-mon[35946]: pgmap v190: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:59:43.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:42 smithi155 ceph-mon[35804]: pgmap v190: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:59:43.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:42 smithi033 ceph-mon[31973]: pgmap v190: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:59:44.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:44 smithi096 ceph-mon[35946]: pgmap v191: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:59:45.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:44 smithi155 ceph-mon[35804]: pgmap v191: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:59:45.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:44 smithi033 ceph-mon[31973]: pgmap v191: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T02:59:45.627 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:59:45.628 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:59:45.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.746+0000 7fe538acc640 1 -- 172.21.15.33:0/2595043723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe534101100 msgr2=0x7fe53410d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:45.747 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.746+0000 7fe538acc640 1 --2- 172.21.15.33:0/2595043723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe534101100 0x7fe53410d610 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fe52c009cd0 tx=0x7fe52c02c0f0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.746+0000 7fe538acc640 1 -- 172.21.15.33:0/2595043723 shutdown_connections 2024-03-19T02:59:45.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.746+0000 7fe538acc640 1 --2- 172.21.15.33:0/2595043723 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe534101100 0x7fe53410d610 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.746+0000 7fe538acc640 1 --2- 172.21.15.33:0/2595043723 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe534100740 0x7fe534100bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.746+0000 7fe538acc640 1 --2- 172.21.15.33:0/2595043723 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe534106750 0x7fe534106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.746+0000 7fe538acc640 1 -- 172.21.15.33:0/2595043723 >> 172.21.15.33:0/2595043723 conn(0x7fe534077880 msgr2=0x7fe534077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:45.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.746+0000 7fe538acc640 1 -- 172.21.15.33:0/2595043723 shutdown_connections 2024-03-19T02:59:45.748 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.746+0000 7fe538acc640 1 -- 172.21.15.33:0/2595043723 wait complete. 2024-03-19T02:59:45.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.747+0000 7fe538acc640 1 Processor -- start 2024-03-19T02:59:45.750 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.747+0000 7fe538acc640 1 -- start start 2024-03-19T02:59:45.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.747+0000 7fe538acc640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe534100740 0x7fe53419a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:45.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.747+0000 7fe538acc640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe534101100 0x7fe53419a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:45.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.747+0000 7fe538acc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe534106750 0x7fe53419eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:45.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.747+0000 7fe538acc640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe534110a10 con 0x7fe534106750 2024-03-19T02:59:45.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.747+0000 7fe538acc640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe534110890 con 0x7fe534100740 2024-03-19T02:59:45.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.747+0000 7fe538acc640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe534110b90 con 0x7fe534101100 2024-03-19T02:59:45.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.747+0000 7fe532d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe534106750 0x7fe53419eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:45.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.747+0000 7fe532d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe534106750 0x7fe53419eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36820/0 (socket says 172.21.15.33:36820) 2024-03-19T02:59:45.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.747+0000 7fe532d9d640 1 -- 172.21.15.33:0/2348345798 learned_addr learned my addr 172.21.15.33:0/2348345798 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:45.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.747+0000 7fe53259c640 1 --2- 172.21.15.33:0/2348345798 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe534100740 0x7fe53419a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:45.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.747+0000 7fe52bfff640 1 --2- 172.21.15.33:0/2348345798 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe534101100 0x7fe53419a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:45.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.747+0000 7fe532d9d640 1 -- 172.21.15.33:0/2348345798 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe534101100 msgr2=0x7fe53419a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:45.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.747+0000 7fe532d9d640 1 --2- 172.21.15.33:0/2348345798 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe534101100 0x7fe53419a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.747+0000 7fe532d9d640 1 -- 172.21.15.33:0/2348345798 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe534100740 msgr2=0x7fe53419a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:45.751 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.747+0000 7fe532d9d640 1 --2- 172.21.15.33:0/2348345798 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe534100740 0x7fe53419a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.747+0000 7fe532d9d640 1 -- 172.21.15.33:0/2348345798 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe53419f210 con 0x7fe534106750 2024-03-19T02:59:45.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.747+0000 7fe53259c640 1 --2- 172.21.15.33:0/2348345798 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe534100740 0x7fe53419a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:45.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.748+0000 7fe532d9d640 1 --2- 172.21.15.33:0/2348345798 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe534106750 0x7fe53419eae0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fe52c004af0 tx=0x7fe52c035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:45.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.748+0000 7fe52b7fe640 1 -- 172.21.15.33:0/2348345798 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe52c039040 con 0x7fe534106750 2024-03-19T02:59:45.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.748+0000 7fe52b7fe640 1 -- 172.21.15.33:0/2348345798 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe52c044070 con 0x7fe534106750 2024-03-19T02:59:45.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.748+0000 7fe538acc640 1 -- 172.21.15.33:0/2348345798 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe53419f4a0 con 0x7fe534106750 2024-03-19T02:59:45.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.748+0000 7fe52b7fe640 1 -- 172.21.15.33:0/2348345798 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe52c0042a0 con 0x7fe534106750 2024-03-19T02:59:45.752 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.748+0000 7fe538acc640 1 -- 172.21.15.33:0/2348345798 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5341a6d90 con 0x7fe534106750 2024-03-19T02:59:45.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.750+0000 7fe52b7fe640 1 -- 172.21.15.33:0/2348345798 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fe52c003970 con 0x7fe534106750 2024-03-19T02:59:45.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.750+0000 7fe53b78a740 1 Processor -- start 2024-03-19T02:59:45.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.751+0000 7fe53b78a740 1 -- start start 2024-03-19T02:59:45.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.751+0000 7fe52b7fe640 1 --2- 172.21.15.33:0/2348345798 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe5200a2970 0x7fe5200a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:45.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.751+0000 7fe53b78a740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0619aba30 0x55c0619abe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:45.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.751+0000 7fe53b78a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0619ac350 0x55c0619b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:45.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.751+0000 7fe53b78a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0619b4f30 0x55c0619b7320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:45.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.751+0000 7fe53b78a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c06190f500 con 0x55c0619b4f30 2024-03-19T02:59:45.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.751+0000 7fe53b78a740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c0618b7be0 con 0x55c0619aba30 2024-03-19T02:59:45.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.751+0000 7fe53b78a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c06190b5a0 con 0x55c0619ac350 2024-03-19T02:59:45.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.751+0000 7fe52bfff640 1 --2- 172.21.15.33:0/2348345798 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe5200a2970 0x7fe5200a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:45.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.751+0000 7fe53259c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0619aba30 0x55c0619abe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:45.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.751+0000 7fe52b7fe640 1 -- 172.21.15.33:0/2348345798 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fe52c0f1430 con 0x7fe534106750 2024-03-19T02:59:45.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.751+0000 7fe53259c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0619aba30 0x55c0619abe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:33772/0 (socket says 172.21.15.33:33772) 2024-03-19T02:59:45.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.751+0000 7fe532d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0619b4f30 0x55c0619b7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:45.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.751+0000 7fe53259c640 1 -- 172.21.15.33:0/2484869263 learned_addr learned my addr 172.21.15.33:0/2484869263 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:45.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.751+0000 7fe52bfff640 1 --2- 172.21.15.33:0/2484869263 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0619ac350 0x55c0619b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:45.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.751+0000 7fe53259c640 1 -- 172.21.15.33:0/2484869263 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0619ac350 msgr2=0x55c0619b49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:45.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.751+0000 7fe53259c640 1 --2- 172.21.15.33:0/2484869263 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0619ac350 0x55c0619b49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.751+0000 7fe53259c640 1 -- 172.21.15.33:0/2484869263 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0619b4f30 msgr2=0x55c0619b7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:45.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.751+0000 7fe53259c640 1 --2- 172.21.15.33:0/2484869263 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0619b4f30 0x55c0619b7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.751+0000 7fe53259c640 1 -- 172.21.15.33:0/2484869263 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0618d9000 con 0x55c0619aba30 2024-03-19T02:59:45.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.751+0000 7fe532d9d640 1 --2- 172.21.15.33:0/2484869263 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0619b4f30 0x55c0619b7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:45.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.751+0000 7fe52bfff640 1 --2- 172.21.15.33:0/2348345798 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe5200a2970 0x7fe5200a4e30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe5340781f0 tx=0x7fe51c009ad0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:45.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.751+0000 7fe53259c640 1 --2- 172.21.15.33:0/2484869263 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0619aba30 0x55c0619abe10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe5240114c0 tx=0x7fe5240118c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69f3df2aebb3385b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:45.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.752+0000 7fe528ff9640 1 -- 172.21.15.33:0/2484869263 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe524004020 con 0x55c0619aba30 2024-03-19T02:59:45.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.752+0000 7fe528ff9640 1 -- 172.21.15.33:0/2484869263 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5240041c0 con 0x55c0619aba30 2024-03-19T02:59:45.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.752+0000 7fe528ff9640 1 -- 172.21.15.33:0/2484869263 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe52400f780 con 0x55c0619aba30 2024-03-19T02:59:45.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.752+0000 7fe53b78a740 1 -- 172.21.15.33:0/2484869263 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0619aba30 msgr2=0x55c0619abe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:45.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.752+0000 7fe53b78a740 1 --2- 172.21.15.33:0/2484869263 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0619aba30 0x55c0619abe10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe5240114c0 tx=0x7fe5240118c0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.757 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.752+0000 7fe53b78a740 1 -- 172.21.15.33:0/2484869263 shutdown_connections 2024-03-19T02:59:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.752+0000 7fe53b78a740 1 --2- 172.21.15.33:0/2484869263 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0619ac350 0x55c0619b49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.752+0000 7fe53b78a740 1 --2- 172.21.15.33:0/2484869263 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0619b4f30 0x55c0619b7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.752+0000 7fe53b78a740 1 --2- 172.21.15.33:0/2484869263 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0619aba30 0x55c0619abe10 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.752+0000 7fe53b78a740 1 -- 172.21.15.33:0/2484869263 >> 172.21.15.33:0/2484869263 conn(0x55c0619a6160 msgr2=0x55c0619a6540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.753+0000 7fe53b78a740 1 -- 172.21.15.33:0/2484869263 shutdown_connections 2024-03-19T02:59:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.753+0000 7fe53b78a740 1 -- 172.21.15.33:0/2484869263 wait complete. 2024-03-19T02:59:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.753+0000 7fe53b78a740 1 -- start start 2024-03-19T02:59:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.753+0000 7fe53b78a740 1 Processor -- start 2024-03-19T02:59:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.753+0000 7fe53b78a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0619aba30 0x55c0619b1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.753+0000 7fe53b78a740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0619ac350 0x55c0619b2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:45.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.753+0000 7fe53b78a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0619b4f30 0x55c0619b28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:45.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.753+0000 7fe53b78a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c0619b7d20 con 0x55c0619b4f30 2024-03-19T02:59:45.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.753+0000 7fe53b78a740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c0618cb160 con 0x55c0619ac350 2024-03-19T02:59:45.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.753+0000 7fe53b78a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c0619b7ea0 con 0x55c0619aba30 2024-03-19T02:59:45.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.753+0000 7fe53b78a740 10 client.0 _ll_register_callbacks cb 0x55c061907a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:59:45.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.753+0000 7fe532d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0619b4f30 0x55c0619b28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:45.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.753+0000 7fe532d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0619b4f30 0x55c0619b28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36840/0 (socket says 172.21.15.33:36840) 2024-03-19T02:59:45.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.753+0000 7fe52bfff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0619ac350 0x55c0619b2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:45.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.753+0000 7fe532d9d640 1 -- 172.21.15.33:0/3752877842 learned_addr learned my addr 172.21.15.33:0/3752877842 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:45.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.753+0000 7fe53259c640 1 --2- 172.21.15.33:0/3752877842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0619aba30 0x55c0619b1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:45.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.754+0000 7fe52bfff640 1 -- 172.21.15.33:0/3752877842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0619aba30 msgr2=0x55c0619b1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:45.762 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.754+0000 7fe52bfff640 1 --2- 172.21.15.33:0/3752877842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0619aba30 0x55c0619b1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.754+0000 7fe52bfff640 1 -- 172.21.15.33:0/3752877842 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0619b4f30 msgr2=0x55c0619b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:45.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.754+0000 7fe52bfff640 1 --2- 172.21.15.33:0/3752877842 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0619b4f30 0x55c0619b28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.754+0000 7fe52bfff640 1 -- 172.21.15.33:0/3752877842 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0616ed7d0 con 0x55c0619ac350 2024-03-19T02:59:45.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.754+0000 7fe532d9d640 1 --2- 172.21.15.33:0/3752877842 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0619b4f30 0x55c0619b28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:45.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.754+0000 7fe52bfff640 1 --2- 172.21.15.33:0/3752877842 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0619ac350 0x55c0619b2360 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x55c0619ab500 tx=0x7fe51c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7becd43e0312db48 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:45.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.754+0000 7fe519ffb640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T02:59:45.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.754+0000 7fe519ffb640 1 -- 172.21.15.33:0/3752877842 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe51c012070 con 0x55c0619ac350 2024-03-19T02:59:45.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.754+0000 7fe519ffb640 1 -- 172.21.15.33:0/3752877842 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fe51c01c040 con 0x55c0619ac350 2024-03-19T02:59:45.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.754+0000 7fe53b78a740 1 -- 172.21.15.33:0/3752877842 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5240041c0 con 0x55c0619ac350 2024-03-19T02:59:45.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.754+0000 7fe519ffb640 1 -- 172.21.15.33:0/3752877842 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe51c031050 con 0x55c0619ac350 2024-03-19T02:59:45.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.754+0000 7fe53b78a740 10 client.26786 fetch_fsmap learned FSMap version 25 2024-03-19T02:59:45.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.754+0000 7fe53b78a740 1 -- 172.21.15.33:0/3752877842 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c0616233f0 con 0x55c0619ac350 2024-03-19T02:59:45.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.754+0000 7fe519ffb640 1 -- 172.21.15.33:0/3752877842 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5240041c0 con 0x55c0619ac350 2024-03-19T02:59:45.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.755+0000 7fe519ffb640 1 -- 172.21.15.33:0/3752877842 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fe51c020050 con 0x55c0619ac350 2024-03-19T02:59:45.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.755+0000 7fe53b78a740 10 client.26786 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:59:45.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.755+0000 7fe53b78a740 10 client.26786 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:59:45.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.755+0000 7fe53b78a740 20 client.26786 populate_metadata read hostname 'smithi033' 2024-03-19T02:59:45.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.755+0000 7fe53b78a740 1 --2- 172.21.15.33:0/3752877842 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55c0619c51a0 0x55c0619c75c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:45.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.755+0000 7fe53b78a740 4 client.26786 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:59:45.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.755+0000 7fe53b78a740 1 -- 172.21.15.33:0/3752877842 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c06191ca40 con 0x55c0619c51a0 2024-03-19T02:59:45.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.755+0000 7fe53259c640 1 --2- 172.21.15.33:0/3752877842 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55c0619c51a0 0x55c0619c75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:45.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.755+0000 7fe53259c640 1 --2- 172.21.15.33:0/3752877842 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55c0619c51a0 0x55c0619c75c0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a770f925b628ee48 server_cookie=7ed2b6c9fc73feb9 in_seq=0 out_seq=0 2024-03-19T02:59:45.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.755+0000 7fe519ffb640 10 client.26786 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:59:45.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.756+0000 7fe519ffb640 1 -- 172.21.15.33:0/3752877842 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c06191ca40 con 0x55c0619c51a0 2024-03-19T02:59:45.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.756+0000 7fe519ffb640 10 client.26786 handle_command_reply: tid=0 2024-03-19T02:59:45.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.756+0000 7fe53b78a740 1 client.26786 shutdown 2024-03-19T02:59:45.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.756+0000 7fe53b78a740 1 -- 172.21.15.33:0/3752877842 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0619ac350 msgr2=0x55c0619b2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:45.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.756+0000 7fe53b78a740 1 --2- 172.21.15.33:0/3752877842 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0619ac350 0x55c0619b2360 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x55c0619ab500 tx=0x7fe51c004730 comp rx=0 tx=0).stop 2024-03-19T02:59:45.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.756+0000 7fe53b78a740 1 -- 172.21.15.33:0/3752877842 shutdown_connections 2024-03-19T02:59:45.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.756+0000 7fe53b78a740 1 --2- 172.21.15.33:0/3752877842 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55c0619c51a0 0x55c0619c75c0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.756+0000 7fe53b78a740 1 --2- 172.21.15.33:0/3752877842 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0619b4f30 0x55c0619b28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.756+0000 7fe519ffb640 0 client.26786 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:59:45.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.756+0000 7fe53b78a740 1 --2- 172.21.15.33:0/3752877842 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0619ac350 0x55c0619b2360 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.756+0000 7fe53b78a740 1 --2- 172.21.15.33:0/3752877842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0619aba30 0x55c0619b1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.756+0000 7fe53b78a740 1 -- 172.21.15.33:0/3752877842 >> 172.21.15.33:0/3752877842 conn(0x55c0619a6160 msgr2=0x55c0619a7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:45.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.756+0000 7fe53b78a740 1 -- 172.21.15.33:0/3752877842 shutdown_connections 2024-03-19T02:59:45.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.756+0000 7fe53b78a740 1 -- 172.21.15.33:0/3752877842 wait complete. 2024-03-19T02:59:45.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.756+0000 7fe53b78a740 20 client.26786 trim_cache size 0 max 16384 2024-03-19T02:59:45.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.774+0000 7fe53b78a740 1 Processor -- start 2024-03-19T02:59:45.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.774+0000 7fe53b78a740 1 -- start start 2024-03-19T02:59:45.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.774+0000 7fe53b78a740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0619aba30 0x55c0619b4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:45.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.774+0000 7fe53b78a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0619ac350 0x55c0619a6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:45.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.774+0000 7fe53b78a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0619b4f30 0x55c0619c0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:45.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.774+0000 7fe53b78a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe5341b35c0 con 0x55c0619b4f30 2024-03-19T02:59:45.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.774+0000 7fe53b78a740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe5341b0e90 con 0x55c0619aba30 2024-03-19T02:59:45.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.774+0000 7fe53b78a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe5341aecc0 con 0x55c0619ac350 2024-03-19T02:59:45.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.774+0000 7fe532d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0619b4f30 0x55c0619c0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:45.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.774+0000 7fe53259c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0619aba30 0x55c0619b4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:45.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.774+0000 7fe532d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0619b4f30 0x55c0619c0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36848/0 (socket says 172.21.15.33:36848) 2024-03-19T02:59:45.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.774+0000 7fe532d9d640 1 -- 172.21.15.33:0/3594401754 learned_addr learned my addr 172.21.15.33:0/3594401754 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:45.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.774+0000 7fe52bfff640 1 --2- 172.21.15.33:0/3594401754 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0619ac350 0x55c0619a6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:45.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.774+0000 7fe53259c640 1 -- 172.21.15.33:0/3594401754 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0619ac350 msgr2=0x55c0619a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:45.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.774+0000 7fe53259c640 1 --2- 172.21.15.33:0/3594401754 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0619ac350 0x55c0619a6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.774+0000 7fe53259c640 1 -- 172.21.15.33:0/3594401754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0619b4f30 msgr2=0x55c0619c0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:45.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.774+0000 7fe53259c640 1 --2- 172.21.15.33:0/3594401754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0619b4f30 0x55c0619c0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.774+0000 7fe53259c640 1 -- 172.21.15.33:0/3594401754 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0615e30f0 con 0x55c0619aba30 2024-03-19T02:59:45.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.774+0000 7fe532d9d640 1 --2- 172.21.15.33:0/3594401754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0619b4f30 0x55c0619c0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:45.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.775+0000 7fe53259c640 1 --2- 172.21.15.33:0/3594401754 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0619aba30 0x55c0619b4550 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe5240096e0 tx=0x7fe524005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a12ad667823ec4d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:45.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.775+0000 7fe51bfff640 1 -- 172.21.15.33:0/3594401754 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe524004150 con 0x55c0619aba30 2024-03-19T02:59:45.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.775+0000 7fe51bfff640 1 -- 172.21.15.33:0/3594401754 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5240054f0 con 0x55c0619aba30 2024-03-19T02:59:45.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.775+0000 7fe51bfff640 1 -- 172.21.15.33:0/3594401754 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe524003c80 con 0x55c0619aba30 2024-03-19T02:59:45.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.775+0000 7fe53b78a740 1 -- 172.21.15.33:0/3594401754 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0619aba30 msgr2=0x55c0619b4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:45.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.775+0000 7fe53b78a740 1 --2- 172.21.15.33:0/3594401754 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0619aba30 0x55c0619b4550 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe5240096e0 tx=0x7fe524005850 comp rx=0 tx=0).stop 2024-03-19T02:59:45.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.775+0000 7fe53b78a740 1 -- 172.21.15.33:0/3594401754 shutdown_connections 2024-03-19T02:59:45.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.775+0000 7fe53b78a740 1 --2- 172.21.15.33:0/3594401754 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0619ac350 0x55c0619a6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.775+0000 7fe53b78a740 1 --2- 172.21.15.33:0/3594401754 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0619b4f30 0x55c0619c0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.775+0000 7fe53b78a740 1 --2- 172.21.15.33:0/3594401754 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0619aba30 0x55c0619b4550 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.775+0000 7fe53b78a740 1 -- 172.21.15.33:0/3594401754 >> 172.21.15.33:0/3594401754 conn(0x55c0619a6160 msgr2=0x55c0619b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:45.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.775+0000 7fe53b78a740 1 -- 172.21.15.33:0/3594401754 shutdown_connections 2024-03-19T02:59:45.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.775+0000 7fe53b78a740 1 -- 172.21.15.33:0/3594401754 wait complete. 2024-03-19T02:59:45.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.776+0000 7fe53b78a740 1 -- start start 2024-03-19T02:59:45.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.776+0000 7fe53b78a740 1 Processor -- start 2024-03-19T02:59:45.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.776+0000 7fe53b78a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0619aba30 0x55c0619bcff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:45.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.776+0000 7fe53b78a740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0619ac350 0x55c0619bd530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:45.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.776+0000 7fe53b78a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0619b4f30 0x55c0619b86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:45.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.776+0000 7fe53b78a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe5341af350 con 0x55c0619aba30 2024-03-19T02:59:45.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.776+0000 7fe53b78a740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe5341b07c0 con 0x55c0619ac350 2024-03-19T02:59:45.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.776+0000 7fe53b78a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe5341afa20 con 0x55c0619b4f30 2024-03-19T02:59:45.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.776+0000 7fe53b78a740 10 client.0 _ll_register_callbacks cb 0x7fe51c013040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:59:45.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.776+0000 7fe53259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0619aba30 0x55c0619bcff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:45.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.776+0000 7fe53259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0619aba30 0x55c0619bcff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36852/0 (socket says 172.21.15.33:36852) 2024-03-19T02:59:45.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.776+0000 7fe53259c640 1 -- 172.21.15.33:0/2448829962 learned_addr learned my addr 172.21.15.33:0/2448829962 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:45.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.776+0000 7fe52bfff640 1 --2- 172.21.15.33:0/2448829962 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0619ac350 0x55c0619bd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:45.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.776+0000 7fe532d9d640 1 --2- 172.21.15.33:0/2448829962 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0619b4f30 0x55c0619b86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:45.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.776+0000 7fe53259c640 1 -- 172.21.15.33:0/2448829962 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0619b4f30 msgr2=0x55c0619b86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:45.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.776+0000 7fe53259c640 1 --2- 172.21.15.33:0/2448829962 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0619b4f30 0x55c0619b86d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.776+0000 7fe53259c640 1 -- 172.21.15.33:0/2448829962 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0619ac350 msgr2=0x55c0619bd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:45.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.776+0000 7fe53259c640 1 --2- 172.21.15.33:0/2448829962 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0619ac350 0x55c0619bd530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.776+0000 7fe53259c640 1 -- 172.21.15.33:0/2448829962 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0619b8ed0 con 0x55c0619aba30 2024-03-19T02:59:45.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.776+0000 7fe532d9d640 1 --2- 172.21.15.33:0/2448829962 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0619b4f30 0x55c0619b86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:59:45.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.777+0000 7fe53259c640 1 --2- 172.21.15.33:0/2448829962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0619aba30 0x55c0619bcff0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fe524009710 tx=0x7fe524009cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cfb399a4f1b1251 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:45.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.777+0000 7fe519ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:59:45.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.777+0000 7fe519ffb640 1 -- 172.21.15.33:0/2448829962 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe524004210 con 0x55c0619aba30 2024-03-19T02:59:45.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.777+0000 7fe519ffb640 1 -- 172.21.15.33:0/2448829962 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fe524031040 con 0x55c0619aba30 2024-03-19T02:59:45.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.777+0000 7fe53b78a740 1 -- 172.21.15.33:0/2448829962 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5240054f0 con 0x55c0619aba30 2024-03-19T02:59:45.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.777+0000 7fe519ffb640 1 -- 172.21.15.33:0/2448829962 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe524011250 con 0x55c0619aba30 2024-03-19T02:59:45.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.777+0000 7fe53b78a740 10 client.16968 fetch_fsmap learned FSMap version 25 2024-03-19T02:59:45.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.777+0000 7fe53b78a740 1 -- 172.21.15.33:0/2448829962 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c061a2b100 con 0x55c0619aba30 2024-03-19T02:59:45.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.777+0000 7fe519ffb640 1 -- 172.21.15.33:0/2448829962 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5240054f0 con 0x55c0619aba30 2024-03-19T02:59:45.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.777+0000 7fe519ffb640 1 -- 172.21.15.33:0/2448829962 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fe52403c030 con 0x55c0619aba30 2024-03-19T02:59:45.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.777+0000 7fe53b78a740 10 client.16968 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:59:45.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.777+0000 7fe53b78a740 10 client.16968 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:59:45.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.777+0000 7fe53b78a740 20 client.16968 populate_metadata read hostname 'smithi033' 2024-03-19T02:59:45.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.777+0000 7fe53b78a740 1 --2- 172.21.15.33:0/2448829962 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55c061a2b6d0 0x55c061a2dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:45.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.777+0000 7fe53b78a740 4 client.16968 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:59:45.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.777+0000 7fe53b78a740 1 -- 172.21.15.33:0/2448829962 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c061916d90 con 0x55c061a2b6d0 2024-03-19T02:59:45.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.778+0000 7fe52bfff640 1 --2- 172.21.15.33:0/2448829962 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55c061a2b6d0 0x55c061a2dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:45.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.778+0000 7fe519ffb640 10 client.16968 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:59:45.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.778+0000 7fe52bfff640 1 --2- 172.21.15.33:0/2448829962 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55c061a2b6d0 0x55c061a2dac0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b835dbf84f24897a server_cookie=f71def5bc0264f5 in_seq=0 out_seq=0 2024-03-19T02:59:45.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.778+0000 7fe519ffb640 1 -- 172.21.15.33:0/2448829962 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c061916d90 con 0x55c061a2b6d0 2024-03-19T02:59:45.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.778+0000 7fe519ffb640 10 client.16968 handle_command_reply: tid=0 2024-03-19T02:59:45.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.778+0000 7fe53b78a740 1 client.16968 shutdown 2024-03-19T02:59:45.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.778+0000 7fe53b78a740 1 -- 172.21.15.33:0/2448829962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0619aba30 msgr2=0x55c0619bcff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:45.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.778+0000 7fe53b78a740 1 --2- 172.21.15.33:0/2448829962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0619aba30 0x55c0619bcff0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fe524009710 tx=0x7fe524009cc0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.778+0000 7fe53b78a740 1 -- 172.21.15.33:0/2448829962 shutdown_connections 2024-03-19T02:59:45.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.778+0000 7fe53b78a740 1 --2- 172.21.15.33:0/2448829962 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55c061a2b6d0 0x55c061a2dac0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.778+0000 7fe53b78a740 1 --2- 172.21.15.33:0/2448829962 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c0619b4f30 0x55c0619b86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.778+0000 7fe53b78a740 1 --2- 172.21.15.33:0/2448829962 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c0619ac350 0x55c0619bd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.778+0000 7fe53b78a740 1 --2- 172.21.15.33:0/2448829962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c0619aba30 0x55c0619bcff0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.778+0000 7fe519ffb640 0 client.16968 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:59:45.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.778+0000 7fe53b78a740 1 -- 172.21.15.33:0/2448829962 >> 172.21.15.33:0/2448829962 conn(0x55c0619a6160 msgr2=0x55c0619b2670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:45.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.778+0000 7fe53b78a740 1 -- 172.21.15.33:0/2448829962 shutdown_connections 2024-03-19T02:59:45.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.778+0000 7fe53b78a740 1 -- 172.21.15.33:0/2448829962 wait complete. 2024-03-19T02:59:45.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.778+0000 7fe53b78a740 20 client.16968 trim_cache size 0 max 16384 2024-03-19T02:59:45.787 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:59:45.787 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-03-19T02:59:45.787 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-03-19T02:59:45.787 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:59:45.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.779+0000 7fe538acc640 1 -- 172.21.15.33:0/2348345798 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe5200a2970 msgr2=0x7fe5200a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:45.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.779+0000 7fe538acc640 1 --2- 172.21.15.33:0/2348345798 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe5200a2970 0x7fe5200a4e30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe5340781f0 tx=0x7fe51c009ad0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.779+0000 7fe538acc640 1 -- 172.21.15.33:0/2348345798 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe534106750 msgr2=0x7fe53419eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:45.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.779+0000 7fe538acc640 1 --2- 172.21.15.33:0/2348345798 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe534106750 0x7fe53419eae0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fe52c004af0 tx=0x7fe52c035ce0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.779+0000 7fe538acc640 1 -- 172.21.15.33:0/2348345798 shutdown_connections 2024-03-19T02:59:45.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.779+0000 7fe538acc640 1 --2- 172.21.15.33:0/2348345798 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe5200a2970 0x7fe5200a4e30 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.779+0000 7fe538acc640 1 --2- 172.21.15.33:0/2348345798 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe534101100 0x7fe53419a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.779+0000 7fe538acc640 1 --2- 172.21.15.33:0/2348345798 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe534106750 0x7fe53419eae0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.779+0000 7fe538acc640 1 --2- 172.21.15.33:0/2348345798 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe534100740 0x7fe53419a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.779+0000 7fe538acc640 1 -- 172.21.15.33:0/2348345798 >> 172.21.15.33:0/2348345798 conn(0x7fe534077880 msgr2=0x7fe53410be00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:45.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.779+0000 7fe538acc640 1 -- 172.21.15.33:0/2348345798 shutdown_connections 2024-03-19T02:59:45.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.779+0000 7fe538acc640 1 -- 172.21.15.33:0/2348345798 wait complete. 2024-03-19T02:59:45.794 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T02:59:45.794 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T02:59:45.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.897+0000 7fdd4e9b9640 1 -- 172.21.15.33:0/624139336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd481017d0 msgr2=0x7fdd48101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:45.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.897+0000 7fdd4e9b9640 1 --2- 172.21.15.33:0/624139336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd481017d0 0x7fdd48101c50 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c009860 tx=0x7fdd3c02bb70 comp rx=0 tx=0).stop 2024-03-19T02:59:45.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.897+0000 7fdd4e9b9640 1 -- 172.21.15.33:0/624139336 shutdown_connections 2024-03-19T02:59:45.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.897+0000 7fdd4e9b9640 1 --2- 172.21.15.33:0/624139336 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdd48102190 0x7fdd4810e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.897+0000 7fdd4e9b9640 1 --2- 172.21.15.33:0/624139336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd481017d0 0x7fdd48101c50 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.897+0000 7fdd4e9b9640 1 --2- 172.21.15.33:0/624139336 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdd481077e0 0x7fdd48107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.897+0000 7fdd4e9b9640 1 -- 172.21.15.33:0/624139336 >> 172.21.15.33:0/624139336 conn(0x7fdd480fd2c0 msgr2=0x7fdd480ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:45.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.897+0000 7fdd4e9b9640 1 -- 172.21.15.33:0/624139336 shutdown_connections 2024-03-19T02:59:45.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.897+0000 7fdd4e9b9640 1 -- 172.21.15.33:0/624139336 wait complete. 2024-03-19T02:59:45.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.897+0000 7fdd4e9b9640 1 Processor -- start 2024-03-19T02:59:45.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.897+0000 7fdd4e9b9640 1 -- start start 2024-03-19T02:59:45.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.898+0000 7fdd4e9b9640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd481017d0 0x7fdd48198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:45.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.898+0000 7fdd4e9b9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdd48102190 0x7fdd48198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:45.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.898+0000 7fdd4e9b9640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdd481077e0 0x7fdd4819c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:45.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.898+0000 7fdd4e9b9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fdd48111aa0 con 0x7fdd48102190 2024-03-19T02:59:45.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.898+0000 7fdd4e9b9640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdd48111920 con 0x7fdd481017d0 2024-03-19T02:59:45.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.898+0000 7fdd4e9b9640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fdd48111c20 con 0x7fdd481077e0 2024-03-19T02:59:45.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.898+0000 7fdd47fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd481017d0 0x7fdd48198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:45.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.898+0000 7fdd477fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdd48102190 0x7fdd48198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:45.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.898+0000 7fdd477fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdd48102190 0x7fdd48198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36862/0 (socket says 172.21.15.33:36862) 2024-03-19T02:59:45.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.898+0000 7fdd47fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd481017d0 0x7fdd48198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:33834/0 (socket says 172.21.15.33:33834) 2024-03-19T02:59:45.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.898+0000 7fdd477fe640 1 -- 172.21.15.33:0/1206163101 learned_addr learned my addr 172.21.15.33:0/1206163101 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:45.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.898+0000 7fdd4cf56640 1 --2- 172.21.15.33:0/1206163101 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdd481077e0 0x7fdd4819c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:45.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.898+0000 7fdd477fe640 1 -- 172.21.15.33:0/1206163101 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdd481077e0 msgr2=0x7fdd4819c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:45.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.898+0000 7fdd477fe640 1 --2- 172.21.15.33:0/1206163101 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdd481077e0 0x7fdd4819c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.898+0000 7fdd477fe640 1 -- 172.21.15.33:0/1206163101 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd481017d0 msgr2=0x7fdd48198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:45.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.898+0000 7fdd477fe640 1 --2- 172.21.15.33:0/1206163101 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd481017d0 0x7fdd48198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:45.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.898+0000 7fdd477fe640 1 -- 172.21.15.33:0/1206163101 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd4819d000 con 0x7fdd48102190 2024-03-19T02:59:45.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.898+0000 7fdd47fff640 1 --2- 172.21.15.33:0/1206163101 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd481017d0 0x7fdd48198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:45.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.898+0000 7fdd477fe640 1 --2- 172.21.15.33:0/1206163101 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdd48102190 0x7fdd48198550 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c037040 tx=0x7fdd3c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:45.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.898+0000 7fdd457fa640 1 -- 172.21.15.33:0/1206163101 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c041020 con 0x7fdd48102190 2024-03-19T02:59:45.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.898+0000 7fdd4e9b9640 1 -- 172.21.15.33:0/1206163101 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd4819d290 con 0x7fdd48102190 2024-03-19T02:59:45.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.898+0000 7fdd457fa640 1 -- 172.21.15.33:0/1206163101 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd3c02c9c0 con 0x7fdd48102190 2024-03-19T02:59:45.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.898+0000 7fdd457fa640 1 -- 172.21.15.33:0/1206163101 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c039040 con 0x7fdd48102190 2024-03-19T02:59:45.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.898+0000 7fdd4e9b9640 1 -- 172.21.15.33:0/1206163101 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd481a4b80 con 0x7fdd48102190 2024-03-19T02:59:45.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.900+0000 7fdd4e9b9640 1 -- 172.21.15.33:0/1206163101 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd48068a60 con 0x7fdd48102190 2024-03-19T02:59:45.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.900+0000 7fdd457fa640 1 -- 172.21.15.33:0/1206163101 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fdd3c044020 con 0x7fdd48102190 2024-03-19T02:59:45.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.901+0000 7fdd457fa640 1 --2- 172.21.15.33:0/1206163101 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fdd300a2970 0x7fdd300a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:45.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.901+0000 7fdd457fa640 1 -- 172.21.15.33:0/1206163101 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7fdd3c0f0890 con 0x7fdd48102190 2024-03-19T02:59:45.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.901+0000 7fdd47fff640 1 --2- 172.21.15.33:0/1206163101 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fdd300a2970 0x7fdd300a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:45.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.901+0000 7fdd47fff640 1 --2- 172.21.15.33:0/1206163101 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fdd300a2970 0x7fdd300a4e30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdd38009f40 tx=0x7fdd38005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:45.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:45.905+0000 7fdd457fa640 1 -- 172.21.15.33:0/1206163101 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdd3c04b020 con 0x7fdd48102190 2024-03-19T02:59:46.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.128+0000 7fdd4e9b9640 1 -- 172.21.15.33:0/1206163101 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd48109d40 con 0x7fdd48102190 2024-03-19T02:59:46.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.128+0000 7fdd457fa640 1 -- 172.21.15.33:0/1206163101 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11867 (secure 0 0 0) 0x7fdd3c0bd390 con 0x7fdd48102190 2024-03-19T02:59:46.130 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T02:59:46.130 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:50:52.201798+0000","modified":"2024-03-19T02:53:11.748697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24488,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/3731232169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":3731232169},{"type":"v1","addr":"172.21.15.96:6839","nonce":3731232169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24457,24488,24509]},"id":1}]} 2024-03-19T02:59:46.130 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-03-19T02:59:46.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.130+0000 7fdd4e9b9640 1 -- 172.21.15.33:0/1206163101 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fdd300a2970 msgr2=0x7fdd300a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:46.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.130+0000 7fdd4e9b9640 1 --2- 172.21.15.33:0/1206163101 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fdd300a2970 0x7fdd300a4e30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdd38009f40 tx=0x7fdd38005f90 comp rx=0 tx=0).stop 2024-03-19T02:59:46.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.131+0000 7fdd4e9b9640 1 -- 172.21.15.33:0/1206163101 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdd48102190 msgr2=0x7fdd48198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:46.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.131+0000 7fdd4e9b9640 1 --2- 172.21.15.33:0/1206163101 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdd48102190 0x7fdd48198550 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c037040 tx=0x7fdd3c004040 comp rx=0 tx=0).stop 2024-03-19T02:59:46.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.131+0000 7fdd4e9b9640 1 -- 172.21.15.33:0/1206163101 shutdown_connections 2024-03-19T02:59:46.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.131+0000 7fdd4e9b9640 1 --2- 172.21.15.33:0/1206163101 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fdd300a2970 0x7fdd300a4e30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.131+0000 7fdd4e9b9640 1 --2- 172.21.15.33:0/1206163101 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdd481077e0 0x7fdd4819c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.131+0000 7fdd4e9b9640 1 --2- 172.21.15.33:0/1206163101 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fdd48102190 0x7fdd48198550 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.131+0000 7fdd4e9b9640 1 --2- 172.21.15.33:0/1206163101 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdd481017d0 0x7fdd48198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.131+0000 7fdd4e9b9640 1 -- 172.21.15.33:0/1206163101 >> 172.21.15.33:0/1206163101 conn(0x7fdd480fd2c0 msgr2=0x7fdd480fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:46.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.131+0000 7fdd4e9b9640 1 -- 172.21.15.33:0/1206163101 shutdown_connections 2024-03-19T02:59:46.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.131+0000 7fdd4e9b9640 1 -- 172.21.15.33:0/1206163101 wait complete. 2024-03-19T02:59:46.143 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T02:59:46.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.247+0000 7f27df6ff640 1 -- 172.21.15.33:0/333434941 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27d81006a0 msgr2=0x7f27d8100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:46.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.247+0000 7f27df6ff640 1 --2- 172.21.15.33:0/333434941 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27d81006a0 0x7f27d8100b20 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f27c8009930 tx=0x7f27c802bc30 comp rx=0 tx=0).stop 2024-03-19T02:59:46.250 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.247+0000 7f27df6ff640 1 -- 172.21.15.33:0/333434941 shutdown_connections 2024-03-19T02:59:46.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.247+0000 7f27df6ff640 1 --2- 172.21.15.33:0/333434941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27d8101060 0x7f27d810d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.247+0000 7f27df6ff640 1 --2- 172.21.15.33:0/333434941 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27d81006a0 0x7f27d8100b20 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.247+0000 7f27df6ff640 1 --2- 172.21.15.33:0/333434941 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27d81066b0 0x7f27d8106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.247+0000 7f27df6ff640 1 -- 172.21.15.33:0/333434941 >> 172.21.15.33:0/333434941 conn(0x7f27d80778d0 msgr2=0x7f27d8077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:46.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.247+0000 7f27df6ff640 1 -- 172.21.15.33:0/333434941 shutdown_connections 2024-03-19T02:59:46.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.247+0000 7f27df6ff640 1 -- 172.21.15.33:0/333434941 wait complete. 2024-03-19T02:59:46.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.247+0000 7f27df6ff640 1 Processor -- start 2024-03-19T02:59:46.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.247+0000 7f27df6ff640 1 -- start start 2024-03-19T02:59:46.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.249+0000 7f27df6ff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27d81006a0 0x7f27d819a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.249+0000 7f27df6ff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27d8101060 0x7f27d819a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.249+0000 7f27df6ff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27d81066b0 0x7f27d819ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.249+0000 7f27df6ff640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f27d8110930 con 0x7f27d8101060 2024-03-19T02:59:46.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.249+0000 7f27df6ff640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f27d81107b0 con 0x7f27d81066b0 2024-03-19T02:59:46.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.249+0000 7f27df6ff640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f27d8110ab0 con 0x7f27d81006a0 2024-03-19T02:59:46.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.249+0000 7f27dcc9a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27d8101060 0x7f27d819a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.251 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.249+0000 7f27dcc9a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27d8101060 0x7f27d819a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36876/0 (socket says 172.21.15.33:36876) 2024-03-19T02:59:46.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.249+0000 7f27dcc9a640 1 -- 172.21.15.33:0/2526849985 learned_addr learned my addr 172.21.15.33:0/2526849985 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:46.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.249+0000 7f27ddc9c640 1 --2- 172.21.15.33:0/2526849985 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27d81066b0 0x7f27d819ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.249+0000 7f27dd49b640 1 --2- 172.21.15.33:0/2526849985 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27d81006a0 0x7f27d819a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.249+0000 7f27dcc9a640 1 -- 172.21.15.33:0/2526849985 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27d81006a0 msgr2=0x7f27d819a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:46.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.249+0000 7f27dcc9a640 1 --2- 172.21.15.33:0/2526849985 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27d81006a0 0x7f27d819a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.249+0000 7f27dcc9a640 1 -- 172.21.15.33:0/2526849985 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27d81066b0 msgr2=0x7f27d819ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:46.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.249+0000 7f27dcc9a640 1 --2- 172.21.15.33:0/2526849985 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27d81066b0 0x7f27d819ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.249+0000 7f27dcc9a640 1 -- 172.21.15.33:0/2526849985 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27d819f150 con 0x7f27d8101060 2024-03-19T02:59:46.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.249+0000 7f27dcc9a640 1 --2- 172.21.15.33:0/2526849985 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27d8101060 0x7f27d819a6f0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f27c8009230 tx=0x7f27c802c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:46.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.249+0000 7f27ce7fc640 1 -- 172.21.15.33:0/2526849985 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c8043020 con 0x7f27d8101060 2024-03-19T02:59:46.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.249+0000 7f27df6ff640 1 -- 172.21.15.33:0/2526849985 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27d819f3e0 con 0x7f27d8101060 2024-03-19T02:59:46.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.249+0000 7f27ce7fc640 1 -- 172.21.15.33:0/2526849985 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f27c802ca70 con 0x7f27d8101060 2024-03-19T02:59:46.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.249+0000 7f27ce7fc640 1 -- 172.21.15.33:0/2526849985 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c8003a50 con 0x7f27d8101060 2024-03-19T02:59:46.252 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.249+0000 7f27df6ff640 1 -- 172.21.15.33:0/2526849985 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27d81a6cd0 con 0x7f27d8101060 2024-03-19T02:59:46.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.251+0000 7f27ce7fc640 1 -- 172.21.15.33:0/2526849985 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f27c8046020 con 0x7f27d8101060 2024-03-19T02:59:46.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.251+0000 7f27e23c8740 1 Processor -- start 2024-03-19T02:59:46.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.251+0000 7f27e23c8740 1 -- start start 2024-03-19T02:59:46.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.251+0000 7f27e23c8740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ce78e1b2d0 0x55ce78e1b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.251+0000 7f27e23c8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce78e1bbf0 0x55ce78e24290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.251+0000 7f27e23c8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ce78e247d0 0x55ce78e26bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.251+0000 7f27e23c8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ce78d7e4b0 con 0x55ce78e1bbf0 2024-03-19T02:59:46.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.251+0000 7f27e23c8740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55ce78d26d70 con 0x55ce78e1b2d0 2024-03-19T02:59:46.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.251+0000 7f27e23c8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ce78d7a550 con 0x55ce78e247d0 2024-03-19T02:59:46.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.251+0000 7f27ce7fc640 1 --2- 172.21.15.33:0/2526849985 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f27c40a2970 0x7f27c40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.251+0000 7f27dd49b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce78e1bbf0 0x55ce78e24290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.252+0000 7f27dd49b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce78e1bbf0 0x55ce78e24290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36890/0 (socket says 172.21.15.33:36890) 2024-03-19T02:59:46.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.252+0000 7f27dd49b640 1 -- 172.21.15.33:0/2257474217 learned_addr learned my addr 172.21.15.33:0/2257474217 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:46.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.252+0000 7f27dcc9a640 1 --2- 172.21.15.33:0/2257474217 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ce78e247d0 0x55ce78e26bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.252+0000 7f27ddc9c640 1 --2- 172.21.15.33:0/2257474217 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ce78e1b2d0 0x55ce78e1b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.252+0000 7f27ddc9c640 1 --2- 172.21.15.33:0/2526849985 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f27c40a2970 0x7f27c40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.252+0000 7f27ce7fc640 1 -- 172.21.15.33:0/2526849985 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f27c80f10a0 con 0x7f27d8101060 2024-03-19T02:59:46.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.252+0000 7f27dd49b640 1 -- 172.21.15.33:0/2257474217 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ce78e247d0 msgr2=0x55ce78e26bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:46.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.252+0000 7f27dd49b640 1 --2- 172.21.15.33:0/2257474217 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ce78e247d0 0x55ce78e26bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.252+0000 7f27dd49b640 1 -- 172.21.15.33:0/2257474217 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ce78e1b2d0 msgr2=0x55ce78e1b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:46.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.252+0000 7f27dd49b640 1 --2- 172.21.15.33:0/2257474217 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ce78e1b2d0 0x55ce78e1b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.252+0000 7f27dd49b640 1 -- 172.21.15.33:0/2257474217 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce78d49210 con 0x55ce78e1bbf0 2024-03-19T02:59:46.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.252+0000 7f27ddc9c640 1 --2- 172.21.15.33:0/2257474217 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ce78e1b2d0 0x55ce78e1b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:46.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.252+0000 7f27dd49b640 1 --2- 172.21.15.33:0/2257474217 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce78e1bbf0 0x55ce78e24290 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f27d4009c70 tx=0x7f27d4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=921b68e8429dae1f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:46.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.252+0000 7f27ddc9c640 1 --2- 172.21.15.33:0/2526849985 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f27c40a2970 0x7f27c40a4e30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f27d0005c40 tx=0x7f27d0009500 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:46.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.252+0000 7f27c37fe640 1 -- 172.21.15.33:0/2257474217 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d4004230 con 0x55ce78e1bbf0 2024-03-19T02:59:46.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.252+0000 7f27c37fe640 1 -- 172.21.15.33:0/2257474217 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f27d4009e20 con 0x55ce78e1bbf0 2024-03-19T02:59:46.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.252+0000 7f27c37fe640 1 -- 172.21.15.33:0/2257474217 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d4019640 con 0x55ce78e1bbf0 2024-03-19T02:59:46.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.253+0000 7f27e23c8740 1 -- 172.21.15.33:0/2257474217 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce78e1bbf0 msgr2=0x55ce78e24290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:46.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.253+0000 7f27e23c8740 1 --2- 172.21.15.33:0/2257474217 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce78e1bbf0 0x55ce78e24290 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f27d4009c70 tx=0x7f27d4011810 comp rx=0 tx=0).stop 2024-03-19T02:59:46.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.253+0000 7f27e23c8740 1 -- 172.21.15.33:0/2257474217 shutdown_connections 2024-03-19T02:59:46.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.253+0000 7f27e23c8740 1 --2- 172.21.15.33:0/2257474217 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ce78e247d0 0x55ce78e26bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.253+0000 7f27e23c8740 1 --2- 172.21.15.33:0/2257474217 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce78e1bbf0 0x55ce78e24290 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.253+0000 7f27e23c8740 1 --2- 172.21.15.33:0/2257474217 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ce78e1b2d0 0x55ce78e1b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.253+0000 7f27e23c8740 1 -- 172.21.15.33:0/2257474217 >> 172.21.15.33:0/2257474217 conn(0x55ce78e15a00 msgr2=0x55ce78e15de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:46.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.253+0000 7f27e23c8740 1 -- 172.21.15.33:0/2257474217 shutdown_connections 2024-03-19T02:59:46.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.253+0000 7f27e23c8740 1 -- 172.21.15.33:0/2257474217 wait complete. 2024-03-19T02:59:46.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.253+0000 7f27e23c8740 1 -- start start 2024-03-19T02:59:46.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.253+0000 7f27e23c8740 1 Processor -- start 2024-03-19T02:59:46.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.254+0000 7f27e23c8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ce78e1b2d0 0x55ce78e217f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.254+0000 7f27e23c8740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ce78e1bbf0 0x55ce78e21d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.254+0000 7f27e23c8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce78e247d0 0x55ce78e22270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.254+0000 7f27e23c8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ce78e275c0 con 0x55ce78e247d0 2024-03-19T02:59:46.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.254+0000 7f27e23c8740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55ce78d3a310 con 0x55ce78e1bbf0 2024-03-19T02:59:46.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.254+0000 7f27e23c8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ce78e27740 con 0x55ce78e1b2d0 2024-03-19T02:59:46.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.254+0000 7f27e23c8740 10 client.0 _ll_register_callbacks cb 0x55ce78d2e2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:59:46.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.254+0000 7f27ddc9c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce78e247d0 0x55ce78e22270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.254+0000 7f27ddc9c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce78e247d0 0x55ce78e22270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36904/0 (socket says 172.21.15.33:36904) 2024-03-19T02:59:46.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.254+0000 7f27ddc9c640 1 -- 172.21.15.33:0/1657013012 learned_addr learned my addr 172.21.15.33:0/1657013012 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:46.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.254+0000 7f27dcc9a640 1 --2- 172.21.15.33:0/1657013012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ce78e1bbf0 0x55ce78e21d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.254+0000 7f27dd49b640 1 --2- 172.21.15.33:0/1657013012 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ce78e1b2d0 0x55ce78e217f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.254+0000 7f27ddc9c640 1 -- 172.21.15.33:0/1657013012 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ce78e1b2d0 msgr2=0x55ce78e217f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:46.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.254+0000 7f27ddc9c640 1 --2- 172.21.15.33:0/1657013012 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ce78e1b2d0 0x55ce78e217f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.254+0000 7f27ddc9c640 1 -- 172.21.15.33:0/1657013012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ce78e1bbf0 msgr2=0x55ce78e21d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:46.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.254+0000 7f27ddc9c640 1 --2- 172.21.15.33:0/1657013012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ce78e1bbf0 0x55ce78e21d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.254+0000 7f27ddc9c640 1 -- 172.21.15.33:0/1657013012 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce78a923f0 con 0x55ce78e247d0 2024-03-19T02:59:46.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.254+0000 7f27ddc9c640 1 --2- 172.21.15.33:0/1657013012 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce78e247d0 0x55ce78e22270 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f27d000fb60 tx=0x7f27d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c586ae6704338db server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:46.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.254+0000 7f27c0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:59:46.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.255+0000 7f27c0ff9640 1 -- 172.21.15.33:0/1657013012 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d00047e0 con 0x55ce78e247d0 2024-03-19T02:59:46.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.255+0000 7f27c0ff9640 1 -- 172.21.15.33:0/1657013012 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f27d0029470 con 0x55ce78e247d0 2024-03-19T02:59:46.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.255+0000 7f27e23c8740 1 -- 172.21.15.33:0/1657013012 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27d4009e20 con 0x55ce78e247d0 2024-03-19T02:59:46.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.255+0000 7f27c0ff9640 1 -- 172.21.15.33:0/1657013012 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f27d0029670 con 0x55ce78e247d0 2024-03-19T02:59:46.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.255+0000 7f27e23c8740 10 client.16986 fetch_fsmap learned FSMap version 25 2024-03-19T02:59:46.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.255+0000 7f27e23c8740 1 -- 172.21.15.33:0/1657013012 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ce78b5c7d0 con 0x55ce78e247d0 2024-03-19T02:59:46.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.255+0000 7f27c0ff9640 1 -- 172.21.15.33:0/1657013012 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f27d4009e20 con 0x55ce78e247d0 2024-03-19T02:59:46.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.255+0000 7f27c0ff9640 1 -- 172.21.15.33:0/1657013012 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f27d001f7c0 con 0x55ce78e247d0 2024-03-19T02:59:46.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.255+0000 7f27e23c8740 10 client.16986 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:59:46.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.255+0000 7f27e23c8740 10 client.16986 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T02:59:46.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.255+0000 7f27e23c8740 20 client.16986 populate_metadata read hostname 'smithi033' 2024-03-19T02:59:46.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.255+0000 7f27e23c8740 1 --2- 172.21.15.33:0/1657013012 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55ce78e34b90 0x55ce78e36fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.255+0000 7f27e23c8740 4 client.16986 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:59:46.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.255+0000 7f27e23c8740 1 -- 172.21.15.33:0/1657013012 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce78ddc3e0 con 0x55ce78e34b90 2024-03-19T02:59:46.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.255+0000 7f27dd49b640 1 --2- 172.21.15.33:0/1657013012 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55ce78e34b90 0x55ce78e36fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.256+0000 7f27c0ff9640 10 client.16986 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T02:59:46.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.256+0000 7f27dd49b640 1 --2- 172.21.15.33:0/1657013012 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55ce78e34b90 0x55ce78e36fe0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a248c252a47b1700 server_cookie=43f62d6d9e697676 in_seq=0 out_seq=0 2024-03-19T02:59:46.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.256+0000 7f27c0ff9640 1 -- 172.21.15.33:0/1657013012 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ce78ddc3e0 con 0x55ce78e34b90 2024-03-19T02:59:46.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.256+0000 7f27c0ff9640 10 client.16986 handle_command_reply: tid=0 2024-03-19T02:59:46.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.256+0000 7f27e23c8740 1 client.16986 shutdown 2024-03-19T02:59:46.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.257+0000 7f27e23c8740 1 -- 172.21.15.33:0/1657013012 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce78e247d0 msgr2=0x55ce78e22270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:46.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.257+0000 7f27e23c8740 1 --2- 172.21.15.33:0/1657013012 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce78e247d0 0x55ce78e22270 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f27d000fb60 tx=0x7f27d0004a40 comp rx=0 tx=0).stop 2024-03-19T02:59:46.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.257+0000 7f27e23c8740 1 -- 172.21.15.33:0/1657013012 shutdown_connections 2024-03-19T02:59:46.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.257+0000 7f27e23c8740 1 --2- 172.21.15.33:0/1657013012 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55ce78e34b90 0x55ce78e36fe0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.257+0000 7f27e23c8740 1 --2- 172.21.15.33:0/1657013012 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce78e247d0 0x55ce78e22270 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.257+0000 7f27e23c8740 1 --2- 172.21.15.33:0/1657013012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ce78e1bbf0 0x55ce78e21d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.257+0000 7f27e23c8740 1 --2- 172.21.15.33:0/1657013012 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ce78e1b2d0 0x55ce78e217f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.257+0000 7f27c0ff9640 0 client.16986 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T02:59:46.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.257+0000 7f27e23c8740 1 -- 172.21.15.33:0/1657013012 >> 172.21.15.33:0/1657013012 conn(0x55ce78e15a00 msgr2=0x55ce78e17750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:46.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.257+0000 7f27e23c8740 1 -- 172.21.15.33:0/1657013012 shutdown_connections 2024-03-19T02:59:46.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.257+0000 7f27e23c8740 1 -- 172.21.15.33:0/1657013012 wait complete. 2024-03-19T02:59:46.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.257+0000 7f27e23c8740 20 client.16986 trim_cache size 0 max 16384 2024-03-19T02:59:46.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.270+0000 7f27e23c8740 1 Processor -- start 2024-03-19T02:59:46.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.270+0000 7f27e23c8740 1 -- start start 2024-03-19T02:59:46.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.270+0000 7f27e23c8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ce78e1b2d0 0x55ce78e30360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.270+0000 7f27e23c8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce78e1bbf0 0x55ce78e3c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.270+0000 7f27e23c8740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ce78e247d0 0x55ce78e2c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.270+0000 7f27e23c8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f27d81b35e0 con 0x55ce78e1bbf0 2024-03-19T02:59:46.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.270+0000 7f27e23c8740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f27d81b0eb0 con 0x55ce78e247d0 2024-03-19T02:59:46.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.270+0000 7f27e23c8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f27d81aece0 con 0x55ce78e1b2d0 2024-03-19T02:59:46.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.270+0000 7f27dcc9a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce78e1bbf0 0x55ce78e3c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.270+0000 7f27dcc9a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce78e1bbf0 0x55ce78e3c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36912/0 (socket says 172.21.15.33:36912) 2024-03-19T02:59:46.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.270+0000 7f27dcc9a640 1 -- 172.21.15.33:0/2796908089 learned_addr learned my addr 172.21.15.33:0/2796908089 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:46.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.270+0000 7f27dd49b640 1 --2- 172.21.15.33:0/2796908089 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ce78e1b2d0 0x55ce78e30360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.277 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.270+0000 7f27ddc9c640 1 --2- 172.21.15.33:0/2796908089 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ce78e247d0 0x55ce78e2c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.270+0000 7f27dcc9a640 1 -- 172.21.15.33:0/2796908089 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ce78e1b2d0 msgr2=0x55ce78e30360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:46.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.270+0000 7f27dcc9a640 1 --2- 172.21.15.33:0/2796908089 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ce78e1b2d0 0x55ce78e30360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.270+0000 7f27dcc9a640 1 -- 172.21.15.33:0/2796908089 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ce78e247d0 msgr2=0x55ce78e2c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:46.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.270+0000 7f27dcc9a640 1 --2- 172.21.15.33:0/2796908089 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ce78e247d0 0x55ce78e2c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.270+0000 7f27dcc9a640 1 -- 172.21.15.33:0/2796908089 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce78e0ffb0 con 0x55ce78e1bbf0 2024-03-19T02:59:46.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.270+0000 7f27dd49b640 1 --2- 172.21.15.33:0/2796908089 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ce78e1b2d0 0x55ce78e30360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:59:46.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.271+0000 7f27dcc9a640 1 --2- 172.21.15.33:0/2796908089 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce78e1bbf0 0x55ce78e3c8b0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f27c8005820 tx=0x7f27c8090180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f89870135d6e3145 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:46.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.271+0000 7f27c2ffd640 1 -- 172.21.15.33:0/2796908089 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c80bce00 con 0x55ce78e1bbf0 2024-03-19T02:59:46.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.271+0000 7f27c2ffd640 1 -- 172.21.15.33:0/2796908089 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f27c80bdba0 con 0x55ce78e1bbf0 2024-03-19T02:59:46.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.271+0000 7f27c2ffd640 1 -- 172.21.15.33:0/2796908089 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c80053d0 con 0x55ce78e1bbf0 2024-03-19T02:59:46.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.271+0000 7f27e23c8740 1 -- 172.21.15.33:0/2796908089 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce78e1bbf0 msgr2=0x55ce78e3c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:46.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.271+0000 7f27e23c8740 1 --2- 172.21.15.33:0/2796908089 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce78e1bbf0 0x55ce78e3c8b0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f27c8005820 tx=0x7f27c8090180 comp rx=0 tx=0).stop 2024-03-19T02:59:46.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.271+0000 7f27e23c8740 1 -- 172.21.15.33:0/2796908089 shutdown_connections 2024-03-19T02:59:46.278 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.271+0000 7f27e23c8740 1 --2- 172.21.15.33:0/2796908089 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ce78e247d0 0x55ce78e2c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.271+0000 7f27e23c8740 1 --2- 172.21.15.33:0/2796908089 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce78e1bbf0 0x55ce78e3c8b0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.271+0000 7f27e23c8740 1 --2- 172.21.15.33:0/2796908089 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ce78e1b2d0 0x55ce78e30360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.271+0000 7f27e23c8740 1 -- 172.21.15.33:0/2796908089 >> 172.21.15.33:0/2796908089 conn(0x55ce78e15a00 msgr2=0x55ce78e217b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:46.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.271+0000 7f27e23c8740 1 -- 172.21.15.33:0/2796908089 shutdown_connections 2024-03-19T02:59:46.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.271+0000 7f27e23c8740 1 -- 172.21.15.33:0/2796908089 wait complete. 2024-03-19T02:59:46.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.272+0000 7f27e23c8740 1 -- start start 2024-03-19T02:59:46.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.272+0000 7f27e23c8740 1 Processor -- start 2024-03-19T02:59:46.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.272+0000 7f27e23c8740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ce78e1b2d0 0x55ce78e2c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.272+0000 7f27e23c8740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ce78e1bbf0 0x55ce78e29960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.272+0000 7f27e23c8740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce78e247d0 0x55ce78e29ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.272+0000 7f27e23c8740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f27d81af370 con 0x55ce78e247d0 2024-03-19T02:59:46.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.272+0000 7f27e23c8740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f27d81b07e0 con 0x55ce78e1b2d0 2024-03-19T02:59:46.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.272+0000 7f27e23c8740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f27d81afa40 con 0x55ce78e1bbf0 2024-03-19T02:59:46.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.272+0000 7f27e23c8740 10 client.0 _ll_register_callbacks cb 0x7f27d0011e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:59:46.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.272+0000 7f27ddc9c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce78e247d0 0x55ce78e29ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.272+0000 7f27dd49b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ce78e1b2d0 0x55ce78e2c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.272+0000 7f27ddc9c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce78e247d0 0x55ce78e29ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36926/0 (socket says 172.21.15.33:36926) 2024-03-19T02:59:46.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.272+0000 7f27ddc9c640 1 -- 172.21.15.33:0/3016484033 learned_addr learned my addr 172.21.15.33:0/3016484033 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:46.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.272+0000 7f27dcc9a640 1 --2- 172.21.15.33:0/3016484033 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ce78e1bbf0 0x55ce78e29960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.272+0000 7f27ddc9c640 1 -- 172.21.15.33:0/3016484033 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ce78e1bbf0 msgr2=0x55ce78e29960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:46.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.272+0000 7f27ddc9c640 1 --2- 172.21.15.33:0/3016484033 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ce78e1bbf0 0x55ce78e29960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.272+0000 7f27ddc9c640 1 -- 172.21.15.33:0/3016484033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ce78e1b2d0 msgr2=0x55ce78e2c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:46.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.272+0000 7f27ddc9c640 1 --2- 172.21.15.33:0/3016484033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ce78e1b2d0 0x55ce78e2c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.273+0000 7f27ddc9c640 1 -- 172.21.15.33:0/3016484033 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce78e2a3e0 con 0x55ce78e247d0 2024-03-19T02:59:46.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.273+0000 7f27ddc9c640 1 --2- 172.21.15.33:0/3016484033 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce78e247d0 0x55ce78e29ea0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f27d0013010 tx=0x7f27d000ff60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b57893412839251 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:46.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.273+0000 7f27c0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:59:46.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.273+0000 7f27c0ff9640 1 -- 172.21.15.33:0/3016484033 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d003a070 con 0x55ce78e247d0 2024-03-19T02:59:46.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.273+0000 7f27c0ff9640 1 -- 172.21.15.33:0/3016484033 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f27d001f960 con 0x55ce78e247d0 2024-03-19T02:59:46.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.273+0000 7f27e23c8740 1 -- 172.21.15.33:0/3016484033 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27c80bdba0 con 0x55ce78e247d0 2024-03-19T02:59:46.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.273+0000 7f27c0ff9640 1 -- 172.21.15.33:0/3016484033 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f27d0003c80 con 0x55ce78e247d0 2024-03-19T02:59:46.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.273+0000 7f27e23c8740 10 client.16992 fetch_fsmap learned FSMap version 25 2024-03-19T02:59:46.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.273+0000 7f27e23c8740 1 -- 172.21.15.33:0/3016484033 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ce78e9bab0 con 0x55ce78e247d0 2024-03-19T02:59:46.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.273+0000 7f27c0ff9640 1 -- 172.21.15.33:0/3016484033 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f27c80bdba0 con 0x55ce78e247d0 2024-03-19T02:59:46.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.273+0000 7f27c0ff9640 1 -- 172.21.15.33:0/3016484033 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f27d0036720 con 0x55ce78e247d0 2024-03-19T02:59:46.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.273+0000 7f27e23c8740 10 client.16992 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:59:46.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.273+0000 7f27e23c8740 10 client.16992 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T02:59:46.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.273+0000 7f27e23c8740 20 client.16992 populate_metadata read hostname 'smithi033' 2024-03-19T02:59:46.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.273+0000 7f27e23c8740 1 --2- 172.21.15.33:0/3016484033 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55ce78e9c230 0x55ce78e9e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.273+0000 7f27e23c8740 4 client.16992 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:59:46.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.273+0000 7f27e23c8740 1 -- 172.21.15.33:0/3016484033 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ce78d8b9f0 con 0x55ce78e9c230 2024-03-19T02:59:46.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.274+0000 7f27dd49b640 1 --2- 172.21.15.33:0/3016484033 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55ce78e9c230 0x55ce78e9e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.274+0000 7f27dd49b640 1 --2- 172.21.15.33:0/3016484033 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55ce78e9c230 0x55ce78e9e650 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a791a1b775dc5e9e server_cookie=f20a74e750dc0bd4 in_seq=0 out_seq=0 2024-03-19T02:59:46.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.274+0000 7f27c0ff9640 10 client.16992 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T02:59:46.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.274+0000 7f27c0ff9640 1 -- 172.21.15.33:0/3016484033 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ce78d8b9f0 con 0x55ce78e9c230 2024-03-19T02:59:46.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.274+0000 7f27c0ff9640 10 client.16992 handle_command_reply: tid=0 2024-03-19T02:59:46.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.274+0000 7f27e23c8740 1 client.16992 shutdown 2024-03-19T02:59:46.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.274+0000 7f27e23c8740 1 -- 172.21.15.33:0/3016484033 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce78e247d0 msgr2=0x55ce78e29ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:46.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.274+0000 7f27e23c8740 1 --2- 172.21.15.33:0/3016484033 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce78e247d0 0x55ce78e29ea0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f27d0013010 tx=0x7f27d000ff60 comp rx=0 tx=0).stop 2024-03-19T02:59:46.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.274+0000 7f27e23c8740 1 -- 172.21.15.33:0/3016484033 shutdown_connections 2024-03-19T02:59:46.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.274+0000 7f27e23c8740 1 --2- 172.21.15.33:0/3016484033 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55ce78e9c230 0x55ce78e9e650 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.274+0000 7f27e23c8740 1 --2- 172.21.15.33:0/3016484033 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ce78e1bbf0 0x55ce78e29960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.274+0000 7f27c0ff9640 0 client.16992 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T02:59:46.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.274+0000 7f27e23c8740 1 --2- 172.21.15.33:0/3016484033 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ce78e247d0 0x55ce78e29ea0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.274+0000 7f27e23c8740 1 --2- 172.21.15.33:0/3016484033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ce78e1b2d0 0x55ce78e2c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.274+0000 7f27e23c8740 1 -- 172.21.15.33:0/3016484033 >> 172.21.15.33:0/3016484033 conn(0x55ce78e15a00 msgr2=0x55ce78e217b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:46.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.274+0000 7f27e23c8740 1 -- 172.21.15.33:0/3016484033 shutdown_connections 2024-03-19T02:59:46.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.274+0000 7f27e23c8740 1 -- 172.21.15.33:0/3016484033 wait complete. 2024-03-19T02:59:46.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.274+0000 7f27e23c8740 20 client.16992 trim_cache size 0 max 16384 2024-03-19T02:59:46.282 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:59:46.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.275+0000 7f27df6ff640 1 -- 172.21.15.33:0/2526849985 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f27c40a2970 msgr2=0x7f27c40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:46.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.275+0000 7f27df6ff640 1 --2- 172.21.15.33:0/2526849985 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f27c40a2970 0x7f27c40a4e30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f27d0005c40 tx=0x7f27d0009500 comp rx=0 tx=0).stop 2024-03-19T02:59:46.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.275+0000 7f27df6ff640 1 -- 172.21.15.33:0/2526849985 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27d8101060 msgr2=0x7f27d819a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:46.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.275+0000 7f27df6ff640 1 --2- 172.21.15.33:0/2526849985 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27d8101060 0x7f27d819a6f0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f27c8009230 tx=0x7f27c802c5d0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.275+0000 7f27df6ff640 1 -- 172.21.15.33:0/2526849985 shutdown_connections 2024-03-19T02:59:46.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.275+0000 7f27df6ff640 1 --2- 172.21.15.33:0/2526849985 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f27c40a2970 0x7f27c40a4e30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.275+0000 7f27df6ff640 1 --2- 172.21.15.33:0/2526849985 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27d81066b0 0x7f27d819ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.275+0000 7f27df6ff640 1 --2- 172.21.15.33:0/2526849985 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f27d8101060 0x7f27d819a6f0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.275+0000 7f27df6ff640 1 --2- 172.21.15.33:0/2526849985 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27d81006a0 0x7f27d819a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.275+0000 7f27df6ff640 1 -- 172.21.15.33:0/2526849985 >> 172.21.15.33:0/2526849985 conn(0x7f27d80778d0 msgr2=0x7f27d810c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:46.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.275+0000 7f27df6ff640 1 -- 172.21.15.33:0/2526849985 shutdown_connections 2024-03-19T02:59:46.284 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.275+0000 7f27df6ff640 1 -- 172.21.15.33:0/2526849985 wait complete. 2024-03-19T02:59:46.287 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T02:59:46.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.390+0000 7f49e1d59640 1 -- 172.21.15.33:0/1362149491 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49dc100970 msgr2=0x7f49dc100df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:46.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.390+0000 7f49e1d59640 1 --2- 172.21.15.33:0/1362149491 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49dc100970 0x7f49dc100df0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f49d0009860 tx=0x7f49d002bb70 comp rx=0 tx=0).stop 2024-03-19T02:59:46.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.390+0000 7f49e1d59640 1 -- 172.21.15.33:0/1362149491 shutdown_connections 2024-03-19T02:59:46.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.390+0000 7f49e1d59640 1 --2- 172.21.15.33:0/1362149491 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49dc100970 0x7f49dc100df0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.390+0000 7f49e1d59640 1 --2- 172.21.15.33:0/1362149491 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49dc101330 0x7f49dc10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.390+0000 7f49e1d59640 1 --2- 172.21.15.33:0/1362149491 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f49dc105660 0x7f49dc105a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.390+0000 7f49e1d59640 1 -- 172.21.15.33:0/1362149491 >> 172.21.15.33:0/1362149491 conn(0x7f49dc0774c0 msgr2=0x7f49dc0778d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:46.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.390+0000 7f49e1d59640 1 -- 172.21.15.33:0/1362149491 shutdown_connections 2024-03-19T02:59:46.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.390+0000 7f49e1d59640 1 -- 172.21.15.33:0/1362149491 wait complete. 2024-03-19T02:59:46.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.391+0000 7f49e1d59640 1 Processor -- start 2024-03-19T02:59:46.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.391+0000 7f49e1d59640 1 -- start start 2024-03-19T02:59:46.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.391+0000 7f49e1d59640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49dc100970 0x7f49dc197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.391+0000 7f49e1d59640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f49dc101330 0x7f49dc1984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.391+0000 7f49e1d59640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49dc105660 0x7f49dc19c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.391+0000 7f49e1d59640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f49dc111a50 con 0x7f49dc101330 2024-03-19T02:59:46.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.391+0000 7f49e1d59640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f49dc1118d0 con 0x7f49dc100970 2024-03-19T02:59:46.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.391+0000 7f49e1d59640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f49dc111bd0 con 0x7f49dc105660 2024-03-19T02:59:46.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.391+0000 7f49daffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f49dc101330 0x7f49dc1984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.391+0000 7f49daffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f49dc101330 0x7f49dc1984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36932/0 (socket says 172.21.15.33:36932) 2024-03-19T02:59:46.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.391+0000 7f49daffd640 1 -- 172.21.15.33:0/1103043962 learned_addr learned my addr 172.21.15.33:0/1103043962 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:46.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.391+0000 7f49db7fe640 1 --2- 172.21.15.33:0/1103043962 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49dc100970 0x7f49dc197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.391+0000 7f49dbfff640 1 --2- 172.21.15.33:0/1103043962 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49dc105660 0x7f49dc19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.392+0000 7f49daffd640 1 -- 172.21.15.33:0/1103043962 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49dc105660 msgr2=0x7f49dc19c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:46.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.392+0000 7f49daffd640 1 --2- 172.21.15.33:0/1103043962 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49dc105660 0x7f49dc19c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.392+0000 7f49daffd640 1 -- 172.21.15.33:0/1103043962 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49dc100970 msgr2=0x7f49dc197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:46.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.392+0000 7f49daffd640 1 --2- 172.21.15.33:0/1103043962 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49dc100970 0x7f49dc197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.392+0000 7f49daffd640 1 -- 172.21.15.33:0/1103043962 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49dc19cf60 con 0x7f49dc101330 2024-03-19T02:59:46.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.392+0000 7f49daffd640 1 --2- 172.21.15.33:0/1103043962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f49dc101330 0x7f49dc1984b0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f49d0037040 tx=0x7f49d0035990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:46.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.392+0000 7f49d8ff9640 1 -- 172.21.15.33:0/1103043962 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d0041020 con 0x7f49dc101330 2024-03-19T02:59:46.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.392+0000 7f49d8ff9640 1 -- 172.21.15.33:0/1103043962 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f49d002c9c0 con 0x7f49dc101330 2024-03-19T02:59:46.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.392+0000 7f49e1d59640 1 -- 172.21.15.33:0/1103043962 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49dc19d1f0 con 0x7f49dc101330 2024-03-19T02:59:46.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.392+0000 7f49d8ff9640 1 -- 172.21.15.33:0/1103043962 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d0039040 con 0x7f49dc101330 2024-03-19T02:59:46.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.392+0000 7f49e1d59640 1 -- 172.21.15.33:0/1103043962 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49dc1a4ae0 con 0x7f49dc101330 2024-03-19T02:59:46.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.394+0000 7f49d8ff9640 1 -- 172.21.15.33:0/1103043962 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f49d0044020 con 0x7f49dc101330 2024-03-19T02:59:46.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.395+0000 7f49e4a12740 1 Processor -- start 2024-03-19T02:59:46.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.395+0000 7f49e4a12740 1 -- start start 2024-03-19T02:59:46.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.395+0000 7f49e4a12740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562f71382d0 0x5562f71386b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.395+0000 7f49e4a12740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562f7138bf0 0x5562f7141290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.395+0000 7f49e4a12740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562f71417d0 0x5562f7143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.395+0000 7f49e4a12740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5562f709b4b0 con 0x5562f71417d0 2024-03-19T02:59:46.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.395+0000 7f49e4a12740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5562f7043d70 con 0x5562f7138bf0 2024-03-19T02:59:46.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.395+0000 7f49e4a12740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5562f7097550 con 0x5562f71382d0 2024-03-19T02:59:46.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.395+0000 7f49d8ff9640 1 --2- 172.21.15.33:0/1103043962 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f49c40a2970 0x7f49c40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.395+0000 7f49daffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562f71417d0 0x5562f7143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.395+0000 7f49db7fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562f7138bf0 0x5562f7141290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.395+0000 7f49dbfff640 1 --2- 172.21.15.33:0/1103043962 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f49c40a2970 0x7f49c40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.395+0000 7f49daffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562f71417d0 0x5562f7143bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36946/0 (socket says 172.21.15.33:36946) 2024-03-19T02:59:46.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.395+0000 7f49daffd640 1 -- 172.21.15.33:0/3059071628 learned_addr learned my addr 172.21.15.33:0/3059071628 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:46.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.395+0000 7f49dbfff640 1 --2- 172.21.15.33:0/3059071628 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562f71382d0 0x5562f71386b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.395+0000 7f49d8ff9640 1 -- 172.21.15.33:0/1103043962 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f49d00f08e0 con 0x7f49dc101330 2024-03-19T02:59:46.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.396+0000 7f49daffd640 1 -- 172.21.15.33:0/3059071628 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562f71382d0 msgr2=0x5562f71386b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:46.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.396+0000 7f49daffd640 1 --2- 172.21.15.33:0/3059071628 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562f71382d0 0x5562f71386b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.396+0000 7f49daffd640 1 -- 172.21.15.33:0/3059071628 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562f7138bf0 msgr2=0x5562f7141290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:46.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.396+0000 7f49daffd640 1 --2- 172.21.15.33:0/3059071628 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562f7138bf0 0x5562f7141290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.396+0000 7f49daffd640 1 -- 172.21.15.33:0/3059071628 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562f7066210 con 0x5562f71417d0 2024-03-19T02:59:46.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.396+0000 7f49db7fe640 1 --2- 172.21.15.33:0/3059071628 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562f7138bf0 0x5562f7141290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:46.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.396+0000 7f49dbfff640 1 --2- 172.21.15.33:0/3059071628 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562f71382d0 0x5562f71386b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:46.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.396+0000 7f49daffd640 1 --2- 172.21.15.33:0/3059071628 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562f71417d0 0x5562f7143bc0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f49d00c4f10 tx=0x7f49d00a77d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92722454bd986155 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:46.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.396+0000 7f49dbfff640 1 --2- 172.21.15.33:0/1103043962 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f49c40a2970 0x7f49c40a4e30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f49d4005fd0 tx=0x7f49d4004ab0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:46.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.396+0000 7f49c1ffb640 1 -- 172.21.15.33:0/3059071628 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d00bc7e0 con 0x5562f71417d0 2024-03-19T02:59:46.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.396+0000 7f49c1ffb640 1 -- 172.21.15.33:0/3059071628 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f49d00bd3a0 con 0x5562f71417d0 2024-03-19T02:59:46.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.396+0000 7f49c1ffb640 1 -- 172.21.15.33:0/3059071628 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d00bc600 con 0x5562f71417d0 2024-03-19T02:59:46.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.397+0000 7f49e4a12740 1 -- 172.21.15.33:0/3059071628 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562f71417d0 msgr2=0x5562f7143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:46.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.397+0000 7f49e4a12740 1 --2- 172.21.15.33:0/3059071628 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562f71417d0 0x5562f7143bc0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f49d00c4f10 tx=0x7f49d00a77d0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.397+0000 7f49e4a12740 1 -- 172.21.15.33:0/3059071628 shutdown_connections 2024-03-19T02:59:46.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.397+0000 7f49e4a12740 1 --2- 172.21.15.33:0/3059071628 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562f71417d0 0x5562f7143bc0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.397+0000 7f49e4a12740 1 --2- 172.21.15.33:0/3059071628 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562f7138bf0 0x5562f7141290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.397+0000 7f49e4a12740 1 --2- 172.21.15.33:0/3059071628 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562f71382d0 0x5562f71386b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.397+0000 7f49e4a12740 1 -- 172.21.15.33:0/3059071628 >> 172.21.15.33:0/3059071628 conn(0x5562f7132a00 msgr2=0x5562f7132de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:46.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.397+0000 7f49e4a12740 1 -- 172.21.15.33:0/3059071628 shutdown_connections 2024-03-19T02:59:46.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.397+0000 7f49e4a12740 1 -- 172.21.15.33:0/3059071628 wait complete. 2024-03-19T02:59:46.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.397+0000 7f49e4a12740 1 -- start start 2024-03-19T02:59:46.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.397+0000 7f49e4a12740 1 Processor -- start 2024-03-19T02:59:46.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.397+0000 7f49e4a12740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562f71382d0 0x5562f713e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.397+0000 7f49e4a12740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562f7138bf0 0x5562f713ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.397+0000 7f49e4a12740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562f71417d0 0x5562f713f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.398+0000 7f49e4a12740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5562f7144380 con 0x5562f7138bf0 2024-03-19T02:59:46.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.398+0000 7f49e4a12740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5562f7057310 con 0x5562f71417d0 2024-03-19T02:59:46.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.398+0000 7f49e4a12740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5562f7144500 con 0x5562f71382d0 2024-03-19T02:59:46.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.398+0000 7f49e4a12740 10 client.0 _ll_register_callbacks cb 0x5562f704b2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:59:46.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.398+0000 7f49daffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562f7138bf0 0x5562f713ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.398+0000 7f49daffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562f7138bf0 0x5562f713ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36956/0 (socket says 172.21.15.33:36956) 2024-03-19T02:59:46.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.398+0000 7f49daffd640 1 -- 172.21.15.33:0/1497236966 learned_addr learned my addr 172.21.15.33:0/1497236966 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:46.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.398+0000 7f49db7fe640 1 --2- 172.21.15.33:0/1497236966 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562f71382d0 0x5562f713e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.398+0000 7f49dbfff640 1 --2- 172.21.15.33:0/1497236966 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562f71417d0 0x5562f713f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.398+0000 7f49daffd640 1 -- 172.21.15.33:0/1497236966 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562f71382d0 msgr2=0x5562f713e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:46.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.398+0000 7f49daffd640 1 --2- 172.21.15.33:0/1497236966 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562f71382d0 0x5562f713e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.398+0000 7f49daffd640 1 -- 172.21.15.33:0/1497236966 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562f71417d0 msgr2=0x5562f713f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:46.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.398+0000 7f49daffd640 1 --2- 172.21.15.33:0/1497236966 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562f71417d0 0x5562f713f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.398+0000 7f49daffd640 1 -- 172.21.15.33:0/1497236966 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562f6daf3f0 con 0x5562f7138bf0 2024-03-19T02:59:46.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.398+0000 7f49daffd640 1 --2- 172.21.15.33:0/1497236966 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562f7138bf0 0x5562f713ed30 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f49d00bc9a0 tx=0x7f49d0035df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f21c6857859eba46 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:46.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.398+0000 7f49baffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:59:46.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.398+0000 7f49baffd640 1 -- 172.21.15.33:0/1497236966 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d00bc600 con 0x5562f7138bf0 2024-03-19T02:59:46.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.398+0000 7f49baffd640 1 -- 172.21.15.33:0/1497236966 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f49d00035f0 con 0x5562f7138bf0 2024-03-19T02:59:46.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.398+0000 7f49e4a12740 1 -- 172.21.15.33:0/1497236966 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49d00bd3a0 con 0x5562f7138bf0 2024-03-19T02:59:46.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.398+0000 7f49baffd640 1 -- 172.21.15.33:0/1497236966 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f49d00bc7e0 con 0x5562f7138bf0 2024-03-19T02:59:46.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.398+0000 7f49e4a12740 10 client.17004 fetch_fsmap learned FSMap version 25 2024-03-19T02:59:46.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.398+0000 7f49e4a12740 1 -- 172.21.15.33:0/1497236966 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5562f6e797d0 con 0x5562f7138bf0 2024-03-19T02:59:46.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.399+0000 7f49baffd640 1 -- 172.21.15.33:0/1497236966 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f49d00bd3a0 con 0x5562f7138bf0 2024-03-19T02:59:46.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.399+0000 7f49baffd640 1 -- 172.21.15.33:0/1497236966 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f49d00a3020 con 0x5562f7138bf0 2024-03-19T02:59:46.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.399+0000 7f49e4a12740 10 client.17004 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:59:46.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.399+0000 7f49e4a12740 10 client.17004 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T02:59:46.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.399+0000 7f49e4a12740 20 client.17004 populate_metadata read hostname 'smithi033' 2024-03-19T02:59:46.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.399+0000 7f49e4a12740 1 --2- 172.21.15.33:0/1497236966 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x5562f71516f0 0x5562f7153ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.399+0000 7f49e4a12740 4 client.17004 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:59:46.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.399+0000 7f49e4a12740 1 -- 172.21.15.33:0/1497236966 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5562f70f93e0 con 0x5562f71516f0 2024-03-19T02:59:46.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.399+0000 7f49db7fe640 1 --2- 172.21.15.33:0/1497236966 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x5562f71516f0 0x5562f7153ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.399+0000 7f49baffd640 10 client.17004 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T02:59:46.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.399+0000 7f49db7fe640 1 --2- 172.21.15.33:0/1497236966 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x5562f71516f0 0x5562f7153ae0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1525ad05e46e6d14 server_cookie=8ba45169732adfc7 in_seq=0 out_seq=0 2024-03-19T02:59:46.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.400+0000 7f49baffd640 1 -- 172.21.15.33:0/1497236966 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5562f70f93e0 con 0x5562f71516f0 2024-03-19T02:59:46.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.400+0000 7f49baffd640 10 client.17004 handle_command_reply: tid=0 2024-03-19T02:59:46.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.400+0000 7f49e4a12740 1 client.17004 shutdown 2024-03-19T02:59:46.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.400+0000 7f49e4a12740 1 -- 172.21.15.33:0/1497236966 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562f7138bf0 msgr2=0x5562f713ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:46.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.400+0000 7f49e4a12740 1 --2- 172.21.15.33:0/1497236966 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562f7138bf0 0x5562f713ed30 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f49d00bc9a0 tx=0x7f49d0035df0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.400+0000 7f49e4a12740 1 -- 172.21.15.33:0/1497236966 shutdown_connections 2024-03-19T02:59:46.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.400+0000 7f49e4a12740 1 --2- 172.21.15.33:0/1497236966 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x5562f71516f0 0x5562f7153ae0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.400+0000 7f49e4a12740 1 --2- 172.21.15.33:0/1497236966 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562f71417d0 0x5562f713f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.400+0000 7f49e4a12740 1 --2- 172.21.15.33:0/1497236966 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562f7138bf0 0x5562f713ed30 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.400+0000 7f49e4a12740 1 --2- 172.21.15.33:0/1497236966 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562f71382d0 0x5562f713e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.400+0000 7f49baffd640 0 client.17004 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T02:59:46.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.400+0000 7f49e4a12740 1 -- 172.21.15.33:0/1497236966 >> 172.21.15.33:0/1497236966 conn(0x5562f7132a00 msgr2=0x5562f7134750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:46.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.401+0000 7f49e4a12740 1 -- 172.21.15.33:0/1497236966 shutdown_connections 2024-03-19T02:59:46.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.401+0000 7f49e4a12740 1 -- 172.21.15.33:0/1497236966 wait complete. 2024-03-19T02:59:46.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.401+0000 7f49e4a12740 20 client.17004 trim_cache size 0 max 16384 2024-03-19T02:59:46.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.415+0000 7f49e4a12740 1 Processor -- start 2024-03-19T02:59:46.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.415+0000 7f49e4a12740 1 -- start start 2024-03-19T02:59:46.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.415+0000 7f49e4a12740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562f71382d0 0x5562f7140f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.415+0000 7f49e4a12740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562f7138bf0 0x5562f714d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.415+0000 7f49e4a12740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562f71417d0 0x5562f7159270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.415+0000 7f49e4a12740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f49dc1b13c0 con 0x5562f7138bf0 2024-03-19T02:59:46.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.415+0000 7f49e4a12740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f49dc1aec90 con 0x5562f71382d0 2024-03-19T02:59:46.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.415+0000 7f49e4a12740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f49dc1acac0 con 0x5562f71417d0 2024-03-19T02:59:46.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.415+0000 7f49daffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562f7138bf0 0x5562f714d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.415+0000 7f49daffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562f7138bf0 0x5562f714d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36970/0 (socket says 172.21.15.33:36970) 2024-03-19T02:59:46.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.415+0000 7f49daffd640 1 -- 172.21.15.33:0/1257423331 learned_addr learned my addr 172.21.15.33:0/1257423331 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:46.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.415+0000 7f49db7fe640 1 --2- 172.21.15.33:0/1257423331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562f71382d0 0x5562f7140f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.415+0000 7f49dbfff640 1 --2- 172.21.15.33:0/1257423331 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562f71417d0 0x5562f7159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.415+0000 7f49daffd640 1 -- 172.21.15.33:0/1257423331 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562f71417d0 msgr2=0x5562f7159270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:46.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.415+0000 7f49daffd640 1 --2- 172.21.15.33:0/1257423331 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562f71417d0 0x5562f7159270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.415+0000 7f49daffd640 1 -- 172.21.15.33:0/1257423331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562f71382d0 msgr2=0x5562f7140f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:46.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.415+0000 7f49daffd640 1 --2- 172.21.15.33:0/1257423331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562f71382d0 0x5562f7140f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.415+0000 7f49daffd640 1 -- 172.21.15.33:0/1257423331 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562f7143bc0 con 0x5562f7138bf0 2024-03-19T02:59:46.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.415+0000 7f49daffd640 1 --2- 172.21.15.33:0/1257423331 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562f7138bf0 0x5562f714d040 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f49d00b9620 tx=0x7f49d0035d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea41e89cf11956cf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:46.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.415+0000 7f49c17fa640 1 -- 172.21.15.33:0/1257423331 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d00bc600 con 0x5562f7138bf0 2024-03-19T02:59:46.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.415+0000 7f49c17fa640 1 -- 172.21.15.33:0/1257423331 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f49d00bd3a0 con 0x5562f7138bf0 2024-03-19T02:59:46.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.415+0000 7f49c17fa640 1 -- 172.21.15.33:0/1257423331 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d00ede70 con 0x5562f7138bf0 2024-03-19T02:59:46.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.415+0000 7f49e4a12740 1 -- 172.21.15.33:0/1257423331 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562f7138bf0 msgr2=0x5562f714d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:46.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.415+0000 7f49e4a12740 1 --2- 172.21.15.33:0/1257423331 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562f7138bf0 0x5562f714d040 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f49d00b9620 tx=0x7f49d0035d80 comp rx=0 tx=0).stop 2024-03-19T02:59:46.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.415+0000 7f49e4a12740 1 -- 172.21.15.33:0/1257423331 shutdown_connections 2024-03-19T02:59:46.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.415+0000 7f49e4a12740 1 --2- 172.21.15.33:0/1257423331 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562f71417d0 0x5562f7159270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.415+0000 7f49e4a12740 1 --2- 172.21.15.33:0/1257423331 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562f7138bf0 0x5562f714d040 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.415+0000 7f49e4a12740 1 --2- 172.21.15.33:0/1257423331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562f71382d0 0x5562f7140f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.415+0000 7f49e4a12740 1 -- 172.21.15.33:0/1257423331 >> 172.21.15.33:0/1257423331 conn(0x5562f7132a00 msgr2=0x5562f7131310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:46.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.415+0000 7f49e4a12740 1 -- 172.21.15.33:0/1257423331 shutdown_connections 2024-03-19T02:59:46.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.415+0000 7f49e4a12740 1 -- 172.21.15.33:0/1257423331 wait complete. 2024-03-19T02:59:46.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.415+0000 7f49e4a12740 1 -- start start 2024-03-19T02:59:46.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.416+0000 7f49e4a12740 1 Processor -- start 2024-03-19T02:59:46.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.416+0000 7f49e4a12740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562f71382d0 0x5562f7144e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.416+0000 7f49e4a12740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562f7138bf0 0x5562f71453d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.416+0000 7f49e4a12740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562f71417d0 0x5562f7145910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.416+0000 7f49e4a12740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f49dc1ad150 con 0x5562f71382d0 2024-03-19T02:59:46.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.416+0000 7f49e4a12740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f49dc1ae5c0 con 0x5562f7138bf0 2024-03-19T02:59:46.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.416+0000 7f49e4a12740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f49dc1ad820 con 0x5562f71417d0 2024-03-19T02:59:46.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.416+0000 7f49db7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562f71382d0 0x5562f7144e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.416+0000 7f49e4a12740 10 client.0 _ll_register_callbacks cb 0x7f49d00d2030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:59:46.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.416+0000 7f49db7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562f71382d0 0x5562f7144e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36980/0 (socket says 172.21.15.33:36980) 2024-03-19T02:59:46.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.416+0000 7f49db7fe640 1 -- 172.21.15.33:0/3845740947 learned_addr learned my addr 172.21.15.33:0/3845740947 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:46.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.416+0000 7f49daffd640 1 --2- 172.21.15.33:0/3845740947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562f7138bf0 0x5562f71453d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.416+0000 7f49dbfff640 1 --2- 172.21.15.33:0/3845740947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562f71417d0 0x5562f7145910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.416+0000 7f49db7fe640 1 -- 172.21.15.33:0/3845740947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562f71417d0 msgr2=0x5562f7145910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:46.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.416+0000 7f49db7fe640 1 --2- 172.21.15.33:0/3845740947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562f71417d0 0x5562f7145910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.416+0000 7f49db7fe640 1 -- 172.21.15.33:0/3845740947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562f7138bf0 msgr2=0x5562f71453d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:46.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.416+0000 7f49db7fe640 1 --2- 172.21.15.33:0/3845740947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562f7138bf0 0x5562f71453d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.416+0000 7f49db7fe640 1 -- 172.21.15.33:0/3845740947 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562f71b68a0 con 0x5562f71382d0 2024-03-19T02:59:46.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.416+0000 7f49db7fe640 1 --2- 172.21.15.33:0/3845740947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562f71382d0 0x5562f7144e90 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x5562f7154210 tx=0x7f49cc0108f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9245bfc3794cda98 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:46.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.417+0000 7f49baffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:59:46.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.417+0000 7f49baffd640 1 -- 172.21.15.33:0/3845740947 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49cc023030 con 0x5562f71382d0 2024-03-19T02:59:46.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.417+0000 7f49baffd640 1 -- 172.21.15.33:0/3845740947 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f49cc01f700 con 0x5562f71382d0 2024-03-19T02:59:46.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.417+0000 7f49e4a12740 1 -- 172.21.15.33:0/3845740947 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49d00bd3a0 con 0x5562f71382d0 2024-03-19T02:59:46.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.417+0000 7f49baffd640 1 -- 172.21.15.33:0/3845740947 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f49cc02e050 con 0x5562f71382d0 2024-03-19T02:59:46.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.417+0000 7f49e4a12740 10 client.17010 fetch_fsmap learned FSMap version 25 2024-03-19T02:59:46.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.417+0000 7f49e4a12740 1 -- 172.21.15.33:0/3845740947 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5562f71b7b10 con 0x5562f71382d0 2024-03-19T02:59:46.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.417+0000 7f49baffd640 1 -- 172.21.15.33:0/3845740947 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f49d00bd3a0 con 0x5562f71382d0 2024-03-19T02:59:46.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.417+0000 7f49baffd640 1 -- 172.21.15.33:0/3845740947 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f49cc01e390 con 0x5562f71382d0 2024-03-19T02:59:46.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.417+0000 7f49e4a12740 10 client.17010 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:59:46.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.417+0000 7f49e4a12740 10 client.17010 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T02:59:46.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.417+0000 7f49e4a12740 20 client.17010 populate_metadata read hostname 'smithi033' 2024-03-19T02:59:46.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.417+0000 7f49e4a12740 1 --2- 172.21.15.33:0/3845740947 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x5562f71b82e0 0x5562f71ba700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.417+0000 7f49e4a12740 4 client.17010 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:59:46.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.417+0000 7f49e4a12740 1 -- 172.21.15.33:0/3845740947 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5562f70a89f0 con 0x5562f71b82e0 2024-03-19T02:59:46.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.417+0000 7f49daffd640 1 --2- 172.21.15.33:0/3845740947 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x5562f71b82e0 0x5562f71ba700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.418+0000 7f49baffd640 10 client.17010 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T02:59:46.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.418+0000 7f49daffd640 1 --2- 172.21.15.33:0/3845740947 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x5562f71b82e0 0x5562f71ba700 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ea04f5732f0c8e6c server_cookie=836cdff134136104 in_seq=0 out_seq=0 2024-03-19T02:59:46.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.418+0000 7f49baffd640 1 -- 172.21.15.33:0/3845740947 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5562f70a89f0 con 0x5562f71b82e0 2024-03-19T02:59:46.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.418+0000 7f49baffd640 10 client.17010 handle_command_reply: tid=0 2024-03-19T02:59:46.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.418+0000 7f49e4a12740 1 client.17010 shutdown 2024-03-19T02:59:46.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.418+0000 7f49e4a12740 1 -- 172.21.15.33:0/3845740947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562f71382d0 msgr2=0x5562f7144e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:46.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.418+0000 7f49e4a12740 1 --2- 172.21.15.33:0/3845740947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562f71382d0 0x5562f7144e90 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x5562f7154210 tx=0x7f49cc0108f0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.418+0000 7f49e4a12740 1 -- 172.21.15.33:0/3845740947 shutdown_connections 2024-03-19T02:59:46.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.418+0000 7f49e4a12740 1 --2- 172.21.15.33:0/3845740947 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x5562f71b82e0 0x5562f71ba700 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.418+0000 7f49e4a12740 1 --2- 172.21.15.33:0/3845740947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5562f71417d0 0x5562f7145910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.418+0000 7f49baffd640 0 client.17010 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T02:59:46.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.418+0000 7f49e4a12740 1 --2- 172.21.15.33:0/3845740947 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5562f7138bf0 0x5562f71453d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.418+0000 7f49e4a12740 1 --2- 172.21.15.33:0/3845740947 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5562f71382d0 0x5562f7144e90 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.418+0000 7f49e4a12740 1 -- 172.21.15.33:0/3845740947 >> 172.21.15.33:0/3845740947 conn(0x5562f7132a00 msgr2=0x5562f713f040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:46.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.418+0000 7f49e4a12740 1 -- 172.21.15.33:0/3845740947 shutdown_connections 2024-03-19T02:59:46.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.418+0000 7f49e4a12740 1 -- 172.21.15.33:0/3845740947 wait complete. 2024-03-19T02:59:46.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.418+0000 7f49e4a12740 20 client.17010 trim_cache size 0 max 16384 2024-03-19T02:59:46.427 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:59:46.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.419+0000 7f49e1d59640 1 -- 172.21.15.33:0/1103043962 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f49c40a2970 msgr2=0x7f49c40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:46.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.419+0000 7f49e1d59640 1 --2- 172.21.15.33:0/1103043962 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f49c40a2970 0x7f49c40a4e30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f49d4005fd0 tx=0x7f49d4004ab0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.419+0000 7f49e1d59640 1 -- 172.21.15.33:0/1103043962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f49dc101330 msgr2=0x7f49dc1984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:46.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.419+0000 7f49e1d59640 1 --2- 172.21.15.33:0/1103043962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f49dc101330 0x7f49dc1984b0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f49d0037040 tx=0x7f49d0035990 comp rx=0 tx=0).stop 2024-03-19T02:59:46.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.419+0000 7f49e1d59640 1 -- 172.21.15.33:0/1103043962 shutdown_connections 2024-03-19T02:59:46.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.419+0000 7f49e1d59640 1 --2- 172.21.15.33:0/1103043962 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f49c40a2970 0x7f49c40a4e30 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.419+0000 7f49e1d59640 1 --2- 172.21.15.33:0/1103043962 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49dc105660 0x7f49dc19c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.419+0000 7f49e1d59640 1 --2- 172.21.15.33:0/1103043962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f49dc101330 0x7f49dc1984b0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.419+0000 7f49e1d59640 1 --2- 172.21.15.33:0/1103043962 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49dc100970 0x7f49dc197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.419+0000 7f49e1d59640 1 -- 172.21.15.33:0/1103043962 >> 172.21.15.33:0/1103043962 conn(0x7f49dc0774c0 msgr2=0x7f49dc0fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:46.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.419+0000 7f49e1d59640 1 -- 172.21.15.33:0/1103043962 shutdown_connections 2024-03-19T02:59:46.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.419+0000 7f49e1d59640 1 -- 172.21.15.33:0/1103043962 wait complete. 2024-03-19T02:59:46.431 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T02:59:46.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.535+0000 7f6c8035b640 1 -- 172.21.15.33:0/1460379611 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c78101310 msgr2=0x7f6c7810d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:46.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.535+0000 7f6c8035b640 1 --2- 172.21.15.33:0/1460379611 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c78101310 0x7f6c7810d4f0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f6c70009ca0 tx=0x7f6c7002c0f0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.535+0000 7f6c8035b640 1 -- 172.21.15.33:0/1460379611 shutdown_connections 2024-03-19T02:59:46.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.535+0000 7f6c8035b640 1 --2- 172.21.15.33:0/1460379611 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c78101310 0x7f6c7810d4f0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.535+0000 7f6c8035b640 1 --2- 172.21.15.33:0/1460379611 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c78100970 0x7f6c78100dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.535+0000 7f6c8035b640 1 --2- 172.21.15.33:0/1460379611 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6c78106750 0x7f6c78106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.535+0000 7f6c8035b640 1 -- 172.21.15.33:0/1460379611 >> 172.21.15.33:0/1460379611 conn(0x7f6c780774c0 msgr2=0x7f6c780778d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:46.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.535+0000 7f6c8035b640 1 -- 172.21.15.33:0/1460379611 shutdown_connections 2024-03-19T02:59:46.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.535+0000 7f6c8035b640 1 -- 172.21.15.33:0/1460379611 wait complete. 2024-03-19T02:59:46.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.536+0000 7f6c8035b640 1 Processor -- start 2024-03-19T02:59:46.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.536+0000 7f6c8035b640 1 -- start start 2024-03-19T02:59:46.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.536+0000 7f6c8035b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c78100970 0x7f6c78102690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.536+0000 7f6c8035b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c78101310 0x7f6c78102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.536+0000 7f6c8035b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6c78106750 0x7f6c78103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.536+0000 7f6c8035b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6c78110980 con 0x7f6c78100970 2024-03-19T02:59:46.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.536+0000 7f6c8035b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6c78110800 con 0x7f6c78101310 2024-03-19T02:59:46.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.536+0000 7f6c8035b640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6c78110b00 con 0x7f6c78106750 2024-03-19T02:59:46.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.536+0000 7f6c7e0f7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c78100970 0x7f6c78102690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.536+0000 7f6c7e0f7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c78100970 0x7f6c78102690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36996/0 (socket says 172.21.15.33:36996) 2024-03-19T02:59:46.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.536+0000 7f6c7e0f7640 1 -- 172.21.15.33:0/1002417480 learned_addr learned my addr 172.21.15.33:0/1002417480 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:46.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.536+0000 7f6c7d8f6640 1 --2- 172.21.15.33:0/1002417480 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c78101310 0x7f6c78102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.536+0000 7f6c7e8f8640 1 --2- 172.21.15.33:0/1002417480 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6c78106750 0x7f6c78103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.536+0000 7f6c7e0f7640 1 -- 172.21.15.33:0/1002417480 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6c78106750 msgr2=0x7f6c78103110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:46.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.536+0000 7f6c7e0f7640 1 --2- 172.21.15.33:0/1002417480 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6c78106750 0x7f6c78103110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.536+0000 7f6c7e0f7640 1 -- 172.21.15.33:0/1002417480 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c78101310 msgr2=0x7f6c78102bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:46.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.536+0000 7f6c7e0f7640 1 --2- 172.21.15.33:0/1002417480 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c78101310 0x7f6c78102bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.536+0000 7f6c7e0f7640 1 -- 172.21.15.33:0/1002417480 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c781a6400 con 0x7f6c78100970 2024-03-19T02:59:46.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.536+0000 7f6c7e0f7640 1 --2- 172.21.15.33:0/1002417480 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c78100970 0x7f6c78102690 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f6c7400e9f0 tx=0x7f6c7400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:46.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.537+0000 7f6c6f7fe640 1 -- 172.21.15.33:0/1002417480 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c7400cd80 con 0x7f6c78100970 2024-03-19T02:59:46.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.537+0000 7f6c8035b640 1 -- 172.21.15.33:0/1002417480 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c781a66f0 con 0x7f6c78100970 2024-03-19T02:59:46.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.537+0000 7f6c6f7fe640 1 -- 172.21.15.33:0/1002417480 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6c740108e0 con 0x7f6c78100970 2024-03-19T02:59:46.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.537+0000 7f6c6f7fe640 1 -- 172.21.15.33:0/1002417480 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c74018920 con 0x7f6c78100970 2024-03-19T02:59:46.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.537+0000 7f6c8035b640 1 -- 172.21.15.33:0/1002417480 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c781a6c40 con 0x7f6c78100970 2024-03-19T02:59:46.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.540+0000 7f6c6f7fe640 1 -- 172.21.15.33:0/1002417480 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f6c74010a90 con 0x7f6c78100970 2024-03-19T02:59:46.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.540+0000 7f6c83025740 1 Processor -- start 2024-03-19T02:59:46.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.540+0000 7f6c83025740 1 -- start start 2024-03-19T02:59:46.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.540+0000 7f6c83025740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3278922d0 0x55a3278926b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.540+0000 7f6c83025740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a327892bf0 0x55a32789b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.540+0000 7f6c83025740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a32789b7d0 0x55a32789dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.540+0000 7f6c83025740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a3277f54b0 con 0x55a3278922d0 2024-03-19T02:59:46.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.540+0000 7f6c83025740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55a32779dd70 con 0x55a32789b7d0 2024-03-19T02:59:46.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.540+0000 7f6c83025740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a3277f1550 con 0x55a327892bf0 2024-03-19T02:59:46.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.540+0000 7f6c7e8f8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3278922d0 0x55a3278926b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.540+0000 7f6c7e8f8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3278922d0 0x55a3278926b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37012/0 (socket says 172.21.15.33:37012) 2024-03-19T02:59:46.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.540+0000 7f6c7e8f8640 1 -- 172.21.15.33:0/2415959891 learned_addr learned my addr 172.21.15.33:0/2415959891 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:46.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.540+0000 7f6c7d8f6640 1 --2- 172.21.15.33:0/2415959891 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a32789b7d0 0x55a32789dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.540+0000 7f6c7e0f7640 1 --2- 172.21.15.33:0/2415959891 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a327892bf0 0x55a32789b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.541+0000 7f6c6f7fe640 1 --2- 172.21.15.33:0/1002417480 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6c640a2970 0x7f6c640a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.541+0000 7f6c7d8f6640 1 -- 172.21.15.33:0/2415959891 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a327892bf0 msgr2=0x55a32789b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:46.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.541+0000 7f6c7d8f6640 1 --2- 172.21.15.33:0/2415959891 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a327892bf0 0x55a32789b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.541+0000 7f6c7d8f6640 1 -- 172.21.15.33:0/2415959891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3278922d0 msgr2=0x55a3278926b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:46.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.541+0000 7f6c7d8f6640 1 --2- 172.21.15.33:0/2415959891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3278922d0 0x55a3278926b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.541+0000 7f6c7d8f6640 1 -- 172.21.15.33:0/2415959891 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3277c0210 con 0x55a32789b7d0 2024-03-19T02:59:46.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.541+0000 7f6c7e8f8640 1 --2- 172.21.15.33:0/2415959891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3278922d0 0x55a3278926b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:46.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.541+0000 7f6c7e8f8640 1 --2- 172.21.15.33:0/1002417480 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6c640a2970 0x7f6c640a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.541+0000 7f6c6f7fe640 1 -- 172.21.15.33:0/1002417480 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f6c740cf460 con 0x7f6c78100970 2024-03-19T02:59:46.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.541+0000 7f6c7d8f6640 1 --2- 172.21.15.33:0/2415959891 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a32789b7d0 0x55a32789dbc0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6c68009bf0 tx=0x7f6c68011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65edd03028486eac server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:46.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.541+0000 7f6c6cbfe640 1 -- 172.21.15.33:0/2415959891 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c68004050 con 0x55a32789b7d0 2024-03-19T02:59:46.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.541+0000 7f6c6cbfe640 1 -- 172.21.15.33:0/2415959891 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6c680054d0 con 0x55a32789b7d0 2024-03-19T02:59:46.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.541+0000 7f6c6cbfe640 1 -- 172.21.15.33:0/2415959891 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c68004cc0 con 0x55a32789b7d0 2024-03-19T02:59:46.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.541+0000 7f6c7e8f8640 1 --2- 172.21.15.33:0/1002417480 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6c640a2970 0x7f6c640a4e30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6c70037040 tx=0x7f6c7002e040 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:46.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.542+0000 7f6c83025740 1 -- 172.21.15.33:0/2415959891 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a32789b7d0 msgr2=0x55a32789dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:46.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.542+0000 7f6c83025740 1 --2- 172.21.15.33:0/2415959891 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a32789b7d0 0x55a32789dbc0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6c68009bf0 tx=0x7f6c68011810 comp rx=0 tx=0).stop 2024-03-19T02:59:46.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.542+0000 7f6c83025740 1 -- 172.21.15.33:0/2415959891 shutdown_connections 2024-03-19T02:59:46.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.542+0000 7f6c83025740 1 --2- 172.21.15.33:0/2415959891 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a327892bf0 0x55a32789b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.542+0000 7f6c83025740 1 --2- 172.21.15.33:0/2415959891 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a32789b7d0 0x55a32789dbc0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.542+0000 7f6c83025740 1 --2- 172.21.15.33:0/2415959891 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3278922d0 0x55a3278926b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.542+0000 7f6c83025740 1 -- 172.21.15.33:0/2415959891 >> 172.21.15.33:0/2415959891 conn(0x55a32788ca00 msgr2=0x55a32788cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:46.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.542+0000 7f6c83025740 1 -- 172.21.15.33:0/2415959891 shutdown_connections 2024-03-19T02:59:46.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.542+0000 7f6c83025740 1 -- 172.21.15.33:0/2415959891 wait complete. 2024-03-19T02:59:46.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.542+0000 7f6c83025740 1 -- start start 2024-03-19T02:59:46.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.542+0000 7f6c83025740 1 Processor -- start 2024-03-19T02:59:46.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.543+0000 7f6c83025740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a3278922d0 0x55a327898840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.543+0000 7f6c83025740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a327892bf0 0x55a327898d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.543+0000 7f6c83025740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a32789b7d0 0x55a3278992c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.543+0000 7f6c83025740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a32789e5c0 con 0x55a32789b7d0 2024-03-19T02:59:46.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.543+0000 7f6c83025740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55a3277b1310 con 0x55a3278922d0 2024-03-19T02:59:46.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.543+0000 7f6c83025740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a32789e740 con 0x55a327892bf0 2024-03-19T02:59:46.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.543+0000 7f6c83025740 10 client.0 _ll_register_callbacks cb 0x55a3277a52d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:59:46.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.543+0000 7f6c7e8f8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a32789b7d0 0x55a3278992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.543+0000 7f6c7e8f8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a32789b7d0 0x55a3278992c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37026/0 (socket says 172.21.15.33:37026) 2024-03-19T02:59:46.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.543+0000 7f6c7e8f8640 1 -- 172.21.15.33:0/3350278727 learned_addr learned my addr 172.21.15.33:0/3350278727 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:46.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.543+0000 7f6c7e0f7640 1 --2- 172.21.15.33:0/3350278727 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a3278922d0 0x55a327898840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.543+0000 7f6c7d8f6640 1 --2- 172.21.15.33:0/3350278727 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a327892bf0 0x55a327898d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.543+0000 7f6c7e8f8640 1 -- 172.21.15.33:0/3350278727 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a327892bf0 msgr2=0x55a327898d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:46.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.543+0000 7f6c7e8f8640 1 --2- 172.21.15.33:0/3350278727 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a327892bf0 0x55a327898d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.543+0000 7f6c7e8f8640 1 -- 172.21.15.33:0/3350278727 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a3278922d0 msgr2=0x55a327898840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:46.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.543+0000 7f6c7e8f8640 1 --2- 172.21.15.33:0/3350278727 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a3278922d0 0x55a327898840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.543+0000 7f6c7e8f8640 1 -- 172.21.15.33:0/3350278727 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3275093f0 con 0x55a32789b7d0 2024-03-19T02:59:46.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.543+0000 7f6c7e0f7640 1 --2- 172.21.15.33:0/3350278727 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a3278922d0 0x55a327898840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:46.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.543+0000 7f6c7e8f8640 1 --2- 172.21.15.33:0/3350278727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a32789b7d0 0x55a3278992c0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f6c7002c700 tx=0x7f6c70005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95b308e963527518 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:46.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.543+0000 7f6c61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:59:46.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.543+0000 7f6c61ffb640 1 -- 172.21.15.33:0/3350278727 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c70052060 con 0x55a32789b7d0 2024-03-19T02:59:46.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.543+0000 7f6c61ffb640 1 -- 172.21.15.33:0/3350278727 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f6c70041d00 con 0x55a32789b7d0 2024-03-19T02:59:46.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.543+0000 7f6c83025740 1 -- 172.21.15.33:0/3350278727 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c680054d0 con 0x55a32789b7d0 2024-03-19T02:59:46.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.544+0000 7f6c61ffb640 1 -- 172.21.15.33:0/3350278727 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f6c70046070 con 0x55a32789b7d0 2024-03-19T02:59:46.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.544+0000 7f6c83025740 10 client.17022 fetch_fsmap learned FSMap version 25 2024-03-19T02:59:46.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.544+0000 7f6c83025740 1 -- 172.21.15.33:0/3350278727 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a3275d37d0 con 0x55a32789b7d0 2024-03-19T02:59:46.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.544+0000 7f6c61ffb640 1 -- 172.21.15.33:0/3350278727 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6c680054d0 con 0x55a32789b7d0 2024-03-19T02:59:46.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.544+0000 7f6c61ffb640 1 -- 172.21.15.33:0/3350278727 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f6c7003f040 con 0x55a32789b7d0 2024-03-19T02:59:46.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.544+0000 7f6c83025740 10 client.17022 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:59:46.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.544+0000 7f6c83025740 10 client.17022 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:59:46.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.544+0000 7f6c83025740 20 client.17022 populate_metadata read hostname 'smithi033' 2024-03-19T02:59:46.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.544+0000 7f6c83025740 1 --2- 172.21.15.33:0/3350278727 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55a3278abba0 0x55a3278adff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.544+0000 7f6c83025740 4 client.17022 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:59:46.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.544+0000 7f6c83025740 1 -- 172.21.15.33:0/3350278727 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a3278533e0 con 0x55a3278abba0 2024-03-19T02:59:46.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.544+0000 7f6c7e0f7640 1 --2- 172.21.15.33:0/3350278727 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55a3278abba0 0x55a3278adff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.544+0000 7f6c61ffb640 10 client.17022 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:59:46.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.544+0000 7f6c7e0f7640 1 --2- 172.21.15.33:0/3350278727 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55a3278abba0 0x55a3278adff0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=647d5453ae62c9f5 server_cookie=86d8d761f0e7be31 in_seq=0 out_seq=0 2024-03-19T02:59:46.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.545+0000 7f6c61ffb640 1 -- 172.21.15.33:0/3350278727 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a3278533e0 con 0x55a3278abba0 2024-03-19T02:59:46.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.545+0000 7f6c61ffb640 10 client.17022 handle_command_reply: tid=0 2024-03-19T02:59:46.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.545+0000 7f6c83025740 1 client.17022 shutdown 2024-03-19T02:59:46.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.546+0000 7f6c83025740 1 -- 172.21.15.33:0/3350278727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a32789b7d0 msgr2=0x55a3278992c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:46.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.546+0000 7f6c83025740 1 --2- 172.21.15.33:0/3350278727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a32789b7d0 0x55a3278992c0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f6c7002c700 tx=0x7f6c70005c50 comp rx=0 tx=0).stop 2024-03-19T02:59:46.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.546+0000 7f6c83025740 1 -- 172.21.15.33:0/3350278727 shutdown_connections 2024-03-19T02:59:46.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.546+0000 7f6c83025740 1 --2- 172.21.15.33:0/3350278727 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55a3278abba0 0x55a3278adff0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.546+0000 7f6c83025740 1 --2- 172.21.15.33:0/3350278727 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a327892bf0 0x55a327898d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.546+0000 7f6c83025740 1 --2- 172.21.15.33:0/3350278727 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a32789b7d0 0x55a3278992c0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.546+0000 7f6c83025740 1 --2- 172.21.15.33:0/3350278727 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a3278922d0 0x55a327898840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.546+0000 7f6c61ffb640 0 client.17022 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:59:46.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.546+0000 7f6c83025740 1 -- 172.21.15.33:0/3350278727 >> 172.21.15.33:0/3350278727 conn(0x55a32788ca00 msgr2=0x55a32788e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:46.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.546+0000 7f6c83025740 1 -- 172.21.15.33:0/3350278727 shutdown_connections 2024-03-19T02:59:46.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.546+0000 7f6c83025740 1 -- 172.21.15.33:0/3350278727 wait complete. 2024-03-19T02:59:46.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.546+0000 7f6c83025740 20 client.17022 trim_cache size 0 max 16384 2024-03-19T02:59:46.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.559+0000 7f6c83025740 1 Processor -- start 2024-03-19T02:59:46.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.559+0000 7f6c83025740 1 -- start start 2024-03-19T02:59:46.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.559+0000 7f6c83025740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a3278922d0 0x55a32789af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.559+0000 7f6c83025740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a327892bf0 0x55a3278a7360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.559+0000 7f6c83025740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a32789b7d0 0x55a3278b38c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.559+0000 7f6c83025740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6c781b3600 con 0x55a327892bf0 2024-03-19T02:59:46.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.559+0000 7f6c83025740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6c781b0ed0 con 0x55a32789b7d0 2024-03-19T02:59:46.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.559+0000 7f6c83025740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6c781aed00 con 0x55a3278922d0 2024-03-19T02:59:46.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.559+0000 7f6c7d8f6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a327892bf0 0x55a3278a7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.559+0000 7f6c7d8f6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a327892bf0 0x55a3278a7360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37038/0 (socket says 172.21.15.33:37038) 2024-03-19T02:59:46.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.559+0000 7f6c7d8f6640 1 -- 172.21.15.33:0/3010659435 learned_addr learned my addr 172.21.15.33:0/3010659435 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:46.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.559+0000 7f6c7e0f7640 1 --2- 172.21.15.33:0/3010659435 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a3278922d0 0x55a32789af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.559+0000 7f6c7e8f8640 1 --2- 172.21.15.33:0/3010659435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a32789b7d0 0x55a3278b38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.559+0000 7f6c7d8f6640 1 -- 172.21.15.33:0/3010659435 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a3278922d0 msgr2=0x55a32789af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:46.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.559+0000 7f6c7d8f6640 1 --2- 172.21.15.33:0/3010659435 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a3278922d0 0x55a32789af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.559+0000 7f6c7d8f6640 1 -- 172.21.15.33:0/3010659435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a32789b7d0 msgr2=0x55a3278b38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:46.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.559+0000 7f6c7d8f6640 1 --2- 172.21.15.33:0/3010659435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a32789b7d0 0x55a3278b38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.559+0000 7f6c7d8f6640 1 -- 172.21.15.33:0/3010659435 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a327894400 con 0x55a327892bf0 2024-03-19T02:59:46.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.560+0000 7f6c7d8f6640 1 --2- 172.21.15.33:0/3010659435 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a327892bf0 0x55a3278a7360 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x55a32788db90 tx=0x7f6c68009930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ebfa20f935480d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:46.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.560+0000 7f6c63fff640 1 -- 172.21.15.33:0/3010659435 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c68004250 con 0x55a327892bf0 2024-03-19T02:59:46.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.560+0000 7f6c63fff640 1 -- 172.21.15.33:0/3010659435 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6c68028070 con 0x55a327892bf0 2024-03-19T02:59:46.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.560+0000 7f6c63fff640 1 -- 172.21.15.33:0/3010659435 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6802d470 con 0x55a327892bf0 2024-03-19T02:59:46.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.560+0000 7f6c83025740 1 -- 172.21.15.33:0/3010659435 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a327892bf0 msgr2=0x55a3278a7360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:46.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.560+0000 7f6c83025740 1 --2- 172.21.15.33:0/3010659435 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a327892bf0 0x55a3278a7360 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x55a32788db90 tx=0x7f6c68009930 comp rx=0 tx=0).stop 2024-03-19T02:59:46.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.560+0000 7f6c83025740 1 -- 172.21.15.33:0/3010659435 shutdown_connections 2024-03-19T02:59:46.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.560+0000 7f6c83025740 1 --2- 172.21.15.33:0/3010659435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a32789b7d0 0x55a3278b38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.560+0000 7f6c83025740 1 --2- 172.21.15.33:0/3010659435 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a327892bf0 0x55a3278a7360 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.560+0000 7f6c83025740 1 --2- 172.21.15.33:0/3010659435 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a3278922d0 0x55a32789af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.560+0000 7f6c83025740 1 -- 172.21.15.33:0/3010659435 >> 172.21.15.33:0/3010659435 conn(0x55a32788ca00 msgr2=0x55a32788cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:46.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.560+0000 7f6c83025740 1 -- 172.21.15.33:0/3010659435 shutdown_connections 2024-03-19T02:59:46.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.560+0000 7f6c83025740 1 -- 172.21.15.33:0/3010659435 wait complete. 2024-03-19T02:59:46.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.561+0000 7f6c83025740 1 -- start start 2024-03-19T02:59:46.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.561+0000 7f6c83025740 1 Processor -- start 2024-03-19T02:59:46.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.561+0000 7f6c83025740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3278922d0 0x55a3278a3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.561+0000 7f6c83025740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a327892bf0 0x55a32789f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.561+0000 7f6c83025740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a32789b7d0 0x55a32789f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.561+0000 7f6c83025740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6c781af390 con 0x55a3278922d0 2024-03-19T02:59:46.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.561+0000 7f6c83025740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6c781b0800 con 0x55a327892bf0 2024-03-19T02:59:46.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.561+0000 7f6c83025740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6c781afa60 con 0x55a32789b7d0 2024-03-19T02:59:46.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.561+0000 7f6c83025740 10 client.0 _ll_register_callbacks cb 0x7f6c74005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:59:46.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.561+0000 7f6c7e0f7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3278922d0 0x55a3278a3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.561+0000 7f6c7e0f7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3278922d0 0x55a3278a3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37042/0 (socket says 172.21.15.33:37042) 2024-03-19T02:59:46.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.561+0000 7f6c7e0f7640 1 -- 172.21.15.33:0/1818142291 learned_addr learned my addr 172.21.15.33:0/1818142291 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:46.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.561+0000 7f6c7d8f6640 1 --2- 172.21.15.33:0/1818142291 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a327892bf0 0x55a32789f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.561+0000 7f6c7e8f8640 1 --2- 172.21.15.33:0/1818142291 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a32789b7d0 0x55a32789f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.561+0000 7f6c7e0f7640 1 -- 172.21.15.33:0/1818142291 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a32789b7d0 msgr2=0x55a32789f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:46.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.561+0000 7f6c7e0f7640 1 --2- 172.21.15.33:0/1818142291 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a32789b7d0 0x55a32789f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.561+0000 7f6c7e0f7640 1 -- 172.21.15.33:0/1818142291 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a327892bf0 msgr2=0x55a32789f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:46.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.561+0000 7f6c7e0f7640 1 --2- 172.21.15.33:0/1818142291 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a327892bf0 0x55a32789f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.561+0000 7f6c7e0f7640 1 -- 172.21.15.33:0/1818142291 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a32789fe60 con 0x55a3278922d0 2024-03-19T02:59:46.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.561+0000 7f6c7e0f7640 1 --2- 172.21.15.33:0/1818142291 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3278922d0 0x55a3278a3cf0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f6c7409b790 tx=0x7f6c740ccf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=706adaacaccb36eb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:46.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.561+0000 7f6c61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:59:46.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.562+0000 7f6c61ffb640 1 -- 172.21.15.33:0/1818142291 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c74010de0 con 0x55a3278922d0 2024-03-19T02:59:46.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.562+0000 7f6c61ffb640 1 -- 172.21.15.33:0/1818142291 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f6c740860b0 con 0x55a3278922d0 2024-03-19T02:59:46.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.562+0000 7f6c83025740 1 -- 172.21.15.33:0/1818142291 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c68028070 con 0x55a3278922d0 2024-03-19T02:59:46.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.562+0000 7f6c61ffb640 1 -- 172.21.15.33:0/1818142291 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f6c7409f8a0 con 0x55a3278922d0 2024-03-19T02:59:46.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.562+0000 7f6c83025740 10 client.17028 fetch_fsmap learned FSMap version 25 2024-03-19T02:59:46.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.562+0000 7f6c83025740 1 -- 172.21.15.33:0/1818142291 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a327911fc0 con 0x55a3278922d0 2024-03-19T02:59:46.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.562+0000 7f6c61ffb640 1 -- 172.21.15.33:0/1818142291 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6c68028070 con 0x55a3278922d0 2024-03-19T02:59:46.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.562+0000 7f6c61ffb640 1 -- 172.21.15.33:0/1818142291 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f6c740ccac0 con 0x55a3278922d0 2024-03-19T02:59:46.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.562+0000 7f6c83025740 10 client.17028 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:59:46.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.562+0000 7f6c83025740 10 client.17028 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:59:46.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.562+0000 7f6c83025740 20 client.17028 populate_metadata read hostname 'smithi033' 2024-03-19T02:59:46.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.562+0000 7f6c83025740 1 --2- 172.21.15.33:0/1818142291 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55a327912690 0x55a327914a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:46.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.562+0000 7f6c83025740 4 client.17028 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T02:59:46.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.562+0000 7f6c83025740 1 -- 172.21.15.33:0/1818142291 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a3278029f0 con 0x55a327912690 2024-03-19T02:59:46.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.562+0000 7f6c7d8f6640 1 --2- 172.21.15.33:0/1818142291 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55a327912690 0x55a327914a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:46.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.563+0000 7f6c61ffb640 10 client.17028 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:59:46.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.563+0000 7f6c7d8f6640 1 --2- 172.21.15.33:0/1818142291 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55a327912690 0x55a327914a80 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1265cbca68b5dc8f server_cookie=2df968b0eaab889f in_seq=0 out_seq=0 2024-03-19T02:59:46.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.563+0000 7f6c61ffb640 1 -- 172.21.15.33:0/1818142291 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a3278029f0 con 0x55a327912690 2024-03-19T02:59:46.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.563+0000 7f6c61ffb640 10 client.17028 handle_command_reply: tid=0 2024-03-19T02:59:46.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.563+0000 7f6c83025740 1 client.17028 shutdown 2024-03-19T02:59:46.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.563+0000 7f6c83025740 1 -- 172.21.15.33:0/1818142291 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3278922d0 msgr2=0x55a3278a3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:46.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.563+0000 7f6c83025740 1 --2- 172.21.15.33:0/1818142291 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3278922d0 0x55a3278a3cf0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f6c7409b790 tx=0x7f6c740ccf90 comp rx=0 tx=0).stop 2024-03-19T02:59:46.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.563+0000 7f6c83025740 1 -- 172.21.15.33:0/1818142291 shutdown_connections 2024-03-19T02:59:46.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.563+0000 7f6c83025740 1 --2- 172.21.15.33:0/1818142291 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55a327912690 0x55a327914a80 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.563+0000 7f6c83025740 1 --2- 172.21.15.33:0/1818142291 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a32789b7d0 0x55a32789f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.563+0000 7f6c83025740 1 --2- 172.21.15.33:0/1818142291 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a327892bf0 0x55a32789f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.563+0000 7f6c61ffb640 0 client.17028 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:59:46.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.563+0000 7f6c83025740 1 --2- 172.21.15.33:0/1818142291 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a3278922d0 0x55a3278a3cf0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.563+0000 7f6c83025740 1 -- 172.21.15.33:0/1818142291 >> 172.21.15.33:0/1818142291 conn(0x55a32788ca00 msgr2=0x55a3278987e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:46.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.563+0000 7f6c83025740 1 -- 172.21.15.33:0/1818142291 shutdown_connections 2024-03-19T02:59:46.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.563+0000 7f6c83025740 1 -- 172.21.15.33:0/1818142291 wait complete. 2024-03-19T02:59:46.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.563+0000 7f6c83025740 20 client.17028 trim_cache size 0 max 16384 2024-03-19T02:59:46.571 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T02:59:46.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.563+0000 7f6c8035b640 1 -- 172.21.15.33:0/1002417480 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6c640a2970 msgr2=0x7f6c640a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:46.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.563+0000 7f6c8035b640 1 --2- 172.21.15.33:0/1002417480 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6c640a2970 0x7f6c640a4e30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6c70037040 tx=0x7f6c7002e040 comp rx=0 tx=0).stop 2024-03-19T02:59:46.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.563+0000 7f6c8035b640 1 -- 172.21.15.33:0/1002417480 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c78100970 msgr2=0x7f6c78102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:46.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.563+0000 7f6c8035b640 1 --2- 172.21.15.33:0/1002417480 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c78100970 0x7f6c78102690 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f6c7400e9f0 tx=0x7f6c7400eeb0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.564+0000 7f6c8035b640 1 -- 172.21.15.33:0/1002417480 shutdown_connections 2024-03-19T02:59:46.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.564+0000 7f6c8035b640 1 --2- 172.21.15.33:0/1002417480 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6c640a2970 0x7f6c640a4e30 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.564+0000 7f6c8035b640 1 --2- 172.21.15.33:0/1002417480 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6c78106750 0x7f6c78103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.564+0000 7f6c8035b640 1 --2- 172.21.15.33:0/1002417480 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c78101310 0x7f6c78102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.564+0000 7f6c8035b640 1 --2- 172.21.15.33:0/1002417480 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6c78100970 0x7f6c78102690 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:46.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.564+0000 7f6c8035b640 1 -- 172.21.15.33:0/1002417480 >> 172.21.15.33:0/1002417480 conn(0x7f6c780774c0 msgr2=0x7f6c78104870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:46.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.564+0000 7f6c8035b640 1 -- 172.21.15.33:0/1002417480 shutdown_connections 2024-03-19T02:59:46.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:46.564+0000 7f6c8035b640 1 -- 172.21.15.33:0/1002417480 wait complete. 2024-03-19T02:59:46.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:46 smithi033 ceph-mon[31973]: pgmap v192: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:59:46.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:46 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1206163101' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:59:46.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:46 smithi096 ceph-mon[35946]: pgmap v192: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:59:46.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:46 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1206163101' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:59:47.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:46 smithi155 ceph-mon[35804]: pgmap v192: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:59:47.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:46 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1206163101' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T02:59:47.576 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T02:59:47.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T02:59:47.577 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T02:59:47.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.697+0000 7f984f885640 1 -- 172.21.15.33:0/4074144619 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f984810e1c0 msgr2=0x7f98481105b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:47.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.697+0000 7f984f885640 1 --2- 172.21.15.33:0/4074144619 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f984810e1c0 0x7f98481105b0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f9844009900 tx=0x7f984402bc30 comp rx=0 tx=0).stop 2024-03-19T02:59:47.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.697+0000 7f984f885640 1 -- 172.21.15.33:0/4074144619 shutdown_connections 2024-03-19T02:59:47.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.697+0000 7f984f885640 1 --2- 172.21.15.33:0/4074144619 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9848069320 0x7f984810dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.697+0000 7f984f885640 1 --2- 172.21.15.33:0/4074144619 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f984810e1c0 0x7f98481105b0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.697+0000 7f984f885640 1 --2- 172.21.15.33:0/4074144619 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9848068a00 0x7f9848068de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.697+0000 7f984f885640 1 -- 172.21.15.33:0/4074144619 >> 172.21.15.33:0/4074144619 conn(0x7f9848077820 msgr2=0x7f9848077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:47.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.697+0000 7f984f885640 1 -- 172.21.15.33:0/4074144619 shutdown_connections 2024-03-19T02:59:47.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.697+0000 7f984f885640 1 -- 172.21.15.33:0/4074144619 wait complete. 2024-03-19T02:59:47.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.698+0000 7f984f885640 1 Processor -- start 2024-03-19T02:59:47.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.698+0000 7f984f885640 1 -- start start 2024-03-19T02:59:47.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.698+0000 7f984f885640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9848068a00 0x7f984819a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.698+0000 7f984f885640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9848069320 0x7f984819a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.698+0000 7f984f885640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f984810e1c0 0x7f984819ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.698+0000 7f984f885640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9848113c30 con 0x7f9848068a00 2024-03-19T02:59:47.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.698+0000 7f984f885640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9848113ab0 con 0x7f984810e1c0 2024-03-19T02:59:47.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.698+0000 7f984f885640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9848113db0 con 0x7f9848069320 2024-03-19T02:59:47.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.698+0000 7f984d621640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9848068a00 0x7f984819a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.698+0000 7f984d621640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9848068a00 0x7f984819a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37058/0 (socket says 172.21.15.33:37058) 2024-03-19T02:59:47.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.698+0000 7f984d621640 1 -- 172.21.15.33:0/2618681593 learned_addr learned my addr 172.21.15.33:0/2618681593 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:47.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.698+0000 7f984ce20640 1 --2- 172.21.15.33:0/2618681593 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9848069320 0x7f984819a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.698+0000 7f984de22640 1 --2- 172.21.15.33:0/2618681593 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f984810e1c0 0x7f984819ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.698+0000 7f984d621640 1 -- 172.21.15.33:0/2618681593 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9848069320 msgr2=0x7f984819a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:47.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.698+0000 7f984d621640 1 --2- 172.21.15.33:0/2618681593 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9848069320 0x7f984819a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.698+0000 7f984d621640 1 -- 172.21.15.33:0/2618681593 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f984810e1c0 msgr2=0x7f984819ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:47.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.698+0000 7f984d621640 1 --2- 172.21.15.33:0/2618681593 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f984810e1c0 0x7f984819ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.698+0000 7f984d621640 1 -- 172.21.15.33:0/2618681593 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f984819f160 con 0x7f9848068a00 2024-03-19T02:59:47.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.698+0000 7f984d621640 1 --2- 172.21.15.33:0/2618681593 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9848068a00 0x7f984819a170 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f9840009d50 tx=0x7f984000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:47.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.699+0000 7f983e7fc640 1 -- 172.21.15.33:0/2618681593 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9840003b70 con 0x7f9848068a00 2024-03-19T02:59:47.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.699+0000 7f983e7fc640 1 -- 172.21.15.33:0/2618681593 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9840003d10 con 0x7f9848068a00 2024-03-19T02:59:47.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.699+0000 7f984f885640 1 -- 172.21.15.33:0/2618681593 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f984819f450 con 0x7f9848068a00 2024-03-19T02:59:47.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.699+0000 7f983e7fc640 1 -- 172.21.15.33:0/2618681593 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98400108f0 con 0x7f9848068a00 2024-03-19T02:59:47.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.699+0000 7f984f885640 1 -- 172.21.15.33:0/2618681593 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98481a6ce0 con 0x7f9848068a00 2024-03-19T02:59:47.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.701+0000 7f983e7fc640 1 -- 172.21.15.33:0/2618681593 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f9840003ec0 con 0x7f9848068a00 2024-03-19T02:59:47.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.702+0000 7f9852479740 1 Processor -- start 2024-03-19T02:59:47.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.702+0000 7f9852479740 1 -- start start 2024-03-19T02:59:47.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.702+0000 7f983e7fc640 1 --2- 172.21.15.33:0/2618681593 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f98340a29f0 0x7f98340a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.708 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.702+0000 7f9852479740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d38236380 0x560d38236760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.702+0000 7f9852479740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d38236ca0 0x560d3823f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.702+0000 7f9852479740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d3823f880 0x560d38241c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.702+0000 7f9852479740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560d3819a160 con 0x560d38236ca0 2024-03-19T02:59:47.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.702+0000 7f9852479740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x560d38142e10 con 0x560d3823f880 2024-03-19T02:59:47.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.702+0000 7f9852479740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560d38196200 con 0x560d38236380 2024-03-19T02:59:47.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.702+0000 7f984ce20640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d38236ca0 0x560d3823f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.702+0000 7f984de22640 1 --2- 172.21.15.33:0/2618681593 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f98340a29f0 0x7f98340a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.702+0000 7f984ce20640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d38236ca0 0x560d3823f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37074/0 (socket says 172.21.15.33:37074) 2024-03-19T02:59:47.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.702+0000 7f984ce20640 1 -- 172.21.15.33:0/841490810 learned_addr learned my addr 172.21.15.33:0/841490810 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:47.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.702+0000 7f983e7fc640 1 -- 172.21.15.33:0/2618681593 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f98400cf820 con 0x7f9848068a00 2024-03-19T02:59:47.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.702+0000 7f984d621640 1 --2- 172.21.15.33:0/841490810 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d38236380 0x560d38236760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.702+0000 7f984de22640 1 --2- 172.21.15.33:0/841490810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d3823f880 0x560d38241c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.703+0000 7f984d621640 1 -- 172.21.15.33:0/841490810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d3823f880 msgr2=0x560d38241c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:47.709 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.703+0000 7f984d621640 1 --2- 172.21.15.33:0/841490810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d3823f880 0x560d38241c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.703+0000 7f984d621640 1 -- 172.21.15.33:0/841490810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d38236ca0 msgr2=0x560d3823f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:47.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.703+0000 7f984d621640 1 --2- 172.21.15.33:0/841490810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d38236ca0 0x560d3823f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.703+0000 7f984d621640 1 -- 172.21.15.33:0/841490810 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d37f767e0 con 0x560d38236380 2024-03-19T02:59:47.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.703+0000 7f984ce20640 1 --2- 172.21.15.33:0/841490810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d38236ca0 0x560d3823f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:47.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.703+0000 7f984de22640 1 --2- 172.21.15.33:0/2618681593 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f98340a29f0 0x7f98340a4eb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9844005940 tx=0x7f9844005b70 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:47.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.703+0000 7f984d621640 1 --2- 172.21.15.33:0/841490810 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d38236380 0x560d38236760 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f98400ccbc0 tx=0x7f984000a440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf85de49dd74f109 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:47.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.703+0000 7f98337fe640 1 -- 172.21.15.33:0/841490810 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9840099c10 con 0x560d38236380 2024-03-19T02:59:47.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.703+0000 7f98337fe640 1 -- 172.21.15.33:0/841490810 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9840009820 con 0x560d38236380 2024-03-19T02:59:47.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.703+0000 7f98337fe640 1 -- 172.21.15.33:0/841490810 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98400a39e0 con 0x560d38236380 2024-03-19T02:59:47.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.704+0000 7f9852479740 1 -- 172.21.15.33:0/841490810 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d38236380 msgr2=0x560d38236760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:47.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.704+0000 7f9852479740 1 --2- 172.21.15.33:0/841490810 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d38236380 0x560d38236760 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f98400ccbc0 tx=0x7f984000a440 comp rx=0 tx=0).stop 2024-03-19T02:59:47.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.704+0000 7f9852479740 1 -- 172.21.15.33:0/841490810 shutdown_connections 2024-03-19T02:59:47.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.704+0000 7f9852479740 1 --2- 172.21.15.33:0/841490810 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d3823f880 0x560d38241c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.710 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.704+0000 7f9852479740 1 --2- 172.21.15.33:0/841490810 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d38236ca0 0x560d3823f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.704+0000 7f9852479740 1 --2- 172.21.15.33:0/841490810 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d38236380 0x560d38236760 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.704+0000 7f9852479740 1 -- 172.21.15.33:0/841490810 >> 172.21.15.33:0/841490810 conn(0x560d38230ab0 msgr2=0x560d38230e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:47.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.704+0000 7f9852479740 1 -- 172.21.15.33:0/841490810 shutdown_connections 2024-03-19T02:59:47.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.704+0000 7f9852479740 1 -- 172.21.15.33:0/841490810 wait complete. 2024-03-19T02:59:47.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.704+0000 7f9852479740 1 -- start start 2024-03-19T02:59:47.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.704+0000 7f9852479740 1 Processor -- start 2024-03-19T02:59:47.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.705+0000 7f9852479740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d38236380 0x560d3823c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.705+0000 7f9852479740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d38236ca0 0x560d3823ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.705+0000 7f9852479740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d3823f880 0x560d3823d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.705+0000 7f9852479740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560d38242670 con 0x560d38236ca0 2024-03-19T02:59:47.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.705+0000 7f9852479740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x560d381561a0 con 0x560d3823f880 2024-03-19T02:59:47.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.705+0000 7f9852479740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560d382427f0 con 0x560d38236380 2024-03-19T02:59:47.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.705+0000 7f9852479740 10 client.0 _ll_register_callbacks cb 0x560d37eadc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:59:47.711 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.705+0000 7f984ce20640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d38236ca0 0x560d3823ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.705+0000 7f984d621640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d38236380 0x560d3823c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.705+0000 7f984ce20640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d38236ca0 0x560d3823ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37080/0 (socket says 172.21.15.33:37080) 2024-03-19T02:59:47.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.705+0000 7f984ce20640 1 -- 172.21.15.33:0/2368671983 learned_addr learned my addr 172.21.15.33:0/2368671983 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:47.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.705+0000 7f984de22640 1 --2- 172.21.15.33:0/2368671983 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d3823f880 0x560d3823d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.705+0000 7f984ce20640 1 -- 172.21.15.33:0/2368671983 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d38236380 msgr2=0x560d3823c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:47.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.705+0000 7f984ce20640 1 --2- 172.21.15.33:0/2368671983 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d38236380 0x560d3823c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.705+0000 7f984ce20640 1 -- 172.21.15.33:0/2368671983 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d3823f880 msgr2=0x560d3823d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:47.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.705+0000 7f984ce20640 1 --2- 172.21.15.33:0/2368671983 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d3823f880 0x560d3823d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.705+0000 7f984ce20640 1 -- 172.21.15.33:0/2368671983 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d37fd6f70 con 0x560d38236ca0 2024-03-19T02:59:47.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.705+0000 7f984ce20640 1 --2- 172.21.15.33:0/2368671983 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d38236ca0 0x560d3823ccc0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f9838012700 tx=0x7f9838012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5e3956c694374cf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:47.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.705+0000 7f9830ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:59:47.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.705+0000 7f9830ff9640 1 -- 172.21.15.33:0/2368671983 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9838004050 con 0x560d38236ca0 2024-03-19T02:59:47.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.705+0000 7f9830ff9640 1 -- 172.21.15.33:0/2368671983 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f98380041f0 con 0x560d38236ca0 2024-03-19T02:59:47.712 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.705+0000 7f9852479740 1 -- 172.21.15.33:0/2368671983 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9840009820 con 0x560d38236ca0 2024-03-19T02:59:47.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.706+0000 7f9830ff9640 1 -- 172.21.15.33:0/2368671983 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f98380043f0 con 0x560d38236ca0 2024-03-19T02:59:47.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.706+0000 7f9852479740 10 client.17040 fetch_fsmap learned FSMap version 25 2024-03-19T02:59:47.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.706+0000 7f9852479740 1 -- 172.21.15.33:0/2368671983 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x560d37e6e170 con 0x560d38236ca0 2024-03-19T02:59:47.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.706+0000 7f9830ff9640 1 -- 172.21.15.33:0/2368671983 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9840009820 con 0x560d38236ca0 2024-03-19T02:59:47.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.706+0000 7f9830ff9640 1 -- 172.21.15.33:0/2368671983 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f9838003ac0 con 0x560d38236ca0 2024-03-19T02:59:47.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.706+0000 7f9852479740 10 client.17040 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:59:47.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.706+0000 7f9852479740 10 client.17040 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:59:47.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.706+0000 7f9852479740 20 client.17040 populate_metadata read hostname 'smithi033' 2024-03-19T02:59:47.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.706+0000 7f9852479740 1 --2- 172.21.15.33:0/2368671983 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x560d3824fb80 0x560d38252000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.706+0000 7f9852479740 4 client.17040 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:59:47.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.706+0000 7f9852479740 1 -- 172.21.15.33:0/2368671983 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d381a76a0 con 0x560d3824fb80 2024-03-19T02:59:47.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.706+0000 7f984d621640 1 --2- 172.21.15.33:0/2368671983 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x560d3824fb80 0x560d38252000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.707+0000 7f9830ff9640 10 client.17040 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:59:47.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.707+0000 7f984d621640 1 --2- 172.21.15.33:0/2368671983 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x560d3824fb80 0x560d38252000 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87a7a32e7ef84da2 server_cookie=cd0ec35f5441baef in_seq=0 out_seq=0 2024-03-19T02:59:47.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.707+0000 7f9830ff9640 1 -- 172.21.15.33:0/2368671983 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560d381a76a0 con 0x560d3824fb80 2024-03-19T02:59:47.713 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.707+0000 7f9830ff9640 10 client.17040 handle_command_reply: tid=0 2024-03-19T02:59:47.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.707+0000 7f9852479740 1 client.17040 shutdown 2024-03-19T02:59:47.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.708+0000 7f9852479740 1 -- 172.21.15.33:0/2368671983 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d38236ca0 msgr2=0x560d3823ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:47.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.708+0000 7f9852479740 1 --2- 172.21.15.33:0/2368671983 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d38236ca0 0x560d3823ccc0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f9838012700 tx=0x7f9838012bc0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.708+0000 7f9852479740 1 -- 172.21.15.33:0/2368671983 shutdown_connections 2024-03-19T02:59:47.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.708+0000 7f9852479740 1 --2- 172.21.15.33:0/2368671983 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x560d3824fb80 0x560d38252000 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.708+0000 7f9852479740 1 --2- 172.21.15.33:0/2368671983 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d3823f880 0x560d3823d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.708+0000 7f9852479740 1 --2- 172.21.15.33:0/2368671983 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d38236ca0 0x560d3823ccc0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.708+0000 7f9852479740 1 --2- 172.21.15.33:0/2368671983 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d38236380 0x560d3823c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.708+0000 7f9830ff9640 0 client.17040 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:59:47.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.708+0000 7f9852479740 1 -- 172.21.15.33:0/2368671983 >> 172.21.15.33:0/2368671983 conn(0x560d38230ab0 msgr2=0x560d38232800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:47.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.708+0000 7f9852479740 1 -- 172.21.15.33:0/2368671983 shutdown_connections 2024-03-19T02:59:47.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.708+0000 7f9852479740 1 -- 172.21.15.33:0/2368671983 wait complete. 2024-03-19T02:59:47.714 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.708+0000 7f9852479740 20 client.17040 trim_cache size 0 max 16384 2024-03-19T02:59:47.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.725+0000 7f9852479740 1 Processor -- start 2024-03-19T02:59:47.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.725+0000 7f9852479740 1 -- start start 2024-03-19T02:59:47.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.726+0000 7f9852479740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d38236380 0x560d3823eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.726+0000 7f9852479740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d38236ca0 0x560d3824b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.726+0000 7f9852479740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d3823f880 0x560d38257910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.726+0000 7f9852479740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f98481b3670 con 0x560d38236380 2024-03-19T02:59:47.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.726+0000 7f9852479740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f98481b0f40 con 0x560d38236ca0 2024-03-19T02:59:47.730 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.726+0000 7f9852479740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f98481aed70 con 0x560d3823f880 2024-03-19T02:59:47.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.726+0000 7f984d621640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d38236380 0x560d3823eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.726+0000 7f984d621640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d38236380 0x560d3823eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37086/0 (socket says 172.21.15.33:37086) 2024-03-19T02:59:47.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.726+0000 7f984d621640 1 -- 172.21.15.33:0/1336265909 learned_addr learned my addr 172.21.15.33:0/1336265909 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:47.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.726+0000 7f984ce20640 1 --2- 172.21.15.33:0/1336265909 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d38236ca0 0x560d3824b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.726+0000 7f984de22640 1 --2- 172.21.15.33:0/1336265909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d3823f880 0x560d38257910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.726+0000 7f984d621640 1 -- 172.21.15.33:0/1336265909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d3823f880 msgr2=0x560d38257910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:47.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.726+0000 7f984d621640 1 --2- 172.21.15.33:0/1336265909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d3823f880 0x560d38257910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.726+0000 7f984d621640 1 -- 172.21.15.33:0/1336265909 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d38236ca0 msgr2=0x560d3824b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:47.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.726+0000 7f984d621640 1 --2- 172.21.15.33:0/1336265909 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d38236ca0 0x560d3824b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.726+0000 7f984d621640 1 -- 172.21.15.33:0/1336265909 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d382308e0 con 0x560d38236380 2024-03-19T02:59:47.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.726+0000 7f984de22640 1 --2- 172.21.15.33:0/1336265909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d3823f880 0x560d38257910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:59:47.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.726+0000 7f984ce20640 1 --2- 172.21.15.33:0/1336265909 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d38236ca0 0x560d3824b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:59:47.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.726+0000 7f984d621640 1 --2- 172.21.15.33:0/1336265909 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d38236380 0x560d3823eeb0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f9840097f30 tx=0x7f9840005f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24d0027b1566519d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:47.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.726+0000 7f9832ffd640 1 -- 172.21.15.33:0/1336265909 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98400a3bd0 con 0x560d38236380 2024-03-19T02:59:47.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.726+0000 7f9832ffd640 1 -- 172.21.15.33:0/1336265909 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9840009820 con 0x560d38236380 2024-03-19T02:59:47.731 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.726+0000 7f9832ffd640 1 -- 172.21.15.33:0/1336265909 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9840099120 con 0x560d38236380 2024-03-19T02:59:47.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.727+0000 7f9852479740 1 -- 172.21.15.33:0/1336265909 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d38236380 msgr2=0x560d3823eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:47.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.727+0000 7f9852479740 1 --2- 172.21.15.33:0/1336265909 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d38236380 0x560d3823eeb0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f9840097f30 tx=0x7f9840005f30 comp rx=0 tx=0).stop 2024-03-19T02:59:47.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.727+0000 7f9852479740 1 -- 172.21.15.33:0/1336265909 shutdown_connections 2024-03-19T02:59:47.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.727+0000 7f9852479740 1 --2- 172.21.15.33:0/1336265909 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d3823f880 0x560d38257910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.727+0000 7f9852479740 1 --2- 172.21.15.33:0/1336265909 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d38236ca0 0x560d3824b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.727+0000 7f9852479740 1 --2- 172.21.15.33:0/1336265909 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d38236380 0x560d3823eeb0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.727+0000 7f9852479740 1 -- 172.21.15.33:0/1336265909 >> 172.21.15.33:0/1336265909 conn(0x560d38230ab0 msgr2=0x560d3822f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:47.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.727+0000 7f9852479740 1 -- 172.21.15.33:0/1336265909 shutdown_connections 2024-03-19T02:59:47.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.727+0000 7f9852479740 1 -- 172.21.15.33:0/1336265909 wait complete. 2024-03-19T02:59:47.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.727+0000 7f9852479740 1 -- start start 2024-03-19T02:59:47.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.727+0000 7f9852479740 1 Processor -- start 2024-03-19T02:59:47.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.728+0000 7f9852479740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d38236380 0x560d382475c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.728+0000 7f9852479740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d38236ca0 0x560d38247b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.728+0000 7f9852479740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d3823f880 0x560d38243a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.732 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.728+0000 7f9852479740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f98481af400 con 0x560d38236380 2024-03-19T02:59:47.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.728+0000 7f9852479740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f98481b0870 con 0x560d3823f880 2024-03-19T02:59:47.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.728+0000 7f9852479740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f98481afad0 con 0x560d38236ca0 2024-03-19T02:59:47.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.728+0000 7f9852479740 10 client.0 _ll_register_callbacks cb 0x7f9838009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:59:47.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.728+0000 7f984d621640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d38236380 0x560d382475c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.728+0000 7f984d621640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d38236380 0x560d382475c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37098/0 (socket says 172.21.15.33:37098) 2024-03-19T02:59:47.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.728+0000 7f984d621640 1 -- 172.21.15.33:0/899681191 learned_addr learned my addr 172.21.15.33:0/899681191 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:47.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.728+0000 7f984de22640 1 --2- 172.21.15.33:0/899681191 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d3823f880 0x560d38243a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.728+0000 7f984ce20640 1 --2- 172.21.15.33:0/899681191 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d38236ca0 0x560d38247b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.728+0000 7f984d621640 1 -- 172.21.15.33:0/899681191 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d38236ca0 msgr2=0x560d38247b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:47.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.728+0000 7f984d621640 1 --2- 172.21.15.33:0/899681191 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d38236ca0 0x560d38247b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.728+0000 7f984d621640 1 -- 172.21.15.33:0/899681191 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d3823f880 msgr2=0x560d38243a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:47.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.728+0000 7f984d621640 1 --2- 172.21.15.33:0/899681191 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d3823f880 0x560d38243a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.728+0000 7f984d621640 1 -- 172.21.15.33:0/899681191 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d38244150 con 0x560d38236380 2024-03-19T02:59:47.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.728+0000 7f984d621640 1 --2- 172.21.15.33:0/899681191 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d38236380 0x560d382475c0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f984009bc90 tx=0x7f98400a3bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b9702129fb86b93 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:47.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.728+0000 7f9830ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:59:47.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.728+0000 7f9830ff9640 1 -- 172.21.15.33:0/899681191 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9840099300 con 0x560d38236380 2024-03-19T02:59:47.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.728+0000 7f9830ff9640 1 -- 172.21.15.33:0/899681191 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7f98400045f0 con 0x560d38236380 2024-03-19T02:59:47.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.728+0000 7f9852479740 1 -- 172.21.15.33:0/899681191 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9840009820 con 0x560d38236380 2024-03-19T02:59:47.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.728+0000 7f9830ff9640 1 -- 172.21.15.33:0/899681191 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9840099520 con 0x560d38236380 2024-03-19T02:59:47.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.728+0000 7f9852479740 10 client.17046 fetch_fsmap learned FSMap version 25 2024-03-19T02:59:47.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.728+0000 7f9852479740 1 -- 172.21.15.33:0/899681191 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x560d382b68e0 con 0x560d38236380 2024-03-19T02:59:47.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.729+0000 7f9830ff9640 1 -- 172.21.15.33:0/899681191 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9840009820 con 0x560d38236380 2024-03-19T02:59:47.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.729+0000 7f9830ff9640 1 -- 172.21.15.33:0/899681191 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f98400ba210 con 0x560d38236380 2024-03-19T02:59:47.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.729+0000 7f9852479740 10 client.17046 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:59:47.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.729+0000 7f9852479740 10 client.17046 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:59:47.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.729+0000 7f9852479740 20 client.17046 populate_metadata read hostname 'smithi033' 2024-03-19T02:59:47.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.729+0000 7f9852479740 1 --2- 172.21.15.33:0/899681191 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x560d382b6b60 0x560d382b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.729+0000 7f9852479740 4 client.17046 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T02:59:47.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.729+0000 7f9852479740 1 -- 172.21.15.33:0/899681191 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560d381a19f0 con 0x560d382b6b60 2024-03-19T02:59:47.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.729+0000 7f984ce20640 1 --2- 172.21.15.33:0/899681191 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x560d382b6b60 0x560d382b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.729+0000 7f984ce20640 1 --2- 172.21.15.33:0/899681191 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x560d382b6b60 0x560d382b8f50 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1211ed5a65a43fd9 server_cookie=b493f8ba6b101bd4 in_seq=0 out_seq=0 2024-03-19T02:59:47.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.729+0000 7f9830ff9640 10 client.17046 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:59:47.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.730+0000 7f9830ff9640 1 -- 172.21.15.33:0/899681191 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560d381a19f0 con 0x560d382b6b60 2024-03-19T02:59:47.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.730+0000 7f9830ff9640 10 client.17046 handle_command_reply: tid=0 2024-03-19T02:59:47.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.730+0000 7f9852479740 1 client.17046 shutdown 2024-03-19T02:59:47.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.730+0000 7f9852479740 1 -- 172.21.15.33:0/899681191 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d38236380 msgr2=0x560d382475c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:47.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.730+0000 7f9852479740 1 --2- 172.21.15.33:0/899681191 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d38236380 0x560d382475c0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f984009bc90 tx=0x7f98400a3bd0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.730+0000 7f9852479740 1 -- 172.21.15.33:0/899681191 shutdown_connections 2024-03-19T02:59:47.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.730+0000 7f9852479740 1 --2- 172.21.15.33:0/899681191 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x560d382b6b60 0x560d382b8f50 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.730+0000 7f9852479740 1 --2- 172.21.15.33:0/899681191 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d38236ca0 0x560d38247b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.730+0000 7f9852479740 1 --2- 172.21.15.33:0/899681191 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d3823f880 0x560d38243a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.730+0000 7f9852479740 1 --2- 172.21.15.33:0/899681191 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d38236380 0x560d382475c0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.731+0000 7f9852479740 1 -- 172.21.15.33:0/899681191 >> 172.21.15.33:0/899681191 conn(0x560d38230ab0 msgr2=0x560d38232700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:47.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.731+0000 7f9830ff9640 0 client.17046 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:59:47.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.731+0000 7f9852479740 1 -- 172.21.15.33:0/899681191 shutdown_connections 2024-03-19T02:59:47.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.731+0000 7f9852479740 1 -- 172.21.15.33:0/899681191 wait complete. 2024-03-19T02:59:47.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.731+0000 7f9852479740 20 client.17046 trim_cache size 0 max 16384 2024-03-19T02:59:47.737 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:59:47.737 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-03-19T02:59:47.737 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "1327895e-80db-46f1-8e2a-c8a765e2bcbf", 2024-03-19T02:59:47.737 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-03-19T02:59:47.737 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:59:47.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.731+0000 7f984f885640 1 -- 172.21.15.33:0/2618681593 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f98340a29f0 msgr2=0x7f98340a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:47.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.731+0000 7f984f885640 1 --2- 172.21.15.33:0/2618681593 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f98340a29f0 0x7f98340a4eb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9844005940 tx=0x7f9844005b70 comp rx=0 tx=0).stop 2024-03-19T02:59:47.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.731+0000 7f984f885640 1 -- 172.21.15.33:0/2618681593 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9848068a00 msgr2=0x7f984819a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:47.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.731+0000 7f984f885640 1 --2- 172.21.15.33:0/2618681593 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9848068a00 0x7f984819a170 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f9840009d50 tx=0x7f984000c6a0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.731+0000 7f984f885640 1 -- 172.21.15.33:0/2618681593 shutdown_connections 2024-03-19T02:59:47.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.731+0000 7f984f885640 1 --2- 172.21.15.33:0/2618681593 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f98340a29f0 0x7f98340a4eb0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.731+0000 7f984f885640 1 --2- 172.21.15.33:0/2618681593 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9848069320 0x7f984819a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.731+0000 7f984f885640 1 --2- 172.21.15.33:0/2618681593 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f984810e1c0 0x7f984819ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.731+0000 7f984f885640 1 --2- 172.21.15.33:0/2618681593 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9848068a00 0x7f984819a170 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.731+0000 7f984f885640 1 -- 172.21.15.33:0/2618681593 >> 172.21.15.33:0/2618681593 conn(0x7f9848077820 msgr2=0x7f984810f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:47.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.731+0000 7f984f885640 1 -- 172.21.15.33:0/2618681593 shutdown_connections 2024-03-19T02:59:47.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.731+0000 7f984f885640 1 -- 172.21.15.33:0/2618681593 wait complete. 2024-03-19T02:59:47.742 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T02:59:47.743 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T02:59:47.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.848+0000 7ffa09359640 1 -- 172.21.15.33:0/2799799429 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa041017b0 msgr2=0x7ffa04101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:47.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.849+0000 7ffa09359640 1 --2- 172.21.15.33:0/2799799429 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa041017b0 0x7ffa04101c30 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8009930 tx=0x7ff9f802bc30 comp rx=0 tx=0).stop 2024-03-19T02:59:47.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.849+0000 7ffa09359640 1 -- 172.21.15.33:0/2799799429 shutdown_connections 2024-03-19T02:59:47.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.849+0000 7ffa09359640 1 --2- 172.21.15.33:0/2799799429 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa04102170 0x7ffa0410e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.849+0000 7ffa09359640 1 --2- 172.21.15.33:0/2799799429 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa041017b0 0x7ffa04101c30 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.849+0000 7ffa09359640 1 --2- 172.21.15.33:0/2799799429 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa041077c0 0x7ffa04107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.849+0000 7ffa09359640 1 -- 172.21.15.33:0/2799799429 >> 172.21.15.33:0/2799799429 conn(0x7ffa04078c20 msgr2=0x7ffa04079030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:47.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.849+0000 7ffa09359640 1 -- 172.21.15.33:0/2799799429 shutdown_connections 2024-03-19T02:59:47.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.849+0000 7ffa09359640 1 -- 172.21.15.33:0/2799799429 wait complete. 2024-03-19T02:59:47.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.849+0000 7ffa09359640 1 Processor -- start 2024-03-19T02:59:47.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.849+0000 7ffa09359640 1 -- start start 2024-03-19T02:59:47.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.850+0000 7ffa09359640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa041017b0 0x7ffa04197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.850+0000 7ffa09359640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa04102170 0x7ffa04198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.850+0000 7ffa09359640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa041077c0 0x7ffa0419c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.850+0000 7ffa09359640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ffa04111a40 con 0x7ffa041077c0 2024-03-19T02:59:47.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.850+0000 7ffa09359640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ffa041118c0 con 0x7ffa04102170 2024-03-19T02:59:47.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.850+0000 7ffa09359640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ffa04111bc0 con 0x7ffa041017b0 2024-03-19T02:59:47.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.850+0000 7ffa037fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa041077c0 0x7ffa0419c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.850+0000 7ffa037fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa041077c0 0x7ffa0419c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37106/0 (socket says 172.21.15.33:37106) 2024-03-19T02:59:47.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.850+0000 7ffa037fe640 1 -- 172.21.15.33:0/2356612326 learned_addr learned my addr 172.21.15.33:0/2356612326 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:47.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.850+0000 7ffa027fc640 1 --2- 172.21.15.33:0/2356612326 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa04102170 0x7ffa04198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.850+0000 7ffa02ffd640 1 --2- 172.21.15.33:0/2356612326 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa041017b0 0x7ffa04197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.850+0000 7ffa037fe640 1 -- 172.21.15.33:0/2356612326 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa041017b0 msgr2=0x7ffa04197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:47.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.850+0000 7ffa037fe640 1 --2- 172.21.15.33:0/2356612326 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa041017b0 0x7ffa04197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.850+0000 7ffa037fe640 1 -- 172.21.15.33:0/2356612326 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa04102170 msgr2=0x7ffa04198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:47.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.850+0000 7ffa037fe640 1 --2- 172.21.15.33:0/2356612326 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa04102170 0x7ffa04198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.850+0000 7ffa037fe640 1 -- 172.21.15.33:0/2356612326 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa0419cfb0 con 0x7ffa041077c0 2024-03-19T02:59:47.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.850+0000 7ffa027fc640 1 --2- 172.21.15.33:0/2356612326 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa04102170 0x7ffa04198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:47.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.850+0000 7ffa037fe640 1 --2- 172.21.15.33:0/2356612326 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa041077c0 0x7ffa0419c880 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc00eb10 tx=0x7ff9fc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:47.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.850+0000 7ff9ebfff640 1 -- 172.21.15.33:0/2356612326 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc004b20 con 0x7ffa041077c0 2024-03-19T02:59:47.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.850+0000 7ffa09359640 1 -- 172.21.15.33:0/2356612326 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa0419d2a0 con 0x7ffa041077c0 2024-03-19T02:59:47.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.850+0000 7ff9ebfff640 1 -- 172.21.15.33:0/2356612326 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff9fc004cc0 con 0x7ffa041077c0 2024-03-19T02:59:47.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.850+0000 7ff9ebfff640 1 -- 172.21.15.33:0/2356612326 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc010690 con 0x7ffa041077c0 2024-03-19T02:59:47.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.850+0000 7ffa09359640 1 -- 172.21.15.33:0/2356612326 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa041a4b30 con 0x7ffa041077c0 2024-03-19T02:59:47.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.852+0000 7ff9ebfff640 1 -- 172.21.15.33:0/2356612326 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7ff9fc005020 con 0x7ffa041077c0 2024-03-19T02:59:47.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.853+0000 7ffa0c012740 1 Processor -- start 2024-03-19T02:59:47.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.853+0000 7ffa0c012740 1 -- start start 2024-03-19T02:59:47.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.853+0000 7ffa0c012740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5589ec82ba30 0x5589ec82be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.853+0000 7ffa0c012740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ec82c350 0x5589ec8349f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.853+0000 7ffa0c012740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5589ec834f30 0x5589ec837320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.853+0000 7ffa0c012740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5589ec78f500 con 0x5589ec82c350 2024-03-19T02:59:47.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.853+0000 7ffa0c012740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5589ec737be0 con 0x5589ec834f30 2024-03-19T02:59:47.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.853+0000 7ffa0c012740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5589ec78b5a0 con 0x5589ec82ba30 2024-03-19T02:59:47.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.853+0000 7ffa02ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ec82c350 0x5589ec8349f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.853+0000 7ff9ebfff640 1 --2- 172.21.15.33:0/2356612326 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7ff9ec0a2970 0x7ff9ec0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.853+0000 7ffa02ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ec82c350 0x5589ec8349f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37120/0 (socket says 172.21.15.33:37120) 2024-03-19T02:59:47.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.853+0000 7ffa02ffd640 1 -- 172.21.15.33:0/1782244336 learned_addr learned my addr 172.21.15.33:0/1782244336 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:47.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.853+0000 7ffa027fc640 1 --2- 172.21.15.33:0/1782244336 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5589ec82ba30 0x5589ec82be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.853+0000 7ffa037fe640 1 --2- 172.21.15.33:0/2356612326 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7ff9ec0a2970 0x7ff9ec0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.853+0000 7ffa027fc640 1 --2- 172.21.15.33:0/1782244336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5589ec834f30 0x5589ec837320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.853+0000 7ff9ebfff640 1 -- 172.21.15.33:0/2356612326 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7ff9fc0cf750 con 0x7ffa041077c0 2024-03-19T02:59:47.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.853+0000 7ffa02ffd640 1 -- 172.21.15.33:0/1782244336 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5589ec82ba30 msgr2=0x5589ec82be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:47.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.853+0000 7ffa02ffd640 1 --2- 172.21.15.33:0/1782244336 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5589ec82ba30 0x5589ec82be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.853+0000 7ffa02ffd640 1 -- 172.21.15.33:0/1782244336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5589ec834f30 msgr2=0x5589ec837320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:47.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.853+0000 7ffa02ffd640 1 --2- 172.21.15.33:0/1782244336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5589ec834f30 0x5589ec837320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.853+0000 7ffa02ffd640 1 -- 172.21.15.33:0/1782244336 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589ec759000 con 0x5589ec82c350 2024-03-19T02:59:47.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.853+0000 7ffa027fc640 1 --2- 172.21.15.33:0/1782244336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5589ec834f30 0x5589ec837320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T02:59:47.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.854+0000 7ffa02ffd640 1 --2- 172.21.15.33:0/1782244336 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ec82c350 0x5589ec8349f0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7ff9f4009c70 tx=0x7ff9f4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c05cfd14b065fb3b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:47.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.854+0000 7ffa037fe640 1 --2- 172.21.15.33:0/2356612326 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7ff9ec0a2970 0x7ff9ec0a4e30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc09b9d0 tx=0x7ff9fc090ed0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:47.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.854+0000 7ff9e97fa640 1 -- 172.21.15.33:0/1782244336 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f4004230 con 0x5589ec82c350 2024-03-19T02:59:47.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.854+0000 7ff9e97fa640 1 -- 172.21.15.33:0/1782244336 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff9f40043d0 con 0x5589ec82c350 2024-03-19T02:59:47.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.854+0000 7ff9e97fa640 1 -- 172.21.15.33:0/1782244336 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f40196d0 con 0x5589ec82c350 2024-03-19T02:59:47.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.854+0000 7ffa0c012740 1 -- 172.21.15.33:0/1782244336 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ec82c350 msgr2=0x5589ec8349f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:47.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.854+0000 7ffa0c012740 1 --2- 172.21.15.33:0/1782244336 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ec82c350 0x5589ec8349f0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7ff9f4009c70 tx=0x7ff9f4011810 comp rx=0 tx=0).stop 2024-03-19T02:59:47.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.854+0000 7ffa0c012740 1 -- 172.21.15.33:0/1782244336 shutdown_connections 2024-03-19T02:59:47.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.854+0000 7ffa0c012740 1 --2- 172.21.15.33:0/1782244336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5589ec834f30 0x5589ec837320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.854+0000 7ffa0c012740 1 --2- 172.21.15.33:0/1782244336 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ec82c350 0x5589ec8349f0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.854+0000 7ffa0c012740 1 --2- 172.21.15.33:0/1782244336 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5589ec82ba30 0x5589ec82be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.854+0000 7ffa0c012740 1 -- 172.21.15.33:0/1782244336 >> 172.21.15.33:0/1782244336 conn(0x5589ec826160 msgr2=0x5589ec826540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:47.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.854+0000 7ffa0c012740 1 -- 172.21.15.33:0/1782244336 shutdown_connections 2024-03-19T02:59:47.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.855+0000 7ffa0c012740 1 -- 172.21.15.33:0/1782244336 wait complete. 2024-03-19T02:59:47.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.855+0000 7ffa0c012740 1 -- start start 2024-03-19T02:59:47.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.855+0000 7ffa0c012740 1 Processor -- start 2024-03-19T02:59:47.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.855+0000 7ffa0c012740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5589ec82ba30 0x5589ec831e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.855+0000 7ffa0c012740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5589ec82c350 0x5589ec8323d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.855+0000 7ffa0c012740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ec834f30 0x5589ec832910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.855+0000 7ffa0c012740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5589ec837d20 con 0x5589ec834f30 2024-03-19T02:59:47.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.855+0000 7ffa0c012740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5589ec74b160 con 0x5589ec82c350 2024-03-19T02:59:47.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.855+0000 7ffa0c012740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5589ec837ea0 con 0x5589ec82ba30 2024-03-19T02:59:47.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.855+0000 7ffa0c012740 10 client.0 _ll_register_callbacks cb 0x5589ec787a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:59:47.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.855+0000 7ffa027fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ec834f30 0x5589ec832910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.855+0000 7ffa027fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ec834f30 0x5589ec832910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37132/0 (socket says 172.21.15.33:37132) 2024-03-19T02:59:47.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.855+0000 7ffa027fc640 1 -- 172.21.15.33:0/1002310283 learned_addr learned my addr 172.21.15.33:0/1002310283 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:47.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.855+0000 7ffa02ffd640 1 --2- 172.21.15.33:0/1002310283 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5589ec82c350 0x5589ec8323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.855+0000 7ffa027fc640 1 --2- 172.21.15.33:0/1002310283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5589ec82ba30 0x5589ec831e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.856+0000 7ffa02ffd640 1 -- 172.21.15.33:0/1002310283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5589ec82ba30 msgr2=0x5589ec831e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:47.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.856+0000 7ffa02ffd640 1 --2- 172.21.15.33:0/1002310283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5589ec82ba30 0x5589ec831e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.856+0000 7ffa02ffd640 1 -- 172.21.15.33:0/1002310283 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ec834f30 msgr2=0x5589ec832910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:47.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.856+0000 7ffa02ffd640 1 --2- 172.21.15.33:0/1002310283 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ec834f30 0x5589ec832910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.856+0000 7ffa02ffd640 1 -- 172.21.15.33:0/1002310283 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589ec56d7d0 con 0x5589ec82c350 2024-03-19T02:59:47.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.856+0000 7ffa027fc640 1 --2- 172.21.15.33:0/1002310283 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ec834f30 0x5589ec832910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T02:59:47.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.856+0000 7ffa02ffd640 1 --2- 172.21.15.33:0/1002310283 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5589ec82c350 0x5589ec8323d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff9f4009c70 tx=0x7ff9f4005750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ae517700137739d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:47.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.856+0000 7ff9e6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T02:59:47.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.856+0000 7ff9e6ff5640 1 -- 172.21.15.33:0/1002310283 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f4027020 con 0x5589ec82c350 2024-03-19T02:59:47.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.856+0000 7ffa0c012740 1 -- 172.21.15.33:0/1002310283 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9f40043d0 con 0x5589ec82c350 2024-03-19T02:59:47.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.856+0000 7ff9e6ff5640 1 -- 172.21.15.33:0/1002310283 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7ff9f4003800 con 0x5589ec82c350 2024-03-19T02:59:47.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.856+0000 7ff9e6ff5640 1 -- 172.21.15.33:0/1002310283 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff9f4032050 con 0x5589ec82c350 2024-03-19T02:59:47.863 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.856+0000 7ffa0c012740 10 client.26879 fetch_fsmap learned FSMap version 25 2024-03-19T02:59:47.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.856+0000 7ffa0c012740 1 -- 172.21.15.33:0/1002310283 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5589ec4a33f0 con 0x5589ec82c350 2024-03-19T02:59:47.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.856+0000 7ff9e6ff5640 1 -- 172.21.15.33:0/1002310283 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff9f40043d0 con 0x5589ec82c350 2024-03-19T02:59:47.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.856+0000 7ff9e6ff5640 1 -- 172.21.15.33:0/1002310283 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7ff9f4018070 con 0x5589ec82c350 2024-03-19T02:59:47.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.856+0000 7ffa0c012740 10 client.26879 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:59:47.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.856+0000 7ffa0c012740 10 client.26879 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:59:47.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.857+0000 7ffa0c012740 20 client.26879 populate_metadata read hostname 'smithi033' 2024-03-19T02:59:47.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.857+0000 7ffa0c012740 1 --2- 172.21.15.33:0/1002310283 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5589ec845120 0x5589ec847510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.857+0000 7ffa0c012740 4 client.26879 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T02:59:47.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.857+0000 7ffa0c012740 1 -- 172.21.15.33:0/1002310283 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5589ec79ca40 con 0x5589ec845120 2024-03-19T02:59:47.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.857+0000 7ffa027fc640 1 --2- 172.21.15.33:0/1002310283 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5589ec845120 0x5589ec847510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.857+0000 7ff9e6ff5640 10 client.26879 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:59:47.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.857+0000 7ffa027fc640 1 --2- 172.21.15.33:0/1002310283 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5589ec845120 0x5589ec847510 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a8785836481bf8 server_cookie=d09fbde48c5c3809 in_seq=0 out_seq=0 2024-03-19T02:59:47.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.858+0000 7ff9e6ff5640 1 -- 172.21.15.33:0/1002310283 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5589ec79ca40 con 0x5589ec845120 2024-03-19T02:59:47.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.858+0000 7ff9e6ff5640 10 client.26879 handle_command_reply: tid=0 2024-03-19T02:59:47.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.858+0000 7ffa0c012740 1 client.26879 shutdown 2024-03-19T02:59:47.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.858+0000 7ffa0c012740 1 -- 172.21.15.33:0/1002310283 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5589ec82c350 msgr2=0x5589ec8323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:47.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.858+0000 7ffa0c012740 1 --2- 172.21.15.33:0/1002310283 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5589ec82c350 0x5589ec8323d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff9f4009c70 tx=0x7ff9f4005750 comp rx=0 tx=0).stop 2024-03-19T02:59:47.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.858+0000 7ffa0c012740 1 -- 172.21.15.33:0/1002310283 shutdown_connections 2024-03-19T02:59:47.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.858+0000 7ffa0c012740 1 --2- 172.21.15.33:0/1002310283 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5589ec845120 0x5589ec847510 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.858+0000 7ffa0c012740 1 --2- 172.21.15.33:0/1002310283 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ec834f30 0x5589ec832910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.858+0000 7ff9e6ff5640 0 client.26879 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:59:47.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.858+0000 7ffa0c012740 1 --2- 172.21.15.33:0/1002310283 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5589ec82c350 0x5589ec8323d0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.858+0000 7ffa0c012740 1 --2- 172.21.15.33:0/1002310283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5589ec82ba30 0x5589ec831e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.858+0000 7ffa0c012740 1 -- 172.21.15.33:0/1002310283 >> 172.21.15.33:0/1002310283 conn(0x5589ec826160 msgr2=0x5589ec827eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:47.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.858+0000 7ffa0c012740 1 -- 172.21.15.33:0/1002310283 shutdown_connections 2024-03-19T02:59:47.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.858+0000 7ffa0c012740 1 -- 172.21.15.33:0/1002310283 wait complete. 2024-03-19T02:59:47.865 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.858+0000 7ffa0c012740 20 client.26879 trim_cache size 0 max 16384 2024-03-19T02:59:47.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.875+0000 7ffa0c012740 1 Processor -- start 2024-03-19T02:59:47.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.875+0000 7ffa0c012740 1 -- start start 2024-03-19T02:59:47.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.875+0000 7ffa0c012740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5589ec82ba30 0x5589ec8345c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.875+0000 7ffa0c012740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ec82c350 0x5589ec830dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.875+0000 7ffa0c012740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5589ec834f30 0x5589ec840af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.875+0000 7ffa0c012740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ffa041b1410 con 0x5589ec82c350 2024-03-19T02:59:47.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.875+0000 7ffa0c012740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ffa041aece0 con 0x5589ec834f30 2024-03-19T02:59:47.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.875+0000 7ffa0c012740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ffa041acb10 con 0x5589ec82ba30 2024-03-19T02:59:47.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.875+0000 7ffa02ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ec82c350 0x5589ec830dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.875+0000 7ffa02ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ec82c350 0x5589ec830dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37146/0 (socket says 172.21.15.33:37146) 2024-03-19T02:59:47.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.875+0000 7ffa02ffd640 1 -- 172.21.15.33:0/2129294686 learned_addr learned my addr 172.21.15.33:0/2129294686 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:47.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.876+0000 7ffa027fc640 1 --2- 172.21.15.33:0/2129294686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5589ec82ba30 0x5589ec8345c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.876+0000 7ffa027fc640 1 --2- 172.21.15.33:0/2129294686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5589ec834f30 0x5589ec840af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.876+0000 7ffa02ffd640 1 -- 172.21.15.33:0/2129294686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5589ec82ba30 msgr2=0x5589ec8345c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:47.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.876+0000 7ffa02ffd640 1 --2- 172.21.15.33:0/2129294686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5589ec82ba30 0x5589ec8345c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.876+0000 7ffa02ffd640 1 -- 172.21.15.33:0/2129294686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5589ec834f30 msgr2=0x5589ec840af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:47.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.876+0000 7ffa02ffd640 1 --2- 172.21.15.33:0/2129294686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5589ec834f30 0x5589ec840af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.876+0000 7ffa02ffd640 1 -- 172.21.15.33:0/2129294686 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589ec4630f0 con 0x5589ec82c350 2024-03-19T02:59:47.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.876+0000 7ffa027fc640 1 --2- 172.21.15.33:0/2129294686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5589ec834f30 0x5589ec840af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T02:59:47.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.876+0000 7ffa02ffd640 1 --2- 172.21.15.33:0/2129294686 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ec82c350 0x5589ec830dd0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7ff9f4004300 tx=0x7ff9f40193e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7976aab655aa8117 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:47.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.876+0000 7ff9e8ff9640 1 -- 172.21.15.33:0/2129294686 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f4027070 con 0x5589ec82c350 2024-03-19T02:59:47.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.876+0000 7ff9e8ff9640 1 -- 172.21.15.33:0/2129294686 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff9f40043a0 con 0x5589ec82c350 2024-03-19T02:59:47.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.876+0000 7ff9e8ff9640 1 -- 172.21.15.33:0/2129294686 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f4003a60 con 0x5589ec82c350 2024-03-19T02:59:47.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.877+0000 7ffa0c012740 1 -- 172.21.15.33:0/2129294686 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ec82c350 msgr2=0x5589ec830dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:47.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.877+0000 7ffa0c012740 1 --2- 172.21.15.33:0/2129294686 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ec82c350 0x5589ec830dd0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7ff9f4004300 tx=0x7ff9f40193e0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.877+0000 7ffa0c012740 1 -- 172.21.15.33:0/2129294686 shutdown_connections 2024-03-19T02:59:47.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.877+0000 7ffa0c012740 1 --2- 172.21.15.33:0/2129294686 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5589ec834f30 0x5589ec840af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.877+0000 7ffa0c012740 1 --2- 172.21.15.33:0/2129294686 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ec82c350 0x5589ec830dd0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.877+0000 7ffa0c012740 1 --2- 172.21.15.33:0/2129294686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5589ec82ba30 0x5589ec8345c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.877+0000 7ffa0c012740 1 -- 172.21.15.33:0/2129294686 >> 172.21.15.33:0/2129294686 conn(0x5589ec826160 msgr2=0x5589ec8326e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:47.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.877+0000 7ffa0c012740 1 -- 172.21.15.33:0/2129294686 shutdown_connections 2024-03-19T02:59:47.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.877+0000 7ffa0c012740 1 -- 172.21.15.33:0/2129294686 wait complete. 2024-03-19T02:59:47.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.877+0000 7ffa0c012740 1 -- start start 2024-03-19T02:59:47.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.877+0000 7ffa0c012740 1 Processor -- start 2024-03-19T02:59:47.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.877+0000 7ffa0c012740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ec82ba30 0x5589ec838a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.877+0000 7ffa0c012740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5589ec82c350 0x5589ec838fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.877+0000 7ffa0c012740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5589ec834f30 0x5589ec8394e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.877+0000 7ffa0c012740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ffa041ad1a0 con 0x5589ec82ba30 2024-03-19T02:59:47.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.877+0000 7ffa0c012740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ffa041ae610 con 0x5589ec82c350 2024-03-19T02:59:47.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.877+0000 7ffa0c012740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ffa041ad870 con 0x5589ec834f30 2024-03-19T02:59:47.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.877+0000 7ffa0c012740 10 client.0 _ll_register_callbacks cb 0x7ff9f4009af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T02:59:47.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.877+0000 7ffa027fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ec82ba30 0x5589ec838a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.877+0000 7ffa027fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ec82ba30 0x5589ec838a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:37160/0 (socket says 172.21.15.33:37160) 2024-03-19T02:59:47.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.877+0000 7ffa027fc640 1 -- 172.21.15.33:0/644307145 learned_addr learned my addr 172.21.15.33:0/644307145 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T02:59:47.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.877+0000 7ffa02ffd640 1 --2- 172.21.15.33:0/644307145 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5589ec82c350 0x5589ec838fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.877+0000 7ffa027fc640 1 --2- 172.21.15.33:0/644307145 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5589ec834f30 0x5589ec8394e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.878+0000 7ffa027fc640 1 -- 172.21.15.33:0/644307145 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5589ec834f30 msgr2=0x5589ec8394e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:47.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.878+0000 7ffa027fc640 1 --2- 172.21.15.33:0/644307145 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5589ec834f30 0x5589ec8394e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.878+0000 7ffa027fc640 1 -- 172.21.15.33:0/644307145 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5589ec82c350 msgr2=0x5589ec838fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T02:59:47.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.878+0000 7ffa027fc640 1 --2- 172.21.15.33:0/644307145 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5589ec82c350 0x5589ec838fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.878+0000 7ffa027fc640 1 -- 172.21.15.33:0/644307145 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589ec8aa390 con 0x5589ec82ba30 2024-03-19T02:59:47.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.878+0000 7ffa027fc640 1 --2- 172.21.15.33:0/644307145 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ec82ba30 0x5589ec838a60 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8009230 tx=0x7ff9f4004330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d96357962c8cc47b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T02:59:47.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.878+0000 7ff9e6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T02:59:47.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.878+0000 7ff9e6ff5640 1 -- 172.21.15.33:0/644307145 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f804fba0 con 0x5589ec82ba30 2024-03-19T02:59:47.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.878+0000 7ff9e6ff5640 1 -- 172.21.15.33:0/644307145 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9408+0+0 (secure 0 0 0) 0x7ff9f804fd40 con 0x5589ec82ba30 2024-03-19T02:59:47.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.878+0000 7ffa0c012740 1 -- 172.21.15.33:0/644307145 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9f40043a0 con 0x5589ec82ba30 2024-03-19T02:59:47.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.878+0000 7ff9e6ff5640 1 -- 172.21.15.33:0/644307145 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff9f8064050 con 0x5589ec82ba30 2024-03-19T02:59:47.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.878+0000 7ffa0c012740 10 client.17064 fetch_fsmap learned FSMap version 25 2024-03-19T02:59:47.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.878+0000 7ffa0c012740 1 -- 172.21.15.33:0/644307145 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5589ec8ab680 con 0x5589ec82ba30 2024-03-19T02:59:47.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.878+0000 7ff9e6ff5640 1 -- 172.21.15.33:0/644307145 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff9f40043a0 con 0x5589ec82ba30 2024-03-19T02:59:47.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.878+0000 7ff9e6ff5640 1 -- 172.21.15.33:0/644307145 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7ff9f8003780 con 0x5589ec82ba30 2024-03-19T02:59:47.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.878+0000 7ffa0c012740 10 client.17064 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T02:59:47.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.878+0000 7ffa0c012740 10 client.17064 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T02:59:47.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.878+0000 7ffa0c012740 20 client.17064 populate_metadata read hostname 'smithi033' 2024-03-19T02:59:47.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.878+0000 7ffa0c012740 1 --2- 172.21.15.33:0/644307145 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5589ec8abdb0 0x5589ec8ae1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T02:59:47.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.878+0000 7ffa0c012740 4 client.17064 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T02:59:47.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.878+0000 7ffa0c012740 1 -- 172.21.15.33:0/644307145 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5589ec796d90 con 0x5589ec8abdb0 2024-03-19T02:59:47.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.879+0000 7ffa02ffd640 1 --2- 172.21.15.33:0/644307145 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5589ec8abdb0 0x5589ec8ae1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T02:59:47.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.879+0000 7ffa02ffd640 1 --2- 172.21.15.33:0/644307145 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5589ec8abdb0 0x5589ec8ae1d0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49056109dc377c64 server_cookie=1a8e89769d8bfe4b in_seq=0 out_seq=0 2024-03-19T02:59:47.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.879+0000 7ff9e6ff5640 10 client.17064 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T02:59:47.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.879+0000 7ff9e6ff5640 1 -- 172.21.15.33:0/644307145 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5589ec796d90 con 0x5589ec8abdb0 2024-03-19T02:59:47.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.879+0000 7ff9e6ff5640 10 client.17064 handle_command_reply: tid=0 2024-03-19T02:59:47.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.879+0000 7ffa0c012740 1 client.17064 shutdown 2024-03-19T02:59:47.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.879+0000 7ffa0c012740 1 -- 172.21.15.33:0/644307145 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ec82ba30 msgr2=0x5589ec838a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:47.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.879+0000 7ffa0c012740 1 --2- 172.21.15.33:0/644307145 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ec82ba30 0x5589ec838a60 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8009230 tx=0x7ff9f4004330 comp rx=0 tx=0).stop 2024-03-19T02:59:47.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.879+0000 7ffa0c012740 1 -- 172.21.15.33:0/644307145 shutdown_connections 2024-03-19T02:59:47.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.879+0000 7ffa0c012740 1 --2- 172.21.15.33:0/644307145 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5589ec8abdb0 0x5589ec8ae1d0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.879+0000 7ffa0c012740 1 --2- 172.21.15.33:0/644307145 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5589ec834f30 0x5589ec8394e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.879+0000 7ff9e6ff5640 0 client.17064 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T02:59:47.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.879+0000 7ffa0c012740 1 --2- 172.21.15.33:0/644307145 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5589ec82c350 0x5589ec838fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.879+0000 7ffa0c012740 1 --2- 172.21.15.33:0/644307145 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5589ec82ba30 0x5589ec838a60 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.879+0000 7ffa0c012740 1 -- 172.21.15.33:0/644307145 >> 172.21.15.33:0/644307145 conn(0x5589ec826160 msgr2=0x5589ec831e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:47.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.879+0000 7ffa0c012740 1 -- 172.21.15.33:0/644307145 shutdown_connections 2024-03-19T02:59:47.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.879+0000 7ffa0c012740 1 -- 172.21.15.33:0/644307145 wait complete. 2024-03-19T02:59:47.886 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.879+0000 7ffa0c012740 20 client.17064 trim_cache size 0 max 16384 2024-03-19T02:59:47.887 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T02:59:47.887 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T02:59:47.887 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-03-19T02:59:47.887 INFO:teuthology.orchestra.run.smithi033.stdout: "1327895e-80db-46f1-8e2a-c8a765e2bcbf": { 2024-03-19T02:59:47.887 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-03-19T02:59:47.887 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "1327895e-80db-46f1-8e2a-c8a765e2bcbf", 2024-03-19T02:59:47.887 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-03-19T02:59:47.887 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T02:59:47.887 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T02:59:47.887 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T02:59:47.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.880+0000 7ffa09359640 1 -- 172.21.15.33:0/2356612326 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7ff9ec0a2970 msgr2=0x7ff9ec0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:47.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.880+0000 7ffa09359640 1 --2- 172.21.15.33:0/2356612326 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7ff9ec0a2970 0x7ff9ec0a4e30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc09b9d0 tx=0x7ff9fc090ed0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.880+0000 7ffa09359640 1 -- 172.21.15.33:0/2356612326 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa041077c0 msgr2=0x7ffa0419c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T02:59:47.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.880+0000 7ffa09359640 1 --2- 172.21.15.33:0/2356612326 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa041077c0 0x7ffa0419c880 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc00eb10 tx=0x7ff9fc00c2d0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.880+0000 7ffa09359640 1 -- 172.21.15.33:0/2356612326 shutdown_connections 2024-03-19T02:59:47.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.880+0000 7ffa09359640 1 --2- 172.21.15.33:0/2356612326 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7ff9ec0a2970 0x7ff9ec0a4e30 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.880+0000 7ffa09359640 1 --2- 172.21.15.33:0/2356612326 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffa041077c0 0x7ffa0419c880 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.880+0000 7ffa09359640 1 --2- 172.21.15.33:0/2356612326 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa04102170 0x7ffa04198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.880+0000 7ffa09359640 1 --2- 172.21.15.33:0/2356612326 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa041017b0 0x7ffa04197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T02:59:47.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.880+0000 7ffa09359640 1 -- 172.21.15.33:0/2356612326 >> 172.21.15.33:0/2356612326 conn(0x7ffa04078c20 msgr2=0x7ffa04104260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T02:59:47.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.880+0000 7ffa09359640 1 -- 172.21.15.33:0/2356612326 shutdown_connections 2024-03-19T02:59:47.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T02:59:47.880+0000 7ffa09359640 1 -- 172.21.15.33:0/2356612326 wait complete. 2024-03-19T02:59:47.891 INFO:tasks.cephfs.filesystem:scrub status for tag:1327895e-80db-46f1-8e2a-c8a765e2bcbf - {'path': '/', 'tag': '1327895e-80db-46f1-8e2a-c8a765e2bcbf', 'options': 'recursive,force'} 2024-03-19T02:59:48.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:48 smithi096 ceph-mon[35946]: pgmap v193: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:59:49.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:48 smithi155 ceph-mon[35804]: pgmap v193: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:59:49.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:48 smithi033 ceph-mon[31973]: pgmap v193: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T02:59:49.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:49 smithi096 ceph-mon[35946]: scrub queued for path: / 2024-03-19T02:59:49.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:49 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T02:59:49.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:49 smithi096 ceph-mon[35946]: scrub summary: active paths [/] 2024-03-19T02:59:49.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:49 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T02:59:50.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:49 smithi155 ceph-mon[35804]: scrub queued for path: / 2024-03-19T02:59:50.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:49 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T02:59:50.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:49 smithi155 ceph-mon[35804]: scrub summary: active paths [/] 2024-03-19T02:59:50.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:49 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T02:59:50.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:49 smithi033 ceph-mon[31973]: scrub queued for path: / 2024-03-19T02:59:50.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:49 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T02:59:50.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:49 smithi033 ceph-mon[31973]: scrub summary: active paths [/] 2024-03-19T02:59:50.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:49 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T02:59:50.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:50 smithi096 ceph-mon[35946]: pgmap v194: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:59:51.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:50 smithi155 ceph-mon[35804]: pgmap v194: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:59:51.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:50 smithi033 ceph-mon[31973]: pgmap v194: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:59:52.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:52 smithi096 ceph-mon[35946]: pgmap v195: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:59:53.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:52 smithi155 ceph-mon[35804]: pgmap v195: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:59:53.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:52 smithi033 ceph-mon[31973]: pgmap v195: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:59:53.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:53 smithi096 ceph-mon[35946]: scrub summary: idle 2024-03-19T02:59:54.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:53 smithi155 ceph-mon[35804]: scrub summary: idle 2024-03-19T02:59:54.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:53 smithi033 ceph-mon[31973]: scrub summary: idle 2024-03-19T02:59:54.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:54 smithi096 ceph-mon[35946]: pgmap v196: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:59:55.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:54 smithi155 ceph-mon[35804]: pgmap v196: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:59:55.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:54 smithi033 ceph-mon[31973]: pgmap v196: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T02:59:56.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:56 smithi096 ceph-mon[35946]: pgmap v197: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:59:57.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:56 smithi155 ceph-mon[35804]: pgmap v197: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:59:57.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:56 smithi033 ceph-mon[31973]: pgmap v197: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:59:58.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 02:59:58 smithi096 ceph-mon[35946]: pgmap v198: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:59:59.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 02:59:58 smithi155 ceph-mon[35804]: pgmap v198: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T02:59:59.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 02:59:58 smithi033 ceph-mon[31973]: pgmap v198: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:00:00.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:00 smithi096 ceph-mon[35946]: pgmap v199: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:00:00.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:00 smithi096 ceph-mon[35946]: overall HEALTH_OK 2024-03-19T03:00:01.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:00 smithi155 ceph-mon[35804]: pgmap v199: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:00:01.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:00 smithi155 ceph-mon[35804]: overall HEALTH_OK 2024-03-19T03:00:01.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:00 smithi033 ceph-mon[31973]: pgmap v199: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:00:01.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:00 smithi033 ceph-mon[31973]: overall HEALTH_OK 2024-03-19T03:00:02.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:02 smithi096 ceph-mon[35946]: pgmap v200: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:00:03.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:02 smithi155 ceph-mon[35804]: pgmap v200: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:00:03.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:02 smithi033 ceph-mon[31973]: pgmap v200: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:00:04.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:04 smithi096 ceph-mon[35946]: pgmap v201: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-19T03:00:05.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:04 smithi155 ceph-mon[35804]: pgmap v201: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-19T03:00:05.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:04 smithi033 ceph-mon[31973]: pgmap v201: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-19T03:00:06.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:06 smithi096 ceph-mon[35946]: pgmap v202: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-19T03:00:07.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:06 smithi155 ceph-mon[35804]: pgmap v202: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-19T03:00:07.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:06 smithi033 ceph-mon[31973]: pgmap v202: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-19T03:00:08.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:08 smithi096 ceph-mon[35946]: pgmap v203: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-19T03:00:08.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:08 smithi096 ceph-mon[35946]: osdmap e99: 12 total, 12 up, 12 in 2024-03-19T03:00:09.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:08 smithi155 ceph-mon[35804]: pgmap v203: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-19T03:00:09.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:08 smithi155 ceph-mon[35804]: osdmap e99: 12 total, 12 up, 12 in 2024-03-19T03:00:09.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:08 smithi033 ceph-mon[31973]: pgmap v203: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-19T03:00:09.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:08 smithi033 ceph-mon[31973]: osdmap e99: 12 total, 12 up, 12 in 2024-03-19T03:00:10.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:10 smithi096 ceph-mon[35946]: pgmap v205: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-19T03:00:10.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:10 smithi096 ceph-mon[35946]: osdmap e100: 12 total, 12 up, 12 in 2024-03-19T03:00:11.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:10 smithi155 ceph-mon[35804]: pgmap v205: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-19T03:00:11.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:10 smithi155 ceph-mon[35804]: osdmap e100: 12 total, 12 up, 12 in 2024-03-19T03:00:11.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:10 smithi033 ceph-mon[31973]: pgmap v205: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-19T03:00:11.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:10 smithi033 ceph-mon[31973]: osdmap e100: 12 total, 12 up, 12 in 2024-03-19T03:00:12.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:12 smithi096 ceph-mon[35946]: pgmap v207: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-19T03:00:13.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:12 smithi155 ceph-mon[35804]: pgmap v207: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-19T03:00:13.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:12 smithi033 ceph-mon[31973]: pgmap v207: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-03-19T03:00:14.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:14 smithi096 ceph-mon[35946]: pgmap v208: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-19T03:00:15.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:14 smithi155 ceph-mon[35804]: pgmap v208: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-19T03:00:15.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:14 smithi033 ceph-mon[31973]: pgmap v208: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-19T03:00:16.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:16 smithi096 ceph-mon[35946]: pgmap v209: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-19T03:00:17.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:16 smithi155 ceph-mon[35804]: pgmap v209: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-19T03:00:17.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:16 smithi033 ceph-mon[31973]: pgmap v209: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-19T03:00:17.892 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T03:00:17.893 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T03:00:18.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.012+0000 7f77ee710640 1 -- 172.21.15.33:0/2499460851 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77e8068a20 msgr2=0x7f77e8068e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.012+0000 7f77ee710640 1 --2- 172.21.15.33:0/2499460851 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77e8068a20 0x7f77e8068e00 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f77d8009900 tx=0x7f77d802bcd0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.012+0000 7f77ee710640 1 -- 172.21.15.33:0/2499460851 shutdown_connections 2024-03-19T03:00:18.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.012+0000 7f77ee710640 1 --2- 172.21.15.33:0/2499460851 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77e8069340 0x7f77e810dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.012+0000 7f77ee710640 1 --2- 172.21.15.33:0/2499460851 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77e810e1c0 0x7f77e81105b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.012+0000 7f77ee710640 1 --2- 172.21.15.33:0/2499460851 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77e8068a20 0x7f77e8068e00 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.012+0000 7f77ee710640 1 -- 172.21.15.33:0/2499460851 >> 172.21.15.33:0/2499460851 conn(0x7f77e8077820 msgr2=0x7f77e8077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:18.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.012+0000 7f77ee710640 1 -- 172.21.15.33:0/2499460851 shutdown_connections 2024-03-19T03:00:18.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.012+0000 7f77ee710640 1 -- 172.21.15.33:0/2499460851 wait complete. 2024-03-19T03:00:18.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.012+0000 7f77ee710640 1 Processor -- start 2024-03-19T03:00:18.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.012+0000 7f77ee710640 1 -- start start 2024-03-19T03:00:18.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.013+0000 7f77ee710640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77e8068a20 0x7f77e819a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.013+0000 7f77ee710640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77e8069340 0x7f77e819a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.013+0000 7f77ee710640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77e810e1c0 0x7f77e819e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.016 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.013+0000 7f77ee710640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f77e8113c90 con 0x7f77e810e1c0 2024-03-19T03:00:18.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.013+0000 7f77ee710640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f77e8113b10 con 0x7f77e8069340 2024-03-19T03:00:18.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.013+0000 7f77ee710640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f77e8113e10 con 0x7f77e8068a20 2024-03-19T03:00:18.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.013+0000 7f77eccad640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77e810e1c0 0x7f77e819e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.013+0000 7f77eccad640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77e810e1c0 0x7f77e819e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42240/0 (socket says 172.21.15.33:42240) 2024-03-19T03:00:18.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.013+0000 7f77eccad640 1 -- 172.21.15.33:0/691636624 learned_addr learned my addr 172.21.15.33:0/691636624 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:18.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.013+0000 7f77e77fe640 1 --2- 172.21.15.33:0/691636624 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77e8069340 0x7f77e819a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.013+0000 7f77e7fff640 1 --2- 172.21.15.33:0/691636624 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77e8068a20 0x7f77e819a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.013+0000 7f77eccad640 1 -- 172.21.15.33:0/691636624 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77e8068a20 msgr2=0x7f77e819a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.013+0000 7f77eccad640 1 --2- 172.21.15.33:0/691636624 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77e8068a20 0x7f77e819a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.013+0000 7f77eccad640 1 -- 172.21.15.33:0/691636624 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77e8069340 msgr2=0x7f77e819a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.013+0000 7f77eccad640 1 --2- 172.21.15.33:0/691636624 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77e8069340 0x7f77e819a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.013+0000 7f77eccad640 1 -- 172.21.15.33:0/691636624 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77e819f120 con 0x7f77e810e1c0 2024-03-19T03:00:18.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.013+0000 7f77eccad640 1 --2- 172.21.15.33:0/691636624 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77e810e1c0 0x7f77e819e9f0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f77e000eb10 tx=0x7f77e000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:18.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.014+0000 7f77e57fa640 1 -- 172.21.15.33:0/691636624 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e0004b20 con 0x7f77e810e1c0 2024-03-19T03:00:18.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.014+0000 7f77e57fa640 1 -- 172.21.15.33:0/691636624 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f77e0004cc0 con 0x7f77e810e1c0 2024-03-19T03:00:18.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.014+0000 7f77ee710640 1 -- 172.21.15.33:0/691636624 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77e819f410 con 0x7f77e810e1c0 2024-03-19T03:00:18.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.014+0000 7f77e57fa640 1 -- 172.21.15.33:0/691636624 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e0010690 con 0x7f77e810e1c0 2024-03-19T03:00:18.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.014+0000 7f77ee710640 1 -- 172.21.15.33:0/691636624 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77e81a6ca0 con 0x7f77e810e1c0 2024-03-19T03:00:18.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.016+0000 7f77e57fa640 1 -- 172.21.15.33:0/691636624 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f77e0005020 con 0x7f77e810e1c0 2024-03-19T03:00:18.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.017+0000 7f77f13de740 1 Processor -- start 2024-03-19T03:00:18.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.017+0000 7f77f13de740 1 -- start start 2024-03-19T03:00:18.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.017+0000 7f77e57fa640 1 --2- 172.21.15.33:0/691636624 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f77d00a2970 0x7f77d00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.017+0000 7f77f13de740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564cb3ffea30 0x564cb3ffee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.017+0000 7f77f13de740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cb3fff350 0x564cb40079f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.017+0000 7f77f13de740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cb4007f30 0x564cb400a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.017+0000 7f77f13de740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564cb3f62500 con 0x564cb3fff350 2024-03-19T03:00:18.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.017+0000 7f77f13de740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x564cb3f0abe0 con 0x564cb4007f30 2024-03-19T03:00:18.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.017+0000 7f77f13de740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564cb3f5e5a0 con 0x564cb3ffea30 2024-03-19T03:00:18.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.017+0000 7f77e77fe640 1 --2- 172.21.15.33:0/691636624 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f77d00a2970 0x7f77d00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.017+0000 7f77e57fa640 1 -- 172.21.15.33:0/691636624 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7f77e00cf950 con 0x7f77e810e1c0 2024-03-19T03:00:18.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.017+0000 7f77e77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cb3fff350 0x564cb40079f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.017+0000 7f77e7fff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564cb3ffea30 0x564cb3ffee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.017+0000 7f77e77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cb3fff350 0x564cb40079f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42254/0 (socket says 172.21.15.33:42254) 2024-03-19T03:00:18.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.017+0000 7f77e77fe640 1 -- 172.21.15.33:0/1625845113 learned_addr learned my addr 172.21.15.33:0/1625845113 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:18.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.017+0000 7f77eccad640 1 --2- 172.21.15.33:0/1625845113 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cb4007f30 0x564cb400a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.018+0000 7f77e7fff640 1 -- 172.21.15.33:0/1625845113 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cb4007f30 msgr2=0x564cb400a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:18.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.018+0000 7f77e7fff640 1 --2- 172.21.15.33:0/1625845113 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cb4007f30 0x564cb400a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.018+0000 7f77e7fff640 1 -- 172.21.15.33:0/1625845113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cb3fff350 msgr2=0x564cb40079f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:18.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.018+0000 7f77e7fff640 1 --2- 172.21.15.33:0/1625845113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cb3fff350 0x564cb40079f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.018+0000 7f77e7fff640 1 -- 172.21.15.33:0/1625845113 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cb3f2c000 con 0x564cb3ffea30 2024-03-19T03:00:18.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.018+0000 7f77eccad640 1 --2- 172.21.15.33:0/1625845113 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cb4007f30 0x564cb400a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:18.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.018+0000 7f77e77fe640 1 --2- 172.21.15.33:0/691636624 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f77d00a2970 0x7f77d00a4e30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f77dc005880 tx=0x7f77dc009b50 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:18.026 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.018+0000 7f77e7fff640 1 --2- 172.21.15.33:0/1625845113 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564cb3ffea30 0x564cb3ffee10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f77d802c210 tx=0x7f77d80095e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b67c4fa25cc94ae server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:18.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.018+0000 7f77ce7fc640 1 -- 172.21.15.33:0/1625845113 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d8004040 con 0x564cb3ffea30 2024-03-19T03:00:18.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.018+0000 7f77ce7fc640 1 -- 172.21.15.33:0/1625845113 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f77d802cd30 con 0x564cb3ffea30 2024-03-19T03:00:18.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.018+0000 7f77ce7fc640 1 -- 172.21.15.33:0/1625845113 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d80053a0 con 0x564cb3ffea30 2024-03-19T03:00:18.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.019+0000 7f77f13de740 1 -- 172.21.15.33:0/1625845113 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564cb3ffea30 msgr2=0x564cb3ffee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.019+0000 7f77f13de740 1 --2- 172.21.15.33:0/1625845113 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564cb3ffea30 0x564cb3ffee10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f77d802c210 tx=0x7f77d80095e0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.019+0000 7f77f13de740 1 -- 172.21.15.33:0/1625845113 shutdown_connections 2024-03-19T03:00:18.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.019+0000 7f77f13de740 1 --2- 172.21.15.33:0/1625845113 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cb4007f30 0x564cb400a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.019+0000 7f77f13de740 1 --2- 172.21.15.33:0/1625845113 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cb3fff350 0x564cb40079f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.019+0000 7f77f13de740 1 --2- 172.21.15.33:0/1625845113 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564cb3ffea30 0x564cb3ffee10 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.019+0000 7f77f13de740 1 -- 172.21.15.33:0/1625845113 >> 172.21.15.33:0/1625845113 conn(0x564cb3ff9160 msgr2=0x564cb3ff9540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:18.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.019+0000 7f77f13de740 1 -- 172.21.15.33:0/1625845113 shutdown_connections 2024-03-19T03:00:18.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.019+0000 7f77f13de740 1 -- 172.21.15.33:0/1625845113 wait complete. 2024-03-19T03:00:18.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.019+0000 7f77f13de740 1 -- start start 2024-03-19T03:00:18.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.019+0000 7f77f13de740 1 Processor -- start 2024-03-19T03:00:18.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.020+0000 7f77f13de740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cb3ffea30 0x564cb4004e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.020+0000 7f77f13de740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564cb3fff350 0x564cb4005350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.020+0000 7f77f13de740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cb4007f30 0x564cb4005890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.020+0000 7f77f13de740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564cb400acc0 con 0x564cb3ffea30 2024-03-19T03:00:18.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.020+0000 7f77f13de740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x564cb3f1e160 con 0x564cb4007f30 2024-03-19T03:00:18.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.020+0000 7f77f13de740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564cb400ae40 con 0x564cb3fff350 2024-03-19T03:00:18.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.020+0000 7f77f13de740 10 client.0 _ll_register_callbacks cb 0x564cb3f5aa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:00:18.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.020+0000 7f77e7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cb3ffea30 0x564cb4004e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.020+0000 7f77e7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cb3ffea30 0x564cb4004e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42270/0 (socket says 172.21.15.33:42270) 2024-03-19T03:00:18.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.020+0000 7f77e7fff640 1 -- 172.21.15.33:0/2439383940 learned_addr learned my addr 172.21.15.33:0/2439383940 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:18.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.020+0000 7f77e77fe640 1 --2- 172.21.15.33:0/2439383940 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564cb3fff350 0x564cb4005350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.020+0000 7f77eccad640 1 --2- 172.21.15.33:0/2439383940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cb4007f30 0x564cb4005890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.020+0000 7f77e7fff640 1 -- 172.21.15.33:0/2439383940 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564cb3fff350 msgr2=0x564cb4005350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:18.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.020+0000 7f77e7fff640 1 --2- 172.21.15.33:0/2439383940 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564cb3fff350 0x564cb4005350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.020+0000 7f77e7fff640 1 -- 172.21.15.33:0/2439383940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cb4007f30 msgr2=0x564cb4005890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:18.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.020+0000 7f77e7fff640 1 --2- 172.21.15.33:0/2439383940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cb4007f30 0x564cb4005890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.020+0000 7f77e7fff640 1 -- 172.21.15.33:0/2439383940 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cb3d407d0 con 0x564cb3ffea30 2024-03-19T03:00:18.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.020+0000 7f77e7fff640 1 --2- 172.21.15.33:0/2439383940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cb3ffea30 0x564cb4004e10 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f77d8036040 tx=0x7f77d8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22f04ba5102bf3d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:18.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.020+0000 7f77c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:00:18.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.020+0000 7f77c77fe640 1 -- 172.21.15.33:0/2439383940 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d8005320 con 0x564cb3ffea30 2024-03-19T03:00:18.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.020+0000 7f77c77fe640 1 -- 172.21.15.33:0/2439383940 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7f77d80054c0 con 0x564cb3ffea30 2024-03-19T03:00:18.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.021+0000 7f77f13de740 1 -- 172.21.15.33:0/2439383940 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77d802cd30 con 0x564cb3ffea30 2024-03-19T03:00:18.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.021+0000 7f77c77fe640 1 -- 172.21.15.33:0/2439383940 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f77d80046c0 con 0x564cb3ffea30 2024-03-19T03:00:18.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.021+0000 7f77f13de740 10 client.17076 fetch_fsmap learned FSMap version 25 2024-03-19T03:00:18.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.021+0000 7f77f13de740 1 -- 172.21.15.33:0/2439383940 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564cb3c763f0 con 0x564cb3ffea30 2024-03-19T03:00:18.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.021+0000 7f77c77fe640 1 -- 172.21.15.33:0/2439383940 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f77d802cd30 con 0x564cb3ffea30 2024-03-19T03:00:18.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.021+0000 7f77c77fe640 1 -- 172.21.15.33:0/2439383940 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f77d8003a30 con 0x564cb3ffea30 2024-03-19T03:00:18.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.021+0000 7f77f13de740 10 client.17076 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:00:18.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.021+0000 7f77f13de740 10 client.17076 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:00:18.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.021+0000 7f77f13de740 20 client.17076 populate_metadata read hostname 'smithi033' 2024-03-19T03:00:18.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.021+0000 7f77f13de740 1 --2- 172.21.15.33:0/2439383940 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x564cb4017f90 0x564cb401a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.029 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.021+0000 7f77f13de740 4 client.17076 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:00:18.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.021+0000 7f77f13de740 1 -- 172.21.15.33:0/2439383940 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564cb3f6fa40 con 0x564cb4017f90 2024-03-19T03:00:18.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.021+0000 7f77e77fe640 1 --2- 172.21.15.33:0/2439383940 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x564cb4017f90 0x564cb401a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.022+0000 7f77e77fe640 1 --2- 172.21.15.33:0/2439383940 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x564cb4017f90 0x564cb401a380 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37cc98ad5345a730 server_cookie=efac5302f64e4fd9 in_seq=0 out_seq=0 2024-03-19T03:00:18.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.022+0000 7f77c77fe640 10 client.17076 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:00:18.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.023+0000 7f77c77fe640 1 -- 172.21.15.33:0/2439383940 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564cb3f6fa40 con 0x564cb4017f90 2024-03-19T03:00:18.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.023+0000 7f77c77fe640 10 client.17076 handle_command_reply: tid=0 2024-03-19T03:00:18.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.023+0000 7f77f13de740 1 client.17076 shutdown 2024-03-19T03:00:18.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.023+0000 7f77f13de740 1 -- 172.21.15.33:0/2439383940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cb3ffea30 msgr2=0x564cb4004e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.023+0000 7f77f13de740 1 --2- 172.21.15.33:0/2439383940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cb3ffea30 0x564cb4004e10 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f77d8036040 tx=0x7f77d8004270 comp rx=0 tx=0).stop 2024-03-19T03:00:18.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.023+0000 7f77f13de740 1 -- 172.21.15.33:0/2439383940 shutdown_connections 2024-03-19T03:00:18.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.023+0000 7f77f13de740 1 --2- 172.21.15.33:0/2439383940 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x564cb4017f90 0x564cb401a380 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.023+0000 7f77f13de740 1 --2- 172.21.15.33:0/2439383940 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564cb3fff350 0x564cb4005350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.023+0000 7f77f13de740 1 --2- 172.21.15.33:0/2439383940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cb4007f30 0x564cb4005890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.023+0000 7f77c77fe640 0 client.17076 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:00:18.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.023+0000 7f77f13de740 1 --2- 172.21.15.33:0/2439383940 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cb3ffea30 0x564cb4004e10 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.023+0000 7f77f13de740 1 -- 172.21.15.33:0/2439383940 >> 172.21.15.33:0/2439383940 conn(0x564cb3ff9160 msgr2=0x564cb3ffaeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:18.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.023+0000 7f77f13de740 1 -- 172.21.15.33:0/2439383940 shutdown_connections 2024-03-19T03:00:18.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.023+0000 7f77f13de740 1 -- 172.21.15.33:0/2439383940 wait complete. 2024-03-19T03:00:18.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.023+0000 7f77f13de740 20 client.17076 trim_cache size 0 max 16384 2024-03-19T03:00:18.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.040+0000 7f77f13de740 1 Processor -- start 2024-03-19T03:00:18.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.040+0000 7f77f13de740 1 -- start start 2024-03-19T03:00:18.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.041+0000 7f77f13de740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564cb3ffea30 0x564cb4007540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.041+0000 7f77f13de740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cb3fff350 0x564cb4013a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.041+0000 7f77f13de740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cb4007f30 0x564cb401f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.041+0000 7f77f13de740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f77e81b3630 con 0x564cb3fff350 2024-03-19T03:00:18.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.041+0000 7f77f13de740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f77e81b0f00 con 0x564cb4007f30 2024-03-19T03:00:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.041+0000 7f77f13de740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f77e81aed70 con 0x564cb3ffea30 2024-03-19T03:00:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.041+0000 7f77e77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cb3fff350 0x564cb4013a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.041+0000 7f77e77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cb3fff350 0x564cb4013a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42282/0 (socket says 172.21.15.33:42282) 2024-03-19T03:00:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.041+0000 7f77e77fe640 1 -- 172.21.15.33:0/1351692914 learned_addr learned my addr 172.21.15.33:0/1351692914 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.041+0000 7f77eccad640 1 --2- 172.21.15.33:0/1351692914 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cb4007f30 0x564cb401f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.041+0000 7f77e7fff640 1 --2- 172.21.15.33:0/1351692914 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564cb3ffea30 0x564cb4007540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.041+0000 7f77e77fe640 1 -- 172.21.15.33:0/1351692914 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564cb3ffea30 msgr2=0x564cb4007540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.041+0000 7f77e77fe640 1 --2- 172.21.15.33:0/1351692914 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564cb3ffea30 0x564cb4007540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.041+0000 7f77e77fe640 1 -- 172.21.15.33:0/1351692914 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cb4007f30 msgr2=0x564cb401f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.041+0000 7f77e77fe640 1 --2- 172.21.15.33:0/1351692914 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cb4007f30 0x564cb401f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.041+0000 7f77e77fe640 1 -- 172.21.15.33:0/1351692914 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cb3fff180 con 0x564cb3fff350 2024-03-19T03:00:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.041+0000 7f77e77fe640 1 --2- 172.21.15.33:0/1351692914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cb3fff350 0x564cb4013a60 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f77dc005850 tx=0x7f77dc021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7576e979ce03722 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.041+0000 7f77cdffb640 1 -- 172.21.15.33:0/1351692914 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77dc009780 con 0x564cb3fff350 2024-03-19T03:00:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.041+0000 7f77cdffb640 1 -- 172.21.15.33:0/1351692914 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f77dc030070 con 0x564cb3fff350 2024-03-19T03:00:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.041+0000 7f77cdffb640 1 -- 172.21.15.33:0/1351692914 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77dc00fcb0 con 0x564cb3fff350 2024-03-19T03:00:18.048 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.042+0000 7f77f13de740 1 -- 172.21.15.33:0/1351692914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cb3fff350 msgr2=0x564cb4013a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.042+0000 7f77f13de740 1 --2- 172.21.15.33:0/1351692914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cb3fff350 0x564cb4013a60 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f77dc005850 tx=0x7f77dc021000 comp rx=0 tx=0).stop 2024-03-19T03:00:18.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.042+0000 7f77f13de740 1 -- 172.21.15.33:0/1351692914 shutdown_connections 2024-03-19T03:00:18.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.042+0000 7f77f13de740 1 --2- 172.21.15.33:0/1351692914 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cb4007f30 0x564cb401f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.042+0000 7f77f13de740 1 --2- 172.21.15.33:0/1351692914 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cb3fff350 0x564cb4013a60 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.042+0000 7f77f13de740 1 --2- 172.21.15.33:0/1351692914 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564cb3ffea30 0x564cb4007540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.042+0000 7f77f13de740 1 -- 172.21.15.33:0/1351692914 >> 172.21.15.33:0/1351692914 conn(0x564cb3ff9160 msgr2=0x564cb3ffa540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:18.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.042+0000 7f77f13de740 1 -- 172.21.15.33:0/1351692914 shutdown_connections 2024-03-19T03:00:18.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.042+0000 7f77f13de740 1 -- 172.21.15.33:0/1351692914 wait complete. 2024-03-19T03:00:18.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.042+0000 7f77f13de740 1 -- start start 2024-03-19T03:00:18.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.042+0000 7f77f13de740 1 Processor -- start 2024-03-19T03:00:18.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.043+0000 7f77f13de740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564cb3ffea30 0x564cb4001900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.043+0000 7f77f13de740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cb3fff350 0x564cb4001e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.043+0000 7f77f13de740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cb4007f30 0x564cb4002380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.043+0000 7f77f13de740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f77e81af3c0 con 0x564cb3fff350 2024-03-19T03:00:18.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.043+0000 7f77f13de740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f77e81b0830 con 0x564cb4007f30 2024-03-19T03:00:18.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.043+0000 7f77f13de740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f77e81afa90 con 0x564cb3ffea30 2024-03-19T03:00:18.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.043+0000 7f77f13de740 10 client.0 _ll_register_callbacks cb 0x7f77dc011500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:00:18.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.043+0000 7f77e77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cb3fff350 0x564cb4001e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.043+0000 7f77e7fff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564cb3ffea30 0x564cb4001900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.043+0000 7f77e77fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cb3fff350 0x564cb4001e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42296/0 (socket says 172.21.15.33:42296) 2024-03-19T03:00:18.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.043+0000 7f77e77fe640 1 -- 172.21.15.33:0/2653559906 learned_addr learned my addr 172.21.15.33:0/2653559906 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:18.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.043+0000 7f77eccad640 1 --2- 172.21.15.33:0/2653559906 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cb4007f30 0x564cb4002380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.043+0000 7f77e77fe640 1 -- 172.21.15.33:0/2653559906 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564cb3ffea30 msgr2=0x564cb4001900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:18.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.043+0000 7f77e77fe640 1 --2- 172.21.15.33:0/2653559906 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564cb3ffea30 0x564cb4001900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.043+0000 7f77e77fe640 1 -- 172.21.15.33:0/2653559906 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cb4007f30 msgr2=0x564cb4002380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:18.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.043+0000 7f77e77fe640 1 --2- 172.21.15.33:0/2653559906 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cb4007f30 0x564cb4002380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.043+0000 7f77e77fe640 1 -- 172.21.15.33:0/2653559906 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cb3c360f0 con 0x564cb3fff350 2024-03-19T03:00:18.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.043+0000 7f77e7fff640 1 --2- 172.21.15.33:0/2653559906 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564cb3ffea30 0x564cb4001900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:18.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.043+0000 7f77eccad640 1 --2- 172.21.15.33:0/2653559906 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cb4007f30 0x564cb4002380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:18.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.043+0000 7f77e77fe640 1 --2- 172.21.15.33:0/2653559906 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cb3fff350 0x564cb4001e40 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f77dc00f860 tx=0x7f77dc00f890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ec7eaca52f137ea server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:18.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.043+0000 7f77c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:00:18.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.043+0000 7f77c77fe640 1 -- 172.21.15.33:0/2653559906 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77dc039060 con 0x564cb3fff350 2024-03-19T03:00:18.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.044+0000 7f77c77fe640 1 -- 172.21.15.33:0/2653559906 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7f77dc026040 con 0x564cb3fff350 2024-03-19T03:00:18.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.044+0000 7f77f13de740 1 -- 172.21.15.33:0/2653559906 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77dc030070 con 0x564cb3fff350 2024-03-19T03:00:18.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.044+0000 7f77c77fe640 1 -- 172.21.15.33:0/2653559906 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f77dc0112c0 con 0x564cb3fff350 2024-03-19T03:00:18.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.044+0000 7f77f13de740 10 client.17082 fetch_fsmap learned FSMap version 25 2024-03-19T03:00:18.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.044+0000 7f77f13de740 1 -- 172.21.15.33:0/2653559906 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564cb407e060 con 0x564cb3fff350 2024-03-19T03:00:18.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.044+0000 7f77c77fe640 1 -- 172.21.15.33:0/2653559906 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f77dc030070 con 0x564cb3fff350 2024-03-19T03:00:18.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.044+0000 7f77c77fe640 1 -- 172.21.15.33:0/2653559906 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f77dc0035f0 con 0x564cb3fff350 2024-03-19T03:00:18.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.044+0000 7f77f13de740 10 client.17082 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:00:18.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.044+0000 7f77f13de740 10 client.17082 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:00:18.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.044+0000 7f77f13de740 20 client.17082 populate_metadata read hostname 'smithi033' 2024-03-19T03:00:18.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.044+0000 7f77f13de740 1 --2- 172.21.15.33:0/2653559906 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x564cb407e7b0 0x564cb4080bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.044+0000 7f77f13de740 4 client.17082 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T03:00:18.051 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.044+0000 7f77f13de740 1 -- 172.21.15.33:0/2653559906 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564cb3f69d90 con 0x564cb407e7b0 2024-03-19T03:00:18.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.044+0000 7f77e7fff640 1 --2- 172.21.15.33:0/2653559906 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x564cb407e7b0 0x564cb4080bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.045+0000 7f77c77fe640 10 client.17082 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:00:18.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.045+0000 7f77e7fff640 1 --2- 172.21.15.33:0/2653559906 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x564cb407e7b0 0x564cb4080bd0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f41a277df1d22df server_cookie=6ce5c2400cd29e72 in_seq=0 out_seq=0 2024-03-19T03:00:18.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.045+0000 7f77c77fe640 1 -- 172.21.15.33:0/2653559906 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564cb3f69d90 con 0x564cb407e7b0 2024-03-19T03:00:18.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.045+0000 7f77c77fe640 10 client.17082 handle_command_reply: tid=0 2024-03-19T03:00:18.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.045+0000 7f77f13de740 1 client.17082 shutdown 2024-03-19T03:00:18.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.045+0000 7f77f13de740 1 -- 172.21.15.33:0/2653559906 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cb3fff350 msgr2=0x564cb4001e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.045+0000 7f77f13de740 1 --2- 172.21.15.33:0/2653559906 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cb3fff350 0x564cb4001e40 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f77dc00f860 tx=0x7f77dc00f890 comp rx=0 tx=0).stop 2024-03-19T03:00:18.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.045+0000 7f77f13de740 1 -- 172.21.15.33:0/2653559906 shutdown_connections 2024-03-19T03:00:18.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.045+0000 7f77f13de740 1 --2- 172.21.15.33:0/2653559906 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x564cb407e7b0 0x564cb4080bd0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.045+0000 7f77f13de740 1 --2- 172.21.15.33:0/2653559906 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cb4007f30 0x564cb4002380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.045+0000 7f77f13de740 1 --2- 172.21.15.33:0/2653559906 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564cb3fff350 0x564cb4001e40 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.045+0000 7f77c77fe640 0 client.17082 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:00:18.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.045+0000 7f77f13de740 1 --2- 172.21.15.33:0/2653559906 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564cb3ffea30 0x564cb4001900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.045+0000 7f77f13de740 1 -- 172.21.15.33:0/2653559906 >> 172.21.15.33:0/2653559906 conn(0x564cb3ff9160 msgr2=0x564cb3ffadd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:18.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.045+0000 7f77f13de740 1 -- 172.21.15.33:0/2653559906 shutdown_connections 2024-03-19T03:00:18.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.045+0000 7f77f13de740 1 -- 172.21.15.33:0/2653559906 wait complete. 2024-03-19T03:00:18.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.045+0000 7f77f13de740 20 client.17082 trim_cache size 0 max 16384 2024-03-19T03:00:18.053 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T03:00:18.053 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-03-19T03:00:18.053 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-03-19T03:00:18.053 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T03:00:18.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.046+0000 7f77ee710640 1 -- 172.21.15.33:0/691636624 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f77d00a2970 msgr2=0x7f77d00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.046+0000 7f77ee710640 1 --2- 172.21.15.33:0/691636624 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f77d00a2970 0x7f77d00a4e30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f77dc005880 tx=0x7f77dc009b50 comp rx=0 tx=0).stop 2024-03-19T03:00:18.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.046+0000 7f77ee710640 1 -- 172.21.15.33:0/691636624 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77e810e1c0 msgr2=0x7f77e819e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.046+0000 7f77ee710640 1 --2- 172.21.15.33:0/691636624 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77e810e1c0 0x7f77e819e9f0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f77e000eb10 tx=0x7f77e000c2d0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.046+0000 7f77ee710640 1 -- 172.21.15.33:0/691636624 shutdown_connections 2024-03-19T03:00:18.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.046+0000 7f77ee710640 1 --2- 172.21.15.33:0/691636624 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f77d00a2970 0x7f77d00a4e30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.046+0000 7f77ee710640 1 --2- 172.21.15.33:0/691636624 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f77e810e1c0 0x7f77e819e9f0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.046+0000 7f77ee710640 1 --2- 172.21.15.33:0/691636624 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f77e8069340 0x7f77e819a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.046+0000 7f77ee710640 1 --2- 172.21.15.33:0/691636624 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77e8068a20 0x7f77e819a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.046+0000 7f77ee710640 1 -- 172.21.15.33:0/691636624 >> 172.21.15.33:0/691636624 conn(0x7f77e8077820 msgr2=0x7f77e810f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:18.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.046+0000 7f77ee710640 1 -- 172.21.15.33:0/691636624 shutdown_connections 2024-03-19T03:00:18.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.046+0000 7f77ee710640 1 -- 172.21.15.33:0/691636624 wait complete. 2024-03-19T03:00:18.058 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T03:00:18.058 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:00:18.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.160+0000 7fd922310640 1 -- 172.21.15.33:0/1652357253 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd91c105180 msgr2=0x7fd91c10cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.160+0000 7fd922310640 1 --2- 172.21.15.33:0/1652357253 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd91c105180 0x7fd91c10cd50 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fd914009ca0 tx=0x7fd91402c0f0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.160+0000 7fd922310640 1 -- 172.21.15.33:0/1652357253 shutdown_connections 2024-03-19T03:00:18.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.160+0000 7fd922310640 1 --2- 172.21.15.33:0/1652357253 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd91c100ab0 0x7fd91c104aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.160+0000 7fd922310640 1 --2- 172.21.15.33:0/1652357253 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd91c105180 0x7fd91c10cd50 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.160+0000 7fd922310640 1 --2- 172.21.15.33:0/1652357253 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd91c100100 0x7fd91c1004e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.160+0000 7fd922310640 1 -- 172.21.15.33:0/1652357253 >> 172.21.15.33:0/1652357253 conn(0x7fd91c076cb0 msgr2=0x7fd91c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:18.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.160+0000 7fd922310640 1 -- 172.21.15.33:0/1652357253 shutdown_connections 2024-03-19T03:00:18.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.160+0000 7fd922310640 1 -- 172.21.15.33:0/1652357253 wait complete. 2024-03-19T03:00:18.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.161+0000 7fd922310640 1 Processor -- start 2024-03-19T03:00:18.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.161+0000 7fd922310640 1 -- start start 2024-03-19T03:00:18.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.161+0000 7fd922310640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd91c100100 0x7fd91c198e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.161+0000 7fd922310640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd91c105180 0x7fd91c199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.161+0000 7fd922310640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd91c1998a0 0x7fd91c1a3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.161+0000 7fd922310640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd91c110960 con 0x7fd91c100100 2024-03-19T03:00:18.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.161+0000 7fd922310640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd91c1107e0 con 0x7fd91c105180 2024-03-19T03:00:18.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.161+0000 7fd922310640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd91c110ae0 con 0x7fd91c1998a0 2024-03-19T03:00:18.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.161+0000 7fd91bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd91c100100 0x7fd91c198e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.161+0000 7fd91bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd91c100100 0x7fd91c198e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42306/0 (socket says 172.21.15.33:42306) 2024-03-19T03:00:18.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.161+0000 7fd91bfff640 1 -- 172.21.15.33:0/430109199 learned_addr learned my addr 172.21.15.33:0/430109199 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:18.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.161+0000 7fd91b7fe640 1 --2- 172.21.15.33:0/430109199 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd91c105180 0x7fd91c199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.161+0000 7fd9208ad640 1 --2- 172.21.15.33:0/430109199 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd91c1998a0 0x7fd91c1a3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.161+0000 7fd91bfff640 1 -- 172.21.15.33:0/430109199 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd91c1998a0 msgr2=0x7fd91c1a3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.161+0000 7fd91bfff640 1 --2- 172.21.15.33:0/430109199 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd91c1998a0 0x7fd91c1a3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.161+0000 7fd91bfff640 1 -- 172.21.15.33:0/430109199 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd91c105180 msgr2=0x7fd91c199360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.161+0000 7fd91bfff640 1 --2- 172.21.15.33:0/430109199 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd91c105180 0x7fd91c199360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.161+0000 7fd91bfff640 1 -- 172.21.15.33:0/430109199 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd91c1a4640 con 0x7fd91c100100 2024-03-19T03:00:18.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.162+0000 7fd91bfff640 1 --2- 172.21.15.33:0/430109199 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd91c100100 0x7fd91c198e20 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fd90c00e9f0 tx=0x7fd90c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:18.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.162+0000 7fd9197fa640 1 -- 172.21.15.33:0/430109199 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd90c00cd80 con 0x7fd91c100100 2024-03-19T03:00:18.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.162+0000 7fd9197fa640 1 -- 172.21.15.33:0/430109199 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd90c0108e0 con 0x7fd91c100100 2024-03-19T03:00:18.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.162+0000 7fd922310640 1 -- 172.21.15.33:0/430109199 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd91c1a4930 con 0x7fd91c100100 2024-03-19T03:00:18.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.162+0000 7fd9197fa640 1 -- 172.21.15.33:0/430109199 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd90c018920 con 0x7fd91c100100 2024-03-19T03:00:18.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.162+0000 7fd922310640 1 -- 172.21.15.33:0/430109199 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd91c1a4e80 con 0x7fd91c100100 2024-03-19T03:00:18.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.163+0000 7fd922310640 1 -- 172.21.15.33:0/430109199 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd91c068a60 con 0x7fd91c100100 2024-03-19T03:00:18.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.164+0000 7fd9197fa640 1 -- 172.21.15.33:0/430109199 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fd90c010a90 con 0x7fd91c100100 2024-03-19T03:00:18.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.164+0000 7fd9197fa640 1 --2- 172.21.15.33:0/430109199 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd9040a29a0 0x7fd9040a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.165+0000 7fd9197fa640 1 -- 172.21.15.33:0/430109199 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7fd90c0cf4e0 con 0x7fd91c100100 2024-03-19T03:00:18.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.165+0000 7fd91b7fe640 1 --2- 172.21.15.33:0/430109199 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd9040a29a0 0x7fd9040a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.165+0000 7fd91b7fe640 1 --2- 172.21.15.33:0/430109199 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd9040a29a0 0x7fd9040a4e60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd91c10e130 tx=0x7fd910005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:18.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.169+0000 7fd9197fa640 1 -- 172.21.15.33:0/430109199 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd90c09b820 con 0x7fd91c100100 2024-03-19T03:00:18.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.391+0000 7fd922310640 1 -- 172.21.15.33:0/430109199 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd91c10dbd0 con 0x7fd91c100100 2024-03-19T03:00:18.393 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.392+0000 7fd9197fa640 1 -- 172.21.15.33:0/430109199 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11867 (secure 0 0 0) 0x7fd90c09f8c0 con 0x7fd91c100100 2024-03-19T03:00:18.393 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T03:00:18.393 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:50:52.201798+0000","modified":"2024-03-19T02:53:11.748697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24488,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/3731232169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":3731232169},{"type":"v1","addr":"172.21.15.96:6839","nonce":3731232169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24457,24488,24509]},"id":1}]} 2024-03-19T03:00:18.393 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-03-19T03:00:18.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.394+0000 7fd922310640 1 -- 172.21.15.33:0/430109199 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd9040a29a0 msgr2=0x7fd9040a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.394+0000 7fd922310640 1 --2- 172.21.15.33:0/430109199 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd9040a29a0 0x7fd9040a4e60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd91c10e130 tx=0x7fd910005f90 comp rx=0 tx=0).stop 2024-03-19T03:00:18.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.394+0000 7fd922310640 1 -- 172.21.15.33:0/430109199 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd91c100100 msgr2=0x7fd91c198e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.394+0000 7fd922310640 1 --2- 172.21.15.33:0/430109199 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd91c100100 0x7fd91c198e20 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fd90c00e9f0 tx=0x7fd90c00eeb0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.394+0000 7fd922310640 1 -- 172.21.15.33:0/430109199 shutdown_connections 2024-03-19T03:00:18.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.394+0000 7fd922310640 1 --2- 172.21.15.33:0/430109199 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd9040a29a0 0x7fd9040a4e60 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.394+0000 7fd922310640 1 --2- 172.21.15.33:0/430109199 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd91c1998a0 0x7fd91c1a3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.394+0000 7fd922310640 1 --2- 172.21.15.33:0/430109199 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd91c105180 0x7fd91c199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.394+0000 7fd922310640 1 --2- 172.21.15.33:0/430109199 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd91c100100 0x7fd91c198e20 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.394+0000 7fd922310640 1 -- 172.21.15.33:0/430109199 >> 172.21.15.33:0/430109199 conn(0x7fd91c076cb0 msgr2=0x7fd91c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:18.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.394+0000 7fd922310640 1 -- 172.21.15.33:0/430109199 shutdown_connections 2024-03-19T03:00:18.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.394+0000 7fd922310640 1 -- 172.21.15.33:0/430109199 wait complete. 2024-03-19T03:00:18.406 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T03:00:18.514 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.509+0000 7fcb0bc85640 1 -- 172.21.15.33:0/219841806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb04102150 msgr2=0x7fcb0410e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.509+0000 7fcb0bc85640 1 --2- 172.21.15.33:0/219841806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb04102150 0x7fcb0410e660 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fcafc009cd0 tx=0x7fcafc02c0f0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.509+0000 7fcb0bc85640 1 -- 172.21.15.33:0/219841806 shutdown_connections 2024-03-19T03:00:18.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.509+0000 7fcb0bc85640 1 --2- 172.21.15.33:0/219841806 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb04102150 0x7fcb0410e660 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.509+0000 7fcb0bc85640 1 --2- 172.21.15.33:0/219841806 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb04101790 0x7fcb04101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.509+0000 7fcb0bc85640 1 --2- 172.21.15.33:0/219841806 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb041077a0 0x7fcb04107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.509+0000 7fcb0bc85640 1 -- 172.21.15.33:0/219841806 >> 172.21.15.33:0/219841806 conn(0x7fcb040fd2c0 msgr2=0x7fcb040ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:18.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.509+0000 7fcb0bc85640 1 -- 172.21.15.33:0/219841806 shutdown_connections 2024-03-19T03:00:18.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.509+0000 7fcb0bc85640 1 -- 172.21.15.33:0/219841806 wait complete. 2024-03-19T03:00:18.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.510+0000 7fcb0bc85640 1 Processor -- start 2024-03-19T03:00:18.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.510+0000 7fcb0bc85640 1 -- start start 2024-03-19T03:00:18.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.510+0000 7fcb0bc85640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb04101790 0x7fcb04197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.510+0000 7fcb0bc85640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb04102150 0x7fcb04198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.510+0000 7fcb0bc85640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb041077a0 0x7fcb0419c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.510+0000 7fcb0bc85640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcb04111a60 con 0x7fcb04101790 2024-03-19T03:00:18.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.510+0000 7fcb0bc85640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcb041118e0 con 0x7fcb04102150 2024-03-19T03:00:18.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.510+0000 7fcb0bc85640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fcb04111be0 con 0x7fcb041077a0 2024-03-19T03:00:18.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.510+0000 7fcb09a21640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb04101790 0x7fcb04197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.510+0000 7fcb09a21640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb04101790 0x7fcb04197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42326/0 (socket says 172.21.15.33:42326) 2024-03-19T03:00:18.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.510+0000 7fcb09a21640 1 -- 172.21.15.33:0/1555525405 learned_addr learned my addr 172.21.15.33:0/1555525405 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:18.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.510+0000 7fcb0a222640 1 --2- 172.21.15.33:0/1555525405 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb041077a0 0x7fcb0419c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.510+0000 7fcb09220640 1 --2- 172.21.15.33:0/1555525405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb04102150 0x7fcb04198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.511+0000 7fcb09a21640 1 -- 172.21.15.33:0/1555525405 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb041077a0 msgr2=0x7fcb0419c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.511+0000 7fcb09a21640 1 --2- 172.21.15.33:0/1555525405 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb041077a0 0x7fcb0419c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.511+0000 7fcb09a21640 1 -- 172.21.15.33:0/1555525405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb04102150 msgr2=0x7fcb04198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.511+0000 7fcb09a21640 1 --2- 172.21.15.33:0/1555525405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb04102150 0x7fcb04198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.511+0000 7fcb09a21640 1 -- 172.21.15.33:0/1555525405 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb0419cf60 con 0x7fcb04101790 2024-03-19T03:00:18.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.511+0000 7fcb09a21640 1 --2- 172.21.15.33:0/1555525405 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb04101790 0x7fcb04197fc0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fcb0000e9f0 tx=0x7fcb0000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:18.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.511+0000 7fcafaffd640 1 -- 172.21.15.33:0/1555525405 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0000cd80 con 0x7fcb04101790 2024-03-19T03:00:18.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.511+0000 7fcafaffd640 1 -- 172.21.15.33:0/1555525405 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcb000054d0 con 0x7fcb04101790 2024-03-19T03:00:18.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.511+0000 7fcb0bc85640 1 -- 172.21.15.33:0/1555525405 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb0419d250 con 0x7fcb04101790 2024-03-19T03:00:18.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.511+0000 7fcafaffd640 1 -- 172.21.15.33:0/1555525405 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb00010490 con 0x7fcb04101790 2024-03-19T03:00:18.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.511+0000 7fcb0bc85640 1 -- 172.21.15.33:0/1555525405 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb041a4ae0 con 0x7fcb04101790 2024-03-19T03:00:18.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.513+0000 7fcafaffd640 1 -- 172.21.15.33:0/1555525405 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fcb00005020 con 0x7fcb04101790 2024-03-19T03:00:18.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.514+0000 7fcb0e87b740 1 Processor -- start 2024-03-19T03:00:18.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.514+0000 7fcb0e87b740 1 -- start start 2024-03-19T03:00:18.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.514+0000 7fcb0e87b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a1dd78b2d0 0x55a1dd78b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.514+0000 7fcb0e87b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a1dd78bbf0 0x55a1dd794290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.514+0000 7fcb0e87b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1dd7947d0 0x55a1dd796bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.514+0000 7fcb0e87b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a1dd6ee4b0 con 0x55a1dd7947d0 2024-03-19T03:00:18.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.514+0000 7fcb0e87b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55a1dd696d70 con 0x55a1dd78bbf0 2024-03-19T03:00:18.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.514+0000 7fcb0e87b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a1dd6ea550 con 0x55a1dd78b2d0 2024-03-19T03:00:18.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.514+0000 7fcafaffd640 1 --2- 172.21.15.33:0/1555525405 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fcaf00a2970 0x7fcaf00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.514+0000 7fcb09220640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1dd7947d0 0x55a1dd796bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.514+0000 7fcb09220640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1dd7947d0 0x55a1dd796bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42342/0 (socket says 172.21.15.33:42342) 2024-03-19T03:00:18.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.514+0000 7fcb0a222640 1 --2- 172.21.15.33:0/1555525405 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fcaf00a2970 0x7fcaf00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.514+0000 7fcb09220640 1 -- 172.21.15.33:0/4270687641 learned_addr learned my addr 172.21.15.33:0/4270687641 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:18.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.514+0000 7fcb09a21640 1 --2- 172.21.15.33:0/4270687641 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a1dd78bbf0 0x55a1dd794290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.514+0000 7fcb0a222640 1 --2- 172.21.15.33:0/4270687641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a1dd78b2d0 0x55a1dd78b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.514+0000 7fcafaffd640 1 -- 172.21.15.33:0/1555525405 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7fcb000cf570 con 0x7fcb04101790 2024-03-19T03:00:18.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.514+0000 7fcb09220640 1 -- 172.21.15.33:0/4270687641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a1dd78b2d0 msgr2=0x55a1dd78b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:18.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.515+0000 7fcb09220640 1 --2- 172.21.15.33:0/4270687641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a1dd78b2d0 0x55a1dd78b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.515+0000 7fcb09220640 1 -- 172.21.15.33:0/4270687641 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a1dd78bbf0 msgr2=0x55a1dd794290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:18.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.515+0000 7fcb09220640 1 --2- 172.21.15.33:0/4270687641 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a1dd78bbf0 0x55a1dd794290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.515+0000 7fcb09220640 1 -- 172.21.15.33:0/4270687641 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1dd6b9210 con 0x55a1dd7947d0 2024-03-19T03:00:18.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.515+0000 7fcb0a222640 1 --2- 172.21.15.33:0/4270687641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a1dd78b2d0 0x55a1dd78b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:18.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.515+0000 7fcb09a21640 1 --2- 172.21.15.33:0/4270687641 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a1dd78bbf0 0x55a1dd794290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:18.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.515+0000 7fcb09220640 1 --2- 172.21.15.33:0/4270687641 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1dd7947d0 0x55a1dd796bc0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4009c90 tx=0x7fcaf4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4444b8e4e67bac2c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:18.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.515+0000 7fcb0a222640 1 --2- 172.21.15.33:0/1555525405 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fcaf00a2970 0x7fcaf00a4e30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcafc005cc0 tx=0x7fcafc02e040 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:18.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.515+0000 7fcaeffff640 1 -- 172.21.15.33:0/4270687641 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf401e070 con 0x55a1dd7947d0 2024-03-19T03:00:18.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.515+0000 7fcaeffff640 1 -- 172.21.15.33:0/4270687641 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcaf4019ae0 con 0x55a1dd7947d0 2024-03-19T03:00:18.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.515+0000 7fcaeffff640 1 -- 172.21.15.33:0/4270687641 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf400e950 con 0x55a1dd7947d0 2024-03-19T03:00:18.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.516+0000 7fcb0e87b740 1 -- 172.21.15.33:0/4270687641 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1dd7947d0 msgr2=0x55a1dd796bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.516+0000 7fcb0e87b740 1 --2- 172.21.15.33:0/4270687641 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1dd7947d0 0x55a1dd796bc0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4009c90 tx=0x7fcaf4011810 comp rx=0 tx=0).stop 2024-03-19T03:00:18.525 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.516+0000 7fcb0e87b740 1 -- 172.21.15.33:0/4270687641 shutdown_connections 2024-03-19T03:00:18.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.516+0000 7fcb0e87b740 1 --2- 172.21.15.33:0/4270687641 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1dd7947d0 0x55a1dd796bc0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.516+0000 7fcb0e87b740 1 --2- 172.21.15.33:0/4270687641 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a1dd78bbf0 0x55a1dd794290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.516+0000 7fcb0e87b740 1 --2- 172.21.15.33:0/4270687641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a1dd78b2d0 0x55a1dd78b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.516+0000 7fcb0e87b740 1 -- 172.21.15.33:0/4270687641 >> 172.21.15.33:0/4270687641 conn(0x55a1dd785a00 msgr2=0x55a1dd785de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:18.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.516+0000 7fcb0e87b740 1 -- 172.21.15.33:0/4270687641 shutdown_connections 2024-03-19T03:00:18.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.516+0000 7fcb0e87b740 1 -- 172.21.15.33:0/4270687641 wait complete. 2024-03-19T03:00:18.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.516+0000 7fcb0e87b740 1 -- start start 2024-03-19T03:00:18.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.516+0000 7fcb0e87b740 1 Processor -- start 2024-03-19T03:00:18.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.517+0000 7fcb0e87b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a1dd78b2d0 0x55a1dd791810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.517+0000 7fcb0e87b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1dd78bbf0 0x55a1dd791d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.517+0000 7fcb0e87b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a1dd7947d0 0x55a1dd792290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.517+0000 7fcb0e87b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55a1dd7975c0 con 0x55a1dd78bbf0 2024-03-19T03:00:18.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.517+0000 7fcb0e87b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55a1dd6aa310 con 0x55a1dd7947d0 2024-03-19T03:00:18.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.517+0000 7fcb0e87b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a1dd797740 con 0x55a1dd78b2d0 2024-03-19T03:00:18.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.517+0000 7fcb0e87b740 10 client.0 _ll_register_callbacks cb 0x55a1dd69e2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:00:18.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.517+0000 7fcb09220640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1dd78bbf0 0x55a1dd791d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.517+0000 7fcb09220640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1dd78bbf0 0x55a1dd791d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42346/0 (socket says 172.21.15.33:42346) 2024-03-19T03:00:18.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.517+0000 7fcb09220640 1 -- 172.21.15.33:0/4187013523 learned_addr learned my addr 172.21.15.33:0/4187013523 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:18.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.517+0000 7fcb09a21640 1 --2- 172.21.15.33:0/4187013523 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a1dd78b2d0 0x55a1dd791810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.517+0000 7fcb0a222640 1 --2- 172.21.15.33:0/4187013523 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a1dd7947d0 0x55a1dd792290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.517+0000 7fcb09220640 1 -- 172.21.15.33:0/4187013523 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a1dd78b2d0 msgr2=0x55a1dd791810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:18.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.517+0000 7fcb09220640 1 --2- 172.21.15.33:0/4187013523 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a1dd78b2d0 0x55a1dd791810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.517+0000 7fcb09220640 1 -- 172.21.15.33:0/4187013523 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a1dd7947d0 msgr2=0x55a1dd792290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:18.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.517+0000 7fcb09220640 1 --2- 172.21.15.33:0/4187013523 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a1dd7947d0 0x55a1dd792290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.517+0000 7fcb09220640 1 -- 172.21.15.33:0/4187013523 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1dd4023f0 con 0x55a1dd78bbf0 2024-03-19T03:00:18.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.517+0000 7fcb09220640 1 --2- 172.21.15.33:0/4187013523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1dd78bbf0 0x55a1dd791d50 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fcaf401c040 tx=0x7fcaf4005db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=757e6d6ba88b6456 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:18.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.517+0000 7fcaed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:00:18.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.517+0000 7fcaed7fa640 1 -- 172.21.15.33:0/4187013523 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf401e050 con 0x55a1dd78bbf0 2024-03-19T03:00:18.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.517+0000 7fcaed7fa640 1 -- 172.21.15.33:0/4187013523 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7fcaf4005450 con 0x55a1dd78bbf0 2024-03-19T03:00:18.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.517+0000 7fcb0e87b740 1 -- 172.21.15.33:0/4187013523 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcaf4019ae0 con 0x55a1dd78bbf0 2024-03-19T03:00:18.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.517+0000 7fcaed7fa640 1 -- 172.21.15.33:0/4187013523 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fcaf4032050 con 0x55a1dd78bbf0 2024-03-19T03:00:18.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.517+0000 7fcb0e87b740 10 client.17100 fetch_fsmap learned FSMap version 25 2024-03-19T03:00:18.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.518+0000 7fcb0e87b740 1 -- 172.21.15.33:0/4187013523 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a1dd4cc7d0 con 0x55a1dd78bbf0 2024-03-19T03:00:18.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.518+0000 7fcaed7fa640 1 -- 172.21.15.33:0/4187013523 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcaf4019ae0 con 0x55a1dd78bbf0 2024-03-19T03:00:18.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.518+0000 7fcaed7fa640 1 -- 172.21.15.33:0/4187013523 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fcaf402d030 con 0x55a1dd78bbf0 2024-03-19T03:00:18.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.518+0000 7fcb0e87b740 10 client.17100 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:00:18.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.518+0000 7fcb0e87b740 10 client.17100 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T03:00:18.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.518+0000 7fcb0e87b740 20 client.17100 populate_metadata read hostname 'smithi033' 2024-03-19T03:00:18.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.518+0000 7fcb0e87b740 1 --2- 172.21.15.33:0/4187013523 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55a1dd7a4af0 0x55a1dd7a6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.518+0000 7fcb0e87b740 4 client.17100 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:00:18.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.518+0000 7fcb0e87b740 1 -- 172.21.15.33:0/4187013523 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a1dd74c3e0 con 0x55a1dd7a4af0 2024-03-19T03:00:18.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.518+0000 7fcb09a21640 1 --2- 172.21.15.33:0/4187013523 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55a1dd7a4af0 0x55a1dd7a6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.518+0000 7fcb09a21640 1 --2- 172.21.15.33:0/4187013523 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55a1dd7a4af0 0x55a1dd7a6ee0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=efe39f22d23eb152 server_cookie=ddba825d08a2b9cf in_seq=0 out_seq=0 2024-03-19T03:00:18.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.518+0000 7fcaed7fa640 10 client.17100 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T03:00:18.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.519+0000 7fcaed7fa640 1 -- 172.21.15.33:0/4187013523 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a1dd74c3e0 con 0x55a1dd7a4af0 2024-03-19T03:00:18.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.519+0000 7fcaed7fa640 10 client.17100 handle_command_reply: tid=0 2024-03-19T03:00:18.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.519+0000 7fcb0e87b740 1 client.17100 shutdown 2024-03-19T03:00:18.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.519+0000 7fcb0e87b740 1 -- 172.21.15.33:0/4187013523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1dd78bbf0 msgr2=0x55a1dd791d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.519+0000 7fcb0e87b740 1 --2- 172.21.15.33:0/4187013523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1dd78bbf0 0x55a1dd791d50 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fcaf401c040 tx=0x7fcaf4005db0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.519+0000 7fcb0e87b740 1 -- 172.21.15.33:0/4187013523 shutdown_connections 2024-03-19T03:00:18.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.519+0000 7fcb0e87b740 1 --2- 172.21.15.33:0/4187013523 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55a1dd7a4af0 0x55a1dd7a6ee0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.519+0000 7fcb0e87b740 1 --2- 172.21.15.33:0/4187013523 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a1dd7947d0 0x55a1dd792290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.519+0000 7fcb0e87b740 1 --2- 172.21.15.33:0/4187013523 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1dd78bbf0 0x55a1dd791d50 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.519+0000 7fcb0e87b740 1 --2- 172.21.15.33:0/4187013523 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a1dd78b2d0 0x55a1dd791810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.519+0000 7fcaed7fa640 0 client.17100 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T03:00:18.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.519+0000 7fcb0e87b740 1 -- 172.21.15.33:0/4187013523 >> 172.21.15.33:0/4187013523 conn(0x55a1dd785a00 msgr2=0x55a1dd787750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:18.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.519+0000 7fcb0e87b740 1 -- 172.21.15.33:0/4187013523 shutdown_connections 2024-03-19T03:00:18.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.519+0000 7fcb0e87b740 1 -- 172.21.15.33:0/4187013523 wait complete. 2024-03-19T03:00:18.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.520+0000 7fcb0e87b740 20 client.17100 trim_cache size 0 max 16384 2024-03-19T03:00:18.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.533+0000 7fcb0e87b740 1 Processor -- start 2024-03-19T03:00:18.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.533+0000 7fcb0e87b740 1 -- start start 2024-03-19T03:00:18.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.533+0000 7fcb0e87b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1dd78b2d0 0x55a1dd7a0360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.533+0000 7fcb0e87b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a1dd78bbf0 0x55a1dd7ac670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.533+0000 7fcb0e87b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a1dd7947d0 0x55a1dd784310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.533+0000 7fcb0e87b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcb041b15c0 con 0x55a1dd78b2d0 2024-03-19T03:00:18.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.533+0000 7fcb0e87b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcb041aee90 con 0x55a1dd78bbf0 2024-03-19T03:00:18.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.533+0000 7fcb0e87b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fcb041acd00 con 0x55a1dd7947d0 2024-03-19T03:00:18.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.533+0000 7fcb09a21640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1dd78b2d0 0x55a1dd7a0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.533+0000 7fcb09a21640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1dd78b2d0 0x55a1dd7a0360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42354/0 (socket says 172.21.15.33:42354) 2024-03-19T03:00:18.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.533+0000 7fcb09a21640 1 -- 172.21.15.33:0/2274088296 learned_addr learned my addr 172.21.15.33:0/2274088296 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:18.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.533+0000 7fcb09220640 1 --2- 172.21.15.33:0/2274088296 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a1dd78bbf0 0x55a1dd7ac670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.533+0000 7fcb0a222640 1 --2- 172.21.15.33:0/2274088296 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a1dd7947d0 0x55a1dd784310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.533+0000 7fcb09a21640 1 -- 172.21.15.33:0/2274088296 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a1dd7947d0 msgr2=0x55a1dd784310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:18.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.533+0000 7fcb09a21640 1 --2- 172.21.15.33:0/2274088296 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a1dd7947d0 0x55a1dd784310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.533+0000 7fcb09a21640 1 -- 172.21.15.33:0/2274088296 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a1dd78bbf0 msgr2=0x55a1dd7ac670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:18.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.533+0000 7fcb09a21640 1 --2- 172.21.15.33:0/2274088296 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a1dd78bbf0 0x55a1dd7ac670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.533+0000 7fcb09a21640 1 -- 172.21.15.33:0/2274088296 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1dd7974a0 con 0x55a1dd78b2d0 2024-03-19T03:00:18.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.533+0000 7fcb09220640 1 --2- 172.21.15.33:0/2274088296 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a1dd78bbf0 0x55a1dd7ac670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:00:18.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.534+0000 7fcb09a21640 1 --2- 172.21.15.33:0/2274088296 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1dd78b2d0 0x55a1dd7a0360 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fcb000cc8f0 tx=0x7fcb00007850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cd67826157acddd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:18.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.534+0000 7fcaef7fe640 1 -- 172.21.15.33:0/2274088296 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb00005680 con 0x55a1dd78b2d0 2024-03-19T03:00:18.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.534+0000 7fcaef7fe640 1 -- 172.21.15.33:0/2274088296 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcb0009bea0 con 0x55a1dd78b2d0 2024-03-19T03:00:18.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.534+0000 7fcaef7fe640 1 -- 172.21.15.33:0/2274088296 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb000046c0 con 0x55a1dd78b2d0 2024-03-19T03:00:18.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.534+0000 7fcb0e87b740 1 -- 172.21.15.33:0/2274088296 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1dd78b2d0 msgr2=0x55a1dd7a0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.534+0000 7fcb0e87b740 1 --2- 172.21.15.33:0/2274088296 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1dd78b2d0 0x55a1dd7a0360 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fcb000cc8f0 tx=0x7fcb00007850 comp rx=0 tx=0).stop 2024-03-19T03:00:18.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.534+0000 7fcb0e87b740 1 -- 172.21.15.33:0/2274088296 shutdown_connections 2024-03-19T03:00:18.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.534+0000 7fcb0e87b740 1 --2- 172.21.15.33:0/2274088296 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a1dd7947d0 0x55a1dd784310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.534+0000 7fcb0e87b740 1 --2- 172.21.15.33:0/2274088296 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a1dd78bbf0 0x55a1dd7ac670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.534+0000 7fcb0e87b740 1 --2- 172.21.15.33:0/2274088296 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1dd78b2d0 0x55a1dd7a0360 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.534+0000 7fcb0e87b740 1 -- 172.21.15.33:0/2274088296 >> 172.21.15.33:0/2274088296 conn(0x55a1dd785a00 msgr2=0x55a1dd790d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:18.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.534+0000 7fcb0e87b740 1 -- 172.21.15.33:0/2274088296 shutdown_connections 2024-03-19T03:00:18.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.534+0000 7fcb0e87b740 1 -- 172.21.15.33:0/2274088296 wait complete. 2024-03-19T03:00:18.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.535+0000 7fcb0e87b740 1 -- start start 2024-03-19T03:00:18.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.535+0000 7fcb0e87b740 1 Processor -- start 2024-03-19T03:00:18.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.535+0000 7fcb0e87b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1dd78b2d0 0x55a1dd7981e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.535+0000 7fcb0e87b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a1dd78bbf0 0x55a1dd798720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.535+0000 7fcb0e87b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a1dd7947d0 0x55a1dd798c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.535+0000 7fcb0e87b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fcb041ad350 con 0x55a1dd78b2d0 2024-03-19T03:00:18.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.535+0000 7fcb0e87b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcb041ae7c0 con 0x55a1dd7947d0 2024-03-19T03:00:18.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.535+0000 7fcb0e87b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fcb041ada20 con 0x55a1dd78bbf0 2024-03-19T03:00:18.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.535+0000 7fcb0e87b740 10 client.0 _ll_register_callbacks cb 0x7fcaf401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:00:18.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.535+0000 7fcb09a21640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1dd78b2d0 0x55a1dd7981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.535+0000 7fcb09a21640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1dd78b2d0 0x55a1dd7981e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42360/0 (socket says 172.21.15.33:42360) 2024-03-19T03:00:18.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.535+0000 7fcb09a21640 1 -- 172.21.15.33:0/214230379 learned_addr learned my addr 172.21.15.33:0/214230379 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:18.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.535+0000 7fcb09220640 1 --2- 172.21.15.33:0/214230379 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a1dd78bbf0 0x55a1dd798720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.535+0000 7fcb0a222640 1 --2- 172.21.15.33:0/214230379 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a1dd7947d0 0x55a1dd798c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.535+0000 7fcb09a21640 1 -- 172.21.15.33:0/214230379 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a1dd78bbf0 msgr2=0x55a1dd798720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:18.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.535+0000 7fcb09a21640 1 --2- 172.21.15.33:0/214230379 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a1dd78bbf0 0x55a1dd798720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.535+0000 7fcb09a21640 1 -- 172.21.15.33:0/214230379 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a1dd7947d0 msgr2=0x55a1dd798c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:18.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.535+0000 7fcb09a21640 1 --2- 172.21.15.33:0/214230379 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a1dd7947d0 0x55a1dd798c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.535+0000 7fcb09a21640 1 -- 172.21.15.33:0/214230379 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1dd809ca0 con 0x55a1dd78b2d0 2024-03-19T03:00:18.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.536+0000 7fcb09a21640 1 --2- 172.21.15.33:0/214230379 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1dd78b2d0 0x55a1dd7981e0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fcb00098170 tx=0x7fcb0009b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85198a140b0c79da server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:18.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.536+0000 7fcaed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:00:18.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.536+0000 7fcaed7fa640 1 -- 172.21.15.33:0/214230379 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb000a3f40 con 0x55a1dd78b2d0 2024-03-19T03:00:18.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.536+0000 7fcaed7fa640 1 -- 172.21.15.33:0/214230379 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7fcb00086050 con 0x55a1dd78b2d0 2024-03-19T03:00:18.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.536+0000 7fcb0e87b740 1 -- 172.21.15.33:0/214230379 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb0009bea0 con 0x55a1dd78b2d0 2024-03-19T03:00:18.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.536+0000 7fcaed7fa640 1 -- 172.21.15.33:0/214230379 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fcb00097e10 con 0x55a1dd78b2d0 2024-03-19T03:00:18.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.536+0000 7fcb0e87b740 10 client.17106 fetch_fsmap learned FSMap version 25 2024-03-19T03:00:18.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.536+0000 7fcb0e87b740 1 -- 172.21.15.33:0/214230379 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a1dd80aee0 con 0x55a1dd78b2d0 2024-03-19T03:00:18.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.536+0000 7fcaed7fa640 1 -- 172.21.15.33:0/214230379 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcb0009bea0 con 0x55a1dd78b2d0 2024-03-19T03:00:18.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.536+0000 7fcaed7fa640 1 -- 172.21.15.33:0/214230379 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fcb00084ce0 con 0x55a1dd78b2d0 2024-03-19T03:00:18.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.536+0000 7fcb0e87b740 10 client.17106 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:00:18.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.536+0000 7fcb0e87b740 10 client.17106 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T03:00:18.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.536+0000 7fcb0e87b740 20 client.17106 populate_metadata read hostname 'smithi033' 2024-03-19T03:00:18.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.536+0000 7fcb0e87b740 1 --2- 172.21.15.33:0/214230379 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55a1dd80b550 0x55a1dd80d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.536+0000 7fcb0e87b740 4 client.17106 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T03:00:18.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.536+0000 7fcb0e87b740 1 -- 172.21.15.33:0/214230379 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a1dd6fb9f0 con 0x55a1dd80b550 2024-03-19T03:00:18.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.537+0000 7fcb09220640 1 --2- 172.21.15.33:0/214230379 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55a1dd80b550 0x55a1dd80d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.537+0000 7fcb09220640 1 --2- 172.21.15.33:0/214230379 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55a1dd80b550 0x55a1dd80d970 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e6431c98e70c4d8e server_cookie=decd78498bf6186c in_seq=0 out_seq=0 2024-03-19T03:00:18.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.537+0000 7fcaed7fa640 10 client.17106 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T03:00:18.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.537+0000 7fcaed7fa640 1 -- 172.21.15.33:0/214230379 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a1dd6fb9f0 con 0x55a1dd80b550 2024-03-19T03:00:18.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.537+0000 7fcaed7fa640 10 client.17106 handle_command_reply: tid=0 2024-03-19T03:00:18.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.537+0000 7fcb0e87b740 1 client.17106 shutdown 2024-03-19T03:00:18.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.537+0000 7fcb0e87b740 1 -- 172.21.15.33:0/214230379 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1dd78b2d0 msgr2=0x55a1dd7981e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.537+0000 7fcb0e87b740 1 --2- 172.21.15.33:0/214230379 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1dd78b2d0 0x55a1dd7981e0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fcb00098170 tx=0x7fcb0009b7a0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.537+0000 7fcb0e87b740 1 -- 172.21.15.33:0/214230379 shutdown_connections 2024-03-19T03:00:18.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.537+0000 7fcb0e87b740 1 --2- 172.21.15.33:0/214230379 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55a1dd80b550 0x55a1dd80d970 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.537+0000 7fcb0e87b740 1 --2- 172.21.15.33:0/214230379 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a1dd78bbf0 0x55a1dd798720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.537+0000 7fcb0e87b740 1 --2- 172.21.15.33:0/214230379 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55a1dd7947d0 0x55a1dd798c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.537+0000 7fcb0e87b740 1 --2- 172.21.15.33:0/214230379 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55a1dd78b2d0 0x55a1dd7981e0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.537+0000 7fcaed7fa640 0 client.17106 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T03:00:18.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.537+0000 7fcb0e87b740 1 -- 172.21.15.33:0/214230379 >> 172.21.15.33:0/214230379 conn(0x55a1dd785a00 msgr2=0x55a1dd792ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:18.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.537+0000 7fcb0e87b740 1 -- 172.21.15.33:0/214230379 shutdown_connections 2024-03-19T03:00:18.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.537+0000 7fcb0e87b740 1 -- 172.21.15.33:0/214230379 wait complete. 2024-03-19T03:00:18.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.537+0000 7fcb0e87b740 20 client.17106 trim_cache size 0 max 16384 2024-03-19T03:00:18.547 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T03:00:18.547 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.538+0000 7fcb0bc85640 1 -- 172.21.15.33:0/1555525405 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fcaf00a2970 msgr2=0x7fcaf00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.538+0000 7fcb0bc85640 1 --2- 172.21.15.33:0/1555525405 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fcaf00a2970 0x7fcaf00a4e30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcafc005cc0 tx=0x7fcafc02e040 comp rx=0 tx=0).stop 2024-03-19T03:00:18.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.538+0000 7fcb0bc85640 1 -- 172.21.15.33:0/1555525405 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb04101790 msgr2=0x7fcb04197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.538+0000 7fcb0bc85640 1 --2- 172.21.15.33:0/1555525405 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb04101790 0x7fcb04197fc0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fcb0000e9f0 tx=0x7fcb0000edf0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.538+0000 7fcb0bc85640 1 -- 172.21.15.33:0/1555525405 shutdown_connections 2024-03-19T03:00:18.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.538+0000 7fcb0bc85640 1 --2- 172.21.15.33:0/1555525405 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fcaf00a2970 0x7fcaf00a4e30 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.538+0000 7fcb0bc85640 1 --2- 172.21.15.33:0/1555525405 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb041077a0 0x7fcb0419c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.538+0000 7fcb0bc85640 1 --2- 172.21.15.33:0/1555525405 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb04102150 0x7fcb04198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.538+0000 7fcb0bc85640 1 --2- 172.21.15.33:0/1555525405 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fcb04101790 0x7fcb04197fc0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.538+0000 7fcb0bc85640 1 -- 172.21.15.33:0/1555525405 >> 172.21.15.33:0/1555525405 conn(0x7fcb040fd2c0 msgr2=0x7fcb040fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:18.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.538+0000 7fcb0bc85640 1 -- 172.21.15.33:0/1555525405 shutdown_connections 2024-03-19T03:00:18.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.538+0000 7fcb0bc85640 1 -- 172.21.15.33:0/1555525405 wait complete. 2024-03-19T03:00:18.549 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T03:00:18.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.653+0000 7fde34e68640 1 -- 172.21.15.33:0/3753019209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde30100a00 msgr2=0x7fde30100e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.653+0000 7fde34e68640 1 --2- 172.21.15.33:0/3753019209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde30100a00 0x7fde30100e80 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fde24009900 tx=0x7fde2402bc30 comp rx=0 tx=0).stop 2024-03-19T03:00:18.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.653+0000 7fde34e68640 1 -- 172.21.15.33:0/3753019209 shutdown_connections 2024-03-19T03:00:18.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.653+0000 7fde34e68640 1 --2- 172.21.15.33:0/3753019209 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde301013c0 0x7fde3010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.653+0000 7fde34e68640 1 --2- 172.21.15.33:0/3753019209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde30100a00 0x7fde30100e80 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.653+0000 7fde34e68640 1 --2- 172.21.15.33:0/3753019209 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde30105680 0x7fde30105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.653+0000 7fde34e68640 1 -- 172.21.15.33:0/3753019209 >> 172.21.15.33:0/3753019209 conn(0x7fde30077700 msgr2=0x7fde30077b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:18.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.653+0000 7fde34e68640 1 -- 172.21.15.33:0/3753019209 shutdown_connections 2024-03-19T03:00:18.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.653+0000 7fde34e68640 1 -- 172.21.15.33:0/3753019209 wait complete. 2024-03-19T03:00:18.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.654+0000 7fde34e68640 1 Processor -- start 2024-03-19T03:00:18.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.654+0000 7fde34e68640 1 -- start start 2024-03-19T03:00:18.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.654+0000 7fde34e68640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde30100a00 0x7fde30198d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.654+0000 7fde34e68640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde301013c0 0x7fde30199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.654+0000 7fde34e68640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde30105680 0x7fde301a3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.654+0000 7fde34e68640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fde30110970 con 0x7fde301013c0 2024-03-19T03:00:18.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.654+0000 7fde34e68640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fde301107f0 con 0x7fde30100a00 2024-03-19T03:00:18.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.654+0000 7fde34e68640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fde30110af0 con 0x7fde30105680 2024-03-19T03:00:18.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.654+0000 7fde2dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde301013c0 0x7fde30199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.654+0000 7fde2dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde301013c0 0x7fde30199290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42366/0 (socket says 172.21.15.33:42366) 2024-03-19T03:00:18.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.654+0000 7fde2dd9b640 1 -- 172.21.15.33:0/3432049225 learned_addr learned my addr 172.21.15.33:0/3432049225 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:18.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.654+0000 7fde2e59c640 1 --2- 172.21.15.33:0/3432049225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde30100a00 0x7fde30198d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.654+0000 7fde2ed9d640 1 --2- 172.21.15.33:0/3432049225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde30105680 0x7fde301a3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.654+0000 7fde2dd9b640 1 -- 172.21.15.33:0/3432049225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde30105680 msgr2=0x7fde301a3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.654+0000 7fde2dd9b640 1 --2- 172.21.15.33:0/3432049225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde30105680 0x7fde301a3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.654+0000 7fde2dd9b640 1 -- 172.21.15.33:0/3432049225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde30100a00 msgr2=0x7fde30198d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.654+0000 7fde2dd9b640 1 --2- 172.21.15.33:0/3432049225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde30100a00 0x7fde30198d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.654+0000 7fde2dd9b640 1 -- 172.21.15.33:0/3432049225 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde301a4310 con 0x7fde301013c0 2024-03-19T03:00:18.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.654+0000 7fde2dd9b640 1 --2- 172.21.15.33:0/3432049225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde301013c0 0x7fde30199290 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fde24006010 tx=0x7fde24005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:18.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.655+0000 7fde1f7fe640 1 -- 172.21.15.33:0/3432049225 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde24043020 con 0x7fde301013c0 2024-03-19T03:00:18.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.655+0000 7fde1f7fe640 1 -- 172.21.15.33:0/3432049225 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fde24004350 con 0x7fde301013c0 2024-03-19T03:00:18.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.655+0000 7fde34e68640 1 -- 172.21.15.33:0/3432049225 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde301a44e0 con 0x7fde301013c0 2024-03-19T03:00:18.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.655+0000 7fde1f7fe640 1 -- 172.21.15.33:0/3432049225 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde24003a10 con 0x7fde301013c0 2024-03-19T03:00:18.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.655+0000 7fde34e68640 1 -- 172.21.15.33:0/3432049225 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde301a4940 con 0x7fde301013c0 2024-03-19T03:00:18.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.657+0000 7fde1f7fe640 1 -- 172.21.15.33:0/3432049225 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fde24046020 con 0x7fde301013c0 2024-03-19T03:00:18.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.657+0000 7fde37a4f740 1 Processor -- start 2024-03-19T03:00:18.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.657+0000 7fde37a4f740 1 -- start start 2024-03-19T03:00:18.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.658+0000 7fde37a4f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2bf63c6e0 0x55c2bf63cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.658+0000 7fde37a4f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c2bf63d000 0x55c2bf6456a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.658+0000 7fde37a4f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c2bf645be0 0x55c2bf647fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.658+0000 7fde37a4f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c2bf59b4b0 con 0x55c2bf63c6e0 2024-03-19T03:00:18.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.658+0000 7fde37a4f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c2bf543d70 con 0x55c2bf645be0 2024-03-19T03:00:18.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.658+0000 7fde37a4f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c2bf597550 con 0x55c2bf63d000 2024-03-19T03:00:18.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.658+0000 7fde2ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2bf63c6e0 0x55c2bf63cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.658+0000 7fde1f7fe640 1 --2- 172.21.15.33:0/3432049225 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fde180a2970 0x7fde180a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.658+0000 7fde2ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2bf63c6e0 0x55c2bf63cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42380/0 (socket says 172.21.15.33:42380) 2024-03-19T03:00:18.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.658+0000 7fde2ed9d640 1 -- 172.21.15.33:0/2461517928 learned_addr learned my addr 172.21.15.33:0/2461517928 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:18.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.658+0000 7fde2e59c640 1 --2- 172.21.15.33:0/2461517928 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c2bf63d000 0x55c2bf6456a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.658+0000 7fde2dd9b640 1 --2- 172.21.15.33:0/2461517928 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c2bf645be0 0x55c2bf647fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.658+0000 7fde2ed9d640 1 --2- 172.21.15.33:0/3432049225 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fde180a2970 0x7fde180a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.658+0000 7fde2ed9d640 1 -- 172.21.15.33:0/2461517928 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c2bf63d000 msgr2=0x55c2bf6456a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:18.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.658+0000 7fde2ed9d640 1 --2- 172.21.15.33:0/2461517928 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c2bf63d000 0x55c2bf6456a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.658+0000 7fde2ed9d640 1 -- 172.21.15.33:0/2461517928 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c2bf645be0 msgr2=0x55c2bf647fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:18.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.658+0000 7fde2ed9d640 1 --2- 172.21.15.33:0/2461517928 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c2bf645be0 0x55c2bf647fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.658+0000 7fde2ed9d640 1 -- 172.21.15.33:0/2461517928 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2bf566210 con 0x55c2bf63c6e0 2024-03-19T03:00:18.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.658+0000 7fde2e59c640 1 --2- 172.21.15.33:0/2461517928 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c2bf63d000 0x55c2bf6456a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:18.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.658+0000 7fde1f7fe640 1 -- 172.21.15.33:0/3432049225 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7fde240f0c10 con 0x7fde301013c0 2024-03-19T03:00:18.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.658+0000 7fde2ed9d640 1 --2- 172.21.15.33:0/2461517928 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2bf63c6e0 0x55c2bf63cac0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fde28005c40 tx=0x7fde2800f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e26e7208784b0a5b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:18.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.658+0000 7fde1cff9640 1 -- 172.21.15.33:0/2461517928 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde28022020 con 0x55c2bf63c6e0 2024-03-19T03:00:18.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.659+0000 7fde1cff9640 1 -- 172.21.15.33:0/2461517928 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fde28013170 con 0x55c2bf63c6e0 2024-03-19T03:00:18.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.659+0000 7fde1cff9640 1 -- 172.21.15.33:0/2461517928 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde28010690 con 0x55c2bf63c6e0 2024-03-19T03:00:18.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.659+0000 7fde2ed9d640 1 --2- 172.21.15.33:0/3432049225 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fde180a2970 0x7fde180a4e30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fde28023000 tx=0x7fde28021000 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:18.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.659+0000 7fde37a4f740 1 -- 172.21.15.33:0/2461517928 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2bf63c6e0 msgr2=0x55c2bf63cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.659+0000 7fde37a4f740 1 --2- 172.21.15.33:0/2461517928 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2bf63c6e0 0x55c2bf63cac0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fde28005c40 tx=0x7fde2800f3c0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.659+0000 7fde37a4f740 1 -- 172.21.15.33:0/2461517928 shutdown_connections 2024-03-19T03:00:18.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.659+0000 7fde37a4f740 1 --2- 172.21.15.33:0/2461517928 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c2bf63d000 0x55c2bf6456a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.659+0000 7fde37a4f740 1 --2- 172.21.15.33:0/2461517928 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c2bf645be0 0x55c2bf647fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.659+0000 7fde37a4f740 1 --2- 172.21.15.33:0/2461517928 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2bf63c6e0 0x55c2bf63cac0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.659+0000 7fde37a4f740 1 -- 172.21.15.33:0/2461517928 >> 172.21.15.33:0/2461517928 conn(0x55c2bf636e10 msgr2=0x55c2bf6371f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:18.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.659+0000 7fde37a4f740 1 -- 172.21.15.33:0/2461517928 shutdown_connections 2024-03-19T03:00:18.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.659+0000 7fde37a4f740 1 -- 172.21.15.33:0/2461517928 wait complete. 2024-03-19T03:00:18.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.660+0000 7fde37a4f740 1 -- start start 2024-03-19T03:00:18.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.660+0000 7fde37a4f740 1 Processor -- start 2024-03-19T03:00:18.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.660+0000 7fde37a4f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2bf63c6e0 0x55c2bf642c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.660+0000 7fde37a4f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c2bf63d000 0x55c2bf6431c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.660+0000 7fde37a4f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c2bf645be0 0x55c2bf643700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.660+0000 7fde37a4f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c2bf6489d0 con 0x55c2bf63c6e0 2024-03-19T03:00:18.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.660+0000 7fde37a4f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c2bf557310 con 0x55c2bf63d000 2024-03-19T03:00:18.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.660+0000 7fde37a4f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c2bf648b50 con 0x55c2bf645be0 2024-03-19T03:00:18.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.660+0000 7fde37a4f740 10 client.0 _ll_register_callbacks cb 0x55c2bf54b2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:00:18.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.660+0000 7fde2e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2bf63c6e0 0x55c2bf642c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.660+0000 7fde2e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2bf63c6e0 0x55c2bf642c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42394/0 (socket says 172.21.15.33:42394) 2024-03-19T03:00:18.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.660+0000 7fde2e59c640 1 -- 172.21.15.33:0/3214388092 learned_addr learned my addr 172.21.15.33:0/3214388092 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:18.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.660+0000 7fde2ed9d640 1 --2- 172.21.15.33:0/3214388092 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c2bf645be0 0x55c2bf643700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.660+0000 7fde2dd9b640 1 --2- 172.21.15.33:0/3214388092 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c2bf63d000 0x55c2bf6431c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.660+0000 7fde2e59c640 1 -- 172.21.15.33:0/3214388092 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c2bf645be0 msgr2=0x55c2bf643700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:18.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.660+0000 7fde2e59c640 1 --2- 172.21.15.33:0/3214388092 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c2bf645be0 0x55c2bf643700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.660+0000 7fde2e59c640 1 -- 172.21.15.33:0/3214388092 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c2bf63d000 msgr2=0x55c2bf6431c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:18.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.660+0000 7fde2e59c640 1 --2- 172.21.15.33:0/3214388092 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c2bf63d000 0x55c2bf6431c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.660+0000 7fde2e59c640 1 -- 172.21.15.33:0/3214388092 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2bf2af3f0 con 0x55c2bf63c6e0 2024-03-19T03:00:18.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.661+0000 7fde2e59c640 1 --2- 172.21.15.33:0/3214388092 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2bf63c6e0 0x55c2bf642c80 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fde200126d0 tx=0x7fde20012ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26fcd8b927fdfbc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:18.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.661+0000 7fde15ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:00:18.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.661+0000 7fde15ffb640 1 -- 172.21.15.33:0/3214388092 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2000f640 con 0x55c2bf63c6e0 2024-03-19T03:00:18.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.661+0000 7fde15ffb640 1 -- 172.21.15.33:0/3214388092 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7fde2000f7e0 con 0x55c2bf63c6e0 2024-03-19T03:00:18.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.661+0000 7fde37a4f740 1 -- 172.21.15.33:0/3214388092 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde28013170 con 0x55c2bf63c6e0 2024-03-19T03:00:18.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.661+0000 7fde15ffb640 1 -- 172.21.15.33:0/3214388092 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fde2000f9e0 con 0x55c2bf63c6e0 2024-03-19T03:00:18.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.661+0000 7fde37a4f740 10 client.17118 fetch_fsmap learned FSMap version 25 2024-03-19T03:00:18.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.661+0000 7fde37a4f740 1 -- 172.21.15.33:0/3214388092 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c2bf3797d0 con 0x55c2bf63c6e0 2024-03-19T03:00:18.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.661+0000 7fde15ffb640 1 -- 172.21.15.33:0/3214388092 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fde28013170 con 0x55c2bf63c6e0 2024-03-19T03:00:18.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.661+0000 7fde15ffb640 1 -- 172.21.15.33:0/3214388092 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fde20010c10 con 0x55c2bf63c6e0 2024-03-19T03:00:18.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.661+0000 7fde37a4f740 10 client.17118 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:00:18.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.661+0000 7fde37a4f740 10 client.17118 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T03:00:18.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.661+0000 7fde37a4f740 20 client.17118 populate_metadata read hostname 'smithi033' 2024-03-19T03:00:18.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.661+0000 7fde37a4f740 1 --2- 172.21.15.33:0/3214388092 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55c2bf656020 0x55c2bf658490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.661+0000 7fde37a4f740 4 client.17118 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:00:18.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.661+0000 7fde37a4f740 1 -- 172.21.15.33:0/3214388092 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c2bf5f93e0 con 0x55c2bf656020 2024-03-19T03:00:18.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.662+0000 7fde2dd9b640 1 --2- 172.21.15.33:0/3214388092 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55c2bf656020 0x55c2bf658490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.662+0000 7fde2dd9b640 1 --2- 172.21.15.33:0/3214388092 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55c2bf656020 0x55c2bf658490 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=40286b77baae7f73 server_cookie=a732e4b75fc27a97 in_seq=0 out_seq=0 2024-03-19T03:00:18.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.662+0000 7fde15ffb640 10 client.17118 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T03:00:18.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.662+0000 7fde15ffb640 1 -- 172.21.15.33:0/3214388092 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c2bf5f93e0 con 0x55c2bf656020 2024-03-19T03:00:18.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.662+0000 7fde15ffb640 10 client.17118 handle_command_reply: tid=0 2024-03-19T03:00:18.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.663+0000 7fde37a4f740 1 client.17118 shutdown 2024-03-19T03:00:18.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.663+0000 7fde37a4f740 1 -- 172.21.15.33:0/3214388092 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2bf63c6e0 msgr2=0x55c2bf642c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.663+0000 7fde37a4f740 1 --2- 172.21.15.33:0/3214388092 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2bf63c6e0 0x55c2bf642c80 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fde200126d0 tx=0x7fde20012ad0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.663+0000 7fde37a4f740 1 -- 172.21.15.33:0/3214388092 shutdown_connections 2024-03-19T03:00:18.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.663+0000 7fde37a4f740 1 --2- 172.21.15.33:0/3214388092 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55c2bf656020 0x55c2bf658490 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.663+0000 7fde37a4f740 1 --2- 172.21.15.33:0/3214388092 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c2bf645be0 0x55c2bf643700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.663+0000 7fde37a4f740 1 --2- 172.21.15.33:0/3214388092 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c2bf63d000 0x55c2bf6431c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.663+0000 7fde15ffb640 0 client.17118 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T03:00:18.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.663+0000 7fde37a4f740 1 --2- 172.21.15.33:0/3214388092 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2bf63c6e0 0x55c2bf642c80 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.663+0000 7fde37a4f740 1 -- 172.21.15.33:0/3214388092 >> 172.21.15.33:0/3214388092 conn(0x55c2bf636e10 msgr2=0x55c2bf631ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:18.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.663+0000 7fde37a4f740 1 -- 172.21.15.33:0/3214388092 shutdown_connections 2024-03-19T03:00:18.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.663+0000 7fde37a4f740 1 -- 172.21.15.33:0/3214388092 wait complete. 2024-03-19T03:00:18.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.663+0000 7fde37a4f740 20 client.17118 trim_cache size 0 max 16384 2024-03-19T03:00:18.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.677+0000 7fde37a4f740 1 Processor -- start 2024-03-19T03:00:18.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.677+0000 7fde37a4f740 1 -- start start 2024-03-19T03:00:18.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.677+0000 7fde37a4f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c2bf63c6e0 0x55c2bf651770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.677+0000 7fde37a4f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2bf63d000 0x55c2bf631a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.677+0000 7fde37a4f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c2bf645be0 0x55c2bf65dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.677+0000 7fde37a4f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fde301af2a0 con 0x55c2bf63d000 2024-03-19T03:00:18.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.677+0000 7fde37a4f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fde301acb70 con 0x55c2bf645be0 2024-03-19T03:00:18.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.677+0000 7fde37a4f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fde301aa9e0 con 0x55c2bf63c6e0 2024-03-19T03:00:18.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.677+0000 7fde2dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2bf63d000 0x55c2bf631a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.677+0000 7fde2dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2bf63d000 0x55c2bf631a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42404/0 (socket says 172.21.15.33:42404) 2024-03-19T03:00:18.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.677+0000 7fde2dd9b640 1 -- 172.21.15.33:0/3396155792 learned_addr learned my addr 172.21.15.33:0/3396155792 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:18.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.677+0000 7fde2ed9d640 1 --2- 172.21.15.33:0/3396155792 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c2bf645be0 0x55c2bf65dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.677+0000 7fde2e59c640 1 --2- 172.21.15.33:0/3396155792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c2bf63c6e0 0x55c2bf651770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.677+0000 7fde2dd9b640 1 -- 172.21.15.33:0/3396155792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c2bf63c6e0 msgr2=0x55c2bf651770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:18.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.677+0000 7fde2dd9b640 1 --2- 172.21.15.33:0/3396155792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c2bf63c6e0 0x55c2bf651770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.677+0000 7fde2dd9b640 1 -- 172.21.15.33:0/3396155792 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c2bf645be0 msgr2=0x55c2bf65dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:18.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.677+0000 7fde2dd9b640 1 --2- 172.21.15.33:0/3396155792 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c2bf645be0 0x55c2bf65dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.677+0000 7fde2dd9b640 1 -- 172.21.15.33:0/3396155792 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2bf62cfb0 con 0x55c2bf63d000 2024-03-19T03:00:18.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.678+0000 7fde2dd9b640 1 --2- 172.21.15.33:0/3396155792 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2bf63d000 0x55c2bf631a20 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fde2402c720 tx=0x7fde240c0040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=341f5d941df4818b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:18.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.678+0000 7fde17fff640 1 -- 172.21.15.33:0/3396155792 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde240c56f0 con 0x55c2bf63d000 2024-03-19T03:00:18.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.678+0000 7fde17fff640 1 -- 172.21.15.33:0/3396155792 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fde240bcf50 con 0x55c2bf63d000 2024-03-19T03:00:18.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.678+0000 7fde17fff640 1 -- 172.21.15.33:0/3396155792 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde240ee1b0 con 0x55c2bf63d000 2024-03-19T03:00:18.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.679+0000 7fde37a4f740 1 -- 172.21.15.33:0/3396155792 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2bf63d000 msgr2=0x55c2bf631a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.679+0000 7fde37a4f740 1 --2- 172.21.15.33:0/3396155792 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2bf63d000 0x55c2bf631a20 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fde2402c720 tx=0x7fde240c0040 comp rx=0 tx=0).stop 2024-03-19T03:00:18.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.679+0000 7fde37a4f740 1 -- 172.21.15.33:0/3396155792 shutdown_connections 2024-03-19T03:00:18.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.679+0000 7fde37a4f740 1 --2- 172.21.15.33:0/3396155792 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c2bf645be0 0x55c2bf65dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.679+0000 7fde37a4f740 1 --2- 172.21.15.33:0/3396155792 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2bf63d000 0x55c2bf631a20 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.679+0000 7fde37a4f740 1 --2- 172.21.15.33:0/3396155792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c2bf63c6e0 0x55c2bf651770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.679+0000 7fde37a4f740 1 -- 172.21.15.33:0/3396155792 >> 172.21.15.33:0/3396155792 conn(0x55c2bf636e10 msgr2=0x55c2bf631310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:18.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.679+0000 7fde37a4f740 1 -- 172.21.15.33:0/3396155792 shutdown_connections 2024-03-19T03:00:18.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.679+0000 7fde37a4f740 1 -- 172.21.15.33:0/3396155792 wait complete. 2024-03-19T03:00:18.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.679+0000 7fde37a4f740 1 -- start start 2024-03-19T03:00:18.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.679+0000 7fde37a4f740 1 Processor -- start 2024-03-19T03:00:18.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.679+0000 7fde37a4f740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c2bf63c6e0 0x55c2bf649cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.679+0000 7fde37a4f740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2bf63d000 0x55c2bf64a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.679+0000 7fde37a4f740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c2bf645be0 0x55c2bf6bb390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.679+0000 7fde37a4f740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fde301ab030 con 0x55c2bf63d000 2024-03-19T03:00:18.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.679+0000 7fde37a4f740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fde301ac4a0 con 0x55c2bf63c6e0 2024-03-19T03:00:18.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.679+0000 7fde37a4f740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fde301ab700 con 0x55c2bf645be0 2024-03-19T03:00:18.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.679+0000 7fde37a4f740 10 client.0 _ll_register_callbacks cb 0x7fde20009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:00:18.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.679+0000 7fde2dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2bf63d000 0x55c2bf64a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.679+0000 7fde2dd9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2bf63d000 0x55c2bf64a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42420/0 (socket says 172.21.15.33:42420) 2024-03-19T03:00:18.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.679+0000 7fde2dd9b640 1 -- 172.21.15.33:0/1712629412 learned_addr learned my addr 172.21.15.33:0/1712629412 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:18.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.679+0000 7fde2e59c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c2bf63c6e0 0x55c2bf649cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.679+0000 7fde2ed9d640 1 --2- 172.21.15.33:0/1712629412 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c2bf645be0 0x55c2bf6bb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.697 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.680+0000 7fde2dd9b640 1 -- 172.21.15.33:0/1712629412 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c2bf645be0 msgr2=0x55c2bf6bb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:18.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.680+0000 7fde2dd9b640 1 --2- 172.21.15.33:0/1712629412 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c2bf645be0 0x55c2bf6bb390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.680+0000 7fde2dd9b640 1 -- 172.21.15.33:0/1712629412 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c2bf63c6e0 msgr2=0x55c2bf649cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:18.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.680+0000 7fde2dd9b640 1 --2- 172.21.15.33:0/1712629412 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c2bf63c6e0 0x55c2bf649cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.680+0000 7fde2dd9b640 1 -- 172.21.15.33:0/1712629412 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2bf6bb8d0 con 0x55c2bf63d000 2024-03-19T03:00:18.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.680+0000 7fde2dd9b640 1 --2- 172.21.15.33:0/1712629412 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2bf63d000 0x55c2bf64a200 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fde240045d0 tx=0x7fde240b9320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2116cbba3b0c8aa server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:18.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.680+0000 7fde15ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:00:18.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.680+0000 7fde15ffb640 1 -- 172.21.15.33:0/1712629412 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde240b95c0 con 0x55c2bf63d000 2024-03-19T03:00:18.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.680+0000 7fde15ffb640 1 -- 172.21.15.33:0/1712629412 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7fde240ee1b0 con 0x55c2bf63d000 2024-03-19T03:00:18.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.680+0000 7fde37a4f740 1 -- 172.21.15.33:0/1712629412 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde240bcf50 con 0x55c2bf63d000 2024-03-19T03:00:18.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.680+0000 7fde15ffb640 1 -- 172.21.15.33:0/1712629412 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fde240a3050 con 0x55c2bf63d000 2024-03-19T03:00:18.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.680+0000 7fde37a4f740 10 client.17124 fetch_fsmap learned FSMap version 25 2024-03-19T03:00:18.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.680+0000 7fde37a4f740 1 -- 172.21.15.33:0/1712629412 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c2bf6bcad0 con 0x55c2bf63d000 2024-03-19T03:00:18.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.680+0000 7fde15ffb640 1 -- 172.21.15.33:0/1712629412 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fde240bcf50 con 0x55c2bf63d000 2024-03-19T03:00:18.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.680+0000 7fde15ffb640 1 -- 172.21.15.33:0/1712629412 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fde24003c40 con 0x55c2bf63d000 2024-03-19T03:00:18.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.680+0000 7fde37a4f740 10 client.17124 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:00:18.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.680+0000 7fde37a4f740 10 client.17124 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T03:00:18.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.680+0000 7fde37a4f740 20 client.17124 populate_metadata read hostname 'smithi033' 2024-03-19T03:00:18.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.680+0000 7fde37a4f740 1 --2- 172.21.15.33:0/1712629412 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55c2bf6bd140 0x55c2bf6bf560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.680+0000 7fde37a4f740 4 client.17124 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T03:00:18.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.680+0000 7fde37a4f740 1 -- 172.21.15.33:0/1712629412 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c2bf5a89f0 con 0x55c2bf6bd140 2024-03-19T03:00:18.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.681+0000 7fde2e59c640 1 --2- 172.21.15.33:0/1712629412 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55c2bf6bd140 0x55c2bf6bf560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.681+0000 7fde2e59c640 1 --2- 172.21.15.33:0/1712629412 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55c2bf6bd140 0x55c2bf6bf560 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2f00d2b2820ef29c server_cookie=cbd49e94b89f35fe in_seq=0 out_seq=0 2024-03-19T03:00:18.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.681+0000 7fde15ffb640 10 client.17124 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T03:00:18.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.681+0000 7fde15ffb640 1 -- 172.21.15.33:0/1712629412 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c2bf5a89f0 con 0x55c2bf6bd140 2024-03-19T03:00:18.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.681+0000 7fde15ffb640 10 client.17124 handle_command_reply: tid=0 2024-03-19T03:00:18.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.681+0000 7fde37a4f740 1 client.17124 shutdown 2024-03-19T03:00:18.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.681+0000 7fde37a4f740 1 -- 172.21.15.33:0/1712629412 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2bf63d000 msgr2=0x55c2bf64a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.681+0000 7fde37a4f740 1 --2- 172.21.15.33:0/1712629412 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2bf63d000 0x55c2bf64a200 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fde240045d0 tx=0x7fde240b9320 comp rx=0 tx=0).stop 2024-03-19T03:00:18.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.681+0000 7fde37a4f740 1 -- 172.21.15.33:0/1712629412 shutdown_connections 2024-03-19T03:00:18.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.681+0000 7fde37a4f740 1 --2- 172.21.15.33:0/1712629412 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55c2bf6bd140 0x55c2bf6bf560 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.681+0000 7fde37a4f740 1 --2- 172.21.15.33:0/1712629412 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c2bf645be0 0x55c2bf6bb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.681+0000 7fde37a4f740 1 --2- 172.21.15.33:0/1712629412 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c2bf63d000 0x55c2bf64a200 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.681+0000 7fde15ffb640 0 client.17124 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T03:00:18.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.681+0000 7fde37a4f740 1 --2- 172.21.15.33:0/1712629412 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c2bf63c6e0 0x55c2bf649cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.681+0000 7fde37a4f740 1 -- 172.21.15.33:0/1712629412 >> 172.21.15.33:0/1712629412 conn(0x55c2bf636e10 msgr2=0x55c2bf631ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:18.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.681+0000 7fde37a4f740 1 -- 172.21.15.33:0/1712629412 shutdown_connections 2024-03-19T03:00:18.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.681+0000 7fde37a4f740 1 -- 172.21.15.33:0/1712629412 wait complete. 2024-03-19T03:00:18.703 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.681+0000 7fde37a4f740 20 client.17124 trim_cache size 0 max 16384 2024-03-19T03:00:18.703 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T03:00:18.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.682+0000 7fde34e68640 1 -- 172.21.15.33:0/3432049225 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fde180a2970 msgr2=0x7fde180a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.682+0000 7fde34e68640 1 --2- 172.21.15.33:0/3432049225 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fde180a2970 0x7fde180a4e30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fde28023000 tx=0x7fde28021000 comp rx=0 tx=0).stop 2024-03-19T03:00:18.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.682+0000 7fde34e68640 1 -- 172.21.15.33:0/3432049225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde301013c0 msgr2=0x7fde30199290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.682+0000 7fde34e68640 1 --2- 172.21.15.33:0/3432049225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde301013c0 0x7fde30199290 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fde24006010 tx=0x7fde24005d90 comp rx=0 tx=0).stop 2024-03-19T03:00:18.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.682+0000 7fde34e68640 1 -- 172.21.15.33:0/3432049225 shutdown_connections 2024-03-19T03:00:18.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.682+0000 7fde34e68640 1 --2- 172.21.15.33:0/3432049225 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fde180a2970 0x7fde180a4e30 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.682+0000 7fde34e68640 1 --2- 172.21.15.33:0/3432049225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fde30105680 0x7fde301a3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.705 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.682+0000 7fde34e68640 1 --2- 172.21.15.33:0/3432049225 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fde301013c0 0x7fde30199290 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.682+0000 7fde34e68640 1 --2- 172.21.15.33:0/3432049225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde30100a00 0x7fde30198d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.682+0000 7fde34e68640 1 -- 172.21.15.33:0/3432049225 >> 172.21.15.33:0/3432049225 conn(0x7fde30077700 msgr2=0x7fde300fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:18.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.682+0000 7fde34e68640 1 -- 172.21.15.33:0/3432049225 shutdown_connections 2024-03-19T03:00:18.706 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.682+0000 7fde34e68640 1 -- 172.21.15.33:0/3432049225 wait complete. 2024-03-19T03:00:18.706 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T03:00:18.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.809+0000 7f47d5685640 1 -- 172.21.15.33:0/3976288364 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f47d010bd80 msgr2=0x7f47d010e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.809+0000 7f47d5685640 1 --2- 172.21.15.33:0/3976288364 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f47d010bd80 0x7f47d010e220 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f47c0009ca0 tx=0x7f47c002c180 comp rx=0 tx=0).stop 2024-03-19T03:00:18.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.809+0000 7f47d5685640 1 -- 172.21.15.33:0/3976288364 shutdown_connections 2024-03-19T03:00:18.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.809+0000 7f47d5685640 1 --2- 172.21.15.33:0/3976288364 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47d0103360 0x7f47d010b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.809+0000 7f47d5685640 1 --2- 172.21.15.33:0/3976288364 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f47d010bd80 0x7f47d010e220 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.809+0000 7f47d5685640 1 --2- 172.21.15.33:0/3976288364 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47d0102a40 0x7f47d0102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.809+0000 7f47d5685640 1 -- 172.21.15.33:0/3976288364 >> 172.21.15.33:0/3976288364 conn(0x7f47d00778d0 msgr2=0x7f47d0077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:18.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.809+0000 7f47d5685640 1 -- 172.21.15.33:0/3976288364 shutdown_connections 2024-03-19T03:00:18.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.809+0000 7f47d5685640 1 -- 172.21.15.33:0/3976288364 wait complete. 2024-03-19T03:00:18.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.810+0000 7f47d5685640 1 Processor -- start 2024-03-19T03:00:18.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.810+0000 7f47d5685640 1 -- start start 2024-03-19T03:00:18.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.810+0000 7f47d5685640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f47d0102a40 0x7f47d0197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.810+0000 7f47d5685640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47d0103360 0x7f47d0198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.810+0000 7f47d5685640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47d010bd80 0x7f47d019c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.810+0000 7f47d5685640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f47d0111a30 con 0x7f47d0102a40 2024-03-19T03:00:18.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.810+0000 7f47d5685640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f47d01118b0 con 0x7f47d0103360 2024-03-19T03:00:18.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.810+0000 7f47d5685640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f47d0111bb0 con 0x7f47d010bd80 2024-03-19T03:00:18.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.810+0000 7f47ceffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f47d0102a40 0x7f47d0197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.811+0000 7f47ceffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f47d0102a40 0x7f47d0197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42434/0 (socket says 172.21.15.33:42434) 2024-03-19T03:00:18.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.811+0000 7f47ceffd640 1 -- 172.21.15.33:0/282059751 learned_addr learned my addr 172.21.15.33:0/282059751 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:18.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.811+0000 7f47ce7fc640 1 --2- 172.21.15.33:0/282059751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47d0103360 0x7f47d0198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.811+0000 7f47cf7fe640 1 --2- 172.21.15.33:0/282059751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47d010bd80 0x7f47d019c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.811+0000 7f47ceffd640 1 -- 172.21.15.33:0/282059751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47d010bd80 msgr2=0x7f47d019c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.811+0000 7f47ceffd640 1 --2- 172.21.15.33:0/282059751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47d010bd80 0x7f47d019c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.811+0000 7f47ceffd640 1 -- 172.21.15.33:0/282059751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47d0103360 msgr2=0x7f47d0198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.811+0000 7f47ceffd640 1 --2- 172.21.15.33:0/282059751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47d0103360 0x7f47d0198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.811+0000 7f47ceffd640 1 -- 172.21.15.33:0/282059751 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d019cf20 con 0x7f47d0102a40 2024-03-19T03:00:18.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.811+0000 7f47cf7fe640 1 --2- 172.21.15.33:0/282059751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47d010bd80 0x7f47d019c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:18.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.811+0000 7f47ce7fc640 1 --2- 172.21.15.33:0/282059751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47d0103360 0x7f47d0198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:18.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.811+0000 7f47ceffd640 1 --2- 172.21.15.33:0/282059751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f47d0102a40 0x7f47d0197f30 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f47c800ea60 tx=0x7f47c800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:18.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.811+0000 7f47c7fff640 1 -- 172.21.15.33:0/282059751 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c800cde0 con 0x7f47d0102a40 2024-03-19T03:00:18.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.811+0000 7f47c7fff640 1 -- 172.21.15.33:0/282059751 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47c80035f0 con 0x7f47d0102a40 2024-03-19T03:00:18.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.811+0000 7f47d5685640 1 -- 172.21.15.33:0/282059751 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47d019d210 con 0x7f47d0102a40 2024-03-19T03:00:18.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.811+0000 7f47c7fff640 1 -- 172.21.15.33:0/282059751 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c8010690 con 0x7f47d0102a40 2024-03-19T03:00:18.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.811+0000 7f47d5685640 1 -- 172.21.15.33:0/282059751 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47d01a4aa0 con 0x7f47d0102a40 2024-03-19T03:00:18.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.813+0000 7f47c7fff640 1 -- 172.21.15.33:0/282059751 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f47c80037a0 con 0x7f47d0102a40 2024-03-19T03:00:18.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.814+0000 7f47d8276740 1 Processor -- start 2024-03-19T03:00:18.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.814+0000 7f47d8276740 1 -- start start 2024-03-19T03:00:18.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.814+0000 7f47d8276740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f8036712d0 0x55f8036716b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.814+0000 7f47d8276740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f803671bf0 0x55f80367a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.814+0000 7f47c7fff640 1 --2- 172.21.15.33:0/282059751 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f47bc0a29f0 0x7f47bc0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.814+0000 7f47d8276740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f80367a7d0 0x55f80367cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.814+0000 7f47d8276740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f8035d44b0 con 0x55f8036712d0 2024-03-19T03:00:18.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.814+0000 7f47d8276740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55f80357cd70 con 0x55f803671bf0 2024-03-19T03:00:18.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.814+0000 7f47d8276740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f8035d0550 con 0x55f80367a7d0 2024-03-19T03:00:18.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.814+0000 7f47cf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f8036712d0 0x55f8036716b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.814+0000 7f47ceffd640 1 --2- 172.21.15.33:0/282059751 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f47bc0a29f0 0x7f47bc0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.814+0000 7f47cf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f8036712d0 0x55f8036716b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42438/0 (socket says 172.21.15.33:42438) 2024-03-19T03:00:18.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.814+0000 7f47cf7fe640 1 -- 172.21.15.33:0/2229881629 learned_addr learned my addr 172.21.15.33:0/2229881629 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:18.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.814+0000 7f47ce7fc640 1 --2- 172.21.15.33:0/2229881629 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f803671bf0 0x55f80367a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.814+0000 7f47cf7fe640 1 --2- 172.21.15.33:0/2229881629 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f80367a7d0 0x55f80367cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.814+0000 7f47c7fff640 1 -- 172.21.15.33:0/282059751 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7f47c80cf410 con 0x7f47d0102a40 2024-03-19T03:00:18.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.815+0000 7f47cf7fe640 1 -- 172.21.15.33:0/2229881629 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f80367a7d0 msgr2=0x55f80367cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:18.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.815+0000 7f47cf7fe640 1 --2- 172.21.15.33:0/2229881629 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f80367a7d0 0x55f80367cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.815+0000 7f47cf7fe640 1 -- 172.21.15.33:0/2229881629 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f803671bf0 msgr2=0x55f80367a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:18.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.815+0000 7f47cf7fe640 1 --2- 172.21.15.33:0/2229881629 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f803671bf0 0x55f80367a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.815+0000 7f47cf7fe640 1 -- 172.21.15.33:0/2229881629 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f80359f210 con 0x55f8036712d0 2024-03-19T03:00:18.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.815+0000 7f47ce7fc640 1 --2- 172.21.15.33:0/2229881629 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f803671bf0 0x55f80367a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.815+0000 7f47cf7fe640 1 --2- 172.21.15.33:0/2229881629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f8036712d0 0x55f8036716b0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f47c0009430 tx=0x7f47c0009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=592c024e4be83842 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.815+0000 7f47ceffd640 1 --2- 172.21.15.33:0/282059751 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f47bc0a29f0 0x7f47bc0a4eb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f47c80cc840 tx=0x7f47c800ef90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.815+0000 7f47c57fa640 1 -- 172.21.15.33:0/2229881629 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c0048070 con 0x55f8036712d0 2024-03-19T03:00:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.815+0000 7f47c57fa640 1 -- 172.21.15.33:0/2229881629 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47c00325f0 con 0x55f8036712d0 2024-03-19T03:00:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.815+0000 7f47c57fa640 1 -- 172.21.15.33:0/2229881629 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c0039040 con 0x55f8036712d0 2024-03-19T03:00:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.816+0000 7f47d8276740 1 -- 172.21.15.33:0/2229881629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f8036712d0 msgr2=0x55f8036716b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.816+0000 7f47d8276740 1 --2- 172.21.15.33:0/2229881629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f8036712d0 0x55f8036716b0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f47c0009430 tx=0x7f47c0009360 comp rx=0 tx=0).stop 2024-03-19T03:00:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.816+0000 7f47d8276740 1 -- 172.21.15.33:0/2229881629 shutdown_connections 2024-03-19T03:00:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.816+0000 7f47d8276740 1 --2- 172.21.15.33:0/2229881629 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f80367a7d0 0x55f80367cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.816+0000 7f47d8276740 1 --2- 172.21.15.33:0/2229881629 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f803671bf0 0x55f80367a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.816+0000 7f47d8276740 1 --2- 172.21.15.33:0/2229881629 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f8036712d0 0x55f8036716b0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.816+0000 7f47d8276740 1 -- 172.21.15.33:0/2229881629 >> 172.21.15.33:0/2229881629 conn(0x55f80366ba00 msgr2=0x55f80366bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.816+0000 7f47d8276740 1 -- 172.21.15.33:0/2229881629 shutdown_connections 2024-03-19T03:00:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.816+0000 7f47d8276740 1 -- 172.21.15.33:0/2229881629 wait complete. 2024-03-19T03:00:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.816+0000 7f47d8276740 1 -- start start 2024-03-19T03:00:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.816+0000 7f47d8276740 1 Processor -- start 2024-03-19T03:00:18.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.817+0000 7f47d8276740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f8036712d0 0x55f803677810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.817+0000 7f47d8276740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f803671bf0 0x55f803677d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.817+0000 7f47d8276740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f80367a7d0 0x55f803678290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.817+0000 7f47d8276740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55f80367d560 con 0x55f80367a7d0 2024-03-19T03:00:18.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.817+0000 7f47d8276740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55f803590310 con 0x55f803671bf0 2024-03-19T03:00:18.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.817+0000 7f47d8276740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55f80367d6e0 con 0x55f8036712d0 2024-03-19T03:00:18.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.817+0000 7f47d8276740 10 client.0 _ll_register_callbacks cb 0x55f8035842d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:00:18.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.817+0000 7f47cf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f80367a7d0 0x55f803678290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.817+0000 7f47cf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f80367a7d0 0x55f803678290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42452/0 (socket says 172.21.15.33:42452) 2024-03-19T03:00:18.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.817+0000 7f47cf7fe640 1 -- 172.21.15.33:0/962273021 learned_addr learned my addr 172.21.15.33:0/962273021 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:18.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.817+0000 7f47cf7fe640 1 --2- 172.21.15.33:0/962273021 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f8036712d0 0x55f803677810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.817+0000 7f47ce7fc640 1 --2- 172.21.15.33:0/962273021 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f803671bf0 0x55f803677d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.817+0000 7f47cf7fe640 1 -- 172.21.15.33:0/962273021 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f8036712d0 msgr2=0x55f803677810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:18.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.817+0000 7f47cf7fe640 1 --2- 172.21.15.33:0/962273021 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f8036712d0 0x55f803677810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.817+0000 7f47cf7fe640 1 -- 172.21.15.33:0/962273021 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f803671bf0 msgr2=0x55f803677d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:18.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.817+0000 7f47cf7fe640 1 --2- 172.21.15.33:0/962273021 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f803671bf0 0x55f803677d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.817+0000 7f47cf7fe640 1 -- 172.21.15.33:0/962273021 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8032e83f0 con 0x55f80367a7d0 2024-03-19T03:00:18.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.817+0000 7f47cf7fe640 1 --2- 172.21.15.33:0/962273021 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f80367a7d0 0x55f803678290 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f47c002c790 tx=0x7f47c0032ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4edde9465aac925 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:18.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.817+0000 7f47b67fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:00:18.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.817+0000 7f47b67fc640 1 -- 172.21.15.33:0/962273021 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c0039030 con 0x55f80367a7d0 2024-03-19T03:00:18.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.817+0000 7f47b67fc640 1 -- 172.21.15.33:0/962273021 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7f47c00347f0 con 0x55f80367a7d0 2024-03-19T03:00:18.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.817+0000 7f47d8276740 1 -- 172.21.15.33:0/962273021 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47c00325f0 con 0x55f80367a7d0 2024-03-19T03:00:18.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.817+0000 7f47b67fc640 1 -- 172.21.15.33:0/962273021 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f47c005c050 con 0x55f80367a7d0 2024-03-19T03:00:18.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.818+0000 7f47d8276740 10 client.17136 fetch_fsmap learned FSMap version 25 2024-03-19T03:00:18.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.818+0000 7f47d8276740 1 -- 172.21.15.33:0/962273021 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f8033b27d0 con 0x55f80367a7d0 2024-03-19T03:00:18.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.818+0000 7f47b67fc640 1 -- 172.21.15.33:0/962273021 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47c00325f0 con 0x55f80367a7d0 2024-03-19T03:00:18.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.818+0000 7f47b67fc640 1 -- 172.21.15.33:0/962273021 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f47c004b050 con 0x55f80367a7d0 2024-03-19T03:00:18.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.818+0000 7f47d8276740 10 client.17136 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:00:18.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.818+0000 7f47d8276740 10 client.17136 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:00:18.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.818+0000 7f47d8276740 20 client.17136 populate_metadata read hostname 'smithi033' 2024-03-19T03:00:18.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.818+0000 7f47d8276740 1 --2- 172.21.15.33:0/962273021 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f80368a9b0 0x55f80368cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.818+0000 7f47d8276740 4 client.17136 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:00:18.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.818+0000 7f47d8276740 1 -- 172.21.15.33:0/962273021 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f8036323e0 con 0x55f80368a9b0 2024-03-19T03:00:18.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.818+0000 7f47ce7fc640 1 --2- 172.21.15.33:0/962273021 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f80368a9b0 0x55f80368cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.818+0000 7f47b67fc640 10 client.17136 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:00:18.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.818+0000 7f47ce7fc640 1 --2- 172.21.15.33:0/962273021 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f80368a9b0 0x55f80368cda0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8abd6a19880e11ba server_cookie=cd186d93c9258a00 in_seq=0 out_seq=0 2024-03-19T03:00:18.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.819+0000 7f47b67fc640 1 -- 172.21.15.33:0/962273021 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f8036323e0 con 0x55f80368a9b0 2024-03-19T03:00:18.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.819+0000 7f47b67fc640 10 client.17136 handle_command_reply: tid=0 2024-03-19T03:00:18.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.819+0000 7f47d8276740 1 client.17136 shutdown 2024-03-19T03:00:18.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.819+0000 7f47d8276740 1 -- 172.21.15.33:0/962273021 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f80367a7d0 msgr2=0x55f803678290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.819+0000 7f47d8276740 1 --2- 172.21.15.33:0/962273021 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f80367a7d0 0x55f803678290 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f47c002c790 tx=0x7f47c0032ed0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.819+0000 7f47d8276740 1 -- 172.21.15.33:0/962273021 shutdown_connections 2024-03-19T03:00:18.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.819+0000 7f47d8276740 1 --2- 172.21.15.33:0/962273021 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f80368a9b0 0x55f80368cda0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.819+0000 7f47d8276740 1 --2- 172.21.15.33:0/962273021 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f80367a7d0 0x55f803678290 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.819+0000 7f47d8276740 1 --2- 172.21.15.33:0/962273021 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f803671bf0 0x55f803677d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.819+0000 7f47b67fc640 0 client.17136 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:00:18.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.819+0000 7f47d8276740 1 --2- 172.21.15.33:0/962273021 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f8036712d0 0x55f803677810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.819+0000 7f47d8276740 1 -- 172.21.15.33:0/962273021 >> 172.21.15.33:0/962273021 conn(0x55f80366ba00 msgr2=0x55f80366d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:18.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.819+0000 7f47d8276740 1 -- 172.21.15.33:0/962273021 shutdown_connections 2024-03-19T03:00:18.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.819+0000 7f47d8276740 1 -- 172.21.15.33:0/962273021 wait complete. 2024-03-19T03:00:18.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.819+0000 7f47d8276740 20 client.17136 trim_cache size 0 max 16384 2024-03-19T03:00:18.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.832+0000 7f47d8276740 1 Processor -- start 2024-03-19T03:00:18.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.832+0000 7f47d8276740 1 -- start start 2024-03-19T03:00:18.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.833+0000 7f47d8276740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f8036712d0 0x55f803679f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.833+0000 7f47d8276740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f803671bf0 0x55f80366ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.833+0000 7f47d8276740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f80367a7d0 0x55f80368fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.833+0000 7f47d8276740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f47d01b15c0 con 0x55f8036712d0 2024-03-19T03:00:18.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.833+0000 7f47d8276740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f47d01aee90 con 0x55f803671bf0 2024-03-19T03:00:18.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.833+0000 7f47d8276740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f47d01acd00 con 0x55f80367a7d0 2024-03-19T03:00:18.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.833+0000 7f47cf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f8036712d0 0x55f803679f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.833+0000 7f47cf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f8036712d0 0x55f803679f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42458/0 (socket says 172.21.15.33:42458) 2024-03-19T03:00:18.840 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.833+0000 7f47cf7fe640 1 -- 172.21.15.33:0/3352398792 learned_addr learned my addr 172.21.15.33:0/3352398792 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:18.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.833+0000 7f47ce7fc640 1 --2- 172.21.15.33:0/3352398792 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f803671bf0 0x55f80366ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.833+0000 7f47cf7fe640 1 --2- 172.21.15.33:0/3352398792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f80367a7d0 0x55f80368fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.833+0000 7f47cf7fe640 1 -- 172.21.15.33:0/3352398792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f80367a7d0 msgr2=0x55f80368fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:18.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.833+0000 7f47cf7fe640 1 --2- 172.21.15.33:0/3352398792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f80367a7d0 0x55f80368fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.833+0000 7f47cf7fe640 1 -- 172.21.15.33:0/3352398792 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f803671bf0 msgr2=0x55f80366ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:18.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.833+0000 7f47cf7fe640 1 --2- 172.21.15.33:0/3352398792 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f803671bf0 0x55f80366ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.833+0000 7f47cf7fe640 1 -- 172.21.15.33:0/3352398792 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f803641880 con 0x55f8036712d0 2024-03-19T03:00:18.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.833+0000 7f47cf7fe640 1 --2- 172.21.15.33:0/3352398792 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f8036712d0 0x55f803679f40 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f47c0051630 tx=0x7f47c0036040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5103479818c0e20 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:18.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.833+0000 7f47c4ff9640 1 -- 172.21.15.33:0/3352398792 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c0067070 con 0x55f8036712d0 2024-03-19T03:00:18.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.834+0000 7f47c4ff9640 1 -- 172.21.15.33:0/3352398792 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47c0003e20 con 0x55f8036712d0 2024-03-19T03:00:18.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.834+0000 7f47c4ff9640 1 -- 172.21.15.33:0/3352398792 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c0039070 con 0x55f8036712d0 2024-03-19T03:00:18.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.834+0000 7f47d8276740 1 -- 172.21.15.33:0/3352398792 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f8036712d0 msgr2=0x55f803679f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.834+0000 7f47d8276740 1 --2- 172.21.15.33:0/3352398792 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f8036712d0 0x55f803679f40 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f47c0051630 tx=0x7f47c0036040 comp rx=0 tx=0).stop 2024-03-19T03:00:18.841 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.834+0000 7f47d8276740 1 -- 172.21.15.33:0/3352398792 shutdown_connections 2024-03-19T03:00:18.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.834+0000 7f47d8276740 1 --2- 172.21.15.33:0/3352398792 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f80367a7d0 0x55f80368fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.834+0000 7f47d8276740 1 --2- 172.21.15.33:0/3352398792 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f803671bf0 0x55f80366ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.834+0000 7f47d8276740 1 --2- 172.21.15.33:0/3352398792 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f8036712d0 0x55f803679f40 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.834+0000 7f47d8276740 1 -- 172.21.15.33:0/3352398792 >> 172.21.15.33:0/3352398792 conn(0x55f80366ba00 msgr2=0x55f8036861f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:18.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.834+0000 7f47d8276740 1 -- 172.21.15.33:0/3352398792 shutdown_connections 2024-03-19T03:00:18.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.834+0000 7f47d8276740 1 -- 172.21.15.33:0/3352398792 wait complete. 2024-03-19T03:00:18.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.834+0000 7f47d8276740 1 -- start start 2024-03-19T03:00:18.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.835+0000 7f47d8276740 1 Processor -- start 2024-03-19T03:00:18.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.835+0000 7f47d8276740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f8036712d0 0x55f803690210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.835+0000 7f47d8276740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f803671bf0 0x55f80368d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.835+0000 7f47d8276740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f80367a7d0 0x55f80368dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.835+0000 7f47d8276740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f47d01ad350 con 0x55f80367a7d0 2024-03-19T03:00:18.842 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.835+0000 7f47d8276740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f47d01ae7c0 con 0x55f803671bf0 2024-03-19T03:00:18.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.835+0000 7f47d8276740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f47d01ada20 con 0x55f8036712d0 2024-03-19T03:00:18.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.835+0000 7f47d8276740 10 client.0 _ll_register_callbacks cb 0x7f47c0046070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:00:18.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.835+0000 7f47cf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f80367a7d0 0x55f80368dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.835+0000 7f47cf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f80367a7d0 0x55f80368dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42464/0 (socket says 172.21.15.33:42464) 2024-03-19T03:00:18.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.835+0000 7f47cf7fe640 1 -- 172.21.15.33:0/2718102318 learned_addr learned my addr 172.21.15.33:0/2718102318 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:18.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.835+0000 7f47ce7fc640 1 --2- 172.21.15.33:0/2718102318 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f803671bf0 0x55f80368d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.835+0000 7f47cf7fe640 1 --2- 172.21.15.33:0/2718102318 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f8036712d0 0x55f803690210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.835+0000 7f47cf7fe640 1 -- 172.21.15.33:0/2718102318 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f8036712d0 msgr2=0x55f803690210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:18.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.835+0000 7f47cf7fe640 1 --2- 172.21.15.33:0/2718102318 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f8036712d0 0x55f803690210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.835+0000 7f47cf7fe640 1 -- 172.21.15.33:0/2718102318 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f803671bf0 msgr2=0x55f80368d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:18.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.835+0000 7f47cf7fe640 1 --2- 172.21.15.33:0/2718102318 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f803671bf0 0x55f80368d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.835+0000 7f47cf7fe640 1 -- 172.21.15.33:0/2718102318 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8036ef950 con 0x55f80367a7d0 2024-03-19T03:00:18.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.835+0000 7f47ce7fc640 1 --2- 172.21.15.33:0/2718102318 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f803671bf0 0x55f80368d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:18.843 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.835+0000 7f47cf7fe640 1 --2- 172.21.15.33:0/2718102318 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f80367a7d0 0x55f80368dc90 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f47c0034850 tx=0x7f47c002c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9921c44f0b399474 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:18.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.835+0000 7f47b67fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:00:18.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.835+0000 7f47b67fc640 1 -- 172.21.15.33:0/2718102318 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c0039070 con 0x55f80367a7d0 2024-03-19T03:00:18.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.836+0000 7f47b67fc640 1 -- 172.21.15.33:0/2718102318 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7f47c0003970 con 0x55f80367a7d0 2024-03-19T03:00:18.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.836+0000 7f47d8276740 1 -- 172.21.15.33:0/2718102318 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47c0003e20 con 0x55f80367a7d0 2024-03-19T03:00:18.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.836+0000 7f47b67fc640 1 -- 172.21.15.33:0/2718102318 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f47c0067040 con 0x55f80367a7d0 2024-03-19T03:00:18.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.836+0000 7f47d8276740 10 client.17142 fetch_fsmap learned FSMap version 25 2024-03-19T03:00:18.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.836+0000 7f47d8276740 1 -- 172.21.15.33:0/2718102318 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f8036f0af0 con 0x55f80367a7d0 2024-03-19T03:00:18.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.836+0000 7f47b67fc640 1 -- 172.21.15.33:0/2718102318 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47c0003e20 con 0x55f80367a7d0 2024-03-19T03:00:18.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.836+0000 7f47b67fc640 1 -- 172.21.15.33:0/2718102318 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f47c004b070 con 0x55f80367a7d0 2024-03-19T03:00:18.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.836+0000 7f47d8276740 10 client.17142 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:00:18.844 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.836+0000 7f47d8276740 10 client.17142 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:00:18.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.836+0000 7f47d8276740 20 client.17142 populate_metadata read hostname 'smithi033' 2024-03-19T03:00:18.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.836+0000 7f47d8276740 1 --2- 172.21.15.33:0/2718102318 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f8036f1160 0x55f8036f3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:18.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.836+0000 7f47d8276740 4 client.17142 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T03:00:18.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.836+0000 7f47d8276740 1 -- 172.21.15.33:0/2718102318 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f8035e19f0 con 0x55f8036f1160 2024-03-19T03:00:18.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.836+0000 7f47ce7fc640 1 --2- 172.21.15.33:0/2718102318 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f8036f1160 0x55f8036f3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:18.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.837+0000 7f47b67fc640 10 client.17142 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:00:18.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.837+0000 7f47ce7fc640 1 --2- 172.21.15.33:0/2718102318 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f8036f1160 0x55f8036f3550 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bd3f2b1a076b2a8 server_cookie=a6537c544cc3a414 in_seq=0 out_seq=0 2024-03-19T03:00:18.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.837+0000 7f47b67fc640 1 -- 172.21.15.33:0/2718102318 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f8035e19f0 con 0x55f8036f1160 2024-03-19T03:00:18.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.837+0000 7f47b67fc640 10 client.17142 handle_command_reply: tid=0 2024-03-19T03:00:18.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.837+0000 7f47d8276740 1 client.17142 shutdown 2024-03-19T03:00:18.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.837+0000 7f47d8276740 1 -- 172.21.15.33:0/2718102318 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f80367a7d0 msgr2=0x55f80368dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.845 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.837+0000 7f47d8276740 1 --2- 172.21.15.33:0/2718102318 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f80367a7d0 0x55f80368dc90 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f47c0034850 tx=0x7f47c002c800 comp rx=0 tx=0).stop 2024-03-19T03:00:18.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.837+0000 7f47d8276740 1 -- 172.21.15.33:0/2718102318 shutdown_connections 2024-03-19T03:00:18.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.837+0000 7f47d8276740 1 --2- 172.21.15.33:0/2718102318 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55f8036f1160 0x55f8036f3550 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.837+0000 7f47d8276740 1 --2- 172.21.15.33:0/2718102318 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55f80367a7d0 0x55f80368dc90 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.837+0000 7f47d8276740 1 --2- 172.21.15.33:0/2718102318 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55f803671bf0 0x55f80368d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.837+0000 7f47b67fc640 0 client.17142 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:00:18.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.837+0000 7f47d8276740 1 --2- 172.21.15.33:0/2718102318 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55f8036712d0 0x55f803690210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.837+0000 7f47d8276740 1 -- 172.21.15.33:0/2718102318 >> 172.21.15.33:0/2718102318 conn(0x55f80366ba00 msgr2=0x55f8036777d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:18.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.837+0000 7f47d8276740 1 -- 172.21.15.33:0/2718102318 shutdown_connections 2024-03-19T03:00:18.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.837+0000 7f47d8276740 1 -- 172.21.15.33:0/2718102318 wait complete. 2024-03-19T03:00:18.846 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.837+0000 7f47d8276740 20 client.17142 trim_cache size 0 max 16384 2024-03-19T03:00:18.847 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T03:00:18.847 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.837+0000 7f47d5685640 1 -- 172.21.15.33:0/282059751 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f47bc0a29f0 msgr2=0x7f47bc0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.837+0000 7f47d5685640 1 --2- 172.21.15.33:0/282059751 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f47bc0a29f0 0x7f47bc0a4eb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f47c80cc840 tx=0x7f47c800ef90 comp rx=0 tx=0).stop 2024-03-19T03:00:18.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.837+0000 7f47d5685640 1 -- 172.21.15.33:0/282059751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f47d0102a40 msgr2=0x7f47d0197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:18.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.837+0000 7f47d5685640 1 --2- 172.21.15.33:0/282059751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f47d0102a40 0x7f47d0197f30 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f47c800ea60 tx=0x7f47c800ef20 comp rx=0 tx=0).stop 2024-03-19T03:00:18.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.837+0000 7f47d5685640 1 -- 172.21.15.33:0/282059751 shutdown_connections 2024-03-19T03:00:18.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.837+0000 7f47d5685640 1 --2- 172.21.15.33:0/282059751 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f47bc0a29f0 0x7f47bc0a4eb0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.838+0000 7f47d5685640 1 --2- 172.21.15.33:0/282059751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47d010bd80 0x7f47d019c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.838+0000 7f47d5685640 1 --2- 172.21.15.33:0/282059751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f47d0103360 0x7f47d0198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.838+0000 7f47d5685640 1 --2- 172.21.15.33:0/282059751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f47d0102a40 0x7f47d0197f30 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:18.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.838+0000 7f47d5685640 1 -- 172.21.15.33:0/282059751 >> 172.21.15.33:0/282059751 conn(0x7f47d00778d0 msgr2=0x7f47d00fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:18.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.838+0000 7f47d5685640 1 -- 172.21.15.33:0/282059751 shutdown_connections 2024-03-19T03:00:18.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:18.838+0000 7f47d5685640 1 -- 172.21.15.33:0/282059751 wait complete. 2024-03-19T03:00:18.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:18 smithi096 ceph-mon[35946]: pgmap v210: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 217 B/s wr, 6 op/s 2024-03-19T03:00:18.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:18 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/430109199' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:00:19.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:18 smithi155 ceph-mon[35804]: pgmap v210: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 217 B/s wr, 6 op/s 2024-03-19T03:00:19.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:18 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/430109199' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:00:19.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:18 smithi033 ceph-mon[31973]: pgmap v210: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 217 B/s wr, 6 op/s 2024-03-19T03:00:19.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:18 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/430109199' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:00:19.850 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T03:00:19.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:00:19.850 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T03:00:19.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.969+0000 7f61904f8640 1 -- 172.21.15.33:0/1992210273 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6188101100 msgr2=0x7f618810d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:19.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.969+0000 7f61904f8640 1 --2- 172.21.15.33:0/1992210273 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6188101100 0x7f618810d610 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f6180009cd0 tx=0x7f618002c0f0 comp rx=0 tx=0).stop 2024-03-19T03:00:19.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.969+0000 7f61904f8640 1 -- 172.21.15.33:0/1992210273 shutdown_connections 2024-03-19T03:00:19.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.969+0000 7f61904f8640 1 --2- 172.21.15.33:0/1992210273 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6188101100 0x7f618810d610 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:19.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.969+0000 7f61904f8640 1 --2- 172.21.15.33:0/1992210273 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6188100740 0x7f6188100bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:19.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.969+0000 7f61904f8640 1 --2- 172.21.15.33:0/1992210273 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6188106750 0x7f6188106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:19.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.969+0000 7f61904f8640 1 -- 172.21.15.33:0/1992210273 >> 172.21.15.33:0/1992210273 conn(0x7f6188077880 msgr2=0x7f6188077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:19.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.969+0000 7f61904f8640 1 -- 172.21.15.33:0/1992210273 shutdown_connections 2024-03-19T03:00:19.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.969+0000 7f61904f8640 1 -- 172.21.15.33:0/1992210273 wait complete. 2024-03-19T03:00:19.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.970+0000 7f61904f8640 1 Processor -- start 2024-03-19T03:00:19.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.970+0000 7f61904f8640 1 -- start start 2024-03-19T03:00:19.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.970+0000 7f61904f8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6188100740 0x7f61881980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:19.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.970+0000 7f61904f8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6188101100 0x7f61881985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:19.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.970+0000 7f61904f8640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6188106750 0x7f618819c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:19.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.970+0000 7f61904f8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6188110a10 con 0x7f6188100740 2024-03-19T03:00:19.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.970+0000 7f61904f8640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6188110890 con 0x7f6188101100 2024-03-19T03:00:19.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.970+0000 7f61904f8640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6188110b90 con 0x7f6188106750 2024-03-19T03:00:19.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.970+0000 7f618e294640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6188100740 0x7f61881980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:19.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.970+0000 7f618e294640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6188100740 0x7f61881980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42484/0 (socket says 172.21.15.33:42484) 2024-03-19T03:00:19.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.970+0000 7f618e294640 1 -- 172.21.15.33:0/912291829 learned_addr learned my addr 172.21.15.33:0/912291829 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:19.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.970+0000 7f618da93640 1 --2- 172.21.15.33:0/912291829 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6188101100 0x7f61881985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:19.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.971+0000 7f618ea95640 1 --2- 172.21.15.33:0/912291829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6188106750 0x7f618819c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:19.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.971+0000 7f618da93640 1 -- 172.21.15.33:0/912291829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6188106750 msgr2=0x7f618819c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:19.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.971+0000 7f618da93640 1 --2- 172.21.15.33:0/912291829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6188106750 0x7f618819c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:19.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.971+0000 7f618da93640 1 -- 172.21.15.33:0/912291829 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6188100740 msgr2=0x7f61881980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:19.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.971+0000 7f618da93640 1 --2- 172.21.15.33:0/912291829 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6188100740 0x7f61881980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:19.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.971+0000 7f618da93640 1 -- 172.21.15.33:0/912291829 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f618819d0a0 con 0x7f6188101100 2024-03-19T03:00:19.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.971+0000 7f618e294640 1 --2- 172.21.15.33:0/912291829 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6188100740 0x7f61881980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:19.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.971+0000 7f618da93640 1 --2- 172.21.15.33:0/912291829 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6188101100 0x7f61881985f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f617800e9f0 tx=0x7f617800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:19.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.971+0000 7f617f7fe640 1 -- 172.21.15.33:0/912291829 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617800ccc0 con 0x7f6188101100 2024-03-19T03:00:19.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.971+0000 7f61904f8640 1 -- 172.21.15.33:0/912291829 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f618819d390 con 0x7f6188101100 2024-03-19T03:00:19.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.971+0000 7f617f7fe640 1 -- 172.21.15.33:0/912291829 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f61780108e0 con 0x7f6188101100 2024-03-19T03:00:19.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.971+0000 7f617f7fe640 1 -- 172.21.15.33:0/912291829 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6178018880 con 0x7f6188101100 2024-03-19T03:00:19.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.971+0000 7f61904f8640 1 -- 172.21.15.33:0/912291829 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61881a4c20 con 0x7f6188101100 2024-03-19T03:00:19.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.973+0000 7f617f7fe640 1 -- 172.21.15.33:0/912291829 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f6178010a90 con 0x7f6188101100 2024-03-19T03:00:19.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.974+0000 7f61931c0740 1 Processor -- start 2024-03-19T03:00:19.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.974+0000 7f61931c0740 1 -- start start 2024-03-19T03:00:19.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.974+0000 7f61931c0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cacd91f380 0x55cacd91f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:19.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.974+0000 7f61931c0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cacd91fca0 0x55cacd928340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:19.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.974+0000 7f61931c0740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cacd928880 0x55cacd92ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:19.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.974+0000 7f61931c0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55cacd883160 con 0x55cacd91f380 2024-03-19T03:00:19.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.974+0000 7f61931c0740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55cacd82be10 con 0x55cacd928880 2024-03-19T03:00:19.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.974+0000 7f61931c0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55cacd87f200 con 0x55cacd91fca0 2024-03-19T03:00:19.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.975+0000 7f617f7fe640 1 --2- 172.21.15.33:0/912291829 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f61740a2970 0x7f61740a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:19.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.975+0000 7f618ea95640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cacd91f380 0x55cacd91f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:19.982 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.975+0000 7f618ea95640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cacd91f380 0x55cacd91f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42488/0 (socket says 172.21.15.33:42488) 2024-03-19T03:00:19.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.975+0000 7f618ea95640 1 -- 172.21.15.33:0/1277920750 learned_addr learned my addr 172.21.15.33:0/1277920750 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:19.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.975+0000 7f618da93640 1 --2- 172.21.15.33:0/1277920750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cacd928880 0x55cacd92ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:19.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.975+0000 7f618e294640 1 --2- 172.21.15.33:0/1277920750 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cacd91fca0 0x55cacd928340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:19.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.975+0000 7f618ea95640 1 --2- 172.21.15.33:0/912291829 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f61740a2970 0x7f61740a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:19.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.975+0000 7f618e294640 1 -- 172.21.15.33:0/1277920750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cacd928880 msgr2=0x55cacd92ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:19.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.975+0000 7f617f7fe640 1 -- 172.21.15.33:0/912291829 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7f61780cf600 con 0x7f6188101100 2024-03-19T03:00:19.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.975+0000 7f618e294640 1 --2- 172.21.15.33:0/1277920750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cacd928880 0x55cacd92ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:19.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.975+0000 7f618e294640 1 -- 172.21.15.33:0/1277920750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cacd91f380 msgr2=0x55cacd91f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:19.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.975+0000 7f618e294640 1 --2- 172.21.15.33:0/1277920750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cacd91f380 0x55cacd91f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:19.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.975+0000 7f618e294640 1 -- 172.21.15.33:0/1277920750 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cacd65f7e0 con 0x55cacd91fca0 2024-03-19T03:00:19.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.975+0000 7f618ea95640 1 --2- 172.21.15.33:0/1277920750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cacd91f380 0x55cacd91f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:19.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.975+0000 7f618e294640 1 --2- 172.21.15.33:0/1277920750 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cacd91fca0 0x55cacd928340 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f61840114f0 tx=0x7f61840119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c94985f6cf1b59 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:19.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.975+0000 7f618ea95640 1 --2- 172.21.15.33:0/912291829 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f61740a2970 0x7f61740a4e30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6180009a60 tx=0x7f618002c0f0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:19.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.975+0000 7f617cbfe640 1 -- 172.21.15.33:0/1277920750 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f618401e070 con 0x55cacd91fca0 2024-03-19T03:00:19.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.975+0000 7f617cbfe640 1 -- 172.21.15.33:0/1277920750 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f618400fb30 con 0x55cacd91fca0 2024-03-19T03:00:19.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.975+0000 7f617cbfe640 1 -- 172.21.15.33:0/1277920750 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6184019c10 con 0x55cacd91fca0 2024-03-19T03:00:19.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.976+0000 7f61931c0740 1 -- 172.21.15.33:0/1277920750 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cacd91fca0 msgr2=0x55cacd928340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:19.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.976+0000 7f61931c0740 1 --2- 172.21.15.33:0/1277920750 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cacd91fca0 0x55cacd928340 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f61840114f0 tx=0x7f61840119b0 comp rx=0 tx=0).stop 2024-03-19T03:00:19.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.976+0000 7f61931c0740 1 -- 172.21.15.33:0/1277920750 shutdown_connections 2024-03-19T03:00:19.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.976+0000 7f61931c0740 1 --2- 172.21.15.33:0/1277920750 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cacd91fca0 0x55cacd928340 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:19.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.976+0000 7f61931c0740 1 --2- 172.21.15.33:0/1277920750 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cacd928880 0x55cacd92ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:19.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.976+0000 7f61931c0740 1 --2- 172.21.15.33:0/1277920750 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cacd91f380 0x55cacd91f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:19.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.976+0000 7f61931c0740 1 -- 172.21.15.33:0/1277920750 >> 172.21.15.33:0/1277920750 conn(0x55cacd919ab0 msgr2=0x55cacd919e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:19.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.976+0000 7f61931c0740 1 -- 172.21.15.33:0/1277920750 shutdown_connections 2024-03-19T03:00:19.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.976+0000 7f61931c0740 1 -- 172.21.15.33:0/1277920750 wait complete. 2024-03-19T03:00:19.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.977+0000 7f61931c0740 1 -- start start 2024-03-19T03:00:19.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.977+0000 7f61931c0740 1 Processor -- start 2024-03-19T03:00:19.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.977+0000 7f61931c0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cacd91f380 0x55cacd9257b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:19.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.977+0000 7f61931c0740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cacd91fca0 0x55cacd925cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:19.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.977+0000 7f61931c0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cacd928880 0x55cacd926230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:19.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.977+0000 7f61931c0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55cacd92b670 con 0x55cacd928880 2024-03-19T03:00:19.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.977+0000 7f61931c0740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55cacd83f1a0 con 0x55cacd91fca0 2024-03-19T03:00:19.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.977+0000 7f61931c0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55cacd92b7f0 con 0x55cacd91f380 2024-03-19T03:00:19.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.977+0000 7f61931c0740 10 client.0 _ll_register_callbacks cb 0x55cacd596c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:00:19.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.977+0000 7f618ea95640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cacd928880 0x55cacd926230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:19.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.977+0000 7f618ea95640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cacd928880 0x55cacd926230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42490/0 (socket says 172.21.15.33:42490) 2024-03-19T03:00:19.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.977+0000 7f618ea95640 1 -- 172.21.15.33:0/1745639477 learned_addr learned my addr 172.21.15.33:0/1745639477 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:19.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.977+0000 7f618e294640 1 --2- 172.21.15.33:0/1745639477 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cacd91f380 0x55cacd9257b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:19.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.977+0000 7f618da93640 1 --2- 172.21.15.33:0/1745639477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cacd91fca0 0x55cacd925cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:19.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.977+0000 7f618ea95640 1 -- 172.21.15.33:0/1745639477 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cacd91f380 msgr2=0x55cacd9257b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:19.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.977+0000 7f618ea95640 1 --2- 172.21.15.33:0/1745639477 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cacd91f380 0x55cacd9257b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:19.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.977+0000 7f618ea95640 1 -- 172.21.15.33:0/1745639477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cacd91fca0 msgr2=0x55cacd925cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:19.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.977+0000 7f618ea95640 1 --2- 172.21.15.33:0/1745639477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cacd91fca0 0x55cacd925cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:19.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.977+0000 7f618ea95640 1 -- 172.21.15.33:0/1745639477 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cacd6bff70 con 0x55cacd928880 2024-03-19T03:00:19.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.977+0000 7f618da93640 1 --2- 172.21.15.33:0/1745639477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cacd91fca0 0x55cacd925cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:19.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.978+0000 7f618ea95640 1 --2- 172.21.15.33:0/1745639477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cacd928880 0x55cacd926230 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f6180005cc0 tx=0x7f61800095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4649d6689a10c267 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:19.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.978+0000 7f6171ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:00:19.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.978+0000 7f6171ffb640 1 -- 172.21.15.33:0/1745639477 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6180052060 con 0x55cacd928880 2024-03-19T03:00:19.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.978+0000 7f6171ffb640 1 -- 172.21.15.33:0/1745639477 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7f6180004450 con 0x55cacd928880 2024-03-19T03:00:19.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.978+0000 7f61931c0740 1 -- 172.21.15.33:0/1745639477 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f618400fb30 con 0x55cacd928880 2024-03-19T03:00:19.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.978+0000 7f6171ffb640 1 -- 172.21.15.33:0/1745639477 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f6180059050 con 0x55cacd928880 2024-03-19T03:00:19.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.978+0000 7f61931c0740 10 client.17154 fetch_fsmap learned FSMap version 25 2024-03-19T03:00:19.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.978+0000 7f61931c0740 1 -- 172.21.15.33:0/1745639477 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55cacd557170 con 0x55cacd928880 2024-03-19T03:00:19.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.978+0000 7f6171ffb640 1 -- 172.21.15.33:0/1745639477 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f618400fb30 con 0x55cacd928880 2024-03-19T03:00:19.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.978+0000 7f6171ffb640 1 -- 172.21.15.33:0/1745639477 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f618003f070 con 0x55cacd928880 2024-03-19T03:00:19.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.978+0000 7f61931c0740 10 client.17154 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:00:19.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.978+0000 7f61931c0740 10 client.17154 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:00:19.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.978+0000 7f61931c0740 20 client.17154 populate_metadata read hostname 'smithi033' 2024-03-19T03:00:19.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.978+0000 7f61931c0740 1 --2- 172.21.15.33:0/1745639477 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55cacd93cf70 0x55cacd93f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:19.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.978+0000 7f61931c0740 4 client.17154 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:00:19.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.978+0000 7f61931c0740 1 -- 172.21.15.33:0/1745639477 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cacd8906a0 con 0x55cacd93cf70 2024-03-19T03:00:19.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.979+0000 7f618e294640 1 --2- 172.21.15.33:0/1745639477 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55cacd93cf70 0x55cacd93f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:19.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.979+0000 7f6171ffb640 10 client.17154 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:00:19.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.979+0000 7f618e294640 1 --2- 172.21.15.33:0/1745639477 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55cacd93cf70 0x55cacd93f3c0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6815f2d551c41fd server_cookie=34e4cfcf607bfa09 in_seq=0 out_seq=0 2024-03-19T03:00:19.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.980+0000 7f6171ffb640 1 -- 172.21.15.33:0/1745639477 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cacd8906a0 con 0x55cacd93cf70 2024-03-19T03:00:19.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.980+0000 7f6171ffb640 10 client.17154 handle_command_reply: tid=0 2024-03-19T03:00:19.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.980+0000 7f61931c0740 1 client.17154 shutdown 2024-03-19T03:00:19.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.980+0000 7f61931c0740 1 -- 172.21.15.33:0/1745639477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cacd928880 msgr2=0x55cacd926230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:19.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.980+0000 7f61931c0740 1 --2- 172.21.15.33:0/1745639477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cacd928880 0x55cacd926230 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f6180005cc0 tx=0x7f61800095c0 comp rx=0 tx=0).stop 2024-03-19T03:00:19.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.980+0000 7f61931c0740 1 -- 172.21.15.33:0/1745639477 shutdown_connections 2024-03-19T03:00:19.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.980+0000 7f61931c0740 1 --2- 172.21.15.33:0/1745639477 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55cacd93cf70 0x55cacd93f3c0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:19.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.980+0000 7f61931c0740 1 --2- 172.21.15.33:0/1745639477 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cacd928880 0x55cacd926230 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:19.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.980+0000 7f61931c0740 1 --2- 172.21.15.33:0/1745639477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cacd91fca0 0x55cacd925cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:19.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.980+0000 7f6171ffb640 0 client.17154 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:00:19.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.980+0000 7f61931c0740 1 --2- 172.21.15.33:0/1745639477 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cacd91f380 0x55cacd9257b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:19.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.980+0000 7f61931c0740 1 -- 172.21.15.33:0/1745639477 >> 172.21.15.33:0/1745639477 conn(0x55cacd919ab0 msgr2=0x55cacd91b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:19.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.980+0000 7f61931c0740 1 -- 172.21.15.33:0/1745639477 shutdown_connections 2024-03-19T03:00:19.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.980+0000 7f61931c0740 1 -- 172.21.15.33:0/1745639477 wait complete. 2024-03-19T03:00:19.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.980+0000 7f61931c0740 20 client.17154 trim_cache size 0 max 16384 2024-03-19T03:00:20.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.997+0000 7f61931c0740 1 Processor -- start 2024-03-19T03:00:20.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.998+0000 7f61931c0740 1 -- start start 2024-03-19T03:00:20.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.998+0000 7f61931c0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cacd91f380 0x55cacd927ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:20.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.998+0000 7f61931c0740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cacd91fca0 0x55cacd938900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:20.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.998+0000 7f61931c0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cacd928880 0x55cacd944c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:20.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.998+0000 7f61931c0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f61881af380 con 0x55cacd91f380 2024-03-19T03:00:20.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.998+0000 7f61931c0740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f61881acc50 con 0x55cacd91fca0 2024-03-19T03:00:20.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.998+0000 7f61931c0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f61881aaac0 con 0x55cacd928880 2024-03-19T03:00:20.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.998+0000 7f618e294640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cacd91f380 0x55cacd927ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:20.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.998+0000 7f618e294640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cacd91f380 0x55cacd927ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42506/0 (socket says 172.21.15.33:42506) 2024-03-19T03:00:20.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.998+0000 7f618e294640 1 -- 172.21.15.33:0/2320597038 learned_addr learned my addr 172.21.15.33:0/2320597038 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:20.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.998+0000 7f618da93640 1 --2- 172.21.15.33:0/2320597038 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cacd91fca0 0x55cacd938900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:20.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.998+0000 7f618ea95640 1 --2- 172.21.15.33:0/2320597038 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cacd928880 0x55cacd944c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:20.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.998+0000 7f618e294640 1 -- 172.21.15.33:0/2320597038 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cacd928880 msgr2=0x55cacd944c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:20.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.998+0000 7f618e294640 1 --2- 172.21.15.33:0/2320597038 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cacd928880 0x55cacd944c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.998+0000 7f618e294640 1 -- 172.21.15.33:0/2320597038 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cacd91fca0 msgr2=0x55cacd938900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:20.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.998+0000 7f618e294640 1 --2- 172.21.15.33:0/2320597038 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cacd91fca0 0x55cacd938900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.004 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.998+0000 7f618e294640 1 -- 172.21.15.33:0/2320597038 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cacd9198e0 con 0x55cacd91f380 2024-03-19T03:00:20.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.998+0000 7f618da93640 1 --2- 172.21.15.33:0/2320597038 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cacd91fca0 0x55cacd938900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:20.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.998+0000 7f618e294640 1 --2- 172.21.15.33:0/2320597038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cacd91f380 0x55cacd927ee0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f6184019800 tx=0x7f6184009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbbd4fbe248a5655 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:20.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.998+0000 7f6173fff640 1 -- 172.21.15.33:0/2320597038 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f618401e040 con 0x55cacd91f380 2024-03-19T03:00:20.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.999+0000 7f6173fff640 1 -- 172.21.15.33:0/2320597038 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f618402b740 con 0x55cacd91f380 2024-03-19T03:00:20.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.999+0000 7f6173fff640 1 -- 172.21.15.33:0/2320597038 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6184019a90 con 0x55cacd91f380 2024-03-19T03:00:20.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.999+0000 7f61931c0740 1 -- 172.21.15.33:0/2320597038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cacd91f380 msgr2=0x55cacd927ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:20.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.999+0000 7f61931c0740 1 --2- 172.21.15.33:0/2320597038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cacd91f380 0x55cacd927ee0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f6184019800 tx=0x7f6184009f60 comp rx=0 tx=0).stop 2024-03-19T03:00:20.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.999+0000 7f61931c0740 1 -- 172.21.15.33:0/2320597038 shutdown_connections 2024-03-19T03:00:20.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.999+0000 7f61931c0740 1 --2- 172.21.15.33:0/2320597038 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cacd928880 0x55cacd944c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.999+0000 7f61931c0740 1 --2- 172.21.15.33:0/2320597038 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cacd91fca0 0x55cacd938900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.999+0000 7f61931c0740 1 --2- 172.21.15.33:0/2320597038 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cacd91f380 0x55cacd927ee0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.999+0000 7f61931c0740 1 -- 172.21.15.33:0/2320597038 >> 172.21.15.33:0/2320597038 conn(0x55cacd919ab0 msgr2=0x55cacd91b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:20.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.999+0000 7f61931c0740 1 -- 172.21.15.33:0/2320597038 shutdown_connections 2024-03-19T03:00:20.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.999+0000 7f61931c0740 1 -- 172.21.15.33:0/2320597038 wait complete. 2024-03-19T03:00:20.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:19.999+0000 7f61931c0740 1 -- start start 2024-03-19T03:00:20.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.000+0000 7f61931c0740 1 Processor -- start 2024-03-19T03:00:20.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.000+0000 7f61931c0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cacd91f380 0x55cacd922b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:20.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.000+0000 7f61931c0740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cacd91fca0 0x55cacd9230a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:20.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.000+0000 7f61931c0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cacd928880 0x55cacd9a22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:20.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.000+0000 7f61931c0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f61881ab110 con 0x55cacd91f380 2024-03-19T03:00:20.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.000+0000 7f61931c0740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f61881ac580 con 0x55cacd91fca0 2024-03-19T03:00:20.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.000+0000 7f61931c0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f61881ab7e0 con 0x55cacd928880 2024-03-19T03:00:20.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.000+0000 7f61931c0740 10 client.0 _ll_register_callbacks cb 0x7f6184005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:00:20.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.000+0000 7f618e294640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cacd91f380 0x55cacd922b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:20.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.000+0000 7f618e294640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cacd91f380 0x55cacd922b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42512/0 (socket says 172.21.15.33:42512) 2024-03-19T03:00:20.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.000+0000 7f618e294640 1 -- 172.21.15.33:0/2729819420 learned_addr learned my addr 172.21.15.33:0/2729819420 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:20.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.000+0000 7f618ea95640 1 --2- 172.21.15.33:0/2729819420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cacd928880 0x55cacd9a22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:20.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.000+0000 7f618da93640 1 --2- 172.21.15.33:0/2729819420 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cacd91fca0 0x55cacd9230a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:20.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.000+0000 7f618e294640 1 -- 172.21.15.33:0/2729819420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cacd928880 msgr2=0x55cacd9a22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:20.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.000+0000 7f618e294640 1 --2- 172.21.15.33:0/2729819420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cacd928880 0x55cacd9a22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.000+0000 7f618e294640 1 -- 172.21.15.33:0/2729819420 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cacd91fca0 msgr2=0x55cacd9230a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:20.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.000+0000 7f618e294640 1 --2- 172.21.15.33:0/2729819420 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cacd91fca0 0x55cacd9230a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.000+0000 7f618e294640 1 -- 172.21.15.33:0/2729819420 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cacd9a2850 con 0x55cacd91f380 2024-03-19T03:00:20.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.000+0000 7f618e294640 1 --2- 172.21.15.33:0/2729819420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cacd91f380 0x55cacd922b60 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f6184013040 tx=0x7f61840095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d129a686c6581c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:20.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.000+0000 7f6171ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:00:20.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.000+0000 7f6171ffb640 1 -- 172.21.15.33:0/2729819420 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f618401e040 con 0x55cacd91f380 2024-03-19T03:00:20.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.001+0000 7f6171ffb640 1 -- 172.21.15.33:0/2729819420 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7f61840055b0 con 0x55cacd91f380 2024-03-19T03:00:20.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.001+0000 7f61931c0740 1 -- 172.21.15.33:0/2729819420 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f618402b740 con 0x55cacd91f380 2024-03-19T03:00:20.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.001+0000 7f6171ffb640 1 -- 172.21.15.33:0/2729819420 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f6184043050 con 0x55cacd91f380 2024-03-19T03:00:20.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.001+0000 7f61931c0740 10 client.17160 fetch_fsmap learned FSMap version 25 2024-03-19T03:00:20.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.001+0000 7f61931c0740 1 -- 172.21.15.33:0/2729819420 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55cacd9a3b90 con 0x55cacd91f380 2024-03-19T03:00:20.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.001+0000 7f6171ffb640 1 -- 172.21.15.33:0/2729819420 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f618402b740 con 0x55cacd91f380 2024-03-19T03:00:20.007 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.001+0000 7f6171ffb640 1 -- 172.21.15.33:0/2729819420 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f618400f680 con 0x55cacd91f380 2024-03-19T03:00:20.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.001+0000 7f61931c0740 10 client.17160 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:00:20.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.001+0000 7f61931c0740 10 client.17160 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:00:20.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.001+0000 7f61931c0740 20 client.17160 populate_metadata read hostname 'smithi033' 2024-03-19T03:00:20.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.001+0000 7f61931c0740 1 --2- 172.21.15.33:0/2729819420 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55cacd9a40f0 0x55cacd9a6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:20.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.001+0000 7f61931c0740 4 client.17160 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T03:00:20.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.001+0000 7f61931c0740 1 -- 172.21.15.33:0/2729819420 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55cacd88a9f0 con 0x55cacd9a40f0 2024-03-19T03:00:20.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.001+0000 7f618da93640 1 --2- 172.21.15.33:0/2729819420 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55cacd9a40f0 0x55cacd9a6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:20.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.002+0000 7f6171ffb640 10 client.17160 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:00:20.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.002+0000 7f618da93640 1 --2- 172.21.15.33:0/2729819420 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55cacd9a40f0 0x55cacd9a6510 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9e1f465db64a9f0 server_cookie=f797401ff78cb95a in_seq=0 out_seq=0 2024-03-19T03:00:20.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.002+0000 7f6171ffb640 1 -- 172.21.15.33:0/2729819420 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55cacd88a9f0 con 0x55cacd9a40f0 2024-03-19T03:00:20.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.002+0000 7f6171ffb640 10 client.17160 handle_command_reply: tid=0 2024-03-19T03:00:20.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.002+0000 7f61931c0740 1 client.17160 shutdown 2024-03-19T03:00:20.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.003+0000 7f61931c0740 1 -- 172.21.15.33:0/2729819420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cacd91f380 msgr2=0x55cacd922b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:20.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.003+0000 7f61931c0740 1 --2- 172.21.15.33:0/2729819420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cacd91f380 0x55cacd922b60 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f6184013040 tx=0x7f61840095c0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.003+0000 7f61931c0740 1 -- 172.21.15.33:0/2729819420 shutdown_connections 2024-03-19T03:00:20.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.003+0000 7f61931c0740 1 --2- 172.21.15.33:0/2729819420 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55cacd9a40f0 0x55cacd9a6510 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.003+0000 7f61931c0740 1 --2- 172.21.15.33:0/2729819420 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cacd928880 0x55cacd9a22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.003+0000 7f61931c0740 1 --2- 172.21.15.33:0/2729819420 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cacd91fca0 0x55cacd9230a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.003+0000 7f61931c0740 1 --2- 172.21.15.33:0/2729819420 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cacd91f380 0x55cacd922b60 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.003+0000 7f61931c0740 1 -- 172.21.15.33:0/2729819420 >> 172.21.15.33:0/2729819420 conn(0x55cacd919ab0 msgr2=0x55cacd925770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:20.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.003+0000 7f6171ffb640 0 client.17160 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:00:20.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.003+0000 7f61931c0740 1 -- 172.21.15.33:0/2729819420 shutdown_connections 2024-03-19T03:00:20.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.003+0000 7f61931c0740 1 -- 172.21.15.33:0/2729819420 wait complete. 2024-03-19T03:00:20.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.003+0000 7f61931c0740 20 client.17160 trim_cache size 0 max 16384 2024-03-19T03:00:20.010 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T03:00:20.011 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-03-19T03:00:20.011 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "9c2358da-c0bb-4313-8c1c-f720a7f64250", 2024-03-19T03:00:20.011 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-03-19T03:00:20.011 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T03:00:20.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.003+0000 7f61904f8640 1 -- 172.21.15.33:0/912291829 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f61740a2970 msgr2=0x7f61740a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:20.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.003+0000 7f61904f8640 1 --2- 172.21.15.33:0/912291829 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f61740a2970 0x7f61740a4e30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6180009a60 tx=0x7f618002c0f0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.003+0000 7f61904f8640 1 -- 172.21.15.33:0/912291829 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6188101100 msgr2=0x7f61881985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:20.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.003+0000 7f61904f8640 1 --2- 172.21.15.33:0/912291829 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6188101100 0x7f61881985f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f617800e9f0 tx=0x7f617800eeb0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.003+0000 7f61904f8640 1 -- 172.21.15.33:0/912291829 shutdown_connections 2024-03-19T03:00:20.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.003+0000 7f61904f8640 1 --2- 172.21.15.33:0/912291829 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f61740a2970 0x7f61740a4e30 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.003+0000 7f61904f8640 1 --2- 172.21.15.33:0/912291829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6188106750 0x7f618819c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.003+0000 7f61904f8640 1 --2- 172.21.15.33:0/912291829 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6188101100 0x7f61881985f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.003+0000 7f61904f8640 1 --2- 172.21.15.33:0/912291829 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6188100740 0x7f61881980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.003+0000 7f61904f8640 1 -- 172.21.15.33:0/912291829 >> 172.21.15.33:0/912291829 conn(0x7f6188077880 msgr2=0x7f618806a230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:20.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.004+0000 7f61904f8640 1 -- 172.21.15.33:0/912291829 shutdown_connections 2024-03-19T03:00:20.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.004+0000 7f61904f8640 1 -- 172.21.15.33:0/912291829 wait complete. 2024-03-19T03:00:20.015 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T03:00:20.016 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T03:00:20.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.119+0000 7f8d3d777640 1 -- 172.21.15.33:0/637223335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d381077c0 msgr2=0x7f8d38107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:20.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.119+0000 7f8d3d777640 1 --2- 172.21.15.33:0/637223335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d381077c0 0x7f8d38107ba0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f8d30009900 tx=0x7f8d3002bc30 comp rx=0 tx=0).stop 2024-03-19T03:00:20.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.119+0000 7f8d3d777640 1 -- 172.21.15.33:0/637223335 shutdown_connections 2024-03-19T03:00:20.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.119+0000 7f8d3d777640 1 --2- 172.21.15.33:0/637223335 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d380ffcc0 0x7f8d38100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.119+0000 7f8d3d777640 1 --2- 172.21.15.33:0/637223335 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d38100680 0x7f8d3810e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.119+0000 7f8d3d777640 1 --2- 172.21.15.33:0/637223335 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d381077c0 0x7f8d38107ba0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.119+0000 7f8d3d777640 1 -- 172.21.15.33:0/637223335 >> 172.21.15.33:0/637223335 conn(0x7f8d38076c40 msgr2=0x7f8d380fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:20.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.119+0000 7f8d3d777640 1 -- 172.21.15.33:0/637223335 shutdown_connections 2024-03-19T03:00:20.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.119+0000 7f8d3d777640 1 -- 172.21.15.33:0/637223335 wait complete. 2024-03-19T03:00:20.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.120+0000 7f8d3d777640 1 Processor -- start 2024-03-19T03:00:20.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.120+0000 7f8d3d777640 1 -- start start 2024-03-19T03:00:20.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.120+0000 7f8d3d777640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d380ffcc0 0x7f8d38198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:20.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.120+0000 7f8d3d777640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d38100680 0x7f8d38198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:20.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.120+0000 7f8d3d777640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d381077c0 0x7f8d3819c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:20.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.120+0000 7f8d3d777640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8d38111a20 con 0x7f8d38100680 2024-03-19T03:00:20.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.120+0000 7f8d3d777640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8d381118a0 con 0x7f8d381077c0 2024-03-19T03:00:20.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.120+0000 7f8d3d777640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8d38111ba0 con 0x7f8d380ffcc0 2024-03-19T03:00:20.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.120+0000 7f8d367fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d38100680 0x7f8d38198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:20.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.120+0000 7f8d367fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d38100680 0x7f8d38198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42532/0 (socket says 172.21.15.33:42532) 2024-03-19T03:00:20.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.120+0000 7f8d367fc640 1 -- 172.21.15.33:0/1186314407 learned_addr learned my addr 172.21.15.33:0/1186314407 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:20.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.121+0000 7f8d377fe640 1 --2- 172.21.15.33:0/1186314407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d381077c0 0x7f8d3819c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:20.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.121+0000 7f8d36ffd640 1 --2- 172.21.15.33:0/1186314407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d380ffcc0 0x7f8d38198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:20.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.121+0000 7f8d367fc640 1 -- 172.21.15.33:0/1186314407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d380ffcc0 msgr2=0x7f8d38198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:20.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.121+0000 7f8d367fc640 1 --2- 172.21.15.33:0/1186314407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d380ffcc0 0x7f8d38198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.121+0000 7f8d367fc640 1 -- 172.21.15.33:0/1186314407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d381077c0 msgr2=0x7f8d3819c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:20.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.121+0000 7f8d367fc640 1 --2- 172.21.15.33:0/1186314407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d381077c0 0x7f8d3819c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.121+0000 7f8d367fc640 1 -- 172.21.15.33:0/1186314407 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d3819cff0 con 0x7f8d38100680 2024-03-19T03:00:20.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.121+0000 7f8d377fe640 1 --2- 172.21.15.33:0/1186314407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d381077c0 0x7f8d3819c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:20.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.121+0000 7f8d367fc640 1 --2- 172.21.15.33:0/1186314407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d38100680 0x7f8d38198540 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f8d2000ea60 tx=0x7f8d2000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:20.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.121+0000 7f8d2ffff640 1 -- 172.21.15.33:0/1186314407 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d2000cde0 con 0x7f8d38100680 2024-03-19T03:00:20.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.121+0000 7f8d3d777640 1 -- 172.21.15.33:0/1186314407 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d3819d2e0 con 0x7f8d38100680 2024-03-19T03:00:20.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.121+0000 7f8d2ffff640 1 -- 172.21.15.33:0/1186314407 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8d200035f0 con 0x7f8d38100680 2024-03-19T03:00:20.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.121+0000 7f8d2ffff640 1 -- 172.21.15.33:0/1186314407 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d20010690 con 0x7f8d38100680 2024-03-19T03:00:20.125 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.121+0000 7f8d3d777640 1 -- 172.21.15.33:0/1186314407 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d381a4b70 con 0x7f8d38100680 2024-03-19T03:00:20.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.123+0000 7f8d2ffff640 1 -- 172.21.15.33:0/1186314407 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f8d200037a0 con 0x7f8d38100680 2024-03-19T03:00:20.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.124+0000 7f8d3fffd740 1 Processor -- start 2024-03-19T03:00:20.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.124+0000 7f8d3fffd740 1 -- start start 2024-03-19T03:00:20.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.124+0000 7f8d2ffff640 1 --2- 172.21.15.33:0/1186314407 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f8d240a2970 0x7f8d240a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:20.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.124+0000 7f8d377fe640 1 --2- 172.21.15.33:0/1186314407 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f8d240a2970 0x7f8d240a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:20.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.124+0000 7f8d2ffff640 1 -- 172.21.15.33:0/1186314407 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7f8d200cf4b0 con 0x7f8d38100680 2024-03-19T03:00:20.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.124+0000 7f8d3fffd740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e01fe7fa30 0x55e01fe7fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:20.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.124+0000 7f8d3fffd740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e01fe80350 0x55e01fe889f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:20.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.124+0000 7f8d3fffd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e01fe88f30 0x55e01fe8b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:20.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.124+0000 7f8d3fffd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e01fde3500 con 0x55e01fe88f30 2024-03-19T03:00:20.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.124+0000 7f8d3fffd740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55e01fd8bbe0 con 0x55e01fe7fa30 2024-03-19T03:00:20.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.124+0000 7f8d3fffd740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e01fddf5a0 con 0x55e01fe80350 2024-03-19T03:00:20.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.124+0000 7f8d367fc640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e01fe80350 0x55e01fe889f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:20.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.124+0000 7f8d36ffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e01fe7fa30 0x55e01fe7fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:20.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.124+0000 7f8d377fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e01fe88f30 0x55e01fe8b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:20.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.124+0000 7f8d367fc640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e01fe80350 0x55e01fe889f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:47208/0 (socket says 172.21.15.33:47208) 2024-03-19T03:00:20.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.124+0000 7f8d367fc640 1 -- 172.21.15.33:0/1476659111 learned_addr learned my addr 172.21.15.33:0/1476659111 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:20.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.125+0000 7f8d36ffd640 1 -- 172.21.15.33:0/1476659111 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e01fe80350 msgr2=0x55e01fe889f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:20.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.125+0000 7f8d36ffd640 1 --2- 172.21.15.33:0/1476659111 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e01fe80350 0x55e01fe889f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.125+0000 7f8d36ffd640 1 -- 172.21.15.33:0/1476659111 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e01fe88f30 msgr2=0x55e01fe8b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:20.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.125+0000 7f8d36ffd640 1 --2- 172.21.15.33:0/1476659111 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e01fe88f30 0x55e01fe8b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.125+0000 7f8d36ffd640 1 -- 172.21.15.33:0/1476659111 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e01fdad000 con 0x55e01fe7fa30 2024-03-19T03:00:20.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.125+0000 7f8d367fc640 1 --2- 172.21.15.33:0/1476659111 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e01fe80350 0x55e01fe889f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:20.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.125+0000 7f8d377fe640 1 --2- 172.21.15.33:0/1186314407 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f8d240a2970 0x7f8d240a4e30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8d28005fd0 tx=0x7f8d28013e50 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:20.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.125+0000 7f8d36ffd640 1 --2- 172.21.15.33:0/1476659111 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e01fe7fa30 0x55e01fe7fe10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8d30000c00 tx=0x7f8d300095f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e50df5145a75d0de server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:20.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.125+0000 7f8d2d7fa640 1 -- 172.21.15.33:0/1476659111 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d30045020 con 0x55e01fe7fa30 2024-03-19T03:00:20.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.125+0000 7f8d2d7fa640 1 -- 172.21.15.33:0/1476659111 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8d3002c7e0 con 0x55e01fe7fa30 2024-03-19T03:00:20.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.125+0000 7f8d2d7fa640 1 -- 172.21.15.33:0/1476659111 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d300054e0 con 0x55e01fe7fa30 2024-03-19T03:00:20.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.126+0000 7f8d3fffd740 1 -- 172.21.15.33:0/1476659111 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e01fe7fa30 msgr2=0x55e01fe7fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:20.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.126+0000 7f8d3fffd740 1 --2- 172.21.15.33:0/1476659111 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e01fe7fa30 0x55e01fe7fe10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8d30000c00 tx=0x7f8d300095f0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.126+0000 7f8d3fffd740 1 -- 172.21.15.33:0/1476659111 shutdown_connections 2024-03-19T03:00:20.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.126+0000 7f8d3fffd740 1 --2- 172.21.15.33:0/1476659111 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e01fe80350 0x55e01fe889f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.126+0000 7f8d3fffd740 1 --2- 172.21.15.33:0/1476659111 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e01fe88f30 0x55e01fe8b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.126+0000 7f8d3fffd740 1 --2- 172.21.15.33:0/1476659111 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e01fe7fa30 0x55e01fe7fe10 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.126+0000 7f8d3fffd740 1 -- 172.21.15.33:0/1476659111 >> 172.21.15.33:0/1476659111 conn(0x55e01fe7a160 msgr2=0x55e01fe7a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:20.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.126+0000 7f8d3fffd740 1 -- 172.21.15.33:0/1476659111 shutdown_connections 2024-03-19T03:00:20.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.126+0000 7f8d3fffd740 1 -- 172.21.15.33:0/1476659111 wait complete. 2024-03-19T03:00:20.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.126+0000 7f8d3fffd740 1 -- start start 2024-03-19T03:00:20.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.126+0000 7f8d3fffd740 1 Processor -- start 2024-03-19T03:00:20.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.126+0000 7f8d3fffd740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e01fe7fa30 0x55e01fe85e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:20.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.126+0000 7f8d3fffd740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e01fe80350 0x55e01fe86390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:20.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.126+0000 7f8d3fffd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e01fe88f30 0x55e01fe868d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:20.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.126+0000 7f8d3fffd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e01fe8bcc0 con 0x55e01fe88f30 2024-03-19T03:00:20.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.126+0000 7f8d3fffd740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55e01fd9f160 con 0x55e01fe80350 2024-03-19T03:00:20.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.126+0000 7f8d3fffd740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e01fe8be40 con 0x55e01fe7fa30 2024-03-19T03:00:20.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.127+0000 7f8d3fffd740 10 client.0 _ll_register_callbacks cb 0x55e01fddba40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:00:20.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.127+0000 7f8d377fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e01fe88f30 0x55e01fe868d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:20.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.127+0000 7f8d36ffd640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e01fe7fa30 0x55e01fe85e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:20.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.127+0000 7f8d377fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e01fe88f30 0x55e01fe868d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42542/0 (socket says 172.21.15.33:42542) 2024-03-19T03:00:20.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.127+0000 7f8d377fe640 1 -- 172.21.15.33:0/2083867283 learned_addr learned my addr 172.21.15.33:0/2083867283 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:20.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.127+0000 7f8d367fc640 1 --2- 172.21.15.33:0/2083867283 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e01fe80350 0x55e01fe86390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:20.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.127+0000 7f8d377fe640 1 -- 172.21.15.33:0/2083867283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e01fe7fa30 msgr2=0x55e01fe85e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:20.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.127+0000 7f8d377fe640 1 --2- 172.21.15.33:0/2083867283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e01fe7fa30 0x55e01fe85e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.127+0000 7f8d377fe640 1 -- 172.21.15.33:0/2083867283 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e01fe80350 msgr2=0x55e01fe86390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:20.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.127+0000 7f8d377fe640 1 --2- 172.21.15.33:0/2083867283 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e01fe80350 0x55e01fe86390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.127+0000 7f8d377fe640 1 -- 172.21.15.33:0/2083867283 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e01fbc17d0 con 0x55e01fe88f30 2024-03-19T03:00:20.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.127+0000 7f8d36ffd640 1 --2- 172.21.15.33:0/2083867283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e01fe7fa30 0x55e01fe85e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:20.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.127+0000 7f8d367fc640 1 --2- 172.21.15.33:0/2083867283 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e01fe80350 0x55e01fe86390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:20.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.127+0000 7f8d377fe640 1 --2- 172.21.15.33:0/2083867283 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e01fe88f30 0x55e01fe868d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f8d28013860 tx=0x7f8d28004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c763ec38c4c68ed8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:20.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.127+0000 7f8d1e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:00:20.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.127+0000 7f8d1e7fc640 1 -- 172.21.15.33:0/2083867283 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d28010630 con 0x55e01fe88f30 2024-03-19T03:00:20.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.127+0000 7f8d1e7fc640 1 -- 172.21.15.33:0/2083867283 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7f8d280107d0 con 0x55e01fe88f30 2024-03-19T03:00:20.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.127+0000 7f8d3fffd740 1 -- 172.21.15.33:0/2083867283 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d3002c7e0 con 0x55e01fe88f30 2024-03-19T03:00:20.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.127+0000 7f8d1e7fc640 1 -- 172.21.15.33:0/2083867283 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8d280109d0 con 0x55e01fe88f30 2024-03-19T03:00:20.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.127+0000 7f8d3fffd740 10 client.17172 fetch_fsmap learned FSMap version 25 2024-03-19T03:00:20.135 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.127+0000 7f8d3fffd740 1 -- 172.21.15.33:0/2083867283 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55e01faf73f0 con 0x55e01fe88f30 2024-03-19T03:00:20.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.128+0000 7f8d1e7fc640 1 -- 172.21.15.33:0/2083867283 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8d3002c7e0 con 0x55e01fe88f30 2024-03-19T03:00:20.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.128+0000 7f8d1e7fc640 1 -- 172.21.15.33:0/2083867283 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f8d2801f070 con 0x55e01fe88f30 2024-03-19T03:00:20.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.128+0000 7f8d3fffd740 10 client.17172 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:00:20.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.128+0000 7f8d3fffd740 10 client.17172 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:00:20.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.128+0000 7f8d3fffd740 20 client.17172 populate_metadata read hostname 'smithi033' 2024-03-19T03:00:20.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.128+0000 7f8d3fffd740 1 --2- 172.21.15.33:0/2083867283 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55e01fe99140 0x55e01fe9b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:20.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.128+0000 7f8d3fffd740 4 client.17172 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:00:20.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.128+0000 7f8d3fffd740 1 -- 172.21.15.33:0/2083867283 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e01fdf0a40 con 0x55e01fe99140 2024-03-19T03:00:20.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.128+0000 7f8d36ffd640 1 --2- 172.21.15.33:0/2083867283 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55e01fe99140 0x55e01fe9b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:20.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.128+0000 7f8d1e7fc640 10 client.17172 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:00:20.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.128+0000 7f8d36ffd640 1 --2- 172.21.15.33:0/2083867283 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55e01fe99140 0x55e01fe9b580 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=edc5d24df5d07a94 server_cookie=e431ca747101df62 in_seq=0 out_seq=0 2024-03-19T03:00:20.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.129+0000 7f8d1e7fc640 1 -- 172.21.15.33:0/2083867283 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e01fdf0a40 con 0x55e01fe99140 2024-03-19T03:00:20.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.129+0000 7f8d1e7fc640 10 client.17172 handle_command_reply: tid=0 2024-03-19T03:00:20.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.129+0000 7f8d3fffd740 1 client.17172 shutdown 2024-03-19T03:00:20.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.130+0000 7f8d3fffd740 1 -- 172.21.15.33:0/2083867283 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e01fe88f30 msgr2=0x55e01fe868d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:20.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.130+0000 7f8d3fffd740 1 --2- 172.21.15.33:0/2083867283 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e01fe88f30 0x55e01fe868d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f8d28013860 tx=0x7f8d28004ab0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.130+0000 7f8d3fffd740 1 -- 172.21.15.33:0/2083867283 shutdown_connections 2024-03-19T03:00:20.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.130+0000 7f8d3fffd740 1 --2- 172.21.15.33:0/2083867283 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55e01fe99140 0x55e01fe9b580 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.130+0000 7f8d3fffd740 1 --2- 172.21.15.33:0/2083867283 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e01fe88f30 0x55e01fe868d0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.130+0000 7f8d3fffd740 1 --2- 172.21.15.33:0/2083867283 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e01fe80350 0x55e01fe86390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.130+0000 7f8d3fffd740 1 --2- 172.21.15.33:0/2083867283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e01fe7fa30 0x55e01fe85e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.130+0000 7f8d1e7fc640 0 client.17172 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:00:20.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.130+0000 7f8d3fffd740 1 -- 172.21.15.33:0/2083867283 >> 172.21.15.33:0/2083867283 conn(0x55e01fe7a160 msgr2=0x55e01fe7beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:20.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.130+0000 7f8d3fffd740 1 -- 172.21.15.33:0/2083867283 shutdown_connections 2024-03-19T03:00:20.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.130+0000 7f8d3fffd740 1 -- 172.21.15.33:0/2083867283 wait complete. 2024-03-19T03:00:20.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.130+0000 7f8d3fffd740 20 client.17172 trim_cache size 0 max 16384 2024-03-19T03:00:20.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.147+0000 7f8d3fffd740 1 Processor -- start 2024-03-19T03:00:20.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.147+0000 7f8d3fffd740 1 -- start start 2024-03-19T03:00:20.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.147+0000 7f8d3fffd740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e01fe7fa30 0x55e01fe91470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:20.149 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.147+0000 7f8d3fffd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e01fe80350 0x55e01fe94a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:20.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.147+0000 7f8d3fffd740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e01fe88f30 0x55e01fe84fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:20.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.147+0000 7f8d3fffd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8d381b1510 con 0x55e01fe80350 2024-03-19T03:00:20.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.147+0000 7f8d3fffd740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8d381aede0 con 0x55e01fe7fa30 2024-03-19T03:00:20.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.147+0000 7f8d3fffd740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8d381acc50 con 0x55e01fe88f30 2024-03-19T03:00:20.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.147+0000 7f8d367fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e01fe80350 0x55e01fe94a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:20.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.147+0000 7f8d377fe640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e01fe88f30 0x55e01fe84fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:20.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.147+0000 7f8d367fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e01fe80350 0x55e01fe94a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42550/0 (socket says 172.21.15.33:42550) 2024-03-19T03:00:20.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.147+0000 7f8d367fc640 1 -- 172.21.15.33:0/364730030 learned_addr learned my addr 172.21.15.33:0/364730030 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:20.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.147+0000 7f8d36ffd640 1 --2- 172.21.15.33:0/364730030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e01fe7fa30 0x55e01fe91470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:20.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.148+0000 7f8d367fc640 1 -- 172.21.15.33:0/364730030 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e01fe88f30 msgr2=0x55e01fe84fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:20.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.148+0000 7f8d367fc640 1 --2- 172.21.15.33:0/364730030 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e01fe88f30 0x55e01fe84fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.148+0000 7f8d367fc640 1 -- 172.21.15.33:0/364730030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e01fe7fa30 msgr2=0x55e01fe91470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:20.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.148+0000 7f8d367fc640 1 --2- 172.21.15.33:0/364730030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e01fe7fa30 0x55e01fe91470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.148+0000 7f8d367fc640 1 -- 172.21.15.33:0/364730030 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e01fab70f0 con 0x55e01fe80350 2024-03-19T03:00:20.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.148+0000 7f8d367fc640 1 --2- 172.21.15.33:0/364730030 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e01fe80350 0x55e01fe94a60 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f8d200cce30 tx=0x7f8d20086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5690982283a700dd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:20.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.148+0000 7f8d2cff9640 1 -- 172.21.15.33:0/364730030 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d2009be40 con 0x55e01fe80350 2024-03-19T03:00:20.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.148+0000 7f8d2cff9640 1 -- 172.21.15.33:0/364730030 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8d2009b740 con 0x55e01fe80350 2024-03-19T03:00:20.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.148+0000 7f8d2cff9640 1 -- 172.21.15.33:0/364730030 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d200cf0b0 con 0x55e01fe80350 2024-03-19T03:00:20.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.149+0000 7f8d3fffd740 1 -- 172.21.15.33:0/364730030 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e01fe80350 msgr2=0x55e01fe94a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:20.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.149+0000 7f8d3fffd740 1 --2- 172.21.15.33:0/364730030 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e01fe80350 0x55e01fe94a60 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f8d200cce30 tx=0x7f8d20086040 comp rx=0 tx=0).stop 2024-03-19T03:00:20.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.149+0000 7f8d3fffd740 1 -- 172.21.15.33:0/364730030 shutdown_connections 2024-03-19T03:00:20.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.149+0000 7f8d3fffd740 1 --2- 172.21.15.33:0/364730030 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e01fe88f30 0x55e01fe84fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.149+0000 7f8d3fffd740 1 --2- 172.21.15.33:0/364730030 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e01fe80350 0x55e01fe94a60 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.149+0000 7f8d3fffd740 1 --2- 172.21.15.33:0/364730030 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e01fe7fa30 0x55e01fe91470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.149+0000 7f8d3fffd740 1 -- 172.21.15.33:0/364730030 >> 172.21.15.33:0/364730030 conn(0x55e01fe7a160 msgr2=0x55e01fe7b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:20.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.149+0000 7f8d3fffd740 1 -- 172.21.15.33:0/364730030 shutdown_connections 2024-03-19T03:00:20.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.149+0000 7f8d3fffd740 1 -- 172.21.15.33:0/364730030 wait complete. 2024-03-19T03:00:20.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.149+0000 7f8d3fffd740 1 -- start start 2024-03-19T03:00:20.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.149+0000 7f8d3fffd740 1 Processor -- start 2024-03-19T03:00:20.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.149+0000 7f8d3fffd740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e01fe7fa30 0x55e01fefb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:20.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.149+0000 7f8d3fffd740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e01fe80350 0x55e01fefbee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:20.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.149+0000 7f8d3fffd740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e01fe88f30 0x55e01fefa6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:20.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.149+0000 7f8d3fffd740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8d381ad2a0 con 0x55e01fe7fa30 2024-03-19T03:00:20.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.149+0000 7f8d3fffd740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8d381ae710 con 0x55e01fe88f30 2024-03-19T03:00:20.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.149+0000 7f8d3fffd740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8d381ad970 con 0x55e01fe80350 2024-03-19T03:00:20.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.149+0000 7f8d36ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e01fe7fa30 0x55e01fefb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:20.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.149+0000 7f8d3fffd740 10 client.0 _ll_register_callbacks cb 0x7f8d280137b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:00:20.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.150+0000 7f8d36ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e01fe7fa30 0x55e01fefb9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:42564/0 (socket says 172.21.15.33:42564) 2024-03-19T03:00:20.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.150+0000 7f8d36ffd640 1 -- 172.21.15.33:0/1745478694 learned_addr learned my addr 172.21.15.33:0/1745478694 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:20.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.150+0000 7f8d377fe640 1 --2- 172.21.15.33:0/1745478694 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e01fe88f30 0x55e01fefa6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:20.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.150+0000 7f8d367fc640 1 --2- 172.21.15.33:0/1745478694 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e01fe80350 0x55e01fefbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:20.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.150+0000 7f8d36ffd640 1 -- 172.21.15.33:0/1745478694 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e01fe80350 msgr2=0x55e01fefbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:20.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.150+0000 7f8d36ffd640 1 --2- 172.21.15.33:0/1745478694 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e01fe80350 0x55e01fefbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.150+0000 7f8d36ffd640 1 -- 172.21.15.33:0/1745478694 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e01fe88f30 msgr2=0x55e01fefa6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:20.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.150+0000 7f8d36ffd640 1 --2- 172.21.15.33:0/1745478694 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e01fe88f30 0x55e01fefa6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.150+0000 7f8d36ffd640 1 -- 172.21.15.33:0/1745478694 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e01fefad30 con 0x55e01fe7fa30 2024-03-19T03:00:20.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.150+0000 7f8d367fc640 1 --2- 172.21.15.33:0/1745478694 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e01fe80350 0x55e01fefbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:00:20.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.150+0000 7f8d36ffd640 1 --2- 172.21.15.33:0/1745478694 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e01fe7fa30 0x55e01fefb9a0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f8d30005150 tx=0x7f8d300050e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e02a86fd0f623435 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:20.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.150+0000 7f8d1e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:00:20.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.150+0000 7f8d1e7fc640 1 -- 172.21.15.33:0/1745478694 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d30053070 con 0x55e01fe7fa30 2024-03-19T03:00:20.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.150+0000 7f8d1e7fc640 1 -- 172.21.15.33:0/1745478694 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7f8d30004240 con 0x55e01fe7fa30 2024-03-19T03:00:20.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.150+0000 7f8d3fffd740 1 -- 172.21.15.33:0/1745478694 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d2009b740 con 0x55e01fe7fa30 2024-03-19T03:00:20.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.150+0000 7f8d1e7fc640 1 -- 172.21.15.33:0/1745478694 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8d3005f050 con 0x55e01fe7fa30 2024-03-19T03:00:20.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.150+0000 7f8d3fffd740 10 client.17178 fetch_fsmap learned FSMap version 25 2024-03-19T03:00:20.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.150+0000 7f8d3fffd740 1 -- 172.21.15.33:0/1745478694 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55e01ff03350 con 0x55e01fe7fa30 2024-03-19T03:00:20.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.150+0000 7f8d1e7fc640 1 -- 172.21.15.33:0/1745478694 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8d2009b740 con 0x55e01fe7fa30 2024-03-19T03:00:20.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.151+0000 7f8d1e7fc640 1 -- 172.21.15.33:0/1745478694 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f8d300035f0 con 0x55e01fe7fa30 2024-03-19T03:00:20.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.151+0000 7f8d3fffd740 10 client.17178 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:00:20.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.151+0000 7f8d3fffd740 10 client.17178 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:00:20.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.151+0000 7f8d3fffd740 20 client.17178 populate_metadata read hostname 'smithi033' 2024-03-19T03:00:20.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.151+0000 7f8d3fffd740 1 --2- 172.21.15.33:0/1745478694 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55e01ff036e0 0x55e01ff05ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:20.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.151+0000 7f8d3fffd740 4 client.17178 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T03:00:20.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.151+0000 7f8d3fffd740 1 -- 172.21.15.33:0/1745478694 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e01fdead90 con 0x55e01ff036e0 2024-03-19T03:00:20.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.151+0000 7f8d367fc640 1 --2- 172.21.15.33:0/1745478694 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55e01ff036e0 0x55e01ff05ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:20.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.151+0000 7f8d367fc640 1 --2- 172.21.15.33:0/1745478694 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55e01ff036e0 0x55e01ff05ad0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a49a6a98993a895 server_cookie=8be16497c5a2f551 in_seq=0 out_seq=0 2024-03-19T03:00:20.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.151+0000 7f8d1e7fc640 10 client.17178 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:00:20.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.151+0000 7f8d1e7fc640 1 -- 172.21.15.33:0/1745478694 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55e01fdead90 con 0x55e01ff036e0 2024-03-19T03:00:20.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.151+0000 7f8d1e7fc640 10 client.17178 handle_command_reply: tid=0 2024-03-19T03:00:20.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.151+0000 7f8d3fffd740 1 client.17178 shutdown 2024-03-19T03:00:20.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.151+0000 7f8d3fffd740 1 -- 172.21.15.33:0/1745478694 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e01fe7fa30 msgr2=0x55e01fefb9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:20.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.151+0000 7f8d3fffd740 1 --2- 172.21.15.33:0/1745478694 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e01fe7fa30 0x55e01fefb9a0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f8d30005150 tx=0x7f8d300050e0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.151+0000 7f8d3fffd740 1 -- 172.21.15.33:0/1745478694 shutdown_connections 2024-03-19T03:00:20.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.151+0000 7f8d3fffd740 1 --2- 172.21.15.33:0/1745478694 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55e01ff036e0 0x55e01ff05ad0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.151+0000 7f8d3fffd740 1 --2- 172.21.15.33:0/1745478694 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e01fe80350 0x55e01fefbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.151+0000 7f8d3fffd740 1 --2- 172.21.15.33:0/1745478694 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e01fe88f30 0x55e01fefa6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.151+0000 7f8d1e7fc640 0 client.17178 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:00:20.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.151+0000 7f8d3fffd740 1 --2- 172.21.15.33:0/1745478694 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e01fe7fa30 0x55e01fefb9a0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.151+0000 7f8d3fffd740 1 -- 172.21.15.33:0/1745478694 >> 172.21.15.33:0/1745478694 conn(0x55e01fe7a160 msgr2=0x55e01fe7be40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:20.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.152+0000 7f8d3fffd740 1 -- 172.21.15.33:0/1745478694 shutdown_connections 2024-03-19T03:00:20.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.152+0000 7f8d3fffd740 1 -- 172.21.15.33:0/1745478694 wait complete. 2024-03-19T03:00:20.159 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.152+0000 7f8d3fffd740 20 client.17178 trim_cache size 0 max 16384 2024-03-19T03:00:20.160 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T03:00:20.160 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T03:00:20.160 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-03-19T03:00:20.160 INFO:teuthology.orchestra.run.smithi033.stdout: "9c2358da-c0bb-4313-8c1c-f720a7f64250": { 2024-03-19T03:00:20.160 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-03-19T03:00:20.160 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "9c2358da-c0bb-4313-8c1c-f720a7f64250", 2024-03-19T03:00:20.160 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-03-19T03:00:20.160 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:00:20.160 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:00:20.160 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T03:00:20.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.152+0000 7f8d3d777640 1 -- 172.21.15.33:0/1186314407 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f8d240a2970 msgr2=0x7f8d240a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:20.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.152+0000 7f8d3d777640 1 --2- 172.21.15.33:0/1186314407 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f8d240a2970 0x7f8d240a4e30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8d28005fd0 tx=0x7f8d28013e50 comp rx=0 tx=0).stop 2024-03-19T03:00:20.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.152+0000 7f8d3d777640 1 -- 172.21.15.33:0/1186314407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d38100680 msgr2=0x7f8d38198540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:20.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.152+0000 7f8d3d777640 1 --2- 172.21.15.33:0/1186314407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d38100680 0x7f8d38198540 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f8d2000ea60 tx=0x7f8d2000ef20 comp rx=0 tx=0).stop 2024-03-19T03:00:20.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.152+0000 7f8d3d777640 1 -- 172.21.15.33:0/1186314407 shutdown_connections 2024-03-19T03:00:20.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.152+0000 7f8d3d777640 1 --2- 172.21.15.33:0/1186314407 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f8d240a2970 0x7f8d240a4e30 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.152+0000 7f8d3d777640 1 --2- 172.21.15.33:0/1186314407 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d381077c0 0x7f8d3819c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.161 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.152+0000 7f8d3d777640 1 --2- 172.21.15.33:0/1186314407 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d38100680 0x7f8d38198540 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.152+0000 7f8d3d777640 1 --2- 172.21.15.33:0/1186314407 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d380ffcc0 0x7f8d38198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:20.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.152+0000 7f8d3d777640 1 -- 172.21.15.33:0/1186314407 >> 172.21.15.33:0/1186314407 conn(0x7f8d38076c40 msgr2=0x7f8d380fe4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:20.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.152+0000 7f8d3d777640 1 -- 172.21.15.33:0/1186314407 shutdown_connections 2024-03-19T03:00:20.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:20.152+0000 7f8d3d777640 1 -- 172.21.15.33:0/1186314407 wait complete. 2024-03-19T03:00:20.166 INFO:tasks.cephfs.filesystem:scrub status for tag:9c2358da-c0bb-4313-8c1c-f720a7f64250 - {'path': '/', 'tag': '9c2358da-c0bb-4313-8c1c-f720a7f64250', 'options': 'recursive,force'} 2024-03-19T03:00:20.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:20 smithi096 ceph-mon[35946]: pgmap v211: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-19T03:00:20.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:20 smithi096 ceph-mon[35946]: scrub queued for path: / 2024-03-19T03:00:20.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:20 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T03:00:20.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:20 smithi096 ceph-mon[35946]: scrub summary: active paths [/] 2024-03-19T03:00:20.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:20 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T03:00:21.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:20 smithi155 ceph-mon[35804]: pgmap v211: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-19T03:00:21.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:20 smithi155 ceph-mon[35804]: scrub queued for path: / 2024-03-19T03:00:21.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:20 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T03:00:21.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:20 smithi155 ceph-mon[35804]: scrub summary: active paths [/] 2024-03-19T03:00:21.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:20 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T03:00:21.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:20 smithi033 ceph-mon[31973]: pgmap v211: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-19T03:00:21.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:20 smithi033 ceph-mon[31973]: scrub queued for path: / 2024-03-19T03:00:21.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:20 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T03:00:21.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:20 smithi033 ceph-mon[31973]: scrub summary: active paths [/] 2024-03-19T03:00:21.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:20 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T03:00:22.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:22 smithi096 ceph-mon[35946]: pgmap v212: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 94 B/s wr, 5 op/s 2024-03-19T03:00:22.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:22 smithi096 ceph-mon[35946]: scrub summary: idle 2024-03-19T03:00:23.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:22 smithi155 ceph-mon[35804]: pgmap v212: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 94 B/s wr, 5 op/s 2024-03-19T03:00:23.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:22 smithi155 ceph-mon[35804]: scrub summary: idle 2024-03-19T03:00:23.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:22 smithi033 ceph-mon[31973]: pgmap v212: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 94 B/s wr, 5 op/s 2024-03-19T03:00:23.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:22 smithi033 ceph-mon[31973]: scrub summary: idle 2024-03-19T03:00:24.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:24 smithi096 ceph-mon[35946]: pgmap v213: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-19T03:00:25.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:24 smithi155 ceph-mon[35804]: pgmap v213: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-19T03:00:25.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:24 smithi033 ceph-mon[31973]: pgmap v213: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-19T03:00:26.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:26 smithi096 ceph-mon[35946]: pgmap v214: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:00:27.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:26 smithi155 ceph-mon[35804]: pgmap v214: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:00:27.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:26 smithi033 ceph-mon[31973]: pgmap v214: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:00:28.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:28 smithi096 ceph-mon[35946]: pgmap v215: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:00:29.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:28 smithi155 ceph-mon[35804]: pgmap v215: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:00:29.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:28 smithi033 ceph-mon[31973]: pgmap v215: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:00:30.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:30 smithi096 ceph-mon[35946]: pgmap v216: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:00:31.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:30 smithi155 ceph-mon[35804]: pgmap v216: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:00:31.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:30 smithi033 ceph-mon[31973]: pgmap v216: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:00:32.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:32 smithi096 ceph-mon[35946]: pgmap v217: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:00:33.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:32 smithi155 ceph-mon[35804]: pgmap v217: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:00:33.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:32 smithi033 ceph-mon[31973]: pgmap v217: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:00:34.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:34 smithi096 ceph-mon[35946]: pgmap v218: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:00:35.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:34 smithi155 ceph-mon[35804]: pgmap v218: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:00:35.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:34 smithi033 ceph-mon[31973]: pgmap v218: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:00:36.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:36 smithi096 ceph-mon[35946]: pgmap v219: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:00:37.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:36 smithi155 ceph-mon[35804]: pgmap v219: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:00:37.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:36 smithi033 ceph-mon[31973]: pgmap v219: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:00:38.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:38 smithi096 ceph-mon[35946]: pgmap v220: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:00:38.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:38 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:00:39.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:38 smithi155 ceph-mon[35804]: pgmap v220: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:00:39.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:38 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:00:39.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:38 smithi033 ceph-mon[31973]: pgmap v220: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:00:39.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:38 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:00:39.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:39 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:00:39.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:39 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:00:39.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:39 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:00:40.016 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:39 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:00:40.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:39 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:00:40.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:39 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:00:40.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:39 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:00:40.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:39 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:00:40.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:39 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:00:41.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:40 smithi096 ceph-mon[35946]: pgmap v221: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:00:41.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:40 smithi155 ceph-mon[35804]: pgmap v221: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:00:41.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:40 smithi033 ceph-mon[31973]: pgmap v221: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:00:43.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:42 smithi096 ceph-mon[35946]: pgmap v222: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:00:43.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:42 smithi155 ceph-mon[35804]: pgmap v222: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:00:43.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:42 smithi033 ceph-mon[31973]: pgmap v222: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:00:45.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:44 smithi096 ceph-mon[35946]: pgmap v223: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:00:45.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:44 smithi155 ceph-mon[35804]: pgmap v223: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:00:45.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:44 smithi033 ceph-mon[31973]: pgmap v223: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:00:47.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:46 smithi096 ceph-mon[35946]: pgmap v224: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:00:47.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:46 smithi155 ceph-mon[35804]: pgmap v224: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:00:47.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:46 smithi033 ceph-mon[31973]: pgmap v224: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:00:49.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:48 smithi096 ceph-mon[35946]: pgmap v225: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:00:49.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:48 smithi155 ceph-mon[35804]: pgmap v225: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:00:49.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:48 smithi033 ceph-mon[31973]: pgmap v225: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:00:50.168 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T03:00:50.168 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T03:00:50.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.284+0000 7fa7649b2640 1 -- 172.21.15.33:0/1761146301 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7601017c0 msgr2=0x7fa760101c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.284+0000 7fa7649b2640 1 --2- 172.21.15.33:0/1761146301 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7601017c0 0x7fa760101c40 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fa748009930 tx=0x7fa74802bc30 comp rx=0 tx=0).stop 2024-03-19T03:00:50.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.284+0000 7fa7649b2640 1 -- 172.21.15.33:0/1761146301 shutdown_connections 2024-03-19T03:00:50.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.284+0000 7fa7649b2640 1 --2- 172.21.15.33:0/1761146301 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa760102180 0x7fa76010e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.284+0000 7fa7649b2640 1 --2- 172.21.15.33:0/1761146301 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7601017c0 0x7fa760101c40 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.284+0000 7fa7649b2640 1 --2- 172.21.15.33:0/1761146301 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7601077d0 0x7fa760107bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.284+0000 7fa7649b2640 1 -- 172.21.15.33:0/1761146301 >> 172.21.15.33:0/1761146301 conn(0x7fa7600fd250 msgr2=0x7fa7600ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:50.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.284+0000 7fa7649b2640 1 -- 172.21.15.33:0/1761146301 shutdown_connections 2024-03-19T03:00:50.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.284+0000 7fa7649b2640 1 -- 172.21.15.33:0/1761146301 wait complete. 2024-03-19T03:00:50.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.285+0000 7fa7649b2640 1 Processor -- start 2024-03-19T03:00:50.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.285+0000 7fa7649b2640 1 -- start start 2024-03-19T03:00:50.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.285+0000 7fa7649b2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7601017c0 0x7fa76019a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.285+0000 7fa7649b2640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa760102180 0x7fa76019a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.285+0000 7fa7649b2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7601077d0 0x7fa76019eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.285+0000 7fa7649b2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa760111a90 con 0x7fa7601017c0 2024-03-19T03:00:50.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.285+0000 7fa7649b2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa760111910 con 0x7fa7601077d0 2024-03-19T03:00:50.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.285+0000 7fa7649b2640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa760111c10 con 0x7fa760102180 2024-03-19T03:00:50.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.286+0000 7fa75e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7601017c0 0x7fa76019a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.286+0000 7fa75e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7601017c0 0x7fa76019a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58658/0 (socket says 172.21.15.33:58658) 2024-03-19T03:00:50.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.286+0000 7fa75e59c640 1 -- 172.21.15.33:0/3625994634 learned_addr learned my addr 172.21.15.33:0/3625994634 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:50.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.286+0000 7fa75ed9d640 1 --2- 172.21.15.33:0/3625994634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7601077d0 0x7fa76019eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.286+0000 7fa757fff640 1 --2- 172.21.15.33:0/3625994634 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa760102180 0x7fa76019a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.286+0000 7fa75e59c640 1 -- 172.21.15.33:0/3625994634 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa760102180 msgr2=0x7fa76019a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.286+0000 7fa75e59c640 1 --2- 172.21.15.33:0/3625994634 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa760102180 0x7fa76019a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.286+0000 7fa75e59c640 1 -- 172.21.15.33:0/3625994634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7601077d0 msgr2=0x7fa76019eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.286+0000 7fa75e59c640 1 --2- 172.21.15.33:0/3625994634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7601077d0 0x7fa76019eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.286+0000 7fa75e59c640 1 -- 172.21.15.33:0/3625994634 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa76019f240 con 0x7fa7601017c0 2024-03-19T03:00:50.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.286+0000 7fa75e59c640 1 --2- 172.21.15.33:0/3625994634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7601017c0 0x7fa76019a250 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fa75000e9f0 tx=0x7fa75000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:50.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.286+0000 7fa7577fe640 1 -- 172.21.15.33:0/3625994634 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75000cd80 con 0x7fa7601017c0 2024-03-19T03:00:50.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.286+0000 7fa7577fe640 1 -- 172.21.15.33:0/3625994634 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa7500108e0 con 0x7fa7601017c0 2024-03-19T03:00:50.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.286+0000 7fa7649b2640 1 -- 172.21.15.33:0/3625994634 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa76019f530 con 0x7fa7601017c0 2024-03-19T03:00:50.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.286+0000 7fa7577fe640 1 -- 172.21.15.33:0/3625994634 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa750018920 con 0x7fa7601017c0 2024-03-19T03:00:50.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.286+0000 7fa7649b2640 1 -- 172.21.15.33:0/3625994634 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7601a6dc0 con 0x7fa7601017c0 2024-03-19T03:00:50.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.288+0000 7fa7577fe640 1 -- 172.21.15.33:0/3625994634 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fa750010a90 con 0x7fa7601017c0 2024-03-19T03:00:50.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.289+0000 7fa7577fe640 1 --2- 172.21.15.33:0/3625994634 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa74c0a2970 0x7fa74c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.289+0000 7fa7675a0740 1 Processor -- start 2024-03-19T03:00:50.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.289+0000 7fa757fff640 1 --2- 172.21.15.33:0/3625994634 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa74c0a2970 0x7fa74c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.289+0000 7fa7577fe640 1 -- 172.21.15.33:0/3625994634 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7fa7500cf540 con 0x7fa7601017c0 2024-03-19T03:00:50.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.289+0000 7fa7675a0740 1 -- start start 2024-03-19T03:00:50.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.290+0000 7fa7675a0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fd9d73a30 0x560fd9d73e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.290+0000 7fa7675a0740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fd9d74350 0x560fd9d7c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.290+0000 7fa7675a0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fd9d7cf30 0x560fd9d7f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.290+0000 7fa7675a0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560fd9cd7500 con 0x560fd9d7cf30 2024-03-19T03:00:50.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.290+0000 7fa7675a0740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x560fd9c7fbe0 con 0x560fd9d74350 2024-03-19T03:00:50.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.290+0000 7fa7675a0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560fd9cd35a0 con 0x560fd9d73a30 2024-03-19T03:00:50.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.290+0000 7fa75e59c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fd9d73a30 0x560fd9d73e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.290+0000 7fa757fff640 1 --2- 172.21.15.33:0/3625994634 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa74c0a2970 0x7fa74c0a4e30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa748005ab0 tx=0x7fa748005fd0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:50.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.290+0000 7fa75ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fd9d7cf30 0x560fd9d7f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.290+0000 7fa757fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fd9d74350 0x560fd9d7c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.290+0000 7fa75e59c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fd9d73a30 0x560fd9d73e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:37958/0 (socket says 172.21.15.33:37958) 2024-03-19T03:00:50.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.290+0000 7fa75e59c640 1 -- 172.21.15.33:0/3155551632 learned_addr learned my addr 172.21.15.33:0/3155551632 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:50.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.290+0000 7fa75e59c640 1 -- 172.21.15.33:0/3155551632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fd9d74350 msgr2=0x560fd9d7c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:50.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.290+0000 7fa75e59c640 1 --2- 172.21.15.33:0/3155551632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fd9d74350 0x560fd9d7c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.290+0000 7fa75e59c640 1 -- 172.21.15.33:0/3155551632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fd9d7cf30 msgr2=0x560fd9d7f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:50.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.290+0000 7fa75e59c640 1 --2- 172.21.15.33:0/3155551632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fd9d7cf30 0x560fd9d7f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.290+0000 7fa75e59c640 1 -- 172.21.15.33:0/3155551632 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fd9ca1000 con 0x560fd9d73a30 2024-03-19T03:00:50.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.290+0000 7fa75e59c640 1 --2- 172.21.15.33:0/3155551632 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fd9d73a30 0x560fd9d73e10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa7500a39e0 tx=0x7fa75009b720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b921197e739fe527 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:50.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.290+0000 7fa754ff9640 1 -- 172.21.15.33:0/3155551632 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7500a3f10 con 0x560fd9d73a30 2024-03-19T03:00:50.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.290+0000 7fa754ff9640 1 -- 172.21.15.33:0/3155551632 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa7500ccae0 con 0x560fd9d73a30 2024-03-19T03:00:50.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.290+0000 7fa754ff9640 1 -- 172.21.15.33:0/3155551632 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75009ac70 con 0x560fd9d73a30 2024-03-19T03:00:50.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.291+0000 7fa7675a0740 1 -- 172.21.15.33:0/3155551632 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fd9d73a30 msgr2=0x560fd9d73e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.291+0000 7fa7675a0740 1 --2- 172.21.15.33:0/3155551632 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fd9d73a30 0x560fd9d73e10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa7500a39e0 tx=0x7fa75009b720 comp rx=0 tx=0).stop 2024-03-19T03:00:50.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.291+0000 7fa7675a0740 1 -- 172.21.15.33:0/3155551632 shutdown_connections 2024-03-19T03:00:50.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.291+0000 7fa7675a0740 1 --2- 172.21.15.33:0/3155551632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fd9d7cf30 0x560fd9d7f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.291+0000 7fa7675a0740 1 --2- 172.21.15.33:0/3155551632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fd9d74350 0x560fd9d7c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.291+0000 7fa7675a0740 1 --2- 172.21.15.33:0/3155551632 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fd9d73a30 0x560fd9d73e10 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.291+0000 7fa7675a0740 1 -- 172.21.15.33:0/3155551632 >> 172.21.15.33:0/3155551632 conn(0x560fd9d6e160 msgr2=0x560fd9d6e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:50.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.291+0000 7fa7675a0740 1 -- 172.21.15.33:0/3155551632 shutdown_connections 2024-03-19T03:00:50.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.291+0000 7fa7675a0740 1 -- 172.21.15.33:0/3155551632 wait complete. 2024-03-19T03:00:50.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.292+0000 7fa7675a0740 1 -- start start 2024-03-19T03:00:50.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.292+0000 7fa7675a0740 1 Processor -- start 2024-03-19T03:00:50.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.292+0000 7fa7675a0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fd9d73a30 0x560fd9d79df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.292+0000 7fa7675a0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fd9d74350 0x560fd9d7a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.292+0000 7fa7675a0740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fd9d7cf30 0x560fd9d7a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.292+0000 7fa7675a0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560fd9d7fcc0 con 0x560fd9d73a30 2024-03-19T03:00:50.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.292+0000 7fa7675a0740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x560fd9c93160 con 0x560fd9d7cf30 2024-03-19T03:00:50.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.292+0000 7fa7675a0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560fd9d7fe40 con 0x560fd9d74350 2024-03-19T03:00:50.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.292+0000 7fa7675a0740 10 client.0 _ll_register_callbacks cb 0x560fd9ccfa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:00:50.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.292+0000 7fa75e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fd9d73a30 0x560fd9d79df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.292+0000 7fa75e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fd9d73a30 0x560fd9d79df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58678/0 (socket says 172.21.15.33:58678) 2024-03-19T03:00:50.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.292+0000 7fa75e59c640 1 -- 172.21.15.33:0/1096801009 learned_addr learned my addr 172.21.15.33:0/1096801009 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:50.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.292+0000 7fa757fff640 1 --2- 172.21.15.33:0/1096801009 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fd9d74350 0x560fd9d7a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.292+0000 7fa75ed9d640 1 --2- 172.21.15.33:0/1096801009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fd9d7cf30 0x560fd9d7a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.292+0000 7fa75e59c640 1 -- 172.21.15.33:0/1096801009 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fd9d74350 msgr2=0x560fd9d7a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:50.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.292+0000 7fa75e59c640 1 --2- 172.21.15.33:0/1096801009 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fd9d74350 0x560fd9d7a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.292+0000 7fa75e59c640 1 -- 172.21.15.33:0/1096801009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fd9d7cf30 msgr2=0x560fd9d7a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:50.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.292+0000 7fa75e59c640 1 --2- 172.21.15.33:0/1096801009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fd9d7cf30 0x560fd9d7a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.292+0000 7fa75e59c640 1 -- 172.21.15.33:0/1096801009 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fd9ab57d0 con 0x560fd9d73a30 2024-03-19T03:00:50.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.293+0000 7fa75e59c640 1 --2- 172.21.15.33:0/1096801009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fd9d73a30 0x560fd9d79df0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fa75000e9c0 tx=0x7fa75000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=992d367ee9f72660 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:50.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.293+0000 7fa745ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:00:50.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.293+0000 7fa745ffb640 1 -- 172.21.15.33:0/1096801009 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7500a3f10 con 0x560fd9d73a30 2024-03-19T03:00:50.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.293+0000 7fa745ffb640 1 -- 172.21.15.33:0/1096801009 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7fa7500afd60 con 0x560fd9d73a30 2024-03-19T03:00:50.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.293+0000 7fa7675a0740 1 -- 172.21.15.33:0/1096801009 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7500ccae0 con 0x560fd9d73a30 2024-03-19T03:00:50.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.293+0000 7fa745ffb640 1 -- 172.21.15.33:0/1096801009 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa7500ad050 con 0x560fd9d73a30 2024-03-19T03:00:50.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.293+0000 7fa7675a0740 10 client.17190 fetch_fsmap learned FSMap version 25 2024-03-19T03:00:50.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.293+0000 7fa7675a0740 1 -- 172.21.15.33:0/1096801009 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x560fd99eb3f0 con 0x560fd9d73a30 2024-03-19T03:00:50.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.293+0000 7fa745ffb640 1 -- 172.21.15.33:0/1096801009 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa7500ccae0 con 0x560fd9d73a30 2024-03-19T03:00:50.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.293+0000 7fa745ffb640 1 -- 172.21.15.33:0/1096801009 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fa750097570 con 0x560fd9d73a30 2024-03-19T03:00:50.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.293+0000 7fa7675a0740 10 client.17190 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:00:50.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.293+0000 7fa7675a0740 10 client.17190 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:00:50.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.293+0000 7fa7675a0740 20 client.17190 populate_metadata read hostname 'smithi033' 2024-03-19T03:00:50.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.293+0000 7fa7675a0740 1 --2- 172.21.15.33:0/1096801009 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x560fd9d8cf70 0x560fd9d8f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.293+0000 7fa7675a0740 4 client.17190 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:00:50.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.293+0000 7fa7675a0740 1 -- 172.21.15.33:0/1096801009 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560fd9ce4a40 con 0x560fd9d8cf70 2024-03-19T03:00:50.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.293+0000 7fa757fff640 1 --2- 172.21.15.33:0/1096801009 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x560fd9d8cf70 0x560fd9d8f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.294+0000 7fa757fff640 1 --2- 172.21.15.33:0/1096801009 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x560fd9d8cf70 0x560fd9d8f360 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0f3d4aa8c520f32 server_cookie=b6252ff42e4cc234 in_seq=0 out_seq=0 2024-03-19T03:00:50.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.294+0000 7fa745ffb640 10 client.17190 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:00:50.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.294+0000 7fa745ffb640 1 -- 172.21.15.33:0/1096801009 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560fd9ce4a40 con 0x560fd9d8cf70 2024-03-19T03:00:50.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.294+0000 7fa745ffb640 10 client.17190 handle_command_reply: tid=0 2024-03-19T03:00:50.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.294+0000 7fa7675a0740 1 client.17190 shutdown 2024-03-19T03:00:50.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.295+0000 7fa7675a0740 1 -- 172.21.15.33:0/1096801009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fd9d73a30 msgr2=0x560fd9d79df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.295+0000 7fa7675a0740 1 --2- 172.21.15.33:0/1096801009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fd9d73a30 0x560fd9d79df0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fa75000e9c0 tx=0x7fa75000cf90 comp rx=0 tx=0).stop 2024-03-19T03:00:50.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.295+0000 7fa7675a0740 1 -- 172.21.15.33:0/1096801009 shutdown_connections 2024-03-19T03:00:50.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.295+0000 7fa7675a0740 1 --2- 172.21.15.33:0/1096801009 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x560fd9d8cf70 0x560fd9d8f360 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.295+0000 7fa7675a0740 1 --2- 172.21.15.33:0/1096801009 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fd9d74350 0x560fd9d7a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.295+0000 7fa7675a0740 1 --2- 172.21.15.33:0/1096801009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fd9d7cf30 0x560fd9d7a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.295+0000 7fa7675a0740 1 --2- 172.21.15.33:0/1096801009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fd9d73a30 0x560fd9d79df0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.295+0000 7fa745ffb640 0 client.17190 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:00:50.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.295+0000 7fa7675a0740 1 -- 172.21.15.33:0/1096801009 >> 172.21.15.33:0/1096801009 conn(0x560fd9d6e160 msgr2=0x560fd9d6feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:50.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.295+0000 7fa7675a0740 1 -- 172.21.15.33:0/1096801009 shutdown_connections 2024-03-19T03:00:50.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.295+0000 7fa7675a0740 1 -- 172.21.15.33:0/1096801009 wait complete. 2024-03-19T03:00:50.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.295+0000 7fa7675a0740 20 client.17190 trim_cache size 0 max 16384 2024-03-19T03:00:50.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.312+0000 7fa7675a0740 1 Processor -- start 2024-03-19T03:00:50.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.312+0000 7fa7675a0740 1 -- start start 2024-03-19T03:00:50.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.312+0000 7fa7675a0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fd9d73a30 0x560fd9d7c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.313+0000 7fa7675a0740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fd9d74350 0x560fd9d6fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.313+0000 7fa7675a0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fd9d7cf30 0x560fd9d92380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.313+0000 7fa7675a0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa7601b37d0 con 0x560fd9d7cf30 2024-03-19T03:00:50.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.313+0000 7fa7675a0740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa7601b10a0 con 0x560fd9d74350 2024-03-19T03:00:50.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.313+0000 7fa7675a0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa7601aef10 con 0x560fd9d73a30 2024-03-19T03:00:50.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.313+0000 7fa75ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fd9d7cf30 0x560fd9d92380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.313+0000 7fa75ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fd9d7cf30 0x560fd9d92380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58686/0 (socket says 172.21.15.33:58686) 2024-03-19T03:00:50.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.313+0000 7fa75ed9d640 1 -- 172.21.15.33:0/549005495 learned_addr learned my addr 172.21.15.33:0/549005495 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:50.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.313+0000 7fa757fff640 1 --2- 172.21.15.33:0/549005495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fd9d74350 0x560fd9d6fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.313+0000 7fa75e59c640 1 --2- 172.21.15.33:0/549005495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fd9d73a30 0x560fd9d7c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.313+0000 7fa75ed9d640 1 -- 172.21.15.33:0/549005495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fd9d73a30 msgr2=0x560fd9d7c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:50.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.313+0000 7fa75ed9d640 1 --2- 172.21.15.33:0/549005495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fd9d73a30 0x560fd9d7c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.313+0000 7fa75ed9d640 1 -- 172.21.15.33:0/549005495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fd9d74350 msgr2=0x560fd9d6fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:50.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.313+0000 7fa75ed9d640 1 --2- 172.21.15.33:0/549005495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fd9d74350 0x560fd9d6fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.313+0000 7fa75ed9d640 1 -- 172.21.15.33:0/549005495 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fd99ab0f0 con 0x560fd9d7cf30 2024-03-19T03:00:50.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.313+0000 7fa75ed9d640 1 --2- 172.21.15.33:0/549005495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fd9d7cf30 0x560fd9d92380 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fa758003c30 tx=0x7fa758004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e354b83c1454c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:50.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.313+0000 7fa747fff640 1 -- 172.21.15.33:0/549005495 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa758010430 con 0x560fd9d7cf30 2024-03-19T03:00:50.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.313+0000 7fa747fff640 1 -- 172.21.15.33:0/549005495 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa758010a80 con 0x560fd9d7cf30 2024-03-19T03:00:50.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.313+0000 7fa747fff640 1 -- 172.21.15.33:0/549005495 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75801bb10 con 0x560fd9d7cf30 2024-03-19T03:00:50.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.314+0000 7fa7675a0740 1 -- 172.21.15.33:0/549005495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fd9d7cf30 msgr2=0x560fd9d92380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.314+0000 7fa7675a0740 1 --2- 172.21.15.33:0/549005495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fd9d7cf30 0x560fd9d92380 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fa758003c30 tx=0x7fa758004410 comp rx=0 tx=0).stop 2024-03-19T03:00:50.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.314+0000 7fa7675a0740 1 -- 172.21.15.33:0/549005495 shutdown_connections 2024-03-19T03:00:50.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.314+0000 7fa7675a0740 1 --2- 172.21.15.33:0/549005495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fd9d7cf30 0x560fd9d92380 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.314+0000 7fa7675a0740 1 --2- 172.21.15.33:0/549005495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fd9d74350 0x560fd9d6fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.314+0000 7fa7675a0740 1 --2- 172.21.15.33:0/549005495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fd9d73a30 0x560fd9d7c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.314+0000 7fa7675a0740 1 -- 172.21.15.33:0/549005495 >> 172.21.15.33:0/549005495 conn(0x560fd9d6e160 msgr2=0x560fd9d6f570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:50.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.314+0000 7fa7675a0740 1 -- 172.21.15.33:0/549005495 shutdown_connections 2024-03-19T03:00:50.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.314+0000 7fa7675a0740 1 -- 172.21.15.33:0/549005495 wait complete. 2024-03-19T03:00:50.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.314+0000 7fa7675a0740 1 -- start start 2024-03-19T03:00:50.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.314+0000 7fa7675a0740 1 Processor -- start 2024-03-19T03:00:50.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.314+0000 7fa7675a0740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fd9d73a30 0x560fd9d8fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.314+0000 7fa7675a0740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fd9d74350 0x560fd9d901b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.314+0000 7fa7675a0740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fd9d7cf30 0x560fd9d906f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.314+0000 7fa7675a0740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa7601af560 con 0x560fd9d74350 2024-03-19T03:00:50.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.314+0000 7fa7675a0740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa7601b09d0 con 0x560fd9d7cf30 2024-03-19T03:00:50.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.314+0000 7fa7675a0740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa7601afc30 con 0x560fd9d73a30 2024-03-19T03:00:50.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.315+0000 7fa7675a0740 10 client.0 _ll_register_callbacks cb 0x7fa758009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:00:50.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.315+0000 7fa757fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fd9d74350 0x560fd9d901b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.315+0000 7fa757fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fd9d74350 0x560fd9d901b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58702/0 (socket says 172.21.15.33:58702) 2024-03-19T03:00:50.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.315+0000 7fa757fff640 1 -- 172.21.15.33:0/388079036 learned_addr learned my addr 172.21.15.33:0/388079036 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:50.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.315+0000 7fa75ed9d640 1 --2- 172.21.15.33:0/388079036 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fd9d7cf30 0x560fd9d906f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.315+0000 7fa75e59c640 1 --2- 172.21.15.33:0/388079036 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fd9d73a30 0x560fd9d8fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.315+0000 7fa757fff640 1 -- 172.21.15.33:0/388079036 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fd9d73a30 msgr2=0x560fd9d8fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:50.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.315+0000 7fa757fff640 1 --2- 172.21.15.33:0/388079036 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fd9d73a30 0x560fd9d8fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.315+0000 7fa757fff640 1 -- 172.21.15.33:0/388079036 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fd9d7cf30 msgr2=0x560fd9d906f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:50.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.315+0000 7fa757fff640 1 --2- 172.21.15.33:0/388079036 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fd9d7cf30 0x560fd9d906f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.315+0000 7fa757fff640 1 -- 172.21.15.33:0/388079036 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fd9df1f10 con 0x560fd9d74350 2024-03-19T03:00:50.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.315+0000 7fa757fff640 1 --2- 172.21.15.33:0/388079036 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fd9d74350 0x560fd9d901b0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fa748009900 tx=0x7fa74803e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfa2e3893f87f402 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:50.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.315+0000 7fa745ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:00:50.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.315+0000 7fa745ffb640 1 -- 172.21.15.33:0/388079036 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa74805a070 con 0x560fd9d74350 2024-03-19T03:00:50.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.315+0000 7fa745ffb640 1 -- 172.21.15.33:0/388079036 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7fa748005040 con 0x560fd9d74350 2024-03-19T03:00:50.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.315+0000 7fa7675a0740 1 -- 172.21.15.33:0/388079036 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa758010a80 con 0x560fd9d74350 2024-03-19T03:00:50.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.315+0000 7fa745ffb640 1 -- 172.21.15.33:0/388079036 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa748066050 con 0x560fd9d74350 2024-03-19T03:00:50.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.315+0000 7fa7675a0740 10 client.17196 fetch_fsmap learned FSMap version 25 2024-03-19T03:00:50.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.315+0000 7fa7675a0740 1 -- 172.21.15.33:0/388079036 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x560fd9df3200 con 0x560fd9d74350 2024-03-19T03:00:50.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.316+0000 7fa745ffb640 1 -- 172.21.15.33:0/388079036 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa758010a80 con 0x560fd9d74350 2024-03-19T03:00:50.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.316+0000 7fa745ffb640 1 -- 172.21.15.33:0/388079036 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fa7480046c0 con 0x560fd9d74350 2024-03-19T03:00:50.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.316+0000 7fa7675a0740 10 client.17196 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:00:50.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.316+0000 7fa7675a0740 10 client.17196 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:00:50.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.316+0000 7fa7675a0740 20 client.17196 populate_metadata read hostname 'smithi033' 2024-03-19T03:00:50.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.316+0000 7fa7675a0740 1 --2- 172.21.15.33:0/388079036 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x560fd9df3960 0x560fd9df5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.316+0000 7fa7675a0740 4 client.17196 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T03:00:50.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.316+0000 7fa7675a0740 1 -- 172.21.15.33:0/388079036 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560fd9cded90 con 0x560fd9df3960 2024-03-19T03:00:50.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.316+0000 7fa75e59c640 1 --2- 172.21.15.33:0/388079036 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x560fd9df3960 0x560fd9df5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.316+0000 7fa745ffb640 10 client.17196 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:00:50.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.316+0000 7fa75e59c640 1 --2- 172.21.15.33:0/388079036 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x560fd9df3960 0x560fd9df5d80 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48404e8c4ac9a92d server_cookie=a60562bc3f1775d4 in_seq=0 out_seq=0 2024-03-19T03:00:50.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.316+0000 7fa745ffb640 1 -- 172.21.15.33:0/388079036 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560fd9cded90 con 0x560fd9df3960 2024-03-19T03:00:50.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.316+0000 7fa745ffb640 10 client.17196 handle_command_reply: tid=0 2024-03-19T03:00:50.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.316+0000 7fa7675a0740 1 client.17196 shutdown 2024-03-19T03:00:50.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.317+0000 7fa7675a0740 1 -- 172.21.15.33:0/388079036 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fd9d74350 msgr2=0x560fd9d901b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.317+0000 7fa7675a0740 1 --2- 172.21.15.33:0/388079036 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fd9d74350 0x560fd9d901b0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fa748009900 tx=0x7fa74803e000 comp rx=0 tx=0).stop 2024-03-19T03:00:50.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.317+0000 7fa7675a0740 1 -- 172.21.15.33:0/388079036 shutdown_connections 2024-03-19T03:00:50.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.317+0000 7fa7675a0740 1 --2- 172.21.15.33:0/388079036 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x560fd9df3960 0x560fd9df5d80 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.317+0000 7fa7675a0740 1 --2- 172.21.15.33:0/388079036 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560fd9d7cf30 0x560fd9d906f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.317+0000 7fa745ffb640 0 client.17196 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:00:50.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.317+0000 7fa7675a0740 1 --2- 172.21.15.33:0/388079036 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560fd9d74350 0x560fd9d901b0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.317+0000 7fa7675a0740 1 --2- 172.21.15.33:0/388079036 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560fd9d73a30 0x560fd9d8fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.317+0000 7fa7675a0740 1 -- 172.21.15.33:0/388079036 >> 172.21.15.33:0/388079036 conn(0x560fd9d6e160 msgr2=0x560fd9d78e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:50.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.317+0000 7fa7675a0740 1 -- 172.21.15.33:0/388079036 shutdown_connections 2024-03-19T03:00:50.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.317+0000 7fa7675a0740 1 -- 172.21.15.33:0/388079036 wait complete. 2024-03-19T03:00:50.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.317+0000 7fa7675a0740 20 client.17196 trim_cache size 0 max 16384 2024-03-19T03:00:50.326 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T03:00:50.326 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-03-19T03:00:50.327 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-03-19T03:00:50.327 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T03:00:50.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.317+0000 7fa7649b2640 1 -- 172.21.15.33:0/3625994634 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa74c0a2970 msgr2=0x7fa74c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.317+0000 7fa7649b2640 1 --2- 172.21.15.33:0/3625994634 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa74c0a2970 0x7fa74c0a4e30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa748005ab0 tx=0x7fa748005fd0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.317+0000 7fa7649b2640 1 -- 172.21.15.33:0/3625994634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7601017c0 msgr2=0x7fa76019a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.317+0000 7fa7649b2640 1 --2- 172.21.15.33:0/3625994634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7601017c0 0x7fa76019a250 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fa75000e9f0 tx=0x7fa75000eeb0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.317+0000 7fa7649b2640 1 -- 172.21.15.33:0/3625994634 shutdown_connections 2024-03-19T03:00:50.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.317+0000 7fa7649b2640 1 --2- 172.21.15.33:0/3625994634 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa74c0a2970 0x7fa74c0a4e30 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.317+0000 7fa7649b2640 1 --2- 172.21.15.33:0/3625994634 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa760102180 0x7fa76019a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.317+0000 7fa7649b2640 1 --2- 172.21.15.33:0/3625994634 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa7601077d0 0x7fa76019eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.317+0000 7fa7649b2640 1 --2- 172.21.15.33:0/3625994634 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa7601017c0 0x7fa76019a250 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.317+0000 7fa7649b2640 1 -- 172.21.15.33:0/3625994634 >> 172.21.15.33:0/3625994634 conn(0x7fa7600fd250 msgr2=0x7fa76010cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:50.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.317+0000 7fa7649b2640 1 -- 172.21.15.33:0/3625994634 shutdown_connections 2024-03-19T03:00:50.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.317+0000 7fa7649b2640 1 -- 172.21.15.33:0/3625994634 wait complete. 2024-03-19T03:00:50.329 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T03:00:50.330 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:00:50.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.431+0000 7f7e1afd3640 1 -- 172.21.15.33:0/3044169200 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e14102170 msgr2=0x7f7e1410e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.435 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.431+0000 7f7e1afd3640 1 --2- 172.21.15.33:0/3044169200 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e14102170 0x7f7e1410e680 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c009cd0 tx=0x7f7e0c02c0f0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.431+0000 7f7e1afd3640 1 -- 172.21.15.33:0/3044169200 shutdown_connections 2024-03-19T03:00:50.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.431+0000 7f7e1afd3640 1 --2- 172.21.15.33:0/3044169200 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e14102170 0x7f7e1410e680 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.431+0000 7f7e1afd3640 1 --2- 172.21.15.33:0/3044169200 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e141017b0 0x7f7e14101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.431+0000 7f7e1afd3640 1 --2- 172.21.15.33:0/3044169200 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e141077c0 0x7f7e14107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.431+0000 7f7e1afd3640 1 -- 172.21.15.33:0/3044169200 >> 172.21.15.33:0/3044169200 conn(0x7f7e140fd2c0 msgr2=0x7f7e140ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:50.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.431+0000 7f7e1afd3640 1 -- 172.21.15.33:0/3044169200 shutdown_connections 2024-03-19T03:00:50.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.431+0000 7f7e1afd3640 1 -- 172.21.15.33:0/3044169200 wait complete. 2024-03-19T03:00:50.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.432+0000 7f7e1afd3640 1 Processor -- start 2024-03-19T03:00:50.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.432+0000 7f7e1afd3640 1 -- start start 2024-03-19T03:00:50.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.432+0000 7f7e1afd3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e141017b0 0x7f7e14197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.432+0000 7f7e1afd3640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e14102170 0x7f7e14198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.432+0000 7f7e1afd3640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e141077c0 0x7f7e1419c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.432+0000 7f7e1afd3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7e14111a80 con 0x7f7e141017b0 2024-03-19T03:00:50.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.432+0000 7f7e1afd3640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7e14111900 con 0x7f7e141077c0 2024-03-19T03:00:50.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.432+0000 7f7e1afd3640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7e14111c00 con 0x7f7e14102170 2024-03-19T03:00:50.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.432+0000 7f7e18d6f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e141017b0 0x7f7e14197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.432+0000 7f7e18d6f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e141017b0 0x7f7e14197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58712/0 (socket says 172.21.15.33:58712) 2024-03-19T03:00:50.437 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.432+0000 7f7e18d6f640 1 -- 172.21.15.33:0/2512217822 learned_addr learned my addr 172.21.15.33:0/2512217822 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:50.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.432+0000 7f7e13fff640 1 --2- 172.21.15.33:0/2512217822 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e14102170 0x7f7e14198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.432+0000 7f7e19570640 1 --2- 172.21.15.33:0/2512217822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e141077c0 0x7f7e1419c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.432+0000 7f7e18d6f640 1 -- 172.21.15.33:0/2512217822 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e14102170 msgr2=0x7f7e14198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.432+0000 7f7e18d6f640 1 --2- 172.21.15.33:0/2512217822 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e14102170 0x7f7e14198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.432+0000 7f7e18d6f640 1 -- 172.21.15.33:0/2512217822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e141077c0 msgr2=0x7f7e1419c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.432+0000 7f7e18d6f640 1 --2- 172.21.15.33:0/2512217822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e141077c0 0x7f7e1419c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.432+0000 7f7e18d6f640 1 -- 172.21.15.33:0/2512217822 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e1419cfb0 con 0x7f7e141017b0 2024-03-19T03:00:50.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.432+0000 7f7e13fff640 1 --2- 172.21.15.33:0/2512217822 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e14102170 0x7f7e14198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:00:50.438 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.433+0000 7f7e18d6f640 1 --2- 172.21.15.33:0/2512217822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e141017b0 0x7f7e14197fc0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f7e0800e9f0 tx=0x7f7e0800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:50.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.433+0000 7f7e11ffb640 1 -- 172.21.15.33:0/2512217822 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e0800cd80 con 0x7f7e141017b0 2024-03-19T03:00:50.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.433+0000 7f7e1afd3640 1 -- 172.21.15.33:0/2512217822 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e1419d2a0 con 0x7f7e141017b0 2024-03-19T03:00:50.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.433+0000 7f7e11ffb640 1 -- 172.21.15.33:0/2512217822 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e080054d0 con 0x7f7e141017b0 2024-03-19T03:00:50.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.433+0000 7f7e11ffb640 1 -- 172.21.15.33:0/2512217822 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e08010490 con 0x7f7e141017b0 2024-03-19T03:00:50.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.433+0000 7f7e1afd3640 1 -- 172.21.15.33:0/2512217822 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e141a4b30 con 0x7f7e141017b0 2024-03-19T03:00:50.439 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.434+0000 7f7e1afd3640 1 -- 172.21.15.33:0/2512217822 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e14068a60 con 0x7f7e141017b0 2024-03-19T03:00:50.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.435+0000 7f7e11ffb640 1 -- 172.21.15.33:0/2512217822 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f7e08005020 con 0x7f7e141017b0 2024-03-19T03:00:50.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.435+0000 7f7e11ffb640 1 --2- 172.21.15.33:0/2512217822 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f7e040a2970 0x7f7e040a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.436+0000 7f7e13fff640 1 --2- 172.21.15.33:0/2512217822 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f7e040a2970 0x7f7e040a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.436+0000 7f7e11ffb640 1 -- 172.21.15.33:0/2512217822 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7f7e080cf620 con 0x7f7e141017b0 2024-03-19T03:00:50.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.436+0000 7f7e13fff640 1 --2- 172.21.15.33:0/2512217822 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f7e040a2970 0x7f7e040a4e30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7e140ff980 tx=0x7f7e00005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:50.441 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.440+0000 7f7e11ffb640 1 -- 172.21.15.33:0/2512217822 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7e0809bf50 con 0x7f7e141017b0 2024-03-19T03:00:50.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.664+0000 7f7e1afd3640 1 -- 172.21.15.33:0/2512217822 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e14199310 con 0x7f7e141017b0 2024-03-19T03:00:50.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.665+0000 7f7e11ffb640 1 -- 172.21.15.33:0/2512217822 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11867 (secure 0 0 0) 0x7f7e0809b850 con 0x7f7e141017b0 2024-03-19T03:00:50.666 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T03:00:50.666 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:50:52.201798+0000","modified":"2024-03-19T02:53:11.748697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24488,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/3731232169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":3731232169},{"type":"v1","addr":"172.21.15.96:6839","nonce":3731232169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24457,24488,24509]},"id":1}]} 2024-03-19T03:00:50.666 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-03-19T03:00:50.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.667+0000 7f7e1afd3640 1 -- 172.21.15.33:0/2512217822 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f7e040a2970 msgr2=0x7f7e040a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.667+0000 7f7e1afd3640 1 --2- 172.21.15.33:0/2512217822 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f7e040a2970 0x7f7e040a4e30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7e140ff980 tx=0x7f7e00005f90 comp rx=0 tx=0).stop 2024-03-19T03:00:50.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.667+0000 7f7e1afd3640 1 -- 172.21.15.33:0/2512217822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e141017b0 msgr2=0x7f7e14197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.667+0000 7f7e1afd3640 1 --2- 172.21.15.33:0/2512217822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e141017b0 0x7f7e14197fc0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f7e0800e9f0 tx=0x7f7e0800edf0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.667+0000 7f7e1afd3640 1 -- 172.21.15.33:0/2512217822 shutdown_connections 2024-03-19T03:00:50.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.667+0000 7f7e1afd3640 1 --2- 172.21.15.33:0/2512217822 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f7e040a2970 0x7f7e040a4e30 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.667+0000 7f7e1afd3640 1 --2- 172.21.15.33:0/2512217822 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e14102170 0x7f7e14198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.667+0000 7f7e1afd3640 1 --2- 172.21.15.33:0/2512217822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e141077c0 0x7f7e1419c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.667+0000 7f7e1afd3640 1 --2- 172.21.15.33:0/2512217822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e141017b0 0x7f7e14197fc0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.667+0000 7f7e1afd3640 1 -- 172.21.15.33:0/2512217822 >> 172.21.15.33:0/2512217822 conn(0x7f7e140fd2c0 msgr2=0x7f7e140fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:50.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.667+0000 7f7e1afd3640 1 -- 172.21.15.33:0/2512217822 shutdown_connections 2024-03-19T03:00:50.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.667+0000 7f7e1afd3640 1 -- 172.21.15.33:0/2512217822 wait complete. 2024-03-19T03:00:50.679 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T03:00:50.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.782+0000 7f64e0cbf640 1 -- 172.21.15.33:0/884265039 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f64dc101060 msgr2=0x7f64dc10d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.782+0000 7f64e0cbf640 1 --2- 172.21.15.33:0/884265039 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f64dc101060 0x7f64dc10d570 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f64cc009cd0 tx=0x7f64cc02c0f0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.782+0000 7f64e0cbf640 1 -- 172.21.15.33:0/884265039 shutdown_connections 2024-03-19T03:00:50.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.782+0000 7f64e0cbf640 1 --2- 172.21.15.33:0/884265039 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f64dc101060 0x7f64dc10d570 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.782+0000 7f64e0cbf640 1 --2- 172.21.15.33:0/884265039 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64dc1006a0 0x7f64dc100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.782+0000 7f64e0cbf640 1 --2- 172.21.15.33:0/884265039 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64dc1066b0 0x7f64dc106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.782+0000 7f64e0cbf640 1 -- 172.21.15.33:0/884265039 >> 172.21.15.33:0/884265039 conn(0x7f64dc0778d0 msgr2=0x7f64dc077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:50.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.783+0000 7f64e0cbf640 1 -- 172.21.15.33:0/884265039 shutdown_connections 2024-03-19T03:00:50.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.783+0000 7f64e0cbf640 1 -- 172.21.15.33:0/884265039 wait complete. 2024-03-19T03:00:50.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.783+0000 7f64e0cbf640 1 Processor -- start 2024-03-19T03:00:50.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.783+0000 7f64e0cbf640 1 -- start start 2024-03-19T03:00:50.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.783+0000 7f64e0cbf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f64dc1006a0 0x7f64dc196e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.783+0000 7f64e0cbf640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64dc101060 0x7f64dc1973d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.783+0000 7f64e0cbf640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64dc1066b0 0x7f64dc19b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.783+0000 7f64e0cbf640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc110970 con 0x7f64dc1006a0 2024-03-19T03:00:50.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.783+0000 7f64e0cbf640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc1107f0 con 0x7f64dc101060 2024-03-19T03:00:50.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.783+0000 7f64e0cbf640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc110af0 con 0x7f64dc1066b0 2024-03-19T03:00:50.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.784+0000 7f64da59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f64dc1006a0 0x7f64dc196e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.784+0000 7f64da59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f64dc1006a0 0x7f64dc196e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58742/0 (socket says 172.21.15.33:58742) 2024-03-19T03:00:50.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.784+0000 7f64da59c640 1 -- 172.21.15.33:0/307831904 learned_addr learned my addr 172.21.15.33:0/307831904 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:50.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.784+0000 7f64dad9d640 1 --2- 172.21.15.33:0/307831904 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64dc1066b0 0x7f64dc19b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.784+0000 7f64d9d9b640 1 --2- 172.21.15.33:0/307831904 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64dc101060 0x7f64dc1973d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.787 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.784+0000 7f64da59c640 1 -- 172.21.15.33:0/307831904 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64dc1066b0 msgr2=0x7f64dc19b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.784+0000 7f64da59c640 1 --2- 172.21.15.33:0/307831904 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64dc1066b0 0x7f64dc19b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.784+0000 7f64da59c640 1 -- 172.21.15.33:0/307831904 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64dc101060 msgr2=0x7f64dc1973d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.784+0000 7f64da59c640 1 --2- 172.21.15.33:0/307831904 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64dc101060 0x7f64dc1973d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.784+0000 7f64da59c640 1 -- 172.21.15.33:0/307831904 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64dc19be30 con 0x7f64dc1006a0 2024-03-19T03:00:50.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.784+0000 7f64da59c640 1 --2- 172.21.15.33:0/307831904 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f64dc1006a0 0x7f64dc196e90 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f64d400e9f0 tx=0x7f64d400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:50.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.784+0000 7f64d37fe640 1 -- 172.21.15.33:0/307831904 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d400cd80 con 0x7f64dc1006a0 2024-03-19T03:00:50.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.784+0000 7f64d37fe640 1 -- 172.21.15.33:0/307831904 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f64d40054d0 con 0x7f64dc1006a0 2024-03-19T03:00:50.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.784+0000 7f64e0cbf640 1 -- 172.21.15.33:0/307831904 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64dc19c120 con 0x7f64dc1006a0 2024-03-19T03:00:50.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.784+0000 7f64d37fe640 1 -- 172.21.15.33:0/307831904 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d4010490 con 0x7f64dc1006a0 2024-03-19T03:00:50.788 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.784+0000 7f64e0cbf640 1 -- 172.21.15.33:0/307831904 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64dc068980 con 0x7f64dc1006a0 2024-03-19T03:00:50.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.786+0000 7f64d37fe640 1 -- 172.21.15.33:0/307831904 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f64d4005020 con 0x7f64dc1006a0 2024-03-19T03:00:50.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.787+0000 7f64e3981740 1 Processor -- start 2024-03-19T03:00:50.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.787+0000 7f64e3981740 1 -- start start 2024-03-19T03:00:50.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.787+0000 7f64e3981740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c75a1732d0 0x55c75a1736b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.787+0000 7f64e3981740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c75a173bf0 0x55c75a17c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.787+0000 7f64e3981740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c75a17c7d0 0x55c75a17ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.787+0000 7f64e3981740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c75a0d64b0 con 0x55c75a17c7d0 2024-03-19T03:00:50.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.787+0000 7f64e3981740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c75a07ed70 con 0x55c75a173bf0 2024-03-19T03:00:50.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.787+0000 7f64e3981740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c75a0d2550 con 0x55c75a1732d0 2024-03-19T03:00:50.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.788+0000 7f64d37fe640 1 --2- 172.21.15.33:0/307831904 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f64c80a2970 0x7f64c80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.788+0000 7f64d9d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c75a17c7d0 0x55c75a17ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.788+0000 7f64dad9d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c75a1732d0 0x55c75a1736b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.788+0000 7f64da59c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c75a173bf0 0x55c75a17c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.788+0000 7f64d9d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c75a17c7d0 0x55c75a17ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58758/0 (socket says 172.21.15.33:58758) 2024-03-19T03:00:50.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.788+0000 7f64dad9d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c75a1732d0 0x55c75a1736b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:38018/0 (socket says 172.21.15.33:38018) 2024-03-19T03:00:50.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.788+0000 7f64d9d9b640 1 -- 172.21.15.33:0/3475803541 learned_addr learned my addr 172.21.15.33:0/3475803541 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:50.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.788+0000 7f64dad9d640 1 --2- 172.21.15.33:0/307831904 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f64c80a2970 0x7f64c80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.788+0000 7f64d9d9b640 1 -- 172.21.15.33:0/3475803541 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c75a1732d0 msgr2=0x55c75a1736b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:50.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.788+0000 7f64d9d9b640 1 --2- 172.21.15.33:0/3475803541 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c75a1732d0 0x55c75a1736b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.788+0000 7f64d37fe640 1 -- 172.21.15.33:0/307831904 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7f64d40cf540 con 0x7f64dc1006a0 2024-03-19T03:00:50.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.788+0000 7f64d9d9b640 1 -- 172.21.15.33:0/3475803541 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c75a173bf0 msgr2=0x55c75a17c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:50.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.788+0000 7f64d9d9b640 1 --2- 172.21.15.33:0/3475803541 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c75a173bf0 0x55c75a17c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.788+0000 7f64d9d9b640 1 -- 172.21.15.33:0/3475803541 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c75a0a1210 con 0x55c75a17c7d0 2024-03-19T03:00:50.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.788+0000 7f64d9d9b640 1 --2- 172.21.15.33:0/3475803541 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c75a17c7d0 0x55c75a17ebc0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f64c4009c90 tx=0x7f64c4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ef7526d22ac4dfb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:50.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.788+0000 7f64dad9d640 1 --2- 172.21.15.33:0/307831904 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f64c80a2970 0x7f64c80a4e30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f64cc005cc0 tx=0x7f64cc02c0f0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:50.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.788+0000 7f64d0ff9640 1 -- 172.21.15.33:0/3475803541 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c401e070 con 0x55c75a17c7d0 2024-03-19T03:00:50.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.788+0000 7f64d0ff9640 1 -- 172.21.15.33:0/3475803541 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f64c4019ae0 con 0x55c75a17c7d0 2024-03-19T03:00:50.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.788+0000 7f64d0ff9640 1 -- 172.21.15.33:0/3475803541 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c400e940 con 0x55c75a17c7d0 2024-03-19T03:00:50.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.789+0000 7f64e3981740 1 -- 172.21.15.33:0/3475803541 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c75a17c7d0 msgr2=0x55c75a17ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.789+0000 7f64e3981740 1 --2- 172.21.15.33:0/3475803541 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c75a17c7d0 0x55c75a17ebc0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f64c4009c90 tx=0x7f64c4011810 comp rx=0 tx=0).stop 2024-03-19T03:00:50.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.789+0000 7f64e3981740 1 -- 172.21.15.33:0/3475803541 shutdown_connections 2024-03-19T03:00:50.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.789+0000 7f64e3981740 1 --2- 172.21.15.33:0/3475803541 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c75a17c7d0 0x55c75a17ebc0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.789+0000 7f64e3981740 1 --2- 172.21.15.33:0/3475803541 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c75a173bf0 0x55c75a17c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.789+0000 7f64e3981740 1 --2- 172.21.15.33:0/3475803541 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c75a1732d0 0x55c75a1736b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.789+0000 7f64e3981740 1 -- 172.21.15.33:0/3475803541 >> 172.21.15.33:0/3475803541 conn(0x55c75a16da00 msgr2=0x55c75a16dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:50.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.789+0000 7f64e3981740 1 -- 172.21.15.33:0/3475803541 shutdown_connections 2024-03-19T03:00:50.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.789+0000 7f64e3981740 1 -- 172.21.15.33:0/3475803541 wait complete. 2024-03-19T03:00:50.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.789+0000 7f64e3981740 1 -- start start 2024-03-19T03:00:50.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.789+0000 7f64e3981740 1 Processor -- start 2024-03-19T03:00:50.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.790+0000 7f64e3981740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c75a1732d0 0x55c75a1797c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.790+0000 7f64e3981740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c75a173bf0 0x55c75a179d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.790+0000 7f64e3981740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c75a17c7d0 0x55c75a17a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.790+0000 7f64e3981740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55c75a17f5c0 con 0x55c75a173bf0 2024-03-19T03:00:50.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.790+0000 7f64e3981740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55c75a092310 con 0x55c75a1732d0 2024-03-19T03:00:50.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.790+0000 7f64e3981740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55c75a17f740 con 0x55c75a17c7d0 2024-03-19T03:00:50.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.790+0000 7f64e3981740 10 client.0 _ll_register_callbacks cb 0x55c75a0862d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:00:50.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.790+0000 7f64d9d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c75a173bf0 0x55c75a179d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.790+0000 7f64d9d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c75a173bf0 0x55c75a179d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58772/0 (socket says 172.21.15.33:58772) 2024-03-19T03:00:50.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.790+0000 7f64d9d9b640 1 -- 172.21.15.33:0/418126080 learned_addr learned my addr 172.21.15.33:0/418126080 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:50.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.790+0000 7f64da59c640 1 --2- 172.21.15.33:0/418126080 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c75a1732d0 0x55c75a1797c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.790+0000 7f64dad9d640 1 --2- 172.21.15.33:0/418126080 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c75a17c7d0 0x55c75a17a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.790+0000 7f64d9d9b640 1 -- 172.21.15.33:0/418126080 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c75a17c7d0 msgr2=0x55c75a17a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:50.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.790+0000 7f64d9d9b640 1 --2- 172.21.15.33:0/418126080 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c75a17c7d0 0x55c75a17a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.790+0000 7f64d9d9b640 1 -- 172.21.15.33:0/418126080 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c75a1732d0 msgr2=0x55c75a1797c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:50.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.790+0000 7f64d9d9b640 1 --2- 172.21.15.33:0/418126080 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c75a1732d0 0x55c75a1797c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.790+0000 7f64d9d9b640 1 -- 172.21.15.33:0/418126080 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c759dea3f0 con 0x55c75a173bf0 2024-03-19T03:00:50.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.790+0000 7f64da59c640 1 --2- 172.21.15.33:0/418126080 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c75a1732d0 0x55c75a1797c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:50.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.790+0000 7f64d9d9b640 1 --2- 172.21.15.33:0/418126080 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c75a173bf0 0x55c75a179d00 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f64c401c040 tx=0x7f64c4019a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b91dca91c14e36e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:50.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.790+0000 7f64c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:00:50.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.790+0000 7f64c1ffb640 1 -- 172.21.15.33:0/418126080 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c401e050 con 0x55c75a173bf0 2024-03-19T03:00:50.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.790+0000 7f64c1ffb640 1 -- 172.21.15.33:0/418126080 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7f64c4005250 con 0x55c75a173bf0 2024-03-19T03:00:50.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.790+0000 7f64e3981740 1 -- 172.21.15.33:0/418126080 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64c4019ae0 con 0x55c75a173bf0 2024-03-19T03:00:50.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.791+0000 7f64c1ffb640 1 -- 172.21.15.33:0/418126080 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f64c4032050 con 0x55c75a173bf0 2024-03-19T03:00:50.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.791+0000 7f64e3981740 10 client.17214 fetch_fsmap learned FSMap version 25 2024-03-19T03:00:50.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.791+0000 7f64e3981740 1 -- 172.21.15.33:0/418126080 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c759eb47d0 con 0x55c75a173bf0 2024-03-19T03:00:50.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.791+0000 7f64c1ffb640 1 -- 172.21.15.33:0/418126080 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f64c4019ae0 con 0x55c75a173bf0 2024-03-19T03:00:50.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.791+0000 7f64c1ffb640 1 -- 172.21.15.33:0/418126080 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f64c400e640 con 0x55c75a173bf0 2024-03-19T03:00:50.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.791+0000 7f64e3981740 10 client.17214 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:00:50.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.791+0000 7f64e3981740 10 client.17214 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T03:00:50.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.791+0000 7f64e3981740 20 client.17214 populate_metadata read hostname 'smithi033' 2024-03-19T03:00:50.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.791+0000 7f64e3981740 1 --2- 172.21.15.33:0/418126080 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55c75a18c9f0 0x55c75a18ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.791+0000 7f64e3981740 4 client.17214 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:00:50.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.791+0000 7f64e3981740 1 -- 172.21.15.33:0/418126080 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c75a1343e0 con 0x55c75a18c9f0 2024-03-19T03:00:50.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.791+0000 7f64da59c640 1 --2- 172.21.15.33:0/418126080 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55c75a18c9f0 0x55c75a18ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.792+0000 7f64da59c640 1 --2- 172.21.15.33:0/418126080 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55c75a18c9f0 0x55c75a18ede0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=69083c69134ce555 server_cookie=f1d2dfcbcf7ddf28 in_seq=0 out_seq=0 2024-03-19T03:00:50.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.792+0000 7f64c1ffb640 10 client.17214 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T03:00:50.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.792+0000 7f64c1ffb640 1 -- 172.21.15.33:0/418126080 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c75a1343e0 con 0x55c75a18c9f0 2024-03-19T03:00:50.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.792+0000 7f64c1ffb640 10 client.17214 handle_command_reply: tid=0 2024-03-19T03:00:50.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.792+0000 7f64e3981740 1 client.17214 shutdown 2024-03-19T03:00:50.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.792+0000 7f64e3981740 1 -- 172.21.15.33:0/418126080 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c75a173bf0 msgr2=0x55c75a179d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.792+0000 7f64e3981740 1 --2- 172.21.15.33:0/418126080 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c75a173bf0 0x55c75a179d00 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f64c401c040 tx=0x7f64c4019a30 comp rx=0 tx=0).stop 2024-03-19T03:00:50.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.792+0000 7f64e3981740 1 -- 172.21.15.33:0/418126080 shutdown_connections 2024-03-19T03:00:50.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.792+0000 7f64e3981740 1 --2- 172.21.15.33:0/418126080 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55c75a18c9f0 0x55c75a18ede0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.792+0000 7f64e3981740 1 --2- 172.21.15.33:0/418126080 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c75a17c7d0 0x55c75a17a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.792+0000 7f64c1ffb640 0 client.17214 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T03:00:50.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.792+0000 7f64e3981740 1 --2- 172.21.15.33:0/418126080 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c75a173bf0 0x55c75a179d00 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.792+0000 7f64e3981740 1 --2- 172.21.15.33:0/418126080 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c75a1732d0 0x55c75a1797c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.792+0000 7f64e3981740 1 -- 172.21.15.33:0/418126080 >> 172.21.15.33:0/418126080 conn(0x55c75a16da00 msgr2=0x55c75a16f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:50.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.793+0000 7f64e3981740 1 -- 172.21.15.33:0/418126080 shutdown_connections 2024-03-19T03:00:50.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.793+0000 7f64e3981740 1 -- 172.21.15.33:0/418126080 wait complete. 2024-03-19T03:00:50.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.793+0000 7f64e3981740 20 client.17214 trim_cache size 0 max 16384 2024-03-19T03:00:50.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.806+0000 7f64e3981740 1 Processor -- start 2024-03-19T03:00:50.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.806+0000 7f64e3981740 1 -- start start 2024-03-19T03:00:50.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.806+0000 7f64e3981740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c75a1732d0 0x55c75a17bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.806+0000 7f64e3981740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c75a173bf0 0x55c75a188360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.806+0000 7f64e3981740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c75a17c7d0 0x55c75a16f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.806+0000 7f64e3981740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc1b14d0 con 0x55c75a173bf0 2024-03-19T03:00:50.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.806+0000 7f64e3981740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc1aeda0 con 0x55c75a17c7d0 2024-03-19T03:00:50.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.806+0000 7f64e3981740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc1acc10 con 0x55c75a1732d0 2024-03-19T03:00:50.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.806+0000 7f64d9d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c75a173bf0 0x55c75a188360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.806+0000 7f64d9d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c75a173bf0 0x55c75a188360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58776/0 (socket says 172.21.15.33:58776) 2024-03-19T03:00:50.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.806+0000 7f64d9d9b640 1 -- 172.21.15.33:0/2144933355 learned_addr learned my addr 172.21.15.33:0/2144933355 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:50.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.806+0000 7f64dad9d640 1 --2- 172.21.15.33:0/2144933355 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c75a17c7d0 0x55c75a16f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.806+0000 7f64da59c640 1 --2- 172.21.15.33:0/2144933355 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c75a1732d0 0x55c75a17bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.806+0000 7f64d9d9b640 1 -- 172.21.15.33:0/2144933355 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c75a1732d0 msgr2=0x55c75a17bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:50.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.806+0000 7f64d9d9b640 1 --2- 172.21.15.33:0/2144933355 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c75a1732d0 0x55c75a17bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.806+0000 7f64d9d9b640 1 -- 172.21.15.33:0/2144933355 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c75a17c7d0 msgr2=0x55c75a16f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:50.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.806+0000 7f64d9d9b640 1 --2- 172.21.15.33:0/2144933355 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c75a17c7d0 0x55c75a16f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.806+0000 7f64d9d9b640 1 -- 172.21.15.33:0/2144933355 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c75a143880 con 0x55c75a173bf0 2024-03-19T03:00:50.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.806+0000 7f64dad9d640 1 --2- 172.21.15.33:0/2144933355 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c75a17c7d0 0x55c75a16f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:00:50.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.806+0000 7f64d9d9b640 1 --2- 172.21.15.33:0/2144933355 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c75a173bf0 0x55c75a188360 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f64c4009be0 tx=0x7f64c400e4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8efef6134da17a5c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:50.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.806+0000 7f64c3fff640 1 -- 172.21.15.33:0/2144933355 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c401e040 con 0x55c75a173bf0 2024-03-19T03:00:50.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.806+0000 7f64c3fff640 1 -- 172.21.15.33:0/2144933355 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f64c401f040 con 0x55c75a173bf0 2024-03-19T03:00:50.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.807+0000 7f64c3fff640 1 -- 172.21.15.33:0/2144933355 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c400e5d0 con 0x55c75a173bf0 2024-03-19T03:00:50.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.807+0000 7f64e3981740 1 -- 172.21.15.33:0/2144933355 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c75a173bf0 msgr2=0x55c75a188360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.807+0000 7f64e3981740 1 --2- 172.21.15.33:0/2144933355 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c75a173bf0 0x55c75a188360 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f64c4009be0 tx=0x7f64c400e4f0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.807+0000 7f64e3981740 1 -- 172.21.15.33:0/2144933355 shutdown_connections 2024-03-19T03:00:50.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.807+0000 7f64e3981740 1 --2- 172.21.15.33:0/2144933355 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c75a17c7d0 0x55c75a16f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.807+0000 7f64e3981740 1 --2- 172.21.15.33:0/2144933355 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c75a173bf0 0x55c75a188360 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.807+0000 7f64e3981740 1 --2- 172.21.15.33:0/2144933355 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c75a1732d0 0x55c75a17bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.807+0000 7f64e3981740 1 -- 172.21.15.33:0/2144933355 >> 172.21.15.33:0/2144933355 conn(0x55c75a16da00 msgr2=0x55c75a16ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:50.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.807+0000 7f64e3981740 1 -- 172.21.15.33:0/2144933355 shutdown_connections 2024-03-19T03:00:50.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.807+0000 7f64e3981740 1 -- 172.21.15.33:0/2144933355 wait complete. 2024-03-19T03:00:50.814 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.807+0000 7f64e3981740 1 -- start start 2024-03-19T03:00:50.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.807+0000 7f64e3981740 1 Processor -- start 2024-03-19T03:00:50.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.808+0000 7f64e3981740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c75a1732d0 0x55c75a1848d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.808+0000 7f64e3981740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c75a173bf0 0x55c75a17fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.808+0000 7f64e3981740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c75a17c7d0 0x55c75a1802a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.808+0000 7f64e3981740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc1ad260 con 0x55c75a1732d0 2024-03-19T03:00:50.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.808+0000 7f64e3981740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc1ae6d0 con 0x55c75a17c7d0 2024-03-19T03:00:50.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.808+0000 7f64e3981740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc1ad930 con 0x55c75a173bf0 2024-03-19T03:00:50.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.808+0000 7f64e3981740 10 client.0 _ll_register_callbacks cb 0x7f64c401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:00:50.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.808+0000 7f64da59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c75a1732d0 0x55c75a1848d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.808+0000 7f64da59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c75a1732d0 0x55c75a1848d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58790/0 (socket says 172.21.15.33:58790) 2024-03-19T03:00:50.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.808+0000 7f64da59c640 1 -- 172.21.15.33:0/2653420849 learned_addr learned my addr 172.21.15.33:0/2653420849 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:50.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.808+0000 7f64d9d9b640 1 --2- 172.21.15.33:0/2653420849 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c75a173bf0 0x55c75a17fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.808+0000 7f64dad9d640 1 --2- 172.21.15.33:0/2653420849 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c75a17c7d0 0x55c75a1802a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.808+0000 7f64da59c640 1 -- 172.21.15.33:0/2653420849 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c75a173bf0 msgr2=0x55c75a17fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:50.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.808+0000 7f64da59c640 1 --2- 172.21.15.33:0/2653420849 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c75a173bf0 0x55c75a17fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.808+0000 7f64da59c640 1 -- 172.21.15.33:0/2653420849 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c75a17c7d0 msgr2=0x55c75a1802a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:50.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.808+0000 7f64da59c640 1 --2- 172.21.15.33:0/2653420849 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c75a17c7d0 0x55c75a1802a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.808+0000 7f64da59c640 1 -- 172.21.15.33:0/2653420849 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c75a1807e0 con 0x55c75a1732d0 2024-03-19T03:00:50.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.808+0000 7f64d9d9b640 1 --2- 172.21.15.33:0/2653420849 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c75a173bf0 0x55c75a17fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:00:50.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.808+0000 7f64dad9d640 1 --2- 172.21.15.33:0/2653420849 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c75a17c7d0 0x55c75a1802a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:00:50.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.808+0000 7f64da59c640 1 --2- 172.21.15.33:0/2653420849 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c75a1732d0 0x55c75a1848d0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f64d409b4b0 tx=0x7f64d409b6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55672c19fb656bb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:50.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.808+0000 7f64c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:00:50.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.808+0000 7f64c1ffb640 1 -- 172.21.15.33:0/2653420849 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d409f900 con 0x55c75a1732d0 2024-03-19T03:00:50.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.808+0000 7f64c1ffb640 1 -- 172.21.15.33:0/2653420849 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7f64d40860b0 con 0x55c75a1732d0 2024-03-19T03:00:50.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.808+0000 7f64e3981740 1 -- 172.21.15.33:0/2653420849 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64c401f040 con 0x55c75a1732d0 2024-03-19T03:00:50.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.809+0000 7f64c1ffb640 1 -- 172.21.15.33:0/2653420849 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f64d40a3f50 con 0x55c75a1732d0 2024-03-19T03:00:50.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.809+0000 7f64e3981740 10 client.17220 fetch_fsmap learned FSMap version 25 2024-03-19T03:00:50.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.809+0000 7f64e3981740 1 -- 172.21.15.33:0/2653420849 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c75a1f28e0 con 0x55c75a1732d0 2024-03-19T03:00:50.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.809+0000 7f64c1ffb640 1 -- 172.21.15.33:0/2653420849 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f64c401f040 con 0x55c75a1732d0 2024-03-19T03:00:50.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.809+0000 7f64c1ffb640 1 -- 172.21.15.33:0/2653420849 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f64d40a3ac0 con 0x55c75a1732d0 2024-03-19T03:00:50.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.809+0000 7f64e3981740 10 client.17220 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:00:50.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.809+0000 7f64e3981740 10 client.17220 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T03:00:50.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.809+0000 7f64e3981740 20 client.17220 populate_metadata read hostname 'smithi033' 2024-03-19T03:00:50.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.809+0000 7f64e3981740 1 --2- 172.21.15.33:0/2653420849 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55c75a1f2fc0 0x55c75a1f53e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.809+0000 7f64e3981740 4 client.17220 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T03:00:50.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.809+0000 7f64e3981740 1 -- 172.21.15.33:0/2653420849 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c75a0e39f0 con 0x55c75a1f2fc0 2024-03-19T03:00:50.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.809+0000 7f64d9d9b640 1 --2- 172.21.15.33:0/2653420849 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55c75a1f2fc0 0x55c75a1f53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.809+0000 7f64c1ffb640 10 client.17220 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T03:00:50.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.809+0000 7f64d9d9b640 1 --2- 172.21.15.33:0/2653420849 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55c75a1f2fc0 0x55c75a1f53e0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b9f377a8569e9c23 server_cookie=958af057d328e660 in_seq=0 out_seq=0 2024-03-19T03:00:50.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.810+0000 7f64c1ffb640 1 -- 172.21.15.33:0/2653420849 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c75a0e39f0 con 0x55c75a1f2fc0 2024-03-19T03:00:50.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.810+0000 7f64c1ffb640 10 client.17220 handle_command_reply: tid=0 2024-03-19T03:00:50.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.810+0000 7f64e3981740 1 client.17220 shutdown 2024-03-19T03:00:50.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.810+0000 7f64e3981740 1 -- 172.21.15.33:0/2653420849 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c75a1732d0 msgr2=0x55c75a1848d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.817 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.810+0000 7f64e3981740 1 --2- 172.21.15.33:0/2653420849 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c75a1732d0 0x55c75a1848d0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f64d409b4b0 tx=0x7f64d409b6b0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.810+0000 7f64e3981740 1 -- 172.21.15.33:0/2653420849 shutdown_connections 2024-03-19T03:00:50.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.810+0000 7f64e3981740 1 --2- 172.21.15.33:0/2653420849 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55c75a1f2fc0 0x55c75a1f53e0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.810+0000 7f64e3981740 1 --2- 172.21.15.33:0/2653420849 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55c75a173bf0 0x55c75a17fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.810+0000 7f64e3981740 1 --2- 172.21.15.33:0/2653420849 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55c75a17c7d0 0x55c75a1802a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.810+0000 7f64e3981740 1 --2- 172.21.15.33:0/2653420849 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55c75a1732d0 0x55c75a1848d0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.810+0000 7f64c1ffb640 0 client.17220 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T03:00:50.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.810+0000 7f64e3981740 1 -- 172.21.15.33:0/2653420849 >> 172.21.15.33:0/2653420849 conn(0x55c75a16da00 msgr2=0x55c75a16f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:50.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.810+0000 7f64e3981740 1 -- 172.21.15.33:0/2653420849 shutdown_connections 2024-03-19T03:00:50.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.810+0000 7f64e3981740 1 -- 172.21.15.33:0/2653420849 wait complete. 2024-03-19T03:00:50.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.810+0000 7f64e3981740 20 client.17220 trim_cache size 0 max 16384 2024-03-19T03:00:50.818 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T03:00:50.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.810+0000 7f64e0cbf640 1 -- 172.21.15.33:0/307831904 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f64c80a2970 msgr2=0x7f64c80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.810+0000 7f64e0cbf640 1 --2- 172.21.15.33:0/307831904 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f64c80a2970 0x7f64c80a4e30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f64cc005cc0 tx=0x7f64cc02c0f0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.810+0000 7f64e0cbf640 1 -- 172.21.15.33:0/307831904 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f64dc1006a0 msgr2=0x7f64dc196e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.810+0000 7f64e0cbf640 1 --2- 172.21.15.33:0/307831904 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f64dc1006a0 0x7f64dc196e90 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f64d400e9f0 tx=0x7f64d400edf0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.810+0000 7f64e0cbf640 1 -- 172.21.15.33:0/307831904 shutdown_connections 2024-03-19T03:00:50.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.810+0000 7f64e0cbf640 1 --2- 172.21.15.33:0/307831904 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f64c80a2970 0x7f64c80a4e30 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.810+0000 7f64e0cbf640 1 --2- 172.21.15.33:0/307831904 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64dc1066b0 0x7f64dc19b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.810+0000 7f64e0cbf640 1 --2- 172.21.15.33:0/307831904 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f64dc101060 0x7f64dc1973d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.810+0000 7f64e0cbf640 1 --2- 172.21.15.33:0/307831904 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f64dc1006a0 0x7f64dc196e90 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.810+0000 7f64e0cbf640 1 -- 172.21.15.33:0/307831904 >> 172.21.15.33:0/307831904 conn(0x7f64dc0778d0 msgr2=0x7f64dc102f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:50.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.811+0000 7f64e0cbf640 1 -- 172.21.15.33:0/307831904 shutdown_connections 2024-03-19T03:00:50.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.811+0000 7f64e0cbf640 1 -- 172.21.15.33:0/307831904 wait complete. 2024-03-19T03:00:50.822 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T03:00:50.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.926+0000 7fd0cfc0b640 1 -- 172.21.15.33:0/2984820639 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0c8101060 msgr2=0x7fd0c810d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.926+0000 7fd0cfc0b640 1 --2- 172.21.15.33:0/2984820639 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0c8101060 0x7fd0c810d570 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fd0c0009cd0 tx=0x7fd0c002c0f0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.927+0000 7fd0cfc0b640 1 -- 172.21.15.33:0/2984820639 shutdown_connections 2024-03-19T03:00:50.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.927+0000 7fd0cfc0b640 1 --2- 172.21.15.33:0/2984820639 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0c81006a0 0x7fd0c8100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.927+0000 7fd0cfc0b640 1 --2- 172.21.15.33:0/2984820639 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0c8101060 0x7fd0c810d570 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.927+0000 7fd0cfc0b640 1 --2- 172.21.15.33:0/2984820639 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0c81066b0 0x7fd0c8106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.927+0000 7fd0cfc0b640 1 -- 172.21.15.33:0/2984820639 >> 172.21.15.33:0/2984820639 conn(0x7fd0c80778d0 msgr2=0x7fd0c8077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:50.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.927+0000 7fd0cfc0b640 1 -- 172.21.15.33:0/2984820639 shutdown_connections 2024-03-19T03:00:50.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.927+0000 7fd0cfc0b640 1 -- 172.21.15.33:0/2984820639 wait complete. 2024-03-19T03:00:50.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.927+0000 7fd0cfc0b640 1 Processor -- start 2024-03-19T03:00:50.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.927+0000 7fd0cfc0b640 1 -- start start 2024-03-19T03:00:50.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.927+0000 7fd0cfc0b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0c81006a0 0x7fd0c819a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.927+0000 7fd0cfc0b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0c8101060 0x7fd0c819a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.927+0000 7fd0cfc0b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0c81066b0 0x7fd0c819ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.927+0000 7fd0cfc0b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd0c8111a70 con 0x7fd0c81006a0 2024-03-19T03:00:50.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.927+0000 7fd0cfc0b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd0c81118f0 con 0x7fd0c8101060 2024-03-19T03:00:50.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.927+0000 7fd0cfc0b640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd0c8111bf0 con 0x7fd0c81066b0 2024-03-19T03:00:50.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.928+0000 7fd0cd9a7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0c81006a0 0x7fd0c819a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.928+0000 7fd0cd1a6640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0c8101060 0x7fd0c819a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.928+0000 7fd0cd9a7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0c81006a0 0x7fd0c819a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58822/0 (socket says 172.21.15.33:58822) 2024-03-19T03:00:50.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.928+0000 7fd0cd9a7640 1 -- 172.21.15.33:0/2393948878 learned_addr learned my addr 172.21.15.33:0/2393948878 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:50.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.928+0000 7fd0cd1a6640 1 --2- 172.21.15.33:0/2393948878 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0c8101060 0x7fd0c819a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:35950/0 (socket says 172.21.15.33:35950) 2024-03-19T03:00:50.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.928+0000 7fd0ce1a8640 1 --2- 172.21.15.33:0/2393948878 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0c81066b0 0x7fd0c819ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.928+0000 7fd0cd9a7640 1 -- 172.21.15.33:0/2393948878 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0c81066b0 msgr2=0x7fd0c819ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.928+0000 7fd0cd9a7640 1 --2- 172.21.15.33:0/2393948878 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0c81066b0 0x7fd0c819ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.928+0000 7fd0cd9a7640 1 -- 172.21.15.33:0/2393948878 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0c8101060 msgr2=0x7fd0c819a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.928+0000 7fd0cd9a7640 1 --2- 172.21.15.33:0/2393948878 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0c8101060 0x7fd0c819a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.928+0000 7fd0cd9a7640 1 -- 172.21.15.33:0/2393948878 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0c819f140 con 0x7fd0c81006a0 2024-03-19T03:00:50.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.928+0000 7fd0cd1a6640 1 --2- 172.21.15.33:0/2393948878 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0c8101060 0x7fd0c819a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:50.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.928+0000 7fd0cd9a7640 1 --2- 172.21.15.33:0/2393948878 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0c81006a0 0x7fd0c819a150 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fd0c400e9f0 tx=0x7fd0c400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:50.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.928+0000 7fd0beffd640 1 -- 172.21.15.33:0/2393948878 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c400cd80 con 0x7fd0c81006a0 2024-03-19T03:00:50.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.928+0000 7fd0beffd640 1 -- 172.21.15.33:0/2393948878 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd0c40108e0 con 0x7fd0c81006a0 2024-03-19T03:00:50.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.928+0000 7fd0beffd640 1 -- 172.21.15.33:0/2393948878 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c4018920 con 0x7fd0c81006a0 2024-03-19T03:00:50.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.928+0000 7fd0cfc0b640 1 -- 172.21.15.33:0/2393948878 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0c819f430 con 0x7fd0c81006a0 2024-03-19T03:00:50.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.928+0000 7fd0cfc0b640 1 -- 172.21.15.33:0/2393948878 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0c81a6cc0 con 0x7fd0c81006a0 2024-03-19T03:00:50.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.931+0000 7fd0beffd640 1 -- 172.21.15.33:0/2393948878 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fd0c4010a90 con 0x7fd0c81006a0 2024-03-19T03:00:50.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.931+0000 7fd0d27f5740 1 Processor -- start 2024-03-19T03:00:50.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.931+0000 7fd0d27f5740 1 -- start start 2024-03-19T03:00:50.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.931+0000 7fd0d27f5740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bcadfba2d0 0x55bcadfba6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.931+0000 7fd0d27f5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcadfbabf0 0x55bcadfc3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.931+0000 7fd0d27f5740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcadfc37d0 0x55bcadfc5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.931+0000 7fd0d27f5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55bcadf1d4b0 con 0x55bcadfbabf0 2024-03-19T03:00:50.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.931+0000 7fd0d27f5740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55bcadec5d70 con 0x55bcadfba2d0 2024-03-19T03:00:50.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.931+0000 7fd0d27f5740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55bcadf19550 con 0x55bcadfc37d0 2024-03-19T03:00:50.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.931+0000 7fd0cd9a7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcadfbabf0 0x55bcadfc3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.931+0000 7fd0ce1a8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bcadfba2d0 0x55bcadfba6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.931+0000 7fd0cd9a7640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcadfbabf0 0x55bcadfc3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58830/0 (socket says 172.21.15.33:58830) 2024-03-19T03:00:50.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.931+0000 7fd0beffd640 1 --2- 172.21.15.33:0/2393948878 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd0b40a2970 0x7fd0b40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.931+0000 7fd0cd9a7640 1 -- 172.21.15.33:0/2988617 learned_addr learned my addr 172.21.15.33:0/2988617 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:50.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.931+0000 7fd0cd1a6640 1 --2- 172.21.15.33:0/2988617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcadfc37d0 0x55bcadfc5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.932+0000 7fd0ce1a8640 1 --2- 172.21.15.33:0/2393948878 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd0b40a2970 0x7fd0b40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.932+0000 7fd0beffd640 1 -- 172.21.15.33:0/2393948878 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7fd0c40cf4e0 con 0x7fd0c81006a0 2024-03-19T03:00:50.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.932+0000 7fd0cd1a6640 1 -- 172.21.15.33:0/2988617 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bcadfba2d0 msgr2=0x55bcadfba6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:50.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.932+0000 7fd0cd1a6640 1 --2- 172.21.15.33:0/2988617 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bcadfba2d0 0x55bcadfba6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.932+0000 7fd0cd1a6640 1 -- 172.21.15.33:0/2988617 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcadfbabf0 msgr2=0x55bcadfc3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:50.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.932+0000 7fd0cd1a6640 1 --2- 172.21.15.33:0/2988617 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcadfbabf0 0x55bcadfc3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.932+0000 7fd0cd1a6640 1 -- 172.21.15.33:0/2988617 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcadee8210 con 0x55bcadfc37d0 2024-03-19T03:00:50.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.932+0000 7fd0cd9a7640 1 --2- 172.21.15.33:0/2988617 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcadfbabf0 0x55bcadfc3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:50.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.932+0000 7fd0cd1a6640 1 --2- 172.21.15.33:0/2988617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcadfc37d0 0x55bcadfc5bc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd0b8009fa0 tx=0x7fd0b80118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=296755b5d8bb54bb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:50.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.932+0000 7fd0ce1a8640 1 --2- 172.21.15.33:0/2393948878 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd0b40a2970 0x7fd0b40a4e30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd0c0005cc0 tx=0x7fd0c002c0f0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:50.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.932+0000 7fd0b3fff640 1 -- 172.21.15.33:0/2988617 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0b8004030 con 0x55bcadfc37d0 2024-03-19T03:00:50.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.932+0000 7fd0b3fff640 1 -- 172.21.15.33:0/2988617 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd0b80054d0 con 0x55bcadfc37d0 2024-03-19T03:00:50.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.932+0000 7fd0b3fff640 1 -- 172.21.15.33:0/2988617 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0b8004c10 con 0x55bcadfc37d0 2024-03-19T03:00:50.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.933+0000 7fd0d27f5740 1 -- 172.21.15.33:0/2988617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcadfc37d0 msgr2=0x55bcadfc5bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.933+0000 7fd0d27f5740 1 --2- 172.21.15.33:0/2988617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcadfc37d0 0x55bcadfc5bc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd0b8009fa0 tx=0x7fd0b80118d0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.933+0000 7fd0d27f5740 1 -- 172.21.15.33:0/2988617 shutdown_connections 2024-03-19T03:00:50.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.933+0000 7fd0d27f5740 1 --2- 172.21.15.33:0/2988617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcadfc37d0 0x55bcadfc5bc0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.933+0000 7fd0d27f5740 1 --2- 172.21.15.33:0/2988617 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcadfbabf0 0x55bcadfc3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.933+0000 7fd0d27f5740 1 --2- 172.21.15.33:0/2988617 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bcadfba2d0 0x55bcadfba6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.933+0000 7fd0d27f5740 1 -- 172.21.15.33:0/2988617 >> 172.21.15.33:0/2988617 conn(0x55bcadfb4a00 msgr2=0x55bcadfb4de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:50.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.933+0000 7fd0d27f5740 1 -- 172.21.15.33:0/2988617 shutdown_connections 2024-03-19T03:00:50.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.933+0000 7fd0d27f5740 1 -- 172.21.15.33:0/2988617 wait complete. 2024-03-19T03:00:50.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.933+0000 7fd0d27f5740 1 -- start start 2024-03-19T03:00:50.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.933+0000 7fd0d27f5740 1 Processor -- start 2024-03-19T03:00:50.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.934+0000 7fd0d27f5740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bcadfba2d0 0x55bcadfc07e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.934+0000 7fd0d27f5740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcadfbabf0 0x55bcadfc0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.934+0000 7fd0d27f5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcadfc37d0 0x55bcadfc1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.934+0000 7fd0d27f5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55bcadfc65c0 con 0x55bcadfc37d0 2024-03-19T03:00:50.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.934+0000 7fd0d27f5740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55bcaded9310 con 0x55bcadfba2d0 2024-03-19T03:00:50.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.934+0000 7fd0d27f5740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55bcadfc6740 con 0x55bcadfbabf0 2024-03-19T03:00:50.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.934+0000 7fd0d27f5740 10 client.0 _ll_register_callbacks cb 0x55bcadecd2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:00:50.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.934+0000 7fd0ce1a8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcadfc37d0 0x55bcadfc1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.934+0000 7fd0cd9a7640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bcadfba2d0 0x55bcadfc07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.934+0000 7fd0ce1a8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcadfc37d0 0x55bcadfc1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58840/0 (socket says 172.21.15.33:58840) 2024-03-19T03:00:50.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.934+0000 7fd0cd1a6640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcadfbabf0 0x55bcadfc0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.934+0000 7fd0ce1a8640 1 -- 172.21.15.33:0/2862011681 learned_addr learned my addr 172.21.15.33:0/2862011681 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:50.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.934+0000 7fd0cd9a7640 1 -- 172.21.15.33:0/2862011681 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcadfbabf0 msgr2=0x55bcadfc0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:50.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.934+0000 7fd0cd9a7640 1 --2- 172.21.15.33:0/2862011681 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcadfbabf0 0x55bcadfc0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.934+0000 7fd0cd9a7640 1 -- 172.21.15.33:0/2862011681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcadfc37d0 msgr2=0x55bcadfc1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:50.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.934+0000 7fd0cd9a7640 1 --2- 172.21.15.33:0/2862011681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcadfc37d0 0x55bcadfc1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.934+0000 7fd0cd9a7640 1 -- 172.21.15.33:0/2862011681 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcadc313f0 con 0x55bcadfba2d0 2024-03-19T03:00:50.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.934+0000 7fd0ce1a8640 1 --2- 172.21.15.33:0/2862011681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcadfc37d0 0x55bcadfc1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:50.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.934+0000 7fd0cd9a7640 1 --2- 172.21.15.33:0/2862011681 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bcadfba2d0 0x55bcadfc07e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd0c4097c80 tx=0x7fd0c4097b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31159cd43f8210e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:50.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.934+0000 7fd0b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T03:00:50.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.934+0000 7fd0b17fa640 1 -- 172.21.15.33:0/2862011681 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c409f8c0 con 0x55bcadfba2d0 2024-03-19T03:00:50.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.934+0000 7fd0d27f5740 1 -- 172.21.15.33:0/2862011681 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0b80054d0 con 0x55bcadfba2d0 2024-03-19T03:00:50.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.934+0000 7fd0b17fa640 1 -- 172.21.15.33:0/2862011681 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7fd0c4086070 con 0x55bcadfba2d0 2024-03-19T03:00:50.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.935+0000 7fd0b17fa640 1 -- 172.21.15.33:0/2862011681 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd0c40a3f10 con 0x55bcadfba2d0 2024-03-19T03:00:50.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.935+0000 7fd0d27f5740 10 client.27053 fetch_fsmap learned FSMap version 25 2024-03-19T03:00:50.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.935+0000 7fd0d27f5740 1 -- 172.21.15.33:0/2862011681 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55bcadcfb7d0 con 0x55bcadfba2d0 2024-03-19T03:00:50.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.935+0000 7fd0b17fa640 1 -- 172.21.15.33:0/2862011681 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd0b80054d0 con 0x55bcadfba2d0 2024-03-19T03:00:50.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.935+0000 7fd0b17fa640 1 -- 172.21.15.33:0/2862011681 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fd0c4010430 con 0x55bcadfba2d0 2024-03-19T03:00:50.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.935+0000 7fd0d27f5740 10 client.27053 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:00:50.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.935+0000 7fd0d27f5740 10 client.27053 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T03:00:50.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.935+0000 7fd0d27f5740 20 client.27053 populate_metadata read hostname 'smithi033' 2024-03-19T03:00:50.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.935+0000 7fd0d27f5740 1 --2- 172.21.15.33:0/2862011681 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55bcadfd3ad0 0x55bcadfd5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.935+0000 7fd0d27f5740 4 client.27053 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:00:50.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.935+0000 7fd0d27f5740 1 -- 172.21.15.33:0/2862011681 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bcadf7b3e0 con 0x55bcadfd3ad0 2024-03-19T03:00:50.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.935+0000 7fd0cd1a6640 1 --2- 172.21.15.33:0/2862011681 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55bcadfd3ad0 0x55bcadfd5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.936+0000 7fd0b17fa640 10 client.27053 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T03:00:50.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.936+0000 7fd0cd1a6640 1 --2- 172.21.15.33:0/2862011681 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55bcadfd3ad0 0x55bcadfd5ec0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7688c15abe7b3723 server_cookie=6b880648b08aa100 in_seq=0 out_seq=0 2024-03-19T03:00:50.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.936+0000 7fd0b17fa640 1 -- 172.21.15.33:0/2862011681 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bcadf7b3e0 con 0x55bcadfd3ad0 2024-03-19T03:00:50.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.936+0000 7fd0b17fa640 10 client.27053 handle_command_reply: tid=0 2024-03-19T03:00:50.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.936+0000 7fd0d27f5740 1 client.27053 shutdown 2024-03-19T03:00:50.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.936+0000 7fd0d27f5740 1 -- 172.21.15.33:0/2862011681 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bcadfba2d0 msgr2=0x55bcadfc07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.936+0000 7fd0d27f5740 1 --2- 172.21.15.33:0/2862011681 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bcadfba2d0 0x55bcadfc07e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd0c4097c80 tx=0x7fd0c4097b20 comp rx=0 tx=0).stop 2024-03-19T03:00:50.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.936+0000 7fd0d27f5740 1 -- 172.21.15.33:0/2862011681 shutdown_connections 2024-03-19T03:00:50.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.936+0000 7fd0d27f5740 1 --2- 172.21.15.33:0/2862011681 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55bcadfd3ad0 0x55bcadfd5ec0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.937+0000 7fd0d27f5740 1 --2- 172.21.15.33:0/2862011681 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcadfbabf0 0x55bcadfc0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.937+0000 7fd0d27f5740 1 --2- 172.21.15.33:0/2862011681 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcadfc37d0 0x55bcadfc1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.937+0000 7fd0d27f5740 1 --2- 172.21.15.33:0/2862011681 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bcadfba2d0 0x55bcadfc07e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.937+0000 7fd0b17fa640 0 client.27053 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T03:00:50.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.937+0000 7fd0d27f5740 1 -- 172.21.15.33:0/2862011681 >> 172.21.15.33:0/2862011681 conn(0x55bcadfb4a00 msgr2=0x55bcadfb6750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:50.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.937+0000 7fd0d27f5740 1 -- 172.21.15.33:0/2862011681 shutdown_connections 2024-03-19T03:00:50.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.937+0000 7fd0d27f5740 1 -- 172.21.15.33:0/2862011681 wait complete. 2024-03-19T03:00:50.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.937+0000 7fd0d27f5740 20 client.27053 trim_cache size 0 max 16384 2024-03-19T03:00:50.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.950+0000 7fd0d27f5740 1 Processor -- start 2024-03-19T03:00:50.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.950+0000 7fd0d27f5740 1 -- start start 2024-03-19T03:00:50.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.950+0000 7fd0d27f5740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcadfba2d0 0x55bcadfcf360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.950+0000 7fd0d27f5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcadfbabf0 0x55bcadfc1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.950+0000 7fd0d27f5740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bcadfc37d0 0x55bcadfdb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.950+0000 7fd0d27f5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd0c81b37c0 con 0x55bcadfbabf0 2024-03-19T03:00:50.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.950+0000 7fd0d27f5740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd0c81b1090 con 0x55bcadfc37d0 2024-03-19T03:00:50.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.950+0000 7fd0d27f5740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd0c81aef00 con 0x55bcadfba2d0 2024-03-19T03:00:50.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.950+0000 7fd0cd1a6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcadfbabf0 0x55bcadfc1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.950+0000 7fd0cd1a6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcadfbabf0 0x55bcadfc1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58844/0 (socket says 172.21.15.33:58844) 2024-03-19T03:00:50.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.950+0000 7fd0cd1a6640 1 -- 172.21.15.33:0/277941965 learned_addr learned my addr 172.21.15.33:0/277941965 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:50.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.951+0000 7fd0ce1a8640 1 --2- 172.21.15.33:0/277941965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bcadfc37d0 0x55bcadfdb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.951+0000 7fd0cd9a7640 1 --2- 172.21.15.33:0/277941965 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcadfba2d0 0x55bcadfcf360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.951+0000 7fd0cd1a6640 1 -- 172.21.15.33:0/277941965 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcadfba2d0 msgr2=0x55bcadfcf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:50.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.951+0000 7fd0cd1a6640 1 --2- 172.21.15.33:0/277941965 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcadfba2d0 0x55bcadfcf360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.951+0000 7fd0cd1a6640 1 -- 172.21.15.33:0/277941965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bcadfc37d0 msgr2=0x55bcadfdb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:50.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.951+0000 7fd0cd1a6640 1 --2- 172.21.15.33:0/277941965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bcadfc37d0 0x55bcadfdb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.951+0000 7fd0cd1a6640 1 -- 172.21.15.33:0/277941965 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcadfbc3c0 con 0x55bcadfbabf0 2024-03-19T03:00:50.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.951+0000 7fd0cd1a6640 1 --2- 172.21.15.33:0/277941965 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcadfbabf0 0x55bcadfc1080 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fd0b8009380 tx=0x7fd0b8005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ba5f35a898057bb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:50.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.951+0000 7fd0b37fe640 1 -- 172.21.15.33:0/277941965 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0b801d040 con 0x55bcadfbabf0 2024-03-19T03:00:50.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.951+0000 7fd0b37fe640 1 -- 172.21.15.33:0/277941965 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd0b802d020 con 0x55bcadfbabf0 2024-03-19T03:00:50.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.951+0000 7fd0b37fe640 1 -- 172.21.15.33:0/277941965 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0b8005040 con 0x55bcadfbabf0 2024-03-19T03:00:50.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.952+0000 7fd0d27f5740 1 -- 172.21.15.33:0/277941965 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcadfbabf0 msgr2=0x55bcadfc1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.952+0000 7fd0d27f5740 1 --2- 172.21.15.33:0/277941965 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcadfbabf0 0x55bcadfc1080 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fd0b8009380 tx=0x7fd0b8005850 comp rx=0 tx=0).stop 2024-03-19T03:00:50.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.952+0000 7fd0d27f5740 1 -- 172.21.15.33:0/277941965 shutdown_connections 2024-03-19T03:00:50.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.952+0000 7fd0d27f5740 1 --2- 172.21.15.33:0/277941965 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bcadfc37d0 0x55bcadfdb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.952+0000 7fd0d27f5740 1 --2- 172.21.15.33:0/277941965 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcadfbabf0 0x55bcadfc1080 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.952+0000 7fd0d27f5740 1 --2- 172.21.15.33:0/277941965 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcadfba2d0 0x55bcadfcf360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.952+0000 7fd0d27f5740 1 -- 172.21.15.33:0/277941965 >> 172.21.15.33:0/277941965 conn(0x55bcadfb4a00 msgr2=0x55bcadfb3ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:50.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.952+0000 7fd0d27f5740 1 -- 172.21.15.33:0/277941965 shutdown_connections 2024-03-19T03:00:50.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.952+0000 7fd0d27f5740 1 -- 172.21.15.33:0/277941965 wait complete. 2024-03-19T03:00:50.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.952+0000 7fd0d27f5740 1 -- start start 2024-03-19T03:00:50.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.952+0000 7fd0d27f5740 1 Processor -- start 2024-03-19T03:00:50.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.952+0000 7fd0d27f5740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bcadfba2d0 0x55bcadfcb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.952+0000 7fd0d27f5740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcadfbabf0 0x55bcadfcbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.952+0000 7fd0d27f5740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcadfc37d0 0x55bcadfc6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.952+0000 7fd0d27f5740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd0c81af550 con 0x55bcadfbabf0 2024-03-19T03:00:50.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.952+0000 7fd0d27f5740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd0c81b09c0 con 0x55bcadfba2d0 2024-03-19T03:00:50.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.952+0000 7fd0d27f5740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd0c81afc20 con 0x55bcadfc37d0 2024-03-19T03:00:50.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.953+0000 7fd0d27f5740 10 client.0 _ll_register_callbacks cb 0x7fd0c400e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:00:50.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.953+0000 7fd0cd1a6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcadfbabf0 0x55bcadfcbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.953+0000 7fd0cd1a6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcadfbabf0 0x55bcadfcbdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58856/0 (socket says 172.21.15.33:58856) 2024-03-19T03:00:50.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.953+0000 7fd0cd1a6640 1 -- 172.21.15.33:0/3133806632 learned_addr learned my addr 172.21.15.33:0/3133806632 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:50.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.953+0000 7fd0cd9a7640 1 --2- 172.21.15.33:0/3133806632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bcadfba2d0 0x55bcadfcb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.953+0000 7fd0ce1a8640 1 --2- 172.21.15.33:0/3133806632 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcadfc37d0 0x55bcadfc6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.953+0000 7fd0cd1a6640 1 -- 172.21.15.33:0/3133806632 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcadfc37d0 msgr2=0x55bcadfc6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:50.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.953+0000 7fd0cd1a6640 1 --2- 172.21.15.33:0/3133806632 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcadfc37d0 0x55bcadfc6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.953+0000 7fd0cd1a6640 1 -- 172.21.15.33:0/3133806632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bcadfba2d0 msgr2=0x55bcadfcb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:50.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.953+0000 7fd0cd1a6640 1 --2- 172.21.15.33:0/3133806632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bcadfba2d0 0x55bcadfcb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.953+0000 7fd0cd1a6640 1 -- 172.21.15.33:0/3133806632 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcadfc7760 con 0x55bcadfbabf0 2024-03-19T03:00:50.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.953+0000 7fd0cd9a7640 1 --2- 172.21.15.33:0/3133806632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bcadfba2d0 0x55bcadfcb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:50.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.953+0000 7fd0cd1a6640 1 --2- 172.21.15.33:0/3133806632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcadfbabf0 0x55bcadfcbdb0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fd0b8009730 tx=0x7fd0b8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a95ae18e75bcd35 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:50.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.953+0000 7fd0b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:00:50.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.953+0000 7fd0b17fa640 1 -- 172.21.15.33:0/3133806632 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0b801d070 con 0x55bcadfbabf0 2024-03-19T03:00:50.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.953+0000 7fd0b17fa640 1 -- 172.21.15.33:0/3133806632 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7fd0b8016040 con 0x55bcadfbabf0 2024-03-19T03:00:50.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.953+0000 7fd0d27f5740 1 -- 172.21.15.33:0/3133806632 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0b802d020 con 0x55bcadfbabf0 2024-03-19T03:00:50.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.953+0000 7fd0b17fa640 1 -- 172.21.15.33:0/3133806632 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd0b8041050 con 0x55bcadfbabf0 2024-03-19T03:00:50.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.953+0000 7fd0d27f5740 10 client.17238 fetch_fsmap learned FSMap version 25 2024-03-19T03:00:50.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.953+0000 7fd0d27f5740 1 -- 172.21.15.33:0/3133806632 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55bcae0399d0 con 0x55bcadfbabf0 2024-03-19T03:00:50.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.954+0000 7fd0b17fa640 1 -- 172.21.15.33:0/3133806632 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd0b802d020 con 0x55bcadfbabf0 2024-03-19T03:00:50.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.954+0000 7fd0b17fa640 1 -- 172.21.15.33:0/3133806632 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fd0b80035f0 con 0x55bcadfbabf0 2024-03-19T03:00:50.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.954+0000 7fd0d27f5740 10 client.17238 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:00:50.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.954+0000 7fd0d27f5740 10 client.17238 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T03:00:50.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.954+0000 7fd0d27f5740 20 client.17238 populate_metadata read hostname 'smithi033' 2024-03-19T03:00:50.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.954+0000 7fd0d27f5740 1 --2- 172.21.15.33:0/3133806632 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55bcae039f80 0x55bcae03c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:50.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.954+0000 7fd0d27f5740 4 client.17238 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T03:00:50.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.954+0000 7fd0d27f5740 1 -- 172.21.15.33:0/3133806632 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bcadf2a9f0 con 0x55bcae039f80 2024-03-19T03:00:50.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.954+0000 7fd0cd9a7640 1 --2- 172.21.15.33:0/3133806632 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55bcae039f80 0x55bcae03c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:50.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.954+0000 7fd0b17fa640 10 client.17238 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T03:00:50.961 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.954+0000 7fd0cd9a7640 1 --2- 172.21.15.33:0/3133806632 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55bcae039f80 0x55bcae03c3a0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2df58506ec0bbb77 server_cookie=7b650bf660565f92 in_seq=0 out_seq=0 2024-03-19T03:00:50.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.954+0000 7fd0b17fa640 1 -- 172.21.15.33:0/3133806632 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bcadf2a9f0 con 0x55bcae039f80 2024-03-19T03:00:50.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.955+0000 7fd0b17fa640 10 client.17238 handle_command_reply: tid=0 2024-03-19T03:00:50.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.955+0000 7fd0d27f5740 1 client.17238 shutdown 2024-03-19T03:00:50.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.955+0000 7fd0d27f5740 1 -- 172.21.15.33:0/3133806632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcadfbabf0 msgr2=0x55bcadfcbdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.955+0000 7fd0d27f5740 1 --2- 172.21.15.33:0/3133806632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcadfbabf0 0x55bcadfcbdb0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fd0b8009730 tx=0x7fd0b8005f90 comp rx=0 tx=0).stop 2024-03-19T03:00:50.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.955+0000 7fd0d27f5740 1 -- 172.21.15.33:0/3133806632 shutdown_connections 2024-03-19T03:00:50.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.955+0000 7fd0d27f5740 1 --2- 172.21.15.33:0/3133806632 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55bcae039f80 0x55bcae03c3a0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.955+0000 7fd0d27f5740 1 --2- 172.21.15.33:0/3133806632 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55bcadfc37d0 0x55bcadfc6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.955+0000 7fd0d27f5740 1 --2- 172.21.15.33:0/3133806632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55bcadfbabf0 0x55bcadfcbdb0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.955+0000 7fd0d27f5740 1 --2- 172.21.15.33:0/3133806632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55bcadfba2d0 0x55bcadfcb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.955+0000 7fd0b17fa640 0 client.17238 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T03:00:50.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.955+0000 7fd0d27f5740 1 -- 172.21.15.33:0/3133806632 >> 172.21.15.33:0/3133806632 conn(0x55bcadfb4a00 msgr2=0x55bcadfb5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:50.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.955+0000 7fd0d27f5740 1 -- 172.21.15.33:0/3133806632 shutdown_connections 2024-03-19T03:00:50.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.955+0000 7fd0d27f5740 1 -- 172.21.15.33:0/3133806632 wait complete. 2024-03-19T03:00:50.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.955+0000 7fd0d27f5740 20 client.17238 trim_cache size 0 max 16384 2024-03-19T03:00:50.963 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T03:00:50.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.955+0000 7fd0cfc0b640 1 -- 172.21.15.33:0/2393948878 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd0b40a2970 msgr2=0x7fd0b40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.955+0000 7fd0cfc0b640 1 --2- 172.21.15.33:0/2393948878 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd0b40a2970 0x7fd0b40a4e30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd0c0005cc0 tx=0x7fd0c002c0f0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.955+0000 7fd0cfc0b640 1 -- 172.21.15.33:0/2393948878 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0c81006a0 msgr2=0x7fd0c819a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:50.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.955+0000 7fd0cfc0b640 1 --2- 172.21.15.33:0/2393948878 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0c81006a0 0x7fd0c819a150 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fd0c400e9f0 tx=0x7fd0c400eeb0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.955+0000 7fd0cfc0b640 1 -- 172.21.15.33:0/2393948878 shutdown_connections 2024-03-19T03:00:50.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.955+0000 7fd0cfc0b640 1 --2- 172.21.15.33:0/2393948878 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd0b40a2970 0x7fd0b40a4e30 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.955+0000 7fd0cfc0b640 1 --2- 172.21.15.33:0/2393948878 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd0c81066b0 0x7fd0c819ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.955+0000 7fd0cfc0b640 1 --2- 172.21.15.33:0/2393948878 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0c8101060 0x7fd0c819a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.955+0000 7fd0cfc0b640 1 --2- 172.21.15.33:0/2393948878 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd0c81006a0 0x7fd0c819a150 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:50.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.955+0000 7fd0cfc0b640 1 -- 172.21.15.33:0/2393948878 >> 172.21.15.33:0/2393948878 conn(0x7fd0c80778d0 msgr2=0x7fd0c810bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:50.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.955+0000 7fd0cfc0b640 1 -- 172.21.15.33:0/2393948878 shutdown_connections 2024-03-19T03:00:50.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:50.956+0000 7fd0cfc0b640 1 -- 172.21.15.33:0/2393948878 wait complete. 2024-03-19T03:00:50.966 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:50 smithi033 ceph-mon[31973]: pgmap v226: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:00:50.966 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:50 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2512217822' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:00:50.967 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T03:00:51.074 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.070+0000 7fe7304d1640 1 -- 172.21.15.33:0/775502184 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7281017f0 msgr2=0x7fe728101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:51.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.070+0000 7fe7304d1640 1 --2- 172.21.15.33:0/775502184 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7281017f0 0x7fe728101c70 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fe718009930 tx=0x7fe71802bc30 comp rx=0 tx=0).stop 2024-03-19T03:00:51.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.071+0000 7fe7304d1640 1 -- 172.21.15.33:0/775502184 shutdown_connections 2024-03-19T03:00:51.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.071+0000 7fe7304d1640 1 --2- 172.21.15.33:0/775502184 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7281021b0 0x7fe72810e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.071+0000 7fe7304d1640 1 --2- 172.21.15.33:0/775502184 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7281017f0 0x7fe728101c70 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.071+0000 7fe7304d1640 1 --2- 172.21.15.33:0/775502184 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe728107800 0x7fe728107be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.071+0000 7fe7304d1640 1 -- 172.21.15.33:0/775502184 >> 172.21.15.33:0/775502184 conn(0x7fe7280fd2c0 msgr2=0x7fe7280ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:51.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.071+0000 7fe7304d1640 1 -- 172.21.15.33:0/775502184 shutdown_connections 2024-03-19T03:00:51.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.071+0000 7fe7304d1640 1 -- 172.21.15.33:0/775502184 wait complete. 2024-03-19T03:00:51.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.071+0000 7fe7304d1640 1 Processor -- start 2024-03-19T03:00:51.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.071+0000 7fe7304d1640 1 -- start start 2024-03-19T03:00:51.075 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.071+0000 7fe7304d1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7281017f0 0x7fe728073b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:51.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.071+0000 7fe7304d1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7281021b0 0x7fe728074060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:51.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.071+0000 7fe7304d1640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe728107800 0x7fe7280745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:51.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.071+0000 7fe7304d1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe728111ac0 con 0x7fe7281021b0 2024-03-19T03:00:51.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.071+0000 7fe7304d1640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe728111940 con 0x7fe7281017f0 2024-03-19T03:00:51.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.071+0000 7fe7304d1640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe728111c40 con 0x7fe728107800 2024-03-19T03:00:51.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.072+0000 7fe72da6c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7281021b0 0x7fe728074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:51.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.072+0000 7fe72da6c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7281021b0 0x7fe728074060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58866/0 (socket says 172.21.15.33:58866) 2024-03-19T03:00:51.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.072+0000 7fe72da6c640 1 -- 172.21.15.33:0/3873682647 learned_addr learned my addr 172.21.15.33:0/3873682647 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:51.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.072+0000 7fe72e26d640 1 --2- 172.21.15.33:0/3873682647 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7281017f0 0x7fe728073b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:51.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.072+0000 7fe72ea6e640 1 --2- 172.21.15.33:0/3873682647 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe728107800 0x7fe7280745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:51.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.072+0000 7fe72da6c640 1 -- 172.21.15.33:0/3873682647 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe728107800 msgr2=0x7fe7280745a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:51.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.072+0000 7fe72da6c640 1 --2- 172.21.15.33:0/3873682647 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe728107800 0x7fe7280745a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.072+0000 7fe72da6c640 1 -- 172.21.15.33:0/3873682647 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7281017f0 msgr2=0x7fe728073b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:51.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.072+0000 7fe72da6c640 1 --2- 172.21.15.33:0/3873682647 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7281017f0 0x7fe728073b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.076 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.072+0000 7fe72da6c640 1 -- 172.21.15.33:0/3873682647 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7281a6520 con 0x7fe7281021b0 2024-03-19T03:00:51.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.072+0000 7fe72e26d640 1 --2- 172.21.15.33:0/3873682647 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7281017f0 0x7fe728073b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:51.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.072+0000 7fe72da6c640 1 --2- 172.21.15.33:0/3873682647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7281021b0 0x7fe728074060 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fe718009230 tx=0x7fe71802c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:51.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.072+0000 7fe71f7fe640 1 -- 172.21.15.33:0/3873682647 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe718043020 con 0x7fe7281021b0 2024-03-19T03:00:51.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.072+0000 7fe7304d1640 1 -- 172.21.15.33:0/3873682647 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7281a67b0 con 0x7fe7281021b0 2024-03-19T03:00:51.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.072+0000 7fe71f7fe640 1 -- 172.21.15.33:0/3873682647 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe71802ca70 con 0x7fe7281021b0 2024-03-19T03:00:51.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.072+0000 7fe71f7fe640 1 -- 172.21.15.33:0/3873682647 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe718003970 con 0x7fe7281021b0 2024-03-19T03:00:51.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.072+0000 7fe7304d1640 1 -- 172.21.15.33:0/3873682647 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7281a6d50 con 0x7fe7281021b0 2024-03-19T03:00:51.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.075+0000 7fe71f7fe640 1 -- 172.21.15.33:0/3873682647 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fe718046020 con 0x7fe7281021b0 2024-03-19T03:00:51.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.075+0000 7fe733197740 1 Processor -- start 2024-03-19T03:00:51.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.075+0000 7fe733197740 1 -- start start 2024-03-19T03:00:51.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.075+0000 7fe733197740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d0cfeb2d0 0x560d0cfeb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:51.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.075+0000 7fe733197740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0cfebbf0 0x560d0cff4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:51.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.075+0000 7fe733197740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d0cff47d0 0x560d0cff6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:51.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.075+0000 7fe733197740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560d0cf4e4b0 con 0x560d0cfebbf0 2024-03-19T03:00:51.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.075+0000 7fe733197740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x560d0cef6d70 con 0x560d0cff47d0 2024-03-19T03:00:51.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.075+0000 7fe733197740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560d0cf4a550 con 0x560d0cfeb2d0 2024-03-19T03:00:51.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.075+0000 7fe72e26d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0cfebbf0 0x560d0cff4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:51.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.075+0000 7fe72e26d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0cfebbf0 0x560d0cff4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58870/0 (socket says 172.21.15.33:58870) 2024-03-19T03:00:51.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.075+0000 7fe72e26d640 1 -- 172.21.15.33:0/672067396 learned_addr learned my addr 172.21.15.33:0/672067396 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:51.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.076+0000 7fe72da6c640 1 --2- 172.21.15.33:0/672067396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d0cff47d0 0x560d0cff6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:51.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.076+0000 7fe72ea6e640 1 --2- 172.21.15.33:0/672067396 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d0cfeb2d0 0x560d0cfeb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:51.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.076+0000 7fe71f7fe640 1 --2- 172.21.15.33:0/3873682647 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe7140a2970 0x7fe7140a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:51.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.076+0000 7fe72e26d640 1 -- 172.21.15.33:0/672067396 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d0cfeb2d0 msgr2=0x560d0cfeb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:51.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.076+0000 7fe72ea6e640 1 --2- 172.21.15.33:0/3873682647 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe7140a2970 0x7fe7140a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:51.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.076+0000 7fe72e26d640 1 --2- 172.21.15.33:0/672067396 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d0cfeb2d0 0x560d0cfeb6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.076+0000 7fe72e26d640 1 -- 172.21.15.33:0/672067396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d0cff47d0 msgr2=0x560d0cff6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:51.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.076+0000 7fe72e26d640 1 --2- 172.21.15.33:0/672067396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d0cff47d0 0x560d0cff6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.076+0000 7fe72e26d640 1 -- 172.21.15.33:0/672067396 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d0cf19210 con 0x560d0cfebbf0 2024-03-19T03:00:51.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.076+0000 7fe72da6c640 1 --2- 172.21.15.33:0/672067396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d0cff47d0 0x560d0cff6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:51.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.076+0000 7fe71f7fe640 1 -- 172.21.15.33:0/3873682647 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7fe7180f1140 con 0x7fe7281021b0 2024-03-19T03:00:51.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.076+0000 7fe72e26d640 1 --2- 172.21.15.33:0/672067396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0cfebbf0 0x560d0cff4290 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fe7200114c0 tx=0x7fe720011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56954a27ec1b46f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:51.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.076+0000 7fe71cbfe640 1 -- 172.21.15.33:0/672067396 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe72000f5c0 con 0x560d0cfebbf0 2024-03-19T03:00:51.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.076+0000 7fe72ea6e640 1 --2- 172.21.15.33:0/3873682647 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe7140a2970 0x7fe7140a4e30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe724009fd0 tx=0x7fe724004ab0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:51.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.076+0000 7fe71cbfe640 1 -- 172.21.15.33:0/672067396 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe72000fc10 con 0x560d0cfebbf0 2024-03-19T03:00:51.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.076+0000 7fe71cbfe640 1 -- 172.21.15.33:0/672067396 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe720019c40 con 0x560d0cfebbf0 2024-03-19T03:00:51.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.077+0000 7fe733197740 1 -- 172.21.15.33:0/672067396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0cfebbf0 msgr2=0x560d0cff4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:51.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.077+0000 7fe733197740 1 --2- 172.21.15.33:0/672067396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0cfebbf0 0x560d0cff4290 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fe7200114c0 tx=0x7fe720011980 comp rx=0 tx=0).stop 2024-03-19T03:00:51.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.077+0000 7fe733197740 1 -- 172.21.15.33:0/672067396 shutdown_connections 2024-03-19T03:00:51.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.077+0000 7fe733197740 1 --2- 172.21.15.33:0/672067396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d0cff47d0 0x560d0cff6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.077+0000 7fe733197740 1 --2- 172.21.15.33:0/672067396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0cfebbf0 0x560d0cff4290 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.077+0000 7fe733197740 1 --2- 172.21.15.33:0/672067396 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d0cfeb2d0 0x560d0cfeb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.077+0000 7fe733197740 1 -- 172.21.15.33:0/672067396 >> 172.21.15.33:0/672067396 conn(0x560d0cfe5a00 msgr2=0x560d0cfe5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:51.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.077+0000 7fe733197740 1 -- 172.21.15.33:0/672067396 shutdown_connections 2024-03-19T03:00:51.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.077+0000 7fe733197740 1 -- 172.21.15.33:0/672067396 wait complete. 2024-03-19T03:00:51.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.077+0000 7fe733197740 1 -- start start 2024-03-19T03:00:51.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.078+0000 7fe733197740 1 Processor -- start 2024-03-19T03:00:51.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.078+0000 7fe733197740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d0cfeb2d0 0x560d0cff1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:51.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.078+0000 7fe733197740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0cfebbf0 0x560d0cff1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:51.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.078+0000 7fe733197740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d0cff47d0 0x560d0cff22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:51.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.078+0000 7fe733197740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560d0cff75c0 con 0x560d0cfebbf0 2024-03-19T03:00:51.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.078+0000 7fe733197740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x560d0cf0a310 con 0x560d0cff47d0 2024-03-19T03:00:51.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.078+0000 7fe733197740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560d0cff7740 con 0x560d0cfeb2d0 2024-03-19T03:00:51.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.078+0000 7fe733197740 10 client.0 _ll_register_callbacks cb 0x560d0cefe2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:00:51.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.078+0000 7fe72da6c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0cfebbf0 0x560d0cff1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:51.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.078+0000 7fe72da6c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0cfebbf0 0x560d0cff1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58880/0 (socket says 172.21.15.33:58880) 2024-03-19T03:00:51.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.078+0000 7fe72da6c640 1 -- 172.21.15.33:0/553780551 learned_addr learned my addr 172.21.15.33:0/553780551 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:51.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.078+0000 7fe72e26d640 1 --2- 172.21.15.33:0/553780551 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d0cfeb2d0 0x560d0cff1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:51.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.078+0000 7fe72ea6e640 1 --2- 172.21.15.33:0/553780551 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d0cff47d0 0x560d0cff22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:51.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.078+0000 7fe72e26d640 1 -- 172.21.15.33:0/553780551 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d0cff47d0 msgr2=0x560d0cff22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:51.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.078+0000 7fe72e26d640 1 --2- 172.21.15.33:0/553780551 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d0cff47d0 0x560d0cff22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.078+0000 7fe72e26d640 1 -- 172.21.15.33:0/553780551 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0cfebbf0 msgr2=0x560d0cff1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:51.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.078+0000 7fe72e26d640 1 --2- 172.21.15.33:0/553780551 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0cfebbf0 0x560d0cff1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.078+0000 7fe72e26d640 1 -- 172.21.15.33:0/553780551 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d0cc623f0 con 0x560d0cfeb2d0 2024-03-19T03:00:51.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.078+0000 7fe72da6c640 1 --2- 172.21.15.33:0/553780551 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0cfebbf0 0x560d0cff1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:51.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.078+0000 7fe72e26d640 1 --2- 172.21.15.33:0/553780551 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d0cfeb2d0 0x560d0cff1840 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe72001b040 tx=0x7fe720009da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56144b56c46cab0b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:51.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.078+0000 7fe711ffb640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-19T03:00:51.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.079+0000 7fe711ffb640 1 -- 172.21.15.33:0/553780551 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe72000f950 con 0x560d0cfeb2d0 2024-03-19T03:00:51.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.079+0000 7fe733197740 1 -- 172.21.15.33:0/553780551 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe72000fc10 con 0x560d0cfeb2d0 2024-03-19T03:00:51.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.079+0000 7fe711ffb640 1 -- 172.21.15.33:0/553780551 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7fe720019940 con 0x560d0cfeb2d0 2024-03-19T03:00:51.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.079+0000 7fe711ffb640 1 -- 172.21.15.33:0/553780551 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe720030050 con 0x560d0cfeb2d0 2024-03-19T03:00:51.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.079+0000 7fe733197740 10 client.26935 fetch_fsmap learned FSMap version 25 2024-03-19T03:00:51.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.079+0000 7fe733197740 1 -- 172.21.15.33:0/553780551 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x560d0cd2c7d0 con 0x560d0cfeb2d0 2024-03-19T03:00:51.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.079+0000 7fe711ffb640 1 -- 172.21.15.33:0/553780551 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe72000fc10 con 0x560d0cfeb2d0 2024-03-19T03:00:51.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.079+0000 7fe711ffb640 1 -- 172.21.15.33:0/553780551 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fe720004750 con 0x560d0cfeb2d0 2024-03-19T03:00:51.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.079+0000 7fe733197740 10 client.26935 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:00:51.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.079+0000 7fe733197740 10 client.26935 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:00:51.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.079+0000 7fe733197740 20 client.26935 populate_metadata read hostname 'smithi033' 2024-03-19T03:00:51.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.079+0000 7fe733197740 1 --2- 172.21.15.33:0/553780551 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x560d0d004ae0 0x560d0d006ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:51.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.079+0000 7fe733197740 4 client.26935 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:00:51.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.079+0000 7fe733197740 1 -- 172.21.15.33:0/553780551 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d0cfac3e0 con 0x560d0d004ae0 2024-03-19T03:00:51.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.079+0000 7fe72da6c640 1 --2- 172.21.15.33:0/553780551 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x560d0d004ae0 0x560d0d006ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:51.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.080+0000 7fe711ffb640 10 client.26935 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:00:51.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.080+0000 7fe72da6c640 1 --2- 172.21.15.33:0/553780551 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x560d0d004ae0 0x560d0d006ed0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f43c9ab39cd7cfc server_cookie=ad9bfaaf80610480 in_seq=0 out_seq=0 2024-03-19T03:00:51.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.080+0000 7fe711ffb640 1 -- 172.21.15.33:0/553780551 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560d0cfac3e0 con 0x560d0d004ae0 2024-03-19T03:00:51.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.080+0000 7fe711ffb640 10 client.26935 handle_command_reply: tid=0 2024-03-19T03:00:51.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.080+0000 7fe733197740 1 client.26935 shutdown 2024-03-19T03:00:51.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.080+0000 7fe733197740 1 -- 172.21.15.33:0/553780551 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d0cfeb2d0 msgr2=0x560d0cff1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:51.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.080+0000 7fe733197740 1 --2- 172.21.15.33:0/553780551 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d0cfeb2d0 0x560d0cff1840 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe72001b040 tx=0x7fe720009da0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.080+0000 7fe733197740 1 -- 172.21.15.33:0/553780551 shutdown_connections 2024-03-19T03:00:51.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.080+0000 7fe733197740 1 --2- 172.21.15.33:0/553780551 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x560d0d004ae0 0x560d0d006ed0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.080+0000 7fe733197740 1 --2- 172.21.15.33:0/553780551 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d0cff47d0 0x560d0cff22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.080+0000 7fe733197740 1 --2- 172.21.15.33:0/553780551 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0cfebbf0 0x560d0cff1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.080+0000 7fe711ffb640 0 client.26935 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:00:51.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.080+0000 7fe733197740 1 --2- 172.21.15.33:0/553780551 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d0cfeb2d0 0x560d0cff1840 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.081+0000 7fe733197740 1 -- 172.21.15.33:0/553780551 >> 172.21.15.33:0/553780551 conn(0x560d0cfe5a00 msgr2=0x560d0cfe7750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:51.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.081+0000 7fe733197740 1 -- 172.21.15.33:0/553780551 shutdown_connections 2024-03-19T03:00:51.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.081+0000 7fe733197740 1 -- 172.21.15.33:0/553780551 wait complete. 2024-03-19T03:00:51.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.081+0000 7fe733197740 20 client.26935 trim_cache size 0 max 16384 2024-03-19T03:00:51.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.094+0000 7fe733197740 1 Processor -- start 2024-03-19T03:00:51.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.094+0000 7fe733197740 1 -- start start 2024-03-19T03:00:51.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.094+0000 7fe733197740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0cfeb2d0 0x560d0d000360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:51.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.094+0000 7fe733197740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d0cfebbf0 0x560d0cff20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:51.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.094+0000 7fe733197740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d0cff47d0 0x560d0cfe4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:51.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.094+0000 7fe733197740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe7281b37f0 con 0x560d0cfeb2d0 2024-03-19T03:00:51.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.094+0000 7fe733197740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe7281b10c0 con 0x560d0cfebbf0 2024-03-19T03:00:51.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.094+0000 7fe733197740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe7281aef30 con 0x560d0cff47d0 2024-03-19T03:00:51.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.094+0000 7fe72e26d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0cfeb2d0 0x560d0d000360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:51.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.094+0000 7fe72e26d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0cfeb2d0 0x560d0d000360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58896/0 (socket says 172.21.15.33:58896) 2024-03-19T03:00:51.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.094+0000 7fe72e26d640 1 -- 172.21.15.33:0/2378649123 learned_addr learned my addr 172.21.15.33:0/2378649123 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:51.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.094+0000 7fe72da6c640 1 --2- 172.21.15.33:0/2378649123 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d0cfebbf0 0x560d0cff20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:51.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.094+0000 7fe72ea6e640 1 --2- 172.21.15.33:0/2378649123 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d0cff47d0 0x560d0cfe4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:51.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.095+0000 7fe72e26d640 1 -- 172.21.15.33:0/2378649123 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d0cff47d0 msgr2=0x560d0cfe4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:51.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.095+0000 7fe72e26d640 1 --2- 172.21.15.33:0/2378649123 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d0cff47d0 0x560d0cfe4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.095+0000 7fe72e26d640 1 -- 172.21.15.33:0/2378649123 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d0cfebbf0 msgr2=0x560d0cff20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:51.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.095+0000 7fe72e26d640 1 --2- 172.21.15.33:0/2378649123 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d0cfebbf0 0x560d0cff20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.095+0000 7fe72e26d640 1 -- 172.21.15.33:0/2378649123 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d0d007240 con 0x560d0cfeb2d0 2024-03-19T03:00:51.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.095+0000 7fe72ea6e640 1 --2- 172.21.15.33:0/2378649123 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d0cff47d0 0x560d0cfe4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:00:51.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.095+0000 7fe72da6c640 1 --2- 172.21.15.33:0/2378649123 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d0cfebbf0 0x560d0cff20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:51.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.095+0000 7fe72e26d640 1 --2- 172.21.15.33:0/2378649123 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0cfeb2d0 0x560d0d000360 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fe720000c00 tx=0x7fe720005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9c138aff1e034c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:51.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.095+0000 7fe713fff640 1 -- 172.21.15.33:0/2378649123 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe720039070 con 0x560d0cfeb2d0 2024-03-19T03:00:51.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.095+0000 7fe713fff640 1 -- 172.21.15.33:0/2378649123 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe720027070 con 0x560d0cfeb2d0 2024-03-19T03:00:51.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.095+0000 7fe713fff640 1 -- 172.21.15.33:0/2378649123 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe720003970 con 0x560d0cfeb2d0 2024-03-19T03:00:51.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.095+0000 7fe733197740 1 -- 172.21.15.33:0/2378649123 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0cfeb2d0 msgr2=0x560d0d000360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:51.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.095+0000 7fe733197740 1 --2- 172.21.15.33:0/2378649123 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0cfeb2d0 0x560d0d000360 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fe720000c00 tx=0x7fe720005f90 comp rx=0 tx=0).stop 2024-03-19T03:00:51.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.095+0000 7fe733197740 1 -- 172.21.15.33:0/2378649123 shutdown_connections 2024-03-19T03:00:51.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.095+0000 7fe733197740 1 --2- 172.21.15.33:0/2378649123 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d0cff47d0 0x560d0cfe4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.096+0000 7fe733197740 1 --2- 172.21.15.33:0/2378649123 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d0cfebbf0 0x560d0cff20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.096+0000 7fe733197740 1 --2- 172.21.15.33:0/2378649123 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0cfeb2d0 0x560d0d000360 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.096+0000 7fe733197740 1 -- 172.21.15.33:0/2378649123 >> 172.21.15.33:0/2378649123 conn(0x560d0cfe5a00 msgr2=0x560d0cfe4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:51.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.096+0000 7fe733197740 1 -- 172.21.15.33:0/2378649123 shutdown_connections 2024-03-19T03:00:51.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.096+0000 7fe733197740 1 -- 172.21.15.33:0/2378649123 wait complete. 2024-03-19T03:00:51.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.096+0000 7fe733197740 1 -- start start 2024-03-19T03:00:51.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.096+0000 7fe733197740 1 Processor -- start 2024-03-19T03:00:51.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.096+0000 7fe733197740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d0cfeb2d0 0x560d0cff7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:51.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.096+0000 7fe733197740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d0cfebbf0 0x560d0cff8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:51.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.096+0000 7fe733197740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0cff47d0 0x560d0d062d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:51.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.096+0000 7fe733197740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe7281af580 con 0x560d0cff47d0 2024-03-19T03:00:51.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.096+0000 7fe733197740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe7281b09f0 con 0x560d0cfebbf0 2024-03-19T03:00:51.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.096+0000 7fe733197740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe7281afc50 con 0x560d0cfeb2d0 2024-03-19T03:00:51.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.096+0000 7fe733197740 10 client.0 _ll_register_callbacks cb 0x7fe7180b9800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:00:51.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.096+0000 7fe72ea6e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0cff47d0 0x560d0d062d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:51.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.096+0000 7fe72ea6e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0cff47d0 0x560d0d062d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58908/0 (socket says 172.21.15.33:58908) 2024-03-19T03:00:51.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.096+0000 7fe72ea6e640 1 -- 172.21.15.33:0/2519319362 learned_addr learned my addr 172.21.15.33:0/2519319362 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:51.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.096+0000 7fe72da6c640 1 --2- 172.21.15.33:0/2519319362 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d0cfebbf0 0x560d0cff8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:51.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.097+0000 7fe72e26d640 1 --2- 172.21.15.33:0/2519319362 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d0cfeb2d0 0x560d0cff7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:51.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.097+0000 7fe72ea6e640 1 -- 172.21.15.33:0/2519319362 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d0cfeb2d0 msgr2=0x560d0cff7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:51.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.097+0000 7fe72ea6e640 1 --2- 172.21.15.33:0/2519319362 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d0cfeb2d0 0x560d0cff7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.097+0000 7fe72ea6e640 1 -- 172.21.15.33:0/2519319362 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d0cfebbf0 msgr2=0x560d0cff8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:51.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.097+0000 7fe72ea6e640 1 --2- 172.21.15.33:0/2519319362 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d0cfebbf0 0x560d0cff8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.097+0000 7fe72ea6e640 1 -- 172.21.15.33:0/2519319362 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d0cfeba20 con 0x560d0cff47d0 2024-03-19T03:00:51.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.097+0000 7fe72e26d640 1 --2- 172.21.15.33:0/2519319362 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d0cfeb2d0 0x560d0cff7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T03:00:51.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.097+0000 7fe72da6c640 1 --2- 172.21.15.33:0/2519319362 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d0cfebbf0 0x560d0cff8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:51.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.097+0000 7fe72ea6e640 1 --2- 172.21.15.33:0/2519319362 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0cff47d0 0x560d0d062d20 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fe724005fd0 tx=0x7fe724004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7219e481bb6c50dd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:51.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.097+0000 7fe711ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:00:51.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.097+0000 7fe711ffb640 1 -- 172.21.15.33:0/2519319362 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe724023d30 con 0x560d0cff47d0 2024-03-19T03:00:51.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.097+0000 7fe711ffb640 1 -- 172.21.15.33:0/2519319362 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7fe724004b20 con 0x560d0cff47d0 2024-03-19T03:00:51.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.097+0000 7fe733197740 1 -- 172.21.15.33:0/2519319362 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe720027070 con 0x560d0cff47d0 2024-03-19T03:00:51.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.097+0000 7fe711ffb640 1 -- 172.21.15.33:0/2519319362 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe724037050 con 0x560d0cff47d0 2024-03-19T03:00:51.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.097+0000 7fe733197740 10 client.17256 fetch_fsmap learned FSMap version 25 2024-03-19T03:00:51.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.097+0000 7fe733197740 1 -- 172.21.15.33:0/2519319362 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x560d0d06a4f0 con 0x560d0cff47d0 2024-03-19T03:00:51.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.097+0000 7fe711ffb640 1 -- 172.21.15.33:0/2519319362 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe720027070 con 0x560d0cff47d0 2024-03-19T03:00:51.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.097+0000 7fe711ffb640 1 -- 172.21.15.33:0/2519319362 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fe72401f040 con 0x560d0cff47d0 2024-03-19T03:00:51.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.097+0000 7fe733197740 10 client.17256 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:00:51.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.097+0000 7fe733197740 10 client.17256 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:00:51.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.097+0000 7fe733197740 20 client.17256 populate_metadata read hostname 'smithi033' 2024-03-19T03:00:51.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.097+0000 7fe733197740 1 --2- 172.21.15.33:0/2519319362 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x560d0d06acf0 0x560d0d06d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:51.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.098+0000 7fe733197740 4 client.17256 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T03:00:51.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.098+0000 7fe733197740 1 -- 172.21.15.33:0/2519319362 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560d0cf5b9f0 con 0x560d0d06acf0 2024-03-19T03:00:51.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.098+0000 7fe72e26d640 1 --2- 172.21.15.33:0/2519319362 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x560d0d06acf0 0x560d0d06d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:51.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.098+0000 7fe711ffb640 10 client.17256 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:00:51.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.098+0000 7fe72e26d640 1 --2- 172.21.15.33:0/2519319362 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x560d0d06acf0 0x560d0d06d110 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20ef7be2919f5c0e server_cookie=ff26b7979127b1cc in_seq=0 out_seq=0 2024-03-19T03:00:51.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.098+0000 7fe711ffb640 1 -- 172.21.15.33:0/2519319362 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560d0cf5b9f0 con 0x560d0d06acf0 2024-03-19T03:00:51.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.098+0000 7fe711ffb640 10 client.17256 handle_command_reply: tid=0 2024-03-19T03:00:51.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.098+0000 7fe733197740 1 client.17256 shutdown 2024-03-19T03:00:51.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.098+0000 7fe733197740 1 -- 172.21.15.33:0/2519319362 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0cff47d0 msgr2=0x560d0d062d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:51.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.098+0000 7fe733197740 1 --2- 172.21.15.33:0/2519319362 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0cff47d0 0x560d0d062d20 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fe724005fd0 tx=0x7fe724004a40 comp rx=0 tx=0).stop 2024-03-19T03:00:51.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.098+0000 7fe733197740 1 -- 172.21.15.33:0/2519319362 shutdown_connections 2024-03-19T03:00:51.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.098+0000 7fe733197740 1 --2- 172.21.15.33:0/2519319362 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x560d0d06acf0 0x560d0d06d110 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.098+0000 7fe733197740 1 --2- 172.21.15.33:0/2519319362 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560d0cff47d0 0x560d0d062d20 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.098+0000 7fe733197740 1 --2- 172.21.15.33:0/2519319362 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560d0cfebbf0 0x560d0cff8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.098+0000 7fe711ffb640 0 client.17256 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:00:51.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.098+0000 7fe733197740 1 --2- 172.21.15.33:0/2519319362 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560d0cfeb2d0 0x560d0cff7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.098+0000 7fe733197740 1 -- 172.21.15.33:0/2519319362 >> 172.21.15.33:0/2519319362 conn(0x560d0cfe5a00 msgr2=0x560d0cfe6a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:51.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.098+0000 7fe733197740 1 -- 172.21.15.33:0/2519319362 shutdown_connections 2024-03-19T03:00:51.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.098+0000 7fe733197740 1 -- 172.21.15.33:0/2519319362 wait complete. 2024-03-19T03:00:51.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.098+0000 7fe733197740 20 client.17256 trim_cache size 0 max 16384 2024-03-19T03:00:51.107 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T03:00:51.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.099+0000 7fe7304d1640 1 -- 172.21.15.33:0/3873682647 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe7140a2970 msgr2=0x7fe7140a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:51.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.099+0000 7fe7304d1640 1 --2- 172.21.15.33:0/3873682647 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe7140a2970 0x7fe7140a4e30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe724009fd0 tx=0x7fe724004ab0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.099+0000 7fe7304d1640 1 -- 172.21.15.33:0/3873682647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7281021b0 msgr2=0x7fe728074060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:51.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.099+0000 7fe7304d1640 1 --2- 172.21.15.33:0/3873682647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7281021b0 0x7fe728074060 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fe718009230 tx=0x7fe71802c5d0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.099+0000 7fe7304d1640 1 -- 172.21.15.33:0/3873682647 shutdown_connections 2024-03-19T03:00:51.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.099+0000 7fe7304d1640 1 --2- 172.21.15.33:0/3873682647 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe7140a2970 0x7fe7140a4e30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.099+0000 7fe7304d1640 1 --2- 172.21.15.33:0/3873682647 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe728107800 0x7fe7280745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.099+0000 7fe7304d1640 1 --2- 172.21.15.33:0/3873682647 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7281021b0 0x7fe728074060 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.099+0000 7fe7304d1640 1 --2- 172.21.15.33:0/3873682647 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7281017f0 0x7fe728073b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:51.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.099+0000 7fe7304d1640 1 -- 172.21.15.33:0/3873682647 >> 172.21.15.33:0/3873682647 conn(0x7fe7280fd2c0 msgr2=0x7fe7280fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:51.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.099+0000 7fe7304d1640 1 -- 172.21.15.33:0/3873682647 shutdown_connections 2024-03-19T03:00:51.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:51.099+0000 7fe7304d1640 1 -- 172.21.15.33:0/3873682647 wait complete. 2024-03-19T03:00:51.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:50 smithi096 ceph-mon[35946]: pgmap v226: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:00:51.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:50 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2512217822' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:00:51.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:50 smithi155 ceph-mon[35804]: pgmap v226: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:00:51.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:50 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2512217822' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:00:52.111 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T03:00:52.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:00:52.111 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T03:00:52.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.229+0000 7f4590636640 1 -- 172.21.15.33:0/1443333489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4588101100 msgr2=0x7f458810d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:52.231 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.230+0000 7f4590636640 1 --2- 172.21.15.33:0/1443333489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4588101100 0x7f458810d610 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f4580009cd0 tx=0x7f458002c0f0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.230+0000 7f4590636640 1 -- 172.21.15.33:0/1443333489 shutdown_connections 2024-03-19T03:00:52.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.230+0000 7f4590636640 1 --2- 172.21.15.33:0/1443333489 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4588101100 0x7f458810d610 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.230+0000 7f4590636640 1 --2- 172.21.15.33:0/1443333489 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4588100740 0x7f4588100bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.230+0000 7f4590636640 1 --2- 172.21.15.33:0/1443333489 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4588106750 0x7f4588106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.230+0000 7f4590636640 1 -- 172.21.15.33:0/1443333489 >> 172.21.15.33:0/1443333489 conn(0x7f4588077880 msgr2=0x7f4588077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:52.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.230+0000 7f4590636640 1 -- 172.21.15.33:0/1443333489 shutdown_connections 2024-03-19T03:00:52.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.230+0000 7f4590636640 1 -- 172.21.15.33:0/1443333489 wait complete. 2024-03-19T03:00:52.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.230+0000 7f4590636640 1 Processor -- start 2024-03-19T03:00:52.232 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.230+0000 7f4590636640 1 -- start start 2024-03-19T03:00:52.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.231+0000 7f4590636640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4588100740 0x7f4588198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.233 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.231+0000 7f4590636640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4588101100 0x7f4588198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.231+0000 7f4590636640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4588106750 0x7f458819c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.231+0000 7f4590636640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4588110a10 con 0x7f4588106750 2024-03-19T03:00:52.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.231+0000 7f4590636640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4588110890 con 0x7f4588101100 2024-03-19T03:00:52.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.231+0000 7f4590636640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4588110b90 con 0x7f4588100740 2024-03-19T03:00:52.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.231+0000 7f458e3d2640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4588100740 0x7f4588198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.231+0000 7f458ebd3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4588106750 0x7f458819c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.231+0000 7f458dbd1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4588101100 0x7f4588198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.231+0000 7f458e3d2640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4588100740 0x7f4588198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:38154/0 (socket says 172.21.15.33:38154) 2024-03-19T03:00:52.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.231+0000 7f458e3d2640 1 -- 172.21.15.33:0/3177235996 learned_addr learned my addr 172.21.15.33:0/3177235996 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:52.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.231+0000 7f458dbd1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4588101100 0x7f4588198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:36032/0 (socket says 172.21.15.33:36032) 2024-03-19T03:00:52.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.231+0000 7f458ebd3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4588106750 0x7f458819c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58932/0 (socket says 172.21.15.33:58932) 2024-03-19T03:00:52.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.231+0000 7f458e3d2640 1 -- 172.21.15.33:0/3177235996 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4588101100 msgr2=0x7f4588198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:52.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.231+0000 7f458e3d2640 1 --2- 172.21.15.33:0/3177235996 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4588101100 0x7f4588198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.231+0000 7f458e3d2640 1 -- 172.21.15.33:0/3177235996 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4588106750 msgr2=0x7f458819c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:52.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.231+0000 7f458e3d2640 1 --2- 172.21.15.33:0/3177235996 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4588106750 0x7f458819c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.231+0000 7f458e3d2640 1 -- 172.21.15.33:0/3177235996 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f458819cff0 con 0x7f4588100740 2024-03-19T03:00:52.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.231+0000 7f458ebd3640 1 --2- 172.21.15.33:0/3177235996 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4588106750 0x7f458819c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:52.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.231+0000 7f458dbd1640 1 --2- 172.21.15.33:0/3177235996 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4588101100 0x7f4588198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:52.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.232+0000 7f458e3d2640 1 --2- 172.21.15.33:0/3177235996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4588100740 0x7f4588198000 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f458400e9f0 tx=0x7f458400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:52.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.232+0000 7f457f7fe640 1 -- 172.21.15.33:0/3177235996 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458400cd80 con 0x7f4588100740 2024-03-19T03:00:52.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.232+0000 7f457f7fe640 1 -- 172.21.15.33:0/3177235996 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f45840108e0 con 0x7f4588100740 2024-03-19T03:00:52.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.232+0000 7f4590636640 1 -- 172.21.15.33:0/3177235996 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f458819d2e0 con 0x7f4588100740 2024-03-19T03:00:52.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.232+0000 7f457f7fe640 1 -- 172.21.15.33:0/3177235996 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4584018920 con 0x7f4588100740 2024-03-19T03:00:52.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.232+0000 7f4590636640 1 -- 172.21.15.33:0/3177235996 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45881a4b70 con 0x7f4588100740 2024-03-19T03:00:52.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.235+0000 7f457f7fe640 1 -- 172.21.15.33:0/3177235996 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f4584010a90 con 0x7f4588100740 2024-03-19T03:00:52.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.235+0000 7f4593226740 1 Processor -- start 2024-03-19T03:00:52.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.236+0000 7f4593226740 1 -- start start 2024-03-19T03:00:52.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.236+0000 7f457f7fe640 1 --2- 172.21.15.33:0/3177235996 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f45740a29f0 0x7f45740a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.236+0000 7f4593226740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be31e9a380 0x55be31e9a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.236+0000 7f4593226740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be31e9aca0 0x55be31ea3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.236+0000 7f4593226740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be31ea3880 0x55be31ea5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.236+0000 7f4593226740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55be31dfe160 con 0x55be31ea3880 2024-03-19T03:00:52.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.236+0000 7f4593226740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55be31da6e10 con 0x55be31e9a380 2024-03-19T03:00:52.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.236+0000 7f4593226740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55be31dfa200 con 0x55be31e9aca0 2024-03-19T03:00:52.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.236+0000 7f458dbd1640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be31e9aca0 0x55be31ea3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.236+0000 7f458e3d2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be31e9a380 0x55be31e9a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.236+0000 7f458ebd3640 1 --2- 172.21.15.33:0/3177235996 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f45740a29f0 0x7f45740a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.236+0000 7f458dbd1640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be31e9aca0 0x55be31ea3340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:38166/0 (socket says 172.21.15.33:38166) 2024-03-19T03:00:52.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.236+0000 7f458dbd1640 1 -- 172.21.15.33:0/4038041889 learned_addr learned my addr 172.21.15.33:0/4038041889 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:52.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.236+0000 7f457f7fe640 1 -- 172.21.15.33:0/3177235996 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7f45840cf440 con 0x7f4588100740 2024-03-19T03:00:52.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.236+0000 7f458ebd3640 1 --2- 172.21.15.33:0/4038041889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be31ea3880 0x55be31ea5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.237+0000 7f458dbd1640 1 -- 172.21.15.33:0/4038041889 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be31e9a380 msgr2=0x55be31e9a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:52.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.237+0000 7f458dbd1640 1 --2- 172.21.15.33:0/4038041889 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be31e9a380 0x55be31e9a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.237+0000 7f458dbd1640 1 -- 172.21.15.33:0/4038041889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be31ea3880 msgr2=0x55be31ea5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:52.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.237+0000 7f458dbd1640 1 --2- 172.21.15.33:0/4038041889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be31ea3880 0x55be31ea5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.237+0000 7f458dbd1640 1 -- 172.21.15.33:0/4038041889 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be31bda7e0 con 0x55be31e9aca0 2024-03-19T03:00:52.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.237+0000 7f458e3d2640 1 --2- 172.21.15.33:0/4038041889 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be31e9a380 0x55be31e9a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:52.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.237+0000 7f458ebd3640 1 --2- 172.21.15.33:0/3177235996 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f45740a29f0 0x7f45740a4eb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4580005cc0 tx=0x7f4580032650 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:52.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.237+0000 7f458dbd1640 1 --2- 172.21.15.33:0/4038041889 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be31e9aca0 0x55be31ea3340 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f45780114f0 tx=0x7f45780119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=292ea50e45f28d96 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:52.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.237+0000 7f457cbfe640 1 -- 172.21.15.33:0/4038041889 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f457801e070 con 0x55be31e9aca0 2024-03-19T03:00:52.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.237+0000 7f457cbfe640 1 -- 172.21.15.33:0/4038041889 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f45780035f0 con 0x55be31e9aca0 2024-03-19T03:00:52.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.237+0000 7f457cbfe640 1 -- 172.21.15.33:0/4038041889 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f457800f7c0 con 0x55be31e9aca0 2024-03-19T03:00:52.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.238+0000 7f4593226740 1 -- 172.21.15.33:0/4038041889 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be31e9aca0 msgr2=0x55be31ea3340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:52.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.238+0000 7f4593226740 1 --2- 172.21.15.33:0/4038041889 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be31e9aca0 0x55be31ea3340 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f45780114f0 tx=0x7f45780119b0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.238+0000 7f4593226740 1 -- 172.21.15.33:0/4038041889 shutdown_connections 2024-03-19T03:00:52.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.238+0000 7f4593226740 1 --2- 172.21.15.33:0/4038041889 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be31e9aca0 0x55be31ea3340 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.238+0000 7f4593226740 1 --2- 172.21.15.33:0/4038041889 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be31ea3880 0x55be31ea5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.238+0000 7f4593226740 1 --2- 172.21.15.33:0/4038041889 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be31e9a380 0x55be31e9a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.238+0000 7f4593226740 1 -- 172.21.15.33:0/4038041889 >> 172.21.15.33:0/4038041889 conn(0x55be31e94ab0 msgr2=0x55be31e94e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:52.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.238+0000 7f4593226740 1 -- 172.21.15.33:0/4038041889 shutdown_connections 2024-03-19T03:00:52.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.238+0000 7f4593226740 1 -- 172.21.15.33:0/4038041889 wait complete. 2024-03-19T03:00:52.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.238+0000 7f4593226740 1 -- start start 2024-03-19T03:00:52.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.238+0000 7f4593226740 1 Processor -- start 2024-03-19T03:00:52.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.239+0000 7f4593226740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be31e9a380 0x55be31ea07a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.239+0000 7f4593226740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be31e9aca0 0x55be31ea0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.239+0000 7f4593226740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be31ea3880 0x55be31ea1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.239+0000 7f4593226740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55be31ea6670 con 0x55be31e9a380 2024-03-19T03:00:52.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.239+0000 7f4593226740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55be31dba1a0 con 0x55be31e9aca0 2024-03-19T03:00:52.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.239+0000 7f4593226740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55be31ea67f0 con 0x55be31ea3880 2024-03-19T03:00:52.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.239+0000 7f458e3d2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be31e9a380 0x55be31ea07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.239+0000 7f458e3d2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be31e9a380 0x55be31ea07a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58940/0 (socket says 172.21.15.33:58940) 2024-03-19T03:00:52.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.239+0000 7f458e3d2640 1 -- 172.21.15.33:0/88591574 learned_addr learned my addr 172.21.15.33:0/88591574 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:52.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.239+0000 7f458dbd1640 1 --2- 172.21.15.33:0/88591574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be31e9aca0 0x55be31ea0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.239+0000 7f4593226740 10 client.0 _ll_register_callbacks cb 0x55be31b11c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:00:52.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.239+0000 7f458ebd3640 1 --2- 172.21.15.33:0/88591574 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be31ea3880 0x55be31ea1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.239+0000 7f458dbd1640 1 -- 172.21.15.33:0/88591574 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be31ea3880 msgr2=0x55be31ea1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:52.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.239+0000 7f458dbd1640 1 --2- 172.21.15.33:0/88591574 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be31ea3880 0x55be31ea1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.246 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.239+0000 7f458dbd1640 1 -- 172.21.15.33:0/88591574 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be31e9a380 msgr2=0x55be31ea07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:52.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.239+0000 7f458dbd1640 1 --2- 172.21.15.33:0/88591574 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be31e9a380 0x55be31ea07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.239+0000 7f458dbd1640 1 -- 172.21.15.33:0/88591574 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be31c3af70 con 0x55be31e9aca0 2024-03-19T03:00:52.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.239+0000 7f458e3d2640 1 --2- 172.21.15.33:0/88591574 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be31e9a380 0x55be31ea07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:52.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.239+0000 7f458dbd1640 1 --2- 172.21.15.33:0/88591574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be31e9aca0 0x55be31ea0ce0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f45780114c0 tx=0x7f45780041c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f05e225fa6d096cc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:52.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.239+0000 7f4571ffb640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T03:00:52.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.240+0000 7f4571ffb640 1 -- 172.21.15.33:0/88591574 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f457801e070 con 0x55be31e9aca0 2024-03-19T03:00:52.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.240+0000 7f4593226740 1 -- 172.21.15.33:0/88591574 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45780035f0 con 0x55be31e9aca0 2024-03-19T03:00:52.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.240+0000 7f4571ffb640 1 -- 172.21.15.33:0/88591574 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7f4578005040 con 0x55be31e9aca0 2024-03-19T03:00:52.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.240+0000 7f4571ffb640 1 -- 172.21.15.33:0/88591574 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4578031050 con 0x55be31e9aca0 2024-03-19T03:00:52.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.240+0000 7f4593226740 10 client.27089 fetch_fsmap learned FSMap version 25 2024-03-19T03:00:52.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.240+0000 7f4593226740 1 -- 172.21.15.33:0/88591574 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55be31ad2170 con 0x55be31e9aca0 2024-03-19T03:00:52.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.240+0000 7f4571ffb640 1 -- 172.21.15.33:0/88591574 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f45780035f0 con 0x55be31e9aca0 2024-03-19T03:00:52.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.240+0000 7f4571ffb640 1 -- 172.21.15.33:0/88591574 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f4578004b70 con 0x55be31e9aca0 2024-03-19T03:00:52.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.240+0000 7f4593226740 10 client.27089 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:00:52.247 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.240+0000 7f4593226740 10 client.27089 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:00:52.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.240+0000 7f4593226740 20 client.27089 populate_metadata read hostname 'smithi033' 2024-03-19T03:00:52.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.240+0000 7f4593226740 1 --2- 172.21.15.33:0/88591574 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55be31eb3af0 0x55be31eb5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.240+0000 7f4593226740 4 client.27089 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:00:52.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.240+0000 7f4593226740 1 -- 172.21.15.33:0/88591574 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55be31e0b6a0 con 0x55be31eb3af0 2024-03-19T03:00:52.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.241+0000 7f458e3d2640 1 --2- 172.21.15.33:0/88591574 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55be31eb3af0 0x55be31eb5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.241+0000 7f458e3d2640 1 --2- 172.21.15.33:0/88591574 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55be31eb3af0 0x55be31eb5ee0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26dc88bf95ba3104 server_cookie=dd912219506a5ba2 in_seq=0 out_seq=0 2024-03-19T03:00:52.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.241+0000 7f4571ffb640 10 client.27089 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:00:52.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.241+0000 7f4571ffb640 1 -- 172.21.15.33:0/88591574 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55be31e0b6a0 con 0x55be31eb3af0 2024-03-19T03:00:52.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.241+0000 7f4571ffb640 10 client.27089 handle_command_reply: tid=0 2024-03-19T03:00:52.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.242+0000 7f4593226740 1 client.27089 shutdown 2024-03-19T03:00:52.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.242+0000 7f4593226740 1 -- 172.21.15.33:0/88591574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be31e9aca0 msgr2=0x55be31ea0ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:52.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.242+0000 7f4593226740 1 --2- 172.21.15.33:0/88591574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be31e9aca0 0x55be31ea0ce0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f45780114c0 tx=0x7f45780041c0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.242+0000 7f4593226740 1 -- 172.21.15.33:0/88591574 shutdown_connections 2024-03-19T03:00:52.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.242+0000 7f4593226740 1 --2- 172.21.15.33:0/88591574 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55be31eb3af0 0x55be31eb5ee0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.248 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.242+0000 7f4593226740 1 --2- 172.21.15.33:0/88591574 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be31ea3880 0x55be31ea1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.242+0000 7f4571ffb640 0 client.27089 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:00:52.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.242+0000 7f4593226740 1 --2- 172.21.15.33:0/88591574 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be31e9aca0 0x55be31ea0ce0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.242+0000 7f4593226740 1 --2- 172.21.15.33:0/88591574 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be31e9a380 0x55be31ea07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.242+0000 7f4593226740 1 -- 172.21.15.33:0/88591574 >> 172.21.15.33:0/88591574 conn(0x55be31e94ab0 msgr2=0x55be31e96800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:52.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.242+0000 7f4593226740 1 -- 172.21.15.33:0/88591574 shutdown_connections 2024-03-19T03:00:52.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.242+0000 7f4593226740 1 -- 172.21.15.33:0/88591574 wait complete. 2024-03-19T03:00:52.249 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.242+0000 7f4593226740 20 client.27089 trim_cache size 0 max 16384 2024-03-19T03:00:52.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.260+0000 7f4593226740 1 Processor -- start 2024-03-19T03:00:52.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.260+0000 7f4593226740 1 -- start start 2024-03-19T03:00:52.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.260+0000 7f4593226740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be31e9a380 0x55be31ea2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.260+0000 7f4593226740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be31e9aca0 0x55be31eaf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.260+0000 7f4593226740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be31ea3880 0x55be31e939b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.260+0000 7f4593226740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f45881b1510 con 0x55be31e9a380 2024-03-19T03:00:52.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.260+0000 7f4593226740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f45881aede0 con 0x55be31e9aca0 2024-03-19T03:00:52.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.260+0000 7f4593226740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f45881acc50 con 0x55be31ea3880 2024-03-19T03:00:52.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.260+0000 7f458e3d2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be31e9a380 0x55be31ea2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.260+0000 7f458e3d2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be31e9a380 0x55be31ea2ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58950/0 (socket says 172.21.15.33:58950) 2024-03-19T03:00:52.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.260+0000 7f458e3d2640 1 -- 172.21.15.33:0/2277255683 learned_addr learned my addr 172.21.15.33:0/2277255683 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:52.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.260+0000 7f458dbd1640 1 --2- 172.21.15.33:0/2277255683 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be31e9aca0 0x55be31eaf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.260+0000 7f458ebd3640 1 --2- 172.21.15.33:0/2277255683 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be31ea3880 0x55be31e939b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.260+0000 7f458e3d2640 1 -- 172.21.15.33:0/2277255683 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be31ea3880 msgr2=0x55be31e939b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:52.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.260+0000 7f458e3d2640 1 --2- 172.21.15.33:0/2277255683 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be31ea3880 0x55be31e939b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.260+0000 7f458e3d2640 1 -- 172.21.15.33:0/2277255683 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be31e9aca0 msgr2=0x55be31eaf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:52.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.260+0000 7f458e3d2640 1 --2- 172.21.15.33:0/2277255683 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be31e9aca0 0x55be31eaf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.261+0000 7f458e3d2640 1 -- 172.21.15.33:0/2277255683 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be31e948e0 con 0x55be31e9a380 2024-03-19T03:00:52.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.261+0000 7f458e3d2640 1 --2- 172.21.15.33:0/2277255683 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be31e9a380 0x55be31ea2ed0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f4584098150 tx=0x7f458401a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33c4160f4df90761 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:52.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.261+0000 7f4573fff640 1 -- 172.21.15.33:0/2277255683 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4584078e60 con 0x55be31e9a380 2024-03-19T03:00:52.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.261+0000 7f4573fff640 1 -- 172.21.15.33:0/2277255683 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f458409f820 con 0x55be31e9a380 2024-03-19T03:00:52.264 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.261+0000 7f4573fff640 1 -- 172.21.15.33:0/2277255683 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4584026040 con 0x55be31e9a380 2024-03-19T03:00:52.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.261+0000 7f4593226740 1 -- 172.21.15.33:0/2277255683 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be31e9a380 msgr2=0x55be31ea2ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:52.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.261+0000 7f4593226740 1 --2- 172.21.15.33:0/2277255683 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be31e9a380 0x55be31ea2ed0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f4584098150 tx=0x7f458401a040 comp rx=0 tx=0).stop 2024-03-19T03:00:52.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.262+0000 7f4593226740 1 -- 172.21.15.33:0/2277255683 shutdown_connections 2024-03-19T03:00:52.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.262+0000 7f4593226740 1 --2- 172.21.15.33:0/2277255683 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be31ea3880 0x55be31e939b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.262+0000 7f4593226740 1 --2- 172.21.15.33:0/2277255683 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be31e9aca0 0x55be31eaf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.262+0000 7f4593226740 1 --2- 172.21.15.33:0/2277255683 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be31e9a380 0x55be31ea2ed0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.262+0000 7f4593226740 1 -- 172.21.15.33:0/2277255683 >> 172.21.15.33:0/2277255683 conn(0x55be31e94ab0 msgr2=0x55be31e932a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:52.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.262+0000 7f4593226740 1 -- 172.21.15.33:0/2277255683 shutdown_connections 2024-03-19T03:00:52.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.262+0000 7f4593226740 1 -- 172.21.15.33:0/2277255683 wait complete. 2024-03-19T03:00:52.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.262+0000 7f4593226740 1 -- start start 2024-03-19T03:00:52.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.262+0000 7f4593226740 1 Processor -- start 2024-03-19T03:00:52.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.262+0000 7f4593226740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be31e9a380 0x55be31e9d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.262+0000 7f4593226740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be31e9aca0 0x55be31e9d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.262+0000 7f4593226740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be31ea3880 0x55be31e9de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.262+0000 7f4593226740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f45881ad2a0 con 0x55be31ea3880 2024-03-19T03:00:52.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.262+0000 7f4593226740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f45881ae710 con 0x55be31e9a380 2024-03-19T03:00:52.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.262+0000 7f4593226740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f45881ad970 con 0x55be31e9aca0 2024-03-19T03:00:52.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.262+0000 7f4593226740 10 client.0 _ll_register_callbacks cb 0x7f457801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:00:52.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.262+0000 7f458ebd3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be31ea3880 0x55be31e9de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.262+0000 7f458e3d2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be31e9a380 0x55be31e9d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.262+0000 7f458e3d2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be31e9a380 0x55be31e9d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:36076/0 (socket says 172.21.15.33:36076) 2024-03-19T03:00:52.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.262+0000 7f458e3d2640 1 -- 172.21.15.33:0/1588290973 learned_addr learned my addr 172.21.15.33:0/1588290973 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:52.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.262+0000 7f458ebd3640 1 --2- 172.21.15.33:0/1588290973 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be31ea3880 0x55be31e9de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58954/0 (socket says 172.21.15.33:58954) 2024-03-19T03:00:52.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.262+0000 7f458dbd1640 1 --2- 172.21.15.33:0/1588290973 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be31e9aca0 0x55be31e9d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.263+0000 7f458ebd3640 1 -- 172.21.15.33:0/1588290973 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be31e9aca0 msgr2=0x55be31e9d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:52.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.263+0000 7f458ebd3640 1 --2- 172.21.15.33:0/1588290973 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be31e9aca0 0x55be31e9d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.263+0000 7f458ebd3640 1 -- 172.21.15.33:0/1588290973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be31e9a380 msgr2=0x55be31e9d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:52.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.263+0000 7f458ebd3640 1 --2- 172.21.15.33:0/1588290973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be31e9a380 0x55be31e9d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.263+0000 7f458ebd3640 1 -- 172.21.15.33:0/1588290973 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be31f18d40 con 0x55be31ea3880 2024-03-19T03:00:52.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.263+0000 7f458e3d2640 1 --2- 172.21.15.33:0/1588290973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be31e9a380 0x55be31e9d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:52.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.263+0000 7f458ebd3640 1 --2- 172.21.15.33:0/1588290973 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be31ea3880 0x55be31e9de30 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f458002c630 tx=0x7f4580005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=376ac43227434a3e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:52.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.263+0000 7f4571ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:00:52.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.263+0000 7f4571ffb640 1 -- 172.21.15.33:0/1588290973 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4580039040 con 0x55be31ea3880 2024-03-19T03:00:52.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.263+0000 7f4571ffb640 1 -- 172.21.15.33:0/1588290973 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7f45800047a0 con 0x55be31ea3880 2024-03-19T03:00:52.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.263+0000 7f4593226740 1 -- 172.21.15.33:0/1588290973 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f458409f820 con 0x55be31ea3880 2024-03-19T03:00:52.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.263+0000 7f4571ffb640 1 -- 172.21.15.33:0/1588290973 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f458005e050 con 0x55be31ea3880 2024-03-19T03:00:52.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.263+0000 7f4593226740 10 client.17274 fetch_fsmap learned FSMap version 25 2024-03-19T03:00:52.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.263+0000 7f4593226740 1 -- 172.21.15.33:0/1588290973 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55be31f1a080 con 0x55be31ea3880 2024-03-19T03:00:52.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.263+0000 7f4571ffb640 1 -- 172.21.15.33:0/1588290973 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f458409f820 con 0x55be31ea3880 2024-03-19T03:00:52.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.263+0000 7f4571ffb640 1 -- 172.21.15.33:0/1588290973 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f458003f070 con 0x55be31ea3880 2024-03-19T03:00:52.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.263+0000 7f4593226740 10 client.17274 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:00:52.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.264+0000 7f4593226740 10 client.17274 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:00:52.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.264+0000 7f4593226740 20 client.17274 populate_metadata read hostname 'smithi033' 2024-03-19T03:00:52.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.264+0000 7f4593226740 1 --2- 172.21.15.33:0/1588290973 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55be31eb3af0 0x55be31f1c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.264+0000 7f4593226740 4 client.17274 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T03:00:52.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.264+0000 7f4593226740 1 -- 172.21.15.33:0/1588290973 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55be31e059f0 con 0x55be31eb3af0 2024-03-19T03:00:52.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.264+0000 7f458e3d2640 1 --2- 172.21.15.33:0/1588290973 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55be31eb3af0 0x55be31f1c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.264+0000 7f458e3d2640 1 --2- 172.21.15.33:0/1588290973 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55be31eb3af0 0x55be31f1c860 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9b3f45ce4139945 server_cookie=e2ab42372868cf2 in_seq=0 out_seq=0 2024-03-19T03:00:52.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.264+0000 7f4571ffb640 10 client.17274 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:00:52.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.265+0000 7f4571ffb640 1 -- 172.21.15.33:0/1588290973 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55be31e059f0 con 0x55be31eb3af0 2024-03-19T03:00:52.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.265+0000 7f4571ffb640 10 client.17274 handle_command_reply: tid=0 2024-03-19T03:00:52.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.265+0000 7f4593226740 1 client.17274 shutdown 2024-03-19T03:00:52.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.265+0000 7f4593226740 1 -- 172.21.15.33:0/1588290973 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be31ea3880 msgr2=0x55be31e9de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:52.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.265+0000 7f4593226740 1 --2- 172.21.15.33:0/1588290973 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be31ea3880 0x55be31e9de30 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f458002c630 tx=0x7f4580005c50 comp rx=0 tx=0).stop 2024-03-19T03:00:52.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.265+0000 7f4593226740 1 -- 172.21.15.33:0/1588290973 shutdown_connections 2024-03-19T03:00:52.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.265+0000 7f4593226740 1 --2- 172.21.15.33:0/1588290973 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55be31eb3af0 0x55be31f1c860 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.265+0000 7f4593226740 1 --2- 172.21.15.33:0/1588290973 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be31e9aca0 0x55be31e9d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.265+0000 7f4593226740 1 --2- 172.21.15.33:0/1588290973 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be31ea3880 0x55be31e9de30 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.265+0000 7f4571ffb640 0 client.17274 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:00:52.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.265+0000 7f4593226740 1 --2- 172.21.15.33:0/1588290973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be31e9a380 0x55be31e9d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.272 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.265+0000 7f4593226740 1 -- 172.21.15.33:0/1588290973 >> 172.21.15.33:0/1588290973 conn(0x55be31e94ab0 msgr2=0x55be31e93b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:52.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.265+0000 7f4593226740 1 -- 172.21.15.33:0/1588290973 shutdown_connections 2024-03-19T03:00:52.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.265+0000 7f4593226740 1 -- 172.21.15.33:0/1588290973 wait complete. 2024-03-19T03:00:52.273 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.266+0000 7f4593226740 20 client.17274 trim_cache size 0 max 16384 2024-03-19T03:00:52.273 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T03:00:52.273 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-03-19T03:00:52.273 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "24b1aad6-69ce-4904-b8e4-9048940c3e6d", 2024-03-19T03:00:52.273 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-03-19T03:00:52.273 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T03:00:52.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.266+0000 7f4590636640 1 -- 172.21.15.33:0/3177235996 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f45740a29f0 msgr2=0x7f45740a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:52.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.266+0000 7f4590636640 1 --2- 172.21.15.33:0/3177235996 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f45740a29f0 0x7f45740a4eb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4580005cc0 tx=0x7f4580032650 comp rx=0 tx=0).stop 2024-03-19T03:00:52.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.266+0000 7f4590636640 1 -- 172.21.15.33:0/3177235996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4588100740 msgr2=0x7f4588198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:52.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.266+0000 7f4590636640 1 --2- 172.21.15.33:0/3177235996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4588100740 0x7f4588198000 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f458400e9f0 tx=0x7f458400eeb0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.266+0000 7f4590636640 1 -- 172.21.15.33:0/3177235996 shutdown_connections 2024-03-19T03:00:52.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.266+0000 7f4590636640 1 --2- 172.21.15.33:0/3177235996 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f45740a29f0 0x7f45740a4eb0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.266+0000 7f4590636640 1 --2- 172.21.15.33:0/3177235996 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4588106750 0x7f458819c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.266+0000 7f4590636640 1 --2- 172.21.15.33:0/3177235996 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4588101100 0x7f4588198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.266+0000 7f4590636640 1 --2- 172.21.15.33:0/3177235996 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4588100740 0x7f4588198000 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.266+0000 7f4590636640 1 -- 172.21.15.33:0/3177235996 >> 172.21.15.33:0/3177235996 conn(0x7f4588077880 msgr2=0x7f458806a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:52.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.266+0000 7f4590636640 1 -- 172.21.15.33:0/3177235996 shutdown_connections 2024-03-19T03:00:52.275 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.266+0000 7f4590636640 1 -- 172.21.15.33:0/3177235996 wait complete. 2024-03-19T03:00:52.278 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T03:00:52.278 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T03:00:52.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.383+0000 7f011017e640 1 -- 172.21.15.33:0/1563118164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0108101050 msgr2=0x7f010810d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:52.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.383+0000 7f011017e640 1 --2- 172.21.15.33:0/1563118164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0108101050 0x7f010810d560 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f0100009cd0 tx=0x7f010002c0f0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.383+0000 7f011017e640 1 -- 172.21.15.33:0/1563118164 shutdown_connections 2024-03-19T03:00:52.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.383+0000 7f011017e640 1 --2- 172.21.15.33:0/1563118164 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0108100690 0x7f0108100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.383+0000 7f011017e640 1 --2- 172.21.15.33:0/1563118164 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0108101050 0x7f010810d560 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.383+0000 7f011017e640 1 --2- 172.21.15.33:0/1563118164 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01081066a0 0x7f0108106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.383+0000 7f011017e640 1 -- 172.21.15.33:0/1563118164 >> 172.21.15.33:0/1563118164 conn(0x7f0108077820 msgr2=0x7f0108077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:52.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.383+0000 7f011017e640 1 -- 172.21.15.33:0/1563118164 shutdown_connections 2024-03-19T03:00:52.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.383+0000 7f011017e640 1 -- 172.21.15.33:0/1563118164 wait complete. 2024-03-19T03:00:52.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.383+0000 7f011017e640 1 Processor -- start 2024-03-19T03:00:52.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.383+0000 7f011017e640 1 -- start start 2024-03-19T03:00:52.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.383+0000 7f011017e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0108100690 0x7f0108198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.383+0000 7f011017e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0108101050 0x7f0108198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.383+0000 7f011017e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01081066a0 0x7f010819c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.383+0000 7f011017e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0108110960 con 0x7f0108101050 2024-03-19T03:00:52.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.383+0000 7f011017e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f01081107e0 con 0x7f01081066a0 2024-03-19T03:00:52.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.383+0000 7f011017e640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f0108110ae0 con 0x7f0108100690 2024-03-19T03:00:52.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.384+0000 7f010d719640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0108101050 0x7f0108198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.384+0000 7f010d719640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0108101050 0x7f0108198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58958/0 (socket says 172.21.15.33:58958) 2024-03-19T03:00:52.387 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.384+0000 7f010d719640 1 -- 172.21.15.33:0/868650382 learned_addr learned my addr 172.21.15.33:0/868650382 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:52.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.384+0000 7f010e71b640 1 --2- 172.21.15.33:0/868650382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01081066a0 0x7f010819c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.384+0000 7f010df1a640 1 --2- 172.21.15.33:0/868650382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0108100690 0x7f0108198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.384+0000 7f010d719640 1 -- 172.21.15.33:0/868650382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0108100690 msgr2=0x7f0108198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:52.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.384+0000 7f010d719640 1 --2- 172.21.15.33:0/868650382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0108100690 0x7f0108198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.384+0000 7f010d719640 1 -- 172.21.15.33:0/868650382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01081066a0 msgr2=0x7f010819c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:52.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.384+0000 7f010d719640 1 --2- 172.21.15.33:0/868650382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01081066a0 0x7f010819c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.384+0000 7f010d719640 1 -- 172.21.15.33:0/868650382 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f010819d020 con 0x7f0108101050 2024-03-19T03:00:52.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.384+0000 7f010d719640 1 --2- 172.21.15.33:0/868650382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0108101050 0x7f0108198570 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f00f800e9f0 tx=0x7f00f800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:52.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.384+0000 7f00feffd640 1 -- 172.21.15.33:0/868650382 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f800cd80 con 0x7f0108101050 2024-03-19T03:00:52.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.384+0000 7f00feffd640 1 -- 172.21.15.33:0/868650382 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f00f80054d0 con 0x7f0108101050 2024-03-19T03:00:52.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.384+0000 7f011017e640 1 -- 172.21.15.33:0/868650382 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f010819d310 con 0x7f0108101050 2024-03-19T03:00:52.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.384+0000 7f00feffd640 1 -- 172.21.15.33:0/868650382 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f8010490 con 0x7f0108101050 2024-03-19T03:00:52.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.384+0000 7f011017e640 1 -- 172.21.15.33:0/868650382 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01081a4ba0 con 0x7f0108101050 2024-03-19T03:00:52.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.386+0000 7f00feffd640 1 -- 172.21.15.33:0/868650382 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f00f8005020 con 0x7f0108101050 2024-03-19T03:00:52.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.387+0000 7f0112d6e740 1 Processor -- start 2024-03-19T03:00:52.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.387+0000 7f0112d6e740 1 -- start start 2024-03-19T03:00:52.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.387+0000 7f00feffd640 1 --2- 172.21.15.33:0/868650382 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f00f40a2970 0x7f00f40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.387+0000 7f0112d6e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5602ceaaca30 0x5602ceaace10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.387+0000 7f0112d6e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602ceaad350 0x5602ceab59f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.387+0000 7f0112d6e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ceab5f30 0x5602ceab8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.387+0000 7f0112d6e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5602cea10500 con 0x5602ceaad350 2024-03-19T03:00:52.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.387+0000 7f0112d6e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5602ce9b8be0 con 0x5602ceab5f30 2024-03-19T03:00:52.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.387+0000 7f0112d6e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5602cea0c5a0 con 0x5602ceaaca30 2024-03-19T03:00:52.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.387+0000 7f010d719640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602ceaad350 0x5602ceab59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.387+0000 7f010e71b640 1 --2- 172.21.15.33:0/868650382 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f00f40a2970 0x7f00f40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.387+0000 7f010d719640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602ceaad350 0x5602ceab59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58960/0 (socket says 172.21.15.33:58960) 2024-03-19T03:00:52.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.388+0000 7f010d719640 1 -- 172.21.15.33:0/4080243752 learned_addr learned my addr 172.21.15.33:0/4080243752 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:52.395 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.388+0000 7f010df1a640 1 --2- 172.21.15.33:0/4080243752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5602ceaaca30 0x5602ceaace10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.388+0000 7f00feffd640 1 -- 172.21.15.33:0/868650382 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7f00f80cf5a0 con 0x7f0108101050 2024-03-19T03:00:52.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.388+0000 7f010e71b640 1 --2- 172.21.15.33:0/4080243752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ceab5f30 0x5602ceab8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.388+0000 7f010df1a640 1 -- 172.21.15.33:0/4080243752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ceab5f30 msgr2=0x5602ceab8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:52.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.388+0000 7f010df1a640 1 --2- 172.21.15.33:0/4080243752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ceab5f30 0x5602ceab8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.388+0000 7f010df1a640 1 -- 172.21.15.33:0/4080243752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602ceaad350 msgr2=0x5602ceab59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:52.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.388+0000 7f010df1a640 1 --2- 172.21.15.33:0/4080243752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602ceaad350 0x5602ceab59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.388+0000 7f010df1a640 1 -- 172.21.15.33:0/4080243752 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602ce9da000 con 0x5602ceaaca30 2024-03-19T03:00:52.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.388+0000 7f010d719640 1 --2- 172.21.15.33:0/4080243752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602ceaad350 0x5602ceab59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:52.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.388+0000 7f010e71b640 1 --2- 172.21.15.33:0/868650382 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f00f40a2970 0x7f00f40a4e30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0100005cc0 tx=0x7f0100032450 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:52.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.388+0000 7f010df1a640 1 --2- 172.21.15.33:0/4080243752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5602ceaaca30 0x5602ceaace10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0104009c20 tx=0x7f0104011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aeaa5ef52bd9922c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:52.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.388+0000 7f00f3fff640 1 -- 172.21.15.33:0/4080243752 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0104004030 con 0x5602ceaaca30 2024-03-19T03:00:52.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.388+0000 7f00f3fff640 1 -- 172.21.15.33:0/4080243752 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f01040035f0 con 0x5602ceaaca30 2024-03-19T03:00:52.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.388+0000 7f00f3fff640 1 -- 172.21.15.33:0/4080243752 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0104019670 con 0x5602ceaaca30 2024-03-19T03:00:52.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.389+0000 7f0112d6e740 1 -- 172.21.15.33:0/4080243752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5602ceaaca30 msgr2=0x5602ceaace10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:52.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.389+0000 7f0112d6e740 1 --2- 172.21.15.33:0/4080243752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5602ceaaca30 0x5602ceaace10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0104009c20 tx=0x7f0104011810 comp rx=0 tx=0).stop 2024-03-19T03:00:52.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.389+0000 7f0112d6e740 1 -- 172.21.15.33:0/4080243752 shutdown_connections 2024-03-19T03:00:52.396 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.389+0000 7f0112d6e740 1 --2- 172.21.15.33:0/4080243752 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ceab5f30 0x5602ceab8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.389+0000 7f0112d6e740 1 --2- 172.21.15.33:0/4080243752 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602ceaad350 0x5602ceab59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.389+0000 7f0112d6e740 1 --2- 172.21.15.33:0/4080243752 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5602ceaaca30 0x5602ceaace10 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.389+0000 7f0112d6e740 1 -- 172.21.15.33:0/4080243752 >> 172.21.15.33:0/4080243752 conn(0x5602ceaa7160 msgr2=0x5602ceaa7540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:52.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.389+0000 7f0112d6e740 1 -- 172.21.15.33:0/4080243752 shutdown_connections 2024-03-19T03:00:52.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.389+0000 7f0112d6e740 1 -- 172.21.15.33:0/4080243752 wait complete. 2024-03-19T03:00:52.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.389+0000 7f0112d6e740 1 -- start start 2024-03-19T03:00:52.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.389+0000 7f0112d6e740 1 Processor -- start 2024-03-19T03:00:52.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.390+0000 7f0112d6e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5602ceaaca30 0x5602ceab2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.390+0000 7f0112d6e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ceaad350 0x5602ceab3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.390+0000 7f0112d6e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602ceab5f30 0x5602ceab3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.390+0000 7f0112d6e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5602ceab8d20 con 0x5602ceab5f30 2024-03-19T03:00:52.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.390+0000 7f0112d6e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5602ce9cc160 con 0x5602ceaad350 2024-03-19T03:00:52.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.390+0000 7f0112d6e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5602ceab8ea0 con 0x5602ceaaca30 2024-03-19T03:00:52.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.390+0000 7f0112d6e740 10 client.0 _ll_register_callbacks cb 0x5602cea08a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:00:52.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.390+0000 7f010e71b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602ceab5f30 0x5602ceab3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.390+0000 7f010e71b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602ceab5f30 0x5602ceab3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58962/0 (socket says 172.21.15.33:58962) 2024-03-19T03:00:52.397 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.390+0000 7f010e71b640 1 -- 172.21.15.33:0/3334124510 learned_addr learned my addr 172.21.15.33:0/3334124510 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:52.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.390+0000 7f010df1a640 1 --2- 172.21.15.33:0/3334124510 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5602ceaaca30 0x5602ceab2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.390+0000 7f010d719640 1 --2- 172.21.15.33:0/3334124510 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ceaad350 0x5602ceab3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.390+0000 7f010e71b640 1 -- 172.21.15.33:0/3334124510 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5602ceaaca30 msgr2=0x5602ceab2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:52.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.390+0000 7f010e71b640 1 --2- 172.21.15.33:0/3334124510 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5602ceaaca30 0x5602ceab2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.390+0000 7f010e71b640 1 -- 172.21.15.33:0/3334124510 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ceaad350 msgr2=0x5602ceab3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:52.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.390+0000 7f010e71b640 1 --2- 172.21.15.33:0/3334124510 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ceaad350 0x5602ceab3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.390+0000 7f010e71b640 1 -- 172.21.15.33:0/3334124510 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602ce7ee7d0 con 0x5602ceab5f30 2024-03-19T03:00:52.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.390+0000 7f010e71b640 1 --2- 172.21.15.33:0/3334124510 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602ceab5f30 0x5602ceab3980 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f0100036000 tx=0x7f010002c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e718105c1d578dc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:52.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.390+0000 7f00f17fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:00:52.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.391+0000 7f00f17fa640 1 -- 172.21.15.33:0/3334124510 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0100039040 con 0x5602ceab5f30 2024-03-19T03:00:52.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.391+0000 7f00f17fa640 1 -- 172.21.15.33:0/3334124510 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7f01000045c0 con 0x5602ceab5f30 2024-03-19T03:00:52.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.391+0000 7f0112d6e740 1 -- 172.21.15.33:0/3334124510 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01040035f0 con 0x5602ceab5f30 2024-03-19T03:00:52.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.391+0000 7f00f17fa640 1 -- 172.21.15.33:0/3334124510 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0100059050 con 0x5602ceab5f30 2024-03-19T03:00:52.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.391+0000 7f0112d6e740 10 client.17286 fetch_fsmap learned FSMap version 25 2024-03-19T03:00:52.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.391+0000 7f0112d6e740 1 -- 172.21.15.33:0/3334124510 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5602ce7243f0 con 0x5602ceab5f30 2024-03-19T03:00:52.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.391+0000 7f00f17fa640 1 -- 172.21.15.33:0/3334124510 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f01040035f0 con 0x5602ceab5f30 2024-03-19T03:00:52.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.391+0000 7f00f17fa640 1 -- 172.21.15.33:0/3334124510 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f010003f030 con 0x5602ceab5f30 2024-03-19T03:00:52.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.391+0000 7f0112d6e740 10 client.17286 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:00:52.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.391+0000 7f0112d6e740 10 client.17286 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:00:52.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.391+0000 7f0112d6e740 20 client.17286 populate_metadata read hostname 'smithi033' 2024-03-19T03:00:52.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.391+0000 7f0112d6e740 1 --2- 172.21.15.33:0/3334124510 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5602ceac62a0 0x5602ceac86f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.391+0000 7f0112d6e740 4 client.17286 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:00:52.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.391+0000 7f0112d6e740 1 -- 172.21.15.33:0/3334124510 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602cea1da40 con 0x5602ceac62a0 2024-03-19T03:00:52.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.391+0000 7f010df1a640 1 --2- 172.21.15.33:0/3334124510 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5602ceac62a0 0x5602ceac86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.392+0000 7f00f17fa640 10 client.17286 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:00:52.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.392+0000 7f010df1a640 1 --2- 172.21.15.33:0/3334124510 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5602ceac62a0 0x5602ceac86f0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e60edbcf1aca34f4 server_cookie=83ab3dc9b0204b59 in_seq=0 out_seq=0 2024-03-19T03:00:52.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.392+0000 7f00f17fa640 1 -- 172.21.15.33:0/3334124510 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5602cea1da40 con 0x5602ceac62a0 2024-03-19T03:00:52.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.392+0000 7f00f17fa640 10 client.17286 handle_command_reply: tid=0 2024-03-19T03:00:52.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.392+0000 7f0112d6e740 1 client.17286 shutdown 2024-03-19T03:00:52.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.392+0000 7f0112d6e740 1 -- 172.21.15.33:0/3334124510 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602ceab5f30 msgr2=0x5602ceab3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:52.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.392+0000 7f0112d6e740 1 --2- 172.21.15.33:0/3334124510 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602ceab5f30 0x5602ceab3980 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f0100036000 tx=0x7f010002c0f0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.393+0000 7f0112d6e740 1 -- 172.21.15.33:0/3334124510 shutdown_connections 2024-03-19T03:00:52.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.393+0000 7f0112d6e740 1 --2- 172.21.15.33:0/3334124510 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5602ceac62a0 0x5602ceac86f0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.393+0000 7f0112d6e740 1 --2- 172.21.15.33:0/3334124510 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602ceab5f30 0x5602ceab3980 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.393+0000 7f0112d6e740 1 --2- 172.21.15.33:0/3334124510 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ceaad350 0x5602ceab3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.393+0000 7f00f17fa640 0 client.17286 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:00:52.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.393+0000 7f0112d6e740 1 --2- 172.21.15.33:0/3334124510 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5602ceaaca30 0x5602ceab2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.393+0000 7f0112d6e740 1 -- 172.21.15.33:0/3334124510 >> 172.21.15.33:0/3334124510 conn(0x5602ceaa7160 msgr2=0x5602ceaa8eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:52.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.393+0000 7f0112d6e740 1 -- 172.21.15.33:0/3334124510 shutdown_connections 2024-03-19T03:00:52.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.393+0000 7f0112d6e740 1 -- 172.21.15.33:0/3334124510 wait complete. 2024-03-19T03:00:52.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.393+0000 7f0112d6e740 20 client.17286 trim_cache size 0 max 16384 2024-03-19T03:00:52.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.410+0000 7f0112d6e740 1 Processor -- start 2024-03-19T03:00:52.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.410+0000 7f0112d6e740 1 -- start start 2024-03-19T03:00:52.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.410+0000 7f0112d6e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602ceaaca30 0x5602ceab5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.410+0000 7f0112d6e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ceaad350 0x5602ceab1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.410+0000 7f0112d6e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5602ceab5f30 0x5602ceaae2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.410+0000 7f0112d6e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f01081b1530 con 0x5602ceaaca30 2024-03-19T03:00:52.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.410+0000 7f0112d6e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f01081aee00 con 0x5602ceaad350 2024-03-19T03:00:52.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.410+0000 7f0112d6e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f01081acc70 con 0x5602ceab5f30 2024-03-19T03:00:52.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.410+0000 7f010df1a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602ceaaca30 0x5602ceab5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.410+0000 7f010df1a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602ceaaca30 0x5602ceab5630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58978/0 (socket says 172.21.15.33:58978) 2024-03-19T03:00:52.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.410+0000 7f010df1a640 1 -- 172.21.15.33:0/901355190 learned_addr learned my addr 172.21.15.33:0/901355190 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:52.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.410+0000 7f010d719640 1 --2- 172.21.15.33:0/901355190 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ceaad350 0x5602ceab1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.410+0000 7f010e71b640 1 --2- 172.21.15.33:0/901355190 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5602ceab5f30 0x5602ceaae2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.410+0000 7f010df1a640 1 -- 172.21.15.33:0/901355190 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5602ceab5f30 msgr2=0x5602ceaae2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:52.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.410+0000 7f010df1a640 1 --2- 172.21.15.33:0/901355190 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5602ceab5f30 0x5602ceaae2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.410+0000 7f010df1a640 1 -- 172.21.15.33:0/901355190 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ceaad350 msgr2=0x5602ceab1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:52.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.410+0000 7f010df1a640 1 --2- 172.21.15.33:0/901355190 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ceaad350 0x5602ceab1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.410+0000 7f010df1a640 1 -- 172.21.15.33:0/901355190 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602ceaad180 con 0x5602ceaaca30 2024-03-19T03:00:52.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.410+0000 7f010e71b640 1 --2- 172.21.15.33:0/901355190 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5602ceab5f30 0x5602ceaae2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:00:52.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.411+0000 7f010df1a640 1 --2- 172.21.15.33:0/901355190 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602ceaaca30 0x5602ceab5630 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f0104009b40 tx=0x7f0104005d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa2944fe6e3fffce server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:52.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.411+0000 7f00f37fe640 1 -- 172.21.15.33:0/901355190 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f010401f040 con 0x5602ceaaca30 2024-03-19T03:00:52.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.411+0000 7f00f37fe640 1 -- 172.21.15.33:0/901355190 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f01040050b0 con 0x5602ceaaca30 2024-03-19T03:00:52.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.411+0000 7f00f37fe640 1 -- 172.21.15.33:0/901355190 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f010402a3d0 con 0x5602ceaaca30 2024-03-19T03:00:52.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.411+0000 7f0112d6e740 1 -- 172.21.15.33:0/901355190 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602ceaaca30 msgr2=0x5602ceab5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:52.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.411+0000 7f0112d6e740 1 --2- 172.21.15.33:0/901355190 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602ceaaca30 0x5602ceab5630 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f0104009b40 tx=0x7f0104005d70 comp rx=0 tx=0).stop 2024-03-19T03:00:52.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.411+0000 7f0112d6e740 1 -- 172.21.15.33:0/901355190 shutdown_connections 2024-03-19T03:00:52.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.411+0000 7f0112d6e740 1 --2- 172.21.15.33:0/901355190 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5602ceab5f30 0x5602ceaae2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.411+0000 7f0112d6e740 1 --2- 172.21.15.33:0/901355190 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ceaad350 0x5602ceab1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.411+0000 7f0112d6e740 1 --2- 172.21.15.33:0/901355190 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602ceaaca30 0x5602ceab5630 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.411+0000 7f0112d6e740 1 -- 172.21.15.33:0/901355190 >> 172.21.15.33:0/901355190 conn(0x5602ceaa7160 msgr2=0x5602ceaa61e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:52.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.411+0000 7f0112d6e740 1 -- 172.21.15.33:0/901355190 shutdown_connections 2024-03-19T03:00:52.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.412+0000 7f0112d6e740 1 -- 172.21.15.33:0/901355190 wait complete. 2024-03-19T03:00:52.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.412+0000 7f0112d6e740 1 -- start start 2024-03-19T03:00:52.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.412+0000 7f0112d6e740 1 Processor -- start 2024-03-19T03:00:52.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.412+0000 7f0112d6e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ceaaca30 0x5602ceab9a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.412+0000 7f0112d6e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602ceaad350 0x5602ceab9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.412+0000 7f0112d6e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5602ceab5f30 0x5602ceaba4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.412+0000 7f0112d6e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f01081ad2c0 con 0x5602ceaad350 2024-03-19T03:00:52.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.412+0000 7f0112d6e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f01081ae730 con 0x5602ceaaca30 2024-03-19T03:00:52.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.412+0000 7f0112d6e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f01081ad990 con 0x5602ceab5f30 2024-03-19T03:00:52.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.412+0000 7f0112d6e740 10 client.0 _ll_register_callbacks cb 0x7f00f8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:00:52.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.412+0000 7f010d719640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602ceaad350 0x5602ceab9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.412+0000 7f010d719640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602ceaad350 0x5602ceab9fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58980/0 (socket says 172.21.15.33:58980) 2024-03-19T03:00:52.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.412+0000 7f010d719640 1 -- 172.21.15.33:0/3141506883 learned_addr learned my addr 172.21.15.33:0/3141506883 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:00:52.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.412+0000 7f010e71b640 1 --2- 172.21.15.33:0/3141506883 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5602ceab5f30 0x5602ceaba4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.412+0000 7f010df1a640 1 --2- 172.21.15.33:0/3141506883 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ceaaca30 0x5602ceab9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.413+0000 7f010d719640 1 -- 172.21.15.33:0/3141506883 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5602ceab5f30 msgr2=0x5602ceaba4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:52.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.413+0000 7f010d719640 1 --2- 172.21.15.33:0/3141506883 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5602ceab5f30 0x5602ceaba4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.413+0000 7f010d719640 1 -- 172.21.15.33:0/3141506883 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ceaaca30 msgr2=0x5602ceab9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:00:52.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.413+0000 7f010d719640 1 --2- 172.21.15.33:0/3141506883 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ceaaca30 0x5602ceab9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.413+0000 7f010d719640 1 -- 172.21.15.33:0/3141506883 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602ce6e40f0 con 0x5602ceaad350 2024-03-19T03:00:52.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.413+0000 7f010df1a640 1 --2- 172.21.15.33:0/3141506883 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ceaaca30 0x5602ceab9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:00:52.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.413+0000 7f010d719640 1 --2- 172.21.15.33:0/3141506883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602ceaad350 0x5602ceab9fb0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f00f8004a40 tx=0x7f00f80973b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfc1fa40f1250fcc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:00:52.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.413+0000 7f00f17fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:00:52.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.413+0000 7f00f17fa640 1 -- 172.21.15.33:0/3141506883 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f8005680 con 0x5602ceaad350 2024-03-19T03:00:52.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.413+0000 7f00f17fa640 1 -- 172.21.15.33:0/3141506883 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9496+0+0 (secure 0 0 0) 0x7f00f80860b0 con 0x5602ceaad350 2024-03-19T03:00:52.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.413+0000 7f0112d6e740 1 -- 172.21.15.33:0/3141506883 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01040050b0 con 0x5602ceaad350 2024-03-19T03:00:52.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.413+0000 7f00f17fa640 1 -- 172.21.15.33:0/3141506883 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f00f809bf30 con 0x5602ceaad350 2024-03-19T03:00:52.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.413+0000 7f0112d6e740 10 client.17292 fetch_fsmap learned FSMap version 25 2024-03-19T03:00:52.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.413+0000 7f0112d6e740 1 -- 172.21.15.33:0/3141506883 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5602ceb2c7f0 con 0x5602ceaad350 2024-03-19T03:00:52.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.413+0000 7f00f17fa640 1 -- 172.21.15.33:0/3141506883 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f01040050b0 con 0x5602ceaad350 2024-03-19T03:00:52.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.413+0000 7f00f17fa640 1 -- 172.21.15.33:0/3141506883 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f00f8097ea0 con 0x5602ceaad350 2024-03-19T03:00:52.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.413+0000 7f0112d6e740 10 client.17292 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:00:52.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.413+0000 7f0112d6e740 10 client.17292 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:00:52.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.413+0000 7f0112d6e740 20 client.17292 populate_metadata read hostname 'smithi033' 2024-03-19T03:00:52.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.413+0000 7f0112d6e740 1 --2- 172.21.15.33:0/3141506883 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5602ceb2cfa0 0x5602ceb2f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:00:52.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.413+0000 7f0112d6e740 4 client.17292 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T03:00:52.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.413+0000 7f0112d6e740 1 -- 172.21.15.33:0/3141506883 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5602cea17d90 con 0x5602ceb2cfa0 2024-03-19T03:00:52.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.414+0000 7f010df1a640 1 --2- 172.21.15.33:0/3141506883 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5602ceb2cfa0 0x5602ceb2f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:00:52.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.414+0000 7f00f17fa640 10 client.17292 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:00:52.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.414+0000 7f010df1a640 1 --2- 172.21.15.33:0/3141506883 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5602ceb2cfa0 0x5602ceb2f3c0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59e1bfbb005f7fd9 server_cookie=1ab57dbe17cf319e in_seq=0 out_seq=0 2024-03-19T03:00:52.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.414+0000 7f00f17fa640 1 -- 172.21.15.33:0/3141506883 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5602cea17d90 con 0x5602ceb2cfa0 2024-03-19T03:00:52.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.414+0000 7f00f17fa640 10 client.17292 handle_command_reply: tid=0 2024-03-19T03:00:52.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.414+0000 7f0112d6e740 1 client.17292 shutdown 2024-03-19T03:00:52.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.414+0000 7f0112d6e740 1 -- 172.21.15.33:0/3141506883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602ceaad350 msgr2=0x5602ceab9fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:52.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.414+0000 7f0112d6e740 1 --2- 172.21.15.33:0/3141506883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602ceaad350 0x5602ceab9fb0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f00f8004a40 tx=0x7f00f80973b0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.414+0000 7f0112d6e740 1 -- 172.21.15.33:0/3141506883 shutdown_connections 2024-03-19T03:00:52.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.414+0000 7f0112d6e740 1 --2- 172.21.15.33:0/3141506883 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5602ceb2cfa0 0x5602ceb2f3c0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.414+0000 7f0112d6e740 1 --2- 172.21.15.33:0/3141506883 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5602ceab5f30 0x5602ceaba4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.414+0000 7f0112d6e740 1 --2- 172.21.15.33:0/3141506883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5602ceaad350 0x5602ceab9fb0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.414+0000 7f00f17fa640 0 client.17292 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:00:52.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.414+0000 7f0112d6e740 1 --2- 172.21.15.33:0/3141506883 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5602ceaaca30 0x5602ceab9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.414+0000 7f0112d6e740 1 -- 172.21.15.33:0/3141506883 >> 172.21.15.33:0/3141506883 conn(0x5602ceaa7160 msgr2=0x5602ceab2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:52.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.414+0000 7f0112d6e740 1 -- 172.21.15.33:0/3141506883 shutdown_connections 2024-03-19T03:00:52.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.414+0000 7f0112d6e740 1 -- 172.21.15.33:0/3141506883 wait complete. 2024-03-19T03:00:52.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.415+0000 7f0112d6e740 20 client.17292 trim_cache size 0 max 16384 2024-03-19T03:00:52.423 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T03:00:52.423 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T03:00:52.423 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-03-19T03:00:52.423 INFO:teuthology.orchestra.run.smithi033.stdout: "24b1aad6-69ce-4904-b8e4-9048940c3e6d": { 2024-03-19T03:00:52.423 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-03-19T03:00:52.423 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "24b1aad6-69ce-4904-b8e4-9048940c3e6d", 2024-03-19T03:00:52.423 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-03-19T03:00:52.423 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:00:52.423 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:00:52.423 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T03:00:52.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.415+0000 7f011017e640 1 -- 172.21.15.33:0/868650382 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f00f40a2970 msgr2=0x7f00f40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:52.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.415+0000 7f011017e640 1 --2- 172.21.15.33:0/868650382 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f00f40a2970 0x7f00f40a4e30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0100005cc0 tx=0x7f0100032450 comp rx=0 tx=0).stop 2024-03-19T03:00:52.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.415+0000 7f011017e640 1 -- 172.21.15.33:0/868650382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0108101050 msgr2=0x7f0108198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:00:52.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.415+0000 7f011017e640 1 --2- 172.21.15.33:0/868650382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0108101050 0x7f0108198570 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f00f800e9f0 tx=0x7f00f800edf0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.415+0000 7f011017e640 1 -- 172.21.15.33:0/868650382 shutdown_connections 2024-03-19T03:00:52.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.415+0000 7f011017e640 1 --2- 172.21.15.33:0/868650382 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f00f40a2970 0x7f00f40a4e30 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.415+0000 7f011017e640 1 --2- 172.21.15.33:0/868650382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01081066a0 0x7f010819c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.415+0000 7f011017e640 1 --2- 172.21.15.33:0/868650382 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0108101050 0x7f0108198570 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.415+0000 7f011017e640 1 --2- 172.21.15.33:0/868650382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0108100690 0x7f0108198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:00:52.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.415+0000 7f011017e640 1 -- 172.21.15.33:0/868650382 >> 172.21.15.33:0/868650382 conn(0x7f0108077820 msgr2=0x7f0108068d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:00:52.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.415+0000 7f011017e640 1 -- 172.21.15.33:0/868650382 shutdown_connections 2024-03-19T03:00:52.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:00:52.415+0000 7f011017e640 1 -- 172.21.15.33:0/868650382 wait complete. 2024-03-19T03:00:52.427 INFO:tasks.cephfs.filesystem:scrub status for tag:24b1aad6-69ce-4904-b8e4-9048940c3e6d - {'path': '/', 'tag': '24b1aad6-69ce-4904-b8e4-9048940c3e6d', 'options': 'recursive,force'} 2024-03-19T03:00:53.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:52 smithi096 ceph-mon[35946]: pgmap v227: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:00:53.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:52 smithi096 ceph-mon[35946]: scrub queued for path: / 2024-03-19T03:00:53.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:52 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T03:00:53.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:52 smithi096 ceph-mon[35946]: scrub summary: active paths [/] 2024-03-19T03:00:53.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:52 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T03:00:53.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:52 smithi096 ceph-mon[35946]: scrub summary: idle 2024-03-19T03:00:53.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:52 smithi155 ceph-mon[35804]: pgmap v227: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:00:53.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:52 smithi155 ceph-mon[35804]: scrub queued for path: / 2024-03-19T03:00:53.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:52 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T03:00:53.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:52 smithi155 ceph-mon[35804]: scrub summary: active paths [/] 2024-03-19T03:00:53.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:52 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T03:00:53.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:52 smithi155 ceph-mon[35804]: scrub summary: idle 2024-03-19T03:00:53.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:52 smithi033 ceph-mon[31973]: pgmap v227: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:00:53.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:52 smithi033 ceph-mon[31973]: scrub queued for path: / 2024-03-19T03:00:53.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:52 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T03:00:53.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:52 smithi033 ceph-mon[31973]: scrub summary: active paths [/] 2024-03-19T03:00:53.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:52 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T03:00:53.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:52 smithi033 ceph-mon[31973]: scrub summary: idle 2024-03-19T03:00:55.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:54 smithi096 ceph-mon[35946]: pgmap v228: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:00:55.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:54 smithi155 ceph-mon[35804]: pgmap v228: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:00:55.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:54 smithi033 ceph-mon[31973]: pgmap v228: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:00:57.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:56 smithi096 ceph-mon[35946]: pgmap v229: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:00:57.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:56 smithi155 ceph-mon[35804]: pgmap v229: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:00:57.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:56 smithi033 ceph-mon[31973]: pgmap v229: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:00:59.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:00:58 smithi096 ceph-mon[35946]: pgmap v230: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:00:59.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:00:58 smithi155 ceph-mon[35804]: pgmap v230: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:00:59.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:00:58 smithi033 ceph-mon[31973]: pgmap v230: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:01:01.110 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:00 smithi096 ceph-mon[35946]: pgmap v231: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:01:01.117 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:00 smithi033 ceph-mon[31973]: pgmap v231: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:01:01.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:00 smithi155 ceph-mon[35804]: pgmap v231: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:01:03.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:02 smithi096 ceph-mon[35946]: pgmap v232: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:01:03.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:02 smithi155 ceph-mon[35804]: pgmap v232: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:01:03.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:02 smithi033 ceph-mon[31973]: pgmap v232: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:01:05.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:04 smithi096 ceph-mon[35946]: pgmap v233: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-19T03:01:05.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:04 smithi155 ceph-mon[35804]: pgmap v233: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-19T03:01:05.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:04 smithi033 ceph-mon[31973]: pgmap v233: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-19T03:01:07.217 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:06 smithi096 ceph-mon[35946]: pgmap v234: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-19T03:01:07.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:06 smithi155 ceph-mon[35804]: pgmap v234: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-19T03:01:07.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:06 smithi033 ceph-mon[31973]: pgmap v234: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-19T03:01:09.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:08 smithi096 ceph-mon[35946]: pgmap v235: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-19T03:01:09.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:08 smithi096 ceph-mon[35946]: osdmap e101: 12 total, 12 up, 12 in 2024-03-19T03:01:09.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:08 smithi155 ceph-mon[35804]: pgmap v235: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-19T03:01:09.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:08 smithi155 ceph-mon[35804]: osdmap e101: 12 total, 12 up, 12 in 2024-03-19T03:01:09.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:08 smithi033 ceph-mon[31973]: pgmap v235: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-19T03:01:09.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:08 smithi033 ceph-mon[31973]: osdmap e101: 12 total, 12 up, 12 in 2024-03-19T03:01:11.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:10 smithi096 ceph-mon[35946]: pgmap v237: 129 pgs: 12 active+clean+snaptrim, 91 active+clean, 26 active+clean+snaptrim_wait; 10 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-19T03:01:11.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:10 smithi155 ceph-mon[35804]: pgmap v237: 129 pgs: 12 active+clean+snaptrim, 91 active+clean, 26 active+clean+snaptrim_wait; 10 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-19T03:01:11.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:10 smithi033 ceph-mon[31973]: pgmap v237: 129 pgs: 12 active+clean+snaptrim, 91 active+clean, 26 active+clean+snaptrim_wait; 10 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-19T03:01:13.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:12 smithi096 ceph-mon[35946]: pgmap v238: 129 pgs: 12 active+clean+snaptrim, 91 active+clean, 26 active+clean+snaptrim_wait; 10 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-19T03:01:13.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:12 smithi155 ceph-mon[35804]: pgmap v238: 129 pgs: 12 active+clean+snaptrim, 91 active+clean, 26 active+clean+snaptrim_wait; 10 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-19T03:01:13.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:12 smithi033 ceph-mon[31973]: pgmap v238: 129 pgs: 12 active+clean+snaptrim, 91 active+clean, 26 active+clean+snaptrim_wait; 10 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-19T03:01:15.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:14 smithi096 ceph-mon[35946]: pgmap v239: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-19T03:01:15.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:14 smithi155 ceph-mon[35804]: pgmap v239: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-19T03:01:15.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:14 smithi033 ceph-mon[31973]: pgmap v239: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-19T03:01:16.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:16 smithi096 ceph-mon[35946]: pgmap v240: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-19T03:01:16.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:16 smithi096 ceph-mon[35946]: osdmap e102: 12 total, 12 up, 12 in 2024-03-19T03:01:16.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:16 smithi155 ceph-mon[35804]: pgmap v240: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-19T03:01:16.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:16 smithi155 ceph-mon[35804]: osdmap e102: 12 total, 12 up, 12 in 2024-03-19T03:01:16.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:16 smithi033 ceph-mon[31973]: pgmap v240: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-19T03:01:16.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:16 smithi033 ceph-mon[31973]: osdmap e102: 12 total, 12 up, 12 in 2024-03-19T03:01:18.466 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:18 smithi096 ceph-mon[35946]: pgmap v242: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 222 B/s wr, 6 op/s 2024-03-19T03:01:18.495 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:18 smithi033 ceph-mon[31973]: pgmap v242: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 222 B/s wr, 6 op/s 2024-03-19T03:01:18.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:18 smithi155 ceph-mon[35804]: pgmap v242: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 222 B/s wr, 6 op/s 2024-03-19T03:01:20.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:20 smithi155 ceph-mon[35804]: pgmap v243: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-19T03:01:20.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:20 smithi033 ceph-mon[31973]: pgmap v243: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-19T03:01:20.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:20 smithi096 ceph-mon[35946]: pgmap v243: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-19T03:01:22.428 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T03:01:22.429 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T03:01:22.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:22 smithi155 ceph-mon[35804]: pgmap v244: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-19T03:01:22.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:22 smithi033 ceph-mon[31973]: pgmap v244: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-19T03:01:22.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.548+0000 7fe603110640 1 -- 172.21.15.33:0/3605211546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe5fc100690 msgr2=0x7fe5fc100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:22.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.548+0000 7fe603110640 1 --2- 172.21.15.33:0/3605211546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe5fc100690 0x7fe5fc100b10 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8009930 tx=0x7fe5e802bc30 comp rx=0 tx=0).stop 2024-03-19T03:01:22.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.548+0000 7fe603110640 1 -- 172.21.15.33:0/3605211546 shutdown_connections 2024-03-19T03:01:22.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.548+0000 7fe603110640 1 --2- 172.21.15.33:0/3605211546 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5fc101050 0x7fe5fc10d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.548+0000 7fe603110640 1 --2- 172.21.15.33:0/3605211546 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe5fc100690 0x7fe5fc100b10 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.548+0000 7fe603110640 1 --2- 172.21.15.33:0/3605211546 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5fc1066a0 0x7fe5fc106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.548+0000 7fe603110640 1 -- 172.21.15.33:0/3605211546 >> 172.21.15.33:0/3605211546 conn(0x7fe5fc077820 msgr2=0x7fe5fc077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:22.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.548+0000 7fe603110640 1 -- 172.21.15.33:0/3605211546 shutdown_connections 2024-03-19T03:01:22.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.548+0000 7fe603110640 1 -- 172.21.15.33:0/3605211546 wait complete. 2024-03-19T03:01:22.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.548+0000 7fe603110640 1 Processor -- start 2024-03-19T03:01:22.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.548+0000 7fe603110640 1 -- start start 2024-03-19T03:01:22.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.549+0000 7fe603110640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe5fc100690 0x7fe5fc19a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:22.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.549+0000 7fe603110640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5fc101050 0x7fe5fc19a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:22.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.549+0000 7fe603110640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5fc1066a0 0x7fe5fc19eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:22.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.549+0000 7fe603110640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe5fc110960 con 0x7fe5fc100690 2024-03-19T03:01:22.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.549+0000 7fe603110640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe5fc1107e0 con 0x7fe5fc1066a0 2024-03-19T03:01:22.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.549+0000 7fe603110640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe5fc110ae0 con 0x7fe5fc101050 2024-03-19T03:01:22.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.549+0000 7fe600eac640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe5fc100690 0x7fe5fc19a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:22.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.549+0000 7fe600eac640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe5fc100690 0x7fe5fc19a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58710/0 (socket says 172.21.15.33:58710) 2024-03-19T03:01:22.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.549+0000 7fe600eac640 1 -- 172.21.15.33:0/2712168604 learned_addr learned my addr 172.21.15.33:0/2712168604 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:22.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.549+0000 7fe5fbfff640 1 --2- 172.21.15.33:0/2712168604 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5fc101050 0x7fe5fc19a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:22.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.549+0000 7fe6016ad640 1 --2- 172.21.15.33:0/2712168604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5fc1066a0 0x7fe5fc19eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:22.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.549+0000 7fe600eac640 1 -- 172.21.15.33:0/2712168604 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5fc101050 msgr2=0x7fe5fc19a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:22.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.549+0000 7fe600eac640 1 --2- 172.21.15.33:0/2712168604 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5fc101050 0x7fe5fc19a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.549+0000 7fe600eac640 1 -- 172.21.15.33:0/2712168604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5fc1066a0 msgr2=0x7fe5fc19eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:22.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.549+0000 7fe600eac640 1 --2- 172.21.15.33:0/2712168604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5fc1066a0 0x7fe5fc19eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.549+0000 7fe600eac640 1 -- 172.21.15.33:0/2712168604 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5fc19f290 con 0x7fe5fc100690 2024-03-19T03:01:22.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.549+0000 7fe600eac640 1 --2- 172.21.15.33:0/2712168604 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe5fc100690 0x7fe5fc19a2f0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fe5f000e9f0 tx=0x7fe5f000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:22.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.550+0000 7fe5f9ffb640 1 -- 172.21.15.33:0/2712168604 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f000ccc0 con 0x7fe5fc100690 2024-03-19T03:01:22.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.550+0000 7fe5f9ffb640 1 -- 172.21.15.33:0/2712168604 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5f000ce60 con 0x7fe5fc100690 2024-03-19T03:01:22.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.550+0000 7fe603110640 1 -- 172.21.15.33:0/2712168604 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5fc19f580 con 0x7fe5fc100690 2024-03-19T03:01:22.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.550+0000 7fe5f9ffb640 1 -- 172.21.15.33:0/2712168604 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f0010430 con 0x7fe5fc100690 2024-03-19T03:01:22.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.550+0000 7fe603110640 1 -- 172.21.15.33:0/2712168604 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5fc1a6e10 con 0x7fe5fc100690 2024-03-19T03:01:22.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.552+0000 7fe5f9ffb640 1 -- 172.21.15.33:0/2712168604 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fe5f0005020 con 0x7fe5fc100690 2024-03-19T03:01:22.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.552+0000 7fe5f9ffb640 1 --2- 172.21.15.33:0/2712168604 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe5ec0a2970 0x7fe5ec0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:22.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.553+0000 7fe605ddb740 1 Processor -- start 2024-03-19T03:01:22.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.553+0000 7fe605ddb740 1 -- start start 2024-03-19T03:01:22.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.553+0000 7fe5fbfff640 1 --2- 172.21.15.33:0/2712168604 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe5ec0a2970 0x7fe5ec0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:22.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.553+0000 7fe5f9ffb640 1 -- 172.21.15.33:0/2712168604 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7fe5f00cf4c0 con 0x7fe5fc100690 2024-03-19T03:01:22.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.553+0000 7fe605ddb740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eb20dd6a30 0x55eb20dd6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:22.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.553+0000 7fe605ddb740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb20dd7350 0x55eb20ddf9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:22.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.553+0000 7fe605ddb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eb20ddff30 0x55eb20de2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:22.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.553+0000 7fe605ddb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55eb20d3a500 con 0x55eb20ddff30 2024-03-19T03:01:22.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.553+0000 7fe605ddb740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55eb20ce2be0 con 0x55eb20dd6a30 2024-03-19T03:01:22.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.553+0000 7fe605ddb740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55eb20d365a0 con 0x55eb20dd7350 2024-03-19T03:01:22.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.553+0000 7fe600eac640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eb20dd6a30 0x55eb20dd6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:22.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.553+0000 7fe6016ad640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eb20ddff30 0x55eb20de2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:22.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.553+0000 7fe600eac640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eb20dd6a30 0x55eb20dd6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:54412/0 (socket says 172.21.15.33:54412) 2024-03-19T03:01:22.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.553+0000 7fe600eac640 1 -- 172.21.15.33:0/1179113860 learned_addr learned my addr 172.21.15.33:0/1179113860 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:22.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.553+0000 7fe6016ad640 1 --2- 172.21.15.33:0/1179113860 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eb20ddff30 0x55eb20de2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58720/0 (socket says 172.21.15.33:58720) 2024-03-19T03:01:22.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.553+0000 7fe5fbfff640 1 --2- 172.21.15.33:0/1179113860 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb20dd7350 0x55eb20ddf9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:22.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.553+0000 7fe5fbfff640 1 --2- 172.21.15.33:0/2712168604 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe5ec0a2970 0x7fe5ec0a4e30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8005ab0 tx=0x7fe5e8005fd0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:22.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.553+0000 7fe600eac640 1 -- 172.21.15.33:0/1179113860 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb20dd7350 msgr2=0x55eb20ddf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:22.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.553+0000 7fe600eac640 1 --2- 172.21.15.33:0/1179113860 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb20dd7350 0x55eb20ddf9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.553+0000 7fe600eac640 1 -- 172.21.15.33:0/1179113860 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eb20ddff30 msgr2=0x55eb20de2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:22.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.553+0000 7fe600eac640 1 --2- 172.21.15.33:0/1179113860 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eb20ddff30 0x55eb20de2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.553+0000 7fe600eac640 1 -- 172.21.15.33:0/1179113860 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb20d04000 con 0x55eb20dd6a30 2024-03-19T03:01:22.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.553+0000 7fe5fbfff640 1 --2- 172.21.15.33:0/1179113860 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb20dd7350 0x55eb20ddf9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:01:22.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.553+0000 7fe6016ad640 1 --2- 172.21.15.33:0/1179113860 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eb20ddff30 0x55eb20de2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:22.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.554+0000 7fe600eac640 1 --2- 172.21.15.33:0/1179113860 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eb20dd6a30 0x55eb20dd6e10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe5f009b780 tx=0x7fe5f009b210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cefe13f1d9afa31 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:22.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.554+0000 7fe5e6ffd640 1 -- 172.21.15.33:0/1179113860 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f00b4070 con 0x55eb20dd6a30 2024-03-19T03:01:22.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.554+0000 7fe5e6ffd640 1 -- 172.21.15.33:0/1179113860 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5f00a3f90 con 0x55eb20dd6a30 2024-03-19T03:01:22.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.554+0000 7fe5e6ffd640 1 -- 172.21.15.33:0/1179113860 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f0097e60 con 0x55eb20dd6a30 2024-03-19T03:01:22.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.554+0000 7fe605ddb740 1 -- 172.21.15.33:0/1179113860 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eb20dd6a30 msgr2=0x55eb20dd6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:22.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.554+0000 7fe605ddb740 1 --2- 172.21.15.33:0/1179113860 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eb20dd6a30 0x55eb20dd6e10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe5f009b780 tx=0x7fe5f009b210 comp rx=0 tx=0).stop 2024-03-19T03:01:22.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.554+0000 7fe605ddb740 1 -- 172.21.15.33:0/1179113860 shutdown_connections 2024-03-19T03:01:22.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.554+0000 7fe605ddb740 1 --2- 172.21.15.33:0/1179113860 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb20dd7350 0x55eb20ddf9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.554+0000 7fe605ddb740 1 --2- 172.21.15.33:0/1179113860 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eb20ddff30 0x55eb20de2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.554+0000 7fe605ddb740 1 --2- 172.21.15.33:0/1179113860 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eb20dd6a30 0x55eb20dd6e10 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.554+0000 7fe605ddb740 1 -- 172.21.15.33:0/1179113860 >> 172.21.15.33:0/1179113860 conn(0x55eb20dd1160 msgr2=0x55eb20dd1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:22.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.555+0000 7fe605ddb740 1 -- 172.21.15.33:0/1179113860 shutdown_connections 2024-03-19T03:01:22.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.555+0000 7fe605ddb740 1 -- 172.21.15.33:0/1179113860 wait complete. 2024-03-19T03:01:22.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.555+0000 7fe605ddb740 1 -- start start 2024-03-19T03:01:22.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.555+0000 7fe605ddb740 1 Processor -- start 2024-03-19T03:01:22.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.555+0000 7fe605ddb740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eb20dd6a30 0x55eb20ddced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:22.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.555+0000 7fe605ddb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eb20dd7350 0x55eb20ddd410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:22.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.555+0000 7fe605ddb740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb20ddff30 0x55eb20ddd950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:22.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.555+0000 7fe605ddb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55eb20de2d20 con 0x55eb20dd7350 2024-03-19T03:01:22.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.555+0000 7fe605ddb740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55eb20cf6160 con 0x55eb20dd6a30 2024-03-19T03:01:22.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.555+0000 7fe605ddb740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55eb20de2ea0 con 0x55eb20ddff30 2024-03-19T03:01:22.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.555+0000 7fe605ddb740 10 client.0 _ll_register_callbacks cb 0x55eb20d32a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:01:22.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.555+0000 7fe5fbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eb20dd7350 0x55eb20ddd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:22.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.555+0000 7fe600eac640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eb20dd6a30 0x55eb20ddced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:22.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.555+0000 7fe600eac640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eb20dd6a30 0x55eb20ddced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:54428/0 (socket says 172.21.15.33:54428) 2024-03-19T03:01:22.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.555+0000 7fe600eac640 1 -- 172.21.15.33:0/1146490163 learned_addr learned my addr 172.21.15.33:0/1146490163 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:22.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.556+0000 7fe6016ad640 1 --2- 172.21.15.33:0/1146490163 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb20ddff30 0x55eb20ddd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:22.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.556+0000 7fe600eac640 1 -- 172.21.15.33:0/1146490163 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb20ddff30 msgr2=0x55eb20ddd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:22.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.556+0000 7fe600eac640 1 --2- 172.21.15.33:0/1146490163 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb20ddff30 0x55eb20ddd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.556+0000 7fe600eac640 1 -- 172.21.15.33:0/1146490163 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eb20dd7350 msgr2=0x55eb20ddd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:22.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.556+0000 7fe600eac640 1 --2- 172.21.15.33:0/1146490163 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eb20dd7350 0x55eb20ddd410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.556+0000 7fe600eac640 1 -- 172.21.15.33:0/1146490163 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb20b187d0 con 0x55eb20dd6a30 2024-03-19T03:01:22.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.556+0000 7fe600eac640 1 --2- 172.21.15.33:0/1146490163 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eb20dd6a30 0x55eb20ddced0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe5f00a3a60 tx=0x7fe5f00048f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f0e7ff705ae1c8f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:22.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.556+0000 7fe5dbfff640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T03:01:22.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.556+0000 7fe5dbfff640 1 -- 172.21.15.33:0/1146490163 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f00b4040 con 0x55eb20dd6a30 2024-03-19T03:01:22.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.556+0000 7fe605ddb740 1 -- 172.21.15.33:0/1146490163 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5f00a3f90 con 0x55eb20dd6a30 2024-03-19T03:01:22.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.556+0000 7fe5dbfff640 1 -- 172.21.15.33:0/1146490163 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7fe5f00cf170 con 0x55eb20dd6a30 2024-03-19T03:01:22.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.556+0000 7fe5dbfff640 1 -- 172.21.15.33:0/1146490163 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe5f0086080 con 0x55eb20dd6a30 2024-03-19T03:01:22.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.556+0000 7fe605ddb740 10 client.27125 fetch_fsmap learned FSMap version 25 2024-03-19T03:01:22.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.556+0000 7fe605ddb740 1 -- 172.21.15.33:0/1146490163 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55eb20a4e3f0 con 0x55eb20dd6a30 2024-03-19T03:01:22.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.556+0000 7fe5dbfff640 1 -- 172.21.15.33:0/1146490163 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5f00a3f90 con 0x55eb20dd6a30 2024-03-19T03:01:22.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.557+0000 7fe5dbfff640 1 -- 172.21.15.33:0/1146490163 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fe5f00bb070 con 0x55eb20dd6a30 2024-03-19T03:01:22.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.557+0000 7fe605ddb740 10 client.27125 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:01:22.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.557+0000 7fe605ddb740 10 client.27125 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:01:22.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.557+0000 7fe605ddb740 20 client.27125 populate_metadata read hostname 'smithi033' 2024-03-19T03:01:22.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.557+0000 7fe605ddb740 1 --2- 172.21.15.33:0/1146490163 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55eb20df00a0 0x55eb20df2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:22.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.557+0000 7fe605ddb740 4 client.27125 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:01:22.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.557+0000 7fe605ddb740 1 -- 172.21.15.33:0/1146490163 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb20d47a40 con 0x55eb20df00a0 2024-03-19T03:01:22.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.557+0000 7fe5fbfff640 1 --2- 172.21.15.33:0/1146490163 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55eb20df00a0 0x55eb20df2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:22.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.557+0000 7fe5fbfff640 1 --2- 172.21.15.33:0/1146490163 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55eb20df00a0 0x55eb20df2490 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8211c10baeed00c9 server_cookie=6c0b5cbe0bbfbffe in_seq=0 out_seq=0 2024-03-19T03:01:22.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.557+0000 7fe5dbfff640 10 client.27125 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:01:22.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.558+0000 7fe5dbfff640 1 -- 172.21.15.33:0/1146490163 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55eb20d47a40 con 0x55eb20df00a0 2024-03-19T03:01:22.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.558+0000 7fe5dbfff640 10 client.27125 handle_command_reply: tid=0 2024-03-19T03:01:22.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.558+0000 7fe605ddb740 1 client.27125 shutdown 2024-03-19T03:01:22.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.558+0000 7fe605ddb740 1 -- 172.21.15.33:0/1146490163 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eb20dd6a30 msgr2=0x55eb20ddced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:22.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.558+0000 7fe605ddb740 1 --2- 172.21.15.33:0/1146490163 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eb20dd6a30 0x55eb20ddced0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe5f00a3a60 tx=0x7fe5f00048f0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.558+0000 7fe605ddb740 1 -- 172.21.15.33:0/1146490163 shutdown_connections 2024-03-19T03:01:22.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.558+0000 7fe605ddb740 1 --2- 172.21.15.33:0/1146490163 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55eb20df00a0 0x55eb20df2490 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.558+0000 7fe605ddb740 1 --2- 172.21.15.33:0/1146490163 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb20ddff30 0x55eb20ddd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.558+0000 7fe5dbfff640 0 client.27125 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:01:22.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.558+0000 7fe605ddb740 1 --2- 172.21.15.33:0/1146490163 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eb20dd7350 0x55eb20ddd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.558+0000 7fe605ddb740 1 --2- 172.21.15.33:0/1146490163 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eb20dd6a30 0x55eb20ddced0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.558+0000 7fe605ddb740 1 -- 172.21.15.33:0/1146490163 >> 172.21.15.33:0/1146490163 conn(0x55eb20dd1160 msgr2=0x55eb20dd2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:22.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.558+0000 7fe605ddb740 1 -- 172.21.15.33:0/1146490163 shutdown_connections 2024-03-19T03:01:22.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.558+0000 7fe605ddb740 1 -- 172.21.15.33:0/1146490163 wait complete. 2024-03-19T03:01:22.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.558+0000 7fe605ddb740 20 client.27125 trim_cache size 0 max 16384 2024-03-19T03:01:22.582 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.576+0000 7fe605ddb740 1 Processor -- start 2024-03-19T03:01:22.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.576+0000 7fe605ddb740 1 -- start start 2024-03-19T03:01:22.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.576+0000 7fe605ddb740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eb20dd6a30 0x55eb20ddf600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:22.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.576+0000 7fe605ddb740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb20dd7350 0x55eb20ddbe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:22.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.576+0000 7fe605ddb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eb20ddff30 0x55eb20dd8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:22.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.576+0000 7fe605ddb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe5fc1b37f0 con 0x55eb20ddff30 2024-03-19T03:01:22.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.576+0000 7fe605ddb740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe5fc1b10c0 con 0x55eb20dd6a30 2024-03-19T03:01:22.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.576+0000 7fe605ddb740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe5fc1aef30 con 0x55eb20dd7350 2024-03-19T03:01:22.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.576+0000 7fe6016ad640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eb20ddff30 0x55eb20dd8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:22.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.576+0000 7fe6016ad640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eb20ddff30 0x55eb20dd8210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58732/0 (socket says 172.21.15.33:58732) 2024-03-19T03:01:22.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.576+0000 7fe6016ad640 1 -- 172.21.15.33:0/2543011435 learned_addr learned my addr 172.21.15.33:0/2543011435 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:22.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.576+0000 7fe600eac640 1 --2- 172.21.15.33:0/2543011435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eb20dd6a30 0x55eb20ddf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:22.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.576+0000 7fe5fbfff640 1 --2- 172.21.15.33:0/2543011435 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb20dd7350 0x55eb20ddbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:22.583 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.576+0000 7fe6016ad640 1 -- 172.21.15.33:0/2543011435 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb20dd7350 msgr2=0x55eb20ddbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:22.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.576+0000 7fe6016ad640 1 --2- 172.21.15.33:0/2543011435 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb20dd7350 0x55eb20ddbe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.576+0000 7fe6016ad640 1 -- 172.21.15.33:0/2543011435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eb20dd6a30 msgr2=0x55eb20ddf600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:22.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.576+0000 7fe6016ad640 1 --2- 172.21.15.33:0/2543011435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eb20dd6a30 0x55eb20ddf600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.576+0000 7fe6016ad640 1 -- 172.21.15.33:0/2543011435 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb20a0e0f0 con 0x55eb20ddff30 2024-03-19T03:01:22.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.576+0000 7fe6016ad640 1 --2- 172.21.15.33:0/2543011435 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eb20ddff30 0x55eb20dd8210 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4012580 tx=0x7fe5f4009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6e24d7a2b12ddb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:22.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.577+0000 7fe5e67fc640 1 -- 172.21.15.33:0/2543011435 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f4022070 con 0x55eb20ddff30 2024-03-19T03:01:22.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.577+0000 7fe5e67fc640 1 -- 172.21.15.33:0/2543011435 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5f4003e20 con 0x55eb20ddff30 2024-03-19T03:01:22.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.577+0000 7fe5e67fc640 1 -- 172.21.15.33:0/2543011435 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f400f7a0 con 0x55eb20ddff30 2024-03-19T03:01:22.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.577+0000 7fe605ddb740 1 -- 172.21.15.33:0/2543011435 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eb20ddff30 msgr2=0x55eb20dd8210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:22.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.577+0000 7fe605ddb740 1 --2- 172.21.15.33:0/2543011435 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eb20ddff30 0x55eb20dd8210 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4012580 tx=0x7fe5f4009500 comp rx=0 tx=0).stop 2024-03-19T03:01:22.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.577+0000 7fe605ddb740 1 -- 172.21.15.33:0/2543011435 shutdown_connections 2024-03-19T03:01:22.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.577+0000 7fe605ddb740 1 --2- 172.21.15.33:0/2543011435 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb20dd7350 0x55eb20ddbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.577+0000 7fe605ddb740 1 --2- 172.21.15.33:0/2543011435 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eb20ddff30 0x55eb20dd8210 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.577+0000 7fe605ddb740 1 --2- 172.21.15.33:0/2543011435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eb20dd6a30 0x55eb20ddf600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.584 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.577+0000 7fe605ddb740 1 -- 172.21.15.33:0/2543011435 >> 172.21.15.33:0/2543011435 conn(0x55eb20dd1160 msgr2=0x55eb20deb980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:22.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.577+0000 7fe605ddb740 1 -- 172.21.15.33:0/2543011435 shutdown_connections 2024-03-19T03:01:22.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.577+0000 7fe605ddb740 1 -- 172.21.15.33:0/2543011435 wait complete. 2024-03-19T03:01:22.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.577+0000 7fe605ddb740 1 -- start start 2024-03-19T03:01:22.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.578+0000 7fe605ddb740 1 Processor -- start 2024-03-19T03:01:22.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.578+0000 7fe605ddb740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb20dd6a30 0x55eb20df2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:22.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.578+0000 7fe605ddb740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eb20dd7350 0x55eb20df34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:22.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.578+0000 7fe605ddb740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eb20ddff30 0x55eb20df39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:22.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.578+0000 7fe605ddb740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe5fc1af580 con 0x55eb20ddff30 2024-03-19T03:01:22.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.578+0000 7fe605ddb740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe5fc1b09f0 con 0x55eb20dd7350 2024-03-19T03:01:22.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.578+0000 7fe605ddb740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe5fc1afc50 con 0x55eb20dd6a30 2024-03-19T03:01:22.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.578+0000 7fe605ddb740 10 client.0 _ll_register_callbacks cb 0x7fe5f00b2070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:01:22.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.578+0000 7fe6016ad640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eb20ddff30 0x55eb20df39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:22.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.578+0000 7fe6016ad640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eb20ddff30 0x55eb20df39f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58744/0 (socket says 172.21.15.33:58744) 2024-03-19T03:01:22.585 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.578+0000 7fe6016ad640 1 -- 172.21.15.33:0/809966963 learned_addr learned my addr 172.21.15.33:0/809966963 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:22.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.578+0000 7fe5fbfff640 1 --2- 172.21.15.33:0/809966963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eb20dd7350 0x55eb20df34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:22.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.578+0000 7fe600eac640 1 --2- 172.21.15.33:0/809966963 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb20dd6a30 0x55eb20df2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:22.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.578+0000 7fe6016ad640 1 -- 172.21.15.33:0/809966963 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb20dd6a30 msgr2=0x55eb20df2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:22.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.578+0000 7fe6016ad640 1 --2- 172.21.15.33:0/809966963 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb20dd6a30 0x55eb20df2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.578+0000 7fe6016ad640 1 -- 172.21.15.33:0/809966963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eb20dd7350 msgr2=0x55eb20df34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:22.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.578+0000 7fe6016ad640 1 --2- 172.21.15.33:0/809966963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eb20dd7350 0x55eb20df34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.578+0000 7fe6016ad640 1 -- 172.21.15.33:0/809966963 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb20e55210 con 0x55eb20ddff30 2024-03-19T03:01:22.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.578+0000 7fe5fbfff640 1 --2- 172.21.15.33:0/809966963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eb20dd7350 0x55eb20df34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:22.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.579+0000 7fe6016ad640 1 --2- 172.21.15.33:0/809966963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eb20ddff30 0x55eb20df39f0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4005c30 tx=0x7fe5f4004540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=769221370d3e80c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:22.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.579+0000 7fe5dbfff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:01:22.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.579+0000 7fe5dbfff640 1 -- 172.21.15.33:0/809966963 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f402d020 con 0x55eb20ddff30 2024-03-19T03:01:22.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.579+0000 7fe5dbfff640 1 -- 172.21.15.33:0/809966963 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7fe5f4004050 con 0x55eb20ddff30 2024-03-19T03:01:22.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.579+0000 7fe605ddb740 1 -- 172.21.15.33:0/809966963 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5f4003e20 con 0x55eb20ddff30 2024-03-19T03:01:22.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.579+0000 7fe5dbfff640 1 -- 172.21.15.33:0/809966963 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe5f4038050 con 0x55eb20ddff30 2024-03-19T03:01:22.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.579+0000 7fe605ddb740 10 client.17310 fetch_fsmap learned FSMap version 25 2024-03-19T03:01:22.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.579+0000 7fe605ddb740 1 -- 172.21.15.33:0/809966963 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55eb20e564e0 con 0x55eb20ddff30 2024-03-19T03:01:22.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.579+0000 7fe5dbfff640 1 -- 172.21.15.33:0/809966963 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5f4003e20 con 0x55eb20ddff30 2024-03-19T03:01:22.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.579+0000 7fe5dbfff640 1 -- 172.21.15.33:0/809966963 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fe5f4029480 con 0x55eb20ddff30 2024-03-19T03:01:22.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.579+0000 7fe605ddb740 10 client.17310 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:01:22.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.579+0000 7fe605ddb740 10 client.17310 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:01:22.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.579+0000 7fe605ddb740 20 client.17310 populate_metadata read hostname 'smithi033' 2024-03-19T03:01:22.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.579+0000 7fe605ddb740 1 --2- 172.21.15.33:0/809966963 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55eb20e56bd0 0x55eb20e58fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:22.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.579+0000 7fe605ddb740 4 client.17310 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T03:01:22.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.579+0000 7fe605ddb740 1 -- 172.21.15.33:0/809966963 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55eb20d41d90 con 0x55eb20e56bd0 2024-03-19T03:01:22.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.580+0000 7fe600eac640 1 --2- 172.21.15.33:0/809966963 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55eb20e56bd0 0x55eb20e58fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:22.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.580+0000 7fe600eac640 1 --2- 172.21.15.33:0/809966963 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55eb20e56bd0 0x55eb20e58fc0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69d9ca0a8b597b5b server_cookie=c73204dc2cf55868 in_seq=0 out_seq=0 2024-03-19T03:01:22.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.580+0000 7fe5dbfff640 10 client.17310 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:01:22.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.580+0000 7fe5dbfff640 1 -- 172.21.15.33:0/809966963 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55eb20d41d90 con 0x55eb20e56bd0 2024-03-19T03:01:22.587 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.580+0000 7fe5dbfff640 10 client.17310 handle_command_reply: tid=0 2024-03-19T03:01:22.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.580+0000 7fe605ddb740 1 client.17310 shutdown 2024-03-19T03:01:22.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.580+0000 7fe605ddb740 1 -- 172.21.15.33:0/809966963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eb20ddff30 msgr2=0x55eb20df39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:22.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.580+0000 7fe605ddb740 1 --2- 172.21.15.33:0/809966963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eb20ddff30 0x55eb20df39f0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4005c30 tx=0x7fe5f4004540 comp rx=0 tx=0).stop 2024-03-19T03:01:22.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.580+0000 7fe605ddb740 1 -- 172.21.15.33:0/809966963 shutdown_connections 2024-03-19T03:01:22.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.580+0000 7fe605ddb740 1 --2- 172.21.15.33:0/809966963 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55eb20e56bd0 0x55eb20e58fc0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.580+0000 7fe605ddb740 1 --2- 172.21.15.33:0/809966963 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55eb20ddff30 0x55eb20df39f0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.580+0000 7fe605ddb740 1 --2- 172.21.15.33:0/809966963 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eb20dd7350 0x55eb20df34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.580+0000 7fe605ddb740 1 --2- 172.21.15.33:0/809966963 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55eb20dd6a30 0x55eb20df2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.580+0000 7fe605ddb740 1 -- 172.21.15.33:0/809966963 >> 172.21.15.33:0/809966963 conn(0x55eb20dd1160 msgr2=0x55eb20ddce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:22.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.580+0000 7fe5dbfff640 0 client.17310 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:01:22.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.580+0000 7fe605ddb740 1 -- 172.21.15.33:0/809966963 shutdown_connections 2024-03-19T03:01:22.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.580+0000 7fe605ddb740 1 -- 172.21.15.33:0/809966963 wait complete. 2024-03-19T03:01:22.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.580+0000 7fe605ddb740 20 client.17310 trim_cache size 0 max 16384 2024-03-19T03:01:22.589 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T03:01:22.589 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-03-19T03:01:22.589 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-03-19T03:01:22.589 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T03:01:22.589 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.581+0000 7fe603110640 1 -- 172.21.15.33:0/2712168604 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe5ec0a2970 msgr2=0x7fe5ec0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:22.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.581+0000 7fe603110640 1 --2- 172.21.15.33:0/2712168604 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe5ec0a2970 0x7fe5ec0a4e30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8005ab0 tx=0x7fe5e8005fd0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.581+0000 7fe603110640 1 -- 172.21.15.33:0/2712168604 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe5fc100690 msgr2=0x7fe5fc19a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:22.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.581+0000 7fe603110640 1 --2- 172.21.15.33:0/2712168604 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe5fc100690 0x7fe5fc19a2f0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fe5f000e9f0 tx=0x7fe5f000edf0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.581+0000 7fe603110640 1 -- 172.21.15.33:0/2712168604 shutdown_connections 2024-03-19T03:01:22.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.581+0000 7fe603110640 1 --2- 172.21.15.33:0/2712168604 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe5ec0a2970 0x7fe5ec0a4e30 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.581+0000 7fe603110640 1 --2- 172.21.15.33:0/2712168604 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5fc101050 0x7fe5fc19a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.581+0000 7fe603110640 1 --2- 172.21.15.33:0/2712168604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5fc1066a0 0x7fe5fc19eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.581+0000 7fe603110640 1 --2- 172.21.15.33:0/2712168604 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe5fc100690 0x7fe5fc19a2f0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.581+0000 7fe603110640 1 -- 172.21.15.33:0/2712168604 >> 172.21.15.33:0/2712168604 conn(0x7fe5fc077820 msgr2=0x7fe5fc10c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:22.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.581+0000 7fe603110640 1 -- 172.21.15.33:0/2712168604 shutdown_connections 2024-03-19T03:01:22.590 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.581+0000 7fe603110640 1 -- 172.21.15.33:0/2712168604 wait complete. 2024-03-19T03:01:22.593 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T03:01:22.594 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:01:22.698 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.695+0000 7f1c20c0b640 1 -- 172.21.15.33:0/1177569800 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c1c101c80 msgr2=0x7f1c1c10d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:22.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.695+0000 7f1c20c0b640 1 --2- 172.21.15.33:0/1177569800 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c1c101c80 0x7f1c1c10d630 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f1c14009cd0 tx=0x7f1c1402c0f0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.695+0000 7f1c20c0b640 1 -- 172.21.15.33:0/1177569800 shutdown_connections 2024-03-19T03:01:22.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.695+0000 7f1c20c0b640 1 --2- 172.21.15.33:0/1177569800 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c1c101c80 0x7f1c1c10d630 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.695+0000 7f1c20c0b640 1 --2- 172.21.15.33:0/1177569800 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c1c1012c0 0x7f1c1c101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.695+0000 7f1c20c0b640 1 --2- 172.21.15.33:0/1177569800 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1c1c105750 0x7f1c1c105b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.695+0000 7f1c20c0b640 1 -- 172.21.15.33:0/1177569800 >> 172.21.15.33:0/1177569800 conn(0x7f1c1c075a30 msgr2=0x7f1c1c075e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:22.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.695+0000 7f1c20c0b640 1 -- 172.21.15.33:0/1177569800 shutdown_connections 2024-03-19T03:01:22.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.695+0000 7f1c20c0b640 1 -- 172.21.15.33:0/1177569800 wait complete. 2024-03-19T03:01:22.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.696+0000 7f1c20c0b640 1 Processor -- start 2024-03-19T03:01:22.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.696+0000 7f1c20c0b640 1 -- start start 2024-03-19T03:01:22.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.696+0000 7f1c20c0b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1c1c1012c0 0x7f1c1c196f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:22.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.696+0000 7f1c20c0b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c1c101c80 0x7f1c1c1974d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:22.699 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.696+0000 7f1c20c0b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c1c105750 0x7f1c1c19b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:22.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.696+0000 7f1c20c0b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1c110a30 con 0x7f1c1c105750 2024-03-19T03:01:22.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.696+0000 7f1c20c0b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1c1108b0 con 0x7f1c1c101c80 2024-03-19T03:01:22.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.696+0000 7f1c20c0b640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1c110bb0 con 0x7f1c1c1012c0 2024-03-19T03:01:22.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.696+0000 7f1c1ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c1c105750 0x7f1c1c19b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:22.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.696+0000 7f1c1ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c1c105750 0x7f1c1c19b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58760/0 (socket says 172.21.15.33:58760) 2024-03-19T03:01:22.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.696+0000 7f1c1ad9d640 1 -- 172.21.15.33:0/3393212323 learned_addr learned my addr 172.21.15.33:0/3393212323 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:22.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.696+0000 7f1c13fff640 1 --2- 172.21.15.33:0/3393212323 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c1c101c80 0x7f1c1c1974d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:22.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.696+0000 7f1c1a59c640 1 --2- 172.21.15.33:0/3393212323 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1c1c1012c0 0x7f1c1c196f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:22.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.697+0000 7f1c1ad9d640 1 -- 172.21.15.33:0/3393212323 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1c1c1012c0 msgr2=0x7f1c1c196f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:22.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.697+0000 7f1c1ad9d640 1 --2- 172.21.15.33:0/3393212323 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1c1c1012c0 0x7f1c1c196f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.697+0000 7f1c1ad9d640 1 -- 172.21.15.33:0/3393212323 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c1c101c80 msgr2=0x7f1c1c1974d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:22.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.697+0000 7f1c1ad9d640 1 --2- 172.21.15.33:0/3393212323 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c1c101c80 0x7f1c1c1974d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.697+0000 7f1c1ad9d640 1 -- 172.21.15.33:0/3393212323 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c1c19bf80 con 0x7f1c1c105750 2024-03-19T03:01:22.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.697+0000 7f1c1ad9d640 1 --2- 172.21.15.33:0/3393212323 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c1c105750 0x7f1c1c19b850 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f1c14005e30 tx=0x7f1c14035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:22.700 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.697+0000 7f1c137fe640 1 -- 172.21.15.33:0/3393212323 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c14044020 con 0x7f1c1c105750 2024-03-19T03:01:22.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.697+0000 7f1c137fe640 1 -- 172.21.15.33:0/3393212323 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c14003e20 con 0x7f1c1c105750 2024-03-19T03:01:22.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.697+0000 7f1c20c0b640 1 -- 172.21.15.33:0/3393212323 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c1c19c210 con 0x7f1c1c105750 2024-03-19T03:01:22.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.697+0000 7f1c137fe640 1 -- 172.21.15.33:0/3393212323 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c14039040 con 0x7f1c1c105750 2024-03-19T03:01:22.701 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.697+0000 7f1c20c0b640 1 -- 172.21.15.33:0/3393212323 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c1c1001b0 con 0x7f1c1c105750 2024-03-19T03:01:22.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.698+0000 7f1c20c0b640 1 -- 172.21.15.33:0/3393212323 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c1c069a60 con 0x7f1c1c105750 2024-03-19T03:01:22.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.699+0000 7f1c137fe640 1 -- 172.21.15.33:0/3393212323 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f1c14047020 con 0x7f1c1c105750 2024-03-19T03:01:22.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.700+0000 7f1c137fe640 1 --2- 172.21.15.33:0/3393212323 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f1c080a2970 0x7f1c080a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:22.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.700+0000 7f1c1a59c640 1 --2- 172.21.15.33:0/3393212323 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f1c080a2970 0x7f1c080a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:22.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.700+0000 7f1c137fe640 1 -- 172.21.15.33:0/3393212323 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f1c140f3d30 con 0x7f1c1c105750 2024-03-19T03:01:22.702 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.700+0000 7f1c1a59c640 1 --2- 172.21.15.33:0/3393212323 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f1c080a2970 0x7f1c080a4e30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c077030 tx=0x7f1c0c005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:22.704 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.703+0000 7f1c137fe640 1 -- 172.21.15.33:0/3393212323 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1c1404e020 con 0x7f1c1c105750 2024-03-19T03:01:22.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:22 smithi096 ceph-mon[35946]: pgmap v244: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-19T03:01:22.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.929+0000 7f1c20c0b640 1 -- 172.21.15.33:0/3393212323 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1c1c100bd0 con 0x7f1c1c105750 2024-03-19T03:01:22.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.929+0000 7f1c137fe640 1 -- 172.21.15.33:0/3393212323 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11867 (secure 0 0 0) 0x7f1c140c07d0 con 0x7f1c1c105750 2024-03-19T03:01:22.931 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T03:01:22.931 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:50:52.201798+0000","modified":"2024-03-19T02:53:11.748697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24488,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/3731232169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":3731232169},{"type":"v1","addr":"172.21.15.96:6839","nonce":3731232169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24457,24488,24509]},"id":1}]} 2024-03-19T03:01:22.931 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-03-19T03:01:22.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.931+0000 7f1c20c0b640 1 -- 172.21.15.33:0/3393212323 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f1c080a2970 msgr2=0x7f1c080a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:22.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.931+0000 7f1c20c0b640 1 --2- 172.21.15.33:0/3393212323 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f1c080a2970 0x7f1c080a4e30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c077030 tx=0x7f1c0c005f90 comp rx=0 tx=0).stop 2024-03-19T03:01:22.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.931+0000 7f1c20c0b640 1 -- 172.21.15.33:0/3393212323 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c1c105750 msgr2=0x7f1c1c19b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:22.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.931+0000 7f1c20c0b640 1 --2- 172.21.15.33:0/3393212323 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c1c105750 0x7f1c1c19b850 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f1c14005e30 tx=0x7f1c14035ce0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.931+0000 7f1c20c0b640 1 -- 172.21.15.33:0/3393212323 shutdown_connections 2024-03-19T03:01:22.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.931+0000 7f1c20c0b640 1 --2- 172.21.15.33:0/3393212323 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f1c080a2970 0x7f1c080a4e30 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.931+0000 7f1c20c0b640 1 --2- 172.21.15.33:0/3393212323 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1c1c105750 0x7f1c1c19b850 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.931+0000 7f1c20c0b640 1 --2- 172.21.15.33:0/3393212323 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c1c101c80 0x7f1c1c1974d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.931+0000 7f1c20c0b640 1 --2- 172.21.15.33:0/3393212323 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1c1c1012c0 0x7f1c1c196f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:22.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.931+0000 7f1c20c0b640 1 -- 172.21.15.33:0/3393212323 >> 172.21.15.33:0/3393212323 conn(0x7f1c1c075a30 msgr2=0x7f1c1c0ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:22.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.932+0000 7f1c20c0b640 1 -- 172.21.15.33:0/3393212323 shutdown_connections 2024-03-19T03:01:22.934 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:22.932+0000 7f1c20c0b640 1 -- 172.21.15.33:0/3393212323 wait complete. 2024-03-19T03:01:22.944 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T03:01:23.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.048+0000 7f42d8a85640 1 -- 172.21.15.33:0/2569146813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f42d4106750 msgr2=0x7f42d4106b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:23.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.048+0000 7f42d8a85640 1 --2- 172.21.15.33:0/2569146813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f42d4106750 0x7f42d4106b30 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f42c4009900 tx=0x7f42c402bc30 comp rx=0 tx=0).stop 2024-03-19T03:01:23.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.048+0000 7f42d8a85640 1 -- 172.21.15.33:0/2569146813 shutdown_connections 2024-03-19T03:01:23.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.048+0000 7f42d8a85640 1 --2- 172.21.15.33:0/2569146813 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f42d4100970 0x7f42d4100dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.048+0000 7f42d8a85640 1 --2- 172.21.15.33:0/2569146813 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42d4101310 0x7f42d410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.048+0000 7f42d8a85640 1 --2- 172.21.15.33:0/2569146813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f42d4106750 0x7f42d4106b30 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.048+0000 7f42d8a85640 1 -- 172.21.15.33:0/2569146813 >> 172.21.15.33:0/2569146813 conn(0x7f42d40774c0 msgr2=0x7f42d40778d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:23.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.048+0000 7f42d8a85640 1 -- 172.21.15.33:0/2569146813 shutdown_connections 2024-03-19T03:01:23.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.048+0000 7f42d8a85640 1 -- 172.21.15.33:0/2569146813 wait complete. 2024-03-19T03:01:23.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.049+0000 7f42d8a85640 1 Processor -- start 2024-03-19T03:01:23.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.049+0000 7f42d8a85640 1 -- start start 2024-03-19T03:01:23.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.049+0000 7f42d8a85640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42d4100970 0x7f42d4197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.049+0000 7f42d8a85640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f42d4101310 0x7f42d41984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.049+0000 7f42d8a85640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f42d4106750 0x7f42d419c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.049+0000 7f42d8a85640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f42d4110980 con 0x7f42d4106750 2024-03-19T03:01:23.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.049+0000 7f42d8a85640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f42d4110800 con 0x7f42d4100970 2024-03-19T03:01:23.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.049+0000 7f42d8a85640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f42d4110b00 con 0x7f42d4101310 2024-03-19T03:01:23.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.049+0000 7f42d2d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f42d4106750 0x7f42d419c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.049+0000 7f42d2d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f42d4106750 0x7f42d419c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58768/0 (socket says 172.21.15.33:58768) 2024-03-19T03:01:23.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.049+0000 7f42d259c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42d4100970 0x7f42d4197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.049+0000 7f42d2d9d640 1 -- 172.21.15.33:0/3966734211 learned_addr learned my addr 172.21.15.33:0/3966734211 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:23.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.049+0000 7f42cbfff640 1 --2- 172.21.15.33:0/3966734211 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f42d4101310 0x7f42d41984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.050+0000 7f42d2d9d640 1 -- 172.21.15.33:0/3966734211 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f42d4101310 msgr2=0x7f42d41984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:23.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.050+0000 7f42d2d9d640 1 --2- 172.21.15.33:0/3966734211 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f42d4101310 0x7f42d41984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.050+0000 7f42d2d9d640 1 -- 172.21.15.33:0/3966734211 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42d4100970 msgr2=0x7f42d4197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:23.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.050+0000 7f42d2d9d640 1 --2- 172.21.15.33:0/3966734211 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42d4100970 0x7f42d4197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.050+0000 7f42d2d9d640 1 -- 172.21.15.33:0/3966734211 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42d419cf90 con 0x7f42d4106750 2024-03-19T03:01:23.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.050+0000 7f42d259c640 1 --2- 172.21.15.33:0/3966734211 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42d4100970 0x7f42d4197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:23.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.050+0000 7f42d2d9d640 1 --2- 172.21.15.33:0/3966734211 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f42d4106750 0x7f42d419c860 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f42cc00eb10 tx=0x7f42cc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:23.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.050+0000 7f42cb7fe640 1 -- 172.21.15.33:0/3966734211 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42cc004b20 con 0x7f42d4106750 2024-03-19T03:01:23.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.050+0000 7f42d8a85640 1 -- 172.21.15.33:0/3966734211 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42d419d280 con 0x7f42d4106750 2024-03-19T03:01:23.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.050+0000 7f42cb7fe640 1 -- 172.21.15.33:0/3966734211 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f42cc004cc0 con 0x7f42d4106750 2024-03-19T03:01:23.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.050+0000 7f42cb7fe640 1 -- 172.21.15.33:0/3966734211 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42cc010690 con 0x7f42d4106750 2024-03-19T03:01:23.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.050+0000 7f42d8a85640 1 -- 172.21.15.33:0/3966734211 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42d41a4b10 con 0x7f42d4106750 2024-03-19T03:01:23.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.052+0000 7f42cb7fe640 1 -- 172.21.15.33:0/3966734211 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f42cc005020 con 0x7f42d4106750 2024-03-19T03:01:23.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.052+0000 7f42db675740 1 Processor -- start 2024-03-19T03:01:23.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.053+0000 7f42db675740 1 -- start start 2024-03-19T03:01:23.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.053+0000 7f42db675740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564486db12d0 0x564486db16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.053+0000 7f42db675740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564486db1bf0 0x564486dba290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.053+0000 7f42db675740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564486dba7d0 0x564486dbcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.053+0000 7f42cb7fe640 1 --2- 172.21.15.33:0/3966734211 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f42c00a2970 0x7f42c00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.053+0000 7f42db675740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564486d144b0 con 0x564486dba7d0 2024-03-19T03:01:23.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.053+0000 7f42db675740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x564486cbcd70 con 0x564486db12d0 2024-03-19T03:01:23.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.053+0000 7f42db675740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564486d10550 con 0x564486db1bf0 2024-03-19T03:01:23.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.053+0000 7f42d2d9d640 1 --2- 172.21.15.33:0/3966734211 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f42c00a2970 0x7f42c00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.053+0000 7f42cbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564486dba7d0 0x564486dbcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.053+0000 7f42d259c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564486db1bf0 0x564486dba290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.053+0000 7f42cbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564486dba7d0 0x564486dbcbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58774/0 (socket says 172.21.15.33:58774) 2024-03-19T03:01:23.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.053+0000 7f42cbfff640 1 -- 172.21.15.33:0/2944699751 learned_addr learned my addr 172.21.15.33:0/2944699751 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:23.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.053+0000 7f42d259c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564486db1bf0 0x564486dba290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:38790/0 (socket says 172.21.15.33:38790) 2024-03-19T03:01:23.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.053+0000 7f42cb7fe640 1 -- 172.21.15.33:0/3966734211 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f42cc0cf780 con 0x7f42d4106750 2024-03-19T03:01:23.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.053+0000 7f42cbfff640 1 --2- 172.21.15.33:0/2944699751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564486db12d0 0x564486db16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.053+0000 7f42cbfff640 1 -- 172.21.15.33:0/2944699751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564486db1bf0 msgr2=0x564486dba290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:23.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.053+0000 7f42cbfff640 1 --2- 172.21.15.33:0/2944699751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564486db1bf0 0x564486dba290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.053+0000 7f42cbfff640 1 -- 172.21.15.33:0/2944699751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564486db12d0 msgr2=0x564486db16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:23.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.053+0000 7f42cbfff640 1 --2- 172.21.15.33:0/2944699751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564486db12d0 0x564486db16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.053+0000 7f42cbfff640 1 -- 172.21.15.33:0/2944699751 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564486cdf210 con 0x564486dba7d0 2024-03-19T03:01:23.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.053+0000 7f42d259c640 1 --2- 172.21.15.33:0/2944699751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564486db1bf0 0x564486dba290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:23.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.054+0000 7f42d2d9d640 1 --2- 172.21.15.33:0/3966734211 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f42c00a2970 0x7f42c00a4e30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f42cc09b710 tx=0x7f42cc090ea0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:23.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.054+0000 7f42cbfff640 1 --2- 172.21.15.33:0/2944699751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564486dba7d0 0x564486dbcbc0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f42bc011810 tx=0x7f42bc0097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8036d54d1f025e51 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:23.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.054+0000 7f42c8ff9640 1 -- 172.21.15.33:0/2944699751 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc004030 con 0x564486dba7d0 2024-03-19T03:01:23.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.054+0000 7f42c8ff9640 1 -- 172.21.15.33:0/2944699751 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f42bc011e50 con 0x564486dba7d0 2024-03-19T03:01:23.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.054+0000 7f42c8ff9640 1 -- 172.21.15.33:0/2944699751 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc003850 con 0x564486dba7d0 2024-03-19T03:01:23.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.054+0000 7f42db675740 1 -- 172.21.15.33:0/2944699751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564486dba7d0 msgr2=0x564486dbcbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:23.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.054+0000 7f42db675740 1 --2- 172.21.15.33:0/2944699751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564486dba7d0 0x564486dbcbc0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f42bc011810 tx=0x7f42bc0097d0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.055+0000 7f42db675740 1 -- 172.21.15.33:0/2944699751 shutdown_connections 2024-03-19T03:01:23.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.055+0000 7f42db675740 1 --2- 172.21.15.33:0/2944699751 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564486db1bf0 0x564486dba290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.055+0000 7f42db675740 1 --2- 172.21.15.33:0/2944699751 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564486dba7d0 0x564486dbcbc0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.055+0000 7f42db675740 1 --2- 172.21.15.33:0/2944699751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564486db12d0 0x564486db16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.055+0000 7f42db675740 1 -- 172.21.15.33:0/2944699751 >> 172.21.15.33:0/2944699751 conn(0x564486daba00 msgr2=0x564486dabde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:23.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.055+0000 7f42db675740 1 -- 172.21.15.33:0/2944699751 shutdown_connections 2024-03-19T03:01:23.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.055+0000 7f42db675740 1 -- 172.21.15.33:0/2944699751 wait complete. 2024-03-19T03:01:23.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.055+0000 7f42db675740 1 -- start start 2024-03-19T03:01:23.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.055+0000 7f42db675740 1 Processor -- start 2024-03-19T03:01:23.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.055+0000 7f42db675740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564486db12d0 0x564486db7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.055+0000 7f42db675740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564486db1bf0 0x564486db7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.055+0000 7f42db675740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564486dba7d0 0x564486db8280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.055+0000 7f42db675740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564486dbd5c0 con 0x564486dba7d0 2024-03-19T03:01:23.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.055+0000 7f42db675740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x564486cd0310 con 0x564486db1bf0 2024-03-19T03:01:23.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.055+0000 7f42db675740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564486dbd740 con 0x564486db12d0 2024-03-19T03:01:23.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.055+0000 7f42db675740 10 client.0 _ll_register_callbacks cb 0x564486cc42d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:01:23.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.055+0000 7f42cbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564486dba7d0 0x564486db8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.055+0000 7f42cbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564486dba7d0 0x564486db8280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58778/0 (socket says 172.21.15.33:58778) 2024-03-19T03:01:23.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.055+0000 7f42d259c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564486db1bf0 0x564486db7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.055+0000 7f42cbfff640 1 -- 172.21.15.33:0/1048257591 learned_addr learned my addr 172.21.15.33:0/1048257591 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:23.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.055+0000 7f42cbfff640 1 --2- 172.21.15.33:0/1048257591 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564486db12d0 0x564486db7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.056+0000 7f42cbfff640 1 -- 172.21.15.33:0/1048257591 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564486db12d0 msgr2=0x564486db7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:23.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.056+0000 7f42cbfff640 1 --2- 172.21.15.33:0/1048257591 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564486db12d0 0x564486db7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.056+0000 7f42cbfff640 1 -- 172.21.15.33:0/1048257591 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564486db1bf0 msgr2=0x564486db7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:23.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.056+0000 7f42cbfff640 1 --2- 172.21.15.33:0/1048257591 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564486db1bf0 0x564486db7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.056+0000 7f42cbfff640 1 -- 172.21.15.33:0/1048257591 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564486a283f0 con 0x564486dba7d0 2024-03-19T03:01:23.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.056+0000 7f42d259c640 1 --2- 172.21.15.33:0/1048257591 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564486db1bf0 0x564486db7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:23.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.056+0000 7f42cbfff640 1 --2- 172.21.15.33:0/1048257591 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564486dba7d0 0x564486db8280 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f42bc001eb0 tx=0x7f42bc009b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d058a50f96336e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:23.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.056+0000 7f42b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:01:23.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.056+0000 7f42b9ffb640 1 -- 172.21.15.33:0/1048257591 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc005030 con 0x564486dba7d0 2024-03-19T03:01:23.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.056+0000 7f42b9ffb640 1 -- 172.21.15.33:0/1048257591 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f42bc019040 con 0x564486dba7d0 2024-03-19T03:01:23.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.056+0000 7f42db675740 1 -- 172.21.15.33:0/1048257591 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42bc011e50 con 0x564486dba7d0 2024-03-19T03:01:23.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.056+0000 7f42b9ffb640 1 -- 172.21.15.33:0/1048257591 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f42bc035050 con 0x564486dba7d0 2024-03-19T03:01:23.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.056+0000 7f42db675740 10 client.17328 fetch_fsmap learned FSMap version 25 2024-03-19T03:01:23.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.056+0000 7f42db675740 1 -- 172.21.15.33:0/1048257591 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564486af27d0 con 0x564486dba7d0 2024-03-19T03:01:23.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.056+0000 7f42b9ffb640 1 -- 172.21.15.33:0/1048257591 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f42bc011e50 con 0x564486dba7d0 2024-03-19T03:01:23.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.057+0000 7f42b9ffb640 1 -- 172.21.15.33:0/1048257591 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f42bc01f4b0 con 0x564486dba7d0 2024-03-19T03:01:23.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.057+0000 7f42db675740 10 client.17328 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:01:23.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.057+0000 7f42db675740 10 client.17328 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T03:01:23.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.057+0000 7f42db675740 20 client.17328 populate_metadata read hostname 'smithi033' 2024-03-19T03:01:23.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.057+0000 7f42db675740 1 --2- 172.21.15.33:0/1048257591 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x564486dcaa10 0x564486dcce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.057+0000 7f42db675740 4 client.17328 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:01:23.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.057+0000 7f42db675740 1 -- 172.21.15.33:0/1048257591 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564486d723e0 con 0x564486dcaa10 2024-03-19T03:01:23.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.057+0000 7f42d259c640 1 --2- 172.21.15.33:0/1048257591 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x564486dcaa10 0x564486dcce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.057+0000 7f42d259c640 1 --2- 172.21.15.33:0/1048257591 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x564486dcaa10 0x564486dcce00 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bd23f839146dfc53 server_cookie=dc600fc15bd804f9 in_seq=0 out_seq=0 2024-03-19T03:01:23.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.057+0000 7f42b9ffb640 10 client.17328 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T03:01:23.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.058+0000 7f42b9ffb640 1 -- 172.21.15.33:0/1048257591 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564486d723e0 con 0x564486dcaa10 2024-03-19T03:01:23.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.058+0000 7f42b9ffb640 10 client.17328 handle_command_reply: tid=0 2024-03-19T03:01:23.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.058+0000 7f42db675740 1 client.17328 shutdown 2024-03-19T03:01:23.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.058+0000 7f42db675740 1 -- 172.21.15.33:0/1048257591 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564486dba7d0 msgr2=0x564486db8280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:23.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.058+0000 7f42db675740 1 --2- 172.21.15.33:0/1048257591 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564486dba7d0 0x564486db8280 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f42bc001eb0 tx=0x7f42bc009b70 comp rx=0 tx=0).stop 2024-03-19T03:01:23.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.058+0000 7f42db675740 1 -- 172.21.15.33:0/1048257591 shutdown_connections 2024-03-19T03:01:23.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.058+0000 7f42db675740 1 --2- 172.21.15.33:0/1048257591 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x564486dcaa10 0x564486dcce00 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.058+0000 7f42db675740 1 --2- 172.21.15.33:0/1048257591 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564486dba7d0 0x564486db8280 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.058+0000 7f42db675740 1 --2- 172.21.15.33:0/1048257591 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564486db1bf0 0x564486db7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.058+0000 7f42b9ffb640 0 client.17328 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T03:01:23.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.058+0000 7f42db675740 1 --2- 172.21.15.33:0/1048257591 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564486db12d0 0x564486db7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.058+0000 7f42db675740 1 -- 172.21.15.33:0/1048257591 >> 172.21.15.33:0/1048257591 conn(0x564486daba00 msgr2=0x564486dad750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:23.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.058+0000 7f42db675740 1 -- 172.21.15.33:0/1048257591 shutdown_connections 2024-03-19T03:01:23.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.058+0000 7f42db675740 1 -- 172.21.15.33:0/1048257591 wait complete. 2024-03-19T03:01:23.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.058+0000 7f42db675740 20 client.17328 trim_cache size 0 max 16384 2024-03-19T03:01:23.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.071+0000 7f42db675740 1 Processor -- start 2024-03-19T03:01:23.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.071+0000 7f42db675740 1 -- start start 2024-03-19T03:01:23.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.071+0000 7f42db675740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564486db12d0 0x564486db9f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.071+0000 7f42db675740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564486db1bf0 0x564486dc6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.071+0000 7f42db675740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564486dba7d0 0x564486dd25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.071+0000 7f42db675740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f42d41b1590 con 0x564486db1bf0 2024-03-19T03:01:23.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.071+0000 7f42db675740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f42d41aee60 con 0x564486dba7d0 2024-03-19T03:01:23.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.071+0000 7f42db675740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f42d41accd0 con 0x564486db12d0 2024-03-19T03:01:23.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.072+0000 7f42d259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564486db1bf0 0x564486dc6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.072+0000 7f42d259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564486db1bf0 0x564486dc6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58784/0 (socket says 172.21.15.33:58784) 2024-03-19T03:01:23.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.072+0000 7f42d259c640 1 -- 172.21.15.33:0/552574249 learned_addr learned my addr 172.21.15.33:0/552574249 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:23.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.072+0000 7f42cbfff640 1 --2- 172.21.15.33:0/552574249 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564486dba7d0 0x564486dd25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.072+0000 7f42cbfff640 1 --2- 172.21.15.33:0/552574249 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564486db12d0 0x564486db9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.072+0000 7f42d259c640 1 -- 172.21.15.33:0/552574249 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564486db12d0 msgr2=0x564486db9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:23.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.072+0000 7f42d259c640 1 --2- 172.21.15.33:0/552574249 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564486db12d0 0x564486db9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.072+0000 7f42d259c640 1 -- 172.21.15.33:0/552574249 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564486dba7d0 msgr2=0x564486dd25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:23.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.072+0000 7f42d259c640 1 --2- 172.21.15.33:0/552574249 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564486dba7d0 0x564486dd25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.072+0000 7f42d259c640 1 -- 172.21.15.33:0/552574249 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564486dcd390 con 0x564486db1bf0 2024-03-19T03:01:23.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.072+0000 7f42d259c640 1 --2- 172.21.15.33:0/552574249 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564486db1bf0 0x564486dc6390 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f42c4006010 tx=0x7f42c4005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a594e5150e4b4bd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:23.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.072+0000 7f42bbfff640 1 -- 172.21.15.33:0/552574249 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c404ab10 con 0x564486db1bf0 2024-03-19T03:01:23.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.072+0000 7f42bbfff640 1 -- 172.21.15.33:0/552574249 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f42c404e030 con 0x564486db1bf0 2024-03-19T03:01:23.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.072+0000 7f42bbfff640 1 -- 172.21.15.33:0/552574249 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c404a7f0 con 0x564486db1bf0 2024-03-19T03:01:23.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.073+0000 7f42db675740 1 -- 172.21.15.33:0/552574249 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564486db1bf0 msgr2=0x564486dc6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:23.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.073+0000 7f42db675740 1 --2- 172.21.15.33:0/552574249 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564486db1bf0 0x564486dc6390 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f42c4006010 tx=0x7f42c4005e40 comp rx=0 tx=0).stop 2024-03-19T03:01:23.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.073+0000 7f42db675740 1 -- 172.21.15.33:0/552574249 shutdown_connections 2024-03-19T03:01:23.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.073+0000 7f42db675740 1 --2- 172.21.15.33:0/552574249 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564486dba7d0 0x564486dd25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.073+0000 7f42db675740 1 --2- 172.21.15.33:0/552574249 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564486db1bf0 0x564486dc6390 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.073+0000 7f42db675740 1 --2- 172.21.15.33:0/552574249 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564486db12d0 0x564486db9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.073+0000 7f42db675740 1 -- 172.21.15.33:0/552574249 >> 172.21.15.33:0/552574249 conn(0x564486daba00 msgr2=0x564486daaba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:23.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.073+0000 7f42db675740 1 -- 172.21.15.33:0/552574249 shutdown_connections 2024-03-19T03:01:23.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.073+0000 7f42db675740 1 -- 172.21.15.33:0/552574249 wait complete. 2024-03-19T03:01:23.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.073+0000 7f42db675740 1 -- start start 2024-03-19T03:01:23.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.073+0000 7f42db675740 1 Processor -- start 2024-03-19T03:01:23.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.073+0000 7f42db675740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564486db12d0 0x564486dc2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.073+0000 7f42db675740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564486db1bf0 0x564486dbdd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.073+0000 7f42db675740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564486dba7d0 0x564486dbe2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.073+0000 7f42db675740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f42d41ad320 con 0x564486db1bf0 2024-03-19T03:01:23.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.073+0000 7f42db675740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f42d41ae790 con 0x564486db12d0 2024-03-19T03:01:23.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.073+0000 7f42db675740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f42d41ad9f0 con 0x564486dba7d0 2024-03-19T03:01:23.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.073+0000 7f42db675740 10 client.0 _ll_register_callbacks cb 0x7f42bc009e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:01:23.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.073+0000 7f42d259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564486db1bf0 0x564486dbdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.074+0000 7f42cbfff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564486db12d0 0x564486dc2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.074+0000 7f42cbfff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564486db12d0 0x564486dc2a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:54508/0 (socket says 172.21.15.33:54508) 2024-03-19T03:01:23.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.074+0000 7f42cbfff640 1 -- 172.21.15.33:0/4024060100 learned_addr learned my addr 172.21.15.33:0/4024060100 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:23.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.074+0000 7f42d259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564486db1bf0 0x564486dbdd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58786/0 (socket says 172.21.15.33:58786) 2024-03-19T03:01:23.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.074+0000 7f42cbfff640 1 --2- 172.21.15.33:0/4024060100 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564486dba7d0 0x564486dbe2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.074+0000 7f42cbfff640 1 -- 172.21.15.33:0/4024060100 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564486dba7d0 msgr2=0x564486dbe2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:23.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.074+0000 7f42cbfff640 1 --2- 172.21.15.33:0/4024060100 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564486dba7d0 0x564486dbe2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.074+0000 7f42cbfff640 1 -- 172.21.15.33:0/4024060100 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564486db1bf0 msgr2=0x564486dbdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:23.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.074+0000 7f42cbfff640 1 --2- 172.21.15.33:0/4024060100 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564486db1bf0 0x564486dbdd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.074+0000 7f42cbfff640 1 -- 172.21.15.33:0/4024060100 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564486dbcbc0 con 0x564486db12d0 2024-03-19T03:01:23.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.074+0000 7f42d259c640 1 --2- 172.21.15.33:0/4024060100 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564486db1bf0 0x564486dbdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:23.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.074+0000 7f42cbfff640 1 --2- 172.21.15.33:0/4024060100 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564486db12d0 0x564486dc2a00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f42bc01fa20 tx=0x7f42bc005040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=293c6febd8be8a2b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:23.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.074+0000 7f42b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T03:01:23.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.074+0000 7f42b9ffb640 1 -- 172.21.15.33:0/4024060100 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc01f6e0 con 0x564486db12d0 2024-03-19T03:01:23.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.074+0000 7f42db675740 1 -- 172.21.15.33:0/4024060100 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42c404e030 con 0x564486db12d0 2024-03-19T03:01:23.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.074+0000 7f42b9ffb640 1 -- 172.21.15.33:0/4024060100 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f42bc011030 con 0x564486db12d0 2024-03-19T03:01:23.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.074+0000 7f42b9ffb640 1 -- 172.21.15.33:0/4024060100 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f42bc009990 con 0x564486db12d0 2024-03-19T03:01:23.081 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.074+0000 7f42db675740 10 client.27155 fetch_fsmap learned FSMap version 25 2024-03-19T03:01:23.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.074+0000 7f42db675740 1 -- 172.21.15.33:0/4024060100 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564486e307b0 con 0x564486db12d0 2024-03-19T03:01:23.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.074+0000 7f42b9ffb640 1 -- 172.21.15.33:0/4024060100 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f42c404e030 con 0x564486db12d0 2024-03-19T03:01:23.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.075+0000 7f42b9ffb640 1 -- 172.21.15.33:0/4024060100 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f42bc0035f0 con 0x564486db12d0 2024-03-19T03:01:23.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.075+0000 7f42db675740 10 client.27155 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:01:23.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.075+0000 7f42db675740 10 client.27155 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T03:01:23.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.075+0000 7f42db675740 20 client.27155 populate_metadata read hostname 'smithi033' 2024-03-19T03:01:23.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.075+0000 7f42db675740 1 --2- 172.21.15.33:0/4024060100 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x564486e30f00 0x564486e33320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.075+0000 7f42db675740 4 client.27155 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T03:01:23.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.075+0000 7f42db675740 1 -- 172.21.15.33:0/4024060100 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564486d219f0 con 0x564486e30f00 2024-03-19T03:01:23.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.075+0000 7f42d259c640 1 --2- 172.21.15.33:0/4024060100 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x564486e30f00 0x564486e33320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.075+0000 7f42b9ffb640 10 client.27155 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T03:01:23.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.075+0000 7f42d259c640 1 --2- 172.21.15.33:0/4024060100 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x564486e30f00 0x564486e33320 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3627f6699aa5f1d8 server_cookie=f57083fe861530f6 in_seq=0 out_seq=0 2024-03-19T03:01:23.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.075+0000 7f42b9ffb640 1 -- 172.21.15.33:0/4024060100 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564486d219f0 con 0x564486e30f00 2024-03-19T03:01:23.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.075+0000 7f42b9ffb640 10 client.27155 handle_command_reply: tid=0 2024-03-19T03:01:23.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.075+0000 7f42db675740 1 client.27155 shutdown 2024-03-19T03:01:23.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.075+0000 7f42db675740 1 -- 172.21.15.33:0/4024060100 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564486db12d0 msgr2=0x564486dc2a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:23.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.075+0000 7f42db675740 1 --2- 172.21.15.33:0/4024060100 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564486db12d0 0x564486dc2a00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f42bc01fa20 tx=0x7f42bc005040 comp rx=0 tx=0).stop 2024-03-19T03:01:23.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.075+0000 7f42db675740 1 -- 172.21.15.33:0/4024060100 shutdown_connections 2024-03-19T03:01:23.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.075+0000 7f42db675740 1 --2- 172.21.15.33:0/4024060100 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x564486e30f00 0x564486e33320 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.075+0000 7f42db675740 1 --2- 172.21.15.33:0/4024060100 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564486dba7d0 0x564486dbe2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.075+0000 7f42db675740 1 --2- 172.21.15.33:0/4024060100 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564486db1bf0 0x564486dbdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.075+0000 7f42db675740 1 --2- 172.21.15.33:0/4024060100 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564486db12d0 0x564486dc2a00 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.075+0000 7f42b9ffb640 0 client.27155 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T03:01:23.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.075+0000 7f42db675740 1 -- 172.21.15.33:0/4024060100 >> 172.21.15.33:0/4024060100 conn(0x564486daba00 msgr2=0x564486db77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:23.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.076+0000 7f42db675740 1 -- 172.21.15.33:0/4024060100 shutdown_connections 2024-03-19T03:01:23.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.076+0000 7f42db675740 1 -- 172.21.15.33:0/4024060100 wait complete. 2024-03-19T03:01:23.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.076+0000 7f42db675740 20 client.27155 trim_cache size 0 max 16384 2024-03-19T03:01:23.084 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T03:01:23.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.076+0000 7f42d8a85640 1 -- 172.21.15.33:0/3966734211 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f42c00a2970 msgr2=0x7f42c00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:23.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.076+0000 7f42d8a85640 1 --2- 172.21.15.33:0/3966734211 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f42c00a2970 0x7f42c00a4e30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f42cc09b710 tx=0x7f42cc090ea0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.076+0000 7f42d8a85640 1 -- 172.21.15.33:0/3966734211 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f42d4106750 msgr2=0x7f42d419c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:23.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.076+0000 7f42d8a85640 1 --2- 172.21.15.33:0/3966734211 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f42d4106750 0x7f42d419c860 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f42cc00eb10 tx=0x7f42cc00c2d0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.076+0000 7f42d8a85640 1 -- 172.21.15.33:0/3966734211 shutdown_connections 2024-03-19T03:01:23.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.076+0000 7f42d8a85640 1 --2- 172.21.15.33:0/3966734211 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f42c00a2970 0x7f42c00a4e30 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.076+0000 7f42d8a85640 1 --2- 172.21.15.33:0/3966734211 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f42d4101310 0x7f42d41984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.076+0000 7f42d8a85640 1 --2- 172.21.15.33:0/3966734211 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f42d4106750 0x7f42d419c860 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.076+0000 7f42d8a85640 1 --2- 172.21.15.33:0/3966734211 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f42d4100970 0x7f42d4197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.076+0000 7f42d8a85640 1 -- 172.21.15.33:0/3966734211 >> 172.21.15.33:0/3966734211 conn(0x7f42d40774c0 msgr2=0x7f42d406a030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:23.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.076+0000 7f42d8a85640 1 -- 172.21.15.33:0/3966734211 shutdown_connections 2024-03-19T03:01:23.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.076+0000 7f42d8a85640 1 -- 172.21.15.33:0/3966734211 wait complete. 2024-03-19T03:01:23.088 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T03:01:23.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.192+0000 7f84584d3640 1 -- 172.21.15.33:0/2967940337 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84501077a0 msgr2=0x7f8450107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:23.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.192+0000 7f84584d3640 1 --2- 172.21.15.33:0/2967940337 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84501077a0 0x7f8450107b80 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f844c009930 tx=0x7f844c02bc30 comp rx=0 tx=0).stop 2024-03-19T03:01:23.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.192+0000 7f84584d3640 1 -- 172.21.15.33:0/2967940337 shutdown_connections 2024-03-19T03:01:23.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.192+0000 7f84584d3640 1 --2- 172.21.15.33:0/2967940337 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8450102150 0x7f845010e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.192+0000 7f84584d3640 1 --2- 172.21.15.33:0/2967940337 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8450101790 0x7f8450101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.192+0000 7f84584d3640 1 --2- 172.21.15.33:0/2967940337 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f84501077a0 0x7f8450107b80 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.192+0000 7f84584d3640 1 -- 172.21.15.33:0/2967940337 >> 172.21.15.33:0/2967940337 conn(0x7f84500fd2c0 msgr2=0x7f84500ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:23.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.192+0000 7f84584d3640 1 -- 172.21.15.33:0/2967940337 shutdown_connections 2024-03-19T03:01:23.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.192+0000 7f84584d3640 1 -- 172.21.15.33:0/2967940337 wait complete. 2024-03-19T03:01:23.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.193+0000 7f84584d3640 1 Processor -- start 2024-03-19T03:01:23.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.193+0000 7f84584d3640 1 -- start start 2024-03-19T03:01:23.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.193+0000 7f84584d3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8450101790 0x7f8450197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.193+0000 7f84584d3640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8450102150 0x7f8450198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.193+0000 7f84584d3640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84501077a0 0x7f845019c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.193+0000 7f84584d3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8450111a20 con 0x7f8450101790 2024-03-19T03:01:23.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.193+0000 7f84584d3640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f84501118a0 con 0x7f8450102150 2024-03-19T03:01:23.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.193+0000 7f84584d3640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8450111ba0 con 0x7f84501077a0 2024-03-19T03:01:23.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.193+0000 7f845626f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8450101790 0x7f8450197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.193+0000 7f845626f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8450101790 0x7f8450197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58800/0 (socket says 172.21.15.33:58800) 2024-03-19T03:01:23.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.193+0000 7f845626f640 1 -- 172.21.15.33:0/82607847 learned_addr learned my addr 172.21.15.33:0/82607847 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:23.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.193+0000 7f8455a6e640 1 --2- 172.21.15.33:0/82607847 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8450102150 0x7f8450198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.193+0000 7f8456a70640 1 --2- 172.21.15.33:0/82607847 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84501077a0 0x7f845019c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.193+0000 7f845626f640 1 -- 172.21.15.33:0/82607847 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84501077a0 msgr2=0x7f845019c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:23.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.193+0000 7f845626f640 1 --2- 172.21.15.33:0/82607847 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84501077a0 0x7f845019c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.193+0000 7f845626f640 1 -- 172.21.15.33:0/82607847 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8450102150 msgr2=0x7f8450198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:23.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.193+0000 7f845626f640 1 --2- 172.21.15.33:0/82607847 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8450102150 0x7f8450198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.193+0000 7f845626f640 1 -- 172.21.15.33:0/82607847 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f845019cfd0 con 0x7f8450101790 2024-03-19T03:01:23.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.194+0000 7f845626f640 1 --2- 172.21.15.33:0/82607847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8450101790 0x7f8450197fe0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f844c009230 tx=0x7f844c02c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:23.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.194+0000 7f84477fe640 1 -- 172.21.15.33:0/82607847 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f844c043020 con 0x7f8450101790 2024-03-19T03:01:23.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.194+0000 7f84584d3640 1 -- 172.21.15.33:0/82607847 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f845019d260 con 0x7f8450101790 2024-03-19T03:01:23.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.194+0000 7f84477fe640 1 -- 172.21.15.33:0/82607847 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f844c02ca70 con 0x7f8450101790 2024-03-19T03:01:23.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.194+0000 7f84477fe640 1 -- 172.21.15.33:0/82607847 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f844c003a50 con 0x7f8450101790 2024-03-19T03:01:23.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.194+0000 7f84584d3640 1 -- 172.21.15.33:0/82607847 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84501a4b50 con 0x7f8450101790 2024-03-19T03:01:23.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.196+0000 7f84477fe640 1 -- 172.21.15.33:0/82607847 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f844c046020 con 0x7f8450101790 2024-03-19T03:01:23.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.196+0000 7f845b195740 1 Processor -- start 2024-03-19T03:01:23.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.197+0000 7f845b195740 1 -- start start 2024-03-19T03:01:23.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.197+0000 7f845b195740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55783386e2d0 0x55783386e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.197+0000 7f845b195740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55783386ebf0 0x557833877290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.197+0000 7f845b195740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5578338777d0 0x557833879bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.197+0000 7f845b195740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5578337d14b0 con 0x55783386e2d0 2024-03-19T03:01:23.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.197+0000 7f845b195740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557833779d70 con 0x5578338777d0 2024-03-19T03:01:23.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.197+0000 7f845b195740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5578337cd550 con 0x55783386ebf0 2024-03-19T03:01:23.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.197+0000 7f84477fe640 1 --2- 172.21.15.33:0/82607847 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f843c0a2970 0x7f843c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.197+0000 7f8456a70640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55783386e2d0 0x55783386e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.197+0000 7f8455a6e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5578338777d0 0x557833879bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.197+0000 7f8456a70640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55783386e2d0 0x55783386e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58808/0 (socket says 172.21.15.33:58808) 2024-03-19T03:01:23.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.197+0000 7f8456a70640 1 -- 172.21.15.33:0/1611356875 learned_addr learned my addr 172.21.15.33:0/1611356875 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:23.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.197+0000 7f845626f640 1 --2- 172.21.15.33:0/1611356875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55783386ebf0 0x557833877290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.197+0000 7f8456a70640 1 --2- 172.21.15.33:0/82607847 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f843c0a2970 0x7f843c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.197+0000 7f84477fe640 1 -- 172.21.15.33:0/82607847 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f844c0f1180 con 0x7f8450101790 2024-03-19T03:01:23.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.197+0000 7f8455a6e640 1 -- 172.21.15.33:0/1611356875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55783386ebf0 msgr2=0x557833877290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:23.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.197+0000 7f8455a6e640 1 --2- 172.21.15.33:0/1611356875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55783386ebf0 0x557833877290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.205 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.197+0000 7f8455a6e640 1 -- 172.21.15.33:0/1611356875 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55783386e2d0 msgr2=0x55783386e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:23.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.197+0000 7f8455a6e640 1 --2- 172.21.15.33:0/1611356875 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55783386e2d0 0x55783386e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.197+0000 7f8455a6e640 1 -- 172.21.15.33:0/1611356875 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55783379c210 con 0x5578338777d0 2024-03-19T03:01:23.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.198+0000 7f8456a70640 1 --2- 172.21.15.33:0/82607847 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f843c0a2970 0x7f843c0a4e30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f84480099e0 tx=0x7f8448004ab0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:23.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.198+0000 7f8455a6e640 1 --2- 172.21.15.33:0/1611356875 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5578338777d0 0x557833879bc0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8440009bf0 tx=0x7f8440011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fc15d83c0c006cb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:23.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.198+0000 7f8444bfe640 1 -- 172.21.15.33:0/1611356875 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f844001e070 con 0x5578338777d0 2024-03-19T03:01:23.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.198+0000 7f8444bfe640 1 -- 172.21.15.33:0/1611356875 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f844000eb20 con 0x5578338777d0 2024-03-19T03:01:23.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.198+0000 7f8444bfe640 1 -- 172.21.15.33:0/1611356875 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8440019c30 con 0x5578338777d0 2024-03-19T03:01:23.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.198+0000 7f845b195740 1 -- 172.21.15.33:0/1611356875 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5578338777d0 msgr2=0x557833879bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:23.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.198+0000 7f845b195740 1 --2- 172.21.15.33:0/1611356875 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5578338777d0 0x557833879bc0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8440009bf0 tx=0x7f8440011440 comp rx=0 tx=0).stop 2024-03-19T03:01:23.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.198+0000 7f845b195740 1 -- 172.21.15.33:0/1611356875 shutdown_connections 2024-03-19T03:01:23.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.198+0000 7f845b195740 1 --2- 172.21.15.33:0/1611356875 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55783386ebf0 0x557833877290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.198+0000 7f845b195740 1 --2- 172.21.15.33:0/1611356875 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5578338777d0 0x557833879bc0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.198+0000 7f845b195740 1 --2- 172.21.15.33:0/1611356875 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55783386e2d0 0x55783386e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.206 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.198+0000 7f845b195740 1 -- 172.21.15.33:0/1611356875 >> 172.21.15.33:0/1611356875 conn(0x557833868a00 msgr2=0x557833868de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:23.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.199+0000 7f845b195740 1 -- 172.21.15.33:0/1611356875 shutdown_connections 2024-03-19T03:01:23.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.199+0000 7f845b195740 1 -- 172.21.15.33:0/1611356875 wait complete. 2024-03-19T03:01:23.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.199+0000 7f845b195740 1 -- start start 2024-03-19T03:01:23.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.199+0000 7f845b195740 1 Processor -- start 2024-03-19T03:01:23.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.199+0000 7f845b195740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55783386e2d0 0x557833874800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.199+0000 7f845b195740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55783386ebf0 0x557833874d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.199+0000 7f845b195740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5578338777d0 0x557833875280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.199+0000 7f845b195740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55783387a5c0 con 0x55783386e2d0 2024-03-19T03:01:23.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.199+0000 7f845b195740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55783378d310 con 0x55783386ebf0 2024-03-19T03:01:23.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.199+0000 7f845b195740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55783387a740 con 0x5578338777d0 2024-03-19T03:01:23.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.199+0000 7f845b195740 10 client.0 _ll_register_callbacks cb 0x5578337812d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:01:23.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.199+0000 7f845626f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55783386e2d0 0x557833874800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.199+0000 7f845626f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55783386e2d0 0x557833874800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58814/0 (socket says 172.21.15.33:58814) 2024-03-19T03:01:23.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.199+0000 7f845626f640 1 -- 172.21.15.33:0/2468540321 learned_addr learned my addr 172.21.15.33:0/2468540321 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:23.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.199+0000 7f8455a6e640 1 --2- 172.21.15.33:0/2468540321 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55783386ebf0 0x557833874d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.207 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.199+0000 7f8456a70640 1 --2- 172.21.15.33:0/2468540321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5578338777d0 0x557833875280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.200+0000 7f845626f640 1 -- 172.21.15.33:0/2468540321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5578338777d0 msgr2=0x557833875280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:23.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.200+0000 7f845626f640 1 --2- 172.21.15.33:0/2468540321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5578338777d0 0x557833875280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.200+0000 7f845626f640 1 -- 172.21.15.33:0/2468540321 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55783386ebf0 msgr2=0x557833874d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:23.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.200+0000 7f845626f640 1 --2- 172.21.15.33:0/2468540321 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55783386ebf0 0x557833874d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.200+0000 7f845626f640 1 -- 172.21.15.33:0/2468540321 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578334e53f0 con 0x55783386e2d0 2024-03-19T03:01:23.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.200+0000 7f8456a70640 1 --2- 172.21.15.33:0/2468540321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5578338777d0 0x557833875280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:01:23.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.200+0000 7f845626f640 1 --2- 172.21.15.33:0/2468540321 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55783386e2d0 0x557833874800 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f844c000c00 tx=0x7f844c08c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d04038340f564122 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:23.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.200+0000 7f8439ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:01:23.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.200+0000 7f8439ffb640 1 -- 172.21.15.33:0/2468540321 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f844c0c5c60 con 0x55783386e2d0 2024-03-19T03:01:23.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.200+0000 7f8439ffb640 1 -- 172.21.15.33:0/2468540321 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f844c0ee720 con 0x55783386e2d0 2024-03-19T03:01:23.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.200+0000 7f845b195740 1 -- 172.21.15.33:0/2468540321 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f844000eb20 con 0x55783386e2d0 2024-03-19T03:01:23.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.200+0000 7f8439ffb640 1 -- 172.21.15.33:0/2468540321 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f844c0ee920 con 0x55783386e2d0 2024-03-19T03:01:23.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.200+0000 7f845b195740 10 client.17346 fetch_fsmap learned FSMap version 25 2024-03-19T03:01:23.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.200+0000 7f845b195740 1 -- 172.21.15.33:0/2468540321 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5578335af7d0 con 0x55783386e2d0 2024-03-19T03:01:23.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.200+0000 7f8439ffb640 1 -- 172.21.15.33:0/2468540321 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f844000eb20 con 0x55783386e2d0 2024-03-19T03:01:23.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.200+0000 7f8439ffb640 1 -- 172.21.15.33:0/2468540321 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f844c0bd720 con 0x55783386e2d0 2024-03-19T03:01:23.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.200+0000 7f845b195740 10 client.17346 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:01:23.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.200+0000 7f845b195740 10 client.17346 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T03:01:23.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.201+0000 7f845b195740 20 client.17346 populate_metadata read hostname 'smithi033' 2024-03-19T03:01:23.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.201+0000 7f845b195740 1 --2- 172.21.15.33:0/2468540321 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x557833887a30 0x557833889e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.201+0000 7f845b195740 4 client.17346 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:01:23.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.201+0000 7f845b195740 1 -- 172.21.15.33:0/2468540321 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55783382f3e0 con 0x557833887a30 2024-03-19T03:01:23.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.201+0000 7f8455a6e640 1 --2- 172.21.15.33:0/2468540321 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x557833887a30 0x557833889e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.201+0000 7f8439ffb640 10 client.17346 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T03:01:23.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.201+0000 7f8455a6e640 1 --2- 172.21.15.33:0/2468540321 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x557833887a30 0x557833889e80 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cfa429200682a8ba server_cookie=c8ee4c1e05d8b944 in_seq=0 out_seq=0 2024-03-19T03:01:23.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.202+0000 7f8439ffb640 1 -- 172.21.15.33:0/2468540321 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55783382f3e0 con 0x557833887a30 2024-03-19T03:01:23.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.202+0000 7f8439ffb640 10 client.17346 handle_command_reply: tid=0 2024-03-19T03:01:23.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.202+0000 7f845b195740 1 client.17346 shutdown 2024-03-19T03:01:23.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.202+0000 7f845b195740 1 -- 172.21.15.33:0/2468540321 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55783386e2d0 msgr2=0x557833874800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:23.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.202+0000 7f845b195740 1 --2- 172.21.15.33:0/2468540321 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55783386e2d0 0x557833874800 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f844c000c00 tx=0x7f844c08c2d0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.202+0000 7f845b195740 1 -- 172.21.15.33:0/2468540321 shutdown_connections 2024-03-19T03:01:23.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.202+0000 7f845b195740 1 --2- 172.21.15.33:0/2468540321 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x557833887a30 0x557833889e80 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.202+0000 7f845b195740 1 --2- 172.21.15.33:0/2468540321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5578338777d0 0x557833875280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.202+0000 7f8439ffb640 0 client.17346 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T03:01:23.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.202+0000 7f845b195740 1 --2- 172.21.15.33:0/2468540321 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55783386ebf0 0x557833874d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.202+0000 7f845b195740 1 --2- 172.21.15.33:0/2468540321 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55783386e2d0 0x557833874800 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.202+0000 7f845b195740 1 -- 172.21.15.33:0/2468540321 >> 172.21.15.33:0/2468540321 conn(0x557833868a00 msgr2=0x55783386a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:23.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.202+0000 7f845b195740 1 -- 172.21.15.33:0/2468540321 shutdown_connections 2024-03-19T03:01:23.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.202+0000 7f845b195740 1 -- 172.21.15.33:0/2468540321 wait complete. 2024-03-19T03:01:23.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.202+0000 7f845b195740 20 client.17346 trim_cache size 0 max 16384 2024-03-19T03:01:23.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.215+0000 7f845b195740 1 Processor -- start 2024-03-19T03:01:23.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.215+0000 7f845b195740 1 -- start start 2024-03-19T03:01:23.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.216+0000 7f845b195740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55783386e2d0 0x557833876f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.216+0000 7f845b195740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55783386ebf0 0x55783386a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.216+0000 7f845b195740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5578338777d0 0x55783388f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.216+0000 7f845b195740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f84501b15c0 con 0x55783386ebf0 2024-03-19T03:01:23.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.216+0000 7f845b195740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f84501aee90 con 0x55783386e2d0 2024-03-19T03:01:23.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.216+0000 7f845b195740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f84501acd00 con 0x5578338777d0 2024-03-19T03:01:23.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.216+0000 7f8455a6e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55783386ebf0 0x55783386a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.216+0000 7f8455a6e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55783386ebf0 0x55783386a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58818/0 (socket says 172.21.15.33:58818) 2024-03-19T03:01:23.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.216+0000 7f8455a6e640 1 -- 172.21.15.33:0/2243223897 learned_addr learned my addr 172.21.15.33:0/2243223897 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:23.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.216+0000 7f845626f640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55783386e2d0 0x557833876f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.216+0000 7f8456a70640 1 --2- 172.21.15.33:0/2243223897 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5578338777d0 0x55783388f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.216+0000 7f8455a6e640 1 -- 172.21.15.33:0/2243223897 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5578338777d0 msgr2=0x55783388f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:23.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.216+0000 7f8455a6e640 1 --2- 172.21.15.33:0/2243223897 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5578338777d0 0x55783388f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.216+0000 7f8455a6e640 1 -- 172.21.15.33:0/2243223897 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55783386e2d0 msgr2=0x557833876f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:23.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.216+0000 7f8455a6e640 1 --2- 172.21.15.33:0/2243223897 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55783386e2d0 0x557833876f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.220 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.216+0000 7f8455a6e640 1 -- 172.21.15.33:0/2243223897 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55783386ea20 con 0x55783386ebf0 2024-03-19T03:01:23.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.216+0000 7f8455a6e640 1 --2- 172.21.15.33:0/2243223897 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55783386ebf0 0x55783386a670 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f844001c040 tx=0x7f84400198b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15cda3b108353c2e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:23.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.216+0000 7f843bfff640 1 -- 172.21.15.33:0/2243223897 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f844001e040 con 0x55783386ebf0 2024-03-19T03:01:23.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.216+0000 7f843bfff640 1 -- 172.21.15.33:0/2243223897 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8440005560 con 0x55783386ebf0 2024-03-19T03:01:23.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.216+0000 7f843bfff640 1 -- 172.21.15.33:0/2243223897 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8440003d10 con 0x55783386ebf0 2024-03-19T03:01:23.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.217+0000 7f845b195740 1 -- 172.21.15.33:0/2243223897 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55783386ebf0 msgr2=0x55783386a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:23.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.217+0000 7f845b195740 1 --2- 172.21.15.33:0/2243223897 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55783386ebf0 0x55783386a670 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f844001c040 tx=0x7f84400198b0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.217+0000 7f845b195740 1 -- 172.21.15.33:0/2243223897 shutdown_connections 2024-03-19T03:01:23.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.217+0000 7f845b195740 1 --2- 172.21.15.33:0/2243223897 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5578338777d0 0x55783388f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.217+0000 7f845b195740 1 --2- 172.21.15.33:0/2243223897 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55783386ebf0 0x55783386a670 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.217+0000 7f845b195740 1 --2- 172.21.15.33:0/2243223897 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55783386e2d0 0x557833876f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.217+0000 7f845b195740 1 -- 172.21.15.33:0/2243223897 >> 172.21.15.33:0/2243223897 conn(0x557833868a00 msgr2=0x557833867310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:23.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.217+0000 7f845b195740 1 -- 172.21.15.33:0/2243223897 shutdown_connections 2024-03-19T03:01:23.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.217+0000 7f845b195740 1 -- 172.21.15.33:0/2243223897 wait complete. 2024-03-19T03:01:23.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.217+0000 7f845b195740 1 -- start start 2024-03-19T03:01:23.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.217+0000 7f845b195740 1 Processor -- start 2024-03-19T03:01:23.221 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.217+0000 7f845b195740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55783386e2d0 0x557833871bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.217+0000 7f845b195740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55783386ebf0 0x557833872100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.217+0000 7f845b195740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5578338777d0 0x5578338ecd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.218+0000 7f845b195740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f84501ad350 con 0x55783386ebf0 2024-03-19T03:01:23.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.218+0000 7f845b195740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f84501ae7c0 con 0x5578338777d0 2024-03-19T03:01:23.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.218+0000 7f845b195740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f84501ada20 con 0x55783386e2d0 2024-03-19T03:01:23.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.218+0000 7f845b195740 10 client.0 _ll_register_callbacks cb 0x7f844c009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:01:23.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.218+0000 7f8455a6e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55783386ebf0 0x557833872100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.218+0000 7f845626f640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55783386e2d0 0x557833871bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.218+0000 7f8455a6e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55783386ebf0 0x557833872100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58820/0 (socket says 172.21.15.33:58820) 2024-03-19T03:01:23.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.218+0000 7f845626f640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55783386e2d0 0x557833871bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:38874/0 (socket says 172.21.15.33:38874) 2024-03-19T03:01:23.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.218+0000 7f8455a6e640 1 -- 172.21.15.33:0/2802567118 learned_addr learned my addr 172.21.15.33:0/2802567118 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:23.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.218+0000 7f8456a70640 1 --2- 172.21.15.33:0/2802567118 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5578338777d0 0x5578338ecd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.218+0000 7f845626f640 1 -- 172.21.15.33:0/2802567118 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5578338777d0 msgr2=0x5578338ecd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:23.222 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.218+0000 7f845626f640 1 --2- 172.21.15.33:0/2802567118 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5578338777d0 0x5578338ecd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.218+0000 7f845626f640 1 -- 172.21.15.33:0/2802567118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55783386ebf0 msgr2=0x557833872100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:23.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.218+0000 7f845626f640 1 --2- 172.21.15.33:0/2802567118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55783386ebf0 0x557833872100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.218+0000 7f845626f640 1 -- 172.21.15.33:0/2802567118 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578338f17b0 con 0x55783386e2d0 2024-03-19T03:01:23.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.218+0000 7f8456a70640 1 --2- 172.21.15.33:0/2802567118 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5578338777d0 0x5578338ecd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:23.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.218+0000 7f8455a6e640 1 --2- 172.21.15.33:0/2802567118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55783386ebf0 0x557833872100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:23.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.218+0000 7f845626f640 1 --2- 172.21.15.33:0/2802567118 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55783386e2d0 0x557833871bc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f844c0bd1c0 tx=0x7f844c0ee930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d54692a4c3743df4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:23.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.218+0000 7f8439ffb640 10 client.0 ms_handle_connect on v2:172.21.15.155:3300/0 2024-03-19T03:01:23.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.220+0000 7f8439ffb640 1 -- 172.21.15.33:0/2802567118 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f844c0c57d0 con 0x55783386e2d0 2024-03-19T03:01:23.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.220+0000 7f8439ffb640 1 -- 172.21.15.33:0/2802567118 <== mon.1 v2:172.21.15.155:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f844c0d6260 con 0x55783386e2d0 2024-03-19T03:01:23.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.220+0000 7f845b195740 1 -- 172.21.15.33:0/2802567118 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8440005560 con 0x55783386e2d0 2024-03-19T03:01:23.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.223+0000 7f8439ffb640 1 -- 172.21.15.33:0/2802567118 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f844c0bd930 con 0x55783386e2d0 2024-03-19T03:01:23.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.223+0000 7f845b195740 10 client.27034 fetch_fsmap learned FSMap version 25 2024-03-19T03:01:23.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.223+0000 7f845b195740 1 -- 172.21.15.33:0/2802567118 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5578338ed480 con 0x55783386e2d0 2024-03-19T03:01:23.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.223+0000 7f8439ffb640 1 -- 172.21.15.33:0/2802567118 <== mon.1 v2:172.21.15.155:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f844c0a9020 con 0x55783386e2d0 2024-03-19T03:01:23.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.224+0000 7f8439ffb640 1 -- 172.21.15.33:0/2802567118 <== mon.1 v2:172.21.15.155:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f844c0c11e0 con 0x55783386e2d0 2024-03-19T03:01:23.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.224+0000 7f845b195740 10 client.27034 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:01:23.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.224+0000 7f845b195740 10 client.27034 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T03:01:23.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.224+0000 7f845b195740 20 client.27034 populate_metadata read hostname 'smithi033' 2024-03-19T03:01:23.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.224+0000 7f845b195740 1 --2- 172.21.15.33:0/2802567118 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x5578338edac0 0x5578338efee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.224+0000 7f845b195740 4 client.27034 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T03:01:23.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.224+0000 7f845b195740 1 -- 172.21.15.33:0/2802567118 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5578337de9f0 con 0x5578338edac0 2024-03-19T03:01:23.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.224+0000 7f8455a6e640 1 --2- 172.21.15.33:0/2802567118 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x5578338edac0 0x5578338efee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.224+0000 7f8439ffb640 10 client.27034 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T03:01:23.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.224+0000 7f8455a6e640 1 --2- 172.21.15.33:0/2802567118 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x5578338edac0 0x5578338efee0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6afc305f79a89017 server_cookie=51fe1ac93b2a3573 in_seq=0 out_seq=0 2024-03-19T03:01:23.227 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.224+0000 7f8439ffb640 1 -- 172.21.15.33:0/2802567118 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5578337de9f0 con 0x5578338edac0 2024-03-19T03:01:23.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.224+0000 7f8439ffb640 10 client.27034 handle_command_reply: tid=0 2024-03-19T03:01:23.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.224+0000 7f845b195740 1 client.27034 shutdown 2024-03-19T03:01:23.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.225+0000 7f845b195740 1 -- 172.21.15.33:0/2802567118 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55783386e2d0 msgr2=0x557833871bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:23.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.225+0000 7f845b195740 1 --2- 172.21.15.33:0/2802567118 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55783386e2d0 0x557833871bc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f844c0bd1c0 tx=0x7f844c0ee930 comp rx=0 tx=0).stop 2024-03-19T03:01:23.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.225+0000 7f845b195740 1 -- 172.21.15.33:0/2802567118 shutdown_connections 2024-03-19T03:01:23.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.225+0000 7f845b195740 1 --2- 172.21.15.33:0/2802567118 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x5578338edac0 0x5578338efee0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.225+0000 7f845b195740 1 --2- 172.21.15.33:0/2802567118 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5578338777d0 0x5578338ecd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.225+0000 7f845b195740 1 --2- 172.21.15.33:0/2802567118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55783386ebf0 0x557833872100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.225+0000 7f845b195740 1 --2- 172.21.15.33:0/2802567118 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55783386e2d0 0x557833871bc0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.225+0000 7f8439ffb640 0 client.27034 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T03:01:23.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.225+0000 7f845b195740 1 -- 172.21.15.33:0/2802567118 >> 172.21.15.33:0/2802567118 conn(0x557833868a00 msgr2=0x55783386a460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:23.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.225+0000 7f845b195740 1 -- 172.21.15.33:0/2802567118 shutdown_connections 2024-03-19T03:01:23.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.225+0000 7f845b195740 1 -- 172.21.15.33:0/2802567118 wait complete. 2024-03-19T03:01:23.228 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.225+0000 7f845b195740 20 client.27034 trim_cache size 0 max 16384 2024-03-19T03:01:23.229 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T03:01:23.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.225+0000 7f84584d3640 1 -- 172.21.15.33:0/82607847 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f843c0a2970 msgr2=0x7f843c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:23.229 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.225+0000 7f84584d3640 1 --2- 172.21.15.33:0/82607847 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f843c0a2970 0x7f843c0a4e30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f84480099e0 tx=0x7f8448004ab0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.225+0000 7f84584d3640 1 -- 172.21.15.33:0/82607847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8450101790 msgr2=0x7f8450197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:23.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.225+0000 7f84584d3640 1 --2- 172.21.15.33:0/82607847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8450101790 0x7f8450197fe0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f844c009230 tx=0x7f844c02c5d0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.225+0000 7f84584d3640 1 -- 172.21.15.33:0/82607847 shutdown_connections 2024-03-19T03:01:23.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.225+0000 7f84584d3640 1 --2- 172.21.15.33:0/82607847 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f843c0a2970 0x7f843c0a4e30 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.225+0000 7f84584d3640 1 --2- 172.21.15.33:0/82607847 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84501077a0 0x7f845019c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.225+0000 7f84584d3640 1 --2- 172.21.15.33:0/82607847 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8450102150 0x7f8450198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.225+0000 7f84584d3640 1 --2- 172.21.15.33:0/82607847 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8450101790 0x7f8450197fe0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.225+0000 7f84584d3640 1 -- 172.21.15.33:0/82607847 >> 172.21.15.33:0/82607847 conn(0x7f84500fd2c0 msgr2=0x7f84500fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:23.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.225+0000 7f84584d3640 1 -- 172.21.15.33:0/82607847 shutdown_connections 2024-03-19T03:01:23.230 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.226+0000 7f84584d3640 1 -- 172.21.15.33:0/82607847 wait complete. 2024-03-19T03:01:23.237 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T03:01:23.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.341+0000 7f1452959640 1 -- 172.21.15.33:0/3651397448 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f144c101060 msgr2=0x7f144c10d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:23.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.341+0000 7f1452959640 1 --2- 172.21.15.33:0/3651397448 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f144c101060 0x7f144c10d570 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f1444009cd0 tx=0x7f144402c0f0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.341+0000 7f1452959640 1 -- 172.21.15.33:0/3651397448 shutdown_connections 2024-03-19T03:01:23.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.341+0000 7f1452959640 1 --2- 172.21.15.33:0/3651397448 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f144c1006a0 0x7f144c100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.341+0000 7f1452959640 1 --2- 172.21.15.33:0/3651397448 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f144c101060 0x7f144c10d570 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.341+0000 7f1452959640 1 --2- 172.21.15.33:0/3651397448 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f144c1066b0 0x7f144c106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.341+0000 7f1452959640 1 -- 172.21.15.33:0/3651397448 >> 172.21.15.33:0/3651397448 conn(0x7f144c0778d0 msgr2=0x7f144c077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:23.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.341+0000 7f1452959640 1 -- 172.21.15.33:0/3651397448 shutdown_connections 2024-03-19T03:01:23.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.341+0000 7f1452959640 1 -- 172.21.15.33:0/3651397448 wait complete. 2024-03-19T03:01:23.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.341+0000 7f1452959640 1 Processor -- start 2024-03-19T03:01:23.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.341+0000 7f1452959640 1 -- start start 2024-03-19T03:01:23.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.342+0000 7f1452959640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f144c1006a0 0x7f144c197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.342+0000 7f1452959640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f144c101060 0x7f144c1984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.342+0000 7f1452959640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f144c1066b0 0x7f144c19c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.342+0000 7f1452959640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f144c110970 con 0x7f144c1006a0 2024-03-19T03:01:23.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.342+0000 7f1452959640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f144c1107f0 con 0x7f144c1066b0 2024-03-19T03:01:23.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.342+0000 7f1452959640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f144c110af0 con 0x7f144c101060 2024-03-19T03:01:23.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.342+0000 7f144bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f144c1006a0 0x7f144c197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.342+0000 7f144bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f144c1006a0 0x7f144c197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58836/0 (socket says 172.21.15.33:58836) 2024-03-19T03:01:23.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.342+0000 7f144bfff640 1 -- 172.21.15.33:0/1769703172 learned_addr learned my addr 172.21.15.33:0/1769703172 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:23.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.342+0000 7f1450ef6640 1 --2- 172.21.15.33:0/1769703172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f144c1066b0 0x7f144c19c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.342+0000 7f144b7fe640 1 --2- 172.21.15.33:0/1769703172 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f144c101060 0x7f144c1984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.342+0000 7f144bfff640 1 -- 172.21.15.33:0/1769703172 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f144c101060 msgr2=0x7f144c1984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:23.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.342+0000 7f144bfff640 1 --2- 172.21.15.33:0/1769703172 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f144c101060 0x7f144c1984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.342+0000 7f144bfff640 1 -- 172.21.15.33:0/1769703172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f144c1066b0 msgr2=0x7f144c19c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:23.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.342+0000 7f144bfff640 1 --2- 172.21.15.33:0/1769703172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f144c1066b0 0x7f144c19c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.342+0000 7f144bfff640 1 -- 172.21.15.33:0/1769703172 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f144c19cf70 con 0x7f144c1006a0 2024-03-19T03:01:23.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.342+0000 7f144b7fe640 1 --2- 172.21.15.33:0/1769703172 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f144c101060 0x7f144c1984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:01:23.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.342+0000 7f144bfff640 1 --2- 172.21.15.33:0/1769703172 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f144c1006a0 0x7f144c197f80 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f143c00ea60 tx=0x7f143c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:23.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.342+0000 7f14497fa640 1 -- 172.21.15.33:0/1769703172 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143c00cde0 con 0x7f144c1006a0 2024-03-19T03:01:23.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.342+0000 7f1452959640 1 -- 172.21.15.33:0/1769703172 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f144c19d260 con 0x7f144c1006a0 2024-03-19T03:01:23.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.342+0000 7f14497fa640 1 -- 172.21.15.33:0/1769703172 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f143c0035f0 con 0x7f144c1006a0 2024-03-19T03:01:23.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.343+0000 7f14497fa640 1 -- 172.21.15.33:0/1769703172 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143c010690 con 0x7f144c1006a0 2024-03-19T03:01:23.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.343+0000 7f1452959640 1 -- 172.21.15.33:0/1769703172 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f144c1a4af0 con 0x7f144c1006a0 2024-03-19T03:01:23.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.345+0000 7f14497fa640 1 -- 172.21.15.33:0/1769703172 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f143c0037a0 con 0x7f144c1006a0 2024-03-19T03:01:23.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.345+0000 7f1455613740 1 Processor -- start 2024-03-19T03:01:23.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.345+0000 7f1455613740 1 -- start start 2024-03-19T03:01:23.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.345+0000 7f1455613740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575a7dd52d0 0x5575a7dd56b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.345+0000 7f1455613740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575a7dd5bf0 0x5575a7dde290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.345+0000 7f1455613740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575a7dde7d0 0x5575a7de0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.345+0000 7f1455613740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5575a7d384b0 con 0x5575a7dd52d0 2024-03-19T03:01:23.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.345+0000 7f1455613740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5575a7ce0d70 con 0x5575a7dde7d0 2024-03-19T03:01:23.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.345+0000 7f1455613740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5575a7d34550 con 0x5575a7dd5bf0 2024-03-19T03:01:23.354 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.345+0000 7f14497fa640 1 --2- 172.21.15.33:0/1769703172 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f14340a2970 0x7f14340a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.345+0000 7f1450ef6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575a7dd52d0 0x5575a7dd56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.345+0000 7f1450ef6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575a7dd52d0 0x5575a7dd56b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58850/0 (socket says 172.21.15.33:58850) 2024-03-19T03:01:23.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.345+0000 7f1450ef6640 1 -- 172.21.15.33:0/2957655758 learned_addr learned my addr 172.21.15.33:0/2957655758 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:23.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.345+0000 7f144b7fe640 1 --2- 172.21.15.33:0/2957655758 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575a7dde7d0 0x5575a7de0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.345+0000 7f144bfff640 1 --2- 172.21.15.33:0/2957655758 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575a7dd5bf0 0x5575a7dde290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.345+0000 7f1450ef6640 1 --2- 172.21.15.33:0/1769703172 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f14340a2970 0x7f14340a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.345+0000 7f14497fa640 1 -- 172.21.15.33:0/1769703172 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f143c0cf4d0 con 0x7f144c1006a0 2024-03-19T03:01:23.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.346+0000 7f1450ef6640 1 -- 172.21.15.33:0/2957655758 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575a7dd5bf0 msgr2=0x5575a7dde290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:23.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.346+0000 7f1450ef6640 1 --2- 172.21.15.33:0/2957655758 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575a7dd5bf0 0x5575a7dde290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.346+0000 7f1450ef6640 1 -- 172.21.15.33:0/2957655758 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575a7dde7d0 msgr2=0x5575a7de0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:23.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.346+0000 7f1450ef6640 1 --2- 172.21.15.33:0/2957655758 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575a7dde7d0 0x5575a7de0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.346+0000 7f1450ef6640 1 -- 172.21.15.33:0/2957655758 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575a7d03210 con 0x5575a7dd52d0 2024-03-19T03:01:23.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.346+0000 7f144b7fe640 1 --2- 172.21.15.33:0/2957655758 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575a7dde7d0 0x5575a7de0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:23.355 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.346+0000 7f1450ef6640 1 --2- 172.21.15.33:0/2957655758 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575a7dd52d0 0x5575a7dd56b0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f1444005e30 tx=0x7f14440326d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4f237bcc08c9a6e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:23.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.346+0000 7f1450ef6640 1 --2- 172.21.15.33:0/1769703172 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f14340a2970 0x7f14340a4e30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1444009ca0 tx=0x7f1444042000 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:23.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.346+0000 7f14327fc640 1 -- 172.21.15.33:0/2957655758 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1444039040 con 0x5575a7dd52d0 2024-03-19T03:01:23.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.346+0000 7f14327fc640 1 -- 172.21.15.33:0/2957655758 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1444032e00 con 0x5575a7dd52d0 2024-03-19T03:01:23.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.346+0000 7f14327fc640 1 -- 172.21.15.33:0/2957655758 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1444004650 con 0x5575a7dd52d0 2024-03-19T03:01:23.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.347+0000 7f1455613740 1 -- 172.21.15.33:0/2957655758 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575a7dd52d0 msgr2=0x5575a7dd56b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:23.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.347+0000 7f1455613740 1 --2- 172.21.15.33:0/2957655758 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575a7dd52d0 0x5575a7dd56b0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f1444005e30 tx=0x7f14440326d0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.347+0000 7f1455613740 1 -- 172.21.15.33:0/2957655758 shutdown_connections 2024-03-19T03:01:23.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.347+0000 7f1455613740 1 --2- 172.21.15.33:0/2957655758 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575a7dd5bf0 0x5575a7dde290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.347+0000 7f1455613740 1 --2- 172.21.15.33:0/2957655758 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575a7dde7d0 0x5575a7de0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.347+0000 7f1455613740 1 --2- 172.21.15.33:0/2957655758 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575a7dd52d0 0x5575a7dd56b0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.347+0000 7f1455613740 1 -- 172.21.15.33:0/2957655758 >> 172.21.15.33:0/2957655758 conn(0x5575a7dcfa00 msgr2=0x5575a7dcfde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:23.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.347+0000 7f1455613740 1 -- 172.21.15.33:0/2957655758 shutdown_connections 2024-03-19T03:01:23.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.347+0000 7f1455613740 1 -- 172.21.15.33:0/2957655758 wait complete. 2024-03-19T03:01:23.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.347+0000 7f1455613740 1 -- start start 2024-03-19T03:01:23.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.347+0000 7f1455613740 1 Processor -- start 2024-03-19T03:01:23.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.348+0000 7f1455613740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575a7dd52d0 0x5575a7ddb890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.348+0000 7f1455613740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575a7dd5bf0 0x5575a7ddbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.348+0000 7f1455613740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575a7dde7d0 0x5575a7ddc310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.348+0000 7f1455613740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5575a7de1560 con 0x5575a7dd5bf0 2024-03-19T03:01:23.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.348+0000 7f1455613740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5575a7cf4310 con 0x5575a7dd52d0 2024-03-19T03:01:23.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.348+0000 7f1455613740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5575a7de16e0 con 0x5575a7dde7d0 2024-03-19T03:01:23.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.348+0000 7f1455613740 10 client.0 _ll_register_callbacks cb 0x5575a7ce82d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:01:23.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.348+0000 7f144b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575a7dd5bf0 0x5575a7ddbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.348+0000 7f144bfff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575a7dd52d0 0x5575a7ddb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.348+0000 7f144b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575a7dd5bf0 0x5575a7ddbdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58868/0 (socket says 172.21.15.33:58868) 2024-03-19T03:01:23.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.348+0000 7f144b7fe640 1 -- 172.21.15.33:0/2737582884 learned_addr learned my addr 172.21.15.33:0/2737582884 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:23.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.348+0000 7f1450ef6640 1 --2- 172.21.15.33:0/2737582884 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575a7dde7d0 0x5575a7ddc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.348+0000 7f144b7fe640 1 -- 172.21.15.33:0/2737582884 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575a7dde7d0 msgr2=0x5575a7ddc310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:23.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.348+0000 7f144b7fe640 1 --2- 172.21.15.33:0/2737582884 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575a7dde7d0 0x5575a7ddc310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.348+0000 7f144b7fe640 1 -- 172.21.15.33:0/2737582884 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575a7dd52d0 msgr2=0x5575a7ddb890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:23.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.348+0000 7f144b7fe640 1 --2- 172.21.15.33:0/2737582884 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575a7dd52d0 0x5575a7ddb890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.348+0000 7f144b7fe640 1 -- 172.21.15.33:0/2737582884 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575a7a4c3f0 con 0x5575a7dd5bf0 2024-03-19T03:01:23.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.348+0000 7f144bfff640 1 --2- 172.21.15.33:0/2737582884 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575a7dd52d0 0x5575a7ddb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:23.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.348+0000 7f144b7fe640 1 --2- 172.21.15.33:0/2737582884 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575a7dd5bf0 0x5575a7ddbdd0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f1440013710 tx=0x7f1440009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f19061c8b5d5f68 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:23.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.348+0000 7f142b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:01:23.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.348+0000 7f142b7fe640 1 -- 172.21.15.33:0/2737582884 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1440005020 con 0x5575a7dd5bf0 2024-03-19T03:01:23.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.348+0000 7f142b7fe640 1 -- 172.21.15.33:0/2737582884 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f14400051c0 con 0x5575a7dd5bf0 2024-03-19T03:01:23.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.348+0000 7f1455613740 1 -- 172.21.15.33:0/2737582884 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5575a7d963e0 con 0x5575a7dd5bf0 2024-03-19T03:01:23.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.349+0000 7f142b7fe640 1 -- 172.21.15.33:0/2737582884 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1440026050 con 0x5575a7dd5bf0 2024-03-19T03:01:23.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.349+0000 7f1455613740 10 client.17364 fetch_fsmap learned FSMap version 25 2024-03-19T03:01:23.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.349+0000 7f1455613740 1 -- 172.21.15.33:0/2737582884 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5575a7b167d0 con 0x5575a7dd5bf0 2024-03-19T03:01:23.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.349+0000 7f142b7fe640 1 -- 172.21.15.33:0/2737582884 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5575a7d963e0 con 0x5575a7dd5bf0 2024-03-19T03:01:23.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.349+0000 7f142b7fe640 1 -- 172.21.15.33:0/2737582884 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f144001a030 con 0x5575a7dd5bf0 2024-03-19T03:01:23.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.349+0000 7f1455613740 10 client.17364 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:01:23.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.349+0000 7f1455613740 10 client.17364 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:01:23.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.349+0000 7f1455613740 20 client.17364 populate_metadata read hostname 'smithi033' 2024-03-19T03:01:23.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.349+0000 7f1455613740 1 --2- 172.21.15.33:0/2737582884 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5575a7deebe0 0x5575a7df1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.349+0000 7f1455613740 4 client.17364 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:01:23.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.349+0000 7f1455613740 1 -- 172.21.15.33:0/2737582884 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5575a7d459f0 con 0x5575a7deebe0 2024-03-19T03:01:23.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.349+0000 7f144bfff640 1 --2- 172.21.15.33:0/2737582884 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5575a7deebe0 0x5575a7df1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.350+0000 7f144bfff640 1 --2- 172.21.15.33:0/2737582884 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5575a7deebe0 0x5575a7df1010 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84c70c2cbcc69e1a server_cookie=6807a8df7da2e20f in_seq=0 out_seq=0 2024-03-19T03:01:23.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.350+0000 7f142b7fe640 10 client.17364 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:01:23.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.350+0000 7f142b7fe640 1 -- 172.21.15.33:0/2737582884 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5575a7d459f0 con 0x5575a7deebe0 2024-03-19T03:01:23.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.350+0000 7f142b7fe640 10 client.17364 handle_command_reply: tid=0 2024-03-19T03:01:23.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.350+0000 7f1455613740 1 client.17364 shutdown 2024-03-19T03:01:23.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.350+0000 7f1455613740 1 -- 172.21.15.33:0/2737582884 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575a7dd5bf0 msgr2=0x5575a7ddbdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:23.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.350+0000 7f1455613740 1 --2- 172.21.15.33:0/2737582884 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575a7dd5bf0 0x5575a7ddbdd0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f1440013710 tx=0x7f1440009f90 comp rx=0 tx=0).stop 2024-03-19T03:01:23.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.350+0000 7f1455613740 1 -- 172.21.15.33:0/2737582884 shutdown_connections 2024-03-19T03:01:23.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.350+0000 7f1455613740 1 --2- 172.21.15.33:0/2737582884 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5575a7deebe0 0x5575a7df1010 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.350+0000 7f1455613740 1 --2- 172.21.15.33:0/2737582884 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575a7dde7d0 0x5575a7ddc310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.350+0000 7f1455613740 1 --2- 172.21.15.33:0/2737582884 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575a7dd5bf0 0x5575a7ddbdd0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.350+0000 7f1455613740 1 --2- 172.21.15.33:0/2737582884 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575a7dd52d0 0x5575a7ddb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.350+0000 7f1455613740 1 -- 172.21.15.33:0/2737582884 >> 172.21.15.33:0/2737582884 conn(0x5575a7dcfa00 msgr2=0x5575a7dd1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:23.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.350+0000 7f142b7fe640 0 client.17364 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:01:23.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.350+0000 7f1455613740 1 -- 172.21.15.33:0/2737582884 shutdown_connections 2024-03-19T03:01:23.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.350+0000 7f1455613740 1 -- 172.21.15.33:0/2737582884 wait complete. 2024-03-19T03:01:23.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.351+0000 7f1455613740 20 client.17364 trim_cache size 0 max 16384 2024-03-19T03:01:23.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.364+0000 7f1455613740 1 Processor -- start 2024-03-19T03:01:23.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.364+0000 7f1455613740 1 -- start start 2024-03-19T03:01:23.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.364+0000 7f1455613740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575a7dd52d0 0x5575a7dddfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.364+0000 7f1455613740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575a7dd5bf0 0x5575a7dea300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.364+0000 7f1455613740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575a7dde7d0 0x5575a7df68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.364+0000 7f1455613740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f144c1b15f0 con 0x5575a7dd52d0 2024-03-19T03:01:23.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.364+0000 7f1455613740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f144c1aeec0 con 0x5575a7dde7d0 2024-03-19T03:01:23.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.364+0000 7f1455613740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f144c1acd30 con 0x5575a7dd5bf0 2024-03-19T03:01:23.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.364+0000 7f144bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575a7dd52d0 0x5575a7dddfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.364+0000 7f144bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575a7dd52d0 0x5575a7dddfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58872/0 (socket says 172.21.15.33:58872) 2024-03-19T03:01:23.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.364+0000 7f144bfff640 1 -- 172.21.15.33:0/71220033 learned_addr learned my addr 172.21.15.33:0/71220033 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:23.371 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.364+0000 7f144b7fe640 1 --2- 172.21.15.33:0/71220033 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575a7dd5bf0 0x5575a7dea300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.364+0000 7f1450ef6640 1 --2- 172.21.15.33:0/71220033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575a7dde7d0 0x5575a7df68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.364+0000 7f144bfff640 1 -- 172.21.15.33:0/71220033 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575a7dd5bf0 msgr2=0x5575a7dea300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:23.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.364+0000 7f144bfff640 1 --2- 172.21.15.33:0/71220033 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575a7dd5bf0 0x5575a7dea300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.364+0000 7f144bfff640 1 -- 172.21.15.33:0/71220033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575a7dde7d0 msgr2=0x5575a7df68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:23.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.364+0000 7f144bfff640 1 --2- 172.21.15.33:0/71220033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575a7dde7d0 0x5575a7df68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.364+0000 7f144bfff640 1 -- 172.21.15.33:0/71220033 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575a7dd0340 con 0x5575a7dd52d0 2024-03-19T03:01:23.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.364+0000 7f1450ef6640 1 --2- 172.21.15.33:0/71220033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575a7dde7d0 0x5575a7df68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:01:23.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.364+0000 7f144b7fe640 1 --2- 172.21.15.33:0/71220033 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575a7dd5bf0 0x5575a7dea300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:01:23.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.364+0000 7f144bfff640 1 --2- 172.21.15.33:0/71220033 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575a7dd52d0 0x5575a7dddfc0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f143c0cc900 tx=0x7f143c086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62150c6e9f73d04b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:23.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.364+0000 7f1431ffb640 1 -- 172.21.15.33:0/71220033 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143c09b0c0 con 0x5575a7dd52d0 2024-03-19T03:01:23.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.364+0000 7f1431ffb640 1 -- 172.21.15.33:0/71220033 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f143c003b30 con 0x5575a7dd52d0 2024-03-19T03:01:23.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.364+0000 7f1431ffb640 1 -- 172.21.15.33:0/71220033 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143c099a60 con 0x5575a7dd52d0 2024-03-19T03:01:23.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.365+0000 7f1455613740 1 -- 172.21.15.33:0/71220033 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575a7dd52d0 msgr2=0x5575a7dddfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:23.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.365+0000 7f1455613740 1 --2- 172.21.15.33:0/71220033 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575a7dd52d0 0x5575a7dddfc0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f143c0cc900 tx=0x7f143c086040 comp rx=0 tx=0).stop 2024-03-19T03:01:23.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.365+0000 7f1455613740 1 -- 172.21.15.33:0/71220033 shutdown_connections 2024-03-19T03:01:23.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.365+0000 7f1455613740 1 --2- 172.21.15.33:0/71220033 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575a7dd5bf0 0x5575a7dea300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.365+0000 7f1455613740 1 --2- 172.21.15.33:0/71220033 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575a7dde7d0 0x5575a7df68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.365+0000 7f1455613740 1 --2- 172.21.15.33:0/71220033 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575a7dd52d0 0x5575a7dddfc0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.365+0000 7f1455613740 1 -- 172.21.15.33:0/71220033 >> 172.21.15.33:0/71220033 conn(0x5575a7dcfa00 msgr2=0x5575a7dd6c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:23.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.365+0000 7f1455613740 1 -- 172.21.15.33:0/71220033 shutdown_connections 2024-03-19T03:01:23.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.365+0000 7f1455613740 1 -- 172.21.15.33:0/71220033 wait complete. 2024-03-19T03:01:23.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.365+0000 7f1455613740 1 -- start start 2024-03-19T03:01:23.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.366+0000 7f1455613740 1 Processor -- start 2024-03-19T03:01:23.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.366+0000 7f1455613740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575a7dd52d0 0x5575a7dd83b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.366+0000 7f1455613740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575a7dd5bf0 0x5575a7dd88f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.366+0000 7f1455613740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575a7dde7d0 0x5575a7dd8e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.366+0000 7f1455613740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f144c1ad380 con 0x5575a7dd5bf0 2024-03-19T03:01:23.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.366+0000 7f1455613740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f144c1ae7f0 con 0x5575a7dd52d0 2024-03-19T03:01:23.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.366+0000 7f1455613740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f144c1ada50 con 0x5575a7dde7d0 2024-03-19T03:01:23.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.366+0000 7f1455613740 10 client.0 _ll_register_callbacks cb 0x7f1440009840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:01:23.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.366+0000 7f144b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575a7dd5bf0 0x5575a7dd88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.366+0000 7f144b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575a7dd5bf0 0x5575a7dd88f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58874/0 (socket says 172.21.15.33:58874) 2024-03-19T03:01:23.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.366+0000 7f144b7fe640 1 -- 172.21.15.33:0/341099244 learned_addr learned my addr 172.21.15.33:0/341099244 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:23.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.366+0000 7f144bfff640 1 --2- 172.21.15.33:0/341099244 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575a7dd52d0 0x5575a7dd83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.366+0000 7f1450ef6640 1 --2- 172.21.15.33:0/341099244 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575a7dde7d0 0x5575a7dd8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.366+0000 7f144b7fe640 1 -- 172.21.15.33:0/341099244 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575a7dde7d0 msgr2=0x5575a7dd8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:23.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.366+0000 7f144b7fe640 1 --2- 172.21.15.33:0/341099244 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575a7dde7d0 0x5575a7dd8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.366+0000 7f144b7fe640 1 -- 172.21.15.33:0/341099244 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575a7dd52d0 msgr2=0x5575a7dd83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:23.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.366+0000 7f144b7fe640 1 --2- 172.21.15.33:0/341099244 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575a7dd52d0 0x5575a7dd83b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.366+0000 7f144b7fe640 1 -- 172.21.15.33:0/341099244 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575a7e53f10 con 0x5575a7dd5bf0 2024-03-19T03:01:23.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.366+0000 7f144b7fe640 1 --2- 172.21.15.33:0/341099244 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575a7dd5bf0 0x5575a7dd88f0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x5575a7ddce60 tx=0x7f1440011950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6d16cdd83429fd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:23.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.366+0000 7f142b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:01:23.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.366+0000 7f142b7fe640 1 -- 172.21.15.33:0/341099244 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f144001b8b0 con 0x5575a7dd5bf0 2024-03-19T03:01:23.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.367+0000 7f142b7fe640 1 -- 172.21.15.33:0/341099244 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f144001ba80 con 0x5575a7dd5bf0 2024-03-19T03:01:23.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.367+0000 7f1455613740 1 -- 172.21.15.33:0/341099244 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5575a7dd67a0 con 0x5575a7dd5bf0 2024-03-19T03:01:23.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.367+0000 7f142b7fe640 1 -- 172.21.15.33:0/341099244 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f144003b050 con 0x5575a7dd5bf0 2024-03-19T03:01:23.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.367+0000 7f1455613740 10 client.17370 fetch_fsmap learned FSMap version 25 2024-03-19T03:01:23.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.367+0000 7f1455613740 1 -- 172.21.15.33:0/341099244 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5575a7e551e0 con 0x5575a7dd5bf0 2024-03-19T03:01:23.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.367+0000 7f142b7fe640 1 -- 172.21.15.33:0/341099244 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5575a7dd67a0 con 0x5575a7dd5bf0 2024-03-19T03:01:23.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.367+0000 7f142b7fe640 1 -- 172.21.15.33:0/341099244 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f144001a040 con 0x5575a7dd5bf0 2024-03-19T03:01:23.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.367+0000 7f1455613740 10 client.17370 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:01:23.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.367+0000 7f1455613740 10 client.17370 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:01:23.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.367+0000 7f1455613740 20 client.17370 populate_metadata read hostname 'smithi033' 2024-03-19T03:01:23.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.367+0000 7f1455613740 1 --2- 172.21.15.33:0/341099244 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5575a7e55980 0x5575a7e57da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:23.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.367+0000 7f1455613740 4 client.17370 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T03:01:23.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.367+0000 7f1455613740 1 -- 172.21.15.33:0/341099244 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5575a7d3fd40 con 0x5575a7e55980 2024-03-19T03:01:23.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.367+0000 7f144bfff640 1 --2- 172.21.15.33:0/341099244 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5575a7e55980 0x5575a7e57da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:23.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.368+0000 7f144bfff640 1 --2- 172.21.15.33:0/341099244 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5575a7e55980 0x5575a7e57da0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67fcf6c781691f14 server_cookie=1c1f05a980ddbdfe in_seq=0 out_seq=0 2024-03-19T03:01:23.376 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.368+0000 7f142b7fe640 10 client.17370 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:01:23.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.368+0000 7f142b7fe640 1 -- 172.21.15.33:0/341099244 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5575a7d3fd40 con 0x5575a7e55980 2024-03-19T03:01:23.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.368+0000 7f142b7fe640 10 client.17370 handle_command_reply: tid=0 2024-03-19T03:01:23.377 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.368+0000 7f1455613740 1 client.17370 shutdown 2024-03-19T03:01:23.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.368+0000 7f1455613740 1 -- 172.21.15.33:0/341099244 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575a7dd5bf0 msgr2=0x5575a7dd88f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:23.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.368+0000 7f1455613740 1 --2- 172.21.15.33:0/341099244 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575a7dd5bf0 0x5575a7dd88f0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x5575a7ddce60 tx=0x7f1440011950 comp rx=0 tx=0).stop 2024-03-19T03:01:23.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.368+0000 7f1455613740 1 -- 172.21.15.33:0/341099244 shutdown_connections 2024-03-19T03:01:23.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.368+0000 7f1455613740 1 --2- 172.21.15.33:0/341099244 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5575a7e55980 0x5575a7e57da0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.368+0000 7f1455613740 1 --2- 172.21.15.33:0/341099244 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5575a7dde7d0 0x5575a7dd8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.368+0000 7f142b7fe640 0 client.17370 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:01:23.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.368+0000 7f1455613740 1 --2- 172.21.15.33:0/341099244 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5575a7dd5bf0 0x5575a7dd88f0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.368+0000 7f1455613740 1 --2- 172.21.15.33:0/341099244 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5575a7dd52d0 0x5575a7dd83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.368+0000 7f1455613740 1 -- 172.21.15.33:0/341099244 >> 172.21.15.33:0/341099244 conn(0x5575a7dcfa00 msgr2=0x5575a7ddb850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:23.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.368+0000 7f1455613740 1 -- 172.21.15.33:0/341099244 shutdown_connections 2024-03-19T03:01:23.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.368+0000 7f1455613740 1 -- 172.21.15.33:0/341099244 wait complete. 2024-03-19T03:01:23.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.368+0000 7f1455613740 20 client.17370 trim_cache size 0 max 16384 2024-03-19T03:01:23.379 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T03:01:23.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.368+0000 7f1452959640 1 -- 172.21.15.33:0/1769703172 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f14340a2970 msgr2=0x7f14340a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:23.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.368+0000 7f1452959640 1 --2- 172.21.15.33:0/1769703172 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f14340a2970 0x7f14340a4e30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1444009ca0 tx=0x7f1444042000 comp rx=0 tx=0).stop 2024-03-19T03:01:23.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.368+0000 7f1452959640 1 -- 172.21.15.33:0/1769703172 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f144c1006a0 msgr2=0x7f144c197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:23.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.368+0000 7f1452959640 1 --2- 172.21.15.33:0/1769703172 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f144c1006a0 0x7f144c197f80 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f143c00ea60 tx=0x7f143c00ef20 comp rx=0 tx=0).stop 2024-03-19T03:01:23.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.369+0000 7f1452959640 1 -- 172.21.15.33:0/1769703172 shutdown_connections 2024-03-19T03:01:23.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.369+0000 7f1452959640 1 --2- 172.21.15.33:0/1769703172 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f14340a2970 0x7f14340a4e30 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.369+0000 7f1452959640 1 --2- 172.21.15.33:0/1769703172 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f144c101060 0x7f144c1984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.369+0000 7f1452959640 1 --2- 172.21.15.33:0/1769703172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f144c1066b0 0x7f144c19c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.369+0000 7f1452959640 1 --2- 172.21.15.33:0/1769703172 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f144c1006a0 0x7f144c197f80 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:23.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.369+0000 7f1452959640 1 -- 172.21.15.33:0/1769703172 >> 172.21.15.33:0/1769703172 conn(0x7f144c0778d0 msgr2=0x7f144c068c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:23.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.369+0000 7f1452959640 1 -- 172.21.15.33:0/1769703172 shutdown_connections 2024-03-19T03:01:23.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:23.369+0000 7f1452959640 1 -- 172.21.15.33:0/1769703172 wait complete. 2024-03-19T03:01:23.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:23 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3393212323' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:01:23.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:23 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3393212323' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:01:23.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:23 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3393212323' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:01:24.382 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T03:01:24.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:01:24.383 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T03:01:24.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.502+0000 7fa348836640 1 -- 172.21.15.33:0/3459807783 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3441006d0 msgr2=0x7fa344100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:24.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.502+0000 7fa348836640 1 --2- 172.21.15.33:0/3459807783 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3441006d0 0x7fa344100b50 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fa338009930 tx=0x7fa33802bc30 comp rx=0 tx=0).stop 2024-03-19T03:01:24.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.502+0000 7fa348836640 1 -- 172.21.15.33:0/3459807783 shutdown_connections 2024-03-19T03:01:24.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.502+0000 7fa348836640 1 --2- 172.21.15.33:0/3459807783 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa344101090 0x7fa34410d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.502+0000 7fa348836640 1 --2- 172.21.15.33:0/3459807783 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3441006d0 0x7fa344100b50 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.502+0000 7fa348836640 1 --2- 172.21.15.33:0/3459807783 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3441066e0 0x7fa344106ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.502+0000 7fa348836640 1 -- 172.21.15.33:0/3459807783 >> 172.21.15.33:0/3459807783 conn(0x7fa344076aa0 msgr2=0x7fa344076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:24.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.502+0000 7fa348836640 1 -- 172.21.15.33:0/3459807783 shutdown_connections 2024-03-19T03:01:24.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.502+0000 7fa348836640 1 -- 172.21.15.33:0/3459807783 wait complete. 2024-03-19T03:01:24.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.503+0000 7fa348836640 1 Processor -- start 2024-03-19T03:01:24.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.503+0000 7fa348836640 1 -- start start 2024-03-19T03:01:24.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.503+0000 7fa348836640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3441006d0 0x7fa344196f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.503+0000 7fa348836640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa344101090 0x7fa344197480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.503+0000 7fa348836640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3441066e0 0x7fa34419b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.503+0000 7fa348836640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa3441109b0 con 0x7fa3441066e0 2024-03-19T03:01:24.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.503+0000 7fa348836640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa344110830 con 0x7fa344101090 2024-03-19T03:01:24.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.503+0000 7fa348836640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa344110b30 con 0x7fa3441006d0 2024-03-19T03:01:24.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.503+0000 7fa342d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3441066e0 0x7fa34419b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.503+0000 7fa342d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3441066e0 0x7fa34419b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58632/0 (socket says 172.21.15.33:58632) 2024-03-19T03:01:24.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.503+0000 7fa342d9d640 1 -- 172.21.15.33:0/3401994883 learned_addr learned my addr 172.21.15.33:0/3401994883 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:24.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.503+0000 7fa341d9b640 1 --2- 172.21.15.33:0/3401994883 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa344101090 0x7fa344197480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.503+0000 7fa34259c640 1 --2- 172.21.15.33:0/3401994883 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3441006d0 0x7fa344196f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.503+0000 7fa342d9d640 1 -- 172.21.15.33:0/3401994883 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3441006d0 msgr2=0x7fa344196f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:24.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.504+0000 7fa342d9d640 1 --2- 172.21.15.33:0/3401994883 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3441006d0 0x7fa344196f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.504+0000 7fa342d9d640 1 -- 172.21.15.33:0/3401994883 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa344101090 msgr2=0x7fa344197480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:24.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.504+0000 7fa342d9d640 1 --2- 172.21.15.33:0/3401994883 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa344101090 0x7fa344197480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.504+0000 7fa342d9d640 1 -- 172.21.15.33:0/3401994883 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa34419bf30 con 0x7fa3441066e0 2024-03-19T03:01:24.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.504+0000 7fa341d9b640 1 --2- 172.21.15.33:0/3401994883 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa344101090 0x7fa344197480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:24.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.504+0000 7fa342d9d640 1 --2- 172.21.15.33:0/3401994883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3441066e0 0x7fa34419b800 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fa33c00d950 tx=0x7fa33c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:24.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.504+0000 7fa3337fe640 1 -- 172.21.15.33:0/3401994883 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa33c017030 con 0x7fa3441066e0 2024-03-19T03:01:24.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.504+0000 7fa348836640 1 -- 172.21.15.33:0/3401994883 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa34419c220 con 0x7fa3441066e0 2024-03-19T03:01:24.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.504+0000 7fa3337fe640 1 -- 172.21.15.33:0/3401994883 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa33c004020 con 0x7fa3441066e0 2024-03-19T03:01:24.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.504+0000 7fa3337fe640 1 -- 172.21.15.33:0/3401994883 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa33c003b20 con 0x7fa3441066e0 2024-03-19T03:01:24.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.504+0000 7fa348836640 1 -- 172.21.15.33:0/3401994883 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3441a3ab0 con 0x7fa3441066e0 2024-03-19T03:01:24.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.506+0000 7fa3337fe640 1 -- 172.21.15.33:0/3401994883 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fa33c01a070 con 0x7fa3441066e0 2024-03-19T03:01:24.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.506+0000 7fa34b423740 1 Processor -- start 2024-03-19T03:01:24.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.507+0000 7fa34b423740 1 -- start start 2024-03-19T03:01:24.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.507+0000 7fa3337fe640 1 --2- 172.21.15.33:0/3401994883 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa32c0a2970 0x7fa32c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.507+0000 7fa34b423740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b663eb2d80 0x55b663eb3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.507+0000 7fa34b423740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b663eb3730 0x55b663ebe550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.507+0000 7fa34b423740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b663ebea90 0x55b663ec0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.507+0000 7fa34b423740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b663e17160 con 0x55b663eb2d80 2024-03-19T03:01:24.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.507+0000 7fa34b423740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55b663dbfe10 con 0x55b663ebea90 2024-03-19T03:01:24.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.507+0000 7fa34b423740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b663e13200 con 0x55b663eb3730 2024-03-19T03:01:24.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.507+0000 7fa3337fe640 1 -- 172.21.15.33:0/3401994883 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7fa33c0ce9c0 con 0x7fa3441066e0 2024-03-19T03:01:24.515 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.507+0000 7fa341d9b640 1 --2- 172.21.15.33:0/3401994883 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa32c0a2970 0x7fa32c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.507+0000 7fa34259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b663eb2d80 0x55b663eb3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.507+0000 7fa34259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b663eb2d80 0x55b663eb3160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58634/0 (socket says 172.21.15.33:58634) 2024-03-19T03:01:24.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.507+0000 7fa34259c640 1 -- 172.21.15.33:0/3785348011 learned_addr learned my addr 172.21.15.33:0/3785348011 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:24.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.507+0000 7fa341d9b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b663eb3730 0x55b663ebe550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.507+0000 7fa342d9d640 1 --2- 172.21.15.33:0/3785348011 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b663ebea90 0x55b663ec0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.507+0000 7fa342d9d640 1 -- 172.21.15.33:0/3785348011 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b663eb3730 msgr2=0x55b663ebe550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:24.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.507+0000 7fa342d9d640 1 --2- 172.21.15.33:0/3785348011 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b663eb3730 0x55b663ebe550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.507+0000 7fa342d9d640 1 -- 172.21.15.33:0/3785348011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b663eb2d80 msgr2=0x55b663eb3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:24.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.507+0000 7fa342d9d640 1 --2- 172.21.15.33:0/3785348011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b663eb2d80 0x55b663eb3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.507+0000 7fa342d9d640 1 -- 172.21.15.33:0/3785348011 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b663bf37e0 con 0x55b663ebea90 2024-03-19T03:01:24.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.508+0000 7fa341d9b640 1 --2- 172.21.15.33:0/3401994883 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa32c0a2970 0x7fa32c0a4e30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa338009900 tx=0x7fa338009580 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:24.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.508+0000 7fa342d9d640 1 --2- 172.21.15.33:0/3785348011 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b663ebea90 0x55b663ec0e80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa33c0cbd40 tx=0x7fa33c09a670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23a0cdeb15db6ec6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:24.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.508+0000 7fa330ff9640 1 -- 172.21.15.33:0/3785348011 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa33c05ebe0 con 0x55b663ebea90 2024-03-19T03:01:24.516 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.508+0000 7fa330ff9640 1 -- 172.21.15.33:0/3785348011 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa33c016800 con 0x55b663ebea90 2024-03-19T03:01:24.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.508+0000 7fa330ff9640 1 -- 172.21.15.33:0/3785348011 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa33c0cbfc0 con 0x55b663ebea90 2024-03-19T03:01:24.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.509+0000 7fa34b423740 1 -- 172.21.15.33:0/3785348011 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b663ebea90 msgr2=0x55b663ec0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:24.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.509+0000 7fa34b423740 1 --2- 172.21.15.33:0/3785348011 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b663ebea90 0x55b663ec0e80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa33c0cbd40 tx=0x7fa33c09a670 comp rx=0 tx=0).stop 2024-03-19T03:01:24.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.509+0000 7fa34b423740 1 -- 172.21.15.33:0/3785348011 shutdown_connections 2024-03-19T03:01:24.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.509+0000 7fa34b423740 1 --2- 172.21.15.33:0/3785348011 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b663eb3730 0x55b663ebe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.509+0000 7fa34b423740 1 --2- 172.21.15.33:0/3785348011 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b663ebea90 0x55b663ec0e80 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.509+0000 7fa34b423740 1 --2- 172.21.15.33:0/3785348011 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b663eb2d80 0x55b663eb3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.509+0000 7fa34b423740 1 -- 172.21.15.33:0/3785348011 >> 172.21.15.33:0/3785348011 conn(0x55b663eadab0 msgr2=0x55b663eade90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:24.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.509+0000 7fa34b423740 1 -- 172.21.15.33:0/3785348011 shutdown_connections 2024-03-19T03:01:24.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.509+0000 7fa34b423740 1 -- 172.21.15.33:0/3785348011 wait complete. 2024-03-19T03:01:24.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.509+0000 7fa34b423740 1 -- start start 2024-03-19T03:01:24.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.509+0000 7fa34b423740 1 Processor -- start 2024-03-19T03:01:24.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.509+0000 7fa34b423740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b663eb2d80 0x55b663ebb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.509+0000 7fa34b423740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b663eb3730 0x55b663ebbec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.509+0000 7fa34b423740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b663ebea90 0x55b663ebc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.517 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.509+0000 7fa34b423740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55b663ec1880 con 0x55b663eb3730 2024-03-19T03:01:24.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.509+0000 7fa34b423740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55b663dd31a0 con 0x55b663ebea90 2024-03-19T03:01:24.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.509+0000 7fa34b423740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55b663ec1a00 con 0x55b663eb2d80 2024-03-19T03:01:24.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.509+0000 7fa34b423740 10 client.0 _ll_register_callbacks cb 0x55b663b2ac60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:01:24.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.509+0000 7fa341d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b663eb3730 0x55b663ebbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.510+0000 7fa342d9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b663ebea90 0x55b663ebc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.510+0000 7fa341d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b663eb3730 0x55b663ebbec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58650/0 (socket says 172.21.15.33:58650) 2024-03-19T03:01:24.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.510+0000 7fa341d9b640 1 -- 172.21.15.33:0/965426825 learned_addr learned my addr 172.21.15.33:0/965426825 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:24.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.510+0000 7fa342d9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b663ebea90 0x55b663ebc400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:34520/0 (socket says 172.21.15.33:34520) 2024-03-19T03:01:24.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.510+0000 7fa34259c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b663eb2d80 0x55b663ebb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.510+0000 7fa342d9d640 1 -- 172.21.15.33:0/965426825 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b663eb2d80 msgr2=0x55b663ebb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:24.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.510+0000 7fa342d9d640 1 --2- 172.21.15.33:0/965426825 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b663eb2d80 0x55b663ebb980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.510+0000 7fa342d9d640 1 -- 172.21.15.33:0/965426825 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b663eb3730 msgr2=0x55b663ebbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:24.518 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.510+0000 7fa342d9d640 1 --2- 172.21.15.33:0/965426825 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b663eb3730 0x55b663ebbec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.510+0000 7fa342d9d640 1 -- 172.21.15.33:0/965426825 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b663c53f70 con 0x55b663ebea90 2024-03-19T03:01:24.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.510+0000 7fa341d9b640 1 --2- 172.21.15.33:0/965426825 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b663eb3730 0x55b663ebbec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:24.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.510+0000 7fa342d9d640 1 --2- 172.21.15.33:0/965426825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b663ebea90 0x55b663ebc400 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa33c098f00 tx=0x7fa33c09b490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=551566f88be24b34 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:24.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.510+0000 7fa329ffb640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T03:01:24.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.510+0000 7fa329ffb640 1 -- 172.21.15.33:0/965426825 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa33c05ebe0 con 0x55b663ebea90 2024-03-19T03:01:24.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.510+0000 7fa34b423740 1 -- 172.21.15.33:0/965426825 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa33c016800 con 0x55b663ebea90 2024-03-19T03:01:24.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.510+0000 7fa329ffb640 1 -- 172.21.15.33:0/965426825 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7fa33c0041d0 con 0x55b663ebea90 2024-03-19T03:01:24.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.510+0000 7fa329ffb640 1 -- 172.21.15.33:0/965426825 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa33c09ac50 con 0x55b663ebea90 2024-03-19T03:01:24.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.510+0000 7fa34b423740 10 client.27203 fetch_fsmap learned FSMap version 25 2024-03-19T03:01:24.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.510+0000 7fa34b423740 1 -- 172.21.15.33:0/965426825 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b663aeb170 con 0x55b663ebea90 2024-03-19T03:01:24.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.511+0000 7fa329ffb640 1 -- 172.21.15.33:0/965426825 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa33c016800 con 0x55b663ebea90 2024-03-19T03:01:24.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.511+0000 7fa329ffb640 1 -- 172.21.15.33:0/965426825 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fa33c0b9070 con 0x55b663ebea90 2024-03-19T03:01:24.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.511+0000 7fa34b423740 10 client.27203 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:01:24.519 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.511+0000 7fa34b423740 10 client.27203 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:01:24.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.511+0000 7fa34b423740 20 client.27203 populate_metadata read hostname 'smithi033' 2024-03-19T03:01:24.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.511+0000 7fa34b423740 1 --2- 172.21.15.33:0/965426825 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55b663eceb90 0x55b663ed0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.511+0000 7fa34b423740 4 client.27203 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:01:24.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.511+0000 7fa34b423740 1 -- 172.21.15.33:0/965426825 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b663e246a0 con 0x55b663eceb90 2024-03-19T03:01:24.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.511+0000 7fa34259c640 1 --2- 172.21.15.33:0/965426825 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55b663eceb90 0x55b663ed0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.511+0000 7fa34259c640 1 --2- 172.21.15.33:0/965426825 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55b663eceb90 0x55b663ed0f80 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82a81e904c3ed730 server_cookie=8fdfa3a88766e754 in_seq=0 out_seq=0 2024-03-19T03:01:24.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.511+0000 7fa329ffb640 10 client.27203 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:01:24.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.512+0000 7fa329ffb640 1 -- 172.21.15.33:0/965426825 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b663e246a0 con 0x55b663eceb90 2024-03-19T03:01:24.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.512+0000 7fa329ffb640 10 client.27203 handle_command_reply: tid=0 2024-03-19T03:01:24.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.512+0000 7fa34b423740 1 client.27203 shutdown 2024-03-19T03:01:24.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.512+0000 7fa34b423740 1 -- 172.21.15.33:0/965426825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b663ebea90 msgr2=0x55b663ebc400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:24.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.512+0000 7fa34b423740 1 --2- 172.21.15.33:0/965426825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b663ebea90 0x55b663ebc400 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa33c098f00 tx=0x7fa33c09b490 comp rx=0 tx=0).stop 2024-03-19T03:01:24.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.512+0000 7fa34b423740 1 -- 172.21.15.33:0/965426825 shutdown_connections 2024-03-19T03:01:24.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.512+0000 7fa34b423740 1 --2- 172.21.15.33:0/965426825 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55b663eceb90 0x55b663ed0f80 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.520 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.512+0000 7fa34b423740 1 --2- 172.21.15.33:0/965426825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b663ebea90 0x55b663ebc400 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.512+0000 7fa329ffb640 0 client.27203 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:01:24.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.512+0000 7fa34b423740 1 --2- 172.21.15.33:0/965426825 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b663eb3730 0x55b663ebbec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.512+0000 7fa34b423740 1 --2- 172.21.15.33:0/965426825 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b663eb2d80 0x55b663ebb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.512+0000 7fa34b423740 1 -- 172.21.15.33:0/965426825 >> 172.21.15.33:0/965426825 conn(0x55b663eadab0 msgr2=0x55b663eaf800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:24.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.512+0000 7fa34b423740 1 -- 172.21.15.33:0/965426825 shutdown_connections 2024-03-19T03:01:24.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.512+0000 7fa34b423740 1 -- 172.21.15.33:0/965426825 wait complete. 2024-03-19T03:01:24.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.513+0000 7fa34b423740 20 client.27203 trim_cache size 0 max 16384 2024-03-19T03:01:24.521 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:24 smithi155 ceph-mon[35804]: pgmap v245: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-19T03:01:24.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:24 smithi033 ceph-mon[31973]: pgmap v245: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-19T03:01:24.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.530+0000 7fa34b423740 1 Processor -- start 2024-03-19T03:01:24.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.530+0000 7fa34b423740 1 -- start start 2024-03-19T03:01:24.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.530+0000 7fa34b423740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b663eb2d80 0x55b663ebe0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.530+0000 7fa34b423740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b663eb3730 0x55b663ed3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.530+0000 7fa34b423740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b663ebea90 0x55b663eb8c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.530+0000 7fa34b423740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa3441af250 con 0x55b663eb2d80 2024-03-19T03:01:24.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.530+0000 7fa34b423740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa3441acb20 con 0x55b663ebea90 2024-03-19T03:01:24.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.530+0000 7fa34b423740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa3441aa990 con 0x55b663eb3730 2024-03-19T03:01:24.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.530+0000 7fa34259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b663eb2d80 0x55b663ebe0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.530+0000 7fa34259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b663eb2d80 0x55b663ebe0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58666/0 (socket says 172.21.15.33:58666) 2024-03-19T03:01:24.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.530+0000 7fa34259c640 1 -- 172.21.15.33:0/3395984369 learned_addr learned my addr 172.21.15.33:0/3395984369 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:24.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.531+0000 7fa342d9d640 1 --2- 172.21.15.33:0/3395984369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b663ebea90 0x55b663eb8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.531+0000 7fa341d9b640 1 --2- 172.21.15.33:0/3395984369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b663eb3730 0x55b663ed3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.531+0000 7fa34259c640 1 -- 172.21.15.33:0/3395984369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b663eb3730 msgr2=0x55b663ed3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:24.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.531+0000 7fa34259c640 1 --2- 172.21.15.33:0/3395984369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b663eb3730 0x55b663ed3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.531+0000 7fa34259c640 1 -- 172.21.15.33:0/3395984369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b663ebea90 msgr2=0x55b663eb8c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:24.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.531+0000 7fa34259c640 1 --2- 172.21.15.33:0/3395984369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b663ebea90 0x55b663eb8c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.531+0000 7fa34259c640 1 -- 172.21.15.33:0/3395984369 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b663ead8e0 con 0x55b663eb2d80 2024-03-19T03:01:24.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.531+0000 7fa34259c640 1 --2- 172.21.15.33:0/3395984369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b663eb2d80 0x55b663ebe0b0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fa334005fd0 tx=0x7fa334009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7372441cd7e4460c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:24.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.531+0000 7fa32bfff640 1 -- 172.21.15.33:0/3395984369 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3340201b0 con 0x55b663eb2d80 2024-03-19T03:01:24.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.531+0000 7fa32bfff640 1 -- 172.21.15.33:0/3395984369 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa334026070 con 0x55b663eb2d80 2024-03-19T03:01:24.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.531+0000 7fa32bfff640 1 -- 172.21.15.33:0/3395984369 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa33401d3f0 con 0x55b663eb2d80 2024-03-19T03:01:24.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.532+0000 7fa34b423740 1 -- 172.21.15.33:0/3395984369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b663eb2d80 msgr2=0x55b663ebe0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:24.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.532+0000 7fa34b423740 1 --2- 172.21.15.33:0/3395984369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b663eb2d80 0x55b663ebe0b0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fa334005fd0 tx=0x7fa334009500 comp rx=0 tx=0).stop 2024-03-19T03:01:24.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.532+0000 7fa34b423740 1 -- 172.21.15.33:0/3395984369 shutdown_connections 2024-03-19T03:01:24.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.532+0000 7fa34b423740 1 --2- 172.21.15.33:0/3395984369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b663eb3730 0x55b663ed3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.532+0000 7fa34b423740 1 --2- 172.21.15.33:0/3395984369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b663ebea90 0x55b663eb8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.532+0000 7fa34b423740 1 --2- 172.21.15.33:0/3395984369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b663eb2d80 0x55b663ebe0b0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.532+0000 7fa34b423740 1 -- 172.21.15.33:0/3395984369 >> 172.21.15.33:0/3395984369 conn(0x55b663eadab0 msgr2=0x55b663eacb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:24.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.532+0000 7fa34b423740 1 -- 172.21.15.33:0/3395984369 shutdown_connections 2024-03-19T03:01:24.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.532+0000 7fa34b423740 1 -- 172.21.15.33:0/3395984369 wait complete. 2024-03-19T03:01:24.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.532+0000 7fa34b423740 1 -- start start 2024-03-19T03:01:24.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.532+0000 7fa34b423740 1 Processor -- start 2024-03-19T03:01:24.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.532+0000 7fa34b423740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b663eb2d80 0x55b663ed1f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.532+0000 7fa34b423740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b663eb3730 0x55b663ed24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.532+0000 7fa34b423740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b663ebea90 0x55b663f33b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.532+0000 7fa34b423740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa3441aafe0 con 0x55b663eb3730 2024-03-19T03:01:24.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.532+0000 7fa34b423740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa3441ac450 con 0x55b663eb2d80 2024-03-19T03:01:24.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.532+0000 7fa34b423740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa3441ab6b0 con 0x55b663ebea90 2024-03-19T03:01:24.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.532+0000 7fa34b423740 10 client.0 _ll_register_callbacks cb 0x7fa334009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:01:24.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.532+0000 7fa341d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b663eb3730 0x55b663ed24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.532+0000 7fa34259c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b663eb2d80 0x55b663ed1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.532+0000 7fa341d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b663eb3730 0x55b663ed24d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58680/0 (socket says 172.21.15.33:58680) 2024-03-19T03:01:24.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.532+0000 7fa341d9b640 1 -- 172.21.15.33:0/1517409958 learned_addr learned my addr 172.21.15.33:0/1517409958 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:24.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.533+0000 7fa342d9d640 1 --2- 172.21.15.33:0/1517409958 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b663ebea90 0x55b663f33b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.533+0000 7fa341d9b640 1 -- 172.21.15.33:0/1517409958 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b663ebea90 msgr2=0x55b663f33b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:24.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.533+0000 7fa341d9b640 1 --2- 172.21.15.33:0/1517409958 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b663ebea90 0x55b663f33b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.533+0000 7fa341d9b640 1 -- 172.21.15.33:0/1517409958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b663eb2d80 msgr2=0x55b663ed1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:24.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.533+0000 7fa341d9b640 1 --2- 172.21.15.33:0/1517409958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b663eb2d80 0x55b663ed1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.533+0000 7fa341d9b640 1 -- 172.21.15.33:0/1517409958 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b663f34070 con 0x55b663eb3730 2024-03-19T03:01:24.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.533+0000 7fa341d9b640 1 --2- 172.21.15.33:0/1517409958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b663eb3730 0x55b663ed24d0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fa33802c240 tx=0x7fa33802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6c063ae46a1e497 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:24.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.533+0000 7fa329ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:01:24.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.533+0000 7fa329ffb640 1 -- 172.21.15.33:0/1517409958 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa338042e10 con 0x55b663eb3730 2024-03-19T03:01:24.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.533+0000 7fa329ffb640 1 -- 172.21.15.33:0/1517409958 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7fa338058480 con 0x55b663eb3730 2024-03-19T03:01:24.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.533+0000 7fa34b423740 1 -- 172.21.15.33:0/1517409958 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa334026070 con 0x55b663eb3730 2024-03-19T03:01:24.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.533+0000 7fa329ffb640 1 -- 172.21.15.33:0/1517409958 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa338061050 con 0x55b663eb3730 2024-03-19T03:01:24.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.533+0000 7fa34b423740 10 client.17388 fetch_fsmap learned FSMap version 25 2024-03-19T03:01:24.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.533+0000 7fa34b423740 1 -- 172.21.15.33:0/1517409958 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b663f35350 con 0x55b663eb3730 2024-03-19T03:01:24.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.533+0000 7fa329ffb640 1 -- 172.21.15.33:0/1517409958 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa334026070 con 0x55b663eb3730 2024-03-19T03:01:24.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.534+0000 7fa329ffb640 1 -- 172.21.15.33:0/1517409958 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fa33804c040 con 0x55b663eb3730 2024-03-19T03:01:24.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.534+0000 7fa34b423740 10 client.17388 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:01:24.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.534+0000 7fa34b423740 10 client.17388 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:01:24.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.534+0000 7fa34b423740 20 client.17388 populate_metadata read hostname 'smithi033' 2024-03-19T03:01:24.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.534+0000 7fa34b423740 1 --2- 172.21.15.33:0/1517409958 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55b663f35aa0 0x55b663f37ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.534+0000 7fa34b423740 4 client.17388 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T03:01:24.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.534+0000 7fa34b423740 1 -- 172.21.15.33:0/1517409958 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b663e1e9f0 con 0x55b663f35aa0 2024-03-19T03:01:24.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.534+0000 7fa34259c640 1 --2- 172.21.15.33:0/1517409958 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55b663f35aa0 0x55b663f37ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.534+0000 7fa329ffb640 10 client.17388 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:01:24.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.534+0000 7fa34259c640 1 --2- 172.21.15.33:0/1517409958 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55b663f35aa0 0x55b663f37ec0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3518bf2f032404f server_cookie=23337b91051d9252 in_seq=0 out_seq=0 2024-03-19T03:01:24.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.535+0000 7fa329ffb640 1 -- 172.21.15.33:0/1517409958 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b663e1e9f0 con 0x55b663f35aa0 2024-03-19T03:01:24.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.535+0000 7fa329ffb640 10 client.17388 handle_command_reply: tid=0 2024-03-19T03:01:24.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.535+0000 7fa34b423740 1 client.17388 shutdown 2024-03-19T03:01:24.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.536+0000 7fa34b423740 1 -- 172.21.15.33:0/1517409958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b663eb3730 msgr2=0x55b663ed24d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:24.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.536+0000 7fa34b423740 1 --2- 172.21.15.33:0/1517409958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b663eb3730 0x55b663ed24d0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fa33802c240 tx=0x7fa33802bc30 comp rx=0 tx=0).stop 2024-03-19T03:01:24.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.536+0000 7fa34b423740 1 -- 172.21.15.33:0/1517409958 shutdown_connections 2024-03-19T03:01:24.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.536+0000 7fa34b423740 1 --2- 172.21.15.33:0/1517409958 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55b663f35aa0 0x55b663f37ec0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.536+0000 7fa34b423740 1 --2- 172.21.15.33:0/1517409958 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55b663ebea90 0x55b663f33b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.536+0000 7fa329ffb640 0 client.17388 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:01:24.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.536+0000 7fa34b423740 1 --2- 172.21.15.33:0/1517409958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55b663eb3730 0x55b663ed24d0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.536+0000 7fa34b423740 1 --2- 172.21.15.33:0/1517409958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b663eb2d80 0x55b663ed1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.536+0000 7fa34b423740 1 -- 172.21.15.33:0/1517409958 >> 172.21.15.33:0/1517409958 conn(0x55b663eadab0 msgr2=0x55b663ec9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:24.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.536+0000 7fa34b423740 1 -- 172.21.15.33:0/1517409958 shutdown_connections 2024-03-19T03:01:24.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.536+0000 7fa34b423740 1 -- 172.21.15.33:0/1517409958 wait complete. 2024-03-19T03:01:24.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.536+0000 7fa34b423740 20 client.17388 trim_cache size 0 max 16384 2024-03-19T03:01:24.544 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T03:01:24.544 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-03-19T03:01:24.544 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "6fe1e4af-f7c9-4695-b9df-8d8f6fe2fb0c", 2024-03-19T03:01:24.544 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-03-19T03:01:24.544 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T03:01:24.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.536+0000 7fa348836640 1 -- 172.21.15.33:0/3401994883 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa32c0a2970 msgr2=0x7fa32c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:24.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.536+0000 7fa348836640 1 --2- 172.21.15.33:0/3401994883 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa32c0a2970 0x7fa32c0a4e30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa338009900 tx=0x7fa338009580 comp rx=0 tx=0).stop 2024-03-19T03:01:24.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.536+0000 7fa348836640 1 -- 172.21.15.33:0/3401994883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3441066e0 msgr2=0x7fa34419b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:24.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.536+0000 7fa348836640 1 --2- 172.21.15.33:0/3401994883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3441066e0 0x7fa34419b800 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fa33c00d950 tx=0x7fa33c00de10 comp rx=0 tx=0).stop 2024-03-19T03:01:24.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.536+0000 7fa348836640 1 -- 172.21.15.33:0/3401994883 shutdown_connections 2024-03-19T03:01:24.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.536+0000 7fa348836640 1 --2- 172.21.15.33:0/3401994883 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa32c0a2970 0x7fa32c0a4e30 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.536+0000 7fa348836640 1 --2- 172.21.15.33:0/3401994883 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa3441066e0 0x7fa34419b800 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.536+0000 7fa348836640 1 --2- 172.21.15.33:0/3401994883 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa344101090 0x7fa344197480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.536+0000 7fa348836640 1 --2- 172.21.15.33:0/3401994883 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3441006d0 0x7fa344196f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.536+0000 7fa348836640 1 -- 172.21.15.33:0/3401994883 >> 172.21.15.33:0/3401994883 conn(0x7fa344076aa0 msgr2=0x7fa3440fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:24.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.536+0000 7fa348836640 1 -- 172.21.15.33:0/3401994883 shutdown_connections 2024-03-19T03:01:24.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.536+0000 7fa348836640 1 -- 172.21.15.33:0/3401994883 wait complete. 2024-03-19T03:01:24.549 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T03:01:24.549 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T03:01:24.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.652+0000 7f34c93c6640 1 -- 172.21.15.33:0/872924069 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34c4100740 msgr2=0x7f34c4100bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:24.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.652+0000 7f34c93c6640 1 --2- 172.21.15.33:0/872924069 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34c4100740 0x7f34c4100bc0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f34b8009930 tx=0x7f34b802bc30 comp rx=0 tx=0).stop 2024-03-19T03:01:24.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.653+0000 7f34c93c6640 1 -- 172.21.15.33:0/872924069 shutdown_connections 2024-03-19T03:01:24.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.653+0000 7f34c93c6640 1 --2- 172.21.15.33:0/872924069 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34c4101100 0x7f34c410d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.653+0000 7f34c93c6640 1 --2- 172.21.15.33:0/872924069 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34c4100740 0x7f34c4100bc0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.653+0000 7f34c93c6640 1 --2- 172.21.15.33:0/872924069 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34c4106750 0x7f34c4106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.653+0000 7f34c93c6640 1 -- 172.21.15.33:0/872924069 >> 172.21.15.33:0/872924069 conn(0x7f34c4077880 msgr2=0x7f34c4077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:24.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.653+0000 7f34c93c6640 1 -- 172.21.15.33:0/872924069 shutdown_connections 2024-03-19T03:01:24.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.653+0000 7f34c93c6640 1 -- 172.21.15.33:0/872924069 wait complete. 2024-03-19T03:01:24.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.653+0000 7f34c93c6640 1 Processor -- start 2024-03-19T03:01:24.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.653+0000 7f34c93c6640 1 -- start start 2024-03-19T03:01:24.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.653+0000 7f34c93c6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34c4100740 0x7f34c41980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.653+0000 7f34c93c6640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34c4101100 0x7f34c4198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.653+0000 7f34c93c6640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34c4106750 0x7f34c419c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.653+0000 7f34c93c6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f34c4111b10 con 0x7f34c4100740 2024-03-19T03:01:24.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.654+0000 7f34c93c6640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f34c4111990 con 0x7f34c4106750 2024-03-19T03:01:24.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.654+0000 7f34c93c6640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f34c4111c90 con 0x7f34c4101100 2024-03-19T03:01:24.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.654+0000 7f34c2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34c4100740 0x7f34c41980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.654+0000 7f34c2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34c4100740 0x7f34c41980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58700/0 (socket says 172.21.15.33:58700) 2024-03-19T03:01:24.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.654+0000 7f34c2ffd640 1 -- 172.21.15.33:0/543267112 learned_addr learned my addr 172.21.15.33:0/543267112 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:24.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.654+0000 7f34c37fe640 1 --2- 172.21.15.33:0/543267112 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34c4106750 0x7f34c419c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.654+0000 7f34c27fc640 1 --2- 172.21.15.33:0/543267112 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34c4101100 0x7f34c4198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.654+0000 7f34c2ffd640 1 -- 172.21.15.33:0/543267112 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34c4101100 msgr2=0x7f34c4198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:24.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.654+0000 7f34c2ffd640 1 --2- 172.21.15.33:0/543267112 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34c4101100 0x7f34c4198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.654+0000 7f34c2ffd640 1 -- 172.21.15.33:0/543267112 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34c4106750 msgr2=0x7f34c419c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:24.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.654+0000 7f34c2ffd640 1 --2- 172.21.15.33:0/543267112 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34c4106750 0x7f34c419c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.654+0000 7f34c2ffd640 1 -- 172.21.15.33:0/543267112 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34c419d0c0 con 0x7f34c4100740 2024-03-19T03:01:24.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.654+0000 7f34c37fe640 1 --2- 172.21.15.33:0/543267112 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34c4106750 0x7f34c419c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:24.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.654+0000 7f34c2ffd640 1 --2- 172.21.15.33:0/543267112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34c4100740 0x7f34c41980d0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f34b400ea60 tx=0x7f34b400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:24.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.654+0000 7f34abfff640 1 -- 172.21.15.33:0/543267112 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b400cde0 con 0x7f34c4100740 2024-03-19T03:01:24.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.654+0000 7f34abfff640 1 -- 172.21.15.33:0/543267112 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f34b40035f0 con 0x7f34c4100740 2024-03-19T03:01:24.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.654+0000 7f34abfff640 1 -- 172.21.15.33:0/543267112 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b4010690 con 0x7f34c4100740 2024-03-19T03:01:24.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.654+0000 7f34c93c6640 1 -- 172.21.15.33:0/543267112 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34c419d3b0 con 0x7f34c4100740 2024-03-19T03:01:24.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.654+0000 7f34c93c6640 1 -- 172.21.15.33:0/543267112 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34c41a4c40 con 0x7f34c4100740 2024-03-19T03:01:24.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.656+0000 7f34abfff640 1 -- 172.21.15.33:0/543267112 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f34b40037a0 con 0x7f34c4100740 2024-03-19T03:01:24.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.657+0000 7f34cbfb3740 1 Processor -- start 2024-03-19T03:01:24.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.657+0000 7f34cbfb3740 1 -- start start 2024-03-19T03:01:24.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.657+0000 7f34cbfb3740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e7e8e06a30 0x55e7e8e06e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.657+0000 7f34cbfb3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7e8e07350 0x55e7e8e0f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.657+0000 7f34cbfb3740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e7e8e0ff30 0x55e7e8e12320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.657+0000 7f34cbfb3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e7e8d6a500 con 0x55e7e8e07350 2024-03-19T03:01:24.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.657+0000 7f34cbfb3740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55e7e8d12be0 con 0x55e7e8e06a30 2024-03-19T03:01:24.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.657+0000 7f34cbfb3740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e7e8d665a0 con 0x55e7e8e0ff30 2024-03-19T03:01:24.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.657+0000 7f34c2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7e8e07350 0x55e7e8e0f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.657+0000 7f34c2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7e8e07350 0x55e7e8e0f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58708/0 (socket says 172.21.15.33:58708) 2024-03-19T03:01:24.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.657+0000 7f34c2ffd640 1 -- 172.21.15.33:0/877253476 learned_addr learned my addr 172.21.15.33:0/877253476 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:24.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.657+0000 7f34abfff640 1 --2- 172.21.15.33:0/543267112 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f34ac0a2970 0x7f34ac0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.657+0000 7f34c37fe640 1 --2- 172.21.15.33:0/877253476 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e7e8e06a30 0x55e7e8e06e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.657+0000 7f34c27fc640 1 --2- 172.21.15.33:0/877253476 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e7e8e0ff30 0x55e7e8e12320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.657+0000 7f34c37fe640 1 --2- 172.21.15.33:0/543267112 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f34ac0a2970 0x7f34ac0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.657+0000 7f34abfff640 1 -- 172.21.15.33:0/543267112 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f34b40cf420 con 0x7f34c4100740 2024-03-19T03:01:24.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.657+0000 7f34c2ffd640 1 -- 172.21.15.33:0/877253476 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e7e8e0ff30 msgr2=0x55e7e8e12320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:24.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.657+0000 7f34c2ffd640 1 --2- 172.21.15.33:0/877253476 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e7e8e0ff30 0x55e7e8e12320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.657+0000 7f34c2ffd640 1 -- 172.21.15.33:0/877253476 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e7e8e06a30 msgr2=0x55e7e8e06e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:24.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.657+0000 7f34c2ffd640 1 --2- 172.21.15.33:0/877253476 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e7e8e06a30 0x55e7e8e06e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.658+0000 7f34c2ffd640 1 -- 172.21.15.33:0/877253476 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7e8d34000 con 0x55e7e8e07350 2024-03-19T03:01:24.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.658+0000 7f34c2ffd640 1 --2- 172.21.15.33:0/877253476 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7e8e07350 0x55e7e8e0f9f0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f34b409b640 tx=0x7f34b400ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92a129fadb14ba8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:24.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.658+0000 7f34a97fa640 1 -- 172.21.15.33:0/877253476 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b409b480 con 0x55e7e8e07350 2024-03-19T03:01:24.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.658+0000 7f34c37fe640 1 --2- 172.21.15.33:0/543267112 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f34ac0a2970 0x7f34ac0a4e30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f34bc005fd0 tx=0x7f34bc004ab0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:24.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.658+0000 7f34a97fa640 1 -- 172.21.15.33:0/877253476 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f34b4003b30 con 0x55e7e8e07350 2024-03-19T03:01:24.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.658+0000 7f34a97fa640 1 -- 172.21.15.33:0/877253476 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b409b010 con 0x55e7e8e07350 2024-03-19T03:01:24.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.659+0000 7f34cbfb3740 1 -- 172.21.15.33:0/877253476 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7e8e07350 msgr2=0x55e7e8e0f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:24.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.659+0000 7f34cbfb3740 1 --2- 172.21.15.33:0/877253476 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7e8e07350 0x55e7e8e0f9f0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f34b409b640 tx=0x7f34b400ef90 comp rx=0 tx=0).stop 2024-03-19T03:01:24.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.659+0000 7f34cbfb3740 1 -- 172.21.15.33:0/877253476 shutdown_connections 2024-03-19T03:01:24.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.659+0000 7f34cbfb3740 1 --2- 172.21.15.33:0/877253476 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e7e8e0ff30 0x55e7e8e12320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.659+0000 7f34cbfb3740 1 --2- 172.21.15.33:0/877253476 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7e8e07350 0x55e7e8e0f9f0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.659+0000 7f34cbfb3740 1 --2- 172.21.15.33:0/877253476 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e7e8e06a30 0x55e7e8e06e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.659+0000 7f34cbfb3740 1 -- 172.21.15.33:0/877253476 >> 172.21.15.33:0/877253476 conn(0x55e7e8e01160 msgr2=0x55e7e8e01540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:24.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.659+0000 7f34cbfb3740 1 -- 172.21.15.33:0/877253476 shutdown_connections 2024-03-19T03:01:24.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.659+0000 7f34cbfb3740 1 -- 172.21.15.33:0/877253476 wait complete. 2024-03-19T03:01:24.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.659+0000 7f34cbfb3740 1 -- start start 2024-03-19T03:01:24.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.659+0000 7f34cbfb3740 1 Processor -- start 2024-03-19T03:01:24.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.659+0000 7f34cbfb3740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e7e8e06a30 0x55e7e8e0cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.659+0000 7f34cbfb3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7e8e07350 0x55e7e8e0d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.659+0000 7f34cbfb3740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e7e8e0ff30 0x55e7e8e0d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.659+0000 7f34cbfb3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55e7e8e12cc0 con 0x55e7e8e07350 2024-03-19T03:01:24.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.659+0000 7f34cbfb3740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55e7e8d26160 con 0x55e7e8e0ff30 2024-03-19T03:01:24.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.659+0000 7f34cbfb3740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55e7e8e12e40 con 0x55e7e8e06a30 2024-03-19T03:01:24.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.659+0000 7f34cbfb3740 10 client.0 _ll_register_callbacks cb 0x55e7e8d62a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:01:24.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.659+0000 7f34c27fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7e8e07350 0x55e7e8e0d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.659+0000 7f34c2ffd640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e7e8e06a30 0x55e7e8e0cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.659+0000 7f34c2ffd640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e7e8e06a30 0x55e7e8e0cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:56562/0 (socket says 172.21.15.33:56562) 2024-03-19T03:01:24.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.659+0000 7f34c2ffd640 1 -- 172.21.15.33:0/194251913 learned_addr learned my addr 172.21.15.33:0/194251913 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:24.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.659+0000 7f34c37fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e7e8e0ff30 0x55e7e8e0d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.660+0000 7f34c27fc640 1 -- 172.21.15.33:0/194251913 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e7e8e06a30 msgr2=0x55e7e8e0cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:24.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.660+0000 7f34c27fc640 1 --2- 172.21.15.33:0/194251913 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e7e8e06a30 0x55e7e8e0cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.660+0000 7f34c27fc640 1 -- 172.21.15.33:0/194251913 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e7e8e0ff30 msgr2=0x55e7e8e0d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:24.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.660+0000 7f34c27fc640 1 --2- 172.21.15.33:0/194251913 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e7e8e0ff30 0x55e7e8e0d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.660+0000 7f34c27fc640 1 -- 172.21.15.33:0/194251913 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7e8b487d0 con 0x55e7e8e07350 2024-03-19T03:01:24.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.660+0000 7f34c2ffd640 1 --2- 172.21.15.33:0/194251913 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e7e8e06a30 0x55e7e8e0cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:24.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.660+0000 7f34c27fc640 1 --2- 172.21.15.33:0/194251913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7e8e07350 0x55e7e8e0d430 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f34b8009230 tx=0x7f34b802ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40ea06dd852a9db6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:24.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.660+0000 7f34a6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:01:24.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.660+0000 7f34a6ff5640 1 -- 172.21.15.33:0/194251913 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b802c5d0 con 0x55e7e8e07350 2024-03-19T03:01:24.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.660+0000 7f34a6ff5640 1 -- 172.21.15.33:0/194251913 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f34b8005030 con 0x55e7e8e07350 2024-03-19T03:01:24.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.660+0000 7f34cbfb3740 1 -- 172.21.15.33:0/194251913 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e7e8d77a40 con 0x55e7e8e07350 2024-03-19T03:01:24.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.660+0000 7f34a6ff5640 1 -- 172.21.15.33:0/194251913 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f34b8005250 con 0x55e7e8e07350 2024-03-19T03:01:24.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.660+0000 7f34cbfb3740 10 client.17400 fetch_fsmap learned FSMap version 25 2024-03-19T03:01:24.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.660+0000 7f34cbfb3740 1 -- 172.21.15.33:0/194251913 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55e7e8a7e3f0 con 0x55e7e8e07350 2024-03-19T03:01:24.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.660+0000 7f34a6ff5640 1 -- 172.21.15.33:0/194251913 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55e7e8d77a40 con 0x55e7e8e07350 2024-03-19T03:01:24.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.660+0000 7f34a6ff5640 1 -- 172.21.15.33:0/194251913 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f34b80035f0 con 0x55e7e8e07350 2024-03-19T03:01:24.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.660+0000 7f34cbfb3740 10 client.17400 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:01:24.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.660+0000 7f34cbfb3740 10 client.17400 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:01:24.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.660+0000 7f34cbfb3740 20 client.17400 populate_metadata read hostname 'smithi033' 2024-03-19T03:01:24.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.661+0000 7f34cbfb3740 1 --2- 172.21.15.33:0/194251913 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55e7e8e20290 0x55e7e8e226f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.661+0000 7f34cbfb3740 4 client.17400 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:01:24.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.661+0000 7f34cbfb3740 1 -- 172.21.15.33:0/194251913 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7e8d71d90 con 0x55e7e8e20290 2024-03-19T03:01:24.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.661+0000 7f34c2ffd640 1 --2- 172.21.15.33:0/194251913 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55e7e8e20290 0x55e7e8e226f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.661+0000 7f34c2ffd640 1 --2- 172.21.15.33:0/194251913 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55e7e8e20290 0x55e7e8e226f0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1bec539a014c658 server_cookie=ef2d23ef3c73778e in_seq=0 out_seq=0 2024-03-19T03:01:24.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.661+0000 7f34a6ff5640 10 client.17400 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:01:24.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.662+0000 7f34a6ff5640 1 -- 172.21.15.33:0/194251913 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e7e8d71d90 con 0x55e7e8e20290 2024-03-19T03:01:24.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.662+0000 7f34a6ff5640 10 client.17400 handle_command_reply: tid=0 2024-03-19T03:01:24.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.662+0000 7f34cbfb3740 1 client.17400 shutdown 2024-03-19T03:01:24.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.662+0000 7f34cbfb3740 1 -- 172.21.15.33:0/194251913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7e8e07350 msgr2=0x55e7e8e0d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:24.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.662+0000 7f34cbfb3740 1 --2- 172.21.15.33:0/194251913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7e8e07350 0x55e7e8e0d430 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f34b8009230 tx=0x7f34b802ca10 comp rx=0 tx=0).stop 2024-03-19T03:01:24.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.662+0000 7f34cbfb3740 1 -- 172.21.15.33:0/194251913 shutdown_connections 2024-03-19T03:01:24.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.662+0000 7f34cbfb3740 1 --2- 172.21.15.33:0/194251913 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55e7e8e20290 0x55e7e8e226f0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.662+0000 7f34cbfb3740 1 --2- 172.21.15.33:0/194251913 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e7e8e0ff30 0x55e7e8e0d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.662+0000 7f34cbfb3740 1 --2- 172.21.15.33:0/194251913 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7e8e07350 0x55e7e8e0d430 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.662+0000 7f34a6ff5640 0 client.17400 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:01:24.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.662+0000 7f34cbfb3740 1 --2- 172.21.15.33:0/194251913 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e7e8e06a30 0x55e7e8e0cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.662+0000 7f34cbfb3740 1 -- 172.21.15.33:0/194251913 >> 172.21.15.33:0/194251913 conn(0x55e7e8e01160 msgr2=0x55e7e8e02eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:24.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.662+0000 7f34cbfb3740 1 -- 172.21.15.33:0/194251913 shutdown_connections 2024-03-19T03:01:24.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.662+0000 7f34cbfb3740 1 -- 172.21.15.33:0/194251913 wait complete. 2024-03-19T03:01:24.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.662+0000 7f34cbfb3740 20 client.17400 trim_cache size 0 max 16384 2024-03-19T03:01:24.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.679+0000 7f34cbfb3740 1 Processor -- start 2024-03-19T03:01:24.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.679+0000 7f34cbfb3740 1 -- start start 2024-03-19T03:01:24.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.679+0000 7f34cbfb3740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e7e8e01160 0x55e7e8e0f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.679+0000 7f34cbfb3740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e7e8e06a30 0x55e7e8e1ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.679+0000 7f34cbfb3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7e8e07350 0x55e7e8e27fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.679+0000 7f34cbfb3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f34c41b15c0 con 0x55e7e8e07350 2024-03-19T03:01:24.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.679+0000 7f34cbfb3740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f34c41aee90 con 0x55e7e8e06a30 2024-03-19T03:01:24.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.679+0000 7f34cbfb3740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f34c41acd00 con 0x55e7e8e01160 2024-03-19T03:01:24.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.679+0000 7f34c37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7e8e07350 0x55e7e8e27fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.679+0000 7f34c37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7e8e07350 0x55e7e8e27fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58716/0 (socket says 172.21.15.33:58716) 2024-03-19T03:01:24.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.679+0000 7f34c37fe640 1 -- 172.21.15.33:0/1665877443 learned_addr learned my addr 172.21.15.33:0/1665877443 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:24.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.680+0000 7f34c27fc640 1 --2- 172.21.15.33:0/1665877443 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e7e8e06a30 0x55e7e8e1ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.680+0000 7f34c2ffd640 1 --2- 172.21.15.33:0/1665877443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e7e8e01160 0x55e7e8e0f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.680+0000 7f34c37fe640 1 -- 172.21.15.33:0/1665877443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e7e8e01160 msgr2=0x55e7e8e0f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:24.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.680+0000 7f34c37fe640 1 --2- 172.21.15.33:0/1665877443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e7e8e01160 0x55e7e8e0f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.680+0000 7f34c37fe640 1 -- 172.21.15.33:0/1665877443 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e7e8e06a30 msgr2=0x55e7e8e1ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:24.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.680+0000 7f34c37fe640 1 --2- 172.21.15.33:0/1665877443 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e7e8e06a30 0x55e7e8e1ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.680+0000 7f34c37fe640 1 -- 172.21.15.33:0/1665877443 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7e8a3e0f0 con 0x55e7e8e07350 2024-03-19T03:01:24.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.680+0000 7f34c37fe640 1 --2- 172.21.15.33:0/1665877443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7e8e07350 0x55e7e8e27fc0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f34bc015000 tx=0x7f34bc009eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4328434d041526be server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:24.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.680+0000 7f34a8ff9640 1 -- 172.21.15.33:0/1665877443 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc01a070 con 0x55e7e8e07350 2024-03-19T03:01:24.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.680+0000 7f34a8ff9640 1 -- 172.21.15.33:0/1665877443 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f34bc004d20 con 0x55e7e8e07350 2024-03-19T03:01:24.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.680+0000 7f34a8ff9640 1 -- 172.21.15.33:0/1665877443 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc021660 con 0x55e7e8e07350 2024-03-19T03:01:24.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.681+0000 7f34cbfb3740 1 -- 172.21.15.33:0/1665877443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7e8e07350 msgr2=0x55e7e8e27fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:24.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.681+0000 7f34cbfb3740 1 --2- 172.21.15.33:0/1665877443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7e8e07350 0x55e7e8e27fc0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f34bc015000 tx=0x7f34bc009eb0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.681+0000 7f34cbfb3740 1 -- 172.21.15.33:0/1665877443 shutdown_connections 2024-03-19T03:01:24.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.681+0000 7f34cbfb3740 1 --2- 172.21.15.33:0/1665877443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7e8e07350 0x55e7e8e27fc0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.681+0000 7f34cbfb3740 1 --2- 172.21.15.33:0/1665877443 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e7e8e06a30 0x55e7e8e1ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.681+0000 7f34cbfb3740 1 --2- 172.21.15.33:0/1665877443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e7e8e01160 0x55e7e8e0f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.681+0000 7f34cbfb3740 1 -- 172.21.15.33:0/1665877443 >> 172.21.15.33:0/1665877443 conn(0x55e7e8e0bb30 msgr2=0x55e7e8e02e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:24.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.681+0000 7f34cbfb3740 1 -- 172.21.15.33:0/1665877443 shutdown_connections 2024-03-19T03:01:24.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.681+0000 7f34cbfb3740 1 -- 172.21.15.33:0/1665877443 wait complete. 2024-03-19T03:01:24.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.681+0000 7f34cbfb3740 1 -- start start 2024-03-19T03:01:24.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.681+0000 7f34cbfb3740 1 Processor -- start 2024-03-19T03:01:24.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.681+0000 7f34cbfb3740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7e8e01160 0x55e7e8e13fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.681+0000 7f34cbfb3740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e7e8e06a30 0x55e7e8e144f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.681+0000 7f34cbfb3740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e7e8e07350 0x55e7e8e89a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.681+0000 7f34cbfb3740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f34c41ad350 con 0x55e7e8e01160 2024-03-19T03:01:24.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.681+0000 7f34cbfb3740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f34c41ae7c0 con 0x55e7e8e06a30 2024-03-19T03:01:24.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.681+0000 7f34cbfb3740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f34c41ada20 con 0x55e7e8e07350 2024-03-19T03:01:24.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.681+0000 7f34cbfb3740 10 client.0 _ll_register_callbacks cb 0x7f34b8005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:01:24.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.681+0000 7f34c2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7e8e01160 0x55e7e8e13fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.681+0000 7f34c2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7e8e01160 0x55e7e8e13fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58730/0 (socket says 172.21.15.33:58730) 2024-03-19T03:01:24.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.681+0000 7f34c2ffd640 1 -- 172.21.15.33:0/1072453084 learned_addr learned my addr 172.21.15.33:0/1072453084 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:24.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.681+0000 7f34c27fc640 1 --2- 172.21.15.33:0/1072453084 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e7e8e06a30 0x55e7e8e144f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.681+0000 7f34c37fe640 1 --2- 172.21.15.33:0/1072453084 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e7e8e07350 0x55e7e8e89a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.688 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.681+0000 7f34c2ffd640 1 -- 172.21.15.33:0/1072453084 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e7e8e07350 msgr2=0x55e7e8e89a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:24.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.681+0000 7f34c2ffd640 1 --2- 172.21.15.33:0/1072453084 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e7e8e07350 0x55e7e8e89a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.681+0000 7f34c2ffd640 1 -- 172.21.15.33:0/1072453084 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e7e8e06a30 msgr2=0x55e7e8e144f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:24.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.681+0000 7f34c2ffd640 1 --2- 172.21.15.33:0/1072453084 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e7e8e06a30 0x55e7e8e144f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.682+0000 7f34c2ffd640 1 -- 172.21.15.33:0/1072453084 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7e8e89f40 con 0x55e7e8e01160 2024-03-19T03:01:24.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.682+0000 7f34c27fc640 1 --2- 172.21.15.33:0/1072453084 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e7e8e06a30 0x55e7e8e144f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:01:24.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.682+0000 7f34c2ffd640 1 --2- 172.21.15.33:0/1072453084 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7e8e01160 0x55e7e8e13fb0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f34b4097390 tx=0x7f34b409b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30e46d1185d83102 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:24.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.682+0000 7f34a6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:01:24.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.682+0000 7f34a6ff5640 1 -- 172.21.15.33:0/1072453084 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b409b1f0 con 0x55e7e8e01160 2024-03-19T03:01:24.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.682+0000 7f34a6ff5640 1 -- 172.21.15.33:0/1072453084 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f34b4004000 con 0x55e7e8e01160 2024-03-19T03:01:24.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.682+0000 7f34cbfb3740 1 -- 172.21.15.33:0/1072453084 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34bc004d20 con 0x55e7e8e01160 2024-03-19T03:01:24.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.682+0000 7f34a6ff5640 1 -- 172.21.15.33:0/1072453084 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f34b4097950 con 0x55e7e8e01160 2024-03-19T03:01:24.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.682+0000 7f34cbfb3740 10 client.17406 fetch_fsmap learned FSMap version 25 2024-03-19T03:01:24.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.682+0000 7f34cbfb3740 1 -- 172.21.15.33:0/1072453084 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55e7e8e8b150 con 0x55e7e8e01160 2024-03-19T03:01:24.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.682+0000 7f34a6ff5640 1 -- 172.21.15.33:0/1072453084 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f34bc004d20 con 0x55e7e8e01160 2024-03-19T03:01:24.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.682+0000 7f34a6ff5640 1 -- 172.21.15.33:0/1072453084 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f34b40cca20 con 0x55e7e8e01160 2024-03-19T03:01:24.689 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.682+0000 7f34cbfb3740 10 client.17406 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:01:24.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.682+0000 7f34cbfb3740 10 client.17406 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:01:24.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.682+0000 7f34cbfb3740 20 client.17406 populate_metadata read hostname 'smithi033' 2024-03-19T03:01:24.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.682+0000 7f34cbfb3740 1 --2- 172.21.15.33:0/1072453084 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55e7e8e0ff30 0x55e7e8e8d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:24.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.682+0000 7f34cbfb3740 4 client.17406 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T03:01:24.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.682+0000 7f34cbfb3740 1 -- 172.21.15.33:0/1072453084 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e7e8cef540 con 0x55e7e8e0ff30 2024-03-19T03:01:24.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.683+0000 7f34c27fc640 1 --2- 172.21.15.33:0/1072453084 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55e7e8e0ff30 0x55e7e8e8d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:24.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.683+0000 7f34c27fc640 1 --2- 172.21.15.33:0/1072453084 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55e7e8e0ff30 0x55e7e8e8d4c0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a235d0646d4aad9 server_cookie=69ec50b98b9ff704 in_seq=0 out_seq=0 2024-03-19T03:01:24.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.683+0000 7f34a6ff5640 10 client.17406 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:01:24.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.683+0000 7f34a6ff5640 1 -- 172.21.15.33:0/1072453084 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55e7e8cef540 con 0x55e7e8e0ff30 2024-03-19T03:01:24.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.683+0000 7f34a6ff5640 10 client.17406 handle_command_reply: tid=0 2024-03-19T03:01:24.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.683+0000 7f34cbfb3740 1 client.17406 shutdown 2024-03-19T03:01:24.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.683+0000 7f34cbfb3740 1 -- 172.21.15.33:0/1072453084 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7e8e01160 msgr2=0x55e7e8e13fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:24.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.683+0000 7f34cbfb3740 1 --2- 172.21.15.33:0/1072453084 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7e8e01160 0x55e7e8e13fb0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f34b4097390 tx=0x7f34b409b480 comp rx=0 tx=0).stop 2024-03-19T03:01:24.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.683+0000 7f34cbfb3740 1 -- 172.21.15.33:0/1072453084 shutdown_connections 2024-03-19T03:01:24.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.683+0000 7f34cbfb3740 1 --2- 172.21.15.33:0/1072453084 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55e7e8e0ff30 0x55e7e8e8d4c0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.683+0000 7f34cbfb3740 1 --2- 172.21.15.33:0/1072453084 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55e7e8e07350 0x55e7e8e89a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.683+0000 7f34cbfb3740 1 --2- 172.21.15.33:0/1072453084 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55e7e8e06a30 0x55e7e8e144f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.683+0000 7f34a6ff5640 0 client.17406 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:01:24.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.683+0000 7f34cbfb3740 1 --2- 172.21.15.33:0/1072453084 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55e7e8e01160 0x55e7e8e13fb0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.683+0000 7f34cbfb3740 1 -- 172.21.15.33:0/1072453084 >> 172.21.15.33:0/1072453084 conn(0x55e7e8e0bb30 msgr2=0x55e7e8e02e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:24.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.683+0000 7f34cbfb3740 1 -- 172.21.15.33:0/1072453084 shutdown_connections 2024-03-19T03:01:24.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.683+0000 7f34cbfb3740 1 -- 172.21.15.33:0/1072453084 wait complete. 2024-03-19T03:01:24.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.683+0000 7f34cbfb3740 20 client.17406 trim_cache size 0 max 16384 2024-03-19T03:01:24.691 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T03:01:24.691 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T03:01:24.692 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-03-19T03:01:24.692 INFO:teuthology.orchestra.run.smithi033.stdout: "6fe1e4af-f7c9-4695-b9df-8d8f6fe2fb0c": { 2024-03-19T03:01:24.692 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-03-19T03:01:24.692 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "6fe1e4af-f7c9-4695-b9df-8d8f6fe2fb0c", 2024-03-19T03:01:24.692 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-03-19T03:01:24.692 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:01:24.692 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:01:24.692 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T03:01:24.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.684+0000 7f34c93c6640 1 -- 172.21.15.33:0/543267112 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f34ac0a2970 msgr2=0x7f34ac0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:24.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.684+0000 7f34c93c6640 1 --2- 172.21.15.33:0/543267112 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f34ac0a2970 0x7f34ac0a4e30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f34bc005fd0 tx=0x7f34bc004ab0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.684+0000 7f34c93c6640 1 -- 172.21.15.33:0/543267112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34c4100740 msgr2=0x7f34c41980d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:24.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.684+0000 7f34c93c6640 1 --2- 172.21.15.33:0/543267112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34c4100740 0x7f34c41980d0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f34b400ea60 tx=0x7f34b400ef20 comp rx=0 tx=0).stop 2024-03-19T03:01:24.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.684+0000 7f34c93c6640 1 -- 172.21.15.33:0/543267112 shutdown_connections 2024-03-19T03:01:24.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.684+0000 7f34c93c6640 1 --2- 172.21.15.33:0/543267112 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f34ac0a2970 0x7f34ac0a4e30 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.684+0000 7f34c93c6640 1 --2- 172.21.15.33:0/543267112 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f34c4101100 0x7f34c4198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.684+0000 7f34c93c6640 1 --2- 172.21.15.33:0/543267112 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34c4106750 0x7f34c419c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.684+0000 7f34c93c6640 1 --2- 172.21.15.33:0/543267112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f34c4100740 0x7f34c41980d0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:24.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.684+0000 7f34c93c6640 1 -- 172.21.15.33:0/543267112 >> 172.21.15.33:0/543267112 conn(0x7f34c4077880 msgr2=0x7f34c4103410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:24.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.684+0000 7f34c93c6640 1 -- 172.21.15.33:0/543267112 shutdown_connections 2024-03-19T03:01:24.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:24.684+0000 7f34c93c6640 1 -- 172.21.15.33:0/543267112 wait complete. 2024-03-19T03:01:24.696 INFO:tasks.cephfs.filesystem:scrub status for tag:6fe1e4af-f7c9-4695-b9df-8d8f6fe2fb0c - {'path': '/', 'tag': '6fe1e4af-f7c9-4695-b9df-8d8f6fe2fb0c', 'options': 'recursive,force'} 2024-03-19T03:01:24.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:24 smithi096 ceph-mon[35946]: pgmap v245: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-19T03:01:25.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:25 smithi155 ceph-mon[35804]: scrub queued for path: / 2024-03-19T03:01:25.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:25 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T03:01:25.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:25 smithi155 ceph-mon[35804]: scrub summary: active paths [/] 2024-03-19T03:01:25.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:25 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T03:01:25.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:25 smithi033 ceph-mon[31973]: scrub queued for path: / 2024-03-19T03:01:25.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:25 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T03:01:25.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:25 smithi033 ceph-mon[31973]: scrub summary: active paths [/] 2024-03-19T03:01:25.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:25 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T03:01:25.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:25 smithi096 ceph-mon[35946]: scrub queued for path: / 2024-03-19T03:01:25.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:25 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T03:01:25.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:25 smithi096 ceph-mon[35946]: scrub summary: active paths [/] 2024-03-19T03:01:25.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:25 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T03:01:26.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:26 smithi155 ceph-mon[35804]: pgmap v246: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-19T03:01:26.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:26 smithi033 ceph-mon[31973]: pgmap v246: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-19T03:01:26.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:26 smithi096 ceph-mon[35946]: pgmap v246: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-19T03:01:28.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:28 smithi155 ceph-mon[35804]: pgmap v247: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-19T03:01:28.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:28 smithi155 ceph-mon[35804]: scrub summary: idle 2024-03-19T03:01:28.517 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:28 smithi096 ceph-mon[35946]: pgmap v247: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-19T03:01:28.517 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:28 smithi096 ceph-mon[35946]: scrub summary: idle 2024-03-19T03:01:28.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:28 smithi033 ceph-mon[31973]: pgmap v247: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-19T03:01:28.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:28 smithi033 ceph-mon[31973]: scrub summary: idle 2024-03-19T03:01:30.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:30 smithi155 ceph-mon[35804]: pgmap v248: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:01:30.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:30 smithi033 ceph-mon[31973]: pgmap v248: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:01:30.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:30 smithi096 ceph-mon[35946]: pgmap v248: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:01:32.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:32 smithi155 ceph-mon[35804]: pgmap v249: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:01:32.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:32 smithi033 ceph-mon[31973]: pgmap v249: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:01:32.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:32 smithi096 ceph-mon[35946]: pgmap v249: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:01:34.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:34 smithi155 ceph-mon[35804]: pgmap v250: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:01:34.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:34 smithi033 ceph-mon[31973]: pgmap v250: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:01:34.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:34 smithi096 ceph-mon[35946]: pgmap v250: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:01:36.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:36 smithi155 ceph-mon[35804]: pgmap v251: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:01:36.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:36 smithi033 ceph-mon[31973]: pgmap v251: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:01:36.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:36 smithi096 ceph-mon[35946]: pgmap v251: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:01:38.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:38 smithi155 ceph-mon[35804]: pgmap v252: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:01:38.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:38 smithi033 ceph-mon[31973]: pgmap v252: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:01:38.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:38 smithi096 ceph-mon[35946]: pgmap v252: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:01:39.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:39 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:01:39.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:39 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:01:39.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:39 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:01:40.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:40 smithi155 ceph-mon[35804]: pgmap v253: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:01:40.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:40 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:01:40.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:40 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:01:40.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:40 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:01:40.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:40 smithi033 ceph-mon[31973]: pgmap v253: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:01:40.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:40 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:01:40.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:40 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:01:40.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:40 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:01:40.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:40 smithi096 ceph-mon[35946]: pgmap v253: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:01:40.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:40 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:01:40.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:40 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:01:40.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:40 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:01:42.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:42 smithi155 ceph-mon[35804]: pgmap v254: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:01:42.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:42 smithi033 ceph-mon[31973]: pgmap v254: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:01:42.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:42 smithi096 ceph-mon[35946]: pgmap v254: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:01:44.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:44 smithi155 ceph-mon[35804]: pgmap v255: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:01:44.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:44 smithi033 ceph-mon[31973]: pgmap v255: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:01:44.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:44 smithi096 ceph-mon[35946]: pgmap v255: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:01:46.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:46 smithi155 ceph-mon[35804]: pgmap v256: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:01:46.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:46 smithi033 ceph-mon[31973]: pgmap v256: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:01:46.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:46 smithi096 ceph-mon[35946]: pgmap v256: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:01:48.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:48 smithi033 ceph-mon[31973]: pgmap v257: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:01:48.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:48 smithi096 ceph-mon[35946]: pgmap v257: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:01:48.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:48 smithi155 ceph-mon[35804]: pgmap v257: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:01:50.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:50 smithi033 ceph-mon[31973]: pgmap v258: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:01:50.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:50 smithi096 ceph-mon[35946]: pgmap v258: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:01:50.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:50 smithi155 ceph-mon[35804]: pgmap v258: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:01:52.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:52 smithi033 ceph-mon[31973]: pgmap v259: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:01:52.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:52 smithi096 ceph-mon[35946]: pgmap v259: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:01:52.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:52 smithi155 ceph-mon[35804]: pgmap v259: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:01:54.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:54 smithi033 ceph-mon[31973]: pgmap v260: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:01:54.568 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:54 smithi155 ceph-mon[35804]: pgmap v260: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:01:54.697 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T03:01:54.697 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T03:01:54.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:54 smithi096 ceph-mon[35946]: pgmap v260: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:01:54.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.813+0000 7f8f7d1f4640 1 -- 172.21.15.33:0/2831486663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f78100740 msgr2=0x7f8f78100bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:54.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.813+0000 7f8f7d1f4640 1 --2- 172.21.15.33:0/2831486663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f78100740 0x7f8f78100bc0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f8f60009930 tx=0x7f8f6002bc30 comp rx=0 tx=0).stop 2024-03-19T03:01:54.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.813+0000 7f8f7d1f4640 1 -- 172.21.15.33:0/2831486663 shutdown_connections 2024-03-19T03:01:54.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.813+0000 7f8f7d1f4640 1 --2- 172.21.15.33:0/2831486663 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f78101100 0x7f8f7810d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.813+0000 7f8f7d1f4640 1 --2- 172.21.15.33:0/2831486663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f78100740 0x7f8f78100bc0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.813+0000 7f8f7d1f4640 1 --2- 172.21.15.33:0/2831486663 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f78106750 0x7f8f78106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.813+0000 7f8f7d1f4640 1 -- 172.21.15.33:0/2831486663 >> 172.21.15.33:0/2831486663 conn(0x7f8f78077880 msgr2=0x7f8f78077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:54.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.813+0000 7f8f7d1f4640 1 -- 172.21.15.33:0/2831486663 shutdown_connections 2024-03-19T03:01:54.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.813+0000 7f8f7d1f4640 1 -- 172.21.15.33:0/2831486663 wait complete. 2024-03-19T03:01:54.816 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.814+0000 7f8f7d1f4640 1 Processor -- start 2024-03-19T03:01:54.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.814+0000 7f8f7d1f4640 1 -- start start 2024-03-19T03:01:54.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.814+0000 7f8f7d1f4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f78100740 0x7f8f78198ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:54.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.814+0000 7f8f7d1f4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f78101100 0x7f8f78199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:54.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.814+0000 7f8f7d1f4640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f78106750 0x7f8f781a3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:54.818 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.814+0000 7f8f7d1f4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8f78110a10 con 0x7f8f78101100 2024-03-19T03:01:54.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.814+0000 7f8f7d1f4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8f78110890 con 0x7f8f78100740 2024-03-19T03:01:54.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.814+0000 7f8f7d1f4640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8f78110b90 con 0x7f8f78106750 2024-03-19T03:01:54.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.815+0000 7f8f76d9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f78100740 0x7f8f78198ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:54.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.815+0000 7f8f6e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f78101100 0x7f8f78199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:54.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.815+0000 7f8f76d9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f78100740 0x7f8f78198ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:33560/0 (socket says 172.21.15.33:33560) 2024-03-19T03:01:54.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.815+0000 7f8f6e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f78101100 0x7f8f78199410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35710/0 (socket says 172.21.15.33:35710) 2024-03-19T03:01:54.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.815+0000 7f8f7759e640 1 --2- 172.21.15.33:0/43461855 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f78106750 0x7f8f781a3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:54.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.815+0000 7f8f76d9d640 1 -- 172.21.15.33:0/43461855 learned_addr learned my addr 172.21.15.33:0/43461855 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:54.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.815+0000 7f8f6e59c640 1 -- 172.21.15.33:0/43461855 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f78106750 msgr2=0x7f8f781a3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:54.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.815+0000 7f8f6e59c640 1 --2- 172.21.15.33:0/43461855 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f78106750 0x7f8f781a3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.815+0000 7f8f6e59c640 1 -- 172.21.15.33:0/43461855 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f78100740 msgr2=0x7f8f78198ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:54.819 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.815+0000 7f8f6e59c640 1 --2- 172.21.15.33:0/43461855 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f78100740 0x7f8f78198ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.815+0000 7f8f6e59c640 1 -- 172.21.15.33:0/43461855 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f781a4490 con 0x7f8f78101100 2024-03-19T03:01:54.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.815+0000 7f8f76d9d640 1 --2- 172.21.15.33:0/43461855 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f78100740 0x7f8f78198ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:54.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.815+0000 7f8f6e59c640 1 --2- 172.21.15.33:0/43461855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f78101100 0x7f8f78199410 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f8f60009230 tx=0x7f8f600045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:54.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.815+0000 7f8f74d99640 1 -- 172.21.15.33:0/43461855 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f60043020 con 0x7f8f78101100 2024-03-19T03:01:54.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.815+0000 7f8f7d1f4640 1 -- 172.21.15.33:0/43461855 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f781a4660 con 0x7f8f78101100 2024-03-19T03:01:54.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.815+0000 7f8f74d99640 1 -- 172.21.15.33:0/43461855 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8f6002ca70 con 0x7f8f78101100 2024-03-19T03:01:54.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.815+0000 7f8f74d99640 1 -- 172.21.15.33:0/43461855 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f600039b0 con 0x7f8f78101100 2024-03-19T03:01:54.820 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.815+0000 7f8f7d1f4640 1 -- 172.21.15.33:0/43461855 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f781a4b70 con 0x7f8f78101100 2024-03-19T03:01:54.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.818+0000 7f8f74d99640 1 -- 172.21.15.33:0/43461855 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f8f60046020 con 0x7f8f78101100 2024-03-19T03:01:54.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.818+0000 7f8f7fde9740 1 Processor -- start 2024-03-19T03:01:54.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.818+0000 7f8f7fde9740 1 -- start start 2024-03-19T03:01:54.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.818+0000 7f8f7fde9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558033dd6a30 0x558033dd6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:54.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.818+0000 7f8f7fde9740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558033dd7350 0x558033ddf9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:54.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.819+0000 7f8f7fde9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558033ddff30 0x558033de2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:54.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.819+0000 7f8f7fde9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558033d3a500 con 0x558033dd6a30 2024-03-19T03:01:54.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.819+0000 7f8f7fde9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x558033ce2be0 con 0x558033ddff30 2024-03-19T03:01:54.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.819+0000 7f8f7fde9740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558033d365a0 con 0x558033dd7350 2024-03-19T03:01:54.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.819+0000 7f8f7759e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558033dd6a30 0x558033dd6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:54.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.819+0000 7f8f7759e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558033dd6a30 0x558033dd6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35724/0 (socket says 172.21.15.33:35724) 2024-03-19T03:01:54.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.819+0000 7f8f7759e640 1 -- 172.21.15.33:0/1337472657 learned_addr learned my addr 172.21.15.33:0/1337472657 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:54.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.819+0000 7f8f6e59c640 1 --2- 172.21.15.33:0/1337472657 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558033ddff30 0x558033de2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:54.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.819+0000 7f8f76d9d640 1 --2- 172.21.15.33:0/1337472657 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558033dd7350 0x558033ddf9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:54.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.819+0000 7f8f74d99640 1 --2- 172.21.15.33:0/43461855 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f8f640a2970 0x7f8f640a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:54.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.819+0000 7f8f7759e640 1 -- 172.21.15.33:0/1337472657 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558033dd7350 msgr2=0x558033ddf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:54.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.819+0000 7f8f7759e640 1 --2- 172.21.15.33:0/1337472657 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558033dd7350 0x558033ddf9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.819+0000 7f8f7759e640 1 -- 172.21.15.33:0/1337472657 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558033ddff30 msgr2=0x558033de2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:54.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.819+0000 7f8f7759e640 1 --2- 172.21.15.33:0/1337472657 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558033ddff30 0x558033de2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.819+0000 7f8f7759e640 1 -- 172.21.15.33:0/1337472657 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558033d04000 con 0x558033dd6a30 2024-03-19T03:01:54.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.819+0000 7f8f6e59c640 1 --2- 172.21.15.33:0/1337472657 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558033ddff30 0x558033de2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:54.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.819+0000 7f8f76d9d640 1 --2- 172.21.15.33:0/1337472657 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558033dd7350 0x558033ddf9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:54.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.819+0000 7f8f74d99640 1 -- 172.21.15.33:0/43461855 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f8f600f10e0 con 0x7f8f78101100 2024-03-19T03:01:54.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.819+0000 7f8f7759e640 1 --2- 172.21.15.33:0/43461855 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f8f640a2970 0x7f8f640a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:54.828 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.819+0000 7f8f7759e640 1 --2- 172.21.15.33:0/1337472657 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558033dd6a30 0x558033dd6e10 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f8f70011590 tx=0x7f8f70011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41d4af8c5d50ac97 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:54.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.819+0000 7f8f6d59a640 1 -- 172.21.15.33:0/1337472657 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f70017050 con 0x558033dd6a30 2024-03-19T03:01:54.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.819+0000 7f8f6d59a640 1 -- 172.21.15.33:0/1337472657 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8f7000fcf0 con 0x558033dd6a30 2024-03-19T03:01:54.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.820+0000 7f8f6d59a640 1 -- 172.21.15.33:0/1337472657 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7001b9f0 con 0x558033dd6a30 2024-03-19T03:01:54.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.820+0000 7f8f7fde9740 1 -- 172.21.15.33:0/1337472657 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558033dd6a30 msgr2=0x558033dd6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:54.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.820+0000 7f8f7759e640 1 --2- 172.21.15.33:0/43461855 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f8f640a2970 0x7f8f640a4e30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8f70022000 tx=0x7f8f70023000 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:54.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.820+0000 7f8f7fde9740 1 --2- 172.21.15.33:0/1337472657 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558033dd6a30 0x558033dd6e10 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f8f70011590 tx=0x7f8f70011a50 comp rx=0 tx=0).stop 2024-03-19T03:01:54.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.820+0000 7f8f7fde9740 1 -- 172.21.15.33:0/1337472657 shutdown_connections 2024-03-19T03:01:54.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.820+0000 7f8f7fde9740 1 --2- 172.21.15.33:0/1337472657 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558033dd7350 0x558033ddf9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.820+0000 7f8f7fde9740 1 --2- 172.21.15.33:0/1337472657 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558033ddff30 0x558033de2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.820+0000 7f8f7fde9740 1 --2- 172.21.15.33:0/1337472657 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558033dd6a30 0x558033dd6e10 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.820+0000 7f8f7fde9740 1 -- 172.21.15.33:0/1337472657 >> 172.21.15.33:0/1337472657 conn(0x558033dd1160 msgr2=0x558033dd1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:54.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.820+0000 7f8f7fde9740 1 -- 172.21.15.33:0/1337472657 shutdown_connections 2024-03-19T03:01:54.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.820+0000 7f8f7fde9740 1 -- 172.21.15.33:0/1337472657 wait complete. 2024-03-19T03:01:54.829 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.821+0000 7f8f7fde9740 1 -- start start 2024-03-19T03:01:54.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.821+0000 7f8f7fde9740 1 Processor -- start 2024-03-19T03:01:54.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.821+0000 7f8f7fde9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558033dd6a30 0x558033ddcee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:54.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.821+0000 7f8f7fde9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558033dd7350 0x558033ddd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:54.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.821+0000 7f8f7fde9740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558033ddff30 0x558033ddd960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:54.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.821+0000 7f8f7fde9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x558033de2d20 con 0x558033dd7350 2024-03-19T03:01:54.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.821+0000 7f8f7fde9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x558033cf6160 con 0x558033dd6a30 2024-03-19T03:01:54.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.821+0000 7f8f7fde9740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x558033de2ea0 con 0x558033ddff30 2024-03-19T03:01:54.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.821+0000 7f8f7fde9740 10 client.0 _ll_register_callbacks cb 0x558033d32a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:01:54.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.821+0000 7f8f6e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558033dd7350 0x558033ddd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:54.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.821+0000 7f8f6e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558033dd7350 0x558033ddd420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35732/0 (socket says 172.21.15.33:35732) 2024-03-19T03:01:54.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.821+0000 7f8f6e59c640 1 -- 172.21.15.33:0/3701755361 learned_addr learned my addr 172.21.15.33:0/3701755361 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:54.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.821+0000 7f8f7759e640 1 --2- 172.21.15.33:0/3701755361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558033ddff30 0x558033ddd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:54.830 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.821+0000 7f8f76d9d640 1 --2- 172.21.15.33:0/3701755361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558033dd6a30 0x558033ddcee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:54.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.821+0000 7f8f6e59c640 1 -- 172.21.15.33:0/3701755361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558033ddff30 msgr2=0x558033ddd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:54.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.821+0000 7f8f6e59c640 1 --2- 172.21.15.33:0/3701755361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558033ddff30 0x558033ddd960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.821+0000 7f8f6e59c640 1 -- 172.21.15.33:0/3701755361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558033dd6a30 msgr2=0x558033ddcee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:54.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.821+0000 7f8f6e59c640 1 --2- 172.21.15.33:0/3701755361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558033dd6a30 0x558033ddcee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.821+0000 7f8f6e59c640 1 -- 172.21.15.33:0/3701755361 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558033b187d0 con 0x558033dd7350 2024-03-19T03:01:54.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.821+0000 7f8f76d9d640 1 --2- 172.21.15.33:0/3701755361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558033dd6a30 0x558033ddcee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:01:54.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.821+0000 7f8f7759e640 1 --2- 172.21.15.33:0/3701755361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558033ddff30 0x558033ddd960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:01:54.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.822+0000 7f8f6e59c640 1 --2- 172.21.15.33:0/3701755361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558033dd7350 0x558033ddd420 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f8f60009900 tx=0x7f8f60000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf1e3fd9d96e67f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:54.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.822+0000 7f8f5e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:01:54.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.822+0000 7f8f5e7fc640 1 -- 172.21.15.33:0/3701755361 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f600bcf60 con 0x558033dd7350 2024-03-19T03:01:54.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.822+0000 7f8f5e7fc640 1 -- 172.21.15.33:0/3701755361 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f8f600f4c80 con 0x558033dd7350 2024-03-19T03:01:54.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.822+0000 7f8f7fde9740 1 -- 172.21.15.33:0/3701755361 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f7000fcf0 con 0x558033dd7350 2024-03-19T03:01:54.831 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.822+0000 7f8f5e7fc640 1 -- 172.21.15.33:0/3701755361 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8f600bcd80 con 0x558033dd7350 2024-03-19T03:01:54.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.822+0000 7f8f7fde9740 10 client.17418 fetch_fsmap learned FSMap version 25 2024-03-19T03:01:54.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.822+0000 7f8f7fde9740 1 -- 172.21.15.33:0/3701755361 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558033a4e3f0 con 0x558033dd7350 2024-03-19T03:01:54.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.822+0000 7f8f5e7fc640 1 -- 172.21.15.33:0/3701755361 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8f7000fcf0 con 0x558033dd7350 2024-03-19T03:01:54.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.822+0000 7f8f5e7fc640 1 -- 172.21.15.33:0/3701755361 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f8f600ee680 con 0x558033dd7350 2024-03-19T03:01:54.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.822+0000 7f8f7fde9740 10 client.17418 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:01:54.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.822+0000 7f8f7fde9740 10 client.17418 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:01:54.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.822+0000 7f8f7fde9740 20 client.17418 populate_metadata read hostname 'smithi033' 2024-03-19T03:01:54.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.822+0000 7f8f7fde9740 1 --2- 172.21.15.33:0/3701755361 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x558033df0270 0x558033df2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:54.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.822+0000 7f8f7fde9740 4 client.17418 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:01:54.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.822+0000 7f8f7fde9740 1 -- 172.21.15.33:0/3701755361 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558033d47a40 con 0x558033df0270 2024-03-19T03:01:54.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.822+0000 7f8f76d9d640 1 --2- 172.21.15.33:0/3701755361 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x558033df0270 0x558033df2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:54.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.823+0000 7f8f76d9d640 1 --2- 172.21.15.33:0/3701755361 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x558033df0270 0x558033df2660 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7478355b2e0ebd04 server_cookie=55fbfacb4c2e001f in_seq=0 out_seq=0 2024-03-19T03:01:54.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.823+0000 7f8f5e7fc640 10 client.17418 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:01:54.832 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.823+0000 7f8f5e7fc640 1 -- 172.21.15.33:0/3701755361 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558033d47a40 con 0x558033df0270 2024-03-19T03:01:54.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.823+0000 7f8f5e7fc640 10 client.17418 handle_command_reply: tid=0 2024-03-19T03:01:54.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.823+0000 7f8f7fde9740 1 client.17418 shutdown 2024-03-19T03:01:54.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.824+0000 7f8f7fde9740 1 -- 172.21.15.33:0/3701755361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558033dd7350 msgr2=0x558033ddd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:54.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.824+0000 7f8f7fde9740 1 --2- 172.21.15.33:0/3701755361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558033dd7350 0x558033ddd420 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f8f60009900 tx=0x7f8f60000bc0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.824+0000 7f8f7fde9740 1 -- 172.21.15.33:0/3701755361 shutdown_connections 2024-03-19T03:01:54.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.824+0000 7f8f7fde9740 1 --2- 172.21.15.33:0/3701755361 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x558033df0270 0x558033df2660 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.824+0000 7f8f7fde9740 1 --2- 172.21.15.33:0/3701755361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558033ddff30 0x558033ddd960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.824+0000 7f8f7fde9740 1 --2- 172.21.15.33:0/3701755361 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558033dd7350 0x558033ddd420 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.824+0000 7f8f5e7fc640 0 client.17418 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:01:54.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.824+0000 7f8f7fde9740 1 --2- 172.21.15.33:0/3701755361 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558033dd6a30 0x558033ddcee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.824+0000 7f8f7fde9740 1 -- 172.21.15.33:0/3701755361 >> 172.21.15.33:0/3701755361 conn(0x558033dd1160 msgr2=0x558033dd2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:54.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.824+0000 7f8f7fde9740 1 -- 172.21.15.33:0/3701755361 shutdown_connections 2024-03-19T03:01:54.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.824+0000 7f8f7fde9740 1 -- 172.21.15.33:0/3701755361 wait complete. 2024-03-19T03:01:54.833 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.824+0000 7f8f7fde9740 20 client.17418 trim_cache size 0 max 16384 2024-03-19T03:01:54.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.841+0000 7f8f7fde9740 1 Processor -- start 2024-03-19T03:01:54.848 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.841+0000 7f8f7fde9740 1 -- start start 2024-03-19T03:01:54.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.841+0000 7f8f7fde9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558033dd6a30 0x558033ddbe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:54.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.842+0000 7f8f7fde9740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558033dd7350 0x558033debac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:54.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.842+0000 7f8f7fde9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558033ddff30 0x558033df7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:54.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.842+0000 7f8f7fde9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8f781b14b0 con 0x558033ddff30 2024-03-19T03:01:54.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.842+0000 7f8f7fde9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8f781aed80 con 0x558033dd6a30 2024-03-19T03:01:54.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.842+0000 7f8f7fde9740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8f781acbf0 con 0x558033dd7350 2024-03-19T03:01:54.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.842+0000 7f8f7759e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558033ddff30 0x558033df7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:54.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.842+0000 7f8f7759e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558033ddff30 0x558033df7ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35740/0 (socket says 172.21.15.33:35740) 2024-03-19T03:01:54.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.842+0000 7f8f7759e640 1 -- 172.21.15.33:0/2460825962 learned_addr learned my addr 172.21.15.33:0/2460825962 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:54.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.842+0000 7f8f6e59c640 1 --2- 172.21.15.33:0/2460825962 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558033dd7350 0x558033debac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:54.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.842+0000 7f8f76d9d640 1 --2- 172.21.15.33:0/2460825962 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558033dd6a30 0x558033ddbe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:54.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.842+0000 7f8f7759e640 1 -- 172.21.15.33:0/2460825962 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558033dd7350 msgr2=0x558033debac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:54.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.842+0000 7f8f7759e640 1 --2- 172.21.15.33:0/2460825962 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558033dd7350 0x558033debac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.849 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.842+0000 7f8f7759e640 1 -- 172.21.15.33:0/2460825962 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558033dd6a30 msgr2=0x558033ddbe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:54.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.842+0000 7f8f7759e640 1 --2- 172.21.15.33:0/2460825962 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558033dd6a30 0x558033ddbe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.842+0000 7f8f7759e640 1 -- 172.21.15.33:0/2460825962 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558033a0e0f0 con 0x558033ddff30 2024-03-19T03:01:54.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.842+0000 7f8f6e59c640 1 --2- 172.21.15.33:0/2460825962 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558033dd7350 0x558033debac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:54.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.842+0000 7f8f76d9d640 1 --2- 172.21.15.33:0/2460825962 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558033dd6a30 0x558033ddbe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:54.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.842+0000 7f8f7759e640 1 --2- 172.21.15.33:0/2460825962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558033ddff30 0x558033df7ee0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f8f70003970 tx=0x7f8f700114f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff661fcc95cef05f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:54.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.842+0000 7f8f6cd99640 1 -- 172.21.15.33:0/2460825962 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f70036070 con 0x558033ddff30 2024-03-19T03:01:54.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.842+0000 7f8f6cd99640 1 -- 172.21.15.33:0/2460825962 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8f70009cd0 con 0x558033ddff30 2024-03-19T03:01:54.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.842+0000 7f8f6cd99640 1 -- 172.21.15.33:0/2460825962 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7001b9c0 con 0x558033ddff30 2024-03-19T03:01:54.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.843+0000 7f8f7fde9740 1 -- 172.21.15.33:0/2460825962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558033ddff30 msgr2=0x558033df7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:54.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.843+0000 7f8f7fde9740 1 --2- 172.21.15.33:0/2460825962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558033ddff30 0x558033df7ee0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f8f70003970 tx=0x7f8f700114f0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.843+0000 7f8f7fde9740 1 -- 172.21.15.33:0/2460825962 shutdown_connections 2024-03-19T03:01:54.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.843+0000 7f8f7fde9740 1 --2- 172.21.15.33:0/2460825962 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558033dd7350 0x558033debac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.843+0000 7f8f7fde9740 1 --2- 172.21.15.33:0/2460825962 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558033ddff30 0x558033df7ee0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.843+0000 7f8f7fde9740 1 --2- 172.21.15.33:0/2460825962 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558033dd6a30 0x558033ddbe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.850 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.843+0000 7f8f7fde9740 1 -- 172.21.15.33:0/2460825962 >> 172.21.15.33:0/2460825962 conn(0x558033dd1160 msgr2=0x558033ddd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:54.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.843+0000 7f8f7fde9740 1 -- 172.21.15.33:0/2460825962 shutdown_connections 2024-03-19T03:01:54.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.843+0000 7f8f7fde9740 1 -- 172.21.15.33:0/2460825962 wait complete. 2024-03-19T03:01:54.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.843+0000 7f8f7fde9740 1 -- start start 2024-03-19T03:01:54.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.844+0000 7f8f7fde9740 1 Processor -- start 2024-03-19T03:01:54.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.844+0000 7f8f7fde9740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558033dd6a30 0x558033de80f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:54.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.844+0000 7f8f7fde9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558033dd7350 0x558033de50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:54.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.844+0000 7f8f7fde9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558033ddff30 0x558033de5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:54.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.844+0000 7f8f7fde9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8f781ad240 con 0x558033ddff30 2024-03-19T03:01:54.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.844+0000 7f8f7fde9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8f781ae6b0 con 0x558033dd7350 2024-03-19T03:01:54.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.844+0000 7f8f7fde9740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8f781ad910 con 0x558033dd6a30 2024-03-19T03:01:54.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.844+0000 7f8f7fde9740 10 client.0 _ll_register_callbacks cb 0x7f8f600057a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:01:54.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.844+0000 7f8f7759e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558033ddff30 0x558033de5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:54.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.844+0000 7f8f7759e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558033ddff30 0x558033de5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35746/0 (socket says 172.21.15.33:35746) 2024-03-19T03:01:54.851 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.844+0000 7f8f7759e640 1 -- 172.21.15.33:0/701925159 learned_addr learned my addr 172.21.15.33:0/701925159 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:54.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.844+0000 7f8f76d9d640 1 --2- 172.21.15.33:0/701925159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558033dd6a30 0x558033de80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:54.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.844+0000 7f8f6e59c640 1 --2- 172.21.15.33:0/701925159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558033dd7350 0x558033de50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:54.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.844+0000 7f8f7759e640 1 -- 172.21.15.33:0/701925159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558033dd6a30 msgr2=0x558033de80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:54.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.844+0000 7f8f7759e640 1 --2- 172.21.15.33:0/701925159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558033dd6a30 0x558033de80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.844+0000 7f8f7759e640 1 -- 172.21.15.33:0/701925159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558033dd7350 msgr2=0x558033de50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:54.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.844+0000 7f8f7759e640 1 --2- 172.21.15.33:0/701925159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558033dd7350 0x558033de50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.844+0000 7f8f7759e640 1 -- 172.21.15.33:0/701925159 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558033de8760 con 0x558033ddff30 2024-03-19T03:01:54.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.844+0000 7f8f7759e640 1 --2- 172.21.15.33:0/701925159 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558033ddff30 0x558033de5600 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f8f70009f90 tx=0x7f8f7001b4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d549114f0afdd91a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:54.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.844+0000 7f8f5e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:01:54.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.844+0000 7f8f5e7fc640 1 -- 172.21.15.33:0/701925159 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f70036030 con 0x558033ddff30 2024-03-19T03:01:54.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.845+0000 7f8f5e7fc640 1 -- 172.21.15.33:0/701925159 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f8f7001b9d0 con 0x558033ddff30 2024-03-19T03:01:54.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.845+0000 7f8f7fde9740 1 -- 172.21.15.33:0/701925159 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f70009cd0 con 0x558033ddff30 2024-03-19T03:01:54.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.845+0000 7f8f5e7fc640 1 -- 172.21.15.33:0/701925159 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8f70043050 con 0x558033ddff30 2024-03-19T03:01:54.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.845+0000 7f8f7fde9740 10 client.17424 fetch_fsmap learned FSMap version 25 2024-03-19T03:01:54.852 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.845+0000 7f8f7fde9740 1 -- 172.21.15.33:0/701925159 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558033e56f70 con 0x558033ddff30 2024-03-19T03:01:54.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.845+0000 7f8f5e7fc640 1 -- 172.21.15.33:0/701925159 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8f70009cd0 con 0x558033ddff30 2024-03-19T03:01:54.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.845+0000 7f8f5e7fc640 1 -- 172.21.15.33:0/701925159 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f8f70030070 con 0x558033ddff30 2024-03-19T03:01:54.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.845+0000 7f8f7fde9740 10 client.17424 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:01:54.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.845+0000 7f8f7fde9740 10 client.17424 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:01:54.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.845+0000 7f8f7fde9740 20 client.17424 populate_metadata read hostname 'smithi033' 2024-03-19T03:01:54.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.845+0000 7f8f7fde9740 1 --2- 172.21.15.33:0/701925159 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x558033e57440 0x558033e59830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:54.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.845+0000 7f8f7fde9740 4 client.17424 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T03:01:54.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.845+0000 7f8f7fde9740 1 -- 172.21.15.33:0/701925159 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558033d41d90 con 0x558033e57440 2024-03-19T03:01:54.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.845+0000 7f8f76d9d640 1 --2- 172.21.15.33:0/701925159 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x558033e57440 0x558033e59830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:54.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.846+0000 7f8f5e7fc640 10 client.17424 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:01:54.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.846+0000 7f8f76d9d640 1 --2- 172.21.15.33:0/701925159 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x558033e57440 0x558033e59830 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=203d7e8a4987d642 server_cookie=48f888a6c5c60513 in_seq=0 out_seq=0 2024-03-19T03:01:54.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.846+0000 7f8f5e7fc640 1 -- 172.21.15.33:0/701925159 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558033d41d90 con 0x558033e57440 2024-03-19T03:01:54.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.846+0000 7f8f5e7fc640 10 client.17424 handle_command_reply: tid=0 2024-03-19T03:01:54.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.846+0000 7f8f7fde9740 1 client.17424 shutdown 2024-03-19T03:01:54.853 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.846+0000 7f8f7fde9740 1 -- 172.21.15.33:0/701925159 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558033ddff30 msgr2=0x558033de5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:54.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.846+0000 7f8f7fde9740 1 --2- 172.21.15.33:0/701925159 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558033ddff30 0x558033de5600 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f8f70009f90 tx=0x7f8f7001b4b0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.846+0000 7f8f7fde9740 1 -- 172.21.15.33:0/701925159 shutdown_connections 2024-03-19T03:01:54.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.846+0000 7f8f7fde9740 1 --2- 172.21.15.33:0/701925159 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x558033e57440 0x558033e59830 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.846+0000 7f8f7fde9740 1 --2- 172.21.15.33:0/701925159 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x558033ddff30 0x558033de5600 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.846+0000 7f8f7fde9740 1 --2- 172.21.15.33:0/701925159 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558033dd7350 0x558033de50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.846+0000 7f8f7fde9740 1 --2- 172.21.15.33:0/701925159 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x558033dd6a30 0x558033de80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.846+0000 7f8f5e7fc640 0 client.17424 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:01:54.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.846+0000 7f8f7fde9740 1 -- 172.21.15.33:0/701925159 >> 172.21.15.33:0/701925159 conn(0x558033dd1160 msgr2=0x558033dd2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:54.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.846+0000 7f8f7fde9740 1 -- 172.21.15.33:0/701925159 shutdown_connections 2024-03-19T03:01:54.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.846+0000 7f8f7fde9740 1 -- 172.21.15.33:0/701925159 wait complete. 2024-03-19T03:01:54.854 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.846+0000 7f8f7fde9740 20 client.17424 trim_cache size 0 max 16384 2024-03-19T03:01:54.855 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T03:01:54.855 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-03-19T03:01:54.855 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-03-19T03:01:54.855 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T03:01:54.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.846+0000 7f8f7d1f4640 1 -- 172.21.15.33:0/43461855 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f8f640a2970 msgr2=0x7f8f640a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:54.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.847+0000 7f8f7d1f4640 1 --2- 172.21.15.33:0/43461855 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f8f640a2970 0x7f8f640a4e30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8f70022000 tx=0x7f8f70023000 comp rx=0 tx=0).stop 2024-03-19T03:01:54.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.847+0000 7f8f7d1f4640 1 -- 172.21.15.33:0/43461855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f78101100 msgr2=0x7f8f78199410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:54.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.847+0000 7f8f7d1f4640 1 --2- 172.21.15.33:0/43461855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f78101100 0x7f8f78199410 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f8f60009230 tx=0x7f8f600045e0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.847+0000 7f8f7d1f4640 1 -- 172.21.15.33:0/43461855 shutdown_connections 2024-03-19T03:01:54.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.847+0000 7f8f7d1f4640 1 --2- 172.21.15.33:0/43461855 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f8f640a2970 0x7f8f640a4e30 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.847+0000 7f8f7d1f4640 1 --2- 172.21.15.33:0/43461855 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8f78106750 0x7f8f781a3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.847+0000 7f8f7d1f4640 1 --2- 172.21.15.33:0/43461855 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8f78101100 0x7f8f78199410 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.847+0000 7f8f7d1f4640 1 --2- 172.21.15.33:0/43461855 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f78100740 0x7f8f78198ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.847+0000 7f8f7d1f4640 1 -- 172.21.15.33:0/43461855 >> 172.21.15.33:0/43461855 conn(0x7f8f78077880 msgr2=0x7f8f78103350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:54.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.847+0000 7f8f7d1f4640 1 -- 172.21.15.33:0/43461855 shutdown_connections 2024-03-19T03:01:54.856 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.847+0000 7f8f7d1f4640 1 -- 172.21.15.33:0/43461855 wait complete. 2024-03-19T03:01:54.859 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T03:01:54.859 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:01:54.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.960+0000 7fbd4edb9640 1 -- 172.21.15.33:0/4284037633 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd48102a50 msgr2=0x7fbd48102e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:54.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.960+0000 7fbd4edb9640 1 --2- 172.21.15.33:0/4284037633 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd48102a50 0x7fbd48102e30 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbd40009860 tx=0x7fbd4002bc00 comp rx=0 tx=0).stop 2024-03-19T03:01:54.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.960+0000 7fbd4edb9640 1 -- 172.21.15.33:0/4284037633 shutdown_connections 2024-03-19T03:01:54.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.960+0000 7fbd4edb9640 1 --2- 172.21.15.33:0/4284037633 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd4810be20 0x7fbd4810e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.960+0000 7fbd4edb9640 1 --2- 172.21.15.33:0/4284037633 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbd48103370 0x7fbd4810b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.960+0000 7fbd4edb9640 1 --2- 172.21.15.33:0/4284037633 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd48102a50 0x7fbd48102e30 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.960+0000 7fbd4edb9640 1 -- 172.21.15.33:0/4284037633 >> 172.21.15.33:0/4284037633 conn(0x7fbd48076aa0 msgr2=0x7fbd48076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:54.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.960+0000 7fbd4edb9640 1 -- 172.21.15.33:0/4284037633 shutdown_connections 2024-03-19T03:01:54.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.960+0000 7fbd4edb9640 1 -- 172.21.15.33:0/4284037633 wait complete. 2024-03-19T03:01:54.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.961+0000 7fbd4edb9640 1 Processor -- start 2024-03-19T03:01:54.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.961+0000 7fbd4edb9640 1 -- start start 2024-03-19T03:01:54.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.961+0000 7fbd4edb9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbd48102a50 0x7fbd48198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:54.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.961+0000 7fbd4edb9640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd48103370 0x7fbd48198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:54.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.961+0000 7fbd4edb9640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd4810be20 0x7fbd4819c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:54.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.961+0000 7fbd4edb9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbd48111a60 con 0x7fbd48102a50 2024-03-19T03:01:54.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.961+0000 7fbd4edb9640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbd481118e0 con 0x7fbd48103370 2024-03-19T03:01:54.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.961+0000 7fbd4edb9640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbd48111be0 con 0x7fbd4810be20 2024-03-19T03:01:54.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.961+0000 7fbd4cb55640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbd48102a50 0x7fbd48198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:54.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.961+0000 7fbd4cb55640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbd48102a50 0x7fbd48198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35754/0 (socket says 172.21.15.33:35754) 2024-03-19T03:01:54.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.961+0000 7fbd4cb55640 1 -- 172.21.15.33:0/1432272748 learned_addr learned my addr 172.21.15.33:0/1432272748 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:54.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.961+0000 7fbd47fff640 1 --2- 172.21.15.33:0/1432272748 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd48103370 0x7fbd48198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:54.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.961+0000 7fbd4d356640 1 --2- 172.21.15.33:0/1432272748 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd4810be20 0x7fbd4819c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:54.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.961+0000 7fbd4cb55640 1 -- 172.21.15.33:0/1432272748 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd4810be20 msgr2=0x7fbd4819c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:54.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.961+0000 7fbd4cb55640 1 --2- 172.21.15.33:0/1432272748 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd4810be20 0x7fbd4819c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.961+0000 7fbd4cb55640 1 -- 172.21.15.33:0/1432272748 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd48103370 msgr2=0x7fbd48198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:54.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.961+0000 7fbd4cb55640 1 --2- 172.21.15.33:0/1432272748 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd48103370 0x7fbd48198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:54.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.961+0000 7fbd4cb55640 1 -- 172.21.15.33:0/1432272748 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd4819cfc0 con 0x7fbd48102a50 2024-03-19T03:01:54.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.962+0000 7fbd4cb55640 1 --2- 172.21.15.33:0/1432272748 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbd48102a50 0x7fbd48198020 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fbd40037040 tx=0x7fbd40035f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:54.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.962+0000 7fbd45ffb640 1 -- 172.21.15.33:0/1432272748 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd40041020 con 0x7fbd48102a50 2024-03-19T03:01:54.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.962+0000 7fbd4edb9640 1 -- 172.21.15.33:0/1432272748 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd4819d250 con 0x7fbd48102a50 2024-03-19T03:01:54.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.962+0000 7fbd45ffb640 1 -- 172.21.15.33:0/1432272748 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbd4002c5a0 con 0x7fbd48102a50 2024-03-19T03:01:54.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.962+0000 7fbd45ffb640 1 -- 172.21.15.33:0/1432272748 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd40039040 con 0x7fbd48102a50 2024-03-19T03:01:54.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.962+0000 7fbd4edb9640 1 -- 172.21.15.33:0/1432272748 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd481a4b40 con 0x7fbd48102a50 2024-03-19T03:01:54.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.963+0000 7fbd4edb9640 1 -- 172.21.15.33:0/1432272748 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd4806a670 con 0x7fbd48102a50 2024-03-19T03:01:54.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.964+0000 7fbd45ffb640 1 -- 172.21.15.33:0/1432272748 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fbd40044020 con 0x7fbd48102a50 2024-03-19T03:01:54.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.964+0000 7fbd45ffb640 1 --2- 172.21.15.33:0/1432272748 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbd340a2970 0x7fbd340a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:54.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.965+0000 7fbd45ffb640 1 -- 172.21.15.33:0/1432272748 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7fbd400f0720 con 0x7fbd48102a50 2024-03-19T03:01:54.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.965+0000 7fbd47fff640 1 --2- 172.21.15.33:0/1432272748 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbd340a2970 0x7fbd340a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:54.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.965+0000 7fbd47fff640 1 --2- 172.21.15.33:0/1432272748 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbd340a2970 0x7fbd340a4e30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbd38009a60 tx=0x7fbd38010040 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:54.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:54.969+0000 7fbd45ffb640 1 -- 172.21.15.33:0/1432272748 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbd4004b020 con 0x7fbd48102a50 2024-03-19T03:01:55.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.190+0000 7fbd4edb9640 1 -- 172.21.15.33:0/1432272748 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd48106f10 con 0x7fbd48102a50 2024-03-19T03:01:55.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.191+0000 7fbd45ffb640 1 -- 172.21.15.33:0/1432272748 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11867 (secure 0 0 0) 0x7fbd400bd1c0 con 0x7fbd48102a50 2024-03-19T03:01:55.192 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T03:01:55.192 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:50:52.201798+0000","modified":"2024-03-19T02:53:11.748697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24488,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/3731232169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":3731232169},{"type":"v1","addr":"172.21.15.96:6839","nonce":3731232169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24457,24488,24509]},"id":1}]} 2024-03-19T03:01:55.193 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-03-19T03:01:55.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.193+0000 7fbd4edb9640 1 -- 172.21.15.33:0/1432272748 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbd340a2970 msgr2=0x7fbd340a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:55.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.193+0000 7fbd4edb9640 1 --2- 172.21.15.33:0/1432272748 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbd340a2970 0x7fbd340a4e30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbd38009a60 tx=0x7fbd38010040 comp rx=0 tx=0).stop 2024-03-19T03:01:55.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.193+0000 7fbd4edb9640 1 -- 172.21.15.33:0/1432272748 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbd48102a50 msgr2=0x7fbd48198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:55.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.193+0000 7fbd4edb9640 1 --2- 172.21.15.33:0/1432272748 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbd48102a50 0x7fbd48198020 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fbd40037040 tx=0x7fbd40035f70 comp rx=0 tx=0).stop 2024-03-19T03:01:55.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.193+0000 7fbd4edb9640 1 -- 172.21.15.33:0/1432272748 shutdown_connections 2024-03-19T03:01:55.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.193+0000 7fbd4edb9640 1 --2- 172.21.15.33:0/1432272748 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbd340a2970 0x7fbd340a4e30 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.193+0000 7fbd4edb9640 1 --2- 172.21.15.33:0/1432272748 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd4810be20 0x7fbd4819c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.193+0000 7fbd4edb9640 1 --2- 172.21.15.33:0/1432272748 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbd48103370 0x7fbd48198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.193+0000 7fbd4edb9640 1 --2- 172.21.15.33:0/1432272748 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbd48102a50 0x7fbd48198020 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.193+0000 7fbd4edb9640 1 -- 172.21.15.33:0/1432272748 >> 172.21.15.33:0/1432272748 conn(0x7fbd48076aa0 msgr2=0x7fbd480fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:55.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.193+0000 7fbd4edb9640 1 -- 172.21.15.33:0/1432272748 shutdown_connections 2024-03-19T03:01:55.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.193+0000 7fbd4edb9640 1 -- 172.21.15.33:0/1432272748 wait complete. 2024-03-19T03:01:55.205 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T03:01:55.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.312+0000 7f7e23063640 1 -- 172.21.15.33:0/232179096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e1c0ffcf0 msgr2=0x7f7e1c100170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:55.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.312+0000 7f7e23063640 1 --2- 172.21.15.33:0/232179096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e1c0ffcf0 0x7f7e1c100170 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f7e10009930 tx=0x7f7e1002bc30 comp rx=0 tx=0).stop 2024-03-19T03:01:55.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.312+0000 7f7e23063640 1 -- 172.21.15.33:0/232179096 shutdown_connections 2024-03-19T03:01:55.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.312+0000 7f7e23063640 1 --2- 172.21.15.33:0/232179096 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e1c1006b0 0x7f7e1c10e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.312+0000 7f7e23063640 1 --2- 172.21.15.33:0/232179096 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e1c0ffcf0 0x7f7e1c100170 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.312+0000 7f7e23063640 1 --2- 172.21.15.33:0/232179096 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e1c1067d0 0x7f7e1c106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.312+0000 7f7e23063640 1 -- 172.21.15.33:0/232179096 >> 172.21.15.33:0/232179096 conn(0x7f7e1c076c30 msgr2=0x7f7e1c0fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:55.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.312+0000 7f7e23063640 1 -- 172.21.15.33:0/232179096 shutdown_connections 2024-03-19T03:01:55.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.312+0000 7f7e23063640 1 -- 172.21.15.33:0/232179096 wait complete. 2024-03-19T03:01:55.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.312+0000 7f7e23063640 1 Processor -- start 2024-03-19T03:01:55.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.312+0000 7f7e23063640 1 -- start start 2024-03-19T03:01:55.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.313+0000 7f7e23063640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e1c0ffcf0 0x7f7e1c198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.313+0000 7f7e23063640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e1c1006b0 0x7f7e1c198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.313+0000 7f7e23063640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e1c1067d0 0x7f7e1c19c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.313+0000 7f7e23063640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1c111b50 con 0x7f7e1c1006b0 2024-03-19T03:01:55.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.313+0000 7f7e23063640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1c1119d0 con 0x7f7e1c1067d0 2024-03-19T03:01:55.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.313+0000 7f7e23063640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1c111cd0 con 0x7f7e1c0ffcf0 2024-03-19T03:01:55.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.313+0000 7f7e1bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e1c1006b0 0x7f7e1c198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.313+0000 7f7e1bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e1c1006b0 0x7f7e1c198690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35774/0 (socket says 172.21.15.33:35774) 2024-03-19T03:01:55.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.313+0000 7f7e1bfff640 1 -- 172.21.15.33:0/705175663 learned_addr learned my addr 172.21.15.33:0/705175663 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:55.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.313+0000 7f7e21600640 1 --2- 172.21.15.33:0/705175663 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e1c1067d0 0x7f7e1c19c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.313+0000 7f7e20dff640 1 --2- 172.21.15.33:0/705175663 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e1c0ffcf0 0x7f7e1c198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.313+0000 7f7e1bfff640 1 -- 172.21.15.33:0/705175663 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e1c0ffcf0 msgr2=0x7f7e1c198150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:55.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.313+0000 7f7e1bfff640 1 --2- 172.21.15.33:0/705175663 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e1c0ffcf0 0x7f7e1c198150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.313+0000 7f7e1bfff640 1 -- 172.21.15.33:0/705175663 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e1c1067d0 msgr2=0x7f7e1c19c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:55.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.313+0000 7f7e1bfff640 1 --2- 172.21.15.33:0/705175663 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e1c1067d0 0x7f7e1c19c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.313+0000 7f7e1bfff640 1 -- 172.21.15.33:0/705175663 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e1c19d0f0 con 0x7f7e1c1006b0 2024-03-19T03:01:55.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.313+0000 7f7e1bfff640 1 --2- 172.21.15.33:0/705175663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e1c1006b0 0x7f7e1c198690 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f7e10009230 tx=0x7f7e1002c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:55.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.313+0000 7f7e19ffb640 1 -- 172.21.15.33:0/705175663 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e10043020 con 0x7f7e1c1006b0 2024-03-19T03:01:55.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.313+0000 7f7e23063640 1 -- 172.21.15.33:0/705175663 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e1c19d380 con 0x7f7e1c1006b0 2024-03-19T03:01:55.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.313+0000 7f7e19ffb640 1 -- 172.21.15.33:0/705175663 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e1002caf0 con 0x7f7e1c1006b0 2024-03-19T03:01:55.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.313+0000 7f7e19ffb640 1 -- 172.21.15.33:0/705175663 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e10003a50 con 0x7f7e1c1006b0 2024-03-19T03:01:55.317 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.314+0000 7f7e23063640 1 -- 172.21.15.33:0/705175663 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e1c1a4c70 con 0x7f7e1c1006b0 2024-03-19T03:01:55.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.316+0000 7f7e19ffb640 1 -- 172.21.15.33:0/705175663 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f7e10046020 con 0x7f7e1c1006b0 2024-03-19T03:01:55.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.316+0000 7f7e25c46740 1 Processor -- start 2024-03-19T03:01:55.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.316+0000 7f7e25c46740 1 -- start start 2024-03-19T03:01:55.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.316+0000 7f7e25c46740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55801bfa22d0 0x55801bfa26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.316+0000 7f7e25c46740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55801bfa2bf0 0x55801bfab290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.316+0000 7f7e25c46740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55801bfab7d0 0x55801bfadbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.316+0000 7f7e25c46740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55801bf054b0 con 0x55801bfa22d0 2024-03-19T03:01:55.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.316+0000 7f7e25c46740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55801beadd70 con 0x55801bfa2bf0 2024-03-19T03:01:55.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.316+0000 7f7e25c46740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55801bf01550 con 0x55801bfab7d0 2024-03-19T03:01:55.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.316+0000 7f7e19ffb640 1 --2- 172.21.15.33:0/705175663 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f7e040a2970 0x7f7e040a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.324 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.317+0000 7f7e21600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55801bfa22d0 0x55801bfa26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.317+0000 7f7e20dff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55801bfa2bf0 0x55801bfab290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.317+0000 7f7e21600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55801bfa22d0 0x55801bfa26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35776/0 (socket says 172.21.15.33:35776) 2024-03-19T03:01:55.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.317+0000 7f7e21600640 1 -- 172.21.15.33:0/600743400 learned_addr learned my addr 172.21.15.33:0/600743400 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:55.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.317+0000 7f7e1bfff640 1 --2- 172.21.15.33:0/600743400 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55801bfab7d0 0x55801bfadbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.317+0000 7f7e21600640 1 --2- 172.21.15.33:0/705175663 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f7e040a2970 0x7f7e040a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.317+0000 7f7e19ffb640 1 -- 172.21.15.33:0/705175663 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f7e100f1170 con 0x7f7e1c1006b0 2024-03-19T03:01:55.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.317+0000 7f7e20dff640 1 -- 172.21.15.33:0/600743400 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55801bfab7d0 msgr2=0x55801bfadbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:55.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.317+0000 7f7e20dff640 1 --2- 172.21.15.33:0/600743400 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55801bfab7d0 0x55801bfadbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.317+0000 7f7e20dff640 1 -- 172.21.15.33:0/600743400 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55801bfa22d0 msgr2=0x55801bfa26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:55.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.317+0000 7f7e20dff640 1 --2- 172.21.15.33:0/600743400 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55801bfa22d0 0x55801bfa26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.317+0000 7f7e20dff640 1 -- 172.21.15.33:0/600743400 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55801bed0210 con 0x55801bfa2bf0 2024-03-19T03:01:55.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.317+0000 7f7e20dff640 1 --2- 172.21.15.33:0/600743400 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55801bfa2bf0 0x55801bfab290 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c005fd0 tx=0x7f7e0c0107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b11f40be3140312 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:55.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.317+0000 7f7e21600640 1 --2- 172.21.15.33:0/705175663 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f7e040a2970 0x7f7e040a4e30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7e14005c40 tx=0x7f7e14009500 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:55.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.317+0000 7f7e02ffd640 1 -- 172.21.15.33:0/600743400 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e0c01c480 con 0x55801bfa2bf0 2024-03-19T03:01:55.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.317+0000 7f7e02ffd640 1 -- 172.21.15.33:0/600743400 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e0c01cad0 con 0x55801bfa2bf0 2024-03-19T03:01:55.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.317+0000 7f7e02ffd640 1 -- 172.21.15.33:0/600743400 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e0c011a40 con 0x55801bfa2bf0 2024-03-19T03:01:55.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.318+0000 7f7e25c46740 1 -- 172.21.15.33:0/600743400 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55801bfa2bf0 msgr2=0x55801bfab290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:55.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.318+0000 7f7e25c46740 1 --2- 172.21.15.33:0/600743400 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55801bfa2bf0 0x55801bfab290 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c005fd0 tx=0x7f7e0c0107a0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.318+0000 7f7e25c46740 1 -- 172.21.15.33:0/600743400 shutdown_connections 2024-03-19T03:01:55.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.318+0000 7f7e25c46740 1 --2- 172.21.15.33:0/600743400 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55801bfab7d0 0x55801bfadbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.318+0000 7f7e25c46740 1 --2- 172.21.15.33:0/600743400 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55801bfa2bf0 0x55801bfab290 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.318+0000 7f7e25c46740 1 --2- 172.21.15.33:0/600743400 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55801bfa22d0 0x55801bfa26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.318+0000 7f7e25c46740 1 -- 172.21.15.33:0/600743400 >> 172.21.15.33:0/600743400 conn(0x55801bf9ca00 msgr2=0x55801bf9cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:55.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.318+0000 7f7e25c46740 1 -- 172.21.15.33:0/600743400 shutdown_connections 2024-03-19T03:01:55.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.318+0000 7f7e25c46740 1 -- 172.21.15.33:0/600743400 wait complete. 2024-03-19T03:01:55.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.318+0000 7f7e25c46740 1 -- start start 2024-03-19T03:01:55.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.318+0000 7f7e25c46740 1 Processor -- start 2024-03-19T03:01:55.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.319+0000 7f7e25c46740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55801bfa22d0 0x55801bfa87d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.319+0000 7f7e25c46740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55801bfa2bf0 0x55801bfa8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.319+0000 7f7e25c46740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55801bfab7d0 0x55801bfa9250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.319+0000 7f7e25c46740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55801bfae5c0 con 0x55801bfa2bf0 2024-03-19T03:01:55.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.319+0000 7f7e25c46740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55801bec1310 con 0x55801bfa22d0 2024-03-19T03:01:55.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.319+0000 7f7e25c46740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55801bfae740 con 0x55801bfab7d0 2024-03-19T03:01:55.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.319+0000 7f7e25c46740 10 client.0 _ll_register_callbacks cb 0x55801beb52d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:01:55.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.319+0000 7f7e1bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55801bfa2bf0 0x55801bfa8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.319+0000 7f7e1bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55801bfa2bf0 0x55801bfa8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35786/0 (socket says 172.21.15.33:35786) 2024-03-19T03:01:55.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.319+0000 7f7e20dff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55801bfa22d0 0x55801bfa87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.319+0000 7f7e1bfff640 1 -- 172.21.15.33:0/1150005668 learned_addr learned my addr 172.21.15.33:0/1150005668 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:55.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.319+0000 7f7e21600640 1 --2- 172.21.15.33:0/1150005668 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55801bfab7d0 0x55801bfa9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.319+0000 7f7e1bfff640 1 -- 172.21.15.33:0/1150005668 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55801bfab7d0 msgr2=0x55801bfa9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:55.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.319+0000 7f7e1bfff640 1 --2- 172.21.15.33:0/1150005668 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55801bfab7d0 0x55801bfa9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.319+0000 7f7e1bfff640 1 -- 172.21.15.33:0/1150005668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55801bfa22d0 msgr2=0x55801bfa87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:55.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.319+0000 7f7e1bfff640 1 --2- 172.21.15.33:0/1150005668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55801bfa22d0 0x55801bfa87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.319+0000 7f7e1bfff640 1 -- 172.21.15.33:0/1150005668 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55801bc193f0 con 0x55801bfa2bf0 2024-03-19T03:01:55.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.319+0000 7f7e20dff640 1 --2- 172.21.15.33:0/1150005668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55801bfa22d0 0x55801bfa87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:55.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.319+0000 7f7e1bfff640 1 --2- 172.21.15.33:0/1150005668 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55801bfa2bf0 0x55801bfa8d10 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f7e100b9970 tx=0x7f7e1008cfe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c69932c5cd607d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:55.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.319+0000 7f7dfbfff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:01:55.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.320+0000 7f7dfbfff640 1 -- 172.21.15.33:0/1150005668 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e100bd050 con 0x55801bfa2bf0 2024-03-19T03:01:55.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.320+0000 7f7dfbfff640 1 -- 172.21.15.33:0/1150005668 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f7e100053b0 con 0x55801bfa2bf0 2024-03-19T03:01:55.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.320+0000 7f7e25c46740 1 -- 172.21.15.33:0/1150005668 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e0c01cad0 con 0x55801bfa2bf0 2024-03-19T03:01:55.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.320+0000 7f7dfbfff640 1 -- 172.21.15.33:0/1150005668 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7e1002cca0 con 0x55801bfa2bf0 2024-03-19T03:01:55.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.320+0000 7f7e25c46740 10 client.17442 fetch_fsmap learned FSMap version 25 2024-03-19T03:01:55.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.320+0000 7f7e25c46740 1 -- 172.21.15.33:0/1150005668 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55801bce37d0 con 0x55801bfa2bf0 2024-03-19T03:01:55.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.320+0000 7f7dfbfff640 1 -- 172.21.15.33:0/1150005668 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e0c01cad0 con 0x55801bfa2bf0 2024-03-19T03:01:55.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.320+0000 7f7dfbfff640 1 -- 172.21.15.33:0/1150005668 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f7e100bd770 con 0x55801bfa2bf0 2024-03-19T03:01:55.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.320+0000 7f7e25c46740 10 client.17442 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:01:55.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.320+0000 7f7e25c46740 10 client.17442 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T03:01:55.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.320+0000 7f7e25c46740 20 client.17442 populate_metadata read hostname 'smithi033' 2024-03-19T03:01:55.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.320+0000 7f7e25c46740 1 --2- 172.21.15.33:0/1150005668 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55801bfbb9a0 0x55801bfbddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.320+0000 7f7e25c46740 4 client.17442 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:01:55.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.320+0000 7f7e25c46740 1 -- 172.21.15.33:0/1150005668 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55801bf633e0 con 0x55801bfbb9a0 2024-03-19T03:01:55.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.320+0000 7f7e20dff640 1 --2- 172.21.15.33:0/1150005668 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55801bfbb9a0 0x55801bfbddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.321+0000 7f7dfbfff640 10 client.17442 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T03:01:55.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.321+0000 7f7e20dff640 1 --2- 172.21.15.33:0/1150005668 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55801bfbb9a0 0x55801bfbddf0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9e897a4e2d6d8d server_cookie=3d632cb2fe8b4702 in_seq=0 out_seq=0 2024-03-19T03:01:55.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.321+0000 7f7dfbfff640 1 -- 172.21.15.33:0/1150005668 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55801bf633e0 con 0x55801bfbb9a0 2024-03-19T03:01:55.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.321+0000 7f7dfbfff640 10 client.17442 handle_command_reply: tid=0 2024-03-19T03:01:55.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.321+0000 7f7e25c46740 1 client.17442 shutdown 2024-03-19T03:01:55.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.321+0000 7f7e25c46740 1 -- 172.21.15.33:0/1150005668 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55801bfa2bf0 msgr2=0x55801bfa8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:55.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.321+0000 7f7e25c46740 1 --2- 172.21.15.33:0/1150005668 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55801bfa2bf0 0x55801bfa8d10 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f7e100b9970 tx=0x7f7e1008cfe0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.322+0000 7f7e25c46740 1 -- 172.21.15.33:0/1150005668 shutdown_connections 2024-03-19T03:01:55.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.322+0000 7f7e25c46740 1 --2- 172.21.15.33:0/1150005668 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55801bfbb9a0 0x55801bfbddf0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.322+0000 7f7e25c46740 1 --2- 172.21.15.33:0/1150005668 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55801bfab7d0 0x55801bfa9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.322+0000 7f7e25c46740 1 --2- 172.21.15.33:0/1150005668 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55801bfa2bf0 0x55801bfa8d10 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.322+0000 7f7dfbfff640 0 client.17442 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T03:01:55.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.322+0000 7f7e25c46740 1 --2- 172.21.15.33:0/1150005668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55801bfa22d0 0x55801bfa87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.322+0000 7f7e25c46740 1 -- 172.21.15.33:0/1150005668 >> 172.21.15.33:0/1150005668 conn(0x55801bf9ca00 msgr2=0x55801bf9e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:55.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.322+0000 7f7e25c46740 1 -- 172.21.15.33:0/1150005668 shutdown_connections 2024-03-19T03:01:55.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.322+0000 7f7e25c46740 1 -- 172.21.15.33:0/1150005668 wait complete. 2024-03-19T03:01:55.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.322+0000 7f7e25c46740 20 client.17442 trim_cache size 0 max 16384 2024-03-19T03:01:55.341 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.335+0000 7f7e25c46740 1 Processor -- start 2024-03-19T03:01:55.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.335+0000 7f7e25c46740 1 -- start start 2024-03-19T03:01:55.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.335+0000 7f7e25c46740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55801bfa22d0 0x55801bfaaf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.335+0000 7f7e25c46740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55801bfa2bf0 0x55801bfb7360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.335+0000 7f7e25c46740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55801bfab7d0 0x55801bf9e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.335+0000 7f7e25c46740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1c1b15c0 con 0x55801bfa2bf0 2024-03-19T03:01:55.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.335+0000 7f7e25c46740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1c1aee90 con 0x55801bfab7d0 2024-03-19T03:01:55.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.335+0000 7f7e25c46740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1c1acd00 con 0x55801bfa22d0 2024-03-19T03:01:55.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.335+0000 7f7e1bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55801bfa2bf0 0x55801bfb7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.335+0000 7f7e1bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55801bfa2bf0 0x55801bfb7360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35790/0 (socket says 172.21.15.33:35790) 2024-03-19T03:01:55.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.335+0000 7f7e1bfff640 1 -- 172.21.15.33:0/2064106703 learned_addr learned my addr 172.21.15.33:0/2064106703 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:55.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.335+0000 7f7e21600640 1 --2- 172.21.15.33:0/2064106703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55801bfab7d0 0x55801bf9e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.335+0000 7f7e20dff640 1 --2- 172.21.15.33:0/2064106703 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55801bfa22d0 0x55801bfaaf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.335+0000 7f7e1bfff640 1 -- 172.21.15.33:0/2064106703 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55801bfa22d0 msgr2=0x55801bfaaf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:55.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.335+0000 7f7e1bfff640 1 --2- 172.21.15.33:0/2064106703 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55801bfa22d0 0x55801bfaaf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.335+0000 7f7e1bfff640 1 -- 172.21.15.33:0/2064106703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55801bfab7d0 msgr2=0x55801bf9e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:55.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.335+0000 7f7e1bfff640 1 --2- 172.21.15.33:0/2064106703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55801bfab7d0 0x55801bf9e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.335+0000 7f7e1bfff640 1 -- 172.21.15.33:0/2064106703 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55801bfa7780 con 0x55801bfa2bf0 2024-03-19T03:01:55.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.335+0000 7f7e21600640 1 --2- 172.21.15.33:0/2064106703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55801bfab7d0 0x55801bf9e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:01:55.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.335+0000 7f7e20dff640 1 --2- 172.21.15.33:0/2064106703 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55801bfa22d0 0x55801bfaaf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:01:55.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.335+0000 7f7e1bfff640 1 --2- 172.21.15.33:0/2064106703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55801bfa2bf0 0x55801bfb7360 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f7e10005e90 tx=0x7f7e10044fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8a7e923ecda2eb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:55.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.336+0000 7f7e027fc640 1 -- 172.21.15.33:0/2064106703 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e100da070 con 0x55801bfa2bf0 2024-03-19T03:01:55.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.336+0000 7f7e027fc640 1 -- 172.21.15.33:0/2064106703 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e100bdc10 con 0x55801bfa2bf0 2024-03-19T03:01:55.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.336+0000 7f7e027fc640 1 -- 172.21.15.33:0/2064106703 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e100bce70 con 0x55801bfa2bf0 2024-03-19T03:01:55.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.336+0000 7f7e25c46740 1 -- 172.21.15.33:0/2064106703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55801bfa2bf0 msgr2=0x55801bfb7360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:55.343 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.336+0000 7f7e25c46740 1 --2- 172.21.15.33:0/2064106703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55801bfa2bf0 0x55801bfb7360 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f7e10005e90 tx=0x7f7e10044fd0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.336+0000 7f7e25c46740 1 -- 172.21.15.33:0/2064106703 shutdown_connections 2024-03-19T03:01:55.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.336+0000 7f7e25c46740 1 --2- 172.21.15.33:0/2064106703 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55801bfab7d0 0x55801bf9e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.336+0000 7f7e25c46740 1 --2- 172.21.15.33:0/2064106703 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55801bfa2bf0 0x55801bfb7360 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.336+0000 7f7e25c46740 1 --2- 172.21.15.33:0/2064106703 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55801bfa22d0 0x55801bfaaf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.336+0000 7f7e25c46740 1 -- 172.21.15.33:0/2064106703 >> 172.21.15.33:0/2064106703 conn(0x55801bf9ca00 msgr2=0x55801bf9cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:55.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.336+0000 7f7e25c46740 1 -- 172.21.15.33:0/2064106703 shutdown_connections 2024-03-19T03:01:55.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.336+0000 7f7e25c46740 1 -- 172.21.15.33:0/2064106703 wait complete. 2024-03-19T03:01:55.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.336+0000 7f7e25c46740 1 -- start start 2024-03-19T03:01:55.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.337+0000 7f7e25c46740 1 Processor -- start 2024-03-19T03:01:55.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.337+0000 7f7e25c46740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55801bfa22d0 0x55801bfafa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.337+0000 7f7e25c46740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55801bfa2bf0 0x55801bfaffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.337+0000 7f7e25c46740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55801bfab7d0 0x55801c020d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.337+0000 7f7e25c46740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1c1ad350 con 0x55801bfab7d0 2024-03-19T03:01:55.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.337+0000 7f7e25c46740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1c1ae7c0 con 0x55801bfa22d0 2024-03-19T03:01:55.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.337+0000 7f7e25c46740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1c1ada20 con 0x55801bfa2bf0 2024-03-19T03:01:55.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.337+0000 7f7e25c46740 10 client.0 _ll_register_callbacks cb 0x7f7e10009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:01:55.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.337+0000 7f7e21600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55801bfab7d0 0x55801c020d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.337+0000 7f7e21600640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55801bfab7d0 0x55801c020d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35796/0 (socket says 172.21.15.33:35796) 2024-03-19T03:01:55.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.337+0000 7f7e21600640 1 -- 172.21.15.33:0/115071224 learned_addr learned my addr 172.21.15.33:0/115071224 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:55.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.337+0000 7f7e20dff640 1 --2- 172.21.15.33:0/115071224 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55801bfa22d0 0x55801bfafa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.337+0000 7f7e1bfff640 1 --2- 172.21.15.33:0/115071224 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55801bfa2bf0 0x55801bfaffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.337+0000 7f7e21600640 1 -- 172.21.15.33:0/115071224 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55801bfa2bf0 msgr2=0x55801bfaffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:55.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.337+0000 7f7e21600640 1 --2- 172.21.15.33:0/115071224 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55801bfa2bf0 0x55801bfaffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.337+0000 7f7e21600640 1 -- 172.21.15.33:0/115071224 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55801bfa22d0 msgr2=0x55801bfafa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:55.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.337+0000 7f7e21600640 1 --2- 172.21.15.33:0/115071224 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55801bfa22d0 0x55801bfafa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.337+0000 7f7e21600640 1 -- 172.21.15.33:0/115071224 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55801bfa2a20 con 0x55801bfab7d0 2024-03-19T03:01:55.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.337+0000 7f7e1bfff640 1 --2- 172.21.15.33:0/115071224 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55801bfa2bf0 0x55801bfaffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:01:55.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.337+0000 7f7e21600640 1 --2- 172.21.15.33:0/115071224 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55801bfab7d0 0x55801c020d30 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f7e14003b60 tx=0x7f7e14004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b977714dbb97f35 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:55.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.337+0000 7f7dfbfff640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:01:55.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.338+0000 7f7dfbfff640 1 -- 172.21.15.33:0/115071224 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e14004d20 con 0x55801bfab7d0 2024-03-19T03:01:55.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.338+0000 7f7dfbfff640 1 -- 172.21.15.33:0/115071224 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f7e14005020 con 0x55801bfab7d0 2024-03-19T03:01:55.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.338+0000 7f7e25c46740 1 -- 172.21.15.33:0/115071224 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e100bdc10 con 0x55801bfab7d0 2024-03-19T03:01:55.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.338+0000 7f7dfbfff640 1 -- 172.21.15.33:0/115071224 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7e14005220 con 0x55801bfab7d0 2024-03-19T03:01:55.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.338+0000 7f7e25c46740 10 client.17448 fetch_fsmap learned FSMap version 25 2024-03-19T03:01:55.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.338+0000 7f7e25c46740 1 -- 172.21.15.33:0/115071224 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55801c022490 con 0x55801bfab7d0 2024-03-19T03:01:55.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.338+0000 7f7dfbfff640 1 -- 172.21.15.33:0/115071224 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e100bdc10 con 0x55801bfab7d0 2024-03-19T03:01:55.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.338+0000 7f7dfbfff640 1 -- 172.21.15.33:0/115071224 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f7e1401d040 con 0x55801bfab7d0 2024-03-19T03:01:55.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.338+0000 7f7e25c46740 10 client.17448 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:01:55.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.338+0000 7f7e25c46740 10 client.17448 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T03:01:55.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.338+0000 7f7e25c46740 20 client.17448 populate_metadata read hostname 'smithi033' 2024-03-19T03:01:55.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.338+0000 7f7e25c46740 1 --2- 172.21.15.33:0/115071224 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55801c022c60 0x55801c025080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.338+0000 7f7e25c46740 4 client.17448 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T03:01:55.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.338+0000 7f7e25c46740 1 -- 172.21.15.33:0/115071224 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55801bf129f0 con 0x55801c022c60 2024-03-19T03:01:55.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.338+0000 7f7e20dff640 1 --2- 172.21.15.33:0/115071224 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55801c022c60 0x55801c025080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.339+0000 7f7e20dff640 1 --2- 172.21.15.33:0/115071224 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55801c022c60 0x55801c025080 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1221d6170e55fbe6 server_cookie=ce284cc436f1e09c in_seq=0 out_seq=0 2024-03-19T03:01:55.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.339+0000 7f7dfbfff640 10 client.17448 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T03:01:55.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.339+0000 7f7dfbfff640 1 -- 172.21.15.33:0/115071224 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55801bf129f0 con 0x55801c022c60 2024-03-19T03:01:55.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.339+0000 7f7dfbfff640 10 client.17448 handle_command_reply: tid=0 2024-03-19T03:01:55.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.339+0000 7f7e25c46740 1 client.17448 shutdown 2024-03-19T03:01:55.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.339+0000 7f7e25c46740 1 -- 172.21.15.33:0/115071224 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55801bfab7d0 msgr2=0x55801c020d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:55.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.339+0000 7f7e25c46740 1 --2- 172.21.15.33:0/115071224 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55801bfab7d0 0x55801c020d30 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f7e14003b60 tx=0x7f7e14004a40 comp rx=0 tx=0).stop 2024-03-19T03:01:55.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.339+0000 7f7e25c46740 1 -- 172.21.15.33:0/115071224 shutdown_connections 2024-03-19T03:01:55.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.339+0000 7f7e25c46740 1 --2- 172.21.15.33:0/115071224 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55801c022c60 0x55801c025080 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.339+0000 7f7e25c46740 1 --2- 172.21.15.33:0/115071224 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55801bfa2bf0 0x55801bfaffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.339+0000 7f7e25c46740 1 --2- 172.21.15.33:0/115071224 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55801bfab7d0 0x55801c020d30 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.339+0000 7f7e25c46740 1 --2- 172.21.15.33:0/115071224 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55801bfa22d0 0x55801bfafa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.339+0000 7f7dfbfff640 0 client.17448 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T03:01:55.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.339+0000 7f7e25c46740 1 -- 172.21.15.33:0/115071224 >> 172.21.15.33:0/115071224 conn(0x55801bf9ca00 msgr2=0x55801bf9bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:55.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.339+0000 7f7e25c46740 1 -- 172.21.15.33:0/115071224 shutdown_connections 2024-03-19T03:01:55.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.339+0000 7f7e25c46740 1 -- 172.21.15.33:0/115071224 wait complete. 2024-03-19T03:01:55.348 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.339+0000 7f7e25c46740 20 client.17448 trim_cache size 0 max 16384 2024-03-19T03:01:55.348 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T03:01:55.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.340+0000 7f7e23063640 1 -- 172.21.15.33:0/705175663 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f7e040a2970 msgr2=0x7f7e040a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:55.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.340+0000 7f7e23063640 1 --2- 172.21.15.33:0/705175663 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f7e040a2970 0x7f7e040a4e30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7e14005c40 tx=0x7f7e14009500 comp rx=0 tx=0).stop 2024-03-19T03:01:55.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.340+0000 7f7e23063640 1 -- 172.21.15.33:0/705175663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e1c1006b0 msgr2=0x7f7e1c198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:55.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.340+0000 7f7e23063640 1 --2- 172.21.15.33:0/705175663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e1c1006b0 0x7f7e1c198690 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f7e10009230 tx=0x7f7e1002c710 comp rx=0 tx=0).stop 2024-03-19T03:01:55.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.340+0000 7f7e23063640 1 -- 172.21.15.33:0/705175663 shutdown_connections 2024-03-19T03:01:55.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.340+0000 7f7e23063640 1 --2- 172.21.15.33:0/705175663 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f7e040a2970 0x7f7e040a4e30 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.340+0000 7f7e23063640 1 --2- 172.21.15.33:0/705175663 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7e1c1067d0 0x7f7e1c19c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.340+0000 7f7e23063640 1 --2- 172.21.15.33:0/705175663 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7e1c1006b0 0x7f7e1c198690 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.340+0000 7f7e23063640 1 --2- 172.21.15.33:0/705175663 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e1c0ffcf0 0x7f7e1c198150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.340+0000 7f7e23063640 1 -- 172.21.15.33:0/705175663 >> 172.21.15.33:0/705175663 conn(0x7f7e1c076c30 msgr2=0x7f7e1c104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:55.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.340+0000 7f7e23063640 1 -- 172.21.15.33:0/705175663 shutdown_connections 2024-03-19T03:01:55.349 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.340+0000 7f7e23063640 1 -- 172.21.15.33:0/705175663 wait complete. 2024-03-19T03:01:55.352 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T03:01:55.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.455+0000 7f91d9b59640 1 -- 172.21.15.33:0/1631705397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91d4102bb0 msgr2=0x7f91d4100260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:55.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.455+0000 7f91d9b59640 1 --2- 172.21.15.33:0/1631705397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91d4102bb0 0x7f91d4100260 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f91c8009900 tx=0x7f91c802bc30 comp rx=0 tx=0).stop 2024-03-19T03:01:55.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.455+0000 7f91d9b59640 1 -- 172.21.15.33:0/1631705397 shutdown_connections 2024-03-19T03:01:55.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.455+0000 7f91d9b59640 1 --2- 172.21.15.33:0/1631705397 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f91d410e4a0 0x7f91d4110890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.456+0000 7f91d9b59640 1 --2- 172.21.15.33:0/1631705397 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91d4102bb0 0x7f91d4100260 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.456+0000 7f91d9b59640 1 --2- 172.21.15.33:0/1631705397 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91d4102740 0x7f91d40ffd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.456+0000 7f91d9b59640 1 -- 172.21.15.33:0/1631705397 >> 172.21.15.33:0/1631705397 conn(0x7f91d4076c50 msgr2=0x7f91d40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:55.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.456+0000 7f91d9b59640 1 -- 172.21.15.33:0/1631705397 shutdown_connections 2024-03-19T03:01:55.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.456+0000 7f91d9b59640 1 -- 172.21.15.33:0/1631705397 wait complete. 2024-03-19T03:01:55.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.456+0000 7f91d9b59640 1 Processor -- start 2024-03-19T03:01:55.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.456+0000 7f91d9b59640 1 -- start start 2024-03-19T03:01:55.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.456+0000 7f91d9b59640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91d4102740 0x7f91d419a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.456+0000 7f91d9b59640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91d4102bb0 0x7f91d419a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.456+0000 7f91d9b59640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f91d410e4a0 0x7f91d419eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.456+0000 7f91d9b59640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f91d4113cd0 con 0x7f91d4102bb0 2024-03-19T03:01:55.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.456+0000 7f91d9b59640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f91d4113b50 con 0x7f91d4102740 2024-03-19T03:01:55.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.456+0000 7f91d9b59640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f91d4113e50 con 0x7f91d410e4a0 2024-03-19T03:01:55.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.457+0000 7f91d2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91d4102bb0 0x7f91d419a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.457+0000 7f91d37fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91d4102740 0x7f91d419a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.457+0000 7f91d2ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91d4102bb0 0x7f91d419a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35816/0 (socket says 172.21.15.33:35816) 2024-03-19T03:01:55.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.457+0000 7f91d2ffd640 1 -- 172.21.15.33:0/2515724504 learned_addr learned my addr 172.21.15.33:0/2515724504 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:55.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.457+0000 7f91d3fff640 1 --2- 172.21.15.33:0/2515724504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f91d410e4a0 0x7f91d419eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.457+0000 7f91d2ffd640 1 -- 172.21.15.33:0/2515724504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f91d410e4a0 msgr2=0x7f91d419eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:55.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.457+0000 7f91d2ffd640 1 --2- 172.21.15.33:0/2515724504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f91d410e4a0 0x7f91d419eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.457+0000 7f91d2ffd640 1 -- 172.21.15.33:0/2515724504 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91d4102740 msgr2=0x7f91d419a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:55.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.457+0000 7f91d2ffd640 1 --2- 172.21.15.33:0/2515724504 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91d4102740 0x7f91d419a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.457+0000 7f91d2ffd640 1 -- 172.21.15.33:0/2515724504 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91d419f240 con 0x7f91d4102bb0 2024-03-19T03:01:55.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.457+0000 7f91d2ffd640 1 --2- 172.21.15.33:0/2515724504 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91d4102bb0 0x7f91d419a790 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f91c8036040 tx=0x7f91c80045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:55.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.457+0000 7f91d0ff9640 1 -- 172.21.15.33:0/2515724504 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c8043020 con 0x7f91d4102bb0 2024-03-19T03:01:55.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.457+0000 7f91d9b59640 1 -- 172.21.15.33:0/2515724504 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91d419f4d0 con 0x7f91d4102bb0 2024-03-19T03:01:55.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.457+0000 7f91d0ff9640 1 -- 172.21.15.33:0/2515724504 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f91c802ca70 con 0x7f91d4102bb0 2024-03-19T03:01:55.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.457+0000 7f91d0ff9640 1 -- 172.21.15.33:0/2515724504 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c8003ae0 con 0x7f91d4102bb0 2024-03-19T03:01:55.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.457+0000 7f91d9b59640 1 -- 172.21.15.33:0/2515724504 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91d41a6dc0 con 0x7f91d4102bb0 2024-03-19T03:01:55.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.459+0000 7f91d0ff9640 1 -- 172.21.15.33:0/2515724504 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f91c8046020 con 0x7f91d4102bb0 2024-03-19T03:01:55.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.460+0000 7f91dc814740 1 Processor -- start 2024-03-19T03:01:55.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.460+0000 7f91dc814740 1 -- start start 2024-03-19T03:01:55.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.460+0000 7f91d0ff9640 1 --2- 172.21.15.33:0/2515724504 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f91bc0a2970 0x7f91bc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.460+0000 7f91d0ff9640 1 -- 172.21.15.33:0/2515724504 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f91c80f11f0 con 0x7f91d4102bb0 2024-03-19T03:01:55.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.460+0000 7f91d3fff640 1 --2- 172.21.15.33:0/2515724504 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f91bc0a2970 0x7f91bc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.460+0000 7f91dc814740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be7ae872d0 0x55be7ae876b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.460+0000 7f91dc814740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be7ae87bf0 0x55be7ae90290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.460+0000 7f91dc814740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7ae907d0 0x55be7ae92bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.460+0000 7f91dc814740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55be7adea4b0 con 0x55be7ae872d0 2024-03-19T03:01:55.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.460+0000 7f91dc814740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55be7ad92d70 con 0x55be7ae87bf0 2024-03-19T03:01:55.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.460+0000 7f91dc814740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55be7ade6550 con 0x55be7ae907d0 2024-03-19T03:01:55.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.460+0000 7f91d37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be7ae872d0 0x55be7ae876b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.460+0000 7f91d37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be7ae872d0 0x55be7ae876b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35830/0 (socket says 172.21.15.33:35830) 2024-03-19T03:01:55.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.460+0000 7f91d37fe640 1 -- 172.21.15.33:0/1679326998 learned_addr learned my addr 172.21.15.33:0/1679326998 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:55.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.460+0000 7f91d2ffd640 1 --2- 172.21.15.33:0/1679326998 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be7ae87bf0 0x55be7ae90290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.460+0000 7f91d3fff640 1 --2- 172.21.15.33:0/1679326998 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7ae907d0 0x55be7ae92bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.461+0000 7f91d37fe640 1 -- 172.21.15.33:0/1679326998 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7ae907d0 msgr2=0x55be7ae92bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:55.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.461+0000 7f91d37fe640 1 --2- 172.21.15.33:0/1679326998 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7ae907d0 0x55be7ae92bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.461+0000 7f91d37fe640 1 -- 172.21.15.33:0/1679326998 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be7ae87bf0 msgr2=0x55be7ae90290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:55.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.461+0000 7f91d37fe640 1 --2- 172.21.15.33:0/1679326998 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be7ae87bf0 0x55be7ae90290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.461+0000 7f91d37fe640 1 -- 172.21.15.33:0/1679326998 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be7adb5210 con 0x55be7ae872d0 2024-03-19T03:01:55.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.461+0000 7f91d3fff640 1 --2- 172.21.15.33:0/1679326998 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7ae907d0 0x55be7ae92bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:01:55.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.461+0000 7f91d3fff640 1 --2- 172.21.15.33:0/2515724504 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f91bc0a2970 0x7f91bc0a4e30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f91d4073fa0 tx=0x7f91cc011810 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:55.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.461+0000 7f91d37fe640 1 --2- 172.21.15.33:0/1679326998 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be7ae872d0 0x55be7ae876b0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f91c4009cf0 tx=0x7f91c4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad923267fc10894b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:55.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.461+0000 7f91b9ffb640 1 -- 172.21.15.33:0/1679326998 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c401e070 con 0x55be7ae872d0 2024-03-19T03:01:55.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.461+0000 7f91b9ffb640 1 -- 172.21.15.33:0/1679326998 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f91c4019ae0 con 0x55be7ae872d0 2024-03-19T03:01:55.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.461+0000 7f91b9ffb640 1 -- 172.21.15.33:0/1679326998 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c400e930 con 0x55be7ae872d0 2024-03-19T03:01:55.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.461+0000 7f91dc814740 1 -- 172.21.15.33:0/1679326998 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be7ae872d0 msgr2=0x55be7ae876b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:55.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.461+0000 7f91dc814740 1 --2- 172.21.15.33:0/1679326998 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be7ae872d0 0x55be7ae876b0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f91c4009cf0 tx=0x7f91c4011810 comp rx=0 tx=0).stop 2024-03-19T03:01:55.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.462+0000 7f91dc814740 1 -- 172.21.15.33:0/1679326998 shutdown_connections 2024-03-19T03:01:55.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.462+0000 7f91dc814740 1 --2- 172.21.15.33:0/1679326998 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7ae907d0 0x55be7ae92bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.462+0000 7f91dc814740 1 --2- 172.21.15.33:0/1679326998 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be7ae87bf0 0x55be7ae90290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.462+0000 7f91dc814740 1 --2- 172.21.15.33:0/1679326998 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be7ae872d0 0x55be7ae876b0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.462+0000 7f91dc814740 1 -- 172.21.15.33:0/1679326998 >> 172.21.15.33:0/1679326998 conn(0x55be7ae81a00 msgr2=0x55be7ae81de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:55.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.462+0000 7f91dc814740 1 -- 172.21.15.33:0/1679326998 shutdown_connections 2024-03-19T03:01:55.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.462+0000 7f91dc814740 1 -- 172.21.15.33:0/1679326998 wait complete. 2024-03-19T03:01:55.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.462+0000 7f91dc814740 1 -- start start 2024-03-19T03:01:55.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.462+0000 7f91dc814740 1 Processor -- start 2024-03-19T03:01:55.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.462+0000 7f91dc814740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be7ae872d0 0x55be7ae8d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.462+0000 7f91dc814740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7ae87bf0 0x55be7ae8dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.462+0000 7f91dc814740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be7ae907d0 0x55be7ae8e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.462+0000 7f91dc814740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55be7ae935c0 con 0x55be7ae872d0 2024-03-19T03:01:55.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.462+0000 7f91dc814740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55be7ada6310 con 0x55be7ae907d0 2024-03-19T03:01:55.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.462+0000 7f91dc814740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55be7ae93740 con 0x55be7ae87bf0 2024-03-19T03:01:55.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.462+0000 7f91dc814740 10 client.0 _ll_register_callbacks cb 0x55be7ad9a2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:01:55.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.462+0000 7f91d37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be7ae872d0 0x55be7ae8d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.462+0000 7f91d37fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be7ae872d0 0x55be7ae8d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35832/0 (socket says 172.21.15.33:35832) 2024-03-19T03:01:55.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.462+0000 7f91d37fe640 1 -- 172.21.15.33:0/178132433 learned_addr learned my addr 172.21.15.33:0/178132433 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:55.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.463+0000 7f91d2ffd640 1 --2- 172.21.15.33:0/178132433 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7ae87bf0 0x55be7ae8dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.463+0000 7f91d3fff640 1 --2- 172.21.15.33:0/178132433 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be7ae907d0 0x55be7ae8e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.463+0000 7f91d37fe640 1 -- 172.21.15.33:0/178132433 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7ae87bf0 msgr2=0x55be7ae8dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:55.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.463+0000 7f91d37fe640 1 --2- 172.21.15.33:0/178132433 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7ae87bf0 0x55be7ae8dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.463+0000 7f91d37fe640 1 -- 172.21.15.33:0/178132433 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be7ae907d0 msgr2=0x55be7ae8e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:55.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.463+0000 7f91d37fe640 1 --2- 172.21.15.33:0/178132433 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be7ae907d0 0x55be7ae8e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.463+0000 7f91d37fe640 1 -- 172.21.15.33:0/178132433 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be7aafe3f0 con 0x55be7ae872d0 2024-03-19T03:01:55.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.463+0000 7f91d37fe640 1 --2- 172.21.15.33:0/178132433 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be7ae872d0 0x55be7ae8d830 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f91c4009f50 tx=0x7f91c4019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7db0a2eab86f221a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:55.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.463+0000 7f91b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:01:55.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.463+0000 7f91b2ffd640 1 -- 172.21.15.33:0/178132433 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c401e050 con 0x55be7ae872d0 2024-03-19T03:01:55.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.463+0000 7f91dc814740 1 -- 172.21.15.33:0/178132433 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91c4019ae0 con 0x55be7ae872d0 2024-03-19T03:01:55.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.463+0000 7f91b2ffd640 1 -- 172.21.15.33:0/178132433 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f91c4005240 con 0x55be7ae872d0 2024-03-19T03:01:55.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.463+0000 7f91b2ffd640 1 -- 172.21.15.33:0/178132433 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f91c4033050 con 0x55be7ae872d0 2024-03-19T03:01:55.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.463+0000 7f91dc814740 10 client.17460 fetch_fsmap learned FSMap version 25 2024-03-19T03:01:55.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.463+0000 7f91dc814740 1 -- 172.21.15.33:0/178132433 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55be7abc87d0 con 0x55be7ae872d0 2024-03-19T03:01:55.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.463+0000 7f91b2ffd640 1 -- 172.21.15.33:0/178132433 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f91c4019ae0 con 0x55be7ae872d0 2024-03-19T03:01:55.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.464+0000 7f91b2ffd640 1 -- 172.21.15.33:0/178132433 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f91c400e6b0 con 0x55be7ae872d0 2024-03-19T03:01:55.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.464+0000 7f91dc814740 10 client.17460 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:01:55.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.464+0000 7f91dc814740 10 client.17460 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T03:01:55.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.464+0000 7f91dc814740 20 client.17460 populate_metadata read hostname 'smithi033' 2024-03-19T03:01:55.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.464+0000 7f91dc814740 1 --2- 172.21.15.33:0/178132433 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55be7aea0b20 0x55be7aea2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.464+0000 7f91dc814740 4 client.17460 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:01:55.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.464+0000 7f91dc814740 1 -- 172.21.15.33:0/178132433 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55be7ae483e0 con 0x55be7aea0b20 2024-03-19T03:01:55.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.464+0000 7f91d2ffd640 1 --2- 172.21.15.33:0/178132433 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55be7aea0b20 0x55be7aea2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.464+0000 7f91d2ffd640 1 --2- 172.21.15.33:0/178132433 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55be7aea0b20 0x55be7aea2f10 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7aa9ff1072ad82e server_cookie=efd22c93df9466ea in_seq=0 out_seq=0 2024-03-19T03:01:55.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.464+0000 7f91b2ffd640 10 client.17460 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T03:01:55.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.465+0000 7f91b2ffd640 1 -- 172.21.15.33:0/178132433 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55be7ae483e0 con 0x55be7aea0b20 2024-03-19T03:01:55.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.465+0000 7f91b2ffd640 10 client.17460 handle_command_reply: tid=0 2024-03-19T03:01:55.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.465+0000 7f91dc814740 1 client.17460 shutdown 2024-03-19T03:01:55.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.465+0000 7f91dc814740 1 -- 172.21.15.33:0/178132433 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be7ae872d0 msgr2=0x55be7ae8d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:55.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.465+0000 7f91dc814740 1 --2- 172.21.15.33:0/178132433 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be7ae872d0 0x55be7ae8d830 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f91c4009f50 tx=0x7f91c4019660 comp rx=0 tx=0).stop 2024-03-19T03:01:55.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.465+0000 7f91dc814740 1 -- 172.21.15.33:0/178132433 shutdown_connections 2024-03-19T03:01:55.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.465+0000 7f91dc814740 1 --2- 172.21.15.33:0/178132433 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55be7aea0b20 0x55be7aea2f10 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.465+0000 7f91dc814740 1 --2- 172.21.15.33:0/178132433 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7ae87bf0 0x55be7ae8dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.465+0000 7f91dc814740 1 --2- 172.21.15.33:0/178132433 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be7ae907d0 0x55be7ae8e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.465+0000 7f91b2ffd640 0 client.17460 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T03:01:55.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.465+0000 7f91dc814740 1 --2- 172.21.15.33:0/178132433 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be7ae872d0 0x55be7ae8d830 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.465+0000 7f91dc814740 1 -- 172.21.15.33:0/178132433 >> 172.21.15.33:0/178132433 conn(0x55be7ae81a00 msgr2=0x55be7ae83750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:55.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.465+0000 7f91dc814740 1 -- 172.21.15.33:0/178132433 shutdown_connections 2024-03-19T03:01:55.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.465+0000 7f91dc814740 1 -- 172.21.15.33:0/178132433 wait complete. 2024-03-19T03:01:55.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.465+0000 7f91dc814740 20 client.17460 trim_cache size 0 max 16384 2024-03-19T03:01:55.485 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.478+0000 7f91dc814740 1 Processor -- start 2024-03-19T03:01:55.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.478+0000 7f91dc814740 1 -- start start 2024-03-19T03:01:55.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.478+0000 7f91dc814740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be7ae872d0 0x55be7ae8caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.479+0000 7f91dc814740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7ae87bf0 0x55be7ae9c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.479+0000 7f91dc814740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be7ae907d0 0x55be7ae8e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.479+0000 7f91dc814740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f91d41b3800 con 0x55be7ae907d0 2024-03-19T03:01:55.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.479+0000 7f91dc814740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f91d41b10d0 con 0x55be7ae872d0 2024-03-19T03:01:55.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.479+0000 7f91dc814740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f91d41aef40 con 0x55be7ae87bf0 2024-03-19T03:01:55.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.479+0000 7f91d3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be7ae907d0 0x55be7ae8e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.486 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.479+0000 7f91d3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be7ae907d0 0x55be7ae8e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35846/0 (socket says 172.21.15.33:35846) 2024-03-19T03:01:55.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.479+0000 7f91d3fff640 1 -- 172.21.15.33:0/3348103403 learned_addr learned my addr 172.21.15.33:0/3348103403 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:55.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.479+0000 7f91d2ffd640 1 --2- 172.21.15.33:0/3348103403 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7ae87bf0 0x55be7ae9c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.479+0000 7f91d37fe640 1 --2- 172.21.15.33:0/3348103403 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be7ae872d0 0x55be7ae8caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.479+0000 7f91d3fff640 1 -- 172.21.15.33:0/3348103403 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7ae87bf0 msgr2=0x55be7ae9c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:55.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.479+0000 7f91d3fff640 1 --2- 172.21.15.33:0/3348103403 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7ae87bf0 0x55be7ae9c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.479+0000 7f91d3fff640 1 -- 172.21.15.33:0/3348103403 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be7ae872d0 msgr2=0x55be7ae8caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:55.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.479+0000 7f91d3fff640 1 --2- 172.21.15.33:0/3348103403 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be7ae872d0 0x55be7ae8caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.479+0000 7f91d3fff640 1 -- 172.21.15.33:0/3348103403 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be7ae86660 con 0x55be7ae907d0 2024-03-19T03:01:55.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.479+0000 7f91d3fff640 1 --2- 172.21.15.33:0/3348103403 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be7ae907d0 0x55be7ae8e0d0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f91cc003ef0 tx=0x7f91cc01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcd34f0136867068 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:55.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.479+0000 7f91b97fa640 1 -- 172.21.15.33:0/3348103403 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91cc017040 con 0x55be7ae907d0 2024-03-19T03:01:55.487 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.479+0000 7f91b97fa640 1 -- 172.21.15.33:0/3348103403 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f91cc019070 con 0x55be7ae907d0 2024-03-19T03:01:55.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.479+0000 7f91b97fa640 1 -- 172.21.15.33:0/3348103403 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91cc005020 con 0x55be7ae907d0 2024-03-19T03:01:55.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.480+0000 7f91dc814740 1 -- 172.21.15.33:0/3348103403 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be7ae907d0 msgr2=0x55be7ae8e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:55.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.480+0000 7f91dc814740 1 --2- 172.21.15.33:0/3348103403 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be7ae907d0 0x55be7ae8e0d0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f91cc003ef0 tx=0x7f91cc01c000 comp rx=0 tx=0).stop 2024-03-19T03:01:55.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.480+0000 7f91dc814740 1 -- 172.21.15.33:0/3348103403 shutdown_connections 2024-03-19T03:01:55.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.480+0000 7f91dc814740 1 --2- 172.21.15.33:0/3348103403 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7ae87bf0 0x55be7ae9c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.480+0000 7f91dc814740 1 --2- 172.21.15.33:0/3348103403 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be7ae907d0 0x55be7ae8e0d0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.480+0000 7f91dc814740 1 --2- 172.21.15.33:0/3348103403 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be7ae872d0 0x55be7ae8caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.480+0000 7f91dc814740 1 -- 172.21.15.33:0/3348103403 >> 172.21.15.33:0/3348103403 conn(0x55be7ae81a00 msgr2=0x55be7ae80ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:55.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.480+0000 7f91dc814740 1 -- 172.21.15.33:0/3348103403 shutdown_connections 2024-03-19T03:01:55.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.480+0000 7f91dc814740 1 -- 172.21.15.33:0/3348103403 wait complete. 2024-03-19T03:01:55.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.480+0000 7f91dc814740 1 -- start start 2024-03-19T03:01:55.488 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.480+0000 7f91dc814740 1 Processor -- start 2024-03-19T03:01:55.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.481+0000 7f91dc814740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be7ae872d0 0x55be7ae98870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.481+0000 7f91dc814740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7ae87bf0 0x55be7ae98db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.481+0000 7f91dc814740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be7ae907d0 0x55be7ae93f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.481+0000 7f91dc814740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f91d41af590 con 0x55be7ae907d0 2024-03-19T03:01:55.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.481+0000 7f91dc814740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f91d41b0a00 con 0x55be7ae872d0 2024-03-19T03:01:55.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.481+0000 7f91dc814740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f91d41afc60 con 0x55be7ae87bf0 2024-03-19T03:01:55.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.481+0000 7f91dc814740 10 client.0 _ll_register_callbacks cb 0x7f91c401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:01:55.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.481+0000 7f91d3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be7ae907d0 0x55be7ae93f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.481+0000 7f91d3fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be7ae907d0 0x55be7ae93f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35852/0 (socket says 172.21.15.33:35852) 2024-03-19T03:01:55.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.481+0000 7f91d3fff640 1 -- 172.21.15.33:0/642712250 learned_addr learned my addr 172.21.15.33:0/642712250 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:55.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.481+0000 7f91d2ffd640 1 --2- 172.21.15.33:0/642712250 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7ae87bf0 0x55be7ae98db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.489 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.481+0000 7f91d37fe640 1 --2- 172.21.15.33:0/642712250 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be7ae872d0 0x55be7ae98870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.481+0000 7f91d3fff640 1 -- 172.21.15.33:0/642712250 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7ae87bf0 msgr2=0x55be7ae98db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:55.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.481+0000 7f91d3fff640 1 --2- 172.21.15.33:0/642712250 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7ae87bf0 0x55be7ae98db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.481+0000 7f91d3fff640 1 -- 172.21.15.33:0/642712250 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be7ae872d0 msgr2=0x55be7ae98870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:55.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.481+0000 7f91d3fff640 1 --2- 172.21.15.33:0/642712250 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be7ae872d0 0x55be7ae98870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.481+0000 7f91d3fff640 1 -- 172.21.15.33:0/642712250 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be7ae94760 con 0x55be7ae907d0 2024-03-19T03:01:55.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.481+0000 7f91d2ffd640 1 --2- 172.21.15.33:0/642712250 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7ae87bf0 0x55be7ae98db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:01:55.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.481+0000 7f91d3fff640 1 --2- 172.21.15.33:0/642712250 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be7ae907d0 0x55be7ae93f30 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f91cc003ec0 tx=0x7f91cc003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e5506d207127396 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:55.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.481+0000 7f91b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:01:55.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.481+0000 7f91b2ffd640 1 -- 172.21.15.33:0/642712250 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91cc017030 con 0x55be7ae907d0 2024-03-19T03:01:55.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.481+0000 7f91b2ffd640 1 -- 172.21.15.33:0/642712250 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f91cc025040 con 0x55be7ae907d0 2024-03-19T03:01:55.490 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.481+0000 7f91dc814740 1 -- 172.21.15.33:0/642712250 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91cc019070 con 0x55be7ae907d0 2024-03-19T03:01:55.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.481+0000 7f91b2ffd640 1 -- 172.21.15.33:0/642712250 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f91cc004e10 con 0x55be7ae907d0 2024-03-19T03:01:55.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.481+0000 7f91dc814740 10 client.17466 fetch_fsmap learned FSMap version 25 2024-03-19T03:01:55.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.481+0000 7f91dc814740 1 -- 172.21.15.33:0/642712250 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55be7af069b0 con 0x55be7ae907d0 2024-03-19T03:01:55.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.482+0000 7f91b2ffd640 1 -- 172.21.15.33:0/642712250 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f91cc019070 con 0x55be7ae907d0 2024-03-19T03:01:55.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.482+0000 7f91b2ffd640 1 -- 172.21.15.33:0/642712250 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f91cc031020 con 0x55be7ae907d0 2024-03-19T03:01:55.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.482+0000 7f91dc814740 10 client.17466 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:01:55.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.482+0000 7f91dc814740 10 client.17466 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T03:01:55.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.482+0000 7f91dc814740 20 client.17466 populate_metadata read hostname 'smithi033' 2024-03-19T03:01:55.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.482+0000 7f91dc814740 1 --2- 172.21.15.33:0/642712250 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55be7af070a0 0x55be7af094c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.482+0000 7f91dc814740 4 client.17466 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T03:01:55.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.482+0000 7f91dc814740 1 -- 172.21.15.33:0/642712250 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55be7adf79f0 con 0x55be7af070a0 2024-03-19T03:01:55.491 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.482+0000 7f91d37fe640 1 --2- 172.21.15.33:0/642712250 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55be7af070a0 0x55be7af094c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.482+0000 7f91b2ffd640 10 client.17466 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T03:01:55.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.482+0000 7f91d37fe640 1 --2- 172.21.15.33:0/642712250 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55be7af070a0 0x55be7af094c0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=67ba8f92f3676c23 server_cookie=a60f33e1c7f41b35 in_seq=0 out_seq=0 2024-03-19T03:01:55.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.482+0000 7f91b2ffd640 1 -- 172.21.15.33:0/642712250 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55be7adf79f0 con 0x55be7af070a0 2024-03-19T03:01:55.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.482+0000 7f91b2ffd640 10 client.17466 handle_command_reply: tid=0 2024-03-19T03:01:55.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.483+0000 7f91dc814740 1 client.17466 shutdown 2024-03-19T03:01:55.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.483+0000 7f91dc814740 1 -- 172.21.15.33:0/642712250 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be7ae907d0 msgr2=0x55be7ae93f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:55.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.483+0000 7f91dc814740 1 --2- 172.21.15.33:0/642712250 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be7ae907d0 0x55be7ae93f30 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f91cc003ec0 tx=0x7f91cc003c80 comp rx=0 tx=0).stop 2024-03-19T03:01:55.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.483+0000 7f91dc814740 1 -- 172.21.15.33:0/642712250 shutdown_connections 2024-03-19T03:01:55.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.483+0000 7f91dc814740 1 --2- 172.21.15.33:0/642712250 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55be7af070a0 0x55be7af094c0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.483+0000 7f91dc814740 1 --2- 172.21.15.33:0/642712250 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55be7ae87bf0 0x55be7ae98db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.483+0000 7f91dc814740 1 --2- 172.21.15.33:0/642712250 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55be7ae907d0 0x55be7ae93f30 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.483+0000 7f91b2ffd640 0 client.17466 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T03:01:55.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.483+0000 7f91dc814740 1 --2- 172.21.15.33:0/642712250 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55be7ae872d0 0x55be7ae98870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.483+0000 7f91dc814740 1 -- 172.21.15.33:0/642712250 >> 172.21.15.33:0/642712250 conn(0x55be7ae81a00 msgr2=0x55be7ae82d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:55.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.483+0000 7f91dc814740 1 -- 172.21.15.33:0/642712250 shutdown_connections 2024-03-19T03:01:55.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.483+0000 7f91dc814740 1 -- 172.21.15.33:0/642712250 wait complete. 2024-03-19T03:01:55.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.483+0000 7f91dc814740 20 client.17466 trim_cache size 0 max 16384 2024-03-19T03:01:55.493 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T03:01:55.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.483+0000 7f91d9b59640 1 -- 172.21.15.33:0/2515724504 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f91bc0a2970 msgr2=0x7f91bc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:55.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.483+0000 7f91d9b59640 1 --2- 172.21.15.33:0/2515724504 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f91bc0a2970 0x7f91bc0a4e30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f91d4073fa0 tx=0x7f91cc011810 comp rx=0 tx=0).stop 2024-03-19T03:01:55.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.483+0000 7f91d9b59640 1 -- 172.21.15.33:0/2515724504 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91d4102bb0 msgr2=0x7f91d419a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:55.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.483+0000 7f91d9b59640 1 --2- 172.21.15.33:0/2515724504 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91d4102bb0 0x7f91d419a790 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f91c8036040 tx=0x7f91c80045e0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.483+0000 7f91d9b59640 1 -- 172.21.15.33:0/2515724504 shutdown_connections 2024-03-19T03:01:55.494 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.483+0000 7f91d9b59640 1 --2- 172.21.15.33:0/2515724504 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f91bc0a2970 0x7f91bc0a4e30 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.483+0000 7f91d9b59640 1 --2- 172.21.15.33:0/2515724504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f91d410e4a0 0x7f91d419eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.483+0000 7f91d9b59640 1 --2- 172.21.15.33:0/2515724504 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f91d4102bb0 0x7f91d419a790 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.483+0000 7f91d9b59640 1 --2- 172.21.15.33:0/2515724504 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91d4102740 0x7f91d419a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.483+0000 7f91d9b59640 1 -- 172.21.15.33:0/2515724504 >> 172.21.15.33:0/2515724504 conn(0x7f91d4076c50 msgr2=0x7f91d410fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:55.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.484+0000 7f91d9b59640 1 -- 172.21.15.33:0/2515724504 shutdown_connections 2024-03-19T03:01:55.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.484+0000 7f91d9b59640 1 -- 172.21.15.33:0/2515724504 wait complete. 2024-03-19T03:01:55.495 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:55 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1432272748' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:01:55.496 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T03:01:55.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.599+0000 7f1237577640 1 -- 172.21.15.33:0/1666581427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1230102a40 msgr2=0x7f1230102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:55.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.599+0000 7f1237577640 1 --2- 172.21.15.33:0/1666581427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1230102a40 0x7f1230102e20 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f1228009ca0 tx=0x7f122802c180 comp rx=0 tx=0).stop 2024-03-19T03:01:55.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.599+0000 7f1237577640 1 -- 172.21.15.33:0/1666581427 shutdown_connections 2024-03-19T03:01:55.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.599+0000 7f1237577640 1 --2- 172.21.15.33:0/1666581427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1230103360 0x7f123010b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.599+0000 7f1237577640 1 --2- 172.21.15.33:0/1666581427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f123010bd80 0x7f123010e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.599+0000 7f1237577640 1 --2- 172.21.15.33:0/1666581427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1230102a40 0x7f1230102e20 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.599+0000 7f1237577640 1 -- 172.21.15.33:0/1666581427 >> 172.21.15.33:0/1666581427 conn(0x7f12300778d0 msgr2=0x7f1230077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:55.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.600+0000 7f1237577640 1 -- 172.21.15.33:0/1666581427 shutdown_connections 2024-03-19T03:01:55.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.600+0000 7f1237577640 1 -- 172.21.15.33:0/1666581427 wait complete. 2024-03-19T03:01:55.603 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.600+0000 7f1237577640 1 Processor -- start 2024-03-19T03:01:55.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.600+0000 7f1237577640 1 -- start start 2024-03-19T03:01:55.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.600+0000 7f1237577640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1230102a40 0x7f1230197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.600+0000 7f1237577640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1230103360 0x7f1230198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.600+0000 7f1237577640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f123010bd80 0x7f123019c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.600+0000 7f1237577640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1230111a30 con 0x7f1230102a40 2024-03-19T03:01:55.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.600+0000 7f1237577640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f12301118b0 con 0x7f123010bd80 2024-03-19T03:01:55.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.600+0000 7f1237577640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1230111bb0 con 0x7f1230103360 2024-03-19T03:01:55.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.600+0000 7f1235313640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1230102a40 0x7f1230197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.600+0000 7f1235313640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1230102a40 0x7f1230197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35860/0 (socket says 172.21.15.33:35860) 2024-03-19T03:01:55.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.600+0000 7f1235313640 1 -- 172.21.15.33:0/2880071782 learned_addr learned my addr 172.21.15.33:0/2880071782 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:55.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.601+0000 7f1234b12640 1 --2- 172.21.15.33:0/2880071782 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1230103360 0x7f1230198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.601+0000 7f1235b14640 1 --2- 172.21.15.33:0/2880071782 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f123010bd80 0x7f123019c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.601+0000 7f1235313640 1 -- 172.21.15.33:0/2880071782 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1230103360 msgr2=0x7f1230198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:55.604 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.601+0000 7f1235313640 1 --2- 172.21.15.33:0/2880071782 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1230103360 0x7f1230198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.601+0000 7f1235313640 1 -- 172.21.15.33:0/2880071782 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f123010bd80 msgr2=0x7f123019c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:55.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.601+0000 7f1235313640 1 --2- 172.21.15.33:0/2880071782 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f123010bd80 0x7f123019c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.601+0000 7f1235313640 1 -- 172.21.15.33:0/2880071782 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f123019cfc0 con 0x7f1230102a40 2024-03-19T03:01:55.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.601+0000 7f1235313640 1 --2- 172.21.15.33:0/2880071782 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1230102a40 0x7f1230197fd0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f1228005e30 tx=0x7f1228035df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:55.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.601+0000 7f122e7fc640 1 -- 172.21.15.33:0/2880071782 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1228039040 con 0x7f1230102a40 2024-03-19T03:01:55.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.601+0000 7f1237577640 1 -- 172.21.15.33:0/2880071782 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f123019d250 con 0x7f1230102a40 2024-03-19T03:01:55.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.601+0000 7f122e7fc640 1 -- 172.21.15.33:0/2880071782 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1228044070 con 0x7f1230102a40 2024-03-19T03:01:55.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.601+0000 7f122e7fc640 1 -- 172.21.15.33:0/2880071782 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1228032850 con 0x7f1230102a40 2024-03-19T03:01:55.605 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.601+0000 7f1237577640 1 -- 172.21.15.33:0/2880071782 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12301a4b40 con 0x7f1230102a40 2024-03-19T03:01:55.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.604+0000 7f122e7fc640 1 -- 172.21.15.33:0/2880071782 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f1228004040 con 0x7f1230102a40 2024-03-19T03:01:55.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.604+0000 7f123a238740 1 Processor -- start 2024-03-19T03:01:55.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.604+0000 7f123a238740 1 -- start start 2024-03-19T03:01:55.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.604+0000 7f123a238740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564f894022d0 0x564f894026b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.604+0000 7f123a238740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f89402bf0 0x564f8940b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.604+0000 7f123a238740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564f8940b7d0 0x564f8940dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.604+0000 7f123a238740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564f893654b0 con 0x564f89402bf0 2024-03-19T03:01:55.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.604+0000 7f123a238740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x564f8930dd70 con 0x564f894022d0 2024-03-19T03:01:55.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.604+0000 7f123a238740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564f89361550 con 0x564f8940b7d0 2024-03-19T03:01:55.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.604+0000 7f1235313640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f89402bf0 0x564f8940b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.604+0000 7f1235313640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f89402bf0 0x564f8940b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35862/0 (socket says 172.21.15.33:35862) 2024-03-19T03:01:55.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.604+0000 7f1235313640 1 -- 172.21.15.33:0/3642279362 learned_addr learned my addr 172.21.15.33:0/3642279362 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:55.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.604+0000 7f1235b14640 1 --2- 172.21.15.33:0/3642279362 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564f894022d0 0x564f894026b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.604+0000 7f1234b12640 1 --2- 172.21.15.33:0/3642279362 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564f8940b7d0 0x564f8940dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.604+0000 7f122e7fc640 1 --2- 172.21.15.33:0/2880071782 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f12180a2970 0x7f12180a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.605+0000 7f1235b14640 1 --2- 172.21.15.33:0/2880071782 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f12180a2970 0x7f12180a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.605+0000 7f1235313640 1 -- 172.21.15.33:0/3642279362 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564f8940b7d0 msgr2=0x564f8940dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:55.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.605+0000 7f1235313640 1 --2- 172.21.15.33:0/3642279362 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564f8940b7d0 0x564f8940dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.605+0000 7f1235313640 1 -- 172.21.15.33:0/3642279362 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564f894022d0 msgr2=0x564f894026b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:55.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.605+0000 7f1235313640 1 --2- 172.21.15.33:0/3642279362 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564f894022d0 0x564f894026b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.605+0000 7f1235313640 1 -- 172.21.15.33:0/3642279362 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f89330210 con 0x564f89402bf0 2024-03-19T03:01:55.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.605+0000 7f1234b12640 1 --2- 172.21.15.33:0/3642279362 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564f8940b7d0 0x564f8940dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:55.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.605+0000 7f122e7fc640 1 -- 172.21.15.33:0/2880071782 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f12280f14d0 con 0x7f1230102a40 2024-03-19T03:01:55.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.605+0000 7f1235313640 1 --2- 172.21.15.33:0/3642279362 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f89402bf0 0x564f8940b290 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f12280ba240 tx=0x7f122802c180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1dc165de9140956 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:55.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.605+0000 7f1235b14640 1 --2- 172.21.15.33:0/2880071782 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f12180a2970 0x7f12180a4e30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1220009fd0 tx=0x7f1220004730 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:55.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.605+0000 7f12177fe640 1 -- 172.21.15.33:0/3642279362 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12280bd1d0 con 0x564f89402bf0 2024-03-19T03:01:55.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.605+0000 7f12177fe640 1 -- 172.21.15.33:0/3642279362 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1228043960 con 0x564f89402bf0 2024-03-19T03:01:55.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.605+0000 7f12177fe640 1 -- 172.21.15.33:0/3642279362 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12280c19c0 con 0x564f89402bf0 2024-03-19T03:01:55.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.606+0000 7f123a238740 1 -- 172.21.15.33:0/3642279362 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f89402bf0 msgr2=0x564f8940b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:55.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.606+0000 7f123a238740 1 --2- 172.21.15.33:0/3642279362 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f89402bf0 0x564f8940b290 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f12280ba240 tx=0x7f122802c180 comp rx=0 tx=0).stop 2024-03-19T03:01:55.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.606+0000 7f123a238740 1 -- 172.21.15.33:0/3642279362 shutdown_connections 2024-03-19T03:01:55.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.606+0000 7f123a238740 1 --2- 172.21.15.33:0/3642279362 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564f8940b7d0 0x564f8940dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.606+0000 7f123a238740 1 --2- 172.21.15.33:0/3642279362 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f89402bf0 0x564f8940b290 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.606+0000 7f123a238740 1 --2- 172.21.15.33:0/3642279362 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564f894022d0 0x564f894026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.606+0000 7f123a238740 1 -- 172.21.15.33:0/3642279362 >> 172.21.15.33:0/3642279362 conn(0x564f893fca00 msgr2=0x564f893fcde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:55.614 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.606+0000 7f123a238740 1 -- 172.21.15.33:0/3642279362 shutdown_connections 2024-03-19T03:01:55.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.606+0000 7f123a238740 1 -- 172.21.15.33:0/3642279362 wait complete. 2024-03-19T03:01:55.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.606+0000 7f123a238740 1 -- start start 2024-03-19T03:01:55.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.606+0000 7f123a238740 1 Processor -- start 2024-03-19T03:01:55.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.607+0000 7f123a238740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564f894022d0 0x564f894087a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.607+0000 7f123a238740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564f89402bf0 0x564f89408ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.607+0000 7f123a238740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f8940b7d0 0x564f89409220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.607+0000 7f123a238740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x564f8940e380 con 0x564f8940b7d0 2024-03-19T03:01:55.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.607+0000 7f123a238740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x564f89321310 con 0x564f89402bf0 2024-03-19T03:01:55.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.607+0000 7f123a238740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x564f8940e500 con 0x564f894022d0 2024-03-19T03:01:55.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.607+0000 7f123a238740 10 client.0 _ll_register_callbacks cb 0x564f893152d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:01:55.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.607+0000 7f1235b14640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f8940b7d0 0x564f89409220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.607+0000 7f1235313640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564f894022d0 0x564f894087a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.607+0000 7f1235313640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564f894022d0 0x564f894087a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:46026/0 (socket says 172.21.15.33:46026) 2024-03-19T03:01:55.615 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.607+0000 7f1235313640 1 -- 172.21.15.33:0/3824810533 learned_addr learned my addr 172.21.15.33:0/3824810533 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:55.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.607+0000 7f1235b14640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f8940b7d0 0x564f89409220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35866/0 (socket says 172.21.15.33:35866) 2024-03-19T03:01:55.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.607+0000 7f1234b12640 1 --2- 172.21.15.33:0/3824810533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564f89402bf0 0x564f89408ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.607+0000 7f1235b14640 1 -- 172.21.15.33:0/3824810533 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564f894022d0 msgr2=0x564f894087a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:55.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.607+0000 7f1235b14640 1 --2- 172.21.15.33:0/3824810533 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564f894022d0 0x564f894087a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.607+0000 7f1235b14640 1 -- 172.21.15.33:0/3824810533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564f89402bf0 msgr2=0x564f89408ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:55.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.607+0000 7f1235b14640 1 --2- 172.21.15.33:0/3824810533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564f89402bf0 0x564f89408ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.607+0000 7f1235b14640 1 -- 172.21.15.33:0/3824810533 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f890793f0 con 0x564f8940b7d0 2024-03-19T03:01:55.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.607+0000 7f1235b14640 1 --2- 172.21.15.33:0/3824810533 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f8940b7d0 0x564f89409220 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f1220010850 tx=0x7f12200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5effc883598ca88e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:55.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.607+0000 7f1214ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:01:55.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.607+0000 7f1214ff9640 1 -- 172.21.15.33:0/3824810533 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f122000fa40 con 0x564f8940b7d0 2024-03-19T03:01:55.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.607+0000 7f1214ff9640 1 -- 172.21.15.33:0/3824810533 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f122000fbe0 con 0x564f8940b7d0 2024-03-19T03:01:55.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.607+0000 7f123a238740 1 -- 172.21.15.33:0/3824810533 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1228043960 con 0x564f8940b7d0 2024-03-19T03:01:55.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.608+0000 7f1214ff9640 1 -- 172.21.15.33:0/3824810533 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f122000fde0 con 0x564f8940b7d0 2024-03-19T03:01:55.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.608+0000 7f123a238740 10 client.17478 fetch_fsmap learned FSMap version 25 2024-03-19T03:01:55.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.608+0000 7f123a238740 1 -- 172.21.15.33:0/3824810533 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564f891437d0 con 0x564f8940b7d0 2024-03-19T03:01:55.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.608+0000 7f1214ff9640 1 -- 172.21.15.33:0/3824810533 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1228043960 con 0x564f8940b7d0 2024-03-19T03:01:55.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.608+0000 7f1214ff9640 1 -- 172.21.15.33:0/3824810533 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f1220020400 con 0x564f8940b7d0 2024-03-19T03:01:55.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.608+0000 7f123a238740 10 client.17478 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:01:55.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.608+0000 7f123a238740 10 client.17478 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:01:55.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.608+0000 7f123a238740 20 client.17478 populate_metadata read hostname 'smithi033' 2024-03-19T03:01:55.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.608+0000 7f123a238740 1 --2- 172.21.15.33:0/3824810533 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x564f8941b920 0x564f8941dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.608+0000 7f123a238740 4 client.17478 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:01:55.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.608+0000 7f123a238740 1 -- 172.21.15.33:0/3824810533 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f893c33e0 con 0x564f8941b920 2024-03-19T03:01:55.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.608+0000 7f1235313640 1 --2- 172.21.15.33:0/3824810533 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x564f8941b920 0x564f8941dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.608+0000 7f1214ff9640 10 client.17478 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:01:55.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.608+0000 7f1235313640 1 --2- 172.21.15.33:0/3824810533 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x564f8941b920 0x564f8941dd40 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ac80897ba39b811 server_cookie=6e13a92ce142a72a in_seq=0 out_seq=0 2024-03-19T03:01:55.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.609+0000 7f1214ff9640 1 -- 172.21.15.33:0/3824810533 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564f893c33e0 con 0x564f8941b920 2024-03-19T03:01:55.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.609+0000 7f1214ff9640 10 client.17478 handle_command_reply: tid=0 2024-03-19T03:01:55.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.609+0000 7f123a238740 1 client.17478 shutdown 2024-03-19T03:01:55.617 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.609+0000 7f123a238740 1 -- 172.21.15.33:0/3824810533 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f8940b7d0 msgr2=0x564f89409220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:55.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.609+0000 7f123a238740 1 --2- 172.21.15.33:0/3824810533 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f8940b7d0 0x564f89409220 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f1220010850 tx=0x7f12200046c0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.609+0000 7f123a238740 1 -- 172.21.15.33:0/3824810533 shutdown_connections 2024-03-19T03:01:55.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.609+0000 7f123a238740 1 --2- 172.21.15.33:0/3824810533 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x564f8941b920 0x564f8941dd40 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.609+0000 7f123a238740 1 --2- 172.21.15.33:0/3824810533 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f8940b7d0 0x564f89409220 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.609+0000 7f123a238740 1 --2- 172.21.15.33:0/3824810533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564f89402bf0 0x564f89408ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.609+0000 7f123a238740 1 --2- 172.21.15.33:0/3824810533 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564f894022d0 0x564f894087a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.609+0000 7f1214ff9640 0 client.17478 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:01:55.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.609+0000 7f123a238740 1 -- 172.21.15.33:0/3824810533 >> 172.21.15.33:0/3824810533 conn(0x564f893fca00 msgr2=0x564f893fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:55.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.609+0000 7f123a238740 1 -- 172.21.15.33:0/3824810533 shutdown_connections 2024-03-19T03:01:55.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.609+0000 7f123a238740 1 -- 172.21.15.33:0/3824810533 wait complete. 2024-03-19T03:01:55.618 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.609+0000 7f123a238740 20 client.17478 trim_cache size 0 max 16384 2024-03-19T03:01:55.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.622+0000 7f123a238740 1 Processor -- start 2024-03-19T03:01:55.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.623+0000 7f123a238740 1 -- start start 2024-03-19T03:01:55.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.623+0000 7f123a238740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564f894022d0 0x564f893fba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.623+0000 7f123a238740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f89402bf0 0x564f894235f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.623+0000 7f123a238740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564f8940b7d0 0x564f89413740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.623+0000 7f123a238740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f12301b1570 con 0x564f89402bf0 2024-03-19T03:01:55.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.623+0000 7f123a238740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f12301aee40 con 0x564f894022d0 2024-03-19T03:01:55.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.623+0000 7f123a238740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f12301accb0 con 0x564f8940b7d0 2024-03-19T03:01:55.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.623+0000 7f1234b12640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f89402bf0 0x564f894235f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.623+0000 7f1234b12640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f89402bf0 0x564f894235f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35878/0 (socket says 172.21.15.33:35878) 2024-03-19T03:01:55.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.623+0000 7f1234b12640 1 -- 172.21.15.33:0/1837399473 learned_addr learned my addr 172.21.15.33:0/1837399473 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:55.630 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.623+0000 7f1235313640 1 --2- 172.21.15.33:0/1837399473 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564f894022d0 0x564f893fba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.623+0000 7f1235b14640 1 --2- 172.21.15.33:0/1837399473 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564f8940b7d0 0x564f89413740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.623+0000 7f1234b12640 1 -- 172.21.15.33:0/1837399473 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564f8940b7d0 msgr2=0x564f89413740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:55.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.623+0000 7f1234b12640 1 --2- 172.21.15.33:0/1837399473 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564f8940b7d0 0x564f89413740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.623+0000 7f1234b12640 1 -- 172.21.15.33:0/1837399473 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564f894022d0 msgr2=0x564f893fba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:55.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.623+0000 7f1234b12640 1 --2- 172.21.15.33:0/1837399473 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564f894022d0 0x564f893fba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.623+0000 7f1234b12640 1 -- 172.21.15.33:0/1837399473 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f893d2880 con 0x564f89402bf0 2024-03-19T03:01:55.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.623+0000 7f1234b12640 1 --2- 172.21.15.33:0/1837399473 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f89402bf0 0x564f894235f0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f1224009ea0 tx=0x7f1224005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0e71ba725f7da0b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:55.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.623+0000 7f1216ffd640 1 -- 172.21.15.33:0/1837399473 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1224022070 con 0x564f89402bf0 2024-03-19T03:01:55.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.623+0000 7f1216ffd640 1 -- 172.21.15.33:0/1837399473 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f12240035f0 con 0x564f89402bf0 2024-03-19T03:01:55.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.623+0000 7f1216ffd640 1 -- 172.21.15.33:0/1837399473 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f122401d6c0 con 0x564f89402bf0 2024-03-19T03:01:55.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.624+0000 7f123a238740 1 -- 172.21.15.33:0/1837399473 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f89402bf0 msgr2=0x564f894235f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:55.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.624+0000 7f123a238740 1 --2- 172.21.15.33:0/1837399473 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f89402bf0 0x564f894235f0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f1224009ea0 tx=0x7f1224005f90 comp rx=0 tx=0).stop 2024-03-19T03:01:55.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.624+0000 7f123a238740 1 -- 172.21.15.33:0/1837399473 shutdown_connections 2024-03-19T03:01:55.631 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.624+0000 7f123a238740 1 --2- 172.21.15.33:0/1837399473 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564f8940b7d0 0x564f89413740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.624+0000 7f123a238740 1 --2- 172.21.15.33:0/1837399473 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f89402bf0 0x564f894235f0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.624+0000 7f123a238740 1 --2- 172.21.15.33:0/1837399473 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564f894022d0 0x564f893fba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.624+0000 7f123a238740 1 -- 172.21.15.33:0/1837399473 >> 172.21.15.33:0/1837399473 conn(0x564f893fca00 msgr2=0x564f893fb310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:55.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.624+0000 7f123a238740 1 -- 172.21.15.33:0/1837399473 shutdown_connections 2024-03-19T03:01:55.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.624+0000 7f123a238740 1 -- 172.21.15.33:0/1837399473 wait complete. 2024-03-19T03:01:55.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.624+0000 7f123a238740 1 -- start start 2024-03-19T03:01:55.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.624+0000 7f123a238740 1 Processor -- start 2024-03-19T03:01:55.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.625+0000 7f123a238740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564f894022d0 0x564f89405610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.625+0000 7f123a238740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f89402bf0 0x564f89405b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.625+0000 7f123a238740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564f8940b7d0 0x564f89406090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.625+0000 7f123a238740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f12301ad300 con 0x564f89402bf0 2024-03-19T03:01:55.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.625+0000 7f123a238740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f12301ae770 con 0x564f894022d0 2024-03-19T03:01:55.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.625+0000 7f123a238740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f12301ad9d0 con 0x564f8940b7d0 2024-03-19T03:01:55.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.625+0000 7f123a238740 10 client.0 _ll_register_callbacks cb 0x7f1220009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:01:55.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.625+0000 7f1234b12640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f89402bf0 0x564f89405b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.625+0000 7f1234b12640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f89402bf0 0x564f89405b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35890/0 (socket says 172.21.15.33:35890) 2024-03-19T03:01:55.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.625+0000 7f1234b12640 1 -- 172.21.15.33:0/1090732719 learned_addr learned my addr 172.21.15.33:0/1090732719 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:55.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.625+0000 7f1235313640 1 --2- 172.21.15.33:0/1090732719 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564f894022d0 0x564f89405610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.625+0000 7f1235b14640 1 --2- 172.21.15.33:0/1090732719 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564f8940b7d0 0x564f89406090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.625+0000 7f1234b12640 1 -- 172.21.15.33:0/1090732719 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564f8940b7d0 msgr2=0x564f89406090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:55.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.625+0000 7f1234b12640 1 --2- 172.21.15.33:0/1090732719 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564f8940b7d0 0x564f89406090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.625+0000 7f1234b12640 1 -- 172.21.15.33:0/1090732719 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564f894022d0 msgr2=0x564f89405610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:55.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.625+0000 7f1234b12640 1 --2- 172.21.15.33:0/1090732719 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564f894022d0 0x564f89405610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.625+0000 7f1234b12640 1 -- 172.21.15.33:0/1090732719 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f893fc830 con 0x564f89402bf0 2024-03-19T03:01:55.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.625+0000 7f1234b12640 1 --2- 172.21.15.33:0/1090732719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f89402bf0 0x564f89405b50 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x564f89403740 tx=0x7f1224003d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bf3950379aba4e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:55.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.625+0000 7f1214ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:01:55.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.625+0000 7f1214ff9640 1 -- 172.21.15.33:0/1090732719 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1224022040 con 0x564f89402bf0 2024-03-19T03:01:55.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.626+0000 7f1214ff9640 1 -- 172.21.15.33:0/1090732719 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f1224004250 con 0x564f89402bf0 2024-03-19T03:01:55.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.626+0000 7f123a238740 1 -- 172.21.15.33:0/1090732719 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12240035f0 con 0x564f89402bf0 2024-03-19T03:01:55.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.626+0000 7f1214ff9640 1 -- 172.21.15.33:0/1090732719 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1224036050 con 0x564f89402bf0 2024-03-19T03:01:55.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.626+0000 7f123a238740 10 client.17484 fetch_fsmap learned FSMap version 25 2024-03-19T03:01:55.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.626+0000 7f123a238740 1 -- 172.21.15.33:0/1090732719 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564f89481ff0 con 0x564f89402bf0 2024-03-19T03:01:55.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.626+0000 7f1214ff9640 1 -- 172.21.15.33:0/1090732719 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f12240035f0 con 0x564f89402bf0 2024-03-19T03:01:55.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.626+0000 7f1214ff9640 1 -- 172.21.15.33:0/1090732719 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f1224031030 con 0x564f89402bf0 2024-03-19T03:01:55.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.626+0000 7f123a238740 10 client.17484 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:01:55.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.626+0000 7f123a238740 10 client.17484 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:01:55.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.626+0000 7f123a238740 20 client.17484 populate_metadata read hostname 'smithi033' 2024-03-19T03:01:55.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.626+0000 7f123a238740 1 --2- 172.21.15.33:0/1090732719 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x564f894826a0 0x564f89484ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:55.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.626+0000 7f123a238740 4 client.17484 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T03:01:55.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.626+0000 7f123a238740 1 -- 172.21.15.33:0/1090732719 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564f893729f0 con 0x564f894826a0 2024-03-19T03:01:55.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.626+0000 7f1235313640 1 --2- 172.21.15.33:0/1090732719 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x564f894826a0 0x564f89484ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:55.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.627+0000 7f1214ff9640 10 client.17484 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:01:55.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.627+0000 7f1235313640 1 --2- 172.21.15.33:0/1090732719 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x564f894826a0 0x564f89484ac0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2acd42016857c4f server_cookie=187786e7bc9611d0 in_seq=0 out_seq=0 2024-03-19T03:01:55.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.627+0000 7f1214ff9640 1 -- 172.21.15.33:0/1090732719 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564f893729f0 con 0x564f894826a0 2024-03-19T03:01:55.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.627+0000 7f1214ff9640 10 client.17484 handle_command_reply: tid=0 2024-03-19T03:01:55.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.627+0000 7f123a238740 1 client.17484 shutdown 2024-03-19T03:01:55.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.627+0000 7f123a238740 1 -- 172.21.15.33:0/1090732719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f89402bf0 msgr2=0x564f89405b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:55.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.627+0000 7f123a238740 1 --2- 172.21.15.33:0/1090732719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f89402bf0 0x564f89405b50 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x564f89403740 tx=0x7f1224003d90 comp rx=0 tx=0).stop 2024-03-19T03:01:55.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.627+0000 7f123a238740 1 -- 172.21.15.33:0/1090732719 shutdown_connections 2024-03-19T03:01:55.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.627+0000 7f123a238740 1 --2- 172.21.15.33:0/1090732719 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x564f894826a0 0x564f89484ac0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.627+0000 7f123a238740 1 --2- 172.21.15.33:0/1090732719 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x564f8940b7d0 0x564f89406090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.627+0000 7f1214ff9640 0 client.17484 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:01:55.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.627+0000 7f123a238740 1 --2- 172.21.15.33:0/1090732719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x564f89402bf0 0x564f89405b50 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.627+0000 7f123a238740 1 --2- 172.21.15.33:0/1090732719 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564f894022d0 0x564f89405610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.627+0000 7f123a238740 1 -- 172.21.15.33:0/1090732719 >> 172.21.15.33:0/1090732719 conn(0x564f893fca00 msgr2=0x564f893fb310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:55.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.627+0000 7f123a238740 1 -- 172.21.15.33:0/1090732719 shutdown_connections 2024-03-19T03:01:55.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.627+0000 7f123a238740 1 -- 172.21.15.33:0/1090732719 wait complete. 2024-03-19T03:01:55.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.627+0000 7f123a238740 20 client.17484 trim_cache size 0 max 16384 2024-03-19T03:01:55.636 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T03:01:55.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.628+0000 7f1237577640 1 -- 172.21.15.33:0/2880071782 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f12180a2970 msgr2=0x7f12180a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:55.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.628+0000 7f1237577640 1 --2- 172.21.15.33:0/2880071782 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f12180a2970 0x7f12180a4e30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1220009fd0 tx=0x7f1220004730 comp rx=0 tx=0).stop 2024-03-19T03:01:55.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.628+0000 7f1237577640 1 -- 172.21.15.33:0/2880071782 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1230102a40 msgr2=0x7f1230197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:55.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.628+0000 7f1237577640 1 --2- 172.21.15.33:0/2880071782 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1230102a40 0x7f1230197fd0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f1228005e30 tx=0x7f1228035df0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.628+0000 7f1237577640 1 -- 172.21.15.33:0/2880071782 shutdown_connections 2024-03-19T03:01:55.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.628+0000 7f1237577640 1 --2- 172.21.15.33:0/2880071782 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f12180a2970 0x7f12180a4e30 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.628+0000 7f1237577640 1 --2- 172.21.15.33:0/2880071782 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1230103360 0x7f1230198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.628+0000 7f1237577640 1 --2- 172.21.15.33:0/2880071782 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f123010bd80 0x7f123019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.628+0000 7f1237577640 1 --2- 172.21.15.33:0/2880071782 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1230102a40 0x7f1230197fd0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:55.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.628+0000 7f1237577640 1 -- 172.21.15.33:0/2880071782 >> 172.21.15.33:0/2880071782 conn(0x7f12300778d0 msgr2=0x7f12300fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:55.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.628+0000 7f1237577640 1 -- 172.21.15.33:0/2880071782 shutdown_connections 2024-03-19T03:01:55.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:55.628+0000 7f1237577640 1 -- 172.21.15.33:0/2880071782 wait complete. 2024-03-19T03:01:55.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:55 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1432272748' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:01:55.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:55 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1432272748' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:01:56.641 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T03:01:56.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:01:56.641 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T03:01:56.669 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:56 smithi033 ceph-mon[31973]: pgmap v261: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:01:56.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:56 smithi096 ceph-mon[35946]: pgmap v261: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:01:56.763 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.759+0000 7f93197f4640 1 -- 172.21.15.33:0/2186733954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9314101100 msgr2=0x7f931410d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:56.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.759+0000 7f93197f4640 1 --2- 172.21.15.33:0/2186733954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9314101100 0x7f931410d610 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f930c009cd0 tx=0x7f930c02c0f0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.759+0000 7f93197f4640 1 -- 172.21.15.33:0/2186733954 shutdown_connections 2024-03-19T03:01:56.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.759+0000 7f93197f4640 1 --2- 172.21.15.33:0/2186733954 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9314101100 0x7f931410d610 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.759+0000 7f93197f4640 1 --2- 172.21.15.33:0/2186733954 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9314100740 0x7f9314100bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.759+0000 7f93197f4640 1 --2- 172.21.15.33:0/2186733954 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9314106750 0x7f9314106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.759+0000 7f93197f4640 1 -- 172.21.15.33:0/2186733954 >> 172.21.15.33:0/2186733954 conn(0x7f9314077880 msgr2=0x7f9314077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:56.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.760+0000 7f93197f4640 1 -- 172.21.15.33:0/2186733954 shutdown_connections 2024-03-19T03:01:56.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.760+0000 7f93197f4640 1 -- 172.21.15.33:0/2186733954 wait complete. 2024-03-19T03:01:56.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.760+0000 7f93197f4640 1 Processor -- start 2024-03-19T03:01:56.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.760+0000 7f93197f4640 1 -- start start 2024-03-19T03:01:56.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.760+0000 7f93197f4640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9314100740 0x7f9314198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.760+0000 7f93197f4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9314101100 0x7f9314198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.760+0000 7f93197f4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9314106750 0x7f931419c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.760+0000 7f93197f4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f9314110a10 con 0x7f9314106750 2024-03-19T03:01:56.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.760+0000 7f93197f4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9314110890 con 0x7f9314101100 2024-03-19T03:01:56.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.760+0000 7f93197f4640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9314110b90 con 0x7f9314100740 2024-03-19T03:01:56.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.761+0000 7f93137fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9314106750 0x7f931419c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.761+0000 7f930bfff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9314101100 0x7f9314198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.761+0000 7f93137fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9314106750 0x7f931419c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35914/0 (socket says 172.21.15.33:35914) 2024-03-19T03:01:56.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.761+0000 7f93137fe640 1 -- 172.21.15.33:0/2911511789 learned_addr learned my addr 172.21.15.33:0/2911511789 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:56.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.761+0000 7f9312ffd640 1 --2- 172.21.15.33:0/2911511789 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9314100740 0x7f9314198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.761+0000 7f93137fe640 1 -- 172.21.15.33:0/2911511789 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9314100740 msgr2=0x7f9314198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:56.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.761+0000 7f93137fe640 1 --2- 172.21.15.33:0/2911511789 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9314100740 0x7f9314198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.761+0000 7f93137fe640 1 -- 172.21.15.33:0/2911511789 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9314101100 msgr2=0x7f9314198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:56.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.761+0000 7f93137fe640 1 --2- 172.21.15.33:0/2911511789 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9314101100 0x7f9314198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.761+0000 7f93137fe640 1 -- 172.21.15.33:0/2911511789 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f931419d030 con 0x7f9314106750 2024-03-19T03:01:56.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.761+0000 7f93137fe640 1 --2- 172.21.15.33:0/2911511789 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9314106750 0x7f931419c900 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f930c004af0 tx=0x7f930c035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:56.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.761+0000 7f9310ff9640 1 -- 172.21.15.33:0/2911511789 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930c039040 con 0x7f9314106750 2024-03-19T03:01:56.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.761+0000 7f93197f4640 1 -- 172.21.15.33:0/2911511789 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f931419d2c0 con 0x7f9314106750 2024-03-19T03:01:56.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.761+0000 7f9310ff9640 1 -- 172.21.15.33:0/2911511789 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f930c044070 con 0x7f9314106750 2024-03-19T03:01:56.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.761+0000 7f9310ff9640 1 -- 172.21.15.33:0/2911511789 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930c0042a0 con 0x7f9314106750 2024-03-19T03:01:56.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.761+0000 7f93197f4640 1 -- 172.21.15.33:0/2911511789 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93141a4bb0 con 0x7f9314106750 2024-03-19T03:01:56.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:56 smithi155 ceph-mon[35804]: pgmap v261: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:01:56.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.763+0000 7f9310ff9640 1 -- 172.21.15.33:0/2911511789 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f930c003970 con 0x7f9314106750 2024-03-19T03:01:56.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.764+0000 7f931c3e9740 1 Processor -- start 2024-03-19T03:01:56.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.764+0000 7f931c3e9740 1 -- start start 2024-03-19T03:01:56.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.764+0000 7f9310ff9640 1 --2- 172.21.15.33:0/2911511789 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f93000a2970 0x7f93000a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.764+0000 7f930bfff640 1 --2- 172.21.15.33:0/2911511789 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f93000a2970 0x7f93000a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.764+0000 7f9310ff9640 1 -- 172.21.15.33:0/2911511789 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f930c0f1490 con 0x7f9314106750 2024-03-19T03:01:56.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.764+0000 7f931c3e9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560adc44a380 0x560adc44a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.764+0000 7f931c3e9740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560adc44aca0 0x560adc453340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.764+0000 7f931c3e9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560adc453880 0x560adc455c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.764+0000 7f931c3e9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560adc3ae160 con 0x560adc44a380 2024-03-19T03:01:56.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.764+0000 7f931c3e9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x560adc356e10 con 0x560adc453880 2024-03-19T03:01:56.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.764+0000 7f931c3e9740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560adc3aa200 con 0x560adc44aca0 2024-03-19T03:01:56.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.765+0000 7f9312ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560adc44a380 0x560adc44a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.765+0000 7f9312ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560adc44a380 0x560adc44a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35928/0 (socket says 172.21.15.33:35928) 2024-03-19T03:01:56.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.765+0000 7f9312ffd640 1 -- 172.21.15.33:0/2019248857 learned_addr learned my addr 172.21.15.33:0/2019248857 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:56.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.765+0000 7f93137fe640 1 --2- 172.21.15.33:0/2019248857 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560adc453880 0x560adc455c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.765+0000 7f930bfff640 1 --2- 172.21.15.33:0/2019248857 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560adc44aca0 0x560adc453340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.765+0000 7f930bfff640 1 --2- 172.21.15.33:0/2911511789 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f93000a2970 0x7f93000a4e30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9314077ff0 tx=0x7f92fc005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.765+0000 7f9312ffd640 1 -- 172.21.15.33:0/2019248857 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560adc44aca0 msgr2=0x560adc453340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.765+0000 7f9312ffd640 1 --2- 172.21.15.33:0/2019248857 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560adc44aca0 0x560adc453340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.765+0000 7f9312ffd640 1 -- 172.21.15.33:0/2019248857 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560adc453880 msgr2=0x560adc455c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.765+0000 7f9312ffd640 1 --2- 172.21.15.33:0/2019248857 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560adc453880 0x560adc455c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.765+0000 7f9312ffd640 1 -- 172.21.15.33:0/2019248857 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560adc18a7e0 con 0x560adc44a380 2024-03-19T03:01:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.765+0000 7f930bfff640 1 --2- 172.21.15.33:0/2019248857 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560adc44aca0 0x560adc453340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:01:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.765+0000 7f9312ffd640 1 --2- 172.21.15.33:0/2019248857 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560adc44a380 0x560adc44a760 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f9304009ca0 tx=0x7f9304011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39d56a436d1200a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.765+0000 7f93097fa640 1 -- 172.21.15.33:0/2019248857 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930401e070 con 0x560adc44a380 2024-03-19T03:01:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.765+0000 7f93097fa640 1 -- 172.21.15.33:0/2019248857 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9304019ae0 con 0x560adc44a380 2024-03-19T03:01:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.765+0000 7f93097fa640 1 -- 172.21.15.33:0/2019248857 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930400e9e0 con 0x560adc44a380 2024-03-19T03:01:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.766+0000 7f931c3e9740 1 -- 172.21.15.33:0/2019248857 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560adc44a380 msgr2=0x560adc44a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.766+0000 7f931c3e9740 1 --2- 172.21.15.33:0/2019248857 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560adc44a380 0x560adc44a760 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f9304009ca0 tx=0x7f9304011810 comp rx=0 tx=0).stop 2024-03-19T03:01:56.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.766+0000 7f931c3e9740 1 -- 172.21.15.33:0/2019248857 shutdown_connections 2024-03-19T03:01:56.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.766+0000 7f931c3e9740 1 --2- 172.21.15.33:0/2019248857 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560adc44aca0 0x560adc453340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.766+0000 7f931c3e9740 1 --2- 172.21.15.33:0/2019248857 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560adc453880 0x560adc455c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.766+0000 7f931c3e9740 1 --2- 172.21.15.33:0/2019248857 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560adc44a380 0x560adc44a760 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.766+0000 7f931c3e9740 1 -- 172.21.15.33:0/2019248857 >> 172.21.15.33:0/2019248857 conn(0x560adc444ab0 msgr2=0x560adc444e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:56.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.766+0000 7f931c3e9740 1 -- 172.21.15.33:0/2019248857 shutdown_connections 2024-03-19T03:01:56.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.766+0000 7f931c3e9740 1 -- 172.21.15.33:0/2019248857 wait complete. 2024-03-19T03:01:56.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.766+0000 7f931c3e9740 1 -- start start 2024-03-19T03:01:56.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.766+0000 7f931c3e9740 1 Processor -- start 2024-03-19T03:01:56.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.767+0000 7f931c3e9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560adc44a380 0x560adc450730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.767+0000 7f931c3e9740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560adc44aca0 0x560adc450c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.767+0000 7f931c3e9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560adc453880 0x560adc4511b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.767+0000 7f931c3e9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x560adc456670 con 0x560adc44a380 2024-03-19T03:01:56.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.767+0000 7f931c3e9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x560adc36a1a0 con 0x560adc453880 2024-03-19T03:01:56.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.767+0000 7f931c3e9740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x560adc4567f0 con 0x560adc44aca0 2024-03-19T03:01:56.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.767+0000 7f931c3e9740 10 client.0 _ll_register_callbacks cb 0x560adc0c1c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:01:56.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.767+0000 7f9312ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560adc44a380 0x560adc450730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.767+0000 7f9312ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560adc44a380 0x560adc450730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35938/0 (socket says 172.21.15.33:35938) 2024-03-19T03:01:56.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.767+0000 7f9312ffd640 1 -- 172.21.15.33:0/3547225615 learned_addr learned my addr 172.21.15.33:0/3547225615 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:56.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.767+0000 7f93137fe640 1 --2- 172.21.15.33:0/3547225615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560adc453880 0x560adc4511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.767+0000 7f930bfff640 1 --2- 172.21.15.33:0/3547225615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560adc44aca0 0x560adc450c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.767+0000 7f9312ffd640 1 -- 172.21.15.33:0/3547225615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560adc44aca0 msgr2=0x560adc450c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:56.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.767+0000 7f9312ffd640 1 --2- 172.21.15.33:0/3547225615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560adc44aca0 0x560adc450c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.767+0000 7f9312ffd640 1 -- 172.21.15.33:0/3547225615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560adc453880 msgr2=0x560adc4511b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:56.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.767+0000 7f9312ffd640 1 --2- 172.21.15.33:0/3547225615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560adc453880 0x560adc4511b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.767+0000 7f9312ffd640 1 -- 172.21.15.33:0/3547225615 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560adc1eaf70 con 0x560adc44a380 2024-03-19T03:01:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.767+0000 7f930bfff640 1 --2- 172.21.15.33:0/3547225615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560adc44aca0 0x560adc450c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:01:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.767+0000 7f9312ffd640 1 --2- 172.21.15.33:0/3547225615 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560adc44a380 0x560adc450730 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f930400e680 tx=0x7f93040099f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=737e229a6e2ccedf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.767+0000 7f92fa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:01:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.767+0000 7f92fa7fc640 1 -- 172.21.15.33:0/3547225615 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930401e050 con 0x560adc44a380 2024-03-19T03:01:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.768+0000 7f931c3e9740 1 -- 172.21.15.33:0/3547225615 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9304019ae0 con 0x560adc44a380 2024-03-19T03:01:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.768+0000 7f92fa7fc640 1 -- 172.21.15.33:0/3547225615 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f9304004020 con 0x560adc44a380 2024-03-19T03:01:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.768+0000 7f92fa7fc640 1 -- 172.21.15.33:0/3547225615 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9304032050 con 0x560adc44a380 2024-03-19T03:01:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.768+0000 7f931c3e9740 10 client.17496 fetch_fsmap learned FSMap version 25 2024-03-19T03:01:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.768+0000 7f931c3e9740 1 -- 172.21.15.33:0/3547225615 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x560adc082170 con 0x560adc44a380 2024-03-19T03:01:56.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.768+0000 7f92fa7fc640 1 -- 172.21.15.33:0/3547225615 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9304019ae0 con 0x560adc44a380 2024-03-19T03:01:56.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.768+0000 7f92fa7fc640 1 -- 172.21.15.33:0/3547225615 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f930400e720 con 0x560adc44a380 2024-03-19T03:01:56.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.768+0000 7f931c3e9740 10 client.17496 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:01:56.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.768+0000 7f931c3e9740 10 client.17496 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:01:56.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.768+0000 7f931c3e9740 20 client.17496 populate_metadata read hostname 'smithi033' 2024-03-19T03:01:56.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.768+0000 7f931c3e9740 1 --2- 172.21.15.33:0/3547225615 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x560adc463ae0 0x560adc465ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.768+0000 7f931c3e9740 4 client.17496 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:01:56.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.768+0000 7f931c3e9740 1 -- 172.21.15.33:0/3547225615 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560adc3bb6a0 con 0x560adc463ae0 2024-03-19T03:01:56.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.768+0000 7f930bfff640 1 --2- 172.21.15.33:0/3547225615 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x560adc463ae0 0x560adc465ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.769+0000 7f930bfff640 1 --2- 172.21.15.33:0/3547225615 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x560adc463ae0 0x560adc465ed0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52ff46ac9ef60f18 server_cookie=fea5d228767d508c in_seq=0 out_seq=0 2024-03-19T03:01:56.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.769+0000 7f92fa7fc640 10 client.17496 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:01:56.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.769+0000 7f92fa7fc640 1 -- 172.21.15.33:0/3547225615 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560adc3bb6a0 con 0x560adc463ae0 2024-03-19T03:01:56.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.769+0000 7f92fa7fc640 10 client.17496 handle_command_reply: tid=0 2024-03-19T03:01:56.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.769+0000 7f931c3e9740 1 client.17496 shutdown 2024-03-19T03:01:56.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.770+0000 7f931c3e9740 1 -- 172.21.15.33:0/3547225615 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560adc44a380 msgr2=0x560adc450730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:56.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.770+0000 7f931c3e9740 1 --2- 172.21.15.33:0/3547225615 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560adc44a380 0x560adc450730 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f930400e680 tx=0x7f93040099f0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.770+0000 7f931c3e9740 1 -- 172.21.15.33:0/3547225615 shutdown_connections 2024-03-19T03:01:56.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.770+0000 7f931c3e9740 1 --2- 172.21.15.33:0/3547225615 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x560adc463ae0 0x560adc465ed0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.770+0000 7f931c3e9740 1 --2- 172.21.15.33:0/3547225615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560adc44aca0 0x560adc450c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.770+0000 7f931c3e9740 1 --2- 172.21.15.33:0/3547225615 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560adc453880 0x560adc4511b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.770+0000 7f931c3e9740 1 --2- 172.21.15.33:0/3547225615 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560adc44a380 0x560adc450730 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.770+0000 7f92fa7fc640 0 client.17496 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:01:56.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.770+0000 7f931c3e9740 1 -- 172.21.15.33:0/3547225615 >> 172.21.15.33:0/3547225615 conn(0x560adc444ab0 msgr2=0x560adc446800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:56.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.770+0000 7f931c3e9740 1 -- 172.21.15.33:0/3547225615 shutdown_connections 2024-03-19T03:01:56.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.770+0000 7f931c3e9740 1 -- 172.21.15.33:0/3547225615 wait complete. 2024-03-19T03:01:56.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.770+0000 7f931c3e9740 20 client.17496 trim_cache size 0 max 16384 2024-03-19T03:01:56.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.788+0000 7f931c3e9740 1 Processor -- start 2024-03-19T03:01:56.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.788+0000 7f931c3e9740 1 -- start start 2024-03-19T03:01:56.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.788+0000 7f931c3e9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560adc44a380 0x560adc452e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.788+0000 7f931c3e9740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560adc44aca0 0x560adc45f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.788+0000 7f931c3e9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560adc453880 0x560adc4439b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.788+0000 7f931c3e9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f93141b1460 con 0x560adc44a380 2024-03-19T03:01:56.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.788+0000 7f931c3e9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f93141aed30 con 0x560adc453880 2024-03-19T03:01:56.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.788+0000 7f931c3e9740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f93141acba0 con 0x560adc44aca0 2024-03-19T03:01:56.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.788+0000 7f9312ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560adc44a380 0x560adc452e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.788+0000 7f9312ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560adc44a380 0x560adc452e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35944/0 (socket says 172.21.15.33:35944) 2024-03-19T03:01:56.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.788+0000 7f9312ffd640 1 -- 172.21.15.33:0/878087483 learned_addr learned my addr 172.21.15.33:0/878087483 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:56.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.788+0000 7f930bfff640 1 --2- 172.21.15.33:0/878087483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560adc44aca0 0x560adc45f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.788+0000 7f93137fe640 1 --2- 172.21.15.33:0/878087483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560adc453880 0x560adc4439b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.788+0000 7f9312ffd640 1 -- 172.21.15.33:0/878087483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560adc44aca0 msgr2=0x560adc45f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:56.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.789+0000 7f9312ffd640 1 --2- 172.21.15.33:0/878087483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560adc44aca0 0x560adc45f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.789+0000 7f9312ffd640 1 -- 172.21.15.33:0/878087483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560adc453880 msgr2=0x560adc4439b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:56.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.789+0000 7f9312ffd640 1 --2- 172.21.15.33:0/878087483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560adc453880 0x560adc4439b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.789+0000 7f9312ffd640 1 -- 172.21.15.33:0/878087483 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560adc466500 con 0x560adc44a380 2024-03-19T03:01:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.789+0000 7f9312ffd640 1 --2- 172.21.15.33:0/878087483 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560adc44a380 0x560adc452e60 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f930401c040 tx=0x7f93040196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8fc8385b04e61ad server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.789+0000 7f9308ff9640 1 -- 172.21.15.33:0/878087483 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930401e040 con 0x560adc44a380 2024-03-19T03:01:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.789+0000 7f9308ff9640 1 -- 172.21.15.33:0/878087483 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f930401f040 con 0x560adc44a380 2024-03-19T03:01:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.789+0000 7f9308ff9640 1 -- 172.21.15.33:0/878087483 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9304005040 con 0x560adc44a380 2024-03-19T03:01:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.790+0000 7f931c3e9740 1 -- 172.21.15.33:0/878087483 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560adc44a380 msgr2=0x560adc452e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.790+0000 7f931c3e9740 1 --2- 172.21.15.33:0/878087483 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560adc44a380 0x560adc452e60 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f930401c040 tx=0x7f93040196c0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.790+0000 7f931c3e9740 1 -- 172.21.15.33:0/878087483 shutdown_connections 2024-03-19T03:01:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.790+0000 7f931c3e9740 1 --2- 172.21.15.33:0/878087483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560adc44aca0 0x560adc45f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.790+0000 7f931c3e9740 1 --2- 172.21.15.33:0/878087483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560adc453880 0x560adc4439b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.790+0000 7f931c3e9740 1 --2- 172.21.15.33:0/878087483 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560adc44a380 0x560adc452e60 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.790+0000 7f931c3e9740 1 -- 172.21.15.33:0/878087483 >> 172.21.15.33:0/878087483 conn(0x560adc444ab0 msgr2=0x560adc4432a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.790+0000 7f931c3e9740 1 -- 172.21.15.33:0/878087483 shutdown_connections 2024-03-19T03:01:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.790+0000 7f931c3e9740 1 -- 172.21.15.33:0/878087483 wait complete. 2024-03-19T03:01:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.790+0000 7f931c3e9740 1 -- start start 2024-03-19T03:01:56.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.790+0000 7f931c3e9740 1 Processor -- start 2024-03-19T03:01:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.790+0000 7f931c3e9740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560adc44a380 0x560adc4544e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.790+0000 7f931c3e9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560adc44aca0 0x560adc454a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.790+0000 7f931c3e9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560adc453880 0x560adc454f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.790+0000 7f931c3e9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f93141ad1f0 con 0x560adc44aca0 2024-03-19T03:01:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.790+0000 7f931c3e9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f93141ae660 con 0x560adc453880 2024-03-19T03:01:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.790+0000 7f931c3e9740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f93141ad8c0 con 0x560adc44a380 2024-03-19T03:01:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.790+0000 7f931c3e9740 10 client.0 _ll_register_callbacks cb 0x7f930401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:01:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.790+0000 7f930bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560adc44aca0 0x560adc454a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.790+0000 7f9312ffd640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560adc44a380 0x560adc4544e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.791+0000 7f930bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560adc44aca0 0x560adc454a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35952/0 (socket says 172.21.15.33:35952) 2024-03-19T03:01:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.791+0000 7f93137fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560adc453880 0x560adc454f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.791+0000 7f930bfff640 1 -- 172.21.15.33:0/4239151694 learned_addr learned my addr 172.21.15.33:0/4239151694 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:56.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.791+0000 7f930bfff640 1 -- 172.21.15.33:0/4239151694 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560adc44a380 msgr2=0x560adc4544e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:56.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.791+0000 7f930bfff640 1 --2- 172.21.15.33:0/4239151694 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560adc44a380 0x560adc4544e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.791+0000 7f930bfff640 1 -- 172.21.15.33:0/4239151694 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560adc453880 msgr2=0x560adc454f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:56.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.791+0000 7f930bfff640 1 --2- 172.21.15.33:0/4239151694 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560adc453880 0x560adc454f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.791+0000 7f930bfff640 1 -- 172.21.15.33:0/4239151694 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560adc459a90 con 0x560adc44aca0 2024-03-19T03:01:56.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.791+0000 7f93137fe640 1 --2- 172.21.15.33:0/4239151694 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560adc453880 0x560adc454f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:56.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.791+0000 7f930bfff640 1 --2- 172.21.15.33:0/4239151694 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560adc44aca0 0x560adc454a20 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f92fc01c7c0 tx=0x7f92fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85ce3bea3648646f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:56.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.791+0000 7f92fa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:01:56.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.791+0000 7f92fa7fc640 1 -- 172.21.15.33:0/4239151694 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92fc004d30 con 0x560adc44aca0 2024-03-19T03:01:56.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.791+0000 7f92fa7fc640 1 -- 172.21.15.33:0/4239151694 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f92fc005040 con 0x560adc44aca0 2024-03-19T03:01:56.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.791+0000 7f931c3e9740 1 -- 172.21.15.33:0/4239151694 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f930401f040 con 0x560adc44aca0 2024-03-19T03:01:56.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.791+0000 7f92fa7fc640 1 -- 172.21.15.33:0/4239151694 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f92fc03c050 con 0x560adc44aca0 2024-03-19T03:01:56.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.791+0000 7f931c3e9740 10 client.17502 fetch_fsmap learned FSMap version 25 2024-03-19T03:01:56.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.791+0000 7f931c3e9740 1 -- 172.21.15.33:0/4239151694 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x560adc4ca0f0 con 0x560adc44aca0 2024-03-19T03:01:56.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.792+0000 7f92fa7fc640 1 -- 172.21.15.33:0/4239151694 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f930401f040 con 0x560adc44aca0 2024-03-19T03:01:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.792+0000 7f92fa7fc640 1 -- 172.21.15.33:0/4239151694 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f92fc02a690 con 0x560adc44aca0 2024-03-19T03:01:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.792+0000 7f931c3e9740 10 client.17502 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:01:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.792+0000 7f931c3e9740 10 client.17502 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:01:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.792+0000 7f931c3e9740 20 client.17502 populate_metadata read hostname 'smithi033' 2024-03-19T03:01:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.792+0000 7f931c3e9740 1 --2- 172.21.15.33:0/4239151694 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x560adc4ca850 0x560adc4ccc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.792+0000 7f931c3e9740 4 client.17502 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T03:01:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.792+0000 7f931c3e9740 1 -- 172.21.15.33:0/4239151694 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560adc3b59f0 con 0x560adc4ca850 2024-03-19T03:01:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.792+0000 7f9312ffd640 1 --2- 172.21.15.33:0/4239151694 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x560adc4ca850 0x560adc4ccc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.792+0000 7f9312ffd640 1 --2- 172.21.15.33:0/4239151694 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x560adc4ca850 0x560adc4ccc70 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3076c94ecb91b6f server_cookie=13ec9d92d7081b95 in_seq=0 out_seq=0 2024-03-19T03:01:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.792+0000 7f92fa7fc640 10 client.17502 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:01:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.793+0000 7f92fa7fc640 1 -- 172.21.15.33:0/4239151694 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560adc3b59f0 con 0x560adc4ca850 2024-03-19T03:01:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.793+0000 7f92fa7fc640 10 client.17502 handle_command_reply: tid=0 2024-03-19T03:01:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.793+0000 7f931c3e9740 1 client.17502 shutdown 2024-03-19T03:01:56.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.794+0000 7f931c3e9740 1 -- 172.21.15.33:0/4239151694 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560adc44aca0 msgr2=0x560adc454a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.794+0000 7f931c3e9740 1 --2- 172.21.15.33:0/4239151694 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560adc44aca0 0x560adc454a20 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f92fc01c7c0 tx=0x7f92fc004730 comp rx=0 tx=0).stop 2024-03-19T03:01:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.794+0000 7f931c3e9740 1 -- 172.21.15.33:0/4239151694 shutdown_connections 2024-03-19T03:01:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.794+0000 7f931c3e9740 1 --2- 172.21.15.33:0/4239151694 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x560adc4ca850 0x560adc4ccc70 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.794+0000 7f931c3e9740 1 --2- 172.21.15.33:0/4239151694 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x560adc453880 0x560adc454f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.794+0000 7f931c3e9740 1 --2- 172.21.15.33:0/4239151694 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x560adc44aca0 0x560adc454a20 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.794+0000 7f92fa7fc640 0 client.17502 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:01:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.794+0000 7f931c3e9740 1 --2- 172.21.15.33:0/4239151694 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x560adc44a380 0x560adc4544e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.794+0000 7f931c3e9740 1 -- 172.21.15.33:0/4239151694 >> 172.21.15.33:0/4239151694 conn(0x560adc444ab0 msgr2=0x560adc443b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.794+0000 7f931c3e9740 1 -- 172.21.15.33:0/4239151694 shutdown_connections 2024-03-19T03:01:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.794+0000 7f931c3e9740 1 -- 172.21.15.33:0/4239151694 wait complete. 2024-03-19T03:01:56.801 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.794+0000 7f931c3e9740 20 client.17502 trim_cache size 0 max 16384 2024-03-19T03:01:56.802 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T03:01:56.802 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-03-19T03:01:56.802 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "facfcb85-f169-4cc9-a7e3-6b970cd4a53c", 2024-03-19T03:01:56.802 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-03-19T03:01:56.802 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T03:01:56.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.794+0000 7f93197f4640 1 -- 172.21.15.33:0/2911511789 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f93000a2970 msgr2=0x7f93000a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:56.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.794+0000 7f93197f4640 1 --2- 172.21.15.33:0/2911511789 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f93000a2970 0x7f93000a4e30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9314077ff0 tx=0x7f92fc005f90 comp rx=0 tx=0).stop 2024-03-19T03:01:56.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.794+0000 7f93197f4640 1 -- 172.21.15.33:0/2911511789 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9314106750 msgr2=0x7f931419c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:56.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.794+0000 7f93197f4640 1 --2- 172.21.15.33:0/2911511789 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9314106750 0x7f931419c900 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f930c004af0 tx=0x7f930c035ce0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.795+0000 7f93197f4640 1 -- 172.21.15.33:0/2911511789 shutdown_connections 2024-03-19T03:01:56.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.795+0000 7f93197f4640 1 --2- 172.21.15.33:0/2911511789 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f93000a2970 0x7f93000a4e30 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.795+0000 7f93197f4640 1 --2- 172.21.15.33:0/2911511789 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f9314106750 0x7f931419c900 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.795+0000 7f93197f4640 1 --2- 172.21.15.33:0/2911511789 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9314101100 0x7f9314198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.795+0000 7f93197f4640 1 --2- 172.21.15.33:0/2911511789 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9314100740 0x7f9314198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.795+0000 7f93197f4640 1 -- 172.21.15.33:0/2911511789 >> 172.21.15.33:0/2911511789 conn(0x7f9314077880 msgr2=0x7f931406a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:56.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.795+0000 7f93197f4640 1 -- 172.21.15.33:0/2911511789 shutdown_connections 2024-03-19T03:01:56.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.795+0000 7f93197f4640 1 -- 172.21.15.33:0/2911511789 wait complete. 2024-03-19T03:01:56.807 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T03:01:56.807 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T03:01:56.954 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.951+0000 7f4d36509640 1 -- 172.21.15.33:0/991404676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d30100aa0 msgr2=0x7f4d3010e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:56.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.951+0000 7f4d36509640 1 --2- 172.21.15.33:0/991404676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d30100aa0 0x7f4d3010e6a0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f4d28009cd0 tx=0x7f4d2802c180 comp rx=0 tx=0).stop 2024-03-19T03:01:56.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.951+0000 7f4d36509640 1 -- 172.21.15.33:0/991404676 shutdown_connections 2024-03-19T03:01:56.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.951+0000 7f4d36509640 1 --2- 172.21.15.33:0/991404676 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d301000e0 0x7f4d30100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.951+0000 7f4d36509640 1 --2- 172.21.15.33:0/991404676 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d30100aa0 0x7f4d3010e6a0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.951+0000 7f4d36509640 1 --2- 172.21.15.33:0/991404676 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d30107a20 0x7f4d30107e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.951+0000 7f4d36509640 1 -- 172.21.15.33:0/991404676 >> 172.21.15.33:0/991404676 conn(0x7f4d30075a60 msgr2=0x7f4d30075e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:56.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.951+0000 7f4d36509640 1 -- 172.21.15.33:0/991404676 shutdown_connections 2024-03-19T03:01:56.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.951+0000 7f4d36509640 1 -- 172.21.15.33:0/991404676 wait complete. 2024-03-19T03:01:56.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.952+0000 7f4d36509640 1 Processor -- start 2024-03-19T03:01:56.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.952+0000 7f4d36509640 1 -- start start 2024-03-19T03:01:56.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.952+0000 7f4d36509640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d301000e0 0x7f4d30198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.955 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.952+0000 7f4d36509640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d30100aa0 0x7f4d30198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.952+0000 7f4d36509640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d30107a20 0x7f4d3019c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.952+0000 7f4d36509640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4d30111b30 con 0x7f4d30100aa0 2024-03-19T03:01:56.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.952+0000 7f4d36509640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4d301119b0 con 0x7f4d30107a20 2024-03-19T03:01:56.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.952+0000 7f4d36509640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4d30111cb0 con 0x7f4d301000e0 2024-03-19T03:01:56.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.952+0000 7f4d2f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d30100aa0 0x7f4d30198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.952+0000 7f4d2f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d30100aa0 0x7f4d30198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35978/0 (socket says 172.21.15.33:35978) 2024-03-19T03:01:56.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.952+0000 7f4d2f7fe640 1 -- 172.21.15.33:0/2937385174 learned_addr learned my addr 172.21.15.33:0/2937385174 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:56.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.952+0000 7f4d2ffff640 1 --2- 172.21.15.33:0/2937385174 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d301000e0 0x7f4d30198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.952+0000 7f4d34aa6640 1 --2- 172.21.15.33:0/2937385174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d30107a20 0x7f4d3019c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.952+0000 7f4d2f7fe640 1 -- 172.21.15.33:0/2937385174 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d301000e0 msgr2=0x7f4d30198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:56.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.952+0000 7f4d2f7fe640 1 --2- 172.21.15.33:0/2937385174 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d301000e0 0x7f4d30198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.952+0000 7f4d2f7fe640 1 -- 172.21.15.33:0/2937385174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d30107a20 msgr2=0x7f4d3019c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:56.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.952+0000 7f4d2f7fe640 1 --2- 172.21.15.33:0/2937385174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d30107a20 0x7f4d3019c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.952+0000 7f4d2f7fe640 1 -- 172.21.15.33:0/2937385174 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d3019d000 con 0x7f4d30100aa0 2024-03-19T03:01:56.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.953+0000 7f4d2f7fe640 1 --2- 172.21.15.33:0/2937385174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d30100aa0 0x7f4d30198550 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f4d2400e9f0 tx=0x7f4d2400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:56.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.953+0000 7f4d2d7fa640 1 -- 172.21.15.33:0/2937385174 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2400cd80 con 0x7f4d30100aa0 2024-03-19T03:01:56.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.953+0000 7f4d36509640 1 -- 172.21.15.33:0/2937385174 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d3019d2f0 con 0x7f4d30100aa0 2024-03-19T03:01:56.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.953+0000 7f4d2d7fa640 1 -- 172.21.15.33:0/2937385174 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4d240054d0 con 0x7f4d30100aa0 2024-03-19T03:01:56.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.953+0000 7f4d2d7fa640 1 -- 172.21.15.33:0/2937385174 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d24010490 con 0x7f4d30100aa0 2024-03-19T03:01:56.957 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.953+0000 7f4d36509640 1 -- 172.21.15.33:0/2937385174 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d301a4b80 con 0x7f4d30100aa0 2024-03-19T03:01:56.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.955+0000 7f4d2d7fa640 1 -- 172.21.15.33:0/2937385174 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f4d24005020 con 0x7f4d30100aa0 2024-03-19T03:01:56.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.955+0000 7f4d391d4740 1 Processor -- start 2024-03-19T03:01:56.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.955+0000 7f4d391d4740 1 -- start start 2024-03-19T03:01:56.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.955+0000 7f4d391d4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56264f762a30 0x56264f762e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.956+0000 7f4d391d4740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56264f763350 0x56264f76b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.956+0000 7f4d391d4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56264f76bf30 0x56264f76e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.956+0000 7f4d391d4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56264f6c6500 con 0x56264f762a30 2024-03-19T03:01:56.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.956+0000 7f4d391d4740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x56264f66ebe0 con 0x56264f763350 2024-03-19T03:01:56.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.956+0000 7f4d391d4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56264f6c25a0 con 0x56264f76bf30 2024-03-19T03:01:56.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.956+0000 7f4d34aa6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56264f762a30 0x56264f762e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.956+0000 7f4d34aa6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56264f762a30 0x56264f762e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35990/0 (socket says 172.21.15.33:35990) 2024-03-19T03:01:56.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.956+0000 7f4d34aa6640 1 -- 172.21.15.33:0/3398035381 learned_addr learned my addr 172.21.15.33:0/3398035381 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:56.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.956+0000 7f4d2ffff640 1 --2- 172.21.15.33:0/3398035381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56264f763350 0x56264f76b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.956+0000 7f4d2f7fe640 1 --2- 172.21.15.33:0/3398035381 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56264f76bf30 0x56264f76e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.956+0000 7f4d2d7fa640 1 --2- 172.21.15.33:0/2937385174 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4d180a2970 0x7f4d180a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.956+0000 7f4d34aa6640 1 --2- 172.21.15.33:0/2937385174 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4d180a2970 0x7f4d180a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.956+0000 7f4d2ffff640 1 -- 172.21.15.33:0/3398035381 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56264f76bf30 msgr2=0x56264f76e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:56.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.956+0000 7f4d2ffff640 1 --2- 172.21.15.33:0/3398035381 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56264f76bf30 0x56264f76e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.956+0000 7f4d2ffff640 1 -- 172.21.15.33:0/3398035381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56264f762a30 msgr2=0x56264f762e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:56.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.956+0000 7f4d2ffff640 1 --2- 172.21.15.33:0/3398035381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56264f762a30 0x56264f762e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.956+0000 7f4d2ffff640 1 -- 172.21.15.33:0/3398035381 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56264f690000 con 0x56264f763350 2024-03-19T03:01:56.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.956+0000 7f4d34aa6640 1 --2- 172.21.15.33:0/3398035381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56264f762a30 0x56264f762e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:56.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.956+0000 7f4d2d7fa640 1 -- 172.21.15.33:0/2937385174 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f4d240cf600 con 0x7f4d30100aa0 2024-03-19T03:01:56.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.956+0000 7f4d2ffff640 1 --2- 172.21.15.33:0/3398035381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56264f763350 0x56264f76b9f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4d20009c20 tx=0x7f4d20011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4dc3a0bf15cf8a93 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:56.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.956+0000 7f4d34aa6640 1 --2- 172.21.15.33:0/2937385174 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4d180a2970 0x7f4d180a4e30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4d28009430 tx=0x7f4d2802c180 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:56.967 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.956+0000 7f4d167fc640 1 -- 172.21.15.33:0/3398035381 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2001e070 con 0x56264f763350 2024-03-19T03:01:56.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.956+0000 7f4d167fc640 1 -- 172.21.15.33:0/3398035381 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4d20009dd0 con 0x56264f763350 2024-03-19T03:01:56.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.956+0000 7f4d167fc640 1 -- 172.21.15.33:0/3398035381 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d20018710 con 0x56264f763350 2024-03-19T03:01:56.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.957+0000 7f4d391d4740 1 -- 172.21.15.33:0/3398035381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56264f763350 msgr2=0x56264f76b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:56.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.957+0000 7f4d391d4740 1 --2- 172.21.15.33:0/3398035381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56264f763350 0x56264f76b9f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4d20009c20 tx=0x7f4d20011810 comp rx=0 tx=0).stop 2024-03-19T03:01:56.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.957+0000 7f4d391d4740 1 -- 172.21.15.33:0/3398035381 shutdown_connections 2024-03-19T03:01:56.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.957+0000 7f4d391d4740 1 --2- 172.21.15.33:0/3398035381 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56264f76bf30 0x56264f76e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.957+0000 7f4d391d4740 1 --2- 172.21.15.33:0/3398035381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56264f763350 0x56264f76b9f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.957+0000 7f4d391d4740 1 --2- 172.21.15.33:0/3398035381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56264f762a30 0x56264f762e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.957+0000 7f4d391d4740 1 -- 172.21.15.33:0/3398035381 >> 172.21.15.33:0/3398035381 conn(0x56264f75d160 msgr2=0x56264f75d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:56.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.957+0000 7f4d391d4740 1 -- 172.21.15.33:0/3398035381 shutdown_connections 2024-03-19T03:01:56.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.957+0000 7f4d391d4740 1 -- 172.21.15.33:0/3398035381 wait complete. 2024-03-19T03:01:56.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.957+0000 7f4d391d4740 1 -- start start 2024-03-19T03:01:56.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.958+0000 7f4d391d4740 1 Processor -- start 2024-03-19T03:01:56.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.958+0000 7f4d391d4740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56264f762a30 0x56264f768e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.958+0000 7f4d391d4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56264f763350 0x56264f7693c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.958+0000 7f4d391d4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56264f76bf30 0x56264f769900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.958+0000 7f4d391d4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56264f76ed20 con 0x56264f76bf30 2024-03-19T03:01:56.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.958+0000 7f4d391d4740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x56264f682160 con 0x56264f762a30 2024-03-19T03:01:56.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.958+0000 7f4d391d4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56264f76eea0 con 0x56264f763350 2024-03-19T03:01:56.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.958+0000 7f4d391d4740 10 client.0 _ll_register_callbacks cb 0x56264f6bea40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:01:56.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.958+0000 7f4d34aa6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56264f76bf30 0x56264f769900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.958+0000 7f4d2ffff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56264f762a30 0x56264f768e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.958+0000 7f4d34aa6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56264f76bf30 0x56264f769900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:35996/0 (socket says 172.21.15.33:35996) 2024-03-19T03:01:56.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.958+0000 7f4d34aa6640 1 -- 172.21.15.33:0/2914125779 learned_addr learned my addr 172.21.15.33:0/2914125779 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:56.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.958+0000 7f4d2f7fe640 1 --2- 172.21.15.33:0/2914125779 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56264f763350 0x56264f7693c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.958+0000 7f4d2ffff640 1 -- 172.21.15.33:0/2914125779 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56264f763350 msgr2=0x56264f7693c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:56.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.958+0000 7f4d2ffff640 1 --2- 172.21.15.33:0/2914125779 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56264f763350 0x56264f7693c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.958+0000 7f4d2ffff640 1 -- 172.21.15.33:0/2914125779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56264f76bf30 msgr2=0x56264f769900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:56.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.958+0000 7f4d2ffff640 1 --2- 172.21.15.33:0/2914125779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56264f76bf30 0x56264f769900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.958+0000 7f4d2ffff640 1 -- 172.21.15.33:0/2914125779 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56264f4a47d0 con 0x56264f762a30 2024-03-19T03:01:56.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.958+0000 7f4d34aa6640 1 --2- 172.21.15.33:0/2914125779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56264f76bf30 0x56264f769900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:01:56.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.959+0000 7f4d2ffff640 1 --2- 172.21.15.33:0/2914125779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56264f762a30 0x56264f768e80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4d2001c040 tx=0x7f4d20009850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=946e607f634e17b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:56.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.959+0000 7f4d0f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T03:01:56.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.959+0000 7f4d0f7fe640 1 -- 172.21.15.33:0/2914125779 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2001e040 con 0x56264f762a30 2024-03-19T03:01:56.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.959+0000 7f4d391d4740 1 -- 172.21.15.33:0/2914125779 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d20009dd0 con 0x56264f762a30 2024-03-19T03:01:56.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.959+0000 7f4d0f7fe640 1 -- 172.21.15.33:0/2914125779 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f4d20005180 con 0x56264f762a30 2024-03-19T03:01:56.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.959+0000 7f4d0f7fe640 1 -- 172.21.15.33:0/2914125779 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4d20032050 con 0x56264f762a30 2024-03-19T03:01:56.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.959+0000 7f4d391d4740 10 client.27335 fetch_fsmap learned FSMap version 25 2024-03-19T03:01:56.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.959+0000 7f4d391d4740 1 -- 172.21.15.33:0/2914125779 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56264f3da3f0 con 0x56264f762a30 2024-03-19T03:01:56.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.959+0000 7f4d0f7fe640 1 -- 172.21.15.33:0/2914125779 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4d20009dd0 con 0x56264f762a30 2024-03-19T03:01:56.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.959+0000 7f4d0f7fe640 1 -- 172.21.15.33:0/2914125779 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f4d20021030 con 0x56264f762a30 2024-03-19T03:01:56.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.959+0000 7f4d391d4740 10 client.27335 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:01:56.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.959+0000 7f4d391d4740 10 client.27335 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:01:56.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.959+0000 7f4d391d4740 20 client.27335 populate_metadata read hostname 'smithi033' 2024-03-19T03:01:56.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.959+0000 7f4d391d4740 1 --2- 172.21.15.33:0/2914125779 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x56264f77c1a0 0x56264f77e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.959+0000 7f4d391d4740 4 client.27335 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:01:56.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.959+0000 7f4d391d4740 1 -- 172.21.15.33:0/2914125779 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56264f6d3a40 con 0x56264f77c1a0 2024-03-19T03:01:56.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.959+0000 7f4d2f7fe640 1 --2- 172.21.15.33:0/2914125779 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x56264f77c1a0 0x56264f77e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.960+0000 7f4d2f7fe640 1 --2- 172.21.15.33:0/2914125779 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x56264f77c1a0 0x56264f77e590 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86ed7d7a87df518 server_cookie=60ef39646f866e94 in_seq=0 out_seq=0 2024-03-19T03:01:56.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.960+0000 7f4d0f7fe640 10 client.27335 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:01:56.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.960+0000 7f4d0f7fe640 1 -- 172.21.15.33:0/2914125779 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56264f6d3a40 con 0x56264f77c1a0 2024-03-19T03:01:56.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.960+0000 7f4d0f7fe640 10 client.27335 handle_command_reply: tid=0 2024-03-19T03:01:56.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.960+0000 7f4d391d4740 1 client.27335 shutdown 2024-03-19T03:01:56.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.961+0000 7f4d391d4740 1 -- 172.21.15.33:0/2914125779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56264f762a30 msgr2=0x56264f768e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:56.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.961+0000 7f4d391d4740 1 --2- 172.21.15.33:0/2914125779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56264f762a30 0x56264f768e80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4d2001c040 tx=0x7f4d20009850 comp rx=0 tx=0).stop 2024-03-19T03:01:56.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.961+0000 7f4d391d4740 1 -- 172.21.15.33:0/2914125779 shutdown_connections 2024-03-19T03:01:56.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.961+0000 7f4d391d4740 1 --2- 172.21.15.33:0/2914125779 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x56264f77c1a0 0x56264f77e590 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.961+0000 7f4d391d4740 1 --2- 172.21.15.33:0/2914125779 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56264f763350 0x56264f7693c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.961+0000 7f4d0f7fe640 0 client.27335 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:01:56.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.961+0000 7f4d391d4740 1 --2- 172.21.15.33:0/2914125779 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56264f76bf30 0x56264f769900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.961+0000 7f4d391d4740 1 --2- 172.21.15.33:0/2914125779 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56264f762a30 0x56264f768e80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.961+0000 7f4d391d4740 1 -- 172.21.15.33:0/2914125779 >> 172.21.15.33:0/2914125779 conn(0x56264f75d160 msgr2=0x56264f75eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:56.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.961+0000 7f4d391d4740 1 -- 172.21.15.33:0/2914125779 shutdown_connections 2024-03-19T03:01:56.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.961+0000 7f4d391d4740 1 -- 172.21.15.33:0/2914125779 wait complete. 2024-03-19T03:01:56.972 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.961+0000 7f4d391d4740 20 client.27335 trim_cache size 0 max 16384 2024-03-19T03:01:56.984 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.978+0000 7f4d391d4740 1 Processor -- start 2024-03-19T03:01:56.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.978+0000 7f4d391d4740 1 -- start start 2024-03-19T03:01:56.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.978+0000 7f4d391d4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56264f762a30 0x56264f76b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.978+0000 7f4d391d4740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56264f763350 0x56264f7643a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.978+0000 7f4d391d4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56264f76bf30 0x56264f777af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.978+0000 7f4d391d4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4d301b1540 con 0x56264f76bf30 2024-03-19T03:01:56.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.978+0000 7f4d391d4740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4d301aee10 con 0x56264f763350 2024-03-19T03:01:56.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.978+0000 7f4d391d4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4d301acc80 con 0x56264f762a30 2024-03-19T03:01:56.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.978+0000 7f4d34aa6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56264f76bf30 0x56264f777af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.978+0000 7f4d34aa6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56264f76bf30 0x56264f777af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36008/0 (socket says 172.21.15.33:36008) 2024-03-19T03:01:56.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.978+0000 7f4d34aa6640 1 -- 172.21.15.33:0/1769671183 learned_addr learned my addr 172.21.15.33:0/1769671183 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:56.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.978+0000 7f4d2ffff640 1 --2- 172.21.15.33:0/1769671183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56264f762a30 0x56264f76b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.978+0000 7f4d2f7fe640 1 --2- 172.21.15.33:0/1769671183 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56264f763350 0x56264f7643a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.985 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.978+0000 7f4d34aa6640 1 -- 172.21.15.33:0/1769671183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56264f762a30 msgr2=0x56264f76b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:56.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.979+0000 7f4d34aa6640 1 --2- 172.21.15.33:0/1769671183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56264f762a30 0x56264f76b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.979+0000 7f4d34aa6640 1 -- 172.21.15.33:0/1769671183 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56264f763350 msgr2=0x56264f7643a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:56.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.979+0000 7f4d34aa6640 1 --2- 172.21.15.33:0/1769671183 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56264f763350 0x56264f7643a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.979+0000 7f4d34aa6640 1 -- 172.21.15.33:0/1769671183 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56264f39a0f0 con 0x56264f76bf30 2024-03-19T03:01:56.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.979+0000 7f4d34aa6640 1 --2- 172.21.15.33:0/1769671183 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56264f76bf30 0x56264f777af0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f4d28032940 tx=0x7f4d28009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=162e7ba8b2b79242 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:56.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.979+0000 7f4d15ffb640 1 -- 172.21.15.33:0/1769671183 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d28054070 con 0x56264f76bf30 2024-03-19T03:01:56.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.979+0000 7f4d15ffb640 1 -- 172.21.15.33:0/1769671183 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4d28034bb0 con 0x56264f76bf30 2024-03-19T03:01:56.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.979+0000 7f4d15ffb640 1 -- 172.21.15.33:0/1769671183 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d28039040 con 0x56264f76bf30 2024-03-19T03:01:56.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.979+0000 7f4d391d4740 1 -- 172.21.15.33:0/1769671183 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56264f76bf30 msgr2=0x56264f777af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:56.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.979+0000 7f4d391d4740 1 --2- 172.21.15.33:0/1769671183 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56264f76bf30 0x56264f777af0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f4d28032940 tx=0x7f4d28009360 comp rx=0 tx=0).stop 2024-03-19T03:01:56.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.979+0000 7f4d391d4740 1 -- 172.21.15.33:0/1769671183 shutdown_connections 2024-03-19T03:01:56.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.979+0000 7f4d391d4740 1 --2- 172.21.15.33:0/1769671183 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56264f76bf30 0x56264f777af0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.979+0000 7f4d391d4740 1 --2- 172.21.15.33:0/1769671183 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56264f763350 0x56264f7643a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.986 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.979+0000 7f4d391d4740 1 --2- 172.21.15.33:0/1769671183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56264f762a30 0x56264f76b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.979+0000 7f4d391d4740 1 -- 172.21.15.33:0/1769671183 >> 172.21.15.33:0/1769671183 conn(0x56264f75d160 msgr2=0x56264f75b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:56.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.980+0000 7f4d391d4740 1 -- 172.21.15.33:0/1769671183 shutdown_connections 2024-03-19T03:01:56.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.980+0000 7f4d391d4740 1 -- 172.21.15.33:0/1769671183 wait complete. 2024-03-19T03:01:56.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.980+0000 7f4d391d4740 1 -- start start 2024-03-19T03:01:56.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.980+0000 7f4d391d4740 1 Processor -- start 2024-03-19T03:01:56.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.980+0000 7f4d391d4740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56264f762a30 0x56264f774120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.980+0000 7f4d391d4740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56264f763350 0x56264f76f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.980+0000 7f4d391d4740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56264f76bf30 0x56264f76fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.980+0000 7f4d391d4740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4d301ad2d0 con 0x56264f76bf30 2024-03-19T03:01:56.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.980+0000 7f4d391d4740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4d301ae740 con 0x56264f763350 2024-03-19T03:01:56.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.980+0000 7f4d391d4740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4d301ad9a0 con 0x56264f762a30 2024-03-19T03:01:56.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.980+0000 7f4d391d4740 10 client.0 _ll_register_callbacks cb 0x7f4d2001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:01:56.987 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.980+0000 7f4d34aa6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56264f76bf30 0x56264f76fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.980+0000 7f4d34aa6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56264f76bf30 0x56264f76fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:36018/0 (socket says 172.21.15.33:36018) 2024-03-19T03:01:56.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.980+0000 7f4d34aa6640 1 -- 172.21.15.33:0/307443737 learned_addr learned my addr 172.21.15.33:0/307443737 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:01:56.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.980+0000 7f4d2f7fe640 1 --2- 172.21.15.33:0/307443737 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56264f763350 0x56264f76f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.980+0000 7f4d2ffff640 1 --2- 172.21.15.33:0/307443737 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56264f762a30 0x56264f774120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.981+0000 7f4d34aa6640 1 -- 172.21.15.33:0/307443737 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56264f762a30 msgr2=0x56264f774120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:56.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.981+0000 7f4d34aa6640 1 --2- 172.21.15.33:0/307443737 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56264f762a30 0x56264f774120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.981+0000 7f4d34aa6640 1 -- 172.21.15.33:0/307443737 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56264f763350 msgr2=0x56264f76f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:01:56.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.981+0000 7f4d34aa6640 1 --2- 172.21.15.33:0/307443737 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56264f763350 0x56264f76f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.981+0000 7f4d34aa6640 1 -- 172.21.15.33:0/307443737 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56264f76ff70 con 0x56264f76bf30 2024-03-19T03:01:56.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.981+0000 7f4d34aa6640 1 --2- 172.21.15.33:0/307443737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56264f76bf30 0x56264f76fa30 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f4d280323c0 tx=0x7f4d280323f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=832613a36510682f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:01:56.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.981+0000 7f4d0f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:01:56.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.981+0000 7f4d0f7fe640 1 -- 172.21.15.33:0/307443737 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2805d060 con 0x56264f76bf30 2024-03-19T03:01:56.988 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.981+0000 7f4d0f7fe640 1 -- 172.21.15.33:0/307443737 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9496+0+0 (secure 0 0 0) 0x7f4d2804fd30 con 0x56264f76bf30 2024-03-19T03:01:56.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.981+0000 7f4d391d4740 1 -- 172.21.15.33:0/307443737 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d28034bb0 con 0x56264f76bf30 2024-03-19T03:01:56.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.981+0000 7f4d0f7fe640 1 -- 172.21.15.33:0/307443737 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4d28064050 con 0x56264f76bf30 2024-03-19T03:01:56.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.981+0000 7f4d391d4740 10 client.17520 fetch_fsmap learned FSMap version 25 2024-03-19T03:01:56.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.981+0000 7f4d391d4740 1 -- 172.21.15.33:0/307443737 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56264f7e2100 con 0x56264f76bf30 2024-03-19T03:01:56.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.981+0000 7f4d0f7fe640 1 -- 172.21.15.33:0/307443737 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4d28034bb0 con 0x56264f76bf30 2024-03-19T03:01:56.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.981+0000 7f4d0f7fe640 1 -- 172.21.15.33:0/307443737 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f4d2803f070 con 0x56264f76bf30 2024-03-19T03:01:56.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.982+0000 7f4d391d4740 10 client.17520 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:01:56.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.982+0000 7f4d391d4740 10 client.17520 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:01:56.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.982+0000 7f4d391d4740 20 client.17520 populate_metadata read hostname 'smithi033' 2024-03-19T03:01:56.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.982+0000 7f4d391d4740 1 --2- 172.21.15.33:0/307443737 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x56264f7e27a0 0x56264f7e4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:01:56.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.982+0000 7f4d391d4740 4 client.17520 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T03:01:56.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.982+0000 7f4d391d4740 1 -- 172.21.15.33:0/307443737 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56264f6cdd90 con 0x56264f7e27a0 2024-03-19T03:01:56.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.982+0000 7f4d2ffff640 1 --2- 172.21.15.33:0/307443737 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x56264f7e27a0 0x56264f7e4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:01:56.989 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.982+0000 7f4d2ffff640 1 --2- 172.21.15.33:0/307443737 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x56264f7e27a0 0x56264f7e4bc0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d288649ca7025f5 server_cookie=a6dc89d4964a4d46 in_seq=0 out_seq=0 2024-03-19T03:01:56.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.982+0000 7f4d0f7fe640 10 client.17520 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:01:56.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.982+0000 7f4d0f7fe640 1 -- 172.21.15.33:0/307443737 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56264f6cdd90 con 0x56264f7e27a0 2024-03-19T03:01:56.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.982+0000 7f4d0f7fe640 10 client.17520 handle_command_reply: tid=0 2024-03-19T03:01:56.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.982+0000 7f4d391d4740 1 client.17520 shutdown 2024-03-19T03:01:56.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.982+0000 7f4d391d4740 1 -- 172.21.15.33:0/307443737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56264f76bf30 msgr2=0x56264f76fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:56.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.982+0000 7f4d391d4740 1 --2- 172.21.15.33:0/307443737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56264f76bf30 0x56264f76fa30 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f4d280323c0 tx=0x7f4d280323f0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.982+0000 7f4d391d4740 1 -- 172.21.15.33:0/307443737 shutdown_connections 2024-03-19T03:01:56.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.982+0000 7f4d391d4740 1 --2- 172.21.15.33:0/307443737 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x56264f7e27a0 0x56264f7e4bc0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.982+0000 7f4d391d4740 1 --2- 172.21.15.33:0/307443737 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56264f76bf30 0x56264f76fa30 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.982+0000 7f4d391d4740 1 --2- 172.21.15.33:0/307443737 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56264f763350 0x56264f76f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.982+0000 7f4d0f7fe640 0 client.17520 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:01:56.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.982+0000 7f4d391d4740 1 --2- 172.21.15.33:0/307443737 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56264f762a30 0x56264f774120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.982+0000 7f4d391d4740 1 -- 172.21.15.33:0/307443737 >> 172.21.15.33:0/307443737 conn(0x56264f75d160 msgr2=0x56264f75c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:56.990 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.983+0000 7f4d391d4740 1 -- 172.21.15.33:0/307443737 shutdown_connections 2024-03-19T03:01:56.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.983+0000 7f4d391d4740 1 -- 172.21.15.33:0/307443737 wait complete. 2024-03-19T03:01:56.991 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.983+0000 7f4d391d4740 20 client.17520 trim_cache size 0 max 16384 2024-03-19T03:01:56.991 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T03:01:56.991 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T03:01:56.991 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-03-19T03:01:56.992 INFO:teuthology.orchestra.run.smithi033.stdout: "facfcb85-f169-4cc9-a7e3-6b970cd4a53c": { 2024-03-19T03:01:56.992 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-03-19T03:01:56.992 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "facfcb85-f169-4cc9-a7e3-6b970cd4a53c", 2024-03-19T03:01:56.992 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-03-19T03:01:56.992 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:01:56.992 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:01:56.992 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T03:01:56.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.983+0000 7f4d36509640 1 -- 172.21.15.33:0/2937385174 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4d180a2970 msgr2=0x7f4d180a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:56.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.983+0000 7f4d36509640 1 --2- 172.21.15.33:0/2937385174 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4d180a2970 0x7f4d180a4e30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4d28009430 tx=0x7f4d2802c180 comp rx=0 tx=0).stop 2024-03-19T03:01:56.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.983+0000 7f4d36509640 1 -- 172.21.15.33:0/2937385174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d30100aa0 msgr2=0x7f4d30198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:01:56.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.983+0000 7f4d36509640 1 --2- 172.21.15.33:0/2937385174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d30100aa0 0x7f4d30198550 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f4d2400e9f0 tx=0x7f4d2400edf0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.983+0000 7f4d36509640 1 -- 172.21.15.33:0/2937385174 shutdown_connections 2024-03-19T03:01:56.993 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.983+0000 7f4d36509640 1 --2- 172.21.15.33:0/2937385174 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4d180a2970 0x7f4d180a4e30 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.983+0000 7f4d36509640 1 --2- 172.21.15.33:0/2937385174 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d30107a20 0x7f4d3019c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.983+0000 7f4d36509640 1 --2- 172.21.15.33:0/2937385174 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d30100aa0 0x7f4d30198550 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.983+0000 7f4d36509640 1 --2- 172.21.15.33:0/2937385174 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d301000e0 0x7f4d30198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:01:56.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.983+0000 7f4d36509640 1 -- 172.21.15.33:0/2937385174 >> 172.21.15.33:0/2937385174 conn(0x7f4d30075a60 msgr2=0x7f4d300fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:01:56.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.983+0000 7f4d36509640 1 -- 172.21.15.33:0/2937385174 shutdown_connections 2024-03-19T03:01:56.994 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:01:56.983+0000 7f4d36509640 1 -- 172.21.15.33:0/2937385174 wait complete. 2024-03-19T03:01:56.995 INFO:tasks.cephfs.filesystem:scrub status for tag:facfcb85-f169-4cc9-a7e3-6b970cd4a53c - {'path': '/', 'tag': 'facfcb85-f169-4cc9-a7e3-6b970cd4a53c', 'options': 'recursive,force'} 2024-03-19T03:01:58.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:58 smithi096 ceph-mon[35946]: scrub queued for path: / 2024-03-19T03:01:58.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:58 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T03:01:58.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:58 smithi096 ceph-mon[35946]: scrub summary: active paths [/] 2024-03-19T03:01:58.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:58 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T03:01:58.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:58 smithi096 ceph-mon[35946]: pgmap v262: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:01:58.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:01:58 smithi096 ceph-mon[35946]: scrub summary: idle 2024-03-19T03:01:58.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:58 smithi155 ceph-mon[35804]: scrub queued for path: / 2024-03-19T03:01:58.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:58 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T03:01:58.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:58 smithi155 ceph-mon[35804]: scrub summary: active paths [/] 2024-03-19T03:01:58.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:58 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T03:01:58.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:58 smithi155 ceph-mon[35804]: pgmap v262: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:01:58.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:01:58 smithi155 ceph-mon[35804]: scrub summary: idle 2024-03-19T03:01:58.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:58 smithi033 ceph-mon[31973]: scrub queued for path: / 2024-03-19T03:01:58.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:58 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T03:01:58.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:58 smithi033 ceph-mon[31973]: scrub summary: active paths [/] 2024-03-19T03:01:58.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:58 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T03:01:58.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:58 smithi033 ceph-mon[31973]: pgmap v262: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:01:58.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:01:58 smithi033 ceph-mon[31973]: scrub summary: idle 2024-03-19T03:02:00.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:00 smithi096 ceph-mon[35946]: pgmap v263: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:02:00.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:00 smithi155 ceph-mon[35804]: pgmap v263: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:02:00.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:00 smithi033 ceph-mon[31973]: pgmap v263: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:02:02.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:02 smithi096 ceph-mon[35946]: pgmap v264: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:02:02.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:02 smithi155 ceph-mon[35804]: pgmap v264: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:02:02.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:02 smithi033 ceph-mon[31973]: pgmap v264: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:02:04.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:04 smithi096 ceph-mon[35946]: pgmap v265: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-19T03:02:04.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:04 smithi155 ceph-mon[35804]: pgmap v265: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-19T03:02:04.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:04 smithi033 ceph-mon[31973]: pgmap v265: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-19T03:02:06.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:06 smithi096 ceph-mon[35946]: pgmap v266: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-19T03:02:06.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:06 smithi155 ceph-mon[35804]: pgmap v266: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-19T03:02:06.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:06 smithi033 ceph-mon[31973]: pgmap v266: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-19T03:02:08.580 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:08 smithi033 ceph-mon[31973]: pgmap v267: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-19T03:02:08.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:08 smithi096 ceph-mon[35946]: pgmap v267: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-19T03:02:08.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:08 smithi155 ceph-mon[35804]: pgmap v267: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-19T03:02:09.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:09 smithi096 ceph-mon[35946]: osdmap e103: 12 total, 12 up, 12 in 2024-03-19T03:02:09.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:09 smithi155 ceph-mon[35804]: osdmap e103: 12 total, 12 up, 12 in 2024-03-19T03:02:09.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:09 smithi033 ceph-mon[31973]: osdmap e103: 12 total, 12 up, 12 in 2024-03-19T03:02:10.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:10 smithi096 ceph-mon[35946]: pgmap v269: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-19T03:02:10.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:10 smithi155 ceph-mon[35804]: pgmap v269: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-19T03:02:10.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:10 smithi033 ceph-mon[31973]: pgmap v269: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-19T03:02:12.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:12 smithi096 ceph-mon[35946]: pgmap v270: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-19T03:02:12.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:12 smithi155 ceph-mon[35804]: pgmap v270: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-19T03:02:12.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:12 smithi033 ceph-mon[31973]: pgmap v270: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-19T03:02:14.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:14 smithi096 ceph-mon[35946]: pgmap v271: 129 pgs: 129 active+clean; 1.9 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-19T03:02:14.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:14 smithi155 ceph-mon[35804]: pgmap v271: 129 pgs: 129 active+clean; 1.9 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-19T03:02:14.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:14 smithi033 ceph-mon[31973]: pgmap v271: 129 pgs: 129 active+clean; 1.9 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-19T03:02:16.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:16 smithi155 ceph-mon[35804]: pgmap v272: 129 pgs: 129 active+clean; 1.9 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-19T03:02:16.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:16 smithi155 ceph-mon[35804]: osdmap e104: 12 total, 12 up, 12 in 2024-03-19T03:02:16.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:16 smithi033 ceph-mon[31973]: pgmap v272: 129 pgs: 129 active+clean; 1.9 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-19T03:02:16.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:16 smithi033 ceph-mon[31973]: osdmap e104: 12 total, 12 up, 12 in 2024-03-19T03:02:16.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:16 smithi096 ceph-mon[35946]: pgmap v272: 129 pgs: 129 active+clean; 1.9 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-19T03:02:16.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:16 smithi096 ceph-mon[35946]: osdmap e104: 12 total, 12 up, 12 in 2024-03-19T03:02:18.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:18 smithi155 ceph-mon[35804]: pgmap v274: 129 pgs: 129 active+clean; 1.9 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 116 B/s wr, 3 op/s 2024-03-19T03:02:18.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:18 smithi033 ceph-mon[31973]: pgmap v274: 129 pgs: 129 active+clean; 1.9 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 116 B/s wr, 3 op/s 2024-03-19T03:02:18.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:18 smithi096 ceph-mon[35946]: pgmap v274: 129 pgs: 129 active+clean; 1.9 MiB data, 786 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 116 B/s wr, 3 op/s 2024-03-19T03:02:20.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:20 smithi155 ceph-mon[35804]: pgmap v275: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-19T03:02:20.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:20 smithi033 ceph-mon[31973]: pgmap v275: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-19T03:02:20.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:20 smithi096 ceph-mon[35946]: pgmap v275: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-19T03:02:22.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:22 smithi155 ceph-mon[35804]: pgmap v276: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-19T03:02:22.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:22 smithi033 ceph-mon[31973]: pgmap v276: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-19T03:02:22.718 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:22 smithi096 ceph-mon[35946]: pgmap v276: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-19T03:02:24.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:24 smithi155 ceph-mon[35804]: pgmap v277: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-19T03:02:24.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:24 smithi033 ceph-mon[31973]: pgmap v277: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-19T03:02:24.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:24 smithi096 ceph-mon[35946]: pgmap v277: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-19T03:02:26.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:26 smithi155 ceph-mon[35804]: pgmap v278: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-19T03:02:26.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:26 smithi033 ceph-mon[31973]: pgmap v278: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-19T03:02:26.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:26 smithi096 ceph-mon[35946]: pgmap v278: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-19T03:02:26.996 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T03:02:26.997 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T03:02:27.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.114+0000 7f826a762640 1 -- 172.21.15.33:0/1406722620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f826410be30 msgr2=0x7f826410e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.114+0000 7f826a762640 1 --2- 172.21.15.33:0/1406722620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f826410be30 0x7f826410e260 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f825c009ca0 tx=0x7f825c02c180 comp rx=0 tx=0).stop 2024-03-19T03:02:27.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.114+0000 7f826a762640 1 -- 172.21.15.33:0/1406722620 shutdown_connections 2024-03-19T03:02:27.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.114+0000 7f826a762640 1 --2- 172.21.15.33:0/1406722620 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8264103380 0x7f826410b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.114+0000 7f826a762640 1 --2- 172.21.15.33:0/1406722620 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f826410be30 0x7f826410e260 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.114+0000 7f826a762640 1 --2- 172.21.15.33:0/1406722620 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8264102a60 0x7f8264102e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.114+0000 7f826a762640 1 -- 172.21.15.33:0/1406722620 >> 172.21.15.33:0/1406722620 conn(0x7f8264076aa0 msgr2=0x7f8264076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:27.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.114+0000 7f826a762640 1 -- 172.21.15.33:0/1406722620 shutdown_connections 2024-03-19T03:02:27.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.114+0000 7f826a762640 1 -- 172.21.15.33:0/1406722620 wait complete. 2024-03-19T03:02:27.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.115+0000 7f826a762640 1 Processor -- start 2024-03-19T03:02:27.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.115+0000 7f826a762640 1 -- start start 2024-03-19T03:02:27.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.115+0000 7f826a762640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8264102a60 0x7f82641006b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.115+0000 7f826a762640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8264103380 0x7f8264100bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.115+0000 7f826a762640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f826410be30 0x7f8264101130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.115+0000 7f826a762640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8264111a70 con 0x7f8264102a60 2024-03-19T03:02:27.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.115+0000 7f826a762640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f82641118f0 con 0x7f826410be30 2024-03-19T03:02:27.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.115+0000 7f826a762640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8264111bf0 con 0x7f8264103380 2024-03-19T03:02:27.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.115+0000 7f8263fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8264102a60 0x7f82641006b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.116+0000 7f8263fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8264102a60 0x7f82641006b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38462/0 (socket says 172.21.15.33:38462) 2024-03-19T03:02:27.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.116+0000 7f8263fff640 1 -- 172.21.15.33:0/3444278979 learned_addr learned my addr 172.21.15.33:0/3444278979 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:27.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.116+0000 7f82637fe640 1 --2- 172.21.15.33:0/3444278979 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8264103380 0x7f8264100bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.116+0000 7f8268cff640 1 --2- 172.21.15.33:0/3444278979 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f826410be30 0x7f8264101130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.119 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.116+0000 7f8263fff640 1 -- 172.21.15.33:0/3444278979 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8264103380 msgr2=0x7f8264100bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.116+0000 7f8263fff640 1 --2- 172.21.15.33:0/3444278979 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8264103380 0x7f8264100bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.116+0000 7f8263fff640 1 -- 172.21.15.33:0/3444278979 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f826410be30 msgr2=0x7f8264101130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.116+0000 7f8263fff640 1 --2- 172.21.15.33:0/3444278979 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f826410be30 0x7f8264101130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.116+0000 7f8263fff640 1 -- 172.21.15.33:0/3444278979 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82641a6450 con 0x7f8264102a60 2024-03-19T03:02:27.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.116+0000 7f8263fff640 1 --2- 172.21.15.33:0/3444278979 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8264102a60 0x7f82641006b0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f825400ea60 tx=0x7f825400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:27.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.116+0000 7f82617fa640 1 -- 172.21.15.33:0/3444278979 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825400cde0 con 0x7f8264102a60 2024-03-19T03:02:27.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.116+0000 7f826a762640 1 -- 172.21.15.33:0/3444278979 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82641a6740 con 0x7f8264102a60 2024-03-19T03:02:27.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.116+0000 7f82617fa640 1 -- 172.21.15.33:0/3444278979 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f82540035f0 con 0x7f8264102a60 2024-03-19T03:02:27.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.116+0000 7f82617fa640 1 -- 172.21.15.33:0/3444278979 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8254010690 con 0x7f8264102a60 2024-03-19T03:02:27.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.116+0000 7f826a762640 1 -- 172.21.15.33:0/3444278979 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82641a6c90 con 0x7f8264102a60 2024-03-19T03:02:27.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.119+0000 7f82617fa640 1 -- 172.21.15.33:0/3444278979 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f82540037a0 con 0x7f8264102a60 2024-03-19T03:02:27.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.119+0000 7f826d42d740 1 Processor -- start 2024-03-19T03:02:27.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.119+0000 7f826d42d740 1 -- start start 2024-03-19T03:02:27.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.119+0000 7f826d42d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5567e2fb4a30 0x5567e2fb4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.119+0000 7f826d42d740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5567e2fb5350 0x5567e2fbd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.119+0000 7f826d42d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5567e2fbdf30 0x5567e2fc0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.119+0000 7f826d42d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5567e2f18500 con 0x5567e2fb4a30 2024-03-19T03:02:27.127 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.119+0000 7f826d42d740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5567e2ec0be0 con 0x5567e2fb5350 2024-03-19T03:02:27.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.119+0000 7f826d42d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5567e2f145a0 con 0x5567e2fbdf30 2024-03-19T03:02:27.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.120+0000 7f8268cff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5567e2fb4a30 0x5567e2fb4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.120+0000 7f8268cff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5567e2fb4a30 0x5567e2fb4e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38472/0 (socket says 172.21.15.33:38472) 2024-03-19T03:02:27.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.120+0000 7f8268cff640 1 -- 172.21.15.33:0/4228959118 learned_addr learned my addr 172.21.15.33:0/4228959118 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:27.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.120+0000 7f8263fff640 1 --2- 172.21.15.33:0/4228959118 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5567e2fb5350 0x5567e2fbd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.120+0000 7f82637fe640 1 --2- 172.21.15.33:0/4228959118 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5567e2fbdf30 0x5567e2fc0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.120+0000 7f82617fa640 1 --2- 172.21.15.33:0/3444278979 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f824c0a29a0 0x7f824c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.120+0000 7f8268cff640 1 -- 172.21.15.33:0/4228959118 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5567e2fbdf30 msgr2=0x5567e2fc0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:27.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.120+0000 7f8268cff640 1 --2- 172.21.15.33:0/4228959118 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5567e2fbdf30 0x5567e2fc0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.120+0000 7f8268cff640 1 -- 172.21.15.33:0/4228959118 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5567e2fb5350 msgr2=0x5567e2fbd9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:27.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.120+0000 7f8268cff640 1 --2- 172.21.15.33:0/4228959118 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5567e2fb5350 0x5567e2fbd9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.120+0000 7f8268cff640 1 -- 172.21.15.33:0/4228959118 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567e2ee2000 con 0x5567e2fb4a30 2024-03-19T03:02:27.128 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.120+0000 7f82617fa640 1 -- 172.21.15.33:0/3444278979 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f82540cf4d0 con 0x7f8264102a60 2024-03-19T03:02:27.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.120+0000 7f8268cff640 1 --2- 172.21.15.33:0/3444278979 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f824c0a29a0 0x7f824c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.120+0000 7f8268cff640 1 --2- 172.21.15.33:0/4228959118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5567e2fb4a30 0x5567e2fb4e10 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f825c004af0 tx=0x7f825c009980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec60b45ad04277e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:27.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.120+0000 7f824a7fc640 1 -- 172.21.15.33:0/4228959118 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825c039040 con 0x5567e2fb4a30 2024-03-19T03:02:27.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.120+0000 7f824a7fc640 1 -- 172.21.15.33:0/4228959118 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f825c0325f0 con 0x5567e2fb4a30 2024-03-19T03:02:27.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.120+0000 7f824a7fc640 1 -- 172.21.15.33:0/4228959118 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825c034680 con 0x5567e2fb4a30 2024-03-19T03:02:27.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.120+0000 7f8268cff640 1 --2- 172.21.15.33:0/3444278979 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f824c0a29a0 0x7f824c0a4e60 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f825c04c000 tx=0x7f825c045000 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:27.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.121+0000 7f826d42d740 1 -- 172.21.15.33:0/4228959118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5567e2fb4a30 msgr2=0x5567e2fb4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.121+0000 7f826d42d740 1 --2- 172.21.15.33:0/4228959118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5567e2fb4a30 0x5567e2fb4e10 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f825c004af0 tx=0x7f825c009980 comp rx=0 tx=0).stop 2024-03-19T03:02:27.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.121+0000 7f826d42d740 1 -- 172.21.15.33:0/4228959118 shutdown_connections 2024-03-19T03:02:27.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.121+0000 7f826d42d740 1 --2- 172.21.15.33:0/4228959118 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5567e2fbdf30 0x5567e2fc0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.121+0000 7f826d42d740 1 --2- 172.21.15.33:0/4228959118 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5567e2fb5350 0x5567e2fbd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.121+0000 7f826d42d740 1 --2- 172.21.15.33:0/4228959118 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5567e2fb4a30 0x5567e2fb4e10 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.121+0000 7f826d42d740 1 -- 172.21.15.33:0/4228959118 >> 172.21.15.33:0/4228959118 conn(0x5567e2faf160 msgr2=0x5567e2faf540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:27.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.121+0000 7f826d42d740 1 -- 172.21.15.33:0/4228959118 shutdown_connections 2024-03-19T03:02:27.129 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.121+0000 7f826d42d740 1 -- 172.21.15.33:0/4228959118 wait complete. 2024-03-19T03:02:27.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.122+0000 7f826d42d740 1 -- start start 2024-03-19T03:02:27.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.122+0000 7f826d42d740 1 Processor -- start 2024-03-19T03:02:27.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.122+0000 7f826d42d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5567e2fb4a30 0x5567e2fbaf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.122+0000 7f826d42d740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5567e2fb5350 0x5567e2fbb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.122+0000 7f826d42d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5567e2fbdf30 0x5567e2fbb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.122+0000 7f826d42d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5567e2fc0d20 con 0x5567e2fbdf30 2024-03-19T03:02:27.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.122+0000 7f826d42d740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5567e2ed4160 con 0x5567e2fb5350 2024-03-19T03:02:27.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.122+0000 7f826d42d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5567e2fc0ea0 con 0x5567e2fb4a30 2024-03-19T03:02:27.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.122+0000 7f826d42d740 10 client.0 _ll_register_callbacks cb 0x5567e2f10a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:02:27.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.122+0000 7f8268cff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5567e2fbdf30 0x5567e2fbb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.122+0000 7f8263fff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5567e2fb4a30 0x5567e2fbaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.122+0000 7f82637fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5567e2fb5350 0x5567e2fbb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.122+0000 7f8268cff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5567e2fbdf30 0x5567e2fbb9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38474/0 (socket says 172.21.15.33:38474) 2024-03-19T03:02:27.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.122+0000 7f8268cff640 1 -- 172.21.15.33:0/3375701937 learned_addr learned my addr 172.21.15.33:0/3375701937 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:27.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.123+0000 7f8268cff640 1 -- 172.21.15.33:0/3375701937 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5567e2fb4a30 msgr2=0x5567e2fbaf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:27.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.123+0000 7f8268cff640 1 --2- 172.21.15.33:0/3375701937 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5567e2fb4a30 0x5567e2fbaf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.123+0000 7f8268cff640 1 -- 172.21.15.33:0/3375701937 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5567e2fb5350 msgr2=0x5567e2fbb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:27.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.123+0000 7f8268cff640 1 --2- 172.21.15.33:0/3375701937 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5567e2fb5350 0x5567e2fbb4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.123+0000 7f8268cff640 1 -- 172.21.15.33:0/3375701937 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567e2cf67d0 con 0x5567e2fbdf30 2024-03-19T03:02:27.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.123+0000 7f82637fe640 1 --2- 172.21.15.33:0/3375701937 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5567e2fb5350 0x5567e2fbb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:02:27.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.123+0000 7f8268cff640 1 --2- 172.21.15.33:0/3375701937 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5567e2fbdf30 0x5567e2fbb9f0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f825c003b20 tx=0x7f825c009540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=412be865ce9e6749 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:27.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.123+0000 7f82437fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:02:27.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.123+0000 7f82437fe640 1 -- 172.21.15.33:0/3375701937 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825c039070 con 0x5567e2fbdf30 2024-03-19T03:02:27.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.123+0000 7f82437fe640 1 -- 172.21.15.33:0/3375701937 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f825c004690 con 0x5567e2fbdf30 2024-03-19T03:02:27.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.123+0000 7f826d42d740 1 -- 172.21.15.33:0/3375701937 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f825c0325f0 con 0x5567e2fbdf30 2024-03-19T03:02:27.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.123+0000 7f82437fe640 1 -- 172.21.15.33:0/3375701937 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f825c003e50 con 0x5567e2fbdf30 2024-03-19T03:02:27.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.123+0000 7f826d42d740 10 client.17532 fetch_fsmap learned FSMap version 25 2024-03-19T03:02:27.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.123+0000 7f826d42d740 1 -- 172.21.15.33:0/3375701937 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5567e2c2c3f0 con 0x5567e2fbdf30 2024-03-19T03:02:27.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.123+0000 7f82437fe640 1 -- 172.21.15.33:0/3375701937 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f825c0325f0 con 0x5567e2fbdf30 2024-03-19T03:02:27.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.124+0000 7f82437fe640 1 -- 172.21.15.33:0/3375701937 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f825c03f040 con 0x5567e2fbdf30 2024-03-19T03:02:27.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.124+0000 7f826d42d740 10 client.17532 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:02:27.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.124+0000 7f826d42d740 10 client.17532 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:02:27.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.124+0000 7f826d42d740 20 client.17532 populate_metadata read hostname 'smithi033' 2024-03-19T03:02:27.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.124+0000 7f826d42d740 1 --2- 172.21.15.33:0/3375701937 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5567e2fce160 0x5567e2fd0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.124+0000 7f826d42d740 4 client.17532 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:02:27.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.124+0000 7f826d42d740 1 -- 172.21.15.33:0/3375701937 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5567e2f25a40 con 0x5567e2fce160 2024-03-19T03:02:27.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.124+0000 7f8263fff640 1 --2- 172.21.15.33:0/3375701937 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5567e2fce160 0x5567e2fd0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.124+0000 7f8263fff640 1 --2- 172.21.15.33:0/3375701937 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5567e2fce160 0x5567e2fd0550 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b39d9d15931856a server_cookie=9566a68336e3d14e in_seq=0 out_seq=0 2024-03-19T03:02:27.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.124+0000 7f82437fe640 10 client.17532 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:02:27.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.125+0000 7f82437fe640 1 -- 172.21.15.33:0/3375701937 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5567e2f25a40 con 0x5567e2fce160 2024-03-19T03:02:27.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.125+0000 7f82437fe640 10 client.17532 handle_command_reply: tid=0 2024-03-19T03:02:27.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.125+0000 7f826d42d740 1 client.17532 shutdown 2024-03-19T03:02:27.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.125+0000 7f826d42d740 1 -- 172.21.15.33:0/3375701937 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5567e2fbdf30 msgr2=0x5567e2fbb9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.125+0000 7f826d42d740 1 --2- 172.21.15.33:0/3375701937 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5567e2fbdf30 0x5567e2fbb9f0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f825c003b20 tx=0x7f825c009540 comp rx=0 tx=0).stop 2024-03-19T03:02:27.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.125+0000 7f826d42d740 1 -- 172.21.15.33:0/3375701937 shutdown_connections 2024-03-19T03:02:27.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.125+0000 7f826d42d740 1 --2- 172.21.15.33:0/3375701937 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5567e2fce160 0x5567e2fd0550 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.125+0000 7f826d42d740 1 --2- 172.21.15.33:0/3375701937 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5567e2fbdf30 0x5567e2fbb9f0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.125+0000 7f826d42d740 1 --2- 172.21.15.33:0/3375701937 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5567e2fb5350 0x5567e2fbb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.125+0000 7f82437fe640 0 client.17532 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:02:27.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.125+0000 7f826d42d740 1 --2- 172.21.15.33:0/3375701937 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5567e2fb4a30 0x5567e2fbaf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.125+0000 7f826d42d740 1 -- 172.21.15.33:0/3375701937 >> 172.21.15.33:0/3375701937 conn(0x5567e2faf160 msgr2=0x5567e2fb0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:27.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.125+0000 7f826d42d740 1 -- 172.21.15.33:0/3375701937 shutdown_connections 2024-03-19T03:02:27.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.125+0000 7f826d42d740 1 -- 172.21.15.33:0/3375701937 wait complete. 2024-03-19T03:02:27.133 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.125+0000 7f826d42d740 20 client.17532 trim_cache size 0 max 16384 2024-03-19T03:02:27.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.143+0000 7f826d42d740 1 Processor -- start 2024-03-19T03:02:27.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.143+0000 7f826d42d740 1 -- start start 2024-03-19T03:02:27.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.143+0000 7f826d42d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5567e2fb4a30 0x5567e2fb9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.143+0000 7f826d42d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5567e2fb5350 0x5567e2fc9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.143+0000 7f826d42d740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5567e2fbdf30 0x5567e2fbb810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.143+0000 7f826d42d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f82641b3700 con 0x5567e2fb5350 2024-03-19T03:02:27.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.143+0000 7f826d42d740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f82641b0fd0 con 0x5567e2fbdf30 2024-03-19T03:02:27.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.143+0000 7f826d42d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f82641aee40 con 0x5567e2fb4a30 2024-03-19T03:02:27.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.143+0000 7f82637fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5567e2fb5350 0x5567e2fc9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.143+0000 7f82637fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5567e2fb5350 0x5567e2fc9ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38480/0 (socket says 172.21.15.33:38480) 2024-03-19T03:02:27.150 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.143+0000 7f82637fe640 1 -- 172.21.15.33:0/2593410410 learned_addr learned my addr 172.21.15.33:0/2593410410 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:27.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.143+0000 7f8268cff640 1 --2- 172.21.15.33:0/2593410410 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5567e2fbdf30 0x5567e2fbb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.143+0000 7f8263fff640 1 --2- 172.21.15.33:0/2593410410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5567e2fb4a30 0x5567e2fb9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.143+0000 7f82637fe640 1 -- 172.21.15.33:0/2593410410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5567e2fb4a30 msgr2=0x5567e2fb9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:27.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.143+0000 7f82637fe640 1 --2- 172.21.15.33:0/2593410410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5567e2fb4a30 0x5567e2fb9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.143+0000 7f82637fe640 1 -- 172.21.15.33:0/2593410410 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5567e2fbdf30 msgr2=0x5567e2fbb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:27.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.143+0000 7f82637fe640 1 --2- 172.21.15.33:0/2593410410 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5567e2fbdf30 0x5567e2fbb810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.143+0000 7f82637fe640 1 -- 172.21.15.33:0/2593410410 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567e2bec0f0 con 0x5567e2fb5350 2024-03-19T03:02:27.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.144+0000 7f82637fe640 1 --2- 172.21.15.33:0/2593410410 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5567e2fb5350 0x5567e2fc9ac0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f8258009f90 tx=0x7f8258005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd63722e422fee76 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:27.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.144+0000 7f8249ffb640 1 -- 172.21.15.33:0/2593410410 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8258012190 con 0x5567e2fb5350 2024-03-19T03:02:27.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.144+0000 7f8249ffb640 1 -- 172.21.15.33:0/2593410410 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f82580054d0 con 0x5567e2fb5350 2024-03-19T03:02:27.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.144+0000 7f8249ffb640 1 -- 172.21.15.33:0/2593410410 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825800f480 con 0x5567e2fb5350 2024-03-19T03:02:27.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.144+0000 7f826d42d740 1 -- 172.21.15.33:0/2593410410 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5567e2fb5350 msgr2=0x5567e2fc9ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.144+0000 7f826d42d740 1 --2- 172.21.15.33:0/2593410410 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5567e2fb5350 0x5567e2fc9ac0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f8258009f90 tx=0x7f8258005f90 comp rx=0 tx=0).stop 2024-03-19T03:02:27.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.144+0000 7f826d42d740 1 -- 172.21.15.33:0/2593410410 shutdown_connections 2024-03-19T03:02:27.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.144+0000 7f826d42d740 1 --2- 172.21.15.33:0/2593410410 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5567e2fbdf30 0x5567e2fbb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.144+0000 7f826d42d740 1 --2- 172.21.15.33:0/2593410410 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5567e2fb5350 0x5567e2fc9ac0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.144+0000 7f826d42d740 1 --2- 172.21.15.33:0/2593410410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5567e2fb4a30 0x5567e2fb9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.144+0000 7f826d42d740 1 -- 172.21.15.33:0/2593410410 >> 172.21.15.33:0/2593410410 conn(0x5567e2faf160 msgr2=0x5567e2fae1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:27.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.144+0000 7f826d42d740 1 -- 172.21.15.33:0/2593410410 shutdown_connections 2024-03-19T03:02:27.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.144+0000 7f826d42d740 1 -- 172.21.15.33:0/2593410410 wait complete. 2024-03-19T03:02:27.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.145+0000 7f826d42d740 1 -- start start 2024-03-19T03:02:27.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.145+0000 7f826d42d740 1 Processor -- start 2024-03-19T03:02:27.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.145+0000 7f826d42d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5567e2fb4a30 0x5567e2fc62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.145+0000 7f826d42d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5567e2fb5350 0x5567e2fc1690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.145+0000 7f826d42d740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5567e2fbdf30 0x5567e2fc1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.145+0000 7f826d42d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f82641af490 con 0x5567e2fb5350 2024-03-19T03:02:27.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.145+0000 7f826d42d740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f82641b0900 con 0x5567e2fbdf30 2024-03-19T03:02:27.152 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.145+0000 7f826d42d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f82641afb60 con 0x5567e2fb4a30 2024-03-19T03:02:27.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.145+0000 7f826d42d740 10 client.0 _ll_register_callbacks cb 0x7f8254097a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:02:27.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.145+0000 7f82637fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5567e2fb5350 0x5567e2fc1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.145+0000 7f82637fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5567e2fb5350 0x5567e2fc1690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38496/0 (socket says 172.21.15.33:38496) 2024-03-19T03:02:27.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.145+0000 7f82637fe640 1 -- 172.21.15.33:0/210288882 learned_addr learned my addr 172.21.15.33:0/210288882 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:27.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.145+0000 7f8263fff640 1 --2- 172.21.15.33:0/210288882 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5567e2fb4a30 0x5567e2fc62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.145+0000 7f8268cff640 1 --2- 172.21.15.33:0/210288882 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5567e2fbdf30 0x5567e2fc1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.146+0000 7f82637fe640 1 -- 172.21.15.33:0/210288882 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5567e2fb4a30 msgr2=0x5567e2fc62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:27.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.146+0000 7f82637fe640 1 --2- 172.21.15.33:0/210288882 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5567e2fb4a30 0x5567e2fc62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.146+0000 7f82637fe640 1 -- 172.21.15.33:0/210288882 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5567e2fbdf30 msgr2=0x5567e2fc1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:27.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.146+0000 7f82637fe640 1 --2- 172.21.15.33:0/210288882 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5567e2fbdf30 0x5567e2fc1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.146+0000 7f82637fe640 1 -- 172.21.15.33:0/210288882 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567e2fc22c0 con 0x5567e2fb5350 2024-03-19T03:02:27.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.146+0000 7f82637fe640 1 --2- 172.21.15.33:0/210288882 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5567e2fb5350 0x5567e2fc1690 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f8258009e50 tx=0x7f8258005680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcbc6e413ae6d658 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:27.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.146+0000 7f82437fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:02:27.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.146+0000 7f82437fe640 1 -- 172.21.15.33:0/210288882 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8258004780 con 0x5567e2fb5350 2024-03-19T03:02:27.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.146+0000 7f82437fe640 1 -- 172.21.15.33:0/210288882 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f8258005020 con 0x5567e2fb5350 2024-03-19T03:02:27.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.146+0000 7f826d42d740 1 -- 172.21.15.33:0/210288882 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82580054d0 con 0x5567e2fb5350 2024-03-19T03:02:27.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.146+0000 7f82437fe640 1 -- 172.21.15.33:0/210288882 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8258005220 con 0x5567e2fb5350 2024-03-19T03:02:27.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.146+0000 7f826d42d740 10 client.17538 fetch_fsmap learned FSMap version 25 2024-03-19T03:02:27.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.146+0000 7f826d42d740 1 -- 172.21.15.33:0/210288882 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5567e303c8a0 con 0x5567e2fb5350 2024-03-19T03:02:27.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.146+0000 7f82437fe640 1 -- 172.21.15.33:0/210288882 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f82580054d0 con 0x5567e2fb5350 2024-03-19T03:02:27.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.146+0000 7f82437fe640 1 -- 172.21.15.33:0/210288882 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f8258010430 con 0x5567e2fb5350 2024-03-19T03:02:27.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.146+0000 7f826d42d740 10 client.17538 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:02:27.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.146+0000 7f826d42d740 10 client.17538 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:02:27.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.146+0000 7f826d42d740 20 client.17538 populate_metadata read hostname 'smithi033' 2024-03-19T03:02:27.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.146+0000 7f826d42d740 1 --2- 172.21.15.33:0/210288882 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5567e303cff0 0x5567e303f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.146+0000 7f826d42d740 4 client.17538 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T03:02:27.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.146+0000 7f826d42d740 1 -- 172.21.15.33:0/210288882 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5567e2f1fd90 con 0x5567e303cff0 2024-03-19T03:02:27.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.147+0000 7f8263fff640 1 --2- 172.21.15.33:0/210288882 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5567e303cff0 0x5567e303f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.147+0000 7f8263fff640 1 --2- 172.21.15.33:0/210288882 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5567e303cff0 0x5567e303f410 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=766287c423f64c55 server_cookie=b74a3bd0f814069d in_seq=0 out_seq=0 2024-03-19T03:02:27.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.147+0000 7f82437fe640 10 client.17538 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:02:27.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.147+0000 7f82437fe640 1 -- 172.21.15.33:0/210288882 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5567e2f1fd90 con 0x5567e303cff0 2024-03-19T03:02:27.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.147+0000 7f82437fe640 10 client.17538 handle_command_reply: tid=0 2024-03-19T03:02:27.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.147+0000 7f826d42d740 1 client.17538 shutdown 2024-03-19T03:02:27.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.147+0000 7f826d42d740 1 -- 172.21.15.33:0/210288882 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5567e2fb5350 msgr2=0x5567e2fc1690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.147+0000 7f826d42d740 1 --2- 172.21.15.33:0/210288882 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5567e2fb5350 0x5567e2fc1690 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f8258009e50 tx=0x7f8258005680 comp rx=0 tx=0).stop 2024-03-19T03:02:27.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.147+0000 7f826d42d740 1 -- 172.21.15.33:0/210288882 shutdown_connections 2024-03-19T03:02:27.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.147+0000 7f826d42d740 1 --2- 172.21.15.33:0/210288882 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5567e303cff0 0x5567e303f410 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.147+0000 7f826d42d740 1 --2- 172.21.15.33:0/210288882 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5567e2fbdf30 0x5567e2fc1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.147+0000 7f826d42d740 1 --2- 172.21.15.33:0/210288882 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5567e2fb5350 0x5567e2fc1690 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.147+0000 7f826d42d740 1 --2- 172.21.15.33:0/210288882 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5567e2fb4a30 0x5567e2fc62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.147+0000 7f82437fe640 0 client.17538 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:02:27.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.147+0000 7f826d42d740 1 -- 172.21.15.33:0/210288882 >> 172.21.15.33:0/210288882 conn(0x5567e2faf160 msgr2=0x5567e2fb0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:27.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.147+0000 7f826d42d740 1 -- 172.21.15.33:0/210288882 shutdown_connections 2024-03-19T03:02:27.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.147+0000 7f826d42d740 1 -- 172.21.15.33:0/210288882 wait complete. 2024-03-19T03:02:27.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.148+0000 7f826d42d740 20 client.17538 trim_cache size 0 max 16384 2024-03-19T03:02:27.156 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T03:02:27.156 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-03-19T03:02:27.156 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-03-19T03:02:27.156 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T03:02:27.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.148+0000 7f826a762640 1 -- 172.21.15.33:0/3444278979 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f824c0a29a0 msgr2=0x7f824c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.148+0000 7f826a762640 1 --2- 172.21.15.33:0/3444278979 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f824c0a29a0 0x7f824c0a4e60 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f825c04c000 tx=0x7f825c045000 comp rx=0 tx=0).stop 2024-03-19T03:02:27.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.148+0000 7f826a762640 1 -- 172.21.15.33:0/3444278979 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8264102a60 msgr2=0x7f82641006b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.148+0000 7f826a762640 1 --2- 172.21.15.33:0/3444278979 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8264102a60 0x7f82641006b0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f825400ea60 tx=0x7f825400ef20 comp rx=0 tx=0).stop 2024-03-19T03:02:27.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.148+0000 7f826a762640 1 -- 172.21.15.33:0/3444278979 shutdown_connections 2024-03-19T03:02:27.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.148+0000 7f826a762640 1 --2- 172.21.15.33:0/3444278979 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f824c0a29a0 0x7f824c0a4e60 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.157 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.148+0000 7f826a762640 1 --2- 172.21.15.33:0/3444278979 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8264103380 0x7f8264100bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.148+0000 7f826a762640 1 --2- 172.21.15.33:0/3444278979 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f826410be30 0x7f8264101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.148+0000 7f826a762640 1 --2- 172.21.15.33:0/3444278979 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8264102a60 0x7f82641006b0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.148+0000 7f826a762640 1 -- 172.21.15.33:0/3444278979 >> 172.21.15.33:0/3444278979 conn(0x7f8264076aa0 msgr2=0x7f82640fd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:27.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.148+0000 7f826a762640 1 -- 172.21.15.33:0/3444278979 shutdown_connections 2024-03-19T03:02:27.158 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.148+0000 7f826a762640 1 -- 172.21.15.33:0/3444278979 wait complete. 2024-03-19T03:02:27.160 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T03:02:27.161 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:02:27.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.263+0000 7f22b39a9640 1 -- 172.21.15.33:0/1778256381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22ac100720 msgr2=0x7f22ac100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.263+0000 7f22b39a9640 1 --2- 172.21.15.33:0/1778256381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22ac100720 0x7f22ac100ba0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f229c009930 tx=0x7f229c02bc30 comp rx=0 tx=0).stop 2024-03-19T03:02:27.267 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.263+0000 7f22b39a9640 1 -- 172.21.15.33:0/1778256381 shutdown_connections 2024-03-19T03:02:27.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.263+0000 7f22b39a9640 1 --2- 172.21.15.33:0/1778256381 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f22ac1010e0 0x7f22ac10d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.263+0000 7f22b39a9640 1 --2- 172.21.15.33:0/1778256381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22ac100720 0x7f22ac100ba0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.263+0000 7f22b39a9640 1 --2- 172.21.15.33:0/1778256381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22ac106730 0x7f22ac106b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.263+0000 7f22b39a9640 1 -- 172.21.15.33:0/1778256381 >> 172.21.15.33:0/1778256381 conn(0x7f22ac077820 msgr2=0x7f22ac077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:27.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.263+0000 7f22b39a9640 1 -- 172.21.15.33:0/1778256381 shutdown_connections 2024-03-19T03:02:27.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.264+0000 7f22b39a9640 1 -- 172.21.15.33:0/1778256381 wait complete. 2024-03-19T03:02:27.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.264+0000 7f22b39a9640 1 Processor -- start 2024-03-19T03:02:27.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.264+0000 7f22b39a9640 1 -- start start 2024-03-19T03:02:27.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.264+0000 7f22b39a9640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22ac100720 0x7f22ac198e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.264+0000 7f22b39a9640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f22ac1010e0 0x7f22ac1993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.264+0000 7f22b39a9640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22ac106730 0x7f22ac1a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.264+0000 7f22b39a9640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f22ac1109f0 con 0x7f22ac106730 2024-03-19T03:02:27.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.264+0000 7f22b39a9640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f22ac110870 con 0x7f22ac100720 2024-03-19T03:02:27.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.264+0000 7f22b39a9640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f22ac110b70 con 0x7f22ac1010e0 2024-03-19T03:02:27.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.264+0000 7f22b1f46640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22ac106730 0x7f22ac1a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.264+0000 7f22b1745640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22ac100720 0x7f22ac198e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.264+0000 7f22b1f46640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22ac106730 0x7f22ac1a3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38510/0 (socket says 172.21.15.33:38510) 2024-03-19T03:02:27.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.264+0000 7f22b1f46640 1 -- 172.21.15.33:0/1126190424 learned_addr learned my addr 172.21.15.33:0/1126190424 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:27.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.264+0000 7f22b0f44640 1 --2- 172.21.15.33:0/1126190424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f22ac1010e0 0x7f22ac1993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.265+0000 7f22b1f46640 1 -- 172.21.15.33:0/1126190424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f22ac1010e0 msgr2=0x7f22ac1993b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.265+0000 7f22b1f46640 1 --2- 172.21.15.33:0/1126190424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f22ac1010e0 0x7f22ac1993b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.265+0000 7f22b1f46640 1 -- 172.21.15.33:0/1126190424 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22ac100720 msgr2=0x7f22ac198e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.265+0000 7f22b1f46640 1 --2- 172.21.15.33:0/1126190424 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22ac100720 0x7f22ac198e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.265+0000 7f22b1f46640 1 -- 172.21.15.33:0/1126190424 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22ac1a4430 con 0x7f22ac106730 2024-03-19T03:02:27.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.265+0000 7f22b1745640 1 --2- 172.21.15.33:0/1126190424 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22ac100720 0x7f22ac198e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:02:27.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.265+0000 7f22b1f46640 1 --2- 172.21.15.33:0/1126190424 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22ac106730 0x7f22ac1a3ef0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f22a800e970 tx=0x7f22a800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:27.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.265+0000 7f22a27fc640 1 -- 172.21.15.33:0/1126190424 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a800ccc0 con 0x7f22ac106730 2024-03-19T03:02:27.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.265+0000 7f22a27fc640 1 -- 172.21.15.33:0/1126190424 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22a800ce60 con 0x7f22ac106730 2024-03-19T03:02:27.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.265+0000 7f22b39a9640 1 -- 172.21.15.33:0/1126190424 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22ac1a4600 con 0x7f22ac106730 2024-03-19T03:02:27.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.265+0000 7f22a27fc640 1 -- 172.21.15.33:0/1126190424 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a8010430 con 0x7f22ac106730 2024-03-19T03:02:27.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.265+0000 7f22b39a9640 1 -- 172.21.15.33:0/1126190424 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22ac1a4b20 con 0x7f22ac106730 2024-03-19T03:02:27.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.266+0000 7f22b39a9640 1 -- 172.21.15.33:0/1126190424 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22ac068610 con 0x7f22ac106730 2024-03-19T03:02:27.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.267+0000 7f22a27fc640 1 -- 172.21.15.33:0/1126190424 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f22a8004f90 con 0x7f22ac106730 2024-03-19T03:02:27.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.268+0000 7f22a27fc640 1 --2- 172.21.15.33:0/1126190424 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f22980a2970 0x7f22980a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.268+0000 7f22b1745640 1 --2- 172.21.15.33:0/1126190424 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f22980a2970 0x7f22980a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.268+0000 7f22a27fc640 1 -- 172.21.15.33:0/1126190424 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f22a80cf5e0 con 0x7f22ac106730 2024-03-19T03:02:27.271 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.268+0000 7f22b1745640 1 --2- 172.21.15.33:0/1126190424 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f22980a2970 0x7f22980a4e30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f22a40099a0 tx=0x7f22a4009500 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:27.274 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.273+0000 7f22a27fc640 1 -- 172.21.15.33:0/1126190424 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f22a809bf70 con 0x7f22ac106730 2024-03-19T03:02:27.500 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.499+0000 7f22b39a9640 1 -- 172.21.15.33:0/1126190424 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f22ac108cb0 con 0x7f22ac106730 2024-03-19T03:02:27.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.500+0000 7f22a27fc640 1 -- 172.21.15.33:0/1126190424 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11867 (secure 0 0 0) 0x7f22a809b870 con 0x7f22ac106730 2024-03-19T03:02:27.501 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T03:02:27.501 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:50:52.201798+0000","modified":"2024-03-19T02:53:11.748697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24488,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/3731232169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":3731232169},{"type":"v1","addr":"172.21.15.96:6839","nonce":3731232169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24457,24488,24509]},"id":1}]} 2024-03-19T03:02:27.502 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-03-19T03:02:27.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.502+0000 7f22b39a9640 1 -- 172.21.15.33:0/1126190424 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f22980a2970 msgr2=0x7f22980a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.502+0000 7f22b39a9640 1 --2- 172.21.15.33:0/1126190424 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f22980a2970 0x7f22980a4e30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f22a40099a0 tx=0x7f22a4009500 comp rx=0 tx=0).stop 2024-03-19T03:02:27.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.502+0000 7f22b39a9640 1 -- 172.21.15.33:0/1126190424 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22ac106730 msgr2=0x7f22ac1a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.502+0000 7f22b39a9640 1 --2- 172.21.15.33:0/1126190424 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22ac106730 0x7f22ac1a3ef0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f22a800e970 tx=0x7f22a800ed70 comp rx=0 tx=0).stop 2024-03-19T03:02:27.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.502+0000 7f22b39a9640 1 -- 172.21.15.33:0/1126190424 shutdown_connections 2024-03-19T03:02:27.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.502+0000 7f22b39a9640 1 --2- 172.21.15.33:0/1126190424 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f22980a2970 0x7f22980a4e30 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.502+0000 7f22b39a9640 1 --2- 172.21.15.33:0/1126190424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f22ac1010e0 0x7f22ac1993b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.502+0000 7f22b39a9640 1 --2- 172.21.15.33:0/1126190424 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f22ac106730 0x7f22ac1a3ef0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.502+0000 7f22b39a9640 1 --2- 172.21.15.33:0/1126190424 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f22ac100720 0x7f22ac198e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.502+0000 7f22b39a9640 1 -- 172.21.15.33:0/1126190424 >> 172.21.15.33:0/1126190424 conn(0x7f22ac077820 msgr2=0x7f22ac103020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:27.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.502+0000 7f22b39a9640 1 -- 172.21.15.33:0/1126190424 shutdown_connections 2024-03-19T03:02:27.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.502+0000 7f22b39a9640 1 -- 172.21.15.33:0/1126190424 wait complete. 2024-03-19T03:02:27.514 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T03:02:27.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.617+0000 7f108633b640 1 -- 172.21.15.33:0/1087752353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10801006a0 msgr2=0x7f1080100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.617+0000 7f108633b640 1 --2- 172.21.15.33:0/1087752353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10801006a0 0x7f1080100b20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1074009900 tx=0x7f107402bc30 comp rx=0 tx=0).stop 2024-03-19T03:02:27.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.617+0000 7f108633b640 1 -- 172.21.15.33:0/1087752353 shutdown_connections 2024-03-19T03:02:27.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.617+0000 7f108633b640 1 --2- 172.21.15.33:0/1087752353 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1080101060 0x7f108010d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.617+0000 7f108633b640 1 --2- 172.21.15.33:0/1087752353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10801006a0 0x7f1080100b20 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.621 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.617+0000 7f108633b640 1 --2- 172.21.15.33:0/1087752353 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10801066b0 0x7f1080106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.617+0000 7f108633b640 1 -- 172.21.15.33:0/1087752353 >> 172.21.15.33:0/1087752353 conn(0x7f10800778d0 msgr2=0x7f1080077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:27.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.617+0000 7f108633b640 1 -- 172.21.15.33:0/1087752353 shutdown_connections 2024-03-19T03:02:27.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.617+0000 7f108633b640 1 -- 172.21.15.33:0/1087752353 wait complete. 2024-03-19T03:02:27.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.618+0000 7f108633b640 1 Processor -- start 2024-03-19T03:02:27.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.618+0000 7f108633b640 1 -- start start 2024-03-19T03:02:27.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.618+0000 7f108633b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10801006a0 0x7f1080196e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.622 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.618+0000 7f108633b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1080101060 0x7f10801973d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.626 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.618+0000 7f108633b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10801066b0 0x7f108019b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.618+0000 7f108633b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1080110970 con 0x7f1080101060 2024-03-19T03:02:27.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.618+0000 7f108633b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f10801107f0 con 0x7f10801066b0 2024-03-19T03:02:27.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.618+0000 7f108633b640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1080110af0 con 0x7f10801006a0 2024-03-19T03:02:27.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.619+0000 7f107f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1080101060 0x7f10801973d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.619+0000 7f107f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1080101060 0x7f10801973d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38518/0 (socket says 172.21.15.33:38518) 2024-03-19T03:02:27.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.619+0000 7f107f7fe640 1 -- 172.21.15.33:0/3486093071 learned_addr learned my addr 172.21.15.33:0/3486093071 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:27.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.619+0000 7f10848d8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10801066b0 0x7f108019b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.627 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.619+0000 7f107ffff640 1 --2- 172.21.15.33:0/3486093071 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10801006a0 0x7f1080196e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.619+0000 7f107f7fe640 1 -- 172.21.15.33:0/3486093071 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10801006a0 msgr2=0x7f1080196e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.619+0000 7f107f7fe640 1 --2- 172.21.15.33:0/3486093071 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10801006a0 0x7f1080196e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.619+0000 7f107f7fe640 1 -- 172.21.15.33:0/3486093071 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10801066b0 msgr2=0x7f108019b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.619+0000 7f107f7fe640 1 --2- 172.21.15.33:0/3486093071 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10801066b0 0x7f108019b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.619+0000 7f107f7fe640 1 -- 172.21.15.33:0/3486093071 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f108019be80 con 0x7f1080101060 2024-03-19T03:02:27.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.619+0000 7f10848d8640 1 --2- 172.21.15.33:0/3486093071 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10801066b0 0x7f108019b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:02:27.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.619+0000 7f107ffff640 1 --2- 172.21.15.33:0/3486093071 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10801006a0 0x7f1080196e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:02:27.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.619+0000 7f107f7fe640 1 --2- 172.21.15.33:0/3486093071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1080101060 0x7f10801973d0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f1074006010 tx=0x7f1074005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:27.628 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.619+0000 7f107d7fa640 1 -- 172.21.15.33:0/3486093071 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1074043020 con 0x7f1080101060 2024-03-19T03:02:27.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.619+0000 7f108633b640 1 -- 172.21.15.33:0/3486093071 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f108019c110 con 0x7f1080101060 2024-03-19T03:02:27.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.619+0000 7f107d7fa640 1 -- 172.21.15.33:0/3486093071 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1074004350 con 0x7f1080101060 2024-03-19T03:02:27.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.619+0000 7f107d7fa640 1 -- 172.21.15.33:0/3486093071 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1074003a10 con 0x7f1080101060 2024-03-19T03:02:27.629 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.619+0000 7f108633b640 1 -- 172.21.15.33:0/3486093071 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10801a3a00 con 0x7f1080101060 2024-03-19T03:02:27.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.622+0000 7f107d7fa640 1 -- 172.21.15.33:0/3486093071 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f1074046020 con 0x7f1080101060 2024-03-19T03:02:27.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.622+0000 7f1089001740 1 Processor -- start 2024-03-19T03:02:27.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.622+0000 7f1089001740 1 -- start start 2024-03-19T03:02:27.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.622+0000 7f1089001740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561082eefbe0 0x561082eeffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.622+0000 7f1089001740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561082ef0500 0x561082efd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.622+0000 7f1089001740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561082efddd0 0x561082f001c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.622+0000 7f1089001740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561082e534b0 con 0x561082efddd0 2024-03-19T03:02:27.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.622+0000 7f1089001740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x561082dfbd70 con 0x561082ef0500 2024-03-19T03:02:27.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.622+0000 7f1089001740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x561082e4f550 con 0x561082eefbe0 2024-03-19T03:02:27.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.623+0000 7f107f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561082efddd0 0x561082f001c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.623+0000 7f107ffff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561082ef0500 0x561082efd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.623+0000 7f107ffff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561082ef0500 0x561082efd890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:40748/0 (socket says 172.21.15.33:40748) 2024-03-19T03:02:27.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.623+0000 7f107f7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561082efddd0 0x561082f001c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38526/0 (socket says 172.21.15.33:38526) 2024-03-19T03:02:27.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.623+0000 7f107ffff640 1 -- 172.21.15.33:0/2434965973 learned_addr learned my addr 172.21.15.33:0/2434965973 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:27.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.623+0000 7f10848d8640 1 --2- 172.21.15.33:0/2434965973 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561082eefbe0 0x561082eeffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.623+0000 7f107d7fa640 1 --2- 172.21.15.33:0/3486093071 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f10680a2970 0x7f10680a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.623+0000 7f107ffff640 1 -- 172.21.15.33:0/2434965973 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561082eefbe0 msgr2=0x561082eeffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:27.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.623+0000 7f107ffff640 1 --2- 172.21.15.33:0/2434965973 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561082eefbe0 0x561082eeffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.623+0000 7f10848d8640 1 --2- 172.21.15.33:0/3486093071 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f10680a2970 0x7f10680a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.623+0000 7f107ffff640 1 -- 172.21.15.33:0/2434965973 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561082efddd0 msgr2=0x561082f001c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:27.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.623+0000 7f107ffff640 1 --2- 172.21.15.33:0/2434965973 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561082efddd0 0x561082f001c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.623+0000 7f107ffff640 1 -- 172.21.15.33:0/2434965973 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561082e1e210 con 0x561082ef0500 2024-03-19T03:02:27.637 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.623+0000 7f107d7fa640 1 -- 172.21.15.33:0/3486093071 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f10740f0bb0 con 0x7f1080101060 2024-03-19T03:02:27.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.623+0000 7f107ffff640 1 --2- 172.21.15.33:0/2434965973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561082ef0500 0x561082efd890 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f10700114c0 tx=0x7f1070011980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4987548b816324d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:27.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.623+0000 7f10667fc640 1 -- 172.21.15.33:0/2434965973 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1070004020 con 0x561082ef0500 2024-03-19T03:02:27.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.623+0000 7f10667fc640 1 -- 172.21.15.33:0/2434965973 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f10700041c0 con 0x561082ef0500 2024-03-19T03:02:27.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.623+0000 7f10848d8640 1 --2- 172.21.15.33:0/3486093071 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f10680a2970 0x7f10680a4e30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f10780049d0 tx=0x7f1078004ab0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:27.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.623+0000 7f10667fc640 1 -- 172.21.15.33:0/2434965973 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107000f8a0 con 0x561082ef0500 2024-03-19T03:02:27.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.624+0000 7f1089001740 1 -- 172.21.15.33:0/2434965973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561082ef0500 msgr2=0x561082efd890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.624+0000 7f1089001740 1 --2- 172.21.15.33:0/2434965973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561082ef0500 0x561082efd890 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f10700114c0 tx=0x7f1070011980 comp rx=0 tx=0).stop 2024-03-19T03:02:27.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.624+0000 7f1089001740 1 -- 172.21.15.33:0/2434965973 shutdown_connections 2024-03-19T03:02:27.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.624+0000 7f1089001740 1 --2- 172.21.15.33:0/2434965973 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561082efddd0 0x561082f001c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.624+0000 7f1089001740 1 --2- 172.21.15.33:0/2434965973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561082ef0500 0x561082efd890 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.624+0000 7f1089001740 1 --2- 172.21.15.33:0/2434965973 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561082eefbe0 0x561082eeffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.624+0000 7f1089001740 1 -- 172.21.15.33:0/2434965973 >> 172.21.15.33:0/2434965973 conn(0x561082eeaa00 msgr2=0x561082eeade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:27.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.624+0000 7f1089001740 1 -- 172.21.15.33:0/2434965973 shutdown_connections 2024-03-19T03:02:27.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.624+0000 7f1089001740 1 -- 172.21.15.33:0/2434965973 wait complete. 2024-03-19T03:02:27.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.624+0000 7f1089001740 1 -- start start 2024-03-19T03:02:27.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.625+0000 7f1089001740 1 Processor -- start 2024-03-19T03:02:27.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.625+0000 7f1089001740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561082eefbe0 0x561082efada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.625+0000 7f1089001740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561082ef0500 0x561082efb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.625+0000 7f1089001740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561082efddd0 0x561082efb820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.625+0000 7f1089001740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561082f00a00 con 0x561082efddd0 2024-03-19T03:02:27.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.625+0000 7f1089001740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x561082e0f310 con 0x561082eefbe0 2024-03-19T03:02:27.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.625+0000 7f1089001740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x561082f00b80 con 0x561082ef0500 2024-03-19T03:02:27.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.625+0000 7f1089001740 10 client.0 _ll_register_callbacks cb 0x561082e032d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:02:27.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.625+0000 7f10848d8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561082efddd0 0x561082efb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.625+0000 7f107ffff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561082eefbe0 0x561082efada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.639 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.625+0000 7f10848d8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561082efddd0 0x561082efb820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38530/0 (socket says 172.21.15.33:38530) 2024-03-19T03:02:27.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.625+0000 7f10848d8640 1 -- 172.21.15.33:0/2782691297 learned_addr learned my addr 172.21.15.33:0/2782691297 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:27.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.625+0000 7f107f7fe640 1 --2- 172.21.15.33:0/2782691297 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561082ef0500 0x561082efb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.625+0000 7f107ffff640 1 -- 172.21.15.33:0/2782691297 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561082ef0500 msgr2=0x561082efb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:27.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.625+0000 7f107ffff640 1 --2- 172.21.15.33:0/2782691297 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561082ef0500 0x561082efb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.625+0000 7f107ffff640 1 -- 172.21.15.33:0/2782691297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561082efddd0 msgr2=0x561082efb820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:27.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.625+0000 7f107ffff640 1 --2- 172.21.15.33:0/2782691297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561082efddd0 0x561082efb820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.625+0000 7f107ffff640 1 -- 172.21.15.33:0/2782691297 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561082b673f0 con 0x561082eefbe0 2024-03-19T03:02:27.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.625+0000 7f107f7fe640 1 --2- 172.21.15.33:0/2782691297 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561082ef0500 0x561082efb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:02:27.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.625+0000 7f107ffff640 1 --2- 172.21.15.33:0/2782691297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561082eefbe0 0x561082efada0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1070005f60 tx=0x7f1070003a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3525180ed8095b26 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:27.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.625+0000 7f105f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T03:02:27.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.625+0000 7f105f7fe640 1 -- 172.21.15.33:0/2782691297 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1070025020 con 0x561082eefbe0 2024-03-19T03:02:27.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.626+0000 7f1089001740 1 -- 172.21.15.33:0/2782691297 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10700041c0 con 0x561082eefbe0 2024-03-19T03:02:27.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.626+0000 7f105f7fe640 1 -- 172.21.15.33:0/2782691297 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f1070005040 con 0x561082eefbe0 2024-03-19T03:02:27.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.626+0000 7f105f7fe640 1 -- 172.21.15.33:0/2782691297 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1070030050 con 0x561082eefbe0 2024-03-19T03:02:27.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.626+0000 7f1089001740 10 client.27377 fetch_fsmap learned FSMap version 25 2024-03-19T03:02:27.640 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.626+0000 7f1089001740 1 -- 172.21.15.33:0/2782691297 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x561082c317d0 con 0x561082eefbe0 2024-03-19T03:02:27.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.626+0000 7f105f7fe640 1 -- 172.21.15.33:0/2782691297 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f10700041c0 con 0x561082eefbe0 2024-03-19T03:02:27.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.626+0000 7f105f7fe640 1 -- 172.21.15.33:0/2782691297 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f1070004830 con 0x561082eefbe0 2024-03-19T03:02:27.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.626+0000 7f1089001740 10 client.27377 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:02:27.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.626+0000 7f1089001740 10 client.27377 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T03:02:27.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.626+0000 7f1089001740 20 client.27377 populate_metadata read hostname 'smithi033' 2024-03-19T03:02:27.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.626+0000 7f1089001740 1 --2- 172.21.15.33:0/2782691297 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x561082f0de60 0x561082f10250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.626+0000 7f1089001740 4 client.27377 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:02:27.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.626+0000 7f1089001740 1 -- 172.21.15.33:0/2782691297 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561082eb13e0 con 0x561082f0de60 2024-03-19T03:02:27.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.626+0000 7f107f7fe640 1 --2- 172.21.15.33:0/2782691297 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x561082f0de60 0x561082f10250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.627+0000 7f105f7fe640 10 client.27377 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T03:02:27.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.627+0000 7f107f7fe640 1 --2- 172.21.15.33:0/2782691297 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x561082f0de60 0x561082f10250 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b22b0ac686f774d3 server_cookie=73352ff704ed00d1 in_seq=0 out_seq=0 2024-03-19T03:02:27.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.627+0000 7f105f7fe640 1 -- 172.21.15.33:0/2782691297 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561082eb13e0 con 0x561082f0de60 2024-03-19T03:02:27.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.627+0000 7f105f7fe640 10 client.27377 handle_command_reply: tid=0 2024-03-19T03:02:27.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.627+0000 7f1089001740 1 client.27377 shutdown 2024-03-19T03:02:27.641 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.627+0000 7f1089001740 1 -- 172.21.15.33:0/2782691297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561082eefbe0 msgr2=0x561082efada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.627+0000 7f1089001740 1 --2- 172.21.15.33:0/2782691297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561082eefbe0 0x561082efada0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1070005f60 tx=0x7f1070003a80 comp rx=0 tx=0).stop 2024-03-19T03:02:27.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.627+0000 7f1089001740 1 -- 172.21.15.33:0/2782691297 shutdown_connections 2024-03-19T03:02:27.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.627+0000 7f1089001740 1 --2- 172.21.15.33:0/2782691297 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x561082f0de60 0x561082f10250 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.627+0000 7f1089001740 1 --2- 172.21.15.33:0/2782691297 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561082ef0500 0x561082efb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.627+0000 7f105f7fe640 0 client.27377 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T03:02:27.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.627+0000 7f1089001740 1 --2- 172.21.15.33:0/2782691297 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561082efddd0 0x561082efb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.627+0000 7f1089001740 1 --2- 172.21.15.33:0/2782691297 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561082eefbe0 0x561082efada0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.627+0000 7f1089001740 1 -- 172.21.15.33:0/2782691297 >> 172.21.15.33:0/2782691297 conn(0x561082eeaa00 msgr2=0x561082eec750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:27.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.628+0000 7f1089001740 1 -- 172.21.15.33:0/2782691297 shutdown_connections 2024-03-19T03:02:27.642 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.628+0000 7f1089001740 1 -- 172.21.15.33:0/2782691297 wait complete. 2024-03-19T03:02:27.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.628+0000 7f1089001740 20 client.27377 trim_cache size 0 max 16384 2024-03-19T03:02:27.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.641+0000 7f1089001740 1 Processor -- start 2024-03-19T03:02:27.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.641+0000 7f1089001740 1 -- start start 2024-03-19T03:02:27.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.641+0000 7f1089001740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561082eefbe0 0x561082efd4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.641+0000 7f1089001740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561082ef0500 0x561082ef8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.641+0000 7f1089001740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561082efddd0 0x561082f097e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.641+0000 7f1089001740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f10801af2e0 con 0x561082eefbe0 2024-03-19T03:02:27.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.641+0000 7f1089001740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f10801acbb0 con 0x561082efddd0 2024-03-19T03:02:27.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.641+0000 7f1089001740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f10801aaa20 con 0x561082ef0500 2024-03-19T03:02:27.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.641+0000 7f107ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561082eefbe0 0x561082efd4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.641+0000 7f107ffff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561082eefbe0 0x561082efd4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38540/0 (socket says 172.21.15.33:38540) 2024-03-19T03:02:27.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.641+0000 7f107ffff640 1 -- 172.21.15.33:0/3280902441 learned_addr learned my addr 172.21.15.33:0/3280902441 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:27.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.641+0000 7f10848d8640 1 --2- 172.21.15.33:0/3280902441 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561082efddd0 0x561082f097e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.646 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.641+0000 7f107f7fe640 1 --2- 172.21.15.33:0/3280902441 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561082ef0500 0x561082ef8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.642+0000 7f107ffff640 1 -- 172.21.15.33:0/3280902441 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561082ef0500 msgr2=0x561082ef8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:27.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.642+0000 7f107ffff640 1 --2- 172.21.15.33:0/3280902441 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561082ef0500 0x561082ef8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.642+0000 7f107ffff640 1 -- 172.21.15.33:0/3280902441 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561082efddd0 msgr2=0x561082f097e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:27.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.642+0000 7f107ffff640 1 --2- 172.21.15.33:0/3280902441 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561082efddd0 0x561082f097e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.642+0000 7f107ffff640 1 -- 172.21.15.33:0/3280902441 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561082f09d20 con 0x561082eefbe0 2024-03-19T03:02:27.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.642+0000 7f107f7fe640 1 --2- 172.21.15.33:0/3280902441 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561082ef0500 0x561082ef8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:02:27.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.642+0000 7f107ffff640 1 --2- 172.21.15.33:0/3280902441 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561082eefbe0 0x561082efd4d0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f10700058c0 tx=0x7f1070005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=130a748d5aaa6e96 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:27.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.642+0000 7f1065ffb640 1 -- 172.21.15.33:0/3280902441 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1070039070 con 0x561082eefbe0 2024-03-19T03:02:27.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.642+0000 7f1065ffb640 1 -- 172.21.15.33:0/3280902441 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1070003af0 con 0x561082eefbe0 2024-03-19T03:02:27.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.642+0000 7f1065ffb640 1 -- 172.21.15.33:0/3280902441 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1070025070 con 0x561082eefbe0 2024-03-19T03:02:27.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.642+0000 7f1089001740 1 -- 172.21.15.33:0/3280902441 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561082eefbe0 msgr2=0x561082efd4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.642+0000 7f1089001740 1 --2- 172.21.15.33:0/3280902441 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561082eefbe0 0x561082efd4d0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f10700058c0 tx=0x7f1070005d30 comp rx=0 tx=0).stop 2024-03-19T03:02:27.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.643+0000 7f1089001740 1 -- 172.21.15.33:0/3280902441 shutdown_connections 2024-03-19T03:02:27.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.643+0000 7f1089001740 1 --2- 172.21.15.33:0/3280902441 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561082ef0500 0x561082ef8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.643+0000 7f1089001740 1 --2- 172.21.15.33:0/3280902441 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561082efddd0 0x561082f097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.647 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.643+0000 7f1089001740 1 --2- 172.21.15.33:0/3280902441 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561082eefbe0 0x561082efd4d0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.643+0000 7f1089001740 1 -- 172.21.15.33:0/3280902441 >> 172.21.15.33:0/3280902441 conn(0x561082eeaa00 msgr2=0x561082ee9ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:27.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.643+0000 7f1089001740 1 -- 172.21.15.33:0/3280902441 shutdown_connections 2024-03-19T03:02:27.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.643+0000 7f1089001740 1 -- 172.21.15.33:0/3280902441 wait complete. 2024-03-19T03:02:27.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.643+0000 7f1089001740 1 -- start start 2024-03-19T03:02:27.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.643+0000 7f1089001740 1 Processor -- start 2024-03-19T03:02:27.648 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.643+0000 7f1089001740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561082eefbe0 0x561082ef9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.643+0000 7f1089001740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561082ef0500 0x561082efa420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.643+0000 7f1089001740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561082efddd0 0x561082efa960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.643+0000 7f1089001740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f10801ab070 con 0x561082efddd0 2024-03-19T03:02:27.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.643+0000 7f1089001740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f10801ac4e0 con 0x561082ef0500 2024-03-19T03:02:27.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.643+0000 7f1089001740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f10801ab740 con 0x561082eefbe0 2024-03-19T03:02:27.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.643+0000 7f1089001740 10 client.0 _ll_register_callbacks cb 0x7f10740b9020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:02:27.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.643+0000 7f10848d8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561082efddd0 0x561082efa960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.643+0000 7f10848d8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561082efddd0 0x561082efa960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38556/0 (socket says 172.21.15.33:38556) 2024-03-19T03:02:27.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.643+0000 7f10848d8640 1 -- 172.21.15.33:0/2388239773 learned_addr learned my addr 172.21.15.33:0/2388239773 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:27.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.643+0000 7f107ffff640 1 --2- 172.21.15.33:0/2388239773 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561082eefbe0 0x561082ef9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.644+0000 7f107f7fe640 1 --2- 172.21.15.33:0/2388239773 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561082ef0500 0x561082efa420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.644+0000 7f10848d8640 1 -- 172.21.15.33:0/2388239773 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561082eefbe0 msgr2=0x561082ef9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:27.653 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.644+0000 7f10848d8640 1 --2- 172.21.15.33:0/2388239773 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561082eefbe0 0x561082ef9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.644+0000 7f10848d8640 1 -- 172.21.15.33:0/2388239773 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561082ef0500 msgr2=0x561082efa420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:27.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.644+0000 7f10848d8640 1 --2- 172.21.15.33:0/2388239773 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561082ef0500 0x561082efa420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.644+0000 7f10848d8640 1 -- 172.21.15.33:0/2388239773 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561082f05080 con 0x561082efddd0 2024-03-19T03:02:27.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.644+0000 7f10848d8640 1 --2- 172.21.15.33:0/2388239773 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561082efddd0 0x561082efa960 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f1078023bc0 tx=0x7f1078004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78ccd5c4bc9b5e1a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:27.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.644+0000 7f105f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:02:27.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.644+0000 7f105f7fe640 1 -- 172.21.15.33:0/2388239773 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1078031060 con 0x561082efddd0 2024-03-19T03:02:27.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.644+0000 7f105f7fe640 1 -- 172.21.15.33:0/2388239773 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f107800fd40 con 0x561082efddd0 2024-03-19T03:02:27.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.644+0000 7f1089001740 1 -- 172.21.15.33:0/2388239773 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1070003af0 con 0x561082efddd0 2024-03-19T03:02:27.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.644+0000 7f105f7fe640 1 -- 172.21.15.33:0/2388239773 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1078038050 con 0x561082efddd0 2024-03-19T03:02:27.654 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.644+0000 7f1089001740 10 client.17562 fetch_fsmap learned FSMap version 25 2024-03-19T03:02:27.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.644+0000 7f1089001740 1 -- 172.21.15.33:0/2388239773 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x561082f06380 con 0x561082efddd0 2024-03-19T03:02:27.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.644+0000 7f105f7fe640 1 -- 172.21.15.33:0/2388239773 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1070003af0 con 0x561082efddd0 2024-03-19T03:02:27.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.644+0000 7f105f7fe640 1 -- 172.21.15.33:0/2388239773 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f107801f040 con 0x561082efddd0 2024-03-19T03:02:27.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.645+0000 7f1089001740 10 client.17562 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:02:27.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.645+0000 7f1089001740 10 client.17562 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T03:02:27.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.645+0000 7f1089001740 20 client.17562 populate_metadata read hostname 'smithi033' 2024-03-19T03:02:27.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.645+0000 7f1089001740 1 --2- 172.21.15.33:0/2388239773 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x561082f743b0 0x561082f767d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.645+0000 7f1089001740 4 client.17562 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T03:02:27.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.645+0000 7f1089001740 1 -- 172.21.15.33:0/2388239773 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561082e609f0 con 0x561082f743b0 2024-03-19T03:02:27.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.645+0000 7f107ffff640 1 --2- 172.21.15.33:0/2388239773 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x561082f743b0 0x561082f767d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.645+0000 7f105f7fe640 10 client.17562 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T03:02:27.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.645+0000 7f107ffff640 1 --2- 172.21.15.33:0/2388239773 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x561082f743b0 0x561082f767d0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=22bb651a6ee82e70 server_cookie=16b90215bd8c3552 in_seq=0 out_seq=0 2024-03-19T03:02:27.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.645+0000 7f105f7fe640 1 -- 172.21.15.33:0/2388239773 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561082e609f0 con 0x561082f743b0 2024-03-19T03:02:27.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.645+0000 7f105f7fe640 10 client.17562 handle_command_reply: tid=0 2024-03-19T03:02:27.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.645+0000 7f1089001740 1 client.17562 shutdown 2024-03-19T03:02:27.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.645+0000 7f1089001740 1 -- 172.21.15.33:0/2388239773 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561082efddd0 msgr2=0x561082efa960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.645+0000 7f1089001740 1 --2- 172.21.15.33:0/2388239773 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561082efddd0 0x561082efa960 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f1078023bc0 tx=0x7f1078004a40 comp rx=0 tx=0).stop 2024-03-19T03:02:27.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.645+0000 7f1089001740 1 -- 172.21.15.33:0/2388239773 shutdown_connections 2024-03-19T03:02:27.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.645+0000 7f1089001740 1 --2- 172.21.15.33:0/2388239773 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x561082f743b0 0x561082f767d0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.645+0000 7f1089001740 1 --2- 172.21.15.33:0/2388239773 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561082efddd0 0x561082efa960 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.645+0000 7f1089001740 1 --2- 172.21.15.33:0/2388239773 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561082ef0500 0x561082efa420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.645+0000 7f105f7fe640 0 client.17562 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T03:02:27.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.645+0000 7f1089001740 1 --2- 172.21.15.33:0/2388239773 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561082eefbe0 0x561082ef9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.645+0000 7f1089001740 1 -- 172.21.15.33:0/2388239773 >> 172.21.15.33:0/2388239773 conn(0x561082eeaa00 msgr2=0x561082eeba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:27.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.646+0000 7f1089001740 1 -- 172.21.15.33:0/2388239773 shutdown_connections 2024-03-19T03:02:27.656 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.646+0000 7f1089001740 1 -- 172.21.15.33:0/2388239773 wait complete. 2024-03-19T03:02:27.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.646+0000 7f1089001740 20 client.17562 trim_cache size 0 max 16384 2024-03-19T03:02:27.657 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T03:02:27.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.646+0000 7f108633b640 1 -- 172.21.15.33:0/3486093071 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f10680a2970 msgr2=0x7f10680a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.646+0000 7f108633b640 1 --2- 172.21.15.33:0/3486093071 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f10680a2970 0x7f10680a4e30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f10780049d0 tx=0x7f1078004ab0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.646+0000 7f108633b640 1 -- 172.21.15.33:0/3486093071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1080101060 msgr2=0x7f10801973d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.646+0000 7f108633b640 1 --2- 172.21.15.33:0/3486093071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1080101060 0x7f10801973d0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f1074006010 tx=0x7f1074005d90 comp rx=0 tx=0).stop 2024-03-19T03:02:27.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.646+0000 7f108633b640 1 -- 172.21.15.33:0/3486093071 shutdown_connections 2024-03-19T03:02:27.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.646+0000 7f108633b640 1 --2- 172.21.15.33:0/3486093071 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f10680a2970 0x7f10680a4e30 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.646+0000 7f108633b640 1 --2- 172.21.15.33:0/3486093071 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f10801066b0 0x7f108019b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.646+0000 7f108633b640 1 --2- 172.21.15.33:0/3486093071 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1080101060 0x7f10801973d0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.646+0000 7f108633b640 1 --2- 172.21.15.33:0/3486093071 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10801006a0 0x7f1080196e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.646+0000 7f108633b640 1 -- 172.21.15.33:0/3486093071 >> 172.21.15.33:0/3486093071 conn(0x7f10800778d0 msgr2=0x7f1080103090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:27.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.646+0000 7f108633b640 1 -- 172.21.15.33:0/3486093071 shutdown_connections 2024-03-19T03:02:27.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.646+0000 7f108633b640 1 -- 172.21.15.33:0/3486093071 wait complete. 2024-03-19T03:02:27.660 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T03:02:27.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.763+0000 7ffbe8ae8640 1 -- 172.21.15.33:0/3653939974 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffbe4100970 msgr2=0x7ffbe4100dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.763+0000 7ffbe8ae8640 1 --2- 172.21.15.33:0/3653939974 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffbe4100970 0x7ffbe4100dd0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc009860 tx=0x7ffbcc02bb70 comp rx=0 tx=0).stop 2024-03-19T03:02:27.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.763+0000 7ffbe8ae8640 1 -- 172.21.15.33:0/3653939974 shutdown_connections 2024-03-19T03:02:27.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.763+0000 7ffbe8ae8640 1 --2- 172.21.15.33:0/3653939974 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbe4101310 0x7ffbe410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.763+0000 7ffbe8ae8640 1 --2- 172.21.15.33:0/3653939974 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffbe4100970 0x7ffbe4100dd0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.763+0000 7ffbe8ae8640 1 --2- 172.21.15.33:0/3653939974 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffbe4106750 0x7ffbe4106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.763+0000 7ffbe8ae8640 1 -- 172.21.15.33:0/3653939974 >> 172.21.15.33:0/3653939974 conn(0x7ffbe40774c0 msgr2=0x7ffbe40778d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:27.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.763+0000 7ffbe8ae8640 1 -- 172.21.15.33:0/3653939974 shutdown_connections 2024-03-19T03:02:27.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.763+0000 7ffbe8ae8640 1 -- 172.21.15.33:0/3653939974 wait complete. 2024-03-19T03:02:27.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.764+0000 7ffbe8ae8640 1 Processor -- start 2024-03-19T03:02:27.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.764+0000 7ffbe8ae8640 1 -- start start 2024-03-19T03:02:27.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.764+0000 7ffbe8ae8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffbe4100970 0x7ffbe4197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.764+0000 7ffbe8ae8640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbe4101310 0x7ffbe41984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.764+0000 7ffbe8ae8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffbe4106750 0x7ffbe419c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.764+0000 7ffbe8ae8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe4111aa0 con 0x7ffbe4106750 2024-03-19T03:02:27.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.764+0000 7ffbe8ae8640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe4111920 con 0x7ffbe4100970 2024-03-19T03:02:27.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.764+0000 7ffbe8ae8640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe4111c20 con 0x7ffbe4101310 2024-03-19T03:02:27.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.764+0000 7ffbe2d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffbe4106750 0x7ffbe419c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.764+0000 7ffbe259c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffbe4100970 0x7ffbe4197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.764+0000 7ffbe2d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffbe4106750 0x7ffbe419c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38570/0 (socket says 172.21.15.33:38570) 2024-03-19T03:02:27.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.764+0000 7ffbe2d9d640 1 -- 172.21.15.33:0/2247006811 learned_addr learned my addr 172.21.15.33:0/2247006811 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:27.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.764+0000 7ffbdbfff640 1 --2- 172.21.15.33:0/2247006811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbe4101310 0x7ffbe41984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.764+0000 7ffbe2d9d640 1 -- 172.21.15.33:0/2247006811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbe4101310 msgr2=0x7ffbe41984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.764+0000 7ffbe2d9d640 1 --2- 172.21.15.33:0/2247006811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbe4101310 0x7ffbe41984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.764+0000 7ffbe2d9d640 1 -- 172.21.15.33:0/2247006811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffbe4100970 msgr2=0x7ffbe4197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.764+0000 7ffbe2d9d640 1 --2- 172.21.15.33:0/2247006811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffbe4100970 0x7ffbe4197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.764+0000 7ffbe2d9d640 1 -- 172.21.15.33:0/2247006811 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbe419cf90 con 0x7ffbe4106750 2024-03-19T03:02:27.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.764+0000 7ffbe259c640 1 --2- 172.21.15.33:0/2247006811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffbe4100970 0x7ffbe4197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:02:27.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.765+0000 7ffbe2d9d640 1 --2- 172.21.15.33:0/2247006811 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffbe4106750 0x7ffbe419c860 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc00eb10 tx=0x7ffbdc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:27.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.765+0000 7ffbdb7fe640 1 -- 172.21.15.33:0/2247006811 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc004b20 con 0x7ffbe4106750 2024-03-19T03:02:27.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.765+0000 7ffbdb7fe640 1 -- 172.21.15.33:0/2247006811 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffbdc004cc0 con 0x7ffbe4106750 2024-03-19T03:02:27.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.765+0000 7ffbe8ae8640 1 -- 172.21.15.33:0/2247006811 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbe419d280 con 0x7ffbe4106750 2024-03-19T03:02:27.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.765+0000 7ffbdb7fe640 1 -- 172.21.15.33:0/2247006811 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc010690 con 0x7ffbe4106750 2024-03-19T03:02:27.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.765+0000 7ffbe8ae8640 1 -- 172.21.15.33:0/2247006811 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbe41a4b10 con 0x7ffbe4106750 2024-03-19T03:02:27.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.767+0000 7ffbdb7fe640 1 -- 172.21.15.33:0/2247006811 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7ffbdc005020 con 0x7ffbe4106750 2024-03-19T03:02:27.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.767+0000 7ffbeb7ab740 1 Processor -- start 2024-03-19T03:02:27.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.767+0000 7ffbeb7ab740 1 -- start start 2024-03-19T03:02:27.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.767+0000 7ffbeb7ab740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56195f7c22d0 0x56195f7c26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.767+0000 7ffbeb7ab740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56195f7c2bf0 0x56195f7cb290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.767+0000 7ffbeb7ab740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56195f7cb7d0 0x56195f7cdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.767+0000 7ffbeb7ab740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56195f7254b0 con 0x56195f7c22d0 2024-03-19T03:02:27.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.768+0000 7ffbeb7ab740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x56195f6cdd70 con 0x56195f7c2bf0 2024-03-19T03:02:27.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.768+0000 7ffbeb7ab740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56195f721550 con 0x56195f7cb7d0 2024-03-19T03:02:27.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.768+0000 7ffbdb7fe640 1 --2- 172.21.15.33:0/2247006811 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7ffbd00a2970 0x7ffbd00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.776 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.768+0000 7ffbdbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56195f7c22d0 0x56195f7c26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.768+0000 7ffbe259c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56195f7c2bf0 0x56195f7cb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.768+0000 7ffbdbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56195f7c22d0 0x56195f7c26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38578/0 (socket says 172.21.15.33:38578) 2024-03-19T03:02:27.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.768+0000 7ffbdbfff640 1 -- 172.21.15.33:0/2577780983 learned_addr learned my addr 172.21.15.33:0/2577780983 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:27.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.768+0000 7ffbe2d9d640 1 --2- 172.21.15.33:0/2247006811 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7ffbd00a2970 0x7ffbd00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.768+0000 7ffbdbfff640 1 --2- 172.21.15.33:0/2577780983 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56195f7cb7d0 0x56195f7cdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.768+0000 7ffbdb7fe640 1 -- 172.21.15.33:0/2247006811 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7ffbdc0cf8f0 con 0x7ffbe4106750 2024-03-19T03:02:27.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.768+0000 7ffbe259c640 1 -- 172.21.15.33:0/2577780983 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56195f7cb7d0 msgr2=0x56195f7cdbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:27.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.768+0000 7ffbe259c640 1 --2- 172.21.15.33:0/2577780983 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56195f7cb7d0 0x56195f7cdbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.768+0000 7ffbe259c640 1 -- 172.21.15.33:0/2577780983 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56195f7c22d0 msgr2=0x56195f7c26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:27.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.768+0000 7ffbe259c640 1 --2- 172.21.15.33:0/2577780983 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56195f7c22d0 0x56195f7c26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.768+0000 7ffbe259c640 1 -- 172.21.15.33:0/2577780983 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56195f6f0210 con 0x56195f7c2bf0 2024-03-19T03:02:27.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.768+0000 7ffbdbfff640 1 --2- 172.21.15.33:0/2577780983 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56195f7c22d0 0x56195f7c26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:02:27.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.768+0000 7ffbe259c640 1 --2- 172.21.15.33:0/2577780983 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56195f7c2bf0 0x56195f7cb290 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ffbd4005fd0 tx=0x7ffbd40118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b53aa642e60b1f74 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:27.777 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.768+0000 7ffbe2d9d640 1 --2- 172.21.15.33:0/2247006811 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7ffbd00a2970 0x7ffbd00a4e30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc09b880 tx=0x7ffbdc01b8f0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:27.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.768+0000 7ffbd8ff9640 1 -- 172.21.15.33:0/2577780983 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd4004020 con 0x56195f7c2bf0 2024-03-19T03:02:27.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.768+0000 7ffbd8ff9640 1 -- 172.21.15.33:0/2577780983 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffbd40041c0 con 0x56195f7c2bf0 2024-03-19T03:02:27.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.768+0000 7ffbd8ff9640 1 -- 172.21.15.33:0/2577780983 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd400f870 con 0x56195f7c2bf0 2024-03-19T03:02:27.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.769+0000 7ffbeb7ab740 1 -- 172.21.15.33:0/2577780983 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56195f7c2bf0 msgr2=0x56195f7cb290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.769+0000 7ffbeb7ab740 1 --2- 172.21.15.33:0/2577780983 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56195f7c2bf0 0x56195f7cb290 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ffbd4005fd0 tx=0x7ffbd40118d0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.769+0000 7ffbeb7ab740 1 -- 172.21.15.33:0/2577780983 shutdown_connections 2024-03-19T03:02:27.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.769+0000 7ffbeb7ab740 1 --2- 172.21.15.33:0/2577780983 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56195f7cb7d0 0x56195f7cdbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.769+0000 7ffbeb7ab740 1 --2- 172.21.15.33:0/2577780983 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56195f7c2bf0 0x56195f7cb290 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.769+0000 7ffbeb7ab740 1 --2- 172.21.15.33:0/2577780983 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56195f7c22d0 0x56195f7c26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.769+0000 7ffbeb7ab740 1 -- 172.21.15.33:0/2577780983 >> 172.21.15.33:0/2577780983 conn(0x56195f7bca00 msgr2=0x56195f7bcde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:27.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.769+0000 7ffbeb7ab740 1 -- 172.21.15.33:0/2577780983 shutdown_connections 2024-03-19T03:02:27.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.769+0000 7ffbeb7ab740 1 -- 172.21.15.33:0/2577780983 wait complete. 2024-03-19T03:02:27.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.769+0000 7ffbeb7ab740 1 -- start start 2024-03-19T03:02:27.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.769+0000 7ffbeb7ab740 1 Processor -- start 2024-03-19T03:02:27.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.770+0000 7ffbeb7ab740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56195f7c22d0 0x56195f7c8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.770+0000 7ffbeb7ab740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56195f7c2bf0 0x56195f7c8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.770+0000 7ffbeb7ab740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56195f7cb7d0 0x56195f7c9210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.770+0000 7ffbeb7ab740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56195f7ce5c0 con 0x56195f7c2bf0 2024-03-19T03:02:27.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.770+0000 7ffbeb7ab740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x56195f6e1310 con 0x56195f7c22d0 2024-03-19T03:02:27.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.770+0000 7ffbeb7ab740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56195f7ce740 con 0x56195f7cb7d0 2024-03-19T03:02:27.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.770+0000 7ffbeb7ab740 10 client.0 _ll_register_callbacks cb 0x56195f6d52d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:02:27.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.770+0000 7ffbe259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56195f7c2bf0 0x56195f7c8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.770+0000 7ffbdbfff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56195f7c22d0 0x56195f7c8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.770+0000 7ffbe259c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56195f7c2bf0 0x56195f7c8cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38582/0 (socket says 172.21.15.33:38582) 2024-03-19T03:02:27.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.770+0000 7ffbe259c640 1 -- 172.21.15.33:0/1876610029 learned_addr learned my addr 172.21.15.33:0/1876610029 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:27.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.770+0000 7ffbdbfff640 1 --2- 172.21.15.33:0/1876610029 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56195f7cb7d0 0x56195f7c9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.770+0000 7ffbe259c640 1 -- 172.21.15.33:0/1876610029 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56195f7cb7d0 msgr2=0x56195f7c9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:27.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.770+0000 7ffbe259c640 1 --2- 172.21.15.33:0/1876610029 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56195f7cb7d0 0x56195f7c9210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.770+0000 7ffbe259c640 1 -- 172.21.15.33:0/1876610029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56195f7c22d0 msgr2=0x56195f7c8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:27.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.770+0000 7ffbe259c640 1 --2- 172.21.15.33:0/1876610029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56195f7c22d0 0x56195f7c8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.770+0000 7ffbe259c640 1 -- 172.21.15.33:0/1876610029 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56195f4393f0 con 0x56195f7c2bf0 2024-03-19T03:02:27.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.770+0000 7ffbdbfff640 1 --2- 172.21.15.33:0/1876610029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56195f7c22d0 0x56195f7c8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:02:27.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.770+0000 7ffbe259c640 1 --2- 172.21.15.33:0/1876610029 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56195f7c2bf0 0x56195f7c8cd0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7ffbd401b040 tx=0x7ffbd4003b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18c2e659e436ff11 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:27.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.770+0000 7ffbc9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:02:27.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.770+0000 7ffbc9ffb640 1 -- 172.21.15.33:0/1876610029 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd4027020 con 0x56195f7c2bf0 2024-03-19T03:02:27.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.770+0000 7ffbc9ffb640 1 -- 172.21.15.33:0/1876610029 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7ffbd4005040 con 0x56195f7c2bf0 2024-03-19T03:02:27.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.770+0000 7ffbeb7ab740 1 -- 172.21.15.33:0/1876610029 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbd40041c0 con 0x56195f7c2bf0 2024-03-19T03:02:27.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.771+0000 7ffbc9ffb640 1 -- 172.21.15.33:0/1876610029 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ffbd4032050 con 0x56195f7c2bf0 2024-03-19T03:02:27.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.771+0000 7ffbeb7ab740 10 client.17574 fetch_fsmap learned FSMap version 25 2024-03-19T03:02:27.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.771+0000 7ffbeb7ab740 1 -- 172.21.15.33:0/1876610029 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56195f5037d0 con 0x56195f7c2bf0 2024-03-19T03:02:27.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.771+0000 7ffbc9ffb640 1 -- 172.21.15.33:0/1876610029 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffbd40041c0 con 0x56195f7c2bf0 2024-03-19T03:02:27.780 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.771+0000 7ffbc9ffb640 1 -- 172.21.15.33:0/1876610029 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7ffbd4004b40 con 0x56195f7c2bf0 2024-03-19T03:02:27.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.771+0000 7ffbeb7ab740 10 client.17574 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:02:27.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.771+0000 7ffbeb7ab740 10 client.17574 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T03:02:27.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.771+0000 7ffbeb7ab740 20 client.17574 populate_metadata read hostname 'smithi033' 2024-03-19T03:02:27.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.771+0000 7ffbeb7ab740 1 --2- 172.21.15.33:0/1876610029 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x56195f7dbaf0 0x56195f7ddee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.771+0000 7ffbeb7ab740 4 client.17574 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:02:27.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.771+0000 7ffbeb7ab740 1 -- 172.21.15.33:0/1876610029 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56195f7833e0 con 0x56195f7dbaf0 2024-03-19T03:02:27.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.771+0000 7ffbdbfff640 1 --2- 172.21.15.33:0/1876610029 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x56195f7dbaf0 0x56195f7ddee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.772+0000 7ffbdbfff640 1 --2- 172.21.15.33:0/1876610029 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x56195f7dbaf0 0x56195f7ddee0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a6a51d3ea9b6f9e server_cookie=6049606de18f9329 in_seq=0 out_seq=0 2024-03-19T03:02:27.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.772+0000 7ffbc9ffb640 10 client.17574 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T03:02:27.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.772+0000 7ffbc9ffb640 1 -- 172.21.15.33:0/1876610029 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56195f7833e0 con 0x56195f7dbaf0 2024-03-19T03:02:27.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.772+0000 7ffbc9ffb640 10 client.17574 handle_command_reply: tid=0 2024-03-19T03:02:27.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.772+0000 7ffbeb7ab740 1 client.17574 shutdown 2024-03-19T03:02:27.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.772+0000 7ffbeb7ab740 1 -- 172.21.15.33:0/1876610029 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56195f7c2bf0 msgr2=0x56195f7c8cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.772+0000 7ffbeb7ab740 1 --2- 172.21.15.33:0/1876610029 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56195f7c2bf0 0x56195f7c8cd0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7ffbd401b040 tx=0x7ffbd4003b40 comp rx=0 tx=0).stop 2024-03-19T03:02:27.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.772+0000 7ffbeb7ab740 1 -- 172.21.15.33:0/1876610029 shutdown_connections 2024-03-19T03:02:27.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.772+0000 7ffbeb7ab740 1 --2- 172.21.15.33:0/1876610029 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x56195f7dbaf0 0x56195f7ddee0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.772+0000 7ffbeb7ab740 1 --2- 172.21.15.33:0/1876610029 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56195f7cb7d0 0x56195f7c9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.772+0000 7ffbeb7ab740 1 --2- 172.21.15.33:0/1876610029 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56195f7c2bf0 0x56195f7c8cd0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.772+0000 7ffbc9ffb640 0 client.17574 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T03:02:27.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.772+0000 7ffbeb7ab740 1 --2- 172.21.15.33:0/1876610029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56195f7c22d0 0x56195f7c8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.772+0000 7ffbeb7ab740 1 -- 172.21.15.33:0/1876610029 >> 172.21.15.33:0/1876610029 conn(0x56195f7bca00 msgr2=0x56195f7be750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:27.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.772+0000 7ffbeb7ab740 1 -- 172.21.15.33:0/1876610029 shutdown_connections 2024-03-19T03:02:27.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.772+0000 7ffbeb7ab740 1 -- 172.21.15.33:0/1876610029 wait complete. 2024-03-19T03:02:27.782 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.772+0000 7ffbeb7ab740 20 client.17574 trim_cache size 0 max 16384 2024-03-19T03:02:27.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.786+0000 7ffbeb7ab740 1 Processor -- start 2024-03-19T03:02:27.792 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.786+0000 7ffbeb7ab740 1 -- start start 2024-03-19T03:02:27.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.786+0000 7ffbeb7ab740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56195f7c22d0 0x56195f7caec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.786+0000 7ffbeb7ab740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56195f7c2bf0 0x56195f7d73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.786+0000 7ffbeb7ab740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56195f7cb7d0 0x56195f7e3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.786+0000 7ffbeb7ab740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe41b1550 con 0x56195f7c22d0 2024-03-19T03:02:27.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.786+0000 7ffbeb7ab740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe41aee20 con 0x56195f7cb7d0 2024-03-19T03:02:27.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.786+0000 7ffbeb7ab740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe41acc90 con 0x56195f7c2bf0 2024-03-19T03:02:27.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.786+0000 7ffbdbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56195f7c22d0 0x56195f7caec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.786+0000 7ffbdbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56195f7c22d0 0x56195f7caec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38586/0 (socket says 172.21.15.33:38586) 2024-03-19T03:02:27.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.786+0000 7ffbdbfff640 1 -- 172.21.15.33:0/2760197473 learned_addr learned my addr 172.21.15.33:0/2760197473 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:27.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.786+0000 7ffbe259c640 1 --2- 172.21.15.33:0/2760197473 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56195f7c2bf0 0x56195f7d73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.786+0000 7ffbdbfff640 1 --2- 172.21.15.33:0/2760197473 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56195f7cb7d0 0x56195f7e3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.786+0000 7ffbdbfff640 1 -- 172.21.15.33:0/2760197473 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56195f7c2bf0 msgr2=0x56195f7d73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:27.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.786+0000 7ffbdbfff640 1 --2- 172.21.15.33:0/2760197473 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56195f7c2bf0 0x56195f7d73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.786+0000 7ffbdbfff640 1 -- 172.21.15.33:0/2760197473 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56195f7cb7d0 msgr2=0x56195f7e3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:27.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.786+0000 7ffbdbfff640 1 --2- 172.21.15.33:0/2760197473 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56195f7cb7d0 0x56195f7e3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.786+0000 7ffbdbfff640 1 -- 172.21.15.33:0/2760197473 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56195f7c2a20 con 0x56195f7c22d0 2024-03-19T03:02:27.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.786+0000 7ffbe259c640 1 --2- 172.21.15.33:0/2760197473 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56195f7c2bf0 0x56195f7d73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:02:27.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.786+0000 7ffbdbfff640 1 --2- 172.21.15.33:0/2760197473 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56195f7c22d0 0x56195f7caec0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc037040 tx=0x7ffbcc0056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d801b3accf72e511 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:27.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.787+0000 7ffbcbfff640 1 -- 172.21.15.33:0/2760197473 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc039040 con 0x56195f7c22d0 2024-03-19T03:02:27.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.787+0000 7ffbcbfff640 1 -- 172.21.15.33:0/2760197473 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffbcc051030 con 0x56195f7c22d0 2024-03-19T03:02:27.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.787+0000 7ffbcbfff640 1 -- 172.21.15.33:0/2760197473 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc04c740 con 0x56195f7c22d0 2024-03-19T03:02:27.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.787+0000 7ffbeb7ab740 1 -- 172.21.15.33:0/2760197473 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56195f7c22d0 msgr2=0x56195f7caec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.787+0000 7ffbeb7ab740 1 --2- 172.21.15.33:0/2760197473 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56195f7c22d0 0x56195f7caec0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc037040 tx=0x7ffbcc0056d0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.787+0000 7ffbeb7ab740 1 -- 172.21.15.33:0/2760197473 shutdown_connections 2024-03-19T03:02:27.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.787+0000 7ffbeb7ab740 1 --2- 172.21.15.33:0/2760197473 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56195f7c2bf0 0x56195f7d73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.787+0000 7ffbeb7ab740 1 --2- 172.21.15.33:0/2760197473 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56195f7cb7d0 0x56195f7e3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.794 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.787+0000 7ffbeb7ab740 1 --2- 172.21.15.33:0/2760197473 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56195f7c22d0 0x56195f7caec0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.787+0000 7ffbeb7ab740 1 -- 172.21.15.33:0/2760197473 >> 172.21.15.33:0/2760197473 conn(0x56195f7bca00 msgr2=0x56195f7b7a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:27.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.787+0000 7ffbeb7ab740 1 -- 172.21.15.33:0/2760197473 shutdown_connections 2024-03-19T03:02:27.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.787+0000 7ffbeb7ab740 1 -- 172.21.15.33:0/2760197473 wait complete. 2024-03-19T03:02:27.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.787+0000 7ffbeb7ab740 1 -- start start 2024-03-19T03:02:27.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.788+0000 7ffbeb7ab740 1 Processor -- start 2024-03-19T03:02:27.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.788+0000 7ffbeb7ab740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56195f7c22d0 0x56195f7d39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.788+0000 7ffbeb7ab740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56195f7c2bf0 0x56195f7c4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.788+0000 7ffbeb7ab740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56195f7cb7d0 0x56195f7c5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.788+0000 7ffbeb7ab740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe41ad2e0 con 0x56195f7cb7d0 2024-03-19T03:02:27.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.788+0000 7ffbeb7ab740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe41ae750 con 0x56195f7c22d0 2024-03-19T03:02:27.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.788+0000 7ffbeb7ab740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe41ad9b0 con 0x56195f7c2bf0 2024-03-19T03:02:27.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.788+0000 7ffbeb7ab740 10 client.0 _ll_register_callbacks cb 0x7ffbd4005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:02:27.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.788+0000 7ffbdbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56195f7cb7d0 0x56195f7c5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.788+0000 7ffbdbfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56195f7cb7d0 0x56195f7c5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38596/0 (socket says 172.21.15.33:38596) 2024-03-19T03:02:27.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.788+0000 7ffbdbfff640 1 -- 172.21.15.33:0/1280808603 learned_addr learned my addr 172.21.15.33:0/1280808603 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:27.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.788+0000 7ffbdbfff640 1 --2- 172.21.15.33:0/1280808603 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56195f7c22d0 0x56195f7d39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.788+0000 7ffbe259c640 1 --2- 172.21.15.33:0/1280808603 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56195f7c2bf0 0x56195f7c4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.788+0000 7ffbdbfff640 1 -- 172.21.15.33:0/1280808603 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56195f7c2bf0 msgr2=0x56195f7c4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:27.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.788+0000 7ffbdbfff640 1 --2- 172.21.15.33:0/1280808603 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56195f7c2bf0 0x56195f7c4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.788+0000 7ffbdbfff640 1 -- 172.21.15.33:0/1280808603 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56195f7c22d0 msgr2=0x56195f7d39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:27.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.788+0000 7ffbdbfff640 1 --2- 172.21.15.33:0/1280808603 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56195f7c22d0 0x56195f7d39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.788+0000 7ffbdbfff640 1 -- 172.21.15.33:0/1280808603 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56195f7c5890 con 0x56195f7cb7d0 2024-03-19T03:02:27.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.788+0000 7ffbdbfff640 1 --2- 172.21.15.33:0/1280808603 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56195f7cb7d0 0x56195f7c5350 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc005b60 tx=0x7ffbcc04f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99d72a4392d52b5b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:27.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.788+0000 7ffbc9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:02:27.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.788+0000 7ffbc9ffb640 1 -- 172.21.15.33:0/1280808603 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc039070 con 0x56195f7cb7d0 2024-03-19T03:02:27.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.789+0000 7ffbeb7ab740 1 -- 172.21.15.33:0/1280808603 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbcc051030 con 0x56195f7cb7d0 2024-03-19T03:02:27.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.789+0000 7ffbc9ffb640 1 -- 172.21.15.33:0/1280808603 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7ffbcc02c7f0 con 0x56195f7cb7d0 2024-03-19T03:02:27.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.789+0000 7ffbc9ffb640 1 -- 172.21.15.33:0/1280808603 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ffbcc02c9f0 con 0x56195f7cb7d0 2024-03-19T03:02:27.796 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.789+0000 7ffbeb7ab740 10 client.17580 fetch_fsmap learned FSMap version 25 2024-03-19T03:02:27.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.789+0000 7ffbeb7ab740 1 -- 172.21.15.33:0/1280808603 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56195f841aa0 con 0x56195f7cb7d0 2024-03-19T03:02:27.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.789+0000 7ffbc9ffb640 1 -- 172.21.15.33:0/1280808603 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffbcc051030 con 0x56195f7cb7d0 2024-03-19T03:02:27.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.789+0000 7ffbc9ffb640 1 -- 172.21.15.33:0/1280808603 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7ffbcc060070 con 0x56195f7cb7d0 2024-03-19T03:02:27.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.789+0000 7ffbeb7ab740 10 client.17580 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:02:27.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.789+0000 7ffbeb7ab740 10 client.17580 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T03:02:27.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.789+0000 7ffbeb7ab740 20 client.17580 populate_metadata read hostname 'smithi033' 2024-03-19T03:02:27.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.789+0000 7ffbeb7ab740 1 --2- 172.21.15.33:0/1280808603 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x56195f842180 0x56195f8445a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.789+0000 7ffbeb7ab740 4 client.17580 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T03:02:27.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.789+0000 7ffbeb7ab740 1 -- 172.21.15.33:0/1280808603 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56195f7329f0 con 0x56195f842180 2024-03-19T03:02:27.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.789+0000 7ffbe259c640 1 --2- 172.21.15.33:0/1280808603 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x56195f842180 0x56195f8445a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.790+0000 7ffbc9ffb640 10 client.17580 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T03:02:27.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.790+0000 7ffbe259c640 1 --2- 172.21.15.33:0/1280808603 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x56195f842180 0x56195f8445a0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=61f7fd8a386e94a0 server_cookie=e3cf322b60943d3d in_seq=0 out_seq=0 2024-03-19T03:02:27.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.790+0000 7ffbc9ffb640 1 -- 172.21.15.33:0/1280808603 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56195f7329f0 con 0x56195f842180 2024-03-19T03:02:27.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.790+0000 7ffbc9ffb640 10 client.17580 handle_command_reply: tid=0 2024-03-19T03:02:27.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.790+0000 7ffbeb7ab740 1 client.17580 shutdown 2024-03-19T03:02:27.797 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.790+0000 7ffbeb7ab740 1 -- 172.21.15.33:0/1280808603 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56195f7cb7d0 msgr2=0x56195f7c5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.790+0000 7ffbeb7ab740 1 --2- 172.21.15.33:0/1280808603 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56195f7cb7d0 0x56195f7c5350 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc005b60 tx=0x7ffbcc04f040 comp rx=0 tx=0).stop 2024-03-19T03:02:27.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.790+0000 7ffbeb7ab740 1 -- 172.21.15.33:0/1280808603 shutdown_connections 2024-03-19T03:02:27.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.790+0000 7ffbeb7ab740 1 --2- 172.21.15.33:0/1280808603 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x56195f842180 0x56195f8445a0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.790+0000 7ffbeb7ab740 1 --2- 172.21.15.33:0/1280808603 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x56195f7c2bf0 0x56195f7c4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.790+0000 7ffbeb7ab740 1 --2- 172.21.15.33:0/1280808603 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56195f7cb7d0 0x56195f7c5350 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.790+0000 7ffbc9ffb640 0 client.17580 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T03:02:27.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.790+0000 7ffbeb7ab740 1 --2- 172.21.15.33:0/1280808603 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x56195f7c22d0 0x56195f7d39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.790+0000 7ffbeb7ab740 1 -- 172.21.15.33:0/1280808603 >> 172.21.15.33:0/1280808603 conn(0x56195f7bca00 msgr2=0x56195f7bb310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:27.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.790+0000 7ffbeb7ab740 1 -- 172.21.15.33:0/1280808603 shutdown_connections 2024-03-19T03:02:27.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.790+0000 7ffbeb7ab740 1 -- 172.21.15.33:0/1280808603 wait complete. 2024-03-19T03:02:27.798 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.790+0000 7ffbeb7ab740 20 client.17580 trim_cache size 0 max 16384 2024-03-19T03:02:27.799 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T03:02:27.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.790+0000 7ffbe8ae8640 1 -- 172.21.15.33:0/2247006811 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7ffbd00a2970 msgr2=0x7ffbd00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.790+0000 7ffbe8ae8640 1 --2- 172.21.15.33:0/2247006811 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7ffbd00a2970 0x7ffbd00a4e30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc09b880 tx=0x7ffbdc01b8f0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.799 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.791+0000 7ffbe8ae8640 1 -- 172.21.15.33:0/2247006811 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffbe4106750 msgr2=0x7ffbe419c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.791+0000 7ffbe8ae8640 1 --2- 172.21.15.33:0/2247006811 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffbe4106750 0x7ffbe419c860 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc00eb10 tx=0x7ffbdc00c2d0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.791+0000 7ffbe8ae8640 1 -- 172.21.15.33:0/2247006811 shutdown_connections 2024-03-19T03:02:27.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.791+0000 7ffbe8ae8640 1 --2- 172.21.15.33:0/2247006811 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7ffbd00a2970 0x7ffbd00a4e30 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.791+0000 7ffbe8ae8640 1 --2- 172.21.15.33:0/2247006811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbe4101310 0x7ffbe41984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.791+0000 7ffbe8ae8640 1 --2- 172.21.15.33:0/2247006811 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7ffbe4106750 0x7ffbe419c860 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.791+0000 7ffbe8ae8640 1 --2- 172.21.15.33:0/2247006811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffbe4100970 0x7ffbe4197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.791+0000 7ffbe8ae8640 1 -- 172.21.15.33:0/2247006811 >> 172.21.15.33:0/2247006811 conn(0x7ffbe40774c0 msgr2=0x7ffbe40ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:27.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.791+0000 7ffbe8ae8640 1 -- 172.21.15.33:0/2247006811 shutdown_connections 2024-03-19T03:02:27.800 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.791+0000 7ffbe8ae8640 1 -- 172.21.15.33:0/2247006811 wait complete. 2024-03-19T03:02:27.802 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T03:02:27.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.906+0000 7f7c52675640 1 -- 172.21.15.33:0/4216985059 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c4c102170 msgr2=0x7f7c4c10e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.906+0000 7f7c52675640 1 --2- 172.21.15.33:0/4216985059 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c4c102170 0x7f7c4c10e680 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009cd0 tx=0x7f7c4402c0f0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.906+0000 7f7c52675640 1 -- 172.21.15.33:0/4216985059 shutdown_connections 2024-03-19T03:02:27.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.906+0000 7f7c52675640 1 --2- 172.21.15.33:0/4216985059 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c4c1017b0 0x7f7c4c101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.910 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.906+0000 7f7c52675640 1 --2- 172.21.15.33:0/4216985059 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c4c102170 0x7f7c4c10e680 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.906+0000 7f7c52675640 1 --2- 172.21.15.33:0/4216985059 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c4c1077c0 0x7f7c4c107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.906+0000 7f7c52675640 1 -- 172.21.15.33:0/4216985059 >> 172.21.15.33:0/4216985059 conn(0x7f7c4c0fd2e0 msgr2=0x7f7c4c0ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:27.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.907+0000 7f7c52675640 1 -- 172.21.15.33:0/4216985059 shutdown_connections 2024-03-19T03:02:27.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.907+0000 7f7c52675640 1 -- 172.21.15.33:0/4216985059 wait complete. 2024-03-19T03:02:27.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.907+0000 7f7c52675640 1 Processor -- start 2024-03-19T03:02:27.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.907+0000 7f7c52675640 1 -- start start 2024-03-19T03:02:27.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.907+0000 7f7c52675640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c4c1017b0 0x7f7c4c197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.907+0000 7f7c52675640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c4c102170 0x7f7c4c1984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.907+0000 7f7c52675640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c4c1077c0 0x7f7c4c19c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.907+0000 7f7c52675640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c111a40 con 0x7f7c4c1077c0 2024-03-19T03:02:27.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.907+0000 7f7c52675640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c1118c0 con 0x7f7c4c102170 2024-03-19T03:02:27.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.907+0000 7f7c52675640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c111bc0 con 0x7f7c4c1017b0 2024-03-19T03:02:27.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.908+0000 7f7c50c12640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c4c1077c0 0x7f7c4c19c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.908+0000 7f7c50c12640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c4c1077c0 0x7f7c4c19c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38620/0 (socket says 172.21.15.33:38620) 2024-03-19T03:02:27.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.908+0000 7f7c50c12640 1 -- 172.21.15.33:0/1377144729 learned_addr learned my addr 172.21.15.33:0/1377144729 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:27.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.908+0000 7f7c4bfff640 1 --2- 172.21.15.33:0/1377144729 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c4c1017b0 0x7f7c4c197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.908+0000 7f7c4b7fe640 1 --2- 172.21.15.33:0/1377144729 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c4c102170 0x7f7c4c1984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.908+0000 7f7c50c12640 1 -- 172.21.15.33:0/1377144729 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c4c1017b0 msgr2=0x7f7c4c197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.908+0000 7f7c50c12640 1 --2- 172.21.15.33:0/1377144729 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c4c1017b0 0x7f7c4c197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.908+0000 7f7c50c12640 1 -- 172.21.15.33:0/1377144729 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c4c102170 msgr2=0x7f7c4c1984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.908+0000 7f7c50c12640 1 --2- 172.21.15.33:0/1377144729 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c4c102170 0x7f7c4c1984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.908+0000 7f7c50c12640 1 -- 172.21.15.33:0/1377144729 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c4c19cf70 con 0x7f7c4c1077c0 2024-03-19T03:02:27.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.908+0000 7f7c50c12640 1 --2- 172.21.15.33:0/1377144729 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c4c1077c0 0x7f7c4c19c840 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f7c44004af0 tx=0x7f7c44035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:27.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.908+0000 7f7c497fa640 1 -- 172.21.15.33:0/1377144729 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c44039040 con 0x7f7c4c1077c0 2024-03-19T03:02:27.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.908+0000 7f7c497fa640 1 -- 172.21.15.33:0/1377144729 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c44044070 con 0x7f7c4c1077c0 2024-03-19T03:02:27.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.908+0000 7f7c52675640 1 -- 172.21.15.33:0/1377144729 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c4c19d200 con 0x7f7c4c1077c0 2024-03-19T03:02:27.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.908+0000 7f7c497fa640 1 -- 172.21.15.33:0/1377144729 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c440042a0 con 0x7f7c4c1077c0 2024-03-19T03:02:27.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.908+0000 7f7c52675640 1 -- 172.21.15.33:0/1377144729 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c4c1a4af0 con 0x7f7c4c1077c0 2024-03-19T03:02:27.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.911+0000 7f7c497fa640 1 -- 172.21.15.33:0/1377144729 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f7c44003970 con 0x7f7c4c1077c0 2024-03-19T03:02:27.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.911+0000 7f7c5525b740 1 Processor -- start 2024-03-19T03:02:27.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.911+0000 7f7c5525b740 1 -- start start 2024-03-19T03:02:27.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.911+0000 7f7c497fa640 1 --2- 172.21.15.33:0/1377144729 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f7c380a2970 0x7f7c380a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.911+0000 7f7c5525b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561e8fbbc2d0 0x561e8fbbc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.911+0000 7f7c5525b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561e8fbbcbf0 0x561e8fbc5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.911+0000 7f7c5525b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561e8fbc57d0 0x561e8fbc7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.911+0000 7f7c5525b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561e8fb1f4b0 con 0x561e8fbbc2d0 2024-03-19T03:02:27.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.911+0000 7f7c5525b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x561e8fac7d70 con 0x561e8fbc57d0 2024-03-19T03:02:27.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.911+0000 7f7c5525b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x561e8fb1b550 con 0x561e8fbbcbf0 2024-03-19T03:02:27.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.911+0000 7f7c4b7fe640 1 --2- 172.21.15.33:0/1377144729 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f7c380a2970 0x7f7c380a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.911+0000 7f7c4bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561e8fbbc2d0 0x561e8fbbc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.911+0000 7f7c497fa640 1 -- 172.21.15.33:0/1377144729 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f7c440f1510 con 0x7f7c4c1077c0 2024-03-19T03:02:27.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.911+0000 7f7c4bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561e8fbbc2d0 0x561e8fbbc6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38628/0 (socket says 172.21.15.33:38628) 2024-03-19T03:02:27.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.911+0000 7f7c4bfff640 1 -- 172.21.15.33:0/3153165724 learned_addr learned my addr 172.21.15.33:0/3153165724 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:27.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.911+0000 7f7c50c12640 1 --2- 172.21.15.33:0/3153165724 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561e8fbc57d0 0x561e8fbc7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.911+0000 7f7c4b7fe640 1 --2- 172.21.15.33:0/3153165724 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561e8fbbcbf0 0x561e8fbc5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.912+0000 7f7c4bfff640 1 -- 172.21.15.33:0/3153165724 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561e8fbbcbf0 msgr2=0x561e8fbc5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:27.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.912+0000 7f7c4bfff640 1 --2- 172.21.15.33:0/3153165724 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561e8fbbcbf0 0x561e8fbc5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.912+0000 7f7c4bfff640 1 -- 172.21.15.33:0/3153165724 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561e8fbc57d0 msgr2=0x561e8fbc7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:27.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.912+0000 7f7c4bfff640 1 --2- 172.21.15.33:0/3153165724 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561e8fbc57d0 0x561e8fbc7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.912+0000 7f7c4bfff640 1 -- 172.21.15.33:0/3153165724 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e8faea210 con 0x561e8fbbc2d0 2024-03-19T03:02:27.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.912+0000 7f7c4bfff640 1 --2- 172.21.15.33:0/3153165724 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561e8fbbc2d0 0x561e8fbbc6b0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c009c70 tx=0x7f7c3c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a590d790fa6b36 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:27.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.912+0000 7f7c4b7fe640 1 --2- 172.21.15.33:0/1377144729 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f7c380a2970 0x7f7c380a4e30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x561e8fbbbda0 tx=0x7f7c34009920 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:27.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.912+0000 7f7c427fc640 1 -- 172.21.15.33:0/3153165724 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c004230 con 0x561e8fbbc2d0 2024-03-19T03:02:27.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.912+0000 7f7c427fc640 1 -- 172.21.15.33:0/3153165724 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c3c0043d0 con 0x561e8fbbc2d0 2024-03-19T03:02:27.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.912+0000 7f7c427fc640 1 -- 172.21.15.33:0/3153165724 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c0196b0 con 0x561e8fbbc2d0 2024-03-19T03:02:27.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.913+0000 7f7c5525b740 1 -- 172.21.15.33:0/3153165724 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561e8fbbc2d0 msgr2=0x561e8fbbc6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.913+0000 7f7c5525b740 1 --2- 172.21.15.33:0/3153165724 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561e8fbbc2d0 0x561e8fbbc6b0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c009c70 tx=0x7f7c3c011810 comp rx=0 tx=0).stop 2024-03-19T03:02:27.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.913+0000 7f7c5525b740 1 -- 172.21.15.33:0/3153165724 shutdown_connections 2024-03-19T03:02:27.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.913+0000 7f7c5525b740 1 --2- 172.21.15.33:0/3153165724 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561e8fbbcbf0 0x561e8fbc5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.913+0000 7f7c5525b740 1 --2- 172.21.15.33:0/3153165724 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561e8fbc57d0 0x561e8fbc7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.913+0000 7f7c5525b740 1 --2- 172.21.15.33:0/3153165724 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561e8fbbc2d0 0x561e8fbbc6b0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.913+0000 7f7c5525b740 1 -- 172.21.15.33:0/3153165724 >> 172.21.15.33:0/3153165724 conn(0x561e8fbb6a00 msgr2=0x561e8fbb6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:27.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.913+0000 7f7c5525b740 1 -- 172.21.15.33:0/3153165724 shutdown_connections 2024-03-19T03:02:27.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.913+0000 7f7c5525b740 1 -- 172.21.15.33:0/3153165724 wait complete. 2024-03-19T03:02:27.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.913+0000 7f7c5525b740 1 -- start start 2024-03-19T03:02:27.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.913+0000 7f7c5525b740 1 Processor -- start 2024-03-19T03:02:27.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.913+0000 7f7c5525b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561e8fbbc2d0 0x561e8fbc2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.913+0000 7f7c5525b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561e8fbbcbf0 0x561e8fbc2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.913+0000 7f7c5525b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561e8fbc57d0 0x561e8fbc3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.913+0000 7f7c5525b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x561e8fbc85c0 con 0x561e8fbbcbf0 2024-03-19T03:02:27.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.913+0000 7f7c5525b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x561e8fadb310 con 0x561e8fbc57d0 2024-03-19T03:02:27.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.913+0000 7f7c5525b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x561e8fbc8740 con 0x561e8fbbc2d0 2024-03-19T03:02:27.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.913+0000 7f7c5525b740 10 client.0 _ll_register_callbacks cb 0x561e8facf2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:02:27.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.914+0000 7f7c50c12640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561e8fbc57d0 0x561e8fbc3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.914+0000 7f7c50c12640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561e8fbc57d0 0x561e8fbc3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:40898/0 (socket says 172.21.15.33:40898) 2024-03-19T03:02:27.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.914+0000 7f7c4b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561e8fbbcbf0 0x561e8fbc2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.922 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.914+0000 7f7c50c12640 1 -- 172.21.15.33:0/2185671997 learned_addr learned my addr 172.21.15.33:0/2185671997 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:27.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.914+0000 7f7c4bfff640 1 --2- 172.21.15.33:0/2185671997 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561e8fbbc2d0 0x561e8fbc2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.914+0000 7f7c50c12640 1 -- 172.21.15.33:0/2185671997 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561e8fbbc2d0 msgr2=0x561e8fbc2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:27.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.914+0000 7f7c50c12640 1 --2- 172.21.15.33:0/2185671997 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561e8fbbc2d0 0x561e8fbc2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.914+0000 7f7c50c12640 1 -- 172.21.15.33:0/2185671997 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561e8fbbcbf0 msgr2=0x561e8fbc2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:27.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.914+0000 7f7c50c12640 1 --2- 172.21.15.33:0/2185671997 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561e8fbbcbf0 0x561e8fbc2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.914+0000 7f7c50c12640 1 -- 172.21.15.33:0/2185671997 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e8f8333f0 con 0x561e8fbc57d0 2024-03-19T03:02:27.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.914+0000 7f7c4b7fe640 1 --2- 172.21.15.33:0/2185671997 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561e8fbbcbf0 0x561e8fbc2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:02:27.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.914+0000 7f7c50c12640 1 --2- 172.21.15.33:0/2185671997 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561e8fbc57d0 0x561e8fbc3290 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7c440bd860 tx=0x7f7c440bd230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=527528a249aa41a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:27.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.914+0000 7f7c2b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T03:02:27.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.914+0000 7f7c2b7fe640 1 -- 172.21.15.33:0/2185671997 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c44034700 con 0x561e8fbc57d0 2024-03-19T03:02:27.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.914+0000 7f7c5525b740 1 -- 172.21.15.33:0/2185671997 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c3c0043d0 con 0x561e8fbc57d0 2024-03-19T03:02:27.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.914+0000 7f7c2b7fe640 1 -- 172.21.15.33:0/2185671997 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f7c440f1170 con 0x561e8fbc57d0 2024-03-19T03:02:27.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.914+0000 7f7c2b7fe640 1 -- 172.21.15.33:0/2185671997 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7c440d6050 con 0x561e8fbc57d0 2024-03-19T03:02:27.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.914+0000 7f7c5525b740 10 client.27413 fetch_fsmap learned FSMap version 25 2024-03-19T03:02:27.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.914+0000 7f7c5525b740 1 -- 172.21.15.33:0/2185671997 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x561e8f8fd7d0 con 0x561e8fbc57d0 2024-03-19T03:02:27.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.914+0000 7f7c2b7fe640 1 -- 172.21.15.33:0/2185671997 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c3c0043d0 con 0x561e8fbc57d0 2024-03-19T03:02:27.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.915+0000 7f7c2b7fe640 1 -- 172.21.15.33:0/2185671997 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f7c4403c030 con 0x561e8fbc57d0 2024-03-19T03:02:27.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.915+0000 7f7c5525b740 10 client.27413 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:02:27.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.915+0000 7f7c5525b740 10 client.27413 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:02:27.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.915+0000 7f7c5525b740 20 client.27413 populate_metadata read hostname 'smithi033' 2024-03-19T03:02:27.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.915+0000 7f7c5525b740 1 --2- 172.21.15.33:0/2185671997 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x561e8fbd5b10 0x561e8fbd7f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.915+0000 7f7c5525b740 4 client.27413 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:02:27.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.915+0000 7f7c5525b740 1 -- 172.21.15.33:0/2185671997 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561e8fb7d3e0 con 0x561e8fbd5b10 2024-03-19T03:02:27.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.915+0000 7f7c4bfff640 1 --2- 172.21.15.33:0/2185671997 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x561e8fbd5b10 0x561e8fbd7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.915+0000 7f7c4bfff640 1 --2- 172.21.15.33:0/2185671997 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x561e8fbd5b10 0x561e8fbd7f70 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2bcdc2c8f45d304 server_cookie=2de18410a9c550a1 in_seq=0 out_seq=0 2024-03-19T03:02:27.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.915+0000 7f7c2b7fe640 10 client.27413 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:02:27.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.916+0000 7f7c2b7fe640 1 -- 172.21.15.33:0/2185671997 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561e8fb7d3e0 con 0x561e8fbd5b10 2024-03-19T03:02:27.924 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.916+0000 7f7c2b7fe640 10 client.27413 handle_command_reply: tid=0 2024-03-19T03:02:27.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.916+0000 7f7c5525b740 1 client.27413 shutdown 2024-03-19T03:02:27.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.916+0000 7f7c5525b740 1 -- 172.21.15.33:0/2185671997 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561e8fbc57d0 msgr2=0x561e8fbc3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.916+0000 7f7c5525b740 1 --2- 172.21.15.33:0/2185671997 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561e8fbc57d0 0x561e8fbc3290 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7c440bd860 tx=0x7f7c440bd230 comp rx=0 tx=0).stop 2024-03-19T03:02:27.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.916+0000 7f7c5525b740 1 -- 172.21.15.33:0/2185671997 shutdown_connections 2024-03-19T03:02:27.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.916+0000 7f7c5525b740 1 --2- 172.21.15.33:0/2185671997 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x561e8fbd5b10 0x561e8fbd7f70 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.916+0000 7f7c5525b740 1 --2- 172.21.15.33:0/2185671997 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561e8fbc57d0 0x561e8fbc3290 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.916+0000 7f7c5525b740 1 --2- 172.21.15.33:0/2185671997 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561e8fbbcbf0 0x561e8fbc2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.916+0000 7f7c5525b740 1 --2- 172.21.15.33:0/2185671997 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561e8fbbc2d0 0x561e8fbc2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.916+0000 7f7c2b7fe640 0 client.27413 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:02:27.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.916+0000 7f7c5525b740 1 -- 172.21.15.33:0/2185671997 >> 172.21.15.33:0/2185671997 conn(0x561e8fbb6a00 msgr2=0x561e8fbb8750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:27.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.916+0000 7f7c5525b740 1 -- 172.21.15.33:0/2185671997 shutdown_connections 2024-03-19T03:02:27.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.916+0000 7f7c5525b740 1 -- 172.21.15.33:0/2185671997 wait complete. 2024-03-19T03:02:27.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.916+0000 7f7c5525b740 20 client.27413 trim_cache size 0 max 16384 2024-03-19T03:02:27.936 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.930+0000 7f7c5525b740 1 Processor -- start 2024-03-19T03:02:27.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.930+0000 7f7c5525b740 1 -- start start 2024-03-19T03:02:27.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.930+0000 7f7c5525b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561e8fbbc2d0 0x561e8fbc4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.930+0000 7f7c5525b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561e8fbbcbf0 0x561e8fbd1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.930+0000 7f7c5525b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561e8fbc57d0 0x561e8fbb6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.930+0000 7f7c5525b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c1b1500 con 0x561e8fbc57d0 2024-03-19T03:02:27.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.930+0000 7f7c5525b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c1aedd0 con 0x561e8fbbcbf0 2024-03-19T03:02:27.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.930+0000 7f7c5525b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c1acc40 con 0x561e8fbbc2d0 2024-03-19T03:02:27.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.930+0000 7f7c50c12640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561e8fbc57d0 0x561e8fbb6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.930+0000 7f7c4b7fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561e8fbbcbf0 0x561e8fbd1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.930+0000 7f7c50c12640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561e8fbc57d0 0x561e8fbb6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38632/0 (socket says 172.21.15.33:38632) 2024-03-19T03:02:27.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.930+0000 7f7c50c12640 1 -- 172.21.15.33:0/2135964669 learned_addr learned my addr 172.21.15.33:0/2135964669 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:27.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.930+0000 7f7c4bfff640 1 --2- 172.21.15.33:0/2135964669 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561e8fbbc2d0 0x561e8fbc4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.930+0000 7f7c50c12640 1 -- 172.21.15.33:0/2135964669 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561e8fbbc2d0 msgr2=0x561e8fbc4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:27.937 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.930+0000 7f7c50c12640 1 --2- 172.21.15.33:0/2135964669 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561e8fbbc2d0 0x561e8fbc4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.938 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.930+0000 7f7c50c12640 1 -- 172.21.15.33:0/2135964669 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561e8fbbcbf0 msgr2=0x561e8fbd1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:27.938 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.930+0000 7f7c50c12640 1 --2- 172.21.15.33:0/2135964669 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561e8fbbcbf0 0x561e8fbd1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.938 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.930+0000 7f7c50c12640 1 -- 172.21.15.33:0/2135964669 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e8fbc17c0 con 0x561e8fbc57d0 2024-03-19T03:02:27.938 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.930+0000 7f7c50c12640 1 --2- 172.21.15.33:0/2135964669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561e8fbc57d0 0x561e8fbb6de0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f7c4402c6d0 tx=0x7f7c4402c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afd460200096eacc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:27.938 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.930+0000 7f7c41ffb640 1 -- 172.21.15.33:0/2135964669 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c44034700 con 0x561e8fbc57d0 2024-03-19T03:02:27.938 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.930+0000 7f7c41ffb640 1 -- 172.21.15.33:0/2135964669 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c440098d0 con 0x561e8fbc57d0 2024-03-19T03:02:27.938 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.930+0000 7f7c41ffb640 1 -- 172.21.15.33:0/2135964669 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c440a80b0 con 0x561e8fbc57d0 2024-03-19T03:02:27.938 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.931+0000 7f7c5525b740 1 -- 172.21.15.33:0/2135964669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561e8fbc57d0 msgr2=0x561e8fbb6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.938 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.931+0000 7f7c5525b740 1 --2- 172.21.15.33:0/2135964669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561e8fbc57d0 0x561e8fbb6de0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f7c4402c6d0 tx=0x7f7c4402c0f0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.938 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.931+0000 7f7c5525b740 1 -- 172.21.15.33:0/2135964669 shutdown_connections 2024-03-19T03:02:27.938 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.931+0000 7f7c5525b740 1 --2- 172.21.15.33:0/2135964669 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561e8fbc57d0 0x561e8fbb6de0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.938 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.931+0000 7f7c5525b740 1 --2- 172.21.15.33:0/2135964669 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561e8fbbcbf0 0x561e8fbd1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.938 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.931+0000 7f7c5525b740 1 --2- 172.21.15.33:0/2135964669 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561e8fbbc2d0 0x561e8fbc4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.931+0000 7f7c5525b740 1 -- 172.21.15.33:0/2135964669 >> 172.21.15.33:0/2135964669 conn(0x561e8fbb6a00 msgr2=0x561e8fbb5ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:27.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.931+0000 7f7c5525b740 1 -- 172.21.15.33:0/2135964669 shutdown_connections 2024-03-19T03:02:27.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.931+0000 7f7c5525b740 1 -- 172.21.15.33:0/2135964669 wait complete. 2024-03-19T03:02:27.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.931+0000 7f7c5525b740 1 -- start start 2024-03-19T03:02:27.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.932+0000 7f7c5525b740 1 Processor -- start 2024-03-19T03:02:27.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.932+0000 7f7c5525b740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561e8fbbc2d0 0x561e8fbc9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.932+0000 7f7c5525b740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561e8fbbcbf0 0x561e8fbc97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.932+0000 7f7c5525b740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561e8fbc57d0 0x561e8fbc9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.932+0000 7f7c5525b740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c1ad290 con 0x561e8fbbc2d0 2024-03-19T03:02:27.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.932+0000 7f7c5525b740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c1ae700 con 0x561e8fbbcbf0 2024-03-19T03:02:27.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.932+0000 7f7c5525b740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c1ad960 con 0x561e8fbc57d0 2024-03-19T03:02:27.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.932+0000 7f7c5525b740 10 client.0 _ll_register_callbacks cb 0x7f7c44003eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:02:27.939 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.932+0000 7f7c4bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561e8fbbc2d0 0x561e8fbc9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.932+0000 7f7c4bfff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561e8fbbc2d0 0x561e8fbc9290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38646/0 (socket says 172.21.15.33:38646) 2024-03-19T03:02:27.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.932+0000 7f7c4bfff640 1 -- 172.21.15.33:0/3447342527 learned_addr learned my addr 172.21.15.33:0/3447342527 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:27.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.932+0000 7f7c4b7fe640 1 --2- 172.21.15.33:0/3447342527 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561e8fbbcbf0 0x561e8fbc97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.932+0000 7f7c50c12640 1 --2- 172.21.15.33:0/3447342527 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561e8fbc57d0 0x561e8fbc9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.932+0000 7f7c4bfff640 1 -- 172.21.15.33:0/3447342527 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561e8fbc57d0 msgr2=0x561e8fbc9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:27.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.932+0000 7f7c4bfff640 1 --2- 172.21.15.33:0/3447342527 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561e8fbc57d0 0x561e8fbc9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.932+0000 7f7c4bfff640 1 -- 172.21.15.33:0/3447342527 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561e8fbbcbf0 msgr2=0x561e8fbc97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:27.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.932+0000 7f7c4bfff640 1 --2- 172.21.15.33:0/3447342527 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561e8fbbcbf0 0x561e8fbc97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.932+0000 7f7c4bfff640 1 -- 172.21.15.33:0/3447342527 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e8fc3ae70 con 0x561e8fbbc2d0 2024-03-19T03:02:27.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.932+0000 7f7c50c12640 1 --2- 172.21.15.33:0/3447342527 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561e8fbc57d0 0x561e8fbc9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:02:27.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.932+0000 7f7c4bfff640 1 --2- 172.21.15.33:0/3447342527 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561e8fbbc2d0 0x561e8fbc9290 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c000c00 tx=0x7f7c3c009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86fe7bbc3ee34e5b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:27.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.932+0000 7f7c2b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:02:27.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.932+0000 7f7c2b7fe640 1 -- 172.21.15.33:0/3447342527 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c033060 con 0x561e8fbbc2d0 2024-03-19T03:02:27.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.932+0000 7f7c2b7fe640 1 -- 172.21.15.33:0/3447342527 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f7c3c01f040 con 0x561e8fbbc2d0 2024-03-19T03:02:27.940 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.932+0000 7f7c5525b740 1 -- 172.21.15.33:0/3447342527 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561e8fb2c9f0 con 0x561e8fbbc2d0 2024-03-19T03:02:27.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.933+0000 7f7c2b7fe640 1 -- 172.21.15.33:0/3447342527 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7c3c03a050 con 0x561e8fbbc2d0 2024-03-19T03:02:27.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.933+0000 7f7c5525b740 10 client.17598 fetch_fsmap learned FSMap version 25 2024-03-19T03:02:27.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.933+0000 7f7c5525b740 1 -- 172.21.15.33:0/3447342527 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x561e8fc3c150 con 0x561e8fbbc2d0 2024-03-19T03:02:27.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.933+0000 7f7c2b7fe640 1 -- 172.21.15.33:0/3447342527 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x561e8fb2c9f0 con 0x561e8fbbc2d0 2024-03-19T03:02:27.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.933+0000 7f7c2b7fe640 1 -- 172.21.15.33:0/3447342527 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f7c3c018070 con 0x561e8fbbc2d0 2024-03-19T03:02:27.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.933+0000 7f7c5525b740 10 client.17598 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:02:27.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.933+0000 7f7c5525b740 10 client.17598 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:02:27.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.933+0000 7f7c5525b740 20 client.17598 populate_metadata read hostname 'smithi033' 2024-03-19T03:02:27.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.933+0000 7f7c5525b740 1 --2- 172.21.15.33:0/3447342527 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x561e8fc3c6e0 0x561e8fc3eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:27.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.933+0000 7f7c5525b740 4 client.17598 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T03:02:27.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.933+0000 7f7c5525b740 1 -- 172.21.15.33:0/3447342527 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561e8fb26d40 con 0x561e8fc3c6e0 2024-03-19T03:02:27.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.933+0000 7f7c4b7fe640 1 --2- 172.21.15.33:0/3447342527 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x561e8fc3c6e0 0x561e8fc3eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:27.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.934+0000 7f7c2b7fe640 10 client.17598 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:02:27.941 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.934+0000 7f7c4b7fe640 1 --2- 172.21.15.33:0/3447342527 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x561e8fc3c6e0 0x561e8fc3eb00 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49e05ae2295d3db4 server_cookie=44122e3ffaed6aec in_seq=0 out_seq=0 2024-03-19T03:02:27.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.934+0000 7f7c2b7fe640 1 -- 172.21.15.33:0/3447342527 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561e8fb26d40 con 0x561e8fc3c6e0 2024-03-19T03:02:27.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.934+0000 7f7c2b7fe640 10 client.17598 handle_command_reply: tid=0 2024-03-19T03:02:27.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.934+0000 7f7c5525b740 1 client.17598 shutdown 2024-03-19T03:02:27.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.934+0000 7f7c5525b740 1 -- 172.21.15.33:0/3447342527 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561e8fbbc2d0 msgr2=0x561e8fbc9290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.934+0000 7f7c5525b740 1 --2- 172.21.15.33:0/3447342527 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561e8fbbc2d0 0x561e8fbc9290 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c000c00 tx=0x7f7c3c009f60 comp rx=0 tx=0).stop 2024-03-19T03:02:27.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.934+0000 7f7c5525b740 1 -- 172.21.15.33:0/3447342527 shutdown_connections 2024-03-19T03:02:27.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.934+0000 7f7c5525b740 1 --2- 172.21.15.33:0/3447342527 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x561e8fc3c6e0 0x561e8fc3eb00 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.934+0000 7f7c5525b740 1 --2- 172.21.15.33:0/3447342527 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x561e8fbc57d0 0x561e8fbc9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.934+0000 7f7c5525b740 1 --2- 172.21.15.33:0/3447342527 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561e8fbbcbf0 0x561e8fbc97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.934+0000 7f7c5525b740 1 --2- 172.21.15.33:0/3447342527 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x561e8fbbc2d0 0x561e8fbc9290 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.934+0000 7f7c2b7fe640 0 client.17598 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:02:27.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.934+0000 7f7c5525b740 1 -- 172.21.15.33:0/3447342527 >> 172.21.15.33:0/3447342527 conn(0x561e8fbb6a00 msgr2=0x561e8fbb5ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:27.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.934+0000 7f7c5525b740 1 -- 172.21.15.33:0/3447342527 shutdown_connections 2024-03-19T03:02:27.942 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.934+0000 7f7c5525b740 1 -- 172.21.15.33:0/3447342527 wait complete. 2024-03-19T03:02:27.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.934+0000 7f7c5525b740 20 client.17598 trim_cache size 0 max 16384 2024-03-19T03:02:27.943 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T03:02:27.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.934+0000 7f7c52675640 1 -- 172.21.15.33:0/1377144729 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f7c380a2970 msgr2=0x7f7c380a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.934+0000 7f7c52675640 1 --2- 172.21.15.33:0/1377144729 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f7c380a2970 0x7f7c380a4e30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x561e8fbbbda0 tx=0x7f7c34009920 comp rx=0 tx=0).stop 2024-03-19T03:02:27.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.934+0000 7f7c52675640 1 -- 172.21.15.33:0/1377144729 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c4c1077c0 msgr2=0x7f7c4c19c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:27.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.934+0000 7f7c52675640 1 --2- 172.21.15.33:0/1377144729 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c4c1077c0 0x7f7c4c19c840 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f7c44004af0 tx=0x7f7c44035ce0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.935+0000 7f7c52675640 1 -- 172.21.15.33:0/1377144729 shutdown_connections 2024-03-19T03:02:27.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.935+0000 7f7c52675640 1 --2- 172.21.15.33:0/1377144729 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f7c380a2970 0x7f7c380a4e30 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.935+0000 7f7c52675640 1 --2- 172.21.15.33:0/1377144729 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f7c4c1077c0 0x7f7c4c19c840 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.935+0000 7f7c52675640 1 --2- 172.21.15.33:0/1377144729 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7c4c102170 0x7f7c4c1984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.935+0000 7f7c52675640 1 --2- 172.21.15.33:0/1377144729 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7c4c1017b0 0x7f7c4c197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:27.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.935+0000 7f7c52675640 1 -- 172.21.15.33:0/1377144729 >> 172.21.15.33:0/1377144729 conn(0x7f7c4c0fd2e0 msgr2=0x7f7c4c0fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:27.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.935+0000 7f7c52675640 1 -- 172.21.15.33:0/1377144729 shutdown_connections 2024-03-19T03:02:27.944 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:27.935+0000 7f7c52675640 1 -- 172.21.15.33:0/1377144729 wait complete. 2024-03-19T03:02:28.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:28 smithi155 ceph-mon[35804]: pgmap v279: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-19T03:02:28.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:28 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1126190424' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:02:28.517 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:28 smithi096 ceph-mon[35946]: pgmap v279: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-19T03:02:28.517 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:28 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1126190424' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:02:28.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:28 smithi033 ceph-mon[31973]: pgmap v279: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-19T03:02:28.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:28 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1126190424' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:02:28.947 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T03:02:28.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:02:28.948 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T03:02:29.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.066+0000 7fbcad3ab640 1 -- 172.21.15.33:0/3429201381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbca8102170 msgr2=0x7fbca810e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:29.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.066+0000 7fbcad3ab640 1 --2- 172.21.15.33:0/3429201381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbca8102170 0x7fbca810e680 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fbc98009cd0 tx=0x7fbc9802c0f0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.066+0000 7fbcad3ab640 1 -- 172.21.15.33:0/3429201381 shutdown_connections 2024-03-19T03:02:29.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.066+0000 7fbcad3ab640 1 --2- 172.21.15.33:0/3429201381 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbca81017b0 0x7fbca8101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.066+0000 7fbcad3ab640 1 --2- 172.21.15.33:0/3429201381 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbca8102170 0x7fbca810e680 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.066+0000 7fbcad3ab640 1 --2- 172.21.15.33:0/3429201381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbca81077c0 0x7fbca8107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.066+0000 7fbcad3ab640 1 -- 172.21.15.33:0/3429201381 >> 172.21.15.33:0/3429201381 conn(0x7fbca8078c20 msgr2=0x7fbca8079030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:29.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.066+0000 7fbcad3ab640 1 -- 172.21.15.33:0/3429201381 shutdown_connections 2024-03-19T03:02:29.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.067+0000 7fbcad3ab640 1 -- 172.21.15.33:0/3429201381 wait complete. 2024-03-19T03:02:29.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.067+0000 7fbcad3ab640 1 Processor -- start 2024-03-19T03:02:29.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.067+0000 7fbcad3ab640 1 -- start start 2024-03-19T03:02:29.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.067+0000 7fbcad3ab640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbca81017b0 0x7fbca8197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.067+0000 7fbcad3ab640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbca8102170 0x7fbca8198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.067+0000 7fbcad3ab640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbca81077c0 0x7fbca819c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.067+0000 7fbcad3ab640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbca8111a80 con 0x7fbca81017b0 2024-03-19T03:02:29.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.068+0000 7fbcad3ab640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbca8111900 con 0x7fbca8102170 2024-03-19T03:02:29.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.068+0000 7fbcad3ab640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbca8111c00 con 0x7fbca81077c0 2024-03-19T03:02:29.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.068+0000 7fbca6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbca81017b0 0x7fbca8197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.068+0000 7fbca6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbca81017b0 0x7fbca8197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38658/0 (socket says 172.21.15.33:38658) 2024-03-19T03:02:29.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.068+0000 7fbca6ffd640 1 -- 172.21.15.33:0/2828913104 learned_addr learned my addr 172.21.15.33:0/2828913104 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:29.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.068+0000 7fbca67fc640 1 --2- 172.21.15.33:0/2828913104 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbca8102170 0x7fbca8198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.068+0000 7fbca77fe640 1 --2- 172.21.15.33:0/2828913104 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbca81077c0 0x7fbca819c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.068+0000 7fbca6ffd640 1 -- 172.21.15.33:0/2828913104 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbca81077c0 msgr2=0x7fbca819c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:29.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.068+0000 7fbca6ffd640 1 --2- 172.21.15.33:0/2828913104 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbca81077c0 0x7fbca819c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.068+0000 7fbca6ffd640 1 -- 172.21.15.33:0/2828913104 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbca8102170 msgr2=0x7fbca8198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:29.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.068+0000 7fbca6ffd640 1 --2- 172.21.15.33:0/2828913104 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbca8102170 0x7fbca8198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.068+0000 7fbca6ffd640 1 -- 172.21.15.33:0/2828913104 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbca819cfb0 con 0x7fbca81017b0 2024-03-19T03:02:29.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.068+0000 7fbca67fc640 1 --2- 172.21.15.33:0/2828913104 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbca8102170 0x7fbca8198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:02:29.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.068+0000 7fbca6ffd640 1 --2- 172.21.15.33:0/2828913104 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbca81017b0 0x7fbca8197fc0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fbca000e9f0 tx=0x7fbca000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:29.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.068+0000 7fbc8ffff640 1 -- 172.21.15.33:0/2828913104 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca000cd80 con 0x7fbca81017b0 2024-03-19T03:02:29.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.068+0000 7fbc8ffff640 1 -- 172.21.15.33:0/2828913104 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbca00108e0 con 0x7fbca81017b0 2024-03-19T03:02:29.084 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.068+0000 7fbcad3ab640 1 -- 172.21.15.33:0/2828913104 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbca819d2a0 con 0x7fbca81017b0 2024-03-19T03:02:29.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.068+0000 7fbc8ffff640 1 -- 172.21.15.33:0/2828913104 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca0018920 con 0x7fbca81017b0 2024-03-19T03:02:29.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.068+0000 7fbcad3ab640 1 -- 172.21.15.33:0/2828913104 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbca81a4b30 con 0x7fbca81017b0 2024-03-19T03:02:29.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.071+0000 7fbc8ffff640 1 -- 172.21.15.33:0/2828913104 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fbca0010a90 con 0x7fbca81017b0 2024-03-19T03:02:29.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.071+0000 7fbcaff8a740 1 Processor -- start 2024-03-19T03:02:29.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.071+0000 7fbcaff8a740 1 -- start start 2024-03-19T03:02:29.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.072+0000 7fbcaff8a740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55587bd77380 0x55587bd77760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.072+0000 7fbc8ffff640 1 --2- 172.21.15.33:0/2828913104 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbc900a2970 0x7fbc900a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.072+0000 7fbcaff8a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55587bd77ca0 0x55587bd80340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.072+0000 7fbcaff8a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55587bd80880 0x55587bd82c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.072+0000 7fbcaff8a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55587bcdb160 con 0x55587bd80880 2024-03-19T03:02:29.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.072+0000 7fbcaff8a740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55587bc83e10 con 0x55587bd77380 2024-03-19T03:02:29.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.072+0000 7fbcaff8a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55587bcd7200 con 0x55587bd77ca0 2024-03-19T03:02:29.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.072+0000 7fbca6ffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55587bd77380 0x55587bd77760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.072+0000 7fbca77fe640 1 --2- 172.21.15.33:0/2828913104 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbc900a2970 0x7fbc900a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.072+0000 7fbca67fc640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55587bd77ca0 0x55587bd80340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.072+0000 7fbca6ffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55587bd77380 0x55587bd77760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:40944/0 (socket says 172.21.15.33:40944) 2024-03-19T03:02:29.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.072+0000 7fbca6ffd640 1 -- 172.21.15.33:0/949962373 learned_addr learned my addr 172.21.15.33:0/949962373 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:29.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.072+0000 7fbca77fe640 1 --2- 172.21.15.33:0/949962373 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55587bd80880 0x55587bd82c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.072+0000 7fbc8ffff640 1 -- 172.21.15.33:0/2828913104 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7fbca00cf4e0 con 0x7fbca81017b0 2024-03-19T03:02:29.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.072+0000 7fbca6ffd640 1 -- 172.21.15.33:0/949962373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55587bd77ca0 msgr2=0x55587bd80340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:29.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.072+0000 7fbca6ffd640 1 --2- 172.21.15.33:0/949962373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55587bd77ca0 0x55587bd80340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.072+0000 7fbca6ffd640 1 -- 172.21.15.33:0/949962373 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55587bd80880 msgr2=0x55587bd82c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:29.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.072+0000 7fbca6ffd640 1 --2- 172.21.15.33:0/949962373 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55587bd80880 0x55587bd82c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.086 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.072+0000 7fbca6ffd640 1 -- 172.21.15.33:0/949962373 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55587bab77e0 con 0x55587bd77380 2024-03-19T03:02:29.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.072+0000 7fbca67fc640 1 --2- 172.21.15.33:0/949962373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55587bd77ca0 0x55587bd80340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:02:29.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.072+0000 7fbca77fe640 1 --2- 172.21.15.33:0/2828913104 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbc900a2970 0x7fbc900a4e30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbc98005cc0 tx=0x7fbc980098e0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:29.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.072+0000 7fbca6ffd640 1 --2- 172.21.15.33:0/949962373 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55587bd77380 0x55587bd77760 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fbca009a6e0 tx=0x7fbca001de60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5caf7fb01913a031 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:29.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.073+0000 7fbc8d7fa640 1 -- 172.21.15.33:0/949962373 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca0099fd0 con 0x55587bd77380 2024-03-19T03:02:29.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.073+0000 7fbc8d7fa640 1 -- 172.21.15.33:0/949962373 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbca0010de0 con 0x55587bd77380 2024-03-19T03:02:29.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.073+0000 7fbc8d7fa640 1 -- 172.21.15.33:0/949962373 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca0099de0 con 0x55587bd77380 2024-03-19T03:02:29.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.073+0000 7fbcaff8a740 1 -- 172.21.15.33:0/949962373 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55587bd77380 msgr2=0x55587bd77760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:29.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.073+0000 7fbcaff8a740 1 --2- 172.21.15.33:0/949962373 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55587bd77380 0x55587bd77760 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fbca009a6e0 tx=0x7fbca001de60 comp rx=0 tx=0).stop 2024-03-19T03:02:29.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.073+0000 7fbcaff8a740 1 -- 172.21.15.33:0/949962373 shutdown_connections 2024-03-19T03:02:29.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.073+0000 7fbcaff8a740 1 --2- 172.21.15.33:0/949962373 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55587bd77ca0 0x55587bd80340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.073+0000 7fbcaff8a740 1 --2- 172.21.15.33:0/949962373 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55587bd80880 0x55587bd82c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.073+0000 7fbcaff8a740 1 --2- 172.21.15.33:0/949962373 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55587bd77380 0x55587bd77760 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.073+0000 7fbcaff8a740 1 -- 172.21.15.33:0/949962373 >> 172.21.15.33:0/949962373 conn(0x55587bd71ab0 msgr2=0x55587bd71e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:29.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.073+0000 7fbcaff8a740 1 -- 172.21.15.33:0/949962373 shutdown_connections 2024-03-19T03:02:29.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.073+0000 7fbcaff8a740 1 -- 172.21.15.33:0/949962373 wait complete. 2024-03-19T03:02:29.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.074+0000 7fbcaff8a740 1 -- start start 2024-03-19T03:02:29.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.074+0000 7fbcaff8a740 1 Processor -- start 2024-03-19T03:02:29.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.074+0000 7fbcaff8a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55587bd77380 0x55587bd7d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.074+0000 7fbcaff8a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55587bd77ca0 0x55587bd7dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.074+0000 7fbcaff8a740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55587bd80880 0x55587bd7e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.074+0000 7fbcaff8a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55587bd83610 con 0x55587bd77380 2024-03-19T03:02:29.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.074+0000 7fbcaff8a740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55587bc971a0 con 0x55587bd80880 2024-03-19T03:02:29.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.074+0000 7fbcaff8a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55587bd83790 con 0x55587bd77ca0 2024-03-19T03:02:29.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.074+0000 7fbcaff8a740 10 client.0 _ll_register_callbacks cb 0x55587b9eec60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:02:29.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.074+0000 7fbca6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55587bd77380 0x55587bd7d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.074+0000 7fbca6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55587bd77380 0x55587bd7d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38678/0 (socket says 172.21.15.33:38678) 2024-03-19T03:02:29.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.074+0000 7fbca6ffd640 1 -- 172.21.15.33:0/246646389 learned_addr learned my addr 172.21.15.33:0/246646389 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:29.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.074+0000 7fbca77fe640 1 --2- 172.21.15.33:0/246646389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55587bd80880 0x55587bd7e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.074+0000 7fbca67fc640 1 --2- 172.21.15.33:0/246646389 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55587bd77ca0 0x55587bd7dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.074+0000 7fbca6ffd640 1 -- 172.21.15.33:0/246646389 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55587bd77ca0 msgr2=0x55587bd7dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:29.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.074+0000 7fbca6ffd640 1 --2- 172.21.15.33:0/246646389 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55587bd77ca0 0x55587bd7dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.074+0000 7fbca6ffd640 1 -- 172.21.15.33:0/246646389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55587bd80880 msgr2=0x55587bd7e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:29.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.074+0000 7fbca6ffd640 1 --2- 172.21.15.33:0/246646389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55587bd80880 0x55587bd7e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.074+0000 7fbca6ffd640 1 -- 172.21.15.33:0/246646389 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55587bb17f70 con 0x55587bd77380 2024-03-19T03:02:29.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.075+0000 7fbca6ffd640 1 --2- 172.21.15.33:0/246646389 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55587bd77380 0x55587bd7d820 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fbca000c6a0 tx=0x7fbca0086290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd441f5a11c03596 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:29.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.075+0000 7fbc8aff5640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:02:29.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.075+0000 7fbc8aff5640 1 -- 172.21.15.33:0/246646389 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca009b820 con 0x55587bd77380 2024-03-19T03:02:29.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.075+0000 7fbc8aff5640 1 -- 172.21.15.33:0/246646389 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7fbca0099de0 con 0x55587bd77380 2024-03-19T03:02:29.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.075+0000 7fbcaff8a740 1 -- 172.21.15.33:0/246646389 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55587bce86a0 con 0x55587bd77380 2024-03-19T03:02:29.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.075+0000 7fbc8aff5640 1 -- 172.21.15.33:0/246646389 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fbca009f8c0 con 0x55587bd77380 2024-03-19T03:02:29.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.075+0000 7fbcaff8a740 10 client.17610 fetch_fsmap learned FSMap version 25 2024-03-19T03:02:29.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.075+0000 7fbcaff8a740 1 -- 172.21.15.33:0/246646389 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55587b9af170 con 0x55587bd77380 2024-03-19T03:02:29.089 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.075+0000 7fbc8aff5640 1 -- 172.21.15.33:0/246646389 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55587bce86a0 con 0x55587bd77380 2024-03-19T03:02:29.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.075+0000 7fbc8aff5640 1 -- 172.21.15.33:0/246646389 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fbca00a3a80 con 0x55587bd77380 2024-03-19T03:02:29.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.075+0000 7fbcaff8a740 10 client.17610 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:02:29.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.075+0000 7fbcaff8a740 10 client.17610 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:02:29.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.075+0000 7fbcaff8a740 20 client.17610 populate_metadata read hostname 'smithi033' 2024-03-19T03:02:29.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.075+0000 7fbcaff8a740 1 --2- 172.21.15.33:0/246646389 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55587bd909c0 0x55587bd92db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.075+0000 7fbcaff8a740 4 client.17610 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:02:29.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.075+0000 7fbcaff8a740 1 -- 172.21.15.33:0/246646389 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55587bce29f0 con 0x55587bd909c0 2024-03-19T03:02:29.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.075+0000 7fbca67fc640 1 --2- 172.21.15.33:0/246646389 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55587bd909c0 0x55587bd92db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.076+0000 7fbca67fc640 1 --2- 172.21.15.33:0/246646389 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55587bd909c0 0x55587bd92db0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72ca6ff445d0382b server_cookie=2f1ade8c5da6edbb in_seq=0 out_seq=0 2024-03-19T03:02:29.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.076+0000 7fbc8aff5640 10 client.17610 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:02:29.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.076+0000 7fbc8aff5640 1 -- 172.21.15.33:0/246646389 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55587bce29f0 con 0x55587bd909c0 2024-03-19T03:02:29.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.076+0000 7fbc8aff5640 10 client.17610 handle_command_reply: tid=0 2024-03-19T03:02:29.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.076+0000 7fbcaff8a740 1 client.17610 shutdown 2024-03-19T03:02:29.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.077+0000 7fbcaff8a740 1 -- 172.21.15.33:0/246646389 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55587bd77380 msgr2=0x55587bd7d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:29.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.077+0000 7fbcaff8a740 1 --2- 172.21.15.33:0/246646389 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55587bd77380 0x55587bd7d820 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fbca000c6a0 tx=0x7fbca0086290 comp rx=0 tx=0).stop 2024-03-19T03:02:29.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.077+0000 7fbcaff8a740 1 -- 172.21.15.33:0/246646389 shutdown_connections 2024-03-19T03:02:29.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.077+0000 7fbcaff8a740 1 --2- 172.21.15.33:0/246646389 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55587bd909c0 0x55587bd92db0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.077+0000 7fbcaff8a740 1 --2- 172.21.15.33:0/246646389 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55587bd77ca0 0x55587bd7dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.077+0000 7fbcaff8a740 1 --2- 172.21.15.33:0/246646389 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55587bd80880 0x55587bd7e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.077+0000 7fbc8aff5640 0 client.17610 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:02:29.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.077+0000 7fbcaff8a740 1 --2- 172.21.15.33:0/246646389 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55587bd77380 0x55587bd7d820 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.077+0000 7fbcaff8a740 1 -- 172.21.15.33:0/246646389 >> 172.21.15.33:0/246646389 conn(0x55587bd71ab0 msgr2=0x55587bd73800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:29.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.077+0000 7fbcaff8a740 1 -- 172.21.15.33:0/246646389 shutdown_connections 2024-03-19T03:02:29.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.077+0000 7fbcaff8a740 1 -- 172.21.15.33:0/246646389 wait complete. 2024-03-19T03:02:29.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.077+0000 7fbcaff8a740 20 client.17610 trim_cache size 0 max 16384 2024-03-19T03:02:29.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.094+0000 7fbcaff8a740 1 Processor -- start 2024-03-19T03:02:29.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.094+0000 7fbcaff8a740 1 -- start start 2024-03-19T03:02:29.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.095+0000 7fbcaff8a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55587bd71ab0 0x55587bd8c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.095+0000 7fbcaff8a740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55587bd77380 0x55587bd983d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.095+0000 7fbcaff8a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55587bd77ca0 0x55587bd88a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.095+0000 7fbcaff8a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbca81b1560 con 0x55587bd71ab0 2024-03-19T03:02:29.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.095+0000 7fbcaff8a740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbca81aee30 con 0x55587bd77380 2024-03-19T03:02:29.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.095+0000 7fbcaff8a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbca81acca0 con 0x55587bd77ca0 2024-03-19T03:02:29.107 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.095+0000 7fbca6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55587bd71ab0 0x55587bd8c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.095+0000 7fbca6ffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55587bd71ab0 0x55587bd8c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38690/0 (socket says 172.21.15.33:38690) 2024-03-19T03:02:29.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.095+0000 7fbca6ffd640 1 -- 172.21.15.33:0/3922327172 learned_addr learned my addr 172.21.15.33:0/3922327172 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:29.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.095+0000 7fbca67fc640 1 --2- 172.21.15.33:0/3922327172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55587bd77380 0x55587bd983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.095+0000 7fbca77fe640 1 --2- 172.21.15.33:0/3922327172 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55587bd77ca0 0x55587bd88a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.095+0000 7fbca6ffd640 1 -- 172.21.15.33:0/3922327172 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55587bd77ca0 msgr2=0x55587bd88a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:29.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.095+0000 7fbca6ffd640 1 --2- 172.21.15.33:0/3922327172 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55587bd77ca0 0x55587bd88a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.095+0000 7fbca6ffd640 1 -- 172.21.15.33:0/3922327172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55587bd77380 msgr2=0x55587bd983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:29.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.095+0000 7fbca6ffd640 1 --2- 172.21.15.33:0/3922327172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55587bd77380 0x55587bd983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.095+0000 7fbca6ffd640 1 -- 172.21.15.33:0/3922327172 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55587bd718e0 con 0x55587bd71ab0 2024-03-19T03:02:29.108 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.095+0000 7fbca6ffd640 1 --2- 172.21.15.33:0/3922327172 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55587bd71ab0 0x55587bd8c3e0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fbca000e9c0 tx=0x7fbca0086300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab8075f64f81b837 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:29.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.095+0000 7fbc8cff9640 1 -- 172.21.15.33:0/3922327172 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca00b9070 con 0x55587bd71ab0 2024-03-19T03:02:29.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.095+0000 7fbc8cff9640 1 -- 172.21.15.33:0/3922327172 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbca00860b0 con 0x55587bd71ab0 2024-03-19T03:02:29.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.095+0000 7fbc8cff9640 1 -- 172.21.15.33:0/3922327172 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca0099de0 con 0x55587bd71ab0 2024-03-19T03:02:29.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.096+0000 7fbcaff8a740 1 -- 172.21.15.33:0/3922327172 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55587bd71ab0 msgr2=0x55587bd8c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:29.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.096+0000 7fbcaff8a740 1 --2- 172.21.15.33:0/3922327172 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55587bd71ab0 0x55587bd8c3e0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fbca000e9c0 tx=0x7fbca0086300 comp rx=0 tx=0).stop 2024-03-19T03:02:29.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.096+0000 7fbcaff8a740 1 -- 172.21.15.33:0/3922327172 shutdown_connections 2024-03-19T03:02:29.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.096+0000 7fbcaff8a740 1 --2- 172.21.15.33:0/3922327172 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55587bd77ca0 0x55587bd88a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.096+0000 7fbcaff8a740 1 --2- 172.21.15.33:0/3922327172 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55587bd77380 0x55587bd983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.096+0000 7fbcaff8a740 1 --2- 172.21.15.33:0/3922327172 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55587bd71ab0 0x55587bd8c3e0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.096+0000 7fbcaff8a740 1 -- 172.21.15.33:0/3922327172 >> 172.21.15.33:0/3922327172 conn(0x55587bd7c460 msgr2=0x55587bd702a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:29.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.096+0000 7fbcaff8a740 1 -- 172.21.15.33:0/3922327172 shutdown_connections 2024-03-19T03:02:29.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.096+0000 7fbcaff8a740 1 -- 172.21.15.33:0/3922327172 wait complete. 2024-03-19T03:02:29.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.096+0000 7fbcaff8a740 1 -- start start 2024-03-19T03:02:29.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.096+0000 7fbcaff8a740 1 Processor -- start 2024-03-19T03:02:29.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.097+0000 7fbcaff8a740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55587bd71ab0 0x55587bd85420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.097+0000 7fbcaff8a740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55587bd77380 0x55587bd85960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.097+0000 7fbcaff8a740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55587bd77ca0 0x55587bd85ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.097+0000 7fbcaff8a740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbca81ad2f0 con 0x55587bd77380 2024-03-19T03:02:29.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.097+0000 7fbcaff8a740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbca81ae760 con 0x55587bd71ab0 2024-03-19T03:02:29.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.097+0000 7fbcaff8a740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbca81ad9c0 con 0x55587bd77ca0 2024-03-19T03:02:29.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.097+0000 7fbcaff8a740 10 client.0 _ll_register_callbacks cb 0x7fbca0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:02:29.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.097+0000 7fbca67fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55587bd77380 0x55587bd85960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.097+0000 7fbca6ffd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55587bd71ab0 0x55587bd85420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.097+0000 7fbca67fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55587bd77380 0x55587bd85960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38700/0 (socket says 172.21.15.33:38700) 2024-03-19T03:02:29.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.097+0000 7fbca67fc640 1 -- 172.21.15.33:0/3287737822 learned_addr learned my addr 172.21.15.33:0/3287737822 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:29.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.097+0000 7fbca77fe640 1 --2- 172.21.15.33:0/3287737822 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55587bd77ca0 0x55587bd85ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.097+0000 7fbca6ffd640 1 -- 172.21.15.33:0/3287737822 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55587bd77ca0 msgr2=0x55587bd85ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:29.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.097+0000 7fbca6ffd640 1 --2- 172.21.15.33:0/3287737822 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55587bd77ca0 0x55587bd85ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.097+0000 7fbca6ffd640 1 -- 172.21.15.33:0/3287737822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55587bd77380 msgr2=0x55587bd85960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:29.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.097+0000 7fbca6ffd640 1 --2- 172.21.15.33:0/3287737822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55587bd77380 0x55587bd85960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.097+0000 7fbca6ffd640 1 -- 172.21.15.33:0/3287737822 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55587bdfaf20 con 0x55587bd71ab0 2024-03-19T03:02:29.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.097+0000 7fbca67fc640 1 --2- 172.21.15.33:0/3287737822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55587bd77380 0x55587bd85960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:02:29.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.097+0000 7fbca6ffd640 1 --2- 172.21.15.33:0/3287737822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55587bd71ab0 0x55587bd85420 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbca000e9c0 tx=0x7fbca009a010 comp rx=0 tx=0).ready entity=mon.2 client_cookie=223c5846469dfca6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:29.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.097+0000 7fbc8aff5640 10 client.0 ms_handle_connect on v2:172.21.15.96:3300/0 2024-03-19T03:02:29.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.097+0000 7fbc8aff5640 1 -- 172.21.15.33:0/3287737822 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca00b9070 con 0x55587bd71ab0 2024-03-19T03:02:29.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.097+0000 7fbc8aff5640 1 -- 172.21.15.33:0/3287737822 <== mon.2 v2:172.21.15.96:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7fbca0010630 con 0x55587bd71ab0 2024-03-19T03:02:29.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.097+0000 7fbcaff8a740 1 -- 172.21.15.33:0/3287737822 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbca00860b0 con 0x55587bd71ab0 2024-03-19T03:02:29.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.098+0000 7fbc8aff5640 1 -- 172.21.15.33:0/3287737822 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fbca009a510 con 0x55587bd71ab0 2024-03-19T03:02:29.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.098+0000 7fbcaff8a740 10 client.27437 fetch_fsmap learned FSMap version 25 2024-03-19T03:02:29.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.098+0000 7fbcaff8a740 1 -- 172.21.15.33:0/3287737822 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55587bdfaf20 con 0x55587bd71ab0 2024-03-19T03:02:29.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.098+0000 7fbc8aff5640 1 -- 172.21.15.33:0/3287737822 <== mon.2 v2:172.21.15.96:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbca00860b0 con 0x55587bd71ab0 2024-03-19T03:02:29.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.098+0000 7fbc8aff5640 1 -- 172.21.15.33:0/3287737822 <== mon.2 v2:172.21.15.96:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fbca009ac70 con 0x55587bd71ab0 2024-03-19T03:02:29.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.098+0000 7fbcaff8a740 10 client.27437 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:02:29.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.098+0000 7fbcaff8a740 10 client.27437 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:02:29.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.098+0000 7fbcaff8a740 20 client.27437 populate_metadata read hostname 'smithi033' 2024-03-19T03:02:29.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.098+0000 7fbcaff8a740 1 --2- 172.21.15.33:0/3287737822 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55587bd80880 0x55587bdfe4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.098+0000 7fbcaff8a740 4 client.27437 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T03:02:29.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.098+0000 7fbcaff8a740 1 -- 172.21.15.33:0/3287737822 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55587bc60be0 con 0x55587bd80880 2024-03-19T03:02:29.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.098+0000 7fbca67fc640 1 --2- 172.21.15.33:0/3287737822 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55587bd80880 0x55587bdfe4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.098+0000 7fbc8aff5640 10 client.27437 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:02:29.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.098+0000 7fbca67fc640 1 --2- 172.21.15.33:0/3287737822 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55587bd80880 0x55587bdfe4c0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4f8c08c3a72a700 server_cookie=460bf6dfdecb098d in_seq=0 out_seq=0 2024-03-19T03:02:29.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.099+0000 7fbc8aff5640 1 -- 172.21.15.33:0/3287737822 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55587bc60be0 con 0x55587bd80880 2024-03-19T03:02:29.112 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.099+0000 7fbc8aff5640 10 client.27437 handle_command_reply: tid=0 2024-03-19T03:02:29.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.099+0000 7fbcaff8a740 1 client.27437 shutdown 2024-03-19T03:02:29.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.100+0000 7fbcaff8a740 1 -- 172.21.15.33:0/3287737822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55587bd71ab0 msgr2=0x55587bd85420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:29.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.100+0000 7fbcaff8a740 1 --2- 172.21.15.33:0/3287737822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55587bd71ab0 0x55587bd85420 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbca000e9c0 tx=0x7fbca009a010 comp rx=0 tx=0).stop 2024-03-19T03:02:29.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.100+0000 7fbcaff8a740 1 -- 172.21.15.33:0/3287737822 shutdown_connections 2024-03-19T03:02:29.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.100+0000 7fbcaff8a740 1 --2- 172.21.15.33:0/3287737822 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55587bd80880 0x55587bdfe4c0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.100+0000 7fbcaff8a740 1 --2- 172.21.15.33:0/3287737822 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55587bd77ca0 0x55587bd85ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.100+0000 7fbcaff8a740 1 --2- 172.21.15.33:0/3287737822 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55587bd77380 0x55587bd85960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.100+0000 7fbcaff8a740 1 --2- 172.21.15.33:0/3287737822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55587bd71ab0 0x55587bd85420 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.100+0000 7fbcaff8a740 1 -- 172.21.15.33:0/3287737822 >> 172.21.15.33:0/3287737822 conn(0x55587bd7c460 msgr2=0x55587bd70b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:29.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.100+0000 7fbc8aff5640 0 client.27437 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:02:29.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.100+0000 7fbcaff8a740 1 -- 172.21.15.33:0/3287737822 shutdown_connections 2024-03-19T03:02:29.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.100+0000 7fbcaff8a740 1 -- 172.21.15.33:0/3287737822 wait complete. 2024-03-19T03:02:29.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.100+0000 7fbcaff8a740 20 client.27437 trim_cache size 0 max 16384 2024-03-19T03:02:29.114 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T03:02:29.114 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-03-19T03:02:29.114 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "9b4ac6b4-8a4d-4eb6-9e68-8ed735b35e08", 2024-03-19T03:02:29.114 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-03-19T03:02:29.114 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T03:02:29.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.100+0000 7fbcad3ab640 1 -- 172.21.15.33:0/2828913104 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbc900a2970 msgr2=0x7fbc900a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:29.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.100+0000 7fbcad3ab640 1 --2- 172.21.15.33:0/2828913104 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbc900a2970 0x7fbc900a4e30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbc98005cc0 tx=0x7fbc980098e0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.100+0000 7fbcad3ab640 1 -- 172.21.15.33:0/2828913104 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbca81017b0 msgr2=0x7fbca8197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:29.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.100+0000 7fbcad3ab640 1 --2- 172.21.15.33:0/2828913104 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbca81017b0 0x7fbca8197fc0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fbca000e9f0 tx=0x7fbca000eeb0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.100+0000 7fbcad3ab640 1 -- 172.21.15.33:0/2828913104 shutdown_connections 2024-03-19T03:02:29.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.100+0000 7fbcad3ab640 1 --2- 172.21.15.33:0/2828913104 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbc900a2970 0x7fbc900a4e30 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.100+0000 7fbcad3ab640 1 --2- 172.21.15.33:0/2828913104 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbca81077c0 0x7fbca819c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.100+0000 7fbcad3ab640 1 --2- 172.21.15.33:0/2828913104 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbca8102170 0x7fbca8198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.100+0000 7fbcad3ab640 1 --2- 172.21.15.33:0/2828913104 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbca81017b0 0x7fbca8197fc0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.100+0000 7fbcad3ab640 1 -- 172.21.15.33:0/2828913104 >> 172.21.15.33:0/2828913104 conn(0x7fbca8078c20 msgr2=0x7fbca81041c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:29.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.101+0000 7fbcad3ab640 1 -- 172.21.15.33:0/2828913104 shutdown_connections 2024-03-19T03:02:29.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.101+0000 7fbcad3ab640 1 -- 172.21.15.33:0/2828913104 wait complete. 2024-03-19T03:02:29.117 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T03:02:29.117 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T03:02:29.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.220+0000 7f3754d97640 1 -- 172.21.15.33:0/770611086 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3750100f00 msgr2=0x7f375010dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:29.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.220+0000 7f3754d97640 1 --2- 172.21.15.33:0/770611086 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3750100f00 0x7f375010dc50 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f3738009900 tx=0x7f373802bc30 comp rx=0 tx=0).stop 2024-03-19T03:02:29.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.220+0000 7f3754d97640 1 -- 172.21.15.33:0/770611086 shutdown_connections 2024-03-19T03:02:29.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.220+0000 7f3754d97640 1 --2- 172.21.15.33:0/770611086 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f375010e190 0x7f3750110580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.220+0000 7f3754d97640 1 --2- 172.21.15.33:0/770611086 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3750100f00 0x7f375010dc50 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.220+0000 7f3754d97640 1 --2- 172.21.15.33:0/770611086 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f37501005e0 0x7f37501009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.224 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.220+0000 7f3754d97640 1 -- 172.21.15.33:0/770611086 >> 172.21.15.33:0/770611086 conn(0x7f3750076aa0 msgr2=0x7f3750076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:29.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.220+0000 7f3754d97640 1 -- 172.21.15.33:0/770611086 shutdown_connections 2024-03-19T03:02:29.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.220+0000 7f3754d97640 1 -- 172.21.15.33:0/770611086 wait complete. 2024-03-19T03:02:29.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.221+0000 7f3754d97640 1 Processor -- start 2024-03-19T03:02:29.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.221+0000 7f3754d97640 1 -- start start 2024-03-19T03:02:29.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.221+0000 7f3754d97640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f37501005e0 0x7f375019a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.221+0000 7f3754d97640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3750100f00 0x7f375019a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.221+0000 7f3754d97640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f375010e190 0x7f375019eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.221+0000 7f3754d97640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3750113ca0 con 0x7f37501005e0 2024-03-19T03:02:29.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.221+0000 7f3754d97640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3750113b20 con 0x7f375010e190 2024-03-19T03:02:29.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.221+0000 7f3754d97640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3750113e20 con 0x7f3750100f00 2024-03-19T03:02:29.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.221+0000 7f374e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f37501005e0 0x7f375019a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.221+0000 7f374e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f37501005e0 0x7f375019a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38730/0 (socket says 172.21.15.33:38730) 2024-03-19T03:02:29.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.221+0000 7f374e59c640 1 -- 172.21.15.33:0/790362553 learned_addr learned my addr 172.21.15.33:0/790362553 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:29.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.221+0000 7f374ed9d640 1 --2- 172.21.15.33:0/790362553 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f375010e190 0x7f375019eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.225 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.221+0000 7f3745d9b640 1 --2- 172.21.15.33:0/790362553 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3750100f00 0x7f375019a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.222+0000 7f374e59c640 1 -- 172.21.15.33:0/790362553 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3750100f00 msgr2=0x7f375019a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:29.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.222+0000 7f374e59c640 1 --2- 172.21.15.33:0/790362553 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3750100f00 0x7f375019a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.222+0000 7f374e59c640 1 -- 172.21.15.33:0/790362553 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f375010e190 msgr2=0x7f375019eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:29.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.222+0000 7f374e59c640 1 --2- 172.21.15.33:0/790362553 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f375010e190 0x7f375019eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.222+0000 7f374e59c640 1 -- 172.21.15.33:0/790362553 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f375019f210 con 0x7f37501005e0 2024-03-19T03:02:29.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.222+0000 7f374e59c640 1 --2- 172.21.15.33:0/790362553 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f37501005e0 0x7f375019a220 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f374000e9f0 tx=0x7f374000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:29.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.222+0000 7f3747fff640 1 -- 172.21.15.33:0/790362553 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f374000cd80 con 0x7f37501005e0 2024-03-19T03:02:29.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.222+0000 7f3747fff640 1 -- 172.21.15.33:0/790362553 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f37400054d0 con 0x7f37501005e0 2024-03-19T03:02:29.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.222+0000 7f3754d97640 1 -- 172.21.15.33:0/790362553 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f375019f500 con 0x7f37501005e0 2024-03-19T03:02:29.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.222+0000 7f3747fff640 1 -- 172.21.15.33:0/790362553 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3740010490 con 0x7f37501005e0 2024-03-19T03:02:29.226 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.222+0000 7f3754d97640 1 -- 172.21.15.33:0/790362553 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37501a6d90 con 0x7f37501005e0 2024-03-19T03:02:29.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.225+0000 7f3757987740 1 Processor -- start 2024-03-19T03:02:29.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.225+0000 7f3757987740 1 -- start start 2024-03-19T03:02:29.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.225+0000 7f3747fff640 1 -- 172.21.15.33:0/790362553 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f3740005020 con 0x7f37501005e0 2024-03-19T03:02:29.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.225+0000 7f3757987740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625c09d5a30 0x5625c09d5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.225+0000 7f3757987740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625c09d6350 0x5625c09de9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.225+0000 7f3757987740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5625c09def30 0x5625c09e1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.234 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.225+0000 7f3757987740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5625c0939500 con 0x5625c09d5a30 2024-03-19T03:02:29.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.225+0000 7f3757987740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5625c08e1be0 con 0x5625c09def30 2024-03-19T03:02:29.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.225+0000 7f3757987740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5625c09355a0 con 0x5625c09d6350 2024-03-19T03:02:29.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.225+0000 7f374ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625c09d5a30 0x5625c09d5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.225+0000 7f374ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625c09d5a30 0x5625c09d5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38744/0 (socket says 172.21.15.33:38744) 2024-03-19T03:02:29.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.225+0000 7f374ed9d640 1 -- 172.21.15.33:0/3514291370 learned_addr learned my addr 172.21.15.33:0/3514291370 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:29.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.225+0000 7f374e59c640 1 --2- 172.21.15.33:0/3514291370 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625c09d6350 0x5625c09de9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.225+0000 7f3745d9b640 1 --2- 172.21.15.33:0/3514291370 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5625c09def30 0x5625c09e1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.226+0000 7f3747fff640 1 --2- 172.21.15.33:0/790362553 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f373c0a2970 0x7f373c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.226+0000 7f374ed9d640 1 -- 172.21.15.33:0/3514291370 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625c09d6350 msgr2=0x5625c09de9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:29.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.226+0000 7f374ed9d640 1 --2- 172.21.15.33:0/3514291370 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625c09d6350 0x5625c09de9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.226+0000 7f374ed9d640 1 -- 172.21.15.33:0/3514291370 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5625c09def30 msgr2=0x5625c09e1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:29.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.226+0000 7f374ed9d640 1 --2- 172.21.15.33:0/3514291370 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5625c09def30 0x5625c09e1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.226+0000 7f374ed9d640 1 -- 172.21.15.33:0/3514291370 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625c0903000 con 0x5625c09d5a30 2024-03-19T03:02:29.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.226+0000 7f3745d9b640 1 --2- 172.21.15.33:0/3514291370 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5625c09def30 0x5625c09e1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:02:29.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.226+0000 7f3747fff640 1 -- 172.21.15.33:0/790362553 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f37400cf5a0 con 0x7f37501005e0 2024-03-19T03:02:29.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.226+0000 7f374ed9d640 1 --2- 172.21.15.33:0/790362553 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f373c0a2970 0x7f373c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.226+0000 7f374ed9d640 1 --2- 172.21.15.33:0/3514291370 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625c09d5a30 0x5625c09d5e10 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f3748009f30 tx=0x7f37480118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d50e32bd5a150e4f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:29.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.226+0000 7f3744d99640 1 -- 172.21.15.33:0/3514291370 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3748003b70 con 0x5625c09d5a30 2024-03-19T03:02:29.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.226+0000 7f3744d99640 1 -- 172.21.15.33:0/3514291370 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3748004530 con 0x5625c09d5a30 2024-03-19T03:02:29.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.226+0000 7f3744d99640 1 -- 172.21.15.33:0/3514291370 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f374800fde0 con 0x5625c09d5a30 2024-03-19T03:02:29.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.227+0000 7f3757987740 1 -- 172.21.15.33:0/3514291370 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625c09d5a30 msgr2=0x5625c09d5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:29.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.227+0000 7f374ed9d640 1 --2- 172.21.15.33:0/790362553 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f373c0a2970 0x7f373c0a4e30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f374801d000 tx=0x7f3748019540 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:29.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.227+0000 7f3757987740 1 --2- 172.21.15.33:0/3514291370 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625c09d5a30 0x5625c09d5e10 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f3748009f30 tx=0x7f37480118d0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.227+0000 7f3757987740 1 -- 172.21.15.33:0/3514291370 shutdown_connections 2024-03-19T03:02:29.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.227+0000 7f3757987740 1 --2- 172.21.15.33:0/3514291370 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625c09d6350 0x5625c09de9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.227+0000 7f3757987740 1 --2- 172.21.15.33:0/3514291370 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5625c09def30 0x5625c09e1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.227+0000 7f3757987740 1 --2- 172.21.15.33:0/3514291370 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625c09d5a30 0x5625c09d5e10 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.227+0000 7f3757987740 1 -- 172.21.15.33:0/3514291370 >> 172.21.15.33:0/3514291370 conn(0x5625c09d0160 msgr2=0x5625c09d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:29.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.227+0000 7f3757987740 1 -- 172.21.15.33:0/3514291370 shutdown_connections 2024-03-19T03:02:29.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.227+0000 7f3757987740 1 -- 172.21.15.33:0/3514291370 wait complete. 2024-03-19T03:02:29.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.227+0000 7f3757987740 1 -- start start 2024-03-19T03:02:29.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.227+0000 7f3757987740 1 Processor -- start 2024-03-19T03:02:29.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.228+0000 7f3757987740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625c09d5a30 0x5625c09dbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.228+0000 7f3757987740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5625c09d6350 0x5625c09dc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.228+0000 7f3757987740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625c09def30 0x5625c09dc830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.228+0000 7f3757987740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5625c09e1d20 con 0x5625c09d5a30 2024-03-19T03:02:29.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.228+0000 7f3757987740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5625c08f5160 con 0x5625c09d6350 2024-03-19T03:02:29.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.228+0000 7f3757987740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5625c09e1ea0 con 0x5625c09def30 2024-03-19T03:02:29.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.228+0000 7f374e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625c09d5a30 0x5625c09dbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.228+0000 7f3757987740 10 client.0 _ll_register_callbacks cb 0x5625c0931a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:02:29.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.228+0000 7f374e59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625c09d5a30 0x5625c09dbdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38752/0 (socket says 172.21.15.33:38752) 2024-03-19T03:02:29.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.228+0000 7f374e59c640 1 -- 172.21.15.33:0/724990953 learned_addr learned my addr 172.21.15.33:0/724990953 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:29.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.228+0000 7f374ed9d640 1 --2- 172.21.15.33:0/724990953 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625c09def30 0x5625c09dc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.228+0000 7f3745d9b640 1 --2- 172.21.15.33:0/724990953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5625c09d6350 0x5625c09dc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.228+0000 7f374e59c640 1 -- 172.21.15.33:0/724990953 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625c09def30 msgr2=0x5625c09dc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:29.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.228+0000 7f374e59c640 1 --2- 172.21.15.33:0/724990953 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625c09def30 0x5625c09dc830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.228+0000 7f374e59c640 1 -- 172.21.15.33:0/724990953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5625c09d6350 msgr2=0x5625c09dc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:29.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.228+0000 7f374e59c640 1 --2- 172.21.15.33:0/724990953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5625c09d6350 0x5625c09dc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.228+0000 7f374e59c640 1 -- 172.21.15.33:0/724990953 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625c07177d0 con 0x5625c09d5a30 2024-03-19T03:02:29.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.228+0000 7f3745d9b640 1 --2- 172.21.15.33:0/724990953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5625c09d6350 0x5625c09dc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:02:29.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.228+0000 7f374e59c640 1 --2- 172.21.15.33:0/724990953 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625c09d5a30 0x5625c09dbdb0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f374000e9c0 tx=0x7f3740004b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=825785b3f6ba1346 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:29.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.228+0000 7f3735ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:02:29.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.228+0000 7f3735ffb640 1 -- 172.21.15.33:0/724990953 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f374009b190 con 0x5625c09d5a30 2024-03-19T03:02:29.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.228+0000 7f3735ffb640 1 -- 172.21.15.33:0/724990953 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f3740086070 con 0x5625c09d5a30 2024-03-19T03:02:29.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.228+0000 7f3757987740 1 -- 172.21.15.33:0/724990953 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3748004530 con 0x5625c09d5a30 2024-03-19T03:02:29.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.229+0000 7f3735ffb640 1 -- 172.21.15.33:0/724990953 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3740005310 con 0x5625c09d5a30 2024-03-19T03:02:29.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.229+0000 7f3757987740 10 client.17628 fetch_fsmap learned FSMap version 25 2024-03-19T03:02:29.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.229+0000 7f3757987740 1 -- 172.21.15.33:0/724990953 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5625c064d3f0 con 0x5625c09d5a30 2024-03-19T03:02:29.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.229+0000 7f3735ffb640 1 -- 172.21.15.33:0/724990953 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3748004530 con 0x5625c09d5a30 2024-03-19T03:02:29.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.229+0000 7f3735ffb640 1 -- 172.21.15.33:0/724990953 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f374009ba90 con 0x5625c09d5a30 2024-03-19T03:02:29.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.229+0000 7f3757987740 10 client.17628 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:02:29.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.229+0000 7f3757987740 10 client.17628 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:02:29.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.229+0000 7f3757987740 20 client.17628 populate_metadata read hostname 'smithi033' 2024-03-19T03:02:29.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.229+0000 7f3757987740 1 --2- 172.21.15.33:0/724990953 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5625c09eeff0 0x5625c09f1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.229+0000 7f3757987740 4 client.17628 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:02:29.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.229+0000 7f3757987740 1 -- 172.21.15.33:0/724990953 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625c0946a40 con 0x5625c09eeff0 2024-03-19T03:02:29.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.229+0000 7f3745d9b640 1 --2- 172.21.15.33:0/724990953 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5625c09eeff0 0x5625c09f1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.230+0000 7f3735ffb640 10 client.17628 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:02:29.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.230+0000 7f3745d9b640 1 --2- 172.21.15.33:0/724990953 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5625c09eeff0 0x5625c09f1460 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e2bbb63198875dd server_cookie=ea23c3f97ba314ed in_seq=0 out_seq=0 2024-03-19T03:02:29.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.230+0000 7f3735ffb640 1 -- 172.21.15.33:0/724990953 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5625c0946a40 con 0x5625c09eeff0 2024-03-19T03:02:29.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.230+0000 7f3735ffb640 10 client.17628 handle_command_reply: tid=0 2024-03-19T03:02:29.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.230+0000 7f3757987740 1 client.17628 shutdown 2024-03-19T03:02:29.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.231+0000 7f3757987740 1 -- 172.21.15.33:0/724990953 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625c09d5a30 msgr2=0x5625c09dbdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:29.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.231+0000 7f3757987740 1 --2- 172.21.15.33:0/724990953 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625c09d5a30 0x5625c09dbdb0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f374000e9c0 tx=0x7f3740004b10 comp rx=0 tx=0).stop 2024-03-19T03:02:29.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.231+0000 7f3757987740 1 -- 172.21.15.33:0/724990953 shutdown_connections 2024-03-19T03:02:29.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.231+0000 7f3757987740 1 --2- 172.21.15.33:0/724990953 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5625c09eeff0 0x5625c09f1460 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.231+0000 7f3757987740 1 --2- 172.21.15.33:0/724990953 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625c09def30 0x5625c09dc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.231+0000 7f3757987740 1 --2- 172.21.15.33:0/724990953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5625c09d6350 0x5625c09dc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.231+0000 7f3757987740 1 --2- 172.21.15.33:0/724990953 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625c09d5a30 0x5625c09dbdb0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.231+0000 7f3757987740 1 -- 172.21.15.33:0/724990953 >> 172.21.15.33:0/724990953 conn(0x5625c09d0160 msgr2=0x5625c09d1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:29.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.231+0000 7f3735ffb640 0 client.17628 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:02:29.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.231+0000 7f3757987740 1 -- 172.21.15.33:0/724990953 shutdown_connections 2024-03-19T03:02:29.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.231+0000 7f3757987740 1 -- 172.21.15.33:0/724990953 wait complete. 2024-03-19T03:02:29.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.231+0000 7f3757987740 20 client.17628 trim_cache size 0 max 16384 2024-03-19T03:02:29.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.248+0000 7f3757987740 1 Processor -- start 2024-03-19T03:02:29.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.248+0000 7f3757987740 1 -- start start 2024-03-19T03:02:29.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.248+0000 7f3757987740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5625c09d5a30 0x5625c09de4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.248+0000 7f3757987740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625c09d6350 0x5625c09eaac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.248+0000 7f3757987740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625c09def30 0x5625c09f6d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.248+0000 7f3757987740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f37501b3770 con 0x5625c09def30 2024-03-19T03:02:29.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.248+0000 7f3757987740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f37501b1040 con 0x5625c09d5a30 2024-03-19T03:02:29.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.248+0000 7f3757987740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f37501aeeb0 con 0x5625c09d6350 2024-03-19T03:02:29.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.249+0000 7f374ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625c09def30 0x5625c09f6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.249+0000 7f3745d9b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625c09d6350 0x5625c09eaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.249+0000 7f374ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625c09def30 0x5625c09f6d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38762/0 (socket says 172.21.15.33:38762) 2024-03-19T03:02:29.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.249+0000 7f374e59c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5625c09d5a30 0x5625c09de4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.249+0000 7f374ed9d640 1 -- 172.21.15.33:0/3692507946 learned_addr learned my addr 172.21.15.33:0/3692507946 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:29.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.249+0000 7f374e59c640 1 -- 172.21.15.33:0/3692507946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625c09d6350 msgr2=0x5625c09eaac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:29.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.249+0000 7f374e59c640 1 --2- 172.21.15.33:0/3692507946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625c09d6350 0x5625c09eaac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.249+0000 7f374e59c640 1 -- 172.21.15.33:0/3692507946 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625c09def30 msgr2=0x5625c09f6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:29.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.249+0000 7f374e59c640 1 --2- 172.21.15.33:0/3692507946 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625c09def30 0x5625c09f6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.249+0000 7f374e59c640 1 -- 172.21.15.33:0/3692507946 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625c060d0f0 con 0x5625c09d5a30 2024-03-19T03:02:29.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.249+0000 7f374ed9d640 1 --2- 172.21.15.33:0/3692507946 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625c09def30 0x5625c09f6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:02:29.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.249+0000 7f374e59c640 1 --2- 172.21.15.33:0/3692507946 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5625c09d5a30 0x5625c09de4e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f37400a3aa0 tx=0x7f37400974d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad15da7a372053ff server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:29.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.249+0000 7f3737fff640 1 -- 172.21.15.33:0/3692507946 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f374009b190 con 0x5625c09d5a30 2024-03-19T03:02:29.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.249+0000 7f3737fff640 1 -- 172.21.15.33:0/3692507946 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3740005680 con 0x5625c09d5a30 2024-03-19T03:02:29.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.249+0000 7f3737fff640 1 -- 172.21.15.33:0/3692507946 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f374009bf30 con 0x5625c09d5a30 2024-03-19T03:02:29.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.250+0000 7f3757987740 1 -- 172.21.15.33:0/3692507946 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5625c09d5a30 msgr2=0x5625c09de4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:29.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.250+0000 7f3757987740 1 --2- 172.21.15.33:0/3692507946 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5625c09d5a30 0x5625c09de4e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f37400a3aa0 tx=0x7f37400974d0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.250+0000 7f3757987740 1 -- 172.21.15.33:0/3692507946 shutdown_connections 2024-03-19T03:02:29.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.250+0000 7f3757987740 1 --2- 172.21.15.33:0/3692507946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625c09d6350 0x5625c09eaac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.250+0000 7f3757987740 1 --2- 172.21.15.33:0/3692507946 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625c09def30 0x5625c09f6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.250+0000 7f3757987740 1 --2- 172.21.15.33:0/3692507946 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5625c09d5a30 0x5625c09de4e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.250+0000 7f3757987740 1 -- 172.21.15.33:0/3692507946 >> 172.21.15.33:0/3692507946 conn(0x5625c09d0160 msgr2=0x5625c09d1510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:29.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.250+0000 7f3757987740 1 -- 172.21.15.33:0/3692507946 shutdown_connections 2024-03-19T03:02:29.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.250+0000 7f3757987740 1 -- 172.21.15.33:0/3692507946 wait complete. 2024-03-19T03:02:29.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.250+0000 7f3757987740 1 -- start start 2024-03-19T03:02:29.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.250+0000 7f3757987740 1 Processor -- start 2024-03-19T03:02:29.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.251+0000 7f3757987740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5625c09d5a30 0x5625c09e73b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.251+0000 7f3757987740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625c09d6350 0x5625c09d86e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.251+0000 7f3757987740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625c09def30 0x5625c09d8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.251+0000 7f3757987740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f37501af500 con 0x5625c09d6350 2024-03-19T03:02:29.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.251+0000 7f3757987740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f37501b0970 con 0x5625c09d5a30 2024-03-19T03:02:29.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.251+0000 7f3757987740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f37501afbd0 con 0x5625c09def30 2024-03-19T03:02:29.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.251+0000 7f3757987740 10 client.0 _ll_register_callbacks cb 0x7f37400d5240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:02:29.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.251+0000 7f3745d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625c09d6350 0x5625c09d86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.251+0000 7f374e59c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5625c09d5a30 0x5625c09e73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.251+0000 7f3745d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625c09d6350 0x5625c09d86e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:38774/0 (socket says 172.21.15.33:38774) 2024-03-19T03:02:29.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.251+0000 7f3745d9b640 1 -- 172.21.15.33:0/4248226009 learned_addr learned my addr 172.21.15.33:0/4248226009 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:29.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.251+0000 7f374ed9d640 1 --2- 172.21.15.33:0/4248226009 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625c09def30 0x5625c09d8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.251+0000 7f3745d9b640 1 -- 172.21.15.33:0/4248226009 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625c09def30 msgr2=0x5625c09d8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:29.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.251+0000 7f3745d9b640 1 --2- 172.21.15.33:0/4248226009 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625c09def30 0x5625c09d8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.251+0000 7f3745d9b640 1 -- 172.21.15.33:0/4248226009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5625c09d5a30 msgr2=0x5625c09e73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:29.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.251+0000 7f3745d9b640 1 --2- 172.21.15.33:0/4248226009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5625c09d5a30 0x5625c09e73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.251+0000 7f3745d9b640 1 -- 172.21.15.33:0/4248226009 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625c09d9470 con 0x5625c09d6350 2024-03-19T03:02:29.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.251+0000 7f374e59c640 1 --2- 172.21.15.33:0/4248226009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5625c09d5a30 0x5625c09e73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:02:29.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.251+0000 7f3745d9b640 1 --2- 172.21.15.33:0/4248226009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625c09d6350 0x5625c09d86e0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f3738009900 tx=0x7f3738005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca5bc345737662a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:29.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.251+0000 7f3735ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:02:29.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.251+0000 7f3735ffb640 1 -- 172.21.15.33:0/4248226009 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f373802c9f0 con 0x5625c09d6350 2024-03-19T03:02:29.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.251+0000 7f3735ffb640 1 -- 172.21.15.33:0/4248226009 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f373802cb90 con 0x5625c09d6350 2024-03-19T03:02:29.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.251+0000 7f3757987740 1 -- 172.21.15.33:0/4248226009 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3740005680 con 0x5625c09d6350 2024-03-19T03:02:29.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.251+0000 7f3735ffb640 1 -- 172.21.15.33:0/4248226009 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3738056050 con 0x5625c09d6350 2024-03-19T03:02:29.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.251+0000 7f3757987740 10 client.17634 fetch_fsmap learned FSMap version 25 2024-03-19T03:02:29.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.251+0000 7f3757987740 1 -- 172.21.15.33:0/4248226009 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5625c0a55450 con 0x5625c09d6350 2024-03-19T03:02:29.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.252+0000 7f3735ffb640 1 -- 172.21.15.33:0/4248226009 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3740005680 con 0x5625c09d6350 2024-03-19T03:02:29.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.252+0000 7f3735ffb640 1 -- 172.21.15.33:0/4248226009 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f3738004cd0 con 0x5625c09d6350 2024-03-19T03:02:29.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.252+0000 7f3757987740 10 client.17634 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:02:29.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.252+0000 7f3757987740 10 client.17634 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:02:29.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.252+0000 7f3757987740 20 client.17634 populate_metadata read hostname 'smithi033' 2024-03-19T03:02:29.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.252+0000 7f3757987740 1 --2- 172.21.15.33:0/4248226009 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5625c0a55c50 0x5625c0a58070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:29.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.252+0000 7f3757987740 4 client.17634 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T03:02:29.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.252+0000 7f3757987740 1 -- 172.21.15.33:0/4248226009 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5625c0940d90 con 0x5625c0a55c50 2024-03-19T03:02:29.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.252+0000 7f374e59c640 1 --2- 172.21.15.33:0/4248226009 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5625c0a55c50 0x5625c0a58070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:29.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.252+0000 7f374e59c640 1 --2- 172.21.15.33:0/4248226009 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5625c0a55c50 0x5625c0a58070 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1fc0b12cc8964c9 server_cookie=ec34456f86a7e527 in_seq=0 out_seq=0 2024-03-19T03:02:29.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.252+0000 7f3735ffb640 10 client.17634 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:02:29.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.252+0000 7f3735ffb640 1 -- 172.21.15.33:0/4248226009 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5625c0940d90 con 0x5625c0a55c50 2024-03-19T03:02:29.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.252+0000 7f3735ffb640 10 client.17634 handle_command_reply: tid=0 2024-03-19T03:02:29.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.253+0000 7f3757987740 1 client.17634 shutdown 2024-03-19T03:02:29.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.253+0000 7f3757987740 1 -- 172.21.15.33:0/4248226009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625c09d6350 msgr2=0x5625c09d86e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:29.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.253+0000 7f3757987740 1 --2- 172.21.15.33:0/4248226009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625c09d6350 0x5625c09d86e0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f3738009900 tx=0x7f3738005c00 comp rx=0 tx=0).stop 2024-03-19T03:02:29.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.253+0000 7f3757987740 1 -- 172.21.15.33:0/4248226009 shutdown_connections 2024-03-19T03:02:29.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.253+0000 7f3757987740 1 --2- 172.21.15.33:0/4248226009 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5625c0a55c50 0x5625c0a58070 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.253+0000 7f3757987740 1 --2- 172.21.15.33:0/4248226009 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625c09def30 0x5625c09d8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.253+0000 7f3757987740 1 --2- 172.21.15.33:0/4248226009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5625c09d6350 0x5625c09d86e0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.253+0000 7f3757987740 1 --2- 172.21.15.33:0/4248226009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5625c09d5a30 0x5625c09e73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.253+0000 7f3735ffb640 0 client.17634 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:02:29.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.253+0000 7f3757987740 1 -- 172.21.15.33:0/4248226009 >> 172.21.15.33:0/4248226009 conn(0x5625c09d0160 msgr2=0x5625c09dc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:29.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.253+0000 7f3757987740 1 -- 172.21.15.33:0/4248226009 shutdown_connections 2024-03-19T03:02:29.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.253+0000 7f3757987740 1 -- 172.21.15.33:0/4248226009 wait complete. 2024-03-19T03:02:29.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.253+0000 7f3757987740 20 client.17634 trim_cache size 0 max 16384 2024-03-19T03:02:29.263 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T03:02:29.263 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T03:02:29.264 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-03-19T03:02:29.264 INFO:teuthology.orchestra.run.smithi033.stdout: "9b4ac6b4-8a4d-4eb6-9e68-8ed735b35e08": { 2024-03-19T03:02:29.264 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-03-19T03:02:29.264 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "9b4ac6b4-8a4d-4eb6-9e68-8ed735b35e08", 2024-03-19T03:02:29.264 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-03-19T03:02:29.264 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:02:29.264 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:02:29.264 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T03:02:29.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.253+0000 7f3754d97640 1 -- 172.21.15.33:0/790362553 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f373c0a2970 msgr2=0x7f373c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:29.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.253+0000 7f3754d97640 1 --2- 172.21.15.33:0/790362553 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f373c0a2970 0x7f373c0a4e30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f374801d000 tx=0x7f3748019540 comp rx=0 tx=0).stop 2024-03-19T03:02:29.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.253+0000 7f3754d97640 1 -- 172.21.15.33:0/790362553 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f37501005e0 msgr2=0x7f375019a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:29.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.253+0000 7f3754d97640 1 --2- 172.21.15.33:0/790362553 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f37501005e0 0x7f375019a220 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f374000e9f0 tx=0x7f374000edf0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.265 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.253+0000 7f3754d97640 1 -- 172.21.15.33:0/790362553 shutdown_connections 2024-03-19T03:02:29.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.253+0000 7f3754d97640 1 --2- 172.21.15.33:0/790362553 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f373c0a2970 0x7f373c0a4e30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.253+0000 7f3754d97640 1 --2- 172.21.15.33:0/790362553 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3750100f00 0x7f375019a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.253+0000 7f3754d97640 1 --2- 172.21.15.33:0/790362553 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f375010e190 0x7f375019eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.253+0000 7f3754d97640 1 --2- 172.21.15.33:0/790362553 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f37501005e0 0x7f375019a220 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:29.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.253+0000 7f3754d97640 1 -- 172.21.15.33:0/790362553 >> 172.21.15.33:0/790362553 conn(0x7f3750076aa0 msgr2=0x7f375010f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:29.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.253+0000 7f3754d97640 1 -- 172.21.15.33:0/790362553 shutdown_connections 2024-03-19T03:02:29.266 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:29.253+0000 7f3754d97640 1 -- 172.21.15.33:0/790362553 wait complete. 2024-03-19T03:02:29.266 INFO:tasks.cephfs.filesystem:scrub status for tag:9b4ac6b4-8a4d-4eb6-9e68-8ed735b35e08 - {'path': '/', 'tag': '9b4ac6b4-8a4d-4eb6-9e68-8ed735b35e08', 'options': 'recursive,force'} 2024-03-19T03:02:30.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:30 smithi155 ceph-mon[35804]: scrub queued for path: / 2024-03-19T03:02:30.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:30 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T03:02:30.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:30 smithi155 ceph-mon[35804]: scrub summary: active paths [/] 2024-03-19T03:02:30.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:30 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T03:02:30.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:30 smithi155 ceph-mon[35804]: pgmap v280: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:02:30.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:30 smithi033 ceph-mon[31973]: scrub queued for path: / 2024-03-19T03:02:30.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:30 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T03:02:30.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:30 smithi033 ceph-mon[31973]: scrub summary: active paths [/] 2024-03-19T03:02:30.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:30 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T03:02:30.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:30 smithi033 ceph-mon[31973]: pgmap v280: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:02:30.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:30 smithi096 ceph-mon[35946]: scrub queued for path: / 2024-03-19T03:02:30.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:30 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T03:02:30.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:30 smithi096 ceph-mon[35946]: scrub summary: active paths [/] 2024-03-19T03:02:30.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:30 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T03:02:30.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:30 smithi096 ceph-mon[35946]: pgmap v280: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:02:32.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:32 smithi155 ceph-mon[35804]: pgmap v281: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:02:32.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:32 smithi033 ceph-mon[31973]: pgmap v281: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:02:32.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:32 smithi096 ceph-mon[35946]: pgmap v281: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:02:33.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:33 smithi155 ceph-mon[35804]: scrub summary: idle 2024-03-19T03:02:33.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:33 smithi033 ceph-mon[31973]: scrub summary: idle 2024-03-19T03:02:33.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:33 smithi096 ceph-mon[35946]: scrub summary: idle 2024-03-19T03:02:34.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:34 smithi155 ceph-mon[35804]: pgmap v282: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:02:34.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:34 smithi033 ceph-mon[31973]: pgmap v282: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:02:34.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:34 smithi096 ceph-mon[35946]: pgmap v282: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:02:36.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:36 smithi155 ceph-mon[35804]: pgmap v283: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:02:36.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:36 smithi033 ceph-mon[31973]: pgmap v283: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:02:36.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:36 smithi096 ceph-mon[35946]: pgmap v283: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:02:38.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:38 smithi155 ceph-mon[35804]: pgmap v284: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:02:38.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:38 smithi033 ceph-mon[31973]: pgmap v284: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:02:38.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:38 smithi096 ceph-mon[35946]: pgmap v284: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:02:40.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:40 smithi155 ceph-mon[35804]: pgmap v285: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:02:40.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:40 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:02:40.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:40 smithi033 ceph-mon[31973]: pgmap v285: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:02:40.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:40 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:02:40.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:40 smithi096 ceph-mon[35946]: pgmap v285: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:02:40.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:40 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:02:41.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:41 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:02:41.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:41 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:02:41.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:41 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:02:41.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:41 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:02:41.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:41 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:02:41.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:41 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:02:41.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:41 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:02:41.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:41 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:02:41.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:41 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:02:42.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:42 smithi096 ceph-mon[35946]: pgmap v286: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:02:42.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:42 smithi155 ceph-mon[35804]: pgmap v286: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:02:42.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:42 smithi033 ceph-mon[31973]: pgmap v286: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-19T03:02:44.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:44 smithi096 ceph-mon[35946]: pgmap v287: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:02:44.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:44 smithi155 ceph-mon[35804]: pgmap v287: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:02:44.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:44 smithi033 ceph-mon[31973]: pgmap v287: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-19T03:02:46.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:46 smithi096 ceph-mon[35946]: pgmap v288: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:02:46.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:46 smithi155 ceph-mon[35804]: pgmap v288: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:02:46.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:46 smithi033 ceph-mon[31973]: pgmap v288: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:02:48.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:48 smithi096 ceph-mon[35946]: pgmap v289: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:02:48.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:48 smithi155 ceph-mon[35804]: pgmap v289: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:02:48.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:48 smithi033 ceph-mon[31973]: pgmap v289: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:02:50.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:50 smithi096 ceph-mon[35946]: pgmap v290: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:02:50.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:50 smithi155 ceph-mon[35804]: pgmap v290: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:02:50.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:50 smithi033 ceph-mon[31973]: pgmap v290: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:02:52.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:52 smithi096 ceph-mon[35946]: pgmap v291: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:02:52.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:52 smithi155 ceph-mon[35804]: pgmap v291: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:02:52.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:52 smithi033 ceph-mon[31973]: pgmap v291: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:02:54.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:54 smithi096 ceph-mon[35946]: pgmap v292: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:02:54.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:54 smithi155 ceph-mon[35804]: pgmap v292: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:02:54.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:54 smithi033 ceph-mon[31973]: pgmap v292: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:02:56.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:56 smithi096 ceph-mon[35946]: pgmap v293: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:02:56.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:56 smithi155 ceph-mon[35804]: pgmap v293: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:02:56.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:56 smithi033 ceph-mon[31973]: pgmap v293: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:02:58.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:02:58 smithi155 ceph-mon[35804]: pgmap v294: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:02:58.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:02:58 smithi033 ceph-mon[31973]: pgmap v294: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:02:58.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:02:58 smithi096 ceph-mon[35946]: pgmap v294: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:02:59.267 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T03:02:59.268 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T03:02:59.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.383+0000 7f3b0e8c6640 1 -- 172.21.15.33:0/485319936 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b081017c0 msgr2=0x7f3b08101c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:59.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.383+0000 7f3b0e8c6640 1 --2- 172.21.15.33:0/485319936 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b081017c0 0x7f3b08101c40 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f3af8009930 tx=0x7f3af802bc30 comp rx=0 tx=0).stop 2024-03-19T03:02:59.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.383+0000 7f3b0e8c6640 1 -- 172.21.15.33:0/485319936 shutdown_connections 2024-03-19T03:02:59.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.383+0000 7f3b0e8c6640 1 --2- 172.21.15.33:0/485319936 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3b08102180 0x7f3b0810e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.383+0000 7f3b0e8c6640 1 --2- 172.21.15.33:0/485319936 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b081017c0 0x7f3b08101c40 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.388 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.383+0000 7f3b0e8c6640 1 --2- 172.21.15.33:0/485319936 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b081077d0 0x7f3b08107bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.383+0000 7f3b0e8c6640 1 -- 172.21.15.33:0/485319936 >> 172.21.15.33:0/485319936 conn(0x7f3b080fd250 msgr2=0x7f3b080ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:59.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.383+0000 7f3b0e8c6640 1 -- 172.21.15.33:0/485319936 shutdown_connections 2024-03-19T03:02:59.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.383+0000 7f3b0e8c6640 1 -- 172.21.15.33:0/485319936 wait complete. 2024-03-19T03:02:59.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.384+0000 7f3b0e8c6640 1 Processor -- start 2024-03-19T03:02:59.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.384+0000 7f3b0e8c6640 1 -- start start 2024-03-19T03:02:59.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.384+0000 7f3b0e8c6640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3b081017c0 0x7f3b08073b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.384+0000 7f3b0e8c6640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b08102180 0x7f3b08074060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.384+0000 7f3b0e8c6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b081077d0 0x7f3b080745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.384+0000 7f3b0e8c6640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3b08111a90 con 0x7f3b081077d0 2024-03-19T03:02:59.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.384+0000 7f3b0e8c6640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3b08111910 con 0x7f3b08102180 2024-03-19T03:02:59.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.384+0000 7f3b0e8c6640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3b08111c10 con 0x7f3b081017c0 2024-03-19T03:02:59.389 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.385+0000 7f3b0ce63640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b081077d0 0x7f3b080745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.385+0000 7f3b077fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b08102180 0x7f3b08074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.385+0000 7f3b0ce63640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b081077d0 0x7f3b080745a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56906/0 (socket says 172.21.15.33:56906) 2024-03-19T03:02:59.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.385+0000 7f3b0ce63640 1 -- 172.21.15.33:0/209899912 learned_addr learned my addr 172.21.15.33:0/209899912 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:59.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.385+0000 7f3b07fff640 1 --2- 172.21.15.33:0/209899912 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3b081017c0 0x7f3b08073b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.385+0000 7f3b0ce63640 1 -- 172.21.15.33:0/209899912 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3b081017c0 msgr2=0x7f3b08073b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:59.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.385+0000 7f3b0ce63640 1 --2- 172.21.15.33:0/209899912 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3b081017c0 0x7f3b08073b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.385+0000 7f3b0ce63640 1 -- 172.21.15.33:0/209899912 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b08102180 msgr2=0x7f3b08074060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:59.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.385+0000 7f3b0ce63640 1 --2- 172.21.15.33:0/209899912 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b08102180 0x7f3b08074060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.385+0000 7f3b0ce63640 1 -- 172.21.15.33:0/209899912 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b081a64f0 con 0x7f3b081077d0 2024-03-19T03:02:59.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.385+0000 7f3b077fe640 1 --2- 172.21.15.33:0/209899912 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b08102180 0x7f3b08074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:02:59.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.385+0000 7f3b0ce63640 1 --2- 172.21.15.33:0/209899912 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b081077d0 0x7f3b080745a0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f3b0000eb10 tx=0x7f3b0000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:59.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.385+0000 7f3b057fa640 1 -- 172.21.15.33:0/209899912 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b00004b20 con 0x7f3b081077d0 2024-03-19T03:02:59.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.385+0000 7f3b057fa640 1 -- 172.21.15.33:0/209899912 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3b00004cc0 con 0x7f3b081077d0 2024-03-19T03:02:59.390 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.385+0000 7f3b0e8c6640 1 -- 172.21.15.33:0/209899912 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b081a67e0 con 0x7f3b081077d0 2024-03-19T03:02:59.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.385+0000 7f3b057fa640 1 -- 172.21.15.33:0/209899912 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b00010690 con 0x7f3b081077d0 2024-03-19T03:02:59.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.385+0000 7f3b0e8c6640 1 -- 172.21.15.33:0/209899912 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b081a6d00 con 0x7f3b081077d0 2024-03-19T03:02:59.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.387+0000 7f3b057fa640 1 -- 172.21.15.33:0/209899912 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f3b00005020 con 0x7f3b081077d0 2024-03-19T03:02:59.391 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.388+0000 7f3b1158d740 1 Processor -- start 2024-03-19T03:02:59.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.388+0000 7f3b1158d740 1 -- start start 2024-03-19T03:02:59.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.388+0000 7f3b057fa640 1 --2- 172.21.15.33:0/209899912 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f3af40a2970 0x7f3af40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.388+0000 7f3b1158d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b2b892a30 0x557b2b892e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.392 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.388+0000 7f3b1158d740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b2b893350 0x557b2b89b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.388+0000 7f3b1158d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b2b89bf30 0x557b2b89e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.388+0000 7f3b1158d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557b2b7f6500 con 0x557b2b892a30 2024-03-19T03:02:59.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.388+0000 7f3b1158d740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557b2b79ebe0 con 0x557b2b893350 2024-03-19T03:02:59.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.388+0000 7f3b1158d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x557b2b7f25a0 con 0x557b2b89bf30 2024-03-19T03:02:59.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.388+0000 7f3b057fa640 1 -- 172.21.15.33:0/209899912 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f3b000cf900 con 0x7f3b081077d0 2024-03-19T03:02:59.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.388+0000 7f3b077fe640 1 --2- 172.21.15.33:0/209899912 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f3af40a2970 0x7f3af40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.388+0000 7f3b07fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b2b892a30 0x557b2b892e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.388+0000 7f3b077fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b2b893350 0x557b2b89b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.388+0000 7f3b07fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b2b892a30 0x557b2b892e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56912/0 (socket says 172.21.15.33:56912) 2024-03-19T03:02:59.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.388+0000 7f3b07fff640 1 -- 172.21.15.33:0/3115585181 learned_addr learned my addr 172.21.15.33:0/3115585181 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:59.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.388+0000 7f3b0ce63640 1 --2- 172.21.15.33:0/3115585181 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b2b89bf30 0x557b2b89e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.398 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.389+0000 7f3b077fe640 1 -- 172.21.15.33:0/3115585181 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b2b89bf30 msgr2=0x557b2b89e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:59.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.389+0000 7f3b077fe640 1 --2- 172.21.15.33:0/3115585181 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b2b89bf30 0x557b2b89e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.389+0000 7f3b077fe640 1 -- 172.21.15.33:0/3115585181 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b2b892a30 msgr2=0x557b2b892e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:59.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.389+0000 7f3b077fe640 1 --2- 172.21.15.33:0/3115585181 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b2b892a30 0x557b2b892e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.389+0000 7f3b077fe640 1 -- 172.21.15.33:0/3115585181 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b2b7c0000 con 0x557b2b893350 2024-03-19T03:02:59.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.389+0000 7f3b07fff640 1 --2- 172.21.15.33:0/3115585181 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b2b892a30 0x557b2b892e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:02:59.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.389+0000 7f3b077fe640 1 --2- 172.21.15.33:0/209899912 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f3af40a2970 0x7f3af40a4e30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3af8005ab0 tx=0x7f3af802c680 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:59.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.389+0000 7f3b077fe640 1 --2- 172.21.15.33:0/3115585181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b2b893350 0x557b2b89b9f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3af8009230 tx=0x7f3af8048000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4f80e321efa66dc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:59.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.389+0000 7f3afe7fc640 1 -- 172.21.15.33:0/3115585181 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af80036b0 con 0x557b2b893350 2024-03-19T03:02:59.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.389+0000 7f3afe7fc640 1 -- 172.21.15.33:0/3115585181 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3af8003d00 con 0x557b2b893350 2024-03-19T03:02:59.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.389+0000 7f3afe7fc640 1 -- 172.21.15.33:0/3115585181 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af8034980 con 0x557b2b893350 2024-03-19T03:02:59.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.390+0000 7f3b1158d740 1 -- 172.21.15.33:0/3115585181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b2b893350 msgr2=0x557b2b89b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:59.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.390+0000 7f3b1158d740 1 --2- 172.21.15.33:0/3115585181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b2b893350 0x557b2b89b9f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3af8009230 tx=0x7f3af8048000 comp rx=0 tx=0).stop 2024-03-19T03:02:59.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.390+0000 7f3b1158d740 1 -- 172.21.15.33:0/3115585181 shutdown_connections 2024-03-19T03:02:59.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.390+0000 7f3b1158d740 1 --2- 172.21.15.33:0/3115585181 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b2b89bf30 0x557b2b89e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.390+0000 7f3b1158d740 1 --2- 172.21.15.33:0/3115585181 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b2b893350 0x557b2b89b9f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.390+0000 7f3b1158d740 1 --2- 172.21.15.33:0/3115585181 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b2b892a30 0x557b2b892e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.390+0000 7f3b1158d740 1 -- 172.21.15.33:0/3115585181 >> 172.21.15.33:0/3115585181 conn(0x557b2b88d160 msgr2=0x557b2b88d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:59.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.390+0000 7f3b1158d740 1 -- 172.21.15.33:0/3115585181 shutdown_connections 2024-03-19T03:02:59.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.390+0000 7f3b1158d740 1 -- 172.21.15.33:0/3115585181 wait complete. 2024-03-19T03:02:59.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.390+0000 7f3b1158d740 1 -- start start 2024-03-19T03:02:59.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.390+0000 7f3b1158d740 1 Processor -- start 2024-03-19T03:02:59.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.391+0000 7f3b1158d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b2b892a30 0x557b2b898fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.391+0000 7f3b1158d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b2b893350 0x557b2b8994e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.391+0000 7f3b1158d740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b2b89bf30 0x557b2b899a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.391+0000 7f3b1158d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557b2b89ecc0 con 0x557b2b893350 2024-03-19T03:02:59.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.391+0000 7f3b1158d740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557b2b7b2160 con 0x557b2b89bf30 2024-03-19T03:02:59.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.391+0000 7f3b1158d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x557b2b89ee40 con 0x557b2b892a30 2024-03-19T03:02:59.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.391+0000 7f3b1158d740 10 client.0 _ll_register_callbacks cb 0x557b2b7eea40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:02:59.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.391+0000 7f3b077fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b2b893350 0x557b2b8994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.391+0000 7f3b07fff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b2b892a30 0x557b2b898fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.391+0000 7f3b077fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b2b893350 0x557b2b8994e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56926/0 (socket says 172.21.15.33:56926) 2024-03-19T03:02:59.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.391+0000 7f3b077fe640 1 -- 172.21.15.33:0/2537243517 learned_addr learned my addr 172.21.15.33:0/2537243517 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:59.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.391+0000 7f3b0ce63640 1 --2- 172.21.15.33:0/2537243517 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b2b89bf30 0x557b2b899a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.391+0000 7f3b077fe640 1 -- 172.21.15.33:0/2537243517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b2b892a30 msgr2=0x557b2b898fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:59.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.391+0000 7f3b077fe640 1 --2- 172.21.15.33:0/2537243517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b2b892a30 0x557b2b898fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.391+0000 7f3b077fe640 1 -- 172.21.15.33:0/2537243517 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b2b89bf30 msgr2=0x557b2b899a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:59.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.391+0000 7f3b077fe640 1 --2- 172.21.15.33:0/2537243517 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b2b89bf30 0x557b2b899a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.391+0000 7f3b077fe640 1 -- 172.21.15.33:0/2537243517 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b2b5d47d0 con 0x557b2b893350 2024-03-19T03:02:59.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.391+0000 7f3b077fe640 1 --2- 172.21.15.33:0/2537243517 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b2b893350 0x557b2b8994e0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f3af8004650 tx=0x7f3af8005890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f482ee33adcbc79 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:59.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.391+0000 7f3ae77fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:02:59.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.391+0000 7f3ae77fe640 1 -- 172.21.15.33:0/2537243517 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af8034650 con 0x557b2b893350 2024-03-19T03:02:59.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.391+0000 7f3ae77fe640 1 -- 172.21.15.33:0/2537243517 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f3af80036b0 con 0x557b2b893350 2024-03-19T03:02:59.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.391+0000 7f3b1158d740 1 -- 172.21.15.33:0/2537243517 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3af8003d00 con 0x557b2b893350 2024-03-19T03:02:59.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.392+0000 7f3ae77fe640 1 -- 172.21.15.33:0/2537243517 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3af80038d0 con 0x557b2b893350 2024-03-19T03:02:59.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.392+0000 7f3b1158d740 10 client.17646 fetch_fsmap learned FSMap version 25 2024-03-19T03:02:59.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.392+0000 7f3b1158d740 1 -- 172.21.15.33:0/2537243517 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x557b2b50a3f0 con 0x557b2b893350 2024-03-19T03:02:59.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.392+0000 7f3ae77fe640 1 -- 172.21.15.33:0/2537243517 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3af8003d00 con 0x557b2b893350 2024-03-19T03:02:59.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.392+0000 7f3ae77fe640 1 -- 172.21.15.33:0/2537243517 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f3af80566a0 con 0x557b2b893350 2024-03-19T03:02:59.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.392+0000 7f3b1158d740 10 client.17646 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:02:59.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.392+0000 7f3b1158d740 10 client.17646 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:02:59.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.392+0000 7f3b1158d740 20 client.17646 populate_metadata read hostname 'smithi033' 2024-03-19T03:02:59.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.392+0000 7f3b1158d740 1 --2- 172.21.15.33:0/2537243517 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557b2b8ac180 0x557b2b8ae570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.392+0000 7f3b1158d740 4 client.17646 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:02:59.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.392+0000 7f3b1158d740 1 -- 172.21.15.33:0/2537243517 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b2b803a40 con 0x557b2b8ac180 2024-03-19T03:02:59.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.392+0000 7f3b07fff640 1 --2- 172.21.15.33:0/2537243517 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557b2b8ac180 0x557b2b8ae570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.393+0000 7f3b07fff640 1 --2- 172.21.15.33:0/2537243517 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557b2b8ac180 0x557b2b8ae570 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47e59edf48b0f0f server_cookie=ef0735954884c509 in_seq=0 out_seq=0 2024-03-19T03:02:59.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.393+0000 7f3ae77fe640 10 client.17646 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:02:59.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.393+0000 7f3ae77fe640 1 -- 172.21.15.33:0/2537243517 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557b2b803a40 con 0x557b2b8ac180 2024-03-19T03:02:59.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.393+0000 7f3ae77fe640 10 client.17646 handle_command_reply: tid=0 2024-03-19T03:02:59.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.393+0000 7f3b1158d740 1 client.17646 shutdown 2024-03-19T03:02:59.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.394+0000 7f3b1158d740 1 -- 172.21.15.33:0/2537243517 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b2b893350 msgr2=0x557b2b8994e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:59.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.394+0000 7f3b1158d740 1 --2- 172.21.15.33:0/2537243517 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b2b893350 0x557b2b8994e0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f3af8004650 tx=0x7f3af8005890 comp rx=0 tx=0).stop 2024-03-19T03:02:59.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.394+0000 7f3b1158d740 1 -- 172.21.15.33:0/2537243517 shutdown_connections 2024-03-19T03:02:59.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.394+0000 7f3b1158d740 1 --2- 172.21.15.33:0/2537243517 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557b2b8ac180 0x557b2b8ae570 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.405 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.394+0000 7f3b1158d740 1 --2- 172.21.15.33:0/2537243517 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b2b89bf30 0x557b2b899a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.394+0000 7f3b1158d740 1 --2- 172.21.15.33:0/2537243517 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b2b893350 0x557b2b8994e0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.394+0000 7f3ae77fe640 0 client.17646 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:02:59.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.394+0000 7f3b1158d740 1 --2- 172.21.15.33:0/2537243517 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b2b892a30 0x557b2b898fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.394+0000 7f3b1158d740 1 -- 172.21.15.33:0/2537243517 >> 172.21.15.33:0/2537243517 conn(0x557b2b88d160 msgr2=0x557b2b88eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:59.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.394+0000 7f3b1158d740 1 -- 172.21.15.33:0/2537243517 shutdown_connections 2024-03-19T03:02:59.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.394+0000 7f3b1158d740 1 -- 172.21.15.33:0/2537243517 wait complete. 2024-03-19T03:02:59.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.394+0000 7f3b1158d740 20 client.17646 trim_cache size 0 max 16384 2024-03-19T03:02:59.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.411+0000 7f3b1158d740 1 Processor -- start 2024-03-19T03:02:59.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.411+0000 7f3b1158d740 1 -- start start 2024-03-19T03:02:59.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.411+0000 7f3b1158d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b2b892a30 0x557b2b89b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.411+0000 7f3b1158d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b2b893350 0x557b2b88c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.411+0000 7f3b1158d740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b2b89bf30 0x557b2b899840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.412+0000 7f3b1158d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3b081b37e0 con 0x557b2b893350 2024-03-19T03:02:59.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.412+0000 7f3b1158d740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3b081b10b0 con 0x557b2b89bf30 2024-03-19T03:02:59.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.412+0000 7f3b1158d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3b081aef20 con 0x557b2b892a30 2024-03-19T03:02:59.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.412+0000 7f3b077fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b2b893350 0x557b2b88c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.412+0000 7f3b077fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b2b893350 0x557b2b88c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56938/0 (socket says 172.21.15.33:56938) 2024-03-19T03:02:59.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.412+0000 7f3b077fe640 1 -- 172.21.15.33:0/1896718331 learned_addr learned my addr 172.21.15.33:0/1896718331 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:59.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.412+0000 7f3b07fff640 1 --2- 172.21.15.33:0/1896718331 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b2b892a30 0x557b2b89b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.412+0000 7f3b0ce63640 1 --2- 172.21.15.33:0/1896718331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b2b89bf30 0x557b2b899840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.412+0000 7f3b077fe640 1 -- 172.21.15.33:0/1896718331 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b2b892a30 msgr2=0x557b2b89b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:59.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.412+0000 7f3b077fe640 1 --2- 172.21.15.33:0/1896718331 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b2b892a30 0x557b2b89b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.412+0000 7f3b077fe640 1 -- 172.21.15.33:0/1896718331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b2b89bf30 msgr2=0x557b2b899840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:59.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.412+0000 7f3b077fe640 1 --2- 172.21.15.33:0/1896718331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b2b89bf30 0x557b2b899840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.412+0000 7f3b077fe640 1 -- 172.21.15.33:0/1896718331 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b2b4ca0f0 con 0x557b2b893350 2024-03-19T03:02:59.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.412+0000 7f3b077fe640 1 --2- 172.21.15.33:0/1896718331 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b2b893350 0x557b2b88c1e0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f3af8056b90 tx=0x7f3af802c6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d896551b8904d98 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:59.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.412+0000 7f3afdffb640 1 -- 172.21.15.33:0/1896718331 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af80036b0 con 0x557b2b893350 2024-03-19T03:02:59.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.412+0000 7f3afdffb640 1 -- 172.21.15.33:0/1896718331 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3af8042070 con 0x557b2b893350 2024-03-19T03:02:59.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.412+0000 7f3afdffb640 1 -- 172.21.15.33:0/1896718331 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af8056430 con 0x557b2b893350 2024-03-19T03:02:59.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.413+0000 7f3b1158d740 1 -- 172.21.15.33:0/1896718331 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b2b893350 msgr2=0x557b2b88c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:59.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.413+0000 7f3b1158d740 1 --2- 172.21.15.33:0/1896718331 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b2b893350 0x557b2b88c1e0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f3af8056b90 tx=0x7f3af802c6f0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.413+0000 7f3b1158d740 1 -- 172.21.15.33:0/1896718331 shutdown_connections 2024-03-19T03:02:59.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.413+0000 7f3b1158d740 1 --2- 172.21.15.33:0/1896718331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b2b89bf30 0x557b2b899840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.413+0000 7f3b1158d740 1 --2- 172.21.15.33:0/1896718331 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b2b893350 0x557b2b88c1e0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.413+0000 7f3b1158d740 1 --2- 172.21.15.33:0/1896718331 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b2b892a30 0x557b2b89b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.413+0000 7f3b1158d740 1 -- 172.21.15.33:0/1896718331 >> 172.21.15.33:0/1896718331 conn(0x557b2b88d160 msgr2=0x557b2b897ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:59.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.413+0000 7f3b1158d740 1 -- 172.21.15.33:0/1896718331 shutdown_connections 2024-03-19T03:02:59.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.413+0000 7f3b1158d740 1 -- 172.21.15.33:0/1896718331 wait complete. 2024-03-19T03:02:59.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.413+0000 7f3b1158d740 1 -- start start 2024-03-19T03:02:59.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.413+0000 7f3b1158d740 1 Processor -- start 2024-03-19T03:02:59.417 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.414+0000 7f3b1158d740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b2b892a30 0x557b2b8b16c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.414+0000 7f3b1158d740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b2b893350 0x557b2b8aea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.414+0000 7f3b1158d740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b2b89bf30 0x557b2b8aef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.414+0000 7f3b1158d740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3b081af570 con 0x557b2b893350 2024-03-19T03:02:59.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.414+0000 7f3b1158d740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3b081b09e0 con 0x557b2b892a30 2024-03-19T03:02:59.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.414+0000 7f3b1158d740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3b081afc40 con 0x557b2b89bf30 2024-03-19T03:02:59.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.414+0000 7f3b1158d740 10 client.0 _ll_register_callbacks cb 0x7f3af00097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:02:59.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.414+0000 7f3b077fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b2b893350 0x557b2b8aea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.414+0000 7f3b077fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b2b893350 0x557b2b8aea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56944/0 (socket says 172.21.15.33:56944) 2024-03-19T03:02:59.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.414+0000 7f3b077fe640 1 -- 172.21.15.33:0/733719396 learned_addr learned my addr 172.21.15.33:0/733719396 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:59.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.414+0000 7f3b0ce63640 1 --2- 172.21.15.33:0/733719396 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b2b89bf30 0x557b2b8aef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.414+0000 7f3b07fff640 1 --2- 172.21.15.33:0/733719396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b2b892a30 0x557b2b8b16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.414+0000 7f3b077fe640 1 -- 172.21.15.33:0/733719396 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b2b89bf30 msgr2=0x557b2b8aef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:59.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.414+0000 7f3b077fe640 1 --2- 172.21.15.33:0/733719396 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b2b89bf30 0x557b2b8aef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.414+0000 7f3b077fe640 1 -- 172.21.15.33:0/733719396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b2b892a30 msgr2=0x557b2b8b16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:59.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.414+0000 7f3b077fe640 1 --2- 172.21.15.33:0/733719396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b2b892a30 0x557b2b8b16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.422 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.414+0000 7f3b077fe640 1 -- 172.21.15.33:0/733719396 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b2b8af600 con 0x557b2b893350 2024-03-19T03:02:59.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.414+0000 7f3b077fe640 1 --2- 172.21.15.33:0/733719396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b2b893350 0x557b2b8aea50 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f3af8042040 tx=0x7f3af802cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a6cf24b237b6b8e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:59.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.414+0000 7f3ae77fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:02:59.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.414+0000 7f3ae77fe640 1 -- 172.21.15.33:0/733719396 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af8056bc0 con 0x557b2b893350 2024-03-19T03:02:59.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.414+0000 7f3ae77fe640 1 -- 172.21.15.33:0/733719396 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f3af8049040 con 0x557b2b893350 2024-03-19T03:02:59.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.414+0000 7f3b1158d740 1 -- 172.21.15.33:0/733719396 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3af8042070 con 0x557b2b893350 2024-03-19T03:02:59.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.415+0000 7f3ae77fe640 1 -- 172.21.15.33:0/733719396 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3af8042220 con 0x557b2b893350 2024-03-19T03:02:59.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.415+0000 7f3b1158d740 10 client.17652 fetch_fsmap learned FSMap version 25 2024-03-19T03:02:59.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.415+0000 7f3b1158d740 1 -- 172.21.15.33:0/733719396 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x557b2b911fc0 con 0x557b2b893350 2024-03-19T03:02:59.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.415+0000 7f3ae77fe640 1 -- 172.21.15.33:0/733719396 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3af8042070 con 0x557b2b893350 2024-03-19T03:02:59.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.415+0000 7f3ae77fe640 1 -- 172.21.15.33:0/733719396 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f3af8069460 con 0x557b2b893350 2024-03-19T03:02:59.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.415+0000 7f3b1158d740 10 client.17652 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:02:59.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.415+0000 7f3b1158d740 10 client.17652 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:02:59.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.415+0000 7f3b1158d740 20 client.17652 populate_metadata read hostname 'smithi033' 2024-03-19T03:02:59.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.415+0000 7f3b1158d740 1 --2- 172.21.15.33:0/733719396 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557b2b912600 0x557b2b914a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.415+0000 7f3b1158d740 4 client.17652 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T03:02:59.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.415+0000 7f3b1158d740 1 -- 172.21.15.33:0/733719396 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557b2b7fdd90 con 0x557b2b912600 2024-03-19T03:02:59.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.415+0000 7f3b07fff640 1 --2- 172.21.15.33:0/733719396 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557b2b912600 0x557b2b914a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.416+0000 7f3b07fff640 1 --2- 172.21.15.33:0/733719396 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557b2b912600 0x557b2b914a20 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53232b198739ddbf server_cookie=1b89fbad91ace68d in_seq=0 out_seq=0 2024-03-19T03:02:59.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.416+0000 7f3ae77fe640 10 client.17652 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:02:59.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.416+0000 7f3ae77fe640 1 -- 172.21.15.33:0/733719396 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557b2b7fdd90 con 0x557b2b912600 2024-03-19T03:02:59.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.416+0000 7f3ae77fe640 10 client.17652 handle_command_reply: tid=0 2024-03-19T03:02:59.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.416+0000 7f3b1158d740 1 client.17652 shutdown 2024-03-19T03:02:59.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.416+0000 7f3b1158d740 1 -- 172.21.15.33:0/733719396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b2b893350 msgr2=0x557b2b8aea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:59.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.416+0000 7f3b1158d740 1 --2- 172.21.15.33:0/733719396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b2b893350 0x557b2b8aea50 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f3af8042040 tx=0x7f3af802cdb0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.416+0000 7f3b1158d740 1 -- 172.21.15.33:0/733719396 shutdown_connections 2024-03-19T03:02:59.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.416+0000 7f3b1158d740 1 --2- 172.21.15.33:0/733719396 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557b2b912600 0x557b2b914a20 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.416+0000 7f3b1158d740 1 --2- 172.21.15.33:0/733719396 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557b2b89bf30 0x557b2b8aef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.416+0000 7f3ae77fe640 0 client.17652 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:02:59.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.416+0000 7f3b1158d740 1 --2- 172.21.15.33:0/733719396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557b2b893350 0x557b2b8aea50 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.416+0000 7f3b1158d740 1 --2- 172.21.15.33:0/733719396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557b2b892a30 0x557b2b8b16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.416+0000 7f3b1158d740 1 -- 172.21.15.33:0/733719396 >> 172.21.15.33:0/733719396 conn(0x557b2b88d160 msgr2=0x557b2b8a7970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:59.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.416+0000 7f3b1158d740 1 -- 172.21.15.33:0/733719396 shutdown_connections 2024-03-19T03:02:59.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.416+0000 7f3b1158d740 1 -- 172.21.15.33:0/733719396 wait complete. 2024-03-19T03:02:59.425 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.416+0000 7f3b1158d740 20 client.17652 trim_cache size 0 max 16384 2024-03-19T03:02:59.426 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T03:02:59.426 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-03-19T03:02:59.426 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-03-19T03:02:59.426 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T03:02:59.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.417+0000 7f3b0e8c6640 1 -- 172.21.15.33:0/209899912 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f3af40a2970 msgr2=0x7f3af40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:59.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.417+0000 7f3b0e8c6640 1 --2- 172.21.15.33:0/209899912 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f3af40a2970 0x7f3af40a4e30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3af8005ab0 tx=0x7f3af802c680 comp rx=0 tx=0).stop 2024-03-19T03:02:59.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.417+0000 7f3b0e8c6640 1 -- 172.21.15.33:0/209899912 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b081077d0 msgr2=0x7f3b080745a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:59.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.417+0000 7f3b0e8c6640 1 --2- 172.21.15.33:0/209899912 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b081077d0 0x7f3b080745a0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f3b0000eb10 tx=0x7f3b0000c2d0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.417+0000 7f3b0e8c6640 1 -- 172.21.15.33:0/209899912 shutdown_connections 2024-03-19T03:02:59.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.417+0000 7f3b0e8c6640 1 --2- 172.21.15.33:0/209899912 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f3af40a2970 0x7f3af40a4e30 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.417+0000 7f3b0e8c6640 1 --2- 172.21.15.33:0/209899912 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3b081077d0 0x7f3b080745a0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.417+0000 7f3b0e8c6640 1 --2- 172.21.15.33:0/209899912 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b08102180 0x7f3b08074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.417+0000 7f3b0e8c6640 1 --2- 172.21.15.33:0/209899912 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3b081017c0 0x7f3b08073b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.417+0000 7f3b0e8c6640 1 -- 172.21.15.33:0/209899912 >> 172.21.15.33:0/209899912 conn(0x7f3b080fd250 msgr2=0x7f3b080fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:59.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.417+0000 7f3b0e8c6640 1 -- 172.21.15.33:0/209899912 shutdown_connections 2024-03-19T03:02:59.428 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.417+0000 7f3b0e8c6640 1 -- 172.21.15.33:0/209899912 wait complete. 2024-03-19T03:02:59.429 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T03:02:59.430 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:02:59.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.532+0000 7f6cc6044640 1 -- 172.21.15.33:0/1883800063 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6cc0107820 msgr2=0x7f6cc0107c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:59.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.532+0000 7f6cc6044640 1 --2- 172.21.15.33:0/1883800063 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6cc0107820 0x7f6cc0107c00 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f6cb4009900 tx=0x7f6cb402bcd0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.532+0000 7f6cc6044640 1 -- 172.21.15.33:0/1883800063 shutdown_connections 2024-03-19T03:02:59.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.532+0000 7f6cc6044640 1 --2- 172.21.15.33:0/1883800063 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6cc006a870 0x7f6cc010e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.532+0000 7f6cc6044640 1 --2- 172.21.15.33:0/1883800063 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cc0069ed0 0x7f6cc006a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.532+0000 7f6cc6044640 1 --2- 172.21.15.33:0/1883800063 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6cc0107820 0x7f6cc0107c00 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.532+0000 7f6cc6044640 1 -- 172.21.15.33:0/1883800063 >> 172.21.15.33:0/1883800063 conn(0x7f6cc00777c0 msgr2=0x7f6cc0077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:59.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.532+0000 7f6cc6044640 1 -- 172.21.15.33:0/1883800063 shutdown_connections 2024-03-19T03:02:59.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.532+0000 7f6cc6044640 1 -- 172.21.15.33:0/1883800063 wait complete. 2024-03-19T03:02:59.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.533+0000 7f6cc6044640 1 Processor -- start 2024-03-19T03:02:59.536 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.533+0000 7f6cc6044640 1 -- start start 2024-03-19T03:02:59.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.533+0000 7f6cc6044640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6cc0069ed0 0x7f6cc019a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.533+0000 7f6cc6044640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cc006a870 0x7f6cc019a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.533+0000 7f6cc6044640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6cc0107820 0x7f6cc019ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.533+0000 7f6cc6044640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc01021b0 con 0x7f6cc0069ed0 2024-03-19T03:02:59.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.533+0000 7f6cc6044640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc0102030 con 0x7f6cc006a870 2024-03-19T03:02:59.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.533+0000 7f6cc6044640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc0102330 con 0x7f6cc0107820 2024-03-19T03:02:59.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.533+0000 7f6cbf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6cc0069ed0 0x7f6cc019a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.533+0000 7f6cbf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6cc0069ed0 0x7f6cc019a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56966/0 (socket says 172.21.15.33:56966) 2024-03-19T03:02:59.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.533+0000 7f6cbf7fe640 1 -- 172.21.15.33:0/3717873378 learned_addr learned my addr 172.21.15.33:0/3717873378 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:59.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.533+0000 7f6cbeffd640 1 --2- 172.21.15.33:0/3717873378 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cc006a870 0x7f6cc019a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.533+0000 7f6cbffff640 1 --2- 172.21.15.33:0/3717873378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6cc0107820 0x7f6cc019ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.537 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.533+0000 7f6cbf7fe640 1 -- 172.21.15.33:0/3717873378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6cc0107820 msgr2=0x7f6cc019ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:59.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.533+0000 7f6cbf7fe640 1 --2- 172.21.15.33:0/3717873378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6cc0107820 0x7f6cc019ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.533+0000 7f6cbf7fe640 1 -- 172.21.15.33:0/3717873378 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cc006a870 msgr2=0x7f6cc019a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:59.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.533+0000 7f6cbf7fe640 1 --2- 172.21.15.33:0/3717873378 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cc006a870 0x7f6cc019a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.533+0000 7f6cbf7fe640 1 -- 172.21.15.33:0/3717873378 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cc019f1b0 con 0x7f6cc0069ed0 2024-03-19T03:02:59.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.534+0000 7f6cbf7fe640 1 --2- 172.21.15.33:0/3717873378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6cc0069ed0 0x7f6cc019a1c0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f6cb4005940 tx=0x7f6cb402c690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:59.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.534+0000 7f6cbcff9640 1 -- 172.21.15.33:0/3717873378 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb4043020 con 0x7f6cc0069ed0 2024-03-19T03:02:59.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.534+0000 7f6cbcff9640 1 -- 172.21.15.33:0/3717873378 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6cb4004e00 con 0x7f6cc0069ed0 2024-03-19T03:02:59.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.534+0000 7f6cc6044640 1 -- 172.21.15.33:0/3717873378 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cc019f440 con 0x7f6cc0069ed0 2024-03-19T03:02:59.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.534+0000 7f6cbcff9640 1 -- 172.21.15.33:0/3717873378 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb4003970 con 0x7f6cc0069ed0 2024-03-19T03:02:59.538 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.534+0000 7f6cc6044640 1 -- 172.21.15.33:0/3717873378 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cc01a6d30 con 0x7f6cc0069ed0 2024-03-19T03:02:59.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.535+0000 7f6cc6044640 1 -- 172.21.15.33:0/3717873378 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cc0102f40 con 0x7f6cc0069ed0 2024-03-19T03:02:59.539 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.536+0000 7f6cbcff9640 1 -- 172.21.15.33:0/3717873378 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f6cb4046020 con 0x7f6cc0069ed0 2024-03-19T03:02:59.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.536+0000 7f6cbcff9640 1 --2- 172.21.15.33:0/3717873378 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6ca80a29f0 0x7f6ca80a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.536+0000 7f6cbcff9640 1 -- 172.21.15.33:0/3717873378 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f6cb40f1050 con 0x7f6cc0069ed0 2024-03-19T03:02:59.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.536+0000 7f6cbeffd640 1 --2- 172.21.15.33:0/3717873378 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6ca80a29f0 0x7f6ca80a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.537+0000 7f6cbeffd640 1 --2- 172.21.15.33:0/3717873378 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6ca80a29f0 0x7f6ca80a4eb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6cb00095f0 tx=0x7f6cb0005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:59.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.541+0000 7f6cbcff9640 1 -- 172.21.15.33:0/3717873378 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6cb404d020 con 0x7f6cc0069ed0 2024-03-19T03:02:59.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.764+0000 7f6cc6044640 1 -- 172.21.15.33:0/3717873378 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6cc0068a60 con 0x7f6cc0069ed0 2024-03-19T03:02:59.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.765+0000 7f6cbcff9640 1 -- 172.21.15.33:0/3717873378 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11867 (secure 0 0 0) 0x7f6cb40bdaf0 con 0x7f6cc0069ed0 2024-03-19T03:02:59.767 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T03:02:59.767 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:50:52.201798+0000","modified":"2024-03-19T02:53:11.748697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24488,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/3731232169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":3731232169},{"type":"v1","addr":"172.21.15.96:6839","nonce":3731232169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24457,24488,24509]},"id":1}]} 2024-03-19T03:02:59.767 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-03-19T03:02:59.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.767+0000 7f6cc6044640 1 -- 172.21.15.33:0/3717873378 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6ca80a29f0 msgr2=0x7f6ca80a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:59.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.767+0000 7f6cc6044640 1 --2- 172.21.15.33:0/3717873378 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6ca80a29f0 0x7f6ca80a4eb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6cb00095f0 tx=0x7f6cb0005f90 comp rx=0 tx=0).stop 2024-03-19T03:02:59.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.767+0000 7f6cc6044640 1 -- 172.21.15.33:0/3717873378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6cc0069ed0 msgr2=0x7f6cc019a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:59.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.767+0000 7f6cc6044640 1 --2- 172.21.15.33:0/3717873378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6cc0069ed0 0x7f6cc019a1c0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f6cb4005940 tx=0x7f6cb402c690 comp rx=0 tx=0).stop 2024-03-19T03:02:59.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.767+0000 7f6cc6044640 1 -- 172.21.15.33:0/3717873378 shutdown_connections 2024-03-19T03:02:59.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.767+0000 7f6cc6044640 1 --2- 172.21.15.33:0/3717873378 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f6ca80a29f0 0x7f6ca80a4eb0 secure :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6cb00095f0 tx=0x7f6cb0005f90 comp rx=0 tx=0).stop 2024-03-19T03:02:59.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.767+0000 7f6cc6044640 1 --2- 172.21.15.33:0/3717873378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6cc0107820 0x7f6cc019ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.767+0000 7f6cc6044640 1 --2- 172.21.15.33:0/3717873378 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6cc006a870 0x7f6cc019a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.767+0000 7f6cc6044640 1 --2- 172.21.15.33:0/3717873378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f6cc0069ed0 0x7f6cc019a1c0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.767+0000 7f6cc6044640 1 -- 172.21.15.33:0/3717873378 >> 172.21.15.33:0/3717873378 conn(0x7f6cc00777c0 msgr2=0x7f6cc00ff4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:59.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.767+0000 7f6cc6044640 1 -- 172.21.15.33:0/3717873378 shutdown_connections 2024-03-19T03:02:59.769 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.767+0000 7f6cc6044640 1 -- 172.21.15.33:0/3717873378 wait complete. 2024-03-19T03:02:59.780 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T03:02:59.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.884+0000 7f5e50ea0640 1 -- 172.21.15.33:0/3515306261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e4c105690 msgr2=0x7f5e4c105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:59.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.884+0000 7f5e50ea0640 1 --2- 172.21.15.33:0/3515306261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e4c105690 0x7f5e4c105a70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c009890 tx=0x7f5e3c02bb70 comp rx=0 tx=0).stop 2024-03-19T03:02:59.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.884+0000 7f5e50ea0640 1 -- 172.21.15.33:0/3515306261 shutdown_connections 2024-03-19T03:02:59.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.884+0000 7f5e50ea0640 1 --2- 172.21.15.33:0/3515306261 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e4c100970 0x7f5e4c100dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.884+0000 7f5e50ea0640 1 --2- 172.21.15.33:0/3515306261 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e4c101310 0x7f5e4c10d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.884+0000 7f5e50ea0640 1 --2- 172.21.15.33:0/3515306261 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e4c105690 0x7f5e4c105a70 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.884+0000 7f5e50ea0640 1 -- 172.21.15.33:0/3515306261 >> 172.21.15.33:0/3515306261 conn(0x7f5e4c077500 msgr2=0x7f5e4c077910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:59.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.884+0000 7f5e50ea0640 1 -- 172.21.15.33:0/3515306261 shutdown_connections 2024-03-19T03:02:59.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.884+0000 7f5e50ea0640 1 -- 172.21.15.33:0/3515306261 wait complete. 2024-03-19T03:02:59.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.885+0000 7f5e50ea0640 1 Processor -- start 2024-03-19T03:02:59.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.885+0000 7f5e50ea0640 1 -- start start 2024-03-19T03:02:59.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.885+0000 7f5e50ea0640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e4c100970 0x7f5e4c198ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.885+0000 7f5e50ea0640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e4c101310 0x7f5e4c199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.885+0000 7f5e50ea0640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e4c105690 0x7f5e4c1a3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.885+0000 7f5e50ea0640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5e4c110a30 con 0x7f5e4c101310 2024-03-19T03:02:59.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.885+0000 7f5e50ea0640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5e4c1108b0 con 0x7f5e4c100970 2024-03-19T03:02:59.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.885+0000 7f5e50ea0640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5e4c110bb0 con 0x7f5e4c105690 2024-03-19T03:02:59.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.885+0000 7f5e43fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e4c101310 0x7f5e4c199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.885+0000 7f5e43fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e4c101310 0x7f5e4c199410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56980/0 (socket says 172.21.15.33:56980) 2024-03-19T03:02:59.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.885+0000 7f5e43fff640 1 -- 172.21.15.33:0/4106590013 learned_addr learned my addr 172.21.15.33:0/4106590013 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:59.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.885+0000 7f5e4a59c640 1 --2- 172.21.15.33:0/4106590013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e4c100970 0x7f5e4c198ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.885+0000 7f5e4ad9d640 1 --2- 172.21.15.33:0/4106590013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e4c105690 0x7f5e4c1a3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.885+0000 7f5e43fff640 1 -- 172.21.15.33:0/4106590013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e4c105690 msgr2=0x7f5e4c1a3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:59.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.885+0000 7f5e43fff640 1 --2- 172.21.15.33:0/4106590013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e4c105690 0x7f5e4c1a3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.885+0000 7f5e43fff640 1 -- 172.21.15.33:0/4106590013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e4c100970 msgr2=0x7f5e4c198ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:59.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.885+0000 7f5e43fff640 1 --2- 172.21.15.33:0/4106590013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e4c100970 0x7f5e4c198ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.885+0000 7f5e43fff640 1 -- 172.21.15.33:0/4106590013 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e4c1a4490 con 0x7f5e4c101310 2024-03-19T03:02:59.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.885+0000 7f5e4a59c640 1 --2- 172.21.15.33:0/4106590013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e4c100970 0x7f5e4c198ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:02:59.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.886+0000 7f5e43fff640 1 --2- 172.21.15.33:0/4106590013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e4c101310 0x7f5e4c199410 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f5e3400e9f0 tx=0x7f5e3400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:59.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.886+0000 7f5e437fe640 1 -- 172.21.15.33:0/4106590013 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3400cd80 con 0x7f5e4c101310 2024-03-19T03:02:59.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.886+0000 7f5e437fe640 1 -- 172.21.15.33:0/4106590013 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e340054d0 con 0x7f5e4c101310 2024-03-19T03:02:59.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.886+0000 7f5e50ea0640 1 -- 172.21.15.33:0/4106590013 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e4c1a4660 con 0x7f5e4c101310 2024-03-19T03:02:59.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.886+0000 7f5e437fe640 1 -- 172.21.15.33:0/4106590013 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e34010490 con 0x7f5e4c101310 2024-03-19T03:02:59.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.886+0000 7f5e50ea0640 1 -- 172.21.15.33:0/4106590013 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e4c1a4b00 con 0x7f5e4c101310 2024-03-19T03:02:59.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.888+0000 7f5e437fe640 1 -- 172.21.15.33:0/4106590013 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f5e34005020 con 0x7f5e4c101310 2024-03-19T03:02:59.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.888+0000 7f5e53a85740 1 Processor -- start 2024-03-19T03:02:59.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.889+0000 7f5e53a85740 1 -- start start 2024-03-19T03:02:59.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.889+0000 7f5e53a85740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd44e7a2d0 0x55cd44e7a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.889+0000 7f5e53a85740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cd44e7abf0 0x55cd44e83290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.889+0000 7f5e53a85740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cd44e837d0 0x55cd44e85bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.889+0000 7f5e53a85740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55cd44ddd4b0 con 0x55cd44e7a2d0 2024-03-19T03:02:59.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.889+0000 7f5e53a85740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55cd44d85d70 con 0x55cd44e837d0 2024-03-19T03:02:59.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.889+0000 7f5e53a85740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55cd44dd9550 con 0x55cd44e7abf0 2024-03-19T03:02:59.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.889+0000 7f5e437fe640 1 --2- 172.21.15.33:0/4106590013 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f5e380a2970 0x7f5e380a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.889+0000 7f5e4ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd44e7a2d0 0x55cd44e7a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.889+0000 7f5e4ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd44e7a2d0 0x55cd44e7a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:56994/0 (socket says 172.21.15.33:56994) 2024-03-19T03:02:59.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.889+0000 7f5e4ad9d640 1 -- 172.21.15.33:0/56135625 learned_addr learned my addr 172.21.15.33:0/56135625 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:59.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.889+0000 7f5e4a59c640 1 --2- 172.21.15.33:0/56135625 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cd44e7abf0 0x55cd44e83290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.889+0000 7f5e43fff640 1 --2- 172.21.15.33:0/56135625 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cd44e837d0 0x55cd44e85bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.889+0000 7f5e4ad9d640 1 --2- 172.21.15.33:0/4106590013 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f5e380a2970 0x7f5e380a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.889+0000 7f5e4a59c640 1 -- 172.21.15.33:0/56135625 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cd44e837d0 msgr2=0x55cd44e85bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:59.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.889+0000 7f5e4a59c640 1 --2- 172.21.15.33:0/56135625 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cd44e837d0 0x55cd44e85bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.889+0000 7f5e4a59c640 1 -- 172.21.15.33:0/56135625 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd44e7a2d0 msgr2=0x55cd44e7a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:59.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.889+0000 7f5e4a59c640 1 --2- 172.21.15.33:0/56135625 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd44e7a2d0 0x55cd44e7a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.889+0000 7f5e4a59c640 1 -- 172.21.15.33:0/56135625 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd44da8210 con 0x55cd44e7abf0 2024-03-19T03:02:59.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.889+0000 7f5e437fe640 1 -- 172.21.15.33:0/4106590013 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f5e340cf4c0 con 0x7f5e4c101310 2024-03-19T03:02:59.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.889+0000 7f5e4ad9d640 1 --2- 172.21.15.33:0/56135625 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd44e7a2d0 0x55cd44e7a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:02:59.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.890+0000 7f5e4a59c640 1 --2- 172.21.15.33:0/56135625 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cd44e7abf0 0x55cd44e83290 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c02c180 tx=0x7f5e3c034870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb58d5d71d1de428 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:59.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.890+0000 7f5e4ad9d640 1 --2- 172.21.15.33:0/4106590013 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f5e380a2970 0x7f5e380a4e30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5e44009ba0 tx=0x7f5e44004ab0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:59.898 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.890+0000 7f5e40ff9640 1 -- 172.21.15.33:0/56135625 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c043020 con 0x55cd44e7abf0 2024-03-19T03:02:59.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.890+0000 7f5e40ff9640 1 -- 172.21.15.33:0/56135625 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e3c034e30 con 0x55cd44e7abf0 2024-03-19T03:02:59.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.890+0000 7f5e40ff9640 1 -- 172.21.15.33:0/56135625 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c039040 con 0x55cd44e7abf0 2024-03-19T03:02:59.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.890+0000 7f5e53a85740 1 -- 172.21.15.33:0/56135625 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cd44e7abf0 msgr2=0x55cd44e83290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:59.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.890+0000 7f5e53a85740 1 --2- 172.21.15.33:0/56135625 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cd44e7abf0 0x55cd44e83290 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c02c180 tx=0x7f5e3c034870 comp rx=0 tx=0).stop 2024-03-19T03:02:59.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.890+0000 7f5e53a85740 1 -- 172.21.15.33:0/56135625 shutdown_connections 2024-03-19T03:02:59.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.890+0000 7f5e53a85740 1 --2- 172.21.15.33:0/56135625 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cd44e7abf0 0x55cd44e83290 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.890+0000 7f5e53a85740 1 --2- 172.21.15.33:0/56135625 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cd44e837d0 0x55cd44e85bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.890+0000 7f5e53a85740 1 --2- 172.21.15.33:0/56135625 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd44e7a2d0 0x55cd44e7a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.890+0000 7f5e53a85740 1 -- 172.21.15.33:0/56135625 >> 172.21.15.33:0/56135625 conn(0x55cd44e74a00 msgr2=0x55cd44e74de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:59.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.891+0000 7f5e53a85740 1 -- 172.21.15.33:0/56135625 shutdown_connections 2024-03-19T03:02:59.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.891+0000 7f5e53a85740 1 -- 172.21.15.33:0/56135625 wait complete. 2024-03-19T03:02:59.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.891+0000 7f5e53a85740 1 -- start start 2024-03-19T03:02:59.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.891+0000 7f5e53a85740 1 Processor -- start 2024-03-19T03:02:59.899 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.891+0000 7f5e53a85740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd44e7a2d0 0x55cd44e80820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.891+0000 7f5e53a85740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cd44e7abf0 0x55cd44e80d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.891+0000 7f5e53a85740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cd44e837d0 0x55cd44e812a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.891+0000 7f5e53a85740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55cd44e86560 con 0x55cd44e7a2d0 2024-03-19T03:02:59.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.891+0000 7f5e53a85740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55cd44d99310 con 0x55cd44e7abf0 2024-03-19T03:02:59.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.891+0000 7f5e53a85740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55cd44e866e0 con 0x55cd44e837d0 2024-03-19T03:02:59.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.891+0000 7f5e53a85740 10 client.0 _ll_register_callbacks cb 0x55cd44d8d2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:02:59.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.891+0000 7f5e4a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd44e7a2d0 0x55cd44e80820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.891+0000 7f5e4a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd44e7a2d0 0x55cd44e80820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57004/0 (socket says 172.21.15.33:57004) 2024-03-19T03:02:59.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.891+0000 7f5e4a59c640 1 -- 172.21.15.33:0/936882815 learned_addr learned my addr 172.21.15.33:0/936882815 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:59.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.891+0000 7f5e4ad9d640 1 --2- 172.21.15.33:0/936882815 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cd44e837d0 0x55cd44e812a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.891+0000 7f5e43fff640 1 --2- 172.21.15.33:0/936882815 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cd44e7abf0 0x55cd44e80d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.892+0000 7f5e4a59c640 1 -- 172.21.15.33:0/936882815 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cd44e837d0 msgr2=0x55cd44e812a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:59.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.892+0000 7f5e4a59c640 1 --2- 172.21.15.33:0/936882815 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cd44e837d0 0x55cd44e812a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.892+0000 7f5e4a59c640 1 -- 172.21.15.33:0/936882815 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cd44e7abf0 msgr2=0x55cd44e80d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:59.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.892+0000 7f5e4a59c640 1 --2- 172.21.15.33:0/936882815 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cd44e7abf0 0x55cd44e80d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.892+0000 7f5e4a59c640 1 -- 172.21.15.33:0/936882815 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd44af13f0 con 0x55cd44e7a2d0 2024-03-19T03:02:59.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.892+0000 7f5e4a59c640 1 --2- 172.21.15.33:0/936882815 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd44e7a2d0 0x55cd44e80820 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c005b60 tx=0x7f5e3c02c2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31ae75a522a226f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:59.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.892+0000 7f5e31ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:02:59.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.892+0000 7f5e31ffb640 1 -- 172.21.15.33:0/936882815 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c04f070 con 0x55cd44e7a2d0 2024-03-19T03:02:59.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.892+0000 7f5e31ffb640 1 -- 172.21.15.33:0/936882815 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f5e3c004180 con 0x55cd44e7a2d0 2024-03-19T03:02:59.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.892+0000 7f5e53a85740 1 -- 172.21.15.33:0/936882815 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e3c034e30 con 0x55cd44e7a2d0 2024-03-19T03:02:59.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.892+0000 7f5e31ffb640 1 -- 172.21.15.33:0/936882815 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f5e3c05b050 con 0x55cd44e7a2d0 2024-03-19T03:02:59.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.892+0000 7f5e53a85740 10 client.17670 fetch_fsmap learned FSMap version 25 2024-03-19T03:02:59.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.892+0000 7f5e53a85740 1 -- 172.21.15.33:0/936882815 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55cd44bbb7d0 con 0x55cd44e7a2d0 2024-03-19T03:02:59.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.892+0000 7f5e31ffb640 1 -- 172.21.15.33:0/936882815 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e3c034e30 con 0x55cd44e7a2d0 2024-03-19T03:02:59.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.892+0000 7f5e31ffb640 1 -- 172.21.15.33:0/936882815 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f5e3c03f480 con 0x55cd44e7a2d0 2024-03-19T03:02:59.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.892+0000 7f5e53a85740 10 client.17670 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:02:59.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.893+0000 7f5e53a85740 10 client.17670 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T03:02:59.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.893+0000 7f5e53a85740 20 client.17670 populate_metadata read hostname 'smithi033' 2024-03-19T03:02:59.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.893+0000 7f5e53a85740 1 --2- 172.21.15.33:0/936882815 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55cd44e939a0 0x55cd44e95d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.893+0000 7f5e53a85740 4 client.17670 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:02:59.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.893+0000 7f5e53a85740 1 -- 172.21.15.33:0/936882815 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cd44e3b3e0 con 0x55cd44e939a0 2024-03-19T03:02:59.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.893+0000 7f5e43fff640 1 --2- 172.21.15.33:0/936882815 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55cd44e939a0 0x55cd44e95d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.893+0000 7f5e31ffb640 10 client.17670 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T03:02:59.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.893+0000 7f5e43fff640 1 --2- 172.21.15.33:0/936882815 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55cd44e939a0 0x55cd44e95d90 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=15b2b72fb966dbc9 server_cookie=36dac16dcc37ef5d in_seq=0 out_seq=0 2024-03-19T03:02:59.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.894+0000 7f5e31ffb640 1 -- 172.21.15.33:0/936882815 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cd44e3b3e0 con 0x55cd44e939a0 2024-03-19T03:02:59.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.894+0000 7f5e31ffb640 10 client.17670 handle_command_reply: tid=0 2024-03-19T03:02:59.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.894+0000 7f5e53a85740 1 client.17670 shutdown 2024-03-19T03:02:59.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.894+0000 7f5e53a85740 1 -- 172.21.15.33:0/936882815 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd44e7a2d0 msgr2=0x55cd44e80820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:59.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.894+0000 7f5e53a85740 1 --2- 172.21.15.33:0/936882815 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd44e7a2d0 0x55cd44e80820 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c005b60 tx=0x7f5e3c02c2f0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.894+0000 7f5e53a85740 1 -- 172.21.15.33:0/936882815 shutdown_connections 2024-03-19T03:02:59.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.894+0000 7f5e53a85740 1 --2- 172.21.15.33:0/936882815 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55cd44e939a0 0x55cd44e95d90 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.894+0000 7f5e53a85740 1 --2- 172.21.15.33:0/936882815 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cd44e837d0 0x55cd44e812a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.894+0000 7f5e31ffb640 0 client.17670 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T03:02:59.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.894+0000 7f5e53a85740 1 --2- 172.21.15.33:0/936882815 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cd44e7abf0 0x55cd44e80d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.894+0000 7f5e53a85740 1 --2- 172.21.15.33:0/936882815 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd44e7a2d0 0x55cd44e80820 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.894+0000 7f5e53a85740 1 -- 172.21.15.33:0/936882815 >> 172.21.15.33:0/936882815 conn(0x55cd44e74a00 msgr2=0x55cd44e76750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:59.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.894+0000 7f5e53a85740 1 -- 172.21.15.33:0/936882815 shutdown_connections 2024-03-19T03:02:59.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.894+0000 7f5e53a85740 1 -- 172.21.15.33:0/936882815 wait complete. 2024-03-19T03:02:59.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.894+0000 7f5e53a85740 20 client.17670 trim_cache size 0 max 16384 2024-03-19T03:02:59.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.907+0000 7f5e53a85740 1 Processor -- start 2024-03-19T03:02:59.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.907+0000 7f5e53a85740 1 -- start start 2024-03-19T03:02:59.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.907+0000 7f5e53a85740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cd44e7a2d0 0x55cd44e82f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.907+0000 7f5e53a85740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cd44e7abf0 0x55cd44e810c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.907+0000 7f5e53a85740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd44e837d0 0x55cd44e73a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.907+0000 7f5e53a85740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5e4c1b1500 con 0x55cd44e837d0 2024-03-19T03:02:59.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.907+0000 7f5e53a85740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5e4c1aedd0 con 0x55cd44e7abf0 2024-03-19T03:02:59.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.907+0000 7f5e53a85740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5e4c1acc40 con 0x55cd44e7a2d0 2024-03-19T03:02:59.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.907+0000 7f5e4ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd44e837d0 0x55cd44e73a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.907+0000 7f5e4ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd44e837d0 0x55cd44e73a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57016/0 (socket says 172.21.15.33:57016) 2024-03-19T03:02:59.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.907+0000 7f5e4ad9d640 1 -- 172.21.15.33:0/1521685865 learned_addr learned my addr 172.21.15.33:0/1521685865 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:59.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.908+0000 7f5e43fff640 1 --2- 172.21.15.33:0/1521685865 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cd44e7abf0 0x55cd44e810c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.908+0000 7f5e4a59c640 1 --2- 172.21.15.33:0/1521685865 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cd44e7a2d0 0x55cd44e82f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.914 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.908+0000 7f5e4ad9d640 1 -- 172.21.15.33:0/1521685865 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cd44e7a2d0 msgr2=0x55cd44e82f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:59.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.908+0000 7f5e4ad9d640 1 --2- 172.21.15.33:0/1521685865 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cd44e7a2d0 0x55cd44e82f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.908+0000 7f5e4ad9d640 1 -- 172.21.15.33:0/1521685865 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cd44e7abf0 msgr2=0x55cd44e810c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:59.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.908+0000 7f5e4ad9d640 1 --2- 172.21.15.33:0/1521685865 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cd44e7abf0 0x55cd44e810c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.908+0000 7f5e4ad9d640 1 -- 172.21.15.33:0/1521685865 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd44e7aa20 con 0x55cd44e837d0 2024-03-19T03:02:59.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.908+0000 7f5e4ad9d640 1 --2- 172.21.15.33:0/1521685865 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd44e837d0 0x55cd44e73a20 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f5e4400dbc0 tx=0x7f5e4401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=893beca7d56e773a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:59.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.908+0000 7f5e33fff640 1 -- 172.21.15.33:0/1521685865 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e44004b20 con 0x55cd44e837d0 2024-03-19T03:02:59.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.908+0000 7f5e33fff640 1 -- 172.21.15.33:0/1521685865 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e44004cc0 con 0x55cd44e837d0 2024-03-19T03:02:59.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.908+0000 7f5e33fff640 1 -- 172.21.15.33:0/1521685865 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e44005260 con 0x55cd44e837d0 2024-03-19T03:02:59.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.909+0000 7f5e53a85740 1 -- 172.21.15.33:0/1521685865 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd44e837d0 msgr2=0x55cd44e73a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:59.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.909+0000 7f5e53a85740 1 --2- 172.21.15.33:0/1521685865 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd44e837d0 0x55cd44e73a20 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f5e4400dbc0 tx=0x7f5e4401a000 comp rx=0 tx=0).stop 2024-03-19T03:02:59.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.909+0000 7f5e53a85740 1 -- 172.21.15.33:0/1521685865 shutdown_connections 2024-03-19T03:02:59.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.909+0000 7f5e53a85740 1 --2- 172.21.15.33:0/1521685865 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd44e837d0 0x55cd44e73a20 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.909+0000 7f5e53a85740 1 --2- 172.21.15.33:0/1521685865 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cd44e7abf0 0x55cd44e810c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.909+0000 7f5e53a85740 1 --2- 172.21.15.33:0/1521685865 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cd44e7a2d0 0x55cd44e82f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.909+0000 7f5e53a85740 1 -- 172.21.15.33:0/1521685865 >> 172.21.15.33:0/1521685865 conn(0x55cd44e74a00 msgr2=0x55cd44e73310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:59.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.909+0000 7f5e53a85740 1 -- 172.21.15.33:0/1521685865 shutdown_connections 2024-03-19T03:02:59.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.909+0000 7f5e53a85740 1 -- 172.21.15.33:0/1521685865 wait complete. 2024-03-19T03:02:59.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.909+0000 7f5e53a85740 1 -- start start 2024-03-19T03:02:59.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.909+0000 7f5e53a85740 1 Processor -- start 2024-03-19T03:02:59.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.909+0000 7f5e53a85740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cd44e7a2d0 0x55cd44e8b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.909+0000 7f5e53a85740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cd44e7abf0 0x55cd44e88900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.909+0000 7f5e53a85740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd44e837d0 0x55cd44e88e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.909+0000 7f5e53a85740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5e4c1ad290 con 0x55cd44e837d0 2024-03-19T03:02:59.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.909+0000 7f5e53a85740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5e4c1ae700 con 0x55cd44e7a2d0 2024-03-19T03:02:59.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.909+0000 7f5e53a85740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5e4c1ad960 con 0x55cd44e7abf0 2024-03-19T03:02:59.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.909+0000 7f5e53a85740 10 client.0 _ll_register_callbacks cb 0x7f5e44009c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:02:59.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.909+0000 7f5e4ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd44e837d0 0x55cd44e88e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.916 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.909+0000 7f5e4a59c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cd44e7a2d0 0x55cd44e8b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.909+0000 7f5e4ad9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd44e837d0 0x55cd44e88e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57018/0 (socket says 172.21.15.33:57018) 2024-03-19T03:02:59.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.909+0000 7f5e4ad9d640 1 -- 172.21.15.33:0/1700253413 learned_addr learned my addr 172.21.15.33:0/1700253413 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:02:59.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.909+0000 7f5e43fff640 1 --2- 172.21.15.33:0/1700253413 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cd44e7abf0 0x55cd44e88900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.910+0000 7f5e4ad9d640 1 -- 172.21.15.33:0/1700253413 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cd44e7abf0 msgr2=0x55cd44e88900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:59.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.910+0000 7f5e4ad9d640 1 --2- 172.21.15.33:0/1700253413 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cd44e7abf0 0x55cd44e88900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.910+0000 7f5e4ad9d640 1 -- 172.21.15.33:0/1700253413 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cd44e7a2d0 msgr2=0x55cd44e8b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:02:59.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.910+0000 7f5e4ad9d640 1 --2- 172.21.15.33:0/1700253413 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cd44e7a2d0 0x55cd44e8b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.910+0000 7f5e4ad9d640 1 -- 172.21.15.33:0/1700253413 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd44e8c020 con 0x55cd44e837d0 2024-03-19T03:02:59.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.910+0000 7f5e4a59c640 1 --2- 172.21.15.33:0/1700253413 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cd44e7a2d0 0x55cd44e8b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:02:59.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.910+0000 7f5e43fff640 1 --2- 172.21.15.33:0/1700253413 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cd44e7abf0 0x55cd44e88900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:02:59.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.910+0000 7f5e4ad9d640 1 --2- 172.21.15.33:0/1700253413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd44e837d0 0x55cd44e88e40 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f5e44009f30 tx=0x7f5e4401c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c68f8bc6da0bac8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:02:59.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.910+0000 7f5e31ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:02:59.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.910+0000 7f5e31ffb640 1 -- 172.21.15.33:0/1700253413 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e44036070 con 0x55cd44e837d0 2024-03-19T03:02:59.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.910+0000 7f5e31ffb640 1 -- 172.21.15.33:0/1700253413 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f5e44032660 con 0x55cd44e837d0 2024-03-19T03:02:59.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.910+0000 7f5e53a85740 1 -- 172.21.15.33:0/1700253413 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e44004cc0 con 0x55cd44e837d0 2024-03-19T03:02:59.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.910+0000 7f5e31ffb640 1 -- 172.21.15.33:0/1700253413 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f5e44042050 con 0x55cd44e837d0 2024-03-19T03:02:59.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.910+0000 7f5e53a85740 10 client.17676 fetch_fsmap learned FSMap version 25 2024-03-19T03:02:59.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.910+0000 7f5e53a85740 1 -- 172.21.15.33:0/1700253413 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55cd44efa480 con 0x55cd44e837d0 2024-03-19T03:02:59.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.910+0000 7f5e31ffb640 1 -- 172.21.15.33:0/1700253413 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e44004cc0 con 0x55cd44e837d0 2024-03-19T03:02:59.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.910+0000 7f5e31ffb640 1 -- 172.21.15.33:0/1700253413 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f5e4400b040 con 0x55cd44e837d0 2024-03-19T03:02:59.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.910+0000 7f5e53a85740 10 client.17676 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:02:59.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.910+0000 7f5e53a85740 10 client.17676 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T03:02:59.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.910+0000 7f5e53a85740 20 client.17676 populate_metadata read hostname 'smithi033' 2024-03-19T03:02:59.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.910+0000 7f5e53a85740 1 --2- 172.21.15.33:0/1700253413 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55cd44efab70 0x55cd44efcf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:02:59.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.910+0000 7f5e53a85740 4 client.17676 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T03:02:59.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.910+0000 7f5e53a85740 1 -- 172.21.15.33:0/1700253413 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cd44dea9f0 con 0x55cd44efab70 2024-03-19T03:02:59.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.911+0000 7f5e4a59c640 1 --2- 172.21.15.33:0/1700253413 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55cd44efab70 0x55cd44efcf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:02:59.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.911+0000 7f5e31ffb640 10 client.17676 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T03:02:59.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.911+0000 7f5e4a59c640 1 --2- 172.21.15.33:0/1700253413 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55cd44efab70 0x55cd44efcf90 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5a93df0659e02ee3 server_cookie=893052f9627000e3 in_seq=0 out_seq=0 2024-03-19T03:02:59.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.911+0000 7f5e31ffb640 1 -- 172.21.15.33:0/1700253413 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cd44dea9f0 con 0x55cd44efab70 2024-03-19T03:02:59.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.911+0000 7f5e31ffb640 10 client.17676 handle_command_reply: tid=0 2024-03-19T03:02:59.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.911+0000 7f5e53a85740 1 client.17676 shutdown 2024-03-19T03:02:59.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.911+0000 7f5e53a85740 1 -- 172.21.15.33:0/1700253413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd44e837d0 msgr2=0x55cd44e88e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:59.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.911+0000 7f5e53a85740 1 --2- 172.21.15.33:0/1700253413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd44e837d0 0x55cd44e88e40 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f5e44009f30 tx=0x7f5e4401c8d0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.911+0000 7f5e53a85740 1 -- 172.21.15.33:0/1700253413 shutdown_connections 2024-03-19T03:02:59.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.911+0000 7f5e53a85740 1 --2- 172.21.15.33:0/1700253413 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x55cd44efab70 0x55cd44efcf90 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.911+0000 7f5e53a85740 1 --2- 172.21.15.33:0/1700253413 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55cd44e7abf0 0x55cd44e88900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.911+0000 7f5e53a85740 1 --2- 172.21.15.33:0/1700253413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55cd44e837d0 0x55cd44e88e40 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.911+0000 7f5e31ffb640 0 client.17676 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T03:02:59.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.911+0000 7f5e53a85740 1 --2- 172.21.15.33:0/1700253413 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55cd44e7a2d0 0x55cd44e8b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.911+0000 7f5e53a85740 1 -- 172.21.15.33:0/1700253413 >> 172.21.15.33:0/1700253413 conn(0x55cd44e74a00 msgr2=0x55cd44e73310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:59.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.911+0000 7f5e53a85740 1 -- 172.21.15.33:0/1700253413 shutdown_connections 2024-03-19T03:02:59.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.911+0000 7f5e53a85740 1 -- 172.21.15.33:0/1700253413 wait complete. 2024-03-19T03:02:59.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.911+0000 7f5e53a85740 20 client.17676 trim_cache size 0 max 16384 2024-03-19T03:02:59.920 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T03:02:59.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.912+0000 7f5e50ea0640 1 -- 172.21.15.33:0/4106590013 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f5e380a2970 msgr2=0x7f5e380a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:59.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.912+0000 7f5e50ea0640 1 --2- 172.21.15.33:0/4106590013 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f5e380a2970 0x7f5e380a4e30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5e44009ba0 tx=0x7f5e44004ab0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.912+0000 7f5e50ea0640 1 -- 172.21.15.33:0/4106590013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e4c101310 msgr2=0x7f5e4c199410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:02:59.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.912+0000 7f5e50ea0640 1 --2- 172.21.15.33:0/4106590013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e4c101310 0x7f5e4c199410 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f5e3400e9f0 tx=0x7f5e3400edf0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.912+0000 7f5e50ea0640 1 -- 172.21.15.33:0/4106590013 shutdown_connections 2024-03-19T03:02:59.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.912+0000 7f5e50ea0640 1 --2- 172.21.15.33:0/4106590013 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f5e380a2970 0x7f5e380a4e30 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.912+0000 7f5e50ea0640 1 --2- 172.21.15.33:0/4106590013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e4c105690 0x7f5e4c1a3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.912+0000 7f5e50ea0640 1 --2- 172.21.15.33:0/4106590013 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5e4c101310 0x7f5e4c199410 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.912+0000 7f5e50ea0640 1 --2- 172.21.15.33:0/4106590013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5e4c100970 0x7f5e4c198ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:02:59.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.912+0000 7f5e50ea0640 1 -- 172.21.15.33:0/4106590013 >> 172.21.15.33:0/4106590013 conn(0x7f5e4c077500 msgr2=0x7f5e4c0fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:02:59.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.912+0000 7f5e50ea0640 1 -- 172.21.15.33:0/4106590013 shutdown_connections 2024-03-19T03:02:59.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:02:59.912+0000 7f5e50ea0640 1 -- 172.21.15.33:0/4106590013 wait complete. 2024-03-19T03:02:59.924 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T03:03:00.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.027+0000 7f78416f8640 1 -- 172.21.15.33:0/1751956536 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f783c10e070 msgr2=0x7f783c110460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:00.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.027+0000 7f78416f8640 1 --2- 172.21.15.33:0/1751956536 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f783c10e070 0x7f783c110460 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f782c009be0 tx=0x7f782c02bf80 comp rx=0 tx=0).stop 2024-03-19T03:03:00.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.027+0000 7f78416f8640 1 -- 172.21.15.33:0/1751956536 shutdown_connections 2024-03-19T03:03:00.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.027+0000 7f78416f8640 1 --2- 172.21.15.33:0/1751956536 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f783c10e070 0x7f783c110460 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.027+0000 7f78416f8640 1 --2- 172.21.15.33:0/1751956536 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f783c0744f0 0x7f783c10db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.027+0000 7f78416f8640 1 --2- 172.21.15.33:0/1751956536 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f783c073bd0 0x7f783c073fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.027+0000 7f78416f8640 1 -- 172.21.15.33:0/1751956536 >> 172.21.15.33:0/1751956536 conn(0x7f783c0fd2c0 msgr2=0x7f783c0ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:00.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.027+0000 7f78416f8640 1 -- 172.21.15.33:0/1751956536 shutdown_connections 2024-03-19T03:03:00.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.027+0000 7f78416f8640 1 -- 172.21.15.33:0/1751956536 wait complete. 2024-03-19T03:03:00.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.028+0000 7f78416f8640 1 Processor -- start 2024-03-19T03:03:00.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.028+0000 7f78416f8640 1 -- start start 2024-03-19T03:03:00.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.028+0000 7f78416f8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f783c073bd0 0x7f783c19a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.028+0000 7f78416f8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f783c0744f0 0x7f783c19a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.028+0000 7f78416f8640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f783c10e070 0x7f783c19ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.028+0000 7f78416f8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f783c113c40 con 0x7f783c0744f0 2024-03-19T03:03:00.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.028+0000 7f78416f8640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f783c113ac0 con 0x7f783c073bd0 2024-03-19T03:03:00.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.028+0000 7f78416f8640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f783c113dc0 con 0x7f783c10e070 2024-03-19T03:03:00.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.029+0000 7f783a7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f783c0744f0 0x7f783c19a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.029+0000 7f783affd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f783c073bd0 0x7f783c19a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.029+0000 7f783a7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f783c0744f0 0x7f783c19a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57038/0 (socket says 172.21.15.33:57038) 2024-03-19T03:03:00.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.029+0000 7f783a7fc640 1 -- 172.21.15.33:0/595926328 learned_addr learned my addr 172.21.15.33:0/595926328 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:00.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.029+0000 7f783b7fe640 1 --2- 172.21.15.33:0/595926328 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f783c10e070 0x7f783c19ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.029+0000 7f783affd640 1 -- 172.21.15.33:0/595926328 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f783c10e070 msgr2=0x7f783c19ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:00.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.029+0000 7f783affd640 1 --2- 172.21.15.33:0/595926328 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f783c10e070 0x7f783c19ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.029+0000 7f783affd640 1 -- 172.21.15.33:0/595926328 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f783c0744f0 msgr2=0x7f783c19a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:00.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.029+0000 7f783affd640 1 --2- 172.21.15.33:0/595926328 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f783c0744f0 0x7f783c19a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.029+0000 7f783affd640 1 -- 172.21.15.33:0/595926328 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f783c19f1c0 con 0x7f783c073bd0 2024-03-19T03:03:00.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.029+0000 7f783a7fc640 1 --2- 172.21.15.33:0/595926328 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f783c0744f0 0x7f783c19a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:00.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.029+0000 7f783affd640 1 --2- 172.21.15.33:0/595926328 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f783c073bd0 0x7f783c19a1d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f783400e9f0 tx=0x7f783400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:00.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.029+0000 7f7833fff640 1 -- 172.21.15.33:0/595926328 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f783400ccc0 con 0x7f783c073bd0 2024-03-19T03:03:00.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.029+0000 7f7833fff640 1 -- 172.21.15.33:0/595926328 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78340108e0 con 0x7f783c073bd0 2024-03-19T03:03:00.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.029+0000 7f78416f8640 1 -- 172.21.15.33:0/595926328 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f783c19f4b0 con 0x7f783c073bd0 2024-03-19T03:03:00.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.029+0000 7f7833fff640 1 -- 172.21.15.33:0/595926328 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7834018880 con 0x7f783c073bd0 2024-03-19T03:03:00.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.029+0000 7f78416f8640 1 -- 172.21.15.33:0/595926328 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f783c1a6d40 con 0x7f783c073bd0 2024-03-19T03:03:00.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.031+0000 7f7833fff640 1 -- 172.21.15.33:0/595926328 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f7834010a90 con 0x7f783c073bd0 2024-03-19T03:03:00.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.032+0000 7f78443c2740 1 Processor -- start 2024-03-19T03:03:00.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.032+0000 7f78443c2740 1 -- start start 2024-03-19T03:03:00.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.032+0000 7f7833fff640 1 --2- 172.21.15.33:0/595926328 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f78280a29f0 0x7f78280a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.032+0000 7f783b7fe640 1 --2- 172.21.15.33:0/595926328 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f78280a29f0 0x7f78280a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.032+0000 7f7833fff640 1 -- 172.21.15.33:0/595926328 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f78340cf540 con 0x7f783c073bd0 2024-03-19T03:03:00.040 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.032+0000 7f78443c2740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625014512d0 0x5625014516b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.032+0000 7f78443c2740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562501451bf0 0x56250145a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.032+0000 7f78443c2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250145a7d0 0x56250145cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.032+0000 7f78443c2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5625013b44b0 con 0x56250145a7d0 2024-03-19T03:03:00.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.032+0000 7f78443c2740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x56250135cd70 con 0x562501451bf0 2024-03-19T03:03:00.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.032+0000 7f78443c2740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5625013b0550 con 0x5625014512d0 2024-03-19T03:03:00.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.032+0000 7f783affd640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625014512d0 0x5625014516b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.032+0000 7f783a7fc640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562501451bf0 0x56250145a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.032+0000 7f783affd640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625014512d0 0x5625014516b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.33:46550/0 (socket says 172.21.15.33:46550) 2024-03-19T03:03:00.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.032+0000 7f783affd640 1 -- 172.21.15.33:0/1816139242 learned_addr learned my addr 172.21.15.33:0/1816139242 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:00.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.032+0000 7f783b7fe640 1 --2- 172.21.15.33:0/1816139242 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250145a7d0 0x56250145cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.033+0000 7f783affd640 1 -- 172.21.15.33:0/1816139242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562501451bf0 msgr2=0x56250145a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:00.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.033+0000 7f783affd640 1 --2- 172.21.15.33:0/1816139242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562501451bf0 0x56250145a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.033+0000 7f783b7fe640 1 --2- 172.21.15.33:0/595926328 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f78280a29f0 0x7f78280a4eb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f782c009370 tx=0x7f782c033b90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:00.041 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.033+0000 7f783affd640 1 -- 172.21.15.33:0/1816139242 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250145a7d0 msgr2=0x56250145cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:00.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.033+0000 7f783affd640 1 --2- 172.21.15.33:0/1816139242 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250145a7d0 0x56250145cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.033+0000 7f783affd640 1 -- 172.21.15.33:0/1816139242 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56250137f210 con 0x5625014512d0 2024-03-19T03:03:00.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.033+0000 7f783a7fc640 1 --2- 172.21.15.33:0/1816139242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562501451bf0 0x56250145a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:00.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.033+0000 7f783affd640 1 --2- 172.21.15.33:0/1816139242 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625014512d0 0x5625014516b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f78340981f0 tx=0x7f7834010f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1435fec478fa88af server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:00.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.033+0000 7f78317fa640 1 -- 172.21.15.33:0/1816139242 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78340860c0 con 0x5625014512d0 2024-03-19T03:03:00.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.033+0000 7f78317fa640 1 -- 172.21.15.33:0/1816139242 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78340ccae0 con 0x5625014512d0 2024-03-19T03:03:00.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.033+0000 7f78317fa640 1 -- 172.21.15.33:0/1816139242 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f783409ac70 con 0x5625014512d0 2024-03-19T03:03:00.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.034+0000 7f78443c2740 1 -- 172.21.15.33:0/1816139242 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625014512d0 msgr2=0x5625014516b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:00.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.034+0000 7f78443c2740 1 --2- 172.21.15.33:0/1816139242 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625014512d0 0x5625014516b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f78340981f0 tx=0x7f7834010f30 comp rx=0 tx=0).stop 2024-03-19T03:03:00.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.034+0000 7f78443c2740 1 -- 172.21.15.33:0/1816139242 shutdown_connections 2024-03-19T03:03:00.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.034+0000 7f78443c2740 1 --2- 172.21.15.33:0/1816139242 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250145a7d0 0x56250145cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.034+0000 7f78443c2740 1 --2- 172.21.15.33:0/1816139242 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562501451bf0 0x56250145a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.034+0000 7f78443c2740 1 --2- 172.21.15.33:0/1816139242 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625014512d0 0x5625014516b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.042 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.034+0000 7f78443c2740 1 -- 172.21.15.33:0/1816139242 >> 172.21.15.33:0/1816139242 conn(0x56250144ba00 msgr2=0x56250144bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:00.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.034+0000 7f78443c2740 1 -- 172.21.15.33:0/1816139242 shutdown_connections 2024-03-19T03:03:00.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.034+0000 7f78443c2740 1 -- 172.21.15.33:0/1816139242 wait complete. 2024-03-19T03:03:00.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.034+0000 7f78443c2740 1 -- start start 2024-03-19T03:03:00.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.034+0000 7f78443c2740 1 Processor -- start 2024-03-19T03:03:00.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.034+0000 7f78443c2740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625014512d0 0x562501457830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.034+0000 7f78443c2740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562501451bf0 0x562501457d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.034+0000 7f78443c2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250145a7d0 0x5625014582b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.034+0000 7f78443c2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x56250145d500 con 0x56250145a7d0 2024-03-19T03:03:00.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.034+0000 7f78443c2740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x562501370310 con 0x562501451bf0 2024-03-19T03:03:00.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.034+0000 7f78443c2740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x56250145d680 con 0x5625014512d0 2024-03-19T03:03:00.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.034+0000 7f78443c2740 10 client.0 _ll_register_callbacks cb 0x5625013642d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:03:00.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.035+0000 7f783b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250145a7d0 0x5625014582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.035+0000 7f783affd640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625014512d0 0x562501457830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.043 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.035+0000 7f783b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250145a7d0 0x5625014582b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57064/0 (socket says 172.21.15.33:57064) 2024-03-19T03:03:00.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.035+0000 7f783b7fe640 1 -- 172.21.15.33:0/1794517772 learned_addr learned my addr 172.21.15.33:0/1794517772 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:00.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.035+0000 7f783a7fc640 1 --2- 172.21.15.33:0/1794517772 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562501451bf0 0x562501457d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.035+0000 7f783b7fe640 1 -- 172.21.15.33:0/1794517772 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625014512d0 msgr2=0x562501457830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:00.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.035+0000 7f783b7fe640 1 --2- 172.21.15.33:0/1794517772 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625014512d0 0x562501457830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.035+0000 7f783b7fe640 1 -- 172.21.15.33:0/1794517772 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562501451bf0 msgr2=0x562501457d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:00.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.035+0000 7f783b7fe640 1 --2- 172.21.15.33:0/1794517772 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562501451bf0 0x562501457d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.035+0000 7f783b7fe640 1 -- 172.21.15.33:0/1794517772 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625010c83f0 con 0x56250145a7d0 2024-03-19T03:03:00.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.035+0000 7f783b7fe640 1 --2- 172.21.15.33:0/1794517772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250145a7d0 0x5625014582b0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f782c009be0 tx=0x7f782c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da854ebf61d8327e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:00.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.035+0000 7f78227fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:03:00.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.035+0000 7f78227fc640 1 -- 172.21.15.33:0/1794517772 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782c004250 con 0x56250145a7d0 2024-03-19T03:03:00.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.035+0000 7f78227fc640 1 -- 172.21.15.33:0/1794517772 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f782c0043f0 con 0x56250145a7d0 2024-03-19T03:03:00.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.035+0000 7f78443c2740 1 -- 172.21.15.33:0/1794517772 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78340ccae0 con 0x56250145a7d0 2024-03-19T03:03:00.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.035+0000 7f78227fc640 1 -- 172.21.15.33:0/1794517772 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f782c05b050 con 0x56250145a7d0 2024-03-19T03:03:00.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.035+0000 7f78443c2740 10 client.17688 fetch_fsmap learned FSMap version 25 2024-03-19T03:03:00.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.035+0000 7f78443c2740 1 -- 172.21.15.33:0/1794517772 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5625011927d0 con 0x56250145a7d0 2024-03-19T03:03:00.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.036+0000 7f78227fc640 1 -- 172.21.15.33:0/1794517772 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78340ccae0 con 0x56250145a7d0 2024-03-19T03:03:00.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.036+0000 7f78227fc640 1 -- 172.21.15.33:0/1794517772 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f782c041a30 con 0x56250145a7d0 2024-03-19T03:03:00.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.036+0000 7f78443c2740 10 client.17688 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:03:00.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.036+0000 7f78443c2740 10 client.17688 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T03:03:00.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.036+0000 7f78443c2740 20 client.17688 populate_metadata read hostname 'smithi033' 2024-03-19T03:03:00.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.036+0000 7f78443c2740 1 --2- 172.21.15.33:0/1794517772 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x56250146abf0 0x56250146d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.036+0000 7f78443c2740 4 client.17688 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:03:00.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.036+0000 7f78443c2740 1 -- 172.21.15.33:0/1794517772 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625014123e0 con 0x56250146abf0 2024-03-19T03:03:00.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.036+0000 7f783affd640 1 --2- 172.21.15.33:0/1794517772 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x56250146abf0 0x56250146d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.036+0000 7f783affd640 1 --2- 172.21.15.33:0/1794517772 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x56250146abf0 0x56250146d010 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=957925cb335e7431 server_cookie=d27ec9c3b9e883d in_seq=0 out_seq=0 2024-03-19T03:03:00.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.036+0000 7f78227fc640 10 client.17688 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T03:03:00.045 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.037+0000 7f78227fc640 1 -- 172.21.15.33:0/1794517772 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5625014123e0 con 0x56250146abf0 2024-03-19T03:03:00.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.037+0000 7f78227fc640 10 client.17688 handle_command_reply: tid=0 2024-03-19T03:03:00.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.037+0000 7f78443c2740 1 client.17688 shutdown 2024-03-19T03:03:00.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.037+0000 7f78443c2740 1 -- 172.21.15.33:0/1794517772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250145a7d0 msgr2=0x5625014582b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:00.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.037+0000 7f78443c2740 1 --2- 172.21.15.33:0/1794517772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250145a7d0 0x5625014582b0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f782c009be0 tx=0x7f782c02bf80 comp rx=0 tx=0).stop 2024-03-19T03:03:00.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.037+0000 7f78443c2740 1 -- 172.21.15.33:0/1794517772 shutdown_connections 2024-03-19T03:03:00.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.037+0000 7f78443c2740 1 --2- 172.21.15.33:0/1794517772 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x56250146abf0 0x56250146d010 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.037+0000 7f78443c2740 1 --2- 172.21.15.33:0/1794517772 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250145a7d0 0x5625014582b0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.037+0000 7f78227fc640 0 client.17688 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T03:03:00.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.037+0000 7f78443c2740 1 --2- 172.21.15.33:0/1794517772 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562501451bf0 0x562501457d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.037+0000 7f78443c2740 1 --2- 172.21.15.33:0/1794517772 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625014512d0 0x562501457830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.037+0000 7f78443c2740 1 -- 172.21.15.33:0/1794517772 >> 172.21.15.33:0/1794517772 conn(0x56250144ba00 msgr2=0x56250144d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:00.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.037+0000 7f78443c2740 1 -- 172.21.15.33:0/1794517772 shutdown_connections 2024-03-19T03:03:00.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.037+0000 7f78443c2740 1 -- 172.21.15.33:0/1794517772 wait complete. 2024-03-19T03:03:00.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.037+0000 7f78443c2740 20 client.17688 trim_cache size 0 max 16384 2024-03-19T03:03:00.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.050+0000 7f78443c2740 1 Processor -- start 2024-03-19T03:03:00.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.051+0000 7f78443c2740 1 -- start start 2024-03-19T03:03:00.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.051+0000 7f78443c2740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625014512d0 0x5625014662d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.051+0000 7f78443c2740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562501451bf0 0x5625014728c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.051+0000 7f78443c2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250145a7d0 0x562501457d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.051+0000 7f78443c2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f783c1b36f0 con 0x56250145a7d0 2024-03-19T03:03:00.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.051+0000 7f78443c2740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f783c1b0fc0 con 0x562501451bf0 2024-03-19T03:03:00.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.051+0000 7f78443c2740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f783c1aee30 con 0x5625014512d0 2024-03-19T03:03:00.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.051+0000 7f783b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250145a7d0 0x562501457d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.051+0000 7f783a7fc640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562501451bf0 0x5625014728c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.051+0000 7f783b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250145a7d0 0x562501457d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57070/0 (socket says 172.21.15.33:57070) 2024-03-19T03:03:00.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.051+0000 7f783affd640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625014512d0 0x5625014662d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.051+0000 7f783b7fe640 1 -- 172.21.15.33:0/64332632 learned_addr learned my addr 172.21.15.33:0/64332632 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:00.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.051+0000 7f783a7fc640 1 -- 172.21.15.33:0/64332632 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625014512d0 msgr2=0x5625014662d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:00.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.051+0000 7f783a7fc640 1 --2- 172.21.15.33:0/64332632 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625014512d0 0x5625014662d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.051+0000 7f783a7fc640 1 -- 172.21.15.33:0/64332632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250145a7d0 msgr2=0x562501457d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:00.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.051+0000 7f783a7fc640 1 --2- 172.21.15.33:0/64332632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250145a7d0 0x562501457d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.051+0000 7f783a7fc640 1 -- 172.21.15.33:0/64332632 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56250145cbc0 con 0x562501451bf0 2024-03-19T03:03:00.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.051+0000 7f783b7fe640 1 --2- 172.21.15.33:0/64332632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250145a7d0 0x562501457d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:00.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.051+0000 7f783affd640 1 --2- 172.21.15.33:0/64332632 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625014512d0 0x5625014662d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:00.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.052+0000 7f783a7fc640 1 --2- 172.21.15.33:0/64332632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562501451bf0 0x5625014728c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x562501450da0 tx=0x7f78240039c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f7742a61f96e6bf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:00.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.052+0000 7f7830ff9640 1 -- 172.21.15.33:0/64332632 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7824005020 con 0x562501451bf0 2024-03-19T03:03:00.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.052+0000 7f7830ff9640 1 -- 172.21.15.33:0/64332632 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78240047a0 con 0x562501451bf0 2024-03-19T03:03:00.055 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.052+0000 7f7830ff9640 1 -- 172.21.15.33:0/64332632 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78240107b0 con 0x562501451bf0 2024-03-19T03:03:00.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.053+0000 7f78443c2740 1 -- 172.21.15.33:0/64332632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562501451bf0 msgr2=0x5625014728c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:00.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.053+0000 7f78443c2740 1 --2- 172.21.15.33:0/64332632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562501451bf0 0x5625014728c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x562501450da0 tx=0x7f78240039c0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.053+0000 7f78443c2740 1 -- 172.21.15.33:0/64332632 shutdown_connections 2024-03-19T03:03:00.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.053+0000 7f78443c2740 1 --2- 172.21.15.33:0/64332632 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250145a7d0 0x562501457d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.053+0000 7f78443c2740 1 --2- 172.21.15.33:0/64332632 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562501451bf0 0x5625014728c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.053+0000 7f78443c2740 1 --2- 172.21.15.33:0/64332632 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5625014512d0 0x5625014662d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.053+0000 7f78443c2740 1 -- 172.21.15.33:0/64332632 >> 172.21.15.33:0/64332632 conn(0x56250144ba00 msgr2=0x56250144aba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:00.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.053+0000 7f78443c2740 1 -- 172.21.15.33:0/64332632 shutdown_connections 2024-03-19T03:03:00.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.053+0000 7f78443c2740 1 -- 172.21.15.33:0/64332632 wait complete. 2024-03-19T03:03:00.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.053+0000 7f78443c2740 1 -- start start 2024-03-19T03:03:00.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.053+0000 7f78443c2740 1 Processor -- start 2024-03-19T03:03:00.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.053+0000 7f78443c2740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5625014512d0 0x5625014624a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.053+0000 7f78443c2740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562501451bf0 0x5625014629e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.053+0000 7f78443c2740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250145a7d0 0x56250145e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.053+0000 7f78443c2740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f783c1af480 con 0x56250145a7d0 2024-03-19T03:03:00.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.053+0000 7f78443c2740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f783c1b08f0 con 0x5625014512d0 2024-03-19T03:03:00.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.054+0000 7f78443c2740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f783c1afb50 con 0x562501451bf0 2024-03-19T03:03:00.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.054+0000 7f78443c2740 10 client.0 _ll_register_callbacks cb 0x7f7834005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:03:00.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.054+0000 7f783b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250145a7d0 0x56250145e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.054+0000 7f783b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250145a7d0 0x56250145e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57072/0 (socket says 172.21.15.33:57072) 2024-03-19T03:03:00.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.054+0000 7f783b7fe640 1 -- 172.21.15.33:0/1184423120 learned_addr learned my addr 172.21.15.33:0/1184423120 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:00.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.054+0000 7f783affd640 1 --2- 172.21.15.33:0/1184423120 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5625014512d0 0x5625014624a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.054+0000 7f783a7fc640 1 --2- 172.21.15.33:0/1184423120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562501451bf0 0x5625014629e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.054+0000 7f783b7fe640 1 -- 172.21.15.33:0/1184423120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562501451bf0 msgr2=0x5625014629e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:00.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.054+0000 7f783b7fe640 1 --2- 172.21.15.33:0/1184423120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562501451bf0 0x5625014629e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.054+0000 7f783b7fe640 1 -- 172.21.15.33:0/1184423120 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5625014512d0 msgr2=0x5625014624a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:00.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.054+0000 7f783b7fe640 1 --2- 172.21.15.33:0/1184423120 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5625014512d0 0x5625014624a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.054+0000 7f783b7fe640 1 -- 172.21.15.33:0/1184423120 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56250145f140 con 0x56250145a7d0 2024-03-19T03:03:00.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.054+0000 7f783affd640 1 --2- 172.21.15.33:0/1184423120 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5625014512d0 0x5625014624a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:00.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.054+0000 7f783b7fe640 1 --2- 172.21.15.33:0/1184423120 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250145a7d0 0x56250145e940 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f782c041ed0 tx=0x7f782c033ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db1c0c38305fe313 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:00.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.054+0000 7f78227fc640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:03:00.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.054+0000 7f78227fc640 1 -- 172.21.15.33:0/1184423120 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782c065030 con 0x56250145a7d0 2024-03-19T03:03:00.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.054+0000 7f78227fc640 1 -- 172.21.15.33:0/1184423120 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f782c03f930 con 0x56250145a7d0 2024-03-19T03:03:00.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.054+0000 7f78443c2740 1 -- 172.21.15.33:0/1184423120 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78240047a0 con 0x56250145a7d0 2024-03-19T03:03:00.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.054+0000 7f78227fc640 1 -- 172.21.15.33:0/1184423120 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f782c03fb70 con 0x56250145a7d0 2024-03-19T03:03:00.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.054+0000 7f78443c2740 10 client.17694 fetch_fsmap learned FSMap version 25 2024-03-19T03:03:00.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.055+0000 7f78443c2740 1 -- 172.21.15.33:0/1184423120 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5625014d1920 con 0x56250145a7d0 2024-03-19T03:03:00.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.055+0000 7f78227fc640 1 -- 172.21.15.33:0/1184423120 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78240047a0 con 0x56250145a7d0 2024-03-19T03:03:00.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.055+0000 7f78227fc640 1 -- 172.21.15.33:0/1184423120 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f782c0414f0 con 0x56250145a7d0 2024-03-19T03:03:00.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.055+0000 7f78443c2740 10 client.17694 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:03:00.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.055+0000 7f78443c2740 10 client.17694 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T03:03:00.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.055+0000 7f78443c2740 20 client.17694 populate_metadata read hostname 'smithi033' 2024-03-19T03:03:00.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.055+0000 7f78443c2740 1 --2- 172.21.15.33:0/1184423120 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x5625014d20c0 0x5625014d44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.055+0000 7f78443c2740 4 client.17694 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T03:03:00.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.055+0000 7f78443c2740 1 -- 172.21.15.33:0/1184423120 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5625013c19f0 con 0x5625014d20c0 2024-03-19T03:03:00.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.055+0000 7f783affd640 1 --2- 172.21.15.33:0/1184423120 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x5625014d20c0 0x5625014d44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.055+0000 7f783affd640 1 --2- 172.21.15.33:0/1184423120 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x5625014d20c0 0x5625014d44e0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6fac46bff0026e4b server_cookie=17a108f53c22915b in_seq=0 out_seq=0 2024-03-19T03:03:00.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.055+0000 7f78227fc640 10 client.17694 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T03:03:00.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.056+0000 7f78227fc640 1 -- 172.21.15.33:0/1184423120 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5625013c19f0 con 0x5625014d20c0 2024-03-19T03:03:00.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.056+0000 7f78227fc640 10 client.17694 handle_command_reply: tid=0 2024-03-19T03:03:00.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.056+0000 7f78443c2740 1 client.17694 shutdown 2024-03-19T03:03:00.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.056+0000 7f78443c2740 1 -- 172.21.15.33:0/1184423120 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250145a7d0 msgr2=0x56250145e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:00.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.056+0000 7f78443c2740 1 --2- 172.21.15.33:0/1184423120 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250145a7d0 0x56250145e940 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f782c041ed0 tx=0x7f782c033ca0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.056+0000 7f78443c2740 1 -- 172.21.15.33:0/1184423120 shutdown_connections 2024-03-19T03:03:00.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.056+0000 7f78443c2740 1 --2- 172.21.15.33:0/1184423120 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x5625014d20c0 0x5625014d44e0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.056+0000 7f78443c2740 1 --2- 172.21.15.33:0/1184423120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x562501451bf0 0x5625014629e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.056+0000 7f78443c2740 1 --2- 172.21.15.33:0/1184423120 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x56250145a7d0 0x56250145e940 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.056+0000 7f78227fc640 0 client.17694 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T03:03:00.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.056+0000 7f78443c2740 1 --2- 172.21.15.33:0/1184423120 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5625014512d0 0x5625014624a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.056+0000 7f78443c2740 1 -- 172.21.15.33:0/1184423120 >> 172.21.15.33:0/1184423120 conn(0x56250144ba00 msgr2=0x562501458ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:00.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.056+0000 7f78443c2740 1 -- 172.21.15.33:0/1184423120 shutdown_connections 2024-03-19T03:03:00.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.056+0000 7f78443c2740 1 -- 172.21.15.33:0/1184423120 wait complete. 2024-03-19T03:03:00.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.056+0000 7f78443c2740 20 client.17694 trim_cache size 0 max 16384 2024-03-19T03:03:00.064 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T03:03:00.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.057+0000 7f78416f8640 1 -- 172.21.15.33:0/595926328 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f78280a29f0 msgr2=0x7f78280a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:00.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.057+0000 7f78416f8640 1 --2- 172.21.15.33:0/595926328 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f78280a29f0 0x7f78280a4eb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f782c009370 tx=0x7f782c033b90 comp rx=0 tx=0).stop 2024-03-19T03:03:00.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.057+0000 7f78416f8640 1 -- 172.21.15.33:0/595926328 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f783c073bd0 msgr2=0x7f783c19a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:00.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.057+0000 7f78416f8640 1 --2- 172.21.15.33:0/595926328 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f783c073bd0 0x7f783c19a1d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f783400e9f0 tx=0x7f783400eeb0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.057+0000 7f78416f8640 1 -- 172.21.15.33:0/595926328 shutdown_connections 2024-03-19T03:03:00.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.057+0000 7f78416f8640 1 --2- 172.21.15.33:0/595926328 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f78280a29f0 0x7f78280a4eb0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.065 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.057+0000 7f78416f8640 1 --2- 172.21.15.33:0/595926328 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f783c10e070 0x7f783c19ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.057+0000 7f78416f8640 1 --2- 172.21.15.33:0/595926328 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f783c0744f0 0x7f783c19a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.057+0000 7f78416f8640 1 --2- 172.21.15.33:0/595926328 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f783c073bd0 0x7f783c19a1d0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.057+0000 7f78416f8640 1 -- 172.21.15.33:0/595926328 >> 172.21.15.33:0/595926328 conn(0x7f783c0fd2c0 msgr2=0x7f783c0feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:00.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.057+0000 7f78416f8640 1 -- 172.21.15.33:0/595926328 shutdown_connections 2024-03-19T03:03:00.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.057+0000 7f78416f8640 1 -- 172.21.15.33:0/595926328 wait complete. 2024-03-19T03:03:00.069 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T03:03:00.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.182+0000 7f1baf89a640 1 -- 172.21.15.33:0/1586420479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ba81077a0 msgr2=0x7f1ba8107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:00.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.182+0000 7f1baf89a640 1 --2- 172.21.15.33:0/1586420479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ba81077a0 0x7f1ba8107b80 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4009930 tx=0x7f1ba402bc30 comp rx=0 tx=0).stop 2024-03-19T03:03:00.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.182+0000 7f1baf89a640 1 -- 172.21.15.33:0/1586420479 shutdown_connections 2024-03-19T03:03:00.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.182+0000 7f1baf89a640 1 --2- 172.21.15.33:0/1586420479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ba8101790 0x7f1ba8101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.182+0000 7f1baf89a640 1 --2- 172.21.15.33:0/1586420479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ba8102150 0x7f1ba810e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.182+0000 7f1baf89a640 1 --2- 172.21.15.33:0/1586420479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ba81077a0 0x7f1ba8107b80 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.182+0000 7f1baf89a640 1 -- 172.21.15.33:0/1586420479 >> 172.21.15.33:0/1586420479 conn(0x7f1ba80fd2c0 msgr2=0x7f1ba80ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:00.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.182+0000 7f1baf89a640 1 -- 172.21.15.33:0/1586420479 shutdown_connections 2024-03-19T03:03:00.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.182+0000 7f1baf89a640 1 -- 172.21.15.33:0/1586420479 wait complete. 2024-03-19T03:03:00.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.182+0000 7f1baf89a640 1 Processor -- start 2024-03-19T03:03:00.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.182+0000 7f1baf89a640 1 -- start start 2024-03-19T03:03:00.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.183+0000 7f1baf89a640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ba8101790 0x7f1ba8198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.183+0000 7f1baf89a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ba8102150 0x7f1ba8198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.183+0000 7f1baf89a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ba81077a0 0x7f1ba819c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.183+0000 7f1baf89a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba8111a20 con 0x7f1ba81077a0 2024-03-19T03:03:00.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.183+0000 7f1baf89a640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba81118a0 con 0x7f1ba8102150 2024-03-19T03:03:00.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.183+0000 7f1baf89a640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba8111ba0 con 0x7f1ba8101790 2024-03-19T03:03:00.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.183+0000 7f1bade37640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ba81077a0 0x7f1ba819c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.183+0000 7f1bade37640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ba81077a0 0x7f1ba819c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57088/0 (socket says 172.21.15.33:57088) 2024-03-19T03:03:00.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.183+0000 7f1bade37640 1 -- 172.21.15.33:0/1370012999 learned_addr learned my addr 172.21.15.33:0/1370012999 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:00.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.183+0000 7f1bad636640 1 --2- 172.21.15.33:0/1370012999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ba8101790 0x7f1ba8198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.183+0000 7f1bace35640 1 --2- 172.21.15.33:0/1370012999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ba8102150 0x7f1ba8198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.183+0000 7f1bade37640 1 -- 172.21.15.33:0/1370012999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ba8101790 msgr2=0x7f1ba8198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:00.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.183+0000 7f1bade37640 1 --2- 172.21.15.33:0/1370012999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ba8101790 0x7f1ba8198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.183+0000 7f1bade37640 1 -- 172.21.15.33:0/1370012999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ba8102150 msgr2=0x7f1ba8198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:00.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.183+0000 7f1bade37640 1 --2- 172.21.15.33:0/1370012999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ba8102150 0x7f1ba8198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.183+0000 7f1bade37640 1 -- 172.21.15.33:0/1370012999 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ba819d010 con 0x7f1ba81077a0 2024-03-19T03:03:00.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.183+0000 7f1bade37640 1 --2- 172.21.15.33:0/1370012999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ba81077a0 0x7f1ba819c8e0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f1ba000eb10 tx=0x7f1ba000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:00.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.183+0000 7f1b9e7fc640 1 -- 172.21.15.33:0/1370012999 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba0004b20 con 0x7f1ba81077a0 2024-03-19T03:03:00.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.183+0000 7f1b9e7fc640 1 -- 172.21.15.33:0/1370012999 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ba0004cc0 con 0x7f1ba81077a0 2024-03-19T03:03:00.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.183+0000 7f1baf89a640 1 -- 172.21.15.33:0/1370012999 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ba819d300 con 0x7f1ba81077a0 2024-03-19T03:03:00.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.183+0000 7f1b9e7fc640 1 -- 172.21.15.33:0/1370012999 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba0010690 con 0x7f1ba81077a0 2024-03-19T03:03:00.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.183+0000 7f1baf89a640 1 -- 172.21.15.33:0/1370012999 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ba81a4b90 con 0x7f1ba81077a0 2024-03-19T03:03:00.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.186+0000 7f1b9e7fc640 1 -- 172.21.15.33:0/1370012999 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f1ba0005020 con 0x7f1ba81077a0 2024-03-19T03:03:00.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.186+0000 7f1bb2566740 1 Processor -- start 2024-03-19T03:03:00.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.186+0000 7f1bb2566740 1 -- start start 2024-03-19T03:03:00.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.186+0000 7f1bb2566740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604e59492d0 0x5604e59496b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.186+0000 7f1bb2566740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5604e5949bf0 0x5604e5952290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.186+0000 7f1bb2566740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5604e59527d0 0x5604e5954bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.186+0000 7f1bb2566740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5604e58ac4b0 con 0x5604e59492d0 2024-03-19T03:03:00.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.186+0000 7f1bb2566740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5604e5854d70 con 0x5604e5949bf0 2024-03-19T03:03:00.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.186+0000 7f1bb2566740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5604e58a8550 con 0x5604e59527d0 2024-03-19T03:03:00.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.186+0000 7f1bace35640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604e59492d0 0x5604e59496b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.186+0000 7f1bace35640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604e59492d0 0x5604e59496b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57102/0 (socket says 172.21.15.33:57102) 2024-03-19T03:03:00.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.186+0000 7f1bace35640 1 -- 172.21.15.33:0/3284053853 learned_addr learned my addr 172.21.15.33:0/3284053853 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:00.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.186+0000 7f1bad636640 1 --2- 172.21.15.33:0/3284053853 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5604e5949bf0 0x5604e5952290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.186+0000 7f1bace35640 1 --2- 172.21.15.33:0/3284053853 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5604e59527d0 0x5604e5954bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.187+0000 7f1b9e7fc640 1 --2- 172.21.15.33:0/1370012999 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f1b940a29f0 0x7f1b940a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.187+0000 7f1bace35640 1 -- 172.21.15.33:0/3284053853 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5604e59527d0 msgr2=0x5604e5954bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:00.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.187+0000 7f1bace35640 1 --2- 172.21.15.33:0/3284053853 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5604e59527d0 0x5604e5954bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.187+0000 7f1bace35640 1 -- 172.21.15.33:0/3284053853 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5604e5949bf0 msgr2=0x5604e5952290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:00.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.187+0000 7f1bace35640 1 --2- 172.21.15.33:0/3284053853 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5604e5949bf0 0x5604e5952290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.187+0000 7f1bace35640 1 -- 172.21.15.33:0/3284053853 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604e5877210 con 0x5604e59492d0 2024-03-19T03:03:00.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.187+0000 7f1bade37640 1 --2- 172.21.15.33:0/1370012999 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f1b940a29f0 0x7f1b940a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.187+0000 7f1bace35640 1 --2- 172.21.15.33:0/3284053853 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604e59492d0 0x5604e59496b0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f1b98012530 tx=0x7f1b9800b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5123b1d0b88aa7d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:00.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.187+0000 7f1b9e7fc640 1 -- 172.21.15.33:0/1370012999 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f1ba00cf720 con 0x7f1ba81077a0 2024-03-19T03:03:00.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.187+0000 7f1b937fe640 1 -- 172.21.15.33:0/3284053853 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b98005020 con 0x5604e59492d0 2024-03-19T03:03:00.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.187+0000 7f1b937fe640 1 -- 172.21.15.33:0/3284053853 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1b98005670 con 0x5604e59492d0 2024-03-19T03:03:00.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.187+0000 7f1b937fe640 1 -- 172.21.15.33:0/3284053853 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b98010460 con 0x5604e59492d0 2024-03-19T03:03:00.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.188+0000 7f1bb2566740 1 -- 172.21.15.33:0/3284053853 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604e59492d0 msgr2=0x5604e59496b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:00.195 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.188+0000 7f1bade37640 1 --2- 172.21.15.33:0/1370012999 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f1b940a29f0 0x7f1b940a4eb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1ba009b6b0 tx=0x7f1ba006a7c0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:00.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.188+0000 7f1bb2566740 1 --2- 172.21.15.33:0/3284053853 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604e59492d0 0x5604e59496b0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f1b98012530 tx=0x7f1b9800b650 comp rx=0 tx=0).stop 2024-03-19T03:03:00.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.188+0000 7f1bb2566740 1 -- 172.21.15.33:0/3284053853 shutdown_connections 2024-03-19T03:03:00.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.188+0000 7f1bb2566740 1 --2- 172.21.15.33:0/3284053853 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5604e59527d0 0x5604e5954bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.188+0000 7f1bb2566740 1 --2- 172.21.15.33:0/3284053853 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5604e5949bf0 0x5604e5952290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.188+0000 7f1bb2566740 1 --2- 172.21.15.33:0/3284053853 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604e59492d0 0x5604e59496b0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.188+0000 7f1bb2566740 1 -- 172.21.15.33:0/3284053853 >> 172.21.15.33:0/3284053853 conn(0x5604e5943a00 msgr2=0x5604e5943de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:00.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.188+0000 7f1bb2566740 1 -- 172.21.15.33:0/3284053853 shutdown_connections 2024-03-19T03:03:00.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.188+0000 7f1bb2566740 1 -- 172.21.15.33:0/3284053853 wait complete. 2024-03-19T03:03:00.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.188+0000 7f1bb2566740 1 -- start start 2024-03-19T03:03:00.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.188+0000 7f1bb2566740 1 Processor -- start 2024-03-19T03:03:00.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.189+0000 7f1bb2566740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5604e59492d0 0x5604e594f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.189+0000 7f1bb2566740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5604e5949bf0 0x5604e594fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.189+0000 7f1bb2566740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604e59527d0 0x5604e5950250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.189+0000 7f1bb2566740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5604e59555c0 con 0x5604e59527d0 2024-03-19T03:03:00.196 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.189+0000 7f1bb2566740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5604e5868310 con 0x5604e5949bf0 2024-03-19T03:03:00.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.189+0000 7f1bb2566740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5604e5955740 con 0x5604e59492d0 2024-03-19T03:03:00.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.189+0000 7f1bb2566740 10 client.0 _ll_register_callbacks cb 0x5604e585c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:03:00.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.189+0000 7f1bace35640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604e59527d0 0x5604e5950250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.189+0000 7f1bad636640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5604e5949bf0 0x5604e594fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.189+0000 7f1bace35640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604e59527d0 0x5604e5950250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57108/0 (socket says 172.21.15.33:57108) 2024-03-19T03:03:00.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.189+0000 7f1bace35640 1 -- 172.21.15.33:0/789899154 learned_addr learned my addr 172.21.15.33:0/789899154 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:00.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.189+0000 7f1bace35640 1 --2- 172.21.15.33:0/789899154 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5604e59492d0 0x5604e594f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.189+0000 7f1bace35640 1 -- 172.21.15.33:0/789899154 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5604e59492d0 msgr2=0x5604e594f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:00.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.189+0000 7f1bace35640 1 --2- 172.21.15.33:0/789899154 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5604e59492d0 0x5604e594f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.189+0000 7f1bace35640 1 -- 172.21.15.33:0/789899154 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5604e5949bf0 msgr2=0x5604e594fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:00.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.189+0000 7f1bace35640 1 --2- 172.21.15.33:0/789899154 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5604e5949bf0 0x5604e594fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.189+0000 7f1bace35640 1 -- 172.21.15.33:0/789899154 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604e55c03f0 con 0x5604e59527d0 2024-03-19T03:03:00.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.189+0000 7f1bace35640 1 --2- 172.21.15.33:0/789899154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604e59527d0 0x5604e5950250 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f1b9800b9f0 tx=0x7f1b98004d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5102ed456a448049 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:00.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.189+0000 7f1b90ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:03:00.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.189+0000 7f1b90ff9640 1 -- 172.21.15.33:0/789899154 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b98027040 con 0x5604e59527d0 2024-03-19T03:03:00.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.189+0000 7f1b90ff9640 1 -- 172.21.15.33:0/789899154 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f1b98005290 con 0x5604e59527d0 2024-03-19T03:03:00.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.189+0000 7f1bb2566740 1 -- 172.21.15.33:0/789899154 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b98005670 con 0x5604e59527d0 2024-03-19T03:03:00.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.189+0000 7f1b90ff9640 1 -- 172.21.15.33:0/789899154 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1b98005490 con 0x5604e59527d0 2024-03-19T03:03:00.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.189+0000 7f1bb2566740 10 client.17706 fetch_fsmap learned FSMap version 25 2024-03-19T03:03:00.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.189+0000 7f1bb2566740 1 -- 172.21.15.33:0/789899154 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5604e568a7d0 con 0x5604e59527d0 2024-03-19T03:03:00.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.190+0000 7f1b90ff9640 1 -- 172.21.15.33:0/789899154 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1b98005670 con 0x5604e59527d0 2024-03-19T03:03:00.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.190+0000 7f1b90ff9640 1 -- 172.21.15.33:0/789899154 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f1b9800e490 con 0x5604e59527d0 2024-03-19T03:03:00.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.190+0000 7f1bb2566740 10 client.17706 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:03:00.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.190+0000 7f1bb2566740 10 client.17706 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:03:00.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.190+0000 7f1bb2566740 20 client.17706 populate_metadata read hostname 'smithi033' 2024-03-19T03:03:00.198 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.190+0000 7f1bb2566740 1 --2- 172.21.15.33:0/789899154 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5604e59629a0 0x5604e5964d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.190+0000 7f1bb2566740 4 client.17706 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:03:00.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.190+0000 7f1bb2566740 1 -- 172.21.15.33:0/789899154 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604e590a3e0 con 0x5604e59629a0 2024-03-19T03:03:00.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.190+0000 7f1bad636640 1 --2- 172.21.15.33:0/789899154 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5604e59629a0 0x5604e5964d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.190+0000 7f1b90ff9640 10 client.17706 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:03:00.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.190+0000 7f1bad636640 1 --2- 172.21.15.33:0/789899154 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5604e59629a0 0x5604e5964d90 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26b27c580539a46e server_cookie=d1f53057a8a027a5 in_seq=0 out_seq=0 2024-03-19T03:03:00.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.191+0000 7f1b90ff9640 1 -- 172.21.15.33:0/789899154 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5604e590a3e0 con 0x5604e59629a0 2024-03-19T03:03:00.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.191+0000 7f1b90ff9640 10 client.17706 handle_command_reply: tid=0 2024-03-19T03:03:00.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.191+0000 7f1bb2566740 1 client.17706 shutdown 2024-03-19T03:03:00.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.191+0000 7f1bb2566740 1 -- 172.21.15.33:0/789899154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604e59527d0 msgr2=0x5604e5950250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:00.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.191+0000 7f1bb2566740 1 --2- 172.21.15.33:0/789899154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604e59527d0 0x5604e5950250 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f1b9800b9f0 tx=0x7f1b98004d20 comp rx=0 tx=0).stop 2024-03-19T03:03:00.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.191+0000 7f1bb2566740 1 -- 172.21.15.33:0/789899154 shutdown_connections 2024-03-19T03:03:00.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.191+0000 7f1bb2566740 1 --2- 172.21.15.33:0/789899154 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5604e59629a0 0x5604e5964d90 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.191+0000 7f1bb2566740 1 --2- 172.21.15.33:0/789899154 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604e59527d0 0x5604e5950250 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.191+0000 7f1bb2566740 1 --2- 172.21.15.33:0/789899154 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5604e5949bf0 0x5604e594fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.199 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.191+0000 7f1b90ff9640 0 client.17706 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:03:00.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.191+0000 7f1bb2566740 1 --2- 172.21.15.33:0/789899154 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5604e59492d0 0x5604e594f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.191+0000 7f1bb2566740 1 -- 172.21.15.33:0/789899154 >> 172.21.15.33:0/789899154 conn(0x5604e5943a00 msgr2=0x5604e5945750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:00.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.191+0000 7f1bb2566740 1 -- 172.21.15.33:0/789899154 shutdown_connections 2024-03-19T03:03:00.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.192+0000 7f1bb2566740 1 -- 172.21.15.33:0/789899154 wait complete. 2024-03-19T03:03:00.200 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.192+0000 7f1bb2566740 20 client.17706 trim_cache size 0 max 16384 2024-03-19T03:03:00.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.205+0000 7f1bb2566740 1 Processor -- start 2024-03-19T03:03:00.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.205+0000 7f1bb2566740 1 -- start start 2024-03-19T03:03:00.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.205+0000 7f1bb2566740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604e59492d0 0x5604e5951f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.205+0000 7f1bb2566740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5604e5949bf0 0x5604e595e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.205+0000 7f1bb2566740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5604e59527d0 0x5604e596a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.205+0000 7f1bb2566740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba81b1650 con 0x5604e59492d0 2024-03-19T03:03:00.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.205+0000 7f1bb2566740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba81aef20 con 0x5604e59527d0 2024-03-19T03:03:00.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.205+0000 7f1bb2566740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba81acd90 con 0x5604e5949bf0 2024-03-19T03:03:00.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.205+0000 7f1bace35640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604e59492d0 0x5604e5951f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.208 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.205+0000 7f1bace35640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604e59492d0 0x5604e5951f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57124/0 (socket says 172.21.15.33:57124) 2024-03-19T03:03:00.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.205+0000 7f1bace35640 1 -- 172.21.15.33:0/2633077465 learned_addr learned my addr 172.21.15.33:0/2633077465 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:00.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.205+0000 7f1bad636640 1 --2- 172.21.15.33:0/2633077465 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5604e5949bf0 0x5604e595e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.205+0000 7f1bace35640 1 --2- 172.21.15.33:0/2633077465 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5604e59527d0 0x5604e596a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.205+0000 7f1bace35640 1 -- 172.21.15.33:0/2633077465 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5604e5949bf0 msgr2=0x5604e595e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:00.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.205+0000 7f1bace35640 1 --2- 172.21.15.33:0/2633077465 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5604e5949bf0 0x5604e595e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.205+0000 7f1bace35640 1 -- 172.21.15.33:0/2633077465 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5604e59527d0 msgr2=0x5604e596a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:00.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.205+0000 7f1bace35640 1 --2- 172.21.15.33:0/2633077465 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5604e59527d0 0x5604e596a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.205+0000 7f1bace35640 1 -- 172.21.15.33:0/2633077465 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604e5919880 con 0x5604e59492d0 2024-03-19T03:03:00.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.205+0000 7f1bace35640 1 --2- 172.21.15.33:0/2633077465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604e59492d0 0x5604e5951f00 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f1b9800c010 tx=0x7f1b98001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bc23d5aab8d147f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:00.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.206+0000 7f1b92ffd640 1 -- 172.21.15.33:0/2633077465 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b9800bc90 con 0x5604e59492d0 2024-03-19T03:03:00.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.206+0000 7f1b92ffd640 1 -- 172.21.15.33:0/2633077465 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1b98037070 con 0x5604e59492d0 2024-03-19T03:03:00.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.206+0000 7f1b92ffd640 1 -- 172.21.15.33:0/2633077465 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b9800e490 con 0x5604e59492d0 2024-03-19T03:03:00.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.206+0000 7f1bb2566740 1 -- 172.21.15.33:0/2633077465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604e59492d0 msgr2=0x5604e5951f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:00.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.206+0000 7f1bb2566740 1 --2- 172.21.15.33:0/2633077465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604e59492d0 0x5604e5951f00 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f1b9800c010 tx=0x7f1b98001e70 comp rx=0 tx=0).stop 2024-03-19T03:03:00.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.206+0000 7f1bb2566740 1 -- 172.21.15.33:0/2633077465 shutdown_connections 2024-03-19T03:03:00.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.206+0000 7f1bb2566740 1 --2- 172.21.15.33:0/2633077465 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5604e5949bf0 0x5604e595e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.206+0000 7f1bb2566740 1 --2- 172.21.15.33:0/2633077465 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5604e59527d0 0x5604e596a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.206+0000 7f1bb2566740 1 --2- 172.21.15.33:0/2633077465 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604e59492d0 0x5604e5951f00 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.206+0000 7f1bb2566740 1 -- 172.21.15.33:0/2633077465 >> 172.21.15.33:0/2633077465 conn(0x5604e5943a00 msgr2=0x5604e5942ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:00.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.206+0000 7f1bb2566740 1 -- 172.21.15.33:0/2633077465 shutdown_connections 2024-03-19T03:03:00.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.206+0000 7f1bb2566740 1 -- 172.21.15.33:0/2633077465 wait complete. 2024-03-19T03:03:00.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.207+0000 7f1bb2566740 1 -- start start 2024-03-19T03:03:00.210 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.207+0000 7f1bb2566740 1 Processor -- start 2024-03-19T03:03:00.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.207+0000 7f1bb2566740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5604e59492d0 0x5604e594c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.207+0000 7f1bb2566740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5604e5949bf0 0x5604e594c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.207+0000 7f1bb2566740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604e59527d0 0x5604e594cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.207+0000 7f1bb2566740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba81ad3e0 con 0x5604e59527d0 2024-03-19T03:03:00.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.207+0000 7f1bb2566740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba81ae850 con 0x5604e59492d0 2024-03-19T03:03:00.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.207+0000 7f1bb2566740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba81adab0 con 0x5604e5949bf0 2024-03-19T03:03:00.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.207+0000 7f1bb2566740 10 client.0 _ll_register_callbacks cb 0x7f1ba4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:03:00.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.207+0000 7f1bace35640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604e59527d0 0x5604e594cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.207+0000 7f1bace35640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604e59527d0 0x5604e594cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57132/0 (socket says 172.21.15.33:57132) 2024-03-19T03:03:00.214 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.207+0000 7f1bace35640 1 -- 172.21.15.33:0/3709416832 learned_addr learned my addr 172.21.15.33:0/3709416832 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:00.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.207+0000 7f1bace35640 1 --2- 172.21.15.33:0/3709416832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5604e59492d0 0x5604e594c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.207+0000 7f1bad636640 1 --2- 172.21.15.33:0/3709416832 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5604e5949bf0 0x5604e594c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.208+0000 7f1bace35640 1 -- 172.21.15.33:0/3709416832 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5604e5949bf0 msgr2=0x5604e594c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:00.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.208+0000 7f1bace35640 1 --2- 172.21.15.33:0/3709416832 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5604e5949bf0 0x5604e594c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.208+0000 7f1bace35640 1 -- 172.21.15.33:0/3709416832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5604e59492d0 msgr2=0x5604e594c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:00.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.208+0000 7f1bace35640 1 --2- 172.21.15.33:0/3709416832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5604e59492d0 0x5604e594c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.208+0000 7f1bace35640 1 -- 172.21.15.33:0/3709416832 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604e59cbe30 con 0x5604e59527d0 2024-03-19T03:03:00.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.208+0000 7f1bace35640 1 --2- 172.21.15.33:0/3709416832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604e59527d0 0x5604e594cda0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f1b9800b9a0 tx=0x7f1b9800b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff5879fb96030ee7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:00.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.208+0000 7f1b90ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:03:00.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.208+0000 7f1b90ff9640 1 -- 172.21.15.33:0/3709416832 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b98005730 con 0x5604e59527d0 2024-03-19T03:03:00.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.208+0000 7f1b90ff9640 1 -- 172.21.15.33:0/3709416832 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f1b9800e490 con 0x5604e59527d0 2024-03-19T03:03:00.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.208+0000 7f1bb2566740 1 -- 172.21.15.33:0/3709416832 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b98037070 con 0x5604e59527d0 2024-03-19T03:03:00.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.208+0000 7f1b90ff9640 1 -- 172.21.15.33:0/3709416832 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1b9800e690 con 0x5604e59527d0 2024-03-19T03:03:00.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.208+0000 7f1bb2566740 10 client.17712 fetch_fsmap learned FSMap version 25 2024-03-19T03:03:00.215 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.208+0000 7f1bb2566740 1 -- 172.21.15.33:0/3709416832 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5604e59cd0d0 con 0x5604e59527d0 2024-03-19T03:03:00.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.208+0000 7f1b90ff9640 1 -- 172.21.15.33:0/3709416832 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1b98037070 con 0x5604e59527d0 2024-03-19T03:03:00.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.208+0000 7f1b90ff9640 1 -- 172.21.15.33:0/3709416832 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f1b9800eaa0 con 0x5604e59527d0 2024-03-19T03:03:00.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.208+0000 7f1bb2566740 10 client.17712 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:03:00.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.208+0000 7f1bb2566740 10 client.17712 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:03:00.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.208+0000 7f1bb2566740 20 client.17712 populate_metadata read hostname 'smithi033' 2024-03-19T03:03:00.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.208+0000 7f1bb2566740 1 --2- 172.21.15.33:0/3709416832 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5604e59cd730 0x5604e59cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:00.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.208+0000 7f1bb2566740 4 client.17712 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T03:03:00.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.208+0000 7f1bb2566740 1 -- 172.21.15.33:0/3709416832 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5604e58b99f0 con 0x5604e59cd730 2024-03-19T03:03:00.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.209+0000 7f1bad636640 1 --2- 172.21.15.33:0/3709416832 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5604e59cd730 0x5604e59cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:00.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.209+0000 7f1bad636640 1 --2- 172.21.15.33:0/3709416832 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5604e59cd730 0x5604e59cfb50 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27e40d3cbab27534 server_cookie=1618d0bf15fe6c87 in_seq=0 out_seq=0 2024-03-19T03:03:00.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.209+0000 7f1b90ff9640 10 client.17712 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:03:00.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.209+0000 7f1b90ff9640 1 -- 172.21.15.33:0/3709416832 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5604e58b99f0 con 0x5604e59cd730 2024-03-19T03:03:00.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.209+0000 7f1b90ff9640 10 client.17712 handle_command_reply: tid=0 2024-03-19T03:03:00.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.209+0000 7f1bb2566740 1 client.17712 shutdown 2024-03-19T03:03:00.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.209+0000 7f1bb2566740 1 -- 172.21.15.33:0/3709416832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604e59527d0 msgr2=0x5604e594cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:00.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.209+0000 7f1bb2566740 1 --2- 172.21.15.33:0/3709416832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604e59527d0 0x5604e594cda0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f1b9800b9a0 tx=0x7f1b9800b810 comp rx=0 tx=0).stop 2024-03-19T03:03:00.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.209+0000 7f1bb2566740 1 -- 172.21.15.33:0/3709416832 shutdown_connections 2024-03-19T03:03:00.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.209+0000 7f1bb2566740 1 --2- 172.21.15.33:0/3709416832 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5604e59cd730 0x5604e59cfb50 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.209+0000 7f1bb2566740 1 --2- 172.21.15.33:0/3709416832 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5604e5949bf0 0x5604e594c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.209+0000 7f1bb2566740 1 --2- 172.21.15.33:0/3709416832 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5604e59527d0 0x5604e594cda0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.209+0000 7f1b90ff9640 0 client.17712 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:03:00.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.209+0000 7f1bb2566740 1 --2- 172.21.15.33:0/3709416832 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5604e59492d0 0x5604e594c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.209+0000 7f1bb2566740 1 -- 172.21.15.33:0/3709416832 >> 172.21.15.33:0/3709416832 conn(0x5604e5943a00 msgr2=0x5604e594f790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:00.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.209+0000 7f1bb2566740 1 -- 172.21.15.33:0/3709416832 shutdown_connections 2024-03-19T03:03:00.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.209+0000 7f1bb2566740 1 -- 172.21.15.33:0/3709416832 wait complete. 2024-03-19T03:03:00.217 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.209+0000 7f1bb2566740 20 client.17712 trim_cache size 0 max 16384 2024-03-19T03:03:00.217 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T03:03:00.218 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.210+0000 7f1baf89a640 1 -- 172.21.15.33:0/1370012999 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f1b940a29f0 msgr2=0x7f1b940a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:00.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.210+0000 7f1baf89a640 1 --2- 172.21.15.33:0/1370012999 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f1b940a29f0 0x7f1b940a4eb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1ba009b6b0 tx=0x7f1ba006a7c0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.210+0000 7f1baf89a640 1 -- 172.21.15.33:0/1370012999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ba81077a0 msgr2=0x7f1ba819c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:00.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.210+0000 7f1baf89a640 1 --2- 172.21.15.33:0/1370012999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ba81077a0 0x7f1ba819c8e0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f1ba000eb10 tx=0x7f1ba000c2d0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.210+0000 7f1baf89a640 1 -- 172.21.15.33:0/1370012999 shutdown_connections 2024-03-19T03:03:00.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.210+0000 7f1baf89a640 1 --2- 172.21.15.33:0/1370012999 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f1b940a29f0 0x7f1b940a4eb0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.210+0000 7f1baf89a640 1 --2- 172.21.15.33:0/1370012999 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f1ba81077a0 0x7f1ba819c8e0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.210+0000 7f1baf89a640 1 --2- 172.21.15.33:0/1370012999 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ba8102150 0x7f1ba8198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.210+0000 7f1baf89a640 1 --2- 172.21.15.33:0/1370012999 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ba8101790 0x7f1ba8198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:00.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.210+0000 7f1baf89a640 1 -- 172.21.15.33:0/1370012999 >> 172.21.15.33:0/1370012999 conn(0x7f1ba80fd2c0 msgr2=0x7f1ba80fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:00.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.210+0000 7f1baf89a640 1 -- 172.21.15.33:0/1370012999 shutdown_connections 2024-03-19T03:03:00.219 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:00.210+0000 7f1baf89a640 1 -- 172.21.15.33:0/1370012999 wait complete. 2024-03-19T03:03:00.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:00 smithi155 ceph-mon[35804]: pgmap v295: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:03:00.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:00 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3717873378' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:03:00.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:00 smithi033 ceph-mon[31973]: pgmap v295: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:03:00.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:00 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3717873378' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:03:00.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:00 smithi096 ceph-mon[35946]: pgmap v295: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:03:00.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:00 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3717873378' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:03:01.222 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-19T03:03:01.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:03:01.223 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-19T03:03:01.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.341+0000 7f4c0e6b2640 1 -- 172.21.15.33:0/2152010801 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4c0810be30 msgr2=0x7f4c0810e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:01.344 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.341+0000 7f4c0e6b2640 1 --2- 172.21.15.33:0/2152010801 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4c0810be30 0x7f4c0810e260 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f4c00009ca0 tx=0x7f4c0002c180 comp rx=0 tx=0).stop 2024-03-19T03:03:01.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.341+0000 7f4c0e6b2640 1 -- 172.21.15.33:0/2152010801 shutdown_connections 2024-03-19T03:03:01.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.341+0000 7f4c0e6b2640 1 --2- 172.21.15.33:0/2152010801 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4c0810be30 0x7f4c0810e260 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.341+0000 7f4c0e6b2640 1 --2- 172.21.15.33:0/2152010801 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c08103380 0x7f4c0810b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.341+0000 7f4c0e6b2640 1 --2- 172.21.15.33:0/2152010801 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4c08102a60 0x7f4c08102e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.341+0000 7f4c0e6b2640 1 -- 172.21.15.33:0/2152010801 >> 172.21.15.33:0/2152010801 conn(0x7f4c08076aa0 msgr2=0x7f4c08076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:01.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.341+0000 7f4c0e6b2640 1 -- 172.21.15.33:0/2152010801 shutdown_connections 2024-03-19T03:03:01.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.341+0000 7f4c0e6b2640 1 -- 172.21.15.33:0/2152010801 wait complete. 2024-03-19T03:03:01.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.341+0000 7f4c0e6b2640 1 Processor -- start 2024-03-19T03:03:01.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.342+0000 7f4c0e6b2640 1 -- start start 2024-03-19T03:03:01.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.342+0000 7f4c0e6b2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c08102a60 0x7f4c08198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.342+0000 7f4c0e6b2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4c08103380 0x7f4c081985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.342+0000 7f4c0e6b2640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4c0810be30 0x7f4c0819c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.342+0000 7f4c0e6b2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4c08111a70 con 0x7f4c08103380 2024-03-19T03:03:01.345 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.342+0000 7f4c0e6b2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4c081118f0 con 0x7f4c08102a60 2024-03-19T03:03:01.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.342+0000 7f4c0e6b2640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4c08111bf0 con 0x7f4c0810be30 2024-03-19T03:03:01.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.342+0000 7f4c077fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4c08103380 0x7f4c081985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.342+0000 7f4c077fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4c08103380 0x7f4c081985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57146/0 (socket says 172.21.15.33:57146) 2024-03-19T03:03:01.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.342+0000 7f4c077fe640 1 -- 172.21.15.33:0/2840512495 learned_addr learned my addr 172.21.15.33:0/2840512495 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:01.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.342+0000 7f4c07fff640 1 --2- 172.21.15.33:0/2840512495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c08102a60 0x7f4c08198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.342+0000 7f4c0cc4f640 1 --2- 172.21.15.33:0/2840512495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4c0810be30 0x7f4c0819c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.342+0000 7f4c077fe640 1 -- 172.21.15.33:0/2840512495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4c0810be30 msgr2=0x7f4c0819c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:01.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.342+0000 7f4c077fe640 1 --2- 172.21.15.33:0/2840512495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4c0810be30 0x7f4c0819c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.342+0000 7f4c077fe640 1 -- 172.21.15.33:0/2840512495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c08102a60 msgr2=0x7f4c08198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:01.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.342+0000 7f4c077fe640 1 --2- 172.21.15.33:0/2840512495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c08102a60 0x7f4c08198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.342+0000 7f4c077fe640 1 -- 172.21.15.33:0/2840512495 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c0819d060 con 0x7f4c08103380 2024-03-19T03:03:01.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.343+0000 7f4c077fe640 1 --2- 172.21.15.33:0/2840512495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4c08103380 0x7f4c081985b0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc00e9f0 tx=0x7f4bfc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:01.346 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.343+0000 7f4c057fa640 1 -- 172.21.15.33:0/2840512495 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc00cd80 con 0x7f4c08103380 2024-03-19T03:03:01.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.343+0000 7f4c057fa640 1 -- 172.21.15.33:0/2840512495 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4bfc0108e0 con 0x7f4c08103380 2024-03-19T03:03:01.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.343+0000 7f4c0e6b2640 1 -- 172.21.15.33:0/2840512495 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c0819d350 con 0x7f4c08103380 2024-03-19T03:03:01.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.343+0000 7f4c057fa640 1 -- 172.21.15.33:0/2840512495 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc018920 con 0x7f4c08103380 2024-03-19T03:03:01.347 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.343+0000 7f4c0e6b2640 1 -- 172.21.15.33:0/2840512495 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c081a4be0 con 0x7f4c08103380 2024-03-19T03:03:01.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.345+0000 7f4c057fa640 1 -- 172.21.15.33:0/2840512495 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f4bfc010a90 con 0x7f4c08103380 2024-03-19T03:03:01.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.345+0000 7f4c1137e740 1 Processor -- start 2024-03-19T03:03:01.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.345+0000 7f4c1137e740 1 -- start start 2024-03-19T03:03:01.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.346+0000 7f4c1137e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8f62aa380 0x55d8f62aa760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.346+0000 7f4c057fa640 1 --2- 172.21.15.33:0/2840512495 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4bf00a2970 0x7f4bf00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.346+0000 7f4c1137e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f62aaca0 0x55d8f62b3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.350 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.346+0000 7f4c1137e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8f62b3880 0x55d8f62b5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.346+0000 7f4c1137e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d8f620e160 con 0x55d8f62b3880 2024-03-19T03:03:01.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.346+0000 7f4c1137e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55d8f61b6e10 con 0x55d8f62aaca0 2024-03-19T03:03:01.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.346+0000 7f4c1137e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d8f620a200 con 0x55d8f62aa380 2024-03-19T03:03:01.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.346+0000 7f4c077fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f62aaca0 0x55d8f62b3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.346+0000 7f4c0cc4f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8f62b3880 0x55d8f62b5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.346+0000 7f4c07fff640 1 --2- 172.21.15.33:0/2840512495 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4bf00a2970 0x7f4bf00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.346+0000 7f4c0cc4f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8f62b3880 0x55d8f62b5c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57152/0 (socket says 172.21.15.33:57152) 2024-03-19T03:03:01.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.346+0000 7f4c0cc4f640 1 -- 172.21.15.33:0/4158442050 learned_addr learned my addr 172.21.15.33:0/4158442050 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:01.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.346+0000 7f4c0cc4f640 1 --2- 172.21.15.33:0/4158442050 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8f62aa380 0x55d8f62aa760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.346+0000 7f4c057fa640 1 -- 172.21.15.33:0/2840512495 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f4bfc0cf540 con 0x7f4c08103380 2024-03-19T03:03:01.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.346+0000 7f4c077fe640 1 -- 172.21.15.33:0/4158442050 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8f62aa380 msgr2=0x55d8f62aa760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:01.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.346+0000 7f4c077fe640 1 --2- 172.21.15.33:0/4158442050 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8f62aa380 0x55d8f62aa760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.346+0000 7f4c077fe640 1 -- 172.21.15.33:0/4158442050 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8f62b3880 msgr2=0x55d8f62b5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:01.351 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.346+0000 7f4c077fe640 1 --2- 172.21.15.33:0/4158442050 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8f62b3880 0x55d8f62b5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.346+0000 7f4c077fe640 1 -- 172.21.15.33:0/4158442050 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8f5fea7e0 con 0x55d8f62aaca0 2024-03-19T03:03:01.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.346+0000 7f4c07fff640 1 --2- 172.21.15.33:0/2840512495 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4bf00a2970 0x7f4bf00a4e30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8009bd0 tx=0x7f4bf8005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:01.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.346+0000 7f4c077fe640 1 --2- 172.21.15.33:0/4158442050 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f62aaca0 0x55d8f62b3340 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc0cc860 tx=0x7f4bfc09b5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3cd5d70d0313f69 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:01.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.347+0000 7f4bee7fc640 1 -- 172.21.15.33:0/4158442050 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc010de0 con 0x55d8f62aaca0 2024-03-19T03:03:01.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.347+0000 7f4bee7fc640 1 -- 172.21.15.33:0/4158442050 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4bfc09b820 con 0x55d8f62aaca0 2024-03-19T03:03:01.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.347+0000 7f4bee7fc640 1 -- 172.21.15.33:0/4158442050 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc0ccae0 con 0x55d8f62aaca0 2024-03-19T03:03:01.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.347+0000 7f4c1137e740 1 -- 172.21.15.33:0/4158442050 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f62aaca0 msgr2=0x55d8f62b3340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:01.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.347+0000 7f4c1137e740 1 --2- 172.21.15.33:0/4158442050 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f62aaca0 0x55d8f62b3340 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc0cc860 tx=0x7f4bfc09b5f0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.347+0000 7f4c1137e740 1 -- 172.21.15.33:0/4158442050 shutdown_connections 2024-03-19T03:03:01.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.347+0000 7f4c1137e740 1 --2- 172.21.15.33:0/4158442050 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8f62b3880 0x55d8f62b5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.347+0000 7f4c1137e740 1 --2- 172.21.15.33:0/4158442050 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f62aaca0 0x55d8f62b3340 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.347+0000 7f4c1137e740 1 --2- 172.21.15.33:0/4158442050 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8f62aa380 0x55d8f62aa760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.347+0000 7f4c1137e740 1 -- 172.21.15.33:0/4158442050 >> 172.21.15.33:0/4158442050 conn(0x55d8f62a4ab0 msgr2=0x55d8f62a4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:01.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.347+0000 7f4c1137e740 1 -- 172.21.15.33:0/4158442050 shutdown_connections 2024-03-19T03:03:01.352 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.347+0000 7f4c1137e740 1 -- 172.21.15.33:0/4158442050 wait complete. 2024-03-19T03:03:01.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.348+0000 7f4c1137e740 1 -- start start 2024-03-19T03:03:01.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.348+0000 7f4c1137e740 1 Processor -- start 2024-03-19T03:03:01.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.348+0000 7f4c1137e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8f62aa380 0x55d8f62b0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.348+0000 7f4c1137e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f62aaca0 0x55d8f62b0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.353 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.348+0000 7f4c1137e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8f62b3880 0x55d8f62b11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.348+0000 7f4c1137e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55d8f62b6610 con 0x55d8f62b3880 2024-03-19T03:03:01.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.348+0000 7f4c1137e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55d8f61ca1a0 con 0x55d8f62aaca0 2024-03-19T03:03:01.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.348+0000 7f4c1137e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55d8f62b6790 con 0x55d8f62aa380 2024-03-19T03:03:01.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.348+0000 7f4c1137e740 10 client.0 _ll_register_callbacks cb 0x55d8f5f21c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:03:01.356 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.348+0000 7f4c0cc4f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8f62b3880 0x55d8f62b11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.348+0000 7f4c07fff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8f62aa380 0x55d8f62b0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.348+0000 7f4c0cc4f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8f62b3880 0x55d8f62b11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57154/0 (socket says 172.21.15.33:57154) 2024-03-19T03:03:01.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.348+0000 7f4c0cc4f640 1 -- 172.21.15.33:0/3764127612 learned_addr learned my addr 172.21.15.33:0/3764127612 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:01.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.348+0000 7f4c077fe640 1 --2- 172.21.15.33:0/3764127612 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f62aaca0 0x55d8f62b0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.349+0000 7f4c0cc4f640 1 -- 172.21.15.33:0/3764127612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8f62aa380 msgr2=0x55d8f62b0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:01.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.349+0000 7f4c0cc4f640 1 --2- 172.21.15.33:0/3764127612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8f62aa380 0x55d8f62b0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.349+0000 7f4c0cc4f640 1 -- 172.21.15.33:0/3764127612 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f62aaca0 msgr2=0x55d8f62b0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:01.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.349+0000 7f4c0cc4f640 1 --2- 172.21.15.33:0/3764127612 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f62aaca0 0x55d8f62b0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.349+0000 7f4c0cc4f640 1 -- 172.21.15.33:0/3764127612 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8f604af70 con 0x55d8f62b3880 2024-03-19T03:03:01.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.349+0000 7f4c07fff640 1 --2- 172.21.15.33:0/3764127612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8f62aa380 0x55d8f62b0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:01.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.349+0000 7f4c0cc4f640 1 --2- 172.21.15.33:0/3764127612 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8f62b3880 0x55d8f62b11f0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f4c00005cc0 tx=0x7f4c00032cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11db883280278a11 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:01.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.349+0000 7f4be77fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:03:01.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.349+0000 7f4be77fe640 1 -- 172.21.15.33:0/3764127612 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c00039040 con 0x55d8f62b3880 2024-03-19T03:03:01.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.349+0000 7f4be77fe640 1 -- 172.21.15.33:0/3764127612 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f4c00034440 con 0x55d8f62b3880 2024-03-19T03:03:01.357 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.349+0000 7f4c1137e740 1 -- 172.21.15.33:0/3764127612 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bfc09b820 con 0x55d8f62b3880 2024-03-19T03:03:01.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.349+0000 7f4be77fe640 1 -- 172.21.15.33:0/3764127612 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4c00053050 con 0x55d8f62b3880 2024-03-19T03:03:01.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.349+0000 7f4c1137e740 10 client.17724 fetch_fsmap learned FSMap version 25 2024-03-19T03:03:01.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.349+0000 7f4c1137e740 1 -- 172.21.15.33:0/3764127612 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d8f5ee2170 con 0x55d8f62b3880 2024-03-19T03:03:01.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.349+0000 7f4be77fe640 1 -- 172.21.15.33:0/3764127612 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4bfc09b820 con 0x55d8f62b3880 2024-03-19T03:03:01.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.349+0000 7f4be77fe640 1 -- 172.21.15.33:0/3764127612 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f4c0003f040 con 0x55d8f62b3880 2024-03-19T03:03:01.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.350+0000 7f4c1137e740 10 client.17724 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:03:01.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.350+0000 7f4c1137e740 10 client.17724 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:03:01.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.350+0000 7f4c1137e740 20 client.17724 populate_metadata read hostname 'smithi033' 2024-03-19T03:03:01.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.350+0000 7f4c1137e740 1 --2- 172.21.15.33:0/3764127612 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55d8f62c3b00 0x55d8f62c5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.350+0000 7f4c1137e740 4 client.17724 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:03:01.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.350+0000 7f4c1137e740 1 -- 172.21.15.33:0/3764127612 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d8f621b6a0 con 0x55d8f62c3b00 2024-03-19T03:03:01.358 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.350+0000 7f4c07fff640 1 --2- 172.21.15.33:0/3764127612 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55d8f62c3b00 0x55d8f62c5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.350+0000 7f4c07fff640 1 --2- 172.21.15.33:0/3764127612 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55d8f62c3b00 0x55d8f62c5f60 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f636c2253154819 server_cookie=c0e06e2e1944b9a1 in_seq=0 out_seq=0 2024-03-19T03:03:01.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.350+0000 7f4be77fe640 10 client.17724 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:03:01.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.351+0000 7f4be77fe640 1 -- 172.21.15.33:0/3764127612 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d8f621b6a0 con 0x55d8f62c3b00 2024-03-19T03:03:01.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.351+0000 7f4be77fe640 10 client.17724 handle_command_reply: tid=0 2024-03-19T03:03:01.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.351+0000 7f4c1137e740 1 client.17724 shutdown 2024-03-19T03:03:01.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.351+0000 7f4c1137e740 1 -- 172.21.15.33:0/3764127612 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8f62b3880 msgr2=0x55d8f62b11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:01.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.351+0000 7f4c1137e740 1 --2- 172.21.15.33:0/3764127612 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8f62b3880 0x55d8f62b11f0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f4c00005cc0 tx=0x7f4c00032cd0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.351+0000 7f4c1137e740 1 -- 172.21.15.33:0/3764127612 shutdown_connections 2024-03-19T03:03:01.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.351+0000 7f4c1137e740 1 --2- 172.21.15.33:0/3764127612 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55d8f62c3b00 0x55d8f62c5f60 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.351+0000 7f4c1137e740 1 --2- 172.21.15.33:0/3764127612 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8f62b3880 0x55d8f62b11f0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.351+0000 7f4c1137e740 1 --2- 172.21.15.33:0/3764127612 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f62aaca0 0x55d8f62b0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.351+0000 7f4c1137e740 1 --2- 172.21.15.33:0/3764127612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8f62aa380 0x55d8f62b0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.351+0000 7f4be77fe640 0 client.17724 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:03:01.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.351+0000 7f4c1137e740 1 -- 172.21.15.33:0/3764127612 >> 172.21.15.33:0/3764127612 conn(0x55d8f62a4ab0 msgr2=0x55d8f62a6800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:01.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.351+0000 7f4c1137e740 1 -- 172.21.15.33:0/3764127612 shutdown_connections 2024-03-19T03:03:01.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.351+0000 7f4c1137e740 1 -- 172.21.15.33:0/3764127612 wait complete. 2024-03-19T03:03:01.360 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.351+0000 7f4c1137e740 20 client.17724 trim_cache size 0 max 16384 2024-03-19T03:03:01.372 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.369+0000 7f4c1137e740 1 Processor -- start 2024-03-19T03:03:01.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.369+0000 7f4c1137e740 1 -- start start 2024-03-19T03:03:01.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.369+0000 7f4c1137e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f62aa380 0x55d8f62b2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.369+0000 7f4c1137e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8f62aaca0 0x55d8f62bf410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.369+0000 7f4c1137e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8f62b3880 0x55d8f62cb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.369+0000 7f4c1137e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4c081b1540 con 0x55d8f62b3880 2024-03-19T03:03:01.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.369+0000 7f4c1137e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4c081aee10 con 0x55d8f62aa380 2024-03-19T03:03:01.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.369+0000 7f4c1137e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4c081acc80 con 0x55d8f62aaca0 2024-03-19T03:03:01.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.369+0000 7f4c0cc4f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8f62b3880 0x55d8f62cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.369+0000 7f4c0cc4f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8f62b3880 0x55d8f62cb880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57158/0 (socket says 172.21.15.33:57158) 2024-03-19T03:03:01.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.369+0000 7f4c0cc4f640 1 -- 172.21.15.33:0/7813496 learned_addr learned my addr 172.21.15.33:0/7813496 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:01.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.369+0000 7f4c07fff640 1 --2- 172.21.15.33:0/7813496 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f62aa380 0x55d8f62b2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.369+0000 7f4c077fe640 1 --2- 172.21.15.33:0/7813496 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8f62aaca0 0x55d8f62bf410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.373 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.370+0000 7f4c0cc4f640 1 -- 172.21.15.33:0/7813496 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8f62aaca0 msgr2=0x55d8f62bf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:01.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.370+0000 7f4c0cc4f640 1 --2- 172.21.15.33:0/7813496 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8f62aaca0 0x55d8f62bf410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.370+0000 7f4c0cc4f640 1 -- 172.21.15.33:0/7813496 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f62aa380 msgr2=0x55d8f62b2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:01.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.370+0000 7f4c0cc4f640 1 --2- 172.21.15.33:0/7813496 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f62aa380 0x55d8f62b2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.370+0000 7f4c0cc4f640 1 -- 172.21.15.33:0/7813496 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8f62aece0 con 0x55d8f62b3880 2024-03-19T03:03:01.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.370+0000 7f4c077fe640 1 --2- 172.21.15.33:0/7813496 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8f62aaca0 0x55d8f62bf410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:03:01.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.370+0000 7f4c0cc4f640 1 --2- 172.21.15.33:0/7813496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8f62b3880 0x55d8f62cb880 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f4c0002e010 tx=0x7f4c00005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee98d06c2ad74a18 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:01.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.370+0000 7f4bedffb640 1 -- 172.21.15.33:0/7813496 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c00039040 con 0x55d8f62b3880 2024-03-19T03:03:01.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.370+0000 7f4bedffb640 1 -- 172.21.15.33:0/7813496 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4c00048020 con 0x55d8f62b3880 2024-03-19T03:03:01.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.370+0000 7f4bedffb640 1 -- 172.21.15.33:0/7813496 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c00034440 con 0x55d8f62b3880 2024-03-19T03:03:01.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.370+0000 7f4c1137e740 1 -- 172.21.15.33:0/7813496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8f62b3880 msgr2=0x55d8f62cb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:01.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.370+0000 7f4c1137e740 1 --2- 172.21.15.33:0/7813496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8f62b3880 0x55d8f62cb880 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f4c0002e010 tx=0x7f4c00005c50 comp rx=0 tx=0).stop 2024-03-19T03:03:01.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.371+0000 7f4c1137e740 1 -- 172.21.15.33:0/7813496 shutdown_connections 2024-03-19T03:03:01.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.371+0000 7f4c1137e740 1 --2- 172.21.15.33:0/7813496 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8f62aaca0 0x55d8f62bf410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.371+0000 7f4c1137e740 1 --2- 172.21.15.33:0/7813496 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8f62b3880 0x55d8f62cb880 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.374 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.371+0000 7f4c1137e740 1 --2- 172.21.15.33:0/7813496 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f62aa380 0x55d8f62b2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.371+0000 7f4c1137e740 1 -- 172.21.15.33:0/7813496 >> 172.21.15.33:0/7813496 conn(0x55d8f62a4ab0 msgr2=0x55d8f62b0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:01.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.371+0000 7f4c1137e740 1 -- 172.21.15.33:0/7813496 shutdown_connections 2024-03-19T03:03:01.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.371+0000 7f4c1137e740 1 -- 172.21.15.33:0/7813496 wait complete. 2024-03-19T03:03:01.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.371+0000 7f4c1137e740 1 -- start start 2024-03-19T03:03:01.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.371+0000 7f4c1137e740 1 Processor -- start 2024-03-19T03:03:01.375 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.371+0000 7f4c1137e740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8f62aa380 0x55d8f62adbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.371+0000 7f4c1137e740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f62aaca0 0x55d8f62ae120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.378 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.371+0000 7f4c1137e740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8f62b3880 0x55d8f6322e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.371+0000 7f4c1137e740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4c081ad2d0 con 0x55d8f62aa380 2024-03-19T03:03:01.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.371+0000 7f4c1137e740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4c081ae740 con 0x55d8f62aaca0 2024-03-19T03:03:01.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.371+0000 7f4c1137e740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4c081ad9a0 con 0x55d8f62b3880 2024-03-19T03:03:01.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.371+0000 7f4c1137e740 10 client.0 _ll_register_callbacks cb 0x7f4c00009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:03:01.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.371+0000 7f4c07fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8f62aa380 0x55d8f62adbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.371+0000 7f4c07fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8f62aa380 0x55d8f62adbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57162/0 (socket says 172.21.15.33:57162) 2024-03-19T03:03:01.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.371+0000 7f4c07fff640 1 -- 172.21.15.33:0/873906203 learned_addr learned my addr 172.21.15.33:0/873906203 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:01.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.372+0000 7f4c077fe640 1 --2- 172.21.15.33:0/873906203 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f62aaca0 0x55d8f62ae120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.372+0000 7f4c0cc4f640 1 --2- 172.21.15.33:0/873906203 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8f62b3880 0x55d8f6322e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.372+0000 7f4c07fff640 1 -- 172.21.15.33:0/873906203 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8f62b3880 msgr2=0x55d8f6322e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:01.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.372+0000 7f4c07fff640 1 --2- 172.21.15.33:0/873906203 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8f62b3880 0x55d8f6322e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.372+0000 7f4c07fff640 1 -- 172.21.15.33:0/873906203 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f62aaca0 msgr2=0x55d8f62ae120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:01.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.372+0000 7f4c07fff640 1 --2- 172.21.15.33:0/873906203 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f62aaca0 0x55d8f62ae120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.379 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.372+0000 7f4c07fff640 1 -- 172.21.15.33:0/873906203 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8f63234f0 con 0x55d8f62aa380 2024-03-19T03:03:01.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.372+0000 7f4c07fff640 1 --2- 172.21.15.33:0/873906203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8f62aa380 0x55d8f62adbe0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8009a90 tx=0x7f4bf8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f38025f6767be8a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:01.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.372+0000 7f4be77fe640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:03:01.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.372+0000 7f4be77fe640 1 -- 172.21.15.33:0/873906203 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf80273a0 con 0x55d8f62aa380 2024-03-19T03:03:01.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.372+0000 7f4be77fe640 1 -- 172.21.15.33:0/873906203 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f4bf8027540 con 0x55d8f62aa380 2024-03-19T03:03:01.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.372+0000 7f4c1137e740 1 -- 172.21.15.33:0/873906203 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c00048020 con 0x55d8f62aa380 2024-03-19T03:03:01.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.372+0000 7f4be77fe640 1 -- 172.21.15.33:0/873906203 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4bf8037050 con 0x55d8f62aa380 2024-03-19T03:03:01.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.372+0000 7f4c1137e740 10 client.17730 fetch_fsmap learned FSMap version 25 2024-03-19T03:03:01.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.372+0000 7f4c1137e740 1 -- 172.21.15.33:0/873906203 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d8f632a8d0 con 0x55d8f62aa380 2024-03-19T03:03:01.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.372+0000 7f4be77fe640 1 -- 172.21.15.33:0/873906203 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4c00048020 con 0x55d8f62aa380 2024-03-19T03:03:01.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.373+0000 7f4be77fe640 1 -- 172.21.15.33:0/873906203 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f4bf802f3c0 con 0x55d8f62aa380 2024-03-19T03:03:01.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.373+0000 7f4c1137e740 10 client.17730 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:03:01.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.373+0000 7f4c1137e740 10 client.17730 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:03:01.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.373+0000 7f4c1137e740 20 client.17730 populate_metadata read hostname 'smithi033' 2024-03-19T03:03:01.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.373+0000 7f4c1137e740 1 --2- 172.21.15.33:0/873906203 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55d8f632b000 0x55d8f632d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.380 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.373+0000 7f4c1137e740 4 client.17730 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-19T03:03:01.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.373+0000 7f4c1137e740 1 -- 172.21.15.33:0/873906203 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d8f62159f0 con 0x55d8f632b000 2024-03-19T03:03:01.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.373+0000 7f4c077fe640 1 --2- 172.21.15.33:0/873906203 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55d8f632b000 0x55d8f632d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.373+0000 7f4be77fe640 10 client.17730 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:03:01.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.373+0000 7f4c077fe640 1 --2- 172.21.15.33:0/873906203 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55d8f632b000 0x55d8f632d420 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdca80bb9c6c8c47 server_cookie=4079c532731cdf2f in_seq=0 out_seq=0 2024-03-19T03:03:01.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.374+0000 7f4be77fe640 1 -- 172.21.15.33:0/873906203 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d8f62159f0 con 0x55d8f632b000 2024-03-19T03:03:01.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.374+0000 7f4be77fe640 10 client.17730 handle_command_reply: tid=0 2024-03-19T03:03:01.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.374+0000 7f4c1137e740 1 client.17730 shutdown 2024-03-19T03:03:01.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.374+0000 7f4c1137e740 1 -- 172.21.15.33:0/873906203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8f62aa380 msgr2=0x55d8f62adbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:01.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.374+0000 7f4c1137e740 1 --2- 172.21.15.33:0/873906203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8f62aa380 0x55d8f62adbe0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8009a90 tx=0x7f4bf8014000 comp rx=0 tx=0).stop 2024-03-19T03:03:01.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.374+0000 7f4c1137e740 1 -- 172.21.15.33:0/873906203 shutdown_connections 2024-03-19T03:03:01.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.374+0000 7f4c1137e740 1 --2- 172.21.15.33:0/873906203 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55d8f632b000 0x55d8f632d420 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.374+0000 7f4c1137e740 1 --2- 172.21.15.33:0/873906203 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55d8f62b3880 0x55d8f6322e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.374+0000 7f4c1137e740 1 --2- 172.21.15.33:0/873906203 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55d8f62aaca0 0x55d8f62ae120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.374+0000 7f4c1137e740 1 --2- 172.21.15.33:0/873906203 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55d8f62aa380 0x55d8f62adbe0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.381 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.374+0000 7f4c1137e740 1 -- 172.21.15.33:0/873906203 >> 172.21.15.33:0/873906203 conn(0x55d8f62a4ab0 msgr2=0x55d8f62b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:01.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.374+0000 7f4be77fe640 0 client.17730 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:03:01.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.374+0000 7f4c1137e740 1 -- 172.21.15.33:0/873906203 shutdown_connections 2024-03-19T03:03:01.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.374+0000 7f4c1137e740 1 -- 172.21.15.33:0/873906203 wait complete. 2024-03-19T03:03:01.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.375+0000 7f4c1137e740 20 client.17730 trim_cache size 0 max 16384 2024-03-19T03:03:01.382 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T03:03:01.382 INFO:teuthology.orchestra.run.smithi033.stdout: "return_code": 0, 2024-03-19T03:03:01.382 INFO:teuthology.orchestra.run.smithi033.stdout: "scrub_tag": "290d8887-6c39-4d56-ac5f-c8b3d70a4fb9", 2024-03-19T03:03:01.382 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": "asynchronous" 2024-03-19T03:03:01.382 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T03:03:01.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.375+0000 7f4c0e6b2640 1 -- 172.21.15.33:0/2840512495 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4bf00a2970 msgr2=0x7f4bf00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:01.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.375+0000 7f4c0e6b2640 1 --2- 172.21.15.33:0/2840512495 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4bf00a2970 0x7f4bf00a4e30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8009bd0 tx=0x7f4bf8005f90 comp rx=0 tx=0).stop 2024-03-19T03:03:01.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.375+0000 7f4c0e6b2640 1 -- 172.21.15.33:0/2840512495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4c08103380 msgr2=0x7f4c081985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:01.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.375+0000 7f4c0e6b2640 1 --2- 172.21.15.33:0/2840512495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4c08103380 0x7f4c081985b0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc00e9f0 tx=0x7f4bfc00eeb0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.375+0000 7f4c0e6b2640 1 -- 172.21.15.33:0/2840512495 shutdown_connections 2024-03-19T03:03:01.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.375+0000 7f4c0e6b2640 1 --2- 172.21.15.33:0/2840512495 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4bf00a2970 0x7f4bf00a4e30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.375+0000 7f4c0e6b2640 1 --2- 172.21.15.33:0/2840512495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4c0810be30 0x7f4c0819c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.375+0000 7f4c0e6b2640 1 --2- 172.21.15.33:0/2840512495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4c08103380 0x7f4c081985b0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.375+0000 7f4c0e6b2640 1 --2- 172.21.15.33:0/2840512495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4c08102a60 0x7f4c08198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.375+0000 7f4c0e6b2640 1 -- 172.21.15.33:0/2840512495 >> 172.21.15.33:0/2840512495 conn(0x7f4c08076aa0 msgr2=0x7f4c080fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:01.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.375+0000 7f4c0e6b2640 1 -- 172.21.15.33:0/2840512495 shutdown_connections 2024-03-19T03:03:01.384 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.375+0000 7f4c0e6b2640 1 -- 172.21.15.33:0/2840512495 wait complete. 2024-03-19T03:03:01.387 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T03:03:01.387 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T03:03:01.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.491+0000 7f40ac7df640 1 -- 172.21.15.33:0/3948097443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40a40693e0 msgr2=0x7f40a410e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:01.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.491+0000 7f40ac7df640 1 --2- 172.21.15.33:0/3948097443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40a40693e0 0x7f40a410e680 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f4094009cd0 tx=0x7f409402c0f0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.491+0000 7f40ac7df640 1 -- 172.21.15.33:0/3948097443 shutdown_connections 2024-03-19T03:03:01.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.491+0000 7f40ac7df640 1 --2- 172.21.15.33:0/3948097443 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f40a4068a20 0x7f40a4068ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.495 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.491+0000 7f40ac7df640 1 --2- 172.21.15.33:0/3948097443 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40a40693e0 0x7f40a410e680 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.491+0000 7f40ac7df640 1 --2- 172.21.15.33:0/3948097443 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40a4107870 0x7f40a4107c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.491+0000 7f40ac7df640 1 -- 172.21.15.33:0/3948097443 >> 172.21.15.33:0/3948097443 conn(0x7f40a4077820 msgr2=0x7f40a4077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:01.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.492+0000 7f40ac7df640 1 -- 172.21.15.33:0/3948097443 shutdown_connections 2024-03-19T03:03:01.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.492+0000 7f40ac7df640 1 -- 172.21.15.33:0/3948097443 wait complete. 2024-03-19T03:03:01.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.492+0000 7f40ac7df640 1 Processor -- start 2024-03-19T03:03:01.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.492+0000 7f40ac7df640 1 -- start start 2024-03-19T03:03:01.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.492+0000 7f40ac7df640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40a4068a20 0x7f40a41006e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.492+0000 7f40ac7df640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40a40693e0 0x7f40a4100c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.492+0000 7f40ac7df640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f40a4107870 0x7f40a4101160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.492+0000 7f40ac7df640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f40a4111a70 con 0x7f40a4068a20 2024-03-19T03:03:01.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.492+0000 7f40ac7df640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f40a41118f0 con 0x7f40a40693e0 2024-03-19T03:03:01.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.492+0000 7f40ac7df640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f40a4111bf0 con 0x7f40a4107870 2024-03-19T03:03:01.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.492+0000 7f40aa57b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40a4068a20 0x7f40a41006e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.496 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.492+0000 7f40aa57b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40a4068a20 0x7f40a41006e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57178/0 (socket says 172.21.15.33:57178) 2024-03-19T03:03:01.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.492+0000 7f40aa57b640 1 -- 172.21.15.33:0/3072102611 learned_addr learned my addr 172.21.15.33:0/3072102611 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:01.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.493+0000 7f40a9d7a640 1 --2- 172.21.15.33:0/3072102611 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40a40693e0 0x7f40a4100c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.493+0000 7f40aad7c640 1 --2- 172.21.15.33:0/3072102611 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f40a4107870 0x7f40a4101160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.493+0000 7f40aa57b640 1 -- 172.21.15.33:0/3072102611 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f40a4107870 msgr2=0x7f40a4101160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:01.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.493+0000 7f40aa57b640 1 --2- 172.21.15.33:0/3072102611 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f40a4107870 0x7f40a4101160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.493+0000 7f40aa57b640 1 -- 172.21.15.33:0/3072102611 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40a40693e0 msgr2=0x7f40a4100c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:01.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.493+0000 7f40aa57b640 1 --2- 172.21.15.33:0/3072102611 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40a40693e0 0x7f40a4100c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.493+0000 7f40aa57b640 1 -- 172.21.15.33:0/3072102611 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40a41a63f0 con 0x7f40a4068a20 2024-03-19T03:03:01.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.493+0000 7f40aa57b640 1 --2- 172.21.15.33:0/3072102611 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40a4068a20 0x7f40a41006e0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f409c00e9f0 tx=0x7f409c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:01.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.493+0000 7f40a37fe640 1 -- 172.21.15.33:0/3072102611 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f409c00cd80 con 0x7f40a4068a20 2024-03-19T03:03:01.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.493+0000 7f40a37fe640 1 -- 172.21.15.33:0/3072102611 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f409c0108e0 con 0x7f40a4068a20 2024-03-19T03:03:01.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.493+0000 7f40ac7df640 1 -- 172.21.15.33:0/3072102611 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40a41a66e0 con 0x7f40a4068a20 2024-03-19T03:03:01.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.493+0000 7f40a37fe640 1 -- 172.21.15.33:0/3072102611 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f409c018920 con 0x7f40a4068a20 2024-03-19T03:03:01.497 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.493+0000 7f40ac7df640 1 -- 172.21.15.33:0/3072102611 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40a41a6c30 con 0x7f40a4068a20 2024-03-19T03:03:01.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.495+0000 7f40a37fe640 1 -- 172.21.15.33:0/3072102611 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f409c010a90 con 0x7f40a4068a20 2024-03-19T03:03:01.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.495+0000 7f40af3c9740 1 Processor -- start 2024-03-19T03:03:01.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.496+0000 7f40af3c9740 1 -- start start 2024-03-19T03:03:01.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.496+0000 7f40af3c9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579f2438a30 0x5579f2438e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.496+0000 7f40af3c9740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579f2439350 0x5579f24419f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.496+0000 7f40af3c9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579f2441f30 0x5579f2444320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.496+0000 7f40af3c9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5579f239c500 con 0x5579f2438a30 2024-03-19T03:03:01.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.496+0000 7f40af3c9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5579f2344be0 con 0x5579f2441f30 2024-03-19T03:03:01.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.496+0000 7f40af3c9740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5579f23985a0 con 0x5579f2439350 2024-03-19T03:03:01.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.496+0000 7f40aad7c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579f2438a30 0x5579f2438e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.496+0000 7f40aad7c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579f2438a30 0x5579f2438e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57182/0 (socket says 172.21.15.33:57182) 2024-03-19T03:03:01.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.496+0000 7f40aa57b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579f2439350 0x5579f24419f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.496+0000 7f40aad7c640 1 -- 172.21.15.33:0/240722485 learned_addr learned my addr 172.21.15.33:0/240722485 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:01.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.496+0000 7f40a9d7a640 1 --2- 172.21.15.33:0/240722485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579f2441f30 0x5579f2444320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.496+0000 7f40a37fe640 1 --2- 172.21.15.33:0/3072102611 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f408c0a2970 0x7f408c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.496+0000 7f40aad7c640 1 -- 172.21.15.33:0/240722485 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579f2439350 msgr2=0x5579f24419f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:01.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.496+0000 7f40aad7c640 1 --2- 172.21.15.33:0/240722485 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579f2439350 0x5579f24419f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.496+0000 7f40aad7c640 1 -- 172.21.15.33:0/240722485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579f2441f30 msgr2=0x5579f2444320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:01.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.496+0000 7f40aad7c640 1 --2- 172.21.15.33:0/240722485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579f2441f30 0x5579f2444320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.496+0000 7f40aad7c640 1 -- 172.21.15.33:0/240722485 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579f2366000 con 0x5579f2438a30 2024-03-19T03:03:01.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.496+0000 7f40aa57b640 1 --2- 172.21.15.33:0/240722485 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579f2439350 0x5579f24419f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:01.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.496+0000 7f40aad7c640 1 --2- 172.21.15.33:0/3072102611 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f408c0a2970 0x7f408c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.496+0000 7f40a37fe640 1 -- 172.21.15.33:0/3072102611 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f409c0cf520 con 0x7f40a4068a20 2024-03-19T03:03:01.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.496+0000 7f40aad7c640 1 --2- 172.21.15.33:0/240722485 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579f2438a30 0x5579f2438e10 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f4094004af0 tx=0x7f40940323c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acd4fbeff27f61d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:01.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.497+0000 7f40a0ff9640 1 -- 172.21.15.33:0/240722485 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4094039040 con 0x5579f2438a30 2024-03-19T03:03:01.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.497+0000 7f40a0ff9640 1 -- 172.21.15.33:0/240722485 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4094003b70 con 0x5579f2438a30 2024-03-19T03:03:01.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.497+0000 7f40a0ff9640 1 -- 172.21.15.33:0/240722485 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40940346a0 con 0x5579f2438a30 2024-03-19T03:03:01.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.497+0000 7f40aad7c640 1 --2- 172.21.15.33:0/3072102611 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f408c0a2970 0x7f408c0a4e30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4094006010 tx=0x7f4094045000 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:01.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.497+0000 7f40af3c9740 1 -- 172.21.15.33:0/240722485 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579f2438a30 msgr2=0x5579f2438e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:01.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.497+0000 7f40af3c9740 1 --2- 172.21.15.33:0/240722485 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579f2438a30 0x5579f2438e10 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f4094004af0 tx=0x7f40940323c0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.497+0000 7f40af3c9740 1 -- 172.21.15.33:0/240722485 shutdown_connections 2024-03-19T03:03:01.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.497+0000 7f40af3c9740 1 --2- 172.21.15.33:0/240722485 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579f2439350 0x5579f24419f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.497+0000 7f40af3c9740 1 --2- 172.21.15.33:0/240722485 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579f2441f30 0x5579f2444320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.497+0000 7f40af3c9740 1 --2- 172.21.15.33:0/240722485 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579f2438a30 0x5579f2438e10 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.497+0000 7f40af3c9740 1 -- 172.21.15.33:0/240722485 >> 172.21.15.33:0/240722485 conn(0x5579f2433160 msgr2=0x5579f2433540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:01.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.497+0000 7f40af3c9740 1 -- 172.21.15.33:0/240722485 shutdown_connections 2024-03-19T03:03:01.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.498+0000 7f40af3c9740 1 -- 172.21.15.33:0/240722485 wait complete. 2024-03-19T03:03:01.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.498+0000 7f40af3c9740 1 -- start start 2024-03-19T03:03:01.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.498+0000 7f40af3c9740 1 Processor -- start 2024-03-19T03:03:01.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.498+0000 7f40af3c9740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579f2438a30 0x5579f243ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.498+0000 7f40af3c9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579f2439350 0x5579f243f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.498+0000 7f40af3c9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579f2441f30 0x5579f243f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.498+0000 7f40af3c9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5579f2444d20 con 0x5579f2441f30 2024-03-19T03:03:01.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.498+0000 7f40af3c9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5579f2358160 con 0x5579f2439350 2024-03-19T03:03:01.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.498+0000 7f40af3c9740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5579f2444ea0 con 0x5579f2438a30 2024-03-19T03:03:01.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.498+0000 7f40af3c9740 10 client.0 _ll_register_callbacks cb 0x5579f2394a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:03:01.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.498+0000 7f40aad7c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579f2441f30 0x5579f243f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.498+0000 7f40aad7c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579f2441f30 0x5579f243f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57190/0 (socket says 172.21.15.33:57190) 2024-03-19T03:03:01.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.498+0000 7f40aad7c640 1 -- 172.21.15.33:0/176971403 learned_addr learned my addr 172.21.15.33:0/176971403 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:01.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.498+0000 7f40a9d7a640 1 --2- 172.21.15.33:0/176971403 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579f2439350 0x5579f243f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.498+0000 7f40aa57b640 1 --2- 172.21.15.33:0/176971403 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579f2438a30 0x5579f243ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.498+0000 7f40aad7c640 1 -- 172.21.15.33:0/176971403 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579f2438a30 msgr2=0x5579f243ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:01.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.498+0000 7f40aad7c640 1 --2- 172.21.15.33:0/176971403 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579f2438a30 0x5579f243ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.507 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.498+0000 7f40aad7c640 1 -- 172.21.15.33:0/176971403 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579f2439350 msgr2=0x5579f243f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:01.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.498+0000 7f40aad7c640 1 --2- 172.21.15.33:0/176971403 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579f2439350 0x5579f243f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.498+0000 7f40aad7c640 1 -- 172.21.15.33:0/176971403 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579f217a7d0 con 0x5579f2441f30 2024-03-19T03:03:01.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.499+0000 7f40aad7c640 1 --2- 172.21.15.33:0/176971403 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579f2441f30 0x5579f243f8f0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f4094009ca0 tx=0x7f40940047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e282771b3cacf948 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:01.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.499+0000 7f4089ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:03:01.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.499+0000 7f4089ffb640 1 -- 172.21.15.33:0/176971403 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4094058060 con 0x5579f2441f30 2024-03-19T03:03:01.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.499+0000 7f4089ffb640 1 -- 172.21.15.33:0/176971403 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f4094034bc0 con 0x5579f2441f30 2024-03-19T03:03:01.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.499+0000 7f40af3c9740 1 -- 172.21.15.33:0/176971403 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4094003b70 con 0x5579f2441f30 2024-03-19T03:03:01.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.499+0000 7f4089ffb640 1 -- 172.21.15.33:0/176971403 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4094039070 con 0x5579f2441f30 2024-03-19T03:03:01.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.499+0000 7f40af3c9740 10 client.17742 fetch_fsmap learned FSMap version 25 2024-03-19T03:03:01.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.499+0000 7f40af3c9740 1 -- 172.21.15.33:0/176971403 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5579f20b03f0 con 0x5579f2441f30 2024-03-19T03:03:01.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.499+0000 7f4089ffb640 1 -- 172.21.15.33:0/176971403 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4094003b70 con 0x5579f2441f30 2024-03-19T03:03:01.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.499+0000 7f4089ffb640 1 -- 172.21.15.33:0/176971403 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f409403f040 con 0x5579f2441f30 2024-03-19T03:03:01.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.499+0000 7f40af3c9740 10 client.17742 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:03:01.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.499+0000 7f40af3c9740 10 client.17742 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:03:01.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.499+0000 7f40af3c9740 20 client.17742 populate_metadata read hostname 'smithi033' 2024-03-19T03:03:01.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.499+0000 7f40af3c9740 1 --2- 172.21.15.33:0/176971403 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5579f2452060 0x5579f2454450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.499+0000 7f40af3c9740 4 client.17742 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:03:01.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.499+0000 7f40af3c9740 1 -- 172.21.15.33:0/176971403 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5579f23a9a40 con 0x5579f2452060 2024-03-19T03:03:01.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.500+0000 7f40aa57b640 1 --2- 172.21.15.33:0/176971403 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5579f2452060 0x5579f2454450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.500+0000 7f4089ffb640 10 client.17742 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:03:01.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.500+0000 7f40aa57b640 1 --2- 172.21.15.33:0/176971403 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5579f2452060 0x5579f2454450 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32ccb91071e1a7ad server_cookie=e09af21d36b28be8 in_seq=0 out_seq=0 2024-03-19T03:03:01.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.500+0000 7f4089ffb640 1 -- 172.21.15.33:0/176971403 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5579f23a9a40 con 0x5579f2452060 2024-03-19T03:03:01.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.500+0000 7f4089ffb640 10 client.17742 handle_command_reply: tid=0 2024-03-19T03:03:01.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.500+0000 7f40af3c9740 1 client.17742 shutdown 2024-03-19T03:03:01.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.501+0000 7f40af3c9740 1 -- 172.21.15.33:0/176971403 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579f2441f30 msgr2=0x5579f243f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:01.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.501+0000 7f40af3c9740 1 --2- 172.21.15.33:0/176971403 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579f2441f30 0x5579f243f8f0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f4094009ca0 tx=0x7f40940047c0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.501+0000 7f40af3c9740 1 -- 172.21.15.33:0/176971403 shutdown_connections 2024-03-19T03:03:01.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.501+0000 7f40af3c9740 1 --2- 172.21.15.33:0/176971403 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5579f2452060 0x5579f2454450 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.501+0000 7f40af3c9740 1 --2- 172.21.15.33:0/176971403 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579f2441f30 0x5579f243f8f0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.509 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.501+0000 7f40af3c9740 1 --2- 172.21.15.33:0/176971403 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579f2439350 0x5579f243f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.501+0000 7f4089ffb640 0 client.17742 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:03:01.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.501+0000 7f40af3c9740 1 --2- 172.21.15.33:0/176971403 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579f2438a30 0x5579f243ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.501+0000 7f40af3c9740 1 -- 172.21.15.33:0/176971403 >> 172.21.15.33:0/176971403 conn(0x5579f2433160 msgr2=0x5579f2434eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:01.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.501+0000 7f40af3c9740 1 -- 172.21.15.33:0/176971403 shutdown_connections 2024-03-19T03:03:01.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.501+0000 7f40af3c9740 1 -- 172.21.15.33:0/176971403 wait complete. 2024-03-19T03:03:01.510 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.501+0000 7f40af3c9740 20 client.17742 trim_cache size 0 max 16384 2024-03-19T03:03:01.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.519+0000 7f40af3c9740 1 Processor -- start 2024-03-19T03:03:01.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.519+0000 7f40af3c9740 1 -- start start 2024-03-19T03:03:01.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.519+0000 7f40af3c9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579f2438a30 0x5579f2441530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.519+0000 7f40af3c9740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579f2439350 0x5579f243dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.519+0000 7f40af3c9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579f2441f30 0x5579f2434d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.519+0000 7f40af3c9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f40a41b36b0 con 0x5579f2438a30 2024-03-19T03:03:01.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.519+0000 7f40af3c9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f40a41b0f80 con 0x5579f2441f30 2024-03-19T03:03:01.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.519+0000 7f40af3c9740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f40a41aedf0 con 0x5579f2439350 2024-03-19T03:03:01.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.519+0000 7f40aa57b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579f2438a30 0x5579f2441530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.519+0000 7f40aa57b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579f2438a30 0x5579f2441530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57204/0 (socket says 172.21.15.33:57204) 2024-03-19T03:03:01.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.519+0000 7f40aa57b640 1 -- 172.21.15.33:0/488588506 learned_addr learned my addr 172.21.15.33:0/488588506 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:01.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.519+0000 7f40aad7c640 1 --2- 172.21.15.33:0/488588506 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579f2441f30 0x5579f2434d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.519+0000 7f40a9d7a640 1 --2- 172.21.15.33:0/488588506 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579f2439350 0x5579f243dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.520+0000 7f40aa57b640 1 -- 172.21.15.33:0/488588506 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579f2439350 msgr2=0x5579f243dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:01.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.520+0000 7f40aa57b640 1 --2- 172.21.15.33:0/488588506 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579f2439350 0x5579f243dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.520+0000 7f40aa57b640 1 -- 172.21.15.33:0/488588506 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579f2441f30 msgr2=0x5579f2434d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:01.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.520+0000 7f40aa57b640 1 --2- 172.21.15.33:0/488588506 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579f2441f30 0x5579f2434d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.520+0000 7f40aa57b640 1 -- 172.21.15.33:0/488588506 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579f20700f0 con 0x5579f2438a30 2024-03-19T03:03:01.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.520+0000 7f40aa57b640 1 --2- 172.21.15.33:0/488588506 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579f2438a30 0x5579f2441530 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f409c097c60 tx=0x7f409c0ccf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5111fd9223f65bf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:01.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.520+0000 7f408bfff640 1 -- 172.21.15.33:0/488588506 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f409c09b160 con 0x5579f2438a30 2024-03-19T03:03:01.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.520+0000 7f408bfff640 1 -- 172.21.15.33:0/488588506 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f409c010de0 con 0x5579f2438a30 2024-03-19T03:03:01.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.520+0000 7f408bfff640 1 -- 172.21.15.33:0/488588506 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f409c0861d0 con 0x5579f2438a30 2024-03-19T03:03:01.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.520+0000 7f40af3c9740 1 -- 172.21.15.33:0/488588506 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579f2438a30 msgr2=0x5579f2441530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:01.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.520+0000 7f40af3c9740 1 --2- 172.21.15.33:0/488588506 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579f2438a30 0x5579f2441530 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f409c097c60 tx=0x7f409c0ccf90 comp rx=0 tx=0).stop 2024-03-19T03:03:01.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.520+0000 7f40af3c9740 1 -- 172.21.15.33:0/488588506 shutdown_connections 2024-03-19T03:03:01.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.520+0000 7f40af3c9740 1 --2- 172.21.15.33:0/488588506 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579f2439350 0x5579f243dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.520+0000 7f40af3c9740 1 --2- 172.21.15.33:0/488588506 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579f2441f30 0x5579f2434d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.520+0000 7f40af3c9740 1 --2- 172.21.15.33:0/488588506 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579f2438a30 0x5579f2441530 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.521+0000 7f40af3c9740 1 -- 172.21.15.33:0/488588506 >> 172.21.15.33:0/488588506 conn(0x5579f2433160 msgr2=0x5579f24321e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:01.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.521+0000 7f40af3c9740 1 -- 172.21.15.33:0/488588506 shutdown_connections 2024-03-19T03:03:01.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.521+0000 7f40af3c9740 1 -- 172.21.15.33:0/488588506 wait complete. 2024-03-19T03:03:01.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.521+0000 7f40af3c9740 1 -- start start 2024-03-19T03:03:01.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.521+0000 7f40af3c9740 1 Processor -- start 2024-03-19T03:03:01.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.521+0000 7f40af3c9740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579f2438a30 0x5579f244a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.521+0000 7f40af3c9740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579f2439350 0x5579f24470c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.521+0000 7f40af3c9740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579f2441f30 0x5579f2447600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.521+0000 7f40af3c9740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f40a41af440 con 0x5579f2438a30 2024-03-19T03:03:01.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.521+0000 7f40af3c9740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f40a41b08b0 con 0x5579f2441f30 2024-03-19T03:03:01.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.521+0000 7f40af3c9740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f40a41afb10 con 0x5579f2439350 2024-03-19T03:03:01.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.521+0000 7f40af3c9740 10 client.0 _ll_register_callbacks cb 0x7f40980097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:03:01.528 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.521+0000 7f40aa57b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579f2438a30 0x5579f244a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.521+0000 7f40aa57b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579f2438a30 0x5579f244a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:57220/0 (socket says 172.21.15.33:57220) 2024-03-19T03:03:01.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.521+0000 7f40aa57b640 1 -- 172.21.15.33:0/397845787 learned_addr learned my addr 172.21.15.33:0/397845787 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:01.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.521+0000 7f40aad7c640 1 --2- 172.21.15.33:0/397845787 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579f2441f30 0x5579f2447600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.521+0000 7f40a9d7a640 1 --2- 172.21.15.33:0/397845787 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579f2439350 0x5579f24470c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.521+0000 7f40aa57b640 1 -- 172.21.15.33:0/397845787 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579f2439350 msgr2=0x5579f24470c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:01.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.521+0000 7f40aa57b640 1 --2- 172.21.15.33:0/397845787 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579f2439350 0x5579f24470c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.521+0000 7f40aa57b640 1 -- 172.21.15.33:0/397845787 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579f2441f30 msgr2=0x5579f2447600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:01.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.521+0000 7f40aa57b640 1 --2- 172.21.15.33:0/397845787 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579f2441f30 0x5579f2447600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.522+0000 7f40aa57b640 1 -- 172.21.15.33:0/397845787 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579f2447b40 con 0x5579f2438a30 2024-03-19T03:03:01.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.522+0000 7f40aa57b640 1 --2- 172.21.15.33:0/397845787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579f2438a30 0x5579f244a0f0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f409c09b500 tx=0x7f409c09b1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b366a80b63b29db server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:01.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.522+0000 7f4089ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:03:01.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.522+0000 7f4089ffb640 1 -- 172.21.15.33:0/397845787 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f409c0a3ef0 con 0x5579f2438a30 2024-03-19T03:03:01.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.522+0000 7f4089ffb640 1 -- 172.21.15.33:0/397845787 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9496+0+0 (secure 0 0 0) 0x7f409c097e70 con 0x5579f2438a30 2024-03-19T03:03:01.529 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.522+0000 7f40af3c9740 1 -- 172.21.15.33:0/397845787 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5579f243e520 con 0x5579f2438a30 2024-03-19T03:03:01.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.522+0000 7f4089ffb640 1 -- 172.21.15.33:0/397845787 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f409c09f8a0 con 0x5579f2438a30 2024-03-19T03:03:01.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.522+0000 7f40af3c9740 10 client.17748 fetch_fsmap learned FSMap version 25 2024-03-19T03:03:01.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.522+0000 7f40af3c9740 1 -- 172.21.15.33:0/397845787 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5579f24b8e50 con 0x5579f2438a30 2024-03-19T03:03:01.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.522+0000 7f4089ffb640 1 -- 172.21.15.33:0/397845787 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5579f243e520 con 0x5579f2438a30 2024-03-19T03:03:01.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.522+0000 7f4089ffb640 1 -- 172.21.15.33:0/397845787 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f409c0afb30 con 0x5579f2438a30 2024-03-19T03:03:01.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.522+0000 7f40af3c9740 10 client.17748 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:03:01.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.522+0000 7f40af3c9740 10 client.17748 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:03:01.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.522+0000 7f40af3c9740 20 client.17748 populate_metadata read hostname 'smithi033' 2024-03-19T03:03:01.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.522+0000 7f40af3c9740 1 --2- 172.21.15.33:0/397845787 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5579f24b94e0 0x5579f24bb900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:01.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.522+0000 7f40af3c9740 4 client.17748 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T03:03:01.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.522+0000 7f40af3c9740 1 -- 172.21.15.33:0/397845787 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5579f23a3d90 con 0x5579f24b94e0 2024-03-19T03:03:01.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.523+0000 7f40a9d7a640 1 --2- 172.21.15.33:0/397845787 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5579f24b94e0 0x5579f24bb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:01.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.523+0000 7f40a9d7a640 1 --2- 172.21.15.33:0/397845787 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5579f24b94e0 0x5579f24bb900 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a19f6e554da739 server_cookie=c09acf9ee503c44 in_seq=0 out_seq=0 2024-03-19T03:03:01.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.523+0000 7f4089ffb640 10 client.17748 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:03:01.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.523+0000 7f4089ffb640 1 -- 172.21.15.33:0/397845787 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5579f23a3d90 con 0x5579f24b94e0 2024-03-19T03:03:01.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.523+0000 7f4089ffb640 10 client.17748 handle_command_reply: tid=0 2024-03-19T03:03:01.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.523+0000 7f40af3c9740 1 client.17748 shutdown 2024-03-19T03:03:01.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.523+0000 7f40af3c9740 1 -- 172.21.15.33:0/397845787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579f2438a30 msgr2=0x5579f244a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:01.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.523+0000 7f40af3c9740 1 --2- 172.21.15.33:0/397845787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579f2438a30 0x5579f244a0f0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f409c09b500 tx=0x7f409c09b1e0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.523+0000 7f40af3c9740 1 -- 172.21.15.33:0/397845787 shutdown_connections 2024-03-19T03:03:01.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.523+0000 7f40af3c9740 1 --2- 172.21.15.33:0/397845787 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x5579f24b94e0 0x5579f24bb900 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.523+0000 7f40af3c9740 1 --2- 172.21.15.33:0/397845787 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5579f2439350 0x5579f24470c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.523+0000 7f40af3c9740 1 --2- 172.21.15.33:0/397845787 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5579f2441f30 0x5579f2447600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.523+0000 7f4089ffb640 0 client.17748 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:03:01.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.523+0000 7f40af3c9740 1 --2- 172.21.15.33:0/397845787 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5579f2438a30 0x5579f244a0f0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.523+0000 7f40af3c9740 1 -- 172.21.15.33:0/397845787 >> 172.21.15.33:0/397845787 conn(0x5579f2433160 msgr2=0x5579f24404d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:01.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.523+0000 7f40af3c9740 1 -- 172.21.15.33:0/397845787 shutdown_connections 2024-03-19T03:03:01.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.523+0000 7f40af3c9740 1 -- 172.21.15.33:0/397845787 wait complete. 2024-03-19T03:03:01.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.523+0000 7f40af3c9740 20 client.17748 trim_cache size 0 max 16384 2024-03-19T03:03:01.532 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T03:03:01.532 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-19T03:03:01.532 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": { 2024-03-19T03:03:01.532 INFO:teuthology.orchestra.run.smithi033.stdout: "290d8887-6c39-4d56-ac5f-c8b3d70a4fb9": { 2024-03-19T03:03:01.532 INFO:teuthology.orchestra.run.smithi033.stdout: "path": "/", 2024-03-19T03:03:01.532 INFO:teuthology.orchestra.run.smithi033.stdout: "tag": "290d8887-6c39-4d56-ac5f-c8b3d70a4fb9", 2024-03-19T03:03:01.532 INFO:teuthology.orchestra.run.smithi033.stdout: "options": "recursive,force" 2024-03-19T03:03:01.532 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:01.532 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:01.532 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T03:03:01.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.524+0000 7f40ac7df640 1 -- 172.21.15.33:0/3072102611 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f408c0a2970 msgr2=0x7f408c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:01.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.524+0000 7f40ac7df640 1 --2- 172.21.15.33:0/3072102611 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f408c0a2970 0x7f408c0a4e30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4094006010 tx=0x7f4094045000 comp rx=0 tx=0).stop 2024-03-19T03:03:01.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.524+0000 7f40ac7df640 1 -- 172.21.15.33:0/3072102611 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40a4068a20 msgr2=0x7f40a41006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:01.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.524+0000 7f40ac7df640 1 --2- 172.21.15.33:0/3072102611 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40a4068a20 0x7f40a41006e0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f409c00e9f0 tx=0x7f409c00eeb0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.533 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.524+0000 7f40ac7df640 1 -- 172.21.15.33:0/3072102611 shutdown_connections 2024-03-19T03:03:01.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.524+0000 7f40ac7df640 1 --2- 172.21.15.33:0/3072102611 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f408c0a2970 0x7f408c0a4e30 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.524+0000 7f40ac7df640 1 --2- 172.21.15.33:0/3072102611 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f40a4107870 0x7f40a4101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.524+0000 7f40ac7df640 1 --2- 172.21.15.33:0/3072102611 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40a40693e0 0x7f40a4100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.524+0000 7f40ac7df640 1 --2- 172.21.15.33:0/3072102611 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f40a4068a20 0x7f40a41006e0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:01.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.524+0000 7f40ac7df640 1 -- 172.21.15.33:0/3072102611 >> 172.21.15.33:0/3072102611 conn(0x7f40a4077820 msgr2=0x7f40a40fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:01.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.524+0000 7f40ac7df640 1 -- 172.21.15.33:0/3072102611 shutdown_connections 2024-03-19T03:03:01.534 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:01.524+0000 7f40ac7df640 1 -- 172.21.15.33:0/3072102611 wait complete. 2024-03-19T03:03:01.536 INFO:tasks.cephfs.filesystem:scrub status for tag:290d8887-6c39-4d56-ac5f-c8b3d70a4fb9 - {'path': '/', 'tag': '290d8887-6c39-4d56-ac5f-c8b3d70a4fb9', 'options': 'recursive,force'} 2024-03-19T03:03:02.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:02 smithi155 ceph-mon[35804]: pgmap v296: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-19T03:03:02.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:02 smithi155 ceph-mon[35804]: scrub queued for path: / 2024-03-19T03:03:02.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:02 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T03:03:02.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:02 smithi155 ceph-mon[35804]: scrub summary: active paths [/] 2024-03-19T03:03:02.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:02 smithi155 ceph-mon[35804]: scrub summary: idle+waiting paths [/] 2024-03-19T03:03:02.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:02 smithi033 ceph-mon[31973]: pgmap v296: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-19T03:03:02.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:02 smithi033 ceph-mon[31973]: scrub queued for path: / 2024-03-19T03:03:02.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:02 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T03:03:02.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:02 smithi033 ceph-mon[31973]: scrub summary: active paths [/] 2024-03-19T03:03:02.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:02 smithi033 ceph-mon[31973]: scrub summary: idle+waiting paths [/] 2024-03-19T03:03:02.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:02 smithi096 ceph-mon[35946]: pgmap v296: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-19T03:03:02.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:02 smithi096 ceph-mon[35946]: scrub queued for path: / 2024-03-19T03:03:02.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:02 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T03:03:02.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:02 smithi096 ceph-mon[35946]: scrub summary: active paths [/] 2024-03-19T03:03:02.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:02 smithi096 ceph-mon[35946]: scrub summary: idle+waiting paths [/] 2024-03-19T03:03:03.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:03 smithi155 ceph-mon[35804]: scrub summary: idle 2024-03-19T03:03:03.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:03 smithi033 ceph-mon[31973]: scrub summary: idle 2024-03-19T03:03:03.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:03 smithi096 ceph-mon[35946]: scrub summary: idle 2024-03-19T03:03:04.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:04 smithi155 ceph-mon[35804]: pgmap v297: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-19T03:03:04.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:04 smithi033 ceph-mon[31973]: pgmap v297: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-19T03:03:04.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:04 smithi096 ceph-mon[35946]: pgmap v297: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-19T03:03:06.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:06 smithi155 ceph-mon[35804]: pgmap v298: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-19T03:03:06.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:06 smithi033 ceph-mon[31973]: pgmap v298: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-19T03:03:06.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:06 smithi096 ceph-mon[35946]: pgmap v298: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-19T03:03:08.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:08 smithi155 ceph-mon[35804]: pgmap v299: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-19T03:03:08.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:08 smithi033 ceph-mon[31973]: pgmap v299: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-19T03:03:08.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:08 smithi096 ceph-mon[35946]: pgmap v299: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-19T03:03:09.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:09 smithi155 ceph-mon[35804]: osdmap e105: 12 total, 12 up, 12 in 2024-03-19T03:03:09.778 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:09 smithi033 ceph-mon[31973]: osdmap e105: 12 total, 12 up, 12 in 2024-03-19T03:03:09.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:09 smithi096 ceph-mon[35946]: osdmap e105: 12 total, 12 up, 12 in 2024-03-19T03:03:10.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:10 smithi155 ceph-mon[35804]: pgmap v301: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-03-19T03:03:10.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:10 smithi033 ceph-mon[31973]: pgmap v301: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-03-19T03:03:10.965 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:10 smithi096 ceph-mon[35946]: pgmap v301: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-03-19T03:03:12.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:12 smithi155 ceph-mon[35804]: pgmap v302: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-03-19T03:03:12.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:12 smithi033 ceph-mon[31973]: pgmap v302: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-03-19T03:03:12.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:12 smithi096 ceph-mon[35946]: pgmap v302: 129 pgs: 129 active+clean; 1.9 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-03-19T03:03:14.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:14 smithi155 ceph-mon[35804]: pgmap v303: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-19T03:03:14.779 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:14 smithi033 ceph-mon[31973]: pgmap v303: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-19T03:03:14.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:14 smithi096 ceph-mon[35946]: pgmap v303: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-19T03:03:16.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:16 smithi155 ceph-mon[35804]: pgmap v304: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-19T03:03:16.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:16 smithi155 ceph-mon[35804]: osdmap e106: 12 total, 12 up, 12 in 2024-03-19T03:03:16.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:16 smithi033 ceph-mon[31973]: pgmap v304: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-19T03:03:16.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:16 smithi033 ceph-mon[31973]: osdmap e106: 12 total, 12 up, 12 in 2024-03-19T03:03:16.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:16 smithi096 ceph-mon[35946]: pgmap v304: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-19T03:03:16.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:16 smithi096 ceph-mon[35946]: osdmap e106: 12 total, 12 up, 12 in 2024-03-19T03:03:18.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:18 smithi155 ceph-mon[35804]: pgmap v306: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 118 B/s wr, 3 op/s 2024-03-19T03:03:18.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:18 smithi033 ceph-mon[31973]: pgmap v306: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 118 B/s wr, 3 op/s 2024-03-19T03:03:18.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:18 smithi096 ceph-mon[35946]: pgmap v306: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 118 B/s wr, 3 op/s 2024-03-19T03:03:20.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:20 smithi155 ceph-mon[35804]: pgmap v307: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-19T03:03:20.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:20 smithi033 ceph-mon[31973]: pgmap v307: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-19T03:03:20.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:20 smithi096 ceph-mon[35946]: pgmap v307: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-19T03:03:22.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:22 smithi155 ceph-mon[35804]: pgmap v308: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-19T03:03:22.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:22 smithi033 ceph-mon[31973]: pgmap v308: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-19T03:03:22.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:22 smithi096 ceph-mon[35946]: pgmap v308: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-19T03:03:24.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:24 smithi155 ceph-mon[35804]: pgmap v309: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-19T03:03:24.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:24 smithi033 ceph-mon[31973]: pgmap v309: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-19T03:03:24.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:24 smithi096 ceph-mon[35946]: pgmap v309: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-19T03:03:26.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:26 smithi155 ceph-mon[35804]: pgmap v310: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-19T03:03:26.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:26 smithi033 ceph-mon[31973]: pgmap v310: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-19T03:03:26.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:26 smithi096 ceph-mon[35946]: pgmap v310: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-19T03:03:28.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:28 smithi155 ceph-mon[35804]: pgmap v311: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-19T03:03:28.517 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:28 smithi096 ceph-mon[35946]: pgmap v311: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-19T03:03:28.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:28 smithi033 ceph-mon[31973]: pgmap v311: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-19T03:03:29.617 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-19T03:03:29.734 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.732+0000 7fbe850d1640 1 -- 172.21.15.33:0/2072269375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe80069840 msgr2=0x7fbe8010e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:29.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.732+0000 7fbe850d1640 1 --2- 172.21.15.33:0/2072269375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe80069840 0x7fbe8010e6b0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fbe70009cd0 tx=0x7fbe7002c0f0 comp rx=0 tx=0).stop 2024-03-19T03:03:29.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.732+0000 7fbe850d1640 1 -- 172.21.15.33:0/2072269375 shutdown_connections 2024-03-19T03:03:29.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.733+0000 7fbe850d1640 1 --2- 172.21.15.33:0/2072269375 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe80068ea0 0x7fbe80069300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:29.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.733+0000 7fbe850d1640 1 --2- 172.21.15.33:0/2072269375 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe80069840 0x7fbe8010e6b0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:29.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.733+0000 7fbe850d1640 1 --2- 172.21.15.33:0/2072269375 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe801078f0 0x7fbe80107cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:29.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.733+0000 7fbe850d1640 1 -- 172.21.15.33:0/2072269375 >> 172.21.15.33:0/2072269375 conn(0x7fbe80077570 msgr2=0x7fbe80077980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:29.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.733+0000 7fbe850d1640 1 -- 172.21.15.33:0/2072269375 shutdown_connections 2024-03-19T03:03:29.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.733+0000 7fbe850d1640 1 -- 172.21.15.33:0/2072269375 wait complete. 2024-03-19T03:03:29.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.733+0000 7fbe850d1640 1 Processor -- start 2024-03-19T03:03:29.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.734+0000 7fbe850d1640 1 -- start start 2024-03-19T03:03:29.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.734+0000 7fbe850d1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe80068ea0 0x7fbe80100a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:29.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.734+0000 7fbe850d1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe80069840 0x7fbe80100f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:29.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.734+0000 7fbe850d1640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe801078f0 0x7fbe801014c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:29.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.734+0000 7fbe850d1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbe80111a90 con 0x7fbe80068ea0 2024-03-19T03:03:29.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.734+0000 7fbe850d1640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbe80111910 con 0x7fbe80069840 2024-03-19T03:03:29.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.734+0000 7fbe850d1640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbe80111c10 con 0x7fbe801078f0 2024-03-19T03:03:29.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.734+0000 7fbe7ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe80068ea0 0x7fbe80100a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:29.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.734+0000 7fbe7ed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe80068ea0 0x7fbe80100a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41126/0 (socket says 172.21.15.33:41126) 2024-03-19T03:03:29.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.734+0000 7fbe7ed9d640 1 -- 172.21.15.33:0/2707041641 learned_addr learned my addr 172.21.15.33:0/2707041641 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:29.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.734+0000 7fbe7e59c640 1 --2- 172.21.15.33:0/2707041641 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe80069840 0x7fbe80100f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:29.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.734+0000 7fbe7f59e640 1 --2- 172.21.15.33:0/2707041641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe801078f0 0x7fbe801014c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:29.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.734+0000 7fbe7ed9d640 1 -- 172.21.15.33:0/2707041641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe801078f0 msgr2=0x7fbe801014c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:29.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.734+0000 7fbe7ed9d640 1 --2- 172.21.15.33:0/2707041641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe801078f0 0x7fbe801014c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:29.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.734+0000 7fbe7ed9d640 1 -- 172.21.15.33:0/2707041641 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe80069840 msgr2=0x7fbe80100f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:29.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.734+0000 7fbe7ed9d640 1 --2- 172.21.15.33:0/2707041641 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe80069840 0x7fbe80100f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:29.738 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.734+0000 7fbe7ed9d640 1 -- 172.21.15.33:0/2707041641 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe801a6390 con 0x7fbe80068ea0 2024-03-19T03:03:29.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.734+0000 7fbe7ed9d640 1 --2- 172.21.15.33:0/2707041641 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe80068ea0 0x7fbe80100a40 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fbe7800e9f0 tx=0x7fbe7800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:29.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.734+0000 7fbe77fff640 1 -- 172.21.15.33:0/2707041641 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe7800cd80 con 0x7fbe80068ea0 2024-03-19T03:03:29.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.735+0000 7fbe77fff640 1 -- 172.21.15.33:0/2707041641 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe780108e0 con 0x7fbe80068ea0 2024-03-19T03:03:29.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.735+0000 7fbe850d1640 1 -- 172.21.15.33:0/2707041641 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe801a6680 con 0x7fbe80068ea0 2024-03-19T03:03:29.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.735+0000 7fbe77fff640 1 -- 172.21.15.33:0/2707041641 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe78018920 con 0x7fbe80068ea0 2024-03-19T03:03:29.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.735+0000 7fbe850d1640 1 -- 172.21.15.33:0/2707041641 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe801a6c20 con 0x7fbe80068ea0 2024-03-19T03:03:29.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.736+0000 7fbe850d1640 1 -- 172.21.15.33:0/2707041641 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe8004f580 con 0x7fbe80068ea0 2024-03-19T03:03:29.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.737+0000 7fbe77fff640 1 -- 172.21.15.33:0/2707041641 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fbe78010a90 con 0x7fbe80068ea0 2024-03-19T03:03:29.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.737+0000 7fbe77fff640 1 --2- 172.21.15.33:0/2707041641 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbe6c0a2a90 0x7fbe6c0a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:29.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.737+0000 7fbe7e59c640 1 --2- 172.21.15.33:0/2707041641 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbe6c0a2a90 0x7fbe6c0a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:29.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.737+0000 7fbe77fff640 1 -- 172.21.15.33:0/2707041641 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7fbe780cf470 con 0x7fbe80068ea0 2024-03-19T03:03:29.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.738+0000 7fbe7e59c640 1 --2- 172.21.15.33:0/2707041641 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbe6c0a2a90 0x7fbe6c0a4f50 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbe801024d0 tx=0x7fbe68005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:29.743 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.742+0000 7fbe77fff640 1 -- 172.21.15.33:0/2707041641 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbe7809b7b0 con 0x7fbe80068ea0 2024-03-19T03:03:29.949 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.948+0000 7fbe850d1640 1 -- 172.21.15.33:0/2707041641 --> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fbe80109d70 con 0x7fbe6c0a2a90 2024-03-19T03:03:29.956 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.955+0000 7fbe77fff640 1 -- 172.21.15.33:0/2707041641 <== mgr.15618 v2:172.21.15.33:6800/2362023661 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7fbe68004030 con 0x7fbe6c0a2a90 2024-03-19T03:03:29.957 INFO:teuthology.orchestra.run.smithi033.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-19T00:00:00", "created": "2024-03-19T02:53:27", "first": "2024-03-19T02:54:00", "last": "2024-03-19T03:03:00", "last_pruned": "2024-03-19T03:03:00", "created_count": 10, "pruned_count": 4, "active": true} 2024-03-19T03:03:29.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.957+0000 7fbe850d1640 1 -- 172.21.15.33:0/2707041641 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbe6c0a2a90 msgr2=0x7fbe6c0a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:29.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.957+0000 7fbe850d1640 1 --2- 172.21.15.33:0/2707041641 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbe6c0a2a90 0x7fbe6c0a4f50 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbe801024d0 tx=0x7fbe68005f90 comp rx=0 tx=0).stop 2024-03-19T03:03:29.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.957+0000 7fbe850d1640 1 -- 172.21.15.33:0/2707041641 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe80068ea0 msgr2=0x7fbe80100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:29.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.957+0000 7fbe850d1640 1 --2- 172.21.15.33:0/2707041641 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe80068ea0 0x7fbe80100a40 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fbe7800e9f0 tx=0x7fbe7800eeb0 comp rx=0 tx=0).stop 2024-03-19T03:03:29.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.957+0000 7fbe850d1640 1 -- 172.21.15.33:0/2707041641 shutdown_connections 2024-03-19T03:03:29.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.957+0000 7fbe850d1640 1 --2- 172.21.15.33:0/2707041641 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbe6c0a2a90 0x7fbe6c0a4f50 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:29.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.957+0000 7fbe850d1640 1 --2- 172.21.15.33:0/2707041641 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe801078f0 0x7fbe801014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:29.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.957+0000 7fbe850d1640 1 --2- 172.21.15.33:0/2707041641 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe80069840 0x7fbe80100f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:29.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.957+0000 7fbe850d1640 1 --2- 172.21.15.33:0/2707041641 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe80068ea0 0x7fbe80100a40 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:29.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.957+0000 7fbe850d1640 1 -- 172.21.15.33:0/2707041641 >> 172.21.15.33:0/2707041641 conn(0x7fbe80077570 msgr2=0x7fbe800fe980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:29.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.957+0000 7fbe850d1640 1 -- 172.21.15.33:0/2707041641 shutdown_connections 2024-03-19T03:03:29.960 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:29.957+0000 7fbe850d1640 1 -- 172.21.15.33:0/2707041641 wait complete. 2024-03-19T03:03:29.969 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-19T03:03:30.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.073+0000 7f8d8415b640 1 -- 172.21.15.33:0/265116044 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d7c068ea0 msgr2=0x7f8d7c0692e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:30.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.073+0000 7f8d8415b640 1 --2- 172.21.15.33:0/265116044 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d7c068ea0 0x7f8d7c0692e0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f8d70009930 tx=0x7f8d7002bc30 comp rx=0 tx=0).stop 2024-03-19T03:03:30.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.073+0000 7f8d8415b640 1 -- 172.21.15.33:0/265116044 shutdown_connections 2024-03-19T03:03:30.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.073+0000 7f8d8415b640 1 --2- 172.21.15.33:0/265116044 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d7c069820 0x7f8d7c10e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:30.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.073+0000 7f8d8415b640 1 --2- 172.21.15.33:0/265116044 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d7c068ea0 0x7f8d7c0692e0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:30.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.073+0000 7f8d8415b640 1 --2- 172.21.15.33:0/265116044 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d7c107800 0x7f8d7c107be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:30.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.073+0000 7f8d8415b640 1 -- 172.21.15.33:0/265116044 >> 172.21.15.33:0/265116044 conn(0x7f8d7c077570 msgr2=0x7f8d7c077980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:30.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.074+0000 7f8d8415b640 1 -- 172.21.15.33:0/265116044 shutdown_connections 2024-03-19T03:03:30.077 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.074+0000 7f8d8415b640 1 -- 172.21.15.33:0/265116044 wait complete. 2024-03-19T03:03:30.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.074+0000 7f8d8415b640 1 Processor -- start 2024-03-19T03:03:30.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.074+0000 7f8d8415b640 1 -- start start 2024-03-19T03:03:30.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.074+0000 7f8d8415b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d7c068ea0 0x7f8d7c197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:30.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.074+0000 7f8d8415b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d7c069820 0x7f8d7c1984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:30.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.074+0000 7f8d8415b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d7c107800 0x7f8d7c19c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:30.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.074+0000 7f8d8415b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f8d7c111ab0 con 0x7f8d7c069820 2024-03-19T03:03:30.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.074+0000 7f8d8415b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8d7c111930 con 0x7f8d7c107800 2024-03-19T03:03:30.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.074+0000 7f8d8415b640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8d7c111c30 con 0x7f8d7c068ea0 2024-03-19T03:03:30.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.074+0000 7f8d816f6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d7c069820 0x7f8d7c1984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:30.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.074+0000 7f8d816f6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d7c069820 0x7f8d7c1984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41144/0 (socket says 172.21.15.33:41144) 2024-03-19T03:03:30.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.074+0000 7f8d816f6640 1 -- 172.21.15.33:0/2817628848 learned_addr learned my addr 172.21.15.33:0/2817628848 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:30.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.074+0000 7f8d81ef7640 1 --2- 172.21.15.33:0/2817628848 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d7c068ea0 0x7f8d7c197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:30.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.074+0000 7f8d826f8640 1 --2- 172.21.15.33:0/2817628848 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d7c107800 0x7f8d7c19c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:30.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.075+0000 7f8d816f6640 1 -- 172.21.15.33:0/2817628848 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d7c068ea0 msgr2=0x7f8d7c197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:30.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.075+0000 7f8d816f6640 1 --2- 172.21.15.33:0/2817628848 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d7c068ea0 0x7f8d7c197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:30.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.075+0000 7f8d816f6640 1 -- 172.21.15.33:0/2817628848 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d7c107800 msgr2=0x7f8d7c19c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:30.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.075+0000 7f8d816f6640 1 --2- 172.21.15.33:0/2817628848 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d7c107800 0x7f8d7c19c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:30.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.075+0000 7f8d816f6640 1 -- 172.21.15.33:0/2817628848 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d7c19cfa0 con 0x7f8d7c069820 2024-03-19T03:03:30.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.075+0000 7f8d816f6640 1 --2- 172.21.15.33:0/2817628848 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d7c069820 0x7f8d7c1984f0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f8d70009230 tx=0x7f8d7002c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:30.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.075+0000 7f8d7affd640 1 -- 172.21.15.33:0/2817628848 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d70043020 con 0x7f8d7c069820 2024-03-19T03:03:30.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.075+0000 7f8d8415b640 1 -- 172.21.15.33:0/2817628848 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d7c19d230 con 0x7f8d7c069820 2024-03-19T03:03:30.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.075+0000 7f8d7affd640 1 -- 172.21.15.33:0/2817628848 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8d7002ca70 con 0x7f8d7c069820 2024-03-19T03:03:30.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.075+0000 7f8d7affd640 1 -- 172.21.15.33:0/2817628848 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d70003a50 con 0x7f8d7c069820 2024-03-19T03:03:30.079 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.075+0000 7f8d8415b640 1 -- 172.21.15.33:0/2817628848 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d7c1a4b20 con 0x7f8d7c069820 2024-03-19T03:03:30.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.076+0000 7f8d8415b640 1 -- 172.21.15.33:0/2817628848 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d7c04f580 con 0x7f8d7c069820 2024-03-19T03:03:30.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.077+0000 7f8d7affd640 1 -- 172.21.15.33:0/2817628848 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f8d70046020 con 0x7f8d7c069820 2024-03-19T03:03:30.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.077+0000 7f8d7affd640 1 --2- 172.21.15.33:0/2817628848 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f8d640a2a10 0x7f8d640a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:30.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.078+0000 7f8d81ef7640 1 --2- 172.21.15.33:0/2817628848 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f8d640a2a10 0x7f8d640a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:30.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.078+0000 7f8d7affd640 1 -- 172.21.15.33:0/2817628848 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7f8d700f1170 con 0x7f8d7c069820 2024-03-19T03:03:30.080 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.078+0000 7f8d81ef7640 1 --2- 172.21.15.33:0/2817628848 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f8d640a2a10 0x7f8d640a4ed0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8d74009b50 tx=0x7f8d74005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:30.083 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.082+0000 7f8d7affd640 1 -- 172.21.15.33:0/2817628848 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8d7004d020 con 0x7f8d7c069820 2024-03-19T03:03:30.291 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.290+0000 7f8d8415b640 1 -- 172.21.15.33:0/2817628848 --> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f8d7c109d40 con 0x7f8d640a2a10 2024-03-19T03:03:30.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.297+0000 7f8d7affd640 1 -- 172.21.15.33:0/2817628848 <== mgr.15618 v2:172.21.15.33:6800/2362023661 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f8d7c109d40 con 0x7f8d640a2a10 2024-03-19T03:03:30.298 INFO:teuthology.orchestra.run.smithi033.stdout:/ 1m 6m3h 2024-03-19T03:03:30.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.298+0000 7f8d8415b640 1 -- 172.21.15.33:0/2817628848 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f8d640a2a10 msgr2=0x7f8d640a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:30.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.298+0000 7f8d8415b640 1 --2- 172.21.15.33:0/2817628848 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f8d640a2a10 0x7f8d640a4ed0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8d74009b50 tx=0x7f8d74005f90 comp rx=0 tx=0).stop 2024-03-19T03:03:30.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.299+0000 7f8d8415b640 1 -- 172.21.15.33:0/2817628848 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d7c069820 msgr2=0x7f8d7c1984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:30.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.299+0000 7f8d8415b640 1 --2- 172.21.15.33:0/2817628848 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d7c069820 0x7f8d7c1984f0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f8d70009230 tx=0x7f8d7002c5d0 comp rx=0 tx=0).stop 2024-03-19T03:03:30.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.299+0000 7f8d8415b640 1 -- 172.21.15.33:0/2817628848 shutdown_connections 2024-03-19T03:03:30.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.299+0000 7f8d8415b640 1 --2- 172.21.15.33:0/2817628848 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f8d640a2a10 0x7f8d640a4ed0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:30.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.299+0000 7f8d8415b640 1 --2- 172.21.15.33:0/2817628848 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d7c107800 0x7f8d7c19c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:30.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.299+0000 7f8d8415b640 1 --2- 172.21.15.33:0/2817628848 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f8d7c069820 0x7f8d7c1984f0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:30.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.299+0000 7f8d8415b640 1 --2- 172.21.15.33:0/2817628848 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d7c068ea0 0x7f8d7c197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:30.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.299+0000 7f8d8415b640 1 -- 172.21.15.33:0/2817628848 >> 172.21.15.33:0/2817628848 conn(0x7f8d7c077570 msgr2=0x7f8d7c0feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:30.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.299+0000 7f8d8415b640 1 -- 172.21.15.33:0/2817628848 shutdown_connections 2024-03-19T03:03:30.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:30.299+0000 7f8d8415b640 1 -- 172.21.15.33:0/2817628848 wait complete. 2024-03-19T03:03:30.311 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-19T03:03:30.322 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-19T03:03:30.516 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:30 smithi155 ceph-mon[35804]: pgmap v312: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:03:30.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:30 smithi155 ceph-mon[35804]: from='client.17754 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:03:30.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:30 smithi033 ceph-mon[31973]: pgmap v312: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:03:30.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:30 smithi033 ceph-mon[31973]: from='client.17754 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:03:30.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:30 smithi096 ceph-mon[35946]: pgmap v312: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:03:30.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:30 smithi096 ceph-mon[35946]: from='client.17754 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:03:31.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:31 smithi155 ceph-mon[35804]: from='client.17760 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:03:31.528 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:31 smithi033 ceph-mon[31973]: from='client.17760 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:03:31.537 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-19T03:03:31.537 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-19T03:03:31.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.654+0000 7f565bfd4640 1 -- 172.21.15.33:0/3168587993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5654102a90 msgr2=0x7f5654102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:31.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.654+0000 7f565bfd4640 1 --2- 172.21.15.33:0/3168587993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5654102a90 0x7f5654102e70 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f5650009ca0 tx=0x7f565002c180 comp rx=0 tx=0).stop 2024-03-19T03:03:31.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.654+0000 7f565bfd4640 1 -- 172.21.15.33:0/3168587993 shutdown_connections 2024-03-19T03:03:31.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.654+0000 7f565bfd4640 1 --2- 172.21.15.33:0/3168587993 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56541033b0 0x7f565410b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.654+0000 7f565bfd4640 1 --2- 172.21.15.33:0/3168587993 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f565410be60 0x7f565410e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.654+0000 7f565bfd4640 1 --2- 172.21.15.33:0/3168587993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5654102a90 0x7f5654102e70 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.654+0000 7f565bfd4640 1 -- 172.21.15.33:0/3168587993 >> 172.21.15.33:0/3168587993 conn(0x7f5654077880 msgr2=0x7f5654077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:31.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.654+0000 7f565bfd4640 1 -- 172.21.15.33:0/3168587993 shutdown_connections 2024-03-19T03:03:31.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.654+0000 7f565bfd4640 1 -- 172.21.15.33:0/3168587993 wait complete. 2024-03-19T03:03:31.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.654+0000 7f565bfd4640 1 Processor -- start 2024-03-19T03:03:31.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.654+0000 7f565bfd4640 1 -- start start 2024-03-19T03:03:31.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.655+0000 7f565bfd4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5654102a90 0x7f56541006e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:31.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.655+0000 7f565bfd4640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56541033b0 0x7f5654100c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:31.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.655+0000 7f565bfd4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f565410be60 0x7f5654101160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:31.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.655+0000 7f565bfd4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5654111ae0 con 0x7f565410be60 2024-03-19T03:03:31.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.655+0000 7f565bfd4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5654111960 con 0x7f5654102a90 2024-03-19T03:03:31.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.655+0000 7f565bfd4640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5654111c60 con 0x7f56541033b0 2024-03-19T03:03:31.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.655+0000 7f565a571640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f565410be60 0x7f5654101160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:31.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.655+0000 7f565a571640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f565410be60 0x7f5654101160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41152/0 (socket says 172.21.15.33:41152) 2024-03-19T03:03:31.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.655+0000 7f5659d70640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5654102a90 0x7f56541006e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:31.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.655+0000 7f565a571640 1 -- 172.21.15.33:0/712228843 learned_addr learned my addr 172.21.15.33:0/712228843 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:31.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.655+0000 7f565956f640 1 --2- 172.21.15.33:0/712228843 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56541033b0 0x7f5654100c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:31.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.655+0000 7f565a571640 1 -- 172.21.15.33:0/712228843 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56541033b0 msgr2=0x7f5654100c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:31.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.655+0000 7f565a571640 1 --2- 172.21.15.33:0/712228843 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56541033b0 0x7f5654100c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.655+0000 7f565a571640 1 -- 172.21.15.33:0/712228843 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5654102a90 msgr2=0x7f56541006e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:31.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.655+0000 7f565a571640 1 --2- 172.21.15.33:0/712228843 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5654102a90 0x7f56541006e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.655+0000 7f565a571640 1 -- 172.21.15.33:0/712228843 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56541a6560 con 0x7f565410be60 2024-03-19T03:03:31.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.655+0000 7f5659d70640 1 --2- 172.21.15.33:0/712228843 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5654102a90 0x7f56541006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:31.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.655+0000 7f565a571640 1 --2- 172.21.15.33:0/712228843 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f565410be60 0x7f5654101160 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f564c00e970 tx=0x7f564c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:31.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.655+0000 7f564affd640 1 -- 172.21.15.33:0/712228843 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f564c00ccc0 con 0x7f565410be60 2024-03-19T03:03:31.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.655+0000 7f564affd640 1 -- 172.21.15.33:0/712228843 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f564c00ce60 con 0x7f565410be60 2024-03-19T03:03:31.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.655+0000 7f565bfd4640 1 -- 172.21.15.33:0/712228843 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56541a6850 con 0x7f565410be60 2024-03-19T03:03:31.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.655+0000 7f564affd640 1 -- 172.21.15.33:0/712228843 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f564c010430 con 0x7f565410be60 2024-03-19T03:03:31.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.656+0000 7f565bfd4640 1 -- 172.21.15.33:0/712228843 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56541a6da0 con 0x7f565410be60 2024-03-19T03:03:31.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.657+0000 7f564affd640 1 -- 172.21.15.33:0/712228843 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f564c004ff0 con 0x7f565410be60 2024-03-19T03:03:31.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.658+0000 7f565ebc1740 1 Processor -- start 2024-03-19T03:03:31.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.658+0000 7f565ebc1740 1 -- start start 2024-03-19T03:03:31.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.658+0000 7f564affd640 1 --2- 172.21.15.33:0/712228843 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f56400a29f0 0x7f56400a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:31.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.658+0000 7f565ebc1740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ceb8482a30 0x55ceb8482e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:31.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.658+0000 7f565ebc1740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ceb8483350 0x55ceb848b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:31.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.658+0000 7f565ebc1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb848bf30 0x55ceb848e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:31.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.658+0000 7f565ebc1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ceb83e6500 con 0x55ceb848bf30 2024-03-19T03:03:31.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.658+0000 7f565ebc1740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55ceb838ebe0 con 0x55ceb8482a30 2024-03-19T03:03:31.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.658+0000 7f565ebc1740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ceb83e25a0 con 0x55ceb8483350 2024-03-19T03:03:31.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.658+0000 7f565956f640 1 --2- 172.21.15.33:0/712228843 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f56400a29f0 0x7f56400a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:31.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.658+0000 7f565a571640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb848bf30 0x55ceb848e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:31.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.659+0000 7f5659d70640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ceb8482a30 0x55ceb8482e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:31.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.659+0000 7f565a571640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb848bf30 0x55ceb848e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41164/0 (socket says 172.21.15.33:41164) 2024-03-19T03:03:31.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.659+0000 7f565a571640 1 -- 172.21.15.33:0/1702266369 learned_addr learned my addr 172.21.15.33:0/1702266369 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:31.667 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.659+0000 7f5659d70640 1 --2- 172.21.15.33:0/1702266369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ceb8482a30 0x55ceb8482e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:53422/0 (socket says 172.21.15.33:53422) 2024-03-19T03:03:31.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.659+0000 7f565956f640 1 --2- 172.21.15.33:0/1702266369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ceb8483350 0x55ceb848b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:31.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.659+0000 7f564affd640 1 -- 172.21.15.33:0/712228843 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7f564c0cf4c0 con 0x7f565410be60 2024-03-19T03:03:31.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.659+0000 7f5659d70640 1 -- 172.21.15.33:0/1702266369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ceb8483350 msgr2=0x55ceb848b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:31.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.659+0000 7f5659d70640 1 --2- 172.21.15.33:0/1702266369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ceb8483350 0x55ceb848b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.659+0000 7f5659d70640 1 -- 172.21.15.33:0/1702266369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb848bf30 msgr2=0x55ceb848e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:31.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.659+0000 7f5659d70640 1 --2- 172.21.15.33:0/1702266369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb848bf30 0x55ceb848e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.659+0000 7f5659d70640 1 -- 172.21.15.33:0/1702266369 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ceb83b0000 con 0x55ceb8482a30 2024-03-19T03:03:31.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.659+0000 7f565a571640 1 --2- 172.21.15.33:0/1702266369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb848bf30 0x55ceb848e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:31.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.659+0000 7f565956f640 1 --2- 172.21.15.33:0/712228843 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f56400a29f0 0x7f56400a4eb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5644009710 tx=0x7f5644009af0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:31.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.659+0000 7f5659d70640 1 --2- 172.21.15.33:0/1702266369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ceb8482a30 0x55ceb8482e10 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5650006010 tx=0x7f56500098c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ac1b4595b484b78 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:31.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.659+0000 7f563ffff640 1 -- 172.21.15.33:0/1702266369 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5650043070 con 0x55ceb8482a30 2024-03-19T03:03:31.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.659+0000 7f563ffff640 1 -- 172.21.15.33:0/1702266369 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5650003e20 con 0x55ceb8482a30 2024-03-19T03:03:31.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.659+0000 7f563ffff640 1 -- 172.21.15.33:0/1702266369 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5650039040 con 0x55ceb8482a30 2024-03-19T03:03:31.668 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.660+0000 7f565ebc1740 1 -- 172.21.15.33:0/1702266369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ceb8482a30 msgr2=0x55ceb8482e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:31.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.660+0000 7f565ebc1740 1 --2- 172.21.15.33:0/1702266369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ceb8482a30 0x55ceb8482e10 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5650006010 tx=0x7f56500098c0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.660+0000 7f565ebc1740 1 -- 172.21.15.33:0/1702266369 shutdown_connections 2024-03-19T03:03:31.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.660+0000 7f565ebc1740 1 --2- 172.21.15.33:0/1702266369 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ceb8483350 0x55ceb848b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.660+0000 7f565ebc1740 1 --2- 172.21.15.33:0/1702266369 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb848bf30 0x55ceb848e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.660+0000 7f565ebc1740 1 --2- 172.21.15.33:0/1702266369 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ceb8482a30 0x55ceb8482e10 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.660+0000 7f565ebc1740 1 -- 172.21.15.33:0/1702266369 >> 172.21.15.33:0/1702266369 conn(0x55ceb847d160 msgr2=0x55ceb847d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:31.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.660+0000 7f565ebc1740 1 -- 172.21.15.33:0/1702266369 shutdown_connections 2024-03-19T03:03:31.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.660+0000 7f565ebc1740 1 -- 172.21.15.33:0/1702266369 wait complete. 2024-03-19T03:03:31.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.660+0000 7f565ebc1740 1 -- start start 2024-03-19T03:03:31.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.661+0000 7f565ebc1740 1 Processor -- start 2024-03-19T03:03:31.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.661+0000 7f565ebc1740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ceb8482a30 0x55ceb8488e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:31.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.661+0000 7f565ebc1740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ceb8483350 0x55ceb8489370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:31.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.661+0000 7f565ebc1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb848bf30 0x55ceb84898b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:31.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.661+0000 7f565ebc1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ceb848ecc0 con 0x55ceb848bf30 2024-03-19T03:03:31.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.661+0000 7f565ebc1740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55ceb83a2160 con 0x55ceb8483350 2024-03-19T03:03:31.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.661+0000 7f565ebc1740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ceb848ee40 con 0x55ceb8482a30 2024-03-19T03:03:31.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.661+0000 7f565ebc1740 10 client.0 _ll_register_callbacks cb 0x55ceb83dea40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:03:31.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.661+0000 7f565a571640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb848bf30 0x55ceb84898b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:31.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.661+0000 7f565956f640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ceb8483350 0x55ceb8489370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:31.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.661+0000 7f565a571640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb848bf30 0x55ceb84898b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41174/0 (socket says 172.21.15.33:41174) 2024-03-19T03:03:31.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.661+0000 7f565956f640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ceb8483350 0x55ceb8489370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:53424/0 (socket says 172.21.15.33:53424) 2024-03-19T03:03:31.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.661+0000 7f565a571640 1 -- 172.21.15.33:0/2477063427 learned_addr learned my addr 172.21.15.33:0/2477063427 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:31.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.661+0000 7f5659d70640 1 --2- 172.21.15.33:0/2477063427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ceb8482a30 0x55ceb8488e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:31.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.661+0000 7f565a571640 1 -- 172.21.15.33:0/2477063427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ceb8482a30 msgr2=0x55ceb8488e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:31.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.661+0000 7f565a571640 1 --2- 172.21.15.33:0/2477063427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ceb8482a30 0x55ceb8488e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.661+0000 7f565a571640 1 -- 172.21.15.33:0/2477063427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ceb8483350 msgr2=0x55ceb8489370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:31.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.661+0000 7f565a571640 1 --2- 172.21.15.33:0/2477063427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ceb8483350 0x55ceb8489370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.661+0000 7f565a571640 1 -- 172.21.15.33:0/2477063427 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ceb81c47d0 con 0x55ceb848bf30 2024-03-19T03:03:31.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.661+0000 7f565956f640 1 --2- 172.21.15.33:0/2477063427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ceb8483350 0x55ceb8489370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:31.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.662+0000 7f565a571640 1 --2- 172.21.15.33:0/2477063427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb848bf30 0x55ceb84898b0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f564c09b780 tx=0x7f564c016f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a0db67ecc5a7c8f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:31.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.662+0000 7f563d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:03:31.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.662+0000 7f563d7fa640 1 -- 172.21.15.33:0/2477063427 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f564c09b330 con 0x55ceb848bf30 2024-03-19T03:03:31.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.662+0000 7f563d7fa640 1 -- 172.21.15.33:0/2477063427 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7f564c097e60 con 0x55ceb848bf30 2024-03-19T03:03:31.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.662+0000 7f565ebc1740 1 -- 172.21.15.33:0/2477063427 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5650003e20 con 0x55ceb848bf30 2024-03-19T03:03:31.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.662+0000 7f563d7fa640 1 -- 172.21.15.33:0/2477063427 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f564c063ba0 con 0x55ceb848bf30 2024-03-19T03:03:31.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.662+0000 7f565ebc1740 10 client.17772 fetch_fsmap learned FSMap version 25 2024-03-19T03:03:31.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.662+0000 7f565ebc1740 1 -- 172.21.15.33:0/2477063427 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ceb80fa3f0 con 0x55ceb848bf30 2024-03-19T03:03:31.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.662+0000 7f563d7fa640 1 -- 172.21.15.33:0/2477063427 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5650003e20 con 0x55ceb848bf30 2024-03-19T03:03:31.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.662+0000 7f563d7fa640 1 -- 172.21.15.33:0/2477063427 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f564c0bb040 con 0x55ceb848bf30 2024-03-19T03:03:31.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.662+0000 7f565ebc1740 10 client.17772 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:03:31.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.662+0000 7f565ebc1740 10 client.17772 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:03:31.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.662+0000 7f565ebc1740 20 client.17772 populate_metadata read hostname 'smithi033' 2024-03-19T03:03:31.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.662+0000 7f565ebc1740 1 --2- 172.21.15.33:0/2477063427 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55ceb849c0d0 0x55ceb849e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:31.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.662+0000 7f565ebc1740 4 client.17772 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:03:31.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.662+0000 7f565ebc1740 1 -- 172.21.15.33:0/2477063427 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ceb83f3a40 con 0x55ceb849c0d0 2024-03-19T03:03:31.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.663+0000 7f5659d70640 1 --2- 172.21.15.33:0/2477063427 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55ceb849c0d0 0x55ceb849e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:31.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.663+0000 7f563d7fa640 10 client.17772 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:03:31.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.663+0000 7f5659d70640 1 --2- 172.21.15.33:0/2477063427 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55ceb849c0d0 0x55ceb849e4e0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe3421142802e450 server_cookie=1495f628305037fa in_seq=0 out_seq=0 2024-03-19T03:03:31.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.664+0000 7f563d7fa640 1 -- 172.21.15.33:0/2477063427 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ceb83f3a40 con 0x55ceb849c0d0 2024-03-19T03:03:31.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.664+0000 7f563d7fa640 10 client.17772 handle_command_reply: tid=0 2024-03-19T03:03:31.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.664+0000 7f565ebc1740 1 client.17772 shutdown 2024-03-19T03:03:31.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.664+0000 7f565ebc1740 1 -- 172.21.15.33:0/2477063427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb848bf30 msgr2=0x55ceb84898b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:31.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.664+0000 7f565ebc1740 1 --2- 172.21.15.33:0/2477063427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb848bf30 0x55ceb84898b0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f564c09b780 tx=0x7f564c016f20 comp rx=0 tx=0).stop 2024-03-19T03:03:31.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.664+0000 7f565ebc1740 1 -- 172.21.15.33:0/2477063427 shutdown_connections 2024-03-19T03:03:31.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.664+0000 7f565ebc1740 1 --2- 172.21.15.33:0/2477063427 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55ceb849c0d0 0x55ceb849e4e0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.664+0000 7f565ebc1740 1 --2- 172.21.15.33:0/2477063427 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb848bf30 0x55ceb84898b0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.664+0000 7f565ebc1740 1 --2- 172.21.15.33:0/2477063427 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ceb8483350 0x55ceb8489370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.664+0000 7f565ebc1740 1 --2- 172.21.15.33:0/2477063427 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ceb8482a30 0x55ceb8488e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.664+0000 7f563d7fa640 0 client.17772 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:03:31.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.664+0000 7f565ebc1740 1 -- 172.21.15.33:0/2477063427 >> 172.21.15.33:0/2477063427 conn(0x55ceb847d160 msgr2=0x55ceb847eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:31.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.664+0000 7f565ebc1740 1 -- 172.21.15.33:0/2477063427 shutdown_connections 2024-03-19T03:03:31.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.664+0000 7f565ebc1740 1 -- 172.21.15.33:0/2477063427 wait complete. 2024-03-19T03:03:31.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.664+0000 7f565ebc1740 20 client.17772 trim_cache size 0 max 16384 2024-03-19T03:03:31.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.681+0000 7f565ebc1740 1 Processor -- start 2024-03-19T03:03:31.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.681+0000 7f565ebc1740 1 -- start start 2024-03-19T03:03:31.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.682+0000 7f565ebc1740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ceb8482a30 0x55ceb848b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:31.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.682+0000 7f565ebc1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb8483350 0x55ceb8497a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:31.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.682+0000 7f565ebc1740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ceb848bf30 0x55ceb84a3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:31.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.682+0000 7f565ebc1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f56541b3770 con 0x55ceb8483350 2024-03-19T03:03:31.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.682+0000 7f565ebc1740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f56541b1040 con 0x55ceb8482a30 2024-03-19T03:03:31.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.682+0000 7f565ebc1740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f56541aeeb0 con 0x55ceb848bf30 2024-03-19T03:03:31.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.682+0000 7f565956f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb8483350 0x55ceb8497a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:31.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.682+0000 7f565956f640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb8483350 0x55ceb8497a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41184/0 (socket says 172.21.15.33:41184) 2024-03-19T03:03:31.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.682+0000 7f565956f640 1 -- 172.21.15.33:0/1264979976 learned_addr learned my addr 172.21.15.33:0/1264979976 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:31.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.682+0000 7f5659d70640 1 --2- 172.21.15.33:0/1264979976 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ceb8482a30 0x55ceb848b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:31.685 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.682+0000 7f565a571640 1 --2- 172.21.15.33:0/1264979976 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ceb848bf30 0x55ceb84a3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:31.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.682+0000 7f565956f640 1 -- 172.21.15.33:0/1264979976 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ceb848bf30 msgr2=0x55ceb84a3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:31.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.682+0000 7f565956f640 1 --2- 172.21.15.33:0/1264979976 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ceb848bf30 0x55ceb84a3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.682+0000 7f565956f640 1 -- 172.21.15.33:0/1264979976 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ceb8482a30 msgr2=0x55ceb848b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:31.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.682+0000 7f565956f640 1 --2- 172.21.15.33:0/1264979976 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ceb8482a30 0x55ceb848b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.682+0000 7f565956f640 1 -- 172.21.15.33:0/1264979976 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ceb80ba0f0 con 0x55ceb8483350 2024-03-19T03:03:31.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.682+0000 7f565956f640 1 --2- 172.21.15.33:0/1264979976 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb8483350 0x55ceb8497a60 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f5644013350 tx=0x7f5644021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=975ae88fc54b506c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:31.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.682+0000 7f563f7fe640 1 -- 172.21.15.33:0/1264979976 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56440044c0 con 0x55ceb8483350 2024-03-19T03:03:31.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.682+0000 7f563f7fe640 1 -- 172.21.15.33:0/1264979976 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5644003aa0 con 0x55ceb8483350 2024-03-19T03:03:31.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.682+0000 7f563f7fe640 1 -- 172.21.15.33:0/1264979976 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5644010a70 con 0x55ceb8483350 2024-03-19T03:03:31.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.683+0000 7f565ebc1740 1 -- 172.21.15.33:0/1264979976 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb8483350 msgr2=0x55ceb8497a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:31.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.683+0000 7f565ebc1740 1 --2- 172.21.15.33:0/1264979976 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb8483350 0x55ceb8497a60 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f5644013350 tx=0x7f5644021000 comp rx=0 tx=0).stop 2024-03-19T03:03:31.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.683+0000 7f565ebc1740 1 -- 172.21.15.33:0/1264979976 shutdown_connections 2024-03-19T03:03:31.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.683+0000 7f565ebc1740 1 --2- 172.21.15.33:0/1264979976 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ceb848bf30 0x55ceb84a3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.683+0000 7f565ebc1740 1 --2- 172.21.15.33:0/1264979976 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb8483350 0x55ceb8497a60 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.686 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.683+0000 7f565ebc1740 1 --2- 172.21.15.33:0/1264979976 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ceb8482a30 0x55ceb848b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.683+0000 7f565ebc1740 1 -- 172.21.15.33:0/1264979976 >> 172.21.15.33:0/1264979976 conn(0x55ceb847d160 msgr2=0x55ceb8488df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:31.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.683+0000 7f565ebc1740 1 -- 172.21.15.33:0/1264979976 shutdown_connections 2024-03-19T03:03:31.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.683+0000 7f565ebc1740 1 -- 172.21.15.33:0/1264979976 wait complete. 2024-03-19T03:03:31.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.683+0000 7f565ebc1740 1 -- start start 2024-03-19T03:03:31.687 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.683+0000 7f565ebc1740 1 Processor -- start 2024-03-19T03:03:31.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.684+0000 7f565ebc1740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb8482a30 0x55ceb84943b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:31.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.684+0000 7f565ebc1740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ceb8483350 0x55ceb848f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:31.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.684+0000 7f565ebc1740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ceb848bf30 0x55ceb848fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:31.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.684+0000 7f565ebc1740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f56541af500 con 0x55ceb8482a30 2024-03-19T03:03:31.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.684+0000 7f565ebc1740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f56541b0970 con 0x55ceb848bf30 2024-03-19T03:03:31.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.684+0000 7f565ebc1740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f56541afbd0 con 0x55ceb8483350 2024-03-19T03:03:31.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.684+0000 7f565ebc1740 10 client.0 _ll_register_callbacks cb 0x7f564c005680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:03:31.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.684+0000 7f5659d70640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb8482a30 0x55ceb84943b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:31.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.684+0000 7f5659d70640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb8482a30 0x55ceb84943b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41200/0 (socket says 172.21.15.33:41200) 2024-03-19T03:03:31.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.684+0000 7f5659d70640 1 -- 172.21.15.33:0/2144538801 learned_addr learned my addr 172.21.15.33:0/2144538801 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:31.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.684+0000 7f565956f640 1 --2- 172.21.15.33:0/2144538801 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ceb8483350 0x55ceb848f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:31.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.684+0000 7f565a571640 1 --2- 172.21.15.33:0/2144538801 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ceb848bf30 0x55ceb848fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:31.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.684+0000 7f5659d70640 1 -- 172.21.15.33:0/2144538801 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ceb8483350 msgr2=0x55ceb848f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:31.691 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.684+0000 7f5659d70640 1 --2- 172.21.15.33:0/2144538801 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ceb8483350 0x55ceb848f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.684+0000 7f5659d70640 1 -- 172.21.15.33:0/2144538801 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ceb848bf30 msgr2=0x55ceb848fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:31.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.684+0000 7f5659d70640 1 --2- 172.21.15.33:0/2144538801 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ceb848bf30 0x55ceb848fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.684+0000 7f5659d70640 1 -- 172.21.15.33:0/2144538801 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ceb8490440 con 0x55ceb8482a30 2024-03-19T03:03:31.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.684+0000 7f565a571640 1 --2- 172.21.15.33:0/2144538801 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ceb848bf30 0x55ceb848fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:03:31.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.684+0000 7f5659d70640 1 --2- 172.21.15.33:0/2144538801 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb8482a30 0x55ceb84943b0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f565002f020 tx=0x7f565002c180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e146adcc4977265 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:31.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.684+0000 7f563d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:03:31.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.685+0000 7f563d7fa640 1 -- 172.21.15.33:0/2144538801 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5650039040 con 0x55ceb8482a30 2024-03-19T03:03:31.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.685+0000 7f563d7fa640 1 -- 172.21.15.33:0/2144538801 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7f5650051b10 con 0x55ceb8482a30 2024-03-19T03:03:31.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.685+0000 7f565ebc1740 1 -- 172.21.15.33:0/2144538801 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5644003aa0 con 0x55ceb8482a30 2024-03-19T03:03:31.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.685+0000 7f563d7fa640 1 -- 172.21.15.33:0/2144538801 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f565005f050 con 0x55ceb8482a30 2024-03-19T03:03:31.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.685+0000 7f565ebc1740 10 client.17778 fetch_fsmap learned FSMap version 25 2024-03-19T03:03:31.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.685+0000 7f565ebc1740 1 -- 172.21.15.33:0/2144538801 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ceb8502450 con 0x55ceb8482a30 2024-03-19T03:03:31.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.685+0000 7f563d7fa640 1 -- 172.21.15.33:0/2144538801 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5644003aa0 con 0x55ceb8482a30 2024-03-19T03:03:31.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.685+0000 7f563d7fa640 1 -- 172.21.15.33:0/2144538801 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f5650051070 con 0x55ceb8482a30 2024-03-19T03:03:31.692 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.685+0000 7f565ebc1740 10 client.17778 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:03:31.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.685+0000 7f565ebc1740 10 client.17778 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:03:31.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.685+0000 7f565ebc1740 20 client.17778 populate_metadata read hostname 'smithi033' 2024-03-19T03:03:31.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.685+0000 7f565ebc1740 1 --2- 172.21.15.33:0/2144538801 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55ceb85027b0 0x55ceb8504ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:31.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.685+0000 7f565ebc1740 4 client.17778 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-19T03:03:31.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.685+0000 7f565ebc1740 1 -- 172.21.15.33:0/2144538801 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ceb83edd90 con 0x55ceb85027b0 2024-03-19T03:03:31.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.685+0000 7f565956f640 1 --2- 172.21.15.33:0/2144538801 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55ceb85027b0 0x55ceb8504ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:31.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.686+0000 7f563d7fa640 10 client.17778 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:03:31.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.686+0000 7f565956f640 1 --2- 172.21.15.33:0/2144538801 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55ceb85027b0 0x55ceb8504ba0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fac4c0dd2bb9560 server_cookie=33eef981ccc25e26 in_seq=0 out_seq=0 2024-03-19T03:03:31.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.686+0000 7f563d7fa640 1 -- 172.21.15.33:0/2144538801 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ceb83edd90 con 0x55ceb85027b0 2024-03-19T03:03:31.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.686+0000 7f563d7fa640 10 client.17778 handle_command_reply: tid=0 2024-03-19T03:03:31.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.686+0000 7f565ebc1740 1 client.17778 shutdown 2024-03-19T03:03:31.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.686+0000 7f565ebc1740 1 -- 172.21.15.33:0/2144538801 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb8482a30 msgr2=0x55ceb84943b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:31.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.686+0000 7f565ebc1740 1 --2- 172.21.15.33:0/2144538801 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb8482a30 0x55ceb84943b0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f565002f020 tx=0x7f565002c180 comp rx=0 tx=0).stop 2024-03-19T03:03:31.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.686+0000 7f565ebc1740 1 -- 172.21.15.33:0/2144538801 shutdown_connections 2024-03-19T03:03:31.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.686+0000 7f565ebc1740 1 --2- 172.21.15.33:0/2144538801 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x55ceb85027b0 0x55ceb8504ba0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.686+0000 7f565ebc1740 1 --2- 172.21.15.33:0/2144538801 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ceb8483350 0x55ceb848f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.686+0000 7f565ebc1740 1 --2- 172.21.15.33:0/2144538801 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ceb848bf30 0x55ceb848fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.686+0000 7f565ebc1740 1 --2- 172.21.15.33:0/2144538801 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ceb8482a30 0x55ceb84943b0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.686+0000 7f563d7fa640 0 client.17778 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:03:31.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.686+0000 7f565ebc1740 1 -- 172.21.15.33:0/2144538801 >> 172.21.15.33:0/2144538801 conn(0x55ceb847d160 msgr2=0x55ceb8489680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:31.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.686+0000 7f565ebc1740 1 -- 172.21.15.33:0/2144538801 shutdown_connections 2024-03-19T03:03:31.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.686+0000 7f565ebc1740 1 -- 172.21.15.33:0/2144538801 wait complete. 2024-03-19T03:03:31.694 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.686+0000 7f565ebc1740 20 client.17778 trim_cache size 0 max 16384 2024-03-19T03:03:31.694 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T03:03:31.695 INFO:teuthology.orchestra.run.smithi033.stdout: "status": "no active scrubs running", 2024-03-19T03:03:31.695 INFO:teuthology.orchestra.run.smithi033.stdout: "scrubs": {} 2024-03-19T03:03:31.695 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T03:03:31.695 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.687+0000 7f565bfd4640 1 -- 172.21.15.33:0/712228843 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f56400a29f0 msgr2=0x7f56400a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:31.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.687+0000 7f565bfd4640 1 --2- 172.21.15.33:0/712228843 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f56400a29f0 0x7f56400a4eb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5644009710 tx=0x7f5644009af0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.687+0000 7f565bfd4640 1 -- 172.21.15.33:0/712228843 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f565410be60 msgr2=0x7f5654101160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:31.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.687+0000 7f565bfd4640 1 --2- 172.21.15.33:0/712228843 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f565410be60 0x7f5654101160 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f564c00e970 tx=0x7f564c00ed70 comp rx=0 tx=0).stop 2024-03-19T03:03:31.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.687+0000 7f565bfd4640 1 -- 172.21.15.33:0/712228843 shutdown_connections 2024-03-19T03:03:31.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.687+0000 7f565bfd4640 1 --2- 172.21.15.33:0/712228843 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f56400a29f0 0x7f56400a4eb0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.687+0000 7f565bfd4640 1 --2- 172.21.15.33:0/712228843 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56541033b0 0x7f5654100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.687+0000 7f565bfd4640 1 --2- 172.21.15.33:0/712228843 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f565410be60 0x7f5654101160 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.687+0000 7f565bfd4640 1 --2- 172.21.15.33:0/712228843 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5654102a90 0x7f56541006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.687+0000 7f565bfd4640 1 -- 172.21.15.33:0/712228843 >> 172.21.15.33:0/712228843 conn(0x7f5654077880 msgr2=0x7f56540fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:31.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.687+0000 7f565bfd4640 1 -- 172.21.15.33:0/712228843 shutdown_connections 2024-03-19T03:03:31.696 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.687+0000 7f565bfd4640 1 -- 172.21.15.33:0/712228843 wait complete. 2024-03-19T03:03:31.699 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-19T03:03:31.699 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:03:31.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:31 smithi096 ceph-mon[35946]: from='client.17760 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:03:31.803 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.800+0000 7f96c804b640 1 -- 172.21.15.33:0/2562538606 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f96c01077c0 msgr2=0x7f96c0107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:31.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.800+0000 7f96c804b640 1 --2- 172.21.15.33:0/2562538606 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f96c01077c0 0x7f96c0107ba0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f96bc009930 tx=0x7f96bc02bc30 comp rx=0 tx=0).stop 2024-03-19T03:03:31.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.800+0000 7f96c804b640 1 -- 172.21.15.33:0/2562538606 shutdown_connections 2024-03-19T03:03:31.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.800+0000 7f96c804b640 1 --2- 172.21.15.33:0/2562538606 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96c0102170 0x7f96c010e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.800+0000 7f96c804b640 1 --2- 172.21.15.33:0/2562538606 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96c01017b0 0x7f96c0101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.800+0000 7f96c804b640 1 --2- 172.21.15.33:0/2562538606 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f96c01077c0 0x7f96c0107ba0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.800+0000 7f96c804b640 1 -- 172.21.15.33:0/2562538606 >> 172.21.15.33:0/2562538606 conn(0x7f96c00fd2c0 msgr2=0x7f96c00ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:31.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.800+0000 7f96c804b640 1 -- 172.21.15.33:0/2562538606 shutdown_connections 2024-03-19T03:03:31.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.800+0000 7f96c804b640 1 -- 172.21.15.33:0/2562538606 wait complete. 2024-03-19T03:03:31.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.801+0000 7f96c804b640 1 Processor -- start 2024-03-19T03:03:31.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.801+0000 7f96c804b640 1 -- start start 2024-03-19T03:03:31.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.801+0000 7f96c804b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96c01017b0 0x7f96c0195e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:31.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.801+0000 7f96c804b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96c0102170 0x7f96c0196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:31.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.801+0000 7f96c804b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f96c01077c0 0x7f96c019a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:31.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.801+0000 7f96c804b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f96c010f8c0 con 0x7f96c01077c0 2024-03-19T03:03:31.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.801+0000 7f96c804b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f96c010f740 con 0x7f96c01017b0 2024-03-19T03:03:31.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.801+0000 7f96c804b640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f96c010fa40 con 0x7f96c0102170 2024-03-19T03:03:31.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.801+0000 7f96c65e8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f96c01077c0 0x7f96c019a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:31.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.801+0000 7f96c65e8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f96c01077c0 0x7f96c019a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41210/0 (socket says 172.21.15.33:41210) 2024-03-19T03:03:31.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.801+0000 7f96c65e8640 1 -- 172.21.15.33:0/3725033837 learned_addr learned my addr 172.21.15.33:0/3725033837 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:31.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.801+0000 7f96c5de7640 1 --2- 172.21.15.33:0/3725033837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96c01017b0 0x7f96c0195e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:31.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.801+0000 7f96c55e6640 1 --2- 172.21.15.33:0/3725033837 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96c0102170 0x7f96c0196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:31.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.801+0000 7f96c65e8640 1 -- 172.21.15.33:0/3725033837 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96c0102170 msgr2=0x7f96c0196380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:31.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.801+0000 7f96c65e8640 1 --2- 172.21.15.33:0/3725033837 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96c0102170 0x7f96c0196380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.801+0000 7f96c65e8640 1 -- 172.21.15.33:0/3725033837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96c01017b0 msgr2=0x7f96c0195e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:31.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.801+0000 7f96c65e8640 1 --2- 172.21.15.33:0/3725033837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96c01017b0 0x7f96c0195e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:31.805 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.801+0000 7f96c65e8640 1 -- 172.21.15.33:0/3725033837 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96c019ae30 con 0x7f96c01077c0 2024-03-19T03:03:31.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.802+0000 7f96c65e8640 1 --2- 172.21.15.33:0/3725033837 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f96c01077c0 0x7f96c019a700 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f96b800eb10 tx=0x7f96b800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:31.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.802+0000 7f96b6ffd640 1 -- 172.21.15.33:0/3725033837 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b8004b20 con 0x7f96c01077c0 2024-03-19T03:03:31.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.802+0000 7f96b6ffd640 1 -- 172.21.15.33:0/3725033837 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f96b8004cc0 con 0x7f96c01077c0 2024-03-19T03:03:31.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.802+0000 7f96c804b640 1 -- 172.21.15.33:0/3725033837 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96c019b120 con 0x7f96c01077c0 2024-03-19T03:03:31.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.802+0000 7f96b6ffd640 1 -- 172.21.15.33:0/3725033837 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b8010690 con 0x7f96c01077c0 2024-03-19T03:03:31.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.802+0000 7f96c804b640 1 -- 172.21.15.33:0/3725033837 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96c01a29b0 con 0x7f96c01077c0 2024-03-19T03:03:31.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.803+0000 7f96c804b640 1 -- 172.21.15.33:0/3725033837 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96c0068a60 con 0x7f96c01077c0 2024-03-19T03:03:31.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.804+0000 7f96b6ffd640 1 -- 172.21.15.33:0/3725033837 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f96b8005000 con 0x7f96c01077c0 2024-03-19T03:03:31.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.804+0000 7f96b6ffd640 1 --2- 172.21.15.33:0/3725033837 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f96ac0a29f0 0x7f96ac0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:31.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.805+0000 7f96c5de7640 1 --2- 172.21.15.33:0/3725033837 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f96ac0a29f0 0x7f96ac0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:31.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.805+0000 7f96b6ffd640 1 -- 172.21.15.33:0/3725033837 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7f96b80cfb30 con 0x7f96c01077c0 2024-03-19T03:03:31.807 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.805+0000 7f96c5de7640 1 --2- 172.21.15.33:0/3725033837 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f96ac0a29f0 0x7f96ac0a4eb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f96bc009900 tx=0x7f96bc005a20 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:31.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:31.808+0000 7f96b6ffd640 1 -- 172.21.15.33:0/3725033837 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f96b809c4c0 con 0x7f96c01077c0 2024-03-19T03:03:32.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.031+0000 7f96c804b640 1 -- 172.21.15.33:0/3725033837 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f96c002d1f0 con 0x7f96c01077c0 2024-03-19T03:03:32.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.032+0000 7f96b6ffd640 1 -- 172.21.15.33:0/3725033837 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11867 (secure 0 0 0) 0x7f96b809bdc0 con 0x7f96c01077c0 2024-03-19T03:03:32.033 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T03:03:32.034 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:50:52.201798+0000","modified":"2024-03-19T02:53:11.748697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24488,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/3731232169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":3731232169},{"type":"v1","addr":"172.21.15.96:6839","nonce":3731232169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24457,24488,24509]},"id":1}]} 2024-03-19T03:03:32.034 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-03-19T03:03:32.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.034+0000 7f96c804b640 1 -- 172.21.15.33:0/3725033837 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f96ac0a29f0 msgr2=0x7f96ac0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.034+0000 7f96c804b640 1 --2- 172.21.15.33:0/3725033837 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f96ac0a29f0 0x7f96ac0a4eb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f96bc009900 tx=0x7f96bc005a20 comp rx=0 tx=0).stop 2024-03-19T03:03:32.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.034+0000 7f96c804b640 1 -- 172.21.15.33:0/3725033837 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f96c01077c0 msgr2=0x7f96c019a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.034+0000 7f96c804b640 1 --2- 172.21.15.33:0/3725033837 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f96c01077c0 0x7f96c019a700 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f96b800eb10 tx=0x7f96b800c2d0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.034+0000 7f96c804b640 1 -- 172.21.15.33:0/3725033837 shutdown_connections 2024-03-19T03:03:32.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.034+0000 7f96c804b640 1 --2- 172.21.15.33:0/3725033837 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f96ac0a29f0 0x7f96ac0a4eb0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.034+0000 7f96c804b640 1 --2- 172.21.15.33:0/3725033837 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f96c0102170 0x7f96c0196380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.034+0000 7f96c804b640 1 --2- 172.21.15.33:0/3725033837 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f96c01077c0 0x7f96c019a700 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.034+0000 7f96c804b640 1 --2- 172.21.15.33:0/3725033837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f96c01017b0 0x7f96c0195e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.034+0000 7f96c804b640 1 -- 172.21.15.33:0/3725033837 >> 172.21.15.33:0/3725033837 conn(0x7f96c00fd2c0 msgr2=0x7f96c00fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:32.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.034+0000 7f96c804b640 1 -- 172.21.15.33:0/3725033837 shutdown_connections 2024-03-19T03:03:32.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.034+0000 7f96c804b640 1 -- 172.21.15.33:0/3725033837 wait complete. 2024-03-19T03:03:32.046 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-19T03:03:32.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.150+0000 7fe70484b640 1 -- 172.21.15.33:0/553388729 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe700101070 msgr2=0x7fe70010d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.150+0000 7fe70484b640 1 --2- 172.21.15.33:0/553388729 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe700101070 0x7fe70010d580 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8009cd0 tx=0x7fe6f802c0f0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.150+0000 7fe70484b640 1 -- 172.21.15.33:0/553388729 shutdown_connections 2024-03-19T03:03:32.153 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.150+0000 7fe70484b640 1 --2- 172.21.15.33:0/553388729 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe700101070 0x7fe70010d580 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.150+0000 7fe70484b640 1 --2- 172.21.15.33:0/553388729 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe7001006b0 0x7fe700100b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.150+0000 7fe70484b640 1 --2- 172.21.15.33:0/553388729 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe7001066c0 0x7fe700106aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.150+0000 7fe70484b640 1 -- 172.21.15.33:0/553388729 >> 172.21.15.33:0/553388729 conn(0x7fe700076aa0 msgr2=0x7fe700076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:32.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.150+0000 7fe70484b640 1 -- 172.21.15.33:0/553388729 shutdown_connections 2024-03-19T03:03:32.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.150+0000 7fe70484b640 1 -- 172.21.15.33:0/553388729 wait complete. 2024-03-19T03:03:32.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.151+0000 7fe70484b640 1 Processor -- start 2024-03-19T03:03:32.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.151+0000 7fe70484b640 1 -- start start 2024-03-19T03:03:32.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.151+0000 7fe70484b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe7001006b0 0x7fe70019a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.151+0000 7fe70484b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe700101070 0x7fe70019a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.151+0000 7fe70484b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7001066c0 0x7fe70019ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.151+0000 7fe70484b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe700110980 con 0x7fe7001066c0 2024-03-19T03:03:32.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.151+0000 7fe70484b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe700110800 con 0x7fe700101070 2024-03-19T03:03:32.154 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.151+0000 7fe70484b640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe700110b00 con 0x7fe7001006b0 2024-03-19T03:03:32.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.151+0000 7fe6fed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7001066c0 0x7fe70019ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.151+0000 7fe6fed9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7001066c0 0x7fe70019ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41234/0 (socket says 172.21.15.33:41234) 2024-03-19T03:03:32.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.151+0000 7fe6fed9d640 1 -- 172.21.15.33:0/3309412093 learned_addr learned my addr 172.21.15.33:0/3309412093 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:32.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.151+0000 7fe6fe59c640 1 --2- 172.21.15.33:0/3309412093 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe7001006b0 0x7fe70019a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.151+0000 7fe6f7fff640 1 --2- 172.21.15.33:0/3309412093 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe700101070 0x7fe70019a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.151+0000 7fe6fed9d640 1 -- 172.21.15.33:0/3309412093 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe7001006b0 msgr2=0x7fe70019a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.151+0000 7fe6fed9d640 1 --2- 172.21.15.33:0/3309412093 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe7001006b0 0x7fe70019a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.151+0000 7fe6fed9d640 1 -- 172.21.15.33:0/3309412093 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe700101070 msgr2=0x7fe70019a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.151+0000 7fe6fed9d640 1 --2- 172.21.15.33:0/3309412093 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe700101070 0x7fe70019a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.151+0000 7fe6fed9d640 1 -- 172.21.15.33:0/3309412093 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe70019f130 con 0x7fe7001066c0 2024-03-19T03:03:32.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.151+0000 7fe6fed9d640 1 --2- 172.21.15.33:0/3309412093 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7001066c0 0x7fe70019ea00 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8004af0 tx=0x7fe6f8035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:32.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.152+0000 7fe6f77fe640 1 -- 172.21.15.33:0/3309412093 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f8039040 con 0x7fe7001066c0 2024-03-19T03:03:32.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.152+0000 7fe6f77fe640 1 -- 172.21.15.33:0/3309412093 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe6f8044070 con 0x7fe7001066c0 2024-03-19T03:03:32.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.152+0000 7fe6f77fe640 1 -- 172.21.15.33:0/3309412093 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f80042a0 con 0x7fe7001066c0 2024-03-19T03:03:32.155 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.152+0000 7fe70484b640 1 -- 172.21.15.33:0/3309412093 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe70019f3c0 con 0x7fe7001066c0 2024-03-19T03:03:32.156 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.152+0000 7fe70484b640 1 -- 172.21.15.33:0/3309412093 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7001a6cb0 con 0x7fe7001066c0 2024-03-19T03:03:32.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.154+0000 7fe6f77fe640 1 -- 172.21.15.33:0/3309412093 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fe6f8004440 con 0x7fe7001066c0 2024-03-19T03:03:32.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.154+0000 7fe707438740 1 Processor -- start 2024-03-19T03:03:32.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.154+0000 7fe707438740 1 -- start start 2024-03-19T03:03:32.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.154+0000 7fe707438740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5639ac2052d0 0x5639ac2056b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.154+0000 7fe707438740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5639ac205bf0 0x5639ac20e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.162 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.154+0000 7fe707438740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639ac20e7d0 0x5639ac210bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.154+0000 7fe707438740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5639ac1684b0 con 0x5639ac20e7d0 2024-03-19T03:03:32.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.154+0000 7fe707438740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5639ac110d70 con 0x5639ac205bf0 2024-03-19T03:03:32.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.154+0000 7fe707438740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5639ac164550 con 0x5639ac2052d0 2024-03-19T03:03:32.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.155+0000 7fe6f77fe640 1 --2- 172.21.15.33:0/3309412093 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe6ec0a2970 0x7fe6ec0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.155+0000 7fe6fe59c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5639ac205bf0 0x5639ac20e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.155+0000 7fe6f7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639ac20e7d0 0x5639ac210bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.155+0000 7fe6fe59c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5639ac205bf0 0x5639ac20e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:53486/0 (socket says 172.21.15.33:53486) 2024-03-19T03:03:32.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.155+0000 7fe6fe59c640 1 -- 172.21.15.33:0/3164287968 learned_addr learned my addr 172.21.15.33:0/3164287968 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:32.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.155+0000 7fe6f7fff640 1 --2- 172.21.15.33:0/3164287968 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5639ac2052d0 0x5639ac2056b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.155+0000 7fe6f77fe640 1 -- 172.21.15.33:0/3309412093 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7fe6f80f1630 con 0x7fe7001066c0 2024-03-19T03:03:32.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.155+0000 7fe6fed9d640 1 --2- 172.21.15.33:0/3309412093 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe6ec0a2970 0x7fe6ec0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.155+0000 7fe6fe59c640 1 -- 172.21.15.33:0/3164287968 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5639ac2052d0 msgr2=0x5639ac2056b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:32.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.155+0000 7fe6fe59c640 1 --2- 172.21.15.33:0/3164287968 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5639ac2052d0 0x5639ac2056b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.163 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.155+0000 7fe6fe59c640 1 -- 172.21.15.33:0/3164287968 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639ac20e7d0 msgr2=0x5639ac210bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:32.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.155+0000 7fe6fe59c640 1 --2- 172.21.15.33:0/3164287968 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639ac20e7d0 0x5639ac210bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.155+0000 7fe6fe59c640 1 -- 172.21.15.33:0/3164287968 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639ac133210 con 0x5639ac205bf0 2024-03-19T03:03:32.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.155+0000 7fe6fe59c640 1 --2- 172.21.15.33:0/3164287968 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5639ac205bf0 0x5639ac20e290 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe6f000b710 tx=0x7fe6f0004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad265262293eb520 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:32.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.155+0000 7fe6fed9d640 1 --2- 172.21.15.33:0/3309412093 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe6ec0a2970 0x7fe6ec0a4e30 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe6f80c5c40 tx=0x7fe6f80095c0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:32.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.155+0000 7fe6f4ff9640 1 -- 172.21.15.33:0/3164287968 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f0003bd0 con 0x5639ac205bf0 2024-03-19T03:03:32.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.155+0000 7fe6f4ff9640 1 -- 172.21.15.33:0/3164287968 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe6f0015070 con 0x5639ac205bf0 2024-03-19T03:03:32.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.155+0000 7fe6f4ff9640 1 -- 172.21.15.33:0/3164287968 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f0004c00 con 0x5639ac205bf0 2024-03-19T03:03:32.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.156+0000 7fe707438740 1 -- 172.21.15.33:0/3164287968 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5639ac205bf0 msgr2=0x5639ac20e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.156+0000 7fe707438740 1 --2- 172.21.15.33:0/3164287968 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5639ac205bf0 0x5639ac20e290 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe6f000b710 tx=0x7fe6f0004610 comp rx=0 tx=0).stop 2024-03-19T03:03:32.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.156+0000 7fe707438740 1 -- 172.21.15.33:0/3164287968 shutdown_connections 2024-03-19T03:03:32.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.156+0000 7fe707438740 1 --2- 172.21.15.33:0/3164287968 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639ac20e7d0 0x5639ac210bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.156+0000 7fe707438740 1 --2- 172.21.15.33:0/3164287968 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5639ac205bf0 0x5639ac20e290 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.156+0000 7fe707438740 1 --2- 172.21.15.33:0/3164287968 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5639ac2052d0 0x5639ac2056b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.156+0000 7fe707438740 1 -- 172.21.15.33:0/3164287968 >> 172.21.15.33:0/3164287968 conn(0x5639ac1ffa00 msgr2=0x5639ac1ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:32.164 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.156+0000 7fe707438740 1 -- 172.21.15.33:0/3164287968 shutdown_connections 2024-03-19T03:03:32.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.156+0000 7fe707438740 1 -- 172.21.15.33:0/3164287968 wait complete. 2024-03-19T03:03:32.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.156+0000 7fe707438740 1 -- start start 2024-03-19T03:03:32.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.157+0000 7fe707438740 1 Processor -- start 2024-03-19T03:03:32.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.157+0000 7fe707438740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5639ac2052d0 0x5639ac20b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.157+0000 7fe707438740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639ac205bf0 0x5639ac20bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.157+0000 7fe707438740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5639ac20e7d0 0x5639ac20c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.157+0000 7fe707438740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x5639ac2115c0 con 0x5639ac205bf0 2024-03-19T03:03:32.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.157+0000 7fe707438740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5639ac124310 con 0x5639ac20e7d0 2024-03-19T03:03:32.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.157+0000 7fe707438740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5639ac211740 con 0x5639ac2052d0 2024-03-19T03:03:32.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.157+0000 7fe707438740 10 client.0 _ll_register_callbacks cb 0x5639ac1182d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:03:32.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.157+0000 7fe6fe59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639ac205bf0 0x5639ac20bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.157+0000 7fe6fe59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639ac205bf0 0x5639ac20bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41258/0 (socket says 172.21.15.33:41258) 2024-03-19T03:03:32.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.157+0000 7fe6fe59c640 1 -- 172.21.15.33:0/2636008073 learned_addr learned my addr 172.21.15.33:0/2636008073 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:32.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.157+0000 7fe6f7fff640 1 --2- 172.21.15.33:0/2636008073 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5639ac20e7d0 0x5639ac20c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.157+0000 7fe6f7fff640 1 --2- 172.21.15.33:0/2636008073 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5639ac2052d0 0x5639ac20b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.157+0000 7fe6fe59c640 1 -- 172.21.15.33:0/2636008073 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5639ac2052d0 msgr2=0x5639ac20b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:32.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.157+0000 7fe6fe59c640 1 --2- 172.21.15.33:0/2636008073 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5639ac2052d0 0x5639ac20b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.157+0000 7fe6fe59c640 1 -- 172.21.15.33:0/2636008073 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5639ac20e7d0 msgr2=0x5639ac20c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:32.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.157+0000 7fe6fe59c640 1 --2- 172.21.15.33:0/2636008073 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5639ac20e7d0 0x5639ac20c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.157+0000 7fe6fe59c640 1 -- 172.21.15.33:0/2636008073 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639abe7c3f0 con 0x5639ac205bf0 2024-03-19T03:03:32.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.157+0000 7fe6f7fff640 1 --2- 172.21.15.33:0/2636008073 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5639ac2052d0 0x5639ac20b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:03:32.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.157+0000 7fe6fe59c640 1 --2- 172.21.15.33:0/2636008073 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639ac205bf0 0x5639ac20bd40 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fe6f0005880 tx=0x7fe6f000bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2468e37532b74777 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:32.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.157+0000 7fe6e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:03:32.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.158+0000 7fe6e5ffb640 1 -- 172.21.15.33:0/2636008073 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f0028060 con 0x5639ac205bf0 2024-03-19T03:03:32.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.158+0000 7fe6e5ffb640 1 -- 172.21.15.33:0/2636008073 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7fe6f0011440 con 0x5639ac205bf0 2024-03-19T03:03:32.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.158+0000 7fe707438740 1 -- 172.21.15.33:0/2636008073 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6f0015070 con 0x5639ac205bf0 2024-03-19T03:03:32.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.158+0000 7fe6e5ffb640 1 -- 172.21.15.33:0/2636008073 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe6f002f050 con 0x5639ac205bf0 2024-03-19T03:03:32.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.158+0000 7fe707438740 10 client.17796 fetch_fsmap learned FSMap version 25 2024-03-19T03:03:32.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.158+0000 7fe707438740 1 -- 172.21.15.33:0/2636008073 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5639abf467d0 con 0x5639ac205bf0 2024-03-19T03:03:32.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.158+0000 7fe6e5ffb640 1 -- 172.21.15.33:0/2636008073 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe6f0015070 con 0x5639ac205bf0 2024-03-19T03:03:32.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.158+0000 7fe6e5ffb640 1 -- 172.21.15.33:0/2636008073 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fe6f0022070 con 0x5639ac205bf0 2024-03-19T03:03:32.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.158+0000 7fe707438740 10 client.17796 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:03:32.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.158+0000 7fe707438740 10 client.17796 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T03:03:32.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.158+0000 7fe707438740 20 client.17796 populate_metadata read hostname 'smithi033' 2024-03-19T03:03:32.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.158+0000 7fe707438740 1 --2- 172.21.15.33:0/2636008073 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x5639ac21eac0 0x5639ac220eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.158+0000 7fe707438740 4 client.17796 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:03:32.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.158+0000 7fe707438740 1 -- 172.21.15.33:0/2636008073 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5639ac1c63e0 con 0x5639ac21eac0 2024-03-19T03:03:32.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.158+0000 7fe6f7fff640 1 --2- 172.21.15.33:0/2636008073 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x5639ac21eac0 0x5639ac220eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.159+0000 7fe6e5ffb640 10 client.17796 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T03:03:32.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.159+0000 7fe6f7fff640 1 --2- 172.21.15.33:0/2636008073 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x5639ac21eac0 0x5639ac220eb0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3c2c1a457631e99c server_cookie=9c3c6f06964b18d4 in_seq=0 out_seq=0 2024-03-19T03:03:32.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.159+0000 7fe6e5ffb640 1 -- 172.21.15.33:0/2636008073 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5639ac1c63e0 con 0x5639ac21eac0 2024-03-19T03:03:32.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.159+0000 7fe6e5ffb640 10 client.17796 handle_command_reply: tid=0 2024-03-19T03:03:32.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.159+0000 7fe707438740 1 client.17796 shutdown 2024-03-19T03:03:32.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.159+0000 7fe707438740 1 -- 172.21.15.33:0/2636008073 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639ac205bf0 msgr2=0x5639ac20bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.159+0000 7fe707438740 1 --2- 172.21.15.33:0/2636008073 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639ac205bf0 0x5639ac20bd40 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fe6f0005880 tx=0x7fe6f000bf80 comp rx=0 tx=0).stop 2024-03-19T03:03:32.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.160+0000 7fe707438740 1 -- 172.21.15.33:0/2636008073 shutdown_connections 2024-03-19T03:03:32.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.160+0000 7fe707438740 1 --2- 172.21.15.33:0/2636008073 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x5639ac21eac0 0x5639ac220eb0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.160+0000 7fe707438740 1 --2- 172.21.15.33:0/2636008073 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5639ac20e7d0 0x5639ac20c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.160+0000 7fe6e5ffb640 0 client.17796 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T03:03:32.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.160+0000 7fe707438740 1 --2- 172.21.15.33:0/2636008073 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639ac205bf0 0x5639ac20bd40 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.160+0000 7fe707438740 1 --2- 172.21.15.33:0/2636008073 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5639ac2052d0 0x5639ac20b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.160+0000 7fe707438740 1 -- 172.21.15.33:0/2636008073 >> 172.21.15.33:0/2636008073 conn(0x5639ac1ffa00 msgr2=0x5639ac201750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:32.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.160+0000 7fe707438740 1 -- 172.21.15.33:0/2636008073 shutdown_connections 2024-03-19T03:03:32.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.160+0000 7fe707438740 1 -- 172.21.15.33:0/2636008073 wait complete. 2024-03-19T03:03:32.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.160+0000 7fe707438740 20 client.17796 trim_cache size 0 max 16384 2024-03-19T03:03:32.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.173+0000 7fe707438740 1 Processor -- start 2024-03-19T03:03:32.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.173+0000 7fe707438740 1 -- start start 2024-03-19T03:03:32.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.173+0000 7fe707438740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5639ac2052d0 0x5639ac20dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.173+0000 7fe707438740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5639ac20e7d0 0x5639ac1faa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.173+0000 7fe707438740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639ac21a390 0x5639ac2264b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.173+0000 7fe707438740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe7001b3670 con 0x5639ac21a390 2024-03-19T03:03:32.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.173+0000 7fe707438740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe7001b0f40 con 0x5639ac20e7d0 2024-03-19T03:03:32.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.173+0000 7fe707438740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe7001aedb0 con 0x5639ac2052d0 2024-03-19T03:03:32.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.173+0000 7fe6f7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639ac21a390 0x5639ac2264b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.173+0000 7fe6fe59c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5639ac20e7d0 0x5639ac1faa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.173+0000 7fe6f7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639ac21a390 0x5639ac2264b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41274/0 (socket says 172.21.15.33:41274) 2024-03-19T03:03:32.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.173+0000 7fe6f7fff640 1 -- 172.21.15.33:0/40021396 learned_addr learned my addr 172.21.15.33:0/40021396 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:32.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.173+0000 7fe6f7fff640 1 --2- 172.21.15.33:0/40021396 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5639ac2052d0 0x5639ac20dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.173+0000 7fe6fe59c640 1 -- 172.21.15.33:0/40021396 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5639ac2052d0 msgr2=0x5639ac20dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:32.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.173+0000 7fe6fe59c640 1 --2- 172.21.15.33:0/40021396 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5639ac2052d0 0x5639ac20dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.173+0000 7fe6fe59c640 1 -- 172.21.15.33:0/40021396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639ac21a390 msgr2=0x5639ac2264b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:32.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.173+0000 7fe6fe59c640 1 --2- 172.21.15.33:0/40021396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639ac21a390 0x5639ac2264b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.173+0000 7fe6fe59c640 1 -- 172.21.15.33:0/40021396 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639ac20a7b0 con 0x5639ac20e7d0 2024-03-19T03:03:32.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.173+0000 7fe6f7fff640 1 --2- 172.21.15.33:0/40021396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639ac21a390 0x5639ac2264b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:32.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.174+0000 7fe6fe59c640 1 --2- 172.21.15.33:0/40021396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5639ac20e7d0 0x5639ac1faa30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x5639ac20c890 tx=0x7fe6f0003a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c861151199db6996 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:32.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.174+0000 7fe6e7fff640 1 -- 172.21.15.33:0/40021396 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f0028050 con 0x5639ac20e7d0 2024-03-19T03:03:32.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.174+0000 7fe6e7fff640 1 -- 172.21.15.33:0/40021396 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe6f0015070 con 0x5639ac20e7d0 2024-03-19T03:03:32.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.174+0000 7fe6e7fff640 1 -- 172.21.15.33:0/40021396 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f00118f0 con 0x5639ac20e7d0 2024-03-19T03:03:32.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.174+0000 7fe707438740 1 -- 172.21.15.33:0/40021396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5639ac20e7d0 msgr2=0x5639ac1faa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.174+0000 7fe707438740 1 --2- 172.21.15.33:0/40021396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5639ac20e7d0 0x5639ac1faa30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x5639ac20c890 tx=0x7fe6f0003a00 comp rx=0 tx=0).stop 2024-03-19T03:03:32.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.174+0000 7fe707438740 1 -- 172.21.15.33:0/40021396 shutdown_connections 2024-03-19T03:03:32.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.174+0000 7fe707438740 1 --2- 172.21.15.33:0/40021396 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639ac21a390 0x5639ac2264b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.174+0000 7fe707438740 1 --2- 172.21.15.33:0/40021396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5639ac20e7d0 0x5639ac1faa30 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.174+0000 7fe707438740 1 --2- 172.21.15.33:0/40021396 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5639ac2052d0 0x5639ac20dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.174+0000 7fe707438740 1 -- 172.21.15.33:0/40021396 >> 172.21.15.33:0/40021396 conn(0x5639ac1ffa00 msgr2=0x5639ac20c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:32.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.174+0000 7fe707438740 1 -- 172.21.15.33:0/40021396 shutdown_connections 2024-03-19T03:03:32.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.174+0000 7fe707438740 1 -- 172.21.15.33:0/40021396 wait complete. 2024-03-19T03:03:32.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.175+0000 7fe707438740 1 -- start start 2024-03-19T03:03:32.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.175+0000 7fe707438740 1 Processor -- start 2024-03-19T03:03:32.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.175+0000 7fe707438740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5639ac2052d0 0x5639ac212500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.175+0000 7fe707438740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639ac20e7d0 0x5639ac212a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.175+0000 7fe707438740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5639ac21a390 0x5639ac212f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.175+0000 7fe707438740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe7001af400 con 0x5639ac20e7d0 2024-03-19T03:03:32.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.175+0000 7fe707438740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe7001b0870 con 0x5639ac2052d0 2024-03-19T03:03:32.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.175+0000 7fe707438740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe7001afad0 con 0x5639ac21a390 2024-03-19T03:03:32.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.175+0000 7fe707438740 10 client.0 _ll_register_callbacks cb 0x7fe6e8009a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:03:32.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.175+0000 7fe6fe59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639ac20e7d0 0x5639ac212a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.175+0000 7fe6fe59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639ac20e7d0 0x5639ac212a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41282/0 (socket says 172.21.15.33:41282) 2024-03-19T03:03:32.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.175+0000 7fe6fe59c640 1 -- 172.21.15.33:0/2760897495 learned_addr learned my addr 172.21.15.33:0/2760897495 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:32.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.175+0000 7fe6f7fff640 1 --2- 172.21.15.33:0/2760897495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5639ac2052d0 0x5639ac212500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.175+0000 7fe6f7fff640 1 --2- 172.21.15.33:0/2760897495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5639ac21a390 0x5639ac212f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.175+0000 7fe6fe59c640 1 -- 172.21.15.33:0/2760897495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5639ac21a390 msgr2=0x5639ac212f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:32.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.175+0000 7fe6fe59c640 1 --2- 172.21.15.33:0/2760897495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5639ac21a390 0x5639ac212f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.175+0000 7fe6fe59c640 1 -- 172.21.15.33:0/2760897495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5639ac2052d0 msgr2=0x5639ac212500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:32.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.175+0000 7fe6fe59c640 1 --2- 172.21.15.33:0/2760897495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5639ac2052d0 0x5639ac212500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.175+0000 7fe6fe59c640 1 -- 172.21.15.33:0/2760897495 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639ac283d30 con 0x5639ac20e7d0 2024-03-19T03:03:32.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.175+0000 7fe6f7fff640 1 --2- 172.21.15.33:0/2760897495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5639ac21a390 0x5639ac212f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:03:32.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.175+0000 7fe6fe59c640 1 --2- 172.21.15.33:0/2760897495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639ac20e7d0 0x5639ac212a40 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fe6f00198b0 tx=0x7fe6f000b4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58349a39ea21a3d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:32.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.175+0000 7fe6e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:03:32.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.176+0000 7fe6e5ffb640 1 -- 172.21.15.33:0/2760897495 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f0040060 con 0x5639ac20e7d0 2024-03-19T03:03:32.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.176+0000 7fe6e5ffb640 1 -- 172.21.15.33:0/2760897495 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7fe6f0004100 con 0x5639ac20e7d0 2024-03-19T03:03:32.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.176+0000 7fe707438740 1 -- 172.21.15.33:0/2760897495 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6f0015070 con 0x5639ac20e7d0 2024-03-19T03:03:32.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.176+0000 7fe6e5ffb640 1 -- 172.21.15.33:0/2760897495 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe6f0047050 con 0x5639ac20e7d0 2024-03-19T03:03:32.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.176+0000 7fe707438740 10 client.17802 fetch_fsmap learned FSMap version 25 2024-03-19T03:03:32.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.176+0000 7fe707438740 1 -- 172.21.15.33:0/2760897495 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5639ac285010 con 0x5639ac20e7d0 2024-03-19T03:03:32.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.176+0000 7fe6e5ffb640 1 -- 172.21.15.33:0/2760897495 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe6f0015070 con 0x5639ac20e7d0 2024-03-19T03:03:32.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.176+0000 7fe6e5ffb640 1 -- 172.21.15.33:0/2760897495 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fe6f0022040 con 0x5639ac20e7d0 2024-03-19T03:03:32.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.176+0000 7fe707438740 10 client.17802 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:03:32.184 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.176+0000 7fe707438740 10 client.17802 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-19T03:03:32.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.176+0000 7fe707438740 20 client.17802 populate_metadata read hostname 'smithi033' 2024-03-19T03:03:32.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.176+0000 7fe707438740 1 --2- 172.21.15.33:0/2760897495 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x5639ac2856d0 0x5639ac287af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.176+0000 7fe707438740 4 client.17802 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T03:03:32.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.176+0000 7fe707438740 1 -- 172.21.15.33:0/2760897495 --> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5639ac1759f0 con 0x5639ac2856d0 2024-03-19T03:03:32.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.176+0000 7fe6f7fff640 1 --2- 172.21.15.33:0/2760897495 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x5639ac2856d0 0x5639ac287af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.177+0000 7fe6f7fff640 1 --2- 172.21.15.33:0/2760897495 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x5639ac2856d0 0x5639ac287af0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cb3c0f1c7b1b54b2 server_cookie=5064328d82d65e76 in_seq=0 out_seq=0 2024-03-19T03:03:32.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.177+0000 7fe6e5ffb640 10 client.17802 ms_handle_connect on v2:172.21.15.155:6838/1840534530 2024-03-19T03:03:32.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.177+0000 7fe6e5ffb640 1 -- 172.21.15.33:0/2760897495 <== mds.2 v2:172.21.15.155:6838/1840534530 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5639ac1759f0 con 0x5639ac2856d0 2024-03-19T03:03:32.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.177+0000 7fe6e5ffb640 10 client.17802 handle_command_reply: tid=0 2024-03-19T03:03:32.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.177+0000 7fe707438740 1 client.17802 shutdown 2024-03-19T03:03:32.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.177+0000 7fe707438740 1 -- 172.21.15.33:0/2760897495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639ac20e7d0 msgr2=0x5639ac212a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.177+0000 7fe707438740 1 --2- 172.21.15.33:0/2760897495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639ac20e7d0 0x5639ac212a40 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fe6f00198b0 tx=0x7fe6f000b4c0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.177+0000 7fe707438740 1 -- 172.21.15.33:0/2760897495 shutdown_connections 2024-03-19T03:03:32.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.177+0000 7fe707438740 1 --2- 172.21.15.33:0/2760897495 >> [v2:172.21.15.155:6838/1840534530,v1:172.21.15.155:6839/1840534530] conn(0x5639ac2856d0 0x5639ac287af0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.185 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.177+0000 7fe707438740 1 --2- 172.21.15.33:0/2760897495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5639ac21a390 0x5639ac212f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.177+0000 7fe6e5ffb640 0 client.17802 ms_handle_reset on v2:172.21.15.155:6838/1840534530 2024-03-19T03:03:32.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.177+0000 7fe707438740 1 --2- 172.21.15.33:0/2760897495 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x5639ac20e7d0 0x5639ac212a40 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.177+0000 7fe707438740 1 --2- 172.21.15.33:0/2760897495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5639ac2052d0 0x5639ac212500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.177+0000 7fe707438740 1 -- 172.21.15.33:0/2760897495 >> 172.21.15.33:0/2760897495 conn(0x5639ac1ffa00 msgr2=0x5639ac1fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:32.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.177+0000 7fe707438740 1 -- 172.21.15.33:0/2760897495 shutdown_connections 2024-03-19T03:03:32.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.177+0000 7fe707438740 1 -- 172.21.15.33:0/2760897495 wait complete. 2024-03-19T03:03:32.186 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.177+0000 7fe707438740 20 client.17802 trim_cache size 0 max 16384 2024-03-19T03:03:32.186 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T03:03:32.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.178+0000 7fe70484b640 1 -- 172.21.15.33:0/3309412093 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe6ec0a2970 msgr2=0x7fe6ec0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.178+0000 7fe70484b640 1 --2- 172.21.15.33:0/3309412093 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe6ec0a2970 0x7fe6ec0a4e30 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe6f80c5c40 tx=0x7fe6f80095c0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.178+0000 7fe70484b640 1 -- 172.21.15.33:0/3309412093 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7001066c0 msgr2=0x7fe70019ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.178+0000 7fe70484b640 1 --2- 172.21.15.33:0/3309412093 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7001066c0 0x7fe70019ea00 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8004af0 tx=0x7fe6f8035ce0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.178+0000 7fe70484b640 1 -- 172.21.15.33:0/3309412093 shutdown_connections 2024-03-19T03:03:32.187 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.178+0000 7fe70484b640 1 --2- 172.21.15.33:0/3309412093 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe6ec0a2970 0x7fe6ec0a4e30 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.178+0000 7fe70484b640 1 --2- 172.21.15.33:0/3309412093 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe7001066c0 0x7fe70019ea00 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.178+0000 7fe70484b640 1 --2- 172.21.15.33:0/3309412093 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe700101070 0x7fe70019a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.178+0000 7fe70484b640 1 --2- 172.21.15.33:0/3309412093 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe7001006b0 0x7fe70019a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.178+0000 7fe70484b640 1 -- 172.21.15.33:0/3309412093 >> 172.21.15.33:0/3309412093 conn(0x7fe700076aa0 msgr2=0x7fe7000ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:32.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.178+0000 7fe70484b640 1 -- 172.21.15.33:0/3309412093 shutdown_connections 2024-03-19T03:03:32.188 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.178+0000 7fe70484b640 1 -- 172.21.15.33:0/3309412093 wait complete. 2024-03-19T03:03:32.190 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-19T03:03:32.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.296+0000 7f369cf22640 1 -- 172.21.15.33:0/1353027531 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f36981066c0 msgr2=0x7f3698106aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.296+0000 7f369cf22640 1 --2- 172.21.15.33:0/1353027531 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f36981066c0 0x7f3698106aa0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f3690009930 tx=0x7f369002bc30 comp rx=0 tx=0).stop 2024-03-19T03:03:32.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.296+0000 7f369cf22640 1 -- 172.21.15.33:0/1353027531 shutdown_connections 2024-03-19T03:03:32.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.296+0000 7f369cf22640 1 --2- 172.21.15.33:0/1353027531 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f36981006b0 0x7f3698100b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.296+0000 7f369cf22640 1 --2- 172.21.15.33:0/1353027531 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3698101070 0x7f369810d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.296+0000 7f369cf22640 1 --2- 172.21.15.33:0/1353027531 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f36981066c0 0x7f3698106aa0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.296+0000 7f369cf22640 1 -- 172.21.15.33:0/1353027531 >> 172.21.15.33:0/1353027531 conn(0x7f3698076aa0 msgr2=0x7f3698076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:32.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.296+0000 7f369cf22640 1 -- 172.21.15.33:0/1353027531 shutdown_connections 2024-03-19T03:03:32.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.296+0000 7f369cf22640 1 -- 172.21.15.33:0/1353027531 wait complete. 2024-03-19T03:03:32.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.296+0000 7f369cf22640 1 Processor -- start 2024-03-19T03:03:32.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.296+0000 7f369cf22640 1 -- start start 2024-03-19T03:03:32.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.297+0000 7f369cf22640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f36981006b0 0x7f3698198e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.297+0000 7f369cf22640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3698101070 0x7f3698199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.297+0000 7f369cf22640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36981066c0 0x7f36981a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.297+0000 7f369cf22640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f3698110980 con 0x7f3698101070 2024-03-19T03:03:32.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.297+0000 7f369cf22640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3698110800 con 0x7f36981066c0 2024-03-19T03:03:32.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.297+0000 7f369cf22640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3698110b00 con 0x7f36981006b0 2024-03-19T03:03:32.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.297+0000 7f3695d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3698101070 0x7f3698199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.297+0000 7f3695d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3698101070 0x7f3698199340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41300/0 (socket says 172.21.15.33:41300) 2024-03-19T03:03:32.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.297+0000 7f3695d9b640 1 -- 172.21.15.33:0/1978972323 learned_addr learned my addr 172.21.15.33:0/1978972323 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:32.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.297+0000 7f369659c640 1 --2- 172.21.15.33:0/1978972323 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f36981006b0 0x7f3698198e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.297+0000 7f3696d9d640 1 --2- 172.21.15.33:0/1978972323 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36981066c0 0x7f36981a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.297+0000 7f3695d9b640 1 -- 172.21.15.33:0/1978972323 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f36981006b0 msgr2=0x7f3698198e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.297+0000 7f3695d9b640 1 --2- 172.21.15.33:0/1978972323 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f36981006b0 0x7f3698198e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.297+0000 7f3695d9b640 1 -- 172.21.15.33:0/1978972323 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36981066c0 msgr2=0x7f36981a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.297+0000 7f3695d9b640 1 --2- 172.21.15.33:0/1978972323 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36981066c0 0x7f36981a3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.297+0000 7f3695d9b640 1 -- 172.21.15.33:0/1978972323 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36981a43c0 con 0x7f3698101070 2024-03-19T03:03:32.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.297+0000 7f3695d9b640 1 --2- 172.21.15.33:0/1978972323 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3698101070 0x7f3698199340 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f368000e9f0 tx=0x7f368000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:32.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.297+0000 7f368f7fe640 1 -- 172.21.15.33:0/1978972323 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f368000cd80 con 0x7f3698101070 2024-03-19T03:03:32.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.297+0000 7f369cf22640 1 -- 172.21.15.33:0/1978972323 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36981a4590 con 0x7f3698101070 2024-03-19T03:03:32.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.297+0000 7f368f7fe640 1 -- 172.21.15.33:0/1978972323 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f36800054d0 con 0x7f3698101070 2024-03-19T03:03:32.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.297+0000 7f368f7fe640 1 -- 172.21.15.33:0/1978972323 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3680010490 con 0x7f3698101070 2024-03-19T03:03:32.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.297+0000 7f369cf22640 1 -- 172.21.15.33:0/1978972323 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36981a4ae0 con 0x7f3698101070 2024-03-19T03:03:32.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.300+0000 7f368f7fe640 1 -- 172.21.15.33:0/1978972323 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f3680005020 con 0x7f3698101070 2024-03-19T03:03:32.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.300+0000 7f369fbee740 1 Processor -- start 2024-03-19T03:03:32.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.300+0000 7f369fbee740 1 -- start start 2024-03-19T03:03:32.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.300+0000 7f369fbee740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ddb5d472d0 0x55ddb5d476b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.300+0000 7f369fbee740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ddb5d47bf0 0x55ddb5d50290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.300+0000 7f369fbee740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ddb5d507d0 0x55ddb5d52bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.300+0000 7f369fbee740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ddb5caa4b0 con 0x55ddb5d507d0 2024-03-19T03:03:32.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.300+0000 7f369fbee740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55ddb5c52d70 con 0x55ddb5d472d0 2024-03-19T03:03:32.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.300+0000 7f369fbee740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ddb5ca6550 con 0x55ddb5d47bf0 2024-03-19T03:03:32.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.301+0000 7f368f7fe640 1 --2- 172.21.15.33:0/1978972323 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f36840a2970 0x7f36840a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.301+0000 7f3695d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ddb5d507d0 0x55ddb5d52bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.301+0000 7f3695d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ddb5d507d0 0x55ddb5d52bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41308/0 (socket says 172.21.15.33:41308) 2024-03-19T03:03:32.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.301+0000 7f3695d9b640 1 -- 172.21.15.33:0/1679896364 learned_addr learned my addr 172.21.15.33:0/1679896364 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:32.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.301+0000 7f3696d9d640 1 --2- 172.21.15.33:0/1679896364 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ddb5d472d0 0x55ddb5d476b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.301+0000 7f3696d9d640 1 --2- 172.21.15.33:0/1978972323 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f36840a2970 0x7f36840a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.301+0000 7f369659c640 1 --2- 172.21.15.33:0/1679896364 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ddb5d47bf0 0x55ddb5d50290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.301+0000 7f368f7fe640 1 -- 172.21.15.33:0/1978972323 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7f36800cf5c0 con 0x7f3698101070 2024-03-19T03:03:32.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.301+0000 7f3696d9d640 1 -- 172.21.15.33:0/1679896364 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ddb5d47bf0 msgr2=0x55ddb5d50290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:32.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.301+0000 7f3696d9d640 1 --2- 172.21.15.33:0/1679896364 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ddb5d47bf0 0x55ddb5d50290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.301+0000 7f3696d9d640 1 -- 172.21.15.33:0/1679896364 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ddb5d507d0 msgr2=0x55ddb5d52bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:32.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.301+0000 7f3696d9d640 1 --2- 172.21.15.33:0/1679896364 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ddb5d507d0 0x55ddb5d52bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.301+0000 7f3696d9d640 1 -- 172.21.15.33:0/1679896364 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddb5c75210 con 0x55ddb5d472d0 2024-03-19T03:03:32.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.301+0000 7f3695d9b640 1 --2- 172.21.15.33:0/1679896364 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ddb5d507d0 0x55ddb5d52bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:32.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.301+0000 7f3696d9d640 1 --2- 172.21.15.33:0/1679896364 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ddb5d472d0 0x55ddb5d476b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3688011cc0 tx=0x7f368800f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=198caf8f608189d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:32.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.301+0000 7f3696d9d640 1 --2- 172.21.15.33:0/1978972323 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f36840a2970 0x7f36840a4e30 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f368800fc90 tx=0x7f3688018000 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:32.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.301+0000 7f368cff9640 1 -- 172.21.15.33:0/1679896364 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3688004b20 con 0x55ddb5d472d0 2024-03-19T03:03:32.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.301+0000 7f368cff9640 1 -- 172.21.15.33:0/1679896364 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3688009ce0 con 0x55ddb5d472d0 2024-03-19T03:03:32.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.301+0000 7f368cff9640 1 -- 172.21.15.33:0/1679896364 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36880053c0 con 0x55ddb5d472d0 2024-03-19T03:03:32.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.302+0000 7f369fbee740 1 -- 172.21.15.33:0/1679896364 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ddb5d472d0 msgr2=0x55ddb5d476b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.302+0000 7f369fbee740 1 --2- 172.21.15.33:0/1679896364 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ddb5d472d0 0x55ddb5d476b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3688011cc0 tx=0x7f368800f790 comp rx=0 tx=0).stop 2024-03-19T03:03:32.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.302+0000 7f369fbee740 1 -- 172.21.15.33:0/1679896364 shutdown_connections 2024-03-19T03:03:32.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.302+0000 7f369fbee740 1 --2- 172.21.15.33:0/1679896364 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ddb5d47bf0 0x55ddb5d50290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.302+0000 7f369fbee740 1 --2- 172.21.15.33:0/1679896364 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ddb5d507d0 0x55ddb5d52bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.302+0000 7f369fbee740 1 --2- 172.21.15.33:0/1679896364 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ddb5d472d0 0x55ddb5d476b0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.302+0000 7f369fbee740 1 -- 172.21.15.33:0/1679896364 >> 172.21.15.33:0/1679896364 conn(0x55ddb5d41a00 msgr2=0x55ddb5d41de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:32.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.302+0000 7f369fbee740 1 -- 172.21.15.33:0/1679896364 shutdown_connections 2024-03-19T03:03:32.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.302+0000 7f369fbee740 1 -- 172.21.15.33:0/1679896364 wait complete. 2024-03-19T03:03:32.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.303+0000 7f369fbee740 1 -- start start 2024-03-19T03:03:32.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.303+0000 7f369fbee740 1 Processor -- start 2024-03-19T03:03:32.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.303+0000 7f369fbee740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ddb5d472d0 0x55ddb5d4d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.303+0000 7f369fbee740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ddb5d47bf0 0x55ddb5d4dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.303+0000 7f369fbee740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ddb5d507d0 0x55ddb5d4e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.303+0000 7f369fbee740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x55ddb5d53560 con 0x55ddb5d47bf0 2024-03-19T03:03:32.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.303+0000 7f369fbee740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55ddb5c66310 con 0x55ddb5d472d0 2024-03-19T03:03:32.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.303+0000 7f369fbee740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55ddb5d536e0 con 0x55ddb5d507d0 2024-03-19T03:03:32.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.303+0000 7f369fbee740 10 client.0 _ll_register_callbacks cb 0x55ddb5c5a2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:03:32.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.303+0000 7f3695d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ddb5d47bf0 0x55ddb5d4dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.303+0000 7f3695d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ddb5d47bf0 0x55ddb5d4dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41310/0 (socket says 172.21.15.33:41310) 2024-03-19T03:03:32.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.303+0000 7f3695d9b640 1 -- 172.21.15.33:0/3905052121 learned_addr learned my addr 172.21.15.33:0/3905052121 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:32.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.303+0000 7f369659c640 1 --2- 172.21.15.33:0/3905052121 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ddb5d472d0 0x55ddb5d4d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.303+0000 7f3695d9b640 1 -- 172.21.15.33:0/3905052121 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ddb5d507d0 msgr2=0x55ddb5d4e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:32.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.303+0000 7f3695d9b640 1 --2- 172.21.15.33:0/3905052121 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ddb5d507d0 0x55ddb5d4e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.303+0000 7f3695d9b640 1 -- 172.21.15.33:0/3905052121 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ddb5d472d0 msgr2=0x55ddb5d4d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:32.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.303+0000 7f3695d9b640 1 --2- 172.21.15.33:0/3905052121 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ddb5d472d0 0x55ddb5d4d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.303+0000 7f3695d9b640 1 -- 172.21.15.33:0/3905052121 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddb59be3f0 con 0x55ddb5d47bf0 2024-03-19T03:03:32.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.303+0000 7f369659c640 1 --2- 172.21.15.33:0/3905052121 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ddb5d472d0 0x55ddb5d4d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:32.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.303+0000 7f3695d9b640 1 --2- 172.21.15.33:0/3905052121 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ddb5d47bf0 0x55ddb5d4dda0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f3680098220 tx=0x7f368009b450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b3bbb09effa8c7a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:32.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.303+0000 7f367dffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:03:32.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.304+0000 7f367dffb640 1 -- 172.21.15.33:0/3905052121 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3680005680 con 0x55ddb5d47bf0 2024-03-19T03:03:32.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.304+0000 7f367dffb640 1 -- 172.21.15.33:0/3905052121 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7f3680097ec0 con 0x55ddb5d47bf0 2024-03-19T03:03:32.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.304+0000 7f369fbee740 1 -- 172.21.15.33:0/3905052121 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3688009ce0 con 0x55ddb5d47bf0 2024-03-19T03:03:32.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.304+0000 7f367dffb640 1 -- 172.21.15.33:0/3905052121 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f368009bf50 con 0x55ddb5d47bf0 2024-03-19T03:03:32.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.304+0000 7f369fbee740 10 client.17814 fetch_fsmap learned FSMap version 25 2024-03-19T03:03:32.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.304+0000 7f369fbee740 1 -- 172.21.15.33:0/3905052121 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ddb5a887d0 con 0x55ddb5d47bf0 2024-03-19T03:03:32.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.304+0000 7f367dffb640 1 -- 172.21.15.33:0/3905052121 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3688009ce0 con 0x55ddb5d47bf0 2024-03-19T03:03:32.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.304+0000 7f367dffb640 1 -- 172.21.15.33:0/3905052121 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f36800cf190 con 0x55ddb5d47bf0 2024-03-19T03:03:32.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.304+0000 7f369fbee740 10 client.17814 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:03:32.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.304+0000 7f369fbee740 10 client.17814 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T03:03:32.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.304+0000 7f369fbee740 20 client.17814 populate_metadata read hostname 'smithi033' 2024-03-19T03:03:32.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.304+0000 7f369fbee740 1 --2- 172.21.15.33:0/3905052121 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55ddb5d60b50 0x55ddb5d62fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.304+0000 7f369fbee740 4 client.17814 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:03:32.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.304+0000 7f369fbee740 1 -- 172.21.15.33:0/3905052121 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ddb5d083e0 con 0x55ddb5d60b50 2024-03-19T03:03:32.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.304+0000 7f369659c640 1 --2- 172.21.15.33:0/3905052121 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55ddb5d60b50 0x55ddb5d62fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.305+0000 7f369659c640 1 --2- 172.21.15.33:0/3905052121 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55ddb5d60b50 0x55ddb5d62fc0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3ac1cedd28e1e419 server_cookie=b33a8f5c464c8b46 in_seq=0 out_seq=0 2024-03-19T03:03:32.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.305+0000 7f367dffb640 10 client.17814 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T03:03:32.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.305+0000 7f367dffb640 1 -- 172.21.15.33:0/3905052121 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ddb5d083e0 con 0x55ddb5d60b50 2024-03-19T03:03:32.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.305+0000 7f367dffb640 10 client.17814 handle_command_reply: tid=0 2024-03-19T03:03:32.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.305+0000 7f369fbee740 1 client.17814 shutdown 2024-03-19T03:03:32.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.306+0000 7f369fbee740 1 -- 172.21.15.33:0/3905052121 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ddb5d47bf0 msgr2=0x55ddb5d4dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.306+0000 7f369fbee740 1 --2- 172.21.15.33:0/3905052121 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ddb5d47bf0 0x55ddb5d4dda0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f3680098220 tx=0x7f368009b450 comp rx=0 tx=0).stop 2024-03-19T03:03:32.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.306+0000 7f369fbee740 1 -- 172.21.15.33:0/3905052121 shutdown_connections 2024-03-19T03:03:32.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.306+0000 7f369fbee740 1 --2- 172.21.15.33:0/3905052121 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55ddb5d60b50 0x55ddb5d62fc0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.306+0000 7f369fbee740 1 --2- 172.21.15.33:0/3905052121 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ddb5d507d0 0x55ddb5d4e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.306+0000 7f369fbee740 1 --2- 172.21.15.33:0/3905052121 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ddb5d47bf0 0x55ddb5d4dda0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.306+0000 7f367dffb640 0 client.17814 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T03:03:32.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.306+0000 7f369fbee740 1 --2- 172.21.15.33:0/3905052121 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ddb5d472d0 0x55ddb5d4d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.306+0000 7f369fbee740 1 -- 172.21.15.33:0/3905052121 >> 172.21.15.33:0/3905052121 conn(0x55ddb5d41a00 msgr2=0x55ddb5d43750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:32.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.306+0000 7f369fbee740 1 -- 172.21.15.33:0/3905052121 shutdown_connections 2024-03-19T03:03:32.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.306+0000 7f369fbee740 1 -- 172.21.15.33:0/3905052121 wait complete. 2024-03-19T03:03:32.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.306+0000 7f369fbee740 20 client.17814 trim_cache size 0 max 16384 2024-03-19T03:03:32.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.319+0000 7f369fbee740 1 Processor -- start 2024-03-19T03:03:32.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.319+0000 7f369fbee740 1 -- start start 2024-03-19T03:03:32.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.319+0000 7f369fbee740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ddb5d472d0 0x55ddb5d4ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.319+0000 7f369fbee740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ddb5d47bf0 0x55ddb5d5c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.325 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.319+0000 7f369fbee740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ddb5d507d0 0x55ddb5d436c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.319+0000 7f369fbee740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f36981b1520 con 0x55ddb5d507d0 2024-03-19T03:03:32.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.319+0000 7f369fbee740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f36981aedf0 con 0x55ddb5d47bf0 2024-03-19T03:03:32.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.319+0000 7f369fbee740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f36981acc60 con 0x55ddb5d472d0 2024-03-19T03:03:32.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.319+0000 7f3696d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ddb5d507d0 0x55ddb5d436c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.319+0000 7f3696d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ddb5d507d0 0x55ddb5d436c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41316/0 (socket says 172.21.15.33:41316) 2024-03-19T03:03:32.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.319+0000 7f3696d9d640 1 -- 172.21.15.33:0/3838338464 learned_addr learned my addr 172.21.15.33:0/3838338464 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:32.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.319+0000 7f369659c640 1 --2- 172.21.15.33:0/3838338464 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ddb5d472d0 0x55ddb5d4ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.319+0000 7f3695d9b640 1 --2- 172.21.15.33:0/3838338464 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ddb5d47bf0 0x55ddb5d5c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.319+0000 7f3696d9d640 1 -- 172.21.15.33:0/3838338464 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ddb5d472d0 msgr2=0x55ddb5d4ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:32.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.319+0000 7f3696d9d640 1 --2- 172.21.15.33:0/3838338464 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ddb5d472d0 0x55ddb5d4ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.319+0000 7f3696d9d640 1 -- 172.21.15.33:0/3838338464 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ddb5d47bf0 msgr2=0x55ddb5d5c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:32.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.319+0000 7f3696d9d640 1 --2- 172.21.15.33:0/3838338464 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ddb5d47bf0 0x55ddb5d5c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.319+0000 7f3696d9d640 1 -- 172.21.15.33:0/3838338464 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddb5d4c810 con 0x55ddb5d507d0 2024-03-19T03:03:32.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.319+0000 7f369659c640 1 --2- 172.21.15.33:0/3838338464 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ddb5d472d0 0x55ddb5d4ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:03:32.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.320+0000 7f3696d9d640 1 --2- 172.21.15.33:0/3838338464 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ddb5d507d0 0x55ddb5d436c0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f36880119b0 tx=0x7f3688009540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e661ed2434abb6db server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:32.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.320+0000 7f367ffff640 1 -- 172.21.15.33:0/3838338464 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3688004a00 con 0x55ddb5d507d0 2024-03-19T03:03:32.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.320+0000 7f367ffff640 1 -- 172.21.15.33:0/3838338464 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3688010cb0 con 0x55ddb5d507d0 2024-03-19T03:03:32.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.320+0000 7f367ffff640 1 -- 172.21.15.33:0/3838338464 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3688005500 con 0x55ddb5d507d0 2024-03-19T03:03:32.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.320+0000 7f369fbee740 1 -- 172.21.15.33:0/3838338464 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ddb5d507d0 msgr2=0x55ddb5d436c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.320+0000 7f369fbee740 1 --2- 172.21.15.33:0/3838338464 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ddb5d507d0 0x55ddb5d436c0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f36880119b0 tx=0x7f3688009540 comp rx=0 tx=0).stop 2024-03-19T03:03:32.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.320+0000 7f369fbee740 1 -- 172.21.15.33:0/3838338464 shutdown_connections 2024-03-19T03:03:32.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.320+0000 7f369fbee740 1 --2- 172.21.15.33:0/3838338464 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ddb5d507d0 0x55ddb5d436c0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.320+0000 7f369fbee740 1 --2- 172.21.15.33:0/3838338464 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ddb5d47bf0 0x55ddb5d5c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.320+0000 7f369fbee740 1 --2- 172.21.15.33:0/3838338464 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ddb5d472d0 0x55ddb5d4ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.320+0000 7f369fbee740 1 -- 172.21.15.33:0/3838338464 >> 172.21.15.33:0/3838338464 conn(0x55ddb5d41a00 msgr2=0x55ddb5d41de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:32.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.320+0000 7f369fbee740 1 -- 172.21.15.33:0/3838338464 shutdown_connections 2024-03-19T03:03:32.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.320+0000 7f369fbee740 1 -- 172.21.15.33:0/3838338464 wait complete. 2024-03-19T03:03:32.327 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.321+0000 7f369fbee740 1 -- start start 2024-03-19T03:03:32.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.321+0000 7f369fbee740 1 Processor -- start 2024-03-19T03:03:32.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.321+0000 7f369fbee740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ddb5d472d0 0x55ddb5d54580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.321+0000 7f369fbee740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ddb5d47bf0 0x55ddb5d54ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.321+0000 7f369fbee740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ddb5d507d0 0x55ddb5dc5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.321+0000 7f369fbee740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f36981ad2b0 con 0x55ddb5d507d0 2024-03-19T03:03:32.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.321+0000 7f369fbee740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f36981ae720 con 0x55ddb5d47bf0 2024-03-19T03:03:32.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.321+0000 7f369fbee740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f36981ad980 con 0x55ddb5d472d0 2024-03-19T03:03:32.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.321+0000 7f369fbee740 10 client.0 _ll_register_callbacks cb 0x7f3680004aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:03:32.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.321+0000 7f3696d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ddb5d507d0 0x55ddb5dc5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.321+0000 7f3696d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ddb5d507d0 0x55ddb5dc5ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41332/0 (socket says 172.21.15.33:41332) 2024-03-19T03:03:32.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.321+0000 7f3696d9d640 1 -- 172.21.15.33:0/3114343311 learned_addr learned my addr 172.21.15.33:0/3114343311 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:32.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.321+0000 7f3695d9b640 1 --2- 172.21.15.33:0/3114343311 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ddb5d47bf0 0x55ddb5d54ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.321+0000 7f369659c640 1 --2- 172.21.15.33:0/3114343311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ddb5d472d0 0x55ddb5d54580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.321+0000 7f3696d9d640 1 -- 172.21.15.33:0/3114343311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ddb5d472d0 msgr2=0x55ddb5d54580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:32.328 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.321+0000 7f3696d9d640 1 --2- 172.21.15.33:0/3114343311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ddb5d472d0 0x55ddb5d54580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.321+0000 7f3696d9d640 1 -- 172.21.15.33:0/3114343311 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ddb5d47bf0 msgr2=0x55ddb5d54ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:32.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.321+0000 7f3696d9d640 1 --2- 172.21.15.33:0/3114343311 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ddb5d47bf0 0x55ddb5d54ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.321+0000 7f3696d9d640 1 -- 172.21.15.33:0/3114343311 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddb5d55390 con 0x55ddb5d507d0 2024-03-19T03:03:32.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.321+0000 7f3696d9d640 1 --2- 172.21.15.33:0/3114343311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ddb5d507d0 0x55ddb5dc5ec0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f368800f7d0 tx=0x7f36880095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abe161a60420e97b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:32.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.321+0000 7f367dffb640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:03:32.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.322+0000 7f367dffb640 1 -- 172.21.15.33:0/3114343311 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f368803c020 con 0x55ddb5d507d0 2024-03-19T03:03:32.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.322+0000 7f367dffb640 1 -- 172.21.15.33:0/3114343311 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7f3688009de0 con 0x55ddb5d507d0 2024-03-19T03:03:32.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.322+0000 7f369fbee740 1 -- 172.21.15.33:0/3114343311 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3688010cb0 con 0x55ddb5d507d0 2024-03-19T03:03:32.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.322+0000 7f367dffb640 1 -- 172.21.15.33:0/3114343311 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3688032040 con 0x55ddb5d507d0 2024-03-19T03:03:32.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.322+0000 7f369fbee740 10 client.17820 fetch_fsmap learned FSMap version 25 2024-03-19T03:03:32.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.322+0000 7f369fbee740 1 -- 172.21.15.33:0/3114343311 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ddb5dc7500 con 0x55ddb5d507d0 2024-03-19T03:03:32.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.322+0000 7f367dffb640 1 -- 172.21.15.33:0/3114343311 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3688010cb0 con 0x55ddb5d507d0 2024-03-19T03:03:32.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.322+0000 7f367dffb640 1 -- 172.21.15.33:0/3114343311 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f3688035030 con 0x55ddb5d507d0 2024-03-19T03:03:32.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.322+0000 7f369fbee740 10 client.17820 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:03:32.329 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.322+0000 7f369fbee740 10 client.17820 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-19T03:03:32.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.322+0000 7f369fbee740 20 client.17820 populate_metadata read hostname 'smithi033' 2024-03-19T03:03:32.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.322+0000 7f369fbee740 1 --2- 172.21.15.33:0/3114343311 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55ddb5dc7a70 0x55ddb5dc9e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.322+0000 7f369fbee740 4 client.17820 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T03:03:32.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.322+0000 7f369fbee740 1 -- 172.21.15.33:0/3114343311 --> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ddb5cb79f0 con 0x55ddb5dc7a70 2024-03-19T03:03:32.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.322+0000 7f369659c640 1 --2- 172.21.15.33:0/3114343311 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55ddb5dc7a70 0x55ddb5dc9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.323+0000 7f369659c640 1 --2- 172.21.15.33:0/3114343311 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55ddb5dc7a70 0x55ddb5dc9e60 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=eda7a190fa25acd3 server_cookie=25ff996fc7671301 in_seq=0 out_seq=0 2024-03-19T03:03:32.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.323+0000 7f367dffb640 10 client.17820 ms_handle_connect on v2:172.21.15.96:6836/522588113 2024-03-19T03:03:32.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.323+0000 7f367dffb640 1 -- 172.21.15.33:0/3114343311 <== mds.1 v2:172.21.15.96:6836/522588113 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ddb5cb79f0 con 0x55ddb5dc7a70 2024-03-19T03:03:32.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.323+0000 7f367dffb640 10 client.17820 handle_command_reply: tid=0 2024-03-19T03:03:32.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.323+0000 7f369fbee740 1 client.17820 shutdown 2024-03-19T03:03:32.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.323+0000 7f369fbee740 1 -- 172.21.15.33:0/3114343311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ddb5d507d0 msgr2=0x55ddb5dc5ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.323+0000 7f369fbee740 1 --2- 172.21.15.33:0/3114343311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ddb5d507d0 0x55ddb5dc5ec0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f368800f7d0 tx=0x7f36880095e0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.323+0000 7f369fbee740 1 -- 172.21.15.33:0/3114343311 shutdown_connections 2024-03-19T03:03:32.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.323+0000 7f369fbee740 1 --2- 172.21.15.33:0/3114343311 >> [v2:172.21.15.96:6836/522588113,v1:172.21.15.96:6837/522588113] conn(0x55ddb5dc7a70 0x55ddb5dc9e60 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.330 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.323+0000 7f369fbee740 1 --2- 172.21.15.33:0/3114343311 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x55ddb5d507d0 0x55ddb5dc5ec0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.323+0000 7f369fbee740 1 --2- 172.21.15.33:0/3114343311 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55ddb5d47bf0 0x55ddb5d54ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.323+0000 7f367dffb640 0 client.17820 ms_handle_reset on v2:172.21.15.96:6836/522588113 2024-03-19T03:03:32.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.323+0000 7f369fbee740 1 --2- 172.21.15.33:0/3114343311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55ddb5d472d0 0x55ddb5d54580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.323+0000 7f369fbee740 1 -- 172.21.15.33:0/3114343311 >> 172.21.15.33:0/3114343311 conn(0x55ddb5d41a00 msgr2=0x55ddb5d40ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:32.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.323+0000 7f369fbee740 1 -- 172.21.15.33:0/3114343311 shutdown_connections 2024-03-19T03:03:32.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.323+0000 7f369fbee740 1 -- 172.21.15.33:0/3114343311 wait complete. 2024-03-19T03:03:32.331 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.323+0000 7f369fbee740 20 client.17820 trim_cache size 0 max 16384 2024-03-19T03:03:32.331 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T03:03:32.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.323+0000 7f369cf22640 1 -- 172.21.15.33:0/1978972323 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f36840a2970 msgr2=0x7f36840a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.323+0000 7f369cf22640 1 --2- 172.21.15.33:0/1978972323 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f36840a2970 0x7f36840a4e30 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f368800fc90 tx=0x7f3688018000 comp rx=0 tx=0).stop 2024-03-19T03:03:32.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.324+0000 7f369cf22640 1 -- 172.21.15.33:0/1978972323 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3698101070 msgr2=0x7f3698199340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.324+0000 7f369cf22640 1 --2- 172.21.15.33:0/1978972323 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3698101070 0x7f3698199340 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f368000e9f0 tx=0x7f368000edf0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.324+0000 7f369cf22640 1 -- 172.21.15.33:0/1978972323 shutdown_connections 2024-03-19T03:03:32.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.324+0000 7f369cf22640 1 --2- 172.21.15.33:0/1978972323 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f36840a2970 0x7f36840a4e30 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.324+0000 7f369cf22640 1 --2- 172.21.15.33:0/1978972323 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f36981066c0 0x7f36981a3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.332 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.324+0000 7f369cf22640 1 --2- 172.21.15.33:0/1978972323 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f3698101070 0x7f3698199340 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.324+0000 7f369cf22640 1 --2- 172.21.15.33:0/1978972323 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f36981006b0 0x7f3698198e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.324+0000 7f369cf22640 1 -- 172.21.15.33:0/1978972323 >> 172.21.15.33:0/1978972323 conn(0x7f3698076aa0 msgr2=0x7f36980fe200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:32.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.324+0000 7f369cf22640 1 -- 172.21.15.33:0/1978972323 shutdown_connections 2024-03-19T03:03:32.333 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.324+0000 7f369cf22640 1 -- 172.21.15.33:0/1978972323 wait complete. 2024-03-19T03:03:32.334 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:32 smithi033 ceph-mon[31973]: pgmap v313: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:03:32.334 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:32 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3725033837' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:03:32.335 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-19T03:03:32.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.439+0000 7fc7338e8640 1 -- 172.21.15.33:0/1388258697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc72c102150 msgr2=0x7fc72c10e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.439+0000 7fc7338e8640 1 --2- 172.21.15.33:0/1388258697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc72c102150 0x7fc72c10e660 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fc71c009cd0 tx=0x7fc71c02c0f0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.439+0000 7fc7338e8640 1 -- 172.21.15.33:0/1388258697 shutdown_connections 2024-03-19T03:03:32.443 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.439+0000 7fc7338e8640 1 --2- 172.21.15.33:0/1388258697 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc72c102150 0x7fc72c10e660 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.439+0000 7fc7338e8640 1 --2- 172.21.15.33:0/1388258697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc72c101790 0x7fc72c101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.439+0000 7fc7338e8640 1 --2- 172.21.15.33:0/1388258697 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc72c1077a0 0x7fc72c107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.439+0000 7fc7338e8640 1 -- 172.21.15.33:0/1388258697 >> 172.21.15.33:0/1388258697 conn(0x7fc72c0fd2c0 msgr2=0x7fc72c0ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:32.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.440+0000 7fc7338e8640 1 -- 172.21.15.33:0/1388258697 shutdown_connections 2024-03-19T03:03:32.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.440+0000 7fc7338e8640 1 -- 172.21.15.33:0/1388258697 wait complete. 2024-03-19T03:03:32.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.440+0000 7fc7338e8640 1 Processor -- start 2024-03-19T03:03:32.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.440+0000 7fc7338e8640 1 -- start start 2024-03-19T03:03:32.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.440+0000 7fc7338e8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc72c101790 0x7fc72c197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.444 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.440+0000 7fc7338e8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc72c102150 0x7fc72c1984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.440+0000 7fc7338e8640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc72c1077a0 0x7fc72c19c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.440+0000 7fc7338e8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc72c111a60 con 0x7fc72c101790 2024-03-19T03:03:32.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.440+0000 7fc7338e8640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc72c1118e0 con 0x7fc72c102150 2024-03-19T03:03:32.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.440+0000 7fc7338e8640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc72c111be0 con 0x7fc72c1077a0 2024-03-19T03:03:32.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.440+0000 7fc731684640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc72c101790 0x7fc72c197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.440+0000 7fc731684640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc72c101790 0x7fc72c197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41358/0 (socket says 172.21.15.33:41358) 2024-03-19T03:03:32.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.440+0000 7fc731684640 1 -- 172.21.15.33:0/2654921387 learned_addr learned my addr 172.21.15.33:0/2654921387 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:32.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.440+0000 7fc730e83640 1 --2- 172.21.15.33:0/2654921387 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc72c102150 0x7fc72c1984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.441+0000 7fc731e85640 1 --2- 172.21.15.33:0/2654921387 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc72c1077a0 0x7fc72c19c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.441+0000 7fc731684640 1 -- 172.21.15.33:0/2654921387 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc72c1077a0 msgr2=0x7fc72c19c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.441+0000 7fc731684640 1 --2- 172.21.15.33:0/2654921387 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc72c1077a0 0x7fc72c19c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.441+0000 7fc731684640 1 -- 172.21.15.33:0/2654921387 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc72c102150 msgr2=0x7fc72c1984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.441+0000 7fc731684640 1 --2- 172.21.15.33:0/2654921387 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc72c102150 0x7fc72c1984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.445 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.441+0000 7fc731684640 1 -- 172.21.15.33:0/2654921387 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc72c19cf70 con 0x7fc72c101790 2024-03-19T03:03:32.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.441+0000 7fc731684640 1 --2- 172.21.15.33:0/2654921387 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc72c101790 0x7fc72c197f80 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fc72400e9f0 tx=0x7fc72400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:32.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.441+0000 7fc72a7fc640 1 -- 172.21.15.33:0/2654921387 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72400cd80 con 0x7fc72c101790 2024-03-19T03:03:32.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.441+0000 7fc7338e8640 1 -- 172.21.15.33:0/2654921387 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc72c19d260 con 0x7fc72c101790 2024-03-19T03:03:32.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.441+0000 7fc72a7fc640 1 -- 172.21.15.33:0/2654921387 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc7240054d0 con 0x7fc72c101790 2024-03-19T03:03:32.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.441+0000 7fc72a7fc640 1 -- 172.21.15.33:0/2654921387 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc724010490 con 0x7fc72c101790 2024-03-19T03:03:32.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.441+0000 7fc7338e8640 1 -- 172.21.15.33:0/2654921387 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc72c1a4af0 con 0x7fc72c101790 2024-03-19T03:03:32.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.443+0000 7fc72a7fc640 1 -- 172.21.15.33:0/2654921387 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fc724005020 con 0x7fc72c101790 2024-03-19T03:03:32.454 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.444+0000 7fc7365ac740 1 Processor -- start 2024-03-19T03:03:32.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.444+0000 7fc7365ac740 1 -- start start 2024-03-19T03:03:32.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.444+0000 7fc7365ac740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f55dd5f80 0x557f55dd6360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.444+0000 7fc7365ac740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f55dd68a0 0x557f55de36a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.444+0000 7fc7365ac740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f55de3be0 0x557f55de5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.444+0000 7fc7365ac740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557f55d394b0 con 0x557f55dd68a0 2024-03-19T03:03:32.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.444+0000 7fc7365ac740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557f55ce1d70 con 0x557f55dd5f80 2024-03-19T03:03:32.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.444+0000 7fc7365ac740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x557f55d35550 con 0x557f55de3be0 2024-03-19T03:03:32.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.444+0000 7fc72a7fc640 1 --2- 172.21.15.33:0/2654921387 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fc7140a2970 0x7fc7140a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.444+0000 7fc731684640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f55dd68a0 0x557f55de36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.444+0000 7fc731684640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f55dd68a0 0x557f55de36a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41372/0 (socket says 172.21.15.33:41372) 2024-03-19T03:03:32.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.444+0000 7fc731684640 1 -- 172.21.15.33:0/374864423 learned_addr learned my addr 172.21.15.33:0/374864423 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:32.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.444+0000 7fc731e85640 1 --2- 172.21.15.33:0/374864423 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f55dd5f80 0x557f55dd6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.444+0000 7fc730e83640 1 --2- 172.21.15.33:0/374864423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f55de3be0 0x557f55de5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.444+0000 7fc731e85640 1 --2- 172.21.15.33:0/2654921387 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fc7140a2970 0x7fc7140a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.444+0000 7fc72a7fc640 1 -- 172.21.15.33:0/2654921387 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7fc7240cf570 con 0x7fc72c101790 2024-03-19T03:03:32.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.445+0000 7fc731684640 1 -- 172.21.15.33:0/374864423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f55de3be0 msgr2=0x557f55de5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:32.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.445+0000 7fc731684640 1 --2- 172.21.15.33:0/374864423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f55de3be0 0x557f55de5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.445+0000 7fc731684640 1 -- 172.21.15.33:0/374864423 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f55dd5f80 msgr2=0x557f55dd6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:32.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.445+0000 7fc731684640 1 --2- 172.21.15.33:0/374864423 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f55dd5f80 0x557f55dd6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.445+0000 7fc731684640 1 -- 172.21.15.33:0/374864423 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f55d04210 con 0x557f55dd68a0 2024-03-19T03:03:32.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.445+0000 7fc730e83640 1 --2- 172.21.15.33:0/374864423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f55de3be0 0x557f55de5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:32.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.445+0000 7fc731e85640 1 --2- 172.21.15.33:0/374864423 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f55dd5f80 0x557f55dd6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:32.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.445+0000 7fc731684640 1 --2- 172.21.15.33:0/374864423 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f55dd68a0 0x557f55de36a0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fc724097c00 tx=0x7fc724004990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13b4f5e5cc241c5a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:32.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.445+0000 7fc731e85640 1 --2- 172.21.15.33:0/2654921387 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fc7140a2970 0x7fc7140a4e30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc71c005cc0 tx=0x7fc71c005c50 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:32.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.445+0000 7fc7137fe640 1 -- 172.21.15.33:0/374864423 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc724005310 con 0x557f55dd68a0 2024-03-19T03:03:32.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.445+0000 7fc7137fe640 1 -- 172.21.15.33:0/374864423 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc724005680 con 0x557f55dd68a0 2024-03-19T03:03:32.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.445+0000 7fc7137fe640 1 -- 172.21.15.33:0/374864423 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7240046c0 con 0x557f55dd68a0 2024-03-19T03:03:32.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.446+0000 7fc7365ac740 1 -- 172.21.15.33:0/374864423 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f55dd68a0 msgr2=0x557f55de36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.456 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.446+0000 7fc7365ac740 1 --2- 172.21.15.33:0/374864423 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f55dd68a0 0x557f55de36a0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fc724097c00 tx=0x7fc724004990 comp rx=0 tx=0).stop 2024-03-19T03:03:32.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.446+0000 7fc7365ac740 1 -- 172.21.15.33:0/374864423 shutdown_connections 2024-03-19T03:03:32.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.446+0000 7fc7365ac740 1 --2- 172.21.15.33:0/374864423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f55de3be0 0x557f55de5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.446+0000 7fc7365ac740 1 --2- 172.21.15.33:0/374864423 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f55dd68a0 0x557f55de36a0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.446+0000 7fc7365ac740 1 --2- 172.21.15.33:0/374864423 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f55dd5f80 0x557f55dd6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.446+0000 7fc7365ac740 1 -- 172.21.15.33:0/374864423 >> 172.21.15.33:0/374864423 conn(0x557f55dd0a00 msgr2=0x557f55dd0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:32.457 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.446+0000 7fc7365ac740 1 -- 172.21.15.33:0/374864423 shutdown_connections 2024-03-19T03:03:32.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.446+0000 7fc7365ac740 1 -- 172.21.15.33:0/374864423 wait complete. 2024-03-19T03:03:32.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.446+0000 7fc7365ac740 1 -- start start 2024-03-19T03:03:32.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.446+0000 7fc7365ac740 1 Processor -- start 2024-03-19T03:03:32.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.446+0000 7fc7365ac740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f55dd5f80 0x557f55de0c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.446+0000 7fc7365ac740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f55dd68a0 0x557f55de1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.446+0000 7fc7365ac740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f55de3be0 0x557f55de16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.446+0000 7fc7365ac740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x557f55de6a20 con 0x557f55dd5f80 2024-03-19T03:03:32.458 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.446+0000 7fc7365ac740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x557f55cf5310 con 0x557f55dd68a0 2024-03-19T03:03:32.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.446+0000 7fc7365ac740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x557f55de6ba0 con 0x557f55de3be0 2024-03-19T03:03:32.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.447+0000 7fc7365ac740 10 client.0 _ll_register_callbacks cb 0x557f55ce92d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:03:32.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.447+0000 7fc731684640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f55dd5f80 0x557f55de0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.447+0000 7fc731684640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f55dd5f80 0x557f55de0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41374/0 (socket says 172.21.15.33:41374) 2024-03-19T03:03:32.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.447+0000 7fc731684640 1 -- 172.21.15.33:0/3249249267 learned_addr learned my addr 172.21.15.33:0/3249249267 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:32.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.447+0000 7fc730e83640 1 --2- 172.21.15.33:0/3249249267 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f55dd68a0 0x557f55de1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.447+0000 7fc731e85640 1 --2- 172.21.15.33:0/3249249267 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f55de3be0 0x557f55de16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.447+0000 7fc731684640 1 -- 172.21.15.33:0/3249249267 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f55de3be0 msgr2=0x557f55de16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:32.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.447+0000 7fc731684640 1 --2- 172.21.15.33:0/3249249267 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f55de3be0 0x557f55de16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.447+0000 7fc731684640 1 -- 172.21.15.33:0/3249249267 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f55dd68a0 msgr2=0x557f55de1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:32.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.447+0000 7fc731684640 1 --2- 172.21.15.33:0/3249249267 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f55dd68a0 0x557f55de1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.459 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.447+0000 7fc731684640 1 -- 172.21.15.33:0/3249249267 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f55a4d3f0 con 0x557f55dd5f80 2024-03-19T03:03:32.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.447+0000 7fc731e85640 1 --2- 172.21.15.33:0/3249249267 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f55de3be0 0x557f55de16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:03:32.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.447+0000 7fc730e83640 1 --2- 172.21.15.33:0/3249249267 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f55dd68a0 0x557f55de1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:32.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.447+0000 7fc731684640 1 --2- 172.21.15.33:0/3249249267 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f55dd5f80 0x557f55de0c20 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fc72409b730 tx=0x7fc7240048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db0d7557350559 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:32.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.447+0000 7fc710ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:03:32.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.447+0000 7fc710ff9640 1 -- 172.21.15.33:0/3249249267 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc724005310 con 0x557f55dd5f80 2024-03-19T03:03:32.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.447+0000 7fc710ff9640 1 -- 172.21.15.33:0/3249249267 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7fc7240ccb10 con 0x557f55dd5f80 2024-03-19T03:03:32.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.447+0000 7fc7365ac740 1 -- 172.21.15.33:0/3249249267 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc724005680 con 0x557f55dd5f80 2024-03-19T03:03:32.460 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.447+0000 7fc710ff9640 1 -- 172.21.15.33:0/3249249267 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc7240ccd10 con 0x557f55dd5f80 2024-03-19T03:03:32.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.447+0000 7fc7365ac740 10 client.17832 fetch_fsmap learned FSMap version 25 2024-03-19T03:03:32.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.447+0000 7fc7365ac740 1 -- 172.21.15.33:0/3249249267 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x557f55b177d0 con 0x557f55dd5f80 2024-03-19T03:03:32.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.448+0000 7fc710ff9640 1 -- 172.21.15.33:0/3249249267 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc724005680 con 0x557f55dd5f80 2024-03-19T03:03:32.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.448+0000 7fc710ff9640 1 -- 172.21.15.33:0/3249249267 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc724084ce0 con 0x557f55dd5f80 2024-03-19T03:03:32.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.448+0000 7fc7365ac740 10 client.17832 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:03:32.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.448+0000 7fc7365ac740 10 client.17832 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:03:32.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.448+0000 7fc7365ac740 20 client.17832 populate_metadata read hostname 'smithi033' 2024-03-19T03:03:32.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.448+0000 7fc7365ac740 1 --2- 172.21.15.33:0/3249249267 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557f55df3f00 0x557f55df62f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.448+0000 7fc7365ac740 4 client.17832 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:03:32.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.448+0000 7fc7365ac740 1 -- 172.21.15.33:0/3249249267 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f55d973e0 con 0x557f55df3f00 2024-03-19T03:03:32.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.448+0000 7fc730e83640 1 --2- 172.21.15.33:0/3249249267 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557f55df3f00 0x557f55df62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.448+0000 7fc730e83640 1 --2- 172.21.15.33:0/3249249267 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557f55df3f00 0x557f55df62f0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f71a6a1271edf073 server_cookie=791844f3ec4e8efd in_seq=0 out_seq=0 2024-03-19T03:03:32.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.448+0000 7fc710ff9640 10 client.17832 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:03:32.461 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.449+0000 7fc710ff9640 1 -- 172.21.15.33:0/3249249267 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557f55d973e0 con 0x557f55df3f00 2024-03-19T03:03:32.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.449+0000 7fc710ff9640 10 client.17832 handle_command_reply: tid=0 2024-03-19T03:03:32.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.449+0000 7fc7365ac740 1 client.17832 shutdown 2024-03-19T03:03:32.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.449+0000 7fc7365ac740 1 -- 172.21.15.33:0/3249249267 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f55dd5f80 msgr2=0x557f55de0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.449+0000 7fc7365ac740 1 --2- 172.21.15.33:0/3249249267 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f55dd5f80 0x557f55de0c20 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fc72409b730 tx=0x7fc7240048f0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.449+0000 7fc7365ac740 1 -- 172.21.15.33:0/3249249267 shutdown_connections 2024-03-19T03:03:32.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.449+0000 7fc7365ac740 1 --2- 172.21.15.33:0/3249249267 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557f55df3f00 0x557f55df62f0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.449+0000 7fc7365ac740 1 --2- 172.21.15.33:0/3249249267 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f55de3be0 0x557f55de16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.449+0000 7fc7365ac740 1 --2- 172.21.15.33:0/3249249267 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f55dd68a0 0x557f55de1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.449+0000 7fc710ff9640 0 client.17832 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:03:32.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.449+0000 7fc7365ac740 1 --2- 172.21.15.33:0/3249249267 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f55dd5f80 0x557f55de0c20 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.449+0000 7fc7365ac740 1 -- 172.21.15.33:0/3249249267 >> 172.21.15.33:0/3249249267 conn(0x557f55dd0a00 msgr2=0x557f55dd2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:32.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.449+0000 7fc7365ac740 1 -- 172.21.15.33:0/3249249267 shutdown_connections 2024-03-19T03:03:32.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.449+0000 7fc7365ac740 1 -- 172.21.15.33:0/3249249267 wait complete. 2024-03-19T03:03:32.462 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.449+0000 7fc7365ac740 20 client.17832 trim_cache size 0 max 16384 2024-03-19T03:03:32.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.463+0000 7fc7365ac740 1 Processor -- start 2024-03-19T03:03:32.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.463+0000 7fc7365ac740 1 -- start start 2024-03-19T03:03:32.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.463+0000 7fc7365ac740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f55dd5f80 0x557f55de3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.463+0000 7fc7365ac740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f55dd68a0 0x557f55def860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.463+0000 7fc7365ac740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f55de3be0 0x557f55dd26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.463+0000 7fc7365ac740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc72c1af370 con 0x557f55dd5f80 2024-03-19T03:03:32.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.463+0000 7fc7365ac740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc72c1acc40 con 0x557f55dd68a0 2024-03-19T03:03:32.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.463+0000 7fc7365ac740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc72c1aaab0 con 0x557f55de3be0 2024-03-19T03:03:32.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.463+0000 7fc731684640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f55dd5f80 0x557f55de3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.463+0000 7fc731684640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f55dd5f80 0x557f55de3350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41378/0 (socket says 172.21.15.33:41378) 2024-03-19T03:03:32.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.463+0000 7fc731684640 1 -- 172.21.15.33:0/3661537007 learned_addr learned my addr 172.21.15.33:0/3661537007 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:32.466 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.463+0000 7fc731e85640 1 --2- 172.21.15.33:0/3661537007 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f55de3be0 0x557f55dd26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.463+0000 7fc730e83640 1 --2- 172.21.15.33:0/3661537007 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f55dd68a0 0x557f55def860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.463+0000 7fc731684640 1 -- 172.21.15.33:0/3661537007 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f55de3be0 msgr2=0x557f55dd26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:32.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.463+0000 7fc731684640 1 --2- 172.21.15.33:0/3661537007 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f55de3be0 0x557f55dd26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.463+0000 7fc731684640 1 -- 172.21.15.33:0/3661537007 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f55dd68a0 msgr2=0x557f55def860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:32.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.463+0000 7fc731684640 1 --2- 172.21.15.33:0/3661537007 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f55dd68a0 0x557f55def860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.463+0000 7fc731684640 1 -- 172.21.15.33:0/3661537007 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f55ddfbd0 con 0x557f55dd5f80 2024-03-19T03:03:32.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.463+0000 7fc731684640 1 --2- 172.21.15.33:0/3661537007 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f55dd5f80 0x557f55de3350 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fc724098170 tx=0x7fc7240cf7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f898975ff1a86b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:32.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.463+0000 7fc712ffd640 1 -- 172.21.15.33:0/3661537007 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72409b2e0 con 0x557f55dd5f80 2024-03-19T03:03:32.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.464+0000 7fc712ffd640 1 -- 172.21.15.33:0/3661537007 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc7240afe50 con 0x557f55dd5f80 2024-03-19T03:03:32.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.464+0000 7fc712ffd640 1 -- 172.21.15.33:0/3661537007 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72409b7a0 con 0x557f55dd5f80 2024-03-19T03:03:32.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.464+0000 7fc7365ac740 1 -- 172.21.15.33:0/3661537007 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f55dd5f80 msgr2=0x557f55de3350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.464+0000 7fc7365ac740 1 --2- 172.21.15.33:0/3661537007 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f55dd5f80 0x557f55de3350 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fc724098170 tx=0x7fc7240cf7f0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.464+0000 7fc7365ac740 1 -- 172.21.15.33:0/3661537007 shutdown_connections 2024-03-19T03:03:32.467 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.464+0000 7fc7365ac740 1 --2- 172.21.15.33:0/3661537007 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f55de3be0 0x557f55dd26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.464+0000 7fc7365ac740 1 --2- 172.21.15.33:0/3661537007 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f55dd68a0 0x557f55def860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.464+0000 7fc7365ac740 1 --2- 172.21.15.33:0/3661537007 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f55dd5f80 0x557f55de3350 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.464+0000 7fc7365ac740 1 -- 172.21.15.33:0/3661537007 >> 172.21.15.33:0/3661537007 conn(0x557f55dd0a00 msgr2=0x557f55dd0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:32.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.464+0000 7fc7365ac740 1 -- 172.21.15.33:0/3661537007 shutdown_connections 2024-03-19T03:03:32.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.464+0000 7fc7365ac740 1 -- 172.21.15.33:0/3661537007 wait complete. 2024-03-19T03:03:32.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.464+0000 7fc7365ac740 1 -- start start 2024-03-19T03:03:32.468 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.464+0000 7fc7365ac740 1 Processor -- start 2024-03-19T03:03:32.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.465+0000 7fc7365ac740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f55dd5f80 0x557f55de8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.465+0000 7fc7365ac740 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f55dd68a0 0x557f55de8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.465+0000 7fc7365ac740 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f55de3be0 0x557f55e52f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.465+0000 7fc7365ac740 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fc72c1ab100 con 0x557f55dd68a0 2024-03-19T03:03:32.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.465+0000 7fc7365ac740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc72c1ac570 con 0x557f55dd5f80 2024-03-19T03:03:32.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.465+0000 7fc7365ac740 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fc72c1ab7d0 con 0x557f55de3be0 2024-03-19T03:03:32.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.465+0000 7fc7365ac740 10 client.0 _ll_register_callbacks cb 0x7fc720005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:03:32.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.465+0000 7fc730e83640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f55dd68a0 0x557f55de8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.465+0000 7fc731684640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f55dd5f80 0x557f55de8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.465+0000 7fc730e83640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f55dd68a0 0x557f55de8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:41394/0 (socket says 172.21.15.33:41394) 2024-03-19T03:03:32.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.465+0000 7fc730e83640 1 -- 172.21.15.33:0/1318630934 learned_addr learned my addr 172.21.15.33:0/1318630934 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:32.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.465+0000 7fc731e85640 1 --2- 172.21.15.33:0/1318630934 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f55de3be0 0x557f55e52f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.465+0000 7fc730e83640 1 -- 172.21.15.33:0/1318630934 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f55de3be0 msgr2=0x557f55e52f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:32.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.465+0000 7fc730e83640 1 --2- 172.21.15.33:0/1318630934 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f55de3be0 0x557f55e52f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.465+0000 7fc730e83640 1 -- 172.21.15.33:0/1318630934 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f55dd5f80 msgr2=0x557f55de8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:03:32.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.465+0000 7fc730e83640 1 --2- 172.21.15.33:0/1318630934 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f55dd5f80 0x557f55de8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.465+0000 7fc730e83640 1 -- 172.21.15.33:0/1318630934 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f55e53590 con 0x557f55dd68a0 2024-03-19T03:03:32.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.465+0000 7fc731684640 1 --2- 172.21.15.33:0/1318630934 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f55dd5f80 0x557f55de8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:32.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.465+0000 7fc730e83640 1 --2- 172.21.15.33:0/1318630934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f55dd68a0 0x557f55de8540 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x557f55dd5c70 tx=0x7fc720012bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b80f9fb3334b589 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:32.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.465+0000 7fc710ff9640 10 client.0 ms_handle_connect on v2:172.21.15.33:3300/0 2024-03-19T03:03:32.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.466+0000 7fc710ff9640 1 -- 172.21.15.33:0/1318630934 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc720004250 con 0x557f55dd68a0 2024-03-19T03:03:32.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.466+0000 7fc710ff9640 1 -- 172.21.15.33:0/1318630934 <== mon.0 v2:172.21.15.33:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7fc7200043f0 con 0x557f55dd68a0 2024-03-19T03:03:32.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.466+0000 7fc7365ac740 1 -- 172.21.15.33:0/1318630934 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7240afe50 con 0x557f55dd68a0 2024-03-19T03:03:32.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.466+0000 7fc710ff9640 1 -- 172.21.15.33:0/1318630934 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc720033050 con 0x557f55dd68a0 2024-03-19T03:03:32.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.466+0000 7fc7365ac740 10 client.17838 fetch_fsmap learned FSMap version 25 2024-03-19T03:03:32.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.466+0000 7fc7365ac740 1 -- 172.21.15.33:0/1318630934 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x557f55e5a990 con 0x557f55dd68a0 2024-03-19T03:03:32.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.466+0000 7fc710ff9640 1 -- 172.21.15.33:0/1318630934 <== mon.0 v2:172.21.15.33:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc7240afe50 con 0x557f55dd68a0 2024-03-19T03:03:32.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.466+0000 7fc710ff9640 1 -- 172.21.15.33:0/1318630934 <== mon.0 v2:172.21.15.33:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc720022420 con 0x557f55dd68a0 2024-03-19T03:03:32.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.466+0000 7fc7365ac740 10 client.17838 fetch_fsmap finished waiting for FSMap version 25 2024-03-19T03:03:32.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.466+0000 7fc7365ac740 10 client.17838 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-19T03:03:32.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.466+0000 7fc7365ac740 20 client.17838 populate_metadata read hostname 'smithi033' 2024-03-19T03:03:32.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.466+0000 7fc7365ac740 1 --2- 172.21.15.33:0/1318630934 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557f55e5b140 0x557f55e5d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:32.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.466+0000 7fc7365ac740 4 client.17838 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-19T03:03:32.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.466+0000 7fc7365ac740 1 -- 172.21.15.33:0/1318630934 --> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557f55d469f0 con 0x557f55e5b140 2024-03-19T03:03:32.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.466+0000 7fc731684640 1 --2- 172.21.15.33:0/1318630934 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557f55e5b140 0x557f55e5d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:32.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.467+0000 7fc710ff9640 10 client.17838 ms_handle_connect on v2:172.21.15.96:6838/3731232169 2024-03-19T03:03:32.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.467+0000 7fc731684640 1 --2- 172.21.15.33:0/1318630934 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557f55e5b140 0x557f55e5d560 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e97ef8eaf817f693 server_cookie=8b9d8adff99d1f81 in_seq=0 out_seq=0 2024-03-19T03:03:32.474 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.467+0000 7fc710ff9640 1 -- 172.21.15.33:0/1318630934 <== mds.0 v2:172.21.15.96:6838/3731232169 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557f55d469f0 con 0x557f55e5b140 2024-03-19T03:03:32.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.467+0000 7fc710ff9640 10 client.17838 handle_command_reply: tid=0 2024-03-19T03:03:32.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.467+0000 7fc7365ac740 1 client.17838 shutdown 2024-03-19T03:03:32.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.467+0000 7fc7365ac740 1 -- 172.21.15.33:0/1318630934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f55dd68a0 msgr2=0x557f55de8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.467+0000 7fc7365ac740 1 --2- 172.21.15.33:0/1318630934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f55dd68a0 0x557f55de8540 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x557f55dd5c70 tx=0x7fc720012bb0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.467+0000 7fc7365ac740 1 -- 172.21.15.33:0/1318630934 shutdown_connections 2024-03-19T03:03:32.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.467+0000 7fc7365ac740 1 --2- 172.21.15.33:0/1318630934 >> [v2:172.21.15.96:6838/3731232169,v1:172.21.15.96:6839/3731232169] conn(0x557f55e5b140 0x557f55e5d560 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.467+0000 7fc7365ac740 1 --2- 172.21.15.33:0/1318630934 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x557f55de3be0 0x557f55e52f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.467+0000 7fc7365ac740 1 --2- 172.21.15.33:0/1318630934 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x557f55dd68a0 0x557f55de8540 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.467+0000 7fc7365ac740 1 --2- 172.21.15.33:0/1318630934 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x557f55dd5f80 0x557f55de8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.467+0000 7fc710ff9640 0 client.17838 ms_handle_reset on v2:172.21.15.96:6838/3731232169 2024-03-19T03:03:32.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.467+0000 7fc7365ac740 1 -- 172.21.15.33:0/1318630934 >> 172.21.15.33:0/1318630934 conn(0x557f55dd0a00 msgr2=0x557f55dcfba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:32.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.467+0000 7fc7365ac740 1 -- 172.21.15.33:0/1318630934 shutdown_connections 2024-03-19T03:03:32.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.467+0000 7fc7365ac740 1 -- 172.21.15.33:0/1318630934 wait complete. 2024-03-19T03:03:32.475 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.467+0000 7fc7365ac740 20 client.17838 trim_cache size 0 max 16384 2024-03-19T03:03:32.476 INFO:teuthology.orchestra.run.smithi033.stdout:[] 2024-03-19T03:03:32.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.468+0000 7fc7338e8640 1 -- 172.21.15.33:0/2654921387 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fc7140a2970 msgr2=0x7fc7140a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.476 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.468+0000 7fc7338e8640 1 --2- 172.21.15.33:0/2654921387 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fc7140a2970 0x7fc7140a4e30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc71c005cc0 tx=0x7fc71c005c50 comp rx=0 tx=0).stop 2024-03-19T03:03:32.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.468+0000 7fc7338e8640 1 -- 172.21.15.33:0/2654921387 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc72c101790 msgr2=0x7fc72c197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:32.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.468+0000 7fc7338e8640 1 --2- 172.21.15.33:0/2654921387 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc72c101790 0x7fc72c197f80 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fc72400e9f0 tx=0x7fc72400edf0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.468+0000 7fc7338e8640 1 -- 172.21.15.33:0/2654921387 shutdown_connections 2024-03-19T03:03:32.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.468+0000 7fc7338e8640 1 --2- 172.21.15.33:0/2654921387 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fc7140a2970 0x7fc7140a4e30 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.468+0000 7fc7338e8640 1 --2- 172.21.15.33:0/2654921387 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc72c1077a0 0x7fc72c19c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.468+0000 7fc7338e8640 1 --2- 172.21.15.33:0/2654921387 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc72c102150 0x7fc72c1984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.468+0000 7fc7338e8640 1 --2- 172.21.15.33:0/2654921387 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fc72c101790 0x7fc72c197f80 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:32.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.468+0000 7fc7338e8640 1 -- 172.21.15.33:0/2654921387 >> 172.21.15.33:0/2654921387 conn(0x7fc72c0fd2c0 msgr2=0x7fc72c0fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:32.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.468+0000 7fc7338e8640 1 -- 172.21.15.33:0/2654921387 shutdown_connections 2024-03-19T03:03:32.477 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:32.468+0000 7fc7338e8640 1 -- 172.21.15.33:0/2654921387 wait complete. 2024-03-19T03:03:32.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:32 smithi155 ceph-mon[35804]: pgmap v313: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:03:32.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:32 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3725033837' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:03:32.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:32 smithi096 ceph-mon[35946]: pgmap v313: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:03:32.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:32 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3725033837' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:03:33.480 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-03-19T03:03:33.480 INFO:tasks.fwd_scrub:done joining 2024-03-19T03:03:33.480 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-19T03:03:33.492 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T03:03:33.609 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.606+0000 7fbe488dc640 1 -- 172.21.15.33:0/3080394292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe4410be20 msgr2=0x7fbe4410e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:33.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.606+0000 7fbe488dc640 1 --2- 172.21.15.33:0/3080394292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe4410be20 0x7fbe4410e250 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c009ca0 tx=0x7fbe3c02c180 comp rx=0 tx=0).stop 2024-03-19T03:03:33.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.606+0000 7fbe488dc640 1 -- 172.21.15.33:0/3080394292 shutdown_connections 2024-03-19T03:03:33.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.606+0000 7fbe488dc640 1 --2- 172.21.15.33:0/3080394292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe4410be20 0x7fbe4410e250 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:33.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.606+0000 7fbe488dc640 1 --2- 172.21.15.33:0/3080394292 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe44103370 0x7fbe4410b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:33.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.606+0000 7fbe488dc640 1 --2- 172.21.15.33:0/3080394292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe44102a50 0x7fbe44102e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:33.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.606+0000 7fbe488dc640 1 -- 172.21.15.33:0/3080394292 >> 172.21.15.33:0/3080394292 conn(0x7fbe44076aa0 msgr2=0x7fbe44076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:33.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.606+0000 7fbe488dc640 1 -- 172.21.15.33:0/3080394292 shutdown_connections 2024-03-19T03:03:33.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.606+0000 7fbe488dc640 1 -- 172.21.15.33:0/3080394292 wait complete. 2024-03-19T03:03:33.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.607+0000 7fbe488dc640 1 Processor -- start 2024-03-19T03:03:33.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.607+0000 7fbe488dc640 1 -- start start 2024-03-19T03:03:33.610 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.607+0000 7fbe488dc640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe44102a50 0x7fbe44198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:33.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.607+0000 7fbe488dc640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe44103370 0x7fbe441985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:33.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.607+0000 7fbe488dc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe4410be20 0x7fbe4419c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:33.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.607+0000 7fbe488dc640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbe44111a60 con 0x7fbe4410be20 2024-03-19T03:03:33.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.607+0000 7fbe488dc640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbe441118e0 con 0x7fbe44103370 2024-03-19T03:03:33.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.607+0000 7fbe488dc640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbe44111be0 con 0x7fbe44102a50 2024-03-19T03:03:33.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.607+0000 7fbe42d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe4410be20 0x7fbe4419c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:33.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.607+0000 7fbe42d9d640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe4410be20 0x7fbe4419c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55272/0 (socket says 172.21.15.33:55272) 2024-03-19T03:03:33.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.607+0000 7fbe42d9d640 1 -- 172.21.15.33:0/392437009 learned_addr learned my addr 172.21.15.33:0/392437009 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:33.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.607+0000 7fbe4259c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe44102a50 0x7fbe44198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:33.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.607+0000 7fbe41d9b640 1 --2- 172.21.15.33:0/392437009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe44103370 0x7fbe441985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:33.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.608+0000 7fbe42d9d640 1 -- 172.21.15.33:0/392437009 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe44102a50 msgr2=0x7fbe44198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:33.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.608+0000 7fbe42d9d640 1 --2- 172.21.15.33:0/392437009 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe44102a50 0x7fbe44198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:33.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.608+0000 7fbe42d9d640 1 -- 172.21.15.33:0/392437009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe44103370 msgr2=0x7fbe441985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:33.611 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.608+0000 7fbe42d9d640 1 --2- 172.21.15.33:0/392437009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe44103370 0x7fbe441985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:33.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.608+0000 7fbe42d9d640 1 -- 172.21.15.33:0/392437009 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe4419d050 con 0x7fbe4410be20 2024-03-19T03:03:33.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.608+0000 7fbe41d9b640 1 --2- 172.21.15.33:0/392437009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe44103370 0x7fbe441985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:33.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.608+0000 7fbe42d9d640 1 --2- 172.21.15.33:0/392437009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe4410be20 0x7fbe4419c920 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c02c790 tx=0x7fbe3c035d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:33.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.608+0000 7fbe337fe640 1 -- 172.21.15.33:0/392437009 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c039040 con 0x7fbe4410be20 2024-03-19T03:03:33.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.608+0000 7fbe488dc640 1 -- 172.21.15.33:0/392437009 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe4419d2e0 con 0x7fbe4410be20 2024-03-19T03:03:33.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.608+0000 7fbe337fe640 1 -- 172.21.15.33:0/392437009 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe3c044070 con 0x7fbe4410be20 2024-03-19T03:03:33.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.608+0000 7fbe337fe640 1 -- 172.21.15.33:0/392437009 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c004860 con 0x7fbe4410be20 2024-03-19T03:03:33.612 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.608+0000 7fbe488dc640 1 -- 172.21.15.33:0/392437009 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe441a4bd0 con 0x7fbe4410be20 2024-03-19T03:03:33.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.609+0000 7fbe488dc640 1 -- 172.21.15.33:0/392437009 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe4406a670 con 0x7fbe4410be20 2024-03-19T03:03:33.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.610+0000 7fbe337fe640 1 -- 172.21.15.33:0/392437009 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fbe3c004040 con 0x7fbe4410be20 2024-03-19T03:03:33.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.611+0000 7fbe337fe640 1 --2- 172.21.15.33:0/392437009 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbe2c0a2970 0x7fbe2c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:33.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.611+0000 7fbe337fe640 1 -- 172.21.15.33:0/392437009 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7fbe3c0f1490 con 0x7fbe4410be20 2024-03-19T03:03:33.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.611+0000 7fbe4259c640 1 --2- 172.21.15.33:0/392437009 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbe2c0a2970 0x7fbe2c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:33.613 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.611+0000 7fbe4259c640 1 --2- 172.21.15.33:0/392437009 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbe2c0a2970 0x7fbe2c0a4e30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbe34009bd0 tx=0x7fbe34005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:33.616 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.615+0000 7fbe337fe640 1 -- 172.21.15.33:0/392437009 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbe3c0bdfd0 con 0x7fbe4410be20 2024-03-19T03:03:33.802 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.801+0000 7fbe488dc640 1 -- 172.21.15.33:0/392437009 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbe44106f10 con 0x7fbe4410be20 2024-03-19T03:03:33.804 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.802+0000 7fbe337fe640 1 -- 172.21.15.33:0/392437009 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) ==== 74+0+25226 (secure 0 0 0) 0x7fbe3c0bd8d0 con 0x7fbe4410be20 2024-03-19T03:03:33.804 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T03:03:33.804 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":106,"fsid":"2b7a8938-e59a-11ee-95c9-87774f69a715","created":"2024-03-19T02:42:37.912179+0000","modified":"2024-03-19T03:03:15.236302+0000","last_up_change":"2024-03-19T02:50:01.539392+0000","last_in_change":"2024-03-19T02:49:40.534415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:46:09.963737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:50:50.069641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:50:51.381064+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":105,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:50:55.476667+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":105,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"090a89ab-6b7b-4317-8823-178d4212b2f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6803","nonce":1633124995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6805","nonce":1633124995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6809","nonce":1633124995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6807","nonce":1633124995}]},"public_addr":"172.21.15.33:6803/1633124995","cluster_addr":"172.21.15.33:6805/1633124995","heartbeat_back_addr":"172.21.15.33:6809/1633124995","heartbeat_front_addr":"172.21.15.33:6807/1633124995","state":["exists","up"]},{"osd":1,"uuid":"f50eec55-1956-4486-8593-f2c8fa610e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6801","nonce":340144398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6803","nonce":340144398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6807","nonce":340144398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6805","nonce":340144398}]},"public_addr":"172.21.15.96:6801/340144398","cluster_addr":"172.21.15.96:6803/340144398","heartbeat_back_addr":"172.21.15.96:6807/340144398","heartbeat_front_addr":"172.21.15.96:6805/340144398","state":["exists","up"]},{"osd":2,"uuid":"893a2d62-80d9-4e5e-bc6a-a1c8c681d652","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6801","nonce":3388743876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6803","nonce":3388743876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6807","nonce":3388743876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6805","nonce":3388743876}]},"public_addr":"172.21.15.155:6801/3388743876","cluster_addr":"172.21.15.155:6803/3388743876","heartbeat_back_addr":"172.21.15.155:6807/3388743876","heartbeat_front_addr":"172.21.15.155:6805/3388743876","state":["exists","up"]},{"osd":3,"uuid":"b07e0482-4281-4d32-9dfe-9995d7246175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6811","nonce":677007171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6813","nonce":677007171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6817","nonce":677007171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6815","nonce":677007171}]},"public_addr":"172.21.15.33:6811/677007171","cluster_addr":"172.21.15.33:6813/677007171","heartbeat_back_addr":"172.21.15.33:6817/677007171","heartbeat_front_addr":"172.21.15.33:6815/677007171","state":["exists","up"]},{"osd":4,"uuid":"a988384b-af6b-47c6-8c07-5c9673acee45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6809","nonce":3106158554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6811","nonce":3106158554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6815","nonce":3106158554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6813","nonce":3106158554}]},"public_addr":"172.21.15.96:6809/3106158554","cluster_addr":"172.21.15.96:6811/3106158554","heartbeat_back_addr":"172.21.15.96:6815/3106158554","heartbeat_front_addr":"172.21.15.96:6813/3106158554","state":["exists","up"]},{"osd":5,"uuid":"198dc992-96f8-4c52-96ee-1fd5de148bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6809","nonce":327293324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6811","nonce":327293324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6815","nonce":327293324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6813","nonce":327293324}]},"public_addr":"172.21.15.155:6809/327293324","cluster_addr":"172.21.15.155:6811/327293324","heartbeat_back_addr":"172.21.15.155:6815/327293324","heartbeat_front_addr":"172.21.15.155:6813/327293324","state":["exists","up"]},{"osd":6,"uuid":"a7278ec0-ab2e-49ed-be53-84d8b9e24177","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6819","nonce":873871509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6821","nonce":873871509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6825","nonce":873871509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6823","nonce":873871509}]},"public_addr":"172.21.15.33:6819/873871509","cluster_addr":"172.21.15.33:6821/873871509","heartbeat_back_addr":"172.21.15.33:6825/873871509","heartbeat_front_addr":"172.21.15.33:6823/873871509","state":["exists","up"]},{"osd":7,"uuid":"727e133d-aa24-4ffc-90e2-266eb2aab8c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6817","nonce":2334123891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6819","nonce":2334123891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6823","nonce":2334123891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6821","nonce":2334123891}]},"public_addr":"172.21.15.96:6817/2334123891","cluster_addr":"172.21.15.96:6819/2334123891","heartbeat_back_addr":"172.21.15.96:6823/2334123891","heartbeat_front_addr":"172.21.15.96:6821/2334123891","state":["exists","up"]},{"osd":8,"uuid":"bb20eec0-39e5-4b86-9db5-486b1840cdad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6817","nonce":3498516903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6819","nonce":3498516903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6823","nonce":3498516903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6821","nonce":3498516903}]},"public_addr":"172.21.15.155:6817/3498516903","cluster_addr":"172.21.15.155:6819/3498516903","heartbeat_back_addr":"172.21.15.155:6823/3498516903","heartbeat_front_addr":"172.21.15.155:6821/3498516903","state":["exists","up"]},{"osd":9,"uuid":"179c6228-3e23-4f65-91a6-1ef0005d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6827","nonce":760777213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6829","nonce":760777213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6833","nonce":760777213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6831","nonce":760777213}]},"public_addr":"172.21.15.33:6827/760777213","cluster_addr":"172.21.15.33:6829/760777213","heartbeat_back_addr":"172.21.15.33:6833/760777213","heartbeat_front_addr":"172.21.15.33:6831/760777213","state":["exists","up"]},{"osd":10,"uuid":"be17b478-3c2e-41fb-bede-958333acdf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6825","nonce":1404898482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6827","nonce":1404898482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6831","nonce":1404898482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6829","nonce":1404898482}]},"public_addr":"172.21.15.96:6825/1404898482","cluster_addr":"172.21.15.96:6827/1404898482","heartbeat_back_addr":"172.21.15.96:6831/1404898482","heartbeat_front_addr":"172.21.15.96:6829/1404898482","state":["exists","up"]},{"osd":11,"uuid":"c0903ed2-cc1e-4c18-8731-1f0ee4dd0457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6825","nonce":792352827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6827","nonce":792352827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6831","nonce":792352827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6829","nonce":792352827}]},"public_addr":"172.21.15.155:6825/792352827","cluster_addr":"172.21.15.155:6827/792352827","heartbeat_back_addr":"172.21.15.155:6831/792352827","heartbeat_front_addr":"172.21.15.155:6829/792352827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:17.942801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:42.109449+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:05.981599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:32.069574+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:57.431015+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:22.448592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:48.526515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:14.498352+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:40.175426+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:06.364006+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:32.853161+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:59.073538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.9","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:6800/1453067701":"2024-03-20T02:53:26.937131+0000","172.21.15.33:0/3554132372":"2024-03-20T02:53:26.937131+0000","172.21.15.33:0/4099467400":"2024-03-20T02:53:26.937131+0000","172.21.15.33:0/176174888":"2024-03-20T02:53:26.937131+0000","172.21.15.33:6801/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:6800/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/1067768047":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/729379453":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2586588742":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/2932814165":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/1950732854":"2024-03-20T02:53:26.937131+0000","172.21.15.33:6801/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6801/1453067701":"2024-03-20T02:53:26.937131+0000","172.21.15.33:0/2212564963":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6800/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/3275126058":"2024-03-20T02:43:39.622218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:03:33.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.804+0000 7fbe488dc640 1 -- 172.21.15.33:0/392437009 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbe2c0a2970 msgr2=0x7fbe2c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:33.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.804+0000 7fbe488dc640 1 --2- 172.21.15.33:0/392437009 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbe2c0a2970 0x7fbe2c0a4e30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbe34009bd0 tx=0x7fbe34005f90 comp rx=0 tx=0).stop 2024-03-19T03:03:33.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.804+0000 7fbe488dc640 1 -- 172.21.15.33:0/392437009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe4410be20 msgr2=0x7fbe4419c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:33.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.804+0000 7fbe488dc640 1 --2- 172.21.15.33:0/392437009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe4410be20 0x7fbe4419c920 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c02c790 tx=0x7fbe3c035d50 comp rx=0 tx=0).stop 2024-03-19T03:03:33.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.804+0000 7fbe488dc640 1 -- 172.21.15.33:0/392437009 shutdown_connections 2024-03-19T03:03:33.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.804+0000 7fbe488dc640 1 --2- 172.21.15.33:0/392437009 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbe2c0a2970 0x7fbe2c0a4e30 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:33.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.804+0000 7fbe488dc640 1 --2- 172.21.15.33:0/392437009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbe4410be20 0x7fbe4419c920 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:33.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.804+0000 7fbe488dc640 1 --2- 172.21.15.33:0/392437009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe44103370 0x7fbe441985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:33.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.804+0000 7fbe488dc640 1 --2- 172.21.15.33:0/392437009 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe44102a50 0x7fbe44198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:33.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.804+0000 7fbe488dc640 1 -- 172.21.15.33:0/392437009 >> 172.21.15.33:0/392437009 conn(0x7fbe44076aa0 msgr2=0x7fbe440fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:33.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.804+0000 7fbe488dc640 1 -- 172.21.15.33:0/392437009 shutdown_connections 2024-03-19T03:03:33.806 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.804+0000 7fbe488dc640 1 -- 172.21.15.33:0/392437009 wait complete. 2024-03-19T03:03:33.818 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:46:09.963737+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T02:50:50.069641+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T02:50:51.381064+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '105', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 105, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T02:50:55.476667+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '105', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 105, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T03:03:33.819 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T03:03:33.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.923+0000 7fbbcfc9a640 1 -- 172.21.15.33:0/3095558413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbbc8102a60 msgr2=0x7fbbc8102e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:33.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.923+0000 7fbbcfc9a640 1 --2- 172.21.15.33:0/3095558413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbbc8102a60 0x7fbbc8102e40 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4009900 tx=0x7fbbc402bcd0 comp rx=0 tx=0).stop 2024-03-19T03:03:33.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.923+0000 7fbbcfc9a640 1 -- 172.21.15.33:0/3095558413 shutdown_connections 2024-03-19T03:03:33.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.923+0000 7fbbcfc9a640 1 --2- 172.21.15.33:0/3095558413 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbbc810be30 0x7fbbc810e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:33.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.923+0000 7fbbcfc9a640 1 --2- 172.21.15.33:0/3095558413 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbbc8103380 0x7fbbc810b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:33.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.923+0000 7fbbcfc9a640 1 --2- 172.21.15.33:0/3095558413 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbbc8102a60 0x7fbbc8102e40 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:33.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.923+0000 7fbbcfc9a640 1 -- 172.21.15.33:0/3095558413 >> 172.21.15.33:0/3095558413 conn(0x7fbbc8076aa0 msgr2=0x7fbbc8076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:33.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.923+0000 7fbbcfc9a640 1 -- 172.21.15.33:0/3095558413 shutdown_connections 2024-03-19T03:03:33.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.923+0000 7fbbcfc9a640 1 -- 172.21.15.33:0/3095558413 wait complete. 2024-03-19T03:03:33.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.924+0000 7fbbcfc9a640 1 Processor -- start 2024-03-19T03:03:33.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.924+0000 7fbbcfc9a640 1 -- start start 2024-03-19T03:03:33.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.924+0000 7fbbcfc9a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbbc8102a60 0x7fbbc8197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:33.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.924+0000 7fbbcfc9a640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbbc8103380 0x7fbbc81984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:33.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.924+0000 7fbbcfc9a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbbc810be30 0x7fbbc819c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:33.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.924+0000 7fbbcfc9a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc8111a70 con 0x7fbbc8102a60 2024-03-19T03:03:33.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.924+0000 7fbbcfc9a640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc81118f0 con 0x7fbbc810be30 2024-03-19T03:03:33.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.924+0000 7fbbcfc9a640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc8111bf0 con 0x7fbbc8103380 2024-03-19T03:03:33.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.924+0000 7fbbcda36640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbbc8102a60 0x7fbbc8197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:33.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.924+0000 7fbbcda36640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbbc8102a60 0x7fbbc8197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55294/0 (socket says 172.21.15.33:55294) 2024-03-19T03:03:33.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.924+0000 7fbbcda36640 1 -- 172.21.15.33:0/4163381487 learned_addr learned my addr 172.21.15.33:0/4163381487 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:33.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.924+0000 7fbbcd235640 1 --2- 172.21.15.33:0/4163381487 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbbc8103380 0x7fbbc81984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:33.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.924+0000 7fbbce237640 1 --2- 172.21.15.33:0/4163381487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbbc810be30 0x7fbbc819c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:33.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.924+0000 7fbbcda36640 1 -- 172.21.15.33:0/4163381487 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbbc8103380 msgr2=0x7fbbc81984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:33.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.924+0000 7fbbcda36640 1 --2- 172.21.15.33:0/4163381487 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbbc8103380 0x7fbbc81984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:33.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.924+0000 7fbbcda36640 1 -- 172.21.15.33:0/4163381487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbbc810be30 msgr2=0x7fbbc819c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:33.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.924+0000 7fbbcda36640 1 --2- 172.21.15.33:0/4163381487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbbc810be30 0x7fbbc819c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:33.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.924+0000 7fbbcda36640 1 -- 172.21.15.33:0/4163381487 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbc819cf80 con 0x7fbbc8102a60 2024-03-19T03:03:33.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.924+0000 7fbbce237640 1 --2- 172.21.15.33:0/4163381487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbbc810be30 0x7fbbc819c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:03:33.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.925+0000 7fbbcda36640 1 --2- 172.21.15.33:0/4163381487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbbc8102a60 0x7fbbc8197f90 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4005940 tx=0x7fbbc40045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:33.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.925+0000 7fbbbeffd640 1 -- 172.21.15.33:0/4163381487 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc4043020 con 0x7fbbc8102a60 2024-03-19T03:03:33.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.925+0000 7fbbbeffd640 1 -- 172.21.15.33:0/4163381487 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbbc402dba0 con 0x7fbbc8102a60 2024-03-19T03:03:33.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.925+0000 7fbbcfc9a640 1 -- 172.21.15.33:0/4163381487 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbc819d210 con 0x7fbbc8102a60 2024-03-19T03:03:33.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.925+0000 7fbbbeffd640 1 -- 172.21.15.33:0/4163381487 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc4003a10 con 0x7fbbc8102a60 2024-03-19T03:03:33.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.925+0000 7fbbcfc9a640 1 -- 172.21.15.33:0/4163381487 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbc81a4b00 con 0x7fbbc8102a60 2024-03-19T03:03:33.930 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.926+0000 7fbbcfc9a640 1 -- 172.21.15.33:0/4163381487 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbc806a630 con 0x7fbbc8102a60 2024-03-19T03:03:33.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.927+0000 7fbbbeffd640 1 -- 172.21.15.33:0/4163381487 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fbbc4046020 con 0x7fbbc8102a60 2024-03-19T03:03:33.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.927+0000 7fbbbeffd640 1 --2- 172.21.15.33:0/4163381487 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbbb40a29f0 0x7fbbb40a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:33.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.928+0000 7fbbcd235640 1 --2- 172.21.15.33:0/4163381487 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbbb40a29f0 0x7fbbb40a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:33.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.928+0000 7fbbbeffd640 1 -- 172.21.15.33:0/4163381487 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7fbbc40f1750 con 0x7fbbc8102a60 2024-03-19T03:03:33.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.928+0000 7fbbcd235640 1 --2- 172.21.15.33:0/4163381487 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbbb40a29f0 0x7fbbb40a4eb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbbc80fff20 tx=0x7fbbb8005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:33.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:33.931+0000 7fbbbeffd640 1 -- 172.21.15.33:0/4163381487 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbbc404d020 con 0x7fbbc8102a60 2024-03-19T03:03:34.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.113+0000 7fbbcfc9a640 1 -- 172.21.15.33:0/4163381487 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbbc8106f20 con 0x7fbbc8102a60 2024-03-19T03:03:34.115 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.114+0000 7fbbbeffd640 1 -- 172.21.15.33:0/4163381487 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) ==== 93+0+10 (secure 0 0 0) 0x7fbbc40be1f0 con 0x7fbbc8102a60 2024-03-19T03:03:34.115 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 1 2024-03-19T03:03:34.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.116+0000 7fbbcfc9a640 1 -- 172.21.15.33:0/4163381487 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbbb40a29f0 msgr2=0x7fbbb40a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:34.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.116+0000 7fbbcfc9a640 1 --2- 172.21.15.33:0/4163381487 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbbb40a29f0 0x7fbbb40a4eb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbbc80fff20 tx=0x7fbbb8005f90 comp rx=0 tx=0).stop 2024-03-19T03:03:34.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.116+0000 7fbbcfc9a640 1 -- 172.21.15.33:0/4163381487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbbc8102a60 msgr2=0x7fbbc8197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:34.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.116+0000 7fbbcfc9a640 1 --2- 172.21.15.33:0/4163381487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbbc8102a60 0x7fbbc8197f90 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4005940 tx=0x7fbbc40045e0 comp rx=0 tx=0).stop 2024-03-19T03:03:34.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.116+0000 7fbbcfc9a640 1 -- 172.21.15.33:0/4163381487 shutdown_connections 2024-03-19T03:03:34.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.116+0000 7fbbcfc9a640 1 --2- 172.21.15.33:0/4163381487 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fbbb40a29f0 0x7fbbb40a4eb0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:34.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.116+0000 7fbbcfc9a640 1 --2- 172.21.15.33:0/4163381487 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbbc8103380 0x7fbbc81984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:34.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.116+0000 7fbbcfc9a640 1 --2- 172.21.15.33:0/4163381487 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbbc810be30 0x7fbbc819c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:34.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.116+0000 7fbbcfc9a640 1 --2- 172.21.15.33:0/4163381487 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fbbc8102a60 0x7fbbc8197f90 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:34.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.116+0000 7fbbcfc9a640 1 -- 172.21.15.33:0/4163381487 >> 172.21.15.33:0/4163381487 conn(0x7fbbc8076aa0 msgr2=0x7fbbc80fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:34.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.116+0000 7fbbcfc9a640 1 -- 172.21.15.33:0/4163381487 shutdown_connections 2024-03-19T03:03:34.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.116+0000 7fbbcfc9a640 1 -- 172.21.15.33:0/4163381487 wait complete. 2024-03-19T03:03:34.128 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T03:03:34.236 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.233+0000 7f7531820640 1 -- 172.21.15.33:0/3296438209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f752c1017b0 msgr2=0x7f752c101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:34.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.233+0000 7f7531820640 1 --2- 172.21.15.33:0/3296438209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f752c1017b0 0x7f752c101c30 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f7520009930 tx=0x7f752002bc30 comp rx=0 tx=0).stop 2024-03-19T03:03:34.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.233+0000 7f7531820640 1 -- 172.21.15.33:0/3296438209 shutdown_connections 2024-03-19T03:03:34.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.233+0000 7f7531820640 1 --2- 172.21.15.33:0/3296438209 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f752c102170 0x7f752c10e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:34.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.233+0000 7f7531820640 1 --2- 172.21.15.33:0/3296438209 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f752c1017b0 0x7f752c101c30 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:34.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.233+0000 7f7531820640 1 --2- 172.21.15.33:0/3296438209 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f752c1077c0 0x7f752c107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:34.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.233+0000 7f7531820640 1 -- 172.21.15.33:0/3296438209 >> 172.21.15.33:0/3296438209 conn(0x7f752c078c20 msgr2=0x7f752c079030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:34.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.233+0000 7f7531820640 1 -- 172.21.15.33:0/3296438209 shutdown_connections 2024-03-19T03:03:34.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.233+0000 7f7531820640 1 -- 172.21.15.33:0/3296438209 wait complete. 2024-03-19T03:03:34.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.234+0000 7f7531820640 1 Processor -- start 2024-03-19T03:03:34.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.234+0000 7f7531820640 1 -- start start 2024-03-19T03:03:34.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.234+0000 7f7531820640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f752c1017b0 0x7f752c073b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:34.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.234+0000 7f7531820640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f752c102170 0x7f752c074060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:34.237 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.234+0000 7f7531820640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f752c1077c0 0x7f752c0745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:34.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.234+0000 7f7531820640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f752c111a80 con 0x7f752c1077c0 2024-03-19T03:03:34.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.234+0000 7f7531820640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f752c111900 con 0x7f752c1017b0 2024-03-19T03:03:34.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.234+0000 7f7531820640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f752c111c00 con 0x7f752c102170 2024-03-19T03:03:34.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.234+0000 7f752affd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f752c1017b0 0x7f752c073b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:34.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.234+0000 7f752b7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f752c1077c0 0x7f752c0745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:34.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.234+0000 7f752affd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f752c1017b0 0x7f752c073b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:34034/0 (socket says 172.21.15.33:34034) 2024-03-19T03:03:34.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.234+0000 7f752affd640 1 -- 172.21.15.33:0/3641712955 learned_addr learned my addr 172.21.15.33:0/3641712955 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:34.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.234+0000 7f752a7fc640 1 --2- 172.21.15.33:0/3641712955 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f752c102170 0x7f752c074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:34.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.235+0000 7f752affd640 1 -- 172.21.15.33:0/3641712955 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f752c102170 msgr2=0x7f752c074060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:34.238 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.235+0000 7f752affd640 1 --2- 172.21.15.33:0/3641712955 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f752c102170 0x7f752c074060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:34.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.235+0000 7f752affd640 1 -- 172.21.15.33:0/3641712955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f752c1077c0 msgr2=0x7f752c0745a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:34.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.235+0000 7f752affd640 1 --2- 172.21.15.33:0/3641712955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f752c1077c0 0x7f752c0745a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:34.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.235+0000 7f752affd640 1 -- 172.21.15.33:0/3641712955 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f752c1a6500 con 0x7f752c1017b0 2024-03-19T03:03:34.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.235+0000 7f752b7fe640 1 --2- 172.21.15.33:0/3641712955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f752c1077c0 0x7f752c0745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:34.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.235+0000 7f752affd640 1 --2- 172.21.15.33:0/3641712955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f752c1017b0 0x7f752c073b20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f752400e9f0 tx=0x7f752400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:34.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.235+0000 7f753081e640 1 -- 172.21.15.33:0/3641712955 <== mon.2 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f752400cd80 con 0x7f752c1017b0 2024-03-19T03:03:34.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.235+0000 7f753081e640 1 -- 172.21.15.33:0/3641712955 <== mon.2 v2:172.21.15.96:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f75240054d0 con 0x7f752c1017b0 2024-03-19T03:03:34.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.235+0000 7f7531820640 1 -- 172.21.15.33:0/3641712955 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f752c1a67f0 con 0x7f752c1017b0 2024-03-19T03:03:34.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.235+0000 7f753081e640 1 -- 172.21.15.33:0/3641712955 <== mon.2 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7524010490 con 0x7f752c1017b0 2024-03-19T03:03:34.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.235+0000 7f7531820640 1 -- 172.21.15.33:0/3641712955 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f752c1a6cc0 con 0x7f752c1017b0 2024-03-19T03:03:34.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.236+0000 7f7531820640 1 -- 172.21.15.33:0/3641712955 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f752c068a60 con 0x7f752c1017b0 2024-03-19T03:03:34.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.237+0000 7f753081e640 1 -- 172.21.15.33:0/3641712955 <== mon.2 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f7524005020 con 0x7f752c1017b0 2024-03-19T03:03:34.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.238+0000 7f753081e640 1 --2- 172.21.15.33:0/3641712955 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f75140a2970 0x7f75140a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:34.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.238+0000 7f752a7fc640 1 --2- 172.21.15.33:0/3641712955 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f75140a2970 0x7f75140a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:34.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.238+0000 7f753081e640 1 -- 172.21.15.33:0/3641712955 <== mon.2 v2:172.21.15.96:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7f75240cf660 con 0x7f752c1017b0 2024-03-19T03:03:34.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.238+0000 7f752a7fc640 1 --2- 172.21.15.33:0/3641712955 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f75140a2970 0x7f75140a4e30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7520006010 tx=0x7f752002c670 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:34.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.241+0000 7f753081e640 1 -- 172.21.15.33:0/3641712955 <== mon.2 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f752409bff0 con 0x7f752c1017b0 2024-03-19T03:03:34.426 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.425+0000 7f7531820640 1 -- 172.21.15.33:0/3641712955 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f752c109d40 con 0x7f752c1017b0 2024-03-19T03:03:34.427 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.426+0000 7f753081e640 1 -- 172.21.15.33:0/3641712955 <== mon.2 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v106) ==== 104+0+11 (secure 0 0 0) 0x7f752409b8f0 con 0x7f752c1017b0 2024-03-19T03:03:34.427 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 32 2024-03-19T03:03:34.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.428+0000 7f7531820640 1 -- 172.21.15.33:0/3641712955 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f75140a2970 msgr2=0x7f75140a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:34.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.428+0000 7f7531820640 1 --2- 172.21.15.33:0/3641712955 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f75140a2970 0x7f75140a4e30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7520006010 tx=0x7f752002c670 comp rx=0 tx=0).stop 2024-03-19T03:03:34.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.428+0000 7f7531820640 1 -- 172.21.15.33:0/3641712955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f752c1017b0 msgr2=0x7f752c073b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:34.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.428+0000 7f7531820640 1 --2- 172.21.15.33:0/3641712955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f752c1017b0 0x7f752c073b20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f752400e9f0 tx=0x7f752400edf0 comp rx=0 tx=0).stop 2024-03-19T03:03:34.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.428+0000 7f7531820640 1 -- 172.21.15.33:0/3641712955 shutdown_connections 2024-03-19T03:03:34.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.428+0000 7f7531820640 1 --2- 172.21.15.33:0/3641712955 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f75140a2970 0x7f75140a4e30 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:34.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.428+0000 7f7531820640 1 --2- 172.21.15.33:0/3641712955 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f752c102170 0x7f752c074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:34.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.428+0000 7f7531820640 1 --2- 172.21.15.33:0/3641712955 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f752c1077c0 0x7f752c0745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:34.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.428+0000 7f7531820640 1 --2- 172.21.15.33:0/3641712955 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f752c1017b0 0x7f752c073b20 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:34.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.428+0000 7f7531820640 1 -- 172.21.15.33:0/3641712955 >> 172.21.15.33:0/3641712955 conn(0x7f752c078c20 msgr2=0x7f752c104260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:34.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.428+0000 7f7531820640 1 -- 172.21.15.33:0/3641712955 shutdown_connections 2024-03-19T03:03:34.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.428+0000 7f7531820640 1 -- 172.21.15.33:0/3641712955 wait complete. 2024-03-19T03:03:34.439 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:34 smithi033 ceph-mon[31973]: pgmap v314: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-19T03:03:34.439 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:34 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/392437009' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:03:34.439 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:34 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/4163381487' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:03:34.440 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T03:03:34.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:34 smithi155 ceph-mon[35804]: pgmap v314: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-19T03:03:34.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:34 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/392437009' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:03:34.517 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:34 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/4163381487' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:03:34.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.544+0000 7fa430937640 1 -- 172.21.15.33:0/2942320254 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa42c106750 msgr2=0x7fa42c106b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:34.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.544+0000 7fa430937640 1 --2- 172.21.15.33:0/2942320254 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa42c106750 0x7fa42c106b30 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fa41c009930 tx=0x7fa41c02bc30 comp rx=0 tx=0).stop 2024-03-19T03:03:34.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.544+0000 7fa430937640 1 -- 172.21.15.33:0/2942320254 shutdown_connections 2024-03-19T03:03:34.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.544+0000 7fa430937640 1 --2- 172.21.15.33:0/2942320254 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa42c100740 0x7fa42c100bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:34.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.544+0000 7fa430937640 1 --2- 172.21.15.33:0/2942320254 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa42c101100 0x7fa42c10d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:34.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.544+0000 7fa430937640 1 --2- 172.21.15.33:0/2942320254 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa42c106750 0x7fa42c106b30 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:34.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.544+0000 7fa430937640 1 -- 172.21.15.33:0/2942320254 >> 172.21.15.33:0/2942320254 conn(0x7fa42c077880 msgr2=0x7fa42c077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:34.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.544+0000 7fa430937640 1 -- 172.21.15.33:0/2942320254 shutdown_connections 2024-03-19T03:03:34.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.544+0000 7fa430937640 1 -- 172.21.15.33:0/2942320254 wait complete. 2024-03-19T03:03:34.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.545+0000 7fa430937640 1 Processor -- start 2024-03-19T03:03:34.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.545+0000 7fa430937640 1 -- start start 2024-03-19T03:03:34.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.545+0000 7fa430937640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa42c100740 0x7fa42c198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:34.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.545+0000 7fa430937640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa42c101100 0x7fa42c198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:34.548 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.545+0000 7fa430937640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa42c106750 0x7fa42c19c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:34.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.545+0000 7fa430937640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c111b10 con 0x7fa42c100740 2024-03-19T03:03:34.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.545+0000 7fa430937640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c111990 con 0x7fa42c106750 2024-03-19T03:03:34.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.545+0000 7fa430937640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c111c90 con 0x7fa42c101100 2024-03-19T03:03:34.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.545+0000 7fa42a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa42c100740 0x7fa42c198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:34.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.545+0000 7fa42a59c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa42c100740 0x7fa42c198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55326/0 (socket says 172.21.15.33:55326) 2024-03-19T03:03:34.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.545+0000 7fa42a59c640 1 -- 172.21.15.33:0/1030361426 learned_addr learned my addr 172.21.15.33:0/1030361426 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:34.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.545+0000 7fa42ad9d640 1 --2- 172.21.15.33:0/1030361426 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa42c106750 0x7fa42c19c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:34.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.545+0000 7fa423fff640 1 --2- 172.21.15.33:0/1030361426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa42c101100 0x7fa42c198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:34.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.545+0000 7fa42a59c640 1 -- 172.21.15.33:0/1030361426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa42c101100 msgr2=0x7fa42c198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:34.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.545+0000 7fa42a59c640 1 --2- 172.21.15.33:0/1030361426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa42c101100 0x7fa42c198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:34.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.545+0000 7fa42a59c640 1 -- 172.21.15.33:0/1030361426 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa42c106750 msgr2=0x7fa42c19c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:34.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.545+0000 7fa42a59c640 1 --2- 172.21.15.33:0/1030361426 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa42c106750 0x7fa42c19c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:34.549 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.545+0000 7fa42a59c640 1 -- 172.21.15.33:0/1030361426 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa42c19d040 con 0x7fa42c100740 2024-03-19T03:03:34.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.545+0000 7fa423fff640 1 --2- 172.21.15.33:0/1030361426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa42c101100 0x7fa42c198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:03:34.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.545+0000 7fa42a59c640 1 --2- 172.21.15.33:0/1030361426 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa42c100740 0x7fa42c198050 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fa41c009230 tx=0x7fa41c02c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:34.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.546+0000 7fa4237fe640 1 -- 172.21.15.33:0/1030361426 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41c043020 con 0x7fa42c100740 2024-03-19T03:03:34.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.546+0000 7fa4237fe640 1 -- 172.21.15.33:0/1030361426 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa41c02ca70 con 0x7fa42c100740 2024-03-19T03:03:34.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.546+0000 7fa4237fe640 1 -- 172.21.15.33:0/1030361426 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41c003970 con 0x7fa42c100740 2024-03-19T03:03:34.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.546+0000 7fa430937640 1 -- 172.21.15.33:0/1030361426 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa42c19d2d0 con 0x7fa42c100740 2024-03-19T03:03:34.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.546+0000 7fa430937640 1 -- 172.21.15.33:0/1030361426 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa42c1a4bc0 con 0x7fa42c100740 2024-03-19T03:03:34.550 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.547+0000 7fa430937640 1 -- 172.21.15.33:0/1030361426 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa42c101e80 con 0x7fa42c100740 2024-03-19T03:03:34.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.548+0000 7fa4237fe640 1 -- 172.21.15.33:0/1030361426 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fa41c046020 con 0x7fa42c100740 2024-03-19T03:03:34.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.548+0000 7fa4237fe640 1 --2- 172.21.15.33:0/1030361426 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa4180a2970 0x7fa4180a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:34.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.548+0000 7fa423fff640 1 --2- 172.21.15.33:0/1030361426 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa4180a2970 0x7fa4180a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:34.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.548+0000 7fa4237fe640 1 -- 172.21.15.33:0/1030361426 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7fa41c0f1220 con 0x7fa42c100740 2024-03-19T03:03:34.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.549+0000 7fa423fff640 1 --2- 172.21.15.33:0/1030361426 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa4180a2970 0x7fa4180a4e30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa42c077ed0 tx=0x7fa414005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:34.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.553+0000 7fa4237fe640 1 -- 172.21.15.33:0/1030361426 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa41c04d020 con 0x7fa42c100740 2024-03-19T03:03:34.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:34 smithi096 ceph-mon[35946]: pgmap v314: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-19T03:03:34.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:34 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/392437009' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:03:34.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:34 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/4163381487' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:03:34.736 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.735+0000 7fa430937640 1 -- 172.21.15.33:0/1030361426 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa42c108cb0 con 0x7fa42c100740 2024-03-19T03:03:34.737 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.736+0000 7fa4237fe640 1 -- 172.21.15.33:0/1030361426 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v106) ==== 100+0+11 (secure 0 0 0) 0x7fa41c03c350 con 0x7fa42c100740 2024-03-19T03:03:34.737 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 64 2024-03-19T03:03:34.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.737+0000 7fa430937640 1 -- 172.21.15.33:0/1030361426 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa4180a2970 msgr2=0x7fa4180a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:34.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.737+0000 7fa430937640 1 --2- 172.21.15.33:0/1030361426 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa4180a2970 0x7fa4180a4e30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa42c077ed0 tx=0x7fa414005f90 comp rx=0 tx=0).stop 2024-03-19T03:03:34.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.737+0000 7fa430937640 1 -- 172.21.15.33:0/1030361426 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa42c100740 msgr2=0x7fa42c198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:34.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.737+0000 7fa430937640 1 --2- 172.21.15.33:0/1030361426 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa42c100740 0x7fa42c198050 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fa41c009230 tx=0x7fa41c02c5d0 comp rx=0 tx=0).stop 2024-03-19T03:03:34.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.738+0000 7fa430937640 1 -- 172.21.15.33:0/1030361426 shutdown_connections 2024-03-19T03:03:34.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.738+0000 7fa430937640 1 --2- 172.21.15.33:0/1030361426 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa4180a2970 0x7fa4180a4e30 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:34.739 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.738+0000 7fa430937640 1 --2- 172.21.15.33:0/1030361426 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa42c101100 0x7fa42c198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:34.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.738+0000 7fa430937640 1 --2- 172.21.15.33:0/1030361426 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa42c106750 0x7fa42c19c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:34.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.738+0000 7fa430937640 1 --2- 172.21.15.33:0/1030361426 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa42c100740 0x7fa42c198050 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:34.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.738+0000 7fa430937640 1 -- 172.21.15.33:0/1030361426 >> 172.21.15.33:0/1030361426 conn(0x7fa42c077880 msgr2=0x7fa42c103350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:34.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.738+0000 7fa430937640 1 -- 172.21.15.33:0/1030361426 shutdown_connections 2024-03-19T03:03:34.740 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.738+0000 7fa430937640 1 -- 172.21.15.33:0/1030361426 wait complete. 2024-03-19T03:03:34.750 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T03:03:34.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.855+0000 7f2ad66ea640 1 -- 172.21.15.33:0/1118741993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ad0102170 msgr2=0x7f2ad010e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:34.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.855+0000 7f2ad66ea640 1 --2- 172.21.15.33:0/1118741993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ad0102170 0x7f2ad010e680 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f2ac8009cd0 tx=0x7f2ac802c0f0 comp rx=0 tx=0).stop 2024-03-19T03:03:34.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.855+0000 7f2ad66ea640 1 -- 172.21.15.33:0/1118741993 shutdown_connections 2024-03-19T03:03:34.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.855+0000 7f2ad66ea640 1 --2- 172.21.15.33:0/1118741993 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2ad01017b0 0x7f2ad0101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:34.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.855+0000 7f2ad66ea640 1 --2- 172.21.15.33:0/1118741993 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ad0102170 0x7f2ad010e680 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:34.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.855+0000 7f2ad66ea640 1 --2- 172.21.15.33:0/1118741993 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ad01077c0 0x7f2ad0107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:34.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.855+0000 7f2ad66ea640 1 -- 172.21.15.33:0/1118741993 >> 172.21.15.33:0/1118741993 conn(0x7f2ad0078c20 msgr2=0x7f2ad0079030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:34.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.855+0000 7f2ad66ea640 1 -- 172.21.15.33:0/1118741993 shutdown_connections 2024-03-19T03:03:34.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.855+0000 7f2ad66ea640 1 -- 172.21.15.33:0/1118741993 wait complete. 2024-03-19T03:03:34.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.855+0000 7f2ad66ea640 1 Processor -- start 2024-03-19T03:03:34.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.855+0000 7f2ad66ea640 1 -- start start 2024-03-19T03:03:34.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.856+0000 7f2ad66ea640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ad01017b0 0x7f2ad0197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:34.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.856+0000 7f2ad66ea640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ad0102170 0x7f2ad0198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:34.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.856+0000 7f2ad66ea640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2ad01077c0 0x7f2ad019c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:34.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.856+0000 7f2ad66ea640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad0111a80 con 0x7f2ad0102170 2024-03-19T03:03:34.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.856+0000 7f2ad66ea640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad0111900 con 0x7f2ad01017b0 2024-03-19T03:03:34.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.856+0000 7f2ad66ea640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad0111c00 con 0x7f2ad01077c0 2024-03-19T03:03:34.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.856+0000 7f2acf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ad0102170 0x7f2ad0198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:34.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.856+0000 7f2acf7fe640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ad0102170 0x7f2ad0198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55352/0 (socket says 172.21.15.33:55352) 2024-03-19T03:03:34.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.856+0000 7f2acf7fe640 1 -- 172.21.15.33:0/1636855123 learned_addr learned my addr 172.21.15.33:0/1636855123 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:34.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.856+0000 7f2ad4c87640 1 --2- 172.21.15.33:0/1636855123 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2ad01077c0 0x7f2ad019c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:34.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.856+0000 7f2acffff640 1 --2- 172.21.15.33:0/1636855123 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ad01017b0 0x7f2ad0197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:34.859 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.856+0000 7f2acf7fe640 1 -- 172.21.15.33:0/1636855123 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2ad01077c0 msgr2=0x7f2ad019c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:34.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.856+0000 7f2acf7fe640 1 --2- 172.21.15.33:0/1636855123 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2ad01077c0 0x7f2ad019c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:34.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.856+0000 7f2acf7fe640 1 -- 172.21.15.33:0/1636855123 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ad01017b0 msgr2=0x7f2ad0197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:34.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.856+0000 7f2acf7fe640 1 --2- 172.21.15.33:0/1636855123 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ad01017b0 0x7f2ad0197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:34.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.856+0000 7f2acf7fe640 1 -- 172.21.15.33:0/1636855123 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ad019cfb0 con 0x7f2ad0102170 2024-03-19T03:03:34.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.856+0000 7f2acf7fe640 1 --2- 172.21.15.33:0/1636855123 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ad0102170 0x7f2ad0198500 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f2ac000e9f0 tx=0x7f2ac000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:34.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.856+0000 7f2acd7fa640 1 -- 172.21.15.33:0/1636855123 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac000cd80 con 0x7f2ad0102170 2024-03-19T03:03:34.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.856+0000 7f2acd7fa640 1 -- 172.21.15.33:0/1636855123 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2ac00054d0 con 0x7f2ad0102170 2024-03-19T03:03:34.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.856+0000 7f2acd7fa640 1 -- 172.21.15.33:0/1636855123 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0010490 con 0x7f2ad0102170 2024-03-19T03:03:34.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.856+0000 7f2ad66ea640 1 -- 172.21.15.33:0/1636855123 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ad019d2a0 con 0x7f2ad0102170 2024-03-19T03:03:34.860 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.856+0000 7f2ad66ea640 1 -- 172.21.15.33:0/1636855123 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ad01a4b30 con 0x7f2ad0102170 2024-03-19T03:03:34.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.857+0000 7f2ad66ea640 1 -- 172.21.15.33:0/1636855123 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ad0068a60 con 0x7f2ad0102170 2024-03-19T03:03:34.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.858+0000 7f2acd7fa640 1 -- 172.21.15.33:0/1636855123 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f2ac0005020 con 0x7f2ad0102170 2024-03-19T03:03:34.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.859+0000 7f2acd7fa640 1 --2- 172.21.15.33:0/1636855123 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f2abc0a2970 0x7f2abc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:34.861 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.859+0000 7f2acffff640 1 --2- 172.21.15.33:0/1636855123 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f2abc0a2970 0x7f2abc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:34.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.859+0000 7f2acd7fa640 1 -- 172.21.15.33:0/1636855123 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7f2ac00cf5a0 con 0x7f2ad0102170 2024-03-19T03:03:34.862 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.859+0000 7f2acffff640 1 --2- 172.21.15.33:0/1636855123 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f2abc0a2970 0x7f2abc0a4e30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8009b50 tx=0x7f2ab8005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:34.864 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:34.863+0000 7f2acd7fa640 1 -- 172.21.15.33:0/1636855123 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2ac009bf30 con 0x7f2ad0102170 2024-03-19T03:03:35.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.046+0000 7f2ad66ea640 1 -- 172.21.15.33:0/1636855123 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f2ad002d1f0 con 0x7f2ad0102170 2024-03-19T03:03:35.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.046+0000 7f2acd7fa640 1 -- 172.21.15.33:0/1636855123 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v106) ==== 103+0+11 (secure 0 0 0) 0x7f2ac009b830 con 0x7f2ad0102170 2024-03-19T03:03:35.048 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 32 2024-03-19T03:03:35.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.048+0000 7f2ad66ea640 1 -- 172.21.15.33:0/1636855123 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f2abc0a2970 msgr2=0x7f2abc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:35.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.048+0000 7f2ad66ea640 1 --2- 172.21.15.33:0/1636855123 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f2abc0a2970 0x7f2abc0a4e30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8009b50 tx=0x7f2ab8005f90 comp rx=0 tx=0).stop 2024-03-19T03:03:35.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.048+0000 7f2ad66ea640 1 -- 172.21.15.33:0/1636855123 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ad0102170 msgr2=0x7f2ad0198500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:35.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.048+0000 7f2ad66ea640 1 --2- 172.21.15.33:0/1636855123 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ad0102170 0x7f2ad0198500 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f2ac000e9f0 tx=0x7f2ac000edf0 comp rx=0 tx=0).stop 2024-03-19T03:03:35.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.048+0000 7f2ad66ea640 1 -- 172.21.15.33:0/1636855123 shutdown_connections 2024-03-19T03:03:35.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.048+0000 7f2ad66ea640 1 --2- 172.21.15.33:0/1636855123 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f2abc0a2970 0x7f2abc0a4e30 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:35.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.048+0000 7f2ad66ea640 1 --2- 172.21.15.33:0/1636855123 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2ad01077c0 0x7f2ad019c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:35.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.048+0000 7f2ad66ea640 1 --2- 172.21.15.33:0/1636855123 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2ad0102170 0x7f2ad0198500 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:35.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.048+0000 7f2ad66ea640 1 --2- 172.21.15.33:0/1636855123 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ad01017b0 0x7f2ad0197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:35.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.048+0000 7f2ad66ea640 1 -- 172.21.15.33:0/1636855123 >> 172.21.15.33:0/1636855123 conn(0x7f2ad0078c20 msgr2=0x7f2ad01041c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:35.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.048+0000 7f2ad66ea640 1 -- 172.21.15.33:0/1636855123 shutdown_connections 2024-03-19T03:03:35.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.048+0000 7f2ad66ea640 1 -- 172.21.15.33:0/1636855123 wait complete. 2024-03-19T03:03:35.060 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-19T03:03:35.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.161+0000 7f81c07bc640 1 -- 172.21.15.33:0/3938358958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f81b81066e0 msgr2=0x7f81b8106ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:35.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.161+0000 7f81c07bc640 1 --2- 172.21.15.33:0/3938358958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f81b81066e0 0x7f81b8106ac0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f81b4009930 tx=0x7f81b402bc30 comp rx=0 tx=0).stop 2024-03-19T03:03:35.165 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.161+0000 7f81c07bc640 1 -- 172.21.15.33:0/3938358958 shutdown_connections 2024-03-19T03:03:35.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.161+0000 7f81c07bc640 1 --2- 172.21.15.33:0/3938358958 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81b81006d0 0x7f81b8100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:35.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.161+0000 7f81c07bc640 1 --2- 172.21.15.33:0/3938358958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81b8101090 0x7f81b810d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:35.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.161+0000 7f81c07bc640 1 --2- 172.21.15.33:0/3938358958 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f81b81066e0 0x7f81b8106ac0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:35.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.161+0000 7f81c07bc640 1 -- 172.21.15.33:0/3938358958 >> 172.21.15.33:0/3938358958 conn(0x7f81b8076aa0 msgr2=0x7f81b8076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:35.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.162+0000 7f81c07bc640 1 -- 172.21.15.33:0/3938358958 shutdown_connections 2024-03-19T03:03:35.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.162+0000 7f81c07bc640 1 -- 172.21.15.33:0/3938358958 wait complete. 2024-03-19T03:03:35.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.162+0000 7f81c07bc640 1 Processor -- start 2024-03-19T03:03:35.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.162+0000 7f81c07bc640 1 -- start start 2024-03-19T03:03:35.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.162+0000 7f81c07bc640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81b81006d0 0x7f81b8198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:35.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.162+0000 7f81c07bc640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81b8101090 0x7f81b8198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:35.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.162+0000 7f81c07bc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f81b81066e0 0x7f81b819c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:35.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.162+0000 7f81c07bc640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f81b8110960 con 0x7f81b81066e0 2024-03-19T03:03:35.166 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.162+0000 7f81c07bc640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f81b81107e0 con 0x7f81b81006d0 2024-03-19T03:03:35.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.162+0000 7f81c07bc640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f81b8110ae0 con 0x7f81b8101090 2024-03-19T03:03:35.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.162+0000 7f81bed59640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f81b81066e0 0x7f81b819c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:35.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.162+0000 7f81bed59640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f81b81066e0 0x7f81b819c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:55372/0 (socket says 172.21.15.33:55372) 2024-03-19T03:03:35.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.163+0000 7f81bed59640 1 -- 172.21.15.33:0/621911649 learned_addr learned my addr 172.21.15.33:0/621911649 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:03:35.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.162+0000 7f81be558640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81b81006d0 0x7f81b8198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:35.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.163+0000 7f81bdd57640 1 --2- 172.21.15.33:0/621911649 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81b8101090 0x7f81b8198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:35.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.163+0000 7f81bed59640 1 -- 172.21.15.33:0/621911649 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81b8101090 msgr2=0x7f81b8198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:35.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.163+0000 7f81bed59640 1 --2- 172.21.15.33:0/621911649 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81b8101090 0x7f81b8198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:35.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.163+0000 7f81bed59640 1 -- 172.21.15.33:0/621911649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81b81006d0 msgr2=0x7f81b8198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:35.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.163+0000 7f81bed59640 1 --2- 172.21.15.33:0/621911649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81b81006d0 0x7f81b8198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:35.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.163+0000 7f81bed59640 1 -- 172.21.15.33:0/621911649 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81b819cfd0 con 0x7f81b81066e0 2024-03-19T03:03:35.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.163+0000 7f81be558640 1 --2- 172.21.15.33:0/621911649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81b81006d0 0x7f81b8198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:03:35.167 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.163+0000 7f81bed59640 1 --2- 172.21.15.33:0/621911649 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f81b81066e0 0x7f81b819c8a0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f81b000efd0 tx=0x7f81b000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:35.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.163+0000 7f81af7fe640 1 -- 172.21.15.33:0/621911649 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81b0003b70 con 0x7f81b81066e0 2024-03-19T03:03:35.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.163+0000 7f81af7fe640 1 -- 172.21.15.33:0/621911649 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f81b0003d10 con 0x7f81b81066e0 2024-03-19T03:03:35.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.163+0000 7f81c07bc640 1 -- 172.21.15.33:0/621911649 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81b819d2c0 con 0x7f81b81066e0 2024-03-19T03:03:35.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.163+0000 7f81af7fe640 1 -- 172.21.15.33:0/621911649 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81b0010980 con 0x7f81b81066e0 2024-03-19T03:03:35.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.163+0000 7f81c07bc640 1 -- 172.21.15.33:0/621911649 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81b81a4b50 con 0x7f81b81066e0 2024-03-19T03:03:35.168 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.164+0000 7f81c07bc640 1 -- 172.21.15.33:0/621911649 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81b8101e10 con 0x7f81b81066e0 2024-03-19T03:03:35.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.165+0000 7f81af7fe640 1 -- 172.21.15.33:0/621911649 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f81b0004020 con 0x7f81b81066e0 2024-03-19T03:03:35.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.166+0000 7f81af7fe640 1 --2- 172.21.15.33:0/621911649 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f81a40a2970 0x7f81a40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:03:35.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.166+0000 7f81be558640 1 --2- 172.21.15.33:0/621911649 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f81a40a2970 0x7f81a40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:03:35.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.166+0000 7f81af7fe640 1 -- 172.21.15.33:0/621911649 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9496+0+0 (secure 0 0 0) 0x7f81b00cf6b0 con 0x7f81b81066e0 2024-03-19T03:03:35.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.166+0000 7f81be558640 1 --2- 172.21.15.33:0/621911649 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f81a40a2970 0x7f81a40a4e30 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f81b4005ab0 tx=0x7f81b4005fd0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:03:35.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.170+0000 7f81af7fe640 1 -- 172.21.15.33:0/621911649 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f81b009bfe0 con 0x7f81b81066e0 2024-03-19T03:03:35.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.409+0000 7f81c07bc640 1 -- 172.21.15.33:0/621911649 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f81b8199330 con 0x7f81b81066e0 2024-03-19T03:03:35.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.417+0000 7f81af7fe640 1 -- 172.21.15.33:0/621911649 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v25) ==== 81+0+740362 (secure 0 0 0) 0x7f81b009b8e0 con 0x7f81b81066e0 2024-03-19T03:03:35.430 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T03:03:35.430 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T03:03:35.430 INFO:teuthology.orchestra.run.smithi033.stdout: "epoch": 25, 2024-03-19T03:03:35.430 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.430 INFO:teuthology.orchestra.run.smithi033.stdout: "active_gid": 15618, 2024-03-19T03:03:35.430 INFO:teuthology.orchestra.run.smithi033.stdout: "active_name": "x", 2024-03-19T03:03:35.430 INFO:teuthology.orchestra.run.smithi033.stdout: "active_addrs": { 2024-03-19T03:03:35.430 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-03-19T03:03:35.430 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.431 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-03-19T03:03:35.431 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6800", 2024-03-19T03:03:35.431 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2362023661 2024-03-19T03:03:35.431 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.431 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.431 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v1", 2024-03-19T03:03:35.431 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:6801", 2024-03-19T03:03:35.431 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2362023661 2024-03-19T03:03:35.431 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.431 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-03-19T03:03:35.431 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.431 INFO:teuthology.orchestra.run.smithi033.stdout: "active_addr": "172.21.15.33:6801/2362023661", 2024-03-19T03:03:35.431 INFO:teuthology.orchestra.run.smithi033.stdout: "active_change": "2024-03-19T02:53:26.937353+0000", 2024-03-19T03:03:35.431 INFO:teuthology.orchestra.run.smithi033.stdout: "active_mgr_features": 4540701547738038271, 2024-03-19T03:03:35.431 INFO:teuthology.orchestra.run.smithi033.stdout: "available": true, 2024-03-19T03:03:35.432 INFO:teuthology.orchestra.run.smithi033.stdout: "standbys": [ 2024-03-19T03:03:35.432 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.432 INFO:teuthology.orchestra.run.smithi033.stdout: "gid": 15609, 2024-03-19T03:03:35.432 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "z", 2024-03-19T03:03:35.432 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_features": 4540701547738038271, 2024-03-19T03:03:35.432 INFO:teuthology.orchestra.run.smithi033.stdout: "available_modules": [ 2024-03-19T03:03:35.432 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.432 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "alerts", 2024-03-19T03:03:35.432 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.432 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.432 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.432 INFO:teuthology.orchestra.run.smithi033.stdout: "interval": { 2024-03-19T03:03:35.432 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "interval", 2024-03-19T03:03:35.432 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:35.432 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.433 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.433 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "60", 2024-03-19T03:03:35.433 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.433 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.433 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.433 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "How frequently to reexamine health status", 2024-03-19T03:03:35.433 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.433 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.433 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.433 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.433 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.433 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.433 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.433 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.433 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.434 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.434 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.434 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.434 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.434 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.434 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.434 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.434 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.434 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.435 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.435 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.435 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.435 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.435 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.435 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.435 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.435 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.435 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.435 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.436 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.436 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.436 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.436 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.436 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.436 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.436 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.436 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.436 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.436 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.436 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.436 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.436 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.436 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.436 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.437 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.437 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.437 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.437 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.437 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.437 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.437 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.437 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.437 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.437 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.437 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.437 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.437 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.437 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.437 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.438 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.438 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.438 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.438 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.438 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.438 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.438 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.438 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.438 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.438 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.438 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.438 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.438 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.438 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.438 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.439 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.439 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_destination": { 2024-03-19T03:03:35.439 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_destination", 2024-03-19T03:03:35.439 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.439 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.439 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.439 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.439 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.439 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.439 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.439 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Email address to send alerts to", 2024-03-19T03:03:35.439 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.439 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.439 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.439 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.440 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_from_name": { 2024-03-19T03:03:35.440 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_from_name", 2024-03-19T03:03:35.440 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.440 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.440 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.440 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "Ceph", 2024-03-19T03:03:35.440 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.440 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.440 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.440 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Email From: name", 2024-03-19T03:03:35.440 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.440 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.440 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.440 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.440 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_host": { 2024-03-19T03:03:35.440 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_host", 2024-03-19T03:03:35.441 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.441 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.441 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.441 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.441 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.441 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.441 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.441 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "SMTP server", 2024-03-19T03:03:35.441 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.441 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.441 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.441 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.441 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_password": { 2024-03-19T03:03:35.441 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_password", 2024-03-19T03:03:35.441 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.442 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.442 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.442 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.442 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.442 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.442 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.442 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Password to authenticate with", 2024-03-19T03:03:35.442 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.442 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.442 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.442 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.442 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_port": { 2024-03-19T03:03:35.442 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_port", 2024-03-19T03:03:35.442 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.442 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.443 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.443 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "465", 2024-03-19T03:03:35.443 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.443 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.443 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.443 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "SMTP port", 2024-03-19T03:03:35.443 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.443 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.443 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.443 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.443 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_sender": { 2024-03-19T03:03:35.443 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_sender", 2024-03-19T03:03:35.443 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.443 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.443 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.444 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.444 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.444 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.444 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.444 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "SMTP envelope sender", 2024-03-19T03:03:35.444 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.444 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.444 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.444 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.444 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_ssl": { 2024-03-19T03:03:35.444 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_ssl", 2024-03-19T03:03:35.444 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.444 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.445 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.445 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.445 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.445 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.445 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.445 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-19T03:03:35.445 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.445 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.445 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.445 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.445 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_user": { 2024-03-19T03:03:35.445 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_user", 2024-03-19T03:03:35.445 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.445 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.445 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.446 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.446 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.446 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.446 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.446 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "User to authenticate as", 2024-03-19T03:03:35.446 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.446 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.446 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.446 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.446 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.446 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.446 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.446 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "balancer", 2024-03-19T03:03:35.446 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.447 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.447 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.447 INFO:teuthology.orchestra.run.smithi033.stdout: "active": { 2024-03-19T03:03:35.447 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "active", 2024-03-19T03:03:35.447 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.447 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.447 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.447 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.447 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.447 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.447 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.447 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "automatically balance PGs across cluster", 2024-03-19T03:03:35.447 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.447 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.447 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.448 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.448 INFO:teuthology.orchestra.run.smithi033.stdout: "begin_time": { 2024-03-19T03:03:35.448 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "begin_time", 2024-03-19T03:03:35.448 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.448 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.448 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.448 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0000", 2024-03-19T03:03:35.448 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.448 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.448 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.448 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "beginning time of day to automatically balance", 2024-03-19T03:03:35.448 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-19T03:03:35.448 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.448 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.448 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.448 INFO:teuthology.orchestra.run.smithi033.stdout: "begin_weekday": { 2024-03-19T03:03:35.449 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "begin_weekday", 2024-03-19T03:03:35.449 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "uint", 2024-03-19T03:03:35.449 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.449 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.449 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0", 2024-03-19T03:03:35.449 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "0", 2024-03-19T03:03:35.449 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "6", 2024-03-19T03:03:35.449 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.449 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-19T03:03:35.449 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-19T03:03:35.449 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.449 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.449 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.449 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_compat_max_iterations": { 2024-03-19T03:03:35.449 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "crush_compat_max_iterations", 2024-03-19T03:03:35.449 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "uint", 2024-03-19T03:03:35.450 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.450 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.450 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "25", 2024-03-19T03:03:35.450 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "1", 2024-03-19T03:03:35.450 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "250", 2024-03-19T03:03:35.450 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.450 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-19T03:03:35.450 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.450 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.450 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.450 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.450 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_compat_metrics": { 2024-03-19T03:03:35.450 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "crush_compat_metrics", 2024-03-19T03:03:35.450 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.451 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.451 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.451 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "pgs,objects,bytes", 2024-03-19T03:03:35.451 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.451 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.451 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.451 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-19T03:03:35.451 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-19T03:03:35.451 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.451 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.451 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.451 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_compat_step": { 2024-03-19T03:03:35.451 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "crush_compat_step", 2024-03-19T03:03:35.451 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-03-19T03:03:35.451 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.451 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.451 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0.5", 2024-03-19T03:03:35.452 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "0.001", 2024-03-19T03:03:35.452 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "0.999", 2024-03-19T03:03:35.452 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.452 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "aggressiveness of optimization", 2024-03-19T03:03:35.452 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-19T03:03:35.452 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.452 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.452 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.452 INFO:teuthology.orchestra.run.smithi033.stdout: "end_time": { 2024-03-19T03:03:35.452 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "end_time", 2024-03-19T03:03:35.452 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.452 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.452 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.452 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "2359", 2024-03-19T03:03:35.452 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.452 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.453 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.453 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "ending time of day to automatically balance", 2024-03-19T03:03:35.453 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-19T03:03:35.453 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.453 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.453 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.453 INFO:teuthology.orchestra.run.smithi033.stdout: "end_weekday": { 2024-03-19T03:03:35.453 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "end_weekday", 2024-03-19T03:03:35.453 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "uint", 2024-03-19T03:03:35.453 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.453 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.453 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0", 2024-03-19T03:03:35.453 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "0", 2024-03-19T03:03:35.453 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "6", 2024-03-19T03:03:35.453 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.454 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-19T03:03:35.454 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-19T03:03:35.454 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.454 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.454 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.454 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.454 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.454 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.454 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.454 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.454 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.454 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.454 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.455 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.455 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.455 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.455 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.455 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.455 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.455 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.455 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.455 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.455 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.455 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.455 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.455 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.455 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.455 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.456 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.456 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.456 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.456 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.456 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.456 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.456 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.456 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.456 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.456 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.456 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.456 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.456 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.456 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.457 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.457 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.457 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.457 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.457 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.457 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.457 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.457 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.457 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.457 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.457 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.457 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.457 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.457 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.457 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.458 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.458 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.458 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.458 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.458 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.458 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.458 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.458 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.458 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.458 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.458 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.458 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.458 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.458 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.459 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.459 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.459 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.459 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.459 INFO:teuthology.orchestra.run.smithi033.stdout: "min_score": { 2024-03-19T03:03:35.459 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "min_score", 2024-03-19T03:03:35.459 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-03-19T03:03:35.459 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.459 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.459 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0", 2024-03-19T03:03:35.459 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.459 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.459 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.459 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-19T03:03:35.459 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.459 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.460 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.460 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.460 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": { 2024-03-19T03:03:35.460 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mode", 2024-03-19T03:03:35.460 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.460 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.460 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.460 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "upmap", 2024-03-19T03:03:35.460 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.460 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.460 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.460 INFO:teuthology.orchestra.run.smithi033.stdout: "crush-compat", 2024-03-19T03:03:35.460 INFO:teuthology.orchestra.run.smithi033.stdout: "none", 2024-03-19T03:03:35.460 INFO:teuthology.orchestra.run.smithi033.stdout: "read", 2024-03-19T03:03:35.460 INFO:teuthology.orchestra.run.smithi033.stdout: "upmap", 2024-03-19T03:03:35.461 INFO:teuthology.orchestra.run.smithi033.stdout: "upmap-read" 2024-03-19T03:03:35.461 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.461 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Balancer mode", 2024-03-19T03:03:35.461 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.461 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.461 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.461 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.461 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_ids": { 2024-03-19T03:03:35.461 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "pool_ids", 2024-03-19T03:03:35.461 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.461 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.461 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.461 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.461 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.461 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.462 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.462 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-19T03:03:35.462 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.462 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.462 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.462 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.462 INFO:teuthology.orchestra.run.smithi033.stdout: "sleep_interval": { 2024-03-19T03:03:35.462 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "sleep_interval", 2024-03-19T03:03:35.462 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:35.462 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.462 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.462 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "60", 2024-03-19T03:03:35.462 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.462 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.463 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.463 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-19T03:03:35.463 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.463 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.463 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.463 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.463 INFO:teuthology.orchestra.run.smithi033.stdout: "upmap_max_deviation": { 2024-03-19T03:03:35.463 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "upmap_max_deviation", 2024-03-19T03:03:35.463 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.463 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.463 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.463 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5", 2024-03-19T03:03:35.463 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "1", 2024-03-19T03:03:35.463 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.463 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.464 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-19T03:03:35.464 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-19T03:03:35.464 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.464 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.464 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.464 INFO:teuthology.orchestra.run.smithi033.stdout: "upmap_max_optimizations": { 2024-03-19T03:03:35.464 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "upmap_max_optimizations", 2024-03-19T03:03:35.464 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "uint", 2024-03-19T03:03:35.464 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.464 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.464 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-03-19T03:03:35.464 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.464 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.464 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.464 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-19T03:03:35.465 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.465 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.465 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.465 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.465 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.465 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.465 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.465 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "cephadm", 2024-03-19T03:03:35.465 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.465 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.465 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.465 INFO:teuthology.orchestra.run.smithi033.stdout: "agent_down_multiplier": { 2024-03-19T03:03:35.465 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "agent_down_multiplier", 2024-03-19T03:03:35.465 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-03-19T03:03:35.465 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.466 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.466 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "3.0", 2024-03-19T03:03:35.466 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.466 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.466 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.466 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-19T03:03:35.466 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.466 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.466 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.466 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.466 INFO:teuthology.orchestra.run.smithi033.stdout: "agent_refresh_rate": { 2024-03-19T03:03:35.466 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "agent_refresh_rate", 2024-03-19T03:03:35.466 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:35.466 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.467 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.467 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "20", 2024-03-19T03:03:35.467 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.467 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.467 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.467 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-19T03:03:35.467 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.467 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.467 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.467 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.467 INFO:teuthology.orchestra.run.smithi033.stdout: "agent_starting_port": { 2024-03-19T03:03:35.467 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "agent_starting_port", 2024-03-19T03:03:35.467 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.467 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.467 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.467 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "4721", 2024-03-19T03:03:35.468 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.468 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.468 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.468 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-19T03:03:35.468 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.468 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.468 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.468 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.468 INFO:teuthology.orchestra.run.smithi033.stdout: "allow_ptrace": { 2024-03-19T03:03:35.468 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "allow_ptrace", 2024-03-19T03:03:35.468 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.468 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.468 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.468 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.469 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.469 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.469 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.469 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-19T03:03:35.469 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-19T03:03:35.469 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.469 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.469 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.469 INFO:teuthology.orchestra.run.smithi033.stdout: "autotune_interval": { 2024-03-19T03:03:35.469 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "autotune_interval", 2024-03-19T03:03:35.469 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:35.469 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.469 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.469 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "600", 2024-03-19T03:03:35.469 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.469 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.470 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.470 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-19T03:03:35.470 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.470 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.470 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.470 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.470 INFO:teuthology.orchestra.run.smithi033.stdout: "autotune_memory_target_ratio": { 2024-03-19T03:03:35.470 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "autotune_memory_target_ratio", 2024-03-19T03:03:35.470 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-03-19T03:03:35.470 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.470 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.470 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0.7", 2024-03-19T03:03:35.470 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.470 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.470 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.471 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-19T03:03:35.471 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.471 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.471 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.471 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.471 INFO:teuthology.orchestra.run.smithi033.stdout: "cephadm_log_destination": { 2024-03-19T03:03:35.471 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "cephadm_log_destination", 2024-03-19T03:03:35.471 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.471 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.471 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.471 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.471 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.471 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.471 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.471 INFO:teuthology.orchestra.run.smithi033.stdout: "file", 2024-03-19T03:03:35.472 INFO:teuthology.orchestra.run.smithi033.stdout: "file,syslog", 2024-03-19T03:03:35.472 INFO:teuthology.orchestra.run.smithi033.stdout: "syslog" 2024-03-19T03:03:35.472 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.472 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-19T03:03:35.472 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.472 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.472 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.472 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.472 INFO:teuthology.orchestra.run.smithi033.stdout: "cgroups_split": { 2024-03-19T03:03:35.472 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "cgroups_split", 2024-03-19T03:03:35.472 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.472 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.472 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.472 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.472 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.473 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.473 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.473 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-19T03:03:35.473 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.473 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.473 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.473 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.473 INFO:teuthology.orchestra.run.smithi033.stdout: "config_checks_enabled": { 2024-03-19T03:03:35.473 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "config_checks_enabled", 2024-03-19T03:03:35.473 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.473 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.473 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.473 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.473 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.473 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.474 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.474 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-19T03:03:35.474 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.474 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.474 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.474 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.474 INFO:teuthology.orchestra.run.smithi033.stdout: "config_dashboard": { 2024-03-19T03:03:35.474 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "config_dashboard", 2024-03-19T03:03:35.474 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.474 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.474 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.474 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.474 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.474 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.474 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.475 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-19T03:03:35.475 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.475 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.475 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.475 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.475 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_alertmanager": { 2024-03-19T03:03:35.475 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_alertmanager", 2024-03-19T03:03:35.475 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.475 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.475 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.475 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-19T03:03:35.475 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.475 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.475 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.475 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Prometheus container image", 2024-03-19T03:03:35.475 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.475 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.476 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.476 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.476 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_base": { 2024-03-19T03:03:35.476 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_base", 2024-03-19T03:03:35.476 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.476 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.476 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.476 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-19T03:03:35.476 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.476 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.476 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.476 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Container image name, without the tag", 2024-03-19T03:03:35.476 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.476 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.476 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.477 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.477 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_elasticsearch": { 2024-03-19T03:03:35.477 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_elasticsearch", 2024-03-19T03:03:35.477 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.477 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.477 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.477 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-19T03:03:35.477 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.477 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.477 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.477 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "elasticsearch container image", 2024-03-19T03:03:35.477 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.477 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.477 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.478 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.478 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_grafana": { 2024-03-19T03:03:35.478 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_grafana", 2024-03-19T03:03:35.478 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.478 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.478 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.478 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-19T03:03:35.478 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.478 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.478 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.478 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Prometheus container image", 2024-03-19T03:03:35.478 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.478 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.478 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.478 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.479 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_haproxy": { 2024-03-19T03:03:35.479 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_haproxy", 2024-03-19T03:03:35.479 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.479 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.479 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.479 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-19T03:03:35.479 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.479 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.479 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.479 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "HAproxy container image", 2024-03-19T03:03:35.479 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.479 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.479 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.479 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.480 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_jaeger_agent": { 2024-03-19T03:03:35.480 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_jaeger_agent", 2024-03-19T03:03:35.480 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.480 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.480 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.480 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-19T03:03:35.480 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.480 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.480 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.480 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Jaeger agent container image", 2024-03-19T03:03:35.480 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.480 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.480 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.480 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.480 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_jaeger_collector": { 2024-03-19T03:03:35.481 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_jaeger_collector", 2024-03-19T03:03:35.481 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.481 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.481 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.481 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-19T03:03:35.481 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.481 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.481 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.481 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Jaeger collector container image", 2024-03-19T03:03:35.481 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.481 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.481 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.481 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.481 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_jaeger_query": { 2024-03-19T03:03:35.481 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_jaeger_query", 2024-03-19T03:03:35.481 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.482 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.482 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.482 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-19T03:03:35.482 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.482 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.482 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.482 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Jaeger query container image", 2024-03-19T03:03:35.482 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.482 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.482 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.482 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.482 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_keepalived": { 2024-03-19T03:03:35.482 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_keepalived", 2024-03-19T03:03:35.482 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.482 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.483 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.483 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-19T03:03:35.483 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.483 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.483 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.483 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Keepalived container image", 2024-03-19T03:03:35.483 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.483 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.483 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.483 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.483 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_loki": { 2024-03-19T03:03:35.483 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_loki", 2024-03-19T03:03:35.483 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.483 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.483 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.484 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-19T03:03:35.484 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.484 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.484 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.484 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Loki container image", 2024-03-19T03:03:35.484 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.484 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.484 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.484 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.484 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_node_exporter": { 2024-03-19T03:03:35.485 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_node_exporter", 2024-03-19T03:03:35.485 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.485 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.485 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.485 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-19T03:03:35.485 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.485 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.485 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.485 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Prometheus container image", 2024-03-19T03:03:35.485 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.485 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.485 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.485 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.485 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_nvmeof": { 2024-03-19T03:03:35.485 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_nvmeof", 2024-03-19T03:03:35.485 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.486 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.486 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.486 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-19T03:03:35.486 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.486 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.486 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.486 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Nvme-of container image", 2024-03-19T03:03:35.486 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.486 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.486 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.486 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.486 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_prometheus": { 2024-03-19T03:03:35.486 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_prometheus", 2024-03-19T03:03:35.487 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.487 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.487 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.487 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-19T03:03:35.487 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.487 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.487 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.487 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Prometheus container image", 2024-03-19T03:03:35.487 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.487 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.487 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.487 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.487 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_promtail": { 2024-03-19T03:03:35.487 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_promtail", 2024-03-19T03:03:35.487 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.488 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.488 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.488 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-19T03:03:35.488 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.488 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.488 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.488 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Promtail container image", 2024-03-19T03:03:35.488 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.488 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.488 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.488 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.488 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_snmp_gateway": { 2024-03-19T03:03:35.488 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_snmp_gateway", 2024-03-19T03:03:35.488 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.488 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.489 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.489 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-19T03:03:35.489 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.489 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.489 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.489 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "SNMP Gateway container image", 2024-03-19T03:03:35.489 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.489 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.489 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.489 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.489 INFO:teuthology.orchestra.run.smithi033.stdout: "container_init": { 2024-03-19T03:03:35.489 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_init", 2024-03-19T03:03:35.489 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.489 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.489 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.490 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.490 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.490 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.490 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.490 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Run podman/docker with `--init`", 2024-03-19T03:03:35.490 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.490 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.490 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.490 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.490 INFO:teuthology.orchestra.run.smithi033.stdout: "daemon_cache_timeout": { 2024-03-19T03:03:35.490 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "daemon_cache_timeout", 2024-03-19T03:03:35.490 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:35.490 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.491 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.491 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "600", 2024-03-19T03:03:35.491 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.491 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.491 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.491 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-19T03:03:35.491 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.491 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.491 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.491 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.492 INFO:teuthology.orchestra.run.smithi033.stdout: "default_cephadm_command_timeout": { 2024-03-19T03:03:35.492 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "default_cephadm_command_timeout", 2024-03-19T03:03:35.492 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.492 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.492 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.492 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "900", 2024-03-19T03:03:35.492 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.492 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.492 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.492 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-19T03:03:35.492 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.492 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.492 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.492 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.492 INFO:teuthology.orchestra.run.smithi033.stdout: "default_registry": { 2024-03-19T03:03:35.492 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "default_registry", 2024-03-19T03:03:35.493 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.493 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.493 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.493 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "docker.io", 2024-03-19T03:03:35.493 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.493 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.493 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.493 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-19T03:03:35.493 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.493 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.493 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.493 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.493 INFO:teuthology.orchestra.run.smithi033.stdout: "device_cache_timeout": { 2024-03-19T03:03:35.493 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "device_cache_timeout", 2024-03-19T03:03:35.494 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:35.494 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.494 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.494 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "1800", 2024-03-19T03:03:35.494 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.494 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.494 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.494 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "seconds to cache device inventory", 2024-03-19T03:03:35.494 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.494 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.494 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.494 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.494 INFO:teuthology.orchestra.run.smithi033.stdout: "device_enhanced_scan": { 2024-03-19T03:03:35.494 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "device_enhanced_scan", 2024-03-19T03:03:35.495 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.495 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.495 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.495 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.495 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.495 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.495 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.495 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-19T03:03:35.495 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.495 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.495 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.495 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.495 INFO:teuthology.orchestra.run.smithi033.stdout: "facts_cache_timeout": { 2024-03-19T03:03:35.495 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "facts_cache_timeout", 2024-03-19T03:03:35.495 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:35.495 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.496 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.496 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "60", 2024-03-19T03:03:35.496 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.496 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.496 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.496 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "seconds to cache host facts data", 2024-03-19T03:03:35.496 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.496 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.496 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.496 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.496 INFO:teuthology.orchestra.run.smithi033.stdout: "host_check_interval": { 2024-03-19T03:03:35.496 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "host_check_interval", 2024-03-19T03:03:35.496 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:35.496 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.496 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.496 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "600", 2024-03-19T03:03:35.497 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.497 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.497 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.497 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how frequently to perform a host check", 2024-03-19T03:03:35.497 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.497 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.497 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.497 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.497 INFO:teuthology.orchestra.run.smithi033.stdout: "hw_monitoring": { 2024-03-19T03:03:35.497 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "hw_monitoring", 2024-03-19T03:03:35.497 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.497 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.497 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.497 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.497 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.497 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.498 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.498 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-19T03:03:35.498 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.498 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.498 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.498 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.498 INFO:teuthology.orchestra.run.smithi033.stdout: "inventory_list_all": { 2024-03-19T03:03:35.498 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "inventory_list_all", 2024-03-19T03:03:35.498 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.498 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.498 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.498 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.498 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.498 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.498 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.499 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-19T03:03:35.499 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.499 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.499 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.499 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.499 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.499 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.499 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.499 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.499 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.499 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.499 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.499 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.499 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.499 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.500 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.500 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.500 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.500 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.500 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.500 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.500 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.500 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.500 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.500 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.500 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.500 INFO:teuthology.orchestra.run.smithi033.stdout: "log_refresh_metadata": { 2024-03-19T03:03:35.500 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_refresh_metadata", 2024-03-19T03:03:35.500 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.500 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.501 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.501 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.501 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.501 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.501 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.501 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-19T03:03:35.501 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.501 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.501 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.501 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.501 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.501 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.501 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.501 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.501 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.502 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.502 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.502 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.502 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.502 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-19T03:03:35.502 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.502 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.502 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.502 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.502 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.502 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.502 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.502 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.502 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.503 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.503 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.503 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.503 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.503 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.503 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.503 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.503 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.503 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.503 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.503 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.503 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.503 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.503 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.503 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.504 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.504 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.504 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.504 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.504 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.504 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.504 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.504 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.504 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.504 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.504 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.504 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.504 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.504 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.504 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.505 INFO:teuthology.orchestra.run.smithi033.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-19T03:03:35.505 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-19T03:03:35.505 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.505 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.505 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.505 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.505 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.505 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.505 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.505 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-19T03:03:35.505 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.505 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.505 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.505 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.506 INFO:teuthology.orchestra.run.smithi033.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-19T03:03:35.506 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-19T03:03:35.506 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.506 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.506 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.506 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "*", 2024-03-19T03:03:35.506 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.506 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.506 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.506 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-19T03:03:35.506 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.506 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.506 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.506 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.506 INFO:teuthology.orchestra.run.smithi033.stdout: "max_count_per_host": { 2024-03-19T03:03:35.507 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "max_count_per_host", 2024-03-19T03:03:35.507 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.507 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.507 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.507 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-03-19T03:03:35.507 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.507 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.507 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.507 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "max number of daemons per service per host", 2024-03-19T03:03:35.507 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.507 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.507 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.507 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.507 INFO:teuthology.orchestra.run.smithi033.stdout: "max_osd_draining_count": { 2024-03-19T03:03:35.507 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "max_osd_draining_count", 2024-03-19T03:03:35.508 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.508 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.508 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.508 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-03-19T03:03:35.508 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.508 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.508 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.508 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-19T03:03:35.508 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.508 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.508 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.508 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.508 INFO:teuthology.orchestra.run.smithi033.stdout: "migration_current": { 2024-03-19T03:03:35.508 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "migration_current", 2024-03-19T03:03:35.508 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.509 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.509 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.509 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.509 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.509 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.509 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.509 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "internal - do not modify", 2024-03-19T03:03:35.509 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.509 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.509 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.509 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.509 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": { 2024-03-19T03:03:35.509 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mode", 2024-03-19T03:03:35.509 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.509 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.510 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.510 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "root", 2024-03-19T03:03:35.510 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.510 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.510 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.510 INFO:teuthology.orchestra.run.smithi033.stdout: "cephadm-package", 2024-03-19T03:03:35.510 INFO:teuthology.orchestra.run.smithi033.stdout: "root" 2024-03-19T03:03:35.510 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.510 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "mode for remote execution of cephadm", 2024-03-19T03:03:35.510 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.510 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.510 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.510 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.510 INFO:teuthology.orchestra.run.smithi033.stdout: "oob_default_addr": { 2024-03-19T03:03:35.510 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "oob_default_addr", 2024-03-19T03:03:35.511 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.511 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.511 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.511 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "169.254.1.1", 2024-03-19T03:03:35.511 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.511 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.511 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.511 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-19T03:03:35.511 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.511 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.511 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.511 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.511 INFO:teuthology.orchestra.run.smithi033.stdout: "prometheus_alerts_path": { 2024-03-19T03:03:35.511 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "prometheus_alerts_path", 2024-03-19T03:03:35.512 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.512 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.512 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.512 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-19T03:03:35.512 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.512 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.512 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.512 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-19T03:03:35.512 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.512 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.512 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.512 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.512 INFO:teuthology.orchestra.run.smithi033.stdout: "registry_insecure": { 2024-03-19T03:03:35.512 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "registry_insecure", 2024-03-19T03:03:35.512 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.513 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.513 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.513 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.513 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.513 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.513 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.513 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-19T03:03:35.513 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.513 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.513 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.513 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.513 INFO:teuthology.orchestra.run.smithi033.stdout: "registry_password": { 2024-03-19T03:03:35.513 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "registry_password", 2024-03-19T03:03:35.513 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.513 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.514 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.514 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.514 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.514 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.514 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.514 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-19T03:03:35.514 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.514 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.514 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.514 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.514 INFO:teuthology.orchestra.run.smithi033.stdout: "registry_url": { 2024-03-19T03:03:35.514 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "registry_url", 2024-03-19T03:03:35.514 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.514 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.515 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.515 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.515 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.515 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.515 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.515 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-19T03:03:35.515 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.515 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.515 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.515 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.515 INFO:teuthology.orchestra.run.smithi033.stdout: "registry_username": { 2024-03-19T03:03:35.515 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "registry_username", 2024-03-19T03:03:35.515 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.515 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.515 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.515 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.516 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.516 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.516 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.516 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-19T03:03:35.516 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.516 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.516 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.516 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.516 INFO:teuthology.orchestra.run.smithi033.stdout: "secure_monitoring_stack": { 2024-03-19T03:03:35.516 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "secure_monitoring_stack", 2024-03-19T03:03:35.516 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.516 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.516 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.516 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.516 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.517 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.517 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.517 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-19T03:03:35.517 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.517 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.517 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.517 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.517 INFO:teuthology.orchestra.run.smithi033.stdout: "service_discovery_port": { 2024-03-19T03:03:35.517 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "service_discovery_port", 2024-03-19T03:03:35.517 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.517 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.517 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.517 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "8765", 2024-03-19T03:03:35.517 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.517 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.517 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.518 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "cephadm service discovery port", 2024-03-19T03:03:35.518 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.518 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.518 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.518 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.518 INFO:teuthology.orchestra.run.smithi033.stdout: "ssh_config_file": { 2024-03-19T03:03:35.518 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ssh_config_file", 2024-03-19T03:03:35.518 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.518 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.518 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.518 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.518 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.518 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.518 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.518 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-19T03:03:35.519 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.519 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.519 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.519 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.519 INFO:teuthology.orchestra.run.smithi033.stdout: "use_agent": { 2024-03-19T03:03:35.519 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "use_agent", 2024-03-19T03:03:35.519 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.519 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.519 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.519 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.519 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.519 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.519 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.519 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-19T03:03:35.520 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.520 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.520 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.520 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.520 INFO:teuthology.orchestra.run.smithi033.stdout: "use_repo_digest": { 2024-03-19T03:03:35.520 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "use_repo_digest", 2024-03-19T03:03:35.520 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.520 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.520 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.520 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.520 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.520 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.520 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.520 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-19T03:03:35.520 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.520 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.521 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.521 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.521 INFO:teuthology.orchestra.run.smithi033.stdout: "warn_on_failed_host_check": { 2024-03-19T03:03:35.521 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "warn_on_failed_host_check", 2024-03-19T03:03:35.521 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.521 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.521 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.521 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.521 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.521 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.521 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.521 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "raise a health warning if the host check fails", 2024-03-19T03:03:35.521 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.521 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.521 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.522 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.522 INFO:teuthology.orchestra.run.smithi033.stdout: "warn_on_stray_daemons": { 2024-03-19T03:03:35.522 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "warn_on_stray_daemons", 2024-03-19T03:03:35.522 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.522 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.522 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.522 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.522 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.522 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.522 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.522 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-19T03:03:35.522 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.522 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.522 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.522 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.523 INFO:teuthology.orchestra.run.smithi033.stdout: "warn_on_stray_hosts": { 2024-03-19T03:03:35.523 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "warn_on_stray_hosts", 2024-03-19T03:03:35.523 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.523 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.523 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.523 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.523 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.523 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.523 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.523 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-19T03:03:35.523 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.523 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.523 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.523 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.523 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.524 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.524 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.524 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "crash", 2024-03-19T03:03:35.524 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.524 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.524 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.524 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.524 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.524 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.524 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.524 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.524 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.524 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.524 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.524 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.525 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.525 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.525 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.525 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.525 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.525 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.525 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.525 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.525 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.525 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.525 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.525 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.525 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.525 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.525 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.525 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.526 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.526 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.526 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.526 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.526 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.526 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.526 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.526 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.526 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.526 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.526 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.526 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.526 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.526 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.526 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.527 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.527 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.527 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.527 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.527 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.527 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.527 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.527 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.527 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.527 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.527 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.527 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.527 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.527 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.527 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.528 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.528 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.528 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.528 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.528 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.528 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.528 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.528 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.528 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.528 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.528 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.528 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.528 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.528 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.529 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.529 INFO:teuthology.orchestra.run.smithi033.stdout: "retain_interval": { 2024-03-19T03:03:35.529 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "retain_interval", 2024-03-19T03:03:35.529 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:35.529 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.529 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.529 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "31536000", 2024-03-19T03:03:35.529 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.529 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.529 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.529 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-19T03:03:35.529 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.529 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.529 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.530 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.530 INFO:teuthology.orchestra.run.smithi033.stdout: "warn_recent_interval": { 2024-03-19T03:03:35.530 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "warn_recent_interval", 2024-03-19T03:03:35.530 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:35.530 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.530 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.530 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "1209600", 2024-03-19T03:03:35.530 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.530 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.530 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.530 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-19T03:03:35.530 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.530 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.530 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.531 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.531 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.531 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.531 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.531 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "dashboard", 2024-03-19T03:03:35.531 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.531 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.531 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.531 INFO:teuthology.orchestra.run.smithi033.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-19T03:03:35.531 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-19T03:03:35.531 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.531 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.531 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.531 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-03-19T03:03:35.532 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.532 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.532 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.532 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.532 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.532 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.532 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.532 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.532 INFO:teuthology.orchestra.run.smithi033.stdout: "ALERTMANAGER_API_HOST": { 2024-03-19T03:03:35.532 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-19T03:03:35.532 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.532 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.532 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.532 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.533 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.533 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.533 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.533 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.533 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.533 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.533 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.533 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.533 INFO:teuthology.orchestra.run.smithi033.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-19T03:03:35.533 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-19T03:03:35.533 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.533 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.533 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.533 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.533 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.534 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.534 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.534 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.534 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.534 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.534 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.534 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.534 INFO:teuthology.orchestra.run.smithi033.stdout: "AUDIT_API_ENABLED": { 2024-03-19T03:03:35.534 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "AUDIT_API_ENABLED", 2024-03-19T03:03:35.534 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.534 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.534 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.534 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.534 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.534 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.534 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.535 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.535 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.535 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.535 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.535 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.535 INFO:teuthology.orchestra.run.smithi033.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-19T03:03:35.535 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-19T03:03:35.535 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.535 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.535 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.535 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.535 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.535 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.535 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.535 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.536 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.536 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.536 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.536 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.536 INFO:teuthology.orchestra.run.smithi033.stdout: "ENABLE_BROWSABLE_API": { 2024-03-19T03:03:35.536 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-19T03:03:35.536 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.536 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.536 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.536 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.536 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.536 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.536 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.536 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.537 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.537 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.537 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.537 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.537 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-19T03:03:35.537 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-19T03:03:35.537 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.537 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.537 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.537 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.537 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.537 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.537 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.537 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.537 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.537 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.538 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.538 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.538 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-19T03:03:35.538 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-19T03:03:35.538 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.538 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.538 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.538 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.538 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.538 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.538 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.538 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.538 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.538 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.539 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.539 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.539 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-19T03:03:35.539 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-19T03:03:35.539 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.539 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.539 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.539 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.539 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.539 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.539 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.539 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.539 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.539 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.540 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.540 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.540 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-19T03:03:35.540 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-19T03:03:35.540 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.540 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.540 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.540 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.540 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.540 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.540 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.540 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.540 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.540 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.540 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.541 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.541 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-19T03:03:35.541 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-19T03:03:35.541 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.541 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.541 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.541 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.541 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.541 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.541 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.541 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.541 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.541 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.541 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.541 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.541 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-19T03:03:35.542 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-19T03:03:35.542 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.542 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.542 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.542 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.542 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.542 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.542 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.542 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.542 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.542 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.542 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.542 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.542 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-19T03:03:35.543 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-19T03:03:35.543 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.543 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.543 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.543 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.543 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.543 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.543 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.543 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.543 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.543 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.543 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.543 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.543 INFO:teuthology.orchestra.run.smithi033.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-19T03:03:35.543 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-19T03:03:35.544 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.544 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.544 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.544 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.544 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.544 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.544 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.544 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.544 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.544 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.544 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.544 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.544 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_API_PASSWORD": { 2024-03-19T03:03:35.544 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-19T03:03:35.544 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.544 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.545 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.545 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "admin", 2024-03-19T03:03:35.545 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.545 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.545 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.545 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.545 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.545 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.545 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.545 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.545 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-19T03:03:35.545 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-19T03:03:35.545 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.545 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.545 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.546 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.546 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.546 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.546 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.546 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.546 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.546 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.546 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.546 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.546 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_API_URL": { 2024-03-19T03:03:35.546 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_API_URL", 2024-03-19T03:03:35.546 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.546 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.546 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.547 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.547 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.547 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.547 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.547 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.547 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.547 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.547 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.547 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.547 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_API_USERNAME": { 2024-03-19T03:03:35.547 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-19T03:03:35.547 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.547 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.547 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.547 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "admin", 2024-03-19T03:03:35.548 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.548 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.548 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.548 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.548 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.548 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.548 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.548 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.548 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-19T03:03:35.548 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-19T03:03:35.548 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.548 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.548 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.548 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.548 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.548 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.549 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.549 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.549 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.549 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.549 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.549 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.549 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-19T03:03:35.549 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-19T03:03:35.549 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.549 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.549 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.549 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.549 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.549 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.549 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.550 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.550 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.550 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.550 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.550 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.550 INFO:teuthology.orchestra.run.smithi033.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-19T03:03:35.550 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-19T03:03:35.550 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.550 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.550 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.550 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.550 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.550 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.550 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.550 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.551 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.551 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.551 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.551 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.551 INFO:teuthology.orchestra.run.smithi033.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-19T03:03:35.551 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-19T03:03:35.551 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.551 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.551 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.551 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.551 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.551 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.551 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.551 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.552 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.552 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.552 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.552 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.552 INFO:teuthology.orchestra.run.smithi033.stdout: "MULTICLUSTER_CONFIG": { 2024-03-19T03:03:35.552 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-19T03:03:35.552 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.552 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.552 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.552 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "{}", 2024-03-19T03:03:35.552 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.552 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.552 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.552 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.552 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.553 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.553 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.553 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.553 INFO:teuthology.orchestra.run.smithi033.stdout: "PROMETHEUS_API_HOST": { 2024-03-19T03:03:35.553 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-19T03:03:35.553 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.553 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.553 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.553 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.553 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.553 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.553 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.553 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.553 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.553 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.554 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.554 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.554 INFO:teuthology.orchestra.run.smithi033.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-19T03:03:35.554 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-19T03:03:35.554 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.554 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.554 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.554 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.554 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.554 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.554 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.554 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.554 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.554 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.554 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.555 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.555 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-19T03:03:35.555 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-19T03:03:35.555 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.555 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.555 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.555 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.555 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.555 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.555 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.555 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.555 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.555 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.555 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.555 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.556 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-19T03:03:35.556 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-19T03:03:35.556 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.556 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.556 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.556 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.556 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.556 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.556 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.556 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.556 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.556 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.556 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.556 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.556 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-19T03:03:35.557 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-19T03:03:35.557 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.557 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.557 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.557 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.557 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.557 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.557 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.557 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.557 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.557 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.557 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.557 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.557 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-19T03:03:35.557 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-19T03:03:35.558 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.558 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.558 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.558 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.558 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.558 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.558 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.558 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.558 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.558 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.558 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.558 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.558 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-19T03:03:35.558 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-19T03:03:35.558 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.558 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.559 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.559 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.559 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.559 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.559 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.559 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.559 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.559 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.559 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.559 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.559 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-19T03:03:35.559 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-19T03:03:35.559 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.559 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.559 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.560 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.560 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.560 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.560 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.560 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.560 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.560 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.560 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.560 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.560 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-19T03:03:35.560 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-19T03:03:35.560 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.560 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.560 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.560 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.560 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.561 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.561 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.561 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.561 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.561 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.561 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.561 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.561 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_ENABLED": { 2024-03-19T03:03:35.561 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-19T03:03:35.561 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.561 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.561 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.561 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.561 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.561 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.561 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.562 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.562 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.562 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.562 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.562 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.562 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-19T03:03:35.562 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-19T03:03:35.562 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.562 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.562 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.562 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-19T03:03:35.562 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.562 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.562 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.562 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.563 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.563 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.563 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.563 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.563 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-19T03:03:35.563 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-19T03:03:35.563 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.563 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.563 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.563 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-03-19T03:03:35.563 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.563 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.563 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.563 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.563 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.564 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.564 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.564 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.564 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-19T03:03:35.564 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-19T03:03:35.564 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.564 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.564 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.564 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "8", 2024-03-19T03:03:35.564 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.564 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.564 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.564 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.564 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.564 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.565 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.565 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.565 INFO:teuthology.orchestra.run.smithi033.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-19T03:03:35.565 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-19T03:03:35.565 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.565 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.565 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.565 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "45", 2024-03-19T03:03:35.565 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.565 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.565 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.565 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.565 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.565 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.565 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.566 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.566 INFO:teuthology.orchestra.run.smithi033.stdout: "RGW_API_ACCESS_KEY": { 2024-03-19T03:03:35.566 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-19T03:03:35.566 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.566 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.566 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.566 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.566 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.566 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.566 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.566 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.566 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.566 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.566 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.566 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.567 INFO:teuthology.orchestra.run.smithi033.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-19T03:03:35.567 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-19T03:03:35.567 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.567 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.567 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.567 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "admin", 2024-03-19T03:03:35.567 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.567 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.567 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.567 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.567 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.567 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.567 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.567 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.567 INFO:teuthology.orchestra.run.smithi033.stdout: "RGW_API_SECRET_KEY": { 2024-03-19T03:03:35.567 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-19T03:03:35.568 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.568 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.568 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.568 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.568 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.568 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.568 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.568 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.568 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.568 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.568 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.568 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.568 INFO:teuthology.orchestra.run.smithi033.stdout: "RGW_API_SSL_VERIFY": { 2024-03-19T03:03:35.568 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-19T03:03:35.568 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.568 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.569 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.569 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.569 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.569 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.569 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.569 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.569 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.569 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.569 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.569 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.569 INFO:teuthology.orchestra.run.smithi033.stdout: "UNSAFE_TLS_v1_2": { 2024-03-19T03:03:35.569 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-19T03:03:35.569 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.569 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.569 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.569 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.570 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.570 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.570 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.570 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.570 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.570 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.570 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.570 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.570 INFO:teuthology.orchestra.run.smithi033.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-19T03:03:35.570 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-19T03:03:35.570 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.570 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.570 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.570 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0", 2024-03-19T03:03:35.570 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.571 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.571 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.571 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.571 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.571 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.571 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.571 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.571 INFO:teuthology.orchestra.run.smithi033.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-19T03:03:35.571 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-19T03:03:35.571 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.571 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.571 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.571 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-03-19T03:03:35.571 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.571 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.572 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.572 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.572 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.572 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.572 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.572 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.572 INFO:teuthology.orchestra.run.smithi033.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-19T03:03:35.572 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-19T03:03:35.572 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.572 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.572 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.572 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5", 2024-03-19T03:03:35.572 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.572 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.572 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.572 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.573 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.573 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.573 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.573 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.573 INFO:teuthology.orchestra.run.smithi033.stdout: "cross_origin_url": { 2024-03-19T03:03:35.573 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "cross_origin_url", 2024-03-19T03:03:35.573 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.573 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.573 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.573 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.573 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.573 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.573 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.573 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.574 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.574 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.574 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.574 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.574 INFO:teuthology.orchestra.run.smithi033.stdout: "crt_file": { 2024-03-19T03:03:35.574 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "crt_file", 2024-03-19T03:03:35.574 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.574 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.574 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.574 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.574 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.574 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.574 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.574 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.574 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.575 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.575 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.575 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.575 INFO:teuthology.orchestra.run.smithi033.stdout: "debug": { 2024-03-19T03:03:35.575 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "debug", 2024-03-19T03:03:35.575 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.575 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.575 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.575 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.575 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.575 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.575 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.575 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Enable/disable debug options", 2024-03-19T03:03:35.575 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.576 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.576 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.576 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.576 INFO:teuthology.orchestra.run.smithi033.stdout: "jwt_token_ttl": { 2024-03-19T03:03:35.576 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "jwt_token_ttl", 2024-03-19T03:03:35.576 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.576 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.576 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.576 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "28800", 2024-03-19T03:03:35.576 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.576 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.576 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.576 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.576 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.576 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.577 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.577 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.577 INFO:teuthology.orchestra.run.smithi033.stdout: "key_file": { 2024-03-19T03:03:35.577 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "key_file", 2024-03-19T03:03:35.577 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.577 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.577 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.577 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.577 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.577 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.577 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.577 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.577 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.577 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.577 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.578 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.578 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.578 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.578 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.578 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.578 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.578 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.578 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.578 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.578 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.578 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.578 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.578 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.578 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.579 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.579 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.579 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.579 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.579 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.579 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.579 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.579 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.579 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.579 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.579 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.579 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.579 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.579 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.579 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.580 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.580 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.580 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.580 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.580 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.580 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.580 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.580 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.580 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.580 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.580 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.580 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.580 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.580 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.581 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.581 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.581 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.581 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.581 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.581 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.581 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.581 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.581 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.581 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.581 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.581 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.581 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.581 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.581 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.581 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.582 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.582 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.582 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.582 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.582 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.582 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.582 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.582 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.582 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.582 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.582 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.582 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.582 INFO:teuthology.orchestra.run.smithi033.stdout: "motd": { 2024-03-19T03:03:35.582 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "motd", 2024-03-19T03:03:35.582 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.583 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.583 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.583 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.583 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.583 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.583 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.583 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "The message of the day", 2024-03-19T03:03:35.583 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.583 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.583 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.583 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.583 INFO:teuthology.orchestra.run.smithi033.stdout: "redirect_resolve_ip_addr": { 2024-03-19T03:03:35.583 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "redirect_resolve_ip_addr", 2024-03-19T03:03:35.583 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.583 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.584 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.584 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.584 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.584 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.584 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.584 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.584 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.584 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.584 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.584 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.584 INFO:teuthology.orchestra.run.smithi033.stdout: "server_addr": { 2024-03-19T03:03:35.584 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_addr", 2024-03-19T03:03:35.584 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.585 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.585 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.585 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "::", 2024-03-19T03:03:35.585 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.585 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.585 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.585 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.585 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.585 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.585 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.585 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.585 INFO:teuthology.orchestra.run.smithi033.stdout: "server_port": { 2024-03-19T03:03:35.586 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_port", 2024-03-19T03:03:35.586 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.586 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.586 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.586 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "8080", 2024-03-19T03:03:35.586 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.586 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.586 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.586 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.586 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.586 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.586 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.586 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.586 INFO:teuthology.orchestra.run.smithi033.stdout: "ssl": { 2024-03-19T03:03:35.586 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ssl", 2024-03-19T03:03:35.587 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.587 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.587 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.587 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.587 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.587 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.587 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.587 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.587 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.587 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.587 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.587 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.587 INFO:teuthology.orchestra.run.smithi033.stdout: "ssl_server_port": { 2024-03-19T03:03:35.587 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ssl_server_port", 2024-03-19T03:03:35.587 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.588 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.588 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.588 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "8443", 2024-03-19T03:03:35.588 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.588 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.588 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.588 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.588 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.588 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.588 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.588 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.588 INFO:teuthology.orchestra.run.smithi033.stdout: "standby_behaviour": { 2024-03-19T03:03:35.588 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "standby_behaviour", 2024-03-19T03:03:35.588 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.588 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.589 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.589 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "redirect", 2024-03-19T03:03:35.589 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.589 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.589 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.589 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.589 INFO:teuthology.orchestra.run.smithi033.stdout: "redirect" 2024-03-19T03:03:35.589 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.589 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.589 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.589 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.589 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.589 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.589 INFO:teuthology.orchestra.run.smithi033.stdout: "standby_error_status_code": { 2024-03-19T03:03:35.589 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "standby_error_status_code", 2024-03-19T03:03:35.590 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.590 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.590 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.590 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "500", 2024-03-19T03:03:35.590 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "400", 2024-03-19T03:03:35.590 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "599", 2024-03-19T03:03:35.590 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.590 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.590 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.590 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.590 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.590 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.590 INFO:teuthology.orchestra.run.smithi033.stdout: "url_prefix": { 2024-03-19T03:03:35.590 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "url_prefix", 2024-03-19T03:03:35.591 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.591 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.591 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.591 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.591 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.591 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.591 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.591 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.591 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.591 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.591 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.591 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.591 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.591 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.591 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.592 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "devicehealth", 2024-03-19T03:03:35.592 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.592 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.592 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.592 INFO:teuthology.orchestra.run.smithi033.stdout: "enable_monitoring": { 2024-03-19T03:03:35.592 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "enable_monitoring", 2024-03-19T03:03:35.592 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.592 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.592 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.592 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.592 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.592 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.592 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.592 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "monitor device health metrics", 2024-03-19T03:03:35.593 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.593 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.593 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.593 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.593 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.593 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.593 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.593 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.593 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.593 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.593 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.593 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.593 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.593 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.593 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.594 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.594 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.594 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.594 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.594 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.594 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.594 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.594 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.594 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.594 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.594 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.594 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.594 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.595 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.595 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.595 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.595 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.595 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.595 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.595 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.595 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.595 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.595 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.595 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.595 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.595 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.595 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.595 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.596 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.596 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.596 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.596 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.596 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.596 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.596 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.596 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.596 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.596 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.596 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.596 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.596 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.596 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.596 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.596 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.597 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.597 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.597 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.597 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.597 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.597 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.597 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.597 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.597 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.597 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.597 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.597 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.597 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.597 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.597 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.597 INFO:teuthology.orchestra.run.smithi033.stdout: "mark_out_threshold": { 2024-03-19T03:03:35.597 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mark_out_threshold", 2024-03-19T03:03:35.598 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:35.598 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.598 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.598 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "2419200", 2024-03-19T03:03:35.598 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.598 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.598 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.598 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-19T03:03:35.598 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.598 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.598 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.598 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.598 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": { 2024-03-19T03:03:35.598 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "pool_name", 2024-03-19T03:03:35.598 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.599 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.599 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.599 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "device_health_metrics", 2024-03-19T03:03:35.599 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.599 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.599 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.599 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-19T03:03:35.599 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.599 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.599 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.599 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.599 INFO:teuthology.orchestra.run.smithi033.stdout: "retention_period": { 2024-03-19T03:03:35.599 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "retention_period", 2024-03-19T03:03:35.599 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:35.600 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.600 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.600 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "15552000", 2024-03-19T03:03:35.600 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.600 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.600 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.600 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how long to retain device health metrics", 2024-03-19T03:03:35.600 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.600 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.600 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.600 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.600 INFO:teuthology.orchestra.run.smithi033.stdout: "scrape_frequency": { 2024-03-19T03:03:35.600 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "scrape_frequency", 2024-03-19T03:03:35.600 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:35.600 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.600 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.601 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "86400", 2024-03-19T03:03:35.601 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.601 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.601 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.601 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-19T03:03:35.601 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.601 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.601 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.601 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.601 INFO:teuthology.orchestra.run.smithi033.stdout: "self_heal": { 2024-03-19T03:03:35.601 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "self_heal", 2024-03-19T03:03:35.601 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.601 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.601 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.602 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.602 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.602 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.602 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.602 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-19T03:03:35.602 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.602 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.602 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.602 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.602 INFO:teuthology.orchestra.run.smithi033.stdout: "sleep_interval": { 2024-03-19T03:03:35.602 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "sleep_interval", 2024-03-19T03:03:35.602 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:35.602 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.602 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.603 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "600", 2024-03-19T03:03:35.603 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.603 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.603 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.603 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how frequently to wake up and check device health", 2024-03-19T03:03:35.603 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.603 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.603 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.603 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.603 INFO:teuthology.orchestra.run.smithi033.stdout: "warn_threshold": { 2024-03-19T03:03:35.603 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "warn_threshold", 2024-03-19T03:03:35.603 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:35.603 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.603 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.603 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "7257600", 2024-03-19T03:03:35.604 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.604 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.604 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.604 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-19T03:03:35.604 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.604 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.604 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.604 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.604 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.604 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.604 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.604 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "diskprediction_local", 2024-03-19T03:03:35.604 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.605 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.605 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.605 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.605 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.605 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.605 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.605 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.605 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.605 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.605 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.605 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.605 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.605 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.605 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.605 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.605 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.606 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.606 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.606 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.606 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.606 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.606 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.606 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.606 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.606 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.606 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.606 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.606 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.606 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.607 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.607 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.607 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.607 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.607 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.607 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.607 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.607 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.607 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.607 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.607 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.607 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.607 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.607 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.607 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.608 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.608 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.608 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.608 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.608 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.608 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.608 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.608 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.608 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.608 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.608 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.608 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.608 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.608 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.608 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.608 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.609 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.609 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.609 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.609 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.609 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.609 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.609 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.609 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.609 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.609 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.609 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.609 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.609 INFO:teuthology.orchestra.run.smithi033.stdout: "predict_interval": { 2024-03-19T03:03:35.609 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "predict_interval", 2024-03-19T03:03:35.609 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.610 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.610 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.610 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "86400", 2024-03-19T03:03:35.610 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.610 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.610 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.610 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.610 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.610 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.610 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.610 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.610 INFO:teuthology.orchestra.run.smithi033.stdout: "predictor_model": { 2024-03-19T03:03:35.610 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "predictor_model", 2024-03-19T03:03:35.610 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.610 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.611 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.611 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "prophetstor", 2024-03-19T03:03:35.611 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.611 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.611 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.611 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.611 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.611 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.611 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.611 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.611 INFO:teuthology.orchestra.run.smithi033.stdout: "sleep_interval": { 2024-03-19T03:03:35.611 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "sleep_interval", 2024-03-19T03:03:35.611 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.611 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.611 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.612 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "600", 2024-03-19T03:03:35.612 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.612 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.612 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.612 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.612 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.612 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.612 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.612 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.612 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.612 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.612 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.612 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "influx", 2024-03-19T03:03:35.612 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": false, 2024-03-19T03:03:35.612 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "influxdb python module not found", 2024-03-19T03:03:35.613 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.613 INFO:teuthology.orchestra.run.smithi033.stdout: "batch_size": { 2024-03-19T03:03:35.613 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "batch_size", 2024-03-19T03:03:35.613 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.613 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.613 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.613 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5000", 2024-03-19T03:03:35.613 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.613 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.613 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.613 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-19T03:03:35.613 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.613 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.613 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.613 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.614 INFO:teuthology.orchestra.run.smithi033.stdout: "database": { 2024-03-19T03:03:35.614 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "database", 2024-03-19T03:03:35.614 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.614 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.614 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.614 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "ceph", 2024-03-19T03:03:35.614 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.614 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.614 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.614 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-19T03:03:35.614 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.614 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.614 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.614 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.615 INFO:teuthology.orchestra.run.smithi033.stdout: "hostname": { 2024-03-19T03:03:35.615 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "hostname", 2024-03-19T03:03:35.615 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.615 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.615 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.615 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.615 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.615 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.615 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.615 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "InfluxDB server hostname", 2024-03-19T03:03:35.615 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.615 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.615 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.615 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.616 INFO:teuthology.orchestra.run.smithi033.stdout: "interval": { 2024-03-19T03:03:35.616 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "interval", 2024-03-19T03:03:35.616 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:35.616 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.616 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.616 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "30", 2024-03-19T03:03:35.616 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "5", 2024-03-19T03:03:35.616 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.616 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.616 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-19T03:03:35.616 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.616 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.616 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.616 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.616 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.617 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.617 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.617 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.617 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.617 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.617 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.617 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.617 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.617 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.617 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.617 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.617 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.617 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.617 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.617 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.618 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.618 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.618 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.618 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.618 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.618 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.618 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.618 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.618 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.618 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.618 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.618 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.618 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.618 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.618 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.619 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.619 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.619 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.619 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.619 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.619 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.619 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.619 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.619 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.619 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.619 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.619 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.619 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.619 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.619 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.620 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.620 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.620 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.620 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.620 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.620 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.620 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.620 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.620 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.620 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.620 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.620 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.620 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.620 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.620 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.621 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.621 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.621 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.621 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.621 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.621 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.621 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.621 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.621 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.621 INFO:teuthology.orchestra.run.smithi033.stdout: "password": { 2024-03-19T03:03:35.621 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "password", 2024-03-19T03:03:35.621 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.621 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.621 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.621 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.622 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.622 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.622 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.622 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "password of InfluxDB server user", 2024-03-19T03:03:35.622 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.622 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.622 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.622 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.622 INFO:teuthology.orchestra.run.smithi033.stdout: "port": { 2024-03-19T03:03:35.622 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "port", 2024-03-19T03:03:35.622 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.622 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.622 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.622 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "8086", 2024-03-19T03:03:35.622 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.623 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.623 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.623 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "InfluxDB server port", 2024-03-19T03:03:35.623 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.623 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.623 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.623 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.623 INFO:teuthology.orchestra.run.smithi033.stdout: "ssl": { 2024-03-19T03:03:35.623 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ssl", 2024-03-19T03:03:35.623 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.623 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.623 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.623 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "false", 2024-03-19T03:03:35.623 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.623 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.624 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.624 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-19T03:03:35.624 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.624 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.624 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.624 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.624 INFO:teuthology.orchestra.run.smithi033.stdout: "threads": { 2024-03-19T03:03:35.624 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "threads", 2024-03-19T03:03:35.624 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.624 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.624 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.624 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5", 2024-03-19T03:03:35.624 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "1", 2024-03-19T03:03:35.624 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "32", 2024-03-19T03:03:35.624 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.625 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-19T03:03:35.625 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.625 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.625 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.625 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.625 INFO:teuthology.orchestra.run.smithi033.stdout: "username": { 2024-03-19T03:03:35.625 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "username", 2024-03-19T03:03:35.625 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.625 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.625 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.625 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.625 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.625 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.625 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.625 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "username of InfluxDB server user", 2024-03-19T03:03:35.626 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.626 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.626 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.626 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.626 INFO:teuthology.orchestra.run.smithi033.stdout: "verify_ssl": { 2024-03-19T03:03:35.626 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "verify_ssl", 2024-03-19T03:03:35.626 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.626 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.626 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.626 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "true", 2024-03-19T03:03:35.626 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.626 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.626 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.627 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-19T03:03:35.627 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.627 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.627 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.627 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.627 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.627 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.627 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.627 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "insights", 2024-03-19T03:03:35.627 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.627 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.627 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.627 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.627 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.628 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.628 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.628 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.628 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.628 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.628 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.628 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.628 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.628 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.628 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.628 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.628 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.628 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.628 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.628 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.629 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.629 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.629 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.629 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.629 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.629 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.629 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.629 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.629 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.629 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.629 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.629 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.629 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.629 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.629 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.629 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.630 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.630 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.630 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.630 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.630 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.630 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.630 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.633 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.633 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.633 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.633 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.633 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.633 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.633 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.633 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.634 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.634 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.634 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.634 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.634 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.634 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.634 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.634 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.634 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.634 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.634 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.634 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.634 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.634 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.635 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.635 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.635 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.635 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.635 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.635 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.635 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.635 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.635 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.635 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.635 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.635 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "iostat", 2024-03-19T03:03:35.635 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.635 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.636 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.636 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.636 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.636 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.636 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.636 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.636 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.636 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.636 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.636 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.636 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.636 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.636 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.636 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.637 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.637 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.637 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.637 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.637 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.637 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.637 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.637 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.637 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.637 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.637 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.637 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.637 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.637 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.637 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.638 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.638 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.638 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.638 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.638 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.638 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.638 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.638 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.638 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.638 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.638 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.638 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.638 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.638 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.638 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.639 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.639 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.639 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.639 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.639 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.639 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.639 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.639 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.639 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.639 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.639 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.639 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.639 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.639 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.639 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.640 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.640 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.640 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.640 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.640 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.640 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.640 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.640 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.640 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.640 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.640 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.640 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.640 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.640 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.641 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.641 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "k8sevents", 2024-03-19T03:03:35.641 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.641 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.641 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.641 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_event_retention_days": { 2024-03-19T03:03:35.641 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ceph_event_retention_days", 2024-03-19T03:03:35.641 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.641 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.641 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.641 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "7", 2024-03-19T03:03:35.641 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.641 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.641 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.641 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-19T03:03:35.642 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.642 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.642 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.642 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.642 INFO:teuthology.orchestra.run.smithi033.stdout: "config_check_secs": { 2024-03-19T03:03:35.642 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "config_check_secs", 2024-03-19T03:03:35.642 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.642 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.642 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.642 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-03-19T03:03:35.642 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "10", 2024-03-19T03:03:35.642 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.642 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.642 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-19T03:03:35.643 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.643 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.643 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.643 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.643 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.643 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.643 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.643 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.643 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.643 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.643 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.643 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.643 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.643 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.644 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.644 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.644 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.644 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.644 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.644 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.644 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.644 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.644 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.644 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.644 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.644 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.644 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.644 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.644 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.645 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.645 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.645 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.645 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.645 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.645 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.645 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.645 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.645 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.645 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.645 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.645 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.645 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.645 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.646 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.646 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.646 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.646 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.646 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.646 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.646 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.646 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.646 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.646 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.646 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.646 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.646 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.646 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.646 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.647 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.647 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.647 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.647 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.647 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.647 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.647 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.647 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.647 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.647 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.647 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.647 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.647 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.647 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.647 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.648 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.648 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.648 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.648 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.648 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "localpool", 2024-03-19T03:03:35.648 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.648 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.648 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.648 INFO:teuthology.orchestra.run.smithi033.stdout: "failure_domain": { 2024-03-19T03:03:35.648 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "failure_domain", 2024-03-19T03:03:35.648 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.648 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.648 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.648 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "host", 2024-03-19T03:03:35.648 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.649 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.649 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.649 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "failure domain for any created local pool", 2024-03-19T03:03:35.649 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-19T03:03:35.649 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.649 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.649 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.649 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.649 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.649 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.649 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.649 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.649 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.649 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.649 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.650 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.650 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.650 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.650 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.650 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.650 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.650 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.650 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.650 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.650 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.650 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.650 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.650 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.650 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.651 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.651 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.651 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.651 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.651 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.651 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.651 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.651 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.651 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.651 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.651 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.651 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.651 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.651 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.651 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.652 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.652 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.652 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.652 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.652 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.652 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.652 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.652 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.652 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.652 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.652 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.652 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.652 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.652 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.652 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.653 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.653 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.653 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.653 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.653 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.653 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.653 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.653 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.653 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.653 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.653 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.653 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.653 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.653 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.653 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.654 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.654 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.654 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.654 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": { 2024-03-19T03:03:35.654 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "min_size", 2024-03-19T03:03:35.654 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.654 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.654 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.654 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.654 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.654 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.654 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.654 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "default min_size for any created local pool", 2024-03-19T03:03:35.654 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-19T03:03:35.654 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.655 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.655 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.655 INFO:teuthology.orchestra.run.smithi033.stdout: "num_rep": { 2024-03-19T03:03:35.655 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "num_rep", 2024-03-19T03:03:35.655 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.655 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.655 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.655 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "3", 2024-03-19T03:03:35.655 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.655 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.655 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.655 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "default replica count for any created local pool", 2024-03-19T03:03:35.655 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.656 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.656 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.656 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.656 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": { 2024-03-19T03:03:35.656 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "pg_num", 2024-03-19T03:03:35.656 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.656 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.656 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.656 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "128", 2024-03-19T03:03:35.656 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.656 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.656 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.656 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "default pg_num for any created local pool", 2024-03-19T03:03:35.656 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.656 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.657 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.657 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.657 INFO:teuthology.orchestra.run.smithi033.stdout: "prefix": { 2024-03-19T03:03:35.657 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "prefix", 2024-03-19T03:03:35.657 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.657 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.657 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.657 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.657 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.657 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.657 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.657 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "name prefix for any created local pool", 2024-03-19T03:03:35.657 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.657 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.657 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.657 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.658 INFO:teuthology.orchestra.run.smithi033.stdout: "subtree": { 2024-03-19T03:03:35.658 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "subtree", 2024-03-19T03:03:35.658 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.658 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.658 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.658 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "rack", 2024-03-19T03:03:35.658 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.658 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.658 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.658 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-19T03:03:35.658 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-19T03:03:35.658 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.658 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.658 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.659 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.659 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.659 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.659 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mds_autoscaler", 2024-03-19T03:03:35.659 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.659 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.659 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.659 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.659 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.659 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.659 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.659 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.659 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.659 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.659 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.660 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.660 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.660 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.660 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.660 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.660 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.660 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.660 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.660 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.660 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.660 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.660 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.660 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.660 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.660 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.661 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.661 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.661 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.661 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.661 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.661 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.661 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.661 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.661 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.661 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.661 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.661 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.661 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.661 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.661 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.661 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.662 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.662 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.662 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.662 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.662 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.662 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.662 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.662 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.662 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.662 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.662 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.662 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.662 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.662 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.663 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.663 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.663 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.663 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.663 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.663 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.663 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.663 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.663 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.663 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.663 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.663 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.663 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.663 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.663 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.663 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.664 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.664 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.664 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.664 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.664 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mirroring", 2024-03-19T03:03:35.664 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.664 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.664 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.664 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.664 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.664 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.664 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.664 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.664 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.664 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.665 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.665 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.665 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.665 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.665 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.665 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.665 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.665 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.665 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.665 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.665 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.665 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.665 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.665 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.665 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.666 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.666 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.666 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.666 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.666 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.666 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.666 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.666 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.666 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.666 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.666 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.666 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.666 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.666 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.667 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.667 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.667 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.667 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.667 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.667 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.667 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.667 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.667 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.667 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.667 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.667 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.667 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.667 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.667 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.668 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.668 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.668 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.668 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.668 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.668 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.668 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.668 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.668 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.668 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.668 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.668 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.668 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.668 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.668 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.669 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.669 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.669 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.669 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.669 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.669 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.669 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.669 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "nfs", 2024-03-19T03:03:35.669 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.669 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.669 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.669 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.669 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.669 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.670 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.670 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.670 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.670 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.670 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.670 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.670 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.670 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.670 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.670 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.670 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.670 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.670 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.670 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.670 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.671 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.671 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.671 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.671 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.671 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.671 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.671 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.671 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.671 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.671 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.671 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.671 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.671 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.671 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.671 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.672 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.672 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.672 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.672 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.672 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.672 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.672 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.672 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.672 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.672 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.672 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.672 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.672 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.672 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.672 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.673 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.673 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.673 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.673 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.673 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.673 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.673 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.673 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.673 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.673 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.673 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.673 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.673 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.673 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.673 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.674 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.674 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.674 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.674 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.674 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.674 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.674 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.674 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.674 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.674 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.674 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "orchestrator", 2024-03-19T03:03:35.674 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.674 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.674 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.675 INFO:teuthology.orchestra.run.smithi033.stdout: "fail_fs": { 2024-03-19T03:03:35.675 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "fail_fs", 2024-03-19T03:03:35.675 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.675 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.675 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.675 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.675 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.675 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.675 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.675 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-19T03:03:35.675 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.675 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.675 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.675 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.675 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.676 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.676 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.676 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.676 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.676 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.676 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.676 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.676 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.676 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.676 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.676 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.676 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.676 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.676 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.676 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.677 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.677 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.677 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.677 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.677 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.677 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.677 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.677 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.677 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.677 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.677 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.677 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.677 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.677 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.677 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.677 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.678 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.678 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.678 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.678 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.678 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.678 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.678 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.678 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.678 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.678 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.678 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.678 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.678 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.678 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.678 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.679 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.679 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.679 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.679 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.679 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.679 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.679 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.679 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.679 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.679 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.679 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.679 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.679 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.679 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.679 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.680 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.680 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.680 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.680 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.680 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.680 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.680 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.680 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.680 INFO:teuthology.orchestra.run.smithi033.stdout: "orchestrator": { 2024-03-19T03:03:35.680 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "orchestrator", 2024-03-19T03:03:35.680 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.680 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.680 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.680 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.680 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.681 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.681 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.681 INFO:teuthology.orchestra.run.smithi033.stdout: "cephadm", 2024-03-19T03:03:35.681 INFO:teuthology.orchestra.run.smithi033.stdout: "rook", 2024-03-19T03:03:35.681 INFO:teuthology.orchestra.run.smithi033.stdout: "test_orchestrator" 2024-03-19T03:03:35.681 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.681 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Orchestrator backend", 2024-03-19T03:03:35.681 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.681 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.681 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.681 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.681 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.681 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.681 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.681 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "osd_perf_query", 2024-03-19T03:03:35.682 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.682 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.682 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.682 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.682 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.682 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.682 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.682 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.682 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.682 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.682 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.682 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.682 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.683 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.683 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.683 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.683 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.683 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.683 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.683 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.683 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.683 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.683 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.683 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.683 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.683 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.683 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.683 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.684 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.684 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.684 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.684 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.684 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.684 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.684 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.684 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.684 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.684 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.684 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.684 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.684 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.684 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.685 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.685 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.685 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.685 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.685 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.685 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.685 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.685 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.685 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.685 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.685 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.686 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.686 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.686 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.686 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.686 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.686 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.686 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.686 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.686 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.686 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.686 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.686 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.686 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.686 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.687 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.687 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.687 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.687 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.687 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.687 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.687 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.687 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.687 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.687 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "osd_support", 2024-03-19T03:03:35.687 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.687 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.687 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.687 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.687 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.688 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.688 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.688 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.688 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.688 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.688 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.688 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.688 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.688 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.688 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.688 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.688 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.688 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.688 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.688 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.689 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.689 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.689 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.689 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.689 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.689 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.689 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.689 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.689 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.689 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.689 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.689 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.689 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.689 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.689 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.690 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.690 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.690 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.690 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.690 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.690 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.690 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.690 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.690 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.690 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.690 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.690 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.690 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.690 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.690 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.691 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.691 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.691 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.691 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.691 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.691 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.691 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.691 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.691 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.691 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.691 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.691 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.691 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.691 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.691 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.692 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.692 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.692 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.692 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.692 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.692 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.692 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.692 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.692 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.692 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.692 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.692 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "pg_autoscaler", 2024-03-19T03:03:35.692 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.692 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.692 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.693 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.693 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.693 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.693 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.693 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.693 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.693 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.693 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.693 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.693 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.693 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.693 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.693 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.693 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.693 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.693 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.694 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.694 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.694 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.694 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.694 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.694 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.694 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.694 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.694 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.694 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.694 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.694 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.694 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.694 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.695 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.695 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.695 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.695 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.695 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.695 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.695 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.695 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.695 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.695 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.695 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.695 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.695 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.695 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.695 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.696 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.696 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.696 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.696 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.696 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.696 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.696 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.696 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.696 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.696 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.696 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.696 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.696 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.696 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.696 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.697 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.697 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.697 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.697 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.697 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.697 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.697 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.697 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.697 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.697 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.697 INFO:teuthology.orchestra.run.smithi033.stdout: "sleep_interval": { 2024-03-19T03:03:35.697 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "sleep_interval", 2024-03-19T03:03:35.697 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:35.697 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.697 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.698 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "60", 2024-03-19T03:03:35.698 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.698 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.698 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.698 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.698 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.698 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.698 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.698 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.698 INFO:teuthology.orchestra.run.smithi033.stdout: "threshold": { 2024-03-19T03:03:35.698 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "threshold", 2024-03-19T03:03:35.698 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-03-19T03:03:35.698 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.698 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.698 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "3.0", 2024-03-19T03:03:35.699 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "1.0", 2024-03-19T03:03:35.699 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.699 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.699 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "scaling threshold", 2024-03-19T03:03:35.699 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-19T03:03:35.699 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.699 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.699 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.699 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.699 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.699 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.699 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "progress", 2024-03-19T03:03:35.699 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.699 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.699 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.700 INFO:teuthology.orchestra.run.smithi033.stdout: "allow_pg_recovery_event": { 2024-03-19T03:03:35.700 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "allow_pg_recovery_event", 2024-03-19T03:03:35.700 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.700 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.700 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.700 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.700 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.700 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.700 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.700 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-19T03:03:35.700 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.700 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.700 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.700 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.700 INFO:teuthology.orchestra.run.smithi033.stdout: "enabled": { 2024-03-19T03:03:35.701 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "enabled", 2024-03-19T03:03:35.701 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.701 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.701 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.701 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.701 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.701 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.701 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.701 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.701 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.701 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.701 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.701 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.701 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.701 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.701 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.702 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.702 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.702 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.702 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.702 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.702 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.702 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.702 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.702 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.702 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.702 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.702 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.702 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.702 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.703 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.703 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.703 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.703 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.703 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.703 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.703 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.703 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.703 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.703 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.703 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.703 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.703 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.703 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.704 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.704 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.704 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.704 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.704 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.704 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.704 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.704 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.704 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.704 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.704 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.704 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.704 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.704 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.704 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.704 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.705 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.705 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.705 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.705 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.705 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.705 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.705 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.705 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.705 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.705 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.705 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.705 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.705 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.705 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.705 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.706 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.706 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.706 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.706 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.706 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.706 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.706 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.706 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.706 INFO:teuthology.orchestra.run.smithi033.stdout: "max_completed_events": { 2024-03-19T03:03:35.706 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "max_completed_events", 2024-03-19T03:03:35.706 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.706 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.706 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.706 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "50", 2024-03-19T03:03:35.707 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.707 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.707 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.707 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "number of past completed events to remember", 2024-03-19T03:03:35.707 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.707 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.707 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.707 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.707 INFO:teuthology.orchestra.run.smithi033.stdout: "sleep_interval": { 2024-03-19T03:03:35.707 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "sleep_interval", 2024-03-19T03:03:35.707 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:35.707 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.707 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.707 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5", 2024-03-19T03:03:35.707 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.708 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.708 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.708 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how long the module is going to sleep", 2024-03-19T03:03:35.708 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.708 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.708 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.708 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.708 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.708 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.708 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.708 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "prometheus", 2024-03-19T03:03:35.708 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.708 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.708 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.708 INFO:teuthology.orchestra.run.smithi033.stdout: "cache": { 2024-03-19T03:03:35.708 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "cache", 2024-03-19T03:03:35.709 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.709 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.709 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.709 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.709 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.709 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.709 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.709 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.709 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.709 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.709 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.709 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.709 INFO:teuthology.orchestra.run.smithi033.stdout: "exclude_perf_counters": { 2024-03-19T03:03:35.709 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "exclude_perf_counters", 2024-03-19T03:03:35.709 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.710 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.710 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.710 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.710 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.710 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.710 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.710 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-19T03:03:35.710 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-19T03:03:35.710 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.710 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.710 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.710 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.710 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.710 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.710 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.711 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.711 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.711 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.711 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.711 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.711 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.711 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.711 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.711 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.711 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.711 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.711 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.711 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.711 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.712 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.712 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.712 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.712 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.712 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.712 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.712 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.712 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.712 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.712 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.712 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.712 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.712 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.712 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.712 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.712 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.713 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.713 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.713 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.713 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.713 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.713 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.713 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.713 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.713 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.713 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.713 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.713 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.713 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.713 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.713 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.713 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.714 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.714 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.714 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.714 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.714 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.714 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.714 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.714 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.714 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.714 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.714 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.714 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.714 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.714 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.715 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.715 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.715 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.715 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.715 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.715 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.715 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_stats_pools": { 2024-03-19T03:03:35.715 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rbd_stats_pools", 2024-03-19T03:03:35.715 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.715 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.715 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.715 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.715 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.715 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.715 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.716 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.716 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.716 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.716 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.716 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.716 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-19T03:03:35.716 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-19T03:03:35.716 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.716 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.716 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.716 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "300", 2024-03-19T03:03:35.716 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.716 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.716 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.716 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.717 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.717 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.717 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.717 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.717 INFO:teuthology.orchestra.run.smithi033.stdout: "scrape_interval": { 2024-03-19T03:03:35.717 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "scrape_interval", 2024-03-19T03:03:35.717 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-03-19T03:03:35.717 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.717 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.717 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "15.0", 2024-03-19T03:03:35.717 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.717 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.717 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.717 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.717 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.718 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.718 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.718 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.718 INFO:teuthology.orchestra.run.smithi033.stdout: "server_addr": { 2024-03-19T03:03:35.718 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_addr", 2024-03-19T03:03:35.718 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.718 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.718 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.718 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "::", 2024-03-19T03:03:35.718 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.718 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.718 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.718 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-19T03:03:35.718 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.719 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.719 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.719 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.719 INFO:teuthology.orchestra.run.smithi033.stdout: "server_port": { 2024-03-19T03:03:35.719 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_port", 2024-03-19T03:03:35.719 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.719 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.719 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.719 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "9283", 2024-03-19T03:03:35.719 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.719 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.719 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.719 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-19T03:03:35.719 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.719 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.719 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.720 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.720 INFO:teuthology.orchestra.run.smithi033.stdout: "stale_cache_strategy": { 2024-03-19T03:03:35.720 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "stale_cache_strategy", 2024-03-19T03:03:35.720 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.720 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.720 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.720 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "log", 2024-03-19T03:03:35.720 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.720 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.720 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.720 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.720 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.720 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.720 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.721 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.721 INFO:teuthology.orchestra.run.smithi033.stdout: "standby_behaviour": { 2024-03-19T03:03:35.721 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "standby_behaviour", 2024-03-19T03:03:35.721 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.721 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.721 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.721 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "default", 2024-03-19T03:03:35.721 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.721 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.721 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.721 INFO:teuthology.orchestra.run.smithi033.stdout: "default", 2024-03-19T03:03:35.721 INFO:teuthology.orchestra.run.smithi033.stdout: "error" 2024-03-19T03:03:35.721 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.721 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.721 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.721 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.722 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.722 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.722 INFO:teuthology.orchestra.run.smithi033.stdout: "standby_error_status_code": { 2024-03-19T03:03:35.722 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "standby_error_status_code", 2024-03-19T03:03:35.722 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.722 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.722 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.722 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "500", 2024-03-19T03:03:35.722 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "400", 2024-03-19T03:03:35.722 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "599", 2024-03-19T03:03:35.722 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.722 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.722 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.722 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.723 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.723 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.723 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.723 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.723 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.723 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rbd_support", 2024-03-19T03:03:35.723 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.723 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.723 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.723 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.723 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.723 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.723 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.723 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.723 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.724 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.724 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.724 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.724 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.724 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.724 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.724 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.724 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.724 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.724 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.724 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.724 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.724 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.724 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.724 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.725 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.725 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.725 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.725 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.725 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.725 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.725 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.725 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.725 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.725 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.725 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.725 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.725 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.725 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.725 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.726 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.726 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.726 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.726 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.726 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.726 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.726 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.726 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.726 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.726 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.726 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.726 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.726 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.726 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.726 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.727 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.727 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.727 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.727 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.727 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.727 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.727 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.727 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.727 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.727 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.727 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.727 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.727 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.727 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.727 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.728 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.728 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.728 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.728 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.728 INFO:teuthology.orchestra.run.smithi033.stdout: "max_concurrent_snap_create": { 2024-03-19T03:03:35.728 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "max_concurrent_snap_create", 2024-03-19T03:03:35.728 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.728 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.728 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.728 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-03-19T03:03:35.728 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.728 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.728 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.728 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.728 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.729 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.729 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.729 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.729 INFO:teuthology.orchestra.run.smithi033.stdout: "mirror_snapshot_schedule": { 2024-03-19T03:03:35.729 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mirror_snapshot_schedule", 2024-03-19T03:03:35.729 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.729 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.729 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.729 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.729 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.729 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.729 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.729 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.729 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.729 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.729 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.730 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.730 INFO:teuthology.orchestra.run.smithi033.stdout: "trash_purge_schedule": { 2024-03-19T03:03:35.730 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "trash_purge_schedule", 2024-03-19T03:03:35.730 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.730 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.730 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.730 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.730 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.730 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.730 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.730 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.730 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.730 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.730 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.730 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.731 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.731 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.731 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.731 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "restful", 2024-03-19T03:03:35.731 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.731 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.731 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.731 INFO:teuthology.orchestra.run.smithi033.stdout: "enable_auth": { 2024-03-19T03:03:35.731 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "enable_auth", 2024-03-19T03:03:35.731 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.731 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.731 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.731 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.731 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.731 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.732 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.732 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.732 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.732 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.732 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.732 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.732 INFO:teuthology.orchestra.run.smithi033.stdout: "key_file": { 2024-03-19T03:03:35.732 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "key_file", 2024-03-19T03:03:35.732 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.732 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.732 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.732 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.732 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.732 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.732 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.732 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.733 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.733 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.733 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.733 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.733 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.733 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.733 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.733 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.733 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.733 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.733 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.734 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.734 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.734 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.734 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.734 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.734 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.734 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.734 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.734 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.734 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.734 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.734 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.734 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.734 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.735 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.735 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.735 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.735 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.735 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.735 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.735 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.735 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.735 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.735 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.735 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.735 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.735 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.735 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.735 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.736 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.736 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.736 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.736 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.736 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.736 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.736 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.736 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.736 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.736 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.736 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.736 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.736 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.736 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.736 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.737 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.737 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.737 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.737 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.737 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.737 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.737 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.737 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.737 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.737 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.737 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.737 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.737 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.737 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.737 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.737 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.738 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.738 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.738 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.738 INFO:teuthology.orchestra.run.smithi033.stdout: "server_addr": { 2024-03-19T03:03:35.738 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_addr", 2024-03-19T03:03:35.738 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.738 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.738 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.738 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.738 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.738 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.738 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.738 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.738 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.739 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.739 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.739 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.739 INFO:teuthology.orchestra.run.smithi033.stdout: "server_port": { 2024-03-19T03:03:35.739 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_port", 2024-03-19T03:03:35.739 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.739 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.739 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.739 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.739 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.739 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.739 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.739 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.739 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.739 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.739 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.740 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.740 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.740 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.740 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.740 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rgw", 2024-03-19T03:03:35.740 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.740 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.740 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.740 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.740 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.740 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.740 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.740 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.740 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.740 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.741 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.741 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.741 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.741 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.741 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.741 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.741 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.741 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.741 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.741 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.741 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.741 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.741 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.741 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.741 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.741 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.742 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.742 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.742 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.742 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.742 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.742 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.742 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.742 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.742 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.742 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.742 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.742 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.742 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.742 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.743 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.743 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.743 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.743 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.743 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.743 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.743 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.743 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.743 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.743 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.743 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.743 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.743 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.743 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.743 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.743 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.744 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.744 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.744 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.744 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.744 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.744 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.744 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.744 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.744 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.744 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.744 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.744 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.744 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.744 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.744 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.745 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.745 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.745 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.745 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.745 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.745 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rook", 2024-03-19T03:03:35.745 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.745 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.745 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.745 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.745 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.745 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.745 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.745 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.745 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.745 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.746 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.746 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.746 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.746 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.746 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.746 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.746 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.746 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.746 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.746 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.746 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.746 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.746 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.747 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.747 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.747 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.747 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.747 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.747 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.747 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.747 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.747 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.747 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.747 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.747 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.747 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.747 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.747 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.748 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.748 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.748 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.748 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.748 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.748 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.748 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.748 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.748 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.748 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.748 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.748 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.748 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.748 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.748 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.748 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.749 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.749 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.749 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.749 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.749 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.749 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.749 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.749 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.749 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.749 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.749 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.749 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.749 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.749 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.750 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.750 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.750 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.750 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.750 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.750 INFO:teuthology.orchestra.run.smithi033.stdout: "storage_class": { 2024-03-19T03:03:35.750 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "storage_class", 2024-03-19T03:03:35.750 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.750 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.750 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.750 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "local", 2024-03-19T03:03:35.750 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.750 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.751 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.751 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-19T03:03:35.751 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.751 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.751 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.751 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.751 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.751 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.751 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.751 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "selftest", 2024-03-19T03:03:35.751 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.751 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.751 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.751 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.751 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.752 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.752 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.752 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.752 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.752 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.752 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.752 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.752 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.752 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.752 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.752 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.752 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.752 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.752 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.752 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.753 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.753 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.753 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.753 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.753 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.753 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.753 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.753 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.753 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.753 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.753 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.753 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.753 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.753 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.753 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.754 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.754 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.754 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.754 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.754 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.754 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.754 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.754 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.754 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.754 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.754 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.754 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.754 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.754 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.755 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.755 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.755 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.755 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.755 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.755 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.755 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.755 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.755 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.755 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.755 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.755 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.755 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.755 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.756 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.756 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.756 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.756 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.756 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.756 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.756 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.756 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.756 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.756 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.756 INFO:teuthology.orchestra.run.smithi033.stdout: "roption1": { 2024-03-19T03:03:35.756 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "roption1", 2024-03-19T03:03:35.756 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.756 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.756 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.757 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.757 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.757 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.757 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.757 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.757 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.757 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.757 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.757 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.757 INFO:teuthology.orchestra.run.smithi033.stdout: "roption2": { 2024-03-19T03:03:35.757 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "roption2", 2024-03-19T03:03:35.757 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.757 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.757 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.758 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "xyz", 2024-03-19T03:03:35.758 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.758 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.758 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.758 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.758 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.758 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.758 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.758 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.758 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption1": { 2024-03-19T03:03:35.758 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption1", 2024-03-19T03:03:35.758 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.758 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.758 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.758 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.759 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.759 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.759 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.759 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.759 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.759 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.759 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.759 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.759 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption2": { 2024-03-19T03:03:35.759 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption2", 2024-03-19T03:03:35.759 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.759 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.759 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.760 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.760 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.760 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.760 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.760 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.760 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.760 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.760 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.760 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.760 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption3": { 2024-03-19T03:03:35.760 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption3", 2024-03-19T03:03:35.760 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-03-19T03:03:35.760 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.761 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.761 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.761 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.761 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.761 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.761 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.761 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.761 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.761 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.761 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.761 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption4": { 2024-03-19T03:03:35.761 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption4", 2024-03-19T03:03:35.761 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.761 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.761 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.762 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.762 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.762 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.762 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.762 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.762 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.762 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.762 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.762 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.762 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption5": { 2024-03-19T03:03:35.762 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption5", 2024-03-19T03:03:35.762 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.762 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.762 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.762 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.763 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.763 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.763 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.763 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.763 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.763 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.763 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.763 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.763 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption6": { 2024-03-19T03:03:35.763 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption6", 2024-03-19T03:03:35.763 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.763 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.763 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.763 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.763 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.764 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.764 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.764 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.764 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.764 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.764 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.764 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.764 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption7": { 2024-03-19T03:03:35.764 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption7", 2024-03-19T03:03:35.764 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.764 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.764 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.764 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.764 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "1", 2024-03-19T03:03:35.764 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "42", 2024-03-19T03:03:35.765 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.765 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.765 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.765 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.765 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.765 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.765 INFO:teuthology.orchestra.run.smithi033.stdout: "testkey": { 2024-03-19T03:03:35.765 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "testkey", 2024-03-19T03:03:35.765 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.765 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.765 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.765 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.765 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.765 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.765 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.766 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.766 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.766 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.766 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.766 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.766 INFO:teuthology.orchestra.run.smithi033.stdout: "testlkey": { 2024-03-19T03:03:35.766 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "testlkey", 2024-03-19T03:03:35.766 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.766 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.766 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.766 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.766 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.766 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.766 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.767 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.767 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.767 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.767 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.767 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.767 INFO:teuthology.orchestra.run.smithi033.stdout: "testnewline": { 2024-03-19T03:03:35.767 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "testnewline", 2024-03-19T03:03:35.767 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.767 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.767 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.767 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.767 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.767 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.767 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.767 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.767 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.768 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.768 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.768 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.768 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.768 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.768 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.768 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "snap_schedule", 2024-03-19T03:03:35.768 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.768 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.768 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.768 INFO:teuthology.orchestra.run.smithi033.stdout: "allow_m_granularity": { 2024-03-19T03:03:35.768 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "allow_m_granularity", 2024-03-19T03:03:35.768 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.768 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.768 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.769 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.769 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.769 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.769 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.769 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "allow minute scheduled snapshots", 2024-03-19T03:03:35.769 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.769 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.769 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.769 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.769 INFO:teuthology.orchestra.run.smithi033.stdout: "dump_on_update": { 2024-03-19T03:03:35.769 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "dump_on_update", 2024-03-19T03:03:35.769 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.769 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.769 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.769 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.770 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.770 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.770 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.770 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "dump database to debug log on update", 2024-03-19T03:03:35.770 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.770 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.770 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.770 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.770 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.770 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.770 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.770 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.770 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.770 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.770 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.771 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.771 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.771 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.771 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.771 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.771 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.771 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.771 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.771 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.771 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.771 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.771 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.771 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.771 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.771 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.772 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.772 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.772 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.772 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.772 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.772 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.772 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.772 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.772 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.772 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.772 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.772 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.772 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.772 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.772 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.773 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.773 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.773 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.773 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.773 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.773 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.773 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.773 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.773 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.773 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.773 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.773 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.773 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.773 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.774 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.774 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.774 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.774 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.774 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.774 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.774 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.774 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.774 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.774 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.774 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.774 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.774 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.774 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.775 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.775 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.775 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.775 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.775 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.775 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.775 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.775 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.775 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "stats", 2024-03-19T03:03:35.775 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.775 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.775 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.775 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.775 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.776 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.776 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.776 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.776 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.776 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.776 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.776 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.776 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.776 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.776 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.776 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.776 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.776 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.776 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.776 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.777 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.777 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.777 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.777 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.777 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.777 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.777 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.777 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.777 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.777 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.777 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.777 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.777 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.777 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.777 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.778 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.778 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.778 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.778 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.778 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.778 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.778 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.778 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.778 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.778 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.778 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.778 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.778 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.778 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.778 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.779 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.779 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.779 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.779 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.779 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.779 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.779 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.779 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.779 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.779 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.779 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.779 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.779 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.779 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.779 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.780 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.780 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.780 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.780 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.780 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.780 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.780 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.780 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.780 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.780 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.780 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.780 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "status", 2024-03-19T03:03:35.780 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.780 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.780 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.781 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.781 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.781 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.781 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.781 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.781 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.781 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.781 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.781 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.781 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.781 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.781 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.781 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.781 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.781 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.782 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.782 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.782 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.782 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.782 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.782 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.782 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.782 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.782 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.782 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.782 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.782 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.782 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.782 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.783 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.783 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.783 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.783 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.783 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.783 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.783 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.783 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.783 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.783 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.783 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.783 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.783 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.783 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.783 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.784 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.784 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.784 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.784 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.784 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.784 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.784 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.784 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.784 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.784 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.784 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.784 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.784 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.784 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.784 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.784 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.785 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.785 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.785 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.785 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.785 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.785 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.785 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.785 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.785 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.785 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.785 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.786 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.786 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.786 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "telegraf", 2024-03-19T03:03:35.786 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.786 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.786 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.786 INFO:teuthology.orchestra.run.smithi033.stdout: "address": { 2024-03-19T03:03:35.786 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "address", 2024-03-19T03:03:35.786 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.786 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.786 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.786 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-19T03:03:35.786 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.786 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.786 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.787 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.787 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.787 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.787 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.787 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.787 INFO:teuthology.orchestra.run.smithi033.stdout: "interval": { 2024-03-19T03:03:35.787 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "interval", 2024-03-19T03:03:35.787 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:35.787 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.787 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.787 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "15", 2024-03-19T03:03:35.787 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.787 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.787 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.788 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.788 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.788 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.788 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.788 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.788 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.788 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.788 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.788 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.788 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.788 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.788 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.788 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.788 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.788 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.789 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.789 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.789 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.789 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.789 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.789 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.789 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.789 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.789 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.789 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.789 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.789 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.789 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.789 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.789 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.790 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.790 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.790 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.790 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.790 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.790 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.790 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.790 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.790 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.790 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.790 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.790 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.790 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.790 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.790 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.791 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.791 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.791 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.791 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.791 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.791 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.791 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.791 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.791 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.791 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.791 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.791 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.791 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.791 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.791 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.792 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.792 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.792 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.792 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.792 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.792 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.792 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.792 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.792 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.792 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.792 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.792 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.792 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.792 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.793 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.793 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.793 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.793 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.793 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "telemetry", 2024-03-19T03:03:35.793 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.793 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.793 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.793 INFO:teuthology.orchestra.run.smithi033.stdout: "channel_basic": { 2024-03-19T03:03:35.793 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "channel_basic", 2024-03-19T03:03:35.793 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.793 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.793 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.793 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.793 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.794 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.794 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.794 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-19T03:03:35.794 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.794 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.794 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.794 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.794 INFO:teuthology.orchestra.run.smithi033.stdout: "channel_crash": { 2024-03-19T03:03:35.794 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "channel_crash", 2024-03-19T03:03:35.794 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.794 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.794 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.794 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.794 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.794 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.795 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.795 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-19T03:03:35.795 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.795 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.795 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.795 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.795 INFO:teuthology.orchestra.run.smithi033.stdout: "channel_device": { 2024-03-19T03:03:35.795 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "channel_device", 2024-03-19T03:03:35.795 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.795 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.795 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.795 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.795 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.795 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.795 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.796 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-19T03:03:35.796 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.796 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.796 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.796 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.796 INFO:teuthology.orchestra.run.smithi033.stdout: "channel_ident": { 2024-03-19T03:03:35.796 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "channel_ident", 2024-03-19T03:03:35.796 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.796 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.796 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.796 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.796 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.796 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.796 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.796 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-19T03:03:35.797 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.797 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.797 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.797 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.797 INFO:teuthology.orchestra.run.smithi033.stdout: "channel_perf": { 2024-03-19T03:03:35.797 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "channel_perf", 2024-03-19T03:03:35.797 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.797 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.797 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.797 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.797 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.797 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.797 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.797 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-19T03:03:35.797 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.798 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.798 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.798 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.798 INFO:teuthology.orchestra.run.smithi033.stdout: "contact": { 2024-03-19T03:03:35.798 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "contact", 2024-03-19T03:03:35.798 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.798 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.798 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.798 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.798 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.798 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.798 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.798 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.798 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.798 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.799 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.799 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.799 INFO:teuthology.orchestra.run.smithi033.stdout: "description": { 2024-03-19T03:03:35.799 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "description", 2024-03-19T03:03:35.799 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.799 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.799 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.799 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.799 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.799 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.799 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.799 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.799 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.799 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.800 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.800 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.800 INFO:teuthology.orchestra.run.smithi033.stdout: "device_url": { 2024-03-19T03:03:35.800 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "device_url", 2024-03-19T03:03:35.800 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.800 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.800 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.800 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-19T03:03:35.800 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.800 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.800 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.800 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.800 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.800 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.800 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.801 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.801 INFO:teuthology.orchestra.run.smithi033.stdout: "enabled": { 2024-03-19T03:03:35.801 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "enabled", 2024-03-19T03:03:35.801 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.801 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.801 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.801 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.801 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.801 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.801 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.801 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.801 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.801 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.801 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.801 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.802 INFO:teuthology.orchestra.run.smithi033.stdout: "interval": { 2024-03-19T03:03:35.802 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "interval", 2024-03-19T03:03:35.802 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.802 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.802 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.802 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "24", 2024-03-19T03:03:35.802 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "8", 2024-03-19T03:03:35.802 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.802 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.802 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.802 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.802 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.802 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.802 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.803 INFO:teuthology.orchestra.run.smithi033.stdout: "last_opt_revision": { 2024-03-19T03:03:35.803 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "last_opt_revision", 2024-03-19T03:03:35.803 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.803 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.803 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.803 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "1", 2024-03-19T03:03:35.803 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.803 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.803 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.803 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.803 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.803 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.803 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.803 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.803 INFO:teuthology.orchestra.run.smithi033.stdout: "leaderboard": { 2024-03-19T03:03:35.804 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "leaderboard", 2024-03-19T03:03:35.804 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.804 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.804 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.804 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.804 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.804 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.804 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.804 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.804 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.804 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.804 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.804 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.804 INFO:teuthology.orchestra.run.smithi033.stdout: "leaderboard_description": { 2024-03-19T03:03:35.804 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "leaderboard_description", 2024-03-19T03:03:35.805 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.805 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.805 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.805 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.805 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.805 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.805 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.805 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.805 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.805 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.805 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.805 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.805 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.805 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.805 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.806 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.806 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.806 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.806 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.806 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.806 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.806 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.806 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.806 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.806 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.806 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.806 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.806 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.806 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.806 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.807 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.807 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.807 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.807 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.807 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.807 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.807 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.807 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.807 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.807 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.807 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.807 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.807 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.807 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.807 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.807 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.808 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.808 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.808 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.808 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.808 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.808 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.808 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.808 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.808 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.808 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.808 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.808 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.808 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.808 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.808 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.809 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.809 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.809 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.809 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.809 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.809 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.809 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.809 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.809 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.809 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.809 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.809 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.809 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.809 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.809 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.810 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.810 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.810 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.810 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.810 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.810 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.810 INFO:teuthology.orchestra.run.smithi033.stdout: "organization": { 2024-03-19T03:03:35.810 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "organization", 2024-03-19T03:03:35.810 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.810 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.810 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.810 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.810 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.810 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.810 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.811 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.811 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.811 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.811 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.811 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.811 INFO:teuthology.orchestra.run.smithi033.stdout: "proxy": { 2024-03-19T03:03:35.811 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "proxy", 2024-03-19T03:03:35.811 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.811 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.811 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.811 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.811 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.811 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.811 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.811 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.812 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.812 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.812 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.812 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.812 INFO:teuthology.orchestra.run.smithi033.stdout: "url": { 2024-03-19T03:03:35.812 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "url", 2024-03-19T03:03:35.812 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.812 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.812 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.812 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-19T03:03:35.812 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.812 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.812 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.812 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.813 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.813 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.813 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.813 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.813 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.813 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.813 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.813 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "test_orchestrator", 2024-03-19T03:03:35.813 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.813 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.813 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.813 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.813 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.813 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.813 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.814 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.814 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.814 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.814 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.814 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.814 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.814 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.814 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.814 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.814 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.814 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.814 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.814 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.814 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.815 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.815 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.815 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.815 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.815 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.815 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.815 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.815 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.815 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.815 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.815 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.815 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.815 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.815 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.815 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.815 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.816 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.816 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.816 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.816 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.816 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.816 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.816 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.816 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.816 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.816 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.816 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.816 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.816 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.816 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.816 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.817 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.817 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.817 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.817 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.817 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.817 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.817 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.817 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.817 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.817 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.817 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.817 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.817 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.817 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.817 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.818 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.818 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.818 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.818 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.818 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.818 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.818 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.818 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.818 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.818 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "volumes", 2024-03-19T03:03:35.818 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.818 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.818 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.818 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.819 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.819 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.819 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.819 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.819 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.819 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.819 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.819 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.819 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.819 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.819 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.819 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.819 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.819 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.819 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.819 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.820 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.820 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.820 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.820 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.820 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.820 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.820 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.820 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.820 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.820 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.820 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.820 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.820 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.820 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.820 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.821 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.821 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.821 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.821 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.821 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.821 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.821 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.821 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.821 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.821 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.821 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.821 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.821 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.821 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.821 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.821 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.822 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.822 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.822 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.822 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.822 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.822 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.822 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.822 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.822 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.822 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.822 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.822 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.822 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.822 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.822 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.823 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.823 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.823 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.823 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.823 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.823 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.823 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.823 INFO:teuthology.orchestra.run.smithi033.stdout: "max_concurrent_clones": { 2024-03-19T03:03:35.823 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "max_concurrent_clones", 2024-03-19T03:03:35.823 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.823 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.823 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.823 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "4", 2024-03-19T03:03:35.823 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.823 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.824 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.824 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-19T03:03:35.824 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.824 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.824 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.824 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.824 INFO:teuthology.orchestra.run.smithi033.stdout: "periodic_async_work": { 2024-03-19T03:03:35.824 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "periodic_async_work", 2024-03-19T03:03:35.824 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.824 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.824 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.824 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.824 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.824 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.824 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.825 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Periodically check for async work", 2024-03-19T03:03:35.825 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.825 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.825 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.825 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.825 INFO:teuthology.orchestra.run.smithi033.stdout: "snapshot_clone_delay": { 2024-03-19T03:03:35.825 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "snapshot_clone_delay", 2024-03-19T03:03:35.825 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.825 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.825 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.825 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0", 2024-03-19T03:03:35.825 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.825 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.825 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.825 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-19T03:03:35.826 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.826 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.826 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.826 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.826 INFO:teuthology.orchestra.run.smithi033.stdout: "snapshot_clone_no_wait": { 2024-03-19T03:03:35.826 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "snapshot_clone_no_wait", 2024-03-19T03:03:35.826 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.826 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.826 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.826 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.826 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.826 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.826 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.826 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-19T03:03:35.827 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.827 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.827 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.827 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.827 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.827 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.827 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.827 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "zabbix", 2024-03-19T03:03:35.827 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.827 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.827 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.827 INFO:teuthology.orchestra.run.smithi033.stdout: "discovery_interval": { 2024-03-19T03:03:35.827 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "discovery_interval", 2024-03-19T03:03:35.827 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "uint", 2024-03-19T03:03:35.827 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.828 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.828 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "100", 2024-03-19T03:03:35.828 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.828 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.828 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.828 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.828 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.828 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.828 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.828 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.828 INFO:teuthology.orchestra.run.smithi033.stdout: "identifier": { 2024-03-19T03:03:35.828 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "identifier", 2024-03-19T03:03:35.828 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.828 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.828 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.828 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.829 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.829 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.829 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.829 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.829 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.829 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.829 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.829 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.829 INFO:teuthology.orchestra.run.smithi033.stdout: "interval": { 2024-03-19T03:03:35.829 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "interval", 2024-03-19T03:03:35.829 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:35.829 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.829 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.829 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "60", 2024-03-19T03:03:35.829 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.830 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.830 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.830 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.830 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.830 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.830 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.830 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.830 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.830 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.830 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.830 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.830 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.830 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.830 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.831 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.831 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.831 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.831 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.831 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.831 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.831 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.831 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.831 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.831 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.831 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.831 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.831 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.831 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.831 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.832 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.832 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.832 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.832 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.832 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.832 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.832 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.832 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.832 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.832 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.832 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.832 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.832 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.832 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.832 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.832 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.833 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.833 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.833 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.833 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.833 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.833 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.833 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.833 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.833 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.833 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.833 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.833 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.833 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.833 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.833 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.834 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.834 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.834 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.834 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.834 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.834 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.834 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.834 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.834 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.834 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.834 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.834 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.834 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.834 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.834 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.835 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.835 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.835 INFO:teuthology.orchestra.run.smithi033.stdout: "zabbix_host": { 2024-03-19T03:03:35.835 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "zabbix_host", 2024-03-19T03:03:35.835 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.835 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.835 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.835 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.835 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.835 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.835 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.835 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.835 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.835 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.836 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.836 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.836 INFO:teuthology.orchestra.run.smithi033.stdout: "zabbix_port": { 2024-03-19T03:03:35.836 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "zabbix_port", 2024-03-19T03:03:35.836 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.836 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.836 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.836 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10051", 2024-03-19T03:03:35.836 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.836 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.836 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.836 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.836 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.836 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.836 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.836 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.837 INFO:teuthology.orchestra.run.smithi033.stdout: "zabbix_sender": { 2024-03-19T03:03:35.837 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "zabbix_sender", 2024-03-19T03:03:35.837 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.837 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.837 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.837 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-19T03:03:35.837 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.837 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.837 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.837 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.837 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.837 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.837 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.837 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.837 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.838 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.838 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-03-19T03:03:35.838 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.838 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.838 INFO:teuthology.orchestra.run.smithi033.stdout: "gid": 15612, 2024-03-19T03:03:35.838 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "y", 2024-03-19T03:03:35.838 INFO:teuthology.orchestra.run.smithi033.stdout: "mgr_features": 4540701547738038271, 2024-03-19T03:03:35.838 INFO:teuthology.orchestra.run.smithi033.stdout: "available_modules": [ 2024-03-19T03:03:35.838 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.838 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "alerts", 2024-03-19T03:03:35.838 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.838 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.838 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.838 INFO:teuthology.orchestra.run.smithi033.stdout: "interval": { 2024-03-19T03:03:35.839 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "interval", 2024-03-19T03:03:35.839 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:35.839 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.839 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.839 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "60", 2024-03-19T03:03:35.839 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.839 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.839 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.839 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "How frequently to reexamine health status", 2024-03-19T03:03:35.839 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.839 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.839 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.839 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.839 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.840 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.840 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.840 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.840 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.840 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.840 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.840 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.840 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.840 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.840 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.840 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.840 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.840 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.840 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.840 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.841 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.841 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.841 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.841 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.841 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.841 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.841 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.841 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.841 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.841 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.841 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.841 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.841 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.841 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.841 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.842 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.842 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.842 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.842 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.842 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.842 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.842 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.842 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.842 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.842 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.842 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.842 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.842 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.842 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.842 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.843 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.843 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.843 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.843 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.843 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.843 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.843 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.843 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.843 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.843 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.843 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.843 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.843 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.843 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.843 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.844 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.844 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.844 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.844 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.844 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.844 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.844 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.844 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.844 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.844 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_destination": { 2024-03-19T03:03:35.844 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_destination", 2024-03-19T03:03:35.844 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.844 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.844 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.844 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.845 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.845 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.845 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.845 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Email address to send alerts to", 2024-03-19T03:03:35.845 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.845 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.845 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.845 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.845 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_from_name": { 2024-03-19T03:03:35.845 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_from_name", 2024-03-19T03:03:35.845 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.845 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.845 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.845 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "Ceph", 2024-03-19T03:03:35.845 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.845 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.846 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.846 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Email From: name", 2024-03-19T03:03:35.846 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.846 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.846 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.846 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.846 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_host": { 2024-03-19T03:03:35.846 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_host", 2024-03-19T03:03:35.846 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.846 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.846 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.846 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.846 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.846 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.846 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.847 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "SMTP server", 2024-03-19T03:03:35.847 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.847 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.847 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.847 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.847 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_password": { 2024-03-19T03:03:35.847 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_password", 2024-03-19T03:03:35.847 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.847 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.847 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.847 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.847 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.847 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.847 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.847 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Password to authenticate with", 2024-03-19T03:03:35.847 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.848 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.848 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.848 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.848 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_port": { 2024-03-19T03:03:35.848 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_port", 2024-03-19T03:03:35.848 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.848 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.848 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.848 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "465", 2024-03-19T03:03:35.848 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.848 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.848 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.848 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "SMTP port", 2024-03-19T03:03:35.848 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.848 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.848 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.849 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.849 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_sender": { 2024-03-19T03:03:35.849 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_sender", 2024-03-19T03:03:35.849 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.849 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.849 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.849 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.849 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.849 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.849 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.849 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "SMTP envelope sender", 2024-03-19T03:03:35.849 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.849 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.849 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.849 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.849 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_ssl": { 2024-03-19T03:03:35.850 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_ssl", 2024-03-19T03:03:35.850 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.850 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.850 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.850 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.850 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.850 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.850 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.850 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-19T03:03:35.850 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.850 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.850 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.850 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.850 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_user": { 2024-03-19T03:03:35.851 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_user", 2024-03-19T03:03:35.851 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.851 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.851 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.851 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.851 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.851 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.851 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.851 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "User to authenticate as", 2024-03-19T03:03:35.851 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.851 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.851 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.851 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.851 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.852 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.852 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.852 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "balancer", 2024-03-19T03:03:35.852 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.852 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.852 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.852 INFO:teuthology.orchestra.run.smithi033.stdout: "active": { 2024-03-19T03:03:35.852 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "active", 2024-03-19T03:03:35.852 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.852 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.852 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.852 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.852 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.852 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.852 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.853 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "automatically balance PGs across cluster", 2024-03-19T03:03:35.853 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.853 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.853 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.853 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.853 INFO:teuthology.orchestra.run.smithi033.stdout: "begin_time": { 2024-03-19T03:03:35.853 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "begin_time", 2024-03-19T03:03:35.853 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.853 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.853 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.853 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0000", 2024-03-19T03:03:35.853 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.853 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.853 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.853 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "beginning time of day to automatically balance", 2024-03-19T03:03:35.853 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-19T03:03:35.854 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.854 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.854 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.854 INFO:teuthology.orchestra.run.smithi033.stdout: "begin_weekday": { 2024-03-19T03:03:35.854 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "begin_weekday", 2024-03-19T03:03:35.854 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "uint", 2024-03-19T03:03:35.854 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.854 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.854 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0", 2024-03-19T03:03:35.854 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "0", 2024-03-19T03:03:35.854 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "6", 2024-03-19T03:03:35.854 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.854 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-19T03:03:35.854 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-19T03:03:35.855 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.855 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.855 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.855 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_compat_max_iterations": { 2024-03-19T03:03:35.855 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "crush_compat_max_iterations", 2024-03-19T03:03:35.855 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "uint", 2024-03-19T03:03:35.855 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.855 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.855 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "25", 2024-03-19T03:03:35.855 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "1", 2024-03-19T03:03:35.855 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "250", 2024-03-19T03:03:35.855 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.855 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-19T03:03:35.855 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.855 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.856 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.856 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.856 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_compat_metrics": { 2024-03-19T03:03:35.856 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "crush_compat_metrics", 2024-03-19T03:03:35.856 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.856 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.856 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.856 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "pgs,objects,bytes", 2024-03-19T03:03:35.856 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.856 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.856 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.856 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-19T03:03:35.856 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-19T03:03:35.856 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.856 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.857 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.857 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_compat_step": { 2024-03-19T03:03:35.857 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "crush_compat_step", 2024-03-19T03:03:35.857 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-03-19T03:03:35.857 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.857 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.857 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0.5", 2024-03-19T03:03:35.857 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "0.001", 2024-03-19T03:03:35.857 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "0.999", 2024-03-19T03:03:35.857 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.857 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "aggressiveness of optimization", 2024-03-19T03:03:35.857 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-19T03:03:35.857 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.857 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.857 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.857 INFO:teuthology.orchestra.run.smithi033.stdout: "end_time": { 2024-03-19T03:03:35.858 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "end_time", 2024-03-19T03:03:35.858 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.858 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.858 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.858 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "2359", 2024-03-19T03:03:35.858 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.858 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.858 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.858 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "ending time of day to automatically balance", 2024-03-19T03:03:35.858 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-19T03:03:35.858 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.858 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.858 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.858 INFO:teuthology.orchestra.run.smithi033.stdout: "end_weekday": { 2024-03-19T03:03:35.858 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "end_weekday", 2024-03-19T03:03:35.859 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "uint", 2024-03-19T03:03:35.859 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.859 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.859 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0", 2024-03-19T03:03:35.859 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "0", 2024-03-19T03:03:35.859 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "6", 2024-03-19T03:03:35.859 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.859 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-19T03:03:35.859 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-19T03:03:35.859 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.859 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.859 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.859 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.859 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.859 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.860 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.860 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.860 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.860 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.860 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.860 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.860 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.860 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.860 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.860 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.860 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.860 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.861 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.861 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.861 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.861 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.861 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.861 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.861 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.861 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.861 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.861 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.861 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.861 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.861 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.861 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.862 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.862 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.862 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.862 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.862 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.862 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.862 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.862 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.862 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.862 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.862 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.862 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.862 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.862 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.863 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.863 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.863 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.863 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.863 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.863 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.863 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.863 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.863 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.863 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.863 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.863 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.863 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.863 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.864 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.864 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.864 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.864 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.864 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.864 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.864 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.864 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.864 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.864 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.864 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.864 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.864 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.864 INFO:teuthology.orchestra.run.smithi033.stdout: "min_score": { 2024-03-19T03:03:35.865 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "min_score", 2024-03-19T03:03:35.865 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-03-19T03:03:35.865 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.865 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.865 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0", 2024-03-19T03:03:35.865 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.865 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.865 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.865 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-19T03:03:35.865 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.865 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.865 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.865 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.865 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": { 2024-03-19T03:03:35.865 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mode", 2024-03-19T03:03:35.865 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.866 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.866 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.866 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "upmap", 2024-03-19T03:03:35.866 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.866 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.866 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.866 INFO:teuthology.orchestra.run.smithi033.stdout: "crush-compat", 2024-03-19T03:03:35.866 INFO:teuthology.orchestra.run.smithi033.stdout: "none", 2024-03-19T03:03:35.866 INFO:teuthology.orchestra.run.smithi033.stdout: "read", 2024-03-19T03:03:35.866 INFO:teuthology.orchestra.run.smithi033.stdout: "upmap", 2024-03-19T03:03:35.866 INFO:teuthology.orchestra.run.smithi033.stdout: "upmap-read" 2024-03-19T03:03:35.866 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.866 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Balancer mode", 2024-03-19T03:03:35.866 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.867 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.867 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.867 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.867 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_ids": { 2024-03-19T03:03:35.867 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "pool_ids", 2024-03-19T03:03:35.867 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.867 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.867 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.867 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.867 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.867 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.867 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.867 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-19T03:03:35.867 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.867 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.868 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.868 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.868 INFO:teuthology.orchestra.run.smithi033.stdout: "sleep_interval": { 2024-03-19T03:03:35.868 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "sleep_interval", 2024-03-19T03:03:35.868 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:35.868 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.868 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.868 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "60", 2024-03-19T03:03:35.868 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.868 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.868 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.868 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-19T03:03:35.868 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.868 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.869 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.869 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.869 INFO:teuthology.orchestra.run.smithi033.stdout: "upmap_max_deviation": { 2024-03-19T03:03:35.869 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "upmap_max_deviation", 2024-03-19T03:03:35.869 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.869 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.869 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.869 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5", 2024-03-19T03:03:35.869 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "1", 2024-03-19T03:03:35.869 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.869 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.869 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-19T03:03:35.869 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-19T03:03:35.869 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.869 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.870 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.870 INFO:teuthology.orchestra.run.smithi033.stdout: "upmap_max_optimizations": { 2024-03-19T03:03:35.870 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "upmap_max_optimizations", 2024-03-19T03:03:35.870 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "uint", 2024-03-19T03:03:35.870 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.870 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.870 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-03-19T03:03:35.870 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.870 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.870 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.870 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-19T03:03:35.870 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.870 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.870 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.871 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.871 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.871 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.871 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.871 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "cephadm", 2024-03-19T03:03:35.871 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.871 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.871 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.871 INFO:teuthology.orchestra.run.smithi033.stdout: "agent_down_multiplier": { 2024-03-19T03:03:35.871 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "agent_down_multiplier", 2024-03-19T03:03:35.871 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-03-19T03:03:35.871 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.871 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.871 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "3.0", 2024-03-19T03:03:35.871 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.871 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.872 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.872 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-19T03:03:35.872 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.872 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.872 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.872 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.872 INFO:teuthology.orchestra.run.smithi033.stdout: "agent_refresh_rate": { 2024-03-19T03:03:35.872 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "agent_refresh_rate", 2024-03-19T03:03:35.872 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:35.872 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.872 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.872 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "20", 2024-03-19T03:03:35.872 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.872 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.872 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.873 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-19T03:03:35.873 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.873 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.873 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.873 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.873 INFO:teuthology.orchestra.run.smithi033.stdout: "agent_starting_port": { 2024-03-19T03:03:35.873 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "agent_starting_port", 2024-03-19T03:03:35.873 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.873 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.873 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.873 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "4721", 2024-03-19T03:03:35.873 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.873 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.873 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.873 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-19T03:03:35.874 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.874 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.874 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.874 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.874 INFO:teuthology.orchestra.run.smithi033.stdout: "allow_ptrace": { 2024-03-19T03:03:35.874 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "allow_ptrace", 2024-03-19T03:03:35.874 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.874 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.874 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.874 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.874 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.874 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.874 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.874 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-19T03:03:35.875 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-19T03:03:35.875 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.875 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.875 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.875 INFO:teuthology.orchestra.run.smithi033.stdout: "autotune_interval": { 2024-03-19T03:03:35.875 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "autotune_interval", 2024-03-19T03:03:35.875 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:35.875 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.875 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.875 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "600", 2024-03-19T03:03:35.875 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.875 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.875 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.875 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-19T03:03:35.876 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.876 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.876 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.876 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.876 INFO:teuthology.orchestra.run.smithi033.stdout: "autotune_memory_target_ratio": { 2024-03-19T03:03:35.876 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "autotune_memory_target_ratio", 2024-03-19T03:03:35.876 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-03-19T03:03:35.876 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.876 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.876 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0.7", 2024-03-19T03:03:35.876 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.876 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.876 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.876 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-19T03:03:35.876 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.877 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.877 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.877 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.877 INFO:teuthology.orchestra.run.smithi033.stdout: "cephadm_log_destination": { 2024-03-19T03:03:35.877 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "cephadm_log_destination", 2024-03-19T03:03:35.877 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.877 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.877 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.877 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.877 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.877 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.877 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.877 INFO:teuthology.orchestra.run.smithi033.stdout: "file", 2024-03-19T03:03:35.877 INFO:teuthology.orchestra.run.smithi033.stdout: "file,syslog", 2024-03-19T03:03:35.877 INFO:teuthology.orchestra.run.smithi033.stdout: "syslog" 2024-03-19T03:03:35.878 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.878 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-19T03:03:35.878 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.878 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.878 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.878 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.878 INFO:teuthology.orchestra.run.smithi033.stdout: "cgroups_split": { 2024-03-19T03:03:35.878 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "cgroups_split", 2024-03-19T03:03:35.878 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.878 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.878 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.878 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.878 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.878 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.878 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.879 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-19T03:03:35.879 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.879 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.879 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.879 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.879 INFO:teuthology.orchestra.run.smithi033.stdout: "config_checks_enabled": { 2024-03-19T03:03:35.879 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "config_checks_enabled", 2024-03-19T03:03:35.879 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.879 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.879 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.879 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.879 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.879 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.879 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.880 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-19T03:03:35.880 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.880 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.880 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.880 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.880 INFO:teuthology.orchestra.run.smithi033.stdout: "config_dashboard": { 2024-03-19T03:03:35.880 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "config_dashboard", 2024-03-19T03:03:35.880 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.880 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.880 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.880 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.880 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.880 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.880 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.881 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-19T03:03:35.881 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.881 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.881 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.881 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.881 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_alertmanager": { 2024-03-19T03:03:35.881 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_alertmanager", 2024-03-19T03:03:35.881 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.881 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.881 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.881 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-19T03:03:35.881 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.881 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.881 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.881 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Prometheus container image", 2024-03-19T03:03:35.882 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.882 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.882 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.882 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.882 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_base": { 2024-03-19T03:03:35.882 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_base", 2024-03-19T03:03:35.882 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.882 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.882 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.882 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-19T03:03:35.882 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.882 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.882 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.882 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Container image name, without the tag", 2024-03-19T03:03:35.883 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.883 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.883 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.883 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.883 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_elasticsearch": { 2024-03-19T03:03:35.883 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_elasticsearch", 2024-03-19T03:03:35.883 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.883 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.883 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.883 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-19T03:03:35.883 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.883 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.883 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.883 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "elasticsearch container image", 2024-03-19T03:03:35.883 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.884 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.884 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.884 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.884 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_grafana": { 2024-03-19T03:03:35.884 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_grafana", 2024-03-19T03:03:35.884 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.884 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.884 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.884 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-19T03:03:35.884 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.884 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.884 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.884 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Prometheus container image", 2024-03-19T03:03:35.884 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.884 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.885 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.885 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.885 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_haproxy": { 2024-03-19T03:03:35.885 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_haproxy", 2024-03-19T03:03:35.885 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.885 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.885 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.885 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-19T03:03:35.885 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.885 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.885 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.885 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "HAproxy container image", 2024-03-19T03:03:35.885 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.885 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.886 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.886 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.886 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_jaeger_agent": { 2024-03-19T03:03:35.886 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_jaeger_agent", 2024-03-19T03:03:35.886 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.886 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.886 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.886 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-19T03:03:35.886 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.886 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.886 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.886 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Jaeger agent container image", 2024-03-19T03:03:35.887 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.887 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.887 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.887 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.887 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_jaeger_collector": { 2024-03-19T03:03:35.887 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_jaeger_collector", 2024-03-19T03:03:35.887 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.887 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.887 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.887 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-19T03:03:35.887 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.887 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.887 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.887 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Jaeger collector container image", 2024-03-19T03:03:35.887 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.888 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.888 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.888 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.888 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_jaeger_query": { 2024-03-19T03:03:35.888 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_jaeger_query", 2024-03-19T03:03:35.888 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.888 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.888 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.888 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-19T03:03:35.888 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.888 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.888 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.888 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Jaeger query container image", 2024-03-19T03:03:35.888 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.888 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.889 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.889 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.889 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_keepalived": { 2024-03-19T03:03:35.889 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_keepalived", 2024-03-19T03:03:35.889 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.889 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.889 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.889 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-19T03:03:35.889 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.889 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.889 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.889 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Keepalived container image", 2024-03-19T03:03:35.889 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.889 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.889 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.889 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.890 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_loki": { 2024-03-19T03:03:35.890 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_loki", 2024-03-19T03:03:35.890 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.890 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.890 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.890 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-19T03:03:35.890 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.890 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.890 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.890 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Loki container image", 2024-03-19T03:03:35.890 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.890 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.890 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.890 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.891 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_node_exporter": { 2024-03-19T03:03:35.891 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_node_exporter", 2024-03-19T03:03:35.891 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.891 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.891 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.891 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-19T03:03:35.891 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.891 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.891 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.891 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Prometheus container image", 2024-03-19T03:03:35.891 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.891 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.891 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.891 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.891 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_nvmeof": { 2024-03-19T03:03:35.892 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_nvmeof", 2024-03-19T03:03:35.892 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.892 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.892 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.892 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-19T03:03:35.892 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.892 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.892 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.892 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Nvme-of container image", 2024-03-19T03:03:35.892 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.892 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.892 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.892 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.892 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_prometheus": { 2024-03-19T03:03:35.892 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_prometheus", 2024-03-19T03:03:35.893 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.893 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.893 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.893 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-19T03:03:35.893 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.893 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.893 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.893 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Prometheus container image", 2024-03-19T03:03:35.893 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.893 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.893 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.893 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.893 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_promtail": { 2024-03-19T03:03:35.893 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_promtail", 2024-03-19T03:03:35.893 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.893 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.894 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.894 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-19T03:03:35.894 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.894 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.894 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.894 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Promtail container image", 2024-03-19T03:03:35.894 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.894 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.894 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.894 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.894 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_snmp_gateway": { 2024-03-19T03:03:35.894 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_snmp_gateway", 2024-03-19T03:03:35.894 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.894 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.895 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.895 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-19T03:03:35.895 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.895 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.895 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.895 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "SNMP Gateway container image", 2024-03-19T03:03:35.895 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.895 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.895 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.895 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.895 INFO:teuthology.orchestra.run.smithi033.stdout: "container_init": { 2024-03-19T03:03:35.895 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_init", 2024-03-19T03:03:35.895 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.895 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.895 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.896 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.896 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.896 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.896 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.896 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Run podman/docker with `--init`", 2024-03-19T03:03:35.896 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.896 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.896 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.896 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.896 INFO:teuthology.orchestra.run.smithi033.stdout: "daemon_cache_timeout": { 2024-03-19T03:03:35.896 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "daemon_cache_timeout", 2024-03-19T03:03:35.896 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:35.896 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.896 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.896 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "600", 2024-03-19T03:03:35.897 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.897 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.897 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.897 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-19T03:03:35.897 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.897 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.897 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.897 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.897 INFO:teuthology.orchestra.run.smithi033.stdout: "default_cephadm_command_timeout": { 2024-03-19T03:03:35.897 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "default_cephadm_command_timeout", 2024-03-19T03:03:35.897 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.897 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.897 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.897 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "900", 2024-03-19T03:03:35.897 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.898 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.898 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.898 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-19T03:03:35.898 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.898 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.898 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.898 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.898 INFO:teuthology.orchestra.run.smithi033.stdout: "default_registry": { 2024-03-19T03:03:35.898 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "default_registry", 2024-03-19T03:03:35.898 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.898 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.898 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.898 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "docker.io", 2024-03-19T03:03:35.898 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.899 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.899 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.899 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-19T03:03:35.899 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.899 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.899 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.899 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.899 INFO:teuthology.orchestra.run.smithi033.stdout: "device_cache_timeout": { 2024-03-19T03:03:35.899 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "device_cache_timeout", 2024-03-19T03:03:35.899 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:35.899 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.899 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.899 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "1800", 2024-03-19T03:03:35.899 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.899 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.900 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.900 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "seconds to cache device inventory", 2024-03-19T03:03:35.900 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.900 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.900 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.900 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.900 INFO:teuthology.orchestra.run.smithi033.stdout: "device_enhanced_scan": { 2024-03-19T03:03:35.900 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "device_enhanced_scan", 2024-03-19T03:03:35.900 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.900 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.900 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.900 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.900 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.900 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.901 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.901 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-19T03:03:35.901 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.901 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.901 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.901 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.901 INFO:teuthology.orchestra.run.smithi033.stdout: "facts_cache_timeout": { 2024-03-19T03:03:35.901 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "facts_cache_timeout", 2024-03-19T03:03:35.901 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:35.901 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.901 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.901 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "60", 2024-03-19T03:03:35.901 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.901 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.901 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.902 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "seconds to cache host facts data", 2024-03-19T03:03:35.902 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.902 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.902 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.902 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.902 INFO:teuthology.orchestra.run.smithi033.stdout: "host_check_interval": { 2024-03-19T03:03:35.902 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "host_check_interval", 2024-03-19T03:03:35.902 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:35.902 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.902 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.902 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "600", 2024-03-19T03:03:35.902 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.902 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.903 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.903 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how frequently to perform a host check", 2024-03-19T03:03:35.903 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.903 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.903 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.903 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.903 INFO:teuthology.orchestra.run.smithi033.stdout: "hw_monitoring": { 2024-03-19T03:03:35.903 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "hw_monitoring", 2024-03-19T03:03:35.903 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.903 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.903 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.903 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.903 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.903 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.903 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.904 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-19T03:03:35.904 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.904 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.904 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.904 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.904 INFO:teuthology.orchestra.run.smithi033.stdout: "inventory_list_all": { 2024-03-19T03:03:35.904 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "inventory_list_all", 2024-03-19T03:03:35.904 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.904 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.904 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.904 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.904 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.904 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.904 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.904 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-19T03:03:35.905 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.905 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.905 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.905 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.905 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.905 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.905 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.905 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.905 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.905 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.905 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.905 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.905 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.905 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.905 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.906 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.906 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.906 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.906 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.906 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.906 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.906 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.906 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.906 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.906 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.906 INFO:teuthology.orchestra.run.smithi033.stdout: "log_refresh_metadata": { 2024-03-19T03:03:35.906 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_refresh_metadata", 2024-03-19T03:03:35.906 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.906 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.907 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.907 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.907 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.907 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.907 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.907 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-19T03:03:35.907 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.907 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.907 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.907 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.907 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.907 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.907 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.907 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.907 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.908 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.908 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.908 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.908 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.908 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-19T03:03:35.908 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.908 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.908 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.908 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.908 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.908 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.908 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.908 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.908 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.908 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.909 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.909 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.909 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.909 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.909 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.909 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.909 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.909 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.909 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.909 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.909 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.909 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.909 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.909 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.909 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.910 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.910 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.910 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.910 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.910 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.910 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.910 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.910 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.910 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.910 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.910 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.910 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.910 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.911 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.911 INFO:teuthology.orchestra.run.smithi033.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-19T03:03:35.911 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-19T03:03:35.911 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.911 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.911 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.911 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.911 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.911 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.911 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.911 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-19T03:03:35.911 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.911 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.911 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.911 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.912 INFO:teuthology.orchestra.run.smithi033.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-19T03:03:35.912 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-19T03:03:35.912 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.912 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.912 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.912 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "*", 2024-03-19T03:03:35.912 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.912 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.912 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.912 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-19T03:03:35.912 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.912 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.912 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.912 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.912 INFO:teuthology.orchestra.run.smithi033.stdout: "max_count_per_host": { 2024-03-19T03:03:35.913 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "max_count_per_host", 2024-03-19T03:03:35.913 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.913 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.913 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.913 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-03-19T03:03:35.913 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.913 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.913 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.913 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "max number of daemons per service per host", 2024-03-19T03:03:35.913 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.913 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.913 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.913 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.913 INFO:teuthology.orchestra.run.smithi033.stdout: "max_osd_draining_count": { 2024-03-19T03:03:35.913 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "max_osd_draining_count", 2024-03-19T03:03:35.914 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.914 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.914 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.914 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-03-19T03:03:35.914 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.914 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.914 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.914 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-19T03:03:35.914 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.914 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.914 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.914 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.914 INFO:teuthology.orchestra.run.smithi033.stdout: "migration_current": { 2024-03-19T03:03:35.914 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "migration_current", 2024-03-19T03:03:35.915 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.915 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.915 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.915 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.915 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.915 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.915 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.915 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "internal - do not modify", 2024-03-19T03:03:35.915 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.915 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.915 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.915 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.915 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": { 2024-03-19T03:03:35.915 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mode", 2024-03-19T03:03:35.915 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.915 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.916 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.916 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "root", 2024-03-19T03:03:35.916 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.916 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.916 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.916 INFO:teuthology.orchestra.run.smithi033.stdout: "cephadm-package", 2024-03-19T03:03:35.916 INFO:teuthology.orchestra.run.smithi033.stdout: "root" 2024-03-19T03:03:35.916 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.916 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "mode for remote execution of cephadm", 2024-03-19T03:03:35.916 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.916 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.916 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.916 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.916 INFO:teuthology.orchestra.run.smithi033.stdout: "oob_default_addr": { 2024-03-19T03:03:35.916 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "oob_default_addr", 2024-03-19T03:03:35.916 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.916 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.917 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.917 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "169.254.1.1", 2024-03-19T03:03:35.917 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.917 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.917 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.917 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-19T03:03:35.917 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.917 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.917 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.917 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.917 INFO:teuthology.orchestra.run.smithi033.stdout: "prometheus_alerts_path": { 2024-03-19T03:03:35.917 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "prometheus_alerts_path", 2024-03-19T03:03:35.917 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.917 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.917 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.917 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-19T03:03:35.918 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.918 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.918 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.918 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-19T03:03:35.918 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.918 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.918 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.918 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.918 INFO:teuthology.orchestra.run.smithi033.stdout: "registry_insecure": { 2024-03-19T03:03:35.918 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "registry_insecure", 2024-03-19T03:03:35.918 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.918 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.918 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.918 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.918 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.918 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.919 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.919 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-19T03:03:35.919 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.919 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.919 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.919 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.919 INFO:teuthology.orchestra.run.smithi033.stdout: "registry_password": { 2024-03-19T03:03:35.919 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "registry_password", 2024-03-19T03:03:35.919 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.919 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.919 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.919 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.919 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.919 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.919 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.920 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-19T03:03:35.920 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.920 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.920 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.920 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.920 INFO:teuthology.orchestra.run.smithi033.stdout: "registry_url": { 2024-03-19T03:03:35.920 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "registry_url", 2024-03-19T03:03:35.920 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.920 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.920 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.920 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.920 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.920 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.920 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.920 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-19T03:03:35.920 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.921 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.921 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.921 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.921 INFO:teuthology.orchestra.run.smithi033.stdout: "registry_username": { 2024-03-19T03:03:35.921 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "registry_username", 2024-03-19T03:03:35.921 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.921 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.921 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.921 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.921 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.921 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.921 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.921 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-19T03:03:35.921 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.921 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.922 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.922 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.922 INFO:teuthology.orchestra.run.smithi033.stdout: "secure_monitoring_stack": { 2024-03-19T03:03:35.922 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "secure_monitoring_stack", 2024-03-19T03:03:35.922 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.922 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.922 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.922 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.922 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.922 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.922 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.922 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-19T03:03:35.922 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.922 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.922 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.923 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.923 INFO:teuthology.orchestra.run.smithi033.stdout: "service_discovery_port": { 2024-03-19T03:03:35.923 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "service_discovery_port", 2024-03-19T03:03:35.923 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.923 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.923 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.923 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "8765", 2024-03-19T03:03:35.923 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.923 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.923 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.923 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "cephadm service discovery port", 2024-03-19T03:03:35.923 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.923 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.923 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.924 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.924 INFO:teuthology.orchestra.run.smithi033.stdout: "ssh_config_file": { 2024-03-19T03:03:35.924 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ssh_config_file", 2024-03-19T03:03:35.924 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.924 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.924 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.924 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.924 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.924 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.924 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.924 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-19T03:03:35.924 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.924 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.924 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.924 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.924 INFO:teuthology.orchestra.run.smithi033.stdout: "use_agent": { 2024-03-19T03:03:35.925 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "use_agent", 2024-03-19T03:03:35.925 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.925 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.925 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.925 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.925 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.925 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.925 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.925 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-19T03:03:35.925 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.925 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.925 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.925 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.925 INFO:teuthology.orchestra.run.smithi033.stdout: "use_repo_digest": { 2024-03-19T03:03:35.926 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "use_repo_digest", 2024-03-19T03:03:35.926 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.926 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.926 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.926 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.926 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.926 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.926 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.926 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-19T03:03:35.926 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.926 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.926 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.926 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.926 INFO:teuthology.orchestra.run.smithi033.stdout: "warn_on_failed_host_check": { 2024-03-19T03:03:35.927 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "warn_on_failed_host_check", 2024-03-19T03:03:35.927 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.927 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.927 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.927 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.927 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.927 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.927 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.927 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "raise a health warning if the host check fails", 2024-03-19T03:03:35.927 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.927 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.927 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.927 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.927 INFO:teuthology.orchestra.run.smithi033.stdout: "warn_on_stray_daemons": { 2024-03-19T03:03:35.927 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "warn_on_stray_daemons", 2024-03-19T03:03:35.928 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.928 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.928 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.928 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.928 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.928 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.928 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.928 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-19T03:03:35.928 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.928 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.928 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.928 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.928 INFO:teuthology.orchestra.run.smithi033.stdout: "warn_on_stray_hosts": { 2024-03-19T03:03:35.928 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "warn_on_stray_hosts", 2024-03-19T03:03:35.928 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.928 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.929 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.929 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.929 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.929 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.929 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.929 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-19T03:03:35.929 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.929 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.929 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.929 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.929 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.929 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.929 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.929 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "crash", 2024-03-19T03:03:35.929 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.929 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.930 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.930 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.930 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.930 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.930 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.930 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.930 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.930 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.930 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.930 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.930 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.930 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.930 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.930 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.931 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.931 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.931 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.931 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.931 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.931 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.931 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.931 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.931 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.931 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.931 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.931 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.931 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.931 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.931 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.931 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.932 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.932 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.932 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.932 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.932 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.932 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.932 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.932 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.932 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.932 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.932 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.932 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.932 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.932 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.932 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.933 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.933 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.933 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.933 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.933 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.933 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.933 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.933 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.933 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.933 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.933 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.933 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.933 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.933 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.934 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.934 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.934 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.934 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.934 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.934 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.934 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.934 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.934 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.934 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.934 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.934 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.934 INFO:teuthology.orchestra.run.smithi033.stdout: "retain_interval": { 2024-03-19T03:03:35.934 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "retain_interval", 2024-03-19T03:03:35.934 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:35.935 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.935 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.935 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "31536000", 2024-03-19T03:03:35.935 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.935 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.935 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.935 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-19T03:03:35.935 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.935 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.935 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.935 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.935 INFO:teuthology.orchestra.run.smithi033.stdout: "warn_recent_interval": { 2024-03-19T03:03:35.935 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "warn_recent_interval", 2024-03-19T03:03:35.935 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:35.936 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.936 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.936 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "1209600", 2024-03-19T03:03:35.936 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.936 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.936 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.936 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-19T03:03:35.936 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.936 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.936 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.936 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.936 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.936 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.936 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.936 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "dashboard", 2024-03-19T03:03:35.937 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.937 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.937 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.937 INFO:teuthology.orchestra.run.smithi033.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-19T03:03:35.937 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-19T03:03:35.937 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.937 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.937 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.937 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-03-19T03:03:35.937 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.937 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.937 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.937 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.937 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.937 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.938 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.938 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.938 INFO:teuthology.orchestra.run.smithi033.stdout: "ALERTMANAGER_API_HOST": { 2024-03-19T03:03:35.938 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-19T03:03:35.938 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.938 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.938 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.938 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.938 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.938 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.938 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.938 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.938 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.938 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.938 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.939 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.939 INFO:teuthology.orchestra.run.smithi033.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-19T03:03:35.939 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-19T03:03:35.939 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.939 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.939 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.939 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.939 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.939 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.939 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.939 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.939 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.939 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.939 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.940 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.940 INFO:teuthology.orchestra.run.smithi033.stdout: "AUDIT_API_ENABLED": { 2024-03-19T03:03:35.940 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "AUDIT_API_ENABLED", 2024-03-19T03:03:35.940 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.940 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.940 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.940 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.940 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.940 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.940 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.940 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.940 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.940 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.940 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.940 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.940 INFO:teuthology.orchestra.run.smithi033.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-19T03:03:35.941 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-19T03:03:35.941 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.941 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.941 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.941 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.941 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.941 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.941 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.941 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.941 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.941 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.941 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.941 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.941 INFO:teuthology.orchestra.run.smithi033.stdout: "ENABLE_BROWSABLE_API": { 2024-03-19T03:03:35.941 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-19T03:03:35.942 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.942 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.942 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.942 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.942 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.942 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.942 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.942 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.942 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.942 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.942 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.942 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.942 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-19T03:03:35.942 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-19T03:03:35.942 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.943 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.943 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.943 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.943 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.943 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.943 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.943 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.943 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.943 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.943 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.943 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.943 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-19T03:03:35.943 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-19T03:03:35.943 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.943 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.944 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.944 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.944 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.944 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.944 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.944 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.944 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.944 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.944 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.944 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.944 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-19T03:03:35.944 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-19T03:03:35.944 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.944 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.944 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.944 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.945 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.945 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.945 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.945 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.945 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.945 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.945 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.945 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.945 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-19T03:03:35.945 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-19T03:03:35.945 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.945 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.945 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.945 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.945 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.946 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.946 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.946 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.946 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.946 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.946 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.946 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.946 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-19T03:03:35.946 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-19T03:03:35.946 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.946 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.946 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.946 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.946 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.947 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.947 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.947 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.947 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.947 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.947 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.947 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.947 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-19T03:03:35.947 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-19T03:03:35.947 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.947 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.947 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.947 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.947 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.947 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.948 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.948 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.948 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.948 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.948 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.948 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.948 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-19T03:03:35.948 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-19T03:03:35.948 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.948 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.948 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.948 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.948 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.948 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.948 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.949 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.949 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.949 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.949 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.949 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.949 INFO:teuthology.orchestra.run.smithi033.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-19T03:03:35.949 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-19T03:03:35.949 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.949 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.949 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.949 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.949 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.949 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.949 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.949 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.950 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.950 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.950 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.950 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.950 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_API_PASSWORD": { 2024-03-19T03:03:35.950 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-19T03:03:35.950 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.950 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.950 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.950 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "admin", 2024-03-19T03:03:35.950 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.950 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.950 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.950 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.951 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.951 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.951 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.951 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.951 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-19T03:03:35.951 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-19T03:03:35.951 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.951 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.951 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.951 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.951 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.951 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.951 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.951 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.951 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.952 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.952 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.952 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.952 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_API_URL": { 2024-03-19T03:03:35.952 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_API_URL", 2024-03-19T03:03:35.952 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.952 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.952 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.952 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.952 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.952 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.952 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.952 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.952 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.952 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.953 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.953 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.953 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_API_USERNAME": { 2024-03-19T03:03:35.953 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-19T03:03:35.953 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.953 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.953 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.953 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "admin", 2024-03-19T03:03:35.953 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.953 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.953 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.953 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.953 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.953 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.953 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.954 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.954 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-19T03:03:35.954 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-19T03:03:35.954 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.954 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.954 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.954 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.954 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.954 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.954 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.954 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.954 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.954 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.954 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.955 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.955 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-19T03:03:35.955 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-19T03:03:35.955 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.955 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.955 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.955 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.955 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.955 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.955 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.955 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.955 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.955 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.955 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.955 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.956 INFO:teuthology.orchestra.run.smithi033.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-19T03:03:35.956 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-19T03:03:35.956 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.956 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.956 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.956 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.956 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.956 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.956 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.956 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.956 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.956 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.956 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.956 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.956 INFO:teuthology.orchestra.run.smithi033.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-19T03:03:35.957 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-19T03:03:35.957 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.957 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.957 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.957 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.957 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.957 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.957 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.957 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.957 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.957 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.957 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.957 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.957 INFO:teuthology.orchestra.run.smithi033.stdout: "MULTICLUSTER_CONFIG": { 2024-03-19T03:03:35.957 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-19T03:03:35.958 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.958 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.958 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.958 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "{}", 2024-03-19T03:03:35.958 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.958 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.958 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.958 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.958 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.958 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.958 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.958 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.958 INFO:teuthology.orchestra.run.smithi033.stdout: "PROMETHEUS_API_HOST": { 2024-03-19T03:03:35.958 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-19T03:03:35.959 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.959 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.959 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.959 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.959 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.959 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.959 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.959 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.959 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.959 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.959 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.959 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.959 INFO:teuthology.orchestra.run.smithi033.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-19T03:03:35.959 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-19T03:03:35.960 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.960 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.960 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.960 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.960 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.960 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.960 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.960 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.960 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.960 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.960 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.960 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.960 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-19T03:03:35.960 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-19T03:03:35.961 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.961 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.961 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.961 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.961 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.961 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.961 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.961 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.961 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.961 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.961 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.961 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.961 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-19T03:03:35.961 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-19T03:03:35.961 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.962 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.962 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.962 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.962 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.962 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.962 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.962 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.962 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.962 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.962 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.962 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.962 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-19T03:03:35.962 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-19T03:03:35.963 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.963 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.963 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.963 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.963 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.963 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.963 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.963 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.963 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.963 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.963 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.963 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.963 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-19T03:03:35.963 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-19T03:03:35.964 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.964 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.964 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.964 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.964 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.964 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.964 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.964 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.964 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.964 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.964 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.964 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.964 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-19T03:03:35.964 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-19T03:03:35.964 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.965 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.965 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.965 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.965 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.965 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.965 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.965 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.965 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.965 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.965 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.965 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.965 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-19T03:03:35.965 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-19T03:03:35.965 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.965 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.966 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.966 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.966 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.966 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.966 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.966 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.966 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.966 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.966 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.966 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.966 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-19T03:03:35.966 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-19T03:03:35.966 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.967 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.967 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.967 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.967 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.967 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.967 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.967 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.967 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.967 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.967 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.967 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.967 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_ENABLED": { 2024-03-19T03:03:35.967 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-19T03:03:35.967 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.967 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.967 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.968 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.968 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.968 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.968 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.968 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.968 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.968 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.968 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.968 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.968 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-19T03:03:35.968 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-19T03:03:35.968 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.968 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.968 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.968 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-19T03:03:35.969 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.969 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.969 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.969 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.969 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.969 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.969 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.969 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.969 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-19T03:03:35.969 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-19T03:03:35.969 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.969 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.969 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.969 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-03-19T03:03:35.969 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.969 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.970 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.970 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.970 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.970 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.970 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.970 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.970 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-19T03:03:35.970 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-19T03:03:35.970 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.970 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.970 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.970 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "8", 2024-03-19T03:03:35.970 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.971 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.971 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.971 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.971 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.971 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.971 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.971 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.971 INFO:teuthology.orchestra.run.smithi033.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-19T03:03:35.971 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-19T03:03:35.971 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.971 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.971 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.971 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "45", 2024-03-19T03:03:35.971 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.971 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.972 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.972 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.972 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.972 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.972 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.972 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.972 INFO:teuthology.orchestra.run.smithi033.stdout: "RGW_API_ACCESS_KEY": { 2024-03-19T03:03:35.972 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-19T03:03:35.972 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.972 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.972 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.972 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.972 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.972 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.972 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.973 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.973 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.973 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.973 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.973 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.973 INFO:teuthology.orchestra.run.smithi033.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-19T03:03:35.973 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-19T03:03:35.973 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.973 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.973 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.973 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "admin", 2024-03-19T03:03:35.973 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.973 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.973 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.973 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.974 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.974 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.974 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.974 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.974 INFO:teuthology.orchestra.run.smithi033.stdout: "RGW_API_SECRET_KEY": { 2024-03-19T03:03:35.974 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-19T03:03:35.974 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.974 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.974 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.974 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.974 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.974 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.974 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.974 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.974 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.975 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.975 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.975 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.975 INFO:teuthology.orchestra.run.smithi033.stdout: "RGW_API_SSL_VERIFY": { 2024-03-19T03:03:35.975 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-19T03:03:35.975 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.975 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.975 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.975 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.975 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.975 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.975 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.975 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.975 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.976 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.976 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.976 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.976 INFO:teuthology.orchestra.run.smithi033.stdout: "UNSAFE_TLS_v1_2": { 2024-03-19T03:03:35.976 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-19T03:03:35.976 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.976 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.976 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.976 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.976 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.976 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.976 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.976 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.976 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.976 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.976 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.977 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.977 INFO:teuthology.orchestra.run.smithi033.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-19T03:03:35.977 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-19T03:03:35.977 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.977 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.977 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.977 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0", 2024-03-19T03:03:35.977 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.977 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.977 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.977 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.977 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.977 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.977 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.977 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.978 INFO:teuthology.orchestra.run.smithi033.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-19T03:03:35.978 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-19T03:03:35.978 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.978 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.978 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.978 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-03-19T03:03:35.978 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.978 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.978 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.978 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.978 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.978 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.978 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.978 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.979 INFO:teuthology.orchestra.run.smithi033.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-19T03:03:35.979 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-19T03:03:35.979 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.979 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.979 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.979 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5", 2024-03-19T03:03:35.979 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.979 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.979 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.979 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.979 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.979 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.979 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.979 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.979 INFO:teuthology.orchestra.run.smithi033.stdout: "cross_origin_url": { 2024-03-19T03:03:35.980 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "cross_origin_url", 2024-03-19T03:03:35.980 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.980 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.980 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.980 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.980 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.980 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.980 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.980 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.980 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.980 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.980 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.980 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.980 INFO:teuthology.orchestra.run.smithi033.stdout: "crt_file": { 2024-03-19T03:03:35.981 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "crt_file", 2024-03-19T03:03:35.981 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.981 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.981 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.981 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.981 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.981 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.981 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.981 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.981 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.981 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.981 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.981 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.981 INFO:teuthology.orchestra.run.smithi033.stdout: "debug": { 2024-03-19T03:03:35.981 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "debug", 2024-03-19T03:03:35.982 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.982 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.982 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.982 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.982 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.982 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.982 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.982 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Enable/disable debug options", 2024-03-19T03:03:35.982 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.982 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.982 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.982 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.982 INFO:teuthology.orchestra.run.smithi033.stdout: "jwt_token_ttl": { 2024-03-19T03:03:35.982 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "jwt_token_ttl", 2024-03-19T03:03:35.982 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.983 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.983 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.983 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "28800", 2024-03-19T03:03:35.983 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.983 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.983 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.983 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.983 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.983 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.983 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.983 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.983 INFO:teuthology.orchestra.run.smithi033.stdout: "key_file": { 2024-03-19T03:03:35.983 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "key_file", 2024-03-19T03:03:35.983 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.984 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.984 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.984 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.984 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.984 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.984 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.984 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.984 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.984 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.984 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.984 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.984 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.984 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.984 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.984 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.984 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.985 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.985 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.985 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.985 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.985 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.985 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.985 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.985 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.985 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.985 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.985 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.985 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.985 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.985 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.985 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.986 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.986 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:35.986 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:35.986 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.986 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.986 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.986 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.986 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.986 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.986 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.986 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.986 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.986 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.986 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.987 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.987 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:35.987 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:35.987 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.987 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.987 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.987 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:35.987 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.987 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.987 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.987 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:35.987 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:35.987 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:35.987 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.987 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:35.988 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:35.988 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.988 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.988 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.988 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.988 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.988 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.988 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:35.988 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:35.988 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.988 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.988 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.988 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.988 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.989 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.989 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.989 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.989 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.989 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.989 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.989 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.989 INFO:teuthology.orchestra.run.smithi033.stdout: "motd": { 2024-03-19T03:03:35.989 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "motd", 2024-03-19T03:03:35.989 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.989 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.989 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.989 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.989 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.989 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.990 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.990 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "The message of the day", 2024-03-19T03:03:35.990 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.990 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.990 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.990 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.990 INFO:teuthology.orchestra.run.smithi033.stdout: "redirect_resolve_ip_addr": { 2024-03-19T03:03:35.990 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "redirect_resolve_ip_addr", 2024-03-19T03:03:35.990 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.990 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.990 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.990 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:35.990 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.990 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.991 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.991 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.991 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.991 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.991 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.991 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.991 INFO:teuthology.orchestra.run.smithi033.stdout: "server_addr": { 2024-03-19T03:03:35.991 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_addr", 2024-03-19T03:03:35.991 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.991 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.991 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.991 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "::", 2024-03-19T03:03:35.991 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.991 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.991 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.992 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.992 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.992 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.992 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.992 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.992 INFO:teuthology.orchestra.run.smithi033.stdout: "server_port": { 2024-03-19T03:03:35.992 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_port", 2024-03-19T03:03:35.992 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.992 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.992 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.992 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "8080", 2024-03-19T03:03:35.992 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.992 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.992 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.992 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.993 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.993 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.993 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.993 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.993 INFO:teuthology.orchestra.run.smithi033.stdout: "ssl": { 2024-03-19T03:03:35.993 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ssl", 2024-03-19T03:03:35.993 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.993 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.993 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.993 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.993 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.993 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.993 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.993 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.993 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.994 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.994 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.994 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.994 INFO:teuthology.orchestra.run.smithi033.stdout: "ssl_server_port": { 2024-03-19T03:03:35.994 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ssl_server_port", 2024-03-19T03:03:35.994 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.994 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.994 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.994 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "8443", 2024-03-19T03:03:35.994 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.994 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.994 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.994 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.994 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.994 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.995 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.995 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.995 INFO:teuthology.orchestra.run.smithi033.stdout: "standby_behaviour": { 2024-03-19T03:03:35.995 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "standby_behaviour", 2024-03-19T03:03:35.995 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.995 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.995 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.995 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "redirect", 2024-03-19T03:03:35.995 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.995 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.995 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:35.995 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:35.995 INFO:teuthology.orchestra.run.smithi033.stdout: "redirect" 2024-03-19T03:03:35.995 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:35.995 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.996 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.996 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.996 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.996 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.996 INFO:teuthology.orchestra.run.smithi033.stdout: "standby_error_status_code": { 2024-03-19T03:03:35.996 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "standby_error_status_code", 2024-03-19T03:03:35.996 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:35.996 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.996 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.996 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "500", 2024-03-19T03:03:35.996 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "400", 2024-03-19T03:03:35.996 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "599", 2024-03-19T03:03:35.996 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.996 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.997 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.997 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.997 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.997 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.997 INFO:teuthology.orchestra.run.smithi033.stdout: "url_prefix": { 2024-03-19T03:03:35.997 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "url_prefix", 2024-03-19T03:03:35.997 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.997 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.997 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:35.997 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:35.997 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.997 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.997 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.997 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:35.997 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.998 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.998 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.998 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.998 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:35.998 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.998 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:35.998 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "devicehealth", 2024-03-19T03:03:35.998 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:35.998 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:35.998 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:35.998 INFO:teuthology.orchestra.run.smithi033.stdout: "enable_monitoring": { 2024-03-19T03:03:35.998 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "enable_monitoring", 2024-03-19T03:03:35.998 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:35.998 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.999 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:35.999 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:35.999 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:35.999 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:35.999 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:35.999 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "monitor device health metrics", 2024-03-19T03:03:35.999 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:35.999 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:35.999 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:35.999 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:35.999 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:35.999 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:35.999 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:35.999 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:35.999 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.000 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.000 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.000 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.000 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.000 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.000 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.000 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.000 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.000 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.000 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.000 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.000 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.000 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.000 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.000 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.001 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.001 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.001 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.001 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.001 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.001 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.001 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.001 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.001 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.001 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.001 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.001 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.001 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.001 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.001 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.002 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.002 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.002 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.002 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.002 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.002 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.002 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.002 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.002 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.002 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.002 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.002 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.002 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.003 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.003 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.003 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.003 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.003 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.003 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.003 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.003 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.003 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.003 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.003 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.003 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.003 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.003 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.003 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.004 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.004 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.004 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.004 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.004 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.004 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.004 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.004 INFO:teuthology.orchestra.run.smithi033.stdout: "mark_out_threshold": { 2024-03-19T03:03:36.004 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mark_out_threshold", 2024-03-19T03:03:36.004 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:36.004 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.004 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.004 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "2419200", 2024-03-19T03:03:36.004 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.004 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.005 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.005 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-19T03:03:36.005 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.005 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.005 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.005 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.005 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": { 2024-03-19T03:03:36.005 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "pool_name", 2024-03-19T03:03:36.005 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.005 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.005 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.005 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "device_health_metrics", 2024-03-19T03:03:36.005 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.005 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.005 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.006 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-19T03:03:36.006 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.006 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.006 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.006 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.006 INFO:teuthology.orchestra.run.smithi033.stdout: "retention_period": { 2024-03-19T03:03:36.006 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "retention_period", 2024-03-19T03:03:36.006 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:36.006 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.006 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.006 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "15552000", 2024-03-19T03:03:36.006 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.006 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.006 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.007 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how long to retain device health metrics", 2024-03-19T03:03:36.007 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.007 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.007 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.007 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.007 INFO:teuthology.orchestra.run.smithi033.stdout: "scrape_frequency": { 2024-03-19T03:03:36.007 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "scrape_frequency", 2024-03-19T03:03:36.007 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:36.007 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.007 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.007 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "86400", 2024-03-19T03:03:36.007 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.007 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.007 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.007 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-19T03:03:36.008 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.008 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.008 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.008 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.008 INFO:teuthology.orchestra.run.smithi033.stdout: "self_heal": { 2024-03-19T03:03:36.008 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "self_heal", 2024-03-19T03:03:36.008 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.008 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.008 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.008 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.008 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.008 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.008 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.008 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-19T03:03:36.009 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.009 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.009 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.009 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.009 INFO:teuthology.orchestra.run.smithi033.stdout: "sleep_interval": { 2024-03-19T03:03:36.009 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "sleep_interval", 2024-03-19T03:03:36.009 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:36.009 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.009 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.009 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "600", 2024-03-19T03:03:36.009 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.009 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.009 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.009 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how frequently to wake up and check device health", 2024-03-19T03:03:36.010 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.010 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.010 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.010 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.010 INFO:teuthology.orchestra.run.smithi033.stdout: "warn_threshold": { 2024-03-19T03:03:36.010 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "warn_threshold", 2024-03-19T03:03:36.010 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:36.010 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.010 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.010 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "7257600", 2024-03-19T03:03:36.010 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.010 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.010 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.010 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-19T03:03:36.011 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.011 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.011 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.011 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.011 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.011 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.011 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.011 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "diskprediction_local", 2024-03-19T03:03:36.011 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.011 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.011 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.011 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.011 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.011 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.012 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.012 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.012 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.012 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.012 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.012 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.012 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.012 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.012 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.012 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.012 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.012 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.012 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.012 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.013 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.013 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.013 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.013 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.013 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.013 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.013 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.013 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.013 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.013 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.013 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.013 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.013 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.013 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.013 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.014 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.014 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.014 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.014 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.014 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.014 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.014 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.014 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.014 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.014 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.014 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.014 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.014 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.014 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.015 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.015 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.015 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.015 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.015 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.015 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.015 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.015 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.015 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.015 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.015 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.015 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.015 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.015 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.015 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.016 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.016 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.016 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.016 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.016 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.016 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.016 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.016 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.016 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.016 INFO:teuthology.orchestra.run.smithi033.stdout: "predict_interval": { 2024-03-19T03:03:36.016 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "predict_interval", 2024-03-19T03:03:36.016 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.016 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.016 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.016 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "86400", 2024-03-19T03:03:36.017 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.017 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.017 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.017 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.017 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.017 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.017 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.017 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.017 INFO:teuthology.orchestra.run.smithi033.stdout: "predictor_model": { 2024-03-19T03:03:36.017 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "predictor_model", 2024-03-19T03:03:36.017 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.017 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.017 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.017 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "prophetstor", 2024-03-19T03:03:36.018 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.018 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.018 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.018 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.018 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.018 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.018 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.018 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.018 INFO:teuthology.orchestra.run.smithi033.stdout: "sleep_interval": { 2024-03-19T03:03:36.018 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "sleep_interval", 2024-03-19T03:03:36.018 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.018 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.018 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.018 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "600", 2024-03-19T03:03:36.019 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.019 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.019 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.019 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.019 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.019 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.019 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.019 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.019 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.019 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.019 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.019 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "influx", 2024-03-19T03:03:36.019 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": false, 2024-03-19T03:03:36.019 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "influxdb python module not found", 2024-03-19T03:03:36.019 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.020 INFO:teuthology.orchestra.run.smithi033.stdout: "batch_size": { 2024-03-19T03:03:36.020 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "batch_size", 2024-03-19T03:03:36.020 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.020 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.020 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.020 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5000", 2024-03-19T03:03:36.020 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.020 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.020 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.020 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-19T03:03:36.020 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.020 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.020 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.020 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.021 INFO:teuthology.orchestra.run.smithi033.stdout: "database": { 2024-03-19T03:03:36.021 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "database", 2024-03-19T03:03:36.021 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.021 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.021 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.021 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "ceph", 2024-03-19T03:03:36.021 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.021 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.021 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.021 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-19T03:03:36.021 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.021 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.021 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.021 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.021 INFO:teuthology.orchestra.run.smithi033.stdout: "hostname": { 2024-03-19T03:03:36.022 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "hostname", 2024-03-19T03:03:36.022 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.022 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.022 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.022 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.022 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.022 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.022 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.022 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "InfluxDB server hostname", 2024-03-19T03:03:36.022 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.022 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.022 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.022 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.023 INFO:teuthology.orchestra.run.smithi033.stdout: "interval": { 2024-03-19T03:03:36.023 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "interval", 2024-03-19T03:03:36.023 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:36.023 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.023 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.023 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "30", 2024-03-19T03:03:36.023 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "5", 2024-03-19T03:03:36.023 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.023 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.023 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-19T03:03:36.023 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.023 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.023 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.023 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.024 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.024 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.024 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.024 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.024 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.024 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.024 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.024 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.024 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.024 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.024 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.024 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.024 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.024 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.025 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.025 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.025 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.025 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.025 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.025 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.025 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.025 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.025 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.025 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.025 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.025 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.025 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.025 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.025 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.026 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.026 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.026 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.026 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.026 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.026 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.026 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.026 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.026 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.026 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.026 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.026 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.026 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.026 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.027 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.027 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.027 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.027 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.027 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.027 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.027 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.027 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.027 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.027 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.027 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.027 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.027 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.027 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.028 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.028 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.028 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.028 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.028 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.028 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.028 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.028 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.028 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.028 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.028 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.028 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.028 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.028 INFO:teuthology.orchestra.run.smithi033.stdout: "password": { 2024-03-19T03:03:36.028 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "password", 2024-03-19T03:03:36.029 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.029 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.029 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.029 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.029 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.029 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.029 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.029 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "password of InfluxDB server user", 2024-03-19T03:03:36.029 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.029 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.029 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.029 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.029 INFO:teuthology.orchestra.run.smithi033.stdout: "port": { 2024-03-19T03:03:36.029 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "port", 2024-03-19T03:03:36.029 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.030 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.030 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.030 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "8086", 2024-03-19T03:03:36.030 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.030 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.030 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.030 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "InfluxDB server port", 2024-03-19T03:03:36.030 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.030 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.030 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.030 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.030 INFO:teuthology.orchestra.run.smithi033.stdout: "ssl": { 2024-03-19T03:03:36.030 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ssl", 2024-03-19T03:03:36.030 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.031 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.031 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.031 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "false", 2024-03-19T03:03:36.031 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.031 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.031 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.031 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-19T03:03:36.031 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.031 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.031 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.031 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.031 INFO:teuthology.orchestra.run.smithi033.stdout: "threads": { 2024-03-19T03:03:36.031 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "threads", 2024-03-19T03:03:36.031 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.031 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.031 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.032 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5", 2024-03-19T03:03:36.032 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "1", 2024-03-19T03:03:36.032 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "32", 2024-03-19T03:03:36.032 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.032 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-19T03:03:36.032 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.032 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.032 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.032 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.032 INFO:teuthology.orchestra.run.smithi033.stdout: "username": { 2024-03-19T03:03:36.032 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "username", 2024-03-19T03:03:36.032 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.032 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.032 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.032 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.032 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.033 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.033 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.033 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "username of InfluxDB server user", 2024-03-19T03:03:36.033 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.033 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.033 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.033 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.033 INFO:teuthology.orchestra.run.smithi033.stdout: "verify_ssl": { 2024-03-19T03:03:36.033 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "verify_ssl", 2024-03-19T03:03:36.033 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.033 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.033 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.033 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "true", 2024-03-19T03:03:36.033 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.033 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.034 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.034 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-19T03:03:36.034 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.034 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.034 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.034 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.034 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.034 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.034 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.034 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "insights", 2024-03-19T03:03:36.034 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.034 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.034 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.034 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.034 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.035 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.035 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.035 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.035 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.035 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.035 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.035 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.035 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.035 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.035 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.035 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.035 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.035 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.035 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.035 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.035 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.036 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.036 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.036 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.036 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.036 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.036 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.036 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.036 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.036 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.036 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.036 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.036 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.036 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.036 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.036 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.037 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.037 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.037 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.037 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.037 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.037 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.037 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.037 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.037 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.037 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.037 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.037 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.037 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.037 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.037 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.038 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.038 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.038 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.038 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.038 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.038 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.038 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.038 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.038 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.038 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.038 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.041 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.041 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.041 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.041 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.041 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.041 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.041 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.042 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.042 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.042 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.042 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.042 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.042 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.042 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.042 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "iostat", 2024-03-19T03:03:36.042 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.042 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.042 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.042 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.042 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.043 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.043 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.043 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.043 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.043 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.043 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.043 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.043 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.043 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.043 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.043 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.043 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.043 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.043 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.043 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.044 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.044 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.044 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.044 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.044 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.044 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.044 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.044 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.044 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.044 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.044 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.044 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.044 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.044 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.044 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.044 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.044 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.045 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.045 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.045 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.045 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.045 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.045 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.045 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.045 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.045 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.045 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.045 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.045 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.045 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.045 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.045 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.045 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.046 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.046 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.046 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.046 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.046 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.046 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.046 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.046 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.046 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.046 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.046 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.046 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.046 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.047 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.047 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.047 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.047 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.047 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.047 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.047 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.047 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.047 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.047 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.047 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "k8sevents", 2024-03-19T03:03:36.047 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.047 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.047 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.047 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_event_retention_days": { 2024-03-19T03:03:36.047 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ceph_event_retention_days", 2024-03-19T03:03:36.048 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.048 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.048 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.048 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "7", 2024-03-19T03:03:36.048 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.048 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.048 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.048 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-19T03:03:36.048 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.048 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.048 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.048 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.048 INFO:teuthology.orchestra.run.smithi033.stdout: "config_check_secs": { 2024-03-19T03:03:36.048 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "config_check_secs", 2024-03-19T03:03:36.048 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.048 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.049 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.049 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-03-19T03:03:36.049 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "10", 2024-03-19T03:03:36.049 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.049 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.049 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-19T03:03:36.049 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.049 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.049 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.049 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.049 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.049 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.049 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.049 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.049 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.050 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.050 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.050 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.050 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.050 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.050 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.050 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.050 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.050 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.050 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.050 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.050 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.050 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.050 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.051 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.051 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.051 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.051 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.051 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.051 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.051 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.051 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.051 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.051 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.051 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.051 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.051 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.051 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.052 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.052 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.052 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.052 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.052 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.052 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.052 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.052 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.052 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.052 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.052 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.052 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.052 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.052 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.052 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.052 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.053 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.053 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.053 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.053 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.053 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.053 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.053 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.053 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.053 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.053 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.053 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.053 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.053 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.053 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.054 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.054 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.054 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.054 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.054 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.054 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.054 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.054 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.054 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.054 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.054 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "localpool", 2024-03-19T03:03:36.054 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.054 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.054 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.055 INFO:teuthology.orchestra.run.smithi033.stdout: "failure_domain": { 2024-03-19T03:03:36.055 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "failure_domain", 2024-03-19T03:03:36.055 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.055 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.055 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.055 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "host", 2024-03-19T03:03:36.055 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.055 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.055 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.055 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "failure domain for any created local pool", 2024-03-19T03:03:36.055 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-19T03:03:36.055 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.055 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.055 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.055 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.055 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.056 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.056 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.056 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.056 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.056 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.056 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.056 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.056 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.056 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.056 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.056 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.056 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.056 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.056 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.056 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.057 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.057 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.057 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.057 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.057 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.057 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.057 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.057 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.057 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.057 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.057 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.057 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.057 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.057 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.058 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.058 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.058 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.058 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.058 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.058 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.058 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.058 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.058 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.058 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.058 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.058 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.058 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.058 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.059 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.059 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.059 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.059 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.059 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.059 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.059 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.059 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.059 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.059 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.059 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.059 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.059 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.059 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.059 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.060 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.060 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.060 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.060 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.060 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.060 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.060 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.060 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.060 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.060 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.060 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": { 2024-03-19T03:03:36.060 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "min_size", 2024-03-19T03:03:36.060 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.060 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.060 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.061 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.061 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.061 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.061 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.061 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "default min_size for any created local pool", 2024-03-19T03:03:36.061 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-19T03:03:36.061 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.061 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.061 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.061 INFO:teuthology.orchestra.run.smithi033.stdout: "num_rep": { 2024-03-19T03:03:36.061 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "num_rep", 2024-03-19T03:03:36.061 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.061 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.061 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.061 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "3", 2024-03-19T03:03:36.061 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.062 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.062 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.062 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "default replica count for any created local pool", 2024-03-19T03:03:36.062 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.062 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.062 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.062 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.062 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": { 2024-03-19T03:03:36.062 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "pg_num", 2024-03-19T03:03:36.062 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.062 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.062 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.062 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "128", 2024-03-19T03:03:36.062 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.062 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.063 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.063 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "default pg_num for any created local pool", 2024-03-19T03:03:36.063 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.063 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.063 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.063 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.063 INFO:teuthology.orchestra.run.smithi033.stdout: "prefix": { 2024-03-19T03:03:36.063 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "prefix", 2024-03-19T03:03:36.063 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.063 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.064 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.064 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.064 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.064 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.064 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.064 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "name prefix for any created local pool", 2024-03-19T03:03:36.064 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.064 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.064 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.064 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.064 INFO:teuthology.orchestra.run.smithi033.stdout: "subtree": { 2024-03-19T03:03:36.064 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "subtree", 2024-03-19T03:03:36.064 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.064 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.064 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.064 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "rack", 2024-03-19T03:03:36.065 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.065 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.065 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.065 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-19T03:03:36.065 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-19T03:03:36.065 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.065 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.065 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.065 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.065 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.065 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.065 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mds_autoscaler", 2024-03-19T03:03:36.065 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.065 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.065 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.066 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.066 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.066 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.066 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.066 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.066 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.066 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.066 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.066 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.066 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.066 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.066 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.066 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.066 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.067 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.067 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.067 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.067 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.067 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.067 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.067 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.067 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.067 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.067 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.067 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.067 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.067 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.067 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.067 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.068 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.068 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.068 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.068 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.068 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.068 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.068 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.068 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.068 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.068 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.068 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.068 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.068 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.068 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.068 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.069 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.069 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.069 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.069 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.069 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.069 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.069 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.069 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.069 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.069 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.069 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.069 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.069 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.069 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.069 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.070 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.070 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.070 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.070 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.070 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.070 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.070 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.070 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.070 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.070 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.070 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.070 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.070 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.070 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.070 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mirroring", 2024-03-19T03:03:36.070 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.071 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.071 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.071 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.071 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.071 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.071 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.071 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.071 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.071 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.071 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.071 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.071 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.071 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.071 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.071 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.071 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.072 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.072 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.072 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.072 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.072 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.072 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.072 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.072 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.072 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.072 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.072 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.072 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.072 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.072 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.072 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.073 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.073 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.073 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.073 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.073 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.073 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.073 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.073 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.073 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.073 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.073 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.073 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.073 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.073 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.073 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.073 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.074 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.074 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.074 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.074 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.074 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.074 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.074 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.074 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.074 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.074 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.074 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.074 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.074 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.074 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.074 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.075 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.075 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.075 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.075 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.075 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.075 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.075 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.075 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.075 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.075 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.075 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.075 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.075 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.075 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "nfs", 2024-03-19T03:03:36.075 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.075 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.076 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.076 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.076 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.076 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.076 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.076 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.076 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.076 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.076 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.076 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.076 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.076 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.076 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.076 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.076 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.076 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.077 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.077 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.077 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.077 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.077 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.077 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.077 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.077 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.077 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.077 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.077 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.077 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.077 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.077 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.077 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.078 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.078 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.078 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.078 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.078 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.078 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.078 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.078 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.078 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.078 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.078 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.078 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.078 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.078 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.079 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.079 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.079 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.079 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.079 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.079 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.079 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.079 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.079 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.079 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.079 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.079 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.079 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.079 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.079 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.080 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.080 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.080 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.080 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.080 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.080 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.080 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.080 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.080 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.080 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.080 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.080 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.080 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.080 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.081 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "orchestrator", 2024-03-19T03:03:36.081 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.081 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.081 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.081 INFO:teuthology.orchestra.run.smithi033.stdout: "fail_fs": { 2024-03-19T03:03:36.081 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "fail_fs", 2024-03-19T03:03:36.081 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.081 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.081 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.081 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.081 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.081 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.081 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.081 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-19T03:03:36.081 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.082 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.082 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.082 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.082 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.082 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.082 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.082 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.082 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.082 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.082 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.082 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.082 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.082 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.083 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.083 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.083 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.083 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.083 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.083 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.083 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.083 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.083 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.083 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.083 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.083 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.083 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.083 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.083 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.084 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.084 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.084 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.084 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.084 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.084 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.084 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.084 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.084 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.084 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.084 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.084 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.084 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.084 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.084 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.084 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.084 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.085 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.085 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.085 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.085 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.085 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.085 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.085 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.085 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.085 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.085 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.085 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.085 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.085 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.085 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.085 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.086 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.086 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.086 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.086 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.086 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.086 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.086 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.086 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.086 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.086 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.086 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.086 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.086 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.086 INFO:teuthology.orchestra.run.smithi033.stdout: "orchestrator": { 2024-03-19T03:03:36.086 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "orchestrator", 2024-03-19T03:03:36.087 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.087 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.087 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.087 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.087 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.087 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.087 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.087 INFO:teuthology.orchestra.run.smithi033.stdout: "cephadm", 2024-03-19T03:03:36.087 INFO:teuthology.orchestra.run.smithi033.stdout: "rook", 2024-03-19T03:03:36.087 INFO:teuthology.orchestra.run.smithi033.stdout: "test_orchestrator" 2024-03-19T03:03:36.087 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.087 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Orchestrator backend", 2024-03-19T03:03:36.087 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.087 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.087 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.088 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.088 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.088 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.088 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.088 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "osd_perf_query", 2024-03-19T03:03:36.088 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.088 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.088 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.088 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.088 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.088 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.088 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.088 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.088 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.089 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.089 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.089 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.089 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.089 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.089 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.089 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.089 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.089 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.089 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.089 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.089 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.089 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.089 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.090 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.090 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.090 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.090 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.090 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.090 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.090 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.090 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.090 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.090 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.090 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.090 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.090 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.090 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.091 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.091 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.091 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.091 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.091 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.091 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.091 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.091 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.091 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.091 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.091 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.091 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.091 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.091 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.091 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.092 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.092 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.092 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.092 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.092 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.092 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.092 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.092 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.092 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.092 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.092 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.092 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.092 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.092 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.093 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.093 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.093 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.093 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.093 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.093 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.093 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.093 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.093 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.093 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.093 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "osd_support", 2024-03-19T03:03:36.093 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.093 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.093 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.094 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.094 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.094 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.094 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.094 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.094 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.094 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.094 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.094 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.094 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.094 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.094 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.094 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.094 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.095 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.095 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.095 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.095 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.095 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.095 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.095 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.095 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.095 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.095 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.095 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.095 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.095 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.095 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.096 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.096 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.096 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.096 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.096 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.096 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.096 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.096 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.096 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.096 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.096 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.096 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.096 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.096 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.096 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.097 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.097 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.097 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.097 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.097 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.097 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.097 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.097 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.097 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.097 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.097 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.097 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.097 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.097 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.097 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.097 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.098 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.098 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.098 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.098 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.098 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.098 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.098 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.098 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.098 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.098 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.098 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.098 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.098 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.098 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.099 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "pg_autoscaler", 2024-03-19T03:03:36.099 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.099 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.099 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.099 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.099 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.099 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.099 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.099 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.099 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.099 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.099 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.099 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.099 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.099 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.099 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.100 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.100 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.100 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.100 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.100 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.100 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.100 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.100 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.100 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.100 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.100 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.100 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.100 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.100 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.100 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.101 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.101 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.101 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.101 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.101 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.101 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.101 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.101 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.101 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.101 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.101 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.101 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.101 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.101 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.101 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.101 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.102 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.102 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.102 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.102 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.102 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.102 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.102 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.102 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.102 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.102 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.102 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.102 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.102 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.102 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.102 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.102 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.103 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.103 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.103 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.103 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.103 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.103 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.103 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.103 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.103 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.103 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.103 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.103 INFO:teuthology.orchestra.run.smithi033.stdout: "sleep_interval": { 2024-03-19T03:03:36.103 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "sleep_interval", 2024-03-19T03:03:36.103 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:36.103 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.104 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.104 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "60", 2024-03-19T03:03:36.104 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.104 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.104 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.104 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.104 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.104 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.104 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.104 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.104 INFO:teuthology.orchestra.run.smithi033.stdout: "threshold": { 2024-03-19T03:03:36.104 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "threshold", 2024-03-19T03:03:36.104 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-03-19T03:03:36.104 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.104 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.104 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "3.0", 2024-03-19T03:03:36.105 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "1.0", 2024-03-19T03:03:36.105 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.105 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.105 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "scaling threshold", 2024-03-19T03:03:36.105 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-19T03:03:36.105 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.105 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.105 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.105 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.105 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.105 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.105 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "progress", 2024-03-19T03:03:36.105 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.105 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.105 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.106 INFO:teuthology.orchestra.run.smithi033.stdout: "allow_pg_recovery_event": { 2024-03-19T03:03:36.106 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "allow_pg_recovery_event", 2024-03-19T03:03:36.106 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.106 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.106 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.106 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.106 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.106 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.106 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.106 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-19T03:03:36.106 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.106 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.106 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.106 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.106 INFO:teuthology.orchestra.run.smithi033.stdout: "enabled": { 2024-03-19T03:03:36.107 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "enabled", 2024-03-19T03:03:36.107 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.107 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.107 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.107 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.107 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.107 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.107 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.107 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.107 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.107 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.107 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.107 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.107 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.107 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.108 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.108 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.108 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.108 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.108 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.108 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.108 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.108 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.108 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.108 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.108 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.108 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.108 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.108 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.108 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.108 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.109 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.109 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.109 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.109 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.109 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.109 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.109 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.109 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.109 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.109 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.109 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.109 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.109 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.109 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.109 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.110 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.110 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.110 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.110 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.110 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.110 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.110 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.110 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.110 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.110 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.110 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.110 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.110 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.110 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.110 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.110 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.111 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.111 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.111 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.111 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.111 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.111 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.111 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.111 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.111 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.111 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.111 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.111 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.111 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.111 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.111 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.112 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.112 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.112 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.112 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.112 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.112 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.112 INFO:teuthology.orchestra.run.smithi033.stdout: "max_completed_events": { 2024-03-19T03:03:36.112 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "max_completed_events", 2024-03-19T03:03:36.112 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.112 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.112 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.112 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "50", 2024-03-19T03:03:36.112 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.112 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.112 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.112 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "number of past completed events to remember", 2024-03-19T03:03:36.113 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.113 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.113 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.113 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.113 INFO:teuthology.orchestra.run.smithi033.stdout: "sleep_interval": { 2024-03-19T03:03:36.113 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "sleep_interval", 2024-03-19T03:03:36.113 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:36.113 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.113 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.113 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5", 2024-03-19T03:03:36.113 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.113 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.113 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.113 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how long the module is going to sleep", 2024-03-19T03:03:36.113 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.113 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.114 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.114 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.114 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.114 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.114 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.114 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "prometheus", 2024-03-19T03:03:36.114 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.114 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.114 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.114 INFO:teuthology.orchestra.run.smithi033.stdout: "cache": { 2024-03-19T03:03:36.114 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "cache", 2024-03-19T03:03:36.114 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.114 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.114 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.114 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.115 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.115 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.115 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.115 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.115 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.115 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.115 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.115 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.115 INFO:teuthology.orchestra.run.smithi033.stdout: "exclude_perf_counters": { 2024-03-19T03:03:36.115 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "exclude_perf_counters", 2024-03-19T03:03:36.115 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.115 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.115 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.115 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.115 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.116 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.116 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.116 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-19T03:03:36.116 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-19T03:03:36.116 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.116 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.116 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.116 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.116 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.116 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.116 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.116 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.116 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.116 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.116 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.116 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.117 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.117 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.117 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.117 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.117 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.117 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.117 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.117 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.117 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.117 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.117 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.117 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.117 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.117 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.117 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.117 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.118 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.118 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.118 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.118 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.118 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.118 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.118 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.118 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.118 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.118 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.118 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.118 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.118 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.118 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.119 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.119 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.119 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.119 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.119 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.119 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.119 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.119 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.119 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.119 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.119 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.119 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.119 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.119 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.119 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.120 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.120 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.120 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.120 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.120 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.120 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.120 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.120 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.120 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.120 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.120 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.120 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.120 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.120 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.121 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.121 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.121 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_stats_pools": { 2024-03-19T03:03:36.121 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rbd_stats_pools", 2024-03-19T03:03:36.121 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.121 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.121 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.121 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.121 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.121 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.121 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.121 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.121 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.121 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.121 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.122 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.122 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-19T03:03:36.122 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-19T03:03:36.122 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.122 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.122 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.122 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "300", 2024-03-19T03:03:36.122 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.122 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.122 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.122 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.122 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.122 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.122 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.123 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.123 INFO:teuthology.orchestra.run.smithi033.stdout: "scrape_interval": { 2024-03-19T03:03:36.123 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "scrape_interval", 2024-03-19T03:03:36.123 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-03-19T03:03:36.123 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.123 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.123 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "15.0", 2024-03-19T03:03:36.123 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.123 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.123 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.123 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.123 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.123 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.123 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.123 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.123 INFO:teuthology.orchestra.run.smithi033.stdout: "server_addr": { 2024-03-19T03:03:36.124 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_addr", 2024-03-19T03:03:36.124 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.124 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.124 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.124 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "::", 2024-03-19T03:03:36.124 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.124 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.124 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.124 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-19T03:03:36.124 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.124 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.124 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.124 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.124 INFO:teuthology.orchestra.run.smithi033.stdout: "server_port": { 2024-03-19T03:03:36.124 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_port", 2024-03-19T03:03:36.125 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.125 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.125 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.125 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "9283", 2024-03-19T03:03:36.125 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.125 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.125 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.125 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-19T03:03:36.125 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.125 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.125 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.125 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.125 INFO:teuthology.orchestra.run.smithi033.stdout: "stale_cache_strategy": { 2024-03-19T03:03:36.125 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "stale_cache_strategy", 2024-03-19T03:03:36.125 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.125 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.126 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.126 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "log", 2024-03-19T03:03:36.126 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.126 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.126 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.126 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.126 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.126 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.126 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.126 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.126 INFO:teuthology.orchestra.run.smithi033.stdout: "standby_behaviour": { 2024-03-19T03:03:36.126 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "standby_behaviour", 2024-03-19T03:03:36.126 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.126 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.127 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.127 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "default", 2024-03-19T03:03:36.127 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.127 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.127 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.127 INFO:teuthology.orchestra.run.smithi033.stdout: "default", 2024-03-19T03:03:36.127 INFO:teuthology.orchestra.run.smithi033.stdout: "error" 2024-03-19T03:03:36.127 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.127 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.127 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.127 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.127 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.127 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.127 INFO:teuthology.orchestra.run.smithi033.stdout: "standby_error_status_code": { 2024-03-19T03:03:36.128 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "standby_error_status_code", 2024-03-19T03:03:36.128 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.128 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.128 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.128 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "500", 2024-03-19T03:03:36.128 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "400", 2024-03-19T03:03:36.128 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "599", 2024-03-19T03:03:36.128 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.128 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.128 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.128 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.128 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.128 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.128 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.128 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.129 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.129 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rbd_support", 2024-03-19T03:03:36.129 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.129 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.129 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.129 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.129 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.129 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.129 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.129 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.129 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.129 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.129 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.129 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.129 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.130 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.130 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.130 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.130 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.130 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.130 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.130 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.130 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.130 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.130 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.130 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.130 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.130 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.130 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.130 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.131 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.131 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.131 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.131 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.131 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.131 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.131 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.131 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.131 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.131 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.131 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.131 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.131 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.131 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.131 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.132 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.132 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.132 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.132 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.132 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.132 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.132 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.132 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.132 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.132 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.132 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.132 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.132 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.132 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.132 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.133 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.133 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.133 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.133 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.133 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.133 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.133 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.133 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.133 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.133 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.133 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.133 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.133 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.133 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.133 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.134 INFO:teuthology.orchestra.run.smithi033.stdout: "max_concurrent_snap_create": { 2024-03-19T03:03:36.134 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "max_concurrent_snap_create", 2024-03-19T03:03:36.134 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.134 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.134 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.134 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-03-19T03:03:36.134 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.134 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.134 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.134 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.134 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.134 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.134 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.134 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.135 INFO:teuthology.orchestra.run.smithi033.stdout: "mirror_snapshot_schedule": { 2024-03-19T03:03:36.135 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mirror_snapshot_schedule", 2024-03-19T03:03:36.135 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.135 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.135 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.135 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.135 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.135 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.135 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.135 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.135 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.135 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.135 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.135 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.135 INFO:teuthology.orchestra.run.smithi033.stdout: "trash_purge_schedule": { 2024-03-19T03:03:36.135 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "trash_purge_schedule", 2024-03-19T03:03:36.136 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.136 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.136 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.136 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.136 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.136 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.136 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.136 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.136 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.136 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.136 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.136 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.136 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.136 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.136 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.137 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "restful", 2024-03-19T03:03:36.137 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.137 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.137 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.137 INFO:teuthology.orchestra.run.smithi033.stdout: "enable_auth": { 2024-03-19T03:03:36.137 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "enable_auth", 2024-03-19T03:03:36.137 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.137 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.137 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.137 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.137 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.137 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.137 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.137 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.137 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.138 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.138 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.138 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.138 INFO:teuthology.orchestra.run.smithi033.stdout: "key_file": { 2024-03-19T03:03:36.138 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "key_file", 2024-03-19T03:03:36.138 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.138 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.138 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.138 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.138 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.138 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.138 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.138 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.138 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.139 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.139 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.139 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.139 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.139 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.139 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.139 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.139 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.139 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.139 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.139 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.139 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.139 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.139 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.140 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.140 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.140 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.140 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.140 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.140 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.140 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.140 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.140 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.140 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.140 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.140 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.140 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.140 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.141 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.141 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.141 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.141 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.141 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.141 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.141 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.141 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.141 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.141 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.141 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.141 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.141 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.141 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.141 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.142 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.142 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.142 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.142 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.142 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.142 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.142 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.142 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.142 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.142 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.142 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.142 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.142 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.143 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.143 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.143 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.143 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.143 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.143 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.143 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.143 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.143 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.143 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.143 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.143 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.143 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.143 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.143 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.144 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.144 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.144 INFO:teuthology.orchestra.run.smithi033.stdout: "server_addr": { 2024-03-19T03:03:36.144 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_addr", 2024-03-19T03:03:36.144 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.144 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.144 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.144 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.144 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.144 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.144 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.144 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.144 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.144 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.144 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.144 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.145 INFO:teuthology.orchestra.run.smithi033.stdout: "server_port": { 2024-03-19T03:03:36.145 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_port", 2024-03-19T03:03:36.145 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.145 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.145 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.145 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.145 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.145 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.145 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.145 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.145 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.145 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.145 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.145 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.146 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.146 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.146 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.146 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rgw", 2024-03-19T03:03:36.146 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.146 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.146 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.146 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.146 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.146 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.146 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.146 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.146 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.146 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.147 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.147 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.147 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.147 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.147 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.147 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.147 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.147 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.147 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.147 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.147 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.147 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.147 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.147 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.147 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.148 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.148 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.148 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.148 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.148 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.148 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.148 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.148 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.148 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.148 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.148 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.148 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.148 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.148 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.149 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.149 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.149 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.149 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.149 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.149 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.149 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.149 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.149 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.149 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.149 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.149 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.149 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.149 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.149 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.149 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.150 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.150 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.150 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.150 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.150 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.150 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.150 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.150 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.150 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.150 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.150 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.150 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.150 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.150 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.150 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.151 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.151 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.151 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.151 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.151 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.151 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.151 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rook", 2024-03-19T03:03:36.151 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.151 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.151 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.151 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.151 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.151 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.151 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.151 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.152 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.152 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.152 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.152 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.152 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.152 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.152 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.152 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.152 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.152 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.152 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.152 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.152 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.152 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.152 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.153 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.153 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.153 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.153 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.153 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.153 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.153 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.153 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.153 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.153 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.153 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.153 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.153 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.153 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.153 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.154 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.154 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.154 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.154 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.154 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.154 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.154 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.154 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.154 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.154 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.154 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.154 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.154 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.154 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.155 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.155 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.155 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.155 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.155 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.155 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.155 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.155 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.155 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.155 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.155 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.155 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.155 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.155 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.155 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.156 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.156 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.156 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.156 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.156 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.156 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.156 INFO:teuthology.orchestra.run.smithi033.stdout: "storage_class": { 2024-03-19T03:03:36.156 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "storage_class", 2024-03-19T03:03:36.156 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.156 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.156 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.156 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "local", 2024-03-19T03:03:36.156 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.156 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.156 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.157 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-19T03:03:36.157 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.157 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.157 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.157 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.157 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.157 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.157 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.157 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "selftest", 2024-03-19T03:03:36.157 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.157 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.157 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.157 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.157 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.157 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.158 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.158 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.158 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.158 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.158 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.158 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.158 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.158 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.158 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.158 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.158 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.158 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.158 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.158 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.159 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.159 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.159 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.159 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.159 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.159 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.159 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.159 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.159 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.159 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.159 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.159 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.159 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.159 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.159 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.160 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.160 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.160 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.160 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.160 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.160 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.160 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.160 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.160 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.160 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.160 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.160 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.160 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.160 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.160 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.160 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.161 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.161 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.161 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.161 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.161 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.161 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.161 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.161 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.161 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.161 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.161 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.161 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.161 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.161 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.161 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.162 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.162 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.162 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.162 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.162 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.162 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.162 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.162 INFO:teuthology.orchestra.run.smithi033.stdout: "roption1": { 2024-03-19T03:03:36.162 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "roption1", 2024-03-19T03:03:36.162 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.162 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.162 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.162 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.162 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.162 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.163 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.163 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.163 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.163 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.163 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.163 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.163 INFO:teuthology.orchestra.run.smithi033.stdout: "roption2": { 2024-03-19T03:03:36.163 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "roption2", 2024-03-19T03:03:36.163 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.163 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.163 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.163 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "xyz", 2024-03-19T03:03:36.163 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.163 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.164 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.164 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.164 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.164 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.164 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.164 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.164 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption1": { 2024-03-19T03:03:36.164 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption1", 2024-03-19T03:03:36.164 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.164 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.164 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.164 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.164 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.164 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.164 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.165 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.165 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.165 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.165 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.165 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.165 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption2": { 2024-03-19T03:03:36.165 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption2", 2024-03-19T03:03:36.165 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.165 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.165 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.165 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.165 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.165 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.165 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.165 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.166 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.166 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.166 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.166 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.166 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption3": { 2024-03-19T03:03:36.166 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption3", 2024-03-19T03:03:36.166 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-03-19T03:03:36.166 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.166 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.166 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.166 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.166 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.166 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.166 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.167 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.167 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.167 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.167 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.167 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption4": { 2024-03-19T03:03:36.167 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption4", 2024-03-19T03:03:36.167 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.167 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.167 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.167 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.167 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.167 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.167 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.167 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.167 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.168 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.168 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.168 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.168 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption5": { 2024-03-19T03:03:36.168 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption5", 2024-03-19T03:03:36.168 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.168 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.168 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.168 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.168 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.168 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.168 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.168 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.168 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.168 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.168 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.169 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.169 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption6": { 2024-03-19T03:03:36.169 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption6", 2024-03-19T03:03:36.169 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.169 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.169 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.169 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.169 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.169 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.169 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.169 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.169 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.169 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.169 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.170 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.170 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption7": { 2024-03-19T03:03:36.170 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption7", 2024-03-19T03:03:36.170 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.170 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.170 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.170 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.170 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "1", 2024-03-19T03:03:36.170 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "42", 2024-03-19T03:03:36.170 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.170 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.170 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.170 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.170 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.171 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.171 INFO:teuthology.orchestra.run.smithi033.stdout: "testkey": { 2024-03-19T03:03:36.171 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "testkey", 2024-03-19T03:03:36.171 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.171 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.171 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.171 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.171 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.171 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.171 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.171 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.171 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.171 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.171 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.171 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.172 INFO:teuthology.orchestra.run.smithi033.stdout: "testlkey": { 2024-03-19T03:03:36.172 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "testlkey", 2024-03-19T03:03:36.172 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.172 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.172 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.172 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.172 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.172 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.172 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.172 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.172 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.172 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.172 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.172 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.172 INFO:teuthology.orchestra.run.smithi033.stdout: "testnewline": { 2024-03-19T03:03:36.173 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "testnewline", 2024-03-19T03:03:36.173 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.173 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.173 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.173 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.173 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.173 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.173 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.173 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.173 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.173 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.173 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.173 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.173 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.173 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.174 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.174 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "snap_schedule", 2024-03-19T03:03:36.174 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.174 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.174 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.174 INFO:teuthology.orchestra.run.smithi033.stdout: "allow_m_granularity": { 2024-03-19T03:03:36.174 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "allow_m_granularity", 2024-03-19T03:03:36.174 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.174 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.174 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.174 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.174 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.174 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.174 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.175 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "allow minute scheduled snapshots", 2024-03-19T03:03:36.175 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.175 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.175 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.175 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.175 INFO:teuthology.orchestra.run.smithi033.stdout: "dump_on_update": { 2024-03-19T03:03:36.175 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "dump_on_update", 2024-03-19T03:03:36.175 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.175 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.175 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.175 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.175 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.175 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.175 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.175 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "dump database to debug log on update", 2024-03-19T03:03:36.176 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.176 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.176 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.176 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.176 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.176 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.176 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.176 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.176 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.176 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.176 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.176 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.176 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.176 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.176 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.176 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.177 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.177 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.177 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.177 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.177 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.177 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.177 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.177 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.177 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.177 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.177 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.177 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.177 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.177 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.177 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.178 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.178 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.178 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.178 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.178 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.178 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.178 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.178 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.178 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.178 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.178 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.178 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.178 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.178 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.179 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.179 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.179 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.179 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.179 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.179 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.179 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.179 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.179 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.179 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.179 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.179 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.179 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.179 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.179 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.180 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.180 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.180 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.180 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.180 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.180 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.180 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.180 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.180 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.180 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.180 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.180 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.180 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.180 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.181 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.181 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.181 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.181 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "stats", 2024-03-19T03:03:36.181 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.181 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.181 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.181 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.181 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.181 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.181 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.181 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.181 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.181 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.181 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.182 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.182 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.182 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.182 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.182 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.182 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.182 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.182 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.182 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.182 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.182 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.182 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.182 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.183 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.183 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.183 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.183 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.183 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.183 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.183 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.183 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.183 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.183 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.183 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.183 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.183 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.183 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.183 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.184 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.184 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.184 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.184 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.184 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.184 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.184 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.184 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.184 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.184 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.184 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.184 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.184 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.184 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.184 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.184 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.185 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.185 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.185 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.185 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.185 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.185 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.185 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.185 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.185 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.185 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.185 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.185 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.185 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.185 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.185 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.185 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.186 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.186 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.186 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.186 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.186 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.186 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "status", 2024-03-19T03:03:36.186 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.186 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.186 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.186 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.186 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.186 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.186 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.186 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.186 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.187 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.187 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.187 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.187 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.187 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.187 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.187 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.187 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.187 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.187 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.187 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.187 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.187 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.187 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.188 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.188 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.188 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.188 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.188 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.188 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.188 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.188 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.188 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.188 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.188 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.188 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.188 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.188 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.188 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.189 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.189 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.189 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.189 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.189 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.189 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.189 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.189 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.189 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.189 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.189 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.189 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.189 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.189 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.189 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.189 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.190 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.190 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.190 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.190 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.190 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.190 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.190 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.190 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.190 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.190 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.190 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.190 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.190 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.190 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.191 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.191 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.191 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.191 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.191 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.191 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.191 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.191 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.191 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "telegraf", 2024-03-19T03:03:36.191 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.191 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.191 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.191 INFO:teuthology.orchestra.run.smithi033.stdout: "address": { 2024-03-19T03:03:36.191 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "address", 2024-03-19T03:03:36.191 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.191 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.192 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.192 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-19T03:03:36.192 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.192 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.192 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.192 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.192 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.192 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.192 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.192 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.192 INFO:teuthology.orchestra.run.smithi033.stdout: "interval": { 2024-03-19T03:03:36.192 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "interval", 2024-03-19T03:03:36.192 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:36.192 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.192 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.192 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "15", 2024-03-19T03:03:36.193 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.193 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.193 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.193 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.193 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.193 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.193 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.193 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.193 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.193 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.193 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.193 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.193 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.193 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.194 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.194 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.194 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.194 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.194 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.194 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.194 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.194 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.194 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.194 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.194 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.194 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.194 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.194 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.194 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.195 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.195 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.195 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.195 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.195 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.195 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.195 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.195 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.195 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.195 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.195 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.195 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.195 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.195 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.195 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.196 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.196 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.196 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.196 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.196 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.196 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.196 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.196 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.196 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.196 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.196 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.196 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.196 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.196 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.196 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.197 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.197 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.197 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.197 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.197 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.197 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.197 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.197 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.197 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.197 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.197 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.197 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.197 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.197 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.197 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.198 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.198 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.198 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.198 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.198 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.198 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.198 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.198 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "telemetry", 2024-03-19T03:03:36.198 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.198 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.198 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.198 INFO:teuthology.orchestra.run.smithi033.stdout: "channel_basic": { 2024-03-19T03:03:36.198 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "channel_basic", 2024-03-19T03:03:36.198 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.198 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.199 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.199 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.199 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.199 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.199 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.199 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-19T03:03:36.199 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.199 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.199 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.199 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.199 INFO:teuthology.orchestra.run.smithi033.stdout: "channel_crash": { 2024-03-19T03:03:36.199 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "channel_crash", 2024-03-19T03:03:36.199 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.199 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.199 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.200 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.200 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.200 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.200 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.200 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-19T03:03:36.200 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.200 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.200 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.200 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.200 INFO:teuthology.orchestra.run.smithi033.stdout: "channel_device": { 2024-03-19T03:03:36.200 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "channel_device", 2024-03-19T03:03:36.200 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.200 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.200 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.200 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.201 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.201 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.201 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.201 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-19T03:03:36.201 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.201 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.201 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.201 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.201 INFO:teuthology.orchestra.run.smithi033.stdout: "channel_ident": { 2024-03-19T03:03:36.201 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "channel_ident", 2024-03-19T03:03:36.201 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.201 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.201 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.201 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.201 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.201 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.202 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.202 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-19T03:03:36.202 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.202 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.202 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.202 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.202 INFO:teuthology.orchestra.run.smithi033.stdout: "channel_perf": { 2024-03-19T03:03:36.202 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "channel_perf", 2024-03-19T03:03:36.202 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.202 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.202 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.202 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.202 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.202 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.203 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.203 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-19T03:03:36.203 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.203 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.203 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.203 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.203 INFO:teuthology.orchestra.run.smithi033.stdout: "contact": { 2024-03-19T03:03:36.203 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "contact", 2024-03-19T03:03:36.203 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.203 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.203 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.203 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.203 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.203 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.203 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.204 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.204 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.204 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.204 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.204 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.204 INFO:teuthology.orchestra.run.smithi033.stdout: "description": { 2024-03-19T03:03:36.204 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "description", 2024-03-19T03:03:36.204 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.204 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.204 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.204 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.204 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.204 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.204 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.204 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.204 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.205 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.205 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.205 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.205 INFO:teuthology.orchestra.run.smithi033.stdout: "device_url": { 2024-03-19T03:03:36.205 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "device_url", 2024-03-19T03:03:36.205 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.205 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.205 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.205 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-19T03:03:36.205 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.205 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.205 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.205 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.205 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.205 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.206 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.206 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.206 INFO:teuthology.orchestra.run.smithi033.stdout: "enabled": { 2024-03-19T03:03:36.206 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "enabled", 2024-03-19T03:03:36.206 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.206 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.206 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.206 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.206 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.206 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.206 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.206 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.206 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.206 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.206 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.207 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.207 INFO:teuthology.orchestra.run.smithi033.stdout: "interval": { 2024-03-19T03:03:36.207 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "interval", 2024-03-19T03:03:36.207 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.207 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.207 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.207 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "24", 2024-03-19T03:03:36.207 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "8", 2024-03-19T03:03:36.207 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.207 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.207 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.207 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.207 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.207 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.207 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.208 INFO:teuthology.orchestra.run.smithi033.stdout: "last_opt_revision": { 2024-03-19T03:03:36.208 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "last_opt_revision", 2024-03-19T03:03:36.208 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.208 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.208 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.208 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "1", 2024-03-19T03:03:36.208 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.208 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.208 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.208 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.208 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.208 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.208 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.208 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.208 INFO:teuthology.orchestra.run.smithi033.stdout: "leaderboard": { 2024-03-19T03:03:36.209 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "leaderboard", 2024-03-19T03:03:36.209 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.209 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.209 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.209 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.209 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.209 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.209 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.209 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.209 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.209 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.209 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.209 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.209 INFO:teuthology.orchestra.run.smithi033.stdout: "leaderboard_description": { 2024-03-19T03:03:36.209 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "leaderboard_description", 2024-03-19T03:03:36.209 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.210 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.210 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.210 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.210 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.210 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.210 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.210 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.210 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.210 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.210 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.210 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.210 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.210 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.210 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.211 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.211 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.211 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.211 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.211 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.211 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.211 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.211 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.211 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.211 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.211 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.211 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.211 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.211 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.211 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.212 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.212 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.212 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.212 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.212 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.212 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.212 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.212 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.212 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.212 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.212 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.212 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.212 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.212 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.212 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.212 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.213 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.213 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.213 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.213 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.213 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.213 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.213 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.213 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.213 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.213 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.213 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.213 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.213 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.213 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.213 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.213 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.214 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.214 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.214 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.214 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.214 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.214 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.214 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.214 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.214 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.214 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.214 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.214 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.214 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.214 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.214 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.215 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.215 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.215 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.215 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.215 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.215 INFO:teuthology.orchestra.run.smithi033.stdout: "organization": { 2024-03-19T03:03:36.215 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "organization", 2024-03-19T03:03:36.215 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.215 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.215 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.215 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.215 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.215 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.215 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.215 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.215 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.216 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.216 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.216 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.216 INFO:teuthology.orchestra.run.smithi033.stdout: "proxy": { 2024-03-19T03:03:36.216 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "proxy", 2024-03-19T03:03:36.216 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.216 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.216 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.216 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.216 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.216 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.216 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.216 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.216 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.216 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.216 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.217 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.217 INFO:teuthology.orchestra.run.smithi033.stdout: "url": { 2024-03-19T03:03:36.217 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "url", 2024-03-19T03:03:36.217 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.217 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.217 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.217 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-19T03:03:36.217 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.217 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.217 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.217 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.217 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.217 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.217 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.217 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.217 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.218 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.218 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.218 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "test_orchestrator", 2024-03-19T03:03:36.218 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.218 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.218 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.218 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.218 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.218 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.218 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.218 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.218 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.218 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.218 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.219 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.219 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.219 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.219 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.219 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.219 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.219 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.219 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.219 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.219 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.219 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.219 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.219 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.219 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.219 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.220 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.220 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.220 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.220 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.220 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.220 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.220 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.220 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.220 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.220 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.220 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.220 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.220 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.220 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.220 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.221 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.221 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.221 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.221 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.221 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.221 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.221 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.221 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.221 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.221 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.221 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.221 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.221 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.221 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.221 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.222 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.222 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.222 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.222 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.222 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.222 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.222 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.222 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.222 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.222 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.222 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.222 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.222 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.222 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.223 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.223 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.223 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.223 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.223 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.223 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.223 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "volumes", 2024-03-19T03:03:36.223 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.223 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.223 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.223 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.223 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.223 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.223 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.223 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.224 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.224 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.224 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.224 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.224 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.224 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.224 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.224 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.224 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.224 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.224 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.224 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.224 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.224 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.224 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.225 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.225 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.225 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.225 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.225 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.225 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.225 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.225 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.225 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.225 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.225 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.225 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.225 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.225 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.225 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.225 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.226 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.226 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.226 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.226 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.226 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.226 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.226 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.226 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.226 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.226 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.226 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.226 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.226 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.226 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.227 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.227 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.227 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.227 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.227 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.227 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.227 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.227 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.227 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.227 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.227 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.227 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.227 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.227 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.227 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.227 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.228 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.228 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.228 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.228 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.228 INFO:teuthology.orchestra.run.smithi033.stdout: "max_concurrent_clones": { 2024-03-19T03:03:36.228 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "max_concurrent_clones", 2024-03-19T03:03:36.228 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.228 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.228 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.228 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "4", 2024-03-19T03:03:36.228 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.228 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.228 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.228 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-19T03:03:36.228 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.229 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.229 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.229 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.229 INFO:teuthology.orchestra.run.smithi033.stdout: "periodic_async_work": { 2024-03-19T03:03:36.229 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "periodic_async_work", 2024-03-19T03:03:36.229 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.229 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.229 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.229 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.229 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.229 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.229 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.229 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Periodically check for async work", 2024-03-19T03:03:36.229 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.229 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.229 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.230 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.230 INFO:teuthology.orchestra.run.smithi033.stdout: "snapshot_clone_delay": { 2024-03-19T03:03:36.230 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "snapshot_clone_delay", 2024-03-19T03:03:36.230 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.230 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.230 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.230 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0", 2024-03-19T03:03:36.230 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.230 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.230 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.230 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-19T03:03:36.230 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.230 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.230 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.231 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.231 INFO:teuthology.orchestra.run.smithi033.stdout: "snapshot_clone_no_wait": { 2024-03-19T03:03:36.231 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "snapshot_clone_no_wait", 2024-03-19T03:03:36.231 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.231 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.231 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.231 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.231 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.231 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.231 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.231 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-19T03:03:36.231 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.231 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.231 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.231 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.232 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.232 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.232 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.232 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "zabbix", 2024-03-19T03:03:36.232 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.232 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.232 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.232 INFO:teuthology.orchestra.run.smithi033.stdout: "discovery_interval": { 2024-03-19T03:03:36.232 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "discovery_interval", 2024-03-19T03:03:36.232 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "uint", 2024-03-19T03:03:36.232 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.232 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.232 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "100", 2024-03-19T03:03:36.232 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.232 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.233 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.233 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.233 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.233 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.233 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.233 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.233 INFO:teuthology.orchestra.run.smithi033.stdout: "identifier": { 2024-03-19T03:03:36.233 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "identifier", 2024-03-19T03:03:36.233 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.233 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.233 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.233 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.233 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.233 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.233 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.233 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.234 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.234 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.234 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.234 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.234 INFO:teuthology.orchestra.run.smithi033.stdout: "interval": { 2024-03-19T03:03:36.234 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "interval", 2024-03-19T03:03:36.234 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:36.234 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.234 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.234 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "60", 2024-03-19T03:03:36.234 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.234 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.234 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.234 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.235 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.235 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.235 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.235 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.235 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.235 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.235 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.235 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.235 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.235 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.235 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.235 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.235 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.235 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.235 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.236 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.236 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.236 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.236 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.236 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.236 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.236 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.236 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.236 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.236 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.236 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.236 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.236 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.236 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.236 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.237 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.237 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.237 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.237 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.237 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.237 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.237 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.237 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.237 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.237 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.237 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.237 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.237 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.237 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.238 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.238 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.238 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.238 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.238 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.238 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.238 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.238 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.238 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.238 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.238 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.238 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.238 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.238 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.238 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.239 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.239 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.239 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.239 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.239 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.239 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.239 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.239 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.239 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.239 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.239 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.239 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.239 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.239 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.239 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.239 INFO:teuthology.orchestra.run.smithi033.stdout: "zabbix_host": { 2024-03-19T03:03:36.240 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "zabbix_host", 2024-03-19T03:03:36.240 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.240 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.240 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.240 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.240 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.240 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.240 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.240 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.240 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.240 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.240 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.240 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.240 INFO:teuthology.orchestra.run.smithi033.stdout: "zabbix_port": { 2024-03-19T03:03:36.240 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "zabbix_port", 2024-03-19T03:03:36.241 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.241 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.241 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.241 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10051", 2024-03-19T03:03:36.241 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.241 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.241 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.241 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.241 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.241 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.241 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.241 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.241 INFO:teuthology.orchestra.run.smithi033.stdout: "zabbix_sender": { 2024-03-19T03:03:36.241 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "zabbix_sender", 2024-03-19T03:03:36.241 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.241 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.241 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.242 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-19T03:03:36.242 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.242 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.242 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.242 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.242 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.242 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.242 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.242 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.242 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.242 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.242 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-03-19T03:03:36.242 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.242 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.243 INFO:teuthology.orchestra.run.smithi033.stdout: "modules": [ 2024-03-19T03:03:36.243 INFO:teuthology.orchestra.run.smithi033.stdout: "cephadm", 2024-03-19T03:03:36.243 INFO:teuthology.orchestra.run.smithi033.stdout: "dashboard", 2024-03-19T03:03:36.243 INFO:teuthology.orchestra.run.smithi033.stdout: "iostat", 2024-03-19T03:03:36.243 INFO:teuthology.orchestra.run.smithi033.stdout: "nfs", 2024-03-19T03:03:36.243 INFO:teuthology.orchestra.run.smithi033.stdout: "restful", 2024-03-19T03:03:36.243 INFO:teuthology.orchestra.run.smithi033.stdout: "snap_schedule" 2024-03-19T03:03:36.243 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.243 INFO:teuthology.orchestra.run.smithi033.stdout: "available_modules": [ 2024-03-19T03:03:36.243 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.243 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "alerts", 2024-03-19T03:03:36.243 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.243 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.243 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.243 INFO:teuthology.orchestra.run.smithi033.stdout: "interval": { 2024-03-19T03:03:36.243 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "interval", 2024-03-19T03:03:36.244 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:36.244 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.244 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.244 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "60", 2024-03-19T03:03:36.244 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.244 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.244 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.244 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "How frequently to reexamine health status", 2024-03-19T03:03:36.244 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.244 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.244 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.244 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.244 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.244 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.244 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.245 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.245 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.245 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.245 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.245 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.245 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.245 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.245 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.245 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.245 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.245 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.245 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.245 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.245 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.245 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.246 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.246 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.246 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.246 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.246 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.246 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.246 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.246 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.246 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.246 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.246 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.246 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.246 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.246 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.247 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.247 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.247 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.247 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.247 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.247 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.247 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.247 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.247 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.247 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.247 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.247 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.247 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.247 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.247 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.247 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.248 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.248 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.248 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.248 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.248 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.248 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.248 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.248 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.248 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.248 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.248 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.248 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.248 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.248 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.248 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.248 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.249 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.249 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.249 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.249 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.249 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.249 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.249 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_destination": { 2024-03-19T03:03:36.249 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_destination", 2024-03-19T03:03:36.249 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.249 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.249 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.249 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.249 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.249 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.249 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.249 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Email address to send alerts to", 2024-03-19T03:03:36.250 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.250 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.250 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.250 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.250 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_from_name": { 2024-03-19T03:03:36.250 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_from_name", 2024-03-19T03:03:36.250 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.250 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.250 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.250 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "Ceph", 2024-03-19T03:03:36.250 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.250 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.250 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.250 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Email From: name", 2024-03-19T03:03:36.250 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.251 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.251 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.251 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.251 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_host": { 2024-03-19T03:03:36.251 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_host", 2024-03-19T03:03:36.251 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.251 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.251 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.251 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.251 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.251 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.251 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.251 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "SMTP server", 2024-03-19T03:03:36.251 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.251 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.252 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.252 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.252 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_password": { 2024-03-19T03:03:36.252 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_password", 2024-03-19T03:03:36.252 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.252 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.252 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.252 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.252 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.252 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.252 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.252 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Password to authenticate with", 2024-03-19T03:03:36.252 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.252 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.252 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.252 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.253 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_port": { 2024-03-19T03:03:36.253 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_port", 2024-03-19T03:03:36.253 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.253 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.253 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.253 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "465", 2024-03-19T03:03:36.253 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.253 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.253 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.253 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "SMTP port", 2024-03-19T03:03:36.253 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.253 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.253 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.253 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.253 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_sender": { 2024-03-19T03:03:36.253 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_sender", 2024-03-19T03:03:36.254 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.254 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.254 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.254 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.254 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.254 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.254 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.254 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "SMTP envelope sender", 2024-03-19T03:03:36.254 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.254 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.254 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.254 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.254 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_ssl": { 2024-03-19T03:03:36.254 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_ssl", 2024-03-19T03:03:36.255 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.255 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.255 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.255 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.255 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.255 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.255 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.255 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-19T03:03:36.255 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.255 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.255 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.255 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.255 INFO:teuthology.orchestra.run.smithi033.stdout: "smtp_user": { 2024-03-19T03:03:36.255 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "smtp_user", 2024-03-19T03:03:36.255 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.255 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.256 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.256 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.256 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.256 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.256 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.256 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "User to authenticate as", 2024-03-19T03:03:36.256 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.256 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.256 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.256 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.256 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.256 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.256 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.256 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "balancer", 2024-03-19T03:03:36.256 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.257 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.257 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.257 INFO:teuthology.orchestra.run.smithi033.stdout: "active": { 2024-03-19T03:03:36.257 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "active", 2024-03-19T03:03:36.257 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.257 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.257 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.257 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.257 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.257 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.257 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.257 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "automatically balance PGs across cluster", 2024-03-19T03:03:36.257 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.257 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.257 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.257 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.258 INFO:teuthology.orchestra.run.smithi033.stdout: "begin_time": { 2024-03-19T03:03:36.258 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "begin_time", 2024-03-19T03:03:36.258 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.258 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.258 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.258 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0000", 2024-03-19T03:03:36.258 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.258 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.258 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.258 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "beginning time of day to automatically balance", 2024-03-19T03:03:36.258 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-19T03:03:36.258 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.258 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.258 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.259 INFO:teuthology.orchestra.run.smithi033.stdout: "begin_weekday": { 2024-03-19T03:03:36.259 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "begin_weekday", 2024-03-19T03:03:36.259 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "uint", 2024-03-19T03:03:36.259 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.259 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.259 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0", 2024-03-19T03:03:36.259 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "0", 2024-03-19T03:03:36.259 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "6", 2024-03-19T03:03:36.259 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.259 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-19T03:03:36.259 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-19T03:03:36.259 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.259 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.259 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.259 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_compat_max_iterations": { 2024-03-19T03:03:36.259 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "crush_compat_max_iterations", 2024-03-19T03:03:36.260 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "uint", 2024-03-19T03:03:36.260 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.260 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.260 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "25", 2024-03-19T03:03:36.260 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "1", 2024-03-19T03:03:36.260 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "250", 2024-03-19T03:03:36.260 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.260 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-19T03:03:36.260 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.260 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.260 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.260 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.260 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_compat_metrics": { 2024-03-19T03:03:36.260 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "crush_compat_metrics", 2024-03-19T03:03:36.260 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.261 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.261 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.261 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "pgs,objects,bytes", 2024-03-19T03:03:36.261 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.261 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.261 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.261 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-19T03:03:36.261 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-19T03:03:36.261 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.261 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.261 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.261 INFO:teuthology.orchestra.run.smithi033.stdout: "crush_compat_step": { 2024-03-19T03:03:36.261 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "crush_compat_step", 2024-03-19T03:03:36.261 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-03-19T03:03:36.262 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.262 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.262 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0.5", 2024-03-19T03:03:36.262 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "0.001", 2024-03-19T03:03:36.262 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "0.999", 2024-03-19T03:03:36.262 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.262 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "aggressiveness of optimization", 2024-03-19T03:03:36.262 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-19T03:03:36.262 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.262 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.262 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.262 INFO:teuthology.orchestra.run.smithi033.stdout: "end_time": { 2024-03-19T03:03:36.262 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "end_time", 2024-03-19T03:03:36.262 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.263 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.263 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.263 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "2359", 2024-03-19T03:03:36.263 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.263 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.263 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.263 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "ending time of day to automatically balance", 2024-03-19T03:03:36.263 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-19T03:03:36.263 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.263 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.263 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.263 INFO:teuthology.orchestra.run.smithi033.stdout: "end_weekday": { 2024-03-19T03:03:36.263 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "end_weekday", 2024-03-19T03:03:36.263 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "uint", 2024-03-19T03:03:36.263 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.264 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.264 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0", 2024-03-19T03:03:36.264 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "0", 2024-03-19T03:03:36.264 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "6", 2024-03-19T03:03:36.264 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.264 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-19T03:03:36.264 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-19T03:03:36.264 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.264 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.264 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.264 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.264 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.264 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.264 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.264 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.265 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.265 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.265 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.265 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.265 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.265 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.265 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.265 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.265 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.265 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.265 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.265 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.265 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.265 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.265 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.266 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.266 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.266 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.266 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.266 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.266 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.266 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.266 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.266 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.266 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.266 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.266 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.266 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.266 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.266 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.266 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.267 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.267 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.267 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.267 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.267 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.267 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.267 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.267 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.267 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.267 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.267 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.267 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.267 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.267 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.267 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.267 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.268 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.268 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.268 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.268 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.268 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.268 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.268 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.268 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.268 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.268 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.268 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.268 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.268 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.268 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.268 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.268 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.269 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.269 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.269 INFO:teuthology.orchestra.run.smithi033.stdout: "min_score": { 2024-03-19T03:03:36.269 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "min_score", 2024-03-19T03:03:36.269 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-03-19T03:03:36.269 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.269 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.269 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0", 2024-03-19T03:03:36.269 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.269 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.269 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.269 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-19T03:03:36.269 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.269 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.269 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.269 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.270 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": { 2024-03-19T03:03:36.270 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mode", 2024-03-19T03:03:36.270 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.270 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.270 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.270 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "upmap", 2024-03-19T03:03:36.270 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.270 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.270 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.270 INFO:teuthology.orchestra.run.smithi033.stdout: "crush-compat", 2024-03-19T03:03:36.270 INFO:teuthology.orchestra.run.smithi033.stdout: "none", 2024-03-19T03:03:36.270 INFO:teuthology.orchestra.run.smithi033.stdout: "read", 2024-03-19T03:03:36.270 INFO:teuthology.orchestra.run.smithi033.stdout: "upmap", 2024-03-19T03:03:36.270 INFO:teuthology.orchestra.run.smithi033.stdout: "upmap-read" 2024-03-19T03:03:36.270 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.271 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Balancer mode", 2024-03-19T03:03:36.271 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.271 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.271 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.271 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.271 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_ids": { 2024-03-19T03:03:36.271 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "pool_ids", 2024-03-19T03:03:36.271 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.271 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.271 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.271 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.271 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.271 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.271 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.271 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-19T03:03:36.272 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.272 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.272 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.272 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.272 INFO:teuthology.orchestra.run.smithi033.stdout: "sleep_interval": { 2024-03-19T03:03:36.272 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "sleep_interval", 2024-03-19T03:03:36.272 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:36.272 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.272 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.272 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "60", 2024-03-19T03:03:36.272 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.272 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.272 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.272 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-19T03:03:36.272 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.273 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.273 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.273 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.273 INFO:teuthology.orchestra.run.smithi033.stdout: "upmap_max_deviation": { 2024-03-19T03:03:36.273 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "upmap_max_deviation", 2024-03-19T03:03:36.273 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.273 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.273 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.273 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5", 2024-03-19T03:03:36.273 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "1", 2024-03-19T03:03:36.273 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.273 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.273 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-19T03:03:36.273 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-19T03:03:36.273 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.274 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.274 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.274 INFO:teuthology.orchestra.run.smithi033.stdout: "upmap_max_optimizations": { 2024-03-19T03:03:36.274 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "upmap_max_optimizations", 2024-03-19T03:03:36.274 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "uint", 2024-03-19T03:03:36.274 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.274 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.274 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-03-19T03:03:36.274 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.274 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.274 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.274 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-19T03:03:36.274 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.274 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.275 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.275 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.275 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.275 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.275 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.275 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "cephadm", 2024-03-19T03:03:36.275 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.275 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.275 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.275 INFO:teuthology.orchestra.run.smithi033.stdout: "agent_down_multiplier": { 2024-03-19T03:03:36.275 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "agent_down_multiplier", 2024-03-19T03:03:36.275 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-03-19T03:03:36.275 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.275 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.275 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "3.0", 2024-03-19T03:03:36.276 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.276 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.276 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.276 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-19T03:03:36.276 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.276 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.276 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.276 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.276 INFO:teuthology.orchestra.run.smithi033.stdout: "agent_refresh_rate": { 2024-03-19T03:03:36.276 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "agent_refresh_rate", 2024-03-19T03:03:36.276 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:36.276 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.276 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.276 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "20", 2024-03-19T03:03:36.276 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.276 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.277 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.277 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-19T03:03:36.277 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.277 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.277 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.277 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.277 INFO:teuthology.orchestra.run.smithi033.stdout: "agent_starting_port": { 2024-03-19T03:03:36.277 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "agent_starting_port", 2024-03-19T03:03:36.277 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.277 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.277 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.277 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "4721", 2024-03-19T03:03:36.277 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.277 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.278 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.278 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-19T03:03:36.278 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.278 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.278 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.278 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.278 INFO:teuthology.orchestra.run.smithi033.stdout: "allow_ptrace": { 2024-03-19T03:03:36.278 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "allow_ptrace", 2024-03-19T03:03:36.278 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.278 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.278 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.278 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.278 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.278 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.278 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.278 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-19T03:03:36.278 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-19T03:03:36.279 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.279 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.279 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.279 INFO:teuthology.orchestra.run.smithi033.stdout: "autotune_interval": { 2024-03-19T03:03:36.279 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "autotune_interval", 2024-03-19T03:03:36.279 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:36.279 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.279 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.279 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "600", 2024-03-19T03:03:36.279 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.279 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.279 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.279 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-19T03:03:36.279 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.279 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.279 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.279 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.280 INFO:teuthology.orchestra.run.smithi033.stdout: "autotune_memory_target_ratio": { 2024-03-19T03:03:36.280 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "autotune_memory_target_ratio", 2024-03-19T03:03:36.280 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-03-19T03:03:36.280 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.280 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.280 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0.7", 2024-03-19T03:03:36.280 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.280 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.280 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.280 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-19T03:03:36.280 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.280 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.280 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.280 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.280 INFO:teuthology.orchestra.run.smithi033.stdout: "cephadm_log_destination": { 2024-03-19T03:03:36.281 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "cephadm_log_destination", 2024-03-19T03:03:36.281 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.281 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.281 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.281 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.281 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.281 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.281 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.281 INFO:teuthology.orchestra.run.smithi033.stdout: "file", 2024-03-19T03:03:36.281 INFO:teuthology.orchestra.run.smithi033.stdout: "file,syslog", 2024-03-19T03:03:36.281 INFO:teuthology.orchestra.run.smithi033.stdout: "syslog" 2024-03-19T03:03:36.281 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.281 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-19T03:03:36.281 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.281 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.281 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.282 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.282 INFO:teuthology.orchestra.run.smithi033.stdout: "cgroups_split": { 2024-03-19T03:03:36.282 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "cgroups_split", 2024-03-19T03:03:36.282 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.282 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.282 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.282 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.282 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.282 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.282 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.282 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-19T03:03:36.282 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.282 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.282 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.282 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.282 INFO:teuthology.orchestra.run.smithi033.stdout: "config_checks_enabled": { 2024-03-19T03:03:36.283 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "config_checks_enabled", 2024-03-19T03:03:36.283 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.283 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.283 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.283 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.283 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.283 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.283 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.283 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-19T03:03:36.283 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.283 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.283 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.283 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.283 INFO:teuthology.orchestra.run.smithi033.stdout: "config_dashboard": { 2024-03-19T03:03:36.283 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "config_dashboard", 2024-03-19T03:03:36.283 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.283 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.284 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.284 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.284 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.284 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.284 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.284 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-19T03:03:36.284 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.284 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.284 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.284 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.284 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_alertmanager": { 2024-03-19T03:03:36.284 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_alertmanager", 2024-03-19T03:03:36.284 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.284 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.284 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.284 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-19T03:03:36.285 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.285 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.285 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.285 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Prometheus container image", 2024-03-19T03:03:36.285 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.285 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.285 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.285 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.285 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_base": { 2024-03-19T03:03:36.285 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_base", 2024-03-19T03:03:36.285 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.285 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.285 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.285 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-19T03:03:36.285 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.286 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.286 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.286 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Container image name, without the tag", 2024-03-19T03:03:36.286 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.286 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.286 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.286 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.286 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_elasticsearch": { 2024-03-19T03:03:36.286 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_elasticsearch", 2024-03-19T03:03:36.286 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.286 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.286 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.286 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-19T03:03:36.286 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.287 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.287 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.287 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "elasticsearch container image", 2024-03-19T03:03:36.287 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.287 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.287 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.287 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.287 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_grafana": { 2024-03-19T03:03:36.287 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_grafana", 2024-03-19T03:03:36.287 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.287 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.287 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.287 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-19T03:03:36.287 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.287 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.287 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.288 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Prometheus container image", 2024-03-19T03:03:36.288 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.288 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.288 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.288 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.288 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_haproxy": { 2024-03-19T03:03:36.288 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_haproxy", 2024-03-19T03:03:36.288 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.288 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.288 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.288 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-19T03:03:36.288 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.288 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.288 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.288 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "HAproxy container image", 2024-03-19T03:03:36.288 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.289 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.289 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.289 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.289 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_jaeger_agent": { 2024-03-19T03:03:36.289 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_jaeger_agent", 2024-03-19T03:03:36.289 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.289 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.289 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.289 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-19T03:03:36.289 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.289 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.289 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.289 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Jaeger agent container image", 2024-03-19T03:03:36.289 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.289 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.289 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.290 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.290 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_jaeger_collector": { 2024-03-19T03:03:36.290 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_jaeger_collector", 2024-03-19T03:03:36.290 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.290 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.290 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.290 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-19T03:03:36.290 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.290 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.290 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.290 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Jaeger collector container image", 2024-03-19T03:03:36.290 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.290 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.290 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.290 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.290 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_jaeger_query": { 2024-03-19T03:03:36.291 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_jaeger_query", 2024-03-19T03:03:36.291 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.291 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.291 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.291 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-19T03:03:36.291 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.291 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.291 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.291 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Jaeger query container image", 2024-03-19T03:03:36.291 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.291 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.291 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.291 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.291 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_keepalived": { 2024-03-19T03:03:36.291 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_keepalived", 2024-03-19T03:03:36.292 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.292 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.292 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.292 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-19T03:03:36.292 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.292 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.292 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.292 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Keepalived container image", 2024-03-19T03:03:36.292 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.292 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.292 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.292 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.292 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_loki": { 2024-03-19T03:03:36.292 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_loki", 2024-03-19T03:03:36.292 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.292 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.293 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.293 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-19T03:03:36.293 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.293 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.293 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.293 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Loki container image", 2024-03-19T03:03:36.293 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.293 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.293 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.293 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.293 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_node_exporter": { 2024-03-19T03:03:36.293 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_node_exporter", 2024-03-19T03:03:36.293 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.293 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.293 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.294 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-19T03:03:36.294 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.294 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.294 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.294 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Prometheus container image", 2024-03-19T03:03:36.294 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.294 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.294 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.294 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.294 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_nvmeof": { 2024-03-19T03:03:36.294 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_nvmeof", 2024-03-19T03:03:36.294 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.294 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.294 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.294 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-19T03:03:36.294 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.295 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.295 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.295 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Nvme-of container image", 2024-03-19T03:03:36.295 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.295 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.295 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.295 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.295 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_prometheus": { 2024-03-19T03:03:36.295 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_prometheus", 2024-03-19T03:03:36.295 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.295 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.295 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.295 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-19T03:03:36.295 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.296 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.296 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.296 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Prometheus container image", 2024-03-19T03:03:36.296 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.296 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.296 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.296 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.296 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_promtail": { 2024-03-19T03:03:36.296 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_promtail", 2024-03-19T03:03:36.296 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.296 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.296 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.296 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-19T03:03:36.296 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.296 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.297 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.297 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Promtail container image", 2024-03-19T03:03:36.297 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.297 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.297 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.297 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.297 INFO:teuthology.orchestra.run.smithi033.stdout: "container_image_snmp_gateway": { 2024-03-19T03:03:36.297 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_image_snmp_gateway", 2024-03-19T03:03:36.297 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.297 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.297 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.297 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-19T03:03:36.297 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.297 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.297 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.298 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "SNMP Gateway container image", 2024-03-19T03:03:36.298 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.298 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.298 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.298 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.298 INFO:teuthology.orchestra.run.smithi033.stdout: "container_init": { 2024-03-19T03:03:36.298 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "container_init", 2024-03-19T03:03:36.298 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.298 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.298 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.298 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.298 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.298 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.298 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.299 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Run podman/docker with `--init`", 2024-03-19T03:03:36.299 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.299 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.299 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.299 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.299 INFO:teuthology.orchestra.run.smithi033.stdout: "daemon_cache_timeout": { 2024-03-19T03:03:36.299 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "daemon_cache_timeout", 2024-03-19T03:03:36.299 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:36.299 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.299 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.299 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "600", 2024-03-19T03:03:36.299 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.299 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.299 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.300 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-19T03:03:36.300 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.301 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.301 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.301 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.301 INFO:teuthology.orchestra.run.smithi033.stdout: "default_cephadm_command_timeout": { 2024-03-19T03:03:36.301 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "default_cephadm_command_timeout", 2024-03-19T03:03:36.301 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.301 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.301 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.301 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "900", 2024-03-19T03:03:36.301 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.301 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.301 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.301 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-19T03:03:36.301 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.302 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.302 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.302 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.302 INFO:teuthology.orchestra.run.smithi033.stdout: "default_registry": { 2024-03-19T03:03:36.302 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "default_registry", 2024-03-19T03:03:36.302 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.302 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.302 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.302 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "docker.io", 2024-03-19T03:03:36.302 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.302 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.302 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.302 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-19T03:03:36.303 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.303 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.303 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.303 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.303 INFO:teuthology.orchestra.run.smithi033.stdout: "device_cache_timeout": { 2024-03-19T03:03:36.303 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "device_cache_timeout", 2024-03-19T03:03:36.303 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:36.303 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.303 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.303 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "1800", 2024-03-19T03:03:36.303 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.303 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.303 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.303 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "seconds to cache device inventory", 2024-03-19T03:03:36.303 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.304 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.304 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.304 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.304 INFO:teuthology.orchestra.run.smithi033.stdout: "device_enhanced_scan": { 2024-03-19T03:03:36.304 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "device_enhanced_scan", 2024-03-19T03:03:36.304 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.304 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.304 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.304 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.304 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.304 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.304 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.304 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-19T03:03:36.304 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.304 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.305 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.305 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.305 INFO:teuthology.orchestra.run.smithi033.stdout: "facts_cache_timeout": { 2024-03-19T03:03:36.305 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "facts_cache_timeout", 2024-03-19T03:03:36.305 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:36.305 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.305 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.305 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "60", 2024-03-19T03:03:36.305 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.305 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.305 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.305 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "seconds to cache host facts data", 2024-03-19T03:03:36.305 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.305 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.305 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.306 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.306 INFO:teuthology.orchestra.run.smithi033.stdout: "host_check_interval": { 2024-03-19T03:03:36.306 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "host_check_interval", 2024-03-19T03:03:36.306 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:36.306 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.306 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.306 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "600", 2024-03-19T03:03:36.306 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.306 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.306 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.306 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how frequently to perform a host check", 2024-03-19T03:03:36.306 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.306 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.306 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.306 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.307 INFO:teuthology.orchestra.run.smithi033.stdout: "hw_monitoring": { 2024-03-19T03:03:36.307 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "hw_monitoring", 2024-03-19T03:03:36.307 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.307 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.307 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.307 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.307 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.307 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.307 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.307 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-19T03:03:36.307 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.307 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.307 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.307 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.307 INFO:teuthology.orchestra.run.smithi033.stdout: "inventory_list_all": { 2024-03-19T03:03:36.308 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "inventory_list_all", 2024-03-19T03:03:36.308 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.308 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.308 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.308 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.308 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.308 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.308 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.308 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-19T03:03:36.308 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.308 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.308 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.308 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.308 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.308 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.309 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.309 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.309 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.309 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.309 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.309 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.309 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.309 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.309 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.309 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.309 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.309 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.309 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.309 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.309 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.309 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.310 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.310 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.310 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.310 INFO:teuthology.orchestra.run.smithi033.stdout: "log_refresh_metadata": { 2024-03-19T03:03:36.310 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_refresh_metadata", 2024-03-19T03:03:36.310 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.310 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.310 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.310 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.310 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.310 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.310 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.310 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-19T03:03:36.310 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.310 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.311 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.311 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.311 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.311 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.311 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.311 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.311 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.311 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.311 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.311 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.311 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.311 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-19T03:03:36.311 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.311 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.311 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.311 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.312 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.312 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.312 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.312 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.312 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.312 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.312 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.312 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.312 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.312 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.312 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.312 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.312 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.312 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.312 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.313 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.313 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.313 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.313 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.313 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.313 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.313 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.313 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.313 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.313 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.313 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.313 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.313 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.313 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.313 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.314 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.314 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.314 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.314 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.314 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.314 INFO:teuthology.orchestra.run.smithi033.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-19T03:03:36.314 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-19T03:03:36.314 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.314 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.314 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.314 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.314 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.314 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.314 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.315 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-19T03:03:36.315 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.315 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.315 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.315 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.315 INFO:teuthology.orchestra.run.smithi033.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-19T03:03:36.315 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-19T03:03:36.315 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.315 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.315 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.315 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "*", 2024-03-19T03:03:36.315 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.315 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.315 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.316 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-19T03:03:36.316 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.316 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.316 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.316 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.316 INFO:teuthology.orchestra.run.smithi033.stdout: "max_count_per_host": { 2024-03-19T03:03:36.316 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "max_count_per_host", 2024-03-19T03:03:36.316 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.316 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.316 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.316 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-03-19T03:03:36.316 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.316 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.316 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.316 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "max number of daemons per service per host", 2024-03-19T03:03:36.316 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.317 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.317 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.317 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.317 INFO:teuthology.orchestra.run.smithi033.stdout: "max_osd_draining_count": { 2024-03-19T03:03:36.317 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "max_osd_draining_count", 2024-03-19T03:03:36.317 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.317 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.317 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.317 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-03-19T03:03:36.317 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.317 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.317 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.317 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-19T03:03:36.317 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.317 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.317 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.318 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.318 INFO:teuthology.orchestra.run.smithi033.stdout: "migration_current": { 2024-03-19T03:03:36.318 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "migration_current", 2024-03-19T03:03:36.318 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.318 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.318 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.318 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.318 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.318 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.318 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.318 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "internal - do not modify", 2024-03-19T03:03:36.318 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.318 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.318 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.318 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.319 INFO:teuthology.orchestra.run.smithi033.stdout: "mode": { 2024-03-19T03:03:36.319 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mode", 2024-03-19T03:03:36.319 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.319 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.319 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.319 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "root", 2024-03-19T03:03:36.319 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.319 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.319 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.319 INFO:teuthology.orchestra.run.smithi033.stdout: "cephadm-package", 2024-03-19T03:03:36.319 INFO:teuthology.orchestra.run.smithi033.stdout: "root" 2024-03-19T03:03:36.319 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.319 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "mode for remote execution of cephadm", 2024-03-19T03:03:36.319 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.319 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.319 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.320 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.320 INFO:teuthology.orchestra.run.smithi033.stdout: "oob_default_addr": { 2024-03-19T03:03:36.320 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "oob_default_addr", 2024-03-19T03:03:36.320 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.320 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.320 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.320 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "169.254.1.1", 2024-03-19T03:03:36.320 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.320 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.320 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.320 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-19T03:03:36.320 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.320 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.320 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.320 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.320 INFO:teuthology.orchestra.run.smithi033.stdout: "prometheus_alerts_path": { 2024-03-19T03:03:36.321 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "prometheus_alerts_path", 2024-03-19T03:03:36.321 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.321 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.321 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.321 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-19T03:03:36.321 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.321 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.321 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.321 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-19T03:03:36.321 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.321 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.321 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.321 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.321 INFO:teuthology.orchestra.run.smithi033.stdout: "registry_insecure": { 2024-03-19T03:03:36.322 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "registry_insecure", 2024-03-19T03:03:36.322 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.322 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.322 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.322 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.322 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.322 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.322 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.322 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-19T03:03:36.322 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.322 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.322 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.322 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.322 INFO:teuthology.orchestra.run.smithi033.stdout: "registry_password": { 2024-03-19T03:03:36.323 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "registry_password", 2024-03-19T03:03:36.323 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.323 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.323 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.323 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.323 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.323 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.323 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.323 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-19T03:03:36.323 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.323 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.323 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.323 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.323 INFO:teuthology.orchestra.run.smithi033.stdout: "registry_url": { 2024-03-19T03:03:36.323 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "registry_url", 2024-03-19T03:03:36.324 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.324 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.324 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.324 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.324 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.324 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.324 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.324 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-19T03:03:36.324 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.324 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.324 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.324 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.324 INFO:teuthology.orchestra.run.smithi033.stdout: "registry_username": { 2024-03-19T03:03:36.324 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "registry_username", 2024-03-19T03:03:36.324 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.325 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.325 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.325 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.325 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.325 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.325 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.325 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-19T03:03:36.325 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.325 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.325 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.325 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.325 INFO:teuthology.orchestra.run.smithi033.stdout: "secure_monitoring_stack": { 2024-03-19T03:03:36.325 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "secure_monitoring_stack", 2024-03-19T03:03:36.325 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.325 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.326 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.326 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.326 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.326 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.326 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.326 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-19T03:03:36.326 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.326 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.326 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.326 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.326 INFO:teuthology.orchestra.run.smithi033.stdout: "service_discovery_port": { 2024-03-19T03:03:36.326 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "service_discovery_port", 2024-03-19T03:03:36.326 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.326 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.326 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.327 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "8765", 2024-03-19T03:03:36.327 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.327 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.327 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.327 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "cephadm service discovery port", 2024-03-19T03:03:36.327 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.327 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.327 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.327 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.327 INFO:teuthology.orchestra.run.smithi033.stdout: "ssh_config_file": { 2024-03-19T03:03:36.327 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ssh_config_file", 2024-03-19T03:03:36.327 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.327 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.327 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.327 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.328 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.328 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.328 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.328 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-19T03:03:36.328 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.328 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.328 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.328 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.328 INFO:teuthology.orchestra.run.smithi033.stdout: "use_agent": { 2024-03-19T03:03:36.328 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "use_agent", 2024-03-19T03:03:36.328 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.328 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.328 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.328 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.328 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.329 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.329 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.329 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-19T03:03:36.329 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.329 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.329 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.329 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.329 INFO:teuthology.orchestra.run.smithi033.stdout: "use_repo_digest": { 2024-03-19T03:03:36.329 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "use_repo_digest", 2024-03-19T03:03:36.329 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.329 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.329 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.329 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.329 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.330 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.330 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.330 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-19T03:03:36.330 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.330 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.330 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.330 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.330 INFO:teuthology.orchestra.run.smithi033.stdout: "warn_on_failed_host_check": { 2024-03-19T03:03:36.330 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "warn_on_failed_host_check", 2024-03-19T03:03:36.330 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.330 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.330 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.330 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.330 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.330 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.331 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.331 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "raise a health warning if the host check fails", 2024-03-19T03:03:36.331 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.331 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.331 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.331 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.331 INFO:teuthology.orchestra.run.smithi033.stdout: "warn_on_stray_daemons": { 2024-03-19T03:03:36.331 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "warn_on_stray_daemons", 2024-03-19T03:03:36.331 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.331 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.331 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.331 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.331 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.331 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.331 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.332 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-19T03:03:36.332 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.332 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.332 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.332 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.332 INFO:teuthology.orchestra.run.smithi033.stdout: "warn_on_stray_hosts": { 2024-03-19T03:03:36.332 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "warn_on_stray_hosts", 2024-03-19T03:03:36.332 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.332 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.332 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.332 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.332 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.332 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.332 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.333 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-19T03:03:36.333 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.333 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.333 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.333 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.333 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.333 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.333 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.333 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "crash", 2024-03-19T03:03:36.333 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.333 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.333 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.333 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.333 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.333 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.334 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.334 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.334 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.334 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.334 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.334 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.334 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.334 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.334 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.334 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.334 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.334 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.334 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.334 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.335 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.335 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.335 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.335 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.335 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.335 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.335 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.335 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.335 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.335 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.335 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.335 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.335 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.335 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.335 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.335 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.336 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.336 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.336 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.336 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.336 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.336 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.336 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.336 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.336 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.336 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.336 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.336 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.336 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.336 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.336 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.336 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.337 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.337 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.337 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.337 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.337 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.337 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.337 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.337 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.337 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.337 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.337 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.337 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.337 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.337 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.337 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.338 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.338 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.338 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.338 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.338 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.338 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.338 INFO:teuthology.orchestra.run.smithi033.stdout: "retain_interval": { 2024-03-19T03:03:36.338 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "retain_interval", 2024-03-19T03:03:36.338 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:36.338 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.338 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.338 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "31536000", 2024-03-19T03:03:36.338 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.338 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.339 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.339 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-19T03:03:36.339 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.339 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.339 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.339 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.339 INFO:teuthology.orchestra.run.smithi033.stdout: "warn_recent_interval": { 2024-03-19T03:03:36.339 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "warn_recent_interval", 2024-03-19T03:03:36.339 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:36.339 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.339 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.339 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "1209600", 2024-03-19T03:03:36.339 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.339 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.339 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.339 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-19T03:03:36.339 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.340 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.340 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.340 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.340 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.340 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.340 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.340 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "dashboard", 2024-03-19T03:03:36.340 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.340 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.340 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.340 INFO:teuthology.orchestra.run.smithi033.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-19T03:03:36.340 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-19T03:03:36.340 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.340 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.340 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.340 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-03-19T03:03:36.341 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.341 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.341 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.341 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.341 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.341 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.341 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.341 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.341 INFO:teuthology.orchestra.run.smithi033.stdout: "ALERTMANAGER_API_HOST": { 2024-03-19T03:03:36.341 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-19T03:03:36.341 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.341 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.341 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.341 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.341 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.341 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.342 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.342 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.342 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.342 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.342 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.342 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.342 INFO:teuthology.orchestra.run.smithi033.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-19T03:03:36.342 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-19T03:03:36.342 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.342 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.342 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.342 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.342 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.342 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.343 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.343 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.343 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.343 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.343 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.343 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.343 INFO:teuthology.orchestra.run.smithi033.stdout: "AUDIT_API_ENABLED": { 2024-03-19T03:03:36.343 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "AUDIT_API_ENABLED", 2024-03-19T03:03:36.343 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.343 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.343 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.343 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.343 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.343 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.343 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.344 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.344 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.344 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.344 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.344 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.344 INFO:teuthology.orchestra.run.smithi033.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-19T03:03:36.344 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-19T03:03:36.344 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.344 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.344 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.344 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.344 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.344 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.344 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.344 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.345 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.345 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.345 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.345 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.345 INFO:teuthology.orchestra.run.smithi033.stdout: "ENABLE_BROWSABLE_API": { 2024-03-19T03:03:36.345 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-19T03:03:36.345 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.345 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.345 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.345 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.345 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.345 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.345 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.345 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.345 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.346 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.346 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.346 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.346 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-19T03:03:36.346 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-19T03:03:36.346 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.346 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.346 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.346 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.346 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.346 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.346 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.346 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.346 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.347 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.347 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.347 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.347 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-19T03:03:36.347 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-19T03:03:36.347 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.347 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.347 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.347 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.347 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.347 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.347 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.347 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.347 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.347 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.348 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.348 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.348 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-19T03:03:36.348 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-19T03:03:36.348 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.348 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.348 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.348 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.348 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.348 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.348 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.348 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.348 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.348 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.348 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.348 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.349 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-19T03:03:36.349 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-19T03:03:36.349 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.349 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.349 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.349 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.349 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.349 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.349 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.349 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.349 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.349 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.349 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.349 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.349 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-19T03:03:36.350 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-19T03:03:36.350 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.350 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.350 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.350 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.350 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.350 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.350 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.350 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.350 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.350 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.350 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.350 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.350 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-19T03:03:36.350 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-19T03:03:36.350 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.351 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.351 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.351 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.351 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.351 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.351 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.351 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.351 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.351 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.351 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.351 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.351 INFO:teuthology.orchestra.run.smithi033.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-19T03:03:36.351 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-19T03:03:36.351 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.351 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.351 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.352 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.352 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.352 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.352 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.352 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.352 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.352 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.352 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.352 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.352 INFO:teuthology.orchestra.run.smithi033.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-19T03:03:36.352 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-19T03:03:36.352 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.352 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.352 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.352 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.353 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.353 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.353 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.353 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.353 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.353 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.353 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.353 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.353 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_API_PASSWORD": { 2024-03-19T03:03:36.353 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-19T03:03:36.353 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.353 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.353 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.353 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "admin", 2024-03-19T03:03:36.353 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.353 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.354 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.354 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.354 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.354 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.354 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.354 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.354 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-19T03:03:36.354 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-19T03:03:36.354 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.354 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.354 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.354 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.354 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.354 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.354 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.355 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.355 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.355 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.355 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.355 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.355 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_API_URL": { 2024-03-19T03:03:36.355 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_API_URL", 2024-03-19T03:03:36.355 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.355 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.355 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.355 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.355 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.355 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.355 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.355 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.355 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.356 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.356 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.356 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.356 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_API_USERNAME": { 2024-03-19T03:03:36.356 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-19T03:03:36.356 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.356 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.356 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.356 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "admin", 2024-03-19T03:03:36.356 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.356 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.356 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.356 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.356 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.356 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.356 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.357 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.357 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-19T03:03:36.357 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-19T03:03:36.357 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.357 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.357 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.357 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.357 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.357 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.357 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.357 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.357 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.357 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.357 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.357 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.357 INFO:teuthology.orchestra.run.smithi033.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-19T03:03:36.358 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-19T03:03:36.358 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.358 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.358 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.358 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.358 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:35 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3641712955' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:03:36.358 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:35 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1030361426' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:03:36.358 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:35 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/1636855123' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:03:36.359 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:35 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3641712955' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:03:36.359 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:35 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1030361426' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:03:36.359 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:35 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/1636855123' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:03:36.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.422+0000 7f81c07bc640 1 -- 172.21.15.33:0/621911649 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f81a40a2970 msgr2=0x7f81a40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:36.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.422+0000 7f81c07bc640 1 --2- 172.21.15.33:0/621911649 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f81a40a2970 0x7f81a40a4e30 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f81b4005ab0 tx=0x7f81b4005fd0 comp rx=0 tx=0).stop 2024-03-19T03:03:36.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.422+0000 7f81c07bc640 1 -- 172.21.15.33:0/621911649 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f81b81066e0 msgr2=0x7f81b819c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:03:36.359 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.422+0000 7f81c07bc640 1 --2- 172.21.15.33:0/621911649 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f81b81066e0 0x7f81b819c8a0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f81b000efd0 tx=0x7f81b000c6a0 comp rx=0 tx=0).stop 2024-03-19T03:03:36.359 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.359 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.360 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.360 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.360 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.360 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.360 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.360 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.360 INFO:teuthology.orchestra.run.smithi033.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-19T03:03:36.360 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-19T03:03:36.360 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.360 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.360 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.360 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.360 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.360 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.360 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.361 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.361 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.361 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.361 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.361 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.361 INFO:teuthology.orchestra.run.smithi033.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-19T03:03:36.361 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-19T03:03:36.361 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.361 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.361 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.361 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.361 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.361 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.361 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.361 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.361 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.362 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.362 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.362 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.362 INFO:teuthology.orchestra.run.smithi033.stdout: "MULTICLUSTER_CONFIG": { 2024-03-19T03:03:36.362 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-19T03:03:36.362 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.362 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.362 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.362 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "{}", 2024-03-19T03:03:36.362 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.362 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.362 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.362 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.362 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.362 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.363 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.363 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.363 INFO:teuthology.orchestra.run.smithi033.stdout: "PROMETHEUS_API_HOST": { 2024-03-19T03:03:36.363 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-19T03:03:36.363 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.363 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.363 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.363 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.363 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.363 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.363 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.363 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.363 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.363 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.363 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.363 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.364 INFO:teuthology.orchestra.run.smithi033.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-19T03:03:36.364 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-19T03:03:36.364 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.364 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.364 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.364 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.364 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.364 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.364 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.364 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.364 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.364 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.364 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.364 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.364 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-19T03:03:36.365 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-19T03:03:36.365 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.365 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.365 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.365 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.365 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.365 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.365 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.365 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.365 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.365 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.365 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.365 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.365 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-19T03:03:36.365 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-19T03:03:36.366 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.366 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.366 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.366 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.366 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.366 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.366 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.366 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.366 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.366 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.366 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.366 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.366 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-19T03:03:36.366 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-19T03:03:36.367 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.367 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.367 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.367 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.367 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.367 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.367 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.367 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.367 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.367 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.367 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.367 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.367 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-19T03:03:36.367 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-19T03:03:36.367 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.368 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.368 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.368 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.368 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.368 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.368 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.368 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.368 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.368 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.368 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.368 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.368 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-19T03:03:36.368 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-19T03:03:36.368 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.368 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.369 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.369 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.369 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.369 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.369 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.369 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.369 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.369 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.369 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.369 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.369 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-19T03:03:36.369 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-19T03:03:36.369 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.369 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.369 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.370 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.370 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.370 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.370 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.370 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.370 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.370 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.370 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.370 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.370 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-19T03:03:36.370 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-19T03:03:36.370 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.370 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.370 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.371 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.371 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.371 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.371 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.371 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.371 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.371 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.371 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.371 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.371 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_ENABLED": { 2024-03-19T03:03:36.371 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-19T03:03:36.371 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.372 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.372 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.372 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.372 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.372 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.372 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.372 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.372 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.372 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.372 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.372 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.372 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-19T03:03:36.372 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-19T03:03:36.372 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.372 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.373 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.373 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-19T03:03:36.373 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.373 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.373 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.373 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.373 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.373 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.373 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.373 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.373 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-19T03:03:36.373 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-19T03:03:36.373 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.373 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.373 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.373 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-03-19T03:03:36.374 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.374 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.374 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.374 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.374 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.374 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.374 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.374 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.374 INFO:teuthology.orchestra.run.smithi033.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-19T03:03:36.374 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-19T03:03:36.374 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.374 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.374 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.374 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "8", 2024-03-19T03:03:36.375 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.375 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.375 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.375 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.375 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.375 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.375 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.375 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.375 INFO:teuthology.orchestra.run.smithi033.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-19T03:03:36.375 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-19T03:03:36.375 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.375 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.375 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.375 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "45", 2024-03-19T03:03:36.375 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.376 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.376 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.376 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.376 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.376 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.376 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.376 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.376 INFO:teuthology.orchestra.run.smithi033.stdout: "RGW_API_ACCESS_KEY": { 2024-03-19T03:03:36.376 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-19T03:03:36.376 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.376 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.376 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.376 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.376 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.377 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.377 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.377 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.377 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.377 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.377 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.377 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.377 INFO:teuthology.orchestra.run.smithi033.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-19T03:03:36.377 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-19T03:03:36.377 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.377 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.377 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.377 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "admin", 2024-03-19T03:03:36.377 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.377 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.378 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.378 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.378 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.378 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.378 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.378 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.378 INFO:teuthology.orchestra.run.smithi033.stdout: "RGW_API_SECRET_KEY": { 2024-03-19T03:03:36.378 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-19T03:03:36.378 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.378 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.378 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.378 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.378 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.378 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.379 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.379 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.379 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.379 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.379 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.379 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.379 INFO:teuthology.orchestra.run.smithi033.stdout: "RGW_API_SSL_VERIFY": { 2024-03-19T03:03:36.379 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-19T03:03:36.379 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.379 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.379 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.379 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.379 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.379 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.380 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.380 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.380 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.380 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.380 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.380 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.380 INFO:teuthology.orchestra.run.smithi033.stdout: "UNSAFE_TLS_v1_2": { 2024-03-19T03:03:36.380 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-19T03:03:36.380 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.380 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.380 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.380 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.380 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.380 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.380 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.381 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.381 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.381 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.381 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.381 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.381 INFO:teuthology.orchestra.run.smithi033.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-19T03:03:36.381 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-19T03:03:36.381 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.381 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.381 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.381 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0", 2024-03-19T03:03:36.381 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.381 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.381 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.381 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.382 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.382 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.382 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.382 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.382 INFO:teuthology.orchestra.run.smithi033.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-19T03:03:36.382 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-19T03:03:36.382 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.382 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.382 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.382 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-03-19T03:03:36.382 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.382 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.382 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.382 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.382 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.383 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.383 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.383 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.383 INFO:teuthology.orchestra.run.smithi033.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-19T03:03:36.383 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-19T03:03:36.383 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.383 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.383 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.383 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5", 2024-03-19T03:03:36.383 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.383 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.383 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.383 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.383 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.383 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.384 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.384 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.384 INFO:teuthology.orchestra.run.smithi033.stdout: "cross_origin_url": { 2024-03-19T03:03:36.384 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "cross_origin_url", 2024-03-19T03:03:36.384 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.384 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.384 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.384 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.384 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.384 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.384 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.384 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.384 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.384 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.384 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.385 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.385 INFO:teuthology.orchestra.run.smithi033.stdout: "crt_file": { 2024-03-19T03:03:36.385 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "crt_file", 2024-03-19T03:03:36.385 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.385 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.385 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.385 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.385 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.385 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.385 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.385 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.385 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.385 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.385 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.385 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.385 INFO:teuthology.orchestra.run.smithi033.stdout: "debug": { 2024-03-19T03:03:36.386 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "debug", 2024-03-19T03:03:36.386 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.386 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.386 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.386 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.386 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.386 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.386 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.386 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Enable/disable debug options", 2024-03-19T03:03:36.386 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.386 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.386 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.386 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.386 INFO:teuthology.orchestra.run.smithi033.stdout: "jwt_token_ttl": { 2024-03-19T03:03:36.386 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "jwt_token_ttl", 2024-03-19T03:03:36.387 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.387 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.387 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.387 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "28800", 2024-03-19T03:03:36.387 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.387 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.387 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.387 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.387 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.387 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.387 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.387 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.387 INFO:teuthology.orchestra.run.smithi033.stdout: "key_file": { 2024-03-19T03:03:36.388 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "key_file", 2024-03-19T03:03:36.388 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.388 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.388 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.388 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.388 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.388 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.388 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.388 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.388 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.388 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.388 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.388 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.388 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.388 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.389 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.389 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.389 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.389 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.389 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.389 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.389 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.389 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.389 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.389 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.389 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.389 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.389 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.389 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.389 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.389 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.390 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.390 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.390 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.390 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.390 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.390 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.390 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.390 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.390 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.390 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.390 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.390 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.390 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.390 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.391 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.391 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.391 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.391 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.391 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.391 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.391 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.391 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.391 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.391 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.391 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.391 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.391 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.391 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.391 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.391 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.392 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.392 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.392 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.392 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.392 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.392 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.392 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.392 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.392 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.392 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.392 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.392 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.392 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.392 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.392 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.392 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.392 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.393 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.393 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.393 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.393 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.393 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.393 INFO:teuthology.orchestra.run.smithi033.stdout: "motd": { 2024-03-19T03:03:36.393 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "motd", 2024-03-19T03:03:36.393 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.393 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.393 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.393 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.393 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.393 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.393 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.393 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "The message of the day", 2024-03-19T03:03:36.393 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.394 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.394 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.394 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.394 INFO:teuthology.orchestra.run.smithi033.stdout: "redirect_resolve_ip_addr": { 2024-03-19T03:03:36.394 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "redirect_resolve_ip_addr", 2024-03-19T03:03:36.394 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.394 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.394 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.394 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.394 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.394 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.394 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.394 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.394 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.395 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.395 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.395 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.395 INFO:teuthology.orchestra.run.smithi033.stdout: "server_addr": { 2024-03-19T03:03:36.395 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_addr", 2024-03-19T03:03:36.395 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.395 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.395 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.395 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "::", 2024-03-19T03:03:36.395 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.395 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.395 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.395 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.395 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.395 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.395 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.395 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.396 INFO:teuthology.orchestra.run.smithi033.stdout: "server_port": { 2024-03-19T03:03:36.396 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_port", 2024-03-19T03:03:36.396 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.396 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.396 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.396 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "8080", 2024-03-19T03:03:36.396 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.396 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.396 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.396 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.396 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.396 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.396 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.396 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.396 INFO:teuthology.orchestra.run.smithi033.stdout: "ssl": { 2024-03-19T03:03:36.396 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ssl", 2024-03-19T03:03:36.397 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.397 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.397 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.397 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.397 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.397 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.397 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.397 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.397 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.397 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.397 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.397 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.397 INFO:teuthology.orchestra.run.smithi033.stdout: "ssl_server_port": { 2024-03-19T03:03:36.397 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ssl_server_port", 2024-03-19T03:03:36.397 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.398 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.398 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.398 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "8443", 2024-03-19T03:03:36.398 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.398 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.398 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.398 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.398 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.398 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.398 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.398 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.398 INFO:teuthology.orchestra.run.smithi033.stdout: "standby_behaviour": { 2024-03-19T03:03:36.398 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "standby_behaviour", 2024-03-19T03:03:36.398 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.398 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.399 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.399 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "redirect", 2024-03-19T03:03:36.399 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.399 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.399 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.399 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.399 INFO:teuthology.orchestra.run.smithi033.stdout: "redirect" 2024-03-19T03:03:36.399 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.399 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.399 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.399 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.399 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.399 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.399 INFO:teuthology.orchestra.run.smithi033.stdout: "standby_error_status_code": { 2024-03-19T03:03:36.399 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "standby_error_status_code", 2024-03-19T03:03:36.399 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.400 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.400 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.400 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "500", 2024-03-19T03:03:36.400 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "400", 2024-03-19T03:03:36.400 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "599", 2024-03-19T03:03:36.400 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.400 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.400 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.400 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.400 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.400 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.400 INFO:teuthology.orchestra.run.smithi033.stdout: "url_prefix": { 2024-03-19T03:03:36.400 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "url_prefix", 2024-03-19T03:03:36.400 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.400 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.400 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.401 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.401 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.401 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.401 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.401 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.401 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.401 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.401 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.401 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.401 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.401 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.401 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.401 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "devicehealth", 2024-03-19T03:03:36.401 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.401 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.401 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.402 INFO:teuthology.orchestra.run.smithi033.stdout: "enable_monitoring": { 2024-03-19T03:03:36.402 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "enable_monitoring", 2024-03-19T03:03:36.402 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.402 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.402 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.402 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.402 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.402 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.402 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.402 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "monitor device health metrics", 2024-03-19T03:03:36.402 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.402 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.402 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.402 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.402 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.403 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.403 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.403 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.403 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.403 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.403 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.403 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.403 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.403 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.403 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.403 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.403 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.403 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.403 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.403 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.403 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.404 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.404 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.404 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.404 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.404 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.404 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.404 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.404 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.404 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.404 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.404 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.404 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.404 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.404 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.404 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.404 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.405 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.405 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.405 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.405 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.405 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.405 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.405 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.405 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.405 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.405 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.405 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.405 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.405 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.405 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.405 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.405 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.406 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.406 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.406 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.406 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.406 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.406 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.406 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.406 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.406 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.406 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.406 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.406 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.406 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.406 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.406 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.407 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.407 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.407 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.407 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.407 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.407 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.407 INFO:teuthology.orchestra.run.smithi033.stdout: "mark_out_threshold": { 2024-03-19T03:03:36.407 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mark_out_threshold", 2024-03-19T03:03:36.407 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:36.407 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.407 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.407 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "2419200", 2024-03-19T03:03:36.407 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.407 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.407 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.407 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-19T03:03:36.408 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.408 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.408 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.408 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.408 INFO:teuthology.orchestra.run.smithi033.stdout: "pool_name": { 2024-03-19T03:03:36.408 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "pool_name", 2024-03-19T03:03:36.408 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.408 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.408 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.408 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "device_health_metrics", 2024-03-19T03:03:36.408 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.408 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.408 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.408 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-19T03:03:36.408 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.408 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.409 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.409 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.409 INFO:teuthology.orchestra.run.smithi033.stdout: "retention_period": { 2024-03-19T03:03:36.409 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "retention_period", 2024-03-19T03:03:36.409 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:36.409 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.409 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.409 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "15552000", 2024-03-19T03:03:36.409 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.409 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.409 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.409 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how long to retain device health metrics", 2024-03-19T03:03:36.409 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.409 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.410 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.410 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.410 INFO:teuthology.orchestra.run.smithi033.stdout: "scrape_frequency": { 2024-03-19T03:03:36.410 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "scrape_frequency", 2024-03-19T03:03:36.410 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:36.410 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.410 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.410 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "86400", 2024-03-19T03:03:36.410 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.410 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.410 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.410 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-19T03:03:36.410 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.410 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.411 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.411 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.411 INFO:teuthology.orchestra.run.smithi033.stdout: "self_heal": { 2024-03-19T03:03:36.411 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "self_heal", 2024-03-19T03:03:36.411 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.411 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.411 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.411 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.411 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.411 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.411 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.411 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-19T03:03:36.411 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.411 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.411 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.411 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.412 INFO:teuthology.orchestra.run.smithi033.stdout: "sleep_interval": { 2024-03-19T03:03:36.412 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "sleep_interval", 2024-03-19T03:03:36.412 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:36.412 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.412 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.412 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "600", 2024-03-19T03:03:36.412 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.412 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.412 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.412 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how frequently to wake up and check device health", 2024-03-19T03:03:36.412 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.412 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.412 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.412 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.412 INFO:teuthology.orchestra.run.smithi033.stdout: "warn_threshold": { 2024-03-19T03:03:36.413 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "warn_threshold", 2024-03-19T03:03:36.413 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:36.413 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.413 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.413 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "7257600", 2024-03-19T03:03:36.413 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.413 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.413 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.413 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-19T03:03:36.413 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.413 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.413 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.413 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.413 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.413 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.414 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.414 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "diskprediction_local", 2024-03-19T03:03:36.414 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.414 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.414 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.414 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.414 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.414 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.414 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.414 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.414 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.414 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.414 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.414 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.414 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.414 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.415 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.415 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.415 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.415 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.415 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.415 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.415 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.415 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.415 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.415 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.415 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.415 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.415 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.415 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.415 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.415 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.415 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.416 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.416 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.416 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.416 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.416 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.416 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.416 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.416 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.416 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.416 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.416 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.416 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.416 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.416 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.416 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.417 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.417 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.417 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.417 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.417 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.417 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.417 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.417 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.417 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.417 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.417 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.417 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.417 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.417 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.417 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.417 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.418 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.418 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.418 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.418 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.418 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.418 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.418 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.418 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.418 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.418 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.418 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.418 INFO:teuthology.orchestra.run.smithi033.stdout: "predict_interval": { 2024-03-19T03:03:36.418 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "predict_interval", 2024-03-19T03:03:36.418 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.418 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.419 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.419 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "86400", 2024-03-19T03:03:36.419 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.419 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.419 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.419 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.419 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.419 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.419 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.419 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.419 INFO:teuthology.orchestra.run.smithi033.stdout: "predictor_model": { 2024-03-19T03:03:36.419 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "predictor_model", 2024-03-19T03:03:36.419 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.419 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.419 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.422+0000 7f81c07bc640 1 -- 172.21.15.33:0/621911649 shutdown_connections 2024-03-19T03:03:36.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.422+0000 7f81c07bc640 1 --2- 172.21.15.33:0/621911649 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f81a40a2970 0x7f81a40a4e30 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:36.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.422+0000 7f81c07bc640 1 --2- 172.21.15.33:0/621911649 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81b8101090 0x7f81b8198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:36.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.422+0000 7f81c07bc640 1 --2- 172.21.15.33:0/621911649 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f81b81066e0 0x7f81b819c8a0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:36.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.422+0000 7f81c07bc640 1 --2- 172.21.15.33:0/621911649 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f81b81006d0 0x7f81b8198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:03:36.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.422+0000 7f81c07bc640 1 -- 172.21.15.33:0/621911649 >> 172.21.15.33:0/621911649 conn(0x7f81b8076aa0 msgr2=0x7f81b806a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:03:36.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.422+0000 7f81c07bc640 1 -- 172.21.15.33:0/621911649 shutdown_connections 2024-03-19T03:03:36.421 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "prophetstor", 2024-03-19T03:03:36.421 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.421 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.421 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.421 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.421 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.421 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.421 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.421 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.421 INFO:teuthology.orchestra.run.smithi033.stdout: "sleep_interval": { 2024-03-19T03:03:36.421 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "sleep_interval", 2024-03-19T03:03:36.421 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.422 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.422 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.422 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "600", 2024-03-19T03:03:36.422 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.422 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.422 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.422 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.422 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.422 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.422 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.422 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.422 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.422 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.422 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.422 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "influx", 2024-03-19T03:03:36.423 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": false, 2024-03-19T03:03:36.423 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "influxdb python module not found", 2024-03-19T03:03:36.423 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.423 INFO:teuthology.orchestra.run.smithi033.stdout: "batch_size": { 2024-03-19T03:03:36.423 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "batch_size", 2024-03-19T03:03:36.423 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.423 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.423 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.423 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5000", 2024-03-19T03:03:36.423 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.423 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.423 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.423 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-19T03:03:36.423 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.423 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.423 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.424 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.424 INFO:teuthology.orchestra.run.smithi033.stdout: "database": { 2024-03-19T03:03:36.424 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "database", 2024-03-19T03:03:36.424 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.424 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.424 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.424 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "ceph", 2024-03-19T03:03:36.424 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.424 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.424 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.424 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-19T03:03:36.424 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.424 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.424 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.424 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.424 INFO:teuthology.orchestra.run.smithi033.stdout: "hostname": { 2024-03-19T03:03:36.425 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "hostname", 2024-03-19T03:03:36.425 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.425 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.425 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.425 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.425 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.425 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.425 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.425 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "InfluxDB server hostname", 2024-03-19T03:03:36.425 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.425 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.425 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.425 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.425 INFO:teuthology.orchestra.run.smithi033.stdout: "interval": { 2024-03-19T03:03:36.425 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "interval", 2024-03-19T03:03:36.425 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:36.425 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.426 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.426 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "30", 2024-03-19T03:03:36.426 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "5", 2024-03-19T03:03:36.426 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.426 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.426 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-19T03:03:36.426 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.426 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.426 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.426 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.426 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.426 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.426 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.426 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.427 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.427 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.427 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.427 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.427 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.427 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.427 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.427 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.427 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.427 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.427 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.427 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.427 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.427 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.427 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.427 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.428 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.428 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.428 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.428 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.428 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.428 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.428 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.428 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.428 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.428 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.428 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.428 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.428 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.428 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.428 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.428 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.428 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.429 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.429 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.429 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.429 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.429 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.429 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.429 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.429 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.429 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.429 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.429 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.429 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.429 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.429 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.429 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.429 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.430 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.430 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.430 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.430 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.430 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.430 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.430 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.430 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.430 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.430 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.430 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.430 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.430 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.430 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.430 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.431 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.431 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.431 INFO:teuthology.orchestra.run.smithi033.stdout: "password": { 2024-03-19T03:03:36.431 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "password", 2024-03-19T03:03:36.431 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.431 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.431 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.431 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.431 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.431 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.431 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.431 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "password of InfluxDB server user", 2024-03-19T03:03:36.431 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.431 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.431 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.431 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.431 INFO:teuthology.orchestra.run.smithi033.stdout: "port": { 2024-03-19T03:03:36.432 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "port", 2024-03-19T03:03:36.432 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.432 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.432 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.432 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "8086", 2024-03-19T03:03:36.432 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.432 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.432 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.432 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "InfluxDB server port", 2024-03-19T03:03:36.432 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.432 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.432 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.432 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.432 INFO:teuthology.orchestra.run.smithi033.stdout: "ssl": { 2024-03-19T03:03:36.432 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ssl", 2024-03-19T03:03:36.432 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.433 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.433 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.433 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "false", 2024-03-19T03:03:36.433 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.433 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.433 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.433 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-19T03:03:36.433 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.433 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.433 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.433 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.433 INFO:teuthology.orchestra.run.smithi033.stdout: "threads": { 2024-03-19T03:03:36.433 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "threads", 2024-03-19T03:03:36.433 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.433 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.433 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.433 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5", 2024-03-19T03:03:36.434 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "1", 2024-03-19T03:03:36.434 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "32", 2024-03-19T03:03:36.434 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.434 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-19T03:03:36.434 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.434 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.434 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.434 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.434 INFO:teuthology.orchestra.run.smithi033.stdout: "username": { 2024-03-19T03:03:36.434 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "username", 2024-03-19T03:03:36.434 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.434 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.434 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.434 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.434 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.434 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.435 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.435 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "username of InfluxDB server user", 2024-03-19T03:03:36.435 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.435 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.435 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.435 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.435 INFO:teuthology.orchestra.run.smithi033.stdout: "verify_ssl": { 2024-03-19T03:03:36.435 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "verify_ssl", 2024-03-19T03:03:36.435 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.435 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.435 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.435 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "true", 2024-03-19T03:03:36.435 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.435 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.435 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.436 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-19T03:03:36.436 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.436 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.436 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.436 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.436 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.436 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.436 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.436 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "insights", 2024-03-19T03:03:36.436 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.436 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.436 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.436 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.436 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.436 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.436 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.437 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.437 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.437 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.437 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.437 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.437 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.437 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.437 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.437 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.437 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.437 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.437 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.437 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.437 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.437 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.438 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.438 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.438 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.438 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.438 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.438 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.438 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.438 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.438 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.438 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.438 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.438 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.438 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.438 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.439 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.439 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.439 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.439 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.439 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.439 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.439 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.439 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.439 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.439 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.439 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.439 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.439 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.439 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.439 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.440 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.440 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.440 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.440 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.440 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.440 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.440 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.440 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.440 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.440 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.440 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.440 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.440 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.440 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.440 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.441 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.441 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.441 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.441 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.441 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.441 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.441 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.441 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.441 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.441 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.441 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "iostat", 2024-03-19T03:03:36.441 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.441 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.441 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.441 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.442 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.442 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.442 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.442 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.442 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.442 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.442 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.442 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.442 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.442 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.442 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.442 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.442 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.442 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.442 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.443 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.443 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.443 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.443 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.443 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.443 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.443 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.443 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.443 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.443 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.443 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.443 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.443 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.443 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.443 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.444 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.444 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.444 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.444 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.444 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.444 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.444 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.444 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.444 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.444 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.444 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.444 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.444 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.444 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.444 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.444 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.445 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.445 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.445 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.445 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.445 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.445 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.445 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.445 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.445 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.445 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.445 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.445 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.445 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.445 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.445 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.446 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.446 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.446 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.446 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.446 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.446 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.446 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.446 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.446 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.446 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.446 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.446 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "k8sevents", 2024-03-19T03:03:36.446 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.446 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.446 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.447 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_event_retention_days": { 2024-03-19T03:03:36.447 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "ceph_event_retention_days", 2024-03-19T03:03:36.447 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.447 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.447 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.447 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "7", 2024-03-19T03:03:36.447 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.447 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.447 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.447 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-19T03:03:36.447 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.447 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.447 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.447 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.447 INFO:teuthology.orchestra.run.smithi033.stdout: "config_check_secs": { 2024-03-19T03:03:36.447 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "config_check_secs", 2024-03-19T03:03:36.448 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.448 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.448 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.448 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-03-19T03:03:36.448 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "10", 2024-03-19T03:03:36.448 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.448 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.448 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-19T03:03:36.448 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.448 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.448 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.448 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.448 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.448 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.448 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.448 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.449 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.449 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.449 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.449 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.449 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.449 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.449 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.449 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.449 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.449 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.449 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.449 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.449 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.449 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.449 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.450 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.450 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.450 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.450 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.450 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.450 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.450 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.450 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.450 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.450 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.450 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.450 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.450 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.450 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.450 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.451 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.451 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.451 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.451 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.451 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.451 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.451 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.451 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.451 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.451 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.451 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.451 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.451 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.451 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.451 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.452 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.452 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.452 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.452 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.452 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.452 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.452 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.452 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.452 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.452 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.452 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.452 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.452 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.452 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.452 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.453 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.453 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.453 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.453 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.453 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.453 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.453 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.453 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.453 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.453 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "localpool", 2024-03-19T03:03:36.453 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.453 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.453 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.453 INFO:teuthology.orchestra.run.smithi033.stdout: "failure_domain": { 2024-03-19T03:03:36.453 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "failure_domain", 2024-03-19T03:03:36.454 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.454 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.454 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.454 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "host", 2024-03-19T03:03:36.454 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.454 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.454 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.454 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "failure domain for any created local pool", 2024-03-19T03:03:36.454 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-19T03:03:36.454 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.454 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.454 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.454 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.454 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.454 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.455 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.455 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.455 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.455 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.455 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.455 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.455 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.455 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.455 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.455 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.455 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.455 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.455 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.455 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.455 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.455 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.456 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.456 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.456 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.456 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.456 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.456 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.456 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.456 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.456 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.456 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.456 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.456 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.456 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.456 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.456 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.456 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.457 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.457 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.457 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.457 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.457 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.457 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.457 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.457 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.457 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.457 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.457 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.457 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.457 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.457 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.458 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.458 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.458 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.458 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.458 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.458 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.458 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.458 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.458 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.458 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.458 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.458 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.458 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.458 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.458 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.458 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.459 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.459 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.459 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.459 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.459 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.459 INFO:teuthology.orchestra.run.smithi033.stdout: "min_size": { 2024-03-19T03:03:36.459 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "min_size", 2024-03-19T03:03:36.459 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.459 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.459 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.459 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.459 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.459 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.459 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.459 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "default min_size for any created local pool", 2024-03-19T03:03:36.459 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-19T03:03:36.459 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.460 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.460 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.460 INFO:teuthology.orchestra.run.smithi033.stdout: "num_rep": { 2024-03-19T03:03:36.460 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "num_rep", 2024-03-19T03:03:36.460 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.460 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.460 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.460 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "3", 2024-03-19T03:03:36.460 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.460 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.460 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.460 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "default replica count for any created local pool", 2024-03-19T03:03:36.460 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.460 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.460 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.460 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.461 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_num": { 2024-03-19T03:03:36.461 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "pg_num", 2024-03-19T03:03:36.461 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.461 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.461 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.461 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "128", 2024-03-19T03:03:36.461 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.461 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.461 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.461 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "default pg_num for any created local pool", 2024-03-19T03:03:36.461 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.461 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.461 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.461 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.461 INFO:teuthology.orchestra.run.smithi033.stdout: "prefix": { 2024-03-19T03:03:36.461 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "prefix", 2024-03-19T03:03:36.462 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.462 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.462 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.462 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.462 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.462 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.462 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.462 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "name prefix for any created local pool", 2024-03-19T03:03:36.462 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.462 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.462 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.462 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.462 INFO:teuthology.orchestra.run.smithi033.stdout: "subtree": { 2024-03-19T03:03:36.462 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "subtree", 2024-03-19T03:03:36.462 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.463 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.463 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.463 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "rack", 2024-03-19T03:03:36.463 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.463 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.463 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.463 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-19T03:03:36.463 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-19T03:03:36.463 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.463 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.463 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.463 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.463 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.463 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.463 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mds_autoscaler", 2024-03-19T03:03:36.463 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.463 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.464 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.464 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.464 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.464 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.464 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.464 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.464 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.464 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.464 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.464 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.464 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.464 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.464 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.464 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.464 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.464 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.465 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.465 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.465 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.465 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.465 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.465 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.465 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.465 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.465 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.465 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.465 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.465 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.465 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.465 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.465 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.465 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.465 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.466 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.466 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.466 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.466 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.466 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.466 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.466 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.466 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.466 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.466 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.466 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.466 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.466 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.466 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.467 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.467 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.467 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.467 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.467 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.467 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.467 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.467 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.467 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.467 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.467 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.467 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.467 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.467 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.467 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.468 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.468 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.468 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.468 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.468 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.468 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.468 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.468 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.468 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.468 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.468 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.468 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.468 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mirroring", 2024-03-19T03:03:36.468 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.468 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.468 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.469 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.469 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.469 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.469 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.469 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.469 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.469 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.469 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.469 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.469 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.469 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.469 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.469 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.469 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.469 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.470 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.470 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.470 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.470 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.470 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.470 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.470 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.470 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.470 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.470 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.470 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.470 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.471 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.471 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.471 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.471 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.471 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.471 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.471 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.471 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.471 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.471 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.471 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.472 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.472 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.472 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.472 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.472 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.472 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.472 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.472 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.472 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.472 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.473 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.473 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.473 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.473 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.473 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.473 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.473 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.473 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.473 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.474 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.474 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.474 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.474 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.474 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.474 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.474 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.474 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.474 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.474 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.474 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.475 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.475 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.475 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.475 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.475 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.475 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "nfs", 2024-03-19T03:03:36.475 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.475 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.475 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.475 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.475 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.476 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.476 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.476 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.476 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.476 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.476 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.476 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.476 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.476 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.476 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.476 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.476 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.476 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.477 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.477 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.477 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.477 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.477 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.477 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.477 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.477 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.477 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.477 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.477 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.477 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.477 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.477 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.477 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.478 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.478 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.478 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.478 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.478 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.478 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.478 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.478 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.478 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.478 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.478 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.478 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.478 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.478 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.478 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.479 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.479 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.479 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.479 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.479 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.479 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.479 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.479 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.479 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.479 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.479 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.479 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.479 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.479 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.479 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.480 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.480 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.480 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.480 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.480 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.480 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.480 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.480 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.480 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.480 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.480 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.480 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.480 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.480 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "orchestrator", 2024-03-19T03:03:36.480 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.481 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.481 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.481 INFO:teuthology.orchestra.run.smithi033.stdout: "fail_fs": { 2024-03-19T03:03:36.481 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "fail_fs", 2024-03-19T03:03:36.481 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.481 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.481 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.481 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.481 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.481 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.481 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.481 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-19T03:03:36.481 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.481 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.481 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.482 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.482 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.482 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.482 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.482 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.482 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.482 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.482 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.482 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.482 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.482 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.482 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.482 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.482 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.482 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.483 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.483 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.483 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.483 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.483 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.483 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.483 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.483 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.483 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.483 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.483 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.483 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.483 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.483 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.483 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.484 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:03:35.422+0000 7f81c07bc640 1 -- 172.21.15.33:0/621911649 wait complete. 2024-03-19T03:03:36.485 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.485 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.485 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.485 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.485 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.485 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.485 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.485 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.485 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.485 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.486 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.486 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.486 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.486 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.486 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.486 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.486 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.486 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.486 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.486 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.486 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.486 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.486 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.486 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.487 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.487 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.487 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.487 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.487 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.487 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.487 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.487 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.487 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.487 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.487 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.487 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.487 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.487 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.487 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.488 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.488 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.488 INFO:teuthology.orchestra.run.smithi033.stdout: "orchestrator": { 2024-03-19T03:03:36.488 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "orchestrator", 2024-03-19T03:03:36.488 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.488 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.488 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.488 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.488 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.488 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.488 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.488 INFO:teuthology.orchestra.run.smithi033.stdout: "cephadm", 2024-03-19T03:03:36.488 INFO:teuthology.orchestra.run.smithi033.stdout: "rook", 2024-03-19T03:03:36.488 INFO:teuthology.orchestra.run.smithi033.stdout: "test_orchestrator" 2024-03-19T03:03:36.488 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.489 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Orchestrator backend", 2024-03-19T03:03:36.489 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.489 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.489 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.489 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.489 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.489 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.489 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.489 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "osd_perf_query", 2024-03-19T03:03:36.489 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.489 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.489 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.489 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.489 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.489 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.490 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.490 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.490 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.490 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.490 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.490 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.490 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.490 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.490 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.490 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.490 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.490 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.490 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.490 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.491 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.491 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.491 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.491 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.491 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.491 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.491 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.491 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.491 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.491 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.491 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.491 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.491 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.491 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.491 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.492 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.492 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.492 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.492 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.492 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.492 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.492 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.492 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.492 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.492 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.492 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.492 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.492 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.492 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.492 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.493 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.493 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.493 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.493 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.493 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.493 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.493 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.493 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.493 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.493 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.493 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.493 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.493 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.493 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.493 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.493 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.494 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.494 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.494 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.494 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.494 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.494 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.494 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.494 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.494 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.494 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.494 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "osd_support", 2024-03-19T03:03:36.494 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.494 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.494 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.495 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.495 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.495 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.495 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.495 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.495 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.495 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.495 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.495 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.495 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.495 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.495 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.495 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.495 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.495 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.496 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.496 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.496 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.496 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.496 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.496 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.496 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.496 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.496 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.496 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.496 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.496 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.496 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.497 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.497 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.497 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.497 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.497 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.497 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.497 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.497 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.497 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.497 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.497 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.497 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.497 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.497 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.497 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.498 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.498 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.498 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.498 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.498 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.498 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.498 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.498 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.498 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.498 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.498 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.498 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.498 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.498 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.498 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.499 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.499 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.499 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.499 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.499 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.499 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.499 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.499 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.499 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.499 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.499 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.499 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.499 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.499 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.499 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.500 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "pg_autoscaler", 2024-03-19T03:03:36.500 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.500 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.500 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.500 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.500 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.500 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.500 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.500 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.500 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.500 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.500 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.500 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.500 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.501 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.501 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.501 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.501 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.501 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.501 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.501 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.501 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.501 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.501 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.501 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.501 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.501 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.501 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.502 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.502 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.502 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.502 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.502 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.502 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.502 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.502 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.502 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.502 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.502 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.502 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.502 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.502 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.503 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.503 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.503 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.503 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.503 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.503 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.503 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.503 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.503 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.503 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.503 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.503 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.503 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.503 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.503 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.504 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.504 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.504 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.504 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.504 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.504 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.504 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.504 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.504 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.504 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.504 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.504 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.504 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.504 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.504 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.505 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.505 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.505 INFO:teuthology.orchestra.run.smithi033.stdout: "sleep_interval": { 2024-03-19T03:03:36.505 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "sleep_interval", 2024-03-19T03:03:36.505 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:36.505 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.505 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.505 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "60", 2024-03-19T03:03:36.505 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.505 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.505 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.505 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.505 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.505 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.505 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.506 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.506 INFO:teuthology.orchestra.run.smithi033.stdout: "threshold": { 2024-03-19T03:03:36.506 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "threshold", 2024-03-19T03:03:36.506 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-03-19T03:03:36.506 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.506 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.506 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "3.0", 2024-03-19T03:03:36.506 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "1.0", 2024-03-19T03:03:36.506 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.506 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.506 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "scaling threshold", 2024-03-19T03:03:36.506 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-19T03:03:36.506 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.507 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.507 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.507 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.507 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.507 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.507 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "progress", 2024-03-19T03:03:36.507 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.507 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.507 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.507 INFO:teuthology.orchestra.run.smithi033.stdout: "allow_pg_recovery_event": { 2024-03-19T03:03:36.507 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "allow_pg_recovery_event", 2024-03-19T03:03:36.507 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.507 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.507 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.507 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.508 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.508 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.508 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.508 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-19T03:03:36.508 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.508 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.508 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.508 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.508 INFO:teuthology.orchestra.run.smithi033.stdout: "enabled": { 2024-03-19T03:03:36.508 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "enabled", 2024-03-19T03:03:36.508 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.508 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.508 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.508 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.508 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.509 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.509 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.509 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.509 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.509 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.509 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.509 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.509 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.509 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.509 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.509 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.509 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.509 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.509 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.509 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.510 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.510 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.510 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.510 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.510 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.510 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.510 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.510 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.510 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.510 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.510 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.510 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.510 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.510 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.510 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.511 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.511 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.511 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.511 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.511 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.511 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.511 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.511 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.511 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.511 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.511 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.511 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.511 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.511 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.512 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.512 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.512 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.512 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.512 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.512 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.512 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.512 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.512 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.512 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.512 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.513 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.513 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.513 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.513 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.513 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.513 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.513 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.513 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.513 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.513 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.513 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.513 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.513 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.514 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.514 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.514 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.514 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.514 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.514 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.514 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.514 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.514 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.514 INFO:teuthology.orchestra.run.smithi033.stdout: "max_completed_events": { 2024-03-19T03:03:36.514 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "max_completed_events", 2024-03-19T03:03:36.514 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.514 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.515 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.515 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "50", 2024-03-19T03:03:36.515 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.515 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.515 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.515 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "number of past completed events to remember", 2024-03-19T03:03:36.515 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.515 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.515 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.515 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.515 INFO:teuthology.orchestra.run.smithi033.stdout: "sleep_interval": { 2024-03-19T03:03:36.515 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "sleep_interval", 2024-03-19T03:03:36.515 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:36.515 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.516 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.516 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "5", 2024-03-19T03:03:36.516 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.516 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.516 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.516 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "how long the module is going to sleep", 2024-03-19T03:03:36.516 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.516 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.516 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.516 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.516 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.516 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.516 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.516 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "prometheus", 2024-03-19T03:03:36.516 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.517 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.517 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.517 INFO:teuthology.orchestra.run.smithi033.stdout: "cache": { 2024-03-19T03:03:36.517 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "cache", 2024-03-19T03:03:36.517 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.517 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.517 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.517 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.517 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.517 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.517 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.517 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.517 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.517 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.518 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.518 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.518 INFO:teuthology.orchestra.run.smithi033.stdout: "exclude_perf_counters": { 2024-03-19T03:03:36.518 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "exclude_perf_counters", 2024-03-19T03:03:36.518 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.518 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.518 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.518 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.518 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.518 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.518 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.518 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-19T03:03:36.519 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-19T03:03:36.519 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.519 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.519 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.519 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.519 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.519 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.519 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.519 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.519 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.519 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.520 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.520 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.520 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.520 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.520 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.520 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.520 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.520 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.520 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.520 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.520 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.520 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.520 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.521 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.521 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.521 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.521 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.521 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.521 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.521 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.521 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.521 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.521 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.521 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.522 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.522 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.522 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.522 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.522 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.522 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.522 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.522 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.522 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.522 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.522 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.523 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.523 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.523 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.523 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.523 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.523 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.523 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.523 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.523 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.523 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.523 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.523 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.523 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.524 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.524 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.524 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.524 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.524 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.524 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.524 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.524 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.524 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.524 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.524 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.524 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.524 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.524 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.525 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.525 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_stats_pools": { 2024-03-19T03:03:36.525 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rbd_stats_pools", 2024-03-19T03:03:36.525 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.525 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.525 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.525 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.525 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.525 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.525 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.525 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.525 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.525 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.525 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.525 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.526 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-19T03:03:36.526 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-19T03:03:36.526 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.526 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.526 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.526 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "300", 2024-03-19T03:03:36.526 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.526 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.526 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.526 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.526 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.526 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.526 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.526 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.527 INFO:teuthology.orchestra.run.smithi033.stdout: "scrape_interval": { 2024-03-19T03:03:36.527 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "scrape_interval", 2024-03-19T03:03:36.527 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-03-19T03:03:36.527 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.527 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.527 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "15.0", 2024-03-19T03:03:36.527 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.527 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.527 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.527 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.527 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.527 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.527 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.527 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.527 INFO:teuthology.orchestra.run.smithi033.stdout: "server_addr": { 2024-03-19T03:03:36.528 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_addr", 2024-03-19T03:03:36.528 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.528 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.528 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.528 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "::", 2024-03-19T03:03:36.528 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.528 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.528 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.528 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-19T03:03:36.528 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.528 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.528 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.528 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.528 INFO:teuthology.orchestra.run.smithi033.stdout: "server_port": { 2024-03-19T03:03:36.529 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_port", 2024-03-19T03:03:36.529 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.529 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.529 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.529 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "9283", 2024-03-19T03:03:36.529 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.529 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.529 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.529 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-19T03:03:36.529 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.529 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.529 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.529 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.529 INFO:teuthology.orchestra.run.smithi033.stdout: "stale_cache_strategy": { 2024-03-19T03:03:36.529 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "stale_cache_strategy", 2024-03-19T03:03:36.530 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.530 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.530 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.530 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "log", 2024-03-19T03:03:36.530 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.530 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.530 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.530 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.530 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.530 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.530 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.530 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.530 INFO:teuthology.orchestra.run.smithi033.stdout: "standby_behaviour": { 2024-03-19T03:03:36.531 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "standby_behaviour", 2024-03-19T03:03:36.531 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.531 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.531 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.531 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "default", 2024-03-19T03:03:36.531 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.531 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.531 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.531 INFO:teuthology.orchestra.run.smithi033.stdout: "default", 2024-03-19T03:03:36.531 INFO:teuthology.orchestra.run.smithi033.stdout: "error" 2024-03-19T03:03:36.531 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.531 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.531 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.531 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.531 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.532 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.532 INFO:teuthology.orchestra.run.smithi033.stdout: "standby_error_status_code": { 2024-03-19T03:03:36.532 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "standby_error_status_code", 2024-03-19T03:03:36.532 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.532 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.532 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.532 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "500", 2024-03-19T03:03:36.532 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "400", 2024-03-19T03:03:36.532 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "599", 2024-03-19T03:03:36.532 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.532 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.532 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.532 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.533 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.533 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.533 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.533 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.533 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.533 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rbd_support", 2024-03-19T03:03:36.533 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.533 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.533 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.533 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.533 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.533 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.533 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.533 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.534 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.534 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.534 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.534 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.534 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.534 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.534 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.534 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.534 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.534 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.534 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.534 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.534 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.534 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.534 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.535 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.535 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.535 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.535 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.535 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.535 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.535 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.535 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.535 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.535 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.535 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.535 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.535 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.535 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.536 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.536 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.536 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.536 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.536 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.536 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.536 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.536 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.536 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.536 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.536 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.536 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.536 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.536 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.536 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.537 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.537 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.537 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.537 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.537 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.537 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.537 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.537 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.537 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.537 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.537 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.537 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.537 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.537 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.537 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.538 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.538 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.538 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.538 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.538 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.538 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.538 INFO:teuthology.orchestra.run.smithi033.stdout: "max_concurrent_snap_create": { 2024-03-19T03:03:36.538 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "max_concurrent_snap_create", 2024-03-19T03:03:36.538 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.539 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.539 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.539 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10", 2024-03-19T03:03:36.539 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.539 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.539 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.539 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.539 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.539 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.539 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.539 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.540 INFO:teuthology.orchestra.run.smithi033.stdout: "mirror_snapshot_schedule": { 2024-03-19T03:03:36.540 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "mirror_snapshot_schedule", 2024-03-19T03:03:36.540 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.540 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.540 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.540 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.540 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.540 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.540 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.540 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.540 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.540 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.541 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.541 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.541 INFO:teuthology.orchestra.run.smithi033.stdout: "trash_purge_schedule": { 2024-03-19T03:03:36.541 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "trash_purge_schedule", 2024-03-19T03:03:36.541 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.541 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.541 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.541 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.541 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.541 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.541 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.541 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.541 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.541 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.541 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.542 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.542 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.542 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.542 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.542 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "restful", 2024-03-19T03:03:36.542 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.542 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.542 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.542 INFO:teuthology.orchestra.run.smithi033.stdout: "enable_auth": { 2024-03-19T03:03:36.542 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "enable_auth", 2024-03-19T03:03:36.542 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.542 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.542 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.542 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.543 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.543 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.543 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.543 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.543 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.543 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.543 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.543 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.543 INFO:teuthology.orchestra.run.smithi033.stdout: "key_file": { 2024-03-19T03:03:36.543 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "key_file", 2024-03-19T03:03:36.543 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.544 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.544 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.544 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.544 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.544 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.544 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.544 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.544 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.544 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.544 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.544 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.544 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.544 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.544 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.545 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.545 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.545 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.545 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.545 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.545 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.545 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.545 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.545 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.545 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.545 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.545 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.545 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.545 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.546 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.546 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.546 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.546 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.546 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.546 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.546 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.546 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.546 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.546 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.546 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.546 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.546 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.546 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.546 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.547 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.547 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.547 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.547 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.547 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.547 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.547 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.547 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.547 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.547 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.547 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.548 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.548 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.548 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.548 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.548 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.548 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.548 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.548 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.548 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.548 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.548 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.548 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.548 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.548 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.548 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.549 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.549 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.549 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.549 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.549 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.549 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.549 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.549 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.549 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.549 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.549 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.549 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.549 INFO:teuthology.orchestra.run.smithi033.stdout: "server_addr": { 2024-03-19T03:03:36.549 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_addr", 2024-03-19T03:03:36.550 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.550 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.550 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.550 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.550 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.550 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.550 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.550 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.550 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.550 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.550 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.550 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.550 INFO:teuthology.orchestra.run.smithi033.stdout: "server_port": { 2024-03-19T03:03:36.550 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "server_port", 2024-03-19T03:03:36.551 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.551 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.551 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.551 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.551 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.551 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.551 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.551 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.551 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.551 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.551 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.551 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.551 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.551 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.552 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.552 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rgw", 2024-03-19T03:03:36.552 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.552 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.552 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.552 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.552 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.552 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.552 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.552 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.552 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.552 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.552 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.552 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.553 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.553 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.553 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.553 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.553 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.553 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.553 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.553 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.553 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.553 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.553 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.553 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.553 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.554 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.554 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.554 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.554 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.554 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.554 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.554 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.554 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.554 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.554 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.554 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.554 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.554 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.554 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.555 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.555 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.555 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.555 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.555 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.555 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.555 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.555 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.555 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.555 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.555 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.555 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.555 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.555 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.556 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.556 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.556 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.556 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.556 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.556 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.556 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.556 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.556 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.556 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.556 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.556 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.556 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.556 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.556 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.557 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.557 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.557 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.557 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.557 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.557 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.557 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.557 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.557 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rook", 2024-03-19T03:03:36.557 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.557 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.557 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.557 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.557 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.558 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.558 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.558 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.558 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.558 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.558 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.558 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.558 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.558 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.558 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.558 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.559 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.559 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.559 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.559 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.559 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.559 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.559 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.559 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.559 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.559 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.559 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.559 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.559 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.560 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.560 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.560 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.560 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.560 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.560 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.560 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.560 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.560 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.560 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.560 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.560 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.560 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.561 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.561 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.561 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.561 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.561 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.561 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.561 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.561 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.561 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.561 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.561 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.561 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.561 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.561 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.562 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.562 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.562 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.562 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.562 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.562 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.562 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.562 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.562 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.562 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.562 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.563 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.563 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.563 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.563 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.563 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.563 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.563 INFO:teuthology.orchestra.run.smithi033.stdout: "storage_class": { 2024-03-19T03:03:36.563 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "storage_class", 2024-03-19T03:03:36.563 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.563 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.563 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.563 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "local", 2024-03-19T03:03:36.563 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.564 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.564 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.564 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-19T03:03:36.564 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.564 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.564 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.564 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.564 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.564 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.564 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.564 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "selftest", 2024-03-19T03:03:36.564 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.564 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.564 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.564 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.565 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.565 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.565 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.565 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.565 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.565 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.565 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.565 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.565 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.565 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.565 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.565 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.565 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.566 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.566 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.566 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.566 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.566 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.566 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.566 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.566 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.566 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.567 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.567 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.567 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.567 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.567 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.567 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.567 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.567 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.567 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.567 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.567 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.567 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.567 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.568 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.568 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.568 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.568 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.568 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.568 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.568 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.568 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.568 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.568 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.568 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.568 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.568 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.568 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.569 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.569 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.569 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.569 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.569 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.569 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.569 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.569 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.569 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.569 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.569 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.569 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.569 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.569 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.569 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.570 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.570 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.570 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.570 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.570 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.570 INFO:teuthology.orchestra.run.smithi033.stdout: "roption1": { 2024-03-19T03:03:36.570 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "roption1", 2024-03-19T03:03:36.570 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.570 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.570 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.570 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.570 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.570 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.571 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.571 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.571 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.571 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.571 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.571 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.571 INFO:teuthology.orchestra.run.smithi033.stdout: "roption2": { 2024-03-19T03:03:36.571 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "roption2", 2024-03-19T03:03:36.571 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.571 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.571 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.571 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "xyz", 2024-03-19T03:03:36.571 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.572 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.572 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.572 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.572 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.572 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.572 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.572 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.572 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption1": { 2024-03-19T03:03:36.572 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption1", 2024-03-19T03:03:36.572 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.572 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.572 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.573 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.573 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.573 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.573 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.573 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.573 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.573 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.573 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.573 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.573 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption2": { 2024-03-19T03:03:36.573 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption2", 2024-03-19T03:03:36.573 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.573 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.573 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.574 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.574 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.574 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.574 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.574 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.574 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.574 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.574 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.574 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.574 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption3": { 2024-03-19T03:03:36.574 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption3", 2024-03-19T03:03:36.574 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "float", 2024-03-19T03:03:36.574 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.574 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.574 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.575 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.575 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.575 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.575 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.575 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.575 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.575 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.575 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.575 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption4": { 2024-03-19T03:03:36.575 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption4", 2024-03-19T03:03:36.575 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.575 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.575 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.575 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.575 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.576 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.576 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.576 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.576 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.576 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.576 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.576 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.576 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption5": { 2024-03-19T03:03:36.576 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption5", 2024-03-19T03:03:36.576 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.576 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.576 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.576 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.577 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.577 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.577 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.577 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.577 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.577 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.577 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.577 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.577 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption6": { 2024-03-19T03:03:36.577 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption6", 2024-03-19T03:03:36.577 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.577 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.577 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.577 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.578 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.578 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.578 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.578 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.578 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.578 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.578 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.578 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.578 INFO:teuthology.orchestra.run.smithi033.stdout: "rwoption7": { 2024-03-19T03:03:36.578 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rwoption7", 2024-03-19T03:03:36.578 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.578 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.578 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.578 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.579 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "1", 2024-03-19T03:03:36.579 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "42", 2024-03-19T03:03:36.579 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.579 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.579 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.579 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.579 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.579 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.579 INFO:teuthology.orchestra.run.smithi033.stdout: "testkey": { 2024-03-19T03:03:36.579 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "testkey", 2024-03-19T03:03:36.579 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.579 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.579 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.580 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.580 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.580 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.580 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.580 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.580 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.580 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.580 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.580 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.580 INFO:teuthology.orchestra.run.smithi033.stdout: "testlkey": { 2024-03-19T03:03:36.580 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "testlkey", 2024-03-19T03:03:36.580 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.580 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.580 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.581 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.581 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.581 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.581 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.581 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.581 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.581 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.581 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.581 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.581 INFO:teuthology.orchestra.run.smithi033.stdout: "testnewline": { 2024-03-19T03:03:36.581 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "testnewline", 2024-03-19T03:03:36.581 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.582 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.582 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.582 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.582 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.582 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.582 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.582 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.582 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.582 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.582 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.582 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.582 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.582 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.583 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.583 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "snap_schedule", 2024-03-19T03:03:36.583 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.583 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.583 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.583 INFO:teuthology.orchestra.run.smithi033.stdout: "allow_m_granularity": { 2024-03-19T03:03:36.583 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "allow_m_granularity", 2024-03-19T03:03:36.583 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.583 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.583 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.583 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.583 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.583 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.584 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.584 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "allow minute scheduled snapshots", 2024-03-19T03:03:36.584 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.584 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.584 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.584 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.584 INFO:teuthology.orchestra.run.smithi033.stdout: "dump_on_update": { 2024-03-19T03:03:36.584 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "dump_on_update", 2024-03-19T03:03:36.584 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.584 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.584 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.584 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.584 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.585 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.585 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.585 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "dump database to debug log on update", 2024-03-19T03:03:36.585 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.585 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.585 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.585 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.585 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.585 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.585 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.585 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.585 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.585 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.585 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.585 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.586 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.586 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.586 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.586 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.586 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.586 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.586 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.586 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.586 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.587 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.587 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.587 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.587 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.587 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.587 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.587 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.587 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.587 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.587 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.587 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.587 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.588 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.588 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.588 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.588 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.588 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.588 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.588 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.588 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.588 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.588 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.588 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.588 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.588 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.589 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.589 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.589 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.589 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.589 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.589 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.589 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.589 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.589 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.589 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.589 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.589 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.589 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.590 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.590 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.590 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.590 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.590 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.590 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.590 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.590 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.590 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.590 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.590 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.590 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.591 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.591 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.591 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.591 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.591 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.591 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.591 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "stats", 2024-03-19T03:03:36.591 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.591 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.591 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.592 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.592 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.592 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.592 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.592 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.592 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.592 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.592 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.592 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.592 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.593 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.593 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.593 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.593 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.593 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.593 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.593 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.593 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.593 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.593 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.593 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.593 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.593 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.594 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.594 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.594 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.594 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.594 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.594 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.594 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.594 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.594 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.594 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.594 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.594 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.594 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.594 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.595 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.595 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.595 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.595 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.595 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.595 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.595 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.595 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.595 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.595 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.595 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.595 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.595 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.595 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.596 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.596 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.596 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.596 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.596 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.596 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.596 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.596 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.596 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.596 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.596 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.596 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.596 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.596 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.597 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.597 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.597 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.597 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.597 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.597 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.597 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.597 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.597 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "status", 2024-03-19T03:03:36.597 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.597 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.597 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.597 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.597 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.597 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.598 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.598 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.598 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.598 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.598 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.598 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.598 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.598 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.598 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.598 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.598 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.598 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.598 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.598 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.599 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.599 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.599 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.599 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.599 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.599 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.599 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.599 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.599 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.599 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.599 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.599 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.599 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.600 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.600 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.600 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.600 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.600 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.600 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.600 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.600 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.601 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.601 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.601 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.601 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.601 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.601 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.601 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.601 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.601 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.601 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.602 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.602 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.602 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.602 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.602 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.602 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.602 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.602 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.602 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.602 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.602 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.602 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.602 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.602 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.603 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.603 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.603 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.603 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.603 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.603 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.603 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.603 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.603 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.603 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.603 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.603 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "telegraf", 2024-03-19T03:03:36.603 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.604 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.604 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.604 INFO:teuthology.orchestra.run.smithi033.stdout: "address": { 2024-03-19T03:03:36.604 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "address", 2024-03-19T03:03:36.604 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.604 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.604 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.604 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-19T03:03:36.604 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.604 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.604 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.604 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.604 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.605 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.605 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.605 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.605 INFO:teuthology.orchestra.run.smithi033.stdout: "interval": { 2024-03-19T03:03:36.605 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "interval", 2024-03-19T03:03:36.605 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:36.605 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.605 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.605 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "15", 2024-03-19T03:03:36.605 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.605 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.605 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.605 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.605 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.605 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.605 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.606 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.606 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.606 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.606 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.606 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.606 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.606 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.606 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.606 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.606 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.607 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.607 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.607 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.607 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.607 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.607 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.607 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.607 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.607 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.607 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.607 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.607 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.607 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.607 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.608 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.608 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.608 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.608 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.608 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.608 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.608 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.608 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.608 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.608 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.608 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.608 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.608 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.608 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.608 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.609 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.609 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.609 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.609 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.609 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.609 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.609 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.609 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.609 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.609 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.609 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.609 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.609 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.609 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.610 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.610 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.610 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.610 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.610 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.610 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.610 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.610 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.610 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.610 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.610 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.610 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.610 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.610 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.611 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.611 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.611 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.611 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.611 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.611 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.611 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.611 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "telemetry", 2024-03-19T03:03:36.611 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.611 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.611 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.611 INFO:teuthology.orchestra.run.smithi033.stdout: "channel_basic": { 2024-03-19T03:03:36.611 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "channel_basic", 2024-03-19T03:03:36.611 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.612 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.612 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.612 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.612 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.612 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.612 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.612 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-19T03:03:36.612 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.612 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.612 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.612 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.612 INFO:teuthology.orchestra.run.smithi033.stdout: "channel_crash": { 2024-03-19T03:03:36.612 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "channel_crash", 2024-03-19T03:03:36.612 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.613 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.613 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.613 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.613 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.613 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.613 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.613 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-19T03:03:36.613 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.613 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.613 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.613 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.613 INFO:teuthology.orchestra.run.smithi033.stdout: "channel_device": { 2024-03-19T03:03:36.613 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "channel_device", 2024-03-19T03:03:36.613 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.613 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.614 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.614 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.614 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.614 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.614 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.614 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-19T03:03:36.614 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.614 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.614 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.614 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.614 INFO:teuthology.orchestra.run.smithi033.stdout: "channel_ident": { 2024-03-19T03:03:36.614 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "channel_ident", 2024-03-19T03:03:36.614 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.615 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.615 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.615 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.615 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.615 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.615 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.615 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-19T03:03:36.615 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.615 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.615 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.615 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.615 INFO:teuthology.orchestra.run.smithi033.stdout: "channel_perf": { 2024-03-19T03:03:36.615 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "channel_perf", 2024-03-19T03:03:36.615 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.615 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.616 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.616 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.616 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.616 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.616 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.616 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-19T03:03:36.616 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.616 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.616 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.616 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.616 INFO:teuthology.orchestra.run.smithi033.stdout: "contact": { 2024-03-19T03:03:36.616 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "contact", 2024-03-19T03:03:36.616 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.616 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.617 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.617 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.617 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.617 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.617 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.617 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.617 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.617 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.617 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.617 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.617 INFO:teuthology.orchestra.run.smithi033.stdout: "description": { 2024-03-19T03:03:36.617 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "description", 2024-03-19T03:03:36.617 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.617 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.617 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.618 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.618 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.618 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.618 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.618 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.618 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.618 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.618 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.618 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.618 INFO:teuthology.orchestra.run.smithi033.stdout: "device_url": { 2024-03-19T03:03:36.618 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "device_url", 2024-03-19T03:03:36.618 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.618 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.618 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.618 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-19T03:03:36.619 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.619 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.619 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.619 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.619 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.619 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.619 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.619 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.619 INFO:teuthology.orchestra.run.smithi033.stdout: "enabled": { 2024-03-19T03:03:36.619 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "enabled", 2024-03-19T03:03:36.619 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.619 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.619 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.619 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.619 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.619 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.620 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.620 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.620 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.620 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.620 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.620 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.620 INFO:teuthology.orchestra.run.smithi033.stdout: "interval": { 2024-03-19T03:03:36.620 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "interval", 2024-03-19T03:03:36.620 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.620 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.620 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.620 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "24", 2024-03-19T03:03:36.620 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "8", 2024-03-19T03:03:36.620 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.620 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.621 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.621 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.621 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.621 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.621 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.621 INFO:teuthology.orchestra.run.smithi033.stdout: "last_opt_revision": { 2024-03-19T03:03:36.621 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "last_opt_revision", 2024-03-19T03:03:36.621 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.621 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.621 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.621 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "1", 2024-03-19T03:03:36.621 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.621 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.621 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.621 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.622 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.622 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.622 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.622 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.622 INFO:teuthology.orchestra.run.smithi033.stdout: "leaderboard": { 2024-03-19T03:03:36.622 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "leaderboard", 2024-03-19T03:03:36.622 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.622 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.622 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.622 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.622 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.622 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.622 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.622 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.622 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.623 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.623 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.623 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.623 INFO:teuthology.orchestra.run.smithi033.stdout: "leaderboard_description": { 2024-03-19T03:03:36.623 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "leaderboard_description", 2024-03-19T03:03:36.623 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.623 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.623 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.623 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.623 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.623 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.623 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.623 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.623 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.623 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.624 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.624 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.624 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.624 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.624 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.624 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.624 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.624 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.624 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.624 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.624 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.624 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.624 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.624 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.625 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.625 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.625 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.625 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.625 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.625 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.625 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.625 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.625 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.625 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.625 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.625 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.625 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.625 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.626 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.626 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.626 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.626 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.626 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.626 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.626 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.626 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.626 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.626 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.626 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.626 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.626 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.626 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.627 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.627 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.627 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.627 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.627 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.627 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.627 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.627 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.627 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.627 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.627 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.627 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.627 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.627 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.627 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.628 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.628 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.628 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.628 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.628 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.628 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.628 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.628 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.628 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.628 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.628 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.628 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.628 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.628 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.628 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.629 INFO:teuthology.orchestra.run.smithi033.stdout: "organization": { 2024-03-19T03:03:36.629 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "organization", 2024-03-19T03:03:36.629 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.629 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.629 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.629 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.629 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.629 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.629 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.629 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.629 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.629 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.629 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.629 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.629 INFO:teuthology.orchestra.run.smithi033.stdout: "proxy": { 2024-03-19T03:03:36.630 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "proxy", 2024-03-19T03:03:36.630 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.630 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.630 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.630 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.630 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.630 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.630 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.630 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.630 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.630 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.630 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.630 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.630 INFO:teuthology.orchestra.run.smithi033.stdout: "url": { 2024-03-19T03:03:36.631 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "url", 2024-03-19T03:03:36.631 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.631 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.631 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.631 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-19T03:03:36.631 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.631 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.631 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.631 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.631 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.631 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.631 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.631 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.632 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.632 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.632 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.632 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "test_orchestrator", 2024-03-19T03:03:36.632 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.632 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.632 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.632 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.632 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.632 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.632 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.632 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.632 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.632 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.632 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.633 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.633 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.633 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.633 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.633 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.633 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.633 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.633 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.633 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.633 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.633 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.633 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.634 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.634 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.634 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.634 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.634 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.634 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.634 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.634 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.634 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.634 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.634 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.634 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.634 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.634 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.634 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.635 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.635 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.635 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.635 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.635 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.635 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.635 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.635 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.635 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.635 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.635 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.635 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.635 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.635 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.635 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.636 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.636 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.636 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.636 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.636 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.636 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.636 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.636 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.636 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.636 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.636 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.636 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.636 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.636 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.637 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.637 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.637 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.637 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.637 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.637 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.637 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.637 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.637 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.637 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "volumes", 2024-03-19T03:03:36.637 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.637 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.637 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.637 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.637 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.637 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.638 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.638 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.638 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.638 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.638 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.638 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.638 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.638 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.638 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.638 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.638 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.638 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.638 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.638 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.639 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.639 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.639 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.639 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.639 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.639 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.639 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.639 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.639 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.639 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.639 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.639 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.639 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.639 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.639 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.639 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.640 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.640 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.640 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.640 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.640 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.640 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.640 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.640 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.640 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.640 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.640 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.640 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.640 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.640 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.640 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.641 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.641 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.641 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.641 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.641 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.641 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.641 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.641 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.641 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.641 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.641 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.641 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.641 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.641 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.641 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.642 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.642 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.642 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.642 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.642 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.642 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.642 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.642 INFO:teuthology.orchestra.run.smithi033.stdout: "max_concurrent_clones": { 2024-03-19T03:03:36.642 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "max_concurrent_clones", 2024-03-19T03:03:36.642 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.642 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.642 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.642 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "4", 2024-03-19T03:03:36.642 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.642 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.643 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.643 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-19T03:03:36.643 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.643 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.643 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.643 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.643 INFO:teuthology.orchestra.run.smithi033.stdout: "periodic_async_work": { 2024-03-19T03:03:36.643 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "periodic_async_work", 2024-03-19T03:03:36.643 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.643 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.643 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.643 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.643 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.643 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.643 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.644 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Periodically check for async work", 2024-03-19T03:03:36.644 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.644 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.644 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.644 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.644 INFO:teuthology.orchestra.run.smithi033.stdout: "snapshot_clone_delay": { 2024-03-19T03:03:36.644 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "snapshot_clone_delay", 2024-03-19T03:03:36.644 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.644 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.644 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.644 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "0", 2024-03-19T03:03:36.644 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.644 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.644 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.644 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-19T03:03:36.644 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.645 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.645 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.645 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.645 INFO:teuthology.orchestra.run.smithi033.stdout: "snapshot_clone_no_wait": { 2024-03-19T03:03:36.645 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "snapshot_clone_no_wait", 2024-03-19T03:03:36.645 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.645 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.645 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.645 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "True", 2024-03-19T03:03:36.645 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.645 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.645 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.645 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-19T03:03:36.645 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.645 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.646 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.646 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.646 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.646 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.646 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.646 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "zabbix", 2024-03-19T03:03:36.646 INFO:teuthology.orchestra.run.smithi033.stdout: "can_run": true, 2024-03-19T03:03:36.646 INFO:teuthology.orchestra.run.smithi033.stdout: "error_string": "", 2024-03-19T03:03:36.646 INFO:teuthology.orchestra.run.smithi033.stdout: "module_options": { 2024-03-19T03:03:36.646 INFO:teuthology.orchestra.run.smithi033.stdout: "discovery_interval": { 2024-03-19T03:03:36.646 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "discovery_interval", 2024-03-19T03:03:36.646 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "uint", 2024-03-19T03:03:36.646 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.646 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.646 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "100", 2024-03-19T03:03:36.647 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.647 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.647 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.647 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.647 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.647 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.647 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.647 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.647 INFO:teuthology.orchestra.run.smithi033.stdout: "identifier": { 2024-03-19T03:03:36.647 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "identifier", 2024-03-19T03:03:36.647 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.647 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.647 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.647 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.647 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.648 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.648 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.648 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.648 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.648 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.648 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.648 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.648 INFO:teuthology.orchestra.run.smithi033.stdout: "interval": { 2024-03-19T03:03:36.648 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "interval", 2024-03-19T03:03:36.648 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "secs", 2024-03-19T03:03:36.648 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.648 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.648 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "60", 2024-03-19T03:03:36.648 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.648 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.648 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.649 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.649 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.649 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.649 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.649 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.649 INFO:teuthology.orchestra.run.smithi033.stdout: "log_level": { 2024-03-19T03:03:36.649 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_level", 2024-03-19T03:03:36.649 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.649 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.649 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.649 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.649 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.649 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.649 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.649 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.649 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.650 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.650 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.650 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.650 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.650 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.650 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.650 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.650 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.650 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.650 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.650 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster": { 2024-03-19T03:03:36.650 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster", 2024-03-19T03:03:36.650 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.650 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.651 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.651 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.651 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.651 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.651 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.651 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.651 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.651 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.651 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.651 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.651 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_cluster_level": { 2024-03-19T03:03:36.651 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_cluster_level", 2024-03-19T03:03:36.651 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.651 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.651 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.651 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "info", 2024-03-19T03:03:36.652 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.652 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.652 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [ 2024-03-19T03:03:36.652 INFO:teuthology.orchestra.run.smithi033.stdout: "", 2024-03-19T03:03:36.652 INFO:teuthology.orchestra.run.smithi033.stdout: "critical", 2024-03-19T03:03:36.652 INFO:teuthology.orchestra.run.smithi033.stdout: "debug", 2024-03-19T03:03:36.652 INFO:teuthology.orchestra.run.smithi033.stdout: "error", 2024-03-19T03:03:36.652 INFO:teuthology.orchestra.run.smithi033.stdout: "info", 2024-03-19T03:03:36.652 INFO:teuthology.orchestra.run.smithi033.stdout: "warning" 2024-03-19T03:03:36.652 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.652 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.652 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.652 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.652 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.652 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.653 INFO:teuthology.orchestra.run.smithi033.stdout: "log_to_file": { 2024-03-19T03:03:36.653 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "log_to_file", 2024-03-19T03:03:36.653 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "bool", 2024-03-19T03:03:36.653 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.653 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 1, 2024-03-19T03:03:36.653 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "False", 2024-03-19T03:03:36.653 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.653 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.653 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.653 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.653 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.653 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.653 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.653 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.653 INFO:teuthology.orchestra.run.smithi033.stdout: "zabbix_host": { 2024-03-19T03:03:36.654 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "zabbix_host", 2024-03-19T03:03:36.654 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.654 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.654 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.654 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "", 2024-03-19T03:03:36.654 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.654 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.654 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.654 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.654 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.654 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.654 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.654 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.654 INFO:teuthology.orchestra.run.smithi033.stdout: "zabbix_port": { 2024-03-19T03:03:36.655 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "zabbix_port", 2024-03-19T03:03:36.655 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "int", 2024-03-19T03:03:36.655 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.655 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.655 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "10051", 2024-03-19T03:03:36.655 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.655 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.655 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.655 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.655 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.655 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.655 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.655 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.655 INFO:teuthology.orchestra.run.smithi033.stdout: "zabbix_sender": { 2024-03-19T03:03:36.655 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "zabbix_sender", 2024-03-19T03:03:36.655 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "str", 2024-03-19T03:03:36.656 INFO:teuthology.orchestra.run.smithi033.stdout: "level": "advanced", 2024-03-19T03:03:36.656 INFO:teuthology.orchestra.run.smithi033.stdout: "flags": 0, 2024-03-19T03:03:36.656 INFO:teuthology.orchestra.run.smithi033.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-19T03:03:36.656 INFO:teuthology.orchestra.run.smithi033.stdout: "min": "", 2024-03-19T03:03:36.656 INFO:teuthology.orchestra.run.smithi033.stdout: "max": "", 2024-03-19T03:03:36.656 INFO:teuthology.orchestra.run.smithi033.stdout: "enum_allowed": [], 2024-03-19T03:03:36.656 INFO:teuthology.orchestra.run.smithi033.stdout: "desc": "", 2024-03-19T03:03:36.656 INFO:teuthology.orchestra.run.smithi033.stdout: "long_desc": "", 2024-03-19T03:03:36.656 INFO:teuthology.orchestra.run.smithi033.stdout: "tags": [], 2024-03-19T03:03:36.656 INFO:teuthology.orchestra.run.smithi033.stdout: "see_also": [] 2024-03-19T03:03:36.656 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.656 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.656 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.656 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.656 INFO:teuthology.orchestra.run.smithi033.stdout: "services": { 2024-03-19T03:03:36.657 INFO:teuthology.orchestra.run.smithi033.stdout: "dashboard": "https://172.21.15.33:8443/" 2024-03-19T03:03:36.657 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.657 INFO:teuthology.orchestra.run.smithi033.stdout: "always_on_modules": { 2024-03-19T03:03:36.657 INFO:teuthology.orchestra.run.smithi033.stdout: "octopus": [ 2024-03-19T03:03:36.657 INFO:teuthology.orchestra.run.smithi033.stdout: "balancer", 2024-03-19T03:03:36.657 INFO:teuthology.orchestra.run.smithi033.stdout: "crash", 2024-03-19T03:03:36.657 INFO:teuthology.orchestra.run.smithi033.stdout: "devicehealth", 2024-03-19T03:03:36.657 INFO:teuthology.orchestra.run.smithi033.stdout: "orchestrator", 2024-03-19T03:03:36.657 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscaler", 2024-03-19T03:03:36.657 INFO:teuthology.orchestra.run.smithi033.stdout: "progress", 2024-03-19T03:03:36.657 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_support", 2024-03-19T03:03:36.657 INFO:teuthology.orchestra.run.smithi033.stdout: "status", 2024-03-19T03:03:36.657 INFO:teuthology.orchestra.run.smithi033.stdout: "telemetry", 2024-03-19T03:03:36.657 INFO:teuthology.orchestra.run.smithi033.stdout: "volumes" 2024-03-19T03:03:36.657 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.657 INFO:teuthology.orchestra.run.smithi033.stdout: "pacific": [ 2024-03-19T03:03:36.658 INFO:teuthology.orchestra.run.smithi033.stdout: "balancer", 2024-03-19T03:03:36.658 INFO:teuthology.orchestra.run.smithi033.stdout: "crash", 2024-03-19T03:03:36.658 INFO:teuthology.orchestra.run.smithi033.stdout: "devicehealth", 2024-03-19T03:03:36.658 INFO:teuthology.orchestra.run.smithi033.stdout: "orchestrator", 2024-03-19T03:03:36.658 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscaler", 2024-03-19T03:03:36.658 INFO:teuthology.orchestra.run.smithi033.stdout: "progress", 2024-03-19T03:03:36.658 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_support", 2024-03-19T03:03:36.658 INFO:teuthology.orchestra.run.smithi033.stdout: "status", 2024-03-19T03:03:36.658 INFO:teuthology.orchestra.run.smithi033.stdout: "telemetry", 2024-03-19T03:03:36.658 INFO:teuthology.orchestra.run.smithi033.stdout: "volumes" 2024-03-19T03:03:36.658 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.658 INFO:teuthology.orchestra.run.smithi033.stdout: "quincy": [ 2024-03-19T03:03:36.658 INFO:teuthology.orchestra.run.smithi033.stdout: "balancer", 2024-03-19T03:03:36.658 INFO:teuthology.orchestra.run.smithi033.stdout: "crash", 2024-03-19T03:03:36.658 INFO:teuthology.orchestra.run.smithi033.stdout: "devicehealth", 2024-03-19T03:03:36.659 INFO:teuthology.orchestra.run.smithi033.stdout: "orchestrator", 2024-03-19T03:03:36.659 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscaler", 2024-03-19T03:03:36.659 INFO:teuthology.orchestra.run.smithi033.stdout: "progress", 2024-03-19T03:03:36.659 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_support", 2024-03-19T03:03:36.659 INFO:teuthology.orchestra.run.smithi033.stdout: "status", 2024-03-19T03:03:36.659 INFO:teuthology.orchestra.run.smithi033.stdout: "telemetry", 2024-03-19T03:03:36.659 INFO:teuthology.orchestra.run.smithi033.stdout: "volumes" 2024-03-19T03:03:36.659 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.659 INFO:teuthology.orchestra.run.smithi033.stdout: "reef": [ 2024-03-19T03:03:36.659 INFO:teuthology.orchestra.run.smithi033.stdout: "balancer", 2024-03-19T03:03:36.659 INFO:teuthology.orchestra.run.smithi033.stdout: "crash", 2024-03-19T03:03:36.659 INFO:teuthology.orchestra.run.smithi033.stdout: "devicehealth", 2024-03-19T03:03:36.659 INFO:teuthology.orchestra.run.smithi033.stdout: "orchestrator", 2024-03-19T03:03:36.659 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscaler", 2024-03-19T03:03:36.659 INFO:teuthology.orchestra.run.smithi033.stdout: "progress", 2024-03-19T03:03:36.659 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_support", 2024-03-19T03:03:36.660 INFO:teuthology.orchestra.run.smithi033.stdout: "status", 2024-03-19T03:03:36.660 INFO:teuthology.orchestra.run.smithi033.stdout: "telemetry", 2024-03-19T03:03:36.660 INFO:teuthology.orchestra.run.smithi033.stdout: "volumes" 2024-03-19T03:03:36.660 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2024-03-19T03:03:36.660 INFO:teuthology.orchestra.run.smithi033.stdout: "squid": [ 2024-03-19T03:03:36.660 INFO:teuthology.orchestra.run.smithi033.stdout: "balancer", 2024-03-19T03:03:36.660 INFO:teuthology.orchestra.run.smithi033.stdout: "crash", 2024-03-19T03:03:36.660 INFO:teuthology.orchestra.run.smithi033.stdout: "devicehealth", 2024-03-19T03:03:36.660 INFO:teuthology.orchestra.run.smithi033.stdout: "orchestrator", 2024-03-19T03:03:36.660 INFO:teuthology.orchestra.run.smithi033.stdout: "pg_autoscaler", 2024-03-19T03:03:36.660 INFO:teuthology.orchestra.run.smithi033.stdout: "progress", 2024-03-19T03:03:36.660 INFO:teuthology.orchestra.run.smithi033.stdout: "rbd_support", 2024-03-19T03:03:36.660 INFO:teuthology.orchestra.run.smithi033.stdout: "status", 2024-03-19T03:03:36.660 INFO:teuthology.orchestra.run.smithi033.stdout: "telemetry", 2024-03-19T03:03:36.660 INFO:teuthology.orchestra.run.smithi033.stdout: "volumes" 2024-03-19T03:03:36.660 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-03-19T03:03:36.661 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.661 INFO:teuthology.orchestra.run.smithi033.stdout: "last_failure_osd_epoch": 98, 2024-03-19T03:03:36.661 INFO:teuthology.orchestra.run.smithi033.stdout: "active_clients": [ 2024-03-19T03:03:36.661 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.661 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "devicehealth", 2024-03-19T03:03:36.661 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-03-19T03:03:36.661 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.661 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-03-19T03:03:36.661 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:0", 2024-03-19T03:03:36.661 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 628261077 2024-03-19T03:03:36.661 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.661 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-03-19T03:03:36.661 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.661 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.661 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "libcephsqlite", 2024-03-19T03:03:36.661 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-03-19T03:03:36.662 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.662 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-03-19T03:03:36.662 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:0", 2024-03-19T03:03:36.662 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3940264209 2024-03-19T03:03:36.662 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.662 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-03-19T03:03:36.662 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.662 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.662 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "rbd_support", 2024-03-19T03:03:36.662 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-03-19T03:03:36.662 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.662 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-03-19T03:03:36.662 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:0", 2024-03-19T03:03:36.662 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3419999957 2024-03-19T03:03:36.662 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.663 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-03-19T03:03:36.663 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.663 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.663 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "snap_schedule", 2024-03-19T03:03:36.663 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-03-19T03:03:36.663 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.663 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-03-19T03:03:36.663 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:0", 2024-03-19T03:03:36.663 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 3671812255 2024-03-19T03:03:36.663 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.663 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-03-19T03:03:36.663 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.663 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.663 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "snap_schedule", 2024-03-19T03:03:36.663 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-03-19T03:03:36.663 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.664 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-03-19T03:03:36.664 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:0", 2024-03-19T03:03:36.664 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 1292438416 2024-03-19T03:03:36.664 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.664 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-03-19T03:03:36.664 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:36.664 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.664 INFO:teuthology.orchestra.run.smithi033.stdout: "name": "volumes", 2024-03-19T03:03:36.664 INFO:teuthology.orchestra.run.smithi033.stdout: "addrvec": [ 2024-03-19T03:03:36.664 INFO:teuthology.orchestra.run.smithi033.stdout: { 2024-03-19T03:03:36.664 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "v2", 2024-03-19T03:03:36.664 INFO:teuthology.orchestra.run.smithi033.stdout: "addr": "172.21.15.33:0", 2024-03-19T03:03:36.664 INFO:teuthology.orchestra.run.smithi033.stdout: "nonce": 2515983875 2024-03-19T03:03:36.664 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.664 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-03-19T03:03:36.664 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:36.665 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-03-19T03:03:36.665 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T03:03:36.665 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:35 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3641712955' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:03:36.665 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:35 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1030361426' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:03:36.665 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:35 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/1636855123' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:03:36.672 DEBUG:tasks.check_counter:Getting stats from a 2024-03-19T03:03:36.673 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-19T03:03:36.673 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:36 smithi033 ceph-mon[31973]: pgmap v315: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-19T03:03:36.673 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:36 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/621911649' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-19T03:03:36.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:36 smithi096 ceph-mon[35946]: pgmap v315: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-19T03:03:36.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:36 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/621911649' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-19T03:03:36.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:36 smithi155 ceph-mon[35804]: pgmap v315: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-19T03:03:36.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:36 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/621911649' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-19T03:03:36.932 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:03:38.442 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:38 smithi033 ceph-mon[31973]: pgmap v316: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-19T03:03:38.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:38 smithi096 ceph-mon[35946]: pgmap v316: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-19T03:03:38.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:38 smithi155 ceph-mon[35804]: pgmap v316: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-19T03:03:38.985 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T03:03:38.985 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T03:03:38.985 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 0, 2024-03-19T03:03:38.985 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 2, 2024-03-19T03:03:38.985 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 0, 2024-03-19T03:03:38.985 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 0, 2024-03-19T03:03:38.986 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 4, 2024-03-19T03:03:38.986 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 0, 2024-03-19T03:03:38.986 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.001046425, 2024-03-19T03:03:38.986 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.000000000, 2024-03-19T03:03:38.986 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.000105745, 2024-03-19T03:03:38.986 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-19T03:03:38.986 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:03:38.986 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-19T03:03:38.986 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-19T03:03:38.986 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-19T03:03:38.986 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:38.986 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:03:38.986 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-19T03:03:38.987 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-19T03:03:38.987 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-19T03:03:38.987 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:38.987 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-19T03:03:38.987 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 0 2024-03-19T03:03:38.987 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:38.987 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T03:03:38.987 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 6, 2024-03-19T03:03:38.987 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 161, 2024-03-19T03:03:38.987 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 3312, 2024-03-19T03:03:38.987 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 184000, 2024-03-19T03:03:38.987 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 5, 2024-03-19T03:03:38.987 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 3, 2024-03-19T03:03:38.987 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.013859046, 2024-03-19T03:03:38.987 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.009588920, 2024-03-19T03:03:38.988 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.002112096, 2024-03-19T03:03:38.988 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-19T03:03:38.988 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:03:38.988 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 159, 2024-03-19T03:03:38.988 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.009460761, 2024-03-19T03:03:38.988 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000059501 2024-03-19T03:03:38.988 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:38.988 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:03:38.988 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-19T03:03:38.988 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-19T03:03:38.988 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-19T03:03:38.988 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:38.988 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-19T03:03:38.988 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 184000 2024-03-19T03:03:38.988 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:38.989 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T03:03:38.989 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 218, 2024-03-19T03:03:38.989 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 208, 2024-03-19T03:03:38.989 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 1436512, 2024-03-19T03:03:38.989 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 127904, 2024-03-19T03:03:38.989 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 2, 2024-03-19T03:03:38.989 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 1, 2024-03-19T03:03:38.989 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.043877604, 2024-03-19T03:03:38.989 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.013599574, 2024-03-19T03:03:38.989 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.035000171, 2024-03-19T03:03:38.989 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.004185451, 2024-03-19T03:03:38.989 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:03:38.989 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 207, 2024-03-19T03:03:38.989 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.008892249, 2024-03-19T03:03:38.990 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000042957 2024-03-19T03:03:38.990 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:38.990 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:03:38.990 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-19T03:03:38.990 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-19T03:03:38.990 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-19T03:03:38.990 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:38.990 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 1436512, 2024-03-19T03:03:38.990 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 127904 2024-03-19T03:03:38.990 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:38.990 INFO:teuthology.orchestra.run.smithi033.stdout: "cct": { 2024-03-19T03:03:38.990 INFO:teuthology.orchestra.run.smithi033.stdout: "total_workers": 0, 2024-03-19T03:03:38.990 INFO:teuthology.orchestra.run.smithi033.stdout: "unhealthy_workers": 0 2024-03-19T03:03:38.990 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:38.990 INFO:teuthology.orchestra.run.smithi033.stdout: "mempool": { 2024-03-19T03:03:38.991 INFO:teuthology.orchestra.run.smithi033.stdout: "bloom_filter_bytes": 0, 2024-03-19T03:03:38.991 INFO:teuthology.orchestra.run.smithi033.stdout: "bloom_filter_items": 0, 2024-03-19T03:03:38.991 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T03:03:38.991 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_items": 0, 2024-03-19T03:03:38.991 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T03:03:38.991 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_data_items": 0, 2024-03-19T03:03:38.991 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T03:03:38.991 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T03:03:38.991 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T03:03:38.991 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T03:03:38.991 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T03:03:38.991 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_other_items": 0, 2024-03-19T03:03:38.991 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T03:03:38.991 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T03:03:38.991 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_bytes": 0, 2024-03-19T03:03:38.992 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_items": 0, 2024-03-19T03:03:38.992 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blob_bytes": 0, 2024-03-19T03:03:38.992 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blob_items": 0, 2024-03-19T03:03:38.992 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T03:03:38.992 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T03:03:38.992 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T03:03:38.992 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T03:03:38.992 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T03:03:38.992 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_items": 0, 2024-03-19T03:03:38.992 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_txc_bytes": 0, 2024-03-19T03:03:38.992 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_txc_items": 0, 2024-03-19T03:03:38.992 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T03:03:38.992 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T03:03:38.992 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_bytes": 0, 2024-03-19T03:03:38.992 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_items": 0, 2024-03-19T03:03:38.992 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_bytes": 0, 2024-03-19T03:03:38.993 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_items": 0, 2024-03-19T03:03:38.993 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T03:03:38.993 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_reader_items": 0, 2024-03-19T03:03:38.993 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T03:03:38.993 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_writer_items": 0, 2024-03-19T03:03:38.993 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_anon_bytes": 148933, 2024-03-19T03:03:38.993 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_anon_items": 30, 2024-03-19T03:03:38.993 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_meta_bytes": 0, 2024-03-19T03:03:38.993 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_meta_items": 0, 2024-03-19T03:03:38.993 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_bytes": 0, 2024-03-19T03:03:38.993 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_items": 0, 2024-03-19T03:03:38.993 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mapbl_bytes": 0, 2024-03-19T03:03:38.993 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mapbl_items": 0, 2024-03-19T03:03:38.993 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pglog_bytes": 0, 2024-03-19T03:03:38.993 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pglog_items": 0, 2024-03-19T03:03:38.993 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_bytes": 0, 2024-03-19T03:03:38.994 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_items": 0, 2024-03-19T03:03:38.994 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T03:03:38.994 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_mapping_items": 0, 2024-03-19T03:03:38.994 INFO:teuthology.orchestra.run.smithi033.stdout: "pgmap_bytes": 0, 2024-03-19T03:03:38.994 INFO:teuthology.orchestra.run.smithi033.stdout: "pgmap_items": 0, 2024-03-19T03:03:38.994 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_co_bytes": 928, 2024-03-19T03:03:38.994 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_co_items": 2, 2024-03-19T03:03:38.994 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_1_bytes": 0, 2024-03-19T03:03:38.994 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_1_items": 0, 2024-03-19T03:03:38.994 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_2_bytes": 0, 2024-03-19T03:03:38.994 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_2_items": 0 2024-03-19T03:03:38.994 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:38.994 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T03:03:38.994 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-03-19T03:03:38.994 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 104857600, 2024-03-19T03:03:38.995 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-03-19T03:03:38.995 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 220, 2024-03-19T03:03:38.995 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 1405791, 2024-03-19T03:03:38.995 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-03-19T03:03:38.995 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 220, 2024-03-19T03:03:38.995 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 0, 2024-03-19T03:03:38.995 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 0, 2024-03-19T03:03:38.995 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 220, 2024-03-19T03:03:38.995 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 1405791, 2024-03-19T03:03:38.995 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-03-19T03:03:38.995 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-19T03:03:38.995 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-19T03:03:38.995 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-19T03:03:38.995 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:38.995 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:38.996 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T03:03:39.639 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-03-19T03:03:39.639 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-03-19T03:03:39.639 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2024-03-19T03:03:39.639 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2024-03-19T03:03:39.639 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-03-19T03:03:39.640 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-03-19T03:03:39.640 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-03-19T03:03:39.640 DEBUG:tasks.check_counter:Getting stats from d 2024-03-19T03:03:39.640 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-19T03:03:39.898 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:03:40.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:40 smithi033 ceph-mon[31973]: pgmap v317: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-19T03:03:40.715 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:40 smithi096 ceph-mon[35946]: pgmap v317: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-19T03:03:40.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:40 smithi155 ceph-mon[35804]: pgmap v317: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-19T03:03:41.369 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:41 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:03:41.369 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:41 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:03:41.369 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:41 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:03:41.551 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:41 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:03:41.552 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:41 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:03:41.552 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:41 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:03:41.620 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:41 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:03:41.620 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:41 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:03:41.620 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:41 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:03:41.622 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T03:03:41.622 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T03:03:41.622 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 225, 2024-03-19T03:03:41.622 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 215, 2024-03-19T03:03:41.623 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 1438480, 2024-03-19T03:03:41.623 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 131904, 2024-03-19T03:03:41.623 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 4, 2024-03-19T03:03:41.623 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 0, 2024-03-19T03:03:41.623 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.042397300, 2024-03-19T03:03:41.623 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.013653976, 2024-03-19T03:03:41.623 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.032221423, 2024-03-19T03:03:41.623 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.004393284, 2024-03-19T03:03:41.623 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:03:41.623 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 214, 2024-03-19T03:03:41.623 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.009922782, 2024-03-19T03:03:41.623 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000046368 2024-03-19T03:03:41.623 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:41.623 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:03:41.624 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-19T03:03:41.624 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-19T03:03:41.624 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-19T03:03:41.624 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:41.624 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 1438480, 2024-03-19T03:03:41.624 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 131904 2024-03-19T03:03:41.624 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:41.624 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T03:03:41.624 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 2, 2024-03-19T03:03:41.624 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 163, 2024-03-19T03:03:41.624 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 320, 2024-03-19T03:03:41.624 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 184256, 2024-03-19T03:03:41.624 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 5, 2024-03-19T03:03:41.624 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 3, 2024-03-19T03:03:41.624 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.013902807, 2024-03-19T03:03:41.624 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.009607547, 2024-03-19T03:03:41.624 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.001755060, 2024-03-19T03:03:41.625 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-19T03:03:41.625 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:03:41.625 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 160, 2024-03-19T03:03:41.625 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.012992923, 2024-03-19T03:03:41.625 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000081205 2024-03-19T03:03:41.625 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:41.625 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:03:41.625 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-19T03:03:41.625 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-19T03:03:41.625 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-19T03:03:41.625 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:41.625 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-19T03:03:41.625 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 184256 2024-03-19T03:03:41.625 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:41.626 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T03:03:41.626 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 4, 2024-03-19T03:03:41.626 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 4, 2024-03-19T03:03:41.626 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 2992, 2024-03-19T03:03:41.626 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 448, 2024-03-19T03:03:41.626 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 2, 2024-03-19T03:03:41.626 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 1, 2024-03-19T03:03:41.626 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.000598256, 2024-03-19T03:03:41.626 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.000033443, 2024-03-19T03:03:41.626 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.000542119, 2024-03-19T03:03:41.626 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-19T03:03:41.626 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:03:41.626 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 3, 2024-03-19T03:03:41.626 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000719719, 2024-03-19T03:03:41.627 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000239906 2024-03-19T03:03:41.627 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:41.627 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:03:41.627 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-19T03:03:41.627 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-19T03:03:41.627 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-19T03:03:41.627 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:41.627 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-19T03:03:41.627 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 448 2024-03-19T03:03:41.627 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:41.627 INFO:teuthology.orchestra.run.smithi033.stdout: "cct": { 2024-03-19T03:03:41.627 INFO:teuthology.orchestra.run.smithi033.stdout: "total_workers": 0, 2024-03-19T03:03:41.627 INFO:teuthology.orchestra.run.smithi033.stdout: "unhealthy_workers": 0 2024-03-19T03:03:41.627 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:41.627 INFO:teuthology.orchestra.run.smithi033.stdout: "mempool": { 2024-03-19T03:03:41.628 INFO:teuthology.orchestra.run.smithi033.stdout: "bloom_filter_bytes": 0, 2024-03-19T03:03:41.628 INFO:teuthology.orchestra.run.smithi033.stdout: "bloom_filter_items": 0, 2024-03-19T03:03:41.628 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T03:03:41.628 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_items": 0, 2024-03-19T03:03:41.628 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T03:03:41.628 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_data_items": 0, 2024-03-19T03:03:41.628 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T03:03:41.628 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T03:03:41.628 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T03:03:41.628 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T03:03:41.628 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T03:03:41.628 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_other_items": 0, 2024-03-19T03:03:41.628 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T03:03:41.628 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T03:03:41.628 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_bytes": 0, 2024-03-19T03:03:41.629 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_items": 0, 2024-03-19T03:03:41.629 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blob_bytes": 0, 2024-03-19T03:03:41.629 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blob_items": 0, 2024-03-19T03:03:41.629 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T03:03:41.629 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T03:03:41.629 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T03:03:41.629 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T03:03:41.629 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T03:03:41.629 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_items": 0, 2024-03-19T03:03:41.629 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_txc_bytes": 0, 2024-03-19T03:03:41.629 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_txc_items": 0, 2024-03-19T03:03:41.629 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T03:03:41.629 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T03:03:41.629 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_bytes": 0, 2024-03-19T03:03:41.629 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_items": 0, 2024-03-19T03:03:41.630 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_bytes": 0, 2024-03-19T03:03:41.630 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_items": 0, 2024-03-19T03:03:41.630 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T03:03:41.630 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_reader_items": 0, 2024-03-19T03:03:41.630 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T03:03:41.630 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_writer_items": 0, 2024-03-19T03:03:41.630 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_anon_bytes": 148731, 2024-03-19T03:03:41.630 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_anon_items": 31, 2024-03-19T03:03:41.630 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_meta_bytes": 0, 2024-03-19T03:03:41.630 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_meta_items": 0, 2024-03-19T03:03:41.630 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_bytes": 0, 2024-03-19T03:03:41.630 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_items": 0, 2024-03-19T03:03:41.630 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mapbl_bytes": 0, 2024-03-19T03:03:41.630 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mapbl_items": 0, 2024-03-19T03:03:41.631 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pglog_bytes": 0, 2024-03-19T03:03:41.631 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pglog_items": 0, 2024-03-19T03:03:41.631 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_bytes": 0, 2024-03-19T03:03:41.631 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_items": 0, 2024-03-19T03:03:41.631 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T03:03:41.631 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_mapping_items": 0, 2024-03-19T03:03:41.631 INFO:teuthology.orchestra.run.smithi033.stdout: "pgmap_bytes": 0, 2024-03-19T03:03:41.631 INFO:teuthology.orchestra.run.smithi033.stdout: "pgmap_items": 0, 2024-03-19T03:03:41.631 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_co_bytes": 928, 2024-03-19T03:03:41.631 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_co_items": 2, 2024-03-19T03:03:41.631 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_1_bytes": 0, 2024-03-19T03:03:41.631 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_1_items": 0, 2024-03-19T03:03:41.631 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_2_bytes": 0, 2024-03-19T03:03:41.631 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_2_items": 0 2024-03-19T03:03:41.631 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:41.632 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T03:03:41.632 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-03-19T03:03:41.632 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 104857600, 2024-03-19T03:03:41.632 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-03-19T03:03:41.632 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 227, 2024-03-19T03:03:41.632 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 1406765, 2024-03-19T03:03:41.632 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-03-19T03:03:41.632 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 227, 2024-03-19T03:03:41.632 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 0, 2024-03-19T03:03:41.632 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 0, 2024-03-19T03:03:41.632 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 227, 2024-03-19T03:03:41.632 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 1406765, 2024-03-19T03:03:41.632 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-03-19T03:03:41.632 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-19T03:03:41.632 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-19T03:03:41.633 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-19T03:03:41.633 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:41.633 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:41.633 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T03:03:42.322 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-19T03:03:42.322 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-19T03:03:42.322 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-03-19T03:03:42.322 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-03-19T03:03:42.322 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-03-19T03:03:42.322 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-03-19T03:03:42.322 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-03-19T03:03:42.323 DEBUG:tasks.check_counter:Getting stats from g 2024-03-19T03:03:42.323 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-19T03:03:42.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:42 smithi033 ceph-mon[31973]: pgmap v318: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-19T03:03:42.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:42 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:03:42.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:42 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:03:42.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:42 smithi096 ceph-mon[35946]: pgmap v318: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-19T03:03:42.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:42 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:03:42.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:42 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:03:42.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:42 smithi155 ceph-mon[35804]: pgmap v318: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-19T03:03:42.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:42 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:03:42.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:42 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:03:43.114 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:03:43.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:43 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:03:43.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:43 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:03:43.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:43 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:43.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:43 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:43.529 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:43 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:43.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:43 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:43.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:43 smithi033 ceph-mon[31973]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-03-19T03:03:43.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:43 smithi033 ceph-mon[31973]: Unable to set osd_memory_target on smithi155 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-03-19T03:03:43.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:43 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:43.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:43 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:43.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:43 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:43.530 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:43 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:43.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:43 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:03:43.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:43 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:03:43.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:43 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:43.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:43 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:43.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:43 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:43.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:43 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:43.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:43 smithi096 ceph-mon[35946]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-03-19T03:03:43.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:43 smithi096 ceph-mon[35946]: Unable to set osd_memory_target on smithi155 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-03-19T03:03:43.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:43 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:43.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:43 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:43.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:43 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:43.717 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:43 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:43.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:43 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:03:43.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:43 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:03:43.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:43 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:43.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:43 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:43.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:43 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:43.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:43 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:43.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:43 smithi155 ceph-mon[35804]: Adjusting osd_memory_target on smithi155 to 176.2M 2024-03-19T03:03:43.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:43 smithi155 ceph-mon[35804]: Unable to set osd_memory_target on smithi155 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-03-19T03:03:43.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:43 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:43.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:43 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:43.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:43 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:43.768 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:43 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:44.654 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:44 smithi033 ceph-mon[31973]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-03-19T03:03:44.655 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:44 smithi033 ceph-mon[31973]: Unable to set osd_memory_target on smithi096 to 184783872: error parsing value: Value '184783872' is below minimum 939524096 2024-03-19T03:03:44.655 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:44 smithi033 ceph-mon[31973]: pgmap v319: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-19T03:03:44.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:44 smithi096 ceph-mon[35946]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-03-19T03:03:44.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:44 smithi096 ceph-mon[35946]: Unable to set osd_memory_target on smithi096 to 184783872: error parsing value: Value '184783872' is below minimum 939524096 2024-03-19T03:03:44.716 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:44 smithi096 ceph-mon[35946]: pgmap v319: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-19T03:03:44.766 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:44 smithi155 ceph-mon[35804]: Adjusting osd_memory_target on smithi096 to 176.2M 2024-03-19T03:03:44.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:44 smithi155 ceph-mon[35804]: Unable to set osd_memory_target on smithi096 to 184783872: error parsing value: Value '184783872' is below minimum 939524096 2024-03-19T03:03:44.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:44 smithi155 ceph-mon[35804]: pgmap v319: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-19T03:03:44.914 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T03:03:44.914 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T03:03:44.914 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 0, 2024-03-19T03:03:44.915 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 2, 2024-03-19T03:03:44.915 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 0, 2024-03-19T03:03:44.915 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 0, 2024-03-19T03:03:44.915 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 4, 2024-03-19T03:03:44.915 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 0, 2024-03-19T03:03:44.915 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.001292088, 2024-03-19T03:03:44.915 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.000000000, 2024-03-19T03:03:44.915 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.000240622, 2024-03-19T03:03:44.915 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-19T03:03:44.915 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:03:44.915 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-19T03:03:44.915 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-19T03:03:44.915 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-19T03:03:44.915 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:44.915 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:03:44.915 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-19T03:03:44.916 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-19T03:03:44.916 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-19T03:03:44.916 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:44.916 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-19T03:03:44.916 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 0 2024-03-19T03:03:44.916 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:44.916 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T03:03:44.916 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 6, 2024-03-19T03:03:44.916 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 163, 2024-03-19T03:03:44.916 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 3312, 2024-03-19T03:03:44.916 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 184352, 2024-03-19T03:03:44.916 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 5, 2024-03-19T03:03:44.916 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 3, 2024-03-19T03:03:44.916 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.014567807, 2024-03-19T03:03:44.916 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.010164090, 2024-03-19T03:03:44.917 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.002125019, 2024-03-19T03:03:44.917 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-19T03:03:44.917 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:03:44.917 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 161, 2024-03-19T03:03:44.917 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.011815366, 2024-03-19T03:03:44.917 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000073387 2024-03-19T03:03:44.917 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:44.917 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:03:44.917 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-19T03:03:44.917 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-19T03:03:44.917 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-19T03:03:44.917 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:44.917 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-19T03:03:44.917 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 184352 2024-03-19T03:03:44.917 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:44.918 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T03:03:44.918 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 220, 2024-03-19T03:03:44.918 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 210, 2024-03-19T03:03:44.918 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 1437056, 2024-03-19T03:03:44.918 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 129184, 2024-03-19T03:03:44.918 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 2, 2024-03-19T03:03:44.918 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 1, 2024-03-19T03:03:44.918 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.043036001, 2024-03-19T03:03:44.918 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.013795516, 2024-03-19T03:03:44.918 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.033435100, 2024-03-19T03:03:44.918 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.004614734, 2024-03-19T03:03:44.918 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:03:44.918 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 209, 2024-03-19T03:03:44.918 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.010253083, 2024-03-19T03:03:44.918 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000049057 2024-03-19T03:03:44.919 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:44.919 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:03:44.919 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-19T03:03:44.919 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-19T03:03:44.919 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-19T03:03:44.919 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:44.919 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 1437056, 2024-03-19T03:03:44.919 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 129184 2024-03-19T03:03:44.919 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:44.919 INFO:teuthology.orchestra.run.smithi033.stdout: "cct": { 2024-03-19T03:03:44.919 INFO:teuthology.orchestra.run.smithi033.stdout: "total_workers": 0, 2024-03-19T03:03:44.919 INFO:teuthology.orchestra.run.smithi033.stdout: "unhealthy_workers": 0 2024-03-19T03:03:44.919 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:44.919 INFO:teuthology.orchestra.run.smithi033.stdout: "mempool": { 2024-03-19T03:03:44.919 INFO:teuthology.orchestra.run.smithi033.stdout: "bloom_filter_bytes": 0, 2024-03-19T03:03:44.919 INFO:teuthology.orchestra.run.smithi033.stdout: "bloom_filter_items": 0, 2024-03-19T03:03:44.919 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T03:03:44.920 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_items": 0, 2024-03-19T03:03:44.920 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T03:03:44.920 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_data_items": 0, 2024-03-19T03:03:44.920 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T03:03:44.920 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T03:03:44.920 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T03:03:44.920 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T03:03:44.920 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T03:03:44.920 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_other_items": 0, 2024-03-19T03:03:44.920 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T03:03:44.920 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T03:03:44.920 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_bytes": 0, 2024-03-19T03:03:44.920 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_items": 0, 2024-03-19T03:03:44.920 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blob_bytes": 0, 2024-03-19T03:03:44.920 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blob_items": 0, 2024-03-19T03:03:44.921 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T03:03:44.921 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T03:03:44.921 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T03:03:44.921 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T03:03:44.921 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T03:03:44.921 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_items": 0, 2024-03-19T03:03:44.921 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_txc_bytes": 0, 2024-03-19T03:03:44.921 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_txc_items": 0, 2024-03-19T03:03:44.921 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T03:03:44.921 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T03:03:44.921 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_bytes": 0, 2024-03-19T03:03:44.921 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_items": 0, 2024-03-19T03:03:44.921 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_bytes": 0, 2024-03-19T03:03:44.921 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_items": 0, 2024-03-19T03:03:44.921 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T03:03:44.922 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_reader_items": 0, 2024-03-19T03:03:44.922 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T03:03:44.922 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_writer_items": 0, 2024-03-19T03:03:44.922 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_anon_bytes": 148977, 2024-03-19T03:03:44.922 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_anon_items": 30, 2024-03-19T03:03:44.922 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_meta_bytes": 0, 2024-03-19T03:03:44.922 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_meta_items": 0, 2024-03-19T03:03:44.922 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_bytes": 0, 2024-03-19T03:03:44.922 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_items": 0, 2024-03-19T03:03:44.922 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mapbl_bytes": 0, 2024-03-19T03:03:44.922 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mapbl_items": 0, 2024-03-19T03:03:44.922 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pglog_bytes": 0, 2024-03-19T03:03:44.922 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pglog_items": 0, 2024-03-19T03:03:44.922 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_bytes": 0, 2024-03-19T03:03:44.923 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_items": 0, 2024-03-19T03:03:44.923 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T03:03:44.923 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_mapping_items": 0, 2024-03-19T03:03:44.923 INFO:teuthology.orchestra.run.smithi033.stdout: "pgmap_bytes": 0, 2024-03-19T03:03:44.923 INFO:teuthology.orchestra.run.smithi033.stdout: "pgmap_items": 0, 2024-03-19T03:03:44.923 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_co_bytes": 928, 2024-03-19T03:03:44.923 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_co_items": 2, 2024-03-19T03:03:44.923 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_1_bytes": 0, 2024-03-19T03:03:44.923 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_1_items": 0, 2024-03-19T03:03:44.923 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_2_bytes": 0, 2024-03-19T03:03:44.923 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_2_items": 0 2024-03-19T03:03:44.924 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:44.924 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T03:03:44.924 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-03-19T03:03:44.924 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 104857600, 2024-03-19T03:03:44.924 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-03-19T03:03:44.924 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 222, 2024-03-19T03:03:44.924 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 1406051, 2024-03-19T03:03:44.924 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-03-19T03:03:44.924 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 222, 2024-03-19T03:03:44.924 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 0, 2024-03-19T03:03:44.924 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 0, 2024-03-19T03:03:44.924 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 222, 2024-03-19T03:03:44.924 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 1406051, 2024-03-19T03:03:44.924 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-03-19T03:03:44.925 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-19T03:03:44.925 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-19T03:03:44.925 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-19T03:03:44.925 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:44.925 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:44.925 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T03:03:45.552 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-03-19T03:03:45.552 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-03-19T03:03:45.552 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2024-03-19T03:03:45.552 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2024-03-19T03:03:45.552 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-03-19T03:03:45.552 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-03-19T03:03:45.552 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-03-19T03:03:45.552 DEBUG:tasks.check_counter:Getting stats from j 2024-03-19T03:03:45.553 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-19T03:03:45.819 INFO:teuthology.orchestra.run.smithi033.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:03:47.073 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:46 smithi033 ceph-mon[31973]: pgmap v320: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:03:47.073 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:46 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:03:47.073 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:46 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:03:47.073 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:46 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:47.073 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:46 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:47.073 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:46 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:47.073 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:46 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:47.073 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:46 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:03:47.073 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:46 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:03:47.073 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:46 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:03:47.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:46 smithi096 ceph-mon[35946]: pgmap v320: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:03:47.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:46 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:03:47.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:46 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:03:47.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:46 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:47.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:46 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:47.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:46 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:47.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:46 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:47.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:46 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:03:47.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:46 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:03:47.217 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:46 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:03:47.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:46 smithi155 ceph-mon[35804]: pgmap v320: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:03:47.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:46 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:03:47.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:46 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:03:47.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:46 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:47.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:46 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:47.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:46 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:47.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:46 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:03:47.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:46 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:03:47.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:46 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:03:47.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:46 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' 2024-03-19T03:03:47.497 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-03-19T03:03:47.497 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T03:03:47.497 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 0, 2024-03-19T03:03:47.497 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 2, 2024-03-19T03:03:47.497 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 0, 2024-03-19T03:03:47.497 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 0, 2024-03-19T03:03:47.497 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 4, 2024-03-19T03:03:47.497 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 0, 2024-03-19T03:03:47.498 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.001142149, 2024-03-19T03:03:47.498 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.000000000, 2024-03-19T03:03:47.498 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.000269877, 2024-03-19T03:03:47.498 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-19T03:03:47.498 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:03:47.498 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-19T03:03:47.498 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-19T03:03:47.498 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-19T03:03:47.498 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:47.498 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:03:47.498 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-19T03:03:47.499 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-19T03:03:47.499 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-19T03:03:47.499 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:47.499 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-19T03:03:47.499 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 0 2024-03-19T03:03:47.499 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:47.499 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T03:03:47.499 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 231, 2024-03-19T03:03:47.499 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 219, 2024-03-19T03:03:47.499 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 1442016, 2024-03-19T03:03:47.499 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 133632, 2024-03-19T03:03:47.499 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 2, 2024-03-19T03:03:47.499 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 1, 2024-03-19T03:03:47.499 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.043889925, 2024-03-19T03:03:47.499 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.014921093, 2024-03-19T03:03:47.500 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.032230071, 2024-03-19T03:03:47.500 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.004458448, 2024-03-19T03:03:47.500 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:03:47.500 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 219, 2024-03-19T03:03:47.500 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.011239888, 2024-03-19T03:03:47.500 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000051323 2024-03-19T03:03:47.500 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:47.500 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:03:47.500 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-19T03:03:47.500 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-19T03:03:47.500 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-19T03:03:47.500 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:47.500 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 1442016, 2024-03-19T03:03:47.500 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 133632 2024-03-19T03:03:47.500 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:47.501 INFO:teuthology.orchestra.run.smithi033.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T03:03:47.501 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_messages": 2, 2024-03-19T03:03:47.501 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages": 165, 2024-03-19T03:03:47.501 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_bytes": 320, 2024-03-19T03:03:47.501 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_bytes": 184608, 2024-03-19T03:03:47.501 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_created_connections": 5, 2024-03-19T03:03:47.501 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_active_connections": 3, 2024-03-19T03:03:47.501 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_total_time": 0.014198120, 2024-03-19T03:03:47.501 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_send_time": 0.009936487, 2024-03-19T03:03:47.501 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_recv_time": 0.001928104, 2024-03-19T03:03:47.501 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-19T03:03:47.501 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:03:47.501 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 162, 2024-03-19T03:03:47.501 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.009844571, 2024-03-19T03:03:47.501 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000060768 2024-03-19T03:03:47.502 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:47.502 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:03:47.502 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-19T03:03:47.502 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-19T03:03:47.502 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-19T03:03:47.502 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:47.502 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-19T03:03:47.502 INFO:teuthology.orchestra.run.smithi033.stdout: "msgr_send_encrypted_bytes": 184608 2024-03-19T03:03:47.502 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:47.502 INFO:teuthology.orchestra.run.smithi033.stdout: "cct": { 2024-03-19T03:03:47.502 INFO:teuthology.orchestra.run.smithi033.stdout: "total_workers": 0, 2024-03-19T03:03:47.502 INFO:teuthology.orchestra.run.smithi033.stdout: "unhealthy_workers": 0 2024-03-19T03:03:47.502 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:47.503 INFO:teuthology.orchestra.run.smithi033.stdout: "mempool": { 2024-03-19T03:03:47.503 INFO:teuthology.orchestra.run.smithi033.stdout: "bloom_filter_bytes": 0, 2024-03-19T03:03:47.503 INFO:teuthology.orchestra.run.smithi033.stdout: "bloom_filter_items": 0, 2024-03-19T03:03:47.503 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T03:03:47.503 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_alloc_items": 0, 2024-03-19T03:03:47.503 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T03:03:47.503 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_data_items": 0, 2024-03-19T03:03:47.503 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T03:03:47.503 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T03:03:47.503 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T03:03:47.503 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T03:03:47.503 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T03:03:47.503 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_other_items": 0, 2024-03-19T03:03:47.503 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T03:03:47.504 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T03:03:47.504 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_bytes": 0, 2024-03-19T03:03:47.504 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_extent_items": 0, 2024-03-19T03:03:47.504 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blob_bytes": 0, 2024-03-19T03:03:47.504 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_blob_items": 0, 2024-03-19T03:03:47.504 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T03:03:47.504 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T03:03:47.504 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T03:03:47.504 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T03:03:47.504 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T03:03:47.504 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_fsck_items": 0, 2024-03-19T03:03:47.504 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_txc_bytes": 0, 2024-03-19T03:03:47.504 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_txc_items": 0, 2024-03-19T03:03:47.504 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T03:03:47.504 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T03:03:47.505 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_bytes": 0, 2024-03-19T03:03:47.505 INFO:teuthology.orchestra.run.smithi033.stdout: "bluestore_writing_items": 0, 2024-03-19T03:03:47.505 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_bytes": 0, 2024-03-19T03:03:47.505 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_items": 0, 2024-03-19T03:03:47.505 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T03:03:47.505 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_reader_items": 0, 2024-03-19T03:03:47.505 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T03:03:47.505 INFO:teuthology.orchestra.run.smithi033.stdout: "bluefs_file_writer_items": 0, 2024-03-19T03:03:47.505 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_anon_bytes": 148754, 2024-03-19T03:03:47.505 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_anon_items": 31, 2024-03-19T03:03:47.505 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_meta_bytes": 0, 2024-03-19T03:03:47.505 INFO:teuthology.orchestra.run.smithi033.stdout: "buffer_meta_items": 0, 2024-03-19T03:03:47.505 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_bytes": 0, 2024-03-19T03:03:47.505 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_items": 0, 2024-03-19T03:03:47.505 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mapbl_bytes": 0, 2024-03-19T03:03:47.506 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_mapbl_items": 0, 2024-03-19T03:03:47.506 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pglog_bytes": 0, 2024-03-19T03:03:47.506 INFO:teuthology.orchestra.run.smithi033.stdout: "osd_pglog_items": 0, 2024-03-19T03:03:47.506 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_bytes": 0, 2024-03-19T03:03:47.506 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_items": 0, 2024-03-19T03:03:47.506 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T03:03:47.506 INFO:teuthology.orchestra.run.smithi033.stdout: "osdmap_mapping_items": 0, 2024-03-19T03:03:47.506 INFO:teuthology.orchestra.run.smithi033.stdout: "pgmap_bytes": 0, 2024-03-19T03:03:47.506 INFO:teuthology.orchestra.run.smithi033.stdout: "pgmap_items": 0, 2024-03-19T03:03:47.506 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_co_bytes": 928, 2024-03-19T03:03:47.506 INFO:teuthology.orchestra.run.smithi033.stdout: "mds_co_items": 2, 2024-03-19T03:03:47.506 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_1_bytes": 0, 2024-03-19T03:03:47.506 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_1_items": 0, 2024-03-19T03:03:47.507 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_2_bytes": 0, 2024-03-19T03:03:47.507 INFO:teuthology.orchestra.run.smithi033.stdout: "unittest_2_items": 0 2024-03-19T03:03:47.507 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2024-03-19T03:03:47.507 INFO:teuthology.orchestra.run.smithi033.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T03:03:47.507 INFO:teuthology.orchestra.run.smithi033.stdout: "val": 0, 2024-03-19T03:03:47.507 INFO:teuthology.orchestra.run.smithi033.stdout: "max": 104857600, 2024-03-19T03:03:47.507 INFO:teuthology.orchestra.run.smithi033.stdout: "get_started": 0, 2024-03-19T03:03:47.507 INFO:teuthology.orchestra.run.smithi033.stdout: "get": 229, 2024-03-19T03:03:47.507 INFO:teuthology.orchestra.run.smithi033.stdout: "get_sum": 1407025, 2024-03-19T03:03:47.507 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_fail": 0, 2024-03-19T03:03:47.507 INFO:teuthology.orchestra.run.smithi033.stdout: "get_or_fail_success": 229, 2024-03-19T03:03:47.507 INFO:teuthology.orchestra.run.smithi033.stdout: "take": 0, 2024-03-19T03:03:47.507 INFO:teuthology.orchestra.run.smithi033.stdout: "take_sum": 0, 2024-03-19T03:03:47.507 INFO:teuthology.orchestra.run.smithi033.stdout: "put": 229, 2024-03-19T03:03:47.508 INFO:teuthology.orchestra.run.smithi033.stdout: "put_sum": 1407025, 2024-03-19T03:03:47.508 INFO:teuthology.orchestra.run.smithi033.stdout: "wait": { 2024-03-19T03:03:47.508 INFO:teuthology.orchestra.run.smithi033.stdout: "avgcount": 0, 2024-03-19T03:03:47.508 INFO:teuthology.orchestra.run.smithi033.stdout: "sum": 0.000000000, 2024-03-19T03:03:47.508 INFO:teuthology.orchestra.run.smithi033.stdout: "avgtime": 0.000000000 2024-03-19T03:03:47.508 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:47.508 INFO:teuthology.orchestra.run.smithi033.stdout: } 2024-03-19T03:03:47.508 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-03-19T03:03:48.150 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:47 smithi033 ceph-mon[31973]: Adjusting osd_memory_target on smithi033 to 177.2M 2024-03-19T03:03:48.150 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:47 smithi033 ceph-mon[31973]: Unable to set osd_memory_target on smithi033 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-03-19T03:03:48.180 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-19T03:03:48.180 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-19T03:03:48.180 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-03-19T03:03:48.180 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-03-19T03:03:48.180 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-03-19T03:03:48.180 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-03-19T03:03:48.180 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-03-19T03:03:48.180 DEBUG:tasks.check_counter:Getting stats from b 2024-03-19T03:03:48.181 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-19T03:03:48.205 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:47 smithi096 ceph-mon[35946]: Adjusting osd_memory_target on smithi033 to 177.2M 2024-03-19T03:03:48.206 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:47 smithi096 ceph-mon[35946]: Unable to set osd_memory_target on smithi033 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-03-19T03:03:48.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:47 smithi155 ceph-mon[35804]: Adjusting osd_memory_target on smithi033 to 177.2M 2024-03-19T03:03:48.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:47 smithi155 ceph-mon[35804]: Unable to set osd_memory_target on smithi033 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-03-19T03:03:48.432 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T03:03:49.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:48 smithi096 ceph-mon[35946]: pgmap v321: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:03:49.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:48 smithi155 ceph-mon[35804]: pgmap v321: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:03:49.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:48 smithi033 ceph-mon[31973]: pgmap v321: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:03:50.358 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-03-19T03:03:50.358 INFO:teuthology.orchestra.run.smithi096.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T03:03:50.358 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_messages": 2058, 2024-03-19T03:03:50.358 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages": 1872, 2024-03-19T03:03:50.358 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_bytes": 701381, 2024-03-19T03:03:50.358 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_bytes": 2870602, 2024-03-19T03:03:50.358 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_created_connections": 155, 2024-03-19T03:03:50.358 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_active_connections": 4, 2024-03-19T03:03:50.358 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_total_time": 0.175932668, 2024-03-19T03:03:50.358 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_send_time": 0.042072542, 2024-03-19T03:03:50.359 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_recv_time": 0.073018034, 2024-03-19T03:03:50.359 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_fast_dispatch_time": 0.045315168, 2024-03-19T03:03:50.359 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:03:50.359 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 1870, 2024-03-19T03:03:50.359 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.211930692, 2024-03-19T03:03:50.359 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000113331 2024-03-19T03:03:50.359 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.359 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:03:50.359 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 1261, 2024-03-19T03:03:50.359 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000232702, 2024-03-19T03:03:50.359 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000184 2024-03-19T03:03:50.359 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.359 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_encrypted_bytes": 43152, 2024-03-19T03:03:50.359 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_encrypted_bytes": 2896 2024-03-19T03:03:50.360 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.360 INFO:teuthology.orchestra.run.smithi096.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T03:03:50.360 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_messages": 2304, 2024-03-19T03:03:50.360 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages": 1803, 2024-03-19T03:03:50.360 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_bytes": 2136173, 2024-03-19T03:03:50.360 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_bytes": 992152, 2024-03-19T03:03:50.360 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_created_connections": 13, 2024-03-19T03:03:50.360 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_active_connections": 7, 2024-03-19T03:03:50.360 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_total_time": 0.178893897, 2024-03-19T03:03:50.360 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_send_time": 0.044733446, 2024-03-19T03:03:50.360 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_recv_time": 0.079041847, 2024-03-19T03:03:50.360 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_fast_dispatch_time": 0.056675295, 2024-03-19T03:03:50.360 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:03:50.360 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 1803, 2024-03-19T03:03:50.360 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.440455282, 2024-03-19T03:03:50.361 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000244290 2024-03-19T03:03:50.361 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.361 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:03:50.361 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 1111, 2024-03-19T03:03:50.361 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000187522, 2024-03-19T03:03:50.361 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000168 2024-03-19T03:03:50.361 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.361 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_encrypted_bytes": 1556512, 2024-03-19T03:03:50.361 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_encrypted_bytes": 507952 2024-03-19T03:03:50.361 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.362 INFO:teuthology.orchestra.run.smithi096.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T03:03:50.362 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_messages": 3156, 2024-03-19T03:03:50.362 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages": 2429, 2024-03-19T03:03:50.362 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_bytes": 1035446, 2024-03-19T03:03:50.362 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_bytes": 1082489, 2024-03-19T03:03:50.362 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_created_connections": 22, 2024-03-19T03:03:50.362 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_active_connections": 6, 2024-03-19T03:03:50.362 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_total_time": 0.229210380, 2024-03-19T03:03:50.362 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_send_time": 0.067455882, 2024-03-19T03:03:50.362 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_recv_time": 0.074041424, 2024-03-19T03:03:50.362 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_fast_dispatch_time": 0.079971401, 2024-03-19T03:03:50.362 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:03:50.362 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 2426, 2024-03-19T03:03:50.362 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.682369750, 2024-03-19T03:03:50.363 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000281273 2024-03-19T03:03:50.363 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.363 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:03:50.363 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 2009, 2024-03-19T03:03:50.363 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000310719, 2024-03-19T03:03:50.363 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000154 2024-03-19T03:03:50.363 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.363 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_encrypted_bytes": 351328, 2024-03-19T03:03:50.363 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_encrypted_bytes": 227040 2024-03-19T03:03:50.364 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.364 INFO:teuthology.orchestra.run.smithi096.stdout: "cct": { 2024-03-19T03:03:50.364 INFO:teuthology.orchestra.run.smithi096.stdout: "total_workers": 1, 2024-03-19T03:03:50.364 INFO:teuthology.orchestra.run.smithi096.stdout: "unhealthy_workers": 0 2024-03-19T03:03:50.364 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.364 INFO:teuthology.orchestra.run.smithi096.stdout: "finisher-MDSRank": { 2024-03-19T03:03:50.364 INFO:teuthology.orchestra.run.smithi096.stdout: "queue_len": 0, 2024-03-19T03:03:50.364 INFO:teuthology.orchestra.run.smithi096.stdout: "complete_latency": { 2024-03-19T03:03:50.364 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 523, 2024-03-19T03:03:50.364 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.151102246, 2024-03-19T03:03:50.364 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000288914 2024-03-19T03:03:50.364 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-03-19T03:03:50.364 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.364 INFO:teuthology.orchestra.run.smithi096.stdout: "finisher-PurgeQueue": { 2024-03-19T03:03:50.364 INFO:teuthology.orchestra.run.smithi096.stdout: "queue_len": 0, 2024-03-19T03:03:50.365 INFO:teuthology.orchestra.run.smithi096.stdout: "complete_latency": { 2024-03-19T03:03:50.365 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 2824, 2024-03-19T03:03:50.365 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.115017811, 2024-03-19T03:03:50.365 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000040728 2024-03-19T03:03:50.365 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-03-19T03:03:50.365 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.365 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-03-19T03:03:50.365 INFO:teuthology.orchestra.run.smithi096.stdout: "request": 174, 2024-03-19T03:03:50.365 INFO:teuthology.orchestra.run.smithi096.stdout: "reply": 174, 2024-03-19T03:03:50.365 INFO:teuthology.orchestra.run.smithi096.stdout: "reply_latency": { 2024-03-19T03:03:50.365 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 174, 2024-03-19T03:03:50.365 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.149113186, 2024-03-19T03:03:50.365 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000856972 2024-03-19T03:03:50.365 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.366 INFO:teuthology.orchestra.run.smithi096.stdout: "slow_reply": 0, 2024-03-19T03:03:50.366 INFO:teuthology.orchestra.run.smithi096.stdout: "forward": 0, 2024-03-19T03:03:50.366 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_fetch_complete": 0, 2024-03-19T03:03:50.366 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_fetch_keys": 0, 2024-03-19T03:03:50.366 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_commit": 12, 2024-03-19T03:03:50.366 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_split": 0, 2024-03-19T03:03:50.366 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_merge": 0, 2024-03-19T03:03:50.366 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes": 35, 2024-03-19T03:03:50.366 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes_top": 1, 2024-03-19T03:03:50.366 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes_bottom": 34, 2024-03-19T03:03:50.366 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes_pin_tail": 0, 2024-03-19T03:03:50.366 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes_pinned": 33, 2024-03-19T03:03:50.366 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes_expired": 0, 2024-03-19T03:03:50.366 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes_with_caps": 2, 2024-03-19T03:03:50.367 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": 3, 2024-03-19T03:03:50.367 INFO:teuthology.orchestra.run.smithi096.stdout: "subtrees": 2, 2024-03-19T03:03:50.367 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse": 245, 2024-03-19T03:03:50.367 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse_hit": 232, 2024-03-19T03:03:50.367 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse_forward": 0, 2024-03-19T03:03:50.367 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse_discover": 0, 2024-03-19T03:03:50.367 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse_dir_fetch": 0, 2024-03-19T03:03:50.367 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse_remote_ino": 0, 2024-03-19T03:03:50.367 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse_lock": 0, 2024-03-19T03:03:50.367 INFO:teuthology.orchestra.run.smithi096.stdout: "load_cent": 0, 2024-03-19T03:03:50.367 INFO:teuthology.orchestra.run.smithi096.stdout: "q": 0, 2024-03-19T03:03:50.367 INFO:teuthology.orchestra.run.smithi096.stdout: "exported": 0, 2024-03-19T03:03:50.367 INFO:teuthology.orchestra.run.smithi096.stdout: "exported_inodes": 0, 2024-03-19T03:03:50.367 INFO:teuthology.orchestra.run.smithi096.stdout: "imported": 0, 2024-03-19T03:03:50.367 INFO:teuthology.orchestra.run.smithi096.stdout: "imported_inodes": 0, 2024-03-19T03:03:50.368 INFO:teuthology.orchestra.run.smithi096.stdout: "openino_dir_fetch": 0, 2024-03-19T03:03:50.368 INFO:teuthology.orchestra.run.smithi096.stdout: "openino_backtrace_fetch": 222, 2024-03-19T03:03:50.368 INFO:teuthology.orchestra.run.smithi096.stdout: "openino_peer_discover": 0, 2024-03-19T03:03:50.368 INFO:teuthology.orchestra.run.smithi096.stdout: "root_rfiles": 4, 2024-03-19T03:03:50.368 INFO:teuthology.orchestra.run.smithi096.stdout: "root_rbytes": 234, 2024-03-19T03:03:50.368 INFO:teuthology.orchestra.run.smithi096.stdout: "root_rsnaps": 6, 2024-03-19T03:03:50.368 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_backtrace_fetch": 222, 2024-03-19T03:03:50.368 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_set_tag": 0, 2024-03-19T03:03:50.368 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T03:03:50.368 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_inotable_repaired": 0, 2024-03-19T03:03:50.368 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_dir_inodes": 142, 2024-03-19T03:03:50.368 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_dir_base_inodes": 19, 2024-03-19T03:03:50.368 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_dirfrag_rstats": 123, 2024-03-19T03:03:50.368 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_file_inodes": 80, 2024-03-19T03:03:50.368 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_inode_file_caps": 0, 2024-03-19T03:03:50.369 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_cap_op_revoke": 17, 2024-03-19T03:03:50.369 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_cap_op_grant": 73, 2024-03-19T03:03:50.369 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T03:03:50.369 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T03:03:50.369 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T03:03:50.369 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_client_caps": 83, 2024-03-19T03:03:50.369 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_client_caps_dirty": 59, 2024-03-19T03:03:50.369 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_client_cap_release": 3, 2024-03-19T03:03:50.369 INFO:teuthology.orchestra.run.smithi096.stdout: "process_request_cap_release": 33 2024-03-19T03:03:50.369 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.369 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_cache": { 2024-03-19T03:03:50.369 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_update": 8, 2024-03-19T03:03:50.369 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_update_receipt": 0, 2024-03-19T03:03:50.369 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_try_discover": 0, 2024-03-19T03:03:50.369 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_send_discover": 0, 2024-03-19T03:03:50.370 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_handle_discover": 16, 2024-03-19T03:03:50.370 INFO:teuthology.orchestra.run.smithi096.stdout: "num_strays": 2, 2024-03-19T03:03:50.370 INFO:teuthology.orchestra.run.smithi096.stdout: "num_strays_delayed": 0, 2024-03-19T03:03:50.370 INFO:teuthology.orchestra.run.smithi096.stdout: "num_strays_enqueuing": 0, 2024-03-19T03:03:50.370 INFO:teuthology.orchestra.run.smithi096.stdout: "strays_created": 5, 2024-03-19T03:03:50.370 INFO:teuthology.orchestra.run.smithi096.stdout: "strays_enqueued": 3, 2024-03-19T03:03:50.370 INFO:teuthology.orchestra.run.smithi096.stdout: "strays_reintegrated": 0, 2024-03-19T03:03:50.370 INFO:teuthology.orchestra.run.smithi096.stdout: "strays_migrated": 0, 2024-03-19T03:03:50.370 INFO:teuthology.orchestra.run.smithi096.stdout: "num_recovering_processing": 0, 2024-03-19T03:03:50.370 INFO:teuthology.orchestra.run.smithi096.stdout: "num_recovering_enqueued": 0, 2024-03-19T03:03:50.370 INFO:teuthology.orchestra.run.smithi096.stdout: "num_recovering_prioritized": 0, 2024-03-19T03:03:50.370 INFO:teuthology.orchestra.run.smithi096.stdout: "recovery_started": 0, 2024-03-19T03:03:50.371 INFO:teuthology.orchestra.run.smithi096.stdout: "recovery_completed": 0, 2024-03-19T03:03:50.371 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_quiesce_path": 0, 2024-03-19T03:03:50.371 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_quiesce_inode": 0, 2024-03-19T03:03:50.371 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_enqueue_scrub": 19, 2024-03-19T03:03:50.371 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_exportdir": 0, 2024-03-19T03:03:50.371 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_flush": 0, 2024-03-19T03:03:50.371 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_fragmentdir": 0, 2024-03-19T03:03:50.371 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_fragstats": 0, 2024-03-19T03:03:50.371 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_inodestats": 0 2024-03-19T03:03:50.371 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.371 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_log": { 2024-03-19T03:03:50.371 INFO:teuthology.orchestra.run.smithi096.stdout: "evlrg": 0, 2024-03-19T03:03:50.371 INFO:teuthology.orchestra.run.smithi096.stdout: "evadd": 195, 2024-03-19T03:03:50.371 INFO:teuthology.orchestra.run.smithi096.stdout: "evex": 0, 2024-03-19T03:03:50.371 INFO:teuthology.orchestra.run.smithi096.stdout: "evtrm": 0, 2024-03-19T03:03:50.372 INFO:teuthology.orchestra.run.smithi096.stdout: "ev": 195, 2024-03-19T03:03:50.372 INFO:teuthology.orchestra.run.smithi096.stdout: "evexg": 0, 2024-03-19T03:03:50.372 INFO:teuthology.orchestra.run.smithi096.stdout: "evexd": 0, 2024-03-19T03:03:50.372 INFO:teuthology.orchestra.run.smithi096.stdout: "segadd": 2, 2024-03-19T03:03:50.372 INFO:teuthology.orchestra.run.smithi096.stdout: "segex": 0, 2024-03-19T03:03:50.372 INFO:teuthology.orchestra.run.smithi096.stdout: "segtrm": 0, 2024-03-19T03:03:50.372 INFO:teuthology.orchestra.run.smithi096.stdout: "seg": 2, 2024-03-19T03:03:50.372 INFO:teuthology.orchestra.run.smithi096.stdout: "segmjr": 2, 2024-03-19T03:03:50.372 INFO:teuthology.orchestra.run.smithi096.stdout: "segexg": 0, 2024-03-19T03:03:50.372 INFO:teuthology.orchestra.run.smithi096.stdout: "segexd": 0, 2024-03-19T03:03:50.372 INFO:teuthology.orchestra.run.smithi096.stdout: "expos": 4194304, 2024-03-19T03:03:50.372 INFO:teuthology.orchestra.run.smithi096.stdout: "wrpos": 4824197, 2024-03-19T03:03:50.372 INFO:teuthology.orchestra.run.smithi096.stdout: "rdpos": 0, 2024-03-19T03:03:50.372 INFO:teuthology.orchestra.run.smithi096.stdout: "jlat": { 2024-03-19T03:03:50.372 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 163, 2024-03-19T03:03:50.373 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.318913074, 2024-03-19T03:03:50.373 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.001956521 2024-03-19T03:03:50.373 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.373 INFO:teuthology.orchestra.run.smithi096.stdout: "replayed": 0 2024-03-19T03:03:50.373 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.373 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_mem": { 2024-03-19T03:03:50.373 INFO:teuthology.orchestra.run.smithi096.stdout: "ino": 27, 2024-03-19T03:03:50.373 INFO:teuthology.orchestra.run.smithi096.stdout: "ino+": 49, 2024-03-19T03:03:50.373 INFO:teuthology.orchestra.run.smithi096.stdout: "ino-": 22, 2024-03-19T03:03:50.373 INFO:teuthology.orchestra.run.smithi096.stdout: "dir": 20, 2024-03-19T03:03:50.373 INFO:teuthology.orchestra.run.smithi096.stdout: "dir+": 20, 2024-03-19T03:03:50.373 INFO:teuthology.orchestra.run.smithi096.stdout: "dir-": 0, 2024-03-19T03:03:50.373 INFO:teuthology.orchestra.run.smithi096.stdout: "dn": 35, 2024-03-19T03:03:50.373 INFO:teuthology.orchestra.run.smithi096.stdout: "dn+": 38, 2024-03-19T03:03:50.373 INFO:teuthology.orchestra.run.smithi096.stdout: "dn-": 3, 2024-03-19T03:03:50.374 INFO:teuthology.orchestra.run.smithi096.stdout: "cap": 3, 2024-03-19T03:03:50.374 INFO:teuthology.orchestra.run.smithi096.stdout: "cap+": 22, 2024-03-19T03:03:50.374 INFO:teuthology.orchestra.run.smithi096.stdout: "cap-": 19, 2024-03-19T03:03:50.374 INFO:teuthology.orchestra.run.smithi096.stdout: "rss": 56436, 2024-03-19T03:03:50.374 INFO:teuthology.orchestra.run.smithi096.stdout: "heap": 231804 2024-03-19T03:03:50.374 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.374 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_server": { 2024-03-19T03:03:50.374 INFO:teuthology.orchestra.run.smithi096.stdout: "dispatch_client_request": 230, 2024-03-19T03:03:50.374 INFO:teuthology.orchestra.run.smithi096.stdout: "dispatch_server_request": 0, 2024-03-19T03:03:50.374 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_client_request": 174, 2024-03-19T03:03:50.374 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_client_session": 35, 2024-03-19T03:03:50.374 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_peer_request": 0, 2024-03-19T03:03:50.374 INFO:teuthology.orchestra.run.smithi096.stdout: "req_create_latency": { 2024-03-19T03:03:50.374 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 7, 2024-03-19T03:03:50.374 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.009091587, 2024-03-19T03:03:50.375 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.001298798 2024-03-19T03:03:50.375 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.375 INFO:teuthology.orchestra.run.smithi096.stdout: "req_getattr_latency": { 2024-03-19T03:03:50.375 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 110, 2024-03-19T03:03:50.375 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.020458532, 2024-03-19T03:03:50.375 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000185986 2024-03-19T03:03:50.375 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.375 INFO:teuthology.orchestra.run.smithi096.stdout: "req_getfilelock_latency": { 2024-03-19T03:03:50.375 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:50.375 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:50.375 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:50.375 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.375 INFO:teuthology.orchestra.run.smithi096.stdout: "req_link_latency": { 2024-03-19T03:03:50.375 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:50.375 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:50.376 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:50.376 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.376 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lookup_latency": { 2024-03-19T03:03:50.376 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 3, 2024-03-19T03:03:50.376 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000871493, 2024-03-19T03:03:50.376 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000290497 2024-03-19T03:03:50.376 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.376 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lookuphash_latency": { 2024-03-19T03:03:50.376 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:50.376 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:50.376 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:50.376 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.376 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lookupino_latency": { 2024-03-19T03:03:50.376 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:50.376 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:50.376 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:50.377 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.377 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lookupname_latency": { 2024-03-19T03:03:50.377 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:50.377 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:50.377 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:50.377 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.377 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lookupparent_latency": { 2024-03-19T03:03:50.377 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:50.377 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:50.377 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:50.377 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.377 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lookupsnap_latency": { 2024-03-19T03:03:50.377 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:50.377 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:50.377 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:50.378 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.378 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lssnap_latency": { 2024-03-19T03:03:50.378 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 10, 2024-03-19T03:03:50.378 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.001494854, 2024-03-19T03:03:50.378 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000149485 2024-03-19T03:03:50.378 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.378 INFO:teuthology.orchestra.run.smithi096.stdout: "req_mkdir_latency": { 2024-03-19T03:03:50.378 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 8, 2024-03-19T03:03:50.378 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.010437569, 2024-03-19T03:03:50.378 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.001304696 2024-03-19T03:03:50.378 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.378 INFO:teuthology.orchestra.run.smithi096.stdout: "req_mknod_latency": { 2024-03-19T03:03:50.378 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:50.378 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:50.379 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:50.379 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.379 INFO:teuthology.orchestra.run.smithi096.stdout: "req_mksnap_latency": { 2024-03-19T03:03:50.379 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 10, 2024-03-19T03:03:50.379 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.041175403, 2024-03-19T03:03:50.379 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.004117540 2024-03-19T03:03:50.379 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.379 INFO:teuthology.orchestra.run.smithi096.stdout: "req_open_latency": { 2024-03-19T03:03:50.379 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 3, 2024-03-19T03:03:50.379 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.001834986, 2024-03-19T03:03:50.379 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000611662 2024-03-19T03:03:50.379 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.379 INFO:teuthology.orchestra.run.smithi096.stdout: "req_readdir_latency": { 2024-03-19T03:03:50.379 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 2, 2024-03-19T03:03:50.379 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.006855230, 2024-03-19T03:03:50.380 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.003427615 2024-03-19T03:03:50.380 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.380 INFO:teuthology.orchestra.run.smithi096.stdout: "req_rename_latency": { 2024-03-19T03:03:50.380 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 2, 2024-03-19T03:03:50.380 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.003021045, 2024-03-19T03:03:50.380 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.001510522 2024-03-19T03:03:50.380 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.380 INFO:teuthology.orchestra.run.smithi096.stdout: "req_renamesnap_latency": { 2024-03-19T03:03:50.380 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:50.380 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:50.380 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:50.380 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.380 INFO:teuthology.orchestra.run.smithi096.stdout: "req_snapdiff_latency": { 2024-03-19T03:03:50.380 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:50.380 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:50.381 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:50.381 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.381 INFO:teuthology.orchestra.run.smithi096.stdout: "req_rmdir_latency": { 2024-03-19T03:03:50.381 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 2, 2024-03-19T03:03:50.381 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.011891966, 2024-03-19T03:03:50.381 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.005945983 2024-03-19T03:03:50.381 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.381 INFO:teuthology.orchestra.run.smithi096.stdout: "req_rmsnap_latency": { 2024-03-19T03:03:50.381 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 4, 2024-03-19T03:03:50.381 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.015639317, 2024-03-19T03:03:50.381 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.003909829 2024-03-19T03:03:50.381 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.381 INFO:teuthology.orchestra.run.smithi096.stdout: "req_rmxattr_latency": { 2024-03-19T03:03:50.381 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:50.381 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:50.382 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:50.382 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.382 INFO:teuthology.orchestra.run.smithi096.stdout: "req_setattr_latency": { 2024-03-19T03:03:50.382 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 2, 2024-03-19T03:03:50.382 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.001036581, 2024-03-19T03:03:50.382 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000518290 2024-03-19T03:03:50.382 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.382 INFO:teuthology.orchestra.run.smithi096.stdout: "req_setdirlayout_latency": { 2024-03-19T03:03:50.382 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:50.382 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:50.382 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:50.382 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.382 INFO:teuthology.orchestra.run.smithi096.stdout: "req_setfilelock_latency": { 2024-03-19T03:03:50.382 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:50.383 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:50.383 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:50.383 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.383 INFO:teuthology.orchestra.run.smithi096.stdout: "req_setlayout_latency": { 2024-03-19T03:03:50.383 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:50.383 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:50.383 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:50.383 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.383 INFO:teuthology.orchestra.run.smithi096.stdout: "req_setxattr_latency": { 2024-03-19T03:03:50.383 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 8, 2024-03-19T03:03:50.383 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.019462394, 2024-03-19T03:03:50.383 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.002432799 2024-03-19T03:03:50.383 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.383 INFO:teuthology.orchestra.run.smithi096.stdout: "req_symlink_latency": { 2024-03-19T03:03:50.383 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:50.384 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:50.384 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:50.384 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.384 INFO:teuthology.orchestra.run.smithi096.stdout: "req_unlink_latency": { 2024-03-19T03:03:50.384 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 3, 2024-03-19T03:03:50.384 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.005842229, 2024-03-19T03:03:50.384 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.001947409 2024-03-19T03:03:50.384 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.384 INFO:teuthology.orchestra.run.smithi096.stdout: "cap_revoke_eviction": 0, 2024-03-19T03:03:50.384 INFO:teuthology.orchestra.run.smithi096.stdout: "cap_acquisition_throttle": 0, 2024-03-19T03:03:50.384 INFO:teuthology.orchestra.run.smithi096.stdout: "req_getvxattr_latency": { 2024-03-19T03:03:50.384 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:50.384 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:50.384 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:50.384 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-03-19T03:03:50.385 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.385 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_sessions": { 2024-03-19T03:03:50.385 INFO:teuthology.orchestra.run.smithi096.stdout: "session_count": 2, 2024-03-19T03:03:50.385 INFO:teuthology.orchestra.run.smithi096.stdout: "session_add": 6, 2024-03-19T03:03:50.385 INFO:teuthology.orchestra.run.smithi096.stdout: "session_remove": 4, 2024-03-19T03:03:50.385 INFO:teuthology.orchestra.run.smithi096.stdout: "sessions_open": 2, 2024-03-19T03:03:50.385 INFO:teuthology.orchestra.run.smithi096.stdout: "sessions_stale": 0, 2024-03-19T03:03:50.385 INFO:teuthology.orchestra.run.smithi096.stdout: "total_load": 27, 2024-03-19T03:03:50.385 INFO:teuthology.orchestra.run.smithi096.stdout: "average_load": 13, 2024-03-19T03:03:50.385 INFO:teuthology.orchestra.run.smithi096.stdout: "avg_session_uptime": 611, 2024-03-19T03:03:50.385 INFO:teuthology.orchestra.run.smithi096.stdout: "mdthresh_evicted": 0 2024-03-19T03:03:50.385 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.385 INFO:teuthology.orchestra.run.smithi096.stdout: "mempool": { 2024-03-19T03:03:50.385 INFO:teuthology.orchestra.run.smithi096.stdout: "bloom_filter_bytes": 0, 2024-03-19T03:03:50.385 INFO:teuthology.orchestra.run.smithi096.stdout: "bloom_filter_items": 0, 2024-03-19T03:03:50.386 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T03:03:50.386 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_alloc_items": 0, 2024-03-19T03:03:50.386 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T03:03:50.386 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_data_items": 0, 2024-03-19T03:03:50.386 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T03:03:50.386 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T03:03:50.386 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T03:03:50.386 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T03:03:50.386 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T03:03:50.386 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_other_items": 0, 2024-03-19T03:03:50.386 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T03:03:50.386 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T03:03:50.386 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_extent_bytes": 0, 2024-03-19T03:03:50.386 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_extent_items": 0, 2024-03-19T03:03:50.386 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_blob_bytes": 0, 2024-03-19T03:03:50.387 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_blob_items": 0, 2024-03-19T03:03:50.387 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T03:03:50.387 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T03:03:50.387 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T03:03:50.387 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T03:03:50.387 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T03:03:50.387 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_fsck_items": 0, 2024-03-19T03:03:50.387 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_txc_bytes": 0, 2024-03-19T03:03:50.387 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_txc_items": 0, 2024-03-19T03:03:50.387 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T03:03:50.387 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T03:03:50.387 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_writing_bytes": 0, 2024-03-19T03:03:50.387 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_writing_items": 0, 2024-03-19T03:03:50.387 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_bytes": 0, 2024-03-19T03:03:50.387 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_items": 0, 2024-03-19T03:03:50.388 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T03:03:50.388 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_file_reader_items": 0, 2024-03-19T03:03:50.388 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T03:03:50.388 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_file_writer_items": 0, 2024-03-19T03:03:50.388 INFO:teuthology.orchestra.run.smithi096.stdout: "buffer_anon_bytes": 338749, 2024-03-19T03:03:50.388 INFO:teuthology.orchestra.run.smithi096.stdout: "buffer_anon_items": 132, 2024-03-19T03:03:50.388 INFO:teuthology.orchestra.run.smithi096.stdout: "buffer_meta_bytes": 352, 2024-03-19T03:03:50.388 INFO:teuthology.orchestra.run.smithi096.stdout: "buffer_meta_items": 4, 2024-03-19T03:03:50.388 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_bytes": 0, 2024-03-19T03:03:50.388 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_items": 0, 2024-03-19T03:03:50.388 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_mapbl_bytes": 0, 2024-03-19T03:03:50.388 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_mapbl_items": 0, 2024-03-19T03:03:50.388 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_pglog_bytes": 0, 2024-03-19T03:03:50.388 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_pglog_items": 0, 2024-03-19T03:03:50.388 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_bytes": 9512, 2024-03-19T03:03:50.389 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_items": 182, 2024-03-19T03:03:50.389 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T03:03:50.389 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_mapping_items": 0, 2024-03-19T03:03:50.389 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap_bytes": 0, 2024-03-19T03:03:50.389 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap_items": 0, 2024-03-19T03:03:50.389 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_co_bytes": 133756, 2024-03-19T03:03:50.389 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_co_items": 971, 2024-03-19T03:03:50.389 INFO:teuthology.orchestra.run.smithi096.stdout: "unittest_1_bytes": 0, 2024-03-19T03:03:50.389 INFO:teuthology.orchestra.run.smithi096.stdout: "unittest_1_items": 0, 2024-03-19T03:03:50.389 INFO:teuthology.orchestra.run.smithi096.stdout: "unittest_2_bytes": 0, 2024-03-19T03:03:50.389 INFO:teuthology.orchestra.run.smithi096.stdout: "unittest_2_items": 0 2024-03-19T03:03:50.389 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.389 INFO:teuthology.orchestra.run.smithi096.stdout: "objecter": { 2024-03-19T03:03:50.389 INFO:teuthology.orchestra.run.smithi096.stdout: "op_active": 0, 2024-03-19T03:03:50.389 INFO:teuthology.orchestra.run.smithi096.stdout: "op_laggy": 0, 2024-03-19T03:03:50.390 INFO:teuthology.orchestra.run.smithi096.stdout: "op_send": 4102, 2024-03-19T03:03:50.390 INFO:teuthology.orchestra.run.smithi096.stdout: "op_send_bytes": 649282, 2024-03-19T03:03:50.390 INFO:teuthology.orchestra.run.smithi096.stdout: "op_resend": 37, 2024-03-19T03:03:50.390 INFO:teuthology.orchestra.run.smithi096.stdout: "op_reply": 4102, 2024-03-19T03:03:50.390 INFO:teuthology.orchestra.run.smithi096.stdout: "op_latency": { 2024-03-19T03:03:50.390 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 4102, 2024-03-19T03:03:50.390 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 56.671612972, 2024-03-19T03:03:50.390 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.013815605 2024-03-19T03:03:50.390 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.390 INFO:teuthology.orchestra.run.smithi096.stdout: "op_inflight": 0, 2024-03-19T03:03:50.390 INFO:teuthology.orchestra.run.smithi096.stdout: "oplen_avg": { 2024-03-19T03:03:50.390 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 4102, 2024-03-19T03:03:50.390 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 4157 2024-03-19T03:03:50.390 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.390 INFO:teuthology.orchestra.run.smithi096.stdout: "op": 4102, 2024-03-19T03:03:50.391 INFO:teuthology.orchestra.run.smithi096.stdout: "op_r": 263, 2024-03-19T03:03:50.391 INFO:teuthology.orchestra.run.smithi096.stdout: "op_w": 3839, 2024-03-19T03:03:50.391 INFO:teuthology.orchestra.run.smithi096.stdout: "op_rmw": 0, 2024-03-19T03:03:50.391 INFO:teuthology.orchestra.run.smithi096.stdout: "op_pg": 0, 2024-03-19T03:03:50.391 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_stat": 0, 2024-03-19T03:03:50.391 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_create": 11, 2024-03-19T03:03:50.391 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_read": 22, 2024-03-19T03:03:50.391 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_write": 166, 2024-03-19T03:03:50.391 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_writefull": 33, 2024-03-19T03:03:50.391 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_writesame": 0, 2024-03-19T03:03:50.391 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_append": 0, 2024-03-19T03:03:50.391 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_zero": 0, 2024-03-19T03:03:50.391 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_truncate": 0, 2024-03-19T03:03:50.391 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_delete": 3587, 2024-03-19T03:03:50.391 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_mapext": 0, 2024-03-19T03:03:50.392 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_sparse_read": 0, 2024-03-19T03:03:50.392 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_clonerange": 0, 2024-03-19T03:03:50.392 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_getxattr": 241, 2024-03-19T03:03:50.392 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_setxattr": 22, 2024-03-19T03:03:50.392 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_cmpxattr": 0, 2024-03-19T03:03:50.392 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_rmxattr": 0, 2024-03-19T03:03:50.392 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_resetxattrs": 0, 2024-03-19T03:03:50.392 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_call": 0, 2024-03-19T03:03:50.392 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_watch": 0, 2024-03-19T03:03:50.392 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_notify": 0, 2024-03-19T03:03:50.392 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T03:03:50.392 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_pgls": 0, 2024-03-19T03:03:50.392 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_pgls_filter": 0, 2024-03-19T03:03:50.392 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_other": 0, 2024-03-19T03:03:50.392 INFO:teuthology.orchestra.run.smithi096.stdout: "linger_active": 0, 2024-03-19T03:03:50.392 INFO:teuthology.orchestra.run.smithi096.stdout: "linger_send": 0, 2024-03-19T03:03:50.393 INFO:teuthology.orchestra.run.smithi096.stdout: "linger_resend": 0, 2024-03-19T03:03:50.393 INFO:teuthology.orchestra.run.smithi096.stdout: "linger_ping": 0, 2024-03-19T03:03:50.393 INFO:teuthology.orchestra.run.smithi096.stdout: "poolop_active": 0, 2024-03-19T03:03:50.393 INFO:teuthology.orchestra.run.smithi096.stdout: "poolop_send": 0, 2024-03-19T03:03:50.393 INFO:teuthology.orchestra.run.smithi096.stdout: "poolop_resend": 0, 2024-03-19T03:03:50.393 INFO:teuthology.orchestra.run.smithi096.stdout: "poolstat_active": 0, 2024-03-19T03:03:50.393 INFO:teuthology.orchestra.run.smithi096.stdout: "poolstat_send": 0, 2024-03-19T03:03:50.393 INFO:teuthology.orchestra.run.smithi096.stdout: "poolstat_resend": 0, 2024-03-19T03:03:50.393 INFO:teuthology.orchestra.run.smithi096.stdout: "statfs_active": 0, 2024-03-19T03:03:50.393 INFO:teuthology.orchestra.run.smithi096.stdout: "statfs_send": 0, 2024-03-19T03:03:50.393 INFO:teuthology.orchestra.run.smithi096.stdout: "statfs_resend": 0, 2024-03-19T03:03:50.393 INFO:teuthology.orchestra.run.smithi096.stdout: "command_active": 0, 2024-03-19T03:03:50.393 INFO:teuthology.orchestra.run.smithi096.stdout: "command_send": 0, 2024-03-19T03:03:50.393 INFO:teuthology.orchestra.run.smithi096.stdout: "command_resend": 0, 2024-03-19T03:03:50.393 INFO:teuthology.orchestra.run.smithi096.stdout: "map_epoch": 106, 2024-03-19T03:03:50.394 INFO:teuthology.orchestra.run.smithi096.stdout: "map_full": 0, 2024-03-19T03:03:50.394 INFO:teuthology.orchestra.run.smithi096.stdout: "map_inc": 20, 2024-03-19T03:03:50.394 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_sessions": 12, 2024-03-19T03:03:50.394 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_session_open": 12, 2024-03-19T03:03:50.394 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_session_close": 0, 2024-03-19T03:03:50.394 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_laggy": 0, 2024-03-19T03:03:50.394 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_wr": 60, 2024-03-19T03:03:50.394 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_rd": 0, 2024-03-19T03:03:50.394 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_del": 15 2024-03-19T03:03:50.394 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.394 INFO:teuthology.orchestra.run.smithi096.stdout: "oft": { 2024-03-19T03:03:50.394 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_total_objs": 1, 2024-03-19T03:03:50.394 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_total_kv_pairs": 0, 2024-03-19T03:03:50.394 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_total_updates": 0, 2024-03-19T03:03:50.395 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_total_removes": 0 2024-03-19T03:03:50.395 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.395 INFO:teuthology.orchestra.run.smithi096.stdout: "osd-slow-ops": { 2024-03-19T03:03:50.395 INFO:teuthology.orchestra.run.smithi096.stdout: "slow_ops_count": 0 2024-03-19T03:03:50.395 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.395 INFO:teuthology.orchestra.run.smithi096.stdout: "purge_queue": { 2024-03-19T03:03:50.395 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_executing_ops": 0, 2024-03-19T03:03:50.395 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_executing_ops_high_water": 2561, 2024-03-19T03:03:50.395 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_executing": 0, 2024-03-19T03:03:50.395 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_executing_high_water": 1, 2024-03-19T03:03:50.395 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_executed_ops": 3075, 2024-03-19T03:03:50.395 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_executed": 3, 2024-03-19T03:03:50.395 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_item_in_journal": 0 2024-03-19T03:03:50.395 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.395 INFO:teuthology.orchestra.run.smithi096.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T03:03:50.395 INFO:teuthology.orchestra.run.smithi096.stdout: "val": 0, 2024-03-19T03:03:50.396 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 104857600, 2024-03-19T03:03:50.396 INFO:teuthology.orchestra.run.smithi096.stdout: "get_started": 0, 2024-03-19T03:03:50.396 INFO:teuthology.orchestra.run.smithi096.stdout: "get": 7514, 2024-03-19T03:03:50.396 INFO:teuthology.orchestra.run.smithi096.stdout: "get_sum": 3130552, 2024-03-19T03:03:50.396 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_fail": 0, 2024-03-19T03:03:50.396 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_success": 7514, 2024-03-19T03:03:50.396 INFO:teuthology.orchestra.run.smithi096.stdout: "take": 0, 2024-03-19T03:03:50.396 INFO:teuthology.orchestra.run.smithi096.stdout: "take_sum": 0, 2024-03-19T03:03:50.396 INFO:teuthology.orchestra.run.smithi096.stdout: "put": 7514, 2024-03-19T03:03:50.396 INFO:teuthology.orchestra.run.smithi096.stdout: "put_sum": 3130552, 2024-03-19T03:03:50.396 INFO:teuthology.orchestra.run.smithi096.stdout: "wait": { 2024-03-19T03:03:50.396 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:50.396 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:50.396 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:50.396 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-03-19T03:03:50.397 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.397 INFO:teuthology.orchestra.run.smithi096.stdout: "throttle-objecter_bytes": { 2024-03-19T03:03:50.397 INFO:teuthology.orchestra.run.smithi096.stdout: "val": 0, 2024-03-19T03:03:50.397 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 104857600, 2024-03-19T03:03:50.397 INFO:teuthology.orchestra.run.smithi096.stdout: "get_started": 0, 2024-03-19T03:03:50.397 INFO:teuthology.orchestra.run.smithi096.stdout: "get": 0, 2024-03-19T03:03:50.397 INFO:teuthology.orchestra.run.smithi096.stdout: "get_sum": 0, 2024-03-19T03:03:50.397 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_fail": 0, 2024-03-19T03:03:50.397 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_success": 0, 2024-03-19T03:03:50.397 INFO:teuthology.orchestra.run.smithi096.stdout: "take": 4102, 2024-03-19T03:03:50.398 INFO:teuthology.orchestra.run.smithi096.stdout: "take_sum": 649625, 2024-03-19T03:03:50.398 INFO:teuthology.orchestra.run.smithi096.stdout: "put": 4102, 2024-03-19T03:03:50.398 INFO:teuthology.orchestra.run.smithi096.stdout: "put_sum": 649625, 2024-03-19T03:03:50.398 INFO:teuthology.orchestra.run.smithi096.stdout: "wait": { 2024-03-19T03:03:50.398 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:50.398 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:50.398 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:50.398 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-03-19T03:03:50.398 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.399 INFO:teuthology.orchestra.run.smithi096.stdout: "throttle-objecter_ops": { 2024-03-19T03:03:50.399 INFO:teuthology.orchestra.run.smithi096.stdout: "val": 0, 2024-03-19T03:03:50.399 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1024, 2024-03-19T03:03:50.399 INFO:teuthology.orchestra.run.smithi096.stdout: "get_started": 0, 2024-03-19T03:03:50.399 INFO:teuthology.orchestra.run.smithi096.stdout: "get": 0, 2024-03-19T03:03:50.399 INFO:teuthology.orchestra.run.smithi096.stdout: "get_sum": 0, 2024-03-19T03:03:50.399 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_fail": 0, 2024-03-19T03:03:50.399 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_success": 0, 2024-03-19T03:03:50.399 INFO:teuthology.orchestra.run.smithi096.stdout: "take": 4102, 2024-03-19T03:03:50.399 INFO:teuthology.orchestra.run.smithi096.stdout: "take_sum": 4102, 2024-03-19T03:03:50.399 INFO:teuthology.orchestra.run.smithi096.stdout: "put": 4102, 2024-03-19T03:03:50.399 INFO:teuthology.orchestra.run.smithi096.stdout: "put_sum": 4102, 2024-03-19T03:03:50.399 INFO:teuthology.orchestra.run.smithi096.stdout: "wait": { 2024-03-19T03:03:50.399 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:50.399 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:50.400 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:50.400 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-03-19T03:03:50.400 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.400 INFO:teuthology.orchestra.run.smithi096.stdout: "throttle-write_buf_throttle": { 2024-03-19T03:03:50.400 INFO:teuthology.orchestra.run.smithi096.stdout: "val": 0, 2024-03-19T03:03:50.400 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 3758096384, 2024-03-19T03:03:50.400 INFO:teuthology.orchestra.run.smithi096.stdout: "get_started": 0, 2024-03-19T03:03:50.400 INFO:teuthology.orchestra.run.smithi096.stdout: "get": 3, 2024-03-19T03:03:50.400 INFO:teuthology.orchestra.run.smithi096.stdout: "get_sum": 343, 2024-03-19T03:03:50.400 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_fail": 0, 2024-03-19T03:03:50.400 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_success": 3, 2024-03-19T03:03:50.400 INFO:teuthology.orchestra.run.smithi096.stdout: "take": 0, 2024-03-19T03:03:50.400 INFO:teuthology.orchestra.run.smithi096.stdout: "take_sum": 0, 2024-03-19T03:03:50.400 INFO:teuthology.orchestra.run.smithi096.stdout: "put": 3, 2024-03-19T03:03:50.400 INFO:teuthology.orchestra.run.smithi096.stdout: "put_sum": 343, 2024-03-19T03:03:50.401 INFO:teuthology.orchestra.run.smithi096.stdout: "wait": { 2024-03-19T03:03:50.401 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:50.401 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:50.401 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:50.401 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-03-19T03:03:50.401 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:50.401 INFO:teuthology.orchestra.run.smithi096.stdout: "throttle-write_buf_throttle-0x55f34df4e500": { 2024-03-19T03:03:50.401 INFO:teuthology.orchestra.run.smithi096.stdout: "val": 0, 2024-03-19T03:03:50.401 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 3758096384, 2024-03-19T03:03:50.401 INFO:teuthology.orchestra.run.smithi096.stdout: "get_started": 0, 2024-03-19T03:03:50.401 INFO:teuthology.orchestra.run.smithi096.stdout: "get": 195, 2024-03-19T03:03:50.401 INFO:teuthology.orchestra.run.smithi096.stdout: "get_sum": 629893, 2024-03-19T03:03:50.401 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_fail": 0, 2024-03-19T03:03:50.401 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_success": 195, 2024-03-19T03:03:50.401 INFO:teuthology.orchestra.run.smithi096.stdout: "take": 0, 2024-03-19T03:03:50.401 INFO:teuthology.orchestra.run.smithi096.stdout: "take_sum": 0, 2024-03-19T03:03:50.402 INFO:teuthology.orchestra.run.smithi096.stdout: "put": 163, 2024-03-19T03:03:50.402 INFO:teuthology.orchestra.run.smithi096.stdout: "put_sum": 629893, 2024-03-19T03:03:50.402 INFO:teuthology.orchestra.run.smithi096.stdout: "wait": { 2024-03-19T03:03:50.402 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:50.402 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:50.402 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:50.402 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-03-19T03:03:50.402 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-03-19T03:03:50.402 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-03-19T03:03:51.189 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:50 smithi096 ceph-mon[35946]: pgmap v322: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:03:51.223 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-03-19T03:03:51.223 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-03-19T03:03:51.223 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=8 2024-03-19T03:03:51.224 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=0 2024-03-19T03:03:51.224 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-03-19T03:03:51.224 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=10 2024-03-19T03:03:51.224 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=4 2024-03-19T03:03:51.224 DEBUG:tasks.check_counter:Getting stats from e 2024-03-19T03:03:51.224 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-19T03:03:51.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:50 smithi155 ceph-mon[35804]: pgmap v322: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:03:51.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:50 smithi033 ceph-mon[31973]: pgmap v322: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:03:51.563 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T03:03:53.118 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:52 smithi096 ceph-mon[35946]: pgmap v323: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:03:53.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:52 smithi155 ceph-mon[35804]: pgmap v323: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:03:53.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:52 smithi033 ceph-mon[31973]: pgmap v323: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:03:53.421 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-03-19T03:03:53.421 INFO:teuthology.orchestra.run.smithi096.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T03:03:53.421 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_messages": 1600, 2024-03-19T03:03:53.421 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages": 1600, 2024-03-19T03:03:53.421 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_bytes": 1106046, 2024-03-19T03:03:53.421 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_bytes": 507986, 2024-03-19T03:03:53.421 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_created_connections": 6, 2024-03-19T03:03:53.421 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_active_connections": 2, 2024-03-19T03:03:53.421 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_total_time": 0.103540838, 2024-03-19T03:03:53.421 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_send_time": 0.035994310, 2024-03-19T03:03:53.421 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_recv_time": 0.025755381, 2024-03-19T03:03:53.422 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_fast_dispatch_time": 0.033371155, 2024-03-19T03:03:53.422 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:03:53.422 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 1599, 2024-03-19T03:03:53.422 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.034980721, 2024-03-19T03:03:53.422 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000021876 2024-03-19T03:03:53.422 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.422 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:03:53.422 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.422 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.422 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.422 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.422 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-19T03:03:53.422 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_encrypted_bytes": 448 2024-03-19T03:03:53.422 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.423 INFO:teuthology.orchestra.run.smithi096.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T03:03:53.423 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_messages": 251, 2024-03-19T03:03:53.423 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages": 240, 2024-03-19T03:03:53.423 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_bytes": 1469064, 2024-03-19T03:03:53.423 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_bytes": 137802, 2024-03-19T03:03:53.423 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_created_connections": 4, 2024-03-19T03:03:53.423 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_active_connections": 3, 2024-03-19T03:03:53.423 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_total_time": 0.033262371, 2024-03-19T03:03:53.423 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_send_time": 0.008402755, 2024-03-19T03:03:53.423 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_recv_time": 0.031679948, 2024-03-19T03:03:53.423 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_fast_dispatch_time": 0.003679052, 2024-03-19T03:03:53.423 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:03:53.423 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 239, 2024-03-19T03:03:53.424 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.009214840, 2024-03-19T03:03:53.424 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000038555 2024-03-19T03:03:53.424 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.424 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:03:53.424 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.424 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.424 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.424 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.424 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_encrypted_bytes": 1467408, 2024-03-19T03:03:53.424 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_encrypted_bytes": 136464 2024-03-19T03:03:53.424 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.424 INFO:teuthology.orchestra.run.smithi096.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T03:03:53.424 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_messages": 757, 2024-03-19T03:03:53.424 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages": 921, 2024-03-19T03:03:53.424 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_bytes": 186121, 2024-03-19T03:03:53.425 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_bytes": 685835, 2024-03-19T03:03:53.425 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_created_connections": 6, 2024-03-19T03:03:53.425 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_active_connections": 4, 2024-03-19T03:03:53.425 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_total_time": 0.056710655, 2024-03-19T03:03:53.425 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_send_time": 0.018617606, 2024-03-19T03:03:53.425 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_recv_time": 0.012305078, 2024-03-19T03:03:53.425 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_fast_dispatch_time": 0.020134816, 2024-03-19T03:03:53.425 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:03:53.425 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 918, 2024-03-19T03:03:53.425 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.027995581, 2024-03-19T03:03:53.425 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000030496 2024-03-19T03:03:53.425 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.425 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:03:53.425 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.425 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.426 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.426 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.426 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-19T03:03:53.426 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_encrypted_bytes": 445744 2024-03-19T03:03:53.426 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.426 INFO:teuthology.orchestra.run.smithi096.stdout: "cct": { 2024-03-19T03:03:53.426 INFO:teuthology.orchestra.run.smithi096.stdout: "total_workers": 1, 2024-03-19T03:03:53.426 INFO:teuthology.orchestra.run.smithi096.stdout: "unhealthy_workers": 0 2024-03-19T03:03:53.426 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.426 INFO:teuthology.orchestra.run.smithi096.stdout: "finisher-MDSRank": { 2024-03-19T03:03:53.426 INFO:teuthology.orchestra.run.smithi096.stdout: "queue_len": 0, 2024-03-19T03:03:53.426 INFO:teuthology.orchestra.run.smithi096.stdout: "complete_latency": { 2024-03-19T03:03:53.426 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 3917, 2024-03-19T03:03:53.426 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.078456553, 2024-03-19T03:03:53.427 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000020029 2024-03-19T03:03:53.427 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-03-19T03:03:53.427 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.427 INFO:teuthology.orchestra.run.smithi096.stdout: "finisher-PurgeQueue": { 2024-03-19T03:03:53.427 INFO:teuthology.orchestra.run.smithi096.stdout: "queue_len": 0, 2024-03-19T03:03:53.427 INFO:teuthology.orchestra.run.smithi096.stdout: "complete_latency": { 2024-03-19T03:03:53.427 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.427 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.427 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.427 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-03-19T03:03:53.427 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.427 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-03-19T03:03:53.427 INFO:teuthology.orchestra.run.smithi096.stdout: "request": 0, 2024-03-19T03:03:53.427 INFO:teuthology.orchestra.run.smithi096.stdout: "reply": 0, 2024-03-19T03:03:53.428 INFO:teuthology.orchestra.run.smithi096.stdout: "reply_latency": { 2024-03-19T03:03:53.428 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.428 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.428 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.428 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.428 INFO:teuthology.orchestra.run.smithi096.stdout: "slow_reply": 0, 2024-03-19T03:03:53.428 INFO:teuthology.orchestra.run.smithi096.stdout: "forward": 0, 2024-03-19T03:03:53.428 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_fetch_complete": 0, 2024-03-19T03:03:53.428 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_fetch_keys": 0, 2024-03-19T03:03:53.428 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_commit": 0, 2024-03-19T03:03:53.428 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_split": 0, 2024-03-19T03:03:53.428 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_merge": 0, 2024-03-19T03:03:53.428 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes": 27, 2024-03-19T03:03:53.428 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes_top": 0, 2024-03-19T03:03:53.428 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes_bottom": 27, 2024-03-19T03:03:53.429 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes_pin_tail": 0, 2024-03-19T03:03:53.429 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes_pinned": 27, 2024-03-19T03:03:53.429 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes_expired": 0, 2024-03-19T03:03:53.429 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes_with_caps": 0, 2024-03-19T03:03:53.429 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": 0, 2024-03-19T03:03:53.429 INFO:teuthology.orchestra.run.smithi096.stdout: "subtrees": 2, 2024-03-19T03:03:53.429 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse": 0, 2024-03-19T03:03:53.429 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse_hit": 0, 2024-03-19T03:03:53.429 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse_forward": 0, 2024-03-19T03:03:53.429 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse_discover": 0, 2024-03-19T03:03:53.429 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse_dir_fetch": 0, 2024-03-19T03:03:53.429 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse_remote_ino": 0, 2024-03-19T03:03:53.429 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse_lock": 0, 2024-03-19T03:03:53.429 INFO:teuthology.orchestra.run.smithi096.stdout: "load_cent": 0, 2024-03-19T03:03:53.430 INFO:teuthology.orchestra.run.smithi096.stdout: "q": 0, 2024-03-19T03:03:53.430 INFO:teuthology.orchestra.run.smithi096.stdout: "exported": 0, 2024-03-19T03:03:53.430 INFO:teuthology.orchestra.run.smithi096.stdout: "exported_inodes": 0, 2024-03-19T03:03:53.430 INFO:teuthology.orchestra.run.smithi096.stdout: "imported": 0, 2024-03-19T03:03:53.430 INFO:teuthology.orchestra.run.smithi096.stdout: "imported_inodes": 0, 2024-03-19T03:03:53.430 INFO:teuthology.orchestra.run.smithi096.stdout: "openino_dir_fetch": 0, 2024-03-19T03:03:53.430 INFO:teuthology.orchestra.run.smithi096.stdout: "openino_backtrace_fetch": 0, 2024-03-19T03:03:53.430 INFO:teuthology.orchestra.run.smithi096.stdout: "openino_peer_discover": 0, 2024-03-19T03:03:53.430 INFO:teuthology.orchestra.run.smithi096.stdout: "root_rfiles": 4, 2024-03-19T03:03:53.430 INFO:teuthology.orchestra.run.smithi096.stdout: "root_rbytes": 234, 2024-03-19T03:03:53.430 INFO:teuthology.orchestra.run.smithi096.stdout: "root_rsnaps": 6, 2024-03-19T03:03:53.430 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_backtrace_fetch": 0, 2024-03-19T03:03:53.430 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_set_tag": 0, 2024-03-19T03:03:53.430 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T03:03:53.431 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_inotable_repaired": 0, 2024-03-19T03:03:53.431 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_dir_inodes": 0, 2024-03-19T03:03:53.431 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T03:03:53.431 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T03:03:53.431 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_file_inodes": 0, 2024-03-19T03:03:53.431 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_inode_file_caps": 0, 2024-03-19T03:03:53.431 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_cap_op_revoke": 0, 2024-03-19T03:03:53.431 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_cap_op_grant": 0, 2024-03-19T03:03:53.431 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T03:03:53.431 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T03:03:53.431 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T03:03:53.431 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_client_caps": 0, 2024-03-19T03:03:53.431 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_client_caps_dirty": 0, 2024-03-19T03:03:53.431 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_client_cap_release": 0, 2024-03-19T03:03:53.432 INFO:teuthology.orchestra.run.smithi096.stdout: "process_request_cap_release": 0 2024-03-19T03:03:53.432 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.432 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_cache": { 2024-03-19T03:03:53.432 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_update": 0, 2024-03-19T03:03:53.432 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_update_receipt": 0, 2024-03-19T03:03:53.432 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_try_discover": 0, 2024-03-19T03:03:53.432 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_send_discover": 0, 2024-03-19T03:03:53.432 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_handle_discover": 0, 2024-03-19T03:03:53.432 INFO:teuthology.orchestra.run.smithi096.stdout: "num_strays": 0, 2024-03-19T03:03:53.432 INFO:teuthology.orchestra.run.smithi096.stdout: "num_strays_delayed": 0, 2024-03-19T03:03:53.432 INFO:teuthology.orchestra.run.smithi096.stdout: "num_strays_enqueuing": 0, 2024-03-19T03:03:53.432 INFO:teuthology.orchestra.run.smithi096.stdout: "strays_created": 0, 2024-03-19T03:03:53.432 INFO:teuthology.orchestra.run.smithi096.stdout: "strays_enqueued": 0, 2024-03-19T03:03:53.432 INFO:teuthology.orchestra.run.smithi096.stdout: "strays_reintegrated": 0, 2024-03-19T03:03:53.432 INFO:teuthology.orchestra.run.smithi096.stdout: "strays_migrated": 0, 2024-03-19T03:03:53.433 INFO:teuthology.orchestra.run.smithi096.stdout: "num_recovering_processing": 0, 2024-03-19T03:03:53.433 INFO:teuthology.orchestra.run.smithi096.stdout: "num_recovering_enqueued": 0, 2024-03-19T03:03:53.433 INFO:teuthology.orchestra.run.smithi096.stdout: "num_recovering_prioritized": 0, 2024-03-19T03:03:53.433 INFO:teuthology.orchestra.run.smithi096.stdout: "recovery_started": 0, 2024-03-19T03:03:53.433 INFO:teuthology.orchestra.run.smithi096.stdout: "recovery_completed": 0, 2024-03-19T03:03:53.433 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_quiesce_path": 0, 2024-03-19T03:03:53.433 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_quiesce_inode": 0, 2024-03-19T03:03:53.433 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T03:03:53.433 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_exportdir": 0, 2024-03-19T03:03:53.433 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_flush": 0, 2024-03-19T03:03:53.433 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_fragmentdir": 0, 2024-03-19T03:03:53.433 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_fragstats": 0, 2024-03-19T03:03:53.433 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_inodestats": 0 2024-03-19T03:03:53.433 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.433 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_log": { 2024-03-19T03:03:53.434 INFO:teuthology.orchestra.run.smithi096.stdout: "evlrg": 0, 2024-03-19T03:03:53.434 INFO:teuthology.orchestra.run.smithi096.stdout: "evadd": 0, 2024-03-19T03:03:53.434 INFO:teuthology.orchestra.run.smithi096.stdout: "evex": 0, 2024-03-19T03:03:53.434 INFO:teuthology.orchestra.run.smithi096.stdout: "evtrm": 0, 2024-03-19T03:03:53.434 INFO:teuthology.orchestra.run.smithi096.stdout: "ev": 195, 2024-03-19T03:03:53.434 INFO:teuthology.orchestra.run.smithi096.stdout: "evexg": 0, 2024-03-19T03:03:53.434 INFO:teuthology.orchestra.run.smithi096.stdout: "evexd": 0, 2024-03-19T03:03:53.434 INFO:teuthology.orchestra.run.smithi096.stdout: "segadd": 0, 2024-03-19T03:03:53.434 INFO:teuthology.orchestra.run.smithi096.stdout: "segex": 0, 2024-03-19T03:03:53.434 INFO:teuthology.orchestra.run.smithi096.stdout: "segtrm": 0, 2024-03-19T03:03:53.434 INFO:teuthology.orchestra.run.smithi096.stdout: "seg": 2, 2024-03-19T03:03:53.434 INFO:teuthology.orchestra.run.smithi096.stdout: "segmjr": 2, 2024-03-19T03:03:53.434 INFO:teuthology.orchestra.run.smithi096.stdout: "segexg": 0, 2024-03-19T03:03:53.435 INFO:teuthology.orchestra.run.smithi096.stdout: "segexd": 0, 2024-03-19T03:03:53.435 INFO:teuthology.orchestra.run.smithi096.stdout: "expos": 4194304, 2024-03-19T03:03:53.435 INFO:teuthology.orchestra.run.smithi096.stdout: "wrpos": 4824197, 2024-03-19T03:03:53.435 INFO:teuthology.orchestra.run.smithi096.stdout: "rdpos": 4824061, 2024-03-19T03:03:53.435 INFO:teuthology.orchestra.run.smithi096.stdout: "jlat": { 2024-03-19T03:03:53.435 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.435 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.435 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.435 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.435 INFO:teuthology.orchestra.run.smithi096.stdout: "replayed": 195 2024-03-19T03:03:53.435 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.435 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_mem": { 2024-03-19T03:03:53.435 INFO:teuthology.orchestra.run.smithi096.stdout: "ino": 18, 2024-03-19T03:03:53.435 INFO:teuthology.orchestra.run.smithi096.stdout: "ino+": 21, 2024-03-19T03:03:53.436 INFO:teuthology.orchestra.run.smithi096.stdout: "ino-": 3, 2024-03-19T03:03:53.436 INFO:teuthology.orchestra.run.smithi096.stdout: "dir": 11, 2024-03-19T03:03:53.436 INFO:teuthology.orchestra.run.smithi096.stdout: "dir+": 11, 2024-03-19T03:03:53.436 INFO:teuthology.orchestra.run.smithi096.stdout: "dir-": 0, 2024-03-19T03:03:53.436 INFO:teuthology.orchestra.run.smithi096.stdout: "dn": 27, 2024-03-19T03:03:53.436 INFO:teuthology.orchestra.run.smithi096.stdout: "dn+": 27, 2024-03-19T03:03:53.436 INFO:teuthology.orchestra.run.smithi096.stdout: "dn-": 0, 2024-03-19T03:03:53.436 INFO:teuthology.orchestra.run.smithi096.stdout: "cap": 0, 2024-03-19T03:03:53.436 INFO:teuthology.orchestra.run.smithi096.stdout: "cap+": 0, 2024-03-19T03:03:53.436 INFO:teuthology.orchestra.run.smithi096.stdout: "cap-": 0, 2024-03-19T03:03:53.436 INFO:teuthology.orchestra.run.smithi096.stdout: "rss": 50728, 2024-03-19T03:03:53.436 INFO:teuthology.orchestra.run.smithi096.stdout: "heap": 215420 2024-03-19T03:03:53.436 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.436 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_server": { 2024-03-19T03:03:53.436 INFO:teuthology.orchestra.run.smithi096.stdout: "dispatch_client_request": 0, 2024-03-19T03:03:53.437 INFO:teuthology.orchestra.run.smithi096.stdout: "dispatch_server_request": 0, 2024-03-19T03:03:53.437 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_client_request": 0, 2024-03-19T03:03:53.437 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_client_session": 0, 2024-03-19T03:03:53.437 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_peer_request": 0, 2024-03-19T03:03:53.437 INFO:teuthology.orchestra.run.smithi096.stdout: "req_create_latency": { 2024-03-19T03:03:53.437 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.437 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.437 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.437 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.437 INFO:teuthology.orchestra.run.smithi096.stdout: "req_getattr_latency": { 2024-03-19T03:03:53.437 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.437 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.437 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.437 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.438 INFO:teuthology.orchestra.run.smithi096.stdout: "req_getfilelock_latency": { 2024-03-19T03:03:53.438 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.438 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.438 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.438 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.438 INFO:teuthology.orchestra.run.smithi096.stdout: "req_link_latency": { 2024-03-19T03:03:53.438 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.438 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.438 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.438 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.438 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lookup_latency": { 2024-03-19T03:03:53.438 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.438 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.438 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.439 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.439 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lookuphash_latency": { 2024-03-19T03:03:53.439 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.439 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.439 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.439 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.439 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lookupino_latency": { 2024-03-19T03:03:53.439 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.439 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.441 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.441 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.441 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lookupname_latency": { 2024-03-19T03:03:53.441 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.441 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.441 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.442 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.442 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lookupparent_latency": { 2024-03-19T03:03:53.442 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.442 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.442 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.442 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.442 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lookupsnap_latency": { 2024-03-19T03:03:53.442 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.442 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.442 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.442 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.442 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lssnap_latency": { 2024-03-19T03:03:53.442 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.443 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.443 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.443 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.443 INFO:teuthology.orchestra.run.smithi096.stdout: "req_mkdir_latency": { 2024-03-19T03:03:53.443 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.444 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.444 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.444 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.444 INFO:teuthology.orchestra.run.smithi096.stdout: "req_mknod_latency": { 2024-03-19T03:03:53.444 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.444 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.444 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.444 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.444 INFO:teuthology.orchestra.run.smithi096.stdout: "req_mksnap_latency": { 2024-03-19T03:03:53.444 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.444 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.444 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.444 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.444 INFO:teuthology.orchestra.run.smithi096.stdout: "req_open_latency": { 2024-03-19T03:03:53.444 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.445 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.445 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.445 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.445 INFO:teuthology.orchestra.run.smithi096.stdout: "req_readdir_latency": { 2024-03-19T03:03:53.445 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.445 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.445 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.445 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.445 INFO:teuthology.orchestra.run.smithi096.stdout: "req_rename_latency": { 2024-03-19T03:03:53.445 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.445 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.445 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.445 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.445 INFO:teuthology.orchestra.run.smithi096.stdout: "req_renamesnap_latency": { 2024-03-19T03:03:53.446 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.446 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.446 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.446 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.446 INFO:teuthology.orchestra.run.smithi096.stdout: "req_snapdiff_latency": { 2024-03-19T03:03:53.446 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.446 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.446 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.446 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.446 INFO:teuthology.orchestra.run.smithi096.stdout: "req_rmdir_latency": { 2024-03-19T03:03:53.446 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.446 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.446 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.446 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.447 INFO:teuthology.orchestra.run.smithi096.stdout: "req_rmsnap_latency": { 2024-03-19T03:03:53.447 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.447 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.447 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.447 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.447 INFO:teuthology.orchestra.run.smithi096.stdout: "req_rmxattr_latency": { 2024-03-19T03:03:53.447 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.447 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.447 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.447 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.447 INFO:teuthology.orchestra.run.smithi096.stdout: "req_setattr_latency": { 2024-03-19T03:03:53.447 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.447 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.447 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.447 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.447 INFO:teuthology.orchestra.run.smithi096.stdout: "req_setdirlayout_latency": { 2024-03-19T03:03:53.448 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.448 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.448 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.448 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.448 INFO:teuthology.orchestra.run.smithi096.stdout: "req_setfilelock_latency": { 2024-03-19T03:03:53.448 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.448 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.448 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.448 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.448 INFO:teuthology.orchestra.run.smithi096.stdout: "req_setlayout_latency": { 2024-03-19T03:03:53.448 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.448 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.448 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.448 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.448 INFO:teuthology.orchestra.run.smithi096.stdout: "req_setxattr_latency": { 2024-03-19T03:03:53.449 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.449 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.449 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.449 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.449 INFO:teuthology.orchestra.run.smithi096.stdout: "req_symlink_latency": { 2024-03-19T03:03:53.449 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.449 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.449 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.449 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.449 INFO:teuthology.orchestra.run.smithi096.stdout: "req_unlink_latency": { 2024-03-19T03:03:53.449 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.449 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.449 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.449 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.449 INFO:teuthology.orchestra.run.smithi096.stdout: "cap_revoke_eviction": 0, 2024-03-19T03:03:53.449 INFO:teuthology.orchestra.run.smithi096.stdout: "cap_acquisition_throttle": 0, 2024-03-19T03:03:53.449 INFO:teuthology.orchestra.run.smithi096.stdout: "req_getvxattr_latency": { 2024-03-19T03:03:53.450 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.450 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.450 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.450 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-03-19T03:03:53.450 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.450 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_sessions": { 2024-03-19T03:03:53.450 INFO:teuthology.orchestra.run.smithi096.stdout: "session_count": 2, 2024-03-19T03:03:53.450 INFO:teuthology.orchestra.run.smithi096.stdout: "session_add": 6, 2024-03-19T03:03:53.450 INFO:teuthology.orchestra.run.smithi096.stdout: "session_remove": 4, 2024-03-19T03:03:53.450 INFO:teuthology.orchestra.run.smithi096.stdout: "sessions_open": 4, 2024-03-19T03:03:53.450 INFO:teuthology.orchestra.run.smithi096.stdout: "sessions_stale": 0, 2024-03-19T03:03:53.450 INFO:teuthology.orchestra.run.smithi096.stdout: "total_load": 0, 2024-03-19T03:03:53.450 INFO:teuthology.orchestra.run.smithi096.stdout: "average_load": 0, 2024-03-19T03:03:53.450 INFO:teuthology.orchestra.run.smithi096.stdout: "avg_session_uptime": 4956, 2024-03-19T03:03:53.451 INFO:teuthology.orchestra.run.smithi096.stdout: "mdthresh_evicted": 0 2024-03-19T03:03:53.451 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.451 INFO:teuthology.orchestra.run.smithi096.stdout: "mempool": { 2024-03-19T03:03:53.451 INFO:teuthology.orchestra.run.smithi096.stdout: "bloom_filter_bytes": 0, 2024-03-19T03:03:53.451 INFO:teuthology.orchestra.run.smithi096.stdout: "bloom_filter_items": 0, 2024-03-19T03:03:53.451 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T03:03:53.451 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_alloc_items": 0, 2024-03-19T03:03:53.451 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T03:03:53.451 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_data_items": 0, 2024-03-19T03:03:53.451 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T03:03:53.451 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T03:03:53.451 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T03:03:53.451 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T03:03:53.451 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T03:03:53.451 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_other_items": 0, 2024-03-19T03:03:53.452 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T03:03:53.452 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T03:03:53.452 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_extent_bytes": 0, 2024-03-19T03:03:53.452 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_extent_items": 0, 2024-03-19T03:03:53.452 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_blob_bytes": 0, 2024-03-19T03:03:53.452 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_blob_items": 0, 2024-03-19T03:03:53.452 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T03:03:53.452 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T03:03:53.452 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T03:03:53.452 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T03:03:53.452 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T03:03:53.452 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_fsck_items": 0, 2024-03-19T03:03:53.452 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_txc_bytes": 0, 2024-03-19T03:03:53.452 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_txc_items": 0, 2024-03-19T03:03:53.452 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T03:03:53.453 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T03:03:53.453 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_writing_bytes": 0, 2024-03-19T03:03:53.453 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_writing_items": 0, 2024-03-19T03:03:53.453 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_bytes": 0, 2024-03-19T03:03:53.453 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_items": 0, 2024-03-19T03:03:53.453 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T03:03:53.453 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_file_reader_items": 0, 2024-03-19T03:03:53.453 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T03:03:53.453 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_file_writer_items": 0, 2024-03-19T03:03:53.453 INFO:teuthology.orchestra.run.smithi096.stdout: "buffer_anon_bytes": 190201, 2024-03-19T03:03:53.453 INFO:teuthology.orchestra.run.smithi096.stdout: "buffer_anon_items": 53, 2024-03-19T03:03:53.453 INFO:teuthology.orchestra.run.smithi096.stdout: "buffer_meta_bytes": 0, 2024-03-19T03:03:53.453 INFO:teuthology.orchestra.run.smithi096.stdout: "buffer_meta_items": 0, 2024-03-19T03:03:53.453 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_bytes": 0, 2024-03-19T03:03:53.453 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_items": 0, 2024-03-19T03:03:53.454 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_mapbl_bytes": 0, 2024-03-19T03:03:53.454 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_mapbl_items": 0, 2024-03-19T03:03:53.454 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_pglog_bytes": 0, 2024-03-19T03:03:53.454 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_pglog_items": 0, 2024-03-19T03:03:53.454 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_bytes": 9512, 2024-03-19T03:03:53.454 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_items": 182, 2024-03-19T03:03:53.454 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T03:03:53.454 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_mapping_items": 0, 2024-03-19T03:03:53.454 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap_bytes": 0, 2024-03-19T03:03:53.454 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap_items": 0, 2024-03-19T03:03:53.454 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_co_bytes": 90932, 2024-03-19T03:03:53.454 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_co_items": 489, 2024-03-19T03:03:53.454 INFO:teuthology.orchestra.run.smithi096.stdout: "unittest_1_bytes": 0, 2024-03-19T03:03:53.455 INFO:teuthology.orchestra.run.smithi096.stdout: "unittest_1_items": 0, 2024-03-19T03:03:53.455 INFO:teuthology.orchestra.run.smithi096.stdout: "unittest_2_bytes": 0, 2024-03-19T03:03:53.455 INFO:teuthology.orchestra.run.smithi096.stdout: "unittest_2_items": 0 2024-03-19T03:03:53.455 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.455 INFO:teuthology.orchestra.run.smithi096.stdout: "objecter": { 2024-03-19T03:03:53.455 INFO:teuthology.orchestra.run.smithi096.stdout: "op_active": 0, 2024-03-19T03:03:53.455 INFO:teuthology.orchestra.run.smithi096.stdout: "op_laggy": 0, 2024-03-19T03:03:53.455 INFO:teuthology.orchestra.run.smithi096.stdout: "op_send": 2355, 2024-03-19T03:03:53.455 INFO:teuthology.orchestra.run.smithi096.stdout: "op_send_bytes": 26, 2024-03-19T03:03:53.455 INFO:teuthology.orchestra.run.smithi096.stdout: "op_resend": 4, 2024-03-19T03:03:53.455 INFO:teuthology.orchestra.run.smithi096.stdout: "op_reply": 2355, 2024-03-19T03:03:53.455 INFO:teuthology.orchestra.run.smithi096.stdout: "op_latency": { 2024-03-19T03:03:53.455 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 2355, 2024-03-19T03:03:53.455 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.793980551, 2024-03-19T03:03:53.455 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000337146 2024-03-19T03:03:53.456 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.456 INFO:teuthology.orchestra.run.smithi096.stdout: "op_inflight": 0, 2024-03-19T03:03:53.456 INFO:teuthology.orchestra.run.smithi096.stdout: "oplen_avg": { 2024-03-19T03:03:53.456 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 2355, 2024-03-19T03:03:53.456 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 2356 2024-03-19T03:03:53.456 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.456 INFO:teuthology.orchestra.run.smithi096.stdout: "op": 2355, 2024-03-19T03:03:53.456 INFO:teuthology.orchestra.run.smithi096.stdout: "op_r": 2355, 2024-03-19T03:03:53.456 INFO:teuthology.orchestra.run.smithi096.stdout: "op_w": 0, 2024-03-19T03:03:53.456 INFO:teuthology.orchestra.run.smithi096.stdout: "op_rmw": 0, 2024-03-19T03:03:53.456 INFO:teuthology.orchestra.run.smithi096.stdout: "op_pg": 0, 2024-03-19T03:03:53.456 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_stat": 1508, 2024-03-19T03:03:53.456 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_create": 0, 2024-03-19T03:03:53.456 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_read": 844, 2024-03-19T03:03:53.456 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_write": 0, 2024-03-19T03:03:53.457 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_writefull": 0, 2024-03-19T03:03:53.457 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_writesame": 0, 2024-03-19T03:03:53.457 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_append": 0, 2024-03-19T03:03:53.457 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_zero": 0, 2024-03-19T03:03:53.457 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_truncate": 0, 2024-03-19T03:03:53.457 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_delete": 0, 2024-03-19T03:03:53.457 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_mapext": 0, 2024-03-19T03:03:53.457 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_sparse_read": 0, 2024-03-19T03:03:53.457 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_clonerange": 0, 2024-03-19T03:03:53.457 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_getxattr": 2, 2024-03-19T03:03:53.457 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_setxattr": 0, 2024-03-19T03:03:53.457 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_cmpxattr": 0, 2024-03-19T03:03:53.457 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_rmxattr": 0, 2024-03-19T03:03:53.457 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_resetxattrs": 0, 2024-03-19T03:03:53.458 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_call": 0, 2024-03-19T03:03:53.458 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_watch": 0, 2024-03-19T03:03:53.458 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_notify": 0, 2024-03-19T03:03:53.458 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T03:03:53.458 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_pgls": 0, 2024-03-19T03:03:53.458 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_pgls_filter": 0, 2024-03-19T03:03:53.458 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_other": 0, 2024-03-19T03:03:53.458 INFO:teuthology.orchestra.run.smithi096.stdout: "linger_active": 0, 2024-03-19T03:03:53.458 INFO:teuthology.orchestra.run.smithi096.stdout: "linger_send": 0, 2024-03-19T03:03:53.458 INFO:teuthology.orchestra.run.smithi096.stdout: "linger_resend": 0, 2024-03-19T03:03:53.458 INFO:teuthology.orchestra.run.smithi096.stdout: "linger_ping": 0, 2024-03-19T03:03:53.458 INFO:teuthology.orchestra.run.smithi096.stdout: "poolop_active": 0, 2024-03-19T03:03:53.458 INFO:teuthology.orchestra.run.smithi096.stdout: "poolop_send": 0, 2024-03-19T03:03:53.458 INFO:teuthology.orchestra.run.smithi096.stdout: "poolop_resend": 0, 2024-03-19T03:03:53.458 INFO:teuthology.orchestra.run.smithi096.stdout: "poolstat_active": 0, 2024-03-19T03:03:53.459 INFO:teuthology.orchestra.run.smithi096.stdout: "poolstat_send": 0, 2024-03-19T03:03:53.459 INFO:teuthology.orchestra.run.smithi096.stdout: "poolstat_resend": 0, 2024-03-19T03:03:53.459 INFO:teuthology.orchestra.run.smithi096.stdout: "statfs_active": 0, 2024-03-19T03:03:53.459 INFO:teuthology.orchestra.run.smithi096.stdout: "statfs_send": 0, 2024-03-19T03:03:53.459 INFO:teuthology.orchestra.run.smithi096.stdout: "statfs_resend": 0, 2024-03-19T03:03:53.459 INFO:teuthology.orchestra.run.smithi096.stdout: "command_active": 0, 2024-03-19T03:03:53.459 INFO:teuthology.orchestra.run.smithi096.stdout: "command_send": 0, 2024-03-19T03:03:53.459 INFO:teuthology.orchestra.run.smithi096.stdout: "command_resend": 0, 2024-03-19T03:03:53.459 INFO:teuthology.orchestra.run.smithi096.stdout: "map_epoch": 106, 2024-03-19T03:03:53.459 INFO:teuthology.orchestra.run.smithi096.stdout: "map_full": 0, 2024-03-19T03:03:53.459 INFO:teuthology.orchestra.run.smithi096.stdout: "map_inc": 13, 2024-03-19T03:03:53.459 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_sessions": 5, 2024-03-19T03:03:53.459 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_session_open": 5, 2024-03-19T03:03:53.459 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_session_close": 0, 2024-03-19T03:03:53.459 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_laggy": 0, 2024-03-19T03:03:53.459 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_wr": 0, 2024-03-19T03:03:53.460 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_rd": 2, 2024-03-19T03:03:53.460 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_del": 0 2024-03-19T03:03:53.460 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.460 INFO:teuthology.orchestra.run.smithi096.stdout: "oft": { 2024-03-19T03:03:53.460 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_total_objs": 0, 2024-03-19T03:03:53.460 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_total_kv_pairs": 0, 2024-03-19T03:03:53.460 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_total_updates": 0, 2024-03-19T03:03:53.460 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_total_removes": 0 2024-03-19T03:03:53.460 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.460 INFO:teuthology.orchestra.run.smithi096.stdout: "osd-slow-ops": { 2024-03-19T03:03:53.460 INFO:teuthology.orchestra.run.smithi096.stdout: "slow_ops_count": 0 2024-03-19T03:03:53.460 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.460 INFO:teuthology.orchestra.run.smithi096.stdout: "purge_queue": { 2024-03-19T03:03:53.460 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_executing_ops": 0, 2024-03-19T03:03:53.460 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_executing_ops_high_water": 0, 2024-03-19T03:03:53.461 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_executing": 0, 2024-03-19T03:03:53.461 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_executing_high_water": 0, 2024-03-19T03:03:53.461 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_executed_ops": 0, 2024-03-19T03:03:53.461 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_executed": 0, 2024-03-19T03:03:53.461 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_item_in_journal": 0 2024-03-19T03:03:53.461 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.461 INFO:teuthology.orchestra.run.smithi096.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T03:03:53.461 INFO:teuthology.orchestra.run.smithi096.stdout: "val": 0, 2024-03-19T03:03:53.461 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 104857600, 2024-03-19T03:03:53.461 INFO:teuthology.orchestra.run.smithi096.stdout: "get_started": 0, 2024-03-19T03:03:53.461 INFO:teuthology.orchestra.run.smithi096.stdout: "get": 2604, 2024-03-19T03:03:53.461 INFO:teuthology.orchestra.run.smithi096.stdout: "get_sum": 2511236, 2024-03-19T03:03:53.461 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_fail": 0, 2024-03-19T03:03:53.461 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_success": 2604, 2024-03-19T03:03:53.461 INFO:teuthology.orchestra.run.smithi096.stdout: "take": 0, 2024-03-19T03:03:53.462 INFO:teuthology.orchestra.run.smithi096.stdout: "take_sum": 0, 2024-03-19T03:03:53.462 INFO:teuthology.orchestra.run.smithi096.stdout: "put": 2604, 2024-03-19T03:03:53.462 INFO:teuthology.orchestra.run.smithi096.stdout: "put_sum": 2511236, 2024-03-19T03:03:53.462 INFO:teuthology.orchestra.run.smithi096.stdout: "wait": { 2024-03-19T03:03:53.462 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.462 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.462 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.462 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-03-19T03:03:53.462 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.462 INFO:teuthology.orchestra.run.smithi096.stdout: "throttle-objecter_bytes": { 2024-03-19T03:03:53.462 INFO:teuthology.orchestra.run.smithi096.stdout: "val": 0, 2024-03-19T03:03:53.462 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 104857600, 2024-03-19T03:03:53.462 INFO:teuthology.orchestra.run.smithi096.stdout: "get_started": 0, 2024-03-19T03:03:53.462 INFO:teuthology.orchestra.run.smithi096.stdout: "get": 0, 2024-03-19T03:03:53.463 INFO:teuthology.orchestra.run.smithi096.stdout: "get_sum": 0, 2024-03-19T03:03:53.463 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_fail": 0, 2024-03-19T03:03:53.463 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_success": 0, 2024-03-19T03:03:53.463 INFO:teuthology.orchestra.run.smithi096.stdout: "take": 2355, 2024-03-19T03:03:53.463 INFO:teuthology.orchestra.run.smithi096.stdout: "take_sum": 629903, 2024-03-19T03:03:53.463 INFO:teuthology.orchestra.run.smithi096.stdout: "put": 2355, 2024-03-19T03:03:53.463 INFO:teuthology.orchestra.run.smithi096.stdout: "put_sum": 629903, 2024-03-19T03:03:53.463 INFO:teuthology.orchestra.run.smithi096.stdout: "wait": { 2024-03-19T03:03:53.463 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.463 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.463 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.463 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-03-19T03:03:53.463 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.463 INFO:teuthology.orchestra.run.smithi096.stdout: "throttle-objecter_ops": { 2024-03-19T03:03:53.463 INFO:teuthology.orchestra.run.smithi096.stdout: "val": 0, 2024-03-19T03:03:53.464 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1024, 2024-03-19T03:03:53.464 INFO:teuthology.orchestra.run.smithi096.stdout: "get_started": 0, 2024-03-19T03:03:53.464 INFO:teuthology.orchestra.run.smithi096.stdout: "get": 0, 2024-03-19T03:03:53.464 INFO:teuthology.orchestra.run.smithi096.stdout: "get_sum": 0, 2024-03-19T03:03:53.464 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_fail": 0, 2024-03-19T03:03:53.464 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_success": 0, 2024-03-19T03:03:53.464 INFO:teuthology.orchestra.run.smithi096.stdout: "take": 2355, 2024-03-19T03:03:53.464 INFO:teuthology.orchestra.run.smithi096.stdout: "take_sum": 2355, 2024-03-19T03:03:53.464 INFO:teuthology.orchestra.run.smithi096.stdout: "put": 2355, 2024-03-19T03:03:53.464 INFO:teuthology.orchestra.run.smithi096.stdout: "put_sum": 2355, 2024-03-19T03:03:53.464 INFO:teuthology.orchestra.run.smithi096.stdout: "wait": { 2024-03-19T03:03:53.464 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.464 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.464 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.465 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-03-19T03:03:53.465 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.465 INFO:teuthology.orchestra.run.smithi096.stdout: "throttle-write_buf_throttle": { 2024-03-19T03:03:53.465 INFO:teuthology.orchestra.run.smithi096.stdout: "val": 0, 2024-03-19T03:03:53.465 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 3758096384, 2024-03-19T03:03:53.465 INFO:teuthology.orchestra.run.smithi096.stdout: "get_started": 0, 2024-03-19T03:03:53.465 INFO:teuthology.orchestra.run.smithi096.stdout: "get": 0, 2024-03-19T03:03:53.465 INFO:teuthology.orchestra.run.smithi096.stdout: "get_sum": 0, 2024-03-19T03:03:53.465 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_fail": 0, 2024-03-19T03:03:53.465 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_success": 0, 2024-03-19T03:03:53.465 INFO:teuthology.orchestra.run.smithi096.stdout: "take": 0, 2024-03-19T03:03:53.465 INFO:teuthology.orchestra.run.smithi096.stdout: "take_sum": 0, 2024-03-19T03:03:53.465 INFO:teuthology.orchestra.run.smithi096.stdout: "put": 0, 2024-03-19T03:03:53.465 INFO:teuthology.orchestra.run.smithi096.stdout: "put_sum": 0, 2024-03-19T03:03:53.465 INFO:teuthology.orchestra.run.smithi096.stdout: "wait": { 2024-03-19T03:03:53.466 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.466 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.466 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.466 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-03-19T03:03:53.466 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:53.466 INFO:teuthology.orchestra.run.smithi096.stdout: "throttle-write_buf_throttle-0x55c37a3f46e0": { 2024-03-19T03:03:53.466 INFO:teuthology.orchestra.run.smithi096.stdout: "val": 0, 2024-03-19T03:03:53.466 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 3758096384, 2024-03-19T03:03:53.466 INFO:teuthology.orchestra.run.smithi096.stdout: "get_started": 0, 2024-03-19T03:03:53.466 INFO:teuthology.orchestra.run.smithi096.stdout: "get": 0, 2024-03-19T03:03:53.466 INFO:teuthology.orchestra.run.smithi096.stdout: "get_sum": 0, 2024-03-19T03:03:53.466 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_fail": 0, 2024-03-19T03:03:53.466 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_success": 0, 2024-03-19T03:03:53.466 INFO:teuthology.orchestra.run.smithi096.stdout: "take": 0, 2024-03-19T03:03:53.467 INFO:teuthology.orchestra.run.smithi096.stdout: "take_sum": 0, 2024-03-19T03:03:53.467 INFO:teuthology.orchestra.run.smithi096.stdout: "put": 0, 2024-03-19T03:03:53.467 INFO:teuthology.orchestra.run.smithi096.stdout: "put_sum": 0, 2024-03-19T03:03:53.467 INFO:teuthology.orchestra.run.smithi096.stdout: "wait": { 2024-03-19T03:03:53.467 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:53.467 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:53.467 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:53.467 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-03-19T03:03:53.467 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-03-19T03:03:53.467 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-03-19T03:03:54.318 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-19T03:03:54.319 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-19T03:03:54.319 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-03-19T03:03:54.319 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-03-19T03:03:54.319 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=6 2024-03-19T03:03:54.319 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-03-19T03:03:54.319 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-03-19T03:03:54.319 DEBUG:tasks.check_counter:Getting stats from h 2024-03-19T03:03:54.319 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-19T03:03:54.568 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T03:03:54.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:54 smithi096 ceph-mon[35946]: pgmap v324: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:03:55.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:54 smithi155 ceph-mon[35804]: pgmap v324: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:03:55.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:54 smithi033 ceph-mon[31973]: pgmap v324: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:03:56.456 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-03-19T03:03:56.457 INFO:teuthology.orchestra.run.smithi096.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T03:03:56.457 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_messages": 92, 2024-03-19T03:03:56.457 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages": 124, 2024-03-19T03:03:56.457 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_bytes": 28048, 2024-03-19T03:03:56.457 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_bytes": 87580, 2024-03-19T03:03:56.457 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_created_connections": 14, 2024-03-19T03:03:56.457 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_active_connections": 4, 2024-03-19T03:03:56.457 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_total_time": 0.010903179, 2024-03-19T03:03:56.457 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_send_time": 0.002214987, 2024-03-19T03:03:56.457 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_recv_time": 0.005379330, 2024-03-19T03:03:56.458 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_fast_dispatch_time": 0.000652379, 2024-03-19T03:03:56.458 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:03:56.458 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 120, 2024-03-19T03:03:56.458 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.017357570, 2024-03-19T03:03:56.458 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000144646 2024-03-19T03:03:56.458 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.458 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:03:56.458 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 133, 2024-03-19T03:03:56.458 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000024085, 2024-03-19T03:03:56.458 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000181 2024-03-19T03:03:56.458 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.458 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-19T03:03:56.458 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_encrypted_bytes": 0 2024-03-19T03:03:56.459 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.459 INFO:teuthology.orchestra.run.smithi096.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T03:03:56.459 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_messages": 757, 2024-03-19T03:03:56.459 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages": 972, 2024-03-19T03:03:56.459 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_bytes": 1628114, 2024-03-19T03:03:56.459 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_bytes": 689112, 2024-03-19T03:03:56.459 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_created_connections": 7, 2024-03-19T03:03:56.459 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_active_connections": 5, 2024-03-19T03:03:56.459 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_total_time": 0.082429354, 2024-03-19T03:03:56.459 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_send_time": 0.032436517, 2024-03-19T03:03:56.459 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_recv_time": 0.058505666, 2024-03-19T03:03:56.459 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_fast_dispatch_time": 0.006324951, 2024-03-19T03:03:56.459 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:03:56.459 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 976, 2024-03-19T03:03:56.460 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.045981996, 2024-03-19T03:03:56.460 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000047112 2024-03-19T03:03:56.460 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.460 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:03:56.460 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 1033, 2024-03-19T03:03:56.460 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000180249, 2024-03-19T03:03:56.460 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000174 2024-03-19T03:03:56.460 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.460 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_encrypted_bytes": 1483648, 2024-03-19T03:03:56.460 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_encrypted_bytes": 453728 2024-03-19T03:03:56.460 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.461 INFO:teuthology.orchestra.run.smithi096.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T03:03:56.461 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_messages": 59, 2024-03-19T03:03:56.461 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages": 100, 2024-03-19T03:03:56.461 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_bytes": 10938, 2024-03-19T03:03:56.461 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_bytes": 585709, 2024-03-19T03:03:56.461 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_created_connections": 48, 2024-03-19T03:03:56.461 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_active_connections": 8, 2024-03-19T03:03:56.461 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_total_time": 0.019686001, 2024-03-19T03:03:56.461 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_send_time": 0.002341664, 2024-03-19T03:03:56.461 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_recv_time": 0.014879379, 2024-03-19T03:03:56.461 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_fast_dispatch_time": 0.000776873, 2024-03-19T03:03:56.461 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:03:56.461 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 95, 2024-03-19T03:03:56.461 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.009679188, 2024-03-19T03:03:56.461 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000101886 2024-03-19T03:03:56.462 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.462 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:03:56.462 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 60, 2024-03-19T03:03:56.462 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000006769, 2024-03-19T03:03:56.462 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000112 2024-03-19T03:03:56.462 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.462 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_encrypted_bytes": 2928, 2024-03-19T03:03:56.462 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_encrypted_bytes": 135936 2024-03-19T03:03:56.462 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.462 INFO:teuthology.orchestra.run.smithi096.stdout: "cct": { 2024-03-19T03:03:56.462 INFO:teuthology.orchestra.run.smithi096.stdout: "total_workers": 1, 2024-03-19T03:03:56.462 INFO:teuthology.orchestra.run.smithi096.stdout: "unhealthy_workers": 0 2024-03-19T03:03:56.462 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.463 INFO:teuthology.orchestra.run.smithi096.stdout: "finisher-MDSRank": { 2024-03-19T03:03:56.463 INFO:teuthology.orchestra.run.smithi096.stdout: "queue_len": 0, 2024-03-19T03:03:56.463 INFO:teuthology.orchestra.run.smithi096.stdout: "complete_latency": { 2024-03-19T03:03:56.463 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 55, 2024-03-19T03:03:56.463 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.001606684, 2024-03-19T03:03:56.463 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000029212 2024-03-19T03:03:56.463 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-03-19T03:03:56.463 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.463 INFO:teuthology.orchestra.run.smithi096.stdout: "finisher-PurgeQueue": { 2024-03-19T03:03:56.463 INFO:teuthology.orchestra.run.smithi096.stdout: "queue_len": 0, 2024-03-19T03:03:56.463 INFO:teuthology.orchestra.run.smithi096.stdout: "complete_latency": { 2024-03-19T03:03:56.463 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 2, 2024-03-19T03:03:56.463 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000044474, 2024-03-19T03:03:56.463 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000022237 2024-03-19T03:03:56.463 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-03-19T03:03:56.464 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.464 INFO:teuthology.orchestra.run.smithi096.stdout: "mds": { 2024-03-19T03:03:56.464 INFO:teuthology.orchestra.run.smithi096.stdout: "request": 7, 2024-03-19T03:03:56.464 INFO:teuthology.orchestra.run.smithi096.stdout: "reply": 5, 2024-03-19T03:03:56.464 INFO:teuthology.orchestra.run.smithi096.stdout: "reply_latency": { 2024-03-19T03:03:56.464 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 5, 2024-03-19T03:03:56.464 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.001393410, 2024-03-19T03:03:56.464 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000278682 2024-03-19T03:03:56.464 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.464 INFO:teuthology.orchestra.run.smithi096.stdout: "slow_reply": 0, 2024-03-19T03:03:56.465 INFO:teuthology.orchestra.run.smithi096.stdout: "forward": 2, 2024-03-19T03:03:56.465 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_fetch_complete": 0, 2024-03-19T03:03:56.465 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_fetch_keys": 0, 2024-03-19T03:03:56.465 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_commit": 11, 2024-03-19T03:03:56.465 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_split": 0, 2024-03-19T03:03:56.465 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_merge": 0, 2024-03-19T03:03:56.465 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes": 16, 2024-03-19T03:03:56.465 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes_top": 0, 2024-03-19T03:03:56.465 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes_bottom": 16, 2024-03-19T03:03:56.465 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes_pin_tail": 0, 2024-03-19T03:03:56.465 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes_pinned": 13, 2024-03-19T03:03:56.465 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes_expired": 4, 2024-03-19T03:03:56.465 INFO:teuthology.orchestra.run.smithi096.stdout: "inodes_with_caps": 3, 2024-03-19T03:03:56.466 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": 3, 2024-03-19T03:03:56.466 INFO:teuthology.orchestra.run.smithi096.stdout: "subtrees": 3, 2024-03-19T03:03:56.466 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse": 16, 2024-03-19T03:03:56.466 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse_hit": 11, 2024-03-19T03:03:56.466 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse_forward": 2, 2024-03-19T03:03:56.466 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse_discover": 3, 2024-03-19T03:03:56.466 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse_dir_fetch": 0, 2024-03-19T03:03:56.466 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse_remote_ino": 0, 2024-03-19T03:03:56.466 INFO:teuthology.orchestra.run.smithi096.stdout: "traverse_lock": 0, 2024-03-19T03:03:56.466 INFO:teuthology.orchestra.run.smithi096.stdout: "load_cent": 0, 2024-03-19T03:03:56.466 INFO:teuthology.orchestra.run.smithi096.stdout: "q": 0, 2024-03-19T03:03:56.466 INFO:teuthology.orchestra.run.smithi096.stdout: "exported": 0, 2024-03-19T03:03:56.466 INFO:teuthology.orchestra.run.smithi096.stdout: "exported_inodes": 0, 2024-03-19T03:03:56.466 INFO:teuthology.orchestra.run.smithi096.stdout: "imported": 0, 2024-03-19T03:03:56.467 INFO:teuthology.orchestra.run.smithi096.stdout: "imported_inodes": 0, 2024-03-19T03:03:56.467 INFO:teuthology.orchestra.run.smithi096.stdout: "openino_dir_fetch": 0, 2024-03-19T03:03:56.467 INFO:teuthology.orchestra.run.smithi096.stdout: "openino_backtrace_fetch": 0, 2024-03-19T03:03:56.467 INFO:teuthology.orchestra.run.smithi096.stdout: "openino_peer_discover": 0, 2024-03-19T03:03:56.467 INFO:teuthology.orchestra.run.smithi096.stdout: "root_rfiles": 4, 2024-03-19T03:03:56.467 INFO:teuthology.orchestra.run.smithi096.stdout: "root_rbytes": 234, 2024-03-19T03:03:56.467 INFO:teuthology.orchestra.run.smithi096.stdout: "root_rsnaps": 6, 2024-03-19T03:03:56.467 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_backtrace_fetch": 0, 2024-03-19T03:03:56.467 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_set_tag": 0, 2024-03-19T03:03:56.467 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T03:03:56.467 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_inotable_repaired": 0, 2024-03-19T03:03:56.467 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_dir_inodes": 0, 2024-03-19T03:03:56.467 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T03:03:56.467 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T03:03:56.467 INFO:teuthology.orchestra.run.smithi096.stdout: "scrub_file_inodes": 0, 2024-03-19T03:03:56.468 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_inode_file_caps": 0, 2024-03-19T03:03:56.468 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_cap_op_revoke": 0, 2024-03-19T03:03:56.468 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_cap_op_grant": 0, 2024-03-19T03:03:56.468 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T03:03:56.468 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T03:03:56.468 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T03:03:56.468 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_client_caps": 0, 2024-03-19T03:03:56.468 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_client_caps_dirty": 0, 2024-03-19T03:03:56.468 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_client_cap_release": 0, 2024-03-19T03:03:56.468 INFO:teuthology.orchestra.run.smithi096.stdout: "process_request_cap_release": 0 2024-03-19T03:03:56.468 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.468 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_cache": { 2024-03-19T03:03:56.468 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_update": 0, 2024-03-19T03:03:56.468 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_update_receipt": 11, 2024-03-19T03:03:56.468 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_try_discover": 7, 2024-03-19T03:03:56.469 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_send_discover": 8, 2024-03-19T03:03:56.469 INFO:teuthology.orchestra.run.smithi096.stdout: "dir_handle_discover": 0, 2024-03-19T03:03:56.469 INFO:teuthology.orchestra.run.smithi096.stdout: "num_strays": 0, 2024-03-19T03:03:56.469 INFO:teuthology.orchestra.run.smithi096.stdout: "num_strays_delayed": 0, 2024-03-19T03:03:56.469 INFO:teuthology.orchestra.run.smithi096.stdout: "num_strays_enqueuing": 0, 2024-03-19T03:03:56.469 INFO:teuthology.orchestra.run.smithi096.stdout: "strays_created": 0, 2024-03-19T03:03:56.469 INFO:teuthology.orchestra.run.smithi096.stdout: "strays_enqueued": 0, 2024-03-19T03:03:56.469 INFO:teuthology.orchestra.run.smithi096.stdout: "strays_reintegrated": 0, 2024-03-19T03:03:56.469 INFO:teuthology.orchestra.run.smithi096.stdout: "strays_migrated": 0, 2024-03-19T03:03:56.469 INFO:teuthology.orchestra.run.smithi096.stdout: "num_recovering_processing": 0, 2024-03-19T03:03:56.469 INFO:teuthology.orchestra.run.smithi096.stdout: "num_recovering_enqueued": 0, 2024-03-19T03:03:56.469 INFO:teuthology.orchestra.run.smithi096.stdout: "num_recovering_prioritized": 0, 2024-03-19T03:03:56.469 INFO:teuthology.orchestra.run.smithi096.stdout: "recovery_started": 0, 2024-03-19T03:03:56.469 INFO:teuthology.orchestra.run.smithi096.stdout: "recovery_completed": 0, 2024-03-19T03:03:56.470 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_quiesce_path": 0, 2024-03-19T03:03:56.470 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_quiesce_inode": 0, 2024-03-19T03:03:56.470 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T03:03:56.470 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_exportdir": 0, 2024-03-19T03:03:56.470 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_flush": 0, 2024-03-19T03:03:56.470 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_fragmentdir": 0, 2024-03-19T03:03:56.470 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_fragstats": 0, 2024-03-19T03:03:56.470 INFO:teuthology.orchestra.run.smithi096.stdout: "ireq_inodestats": 0 2024-03-19T03:03:56.470 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.470 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_log": { 2024-03-19T03:03:56.470 INFO:teuthology.orchestra.run.smithi096.stdout: "evlrg": 0, 2024-03-19T03:03:56.470 INFO:teuthology.orchestra.run.smithi096.stdout: "evadd": 7, 2024-03-19T03:03:56.470 INFO:teuthology.orchestra.run.smithi096.stdout: "evex": 0, 2024-03-19T03:03:56.470 INFO:teuthology.orchestra.run.smithi096.stdout: "evtrm": 0, 2024-03-19T03:03:56.470 INFO:teuthology.orchestra.run.smithi096.stdout: "ev": 7, 2024-03-19T03:03:56.471 INFO:teuthology.orchestra.run.smithi096.stdout: "evexg": 0, 2024-03-19T03:03:56.471 INFO:teuthology.orchestra.run.smithi096.stdout: "evexd": 0, 2024-03-19T03:03:56.471 INFO:teuthology.orchestra.run.smithi096.stdout: "segadd": 2, 2024-03-19T03:03:56.471 INFO:teuthology.orchestra.run.smithi096.stdout: "segex": 0, 2024-03-19T03:03:56.471 INFO:teuthology.orchestra.run.smithi096.stdout: "segtrm": 0, 2024-03-19T03:03:56.471 INFO:teuthology.orchestra.run.smithi096.stdout: "seg": 2, 2024-03-19T03:03:56.471 INFO:teuthology.orchestra.run.smithi096.stdout: "segmjr": 2, 2024-03-19T03:03:56.471 INFO:teuthology.orchestra.run.smithi096.stdout: "segexg": 0, 2024-03-19T03:03:56.471 INFO:teuthology.orchestra.run.smithi096.stdout: "segexd": 0, 2024-03-19T03:03:56.471 INFO:teuthology.orchestra.run.smithi096.stdout: "expos": 4194304, 2024-03-19T03:03:56.471 INFO:teuthology.orchestra.run.smithi096.stdout: "wrpos": 4196449, 2024-03-19T03:03:56.471 INFO:teuthology.orchestra.run.smithi096.stdout: "rdpos": 0, 2024-03-19T03:03:56.471 INFO:teuthology.orchestra.run.smithi096.stdout: "jlat": { 2024-03-19T03:03:56.471 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 6, 2024-03-19T03:03:56.471 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.009264307, 2024-03-19T03:03:56.472 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.001544051 2024-03-19T03:03:56.472 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.472 INFO:teuthology.orchestra.run.smithi096.stdout: "replayed": 0 2024-03-19T03:03:56.472 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.472 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_mem": { 2024-03-19T03:03:56.472 INFO:teuthology.orchestra.run.smithi096.stdout: "ino": 20, 2024-03-19T03:03:56.472 INFO:teuthology.orchestra.run.smithi096.stdout: "ino+": 22, 2024-03-19T03:03:56.472 INFO:teuthology.orchestra.run.smithi096.stdout: "ino-": 2, 2024-03-19T03:03:56.472 INFO:teuthology.orchestra.run.smithi096.stdout: "dir": 19, 2024-03-19T03:03:56.472 INFO:teuthology.orchestra.run.smithi096.stdout: "dir+": 22, 2024-03-19T03:03:56.472 INFO:teuthology.orchestra.run.smithi096.stdout: "dir-": 3, 2024-03-19T03:03:56.472 INFO:teuthology.orchestra.run.smithi096.stdout: "dn": 16, 2024-03-19T03:03:56.472 INFO:teuthology.orchestra.run.smithi096.stdout: "dn+": 20, 2024-03-19T03:03:56.472 INFO:teuthology.orchestra.run.smithi096.stdout: "dn-": 4, 2024-03-19T03:03:56.472 INFO:teuthology.orchestra.run.smithi096.stdout: "cap": 3, 2024-03-19T03:03:56.473 INFO:teuthology.orchestra.run.smithi096.stdout: "cap+": 5, 2024-03-19T03:03:56.473 INFO:teuthology.orchestra.run.smithi096.stdout: "cap-": 2, 2024-03-19T03:03:56.473 INFO:teuthology.orchestra.run.smithi096.stdout: "rss": 54976, 2024-03-19T03:03:56.473 INFO:teuthology.orchestra.run.smithi096.stdout: "heap": 223612 2024-03-19T03:03:56.473 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.473 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_server": { 2024-03-19T03:03:56.473 INFO:teuthology.orchestra.run.smithi096.stdout: "dispatch_client_request": 7, 2024-03-19T03:03:56.473 INFO:teuthology.orchestra.run.smithi096.stdout: "dispatch_server_request": 0, 2024-03-19T03:03:56.473 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_client_request": 7, 2024-03-19T03:03:56.473 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_client_session": 17, 2024-03-19T03:03:56.473 INFO:teuthology.orchestra.run.smithi096.stdout: "handle_peer_request": 0, 2024-03-19T03:03:56.473 INFO:teuthology.orchestra.run.smithi096.stdout: "req_create_latency": { 2024-03-19T03:03:56.473 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.473 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.473 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.474 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.474 INFO:teuthology.orchestra.run.smithi096.stdout: "req_getattr_latency": { 2024-03-19T03:03:56.474 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 4, 2024-03-19T03:03:56.474 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.001046379, 2024-03-19T03:03:56.474 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000261594 2024-03-19T03:03:56.474 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.474 INFO:teuthology.orchestra.run.smithi096.stdout: "req_getfilelock_latency": { 2024-03-19T03:03:56.474 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.474 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.474 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.474 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.474 INFO:teuthology.orchestra.run.smithi096.stdout: "req_link_latency": { 2024-03-19T03:03:56.474 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.474 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.475 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.475 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.475 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lookup_latency": { 2024-03-19T03:03:56.475 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 1, 2024-03-19T03:03:56.475 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000347031, 2024-03-19T03:03:56.475 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000347031 2024-03-19T03:03:56.475 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.475 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lookuphash_latency": { 2024-03-19T03:03:56.475 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.475 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.475 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.475 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.475 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lookupino_latency": { 2024-03-19T03:03:56.475 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.476 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.476 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.476 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.476 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lookupname_latency": { 2024-03-19T03:03:56.476 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.476 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.476 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.476 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.476 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lookupparent_latency": { 2024-03-19T03:03:56.476 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.476 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.476 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.476 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.476 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lookupsnap_latency": { 2024-03-19T03:03:56.476 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.477 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.477 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.477 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.477 INFO:teuthology.orchestra.run.smithi096.stdout: "req_lssnap_latency": { 2024-03-19T03:03:56.477 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.477 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.477 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.477 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.477 INFO:teuthology.orchestra.run.smithi096.stdout: "req_mkdir_latency": { 2024-03-19T03:03:56.477 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.478 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.478 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.478 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.478 INFO:teuthology.orchestra.run.smithi096.stdout: "req_mknod_latency": { 2024-03-19T03:03:56.478 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.478 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.478 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.478 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.478 INFO:teuthology.orchestra.run.smithi096.stdout: "req_mksnap_latency": { 2024-03-19T03:03:56.478 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.479 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.479 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.479 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.479 INFO:teuthology.orchestra.run.smithi096.stdout: "req_open_latency": { 2024-03-19T03:03:56.479 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.479 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.479 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.479 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.479 INFO:teuthology.orchestra.run.smithi096.stdout: "req_readdir_latency": { 2024-03-19T03:03:56.479 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.479 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.479 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.479 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.479 INFO:teuthology.orchestra.run.smithi096.stdout: "req_rename_latency": { 2024-03-19T03:03:56.479 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.480 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.480 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.480 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.480 INFO:teuthology.orchestra.run.smithi096.stdout: "req_renamesnap_latency": { 2024-03-19T03:03:56.480 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.480 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.480 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.480 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.480 INFO:teuthology.orchestra.run.smithi096.stdout: "req_snapdiff_latency": { 2024-03-19T03:03:56.480 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.480 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.480 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.480 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.480 INFO:teuthology.orchestra.run.smithi096.stdout: "req_rmdir_latency": { 2024-03-19T03:03:56.481 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.481 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.481 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.481 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.481 INFO:teuthology.orchestra.run.smithi096.stdout: "req_rmsnap_latency": { 2024-03-19T03:03:56.481 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.481 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.481 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.481 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.481 INFO:teuthology.orchestra.run.smithi096.stdout: "req_rmxattr_latency": { 2024-03-19T03:03:56.481 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.481 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.481 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.481 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.481 INFO:teuthology.orchestra.run.smithi096.stdout: "req_setattr_latency": { 2024-03-19T03:03:56.481 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.482 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.482 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.482 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.482 INFO:teuthology.orchestra.run.smithi096.stdout: "req_setdirlayout_latency": { 2024-03-19T03:03:56.482 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.482 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.482 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.482 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.482 INFO:teuthology.orchestra.run.smithi096.stdout: "req_setfilelock_latency": { 2024-03-19T03:03:56.482 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.482 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.482 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.482 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.482 INFO:teuthology.orchestra.run.smithi096.stdout: "req_setlayout_latency": { 2024-03-19T03:03:56.483 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.483 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.483 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.483 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.483 INFO:teuthology.orchestra.run.smithi096.stdout: "req_setxattr_latency": { 2024-03-19T03:03:56.483 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.483 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.483 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.483 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.483 INFO:teuthology.orchestra.run.smithi096.stdout: "req_symlink_latency": { 2024-03-19T03:03:56.483 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.483 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.483 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.483 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.484 INFO:teuthology.orchestra.run.smithi096.stdout: "req_unlink_latency": { 2024-03-19T03:03:56.484 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.484 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.484 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.484 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.484 INFO:teuthology.orchestra.run.smithi096.stdout: "cap_revoke_eviction": 0, 2024-03-19T03:03:56.484 INFO:teuthology.orchestra.run.smithi096.stdout: "cap_acquisition_throttle": 0, 2024-03-19T03:03:56.484 INFO:teuthology.orchestra.run.smithi096.stdout: "req_getvxattr_latency": { 2024-03-19T03:03:56.484 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.484 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.484 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.484 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-03-19T03:03:56.484 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.484 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_sessions": { 2024-03-19T03:03:56.484 INFO:teuthology.orchestra.run.smithi096.stdout: "session_count": 1, 2024-03-19T03:03:56.485 INFO:teuthology.orchestra.run.smithi096.stdout: "session_add": 3, 2024-03-19T03:03:56.485 INFO:teuthology.orchestra.run.smithi096.stdout: "session_remove": 2, 2024-03-19T03:03:56.485 INFO:teuthology.orchestra.run.smithi096.stdout: "sessions_open": 1, 2024-03-19T03:03:56.485 INFO:teuthology.orchestra.run.smithi096.stdout: "sessions_stale": 0, 2024-03-19T03:03:56.485 INFO:teuthology.orchestra.run.smithi096.stdout: "total_load": 4, 2024-03-19T03:03:56.485 INFO:teuthology.orchestra.run.smithi096.stdout: "average_load": 1, 2024-03-19T03:03:56.485 INFO:teuthology.orchestra.run.smithi096.stdout: "avg_session_uptime": 645, 2024-03-19T03:03:56.485 INFO:teuthology.orchestra.run.smithi096.stdout: "mdthresh_evicted": 0 2024-03-19T03:03:56.485 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.485 INFO:teuthology.orchestra.run.smithi096.stdout: "mempool": { 2024-03-19T03:03:56.485 INFO:teuthology.orchestra.run.smithi096.stdout: "bloom_filter_bytes": 0, 2024-03-19T03:03:56.485 INFO:teuthology.orchestra.run.smithi096.stdout: "bloom_filter_items": 0, 2024-03-19T03:03:56.485 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T03:03:56.485 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_alloc_items": 0, 2024-03-19T03:03:56.485 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T03:03:56.486 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_data_items": 0, 2024-03-19T03:03:56.486 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T03:03:56.486 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T03:03:56.486 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T03:03:56.486 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T03:03:56.486 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T03:03:56.486 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_other_items": 0, 2024-03-19T03:03:56.486 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T03:03:56.486 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T03:03:56.486 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_extent_bytes": 0, 2024-03-19T03:03:56.486 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_extent_items": 0, 2024-03-19T03:03:56.486 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_blob_bytes": 0, 2024-03-19T03:03:56.486 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_blob_items": 0, 2024-03-19T03:03:56.486 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T03:03:56.486 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T03:03:56.487 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T03:03:56.487 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T03:03:56.487 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T03:03:56.487 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_fsck_items": 0, 2024-03-19T03:03:56.487 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_txc_bytes": 0, 2024-03-19T03:03:56.487 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_txc_items": 0, 2024-03-19T03:03:56.487 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T03:03:56.487 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T03:03:56.487 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_writing_bytes": 0, 2024-03-19T03:03:56.487 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_writing_items": 0, 2024-03-19T03:03:56.487 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_bytes": 0, 2024-03-19T03:03:56.487 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_items": 0, 2024-03-19T03:03:56.487 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T03:03:56.487 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_file_reader_items": 0, 2024-03-19T03:03:56.487 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T03:03:56.488 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_file_writer_items": 0, 2024-03-19T03:03:56.488 INFO:teuthology.orchestra.run.smithi096.stdout: "buffer_anon_bytes": 264011, 2024-03-19T03:03:56.488 INFO:teuthology.orchestra.run.smithi096.stdout: "buffer_anon_items": 96, 2024-03-19T03:03:56.488 INFO:teuthology.orchestra.run.smithi096.stdout: "buffer_meta_bytes": 0, 2024-03-19T03:03:56.488 INFO:teuthology.orchestra.run.smithi096.stdout: "buffer_meta_items": 0, 2024-03-19T03:03:56.488 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_bytes": 0, 2024-03-19T03:03:56.488 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_items": 0, 2024-03-19T03:03:56.488 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_mapbl_bytes": 0, 2024-03-19T03:03:56.488 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_mapbl_items": 0, 2024-03-19T03:03:56.488 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_pglog_bytes": 0, 2024-03-19T03:03:56.488 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_pglog_items": 0, 2024-03-19T03:03:56.488 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_bytes": 9512, 2024-03-19T03:03:56.488 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_items": 182, 2024-03-19T03:03:56.488 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T03:03:56.488 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_mapping_items": 0, 2024-03-19T03:03:56.489 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap_bytes": 0, 2024-03-19T03:03:56.489 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap_items": 0, 2024-03-19T03:03:56.489 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_co_bytes": 88021, 2024-03-19T03:03:56.489 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_co_items": 348, 2024-03-19T03:03:56.489 INFO:teuthology.orchestra.run.smithi096.stdout: "unittest_1_bytes": 0, 2024-03-19T03:03:56.489 INFO:teuthology.orchestra.run.smithi096.stdout: "unittest_1_items": 0, 2024-03-19T03:03:56.489 INFO:teuthology.orchestra.run.smithi096.stdout: "unittest_2_bytes": 0, 2024-03-19T03:03:56.489 INFO:teuthology.orchestra.run.smithi096.stdout: "unittest_2_items": 0 2024-03-19T03:03:56.489 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.489 INFO:teuthology.orchestra.run.smithi096.stdout: "objecter": { 2024-03-19T03:03:56.489 INFO:teuthology.orchestra.run.smithi096.stdout: "op_active": 0, 2024-03-19T03:03:56.489 INFO:teuthology.orchestra.run.smithi096.stdout: "op_laggy": 0, 2024-03-19T03:03:56.489 INFO:teuthology.orchestra.run.smithi096.stdout: "op_send": 44, 2024-03-19T03:03:56.489 INFO:teuthology.orchestra.run.smithi096.stdout: "op_send_bytes": 12756, 2024-03-19T03:03:56.489 INFO:teuthology.orchestra.run.smithi096.stdout: "op_resend": 30, 2024-03-19T03:03:56.490 INFO:teuthology.orchestra.run.smithi096.stdout: "op_reply": 44, 2024-03-19T03:03:56.490 INFO:teuthology.orchestra.run.smithi096.stdout: "op_latency": { 2024-03-19T03:03:56.490 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 44, 2024-03-19T03:03:56.490 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.214994739, 2024-03-19T03:03:56.490 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.004886244 2024-03-19T03:03:56.490 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.490 INFO:teuthology.orchestra.run.smithi096.stdout: "op_inflight": 0, 2024-03-19T03:03:56.490 INFO:teuthology.orchestra.run.smithi096.stdout: "oplen_avg": { 2024-03-19T03:03:56.490 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 44, 2024-03-19T03:03:56.490 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 67 2024-03-19T03:03:56.490 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.490 INFO:teuthology.orchestra.run.smithi096.stdout: "op": 44, 2024-03-19T03:03:56.490 INFO:teuthology.orchestra.run.smithi096.stdout: "op_r": 0, 2024-03-19T03:03:56.490 INFO:teuthology.orchestra.run.smithi096.stdout: "op_w": 44, 2024-03-19T03:03:56.490 INFO:teuthology.orchestra.run.smithi096.stdout: "op_rmw": 0, 2024-03-19T03:03:56.491 INFO:teuthology.orchestra.run.smithi096.stdout: "op_pg": 0, 2024-03-19T03:03:56.491 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_stat": 0, 2024-03-19T03:03:56.491 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_create": 10, 2024-03-19T03:03:56.491 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_read": 0, 2024-03-19T03:03:56.491 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_write": 6, 2024-03-19T03:03:56.491 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_writefull": 8, 2024-03-19T03:03:56.491 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_writesame": 0, 2024-03-19T03:03:56.491 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_append": 0, 2024-03-19T03:03:56.491 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_zero": 0, 2024-03-19T03:03:56.491 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_truncate": 0, 2024-03-19T03:03:56.491 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_delete": 6, 2024-03-19T03:03:56.491 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_mapext": 0, 2024-03-19T03:03:56.491 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_sparse_read": 0, 2024-03-19T03:03:56.491 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_clonerange": 0, 2024-03-19T03:03:56.491 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_getxattr": 0, 2024-03-19T03:03:56.492 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_setxattr": 20, 2024-03-19T03:03:56.492 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_cmpxattr": 0, 2024-03-19T03:03:56.492 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_rmxattr": 0, 2024-03-19T03:03:56.492 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_resetxattrs": 0, 2024-03-19T03:03:56.492 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_call": 0, 2024-03-19T03:03:56.492 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_watch": 0, 2024-03-19T03:03:56.492 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_notify": 0, 2024-03-19T03:03:56.492 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T03:03:56.492 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_pgls": 0, 2024-03-19T03:03:56.492 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_pgls_filter": 0, 2024-03-19T03:03:56.492 INFO:teuthology.orchestra.run.smithi096.stdout: "osdop_other": 0, 2024-03-19T03:03:56.492 INFO:teuthology.orchestra.run.smithi096.stdout: "linger_active": 0, 2024-03-19T03:03:56.492 INFO:teuthology.orchestra.run.smithi096.stdout: "linger_send": 0, 2024-03-19T03:03:56.492 INFO:teuthology.orchestra.run.smithi096.stdout: "linger_resend": 0, 2024-03-19T03:03:56.492 INFO:teuthology.orchestra.run.smithi096.stdout: "linger_ping": 0, 2024-03-19T03:03:56.493 INFO:teuthology.orchestra.run.smithi096.stdout: "poolop_active": 0, 2024-03-19T03:03:56.493 INFO:teuthology.orchestra.run.smithi096.stdout: "poolop_send": 0, 2024-03-19T03:03:56.493 INFO:teuthology.orchestra.run.smithi096.stdout: "poolop_resend": 0, 2024-03-19T03:03:56.493 INFO:teuthology.orchestra.run.smithi096.stdout: "poolstat_active": 0, 2024-03-19T03:03:56.493 INFO:teuthology.orchestra.run.smithi096.stdout: "poolstat_send": 0, 2024-03-19T03:03:56.493 INFO:teuthology.orchestra.run.smithi096.stdout: "poolstat_resend": 0, 2024-03-19T03:03:56.493 INFO:teuthology.orchestra.run.smithi096.stdout: "statfs_active": 0, 2024-03-19T03:03:56.493 INFO:teuthology.orchestra.run.smithi096.stdout: "statfs_send": 0, 2024-03-19T03:03:56.493 INFO:teuthology.orchestra.run.smithi096.stdout: "statfs_resend": 0, 2024-03-19T03:03:56.493 INFO:teuthology.orchestra.run.smithi096.stdout: "command_active": 0, 2024-03-19T03:03:56.493 INFO:teuthology.orchestra.run.smithi096.stdout: "command_send": 0, 2024-03-19T03:03:56.493 INFO:teuthology.orchestra.run.smithi096.stdout: "command_resend": 0, 2024-03-19T03:03:56.493 INFO:teuthology.orchestra.run.smithi096.stdout: "map_epoch": 106, 2024-03-19T03:03:56.493 INFO:teuthology.orchestra.run.smithi096.stdout: "map_full": 0, 2024-03-19T03:03:56.493 INFO:teuthology.orchestra.run.smithi096.stdout: "map_inc": 13, 2024-03-19T03:03:56.493 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_sessions": 9, 2024-03-19T03:03:56.494 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_session_open": 9, 2024-03-19T03:03:56.494 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_session_close": 0, 2024-03-19T03:03:56.494 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_laggy": 0, 2024-03-19T03:03:56.494 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_wr": 16, 2024-03-19T03:03:56.494 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_rd": 0, 2024-03-19T03:03:56.494 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_del": 1 2024-03-19T03:03:56.494 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.494 INFO:teuthology.orchestra.run.smithi096.stdout: "oft": { 2024-03-19T03:03:56.494 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_total_objs": 1, 2024-03-19T03:03:56.494 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_total_kv_pairs": 0, 2024-03-19T03:03:56.494 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_total_updates": 0, 2024-03-19T03:03:56.494 INFO:teuthology.orchestra.run.smithi096.stdout: "omap_total_removes": 0 2024-03-19T03:03:56.494 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.494 INFO:teuthology.orchestra.run.smithi096.stdout: "osd-slow-ops": { 2024-03-19T03:03:56.495 INFO:teuthology.orchestra.run.smithi096.stdout: "slow_ops_count": 0 2024-03-19T03:03:56.495 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.495 INFO:teuthology.orchestra.run.smithi096.stdout: "purge_queue": { 2024-03-19T03:03:56.495 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_executing_ops": 0, 2024-03-19T03:03:56.495 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_executing_ops_high_water": 0, 2024-03-19T03:03:56.495 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_executing": 0, 2024-03-19T03:03:56.495 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_executing_high_water": 0, 2024-03-19T03:03:56.495 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_executed_ops": 0, 2024-03-19T03:03:56.495 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_executed": 0, 2024-03-19T03:03:56.495 INFO:teuthology.orchestra.run.smithi096.stdout: "pq_item_in_journal": 0 2024-03-19T03:03:56.495 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.495 INFO:teuthology.orchestra.run.smithi096.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T03:03:56.495 INFO:teuthology.orchestra.run.smithi096.stdout: "val": 0, 2024-03-19T03:03:56.495 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 104857600, 2024-03-19T03:03:56.495 INFO:teuthology.orchestra.run.smithi096.stdout: "get_started": 0, 2024-03-19T03:03:56.495 INFO:teuthology.orchestra.run.smithi096.stdout: "get": 904, 2024-03-19T03:03:56.496 INFO:teuthology.orchestra.run.smithi096.stdout: "get_sum": 1568306, 2024-03-19T03:03:56.496 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_fail": 0, 2024-03-19T03:03:56.496 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_success": 904, 2024-03-19T03:03:56.496 INFO:teuthology.orchestra.run.smithi096.stdout: "take": 0, 2024-03-19T03:03:56.496 INFO:teuthology.orchestra.run.smithi096.stdout: "take_sum": 0, 2024-03-19T03:03:56.496 INFO:teuthology.orchestra.run.smithi096.stdout: "put": 904, 2024-03-19T03:03:56.496 INFO:teuthology.orchestra.run.smithi096.stdout: "put_sum": 1568306, 2024-03-19T03:03:56.496 INFO:teuthology.orchestra.run.smithi096.stdout: "wait": { 2024-03-19T03:03:56.496 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.496 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.496 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.496 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-03-19T03:03:56.496 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.496 INFO:teuthology.orchestra.run.smithi096.stdout: "throttle-objecter_bytes": { 2024-03-19T03:03:56.497 INFO:teuthology.orchestra.run.smithi096.stdout: "val": 0, 2024-03-19T03:03:56.497 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 104857600, 2024-03-19T03:03:56.497 INFO:teuthology.orchestra.run.smithi096.stdout: "get_started": 0, 2024-03-19T03:03:56.497 INFO:teuthology.orchestra.run.smithi096.stdout: "get": 0, 2024-03-19T03:03:56.497 INFO:teuthology.orchestra.run.smithi096.stdout: "get_sum": 0, 2024-03-19T03:03:56.497 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_fail": 0, 2024-03-19T03:03:56.497 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_success": 0, 2024-03-19T03:03:56.497 INFO:teuthology.orchestra.run.smithi096.stdout: "take": 44, 2024-03-19T03:03:56.497 INFO:teuthology.orchestra.run.smithi096.stdout: "take_sum": 12756, 2024-03-19T03:03:56.497 INFO:teuthology.orchestra.run.smithi096.stdout: "put": 44, 2024-03-19T03:03:56.497 INFO:teuthology.orchestra.run.smithi096.stdout: "put_sum": 12756, 2024-03-19T03:03:56.497 INFO:teuthology.orchestra.run.smithi096.stdout: "wait": { 2024-03-19T03:03:56.497 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.497 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.498 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.498 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-03-19T03:03:56.498 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.498 INFO:teuthology.orchestra.run.smithi096.stdout: "throttle-objecter_ops": { 2024-03-19T03:03:56.498 INFO:teuthology.orchestra.run.smithi096.stdout: "val": 0, 2024-03-19T03:03:56.498 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 1024, 2024-03-19T03:03:56.498 INFO:teuthology.orchestra.run.smithi096.stdout: "get_started": 0, 2024-03-19T03:03:56.498 INFO:teuthology.orchestra.run.smithi096.stdout: "get": 0, 2024-03-19T03:03:56.498 INFO:teuthology.orchestra.run.smithi096.stdout: "get_sum": 0, 2024-03-19T03:03:56.498 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_fail": 0, 2024-03-19T03:03:56.498 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_success": 0, 2024-03-19T03:03:56.498 INFO:teuthology.orchestra.run.smithi096.stdout: "take": 44, 2024-03-19T03:03:56.498 INFO:teuthology.orchestra.run.smithi096.stdout: "take_sum": 44, 2024-03-19T03:03:56.498 INFO:teuthology.orchestra.run.smithi096.stdout: "put": 44, 2024-03-19T03:03:56.499 INFO:teuthology.orchestra.run.smithi096.stdout: "put_sum": 44, 2024-03-19T03:03:56.499 INFO:teuthology.orchestra.run.smithi096.stdout: "wait": { 2024-03-19T03:03:56.499 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.499 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.499 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.499 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-03-19T03:03:56.499 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.499 INFO:teuthology.orchestra.run.smithi096.stdout: "throttle-write_buf_throttle": { 2024-03-19T03:03:56.499 INFO:teuthology.orchestra.run.smithi096.stdout: "val": 0, 2024-03-19T03:03:56.499 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 3758096384, 2024-03-19T03:03:56.499 INFO:teuthology.orchestra.run.smithi096.stdout: "get_started": 0, 2024-03-19T03:03:56.499 INFO:teuthology.orchestra.run.smithi096.stdout: "get": 0, 2024-03-19T03:03:56.499 INFO:teuthology.orchestra.run.smithi096.stdout: "get_sum": 0, 2024-03-19T03:03:56.499 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_fail": 0, 2024-03-19T03:03:56.499 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_success": 0, 2024-03-19T03:03:56.500 INFO:teuthology.orchestra.run.smithi096.stdout: "take": 0, 2024-03-19T03:03:56.500 INFO:teuthology.orchestra.run.smithi096.stdout: "take_sum": 0, 2024-03-19T03:03:56.500 INFO:teuthology.orchestra.run.smithi096.stdout: "put": 0, 2024-03-19T03:03:56.500 INFO:teuthology.orchestra.run.smithi096.stdout: "put_sum": 0, 2024-03-19T03:03:56.500 INFO:teuthology.orchestra.run.smithi096.stdout: "wait": { 2024-03-19T03:03:56.500 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.500 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.500 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.500 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-03-19T03:03:56.500 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:56.500 INFO:teuthology.orchestra.run.smithi096.stdout: "throttle-write_buf_throttle-0x55cf38926500": { 2024-03-19T03:03:56.500 INFO:teuthology.orchestra.run.smithi096.stdout: "val": 0, 2024-03-19T03:03:56.500 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 3758096384, 2024-03-19T03:03:56.500 INFO:teuthology.orchestra.run.smithi096.stdout: "get_started": 0, 2024-03-19T03:03:56.500 INFO:teuthology.orchestra.run.smithi096.stdout: "get": 7, 2024-03-19T03:03:56.501 INFO:teuthology.orchestra.run.smithi096.stdout: "get_sum": 2145, 2024-03-19T03:03:56.501 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_fail": 0, 2024-03-19T03:03:56.501 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_success": 7, 2024-03-19T03:03:56.501 INFO:teuthology.orchestra.run.smithi096.stdout: "take": 0, 2024-03-19T03:03:56.501 INFO:teuthology.orchestra.run.smithi096.stdout: "take_sum": 0, 2024-03-19T03:03:56.501 INFO:teuthology.orchestra.run.smithi096.stdout: "put": 6, 2024-03-19T03:03:56.501 INFO:teuthology.orchestra.run.smithi096.stdout: "put_sum": 2145, 2024-03-19T03:03:56.501 INFO:teuthology.orchestra.run.smithi096.stdout: "wait": { 2024-03-19T03:03:56.501 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:56.501 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:56.501 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:56.501 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-03-19T03:03:56.501 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-03-19T03:03:56.501 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-03-19T03:03:57.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:56 smithi096 ceph-mon[35946]: pgmap v325: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:03:57.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:56 smithi155 ceph-mon[35804]: pgmap v325: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:03:57.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:56 smithi033 ceph-mon[31973]: pgmap v325: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:03:57.310 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-03-19T03:03:57.310 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-03-19T03:03:57.310 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2024-03-19T03:03:57.310 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=11 2024-03-19T03:03:57.310 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-03-19T03:03:57.310 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-03-19T03:03:57.310 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-03-19T03:03:57.310 DEBUG:tasks.check_counter:Getting stats from k 2024-03-19T03:03:57.311 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-19T03:03:57.569 INFO:teuthology.orchestra.run.smithi096.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T03:03:59.073 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:03:58 smithi096 ceph-mon[35946]: pgmap v326: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:03:59.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:03:58 smithi155 ceph-mon[35804]: pgmap v326: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:03:59.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:03:58 smithi033 ceph-mon[31973]: pgmap v326: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:03:59.346 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-03-19T03:03:59.346 INFO:teuthology.orchestra.run.smithi096.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T03:03:59.346 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_messages": 0, 2024-03-19T03:03:59.346 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages": 2, 2024-03-19T03:03:59.346 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_bytes": 0, 2024-03-19T03:03:59.347 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_bytes": 0, 2024-03-19T03:03:59.347 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_created_connections": 4, 2024-03-19T03:03:59.347 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_active_connections": 0, 2024-03-19T03:03:59.347 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_total_time": 0.001122927, 2024-03-19T03:03:59.347 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_send_time": 0.000000000, 2024-03-19T03:03:59.347 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_recv_time": 0.000335429, 2024-03-19T03:03:59.347 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-19T03:03:59.347 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:03:59.347 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:59.347 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:59.347 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:59.347 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:59.347 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:03:59.347 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:59.348 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:59.348 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:59.348 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:59.348 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-19T03:03:59.348 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_encrypted_bytes": 0 2024-03-19T03:03:59.348 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:59.348 INFO:teuthology.orchestra.run.smithi096.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T03:03:59.348 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_messages": 6, 2024-03-19T03:03:59.348 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages": 171, 2024-03-19T03:03:59.348 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_bytes": 3312, 2024-03-19T03:03:59.348 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_bytes": 185760, 2024-03-19T03:03:59.348 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_created_connections": 5, 2024-03-19T03:03:59.348 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_active_connections": 3, 2024-03-19T03:03:59.348 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_total_time": 0.007882191, 2024-03-19T03:03:59.349 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_send_time": 0.004721111, 2024-03-19T03:03:59.349 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_recv_time": 0.001649006, 2024-03-19T03:03:59.349 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-19T03:03:59.349 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:03:59.349 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 169, 2024-03-19T03:03:59.349 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.014046614, 2024-03-19T03:03:59.349 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000083116 2024-03-19T03:03:59.349 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:59.349 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:03:59.349 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:59.349 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:59.349 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:59.349 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:59.349 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-19T03:03:59.349 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_encrypted_bytes": 185760 2024-03-19T03:03:59.349 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:59.350 INFO:teuthology.orchestra.run.smithi096.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T03:03:59.350 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_messages": 231, 2024-03-19T03:03:59.350 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages": 221, 2024-03-19T03:03:59.350 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_bytes": 1440112, 2024-03-19T03:03:59.350 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_bytes": 135744, 2024-03-19T03:03:59.350 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_created_connections": 2, 2024-03-19T03:03:59.350 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_active_connections": 1, 2024-03-19T03:03:59.350 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_total_time": 0.034428343, 2024-03-19T03:03:59.350 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_send_time": 0.008447059, 2024-03-19T03:03:59.350 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_recv_time": 0.031568949, 2024-03-19T03:03:59.350 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_running_fast_dispatch_time": 0.003989941, 2024-03-19T03:03:59.350 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:03:59.350 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 220, 2024-03-19T03:03:59.350 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.006696892, 2024-03-19T03:03:59.351 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000030440 2024-03-19T03:03:59.351 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:59.351 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:03:59.351 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:59.351 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:59.351 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:59.351 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:59.351 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_recv_encrypted_bytes": 1440112, 2024-03-19T03:03:59.351 INFO:teuthology.orchestra.run.smithi096.stdout: "msgr_send_encrypted_bytes": 135744 2024-03-19T03:03:59.351 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:59.351 INFO:teuthology.orchestra.run.smithi096.stdout: "cct": { 2024-03-19T03:03:59.351 INFO:teuthology.orchestra.run.smithi096.stdout: "total_workers": 0, 2024-03-19T03:03:59.351 INFO:teuthology.orchestra.run.smithi096.stdout: "unhealthy_workers": 0 2024-03-19T03:03:59.351 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:59.351 INFO:teuthology.orchestra.run.smithi096.stdout: "mempool": { 2024-03-19T03:03:59.352 INFO:teuthology.orchestra.run.smithi096.stdout: "bloom_filter_bytes": 0, 2024-03-19T03:03:59.352 INFO:teuthology.orchestra.run.smithi096.stdout: "bloom_filter_items": 0, 2024-03-19T03:03:59.352 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T03:03:59.352 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_alloc_items": 0, 2024-03-19T03:03:59.352 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T03:03:59.352 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_data_items": 0, 2024-03-19T03:03:59.352 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T03:03:59.352 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T03:03:59.352 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T03:03:59.352 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T03:03:59.352 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T03:03:59.352 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_other_items": 0, 2024-03-19T03:03:59.352 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T03:03:59.352 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T03:03:59.353 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_extent_bytes": 0, 2024-03-19T03:03:59.353 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_extent_items": 0, 2024-03-19T03:03:59.353 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_blob_bytes": 0, 2024-03-19T03:03:59.353 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_blob_items": 0, 2024-03-19T03:03:59.353 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T03:03:59.353 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T03:03:59.353 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T03:03:59.353 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T03:03:59.353 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T03:03:59.353 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_fsck_items": 0, 2024-03-19T03:03:59.353 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_txc_bytes": 0, 2024-03-19T03:03:59.353 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_txc_items": 0, 2024-03-19T03:03:59.353 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T03:03:59.353 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T03:03:59.354 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_writing_bytes": 0, 2024-03-19T03:03:59.354 INFO:teuthology.orchestra.run.smithi096.stdout: "bluestore_writing_items": 0, 2024-03-19T03:03:59.354 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_bytes": 0, 2024-03-19T03:03:59.354 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_items": 0, 2024-03-19T03:03:59.354 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T03:03:59.354 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_file_reader_items": 0, 2024-03-19T03:03:59.354 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T03:03:59.354 INFO:teuthology.orchestra.run.smithi096.stdout: "bluefs_file_writer_items": 0, 2024-03-19T03:03:59.354 INFO:teuthology.orchestra.run.smithi096.stdout: "buffer_anon_bytes": 148732, 2024-03-19T03:03:59.354 INFO:teuthology.orchestra.run.smithi096.stdout: "buffer_anon_items": 31, 2024-03-19T03:03:59.354 INFO:teuthology.orchestra.run.smithi096.stdout: "buffer_meta_bytes": 0, 2024-03-19T03:03:59.354 INFO:teuthology.orchestra.run.smithi096.stdout: "buffer_meta_items": 0, 2024-03-19T03:03:59.354 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_bytes": 0, 2024-03-19T03:03:59.354 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_items": 0, 2024-03-19T03:03:59.355 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_mapbl_bytes": 0, 2024-03-19T03:03:59.355 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_mapbl_items": 0, 2024-03-19T03:03:59.355 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_pglog_bytes": 0, 2024-03-19T03:03:59.355 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_pglog_items": 0, 2024-03-19T03:03:59.355 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_bytes": 0, 2024-03-19T03:03:59.355 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_items": 0, 2024-03-19T03:03:59.355 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T03:03:59.355 INFO:teuthology.orchestra.run.smithi096.stdout: "osdmap_mapping_items": 0, 2024-03-19T03:03:59.355 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap_bytes": 0, 2024-03-19T03:03:59.355 INFO:teuthology.orchestra.run.smithi096.stdout: "pgmap_items": 0, 2024-03-19T03:03:59.355 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_co_bytes": 928, 2024-03-19T03:03:59.355 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_co_items": 2, 2024-03-19T03:03:59.355 INFO:teuthology.orchestra.run.smithi096.stdout: "unittest_1_bytes": 0, 2024-03-19T03:03:59.355 INFO:teuthology.orchestra.run.smithi096.stdout: "unittest_1_items": 0, 2024-03-19T03:03:59.355 INFO:teuthology.orchestra.run.smithi096.stdout: "unittest_2_bytes": 0, 2024-03-19T03:03:59.356 INFO:teuthology.orchestra.run.smithi096.stdout: "unittest_2_items": 0 2024-03-19T03:03:59.356 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-03-19T03:03:59.356 INFO:teuthology.orchestra.run.smithi096.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T03:03:59.356 INFO:teuthology.orchestra.run.smithi096.stdout: "val": 0, 2024-03-19T03:03:59.356 INFO:teuthology.orchestra.run.smithi096.stdout: "max": 104857600, 2024-03-19T03:03:59.356 INFO:teuthology.orchestra.run.smithi096.stdout: "get_started": 0, 2024-03-19T03:03:59.356 INFO:teuthology.orchestra.run.smithi096.stdout: "get": 233, 2024-03-19T03:03:59.356 INFO:teuthology.orchestra.run.smithi096.stdout: "get_sum": 1407545, 2024-03-19T03:03:59.356 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_fail": 0, 2024-03-19T03:03:59.356 INFO:teuthology.orchestra.run.smithi096.stdout: "get_or_fail_success": 233, 2024-03-19T03:03:59.356 INFO:teuthology.orchestra.run.smithi096.stdout: "take": 0, 2024-03-19T03:03:59.356 INFO:teuthology.orchestra.run.smithi096.stdout: "take_sum": 0, 2024-03-19T03:03:59.356 INFO:teuthology.orchestra.run.smithi096.stdout: "put": 233, 2024-03-19T03:03:59.356 INFO:teuthology.orchestra.run.smithi096.stdout: "put_sum": 1407545, 2024-03-19T03:03:59.357 INFO:teuthology.orchestra.run.smithi096.stdout: "wait": { 2024-03-19T03:03:59.357 INFO:teuthology.orchestra.run.smithi096.stdout: "avgcount": 0, 2024-03-19T03:03:59.357 INFO:teuthology.orchestra.run.smithi096.stdout: "sum": 0.000000000, 2024-03-19T03:03:59.357 INFO:teuthology.orchestra.run.smithi096.stdout: "avgtime": 0.000000000 2024-03-19T03:03:59.357 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-03-19T03:03:59.357 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-03-19T03:03:59.357 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-03-19T03:04:00.078 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-03-19T03:04:00.078 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-03-19T03:04:00.078 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2024-03-19T03:04:00.078 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2024-03-19T03:04:00.078 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-03-19T03:04:00.078 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-03-19T03:04:00.078 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-03-19T03:04:00.078 DEBUG:tasks.check_counter:Getting stats from c 2024-03-19T03:04:00.079 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-19T03:04:00.354 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T03:04:01.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:00 smithi096 ceph-mon[35946]: pgmap v327: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:04:01.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:00 smithi155 ceph-mon[35804]: pgmap v327: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:04:01.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:00 smithi033 ceph-mon[31973]: pgmap v327: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-19T03:04:02.085 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-03-19T03:04:02.085 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T03:04:02.085 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 1010, 2024-03-19T03:04:02.085 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 999, 2024-03-19T03:04:02.085 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 1723888, 2024-03-19T03:04:02.086 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 379578, 2024-03-19T03:04:02.086 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 5, 2024-03-19T03:04:02.086 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 1, 2024-03-19T03:04:02.086 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.088299210, 2024-03-19T03:04:02.086 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.025126460, 2024-03-19T03:04:02.086 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.043724762, 2024-03-19T03:04:02.086 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.023144783, 2024-03-19T03:04:02.086 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:04:02.086 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 998, 2024-03-19T03:04:02.086 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.031163661, 2024-03-19T03:04:02.086 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000031226 2024-03-19T03:04:02.086 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.086 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:04:02.086 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.087 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.087 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.087 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.087 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 1467520, 2024-03-19T03:04:02.087 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 136944 2024-03-19T03:04:02.087 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.087 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T03:04:02.087 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 4, 2024-03-19T03:04:02.087 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 168, 2024-03-19T03:04:02.087 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 876, 2024-03-19T03:04:02.087 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 448058, 2024-03-19T03:04:02.087 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 6, 2024-03-19T03:04:02.087 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 4, 2024-03-19T03:04:02.087 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.007937809, 2024-03-19T03:04:02.088 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.004528393, 2024-03-19T03:04:02.088 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.001925019, 2024-03-19T03:04:02.088 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.000069119, 2024-03-19T03:04:02.088 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:04:02.088 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 165, 2024-03-19T03:04:02.088 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.011038171, 2024-03-19T03:04:02.088 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000066898 2024-03-19T03:04:02.088 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.088 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:04:02.088 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.088 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.088 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.088 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.089 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-19T03:04:02.089 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 447360 2024-03-19T03:04:02.089 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.089 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T03:04:02.089 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 1425, 2024-03-19T03:04:02.089 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 1425, 2024-03-19T03:04:02.089 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 367556, 2024-03-19T03:04:02.089 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 452333, 2024-03-19T03:04:02.089 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 4, 2024-03-19T03:04:02.089 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 3, 2024-03-19T03:04:02.089 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.071131120, 2024-03-19T03:04:02.089 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.021005984, 2024-03-19T03:04:02.089 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.018309713, 2024-03-19T03:04:02.090 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.027468755, 2024-03-19T03:04:02.090 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:04:02.090 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 1424, 2024-03-19T03:04:02.090 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.031763036, 2024-03-19T03:04:02.090 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000022305 2024-03-19T03:04:02.090 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.090 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:04:02.090 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.090 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.090 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.090 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.091 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-19T03:04:02.091 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 448 2024-03-19T03:04:02.091 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.091 INFO:teuthology.orchestra.run.smithi155.stdout: "cct": { 2024-03-19T03:04:02.091 INFO:teuthology.orchestra.run.smithi155.stdout: "total_workers": 1, 2024-03-19T03:04:02.091 INFO:teuthology.orchestra.run.smithi155.stdout: "unhealthy_workers": 0 2024-03-19T03:04:02.091 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.091 INFO:teuthology.orchestra.run.smithi155.stdout: "finisher-MDSRank": { 2024-03-19T03:04:02.091 INFO:teuthology.orchestra.run.smithi155.stdout: "queue_len": 0, 2024-03-19T03:04:02.091 INFO:teuthology.orchestra.run.smithi155.stdout: "complete_latency": { 2024-03-19T03:04:02.091 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 3713, 2024-03-19T03:04:02.091 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.071379622, 2024-03-19T03:04:02.091 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000019224 2024-03-19T03:04:02.091 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-19T03:04:02.092 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.092 INFO:teuthology.orchestra.run.smithi155.stdout: "finisher-PurgeQueue": { 2024-03-19T03:04:02.092 INFO:teuthology.orchestra.run.smithi155.stdout: "queue_len": 0, 2024-03-19T03:04:02.092 INFO:teuthology.orchestra.run.smithi155.stdout: "complete_latency": { 2024-03-19T03:04:02.092 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.092 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.092 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.092 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-19T03:04:02.092 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.092 INFO:teuthology.orchestra.run.smithi155.stdout: "mds": { 2024-03-19T03:04:02.092 INFO:teuthology.orchestra.run.smithi155.stdout: "request": 0, 2024-03-19T03:04:02.092 INFO:teuthology.orchestra.run.smithi155.stdout: "reply": 0, 2024-03-19T03:04:02.092 INFO:teuthology.orchestra.run.smithi155.stdout: "reply_latency": { 2024-03-19T03:04:02.092 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.093 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.093 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.093 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.093 INFO:teuthology.orchestra.run.smithi155.stdout: "slow_reply": 0, 2024-03-19T03:04:02.093 INFO:teuthology.orchestra.run.smithi155.stdout: "forward": 0, 2024-03-19T03:04:02.093 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_fetch_complete": 0, 2024-03-19T03:04:02.093 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_fetch_keys": 0, 2024-03-19T03:04:02.093 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_commit": 0, 2024-03-19T03:04:02.093 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_split": 0, 2024-03-19T03:04:02.093 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_merge": 0, 2024-03-19T03:04:02.093 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes": 0, 2024-03-19T03:04:02.094 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_top": 0, 2024-03-19T03:04:02.094 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_bottom": 0, 2024-03-19T03:04:02.094 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_pin_tail": 0, 2024-03-19T03:04:02.094 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_pinned": 0, 2024-03-19T03:04:02.094 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_expired": 0, 2024-03-19T03:04:02.094 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_with_caps": 0, 2024-03-19T03:04:02.094 INFO:teuthology.orchestra.run.smithi155.stdout: "caps": 0, 2024-03-19T03:04:02.094 INFO:teuthology.orchestra.run.smithi155.stdout: "subtrees": 1, 2024-03-19T03:04:02.095 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse": 0, 2024-03-19T03:04:02.095 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_hit": 0, 2024-03-19T03:04:02.095 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_forward": 0, 2024-03-19T03:04:02.095 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_discover": 0, 2024-03-19T03:04:02.095 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_dir_fetch": 0, 2024-03-19T03:04:02.095 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_remote_ino": 0, 2024-03-19T03:04:02.095 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_lock": 0, 2024-03-19T03:04:02.095 INFO:teuthology.orchestra.run.smithi155.stdout: "load_cent": 0, 2024-03-19T03:04:02.095 INFO:teuthology.orchestra.run.smithi155.stdout: "q": 0, 2024-03-19T03:04:02.095 INFO:teuthology.orchestra.run.smithi155.stdout: "exported": 0, 2024-03-19T03:04:02.095 INFO:teuthology.orchestra.run.smithi155.stdout: "exported_inodes": 0, 2024-03-19T03:04:02.095 INFO:teuthology.orchestra.run.smithi155.stdout: "imported": 0, 2024-03-19T03:04:02.095 INFO:teuthology.orchestra.run.smithi155.stdout: "imported_inodes": 0, 2024-03-19T03:04:02.096 INFO:teuthology.orchestra.run.smithi155.stdout: "openino_dir_fetch": 0, 2024-03-19T03:04:02.096 INFO:teuthology.orchestra.run.smithi155.stdout: "openino_backtrace_fetch": 0, 2024-03-19T03:04:02.096 INFO:teuthology.orchestra.run.smithi155.stdout: "openino_peer_discover": 0, 2024-03-19T03:04:02.096 INFO:teuthology.orchestra.run.smithi155.stdout: "root_rfiles": 0, 2024-03-19T03:04:02.096 INFO:teuthology.orchestra.run.smithi155.stdout: "root_rbytes": 0, 2024-03-19T03:04:02.096 INFO:teuthology.orchestra.run.smithi155.stdout: "root_rsnaps": 0, 2024-03-19T03:04:02.096 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_backtrace_fetch": 0, 2024-03-19T03:04:02.096 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_set_tag": 0, 2024-03-19T03:04:02.096 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T03:04:02.096 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_inotable_repaired": 0, 2024-03-19T03:04:02.096 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_dir_inodes": 0, 2024-03-19T03:04:02.096 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T03:04:02.096 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T03:04:02.096 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_file_inodes": 0, 2024-03-19T03:04:02.096 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_inode_file_caps": 0, 2024-03-19T03:04:02.097 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_revoke": 0, 2024-03-19T03:04:02.097 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_grant": 0, 2024-03-19T03:04:02.097 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T03:04:02.097 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T03:04:02.097 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T03:04:02.097 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_caps": 0, 2024-03-19T03:04:02.097 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_caps_dirty": 0, 2024-03-19T03:04:02.097 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_cap_release": 0, 2024-03-19T03:04:02.097 INFO:teuthology.orchestra.run.smithi155.stdout: "process_request_cap_release": 0 2024-03-19T03:04:02.097 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.097 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_cache": { 2024-03-19T03:04:02.097 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_update": 0, 2024-03-19T03:04:02.097 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_update_receipt": 0, 2024-03-19T03:04:02.097 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_try_discover": 0, 2024-03-19T03:04:02.098 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_send_discover": 0, 2024-03-19T03:04:02.098 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_handle_discover": 0, 2024-03-19T03:04:02.098 INFO:teuthology.orchestra.run.smithi155.stdout: "num_strays": 0, 2024-03-19T03:04:02.098 INFO:teuthology.orchestra.run.smithi155.stdout: "num_strays_delayed": 0, 2024-03-19T03:04:02.098 INFO:teuthology.orchestra.run.smithi155.stdout: "num_strays_enqueuing": 0, 2024-03-19T03:04:02.098 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_created": 0, 2024-03-19T03:04:02.098 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_enqueued": 0, 2024-03-19T03:04:02.098 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_reintegrated": 0, 2024-03-19T03:04:02.098 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_migrated": 0, 2024-03-19T03:04:02.098 INFO:teuthology.orchestra.run.smithi155.stdout: "num_recovering_processing": 0, 2024-03-19T03:04:02.098 INFO:teuthology.orchestra.run.smithi155.stdout: "num_recovering_enqueued": 0, 2024-03-19T03:04:02.099 INFO:teuthology.orchestra.run.smithi155.stdout: "num_recovering_prioritized": 0, 2024-03-19T03:04:02.099 INFO:teuthology.orchestra.run.smithi155.stdout: "recovery_started": 0, 2024-03-19T03:04:02.099 INFO:teuthology.orchestra.run.smithi155.stdout: "recovery_completed": 0, 2024-03-19T03:04:02.099 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_quiesce_path": 0, 2024-03-19T03:04:02.099 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_quiesce_inode": 0, 2024-03-19T03:04:02.099 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T03:04:02.099 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_exportdir": 0, 2024-03-19T03:04:02.099 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_flush": 0, 2024-03-19T03:04:02.099 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_fragmentdir": 0, 2024-03-19T03:04:02.099 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_fragstats": 0, 2024-03-19T03:04:02.099 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_inodestats": 0 2024-03-19T03:04:02.099 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.099 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_log": { 2024-03-19T03:04:02.099 INFO:teuthology.orchestra.run.smithi155.stdout: "evlrg": 0, 2024-03-19T03:04:02.100 INFO:teuthology.orchestra.run.smithi155.stdout: "evadd": 0, 2024-03-19T03:04:02.100 INFO:teuthology.orchestra.run.smithi155.stdout: "evex": 0, 2024-03-19T03:04:02.100 INFO:teuthology.orchestra.run.smithi155.stdout: "evtrm": 0, 2024-03-19T03:04:02.100 INFO:teuthology.orchestra.run.smithi155.stdout: "ev": 7, 2024-03-19T03:04:02.100 INFO:teuthology.orchestra.run.smithi155.stdout: "evexg": 0, 2024-03-19T03:04:02.100 INFO:teuthology.orchestra.run.smithi155.stdout: "evexd": 0, 2024-03-19T03:04:02.100 INFO:teuthology.orchestra.run.smithi155.stdout: "segadd": 0, 2024-03-19T03:04:02.100 INFO:teuthology.orchestra.run.smithi155.stdout: "segex": 0, 2024-03-19T03:04:02.100 INFO:teuthology.orchestra.run.smithi155.stdout: "segtrm": 0, 2024-03-19T03:04:02.100 INFO:teuthology.orchestra.run.smithi155.stdout: "seg": 2, 2024-03-19T03:04:02.100 INFO:teuthology.orchestra.run.smithi155.stdout: "segmjr": 2, 2024-03-19T03:04:02.100 INFO:teuthology.orchestra.run.smithi155.stdout: "segexg": 0, 2024-03-19T03:04:02.100 INFO:teuthology.orchestra.run.smithi155.stdout: "segexd": 0, 2024-03-19T03:04:02.100 INFO:teuthology.orchestra.run.smithi155.stdout: "expos": 4194304, 2024-03-19T03:04:02.100 INFO:teuthology.orchestra.run.smithi155.stdout: "wrpos": 4196449, 2024-03-19T03:04:02.101 INFO:teuthology.orchestra.run.smithi155.stdout: "rdpos": 4196308, 2024-03-19T03:04:02.101 INFO:teuthology.orchestra.run.smithi155.stdout: "jlat": { 2024-03-19T03:04:02.101 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.101 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.101 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.101 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.101 INFO:teuthology.orchestra.run.smithi155.stdout: "replayed": 7 2024-03-19T03:04:02.101 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.101 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_mem": { 2024-03-19T03:04:02.101 INFO:teuthology.orchestra.run.smithi155.stdout: "ino": 3, 2024-03-19T03:04:02.101 INFO:teuthology.orchestra.run.smithi155.stdout: "ino+": 3, 2024-03-19T03:04:02.101 INFO:teuthology.orchestra.run.smithi155.stdout: "ino-": 0, 2024-03-19T03:04:02.101 INFO:teuthology.orchestra.run.smithi155.stdout: "dir": 1, 2024-03-19T03:04:02.101 INFO:teuthology.orchestra.run.smithi155.stdout: "dir+": 1, 2024-03-19T03:04:02.101 INFO:teuthology.orchestra.run.smithi155.stdout: "dir-": 0, 2024-03-19T03:04:02.101 INFO:teuthology.orchestra.run.smithi155.stdout: "dn": 0, 2024-03-19T03:04:02.102 INFO:teuthology.orchestra.run.smithi155.stdout: "dn+": 0, 2024-03-19T03:04:02.102 INFO:teuthology.orchestra.run.smithi155.stdout: "dn-": 0, 2024-03-19T03:04:02.102 INFO:teuthology.orchestra.run.smithi155.stdout: "cap": 0, 2024-03-19T03:04:02.102 INFO:teuthology.orchestra.run.smithi155.stdout: "cap+": 0, 2024-03-19T03:04:02.102 INFO:teuthology.orchestra.run.smithi155.stdout: "cap-": 0, 2024-03-19T03:04:02.102 INFO:teuthology.orchestra.run.smithi155.stdout: "rss": 50436, 2024-03-19T03:04:02.102 INFO:teuthology.orchestra.run.smithi155.stdout: "heap": 215420 2024-03-19T03:04:02.102 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.102 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_server": { 2024-03-19T03:04:02.102 INFO:teuthology.orchestra.run.smithi155.stdout: "dispatch_client_request": 0, 2024-03-19T03:04:02.102 INFO:teuthology.orchestra.run.smithi155.stdout: "dispatch_server_request": 0, 2024-03-19T03:04:02.102 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_request": 0, 2024-03-19T03:04:02.102 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_session": 0, 2024-03-19T03:04:02.102 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_peer_request": 0, 2024-03-19T03:04:02.102 INFO:teuthology.orchestra.run.smithi155.stdout: "req_create_latency": { 2024-03-19T03:04:02.103 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.103 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.103 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.103 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.103 INFO:teuthology.orchestra.run.smithi155.stdout: "req_getattr_latency": { 2024-03-19T03:04:02.103 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.103 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.103 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.103 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.103 INFO:teuthology.orchestra.run.smithi155.stdout: "req_getfilelock_latency": { 2024-03-19T03:04:02.103 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.103 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.103 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.103 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.103 INFO:teuthology.orchestra.run.smithi155.stdout: "req_link_latency": { 2024-03-19T03:04:02.103 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.104 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.104 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.104 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.104 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookup_latency": { 2024-03-19T03:04:02.104 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.104 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.104 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.104 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.104 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookuphash_latency": { 2024-03-19T03:04:02.104 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.104 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.104 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.104 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.104 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupino_latency": { 2024-03-19T03:04:02.104 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.105 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.105 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.105 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.105 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupname_latency": { 2024-03-19T03:04:02.105 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.105 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.105 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.105 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.105 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupparent_latency": { 2024-03-19T03:04:02.105 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.105 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.105 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.105 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.105 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupsnap_latency": { 2024-03-19T03:04:02.105 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.106 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.106 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.106 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.106 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lssnap_latency": { 2024-03-19T03:04:02.106 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.106 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.106 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.106 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.106 INFO:teuthology.orchestra.run.smithi155.stdout: "req_mkdir_latency": { 2024-03-19T03:04:02.106 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.107 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.107 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.107 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.107 INFO:teuthology.orchestra.run.smithi155.stdout: "req_mknod_latency": { 2024-03-19T03:04:02.107 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.107 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.107 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.107 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.107 INFO:teuthology.orchestra.run.smithi155.stdout: "req_mksnap_latency": { 2024-03-19T03:04:02.107 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.108 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.108 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.108 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.108 INFO:teuthology.orchestra.run.smithi155.stdout: "req_open_latency": { 2024-03-19T03:04:02.108 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.108 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.108 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.108 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.108 INFO:teuthology.orchestra.run.smithi155.stdout: "req_readdir_latency": { 2024-03-19T03:04:02.108 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.108 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.108 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.108 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.108 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rename_latency": { 2024-03-19T03:04:02.108 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.109 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.109 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.109 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.109 INFO:teuthology.orchestra.run.smithi155.stdout: "req_renamesnap_latency": { 2024-03-19T03:04:02.109 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.109 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.109 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.109 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.109 INFO:teuthology.orchestra.run.smithi155.stdout: "req_snapdiff_latency": { 2024-03-19T03:04:02.109 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.109 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.109 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.109 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.109 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rmdir_latency": { 2024-03-19T03:04:02.109 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.110 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.110 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.110 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.110 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rmsnap_latency": { 2024-03-19T03:04:02.110 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.110 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.110 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.110 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.110 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rmxattr_latency": { 2024-03-19T03:04:02.110 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.110 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.110 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.110 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.110 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setattr_latency": { 2024-03-19T03:04:02.111 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.111 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.111 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.111 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.111 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setdirlayout_latency": { 2024-03-19T03:04:02.111 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.111 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.111 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.111 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.111 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setfilelock_latency": { 2024-03-19T03:04:02.111 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.111 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.111 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.111 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.111 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setlayout_latency": { 2024-03-19T03:04:02.111 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.112 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.112 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.112 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.112 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setxattr_latency": { 2024-03-19T03:04:02.112 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.112 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.112 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.112 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.112 INFO:teuthology.orchestra.run.smithi155.stdout: "req_symlink_latency": { 2024-03-19T03:04:02.112 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.112 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.112 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.112 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.112 INFO:teuthology.orchestra.run.smithi155.stdout: "req_unlink_latency": { 2024-03-19T03:04:02.112 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.113 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.113 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.113 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.113 INFO:teuthology.orchestra.run.smithi155.stdout: "cap_revoke_eviction": 0, 2024-03-19T03:04:02.113 INFO:teuthology.orchestra.run.smithi155.stdout: "cap_acquisition_throttle": 0, 2024-03-19T03:04:02.113 INFO:teuthology.orchestra.run.smithi155.stdout: "req_getvxattr_latency": { 2024-03-19T03:04:02.113 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.113 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.113 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.113 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-19T03:04:02.113 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.113 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_sessions": { 2024-03-19T03:04:02.113 INFO:teuthology.orchestra.run.smithi155.stdout: "session_count": 1, 2024-03-19T03:04:02.113 INFO:teuthology.orchestra.run.smithi155.stdout: "session_add": 3, 2024-03-19T03:04:02.113 INFO:teuthology.orchestra.run.smithi155.stdout: "session_remove": 2, 2024-03-19T03:04:02.114 INFO:teuthology.orchestra.run.smithi155.stdout: "sessions_open": 3, 2024-03-19T03:04:02.114 INFO:teuthology.orchestra.run.smithi155.stdout: "sessions_stale": 0, 2024-03-19T03:04:02.114 INFO:teuthology.orchestra.run.smithi155.stdout: "total_load": 0, 2024-03-19T03:04:02.114 INFO:teuthology.orchestra.run.smithi155.stdout: "average_load": 0, 2024-03-19T03:04:02.114 INFO:teuthology.orchestra.run.smithi155.stdout: "avg_session_uptime": 7088, 2024-03-19T03:04:02.114 INFO:teuthology.orchestra.run.smithi155.stdout: "mdthresh_evicted": 0 2024-03-19T03:04:02.114 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.114 INFO:teuthology.orchestra.run.smithi155.stdout: "mempool": { 2024-03-19T03:04:02.114 INFO:teuthology.orchestra.run.smithi155.stdout: "bloom_filter_bytes": 0, 2024-03-19T03:04:02.114 INFO:teuthology.orchestra.run.smithi155.stdout: "bloom_filter_items": 0, 2024-03-19T03:04:02.114 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T03:04:02.114 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_alloc_items": 0, 2024-03-19T03:04:02.114 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T03:04:02.114 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_data_items": 0, 2024-03-19T03:04:02.114 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T03:04:02.115 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T03:04:02.115 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T03:04:02.115 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T03:04:02.115 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T03:04:02.115 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_other_items": 0, 2024-03-19T03:04:02.115 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T03:04:02.115 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T03:04:02.115 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_extent_bytes": 0, 2024-03-19T03:04:02.115 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_extent_items": 0, 2024-03-19T03:04:02.115 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_blob_bytes": 0, 2024-03-19T03:04:02.115 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_blob_items": 0, 2024-03-19T03:04:02.115 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T03:04:02.115 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T03:04:02.115 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T03:04:02.115 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T03:04:02.116 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T03:04:02.116 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_fsck_items": 0, 2024-03-19T03:04:02.116 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_txc_bytes": 0, 2024-03-19T03:04:02.116 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_txc_items": 0, 2024-03-19T03:04:02.116 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T03:04:02.116 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T03:04:02.116 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_bytes": 0, 2024-03-19T03:04:02.116 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_items": 0, 2024-03-19T03:04:02.116 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_bytes": 0, 2024-03-19T03:04:02.116 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_items": 0, 2024-03-19T03:04:02.116 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T03:04:02.116 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_reader_items": 0, 2024-03-19T03:04:02.116 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T03:04:02.116 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_writer_items": 0, 2024-03-19T03:04:02.116 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_anon_bytes": 181303, 2024-03-19T03:04:02.116 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_anon_items": 46, 2024-03-19T03:04:02.117 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_meta_bytes": 0, 2024-03-19T03:04:02.117 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_meta_items": 0, 2024-03-19T03:04:02.117 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_bytes": 0, 2024-03-19T03:04:02.117 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_items": 0, 2024-03-19T03:04:02.117 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_mapbl_bytes": 0, 2024-03-19T03:04:02.117 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_mapbl_items": 0, 2024-03-19T03:04:02.117 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_pglog_bytes": 0, 2024-03-19T03:04:02.117 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_pglog_items": 0, 2024-03-19T03:04:02.117 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_bytes": 9512, 2024-03-19T03:04:02.117 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_items": 182, 2024-03-19T03:04:02.117 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T03:04:02.117 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_mapping_items": 0, 2024-03-19T03:04:02.117 INFO:teuthology.orchestra.run.smithi155.stdout: "pgmap_bytes": 0, 2024-03-19T03:04:02.117 INFO:teuthology.orchestra.run.smithi155.stdout: "pgmap_items": 0, 2024-03-19T03:04:02.117 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_co_bytes": 8464, 2024-03-19T03:04:02.118 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_co_items": 13, 2024-03-19T03:04:02.118 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_1_bytes": 0, 2024-03-19T03:04:02.118 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_1_items": 0, 2024-03-19T03:04:02.118 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_2_bytes": 0, 2024-03-19T03:04:02.118 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_2_items": 0 2024-03-19T03:04:02.118 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.118 INFO:teuthology.orchestra.run.smithi155.stdout: "objecter": { 2024-03-19T03:04:02.118 INFO:teuthology.orchestra.run.smithi155.stdout: "op_active": 0, 2024-03-19T03:04:02.118 INFO:teuthology.orchestra.run.smithi155.stdout: "op_laggy": 0, 2024-03-19T03:04:02.118 INFO:teuthology.orchestra.run.smithi155.stdout: "op_send": 2186, 2024-03-19T03:04:02.118 INFO:teuthology.orchestra.run.smithi155.stdout: "op_send_bytes": 21, 2024-03-19T03:04:02.118 INFO:teuthology.orchestra.run.smithi155.stdout: "op_resend": 3, 2024-03-19T03:04:02.118 INFO:teuthology.orchestra.run.smithi155.stdout: "op_reply": 2186, 2024-03-19T03:04:02.118 INFO:teuthology.orchestra.run.smithi155.stdout: "op_latency": { 2024-03-19T03:04:02.119 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 2186, 2024-03-19T03:04:02.119 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.923981353, 2024-03-19T03:04:02.119 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000422681 2024-03-19T03:04:02.119 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.119 INFO:teuthology.orchestra.run.smithi155.stdout: "op_inflight": 0, 2024-03-19T03:04:02.119 INFO:teuthology.orchestra.run.smithi155.stdout: "oplen_avg": { 2024-03-19T03:04:02.119 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 2186, 2024-03-19T03:04:02.119 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 2187 2024-03-19T03:04:02.119 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.119 INFO:teuthology.orchestra.run.smithi155.stdout: "op": 2186, 2024-03-19T03:04:02.119 INFO:teuthology.orchestra.run.smithi155.stdout: "op_r": 2186, 2024-03-19T03:04:02.119 INFO:teuthology.orchestra.run.smithi155.stdout: "op_w": 0, 2024-03-19T03:04:02.119 INFO:teuthology.orchestra.run.smithi155.stdout: "op_rmw": 0, 2024-03-19T03:04:02.119 INFO:teuthology.orchestra.run.smithi155.stdout: "op_pg": 0, 2024-03-19T03:04:02.120 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_stat": 1414, 2024-03-19T03:04:02.120 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_create": 0, 2024-03-19T03:04:02.120 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_read": 770, 2024-03-19T03:04:02.120 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_write": 0, 2024-03-19T03:04:02.120 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_writefull": 0, 2024-03-19T03:04:02.120 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_writesame": 0, 2024-03-19T03:04:02.120 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_append": 0, 2024-03-19T03:04:02.120 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_zero": 0, 2024-03-19T03:04:02.120 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_truncate": 0, 2024-03-19T03:04:02.120 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_delete": 0, 2024-03-19T03:04:02.120 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_mapext": 0, 2024-03-19T03:04:02.120 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_sparse_read": 0, 2024-03-19T03:04:02.120 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_clonerange": 0, 2024-03-19T03:04:02.120 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_getxattr": 1, 2024-03-19T03:04:02.120 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_setxattr": 0, 2024-03-19T03:04:02.120 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_cmpxattr": 0, 2024-03-19T03:04:02.121 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_rmxattr": 0, 2024-03-19T03:04:02.121 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_resetxattrs": 0, 2024-03-19T03:04:02.121 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_call": 0, 2024-03-19T03:04:02.121 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_watch": 0, 2024-03-19T03:04:02.121 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_notify": 0, 2024-03-19T03:04:02.121 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T03:04:02.121 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_pgls": 0, 2024-03-19T03:04:02.121 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_pgls_filter": 0, 2024-03-19T03:04:02.121 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_other": 0, 2024-03-19T03:04:02.121 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_active": 0, 2024-03-19T03:04:02.121 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_send": 0, 2024-03-19T03:04:02.121 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_resend": 0, 2024-03-19T03:04:02.121 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_ping": 0, 2024-03-19T03:04:02.121 INFO:teuthology.orchestra.run.smithi155.stdout: "poolop_active": 0, 2024-03-19T03:04:02.121 INFO:teuthology.orchestra.run.smithi155.stdout: "poolop_send": 0, 2024-03-19T03:04:02.122 INFO:teuthology.orchestra.run.smithi155.stdout: "poolop_resend": 0, 2024-03-19T03:04:02.122 INFO:teuthology.orchestra.run.smithi155.stdout: "poolstat_active": 0, 2024-03-19T03:04:02.122 INFO:teuthology.orchestra.run.smithi155.stdout: "poolstat_send": 0, 2024-03-19T03:04:02.122 INFO:teuthology.orchestra.run.smithi155.stdout: "poolstat_resend": 0, 2024-03-19T03:04:02.122 INFO:teuthology.orchestra.run.smithi155.stdout: "statfs_active": 0, 2024-03-19T03:04:02.122 INFO:teuthology.orchestra.run.smithi155.stdout: "statfs_send": 0, 2024-03-19T03:04:02.122 INFO:teuthology.orchestra.run.smithi155.stdout: "statfs_resend": 0, 2024-03-19T03:04:02.122 INFO:teuthology.orchestra.run.smithi155.stdout: "command_active": 0, 2024-03-19T03:04:02.122 INFO:teuthology.orchestra.run.smithi155.stdout: "command_send": 0, 2024-03-19T03:04:02.122 INFO:teuthology.orchestra.run.smithi155.stdout: "command_resend": 0, 2024-03-19T03:04:02.122 INFO:teuthology.orchestra.run.smithi155.stdout: "map_epoch": 106, 2024-03-19T03:04:02.122 INFO:teuthology.orchestra.run.smithi155.stdout: "map_full": 0, 2024-03-19T03:04:02.122 INFO:teuthology.orchestra.run.smithi155.stdout: "map_inc": 13, 2024-03-19T03:04:02.122 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_sessions": 4, 2024-03-19T03:04:02.123 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_session_open": 4, 2024-03-19T03:04:02.123 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_session_close": 0, 2024-03-19T03:04:02.123 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_laggy": 0, 2024-03-19T03:04:02.123 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_wr": 0, 2024-03-19T03:04:02.123 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_rd": 2, 2024-03-19T03:04:02.123 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_del": 0 2024-03-19T03:04:02.123 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.123 INFO:teuthology.orchestra.run.smithi155.stdout: "oft": { 2024-03-19T03:04:02.123 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_objs": 0, 2024-03-19T03:04:02.123 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_kv_pairs": 0, 2024-03-19T03:04:02.123 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_updates": 0, 2024-03-19T03:04:02.123 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_removes": 0 2024-03-19T03:04:02.123 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.123 INFO:teuthology.orchestra.run.smithi155.stdout: "osd-slow-ops": { 2024-03-19T03:04:02.123 INFO:teuthology.orchestra.run.smithi155.stdout: "slow_ops_count": 0 2024-03-19T03:04:02.124 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.124 INFO:teuthology.orchestra.run.smithi155.stdout: "purge_queue": { 2024-03-19T03:04:02.124 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing_ops": 0, 2024-03-19T03:04:02.124 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing_ops_high_water": 0, 2024-03-19T03:04:02.124 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing": 0, 2024-03-19T03:04:02.124 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing_high_water": 0, 2024-03-19T03:04:02.124 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executed_ops": 0, 2024-03-19T03:04:02.124 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executed": 0, 2024-03-19T03:04:02.124 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_item_in_journal": 0 2024-03-19T03:04:02.124 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.124 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T03:04:02.124 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-19T03:04:02.124 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 104857600, 2024-03-19T03:04:02.124 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-19T03:04:02.124 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 2435, 2024-03-19T03:04:02.125 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 1857529, 2024-03-19T03:04:02.125 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-19T03:04:02.125 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 2435, 2024-03-19T03:04:02.125 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2024-03-19T03:04:02.125 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2024-03-19T03:04:02.125 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 2435, 2024-03-19T03:04:02.125 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 1857529, 2024-03-19T03:04:02.125 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-19T03:04:02.125 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.125 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.125 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.125 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-19T03:04:02.125 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.125 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-objecter_bytes": { 2024-03-19T03:04:02.125 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-19T03:04:02.126 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 104857600, 2024-03-19T03:04:02.126 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-19T03:04:02.126 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2024-03-19T03:04:02.126 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2024-03-19T03:04:02.126 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-19T03:04:02.126 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2024-03-19T03:04:02.126 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 2186, 2024-03-19T03:04:02.126 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 2150, 2024-03-19T03:04:02.126 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 2186, 2024-03-19T03:04:02.126 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 2150, 2024-03-19T03:04:02.126 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-19T03:04:02.126 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.126 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.126 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.126 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-19T03:04:02.127 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.127 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-objecter_ops": { 2024-03-19T03:04:02.127 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-19T03:04:02.127 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 1024, 2024-03-19T03:04:02.127 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-19T03:04:02.127 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2024-03-19T03:04:02.127 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2024-03-19T03:04:02.127 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-19T03:04:02.127 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2024-03-19T03:04:02.127 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 2186, 2024-03-19T03:04:02.127 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 2186, 2024-03-19T03:04:02.127 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 2186, 2024-03-19T03:04:02.127 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 2186, 2024-03-19T03:04:02.127 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-19T03:04:02.127 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.128 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.128 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.128 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-19T03:04:02.128 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.128 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-write_buf_throttle": { 2024-03-19T03:04:02.128 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-19T03:04:02.128 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 3758096384, 2024-03-19T03:04:02.128 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-19T03:04:02.128 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2024-03-19T03:04:02.128 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2024-03-19T03:04:02.128 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-19T03:04:02.128 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2024-03-19T03:04:02.128 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2024-03-19T03:04:02.128 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2024-03-19T03:04:02.128 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 0, 2024-03-19T03:04:02.128 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 0, 2024-03-19T03:04:02.129 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-19T03:04:02.129 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.129 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.129 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.129 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-19T03:04:02.129 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:02.129 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-write_buf_throttle-0x55e4a1fce6e0": { 2024-03-19T03:04:02.129 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-19T03:04:02.129 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 3758096384, 2024-03-19T03:04:02.129 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-19T03:04:02.129 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2024-03-19T03:04:02.129 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2024-03-19T03:04:02.129 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-19T03:04:02.129 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2024-03-19T03:04:02.129 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2024-03-19T03:04:02.130 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2024-03-19T03:04:02.130 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 0, 2024-03-19T03:04:02.130 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 0, 2024-03-19T03:04:02.130 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-19T03:04:02.130 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:02.130 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:02.130 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:02.130 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-19T03:04:02.130 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-19T03:04:02.130 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-03-19T03:04:02.854 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-19T03:04:02.854 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-03-19T03:04:02.854 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-03-19T03:04:02.854 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2024-03-19T03:04:02.854 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=0 2024-03-19T03:04:02.854 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-03-19T03:04:02.854 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-03-19T03:04:02.854 DEBUG:tasks.check_counter:Getting stats from f 2024-03-19T03:04:02.855 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-19T03:04:03.114 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T03:04:03.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:02 smithi096 ceph-mon[35946]: pgmap v328: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:04:03.236 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:02 smithi155 ceph-mon[35804]: pgmap v328: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:04:03.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:02 smithi033 ceph-mon[31973]: pgmap v328: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-19T03:04:04.775 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-03-19T03:04:04.775 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T03:04:04.775 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 253, 2024-03-19T03:04:04.776 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 240, 2024-03-19T03:04:04.776 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 1471618, 2024-03-19T03:04:04.776 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 138356, 2024-03-19T03:04:04.776 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 5, 2024-03-19T03:04:04.776 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 1, 2024-03-19T03:04:04.776 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.035036478, 2024-03-19T03:04:04.776 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.008952459, 2024-03-19T03:04:04.776 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.031628980, 2024-03-19T03:04:04.776 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.003697920, 2024-03-19T03:04:04.776 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:04:04.776 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 240, 2024-03-19T03:04:04.776 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.009105455, 2024-03-19T03:04:04.776 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000037939 2024-03-19T03:04:04.776 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.776 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:04:04.776 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.777 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.777 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.777 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.777 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 1470784, 2024-03-19T03:04:04.777 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 138032 2024-03-19T03:04:04.777 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.777 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T03:04:04.777 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 657, 2024-03-19T03:04:04.777 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 822, 2024-03-19T03:04:04.777 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 161472, 2024-03-19T03:04:04.777 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 657447, 2024-03-19T03:04:04.777 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 7, 2024-03-19T03:04:04.777 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 5, 2024-03-19T03:04:04.777 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.051243951, 2024-03-19T03:04:04.778 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.015856265, 2024-03-19T03:04:04.778 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.012045900, 2024-03-19T03:04:04.778 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.017772958, 2024-03-19T03:04:04.778 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:04:04.778 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 819, 2024-03-19T03:04:04.778 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.026275838, 2024-03-19T03:04:04.778 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000032082 2024-03-19T03:04:04.778 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.778 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:04:04.778 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.778 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.778 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.778 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.779 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-19T03:04:04.779 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 449152 2024-03-19T03:04:04.779 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.779 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T03:04:04.779 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 1537, 2024-03-19T03:04:04.779 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 1539, 2024-03-19T03:04:04.779 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 461295, 2024-03-19T03:04:04.779 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 488824, 2024-03-19T03:04:04.779 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 4, 2024-03-19T03:04:04.779 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 3, 2024-03-19T03:04:04.779 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.092912842, 2024-03-19T03:04:04.779 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.026706124, 2024-03-19T03:04:04.779 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.025688626, 2024-03-19T03:04:04.779 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.032352249, 2024-03-19T03:04:04.780 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:04:04.780 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 1537, 2024-03-19T03:04:04.780 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.037818181, 2024-03-19T03:04:04.780 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000024605 2024-03-19T03:04:04.780 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.780 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:04:04.780 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.780 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.780 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.780 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.780 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-19T03:04:04.780 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 0 2024-03-19T03:04:04.780 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.780 INFO:teuthology.orchestra.run.smithi155.stdout: "cct": { 2024-03-19T03:04:04.781 INFO:teuthology.orchestra.run.smithi155.stdout: "total_workers": 1, 2024-03-19T03:04:04.781 INFO:teuthology.orchestra.run.smithi155.stdout: "unhealthy_workers": 0 2024-03-19T03:04:04.781 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.781 INFO:teuthology.orchestra.run.smithi155.stdout: "finisher-MDSRank": { 2024-03-19T03:04:04.781 INFO:teuthology.orchestra.run.smithi155.stdout: "queue_len": 0, 2024-03-19T03:04:04.781 INFO:teuthology.orchestra.run.smithi155.stdout: "complete_latency": { 2024-03-19T03:04:04.781 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 3616, 2024-03-19T03:04:04.781 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.068575338, 2024-03-19T03:04:04.781 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000018964 2024-03-19T03:04:04.781 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-19T03:04:04.781 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.781 INFO:teuthology.orchestra.run.smithi155.stdout: "finisher-PurgeQueue": { 2024-03-19T03:04:04.781 INFO:teuthology.orchestra.run.smithi155.stdout: "queue_len": 0, 2024-03-19T03:04:04.781 INFO:teuthology.orchestra.run.smithi155.stdout: "complete_latency": { 2024-03-19T03:04:04.781 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.781 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.782 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.782 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-19T03:04:04.782 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.782 INFO:teuthology.orchestra.run.smithi155.stdout: "mds": { 2024-03-19T03:04:04.782 INFO:teuthology.orchestra.run.smithi155.stdout: "request": 0, 2024-03-19T03:04:04.782 INFO:teuthology.orchestra.run.smithi155.stdout: "reply": 0, 2024-03-19T03:04:04.782 INFO:teuthology.orchestra.run.smithi155.stdout: "reply_latency": { 2024-03-19T03:04:04.782 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.782 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.782 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.782 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.782 INFO:teuthology.orchestra.run.smithi155.stdout: "slow_reply": 0, 2024-03-19T03:04:04.782 INFO:teuthology.orchestra.run.smithi155.stdout: "forward": 0, 2024-03-19T03:04:04.782 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_fetch_complete": 0, 2024-03-19T03:04:04.783 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_fetch_keys": 0, 2024-03-19T03:04:04.783 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_commit": 0, 2024-03-19T03:04:04.783 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_split": 0, 2024-03-19T03:04:04.783 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_merge": 0, 2024-03-19T03:04:04.783 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes": 0, 2024-03-19T03:04:04.783 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_top": 0, 2024-03-19T03:04:04.783 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_bottom": 0, 2024-03-19T03:04:04.783 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_pin_tail": 0, 2024-03-19T03:04:04.783 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_pinned": 0, 2024-03-19T03:04:04.783 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_expired": 0, 2024-03-19T03:04:04.783 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_with_caps": 0, 2024-03-19T03:04:04.783 INFO:teuthology.orchestra.run.smithi155.stdout: "caps": 0, 2024-03-19T03:04:04.783 INFO:teuthology.orchestra.run.smithi155.stdout: "subtrees": 1, 2024-03-19T03:04:04.783 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse": 0, 2024-03-19T03:04:04.783 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_hit": 0, 2024-03-19T03:04:04.784 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_forward": 0, 2024-03-19T03:04:04.784 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_discover": 0, 2024-03-19T03:04:04.784 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_dir_fetch": 0, 2024-03-19T03:04:04.784 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_remote_ino": 0, 2024-03-19T03:04:04.784 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_lock": 0, 2024-03-19T03:04:04.784 INFO:teuthology.orchestra.run.smithi155.stdout: "load_cent": 0, 2024-03-19T03:04:04.784 INFO:teuthology.orchestra.run.smithi155.stdout: "q": 0, 2024-03-19T03:04:04.784 INFO:teuthology.orchestra.run.smithi155.stdout: "exported": 0, 2024-03-19T03:04:04.784 INFO:teuthology.orchestra.run.smithi155.stdout: "exported_inodes": 0, 2024-03-19T03:04:04.784 INFO:teuthology.orchestra.run.smithi155.stdout: "imported": 0, 2024-03-19T03:04:04.784 INFO:teuthology.orchestra.run.smithi155.stdout: "imported_inodes": 0, 2024-03-19T03:04:04.784 INFO:teuthology.orchestra.run.smithi155.stdout: "openino_dir_fetch": 0, 2024-03-19T03:04:04.784 INFO:teuthology.orchestra.run.smithi155.stdout: "openino_backtrace_fetch": 0, 2024-03-19T03:04:04.784 INFO:teuthology.orchestra.run.smithi155.stdout: "openino_peer_discover": 0, 2024-03-19T03:04:04.784 INFO:teuthology.orchestra.run.smithi155.stdout: "root_rfiles": 0, 2024-03-19T03:04:04.784 INFO:teuthology.orchestra.run.smithi155.stdout: "root_rbytes": 0, 2024-03-19T03:04:04.785 INFO:teuthology.orchestra.run.smithi155.stdout: "root_rsnaps": 0, 2024-03-19T03:04:04.785 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_backtrace_fetch": 0, 2024-03-19T03:04:04.785 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_set_tag": 0, 2024-03-19T03:04:04.785 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T03:04:04.785 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_inotable_repaired": 0, 2024-03-19T03:04:04.785 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_dir_inodes": 0, 2024-03-19T03:04:04.785 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T03:04:04.785 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T03:04:04.785 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_file_inodes": 0, 2024-03-19T03:04:04.785 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_inode_file_caps": 0, 2024-03-19T03:04:04.785 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_revoke": 0, 2024-03-19T03:04:04.785 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_grant": 0, 2024-03-19T03:04:04.785 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T03:04:04.785 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T03:04:04.785 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T03:04:04.785 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_caps": 0, 2024-03-19T03:04:04.786 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_caps_dirty": 0, 2024-03-19T03:04:04.786 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_cap_release": 0, 2024-03-19T03:04:04.786 INFO:teuthology.orchestra.run.smithi155.stdout: "process_request_cap_release": 0 2024-03-19T03:04:04.786 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.786 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_cache": { 2024-03-19T03:04:04.786 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_update": 0, 2024-03-19T03:04:04.786 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_update_receipt": 0, 2024-03-19T03:04:04.786 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_try_discover": 0, 2024-03-19T03:04:04.786 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_send_discover": 0, 2024-03-19T03:04:04.786 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_handle_discover": 0, 2024-03-19T03:04:04.786 INFO:teuthology.orchestra.run.smithi155.stdout: "num_strays": 0, 2024-03-19T03:04:04.786 INFO:teuthology.orchestra.run.smithi155.stdout: "num_strays_delayed": 0, 2024-03-19T03:04:04.786 INFO:teuthology.orchestra.run.smithi155.stdout: "num_strays_enqueuing": 0, 2024-03-19T03:04:04.786 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_created": 0, 2024-03-19T03:04:04.787 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_enqueued": 0, 2024-03-19T03:04:04.787 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_reintegrated": 0, 2024-03-19T03:04:04.787 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_migrated": 0, 2024-03-19T03:04:04.787 INFO:teuthology.orchestra.run.smithi155.stdout: "num_recovering_processing": 0, 2024-03-19T03:04:04.787 INFO:teuthology.orchestra.run.smithi155.stdout: "num_recovering_enqueued": 0, 2024-03-19T03:04:04.787 INFO:teuthology.orchestra.run.smithi155.stdout: "num_recovering_prioritized": 0, 2024-03-19T03:04:04.787 INFO:teuthology.orchestra.run.smithi155.stdout: "recovery_started": 0, 2024-03-19T03:04:04.787 INFO:teuthology.orchestra.run.smithi155.stdout: "recovery_completed": 0, 2024-03-19T03:04:04.787 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_quiesce_path": 0, 2024-03-19T03:04:04.787 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_quiesce_inode": 0, 2024-03-19T03:04:04.787 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T03:04:04.787 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_exportdir": 0, 2024-03-19T03:04:04.787 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_flush": 0, 2024-03-19T03:04:04.787 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_fragmentdir": 0, 2024-03-19T03:04:04.787 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_fragstats": 0, 2024-03-19T03:04:04.787 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_inodestats": 0 2024-03-19T03:04:04.788 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.788 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_log": { 2024-03-19T03:04:04.788 INFO:teuthology.orchestra.run.smithi155.stdout: "evlrg": 0, 2024-03-19T03:04:04.788 INFO:teuthology.orchestra.run.smithi155.stdout: "evadd": 0, 2024-03-19T03:04:04.788 INFO:teuthology.orchestra.run.smithi155.stdout: "evex": 0, 2024-03-19T03:04:04.788 INFO:teuthology.orchestra.run.smithi155.stdout: "evtrm": 0, 2024-03-19T03:04:04.788 INFO:teuthology.orchestra.run.smithi155.stdout: "ev": 6, 2024-03-19T03:04:04.788 INFO:teuthology.orchestra.run.smithi155.stdout: "evexg": 0, 2024-03-19T03:04:04.788 INFO:teuthology.orchestra.run.smithi155.stdout: "evexd": 0, 2024-03-19T03:04:04.788 INFO:teuthology.orchestra.run.smithi155.stdout: "segadd": 0, 2024-03-19T03:04:04.788 INFO:teuthology.orchestra.run.smithi155.stdout: "segex": 0, 2024-03-19T03:04:04.788 INFO:teuthology.orchestra.run.smithi155.stdout: "segtrm": 0, 2024-03-19T03:04:04.788 INFO:teuthology.orchestra.run.smithi155.stdout: "seg": 2, 2024-03-19T03:04:04.788 INFO:teuthology.orchestra.run.smithi155.stdout: "segmjr": 2, 2024-03-19T03:04:04.788 INFO:teuthology.orchestra.run.smithi155.stdout: "segexg": 0, 2024-03-19T03:04:04.789 INFO:teuthology.orchestra.run.smithi155.stdout: "segexd": 0, 2024-03-19T03:04:04.789 INFO:teuthology.orchestra.run.smithi155.stdout: "expos": 4194304, 2024-03-19T03:04:04.789 INFO:teuthology.orchestra.run.smithi155.stdout: "wrpos": 4196308, 2024-03-19T03:04:04.789 INFO:teuthology.orchestra.run.smithi155.stdout: "rdpos": 4196167, 2024-03-19T03:04:04.789 INFO:teuthology.orchestra.run.smithi155.stdout: "jlat": { 2024-03-19T03:04:04.789 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.789 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.789 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.789 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.789 INFO:teuthology.orchestra.run.smithi155.stdout: "replayed": 6 2024-03-19T03:04:04.789 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.789 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_mem": { 2024-03-19T03:04:04.789 INFO:teuthology.orchestra.run.smithi155.stdout: "ino": 3, 2024-03-19T03:04:04.789 INFO:teuthology.orchestra.run.smithi155.stdout: "ino+": 3, 2024-03-19T03:04:04.790 INFO:teuthology.orchestra.run.smithi155.stdout: "ino-": 0, 2024-03-19T03:04:04.790 INFO:teuthology.orchestra.run.smithi155.stdout: "dir": 1, 2024-03-19T03:04:04.790 INFO:teuthology.orchestra.run.smithi155.stdout: "dir+": 1, 2024-03-19T03:04:04.790 INFO:teuthology.orchestra.run.smithi155.stdout: "dir-": 0, 2024-03-19T03:04:04.790 INFO:teuthology.orchestra.run.smithi155.stdout: "dn": 0, 2024-03-19T03:04:04.790 INFO:teuthology.orchestra.run.smithi155.stdout: "dn+": 0, 2024-03-19T03:04:04.790 INFO:teuthology.orchestra.run.smithi155.stdout: "dn-": 0, 2024-03-19T03:04:04.790 INFO:teuthology.orchestra.run.smithi155.stdout: "cap": 0, 2024-03-19T03:04:04.790 INFO:teuthology.orchestra.run.smithi155.stdout: "cap+": 0, 2024-03-19T03:04:04.790 INFO:teuthology.orchestra.run.smithi155.stdout: "cap-": 0, 2024-03-19T03:04:04.790 INFO:teuthology.orchestra.run.smithi155.stdout: "rss": 50588, 2024-03-19T03:04:04.790 INFO:teuthology.orchestra.run.smithi155.stdout: "heap": 215420 2024-03-19T03:04:04.790 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.791 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_server": { 2024-03-19T03:04:04.791 INFO:teuthology.orchestra.run.smithi155.stdout: "dispatch_client_request": 0, 2024-03-19T03:04:04.791 INFO:teuthology.orchestra.run.smithi155.stdout: "dispatch_server_request": 0, 2024-03-19T03:04:04.791 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_request": 0, 2024-03-19T03:04:04.791 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_session": 0, 2024-03-19T03:04:04.791 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_peer_request": 0, 2024-03-19T03:04:04.791 INFO:teuthology.orchestra.run.smithi155.stdout: "req_create_latency": { 2024-03-19T03:04:04.791 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.791 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.791 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.791 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.791 INFO:teuthology.orchestra.run.smithi155.stdout: "req_getattr_latency": { 2024-03-19T03:04:04.791 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.791 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.791 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.792 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.792 INFO:teuthology.orchestra.run.smithi155.stdout: "req_getfilelock_latency": { 2024-03-19T03:04:04.792 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.792 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.792 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.792 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.792 INFO:teuthology.orchestra.run.smithi155.stdout: "req_link_latency": { 2024-03-19T03:04:04.792 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.792 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.792 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.792 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.792 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookup_latency": { 2024-03-19T03:04:04.792 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.792 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.792 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.793 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.793 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookuphash_latency": { 2024-03-19T03:04:04.793 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.793 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.793 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.793 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.793 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupino_latency": { 2024-03-19T03:04:04.793 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.793 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.793 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.793 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.793 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupname_latency": { 2024-03-19T03:04:04.793 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.793 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.793 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.794 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.794 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupparent_latency": { 2024-03-19T03:04:04.794 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.794 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.794 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.794 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.794 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupsnap_latency": { 2024-03-19T03:04:04.794 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.794 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.794 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.794 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.794 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lssnap_latency": { 2024-03-19T03:04:04.794 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.794 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.795 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.795 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.795 INFO:teuthology.orchestra.run.smithi155.stdout: "req_mkdir_latency": { 2024-03-19T03:04:04.795 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.795 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.795 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.796 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.796 INFO:teuthology.orchestra.run.smithi155.stdout: "req_mknod_latency": { 2024-03-19T03:04:04.796 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.796 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.796 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.796 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.796 INFO:teuthology.orchestra.run.smithi155.stdout: "req_mksnap_latency": { 2024-03-19T03:04:04.796 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.796 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.796 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.796 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.796 INFO:teuthology.orchestra.run.smithi155.stdout: "req_open_latency": { 2024-03-19T03:04:04.796 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.796 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.796 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.796 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.797 INFO:teuthology.orchestra.run.smithi155.stdout: "req_readdir_latency": { 2024-03-19T03:04:04.797 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.797 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.797 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.797 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.797 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rename_latency": { 2024-03-19T03:04:04.797 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.797 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.797 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.797 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.797 INFO:teuthology.orchestra.run.smithi155.stdout: "req_renamesnap_latency": { 2024-03-19T03:04:04.797 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.797 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.797 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.797 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.798 INFO:teuthology.orchestra.run.smithi155.stdout: "req_snapdiff_latency": { 2024-03-19T03:04:04.798 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.798 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.798 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.798 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.798 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rmdir_latency": { 2024-03-19T03:04:04.798 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.798 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.798 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.798 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.798 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rmsnap_latency": { 2024-03-19T03:04:04.798 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.798 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.798 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.799 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.799 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rmxattr_latency": { 2024-03-19T03:04:04.799 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.799 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.799 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.799 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.799 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setattr_latency": { 2024-03-19T03:04:04.799 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.799 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.799 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.799 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.799 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setdirlayout_latency": { 2024-03-19T03:04:04.799 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.799 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.800 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.800 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.800 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setfilelock_latency": { 2024-03-19T03:04:04.800 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.800 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.800 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.800 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.800 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setlayout_latency": { 2024-03-19T03:04:04.800 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.800 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.800 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.800 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.800 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setxattr_latency": { 2024-03-19T03:04:04.800 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.800 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.801 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.801 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.801 INFO:teuthology.orchestra.run.smithi155.stdout: "req_symlink_latency": { 2024-03-19T03:04:04.801 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.801 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.801 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.801 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.801 INFO:teuthology.orchestra.run.smithi155.stdout: "req_unlink_latency": { 2024-03-19T03:04:04.801 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.801 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.801 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.801 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.801 INFO:teuthology.orchestra.run.smithi155.stdout: "cap_revoke_eviction": 0, 2024-03-19T03:04:04.801 INFO:teuthology.orchestra.run.smithi155.stdout: "cap_acquisition_throttle": 0, 2024-03-19T03:04:04.801 INFO:teuthology.orchestra.run.smithi155.stdout: "req_getvxattr_latency": { 2024-03-19T03:04:04.801 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.801 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.802 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.802 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-19T03:04:04.802 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.802 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_sessions": { 2024-03-19T03:04:04.802 INFO:teuthology.orchestra.run.smithi155.stdout: "session_count": 2, 2024-03-19T03:04:04.802 INFO:teuthology.orchestra.run.smithi155.stdout: "session_add": 3, 2024-03-19T03:04:04.802 INFO:teuthology.orchestra.run.smithi155.stdout: "session_remove": 1, 2024-03-19T03:04:04.802 INFO:teuthology.orchestra.run.smithi155.stdout: "sessions_open": 3, 2024-03-19T03:04:04.802 INFO:teuthology.orchestra.run.smithi155.stdout: "sessions_stale": 0, 2024-03-19T03:04:04.802 INFO:teuthology.orchestra.run.smithi155.stdout: "total_load": 0, 2024-03-19T03:04:04.802 INFO:teuthology.orchestra.run.smithi155.stdout: "average_load": 0, 2024-03-19T03:04:04.802 INFO:teuthology.orchestra.run.smithi155.stdout: "avg_session_uptime": 3862, 2024-03-19T03:04:04.802 INFO:teuthology.orchestra.run.smithi155.stdout: "mdthresh_evicted": 0 2024-03-19T03:04:04.802 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.803 INFO:teuthology.orchestra.run.smithi155.stdout: "mempool": { 2024-03-19T03:04:04.803 INFO:teuthology.orchestra.run.smithi155.stdout: "bloom_filter_bytes": 0, 2024-03-19T03:04:04.803 INFO:teuthology.orchestra.run.smithi155.stdout: "bloom_filter_items": 0, 2024-03-19T03:04:04.803 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T03:04:04.803 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_alloc_items": 0, 2024-03-19T03:04:04.803 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T03:04:04.803 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_data_items": 0, 2024-03-19T03:04:04.803 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T03:04:04.803 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T03:04:04.803 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T03:04:04.803 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T03:04:04.803 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T03:04:04.803 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_other_items": 0, 2024-03-19T03:04:04.803 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T03:04:04.804 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T03:04:04.804 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_extent_bytes": 0, 2024-03-19T03:04:04.804 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_extent_items": 0, 2024-03-19T03:04:04.804 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_blob_bytes": 0, 2024-03-19T03:04:04.804 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_blob_items": 0, 2024-03-19T03:04:04.804 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T03:04:04.804 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T03:04:04.804 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T03:04:04.804 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T03:04:04.804 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T03:04:04.804 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_fsck_items": 0, 2024-03-19T03:04:04.804 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_txc_bytes": 0, 2024-03-19T03:04:04.804 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_txc_items": 0, 2024-03-19T03:04:04.804 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T03:04:04.805 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T03:04:04.805 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_bytes": 0, 2024-03-19T03:04:04.805 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_items": 0, 2024-03-19T03:04:04.805 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_bytes": 0, 2024-03-19T03:04:04.805 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_items": 0, 2024-03-19T03:04:04.805 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T03:04:04.805 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_reader_items": 0, 2024-03-19T03:04:04.805 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T03:04:04.805 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_writer_items": 0, 2024-03-19T03:04:04.805 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_anon_bytes": 189463, 2024-03-19T03:04:04.805 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_anon_items": 52, 2024-03-19T03:04:04.805 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_meta_bytes": 0, 2024-03-19T03:04:04.805 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_meta_items": 0, 2024-03-19T03:04:04.805 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_bytes": 0, 2024-03-19T03:04:04.805 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_items": 0, 2024-03-19T03:04:04.806 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_mapbl_bytes": 0, 2024-03-19T03:04:04.806 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_mapbl_items": 0, 2024-03-19T03:04:04.806 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_pglog_bytes": 0, 2024-03-19T03:04:04.806 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_pglog_items": 0, 2024-03-19T03:04:04.806 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_bytes": 9512, 2024-03-19T03:04:04.806 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_items": 182, 2024-03-19T03:04:04.806 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T03:04:04.806 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_mapping_items": 0, 2024-03-19T03:04:04.806 INFO:teuthology.orchestra.run.smithi155.stdout: "pgmap_bytes": 0, 2024-03-19T03:04:04.806 INFO:teuthology.orchestra.run.smithi155.stdout: "pgmap_items": 0, 2024-03-19T03:04:04.806 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_co_bytes": 8464, 2024-03-19T03:04:04.806 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_co_items": 13, 2024-03-19T03:04:04.806 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_1_bytes": 0, 2024-03-19T03:04:04.806 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_1_items": 0, 2024-03-19T03:04:04.806 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_2_bytes": 0, 2024-03-19T03:04:04.807 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_2_items": 0 2024-03-19T03:04:04.807 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.807 INFO:teuthology.orchestra.run.smithi155.stdout: "objecter": { 2024-03-19T03:04:04.807 INFO:teuthology.orchestra.run.smithi155.stdout: "op_active": 0, 2024-03-19T03:04:04.807 INFO:teuthology.orchestra.run.smithi155.stdout: "op_laggy": 0, 2024-03-19T03:04:04.807 INFO:teuthology.orchestra.run.smithi155.stdout: "op_send": 2193, 2024-03-19T03:04:04.807 INFO:teuthology.orchestra.run.smithi155.stdout: "op_send_bytes": 21, 2024-03-19T03:04:04.807 INFO:teuthology.orchestra.run.smithi155.stdout: "op_resend": 3, 2024-03-19T03:04:04.807 INFO:teuthology.orchestra.run.smithi155.stdout: "op_reply": 2193, 2024-03-19T03:04:04.807 INFO:teuthology.orchestra.run.smithi155.stdout: "op_latency": { 2024-03-19T03:04:04.807 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 2193, 2024-03-19T03:04:04.807 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 1.155976685, 2024-03-19T03:04:04.807 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000527121 2024-03-19T03:04:04.807 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.807 INFO:teuthology.orchestra.run.smithi155.stdout: "op_inflight": 0, 2024-03-19T03:04:04.808 INFO:teuthology.orchestra.run.smithi155.stdout: "oplen_avg": { 2024-03-19T03:04:04.808 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 2193, 2024-03-19T03:04:04.808 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 2194 2024-03-19T03:04:04.808 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.808 INFO:teuthology.orchestra.run.smithi155.stdout: "op": 2193, 2024-03-19T03:04:04.808 INFO:teuthology.orchestra.run.smithi155.stdout: "op_r": 2193, 2024-03-19T03:04:04.808 INFO:teuthology.orchestra.run.smithi155.stdout: "op_w": 0, 2024-03-19T03:04:04.808 INFO:teuthology.orchestra.run.smithi155.stdout: "op_rmw": 0, 2024-03-19T03:04:04.808 INFO:teuthology.orchestra.run.smithi155.stdout: "op_pg": 0, 2024-03-19T03:04:04.808 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_stat": 1418, 2024-03-19T03:04:04.808 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_create": 0, 2024-03-19T03:04:04.808 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_read": 773, 2024-03-19T03:04:04.808 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_write": 0, 2024-03-19T03:04:04.808 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_writefull": 0, 2024-03-19T03:04:04.808 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_writesame": 0, 2024-03-19T03:04:04.809 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_append": 0, 2024-03-19T03:04:04.809 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_zero": 0, 2024-03-19T03:04:04.809 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_truncate": 0, 2024-03-19T03:04:04.809 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_delete": 0, 2024-03-19T03:04:04.809 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_mapext": 0, 2024-03-19T03:04:04.809 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_sparse_read": 0, 2024-03-19T03:04:04.809 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_clonerange": 0, 2024-03-19T03:04:04.809 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_getxattr": 1, 2024-03-19T03:04:04.809 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_setxattr": 0, 2024-03-19T03:04:04.809 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_cmpxattr": 0, 2024-03-19T03:04:04.809 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_rmxattr": 0, 2024-03-19T03:04:04.809 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_resetxattrs": 0, 2024-03-19T03:04:04.809 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_call": 0, 2024-03-19T03:04:04.809 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_watch": 0, 2024-03-19T03:04:04.809 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_notify": 0, 2024-03-19T03:04:04.810 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T03:04:04.810 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_pgls": 0, 2024-03-19T03:04:04.810 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_pgls_filter": 0, 2024-03-19T03:04:04.810 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_other": 0, 2024-03-19T03:04:04.810 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_active": 0, 2024-03-19T03:04:04.810 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_send": 0, 2024-03-19T03:04:04.810 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_resend": 0, 2024-03-19T03:04:04.810 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_ping": 0, 2024-03-19T03:04:04.810 INFO:teuthology.orchestra.run.smithi155.stdout: "poolop_active": 0, 2024-03-19T03:04:04.810 INFO:teuthology.orchestra.run.smithi155.stdout: "poolop_send": 0, 2024-03-19T03:04:04.810 INFO:teuthology.orchestra.run.smithi155.stdout: "poolop_resend": 0, 2024-03-19T03:04:04.810 INFO:teuthology.orchestra.run.smithi155.stdout: "poolstat_active": 0, 2024-03-19T03:04:04.810 INFO:teuthology.orchestra.run.smithi155.stdout: "poolstat_send": 0, 2024-03-19T03:04:04.810 INFO:teuthology.orchestra.run.smithi155.stdout: "poolstat_resend": 0, 2024-03-19T03:04:04.810 INFO:teuthology.orchestra.run.smithi155.stdout: "statfs_active": 0, 2024-03-19T03:04:04.811 INFO:teuthology.orchestra.run.smithi155.stdout: "statfs_send": 0, 2024-03-19T03:04:04.811 INFO:teuthology.orchestra.run.smithi155.stdout: "statfs_resend": 0, 2024-03-19T03:04:04.811 INFO:teuthology.orchestra.run.smithi155.stdout: "command_active": 0, 2024-03-19T03:04:04.811 INFO:teuthology.orchestra.run.smithi155.stdout: "command_send": 0, 2024-03-19T03:04:04.811 INFO:teuthology.orchestra.run.smithi155.stdout: "command_resend": 0, 2024-03-19T03:04:04.811 INFO:teuthology.orchestra.run.smithi155.stdout: "map_epoch": 106, 2024-03-19T03:04:04.811 INFO:teuthology.orchestra.run.smithi155.stdout: "map_full": 0, 2024-03-19T03:04:04.811 INFO:teuthology.orchestra.run.smithi155.stdout: "map_inc": 13, 2024-03-19T03:04:04.811 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_sessions": 5, 2024-03-19T03:04:04.811 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_session_open": 5, 2024-03-19T03:04:04.811 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_session_close": 0, 2024-03-19T03:04:04.811 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_laggy": 0, 2024-03-19T03:04:04.811 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_wr": 0, 2024-03-19T03:04:04.811 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_rd": 2, 2024-03-19T03:04:04.811 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_del": 0 2024-03-19T03:04:04.811 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.812 INFO:teuthology.orchestra.run.smithi155.stdout: "oft": { 2024-03-19T03:04:04.812 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_objs": 0, 2024-03-19T03:04:04.812 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_kv_pairs": 0, 2024-03-19T03:04:04.812 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_updates": 0, 2024-03-19T03:04:04.812 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_removes": 0 2024-03-19T03:04:04.812 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.812 INFO:teuthology.orchestra.run.smithi155.stdout: "osd-slow-ops": { 2024-03-19T03:04:04.812 INFO:teuthology.orchestra.run.smithi155.stdout: "slow_ops_count": 0 2024-03-19T03:04:04.812 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.812 INFO:teuthology.orchestra.run.smithi155.stdout: "purge_queue": { 2024-03-19T03:04:04.812 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing_ops": 0, 2024-03-19T03:04:04.812 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing_ops_high_water": 0, 2024-03-19T03:04:04.812 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing": 0, 2024-03-19T03:04:04.812 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing_high_water": 0, 2024-03-19T03:04:04.812 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executed_ops": 0, 2024-03-19T03:04:04.812 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executed": 0, 2024-03-19T03:04:04.812 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_item_in_journal": 0 2024-03-19T03:04:04.813 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.813 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T03:04:04.813 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-19T03:04:04.813 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 104857600, 2024-03-19T03:04:04.813 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-19T03:04:04.813 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 2443, 2024-03-19T03:04:04.813 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 1858822, 2024-03-19T03:04:04.813 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-19T03:04:04.813 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 2443, 2024-03-19T03:04:04.813 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2024-03-19T03:04:04.813 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2024-03-19T03:04:04.813 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 2443, 2024-03-19T03:04:04.813 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 1858822, 2024-03-19T03:04:04.813 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-19T03:04:04.813 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.813 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.814 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.814 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-19T03:04:04.814 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.814 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-objecter_bytes": { 2024-03-19T03:04:04.814 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-19T03:04:04.814 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 104857600, 2024-03-19T03:04:04.814 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-19T03:04:04.814 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2024-03-19T03:04:04.814 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2024-03-19T03:04:04.814 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-19T03:04:04.814 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2024-03-19T03:04:04.814 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 2193, 2024-03-19T03:04:04.814 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 2009, 2024-03-19T03:04:04.814 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 2193, 2024-03-19T03:04:04.815 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 2009, 2024-03-19T03:04:04.815 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-19T03:04:04.815 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.815 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.815 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.815 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-19T03:04:04.815 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.815 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-objecter_ops": { 2024-03-19T03:04:04.815 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-19T03:04:04.815 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 1024, 2024-03-19T03:04:04.815 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-19T03:04:04.815 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2024-03-19T03:04:04.815 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2024-03-19T03:04:04.815 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-19T03:04:04.815 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2024-03-19T03:04:04.816 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 2193, 2024-03-19T03:04:04.816 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 2193, 2024-03-19T03:04:04.816 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 2193, 2024-03-19T03:04:04.816 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 2193, 2024-03-19T03:04:04.816 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-19T03:04:04.816 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.816 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.816 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.816 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-19T03:04:04.816 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.816 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-write_buf_throttle": { 2024-03-19T03:04:04.816 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-19T03:04:04.816 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 3758096384, 2024-03-19T03:04:04.816 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-19T03:04:04.816 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2024-03-19T03:04:04.816 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2024-03-19T03:04:04.817 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-19T03:04:04.817 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2024-03-19T03:04:04.817 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2024-03-19T03:04:04.817 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2024-03-19T03:04:04.817 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 0, 2024-03-19T03:04:04.817 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 0, 2024-03-19T03:04:04.817 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-19T03:04:04.817 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.817 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.817 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.817 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-19T03:04:04.817 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:04.817 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-write_buf_throttle-0x55ad372d46e0": { 2024-03-19T03:04:04.817 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-19T03:04:04.817 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 3758096384, 2024-03-19T03:04:04.818 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-19T03:04:04.818 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2024-03-19T03:04:04.818 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2024-03-19T03:04:04.818 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-19T03:04:04.818 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2024-03-19T03:04:04.818 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2024-03-19T03:04:04.818 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2024-03-19T03:04:04.818 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 0, 2024-03-19T03:04:04.818 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 0, 2024-03-19T03:04:04.818 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-19T03:04:04.818 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:04.818 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:04.818 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:04.818 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-19T03:04:04.818 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-19T03:04:04.819 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-03-19T03:04:05.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:04 smithi096 ceph-mon[35946]: pgmap v329: 129 pgs: 129 active+clean; 1.9 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-19T03:04:05.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:04 smithi155 ceph-mon[35804]: pgmap v329: 129 pgs: 129 active+clean; 1.9 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-19T03:04:05.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:04 smithi033 ceph-mon[31973]: pgmap v329: 129 pgs: 129 active+clean; 1.9 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-19T03:04:05.483 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-19T03:04:05.483 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-19T03:04:05.483 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-03-19T03:04:05.484 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-03-19T03:04:05.484 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-03-19T03:04:05.484 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-03-19T03:04:05.484 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-03-19T03:04:05.484 DEBUG:tasks.check_counter:Getting stats from i 2024-03-19T03:04:05.484 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-19T03:04:05.742 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T03:04:06.891 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:06 smithi155 ceph-mon[35804]: pgmap v330: 129 pgs: 129 active+clean; 1.9 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-19T03:04:07.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:06 smithi096 ceph-mon[35946]: pgmap v330: 129 pgs: 129 active+clean; 1.9 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-19T03:04:07.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:06 smithi033 ceph-mon[31973]: pgmap v330: 129 pgs: 129 active+clean; 1.9 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-19T03:04:07.490 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-03-19T03:04:07.490 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T03:04:07.490 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 58, 2024-03-19T03:04:07.490 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 106, 2024-03-19T03:04:07.490 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 15277, 2024-03-19T03:04:07.490 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 100119, 2024-03-19T03:04:07.491 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 15, 2024-03-19T03:04:07.491 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 4, 2024-03-19T03:04:07.491 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.009699543, 2024-03-19T03:04:07.491 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.001547349, 2024-03-19T03:04:07.491 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.004763759, 2024-03-19T03:04:07.491 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.000402973, 2024-03-19T03:04:07.491 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:04:07.491 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 105, 2024-03-19T03:04:07.491 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.019949922, 2024-03-19T03:04:07.491 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000189999 2024-03-19T03:04:07.491 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.491 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:04:07.491 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 95, 2024-03-19T03:04:07.491 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000012878, 2024-03-19T03:04:07.492 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000135 2024-03-19T03:04:07.492 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.492 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 6640, 2024-03-19T03:04:07.492 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 12784 2024-03-19T03:04:07.492 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.492 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T03:04:07.492 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 343, 2024-03-19T03:04:07.492 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 330, 2024-03-19T03:04:07.492 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 1499811, 2024-03-19T03:04:07.492 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 218085, 2024-03-19T03:04:07.492 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 10, 2024-03-19T03:04:07.492 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 5, 2024-03-19T03:04:07.492 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.039542559, 2024-03-19T03:04:07.493 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.010352842, 2024-03-19T03:04:07.493 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.034850661, 2024-03-19T03:04:07.493 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.004095805, 2024-03-19T03:04:07.493 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:04:07.493 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 329, 2024-03-19T03:04:07.493 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.020182657, 2024-03-19T03:04:07.493 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000061345 2024-03-19T03:04:07.493 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.493 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:04:07.493 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 102, 2024-03-19T03:04:07.493 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000034299, 2024-03-19T03:04:07.493 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000336 2024-03-19T03:04:07.493 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.493 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 1476736, 2024-03-19T03:04:07.494 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 140944 2024-03-19T03:04:07.494 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.494 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T03:04:07.494 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 519, 2024-03-19T03:04:07.494 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 787, 2024-03-19T03:04:07.494 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 152387, 2024-03-19T03:04:07.494 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 1141091, 2024-03-19T03:04:07.494 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 42, 2024-03-19T03:04:07.494 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 5, 2024-03-19T03:04:07.494 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.053115213, 2024-03-19T03:04:07.494 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.019964106, 2024-03-19T03:04:07.494 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.029359224, 2024-03-19T03:04:07.494 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.002432035, 2024-03-19T03:04:07.494 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:04:07.495 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 784, 2024-03-19T03:04:07.495 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.033138385, 2024-03-19T03:04:07.495 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000042268 2024-03-19T03:04:07.495 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.495 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:04:07.495 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 1084, 2024-03-19T03:04:07.495 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000176814, 2024-03-19T03:04:07.495 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000163 2024-03-19T03:04:07.495 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.495 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 1248, 2024-03-19T03:04:07.495 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 451568 2024-03-19T03:04:07.495 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.496 INFO:teuthology.orchestra.run.smithi155.stdout: "cct": { 2024-03-19T03:04:07.496 INFO:teuthology.orchestra.run.smithi155.stdout: "total_workers": 1, 2024-03-19T03:04:07.496 INFO:teuthology.orchestra.run.smithi155.stdout: "unhealthy_workers": 0 2024-03-19T03:04:07.496 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.496 INFO:teuthology.orchestra.run.smithi155.stdout: "finisher-MDSRank": { 2024-03-19T03:04:07.496 INFO:teuthology.orchestra.run.smithi155.stdout: "queue_len": 0, 2024-03-19T03:04:07.496 INFO:teuthology.orchestra.run.smithi155.stdout: "complete_latency": { 2024-03-19T03:04:07.496 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 52, 2024-03-19T03:04:07.496 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.002162517, 2024-03-19T03:04:07.496 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000041586 2024-03-19T03:04:07.496 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-19T03:04:07.496 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.496 INFO:teuthology.orchestra.run.smithi155.stdout: "finisher-PurgeQueue": { 2024-03-19T03:04:07.496 INFO:teuthology.orchestra.run.smithi155.stdout: "queue_len": 0, 2024-03-19T03:04:07.497 INFO:teuthology.orchestra.run.smithi155.stdout: "complete_latency": { 2024-03-19T03:04:07.497 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 2, 2024-03-19T03:04:07.497 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000014365, 2024-03-19T03:04:07.497 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000007182 2024-03-19T03:04:07.497 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-19T03:04:07.497 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.497 INFO:teuthology.orchestra.run.smithi155.stdout: "mds": { 2024-03-19T03:04:07.497 INFO:teuthology.orchestra.run.smithi155.stdout: "request": 6, 2024-03-19T03:04:07.497 INFO:teuthology.orchestra.run.smithi155.stdout: "reply": 3, 2024-03-19T03:04:07.498 INFO:teuthology.orchestra.run.smithi155.stdout: "reply_latency": { 2024-03-19T03:04:07.498 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 3, 2024-03-19T03:04:07.498 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000707186, 2024-03-19T03:04:07.498 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000235728 2024-03-19T03:04:07.498 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.498 INFO:teuthology.orchestra.run.smithi155.stdout: "slow_reply": 0, 2024-03-19T03:04:07.498 INFO:teuthology.orchestra.run.smithi155.stdout: "forward": 3, 2024-03-19T03:04:07.498 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_fetch_complete": 0, 2024-03-19T03:04:07.498 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_fetch_keys": 0, 2024-03-19T03:04:07.498 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_commit": 11, 2024-03-19T03:04:07.498 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_split": 0, 2024-03-19T03:04:07.498 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_merge": 0, 2024-03-19T03:04:07.498 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes": 16, 2024-03-19T03:04:07.499 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_top": 0, 2024-03-19T03:04:07.499 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_bottom": 16, 2024-03-19T03:04:07.499 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_pin_tail": 0, 2024-03-19T03:04:07.499 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_pinned": 13, 2024-03-19T03:04:07.499 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_expired": 4, 2024-03-19T03:04:07.499 INFO:teuthology.orchestra.run.smithi155.stdout: "inodes_with_caps": 1, 2024-03-19T03:04:07.499 INFO:teuthology.orchestra.run.smithi155.stdout: "caps": 1, 2024-03-19T03:04:07.499 INFO:teuthology.orchestra.run.smithi155.stdout: "subtrees": 3, 2024-03-19T03:04:07.499 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse": 14, 2024-03-19T03:04:07.499 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_hit": 10, 2024-03-19T03:04:07.499 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_forward": 0, 2024-03-19T03:04:07.499 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_discover": 3, 2024-03-19T03:04:07.499 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_dir_fetch": 0, 2024-03-19T03:04:07.500 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_remote_ino": 0, 2024-03-19T03:04:07.500 INFO:teuthology.orchestra.run.smithi155.stdout: "traverse_lock": 0, 2024-03-19T03:04:07.500 INFO:teuthology.orchestra.run.smithi155.stdout: "load_cent": 0, 2024-03-19T03:04:07.500 INFO:teuthology.orchestra.run.smithi155.stdout: "q": 0, 2024-03-19T03:04:07.500 INFO:teuthology.orchestra.run.smithi155.stdout: "exported": 0, 2024-03-19T03:04:07.500 INFO:teuthology.orchestra.run.smithi155.stdout: "exported_inodes": 0, 2024-03-19T03:04:07.500 INFO:teuthology.orchestra.run.smithi155.stdout: "imported": 0, 2024-03-19T03:04:07.500 INFO:teuthology.orchestra.run.smithi155.stdout: "imported_inodes": 0, 2024-03-19T03:04:07.500 INFO:teuthology.orchestra.run.smithi155.stdout: "openino_dir_fetch": 0, 2024-03-19T03:04:07.500 INFO:teuthology.orchestra.run.smithi155.stdout: "openino_backtrace_fetch": 0, 2024-03-19T03:04:07.500 INFO:teuthology.orchestra.run.smithi155.stdout: "openino_peer_discover": 0, 2024-03-19T03:04:07.500 INFO:teuthology.orchestra.run.smithi155.stdout: "root_rfiles": 4, 2024-03-19T03:04:07.501 INFO:teuthology.orchestra.run.smithi155.stdout: "root_rbytes": 234, 2024-03-19T03:04:07.501 INFO:teuthology.orchestra.run.smithi155.stdout: "root_rsnaps": 6, 2024-03-19T03:04:07.501 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_backtrace_fetch": 0, 2024-03-19T03:04:07.501 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_set_tag": 0, 2024-03-19T03:04:07.501 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T03:04:07.501 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_inotable_repaired": 0, 2024-03-19T03:04:07.501 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_dir_inodes": 0, 2024-03-19T03:04:07.501 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T03:04:07.501 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T03:04:07.501 INFO:teuthology.orchestra.run.smithi155.stdout: "scrub_file_inodes": 0, 2024-03-19T03:04:07.501 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_inode_file_caps": 0, 2024-03-19T03:04:07.501 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_revoke": 0, 2024-03-19T03:04:07.501 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_grant": 0, 2024-03-19T03:04:07.501 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T03:04:07.501 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T03:04:07.502 INFO:teuthology.orchestra.run.smithi155.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T03:04:07.502 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_caps": 0, 2024-03-19T03:04:07.502 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_caps_dirty": 0, 2024-03-19T03:04:07.502 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_cap_release": 0, 2024-03-19T03:04:07.502 INFO:teuthology.orchestra.run.smithi155.stdout: "process_request_cap_release": 0 2024-03-19T03:04:07.502 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.502 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_cache": { 2024-03-19T03:04:07.502 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_update": 0, 2024-03-19T03:04:07.502 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_update_receipt": 11, 2024-03-19T03:04:07.502 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_try_discover": 7, 2024-03-19T03:04:07.502 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_send_discover": 8, 2024-03-19T03:04:07.502 INFO:teuthology.orchestra.run.smithi155.stdout: "dir_handle_discover": 0, 2024-03-19T03:04:07.502 INFO:teuthology.orchestra.run.smithi155.stdout: "num_strays": 0, 2024-03-19T03:04:07.502 INFO:teuthology.orchestra.run.smithi155.stdout: "num_strays_delayed": 0, 2024-03-19T03:04:07.503 INFO:teuthology.orchestra.run.smithi155.stdout: "num_strays_enqueuing": 0, 2024-03-19T03:04:07.503 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_created": 0, 2024-03-19T03:04:07.503 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_enqueued": 0, 2024-03-19T03:04:07.503 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_reintegrated": 0, 2024-03-19T03:04:07.503 INFO:teuthology.orchestra.run.smithi155.stdout: "strays_migrated": 0, 2024-03-19T03:04:07.503 INFO:teuthology.orchestra.run.smithi155.stdout: "num_recovering_processing": 0, 2024-03-19T03:04:07.503 INFO:teuthology.orchestra.run.smithi155.stdout: "num_recovering_enqueued": 0, 2024-03-19T03:04:07.503 INFO:teuthology.orchestra.run.smithi155.stdout: "num_recovering_prioritized": 0, 2024-03-19T03:04:07.503 INFO:teuthology.orchestra.run.smithi155.stdout: "recovery_started": 0, 2024-03-19T03:04:07.503 INFO:teuthology.orchestra.run.smithi155.stdout: "recovery_completed": 0, 2024-03-19T03:04:07.503 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_quiesce_path": 0, 2024-03-19T03:04:07.503 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_quiesce_inode": 0, 2024-03-19T03:04:07.503 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T03:04:07.503 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_exportdir": 0, 2024-03-19T03:04:07.503 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_flush": 0, 2024-03-19T03:04:07.504 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_fragmentdir": 0, 2024-03-19T03:04:07.504 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_fragstats": 0, 2024-03-19T03:04:07.504 INFO:teuthology.orchestra.run.smithi155.stdout: "ireq_inodestats": 0 2024-03-19T03:04:07.504 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.504 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_log": { 2024-03-19T03:04:07.504 INFO:teuthology.orchestra.run.smithi155.stdout: "evlrg": 0, 2024-03-19T03:04:07.504 INFO:teuthology.orchestra.run.smithi155.stdout: "evadd": 6, 2024-03-19T03:04:07.504 INFO:teuthology.orchestra.run.smithi155.stdout: "evex": 0, 2024-03-19T03:04:07.504 INFO:teuthology.orchestra.run.smithi155.stdout: "evtrm": 0, 2024-03-19T03:04:07.504 INFO:teuthology.orchestra.run.smithi155.stdout: "ev": 6, 2024-03-19T03:04:07.504 INFO:teuthology.orchestra.run.smithi155.stdout: "evexg": 0, 2024-03-19T03:04:07.504 INFO:teuthology.orchestra.run.smithi155.stdout: "evexd": 0, 2024-03-19T03:04:07.504 INFO:teuthology.orchestra.run.smithi155.stdout: "segadd": 2, 2024-03-19T03:04:07.505 INFO:teuthology.orchestra.run.smithi155.stdout: "segex": 0, 2024-03-19T03:04:07.505 INFO:teuthology.orchestra.run.smithi155.stdout: "segtrm": 0, 2024-03-19T03:04:07.505 INFO:teuthology.orchestra.run.smithi155.stdout: "seg": 2, 2024-03-19T03:04:07.505 INFO:teuthology.orchestra.run.smithi155.stdout: "segmjr": 2, 2024-03-19T03:04:07.505 INFO:teuthology.orchestra.run.smithi155.stdout: "segexg": 0, 2024-03-19T03:04:07.505 INFO:teuthology.orchestra.run.smithi155.stdout: "segexd": 0, 2024-03-19T03:04:07.505 INFO:teuthology.orchestra.run.smithi155.stdout: "expos": 4194304, 2024-03-19T03:04:07.505 INFO:teuthology.orchestra.run.smithi155.stdout: "wrpos": 4196308, 2024-03-19T03:04:07.505 INFO:teuthology.orchestra.run.smithi155.stdout: "rdpos": 0, 2024-03-19T03:04:07.505 INFO:teuthology.orchestra.run.smithi155.stdout: "jlat": { 2024-03-19T03:04:07.505 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 5, 2024-03-19T03:04:07.505 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.010119288, 2024-03-19T03:04:07.505 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.002023857 2024-03-19T03:04:07.505 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.506 INFO:teuthology.orchestra.run.smithi155.stdout: "replayed": 0 2024-03-19T03:04:07.506 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.506 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_mem": { 2024-03-19T03:04:07.506 INFO:teuthology.orchestra.run.smithi155.stdout: "ino": 20, 2024-03-19T03:04:07.506 INFO:teuthology.orchestra.run.smithi155.stdout: "ino+": 22, 2024-03-19T03:04:07.506 INFO:teuthology.orchestra.run.smithi155.stdout: "ino-": 2, 2024-03-19T03:04:07.506 INFO:teuthology.orchestra.run.smithi155.stdout: "dir": 19, 2024-03-19T03:04:07.506 INFO:teuthology.orchestra.run.smithi155.stdout: "dir+": 22, 2024-03-19T03:04:07.506 INFO:teuthology.orchestra.run.smithi155.stdout: "dir-": 3, 2024-03-19T03:04:07.506 INFO:teuthology.orchestra.run.smithi155.stdout: "dn": 16, 2024-03-19T03:04:07.506 INFO:teuthology.orchestra.run.smithi155.stdout: "dn+": 20, 2024-03-19T03:04:07.506 INFO:teuthology.orchestra.run.smithi155.stdout: "dn-": 4, 2024-03-19T03:04:07.506 INFO:teuthology.orchestra.run.smithi155.stdout: "cap": 1, 2024-03-19T03:04:07.506 INFO:teuthology.orchestra.run.smithi155.stdout: "cap+": 1, 2024-03-19T03:04:07.507 INFO:teuthology.orchestra.run.smithi155.stdout: "cap-": 0, 2024-03-19T03:04:07.507 INFO:teuthology.orchestra.run.smithi155.stdout: "rss": 54980, 2024-03-19T03:04:07.507 INFO:teuthology.orchestra.run.smithi155.stdout: "heap": 223612 2024-03-19T03:04:07.507 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.507 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_server": { 2024-03-19T03:04:07.507 INFO:teuthology.orchestra.run.smithi155.stdout: "dispatch_client_request": 6, 2024-03-19T03:04:07.507 INFO:teuthology.orchestra.run.smithi155.stdout: "dispatch_server_request": 0, 2024-03-19T03:04:07.507 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_request": 6, 2024-03-19T03:04:07.508 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_client_session": 22, 2024-03-19T03:04:07.508 INFO:teuthology.orchestra.run.smithi155.stdout: "handle_peer_request": 0, 2024-03-19T03:04:07.508 INFO:teuthology.orchestra.run.smithi155.stdout: "req_create_latency": { 2024-03-19T03:04:07.508 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.508 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.508 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.508 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.508 INFO:teuthology.orchestra.run.smithi155.stdout: "req_getattr_latency": { 2024-03-19T03:04:07.508 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 3, 2024-03-19T03:04:07.508 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000707186, 2024-03-19T03:04:07.508 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000235728 2024-03-19T03:04:07.508 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.508 INFO:teuthology.orchestra.run.smithi155.stdout: "req_getfilelock_latency": { 2024-03-19T03:04:07.508 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.508 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.508 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.509 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.509 INFO:teuthology.orchestra.run.smithi155.stdout: "req_link_latency": { 2024-03-19T03:04:07.509 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.509 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.509 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.509 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.509 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookup_latency": { 2024-03-19T03:04:07.509 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.509 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.509 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.509 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.509 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookuphash_latency": { 2024-03-19T03:04:07.509 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.509 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.509 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.510 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.510 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupino_latency": { 2024-03-19T03:04:07.510 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.510 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.510 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.510 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.510 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupname_latency": { 2024-03-19T03:04:07.510 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.510 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.510 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.510 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.510 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupparent_latency": { 2024-03-19T03:04:07.510 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.510 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.511 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.511 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.511 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lookupsnap_latency": { 2024-03-19T03:04:07.511 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.511 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.511 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.511 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.511 INFO:teuthology.orchestra.run.smithi155.stdout: "req_lssnap_latency": { 2024-03-19T03:04:07.511 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.511 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.511 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.511 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.511 INFO:teuthology.orchestra.run.smithi155.stdout: "req_mkdir_latency": { 2024-03-19T03:04:07.512 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.512 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.512 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.512 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.512 INFO:teuthology.orchestra.run.smithi155.stdout: "req_mknod_latency": { 2024-03-19T03:04:07.512 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.512 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.512 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.513 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.513 INFO:teuthology.orchestra.run.smithi155.stdout: "req_mksnap_latency": { 2024-03-19T03:04:07.513 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.513 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.513 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.513 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.513 INFO:teuthology.orchestra.run.smithi155.stdout: "req_open_latency": { 2024-03-19T03:04:07.513 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.513 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.513 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.513 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.513 INFO:teuthology.orchestra.run.smithi155.stdout: "req_readdir_latency": { 2024-03-19T03:04:07.513 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.513 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.513 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.513 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.514 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rename_latency": { 2024-03-19T03:04:07.514 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.514 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.514 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.514 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.514 INFO:teuthology.orchestra.run.smithi155.stdout: "req_renamesnap_latency": { 2024-03-19T03:04:07.514 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.514 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.514 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.514 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.514 INFO:teuthology.orchestra.run.smithi155.stdout: "req_snapdiff_latency": { 2024-03-19T03:04:07.514 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.514 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.514 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.515 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.515 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rmdir_latency": { 2024-03-19T03:04:07.515 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.515 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.515 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.515 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.515 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rmsnap_latency": { 2024-03-19T03:04:07.515 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.515 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.515 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.515 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.515 INFO:teuthology.orchestra.run.smithi155.stdout: "req_rmxattr_latency": { 2024-03-19T03:04:07.515 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.515 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.516 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.516 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.516 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setattr_latency": { 2024-03-19T03:04:07.516 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.516 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.516 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.516 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.516 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setdirlayout_latency": { 2024-03-19T03:04:07.516 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.516 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.516 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.516 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.516 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setfilelock_latency": { 2024-03-19T03:04:07.516 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.516 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.516 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.517 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.517 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setlayout_latency": { 2024-03-19T03:04:07.517 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.517 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.517 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.517 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.517 INFO:teuthology.orchestra.run.smithi155.stdout: "req_setxattr_latency": { 2024-03-19T03:04:07.517 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.517 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.517 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.517 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.517 INFO:teuthology.orchestra.run.smithi155.stdout: "req_symlink_latency": { 2024-03-19T03:04:07.517 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.518 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.518 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.518 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.518 INFO:teuthology.orchestra.run.smithi155.stdout: "req_unlink_latency": { 2024-03-19T03:04:07.518 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.518 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.518 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.518 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.518 INFO:teuthology.orchestra.run.smithi155.stdout: "cap_revoke_eviction": 0, 2024-03-19T03:04:07.518 INFO:teuthology.orchestra.run.smithi155.stdout: "cap_acquisition_throttle": 0, 2024-03-19T03:04:07.518 INFO:teuthology.orchestra.run.smithi155.stdout: "req_getvxattr_latency": { 2024-03-19T03:04:07.518 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.518 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.518 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.519 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-19T03:04:07.519 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.519 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_sessions": { 2024-03-19T03:04:07.519 INFO:teuthology.orchestra.run.smithi155.stdout: "session_count": 2, 2024-03-19T03:04:07.519 INFO:teuthology.orchestra.run.smithi155.stdout: "session_add": 3, 2024-03-19T03:04:07.519 INFO:teuthology.orchestra.run.smithi155.stdout: "session_remove": 1, 2024-03-19T03:04:07.519 INFO:teuthology.orchestra.run.smithi155.stdout: "sessions_open": 2, 2024-03-19T03:04:07.519 INFO:teuthology.orchestra.run.smithi155.stdout: "sessions_stale": 0, 2024-03-19T03:04:07.519 INFO:teuthology.orchestra.run.smithi155.stdout: "total_load": 2, 2024-03-19T03:04:07.519 INFO:teuthology.orchestra.run.smithi155.stdout: "average_load": 0, 2024-03-19T03:04:07.519 INFO:teuthology.orchestra.run.smithi155.stdout: "avg_session_uptime": 628, 2024-03-19T03:04:07.519 INFO:teuthology.orchestra.run.smithi155.stdout: "mdthresh_evicted": 0 2024-03-19T03:04:07.519 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.519 INFO:teuthology.orchestra.run.smithi155.stdout: "mempool": { 2024-03-19T03:04:07.520 INFO:teuthology.orchestra.run.smithi155.stdout: "bloom_filter_bytes": 0, 2024-03-19T03:04:07.520 INFO:teuthology.orchestra.run.smithi155.stdout: "bloom_filter_items": 0, 2024-03-19T03:04:07.520 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T03:04:07.520 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_alloc_items": 0, 2024-03-19T03:04:07.520 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T03:04:07.520 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_data_items": 0, 2024-03-19T03:04:07.520 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T03:04:07.520 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T03:04:07.520 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T03:04:07.520 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T03:04:07.520 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T03:04:07.520 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_other_items": 0, 2024-03-19T03:04:07.520 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T03:04:07.520 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T03:04:07.520 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_extent_bytes": 0, 2024-03-19T03:04:07.521 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_extent_items": 0, 2024-03-19T03:04:07.521 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_blob_bytes": 0, 2024-03-19T03:04:07.521 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_blob_items": 0, 2024-03-19T03:04:07.521 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T03:04:07.521 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T03:04:07.521 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T03:04:07.521 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T03:04:07.521 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T03:04:07.521 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_fsck_items": 0, 2024-03-19T03:04:07.521 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_txc_bytes": 0, 2024-03-19T03:04:07.521 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_txc_items": 0, 2024-03-19T03:04:07.521 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T03:04:07.521 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T03:04:07.521 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_bytes": 0, 2024-03-19T03:04:07.521 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_items": 0, 2024-03-19T03:04:07.522 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_bytes": 0, 2024-03-19T03:04:07.522 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_items": 0, 2024-03-19T03:04:07.522 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T03:04:07.522 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_reader_items": 0, 2024-03-19T03:04:07.522 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T03:04:07.522 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_writer_items": 0, 2024-03-19T03:04:07.522 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_anon_bytes": 255607, 2024-03-19T03:04:07.522 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_anon_items": 93, 2024-03-19T03:04:07.522 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_meta_bytes": 176, 2024-03-19T03:04:07.522 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_meta_items": 2, 2024-03-19T03:04:07.522 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_bytes": 0, 2024-03-19T03:04:07.522 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_items": 0, 2024-03-19T03:04:07.522 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_mapbl_bytes": 0, 2024-03-19T03:04:07.522 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_mapbl_items": 0, 2024-03-19T03:04:07.523 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_pglog_bytes": 0, 2024-03-19T03:04:07.523 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_pglog_items": 0, 2024-03-19T03:04:07.523 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_bytes": 9512, 2024-03-19T03:04:07.523 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_items": 182, 2024-03-19T03:04:07.523 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T03:04:07.523 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_mapping_items": 0, 2024-03-19T03:04:07.523 INFO:teuthology.orchestra.run.smithi155.stdout: "pgmap_bytes": 0, 2024-03-19T03:04:07.523 INFO:teuthology.orchestra.run.smithi155.stdout: "pgmap_items": 0, 2024-03-19T03:04:07.523 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_co_bytes": 87213, 2024-03-19T03:04:07.523 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_co_items": 335, 2024-03-19T03:04:07.523 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_1_bytes": 0, 2024-03-19T03:04:07.523 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_1_items": 0, 2024-03-19T03:04:07.523 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_2_bytes": 0, 2024-03-19T03:04:07.523 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_2_items": 0 2024-03-19T03:04:07.523 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.524 INFO:teuthology.orchestra.run.smithi155.stdout: "objecter": { 2024-03-19T03:04:07.524 INFO:teuthology.orchestra.run.smithi155.stdout: "op_active": 0, 2024-03-19T03:04:07.524 INFO:teuthology.orchestra.run.smithi155.stdout: "op_laggy": 0, 2024-03-19T03:04:07.524 INFO:teuthology.orchestra.run.smithi155.stdout: "op_send": 43, 2024-03-19T03:04:07.524 INFO:teuthology.orchestra.run.smithi155.stdout: "op_send_bytes": 12486, 2024-03-19T03:04:07.524 INFO:teuthology.orchestra.run.smithi155.stdout: "op_resend": 28, 2024-03-19T03:04:07.524 INFO:teuthology.orchestra.run.smithi155.stdout: "op_reply": 43, 2024-03-19T03:04:07.524 INFO:teuthology.orchestra.run.smithi155.stdout: "op_latency": { 2024-03-19T03:04:07.524 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 43, 2024-03-19T03:04:07.524 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.219995575, 2024-03-19T03:04:07.524 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.005116176 2024-03-19T03:04:07.524 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.524 INFO:teuthology.orchestra.run.smithi155.stdout: "op_inflight": 0, 2024-03-19T03:04:07.525 INFO:teuthology.orchestra.run.smithi155.stdout: "oplen_avg": { 2024-03-19T03:04:07.525 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 43, 2024-03-19T03:04:07.525 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 65 2024-03-19T03:04:07.525 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.525 INFO:teuthology.orchestra.run.smithi155.stdout: "op": 43, 2024-03-19T03:04:07.525 INFO:teuthology.orchestra.run.smithi155.stdout: "op_r": 0, 2024-03-19T03:04:07.525 INFO:teuthology.orchestra.run.smithi155.stdout: "op_w": 43, 2024-03-19T03:04:07.525 INFO:teuthology.orchestra.run.smithi155.stdout: "op_rmw": 0, 2024-03-19T03:04:07.525 INFO:teuthology.orchestra.run.smithi155.stdout: "op_pg": 0, 2024-03-19T03:04:07.525 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_stat": 0, 2024-03-19T03:04:07.525 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_create": 10, 2024-03-19T03:04:07.525 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_read": 0, 2024-03-19T03:04:07.525 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_write": 5, 2024-03-19T03:04:07.525 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_writefull": 9, 2024-03-19T03:04:07.525 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_writesame": 0, 2024-03-19T03:04:07.526 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_append": 0, 2024-03-19T03:04:07.526 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_zero": 0, 2024-03-19T03:04:07.526 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_truncate": 0, 2024-03-19T03:04:07.526 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_delete": 6, 2024-03-19T03:04:07.526 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_mapext": 0, 2024-03-19T03:04:07.526 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_sparse_read": 0, 2024-03-19T03:04:07.526 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_clonerange": 0, 2024-03-19T03:04:07.526 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_getxattr": 0, 2024-03-19T03:04:07.526 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_setxattr": 20, 2024-03-19T03:04:07.526 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_cmpxattr": 0, 2024-03-19T03:04:07.526 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_rmxattr": 0, 2024-03-19T03:04:07.526 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_resetxattrs": 0, 2024-03-19T03:04:07.526 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_call": 0, 2024-03-19T03:04:07.526 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_watch": 0, 2024-03-19T03:04:07.526 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_notify": 0, 2024-03-19T03:04:07.527 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T03:04:07.527 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_pgls": 0, 2024-03-19T03:04:07.527 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_pgls_filter": 0, 2024-03-19T03:04:07.527 INFO:teuthology.orchestra.run.smithi155.stdout: "osdop_other": 0, 2024-03-19T03:04:07.527 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_active": 0, 2024-03-19T03:04:07.527 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_send": 0, 2024-03-19T03:04:07.527 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_resend": 0, 2024-03-19T03:04:07.527 INFO:teuthology.orchestra.run.smithi155.stdout: "linger_ping": 0, 2024-03-19T03:04:07.527 INFO:teuthology.orchestra.run.smithi155.stdout: "poolop_active": 0, 2024-03-19T03:04:07.527 INFO:teuthology.orchestra.run.smithi155.stdout: "poolop_send": 0, 2024-03-19T03:04:07.527 INFO:teuthology.orchestra.run.smithi155.stdout: "poolop_resend": 0, 2024-03-19T03:04:07.527 INFO:teuthology.orchestra.run.smithi155.stdout: "poolstat_active": 0, 2024-03-19T03:04:07.527 INFO:teuthology.orchestra.run.smithi155.stdout: "poolstat_send": 0, 2024-03-19T03:04:07.528 INFO:teuthology.orchestra.run.smithi155.stdout: "poolstat_resend": 0, 2024-03-19T03:04:07.528 INFO:teuthology.orchestra.run.smithi155.stdout: "statfs_active": 0, 2024-03-19T03:04:07.528 INFO:teuthology.orchestra.run.smithi155.stdout: "statfs_send": 0, 2024-03-19T03:04:07.528 INFO:teuthology.orchestra.run.smithi155.stdout: "statfs_resend": 0, 2024-03-19T03:04:07.528 INFO:teuthology.orchestra.run.smithi155.stdout: "command_active": 0, 2024-03-19T03:04:07.528 INFO:teuthology.orchestra.run.smithi155.stdout: "command_send": 0, 2024-03-19T03:04:07.528 INFO:teuthology.orchestra.run.smithi155.stdout: "command_resend": 0, 2024-03-19T03:04:07.528 INFO:teuthology.orchestra.run.smithi155.stdout: "map_epoch": 106, 2024-03-19T03:04:07.528 INFO:teuthology.orchestra.run.smithi155.stdout: "map_full": 0, 2024-03-19T03:04:07.528 INFO:teuthology.orchestra.run.smithi155.stdout: "map_inc": 13, 2024-03-19T03:04:07.528 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_sessions": 8, 2024-03-19T03:04:07.528 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_session_open": 8, 2024-03-19T03:04:07.528 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_session_close": 0, 2024-03-19T03:04:07.528 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_laggy": 0, 2024-03-19T03:04:07.529 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_wr": 14, 2024-03-19T03:04:07.529 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_rd": 0, 2024-03-19T03:04:07.529 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_del": 1 2024-03-19T03:04:07.529 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.529 INFO:teuthology.orchestra.run.smithi155.stdout: "oft": { 2024-03-19T03:04:07.529 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_objs": 1, 2024-03-19T03:04:07.529 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_kv_pairs": 0, 2024-03-19T03:04:07.529 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_updates": 0, 2024-03-19T03:04:07.529 INFO:teuthology.orchestra.run.smithi155.stdout: "omap_total_removes": 0 2024-03-19T03:04:07.529 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.529 INFO:teuthology.orchestra.run.smithi155.stdout: "osd-slow-ops": { 2024-03-19T03:04:07.529 INFO:teuthology.orchestra.run.smithi155.stdout: "slow_ops_count": 0 2024-03-19T03:04:07.529 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.529 INFO:teuthology.orchestra.run.smithi155.stdout: "purge_queue": { 2024-03-19T03:04:07.529 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing_ops": 0, 2024-03-19T03:04:07.529 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing_ops_high_water": 0, 2024-03-19T03:04:07.530 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing": 0, 2024-03-19T03:04:07.530 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executing_high_water": 0, 2024-03-19T03:04:07.530 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executed_ops": 0, 2024-03-19T03:04:07.530 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_executed": 0, 2024-03-19T03:04:07.530 INFO:teuthology.orchestra.run.smithi155.stdout: "pq_item_in_journal": 0 2024-03-19T03:04:07.530 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.530 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T03:04:07.530 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-19T03:04:07.530 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 104857600, 2024-03-19T03:04:07.530 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-19T03:04:07.530 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 916, 2024-03-19T03:04:07.530 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 1567310, 2024-03-19T03:04:07.531 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-19T03:04:07.531 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 916, 2024-03-19T03:04:07.531 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2024-03-19T03:04:07.531 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2024-03-19T03:04:07.531 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 916, 2024-03-19T03:04:07.531 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 1567310, 2024-03-19T03:04:07.531 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-19T03:04:07.531 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.531 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.531 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.531 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-19T03:04:07.531 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.531 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-objecter_bytes": { 2024-03-19T03:04:07.531 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-19T03:04:07.532 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 104857600, 2024-03-19T03:04:07.532 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-19T03:04:07.532 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2024-03-19T03:04:07.532 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2024-03-19T03:04:07.532 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-19T03:04:07.532 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2024-03-19T03:04:07.532 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 43, 2024-03-19T03:04:07.532 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 12486, 2024-03-19T03:04:07.532 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 43, 2024-03-19T03:04:07.532 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 12486, 2024-03-19T03:04:07.532 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-19T03:04:07.532 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.532 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.533 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.533 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-19T03:04:07.533 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.533 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-objecter_ops": { 2024-03-19T03:04:07.533 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-19T03:04:07.533 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 1024, 2024-03-19T03:04:07.533 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-19T03:04:07.533 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2024-03-19T03:04:07.533 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2024-03-19T03:04:07.533 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-19T03:04:07.533 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2024-03-19T03:04:07.533 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 43, 2024-03-19T03:04:07.533 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 43, 2024-03-19T03:04:07.533 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 43, 2024-03-19T03:04:07.534 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 43, 2024-03-19T03:04:07.534 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-19T03:04:07.534 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.534 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.534 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.534 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-19T03:04:07.534 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.534 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-write_buf_throttle": { 2024-03-19T03:04:07.534 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-19T03:04:07.534 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 3758096384, 2024-03-19T03:04:07.534 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-19T03:04:07.534 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 0, 2024-03-19T03:04:07.534 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 0, 2024-03-19T03:04:07.534 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-19T03:04:07.535 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 0, 2024-03-19T03:04:07.535 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2024-03-19T03:04:07.535 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2024-03-19T03:04:07.535 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 0, 2024-03-19T03:04:07.535 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 0, 2024-03-19T03:04:07.535 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-19T03:04:07.535 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.535 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.535 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.535 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-19T03:04:07.535 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:07.535 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-write_buf_throttle-0x55735ad32500": { 2024-03-19T03:04:07.535 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-19T03:04:07.535 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 3758096384, 2024-03-19T03:04:07.535 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-19T03:04:07.536 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 6, 2024-03-19T03:04:07.536 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 2004, 2024-03-19T03:04:07.536 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-19T03:04:07.536 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 6, 2024-03-19T03:04:07.536 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2024-03-19T03:04:07.536 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2024-03-19T03:04:07.536 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 5, 2024-03-19T03:04:07.536 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 2004, 2024-03-19T03:04:07.536 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-19T03:04:07.536 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:07.536 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:07.536 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:07.536 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-19T03:04:07.536 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-19T03:04:07.536 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-03-19T03:04:08.251 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-03-19T03:04:08.251 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-03-19T03:04:08.251 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=0 2024-03-19T03:04:08.251 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=11 2024-03-19T03:04:08.251 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-03-19T03:04:08.251 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-03-19T03:04:08.251 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-03-19T03:04:08.252 DEBUG:tasks.check_counter:Getting stats from l 2024-03-19T03:04:08.252 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-19T03:04:08.505 INFO:teuthology.orchestra.run.smithi155.stderr:Inferring config /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T03:04:08.956 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:08 smithi155 ceph-mon[35804]: pgmap v331: 129 pgs: 129 active+clean; 1.9 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-19T03:04:08.956 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:08 smithi155 ceph-mon[35804]: osdmap e107: 12 total, 12 up, 12 in 2024-03-19T03:04:09.095 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:08 smithi033 ceph-mon[31973]: pgmap v331: 129 pgs: 129 active+clean; 1.9 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-19T03:04:09.095 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:08 smithi033 ceph-mon[31973]: osdmap e107: 12 total, 12 up, 12 in 2024-03-19T03:04:09.122 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:08 smithi096 ceph-mon[35946]: pgmap v331: 129 pgs: 129 active+clean; 1.9 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-19T03:04:09.123 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:08 smithi096 ceph-mon[35946]: osdmap e107: 12 total, 12 up, 12 in 2024-03-19T03:04:10.194 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-03-19T03:04:10.194 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T03:04:10.194 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 4, 2024-03-19T03:04:10.195 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 4, 2024-03-19T03:04:10.195 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 2992, 2024-03-19T03:04:10.195 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 448, 2024-03-19T03:04:10.195 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 4, 2024-03-19T03:04:10.195 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 0, 2024-03-19T03:04:10.195 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.001420536, 2024-03-19T03:04:10.195 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.000035397, 2024-03-19T03:04:10.195 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.000643237, 2024-03-19T03:04:10.195 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-19T03:04:10.195 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:04:10.195 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 3, 2024-03-19T03:04:10.195 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000838981, 2024-03-19T03:04:10.195 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000279660 2024-03-19T03:04:10.196 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:10.196 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:04:10.196 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:10.196 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:10.196 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:10.196 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:10.196 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-19T03:04:10.196 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 448 2024-03-19T03:04:10.196 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:10.196 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T03:04:10.196 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 2, 2024-03-19T03:04:10.196 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 171, 2024-03-19T03:04:10.196 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 320, 2024-03-19T03:04:10.197 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 185664, 2024-03-19T03:04:10.197 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 5, 2024-03-19T03:04:10.197 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 3, 2024-03-19T03:04:10.197 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.008134922, 2024-03-19T03:04:10.197 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.005001584, 2024-03-19T03:04:10.197 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.001487098, 2024-03-19T03:04:10.197 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-19T03:04:10.197 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:04:10.197 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 168, 2024-03-19T03:04:10.197 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.011798888, 2024-03-19T03:04:10.197 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000070231 2024-03-19T03:04:10.197 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:10.197 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:04:10.197 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:10.198 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:10.198 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:10.198 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:10.198 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-19T03:04:10.198 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 185664 2024-03-19T03:04:10.198 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:10.198 INFO:teuthology.orchestra.run.smithi155.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T03:04:10.198 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_messages": 234, 2024-03-19T03:04:10.198 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages": 224, 2024-03-19T03:04:10.198 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_bytes": 1440928, 2024-03-19T03:04:10.198 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_bytes": 137664, 2024-03-19T03:04:10.198 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_created_connections": 2, 2024-03-19T03:04:10.198 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_active_connections": 1, 2024-03-19T03:04:10.198 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_total_time": 0.036493478, 2024-03-19T03:04:10.199 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_send_time": 0.008749158, 2024-03-19T03:04:10.199 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_recv_time": 0.033313277, 2024-03-19T03:04:10.199 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_running_fast_dispatch_time": 0.003823510, 2024-03-19T03:04:10.199 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:04:10.199 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 223, 2024-03-19T03:04:10.199 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.008431768, 2024-03-19T03:04:10.199 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000037810 2024-03-19T03:04:10.199 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:10.199 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:04:10.199 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:10.199 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:10.199 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:10.199 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:10.200 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_recv_encrypted_bytes": 1440928, 2024-03-19T03:04:10.200 INFO:teuthology.orchestra.run.smithi155.stdout: "msgr_send_encrypted_bytes": 137664 2024-03-19T03:04:10.200 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:10.200 INFO:teuthology.orchestra.run.smithi155.stdout: "cct": { 2024-03-19T03:04:10.200 INFO:teuthology.orchestra.run.smithi155.stdout: "total_workers": 0, 2024-03-19T03:04:10.200 INFO:teuthology.orchestra.run.smithi155.stdout: "unhealthy_workers": 0 2024-03-19T03:04:10.200 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:10.200 INFO:teuthology.orchestra.run.smithi155.stdout: "mempool": { 2024-03-19T03:04:10.200 INFO:teuthology.orchestra.run.smithi155.stdout: "bloom_filter_bytes": 0, 2024-03-19T03:04:10.200 INFO:teuthology.orchestra.run.smithi155.stdout: "bloom_filter_items": 0, 2024-03-19T03:04:10.200 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T03:04:10.200 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_alloc_items": 0, 2024-03-19T03:04:10.200 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T03:04:10.200 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_data_items": 0, 2024-03-19T03:04:10.201 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T03:04:10.201 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T03:04:10.201 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T03:04:10.201 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T03:04:10.201 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T03:04:10.201 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_other_items": 0, 2024-03-19T03:04:10.201 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T03:04:10.201 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T03:04:10.201 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_extent_bytes": 0, 2024-03-19T03:04:10.201 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_extent_items": 0, 2024-03-19T03:04:10.201 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_blob_bytes": 0, 2024-03-19T03:04:10.201 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_blob_items": 0, 2024-03-19T03:04:10.201 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T03:04:10.201 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T03:04:10.201 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T03:04:10.201 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T03:04:10.202 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T03:04:10.202 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_fsck_items": 0, 2024-03-19T03:04:10.202 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_txc_bytes": 0, 2024-03-19T03:04:10.202 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_txc_items": 0, 2024-03-19T03:04:10.202 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T03:04:10.202 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T03:04:10.202 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_bytes": 0, 2024-03-19T03:04:10.202 INFO:teuthology.orchestra.run.smithi155.stdout: "bluestore_writing_items": 0, 2024-03-19T03:04:10.202 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_bytes": 0, 2024-03-19T03:04:10.202 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_items": 0, 2024-03-19T03:04:10.202 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T03:04:10.202 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_reader_items": 0, 2024-03-19T03:04:10.202 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T03:04:10.202 INFO:teuthology.orchestra.run.smithi155.stdout: "bluefs_file_writer_items": 0, 2024-03-19T03:04:10.202 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_anon_bytes": 148731, 2024-03-19T03:04:10.203 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_anon_items": 31, 2024-03-19T03:04:10.203 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_meta_bytes": 0, 2024-03-19T03:04:10.203 INFO:teuthology.orchestra.run.smithi155.stdout: "buffer_meta_items": 0, 2024-03-19T03:04:10.203 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_bytes": 0, 2024-03-19T03:04:10.203 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_items": 0, 2024-03-19T03:04:10.203 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_mapbl_bytes": 0, 2024-03-19T03:04:10.203 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_mapbl_items": 0, 2024-03-19T03:04:10.203 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_pglog_bytes": 0, 2024-03-19T03:04:10.203 INFO:teuthology.orchestra.run.smithi155.stdout: "osd_pglog_items": 0, 2024-03-19T03:04:10.203 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_bytes": 0, 2024-03-19T03:04:10.203 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_items": 0, 2024-03-19T03:04:10.203 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T03:04:10.203 INFO:teuthology.orchestra.run.smithi155.stdout: "osdmap_mapping_items": 0, 2024-03-19T03:04:10.203 INFO:teuthology.orchestra.run.smithi155.stdout: "pgmap_bytes": 0, 2024-03-19T03:04:10.204 INFO:teuthology.orchestra.run.smithi155.stdout: "pgmap_items": 0, 2024-03-19T03:04:10.204 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_co_bytes": 928, 2024-03-19T03:04:10.204 INFO:teuthology.orchestra.run.smithi155.stdout: "mds_co_items": 2, 2024-03-19T03:04:10.204 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_1_bytes": 0, 2024-03-19T03:04:10.204 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_1_items": 0, 2024-03-19T03:04:10.204 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_2_bytes": 0, 2024-03-19T03:04:10.204 INFO:teuthology.orchestra.run.smithi155.stdout: "unittest_2_items": 0 2024-03-19T03:04:10.204 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-03-19T03:04:10.204 INFO:teuthology.orchestra.run.smithi155.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T03:04:10.204 INFO:teuthology.orchestra.run.smithi155.stdout: "val": 0, 2024-03-19T03:04:10.204 INFO:teuthology.orchestra.run.smithi155.stdout: "max": 104857600, 2024-03-19T03:04:10.204 INFO:teuthology.orchestra.run.smithi155.stdout: "get_started": 0, 2024-03-19T03:04:10.204 INFO:teuthology.orchestra.run.smithi155.stdout: "get": 236, 2024-03-19T03:04:10.204 INFO:teuthology.orchestra.run.smithi155.stdout: "get_sum": 1407935, 2024-03-19T03:04:10.204 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_fail": 0, 2024-03-19T03:04:10.205 INFO:teuthology.orchestra.run.smithi155.stdout: "get_or_fail_success": 236, 2024-03-19T03:04:10.205 INFO:teuthology.orchestra.run.smithi155.stdout: "take": 0, 2024-03-19T03:04:10.205 INFO:teuthology.orchestra.run.smithi155.stdout: "take_sum": 0, 2024-03-19T03:04:10.205 INFO:teuthology.orchestra.run.smithi155.stdout: "put": 236, 2024-03-19T03:04:10.205 INFO:teuthology.orchestra.run.smithi155.stdout: "put_sum": 1407935, 2024-03-19T03:04:10.205 INFO:teuthology.orchestra.run.smithi155.stdout: "wait": { 2024-03-19T03:04:10.205 INFO:teuthology.orchestra.run.smithi155.stdout: "avgcount": 0, 2024-03-19T03:04:10.205 INFO:teuthology.orchestra.run.smithi155.stdout: "sum": 0.000000000, 2024-03-19T03:04:10.205 INFO:teuthology.orchestra.run.smithi155.stdout: "avgtime": 0.000000000 2024-03-19T03:04:10.205 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-19T03:04:10.205 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-03-19T03:04:10.205 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-03-19T03:04:10.997 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:10 smithi155 ceph-mon[35804]: pgmap v333: 129 pgs: 129 active+clean; 1.9 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-19T03:04:11.028 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-03-19T03:04:11.028 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-03-19T03:04:11.028 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2024-03-19T03:04:11.028 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2024-03-19T03:04:11.028 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-03-19T03:04:11.028 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-03-19T03:04:11.029 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-03-19T03:04:11.029 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-19T03:04:11.041 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-19T03:04:11.041 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-19T03:04:11.041 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T03:04:11.063 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-19T03:04:11.063 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T03:04:11.122 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T03:04:11.190 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:10 smithi033 ceph-mon[31973]: pgmap v333: 129 pgs: 129 active+clean; 1.9 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-19T03:04:11.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:10 smithi096 ceph-mon[35946]: pgmap v333: 129 pgs: 129 active+clean; 1.9 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-19T03:04:11.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.275+0000 7f50789d4640 1 -- 172.21.15.33:0/3128263813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50741017b0 msgr2=0x7f5074101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:11.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.275+0000 7f50789d4640 1 --2- 172.21.15.33:0/3128263813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50741017b0 0x7f5074101c30 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f5068009930 tx=0x7f506802bc30 comp rx=0 tx=0).stop 2024-03-19T03:04:11.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.275+0000 7f50789d4640 1 -- 172.21.15.33:0/3128263813 shutdown_connections 2024-03-19T03:04:11.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.275+0000 7f50789d4640 1 --2- 172.21.15.33:0/3128263813 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5074102170 0x7f507410e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:11.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.275+0000 7f50789d4640 1 --2- 172.21.15.33:0/3128263813 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f50741017b0 0x7f5074101c30 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:11.279 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.275+0000 7f50789d4640 1 --2- 172.21.15.33:0/3128263813 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50741077c0 0x7f5074107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:11.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.275+0000 7f50789d4640 1 -- 172.21.15.33:0/3128263813 >> 172.21.15.33:0/3128263813 conn(0x7f50740fd2c0 msgr2=0x7f50740ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:11.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.275+0000 7f50789d4640 1 -- 172.21.15.33:0/3128263813 shutdown_connections 2024-03-19T03:04:11.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.275+0000 7f50789d4640 1 -- 172.21.15.33:0/3128263813 wait complete. 2024-03-19T03:04:11.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.276+0000 7f50789d4640 1 Processor -- start 2024-03-19T03:04:11.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.276+0000 7f50789d4640 1 -- start start 2024-03-19T03:04:11.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.276+0000 7f50789d4640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50741017b0 0x7f5074198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:11.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.276+0000 7f50789d4640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5074102170 0x7f5074198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:11.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.276+0000 7f50789d4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50741077c0 0x7f507419c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:11.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.276+0000 7f50789d4640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5074111a80 con 0x7f5074102170 2024-03-19T03:04:11.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.276+0000 7f50789d4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5074111900 con 0x7f50741077c0 2024-03-19T03:04:11.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.276+0000 7f50789d4640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5074111c00 con 0x7f50741017b0 2024-03-19T03:04:11.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.276+0000 7f5071d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5074102170 0x7f5074198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:11.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.277+0000 7f5071d9b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5074102170 0x7f5074198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50538/0 (socket says 172.21.15.33:50538) 2024-03-19T03:04:11.280 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.277+0000 7f5071d9b640 1 -- 172.21.15.33:0/45676194 learned_addr learned my addr 172.21.15.33:0/45676194 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:04:11.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.277+0000 7f507259c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50741017b0 0x7f5074198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:11.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.277+0000 7f5072d9d640 1 --2- 172.21.15.33:0/45676194 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50741077c0 0x7f507419c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:11.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.277+0000 7f5071d9b640 1 -- 172.21.15.33:0/45676194 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50741017b0 msgr2=0x7f5074198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:11.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.277+0000 7f5071d9b640 1 --2- 172.21.15.33:0/45676194 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50741017b0 0x7f5074198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:11.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.277+0000 7f5071d9b640 1 -- 172.21.15.33:0/45676194 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50741077c0 msgr2=0x7f507419c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:11.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.277+0000 7f5071d9b640 1 --2- 172.21.15.33:0/45676194 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50741077c0 0x7f507419c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:11.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.277+0000 7f5071d9b640 1 -- 172.21.15.33:0/45676194 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f507419d000 con 0x7f5074102170 2024-03-19T03:04:11.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.277+0000 7f507259c640 1 --2- 172.21.15.33:0/45676194 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50741017b0 0x7f5074198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:04:11.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.277+0000 7f5071d9b640 1 --2- 172.21.15.33:0/45676194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5074102170 0x7f5074198550 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f5068009230 tx=0x7f506802c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:11.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.277+0000 7f50637fe640 1 -- 172.21.15.33:0/45676194 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5068043020 con 0x7f5074102170 2024-03-19T03:04:11.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.277+0000 7f50637fe640 1 -- 172.21.15.33:0/45676194 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f506802ca70 con 0x7f5074102170 2024-03-19T03:04:11.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.277+0000 7f50789d4640 1 -- 172.21.15.33:0/45676194 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f507419d290 con 0x7f5074102170 2024-03-19T03:04:11.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.277+0000 7f50637fe640 1 -- 172.21.15.33:0/45676194 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5068003a50 con 0x7f5074102170 2024-03-19T03:04:11.281 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.277+0000 7f50789d4640 1 -- 172.21.15.33:0/45676194 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50741a4b80 con 0x7f5074102170 2024-03-19T03:04:11.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.278+0000 7f50789d4640 1 -- 172.21.15.33:0/45676194 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5074068a60 con 0x7f5074102170 2024-03-19T03:04:11.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.279+0000 7f50637fe640 1 -- 172.21.15.33:0/45676194 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f5068046020 con 0x7f5074102170 2024-03-19T03:04:11.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.280+0000 7f50637fe640 1 --2- 172.21.15.33:0/45676194 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f505c0a2970 0x7f505c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:11.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.280+0000 7f507259c640 1 --2- 172.21.15.33:0/45676194 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f505c0a2970 0x7f505c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:11.282 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.280+0000 7f50637fe640 1 -- 172.21.15.33:0/45676194 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9584+0+0 (secure 0 0 0) 0x7f50680f1190 con 0x7f5074102170 2024-03-19T03:04:11.283 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.280+0000 7f507259c640 1 --2- 172.21.15.33:0/45676194 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f505c0a2970 0x7f505c0a4e30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5064009f40 tx=0x7f5064005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:11.285 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.284+0000 7f50637fe640 1 -- 172.21.15.33:0/45676194 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f506804d020 con 0x7f5074102170 2024-03-19T03:04:11.470 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.467+0000 7f50789d4640 1 -- 172.21.15.33:0/45676194 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5074109d70 con 0x7f5074102170 2024-03-19T03:04:11.471 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.469+0000 7f50637fe640 1 -- 172.21.15.33:0/45676194 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) ==== 74+0+25361 (secure 0 0 0) 0x7f50680bdba0 con 0x7f5074102170 2024-03-19T03:04:11.471 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T03:04:11.471 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":107,"fsid":"2b7a8938-e59a-11ee-95c9-87774f69a715","created":"2024-03-19T02:42:37.912179+0000","modified":"2024-03-19T03:04:07.834958+0000","last_up_change":"2024-03-19T02:50:01.539392+0000","last_in_change":"2024-03-19T02:49:40.534415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:46:09.963737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:50:50.069641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:50:51.381064+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:50:55.476667+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"090a89ab-6b7b-4317-8823-178d4212b2f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6803","nonce":1633124995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6805","nonce":1633124995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6809","nonce":1633124995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6807","nonce":1633124995}]},"public_addr":"172.21.15.33:6803/1633124995","cluster_addr":"172.21.15.33:6805/1633124995","heartbeat_back_addr":"172.21.15.33:6809/1633124995","heartbeat_front_addr":"172.21.15.33:6807/1633124995","state":["exists","up"]},{"osd":1,"uuid":"f50eec55-1956-4486-8593-f2c8fa610e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6801","nonce":340144398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6803","nonce":340144398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6807","nonce":340144398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6805","nonce":340144398}]},"public_addr":"172.21.15.96:6801/340144398","cluster_addr":"172.21.15.96:6803/340144398","heartbeat_back_addr":"172.21.15.96:6807/340144398","heartbeat_front_addr":"172.21.15.96:6805/340144398","state":["exists","up"]},{"osd":2,"uuid":"893a2d62-80d9-4e5e-bc6a-a1c8c681d652","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6801","nonce":3388743876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6803","nonce":3388743876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6807","nonce":3388743876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6805","nonce":3388743876}]},"public_addr":"172.21.15.155:6801/3388743876","cluster_addr":"172.21.15.155:6803/3388743876","heartbeat_back_addr":"172.21.15.155:6807/3388743876","heartbeat_front_addr":"172.21.15.155:6805/3388743876","state":["exists","up"]},{"osd":3,"uuid":"b07e0482-4281-4d32-9dfe-9995d7246175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6811","nonce":677007171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6813","nonce":677007171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6817","nonce":677007171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6815","nonce":677007171}]},"public_addr":"172.21.15.33:6811/677007171","cluster_addr":"172.21.15.33:6813/677007171","heartbeat_back_addr":"172.21.15.33:6817/677007171","heartbeat_front_addr":"172.21.15.33:6815/677007171","state":["exists","up"]},{"osd":4,"uuid":"a988384b-af6b-47c6-8c07-5c9673acee45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6809","nonce":3106158554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6811","nonce":3106158554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6815","nonce":3106158554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6813","nonce":3106158554}]},"public_addr":"172.21.15.96:6809/3106158554","cluster_addr":"172.21.15.96:6811/3106158554","heartbeat_back_addr":"172.21.15.96:6815/3106158554","heartbeat_front_addr":"172.21.15.96:6813/3106158554","state":["exists","up"]},{"osd":5,"uuid":"198dc992-96f8-4c52-96ee-1fd5de148bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6809","nonce":327293324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6811","nonce":327293324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6815","nonce":327293324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6813","nonce":327293324}]},"public_addr":"172.21.15.155:6809/327293324","cluster_addr":"172.21.15.155:6811/327293324","heartbeat_back_addr":"172.21.15.155:6815/327293324","heartbeat_front_addr":"172.21.15.155:6813/327293324","state":["exists","up"]},{"osd":6,"uuid":"a7278ec0-ab2e-49ed-be53-84d8b9e24177","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6819","nonce":873871509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6821","nonce":873871509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6825","nonce":873871509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6823","nonce":873871509}]},"public_addr":"172.21.15.33:6819/873871509","cluster_addr":"172.21.15.33:6821/873871509","heartbeat_back_addr":"172.21.15.33:6825/873871509","heartbeat_front_addr":"172.21.15.33:6823/873871509","state":["exists","up"]},{"osd":7,"uuid":"727e133d-aa24-4ffc-90e2-266eb2aab8c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6817","nonce":2334123891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6819","nonce":2334123891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6823","nonce":2334123891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6821","nonce":2334123891}]},"public_addr":"172.21.15.96:6817/2334123891","cluster_addr":"172.21.15.96:6819/2334123891","heartbeat_back_addr":"172.21.15.96:6823/2334123891","heartbeat_front_addr":"172.21.15.96:6821/2334123891","state":["exists","up"]},{"osd":8,"uuid":"bb20eec0-39e5-4b86-9db5-486b1840cdad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6817","nonce":3498516903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6819","nonce":3498516903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6823","nonce":3498516903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6821","nonce":3498516903}]},"public_addr":"172.21.15.155:6817/3498516903","cluster_addr":"172.21.15.155:6819/3498516903","heartbeat_back_addr":"172.21.15.155:6823/3498516903","heartbeat_front_addr":"172.21.15.155:6821/3498516903","state":["exists","up"]},{"osd":9,"uuid":"179c6228-3e23-4f65-91a6-1ef0005d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6827","nonce":760777213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6829","nonce":760777213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6833","nonce":760777213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6831","nonce":760777213}]},"public_addr":"172.21.15.33:6827/760777213","cluster_addr":"172.21.15.33:6829/760777213","heartbeat_back_addr":"172.21.15.33:6833/760777213","heartbeat_front_addr":"172.21.15.33:6831/760777213","state":["exists","up"]},{"osd":10,"uuid":"be17b478-3c2e-41fb-bede-958333acdf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6825","nonce":1404898482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6827","nonce":1404898482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6831","nonce":1404898482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6829","nonce":1404898482}]},"public_addr":"172.21.15.96:6825/1404898482","cluster_addr":"172.21.15.96:6827/1404898482","heartbeat_back_addr":"172.21.15.96:6831/1404898482","heartbeat_front_addr":"172.21.15.96:6829/1404898482","state":["exists","up"]},{"osd":11,"uuid":"c0903ed2-cc1e-4c18-8731-1f0ee4dd0457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6825","nonce":792352827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6827","nonce":792352827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6831","nonce":792352827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6829","nonce":792352827}]},"public_addr":"172.21.15.155:6825/792352827","cluster_addr":"172.21.15.155:6827/792352827","heartbeat_back_addr":"172.21.15.155:6831/792352827","heartbeat_front_addr":"172.21.15.155:6829/792352827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:17.942801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:42.109449+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:05.981599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:32.069574+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:57.431015+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:22.448592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:48.526515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:14.498352+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:40.175426+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:06.364006+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:32.853161+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:59.073538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.9","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:6800/1453067701":"2024-03-20T02:53:26.937131+0000","172.21.15.33:0/3554132372":"2024-03-20T02:53:26.937131+0000","172.21.15.33:0/4099467400":"2024-03-20T02:53:26.937131+0000","172.21.15.33:0/176174888":"2024-03-20T02:53:26.937131+0000","172.21.15.33:6801/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:6800/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/1067768047":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/729379453":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2586588742":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/2932814165":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/1950732854":"2024-03-20T02:53:26.937131+0000","172.21.15.33:6801/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6801/1453067701":"2024-03-20T02:53:26.937131+0000","172.21.15.33:0/2212564963":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6800/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/3275126058":"2024-03-20T02:43:39.622218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:04:11.472 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.471+0000 7f50789d4640 1 -- 172.21.15.33:0/45676194 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f505c0a2970 msgr2=0x7f505c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:11.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.471+0000 7f50789d4640 1 --2- 172.21.15.33:0/45676194 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f505c0a2970 0x7f505c0a4e30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5064009f40 tx=0x7f5064005f90 comp rx=0 tx=0).stop 2024-03-19T03:04:11.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.471+0000 7f50789d4640 1 -- 172.21.15.33:0/45676194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5074102170 msgr2=0x7f5074198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:11.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.471+0000 7f50789d4640 1 --2- 172.21.15.33:0/45676194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5074102170 0x7f5074198550 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f5068009230 tx=0x7f506802c5d0 comp rx=0 tx=0).stop 2024-03-19T03:04:11.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.471+0000 7f50789d4640 1 -- 172.21.15.33:0/45676194 shutdown_connections 2024-03-19T03:04:11.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.471+0000 7f50789d4640 1 --2- 172.21.15.33:0/45676194 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f505c0a2970 0x7f505c0a4e30 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:11.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.471+0000 7f50789d4640 1 --2- 172.21.15.33:0/45676194 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f50741077c0 0x7f507419c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:11.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.471+0000 7f50789d4640 1 --2- 172.21.15.33:0/45676194 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5074102170 0x7f5074198550 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:11.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.471+0000 7f50789d4640 1 --2- 172.21.15.33:0/45676194 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50741017b0 0x7f5074198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:11.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.471+0000 7f50789d4640 1 -- 172.21.15.33:0/45676194 >> 172.21.15.33:0/45676194 conn(0x7f50740fd2c0 msgr2=0x7f50740fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:11.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.471+0000 7f50789d4640 1 -- 172.21.15.33:0/45676194 shutdown_connections 2024-03-19T03:04:11.473 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.471+0000 7f50789d4640 1 -- 172.21.15.33:0/45676194 wait complete. 2024-03-19T03:04:11.484 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:46:09.963737+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T02:50:50.069641+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T02:50:51.381064+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 107, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T02:50:55.476667+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 107, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T03:04:11.484 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T03:04:11.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.628+0000 7f4d17a0b640 1 -- 172.21.15.33:0/923662127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d10100740 msgr2=0x7f4d10100bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:11.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.628+0000 7f4d17a0b640 1 --2- 172.21.15.33:0/923662127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d10100740 0x7f4d10100bc0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f4d00009930 tx=0x7f4d0002bc30 comp rx=0 tx=0).stop 2024-03-19T03:04:11.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.628+0000 7f4d17a0b640 1 -- 172.21.15.33:0/923662127 shutdown_connections 2024-03-19T03:04:11.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.628+0000 7f4d17a0b640 1 --2- 172.21.15.33:0/923662127 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d10101100 0x7f4d1010d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:11.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.628+0000 7f4d17a0b640 1 --2- 172.21.15.33:0/923662127 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d10100740 0x7f4d10100bc0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:11.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.628+0000 7f4d17a0b640 1 --2- 172.21.15.33:0/923662127 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d10106750 0x7f4d10106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:11.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.628+0000 7f4d17a0b640 1 -- 172.21.15.33:0/923662127 >> 172.21.15.33:0/923662127 conn(0x7f4d10077880 msgr2=0x7f4d10077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:11.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.628+0000 7f4d17a0b640 1 -- 172.21.15.33:0/923662127 shutdown_connections 2024-03-19T03:04:11.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.628+0000 7f4d17a0b640 1 -- 172.21.15.33:0/923662127 wait complete. 2024-03-19T03:04:11.632 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.628+0000 7f4d17a0b640 1 Processor -- start 2024-03-19T03:04:11.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.629+0000 7f4d17a0b640 1 -- start start 2024-03-19T03:04:11.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.629+0000 7f4d17a0b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d10100740 0x7f4d101980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:11.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.629+0000 7f4d17a0b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d10101100 0x7f4d101985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:11.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.629+0000 7f4d17a0b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d10106750 0x7f4d1019c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:11.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.629+0000 7f4d17a0b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4d10110a10 con 0x7f4d10101100 2024-03-19T03:04:11.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.629+0000 7f4d17a0b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4d10110890 con 0x7f4d10106750 2024-03-19T03:04:11.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.629+0000 7f4d17a0b640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4d10110b90 con 0x7f4d10100740 2024-03-19T03:04:11.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.629+0000 7f4d14fa6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d10101100 0x7f4d101985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:11.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.629+0000 7f4d14fa6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d10101100 0x7f4d101985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50558/0 (socket says 172.21.15.33:50558) 2024-03-19T03:04:11.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.629+0000 7f4d14fa6640 1 -- 172.21.15.33:0/4099832040 learned_addr learned my addr 172.21.15.33:0/4099832040 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:04:11.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.629+0000 7f4d157a7640 1 --2- 172.21.15.33:0/4099832040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d10100740 0x7f4d101980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:11.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.629+0000 7f4d15fa8640 1 --2- 172.21.15.33:0/4099832040 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d10106750 0x7f4d1019c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:11.633 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.629+0000 7f4d14fa6640 1 -- 172.21.15.33:0/4099832040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d10100740 msgr2=0x7f4d101980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:11.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.629+0000 7f4d14fa6640 1 --2- 172.21.15.33:0/4099832040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d10100740 0x7f4d101980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:11.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.629+0000 7f4d14fa6640 1 -- 172.21.15.33:0/4099832040 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d10106750 msgr2=0x7f4d1019c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:11.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.629+0000 7f4d14fa6640 1 --2- 172.21.15.33:0/4099832040 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d10106750 0x7f4d1019c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:11.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.629+0000 7f4d14fa6640 1 -- 172.21.15.33:0/4099832040 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d1019d040 con 0x7f4d10101100 2024-03-19T03:04:11.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.629+0000 7f4d157a7640 1 --2- 172.21.15.33:0/4099832040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d10100740 0x7f4d101980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:04:11.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.629+0000 7f4d15fa8640 1 --2- 172.21.15.33:0/4099832040 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d10106750 0x7f4d1019c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:04:11.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.629+0000 7f4d14fa6640 1 --2- 172.21.15.33:0/4099832040 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d10101100 0x7f4d101985e0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f4d00009230 tx=0x7f4d0002c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:11.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.629+0000 7f4d067fc640 1 -- 172.21.15.33:0/4099832040 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d00043020 con 0x7f4d10101100 2024-03-19T03:04:11.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.630+0000 7f4d067fc640 1 -- 172.21.15.33:0/4099832040 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4d0002ca70 con 0x7f4d10101100 2024-03-19T03:04:11.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.630+0000 7f4d17a0b640 1 -- 172.21.15.33:0/4099832040 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d1019d2d0 con 0x7f4d10101100 2024-03-19T03:04:11.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.630+0000 7f4d067fc640 1 -- 172.21.15.33:0/4099832040 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d00003970 con 0x7f4d10101100 2024-03-19T03:04:11.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.630+0000 7f4d17a0b640 1 -- 172.21.15.33:0/4099832040 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d101a4bc0 con 0x7f4d10101100 2024-03-19T03:04:11.634 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.631+0000 7f4d17a0b640 1 -- 172.21.15.33:0/4099832040 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d10068a60 con 0x7f4d10101100 2024-03-19T03:04:11.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.632+0000 7f4d067fc640 1 -- 172.21.15.33:0/4099832040 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f4d00046020 con 0x7f4d10101100 2024-03-19T03:04:11.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.632+0000 7f4d067fc640 1 --2- 172.21.15.33:0/4099832040 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4cfc0a2970 0x7f4cfc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:11.635 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.632+0000 7f4d157a7640 1 --2- 172.21.15.33:0/4099832040 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4cfc0a2970 0x7f4cfc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:11.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.632+0000 7f4d067fc640 1 -- 172.21.15.33:0/4099832040 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9584+0+0 (secure 0 0 0) 0x7f4d000f11d0 con 0x7f4d10101100 2024-03-19T03:04:11.636 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.633+0000 7f4d157a7640 1 --2- 172.21.15.33:0/4099832040 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4cfc0a2970 0x7f4cfc0a4e30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c0095f0 tx=0x7f4d0c005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:11.638 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.637+0000 7f4d067fc640 1 -- 172.21.15.33:0/4099832040 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4d0004d020 con 0x7f4d10101100 2024-03-19T03:04:11.823 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.822+0000 7f4d17a0b640 1 -- 172.21.15.33:0/4099832040 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4d10109cc0 con 0x7f4d10101100 2024-03-19T03:04:11.824 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.822+0000 7f4d067fc640 1 -- 172.21.15.33:0/4099832040 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v107) ==== 93+0+10 (secure 0 0 0) 0x7f4d000bdc20 con 0x7f4d10101100 2024-03-19T03:04:11.824 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 1 2024-03-19T03:04:11.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.824+0000 7f4d17a0b640 1 -- 172.21.15.33:0/4099832040 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4cfc0a2970 msgr2=0x7f4cfc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:11.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.824+0000 7f4d17a0b640 1 --2- 172.21.15.33:0/4099832040 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4cfc0a2970 0x7f4cfc0a4e30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c0095f0 tx=0x7f4d0c005f90 comp rx=0 tx=0).stop 2024-03-19T03:04:11.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.824+0000 7f4d17a0b640 1 -- 172.21.15.33:0/4099832040 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d10101100 msgr2=0x7f4d101985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:11.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.824+0000 7f4d17a0b640 1 --2- 172.21.15.33:0/4099832040 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d10101100 0x7f4d101985e0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f4d00009230 tx=0x7f4d0002c5d0 comp rx=0 tx=0).stop 2024-03-19T03:04:11.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.824+0000 7f4d17a0b640 1 -- 172.21.15.33:0/4099832040 shutdown_connections 2024-03-19T03:04:11.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.824+0000 7f4d17a0b640 1 --2- 172.21.15.33:0/4099832040 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4cfc0a2970 0x7f4cfc0a4e30 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:11.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.824+0000 7f4d17a0b640 1 --2- 172.21.15.33:0/4099832040 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d10106750 0x7f4d1019c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:11.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.824+0000 7f4d17a0b640 1 --2- 172.21.15.33:0/4099832040 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4d10101100 0x7f4d101985e0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:11.826 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.824+0000 7f4d17a0b640 1 --2- 172.21.15.33:0/4099832040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d10100740 0x7f4d101980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:11.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.824+0000 7f4d17a0b640 1 -- 172.21.15.33:0/4099832040 >> 172.21.15.33:0/4099832040 conn(0x7f4d10077880 msgr2=0x7f4d1006a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:11.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.824+0000 7f4d17a0b640 1 -- 172.21.15.33:0/4099832040 shutdown_connections 2024-03-19T03:04:11.827 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.824+0000 7f4d17a0b640 1 -- 172.21.15.33:0/4099832040 wait complete. 2024-03-19T03:04:11.836 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T03:04:11.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.941+0000 7fa0e3812640 1 -- 172.21.15.33:0/414959753 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa0dc102170 msgr2=0x7fa0dc10e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:11.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.941+0000 7fa0e3812640 1 --2- 172.21.15.33:0/414959753 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa0dc102170 0x7fa0dc10e680 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4009cd0 tx=0x7fa0d402c0f0 comp rx=0 tx=0).stop 2024-03-19T03:04:11.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.941+0000 7fa0e3812640 1 -- 172.21.15.33:0/414959753 shutdown_connections 2024-03-19T03:04:11.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.941+0000 7fa0e3812640 1 --2- 172.21.15.33:0/414959753 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa0dc102170 0x7fa0dc10e680 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:11.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.941+0000 7fa0e3812640 1 --2- 172.21.15.33:0/414959753 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0dc1017b0 0x7fa0dc101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:11.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.941+0000 7fa0e3812640 1 --2- 172.21.15.33:0/414959753 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa0dc1077c0 0x7fa0dc107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:11.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.941+0000 7fa0e3812640 1 -- 172.21.15.33:0/414959753 >> 172.21.15.33:0/414959753 conn(0x7fa0dc078c20 msgr2=0x7fa0dc079030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:11.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.941+0000 7fa0e3812640 1 -- 172.21.15.33:0/414959753 shutdown_connections 2024-03-19T03:04:11.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.941+0000 7fa0e3812640 1 -- 172.21.15.33:0/414959753 wait complete. 2024-03-19T03:04:11.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.942+0000 7fa0e3812640 1 Processor -- start 2024-03-19T03:04:11.945 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.942+0000 7fa0e3812640 1 -- start start 2024-03-19T03:04:11.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.942+0000 7fa0e3812640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0dc1017b0 0x7fa0dc197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:11.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.942+0000 7fa0e3812640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa0dc102170 0x7fa0dc1984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:11.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.942+0000 7fa0e3812640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa0dc1077c0 0x7fa0dc19c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:11.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.942+0000 7fa0e3812640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa0dc111a80 con 0x7fa0dc1077c0 2024-03-19T03:04:11.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.942+0000 7fa0e3812640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa0dc111900 con 0x7fa0dc1017b0 2024-03-19T03:04:11.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.942+0000 7fa0e3812640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa0dc111c00 con 0x7fa0dc102170 2024-03-19T03:04:11.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.942+0000 7fa0e1daf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa0dc1077c0 0x7fa0dc19c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:11.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.942+0000 7fa0e15ae640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0dc1017b0 0x7fa0dc197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:11.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.942+0000 7fa0e1daf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa0dc1077c0 0x7fa0dc19c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50582/0 (socket says 172.21.15.33:50582) 2024-03-19T03:04:11.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.942+0000 7fa0e1daf640 1 -- 172.21.15.33:0/87332189 learned_addr learned my addr 172.21.15.33:0/87332189 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:04:11.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.942+0000 7fa0e15ae640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0dc1017b0 0x7fa0dc197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.33:40694/0 (socket says 172.21.15.33:40694) 2024-03-19T03:04:11.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.942+0000 7fa0e0dad640 1 --2- 172.21.15.33:0/87332189 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa0dc102170 0x7fa0dc1984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:11.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.942+0000 7fa0e1daf640 1 -- 172.21.15.33:0/87332189 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa0dc102170 msgr2=0x7fa0dc1984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:11.946 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.942+0000 7fa0e1daf640 1 --2- 172.21.15.33:0/87332189 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa0dc102170 0x7fa0dc1984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:11.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.942+0000 7fa0e1daf640 1 -- 172.21.15.33:0/87332189 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0dc1017b0 msgr2=0x7fa0dc197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:11.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.942+0000 7fa0e1daf640 1 --2- 172.21.15.33:0/87332189 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0dc1017b0 0x7fa0dc197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:11.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.943+0000 7fa0e1daf640 1 -- 172.21.15.33:0/87332189 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0dc19cf70 con 0x7fa0dc1077c0 2024-03-19T03:04:11.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.943+0000 7fa0e15ae640 1 --2- 172.21.15.33:0/87332189 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0dc1017b0 0x7fa0dc197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:04:11.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.943+0000 7fa0e1daf640 1 --2- 172.21.15.33:0/87332189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa0dc1077c0 0x7fa0dc19c840 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4004af0 tx=0x7fa0d4035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:11.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.943+0000 7fa0d27fc640 1 -- 172.21.15.33:0/87332189 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d4039040 con 0x7fa0dc1077c0 2024-03-19T03:04:11.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.943+0000 7fa0d27fc640 1 -- 172.21.15.33:0/87332189 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa0d4044070 con 0x7fa0dc1077c0 2024-03-19T03:04:11.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.943+0000 7fa0e3812640 1 -- 172.21.15.33:0/87332189 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0dc19d200 con 0x7fa0dc1077c0 2024-03-19T03:04:11.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.943+0000 7fa0d27fc640 1 -- 172.21.15.33:0/87332189 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d40042a0 con 0x7fa0dc1077c0 2024-03-19T03:04:11.947 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.943+0000 7fa0e3812640 1 -- 172.21.15.33:0/87332189 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0dc1a4af0 con 0x7fa0dc1077c0 2024-03-19T03:04:11.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.944+0000 7fa0e3812640 1 -- 172.21.15.33:0/87332189 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0dc068a60 con 0x7fa0dc1077c0 2024-03-19T03:04:11.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.945+0000 7fa0d27fc640 1 -- 172.21.15.33:0/87332189 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fa0d4003970 con 0x7fa0dc1077c0 2024-03-19T03:04:11.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.945+0000 7fa0d27fc640 1 --2- 172.21.15.33:0/87332189 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa0c80a29f0 0x7fa0c80a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:11.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.946+0000 7fa0d27fc640 1 -- 172.21.15.33:0/87332189 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9584+0+0 (secure 0 0 0) 0x7fa0d40f14c0 con 0x7fa0dc1077c0 2024-03-19T03:04:11.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.946+0000 7fa0e15ae640 1 --2- 172.21.15.33:0/87332189 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa0c80a29f0 0x7fa0c80a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:11.948 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.946+0000 7fa0e15ae640 1 --2- 172.21.15.33:0/87332189 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa0c80a29f0 0x7fa0c80a4eb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8009f40 tx=0x7fa0d8005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:11.951 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:11.950+0000 7fa0d27fc640 1 -- 172.21.15.33:0/87332189 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa0d40bdf10 con 0x7fa0dc1077c0 2024-03-19T03:04:12.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.132+0000 7fa0e3812640 1 -- 172.21.15.33:0/87332189 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa0dc109d20 con 0x7fa0dc1077c0 2024-03-19T03:04:12.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.132+0000 7fa0d27fc640 1 -- 172.21.15.33:0/87332189 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v107) ==== 104+0+11 (secure 0 0 0) 0x7fa0d40bd810 con 0x7fa0dc1077c0 2024-03-19T03:04:12.134 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 32 2024-03-19T03:04:12.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.135+0000 7fa0e3812640 1 -- 172.21.15.33:0/87332189 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa0c80a29f0 msgr2=0x7fa0c80a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:12.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.135+0000 7fa0e3812640 1 --2- 172.21.15.33:0/87332189 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa0c80a29f0 0x7fa0c80a4eb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8009f40 tx=0x7fa0d8005f90 comp rx=0 tx=0).stop 2024-03-19T03:04:12.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.135+0000 7fa0e3812640 1 -- 172.21.15.33:0/87332189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa0dc1077c0 msgr2=0x7fa0dc19c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:12.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.135+0000 7fa0e3812640 1 --2- 172.21.15.33:0/87332189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa0dc1077c0 0x7fa0dc19c840 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4004af0 tx=0x7fa0d4035ce0 comp rx=0 tx=0).stop 2024-03-19T03:04:12.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.135+0000 7fa0e3812640 1 -- 172.21.15.33:0/87332189 shutdown_connections 2024-03-19T03:04:12.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.135+0000 7fa0e3812640 1 --2- 172.21.15.33:0/87332189 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa0c80a29f0 0x7fa0c80a4eb0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:12.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.135+0000 7fa0e3812640 1 --2- 172.21.15.33:0/87332189 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa0dc102170 0x7fa0dc1984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:12.137 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.135+0000 7fa0e3812640 1 --2- 172.21.15.33:0/87332189 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa0dc1077c0 0x7fa0dc19c840 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:12.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.135+0000 7fa0e3812640 1 --2- 172.21.15.33:0/87332189 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa0dc1017b0 0x7fa0dc197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:12.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.135+0000 7fa0e3812640 1 -- 172.21.15.33:0/87332189 >> 172.21.15.33:0/87332189 conn(0x7fa0dc078c20 msgr2=0x7fa0dc1042c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:12.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.135+0000 7fa0e3812640 1 -- 172.21.15.33:0/87332189 shutdown_connections 2024-03-19T03:04:12.138 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.135+0000 7fa0e3812640 1 -- 172.21.15.33:0/87332189 wait complete. 2024-03-19T03:04:12.146 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:11 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/45676194' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:04:12.146 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:11 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/4099832040' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:04:12.147 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T03:04:12.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:11 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/45676194' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:04:12.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:11 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/4099832040' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:04:12.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.254+0000 7f803349a640 1 -- 172.21.15.33:0/1965412063 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f802c100740 msgr2=0x7f802c100bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:12.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.254+0000 7f803349a640 1 --2- 172.21.15.33:0/1965412063 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f802c100740 0x7f802c100bc0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f801c009930 tx=0x7f801c02bc30 comp rx=0 tx=0).stop 2024-03-19T03:04:12.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.254+0000 7f803349a640 1 -- 172.21.15.33:0/1965412063 shutdown_connections 2024-03-19T03:04:12.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.254+0000 7f803349a640 1 --2- 172.21.15.33:0/1965412063 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f802c101100 0x7f802c10d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:12.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.254+0000 7f803349a640 1 --2- 172.21.15.33:0/1965412063 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f802c100740 0x7f802c100bc0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:12.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.254+0000 7f803349a640 1 --2- 172.21.15.33:0/1965412063 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f802c106750 0x7f802c106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:12.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.254+0000 7f803349a640 1 -- 172.21.15.33:0/1965412063 >> 172.21.15.33:0/1965412063 conn(0x7f802c077880 msgr2=0x7f802c077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:12.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.254+0000 7f803349a640 1 -- 172.21.15.33:0/1965412063 shutdown_connections 2024-03-19T03:04:12.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.254+0000 7f803349a640 1 -- 172.21.15.33:0/1965412063 wait complete. 2024-03-19T03:04:12.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.255+0000 7f803349a640 1 Processor -- start 2024-03-19T03:04:12.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.255+0000 7f803349a640 1 -- start start 2024-03-19T03:04:12.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.255+0000 7f803349a640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f802c100740 0x7f802c198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:12.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.255+0000 7f803349a640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f802c101100 0x7f802c198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:12.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.255+0000 7f803349a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f802c106750 0x7f802c19c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:12.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.255+0000 7f803349a640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f802c111b50 con 0x7f802c100740 2024-03-19T03:04:12.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.255+0000 7f803349a640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f802c1119d0 con 0x7f802c106750 2024-03-19T03:04:12.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.255+0000 7f803349a640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f802c111cd0 con 0x7f802c101100 2024-03-19T03:04:12.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.255+0000 7f8031236640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f802c100740 0x7f802c198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:12.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.255+0000 7f8031236640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f802c100740 0x7f802c198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50606/0 (socket says 172.21.15.33:50606) 2024-03-19T03:04:12.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.255+0000 7f8031236640 1 -- 172.21.15.33:0/688531968 learned_addr learned my addr 172.21.15.33:0/688531968 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:04:12.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.255+0000 7f8031a37640 1 --2- 172.21.15.33:0/688531968 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f802c106750 0x7f802c19c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:12.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.255+0000 7f8030a35640 1 --2- 172.21.15.33:0/688531968 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f802c101100 0x7f802c198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:12.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.255+0000 7f8031236640 1 -- 172.21.15.33:0/688531968 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f802c101100 msgr2=0x7f802c198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:12.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.255+0000 7f8031236640 1 --2- 172.21.15.33:0/688531968 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f802c101100 0x7f802c198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:12.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.255+0000 7f8031236640 1 -- 172.21.15.33:0/688531968 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f802c106750 msgr2=0x7f802c19c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:12.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.255+0000 7f8031236640 1 --2- 172.21.15.33:0/688531968 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f802c106750 0x7f802c19c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:12.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.255+0000 7f8031236640 1 -- 172.21.15.33:0/688531968 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f802c19d020 con 0x7f802c100740 2024-03-19T03:04:12.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.255+0000 7f8031a37640 1 --2- 172.21.15.33:0/688531968 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f802c106750 0x7f802c19c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:04:12.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.256+0000 7f8031236640 1 --2- 172.21.15.33:0/688531968 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f802c100740 0x7f802c198030 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f802800e9f0 tx=0x7f802800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:12.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.256+0000 7f80227fc640 1 -- 172.21.15.33:0/688531968 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f802800cd80 con 0x7f802c100740 2024-03-19T03:04:12.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.256+0000 7f803349a640 1 -- 172.21.15.33:0/688531968 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f802c19d310 con 0x7f802c100740 2024-03-19T03:04:12.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.256+0000 7f80227fc640 1 -- 172.21.15.33:0/688531968 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f80280108e0 con 0x7f802c100740 2024-03-19T03:04:12.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.256+0000 7f80227fc640 1 -- 172.21.15.33:0/688531968 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8028018920 con 0x7f802c100740 2024-03-19T03:04:12.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.256+0000 7f803349a640 1 -- 172.21.15.33:0/688531968 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f802c1a4ba0 con 0x7f802c100740 2024-03-19T03:04:12.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.257+0000 7f803349a640 1 -- 172.21.15.33:0/688531968 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f802c068a60 con 0x7f802c100740 2024-03-19T03:04:12.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.258+0000 7f80227fc640 1 -- 172.21.15.33:0/688531968 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f8028010a90 con 0x7f802c100740 2024-03-19T03:04:12.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.258+0000 7f80227fc640 1 --2- 172.21.15.33:0/688531968 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f80180a2970 0x7f80180a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:12.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.259+0000 7f8030a35640 1 --2- 172.21.15.33:0/688531968 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f80180a2970 0x7f80180a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:12.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.259+0000 7f80227fc640 1 -- 172.21.15.33:0/688531968 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9584+0+0 (secure 0 0 0) 0x7f80280cf9c0 con 0x7f802c100740 2024-03-19T03:04:12.261 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.259+0000 7f8030a35640 1 --2- 172.21.15.33:0/688531968 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f80180a2970 0x7f80180a4e30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f801c005ab0 tx=0x7f801c005fd0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:12.263 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.262+0000 7f80227fc640 1 -- 172.21.15.33:0/688531968 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f802809bc70 con 0x7f802c100740 2024-03-19T03:04:12.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:11 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/45676194' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:04:12.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:11 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/4099832040' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:04:12.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.445+0000 7f803349a640 1 -- 172.21.15.33:0/688531968 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f802c108ce0 con 0x7f802c100740 2024-03-19T03:04:12.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.446+0000 7f80227fc640 1 -- 172.21.15.33:0/688531968 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v107) ==== 100+0+11 (secure 0 0 0) 0x7f80280860a0 con 0x7f802c100740 2024-03-19T03:04:12.447 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 64 2024-03-19T03:04:12.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.447+0000 7f803349a640 1 -- 172.21.15.33:0/688531968 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f80180a2970 msgr2=0x7f80180a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:12.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.447+0000 7f803349a640 1 --2- 172.21.15.33:0/688531968 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f80180a2970 0x7f80180a4e30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f801c005ab0 tx=0x7f801c005fd0 comp rx=0 tx=0).stop 2024-03-19T03:04:12.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.447+0000 7f803349a640 1 -- 172.21.15.33:0/688531968 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f802c100740 msgr2=0x7f802c198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:12.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.447+0000 7f803349a640 1 --2- 172.21.15.33:0/688531968 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f802c100740 0x7f802c198030 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f802800e9f0 tx=0x7f802800eeb0 comp rx=0 tx=0).stop 2024-03-19T03:04:12.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.447+0000 7f803349a640 1 -- 172.21.15.33:0/688531968 shutdown_connections 2024-03-19T03:04:12.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.447+0000 7f803349a640 1 --2- 172.21.15.33:0/688531968 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f80180a2970 0x7f80180a4e30 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:12.449 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.447+0000 7f803349a640 1 --2- 172.21.15.33:0/688531968 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f802c101100 0x7f802c198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:12.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.447+0000 7f803349a640 1 --2- 172.21.15.33:0/688531968 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f802c106750 0x7f802c19c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:12.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.447+0000 7f803349a640 1 --2- 172.21.15.33:0/688531968 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f802c100740 0x7f802c198030 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:12.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.447+0000 7f803349a640 1 -- 172.21.15.33:0/688531968 >> 172.21.15.33:0/688531968 conn(0x7f802c077880 msgr2=0x7f802c1033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:12.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.448+0000 7f803349a640 1 -- 172.21.15.33:0/688531968 shutdown_connections 2024-03-19T03:04:12.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.448+0000 7f803349a640 1 -- 172.21.15.33:0/688531968 wait complete. 2024-03-19T03:04:12.459 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T03:04:12.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.563+0000 7f003ecb2640 1 -- 172.21.15.33:0/2215578230 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00381066a0 msgr2=0x7f0038106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:12.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.564+0000 7f003ecb2640 1 --2- 172.21.15.33:0/2215578230 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00381066a0 0x7f0038106a80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0030009ca0 tx=0x7f003002c0f0 comp rx=0 tx=0).stop 2024-03-19T03:04:12.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.564+0000 7f003ecb2640 1 -- 172.21.15.33:0/2215578230 shutdown_connections 2024-03-19T03:04:12.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.564+0000 7f003ecb2640 1 --2- 172.21.15.33:0/2215578230 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0038100690 0x7f0038100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:12.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.564+0000 7f003ecb2640 1 --2- 172.21.15.33:0/2215578230 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0038101050 0x7f003810d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:12.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.564+0000 7f003ecb2640 1 --2- 172.21.15.33:0/2215578230 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00381066a0 0x7f0038106a80 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:12.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.564+0000 7f003ecb2640 1 -- 172.21.15.33:0/2215578230 >> 172.21.15.33:0/2215578230 conn(0x7f0038077820 msgr2=0x7f0038077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:12.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.564+0000 7f003ecb2640 1 -- 172.21.15.33:0/2215578230 shutdown_connections 2024-03-19T03:04:12.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.564+0000 7f003ecb2640 1 -- 172.21.15.33:0/2215578230 wait complete. 2024-03-19T03:04:12.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.564+0000 7f003ecb2640 1 Processor -- start 2024-03-19T03:04:12.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.564+0000 7f003ecb2640 1 -- start start 2024-03-19T03:04:12.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.564+0000 7f003ecb2640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0038100690 0x7f0038198e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:12.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.564+0000 7f003ecb2640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0038101050 0x7f0038199380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:12.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.564+0000 7f003ecb2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00381066a0 0x7f00381a3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:12.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.564+0000 7f003ecb2640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f0038110960 con 0x7f0038100690 2024-03-19T03:04:12.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.564+0000 7f003ecb2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f00381107e0 con 0x7f00381066a0 2024-03-19T03:04:12.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.564+0000 7f003ecb2640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f0038110ae0 con 0x7f0038101050 2024-03-19T03:04:12.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.564+0000 7f003ca4e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0038100690 0x7f0038198e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:12.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.564+0000 7f003ca4e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0038100690 0x7f0038198e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50626/0 (socket says 172.21.15.33:50626) 2024-03-19T03:04:12.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.564+0000 7f003ca4e640 1 -- 172.21.15.33:0/2277697329 learned_addr learned my addr 172.21.15.33:0/2277697329 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:04:12.568 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.564+0000 7f003d24f640 1 --2- 172.21.15.33:0/2277697329 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00381066a0 0x7f00381a3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:12.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.565+0000 7f0037fff640 1 --2- 172.21.15.33:0/2277697329 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0038101050 0x7f0038199380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:12.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.565+0000 7f003ca4e640 1 -- 172.21.15.33:0/2277697329 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0038101050 msgr2=0x7f0038199380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:12.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.565+0000 7f003ca4e640 1 --2- 172.21.15.33:0/2277697329 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0038101050 0x7f0038199380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:12.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.565+0000 7f003ca4e640 1 -- 172.21.15.33:0/2277697329 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00381066a0 msgr2=0x7f00381a3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:12.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.565+0000 7f003ca4e640 1 --2- 172.21.15.33:0/2277697329 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00381066a0 0x7f00381a3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:12.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.565+0000 7f003ca4e640 1 -- 172.21.15.33:0/2277697329 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00381a4400 con 0x7f0038100690 2024-03-19T03:04:12.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.565+0000 7f003ca4e640 1 --2- 172.21.15.33:0/2277697329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0038100690 0x7f0038198e40 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f0030005e30 tx=0x7f0030035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:12.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.565+0000 7f0035ffb640 1 -- 172.21.15.33:0/2277697329 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0030039040 con 0x7f0038100690 2024-03-19T03:04:12.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.565+0000 7f003ecb2640 1 -- 172.21.15.33:0/2277697329 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00381a45d0 con 0x7f0038100690 2024-03-19T03:04:12.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.565+0000 7f0035ffb640 1 -- 172.21.15.33:0/2277697329 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0030044070 con 0x7f0038100690 2024-03-19T03:04:12.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.565+0000 7f0035ffb640 1 -- 172.21.15.33:0/2277697329 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00300329b0 con 0x7f0038100690 2024-03-19T03:04:12.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.565+0000 7f003ecb2640 1 -- 172.21.15.33:0/2277697329 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00381a4a30 con 0x7f0038100690 2024-03-19T03:04:12.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.566+0000 7f003ecb2640 1 -- 172.21.15.33:0/2277697329 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0038068610 con 0x7f0038100690 2024-03-19T03:04:12.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.567+0000 7f0035ffb640 1 -- 172.21.15.33:0/2277697329 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f0030003970 con 0x7f0038100690 2024-03-19T03:04:12.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.568+0000 7f0035ffb640 1 --2- 172.21.15.33:0/2277697329 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f00200a2970 0x7f00200a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:12.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.568+0000 7f0037fff640 1 --2- 172.21.15.33:0/2277697329 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f00200a2970 0x7f00200a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:12.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.568+0000 7f0035ffb640 1 -- 172.21.15.33:0/2277697329 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9584+0+0 (secure 0 0 0) 0x7f00300f2080 con 0x7f0038100690 2024-03-19T03:04:12.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.569+0000 7f0037fff640 1 --2- 172.21.15.33:0/2277697329 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f00200a2970 0x7f00200a4e30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f00380694d0 tx=0x7f002c005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:12.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.571+0000 7f0035ffb640 1 -- 172.21.15.33:0/2277697329 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f00300bead0 con 0x7f0038100690 2024-03-19T03:04:12.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.754+0000 7f003ecb2640 1 -- 172.21.15.33:0/2277697329 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f0038108c20 con 0x7f0038100690 2024-03-19T03:04:12.756 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.755+0000 7f0035ffb640 1 -- 172.21.15.33:0/2277697329 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v107) ==== 103+0+11 (secure 0 0 0) 0x7f00300be3d0 con 0x7f0038100690 2024-03-19T03:04:12.756 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 32 2024-03-19T03:04:12.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.757+0000 7f003ecb2640 1 -- 172.21.15.33:0/2277697329 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f00200a2970 msgr2=0x7f00200a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:12.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.757+0000 7f003ecb2640 1 --2- 172.21.15.33:0/2277697329 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f00200a2970 0x7f00200a4e30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f00380694d0 tx=0x7f002c005f90 comp rx=0 tx=0).stop 2024-03-19T03:04:12.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.757+0000 7f003ecb2640 1 -- 172.21.15.33:0/2277697329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0038100690 msgr2=0x7f0038198e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:12.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.757+0000 7f003ecb2640 1 --2- 172.21.15.33:0/2277697329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0038100690 0x7f0038198e40 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f0030005e30 tx=0x7f0030035d60 comp rx=0 tx=0).stop 2024-03-19T03:04:12.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.757+0000 7f003ecb2640 1 -- 172.21.15.33:0/2277697329 shutdown_connections 2024-03-19T03:04:12.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.757+0000 7f003ecb2640 1 --2- 172.21.15.33:0/2277697329 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f00200a2970 0x7f00200a4e30 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:12.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.757+0000 7f003ecb2640 1 --2- 172.21.15.33:0/2277697329 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0038101050 0x7f0038199380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:12.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.757+0000 7f003ecb2640 1 --2- 172.21.15.33:0/2277697329 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00381066a0 0x7f00381a3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:12.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.757+0000 7f003ecb2640 1 --2- 172.21.15.33:0/2277697329 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f0038100690 0x7f0038198e40 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:12.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.757+0000 7f003ecb2640 1 -- 172.21.15.33:0/2277697329 >> 172.21.15.33:0/2277697329 conn(0x7f0038077820 msgr2=0x7f0038102ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:12.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.757+0000 7f003ecb2640 1 -- 172.21.15.33:0/2277697329 shutdown_connections 2024-03-19T03:04:12.759 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.757+0000 7f003ecb2640 1 -- 172.21.15.33:0/2277697329 wait complete. 2024-03-19T03:04:12.769 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-19T03:04:12.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.875+0000 7fecb44b8640 1 -- 172.21.15.33:0/978335378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecac1066e0 msgr2=0x7fecac106ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:12.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.875+0000 7fecb44b8640 1 --2- 172.21.15.33:0/978335378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecac1066e0 0x7fecac106ac0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7feca4009930 tx=0x7feca402bc30 comp rx=0 tx=0).stop 2024-03-19T03:04:12.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.875+0000 7fecb44b8640 1 -- 172.21.15.33:0/978335378 shutdown_connections 2024-03-19T03:04:12.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.875+0000 7fecb44b8640 1 --2- 172.21.15.33:0/978335378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecac1006d0 0x7fecac100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:12.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.875+0000 7fecb44b8640 1 --2- 172.21.15.33:0/978335378 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fecac101090 0x7fecac10d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:12.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.875+0000 7fecb44b8640 1 --2- 172.21.15.33:0/978335378 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecac1066e0 0x7fecac106ac0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:12.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.875+0000 7fecb44b8640 1 -- 172.21.15.33:0/978335378 >> 172.21.15.33:0/978335378 conn(0x7fecac076aa0 msgr2=0x7fecac076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:12.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.875+0000 7fecb44b8640 1 -- 172.21.15.33:0/978335378 shutdown_connections 2024-03-19T03:04:12.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.875+0000 7fecb44b8640 1 -- 172.21.15.33:0/978335378 wait complete. 2024-03-19T03:04:12.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.876+0000 7fecb44b8640 1 Processor -- start 2024-03-19T03:04:12.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.876+0000 7fecb44b8640 1 -- start start 2024-03-19T03:04:12.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.876+0000 7fecb44b8640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecac1006d0 0x7fecac0fdee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:12.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.876+0000 7fecb44b8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fecac101090 0x7fecac0fe420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:12.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.876+0000 7fecb44b8640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecac1066e0 0x7fecac0fe960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:12.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.876+0000 7fecb44b8640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fecac1109a0 con 0x7fecac1066e0 2024-03-19T03:04:12.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.876+0000 7fecb44b8640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fecac110820 con 0x7fecac101090 2024-03-19T03:04:12.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.876+0000 7fecb44b8640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fecac110b20 con 0x7fecac1006d0 2024-03-19T03:04:12.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.876+0000 7fecb2a55640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecac1066e0 0x7fecac0fe960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:12.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.876+0000 7fecb2a55640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecac1066e0 0x7fecac0fe960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50658/0 (socket says 172.21.15.33:50658) 2024-03-19T03:04:12.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.876+0000 7fecb2a55640 1 -- 172.21.15.33:0/2093490956 learned_addr learned my addr 172.21.15.33:0/2093490956 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:04:12.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.876+0000 7fecb1a53640 1 --2- 172.21.15.33:0/2093490956 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fecac101090 0x7fecac0fe420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:12.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.876+0000 7fecb2254640 1 --2- 172.21.15.33:0/2093490956 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecac1006d0 0x7fecac0fdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:12.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.877+0000 7fecb2a55640 1 -- 172.21.15.33:0/2093490956 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecac1006d0 msgr2=0x7fecac0fdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:12.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.877+0000 7fecb2a55640 1 --2- 172.21.15.33:0/2093490956 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecac1006d0 0x7fecac0fdee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:12.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.877+0000 7fecb2a55640 1 -- 172.21.15.33:0/2093490956 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fecac101090 msgr2=0x7fecac0fe420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:12.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.877+0000 7fecb2a55640 1 --2- 172.21.15.33:0/2093490956 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fecac101090 0x7fecac0fe420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:12.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.877+0000 7fecb2a55640 1 -- 172.21.15.33:0/2093490956 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecac1a6550 con 0x7fecac1066e0 2024-03-19T03:04:12.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.877+0000 7fecb1a53640 1 --2- 172.21.15.33:0/2093490956 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fecac101090 0x7fecac0fe420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:04:12.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.877+0000 7fecb2a55640 1 --2- 172.21.15.33:0/2093490956 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecac1066e0 0x7fecac0fe960 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fec9c00eb10 tx=0x7fec9c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:12.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.877+0000 7fecab7fe640 1 -- 172.21.15.33:0/2093490956 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec9c004b20 con 0x7fecac1066e0 2024-03-19T03:04:12.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.877+0000 7fecab7fe640 1 -- 172.21.15.33:0/2093490956 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fec9c004cc0 con 0x7fecac1066e0 2024-03-19T03:04:12.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.877+0000 7fecb44b8640 1 -- 172.21.15.33:0/2093490956 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecac1a6840 con 0x7fecac1066e0 2024-03-19T03:04:12.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.877+0000 7fecab7fe640 1 -- 172.21.15.33:0/2093490956 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec9c010690 con 0x7fecac1066e0 2024-03-19T03:04:12.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.877+0000 7fecb44b8640 1 -- 172.21.15.33:0/2093490956 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecac1a6d40 con 0x7fecac1066e0 2024-03-19T03:04:12.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.878+0000 7fecb44b8640 1 -- 172.21.15.33:0/2093490956 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecac101e10 con 0x7fecac1066e0 2024-03-19T03:04:12.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.879+0000 7fecab7fe640 1 -- 172.21.15.33:0/2093490956 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fec9c005000 con 0x7fecac1066e0 2024-03-19T03:04:12.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.880+0000 7fecab7fe640 1 --2- 172.21.15.33:0/2093490956 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fec940a2970 0x7fec940a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:12.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.880+0000 7fecb2254640 1 --2- 172.21.15.33:0/2093490956 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fec940a2970 0x7fec940a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:12.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.880+0000 7fecab7fe640 1 -- 172.21.15.33:0/2093490956 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9584+0+0 (secure 0 0 0) 0x7fec9c0cf9e0 con 0x7fecac1066e0 2024-03-19T03:04:12.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.880+0000 7fecb2254640 1 --2- 172.21.15.33:0/2093490956 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fec940a2970 0x7fec940a4e30 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7feca4005ab0 tx=0x7feca4005fd0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:12.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:12.883+0000 7fecab7fe640 1 -- 172.21.15.33:0/2093490956 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fec9c053740 con 0x7fecac1066e0 2024-03-19T03:04:13.028 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:12 smithi033 ceph-mon[31973]: pgmap v334: 129 pgs: 129 active+clean; 1.9 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-19T03:04:13.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:12 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/87332189' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:04:13.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:12 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/688531968' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:04:13.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:12 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2277697329' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:04:13.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.081+0000 7fecb44b8640 1 -- 172.21.15.33:0/2093490956 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fecac02d1f0 con 0x7fecac1066e0 2024-03-19T03:04:13.082 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.081+0000 7fecab7fe640 1 -- 172.21.15.33:0/2093490956 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v107) ==== 81+0+942 (secure 0 0 0) 0x7fec9c09c2e0 con 0x7fecac1066e0 2024-03-19T03:04:13.082 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:6800/1453067701 2024-03-20T02:53:26.937131+0000 2024-03-19T03:04:13.083 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/3554132372 2024-03-20T02:53:26.937131+0000 2024-03-19T03:04:13.083 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/4099467400 2024-03-20T02:53:26.937131+0000 2024-03-19T03:04:13.083 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/176174888 2024-03-20T02:53:26.937131+0000 2024-03-19T03:04:13.083 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:6801/906645748 2024-03-20T02:43:39.622218+0000 2024-03-19T03:04:13.083 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:6800/906645748 2024-03-20T02:43:39.622218+0000 2024-03-19T03:04:13.083 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/1067768047 2024-03-20T02:43:39.622218+0000 2024-03-19T03:04:13.083 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/729379453 2024-03-20T02:43:11.918076+0000 2024-03-19T03:04:13.083 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/2586588742 2024-03-20T02:43:39.622218+0000 2024-03-19T03:04:13.083 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/2932814165 2024-03-20T02:43:11.918076+0000 2024-03-19T03:04:13.083 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/1950732854 2024-03-20T02:53:26.937131+0000 2024-03-19T03:04:13.083 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:6801/3342667183 2024-03-20T02:43:11.918076+0000 2024-03-19T03:04:13.083 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:6801/1453067701 2024-03-20T02:53:26.937131+0000 2024-03-19T03:04:13.083 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/2212564963 2024-03-20T02:43:11.918076+0000 2024-03-19T03:04:13.083 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:6800/3342667183 2024-03-20T02:43:11.918076+0000 2024-03-19T03:04:13.084 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/3275126058 2024-03-20T02:43:39.622218+0000 2024-03-19T03:04:13.084 INFO:teuthology.orchestra.run.smithi033.stderr:listed 16 entries 2024-03-19T03:04:13.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.083+0000 7fecb44b8640 1 -- 172.21.15.33:0/2093490956 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fec940a2970 msgr2=0x7fec940a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:13.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.083+0000 7fecb44b8640 1 --2- 172.21.15.33:0/2093490956 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fec940a2970 0x7fec940a4e30 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7feca4005ab0 tx=0x7feca4005fd0 comp rx=0 tx=0).stop 2024-03-19T03:04:13.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.083+0000 7fecb44b8640 1 -- 172.21.15.33:0/2093490956 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecac1066e0 msgr2=0x7fecac0fe960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:13.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.083+0000 7fecb44b8640 1 --2- 172.21.15.33:0/2093490956 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecac1066e0 0x7fecac0fe960 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fec9c00eb10 tx=0x7fec9c00c2d0 comp rx=0 tx=0).stop 2024-03-19T03:04:13.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.083+0000 7fecb44b8640 1 -- 172.21.15.33:0/2093490956 shutdown_connections 2024-03-19T03:04:13.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.083+0000 7fecb44b8640 1 --2- 172.21.15.33:0/2093490956 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fec940a2970 0x7fec940a4e30 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:13.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.083+0000 7fecb44b8640 1 --2- 172.21.15.33:0/2093490956 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fecac1066e0 0x7fecac0fe960 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:13.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.083+0000 7fecb44b8640 1 --2- 172.21.15.33:0/2093490956 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fecac101090 0x7fecac0fe420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:13.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.083+0000 7fecb44b8640 1 --2- 172.21.15.33:0/2093490956 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecac1006d0 0x7fecac0fdee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:13.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.083+0000 7fecb44b8640 1 -- 172.21.15.33:0/2093490956 >> 172.21.15.33:0/2093490956 conn(0x7fecac076aa0 msgr2=0x7fecac06a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:13.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.083+0000 7fecb44b8640 1 -- 172.21.15.33:0/2093490956 shutdown_connections 2024-03-19T03:04:13.085 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.083+0000 7fecb44b8640 1 -- 172.21.15.33:0/2093490956 wait complete. 2024-03-19T03:04:13.095 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-19T03:04:13.095 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T03:04:13.112 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T03:04:13.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:12 smithi096 ceph-mon[35946]: pgmap v334: 129 pgs: 129 active+clean; 1.9 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-19T03:04:13.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:12 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/87332189' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:04:13.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:12 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/688531968' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:04:13.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:12 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2277697329' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:04:13.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.253+0000 7fb312d90640 1 -- 172.21.15.33:0/4153284009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb30c106be0 msgr2=0x7fb30c10e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:13.256 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.253+0000 7fb312d90640 1 --2- 172.21.15.33:0/4153284009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb30c106be0 0x7fb30c10e7f0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fb304009ca0 tx=0x7fb30402c0f0 comp rx=0 tx=0).stop 2024-03-19T03:04:13.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.253+0000 7fb312d90640 1 -- 172.21.15.33:0/4153284009 shutdown_connections 2024-03-19T03:04:13.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.253+0000 7fb312d90640 1 --2- 172.21.15.33:0/4153284009 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb30c100dd0 0x7fb30c106500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:13.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.253+0000 7fb312d90640 1 --2- 172.21.15.33:0/4153284009 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb30c106be0 0x7fb30c10e7f0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:13.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.253+0000 7fb312d90640 1 --2- 172.21.15.33:0/4153284009 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb30c100420 0x7fb30c100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:13.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.253+0000 7fb312d90640 1 -- 172.21.15.33:0/4153284009 >> 172.21.15.33:0/4153284009 conn(0x7fb30c075a30 msgr2=0x7fb30c075e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:13.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.253+0000 7fb312d90640 1 -- 172.21.15.33:0/4153284009 shutdown_connections 2024-03-19T03:04:13.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.253+0000 7fb312d90640 1 -- 172.21.15.33:0/4153284009 wait complete. 2024-03-19T03:04:13.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.253+0000 7fb312d90640 1 Processor -- start 2024-03-19T03:04:13.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.254+0000 7fb312d90640 1 -- start start 2024-03-19T03:04:13.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.254+0000 7fb312d90640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb30c100420 0x7fb30c1980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:13.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.254+0000 7fb312d90640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb30c106be0 0x7fb30c198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:13.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.254+0000 7fb312d90640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb30c19c950 0x7fb30c19ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:13.257 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.254+0000 7fb312d90640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fb30c111b80 con 0x7fb30c100420 2024-03-19T03:04:13.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.254+0000 7fb312d90640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb30c111a00 con 0x7fb30c19c950 2024-03-19T03:04:13.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.254+0000 7fb312d90640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb30c111d00 con 0x7fb30c106be0 2024-03-19T03:04:13.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.254+0000 7fb310b2c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb30c100420 0x7fb30c1980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:13.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.254+0000 7fb310b2c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb30c100420 0x7fb30c1980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:50674/0 (socket says 172.21.15.33:50674) 2024-03-19T03:04:13.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.254+0000 7fb310b2c640 1 -- 172.21.15.33:0/2597273608 learned_addr learned my addr 172.21.15.33:0/2597273608 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:04:13.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.254+0000 7fb31132d640 1 --2- 172.21.15.33:0/2597273608 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb30c19c950 0x7fb30c19ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:13.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.254+0000 7fb30bfff640 1 --2- 172.21.15.33:0/2597273608 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb30c106be0 0x7fb30c198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:13.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.254+0000 7fb310b2c640 1 -- 172.21.15.33:0/2597273608 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb30c106be0 msgr2=0x7fb30c198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:13.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.254+0000 7fb310b2c640 1 --2- 172.21.15.33:0/2597273608 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb30c106be0 0x7fb30c198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:13.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.254+0000 7fb310b2c640 1 -- 172.21.15.33:0/2597273608 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb30c19c950 msgr2=0x7fb30c19ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:13.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.254+0000 7fb310b2c640 1 --2- 172.21.15.33:0/2597273608 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb30c19c950 0x7fb30c19ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:13.258 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.254+0000 7fb310b2c640 1 -- 172.21.15.33:0/2597273608 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb30c19d440 con 0x7fb30c100420 2024-03-19T03:04:13.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.254+0000 7fb310b2c640 1 --2- 172.21.15.33:0/2597273608 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb30c100420 0x7fb30c1980e0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc00ea60 tx=0x7fb2fc00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:13.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.254+0000 7fb309ffb640 1 -- 172.21.15.33:0/2597273608 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2fc00cde0 con 0x7fb30c100420 2024-03-19T03:04:13.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.254+0000 7fb309ffb640 1 -- 172.21.15.33:0/2597273608 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb2fc0035f0 con 0x7fb30c100420 2024-03-19T03:04:13.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.254+0000 7fb312d90640 1 -- 172.21.15.33:0/2597273608 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb30c19d730 con 0x7fb30c100420 2024-03-19T03:04:13.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.254+0000 7fb309ffb640 1 -- 172.21.15.33:0/2597273608 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2fc010690 con 0x7fb30c100420 2024-03-19T03:04:13.259 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.255+0000 7fb312d90640 1 -- 172.21.15.33:0/2597273608 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb30c1a4f20 con 0x7fb30c100420 2024-03-19T03:04:13.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.256+0000 7fb312d90640 1 -- 172.21.15.33:0/2597273608 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb30c069950 con 0x7fb30c100420 2024-03-19T03:04:13.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.257+0000 7fb309ffb640 1 -- 172.21.15.33:0/2597273608 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fb2fc0037a0 con 0x7fb30c100420 2024-03-19T03:04:13.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.257+0000 7fb309ffb640 1 --2- 172.21.15.33:0/2597273608 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fb2f80a2970 0x7fb2f80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:13.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.257+0000 7fb309ffb640 1 -- 172.21.15.33:0/2597273608 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9584+0+0 (secure 0 0 0) 0x7fb2fc0cf750 con 0x7fb30c100420 2024-03-19T03:04:13.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.257+0000 7fb30bfff640 1 --2- 172.21.15.33:0/2597273608 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fb2f80a2970 0x7fb2f80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:13.260 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.258+0000 7fb30bfff640 1 --2- 172.21.15.33:0/2597273608 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fb2f80a2970 0x7fb2f80a4e30 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb30c0ffd10 tx=0x7fb2f4005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:13.262 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.261+0000 7fb309ffb640 1 -- 172.21.15.33:0/2597273608 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb2fc053740 con 0x7fb30c100420 2024-03-19T03:04:13.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:12 smithi155 ceph-mon[35804]: pgmap v334: 129 pgs: 129 active+clean; 1.9 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-19T03:04:13.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:12 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/87332189' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:04:13.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:12 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/688531968' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:04:13.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:12 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2277697329' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:04:13.446 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.445+0000 7fb312d90640 1 -- 172.21.15.33:0/2597273608 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb30c068a60 con 0x7fb30c100420 2024-03-19T03:04:13.447 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.446+0000 7fb309ffb640 1 -- 172.21.15.33:0/2597273608 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) ==== 74+0+25361 (secure 0 0 0) 0x7fb2fc09c090 con 0x7fb30c100420 2024-03-19T03:04:13.448 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T03:04:13.448 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":107,"fsid":"2b7a8938-e59a-11ee-95c9-87774f69a715","created":"2024-03-19T02:42:37.912179+0000","modified":"2024-03-19T03:04:07.834958+0000","last_up_change":"2024-03-19T02:50:01.539392+0000","last_in_change":"2024-03-19T02:49:40.534415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:46:09.963737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:50:50.069641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:50:51.381064+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:50:55.476667+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"090a89ab-6b7b-4317-8823-178d4212b2f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6803","nonce":1633124995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6805","nonce":1633124995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6809","nonce":1633124995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6807","nonce":1633124995}]},"public_addr":"172.21.15.33:6803/1633124995","cluster_addr":"172.21.15.33:6805/1633124995","heartbeat_back_addr":"172.21.15.33:6809/1633124995","heartbeat_front_addr":"172.21.15.33:6807/1633124995","state":["exists","up"]},{"osd":1,"uuid":"f50eec55-1956-4486-8593-f2c8fa610e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6801","nonce":340144398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6803","nonce":340144398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6807","nonce":340144398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6805","nonce":340144398}]},"public_addr":"172.21.15.96:6801/340144398","cluster_addr":"172.21.15.96:6803/340144398","heartbeat_back_addr":"172.21.15.96:6807/340144398","heartbeat_front_addr":"172.21.15.96:6805/340144398","state":["exists","up"]},{"osd":2,"uuid":"893a2d62-80d9-4e5e-bc6a-a1c8c681d652","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6801","nonce":3388743876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6803","nonce":3388743876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6807","nonce":3388743876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6805","nonce":3388743876}]},"public_addr":"172.21.15.155:6801/3388743876","cluster_addr":"172.21.15.155:6803/3388743876","heartbeat_back_addr":"172.21.15.155:6807/3388743876","heartbeat_front_addr":"172.21.15.155:6805/3388743876","state":["exists","up"]},{"osd":3,"uuid":"b07e0482-4281-4d32-9dfe-9995d7246175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6811","nonce":677007171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6813","nonce":677007171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6817","nonce":677007171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6815","nonce":677007171}]},"public_addr":"172.21.15.33:6811/677007171","cluster_addr":"172.21.15.33:6813/677007171","heartbeat_back_addr":"172.21.15.33:6817/677007171","heartbeat_front_addr":"172.21.15.33:6815/677007171","state":["exists","up"]},{"osd":4,"uuid":"a988384b-af6b-47c6-8c07-5c9673acee45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6809","nonce":3106158554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6811","nonce":3106158554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6815","nonce":3106158554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6813","nonce":3106158554}]},"public_addr":"172.21.15.96:6809/3106158554","cluster_addr":"172.21.15.96:6811/3106158554","heartbeat_back_addr":"172.21.15.96:6815/3106158554","heartbeat_front_addr":"172.21.15.96:6813/3106158554","state":["exists","up"]},{"osd":5,"uuid":"198dc992-96f8-4c52-96ee-1fd5de148bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6809","nonce":327293324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6811","nonce":327293324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6815","nonce":327293324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6813","nonce":327293324}]},"public_addr":"172.21.15.155:6809/327293324","cluster_addr":"172.21.15.155:6811/327293324","heartbeat_back_addr":"172.21.15.155:6815/327293324","heartbeat_front_addr":"172.21.15.155:6813/327293324","state":["exists","up"]},{"osd":6,"uuid":"a7278ec0-ab2e-49ed-be53-84d8b9e24177","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6819","nonce":873871509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6821","nonce":873871509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6825","nonce":873871509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6823","nonce":873871509}]},"public_addr":"172.21.15.33:6819/873871509","cluster_addr":"172.21.15.33:6821/873871509","heartbeat_back_addr":"172.21.15.33:6825/873871509","heartbeat_front_addr":"172.21.15.33:6823/873871509","state":["exists","up"]},{"osd":7,"uuid":"727e133d-aa24-4ffc-90e2-266eb2aab8c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6817","nonce":2334123891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6819","nonce":2334123891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6823","nonce":2334123891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6821","nonce":2334123891}]},"public_addr":"172.21.15.96:6817/2334123891","cluster_addr":"172.21.15.96:6819/2334123891","heartbeat_back_addr":"172.21.15.96:6823/2334123891","heartbeat_front_addr":"172.21.15.96:6821/2334123891","state":["exists","up"]},{"osd":8,"uuid":"bb20eec0-39e5-4b86-9db5-486b1840cdad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6817","nonce":3498516903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6819","nonce":3498516903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6823","nonce":3498516903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6821","nonce":3498516903}]},"public_addr":"172.21.15.155:6817/3498516903","cluster_addr":"172.21.15.155:6819/3498516903","heartbeat_back_addr":"172.21.15.155:6823/3498516903","heartbeat_front_addr":"172.21.15.155:6821/3498516903","state":["exists","up"]},{"osd":9,"uuid":"179c6228-3e23-4f65-91a6-1ef0005d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6827","nonce":760777213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6829","nonce":760777213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6833","nonce":760777213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6831","nonce":760777213}]},"public_addr":"172.21.15.33:6827/760777213","cluster_addr":"172.21.15.33:6829/760777213","heartbeat_back_addr":"172.21.15.33:6833/760777213","heartbeat_front_addr":"172.21.15.33:6831/760777213","state":["exists","up"]},{"osd":10,"uuid":"be17b478-3c2e-41fb-bede-958333acdf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6825","nonce":1404898482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6827","nonce":1404898482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6831","nonce":1404898482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6829","nonce":1404898482}]},"public_addr":"172.21.15.96:6825/1404898482","cluster_addr":"172.21.15.96:6827/1404898482","heartbeat_back_addr":"172.21.15.96:6831/1404898482","heartbeat_front_addr":"172.21.15.96:6829/1404898482","state":["exists","up"]},{"osd":11,"uuid":"c0903ed2-cc1e-4c18-8731-1f0ee4dd0457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6825","nonce":792352827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6827","nonce":792352827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6831","nonce":792352827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6829","nonce":792352827}]},"public_addr":"172.21.15.155:6825/792352827","cluster_addr":"172.21.15.155:6827/792352827","heartbeat_back_addr":"172.21.15.155:6831/792352827","heartbeat_front_addr":"172.21.15.155:6829/792352827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:17.942801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:42.109449+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:05.981599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:32.069574+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:57.431015+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:22.448592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:48.526515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:14.498352+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:40.175426+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:06.364006+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:32.853161+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:59.073538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.9","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:6800/1453067701":"2024-03-20T02:53:26.937131+0000","172.21.15.33:0/3554132372":"2024-03-20T02:53:26.937131+0000","172.21.15.33:0/4099467400":"2024-03-20T02:53:26.937131+0000","172.21.15.33:0/176174888":"2024-03-20T02:53:26.937131+0000","172.21.15.33:6801/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:6800/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/1067768047":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/729379453":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2586588742":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/2932814165":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/1950732854":"2024-03-20T02:53:26.937131+0000","172.21.15.33:6801/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6801/1453067701":"2024-03-20T02:53:26.937131+0000","172.21.15.33:0/2212564963":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6800/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/3275126058":"2024-03-20T02:43:39.622218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:04:13.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.448+0000 7fb312d90640 1 -- 172.21.15.33:0/2597273608 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fb2f80a2970 msgr2=0x7fb2f80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:13.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.448+0000 7fb312d90640 1 --2- 172.21.15.33:0/2597273608 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fb2f80a2970 0x7fb2f80a4e30 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb30c0ffd10 tx=0x7fb2f4005f90 comp rx=0 tx=0).stop 2024-03-19T03:04:13.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.448+0000 7fb312d90640 1 -- 172.21.15.33:0/2597273608 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb30c100420 msgr2=0x7fb30c1980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:13.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.448+0000 7fb312d90640 1 --2- 172.21.15.33:0/2597273608 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb30c100420 0x7fb30c1980e0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc00ea60 tx=0x7fb2fc00ef20 comp rx=0 tx=0).stop 2024-03-19T03:04:13.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.448+0000 7fb312d90640 1 -- 172.21.15.33:0/2597273608 shutdown_connections 2024-03-19T03:04:13.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.448+0000 7fb312d90640 1 --2- 172.21.15.33:0/2597273608 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fb2f80a2970 0x7fb2f80a4e30 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:13.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.448+0000 7fb312d90640 1 --2- 172.21.15.33:0/2597273608 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb30c106be0 0x7fb30c198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:13.450 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.448+0000 7fb312d90640 1 --2- 172.21.15.33:0/2597273608 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb30c19c950 0x7fb30c19ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:13.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.448+0000 7fb312d90640 1 --2- 172.21.15.33:0/2597273608 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fb30c100420 0x7fb30c1980e0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:13.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.448+0000 7fb312d90640 1 -- 172.21.15.33:0/2597273608 >> 172.21.15.33:0/2597273608 conn(0x7fb30c075a30 msgr2=0x7fb30c0fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:13.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.448+0000 7fb312d90640 1 -- 172.21.15.33:0/2597273608 shutdown_connections 2024-03-19T03:04:13.451 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.448+0000 7fb312d90640 1 -- 172.21.15.33:0/2597273608 wait complete. 2024-03-19T03:04:13.461 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T02:46:09.963737+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T02:50:50.069641+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T02:50:51.381064+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 107, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T02:50:55.476667+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 107, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T03:04:13.461 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T03:04:13.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.566+0000 7fe26fc7e640 1 -- 172.21.15.33:0/437066287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe268101050 msgr2=0x7fe26810d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:13.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.566+0000 7fe26fc7e640 1 --2- 172.21.15.33:0/437066287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe268101050 0x7fe26810d560 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fe260009cd0 tx=0x7fe26002c0f0 comp rx=0 tx=0).stop 2024-03-19T03:04:13.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.566+0000 7fe26fc7e640 1 -- 172.21.15.33:0/437066287 shutdown_connections 2024-03-19T03:04:13.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.566+0000 7fe26fc7e640 1 --2- 172.21.15.33:0/437066287 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe268100690 0x7fe268100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:13.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.566+0000 7fe26fc7e640 1 --2- 172.21.15.33:0/437066287 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe268101050 0x7fe26810d560 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:13.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.566+0000 7fe26fc7e640 1 --2- 172.21.15.33:0/437066287 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe2681066a0 0x7fe268106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:13.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.566+0000 7fe26fc7e640 1 -- 172.21.15.33:0/437066287 >> 172.21.15.33:0/437066287 conn(0x7fe268077820 msgr2=0x7fe268077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:13.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.566+0000 7fe26fc7e640 1 -- 172.21.15.33:0/437066287 shutdown_connections 2024-03-19T03:04:13.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.566+0000 7fe26fc7e640 1 -- 172.21.15.33:0/437066287 wait complete. 2024-03-19T03:04:13.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.566+0000 7fe26fc7e640 1 Processor -- start 2024-03-19T03:04:13.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.566+0000 7fe26fc7e640 1 -- start start 2024-03-19T03:04:13.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.567+0000 7fe26fc7e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe268100690 0x7fe268198dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:13.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.567+0000 7fe26fc7e640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe268101050 0x7fe268199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:13.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.567+0000 7fe26fc7e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2681066a0 0x7fe2681a3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:13.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.567+0000 7fe26fc7e640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe268110920 con 0x7fe268101050 2024-03-19T03:04:13.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.567+0000 7fe26fc7e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe2681107a0 con 0x7fe268100690 2024-03-19T03:04:13.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.567+0000 7fe26fc7e640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe268110aa0 con 0x7fe2681066a0 2024-03-19T03:04:13.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.567+0000 7fe26d219640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe268101050 0x7fe268199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:13.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.567+0000 7fe26d219640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe268101050 0x7fe268199300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51618/0 (socket says 172.21.15.33:51618) 2024-03-19T03:04:13.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.567+0000 7fe26d219640 1 -- 172.21.15.33:0/2350165028 learned_addr learned my addr 172.21.15.33:0/2350165028 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:04:13.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.567+0000 7fe26da1a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe268100690 0x7fe268198dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:13.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.567+0000 7fe26e21b640 1 --2- 172.21.15.33:0/2350165028 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2681066a0 0x7fe2681a3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:13.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.567+0000 7fe26d219640 1 -- 172.21.15.33:0/2350165028 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2681066a0 msgr2=0x7fe2681a3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:13.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.567+0000 7fe26d219640 1 --2- 172.21.15.33:0/2350165028 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2681066a0 0x7fe2681a3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:13.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.567+0000 7fe26d219640 1 -- 172.21.15.33:0/2350165028 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe268100690 msgr2=0x7fe268198dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:13.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.567+0000 7fe26d219640 1 --2- 172.21.15.33:0/2350165028 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe268100690 0x7fe268198dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:13.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.567+0000 7fe26d219640 1 -- 172.21.15.33:0/2350165028 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2681a4380 con 0x7fe268101050 2024-03-19T03:04:13.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.567+0000 7fe26d219640 1 --2- 172.21.15.33:0/2350165028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe268101050 0x7fe268199300 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fe25800d950 tx=0x7fe25800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:13.571 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.567+0000 7fe25effd640 1 -- 172.21.15.33:0/2350165028 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe258017030 con 0x7fe268101050 2024-03-19T03:04:13.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.567+0000 7fe25effd640 1 -- 172.21.15.33:0/2350165028 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe258004020 con 0x7fe268101050 2024-03-19T03:04:13.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.567+0000 7fe26fc7e640 1 -- 172.21.15.33:0/2350165028 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2681a4550 con 0x7fe268101050 2024-03-19T03:04:13.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.567+0000 7fe25effd640 1 -- 172.21.15.33:0/2350165028 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2580047a0 con 0x7fe268101050 2024-03-19T03:04:13.572 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.567+0000 7fe26fc7e640 1 -- 172.21.15.33:0/2350165028 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2681a4ac0 con 0x7fe268101050 2024-03-19T03:04:13.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.568+0000 7fe26fc7e640 1 -- 172.21.15.33:0/2350165028 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe268068610 con 0x7fe268101050 2024-03-19T03:04:13.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.570+0000 7fe25effd640 1 -- 172.21.15.33:0/2350165028 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fe2580041d0 con 0x7fe268101050 2024-03-19T03:04:13.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.571+0000 7fe25effd640 1 --2- 172.21.15.33:0/2350165028 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe2540a29f0 0x7fe2540a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:13.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.571+0000 7fe26da1a640 1 --2- 172.21.15.33:0/2350165028 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe2540a29f0 0x7fe2540a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:13.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.571+0000 7fe25effd640 1 -- 172.21.15.33:0/2350165028 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9584+0+0 (secure 0 0 0) 0x7fe2580cf2a0 con 0x7fe268101050 2024-03-19T03:04:13.573 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.571+0000 7fe26da1a640 1 --2- 172.21.15.33:0/2350165028 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe2540a29f0 0x7fe2540a4eb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe264005d00 tx=0x7fe26400b410 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:13.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.574+0000 7fe25effd640 1 -- 172.21.15.33:0/2350165028 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe25809bc20 con 0x7fe268101050 2024-03-19T03:04:13.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.756+0000 7fe26fc7e640 1 -- 172.21.15.33:0/2350165028 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe268108c20 con 0x7fe268101050 2024-03-19T03:04:13.758 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.757+0000 7fe25effd640 1 -- 172.21.15.33:0/2350165028 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v107) ==== 93+0+10 (secure 0 0 0) 0x7fe25809b520 con 0x7fe268101050 2024-03-19T03:04:13.758 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 1 2024-03-19T03:04:13.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.758+0000 7fe26fc7e640 1 -- 172.21.15.33:0/2350165028 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe2540a29f0 msgr2=0x7fe2540a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:13.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.758+0000 7fe26fc7e640 1 --2- 172.21.15.33:0/2350165028 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe2540a29f0 0x7fe2540a4eb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe264005d00 tx=0x7fe26400b410 comp rx=0 tx=0).stop 2024-03-19T03:04:13.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.758+0000 7fe26fc7e640 1 -- 172.21.15.33:0/2350165028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe268101050 msgr2=0x7fe268199300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:13.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.758+0000 7fe26fc7e640 1 --2- 172.21.15.33:0/2350165028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe268101050 0x7fe268199300 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fe25800d950 tx=0x7fe25800de10 comp rx=0 tx=0).stop 2024-03-19T03:04:13.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.758+0000 7fe26fc7e640 1 -- 172.21.15.33:0/2350165028 shutdown_connections 2024-03-19T03:04:13.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.758+0000 7fe26fc7e640 1 --2- 172.21.15.33:0/2350165028 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe2540a29f0 0x7fe2540a4eb0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:13.760 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.758+0000 7fe26fc7e640 1 --2- 172.21.15.33:0/2350165028 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2681066a0 0x7fe2681a3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:13.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.758+0000 7fe26fc7e640 1 --2- 172.21.15.33:0/2350165028 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe268101050 0x7fe268199300 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:13.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.758+0000 7fe26fc7e640 1 --2- 172.21.15.33:0/2350165028 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe268100690 0x7fe268198dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:13.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.758+0000 7fe26fc7e640 1 -- 172.21.15.33:0/2350165028 >> 172.21.15.33:0/2350165028 conn(0x7fe268077820 msgr2=0x7fe268102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:13.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.759+0000 7fe26fc7e640 1 -- 172.21.15.33:0/2350165028 shutdown_connections 2024-03-19T03:04:13.761 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.759+0000 7fe26fc7e640 1 -- 172.21.15.33:0/2350165028 wait complete. 2024-03-19T03:04:13.771 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T03:04:13.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.875+0000 7fa2930dd640 1 -- 172.21.15.33:0/3605469563 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa28c100740 msgr2=0x7fa28c100bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:13.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.875+0000 7fa2930dd640 1 --2- 172.21.15.33:0/3605469563 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa28c100740 0x7fa28c100bc0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fa278009930 tx=0x7fa27802bc30 comp rx=0 tx=0).stop 2024-03-19T03:04:13.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.876+0000 7fa2930dd640 1 -- 172.21.15.33:0/3605469563 shutdown_connections 2024-03-19T03:04:13.879 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.876+0000 7fa2930dd640 1 --2- 172.21.15.33:0/3605469563 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa28c101100 0x7fa28c10d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:13.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.876+0000 7fa2930dd640 1 --2- 172.21.15.33:0/3605469563 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa28c100740 0x7fa28c100bc0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:13.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.876+0000 7fa2930dd640 1 --2- 172.21.15.33:0/3605469563 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa28c106750 0x7fa28c106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:13.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.876+0000 7fa2930dd640 1 -- 172.21.15.33:0/3605469563 >> 172.21.15.33:0/3605469563 conn(0x7fa28c077880 msgr2=0x7fa28c077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:13.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.876+0000 7fa2930dd640 1 -- 172.21.15.33:0/3605469563 shutdown_connections 2024-03-19T03:04:13.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.876+0000 7fa2930dd640 1 -- 172.21.15.33:0/3605469563 wait complete. 2024-03-19T03:04:13.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.876+0000 7fa2930dd640 1 Processor -- start 2024-03-19T03:04:13.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.876+0000 7fa2930dd640 1 -- start start 2024-03-19T03:04:13.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.876+0000 7fa2930dd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa28c100740 0x7fa28c198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:13.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.876+0000 7fa2930dd640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa28c101100 0x7fa28c1985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:13.880 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.876+0000 7fa2930dd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa28c106750 0x7fa28c19c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:13.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.876+0000 7fa2930dd640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa28c111b10 con 0x7fa28c100740 2024-03-19T03:04:13.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.876+0000 7fa2930dd640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa28c111990 con 0x7fa28c106750 2024-03-19T03:04:13.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.876+0000 7fa2930dd640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa28c111c90 con 0x7fa28c101100 2024-03-19T03:04:13.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.876+0000 7fa290e79640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa28c100740 0x7fa28c198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:13.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.876+0000 7fa290e79640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa28c100740 0x7fa28c198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51626/0 (socket says 172.21.15.33:51626) 2024-03-19T03:04:13.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.876+0000 7fa290e79640 1 -- 172.21.15.33:0/3228402591 learned_addr learned my addr 172.21.15.33:0/3228402591 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:04:13.881 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.877+0000 7fa29167a640 1 --2- 172.21.15.33:0/3228402591 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa28c106750 0x7fa28c19c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:13.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.877+0000 7fa28bfff640 1 --2- 172.21.15.33:0/3228402591 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa28c101100 0x7fa28c1985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:13.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.877+0000 7fa290e79640 1 -- 172.21.15.33:0/3228402591 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa28c101100 msgr2=0x7fa28c1985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:13.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.877+0000 7fa290e79640 1 --2- 172.21.15.33:0/3228402591 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa28c101100 0x7fa28c1985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:13.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.877+0000 7fa290e79640 1 -- 172.21.15.33:0/3228402591 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa28c106750 msgr2=0x7fa28c19c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:13.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.877+0000 7fa290e79640 1 --2- 172.21.15.33:0/3228402591 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa28c106750 0x7fa28c19c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:13.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.877+0000 7fa290e79640 1 -- 172.21.15.33:0/3228402591 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa28c19d060 con 0x7fa28c100740 2024-03-19T03:04:13.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.877+0000 7fa290e79640 1 --2- 172.21.15.33:0/3228402591 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa28c100740 0x7fa28c198070 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fa28000e9f0 tx=0x7fa28000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:13.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.877+0000 7fa289ffb640 1 -- 172.21.15.33:0/3228402591 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa28000cd80 con 0x7fa28c100740 2024-03-19T03:04:13.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.877+0000 7fa2930dd640 1 -- 172.21.15.33:0/3228402591 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa28c19d350 con 0x7fa28c100740 2024-03-19T03:04:13.882 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.877+0000 7fa289ffb640 1 -- 172.21.15.33:0/3228402591 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa2800054d0 con 0x7fa28c100740 2024-03-19T03:04:13.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.877+0000 7fa289ffb640 1 -- 172.21.15.33:0/3228402591 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa280010490 con 0x7fa28c100740 2024-03-19T03:04:13.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.877+0000 7fa2930dd640 1 -- 172.21.15.33:0/3228402591 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa28c1a4be0 con 0x7fa28c100740 2024-03-19T03:04:13.883 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.878+0000 7fa2930dd640 1 -- 172.21.15.33:0/3228402591 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa28c068a60 con 0x7fa28c100740 2024-03-19T03:04:13.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.879+0000 7fa289ffb640 1 -- 172.21.15.33:0/3228402591 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fa280005020 con 0x7fa28c100740 2024-03-19T03:04:13.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.880+0000 7fa289ffb640 1 --2- 172.21.15.33:0/3228402591 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa27c0a2970 0x7fa27c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:13.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.880+0000 7fa289ffb640 1 -- 172.21.15.33:0/3228402591 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9584+0+0 (secure 0 0 0) 0x7fa2800cf610 con 0x7fa28c100740 2024-03-19T03:04:13.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.880+0000 7fa28bfff640 1 --2- 172.21.15.33:0/3228402591 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa27c0a2970 0x7fa27c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:13.884 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.880+0000 7fa28bfff640 1 --2- 172.21.15.33:0/3228402591 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa27c0a2970 0x7fa27c0a4e30 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa278006010 tx=0x7fa27802c670 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:13.885 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:13.884+0000 7fa289ffb640 1 -- 172.21.15.33:0/3228402591 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa28009bf50 con 0x7fa28c100740 2024-03-19T03:04:14.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:13 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2093490956' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T03:04:14.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:13 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2597273608' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:04:14.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:13 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2350165028' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:04:14.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.067+0000 7fa2930dd640 1 -- 172.21.15.33:0/3228402591 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa28c108ce0 con 0x7fa28c100740 2024-03-19T03:04:14.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.068+0000 7fa289ffb640 1 -- 172.21.15.33:0/3228402591 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v107) ==== 104+0+11 (secure 0 0 0) 0x7fa28009b850 con 0x7fa28c100740 2024-03-19T03:04:14.069 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 32 2024-03-19T03:04:14.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.070+0000 7fa2930dd640 1 -- 172.21.15.33:0/3228402591 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa27c0a2970 msgr2=0x7fa27c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:14.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.070+0000 7fa2930dd640 1 --2- 172.21.15.33:0/3228402591 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa27c0a2970 0x7fa27c0a4e30 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa278006010 tx=0x7fa27802c670 comp rx=0 tx=0).stop 2024-03-19T03:04:14.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.070+0000 7fa2930dd640 1 -- 172.21.15.33:0/3228402591 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa28c100740 msgr2=0x7fa28c198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:14.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.070+0000 7fa2930dd640 1 --2- 172.21.15.33:0/3228402591 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa28c100740 0x7fa28c198070 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fa28000e9f0 tx=0x7fa28000edf0 comp rx=0 tx=0).stop 2024-03-19T03:04:14.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.070+0000 7fa2930dd640 1 -- 172.21.15.33:0/3228402591 shutdown_connections 2024-03-19T03:04:14.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.070+0000 7fa2930dd640 1 --2- 172.21.15.33:0/3228402591 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa27c0a2970 0x7fa27c0a4e30 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:14.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.070+0000 7fa2930dd640 1 --2- 172.21.15.33:0/3228402591 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa28c101100 0x7fa28c1985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:14.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.070+0000 7fa2930dd640 1 --2- 172.21.15.33:0/3228402591 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa28c106750 0x7fa28c19c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:14.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.070+0000 7fa2930dd640 1 --2- 172.21.15.33:0/3228402591 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa28c100740 0x7fa28c198070 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:14.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.070+0000 7fa2930dd640 1 -- 172.21.15.33:0/3228402591 >> 172.21.15.33:0/3228402591 conn(0x7fa28c077880 msgr2=0x7fa28c103410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:14.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.070+0000 7fa2930dd640 1 -- 172.21.15.33:0/3228402591 shutdown_connections 2024-03-19T03:04:14.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.070+0000 7fa2930dd640 1 -- 172.21.15.33:0/3228402591 wait complete. 2024-03-19T03:04:14.082 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T03:04:14.190 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.187+0000 7f87ac8b5640 1 -- 172.21.15.33:0/4009864345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87a810be60 msgr2=0x7f87a810e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:14.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.187+0000 7f87ac8b5640 1 --2- 172.21.15.33:0/4009864345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87a810be60 0x7f87a810e2d0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f87a0009ca0 tx=0x7f87a002c180 comp rx=0 tx=0).stop 2024-03-19T03:04:14.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.187+0000 7f87ac8b5640 1 -- 172.21.15.33:0/4009864345 shutdown_connections 2024-03-19T03:04:14.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.187+0000 7f87ac8b5640 1 --2- 172.21.15.33:0/4009864345 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87a810be60 0x7f87a810e2d0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:14.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.187+0000 7f87ac8b5640 1 --2- 172.21.15.33:0/4009864345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87a81033b0 0x7f87a810b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:14.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.187+0000 7f87ac8b5640 1 --2- 172.21.15.33:0/4009864345 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87a8102a90 0x7f87a8102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:14.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.187+0000 7f87ac8b5640 1 -- 172.21.15.33:0/4009864345 >> 172.21.15.33:0/4009864345 conn(0x7f87a8077880 msgr2=0x7f87a8077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:14.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.187+0000 7f87ac8b5640 1 -- 172.21.15.33:0/4009864345 shutdown_connections 2024-03-19T03:04:14.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.187+0000 7f87ac8b5640 1 -- 172.21.15.33:0/4009864345 wait complete. 2024-03-19T03:04:14.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.188+0000 7f87ac8b5640 1 Processor -- start 2024-03-19T03:04:14.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.188+0000 7f87ac8b5640 1 -- start start 2024-03-19T03:04:14.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.188+0000 7f87ac8b5640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87a8102a90 0x7f87a8197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:14.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.188+0000 7f87ac8b5640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87a81033b0 0x7f87a8198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:14.191 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.188+0000 7f87ac8b5640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87a810be60 0x7f87a819c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:14.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.188+0000 7f87ac8b5640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f87a8111ae0 con 0x7f87a8102a90 2024-03-19T03:04:14.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.188+0000 7f87ac8b5640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f87a8111960 con 0x7f87a810be60 2024-03-19T03:04:14.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.188+0000 7f87ac8b5640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f87a8111c60 con 0x7f87a81033b0 2024-03-19T03:04:14.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.188+0000 7f87a659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87a8102a90 0x7f87a8197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:14.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.188+0000 7f87a659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87a8102a90 0x7f87a8197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51644/0 (socket says 172.21.15.33:51644) 2024-03-19T03:04:14.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.188+0000 7f87a659c640 1 -- 172.21.15.33:0/2255807732 learned_addr learned my addr 172.21.15.33:0/2255807732 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:04:14.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.188+0000 7f87a6d9d640 1 --2- 172.21.15.33:0/2255807732 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87a810be60 0x7f87a819c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:14.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.188+0000 7f879ffff640 1 --2- 172.21.15.33:0/2255807732 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87a81033b0 0x7f87a8198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:14.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.188+0000 7f87a659c640 1 -- 172.21.15.33:0/2255807732 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87a81033b0 msgr2=0x7f87a8198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:14.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.188+0000 7f87a659c640 1 --2- 172.21.15.33:0/2255807732 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87a81033b0 0x7f87a8198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:14.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.188+0000 7f87a659c640 1 -- 172.21.15.33:0/2255807732 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87a810be60 msgr2=0x7f87a819c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:14.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.188+0000 7f87a659c640 1 --2- 172.21.15.33:0/2255807732 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87a810be60 0x7f87a819c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:14.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.189+0000 7f87a659c640 1 -- 172.21.15.33:0/2255807732 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87a819cfd0 con 0x7f87a8102a90 2024-03-19T03:04:14.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.189+0000 7f87a659c640 1 --2- 172.21.15.33:0/2255807732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87a8102a90 0x7f87a8197fe0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f879800e9f0 tx=0x7f879800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:14.192 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.189+0000 7f879f7fe640 1 -- 172.21.15.33:0/2255807732 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f879800cd80 con 0x7f87a8102a90 2024-03-19T03:04:14.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.189+0000 7f87ac8b5640 1 -- 172.21.15.33:0/2255807732 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87a819d2c0 con 0x7f87a8102a90 2024-03-19T03:04:14.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.189+0000 7f879f7fe640 1 -- 172.21.15.33:0/2255807732 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f87980054d0 con 0x7f87a8102a90 2024-03-19T03:04:14.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.189+0000 7f879f7fe640 1 -- 172.21.15.33:0/2255807732 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8798010490 con 0x7f87a8102a90 2024-03-19T03:04:14.193 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.189+0000 7f87ac8b5640 1 -- 172.21.15.33:0/2255807732 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87a81a4b50 con 0x7f87a8102a90 2024-03-19T03:04:14.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.190+0000 7f87ac8b5640 1 -- 172.21.15.33:0/2255807732 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87a8068a60 con 0x7f87a8102a90 2024-03-19T03:04:14.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.191+0000 7f879f7fe640 1 -- 172.21.15.33:0/2255807732 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f8798005020 con 0x7f87a8102a90 2024-03-19T03:04:14.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.191+0000 7f879f7fe640 1 --2- 172.21.15.33:0/2255807732 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f87940a2970 0x7f87940a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:14.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.191+0000 7f879ffff640 1 --2- 172.21.15.33:0/2255807732 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f87940a2970 0x7f87940a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:14.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.192+0000 7f879f7fe640 1 -- 172.21.15.33:0/2255807732 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9584+0+0 (secure 0 0 0) 0x7f87980cf5f0 con 0x7f87a8102a90 2024-03-19T03:04:14.194 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.192+0000 7f879ffff640 1 --2- 172.21.15.33:0/2255807732 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f87940a2970 0x7f87940a4e30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f87900095f0 tx=0x7f8790005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:14.197 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.196+0000 7f879f7fe640 1 -- 172.21.15.33:0/2255807732 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f879809bf30 con 0x7f87a8102a90 2024-03-19T03:04:14.215 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:13 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2093490956' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T03:04:14.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:13 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2597273608' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:04:14.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:13 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2350165028' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:04:14.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:13 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2093490956' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T03:04:14.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:13 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2597273608' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:04:14.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:13 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2350165028' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:04:14.382 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.381+0000 7f87ac8b5640 1 -- 172.21.15.33:0/2255807732 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f87a8106f50 con 0x7f87a8102a90 2024-03-19T03:04:14.383 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.381+0000 7f879f7fe640 1 -- 172.21.15.33:0/2255807732 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v107) ==== 100+0+11 (secure 0 0 0) 0x7f879809b830 con 0x7f87a8102a90 2024-03-19T03:04:14.383 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 64 2024-03-19T03:04:14.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.383+0000 7f87ac8b5640 1 -- 172.21.15.33:0/2255807732 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f87940a2970 msgr2=0x7f87940a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:14.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.383+0000 7f87ac8b5640 1 --2- 172.21.15.33:0/2255807732 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f87940a2970 0x7f87940a4e30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f87900095f0 tx=0x7f8790005f90 comp rx=0 tx=0).stop 2024-03-19T03:04:14.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.383+0000 7f87ac8b5640 1 -- 172.21.15.33:0/2255807732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87a8102a90 msgr2=0x7f87a8197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:14.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.383+0000 7f87ac8b5640 1 --2- 172.21.15.33:0/2255807732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87a8102a90 0x7f87a8197fe0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f879800e9f0 tx=0x7f879800edf0 comp rx=0 tx=0).stop 2024-03-19T03:04:14.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.383+0000 7f87ac8b5640 1 -- 172.21.15.33:0/2255807732 shutdown_connections 2024-03-19T03:04:14.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.383+0000 7f87ac8b5640 1 --2- 172.21.15.33:0/2255807732 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f87940a2970 0x7f87940a4e30 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:14.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.383+0000 7f87ac8b5640 1 --2- 172.21.15.33:0/2255807732 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f87a81033b0 0x7f87a8198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:14.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.383+0000 7f87ac8b5640 1 --2- 172.21.15.33:0/2255807732 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87a810be60 0x7f87a819c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:14.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.383+0000 7f87ac8b5640 1 --2- 172.21.15.33:0/2255807732 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f87a8102a90 0x7f87a8197fe0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:14.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.383+0000 7f87ac8b5640 1 -- 172.21.15.33:0/2255807732 >> 172.21.15.33:0/2255807732 conn(0x7f87a8077880 msgr2=0x7f87a80fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:14.385 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.383+0000 7f87ac8b5640 1 -- 172.21.15.33:0/2255807732 shutdown_connections 2024-03-19T03:04:14.386 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.383+0000 7f87ac8b5640 1 -- 172.21.15.33:0/2255807732 wait complete. 2024-03-19T03:04:14.396 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T03:04:14.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.499+0000 7fa8cd113640 1 -- 172.21.15.33:0/4118713814 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8c8101100 msgr2=0x7fa8c810d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:14.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.499+0000 7fa8cd113640 1 --2- 172.21.15.33:0/4118713814 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8c8101100 0x7fa8c810d610 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8009cd0 tx=0x7fa8b802c0f0 comp rx=0 tx=0).stop 2024-03-19T03:04:14.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.499+0000 7fa8cd113640 1 -- 172.21.15.33:0/4118713814 shutdown_connections 2024-03-19T03:04:14.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.499+0000 7fa8cd113640 1 --2- 172.21.15.33:0/4118713814 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8c8100740 0x7fa8c8100bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:14.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.499+0000 7fa8cd113640 1 --2- 172.21.15.33:0/4118713814 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8c8101100 0x7fa8c810d610 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:14.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.499+0000 7fa8cd113640 1 --2- 172.21.15.33:0/4118713814 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8c8106750 0x7fa8c8106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:14.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.499+0000 7fa8cd113640 1 -- 172.21.15.33:0/4118713814 >> 172.21.15.33:0/4118713814 conn(0x7fa8c8077880 msgr2=0x7fa8c8077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:14.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.499+0000 7fa8cd113640 1 -- 172.21.15.33:0/4118713814 shutdown_connections 2024-03-19T03:04:14.501 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.499+0000 7fa8cd113640 1 -- 172.21.15.33:0/4118713814 wait complete. 2024-03-19T03:04:14.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.500+0000 7fa8cd113640 1 Processor -- start 2024-03-19T03:04:14.503 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.500+0000 7fa8cd113640 1 -- start start 2024-03-19T03:04:14.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.500+0000 7fa8cd113640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8c8100740 0x7fa8c8198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:14.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.500+0000 7fa8cd113640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8c8101100 0x7fa8c8198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:14.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.500+0000 7fa8cd113640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8c8106750 0x7fa8c819c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:14.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.500+0000 7fa8cd113640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c8110a10 con 0x7fa8c8101100 2024-03-19T03:04:14.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.500+0000 7fa8cd113640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c8110890 con 0x7fa8c8106750 2024-03-19T03:04:14.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.500+0000 7fa8cd113640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c8110b90 con 0x7fa8c8100740 2024-03-19T03:04:14.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.500+0000 7fa8c659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8c8101100 0x7fa8c8198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:14.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.500+0000 7fa8c659c640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8c8101100 0x7fa8c8198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51668/0 (socket says 172.21.15.33:51668) 2024-03-19T03:04:14.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.500+0000 7fa8c659c640 1 -- 172.21.15.33:0/3536365088 learned_addr learned my addr 172.21.15.33:0/3536365088 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:04:14.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.500+0000 7fa8c759e640 1 --2- 172.21.15.33:0/3536365088 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8c8106750 0x7fa8c819c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:14.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.501+0000 7fa8c6d9d640 1 --2- 172.21.15.33:0/3536365088 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8c8100740 0x7fa8c8198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:14.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.501+0000 7fa8c659c640 1 -- 172.21.15.33:0/3536365088 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8c8100740 msgr2=0x7fa8c8198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:14.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.501+0000 7fa8c659c640 1 --2- 172.21.15.33:0/3536365088 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8c8100740 0x7fa8c8198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:14.504 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.501+0000 7fa8c659c640 1 -- 172.21.15.33:0/3536365088 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8c8106750 msgr2=0x7fa8c819c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:14.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.501+0000 7fa8c659c640 1 --2- 172.21.15.33:0/3536365088 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8c8106750 0x7fa8c819c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:14.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.501+0000 7fa8c659c640 1 -- 172.21.15.33:0/3536365088 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8c819cff0 con 0x7fa8c8101100 2024-03-19T03:04:14.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.501+0000 7fa8c659c640 1 --2- 172.21.15.33:0/3536365088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8c8101100 0x7fa8c8198540 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fa8b000e9f0 tx=0x7fa8b000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:14.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.501+0000 7fa8bffff640 1 -- 172.21.15.33:0/3536365088 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b000cd80 con 0x7fa8c8101100 2024-03-19T03:04:14.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.501+0000 7fa8cd113640 1 -- 172.21.15.33:0/3536365088 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8c819d2e0 con 0x7fa8c8101100 2024-03-19T03:04:14.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.501+0000 7fa8bffff640 1 -- 172.21.15.33:0/3536365088 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa8b00054d0 con 0x7fa8c8101100 2024-03-19T03:04:14.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.501+0000 7fa8bffff640 1 -- 172.21.15.33:0/3536365088 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b0010490 con 0x7fa8c8101100 2024-03-19T03:04:14.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.501+0000 7fa8cd113640 1 -- 172.21.15.33:0/3536365088 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8c81a4b70 con 0x7fa8c8101100 2024-03-19T03:04:14.505 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.502+0000 7fa8cd113640 1 -- 172.21.15.33:0/3536365088 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8c8068a60 con 0x7fa8c8101100 2024-03-19T03:04:14.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.503+0000 7fa8bffff640 1 -- 172.21.15.33:0/3536365088 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fa8b0005000 con 0x7fa8c8101100 2024-03-19T03:04:14.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.504+0000 7fa8bffff640 1 --2- 172.21.15.33:0/3536365088 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa8b40a2970 0x7fa8b40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:14.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.504+0000 7fa8c6d9d640 1 --2- 172.21.15.33:0/3536365088 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa8b40a2970 0x7fa8b40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:14.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.504+0000 7fa8bffff640 1 -- 172.21.15.33:0/3536365088 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9584+0+0 (secure 0 0 0) 0x7fa8b00cf7f0 con 0x7fa8c8101100 2024-03-19T03:04:14.506 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.504+0000 7fa8c6d9d640 1 --2- 172.21.15.33:0/3536365088 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa8b40a2970 0x7fa8b40a4e30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0009b50 tx=0x7fa8c0005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:14.508 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.507+0000 7fa8bffff640 1 -- 172.21.15.33:0/3536365088 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa8b0053740 con 0x7fa8c8101100 2024-03-19T03:04:14.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.689+0000 7fa8cd113640 1 -- 172.21.15.33:0/3536365088 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fa8c81a4dd0 con 0x7fa8c8101100 2024-03-19T03:04:14.690 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.689+0000 7fa8bffff640 1 -- 172.21.15.33:0/3536365088 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v107) ==== 103+0+11 (secure 0 0 0) 0x7fa8b00052f0 con 0x7fa8c8101100 2024-03-19T03:04:14.691 INFO:teuthology.orchestra.run.smithi033.stdout:pg_num: 32 2024-03-19T03:04:14.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.691+0000 7fa8cd113640 1 -- 172.21.15.33:0/3536365088 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa8b40a2970 msgr2=0x7fa8b40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:14.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.691+0000 7fa8cd113640 1 --2- 172.21.15.33:0/3536365088 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa8b40a2970 0x7fa8b40a4e30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0009b50 tx=0x7fa8c0005f90 comp rx=0 tx=0).stop 2024-03-19T03:04:14.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.691+0000 7fa8cd113640 1 -- 172.21.15.33:0/3536365088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8c8101100 msgr2=0x7fa8c8198540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:14.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.691+0000 7fa8cd113640 1 --2- 172.21.15.33:0/3536365088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8c8101100 0x7fa8c8198540 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fa8b000e9f0 tx=0x7fa8b000edf0 comp rx=0 tx=0).stop 2024-03-19T03:04:14.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.691+0000 7fa8cd113640 1 -- 172.21.15.33:0/3536365088 shutdown_connections 2024-03-19T03:04:14.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.691+0000 7fa8cd113640 1 --2- 172.21.15.33:0/3536365088 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fa8b40a2970 0x7fa8b40a4e30 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:14.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.691+0000 7fa8cd113640 1 --2- 172.21.15.33:0/3536365088 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8c8106750 0x7fa8c819c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:14.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.691+0000 7fa8cd113640 1 --2- 172.21.15.33:0/3536365088 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fa8c8101100 0x7fa8c8198540 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:14.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.691+0000 7fa8cd113640 1 --2- 172.21.15.33:0/3536365088 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa8c8100740 0x7fa8c8198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:14.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.691+0000 7fa8cd113640 1 -- 172.21.15.33:0/3536365088 >> 172.21.15.33:0/3536365088 conn(0x7fa8c8077880 msgr2=0x7fa8c80ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:14.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.691+0000 7fa8cd113640 1 -- 172.21.15.33:0/3536365088 shutdown_connections 2024-03-19T03:04:14.693 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.691+0000 7fa8cd113640 1 -- 172.21.15.33:0/3536365088 wait complete. 2024-03-19T03:04:14.703 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-19T03:04:14.809 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.806+0000 7fd2aeaae640 1 -- 172.21.15.33:0/3240482112 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2a81017b0 msgr2=0x7fd2a8101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:14.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.806+0000 7fd2aeaae640 1 --2- 172.21.15.33:0/3240482112 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2a81017b0 0x7fd2a8101c30 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd29c009860 tx=0x7fd29c02bb70 comp rx=0 tx=0).stop 2024-03-19T03:04:14.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.806+0000 7fd2aeaae640 1 -- 172.21.15.33:0/3240482112 shutdown_connections 2024-03-19T03:04:14.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.806+0000 7fd2aeaae640 1 --2- 172.21.15.33:0/3240482112 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2a8102170 0x7fd2a810e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:14.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.806+0000 7fd2aeaae640 1 --2- 172.21.15.33:0/3240482112 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2a81017b0 0x7fd2a8101c30 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:14.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.806+0000 7fd2aeaae640 1 --2- 172.21.15.33:0/3240482112 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2a81077c0 0x7fd2a8107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:14.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.806+0000 7fd2aeaae640 1 -- 172.21.15.33:0/3240482112 >> 172.21.15.33:0/3240482112 conn(0x7fd2a8078c20 msgr2=0x7fd2a8079030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:14.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.806+0000 7fd2aeaae640 1 -- 172.21.15.33:0/3240482112 shutdown_connections 2024-03-19T03:04:14.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.806+0000 7fd2aeaae640 1 -- 172.21.15.33:0/3240482112 wait complete. 2024-03-19T03:04:14.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.807+0000 7fd2aeaae640 1 Processor -- start 2024-03-19T03:04:14.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.807+0000 7fd2aeaae640 1 -- start start 2024-03-19T03:04:14.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.807+0000 7fd2aeaae640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2a81017b0 0x7fd2a8073b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:14.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.807+0000 7fd2aeaae640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2a8102170 0x7fd2a8074060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:14.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.807+0000 7fd2aeaae640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2a81077c0 0x7fd2a80745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:14.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.807+0000 7fd2aeaae640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a8111a80 con 0x7fd2a8102170 2024-03-19T03:04:14.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.807+0000 7fd2aeaae640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a8111900 con 0x7fd2a81077c0 2024-03-19T03:04:14.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.807+0000 7fd2aeaae640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a8111c00 con 0x7fd2a81017b0 2024-03-19T03:04:14.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.807+0000 7fd2a7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2a8102170 0x7fd2a8074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:14.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.807+0000 7fd2a7fff640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2a8102170 0x7fd2a8074060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51686/0 (socket says 172.21.15.33:51686) 2024-03-19T03:04:14.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.807+0000 7fd2a7fff640 1 -- 172.21.15.33:0/4218561698 learned_addr learned my addr 172.21.15.33:0/4218561698 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:04:14.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.807+0000 7fd2ac84a640 1 --2- 172.21.15.33:0/4218561698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2a81017b0 0x7fd2a8073b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:14.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.807+0000 7fd2ad04b640 1 --2- 172.21.15.33:0/4218561698 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2a81077c0 0x7fd2a80745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:14.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.807+0000 7fd2a7fff640 1 -- 172.21.15.33:0/4218561698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2a81017b0 msgr2=0x7fd2a8073b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:14.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.807+0000 7fd2a7fff640 1 --2- 172.21.15.33:0/4218561698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2a81017b0 0x7fd2a8073b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:14.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.807+0000 7fd2a7fff640 1 -- 172.21.15.33:0/4218561698 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2a81077c0 msgr2=0x7fd2a80745a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:14.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.807+0000 7fd2a7fff640 1 --2- 172.21.15.33:0/4218561698 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2a81077c0 0x7fd2a80745a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:14.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.807+0000 7fd2a7fff640 1 -- 172.21.15.33:0/4218561698 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2a81a64a0 con 0x7fd2a8102170 2024-03-19T03:04:14.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.807+0000 7fd2a7fff640 1 --2- 172.21.15.33:0/4218561698 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2a8102170 0x7fd2a8074060 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fd29c037040 tx=0x7fd29c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:14.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.808+0000 7fd2a5ffb640 1 -- 172.21.15.33:0/4218561698 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd29c041020 con 0x7fd2a8102170 2024-03-19T03:04:14.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.808+0000 7fd2aeaae640 1 -- 172.21.15.33:0/4218561698 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2a81a6730 con 0x7fd2a8102170 2024-03-19T03:04:14.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.808+0000 7fd2a5ffb640 1 -- 172.21.15.33:0/4218561698 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd29c02c9c0 con 0x7fd2a8102170 2024-03-19T03:04:14.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.808+0000 7fd2a5ffb640 1 -- 172.21.15.33:0/4218561698 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd29c039040 con 0x7fd2a8102170 2024-03-19T03:04:14.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.808+0000 7fd2aeaae640 1 -- 172.21.15.33:0/4218561698 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2a81a6c20 con 0x7fd2a8102170 2024-03-19T03:04:14.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.809+0000 7fd2aeaae640 1 -- 172.21.15.33:0/4218561698 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2a8068a60 con 0x7fd2a8102170 2024-03-19T03:04:14.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.810+0000 7fd2a5ffb640 1 -- 172.21.15.33:0/4218561698 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fd29c044020 con 0x7fd2a8102170 2024-03-19T03:04:14.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.811+0000 7fd2a5ffb640 1 --2- 172.21.15.33:0/4218561698 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd2900a2970 0x7fd2900a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:14.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.811+0000 7fd2ac84a640 1 --2- 172.21.15.33:0/4218561698 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd2900a2970 0x7fd2900a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:14.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.811+0000 7fd2a5ffb640 1 -- 172.21.15.33:0/4218561698 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9584+0+0 (secure 0 0 0) 0x7fd29c0f0e00 con 0x7fd2a8102170 2024-03-19T03:04:14.813 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.811+0000 7fd2ac84a640 1 --2- 172.21.15.33:0/4218561698 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd2900a2970 0x7fd2900a4e30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0009b50 tx=0x7fd2a0005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:14.815 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:14.814+0000 7fd2a5ffb640 1 -- 172.21.15.33:0/4218561698 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd29c04b020 con 0x7fd2a8102170 2024-03-19T03:04:15.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:15.012+0000 7fd2aeaae640 1 -- 172.21.15.33:0/4218561698 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd2a8075050 con 0x7fd2a8102170 2024-03-19T03:04:15.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:15.013+0000 7fd2a5ffb640 1 -- 172.21.15.33:0/4218561698 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v107) ==== 81+0+942 (secure 0 0 0) 0x7fd2a8075050 con 0x7fd2a8102170 2024-03-19T03:04:15.014 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:6800/1453067701 2024-03-20T02:53:26.937131+0000 2024-03-19T03:04:15.015 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/3554132372 2024-03-20T02:53:26.937131+0000 2024-03-19T03:04:15.015 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/4099467400 2024-03-20T02:53:26.937131+0000 2024-03-19T03:04:15.015 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/176174888 2024-03-20T02:53:26.937131+0000 2024-03-19T03:04:15.015 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:6801/906645748 2024-03-20T02:43:39.622218+0000 2024-03-19T03:04:15.015 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:6800/906645748 2024-03-20T02:43:39.622218+0000 2024-03-19T03:04:15.015 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/1067768047 2024-03-20T02:43:39.622218+0000 2024-03-19T03:04:15.015 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/729379453 2024-03-20T02:43:11.918076+0000 2024-03-19T03:04:15.015 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/2586588742 2024-03-20T02:43:39.622218+0000 2024-03-19T03:04:15.015 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/2932814165 2024-03-20T02:43:11.918076+0000 2024-03-19T03:04:15.015 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/1950732854 2024-03-20T02:53:26.937131+0000 2024-03-19T03:04:15.015 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:6801/3342667183 2024-03-20T02:43:11.918076+0000 2024-03-19T03:04:15.015 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:6801/1453067701 2024-03-20T02:53:26.937131+0000 2024-03-19T03:04:15.016 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/2212564963 2024-03-20T02:43:11.918076+0000 2024-03-19T03:04:15.016 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:6800/3342667183 2024-03-20T02:43:11.918076+0000 2024-03-19T03:04:15.016 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33:0/3275126058 2024-03-20T02:43:39.622218+0000 2024-03-19T03:04:15.017 INFO:teuthology.orchestra.run.smithi033.stderr:listed 16 entries 2024-03-19T03:04:15.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:15.015+0000 7fd2aeaae640 1 -- 172.21.15.33:0/4218561698 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd2900a2970 msgr2=0x7fd2900a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:15.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:15.015+0000 7fd2aeaae640 1 --2- 172.21.15.33:0/4218561698 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd2900a2970 0x7fd2900a4e30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0009b50 tx=0x7fd2a0005f90 comp rx=0 tx=0).stop 2024-03-19T03:04:15.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:15.015+0000 7fd2aeaae640 1 -- 172.21.15.33:0/4218561698 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2a8102170 msgr2=0x7fd2a8074060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:15.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:15.015+0000 7fd2aeaae640 1 --2- 172.21.15.33:0/4218561698 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2a8102170 0x7fd2a8074060 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fd29c037040 tx=0x7fd29c004040 comp rx=0 tx=0).stop 2024-03-19T03:04:15.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:15.015+0000 7fd2aeaae640 1 -- 172.21.15.33:0/4218561698 shutdown_connections 2024-03-19T03:04:15.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:15.015+0000 7fd2aeaae640 1 --2- 172.21.15.33:0/4218561698 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fd2900a2970 0x7fd2900a4e30 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:15.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:15.015+0000 7fd2aeaae640 1 --2- 172.21.15.33:0/4218561698 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd2a81077c0 0x7fd2a80745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:15.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:15.015+0000 7fd2aeaae640 1 --2- 172.21.15.33:0/4218561698 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fd2a8102170 0x7fd2a8074060 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:15.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:15.015+0000 7fd2aeaae640 1 --2- 172.21.15.33:0/4218561698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2a81017b0 0x7fd2a8073b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:15.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:15.015+0000 7fd2aeaae640 1 -- 172.21.15.33:0/4218561698 >> 172.21.15.33:0/4218561698 conn(0x7fd2a8078c20 msgr2=0x7fd2a81041f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:15.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:15.015+0000 7fd2aeaae640 1 -- 172.21.15.33:0/4218561698 shutdown_connections 2024-03-19T03:04:15.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:15.015+0000 7fd2aeaae640 1 -- 172.21.15.33:0/4218561698 wait complete. 2024-03-19T03:04:15.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:14 smithi033 ceph-mon[31973]: pgmap v335: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-19T03:04:15.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:14 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3228402591' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:04:15.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:14 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2255807732' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:04:15.026 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:14 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3536365088' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:04:15.027 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi033.front.sepia.ceph.com... 2024-03-19T03:04:15.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:04:15.028 DEBUG:teuthology.orchestra.run.smithi033:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-19T03:04:15.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi033.stderr:ceph-fuse[85913]: fuse finished with error 0 and tester_r 0 2024-03-19T03:04:15.054 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T03:04:15.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:14 smithi096 ceph-mon[35946]: pgmap v335: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-19T03:04:15.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:14 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3228402591' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:04:15.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:14 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2255807732' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:04:15.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:14 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3536365088' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:04:15.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:14 smithi155 ceph-mon[35804]: pgmap v335: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-19T03:04:15.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:14 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3228402591' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:04:15.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:14 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2255807732' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:04:15.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:14 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3536365088' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:04:16.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:15 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/4218561698' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T03:04:16.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:15 smithi096 ceph-mon[35946]: osdmap e108: 12 total, 12 up, 12 in 2024-03-19T03:04:16.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:15 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/4218561698' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T03:04:16.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:15 smithi155 ceph-mon[35804]: osdmap e108: 12 total, 12 up, 12 in 2024-03-19T03:04:16.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:15 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/4218561698' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T03:04:16.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:15 smithi033 ceph-mon[31973]: osdmap e108: 12 total, 12 up, 12 in 2024-03-19T03:04:17.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:16 smithi096 ceph-mon[35946]: pgmap v336: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-19T03:04:17.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:16 smithi155 ceph-mon[35804]: pgmap v336: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-19T03:04:17.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:16 smithi033 ceph-mon[31973]: pgmap v336: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-19T03:04:19.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:18 smithi096 ceph-mon[35946]: pgmap v338: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 220 B/s wr, 6 op/s 2024-03-19T03:04:19.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:18 smithi155 ceph-mon[35804]: pgmap v338: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 220 B/s wr, 6 op/s 2024-03-19T03:04:19.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:18 smithi033 ceph-mon[31973]: pgmap v338: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 220 B/s wr, 6 op/s 2024-03-19T03:04:21.157 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi033.front.sepia.ceph.com 2024-03-19T03:04:21.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:04:21.157 DEBUG:teuthology.orchestra.run.smithi033:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-19T03:04:21.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:04:21.178 DEBUG:teuthology.orchestra.run.smithi033:> ip netns list 2024-03-19T03:04:21.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:20 smithi096 ceph-mon[35946]: pgmap v339: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 818 B/s wr, 7 op/s 2024-03-19T03:04:21.235 INFO:teuthology.orchestra.run.smithi033.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-19T03:04:21.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:04:21.236 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T03:04:21.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:20 smithi155 ceph-mon[35804]: pgmap v339: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 818 B/s wr, 7 op/s 2024-03-19T03:04:21.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:20 smithi033 ceph-mon[31973]: pgmap v339: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 818 B/s wr, 7 op/s 2024-03-19T03:04:21.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:04:21.309 DEBUG:teuthology.orchestra.run.smithi033:> sudo ip link delete ceph-brx 2024-03-19T03:04:21.420 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-19T03:04:21.430 INFO:tasks.cephadm:Teardown begin 2024-03-19T03:04:21.431 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-19T03:04:21.431 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:04:21.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.536+0000 7f85df93b640 1 -- 172.21.15.33:0/2210809201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85d8107820 msgr2=0x7f85d8107c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:21.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.536+0000 7f85df93b640 1 --2- 172.21.15.33:0/2210809201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85d8107820 0x7f85d8107c00 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f85d4009900 tx=0x7f85d402bcd0 comp rx=0 tx=0).stop 2024-03-19T03:04:21.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.536+0000 7f85df93b640 1 -- 172.21.15.33:0/2210809201 shutdown_connections 2024-03-19T03:04:21.540 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.536+0000 7f85df93b640 1 --2- 172.21.15.33:0/2210809201 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85d806a870 0x7f85d810e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:21.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.536+0000 7f85df93b640 1 --2- 172.21.15.33:0/2210809201 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f85d8069ed0 0x7f85d806a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:21.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.536+0000 7f85df93b640 1 --2- 172.21.15.33:0/2210809201 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85d8107820 0x7f85d8107c00 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:21.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.536+0000 7f85df93b640 1 -- 172.21.15.33:0/2210809201 >> 172.21.15.33:0/2210809201 conn(0x7f85d80777c0 msgr2=0x7f85d8077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:21.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.536+0000 7f85df93b640 1 -- 172.21.15.33:0/2210809201 shutdown_connections 2024-03-19T03:04:21.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.536+0000 7f85df93b640 1 -- 172.21.15.33:0/2210809201 wait complete. 2024-03-19T03:04:21.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.537+0000 7f85df93b640 1 Processor -- start 2024-03-19T03:04:21.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.537+0000 7f85df93b640 1 -- start start 2024-03-19T03:04:21.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.537+0000 7f85df93b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85d8069ed0 0x7f85d8100750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:21.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.537+0000 7f85df93b640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85d806a870 0x7f85d8100c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:21.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.537+0000 7f85df93b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f85d8107820 0x7f85d81011d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:21.541 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.537+0000 7f85df93b640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f85d8111a90 con 0x7f85d806a870 2024-03-19T03:04:21.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.537+0000 7f85df93b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f85d8111910 con 0x7f85d8107820 2024-03-19T03:04:21.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.537+0000 7f85df93b640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f85d8111c10 con 0x7f85d8069ed0 2024-03-19T03:04:21.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.537+0000 7f85dced6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85d806a870 0x7f85d8100c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:21.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.537+0000 7f85dced6640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85d806a870 0x7f85d8100c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51716/0 (socket says 172.21.15.33:51716) 2024-03-19T03:04:21.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.537+0000 7f85dced6640 1 -- 172.21.15.33:0/2790159238 learned_addr learned my addr 172.21.15.33:0/2790159238 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:04:21.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.537+0000 7f85dded8640 1 --2- 172.21.15.33:0/2790159238 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f85d8107820 0x7f85d81011d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:21.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.537+0000 7f85dd6d7640 1 --2- 172.21.15.33:0/2790159238 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85d8069ed0 0x7f85d8100750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:21.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.537+0000 7f85dced6640 1 -- 172.21.15.33:0/2790159238 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85d8069ed0 msgr2=0x7f85d8100750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:21.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.537+0000 7f85dced6640 1 --2- 172.21.15.33:0/2790159238 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85d8069ed0 0x7f85d8100750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:21.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.537+0000 7f85dced6640 1 -- 172.21.15.33:0/2790159238 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f85d8107820 msgr2=0x7f85d81011d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:21.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.537+0000 7f85dced6640 1 --2- 172.21.15.33:0/2790159238 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f85d8107820 0x7f85d81011d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:21.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.537+0000 7f85dced6640 1 -- 172.21.15.33:0/2790159238 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85d81a64d0 con 0x7f85d806a870 2024-03-19T03:04:21.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.538+0000 7f85dced6640 1 --2- 172.21.15.33:0/2790159238 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85d806a870 0x7f85d8100c90 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f85c800e9f0 tx=0x7f85c800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:21.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.538+0000 7f85ce7fc640 1 -- 172.21.15.33:0/2790159238 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c800cd80 con 0x7f85d806a870 2024-03-19T03:04:21.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.538+0000 7f85ce7fc640 1 -- 172.21.15.33:0/2790159238 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85c80108e0 con 0x7f85d806a870 2024-03-19T03:04:21.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.538+0000 7f85df93b640 1 -- 172.21.15.33:0/2790159238 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85d81a67c0 con 0x7f85d806a870 2024-03-19T03:04:21.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.538+0000 7f85ce7fc640 1 -- 172.21.15.33:0/2790159238 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c8018920 con 0x7f85d806a870 2024-03-19T03:04:21.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.538+0000 7f85df93b640 1 -- 172.21.15.33:0/2790159238 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85d81a6d10 con 0x7f85d806a870 2024-03-19T03:04:21.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.539+0000 7f85df93b640 1 -- 172.21.15.33:0/2790159238 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85d8068a60 con 0x7f85d806a870 2024-03-19T03:04:21.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.540+0000 7f85ce7fc640 1 -- 172.21.15.33:0/2790159238 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f85c8010a90 con 0x7f85d806a870 2024-03-19T03:04:21.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.540+0000 7f85ce7fc640 1 --2- 172.21.15.33:0/2790159238 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f85c40a2970 0x7f85c40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:21.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.541+0000 7f85dd6d7640 1 --2- 172.21.15.33:0/2790159238 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f85c40a2970 0x7f85c40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:21.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.541+0000 7f85ce7fc640 1 -- 172.21.15.33:0/2790159238 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9496+0+0 (secure 0 0 0) 0x7f85c80cf490 con 0x7f85d806a870 2024-03-19T03:04:21.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.541+0000 7f85dd6d7640 1 --2- 172.21.15.33:0/2790159238 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f85c40a2970 0x7f85c40a4e30 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f85d4006010 tx=0x7f85d4005e40 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:21.546 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.545+0000 7f85ce7fc640 1 -- 172.21.15.33:0/2790159238 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f85c809b870 con 0x7f85d806a870 2024-03-19T03:04:21.771 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.769+0000 7f85df93b640 1 -- 172.21.15.33:0/2790159238 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f85d8109cc0 con 0x7f85d806a870 2024-03-19T03:04:21.772 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.770+0000 7f85ce7fc640 1 -- 172.21.15.33:0/2790159238 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11867 (secure 0 0 0) 0x7f85c809f910 con 0x7f85d806a870 2024-03-19T03:04:21.772 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T03:04:21.772 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:50:52.201798+0000","modified":"2024-03-19T02:53:11.748697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24488,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/3731232169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":3731232169},{"type":"v1","addr":"172.21.15.96:6839","nonce":3731232169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24457,24488,24509]},"id":1}]} 2024-03-19T03:04:21.772 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-03-19T03:04:21.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.772+0000 7f85df93b640 1 -- 172.21.15.33:0/2790159238 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f85c40a2970 msgr2=0x7f85c40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:21.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.772+0000 7f85df93b640 1 --2- 172.21.15.33:0/2790159238 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f85c40a2970 0x7f85c40a4e30 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f85d4006010 tx=0x7f85d4005e40 comp rx=0 tx=0).stop 2024-03-19T03:04:21.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.772+0000 7f85df93b640 1 -- 172.21.15.33:0/2790159238 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85d806a870 msgr2=0x7f85d8100c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:21.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.772+0000 7f85df93b640 1 --2- 172.21.15.33:0/2790159238 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85d806a870 0x7f85d8100c90 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f85c800e9f0 tx=0x7f85c800eeb0 comp rx=0 tx=0).stop 2024-03-19T03:04:21.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.772+0000 7f85df93b640 1 -- 172.21.15.33:0/2790159238 shutdown_connections 2024-03-19T03:04:21.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.772+0000 7f85df93b640 1 --2- 172.21.15.33:0/2790159238 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f85c40a2970 0x7f85c40a4e30 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:21.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.772+0000 7f85df93b640 1 --2- 172.21.15.33:0/2790159238 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f85d8107820 0x7f85d81011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:21.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.772+0000 7f85df93b640 1 --2- 172.21.15.33:0/2790159238 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f85d806a870 0x7f85d8100c90 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:21.774 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.772+0000 7f85df93b640 1 --2- 172.21.15.33:0/2790159238 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85d8069ed0 0x7f85d8100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:21.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.772+0000 7f85df93b640 1 -- 172.21.15.33:0/2790159238 >> 172.21.15.33:0/2790159238 conn(0x7f85d80777c0 msgr2=0x7f85d80ff520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:21.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.772+0000 7f85df93b640 1 -- 172.21.15.33:0/2790159238 shutdown_connections 2024-03-19T03:04:21.775 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.772+0000 7f85df93b640 1 -- 172.21.15.33:0/2790159238 wait complete. 2024-03-19T03:04:21.786 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:04:21.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.888+0000 7f4dd2697640 1 -- 172.21.15.33:0/3388622614 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4dcc1010e0 msgr2=0x7f4dcc10d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:21.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.888+0000 7f4dd2697640 1 --2- 172.21.15.33:0/3388622614 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4dcc1010e0 0x7f4dcc10d5f0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4009cd0 tx=0x7f4dc402c0f0 comp rx=0 tx=0).stop 2024-03-19T03:04:21.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.888+0000 7f4dd2697640 1 -- 172.21.15.33:0/3388622614 shutdown_connections 2024-03-19T03:04:21.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.888+0000 7f4dd2697640 1 --2- 172.21.15.33:0/3388622614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4dcc100720 0x7f4dcc100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:21.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.888+0000 7f4dd2697640 1 --2- 172.21.15.33:0/3388622614 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4dcc1010e0 0x7f4dcc10d5f0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:21.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.888+0000 7f4dd2697640 1 --2- 172.21.15.33:0/3388622614 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4dcc106730 0x7f4dcc106b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:21.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.888+0000 7f4dd2697640 1 -- 172.21.15.33:0/3388622614 >> 172.21.15.33:0/3388622614 conn(0x7f4dcc077820 msgr2=0x7f4dcc077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:21.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.888+0000 7f4dd2697640 1 -- 172.21.15.33:0/3388622614 shutdown_connections 2024-03-19T03:04:21.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.888+0000 7f4dd2697640 1 -- 172.21.15.33:0/3388622614 wait complete. 2024-03-19T03:04:21.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.888+0000 7f4dd2697640 1 Processor -- start 2024-03-19T03:04:21.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.888+0000 7f4dd2697640 1 -- start start 2024-03-19T03:04:21.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.889+0000 7f4dd2697640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4dcc100720 0x7f4dcc198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:21.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.889+0000 7f4dd2697640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4dcc1010e0 0x7f4dcc1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:21.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.889+0000 7f4dd2697640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4dcc106730 0x7f4dcc19c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:21.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.889+0000 7f4dd2697640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f4dcc1109f0 con 0x7f4dcc106730 2024-03-19T03:04:21.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.889+0000 7f4dd2697640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4dcc110870 con 0x7f4dcc1010e0 2024-03-19T03:04:21.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.889+0000 7f4dd2697640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4dcc110b70 con 0x7f4dcc100720 2024-03-19T03:04:21.892 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.889+0000 7f4dd0c34640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4dcc106730 0x7f4dcc19c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:21.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.889+0000 7f4dd0c34640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4dcc106730 0x7f4dcc19c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51736/0 (socket says 172.21.15.33:51736) 2024-03-19T03:04:21.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.889+0000 7f4dcb7fe640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4dcc1010e0 0x7f4dcc1985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:21.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.889+0000 7f4dd0c34640 1 -- 172.21.15.33:0/3682431688 learned_addr learned my addr 172.21.15.33:0/3682431688 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:04:21.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.889+0000 7f4dcbfff640 1 --2- 172.21.15.33:0/3682431688 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4dcc100720 0x7f4dcc198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:21.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.889+0000 7f4dd0c34640 1 -- 172.21.15.33:0/3682431688 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4dcc100720 msgr2=0x7f4dcc198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:21.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.889+0000 7f4dd0c34640 1 --2- 172.21.15.33:0/3682431688 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4dcc100720 0x7f4dcc198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:21.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.889+0000 7f4dd0c34640 1 -- 172.21.15.33:0/3682431688 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4dcc1010e0 msgr2=0x7f4dcc1985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:21.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.889+0000 7f4dd0c34640 1 --2- 172.21.15.33:0/3682431688 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4dcc1010e0 0x7f4dcc1985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:21.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.889+0000 7f4dd0c34640 1 -- 172.21.15.33:0/3682431688 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dcc19d000 con 0x7f4dcc106730 2024-03-19T03:04:21.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.889+0000 7f4dcb7fe640 1 --2- 172.21.15.33:0/3682431688 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4dcc1010e0 0x7f4dcc1985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:04:21.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.889+0000 7f4dcbfff640 1 --2- 172.21.15.33:0/3682431688 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4dcc100720 0x7f4dcc198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:04:21.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.889+0000 7f4dd0c34640 1 --2- 172.21.15.33:0/3682431688 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4dcc106730 0x7f4dcc19c8d0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4004af0 tx=0x7f4dc4035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:21.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.889+0000 7f4dc97fa640 1 -- 172.21.15.33:0/3682431688 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc4039040 con 0x7f4dcc106730 2024-03-19T03:04:21.893 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.889+0000 7f4dd2697640 1 -- 172.21.15.33:0/3682431688 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dcc19d290 con 0x7f4dcc106730 2024-03-19T03:04:21.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.889+0000 7f4dc97fa640 1 -- 172.21.15.33:0/3682431688 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4dc4044070 con 0x7f4dcc106730 2024-03-19T03:04:21.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.889+0000 7f4dc97fa640 1 -- 172.21.15.33:0/3682431688 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc40042a0 con 0x7f4dcc106730 2024-03-19T03:04:21.894 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.890+0000 7f4dd2697640 1 -- 172.21.15.33:0/3682431688 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dcc1a4b80 con 0x7f4dcc106730 2024-03-19T03:04:21.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.890+0000 7f4dd2697640 1 -- 172.21.15.33:0/3682431688 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4dcc068610 con 0x7f4dcc106730 2024-03-19T03:04:21.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.891+0000 7f4dc97fa640 1 -- 172.21.15.33:0/3682431688 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f4dc4004440 con 0x7f4dcc106730 2024-03-19T03:04:21.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.892+0000 7f4dc97fa640 1 --2- 172.21.15.33:0/3682431688 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4db40a2970 0x7f4db40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:21.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.892+0000 7f4dcbfff640 1 --2- 172.21.15.33:0/3682431688 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4db40a2970 0x7f4db40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:21.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.892+0000 7f4dc97fa640 1 -- 172.21.15.33:0/3682431688 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9496+0+0 (secure 0 0 0) 0x7f4dc40f1630 con 0x7f4dcc106730 2024-03-19T03:04:21.895 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.892+0000 7f4dcbfff640 1 --2- 172.21.15.33:0/3682431688 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4db40a2970 0x7f4db40a4e30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc005d00 tx=0x7f4dbc00b410 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:21.897 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:21.896+0000 7f4dc97fa640 1 -- 172.21.15.33:0/3682431688 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4dc40be0d0 con 0x7f4dcc106730 2024-03-19T03:04:22.120 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.119+0000 7f4dd2697640 1 -- 172.21.15.33:0/3682431688 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4dcc1a4d50 con 0x7f4dcc106730 2024-03-19T03:04:22.121 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.119+0000 7f4dc97fa640 1 -- 172.21.15.33:0/3682431688 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11867 (secure 0 0 0) 0x7f4dc40bd9d0 con 0x7f4dcc106730 2024-03-19T03:04:22.121 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T03:04:22.121 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6835/973717150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6834","nonce":973717150},{"type":"v1","addr":"172.21.15.33:6835","nonce":973717150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14538,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6837/3063370040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6836","nonce":3063370040},{"type":"v1","addr":"172.21.15.33:6837","nonce":3063370040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6839/1789650898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6838","nonce":1789650898},{"type":"v1","addr":"172.21.15.33:6839","nonce":1789650898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.33:6841/1349324757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6840","nonce":1349324757},{"type":"v1","addr":"172.21.15.33:6841","nonce":1349324757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24391,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6833/268774752","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":268774752},{"type":"v1","addr":"172.21.15.155:6833","nonce":268774752}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/2137965374","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":2137965374},{"type":"v1","addr":"172.21.15.96:6833","nonce":2137965374}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T02:50:52.201798+0000","modified":"2024-03-19T02:53:11.748697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24488,"mds_2":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24427":{"gid":24427,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6835/1274854977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":1274854977},{"type":"v1","addr":"172.21.15.155:6835","nonce":1274854977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.155:6837/1429349330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6836","nonce":1429349330},{"type":"v1","addr":"172.21.15.155:6837","nonce":1429349330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.96:6835/3551193617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":3551193617},{"type":"v1","addr":"172.21.15.96:6835","nonce":3551193617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.155:6839/1840534530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6838","nonce":1840534530},{"type":"v1","addr":"172.21.15.155:6839","nonce":1840534530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.96:6837/522588113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6836","nonce":522588113},{"type":"v1","addr":"172.21.15.96:6837","nonce":522588113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6839/3731232169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6838","nonce":3731232169},{"type":"v1","addr":"172.21.15.96:6839","nonce":3731232169}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24457,24488,24509]},"id":1}]} 2024-03-19T03:04:22.121 INFO:teuthology.orchestra.run.smithi033.stderr:dumped fsmap epoch 25 2024-03-19T03:04:22.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.121+0000 7f4dd2697640 1 -- 172.21.15.33:0/3682431688 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4db40a2970 msgr2=0x7f4db40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:22.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.121+0000 7f4dd2697640 1 --2- 172.21.15.33:0/3682431688 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4db40a2970 0x7f4db40a4e30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc005d00 tx=0x7f4dbc00b410 comp rx=0 tx=0).stop 2024-03-19T03:04:22.123 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.121+0000 7f4dd2697640 1 -- 172.21.15.33:0/3682431688 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4dcc106730 msgr2=0x7f4dcc19c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:22.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.121+0000 7f4dd2697640 1 --2- 172.21.15.33:0/3682431688 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4dcc106730 0x7f4dcc19c8d0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4004af0 tx=0x7f4dc4035ce0 comp rx=0 tx=0).stop 2024-03-19T03:04:22.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.121+0000 7f4dd2697640 1 -- 172.21.15.33:0/3682431688 shutdown_connections 2024-03-19T03:04:22.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.121+0000 7f4dd2697640 1 --2- 172.21.15.33:0/3682431688 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f4db40a2970 0x7f4db40a4e30 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:22.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.122+0000 7f4dd2697640 1 --2- 172.21.15.33:0/3682431688 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f4dcc106730 0x7f4dcc19c8d0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:22.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.122+0000 7f4dd2697640 1 --2- 172.21.15.33:0/3682431688 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4dcc1010e0 0x7f4dcc1985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:22.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.122+0000 7f4dd2697640 1 --2- 172.21.15.33:0/3682431688 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4dcc100720 0x7f4dcc198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:22.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.122+0000 7f4dd2697640 1 -- 172.21.15.33:0/3682431688 >> 172.21.15.33:0/3682431688 conn(0x7f4dcc077820 msgr2=0x7f4dcc0ff980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:22.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.122+0000 7f4dd2697640 1 -- 172.21.15.33:0/3682431688 shutdown_connections 2024-03-19T03:04:22.124 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.122+0000 7f4dd2697640 1 -- 172.21.15.33:0/3682431688 wait complete. 2024-03-19T03:04:22.133 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:21 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2790159238' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:04:22.134 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T03:04:22.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:21 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2790159238' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:04:22.239 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.236+0000 7fe9916bf640 1 -- 172.21.15.33:0/3702784738 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe98c101080 msgr2=0x7fe98c10d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:22.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.236+0000 7fe9916bf640 1 --2- 172.21.15.33:0/3702784738 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe98c101080 0x7fe98c10d590 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fe984009cd0 tx=0x7fe98402c0f0 comp rx=0 tx=0).stop 2024-03-19T03:04:22.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.236+0000 7fe9916bf640 1 -- 172.21.15.33:0/3702784738 shutdown_connections 2024-03-19T03:04:22.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.236+0000 7fe9916bf640 1 --2- 172.21.15.33:0/3702784738 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe98c1006c0 0x7fe98c100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:22.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.236+0000 7fe9916bf640 1 --2- 172.21.15.33:0/3702784738 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe98c101080 0x7fe98c10d590 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:22.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.236+0000 7fe9916bf640 1 --2- 172.21.15.33:0/3702784738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe98c1066d0 0x7fe98c106ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:22.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.236+0000 7fe9916bf640 1 -- 172.21.15.33:0/3702784738 >> 172.21.15.33:0/3702784738 conn(0x7fe98c076aa0 msgr2=0x7fe98c076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:22.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.236+0000 7fe9916bf640 1 -- 172.21.15.33:0/3702784738 shutdown_connections 2024-03-19T03:04:22.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.236+0000 7fe9916bf640 1 -- 172.21.15.33:0/3702784738 wait complete. 2024-03-19T03:04:22.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.236+0000 7fe9916bf640 1 Processor -- start 2024-03-19T03:04:22.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.237+0000 7fe9916bf640 1 -- start start 2024-03-19T03:04:22.240 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.237+0000 7fe9916bf640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe98c1006c0 0x7fe98c197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:22.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.237+0000 7fe9916bf640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe98c101080 0x7fe98c198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:22.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.237+0000 7fe9916bf640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe98c1066d0 0x7fe98c19c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:22.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.237+0000 7fe9916bf640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7fe98c110990 con 0x7fe98c1006c0 2024-03-19T03:04:22.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.237+0000 7fe9916bf640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe98c110810 con 0x7fe98c101080 2024-03-19T03:04:22.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.237+0000 7fe9916bf640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe98c110b10 con 0x7fe98c1066d0 2024-03-19T03:04:22.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.237+0000 7fe98affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe98c1006c0 0x7fe98c197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:22.241 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.237+0000 7fe98affd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe98c1006c0 0x7fe98c197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51768/0 (socket says 172.21.15.33:51768) 2024-03-19T03:04:22.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.237+0000 7fe98affd640 1 -- 172.21.15.33:0/3620881718 learned_addr learned my addr 172.21.15.33:0/3620881718 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:04:22.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.237+0000 7fe98a7fc640 1 --2- 172.21.15.33:0/3620881718 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe98c101080 0x7fe98c198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:22.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.237+0000 7fe98b7fe640 1 --2- 172.21.15.33:0/3620881718 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe98c1066d0 0x7fe98c19c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:22.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.237+0000 7fe98affd640 1 -- 172.21.15.33:0/3620881718 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe98c1066d0 msgr2=0x7fe98c19c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:22.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.237+0000 7fe98affd640 1 --2- 172.21.15.33:0/3620881718 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe98c1066d0 0x7fe98c19c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:22.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.237+0000 7fe98affd640 1 -- 172.21.15.33:0/3620881718 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe98c101080 msgr2=0x7fe98c198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:22.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.237+0000 7fe98affd640 1 --2- 172.21.15.33:0/3620881718 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe98c101080 0x7fe98c198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:22.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.237+0000 7fe98affd640 1 -- 172.21.15.33:0/3620881718 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe98c19cfc0 con 0x7fe98c1006c0 2024-03-19T03:04:22.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.238+0000 7fe98affd640 1 --2- 172.21.15.33:0/3620881718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe98c1006c0 0x7fe98c197fd0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fe97c00e9f0 tx=0x7fe97c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:22.242 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.238+0000 7fe973fff640 1 -- 172.21.15.33:0/3620881718 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97c00cd80 con 0x7fe98c1006c0 2024-03-19T03:04:22.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.238+0000 7fe973fff640 1 -- 172.21.15.33:0/3620881718 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe97c0108e0 con 0x7fe98c1006c0 2024-03-19T03:04:22.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.238+0000 7fe9916bf640 1 -- 172.21.15.33:0/3620881718 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe98c19d2b0 con 0x7fe98c1006c0 2024-03-19T03:04:22.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.238+0000 7fe973fff640 1 -- 172.21.15.33:0/3620881718 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97c018920 con 0x7fe98c1006c0 2024-03-19T03:04:22.243 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.238+0000 7fe9916bf640 1 -- 172.21.15.33:0/3620881718 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe98c1a4b40 con 0x7fe98c1006c0 2024-03-19T03:04:22.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.239+0000 7fe9916bf640 1 -- 172.21.15.33:0/3620881718 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe98c101e00 con 0x7fe98c1006c0 2024-03-19T03:04:22.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.240+0000 7fe973fff640 1 -- 172.21.15.33:0/3620881718 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7fe97c010a90 con 0x7fe98c1006c0 2024-03-19T03:04:22.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.240+0000 7fe973fff640 1 --2- 172.21.15.33:0/3620881718 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe9740a2970 0x7fe9740a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:22.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.241+0000 7fe98a7fc640 1 --2- 172.21.15.33:0/3620881718 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe9740a2970 0x7fe9740a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:22.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.241+0000 7fe973fff640 1 -- 172.21.15.33:0/3620881718 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9496+0+0 (secure 0 0 0) 0x7fe97c0cf930 con 0x7fe98c1006c0 2024-03-19T03:04:22.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.241+0000 7fe98a7fc640 1 --2- 172.21.15.33:0/3620881718 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe9740a2970 0x7fe9740a4e30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe98c104e50 tx=0x7fe980005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:22.245 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.244+0000 7fe973fff640 1 -- 172.21.15.33:0/3620881718 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe97c09bc70 con 0x7fe98c1006c0 2024-03-19T03:04:22.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:21 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2790159238' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:04:22.429 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.427+0000 7fe9916bf640 1 -- 172.21.15.33:0/3620881718 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe98c068a60 con 0x7fe98c1006c0 2024-03-19T03:04:22.430 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.429+0000 7fe973fff640 1 -- 172.21.15.33:0/3620881718 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) ==== 74+0+25226 (secure 0 0 0) 0x7fe97c086090 con 0x7fe98c1006c0 2024-03-19T03:04:22.431 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-19T03:04:22.431 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":108,"fsid":"2b7a8938-e59a-11ee-95c9-87774f69a715","created":"2024-03-19T02:42:37.912179+0000","modified":"2024-03-19T03:04:15.250184+0000","last_up_change":"2024-03-19T02:50:01.539392+0000","last_in_change":"2024-03-19T02:49:40.534415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T02:46:09.963737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T02:50:50.069641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T02:50:51.381064+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T02:50:55.476667+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"090a89ab-6b7b-4317-8823-178d4212b2f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6803","nonce":1633124995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6805","nonce":1633124995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6809","nonce":1633124995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1633124995},{"type":"v1","addr":"172.21.15.33:6807","nonce":1633124995}]},"public_addr":"172.21.15.33:6803/1633124995","cluster_addr":"172.21.15.33:6805/1633124995","heartbeat_back_addr":"172.21.15.33:6809/1633124995","heartbeat_front_addr":"172.21.15.33:6807/1633124995","state":["exists","up"]},{"osd":1,"uuid":"f50eec55-1956-4486-8593-f2c8fa610e88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6801","nonce":340144398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6803","nonce":340144398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6807","nonce":340144398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":340144398},{"type":"v1","addr":"172.21.15.96:6805","nonce":340144398}]},"public_addr":"172.21.15.96:6801/340144398","cluster_addr":"172.21.15.96:6803/340144398","heartbeat_back_addr":"172.21.15.96:6807/340144398","heartbeat_front_addr":"172.21.15.96:6805/340144398","state":["exists","up"]},{"osd":2,"uuid":"893a2d62-80d9-4e5e-bc6a-a1c8c681d652","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6801","nonce":3388743876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6803","nonce":3388743876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6807","nonce":3388743876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":3388743876},{"type":"v1","addr":"172.21.15.155:6805","nonce":3388743876}]},"public_addr":"172.21.15.155:6801/3388743876","cluster_addr":"172.21.15.155:6803/3388743876","heartbeat_back_addr":"172.21.15.155:6807/3388743876","heartbeat_front_addr":"172.21.15.155:6805/3388743876","state":["exists","up"]},{"osd":3,"uuid":"b07e0482-4281-4d32-9dfe-9995d7246175","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6811","nonce":677007171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6813","nonce":677007171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6817","nonce":677007171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":677007171},{"type":"v1","addr":"172.21.15.33:6815","nonce":677007171}]},"public_addr":"172.21.15.33:6811/677007171","cluster_addr":"172.21.15.33:6813/677007171","heartbeat_back_addr":"172.21.15.33:6817/677007171","heartbeat_front_addr":"172.21.15.33:6815/677007171","state":["exists","up"]},{"osd":4,"uuid":"a988384b-af6b-47c6-8c07-5c9673acee45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6809","nonce":3106158554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6811","nonce":3106158554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6815","nonce":3106158554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":3106158554},{"type":"v1","addr":"172.21.15.96:6813","nonce":3106158554}]},"public_addr":"172.21.15.96:6809/3106158554","cluster_addr":"172.21.15.96:6811/3106158554","heartbeat_back_addr":"172.21.15.96:6815/3106158554","heartbeat_front_addr":"172.21.15.96:6813/3106158554","state":["exists","up"]},{"osd":5,"uuid":"198dc992-96f8-4c52-96ee-1fd5de148bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6809","nonce":327293324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6811","nonce":327293324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6815","nonce":327293324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":327293324},{"type":"v1","addr":"172.21.15.155:6813","nonce":327293324}]},"public_addr":"172.21.15.155:6809/327293324","cluster_addr":"172.21.15.155:6811/327293324","heartbeat_back_addr":"172.21.15.155:6815/327293324","heartbeat_front_addr":"172.21.15.155:6813/327293324","state":["exists","up"]},{"osd":6,"uuid":"a7278ec0-ab2e-49ed-be53-84d8b9e24177","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6819","nonce":873871509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6821","nonce":873871509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6825","nonce":873871509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":873871509},{"type":"v1","addr":"172.21.15.33:6823","nonce":873871509}]},"public_addr":"172.21.15.33:6819/873871509","cluster_addr":"172.21.15.33:6821/873871509","heartbeat_back_addr":"172.21.15.33:6825/873871509","heartbeat_front_addr":"172.21.15.33:6823/873871509","state":["exists","up"]},{"osd":7,"uuid":"727e133d-aa24-4ffc-90e2-266eb2aab8c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6817","nonce":2334123891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6819","nonce":2334123891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6823","nonce":2334123891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2334123891},{"type":"v1","addr":"172.21.15.96:6821","nonce":2334123891}]},"public_addr":"172.21.15.96:6817/2334123891","cluster_addr":"172.21.15.96:6819/2334123891","heartbeat_back_addr":"172.21.15.96:6823/2334123891","heartbeat_front_addr":"172.21.15.96:6821/2334123891","state":["exists","up"]},{"osd":8,"uuid":"bb20eec0-39e5-4b86-9db5-486b1840cdad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6817","nonce":3498516903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6819","nonce":3498516903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6823","nonce":3498516903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":3498516903},{"type":"v1","addr":"172.21.15.155:6821","nonce":3498516903}]},"public_addr":"172.21.15.155:6817/3498516903","cluster_addr":"172.21.15.155:6819/3498516903","heartbeat_back_addr":"172.21.15.155:6823/3498516903","heartbeat_front_addr":"172.21.15.155:6821/3498516903","state":["exists","up"]},{"osd":9,"uuid":"179c6228-3e23-4f65-91a6-1ef0005d6904","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6827","nonce":760777213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6829","nonce":760777213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6832","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6833","nonce":760777213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":760777213},{"type":"v1","addr":"172.21.15.33:6831","nonce":760777213}]},"public_addr":"172.21.15.33:6827/760777213","cluster_addr":"172.21.15.33:6829/760777213","heartbeat_back_addr":"172.21.15.33:6833/760777213","heartbeat_front_addr":"172.21.15.33:6831/760777213","state":["exists","up"]},{"osd":10,"uuid":"be17b478-3c2e-41fb-bede-958333acdf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6825","nonce":1404898482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6827","nonce":1404898482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6831","nonce":1404898482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1404898482},{"type":"v1","addr":"172.21.15.96:6829","nonce":1404898482}]},"public_addr":"172.21.15.96:6825/1404898482","cluster_addr":"172.21.15.96:6827/1404898482","heartbeat_back_addr":"172.21.15.96:6831/1404898482","heartbeat_front_addr":"172.21.15.96:6829/1404898482","state":["exists","up"]},{"osd":11,"uuid":"c0903ed2-cc1e-4c18-8731-1f0ee4dd0457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6825","nonce":792352827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6827","nonce":792352827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6831","nonce":792352827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":792352827},{"type":"v1","addr":"172.21.15.155:6829","nonce":792352827}]},"public_addr":"172.21.15.155:6825/792352827","cluster_addr":"172.21.15.155:6827/792352827","heartbeat_back_addr":"172.21.15.155:6831/792352827","heartbeat_front_addr":"172.21.15.155:6829/792352827","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:17.942801+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:45:42.109449+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:05.981599+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:32.069574+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:46:57.431015+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:22.448592+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:47:48.526515+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:14.498352+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:48:40.175426+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:06.364006+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:32.853161+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T02:49:59.073538+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.9","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.33:6800/1453067701":"2024-03-20T02:53:26.937131+0000","172.21.15.33:0/3554132372":"2024-03-20T02:53:26.937131+0000","172.21.15.33:0/4099467400":"2024-03-20T02:53:26.937131+0000","172.21.15.33:0/176174888":"2024-03-20T02:53:26.937131+0000","172.21.15.33:6801/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:6800/906645748":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/1067768047":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/729379453":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/2586588742":"2024-03-20T02:43:39.622218+0000","172.21.15.33:0/2932814165":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/1950732854":"2024-03-20T02:53:26.937131+0000","172.21.15.33:6801/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6801/1453067701":"2024-03-20T02:53:26.937131+0000","172.21.15.33:0/2212564963":"2024-03-20T02:43:11.918076+0000","172.21.15.33:6800/3342667183":"2024-03-20T02:43:11.918076+0000","172.21.15.33:0/3275126058":"2024-03-20T02:43:39.622218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:04:22.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.431+0000 7fe9916bf640 1 -- 172.21.15.33:0/3620881718 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe9740a2970 msgr2=0x7fe9740a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:22.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.431+0000 7fe9916bf640 1 --2- 172.21.15.33:0/3620881718 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe9740a2970 0x7fe9740a4e30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe98c104e50 tx=0x7fe980005f90 comp rx=0 tx=0).stop 2024-03-19T03:04:22.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.431+0000 7fe9916bf640 1 -- 172.21.15.33:0/3620881718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe98c1006c0 msgr2=0x7fe98c197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:22.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.431+0000 7fe9916bf640 1 --2- 172.21.15.33:0/3620881718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe98c1006c0 0x7fe98c197fd0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fe97c00e9f0 tx=0x7fe97c00eeb0 comp rx=0 tx=0).stop 2024-03-19T03:04:22.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.431+0000 7fe9916bf640 1 -- 172.21.15.33:0/3620881718 shutdown_connections 2024-03-19T03:04:22.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.431+0000 7fe9916bf640 1 --2- 172.21.15.33:0/3620881718 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fe9740a2970 0x7fe9740a4e30 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:22.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.431+0000 7fe9916bf640 1 --2- 172.21.15.33:0/3620881718 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe98c1066d0 0x7fe98c19c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:22.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.431+0000 7fe9916bf640 1 --2- 172.21.15.33:0/3620881718 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe98c101080 0x7fe98c198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:22.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.431+0000 7fe9916bf640 1 --2- 172.21.15.33:0/3620881718 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7fe98c1006c0 0x7fe98c197fd0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:22.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.431+0000 7fe9916bf640 1 -- 172.21.15.33:0/3620881718 >> 172.21.15.33:0/3620881718 conn(0x7fe98c076aa0 msgr2=0x7fe98c06a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:22.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.431+0000 7fe9916bf640 1 -- 172.21.15.33:0/3620881718 shutdown_connections 2024-03-19T03:04:22.433 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.431+0000 7fe9916bf640 1 -- 172.21.15.33:0/3620881718 wait complete. 2024-03-19T03:04:22.444 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-19T03:04:22.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.547+0000 7faeb9631640 1 -- 172.21.15.33:0/1682678224 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faeb4100690 msgr2=0x7faeb4100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:22.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.547+0000 7faeb9631640 1 --2- 172.21.15.33:0/1682678224 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faeb4100690 0x7faeb4100b10 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7faea8009900 tx=0x7faea802bc30 comp rx=0 tx=0).stop 2024-03-19T03:04:22.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.547+0000 7faeb9631640 1 -- 172.21.15.33:0/1682678224 shutdown_connections 2024-03-19T03:04:22.551 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.547+0000 7faeb9631640 1 --2- 172.21.15.33:0/1682678224 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faeb4101050 0x7faeb410d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:22.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.547+0000 7faeb9631640 1 --2- 172.21.15.33:0/1682678224 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faeb4100690 0x7faeb4100b10 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:22.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.547+0000 7faeb9631640 1 --2- 172.21.15.33:0/1682678224 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faeb41066a0 0x7faeb4106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:22.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.547+0000 7faeb9631640 1 -- 172.21.15.33:0/1682678224 >> 172.21.15.33:0/1682678224 conn(0x7faeb4077820 msgr2=0x7faeb4077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:22.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.547+0000 7faeb9631640 1 -- 172.21.15.33:0/1682678224 shutdown_connections 2024-03-19T03:04:22.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.547+0000 7faeb9631640 1 -- 172.21.15.33:0/1682678224 wait complete. 2024-03-19T03:04:22.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.547+0000 7faeb9631640 1 Processor -- start 2024-03-19T03:04:22.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.548+0000 7faeb9631640 1 -- start start 2024-03-19T03:04:22.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.548+0000 7faeb9631640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faeb4100690 0x7faeb4102c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:22.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.548+0000 7faeb9631640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faeb4101050 0x7faeb4103170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:22.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.548+0000 7faeb9631640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faeb41066a0 0x7faeb41036b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:22.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.548+0000 7faeb9631640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7faeb41109b0 con 0x7faeb4101050 2024-03-19T03:04:22.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.548+0000 7faeb9631640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faeb4110830 con 0x7faeb41066a0 2024-03-19T03:04:22.552 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.548+0000 7faeb9631640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7faeb4110b30 con 0x7faeb4100690 2024-03-19T03:04:22.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.548+0000 7faeb27fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faeb4101050 0x7faeb4103170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:22.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.548+0000 7faeb27fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faeb4101050 0x7faeb4103170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51786/0 (socket says 172.21.15.33:51786) 2024-03-19T03:04:22.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.548+0000 7faeb27fc640 1 -- 172.21.15.33:0/2329752719 learned_addr learned my addr 172.21.15.33:0/2329752719 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:04:22.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.548+0000 7faeb2ffd640 1 --2- 172.21.15.33:0/2329752719 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faeb4100690 0x7faeb4102c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:22.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.548+0000 7faeb37fe640 1 --2- 172.21.15.33:0/2329752719 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faeb41066a0 0x7faeb41036b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:22.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.548+0000 7faeb27fc640 1 -- 172.21.15.33:0/2329752719 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faeb4100690 msgr2=0x7faeb4102c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:22.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.548+0000 7faeb27fc640 1 --2- 172.21.15.33:0/2329752719 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faeb4100690 0x7faeb4102c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:22.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.548+0000 7faeb27fc640 1 -- 172.21.15.33:0/2329752719 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faeb41066a0 msgr2=0x7faeb41036b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:22.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.548+0000 7faeb27fc640 1 --2- 172.21.15.33:0/2329752719 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faeb41066a0 0x7faeb41036b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:22.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.548+0000 7faeb27fc640 1 -- 172.21.15.33:0/2329752719 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faeb41a6580 con 0x7faeb4101050 2024-03-19T03:04:22.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.548+0000 7faeb27fc640 1 --2- 172.21.15.33:0/2329752719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faeb4101050 0x7faeb4103170 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7faea8000c00 tx=0x7faea8005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:22.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.549+0000 7fae9bfff640 1 -- 172.21.15.33:0/2329752719 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea8043020 con 0x7faeb4101050 2024-03-19T03:04:22.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.549+0000 7faeb9631640 1 -- 172.21.15.33:0/2329752719 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faeb41a6810 con 0x7faeb4101050 2024-03-19T03:04:22.553 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.549+0000 7fae9bfff640 1 -- 172.21.15.33:0/2329752719 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faea802ca90 con 0x7faeb4101050 2024-03-19T03:04:22.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.549+0000 7fae9bfff640 1 -- 172.21.15.33:0/2329752719 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea80039e0 con 0x7faeb4101050 2024-03-19T03:04:22.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.549+0000 7faeb9631640 1 -- 172.21.15.33:0/2329752719 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faeb41a6d60 con 0x7faeb4101050 2024-03-19T03:04:22.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.550+0000 7faeb9631640 1 -- 172.21.15.33:0/2329752719 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faeb4068610 con 0x7faeb4101050 2024-03-19T03:04:22.554 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.551+0000 7fae9bfff640 1 -- 172.21.15.33:0/2329752719 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7faea8046020 con 0x7faeb4101050 2024-03-19T03:04:22.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.551+0000 7fae9bfff640 1 --2- 172.21.15.33:0/2329752719 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fae9c0a2970 0x7fae9c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:22.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.551+0000 7faeb2ffd640 1 --2- 172.21.15.33:0/2329752719 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fae9c0a2970 0x7fae9c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:22.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.551+0000 7fae9bfff640 1 -- 172.21.15.33:0/2329752719 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9496+0+0 (secure 0 0 0) 0x7faea80f1250 con 0x7faeb4101050 2024-03-19T03:04:22.555 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.552+0000 7faeb2ffd640 1 --2- 172.21.15.33:0/2329752719 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fae9c0a2970 0x7fae9c0a4e30 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7faea4009b50 tx=0x7faea4005f90 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:22.557 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.556+0000 7fae9bfff640 1 -- 172.21.15.33:0/2329752719 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7faea804d020 con 0x7faeb4101050 2024-03-19T03:04:22.778 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.776+0000 7faeb9631640 1 -- 172.21.15.33:0/2329752719 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7faeb4109c30 con 0x7faeb4101050 2024-03-19T03:04:22.779 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.778+0000 7fae9bfff640 1 -- 172.21.15.33:0/2329752719 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v26) ==== 161+0+0 (secure 0 0 0) 0x7faea80bdcf0 con 0x7faeb4101050 2024-03-19T03:04:22.779 INFO:teuthology.orchestra.run.smithi033.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-19T03:04:22.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.781+0000 7faeb9631640 1 -- 172.21.15.33:0/2329752719 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fae9c0a2970 msgr2=0x7fae9c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:22.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.781+0000 7faeb9631640 1 --2- 172.21.15.33:0/2329752719 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fae9c0a2970 0x7fae9c0a4e30 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7faea4009b50 tx=0x7faea4005f90 comp rx=0 tx=0).stop 2024-03-19T03:04:22.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.781+0000 7faeb9631640 1 -- 172.21.15.33:0/2329752719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faeb4101050 msgr2=0x7faeb4103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:22.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.781+0000 7faeb9631640 1 --2- 172.21.15.33:0/2329752719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faeb4101050 0x7faeb4103170 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7faea8000c00 tx=0x7faea8005dd0 comp rx=0 tx=0).stop 2024-03-19T03:04:22.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.781+0000 7faeb9631640 1 -- 172.21.15.33:0/2329752719 shutdown_connections 2024-03-19T03:04:22.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.781+0000 7faeb9631640 1 --2- 172.21.15.33:0/2329752719 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7fae9c0a2970 0x7fae9c0a4e30 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:22.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.781+0000 7faeb9631640 1 --2- 172.21.15.33:0/2329752719 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faeb41066a0 0x7faeb41036b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:22.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.781+0000 7faeb9631640 1 --2- 172.21.15.33:0/2329752719 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7faeb4101050 0x7faeb4103170 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:22.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.781+0000 7faeb9631640 1 --2- 172.21.15.33:0/2329752719 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faeb4100690 0x7faeb4102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:22.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.781+0000 7faeb9631640 1 -- 172.21.15.33:0/2329752719 >> 172.21.15.33:0/2329752719 conn(0x7faeb4077820 msgr2=0x7faeb4068d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:22.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.781+0000 7faeb9631640 1 -- 172.21.15.33:0/2329752719 shutdown_connections 2024-03-19T03:04:22.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.781+0000 7faeb9631640 1 -- 172.21.15.33:0/2329752719 wait complete. 2024-03-19T03:04:22.794 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-19T03:04:22.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.899+0000 7f60614f1640 1 -- 172.21.15.33:0/3230347953 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f605c100aa0 msgr2=0x7f605c10e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:22.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.899+0000 7f60614f1640 1 --2- 172.21.15.33:0/3230347953 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f605c100aa0 0x7f605c10e6a0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f6054009930 tx=0x7f605402bc30 comp rx=0 tx=0).stop 2024-03-19T03:04:22.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.899+0000 7f60614f1640 1 -- 172.21.15.33:0/3230347953 shutdown_connections 2024-03-19T03:04:22.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.899+0000 7f60614f1640 1 --2- 172.21.15.33:0/3230347953 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f605c1000e0 0x7f605c100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:22.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.899+0000 7f60614f1640 1 --2- 172.21.15.33:0/3230347953 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f605c100aa0 0x7f605c10e6a0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:22.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.899+0000 7f60614f1640 1 --2- 172.21.15.33:0/3230347953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f605c107a20 0x7f605c107e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:22.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.899+0000 7f60614f1640 1 -- 172.21.15.33:0/3230347953 >> 172.21.15.33:0/3230347953 conn(0x7f605c075a60 msgr2=0x7f605c075e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:22.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.899+0000 7f60614f1640 1 -- 172.21.15.33:0/3230347953 shutdown_connections 2024-03-19T03:04:22.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.899+0000 7f60614f1640 1 -- 172.21.15.33:0/3230347953 wait complete. 2024-03-19T03:04:22.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.899+0000 7f60614f1640 1 Processor -- start 2024-03-19T03:04:22.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.900+0000 7f60614f1640 1 -- start start 2024-03-19T03:04:22.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.900+0000 7f60614f1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f605c1000e0 0x7f605c198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:22.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.900+0000 7f60614f1640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f605c100aa0 0x7f605c198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:22.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.900+0000 7f60614f1640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f605c107a20 0x7f605c19c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:22.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.900+0000 7f60614f1640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f605c111b30 con 0x7f605c100aa0 2024-03-19T03:04:22.903 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.900+0000 7f60614f1640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f605c1119b0 con 0x7f605c1000e0 2024-03-19T03:04:22.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.900+0000 7f60614f1640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f605c111cb0 con 0x7f605c107a20 2024-03-19T03:04:22.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.900+0000 7f605a7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f605c100aa0 0x7f605c198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:22.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.900+0000 7f605a7fc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f605c100aa0 0x7f605c198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:51804/0 (socket says 172.21.15.33:51804) 2024-03-19T03:04:22.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.900+0000 7f605a7fc640 1 -- 172.21.15.33:0/3830656583 learned_addr learned my addr 172.21.15.33:0/3830656583 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:04:22.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.900+0000 7f605b7fe640 1 --2- 172.21.15.33:0/3830656583 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f605c107a20 0x7f605c19c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:22.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.900+0000 7f605affd640 1 --2- 172.21.15.33:0/3830656583 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f605c1000e0 0x7f605c198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:22.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.900+0000 7f605a7fc640 1 -- 172.21.15.33:0/3830656583 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f605c107a20 msgr2=0x7f605c19c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:22.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.900+0000 7f605a7fc640 1 --2- 172.21.15.33:0/3830656583 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f605c107a20 0x7f605c19c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:22.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.900+0000 7f605a7fc640 1 -- 172.21.15.33:0/3830656583 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f605c1000e0 msgr2=0x7f605c198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:22.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.900+0000 7f605a7fc640 1 --2- 172.21.15.33:0/3830656583 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f605c1000e0 0x7f605c198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:22.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.900+0000 7f605a7fc640 1 -- 172.21.15.33:0/3830656583 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f605c19cfd0 con 0x7f605c100aa0 2024-03-19T03:04:22.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.900+0000 7f605a7fc640 1 --2- 172.21.15.33:0/3830656583 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f605c100aa0 0x7f605c198570 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f605000d770 tx=0x7f605000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:22.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.900+0000 7f6043fff640 1 -- 172.21.15.33:0/3830656583 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6050017070 con 0x7f605c100aa0 2024-03-19T03:04:22.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.901+0000 7f60614f1640 1 -- 172.21.15.33:0/3830656583 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f605c19d2c0 con 0x7f605c100aa0 2024-03-19T03:04:22.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.901+0000 7f6043fff640 1 -- 172.21.15.33:0/3830656583 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60500054d0 con 0x7f605c100aa0 2024-03-19T03:04:22.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.901+0000 7f6043fff640 1 -- 172.21.15.33:0/3830656583 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6050004cf0 con 0x7f605c100aa0 2024-03-19T03:04:22.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.901+0000 7f60614f1640 1 -- 172.21.15.33:0/3830656583 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f605c1a4b50 con 0x7f605c100aa0 2024-03-19T03:04:22.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.902+0000 7f60614f1640 1 -- 172.21.15.33:0/3830656583 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f605c068a60 con 0x7f605c100aa0 2024-03-19T03:04:22.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.903+0000 7f6043fff640 1 -- 172.21.15.33:0/3830656583 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f6050004e90 con 0x7f605c100aa0 2024-03-19T03:04:22.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.903+0000 7f6043fff640 1 --2- 172.21.15.33:0/3830656583 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f60440a2970 0x7f60440a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:22.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.903+0000 7f605affd640 1 --2- 172.21.15.33:0/3830656583 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f60440a2970 0x7f60440a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:22.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.903+0000 7f6043fff640 1 -- 172.21.15.33:0/3830656583 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9666+0+0 (secure 0 0 0) 0x7f60500cf100 con 0x7f605c100aa0 2024-03-19T03:04:22.906 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.904+0000 7f605affd640 1 --2- 172.21.15.33:0/3830656583 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f60440a2970 0x7f60440a4e30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f604c005fd0 tx=0x7f604c009500 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:22.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:22.907+0000 7f6043fff640 1 -- 172.21.15.33:0/3830656583 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f605009b9e0 con 0x7f605c100aa0 2024-03-19T03:04:23.134 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.132+0000 7f60614f1640 1 -- 172.21.15.33:0/3830656583 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f605c109d90 con 0x7f605c100aa0 2024-03-19T03:04:23.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:22 smithi096 ceph-mon[35946]: pgmap v340: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 818 B/s wr, 7 op/s 2024-03-19T03:04:23.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:22 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3682431688' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:04:23.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:22 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3620881718' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:04:23.217 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:22 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2329752719' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-19T03:04:23.217 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:22 smithi096 ceph-mon[35946]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-19T03:04:23.217 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:22 smithi096 ceph-mon[35946]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-19T03:04:23.217 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:22 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2329752719' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-19T03:04:23.217 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:22 smithi096 ceph-mon[35946]: osdmap e109: 12 total, 12 up, 12 in 2024-03-19T03:04:23.217 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:22 smithi096 ceph-mon[35946]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-19T03:04:23.217 INFO:journalctl@ceph.mds.b.smithi096.stdout:Mar 19 03:04:22 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-b[65459]: ignoring --setuser ceph since I am not root 2024-03-19T03:04:23.217 INFO:journalctl@ceph.mds.b.smithi096.stdout:Mar 19 03:04:22 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-b[65459]: ignoring --setgroup ceph since I am not root 2024-03-19T03:04:23.217 INFO:journalctl@ceph.mds.b.smithi096.stdout:Mar 19 03:04:22 smithi096 ceph-mds[65463]: -- 172.21.15.96:0/4212123096 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5562d59ada40 con 0x5562d5980c00 2024-03-19T03:04:23.218 INFO:journalctl@ceph.mds.b.smithi096.stdout:Mar 19 03:04:22 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-b[65459]: starting mds.b at 2024-03-19T03:04:23.218 INFO:journalctl@ceph.mds.e.smithi096.stdout:Mar 19 03:04:22 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-e[64849]: ignoring --setuser ceph since I am not root 2024-03-19T03:04:23.218 INFO:journalctl@ceph.mds.e.smithi096.stdout:Mar 19 03:04:22 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-e[64849]: ignoring --setgroup ceph since I am not root 2024-03-19T03:04:23.218 INFO:journalctl@ceph.mds.e.smithi096.stdout:Mar 19 03:04:22 smithi096 ceph-mds[64853]: -- 172.21.15.96:0/250590149 <== mon.2 v2:172.21.15.96:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5596ba3f3a40 con 0x5596ba3c7000 2024-03-19T03:04:23.218 INFO:journalctl@ceph.mds.e.smithi096.stdout:Mar 19 03:04:22 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-e[64849]: starting mds.e at 2024-03-19T03:04:23.218 INFO:journalctl@ceph.mds.h.smithi096.stdout:Mar 19 03:04:22 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-h[65157]: ignoring --setuser ceph since I am not root 2024-03-19T03:04:23.218 INFO:journalctl@ceph.mds.h.smithi096.stdout:Mar 19 03:04:22 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-h[65157]: ignoring --setgroup ceph since I am not root 2024-03-19T03:04:23.219 INFO:journalctl@ceph.mds.h.smithi096.stdout:Mar 19 03:04:22 smithi096 ceph-mds[65161]: -- 172.21.15.96:0/3738750227 <== mon.0 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bdc6357a40 con 0x55bdc632a800 2024-03-19T03:04:23.219 INFO:journalctl@ceph.mds.h.smithi096.stdout:Mar 19 03:04:22 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-h[65157]: starting mds.h at 2024-03-19T03:04:23.266 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:22 smithi155 ceph-mon[35804]: pgmap v340: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 818 B/s wr, 7 op/s 2024-03-19T03:04:23.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:22 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3682431688' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:04:23.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:22 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3620881718' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:04:23.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:22 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2329752719' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-19T03:04:23.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:22 smithi155 ceph-mon[35804]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-19T03:04:23.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:22 smithi155 ceph-mon[35804]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-19T03:04:23.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:22 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2329752719' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-19T03:04:23.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:22 smithi155 ceph-mon[35804]: osdmap e109: 12 total, 12 up, 12 in 2024-03-19T03:04:23.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:22 smithi155 ceph-mon[35804]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-19T03:04:23.267 INFO:journalctl@ceph.mds.i.smithi155.stdout:Mar 19 03:04:22 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-i[66753]: ignoring --setuser ceph since I am not root 2024-03-19T03:04:23.267 INFO:journalctl@ceph.mds.i.smithi155.stdout:Mar 19 03:04:22 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-i[66753]: ignoring --setgroup ceph since I am not root 2024-03-19T03:04:23.267 INFO:journalctl@ceph.mds.i.smithi155.stdout:Mar 19 03:04:22 smithi155 ceph-mds[66757]: -- 172.21.15.155:0/854643629 <== mon.1 v2:172.21.15.155:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559737739a40 con 0x55973770c800 2024-03-19T03:04:23.268 INFO:journalctl@ceph.mds.i.smithi155.stdout:Mar 19 03:04:22 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-i[66753]: starting mds.i at 2024-03-19T03:04:23.268 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 19 03:04:22 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-c[66446]: ignoring --setuser ceph since I am not root 2024-03-19T03:04:23.268 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 19 03:04:22 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-c[66446]: ignoring --setgroup ceph since I am not root 2024-03-19T03:04:23.268 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 19 03:04:22 smithi155 ceph-mds[66450]: -- 172.21.15.155:0/2034146587 <== mon.0 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556f9cfe9a40 con 0x556f9cfbcc00 2024-03-19T03:04:23.268 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 19 03:04:22 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-c[66446]: starting mds.c at 2024-03-19T03:04:23.268 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 19 03:04:22 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-f[66140]: ignoring --setuser ceph since I am not root 2024-03-19T03:04:23.268 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 19 03:04:22 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-f[66140]: ignoring --setgroup ceph since I am not root 2024-03-19T03:04:23.268 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 19 03:04:22 smithi155 ceph-mds[66144]: -- 172.21.15.155:0/309029660 <== mon.0 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561272785a40 con 0x561272759000 2024-03-19T03:04:23.268 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 19 03:04:22 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-f[66140]: starting mds.f at 2024-03-19T03:04:23.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:22 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mon-a[31969]: 2024-03-19T03:04:22.777+0000 7f6d5c574700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-19T03:04:23.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:22 smithi033 ceph-mon[31973]: pgmap v340: 129 pgs: 129 active+clean; 1.9 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 818 B/s wr, 7 op/s 2024-03-19T03:04:23.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:22 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3682431688' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:04:23.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:22 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3620881718' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:04:23.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:22 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2329752719' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-19T03:04:23.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:22 smithi033 ceph-mon[31973]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-19T03:04:23.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:22 smithi033 ceph-mon[31973]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-19T03:04:23.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:22 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2329752719' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-19T03:04:23.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:22 smithi033 ceph-mon[31973]: osdmap e109: 12 total, 12 up, 12 in 2024-03-19T03:04:23.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:22 smithi033 ceph-mon[31973]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-19T03:04:23.781 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.780+0000 7f6043fff640 1 -- 172.21.15.33:0/3830656583 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v27) ==== 104+0+0 (secure 0 0 0) 0x7f605009b2e0 con 0x7f605c100aa0 2024-03-19T03:04:23.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.783+0000 7f60614f1640 1 -- 172.21.15.33:0/3830656583 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f60440a2970 msgr2=0x7f60440a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:23.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.783+0000 7f60614f1640 1 --2- 172.21.15.33:0/3830656583 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f60440a2970 0x7f60440a4e30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f604c005fd0 tx=0x7f604c009500 comp rx=0 tx=0).stop 2024-03-19T03:04:23.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.783+0000 7f60614f1640 1 -- 172.21.15.33:0/3830656583 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f605c100aa0 msgr2=0x7f605c198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:23.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.783+0000 7f60614f1640 1 --2- 172.21.15.33:0/3830656583 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f605c100aa0 0x7f605c198570 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f605000d770 tx=0x7f605000dc30 comp rx=0 tx=0).stop 2024-03-19T03:04:23.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.783+0000 7f60614f1640 1 -- 172.21.15.33:0/3830656583 shutdown_connections 2024-03-19T03:04:23.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.783+0000 7f60614f1640 1 --2- 172.21.15.33:0/3830656583 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f60440a2970 0x7f60440a4e30 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:23.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.783+0000 7f60614f1640 1 --2- 172.21.15.33:0/3830656583 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f605c107a20 0x7f605c19c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:23.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.783+0000 7f60614f1640 1 --2- 172.21.15.33:0/3830656583 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f605c100aa0 0x7f605c198570 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:23.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.783+0000 7f60614f1640 1 --2- 172.21.15.33:0/3830656583 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f605c1000e0 0x7f605c198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:23.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.783+0000 7f60614f1640 1 -- 172.21.15.33:0/3830656583 >> 172.21.15.33:0/3830656583 conn(0x7f605c075a60 msgr2=0x7f605c0fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:23.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.783+0000 7f60614f1640 1 -- 172.21.15.33:0/3830656583 shutdown_connections 2024-03-19T03:04:23.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.783+0000 7f60614f1640 1 -- 172.21.15.33:0/3830656583 wait complete. 2024-03-19T03:04:23.800 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-19T03:04:23.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.913+0000 7f5d8b377640 1 -- 172.21.15.33:0/2054501288 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d841006d0 msgr2=0x7f5d84100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:23.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.913+0000 7f5d8b377640 1 --2- 172.21.15.33:0/2054501288 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d841006d0 0x7f5d84100b50 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f5d74009930 tx=0x7f5d7402bc30 comp rx=0 tx=0).stop 2024-03-19T03:04:23.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.913+0000 7f5d8b377640 1 -- 172.21.15.33:0/2054501288 shutdown_connections 2024-03-19T03:04:23.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.913+0000 7f5d8b377640 1 --2- 172.21.15.33:0/2054501288 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d84101090 0x7f5d8410d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:23.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.913+0000 7f5d8b377640 1 --2- 172.21.15.33:0/2054501288 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d841006d0 0x7f5d84100b50 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:23.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.913+0000 7f5d8b377640 1 --2- 172.21.15.33:0/2054501288 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d841066e0 0x7f5d84106ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:23.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.913+0000 7f5d8b377640 1 -- 172.21.15.33:0/2054501288 >> 172.21.15.33:0/2054501288 conn(0x7f5d84076aa0 msgr2=0x7f5d84076eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:23.917 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.913+0000 7f5d8b377640 1 -- 172.21.15.33:0/2054501288 shutdown_connections 2024-03-19T03:04:23.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.913+0000 7f5d8b377640 1 -- 172.21.15.33:0/2054501288 wait complete. 2024-03-19T03:04:23.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.914+0000 7f5d8b377640 1 Processor -- start 2024-03-19T03:04:23.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.914+0000 7f5d8b377640 1 -- start start 2024-03-19T03:04:23.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.914+0000 7f5d8b377640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d841006d0 0x7f5d84197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:23.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.914+0000 7f5d8b377640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d84101090 0x7f5d84198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:23.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.914+0000 7f5d8b377640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d841066e0 0x7f5d8419c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:23.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.914+0000 7f5d8b377640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f5d841109a0 con 0x7f5d841066e0 2024-03-19T03:04:23.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.914+0000 7f5d8b377640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5d84110820 con 0x7f5d84101090 2024-03-19T03:04:23.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.914+0000 7f5d8b377640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5d84110b20 con 0x7f5d841006d0 2024-03-19T03:04:23.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.914+0000 7f5d89914640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d841066e0 0x7f5d8419c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:23.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.914+0000 7f5d89914640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d841066e0 0x7f5d8419c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58258/0 (socket says 172.21.15.33:58258) 2024-03-19T03:04:23.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.914+0000 7f5d89914640 1 -- 172.21.15.33:0/2342638893 learned_addr learned my addr 172.21.15.33:0/2342638893 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:04:23.918 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.914+0000 7f5d89113640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d841006d0 0x7f5d84197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:23.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.914+0000 7f5d88912640 1 --2- 172.21.15.33:0/2342638893 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d84101090 0x7f5d84198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:23.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.915+0000 7f5d89914640 1 -- 172.21.15.33:0/2342638893 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d841006d0 msgr2=0x7f5d84197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:23.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.915+0000 7f5d89914640 1 --2- 172.21.15.33:0/2342638893 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d841006d0 0x7f5d84197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:23.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.915+0000 7f5d89914640 1 -- 172.21.15.33:0/2342638893 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d84101090 msgr2=0x7f5d84198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:23.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.915+0000 7f5d89914640 1 --2- 172.21.15.33:0/2342638893 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d84101090 0x7f5d84198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:23.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.915+0000 7f5d89914640 1 -- 172.21.15.33:0/2342638893 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d8419cfe0 con 0x7f5d841066e0 2024-03-19T03:04:23.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.915+0000 7f5d89914640 1 --2- 172.21.15.33:0/2342638893 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d841066e0 0x7f5d8419c8b0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c00eaa0 tx=0x7f5d7c00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:23.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.915+0000 7f5d7a7fc640 1 -- 172.21.15.33:0/2342638893 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d7c004d20 con 0x7f5d841066e0 2024-03-19T03:04:23.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.915+0000 7f5d8b377640 1 -- 172.21.15.33:0/2342638893 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d8419d2d0 con 0x7f5d841066e0 2024-03-19T03:04:23.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.915+0000 7f5d7a7fc640 1 -- 172.21.15.33:0/2342638893 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5d7c005020 con 0x7f5d841066e0 2024-03-19T03:04:23.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.915+0000 7f5d7a7fc640 1 -- 172.21.15.33:0/2342638893 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d7c0116b0 con 0x7f5d841066e0 2024-03-19T03:04:23.919 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.915+0000 7f5d8b377640 1 -- 172.21.15.33:0/2342638893 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d841a4b60 con 0x7f5d841066e0 2024-03-19T03:04:23.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.916+0000 7f5d8b377640 1 -- 172.21.15.33:0/2342638893 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d84101e10 con 0x7f5d841066e0 2024-03-19T03:04:23.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.917+0000 7f5d7a7fc640 1 -- 172.21.15.33:0/2342638893 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f5d7c0051d0 con 0x7f5d841066e0 2024-03-19T03:04:23.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.918+0000 7f5d7a7fc640 1 --2- 172.21.15.33:0/2342638893 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f5d700a2970 0x7f5d700a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:23.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.918+0000 7f5d89113640 1 --2- 172.21.15.33:0/2342638893 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f5d700a2970 0x7f5d700a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:23.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.918+0000 7f5d7a7fc640 1 -- 172.21.15.33:0/2342638893 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9666+0+0 (secure 0 0 0) 0x7f5d7c0cf460 con 0x7f5d841066e0 2024-03-19T03:04:23.921 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.918+0000 7f5d89113640 1 --2- 172.21.15.33:0/2342638893 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f5d700a2970 0x7f5d700a4e30 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5d8000ab60 tx=0x7f5d8000a2d0 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:23.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:23.922+0000 7f5d7a7fc640 1 -- 172.21.15.33:0/2342638893 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5d7c016030 con 0x7f5d841066e0 2024-03-19T03:04:24.106 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:24.104+0000 7f5d8b377640 1 -- 172.21.15.33:0/2342638893 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5d84109ca0 con 0x7f5d841066e0 2024-03-19T03:04:24.168 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:23 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3830656583' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-19T03:04:24.168 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:23 smithi155 ceph-mon[35804]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-19T03:04:24.168 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:23 smithi155 ceph-mon[35804]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-19T03:04:24.168 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:23 smithi155 ceph-mon[35804]: Cluster is now healthy 2024-03-19T03:04:24.168 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:23 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/3830656583' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-19T03:04:24.168 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:23 smithi155 ceph-mon[35804]: mds.? [v2:172.21.15.155:6835/1249509621,v1:172.21.15.155:6838/1249509621] up:boot 2024-03-19T03:04:24.168 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:23 smithi155 ceph-mon[35804]: mds.? [v2:172.21.15.96:6838/1590027926,v1:172.21.15.96:6839/1590027926] up:boot 2024-03-19T03:04:24.168 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:23 smithi155 ceph-mon[35804]: mds.? [v2:172.21.15.155:6834/2142410105,v1:172.21.15.155:6837/2142410105] up:boot 2024-03-19T03:04:24.168 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:23 smithi155 ceph-mon[35804]: mds.? [v2:172.21.15.96:6835/1694329426,v1:172.21.15.96:6836/1694329426] up:boot 2024-03-19T03:04:24.169 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:23 smithi155 ceph-mon[35804]: mds.? [v2:172.21.15.96:6834/2793047534,v1:172.21.15.96:6837/2793047534] up:boot 2024-03-19T03:04:24.169 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:23 smithi155 ceph-mon[35804]: mds.? [v2:172.21.15.155:6836/3377531680,v1:172.21.15.155:6839/3377531680] up:boot 2024-03-19T03:04:24.169 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:23 smithi155 ceph-mon[35804]: fsmap 12 up:standby 2024-03-19T03:04:24.169 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:23 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-19T03:04:24.169 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:23 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-19T03:04:24.169 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:23 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-19T03:04:24.169 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:23 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-19T03:04:24.169 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:23 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-19T03:04:24.169 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:23 smithi155 ceph-mon[35804]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-19T03:04:24.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:23 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3830656583' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-19T03:04:24.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:23 smithi096 ceph-mon[35946]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-19T03:04:24.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:23 smithi096 ceph-mon[35946]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-19T03:04:24.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:23 smithi096 ceph-mon[35946]: Cluster is now healthy 2024-03-19T03:04:24.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:23 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/3830656583' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-19T03:04:24.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:23 smithi096 ceph-mon[35946]: mds.? [v2:172.21.15.155:6835/1249509621,v1:172.21.15.155:6838/1249509621] up:boot 2024-03-19T03:04:24.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:23 smithi096 ceph-mon[35946]: mds.? [v2:172.21.15.96:6838/1590027926,v1:172.21.15.96:6839/1590027926] up:boot 2024-03-19T03:04:24.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:23 smithi096 ceph-mon[35946]: mds.? [v2:172.21.15.155:6834/2142410105,v1:172.21.15.155:6837/2142410105] up:boot 2024-03-19T03:04:24.217 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:23 smithi096 ceph-mon[35946]: mds.? [v2:172.21.15.96:6835/1694329426,v1:172.21.15.96:6836/1694329426] up:boot 2024-03-19T03:04:24.217 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:23 smithi096 ceph-mon[35946]: mds.? [v2:172.21.15.96:6834/2793047534,v1:172.21.15.96:6837/2793047534] up:boot 2024-03-19T03:04:24.217 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:23 smithi096 ceph-mon[35946]: mds.? [v2:172.21.15.155:6836/3377531680,v1:172.21.15.155:6839/3377531680] up:boot 2024-03-19T03:04:24.217 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:23 smithi096 ceph-mon[35946]: fsmap 12 up:standby 2024-03-19T03:04:24.217 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:23 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-19T03:04:24.217 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:23 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-19T03:04:24.217 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:23 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-19T03:04:24.217 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:23 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-19T03:04:24.217 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:23 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-19T03:04:24.217 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:23 smithi096 ceph-mon[35946]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-19T03:04:24.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:23 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3830656583' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-19T03:04:24.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:23 smithi033 ceph-mon[31973]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-19T03:04:24.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:23 smithi033 ceph-mon[31973]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-19T03:04:24.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:23 smithi033 ceph-mon[31973]: Cluster is now healthy 2024-03-19T03:04:24.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:23 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/3830656583' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-19T03:04:24.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:23 smithi033 ceph-mon[31973]: mds.? [v2:172.21.15.155:6835/1249509621,v1:172.21.15.155:6838/1249509621] up:boot 2024-03-19T03:04:24.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:23 smithi033 ceph-mon[31973]: mds.? [v2:172.21.15.96:6838/1590027926,v1:172.21.15.96:6839/1590027926] up:boot 2024-03-19T03:04:24.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:23 smithi033 ceph-mon[31973]: mds.? [v2:172.21.15.155:6834/2142410105,v1:172.21.15.155:6837/2142410105] up:boot 2024-03-19T03:04:24.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:23 smithi033 ceph-mon[31973]: mds.? [v2:172.21.15.96:6835/1694329426,v1:172.21.15.96:6836/1694329426] up:boot 2024-03-19T03:04:24.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:23 smithi033 ceph-mon[31973]: mds.? [v2:172.21.15.96:6834/2793047534,v1:172.21.15.96:6837/2793047534] up:boot 2024-03-19T03:04:24.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:23 smithi033 ceph-mon[31973]: mds.? [v2:172.21.15.155:6836/3377531680,v1:172.21.15.155:6839/3377531680] up:boot 2024-03-19T03:04:24.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:23 smithi033 ceph-mon[31973]: fsmap 12 up:standby 2024-03-19T03:04:24.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:23 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-19T03:04:24.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:23 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-19T03:04:24.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:23 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-19T03:04:24.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:23 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-19T03:04:24.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:23 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-19T03:04:24.280 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:23 smithi033 ceph-mon[31973]: from='mgr.15618 172.21.15.33:0/190755344' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-19T03:04:24.896 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:24.895+0000 7f5d7a7fc640 1 -- 172.21.15.33:0/2342638893 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v110) ==== 181+0+0 (secure 0 0 0) 0x7f5d7c097020 con 0x7f5d841066e0 2024-03-19T03:04:24.896 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'cephfs_metadata' removed 2024-03-19T03:04:24.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:24.898+0000 7f5d8b377640 1 -- 172.21.15.33:0/2342638893 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f5d700a2970 msgr2=0x7f5d700a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:24.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:24.898+0000 7f5d8b377640 1 --2- 172.21.15.33:0/2342638893 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f5d700a2970 0x7f5d700a4e30 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5d8000ab60 tx=0x7f5d8000a2d0 comp rx=0 tx=0).stop 2024-03-19T03:04:24.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:24.899+0000 7f5d8b377640 1 -- 172.21.15.33:0/2342638893 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d841066e0 msgr2=0x7f5d8419c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:24.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:24.899+0000 7f5d8b377640 1 --2- 172.21.15.33:0/2342638893 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d841066e0 0x7f5d8419c8b0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c00eaa0 tx=0x7f5d7c00eea0 comp rx=0 tx=0).stop 2024-03-19T03:04:24.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:24.899+0000 7f5d8b377640 1 -- 172.21.15.33:0/2342638893 shutdown_connections 2024-03-19T03:04:24.900 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:24.899+0000 7f5d8b377640 1 --2- 172.21.15.33:0/2342638893 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f5d700a2970 0x7f5d700a4e30 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:24.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:24.899+0000 7f5d8b377640 1 --2- 172.21.15.33:0/2342638893 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f5d841066e0 0x7f5d8419c8b0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:24.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:24.899+0000 7f5d8b377640 1 --2- 172.21.15.33:0/2342638893 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d84101090 0x7f5d84198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:24.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:24.899+0000 7f5d8b377640 1 --2- 172.21.15.33:0/2342638893 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5d841006d0 0x7f5d84197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:24.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:24.899+0000 7f5d8b377640 1 -- 172.21.15.33:0/2342638893 >> 172.21.15.33:0/2342638893 conn(0x7f5d84076aa0 msgr2=0x7f5d8406a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:24.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:24.899+0000 7f5d8b377640 1 -- 172.21.15.33:0/2342638893 shutdown_connections 2024-03-19T03:04:24.901 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:24.899+0000 7f5d8b377640 1 -- 172.21.15.33:0/2342638893 wait complete. 2024-03-19T03:04:24.914 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-19T03:04:25.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.026+0000 7f2b3f861640 1 -- 172.21.15.33:0/4405082 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b38106750 msgr2=0x7f2b38106b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:25.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.026+0000 7f2b3f861640 1 --2- 172.21.15.33:0/4405082 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b38106750 0x7f2b38106b30 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f2b34009930 tx=0x7f2b3402bc30 comp rx=0 tx=0).stop 2024-03-19T03:04:25.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.026+0000 7f2b3f861640 1 -- 172.21.15.33:0/4405082 shutdown_connections 2024-03-19T03:04:25.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.026+0000 7f2b3f861640 1 --2- 172.21.15.33:0/4405082 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b38101100 0x7f2b3810d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:25.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.026+0000 7f2b3f861640 1 --2- 172.21.15.33:0/4405082 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b38100740 0x7f2b38100bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:25.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.026+0000 7f2b3f861640 1 --2- 172.21.15.33:0/4405082 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b38106750 0x7f2b38106b30 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:25.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.026+0000 7f2b3f861640 1 -- 172.21.15.33:0/4405082 >> 172.21.15.33:0/4405082 conn(0x7f2b38077880 msgr2=0x7f2b38077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:25.030 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.026+0000 7f2b3f861640 1 -- 172.21.15.33:0/4405082 shutdown_connections 2024-03-19T03:04:25.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.026+0000 7f2b3f861640 1 -- 172.21.15.33:0/4405082 wait complete. 2024-03-19T03:04:25.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.027+0000 7f2b3f861640 1 Processor -- start 2024-03-19T03:04:25.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.027+0000 7f2b3f861640 1 -- start start 2024-03-19T03:04:25.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.027+0000 7f2b3f861640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b38100740 0x7f2b38198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:25.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.027+0000 7f2b3f861640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b38101100 0x7f2b381985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:25.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.027+0000 7f2b3f861640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b38106750 0x7f2b3819c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:25.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.027+0000 7f2b3f861640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f2b381109d0 con 0x7f2b38101100 2024-03-19T03:04:25.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.027+0000 7f2b3f861640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2b38110850 con 0x7f2b38106750 2024-03-19T03:04:25.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.027+0000 7f2b3f861640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2b38110b50 con 0x7f2b38100740 2024-03-19T03:04:25.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.027+0000 7f2b3cdfc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b38101100 0x7f2b381985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:25.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.027+0000 7f2b3cdfc640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b38101100 0x7f2b381985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58276/0 (socket says 172.21.15.33:58276) 2024-03-19T03:04:25.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.027+0000 7f2b3cdfc640 1 -- 172.21.15.33:0/149949577 learned_addr learned my addr 172.21.15.33:0/149949577 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:04:25.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.028+0000 7f2b3ddfe640 1 --2- 172.21.15.33:0/149949577 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b38106750 0x7f2b3819c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:25.031 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.028+0000 7f2b3d5fd640 1 --2- 172.21.15.33:0/149949577 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b38100740 0x7f2b38198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:25.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.028+0000 7f2b3cdfc640 1 -- 172.21.15.33:0/149949577 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b38100740 msgr2=0x7f2b38198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:25.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.028+0000 7f2b3cdfc640 1 --2- 172.21.15.33:0/149949577 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b38100740 0x7f2b38198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:25.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.028+0000 7f2b3cdfc640 1 -- 172.21.15.33:0/149949577 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b38106750 msgr2=0x7f2b3819c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:25.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.028+0000 7f2b3cdfc640 1 --2- 172.21.15.33:0/149949577 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b38106750 0x7f2b3819c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:25.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.028+0000 7f2b3cdfc640 1 -- 172.21.15.33:0/149949577 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b3819d020 con 0x7f2b38101100 2024-03-19T03:04:25.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.028+0000 7f2b3d5fd640 1 --2- 172.21.15.33:0/149949577 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b38100740 0x7f2b38198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:04:25.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.028+0000 7f2b3cdfc640 1 --2- 172.21.15.33:0/149949577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b38101100 0x7f2b381985c0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f2b2800d950 tx=0x7f2b2800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:25.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.028+0000 7f2b2e7fc640 1 -- 172.21.15.33:0/149949577 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b28017030 con 0x7f2b38101100 2024-03-19T03:04:25.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.028+0000 7f2b3f861640 1 -- 172.21.15.33:0/149949577 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b3819d310 con 0x7f2b38101100 2024-03-19T03:04:25.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.028+0000 7f2b2e7fc640 1 -- 172.21.15.33:0/149949577 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2b28004020 con 0x7f2b38101100 2024-03-19T03:04:25.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.028+0000 7f2b2e7fc640 1 -- 172.21.15.33:0/149949577 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b280047a0 con 0x7f2b38101100 2024-03-19T03:04:25.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.028+0000 7f2b3f861640 1 -- 172.21.15.33:0/149949577 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b381a4ba0 con 0x7f2b38101100 2024-03-19T03:04:25.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.029+0000 7f2b3f861640 1 -- 172.21.15.33:0/149949577 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b38068a60 con 0x7f2b38101100 2024-03-19T03:04:25.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.030+0000 7f2b2e7fc640 1 -- 172.21.15.33:0/149949577 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f2b280041d0 con 0x7f2b38101100 2024-03-19T03:04:25.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.031+0000 7f2b2e7fc640 1 --2- 172.21.15.33:0/149949577 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f2b240a2970 0x7f2b240a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:25.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.031+0000 7f2b3d5fd640 1 --2- 172.21.15.33:0/149949577 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f2b240a2970 0x7f2b240a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:25.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.031+0000 7f2b2e7fc640 1 -- 172.21.15.33:0/149949577 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9141+0+0 (secure 0 0 0) 0x7f2b280cf1f0 con 0x7f2b38101100 2024-03-19T03:04:25.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.032+0000 7f2b3d5fd640 1 --2- 172.21.15.33:0/149949577 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f2b240a2970 0x7f2b240a4e30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2b34006010 tx=0x7f2b34009610 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:25.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.035+0000 7f2b2e7fc640 1 -- 172.21.15.33:0/149949577 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2b2809bce0 con 0x7f2b38101100 2024-03-19T03:04:25.189 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:24 smithi155 ceph-mon[35804]: pgmap v342: 129 pgs: 129 active+clean; 1.9 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 895 B/s wr, 8 op/s 2024-03-19T03:04:25.189 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:24 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2342638893' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T03:04:25.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:24 smithi096 ceph-mon[35946]: pgmap v342: 129 pgs: 129 active+clean; 1.9 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 895 B/s wr, 8 op/s 2024-03-19T03:04:25.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:24 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2342638893' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T03:04:25.223 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.222+0000 7f2b3f861640 1 -- 172.21.15.33:0/149949577 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f2b38109cf0 con 0x7f2b38101100 2024-03-19T03:04:25.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:24 smithi033 ceph-mon[31973]: pgmap v342: 129 pgs: 129 active+clean; 1.9 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 895 B/s wr, 8 op/s 2024-03-19T03:04:25.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:24 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2342638893' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T03:04:25.902 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.900+0000 7f2b2e7fc640 1 -- 172.21.15.33:0/149949577 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v111) ==== 169+0+0 (secure 0 0 0) 0x7f2b280cce20 con 0x7f2b38101100 2024-03-19T03:04:25.902 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'cephfs_data' removed 2024-03-19T03:04:25.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.903+0000 7f2b3f861640 1 -- 172.21.15.33:0/149949577 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f2b240a2970 msgr2=0x7f2b240a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:25.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.903+0000 7f2b3f861640 1 --2- 172.21.15.33:0/149949577 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f2b240a2970 0x7f2b240a4e30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2b34006010 tx=0x7f2b34009610 comp rx=0 tx=0).stop 2024-03-19T03:04:25.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.903+0000 7f2b3f861640 1 -- 172.21.15.33:0/149949577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b38101100 msgr2=0x7f2b381985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:25.904 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.903+0000 7f2b3f861640 1 --2- 172.21.15.33:0/149949577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b38101100 0x7f2b381985c0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f2b2800d950 tx=0x7f2b2800de10 comp rx=0 tx=0).stop 2024-03-19T03:04:25.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.903+0000 7f2b3f861640 1 -- 172.21.15.33:0/149949577 shutdown_connections 2024-03-19T03:04:25.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.903+0000 7f2b3f861640 1 --2- 172.21.15.33:0/149949577 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f2b240a2970 0x7f2b240a4e30 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:25.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.903+0000 7f2b3f861640 1 --2- 172.21.15.33:0/149949577 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b38106750 0x7f2b3819c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:25.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.903+0000 7f2b3f861640 1 --2- 172.21.15.33:0/149949577 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f2b38101100 0x7f2b381985c0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:25.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.903+0000 7f2b3f861640 1 --2- 172.21.15.33:0/149949577 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b38100740 0x7f2b38198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:25.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.903+0000 7f2b3f861640 1 -- 172.21.15.33:0/149949577 >> 172.21.15.33:0/149949577 conn(0x7f2b38077880 msgr2=0x7f2b3806a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:25.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.903+0000 7f2b3f861640 1 -- 172.21.15.33:0/149949577 shutdown_connections 2024-03-19T03:04:25.905 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:25.903+0000 7f2b3f861640 1 -- 172.21.15.33:0/149949577 wait complete. 2024-03-19T03:04:25.920 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-19T03:04:26.032 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.029+0000 7f12d13f3640 1 -- 172.21.15.33:0/2817685967 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f12cc100390 msgr2=0x7f12cc100770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:26.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.029+0000 7f12d13f3640 1 --2- 172.21.15.33:0/2817685967 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f12cc100390 0x7f12cc100770 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f12bc009900 tx=0x7f12bc02bc30 comp rx=0 tx=0).stop 2024-03-19T03:04:26.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.029+0000 7f12d13f3640 1 -- 172.21.15.33:0/2817685967 shutdown_connections 2024-03-19T03:04:26.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.029+0000 7f12d13f3640 1 --2- 172.21.15.33:0/2817685967 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12cc106b50 0x7f12cc10e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:26.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.029+0000 7f12d13f3640 1 --2- 172.21.15.33:0/2817685967 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f12cc100d40 0x7f12cc106470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:26.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.029+0000 7f12d13f3640 1 --2- 172.21.15.33:0/2817685967 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f12cc100390 0x7f12cc100770 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:26.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.029+0000 7f12d13f3640 1 -- 172.21.15.33:0/2817685967 >> 172.21.15.33:0/2817685967 conn(0x7f12cc075980 msgr2=0x7f12cc075d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:26.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.029+0000 7f12d13f3640 1 -- 172.21.15.33:0/2817685967 shutdown_connections 2024-03-19T03:04:26.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.029+0000 7f12d13f3640 1 -- 172.21.15.33:0/2817685967 wait complete. 2024-03-19T03:04:26.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.029+0000 7f12d13f3640 1 Processor -- start 2024-03-19T03:04:26.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.030+0000 7f12d13f3640 1 -- start start 2024-03-19T03:04:26.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.030+0000 7f12d13f3640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f12cc100390 0x7f12cc198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:26.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.030+0000 7f12d13f3640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f12cc106b50 0x7f12cc198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:26.033 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.030+0000 7f12d13f3640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12cc19c8f0 0x7f12cc19cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:26.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.030+0000 7f12d13f3640 1 -- --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_getmap magic: 0 -- 0x7f12cc111af0 con 0x7f12cc100390 2024-03-19T03:04:26.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.030+0000 7f12d13f3640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f12cc111970 con 0x7f12cc106b50 2024-03-19T03:04:26.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.030+0000 7f12d13f3640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f12cc111c70 con 0x7f12cc19c8f0 2024-03-19T03:04:26.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.030+0000 7f12caffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f12cc100390 0x7f12cc198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:26.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.030+0000 7f12caffd640 1 --2- >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f12cc100390 0x7f12cc198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.33:3300/0 says I am v2:172.21.15.33:58296/0 (socket says 172.21.15.33:58296) 2024-03-19T03:04:26.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.030+0000 7f12caffd640 1 -- 172.21.15.33:0/871506479 learned_addr learned my addr 172.21.15.33:0/871506479 (peer_addr_for_me v2:172.21.15.33:0/0) 2024-03-19T03:04:26.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.030+0000 7f12cb7fe640 1 --2- 172.21.15.33:0/871506479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12cc19c8f0 0x7f12cc19cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:26.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.030+0000 7f12ca7fc640 1 --2- 172.21.15.33:0/871506479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f12cc106b50 0x7f12cc198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:26.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.030+0000 7f12caffd640 1 -- 172.21.15.33:0/871506479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12cc19c8f0 msgr2=0x7f12cc19cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:26.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.030+0000 7f12caffd640 1 --2- 172.21.15.33:0/871506479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12cc19c8f0 0x7f12cc19cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:26.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.030+0000 7f12caffd640 1 -- 172.21.15.33:0/871506479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f12cc106b50 msgr2=0x7f12cc198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:26.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.030+0000 7f12caffd640 1 --2- 172.21.15.33:0/871506479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f12cc106b50 0x7f12cc198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:26.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.030+0000 7f12caffd640 1 -- 172.21.15.33:0/871506479 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12cc19d3e0 con 0x7f12cc100390 2024-03-19T03:04:26.034 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.030+0000 7f12cb7fe640 1 --2- 172.21.15.33:0/871506479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12cc19c8f0 0x7f12cc19cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:04:26.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.030+0000 7f12ca7fc640 1 --2- 172.21.15.33:0/871506479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f12cc106b50 0x7f12cc198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:04:26.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.030+0000 7f12caffd640 1 --2- 172.21.15.33:0/871506479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f12cc100390 0x7f12cc198030 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f12bc005940 tx=0x7f12bc02c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:26.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.031+0000 7f12b3fff640 1 -- 172.21.15.33:0/871506479 <== mon.0 v2:172.21.15.33:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12bc043020 con 0x7f12cc100390 2024-03-19T03:04:26.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.031+0000 7f12d13f3640 1 -- 172.21.15.33:0/871506479 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12cc19d670 con 0x7f12cc100390 2024-03-19T03:04:26.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.031+0000 7f12b3fff640 1 -- 172.21.15.33:0/871506479 <== mon.0 v2:172.21.15.33:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f12bc02cbe0 con 0x7f12cc100390 2024-03-19T03:04:26.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.031+0000 7f12b3fff640 1 -- 172.21.15.33:0/871506479 <== mon.0 v2:172.21.15.33:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12bc003bf0 con 0x7f12cc100390 2024-03-19T03:04:26.035 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.031+0000 7f12d13f3640 1 -- 172.21.15.33:0/871506479 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12cc1a4e90 con 0x7f12cc100390 2024-03-19T03:04:26.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.032+0000 7f12d13f3640 1 -- 172.21.15.33:0/871506479 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12cc068a60 con 0x7f12cc100390 2024-03-19T03:04:26.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.033+0000 7f12b3fff640 1 -- 172.21.15.33:0/871506479 <== mon.0 v2:172.21.15.33:3300/0 4 ==== mgrmap(e 25) ==== 137421+0+0 (secure 0 0 0) 0x7f12bc046020 con 0x7f12cc100390 2024-03-19T03:04:26.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.033+0000 7f12b3fff640 1 --2- 172.21.15.33:0/871506479 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f12b40a2970 0x7f12b40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:04:26.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.033+0000 7f12ca7fc640 1 --2- 172.21.15.33:0/871506479 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f12b40a2970 0x7f12b40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:04:26.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.033+0000 7f12b3fff640 1 -- 172.21.15.33:0/871506479 <== mon.0 v2:172.21.15.33:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8582+0+0 (secure 0 0 0) 0x7f12bc0f10e0 con 0x7f12cc100390 2024-03-19T03:04:26.036 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.034+0000 7f12ca7fc640 1 --2- 172.21.15.33:0/871506479 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f12b40a2970 0x7f12b40a4e30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f12c00058c0 tx=0x7f12c0005850 comp rx=0 tx=0).ready entity=mgr.15618 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:04:26.039 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.038+0000 7f12b3fff640 1 -- 172.21.15.33:0/871506479 <== mon.0 v2:172.21.15.33:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f12bc04d020 con 0x7f12cc100390 2024-03-19T03:04:26.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:25 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/2342638893' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-19T03:04:26.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:25 smithi096 ceph-mon[35946]: osdmap e110: 12 total, 12 up, 12 in 2024-03-19T03:04:26.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:25 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/149949577' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T03:04:26.235 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.234+0000 7f12d13f3640 1 -- 172.21.15.33:0/871506479 --> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f12cc109d70 con 0x7f12cc100390 2024-03-19T03:04:26.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:25 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/2342638893' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-19T03:04:26.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:25 smithi155 ceph-mon[35804]: osdmap e110: 12 total, 12 up, 12 in 2024-03-19T03:04:26.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:25 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/149949577' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T03:04:26.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:25 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/2342638893' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-19T03:04:26.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:25 smithi033 ceph-mon[31973]: osdmap e110: 12 total, 12 up, 12 in 2024-03-19T03:04:26.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:25 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/149949577' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T03:04:26.908 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.906+0000 7f12b3fff640 1 -- 172.21.15.33:0/871506479 <== mon.0 v2:172.21.15.33:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v112) ==== 178+0+0 (secure 0 0 0) 0x7f12bc0bdf10 con 0x7f12cc100390 2024-03-19T03:04:26.908 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'cephfs_data_ec' removed 2024-03-19T03:04:26.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.910+0000 7f12d13f3640 1 -- 172.21.15.33:0/871506479 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f12b40a2970 msgr2=0x7f12b40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:26.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.910+0000 7f12d13f3640 1 --2- 172.21.15.33:0/871506479 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f12b40a2970 0x7f12b40a4e30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f12c00058c0 tx=0x7f12c0005850 comp rx=0 tx=0).stop 2024-03-19T03:04:26.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.910+0000 7f12d13f3640 1 -- 172.21.15.33:0/871506479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f12cc100390 msgr2=0x7f12cc198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:04:26.911 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.910+0000 7f12d13f3640 1 --2- 172.21.15.33:0/871506479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f12cc100390 0x7f12cc198030 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f12bc005940 tx=0x7f12bc02c7e0 comp rx=0 tx=0).stop 2024-03-19T03:04:26.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.910+0000 7f12d13f3640 1 -- 172.21.15.33:0/871506479 shutdown_connections 2024-03-19T03:04:26.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.910+0000 7f12d13f3640 1 --2- 172.21.15.33:0/871506479 >> [v2:172.21.15.33:6800/2362023661,v1:172.21.15.33:6801/2362023661] conn(0x7f12b40a2970 0x7f12b40a4e30 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:26.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.910+0000 7f12d13f3640 1 --2- 172.21.15.33:0/871506479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12cc19c8f0 0x7f12cc19cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:26.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.910+0000 7f12d13f3640 1 --2- 172.21.15.33:0/871506479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f12cc106b50 0x7f12cc198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:26.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.910+0000 7f12d13f3640 1 --2- 172.21.15.33:0/871506479 >> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] conn(0x7f12cc100390 0x7f12cc198030 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:04:26.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.910+0000 7f12d13f3640 1 -- 172.21.15.33:0/871506479 >> 172.21.15.33:0/871506479 conn(0x7f12cc075980 msgr2=0x7f12cc0fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:04:26.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.910+0000 7f12d13f3640 1 -- 172.21.15.33:0/871506479 shutdown_connections 2024-03-19T03:04:26.912 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-19T03:04:26.911+0000 7f12d13f3640 1 -- 172.21.15.33:0/871506479 wait complete. 2024-03-19T03:04:26.931 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-19T03:04:26.963 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-19T03:04:26.997 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-19T03:04:27.028 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-19T03:04:27.028 DEBUG:teuthology.orchestra.run.smithi033:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-19T03:04:27.044 DEBUG:teuthology.orchestra.run.smithi096:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-19T03:04:27.059 DEBUG:teuthology.orchestra.run.smithi155:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-19T03:04:27.083 INFO:tasks.cephadm:Stopping all daemons... 2024-03-19T03:04:27.083 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-19T03:04:27.083 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mon.a 2024-03-19T03:04:27.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:26 smithi096 ceph-mon[35946]: pgmap v344: 97 pgs: 97 active+clean; 577 KiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:04:27.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:26 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/149949577' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-19T03:04:27.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:26 smithi096 ceph-mon[35946]: osdmap e111: 12 total, 12 up, 12 in 2024-03-19T03:04:27.216 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:26 smithi096 ceph-mon[35946]: from='client.? 172.21.15.33:0/871506479' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T03:04:27.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:26 smithi155 ceph-mon[35804]: pgmap v344: 97 pgs: 97 active+clean; 577 KiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:04:27.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:26 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/149949577' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-19T03:04:27.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:26 smithi155 ceph-mon[35804]: osdmap e111: 12 total, 12 up, 12 in 2024-03-19T03:04:27.267 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:26 smithi155 ceph-mon[35804]: from='client.? 172.21.15.33:0/871506479' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T03:04:27.278 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:26 smithi033 ceph-mon[31973]: pgmap v344: 97 pgs: 97 active+clean; 577 KiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:04:27.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:26 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/149949577' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-19T03:04:27.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:26 smithi033 ceph-mon[31973]: osdmap e111: 12 total, 12 up, 12 in 2024-03-19T03:04:27.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:26 smithi033 ceph-mon[31973]: from='client.? 172.21.15.33:0/871506479' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T03:04:27.279 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:27 smithi033 systemd[1]: Stopping Ceph mon.a for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T03:04:27.689 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:27 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mon-a[31969]: 2024-03-19T03:04:27.394+0000 7f6d61d7f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:04:27.689 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:27 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mon-a[31969]: 2024-03-19T03:04:27.394+0000 7f6d61d7f700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-19T03:04:27.689 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:27 smithi033 podman[101260]: 2024-03-19 03:04:27.416699265 +0000 UTC m=+0.134036009 container died 24ae7c13ccaa60b50b98a22bf85d87f102639af4a19cc16cb6f753ec99f1ba13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0) 2024-03-19T03:04:28.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:27 smithi033 podman[101260]: 2024-03-19 03:04:27.787117769 +0000 UTC m=+0.504454514 container cleanup 24ae7c13ccaa60b50b98a22bf85d87f102639af4a19cc16cb6f753ec99f1ba13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T03:04:28.029 INFO:journalctl@ceph.mon.a.smithi033.stdout:Mar 19 03:04:27 smithi033 bash[101260]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mon-a 2024-03-19T03:04:28.264 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mon.a.service' 2024-03-19T03:04:28.310 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:04:28.310 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-19T03:04:28.310 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-19T03:04:28.311 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mon.b 2024-03-19T03:04:28.639 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:28 smithi096 systemd[1]: Stopping Ceph mon.b for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T03:04:28.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:28 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mon-b[35942]: 2024-03-19T03:04:28.636+0000 7f729d9fd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:04:28.966 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:28 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mon-b[35942]: 2024-03-19T03:04:28.636+0000 7f729d9fd700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-19T03:04:28.967 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:28 smithi096 podman[70850]: 2024-03-19 03:04:28.883176661 +0000 UTC m=+0.376460535 container died 787ee6c286fe9c4ac3783c5fb2474b6e9374f6768b9d40a8271e16690532e09d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T03:04:29.401 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:29 smithi096 podman[70850]: 2024-03-19 03:04:29.304725957 +0000 UTC m=+0.798009824 container cleanup 787ee6c286fe9c4ac3783c5fb2474b6e9374f6768b9d40a8271e16690532e09d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mon-b, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-19T03:04:29.401 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:29 smithi096 bash[70850]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mon-b 2024-03-19T03:04:29.660 INFO:journalctl@ceph.mon.b.smithi096.stdout:Mar 19 03:04:29 smithi096 podman[70865]: 2024-03-19 03:04:29.539982504 +0000 UTC m=+0.651157550 container remove 787ee6c286fe9c4ac3783c5fb2474b6e9374f6768b9d40a8271e16690532e09d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mon-b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0) 2024-03-19T03:04:29.673 DEBUG:teuthology.orchestra.run.smithi096:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mon.b.service' 2024-03-19T03:04:29.726 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:04:29.726 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-19T03:04:29.726 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-19T03:04:29.726 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mon.c 2024-03-19T03:04:30.017 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:29 smithi155 systemd[1]: Stopping Ceph mon.c for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T03:04:30.484 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:30 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mon-c[35798]: 2024-03-19T03:04:30.050+0000 7f5578a1a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:04:30.484 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:30 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mon-c[35798]: 2024-03-19T03:04:30.050+0000 7f5578a1a700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-19T03:04:30.484 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:30 smithi155 podman[72114]: 2024-03-19 03:04:30.174100644 +0000 UTC m=+0.252570421 container died 326f798dcb9b206a7fc645e4c63fdb1afc46b85c37b3b99a67c99d2242a36bc1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mon-c, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-19T03:04:30.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:30 smithi155 podman[72114]: 2024-03-19 03:04:30.577933067 +0000 UTC m=+0.656402887 container cleanup 326f798dcb9b206a7fc645e4c63fdb1afc46b85c37b3b99a67c99d2242a36bc1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mon-c, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T03:04:30.767 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:30 smithi155 bash[72114]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mon-c 2024-03-19T03:04:31.044 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mon.c.service' 2024-03-19T03:04:31.068 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:30 smithi155 podman[72128]: 2024-03-19 03:04:30.804348795 +0000 UTC m=+0.626805008 container remove 326f798dcb9b206a7fc645e4c63fdb1afc46b85c37b3b99a67c99d2242a36bc1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mon-c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-19T03:04:31.069 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:31 smithi155 systemd[1]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mon.c.service: Deactivated successfully. 2024-03-19T03:04:31.069 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:31 smithi155 systemd[1]: Stopped Ceph mon.c for 2b7a8938-e59a-11ee-95c9-87774f69a715. 2024-03-19T03:04:31.069 INFO:journalctl@ceph.mon.c.smithi155.stdout:Mar 19 03:04:31 smithi155 systemd[1]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mon.c.service: Consumed 11.250s CPU time. 2024-03-19T03:04:31.120 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:04:31.120 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-19T03:04:31.120 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-19T03:04:31.121 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mgr.x 2024-03-19T03:04:31.450 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 03:04:31 smithi033 systemd[1]: Stopping Ceph mgr.x for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T03:04:31.761 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 03:04:31 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T03:04:31.448+0000 7f1cd62a4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:04:31.761 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 03:04:31 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x[32204]: 2024-03-19T03:04:31.448+0000 7f1cd62a4700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-19T03:04:31.761 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 03:04:31 smithi033 podman[101381]: 2024-03-19 03:04:31.48120709 +0000 UTC m=+0.167819940 container died ed11015124be9cc4cc29ee053e312c8fc9f394c5a33d6aa081bfd911c744dbfd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T03:04:32.029 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 03:04:31 smithi033 podman[101381]: 2024-03-19 03:04:31.876147742 +0000 UTC m=+0.562760590 container cleanup ed11015124be9cc4cc29ee053e312c8fc9f394c5a33d6aa081bfd911c744dbfd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-19T03:04:32.029 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 03:04:31 smithi033 bash[101381]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x 2024-03-19T03:04:32.330 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mgr.x.service' 2024-03-19T03:04:32.354 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 03:04:32 smithi033 podman[101395]: 2024-03-19 03:04:32.103140554 +0000 UTC m=+0.620089450 container remove ed11015124be9cc4cc29ee053e312c8fc9f394c5a33d6aa081bfd911c744dbfd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T03:04:32.354 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 03:04:32 smithi033 systemd[1]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mgr.x.service: Deactivated successfully. 2024-03-19T03:04:32.354 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 03:04:32 smithi033 systemd[1]: Stopped Ceph mgr.x for 2b7a8938-e59a-11ee-95c9-87774f69a715. 2024-03-19T03:04:32.354 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Mar 19 03:04:32 smithi033 systemd[1]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mgr.x.service: Consumed 1min 28.280s CPU time. 2024-03-19T03:04:32.367 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:04:32.367 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-19T03:04:32.368 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-19T03:04:32.368 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mgr.y 2024-03-19T03:04:32.706 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 03:04:32 smithi096 systemd[1]: Stopping Ceph mgr.y for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T03:04:32.966 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 03:04:32 smithi096 podman[70985]: 2024-03-19 03:04:32.705347563 +0000 UTC m=+0.140647630 container died 9e45c9a1db3572c2db31df1181993964559fd9132323770875dd591f45127423 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T03:04:33.362 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 03:04:33 smithi096 podman[70985]: 2024-03-19 03:04:33.118816562 +0000 UTC m=+0.554116629 container cleanup 9e45c9a1db3572c2db31df1181993964559fd9132323770875dd591f45127423 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True) 2024-03-19T03:04:33.363 INFO:journalctl@ceph.mgr.y.smithi096.stdout:Mar 19 03:04:33 smithi096 bash[70985]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-y 2024-03-19T03:04:33.487 DEBUG:teuthology.orchestra.run.smithi096:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mgr.y.service' 2024-03-19T03:04:33.525 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:04:33.526 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-19T03:04:33.526 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-19T03:04:33.526 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mgr.z 2024-03-19T03:04:33.937 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 03:04:33 smithi155 systemd[1]: Stopping Ceph mgr.z for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T03:04:34.243 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 03:04:33 smithi155 podman[72239]: 2024-03-19 03:04:33.936755482 +0000 UTC m=+0.214830212 container died eabf96d16ca6f70d8350d501a4d2e02b1c2c309d16113004bddaa6671132566a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T03:04:34.517 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 03:04:34 smithi155 podman[72239]: 2024-03-19 03:04:34.358395296 +0000 UTC m=+0.636470033 container cleanup eabf96d16ca6f70d8350d501a4d2e02b1c2c309d16113004bddaa6671132566a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-19T03:04:34.517 INFO:journalctl@ceph.mgr.z.smithi155.stdout:Mar 19 03:04:34 smithi155 bash[72239]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mgr-z 2024-03-19T03:04:34.778 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mgr.z.service' 2024-03-19T03:04:34.815 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:04:34.815 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-19T03:04:34.815 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-19T03:04:34.815 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.0 2024-03-19T03:04:35.114 INFO:journalctl@ceph.osd.0.smithi033.stdout:Mar 19 03:04:34 smithi033 systemd[1]: Stopping Ceph osd.0 for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T03:04:35.529 INFO:journalctl@ceph.osd.0.smithi033.stdout:Mar 19 03:04:35 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-0[46485]: 2024-03-19T03:04:35.113+0000 7fb1046d4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:04:35.529 INFO:journalctl@ceph.osd.0.smithi033.stdout:Mar 19 03:04:35 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-0[46485]: 2024-03-19T03:04:35.113+0000 7fb1046d4700 -1 osd.0 112 *** Got signal Terminated *** 2024-03-19T03:04:35.529 INFO:journalctl@ceph.osd.0.smithi033.stdout:Mar 19 03:04:35 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-0[46485]: 2024-03-19T03:04:35.113+0000 7fb1046d4700 -1 osd.0 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T03:04:40.766 INFO:journalctl@ceph.osd.0.smithi033.stdout:Mar 19 03:04:40 smithi033 podman[101504]: 2024-03-19 03:04:40.466629898 +0000 UTC m=+5.463556455 container died 3595ce771f88fc27ceb87894d76b2c2f3ceb66460a1dc1795032072ff02bd14b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2024-03-19T03:04:41.029 INFO:journalctl@ceph.osd.0.smithi033.stdout:Mar 19 03:04:40 smithi033 podman[101504]: 2024-03-19 03:04:40.855984719 +0000 UTC m=+5.852911278 container cleanup 3595ce771f88fc27ceb87894d76b2c2f3ceb66460a1dc1795032072ff02bd14b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-19T03:04:41.029 INFO:journalctl@ceph.osd.0.smithi033.stdout:Mar 19 03:04:40 smithi033 bash[101504]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-0 2024-03-19T03:04:41.472 INFO:journalctl@ceph.osd.0.smithi033.stdout:Mar 19 03:04:41 smithi033 podman[101521]: 2024-03-19 03:04:41.116105127 +0000 UTC m=+0.643570720 container remove 3595ce771f88fc27ceb87894d76b2c2f3ceb66460a1dc1795032072ff02bd14b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS) 2024-03-19T03:04:41.754 INFO:journalctl@ceph.osd.0.smithi033.stdout:Mar 19 03:04:41 smithi033 podman[101614]: 2024-03-19 03:04:41.374354517 +0000 UTC m=+0.018629214 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:04:41.754 INFO:journalctl@ceph.osd.0.smithi033.stdout:Mar 19 03:04:41 smithi033 podman[101614]: 2024-03-19 03:04:41.48396155 +0000 UTC m=+0.128236285 container create 0298f68d72d186e88261e57d9266015993ae7e24d626585b3671da749cfc4d55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-0-deactivate, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-19T03:04:42.029 INFO:journalctl@ceph.osd.0.smithi033.stdout:Mar 19 03:04:41 smithi033 podman[101614]: 2024-03-19 03:04:41.753314908 +0000 UTC m=+0.397589600 container init 0298f68d72d186e88261e57d9266015993ae7e24d626585b3671da749cfc4d55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2) 2024-03-19T03:04:42.029 INFO:journalctl@ceph.osd.0.smithi033.stdout:Mar 19 03:04:41 smithi033 podman[101614]: 2024-03-19 03:04:41.757046522 +0000 UTC m=+0.401321217 container start 0298f68d72d186e88261e57d9266015993ae7e24d626585b3671da749cfc4d55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-19T03:04:42.029 INFO:journalctl@ceph.osd.0.smithi033.stdout:Mar 19 03:04:41 smithi033 podman[101614]: 2024-03-19 03:04:41.861828472 +0000 UTC m=+0.506103161 container attach 0298f68d72d186e88261e57d9266015993ae7e24d626585b3671da749cfc4d55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-0-deactivate, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-19T03:04:42.338 INFO:journalctl@ceph.osd.0.smithi033.stdout:Mar 19 03:04:42 smithi033 podman[101614]: 2024-03-19 03:04:42.052623935 +0000 UTC m=+0.696898633 container died 0298f68d72d186e88261e57d9266015993ae7e24d626585b3671da749cfc4d55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-19T03:04:42.667 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.0.service' 2024-03-19T03:04:42.708 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:04:42.708 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-19T03:04:42.709 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-19T03:04:42.709 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.1 2024-03-19T03:04:43.032 INFO:journalctl@ceph.osd.1.smithi096.stdout:Mar 19 03:04:42 smithi096 systemd[1]: Stopping Ceph osd.1 for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T03:04:43.466 INFO:journalctl@ceph.osd.1.smithi096.stdout:Mar 19 03:04:43 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-1[40541]: 2024-03-19T03:04:43.030+0000 7f00bcc1e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:04:43.466 INFO:journalctl@ceph.osd.1.smithi096.stdout:Mar 19 03:04:43 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-1[40541]: 2024-03-19T03:04:43.030+0000 7f00bcc1e700 -1 osd.1 112 *** Got signal Terminated *** 2024-03-19T03:04:43.466 INFO:journalctl@ceph.osd.1.smithi096.stdout:Mar 19 03:04:43 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-1[40541]: 2024-03-19T03:04:43.030+0000 7f00bcc1e700 -1 osd.1 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T03:04:48.899 INFO:journalctl@ceph.osd.1.smithi096.stdout:Mar 19 03:04:48 smithi096 podman[71111]: 2024-03-19 03:04:48.576086664 +0000 UTC m=+5.680291295 container died 3c14e4d75f7b12abff1af545df519fc42d2e5fd153ed923be15d28975abdf7bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-1, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T03:04:49.216 INFO:journalctl@ceph.osd.1.smithi096.stdout:Mar 19 03:04:48 smithi096 podman[71111]: 2024-03-19 03:04:48.980867367 +0000 UTC m=+6.085072043 container cleanup 3c14e4d75f7b12abff1af545df519fc42d2e5fd153ed923be15d28975abdf7bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-1, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True) 2024-03-19T03:04:49.216 INFO:journalctl@ceph.osd.1.smithi096.stdout:Mar 19 03:04:48 smithi096 bash[71111]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-1 2024-03-19T03:04:49.591 INFO:journalctl@ceph.osd.1.smithi096.stdout:Mar 19 03:04:49 smithi096 podman[71128]: 2024-03-19 03:04:49.249016113 +0000 UTC m=+0.670450310 container remove 3c14e4d75f7b12abff1af545df519fc42d2e5fd153ed923be15d28975abdf7bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-1, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True) 2024-03-19T03:04:49.846 INFO:journalctl@ceph.osd.1.smithi096.stdout:Mar 19 03:04:49 smithi096 podman[71222]: 2024-03-19 03:04:49.493421986 +0000 UTC m=+0.018006937 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:04:49.846 INFO:journalctl@ceph.osd.1.smithi096.stdout:Mar 19 03:04:49 smithi096 podman[71222]: 2024-03-19 03:04:49.599680456 +0000 UTC m=+0.124265400 container create eb3457a8450e24d0c01769eb7de574f67bf302367ff52b2f4f18974d385cd3c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-19T03:04:49.846 INFO:journalctl@ceph.osd.1.smithi096.stdout:Mar 19 03:04:49 smithi096 podman[71222]: 2024-03-19 03:04:49.845302003 +0000 UTC m=+0.369886950 container init eb3457a8450e24d0c01769eb7de574f67bf302367ff52b2f4f18974d385cd3c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-19T03:04:50.145 INFO:journalctl@ceph.osd.1.smithi096.stdout:Mar 19 03:04:49 smithi096 podman[71222]: 2024-03-19 03:04:49.849073018 +0000 UTC m=+0.373657964 container start eb3457a8450e24d0c01769eb7de574f67bf302367ff52b2f4f18974d385cd3c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-19T03:04:50.145 INFO:journalctl@ceph.osd.1.smithi096.stdout:Mar 19 03:04:49 smithi096 podman[71222]: 2024-03-19 03:04:49.927872739 +0000 UTC m=+0.452457694 container attach eb3457a8450e24d0c01769eb7de574f67bf302367ff52b2f4f18974d385cd3c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-19T03:04:50.430 INFO:journalctl@ceph.osd.1.smithi096.stdout:Mar 19 03:04:50 smithi096 podman[71222]: 2024-03-19 03:04:50.145243297 +0000 UTC m=+0.669828255 container died eb3457a8450e24d0c01769eb7de574f67bf302367ff52b2f4f18974d385cd3c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-19T03:04:50.802 DEBUG:teuthology.orchestra.run.smithi096:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.1.service' 2024-03-19T03:04:50.845 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:04:50.846 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-19T03:04:50.846 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-19T03:04:50.846 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.2 2024-03-19T03:04:51.267 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 19 03:04:50 smithi155 systemd[1]: Stopping Ceph osd.2 for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T03:04:51.767 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 19 03:04:51 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-2[41086]: 2024-03-19T03:04:51.337+0000 7fb20c58a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:04:51.767 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 19 03:04:51 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-2[41086]: 2024-03-19T03:04:51.337+0000 7fb20c58a700 -1 osd.2 112 *** Got signal Terminated *** 2024-03-19T03:04:51.767 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 19 03:04:51 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-2[41086]: 2024-03-19T03:04:51.337+0000 7fb20c58a700 -1 osd.2 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T03:04:57.008 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 19 03:04:56 smithi155 podman[72363]: 2024-03-19 03:04:56.697545333 +0000 UTC m=+5.656792214 container died 9abbde5ee2550df43469c2e490f4c8e0f05b996e06f777abef848477ec71f423 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T03:04:57.267 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 19 03:04:57 smithi155 podman[72363]: 2024-03-19 03:04:57.097348587 +0000 UTC m=+6.056595477 container cleanup 9abbde5ee2550df43469c2e490f4c8e0f05b996e06f777abef848477ec71f423 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-19T03:04:57.267 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 19 03:04:57 smithi155 bash[72363]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-2 2024-03-19T03:04:57.633 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 19 03:04:57 smithi155 podman[72380]: 2024-03-19 03:04:57.332192845 +0000 UTC m=+0.634943731 container remove 9abbde5ee2550df43469c2e490f4c8e0f05b996e06f777abef848477ec71f423 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2) 2024-03-19T03:04:57.978 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 19 03:04:57 smithi155 podman[72471]: 2024-03-19 03:04:57.633055231 +0000 UTC m=+0.109259347 container create b25a80f4af5fd71f77eb61a8de4bf43b9626a5be06293201e1491e51cc8236af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-19T03:04:57.978 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 19 03:04:57 smithi155 podman[72471]: 2024-03-19 03:04:57.542403073 +0000 UTC m=+0.018607197 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:04:57.978 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 19 03:04:57 smithi155 podman[72471]: 2024-03-19 03:04:57.877704452 +0000 UTC m=+0.353908580 container init b25a80f4af5fd71f77eb61a8de4bf43b9626a5be06293201e1491e51cc8236af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-19T03:04:57.978 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 19 03:04:57 smithi155 podman[72471]: 2024-03-19 03:04:57.88127546 +0000 UTC m=+0.357479573 container start b25a80f4af5fd71f77eb61a8de4bf43b9626a5be06293201e1491e51cc8236af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T03:04:57.978 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 19 03:04:57 smithi155 podman[72471]: 2024-03-19 03:04:57.977274996 +0000 UTC m=+0.453479114 container attach b25a80f4af5fd71f77eb61a8de4bf43b9626a5be06293201e1491e51cc8236af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T03:04:58.267 INFO:journalctl@ceph.osd.2.smithi155.stdout:Mar 19 03:04:58 smithi155 podman[72471]: 2024-03-19 03:04:58.172019018 +0000 UTC m=+0.648223131 container died b25a80f4af5fd71f77eb61a8de4bf43b9626a5be06293201e1491e51cc8236af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-2-deactivate, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True) 2024-03-19T03:04:58.866 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.2.service' 2024-03-19T03:04:58.909 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:04:58.909 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-19T03:04:58.909 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-19T03:04:58.909 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.3 2024-03-19T03:04:59.236 INFO:journalctl@ceph.osd.3.smithi033.stdout:Mar 19 03:04:58 smithi033 systemd[1]: Stopping Ceph osd.3 for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T03:04:59.529 INFO:journalctl@ceph.osd.3.smithi033.stdout:Mar 19 03:04:59 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-3[53426]: 2024-03-19T03:04:59.234+0000 7f4429cf4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:04:59.530 INFO:journalctl@ceph.osd.3.smithi033.stdout:Mar 19 03:04:59 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-3[53426]: 2024-03-19T03:04:59.234+0000 7f4429cf4700 -1 osd.3 112 *** Got signal Terminated *** 2024-03-19T03:04:59.530 INFO:journalctl@ceph.osd.3.smithi033.stdout:Mar 19 03:04:59 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-3[53426]: 2024-03-19T03:04:59.234+0000 7f4429cf4700 -1 osd.3 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T03:05:01.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:01 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:01.350+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:02.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:01 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:01.981+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:02.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:02 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:02.392+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:03.344 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:03 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:03.023+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:03.767 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:03 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T03:05:03.342+0000 7f09d6b84700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:40.061449+0000 front 2024-03-19T03:04:40.061565+0000 (oldest deadline 2024-03-19T03:05:02.361449+0000) 2024-03-19T03:05:03.778 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:03 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:03.439+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:04.266 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:04 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:04.005+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:04.766 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:04 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T03:05:04.298+0000 7f09d6b84700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:40.061449+0000 front 2024-03-19T03:04:40.061565+0000 (oldest deadline 2024-03-19T03:05:02.361449+0000) 2024-03-19T03:05:04.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:04 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:04.484+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:04.780 INFO:journalctl@ceph.osd.3.smithi033.stdout:Mar 19 03:05:04 smithi033 podman[101733]: 2024-03-19 03:05:04.666309887 +0000 UTC m=+5.553746013 container died d6d277fb8018c76de554f7607598dc2115f9ef087ff673ba4ceb3c196f0fef06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-3, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-19T03:05:05.279 INFO:journalctl@ceph.osd.3.smithi033.stdout:Mar 19 03:05:05 smithi033 podman[101733]: 2024-03-19 03:05:05.034743318 +0000 UTC m=+5.922179443 container cleanup d6d277fb8018c76de554f7607598dc2115f9ef087ff673ba4ceb3c196f0fef06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-3, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-19T03:05:05.279 INFO:journalctl@ceph.osd.3.smithi033.stdout:Mar 19 03:05:05 smithi033 bash[101733]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-3 2024-03-19T03:05:05.291 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:05 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:05.030+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:05.662 INFO:journalctl@ceph.osd.3.smithi033.stdout:Mar 19 03:05:05 smithi033 podman[101750]: 2024-03-19 03:05:05.319491782 +0000 UTC m=+0.647696926 container remove d6d277fb8018c76de554f7607598dc2115f9ef087ff673ba4ceb3c196f0fef06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-3, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-19T03:05:05.662 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:05 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:05.472+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:05.766 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:05 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T03:05:05.290+0000 7f09d6b84700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:40.061449+0000 front 2024-03-19T03:04:40.061565+0000 (oldest deadline 2024-03-19T03:05:02.361449+0000) 2024-03-19T03:05:05.949 INFO:journalctl@ceph.osd.3.smithi033.stdout:Mar 19 03:05:05 smithi033 podman[101838]: 2024-03-19 03:05:05.564288682 +0000 UTC m=+0.017984385 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:05:05.950 INFO:journalctl@ceph.osd.3.smithi033.stdout:Mar 19 03:05:05 smithi033 podman[101838]: 2024-03-19 03:05:05.678581819 +0000 UTC m=+0.132277503 container create 25e78608406262d60a3dc90b93b5e0c2096d8f6033873a018cead29c12c4d4fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-19T03:05:05.950 INFO:journalctl@ceph.osd.3.smithi033.stdout:Mar 19 03:05:05 smithi033 podman[101838]: 2024-03-19 03:05:05.948650274 +0000 UTC m=+0.402345967 container init 25e78608406262d60a3dc90b93b5e0c2096d8f6033873a018cead29c12c4d4fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T03:05:06.216 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:05 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:05.949+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:06.246 INFO:journalctl@ceph.osd.3.smithi033.stdout:Mar 19 03:05:05 smithi033 podman[101838]: 2024-03-19 03:05:05.952555919 +0000 UTC m=+0.406251607 container start 25e78608406262d60a3dc90b93b5e0c2096d8f6033873a018cead29c12c4d4fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-19T03:05:06.246 INFO:journalctl@ceph.osd.3.smithi033.stdout:Mar 19 03:05:06 smithi033 podman[101838]: 2024-03-19 03:05:06.065277097 +0000 UTC m=+0.518972784 container attach 25e78608406262d60a3dc90b93b5e0c2096d8f6033873a018cead29c12c4d4fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5) 2024-03-19T03:05:06.517 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:06 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T03:05:06.243+0000 7f09d6b84700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:40.061449+0000 front 2024-03-19T03:04:40.061565+0000 (oldest deadline 2024-03-19T03:05:02.361449+0000) 2024-03-19T03:05:06.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:06 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:06.078+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:06.526 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:06 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:06.437+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:06.527 INFO:journalctl@ceph.osd.3.smithi033.stdout:Mar 19 03:05:06 smithi033 podman[101838]: 2024-03-19 03:05:06.245598722 +0000 UTC m=+0.699294415 container died 25e78608406262d60a3dc90b93b5e0c2096d8f6033873a018cead29c12c4d4fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-19T03:05:06.849 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.3.service' 2024-03-19T03:05:06.890 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:05:06.891 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-19T03:05:06.891 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-19T03:05:06.891 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.4 2024-03-19T03:05:07.216 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:06 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:06.943+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:07.216 INFO:journalctl@ceph.osd.4.smithi096.stdout:Mar 19 03:05:06 smithi096 systemd[1]: Stopping Ceph osd.4 for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T03:05:07.517 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:07 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T03:05:07.271+0000 7f09d6b84700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:40.061449+0000 front 2024-03-19T03:04:40.061565+0000 (oldest deadline 2024-03-19T03:05:02.361449+0000) 2024-03-19T03:05:07.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:07 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:07.083+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:07.716 INFO:journalctl@ceph.osd.4.smithi096.stdout:Mar 19 03:05:07 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-4[47074]: 2024-03-19T03:05:07.215+0000 7f7becf96700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:05:07.716 INFO:journalctl@ceph.osd.4.smithi096.stdout:Mar 19 03:05:07 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-4[47074]: 2024-03-19T03:05:07.215+0000 7f7becf96700 -1 osd.4 112 *** Got signal Terminated *** 2024-03-19T03:05:07.716 INFO:journalctl@ceph.osd.4.smithi096.stdout:Mar 19 03:05:07 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-4[47074]: 2024-03-19T03:05:07.215+0000 7f7becf96700 -1 osd.4 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T03:05:07.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:07 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:07.429+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:08.216 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:07 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:07.941+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:08.517 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:08 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T03:05:08.266+0000 7f09d6b84700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:40.061449+0000 front 2024-03-19T03:04:40.061565+0000 (oldest deadline 2024-03-19T03:05:02.361449+0000) 2024-03-19T03:05:08.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:08 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:08.094+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:08.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:08 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:08.430+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:09.216 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:08 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:08.940+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:09.460 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:09 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T03:05:09.240+0000 7f09d6b84700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:40.061449+0000 front 2024-03-19T03:04:40.061565+0000 (oldest deadline 2024-03-19T03:05:02.361449+0000) 2024-03-19T03:05:09.460 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:09 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:09.081+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:09.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:09 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:09.416+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:09.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:09 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:09.416+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:10.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:09 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:09.980+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:10.517 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:10 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T03:05:10.207+0000 7f09d6b84700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:40.061449+0000 front 2024-03-19T03:04:40.061565+0000 (oldest deadline 2024-03-19T03:05:02.361449+0000) 2024-03-19T03:05:10.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:10 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:10.036+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:10.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:10 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:10.036+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:10.680 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:10 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:10.378+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:10.680 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:10 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:10.378+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:11.216 INFO:journalctl@ceph.osd.4.smithi096.stdout:Mar 19 03:05:11 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-4[47074]: 2024-03-19T03:05:11.002+0000 7f7be958a700 -1 osd.4 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.540615+0000 front 2024-03-19T03:04:47.540587+0000 (oldest deadline 2024-03-19T03:05:10.440730+0000) 2024-03-19T03:05:11.216 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:10 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:10.963+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:11.216 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:10 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:10.950+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:44.221520+0000 front 2024-03-19T03:04:44.221546+0000 (oldest deadline 2024-03-19T03:05:10.121657+0000) 2024-03-19T03:05:11.267 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:11 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T03:05:11.242+0000 7f09d6b84700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:40.061449+0000 front 2024-03-19T03:04:40.061565+0000 (oldest deadline 2024-03-19T03:05:02.361449+0000) 2024-03-19T03:05:11.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:11 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:11.003+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:11.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:11 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:11.003+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:11.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:11 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:11.361+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:11.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:11 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:11.361+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:12.216 INFO:journalctl@ceph.osd.4.smithi096.stdout:Mar 19 03:05:11 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-4[47074]: 2024-03-19T03:05:11.993+0000 7f7be958a700 -1 osd.4 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.540615+0000 front 2024-03-19T03:04:47.540587+0000 (oldest deadline 2024-03-19T03:05:10.440730+0000) 2024-03-19T03:05:12.216 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:11 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:11.961+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:44.221520+0000 front 2024-03-19T03:04:44.221546+0000 (oldest deadline 2024-03-19T03:05:10.121657+0000) 2024-03-19T03:05:12.217 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:11 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:11.987+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:12.279 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:11 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6[60108]: 2024-03-19T03:05:11.972+0000 7f163d4a0700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.628182+0000 front 2024-03-19T03:04:46.628266+0000 (oldest deadline 2024-03-19T03:05:11.829002+0000) 2024-03-19T03:05:12.517 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:12 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T03:05:12.216+0000 7f09d6b84700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:40.061449+0000 front 2024-03-19T03:04:40.061565+0000 (oldest deadline 2024-03-19T03:05:02.361449+0000) 2024-03-19T03:05:12.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:12 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:12.028+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:12.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:12 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:12.028+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:12.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:12 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:12.377+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:12.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:12 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:12.377+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:12.966 INFO:journalctl@ceph.osd.4.smithi096.stdout:Mar 19 03:05:12 smithi096 podman[71339]: 2024-03-19 03:05:12.697790465 +0000 UTC m=+5.610802358 container died 51f69ec22f02bd41dc268f0cccedffd940e9a3fe958ac13cb15d8c8c0825d1c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5) 2024-03-19T03:05:13.267 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:13 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T03:05:13.194+0000 7f09d6b84700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:40.061449+0000 front 2024-03-19T03:04:40.061565+0000 (oldest deadline 2024-03-19T03:05:02.361449+0000) 2024-03-19T03:05:13.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:13 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:13.007+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:13.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:13 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:13.007+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:13.278 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:12 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6[60108]: 2024-03-19T03:05:12.996+0000 7f163d4a0700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.628182+0000 front 2024-03-19T03:04:46.628266+0000 (oldest deadline 2024-03-19T03:05:11.829002+0000) 2024-03-19T03:05:13.377 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:12 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:12.994+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:44.221520+0000 front 2024-03-19T03:04:44.221546+0000 (oldest deadline 2024-03-19T03:05:10.121657+0000) 2024-03-19T03:05:13.377 INFO:journalctl@ceph.osd.4.smithi096.stdout:Mar 19 03:05:13 smithi096 podman[71339]: 2024-03-19 03:05:13.099688098 +0000 UTC m=+6.012699989 container cleanup 51f69ec22f02bd41dc268f0cccedffd940e9a3fe958ac13cb15d8c8c0825d1c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-4, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T03:05:13.377 INFO:journalctl@ceph.osd.4.smithi096.stdout:Mar 19 03:05:13 smithi096 bash[71339]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-4 2024-03-19T03:05:13.377 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:12 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:12.990+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:13.378 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:12 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:12.990+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:13.710 INFO:journalctl@ceph.osd.4.smithi096.stdout:Mar 19 03:05:13 smithi096 podman[71358]: 2024-03-19 03:05:13.376132122 +0000 UTC m=+0.680918437 container remove 51f69ec22f02bd41dc268f0cccedffd940e9a3fe958ac13cb15d8c8c0825d1c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-4, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T03:05:13.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:13 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:13.332+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:13.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:13 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:13.332+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:13.966 INFO:journalctl@ceph.osd.4.smithi096.stdout:Mar 19 03:05:13 smithi096 podman[71446]: 2024-03-19 03:05:13.611877727 +0000 UTC m=+0.017098635 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:05:13.966 INFO:journalctl@ceph.osd.4.smithi096.stdout:Mar 19 03:05:13 smithi096 podman[71446]: 2024-03-19 03:05:13.710320617 +0000 UTC m=+0.115541570 container create 321ddb31e7253e980fdca8a4ef62ce3e7adfed7b65460640bc0a15b7dbff1e73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-19T03:05:13.966 INFO:journalctl@ceph.osd.4.smithi096.stdout:Mar 19 03:05:13 smithi096 podman[71446]: 2024-03-19 03:05:13.921596534 +0000 UTC m=+0.326817457 container init 321ddb31e7253e980fdca8a4ef62ce3e7adfed7b65460640bc0a15b7dbff1e73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD) 2024-03-19T03:05:13.966 INFO:journalctl@ceph.osd.4.smithi096.stdout:Mar 19 03:05:13 smithi096 podman[71446]: 2024-03-19 03:05:13.924844787 +0000 UTC m=+0.330065695 container start 321ddb31e7253e980fdca8a4ef62ce3e7adfed7b65460640bc0a15b7dbff1e73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-4-deactivate, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T03:05:14.059 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:13 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:13.789+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:42.502712+0000 front 2024-03-19T03:04:42.502587+0000 (oldest deadline 2024-03-19T03:05:13.703191+0000) 2024-03-19T03:05:14.227 INFO:journalctl@ceph.osd.4.smithi096.stdout:Mar 19 03:05:14 smithi096 podman[71446]: 2024-03-19 03:05:14.004591036 +0000 UTC m=+0.409811956 container attach 321ddb31e7253e980fdca8a4ef62ce3e7adfed7b65460640bc0a15b7dbff1e73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-19T03:05:14.228 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:13 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:13.972+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:14.228 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:13 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:13.972+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:14.228 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:14 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:14.030+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:44.221520+0000 front 2024-03-19T03:04:44.221546+0000 (oldest deadline 2024-03-19T03:05:10.121657+0000) 2024-03-19T03:05:14.279 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:13 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6[60108]: 2024-03-19T03:05:13.954+0000 7f163d4a0700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.628182+0000 front 2024-03-19T03:04:46.628266+0000 (oldest deadline 2024-03-19T03:05:11.829002+0000) 2024-03-19T03:05:14.517 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:14 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T03:05:14.234+0000 7f09d6b84700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:40.061449+0000 front 2024-03-19T03:04:40.061565+0000 (oldest deadline 2024-03-19T03:05:02.361449+0000) 2024-03-19T03:05:14.517 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:14 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T03:05:14.234+0000 7f09d6b84700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.662638+0000 front 2024-03-19T03:04:47.662715+0000 (oldest deadline 2024-03-19T03:05:13.562615+0000) 2024-03-19T03:05:14.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:14 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:14.058+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:14.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:14 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:14.058+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:14.608 INFO:journalctl@ceph.osd.4.smithi096.stdout:Mar 19 03:05:14 smithi096 podman[71446]: 2024-03-19 03:05:14.227094141 +0000 UTC m=+0.632315056 container died 321ddb31e7253e980fdca8a4ef62ce3e7adfed7b65460640bc0a15b7dbff1e73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-19T03:05:14.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:14 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:14.302+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:14.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:14 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:14.302+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:14.953 DEBUG:teuthology.orchestra.run.smithi096:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.4.service' 2024-03-19T03:05:14.995 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:05:14.996 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-19T03:05:14.996 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-19T03:05:14.996 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.5 2024-03-19T03:05:15.097 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:15 smithi155 systemd[1]: Stopping Ceph osd.5 for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T03:05:15.097 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:14 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:14.791+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:42.502712+0000 front 2024-03-19T03:04:42.502587+0000 (oldest deadline 2024-03-19T03:05:13.703191+0000) 2024-03-19T03:05:15.216 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:15 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:15.025+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:44.221520+0000 front 2024-03-19T03:04:44.221546+0000 (oldest deadline 2024-03-19T03:05:10.121657+0000) 2024-03-19T03:05:15.216 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:14 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:14.960+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:15.216 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:14 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:14.960+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:15.279 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:14 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6[60108]: 2024-03-19T03:05:14.980+0000 7f163d4a0700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.628182+0000 front 2024-03-19T03:04:46.628266+0000 (oldest deadline 2024-03-19T03:05:11.829002+0000) 2024-03-19T03:05:15.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:15 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:15.095+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:15.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:15 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:15.095+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:15.517 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:15 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T03:05:15.195+0000 7f09d6b84700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:40.061449+0000 front 2024-03-19T03:04:40.061565+0000 (oldest deadline 2024-03-19T03:05:02.361449+0000) 2024-03-19T03:05:15.517 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:15 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T03:05:15.195+0000 7f09d6b84700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.662638+0000 front 2024-03-19T03:04:47.662715+0000 (oldest deadline 2024-03-19T03:05:13.562615+0000) 2024-03-19T03:05:15.517 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:15 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T03:05:15.309+0000 7f09da590700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:05:15.517 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:15 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T03:05:15.309+0000 7f09da590700 -1 osd.5 112 *** Got signal Terminated *** 2024-03-19T03:05:15.517 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:15 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T03:05:15.309+0000 7f09da590700 -1 osd.5 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T03:05:15.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:15 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:15.329+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:15.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:15 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:15.329+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:16.055 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:15 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:15.773+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:42.502712+0000 front 2024-03-19T03:04:42.502587+0000 (oldest deadline 2024-03-19T03:05:13.703191+0000) 2024-03-19T03:05:16.216 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:16 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:16.022+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:44.221520+0000 front 2024-03-19T03:04:44.221546+0000 (oldest deadline 2024-03-19T03:05:10.121657+0000) 2024-03-19T03:05:16.216 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:15 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:15.916+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:16.216 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:15 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:15.916+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:16.279 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:15 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6[60108]: 2024-03-19T03:05:15.983+0000 7f163d4a0700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.628182+0000 front 2024-03-19T03:04:46.628266+0000 (oldest deadline 2024-03-19T03:05:11.829002+0000) 2024-03-19T03:05:16.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:16 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:16.053+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:16.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:16 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:16.053+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:16.517 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:16 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T03:05:16.242+0000 7f09d6b84700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:40.061449+0000 front 2024-03-19T03:04:40.061565+0000 (oldest deadline 2024-03-19T03:05:02.361449+0000) 2024-03-19T03:05:16.518 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:16 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T03:05:16.242+0000 7f09d6b84700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.662638+0000 front 2024-03-19T03:04:47.662715+0000 (oldest deadline 2024-03-19T03:05:13.562615+0000) 2024-03-19T03:05:16.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:16 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:16.316+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:16.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:16 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:16.316+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:17.075 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:16 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:16.815+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:42.502712+0000 front 2024-03-19T03:04:42.502587+0000 (oldest deadline 2024-03-19T03:05:13.703191+0000) 2024-03-19T03:05:17.216 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:17 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:17.049+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:44.221520+0000 front 2024-03-19T03:04:44.221546+0000 (oldest deadline 2024-03-19T03:05:10.121657+0000) 2024-03-19T03:05:17.216 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:16 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:16.903+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:17.216 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:16 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:16.903+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:17.279 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:16 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6[60108]: 2024-03-19T03:05:16.981+0000 7f163d4a0700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.628182+0000 front 2024-03-19T03:04:46.628266+0000 (oldest deadline 2024-03-19T03:05:11.829002+0000) 2024-03-19T03:05:17.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:17 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:17.074+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:17.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:17 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:17.074+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:17.517 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:17 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T03:05:17.197+0000 7f09d6b84700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:40.061449+0000 front 2024-03-19T03:04:40.061565+0000 (oldest deadline 2024-03-19T03:05:02.361449+0000) 2024-03-19T03:05:17.518 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:17 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T03:05:17.197+0000 7f09d6b84700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.662638+0000 front 2024-03-19T03:04:47.662715+0000 (oldest deadline 2024-03-19T03:05:13.562615+0000) 2024-03-19T03:05:17.518 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:17 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T03:05:17.197+0000 7f09d6b84700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:53.562994+0000 front 2024-03-19T03:04:53.562966+0000 (oldest deadline 2024-03-19T03:05:17.063186+0000) 2024-03-19T03:05:17.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:17 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:17.363+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:17.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:17 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:17.363+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:17.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:17 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:17.363+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:18.200 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:17 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:17.833+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:42.502712+0000 front 2024-03-19T03:04:42.502587+0000 (oldest deadline 2024-03-19T03:05:13.703191+0000) 2024-03-19T03:05:18.200 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:18 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:18.070+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:18.200 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:18 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:18.070+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:18.216 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:18 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:18.062+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:44.221520+0000 front 2024-03-19T03:04:44.221546+0000 (oldest deadline 2024-03-19T03:05:10.121657+0000) 2024-03-19T03:05:18.216 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:18 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:18.062+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.422503+0000 front 2024-03-19T03:04:52.422481+0000 (oldest deadline 2024-03-19T03:05:17.122606+0000) 2024-03-19T03:05:18.216 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:17 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:17.924+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:18.216 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:17 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:17.924+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:18.279 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:17 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6[60108]: 2024-03-19T03:05:17.973+0000 7f163d4a0700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.628182+0000 front 2024-03-19T03:04:46.628266+0000 (oldest deadline 2024-03-19T03:05:11.829002+0000) 2024-03-19T03:05:18.517 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:18 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T03:05:18.198+0000 7f09d6b84700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:40.061449+0000 front 2024-03-19T03:04:40.061565+0000 (oldest deadline 2024-03-19T03:05:02.361449+0000) 2024-03-19T03:05:18.517 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:18 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T03:05:18.198+0000 7f09d6b84700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.662638+0000 front 2024-03-19T03:04:47.662715+0000 (oldest deadline 2024-03-19T03:05:13.562615+0000) 2024-03-19T03:05:18.517 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:18 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T03:05:18.198+0000 7f09d6b84700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:53.562994+0000 front 2024-03-19T03:04:53.562966+0000 (oldest deadline 2024-03-19T03:05:17.063186+0000) 2024-03-19T03:05:18.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:18 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:18.351+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:18.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:18 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:18.351+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:18.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:18 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:18.351+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:19.165 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:18 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:18.877+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:42.502712+0000 front 2024-03-19T03:04:42.502587+0000 (oldest deadline 2024-03-19T03:05:13.703191+0000) 2024-03-19T03:05:19.165 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:18 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:18.877+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:53.703569+0000 front 2024-03-19T03:04:53.703647+0000 (oldest deadline 2024-03-19T03:05:18.403703+0000) 2024-03-19T03:05:19.165 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:19 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:19.029+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:19.165 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:19 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:19.029+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:19.165 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:19 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:19.029+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:19.216 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:19 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:19.070+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:44.221520+0000 front 2024-03-19T03:04:44.221546+0000 (oldest deadline 2024-03-19T03:05:10.121657+0000) 2024-03-19T03:05:19.216 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:19 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:19.070+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.422503+0000 front 2024-03-19T03:04:52.422481+0000 (oldest deadline 2024-03-19T03:05:17.122606+0000) 2024-03-19T03:05:19.216 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:18 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:18.934+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:19.216 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:18 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:18.934+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:19.279 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:19 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6[60108]: 2024-03-19T03:05:19.005+0000 7f163d4a0700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.628182+0000 front 2024-03-19T03:04:46.628266+0000 (oldest deadline 2024-03-19T03:05:11.829002+0000) 2024-03-19T03:05:19.518 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:19 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T03:05:19.164+0000 7f09d6b84700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:40.061449+0000 front 2024-03-19T03:04:40.061565+0000 (oldest deadline 2024-03-19T03:05:02.361449+0000) 2024-03-19T03:05:19.518 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:19 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T03:05:19.164+0000 7f09d6b84700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.662638+0000 front 2024-03-19T03:04:47.662715+0000 (oldest deadline 2024-03-19T03:05:13.562615+0000) 2024-03-19T03:05:19.518 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:19 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T03:05:19.164+0000 7f09d6b84700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:53.562994+0000 front 2024-03-19T03:04:53.562966+0000 (oldest deadline 2024-03-19T03:05:17.063186+0000) 2024-03-19T03:05:19.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:19 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:19.392+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:19.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:19 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:19.392+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:19.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:19 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:19.392+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:20.206 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:19 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:19.864+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:42.502712+0000 front 2024-03-19T03:04:42.502587+0000 (oldest deadline 2024-03-19T03:05:13.703191+0000) 2024-03-19T03:05:20.206 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:19 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:19.864+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:53.703569+0000 front 2024-03-19T03:04:53.703647+0000 (oldest deadline 2024-03-19T03:05:18.403703+0000) 2024-03-19T03:05:20.206 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:20 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:20.015+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:20.206 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:20 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:20.015+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:20.207 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:20 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:20.015+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:20.216 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:20 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:20.117+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:44.221520+0000 front 2024-03-19T03:04:44.221546+0000 (oldest deadline 2024-03-19T03:05:10.121657+0000) 2024-03-19T03:05:20.216 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:20 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:20.117+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.422503+0000 front 2024-03-19T03:04:52.422481+0000 (oldest deadline 2024-03-19T03:05:17.122606+0000) 2024-03-19T03:05:20.216 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:19 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:19.934+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:20.216 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:19 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:19.934+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:20.424 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:20 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6[60108]: 2024-03-19T03:05:20.028+0000 7f163d4a0700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.628182+0000 front 2024-03-19T03:04:46.628266+0000 (oldest deadline 2024-03-19T03:05:11.829002+0000) 2024-03-19T03:05:20.517 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:20 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T03:05:20.204+0000 7f09d6b84700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:40.061449+0000 front 2024-03-19T03:04:40.061565+0000 (oldest deadline 2024-03-19T03:05:02.361449+0000) 2024-03-19T03:05:20.517 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:20 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T03:05:20.204+0000 7f09d6b84700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.662638+0000 front 2024-03-19T03:04:47.662715+0000 (oldest deadline 2024-03-19T03:05:13.562615+0000) 2024-03-19T03:05:20.517 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:20 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5[47575]: 2024-03-19T03:05:20.204+0000 7f09d6b84700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:53.562994+0000 front 2024-03-19T03:04:53.562966+0000 (oldest deadline 2024-03-19T03:05:17.063186+0000) 2024-03-19T03:05:20.733 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:20 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:20.423+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:20.733 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:20 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:20.423+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:20.734 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:20 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:20.423+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:21.017 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:20 smithi155 podman[72593]: 2024-03-19 03:05:20.759249248 +0000 UTC m=+5.573663702 container died a99921a4cecfcc58cb68db0967821c6421024ef06ba923c2af5397ce23bcf8d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-19T03:05:21.017 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:20 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:20.839+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:42.502712+0000 front 2024-03-19T03:04:42.502587+0000 (oldest deadline 2024-03-19T03:05:13.703191+0000) 2024-03-19T03:05:21.017 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:20 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:20.839+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:53.703569+0000 front 2024-03-19T03:04:53.703647+0000 (oldest deadline 2024-03-19T03:05:18.403703+0000) 2024-03-19T03:05:21.399 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:21 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6[60108]: 2024-03-19T03:05:21.055+0000 7f163d4a0700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.628182+0000 front 2024-03-19T03:04:46.628266+0000 (oldest deadline 2024-03-19T03:05:11.829002+0000) 2024-03-19T03:05:21.399 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:21 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6[60108]: 2024-03-19T03:05:21.055+0000 7f163d4a0700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.330201+0000 front 2024-03-19T03:04:55.330395+0000 (oldest deadline 2024-03-19T03:05:20.030348+0000) 2024-03-19T03:05:21.412 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:21 smithi155 podman[72593]: 2024-03-19 03:05:21.143607626 +0000 UTC m=+5.958022098 container cleanup a99921a4cecfcc58cb68db0967821c6421024ef06ba923c2af5397ce23bcf8d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T03:05:21.412 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:21 smithi155 bash[72593]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5 2024-03-19T03:05:21.413 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:21 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:21.018+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:21.413 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:21 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:21.018+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:21.413 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:21 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:21.018+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:21.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:21 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:21.075+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:44.221520+0000 front 2024-03-19T03:04:44.221546+0000 (oldest deadline 2024-03-19T03:05:10.121657+0000) 2024-03-19T03:05:21.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:21 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:21.075+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.422503+0000 front 2024-03-19T03:04:52.422481+0000 (oldest deadline 2024-03-19T03:05:17.122606+0000) 2024-03-19T03:05:21.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:20 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:20.978+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:21.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:20 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:20.979+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:21.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:20 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:20.979+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:21.754 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:21 smithi155 podman[72610]: 2024-03-19 03:05:21.411670358 +0000 UTC m=+0.652667484 container remove a99921a4cecfcc58cb68db0967821c6421024ef06ba923c2af5397ce23bcf8d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-19T03:05:21.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:21 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:21.397+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:21.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:21 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:21.397+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:21.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:21 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:21.397+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:22.017 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:21 smithi155 podman[72703]: 2024-03-19 03:05:21.657849548 +0000 UTC m=+0.018923316 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:05:22.017 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:21 smithi155 podman[72703]: 2024-03-19 03:05:21.795876572 +0000 UTC m=+0.156950326 container create 08556ee52e97fccd3a8834c99b0eb3626125ba293cf43ae4094ce2f7834c9b79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-19T03:05:22.017 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:21 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:21.846+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:42.502712+0000 front 2024-03-19T03:04:42.502587+0000 (oldest deadline 2024-03-19T03:05:13.703191+0000) 2024-03-19T03:05:22.017 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:21 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:21.846+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:53.703569+0000 front 2024-03-19T03:04:53.703647+0000 (oldest deadline 2024-03-19T03:05:18.403703+0000) 2024-03-19T03:05:22.018 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:21 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:21.987+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:22.018 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:21 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:21.987+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:22.018 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:21 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:21.988+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:22.279 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:22 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6[60108]: 2024-03-19T03:05:22.022+0000 7f163d4a0700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.628182+0000 front 2024-03-19T03:04:46.628266+0000 (oldest deadline 2024-03-19T03:05:11.829002+0000) 2024-03-19T03:05:22.279 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:22 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6[60108]: 2024-03-19T03:05:22.022+0000 7f163d4a0700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.330201+0000 front 2024-03-19T03:04:55.330395+0000 (oldest deadline 2024-03-19T03:05:20.030348+0000) 2024-03-19T03:05:22.360 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:22 smithi155 podman[72703]: 2024-03-19 03:05:22.065176534 +0000 UTC m=+0.426250300 container init 08556ee52e97fccd3a8834c99b0eb3626125ba293cf43ae4094ce2f7834c9b79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T03:05:22.360 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:22 smithi155 podman[72703]: 2024-03-19 03:05:22.068239551 +0000 UTC m=+0.429313316 container start 08556ee52e97fccd3a8834c99b0eb3626125ba293cf43ae4094ce2f7834c9b79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T03:05:22.360 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:22 smithi155 podman[72703]: 2024-03-19 03:05:22.140238322 +0000 UTC m=+0.501312093 container attach 08556ee52e97fccd3a8834c99b0eb3626125ba293cf43ae4094ce2f7834c9b79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5) 2024-03-19T03:05:22.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:22 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:22.071+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:44.221520+0000 front 2024-03-19T03:04:44.221546+0000 (oldest deadline 2024-03-19T03:05:10.121657+0000) 2024-03-19T03:05:22.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:22 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:22.071+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.422503+0000 front 2024-03-19T03:04:52.422481+0000 (oldest deadline 2024-03-19T03:05:17.122606+0000) 2024-03-19T03:05:22.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:22 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:22.020+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:22.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:22 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:22.020+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:22.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:22 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:22.020+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:22.711 INFO:journalctl@ceph.osd.5.smithi155.stdout:Mar 19 03:05:22 smithi155 podman[72703]: 2024-03-19 03:05:22.359635601 +0000 UTC m=+0.720709366 container died 08556ee52e97fccd3a8834c99b0eb3626125ba293cf43ae4094ce2f7834c9b79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-19T03:05:22.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:22 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:22.385+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:22.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:22 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:22.385+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:22.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:22 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:22.385+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:22.994 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:22 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:22.835+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:42.502712+0000 front 2024-03-19T03:04:42.502587+0000 (oldest deadline 2024-03-19T03:05:13.703191+0000) 2024-03-19T03:05:22.994 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:22 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:22.835+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:53.703569+0000 front 2024-03-19T03:04:53.703647+0000 (oldest deadline 2024-03-19T03:05:18.403703+0000) 2024-03-19T03:05:23.059 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.5.service' 2024-03-19T03:05:23.100 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:05:23.101 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-19T03:05:23.101 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-19T03:05:23.101 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.6 2024-03-19T03:05:23.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:22 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:22.993+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:23.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:22 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:22.993+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:23.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:22 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:22.993+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:23.278 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:23 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6[60108]: 2024-03-19T03:05:23.009+0000 7f163d4a0700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.628182+0000 front 2024-03-19T03:04:46.628266+0000 (oldest deadline 2024-03-19T03:05:11.829002+0000) 2024-03-19T03:05:23.279 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:23 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6[60108]: 2024-03-19T03:05:23.009+0000 7f163d4a0700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.330201+0000 front 2024-03-19T03:04:55.330395+0000 (oldest deadline 2024-03-19T03:05:20.030348+0000) 2024-03-19T03:05:23.279 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:23 smithi033 systemd[1]: Stopping Ceph osd.6 for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T03:05:23.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:23 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:23.050+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:44.221520+0000 front 2024-03-19T03:04:44.221546+0000 (oldest deadline 2024-03-19T03:05:10.121657+0000) 2024-03-19T03:05:23.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:23 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:23.050+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.422503+0000 front 2024-03-19T03:04:52.422481+0000 (oldest deadline 2024-03-19T03:05:17.122606+0000) 2024-03-19T03:05:23.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:23 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:23.018+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:23.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:23 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:23.018+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:23.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:23 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:23.018+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:23.779 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:23 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6[60108]: 2024-03-19T03:05:23.420+0000 7f1640eac700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:05:23.779 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:23 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6[60108]: 2024-03-19T03:05:23.420+0000 7f1640eac700 -1 osd.6 112 *** Got signal Terminated *** 2024-03-19T03:05:23.779 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:23 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6[60108]: 2024-03-19T03:05:23.420+0000 7f1640eac700 -1 osd.6 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T03:05:23.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:23 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:23.377+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:23.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:23 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:23.377+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:23.780 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:23 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:23.377+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:24.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:23 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:23.815+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:42.502712+0000 front 2024-03-19T03:04:42.502587+0000 (oldest deadline 2024-03-19T03:05:13.703191+0000) 2024-03-19T03:05:24.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:23 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:23.815+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:53.703569+0000 front 2024-03-19T03:04:53.703647+0000 (oldest deadline 2024-03-19T03:05:18.403703+0000) 2024-03-19T03:05:24.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:24 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:24.019+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:24.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:24 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:24.019+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:24.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:24 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:24.019+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:24.279 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:23 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6[60108]: 2024-03-19T03:05:23.969+0000 7f163d4a0700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.628182+0000 front 2024-03-19T03:04:46.628266+0000 (oldest deadline 2024-03-19T03:05:11.829002+0000) 2024-03-19T03:05:24.279 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:23 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6[60108]: 2024-03-19T03:05:23.969+0000 7f163d4a0700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.330201+0000 front 2024-03-19T03:04:55.330395+0000 (oldest deadline 2024-03-19T03:05:20.030348+0000) 2024-03-19T03:05:24.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:24 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:24.073+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:44.221520+0000 front 2024-03-19T03:04:44.221546+0000 (oldest deadline 2024-03-19T03:05:10.121657+0000) 2024-03-19T03:05:24.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:24 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:24.073+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.422503+0000 front 2024-03-19T03:04:52.422481+0000 (oldest deadline 2024-03-19T03:05:17.122606+0000) 2024-03-19T03:05:24.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:23 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:23.982+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:24.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:23 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:23.982+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:24.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:23 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:23.982+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:24.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:24 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:24.368+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:24.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:24 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:24.368+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:24.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:24 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:24.368+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:25.054 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:24 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:24.802+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:42.502712+0000 front 2024-03-19T03:04:42.502587+0000 (oldest deadline 2024-03-19T03:05:13.703191+0000) 2024-03-19T03:05:25.054 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:24 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:24.802+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:53.703569+0000 front 2024-03-19T03:04:53.703647+0000 (oldest deadline 2024-03-19T03:05:18.403703+0000) 2024-03-19T03:05:25.278 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:24 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6[60108]: 2024-03-19T03:05:24.944+0000 7f163d4a0700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.628182+0000 front 2024-03-19T03:04:46.628266+0000 (oldest deadline 2024-03-19T03:05:11.829002+0000) 2024-03-19T03:05:25.279 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:24 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6[60108]: 2024-03-19T03:05:24.944+0000 7f163d4a0700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.330201+0000 front 2024-03-19T03:04:55.330395+0000 (oldest deadline 2024-03-19T03:05:20.030348+0000) 2024-03-19T03:05:25.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:25 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:25.112+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:44.221520+0000 front 2024-03-19T03:04:44.221546+0000 (oldest deadline 2024-03-19T03:05:10.121657+0000) 2024-03-19T03:05:25.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:25 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:25.112+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.422503+0000 front 2024-03-19T03:04:52.422481+0000 (oldest deadline 2024-03-19T03:05:17.122606+0000) 2024-03-19T03:05:25.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:25 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:25.010+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:25.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:25 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:25.010+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:25.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:25 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:25.010+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:25.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:25 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:25.053+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:25.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:25 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:25.053+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:25.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:25 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:25.053+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:25.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:25 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:25.386+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:25.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:25 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:25.386+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:25.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:25 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:25.386+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:26.102 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:25 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:25.839+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:42.502712+0000 front 2024-03-19T03:04:42.502587+0000 (oldest deadline 2024-03-19T03:05:13.703191+0000) 2024-03-19T03:05:26.103 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:25 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:25.839+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:53.703569+0000 front 2024-03-19T03:04:53.703647+0000 (oldest deadline 2024-03-19T03:05:18.403703+0000) 2024-03-19T03:05:26.279 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:25 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6[60108]: 2024-03-19T03:05:25.900+0000 7f163d4a0700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.628182+0000 front 2024-03-19T03:04:46.628266+0000 (oldest deadline 2024-03-19T03:05:11.829002+0000) 2024-03-19T03:05:26.279 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:25 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6[60108]: 2024-03-19T03:05:25.900+0000 7f163d4a0700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.330201+0000 front 2024-03-19T03:04:55.330395+0000 (oldest deadline 2024-03-19T03:05:20.030348+0000) 2024-03-19T03:05:26.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:26 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:26.104+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:44.221520+0000 front 2024-03-19T03:04:44.221546+0000 (oldest deadline 2024-03-19T03:05:10.121657+0000) 2024-03-19T03:05:26.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:26 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:26.104+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.422503+0000 front 2024-03-19T03:04:52.422481+0000 (oldest deadline 2024-03-19T03:05:17.122606+0000) 2024-03-19T03:05:26.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:26 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:26.026+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:26.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:26 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:26.026+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:26.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:26 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:26.026+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:26.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:26 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:26.101+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:26.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:26 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:26.101+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:26.518 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:26 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:26.101+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:26.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:26 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:26.384+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:26.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:26 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:26.384+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:26.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:26 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:26.384+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:27.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:27 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:27.067+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:27.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:27 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:27.067+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:27.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:27 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:27.067+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:27.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:26 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:26.852+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:42.502712+0000 front 2024-03-19T03:04:42.502587+0000 (oldest deadline 2024-03-19T03:05:13.703191+0000) 2024-03-19T03:05:27.268 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:26 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:26.852+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:53.703569+0000 front 2024-03-19T03:04:53.703647+0000 (oldest deadline 2024-03-19T03:05:18.403703+0000) 2024-03-19T03:05:27.268 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:26 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:26.852+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:00.005093+0000 front 2024-03-19T03:05:00.005105+0000 (oldest deadline 2024-03-19T03:05:25.905030+0000) 2024-03-19T03:05:27.278 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:26 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6[60108]: 2024-03-19T03:05:26.870+0000 7f163d4a0700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.628182+0000 front 2024-03-19T03:04:46.628266+0000 (oldest deadline 2024-03-19T03:05:11.829002+0000) 2024-03-19T03:05:27.278 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:26 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6[60108]: 2024-03-19T03:05:26.870+0000 7f163d4a0700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.330201+0000 front 2024-03-19T03:04:55.330395+0000 (oldest deadline 2024-03-19T03:05:20.030348+0000) 2024-03-19T03:05:27.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:27 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:27.107+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:44.221520+0000 front 2024-03-19T03:04:44.221546+0000 (oldest deadline 2024-03-19T03:05:10.121657+0000) 2024-03-19T03:05:27.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:27 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:27.107+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.422503+0000 front 2024-03-19T03:04:52.422481+0000 (oldest deadline 2024-03-19T03:05:17.122606+0000) 2024-03-19T03:05:27.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:27 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:27.046+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:27.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:27 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:27.046+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:27.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:27 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:27.046+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:27.778 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:27 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:27.433+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:27.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:27 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:27.433+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:27.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:27 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:27.433+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:28.118 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:27 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:27.851+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:42.502712+0000 front 2024-03-19T03:04:42.502587+0000 (oldest deadline 2024-03-19T03:05:13.703191+0000) 2024-03-19T03:05:28.118 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:27 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:27.851+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:53.703569+0000 front 2024-03-19T03:04:53.703647+0000 (oldest deadline 2024-03-19T03:05:18.403703+0000) 2024-03-19T03:05:28.118 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:27 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:27.851+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:00.005093+0000 front 2024-03-19T03:05:00.005105+0000 (oldest deadline 2024-03-19T03:05:25.905030+0000) 2024-03-19T03:05:28.279 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:27 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6[60108]: 2024-03-19T03:05:27.865+0000 7f163d4a0700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.628182+0000 front 2024-03-19T03:04:46.628266+0000 (oldest deadline 2024-03-19T03:05:11.829002+0000) 2024-03-19T03:05:28.279 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:27 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6[60108]: 2024-03-19T03:05:27.865+0000 7f163d4a0700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.330201+0000 front 2024-03-19T03:04:55.330395+0000 (oldest deadline 2024-03-19T03:05:20.030348+0000) 2024-03-19T03:05:28.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:28 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:28.074+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:44.221520+0000 front 2024-03-19T03:04:44.221546+0000 (oldest deadline 2024-03-19T03:05:10.121657+0000) 2024-03-19T03:05:28.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:28 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:28.074+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.422503+0000 front 2024-03-19T03:04:52.422481+0000 (oldest deadline 2024-03-19T03:05:17.122606+0000) 2024-03-19T03:05:28.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:28 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:28.074+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.924146+0000 front 2024-03-19T03:05:02.924238+0000 (oldest deadline 2024-03-19T03:05:27.624257+0000) 2024-03-19T03:05:28.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:28 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:28.055+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:28.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:28 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:28.055+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:28.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:28 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:28.055+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:28.488 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:28 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:28.115+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:28.488 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:28 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:28.115+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:28.488 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:28 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:28.115+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:28.771 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:28 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:28.451+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:28.771 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:28 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:28.451+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:28.771 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:28 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:28.451+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:29.029 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:28 smithi033 podman[101955]: 2024-03-19 03:05:28.770583649 +0000 UTC m=+5.475370333 container died 769f7bd71944b10b1cc89b10293f377a592d9b4f24fcf655d74fb11b2d7467ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307) 2024-03-19T03:05:29.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:28 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:28.873+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:42.502712+0000 front 2024-03-19T03:04:42.502587+0000 (oldest deadline 2024-03-19T03:05:13.703191+0000) 2024-03-19T03:05:29.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:28 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:28.873+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:53.703569+0000 front 2024-03-19T03:04:53.703647+0000 (oldest deadline 2024-03-19T03:05:18.403703+0000) 2024-03-19T03:05:29.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:28 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:28.873+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:00.005093+0000 front 2024-03-19T03:05:00.005105+0000 (oldest deadline 2024-03-19T03:05:25.905030+0000) 2024-03-19T03:05:29.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:29 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:29.087+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:29.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:29 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:29.087+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:29.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:29 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:29.087+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:29.406 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:29 smithi033 podman[101955]: 2024-03-19 03:05:29.154166171 +0000 UTC m=+5.858952848 container cleanup 769f7bd71944b10b1cc89b10293f377a592d9b4f24fcf655d74fb11b2d7467ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T03:05:29.406 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:29 smithi033 bash[101955]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6 2024-03-19T03:05:29.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:29 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:29.097+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:44.221520+0000 front 2024-03-19T03:04:44.221546+0000 (oldest deadline 2024-03-19T03:05:10.121657+0000) 2024-03-19T03:05:29.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:29 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:29.097+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.422503+0000 front 2024-03-19T03:04:52.422481+0000 (oldest deadline 2024-03-19T03:05:17.122606+0000) 2024-03-19T03:05:29.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:29 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:29.097+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.924146+0000 front 2024-03-19T03:05:02.924238+0000 (oldest deadline 2024-03-19T03:05:27.624257+0000) 2024-03-19T03:05:29.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:29 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:29.011+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:29.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:29 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:29.011+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:29.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:29 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:29.011+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:29.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:29 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:29.011+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.318466+0000 front 2024-03-19T03:05:02.318512+0000 (oldest deadline 2024-03-19T03:05:28.119022+0000) 2024-03-19T03:05:29.721 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:29 smithi033 podman[101973]: 2024-03-19 03:05:29.405739123 +0000 UTC m=+0.630016251 container remove 769f7bd71944b10b1cc89b10293f377a592d9b4f24fcf655d74fb11b2d7467ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-19T03:05:29.721 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:29 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:29.448+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:29.721 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:29 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:29.448+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:29.721 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:29 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:29.448+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:30.029 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:29 smithi033 podman[102062]: 2024-03-19 03:05:29.623425114 +0000 UTC m=+0.017364879 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:05:30.029 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:29 smithi033 podman[102062]: 2024-03-19 03:05:29.723376915 +0000 UTC m=+0.117316713 container create e37a89b68c79812f165f7a24b8285c07c2cafa14373af1e675f24fd1a41d6cfb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-19T03:05:30.029 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:29 smithi033 podman[102062]: 2024-03-19 03:05:29.959695667 +0000 UTC m=+0.353635445 container init e37a89b68c79812f165f7a24b8285c07c2cafa14373af1e675f24fd1a41d6cfb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-19T03:05:30.029 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:29 smithi033 podman[102062]: 2024-03-19 03:05:29.963042999 +0000 UTC m=+0.356982759 container start e37a89b68c79812f165f7a24b8285c07c2cafa14373af1e675f24fd1a41d6cfb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-19T03:05:30.135 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:29 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:29.850+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:42.502712+0000 front 2024-03-19T03:04:42.502587+0000 (oldest deadline 2024-03-19T03:05:13.703191+0000) 2024-03-19T03:05:30.135 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:29 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:29.850+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:53.703569+0000 front 2024-03-19T03:04:53.703647+0000 (oldest deadline 2024-03-19T03:05:18.403703+0000) 2024-03-19T03:05:30.135 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:29 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:29.850+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:00.005093+0000 front 2024-03-19T03:05:00.005105+0000 (oldest deadline 2024-03-19T03:05:25.905030+0000) 2024-03-19T03:05:30.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:30 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:30.085+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:44.221520+0000 front 2024-03-19T03:04:44.221546+0000 (oldest deadline 2024-03-19T03:05:10.121657+0000) 2024-03-19T03:05:30.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:30 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:30.085+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.422503+0000 front 2024-03-19T03:04:52.422481+0000 (oldest deadline 2024-03-19T03:05:17.122606+0000) 2024-03-19T03:05:30.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:30 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:30.085+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.924146+0000 front 2024-03-19T03:05:02.924238+0000 (oldest deadline 2024-03-19T03:05:27.624257+0000) 2024-03-19T03:05:30.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:30 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:30.045+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:30.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:30 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:30.045+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:30.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:30 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:30.045+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:30.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:30 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:30.045+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.318466+0000 front 2024-03-19T03:05:02.318512+0000 (oldest deadline 2024-03-19T03:05:28.119022+0000) 2024-03-19T03:05:30.473 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:30 smithi033 podman[102062]: 2024-03-19 03:05:30.051013051 +0000 UTC m=+0.444952812 container attach e37a89b68c79812f165f7a24b8285c07c2cafa14373af1e675f24fd1a41d6cfb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T03:05:30.473 INFO:journalctl@ceph.osd.6.smithi033.stdout:Mar 19 03:05:30 smithi033 podman[102062]: 2024-03-19 03:05:30.257731769 +0000 UTC m=+0.651671538 container died e37a89b68c79812f165f7a24b8285c07c2cafa14373af1e675f24fd1a41d6cfb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-19T03:05:30.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:30 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:30.133+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:30.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:30 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:30.133+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:30.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:30 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:30.133+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:30.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:30 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:30.471+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:30.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:30 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:30.471+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:30.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:30 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:30.471+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:30.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:30 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:30.471+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.746043+0000 front 2024-03-19T03:05:02.746074+0000 (oldest deadline 2024-03-19T03:05:30.346883+0000) 2024-03-19T03:05:30.895 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.6.service' 2024-03-19T03:05:30.940 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:05:30.940 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-19T03:05:30.941 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-19T03:05:30.941 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.7 2024-03-19T03:05:31.131 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:30 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:30.825+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:42.502712+0000 front 2024-03-19T03:04:42.502587+0000 (oldest deadline 2024-03-19T03:05:13.703191+0000) 2024-03-19T03:05:31.131 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:30 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:30.825+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:53.703569+0000 front 2024-03-19T03:04:53.703647+0000 (oldest deadline 2024-03-19T03:05:18.403703+0000) 2024-03-19T03:05:31.131 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:30 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:30.825+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:00.005093+0000 front 2024-03-19T03:05:00.005105+0000 (oldest deadline 2024-03-19T03:05:25.905030+0000) 2024-03-19T03:05:31.253 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:31 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:31.032+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:31.253 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:31 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:31.032+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:31.253 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:31 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:31.032+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:31.253 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:31 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:31.032+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.318466+0000 front 2024-03-19T03:05:02.318512+0000 (oldest deadline 2024-03-19T03:05:28.119022+0000) 2024-03-19T03:05:31.253 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:30 smithi096 systemd[1]: Stopping Ceph osd.7 for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T03:05:31.253 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:31 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:31.078+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:44.221520+0000 front 2024-03-19T03:04:44.221546+0000 (oldest deadline 2024-03-19T03:05:10.121657+0000) 2024-03-19T03:05:31.253 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:31 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:31.078+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.422503+0000 front 2024-03-19T03:04:52.422481+0000 (oldest deadline 2024-03-19T03:05:17.122606+0000) 2024-03-19T03:05:31.253 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:31 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:31.078+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.924146+0000 front 2024-03-19T03:05:02.924238+0000 (oldest deadline 2024-03-19T03:05:27.624257+0000) 2024-03-19T03:05:31.254 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:31 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:31.251+0000 7f248f7b1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:05:31.254 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:31 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:31.251+0000 7f248f7b1700 -1 osd.7 112 *** Got signal Terminated *** 2024-03-19T03:05:31.254 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:31 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:31.251+0000 7f248f7b1700 -1 osd.7 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T03:05:31.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:31 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:31.129+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:31.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:31 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:31.129+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:31.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:31 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:31.129+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:31.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:31 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:31.466+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:31.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:31 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:31.466+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:31.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:31 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:31.466+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:31.780 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:31 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:31.466+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.746043+0000 front 2024-03-19T03:05:02.746074+0000 (oldest deadline 2024-03-19T03:05:30.346883+0000) 2024-03-19T03:05:32.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:31 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:31.868+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:42.502712+0000 front 2024-03-19T03:04:42.502587+0000 (oldest deadline 2024-03-19T03:05:13.703191+0000) 2024-03-19T03:05:32.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:31 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:31.869+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:53.703569+0000 front 2024-03-19T03:04:53.703647+0000 (oldest deadline 2024-03-19T03:05:18.403703+0000) 2024-03-19T03:05:32.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:31 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:31.869+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:00.005093+0000 front 2024-03-19T03:05:00.005105+0000 (oldest deadline 2024-03-19T03:05:25.905030+0000) 2024-03-19T03:05:32.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:32 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:32.087+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:32.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:32 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:32.087+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:32.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:32 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:32.087+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:32.457 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:32 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:32.126+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:44.221520+0000 front 2024-03-19T03:04:44.221546+0000 (oldest deadline 2024-03-19T03:05:10.121657+0000) 2024-03-19T03:05:32.457 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:32 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:32.126+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.422503+0000 front 2024-03-19T03:04:52.422481+0000 (oldest deadline 2024-03-19T03:05:17.122606+0000) 2024-03-19T03:05:32.457 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:32 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:32.126+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.924146+0000 front 2024-03-19T03:05:02.924238+0000 (oldest deadline 2024-03-19T03:05:27.624257+0000) 2024-03-19T03:05:32.457 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:32 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:32.048+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:32.458 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:32 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:32.048+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:32.458 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:32 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:32.048+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:32.458 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:32 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:32.048+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.318466+0000 front 2024-03-19T03:05:02.318512+0000 (oldest deadline 2024-03-19T03:05:28.119022+0000) 2024-03-19T03:05:32.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:32 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:32.448+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:32.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:32 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:32.448+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:32.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:32 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:32.448+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:32.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:32 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:32.448+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.746043+0000 front 2024-03-19T03:05:02.746074+0000 (oldest deadline 2024-03-19T03:05:30.346883+0000) 2024-03-19T03:05:33.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:32 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:32.888+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:42.502712+0000 front 2024-03-19T03:04:42.502587+0000 (oldest deadline 2024-03-19T03:05:13.703191+0000) 2024-03-19T03:05:33.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:32 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:32.888+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:53.703569+0000 front 2024-03-19T03:04:53.703647+0000 (oldest deadline 2024-03-19T03:05:18.403703+0000) 2024-03-19T03:05:33.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:32 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:32.888+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:00.005093+0000 front 2024-03-19T03:05:00.005105+0000 (oldest deadline 2024-03-19T03:05:25.905030+0000) 2024-03-19T03:05:33.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:33 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:33.063+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:33.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:33 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:33.063+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:33.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:33 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:33.063+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:33.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:33 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:33.063+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:05:33.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:33 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:33.081+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:44.221520+0000 front 2024-03-19T03:04:44.221546+0000 (oldest deadline 2024-03-19T03:05:10.121657+0000) 2024-03-19T03:05:33.467 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:33 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:33.081+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.422503+0000 front 2024-03-19T03:04:52.422481+0000 (oldest deadline 2024-03-19T03:05:17.122606+0000) 2024-03-19T03:05:33.467 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:33 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:33.081+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.924146+0000 front 2024-03-19T03:05:02.924238+0000 (oldest deadline 2024-03-19T03:05:27.624257+0000) 2024-03-19T03:05:33.468 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:33 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:33.027+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:33.468 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:33 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:33.027+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:33.468 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:33 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:33.027+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:33.468 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:33 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:33.027+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.318466+0000 front 2024-03-19T03:05:02.318512+0000 (oldest deadline 2024-03-19T03:05:28.119022+0000) 2024-03-19T03:05:33.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:33 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:33.427+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:33.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:33 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:33.427+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:33.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:33 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:33.427+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:33.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:33 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:33.427+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.746043+0000 front 2024-03-19T03:05:02.746074+0000 (oldest deadline 2024-03-19T03:05:30.346883+0000) 2024-03-19T03:05:34.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:33 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:33.854+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:42.502712+0000 front 2024-03-19T03:04:42.502587+0000 (oldest deadline 2024-03-19T03:05:13.703191+0000) 2024-03-19T03:05:34.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:33 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:33.854+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:53.703569+0000 front 2024-03-19T03:04:53.703647+0000 (oldest deadline 2024-03-19T03:05:18.403703+0000) 2024-03-19T03:05:34.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:33 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:33.854+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:00.005093+0000 front 2024-03-19T03:05:00.005105+0000 (oldest deadline 2024-03-19T03:05:25.905030+0000) 2024-03-19T03:05:34.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:33 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:33.854+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.806257+0000 front 2024-03-19T03:05:08.806160+0000 (oldest deadline 2024-03-19T03:05:32.906260+0000) 2024-03-19T03:05:34.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:34 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:34.068+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:34.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:34 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:34.068+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:34.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:34 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:34.068+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:34.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:34 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:34.068+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:05:34.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:34 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:34.100+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:44.221520+0000 front 2024-03-19T03:04:44.221546+0000 (oldest deadline 2024-03-19T03:05:10.121657+0000) 2024-03-19T03:05:34.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:34 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:34.100+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.422503+0000 front 2024-03-19T03:04:52.422481+0000 (oldest deadline 2024-03-19T03:05:17.122606+0000) 2024-03-19T03:05:34.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:34 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:34.100+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.924146+0000 front 2024-03-19T03:05:02.924238+0000 (oldest deadline 2024-03-19T03:05:27.624257+0000) 2024-03-19T03:05:34.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:34 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:34.100+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:10.425395+0000 front 2024-03-19T03:05:10.425428+0000 (oldest deadline 2024-03-19T03:05:33.325602+0000) 2024-03-19T03:05:34.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:34 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:34.049+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:34.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:34 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:34.049+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:34.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:34 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:34.049+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:34.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:34 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:34.049+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.318466+0000 front 2024-03-19T03:05:02.318512+0000 (oldest deadline 2024-03-19T03:05:28.119022+0000) 2024-03-19T03:05:34.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:34 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:34.049+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.119327+0000 front 2024-03-19T03:05:08.119368+0000 (oldest deadline 2024-03-19T03:05:34.019391+0000) 2024-03-19T03:05:34.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:34 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:34.476+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:34.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:34 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:34.476+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:34.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:34 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:34.476+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:34.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:34 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:34.476+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.746043+0000 front 2024-03-19T03:05:02.746074+0000 (oldest deadline 2024-03-19T03:05:30.346883+0000) 2024-03-19T03:05:35.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:34 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:34.818+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:42.502712+0000 front 2024-03-19T03:04:42.502587+0000 (oldest deadline 2024-03-19T03:05:13.703191+0000) 2024-03-19T03:05:35.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:34 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:34.818+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:53.703569+0000 front 2024-03-19T03:04:53.703647+0000 (oldest deadline 2024-03-19T03:05:18.403703+0000) 2024-03-19T03:05:35.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:34 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:34.818+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:00.005093+0000 front 2024-03-19T03:05:00.005105+0000 (oldest deadline 2024-03-19T03:05:25.905030+0000) 2024-03-19T03:05:35.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:34 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:34.818+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.806257+0000 front 2024-03-19T03:05:08.806160+0000 (oldest deadline 2024-03-19T03:05:32.906260+0000) 2024-03-19T03:05:35.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:35 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:35.049+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:35.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:35 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:35.049+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:35.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:35 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:35.049+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:35.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:35 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:35.049+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:05:35.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:35 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:35.084+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:44.221520+0000 front 2024-03-19T03:04:44.221546+0000 (oldest deadline 2024-03-19T03:05:10.121657+0000) 2024-03-19T03:05:35.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:35 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:35.084+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.422503+0000 front 2024-03-19T03:04:52.422481+0000 (oldest deadline 2024-03-19T03:05:17.122606+0000) 2024-03-19T03:05:35.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:35 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:35.084+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.924146+0000 front 2024-03-19T03:05:02.924238+0000 (oldest deadline 2024-03-19T03:05:27.624257+0000) 2024-03-19T03:05:35.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:35 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:35.084+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:10.425395+0000 front 2024-03-19T03:05:10.425428+0000 (oldest deadline 2024-03-19T03:05:33.325602+0000) 2024-03-19T03:05:35.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:35 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:35.051+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:35.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:35 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:35.051+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:35.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:35 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:35.051+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:35.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:35 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:35.051+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.318466+0000 front 2024-03-19T03:05:02.318512+0000 (oldest deadline 2024-03-19T03:05:28.119022+0000) 2024-03-19T03:05:35.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:35 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:35.051+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.119327+0000 front 2024-03-19T03:05:08.119368+0000 (oldest deadline 2024-03-19T03:05:34.019391+0000) 2024-03-19T03:05:35.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:35 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:35.437+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:35.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:35 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:35.437+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:35.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:35 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:35.437+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:35.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:35 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:35.437+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.746043+0000 front 2024-03-19T03:05:02.746074+0000 (oldest deadline 2024-03-19T03:05:30.346883+0000) 2024-03-19T03:05:36.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:35 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:35.841+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:42.502712+0000 front 2024-03-19T03:04:42.502587+0000 (oldest deadline 2024-03-19T03:05:13.703191+0000) 2024-03-19T03:05:36.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:35 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:35.841+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:53.703569+0000 front 2024-03-19T03:04:53.703647+0000 (oldest deadline 2024-03-19T03:05:18.403703+0000) 2024-03-19T03:05:36.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:35 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:35.841+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:00.005093+0000 front 2024-03-19T03:05:00.005105+0000 (oldest deadline 2024-03-19T03:05:25.905030+0000) 2024-03-19T03:05:36.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:35 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:35.841+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.806257+0000 front 2024-03-19T03:05:08.806160+0000 (oldest deadline 2024-03-19T03:05:32.906260+0000) 2024-03-19T03:05:36.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:36 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:36.058+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:36.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:36 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:36.058+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:36.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:36 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:36.058+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:36.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:36 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:36.058+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:05:36.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:36 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:36.109+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:44.221520+0000 front 2024-03-19T03:04:44.221546+0000 (oldest deadline 2024-03-19T03:05:10.121657+0000) 2024-03-19T03:05:36.466 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:36 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:36.109+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.422503+0000 front 2024-03-19T03:04:52.422481+0000 (oldest deadline 2024-03-19T03:05:17.122606+0000) 2024-03-19T03:05:36.467 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:36 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:36.109+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.924146+0000 front 2024-03-19T03:05:02.924238+0000 (oldest deadline 2024-03-19T03:05:27.624257+0000) 2024-03-19T03:05:36.467 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:36 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7[53624]: 2024-03-19T03:05:36.109+0000 7f2487d9d700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:10.425395+0000 front 2024-03-19T03:05:10.425428+0000 (oldest deadline 2024-03-19T03:05:33.325602+0000) 2024-03-19T03:05:36.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:36 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:36.056+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:36.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:36 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:36.056+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:36.468 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:36 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:36.056+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:36.468 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:36 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:36.056+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.318466+0000 front 2024-03-19T03:05:02.318512+0000 (oldest deadline 2024-03-19T03:05:28.119022+0000) 2024-03-19T03:05:36.468 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:36 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:36.056+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.119327+0000 front 2024-03-19T03:05:08.119368+0000 (oldest deadline 2024-03-19T03:05:34.019391+0000) 2024-03-19T03:05:36.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:36 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:36.459+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:36.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:36 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:36.459+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:36.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:36 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:36.459+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:36.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:36 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:36.459+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.746043+0000 front 2024-03-19T03:05:02.746074+0000 (oldest deadline 2024-03-19T03:05:30.346883+0000) 2024-03-19T03:05:37.069 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:36 smithi096 podman[71553]: 2024-03-19 03:05:36.74950594 +0000 UTC m=+5.616855139 container died 92c3bf15354667e5287998c98964958b2f4094e7f9a8ee5c8049a35798c00f27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-19T03:05:37.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:36 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:36.851+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:42.502712+0000 front 2024-03-19T03:04:42.502587+0000 (oldest deadline 2024-03-19T03:05:13.703191+0000) 2024-03-19T03:05:37.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:36 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:36.851+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:53.703569+0000 front 2024-03-19T03:04:53.703647+0000 (oldest deadline 2024-03-19T03:05:18.403703+0000) 2024-03-19T03:05:37.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:36 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:36.851+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:00.005093+0000 front 2024-03-19T03:05:00.005105+0000 (oldest deadline 2024-03-19T03:05:25.905030+0000) 2024-03-19T03:05:37.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:36 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:36.851+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.806257+0000 front 2024-03-19T03:05:08.806160+0000 (oldest deadline 2024-03-19T03:05:32.906260+0000) 2024-03-19T03:05:37.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:37 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:37.020+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:37.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:37 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:37.020+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:37.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:37 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:37.020+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:37.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:37 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:37.020+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:05:37.437 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:37 smithi096 podman[71553]: 2024-03-19 03:05:37.168107319 +0000 UTC m=+6.035456519 container cleanup 92c3bf15354667e5287998c98964958b2f4094e7f9a8ee5c8049a35798c00f27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T03:05:37.437 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:37 smithi096 bash[71553]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7 2024-03-19T03:05:37.437 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:37 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:37.100+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:37.437 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:37 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:37.100+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:37.437 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:37 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:37.100+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:37.437 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:37 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:37.100+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.318466+0000 front 2024-03-19T03:05:02.318512+0000 (oldest deadline 2024-03-19T03:05:28.119022+0000) 2024-03-19T03:05:37.437 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:37 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:37.100+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.119327+0000 front 2024-03-19T03:05:08.119368+0000 (oldest deadline 2024-03-19T03:05:34.019391+0000) 2024-03-19T03:05:37.716 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:37 smithi096 podman[71582]: 2024-03-19 03:05:37.435711525 +0000 UTC m=+0.679730883 container remove 92c3bf15354667e5287998c98964958b2f4094e7f9a8ee5c8049a35798c00f27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-19T03:05:37.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:37 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:37.466+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:37.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:37 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:37.466+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:37.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:37 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:37.466+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:37.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:37 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:37.466+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.746043+0000 front 2024-03-19T03:05:02.746074+0000 (oldest deadline 2024-03-19T03:05:30.346883+0000) 2024-03-19T03:05:37.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:37 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:37.466+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:10.347321+0000 front 2024-03-19T03:05:10.347272+0000 (oldest deadline 2024-03-19T03:05:37.347726+0000) 2024-03-19T03:05:38.098 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:37 smithi096 podman[71676]: 2024-03-19 03:05:37.680131906 +0000 UTC m=+0.018623186 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:05:38.099 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:37 smithi096 podman[71676]: 2024-03-19 03:05:37.811694286 +0000 UTC m=+0.150185549 container create 8c621fe3ff68ae912abae7bf276c73f075d9ea55f67de8378b346b09b4449961 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7-deactivate, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-19T03:05:38.099 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:38 smithi096 podman[71676]: 2024-03-19 03:05:38.097921989 +0000 UTC m=+0.436413261 container init 8c621fe3ff68ae912abae7bf276c73f075d9ea55f67de8378b346b09b4449961 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7-deactivate, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T03:05:38.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:37 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:37.871+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:42.502712+0000 front 2024-03-19T03:04:42.502587+0000 (oldest deadline 2024-03-19T03:05:13.703191+0000) 2024-03-19T03:05:38.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:37 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:37.871+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:53.703569+0000 front 2024-03-19T03:04:53.703647+0000 (oldest deadline 2024-03-19T03:05:18.403703+0000) 2024-03-19T03:05:38.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:37 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:37.871+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:00.005093+0000 front 2024-03-19T03:05:00.005105+0000 (oldest deadline 2024-03-19T03:05:25.905030+0000) 2024-03-19T03:05:38.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:37 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:37.871+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.806257+0000 front 2024-03-19T03:05:08.806160+0000 (oldest deadline 2024-03-19T03:05:32.906260+0000) 2024-03-19T03:05:38.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:38 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:38.036+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:38.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:38 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:38.036+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:38.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:38 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:38.036+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:38.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:38 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:38.036+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:05:38.393 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:38 smithi096 podman[71676]: 2024-03-19 03:05:38.10105213 +0000 UTC m=+0.439543393 container start 8c621fe3ff68ae912abae7bf276c73f075d9ea55f67de8378b346b09b4449961 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-19T03:05:38.393 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:38 smithi096 podman[71676]: 2024-03-19 03:05:38.206206419 +0000 UTC m=+0.544697694 container attach 8c621fe3ff68ae912abae7bf276c73f075d9ea55f67de8378b346b09b4449961 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True) 2024-03-19T03:05:38.394 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:38 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:38.135+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:38.394 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:38 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:38.135+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:38.394 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:38 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:38.135+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:38.394 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:38 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:38.135+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.318466+0000 front 2024-03-19T03:05:02.318512+0000 (oldest deadline 2024-03-19T03:05:28.119022+0000) 2024-03-19T03:05:38.394 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:38 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:38.135+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.119327+0000 front 2024-03-19T03:05:08.119368+0000 (oldest deadline 2024-03-19T03:05:34.019391+0000) 2024-03-19T03:05:38.716 INFO:journalctl@ceph.osd.7.smithi096.stdout:Mar 19 03:05:38 smithi096 podman[71676]: 2024-03-19 03:05:38.390037714 +0000 UTC m=+0.728528984 container died 8c621fe3ff68ae912abae7bf276c73f075d9ea55f67de8378b346b09b4449961 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-7-deactivate, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-19T03:05:38.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:38 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:38.447+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:38.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:38 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:38.447+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:38.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:38 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:38.447+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:38.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:38 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:38.447+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.746043+0000 front 2024-03-19T03:05:02.746074+0000 (oldest deadline 2024-03-19T03:05:30.346883+0000) 2024-03-19T03:05:38.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:38 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:38.447+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:10.347321+0000 front 2024-03-19T03:05:10.347272+0000 (oldest deadline 2024-03-19T03:05:37.347726+0000) 2024-03-19T03:05:39.085 DEBUG:teuthology.orchestra.run.smithi096:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.7.service' 2024-03-19T03:05:39.131 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:05:39.131 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-19T03:05:39.131 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-19T03:05:39.131 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.8 2024-03-19T03:05:39.161 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:38 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:38.880+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:42.502712+0000 front 2024-03-19T03:04:42.502587+0000 (oldest deadline 2024-03-19T03:05:13.703191+0000) 2024-03-19T03:05:39.161 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:38 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:38.880+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:53.703569+0000 front 2024-03-19T03:04:53.703647+0000 (oldest deadline 2024-03-19T03:05:18.403703+0000) 2024-03-19T03:05:39.161 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:38 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:38.880+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:00.005093+0000 front 2024-03-19T03:05:00.005105+0000 (oldest deadline 2024-03-19T03:05:25.905030+0000) 2024-03-19T03:05:39.161 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:38 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:38.880+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.806257+0000 front 2024-03-19T03:05:08.806160+0000 (oldest deadline 2024-03-19T03:05:32.906260+0000) 2024-03-19T03:05:39.161 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:39 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:39.012+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:39.162 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:39 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:39.012+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:39.162 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:39 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:39.012+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:39.162 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:39 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:39.012+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:05:39.448 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:39 smithi155 systemd[1]: Stopping Ceph osd.8 for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T03:05:39.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:39 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:39.129+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:39.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:39 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:39.130+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:39.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:39 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:39.130+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:39.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:39 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:39.130+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.318466+0000 front 2024-03-19T03:05:02.318512+0000 (oldest deadline 2024-03-19T03:05:28.119022+0000) 2024-03-19T03:05:39.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:39 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:39.130+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.119327+0000 front 2024-03-19T03:05:08.119368+0000 (oldest deadline 2024-03-19T03:05:34.019391+0000) 2024-03-19T03:05:39.766 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:39 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:39.446+0000 7fec65e96700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:05:39.767 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:39 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:39.446+0000 7fec65e96700 -1 osd.8 112 *** Got signal Terminated *** 2024-03-19T03:05:39.767 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:39 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:39.446+0000 7fec65e96700 -1 osd.8 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T03:05:39.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:39 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:39.431+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:39.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:39 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:39.431+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:39.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:39 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:39.431+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:39.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:39 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:39.431+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.746043+0000 front 2024-03-19T03:05:02.746074+0000 (oldest deadline 2024-03-19T03:05:30.346883+0000) 2024-03-19T03:05:39.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:39 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:39.431+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:10.347321+0000 front 2024-03-19T03:05:10.347272+0000 (oldest deadline 2024-03-19T03:05:37.347726+0000) 2024-03-19T03:05:40.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:39 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:39.866+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:42.502712+0000 front 2024-03-19T03:04:42.502587+0000 (oldest deadline 2024-03-19T03:05:13.703191+0000) 2024-03-19T03:05:40.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:39 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:39.866+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:53.703569+0000 front 2024-03-19T03:04:53.703647+0000 (oldest deadline 2024-03-19T03:05:18.403703+0000) 2024-03-19T03:05:40.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:39 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:39.866+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:00.005093+0000 front 2024-03-19T03:05:00.005105+0000 (oldest deadline 2024-03-19T03:05:25.905030+0000) 2024-03-19T03:05:40.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:39 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:39.866+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.806257+0000 front 2024-03-19T03:05:08.806160+0000 (oldest deadline 2024-03-19T03:05:32.906260+0000) 2024-03-19T03:05:40.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:40 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:40.054+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:40.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:40 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:40.054+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:40.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:40 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:40.054+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:40.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:40 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:40.054+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:05:40.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:40 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:40.135+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:40.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:40 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:40.135+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:40.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:40 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:40.135+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:40.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:40 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:40.135+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.318466+0000 front 2024-03-19T03:05:02.318512+0000 (oldest deadline 2024-03-19T03:05:28.119022+0000) 2024-03-19T03:05:40.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:40 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:40.135+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.119327+0000 front 2024-03-19T03:05:08.119368+0000 (oldest deadline 2024-03-19T03:05:34.019391+0000) 2024-03-19T03:05:40.680 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:40 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:40.418+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:40.680 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:40 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:40.418+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:40.680 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:40 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:40.418+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:40.680 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:40 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:40.418+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.746043+0000 front 2024-03-19T03:05:02.746074+0000 (oldest deadline 2024-03-19T03:05:30.346883+0000) 2024-03-19T03:05:40.680 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:40 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:40.418+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:10.347321+0000 front 2024-03-19T03:05:10.347272+0000 (oldest deadline 2024-03-19T03:05:37.347726+0000) 2024-03-19T03:05:41.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:40 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:40.817+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:42.502712+0000 front 2024-03-19T03:04:42.502587+0000 (oldest deadline 2024-03-19T03:05:13.703191+0000) 2024-03-19T03:05:41.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:40 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:40.817+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:53.703569+0000 front 2024-03-19T03:04:53.703647+0000 (oldest deadline 2024-03-19T03:05:18.403703+0000) 2024-03-19T03:05:41.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:40 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:40.817+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:00.005093+0000 front 2024-03-19T03:05:00.005105+0000 (oldest deadline 2024-03-19T03:05:25.905030+0000) 2024-03-19T03:05:41.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:40 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:40.817+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.806257+0000 front 2024-03-19T03:05:08.806160+0000 (oldest deadline 2024-03-19T03:05:32.906260+0000) 2024-03-19T03:05:41.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:41 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:41.048+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:41.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:41 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:41.048+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:41.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:41 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:41.048+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:41.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:41 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:41.048+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:05:41.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:41 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:41.124+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:41.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:41 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:41.124+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:41.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:41 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:41.124+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:41.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:41 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:41.124+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.318466+0000 front 2024-03-19T03:05:02.318512+0000 (oldest deadline 2024-03-19T03:05:28.119022+0000) 2024-03-19T03:05:41.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:41 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:41.124+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.119327+0000 front 2024-03-19T03:05:08.119368+0000 (oldest deadline 2024-03-19T03:05:34.019391+0000) 2024-03-19T03:05:41.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:41 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:41.371+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:41.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:41 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:41.371+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:41.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:41 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:41.371+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:41.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:41 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:41.371+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.746043+0000 front 2024-03-19T03:05:02.746074+0000 (oldest deadline 2024-03-19T03:05:30.346883+0000) 2024-03-19T03:05:41.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:41 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:41.371+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:10.347321+0000 front 2024-03-19T03:05:10.347272+0000 (oldest deadline 2024-03-19T03:05:37.347726+0000) 2024-03-19T03:05:42.216 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:42 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:42.112+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:42.216 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:42 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:42.112+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:42.216 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:42 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:42.112+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:42.216 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:42 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:42.112+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.318466+0000 front 2024-03-19T03:05:02.318512+0000 (oldest deadline 2024-03-19T03:05:28.119022+0000) 2024-03-19T03:05:42.216 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:42 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:42.112+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.119327+0000 front 2024-03-19T03:05:08.119368+0000 (oldest deadline 2024-03-19T03:05:34.019391+0000) 2024-03-19T03:05:42.216 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:42 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:42.112+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:15.621007+0000 front 2024-03-19T03:05:15.620954+0000 (oldest deadline 2024-03-19T03:05:41.421555+0000) 2024-03-19T03:05:42.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:41 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:41.865+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:42.502712+0000 front 2024-03-19T03:04:42.502587+0000 (oldest deadline 2024-03-19T03:05:13.703191+0000) 2024-03-19T03:05:42.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:41 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:41.865+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:53.703569+0000 front 2024-03-19T03:04:53.703647+0000 (oldest deadline 2024-03-19T03:05:18.403703+0000) 2024-03-19T03:05:42.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:41 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:41.865+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:00.005093+0000 front 2024-03-19T03:05:00.005105+0000 (oldest deadline 2024-03-19T03:05:25.905030+0000) 2024-03-19T03:05:42.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:41 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:41.865+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.806257+0000 front 2024-03-19T03:05:08.806160+0000 (oldest deadline 2024-03-19T03:05:32.906260+0000) 2024-03-19T03:05:42.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:42 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:42.092+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:42.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:42 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:42.092+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:42.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:42 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:42.092+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:42.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:42 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:42.092+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:05:42.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:42 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:42.387+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:42.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:42 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:42.387+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:42.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:42 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:42.387+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:42.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:42 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:42.387+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.746043+0000 front 2024-03-19T03:05:02.746074+0000 (oldest deadline 2024-03-19T03:05:30.346883+0000) 2024-03-19T03:05:42.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:42 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:42.387+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:10.347321+0000 front 2024-03-19T03:05:10.347272+0000 (oldest deadline 2024-03-19T03:05:37.347726+0000) 2024-03-19T03:05:43.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:42 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:42.914+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:42.502712+0000 front 2024-03-19T03:04:42.502587+0000 (oldest deadline 2024-03-19T03:05:13.703191+0000) 2024-03-19T03:05:43.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:42 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:42.914+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:53.703569+0000 front 2024-03-19T03:04:53.703647+0000 (oldest deadline 2024-03-19T03:05:18.403703+0000) 2024-03-19T03:05:43.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:42 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:42.914+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:00.005093+0000 front 2024-03-19T03:05:00.005105+0000 (oldest deadline 2024-03-19T03:05:25.905030+0000) 2024-03-19T03:05:43.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:42 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:42.915+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.806257+0000 front 2024-03-19T03:05:08.806160+0000 (oldest deadline 2024-03-19T03:05:32.906260+0000) 2024-03-19T03:05:43.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:43 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:43.065+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:43.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:43 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:43.065+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:43.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:43 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:43.065+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:43.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:43 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:43.065+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:05:43.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:43 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:43.065+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:16.909958+0000 front 2024-03-19T03:05:16.910060+0000 (oldest deadline 2024-03-19T03:05:42.810008+0000) 2024-03-19T03:05:43.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:43 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:43.084+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:43.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:43 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:43.084+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:43.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:43 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:43.084+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:43.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:43 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:43.084+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.318466+0000 front 2024-03-19T03:05:02.318512+0000 (oldest deadline 2024-03-19T03:05:28.119022+0000) 2024-03-19T03:05:43.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:43 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:43.084+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.119327+0000 front 2024-03-19T03:05:08.119368+0000 (oldest deadline 2024-03-19T03:05:34.019391+0000) 2024-03-19T03:05:43.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:43 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:43.084+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:15.621007+0000 front 2024-03-19T03:05:15.620954+0000 (oldest deadline 2024-03-19T03:05:41.421555+0000) 2024-03-19T03:05:43.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:43 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:43.362+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:43.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:43 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:43.362+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:43.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:43 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:43.362+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:43.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:43 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:43.362+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.746043+0000 front 2024-03-19T03:05:02.746074+0000 (oldest deadline 2024-03-19T03:05:30.346883+0000) 2024-03-19T03:05:43.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:43 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:43.362+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:10.347321+0000 front 2024-03-19T03:05:10.347272+0000 (oldest deadline 2024-03-19T03:05:37.347726+0000) 2024-03-19T03:05:44.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:43 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:43.894+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:42.502712+0000 front 2024-03-19T03:04:42.502587+0000 (oldest deadline 2024-03-19T03:05:13.703191+0000) 2024-03-19T03:05:44.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:43 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:43.894+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:53.703569+0000 front 2024-03-19T03:04:53.703647+0000 (oldest deadline 2024-03-19T03:05:18.403703+0000) 2024-03-19T03:05:44.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:43 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:43.894+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:00.005093+0000 front 2024-03-19T03:05:00.005105+0000 (oldest deadline 2024-03-19T03:05:25.905030+0000) 2024-03-19T03:05:44.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:43 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8[54265]: 2024-03-19T03:05:43.894+0000 7fec5e482700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.806257+0000 front 2024-03-19T03:05:08.806160+0000 (oldest deadline 2024-03-19T03:05:32.906260+0000) 2024-03-19T03:05:44.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:44 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:44.058+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:44.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:44 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:44.058+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:44.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:44 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:44.058+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:44.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:44 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:44.058+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:05:44.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:44 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:44.058+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:16.909958+0000 front 2024-03-19T03:05:16.910060+0000 (oldest deadline 2024-03-19T03:05:42.810008+0000) 2024-03-19T03:05:44.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:44 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:44.048+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:44.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:44 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:44.048+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:44.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:44 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:44.048+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:44.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:44 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:44.048+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.318466+0000 front 2024-03-19T03:05:02.318512+0000 (oldest deadline 2024-03-19T03:05:28.119022+0000) 2024-03-19T03:05:44.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:44 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:44.048+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.119327+0000 front 2024-03-19T03:05:08.119368+0000 (oldest deadline 2024-03-19T03:05:34.019391+0000) 2024-03-19T03:05:44.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:44 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:44.048+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:15.621007+0000 front 2024-03-19T03:05:15.620954+0000 (oldest deadline 2024-03-19T03:05:41.421555+0000) 2024-03-19T03:05:44.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:44 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:44.323+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:44.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:44 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:44.323+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:44.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:44 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:44.323+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:44.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:44 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:44.323+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.746043+0000 front 2024-03-19T03:05:02.746074+0000 (oldest deadline 2024-03-19T03:05:30.346883+0000) 2024-03-19T03:05:44.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:44 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:44.323+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:10.347321+0000 front 2024-03-19T03:05:10.347272+0000 (oldest deadline 2024-03-19T03:05:37.347726+0000) 2024-03-19T03:05:45.267 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:44 smithi155 podman[72824]: 2024-03-19 03:05:44.974131754 +0000 UTC m=+5.646440062 container died e95fc8bf30fdae9cfd49d77c7f1827e86cf147d4f562b6403d74333d040a31ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-19T03:05:45.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:45 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:45.075+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:45.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:45 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:45.075+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:45.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:45 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:45.075+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:45.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:45 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:45.075+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:05:45.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:45 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:45.075+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:16.909958+0000 front 2024-03-19T03:05:16.910060+0000 (oldest deadline 2024-03-19T03:05:42.810008+0000) 2024-03-19T03:05:45.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:45 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:45.061+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:45.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:45 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:45.061+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:45.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:45 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:45.061+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:45.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:45 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:45.061+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.318466+0000 front 2024-03-19T03:05:02.318512+0000 (oldest deadline 2024-03-19T03:05:28.119022+0000) 2024-03-19T03:05:45.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:45 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:45.061+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.119327+0000 front 2024-03-19T03:05:08.119368+0000 (oldest deadline 2024-03-19T03:05:34.019391+0000) 2024-03-19T03:05:45.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:45 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:45.061+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:15.621007+0000 front 2024-03-19T03:05:15.620954+0000 (oldest deadline 2024-03-19T03:05:41.421555+0000) 2024-03-19T03:05:45.632 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:45 smithi155 podman[72824]: 2024-03-19 03:05:45.396663279 +0000 UTC m=+6.068971586 container cleanup e95fc8bf30fdae9cfd49d77c7f1827e86cf147d4f562b6403d74333d040a31ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T03:05:45.632 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:45 smithi155 bash[72824]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8 2024-03-19T03:05:45.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:45 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:45.314+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:45.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:45 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:45.314+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:45.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:45 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:45.314+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:45.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:45 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:45.314+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.746043+0000 front 2024-03-19T03:05:02.746074+0000 (oldest deadline 2024-03-19T03:05:30.346883+0000) 2024-03-19T03:05:45.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:45 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:45.314+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:10.347321+0000 front 2024-03-19T03:05:10.347272+0000 (oldest deadline 2024-03-19T03:05:37.347726+0000) 2024-03-19T03:05:45.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:45 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:45.314+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:20.248774+0000 front 2024-03-19T03:05:20.248891+0000 (oldest deadline 2024-03-19T03:05:44.348776+0000) 2024-03-19T03:05:46.017 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:45 smithi155 podman[72842]: 2024-03-19 03:05:45.631501816 +0000 UTC m=+0.649762292 container remove e95fc8bf30fdae9cfd49d77c7f1827e86cf147d4f562b6403d74333d040a31ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-19T03:05:46.319 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:46 smithi155 podman[72931]: 2024-03-19 03:05:45.971539706 +0000 UTC m=+0.017921965 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:05:46.319 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:46 smithi155 podman[72931]: 2024-03-19 03:05:46.074125601 +0000 UTC m=+0.120507894 container create 6c8df7d7f58a1c64597edab102c61b1221ddf93c467fe6c53beb46f646222782 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8-deactivate, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T03:05:46.319 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:46 smithi155 podman[72931]: 2024-03-19 03:05:46.318263927 +0000 UTC m=+0.364646185 container init 6c8df7d7f58a1c64597edab102c61b1221ddf93c467fe6c53beb46f646222782 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-19T03:05:46.319 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:46 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:46.062+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:46.320 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:46 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:46.062+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:46.320 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:46 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:46.062+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:46.320 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:46 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:46.062+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:05:46.320 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:46 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:46.062+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:16.909958+0000 front 2024-03-19T03:05:16.910060+0000 (oldest deadline 2024-03-19T03:05:42.810008+0000) 2024-03-19T03:05:46.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:46 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:46.061+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:46.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:46 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:46.061+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:46.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:46 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:46.061+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:46.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:46 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:46.061+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.318466+0000 front 2024-03-19T03:05:02.318512+0000 (oldest deadline 2024-03-19T03:05:28.119022+0000) 2024-03-19T03:05:46.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:46 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:46.061+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.119327+0000 front 2024-03-19T03:05:08.119368+0000 (oldest deadline 2024-03-19T03:05:34.019391+0000) 2024-03-19T03:05:46.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:46 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:46.061+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:15.621007+0000 front 2024-03-19T03:05:15.620954+0000 (oldest deadline 2024-03-19T03:05:41.421555+0000) 2024-03-19T03:05:46.614 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:46 smithi155 podman[72931]: 2024-03-19 03:05:46.322026546 +0000 UTC m=+0.368408804 container start 6c8df7d7f58a1c64597edab102c61b1221ddf93c467fe6c53beb46f646222782 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T03:05:46.615 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:46 smithi155 podman[72931]: 2024-03-19 03:05:46.452244487 +0000 UTC m=+0.498626738 container attach 6c8df7d7f58a1c64597edab102c61b1221ddf93c467fe6c53beb46f646222782 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-19T03:05:46.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:46 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:46.315+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:46.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:46 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:46.315+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:46.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:46 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:46.315+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:46.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:46 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:46.315+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.746043+0000 front 2024-03-19T03:05:02.746074+0000 (oldest deadline 2024-03-19T03:05:30.346883+0000) 2024-03-19T03:05:46.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:46 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:46.315+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:10.347321+0000 front 2024-03-19T03:05:10.347272+0000 (oldest deadline 2024-03-19T03:05:37.347726+0000) 2024-03-19T03:05:46.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:46 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:46.315+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:20.248774+0000 front 2024-03-19T03:05:20.248891+0000 (oldest deadline 2024-03-19T03:05:44.348776+0000) 2024-03-19T03:05:46.922 INFO:journalctl@ceph.osd.8.smithi155.stdout:Mar 19 03:05:46 smithi155 podman[72931]: 2024-03-19 03:05:46.613751519 +0000 UTC m=+0.660133780 container died 6c8df7d7f58a1c64597edab102c61b1221ddf93c467fe6c53beb46f646222782 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-8-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-19T03:05:47.215 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:47 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:47.017+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:47.215 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:47 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:47.017+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:47.215 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:47 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:47.017+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:47.215 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:47 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:47.017+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:05:47.215 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:47 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:47.017+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:16.909958+0000 front 2024-03-19T03:05:16.910060+0000 (oldest deadline 2024-03-19T03:05:42.810008+0000) 2024-03-19T03:05:47.245 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.8.service' 2024-03-19T03:05:47.326 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:05:47.326 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-19T03:05:47.326 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-19T03:05:47.326 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.9 2024-03-19T03:05:47.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:47 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:47.015+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:47.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:47 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:47.015+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:47.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:47 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:47.015+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:47.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:47 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:47.015+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.318466+0000 front 2024-03-19T03:05:02.318512+0000 (oldest deadline 2024-03-19T03:05:28.119022+0000) 2024-03-19T03:05:47.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:47 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:47.015+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.119327+0000 front 2024-03-19T03:05:08.119368+0000 (oldest deadline 2024-03-19T03:05:34.019391+0000) 2024-03-19T03:05:47.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:47 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:47.015+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:15.621007+0000 front 2024-03-19T03:05:15.620954+0000 (oldest deadline 2024-03-19T03:05:41.421555+0000) 2024-03-19T03:05:47.658 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:47 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:47.331+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:47.659 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:47 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:47.331+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:47.659 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:47 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:47.331+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:47.659 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:47 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:47.331+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.746043+0000 front 2024-03-19T03:05:02.746074+0000 (oldest deadline 2024-03-19T03:05:30.346883+0000) 2024-03-19T03:05:47.659 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:47 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:47.331+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:10.347321+0000 front 2024-03-19T03:05:10.347272+0000 (oldest deadline 2024-03-19T03:05:37.347726+0000) 2024-03-19T03:05:47.659 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:47 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:47.331+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:20.248774+0000 front 2024-03-19T03:05:20.248891+0000 (oldest deadline 2024-03-19T03:05:44.348776+0000) 2024-03-19T03:05:47.659 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:47 smithi033 systemd[1]: Stopping Ceph osd.9 for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T03:05:47.659 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:47 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:47.657+0000 7fe2d2b03700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:05:47.659 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:47 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:47.657+0000 7fe2d2b03700 -1 osd.9 112 *** Got signal Terminated *** 2024-03-19T03:05:47.660 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:47 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:47.657+0000 7fe2d2b03700 -1 osd.9 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T03:05:48.430 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:48 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:48.018+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:48.430 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:48 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:48.018+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:48.431 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:48 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:48.018+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:48.431 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:48 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:48.018+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:05:48.431 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:48 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:48.018+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:16.909958+0000 front 2024-03-19T03:05:16.910060+0000 (oldest deadline 2024-03-19T03:05:42.810008+0000) 2024-03-19T03:05:48.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:48 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:48.008+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:48.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:48 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:48.008+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:48.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:48 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:48.008+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:48.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:48 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:48.008+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.318466+0000 front 2024-03-19T03:05:02.318512+0000 (oldest deadline 2024-03-19T03:05:28.119022+0000) 2024-03-19T03:05:48.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:48 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:48.008+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.119327+0000 front 2024-03-19T03:05:08.119368+0000 (oldest deadline 2024-03-19T03:05:34.019391+0000) 2024-03-19T03:05:48.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:48 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:48.008+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:15.621007+0000 front 2024-03-19T03:05:15.620954+0000 (oldest deadline 2024-03-19T03:05:41.421555+0000) 2024-03-19T03:05:48.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:48 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:48.283+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:48.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:48 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:48.283+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:48.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:48 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:48.283+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:48.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:48 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:48.283+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.746043+0000 front 2024-03-19T03:05:02.746074+0000 (oldest deadline 2024-03-19T03:05:30.346883+0000) 2024-03-19T03:05:48.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:48 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:48.283+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:10.347321+0000 front 2024-03-19T03:05:10.347272+0000 (oldest deadline 2024-03-19T03:05:37.347726+0000) 2024-03-19T03:05:48.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:48 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:48.283+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:20.248774+0000 front 2024-03-19T03:05:20.248891+0000 (oldest deadline 2024-03-19T03:05:44.348776+0000) 2024-03-19T03:05:49.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:49 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:49.003+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:49.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:49 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:49.003+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:49.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:49 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:49.003+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:49.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:49 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:49.003+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:05:49.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:49 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:49.003+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:16.909958+0000 front 2024-03-19T03:05:16.910060+0000 (oldest deadline 2024-03-19T03:05:42.810008+0000) 2024-03-19T03:05:49.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:49 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:49.032+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:49.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:49 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:49.032+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:49.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:49 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:49.032+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:49.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:49 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:49.032+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.318466+0000 front 2024-03-19T03:05:02.318512+0000 (oldest deadline 2024-03-19T03:05:28.119022+0000) 2024-03-19T03:05:49.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:49 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:49.032+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.119327+0000 front 2024-03-19T03:05:08.119368+0000 (oldest deadline 2024-03-19T03:05:34.019391+0000) 2024-03-19T03:05:49.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:49 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:49.032+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:15.621007+0000 front 2024-03-19T03:05:15.620954+0000 (oldest deadline 2024-03-19T03:05:41.421555+0000) 2024-03-19T03:05:49.529 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:49 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:49.257+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:49.529 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:49 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:49.257+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:49.529 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:49 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:49.257+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:49.529 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:49 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:49.257+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.746043+0000 front 2024-03-19T03:05:02.746074+0000 (oldest deadline 2024-03-19T03:05:30.346883+0000) 2024-03-19T03:05:49.529 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:49 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:49.257+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:10.347321+0000 front 2024-03-19T03:05:10.347272+0000 (oldest deadline 2024-03-19T03:05:37.347726+0000) 2024-03-19T03:05:49.529 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:49 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:49.257+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:20.248774+0000 front 2024-03-19T03:05:20.248891+0000 (oldest deadline 2024-03-19T03:05:44.348776+0000) 2024-03-19T03:05:50.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:50 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:50.015+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:50.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:50 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:50.015+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:50.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:50 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:50.015+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:50.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:50 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:50.015+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:05:50.268 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:50 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:50.015+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:16.909958+0000 front 2024-03-19T03:05:16.910060+0000 (oldest deadline 2024-03-19T03:05:42.810008+0000) 2024-03-19T03:05:50.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:50 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:50.036+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:50.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:50 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:50.036+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:50.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:50 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:50.036+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:50.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:50 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:50.036+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.318466+0000 front 2024-03-19T03:05:02.318512+0000 (oldest deadline 2024-03-19T03:05:28.119022+0000) 2024-03-19T03:05:50.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:50 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:50.036+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.119327+0000 front 2024-03-19T03:05:08.119368+0000 (oldest deadline 2024-03-19T03:05:34.019391+0000) 2024-03-19T03:05:50.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:50 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:50.036+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:15.621007+0000 front 2024-03-19T03:05:15.620954+0000 (oldest deadline 2024-03-19T03:05:41.421555+0000) 2024-03-19T03:05:50.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:50 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:50.306+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:50.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:50 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:50.306+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:50.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:50 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:50.306+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:50.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:50 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:50.306+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.746043+0000 front 2024-03-19T03:05:02.746074+0000 (oldest deadline 2024-03-19T03:05:30.346883+0000) 2024-03-19T03:05:50.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:50 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:50.306+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:10.347321+0000 front 2024-03-19T03:05:10.347272+0000 (oldest deadline 2024-03-19T03:05:37.347726+0000) 2024-03-19T03:05:50.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:50 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:50.306+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:20.248774+0000 front 2024-03-19T03:05:20.248891+0000 (oldest deadline 2024-03-19T03:05:44.348776+0000) 2024-03-19T03:05:51.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:51 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:51.007+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:51.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:51 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:51.007+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:51.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:51 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:51.007+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:51.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:51 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:51.007+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:05:51.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:51 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:51.007+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:16.909958+0000 front 2024-03-19T03:05:16.910060+0000 (oldest deadline 2024-03-19T03:05:42.810008+0000) 2024-03-19T03:05:51.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:50 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:50.998+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:51.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:50 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:50.998+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:51.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:50 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:50.998+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:51.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:50 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:50.998+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.318466+0000 front 2024-03-19T03:05:02.318512+0000 (oldest deadline 2024-03-19T03:05:28.119022+0000) 2024-03-19T03:05:51.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:50 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:50.998+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.119327+0000 front 2024-03-19T03:05:08.119368+0000 (oldest deadline 2024-03-19T03:05:34.019391+0000) 2024-03-19T03:05:51.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:50 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:50.998+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:15.621007+0000 front 2024-03-19T03:05:15.620954+0000 (oldest deadline 2024-03-19T03:05:41.421555+0000) 2024-03-19T03:05:51.529 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:51 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:51.260+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:51.529 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:51 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:51.260+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:51.529 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:51 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:51.260+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:51.529 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:51 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:51.260+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.746043+0000 front 2024-03-19T03:05:02.746074+0000 (oldest deadline 2024-03-19T03:05:30.346883+0000) 2024-03-19T03:05:51.529 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:51 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:51.260+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:10.347321+0000 front 2024-03-19T03:05:10.347272+0000 (oldest deadline 2024-03-19T03:05:37.347726+0000) 2024-03-19T03:05:51.529 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:51 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:51.260+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:20.248774+0000 front 2024-03-19T03:05:20.248891+0000 (oldest deadline 2024-03-19T03:05:44.348776+0000) 2024-03-19T03:05:52.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:52 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:52.004+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:52.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:52 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:52.004+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:52.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:52 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:52.004+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:52.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:52 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:52.004+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:05:52.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:52 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:52.004+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:16.909958+0000 front 2024-03-19T03:05:16.910060+0000 (oldest deadline 2024-03-19T03:05:42.810008+0000) 2024-03-19T03:05:52.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:51 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:51.986+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:52.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:51 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:51.986+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:52.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:51 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:51.986+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:52.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:51 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:51.986+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.318466+0000 front 2024-03-19T03:05:02.318512+0000 (oldest deadline 2024-03-19T03:05:28.119022+0000) 2024-03-19T03:05:52.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:51 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:51.986+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.119327+0000 front 2024-03-19T03:05:08.119368+0000 (oldest deadline 2024-03-19T03:05:34.019391+0000) 2024-03-19T03:05:52.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:51 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:51.986+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:15.621007+0000 front 2024-03-19T03:05:15.620954+0000 (oldest deadline 2024-03-19T03:05:41.421555+0000) 2024-03-19T03:05:52.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:52 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:52.309+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:38.842215+0000 front 2024-03-19T03:04:38.842326+0000 (oldest deadline 2024-03-19T03:05:01.142444+0000) 2024-03-19T03:05:52.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:52 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:52.309+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:43.443473+0000 front 2024-03-19T03:04:43.443531+0000 (oldest deadline 2024-03-19T03:05:08.743475+0000) 2024-03-19T03:05:52.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:52 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:52.309+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:54.644612+0000 front 2024-03-19T03:04:54.644578+0000 (oldest deadline 2024-03-19T03:05:16.944630+0000) 2024-03-19T03:05:52.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:52 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:52.309+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.746043+0000 front 2024-03-19T03:05:02.746074+0000 (oldest deadline 2024-03-19T03:05:30.346883+0000) 2024-03-19T03:05:52.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:52 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:52.309+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:10.347321+0000 front 2024-03-19T03:05:10.347272+0000 (oldest deadline 2024-03-19T03:05:37.347726+0000) 2024-03-19T03:05:52.779 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:52 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9[67019]: 2024-03-19T03:05:52.309+0000 7fe2cb0ef700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:20.248774+0000 front 2024-03-19T03:05:20.248891+0000 (oldest deadline 2024-03-19T03:05:44.348776+0000) 2024-03-19T03:05:53.325 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:53 smithi033 podman[102179]: 2024-03-19 03:05:53.055823831 +0000 UTC m=+5.530046999 container died b3b0df9d95fcde05e4c68e59276c8094fdd9aaf0c30edc477e137194128ae48d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-19T03:05:53.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:52 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:52.994+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:53.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:52 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:52.994+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:53.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:52 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:52.994+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:53.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:52 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:52.994+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.318466+0000 front 2024-03-19T03:05:02.318512+0000 (oldest deadline 2024-03-19T03:05:28.119022+0000) 2024-03-19T03:05:53.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:52 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:52.994+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.119327+0000 front 2024-03-19T03:05:08.119368+0000 (oldest deadline 2024-03-19T03:05:34.019391+0000) 2024-03-19T03:05:53.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:52 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:52.994+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:15.621007+0000 front 2024-03-19T03:05:15.620954+0000 (oldest deadline 2024-03-19T03:05:41.421555+0000) 2024-03-19T03:05:53.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:53 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:53.034+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:53.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:53 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:53.034+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:53.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:53 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:53.034+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:53.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:53 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:53.034+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:05:53.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:53 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:53.034+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:16.909958+0000 front 2024-03-19T03:05:16.910060+0000 (oldest deadline 2024-03-19T03:05:42.810008+0000) 2024-03-19T03:05:53.667 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:53 smithi033 podman[102179]: 2024-03-19 03:05:53.423978355 +0000 UTC m=+5.898201520 container cleanup b3b0df9d95fcde05e4c68e59276c8094fdd9aaf0c30edc477e137194128ae48d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T03:05:53.667 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:53 smithi033 bash[102179]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9 2024-03-19T03:05:54.002 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:53 smithi033 podman[102197]: 2024-03-19 03:05:53.666635452 +0000 UTC m=+0.606415480 container remove b3b0df9d95fcde05e4c68e59276c8094fdd9aaf0c30edc477e137194128ae48d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS) 2024-03-19T03:05:54.262 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:54 smithi033 podman[102288]: 2024-03-19 03:05:53.904400554 +0000 UTC m=+0.018465286 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:05:54.262 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:54 smithi033 podman[102288]: 2024-03-19 03:05:54.017555214 +0000 UTC m=+0.131619929 container create 054019706a41ccfaa944e26d4e5b1d2ef74ecfff3a2ad2d4767bfd323674b421 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-19T03:05:54.262 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:54 smithi033 podman[102288]: 2024-03-19 03:05:54.261912822 +0000 UTC m=+0.375977547 container init 054019706a41ccfaa944e26d4e5b1d2ef74ecfff3a2ad2d4767bfd323674b421 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-19T03:05:54.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:53 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:53.986+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:54.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:53 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:53.986+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:54.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:53 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:53.986+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:54.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:53 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:53.986+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:05:54.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:53 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:53.986+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:16.909958+0000 front 2024-03-19T03:05:16.910060+0000 (oldest deadline 2024-03-19T03:05:42.810008+0000) 2024-03-19T03:05:54.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:54 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:54.000+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:54.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:54 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:54.000+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:54.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:54 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:54.001+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:54.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:54 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:54.001+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.318466+0000 front 2024-03-19T03:05:02.318512+0000 (oldest deadline 2024-03-19T03:05:28.119022+0000) 2024-03-19T03:05:54.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:54 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:54.001+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.119327+0000 front 2024-03-19T03:05:08.119368+0000 (oldest deadline 2024-03-19T03:05:34.019391+0000) 2024-03-19T03:05:54.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:54 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:54.001+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:15.621007+0000 front 2024-03-19T03:05:15.620954+0000 (oldest deadline 2024-03-19T03:05:41.421555+0000) 2024-03-19T03:05:54.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:54 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:54.001+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-03-19T03:05:27.123850+0000 front 2024-03-19T03:05:27.123784+0000 (oldest deadline 2024-03-19T03:05:53.023598+0000) 2024-03-19T03:05:54.529 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:54 smithi033 podman[102288]: 2024-03-19 03:05:54.26512695 +0000 UTC m=+0.379191666 container start 054019706a41ccfaa944e26d4e5b1d2ef74ecfff3a2ad2d4767bfd323674b421 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9-deactivate, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T03:05:54.529 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:54 smithi033 podman[102288]: 2024-03-19 03:05:54.345275788 +0000 UTC m=+0.459340533 container attach 054019706a41ccfaa944e26d4e5b1d2ef74ecfff3a2ad2d4767bfd323674b421 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-19T03:05:54.823 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:54 smithi033 podman[102288]: 2024-03-19 03:05:54.556631777 +0000 UTC m=+0.670696486 container died 054019706a41ccfaa944e26d4e5b1d2ef74ecfff3a2ad2d4767bfd323674b421 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-19T03:05:55.051 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.9.service' 2024-03-19T03:05:55.077 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:55 smithi033 podman[102314]: 2024-03-19 03:05:55.032082441 +0000 UTC m=+0.470508442 container remove 054019706a41ccfaa944e26d4e5b1d2ef74ecfff3a2ad2d4767bfd323674b421 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-19T03:05:55.077 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:55 smithi033 systemd[1]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.9.service: Deactivated successfully. 2024-03-19T03:05:55.077 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:55 smithi033 systemd[1]: Stopped Ceph osd.9 for 2b7a8938-e59a-11ee-95c9-87774f69a715. 2024-03-19T03:05:55.077 INFO:journalctl@ceph.osd.9.smithi033.stdout:Mar 19 03:05:55 smithi033 systemd[1]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.9.service: Consumed 54.927s CPU time. 2024-03-19T03:05:55.093 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:05:55.093 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-19T03:05:55.093 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-19T03:05:55.093 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.10 2024-03-19T03:05:55.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:54 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:54.961+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:55.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:54 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:54.961+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:55.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:54 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:54.961+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:55.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:54 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:54.961+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:05:55.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:54 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:54.961+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:16.909958+0000 front 2024-03-19T03:05:16.910060+0000 (oldest deadline 2024-03-19T03:05:42.810008+0000) 2024-03-19T03:05:55.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:54 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:54.961+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-03-19T03:05:22.810471+0000 front 2024-03-19T03:05:22.810439+0000 (oldest deadline 2024-03-19T03:05:54.610993+0000) 2024-03-19T03:05:55.404 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:54 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:54.996+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:55.404 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:54 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:54.996+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:55.404 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:54 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:54.996+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:55.404 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:54 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:54.996+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.318466+0000 front 2024-03-19T03:05:02.318512+0000 (oldest deadline 2024-03-19T03:05:28.119022+0000) 2024-03-19T03:05:55.404 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:54 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:54.996+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.119327+0000 front 2024-03-19T03:05:08.119368+0000 (oldest deadline 2024-03-19T03:05:34.019391+0000) 2024-03-19T03:05:55.404 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:54 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:54.996+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:15.621007+0000 front 2024-03-19T03:05:15.620954+0000 (oldest deadline 2024-03-19T03:05:41.421555+0000) 2024-03-19T03:05:55.405 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:54 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:54.996+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-03-19T03:05:27.123850+0000 front 2024-03-19T03:05:27.123784+0000 (oldest deadline 2024-03-19T03:05:53.023598+0000) 2024-03-19T03:05:55.405 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:55 smithi096 systemd[1]: Stopping Ceph osd.10 for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T03:05:55.405 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:55 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:55.402+0000 7f25a9a8a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:05:55.405 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:55 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:55.402+0000 7f25a9a8a700 -1 osd.10 112 *** Got signal Terminated *** 2024-03-19T03:05:55.405 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:55 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:55.402+0000 7f25a9a8a700 -1 osd.10 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T03:05:56.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:55 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:55.967+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:56.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:55 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:55.967+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:56.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:55 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:55.967+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:56.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:55 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:55.967+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:05:56.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:55 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:55.967+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:16.909958+0000 front 2024-03-19T03:05:16.910060+0000 (oldest deadline 2024-03-19T03:05:42.810008+0000) 2024-03-19T03:05:56.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:55 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:55.967+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-03-19T03:05:22.810471+0000 front 2024-03-19T03:05:22.810439+0000 (oldest deadline 2024-03-19T03:05:54.610993+0000) 2024-03-19T03:05:56.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:56 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:56.044+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:56.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:56 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:56.044+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:56.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:56 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:56.044+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:56.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:56 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:56.044+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.318466+0000 front 2024-03-19T03:05:02.318512+0000 (oldest deadline 2024-03-19T03:05:28.119022+0000) 2024-03-19T03:05:56.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:56 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:56.044+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.119327+0000 front 2024-03-19T03:05:08.119368+0000 (oldest deadline 2024-03-19T03:05:34.019391+0000) 2024-03-19T03:05:56.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:56 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:56.044+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:15.621007+0000 front 2024-03-19T03:05:15.620954+0000 (oldest deadline 2024-03-19T03:05:41.421555+0000) 2024-03-19T03:05:56.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:56 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:56.044+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-03-19T03:05:27.123850+0000 front 2024-03-19T03:05:27.123784+0000 (oldest deadline 2024-03-19T03:05:53.023598+0000) 2024-03-19T03:05:57.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:57 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:57.008+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:57.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:57 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:57.008+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:57.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:57 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:57.008+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:57.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:57 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:57.008+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:05:57.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:57 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:57.008+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:16.909958+0000 front 2024-03-19T03:05:16.910060+0000 (oldest deadline 2024-03-19T03:05:42.810008+0000) 2024-03-19T03:05:57.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:57 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:57.008+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-03-19T03:05:22.810471+0000 front 2024-03-19T03:05:22.810439+0000 (oldest deadline 2024-03-19T03:05:54.610993+0000) 2024-03-19T03:05:57.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:57 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:57.078+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:57.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:57 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:57.078+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:57.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:57 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:57.078+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:57.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:57 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:57.078+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.318466+0000 front 2024-03-19T03:05:02.318512+0000 (oldest deadline 2024-03-19T03:05:28.119022+0000) 2024-03-19T03:05:57.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:57 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:57.078+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.119327+0000 front 2024-03-19T03:05:08.119368+0000 (oldest deadline 2024-03-19T03:05:34.019391+0000) 2024-03-19T03:05:57.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:57 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:57.078+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:15.621007+0000 front 2024-03-19T03:05:15.620954+0000 (oldest deadline 2024-03-19T03:05:41.421555+0000) 2024-03-19T03:05:57.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:57 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:57.078+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-03-19T03:05:27.123850+0000 front 2024-03-19T03:05:27.123784+0000 (oldest deadline 2024-03-19T03:05:53.023598+0000) 2024-03-19T03:05:58.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:58 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:58.073+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:58.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:58 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:58.073+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:58.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:58 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:58.073+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:58.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:58 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:58.073+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.318466+0000 front 2024-03-19T03:05:02.318512+0000 (oldest deadline 2024-03-19T03:05:28.119022+0000) 2024-03-19T03:05:58.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:58 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:58.074+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.119327+0000 front 2024-03-19T03:05:08.119368+0000 (oldest deadline 2024-03-19T03:05:34.019391+0000) 2024-03-19T03:05:58.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:58 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:58.074+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:15.621007+0000 front 2024-03-19T03:05:15.620954+0000 (oldest deadline 2024-03-19T03:05:41.421555+0000) 2024-03-19T03:05:58.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:58 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:58.074+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-03-19T03:05:27.123850+0000 front 2024-03-19T03:05:27.123784+0000 (oldest deadline 2024-03-19T03:05:53.023598+0000) 2024-03-19T03:05:58.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:58 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:58.050+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:58.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:58 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:58.050+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:58.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:58 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:58.050+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:58.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:58 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:58.050+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:05:58.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:58 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:58.050+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:16.909958+0000 front 2024-03-19T03:05:16.910060+0000 (oldest deadline 2024-03-19T03:05:42.810008+0000) 2024-03-19T03:05:58.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:58 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:58.050+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-03-19T03:05:22.810471+0000 front 2024-03-19T03:05:22.810439+0000 (oldest deadline 2024-03-19T03:05:54.610993+0000) 2024-03-19T03:05:58.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:58 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:58.050+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6820 osd.7 since back 2024-03-19T03:05:34.611667+0000 front 2024-03-19T03:05:34.611611+0000 (oldest deadline 2024-03-19T03:05:57.511523+0000) 2024-03-19T03:05:59.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:59 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:59.002+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:05:59.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:59 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:59.002+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:05:59.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:59 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:59.002+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:05:59.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:59 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:59.002+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:05:59.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:59 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:59.002+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:16.909958+0000 front 2024-03-19T03:05:16.910060+0000 (oldest deadline 2024-03-19T03:05:42.810008+0000) 2024-03-19T03:05:59.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:59 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:59.002+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-03-19T03:05:22.810471+0000 front 2024-03-19T03:05:22.810439+0000 (oldest deadline 2024-03-19T03:05:54.610993+0000) 2024-03-19T03:05:59.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:59 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:59.002+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6820 osd.7 since back 2024-03-19T03:05:34.611667+0000 front 2024-03-19T03:05:34.611611+0000 (oldest deadline 2024-03-19T03:05:57.511523+0000) 2024-03-19T03:05:59.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:59 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:59.038+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:05:59.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:59 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:59.038+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:05:59.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:59 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:59.038+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:05:59.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:59 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:59.038+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.318466+0000 front 2024-03-19T03:05:02.318512+0000 (oldest deadline 2024-03-19T03:05:28.119022+0000) 2024-03-19T03:05:59.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:59 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:59.038+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.119327+0000 front 2024-03-19T03:05:08.119368+0000 (oldest deadline 2024-03-19T03:05:34.019391+0000) 2024-03-19T03:05:59.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:59 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:59.038+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:15.621007+0000 front 2024-03-19T03:05:15.620954+0000 (oldest deadline 2024-03-19T03:05:41.421555+0000) 2024-03-19T03:05:59.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:05:59 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:05:59.038+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-03-19T03:05:27.123850+0000 front 2024-03-19T03:05:27.123784+0000 (oldest deadline 2024-03-19T03:05:53.023598+0000) 2024-03-19T03:06:00.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:59 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:59.993+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:06:00.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:59 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:59.993+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:06:00.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:59 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:59.993+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:06:00.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:59 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:59.993+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:06:00.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:59 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:59.993+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:16.909958+0000 front 2024-03-19T03:05:16.910060+0000 (oldest deadline 2024-03-19T03:05:42.810008+0000) 2024-03-19T03:06:00.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:59 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:59.993+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-03-19T03:05:22.810471+0000 front 2024-03-19T03:05:22.810439+0000 (oldest deadline 2024-03-19T03:05:54.610993+0000) 2024-03-19T03:06:00.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:05:59 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:05:59.993+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6820 osd.7 since back 2024-03-19T03:05:34.611667+0000 front 2024-03-19T03:05:34.611611+0000 (oldest deadline 2024-03-19T03:05:57.511523+0000) 2024-03-19T03:06:00.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:06:00 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:06:00.070+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:39.113604+0000 front 2024-03-19T03:04:39.113658+0000 (oldest deadline 2024-03-19T03:05:05.013574+0000) 2024-03-19T03:06:00.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:06:00 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:06:00.070+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:47.914734+0000 front 2024-03-19T03:04:47.914815+0000 (oldest deadline 2024-03-19T03:05:12.515578+0000) 2024-03-19T03:06:00.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:06:00 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:06:00.070+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:55.317318+0000 front 2024-03-19T03:04:55.317242+0000 (oldest deadline 2024-03-19T03:05:20.017313+0000) 2024-03-19T03:06:00.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:06:00 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:06:00.070+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6814 osd.3 since back 2024-03-19T03:05:02.318466+0000 front 2024-03-19T03:05:02.318512+0000 (oldest deadline 2024-03-19T03:05:28.119022+0000) 2024-03-19T03:06:00.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:06:00 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:06:00.070+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.119327+0000 front 2024-03-19T03:05:08.119368+0000 (oldest deadline 2024-03-19T03:05:34.019391+0000) 2024-03-19T03:06:00.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:06:00 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:06:00.070+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:15.621007+0000 front 2024-03-19T03:05:15.620954+0000 (oldest deadline 2024-03-19T03:05:41.421555+0000) 2024-03-19T03:06:00.467 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:06:00 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10[60244]: 2024-03-19T03:06:00.070+0000 7f25a607e700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-03-19T03:05:27.123850+0000 front 2024-03-19T03:05:27.123784+0000 (oldest deadline 2024-03-19T03:05:53.023598+0000) 2024-03-19T03:06:01.170 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:06:00 smithi096 podman[71781]: 2024-03-19 03:06:00.907835344 +0000 UTC m=+5.627745618 container died 1b7668219643a44bd1e68394d5563a49a1ca0bd1bfc05fb82e6ea05b22cc3c2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-19T03:06:01.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:00 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:00.990+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:06:01.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:00 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:00.990+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:06:01.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:00 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:00.990+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:06:01.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:00 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:00.990+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:06:01.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:00 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:00.990+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:16.909958+0000 front 2024-03-19T03:05:16.910060+0000 (oldest deadline 2024-03-19T03:05:42.810008+0000) 2024-03-19T03:06:01.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:00 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:00.990+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-03-19T03:05:22.810471+0000 front 2024-03-19T03:05:22.810439+0000 (oldest deadline 2024-03-19T03:05:54.610993+0000) 2024-03-19T03:06:01.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:00 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:00.990+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6820 osd.7 since back 2024-03-19T03:05:34.611667+0000 front 2024-03-19T03:05:34.611611+0000 (oldest deadline 2024-03-19T03:05:57.511523+0000) 2024-03-19T03:06:01.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:06:01 smithi096 podman[71781]: 2024-03-19 03:06:01.268554231 +0000 UTC m=+5.988464543 container cleanup 1b7668219643a44bd1e68394d5563a49a1ca0bd1bfc05fb82e6ea05b22cc3c2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-19T03:06:01.466 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:06:01 smithi096 bash[71781]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10 2024-03-19T03:06:01.930 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:06:01 smithi096 podman[71800]: 2024-03-19 03:06:01.586430776 +0000 UTC m=+0.677172078 container remove 1b7668219643a44bd1e68394d5563a49a1ca0bd1bfc05fb82e6ea05b22cc3c2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T03:06:02.216 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:06:01 smithi096 podman[71890]: 2024-03-19 03:06:01.929114903 +0000 UTC m=+0.110365697 container create a549eddbaee2f6154a2c2fe626143f92fe4c5c400444340b5aa132f5637ee836 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T03:06:02.216 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:06:01 smithi096 podman[71890]: 2024-03-19 03:06:01.836457382 +0000 UTC m=+0.017708177 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:06:02.216 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:06:02 smithi096 podman[71890]: 2024-03-19 03:06:02.148243259 +0000 UTC m=+0.329494049 container init a549eddbaee2f6154a2c2fe626143f92fe4c5c400444340b5aa132f5637ee836 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True) 2024-03-19T03:06:02.216 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:06:02 smithi096 podman[71890]: 2024-03-19 03:06:02.151410803 +0000 UTC m=+0.332661597 container start a549eddbaee2f6154a2c2fe626143f92fe4c5c400444340b5aa132f5637ee836 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-19T03:06:02.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:01 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:01.977+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:06:02.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:01 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:01.977+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:06:02.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:01 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:01.977+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:06:02.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:01 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:01.977+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:06:02.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:01 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:01.977+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:16.909958+0000 front 2024-03-19T03:05:16.910060+0000 (oldest deadline 2024-03-19T03:05:42.810008+0000) 2024-03-19T03:06:02.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:01 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:01.977+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-03-19T03:05:22.810471+0000 front 2024-03-19T03:05:22.810439+0000 (oldest deadline 2024-03-19T03:05:54.610993+0000) 2024-03-19T03:06:02.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:01 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:01.977+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6820 osd.7 since back 2024-03-19T03:05:34.611667+0000 front 2024-03-19T03:05:34.611611+0000 (oldest deadline 2024-03-19T03:05:57.511523+0000) 2024-03-19T03:06:02.716 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:06:02 smithi096 podman[71890]: 2024-03-19 03:06:02.239573205 +0000 UTC m=+0.420824003 container attach a549eddbaee2f6154a2c2fe626143f92fe4c5c400444340b5aa132f5637ee836 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10-deactivate, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T03:06:02.716 INFO:journalctl@ceph.osd.10.smithi096.stdout:Mar 19 03:06:02 smithi096 podman[71890]: 2024-03-19 03:06:02.439463793 +0000 UTC m=+0.620714588 container died a549eddbaee2f6154a2c2fe626143f92fe4c5c400444340b5aa132f5637ee836 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-10-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T03:06:03.046 DEBUG:teuthology.orchestra.run.smithi096:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.10.service' 2024-03-19T03:06:03.086 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:06:03.087 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-19T03:06:03.087 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-19T03:06:03.087 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.11 2024-03-19T03:06:03.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:02 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:02.995+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:06:03.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:02 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:02.995+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:06:03.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:02 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:02.995+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:06:03.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:02 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:02.995+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:06:03.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:02 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:02.995+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:16.909958+0000 front 2024-03-19T03:05:16.910060+0000 (oldest deadline 2024-03-19T03:05:42.810008+0000) 2024-03-19T03:06:03.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:02 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:02.995+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-03-19T03:05:22.810471+0000 front 2024-03-19T03:05:22.810439+0000 (oldest deadline 2024-03-19T03:05:54.610993+0000) 2024-03-19T03:06:03.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:02 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:02.995+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6820 osd.7 since back 2024-03-19T03:05:34.611667+0000 front 2024-03-19T03:05:34.611611+0000 (oldest deadline 2024-03-19T03:05:57.511523+0000) 2024-03-19T03:06:03.267 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:03 smithi155 systemd[1]: Stopping Ceph osd.11 for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T03:06:03.767 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:03 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:03.408+0000 7f0a8d0cd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:06:03.767 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:03 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:03.408+0000 7f0a8d0cd700 -1 osd.11 112 *** Got signal Terminated *** 2024-03-19T03:06:03.767 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:03 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:03.408+0000 7f0a8d0cd700 -1 osd.11 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T03:06:04.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:04 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:04.025+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:06:04.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:04 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:04.025+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:06:04.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:04 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:04.025+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:06:04.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:04 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:04.025+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:06:04.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:04 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:04.025+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:16.909958+0000 front 2024-03-19T03:05:16.910060+0000 (oldest deadline 2024-03-19T03:05:42.810008+0000) 2024-03-19T03:06:04.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:04 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:04.025+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-03-19T03:05:22.810471+0000 front 2024-03-19T03:05:22.810439+0000 (oldest deadline 2024-03-19T03:05:54.610993+0000) 2024-03-19T03:06:04.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:04 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:04.025+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6820 osd.7 since back 2024-03-19T03:05:34.611667+0000 front 2024-03-19T03:05:34.611611+0000 (oldest deadline 2024-03-19T03:05:57.511523+0000) 2024-03-19T03:06:05.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:05 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:05.057+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:06:05.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:05 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:05.057+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:06:05.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:05 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:05.057+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:06:05.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:05 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:05.057+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:06:05.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:05 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:05.057+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:16.909958+0000 front 2024-03-19T03:05:16.910060+0000 (oldest deadline 2024-03-19T03:05:42.810008+0000) 2024-03-19T03:06:05.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:05 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:05.057+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-03-19T03:05:22.810471+0000 front 2024-03-19T03:05:22.810439+0000 (oldest deadline 2024-03-19T03:05:54.610993+0000) 2024-03-19T03:06:05.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:05 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:05.057+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6820 osd.7 since back 2024-03-19T03:05:34.611667+0000 front 2024-03-19T03:05:34.611611+0000 (oldest deadline 2024-03-19T03:05:57.511523+0000) 2024-03-19T03:06:06.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:06 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:06.082+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:06:06.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:06 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:06.082+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:06:06.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:06 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:06.082+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:06:06.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:06 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:06.082+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:06:06.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:06 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:06.082+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:16.909958+0000 front 2024-03-19T03:05:16.910060+0000 (oldest deadline 2024-03-19T03:05:42.810008+0000) 2024-03-19T03:06:06.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:06 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:06.082+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-03-19T03:05:22.810471+0000 front 2024-03-19T03:05:22.810439+0000 (oldest deadline 2024-03-19T03:05:54.610993+0000) 2024-03-19T03:06:06.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:06 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:06.082+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6820 osd.7 since back 2024-03-19T03:05:34.611667+0000 front 2024-03-19T03:05:34.611611+0000 (oldest deadline 2024-03-19T03:05:57.511523+0000) 2024-03-19T03:06:07.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:07 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:07.058+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:06:07.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:07 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:07.058+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:06:07.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:07 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:07.058+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:06:07.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:07 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:07.058+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:06:07.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:07 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:07.058+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:16.909958+0000 front 2024-03-19T03:05:16.910060+0000 (oldest deadline 2024-03-19T03:05:42.810008+0000) 2024-03-19T03:06:07.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:07 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:07.058+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-03-19T03:05:22.810471+0000 front 2024-03-19T03:05:22.810439+0000 (oldest deadline 2024-03-19T03:05:54.610993+0000) 2024-03-19T03:06:07.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:07 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:07.058+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6820 osd.7 since back 2024-03-19T03:05:34.611667+0000 front 2024-03-19T03:05:34.611611+0000 (oldest deadline 2024-03-19T03:05:57.511523+0000) 2024-03-19T03:06:08.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:08 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:08.072+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6806 osd.0 since back 2024-03-19T03:04:37.204736+0000 front 2024-03-19T03:04:37.204667+0000 (oldest deadline 2024-03-19T03:05:01.304623+0000) 2024-03-19T03:06:08.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:08 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:08.072+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6804 osd.1 since back 2024-03-19T03:04:46.605869+0000 front 2024-03-19T03:04:46.605930+0000 (oldest deadline 2024-03-19T03:05:09.406142+0000) 2024-03-19T03:06:08.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:08 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:08.072+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6804 osd.2 since back 2024-03-19T03:04:52.807733+0000 front 2024-03-19T03:04:52.807854+0000 (oldest deadline 2024-03-19T03:05:18.707889+0000) 2024-03-19T03:06:08.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:08 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:08.072+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6812 osd.4 since back 2024-03-19T03:05:08.109268+0000 front 2024-03-19T03:05:08.109283+0000 (oldest deadline 2024-03-19T03:05:32.809161+0000) 2024-03-19T03:06:08.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:08 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:08.072+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.155:6812 osd.5 since back 2024-03-19T03:05:16.909958+0000 front 2024-03-19T03:05:16.910060+0000 (oldest deadline 2024-03-19T03:05:42.810008+0000) 2024-03-19T03:06:08.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:08 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:08.072+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.33:6822 osd.6 since back 2024-03-19T03:05:22.810471+0000 front 2024-03-19T03:05:22.810439+0000 (oldest deadline 2024-03-19T03:05:54.610993+0000) 2024-03-19T03:06:08.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:08 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11[61296]: 2024-03-19T03:06:08.072+0000 7f0a856b9700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.96:6820 osd.7 since back 2024-03-19T03:05:34.611667+0000 front 2024-03-19T03:05:34.611611+0000 (oldest deadline 2024-03-19T03:05:57.511523+0000) 2024-03-19T03:06:09.193 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:08 smithi155 podman[73047]: 2024-03-19 03:06:08.922758822 +0000 UTC m=+5.639773808 container died 40ba55b9d04770381818382a963fee671db4fad9cffef260d6cff8ce91b674cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11, GIT_CLEAN=True, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-19T03:06:09.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:09 smithi155 podman[73047]: 2024-03-19 03:06:09.291366743 +0000 UTC m=+6.008381729 container cleanup 40ba55b9d04770381818382a963fee671db4fad9cffef260d6cff8ce91b674cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5) 2024-03-19T03:06:09.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:09 smithi155 bash[73047]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11 2024-03-19T03:06:09.912 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:09 smithi155 podman[73065]: 2024-03-19 03:06:09.554447458 +0000 UTC m=+0.628613747 container remove 40ba55b9d04770381818382a963fee671db4fad9cffef260d6cff8ce91b674cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T03:06:10.213 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:09 smithi155 podman[73156]: 2024-03-19 03:06:09.814347204 +0000 UTC m=+0.017618571 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:06:10.213 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:09 smithi155 podman[73156]: 2024-03-19 03:06:09.918347591 +0000 UTC m=+0.121618987 container create 9ca2e0d10641f7b6dc2194235306a9f6137ecdae949e63557c1aee007e3124ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T03:06:10.213 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:10 smithi155 podman[73156]: 2024-03-19 03:06:10.137952423 +0000 UTC m=+0.341223780 container init 9ca2e0d10641f7b6dc2194235306a9f6137ecdae949e63557c1aee007e3124ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True) 2024-03-19T03:06:10.213 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:10 smithi155 podman[73156]: 2024-03-19 03:06:10.141715758 +0000 UTC m=+0.344987112 container start 9ca2e0d10641f7b6dc2194235306a9f6137ecdae949e63557c1aee007e3124ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-19T03:06:10.213 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:10 smithi155 podman[73156]: 2024-03-19 03:06:10.212597451 +0000 UTC m=+0.415868802 container attach 9ca2e0d10641f7b6dc2194235306a9f6137ecdae949e63557c1aee007e3124ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-19T03:06:10.517 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:10 smithi155 podman[73156]: 2024-03-19 03:06:10.433205896 +0000 UTC m=+0.636477247 container died 9ca2e0d10641f7b6dc2194235306a9f6137ecdae949e63557c1aee007e3124ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T03:06:11.039 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.11.service' 2024-03-19T03:06:11.064 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:11 smithi155 podman[73182]: 2024-03-19 03:06:11.024038214 +0000 UTC m=+0.585419285 container remove 9ca2e0d10641f7b6dc2194235306a9f6137ecdae949e63557c1aee007e3124ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T03:06:11.065 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:11 smithi155 systemd[1]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.11.service: Deactivated successfully. 2024-03-19T03:06:11.065 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:11 smithi155 systemd[1]: Stopped Ceph osd.11 for 2b7a8938-e59a-11ee-95c9-87774f69a715. 2024-03-19T03:06:11.065 INFO:journalctl@ceph.osd.11.smithi155.stdout:Mar 19 03:06:11 smithi155 systemd[1]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@osd.11.service: Consumed 30.843s CPU time. 2024-03-19T03:06:11.080 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:06:11.080 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-19T03:06:11.080 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-19T03:06:11.080 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.a 2024-03-19T03:06:11.377 INFO:journalctl@ceph.mds.a.smithi033.stdout:Mar 19 03:06:11 smithi033 systemd[1]: Stopping Ceph mds.a for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T03:06:11.733 INFO:journalctl@ceph.mds.a.smithi033.stdout:Mar 19 03:06:11 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-a[73836]: 2024-03-19T03:06:11.374+0000 7f0ff5995700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:06:11.734 INFO:journalctl@ceph.mds.a.smithi033.stdout:Mar 19 03:06:11 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-a[73836]: 2024-03-19T03:06:11.374+0000 7f0ff5995700 -1 mds.a *** got signal Terminated *** 2024-03-19T03:06:12.779 INFO:journalctl@ceph.mds.a.smithi033.stdout:Mar 19 03:06:12 smithi033 podman[102404]: 2024-03-19 03:06:12.729139901 +0000 UTC m=+1.442271129 container died 4b3e4c2dbe2052015d1f9f1c0cfb27f97035c965bf34418708ec3f5ec7bb5806 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux ) 2024-03-19T03:06:13.279 INFO:journalctl@ceph.mds.a.smithi033.stdout:Mar 19 03:06:13 smithi033 podman[102404]: 2024-03-19 03:06:13.100352307 +0000 UTC m=+1.813483577 container cleanup 4b3e4c2dbe2052015d1f9f1c0cfb27f97035c965bf34418708ec3f5ec7bb5806 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-a, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-19T03:06:13.279 INFO:journalctl@ceph.mds.a.smithi033.stdout:Mar 19 03:06:13 smithi033 bash[102404]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-a 2024-03-19T03:06:13.532 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.a.service' 2024-03-19T03:06:13.571 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:06:13.571 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-19T03:06:13.571 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-19T03:06:13.571 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.d 2024-03-19T03:06:13.888 INFO:journalctl@ceph.mds.d.smithi033.stdout:Mar 19 03:06:13 smithi033 systemd[1]: Stopping Ceph mds.d for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T03:06:14.279 INFO:journalctl@ceph.mds.d.smithi033.stdout:Mar 19 03:06:13 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-d[72884]: 2024-03-19T03:06:13.887+0000 7f005e3c9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:06:14.279 INFO:journalctl@ceph.mds.d.smithi033.stdout:Mar 19 03:06:13 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-d[72884]: 2024-03-19T03:06:13.887+0000 7f005e3c9700 -1 mds.d *** got signal Terminated *** 2024-03-19T03:06:17.121 INFO:journalctl@ceph.mds.d.smithi033.stdout:Mar 19 03:06:16 smithi033 podman[102535]: 2024-03-19 03:06:16.810352717 +0000 UTC m=+3.014269603 container died fb33411f4f2db1d6ce6a0e823652a54fa2086ded7737949f63daedc0c51972fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-d, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T03:06:17.121 INFO:journalctl@ceph.mds.d.smithi033.stdout:Mar 19 03:06:17 smithi033 podman[102535]: 2024-03-19 03:06:17.120379745 +0000 UTC m=+3.324296630 container cleanup fb33411f4f2db1d6ce6a0e823652a54fa2086ded7737949f63daedc0c51972fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-19T03:06:17.434 INFO:journalctl@ceph.mds.d.smithi033.stdout:Mar 19 03:06:17 smithi033 bash[102535]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-d 2024-03-19T03:06:17.434 INFO:journalctl@ceph.mds.d.smithi033.stdout:Mar 19 03:06:17 smithi033 podman[102549]: 2024-03-19 03:06:17.321668441 +0000 UTC m=+0.508395541 container remove fb33411f4f2db1d6ce6a0e823652a54fa2086ded7737949f63daedc0c51972fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-19T03:06:17.451 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.d.service' 2024-03-19T03:06:17.486 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:06:17.486 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-19T03:06:17.486 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-19T03:06:17.486 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.g 2024-03-19T03:06:17.779 INFO:journalctl@ceph.mds.g.smithi033.stdout:Mar 19 03:06:17 smithi033 systemd[1]: Stopping Ceph mds.g for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T03:06:18.279 INFO:journalctl@ceph.mds.g.smithi033.stdout:Mar 19 03:06:17 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-g[73490]: 2024-03-19T03:06:17.802+0000 7f29eef2d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:06:18.279 INFO:journalctl@ceph.mds.g.smithi033.stdout:Mar 19 03:06:17 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-g[73490]: 2024-03-19T03:06:17.802+0000 7f29eef2d700 -1 mds.g *** got signal Terminated *** 2024-03-19T03:06:22.067 INFO:journalctl@ceph.mds.g.smithi033.stdout:Mar 19 03:06:21 smithi033 podman[102658]: 2024-03-19 03:06:21.813660802 +0000 UTC m=+4.112101681 container died d62af71f141093b3d3e6eef5f216257908dcf8fed9baa4c28d1e088ebc440c95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-g, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-19T03:06:22.426 INFO:journalctl@ceph.mds.g.smithi033.stdout:Mar 19 03:06:22 smithi033 podman[102658]: 2024-03-19 03:06:22.17449915 +0000 UTC m=+4.472940033 container cleanup d62af71f141093b3d3e6eef5f216257908dcf8fed9baa4c28d1e088ebc440c95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-g, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T03:06:22.426 INFO:journalctl@ceph.mds.g.smithi033.stdout:Mar 19 03:06:22 smithi033 bash[102658]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-g 2024-03-19T03:06:22.598 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.g.service' 2024-03-19T03:06:22.632 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:06:22.632 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-19T03:06:22.632 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-19T03:06:22.632 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.j 2024-03-19T03:06:23.029 INFO:journalctl@ceph.mds.j.smithi033.stdout:Mar 19 03:06:22 smithi033 systemd[1]: Stopping Ceph mds.j for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T03:06:23.529 INFO:journalctl@ceph.mds.j.smithi033.stdout:Mar 19 03:06:23 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-j[72494]: 2024-03-19T03:06:23.061+0000 7f21fc6ba700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:06:23.529 INFO:journalctl@ceph.mds.j.smithi033.stdout:Mar 19 03:06:23 smithi033 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-j[72494]: 2024-03-19T03:06:23.061+0000 7f21fc6ba700 -1 mds.j *** got signal Terminated *** 2024-03-19T03:06:26.430 INFO:journalctl@ceph.mds.j.smithi033.stdout:Mar 19 03:06:26 smithi033 podman[102780]: 2024-03-19 03:06:26.169484381 +0000 UTC m=+3.317699360 container died deeb835197b670b7212587fbe8589c99e8b85a812c1fc46ada0d21cb5dffcfda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-j, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-19T03:06:26.688 INFO:journalctl@ceph.mds.j.smithi033.stdout:Mar 19 03:06:26 smithi033 podman[102780]: 2024-03-19 03:06:26.50299164 +0000 UTC m=+3.651206627 container cleanup deeb835197b670b7212587fbe8589c99e8b85a812c1fc46ada0d21cb5dffcfda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T03:06:26.688 INFO:journalctl@ceph.mds.j.smithi033.stdout:Mar 19 03:06:26 smithi033 bash[102780]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-j 2024-03-19T03:06:26.924 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.j.service' 2024-03-19T03:06:26.947 INFO:journalctl@ceph.mds.j.smithi033.stdout:Mar 19 03:06:26 smithi033 podman[102794]: 2024-03-19 03:06:26.687454654 +0000 UTC m=+0.511145699 container remove deeb835197b670b7212587fbe8589c99e8b85a812c1fc46ada0d21cb5dffcfda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-19T03:06:26.947 INFO:journalctl@ceph.mds.j.smithi033.stdout:Mar 19 03:06:26 smithi033 systemd[1]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.j.service: Deactivated successfully. 2024-03-19T03:06:26.948 INFO:journalctl@ceph.mds.j.smithi033.stdout:Mar 19 03:06:26 smithi033 systemd[1]: Stopped Ceph mds.j for 2b7a8938-e59a-11ee-95c9-87774f69a715. 2024-03-19T03:06:26.948 INFO:journalctl@ceph.mds.j.smithi033.stdout:Mar 19 03:06:26 smithi033 systemd[1]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.j.service: Consumed 1.083s CPU time. 2024-03-19T03:06:26.957 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:06:26.958 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-19T03:06:26.958 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-19T03:06:26.958 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.b 2024-03-19T03:06:27.310 INFO:journalctl@ceph.mds.b.smithi096.stdout:Mar 19 03:06:27 smithi096 systemd[1]: Stopping Ceph mds.b for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T03:06:27.716 INFO:journalctl@ceph.mds.b.smithi096.stdout:Mar 19 03:06:27 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-b[65459]: 2024-03-19T03:06:27.307+0000 7f06489ba700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:06:27.716 INFO:journalctl@ceph.mds.b.smithi096.stdout:Mar 19 03:06:27 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-b[65459]: 2024-03-19T03:06:27.307+0000 7f06489ba700 -1 mds.b *** got signal Terminated *** 2024-03-19T03:06:31.127 INFO:journalctl@ceph.mds.b.smithi096.stdout:Mar 19 03:06:30 smithi096 podman[72008]: 2024-03-19 03:06:30.844590217 +0000 UTC m=+3.677132915 container died a09bdf94d17e0c7a9e940c0065916e55bfd61c2039eba3d1fdabd8938242f271 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-b, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-19T03:06:31.466 INFO:journalctl@ceph.mds.b.smithi096.stdout:Mar 19 03:06:31 smithi096 podman[72008]: 2024-03-19 03:06:31.233458078 +0000 UTC m=+4.066000773 container cleanup a09bdf94d17e0c7a9e940c0065916e55bfd61c2039eba3d1fdabd8938242f271 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-b, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T03:06:31.466 INFO:journalctl@ceph.mds.b.smithi096.stdout:Mar 19 03:06:31 smithi096 bash[72008]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-b 2024-03-19T03:06:31.652 DEBUG:teuthology.orchestra.run.smithi096:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.b.service' 2024-03-19T03:06:31.690 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:06:31.690 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-19T03:06:31.691 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-19T03:06:31.691 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.e 2024-03-19T03:06:32.216 INFO:journalctl@ceph.mds.e.smithi096.stdout:Mar 19 03:06:31 smithi096 systemd[1]: Stopping Ceph mds.e for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T03:06:32.216 INFO:journalctl@ceph.mds.e.smithi096.stdout:Mar 19 03:06:32 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-e[64849]: 2024-03-19T03:06:32.011+0000 7f99ce522700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:06:32.216 INFO:journalctl@ceph.mds.e.smithi096.stdout:Mar 19 03:06:32 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-e[64849]: 2024-03-19T03:06:32.011+0000 7f99ce522700 -1 mds.e *** got signal Terminated *** 2024-03-19T03:06:35.165 INFO:journalctl@ceph.mds.e.smithi096.stdout:Mar 19 03:06:34 smithi096 podman[72128]: 2024-03-19 03:06:34.841545241 +0000 UTC m=+2.924055450 container died 8c5733c6cf79fe83ebc3b301cb3074ae25f15f9e1e9f5ed0ac63368afe6bbc0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-e, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T03:06:35.466 INFO:journalctl@ceph.mds.e.smithi096.stdout:Mar 19 03:06:35 smithi096 podman[72128]: 2024-03-19 03:06:35.281058175 +0000 UTC m=+3.363568381 container cleanup 8c5733c6cf79fe83ebc3b301cb3074ae25f15f9e1e9f5ed0ac63368afe6bbc0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-e, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-19T03:06:35.466 INFO:journalctl@ceph.mds.e.smithi096.stdout:Mar 19 03:06:35 smithi096 bash[72128]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-e 2024-03-19T03:06:35.696 DEBUG:teuthology.orchestra.run.smithi096:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.e.service' 2024-03-19T03:06:35.734 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:06:35.734 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-19T03:06:35.734 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-19T03:06:35.735 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.h 2024-03-19T03:06:36.062 INFO:journalctl@ceph.mds.h.smithi096.stdout:Mar 19 03:06:35 smithi096 systemd[1]: Stopping Ceph mds.h for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T03:06:36.466 INFO:journalctl@ceph.mds.h.smithi096.stdout:Mar 19 03:06:36 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-h[65157]: 2024-03-19T03:06:36.060+0000 7f6d235ef700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:06:36.466 INFO:journalctl@ceph.mds.h.smithi096.stdout:Mar 19 03:06:36 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-h[65157]: 2024-03-19T03:06:36.060+0000 7f6d235ef700 -1 mds.h *** got signal Terminated *** 2024-03-19T03:06:39.111 INFO:journalctl@ceph.mds.h.smithi096.stdout:Mar 19 03:06:38 smithi096 podman[72260]: 2024-03-19 03:06:38.84322005 +0000 UTC m=+2.890149981 container died d02dd865bd78e04bef096350c803942dd7c8bd6b751d4ca5fbe1f1cf2cfe8e43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-h, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-19T03:06:39.466 INFO:journalctl@ceph.mds.h.smithi096.stdout:Mar 19 03:06:39 smithi096 podman[72260]: 2024-03-19 03:06:39.201507313 +0000 UTC m=+3.248437297 container cleanup d02dd865bd78e04bef096350c803942dd7c8bd6b751d4ca5fbe1f1cf2cfe8e43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-h, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-19T03:06:39.466 INFO:journalctl@ceph.mds.h.smithi096.stdout:Mar 19 03:06:39 smithi096 bash[72260]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-h 2024-03-19T03:06:39.679 DEBUG:teuthology.orchestra.run.smithi096:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.h.service' 2024-03-19T03:06:39.715 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:06:39.715 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-19T03:06:39.715 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-19T03:06:39.716 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.k 2024-03-19T03:06:40.082 INFO:journalctl@ceph.mds.k.smithi096.stdout:Mar 19 03:06:39 smithi096 systemd[1]: Stopping Ceph mds.k for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T03:06:40.466 INFO:journalctl@ceph.mds.k.smithi096.stdout:Mar 19 03:06:40 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-k[64528]: 2024-03-19T03:06:40.079+0000 7f66deb4b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:06:40.466 INFO:journalctl@ceph.mds.k.smithi096.stdout:Mar 19 03:06:40 smithi096 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-k[64528]: 2024-03-19T03:06:40.079+0000 7f66deb4b700 -1 mds.k *** got signal Terminated *** 2024-03-19T03:06:43.466 INFO:journalctl@ceph.mds.k.smithi096.stdout:Mar 19 03:06:43 smithi096 podman[72381]: 2024-03-19 03:06:43.20526422 +0000 UTC m=+3.278796534 container died b6f149b36ab370a3c8d993026ef26705a6d08f861c1bc99996279d20fa6ebf8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-k, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-19T03:06:43.823 INFO:journalctl@ceph.mds.k.smithi096.stdout:Mar 19 03:06:43 smithi096 podman[72381]: 2024-03-19 03:06:43.55496134 +0000 UTC m=+3.628493653 container cleanup b6f149b36ab370a3c8d993026ef26705a6d08f861c1bc99996279d20fa6ebf8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-19T03:06:43.823 INFO:journalctl@ceph.mds.k.smithi096.stdout:Mar 19 03:06:43 smithi096 bash[72381]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-k 2024-03-19T03:06:43.963 DEBUG:teuthology.orchestra.run.smithi096:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.k.service' 2024-03-19T03:06:43.998 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:06:43.999 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-19T03:06:43.999 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-19T03:06:43.999 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.c 2024-03-19T03:06:44.308 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 19 03:06:44 smithi155 systemd[1]: Stopping Ceph mds.c for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T03:06:44.566 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 19 03:06:44 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-c[66446]: 2024-03-19T03:06:44.306+0000 7fa8f226d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:06:44.566 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 19 03:06:44 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-c[66446]: 2024-03-19T03:06:44.306+0000 7fa8f226d700 -1 mds.c *** got signal Terminated *** 2024-03-19T03:06:47.116 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 19 03:06:46 smithi155 podman[73261]: 2024-03-19 03:06:46.845023259 +0000 UTC m=+2.650281088 container died b6f010f3fa3f0f9ef09dc6674cd81105a9d4d58e479bd90fbd77391726e6f423 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True) 2024-03-19T03:06:47.475 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 19 03:06:47 smithi155 podman[73261]: 2024-03-19 03:06:47.215102099 +0000 UTC m=+3.020359935 container cleanup b6f010f3fa3f0f9ef09dc6674cd81105a9d4d58e479bd90fbd77391726e6f423 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T03:06:47.475 INFO:journalctl@ceph.mds.c.smithi155.stdout:Mar 19 03:06:47 smithi155 bash[73261]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-c 2024-03-19T03:06:47.633 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.c.service' 2024-03-19T03:06:47.671 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:06:47.671 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-19T03:06:47.671 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-19T03:06:47.672 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.f 2024-03-19T03:06:47.995 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 19 03:06:47 smithi155 systemd[1]: Stopping Ceph mds.f for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T03:06:48.266 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 19 03:06:47 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-f[66140]: 2024-03-19T03:06:47.993+0000 7f2ad498d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:06:48.267 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 19 03:06:47 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-f[66140]: 2024-03-19T03:06:47.993+0000 7f2ad498d700 -1 mds.f *** got signal Terminated *** 2024-03-19T03:06:51.103 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 19 03:06:50 smithi155 podman[73395]: 2024-03-19 03:06:50.844949603 +0000 UTC m=+2.959027501 container died 3b39aaf94f1ae1f636a573c0b0fb761c1499b25145fbbea728475deae1bf5bb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-f, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-19T03:06:51.516 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 19 03:06:51 smithi155 podman[73395]: 2024-03-19 03:06:51.160124375 +0000 UTC m=+3.274202314 container cleanup 3b39aaf94f1ae1f636a573c0b0fb761c1499b25145fbbea728475deae1bf5bb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-f, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T03:06:51.516 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 19 03:06:51 smithi155 bash[73395]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-f 2024-03-19T03:06:51.516 INFO:journalctl@ceph.mds.f.smithi155.stdout:Mar 19 03:06:51 smithi155 podman[73409]: 2024-03-19 03:06:51.327716359 +0000 UTC m=+0.485030243 container remove 3b39aaf94f1ae1f636a573c0b0fb761c1499b25145fbbea728475deae1bf5bb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux ) 2024-03-19T03:06:51.531 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.f.service' 2024-03-19T03:06:51.567 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:06:51.567 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-19T03:06:51.567 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-19T03:06:51.567 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.i 2024-03-19T03:06:51.889 INFO:journalctl@ceph.mds.i.smithi155.stdout:Mar 19 03:06:51 smithi155 systemd[1]: Stopping Ceph mds.i for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T03:06:52.267 INFO:journalctl@ceph.mds.i.smithi155.stdout:Mar 19 03:06:51 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-i[66753]: 2024-03-19T03:06:51.887+0000 7f51734ec700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:06:52.267 INFO:journalctl@ceph.mds.i.smithi155.stdout:Mar 19 03:06:51 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-i[66753]: 2024-03-19T03:06:51.887+0000 7f51734ec700 -1 mds.i *** got signal Terminated *** 2024-03-19T03:06:55.123 INFO:journalctl@ceph.mds.i.smithi155.stdout:Mar 19 03:06:54 smithi155 podman[73518]: 2024-03-19 03:06:54.838351623 +0000 UTC m=+3.059213418 container died 3558562598b9b8bbd6737cb2ae39d607eed6d1f6a57f7e02d67f637110dd0505 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-i, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=) 2024-03-19T03:06:55.465 INFO:journalctl@ceph.mds.i.smithi155.stdout:Mar 19 03:06:55 smithi155 podman[73518]: 2024-03-19 03:06:55.22198807 +0000 UTC m=+3.442849865 container cleanup 3558562598b9b8bbd6737cb2ae39d607eed6d1f6a57f7e02d67f637110dd0505 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-i, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T03:06:55.465 INFO:journalctl@ceph.mds.i.smithi155.stdout:Mar 19 03:06:55 smithi155 bash[73518]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-i 2024-03-19T03:06:55.590 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.i.service' 2024-03-19T03:06:55.625 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:06:55.625 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-19T03:06:55.625 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-19T03:06:55.625 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.l 2024-03-19T03:06:55.767 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 19 03:06:55 smithi155 systemd[1]: Stopping Ceph mds.l for 2b7a8938-e59a-11ee-95c9-87774f69a715... 2024-03-19T03:06:56.267 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 19 03:06:55 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-l[65832]: 2024-03-19T03:06:55.925+0000 7f873350d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:06:56.267 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 19 03:06:55 smithi155 ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-l[65832]: 2024-03-19T03:06:55.925+0000 7f873350d700 -1 mds.l *** got signal Terminated *** 2024-03-19T03:07:00.267 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 19 03:06:59 smithi155 podman[73639]: 2024-03-19 03:06:59.931286887 +0000 UTC m=+4.089561386 container died 0c647f0e69330a905303cd215ef449acf725d25b152b54998b3dcbf5b2140cf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-l, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux ) 2024-03-19T03:07:00.571 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 19 03:07:00 smithi155 podman[73639]: 2024-03-19 03:07:00.442263883 +0000 UTC m=+4.600538391 container cleanup 0c647f0e69330a905303cd215ef449acf725d25b152b54998b3dcbf5b2140cf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-l, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-19T03:07:00.571 INFO:journalctl@ceph.mds.l.smithi155.stdout:Mar 19 03:07:00 smithi155 bash[73639]: ceph-2b7a8938-e59a-11ee-95c9-87774f69a715-mds-l 2024-03-19T03:07:00.795 DEBUG:teuthology.orchestra.run.smithi155:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b7a8938-e59a-11ee-95c9-87774f69a715@mds.l.service' 2024-03-19T03:07:00.829 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:07:00.829 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-19T03:07:00.829 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-19T03:07:01.027 INFO:teuthology.orchestra.run.smithi033.stdout:Deleting cluster with fsid: 2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T03:07:03.428 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-19T03:07:03.624 INFO:teuthology.orchestra.run.smithi096.stdout:Deleting cluster with fsid: 2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T03:07:06.100 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-19T03:07:06.294 INFO:teuthology.orchestra.run.smithi155.stdout:Deleting cluster with fsid: 2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T03:07:08.720 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-19T03:07:08.755 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-19T03:07:08.790 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-19T03:07:08.818 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-19T03:07:08.820 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609408/remote/smithi033/crash 2024-03-19T03:07:08.821 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/crash -- . 2024-03-19T03:07:08.866 INFO:teuthology.orchestra.run.smithi033.stderr:tar: /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-19T03:07:08.866 INFO:teuthology.orchestra.run.smithi033.stderr:tar: Error is not recoverable: exiting now 2024-03-19T03:07:08.868 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609408/remote/smithi096/crash 2024-03-19T03:07:08.869 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/crash -- . 2024-03-19T03:07:08.921 INFO:teuthology.orchestra.run.smithi096.stderr:tar: /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-19T03:07:08.921 INFO:teuthology.orchestra.run.smithi096.stderr:tar: Error is not recoverable: exiting now 2024-03-19T03:07:08.923 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609408/remote/smithi155/crash 2024-03-19T03:07:08.924 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/crash -- . 2024-03-19T03:07:08.957 INFO:teuthology.orchestra.run.smithi155.stderr:tar: /var/lib/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-19T03:07:08.957 INFO:teuthology.orchestra.run.smithi155.stderr:tar: Error is not recoverable: exiting now 2024-03-19T03:07:08.958 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-19T03:07:08.958 DEBUG:teuthology.orchestra.run.smithi033:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-19T03:07:08.992 INFO:teuthology.orchestra.run.smithi033.stdout:2024-03-19T02:44:40.661952+0000 mon.a (mon.0) 210 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-19T03:07:08.992 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-19T03:07:08.993 DEBUG:teuthology.orchestra.run.smithi033:> sudo egrep '\[SEC\]' /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-19T03:07:09.066 DEBUG:teuthology.orchestra.run.smithi033:> sudo egrep '\[ERR\]' /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-19T03:07:09.139 DEBUG:teuthology.orchestra.run.smithi033:> sudo egrep '\[WRN\]' /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-19T03:07:09.212 INFO:teuthology.orchestra.run.smithi033.stdout:2024-03-19T02:44:40.661952+0000 mon.a (mon.0) 210 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-19T03:07:09.212 INFO:tasks.cephadm:Compressing logs... 2024-03-19T03:07:09.212 DEBUG:teuthology.orchestra.run.smithi033:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-19T03:07:09.255 DEBUG:teuthology.orchestra.run.smithi096:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-19T03:07:09.258 DEBUG:teuthology.orchestra.run.smithi155:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-19T03:07:09.281 INFO:teuthology.orchestra.run.smithi033.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-19T03:07:09.281 INFO:teuthology.orchestra.run.smithi033.stderr:: No such file or directory 2024-03-19T03:07:09.281 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2024-03-19T03:07:09.281 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-03-19T03:07:09.281 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.0.log: gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.6.log 2024-03-19T03:07:09.282 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.log: gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mon.a.log 2024-03-19T03:07:09.282 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-19T03:07:09.282 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.6.log: /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.g.log 2024-03-19T03:07:09.282 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.j.log 2024-03-19T03:07:09.282 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.3.log 2024-03-19T03:07:09.282 INFO:teuthology.orchestra.run.smithi033.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-03-19T03:07:09.282 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.j.log: 90.2% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-19T03:07:09.284 INFO:teuthology.orchestra.run.smithi155.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-19T03:07:09.284 INFO:teuthology.orchestra.run.smithi155.stderr:: No such file or directory 2024-03-19T03:07:09.285 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.l.log 2024-03-19T03:07:09.285 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-19T03:07:09.285 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.i.log 2024-03-19T03:07:09.287 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.11.log 2024-03-19T03:07:09.287 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mgr.z.log 2024-03-19T03:07:09.287 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.i.log: 92.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-19T03:07:09.287 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.8.log 2024-03-19T03:07:09.287 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.5.log 2024-03-19T03:07:09.287 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mgr.z.log: /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.f.log 2024-03-19T03:07:09.294 INFO:teuthology.orchestra.run.smithi033.stderr: 94.6% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.g.log.gz 2024-03-19T03:07:09.294 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mgr.x.log 2024-03-19T03:07:09.294 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.3.log: 94.5% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.j.log.gz 2024-03-19T03:07:09.294 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.0.log 2024-03-19T03:07:09.294 INFO:teuthology.orchestra.run.smithi033.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-19T03:07:09.294 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-19T03:07:09.294 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph.log 2024-03-19T03:07:09.294 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-19T03:07:09.294 INFO:teuthology.orchestra.run.smithi033.stderr: 84.1%/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph.log: -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-19T03:07:09.294 INFO:teuthology.orchestra.run.smithi033.stderr: 87.4% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-19T03:07:09.294 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.9.log 2024-03-19T03:07:09.294 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.d.log 2024-03-19T03:07:09.297 INFO:teuthology.orchestra.run.smithi096.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-19T03:07:09.297 INFO:teuthology.orchestra.run.smithi096.stderr:: No such file or directory 2024-03-19T03:07:09.297 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.10.log 2024-03-19T03:07:09.297 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-19T03:07:09.297 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.b.log 2024-03-19T03:07:09.297 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.4.log 2024-03-19T03:07:09.297 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.1.log 2024-03-19T03:07:09.297 INFO:teuthology.orchestra.run.smithi096.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-19T03:07:09.298 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.k.log 2024-03-19T03:07:09.298 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.h.log 2024-03-19T03:07:09.298 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.7.log 2024-03-19T03:07:09.298 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.e.log 2024-03-19T03:07:09.298 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.7.log: 90.3% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-19T03:07:09.298 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mgr.y.log 2024-03-19T03:07:09.298 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.e.log: 94.6% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.k.log.gz 2024-03-19T03:07:09.298 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-19T03:07:09.300 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-19T03:07:09.300 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.f.log: 90.3% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-19T03:07:09.300 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph.log 2024-03-19T03:07:09.300 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph.cephadm.log: 83.7% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-19T03:07:09.300 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.2.log 2024-03-19T03:07:09.300 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph.log: 94.7% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.l.log.gz 2024-03-19T03:07:09.300 INFO:teuthology.orchestra.run.smithi155.stderr: 87.5% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-19T03:07:09.300 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.c.log 2024-03-19T03:07:09.300 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-19T03:07:09.300 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.c.log: 91.8% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mgr.z.log.gz 2024-03-19T03:07:09.304 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.a.log 2024-03-19T03:07:09.305 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mgr.y.log: 91.8% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mgr.y.log.gz 2024-03-19T03:07:09.305 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph.log 2024-03-19T03:07:09.306 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-19T03:07:09.306 INFO:teuthology.orchestra.run.smithi096.stderr: 83.8% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-19T03:07:09.309 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph.log: 87.5% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-19T03:07:09.310 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mon.b.log 2024-03-19T03:07:09.310 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.d.log: 94.6% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.d.log.gz 2024-03-19T03:07:09.312 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.a.log: 91.0% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-19T03:07:09.312 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mon.c.log 2024-03-19T03:07:09.316 INFO:teuthology.orchestra.run.smithi033.stderr: 94.6% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.a.log.gz 2024-03-19T03:07:09.324 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mon.b.log: 91.0% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-19T03:07:09.326 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-volume.log: 90.9% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-19T03:07:09.352 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mon.c.log: 94.5% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.f.log.gz 2024-03-19T03:07:09.352 INFO:teuthology.orchestra.run.smithi155.stderr: 92.7% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.i.log.gz 2024-03-19T03:07:09.358 INFO:teuthology.orchestra.run.smithi155.stderr: 94.7% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.c.log.gz 2024-03-19T03:07:09.362 INFO:teuthology.orchestra.run.smithi096.stderr: 92.6% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.h.log.gz 2024-03-19T03:07:09.383 INFO:teuthology.orchestra.run.smithi096.stderr: 93.8% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.e.log.gz 2024-03-19T03:07:09.592 INFO:teuthology.orchestra.run.smithi033.stderr: 89.9% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mgr.x.log.gz 2024-03-19T03:07:09.629 INFO:teuthology.orchestra.run.smithi155.stderr: 92.1% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mon.c.log.gz 2024-03-19T03:07:09.694 INFO:teuthology.orchestra.run.smithi096.stderr: 92.9% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mds.b.log.gz 2024-03-19T03:07:09.779 INFO:teuthology.orchestra.run.smithi096.stderr: 92.0% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mon.b.log.gz 2024-03-19T03:07:10.263 INFO:teuthology.orchestra.run.smithi033.stderr: 90.6% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-mon.a.log.gz 2024-03-19T03:07:13.704 INFO:teuthology.orchestra.run.smithi096.stderr: 92.9% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.10.log.gz 2024-03-19T03:07:14.635 INFO:teuthology.orchestra.run.smithi155.stderr: 93.0% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.11.log.gz 2024-03-19T03:07:14.871 INFO:teuthology.orchestra.run.smithi155.stderr: 93.0% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.8.log.gz 2024-03-19T03:07:15.522 INFO:teuthology.orchestra.run.smithi155.stderr: 92.8% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.2.log.gz 2024-03-19T03:07:15.750 INFO:teuthology.orchestra.run.smithi033.stderr: 92.8% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.0.log.gz 2024-03-19T03:07:15.772 INFO:teuthology.orchestra.run.smithi096.stderr: 92.9% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.1.log.gz 2024-03-19T03:07:16.973 INFO:teuthology.orchestra.run.smithi096.stderr: 93.0% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.7.log.gz 2024-03-19T03:07:17.047 INFO:teuthology.orchestra.run.smithi033.stderr: 92.8% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.3.log.gz 2024-03-19T03:07:17.128 INFO:teuthology.orchestra.run.smithi096.stderr: 92.7% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.4.log.gz 2024-03-19T03:07:17.129 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-03-19T03:07:17.130 INFO:teuthology.orchestra.run.smithi096.stderr:real 0m7.857s 2024-03-19T03:07:17.130 INFO:teuthology.orchestra.run.smithi096.stderr:user 0m26.527s 2024-03-19T03:07:17.130 INFO:teuthology.orchestra.run.smithi096.stderr:sys 0m0.942s 2024-03-19T03:07:18.283 INFO:teuthology.orchestra.run.smithi155.stderr: 92.7% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.5.log.gz 2024-03-19T03:07:18.285 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-03-19T03:07:18.285 INFO:teuthology.orchestra.run.smithi155.stderr:real 0m9.013s 2024-03-19T03:07:18.285 INFO:teuthology.orchestra.run.smithi155.stderr:user 0m25.766s 2024-03-19T03:07:18.285 INFO:teuthology.orchestra.run.smithi155.stderr:sys 0m0.913s 2024-03-19T03:07:19.007 INFO:teuthology.orchestra.run.smithi033.stderr: 92.8% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.6.log.gz 2024-03-19T03:07:19.635 INFO:teuthology.orchestra.run.smithi033.stderr: 96.8% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2024-03-19T03:07:19.696 INFO:teuthology.orchestra.run.smithi033.stderr: 92.8% -- replaced with /var/log/ceph/2b7a8938-e59a-11ee-95c9-87774f69a715/ceph-osd.9.log.gz 2024-03-19T03:07:19.698 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-19T03:07:19.698 INFO:teuthology.orchestra.run.smithi033.stderr:real 0m10.432s 2024-03-19T03:07:19.698 INFO:teuthology.orchestra.run.smithi033.stderr:user 0m44.578s 2024-03-19T03:07:19.698 INFO:teuthology.orchestra.run.smithi033.stderr:sys 0m1.415s 2024-03-19T03:07:19.699 INFO:tasks.cephadm:Archiving logs... 2024-03-19T03:07:19.699 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609408/remote/smithi033/log 2024-03-19T03:07:19.700 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-19T03:07:25.699 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609408/remote/smithi096/log 2024-03-19T03:07:25.700 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-19T03:07:30.144 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609408/remote/smithi155/log 2024-03-19T03:07:30.145 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-19T03:07:34.517 INFO:tasks.cephadm:Removing cluster... 2024-03-19T03:07:34.518 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 --force 2024-03-19T03:07:34.716 INFO:teuthology.orchestra.run.smithi033.stdout:Deleting cluster with fsid: 2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T03:07:35.021 DEBUG:teuthology.orchestra.run.smithi096:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 --force 2024-03-19T03:07:35.212 INFO:teuthology.orchestra.run.smithi096.stdout:Deleting cluster with fsid: 2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T03:07:35.514 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2b7a8938-e59a-11ee-95c9-87774f69a715 --force 2024-03-19T03:07:35.706 INFO:teuthology.orchestra.run.smithi155.stdout:Deleting cluster with fsid: 2b7a8938-e59a-11ee-95c9-87774f69a715 2024-03-19T03:07:35.993 INFO:tasks.cephadm:Removing cephadm ... 2024-03-19T03:07:35.993 DEBUG:teuthology.orchestra.run.smithi033:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-19T03:07:36.010 DEBUG:teuthology.orchestra.run.smithi096:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-19T03:07:36.027 DEBUG:teuthology.orchestra.run.smithi155:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-19T03:07:36.042 INFO:tasks.cephadm:Teardown complete 2024-03-19T03:07:36.043 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-19T03:07:36.055 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-19T03:07:36.055 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-19T03:07:36.058 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-19T03:07:36.069 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-19T03:07:36.171 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-19T03:07:36.182 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-19T03:07:36.182 DEBUG:teuthology.orchestra.run.smithi033:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T03:07:36.184 DEBUG:teuthology.orchestra.run.smithi096:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T03:07:36.186 DEBUG:teuthology.orchestra.run.smithi155:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T03:07:36.195 INFO:teuthology.orchestra.run.smithi033.stderr:bash: line 1: ntpq: command not found 2024-03-19T03:07:36.197 INFO:teuthology.orchestra.run.smithi096.stderr:bash: line 1: ntpq: command not found 2024-03-19T03:07:36.199 INFO:teuthology.orchestra.run.smithi033.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-19T03:07:36.199 INFO:teuthology.orchestra.run.smithi033.stdout:=============================================================================== 2024-03-19T03:07:36.199 INFO:teuthology.orchestra.run.smithi033.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 31 +3067us[+2848us] +/- 108ms 2024-03-19T03:07:36.199 INFO:teuthology.orchestra.run.smithi033.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 32 -1265us[-1485us] +/- 50ms 2024-03-19T03:07:36.199 INFO:teuthology.orchestra.run.smithi033.stdout:^* hv03.front.sepia.ceph.com 3 6 377 31 -309us[ -529us] +/- 74ms 2024-03-19T03:07:36.199 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T03:07:36.200 INFO:teuthology.orchestra.run.smithi155.stderr:bash: line 1: ntpq: command not found 2024-03-19T03:07:36.202 INFO:teuthology.orchestra.run.smithi096.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-19T03:07:36.202 INFO:teuthology.orchestra.run.smithi096.stdout:=============================================================================== 2024-03-19T03:07:36.202 INFO:teuthology.orchestra.run.smithi096.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 31 +3084us[+2878us] +/- 108ms 2024-03-19T03:07:36.202 INFO:teuthology.orchestra.run.smithi096.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 33 -1250us[-1456us] +/- 50ms 2024-03-19T03:07:36.202 INFO:teuthology.orchestra.run.smithi096.stdout:^* hv03.front.sepia.ceph.com 3 6 377 27 -317us[ -523us] +/- 74ms 2024-03-19T03:07:36.202 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T03:07:36.204 INFO:teuthology.orchestra.run.smithi155.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-19T03:07:36.204 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================== 2024-03-19T03:07:36.204 INFO:teuthology.orchestra.run.smithi155.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 27 +3032us[+3032us] +/- 108ms 2024-03-19T03:07:36.204 INFO:teuthology.orchestra.run.smithi155.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 29 -1288us[-1480us] +/- 50ms 2024-03-19T03:07:36.204 INFO:teuthology.orchestra.run.smithi155.stdout:^* hv03.front.sepia.ceph.com 3 6 377 29 -344us[ -535us] +/- 74ms 2024-03-19T03:07:36.204 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T03:07:36.205 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-19T03:07:36.220 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-19T03:07:36.221 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-19T03:07:36.232 DEBUG:teuthology.orchestra.run.smithi033:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-19T03:07:36.325 DEBUG:teuthology.orchestra.run.smithi096:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-19T03:07:36.395 DEBUG:teuthology.orchestra.run.smithi155:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-19T03:07:36.467 DEBUG:teuthology.orchestra.run.smithi033:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-19T03:07:36.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T03:07:36.498 DEBUG:teuthology.orchestra.run.smithi096:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-19T03:07:36.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T03:07:36.528 DEBUG:teuthology.orchestra.run.smithi155:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-19T03:07:36.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T03:07:36.558 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-19T03:07:36.570 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-19T03:07:36.581 INFO:teuthology.task.internal:Duration was 2623.523586 seconds 2024-03-19T03:07:36.581 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-19T03:07:36.592 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-19T03:07:36.593 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-19T03:07:36.595 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-19T03:07:36.598 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-19T03:07:36.636 INFO:teuthology.orchestra.run.smithi033.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-19T03:07:36.638 INFO:teuthology.orchestra.run.smithi096.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-19T03:07:36.640 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-19T03:07:36.998 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-19T03:07:36.998 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi033.front.sepia.ceph.com 2024-03-19T03:07:36.998 DEBUG:teuthology.orchestra.run.smithi033:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-19T03:07:37.018 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi096.front.sepia.ceph.com 2024-03-19T03:07:37.019 DEBUG:teuthology.orchestra.run.smithi096:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-19T03:07:37.059 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi155.front.sepia.ceph.com 2024-03-19T03:07:37.059 DEBUG:teuthology.orchestra.run.smithi155:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-19T03:07:37.086 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-19T03:07:37.086 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-19T03:07:37.088 DEBUG:teuthology.orchestra.run.smithi096:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-19T03:07:37.102 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-19T03:07:37.154 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-19T03:07:37.154 DEBUG:teuthology.orchestra.run.smithi033:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-19T03:07:37.157 DEBUG:teuthology.orchestra.run.smithi096:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-19T03:07:37.170 DEBUG:teuthology.orchestra.run.smithi155:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-19T03:07:37.546 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-19T03:07:37.558 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-19T03:07:37.558 DEBUG:teuthology.orchestra.run.smithi033:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-19T03:07:37.584 DEBUG:teuthology.orchestra.run.smithi096:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-19T03:07:37.611 DEBUG:teuthology.orchestra.run.smithi155:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-19T03:07:37.639 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-19T03:07:37.650 DEBUG:teuthology.orchestra.run.smithi033:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-19T03:07:37.652 DEBUG:teuthology.orchestra.run.smithi096:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-19T03:07:37.654 DEBUG:teuthology.orchestra.run.smithi155:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-19T03:07:37.675 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = core 2024-03-19T03:07:37.677 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern = core 2024-03-19T03:07:37.705 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = core 2024-03-19T03:07:37.739 DEBUG:teuthology.orchestra.run.smithi033:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-19T03:07:37.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T03:07:37.753 DEBUG:teuthology.orchestra.run.smithi096:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-19T03:07:37.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T03:07:37.767 DEBUG:teuthology.orchestra.run.smithi155:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-19T03:07:37.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T03:07:37.793 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-19T03:07:37.805 INFO:teuthology.task.internal:Transferring archived files... 2024-03-19T03:07:37.806 DEBUG:teuthology.misc:Transferring archived files from smithi033:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609408/remote/smithi033 2024-03-19T03:07:37.807 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-19T03:07:37.850 DEBUG:teuthology.misc:Transferring archived files from smithi096:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609408/remote/smithi096 2024-03-19T03:07:37.850 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-19T03:07:37.888 DEBUG:teuthology.misc:Transferring archived files from smithi155:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609408/remote/smithi155 2024-03-19T03:07:37.888 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-19T03:07:37.926 INFO:teuthology.task.internal:Removing archive directory... 2024-03-19T03:07:37.927 DEBUG:teuthology.orchestra.run.smithi033:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-19T03:07:37.929 DEBUG:teuthology.orchestra.run.smithi096:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-19T03:07:37.931 DEBUG:teuthology.orchestra.run.smithi155:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-19T03:07:37.975 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-19T03:07:37.989 INFO:teuthology.task.internal:Not uploading archives. 2024-03-19T03:07:37.989 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-19T03:07:38.000 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-19T03:07:38.001 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-19T03:07:38.003 DEBUG:teuthology.orchestra.run.smithi096:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-19T03:07:38.005 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-19T03:07:38.015 INFO:teuthology.orchestra.run.smithi033.stdout: 265343 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 19 03:07 /home/ubuntu/cephtest 2024-03-19T03:07:38.017 INFO:teuthology.orchestra.run.smithi096.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 19 03:07 /home/ubuntu/cephtest 2024-03-19T03:07:38.029 INFO:teuthology.orchestra.run.smithi155.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 19 03:07 /home/ubuntu/cephtest 2024-03-19T03:07:38.030 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-19T03:07:38.042 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-19T03:07:38.068 INFO:teuthology.nuke:Checking targets against current locks 2024-03-19T03:07:38.092 DEBUG:teuthology.nuke:shortname: smithi033 2024-03-19T03:07:38.093 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-19T03:07:38.115 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi033.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609408', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 02:07:00.733364', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJzq0qnp859Um6mXLmhNb1b49Ng59y4fRCIOrHnHTHcIf6mgk4lvvvbIbPq1uSBQAInIpaTsdJROOs1V07ujsYU='} 2024-03-19T03:07:38.121 DEBUG:teuthology.nuke:shortname: smithi096 2024-03-19T03:07:38.121 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-19T03:07:38.145 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi096.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609408', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 02:07:00.736686', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIhib3wbRbcPlEUnyWqrOG0jfwV6I0Xc3kK/vHWmavG4Dl7nQrkYBBGps05H+an/B7xEk8l0mXnuRd6TmMCyLRw='} 2024-03-19T03:07:38.146 DEBUG:teuthology.nuke:shortname: smithi155 2024-03-19T03:07:38.146 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-19T03:07:38.170 INFO:teuthology.orchestra.console.smithi033:Power off 2024-03-19T03:07:38.171 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-19T03:07:38.182 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi155.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609408', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 02:07:00.735120', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAmfURHQb3AafSy0yHeZWbFutUFquk1+c3wDFUcl75szpwFM2Z41eARRdgpOZE0tdaRbVRlcoWVVw6f0cQs/SAQ='} 2024-03-19T03:07:38.196 DEBUG:teuthology.orchestra.console.smithi033:power off output: Chassis Power Control: Down/Off 2024-03-19T03:07:38.196 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:07:38.213 INFO:teuthology.orchestra.console.smithi096:Power off 2024-03-19T03:07:38.214 DEBUG:teuthology.orchestra.console.smithi096:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-19T03:07:38.226 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is on 2024-03-19T03:07:38.236 DEBUG:teuthology.orchestra.console.smithi096:power off output: Chassis Power Control: Down/Off 2024-03-19T03:07:38.236 DEBUG:teuthology.orchestra.console.smithi096:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:07:38.251 INFO:teuthology.orchestra.console.smithi155:Power off 2024-03-19T03:07:38.251 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-19T03:07:38.262 DEBUG:teuthology.orchestra.console.smithi096:check power output: Chassis Power is on 2024-03-19T03:07:38.275 DEBUG:teuthology.orchestra.console.smithi155:power off output: Chassis Power Control: Down/Off 2024-03-19T03:07:38.275 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:07:38.307 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is on 2024-03-19T03:07:42.228 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:07:42.263 DEBUG:teuthology.orchestra.console.smithi096:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:07:42.307 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:07:42.339 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is on 2024-03-19T03:07:42.374 DEBUG:teuthology.orchestra.console.smithi096:check power output: Chassis Power is on 2024-03-19T03:07:42.418 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is on 2024-03-19T03:07:46.345 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:07:46.374 DEBUG:teuthology.orchestra.console.smithi096:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:07:46.419 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:07:46.456 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is off 2024-03-19T03:07:46.484 DEBUG:teuthology.orchestra.console.smithi096:check power output: Chassis Power is off 2024-03-19T03:07:46.529 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is off 2024-03-19T03:07:46.557 INFO:teuthology.orchestra.console.smithi033:Power off completed 2024-03-19T03:07:46.585 INFO:teuthology.orchestra.console.smithi096:Power off completed 2024-03-19T03:07:46.629 INFO:teuthology.orchestra.console.smithi155:Power off completed 2024-03-19T03:07:46.734 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} duration: 2623.523585796356 failure_reason: '"2024-03-19T02:44:40.661952+0000 mon.a (mon.0) 210 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log' flavor: default owner: scheduled_pdonnell@teuthology success: false 2024-03-19T03:07:46.734 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-19T03:07:46.825 INFO:teuthology.run:FAIL